MiniMed, Newfront Shift AI Budgets to Human Workflows Over Tools

MiniMed与Newfront将AI预算重心从工具转向人工工作流 除非企业在员工队伍重塑与治理方面的投入达到软件许可费用的两倍，否则企业AI的回报将难以显现。 专家小组强调，成功部署AI Agent需要将组织工作流程调整、人力资本和严格治理置于工具费用之上。与会嘉宾分享了具体的部署门槛，例如以反映在损益表（P&L）上的20%至30%财务ROI为目标，并覆盖较大比例的员工群体。为了平衡创新与风险，各公司利用集中的AI指导委员会、快速原型冲刺以及模型路由架构来优化token成本。归根结底，高ROI的用例侧重于增强人类生产力并构建结构化数据基础，从而随着时间的推移产生复合的下游价值。 专家组赞同Gartner的规则，即相对于纯AI软件采购，企业应大力投资于人员、工作流、数据治理和组织重塑（比例达到2:1或更高）。 (0:04) 企业AI指导委员会在评估自下而上的提案时，依据的是其能否解决实际业务问题、提供明确且可衡量的ROI，并在组织治理下安全地扩展规模。 (0:06) 在保险等受监管行业中，团队通过初期避开核心监管领域、引入人机协同（human-in-the-loop）监督，并在1至2周的冲刺中对多达20个想法进行原型设计且设定80%至90%的可接受准确率目标，从而实现快速部署。 (18:41) AI工具的作用在于增强现有的核心软件栈和人力团队，而非淘汰底层企业SaaS系统或取代人类专业经验。 (22:08) 在新闻业等以人际信任为核心的领域中，利用AI来处理繁杂的运营事务和背景调研，以便人类能专注于核心能力。 (23:38)

Enterprise leaders across insurance, publishing, and healthcare argue that high-ROI AI deployments depend far more on data hygiene, workflow redesign, and human oversight than on tooling alone. To deliver measurable P&L returns, organizations are pairing centralized governance with rapid, low-risk pilot testing. Prioritizing Human Capital and Workflow Redesign Over Tooling Panelists concurred with a Gartner report suggesting organizations spend twice as much on data structuring, governance, and people as on AI software. Devin Stevens (Scrunch) observed that AI initiatives fail primarily when teams neglect workflow adaptation and change management, not because the underlying technology breaks. Muthu Sabarethinam (MiniMed) pointed out that dropping software prices make talent and organizational redesign the higher expense, driving a transition toward hybrid roles such as 'product engineers.' Centralized Governance and P&L Deployment Thresholds (25:22) To prevent tool sprawl and mitigate regulatory and brand risks, enterprises rely on centralized steering bodies. Bhaskar Borah explained that Dow Jones filters bottom-up proposals through an executive investing committee focused on real operational bottlenecks, measurable returns, and data safety before procurement. For regulated environments, Lin Yuan stated that Newfront Insurance avoids core state-level regulatory areas during early rollouts, using 1- to 2-week sprints to prototype up to 20 ideas that require human-in-the-loop validation at acceptable accuracy targets of 80% to 90%. Financial thresholds must directly alter financial statements rather than rely on soft productivity metrics. MiniMed enforces a 20% to 30% minimum ROI hurdle rate and mandates that initiatives touch at least 50% of the workforce (5,000+ employees) to justify enterprise rollout. Stevens added that back-office cost reductions carry far higher credibility in ROI models than vendors' projected top-line revenue growth. Build Versus Buy and Controlling Variable Compute Costs Panelists rejected the narrative that generative AI will eliminate core SaaS stacks, noting that systems like CRMs and financial software remain essential infrastructure. On procurement, Stevens cited an MIT study finding commercial AI software purchases achieved twice the implementation success rate of internal builds due to organizational focus and vendor support. Borah noted Dow Jones licenses proprietary archives to hyperscalers for model training, but builds custom internal tools whenever off-the-shelf software risks brand credibility or human editorial oversight. To control unpredictable inference costs, Sabarethinam described routing simple enterprise queries to self-hosted models while reserving costly reasoning models for complex logic, reducing compute overhead by 30% to 40%. Yuan further recommended funneling vendor LLM calls through a centralized enterprise API layer to monitor usage and curb unmonitored API expenses. Unlocking Compounding Value Beyond Initial Targets (40:32) The panel highlighted that successful implementations frequently yield secondary, unexpected returns once a structured data baseline exists. At Newfront, initial AI agents built to ingest unstructured documents created a structured data foundation that subsequently powered deterministic, non-LLM workflows without additional inference costs. Similarly, Stevens noted that automated meeting note-taking originally meant to save personal time ended up feeding customer health scoring models and revealing gaps in internal support documentation, while Borah reported that Dow Jones's Customer 360 project gained cross-departmental adoption well beyond its original scope. The panel concluded that sustainable AI adoption requires initiatives that reach wide segments of the workforce, protect core competencies, and demonstrate tangible P&L impact. However, maintaining cost controls remains contingent on strict architectural oversight, dynamic model routing, and centralized API governance as usage scales across business units.

在Ai4组织的高管研讨中，来自保险、传媒、医疗和企业软件领域的行业领袖指出，企业获取高投资回报率（ROI）AI应用的关键，在于将至少两倍于工具采购的资源投入到数据治理、工作流重组和人机协同机制上，而非盲目追求全自动化的工具堆砌。 治理优先与双倍人员流程投入 研讨会嘉宾一致认同Gartner关于企业在数据结构化、治理和人才上的投入应达到纯AI工具两倍的建议。Scrunch的Devin Stevens指出，AI工具本身无法独立运转，企业项目的失败通常不是因为技术故障，而是未能及时调整工作流或有效推进组织采纳。MiniMed的Muthu Sabarethinam补充道，在底层软件价格快速下滑的背景下，组织重构与人才培养构成了主要支出，传统分工正向融合型的“产品工程师”角色演进。 针对无序实验带来的风险，道琼斯高级副总裁Bhaskar Borah介绍了其集中的AI指导委员会机制。该委员会对自下而上提出的方案设定了三大硬性标准：是否解决真实业务瓶颈、能否在可接受周期内量化ROI，以及能否在不泄露核心IP与客户数据的前提下安全扩展。Borah强调，尽管底层大模型每周都在更迭，企业的治理政策与伦理底线必须保持恒定，合规与治理评估必须先于ROI测算。 量化回报门槛与人机协同试错 (18:41) 在监管严格的行业中，落地AI智能体需要平衡容错率与推进速度。Newfront保险首席AI官Lin Yuan表示，面对全美50个州的繁杂监管，团队在初期实施中避开核心监管重地，优先选择有人工介入（Human-in-the-Loop）的低阻力环节。由于原型设计成本大幅下降，工程与业务团队可以在1至2周内快速构建多达20个原型，在保持人工监督的前提下，80%至90%的准确率即可作为可接受的部署基线，并按3个月、6个月和1年的周期评估业务指标。 对于全面推向生产环境的门槛，Sabarethinam透露MiniMed执行严格的量化指标：项目必须达到20%至30%的最低投资回报率基线，触达至少50%的员工群体（对应5000人以上），且节约效益必须直接体现在损益表（P&L）上，仅凭“节省时间”等软性指标不足以获批。Stevens亦指出，在财务验证中，后台运营成本的实际削减远比供应商宣称的提振20%顶线营收更为可靠。 自研外采抉择与SaaS基础设施定位 (28:18) 关于行业内“SaaS已死”的论调，Stevens予以明确反驳，指出AI工具并未取代CRM、技术支持或财务软件等核心底座，其实际价值在于为现有业务流程带来50倍、100倍甚至200倍的效率提升。他援引一项来自哈佛与麻省理工学院的研究说明，采购成熟商用AI工具的成功率是内部自研的两倍，这一差距主要源于商用合作所具备的专注度、持续技术支持与运营协同。 在自研与采购的边界划分上，Borah解释道琼斯采取分案评估策略：一方面与云厂商签订授权协议，在获得合理补偿的前提下允许其利用内容训练，并采购大厂企业级服务支持内部运营；另一方面，针对威胁品牌信誉、数据安全或人工把关机制的特定场景坚持自主研发。Sabarethinam则指出，面对传统SaaS厂商适配信智流程迟缓的现状，MiniMed在采购中更青睐具备可扩展架构、支持模型持续迭代的原生AI软件。 动态模型路由与集中化成本控制 (34:03) 未受管控的Token消耗容易引发推理预算失控。Stevens提及Scrunch在早期无限制开放Notion AI智能体导致账单意外激增的教训。为此，MiniMed通过部署动态模型路由框架，将常规用户查询引导至自托管或轻量化模型，仅将复杂的多步推理分发给高成本的顶尖大模型，成功将整体算力开销降低了30%至40%。 为防范工具泛滥与账单失控，Yuan提出通过集中化的企业级API平台层汇聚所有第三方应用的LLM调用请求，建立统一的监控与成本过滤机制。道琼斯则通过执行投资委员会把关采购，要求拟引入的AI工具必须具备跨部门赋能的通用性，以此杜绝各业务部门出于“错失恐惧”（FOMO）而采购过多孤岛式单点工具。 低风险场景起步与复合收益释放 (40:32) 针对低风险、高ROI的切入点，Sabarethinam建议企业优先从分析任务、内部报表、资料检索、合规审查以及FDA申报等场景入手，借助结构化的数据底座，可迅速将业务分析师的生产力提升50%至60%。Borah同样指出，道琼斯将AI定位于辅助记者承担繁琐事务的生产力工具，以保护调查报道中不可替代的信任资产与核心叙事能力。 多位嘉宾指出，高回报AI用例在部署后期往往展现出复合效应。Yuan举例说明，Newfront最初仅利用AI智能体提取非结构化保险文档以节省工时，但这层沉淀下来的结构化数据随后催生了大量完全无需调用LLM的确定性下游应用，大幅放大了初期的技术回报。Stevens和Borah也分别以会议记录沉淀客户健康分、客服对话暴露内部文档缺陷，以及“客户360”项目跨部门铺开为例，证明了基础AI实施对企业整体数据治理的连锁正向效益。 该研讨表明，高价值AI智能体项目的成败不在于追求技术上的激进颠覆，而在于能否建立跨部门复用、保护核心业务信誉并产生复合数据价值的长效机制。然而，如何在底层模型高频迭代的外部环境下保持企业治理规范的稳定性，并确保纸面上的时间节省能够切实转化为损益表上的财务收益，仍是各行业高管在规模化推进中需持续权衡的核心挑战。

## Background 该内容由 Ai4 频道于 2026 年 8 月 22 日发布在 YouTube 平台。页面简介显示，本次圆桌研讨会由 Resilience Media 记者 Jonathan Shieber 主持，汇聚了来自 Scrunch、MiniMed、Newfront 和道琼斯（Dow Jones）的高管专家，重点探讨如何为企业精准识别具备最高投资回报率（High ROI）的 AI Agent 用例，并分享评估业务就绪度、量化潜在收益与加速方案落地的实战框架。 ## TL;DR 圆桌嘉宾们深入剖析了企业推进 AI Agent 的核心路径，一致认为在人员、流程与数据治理上的投入远比单纯采购工具更为关键。道琼斯与 MiniMed 等企业分享了设立严密治理委员会、搭建 Customer 360 数据基座，以及在呼叫中心等核心场景坚持“人在回路”（HITL）以斩获 30% 至 40% 综合生产力提升的实战经验。此外，嘉宾们围绕快速原型试错、自研与外采决策逻辑，以及利用集中化网关和模型路由管控 Token 成本展开了深入探讨，强调真正的高 ROI 必须实打实反映在损益表（P&L）的财务指标上，并通过沉淀下游结构化数据资产形成持续的复利效应。 ## Atmosphere & style 整场对话务实干练、紧凑专业且极具一线实战厚度，主持人发问切中要害，频频引导嘉宾直面落地痛点。四位来自医疗、保险与媒体等强监管或重内容行业的高管毫无保留地分享了各自踩过的治理陷阱与账单教训，在冷静剥离 AI 炒作泡沫、坚守安全与信任底线的同时，展现出利用技术重塑端到端业务流程的从容自信，洋溢着高管决策与硬核工程落地深度交织的思辨氛围。 ## Key takeaways - [02:35](https://www.youtube.com/watch?v=e8o5Mdm-Z9g&t=155s) 企业在数据治理、流程改造与员工培训上的投入应远超 AI 工具本身（甚至达到 2 倍以上），因为脱离工作流变革的单点 AI 往往无法实现预期 ROI。 - [06:51](https://www.youtube.com/watch?v=e8o5Mdm-Z9g&t=411s) 道琼斯通过设立 AI 指导委员会，采用自下而上的创意提报机制，并严格基于“解决真实痛点、可量化 ROI、负责任且具扩展性”三大原则进行立项过滤与高管对齐。 - [08:31](https://www.youtube.com/watch?v=e8o5Mdm-Z9g&t=511s) 打通分散数据为 AI 提供上下文可能导致“数据武器化”的安全隐患，企业必须从放任自流的过度自治转向“组织级审批+局部自主”的双层治理架构。 - [11:32](https://www.youtube.com/watch?v=e8o5Mdm-Z9g&t=692s) 医疗健康等强监管行业坚持“患者安全第一”为首要红线，通过从零重塑端到端业务流程并引入人在回路（HITL），在客服呼叫中心等场景实现 30%–40% 的综合生产力跃升。 - [15:14](https://www.youtube.com/watch?v=e8o5Mdm-Z9g&t=914s) 道琼斯以统一数据平台 Customer 360 为底层基石，通过解决数据孤岛来支撑下游的次优行动预测、AI 销售助手等高 ROI 智能体用例落地。 - [18:46](https://www.youtube.com/watch?v=e8o5Mdm-Z9g&t=1126s) 在强监管的保险业务中，Newfront 采取“代码和原型极度廉价”的策略，通过一周快速试错 20 个原型，避开监管核心区并优先寻找人机协同的高收益突破口。 - [22:43](https://www.youtube.com/watch?v=e8o5Mdm-Z9g&t=1363s) AI 的本质是核心系统和专业员工的效率放大器（50x–200x），而非推倒既有企业 SaaS 基础设施的颠覆者。 - [25:15](https://www.youtube.com/watch?v=e8o5Mdm-Z9g&t=1515s) AI 试点的衡量硬指标必须切实反映在损益表（P&L）的真金白银上，后台运营的明确降本往往比虚高的增收预测更具管理层置信度。 - [28:18](https://www.youtube.com/watch?v=e8o5Mdm-Z9g&t=1698s) 企业在核心竞争力与强公信力场景倾向自主研发，而在非核心或通用流程更倾向采购具备高扩展性的 AI 原生软件，且推进的核心难点始终是变革管理。 - [34:33](https://www.youtube.com/watch?v=e8o5Mdm-Z9g&t=2073s) 应对 Token 成本失控的有效手段是采用动态模型路由（将简单任务分流给轻量/开源模型以省去 30%–40% 成本），并建立集中化的企业 GenAI API 网关进行统管。 - [38:45](https://www.youtube.com/watch?v=e8o5Mdm-Z9g&t=2325s) 内部研究报告、合规申报与文案总结等数据结构化与分析任务是低风险、高 ROI 的“低垂果实”，能直接帮分析师团队提升 50%–60% 效率。 - [41:07](https://www.youtube.com/watch?v=e8o5Mdm-Z9g&t=2467s) 高价值 AI Agent 项目具备强大的意外复利效应：单点自动化过程中沉淀的高质量结构化数据层，能反哺下游构建客户健康度或支持确定性规则系统。 ## Detailed walkthrough ### [00:04](https://www.youtube.com/watch?v=e8o5Mdm-Z9g&t=4s) 圆桌开场与重磅嘉宾介绍 本次 Ai4 峰会圆桌由 Resilience Media 记者 Jonathan Shieber 主持。Jonathan 幽默地介绍，今天同台的都是在 AI Agent（智能体）领域深耕已久的资深高管。 Jonathan 依次介绍了四位嘉宾： - **Lin Yuan（袁林）**：Newfront 首席 AI 官兼工程执行副总裁，加入 Newfront 前曾就职于 Meta、Coursera 和 Lyft； - **Bhaskar Borah**：Dow Jones（道琼斯）高级副总裁兼营销数据科学与技术负责人，此前曾任职于 Citi（花旗银行），而道琼斯恰好是主持人 Jonathan 曾供职多年的前东家； - **Devin Stevens**：Scrunch 解决方案工程与实施负责人，此前曾在 Gainsight、Hearsay 以及收购 Hearsay 的 Yext 担任职务； - **Muthu Sabarethinam**：MiniMed（原美敦力 Medtronic 旗下业务，现已独立分拆）数据科学与业务转型副总裁，此前曾担任 Honeywell Connected Enterprise（霍尼韦尔互联企业）的副总裁兼首席数据官，主导了 Forge AI 的开发与落地。 主持人表示，本场讨论的核心是探讨企业如何识别高投资回报率（High ROI）的 AI Agent 用例并落地推进，涵盖决策框架、落地实践以及人在回路（Human-in-the-loop）的角色等。 ```mermaid flowchart LR accTitle: 核心讨论议题框架 A["高 ROI 识别"] --> B["决策与治理结构"] B --> C["落地实施与工作流变革"] C --> D["人在回路 (HITL) 与风控"] ``` --- ### [02:35](https://www.youtube.com/watch?v=e8o5Mdm-Z9g&t=155s) 工具 vs 人与数据治理：Gartner“2倍投入法则”大讨论 Jonathan 抛出开场问题：Gartner 近期发布的一份报告指出，企业在数据结构化、治理、员工培训以及“人”的方面的投入，应该是其在 AI 工具本身投入的 **2 倍**。他向全场嘉宾提问：这个比例到底偏高了，还是恰到好处？ Lin Yuan 率先作答。她认为人员、流程与工具同样重要，至于具体是 2 倍、4 倍还是 8 倍因企业而异。她回顾了 Newfront 的演进历程： 在生成式 AI 爆发之前，团队主要从事传统的机器学习、深度学习、回归分析和统计等项目。当时团队尝试将商业保险和员工福利的工作流程进行标准化，而这些流程充斥着大量非结构化工作和手工操作。传统行业累积了海量数据，但基本都是“脏数据”（dirty data），质量与结构化程度极低。在过去，清洗数据所耗费的投入远超过构建实际产品的成本。直到 ChatGPT (GPT-3.5) 出现后，生成式 AI 才真正解锁了非结构化数据的处理潜力。 随后，Bhaskar Borah 从媒体与内容行业的角度表示赞同： 道琼斯旗下拥有《华尔街日报》（The Wall Street Journal）等标杆品牌，“信任就是我们的核心产品（Trust is our product）”。虽然 AI 极其擅长承担繁重的脏活累活（heavy lifting），但人类监督与高质量的新闻专业主义始终是核心，人类负责最终的判断与故事叙述。因此道琼斯在人员与流程上的投入极大。 Devin Stevens 表达了强烈的肯定立场（"I'm a huge yes"）： 无论具体是 2 倍、2.5 倍还是 3 倍，核心启示在于 **“AI 自身无法独立完成任何事情”**。工具再强大，也必须伴随着工作流变革与战略演进。市面上大量 AI 项目之所以以 ROI 失败告终，往往是因为项目范围（scope）设定错误，或者缺乏对应的工作流改造。 Muthu Sabarethinam 从成本与组织架构两个维度深入剖析： 1. **AI 工具单价持续下降**：企业无需对工具本身的采购成本过度担忧。 2. **岗位角色正在被重塑**：传统的产品经理（PM）与软件工程师（SWE）正在融合为“产品工程师”（Product Engineer），营销等岗位也在发生颠覆性变化。因此企业必须花大精力重新定义岗位职责，让员工在更高维度上驾驭 AI。 --- ### [06:51](https://www.youtube.com/watch?v=e8o5Mdm-Z9g&t=411s) 道琼斯 AI 决策委员会的运作机制与治理原则 Jonathan 向身处道琼斯 AI 指导委员会（AI Steering Committee）的 Bhaskar 发问：该委员会是拍板落实的最终决策者，还是一个前瞻性的战略把关机构？ Bhaskar 详细拆解了道琼斯的 AI 推进模式： - **自下而上的创意收集**：道琼斯内部各业务团队自下而上（bottoms-up）提交 AI 构想与商业论证（Business Case）。 - **指导委员会的三大评估原则**： 1. **解决真实业务痛点**：是否在解决真正的业务问题？ 2. **可靠量化 ROI**：能否在可预期的时间窗口内可靠地衡量投资回报？ 3. **负责任且具扩展性**：能否做到 Responsible AI 并支持规模化落地？ - **推动落地与高管对齐**：通过筛选的项目会被提交给 C-Suite 高管团队，以获取落地试点（POC）的资源和预算对齐。 - **治理重于工具**：Bhaskar 强调技术每周都在变，但政策与基本原则必须保持稳定，在谈 ROI 之前必须建立好治理底座。 ```mermaid flowchart TD accTitle: 道琼斯 AI 决策流程 subgraph 创意与评估 A["一线业务团队<br/>(Bottom-up 提报)"] --> B["AI 指导委员会"] B --> C{"三大评估原则"} C -->|1. 真实痛点<br/>2. 可靠 ROI<br/>3. 责任与扩展| D["高管层 (C-Suite) 审批"] end subgraph 落地与管控 D --> E["POC 试点与预算落地"] F["稳定治理原则与政策底座"] -.-> B end ``` --- ### [08:31](https://www.youtube.com/watch?v=e8o5Mdm-Z9g&t=511s) 避免“数据武器化”：企业治理模式与过度自治的收敛 针对治理与决策结构，其他嘉宾也分享了各自组织的运转方式： Muthu 提出了警示： AI 运行高度依赖上下文（Context），这就迫使企业将原本分散的数据串联打通。这在某种程度上等于 **“把数据武器化”（weaponizing our data）** —— 一旦发生越权或泄露，攻击者就能拿到全链路的完整档案。此外，员工个人的 OneDrive、全量邮件、Teams 聊天记录等非结构化数据都被接入 AI 系统，如果缺乏严格的访问控制与安全框架，极易引发合规灾难。在 MiniMed，虽然有底层基础设施和数据治理团队（如 IT 与 Office 365 负责人）提供专业输入，但最终由高级管理层统一制定授权准则。 Lin Yuan 补充了 Newfront 的跨部门协同机制： 公司由法务、财务、技术及业务核心利益相关者组成联合高管决策群。面对上百个 AI 创意，团队严格依靠顶层原则进行过滤，首要原则就是 **人在回路（Human in the loop）**，并确保算法具备可解释性与责任机制。 Devin 则坦诚分享了 Scrunch 走过的弯路： - **AI 优先与野蛮生长**：Scrunch 提倡“AI 优先”文化，遇到问题首先考虑现有 AI 工具是否能解决。但在过去 3 到 6 个月里，公司放手让一线员工自由尝试，导致工具泛滥、使用方式五花八门，失去了标准化与规模化效应。 - **收归与双层治理**：团队近期不得不收回部分工具权限，转为 **“双层架构”** 治理——在组织级（Organizational Level）必须经过正规委员会与相关负责人审批立项；而在局部场景（如日常会议记笔记等）则保留一定的个人自主权。 ### [11:32](https://www.youtube.com/watch?v=e8o5Mdm-Z9g&t=692s) 从单点工具反思到全面流程再造：医疗安全底线与端到端 ROI 考量 在日常办公场景（如邮件撰写和日程管理）中，企业通常会给予员工极大的自由度来探索和使用 AI；然而一旦涉及具体业务职能，团队往往会因为初期推进过于激进、缺乏克制而被迫刹车，回过头去补建严格的治理体系。面对这种复杂局面，在判断一个业务场景究竟应当自动化、交由智能体处理，还是坚决保留人工干预时，医疗健康行业有着极为严苛的判定红线。MiniMed 的数据科学与业务转型副总裁 Muthu Sabarethinam 指出，决策的第一道关卡是“可行性核验”（feasibility check），对于医疗企业而言，最重要的核心指标永远是“更优质的患者护理”与“患者安全第一”，这一关过不去，任何场景都免谈。 当通过了安全可行性核验后，评估重点便转向商业价值。Muthu 强调，企业绝不应满足于零敲碎打的边际收益，而是要刻意将想法塑造成能够“撼动大局”（needle-moving）的宏大项目——如果 CEO 对此都不觉得有重大意义，那这个用例多半不够大。他将其称之为“超人级用例”（superhuman use cases）或端到端流程转型。由于构建一个微小改进与搭建完整系统的工程量其实相差无几，与其修修补补，不如拿出一张白纸，从零重新思考由 AI 全面接管该流程该如何运转，这样才能获得最高的投入产出比。 ```mermaid flowchart TD accTitle: 医疗AI智能体用例评估与全流程价值落地机制 subgraph S1["第一阶段：安全红线与可行性审查"] A["患者护理与安全核验\n(Patient Safety First)"] -->|不通过| B["坚决否决 / 纯人工处理"] A -->|通过| C["价值评估：拒绝零散修补\n必须能撼动大局 (Needle-moving)"] end subgraph S2["第二阶段：端到端流程重塑与多点增效"] C --> D["端到端流程重构 (Blank-sheet Redesign)"] D --> E["智能接入 (Smart Intake)\n前序筛查省时 5%"] D --> F["通话摘要 (Summarization)\n后处理省时 10%"] D --> G["人机协同机制 (Human-in-the-loop)\n专家做主导，防止放弃思考"] end E & F & G --> H["综合实现 30%-40% 生产力跃升"] ``` --- ### [13:33](https://www.youtube.com/watch?v=e8o5Mdm-Z9g&t=813s) 呼叫中心解构：智能分流、通话摘要与必须保留的人机协同 许多人认为呼叫中心引入 AI 仅仅是为了节省坐席的通话时长，或纯粹追求接听更多电话。然而 MiniMed 的实践证明，从一张白纸重新设计整个流程，会在各个节点挖掘出意想不到的增量杠杆。Muthu 透露，团队配备了 800 名坐席，每年要处理 200 万通电话。他们上线了名为“智能接入”（smart intake）的环节，由 AI 承担每通电话前几分钟的患者信息筛查，在人工坐席介入前就已经节省了 5% 的通话时长；以往坐席在挂断后需要耗费 5 分钟来整理记录，如今引入 AI 通话摘要功能后，又直接削减了 10% 的后续处理耗时。通过全流程重构，这些看似微小的 5% 和 10% 迅速累加，形成了可观的综合 ROI。 针对主持人 Jonathan Shieber 提出的是否在“全面裁撤坐席”的疑问，Muthu 明确澄清这绝非全盘替换员工，而是精准定位一线员工的痛点并为其减负。甚至在技术上，AI 系统完全有能力独立主导整通电话，但团队在测试中发现了一个严重的弊端：如果让 AI 主导全场，人工坐席就会沦为纯粹等待 AI 提示下一步提问的“传声筒”，导致专业人员放弃主动思考。由于医疗领域关乎人命，坐席本身就是业务专家，系统必须坚持“人在回路”（Human-in-the-loop）的设计，让 AI 退居辅助支持角色，这样既能保证高水准的服务体验，又能扎扎实实拿下 30% 至 40% 的整体生产力提升。 - **坐席规模与业务体量**：MiniMed 拥有 800 名坐席，年呼叫量达 200 万通。 - **智能接入降本效果**：由 AI 承担开局筛查，立即为每通电话节省约 5% 的人工介入时间。 - **摘要自动化收益**：替代坐席挂断后 5 分钟的手动记录，再释放 10% 的流程耗时。 - **综合生产力跃升**：在确保坐席保持独立临床思维的前提下，整体业务实现了 30% 到 40% 的生产力增长。 --- ### [15:14](https://www.youtube.com/watch?v=e8o5Mdm-Z9g&t=914s) 道琼斯的底层基石：以 Customer 360 驱动多业务下游智能体 将话题转向道琼斯（Dow Jones）的内部 AI 落地经验时，Bhaskar Borah 重点剖析了“次优行动工具”（next best action）以及“客户全景画像”（Customer 360）的立项逻辑。Bhaskar 直言，向 AI 指导委员会汇报项目时，指标必须真真切切地体现在财务与运营数字上，而不能停留在漂亮的 PPT 汇报层面。Customer 360 在立项之初并不是一个直接产生业务收入的智能体，而是一项核心的数据基建工程。由于道琼斯旗下拥有众多知名品牌和错综复杂的业务线，数据长期割裂在不同的孤岛系统中，如果不先整合并建立起单一体身份图谱（single identity graph）的统一数据平台，任何所谓的大模型智能体都无从谈起。 ```mermaid flowchart TD accTitle: 道琼斯从底层数据平台到智能体业务价值的传导架构 subgraph DataFoundation["底层数据基石 (Data Foundation)"] A1["多业务线/跨品牌孤立数据"] --> B["Customer 360 统一数据平台\n(Single Identity Graph)"] end subgraph AgenticUseCases["赋能下游智能体用例"] B --> C["次优行动预测 (Next Best Action)"] B --> D["AI 驱动型销售助手 (AI Sales Assistants)"] B --> E["智能客户服务 (Agentic Customer Service)"] end subgraph BusinessValue["可量化商业 KPI"] C & D & E --> F["新增业务收入 (Revenue)"] C & D & E --> G["运营支出优化 (Expense Reduction)"] C & D & E --> H["客户获客成本/转化率 (CAC & Acquisition)"] end ``` 这项底层数据基座的核心价值在于其能够激活和赋能各类下游智能体。Bhaskar 解释道，一旦数据底座成型，团队便能够在其上迅速孵化出次优行动推荐、AI 销售助手以及自动化客服系统等实用工具。每一个下游用例都直接绑定了清晰可衡量的商业 KPI，包括营业收入的提升、营运成本的缩减以及客户获客转化率的优化。因此，评估基建型项目的 ROI，本质上就是清晰度量它对全链路下游商业价值的杠杆撬动能力。 --- ### [17:02](https://www.youtube.com/watch?v=e8o5Mdm-Z9g&t=1022s) 自下而上的有机创新：支持工单骤降 80% 带来的跨部门连锁效应 相比于道琼斯严密自上而下的基建推进，Devin Stevens 介绍了 Scrunch 更加自由灵活（甚至略带自发演化色彩）的 AI 探索路径。为了让优秀的构想从团队内部浮现并上升为全公司级项目，Scrunch 建立了常态化的展示机制，员工可以在跨部门例会或专设的 Slack 频道中主动向全员演示“我和我的团队最新做出了什么”。这种机制极大地激发了跨团队的自发复用与技术借用，形成了自下而上的扩散效应。 这种快速推进也伴随着新的组织挑战与连锁反应。Devin 透露，在客服与技术支持场景中开启 AI 工具后，团队以极快的速度将高达 80% 的日常工单转移给 AI 自主解决。然而这个单点突破立刻暴露了上游团队的短板——AI 的回答完全依赖于输入知识的质量，导致产品文档（product documentation）的准确度与完备性瞬间变成了全公司的紧急痛点。这一现象促使团队意识到，真正成功的 AI 落地绝不仅是一个局部的技术替换，它会立刻演变为一个跨职能协作工程；越早将相关职能部门拉入协作轨道，整个组织越能享受到 AI 带来的巨大瀑布式价值红利。 --- ### [18:46](https://www.youtube.com/watch?v=e8o5Mdm-Z9g&t=1126s) 严苛监管下的快速原型战术：代码很便宜，一周试错 20 个想法 在保险这种受到各州法案多重约束的强监管行业，将 AI 智能体推向全公司应用需要截然不同的战术。Newfront Insurance 的首席 AI 官兼工程执行副总裁 Lin Yuan 坦言，美国 50 个州各有各的合规要求，因此高 ROI 用例的首要筛选准则并非直奔核心监管地带，而是“在起步阶段绝对不碰监管核心业务”，优先挑选易于部署、天然支持人机协同的环节作为突破口。 面对如何界定成功阈值（是必须达到 100%、99% 还是 85% 准度即可）的追问，Lin 指出团队已经抛弃了那种关起门来苦思冥想“未来一两年投资什么才能见效”的旧模式。因为在当前大模型环境下，“编写代码的成本非常低，构建原型的成本也极低”（code is cheap, prototyping is cheap）。他们的做法是将工程师和一线业务人员编组，在短短一至两周内集中捣鼓出 20 个具体的项目原型，随后直接拉来未来需要参与人机回路的业务人员进行面对面实操测试。 通过这些可感知的原型，团队快速测算其在 3 个月、6 个月及 1 年内的潜在商业收益——是能帮公司在某一特定保险细分市场建立竞争优势，还是能够直接压降特定工作流的人工成本。在此基础上，结合安全隐私评估，判断针对当前业务 80% 或 90% 的准度是否已经足以为人类员工兜底减负。这种避开合规风险、让 AI 充当人类副驾驶而非全权代理的做法，成为了强监管机构实现稳妥回报的黄金标准。 --- ### [22:08](https://www.youtube.com/watch?v=e8o5Mdm-Z9g&t=1328s) 智能体对撞的前沿思考：当自动化与自动化直接对话 讨论的尾声触及了一个更为激进的未来愿景：从“人机协同”迈向“智能体对智能体”（Agent-to-Agent）的交互生态。主持人 Jonathan 提及业界关于“AI 崛起将导致传统 SaaS 走向消亡”的激进论调，并询问 Devin：当企业不再是帮人类省力，而是构建一套完全由智能体与另一套自动化系统自主对话的纯自动化体系时，ROI 和“人在回路”又将如何被重新定义？Devin 回应称，Scrunch 的策略确实是尽可能将一切能自动化的流程彻底自动化，尽管团队进行了大量极端场景的实验，但行业在真正脱离人工、进入全自动智能体交互的进程中，依然在软件形态与信任机制上面临着未解的探索。 ### [22:43](https://www.youtube.com/watch?v=e8o5Mdm-Z9g&t=1363s) AI 作为杠杆：增强核心工具与人，而非颠覆基础设施 在深入探讨企业的采购策略之前，讨论聚焦于 AI 在企业既有技术栈中的角色。企业并不会因为 AI 的出现就彻底推翻其底层基础设施；无论是客户支持工具、CRM 还是财务软件系统，企业依然必不可少。目前既不存在所谓的“凭感觉写代码”（vibe coding）能凭空替代这些庞大平台，也没有任何独立的 AI Agent 能够将某一核心系统完全剔除。当前 AI 的核心逻辑并不是取代既有工具或替代员工，而是围绕着人与核心系统，提供极高的效能杠杆。 “SaaS 并没有死。”业界讨论的焦点在于，AI 能否让员工在处理以往相同的业务流程时，效率提升 50 倍、100 倍甚至 200 倍。这种提效逻辑将 AI 确立为“放大器”，而非推倒重来的破坏者。 ```mermaid flowchart LR accTitle: AI 赋能核心系统与员工架构图 subgraph Core["核心企业基础设施"] direction TB CRM["CRM 系统"] FIN["财务软件"] SUP["支持工具"] end subgraph Human["人力团队"] EMP["专业员工 / 业务团队"] end AI["AI Agent / 生产力工具"] -->|效率放大 50x-200x| Core AI -->|分担机械性繁重劳动| EMP Core <--> EMP ``` ### [23:38](https://www.youtube.com/watch?v=e8o5Mdm-Z9g&t=1418s) 真实人类智能与新闻信任：道琼斯如何界定 AI 的边界 主持人 Jonathan Shieber 将话题引向道琼斯（Dow Jones），提出对于一家高度依赖真实人类智能（Real Human Intelligence）与深度报道的新闻机构而言，AI Agent 如何参与以“人际互动与信任”为底色的话语体系。道琼斯拥有数以百计乃至千计的记者，新闻业的本质究竟能否被自动化，又该如何衡量采编环节的投资回报率（ROI）？ 来自道琼斯的 Bhaskar Borah 明确指出，对于道琼斯旗下的品牌来说，品牌信任（Trust）与高品质新闻（Premium Journalism）始终处于绝对的核心地位，这是 AI 无论如何都无法替代的。道琼斯引入 AI 系统的根本定位是“专业记者的生产力助手”。记者们在挖掘调查、撰写故事上投入了极大的心血与汗水，AI 负责接管大量繁重的机械性杂务（grunt work），使记者能将宝贵的精力聚焦于内容叙事本身。 ### [25:15](https://www.youtube.com/watch?v=e8o5Mdm-Z9g&t=1515s) 衡量成功的硬指标：P&L 利润表上的真金白银与规模覆盖 在谈及 AI 试点项目如何跨越到全企业推广时，小组深入探讨了衡量的硬性门槛。MiniMed 的 Muthu Sabarethinam 给出了清晰的量化标准： - **最低 20% 至 30% 的投资回报率（ROI）**：这是所有投资的起跑线。 - **覆盖广度与规模化**：目标是让全公司至少 50% 的员工（即至少 5,000 人）使用 AI。单点团队的效率提升非常可观，能触达 50 到 100 人的群体并让他们做得更好，聚合效益会极为惊人。 - **必须落实在 P&L（损益表）上**：Muthu 直言评估标准既不是虚幻的“节约时间”，而是实打实的美元（"It's dollars... it's not time"）。CFO 必须能在账面上看到节省下来的资金流向何处、如何重新投资回业务中。 来自 Newfront Insurance 的 Lin Yuan 与 Scrunch 的 Devin Stevens 补充了对美元指标的不同理解维度： - **可直接度量 vs 战略测算**：团队效率提升 30% 意味着可以用更小的人力规模承接更多业务，这是直接的财务回报；而像“改善客户留存率”并间接拉动收入，则属于战略层面的推算，无法做到严格的一对一映射。 - **后台降本 vs 业务增长的置信度**：Devin 指出内部对后台运营的“降本”（Cost Reduction）抱有高得多的信任度；如果某款新营销工具号称能带来 20% 的收入顶线增长，管理层往往非常审慎甚至怀疑。相对而言，削减内容创作、客服支持或交付服务流程中的固定开支，其 ROI 更具确定性与稳定性。 ### [28:18](https://www.youtube.com/watch?v=e8o5Mdm-Z9g&t=1698s) 自研还是外采（Build vs. Buy）：核心竞争力与架构扩展性 在自研与外购系统的权衡上，道琼斯采取因地制宜的策略。一方面，道琼斯与超大规模云厂商（Hyperscalers）签署内容授权许可协议，利用自身丰富的品牌内容训练模型并获得合理补偿；另一方面，与头部科技企业达成企业级合作，共同构建内部工作流与面向客户的产品。然而，一旦遇到需要高度定制且绝不能牺牲公信力与“人类在环”（Human-in-the-loop）的特定场景，道琼斯就会坚决选择自主研发。 | 决策考量因素 | 倾向自主研发（Build） | 倾向商业采购（Buy） | |:--- |:--- |:--- | | **业务关联** | 触及核心竞争力、关乎品牌公信力与合规审核的利基场景 | 非核心通用流程、需要快速响应前沿技术的场景 | | **厂商属性** | 传统 SaaS 厂商迭代乏力，难以满足特定流程需求 | 具备原生 AI 架构演进能力的现代化新型 AI 软件供应商 | | **核心评估维度** | 内部工程资源、是否能保障 Human-in-the-loop | 供应商的系统架构扩展性（Extensibility）及其与 AI 演进路线的契合度 | Muthu 指出，由于存活了几十年的传统 SaaS 巨头在跟进 AI 步伐上较为迟缓，MiniMed 如今极少再去采购传统的 Salesforce 附加模块，而是更倾向于挑选像 Observatory 这样的新兴 AI 原生软件厂商。在评估新型供应商时，系统架构的“可扩展性”（Extensibility）成为了最关键的决策依据——企业买的不仅是当前的成熟功能，更是其底层架构能否无缝接入企业未来的 AI 演进蓝图。 ### [31:25](https://www.youtube.com/watch?v=e8o5Mdm-Z9g&t=1885s) 变革管理胜过单纯技术：AI 原生方案的落地现实 针对 AI 原生供应商的获客逻辑与价值传递，Devin Stevens 坦言他们开拓的是全新的产品类别，例如分析并影响“AI Agent 如何检索互联网”。但他强调，即便工具和技术本身再优秀，决定成败的往往是企业内部的流程、组织架构与高管支持度。 Devin 援引了麻省理工学院（MIT）前一年的一项研究，该研究显示**采购外部 AI 工具的企业成功率是完全自主构建内部工具的两倍**。造成这种显著差异的根本原因往往不在于底层技术有多大差距，而在于组织对待外部项目的资源倾斜与执行机制完全不同：自研项目往往容易缺乏同等的重视、资源维系和持续支持。 Lin Yuan 也表达了强烈的共鸣。在 Newfront，团队并不对外兜售 AI 产品，而是自研工具赋能内部业务。向业务部门推销方案时，最重要的就是早期将业务利益相关者深度拉入共创体系中。推进 AI 绝不仅仅是交付一个模型或代码库，核心难题始终是变革管理（Change Management）：深刻洞察业务实际工作流，并妥善管理 Agent 与工具对真实员工日常习惯所产生的冲击与改变。 ### [34:03](https://www.youtube.com/watch?v=e8o5Mdm-Z9g&t=2043s) Token 成本膨胀的隐忧与应对策略 主持人随后抛出业界广为流传的痛点：“Token 成本的失控与暴涨，是否已经改变了大家关于自研还是外采的算盘，抑或是改写了整个 ROI 评估模型？” Muthu 作为首位回应者表示，Token 成本确实是一笔必须严加看管的大开支。如果疏于监控，费用在短时间内确实会迅速膨胀到难以收拾的地步。然而，只要建立起精细的治理与控制机制，AI 调用成本其实并非不可承受的高昂负担。当前市场上可供选择的模型层出不穷，企业不仅拥有广泛的第三方模型可选，还可以结合私有化自托管模型（Self-hosted models）来优化整体开销结构。 ### [34:33](https://www.youtube.com/watch?v=e8o5Mdm-Z9g&t=2073s) 模型路由与成本管控：避免昂贵的 AI 账单 讨论从如何控制 AI 模型调用成本切入。嘉宾指出，合理架构的关键在于分流：将简单的日常问题重定向至轻量、快速的模型，仅在真正需要复杂推理时才调用高昂的前沿模型（Frontier Models）。通过教育团队使用“快速模式”（fast mode）并引入能够自动选择模型的路由工具，企业的成本结构可以直接降低 30% 到 40%。“工具就在那里，但如果你不花精力去治理，收到一张巨额账单是轻而易举的事。” Devin Stevens 坦承自己也有过类似教训。他是 Notion 的忠实拥趸，曾狂热地在各个业务场景中全面开启 Notion Agent。然而狂欢之后，账单扑面而来。他强调，尽管模型的单价成本正在不断下降，但团队必须未雨绸缪——不能只看 PoC 或小规模试点时的开销，更要精准测算当使用规模扩展到 50 人、100 人甚至 1000 人时的成本结构。 - **模型路由优化**：自动化模型选择与分流机制可直接削减至少 30%–40% 的 Token 成本支出。 - **规模化成本预测**：评估 AI 方案不能止步于试点阶段，必须对全员推广时的成本增长曲线建立前瞻性预测。 --- ### [35:56](https://www.youtube.com/watch?v=e8o5Mdm-Z9g&t=2156s) 预算硬顶还是收益驱动？跨部门治理与集中化网关 针对主持人 Jonathan Shieber 关于“内部是否该设置刚性支出上限（Hard Cap）”的提问，Lin Yuan 和 Bhaskar Borah 分享了截然不同的治理思路。Lin Yuan 强调核心原则在于 ROI 净值：只要 AI 产生的业务收益（增收或降本）大于消耗的 Token 成本本身，项目就是合理的；同时，她极力推崇开源模型与“模型不可知”（Model Agnostic）的产品架构设计，以保持替换灵活性并压低计算单价。 ```mermaid flowchart TD accTitle: AI Cost and Gateway Governance Architecture subgraph S1["业务请求与治理评估"] A["各业务线 AI 需求"] --> B{"投资委员会评估\n是否具备跨业务复用性？"} B -- "否 (孤立/重复场景)" --> C["拒绝 / 避免 FOMO 盲从"] B -- "是 (平台化通用场景)" --> D["立项并统一接入"] end subgraph S2["集中化 API 平台层"] D --> E["企业统一 GenAI 平台网关"] E --> F["统一 API 密钥与用量监控"] E --> G["动态模型路由与分流"] end subgraph S3["底层模型调用"] G --> H["轻量/开源模型\n(简单问答/分析)"] G --> I["前沿高推理模型\n(复杂逻辑/决策)"] end ``` Bhaskar Borah 指出，虽然资本支出（CapEx）有总预算红线，但最危险的是由 FOMO（错失恐惧症）驱动的野蛮采购。当业务部门为细碎场景提出成百上千个工具采购申请时，投资委员会必须评估该工具是否具备跨业务线的复用能力（"have legs for multiple use cases"），严防在细枝末节上重复造轮子。Muthu Sabarethinam 补充了一个关键技术实践：建立统一的集中式企业 GenAI 平台网关。所有供应商与自研工具的 API 调用必须强制经由该网关，统一管理订阅、流量与鉴权，彻底解决第三方工具各自调用 LLM 导致的成本失控与混乱局面。 - **模型中立原则**：构建与具体底层解耦的 Model-agnostic 架构，充分利用开源模型降低开销。 - **反 FOMO 评审**：设立投资委员会，拒绝单一分散工具，优先投资能支撑多条业务线的通用平台级方案。 - **集中网关管控**：将所有外部 Vendor 及内部调用的 LLM 请求统一收敛到集中 API 层，实现可观测性与成本过滤。 --- ### [38:45](https://www.youtube.com/watch?v=e8o5Mdm-Z9g&t=2325s) 现场问答：低风险、高 ROI 的“低垂果实”用例 在观众问答环节，现场参会者询问是否存在既没有过高法律与合规安全风险、又能快速见效的高 ROI“低垂果实”（low-hanging fruit）。Muthu Sabarethinam 迅速给出了答案：AI 最擅长的各类**分析与结构化生成任务**。 Muthu 表示，无论是日常报告生成、市场营销内容创作，还是复杂的法规申报（如 FDA 文件整理）、合规分析及高管汇报材料准备，只要底层企业数据治理就绪，这类分析用例就能极快落地，且能让业务分析师团队的工作效率迅速提升 50% 到 60%。这类赋能不直接对外暴露风险，却能广泛覆盖公司内部的几乎所有角色。 - **核心落地场景**：涵盖内部研究报告、FDA 申报及合规材料整理、营销文案生成、管理层演示准备。 - **提效幅度**：在具备良好数据底座的前提下，可直接帮助分析师团队提升 50%–60% 的生产力。 --- ### [41:07](https://www.youtube.com/watch?v=e8o5Mdm-Z9g&t=2467s) 超出预期的复利效应：从单点工具到数据飞轮 Jonathan Shieber 总结道，高 ROI 的 AI Agent 场景往往具备三个特质：能够在大规模员工中推广、增强而非危及核心业务能力、具备复利效应。他随即抛出最后一个问题：在项目实施后，是否存在远超最初预期的意外回报（Unexpected ROI）？ | 嘉宾 / 机构 | 初始解决的问题 | 实际产生的意外复利价值 (Compounding ROI) | |:--- |:--- |:--- | | **Lin Yuan** (Newfront) | 替代人工录入/解析非结构化保险保单文档 | 沉淀出高质量数据层，直接支撑下游大量确定性业务应用，无需再依赖大模型即产生长效复利 | | **Bhaskar Borah** (Dow Jones) | 在小范围内试点上线 Customer 360 系统 | 成功验证后迅速被其他业务单元复制并衍生出新用例，应用潜力呈网络化扩散 | | **Devin Stevens** (Scrunch) | 会议自动记笔记与客户支持工单自动化降本 | 会议记录反哺构建了客户健康度评分体系；支持数据沉淀倒逼并优化了全公司的知识库与数据架构 | Lin Yuan 详细拆解了保险非结构化文档录入的案例：起初算账仅考虑“节省了多少人工工时 vs 消耗了多少 Token”；但文档解析后沉淀下来的结构化数据层，后续驱动了大量完全不需要大模型参与的确定性规则系统，带来了持续的复合收益。Devin Stevens 同样补充，原本单纯为了“省去手写麻烦”的会议纪要 AI，最终沉淀成了全量客户沟通档案，直接赋能了客户健康度评分系统，像滚雪球一样推动了整个公司的数据规范化进程。 - **复利回报法则**：成功的 AI Agent 项目不仅提供单点的劳动替代，更关键在于沉淀下游可复用的结构化资产。 - **圆满收官**：在全场掌声中，主持人致谢各位嘉宾与现场观众，AI4 峰会的高 ROI AI Agent 专题研讨圆满落幕。 ## Notable quotes > [02:35](https://www.youtube.com/watch?v=e8o5Mdm-Z9g&t=155s) Trust is our product — Bhaskar Borah > [02:35](https://www.youtube.com/watch?v=e8o5Mdm-Z9g&t=155s) AI 自身无法独立完成任何事情 — Devin Stevens > [08:31](https://www.youtube.com/watch?v=e8o5Mdm-Z9g&t=511s) 把数据武器化 — Muthu Sabarethinam > [18:46](https://www.youtube.com/watch?v=e8o5Mdm-Z9g&t=1126s) 编写代码的成本非常低，构建原型的成本也极低 — Lin Yuan > [22:43](https://www.youtube.com/watch?v=e8o5Mdm-Z9g&t=1363s) SaaS 并没有死 — Devin Stevens > [25:15](https://www.youtube.com/watch?v=e8o5Mdm-Z9g&t=1515s) It's dollars... it's not time — Muthu Sabarethinam > [31:25](https://www.youtube.com/watch?v=e8o5Mdm-Z9g&t=1885s) 采购外部 AI 工具的企业成功率是完全自主构建内部工具的两倍 — Devin Stevens > [34:33](https://www.youtube.com/watch?v=e8o5Mdm-Z9g&t=2073s) 工具就在那里，但如果你不花精力去治理，收到一张巨额账单是轻而易举的事 — Muthu Sabarethinam > [35:56](https://www.youtube.com/watch?v=e8o5Mdm-Z9g&t=2156s) have legs for multiple use cases — Bhaskar Borah ## Mentioned Jonathan Shieber, Resilience Media, Lin Yuan, Newfront, WTW, Meta, Coursera, Lyft, Bhaskar Borah, Dow Jones, Citi, The Wall Street Journal, Devin Stevens, Scrunch, Gainsight, Hearsay, Yext, Muthu Sabarethinam, MiniMed, Medtronic, Honeywell Connected Enterprise, Forge AI, Gartner, ChatGPT, GPT-3.5, OneDrive, Microsoft Teams, Customer 360, Salesforce, Observatory, MIT, Notion, FDA ## Video description Devin Stevens - Head of Solutions Engineering & Implementation, Scrunch Jonathan Shieber - Journalist, Resilience Media Muthu Sabarethinam - VP Data Science & Business Transformation, MiniMed Lin Yuan - Chief AI Officer and EVP of Engineering, Newfront Insurance (now part of WTW) Bhaskar Borah - SVP, Head of Marketing Data Science & Technology, Dow Jones Successfully deploying AI agents begins with identifying the highest impact opportunities within your organization. This panel will explore proven methodologies for pinpointing use cases with maximum return on investment. Experts will discuss frameworks for evaluating business readiness, quantifying potential benefits, and rapidly testing AI agent solutions in real-world scenarios. Gain practical advice on how to transition quickly from identifying opportunities to implementation, ensuring your AI initiatives deliver measurable value and strategic advantage. [Source](https://www.youtube.com/watch?v=e8o5Mdm-Z9g)

[2:35] Bhaskar Borah: Trust is our product [2:35] Devin Stevens: AI 自身无法独立完成任何事情 [8:31] Muthu Sabarethinam: 把数据武器化 [18:46] Lin Yuan: 编写代码的成本非常低，构建原型的成本也极低 [22:43] Devin Stevens: SaaS 并没有死 [25:15] Muthu Sabarethinam: It's dollars... it's not time [31:25] Devin Stevens: 采购外部 AI 工具的企业成功率是完全自主构建内部工具的两倍 [34:33] Muthu Sabarethinam: 工具就在那里，但如果你不花精力去治理，收到一张巨额账单是轻而易举的事 [35:56] Bhaskar Borah: have legs for multiple use cases

[4:00] Devin Stevens: do not do anything by themselves. [22:08] Devin Stevens: SaaS is dead. [25:22] Devin Stevens: cost reduction in back-office operations is treated as far more credible than top-line revenue growth promises from third-party vendor tools. [25:22] Muthu Sabarethinam: soft metrics like time savings are insufficient; financial benefits must translate directly to the P&L statement. [31:25] Devin Stevens: purchasing off-the-shelf AI tools achieved twice the success rate of those attempting internal builds. [34:03] Muthu Sabarethinam: dynamically selects the most cost-effective model per query, cutting compute overhead by 30% to 40%. [36:01] Lin Yuan: As long as an AI application generates more revenue or cost savings than its underlying token consumption, the expenditure remains justified. [36:01] Bhaskar Borah: avoid "fear of missing out" (FOMO), which drives business units to request hundreds of point-solution AI tools for niche tasks. [38:10] Lin Yuan: funneled API traffic through a single, centralized platform layer—where the company manages its own API subscriptions and routing tools—allows all third-party vendor applications to pass their LLM calls through one controlled bottleneck. [38:10] Muthu Sabarethinam: analysis tasks, internal reporting, research, regulatory compliance, and FDA filings are prime, low-risk opportunities. [40:32] Lin Yuan: generated a compounding interest effect on their initial technical investment.

[0:04] Hello everyone. Thanks so much for  showing up this morning. My name is [0:08] John Sheber. Uh I'll be moderating this  panel with uh some very illustrious and [0:15] um experienced uh executives who've been  working on AI agents for a long long time. Uh [0:22] to my right we have Lin Yan. She's the chief AI  officer and EVP of engineering at Newfront. Uh, [0:30] and she joined Newfront after roles at Meta,  Corsera, and Lyft. So, small company, small [0:36] tech companies. Uh, to her right, we have Bascar  Bora, the SVP and head of marketing and data ser [0:43] data service and technology at Dow Jones. Uh, my  former employer. I was a reporter there for many, [0:49] many, many years. Uh, and he previously worked  at city. So another small company with not a [0:56] lot of tech going on there. Uh to his right is  Devin Stevens who leads solutions engineering [1:01] and implementation at Scrunch. Uh who and he  previously held roles at Gainsite and Hearsay [1:08] and after hearsay was acquired by Yex at Yex.  And last but certainly not least we have Muth [1:16] Saberethan. Uh I should have practiced that before  I got on stage. Uh he's the VP of data science and [1:24] business transformation at Minimed uh formerly a  Medtronic company now spun out and prior to that [1:31] he was the VP and chief data officer at Honeywell  connected enterprise where he led the Forge AI [1:39] uh development and implementation. Is that right  sir? There we go. It's a it's a great panel panel. [1:46] Let's give them another round of applause please.  Thanks so much for showing up this morning. [1:53] All right. So, we're going to identify high ROI  AI agent use cases and we're going to take some [2:00] action. But before we do that, um we're going  to talk a little bit about how organizations or [2:07] how these particular organizations structure the  decision-making for how to implement these high [2:12] ROI AI agent use cases. That's a mouthful. Uh  we're going to talk a little bit about some of [2:19] those implementations that they've done. We're  going to talk a little bit about the role for [2:24] humans in all of this process and uh we're  going to get a little weird. Uh so looking [2:30] forward to it. I'm going to kick us off with a  question for the hold panel. Uh there was a a [2:35] report that Gartner did or a little essay that  Gartner did last month where they published a [2:40] piece that recommended investing two times more  in structuring data in governance in training and [2:47] you know investing in people than you should  invest in your AI tools. Uh does that number [2:53] seem high to you all or does that seem right?  Let's go down the row. Lyn, you want to take a [3:00] first crack at this? Should you be investing two  times more in policies and procedures and people [3:06] instead of the AI tools or I think they are I  think they are all equivalently important uh [3:14] but whether it's 2x 4x 8x I'm not sure I think  like uh in our organization is basically how we [3:21] evolved this process like in early days before  the genai we also do like AI projects uh was [3:28] mostly like machine learning based deep learning  based regression an analytics and etc was more [3:34] like the traditional side of like AI and then we  built like we we try to standardize the workflows [3:41] for commercial insurance and the employee benefits  which are traditionally uh more like unstructured [3:48] work uh a lot of like you know manual process  etc and then since the momentum of gen uh chbt [3:56] 3.5 we started realizing okay historically in this  industry we had lots of data but uh we we quite [4:04] often say that they are mostly dirty data. It's  not like a high quality not high structured and [4:10] uh a lot of times like uh the the the the  the you know investment you put in cleaning [4:16] the data is is more than the investment you you  invest in like uh building the actual products. [4:22] But then JAI basically unlocks everything. And no  spoilers. We're gonna get to all that stuff in a [4:27] second. This is just uh like a quick question  about are you spending 2x more on people than [4:32] you are on tools right now or are you finding  yourself spending more on tools than people and [4:36] should that be reversed? Yeah, I mean um at Dow  Jones right we are a powerhouse in business news [4:44] um uh data and information. We have iconic  brands like the Wall Street Journal. So trust [4:51] is our product essentially right and although AI  systems are very good at doing the heavy lifting [4:58] human oversight journalism premium journalism  remains at the core they are responsible for the [5:04] judgment they're responsible for the storytelling  right so that definitely we are putting a lot of [5:09] investments in people processes yeah but not uh  also uh also taking the AI systems along with it [5:17] I agree with that statement yeah Devon how about  do you yes or no? 2x more on people process and [5:23] a little bit less on the AI. Uh I'm a huge yes.  Um and I think you know whether it's two 2 and [5:28] 1/2 three times whatever the specific number is I  think I think there's a huge takeaway there. And [5:32] the takeaway is is that you know AI doesn't do  anything by itself. Um so as much as the tools [5:37] are valuable I think that what you know what we  need to learn from that is it takes changes in [5:42] workflow, it takes changes in strategy. It takes  uh you know scalability around it. And so, um, [5:48] you know, I was reading the other day there's a a  lot of, you know, ROI stories projects fail. Um, [5:53] you know, a lot of those projects fail because  they have the wrong scope or they don't have a [5:58] workflow adjustment. So, um, you know, again, I  think it's a great number whether it's big enough, [6:03] maybe not even big enough. Um, but I think  the message there is clear is is AI doesn't [6:07] do it by itself. We've got to build the the  structure and the tools around it as well. [6:12] Yeah, I think um absolutely true and not because  of two reasons. One is that AI tools are getting [6:19] cheaper. So we don't have to worry too much  about that price points but people roles are [6:23] changing. So we we spending a lot of time  on there's no longer product managers and [6:29] software engineers. They're product engineers. U  you know Andreing talked about the marketing roles [6:34] significantly changing. So that requires a lot  more investment in people in defining new roles [6:39] in the organization that was going to leverage the  AI to do the same processes at a much higher level [6:44] and that's where I think the spend is going for  us. Um Bascar you're on the AI steering committee [6:51] at Dow Jones. Um is that where the buck stops in  terms of implementation? like are you the decision [6:58] makers for how AI will get rolled out or is it  more just sort of a forward-looking organization [7:04] that's trying to gauge where the puck is heading  as it were with AI? Yeah. So, so we have the [7:11] steering committee set up that has multiple uh  stakeholders in it. So, the the AI ideas in at [7:19] DJun actually comes bottoms up from all the teams.  They submit their ideas and business cases to the [7:24] strain committee. Then the strain committee looks  at it from three principles, right? Or essentially [7:31] is it solving a real business problem. Can we  measure ROI uh reliably and uh within a reliable [7:40] time frame and can we do this responsibly or at  scale like and then we help prioritize and take [7:47] it to the seuite uh and get alignment and funding  for the PC's. Right. Yeah. We also so that's where [7:53] it shapes the um direction of AI uh in our company  but also governance right governance is where we [8:02] spend a lot of time uh deliberating within this  training committee because technology changes [8:07] every week our policies and principles cannot  right that's where you have to find the right [8:13] set of policies and principles and um govern using  that so before you even think about ROI you have [8:21] to think about the governance structures that are  in place to make sure that these things get rolled [8:25] out. Does that is that the same is there a similar  structure at each of your organizations? Lynn, [8:31] Mutu, Devon, is that how that works for y'all?  Uh I would say uh data governance probably is [8:39] becoming more and more important simply because AI  requires a lot of context. So we end up stitching [8:45] together a lot of related data together which  I call it as um you know we are essentially [8:50] weaponizing our data right so somebody gets  access to it they get full portfolio of everything [8:55] connected which is very risky and also a lot of  unstructured data like your one drives your entire [9:00] emails your entire teams are being fed into the AI  if you don't have a good strong governance policy [9:06] and your very good security framework to make  sure people only see what they're supposed to [9:11] see when they run their agents you get in a lot  of trouble very fast. So that's what I think is [9:16] the the effort is going for us. And but is there  a single decision-m body that that's responsible [9:21] for saying yes or no at the highest level or do  you devolve that down to group structures? No, [9:29] I I think we have experts informing our senior  most leaders on what the policy should be and [9:35] how we drive it. So at the end of the day we have  the VP of infrastructure and and office 365 and [9:41] uh our data governance teams informing our  leaders but our leaders make the mandate of [9:45] this here's how we're going to run this and who  gets exposure to what Lynn is that similar at your [9:50] organization. Yes. Yes. So we have a collaboration  um uh executives coming from legal, from finance, [9:57] from technology, business stakeholders like  we form a group to make decisions together and [10:02] uh similar to the structure that we also have like  AI projects we call like some principles and we we [10:08] rely on the framework we rely on the principles  first. Some of the principles might include like [10:14] okay human in the loop or we always want to build  something that are responsible and explainable [10:20] etc. And then when we look at okay there's a  100 ideas which ideas you pick it has to be [10:24] aligned with the principles that we set up like  a foundation for like at the very beginning and [10:29] we'll we'll be talking about human in the loop  uh a lot more a little bit later on the panel. [10:34] There's there's a lot to to dive into there but uh  Devon what's how does it work at your organization [10:39] at scrunch? Yeah I would love to tell you we did  it flawlessly and everything's been going perfect [10:44] um but it hasn't. Um we probably you know  we we really start with AI first. you know, [10:48] every time we look at a new tool, we look at a new  solution that, you know, the question is, hey, can [10:53] we solve this with something we already have and a  lot of times, you know, one of our AI tools is the [10:57] the first thing we look at. Um, we let people  go a little wild, uh, over the last 6 months, [11:03] especially the last, you know, 3 months. Uh, and  we've actually had to go back and and pull away [11:07] some of those tools because there there's too much  of it going on. There's too much variation. People [11:12] are kind of losing um that consistency and and a  lot of the scale. Um, so we are kind of thinking [11:18] about it and we we do it really at two different  levels. So we've got kind of an organizational [11:22] level. Um, where we'll go sign off with with the  right committee and the right individuals. Um, [11:27] we do let people do a lot locally. So as we think  about, you know, day-to-day note takingaking and [11:32] email writing and calendar management, um, we do  let people do a lot of those things. But when we [11:37] think about functional initiatives, um we we've  had to go back and put a lot of that governance [11:42] in place because we we got a little bit too happy  and a little bit too fast uh with some of the AI [11:47] tools out there. So, Mutu, with with um Minimemed,  what um and your your AI deployment, what was the [11:56] is there a single metric that that is sort of the  tipping point for a yay or nay on we're going to [12:02] automate or put an agentic system in place or this  is something that that we want humans to handle? [12:08] Um there's not as there's a few metrics. One is  um it's all geared towards better patient care for [12:15] us. that's the most important metric uh because  safety is priority for us in in that front. So [12:20] that is a that's what I call as a feasibility  check but you have to cross that uh before you [12:25] get to the use case but then after that it is  value and I always talk about this is you know [12:31] not incremental we try to be more deliberate about  shaping the idea into something bigger that's [12:36] needle moving like if your CEO doesn't respond to  it as something meaningful like it's probably not [12:40] a big enough uh I call them superhuman use cases  or really um end toend process transformations we [12:47] actually it's better to take a process and rethink  about how this can be run end to end with AI and [12:53] that gives you bigger bang for the buck then  incrementally focusing on parts of the process. [12:57] So those are the two things we end up doing is as  after it passes the reli the feasibility check of [13:02] patient safety then you know much bigger use cases  are better because the amount of work you're going [13:07] to do is almost the same right and so does that  boil down to a metric like time saved for someone [13:14] on in your case like you're we specifically you  use observe AI shout out to a sponsor I guess I [13:21] don't know uh for uh the the services that they  provide and is is that about time saved on calls. [13:27] Is that about having the ability to process more  calls? Is it about like these people didn't die? [13:33] Yeah. Um what what Yeah, it's it started off as  uh you know the most obvious metric is time saved [13:39] on the call. But then as we rethought the entire  process, we found a lot more levers and this is [13:43] why you know as an example we have something  called a smart intake where you know the AI [13:48] takes the first few minutes of every call uh to  screen the patient. So that was automatically [13:53] 5% savings on the call when the agent is not  engaged. We've got 800 agents, 2 million calls [13:57] a year. Then we had call summarization which  typically takes 5 minutes for the agent to [14:02] summarize after the call. AI can do a very good  job of it. Another five, that's another 10% of [14:07] the call. So you can start chipping away. That's  why when you don't re-engineer the entire process, [14:11] you lose those ROI benefits. So it's better to  rethink from a blank sheet of paper how you would [14:16] run it with AI full scale. And then you see so  many spots in the process where you get 5% and [14:21] it just quickly adds up. And and so these aren't  like wholesale replacements of people with with [14:26] agents. This is more, you know, finding the right  pain points that that your employees have. Um [14:34] especially the the ones in the call centers on the  front lines and finding ways to make that their [14:39] lives easier. Yes. And it's it's one system that  drives the entire call. But you're right, it is [14:44] finding those pain points. But we don't we found  that if you let the we could actually easily let [14:49] the AI drive the entire call but we found that the  agents then almost just wait for the AI to prompt [14:55] them the next question. We have to have human in  the loop because it's healthcare and it was not a [15:00] good experience. It's almost like the agents stop  thinking for us. Agents are our experts right? [15:05] So they need to be thinking in the call. We just  need to have the AI support them. That was a much [15:09] better way to get better results and still get to  30 40% productivity. and Bascar, when we talked [15:14] about uh the the ways in which y'all are deploying  AI internally at Dow Jones, you mentioned uh a [15:20] next best action tool and a customer 360 tool.  When when you brought that to the AI or when that [15:28] was brought to the AI steering committee, how  what metrics did y'all use internally to say, [15:33] okay, this is something that we're going to deploy  sort of uh systemwide within Dow Jones? Yeah, I [15:40] mean see for us the metrics have to show up in the  numbers right not just in PowerPoint. So we have [15:48] had to um for different use cases for for this  one. Customer 360 was more of a data foundation [15:57] uh essentially because very early on we realized  that um you know uh without having that that [16:05] foundation in data you cannot build AI intelligent  AI systems right so that's that's critical [16:11] uh because we have so many different brands so  many different business lines and data lives [16:14] in different systems so if you cannot integrate  that create a platform with with a single identity [16:20] graph none of the other agentic work will work.  Yeah. But then what does that enable in terms of [16:28] if we build this out it will enable uh all the  other agentic use cases like next best action [16:35] um AI powered sales assistance and um and customer  service and those are tang those each of those [16:41] downstream use cases have tangible benefits right  business KPIs in terms of either revenue uh or [16:49] expensive or customer acquisition uh rate and all  of that right so how does a foundational piece of [16:55] work like that enables these downstream use cases  and what are the ROI for that right and how can we [17:02] measure them yeah and Devon with with your system  you have a much more I'm not going to say anarctic [17:08] but a much more free form appro approach to to AI  development when does something bubble up from a [17:16] group level to a a sort of companywide initiative  and does it have to be on on sort of a metric of [17:23] of either money your time are those basically  the two things that that we look to save. Yeah. [17:29] Um and you know we're we're still learning a lot  on how to like really surface those. Um so we do [17:34] you know a number of different channels both uh in  different meetings um as well as you know even in [17:40] Slack. Uh we've got a number of different places  where we kind of lead like almost a showcase. Hey [17:44] here's what I've been working on. Here's what my  team's developed. Uh and it's been really good [17:47] to see other people kind of pull in and start  to use those things. Um we do also see kind of [17:52] the the flip side of that where um you know we  we've most folks probably in the room if you do [17:58] support or you know do software solutions we we  obviously turned AI on um in our support channels [18:03] and immediately we were able to shift you know  80% of our support requests um to be solved with [18:08] AI. Uh but what was interesting is it started to  open up challenges in other teams. Suddenly our [18:13] product documentation became a really big deal uh  in making sure that we were giving AI the right [18:19] information. So, um, what what we're seeing is  when it's done well, it really does become almost [18:25] immediately crossf functional, um, and and really  starting that collaboration early, um, has been [18:32] really helpful for us. Uh, we're still kind of  getting the operating rhythm and the cadence, but, [18:37] um, you know, the more exposure, you know, the  more you can get your teams and your your leaders [18:41] up there showing what they've done, um, it's led  us to just a huge, you know, waterfall effect in [18:46] terms of value across the organization. And Lynn,  y'all operate in a highly regulated industry. Um, [18:54] what sorts of how long does something do you does  a an AI tool or potential Agentic solution spend [19:02] in a pilot environment or in a closed environment  before you roll it out systemwide? And and what [19:08] are sort of the the metrics that you use to  determine how successful it needs to be? Like [19:16] does it need to perform 100% all of the time to be  deployed? Is it 99%, is it 85%? Like what what is [19:24] what is a successful use case look like knowing  that there's going to be some failure in the [19:29] system? Right. Right. Like uh insurance is highly  regulated industry unfortunately and there's also [19:35] state level regulations. We have 50 different  even within the US and uh like for us it's like [19:41] uh we I mean like I said like every year there are  quite a lot of ideas out there and then we also [19:49] pair you know engineers and business stakeholders  come up with all of the different ideas we try to [19:56] uh come up with the ones that with when when we  say like a high ROI I think one of the metrics or [20:02] one of the criteria is like you don't touch the  core of the regulation at the very beginning, [20:06] right? You want to work on something that are  easier and also you want to work on something [20:10] that is easy to deploy like a human in the loop.  Like I said, one of the principles that when we [20:16] look at like all the AI ROIs or the AI projects  and then for us it's like u because nowadays like [20:23] prototyping is cheap. It's like we say that we we  say that okay code is cheap, prototyping is cheap. [20:29] So then for us it's like uh very rare. We're like  sitting in the room discussing okay what will be [20:35] you know the long-term like what will be like  something that we invest in a year or two years [20:41] and then you have some impact. Instead what we do  is we just put like all the engineers together. [20:46] Let's prototype this like 20 you know ideas all  together within a week or within two weeks and [20:53] then we use like a very tangible prototypes and  discuss with our stakeholders discuss with those [20:59] humans who are going to be in the loop and then  we calculate like the potential 3 months 6 months [21:07] one year uh business metrics like it's either  can potentially help us build some advantage [21:13] in specific uh segment of the insurance or it  could be like uh helping us like lower the cost [21:19] for certain type of workflows and then we evaluate  obviously security privacy is this like human loop [21:27] good enough and is it like 80% or 90% good enough  for this use case is it easy to achievable and [21:33] etc and then we decide okay these are the things  that we're going to pursue you know in the next [21:38] couple quarters so in in some cases the the  the high ROI use cases are the ones that are [21:45] are sort of least influenced by or or where where  human error is is is is sort of less of a factor, [21:53] right? So, what you're looking for are ways  to make the the the human's job easier across [21:59] the board rather than than sort of building for  an agentic an agentic fully agentic world. Now, [22:08] um, you know, that that makes sense, but Devon,  you know, Scrunch is is sort of building for an [22:14] agentto agent world. What is human in the loop  or what does ROI mean when everything is you're [22:21] talking about automations talking to automations?  Yeah. And we we automate everything we can. Um, [22:28] you know, there was an article and I I don't  even remember where I read it, but a big the [22:31] big headline was SAS is dead. Um, because AI had  come onto the scene. And so um you know as much [22:37] as I think we've experimented we've tried to do  that you know we we don't see a lot of AI tools [22:43] completely replacing our core infrastructure um  you know as we think about both people and systems [22:50] um you know we still need a support tool we still  need a CRM we still need our finance tools um and [22:56] we we don't necessarily you know vibe code or  we don't have an agent that can just take one [23:00] of those off the board um so we really think  about how we can use AI I to really enhance um [23:06] really scale and create a lot of efficiency kind  of around that structure both people and our core [23:12] tools. Um so as much as we've tried a lot of  things um you know I will tell you I don't I [23:17] don't believe SAS is dead um but there is a ton of  value uh in the idea of not replacing the tool not [23:26] replacing the human um but making them you know  50 100 200 times more efficient um doing a lot [23:34] of the same things that they they've already done  before. Yeah. I want to get into build versus buy [23:38] which I will do in a second but before I do that  I wanted to double click on one point and Bascar [23:45] when you think about ROI within Dow Jones so much  of what Dow Jones has been built on is is not just [23:53] human in the loop but real human intelligence  right um reporting uh is essentially a humanto [23:59] human enterprise is that something that's going to  be either automated or way or can be supplemented [24:05] with with any sort of agentic actions? How do you  do you see ROI for like the thousands of reporters [24:14] that or hundreds of reporters maybe now that Dow  Jones employs? Uh and then for is it more of an [24:21] operational thing? I think I might have just  talked myself into a circle, but hopefully you [24:25] got you got where I was going there. Yeah. So, so  see for for our brands trust and and and premium [24:34] journalism still is at the core right AI cannot  replace that. So what we are uh looking at AI or [24:42] approaching AI systems is is as a assistant  essentially right productivity tool for all [24:48] of our qualified uh journalists who spend so much  time who give so much time energy and sweat behind [24:54] uh generating these stories. So AI is uh we are we  have enabled a lot of the productivity tools for [25:02] them which is doing a lot of the heavy lifting but  as I said right uh the journalists still remain at [25:08] the core of storytelling so they can now focus  more on the storytelling uh while um AI does the [25:15] so-called grant work right I I want to get back  to this this notion of what success looks like in [25:22] these implementations and and what the threshold  for success is in these pilots before you um [25:30] you know you you deploy systemwide. Is there a  number like a a hard number for that? Is it 100%, [25:36] is it 80% move through what what what's the number  at minime? We we look at minimum um 20 to 30% ROI. [25:45] It's a threshold for us on the investment. But um  also the scale we are starting to look at how many [25:51] people is this going to reach. uh we want to have  AI used by 50% of our employees if not more. So [25:58] that's at least 5,000 people. So we want to make  sure there is reach because at the end of the day [26:04] um persona based efficiency gains add up very very  quickly. The more people you reach the better. So [26:12] as we just look for those two numbers of can you  reach at least 50 to 100 people? Can you can all [26:17] of them get better at what they do? Um can you  have a 20% ROI minimum? And those are the kind [26:22] of thresholds we at least look for. So 20% ROI  on what though? Like does it matter? Is there [26:27] is there sort of a matrix of of of figures? It  it's both time, it's money, it's whatever. It's [26:35] dollars. Um it [laughter] it's not it's not time.  It and they all eventually it has to show up on [26:40] the P&L. So if my CFO doesn't see the number  Yeah. literally we talk about, okay, when this [26:45] savings is realized, what are they going to do  to reinvest that dollar and where is it going to [26:49] go? So we literally know the dollar that's coming  out where is it going back into the business. We [26:54] talk it we need to be able to count it and that's  essentially has to hit the P&L right Devon Lynn [26:59] is that is that the same it all boils down to  the almighty dollar. Uh yes for dollar numbers. [27:06] Yeah it's it's dollar numbers and also I mean  some in some cases you can measure the dollars [27:12] more directly. It's like okay this is a group of  people you make everyone 30% more efficient. uh [27:17] means that okay we can take more business with a  smaller team etc. In some cases the dollar amount [27:22] is not that direct like okay we want to improve  customer retention and the customer retention will [27:28] potentially help us uh generate how much more  revenue those are more based on some strategic [27:34] calculation it's not like a onetoone mapping. Yeah  yeah definitely the dollar I I will say we kind of [27:41] think about the dollar in two different ways. Um  so when you're thinking about like back office [27:45] internal operations um you know a cost reduction  a savings is a lot more credible to us um than [27:51] what you see a lot more in terms of like potential  growth. Um so if we think there's you know a 20% [27:56] growth in our top line for a new marketing tool  um you know we are far more skeptical of that than [28:02] something that you know may reduce um you know  content creation or our support team or something [28:06] in our implementation or services motions where  you know we feel like that ROI dollar is is a lot [28:11] more likely. It's a lot more stable. It's a lot  more consistent. Basar, what about for y'all? When [28:18] when you think about the um the the the ROI, I'm  sure there's there's dollars there, but when you [28:27] also think about sort of what Agentic systems to  deploy, whether you're building them internally or [28:33] buying them from an outside vendor, what goes into  that that calculation, build versus buy? Uh yeah. [28:41] So that's that's a very hard decision and it's um  we look at it on a case-toase basis. All right. Um [28:48] we do have um licensing agreements with a few of  the um a few of the uh hyperscalers right where [28:58] um we are getting u them to train on our uh  content. We have a very rich content across our [29:05] brands, right? We we we have the deal so that we  can get uh fair compensation for that, right? So, [29:12] and then uh we also have enterprise deals with uh  again some of the big big tech companies uh where [29:19] uh all of our internal um workflows and uh even  outside customerf facing products are being helped [29:28] build by them. But if there are some niche use  cases where customized build is needed where uh [29:36] we are not sure if we buy uh we either compromise  on the trust right credibility and trust which is [29:43] we cannot um and also if we do not feel u our  um human in the loop is getting compromised we [29:51] then go for customization right it's a case to  case basis but um we we do a mix of good mix of [29:59] Again, I think it it sort of comes back to core  competencies, right? Like you you you don't want [30:05] to automate away the core competencies within your  organization. You you want to enhance them. What [30:11] what does that look like in a build versus buy  discussion at Miniad Mudoop? So um it's a it's [30:18] a great question because when we building we're  increasingly finding that the existing SAS vendors [30:26] who've been here for decades don't quite have  the new way of marching us to the fast enough to [30:32] that. So we end up whenever we end up with a buy  decision. We're going with probably new vendors, [30:36] new AI software vendors and observatory is one of  them. You you need to purpose pick a new vendor [30:43] for the new process think that you want to have  and we increasingly spend time on the architecture [30:49] of those solutions because you know they won't  have the fully perfect product today but they [30:54] have a system architecture that lends itself to  getting the AI better in this particular process. [30:59] So that becomes a big part of the decision which  typically um was not before. So we spend more time [31:05] on understanding the extensibility of the vendor's  architecture and how it fits into our AI journey [31:11] and roadmap in the system plugs in place very  nicely. So that's the buy decision. If not we went [31:17] to go to build it's rarely we are going and buying  another Salesforce um module. Yeah. And Devin, as [31:25] a vendor yourself, and Lyn, I'm gonna ask you this  too, with that's sort of AI native, right? What [31:31] is the ROI pitch that you make to your customers  that gets them to Yes. Or is it just you carved [31:38] out this new category in a a new market that that  needs a new solution? Yeah, I mean, it's it's a [31:44] lot of the latter. So, we we are in a new category  and, you know, understanding how AI agents search [31:49] the internet and and how to influence that. Um, so  a lot of what we do is is really building for the [31:55] future, um, you know, kind of thinking about  and and a lot of the companies I work with, [32:00] um, you know, as much as the tooling and the  agent and the technology matters, there's a lot [32:06] of structure, there's a lot of process, there's a  lot of executive visibility that varies in build [32:12] versus buy motions. Um, so even, you know, as  as simple as maybe the AI is just as good or [32:18] the tool is just as good, you really have to think  about how you implement and and how you're putting [32:22] these things in place. Um, MIT did a study last  year. Uh, and I think that they found, you know, [32:29] uh, basically buying AI tools was twice as  successful for the companies that they'd studied [32:36] than trying to build something internally. And I  don't think it's because the the technology was [32:40] that different. I think it's because organizations  approach those solutions very differently. Um, [32:46] so looking at your own company and really  understanding, hey, maybe we can build it, but is [32:50] it going to get the same time, the same attention,  you know, the same, you know, uh, you know, [32:55] reinforcement and support that it would if you'd,  you know, gone and got something through a partner [33:00] off the shelf. Again, it seems we keep coming back  to core competencies, right? And Lynn, how does [33:06] that resonate with you? What are what what's the  pitch that y'all make when you Yeah. So, normally [33:12] we make the pitch to our own customers and the  internal teams cuz we don't sell uh software [33:17] or AI agents. We build to empower our own business  and uh the pitch is really like uh to the business [33:25] leaders or to the business stakeholders. It's  like I mean like I said before and sometimes it's [33:32] pitch sometimes is just like our collaborative  idea. I think bring people like on board early [33:36] on is important and another thing which I very  much agree with is like a lot of times it's not [33:41] just the technology or tool thing is like change  management is really like understand the workflows [33:48] is really understand the impact of those agents  or the AI tools implied on the humans. I think [33:55] those are really really important considerations.  Has has the um the sort of horror stories around [34:03] token costs and the ballooning costs of of  using AI changed your calculus on either [34:09] build versus buy or what ROI looks like with  an AI implementation. Uh that's for everybody, [34:16] but we'll we'll start at the end with you Mu.  Yeah, I mean it certainly plays a huge role, [34:21] but it's not um you have to manage it. it can  get unwieldy very fast. Uh but if you manage it, [34:28] we don't find it very expensive. I think we've  got plenty of choices. We've got our own hosted [34:33] models that we could redirect simple questions  to. Um we've got um ability to use the high praise [34:41] models only for reasoning tasks. So we start  educating people on how to use the fast mode. [34:46] uh we're also increasingly getting products into  the layout landscape that allows for automatic [34:52] selection of models that significantly can reduce  your cost structure by at least 30 40%. So I think [34:57] the tools are available but if you don't put an  effort you can get a really big bill that's easy. [35:04] Yeah, and we we've done the same thing. Um, you  know, we got a little carried away um with carried [35:09] away. Um I'm a super fan of notion. If anybody is  anybody works for notion there, great great tool. [35:14] The notion agents are awesome. Um you know, notion  did a great job uh as as software. You know, [35:19] we turned agents on everywhere. Uh and  then later, you know, kind of found out, [35:23] oh, here here's where the cost comes in  and here's the structure. So, um you know, [35:27] it is really important to think about it's really  important to, you know, kind of understand. Again, [35:30] I think everything's moving more efficient.  we're seeing a lot of those, you know, um, [35:35] unit costs come down. Um, but it is really  important and it's, you know, has to be really [35:40] thought through in terms of what those costs will  be, not just in your pilot, in your PC, but hey, [35:46] if we turn this on for another 50 people, another  hundred or even another thousand. Um, and and [35:51] we're getting a little bit better at, you know,  forecasting and seeing into the future than we [35:56] did in the beginning. Is is the best way to manage  this a hard cap on spend? just like just cut it [36:01] and say you can't spend more than X is that is  that what how are y'all managing cost internally [36:07] Lynn and then Basar as long as the cost generate  more benefit than the cost itself and also like [36:14] me personally I'm a super fan of open models I  think models are just getting cheaper and then [36:19] open models and there they're just quite a lot of  them that you can experiment with and play with [36:26] and then for us like uh we generally build the  layers of the products with model agnostic so that [36:32] you can experiment different uh different models  there. And then in terms of a computing cost or in [36:38] terms like a token cost when you use the frontier  models there are always ways that you can actually [36:44] uh experiment and see the actual cost do the  estimation and then for us I I think the principle [36:50] is basically we always we always build something  that is uh you know generating more revenue or [36:56] saving more cost than the cost of the token by  itself as long as we meet that you know yeah [37:04] I mean there's obvious obviously a cap uh on spend  in terms of like total capex spend for for in the [37:10] budget and all of that right that is there but  it's very important to not get carried out by [37:15] FOMO right um I mean we get so many different  requests uh on can I use this one particular AI [37:22] tool for this particular small use case right  and then there are like hundreds and thousands [37:26] of them across multiple lines of business and  products right so how do we make sure we think [37:33] through how can we select a few but uh use it  at how do you make sure how that you select a [37:39] few what what is the metric there what how how do  you determine that is that made at the board level [37:45] or the AI yeah we as investing committee we look  at does this have uh legs for multiple use cases [37:52] right across business lines we do not want to  replicate u tools and uh systems for small small [38:00] use cases right that your cost will will balloon  significantly if you do that so Those are the [38:04] kind of decisions and then again for more like if  somebody else is doing uh using a particular tool [38:10] why not so that how do you make sure you don't  get carried there with that is very important. [38:14] So yeah, mut maybe I'll just add what we have  found useful is to try to funnel all of the [38:21] uh genai calls into a single platform layer where  you know you have your own API subscriptions, you [38:28] have a tool to manage and let every vendor that  you bring in have those calls go through that and [38:34] that way you get a good filter on you know how to  manage cost because every tool calls the LLMs on [38:39] their own from their own it gets a little unwieldy  and we found that to be very useful way to manage [38:45] So I I I mean we we have about 3 minutes left.  Um I can ask these folks a bunch of questions. I [38:52] have a lot more to ask them, but if anyone in the  audience has a question. Unfortunately, we don't [38:57] have mics, but uh if you would stand up and shout  your question, we will do what we can or I can [39:04] run down. Um I'm just interested in some of your  success success stories and can maybe um each one [39:12] of you share a concrete example of a successful  high ROI use cases that is also lowhanging fruit [39:21] uh without too much of a legal or security risk.  All right. So the best use case without too much [39:29] of a legal or security risk. I feel like we could  have turned this into a case study discussion. Uh [39:34] but we went with a conversation instead. Who wants  to take a crack? What is the Well, why don't we [39:41] we start at the end with you? Yeah. Yeah. I think  all of the I mean just think about what AI is very [39:46] good at doing. Any types of analysis task um you  know reporting trying to um research and file you [39:53] know FDA filings all of those are very very prime  use case. They're very fairly easy to implement. [39:58] So you could actually once you have the the data  in place you could actually have the business [40:04] analyst teams really get 50 60% productive very  fast. I mean everything in everything they do [40:10] um you know consider content creation um it  could be marketing content creation could also be [40:16] uh you know regulatory filings compliance internal  research reports presentations to the executives [40:22] like all of these analysis tasks are prime  productivity uh gainers uh in our in our you [40:27] know scope and it actually touches every persona  in the company just if you enable it so I I I mean [40:32] if I were to sum up like from what I've heard from  the panel it it seems like the The high high ROI [40:40] AI agent use cases that matter are the ones that  have scale that can reach the broadest number of [40:45] employees across the organization and the ones  that that have um an ability to enhance the core [40:53] competencies of the user, whatever that looks  like for your particular organization without [40:58] without sort of touching or or putting those core  competencies at risk in some way, right? Um, was [41:07] there anything that changed in your calculation?  And I'm about to make us run a couple minutes [41:13] over, but did the ROI change after implementation?  Did you find an unexpected return on investment [41:20] that you didn't um, you know, account for  in your initial deployments? Um, uh, Lynn, [41:28] do you want to take a stab at that one? Uh, you  mean higher or lower then? Well, I either one. Um, [41:36] ideally is sort of just an unexpected like higher  I guess. Let's go with positive. Let's accentuate [41:42] the positive. Yeah. Um, I can think of a positive  example is like where we use AI agent to replace [41:50] human beings to ingest the very unstructured  insurance documentation into the system. I think [41:56] the when we initially calculated the ROI was more  about like how many hours it cost the people, how [42:02] many uh how much time or you know it cost or the  tokens it cost the AI and then later you figure [42:08] that with that data layer it can actually generate  a lot more deterministic downstream applications [42:15] that don't require uh genai or don't require like  an open model anymore uh which actually have a [42:22] compound interest on the uh application that we  initially uh built. So that that's a higher than [42:28] anticipated uh uh use case. Pascar, what about  you? I mean higher definitely that uh customer [42:36] 360 I was mentioning once we successfully rolled  out um other business units uh and other use cases [42:43] came to the four ideas then where this had more  legs than we initially thought, right? Which [42:48] always happens when you publicize and successfully  roll out within a small pocket first, right? that [42:53] was like a good positive um um story for us. So  let's say let's add another layer to where the [43:00] high ROI is. It's got to be compounding. It's got  to not touch core competencies and it's got to be [43:05] able to scale across organizations and touch as  many of the employees in your in your company [43:11] as possible. Yeah. We we set out to solve, you  know, what sometimes are really simple challenges [43:16] and then it it actually turned into more value  that that almost wasn't predicted. So, you know, [43:21] note-taking, um, it's it's great. We thought  it would save some time. You're not putting [43:24] as much pin on paper. Um, but those notes became  incredibly valuable into customer health scores. [43:30] We suddenly had a record of every conversation,  who was speaking, and, you know, kind of that ball [43:34] snowball rolling down the hill. You know, our our  support chat was amazing in terms of cost savings, [43:39] but it also helped us identify and we're  far more efficient in our documentation, [43:43] you know, and our data structure now, which has  effects across the whole company. So, you know, [43:48] even though we set out to solve one challenge and  we and we did, um, we saw these other, you know, [43:52] pockets and you can kind of start moving down  those hallways and and find a lot more value [43:56] that you maybe not originally originally scoped  for. And I think we're getting the axe now. So, [44:00] I I think, uh, we're going to have to call  it there. Thank you so much to my panelists. [44:04] Please give them a round of applause. Thank you so  much for showing up at 10:30 in the morning when [44:10] there are other things to do in Vegas. And  thank you to AI4 for an amazing conference.