OpenAI Enterprise Chief Reveals 88% Failure Rate for AI Pilots

OpenAI企业业务负责人透露88%的AI试点项目以失败告终 由于用例选择不当和非技术障碍，企业AI项目的失败率达88%，这要求组织优先考虑高影响、低投入的快速见效项目，并建立严格的评估框架以实现生产部署。 本次演讲由OpenAI大型企业解决方案架构负责人Yogi Pillay发表，并于2025年8月26日由Ai4在YouTube上发布。演讲概述了推动AI走向实际生产部署的实用策略。 约88%的企业AI试点项目因技术未就绪及缺乏内部支持而无法上线，加之安全合规与数据孤岛等非技术阻碍，导致大量项目陷入无休止的POC泥潭。 (0:04) CFO与技术高管不再盲目注资，而是要求AI投入带来明确的底层财务ROI；引入兼顾技术精度与财务ROI的严谨评估框架（evals）是系统性推进生产落地的核心保障。 (5:05) 企业在用例优先级排序上应采用“影响力-工作量”四象限矩阵，优先聚焦高影响、低复杂度的P0级“速赢”（Easy Wins）项目，切忌在AI成熟度不足时直接挑战高难度登月项目。 (8:38) 采用模块化架构设计可避免过早追逐最新模型发布，团队应从损益表（P&L）或核心业务KPI倒推ROI基准，并在早期利用AI辅助测算与敏捷迭代建立组织信心。 (15:12)

Yogi Pillay, Head of Large Enterprise Solutions Architecture at OpenAI, argues that enterprises must escape perpetual proof-of-concept stagnation by pairing rigorous evaluation frameworks with pragmatic, maturity-aligned use-case prioritization tied directly to measurable financial returns. Enterprise AI Hurdles and Pilot Attrition (0:04) According to Pillay, enterprise adoption faces severe friction that prevents models from reaching production, resulting in roughly 88% of AI pilots failing due to technical unreadiness and a lack of internal sponsorship. While creating a minimum viable product is relatively straightforward, deploying at scale exposes governance bottlenecks, security team unfamiliarity with evaluating AI systems versus traditional software, data privacy concerns, and thousands of siloed enterprise data stores. Furthermore, executive leadership—particularly CFOs and CIOs—is shifting away from speculative spending to demand clear proof of bottom-line financial impact. Adoption Misconceptions and Evaluation Rigor (5:05) A primary failure mode among enterprise teams is pursuing technology over business value, such as rushing to integrate bleeding-edge models that may change before an extended launch cycle finishes. Pillay advises building modular architectures so organizations can swap in updated models at deployment time, while using rapid prototyping to engage end-users early and manage change organically. Additionally, large engineering organizations with limited AI experience frequently tackle overly ambitious initiatives before building organizational maturity. To counter these patterns, Pillay cites OpenAI co-founder Greg Brockman's view that evaluation frameworks are indispensable. Implementations require systematic evals measuring both technical baseline benchmarks and financial metrics within a continuous development flywheel, ensuring systems function reliably prior to a production rollout. Prioritization Matrices and Operational Scope (8:38) Before committing resources, organizations must distinguish between deterministic workloads suitable for traditional machine learning or robotic process automation and those truly suited for large language models. Pillay categorizes prospective ideas along an Impact versus Effort matrix. Low-effort, high-impact 'Easy Wins' are designated as P0 priorities to develop institutional capability, while strategic 'Big Bets' involve formal approvals and longer execution horizons. 'Fill-ins' (low impact, low effort) should remain parked for future review, and 'Thankless Tasks' (high effort, low impact) must be avoided entirely. This selection is screened against four pillars: business value, data complexity, risk and regulatory familiarity, and implementation readiness. Operationally, ROI potential divides across three tiers: broad reimagination of business models, growth initiatives that expand revenue lines, and baseline operational efficiency. Pillay advises organizations with minimal production experience to start with operational efficiency to build execution momentum. Establishing Baseline KPIs and Tracking Value (15:12) Translating use cases into financial outcomes requires grounding initiatives in concrete headline Key Performance Indicators derived either from strategic initiatives or worked backward from Profit and Loss (P&L) statements. Pillay illustrates this with a hypothetical enterprise model ('Sam's Sofas') deploying a support agent co-pilot: teams take a measurable baseline metric, such as an average ticket handling time of three minutes, and map expected efficiency gains back to operational cost reductions on the P&L. Rather than treating ROI as a static calculation, organizations should view it as an iterative cycle. Teams can use initial t-shirt sizing (Small, Medium, Large) for early pipeline filtering, employ AI models to establish initial baselines, and refine those projections once runtime performance data is gathered. Pillay emphasizes that publishing these verified, data-driven successes internally reduces organizational resistance and funds future, higher-complexity deployments. Pillay concludes that successful enterprise AI adoption depends on robust upfront evaluations, initial focus on high-impact and low-effort implementations, and iterative financial tracking. While acknowledging the transformative capabilities of modern models, he stresses that long-term enterprise impact requires disciplined execution rather than deploying technology for its own sake.

OpenAI大型企业解决方案架构负责人Yogi Pillay指出，企业AI落地正面临CFO与CIO严格的底线回报审查，约88%的试点因技术准备不足和缺乏内部支持而无法投产；企业必须摆脱无休止的概念验证（POC），通过严谨的评估体系、四象限优先级排序及与损益表挂钩的动态ROI量化方法推进规模化落地。 企业AI落地瓶颈与试点失败根源 (0:04) 当前企业对AI的热情依然高涨，但大多数组织深陷在“永久POC”的循环之中。Pillay援引数据指出，大约88%的AI试点项目最终未能走向生产环境。这背后的核心阻力不仅在于技术准备度不足，更在于非技术层面的障碍：企业安全团队在面对新型AI架构时缺乏评估经验，数据隐私争议频发；大型企业内部成千上万的数据孤岛也严重阻碍了最小可行产品（MVP）的横向扩展。与此同时，CFO与CIO等管理层不再盲目投入预算，转而严格要求每一笔AI支出都必须出具明确的底线回报证明。 技术选型、架构解耦与评估飞轮 (5:05) 在实施策略上，企业常犯的错误包括过早追求过于庞大的目标，或在团队缺乏AI落地成熟度时盲目上马复杂项目。针对OpenAI联合创始人Greg Brockman提出的“评估就是一切”（Evals are all you need），Pillay强调企业必须建立贯穿技术基准与财务指标的严密评估体系（evals），并将其作为持续迭代飞轮的核心。此外，由于AI技术迭代迅速，企业应当警惕“技术凌驾于业务价值”的倾向，不应在生产投产周期长达数月的情况下执念于当下最新模型，而应采用模块化架构实现模型的即插即用；同时应利用快速原型设计尽早引入最终用户，实现自下而上的变革管理。 四象限优先级矩阵与预筛选机制 (8:38) 为了合理规划用例路径，企业首先需要明确大语言模型（LLM）与传统机器学习、机器人流程自动化（RPA）的适用边界，避免将确定性工作流生搬硬套至LLM。Pillay提出了基于“影响与投入”（Impact vs. Effort）的四象限优先级模型：高影响低投入的“速赢用例”（Easy Wins）属于P0级核心切入点，用于快速积累组织信心；高影响高投入的“战略豪赌”（Big Bets）需要长期投入并契合核心战略；低影响低投入的任务可作为候选备选；而低价值高投入的“吃力不讨好任务”则纯属“为了AI而AI”，必须彻底放弃。在正式立项前，团队还需从业务价值、数据与复杂度、风险认知度及实施落地四个支柱展开清单式审查。 将业务用例映射至损益表与衡量基准 (13:48) 量化AI的真实商业回报需要将业务概念转换为具体的财务数据。Pillay将ROI潜力划分为三个维度：重构商业模式的重塑型、拓展收入线的增长型，以及聚焦流程自动化与降本的运营效率型。对于AI成熟度较低的团队，建议从运营效率入手。Pillay以虚构公司“Sam's Sofas”的客服场景为例说明，企业可直接从损益表（P&L）出发倒推运营指标，确立核心KPI（例如平均工单处理时长），并测定基准数据（如单张工单处理需3分钟）。随后，结合AI带来的效率提升假设推导财务收益，在早期阶段可采用S/M/L尺寸分级进行粗略测算，进而系统性评估方案的投资可行性。 动态ROI闭环追踪与组织动能滚雪球 (16:50) ROI的核算不能停留在上线前的一次性静态预估，而应作为贯穿全生命周期的动态迭代飞轮。企业可以在立项探索阶段直接借助AI工具生成初步的ROI测算基准，在应用部署后持续采集运行时性能数据，动态修正财务分析。与此同时，及时向管理层及跨职能部门公开传播早期可量化的成功经验，能够有效消除后续AI系统落地的组织阻力，形成正向飞轮效应，为后续承接更具变革性的高复杂度项目奠定基础。 Pillay总结指出，构建结果导向型AI的核心法则在于：以高质量评估保障部署质量、优先落地高影响低投入用例，并以敏捷方式启动并持续修正ROI模型；尽管严密的财务核算不可或缺，企业领导者亦应认识到前沿AI工具带来的超常生产力跃升，积极将其转化为解决现实业务痛点与服务客户的实际能力。

## Background 本视频由 Ai4 于 2025 年 8 月 26 日发布在 YouTube 平台，主讲人为 OpenAI 大企业解决方案架构负责人 Yogi Pillay。视频围绕“从构想到落地：以结果为导向的实用 AI ROI 评估框架”展开，分享了如何倒推可衡量的业务成果、平衡收益与实施风险，以及将 AI 创意转化为实际商业回报的实战方法论。 ## TL;DR 演讲指出高达 88% 的 AI 试点项目因治理、数据就绪度及组织支持不足而未能投产，强调企业亟需从盲目投入转向底线 ROI 考核。主讲人提出了基于“业务影响与实施难度”的四象限优先级矩阵，建议企业从低投入、高回报的“简易速赢”场景（AGI 象限）切入以积累能力，避开“为 AI 而 AI”的吃力不讨好陷阱。同时，通过结合头条 KPI 与损益表（P&L）对三大价值梯度进行财务量化，并践行“良好评估（Evals）等于成功部署”的原则，实现 AI 价值的持续迭代与复利增长。 ## Atmosphere & style 整场演讲风格专业务实、逻辑缜密，充溢着资深方案架构师在一线大企业实践中提炼出的清醒认知与实战智慧。演讲者穿插着恰到好处的幽默（如调侃‘山姆沙发’和‘为了饭碗命名的 AGI 象限’），使原本严肃的财务测算与企业治理话题变得生动有趣。收尾处由理性的 ROI 框架自然升华至对技术前沿的激昂展望，鼓励大家借助强大工具成为‘超级人类’，兼具严谨的说服力与鼓舞人心的感染力。 ## Key takeaways - [00:04](https://www.youtube.com/watch?v=rLcd2YHf1FI&t=4s) AI 的投资回报率（ROI）并非自动发生，而是需要被系统化工程化设计的，用例优先级排序与财务 ROI 测算是并行开展的核心支柱。 - [02:06](https://www.youtube.com/watch?v=rLcd2YHf1FI&t=126s) 高达 88% 的企业 AI 试点项目未能走向生产环境，阻碍主要来自 GRC 治理、安全评估机制缺失、数据就绪度不足以及缺乏高层业务赞助等非技术因素。 - [05:05](https://www.youtube.com/watch?v=rLcd2YHf1FI&t=305s) 企业在 AI 落地中常陷入忽视评估体系（Evals）、脱离成熟度盲目立项、紧耦合最新模型而忽视业务价值，以及将变革管理当成事后补救等四大误区。 - [05:05](https://www.youtube.com/watch?v=rLcd2YHf1FI&t=305s) 评估驱动开发（Good eval equals good deployments）：上线前需建立技术基准与财务 ROI 基准的双重评估飞轮，以确保系统稳定迭代。 - [05:05](https://www.youtube.com/watch?v=rLcd2YHf1FI&t=305s) 在架构设计上应解耦模型与业务，前期聚焦业务价值流与实施路径，待投产前夕再灵活即插即用接入当时最优的底层模型能力。 - [09:14](https://www.youtube.com/watch?v=rLcd2YHf1FI&t=554s) 企业在筛选用例前必须明确 AI 适用边界，区分 LLM、传统机器学习与确定性 RPA 自动化的不同应用场景，避免盲目引入生成式 AI。 - [10:17](https://www.youtube.com/watch?v=rLcd2YHf1FI&t=617s) 利用四象限矩阵分类用例：P0 级优先攻坚高影响低投入的“简易速赢”（Easy Wins），严格论证“战略押注”（Big Bets），暂存“补充填补”（Fill-ins），坚决杜绝“吃力不讨好”（Thankless Tasks）。 - [11:41](https://www.youtube.com/watch?v=rLcd2YHf1FI&t=701s) 立项审查可围绕业务价值、复杂度与数据就绪度、熟悉度与风险、落地推进路径四大维度，快速定位处于高收益短耗时“甜点区”的 AGI 象限用例。 - [13:48](https://www.youtube.com/watch?v=rLcd2YHf1FI&t=828s) AI 价值回报分为运营自动化、业务增长和业务重塑三大梯队，初期团队应从自动化速赢起步，自下而上逐步强化企业 AI 成熟度。 - [13:48](https://www.youtube.com/watch?v=rLcd2YHf1FI&t=828s) 量化商业价值需锚定损益表（P&L）或核心战略，确立关键头条 KPI（如工单处理时长 AHT），结合基线数据与提效假设测算财务节省。 - [16:50](https://www.youtube.com/watch?v=rLcd2YHf1FI&t=1010s) ROI 估算是一个持续迭代校准的过程，企业应在内部积极共享可量化的阶段性战果，以形成滚雪球般的推进势头并释放团队潜能。 ## Detailed walkthrough ### [00:04](https://www.youtube.com/watch?v=rLcd2YHf1FI&t=4s) 企业 AI 落地的挑战与演讲路线图 OpenAI 大企业解决方案架构团队负责人 Yogi Pillay 展开了关于企业如何构建并扩展 AI 能力（尤其是基于 OpenAI API）的分享。基于跨行业、跨领域的丰富客户实践，核心议题聚焦于如何量化 AI 的商业价值与投资回报率（ROI）。 OpenAI 的愿景是打造造福全人类的通用人工智能（AGI），而实现这一愿景的关键环节，在于推动大型企业逐步迈向“AI 原生化”（AI Nativity）。然而在现实中，大量企业在立项阶段就选错了用例形态与优先级，导致许多 AI 项目中途夭折，始终无法上线投产。 ```mermaid flowchart LR accTitle: AI 企业落地演进路径 subgraph S1["现状与痛点"] A["行业现状与非技术阻碍"] --> B["常见误区与避坑策略"] end subgraph S2["落地与度量"] C["用例优先级排序"] --> D["ROI 与商业价值量化"] end B --> C ``` - 演讲涵盖四大主线：企业 AI 现状、常见认知误区与规避建议、ROI 与商业影响力的评估框架、总结与行动倡议。 - 虽然分享逻辑上按线性展开，但用例优先级评估与 ROI 测算在实际操作中是“一枚硬币的两面”，需要并行开展。 --- ### [02:06](https://www.youtube.com/watch?v=rLcd2YHf1FI&t=126s) 企业落地现状：88% 试点失败与财务 ROI 压力 尽管企业对 AI 抱有极高热情，但绝大多数仍深陷在“无休止的概念验证（Perpetual POCs）”泥潭中。阻碍企业走向生产环境的核心往往不是技术本身，而是非技术层面的多重障碍： ```mermaid flowchart TD accTitle: 企业 AI 落地阻碍与推进闭环 subgraph Blockers["四大非技术阻碍"] G["治理与数据合规"] S["安全漏洞认知不足"] D["跨部门数据孤岛与就绪度差"] P["内部支持与组织就绪度缺失"] end subgraph Outcomes["导致结果与转变"] F["88% Pilot 无法投产"] ROI["CFO / CIO 强化底线 ROI 考核"] M["聚焦核心价值，加速构建飞轮"] end Blockers --> F F --> ROI ROI --> M ``` 1. **治理、风险与数据隐私（GRC）**：缺乏对组织内部数据合规与隐私要求的系统规划。 2. **安全评估机制缺失**：传统安全团队常常不知如何评估企业级 AI 解决方案，无法准确识别 AI 架构相较于传统软件系统的特有漏洞与风险点。 3. **数据就绪度（Data Readiness）不足**：搭建一个 POC 或 MVP 非常容易，但在大型复杂组织中，成千上万的数据分散在各个部门的数据孤岛中，难以在生产级规模下稳定调用。 4. **内部缺乏业务赞助人（Sponsorship）**：统计表明，高达 **88% 的 AI 试点项目最终未能走向生产环境**，其根源往往在于组织就绪度欠缺与缺乏高层持续推动的动力。 与此同时，行业也迎来了积极的转变：CFO 和 CIO 不再盲目“用钱砸向 AI”，而是对 AI 投入产出比进行极为严格的商业审视，更加注重投入预算对企业底线收益（Bottom Line）的真实贡献。只要企业掌握科学的评估框架，即使在拥有复杂遗留系统的大型企业中，AI 用例也能成功落地投产。 --- ### [05:05](https://www.youtube.com/watch?v=rLcd2YHf1FI&t=305s) 企业 AI 的四大常见误区与应对策略 在 AI 落地过程中，企业极易陷入四个常见误区： | 常见误区 | 核心问题表现 | 应对策略与解法 | |:--- |:--- |:--- | | **忽视评估机制（Evals）** | 上线前未建立基线与精度标准，缺乏投产后度量方案 | 践行 Greg Brockman 所说的 *"Evals are all you need"*；将技术指标与 ROI 指标同时纳入评估飞轮中 | | **脱离成熟度盲目立项** | 缺乏 AI 投产经验（如 2 万人研发团队未上线过 AI）却直奔超高难度场景 | 优先以“速赢（Quick Wins）”项目起步，建立组织信心与经验，以此补贴和撬动长远宏大目标 | | **以技术为中心而非价值驱动** | 沉迷于使用最新模型，但实际项目周期长达 8 个月 | 模块化设计架构以解耦模型与业务，前期聚焦业务价值与实施路线图，投产时灵活“即插即用”最新模型 | | **将变革管理当成事后补救** | 忽视人员习惯与流程变革，最终导致大规模推广受阻 | 利用 AI 快速迭代与原型制作的优势，在构建阶段即引入终端用户实时反馈，实现人机协同共创 | - **评估驱动开发**：良好的评估体系等于良好的系统开发。技术基准（Technical Benchmarks）与财务指标（ROI Benchmarks）共同构成了项目全生命周期的评估输入。 - **解耦模型迭代风险**：底层模型演进极快，如果架构紧耦合于某个特定模型，开发周期尚未结束模型就已过时。合理的方案是架构模块化，聚焦业务价值流，待投产前夕直接接入当时最优的模型能力。 - **以原型驱动变革管理**：技术往往是最容易解决的部分，真正的难点在于“人的采纳（Adoption）”。在开发初期就要利用 AI 敏捷重塑输入与输出流程，将用户体验与流程改造无缝嵌入开发过程。 --- ### [08:38](https://www.youtube.com/watch?v=rLcd2YHf1FI&t=518s) 用例生命周期：从构想立项到 ROI 量化 从 AI 概念到生产落地的完整生命周期，第一步且最关键的一步是**构想与范围界定（Ideation and Scoping）**，随后依次推进至开发、测试以及生产阶段。 ```mermaid flowchart LR accTitle: AI 用例全生命周期 A["构想与范围界定<br>(Ideation & Scoping)"] --> B["开发构建<br>(Development)"] B --> C["测试与评估<br>(Testing & Evals)"] C --> D["生产投产与扩展<br>(Production & Scale)"] ``` 在立项之初，用例优先级的排序与财务维度的 ROI 量化紧密交织。企业必须在定义用例价值的同时，将潜在商业回报精准映射为实际的财务数字，以此指导后续开发资源的倾斜与投入。 ### [09:14](https://www.youtube.com/watch?v=rLcd2YHf1FI&t=554s) AI 落地优先级的核心前置条件：从认知到规模化运作 在深入讨论具体的落地评估之前，评估（Eval）、生产上线以及持续维护是不可或缺的基石。在构建优先级排序框架时，首要且最关键的第一步是“真正理解 AI”。企业必须首先明确：组织内部是否真正清楚这项技术的最佳应用场景是什么？并非所有业务流程都适合大语言模型（LLM）。有些场景可能使用传统的机器人流程自动化（RPA）效率更高，而另一些场景可能过于确定性（Deterministic Workload），引入生成式 AI 反而并不合适。因此，通过内部培训、教育和技能赋能，让团队深刻理解传统机器学习（Traditional ML）与大语言模型各自的适用边界，是筛选出正确用例的前提。 在建立技术认知之后，企业需要明确哪些场景值得规模化扩展。整体推进路径需要从摸清现状（Current State）出发，构思并定义出清晰的终态（Defined End State），从而绘制出一条明确的演进路径图。一旦这一飞轮持续运转起来，企业才能真正跨入“运营级 AI（Operational AI）”的成熟阶段。 ```mermaid flowchart LR accTitle: AI Prioritization and Scaling Lifecycle subgraph S1["认知与使能"] A["理解 AI 适用边界<br/>(LLM vs 传统 ML / RPA)"] end subgraph S2["规划与映射"] B["摸清现状 (Current State)"] --> C["定义目标终态 (End State)"] end subgraph S3["规模化飞轮"] D["用例优先级排序"] --> E["进入生产与评估"] E --> F["运营级 AI (Operational AI)"] end A --> B C --> D ``` --- ### [10:17](https://www.youtube.com/watch?v=rLcd2YHf1FI&t=617s) 用例分类的四象限矩阵：识别速赢用例与吃力不讨好的陷阱 在实际评估 AI 用例时，通常可以将其归入四大象限，从而帮助团队做出清晰的取舍： ```mermaid flowchart TD accTitle: AI Use Case Prioritization Matrix subgraph HighImpact["高业务影响 (High Impact)"] EW["🌟 简易速赢 (Easy Wins)<br/>- P0 最高优先级<br/>- 低投入、高回报<br/>- 建立团队 AI 能力"] BB["🚀 战略押注 (Big Bets)<br/>- 核心战略重点<br/>- 上线周期长、流程严谨<br/>- 产生深远变革"] end subgraph LowImpact["低业务影响 (Low Impact)"] FI["⏸️ 补充填补 (Fill-ins)<br/>- 难度低、影响小<br/>- 放入暂存区 (Parking Lot)<br/>- 待优先级变更时再评估"] TT["🚫 吃力不讨好 (Thankless Tasks)<br/>- 投入极高、价值极低<br/>- 为 AI 而 AI<br/>- 必须坚决规避"] end EW --- BB FI --- TT ``` - **简易速赢（Easy Wins）**：属于高影响、低投入的用例，上线门槛较低。这些是团队的 **P0 级** 任务，必须绝对优先安排。尤其是在团队刚开始构建 AI 能力（AI Muscle）的初期阶段，速赢用例是开启 AI 之旅的最佳突破口。 - **战略押注（Big Bets）**：直接绑定企业的核心战略重点。虽然上线周期较长，但其战略价值极其关键。这类项目依然需要重点推进，但内部需要走一套更正式的立项和论证流程。 - **补充填补（Fill-ins）**：实施难度不高，但业务影响力也相对有限。这类用例应当先放进“暂存区”（Parking Lot），随着业务重点在未来的调整，再回头评估是否推进。 - **吃力不讨好（Thankless Tasks）**：这是必须坚决远离的陷阱象限。它们通常需要极高的开发与维护投入，带来的业务价值却极低。这类项目本质上就是“纯粹为了做 AI 而做 AI（AI for AI's sake）”，企业应尽一切可能坚决规避。 --- ### [11:41](https://www.youtube.com/watch?v=rLcd2YHf1FI&t=701s) 优先级核对清单与“AGI 象限”网格定位 在最终批准立项某个具体用例之前，团队必须通盘审查一份结构化的核对清单。虽然清单内容看起来繁多，但在实际操作中，团队通常只需花费 30 到 40 分钟即可完成重点用例的初审，并在后续迭代中逐步完善。核对清单主要涵盖四大维度： 1. **业务价值（Business Value）**：明确该用例的预估投资回报率（ROI）是多少，以及如何科学衡量其业务影响。 2. **复杂度与数据就绪度（Complexity & Data）**：评估是否具备对关键数据系统的访问权限、数据策略和架构是否易于被 LLM 消费和调用，以及业务工作流是否具备明确定义。 3. **熟悉度与风险（Familiarity & Risk）**：从技术适配度来看，该场景是否是大模型擅长解决的问题，同时必须审查是否存在知识产权（IP）或合规监管风险。 4. **落地推进路径（Path to Implementation）**：明确从现状到生产环境的具体路线图，确定项目上线后的业务负责人（Owner），以及项目长期的生命周期与维护规划。 在完成上述评估后，可以将所有候选用例绘制到以“上线耗时（X 轴）”和“业务影响（Y 轴）”构成的二维网格上。演讲者幽默地将左上角那个耗时最短、收益最高的黄金区域命名为 **AGI 象限**，并打趣道：“看到了吧？这个名字完全是我自己想出来的，没用任何 AI，这纯粹是为了保住我的工作（#jobsecurity）。”这一高价值、低耗时的区域正是企业开启 AI 旅程的“甜点区”（Sweet Spot），特别适合处于 AI 成熟度初期、正在逐步积累经验的团队。 --- ### [13:48](https://www.youtube.com/watch?v=rLcd2YHf1FI&t=828s) 量化业务回报：三大价值分桶、头条 KPI 与“山姆沙发”案例 确定了用例优先级后，下一步是将预期价值拆解为具体的“真金白银（Dollars and Cents）”，为决策引入更严谨的量化维度。AI 的价值回报通常可以分为三大梯队： - **业务重塑（Reimagination）**：涉及组织内重大战略变革的大型押注。 - **业务增长（Business Growth）**：周期相对较短、影响显著、能够直接推动业务增长的用例。 - **运营自动化（Automating Operations）**：见效最快、容易落地的生产力速赢用例。 用例的选择与企业的 **AI 成熟度（AI Maturity）** 密切相关。如果团队已经成功将大量模型推向生产环境，就可以大胆向“业务重塑”迈进；但如果是第一次尝试，或者团队刚经历过失败用例的阵痛，最佳策略是从“运营自动化”开始，自下而上逐步强化团队能力。 ```mermaid flowchart TD accTitle: Value Estimation and Sizing Flow subgraph Step1["1. 锚定价值来源"] A["损益表 (P&L) 或战略举措"] end subgraph Step2["2. 确立头条 KPI (Headline KPI)"] B["例如：客服工单平均处理时间 (AHT)"] end subgraph Step3["3. 快速规模估算 (Sizing)"] C["设定内部基线 (如 3 分钟/单)"] --> D["引入 AI 提效假设 (提升百分比)"] D --> E["反推财务影响 (T 恤尺码 / 美元量化)"] end A --> B --> Step3 ``` 在量化过程中，首先要确立衡量成效的 **头条 KPI（Headline KPI）**。演讲者以一家虚构的公司 **“山姆沙发”（Sam's Sofas）** 为例进行讲解，并开玩笑说“这听起来像是山姆·奥特曼（Sam Altman）的副业”。企业可以通过损益表（P&L）或核心战略举措来反推这些指标。即使在立项初期只能估算出类似“T 恤尺码（S/M/L/XL）”这样的大致级别，也足以支持决策。 在“客服支持 Copilot”的用例中： - **头条 KPI** 设为工单平均处理时长（Average Ticket Handling Time）； - **现状基线** 基于内部数据测算为每张工单 3 分钟； - **效益测算** 结合 AI 带来的提效预期以及 P&L 中的人力成本数据，精确计算出整体财务节省。 演讲者特别提醒，虽然越往“业务重塑”层面走，纸面上的潜在 ROI 往往高得惊人，但切忌盲目掉入只做宏大项目的陷阱，必须始终结合团队的实际成熟度稳步推进。 --- ### [16:50](https://www.youtube.com/watch?v=rLcd2YHf1FI&t=1010s) 价值追踪三原则与总结倡议 在价值追踪的执行层面，演讲者总结了推进 AI 投资回报的核心实践：优先聚焦“低投入、高影响”的用例；将 ROI 估算视为一个持续迭代的过程，甚至可以借助 AI 工具来生成初始的 ROI 基线，再不断校准；同时，务必在内部积极分享可量化的阶段性战果，向管理层与其他团队同步成功经验，从而形成滚雪球般的推进势头。 演讲最后提炼了三条最核心的精髓： - **出色的评估等于成功的部署（Good eval equals good deployments）**：必须在此投入充足时间并切实做好。 - **聚焦高影响、低投入的用例**：在能力建设期找准切入点。 - **从简单起步，随时间不断修正 ROI 分析**。 演讲者在结语中呼吁，ROI 和业务影响固然至关重要，但当下也是技术史上最令人振奋的时代之一。我们所掌握的技术工具前所未有地赋予了人们超越常人的超级能力（Superhumans）。企业与技术人员应当充分利用这些强大工具，大胆尝试，构建出真正具有突破性的创新成果。 ### [18:06](https://www.youtube.com/watch?v=rLcd2YHf1FI&t=1086s) 释放超级人类潜能：将 AI 构想转化为现实 在演讲的结尾部分，演讲者向现场听众发出了充满激情的行动号召。他强调，当前正是利用人工智能技术重塑业务、赋能团队以及突破创新极限的绝佳时机。无论是为了更好地服务和发展自身社区，为客户带来令人惊喜的卓越体验，还是最终将那些在心中构思已久、梦寐以求的创新点子落地实施，企业与个人都应当紧紧抓住这一时代机遇。 演讲者指出，AI 技术赋予了大家前所未有的能力，这正是每个人去超越极限、真正成为“超级人类”（superhumans）的时刻。他鼓励大家将前面所讨论的投资回报率（ROI）框架与实操策略付诸实践，从明确的业务成果出发，脚踏实地地把 AI 的潜力转化为切实的商业影响力和 compounding（复利）价值。最后，演讲者向在场的所有听众表达了诚挚的感谢。 - **核心行动指南**：积极拥抱 AI 工具，提升团队效能，服务客户与社区，并将积压的创新构想快速落地。 - **金句结语**："This is the moment to really go be superhumans."（这正是真正去成为超级人类的时刻。） ## Notable quotes > [05:05](https://www.youtube.com/watch?v=rLcd2YHf1FI&t=305s) Evals are all you need — Greg Brockman (引用) > [11:41](https://www.youtube.com/watch?v=rLcd2YHf1FI&t=701s) 看到了吧？这个名字完全是我自己想出来的，没用任何 AI，这纯粹是为了保住我的工作（#jobsecurity）。 — Yogi Pillay > [13:48](https://www.youtube.com/watch?v=rLcd2YHf1FI&t=828s) 这听起来像是山姆·奥特曼（Sam Altman）的副业 — Yogi Pillay > [16:50](https://www.youtube.com/watch?v=rLcd2YHf1FI&t=1010s) Good eval equals good deployments — Yogi Pillay > [18:06](https://www.youtube.com/watch?v=rLcd2YHf1FI&t=1086s) This is the moment to really go be superhumans. — Yogi Pillay > [02:06](https://www.youtube.com/watch?v=rLcd2YHf1FI&t=126s) 88% 的 AI 试点项目最终未能走向生产环境 — Yogi Pillay > [10:17](https://www.youtube.com/watch?v=rLcd2YHf1FI&t=617s) 纯粹为了做 AI 而做 AI（AI for AI's sake） — Yogi Pillay > [16:50](https://www.youtube.com/watch?v=rLcd2YHf1FI&t=1010s) 出色的评估等于成功的部署 — Yogi Pillay ## Mentioned OpenAI, Yogi Pillay, Greg Brockman, Sam Altman, 山姆沙发 (Sam's Sofas), OpenAI API, RPA, LLM ## Video description Yogi Pillay, Head of Large Enterprise Solutions Architecture, OpenAI Abstracts AI ROI isn’t automatic; it’s engineered. This session outlines how to work backward from measurable outcomes, estimate upside, effort, and risk, and stack-rank use cases using a simple value/feasibility lens. Expect concrete metrics for tracking progress, examples drawn from large enterprises (without specifics), and tactics for sustaining improvement beyond launch. Attendees will leave with reusable frameworks, alignment questions, and a playbook to translate AI ideas into compounding business results fast. [Source](https://www.youtube.com/watch?v=rLcd2YHf1FI)

[5:05] Greg Brockman (引用): Evals are all you need [11:41] Yogi Pillay: 看到了吧？这个名字完全是我自己想出来的，没用任何 AI，这纯粹是为了保住我的工作（#jobsecurity）。 [13:48] Yogi Pillay: 这听起来像是山姆·奥特曼（Sam Altman）的副业 [16:50] Yogi Pillay: Good eval equals good deployments [18:06] Yogi Pillay: This is the moment to really go be superhumans. [2:06] Yogi Pillay: 88% 的 AI 试点项目最终未能走向生产环境 [10:17] Yogi Pillay: 纯粹为了做 AI 而做 AI（AI for AI's sake） [16:50] Yogi Pillay: 出色的评估等于成功的部署

[5:05] Greg Brockman: Evals are all you need [8:38] Yogi Pillay: No AI, #jobsecurity [15:12] Yogi Pillay: it looks like Sam Altman's side gig maybe [17:24] Yogi Pillay: Good Evals Equal Good Deployments [17:24] Yogi Pillay: Focus on High-Impact, Low-Effort Use Cases [17:24] Yogi Pillay: Start Simple and Refine Iteratively [17:24] Yogi Pillay: at times feel like superhumans

[0:04] My name is Yogi Ple. I lead the [0:06] enterprise solutions architecture team [0:08] over at OpenAI. Our team is primarily [0:11] responsible for helping customers build [0:13] and scale, especially using our API. And [0:17] I'm really excited to spend the next 20 [0:18] minutes or so sharing some learnings and [0:21] insights that we've gathered from [0:22] working with customers across verticals [0:24] and segments. Um especially as it [0:27] relates to ROI and business impact and [0:29] trying to quantify some of the value [0:31] that we're all hoping to gain from from [0:33] AI. [0:35] So our mission at OpenAI is to bring AGI [0:38] to the world that benefits all of [0:40] humanity. And I think a big part of this [0:42] is really unlocking AI in the enterprise [0:45] and helping especially large enterprise [0:47] organizations to get to this place of AI [0:50] nativity. But a challenge that we're [0:52] seeing is uh companies picking and [0:55] selecting the wrong use cases or or [0:57] prioritizing the wrong shapes of ideas [0:59] to focus on resulting in many use cases [1:02] that never reach production and uh fail [1:04] to launch. So over the next 20 minutes [1:07] or so, I'm going to share with you [1:08] learnings from the thousands of [1:09] deployments that we've worked with uh [1:12] with again cross with customers across [1:14] various segments across various [1:15] industries to share some of the um some [1:18] of the key insights and learnings that [1:20] we've taken back. Quick road map. We [1:23] have a lot to cover and it's just 20 [1:24] minutes. So I'll try to um squeeze in as [1:27] much as I can here, but we'll start off [1:28] by talking through and contextualizing [1:30] you all with what we're seeing in the [1:32] enterprise right now um and the state of [1:34] AI. We'll then then talk talk through [1:36] some of the misconceptions as it relates [1:39] to AI and some some um common pitfalls [1:41] that we see uh customers fall through. [1:44] Um and then we'll talk through how to [1:46] actually think about ROI and business [1:48] impact. And in the talk today, we'll [1:50] kind of go through this in a linear way, [1:52] but just know that this is two sides of [1:54] the same coin. Um and and in in reality, [1:57] you're probably going to be doing this [1:58] in parallel. And so we'll talk through [2:00] some frameworks and share some examples [2:01] there. and then we'll close with a quick [2:03] summary and call to action. [2:06] Cool. So some of the trends that we're [2:08] seeing um right now within um AI and the [2:11] enterprise is we're still seeing a lot [2:13] we're seeing a lot of excitement but a [2:15] lot of enterprises still remain stuck [2:18] early on in you know in this state of [2:21] perpetual PC's and I think a big reason [2:23] for this and we'll talk more about this [2:25] is is not selecting the right use cases [2:27] but also all of the non-technical [2:29] hurdles that are involved in taking AI [2:31] use cases into production. So [2:34] specifically when we're thinking about [2:35] governance, risk and security, those are [2:37] generally big blockers internally, not [2:40] thinking through what the data privacy u [2:42] concerns of the organization are. Um [2:44] security for large enterprise especially [2:47] continues to be a big blocker where [2:49] security teams aren't even sure exactly [2:50] how to evaluate what an enterprise [2:52] solution might look like. And so a lot [2:54] of uh a lot of the time spent is [2:56] enabling these teams to think about [2:57] where the actual vulnerabilities lie [2:59] within AI systems compared to [3:01] traditional software systems. Data [3:03] readiness is a huge bucket and a huge [3:06] point of contention as well because uh [3:08] it's really easy to spin up a PC. It's [3:11] really easy to spin up an MVP. But as we [3:13] try to scale this and put it into [3:15] production, we realize that you know [3:17] large organizations have thousands of [3:19] data stores in across silos in various [3:21] um um departments and various places [3:23] within the organization. Uh so as we're [3:25] going to talk about spending the time [3:27] early on to really understand your [3:29] landscape, evaluate your maturity and [3:31] then decide on what use cases make sense [3:33] is so critical. And we'll talk through [3:35] some strategies in a moment as to how we [3:37] can do that. [3:38] As I mentioned, 88% of pilots fail to [3:40] get to production for the reasons we [3:42] talked about, but also because of uh [3:45] lack of readiness and lack of internal [3:47] sponsorship. And we'll also talk through [3:49] some strategies around how we can create [3:51] enthusiasm internally to build the [3:53] momentum that helps drive a lot of these [3:56] uh use cases into production. [3:59] The next one here, I think, is actually [4:00] a net positive. We're starting to see a [4:02] lot more ROI pressure being placed on [4:05] the types and shapes of use cases that [4:07] we're focusing on. I think just a couple [4:09] years ago it was this world where money [4:12] was thrown at the problem and and [4:14] everyone thought if we spent enough we [4:15] would just become AI and we all know [4:17] that that's not the case. So I think [4:19] rightfully so a lot of CFOs and CIOS are [4:22] placing a lot more diligence on [4:24] understanding exactly what those dollars [4:26] in are going to equate to from a bottom [4:29] line perspective. And so now more than [4:31] ever, it's really really important to [4:32] really focus and prioritize the use [4:35] cases that are actually going to impact [4:36] your bottom line. Lastly here with the [4:39] organizations that are focusing on these [4:41] things and you know going through the [4:42] steps to prioritize and focus on the [4:44] right use cases, we're seeing momentum [4:46] build and we're starting to see a lot [4:48] more of these uh use cases actually land [4:51] in production even in in fairly complex [4:53] organizations that have robust systems. [4:55] So, I think we're we're starting to see [4:56] the light at the end of the tunnel, but [4:58] we still have a long way to go. And I [5:00] think frameworks like we'll cover today [5:01] will help get us there in an accelerated [5:03] fashion. [5:05] Next, we'll talk through some misconcept [5:07] misconceptions as it relates to AI. [5:09] Specifically, some of the pitfalls to [5:10] avoid and then I'll share some antidotes [5:12] for what maybe to do instead. [5:15] First and foremost, evaluations. I [5:18] cannot emphasize how important [5:19] evaluations are in uh ensuring a [5:23] productive launch uh at the end of your [5:25] cycle. Thinking through what your [5:27] baselines need to be, thinking through [5:28] what uh uh your accuracy levels need to [5:31] be, thinking through how you're going to [5:32] measure the system once it's in [5:34] production is so critical. Greg Brockman [5:37] famously tweeted, "Eval are all you [5:39] need." And I think that that hasn't [5:40] stand um stood more true. [5:43] Um generally good eval equals good [5:45] development. And with the ROI frameworks [5:48] that we'll cover today, this can be [5:49] another input into your evaluation [5:51] pipeline as well. So that not only are [5:53] you measuring your technical benchmarks, [5:55] but you're also measuring your ROI [5:57] benchmarks in this evaluation flywheel. [6:00] The next one here is overly ambitious [6:02] goals without the organizational AI [6:04] maturity um in place already. A lot of [6:07] organizations will make the mistake of [6:10] focusing on the highest impact um use [6:13] case. And while that's a great uh [6:15] ambition, if you have, you know, let's [6:17] say a 20,000 person dev team, but have [6:20] never taken an AI product into [6:22] production, that's going to be a bit [6:23] challenging to to take on something so [6:25] big. And so the recommendation here is [6:28] really to focus on some of these quick [6:29] wins. And we'll talk more about that in [6:31] a moment. And from a strategy [6:33] perspective, you can focus on these [6:34] quick wins to subsidize some of these [6:36] ambitious goals. So the the answer here [6:38] isn't to totally uh ignore them, but [6:41] think strategically about how and when [6:42] you want to prioritize these big [6:44] ambitious moonshots. [6:47] Another one that we're seeing quite a [6:49] bit is tech versus value first projects. [6:52] I think we all fall into the trap AI is [6:54] cool. It's a really fun thing to to be [6:56] working on. And so I can't tell you how [6:58] many times I've had conversations with [7:00] enterprise customers where they want to [7:02] leverage the latest model to do [7:03] something. But in reality, we're 8 [7:05] months away from from production. And so [7:08] I think taking a step back, [7:10] understanding what value that AI is [7:12] going to bring to the organization, then [7:14] thinking about what the roadmap to produ [7:15] production is going to be and building [7:17] out your architecture in a module away [7:19] so that you can leverage whatever the [7:20] newest tech is when it's ready to go [7:22] into production is the right approach [7:24] here. So focus less on the tech because [7:26] it's changing so fast. Focus on the [7:28] value. Focus on what the roadmap's going [7:29] to be and then plug and play once you're [7:31] ready to go into production based on [7:32] whatever the best model is at that time [7:34] and point. [7:36] And then the last one here which I think [7:37] is is probably one of the most critical [7:39] ones is uh tech is the easy part. We've [7:42] seen this with the PC's we've built. [7:44] We've seen this with the MVPs. Adoption [7:46] is so critical and is really really [7:48] difficult to do at scale. So thinking to [7:50] the people process change management is [7:53] so critical and with the value that AI [7:56] brings which is the ab ability to [7:58] quickly iterate and prototype we should [8:00] be able to use that to our advantage and [8:02] as we're developing these new systems [8:04] think about the inputs the outputs use [8:06] AI to create a new way of doing that [8:08] thing and then as you're iterating on [8:11] that journey bring your users into the [8:13] fold and you can iterate on PC's and [8:15] MVPs based on real-time feedback that [8:18] you're getting from your end users. [8:19] users so that you kind of account for [8:20] the people management and change [8:22] management process as you're going [8:23] through the build itself versus thinking [8:25] about it as an afterthought. [8:28] So those are some of the uh areas that I [8:31] would focus on and and I would think [8:33] about um and then some of the context [8:34] for what we're seeing in the space right [8:36] now. Let's now talk about how to think [8:38] about use case prioritization and [8:41] mapping those use cases as some tangible [8:43] dolls and cents ROI. Like I mentioned, [8:45] we're going to talk through this in a [8:46] bit of a linear way, but just know in [8:48] practice, these are two sides of the [8:50] same coin. So, you're going to be likely [8:52] as you're prioritizing use cases, at the [8:55] same time evaluating what the ROI of [8:56] those potential, the ROI potential of [8:58] those use cases are. [9:01] So, we think about the um the use case [9:03] uh journey here from ideation to [9:06] production. This is sort of what it [9:07] looks like. So, we start off with [9:08] ideation and scoping, which is what [9:10] we're going to primarily focus on now. [9:11] We then go into development, testing and [9:14] eval uh evaluation, production and then [9:16] ongoing maintenance. [9:18] And then within the prioritization [9:20] framework here, there's a couple steps [9:22] that we need to consider. The first step [9:25] and arguably the most important is [9:27] understanding AI. Does the organization [9:30] understand what good uses of the [9:32] technology are? there are some use cases [9:34] that maybe RPA might be a better use for [9:36] or it could be a too too much of a [9:38] deterministic workload where AI is not [9:40] necessarily the right fit to solve that [9:42] problem. So going through trainings [9:44] thinking through education and [9:45] enablement within the organization so [9:47] that they understand when to use [9:49] traditional ML versus when to use uh an [9:51] NLM is absolutely critical in selecting [9:53] the right use cases. Next we prioritize [9:56] what we what we're going to scale and [9:58] that's what we'll talk about and go more [9:59] into more detail about today. The steps [10:01] from there are really understanding the [10:03] current state, thinking through what our [10:05] defined end state needs to be so that we [10:07] have a journey that we're mapping and [10:09] then once you create that flywheel on an [10:12] ongoing basis is sort of when you get to [10:14] this level of operational AI. [10:17] So if you think about use case [10:18] prioritization, I think the use cases [10:21] generally fall into one of these four [10:22] quadrants where you have maybe easy [10:25] wins. So these are use cases that are [10:28] high impact and fairly low effort to get [10:30] done and get into production. These are [10:32] the P zeros. We should absolutely be [10:34] prioritizing these. Um and especially as [10:36] you're building your AI muscle, this is [10:38] a really good focus area to um to start [10:40] your journey. The next one we have are [10:43] these big bets. So you can think of [10:45] these as use cases that are tied to core [10:48] strategic priorities. It's going to be a [10:50] longer time to get into production, but [10:52] they're still critically important. And [10:53] so these should be prioritized, but we [10:55] needed to go through a bit more formal [10:57] process to get them um uh identified and [11:00] get them actually worked on and and pro [11:01] and uh focused on internally. [11:04] The next bucket here are your fill-ins. [11:06] So you can think of these as not super [11:08] hard to do, but also not super [11:10] impactful. And so these you probably [11:11] want to put in the parking lot as your [11:13] business priorities change over time. [11:16] Maybe come back to them and revisit them [11:17] uh and think about bringing them to, you [11:19] know, into production at that point. And [11:21] then the last one here which is the [11:23] bucket or the quadrant to absolutely [11:25] stay away from are these uh tankless [11:27] tasks. So you can think of these as [11:29] super high uh impact excuse me super [11:31] high effort but also super low value. [11:33] And this is the bucket that is just [11:35] doing AI for AI sake. And so this is the [11:37] one where we likely want to stay away [11:38] from the best we can. [11:41] Now when you think of the checklist that [11:43] you want to go through. I know this is a [11:44] a a text heavy slide. Um there's a [11:47] couple different areas that we want to [11:48] ensure that we're focusing on before we [11:50] actually have sign off to prioritize any [11:52] specific use case. First and foremost, [11:54] business value. What is the ROI of this [11:57] this um use case? How are we measuring [12:00] the impact that it's going to have and [12:01] making sure that we have those questions [12:03] answered? The second, as we talked [12:05] about, is complexity and data. Do we [12:07] have data access to the right data [12:08] systems? Is our data strategy and [12:10] structure in a place that can be easily [12:12] consumed and leveraged by LLMs? and are [12:15] there clearly defined workflows. [12:17] The next one here is familiarity and [12:19] risk. So is this like we talked about [12:21] from an education perspective a use case [12:23] that is good for the LLM to actually do. [12:26] Are they IP or regulatory um concerns [12:28] that we need to consider and think [12:30] about? And then the last one is path to [12:32] implementation. So once we have all of [12:34] these questions answered, um what is [12:36] going to be our roadmap to actually go [12:37] from where we are right now to [12:39] eventually getting into production and [12:41] who are who's going to be the owners? [12:43] who's going to take on this project [12:44] after it gets to production and what is [12:45] going to be the lifeline and the [12:47] longevity of this project um following [12:49] all of this being done. Once we go [12:51] through those steps and I know it feels [12:53] like a lot of stuff but you can you can [12:55] do this in 30 40 minutes especially for [12:57] priority use cases and then iterate on [13:00] it over time but once we go through all [13:02] of these steps we have identified some [13:04] use cases [13:06] this is really where we want to focus [13:07] and prioritize. So, we have a list of [13:09] use cases. We want to plot them on this [13:11] um on this grid where the x-axis here is [13:14] how long it's going to take and the [13:16] y-axis is the amount of impact that it's [13:18] going to have. And the area we really [13:20] want to focus in on is that AGI area. [13:22] See what I did there? That was all me, [13:24] by the way. No AI # jobsecurity. Um, but [13:28] that's really the area that we want to [13:29] focus in on is that AGI quadrant because [13:32] those are going to be the highest value [13:34] use cases for us to prioritize, but also [13:36] the ones that are going to take the [13:37] lowest amount e effort and I think [13:39] especially based on the AI maturity of [13:41] the organization if this is a new muscle [13:43] that you're building that is the sweet [13:45] spot to really start this journey. [13:48] Cool. Now that we have this list of use [13:51] cases that we prioritized, let's now [13:52] talk about how we can actually attach [13:54] some value to that use cases and boil it [13:56] down to some dollars and cents [13:58] perspective that allows us to add this [14:00] additional dimension in defining what [14:03] use cases we're prioritizing. [14:06] The first thing we want to understand is [14:07] which bucket does the ROI sort of value [14:10] fall into? And the three buckets here [14:11] are reimagination. So this is these huge [14:14] strategic bets within an organization [14:16] that we want to maybe prioritize. The [14:18] second one are the uh like you know [14:21] shorter term um lesser impact that's [14:25] still impactful from an RAI perspective [14:27] where we're growing the business. And [14:28] the third one is operating operations. [14:30] So these are really quick wins that we [14:31] can get fairly simple to get into [14:33] production. Once we define where the use [14:36] case falls, we can then start to map [14:38] some ROI and work backwards into that. [14:40] And again, a key indicator here in [14:42] helping you to understand where you [14:43] should start or continue the journey is [14:45] very much your AI maturity. Have you [14:47] taken many use cases to production? [14:49] Maybe we can start to focus on [14:50] reimagining the business. If this is the [14:52] first or second one or if there's many [14:53] failed use cases that you're coming back [14:55] from, maybe we start with automating [14:56] operations first and then work up as we [14:58] build the muscle. [15:00] So once we identify this, the next step [15:02] here is um generating what our headline [15:04] KPI is. And this is really the metric we [15:07] want to use to measure this specific use [15:09] case. So, for example, if I look at [15:12] automating operations, maybe I want [15:14] faster resolutions for my customer [15:15] service tickets. The metric I'm going to [15:17] be measuring and the headline KPI that [15:19] I'm going to be focused on is the [15:20] average ticket ticket handling time. And [15:23] there's different ways that you can come [15:24] up with these KPIs. It's going to be [15:26] very much bespoke and based on your [15:28] organization. But one way that you might [15:30] be able to do this is we have this [15:32] fictitious company Sam Sofas. It looks [15:34] like Sam Alman's side gig maybe. Um but [15:38] in in this example here, there's a few [15:40] different places we can pull these ROI [15:41] numbers from. We can start with the [15:44] sofa, you know, the strategic um [15:46] initiatives that we want to focus in on, [15:48] create a KPI from there, or we can even [15:50] start from your profit and loss and that [15:51] gives us some metrics that we can work [15:52] backwards into. Once we have some [15:55] numbers and some general areas for how [15:57] we're going to approach this, we can [15:59] then look at what the pipeline is here [16:01] and like really start to size these [16:03] ideas really quickly. Even at minimum, [16:05] if you're getting to t-shirt sizes, [16:07] that's good enough for helping us to [16:08] prioritize which use cases we should be [16:10] focused on. So, we identify the um [16:12] headline KPI. Um for example, if we look [16:15] at support agent co-pilot, which is the [16:17] average ticket handling time, we come up [16:19] with some baseline based on our internal [16:21] data. Um right now, it's taking three [16:23] minutes for a ticket. We have some [16:24] assumption for what value AI is going to [16:26] add. And then we can do a calculation [16:28] based on the profit and loss numbers [16:29] that we saw on the previous slide to [16:31] determine what the financial impact is [16:33] going to be. And you'll notice as you go [16:35] higher here or lower down in the stack [16:37] and look at more of these [16:38] transformational ideas, the ROI is [16:40] significantly higher, but you don't want [16:41] to fall for the trap of just focusing [16:43] there because you really want to think [16:44] about what the maturity is of the [16:46] organization and uh start at the place [16:48] that makes the most sense. [16:50] So really quick summary when you think [16:52] about the value checking uh value [16:56] tracking checklist here, you want to [16:57] make sure that you're prioritizing these [16:59] loweffort high impact use cases. We want [17:01] to treat ROI estimations as iterative. [17:04] Get to a starting place. Leverage AI to [17:06] actually help create some of these ROI [17:08] baselines to start with and then iterate [17:11] as often as you can. And then share [17:13] quantifiable results as you have wins [17:15] internally. Share this with leadership. [17:17] Share this with other teams because that [17:18] builds momentum and it starts to [17:20] snowball as you're trying to get more [17:21] use cases into production. [17:24] So if there's only three things you take [17:26] away from our time here together, I'll [17:27] be a little disappointed. But this would [17:29] be those three. uh good eval equals good [17:31] deployments. Start there and spend the [17:33] time and getting that right. Second, [17:35] high impact, loweffort use cases. And [17:38] then third, start simple and refine your [17:41] ROI analysis over time. [17:44] Cool. So, I'll leave you guys with this. [17:46] ROI matters. Business impact matters. [17:49] But we're also in one of the most [17:51] exciting technology moments in history, [17:53] and we have access to some of the most [17:54] powerful tools that the world has ever [17:56] known. They make us at times feel like [17:59] superhumans. So I encourage you to [18:02] harness them, apply them, go build [18:04] something bold, use them to uplift your [18:06] community, delight your customers, or [18:08] finally launch the idea you've been [18:09] dreaming about. This is the moment to [18:11] really go be superhumans. Thanks so [18:13] much.