EPAM Says Measuring Standalone AI Return Distorts Enterprise Budgets

What's the ROI on Your AI? How to Get the Right Answer to This Wrong Question With EPAM Systems — Ai4

Ai4

Isolating return on artificial intelligence fails because enterprise gains stem from operational redesign and shared data infrastructure rather than standalone algorithms.

AI-assisted summary of What's the ROI on Your AI? How to Get the Right Answer to This Wrong Question With EPAM Systems — original source. Check the original for full context.

Source published: 2026-08-19

Detailed notes

## Background Published by Ai4 on YouTube on August 19, 2026, this presentation features EPAM Systems leaders John Capriotti (VP, AI-First Business Design) and Ira Livshits (Principal Data Scientist). According to the description, the session explores why the common executive question "What's our ROI on AI?" consistently fails to yield actionable answers and provides a defensible methodology for evaluating true business value.

## TL;DR Capriotti and Livshits argue that calculating a blanket ROI on AI is as misguided as calculating the ROI of HTML, as AI functions as a foundational enabler rather than an isolated financial asset. Deconstructing three real-world implementations—an AI legal assistant, a conversational career advisor, and an enterprise-wide business transformation—they show how standard ROI calculations distort reality when failing to model capacity reinvestment horizons, knowledge graph infrastructure amortization, and the heavy role of non-AI operational hygiene. To avoid killing high-impact initiatives prematurely or greenlighting deceptively cheap add-ons, enterprises must establish cost-attribution and portfolio-allocation models before writing code, anchoring measurement in overarching business outcomes rather than isolated algorithm performance.

## Atmosphere & style Taking the stage in full Star Trek attire as Kirk and Spock, the duo establishes a warm, dynamic rapport that pairs intuitive executive storytelling with rigorous analytical realism. The presentation feels energetic, irreverent, and pragmatic, punctuated by humorous client-call anecdotes and sharp critiques of corporate AI hype. The tone smoothly navigates between approachable banter and deep architectural nuance, leaving the audience with an unvarnished, clear-eyed view of enterprise transformation.

## Key takeaways - [02:18](https://www.youtube.com/watch?v=dqQiZW82sB0&t=138s) Enterprise AI has matured from subsidized proof-of-concept tests and adoption pilots into scaled production, driving executive mandates for strict financial accountability. - [02:18](https://www.youtube.com/watch?v=dqQiZW82sB0&t=138s) Asking for the blanket 'ROI of AI' is a fundamentally flawed question akin to asking for the ROI of HTML, as AI is an enabling foundational technology rather than a standalone asset. - [04:21](https://www.youtube.com/watch?v=dqQiZW82sB0&t=261s) While the classic ROI formula is mathematically simple, the primary difficulty lies in accurately attributing and time-phasing value and costs across complex architectures. - [05:41](https://www.youtube.com/watch?v=dqQiZW82sB0&t=341s) Calculating direct labor-hour savings from AI point tools is incomplete without modeling how that liberated capacity is strategically reinvested across varying business growth horizons. - [09:31](https://www.youtube.com/watch?v=dqQiZW82sB0&t=571s) Conversational AI interfaces depend heavily on non-AI foundational architecture, such as structured taxonomies and enterprise knowledge graphs, to deliver meaningful value. - [11:27](https://www.youtube.com/watch?v=dqQiZW82sB0&t=687s) Attributing 100% of foundational data and graph infrastructure costs to an initial pilot makes pioneer projects appear financially unviable on paper. - [11:27](https://www.youtube.com/watch?v=dqQiZW82sB0&t=687s) Enterprises need centralized portfolio strategies to amortize shared foundational infrastructure costs across multiple downstream AI initiatives. - [14:43](https://www.youtube.com/watch?v=dqQiZW82sB0&t=883s) Executive perceptions of current-state workflows are routinely off by at least 15% when audited against how employees actually execute daily tasks. - [14:43](https://www.youtube.com/watch?v=dqQiZW82sB0&t=883s) In large-scale transformation, the highest value typically comes from basic operational hygiene, workflow re-engineering, and data cleanup rather than complex AI algorithms. - [14:43](https://www.youtube.com/watch?v=dqQiZW82sB0&t=883s) Funding holistic business design out of dedicated AI budgets leads to misleading ROI figures by crediting standard process improvements solely to AI technology. - [18:17](https://www.youtube.com/watch?v=dqQiZW82sB0&t=1097s) As initiatives move from narrow point solutions to comprehensive business design, the share of direct AI shrinks and ROI attribution becomes increasingly subjective. - [18:17](https://www.youtube.com/watch?v=dqQiZW82sB0&t=1097s) Organizations should stop attempting to isolate algorithm performance and instead evaluate AI success through concrete, overarching business outcomes.

## Detailed walkthrough ### [00:04](https://www.youtube.com/watch?v=dqQiZW82sB0&t=4s) Introductions, AI Pairing, and Kirk & Spock

Opening the presentation at the Ai4 conference, John Capriotti and Ira Livshits from EPAM Systems take the stage in full Star Trek attire. To explain their wardrobe choice, Ira shares how they prompted an AI to analyze detailed personal and professional descriptions of their long-standing daily working relationship and suggest matching collaborator archetypes. Ira had described John as an inspiring leader with a background blending cognitive psychology and modern technology, a people-first advocate who elevates his teams, highly passionate—though "sometimes can be easily confused with a hothead"—and exceptionally eloquent, frequently using his verbal quickness to poke fun at her.

John, in turn, described Ira as a rigorous mathematician and analytical thinker capable of applying immense mathematical precision to business problems like AI valuation, alongside her deep data science background. The AI returned several iconic pairings, including Ren & Stimpy, Abbott & Costello, Pinky and the Brain, and ultimately Captain Kirk and Mr. Spock—a balance of intuition and analytical rigor that framed their approach to the day's topic.

[02:18](https://www.youtube.com/watch?v=dqQiZW82sB0&t=138s) The Maturation of AI and the "ROI of HTML" Fallacy

John contextualizes why calculating return on investment has suddenly become the most urgent issue across enterprise boardrooms. In the early stages of generative AI, companies built rapid proofs of concept where financial costs were largely negligible or subsidized by technology partners, and success was measured solely as a binary: did the concept work or not? As initiatives progressed to pilots, the focus shifted to small-scale usability, qualitative user adoption, and functional viability. Today, however, enterprise AI is live in production, fully scaled, and accumulating substantial infrastructure and computational run costs, prompting CFOs and executive leadership to demand strict financial accountability.

```mermaid flowchart LR accTitle: Evolution of Enterprise AI Maturity subgraph S1["1. Proof of Concept"] A["Vendor Subsidized"] --> B["Binary Feasibility"] end subgraph S2["2. Pilot Phase"] C["Small-Scale Test"] --> D["User Adoption"] end subgraph S3["3. Production at Scale"] E["Scaled Computing Costs"] --> F["Executive ROI Mandate"] end S1 --> S2 --> S3 ```

During a client meeting where a CEO bluntly asked, "What is the ROI of AI?", John played a recurring game he shares with Ira: trying to make her laugh so hard on client video calls that she is forced to turn off her camera and microphone. John texted her privately: *"That's a dumb question. It's like asking, 'What's the ROI of HTML?'"* While the question is fundamentally flawed because AI is an enabling foundational technology rather than an isolated financial asset, enterprise leaders will not stop asking it. The real challenge is determining how to provide the right answer to the wrong question using a clear, defensible methodology.

- **Proof of Concept (POC) stage**: Focused on raw technical feasibility; costs were low and often vendor-subsidized. - **Pilot stage**: Evaluated small-scale adoption and workflow usability. - **Scaled Production stage**: High recurring operating costs (tokens, compute, integration) driving board-level ROI pressure. - **The Core Thesis**: Asking for the blanket ROI of AI is as meaningless as asking for the ROI of HTML; it must instead be grounded in specific business use cases and financial attribution models.

[04:21](https://www.youtube.com/watch?v=dqQiZW82sB0&t=261s) Deconstructing the Classic ROI Equation

Rather than designing an overly academic formula burdened with integrals and derivatives, Ira and John emphasize the time-tested business formula:

$$\text{ROI} = \frac{\text{Value} - \text{Cost}}{\text{Cost}}$$

While the formula itself is conceptually straightforward—low cost combined with high value yields a healthy return—the complexity lies in measuring, attributing, and time-phasing both the value generated and the true operational costs across distinct AI architectures. To illustrate this, the speakers introduce three real-world enterprise implementations executed over the preceding 6 to 12 months:

1. **AI Legal Assistant** for an international document and records management firm. 2. **Conversational Professional Development Advisor** for a professional services membership organization. 3. **Enterprise-Wide AI Transformation** operating at a systemic business level.

[05:41](https://www.youtube.com/watch?v=dqQiZW82sB0&t=341s) Case 1: The AI Legal Assistant and Value Reinvestment

For the international document and records management company, onboarding new clients represented a major operational bottleneck. The onboarding workflow required specialized attorneys to interview each client, identify specific vertical compliance and document retention regulations, research applicable case law, and manually draft bespoke contractual terms. Because legal talent is expensive and scarce, onboarding capacity was strictly constrained. To eliminate this bottleneck without drastically expanding headcount, the team developed an AI legal assistant that joins intake discussions, automatically extracts key requirements, identifies the relevant legal parameters, and drafts the necessary contract language for the attorney's final review.

From a pure cost-accounting perspective, this implementation behaves predictably: the development cost of the engineering team combined with ongoing inference and API run costs follow steady, forecastable utilization patterns. On the value side, the assistant immediately cut attorney onboarding time by 50%, freeing up half of their billable capacity.

```mermaid flowchart TD accTitle: Legal Assistant Value Attribution and Reinvestment A["AI Legal Assistant"] --> B["50% Attorney Time Freed"] B --> C{"Strategic Reinvestment"} C --> D["Immediate Onboarding Capacity (Short Maturation / Fast ROI)"] C --> E["Existing Client Retention (Medium Maturation)"] C --> F["New Industry Verticals (Long Maturation Horizon)"] C --> G["Geographic Expansion (Long Maturation Horizon)"] ```

However, John and Ira point out that stopping at direct labor-hour savings produces an inaccurate, incomplete ROI calculation. The company did not simply send its legal team out to play golf; leadership systematically reinvested that freed capacity across four strategic growth objectives:

- **Onboarding new pipeline clients**: Delivers rapid financial return, raising the question of whether to count first-year revenue or lifetime customer value in the ROI equation. - **Existing client retention**: Protects baseline recurring revenue over medium-term horizons. - **Expanding into new industry verticals**: Requires attorneys to research unfamiliar regulatory environments, demanding a longer maturation runway before financial returns materialize. - **Entering new geographic markets**: Involves complex international jurisdictions, creating an extended time lag between capacity reinvestment and realized value.

Calculating true value requires not just measuring raw time saved, but modeling how that capacity is reinvested and selecting the appropriate time intervals required for each specific business objective to mature.

[09:31](https://www.youtube.com/watch?v=dqQiZW82sB0&t=571s) Case 2: Conversational Career Advisor and Foundational Data Pitfalls

The second case examines a professional services membership organization whose business model relies on sustained member engagement and educational value delivery. The organization suffered from a steep drop-off in member engagement after the first few months of onboarding, leaving high-value training coursework, certifications, and professional advancement resources underutilized. To solve this, the organization sought to build a conversational career advisor that could evaluate a member's current career stage and map out personalized learning journeys, relevant courses, and certifications to reach their target roles.

The organization had previously attempted to build this capability internally through a personalization proof of concept. While the initial POC appeared viable on the surface, the pilot failed when deployed to actual users because the underlying personalization was paper-thin. The conversational advisor lacked any meaningful understanding of the members because the organization had neglected the essential foundational architecture: a structured skills taxonomy and an enterprise knowledge graph connecting member profiles, competencies, learning assets, and career pathways.

[11:27](https://www.youtube.com/watch?v=dqQiZW82sB0&t=687s) Distributing Foundational AI Costs: The Knowledge Graph Dilemma

Building an intelligent career advisory tool required creating an entire "universe of skills" relevant to the profession, mapping where members currently stood, and charting the skill gaps between their current roles and career aspirations. Achieving this meant first establishing a comprehensive skills taxonomy and mapping skills content directly to a skill dataset. This heavy lifting culminated in an enterprise knowledge graph—by far the most time-consuming and expensive component of the entire initiative. By contrast, spinning up a conversational chatbot interface in the modern AI ecosystem takes almost no time or money. In practical terms, the bulk of the expenditure went toward building a non-AI foundational system that connected the organizational universe.

```mermaid flowchart TD accTitle: Foundational Cost Redistribution in AI Portfolios

subgraph S1["Foundational Investment"] KG["Enterprise Knowledge Graph (High Cost / Non-AI Core)"] end

subgraph S2["Direct AI Project"] P1["Career Advisor Pilot (Low-cost Chatbot UI)"] V1["Modest Member Value (Looks ROI Negative if fully loaded)"] end

subgraph S3["Future Downstream Projects"] P2["Project B"] P3["Project C"] P4["Project D"] end

KG -->|"100% cost burden (Flawed)"| P1 P1 --> V1 KG -.->|"Amortized cost allocation (Ideal)"| P2 KG -.->|"Amortized cost allocation (Ideal)"| P3 KG -.->|"Amortized cost allocation (Ideal)"| P4 ```

When the accounting department attributes the entirety of that foundational graph development exclusively to the career advisor project, the math looks disastrous. The measurable returns for that single application—member engagement, satisfaction scores, and usage interaction—are modest. While the tool met its explicit business objectives, the loaded costs were massive, making the project look like an utter failure on paper. Yet that same knowledge graph forms an enduring foundation that is reused across every subsequent AI project the enterprise builds. If future projects leverage the graph for free, they will look deceptively cheap and massively profitable, while the pioneer project looks like a loss.

To prevent skewing executive decision-making, an organization requires a mature portfolio strategy that identifies downstream projects upfront and redistributes foundational infrastructure costs across them. In reality, most enterprises operate in a decentralized, experimental fashion without centralized AI product management. Without proper cost-attribution modeling, leadership risks prematurely killing off high-impact foundational initiatives because they appear ROI-negative, while greenlighting superficial, inexpensive add-ons whose true infrastructural dependencies were hidden.

- **Foundational investment:** Graph construction and skills taxonomy mapping accounted for nearly all project spend, while LLM/chatbot interfaces represented minimal incremental cost. - **Value misalignment:** The career advisor satisfied business goals (member engagement/satisfaction), but looked financially unviable when saddled with 100% of the foundational graph's development costs. - **Portfolio risk:** Failing to amortize foundational assets across future AI projects leads companies to terminate genuinely effective tools while investing in falsely "cheap" follow-ons.

---

[14:43](https://www.youtube.com/watch?v=dqQiZW82sB0&t=883s) AI-First Business Design: When AI Gets the Budget but Not the Work

The most controversial and complex scenario arises during broad "AI-first business transformation" or business design programs. Here, asking "What's the ROI on AI?" becomes both too narrow and too vague. In these engagements, funding is typically drawn straight from a dedicated corporate AI budget. However, effective transformation demands an exhaustive audit of current-state data, personnel, processes, and technology. Leadership invariably insists they already possess accurate process maps, yet audits consistently discover those assumptions are "always at least 15% off"—primarily because executive perceptions never match what employees on the ground are actually doing.

Once those workflows are accurately mapped, a paradox emerges: pure AI components end up representing only a tiny fraction of the final architecture and realized value. The most immediate, cost-effective, and impactful breakthroughs almost always stem from basic operational hygiene—cleaning broken data pipelines, streamlining business workflows, redefining job responsibilities, or performing standard systems integration. The team operates under the principle that the work is "AI-first, but not AI-only and not AI-forced." If an integration or a simple schema update solves a bottleneck faster and cheaper than a model, they implement it immediately.

This reality breaks traditional AI ROI formulas. If a transformation program is funded by an AI budget and led by an AI team, companies routinely attribute every dollar of expenditure—and every dollar of generated value—to "AI." In reality, outside of minor components like extracting text from documents, the work is "traditional, honest, let's fix the data, let's fix the processes, let's fix the people's roles." While holistic business design yields the largest total enterprise value, that value is least connected to AI algorithms. Without an attribution model established *before* launching the initiative, the resulting ROI calculations become purely subjective exercises arranged to fit whichever internal political narrative the sponsor wants to advance.

---

[18:17](https://www.youtube.com/watch?v=dqQiZW82sB0&t=1097s) Measuring the Spectrum of AI: From Direct Tools to Business Outcomes

Reflecting across their client portfolio, the speakers illustrate how ROI measurement evolves across three distinct archetypes: a targeted legal assistant, an organizational career advisor, and a comprehensive AI-first business design engagement. The further an initiative moves from a narrow, point-solution tool into broad organizational redesign, the more divorced the work becomes from pure AI mechanics—and the more subjective the ROI metric gets.

To navigate this spectrum successfully, enterprises must invest upfront in defining clear measurement criteria and cost-allocation mechanisms before writing code or redesigning workflows. The core lesson drawn from dozens of enterprise deployments is that asking "What is the ROI on our AI?" is fundamentally the wrong starting point. Instead, leadership must anchor their evaluations in overarching business outcomes—solving the attribution debate by measuring organizational impact rather than artificially isolating algorithm performance.

## Notable quotes > [00:04](https://www.youtube.com/watch?v=dqQiZW82sB0&t=4s) sometimes can be easily confused with a hothead — Ira Livshits

> [02:18](https://www.youtube.com/watch?v=dqQiZW82sB0&t=138s) What is the ROI of AI?

> [02:18](https://www.youtube.com/watch?v=dqQiZW82sB0&t=138s) That's a dumb question. It's like asking, 'What's the ROI of HTML?' — John Capriotti

> [11:27](https://www.youtube.com/watch?v=dqQiZW82sB0&t=687s) universe of skills

> [14:43](https://www.youtube.com/watch?v=dqQiZW82sB0&t=883s) always at least 15% off — John Capriotti

> [14:43](https://www.youtube.com/watch?v=dqQiZW82sB0&t=883s) AI-first, but not AI-only and not AI-forced. — John Capriotti

> [14:43](https://www.youtube.com/watch?v=dqQiZW82sB0&t=883s) traditional, honest, let's fix the data, let's fix the processes, let's fix the people's roles. — John Capriotti

> [18:17](https://www.youtube.com/watch?v=dqQiZW82sB0&t=1097s) What is the ROI on our AI?

> What's our ROI on AI?

> The answer should drive confident decisions—but consistently fails. Not because AI doesn't work or you can't measure it, but because the question itself is fundamentally broken.

## Mentioned John Capriotti, Ira Livshits, EPAM Systems, Ai4, Star Trek, Captain Kirk, Mr. Spock, Ren & Stimpy, Abbott & Costello, Pinky and the Brain, HTML

## Video description John Capriotti - VP, AI-First Business Design, EPAM Systems Ira Livshits - Principal Data Scientist, EPAM Systems

You've asked it. Your board has asked it. Your CFO wants to know: "What's our ROI on AI?"

The answer should drive confident decisions—but consistently fails. Not because AI doesn't work or you can't measure it, but because the question itself is fundamentally broken.

In 20 minutes, we'll show you why and give you something better: a question you can actually answer and a method you can defend.

[Source](https://www.youtube.com/watch?v=dqQiZW82sB0)

中文简报

EPAM称单独衡量AI回报会扭曲企业预算孤立计算人工智能回报之所以行不通，是因为企业收益源于运营流程重组与共享数据基础设施，而非独立的算法本身。 Capriotti与Livshits指出，笼统计算AI的投资回报率（ROI）就像计算HTML的ROI一样是一种误导，因为AI扮演的是底层赋能者的角色，而非孤立的金融资产。通过解构三个实际落地案例——AI法律助理、对话式职业顾问和企业级业务转型——他们展示了当标准ROI计算未能对产能再投资周期、知识图谱基础设施摊销以及非AI基础运营规范的关键作用进行建模时，会如何脱离实际。为了避免过早扼杀高影响力的项目或批准看似廉价的附加工具，企业必须在编写代码之前建立成本归因与投资组合配置模型，将衡量标准锚定在整体业务成果上，而非孤立的算法表现。随着企业 AI 从低成本的概念验证走向规模化生产，高管层开始严抓财务问责；但直接询问“AI 的 ROI”如同询问“HTML 的 ROI”一样是个伪命题，因为 AI 是底层赋能技术而非孤立的财务资产。 (2:18) 单纯计算 AI 点工具节省的工时无法体现完整价值，企业必须建立模型，评估释放出的产能如何策略性地再投资于不同周期的业务增长，如快速客户入职、客户留存或长期跨地域扩张。 (5:41) 对话式 AI 往往高度依赖企业知识图谱等非 AI 基础设施，若将 100% 的基建成本归因于首个试点，会使先锋项目在账面上显得严重亏损，因此企业需要制定组合策略来分摊共享架构成本。 (11:27) 企业审计显示管理层对实际工作流的认知通常存在至少 15% 的偏差；在全面的 AI-First 业务重塑中，绝大部分回报往往来自流程理顺和数据清洗等基础运营优化，而非 AI 算法本身。 (14:43)

Reading summary

John Capriotti and Ira Livshits of EPAM Systems argue that asking for an aggregate "ROI of AI" is fundamentally flawed because AI is an enabling technology whose return depends on how capacity is reinvested, how foundational costs are distributed, and how non-AI operational fixes are accounted for. The Flawed Premise of Aggregate AI ROI (2:18) As enterprise AI transitioned from subsidized proofs of concept and usability pilots into production at scale, high computational and infrastructure costs led executive leadership to demand strict financial accountability. Capriotti likens asking for the blanket "ROI of AI" to asking for the "ROI of HTML": AI is an underlying capability rather than an isolated asset. Although the basic business formula of value minus cost divided by cost remains the standard, establishing accurate returns requires modeling specific use cases rather than treating AI as a monolith. Time Horizons in Value Reinvestment (5:41) Direct labor savings alone do not capture full financial return. In a deployment for an international document and records management firm, an AI assistant halved attorney onboarding time by drafting contracts and identifying compliance rules. However, calculating true ROI depended on how that reclaimed 50% capacity was reinvested. The company redeployed attorneys into four distinct initiatives with varying payback periods: immediate pipeline client onboarding, medium-term client retention, and longer-term entries into new industry verticals and international jurisdictions. Accurate attribution requires selecting time intervals that match these distinct maturation horizons. Amortizing Foundational Infrastructure Across Projects (11:27) When a professional services membership organization developed a conversational career advisor, the primary expenditure went into establishing a skills taxonomy and building an enterprise knowledge graph, whereas the conversational interface required minimal time and money. If the entire cost of that foundational, non-AI infrastructure is charged solely to the initial advisor pilot, the application appears financially underwater despite meeting engagement goals. Capriotti and Livshits stress that without a centralized portfolio strategy that amortizes shared foundational assets across downstream projects, companies risk canceling viable tools while falsely viewing follow-on applications as highly profitable. Operational Hygiene in AI Transformations (14:43) Broad "AI-first business design" initiatives frequently receive dedicated corporate AI funding, yet audits of personnel, technology, and workflows regularly reveal that executive assumptions about processes are off by at least 15%. Resolving these operational issues typically reveals that pure AI models represent only a tiny fraction of the final implementation. Most value comes from standard operational hygiene, such as fixing broken data pipelines, updating schemas, and clarifying employee roles. If all expenses and gains are credited solely to an AI budget, ROI calculations become subjective narratives disconnected from actual technical drivers. Capriotti and Livshits conclude that as implementations move from isolated point solutions to broad business redesign, the direct technical contribution of AI decreases while the subjectivity of the ROI metric grows. To prevent skewed decision-making, organizations must establish cost-allocation frameworks and define target business outcomes before deploying code.

精炼阅读摘要

EPAM Systems 的 John Capriotti 与 Ira Livshits 认为，笼统询问“AI 的投资回报率（ROI）”在本质上就像询问“HTML 的 ROI”一样是个伪命题；企业不应孤立评估底层算法模型，而必须依据具体的业务场景、成本分摊机制与价值再投资路径来建立财务归因模型。从概念验证到规模化生产的财务压力 (2:18) 在生成式 AI 发展初期，企业通常停留在由技术合作伙伴补贴、财务成本极低的概念验证阶段，成功的衡量标准仅是技术上“可行与否”。随后的小规模试点则主要关注用户采纳度与工作流可用性。然而，随着 AI 系统全面接入生产环境，不断累积的高昂算力、推理 Token 及系统集成等日常运行成本，促使首席财务官与董事会开始施加严苛的财务问责压力。 Capriotti 指出，直接向管理层询问“AI 的 ROI 是多少”脱离了实际，因为 AI 是一项基础赋能技术而非孤立的资产。虽然基础商业公式“ROI = (价值 - 成本) / 成本”看似简单，但随着 AI 架构日益复杂，如何对产生的实际价值进行时间跨度建模，以及如何在系统层面分摊运营成本，已成为解答这一财务问题的核心难点。工时节省的局限与业务能力再投资周期 (5:41) 在一家国际文档与档案管理公司的案例中，由于合规法规繁杂且需定制合同条款，稀缺且昂贵的专业律师团队构成了新客户入驻的运营瓶颈。该团队部署的 AI 法律助手可自动提取需求并起草合同供律师终审，直接将律师的入驻耗时缩短了 50%。该项目的工程开发与推理 API 成本均高度可控且可预测。然而，讲者强调如果仅将工时节省计为收益，会导致 ROI 计算失真。企业并未因效率提升而削减人力，而是将释放出的 50% 产能重新投资到四项增长目标中：消化待入驻新客户管线（短期兑现收入）、维护存量客户以巩固续费（中期见效）、拓展全新行业垂类以及开拓跨国业务（因需调研新法域而具备较长的回报滞后期）。因此，精准核算 AI 价值不仅要统计直接节省的时间，更要结合各项业务再投资所需的成熟周期来评估实际回报。基础数据资产导致的单项目成本错配 (9:31) 某专业服务会员机构为了解决新会员入驻数月后参与度锐减的问题，试图打造一款能够根据职业阶段规划学习路径的对话式职业发展顾问。此前该机构内部的类似试点以失败告终，根本原因在于缺乏底层的技能分类法与企业知识图谱，导致个性化推荐停留在表面。为构建这一系统，团队耗费了绝大部分资金与时间去搭建庞大的非 AI 基础系统——企业知识图谱，而前端对话界面的开发成本却微不足道。财务部门若将知识图谱的全部建设成本全额分摊到该职业顾问项目上，即便其达成了提升会员满意度与活跃度的目标，账面 ROI 也会呈现严重亏损。该图谱实际上属于可在未来下游项目中复用的持久基础资产；若不采用跨产品组合的分摊策略，后续复用图谱的项目看起来极其廉价且高回报，而先行项目则被判定为失败。企业若缺乏集中的 AI 产品管理与组合分摊机制，极易因账面亏损而过早扼杀真正具有战略价值的基础设施项目。全面业务转型中的预算与价值归因悖论 (14:43) 在规模更大的“AI 优先业务转型”中，项目的资金通常直接划拨自企业的 AI 专项预算。然而项目前期的全面审计显示，高管层对业务流程现状的认知与一线员工的实际操作“始终存在至少 15% 的偏差”。在此类复杂场景下，团队坚持“AI 优先，但非唯 AI 且不强推 AI”的原则，一旦发现传统集成或模式调整能以更低成本解决瓶颈，便会优先采用。最终结果表明，真正的 AI 算法与纯模型组件在整体架构和创造的价值中仅占极小比例，绝大多数突破来自于清理数据流水线、重塑流程与明确岗位职责等传统数字化工作。若企业依然机械地把 AI 专项预算下的所有投入与收益统统归功于“AI 的 ROI”，整个指标就会沦为主观的内部政治工具。业务设计转型虽然能带来最高的企业总价值，但其回报与 AI 算法本身的关系最为疏离。从孤立的直接工具（如法律助手）、平台型应用（如职业顾问）到系统性的业务重塑，项目范围越宽广，纯 AI 技术在其中的比重就越低，其 ROI 的衡量难度与主观性也成倍增加。企业必须在启动技术开发与流程改造前，提前确立财务分摊机制与衡量标准，将评估重心从孤立的算法表现转向组织维度的业务成果。

Detailed notes — zh

## Background 本视频由 Ai4 频道于 2026 年 8 月 19 日发布在 YouTube 平台。内容围绕 EPAM Systems 的 AI 原生业务设计副总裁 John Capriotti 与首席数据科学家 Ira Livshits 的联合演讲展开，旨在解答企业高管和财务层关于“AI 的 ROI 到底是多少”的普遍困惑，指出该提问本身的根本缺陷并给出可落地的应对方法。 ## TL;DR 演讲通过三个由浅入深的实际交付案例（AI 法律助手、职业发展顾问、AI 原生业务重塑），系统拆解了企业计算 AI 投资回报率（ROI）时的归因难点与常见陷阱。主讲人指出，单点工具的价值取决于工时节省后的业务再投资方向，底层知识图谱等高昂基建成本需在企业级产品组合中跨项目分摊，而在全盘业务重塑中，大部分回报往往来自伴随 AI 调研而来的传统流程与数据优化。最终结论强调，企业必须停止纠结于孤立计算“纯 AI 的 ROI”，转而将衡量基准牢牢锚定在最终达成的业务成果（Business Outcomes）上。 ## Atmosphere & style 两位主讲人以《星际迷航》中“柯克船长与斯波克”的经典搭档形象亮相，一人热情幽默、辞锋犀利，另一人严谨沉着、深耕数理逻辑，彼此互动默契且充满幽默感。整场演讲氛围兼具极客趣味与商业严肃性，既有对行业内“孤立询问技术 ROI”等盲目行为的辛辣讽刺，又以极其务实、直面痛点的企业级交付经验为听众提供了清晰解题思路。 ## Key takeaways - [00:04](https://www.youtube.com/watch?v=dqQiZW82sB0&t=4s) 企业 AI 应用已从早期由技术伙伴出资的概念验证（PoC）和小规模试点（Pilot），迈入全面投产与规模化阶段，算力与运营成本的累积引发了高管层对 ROI 的紧迫问责。 - [03:13](https://www.youtube.com/watch?v=dqQiZW82sB0&t=193s) 直接询问“AI 的 ROI 是多少”在逻辑上是错误的提问，其荒谬程度类似于追问“HTML 的投资回报率是多少”，但企业依然需要找到合理解释业务成效的方法。 - [03:13](https://www.youtube.com/watch?v=dqQiZW82sB0&t=193s) AI 的 ROI 评估最终回归于古典的“价值 / 成本”公式，但在真实企业落地时，价值的归因与成本的核算极具复杂性与多维性。 - [05:41](https://www.youtube.com/watch?v=dqQiZW82sB0&t=341s) 在工具型应用（如 AI 法律助理）中，表面价值是节省了 50% 的工时薪酬，但更深层的价值在于企业将释放的时间再投资于获客、客户留存与新市场拓展。 - [05:41](https://www.youtube.com/watch?v=dqQiZW82sB0&t=341s) 衡量 AI 的价值必须结合时间维度，因为不同业务目标（如即时签署新客户 vs. 跨法域开拓海外市场）对应的价值成熟周期长短差异巨大。 - [11:27](https://www.youtube.com/watch?v=dqQiZW82sB0&t=687s) 在复杂场景（如职业规划顾问）中，聊天机器人界面成本极低，而耗资最巨的底层核心是构建通用的专业技能分类体系与知识图谱底座。 - [11:27](https://www.youtube.com/watch?v=dqQiZW82sB0&t=687s) 若将高昂的基础设施（知识图谱）成本全部摊在首个 AI 试点上，会导致账面 ROI 极差并被误判为失败，而后续复用该能力的项目则会显得轻松坐享其成。 - [11:27](https://www.youtube.com/watch?v=dqQiZW82sB0&t=687s) 企业往往因缺乏集中的 AI 产品组合战略而采取孤岛式推进，容易导致极具战略复用价值的底座项目因账面 ROI 不好看而被错误砍掉。 - [14:43](https://www.youtube.com/watch?v=dqQiZW82sB0&t=883s) 在 AI 原生业务重塑项目中，管理层对内部实际流程的认知偏差通常不低于 15%，一线真实操作与高层预想脱节是导致 AI 方案跑偏的主因。 - [14:43](https://www.youtube.com/watch?v=dqQiZW82sB0&t=883s) “AI 优先”不等于“唯 AI 论”或“强行 AI”；许多业务痛点最经济、最见效的解法依然是传统系统集成、流程优化或岗位职责理顺。 - [14:43](https://www.youtube.com/watch?v=dqQiZW82sB0&t=883s) 当项目由 AI 预算立项但大部分产出由传统数字化重组贡献时，企业必须提前设计好跨模块归因模型，避免主观粉饰或过度争夺技术功劳。 - [18:17](https://www.youtube.com/watch?v=dqQiZW82sB0&t=1097s) AI 项目距离单纯的工具/算法越远、与企业复杂业务重构结合越深，其 ROI 的衡量难度与主观性就越高。 - [18:17](https://www.youtube.com/watch?v=dqQiZW82sB0&t=1097s) 破除 AI 投资焦虑的核心解法是跳出对单一“纯 AI 技术 ROI”的狭隘纠结，转向以最终达成的业务成果（Business Outcomes）为核心来综合评估成效。 ## Detailed walkthrough ### [00:04](https://www.youtube.com/watch?v=dqQiZW82sB0&t=4s) 开场与搭档默契：为什么台上站着“柯克船长与斯波克”？来自 EPAM Systems 的 AI 原生业务设计副总裁 John Capriotti 与首席数据科学家 Ira Livshits 登台分享。他们直奔主题：当下无数企业高管和董事会反复抛出的同一个问题——“AI 的投资回报率（ROI）到底是多少？”为了给严肃的技术与商业探讨增添趣味，两位长期并肩作战的搭档特意穿上了《星际迷航》（*Star Trek*）的戏服。在筹备演讲时，他们把彼此的性格特征输入给 AI，让大模型匹配出最契合的协作角色组合。 Ira 对 John 的描述是一位热衷于将认知心理学与现代科技完美融合的优秀领导者，待人极具亲和力，善于赋能和激励团队，性格充满热情——不过偶尔也会被人误以为“脾气火爆”；同时他言辞极为犀利雄辩，且总爱用幽默来打趣 Ira。而 John 则将 Ira 描述为一位具备极强分析思维的数学家与数据科学家，既冷静沉着又富有激情，能够用极度严谨的态度去拆解诸如“AI 的 ROI”这样复杂的商业命题。在 AI 给出的众多备选组合中（例如《莱恩和史丁比》、雅培与科斯特洛、《粉红猪小妹和大头脑》），两人最终选定了经典的“柯克船长（Kirk）与斯波克（Spock）”。随后，John 梳理了企业 AI 应用演进的三个阶段，指出了当下 ROI 焦虑爆发的根本原因： ```mermaid flowchart LR accTitle: AI 企业应用发展阶段演进 subgraph S1["早期阶段"] A["概念验证 (PoC) 成本由科技伙伴承担 仅验证可行性"] end subgraph S2["试点阶段"] B["小规模 Pilot 关注可用性与采纳度 验证局部效果"] end subgraph S3["现今阶段"] C["全面投产与规模化 成本持续累积 财务问责与 ROI 压力爆发"] end A --> B --> C ``` - **概念验证阶段（PoC）**：成本并不是首要问题，很多前期工作由技术合作伙伴出资支持，核心关注点仅仅是概念在技术上能否走通。 - **试点阶段（Pilots）**：重点转移到小规模场景下的可用性与员工/客户采纳度，检验系统在局部范围内是否有效。 - **全面投产阶段（Production & Scale）**：AI 已经正式上线并进入规模化运作，算力与运营成本不断累积，高管层对财务回报的紧迫追问随之而来。 --- ### [03:13](https://www.youtube.com/watch?v=dqQiZW82sB0&t=193s) 荒谬的提问与古典公式：“HTML 的 ROI 是多少？” John 分享了一次真实的客户会议经历。当时一位客户 CEO 直截了当地质问团队：“我们做 AI 的 ROI 到底是多少？”在开会时，John 经常喜欢和 Ira 玩一个小游戏——看能不能在视频会议中把 Ira 逗笑到不得不关掉麦克风和摄像头的地步。听到那个问题后，John 立即给 Ira 发了条私信：“这问题太蠢了，简直就像在问‘HTML 的 ROI 是多少？’”果不其然，Ira 当场忍不住笑意，迅速关闭了摄像头和麦克风。尽管将整项通用技术孤立地询问 ROI 在逻辑上并不合理，但身处商业环境中的技术与业务团队却无法回避高管的追问。因此，本次演讲的核心就是如何“给这个错误的问题找到正确的答案”。作为数学家的 Ira 本打算列出一套充斥着微积分与导数的复杂数学模型来展示专业度，但被 John 坚决叫停，要求将逻辑简化至最清晰、最直观的本质。最终提炼出的 ROI 衡量标准回归到了经过历史检验的经典双参数框架： $$\text{ROI} = \frac{\text{价值 (Value)}}{\text{成本 (Cost)}}$$ 当成本足够小、价值足够大时，业务就处于理想状态。然而，这套看似简单的公式在实际落地归因时极具挑战。他们精选了过去 6 到 12 个月内完成的三个截然不同的实战项目，逐一拆解价值与成本在不同场景下的真实计算方式。 --- ### [05:41](https://www.youtube.com/watch?v=dqQiZW82sB0&t=341s) 案例一：国际文档管理公司的 AI 法律助理第一个项目是为一家负责文档与档案管理（涵盖合规保留期设定）的国际企业构建 AI 法律助理。该企业在吸纳新客户的入职流程（Onboarding）中存在严重的交付瓶颈：每当新客户接入时，必须由专业律师与客户团队深入沟通，理解其特定垂直行业的文档合规要求；随后律师需要检索相关判例法和法律参数，撰写文档保留与安全条款，并最终生成定制化合同。由于资深律师薪酬高昂且精力有限，法律审核环节直接限制了公司的扩客速度。针对该痛点，团队构建了一套 AI 法律助理系统。该系统能够自动旁听客户对接会议、提取关键合规信息、精准匹配对应法条与参数，并自动生成合同条款供律师审阅，大幅缩短了人工草拟和检索的时间。 ```mermaid flowchart TD accTitle: AI 法律助理价值与再投资链条 subgraph Input["AI 法律助理介入"] A["自动旁听客户沟通"] --> B["提取关键合规信息"] B --> C["匹配法条并生成合同文本"] end subgraph DirectValue["直接价值"] C --> D["律师入职耗时缩减 50% 直接节约工时成本"] end subgraph Reinvestment["时间再投资方向 (业务价值)"] D --> E1["加速拓展新客户"] D --> E2["维护存量客户以提高留存"] D --> E3["开拓全新行业垂直领域"] D --> E4["进军全新海外地理市场"] end ``` 在对该案例进行 ROI 拆解时，Ira 指出这是公式能够直接适用的典型场景，但价值计算绝不能仅仅停留在表面： - **成本端（Cost）**：构成非常清晰且可预测，主要包含构建 AI 法律助理的工程团队成本，以及系统日常运行所产生的 AI 推理算力费用，使用模式稳定可控。 - **表面价值（Direct Value）**：AI 帮助律师将新客户接入耗时直接砍掉一半（释放了 50% 的时间），折算出的工时薪酬节省是立即可见的硬性成本节约。 - **深度价值归因（Value Maturation & Reinvestment）**：如果计算只停留在节省的工时成本，那就大错特错了。企业释放律师产能并不是为了让他们去打高尔夫，而是将时间再投资到四个核心业务目标中： 1. **接入更多新客户**：价值立竿见影，但计算时需明确是仅计入首年签约金额，还是按客户全生命周期留存价值折算。 2. **提高存量客户留存率**：增强客户关系维护以减少流失。 3. **进军全新垂直行业**：律师需要花费更长周期去摸索新行业的合规体系，价值释放周期较长。 4. **拓展新的海外区域市场**：涉及跨法域合规，价值成熟需要更长的时间窗口。因此，衡量 AI 的真正价值必须综合考量“节约了什么”、“节省下来的资源被重新投向何处”以及“不同业务目标对应的价值成熟时间周期”。 --- ### [09:36](https://www.youtube.com/watch?v=dqQiZW82sB0&t=576s) 案例二：专业服务机构的职业发展对话顾问第二个案例是一家以会员制为核心的专业服务机构。该机构的商业成功完全取决于会员的活跃度以及机构能为会员提供的专业价值。当时客户面临的严峻商业挑战是：会员在入会最初几个月后活跃度急剧下滑（Engagement Drop-off），导致机构精心研发的优质专业进阶课程与认证项目无人问津，会员权益无法转化为实际价值。为了解决这一问题，机构希望推出一款对话式职业发展顾问，根据会员当前职业现状与理想目标，动态规划成长路径并推荐匹配的课程与认证。此前，客户曾尝试自行构建该系统并做过一次 PoC，由于当时概念验证中的“个性化”程度极浅，PoC 顺利过关；但当项目推向真实试点（Pilot）时却遭遇惨败。失败的核心根源在于系统底层严重缺乏基础设施支撑——既没有建立起标准化的技能分类体系（Skills Taxonomy），也没有构建能够将技能、课程、认证与职业路径串联起来的知识图谱（Knowledge Graph）。AI 职业顾问对会员的真实能力与需求一无所知，根本无法给出有价值的专业建议。这种基建层面的缺失，使得该项目的成本投入与价值核算逻辑与前一个案例截然不同。 ### [11:27](https://www.youtube.com/watch?v=dqQiZW82sB0&t=687s) 技能图谱与基础能力的成本分摊困境在职业导师类 AI 工具的构建中，核心难点在于建立一套与专业领域紧密关联且切实可用的技能全集（Universe of Skills），以便精准识别每位成员所处的技能坐标，并计算出他们“当前位置与目标位置之间的技能差距”。为了实现这一目标，团队必须建立详尽的技能分类体系（Taxonomy），并将海量的技能内容与底层数据映射到图谱中。这项知识图谱（Knowledge Graph）的搭建构成了整个项目中最耗时、最昂贵、投入最大的部分。相比之下，搭建上层的聊天机器人界面在如今几乎不需要花费多少时间和金钱。这就导致在成本结构上，团队实质上投入巨资构建了一个连接整个组织运作的非 AI 基础底座，而所有这些前期成本却全部被武断地归摊到了这单个 AI 试点项目上。如果仅从该项目的直接业务产出来看，其收益主要体现为成员参与度、满意度以及对新工具的日常使用。尽管项目圆满达成了既定的业务目标，但与极其高昂的建设成本相比，单看该项目的账面投资回报率并不理想。这是否意味着项目失败了？答案完全取决于评估的视角。作为核心底层能力的知识图谱具有极强的复用性，事实上它已经被后续构建的每一个企业级 AI 项目反复使用。如果将全部地基成本强行扣在第一个项目头上，它就会被贴上“失败”的标签，而后续顺风顺水搭便车的项目则会显得“极度成功”，这种割裂的核算方式既不公允，也脱离了业务实际。 ```mermaid flowchart TD accTitle: 基础能力成本分摊与 AI 投资回报率机制 subgraph S1["底层基础设施建设（高成本/重投入）"] A["业务与技能分类体系 (Taxonomy)"] --> B["知识图谱底层底座 (Knowledge Graph)"] end subgraph S2["上层应用场景矩阵"] B -.->|"不合理模式：独自承担全部成本"| C["首个试点项目（职业导师 AI） 账面 ROI 极差 / 被误判为失败"] B ==>|"合理模式：战略分摊基础成本"| C B ==>|"能力复用与成本分摊"| D["后续 AI 项目 A（智能招聘）"] B ==>|"能力复用与成本分摊"| E["后续 AI 项目 B（员工培训）"] end ``` 针对这一矛盾，真正有效的解法在于企业必须具备顶层的 AI 战略规划，清晰预判哪些后续项目将复用这项基础能力，并在组合层面合理重新分摊成本。然而现实中的最大阻碍在于，许多企业在开发 AI 产品时往往采取分散、割裂甚至纯实验性的推进方式，缺乏集中的 AI 产品组合管理视角，导致根本无法在前期将底层图谱的成本系统性地分摊开来。这种核算缺陷带来了极大的决策隐患：企业很可能会因为“账面 ROI 不好看”而中途砍掉那些极具战略价值、高效的核心基础项目；反过来，又可能盲目投资那些看似便宜但实际综合成本高昂的表面工程。 - **图谱投入与接口成本对比**：聊天机器人界面的接入在当前技术条件下耗时极短且边际成本微弱，但底层的技能分类体系与知识图谱构建占据了项目绝大部分的资金与工时。 - **孤岛式实验的弊端**：缺乏中心化投资组合战略会导致基础能力沉淀被当成单点沉没成本，使高价值项目面临被错误叫停的风险。 --- ### [14:43](https://www.youtube.com/watch?v=dqQiZW82sB0&t=883s) AI 原生业务重塑与“非强制 AI”的价值归属在探讨企业级 AI 时，最具争议但也最受推崇的场景是“AI 原生业务重塑”（AI-First Business Transformation/Design）。这类项目通常由专门的 AI 预算立项，其推进前提是对企业现状的“数据、人员、流程和技术”进行极其详尽、精准且及时的全盘梳理。风趣的是，几乎每家企业在启动时都坚信自己对内部流程了如指掌，但在实际调研中，没有一家企业的自我认知偏差低于 15%，通常偏差甚至更大。其根源在于：管理层以为的流程运作方式，与基层员工每天实际执行的工作细节完全是两码事。如果无法在现状梳理中看清真实业务流，后续的所有 AI 部署都会彻底跑偏。 ```mermaid flowchart LR accTitle: AI 优先业务转型的价值创造与归因结构 subgraph Input["全额立项"] A["AI 专项预算 / 转型项目"] end subgraph Process["现状深度梳理（纠偏 >15%）"] B["数据架构优化"] C["业务流程重组"] D["人员职责明晰"] E["局部 AI/文档提取"] end subgraph Output["业务价值释放"] F["低成本、即时见效的高额业务回报"] end A --> B & C & D & E B & C & D -->|贡献绝大部分核心价值| F E -->|仅占最终方案极小部分| F ``` 在理清真实业务流程之后，团队往往会得出一个出人意料的结论：AI 最终往往只占整体解决方案和价值释放中非常小的一部分。许多业务堵点最直接、成本最低、见效最快的解法，其实是传统系统集成、数据模型微调、岗位职责梳理或基础业务流程优化。EPAM 在推进此类项目时始终向客户明确一个核心原则：“这是一个 AI 项目，而且是 AI 优先的（AI-first）。但它绝不是‘唯 AI 论’（AI only），也不是‘强行 AI’（AI forced）”。只要非 AI 的传统方案能创造更高价值、更简单地解决痛点，团队就会优先采纳。这种做法随即引发了极其棘手的价值归因难题：整个项目的启动资金明明扣自 AI 专项预算，整个咨询团队也是以 AI 为导向进场的，但除了少数如文档信息抽取等环节确实运用了 AI 之外，绝大多数成本和产出都落在修补数据、优化流程、调整岗位职责等“诚实而传统的数字化改造”上。如果单纯为了迎合汇报需求，人们完全可以把所有功劳都强行包装成“AI 带来的巨大 ROI”；但从技术本质来看，这根本不能算作纯 AI 的功劳。为了避免这种主观粉饰或互相扯皮，企业必须在项目启动前就设计好清晰的价值归因模型，明确区分不同优化手段的贡献，才能做出理性的投资决策。 - **企业流程认知偏差**：企业管理层汇报的业务流程与一线实际操作之间普遍存在至少 15% 以上的脱节。 - **AI 优先而非强行 AI**：优先探索 AI 潜能，但坚决拒绝为用而用；低成本的系统集成与流程重构往往贡献了转型中最大份额的 ROI。 - **归因设计前置**：在耗费 AI 专项预算进行全盘重塑前，必须提前确立跨职能、跨技术模块的价值核算标准。 --- ### [18:17](https://www.youtube.com/watch?v=dqQiZW82sB0&t=1097s) 从具体工具到业务成果：重新审视 AI 的衡量标准演讲通过三个来自实际业务组合的真实案例（法律助手、职业规划顾问、AI 原生业务重塑），展示了 AI 衡量维度的演变规律。一个清晰的趋势是：项目距离单纯的 AI 算法/工具越远、与复杂的企业业务生态融合越深，ROI 的计算就越具挑战性，评价指标的主观性也就越强。 ```mermaid flowchart LR accTitle: 案例复杂度与 ROI 主观性递增关系 A["案例 1：法律助手 （工具级应用 / 相对直接）"] --> B["案例 2：职业规划顾问 （涉及技能图谱沉淀 / 中度复杂）"] B --> C["案例 3：业务架构重塑 （全盘组织重构 / 极度主观）"] ``` 当面临越来越主观的衡量环境时，企业越需要在前期把业务定义和归因逻辑梳理清楚。在过去数月完成的 50 多个交付项目中，存在着各种各样的业务形态与度量方式，但所有实践最终都指向同一个核心结论：在董事会和管理层中广泛流传的“我们的 AI 投资回报率是多少？”本身就是一个错误的问题。真正能够穿透主观争议、驱动自信决策的衡量方式，应当直接锚定在最终的**业务成果（Business Outcomes）**上。只要跳出对“AI 单一技术归属”的狭隘纠结，转向衡量具体业务目标的达成，ROI 的度量困境便能迎刃而解。 - **三阶段递进模型**：随着 AI 项目从单点工具演变为组织级业务设计，ROI 衡量的客观性逐渐减弱，主观性与定义难度成倍上升。 - **重新定义核心命题**：摆脱对“纯 AI ROI”的微观抠算，转向以业务成果为中心的全口径成效评估，是破除当前 AI 投资焦虑的关键出路。 ## Notable quotes > [00:00](https://www.youtube.com/watch?v=dqQiZW82sB0&t=0s) What's our ROI on AI? — John Capriotti > [03:13](https://www.youtube.com/watch?v=dqQiZW82sB0&t=193s) 这问题太蠢了，简直就像在问‘HTML 的 ROI 是多少？’ — John Capriotti > [03:13](https://www.youtube.com/watch?v=dqQiZW82sB0&t=193s) 给这个错误的问题找到正确的答案 — John Capriotti > [03:13](https://www.youtube.com/watch?v=dqQiZW82sB0&t=193s) ROI = 价值 (Value) / 成本 (Cost) — Ira Livshits > [14:43](https://www.youtube.com/watch?v=dqQiZW82sB0&t=883s) 这是一个 AI 项目，而且是 AI 优先的（AI-first）。但它绝不是‘唯 AI 论’（AI only），也不是‘强行 AI’（AI forced） — John Capriotti > [18:17](https://www.youtube.com/watch?v=dqQiZW82sB0&t=1097s) 在董事会和管理层中广泛流传的‘我们的 AI 投资回报率是多少？’本身就是一个错误的问题 — John Capriotti > [18:17](https://www.youtube.com/watch?v=dqQiZW82sB0&t=1097s) 真正能够穿透主观争议、驱动自信决策的衡量方式，应当直接锚定在最终的业务成果（Business Outcomes）上 — John Capriotti ## Mentioned EPAM Systems, John Capriotti, Ira Livshits, Star Trek, Kirk, Spock, Ren & Stimpy, Abbott and Costello, Peppa Pig, HTML ## Video description John Capriotti - VP, AI-First Business Design, EPAM Systems Ira Livshits - Principal Data Scientist, EPAM Systems You've asked it. Your board has asked it. Your CFO wants to know: "What's our ROI on AI?" The answer should drive confident decisions—but consistently fails. Not because AI doesn't work or you can't measure it, but because the question itself is fundamentally broken. In 20 minutes, we'll show you why and give you something better: a question you can actually answer and a method you can defend. [Source](https://www.youtube.com/watch?v=dqQiZW82sB0)

Source quotes

[0:00] John Capriotti: What's our ROI on AI? [3:13] John Capriotti: 这问题太蠢了，简直就像在问‘HTML 的 ROI 是多少？’ [3:13] John Capriotti: 给这个错误的问题找到正确的答案 [3:13] Ira Livshits: ROI = 价值 (Value) / 成本 (Cost) [14:43] John Capriotti: 这是一个 AI 项目，而且是 AI 优先的（AI-first）。但它绝不是‘唯 AI 论’（AI only），也不是‘强行 AI’（AI forced） [18:17] John Capriotti: 在董事会和管理层中广泛流传的‘我们的 AI 投资回报率是多少？’本身就是一个错误的问题 [18:17] John Capriotti: 真正能够穿透主观争议、驱动自信决策的衡量方式，应当直接锚定在最终的业务成果（Business Outcomes）上

Source quotes

[0:04] Ira Livshits: sometimes can be easily confused with a hothead [2:18] What is the ROI of AI? [2:18] John Capriotti: That's a dumb question. It's like asking, 'What's the ROI of HTML?' [11:27] universe of skills [14:43] John Capriotti: always at least 15% off [14:43] John Capriotti: AI-first, but not AI-only and not AI-forced. [14:43] John Capriotti: traditional, honest, let's fix the data, let's fix the processes, let's fix the people's roles. [18:17] What is the ROI on our AI? What's our ROI on AI? The answer should drive confident decisions—but consistently fails. Not because AI doesn't work or you can't measure it, but because the question itself is fundamentally broken.

Original transcript

[0:04] What we're going to talk today about is the question that is being asked by so many of [0:08] our leaders at so many of our companies, which is what is the ROI of AI? Literally, that's the [0:14] depth of the question that we're hearing from many of our uh of our client leaders. Um, let [0:19] us introduce ourselves. So, I'm John Capriyotti. This is a she's gonna walk us through why we are [0:24] wearing Star Trek outfits. Oh, thank you, John. Hello, everyone. very nice to to see you here. So, [0:30] we are at AI conference. We thought we'll make it a little bit fun for ourselves [0:34] and hopefully for you and we decided that we'll give descriptions. John and I have been working [0:39] together for basically on the daily for a lot. Yeah, you can hear the sadness in his voice. [0:46] We decided to to sort of give descriptions of each other to AI and ask it to suggest [0:54] pairs of collaboration collaborators that fit our descriptions. So for example, what I said about [1:00] John that he's an amazing leader with passion to cognitive psychology and modern technology [1:06] that he blends perfectly. He's a great people's person. elevates, supports people. She is very [1:17] enthusiastic and passionate to in what he does. Sometimes can be easily confused with hotad. Wow. [1:25] He is very eloquent. He's really good with words which he very often uses to make fun of me. Floyd [1:30] Kirkensbach. [clears throat] I described Era as an incredible analytic thinker, uh a mathematician, [1:38] someone who is uh calculated but passionate at the same time and able to apply very specific and very [1:46] um intense rigor to the analysis of situations like the ROI of AI. And as a data scientist, [1:52] I I added some details about her history. Um we got some very interesting pairs. We got if [1:57] anybody knows any of these they might they might they might resonate. Ren and Stimpy, [2:03] Abbott and Costello, uh Pinky in the Brain. We settled on Spock and Kirk. That's why we're you [2:10] see us that way. And um that's all about us. So as we thought about this topic, we we wanted to [2:18] really dissect it in a way that like made it made us understand why it was so urgent today. [2:24] And what we really deconstructed was the early days to today. What's happened in the space of AI? [2:30] And we went from early PC's where the cost wasn't an issue. Um technology partners were paying for a [2:37] lot of this work. Uh and the answer was either the we proved the concept or we didn't. That [2:42] moved to pilots. Pilots were all about whether or not the thing was usable on a small scale. [2:48] uh was it adopted by uh by the the people that we had designed it for by the companies we designed [2:53] it for uh and then did it work at that small scale. Today we're at a much different place [2:59] where it is in production it's live it's scaled and the cost is uh adding up and the questions [3:06] around the financials are what are driving the ROI question. So with that said, [snorts] and I [3:13] were in a meeting. We had uh a CEO ask our uh team, well, what's the ROI of AI? Literally, [3:20] that was the question. Now, we have a little game that I like to play with, which is, [3:26] can I make a laugh enough on a call where she has to shut down the camera and shut off her mic? So, [3:31] I immediately texted her. That's a dumb question. It's like asking, "What's the ROI of HTML?" [3:38] And while it was a dumb question, she did shut the mic off and her camera. She laughed. Um, [3:43] but it really it begged the question of is it even a reasonable question to ask? What's the [3:48] ROI of AI? And and our answer is no, it's not. It's the wrong question, but we're not going to [3:54] stop being asked it by the people who are asking. So, how do you get the right answer to this wrong [3:59] question? That's the whole topic we're going to cover today. We're going to walk through an [4:04] age-old equation for ROI. She's a mathematician. She's going to walk us through it. And then we're [4:09] going to apply it to three very specific, very distinct programs that we've completed in the last [4:14] 6 to 12 months and look at how those programs and their differences mean different things [4:21] for an ROI calculation. Thank you, John. I am a mathematician. So I had a little bit of temptation [4:28] to come up with a complex formula. Maybe throw in some derivatives integrate and integrals just to [4:34] make me feel better about myself maybe. But John was very adamant. Let's keep the things to the [4:40] minimum. Let's make it very clear and clean. So basically we have a formula that is time- tested [4:45] old has nothing to do with AI booked for ages with just two parameters. It's value and cost. And we [4:51] know that cost is small, value high, life is good. Yeah. So what we'll spend time talking about here [4:59] is the complexity of calculating value and cost and how to attribute it to it. The simplicity [5:05] simplicity of the equation actually becomes much more interesting and more complicated when you [5:09] apply it to these three cases. So the first case we're going to look at is AI as a legal assistant [5:13] for a company that is responsible for document and records management which includes retention. We're [5:19] going to talk about conversational professional development assistant for a professional services [5:25] organization that is helping its members determine career path and appropriate um uh courses to [5:30] achieve that objective. And lastly, we're going to look at uh next generation AI transformation [5:36] at a business level and what that means and what that looks like. So we'll start with the [5:41] first one which is the legal assistant for uh an international document management company. In this [5:46] case, this record management company onboards new customers and the new customer onboarding [5:50] process involves an attorney sitting with that uh with that organization or that that client [5:56] and understanding the specific needs of their particular vertical industry documents. That [6:02] lawyer then takes that information, looks for the relevant case law or the relevant legal parameters [6:08] around document retention, security, etc. and then creates a contract and contract terms aligned with [6:14] all of those things. It's a pretty involved process. Lawyers are not inexpensive today. [6:20] That'll change. Um and that particular process became a bottleneck in onboarding. So the the the [6:27] challenge was how do we get around this bottleneck without adding tremendous additional cost and [6:31] additional uh legal capacity. And the business objective again was to onboard more customers, [6:37] create more time, make that available. The answer was this AI legal assistant. And this legal [6:41] assistant that was created automatically listened to the conversation, extracted the relevant [6:46] information, found the right finds the right uh legal parameters and then creates the contract [6:52] language that it inserts into a document that the lawyer then uses much much faster, much cleaner, [6:58] much uh much improved in terms of the time it takes to do these things. So let's let's [7:03] deconstruct this from an ROI perspective. So this is an example when the formula actually works. [7:09] The cost is very clear. The cost is of the team that build this legal system and the cost of AI [7:15] that you have to spend on running it. The pattern of usage of AI here is very clean and clear. You [7:22] can predict that the costs for a long time to to be the value becomes much more interesting [7:28] story. So the obvious value the AI legal assistant help cut on boarding time for lawyer in half. So [7:36] there is 50% time freed up which saves the certain amount of dollars for the company which is a clear [7:45] cut cost uh value. You could stop there and that would give you an ROI. That's insufficient. It's [7:52] in and and you you can't stop there to actually get the right calculation. Yes. So the company [7:58] had other ideas than just to let the lawyer go and golf. Instead they had three four business [8:04] objectives of how to reinvest lawyers time and the the objectives were on board new clients [8:12] client all existing client retention expanding to new verticals expanding to new geographical [8:21] locations. The first question in the value that we should ask ourselves is like they all have [8:28] different components of that to them. So how do you distribute in your value formula? How do you [8:35] assign values to those four? That's part number one. Part number two here is the time it takes [8:41] for this value to mature. When you're on board a new client, you get to see the value right away. [8:47] You have to ask yourself, do I count just the first year signing bonus or the average time [8:52] when we retain the client? That's number one. When it comes for example to seeing the effects [8:57] of a new vertical that's a much longer story it much it takes a lawyer much longer to get there [9:03] and then to onboard new clients. The same is with geography. So you have to decide how you measure [9:09] time how much you allow time for maturing and how do you put it in the value formula. So the [9:15] factors are what is saved great what is done with that savings as it's reinvested and then what is [9:21] the appropriate time interval against which we are measuring these improvements so that [9:26] we're getting an accurate calculation. The time interval varies by business objective. It's an [9:31] important calculation. We'll go on to the next one. So this is the conversational professional [9:36] development adviser uh for a professional services organization has it's it's driven by membership uh [9:41] and their success is a function of how engaged the members are and how much value they can provide to [9:48] their members. In this particular instance, there were some specific B business problems [9:54] uh associated with their current state, which was that their membership engagement was extremely [10:00] low, particularly after the first few months. And that engagement was low and it was it was actually [10:06] preventing some of their really good coursework and professional advancement capabilities uh from [10:12] being realized by these particular members. Uh and that was a problem. So the idea was let's [10:17] provide a career advisor so people can go from where they are today to where they want to go [10:22] um and actually map it to the appropriate courses, career path and uh and certifications that go [10:28] along with it. Let's provide that to members. It'll solve these problems. That's what we did. [10:33] Uh it was deemed successful and on the surface it seems very very similar to the one we just [10:38] talked about. It's an assistant. It's most it's AI specific. There is a big difference [10:44] though between the two. In this particular case, there was a foundational element that was not in [10:49] place. In fact, this company attempted to do this themselves. They did a P for personalization. The [10:54] PC worked because personalization was very very thin at that point. Uh when they went to pilot, [11:00] it failed because of that thinness. And that thinness was based on a lack of skills taxonomy [11:05] and a lack of a knowledge graph underpinning and connecting all of these pieces. uh is going to [11:10] walk us through like the challenges of that and how we would then walk uh how we would calculate [11:14] ROI accordingly. Thank you. So the reason why the pilot didn't work because the career advisor knew [11:21] nothing about the members that is actually useful. So what needed to be established is [11:27] a universe of skill that is relatable and useful in this profession. Being able to identify what [11:33] member belongs in this universe of skills and what skill gap he has from what he current where they [11:41] currently are and where they want to be. And to be able to build all of that, we had to a [11:49] establish a skills taxonomy and then map skills content, everything else to the skills uh to [11:56] skill data set resulting in the graphs. The graph was the biggest investment, most time consuming, [12:04] most expensive part of this thing. You can now I mean you all know now that creating a chatbot [12:12] is does not take time or money. So costwise we created a non AI tool that basically connected [12:23] the entire universal organization works. Now all of this cost was attributed to this particular [12:29] project. If you look at the value, the values are members engagement, member satisfaction, members [12:36] engagement with with the tool that we built. Not a huge value. We've met the business objectives [12:43] here, but the cost was tremendous. Does it mean this projects fail? It's really dependent how you [12:50] look at it. their foundational capabilities, this knowledge graph will be reused and is [12:57] reused in the in every next AI project that this company builds. So if the costs are attributed [13:05] only to that that's in that's not a successful project and everything after that will be much [13:12] more successful. Is it fair? Not is it actually relevant? No. So what would be a good solution [13:18] here? If the organization actually has a strategy and knows which next project will be reusing this [13:25] capability, maybe redistribute the cost and this will give a much better representation of [13:31] what happened and much better advice in decision making. Yeah. So the challenge there is there's a [13:37] very um inconsistent distributed way of actually creating product with AI in these organizations [13:42] and probably in many of yours. um that distributed or almost experimental nature means there's not a [13:48] lot of centralized portfolios of AI product that you can then say this knowledge graph will feed [13:54] all of them let's distribute the cost across in any real consistent way um by having that strategy [14:01] and by connecting all those dots up front which is not easy you're able to distribute the cost [14:06] correctly you are then able to actually determine whether or not the thing you just did was worth it [14:11] and should you persist or should you kill it and if If you don't do that properly, you will end up [14:16] terminating projects and programs and products that are extremely effective but don't look ROI [14:21] positive and you'll invest in ones that seem to be very inexpensive but in fact when you actually [14:27] look at the real cost it's it's it's a lot more money. So very important to understand how to [14:32] distribute foundational element costs on which lots of AI product and capability can be built. Uh [14:38] it's a challenge but it's a challenge that's worth addressing so you can make the right decisions. [14:43] The next example is probably my favorite. Uh it's the one that is maybe most controversial and it's [14:49] when you have an AI first business transformation or business design. And that's a fairly broad [14:55] topic. Um and with a question that is as narrow as what's the ROI and as general as what's the ROI, [15:04] which is interesting. How do you get to the right answer here? So let's walk through what [15:09] this looks like. In this particular instance, the budget was earmarked from an AI budget. [15:15] It's AI transformation. The program itself involved and should involve when you do this [15:20] kind of transformation, a very detailed and very accurate and very timely understanding of current [15:26] state data, people, process, technology has to be understood by understanding that. And by the way, [15:32] every company thinks they already have that. And there has not been a company that I've worked with [15:36] that got it right. It is always at least 15% off. Usually, it's more than that. And it's because [15:42] the people who think they know the process tell you one thing and the people doing the work tell [15:45] you something else. You have to get it right or you're going to get the AI pieces wrong. When you [15:51] go through that process, what we found is that AI becomes a very very small part of the end solution [15:57] and the end value realized. In fact, there are far more valuable, less expensive, less complicated, [16:05] more immediate solutions to a lot of the business process challenges that we identify [16:10] that really drive the value. So we, you know, we'll tell clients during something like this, [16:14] this is an AI project and it's AI first. It is not AI only and it is not AI forced, which means if we [16:20] could use some other capability, if we can do an integration, a different change to a data model, [16:25] something like that, we're going to do it. It's going to create value and it's going to solve a [16:29] problem. How do you factor that into the value of AI when it's an AI budget and it's AI first [16:35] business transformation and you get a lot of different answers depending on who you asked [16:39] and you get a lot of different an you can you can create a different answer that serves your purpose [16:44] if you want to. What I mean by that is [clears throat] that well you're we should go to the [16:50] calculations slide. Yeah, that's all right. Then the conflations will be just really very clear not [16:55] clear the opposite of clear but it's very simple to explain the cost of the cost were our team that [17:03] was a part and the team started with AI first so let's attribute to AI everything every other cost [17:10] with little tiny exceptions of maybe extracting information from the document that we could have [17:16] used AI and and and we have done so everything else is traditional, honest, let's fix the data, [17:24] let's fix the processes, let's fix the role, the people's roles. That was the cost. Whether you [17:31] want to attribute it to AI, go ahead. Is it really AI? Absolutely not. In in our experience so far, [17:40] these have the biggest potential for value generation, but it's the least connected [17:46] directly to AI because it's an AI budget. How do you calculate it? How do you determine [17:51] what's valuable or what's assigned to where? And depending on what you want to do next, you might [17:56] want to attribute it to to AI. You might want to attribute to some other form of optimization. Um, [18:02] but it is a necessary thing to figure out ideally before you even start this project. [18:07] So you have an attribution design and model in place already and you're you're attributing [18:12] the value to the right things and you're able to then move forward with with with good decisions. [18:17] So these are the three specific examples we walk through. It's a legal assistant, a career advisor, [18:21] and this business design approach. The further removed from the direct connection to AI, [18:28] the more challenging the ROI calculation and the more subjective it becomes. So with these [18:34] three examples, real world from our portfolio, distinct business problems and in increasingly [18:41] um subjective measurement and criteria. And as you get more subjective, the more you do upfront [18:46] to define things, the easier things are going to be as you move forward. That's the sum total of [18:50] what we uh wanted to share today. We are certainly open to questions and conversation. Would love to [18:55] talk to anybody afterwards. These are three of probably 50 things that we've completed in the [19:00] last few months, several months. Um and there are lots of different examples and lots of different [19:05] flavors of how you can do this. But the question itself is still the wrong question. It should be [19:10] based on business outcomes, which answers a lot of the subjectivity. A lot of places [19:16] aren't there yet. Hopefully, we'll get there. Thank you. Thank you very much. [applause]