Pay-i CEO Tepper Outlines Framework to Quantify AI Agent ROI

Measuring the Business Value of Agents With Pay-I, Inc — Ai4

Ai4

Enterprise AI agents only drive productivity when task completion rates exceed human verification time relative to manual effort, as variable token costs and latency disrupt traditional financial models.

AI-assisted summary of Measuring the Business Value of Agents With Pay-I, Inc — original source. Check the original for full context.

Source published:

Detailed notes

## Background This content was published on YouTube by the Ai4 channel on August 26, 2025. It features a talk by David Tepper, CEO of Pay-I, Inc., who addresses how enterprises can move past the hype of generative AI and accurately measure, quantify, and manage the real business value, cost, and ROI of AI agents.

## TL;DR David Tepper breaks down agentic software into workflows, pipelines, and fully autonomous systems, emphasizing that evaluating their worth requires focusing on three core buckets: financial value, time savings, and productivity. He explores the hidden cost and latency complexities of Large Language Models, including token pricing discrepancies, multimodal overhead, and context-window degradation. To help organizations systematically track ROI, Tepper introduces version-bound KPIs, mathematical formulas for time savings and verification overhead, and outlines how Pay-i helps businesses govern AI spend and measure real-world performance.

## Atmosphere & style The session carries an authoritative yet accessible, pragmatically grounded tone typical of an experienced enterprise software leader. Tepper blends technical expertise with dry humor and self-deprecating remarks about the stress and fast-paced nature of the AI field. The delivery feels like a high-level executive briefing that balances rigorous mathematical frameworks with practical advice for navigating the complexities of AI adoption.

## Key takeaways - [00:04](https://www.youtube.com/watch?v=0mbItBdqlko&t=4s) The speaker, David Tepper, brings deep enterprise AI expertise, having spent 19 years at Microsoft managing internal AI platforms and advising Fortune 500 boards, the U.S. Congress, and the UN. - [01:28](https://www.youtube.com/watch?v=0mbItBdqlko&t=88s) Agentic software falls into three distinct architectures: Workflows (code calling LLMs), Pipelines (fixed multi-step sequences like standard RAG), and Agentic Systems (autonomous runtime decisions and tool selection). - [02:48](https://www.youtube.com/watch?v=0mbItBdqlko&t=168s) Enterprise business value for AI agents should be measured across three core, quantifiable buckets: Financial Value (revenue/cost savings), Time Savings, and Productivity Gains. - [04:34](https://www.youtube.com/watch?v=0mbItBdqlko&t=274s) Calculating agent costs is highly complex due to tokenizer differences, undisclosed backend vision pricing models, SDK disparities, and compound execution overhead from multi-step reasoning. - [07:25](https://www.youtube.com/watch?v=0mbItBdqlko&t=445s) Evaluating AI time savings requires analyzing specific sub-components of GenAI latency, including Time to First Token (TTFT), Inter-Token Latency (ITL), Time to First Completion Token, and Output Tokens Per Second (OTPS). - [09:27](https://www.youtube.com/watch?v=0mbItBdqlko&t=567s) Agents must be evaluated by specific use cases and versioned strictly, as minor changes to system prompts, underlying data, or models drastically alter gross margins and operational KPIs. - [12:00](https://www.youtube.com/watch?v=0mbItBdqlko&t=720s) Over 99% of enterprise production KPIs for agents fall into four structural metric types: Booleans, Numbers, Percentages, and Likert rating scales. - [13:19](https://www.youtube.com/watch?v=0mbItBdqlko&t=799s) Quantifying true time savings requires incorporating a quality/satisfaction factor to account for agent failures or tasks completed without meaningful efficiency gains. - [14:53](https://www.youtube.com/watch?v=0mbItBdqlko&t=893s) An agent genuinely increases productivity only if its probability of success exceeds the time required to verify the output relative to the time needed for a human to perform the task from scratch. - [15:36](https://www.youtube.com/watch?v=0mbItBdqlko&t=936s) Pay-i provides a GenAI FinOps platform for tracking spend, setting governance, preventing runaway agent costs, and measuring ROI, including a strategic partnership with AWS on their VALUE initiative.

## Detailed walkthrough ### [00:04](https://www.youtube.com/watch?v=0mbItBdqlko&t=4s) Introduction and Background of David Tepper

David Tepper introduces the session, *Measuring the Business Value of Agents*, thanking attendees for navigating a last-minute room change. Before jumping into the core framework, Tepper establishes his long-standing background in artificial intelligence. He has been working with Generative AI since 2009 and holds a patent filed back in 2011. During a 19-year tenure at Microsoft, he managed the internal AI software platform used by Azure’s 45,000 employees. He eventually turned down an offer to report directly to Microsoft's CTO—a role overseeing internal AI adoption across the entire enterprise—in order to found Pay-i.

Throughout his career, Tepper served as a primary AI advisor to Microsoft leadership, Fortune 500 executives, the U.S. Congress, and recently spoke at the United Nations regarding AI's impact on developing nation-states. He jokes self-deprecatingly about the complexity of the space showing a picture of himself prior to starting his startup journey, joking that he lost all his hair along the way.

- **Microsoft Tenure:** 19 years leading internal AI developer tool platforms. - **Enterprise Scale:** Managed internal AI infrastructure serving 45,000 Azure employees. - **Global Policy & Executive Briefings:** Advised Fortune 500 boards, U.S. Congress, and the United Nations.

---

[01:28](https://www.youtube.com/watch?v=0mbItBdqlko&t=88s) Defining "Agent": Workflows, Pipelines, and Agentic Systems

Tepper tackles the term "agent," pointing out that it has become an overused buzzword in tech marketing where "everything seems to be agentic." To establish a rigorous baseline, he categorizes agentic software architecture into three distinct operational models:

1. **Workflows:** Traditional software programs that make discrete calls to Large Language Models (LLMs) to handle unstructured-to-structured data transformations. Once structured, conventional code executes the remaining business logic. 2. **Pipelines:** Predetermined multi-step sequences incorporating LLMs, such as standard Retrieval-Augmented Generation (RAG) chatbots or fixed data-processing pipelines. These systems lack real autonomy because they follow a fixed path without making dynamic runtime decisions. 3. **Agentic Systems:** Truly autonomous systems that evaluate choices dynamically at runtime. They select and utilize tools interactively, resulting in non-deterministic execution paths and variable response times.

``` +-------------------------------------------------------------------+ | SOFTWARE MODELS | +-------------------------------------------------------------------+ | 1. WORKFLOW: Code -> LLM (Unstructured to Structured) -> Code | | 2. PIPELINE: Fixed Multi-Step Sequence (e.g., Standard RAG) | | 3. AGENTIC: Dynamic Runtime Decisions & Variable Tool Selection| +-------------------------------------------------------------------+ ```

---

[02:48](https://www.youtube.com/watch?v=0mbItBdqlko&t=168s) The Three Buckets of Business Value

To evaluate AI deployments, organizations must move beyond qualitative feelings—such as the "warm and fuzzy feeling" in non-profit or caregiving scenarios—and focus on universally quantifiable metrics. Based on interviews with hundreds of enterprise customers, Pay-i classifies business value into three primary categories:

- **Financial Value (Raw Currency):** Direct top-line revenue expansion or bottom-line cost reduction (e.g., automation savings). - **Time Savings:** Quantifiable reduction in operational effort required to complete specific tasks compared to legacy manual baselines. - **Productivity Gains:** Expressed as output per unit of time (e.g., increase in customer service tickets closed per representative per day).

---

[04:34](https://www.youtube.com/watch?v=0mbItBdqlko&t=274s) The Hidden Complexities of LLM and Agent Costing

Calculating standard ROI appears simple (Net Profit divided by Cost), but tracking the *cost* side of an AI agent is surprisingly intricate. Unlike standard software subscriptions, AI consumption costs fluctuate based on tokenizer variations, provider pricing models, and multimodal workloads.

For instance, multimodal vision calls (such as OpenAI's GPT-4o) depend on proprietary backend image-tiling calculations with variable pricing metrics. In API billing statements, token counts for vision calls are blended together, obscuring the precise cost attribution for visual processing versus standard text generation. Tepper notes that he maintains an 80-slide presentation dedicated solely to decoding GPT-4o cost management.

``` +-------------------------------------------------------------------+ | AGENT COST CALCULATION FACTORS | +-------------------------------------------------------------------+ | * Tokenizer Differences (Variation across model vendors) | | * Multimodal Inputs (Blended visual/text API pricing models) | | * SDK Discrepancies (Pricing variance based on access method) | | * Reserved Instance Tracking (Requires multi-dimensional math) | | * Compound Execution (Multi-step reasoning and tool calling) | +-------------------------------------------------------------------+ ```

- **SDK Pricing Variations:** Calling the exact same model via different SDKs or API abstractions can yield distinct pricing profiles. - **Reserved Capacity Complexity:** Allocating static reserved instances across dynamic dynamic agent requests requires multi-dimensional modeling. - **Delayed Provider Tooling:** OpenAI posted a note promising a dedicated tracking feature for reserved instance attribution in "Q2 of 2023," a milestone that remains unreleased years later. - **Compound Execution Overhead:** A single user request to an agent triggers multiple internal reasoning steps, sub-queries, and tool invocations, each incurring variable token costs.

---

[07:25](https://www.youtube.com/watch?v=0mbItBdqlko&t=445s) Dissecting GenAI Latency Across Execution Phases

When measuring time savings, basic end-to-end response time fails to capture the user experience or system behavior. Generative AI requires analyzing four distinct sub-components of latency:

1. **Time to First Token (TTFT):** The duration until initial output streaming begins. High TTFT impacts perceived responsiveness. 2. **Inter-Token Latency (ITL):** The speed at which subsequent individual tokens stream. ITL is typically much faster than TTFT because the model has already completed its initial prompt evaluation. 3. **Time to First Completion Token (TTFCT):** In reasoning models, this represents the internal thinking time spent solving a problem before emitting the user-facing answer. 4. **Output Tokens Per Second (OTPS):** The streaming generation throughput. Long-context models like Google Gemini can slow down noticeably as generation approaches context limits due to the compute cost of processing growing context windows.

To illustrate how reasoning models operate before emitting tokens, Tepper uses a maze analogy: if an LLM is asked to output the single fastest path through a maze with multiple entry points, it cannot stream the correct initial step until it has solved the entire path internally.

---

[09:27](https://www.youtube.com/watch?v=0mbItBdqlko&t=567s) Structuring Agent### [00:04](https://www.youtube.com/watch?v=0mbItBdqlko&t=4s) Introduction and Speaker Background

David Tepper, CEO of Pay-I, Inc., introduces the session on measuring the business value of AI agents, acknowledging a last-minute room change that attendees had to navigate. He sets an agenda covering his background, definitions of agents and business value, methodologies for measuring their combination, and concrete real-world examples. Tepper shares his extensive history in the generative AI space, noting that he began working on GenAI in 2009 and filed his first patent in 2011. He spent roughly 19 years at Microsoft, where he managed the internal software platform that enabled Azure's 45,000 employees to leverage generative AI. He eventually turned down a role reporting directly to Microsoft's CTO—a position where he would have overseen internal AI adoption across the entire company—in order to launch Pay-I.

During his tenure at Microsoft, Tepper regularly briefed internal executives on how generative AI would transform products and services. He also represented the company externally, speaking on GenAI to Fortune 500 leadership, academic institutions, the U.S. Congress, and the United Nations, where he presented six months prior on how AI will impact developing nation-states. He jokes about how AI transformed from a niche area into a massive industry explosion, humorously pointing out a photo of himself with hair prior to taking on the complicated, stress-inducing field.

- **Speaker Profile**: David Tepper, CEO of Pay-I, Inc. - **Key Milestones**: Working in GenAI since 2009; first patent filed in 2011; 19-year tenure at Microsoft. - **Key Engagements**: Briefed U.S. Congress, Fortune 500 boards, and the United Nations on AI impacts.

---

[01:28](https://www.youtube.com/watch?v=0mbItBdqlko&t=88s) Defining an Agent: Workflows, Pipelines, and Autonomous Agents

Addressing the hype surrounding the word "agent," Tepper jokes that the term has become so overloaded that people treat everything—including the podium or audience members—as "agentic." To cut through the buzzwords, he establishes a taxonomy that categorizes agentic software into three distinct technical definitions:

1. **Workflows**: Traditional software applications that invoke a Large Language Model (LLM) at specific points in their process. This pattern is commonly used to parse unstructured data into structured formats so traditional code can execute subsequent tasks. 2. **Pipelines**: Predetermined sequences of execution steps where LLMs handle specific parts. A standard example is a Retrieval-Augmented Generation (RAG) chatbot; while it uses LLMs, it is not strictly autonomous or self-directing because it executes a fixed, repeatable path. 3. **Truly Agentic Systems**: Autonomous systems that make runtime decisions, dynamically select and utilize different tools, and feature non-deterministic execution times and paths.

Tepper emphasizes that Pay-I's methodology for measuring business value applies universally across all three definitions, ensuring enterprises can evaluate value regardless of which architectural pattern they implement.

``` +-----------------------------------------------------------------+ | AGENTIC TAXONOMY | +-----------------------------------------------------------------+ | 1. WORKFLOW | | [Unstructured Data] --> [LLM Parsing] --> [Structured Code] | | | | 2. PIPELINE | | [User Query] --> [RAG Retrieval] --> [LLM] --> [Response] | | (Fixed, predetermined execution path) | | | | 3. TRULY AGENTIC | | [Goal] --> [Runtime Decision] --> [Tool Choice] --> [Output] | | (Dynamic execution, non-deterministic path and duration) | +-----------------------------------------------------------------+ ```

---

[02:48](https://www.youtube.com/watch?v=0mbItBdqlko&t=168s) Defining Business Value and its Three Buckets

Moving from technical definitions to business metrics, Tepper defines business value through quantifiable components rather than subjective "warm and fuzzy feelings." Based on conversations with hundreds of enterprise customers—primarily through the Office of the CIO—Pay-I categorizes quantifiable business value into three core buckets:

- **Financial Value (Raw Currency)**: Value generated via direct monetary metrics—either driving net-new revenue streams or cutting operational costs (jokingly referencing the popular tech trope "#AI is taking all of our jobs"). - **Time Savings**: Reductions in human task duration, measured by comparing the baseline time required for a human to complete an action versus the accelerated timeframe achieved using agentic tools. - **Productivity**: Measured as output generated per unit of time (e.g., the volume of customer support tickets processed within a single work shift).

---

[04:35](https://www.youtube.com/watch?v=0mbItBdqlko&t=275s) The Complexity of Calculating Agent Costs

Tepper breaks down the standard Return on Investment equation ($\text{ROI} = \frac{\text{Net Profit}}{\text{Cost}}$), explaining that while net profit stems from savings or revenue, calculating the *cost* side for LLMs and agents is notoriously complex. Standard software pricing model assumptions break down with LLMs: two models with identical published pricing doing identical tasks can yield wildly different costs because they utilize different tokenizers, or do not charge on a per-token basis at all.

Multi-modal operations introduce hidden layers of variable pricing. Using OpenAI's GPT-4o vision pricing algorithm as an example, Tepper explains that back-end calculations vary dynamically without explicit disclosure to developers. When receiving an API response, the system returns a unified token count without breaking down how many tokens were consumed by visual processing versus text processing, making fine-grained cost allocation difficult.

``` +-----------------------------------------------------------------+ | LLM AGENT COST COMPLEXITY | +-----------------------------------------------------------------+ | MULTI-MODAL HIDDEN COSTS: | | Image Input --> [Variable Back-end Algorithm] --> API Response| | (API returns total tokens without separating vision vs text) | | | | API & SDK DISPARITIES: | | Model X via SDK A <-- Price Discrepancy --> Model X via SDK B| | | | RESERVED INSTANCES: | | Requires multi-dimensional modeling to attribute cost per call| +-----------------------------------------------------------------+ ```

Tepper notes that he maintains an 80-slide deck dedicated solely to calculating the true cost of GPT-4o calls. He highlights several billing edge cases: - Calling the exact same model via different SDKs can result in varying price points. - Model snapshots carry distinct pricing tiers that change over time. - Calculating the unit cost per API call on OpenAI Reserved Instances requires complex, multi-dimensional financial modeling. - He cites a quote from OpenAI's website promising a tracking system for reserved capacity in "Q2 of 2023"—a deadline that passed long ago without resolution.

Furthermore, an agent's operational cost cannot be computed from a single API call. Agents execute non-deterministic sequences of reasoning steps, tool calls, and model invocations, leading to fluctuating token counts and dynamic costs across every execution run.

---

[07:25](https://www.youtube.com/watch?v=0mbItBdqlko&t=445s) Understanding Generative AI Latency Metrics

To accurately calculate time savings, organizations must understand the unique nuances of GenAI latency. Unlike standard software applications where end-to-end latency is the primary metric, generative models introduce several granular latency types:

- **Time to First Token (TTFT)**: Crucial for streaming responses, TTFT measures how quickly the user perceives responsiveness. Tepper shares a counterintuitive insight: research shows LLMs frequently solve the core logic of a prompt *before* outputting the very first token, spending the remainder of the streaming time generating explanations. He compares this to solving a complex maze with multiple exits: the model must calculate the full path internally before it can begin outputting the first correct direction. - **Inter-Token Latency**: The speed at which subsequent tokens are output during streaming, which is typically significantly faster than TTFT because the core logic has already been resolved. - **Time to First Completion Token**: The duration a reasoning model spends "thinking" or processing internal step-by-step logic before it generates the first visible completion token. - **Output Tokens Per Second (OTPS)**: A metric that degrades as context windows grow. Tepper points out Google's Gemini as an example, noting that models with ultra-large context windows (e.g., 1 million tokens) can experience significant performance degradation near the end of the context window due to the computational overhead of processing vast amounts of prior context for each new token generated.

---

[09:28](https://www.youtube.com/watch?v=0mbItBdqlko&t=568s) Structuring Use Cases, Versioning, and Key Performance Indicators

Because the underlying mechanics of GenAI are complex, Tepper advises organizations to evaluate agents by **use cases and tangible outcomes** rather than underlying technology architectures. A use case defines the core objective—such as summarizing a contract or booking a travel itinerary—and must be measured by execution time, total cost, and degree of success.

Tepper cautions that generative AI systems are hyper-sensitive to minor configuration changes. Altering an English prompt, updating underlying data, swapping a model, or tweaking application code drastically shifts performance dynamics. A developer inadvertently adding a single comma to a system prompt can compromise product gross margins, increase reasoning latency, or lower response accuracy. Because of this sensitivity, every metric must be rigorously tied to a specific **version** of the agent.

``` +-----------------------------------------------------------------+ | AGENT VERSION CONTROL | +-----------------------------------------------------------------+ | [System Prompt v1.2] + [Model Version] + [Data Pipeline] | | | | | v | | Evaluated against Quantifiable KPIs (Booleans, Numbers, etc.) | | | | | v | | Yields Granular ROI Metrics (Cost, Latency, Accuracy) | +-----------------------------------------------------------------+ ```

When evaluating enterprise use cases, Tepper excludes terms like "copilot" or "agent," noting that those represent technical tools rather than outcome-driven scenarios. Based on Pay-I's study of enterprise deployments, industry-standard KPIs fall into four core structural types:

- **Booleans**: Binary success/failure flags (e.g., true/false). - **Numbers**: Discrete numeric counts (e.g., count of items processed). - **Percentages**: Accuracy or completion rates (e.g., % of tickets resolved without human intervention). - **Liker / Likert Scales**: Rating scales (e.g., 1-to-5 or 1-to-10 quality ratings).

Tepper notes that these four KPI formats encapsulate over 99% of all production metrics used by enterprises today. By continuously binding versioned execution data to these quantified KPIs, enterprises can measure business value accurately and make data-driven engineering decisions.

[13:19](https://www.youtube.com/watch?v=0mbItBdqlko&t=799s) Calculating Financial Value and Time Savings Equations

David Tepper returns to the core ROI equation to demonstrate how business logic and quantifiable KPIs—such as click-through rates and conversion rates—populate the numerator (e.g., cost savings, revenue generation) and denominator. Beyond basic ROI, evaluating real efficiency requires a more nuanced time-savings equation:

$$\text{Time Savings} = \sum_{i} \left( n_i - (s_i \cdot a_i) \right)$$

In this equation, $s$ serves as a binary satisfaction indicator ($1$ if completed satisfactorily, $0$ if not), $n$ is the standard human execution time, and $a$ is the agent's completion time. Summing these metrics across all items (such as processed documents) accounts for scenarios where an agent is performing exceptionally, completely failing, or succeeding without yielding meaningful time savings.

- **Quality Rating Impact:** Evaluates task satisfaction ($s$) across variables like document size to quantify true time savings across batch operations.

---

[14:53](https://www.youtube.com/watch?v=0mbItBdqlko&t=893s) Defining the Golden Rule of Agentic Productivity

When evaluating true productivity gains, Tepper summarizes the governing principle behind AI-driven workforce efficiency into a core mathematical rule:

$$\text{Productivity Gain Condition: } P(\text{Success}) > \frac{T_{\text{verify}}}{T_{\text{human}}}$$

An agent genuinely increases human productivity only if its probability of completing a task successfully $P(\text{Success})$ exceeds the time required for a human to verify the output ($T_{\text{verify}}$) divided by the time it takes for a human to perform the task manually from scratch ($T_{\text{human}}$). If verification overhead is too high relative to task duration or the agent's success rate is low, the system creates a net loss in productivity.

- **Key Takeaway:** If verification time relative to manual effort outweighs the probability of task success, the agent acts as a net drag on organizational efficiency.

---

[15:36](https://www.youtube.com/watch?v=0mbItBdqlko&t=936s) Summary of Agent Architectures, Business Value, and Pay-i Overview

Summarizing the talk, AI agents fall into three primary categories: simple LLM-enhanced workflows, multi-step sequential pipelines, and fully autonomous agentic systems utilizing Model Context Protocol (MCP) servers and external tools. Quantifying their enterprise value requires categorizing outcomes into direct financial value, time savings, and productivity gains. Managing these deployments in production demands tracking operational variables like failure rates and capacity planning.

``` +---------------------------------------------------------------+ | Agent Architectures | | [LLM Workflows] -> [Sequential Pipelines] -> [MCP Agents] | +-------------------------------+-------------------------------+ | v +---------------------------------------------------------------+ | Business Value KPIs | | [Financial Value] | [Time Savings] | [Productivity] | +-------------------------------+-------------------------------+ | v +---------------------------------------------------------------+ | Pay-i Platform | | [Real-Time Spend] | [FinOps Governance] | [AWS VALUE] | +---------------------------------------------------------------+ ```

Pay-i provides enterprise solutions to track and optimize these metrics, highlighted by recent recognition from *Forbes* and the FinOps Foundation as a dedicated Generative AI FinOps platform. Features include financial governance and real-time spend protection to prevent runaway costs from looping agents. Additionally, Pay-i launched an official partnership with AWS on their "VALUE" initiative (featuring an acronym structure reminiscent of Marvel's S.H.I.E.L.D.), pairing AWS teams with Pay-i tools to guide enterprises through use-case definition, KPI selection, and ROI optimization.

- **FinOps Recognition:** Identified by the FinOps Foundation as the premier GenAI FinOps solution for cost and ROI management. - **AWS Partnership:** Official collaboration via the AWS "VALUE" program to guide enterprise AI adoption and measure ROI. - **Exhibitor Details:** Pay-i demonstrated these capability suites live at Booth 246.

---

[18:06](https://www.youtube.com/watch?v=0mbItBdqlko&t=1086s) Q&A: ROI vs. Deployment Focus and Prioritizing Value Metrics

During the Q&A session, Tepper addresses questions regarding Pay-i's core focus and how organizations should prioritize competing value metrics:

* **ROI vs. Deployment Focus:** Tepper clarifies that Pay-i focuses explicitly on ROI rather than general agent deployment or building tools. While numerous vendors assist in building or deploying agents, Pay-i identifies which specific workflows yield tangible business returns. * **Prioritizing Value Metrics:** When choosing between financial value, time savings, and productivity, prioritization depends entirely on the specific use case and business objectives. For revenue-generating applications, direct financial metrics take precedence; for internal operational tools, time savings and productivity metrics dominate. Organizations should measure all three dimensions across their portfolios, as there is no single "silver bullet" metric for every AI initiative.

## Notable quotes > The hype era of “trust me, it’s working” is ending, and in its place you’ll need the tools to manage the workforce of tomorrow. — David Tepper

> [01:28](https://www.youtube.com/watch?v=0mbItBdqlko&t=88s) everything seems to be agentic. — David Tepper

> [02:48](https://www.youtube.com/watch?v=0mbItBdqlko&t=168s) warm and fuzzy feeling — David Tepper

> [02:48](https://www.youtube.com/watch?v=0mbItBdqlko&t=168s) #AI is taking all of our jobs — David Tepper

> [04:35](https://www.youtube.com/watch?v=0mbItBdqlko&t=275s) Q2 of 2023 — David Tepper

## Mentioned David Tepper, Pay-I, Inc., Microsoft, Azure, U.S. Congress, United Nations, OpenAI, GPT-4o, Google Gemini, Forbes, FinOps Foundation, AWS, Model Context Protocol (MCP)

## Video description David Tepper, CEO, Pay-I, Inc.

Abstracts

The new world of software promises to be teeming with agents and AI powered tools… but how will that impact your business? You've built the POCs and seen the demos, but scaling to production demands answers: What is the real cost per outcome? Which use cases deliver provable ROI? As enterprises race to deploy AI agents and chatbots, boards are demanding quantifiable business value metrics. The hype era of “trust me, it’s working” is ending, and in its place you’ll need the tools to manage the workforce of tomorrow. I’ve been working on AI solutions for the largest companies in the world since 2009, and in this talk I’ll show you how Pay-i is helping companies quantify the real value of every agentic workflow, align teams around the new metrics that drive business growth, and forecast margins before shipping.

[Source](https://www.youtube.com/watch?v=0mbItBdqlko)

中文简报

Pay-i CEO Tepper 概述 AI Agent 投资回报率量化框架 由于可变 Token 成本和延迟打破了传统的财务模型,企业级 AI Agent 唯有在任务完成率相对于纯人工投入超过人工核验耗时的情况下,才能真正推动生产力提升。 David Tepper 将 Agent 软件细分为工作流、流水线以及完全自主系统,并强调评估其价值需要聚焦于三大核心维度:财务价值、节省时间与生产力提升。他探讨了大语言模型隐藏的成本与延迟复杂性,涵盖 Token 定价差异、多模态开销以及上下文窗口衰减等问题。为帮助企业系统化追踪投资回报率(ROI),Tepper 提出了与版本绑定的关键绩效指标(KPI),以及用于计算时间节省与核验开销的数学公式,并概述了 Pay-i 如何帮助企业监管 AI 支出并衡量实际落地成效。 前微软内部 AI 平台负责人、Pay-I 首席执行官 David Tepper 将 AI 软件架构划分为三种形态:调用大模型处理非结构化数据的工作流(Workflows)、如标准 RAG 等固定多步执行的流水线(Pipelines),以及具备动态运行时决策与工具选择的自主智能体系统(Agentic Systems)。 (1:28) 企业衡量 AI 智能体业务价值应聚焦于三大核心量化维度:财务价值(直接增收或降本)、时间节省(相比人工基准的工时缩减)与生产力提升(单位时间的产出量)。 (2:48) 智能体成本核算极具复杂性,受到分词器差异、多模态视觉隐藏计费、SDK 定价差异以及多步推理复合开销等多重因素影响,不能按普通 API 调用的单价简单估算。 (4:35) 评估生成式 AI 真实延迟需要细分首 Token 时间(TTFT)、Token 间延迟(ITL)及首个完成 Token 时间(TTFCT),其中推理模型在输出首个 Token 前通常需要先在内部完成完整逻辑路径推演。 (7:25) Tepper 提出了智能体生产力黄金法则:只有当智能体任务成功概率大于人工验证耗时与从零人工完成耗时的比值时,智能体才能实现真正的净生产力提升,否则反而会拖累组织效率。 (14:53)

Reading summary

David Tepper, CEO of Pay-I, argues that measuring the business value of AI agents requires moving beyond qualitative sentiment to evaluate systems across three quantifiable buckets: financial return, time savings, and productivity. Because agent behavior is non-deterministic and highly sensitive to configuration tweaks, organizations must rigorously track hidden compute costs, granular latency phases, and the mathematical overhead of human verification. Taxonomy of Agent Architectures and Value Buckets (1:28) To cut through industry buzzwords, Tepper categorizes agentic software into three architectural models: traditional workflows that call LLMs for data conversion, deterministic multi-step pipelines such as standard retrieval-augmented generation (RAG), and autonomous agentic systems that dynamically select tools and execute non-deterministic paths at runtime. Regardless of the underlying technical pattern, enterprise evaluation must replace qualitative impressions with quantifiable business metrics. Based on enterprise engagements through the Office of the CIO, Pay-i classifies quantifiable value into three primary categories: direct financial value (top-line revenue expansion or bottom-line labor and operational cost reductions), time savings against manual baselines, and productivity measured as output generated per unit of time. Hidden Cost Realities and Multi-Phase Latency (4:34) Calculating an agent's return on investment is complicated by pricing mechanics that depart from traditional software. Model costs vary based on tokenizer differences, API access methods, and compound tool-calling sequences. In multimodal workloads like OpenAI's GPT-4o, backend image-tiling algorithms blend vision and text processing into a single token count, obscuring granular cost attribution. Additionally, Tepper points out that calculating per-call costs on reserved capacity requires multi-dimensional modeling, noting that OpenAI has yet to release a dedicated tracking feature originally promised for Q2 2023. Evaluating time savings similarly requires breaking end-to-end response times into four distinct latency components: Time to First Token (TTFT), Inter-Token Latency (ITL), Time to First Completion Token (TTFCT), and Output Tokens Per Second (OTPS). Reasoning models often solve an entire problem internally before emitting their initial token—similar to computing a full path through a maze—while ultra-large context architectures, such as Google Gemini, experience throughput degradation as context windows fill up. Versioned Key Performance Indicators and Outcome Tracking (9:27) Tepper advises organizations to evaluate deployments by concrete use cases and business outcomes rather than technology labels like 'copilot' or 'agent.' Because generative AI systems are hyper-sensitive to configuration adjustments—where adding a single comma in a system prompt can degrade gross margins, increase latency, or lower accuracy—performance metrics must be strictly tied to specific versions of prompts, models, and code pipelines. Across production enterprise deployments, Pay-i finds that over 99% of evaluated key performance indicators fall into four structural formats: booleans (binary success/failure flags), numbers (discrete counts), percentages (accuracy or completion rates), and Likert scales (numerical quality ratings). Continuous binding of versioned execution data to these KPI types enables organizations to make data-driven engineering decisions. Mathematical Thresholds Governing Time and Productivity Gains (13:19) Measuring efficiency requires accounting for failure states and human oversight rather than comparing raw operational speeds. Tepper expresses total time savings across operations through a formula that weights agent execution against satisfaction: the sum across tasks of human execution time minus the product of agent duration and a binary satisfaction indicator. This prevents failed or unhelpful agent runs from registering as operational gains. To establish whether an agent truly improves human productivity, Tepper defines a governing condition: the agent's probability of success must exceed the time required for human verification divided by the time it takes for a human to perform the task manually from scratch. If an agent's accuracy is too low or verification requires too much effort relative to manual execution, deploying the agent creates a net loss in organizational productivity. Tepper emphasizes that value prioritization depends on specific operational objectives: revenue-facing applications prioritize raw currency, while internal tools center on time and productivity. Because there is no single universal metric, organizations must track all three dimensions simultaneously to prevent runaway agent costs and ensure net-positive operational returns.

精炼阅读摘要

Pay-i 创始人 David Tepper 指出,企业衡量 AI Agent 的商业价值必须摆脱主观感受,将其落实为可精确度量的财务收益、时间节约与生产力指标,并警惕隐性成本和验证开销带来的效率拖累。 Agent 架构的三种技术形态与边界 (1:28) Tepper 强调,“Agent”已被科技营销过度泛化。为了建立严谨的商业价值衡量标准,他将基于大语言模型(LLM)的软件架构划分为三种形态:第一种是工作流(Workflows),传统软件仅在特定节点调用 LLM,负责将非结构化数据解析为结构化数据,后续业务逻辑仍由传统代码接管;第二种是流水线(Pipelines),由预设的多步骤序列组成,例如标准的检索增强生成(RAG)聊天机器人,这类系统沿用固定路径执行,不具备运行时的自主决策能力;第三种是真正的 Agent 系统(Agentic Systems),具有自主性,能在运行时根据环境动态评估决策、自主调用外部工具(如 MCP 服务器),其执行路径和响应耗时具有非确定性。Tepper 认为,企业的价值衡量框架必须统一适用于这三种不同模式。 成本计量黑盒与生成式延迟细分 (4:35) 计算投资回报率(ROI = 净利润 / 成本)虽然原理简单,但 Agent 的真实成本核算极为复杂。由于不同模型分词器(Tokenizer)各异,即便是标价相同的模型在处理相同任务时实际开销也会相差巨大;甚至通过不同 SDK 或 API 抽象层调用同一模型,其结算单价也存在差异。多模态计算进一步加剧了成本黑盒,例如 OpenAI 的 GPT-4o 视觉调用依赖底层专有的图像切片算法,但在 API 账单中视觉与文本代币(Token)被混在一起结算,导致精准成本归因极其困难。Tepper 透露他为此专门制作了长达 80 页的 PPT 来解析 GPT-4o 成本。此外,静态预留实例分配到动态请求需要复杂的多维数学建模,而 OpenAI 曾承诺在 2023 年第二季度上线的预留实例跟踪功能至今未能兑现;加之 Agent 包含链式推理和多步工具调用,单次运行的代币消耗剧烈波动。 在时间节约层面,单纯统计端到端耗时无法反映实际体验,必须拆解为四类延迟指标:首字时间(TTFT)、字间延迟(ITL)、推理模型的首个完成标记时间(TTFCT)以及每秒输出代币数(OTPS)。Tepper 指出,LLM 往往在输出首个标记前就已在内部完成了核心逻辑推理,如同在多出口迷宫中计算出完整路径后才生成起点指引;而类似 Google Gemini 这类长上下文模型,在接近其上下文窗口极限(如 100 万代币)时,会由于处理庞大历史上下文的计算开销而出现吞吐率(OTPS)大幅下降。 用例导向、版本绑定与标准化指标体系 (9:28) 鉴于底层技术的复杂性,Tepper 建议企业必须摆脱“Copilot”等技术标签,转向以具体业务用例(如合同摘要生成或行程预订)和产出为导向进行度量。生成式 AI 具有极高的配置敏感性:在英文提示词中无意添加一个逗号、变更微调数据或切换模型版本,都会显著影响推理延迟、准确率乃至毛利率。因此,企业的所有业务与技术指标必须严格绑定到具体的 Agent 版本(涵盖提示词版本、模型版本与数据管道)。 在量化指标的选择上,Tepper 将价值归纳为财务价值(直接增收或降本)、时间节约和生产力提升(单位时间产出,如每位客服日均结单量)。根据 Pay-i 对数百家企业客户的调研,当前业界超过 99% 的生产环境 KPI 都可以归纳为四种基础数据类型:布尔值(成功/失败判定)、离散数值(处理数量计数)、百分比(如无人工干预解决率)以及李克特量表(Likert Scales,质量等级评分)。 时间节约方程与生产力增益黄金法则 (13:19) 为了精细度量效率提升,Tepper 提出了改进的时间节约方程:$\text{Time Savings} = \sum_{i} (n_i - (s_i \cdot a_i))$。其中 $n_i$ 代表人类手动完成第 $i$ 项任务的基准工时,$a_i$ 为 Agent 执行耗时,$s_i$ 为二元满意度指标(任务达成且令人满意取 1,否则取 0)。该公式剔除了 Agent 虽执行迅速但结果无效的伪效率,准确反映出多批次任务中的实际净节省时间。 对于真实的生产力增益,Tepper 确立了 Agent 生产力黄金法则(Golden Rule of Agentic Productivity),即只有当 Agent 成功完成任务的概率 $P(\text{Success})$ 大于人类核验输出所需时间($T_{\text{verify}}$)与人类从头手动执行所需时间($T_{\text{human}}$)的比值时,系统才能产生真正的生产力收益:$P(\text{Success}) > \frac{T_{\text{verify}}}{T_{\text{human}}}$。若人工核验成本相对手动执行时间过长,或者 Agent 的成功率较低,系统将引发负生产力,最终成为组织的运作负担。 Tepper 总结指出,Pay-i 与 AWS“VALUE”计划合作并获 FinOps 基金会认可,专注于解决特定工作流的实际 ROI 测算与防范循环失控支出,而非通用构建工具。企业在推进 Agent 落地时不存在通用指标,必须根据用例权衡财务、时间与生产力维度;若无法严格量化配置敏感性、隐性调用成本及人工核验代价,盲目部署 Agent 将面临预算超支与效率反向拖累的双重风险。

Detailed notes — zh

## Background 本文来源于 YouTube 频道 Ai4 于 2025 年 8 月 26 日发布的演讲视频,标题为《Measuring the Business Value of Agents With Pay-I, Inc》。视频页面简介指出,演讲者 Pay-I, Inc. 的 CEO David Tepper 探讨了在企业跨越概念验证(POC)阶段后,如何量化 AI Agent 的真实成本与 ROI,建立量化业务价值指标并在项目上线前预测毛利率。 ## TL;DR 演讲者 David Tepper 首先将 Agent 划分为工作流(Workflow)、流水线(Pipeline)与自主智能体(Dynamic Agent)三种架构形态,并指出了直接计算 LLM 成本与延迟的极高复杂性。随后,他提出了评估 Agent 商业价值的三大核心维度——财务价值、时间节省与生产率提升,并给出了具体的时间计算公式与生产力判定法则。最后,他强调企业应通过按用例拆分、严格版本控制以及量化 KPI 来精准测量 AI 投资回报率,并介绍了 Pay-I 平台及其与 AWS 合作的 VALUE 计划。 ## Atmosphere & style 演讲氛围专业且兼具幽默感,David Tepper 凭借其在微软多年的丰富履历与行业洞察,以轻松自如的台风娓娓道来。面对极其复杂的 Token 计费与算法架构,他善于用自嘲(如谈及 AI 复杂度导致脱发)和直白的金句化繁为简,展现出资深技术专家与创业者的严谨逻辑与敏锐商业嗅觉。整体叙事节奏紧凑,既有硬核的方法论推导,又有接地气的企业落地经验分享,给听众带来极高的信息密度与实用启发。 ## Key takeaways - [00:04](https://www.youtube.com/watch?v=0mbItBdqlko&t=4s) 演讲者 David Tepper 拥有超过 15 年的 AI 领域经验,曾任微软高管管理覆盖 4.5 万 Azure 员工的 GenAI 平台,后创立 Pay-I 专注于量化 AI Agent 的真实商业价值与 ROI。 - [01:28](https://www.youtube.com/watch?v=0mbItBdqlko&t=88s) 行业中 Agent 概念被过度滥用,其技术形态应严谨划分为三种:工作流(Workflow)、管线(Pipeline)与自主智能体(Dynamic Agent)。 - [02:48](https://www.youtube.com/watch?v=0mbItBdqlko&t=168s) 商业价值可归纳为三大核心量化维度:财务价值(直接收益/成本节省)、时间节省(任务缩短)与生产率提升(单位产出增加)。 - [04:21](https://www.youtube.com/watch?v=0mbItBdqlko&t=261s) 大模型成本体系极其复杂,涉及不同的 Tokenizer、多模态不透明计费、SDK/Snapshot 差异以及 Agent 动态多轮调用带来的不确定性,使得计算真实单次任务开销极其困难。 - [07:25](https://www.youtube.com/watch?v=0mbItBdqlko&t=445s) 衡量时间节省的前提是拆解 GenAI 的各种延迟指标,包括首 Token 时间(TTFT)、Token 间延迟、首完成 Token 时间及每秒输出 Token 数(OTPS)。 - [09:27](https://www.youtube.com/watch?v=0mbItBdqlko&t=567s) 推行 Agent 评估的三步法:首先按“用例(Use Cases)与业务成果”拆分,其次建立严密的“版本控制(Versioning)”,最后使用布尔值、数值、百分比及李克特量表这四大标准化 KPI 进行量化。 - [13:19](https://www.youtube.com/watch?v=0mbItBdqlko&t=799s) 提出了时间节省计算模型,通过组合“任务满意度二进制变量”、“人工常规耗时”与“智能体耗时”来精确评估累计节省的时间,揭示虚假繁荣。 - [14:53](https://www.youtube.com/watch?v=0mbItBdqlko&t=893s) 生产力提升的核心黄金法则:只有当智能体“成功完成任务的概率”大于“人工验证耗时除以人工亲自执行耗时”时,Agent 才真正提高了生产力。 - [16:15](https://www.youtube.com/watch?v=0mbItBdqlko&t=975s) Pay-I 作为首个被 FinOps 基金会认定的 GenAI FinOps 解决方案,提供实时支出保护与跨版本数据采集功能,并与 AWS 联合推出了 VALUE 计划帮助企业落地 AI ROI 评估。 - [18:05](https://www.youtube.com/watch?v=0mbItBdqlko&t=1085s) 财务价值、时间节省与生产率三大维度没有固定的优先级,企业应根据具体用例的目标(如开源、节流或提效)灵活组合测量。 ## Detailed walkthrough ### [00:04](https://www.youtube.com/watch?v=0mbItBdqlko&t=4s) 演讲开场与个人背景 演讲者 David Tepper(Pay-I 公司 CEO)拉开了关于“评估 AI Agent 商业价值”的演讲序幕。在因临时更换场地而进行简短的暖场交流后,他概述了本次演讲的四大核心议题:介绍个人背景、定义 AI Agent(智能体)、阐释商业价值,以及如何将两者结合进行量化测量与案例分析。 ``` David Tepper 的职业履历 ──────────────────────────────────────────────────────────── [微软 19 年履历] ──► 负责 Azure 4.5 万员工的 GenAI 内部平台 拒绝微软 CTO 团队的高管要约(全公司 GenAI) 离职创业成立 Pay-I, Inc. ──────────────────────────────────────────────────────────── [行业影响力] ──► 微软内部 GenAI 战略主讲人 向 Fortune 500 企业及美国国会演讲 向联合国(UN)汇报 GenAI 对发展中国家的影响 ``` David Tepper 拥有深厚的生成式 AI(GenAI)技术背景。他自 2009 年起便深耕该领域,早在 2011 年就获得了首项相关专利。在微软任职的 19 年间,他曾领导内部软件平台,服务 Azure 全球 4.5 万名员工使用生成式 AI 技术。为了创立 Pay-I,他甚至拒绝了直接向微软 CTO 汇报、统筹全公司内部 AI 应用程序的要约。在微软期间,他不仅向公司高层规划 GenAI 的战略布局,还代表微软面向《财富》500 强企业、知名高校及美国国会发表演讲;半年前,他还在联合国发表演讲,探讨生成式 AI 对发展中国家的深远影响。他开玩笑说,生成式 AI 曾经是一个极度小众的领域,而如今技术迎来爆发式增长,他选择脱离大企业的束缚,踏上创业之旅,甚至打趣道由于该领域过于复杂,自己“在探索过程中头发都掉光了”。 - 演讲者曾于 2011 年获得个人首项 AI 专利。 - 曾在微软内部管理覆盖 45,000 名 Azure 员工的生成式 AI 平台。 - 约半年前受邀在联合国发表演讲,探讨生成式 AI 对发展中国家的影响。 --- ### [01:28](https://www.youtube.com/watch?v=0mbItBdqlko&t=88s) 定义 AI Agent 的三种技术形态 针对当前行业中 Agent 概念被过度滥用(Overloaded)的现象,David Tepper 幽默地调侃道:“如今似乎连讲台都能被称为 Agent,甚至台下的观众也人人都是 Agent。”为了建立严谨的分析框架,他将广义上的 Agent 划分为三种截然不同的技术架构形态: ``` +-----------------------------------------------------------------+ | Agent 的三种技术架构形态 | +-----------------------------------------------------------------+ | 1. Workflow (工作流): 传统代码调用 LLM,非结构化转结构化 | | 2. Pipeline (流水线): 预设固定步骤,如 RAG 聊天机器人/数据处理| | 3. Dynamic Agent: 运行时自主决策,多工具调用,执行路径不确定| +-----------------------------------------------------------------+ ``` 1. **Workflow(工作流):** 本质上是传统的软件程序,仅在某些节点将大语言模型(LLM)作为组件调用。最常见的应用场景是将非结构化数据转化为结构化数据,以便后续的传统代码进行处理。 2. **Pipeline(流水线):** 由预先设定好的步骤组成,LLM 参与其中。典型的例子是具备检索增强生成(RAG)功能的聊天机器人或数据处理流水线。这种形态不具备真正的“自主性”,因为它并不自行做出决策,而是始终严格执行预设的流程。 3. **Dynamic Agent(动态智能体):** 这才是真正具备 Agentic 特性的系统。它能够在运行时(Runtime)根据实时情况自主做出决策、灵活调用多种工具,且执行的时间长短和路径是非确定性(Non-deterministic)的。 David Tepper 强调,无论企业目前采用的是上述哪一种技术形态,本次演讲所介绍的商业价值测量技术与评估方法都是完全通用且适用的。 - **Workflow:** 核心功能在于非结构化数据向结构化数据的转化。 - **Pipeline:** 典型应用包括 RAG 聊天机器人与自动化数据处理流。 - **Dynamic Agent:** 核心特征在于运行时自主决策与非确定性执行路径。 --- ### [02:48](https://www.youtube.com/watch?v=0mbItBdqlko&t=168s) 商业价值的三大可量化维度 在厘清 Agent 的定义后,话题转向了“商业价值”的界定。David Tepper 指出,商业价值在不同行业中的感知截然不同——例如养老护理行业的价值可能体### 演讲背景与演讲者背景介绍 Pay-I, Inc. 的 CEO David Tepper 在本次演讲中聚焦于“衡量 AI Agent 的商业价值”。随着各大企业跨越概念验证(POC)阶段并进入生产部署,如何量化 AI 智能体(Agent)的真实投资回报率(ROI)、单次任务成本以及对业务增长的贡献,已成为企业高层和董事会最关注的焦点问题。 演讲者 David Tepper 拥有极其深厚的 AI 行业履历: - 他自 2009 年起便深耕生成式 AI(GenAI)领域,并在 2011 年获得了个人首项相关专利。 - 在微软(Microsoft)效力约 19 年间,他领导构建了 Azure 内部供 45,000 名员工使用 GenAI 的软件平台,并曾多次代表微软向 Fortune 500 强企业高管、顶尖大学、美国国会乃至联合国(讲解 GenAI 对发展中国家的影响)进行专题汇报。 - 为创立 Pay-I,他放弃了直接向微软 CTO 汇报并统领微软全公司内部 AI 工作的机会。他调侃道,AI 领域极其复杂,自己在这个过程中“甚至脱光了头发”。 --- - **演讲者姓名与职位**:David Tepper,Pay-I, Inc. CEO。 - **履历要点**:微软 19 年老兵;曾管理 Azure 内部 4.5 万人的 GenAI 平台;放弃向微软 CTO 汇报的职位出来创业。 - **演讲核心目标**:帮助企业量化 Agent 系统的真实价值、对齐衡量指标并在上线前预测毛利率。 --- ### [01:28](https://www.youtube.com/watch?v=0mbItBdqlko&t=88s) 重新定义 AI Agent:三大演进形态 “Agent”在当下的科技界已成为一个高度泛化的词汇(Overloaded term),几乎所有软件项目都在争相贴上“Agentic”的标签。为了建立清晰的讨论基准,David 将广义上的 Agent 明确划分为以下三种形态,并强调本演讲所讲的价值衡量方法论全面适用于这三种类型: ``` +------------------------------------------------------------------+ | 三种 Agent 技术形态 | +------------------------------------------------------------------+ | 1. 工作流 (Workflow) | | 传统软件 + LLM 节点 -> 用于非结构化数据转结构化数据 | +------------------------------------------------------------------+ | 2. 管线 (Pipeline) | | 预设多步流程 (如 RAG / 问答 Bot) -> 路径确定,无自主决策 | +------------------------------------------------------------------+ | 3. 智能体 (Agentic) | | 运行时自主决策 + 工具调用 -> 执行路径与耗时非确定性 (Non-deterministic) | +------------------------------------------------------------------+ ``` 1. **工作流(Workflow)**:传统软件在特定环节调用大语言模型(LLM),最常见的应用场景是将非结构化数据转化为结构化数据,以便后续的代码进行处理。 2. **管线(Pipeline)**:预设固定的多步骤流程,LLM 作为其中的一部分。例如典型的 RAG(检索增强生成)聊天机器人。这类系统不具备自主决策能力,每次执行都遵循严格相同的步骤。 3. **自主智能体(Agentic)**:在运行时根据目标自主决策并动态选择调用不同的工具,其执行路径和耗时长短均具有非确定性(Non-deterministic)。 --- ### [02:48](https://www.youtube.com/watch?v=0mbItBdqlko&t=168s) 商业价值的三大可量化维度 不同行业对“商业价值”的感知可能大相径庭(如养老服务可能看重情感抚慰),但要建立通用的通用衡量体系,就必须聚焦于可量化的指标。Pay-I 在访谈了数百家 Enterprise 企业(多来自 CIO 办公室)后,总结出商业价值的三大核心归宿: - **财务价值(Financial Value)**:以直接货币计量的收益,包括开辟新营收或实现成本节省(如自动化替代人工)。 - **时间节省(Time Savings)**:将原本需要数分钟或数小时的任务缩短至几秒钟,显著提升执行效率。 - **生产率提升(Productivity)**:单位时间内的产出数量(例如:客服人员或系统每天能够处理的工单数量)。 --- ### [04:21](https://www.youtube.com/watch?v=0mbItBdqlko&t=261s) 计算 ROI 的巨坑:极其复杂的 LLM 成本体系 计算 ROI 的基本公式极度简单($\text{ROI} = \frac{\text{净利润}}{\text{成本}}$),但对于 AI Agent 而言,理解与计算“成本”却极其复杂,远非简单地将两款售价相同的模型视为同等成本。 ``` +------------------------------------------------------------------+ | LLM / Agent 成本的不确定性 | +------------------------------------------------------------------+ | [不同 Tokenizer] ----> 相同文本生成不同数量 Token | | [多模态输入 (Vision)] -> 隐藏计算公式 (不透明的 Token 扣减) | | [不同 SDK / Snapshot] -> 同一模型调用价格存在差异 | | [Agent 动态多轮调用] -> 每次推理耗时、Tool 调用与 Token 消耗完全不同 | +------------------------------------------------------------------+ ``` 大模型成本核算之所以极其棘手,主要原因包括: 1. **Tokenizer 与计费模式差异**:不同模型使用的 Tokenizer 不同,甚至有些模型不按 Token 计费。将系统从 Model A 切换到 Model B 时,无法事先简单预判成本是上升还是下降。 2. **多模态(Vision)计费不透明**:以 OpenAI GPT-4o 的视觉计费为例,其后台图像转换公式经常在不通知用户的情况下变更,且返回的账单仅显示“消耗了 X 个 Token”,完全不拆分图像与文本的占比。 3. **SDK 与版本快照差异**:同一个模型通过不同的 SDK 调用,价格可能完全不同;不同的 Snapshot 版本计费也有差异。 4. **预留实例(Reserved Instances)复杂度**:在 OpenAI 购买预留实例后,要计算单次 API 调用的真实成本,甚至需要建立多维数学模型。David 调侃道,OpenAI 官网上曾写着“将在 2023 年 Q2 推出成本追踪系统”,结果严重拖延至今。 5. **Agent 单次请求 vs 整体工作流**:单次模型调用的成本不等于 Agent 的成本。一个 Agent 在完成一项任务时会发起一系列连续调用,由于推理深度、工具调用次数和 Token 量的波动,每一次任务的实际开销都不尽相同。 - **案例/名言**:David 提到他有一份长达 80 页的 PPT,专门用来讲解如何计算 GPT-4o 的真实调用成本;他同时提醒听众可以私下来找他听关于 Grok 利用 Token 计算成本的奇葩趣味故事。 --- ### [07:25](https://www.youtube.com/watch?v=0mbItBdqlko&t=445s) GenAI 延迟的多种形态及其对时间节省的影响 衡量“时间节省”的前提是准确度量 Agent 系统的延迟。与传统软件单一的“端到端延迟(End-to-End Latency)”不同,GenAI 引入了多种必须拆解评估的延迟指标: - **首 Token 时间(Time to First Token, TTFT)**:在流式输出(Streaming)系统中,用户多久能看到 Agent 开始响应。这直接决定了交互的“敏捷感”。 - **Token 间延迟(Inter-token Latency)**:流式输出过程中每个 Token 之间的间隔,通常比 TTFT 快得多。 - **首完成 Token 时间(Time to First Completion Token)**:在推理模型(Reasoning Models)中,模型在开始给出最终解答前“思考”了多久。 - **每秒输出 Token 数(Output Tokens Per Second, OTPS)**:模型生成文本的速度。特别需要注意的是,某些大上下文模型(如拥有百万 Context Window 的 Google Gemini)随着输入上下文的增长,后续生成的 Token 处理速度会急剧下降并出现卡顿。 > **关于推理模型的有趣事实**:很多数据表明,模型在返回第一个 Token 之前,实际上就已经在内部算出了答案,后续的流式输出只是在向用户进行解释。就如同求解一个多入口多出口的迷宫,模型必须先在内部把整条最佳路径跑通(Solve the maze),然后才能开始输出第一步。 --- ### [09:27](https://www.youtube.com/watch?v=0mbItBdqlko&t=567s) 落地三步法:用 KPI 和版本控制破解测量难题 针对 Agent 准确率不一、可能需要人工复核(Human-in-the-loop)而影响实际生产率的难题,David 提出了将复杂问题拆解测量的标准流程: #### 第一步:按用例(Use Cases)拆分 Agent 不要关注底层用了什么技术(是 Workflow 还是 True Agent),只关注**最终交付的业务成果(Outcome)**(如:文档摘要、预订行程、处理工单等)。从成果视角评估:完成该用例花了多久?花了多少钱?成功质量如何? #### 第二步:引入严密的版本控制(Versioning) GenAI 系统极其敏感——修改提示词(Prompt)、更换模型、更新知识库数据或后端代码,都会彻底改变 Agent 的性能 profile。甚至工程师在提示词里多加一个逗号,都可能直接改变产品的毛利率、推理耗时和回答准确率。因此,**所有 KPI 必须与 Agent 的特定版本绑定**。 #### 第三步:选择恰当的量化 KPI 常见且高频的企业级用例(如客户服务、文档抽取等)对应的 KPI 最终可以被归纳为四大标准化数据类型: | 动态数据类型 | 说明与典型示例 | |:--- |:--- | | **布尔值 (Boolean)** | 是/否,True/False(如:任务是否一次性成功成功执行) | | **数值 (Numbers)** | 标量数值(如:处理该任务消耗的平均成本、总耗时秒数) | | **百分比 (Percentages)** | 比例或率(如:人工接管率、准确率、自动化覆盖率) | | **李克特量表 (Likert Scale)** | 分级打分(如:1-5 星级满意度、1-10 分用户评价) | David 指出,这四种类型几乎涵盖了目前生产环境中 99% 以上的 KPI 量化评估需求。将收集到的 Agent 运行数据与这些 KPI 结合,企业才能真正告别“感觉良好”的盲目时代,精准评估出 AI Agent 的真实商业价值。 --- - **用例与工具的区分**:Copilot 或 Agent 本身只是“工具(Tools)”,并非“用例(Use Cases)”,因为工具不直接等于业务结果。 - **KPI 的本质**:“ROI 本身不是 KPI,KPI 是用来计算出 ROI 的数据过程”;“满意感觉(Feeling)不是 KPI”。<FollowUp label="想了解演讲的后半部分内容(如具体的 Pay-I 客户案例或 Margin 预测方法)吗?" query="请继续提供该视频后半部分的详细讲解与逐段总结。"/> ### [13:19](https://www.youtube.com/watch?v=0mbItBdqlko&t=799s) 评估 AI 智能体的财务价值与时间节省模型 David Tepper 首先将分析带回基础的投资回报率(ROI)计算公式。他指出,计算公式的分子通常代表收益(如成本节约、增加营收,甚至是裁员带来的开支减少),而分母则对应投入成本。通过将具体的业务逻辑转化为对关键绩效指标(KPIs,如点击率、转化率等)的量化测量,企业能够准确填入这些数据并算出明确的利润与 ROI。 接着,Tepper 引入了一个更为精确的“时间节省计算公式”。该公式看似复杂,其实结构清晰: ``` ┌────────────────────────────────────────────────────────┐ │ 时间节省总量 = Σ [ s × (n - a) ] │ ├────────────────────────────────────────────────────────┤ │ 其中: │ │ • s: 任务是否满意的指示变量 (1 代表成功,0 代表失败) │ │ • n: 人工完成该任务所需的常规时间 │ │ • a: 智能体完成该任务所需的时间 │ │ • Σ: 针对所有处理任务(如所有文档)的求和 │ └────────────────────────────────────────────────────────┘ ``` 这个简单公式能非常敏锐地揭示智能体的真实表现:它不仅能识别出那些表现出色、节省大量时间的智能体,也能暴露完全失败并白白浪费时间的智能体。更重要的是,它能揭示出最隐蔽的“虚假繁荣”情况——即智能体虽然成功完成了任务,但实际节省的时间远低于预期。以文档摘要为例,通过评估输出的质量评级(决定 $s$ 值)以及文档的规模大小(决定 $n$ 与 $a$),企业便能跨所有文档精准计算出累计节省的时间。 - **ROI 分子构建**:包含节约成本、新增收入或人员结构调整带来的直接财务收益。 - **时间节省模型参数**:$s$ 为满意度二进制变量(0 或 1),$n$ 为人工基线耗时,$a$ 为智能体实际耗时。 ### [14:53](https://www.youtube.com/watch?v=0mbItBdqlko&t=893s) 生产力提升的核心法则与智能体分类总结 在谈及更为复杂的生产力评估时,由于演讲时间所限,Tepper 决定跳过繁琐推导,直接给出关于生产力最核心的“金句法则”(sound bite)。他强调,如果大家今天只能记住关于生产力的一件事,那就是这个核心不等式:**只有当 AI 智能体“成功完成任务的概率”,大于“人类验证该任务所需的时间除以人类亲自做该任务所需的时间”时,智能体才真正提高了用户的生产力。** 他将这一核心结论反复重述,强调验证成本在衡量 AI 价值时的决定性作用。随后,Tepper 对整场演讲的核心框架进行了系统性梳理,重新提炼了三大要点: 1. **智能体的三级形态**: - **基础工作流**:传统软件直接调用 LLM。 - **预设管道**:按照预先确定的步骤运行,并在特定节点嵌入 LLM。 - **自主智能体(Agentic)**:具备自主决策能力,能够主动使用 MCP(Model Context Protocol)服务器及各类外部工具。 2. **业务价值的三大维度**:包括价值创造(财务收益)、时间节省与生产力提升。 3. **量化评估方法论**:将智能体拆解为由成果驱动的用例,对成果进行精确测量,并运用量化 KPI 综合计算最终的业务价值。 - **生产力黄金法则**:$P(\text{成功}) > \frac{T_{\text{验证}}}{T_{\text{执行}}}$(智能体成功概率必须超过“验证耗时/亲自执行耗时”的比例)。 - **工具与协议支持**:自主智能体依靠 MCP 服务器和外部工具链实现复杂决策。 ### [16:15](https://www.youtube.com/watch?v=0mbItBdqlko&t=975s) Pay-I 平台架构、FinOps 治理与 AWS 官方合作 Tepper 指出,理论公式只是第一步,实际落地需要一套能够自动化收集数据、跨不同版本汇总分析的完整系统,同时还需考虑失败率管理和容量规划等日常部署中的现实挑战。这正是 Pay-I 公司的核心价值所在。Pay-I 最近被《福布斯》(Forbes)报道为管理生成式 AI ROI 的首选平台,并被 FinOps 基金会认定为目前唯一的生成式 AI FinOps 解决方案。公司不仅研发工具,还在开创一套完整的行业标准实践。 除了评估业务价值外,Pay-I 平台还提供财务治理和实时支出保护功能,防止智能体出现“发疯”(go bananas)导致成本异常飙升的情况。Tepper 在现场宣布了一项重大合作:**Pay-I 已与 AWS 达成官方战略合作**,共同推出名为“VALUE”的专项计划(其简称类似《复仇者联盟》中的神盾局 S.H.I.E.L.D.)。通过该计划,AWS 团队将全程协助企业进行生成式 AI 的价值实现,涵盖用例定义、KPI 识别、利用 Pay-I 筛选具备真实 ROI 的项目以及后续优化。这能帮助企业在 LinkedIn 上发布智能体成果时,摆脱虚浮的炒作,具备足够的底气与信心。 - **行业认可**:获得 Forbes 报道以及 FinOps Foundation 唯一认证的 GenAI FinOps 解决方案。 - **核心功能**:实时支出保护(防止成本暴涨)、跨版本数据采集、失败率与容量规划。 - **AWS VALUE 计划**:联合 AWS 团队提供从用例拆解、KPI 确定到利用 Pay-I 验证 ROI 的全流程服务。 - **展位信息**:Pay-I 在大会现场设置了第 246 号展位(Booth 246)提供演示。 ### [18:05](https://www.youtube.com/watch?v=0mbItBdqlko&t=1085s) 问答环节:聚焦 ROI 与三大价值维度的优先级选择 在演讲最后的问答环节,Tepper 回答了现场观众提出的两个核心问题: 1. **关于定位(专注于 ROI 还是部署)**:Tepper 明确表示 Pay-I 的核心焦点是 **ROI**。市面上所有人都在推销或构建各种智能体,但 Pay-I 关注的是哪些智能体能够为企业带来切实的业务回报。 2. **价值维度的优先级选择**:观众问及在财务价值、时间节省和生产力这三种价值衡量方式之间应如何排序。Tepper 解释称,这完全取决于具体的用例和企业的目标——如果是为了直接赚钱,就优先看财务价值;如果是为了提升效率,则侧重时间节省。但在实际操作中,企业不仅可以而且**应该同时测量这三个维度**。当客户询问“使用 ChatGPT 到底能为公司带来什么价值”时,关键在于企业自身如何定义“价值”。这极其依赖具体业务场景,并不存在一劳永逸的“万灵药”(silver bullet)。 - **业务焦点**:不提供智能体构建,专注衡量智能体的真实业务回报(ROI)。 - **优先级原则**:无通用套路,需根据用例目标(开源、节流或提效)灵活组合测量财务、时间与生产力三大指标。 ## Notable quotes > [01:28](https://www.youtube.com/watch?v=0mbItBdqlko&t=88s) 如今似乎连讲台都能被称为 Agent,甚至台下的观众也人人都是 Agent。 — David Tepper > [00:04](https://www.youtube.com/watch?v=0mbItBdqlko&t=4s) 我在探索过程中头发都掉光了。 — David Tepper > [09:27](https://www.youtube.com/watch?v=0mbItBdqlko&t=567s) ROI 本身不是 KPI,KPI 是用来计算出 ROI 的数据过程。 — David Tepper > [09:27](https://www.youtube.com/watch?v=0mbItBdqlko&t=567s) 满意感觉(Feeling)不是 KPI。 — David Tepper > [14:53](https://www.youtube.com/watch?v=0mbItBdqlko&t=893s) 只有当 AI 智能体‘成功完成任务的概率’,大于‘人类验证该任务所需的时间除以人类亲自做该任务所需的时间’时,智能体才真正提高了用户的生产力。 — David Tepper > [04:21](https://www.youtube.com/watch?v=0mbItBdqlko&t=261s) 将系统从 Model A 切换到 Model B 时,无法事先简单预判成本是上升还是下降。 — David Tepper > [09:27](https://www.youtube.com/watch?v=0mbItBdqlko&t=567s) 修改提示词(Prompt)、更换模型、更新知识库数据或后端代码,都会彻底改变 Agent 的性能 profile。 — David Tepper > [09:27](https://www.youtube.com/watch?v=0mbItBdqlko&t=567s) 甚至工程师在提示词里多加一个逗号,都可能直接改变产品的毛利率、推理耗时和回答准确率。 — David Tepper > [16:15](https://www.youtube.com/watch?v=0mbItBdqlko&t=975s) 防止智能体出现‘发疯’(go bananas)导致成本异常飙升的情况。 — David Tepper > [18:05](https://www.youtube.com/watch?v=0mbItBdqlko&t=1085s) 当客户询问‘使用 ChatGPT 到底能为公司带来什么价值’时,关键在于企业自身如何定义‘价值’。 — David Tepper ## Mentioned David Tepper, Pay-I, Inc., Microsoft, Azure, United Nations, Fortune 500, OpenAI, GPT-4o, Grok, Google Gemini, AWS, VALUE Initiative, FinOps Foundation, Forbes, Model Context Protocol (MCP), ChatGPT, RAG ## Video description David Tepper, CEO, Pay-I, Inc. Abstracts The new world of software promises to be teeming with agents and AI powered tools… but how will that impact your business? You've built the POCs and seen the demos, but scaling to production demands answers: What is the real cost per outcome? Which use cases deliver provable ROI? As enterprises race to deploy AI agents and chatbots, boards are demanding quantifiable business value metrics. The hype era of “trust me, it’s working” is ending, and in its place you’ll need the tools to manage the workforce of tomorrow. I’ve been working on AI solutions for the largest companies in the world since 2009, and in this talk I’ll show you how Pay-i is helping companies quantify the real value of every agentic workflow, align teams around the new metrics that drive business growth, and forecast margins before shipping. [Source](https://www.youtube.com/watch?v=0mbItBdqlko)

Source quotes

[1:28] David Tepper: 如今似乎连讲台都能被称为 Agent,甚至台下的观众也人人都是 Agent。 [0:04] David Tepper: 我在探索过程中头发都掉光了。 [9:27] David Tepper: ROI 本身不是 KPI,KPI 是用来计算出 ROI 的数据过程。 [9:27] David Tepper: 满意感觉(Feeling)不是 KPI。 [14:53] David Tepper: 只有当 AI 智能体‘成功完成任务的概率’,大于‘人类验证该任务所需的时间除以人类亲自做该任务所需的时间’时,智能体才真正提高了用户的生产力。 [4:21] David Tepper: 将系统从 Model A 切换到 Model B 时,无法事先简单预判成本是上升还是下降。 [9:27] David Tepper: 修改提示词(Prompt)、更换模型、更新知识库数据或后端代码,都会彻底改变 Agent 的性能 profile。 [9:27] David Tepper: 甚至工程师在提示词里多加一个逗号,都可能直接改变产品的毛利率、推理耗时和回答准确率。 [16:15] David Tepper: 防止智能体出现‘发疯’(go bananas)导致成本异常飙升的情况。 [18:05] David Tepper: 当客户询问‘使用 ChatGPT 到底能为公司带来什么价值’时,关键在于企业自身如何定义‘价值’。

Source quotes

David Tepper: The hype era of “trust me, it’s working” is ending, and in its place you’ll need the tools to manage the workforce of tomorrow. [1:28] David Tepper: everything seems to be agentic. [2:48] David Tepper: warm and fuzzy feeling [2:48] David Tepper: #AI is taking all of our jobs [4:35] David Tepper: Q2 of 2023

Original transcript

[0:04] Uh, welcome to measuring the business [0:05] value of agents. Hopefully you're in the [0:07] right spot. Uh, they did a rim a quick [0:08] room change on us here. So glad you all [0:10] found it. Uh, quick agenda. I'll go over [0:12] a little bit about me and why I'm [0:13] talking to you guys today. Obviously, [0:15] we'll discuss what is an agent, what is [0:17] business value, how do you actually go [0:18] about measuring those two in combination [0:20] with one another? You know, the the [0:21] point of the talk and then go over a few [0:23] examples. Uh, first off, who the heck is [0:25] this guy um uh standing in front of you [0:27] today? So um uh I have been working on [0:30] Genai uh since 2009. My first patent is [0:33] back in 2011. I worked at Microsoft for [0:35] about 19 years. I ran the internal [0:37] software platform for how Azure's 45,000 [0:39] employees leverage Genai internal to the [0:42] company. And then in order to pursue [0:43] payi, I turned down the role to report [0:45] to the CTO of Microsoft and run internal [0:47] use across all of Microsoft. While I was [0:49] there, I was a guy that was giving a lot [0:50] of the talks to the executives at [0:51] Microsoft about how they should be [0:53] thinking about Genai, how it's going to [0:54] impact their products and services. I [0:56] represented Microsoft in speaking about [0:57] Jenny to other Fortune 500 companies, [0:59] universities and to the US Congress and [1:01] about six months ago I spoke in front of [1:03] the United Nations about how it's going [1:04] to impact developing nation states and [1:06] so this has been my area of expertise [1:08] for pretty much my whole career and it [1:10] was this really niche little thing that [1:11] a lot of people weren't paying attention [1:12] to and then I don't know if you paid [1:13] attention recently of of course you all [1:14] have uh it has exploded and with that I [1:17] found myself bound by the shackles of [1:18] corporate America and decided to set [1:20] forth uh out on my own adventure here. [1:22] Um, this is a picture of me before I [1:24] went down this journey, by the way, [1:25] because this area is really complicated. [1:26] I lost all my hair in the process. So, [1:28] first off, what the heck is an agent, [1:29] right? Um, these days, uh, the answer to [1:32] that seems to be everything. Um, uh, [1:34] this this podium is agentic. I'm an [1:36] agent right now. Raise your hand if you [1:38] guys are an agent out in the audience. I [1:39] see a few agents out there. Yes, thank [1:41] you all for being agents. I appreciate [1:42] that. Uh, it's a super overloaded term [1:44] because everybody wants to be a cool kid [1:45] and say everything they're building is [1:46] an agent. So, let's actually start to [1:48] define some of that. Um, I'm going to [1:50] define an agent as one of three things. [1:52] The first of those is simply a normal [1:54] piece of software that calls an LLM as [1:57] part of its workflow. And uh, this is [1:59] most commonly used when you're [2:01] converting unstructured data into [2:02] structured data so that you can have [2:04] like traditional code then operate on [2:05] that. We call that one a workflow. The [2:07] second one is a pipeline. This is a sort [2:09] of a predetermined set of steps of which [2:12] LLMs are a part of that. A chatbased [2:14] like a chatbot is a really common use [2:16] case for this. It has like rag, right? [2:18] It's not agentic in the sense that it's [2:20] not making decisions for itself. It's [2:22] pretty much always following the exact [2:23] same set of steps. That's just a [2:25] pipeline. If you're processing data, [2:26] right? That's another example. And then [2:28] the third one is something that's [2:29] actually agentic. This is making its [2:31] decisions at runtime. This is using a [2:33] variety of different tools, right? It [2:35] has a non-deterministic execution time [2:36] length, things like that. Right? Now, [2:38] the good news is that in this talk, I'm [2:40] actually going to cover all three of [2:41] these. So, the techniques that we're [2:43] talking about for measuring business [2:44] value apply regardless of your [2:46] definition of an agent. Uh so that now [2:48] requires us to have a common definition [2:50] of what we talk about when we talk about [2:52] business value. And so um uh business [2:54] value is a thing that feels a little [2:56] different depending on which business [2:58] you're in. For example, if you take care [2:59] of the elderly, the value could be like [3:01] the warm and fuzzy feeling that you get [3:03] uh when you're helping people along with [3:04] their day. But uh in order for us to be [3:06] able to speak more generally about [3:07] business value, we have to talk about [3:08] the things that are quantifiable and [3:10] therefore they are sort of uh uh [3:13] universal across a variety of different [3:14] businesses. And so that you know uh we [3:17] can give you some examples at PI we've [3:18] talked to literally hundreds of [3:20] enterprises to understand what business [3:21] value means to them. And it turns out [3:23] that this also leads to three buckets of [3:26] quantifiable business value. And by all [3:28] means if after you see these three you [3:29] think of one that we haven't touched on [3:31] here I'd love to hear about it. The [3:33] first one is really just financial value [3:35] or or uh uh value generated in raw [3:37] currency. This is uh different sources [3:40] right? you're either making new revenue, [3:42] um you're you're saving a bunch, hash AI [3:45] is taking all of our jobs, uh as an [3:47] example, right? The second one is time [3:49] savings. So, I used to take x number of [3:51] uh minutes to do something, right? And [3:53] now I can do it substantially more [3:55] quickly as a result of leveraging all [3:56] these agents. And the third one is [3:58] productivity. So, uh productivity has [4:01] two units. It's the thing that I'm doing [4:03] and then the unit of time it takes me to [4:05] do it. So the easiest example, the [4:07] generic one we're using here is I'm [4:08] processing tickets and how many tickets [4:10] can I process in a given day. And so [4:12] these are the three ways that we've [4:14] discovered the companies are really [4:15] thinking about business value and this [4:16] is going from typically the office of [4:18] the CIO. And so you know what makes [4:21] measurement of this really difficult for [4:23] agents um and in some way agents are [4:25] like employees. How do you think about [4:27] the value of a human employee right? How [4:30] would you how would you measure that? Uh [4:31] well so for each of the three types of [4:33] business value let's go through some [4:34] examples. This is the world's most basic [4:36] ROI equation. Uh we talked earlier about [4:38] how net profit in this case can come [4:40] from pure savings, you know, and a bunch [4:41] of other things. Uh but the other part [4:43] of this equation is cost. And so that [4:45] means you have to understand the cost of [4:46] an agent. And unfortunately, this can be [4:48] remarkably complex. So, uh normally if [4:50] you look at two products and if you [4:52] think they have the exact same price and [4:54] you do the exact same things with them, [4:57] well then the cost must be identical. [4:58] But that is not the case with large [5:00] language models. uh different LLM use [5:02] different tokenizers and not all of them [5:04] even charge you based on tokens. And so [5:05] if you're going to make a switch from [5:07] one model over to another, you actually [5:09] don't know a priori whether or not that [5:11] switch is going to end up being more or [5:12] less expensive. Uh catch me afterward. [5:14] I'll tell you a really fun story about [5:16] gro uh and how it uses tokens to crazy [5:18] effect on this. Um you also are charged [5:20] for things apart from text. So models [5:22] these days are multimodal which means [5:24] that they've accept a variety of [5:26] different types of inputs. They have a [5:27] whole bunch of different types of [5:28] outputs. This is uh the image the the [5:31] vision pricing algorithm for GPT40 from [5:34] just the most recent snapshot of GPT40. [5:36] Earlier snapshots have a totally [5:38] different equation. Those numbers that [5:39] are marked variable, they change those [5:41] all the time. They don't even tell you. [5:42] Those are hidden uh calculations that go [5:44] on in the back end is basically how they [5:45] charge you for sending images into a [5:47] model independent from the token price [5:49] of that model. And when you get the [5:50] responses back, it says, "Oh, you used a [5:52] thousand tokens." They don't tell you [5:53] what percentage of that is images. So [5:54] you have no idea when you're getting [5:56] charged for vision systems. Um, and in [5:58] fact, uh, understanding costs for an LLM [6:00] in general are it's like remarkably [6:02] complicated. Uh, I have an 80 slide deck [6:04] that goes over how to calculate the cost [6:06] for just GPT40. Uh, there's some fun [6:08] things like the exact same model will [6:10] have a different price if called from a [6:12] different SDK, right? That should never [6:14] be the case. Obviously, different models [6:16] will have different snapshots. Those [6:17] snapshots have different pricing. Um, [6:19] the there's some fun stuff here. the [6:20] reserved instances. If you uh buy [6:22] reserved instances from OpenAI, the only [6:24] way to understand how much an individual [6:26] API call costs is to do [6:28] multi-dimensional modeling. And it's so [6:30] complicated that this leads to my [6:32] favorite quote on OpenAI's website, [6:33] which is they're going to release a [6:34] tracking system for this in Q2 of 2023. [6:37] Uh which is clearly a little bit late uh [6:39] on them releasing that. So, this is this [6:42] is a whole lot of fun, especially if [6:43] you're trying to leverage a bunch of [6:44] different models from a bunch of [6:46] different sources. You get a bill. Yeah, [6:47] they'll sell they'll tell you this is [6:48] how much you owe. But if you're trying [6:49] to understand why the bill is the way [6:51] that it is, that's very challenging. It [6:53] takes a lot of effort on that. Also, the [6:55] request the cost of an individual [6:56] request like I'm making one call to a [6:58] model is not the cost of your agent. [7:01] Your agent is going to have a host of [7:02] different requests that it's making, [7:04] right? Especially if it's a pipeline or [7:05] if it's actually truly agentic. It's [7:07] going to be doing a bunch of different [7:08] things. And every single time it makes a [7:10] call, that cost is going to be [7:12] different, right? Whenever you use [7:13] Genai, it's a different amount of [7:14] reasoning effort, different number of [7:16] tokens, different tools that are being [7:17] used, right? And that's for every single [7:19] request. And so understanding that in [7:21] the context of your entire agent is [7:22] really the cost, right? And so uh the [7:25] second one of these is time savings, [7:27] right? So to calculate time savings, you [7:29] have to understand uh how much time the [7:31] agent is taking itself. And this leads [7:33] you to the topic of Genai latency, which [7:35] is actually a pretty cool one because [7:37] we're all used to this one here, which [7:38] is endto-end latency. How long does [7:40] something take from start to finish? But [7:42] Genaii introduces a bunch of additional [7:44] types of latencies that are also really [7:46] important. The first of these is time to [7:48] first token. So if you're building a [7:50] streaming system or a system that's [7:51] leveraging a streaming model, the time [7:52] to first token is going to be the [7:54] snappiness of that. When do your user [7:56] they actually start to experience your [7:58] agent start to do things because it'll [7:59] start to stream. They'll actually start [8:00] to see the results of that. Now really [8:02] interesting factoid kind of unrelated to [8:04] the talk. time to first token is really [8:05] interesting with Genai because there's a [8:07] lot of data that shows that the models [8:09] will actually answer your question be [8:11] like before the first token ever comes [8:13] back and then they spend all the time [8:15] streaming out the results in order to [8:16] explain it to you and a way to convince [8:18] yourself of that imagine you have a maze [8:19] and the maze has multiple entrances and [8:21] exits and you ask it hey I want the [8:23] fastest route through this maze the only [8:25] way for it to become uh to accurately [8:27] give you that answer is it has to solve [8:28] the maze before it starts giving you the [8:30] path out right and so that's where [8:32] reasoning models come in is because it [8:34] gives them time to work on things before [8:35] they start to give you the result. Um [8:37] and and and as you're streaming all that [8:39] information, you have inter token [8:40] latency. Inter token latency is [8:41] traditionally much uh faster than the [8:44] time to first token. And that's the [8:45] reason why because it's not actually [8:46] resolving uh in that process. And then [8:49] you have time to first completion token. [8:51] If you are reasoning, how long does it [8:52] think for before it starts to give you [8:54] the completion? In other words, the [8:56] thing that you're trying to resolve. And [8:57] then last but not least, you have output [8:58] tokens per second or OTPS. This one is [9:01] only interesting because different [9:03] models will slow down the more tokens [9:05] you put into them. And that means that [9:06] if they're streaming large outputs, [9:08] Gemini is famous for this, right? It has [9:09] like a million token context window. [9:11] That thing will start to chug like crazy [9:13] once you start to get near the end of [9:15] that. Uh because it's processing so much [9:17] to output every cont, you know, [9:18] continuous uh token that comes [9:20] afterward. And so genai latency is by [9:22] itself a very complicated topic. And so [9:24] understanding time savings, you have to [9:25] account for these things as well. And [9:27] then last but not least, we have [9:28] productivity. Uh let's imagine a human [9:31] can process 500 tickets per day. Uh and [9:33] an agent can theoretically process a [9:35] thousand tickets per day. Uh but some [9:37] variable percentage of these tickets [9:39] that the agent produces are wrong and [9:41] the human has to verify and check. So [9:43] what is the impact to productivity when [9:46] humans have to review all of the work [9:48] that these agents are doing, right? Um [9:51] so in order to tackle this, you have to [9:53] subdivide the problem because it's those [9:55] are all pretty big. First step is to [9:57] split out your agent into use cases. It [10:00] doesn't matter if it's a workflow, a [10:02] pipeline, something that's truly [10:03] agentic. What matters isn't the [10:04] technology, it's the outcomes. And so, [10:07] uh, you can think of accomplishing a use [10:08] case as the objective of your agent. [10:11] Whether you're summarizing a document, [10:12] you know, you're you're booking a [10:14] vacation, things like that. Um, [10:15] understanding use cases again is [10:17] thinking about it from this outcome. How [10:18] long did it take to achieve that [10:19] outcome? How expensive was it? How [10:22] successful was it? And that is not [10:23] binary success or failure, right? there [10:26] there's gradients in there. Um but the [10:27] thing is that Genai is remarkably [10:29] sensitive. If you change the prompts [10:31] which are just like English sentences or [10:34] human language sentences, uh it [10:35] completely changes all of the dynamics [10:37] of your agent, right? If you change the [10:39] models, if you change the data, you [10:40] change the code behind it. And so this [10:42] all also has to be versioned um in order [10:45] to uh to understand the deltas between [10:47] hey you know we we're shipping an agent [10:49] and and and we switch to this new model [10:51] we switch to different data or you have [10:52] an engineer that's just playing around [10:53] with things they put a comma in your [10:55] prompt it can completely change the [10:57] gross margin profile of your product [10:58] totally change how long your agent is [11:00] reasoning before it replies and the [11:02] accuracy of the results that you get out [11:04] from there. These are some example use [11:06] cases. You'll notice if you look through [11:07] it, um we we we've spoken with hundreds [11:10] of enterprises, by the way. And these [11:11] are these are far and away the most [11:12] common today. I think these will evolve [11:14] over time. Uh I'm not listing co-pilots [11:17] here. I'm not also listing the word [11:18] agents because those are tools. Tools [11:21] are not uh inherently outcome driven. [11:22] They're not scenarios. And so every [11:24] single one of these, you can convince [11:25] yourself that there is actually a [11:27] tangible outcome that can come out from [11:28] doing that. And so for each of those [11:30] tangible outcomes, you then want to [11:32] track quantifiable KPIs. Uh a KPI is uh [11:37] ROI by itself is not a KPI. KPIs are [11:39] things that you use to calculate ROI to [11:42] get yourself there. A KPI is also not a [11:44] feeling. Uh thanks Chad GPT. I really [11:46] enjoyed that that document summary. Like [11:48] that was great. Like that's that's not [11:49] measurable, right? And so you have to [11:51] take uh the uh the use cases that you've [11:54] got and then address those with KPIs [11:57] that you can in some way quantify. Now [11:59] these are three of the most common use [12:01] cases that we see and uh some example [12:04] KPIs for these and uh the quantifi [12:08] quantification of these these are again [12:09] some examples of how you could choose to [12:11] quantify those. Now uh the the [12:14] interesting thing here is that every KPI [12:16] just like with you know we mentioned [12:18] earlier is associated to a version of [12:21] your agent because as you change things [12:24] the numbers will change and you want to [12:25] be able to say like look I changed the [12:27] model and this was the impact on user [12:29] satisfaction. I changed the prompt and [12:31] all of a sudden it's taking way longer [12:33] for it to accomplish its objective. Now, [12:36] uh, when we've been looking at, uh, KPIs [12:39] for all of those use cases, an [12:41] interesting observation, which we didn't [12:42] expect and, uh, and by all means, again, [12:45] if you, if you have an alternative, I'd [12:47] love to hear it. It turns out that in [12:48] terms of industry standard KPIs, they [12:50] tend to fall into four buckets. You have [12:52] booleans, like true false, you have [12:54] numbers, just, you know, pick a number, [12:56] percentages, and then liker like one to [12:57] five star, one to 10 star type of thing. [12:59] that seems to encapsulate 99% or more of [13:03] all of the different types of ways to [13:05] quantify KPIs associated with these use [13:07] cases that's actually in production [13:09] today. And then step three is to combine [13:12] uh all the data that you've got that [13:14] you're collecting about these agents [13:16] with the KPIs so that you can understand [13:19] you know sort of what the impact is and [13:20] uh let's look at these uh one at a time. [13:22] So the first one is financial value. us [13:24] back to our basic ROI equation. And we [13:26] talked earlier about how you get to the [13:27] numerator, right? Like savings, revenue, [13:29] firing everybody, things like that. Uh [13:30] and and a little bit about the [13:31] denominator. But now you can also [13:33] understand like by taking my business [13:35] logic and looking at the the [13:37] quantification of these KPIs, my [13:39] click-through rate, my conversion rate, [13:40] and so on, that can tell you how to [13:43] populate this uh this equation. You [13:45] understand what your profit numbers are [13:47] as a result of that. So again, you're [13:48] taking the things that you're [13:49] understanding about the outcomes, you're [13:51] turning that into numbers that you can [13:52] then push into equations to understand [13:54] your ROI. Uh to calculate time savings, [13:57] a bit more involved. This is a new [13:58] equation. [13:59] So uh this equation seems more [14:01] complicated than it is. Uh the s here is [14:04] simply an indicator of whether or not [14:05] the task was completed to satisfaction. [14:07] So you know zero if it was not, one if [14:10] it was, and so uh and then n is just the [14:12] normal time it takes to complete the [14:13] task and then a is the amount of time it [14:15] takes for the agent to complete the [14:16] task. And so the idea is that this [14:18] simple uh single equation can uh justify [14:21] whether you've got an agent that's just, [14:23] you know, doing everything awesome, [14:24] saving tons of time, or it's failing [14:27] completely, and it's just wasting a [14:28] bunch of time, or the more nefarious [14:29] one, which is it's succeeding, but it's [14:31] not actually saving me as much time as I [14:33] hoped, right? And this this this will [14:35] explain all that. So looking at our [14:36] document summarization example, uh you [14:38] have a quality rating, which is an [14:40] indication of whether or not it was [14:41] completed to satisfaction. You have the [14:43] size of the document that it's [14:44] summarizing and that can help you fill [14:46] in all of this to understand your time [14:47] savings. And the summation here is [14:49] simply across all of the documents that [14:51] you're summarizing for example. And then [14:53] productivity is a bit more involved. Um [14:56] and so uh I have only a few minutes left [14:58] so I'm going to skip over the majority [14:59] of this. I want to leave you with one [15:00] sound bite here. The most important [15:02] sound bite uh for productivity. The the [15:04] agent is increasing user productivity if [15:08] the probability of it completing its [15:10] task successfully is greater than the [15:13] time it takes for the human to verify [15:15] the task divided by the time it takes [15:17] for the human to do the task. I'll say [15:19] that one more time. The the agent is [15:23] increasing productivity if the [15:25] probability of it completing it [15:27] successfully is greater than the time it [15:29] takes to verify the task divided by the [15:31] time it takes for the human to do the [15:32] task. If you take away one thing about [15:34] productivity, it should be that. So, I [15:36] know I went really quickly quickly [15:37] through a lot of stuff here. So, just in [15:39] summary, uh we defined agents as either [15:41] a workflow where uh normal software [15:43] leverages an LLM, a pipeline of [15:46] predetermined steps, uh that you know [15:48] leverage LLM as part of those steps or [15:50] something that's truly agentic and will [15:51] actually make decisions use MCP servers [15:53] or other types of tools and things like [15:54] that. Uh we went over business value uh [15:57] measured either in value generation, [15:59] time savings and productivity. And then [16:01] we discussed uh splitting an agent into [16:03] use cases driven by outcomes, taking [16:05] measurements on those outcomes and then [16:07] different categories of quantifiable [16:09] KPIs to use in those measurements and [16:11] then finally how to combine all of those [16:13] into the task of tackling business [16:15] value. Now this is just the beginning uh [16:18] because in order to make fun to make use [16:20] of all these fun equations, you actually [16:21] have to calculate and collect all of [16:23] that data, right? uh and so you need a [16:25] system which which measures all those [16:26] things and then uh accordingly puts it [16:28] all together uh across your versions but [16:30] also takes into account a bunch of [16:32] things that I didn't have time to talk [16:33] to you about today like failure rates [16:35] capacity planning and things like that [16:37] which are you the real everyday I'm [16:39] actually deploying an agent out there [16:40] and I'm trying to see how successful [16:41] this is and so that's obviously where uh [16:44] PI comes in uh we were recently featured [16:46] on uh Forbes as the go-to platform for [16:49] managing Genai ROI uh we're recognized [16:51] by the PHOPS Foundation as the only [16:53] Genai PHOps solution and uh we're not [16:55] just developing the tools to do this, [16:56] we're actually developing the practices [16:58] behind this. And so a lot of the stuff [17:00] that you saw today, you know, utterly [17:02] unique uh to the platform. And so in [17:04] addition to understanding the business [17:05] value for your agents, we also give you [17:07] things like uh financial governance, [17:08] real-time spend protection, so you don't [17:10] have an agent that goes bananas and [17:11] spikes your costs like crazy. Uh and [17:13] we're also working with some of the [17:14] world's largest enterprises on this. So [17:15] I'm happy to announce today that we're [17:17] partnered officially with AWS. Um they [17:19] are uh bringing a program called value [17:21] which has an acronym shim similar to [17:23] like shield from Avengers uh if you guys [17:25] have seen that and um AWS team members [17:28] will actually work with you on every [17:29] step of the Genai value realization [17:31] journey starting with defining your use [17:34] cases identifying KPIs and then using PI [17:36] to tell you which of those use cases [17:38] actually has real ROI and from there [17:41] work with you to optimize uh so that [17:43] when you post on LinkedIn that [17:44] everything that your company is doing is [17:45] agents you're avoiding the hype and [17:47] actually can uh can say that with some [17:49] confidence. Um, so if you're uh on AWS [17:52] today, reach out to your AWS rep. Uh, [17:53] but also you can come directly to our [17:55] booth. We're in booth 246. Be happy to [17:57] show you a demo and um, I would love to [18:00] show you uh, PI in all of its glory and [18:02] also how you can get agents to be [18:04] actually providing ROI for your [18:05] business. I think we have about two [18:06] minutes left for questions. I know I [18:08] went very quickly through all that. Uh, [18:09] the question was, are we focused on ROI [18:12] or deployment? Uh, we are focused on [18:14] ROI. So uh all about understanding the [18:17] business implications of of leveraging [18:19] an agent. Everybody will sell you an [18:21] agent, right? And you can go build all [18:22] sorts of agents, but which one of those [18:24] are actually tangibly resulting in stuff [18:27] back to your business, right? Like [18:28] that's what we care about. [18:30] >> Yeah, that's a great question. So the [18:31] question was if you know of the three [18:32] ways of thinking about productivity um [18:34] or sorry thinking about value, excuse [18:36] me, which are uh financial value, uh [18:38] time savings and productivity, how do [18:39] you prioritize uh amongst those three? [18:42] And the answer is for every use case uh [18:44] it really is going to depend. Um so if [18:47] you are trying to make money with [18:48] something then it's going to be [18:49] financial. If you're trying to save time [18:51] you know what I mean like it's going to [18:52] matter uh depending but you can actually [18:54] measure all three. And in fact you [18:55] should uh if one of the most common [18:58] questions we get is like hey uh we're [19:00] trying to use chat GPT to uh to do X Y [19:03] and Z. What kind of value does that [19:04] bring back to the company? And I say [19:06] well how do you quantify value? Like [19:08] what does value mean to you at the end [19:09] of the day? And so it it's uh it's very [19:12] specific to your business. There's no [19:13] silver bullet there.