Tetrate Product Chief Links AI Return to Gateway Infrastructure
Ai4
Enterprise AI ROI hinges on combining developer output metrics with centralized gateway architecture to curb unpredictable token spending.
AI-assisted summary of ROI Maxxing Case Study: The Alchemy of Shipping Faster, Proving AI Value, and Controlling Token Cost — original source. Check the original for full context.
Source published:
Detailed notes
## Background This talk by David Wang, Head of Product at Tetrate.io, was presented on YouTube by the Ai4 channel on August 18, 2026. According to the page description, the session addresses the critical challenge of proving AI ROI for engineering leaders, providing a practical playbook on defining ROI, selecting infrastructure choices, and controlling token costs through an end-to-end enterprise case study.
## TL;DR David Wang breaks down the AI ROI equation into organizational outcomes (the numerator) and aggregated inference spend (the denominator). He highlights developer productivity as the single largest area of AI token consumption and explains how unconstrained 'token maxxing' and custom proxy scripts lead to massive cost overruns. To fix this, he advocates for an AI Gateway layer—demonstrated live—to centralize multi-model visibility, enforce security guardrails, enable prompt caching, and provide granular usage analytics across engineering teams.
## Atmosphere & style The presentation is pragmatic, technical, and lightheartedly relatable, with Wang opening by warmly acknowledging the extreme Las Vegas conference weather. Using clever analogies like 'Schrodinger's cat' to describe uncertain AI projects and comparing homegrown proxy maintenance to 'adopting a puppy,' he maintains an engaging, conversational rhythm. The live dashboard demonstration shifts the mood from conceptual strategy to crisp, practical execution.
## Key takeaways - [00:04](https://www.youtube.com/watch?v=MOOt9a6uy1g&t=4s) AI initiatives exist in a 'Schrodinger's cat' superposition of being deployed and dead until ROI is questioned, yielding wildly inconsistent metrics across organizations. - [00:04](https://www.youtube.com/watch?v=MOOt9a6uy1g&t=4s) Tetrate leverages Envoy, the open-source proxy routing major global internet and AI traffic, as the architectural backbone for managing AI traffic and ROI. - [02:44](https://www.youtube.com/watch?v=MOOt9a6uy1g&t=164s) AI spend is shifting from predictable subscription models to volatile, rapidly compounding costs driven by autonomous agents and changing model provider pricing. - [02:44](https://www.youtube.com/watch?v=MOOt9a6uy1g&t=164s) Developer productivity is the primary focal point of enterprise AI token spend because software directly underpins revenue, customer experience, and OPEX. - [04:46](https://www.youtube.com/watch?v=MOOt9a6uy1g&t=286s) The AI ROI equation divides developer-driven outcomes (the numerator) by aggregated inference spend (the denominator). - [04:46](https://www.youtube.com/watch?v=MOOt9a6uy1g&t=286s) Defining the ROI numerator is an organizational leadership challenge, requiring top-down alignment on proxy metrics despite flawed or incomplete data. - [04:46](https://www.youtube.com/watch?v=MOOt9a6uy1g&t=286s) Calculating the ROI denominator is a complex technical challenge due to fragmented token spend across multi-cloud providers, regions, and business units. - [06:44](https://www.youtube.com/watch?v=MOOt9a6uy1g&t=404s) The AI Gateway serves as a crucial centralized control plane between AI consumers and AI resources, enabling visibility, policy enforcement, and compliance. - [10:08](https://www.youtube.com/watch?v=MOOt9a6uy1g&t=608s) Unregulated 'token maxxing' leads to loss of financial control, causing risk-averse executives to overcorrect with total AI lock downs. - [10:08](https://www.youtube.com/watch?v=MOOt9a6uy1g&t=608s) Building homegrown proxy layers creates significant engineering drag and inflates long-term infrastructure maintenance costs up to 5x initial estimates. - [10:08](https://www.youtube.com/watch?v=MOOt9a6uy1g&t=608s) A distributed AI Gateway architecture allows enterprises to enforce global compliance while maintaining local low-latency routing and model flexibility. - [14:35](https://www.youtube.com/watch?v=MOOt9a6uy1g&t=875s) To drive developer adoption, AI Gateways must provide developer value through real-time spend visibility, semantic prompt caching, and system prompt debugging. - [14:35](https://www.youtube.com/watch?v=MOOt9a6uy1g&t=875s) Granular user-level analytics empower engineering managers to spot usage anomalies, runaway agent loops, and cost disparities across teams.
## Detailed walkthrough Here is the detailed walkthrough of Part 1 of the transcript:
[00:04](https://www.youtube.com/watch?v=MOOt9a6uy1g&t=4s) Defining AI ROI: The Theory, Practice, and Schrodinger’s Cat Problem
David Wang, Head of Product at Tetrate.io, opens his talk by setting the agenda for his session, which is split into three main segments: theory (the concept of ROI in the age of AI), practice (observations from working across regulated industries and startups), and a live demo showcasing the actual infrastructure tools. Addressing the audience at the conference in Las Vegas during the lunch hour, Wang acknowledges the extreme weather conditions—braving air traffic issues, extreme heat, and heavy indoor air conditioning—and asks the room a fundamental question: "How's that transformation going?"
He compares the current state of enterprise AI transformation to Quantum Mechanics' famous "Schrodinger's cat" thought experiment: at any given moment, an AI transformation project is simultaneously going well, deployed, and dead. You never actually know which state it is in until someone asks the inevitable question: "What am I getting out of it?" If you ask a room of 20 to 40 people that question, Wang notes you will receive 80 different answers. The responses range wildly in data types—they can be an integer (a hard number), a string (a sentence of pure "word salad"), or simply a string of curse words from stakeholders who have no idea how to quantify their results.
Wang introduces his personal background to explain why he finds this problem fascinating. Having previously worked as a management consultant at McKinsey serving major retail banks and retailers, and later at enterprise infrastructure platforms like Databricks and MuleSoft, he views solving AI ROI as both equal parts art and science. Now leading product at Tetrate, Wang benefits from a unique vantage point: Tetrate is the primary driving force behind Envoy, the open-source proxy project that routes a massive portion of the world's internet application traffic—and, increasingly, the majority of global AI traffic. Mastering this underlying traffic, Wang argues, gives teams the visibility and leverage needed to master AI ROI.
- **Speaker Background:** David Wang (Head of Product at Tetrate; ex-McKinsey consultant; former roles at Databricks and MuleSoft). - **Core Technology Platform:** Envoy (the open-source edge/service proxy project driven heavily by Tetrate that routes global application and AI traffic). - **The Schrodinger's Cat Metaphor:** An AI initiative exists in a superposition of being successfully deployed and completely dead until someone queries its ROI.
[02:44](https://www.youtube.com/watch?v=MOOt9a6uy1g&t=164s) The Growth of AI Spend and the Primacy of Developer Productivity
Wang asserts that there are "37 billion reasons" to care about AI ROI today as AI spending accelerates rapidly across every sector. Historically, software and IT spending was small, slow, and highly predictable—primarily composed of flat, recurring user subscriptions for a trickling set of narrow use cases. In the modern AI paradigm, however, spend has become massive, unpredictable, and frequently doubles month-over-month. Wang attributes this explosion in costs to two primary factors: 1. **Changing Usage Patterns:** The shift from human users to autonomous AI agents. Agents operate continuously and consume tokens at a drastically higher rate than humans. 2. **Dynamic Pricing Models:** Frontier AI model labs (such as OpenAI and Anthropic) continuously tweak and restructure their pricing tiers to recover the exorbitant capital expenditure required to train next-generation foundation models.
Because AI spend scales unpredictably, technical leaders and executives inevitably face demands from their CFOs asking, "What did we get for the spend?" When evaluating where token spend is concentrated, Wang emphasizes that developer productivity is the single most critical area to monitor. The vast majority of enterprise AI tokens are currently spent on AI-assisted coding and software engineering.
Reflecting on the previous technology wave—digital transformation—Wang points out that software was placed directly at the intersection of every critical business function: driving revenue, shaping customer experience, and controlling operational expenditure (OPEX). Because modern businesses run fundamentally on software, making that software better, more reliable, and faster to deliver hinges directly on developer performance. As developers increasingly rely on AI to write and ship code, understanding and optimizing AI-driven developer productivity becomes the focal point of answering the overall ROI question.
- **Shift in AI Spend Characteristics:** Transformed from small, slow, predictable subscription costs to massive, volatile, month-over-month doubling usage. - **Drivers of Cost Volatility:** Autonomous AI agents (high token consumption) and frontier model lab pricing shifts. - **The Developer Spend Focus:** Software sits at the intersection of Revenue, Customer Experience, and OPEX; thus, AI token spend is heavily concentrated in developer tooling and code generation.
[04:46](https://www.youtube.com/watch?v=MOOt9a6uy1g&t=286s) The AI ROI Equation: Leadership Challenges vs. Infrastructure Technical Barriers
At its most fundamental level, Return on Investment (ROI) is expressed mathematically as outcomes divided by spend. In the context of developer productivity, Wang defines this ratio as the business outcomes driven by developers divided by the aggregated inference spend required to power their AI tools:
$$\text{ROI} = \frac{\text{Outcomes Driven by Developers}}{\text{Aggregated Inference Spend}}$$
When Wang polls the room to ask how many organizations have attempted to measure this ratio, he notes that while almost everyone tries, most teams hit severe roadblocks. He breaks the measurement challenge into two distinct halves:
``` ┌─────────────────────────────────────────┐ │ AI ROI Equation │ └────────────────────┬────────────────────┘ │ ┌──────────────────────┴──────────────────────┐ ▼ ▼ ┌──────────────────────────┐ ┌──────────────────────────┐ │ Numerator: RETURNS │ │ Denominator: INVESTMENT │ ├──────────────────────────┤ ├──────────────────────────┤ │ Leadership/Org Problem │ │ Technical/Infra Problem │ │ │ │ │ │ • Defining developer │ │ • Fragmented spend │ │ "outcomes" │ │ • Multiple providers │ │ • Disagreeing on metrics │ │ • Multi-geo business │ │ • Debating data flaws │ │ units │ └──────────────────────────┘ └──────────────────────────┘ ```
1. **The Numerator (Returns / Outcomes):** Wang argues that defining the numerator is less of a technical issue and far more of a leadership and organizational alignment problem. Teams struggle to agree on what constitutes a developer "outcome." Is it feature delivery speed, direct revenue generated by a feature, or intangibles like uptime and customer satisfaction? Furthermore, picking the right proxy metrics triggers endless debate: Should teams track Pull Requests (PRs), resolved Jira story points, ticket resolution time, or time-to-first-review? 2. **The Denominator (Investment / Aggregated Inference Spend):** The denominator presents a severe technical and infrastructure challenge. AI inference spend in large enterprises is deeply fragmented across various cloud providers, business units, geographic regions, and rapidly shifting model choices. Consolidating this raw data to get a single, accurate view of total token consumption is extremely difficult without centralized tooling.
- **Numerator Drivers:** Organizational consensus on feature delivery, customer satisfaction, or bug resolution/uptime metrics. - **Denominator Drivers:** Technical aggregation of multi-cloud, multi-model, multi-region token inference spend. - **Common Organizational Pitfall:** Teams get paralyzed arguing over the imperfections of data sources (e.g., whether PR count is a "flawed metric") rather than picking a proxy metric top-down and continuously improving its baseline.
[06:44](https://www.youtube.com/watch?v=MOOt9a6uy1g&t=404s) Unpacking the Value Tree and the Essential Role of the AI Gateway
To solve the numerator challenge, Wang advises leaders to treat return measurement as a two-part decomposition process: separating overarching **value drivers** from tactical **proxy metrics**. Organizations must first select the primary business value driver that matters most to their current strategic goals—whether that is accelerating feature delivery, boosting customer satisfaction, or fixing bugs to maintain system uptime. Once the top-level driver is locked in, leadership must enforce top-down alignment around specific proxy metrics pulled from well-known operational data sources, such as version control systems (e.g., GitHub PRs) or customer support platforms (e.g., ticket resolution times).
Rather than wasting organizational energy arguing over the edge-case deficiencies of a specific metric, successful technical leaders "put their foot down," pick a baseline, and focus their effort on refining the quality of that measurement over time.
Turning to the denominator (tracking and controlling inference costs), Wang highlights the pivotal role of infrastructure. Just as API Gateways were fundamental to the digital transformation era by acting as centralized control planes for REST services, the **AI Gateway** has emerged as the critical architectural component for AI transformations.
``` ┌───────────────────────────┐ │ AI Consumers │ │ (Developers, AI Agents, │ │ Citizen Developers) │ └─────────────┬─────────────┘ │ ▼ ┌───────────────────────────┐ │ AI GATEWAY │ ◄── [ Single Control Plane ] │ • Aggregate Spend Data │ • Real-time Cost Allocation │ • Enforce Guardrails │ • Audit & Compliance Logging │ • Cache Frequent Prompts │ • Traffic & Policy Control └─────────────┬─────────────┘ │ ▼ ┌───────────────────────────┐ │ AI Resources │ │ (Azure OpenAI, Anthropic,│ │ Gemini, Open-Weights) │ └───────────────────────────┘ ```
Deployed directly between AI consumers (developers, autonomous agents, and citizen developers) and underlying AI resources (frontier LLMs, local models, and Model Context Protocol/MCP servers), an AI Gateway provides three indispensable enterprise capabilities: - **Centralized Visibility:** Aggregating usage and cost metrics across all teams and providers in a single pane of glass. - **Policy Enforcement:** Managing security, rate limits, access guardrails, and context windows. - **Auditing & Compliance:** Providing immutable logs, which are strictly mandatory for teams operating within regulated industries (such as banking and healthcare).
[10:08](https://www.youtube.com/watch?v=MOOt9a6uy1g&t=608s) Synthetic Case Study: From "Token Maxxing" Chaos to Unified Infrastructure
Wang transitions from theory to practical application by sharing a case study drawn from a composite of Tetrate’s enterprise customers. In the early stages of their AI journeys, these organizations shared a common backdrop characterized by unconstrained "token maxxing." Developers across disparate business units were adopting any AI tool available, running queries against whatever models they could access, and charging expenses directly to corporate credit cards or isolated purchase orders.
This created an "any-to-any" chaotic environment with zero central control, leaving executives completely unable to track spend or measure business returns. In reaction to this loss of control, many risk-averse enterprise leaders overcorrected by locking down environments entirely, forbidding developers from using external models out of fear of data leakage or unchecked costs. In both extremes—unbridled chaos or total lockdown—developers suffered, losing either structural freedom or access to modern tooling.
``` EARLY CHAOTIC STATE TARGET UNIFIED STATE ┌─────────────────────────────┐ ┌─────────────────────────────┐ │ Fragmented Developers │ │ Fragmented Developers │ └──────┬──────────────┬───────┘ └──────────────┬──────────────┘ │ │ │ ▼ ▼ ▼ ┌────────────┐ ┌────────────┐ ┌─────────────────────────────┐ │ Custom Glue│ │ Homegrown │ │ UNIFIED DISTRIBUTED │ │ Code / VIs │ │ Proxy Infra│ │ AI GATEWAY LAYER │ └──────┬─────┘ └──────┬─────┘ └──────────────┬──────────────┘ │ │ │ ▼ ▼ ▼ ┌────────────────────────────┐ ┌─────────────────────────────┐ │ Isolated Vendor Silos & │ │ Multi-Provider AI Resources │ │ Credit Card Procurement │ │ (Azure, Anthropic, Open- │ └────────────────────────────┘ │ Weight/Self-Hosted Models) │ └─────────────────────────────┘ ```
Furthermore, teams that attempted to build their own internal proxy layers quickly ran into a compounding infrastructure tax: - **Product Engineering Drain:** Engineers reassigned from building revenue-generating, customer-facing features to maintaining internal AI routing scripts, fixing CVE security vulnerabilities in homegrown proxies, and managing custom integrations. Wang compares building custom internal infrastructure to "adopting a puppy or a cat"—it requires continuous, long-term care and maintenance. - **The 5x Cost Multiplier:** The total long-term cost of maintaining a patchwork of custom internal infrastructure ultimately ballooned to over **5x initial estimates**, while locking the enterprise into specific vendor ecosystems.
To escape this trap, leading enterprises execute a deliberate strategic shift: they replace disparate, homegrown routing scripts with a single, globally distributed, centrally managed AI Gateway layer. This architecture allows multinational enterprises to maintain local low-latency routing across international regions while maintaining central control over policies, governance, and cost tracking. Additionally, it enables teams to flexibly blend proprietary frontier models (e.g., Azure OpenAI, Anthropic, Google Gemini) with self-hosted open-weight models (running on private hardware for strategic cost containment and vendor independence).
- **"Token Maxxing" Phase:** Unregulated tool adoption, credit-card purchasing, and lack of auditability. - **Homegrown Infra Pitfalls:** Heavy maintenance overhead, CVE patching, distraction from core product engineering, and a 5x cost overrun compared to initial build projections. - **Target Enterprise Architecture:** A distributed, centrally managed gateway layer capable of dynamically switching between commercial LLM APIs and private open-weight model clusters.
[14:35](https://www.youtube.com/watch?v=MOOt9a6uy1g&t=875s) Live Demo: Developer Experience, System Prompts, and Manager Analytics
With seven minutes remaining in his presentation, Wang jumps into a live demonstration to show how an AI Gateway operates in practice across three distinct user perspectives: the developer, the admin/manager, and the executive.
#### The Developer View Demonstrating from his own laptop, Wang opens his local coding environment (Cloud Code / Cursor-style setup) where he has run several test prompts. He poses a fundamental question: Why would a developer willingly route their queries through an enterprise AI Gateway instead of bypassing it? Governance alone will not drive adoption; the gateway must provide tangible value back to the developer.
Switching to the gateway's personal developer console dashboard, Wang illustrates the immediate benefits: - **Cost & Token Transparency:** Real-time visibility into personal token counts, API request rates, and total dollar expenditure. - **Semantic Prompt Caching:** Clear metrics showing cache hit rates, allowing developers to structure repetitive queries to leverage cached context, saving both time and budget. - **System Prompt Inspection & Debugging:** The dashboard reveals the actual underlying system prompts injected by developer extensions—which are often massive in size. Wang notes that when upgrading an underlying model (e.g., moving from Claude 3.5 Sonnet or Opus 47 to a newer release), agents frequently break or behave unpredictably due to subtle system prompt interactions. Having full, self-service visibility into the exact system prompt payloads drastically simplifies prompt engineering and troubleshooting.
#### The Admin / Engineering Manager View Next, Wang switches to the engineering manager perspective. Using the admin dashboard, a team lead can monitor team-wide token consumption and usage trends over selected time windows. Filtering down to the "Agent Router Engineering Team," Wang inspects per-user token and cost distributions: - High-volume users (e.g., an engineer named Mr. Inas) are highlighted consuming a substantial portion of the team's token budget and overall spend. - Lower-volume users (e.g., an engineer named Aldi) are shown consuming roughly one-fifth of the tokens consumed by top-tier power users (like Ambari).
This granular, per-user breakdown allows engineering managers to identify training opportunities, optimize developer workflows, and catch runaway agent loops before they turn into unexpected monthly invoice shocks.
- **Developer Portal Capabilities:** Personal usage metrics, token costs, prompt caching hit-rates, and full system prompt payload inspection. - **Admin Portal Capabilities:** Multi-tenant usage tracking, team-level breakdowns, user-by-user token consumption comparisons, and cost-anomaly detection.
## Notable quotes > [00:04](https://www.youtube.com/watch?v=MOOt9a6uy1g&t=4s) How's that transformation going? — David Wang
> [00:04](https://www.youtube.com/watch?v=MOOt9a6uy1g&t=4s) What am I getting out of it? — David Wang
> [02:44](https://www.youtube.com/watch?v=MOOt9a6uy1g&t=164s) 37 billion reasons — David Wang
> [02:44](https://www.youtube.com/watch?v=MOOt9a6uy1g&t=164s) What did we get for the spend? — David Wang
> [06:44](https://www.youtube.com/watch?v=MOOt9a6uy1g&t=404s) put their foot down — David Wang
> [10:08](https://www.youtube.com/watch?v=MOOt9a6uy1g&t=608s) adopting a puppy or a cat — David Wang
## Mentioned David Wang, Tetrate.io, McKinsey, Databricks, MuleSoft, Envoy, OpenAI, Anthropic, GitHub, Jira, Azure OpenAI, Google Gemini, Cloud Code, Cursor, Claude 3.5 Sonnet, Opus 47, Model Context Protocol
## Video description David Wang - Head of Product, Tetrate.io
Every AI initiative now has to answer one question: what did we get for the spend? That question is harder than it sounds: ROI is part measurement problem, part organizational problem, and many teams struggle with both. This talk gives AI engineering leaders and technical executives a working playbook: how to define AI ROI so it survives a budget review, which infrastructure choices actually move the number, and how to run initiatives that ship fast without becoming a token-cost surprise. We'll walk through a real, anonymized case study end to end to go beyond theory, and into actionable practice.
[Source](https://www.youtube.com/watch?v=MOOt9a6uy1g)