TrueFoundry Cuts Health Giant's AI Bill to $15M From $300M Quote

We Saved an Enterprise $100M in Token and Model Costs. Steal Our Playbook With TrueFoundry — Ai4

Ai4

Enterprises must abandon hundreds of fragmented pilots and fund massive anchor deployments using tiered routing and self-hosted models to eliminate runaway vendor fees.

AI-assisted summary of We Saved an Enterprise $100M in Token and Model Costs. Steal Our Playbook With TrueFoundry — original source. Check the original for full context.

Source published:

Detailed notes

## Background Published on YouTube by Ai4 on August 17, 2026, this presentation features Nikunj Bajaj, Co-founder & CEO of TrueFoundry, addressing the challenge of runaway token bills and unmonitored agent pilots. Drawing on large enterprise deployments at companies such as CVS and Siemens Healthineers, Bajaj presents a playbook for starting with major anchor use cases to govern AI spend and maximize business ROI.

## TL;DR Nikunj Bajaj details how a Fortune 50 healthcare enterprise transformed a 500-million-call IVR infrastructure, reducing an initial $300 million vendor estimate down to an actual $15 million annual operating cost across 9,000 stores. He cautions against launching hundreds of disconnected micro-pilots and instead advocates for high-value anchor initiatives that fund a reusable enterprise control plane. By combining tiered semantic routing, aggressive caching, self-hosted open-source models, and strict token attribution, organizations can handle over 90% of requests off expensive frontier models while delivering sub-two-second latency.

## Atmosphere & style The presentation opens with an engaging, interactive room poll in Las Vegas, using clever casino and poker analogies to address audience anxiety around escalating AI bills and stalled pilots. Bajaj maintains a crisp, authoritative, and pragmatic delivery, balancing sharp reality checks about industry failure rates with concrete architectural wins. The session feels like a fast-paced, no-nonsense executive masterclass focused on engineering accountability, platform leverage, and financial discipline.

## Key takeaways - [00:04](https://www.youtube.com/watch?v=HezMNx2NZT4&t=4s) Widespread enterprise frustration stems from runaway AI costs and stagnant pilots that fail to generate measurable business ROI. - [00:04](https://www.youtube.com/watch?v=HezMNx2NZT4&t=4s) A Fortune 50 healthcare enterprise slashed its projected $300M annual model spend down to $15M in production by rearchitecting its system. - [01:35](https://www.youtube.com/watch?v=HezMNx2NZT4&t=95s) Applying frontier LLM APIs directly across high-volume workflows creates unacceptable 90-second response latency and unsustainable multi-hundred-million-dollar cost estimates. - [05:00](https://www.youtube.com/watch?v=HezMNx2NZT4&t=300s) The conventional 'start small' strategy produces fragmented, disconnected micro-pilots that fail to generate enough ROI to fund enterprise-grade infrastructure. - [05:00](https://www.youtube.com/watch?v=HezMNx2NZT4&t=300s) Enterprises should fund high-impact anchor use cases ($100M+ potential value) to justify building the core centralized platform plumbing required for all future agents. - [07:08](https://www.youtube.com/watch?v=HezMNx2NZT4&t=428s) Scaling an agentic voice system across 9,000 retail stores proved that multi-step autonomous workflows can achieve sub-2-second latency and a 95% operating cost reduction. - [09:05](https://www.youtube.com/watch?v=HezMNx2NZT4&t=545s) Under the 'Power Plant Effect,' a single large anchor project finances the shared control plane, accelerating downstream agent deployment from months down to days or weeks. - [10:27](https://www.youtube.com/watch?v=HezMNx2NZT4&t=627s) By implementing tiered semantic routing and caching, enterprises can resolve 91% of queries via local models or deterministic rules, reserving expensive frontier models for only 9% of traffic. - [10:27](https://www.youtube.com/watch?v=HezMNx2NZT4&t=627s) Granular token attribution and pre-execution budget caps eliminate 20% to 50% of wasted AI spend by curbing runaway agent loops and developer configuration errors. - [13:19](https://www.youtube.com/watch?v=HezMNx2NZT4&t=799s) Running safety and compliance guardrails in parallel at the gateway level prevents serial latency bottlenecks in time-sensitive conversational applications. - [14:23](https://www.youtube.com/watch?v=HezMNx2NZT4&t=863s) Every autonomous enterprise agent must have a hard budget cap and an explicit kill switch that routes traffic to deterministic fallbacks when loops or hallucinations occur. - [14:23](https://www.youtube.com/watch?v=HezMNx2NZT4&t=863s) Organizations should implement targeted guardrails for concrete use cases rather than delaying production to defend against hundreds of theoretical failure modes. - [15:29](https://www.youtube.com/watch?v=HezMNx2NZT4&t=929s) The core playbook for enterprise generative AI ROI centers on token attribution, intelligent model routing, self-hosting models at scale, and prioritizing anchor use cases.

## Detailed walkthrough ### [00:04](https://www.youtube.com/watch?v=HezMNx2NZT4&t=4s) AI Costs, Poker Tables, and the $300M Dilemma

Nikunj Bajaj kicks off his presentation by polling the room with a set of pointed questions that immediately strike a chord with the enterprise audience. He asks how many people are upset or anxious about escalating AI costs, and roughly half the hands in the room go up. He then asks how many have witnessed countless AI pilots and prototypes launched across their organizations without delivering any measurable business impact—a scenario that again prompts about 50% of the room to raise their hands. Tying the mood to the Las Vegas setting, he jokes about how many people lost money at the poker tables the night before, humorously promising that by the end of the session, they will solve all three problems: runaway AI costs, zero-ROI pilots, and running out of poker money.

He introduces the central case study of the talk: an external, customer-facing use case at a Fortune 50 healthcare enterprise with services familiar to everyone in the room. The organization was initially quoted an eye-watering $300 million by model providers to build and run this single agentic system for one year. By changing their technical strategy and architecture, they successfully shipped the application into production for approximately $15 million.

- **Initial Audience Sentiment**: ~50% frustrated by AI cost escalation; ~50% struggling with pilot stagnation and lack of business impact. - **The Core Enterprise Example**: Fortune 50 healthcare provider facing a $300M vendor quote, eventually brought to production for ~$15M.

---

[01:35](https://www.youtube.com/watch?v=HezMNx2NZT4&t=95s) The 500-Million-Call Problem: Vendor Quotes and the Latency Trap

Prior to the generative AI boom, the healthcare enterprise operated standard interactive voice response (IVR) phone systems across its 9,000 retail stores, fielding roughly 500 million incoming customer phone calls every year. The traditional "press 1 for X, press 2 for Y" legacy software cost them only $2 million to $3 million annually to operate. In 2023, leadership set out with high ambitions to modernize this entire phone infrastructure into an LLM-driven agentic system. When they approached major frontier model providers for pricing, they received a staggering annual quote exceeding $300 million. Even after aggressive shopping and negotiations, the lowest quote remained well above $100 million in net API expenses—a non-starter for any reasonable executive.

Beyond prohibitive pricing, early prototypes built on frontier models suffered from catastrophic latency. It took the prototype over 90 seconds to respond to a customer inquiry over the phone. Bajaj quips that 90 seconds is long enough for people to get married in a Las Vegas chapel down the street, whereas a telephone support interaction strictly requires sub-two-second latency. This crisis mirrors broader industry patterns: an MIT survey revealed that 95% of generative AI pilots generate zero return on investment, while Gartner projects over 40% of all gen AI initiatives will be abandoned. When enterprises blindly run frontier models on every step of an agentic workflow across 500 million calls, API expenses inevitably run into hundreds of millions of dollars.

```mermaid flowchart TD accTitle: Legacy IVR vs Frontier Model Pitfall subgraph Legacy["Legacy IVR"] A["500M Calls / Year"] --> B["Traditional Menu Routing<br/>'Press 1 for refills'"] B --> C["Annual Cost: $2M - $3M<br/>Deterministic & Low Latency"] end

subgraph NaiveLLM["Naive Frontier Model Prototype"] D["500M Calls / Year"] --> E["Direct Frontier Model API<br/>Every Turn & Step"] E --> F["Vendor Quote: $300M+ / Year"] E --> G["Response Latency: 90+ Seconds"] end ```

- **Call Scale**: 500 million customer calls annually across 9,000 store locations. - **Legacy System Spend**: $2M to $3M per year for traditional menu-based IVR. - **Direct Frontier Model Quote**: Over $300M per year (negotiated down to >$100M, still unfeasible). - **Prototype Latency**: 90+ seconds vs. required production target of <2 seconds. - **Industry Failure Rates**: 95% of gen AI pilots yield 0% ROI (MIT study); >40% of gen AI projects predicted to be canceled (Gartner).

---

[05:00](https://www.youtube.com/watch?v=HezMNx2NZT4&t=300s) The Fallacy of "Start Small": The Case for Large Anchor Use Cases

Bajaj challenges conventional enterprise advice that encourages teams to "start small" with isolated generative AI pilots. When hundreds of departmental teams experiment independently on localized task automation, an enterprise ends up with 200 disconnected pilots. None of these micro-pilots generate sufficient financial return or business justification to fund the shared platform and infrastructure—the "plumbing layer"—required to run generative AI robustly in production. He compares this fragmented approach to playing casino slot machines: every small bet feels negligible on its own, but the house keeps taking all the money because the cumulative spend produces no platform equity.

Instead, TrueFoundry advocates for a counterintuitive strategy: find a small number of massive anchor use cases—around five major initiatives—scaled to the enterprise's size. For global enterprises operating at this magnitude, an anchor use case should represent $100 million or more in potential business impact. Justifying the initial investment through one high-value anchor use case funds the entire shared plumbing layer. Once that core control plane is deployed, every subsequent team and lighter agent workflow can be built on top of the existing infrastructure without constantly reinventing the wheel.

- **The Pitfall of Micro-Pilots**: 200 disparate pilots create duplicate effort without sufficient ROI to justify infrastructure investments. - **The Anchor Strategy**: Identify ~5 high-impact initiatives (e.g., $100M+ potential impact) where just one use case justifies building the entire enterprise platform layer. - **Platform Leverage**: Once the platform plumbing is established, all secondary applications launch with minimal marginal overhead.

---

[07:08](https://www.youtube.com/watch?v=HezMNx2NZT4&t=428s) Deploying to 9,000 Stores: Sub-2-Second Agentic Care

Six months after partnering with TrueFoundry, the healthcare enterprise replaced its legacy phone tree with a fully agentic conversational system in production. Bajaj illustrates the experience through a real-world scenario: a 90-year-old patient calls in and simply says, "Refill my heart medicine." With no structured menu or multi-step navigation required, the agent identifies the specific medication referenced, checks prescription dates and available refill counts against backend health records, validates the order, submits the refill request, and confirms everything back to the caller in under two seconds.

Within 18 months, this voice agent system scaled nationally across all 9,000 retail store locations. For finance leaders, the economic reality was stark: a project initially quoted at $300 million per year was launched and maintained at a steady-state run rate of just $15 million annually—a 95% total cost reduction achieved while delivering real-time performance and superior patient convenience.

- **Time to Initial Production**: Under 6 months from project kickoff to live customer deployment. - **Full National Rollout**: Live across all 9,000 US stores within 1.5 years. - **Real-Time Performance**: Multi-step intent parsing, prescription validation, and order fulfillment completed in under 2 seconds. - **Financial Comparison**: $300M quoted vendor cost reduced to a $15M annual recurring operating cost (95% cost reduction).

---

[09:05](https://www.youtube.com/watch?v=HezMNx2NZT4&t=545s) The Power Plant Effect: How One Anchor Funds the Enterprise

Bajaj illustrates the enterprise leverage of an anchor deployment using a power plant analogy: "You don't build a power plant to charge your mobile phone... You build it to run the factory and then all the phones in the building charge for free." By justifying the centralized plumbing layer on the back of the 500-million-call IVR project, the enterprise unlocked massive downstream velocity. Within roughly a year, they had more than 15 agentic AI applications running in production—each delivering over $10 million in business value—alongside dozens of smaller applications delivering hundreds of thousands to a million dollars each.

Pilots that previously lingered in development for months or years began shipping in days or weeks. Furthermore, this architectural shift improved core customer service containment rates rather than degrading them. The platform gave engineering teams an agile foundation that allowed them to adopt, benchmark, and integrate new model releases into their ecosystem in under 24 hours.

```mermaid flowchart TD accTitle: The Power Plant Platform Effect subgraph Anchor["Foundational Anchor Investment"] A["500M Call Agentic Voice System"] --> B["Enterprise Plumbing Layer & Control Plane"] end

subgraph Compound["Compounding Production Agents"] B --> C["15+ Major GenAI Applications<br/>(>$10M Business Impact Each)"] B --> D["Dozens of Departmental Agents<br/>($100k - $1M Impact Each)"] B --> E["<24h Adoption of New Frontier/Open Models"] B --> F["Deployment Velocity: Days/Weeks vs Months/Years"] end ```

- **Downstream Applications**: 15+ production AI agents generating >$10M each in business impact; dozens more generating $100K–$1M. - **Deployment Cycle**: Pilot-to-production turnaround compressed from months/years down to days/weeks. - **Model Adaptability**: Rapid evaluation and rollout of new foundational models in <24 hours. - **Business Performance**: Higher automated call containment rates alongside dramatic operational cost reductions.

---

[10:27](https://www.youtube.com/watch?v=HezMNx2NZT4&t=627s) Becoming the House: Tiered Routing, Caching, and Control Planes

In gambling, the house always wins; in generative AI, relying exclusively on proprietary frontier model APIs turns model providers into the house. Bajaj argues that enterprises must "become the house" by taking ownership of their architecture and traffic routing. The team implemented five core levers via a single agent control plane: 1. **Open-Source Infrastructure**: Hosting and directing baseline traffic to self-managed open-source models. 2. **Tiered Semantic Routing**: Directing complex edge cases to frontier models while routing predictable, high-frequency queries to smaller, specialized models. 3. **Semantic Caching and Pre-fetching**: Ensuring identical or near-identical prompts never hit an LLM API repeatedly. 4. **Policy Engines & Budget Caps**: Enforcing automated limits to prevent runaway agent loops. 5. **GPU Auto-scaling**: Dynamically scaling compute resources to match real-time voice call loads.

Through aggressive semantic caching and tiered routing, only 9% of all incoming requests ever reach a frontier model—91% of calls are resolved via deterministic rules, caches, or lightweight local models. As Bajaj emphasizes, "the cheapest token is obviously the one that you never have to send to a model." Additionally, instituting granular token attribution gives visibility into exactly which user, team, or agent spends money. Drawing on his previous experience at Meta, where internal platforms displayed exact training run costs (e.g., "$4,500 for this run") before execution, he notes that transparency and budget controls routinely eliminate 20% to 50% of wasted AI spend by curbing unintended developer errors and runaway agent loops.

- **Traffic Distribution**: ~91% of requests resolved via rules, caches, or small open models; only ~9% reach expensive frontier models. - **Cost Avoidance via Governance**: Catching human configuration mistakes and runaway loops accounts for 20% to 50% in immediate savings across client deployments. - **Token Attribution**: Attributing every generated token to specific teams and workflows enforces accountability before execution.

[13:19](https://www.youtube.com/watch?v=HezMNx2NZT4&t=799s) Developer Behavioral Change and Latency Optimization

When engineers are held accountable for compute resources, their core behavior shifts naturally. In traditional machine learning workflows, developers routinely ask themselves whether a job can initially run on a smaller slice of data, with fewer parameters, or with pruned hyperparameters to keep infrastructure costs in check. The exact same psychological and operational dynamic applies to generative AI. By attributing every generated token directly to the responsible team and displaying that usage on an internal leaderboard, teams gain immediate visibility into their footprint and instinctively optimize to drive costs down. Combined with hosting open-source models on self-managed enterprise GPUs, this cultural shift delivers massive financial yield.

Solving the token cost equation also intersects directly with resolving latency bottlenecks. To eliminate latency overhead without sacrificing performance or safety, enterprises should leverage an intelligent API gateway layer and robust caching mechanisms capable of orchestrating several optimizations simultaneously:

- **Data Pre-fetching:** Loading anticipated context and datasets ahead of runtime to reduce waiting states. - **Semantic & Query Caching:** Reusing previous prompt completions and intermediate outputs for recurring user queries. - **Parallel Guardrails Execution:** Evaluating security, compliance, and toxicity guardrails concurrently in the gateway rather than chaining them in slow, serial pipeline steps.

```mermaid flowchart TD accTitle: Enterprise AI Gateway Architecture subgraph ClientLayer ["Client & Input"] REQ["Agent / App Query"] end

subgraph Gateway ["TrueFoundry Gateway Control Plane"] CACHE{"Cache Hit?"} PREFETCH["Data Pre-fetching"] PARALLEL["Parallel Guardrails Evaluation"] KILLSWITCH{"Budget Limit / Kill Switch"} end

subgraph ExecutionLayer ["Model & Fallback Routing"] LLM["Self-Hosted / Open-Source Models"] FALLBACK["Fallback Rule-Based System"] end

REQ --> CACHE CACHE -- "Miss" --> PREFETCH CACHE -- "Hit" --> REQ PREFETCH --> PARALLEL PARALLEL --> KILLSWITCH KILLSWITCH -- "Normal" --> LLM KILLSWITCH -- "Rogue / Exceeded" --> FALLBACK ```

---

[14:23](https://www.youtube.com/watch?v=HezMNx2NZT4&t=863s) Agent Governance, Kill Switches, and the Use-Case-First Guardrails Strategy

Deploying autonomous agents requires strict runtime boundaries and containment mechanisms. Every single enterprise agent must have a hard budget cap and an explicit kill switch attached to it. When an agent enters an infinite loop, hallucinates uncontrollably, or behaves unexpectedly in production, teams must be able to "pull the trigger" immediately to shut down the rogue process and safely route inbound traffic to deterministic, rule-based fallback systems.

Equally critical is avoiding the trap of building exhaustive guardrails before shipping applications. Organizations that adopt a "guardrails-first" philosophy frequently suffer from analysis paralysis and fail to ship anything to production. Attempting to anticipate 300 theoretical failure modes upfront forces teams to build defenses for non-existent issues. Selling into large, highly regulated enterprises proves that the pragmatic approach is to:

1. **Lead with a clear, valuable use case:** Identify the exact business problem and the concrete risks tied directly to it. 2. **Implement targeted guardrails:** Build defenses strictly addressing those specific, real-world failure modes. 3. **Establish an extensible plumbing layer:** Design modular infrastructure that allows additional policy controls and safety guardrails to be introduced incrementally as scope expands—ensuring teams never "put the cart before the horse."

---

[15:29](https://www.youtube.com/watch?v=HezMNx2NZT4&t=929s) The Enterprise Playbook: Four Core Rules for Generative AI ROI

To scale enterprise AI efficiently and avoid burning tens of millions of dollars on uncontrolled pilot sprawl, organizations should anchor their strategy around four fundamental principles:

- **Attribute every single token:** Track token consumption down to the specific team, user, and agent level to enforce accountability and cost visibility. - **Route to the appropriate model:** Avoid reflexively upgrading prompts to the newest or largest flagship model; route simpler tasks to lightweight, specialized models. - **Own the model layer at scale:** Transition workloads to self-hosted open-source models on enterprise GPU infrastructure as volume expands to achieve superior unit economics. - **Fund anchor use cases over disconnected pilots:** Concentrate budget and engineering horsepower behind transformative, high-impact enterprise deployments rather than hundreds of fragmented, unmonitored experiments.

## Notable quotes > [01:35](https://www.youtube.com/watch?v=HezMNx2NZT4&t=95s) 90 seconds is long enough for people to get married in a Las Vegas chapel down the street — Nikunj Bajaj

> [07:08](https://www.youtube.com/watch?v=HezMNx2NZT4&t=428s) Refill my heart medicine. — Nikunj Bajaj

> [09:05](https://www.youtube.com/watch?v=HezMNx2NZT4&t=545s) You don't build a power plant to charge your mobile phone... You build it to run the factory and then all the phones in the building charge for free. — Nikunj Bajaj

> [10:27](https://www.youtube.com/watch?v=HezMNx2NZT4&t=627s) the cheapest token is obviously the one that you never have to send to a model. — Nikunj Bajaj

> [05:00](https://www.youtube.com/watch?v=HezMNx2NZT4&t=300s) start small — Nikunj Bajaj

> [10:27](https://www.youtube.com/watch?v=HezMNx2NZT4&t=627s) become the house — Nikunj Bajaj

> [10:27](https://www.youtube.com/watch?v=HezMNx2NZT4&t=627s) $4,500 for this run — Nikunj Bajaj

> [14:23](https://www.youtube.com/watch?v=HezMNx2NZT4&t=863s) pull the trigger — Nikunj Bajaj

## Mentioned Nikunj Bajaj, TrueFoundry, Ai4, CVS, Siemens Healthineers, Meta, MIT, Gartner

## Video description Nikunj Bajaj - Co-founder & CEO, TrueFoundry

Your engineering teams are shipping agents faster than anyone can track them. Token bills climb with no clear attribution, and when an agent loops or breaks, there's no way to audit the spend or shut it down. Most enterprises got here following the "start small" advice, which quietly produces hundreds of disconnected pilots, GPU waste, and no shared governance.

Nikunj Bajaj, CEO of TrueFoundry, flips the script and makes the case for starting big. Drawing on production deployments at large global enterprise customers (CVS and Siemens Healthineers) that drove tens to hundreds of millions in impact, he shares the concrete plays used to rein in token spend and maximize ROI.

[Source](https://www.youtube.com/watch?v=HezMNx2NZT4)

中文简报

TrueFoundry将医疗巨头AI账单从3亿美元报价降至1500万美元 企业必须放弃数百个碎片化试点项目，通过分层路由和自托管模型资助大规模主力项目部署，以消除失控的供应商费用。 Nikunj Bajaj详细介绍了某财富50强医疗企业如何重构其处理5亿次通话的IVR基础设施，在覆盖9,000家门店的范围内，将最初3亿美元的供应商预估费用降至实际每年1500万美元的运营成本。他告诫企业不要开展数百个彼此脱节的微型试点，并主张通过高价值的主力项目来支持可复用的企业控制平面的建设。通过结合分层语义路由、激进缓存、自托管开源模型和严格的Token归因，企业可以在分流昂贵前沿模型的前提下处理超过90%的请求，同时将响应延迟控制在两秒以内。 某财富 50 强医疗企业将 9,000 家门店的 5 亿次年呼叫系统重构为 Agent 语音平台，不仅将响应延迟从原型期的 90 秒以上压缩至 2 秒以内，更将高达 3 亿美元的年度供应商报价大幅削减 95% 至 1,500 万美元。 (7:08) 针对 95% 的生成式 AI 试点缺乏业务回报的行业困境，企业应放弃产生大量孤岛的“从小做起”策略，转而聚焦潜在商业价值超 1 亿美元的“锚点用例”，以此论证并资助企业级共享基础设施与控制平面的建设。 (5:00) 借助“发电厂效应”，由核心锚点项目跑通的基础设施底座可赋能全企业，推动后续 15 个以上千万美元级生产应用的落地，并将 Agent 交付周期从数月缩短至数天或数周。 (9:05) 通过引入分层语义路由、语义缓存与自托管开源模型，系统将 91% 的常规请求分流至低成本本地方案，仅 9% 的长尾流量调用昂贵的前沿旗舰模型，配合精细化 Token 归因与断路 Kill Switch 杜绝了死循环与预算失控。 (10:27)

Reading summary

Enterprises can slash generative AI expenses and overcome stalled pilots by funding large anchor deployments rather than micro-experiments, then establishing a centralized control plane to route traffic away from expensive proprietary models. The Enterprise Pilot Stagnation and Latency Crisis (0:04) Enterprises frequently struggle with escalating AI costs and low business returns. Nikunj Bajaj cites industry surveys showing that 95% of generative AI pilots yield zero return on investment (MIT) and that over 40% will ultimately be abandoned (Gartner). A Fortune 50 healthcare enterprise managing 500 million annual IVR calls across 9,000 stores faced vendor quotes exceeding $300 million—later negotiated to over $100 million—to replace legacy systems costing $2 million to $3 million annually. Early prototypes relying directly on frontier model APIs also suffered from catastrophic response times exceeding 90 seconds, far beyond the sub-two-second requirement for telephone customer interactions. Funding Shared Plumbing Through Anchor Use Cases (5:00) Bajaj challenges the conventional practice of starting small with hundreds of isolated departmental experiments, comparing it to playing slot machines without building platform equity. Instead, he advocates selecting a small set of high-impact anchor initiatives—representing $100 million or more in potential business value—to economically justify the centralized infrastructure and control plane. Deploying this shared foundational layer then dramatically reduces the marginal cost and development time for subsequent applications. In the healthcare case study, an agentic voice system handling complex, real-time prescription lookups rolled out across all 9,000 stores in 18 months. The initial $300 million quote was reduced to a steady-state run rate of approximately $15 million per year, yielding a 95% cost reduction. The shared platform subsequently enabled the deployment of over 15 additional production agents delivering more than $10 million in value each, while cutting deployment cycles from months to weeks and allowing new models to be integrated within 24 hours. Architecture Levers: Tiered Routing, Caching, and Pre-fetching (10:27) To reduce reliance on proprietary APIs, enterprises must direct traffic through an intelligent gateway. By combining semantic caching, data pre-fetching, and tiered semantic routing, 91% of incoming requests in the healthcare deployment were resolved via deterministic rules, caches, or self-hosted open-source models, leaving only 9% to touch frontier APIs. Evaluating compliance and toxicity guardrails concurrently in the gateway rather than serially also eliminated latency bottlenecks. Granular token attribution further drove savings by establishing cost transparency for development teams. Presenting compute costs before execution, a practice Bajaj notes from Meta, combined with automated policy limits and hard budget caps, routinely prevents rogue agent loops and cuts unintended enterprise AI spend by 20% to 50%. Targeted Governance and the Production Playbook (14:23) Bajaj cautions against a 'guardrails-first' mentality where teams attempt to anticipate hundreds of theoretical failure modes upfront, arguing it causes analysis paralysis. Instead, organizations should prioritize specific, real-world failure points tied to concrete use cases, while deploying kill switches that can immediately divert aberrant agents to deterministic fallback systems. The resulting operating playbook rests on four core tenets: attributing every token to specific teams or agents, routing queries to the most lightweight suitable model, transitioning high-volume workloads to self-hosted open-source models, and prioritizing enterprise-scale anchor initiatives over disconnected pilots. The presentation demonstrates that substantial cost reductions and real-time latency targets are technically achievable at massive scale. However, the approach relies on having sufficient internal volume, engineering capacity, and enterprise-grade GPU infrastructure to justify hosting open-source models and building out a proprietary control plane.

精炼阅读摘要

TrueFoundry首席执行官Nikunj Bajaj指出，企业盲目依赖前沿大模型API进行分散试验会导致巨额成本与高失败率；通过聚焦高价值核心用例、自建控制平面、实施分层路由与严格治理，企业能够在保障延迟与安全的前提下实现数亿美元的成本缩减。 前沿模型API的成本与延迟困境 (1:35) Bajaj援引数据指出，生成式AI在企业落地中面临极高阻力：MIT的一项调查显示95%的生成式AI试点项目投资回报率为零，Gartner也预测超过40%的相关项目将被放弃。在传统架构下，一家拥有9000家零售门店、年均呼叫量达5亿次的财富50强医疗企业，其传统按键式交互语音应答（IVR）系统年运营成本仅为200万至300万美元。但在2023年尝试直接接入前沿大模型进行现代化改造时，供应商给出的年报价高达3亿美元以上，经谈判后净API开销仍超过1亿美元。 除了经济成本不可承受，初代大模型原型系统在实际交互中暴露出严重的性能短板。在涉及5亿次通话的电话支持场景中，系统要求响应延迟必须控制在2秒以内，而前沿大模型原型的端到端响应时间超过90秒。如果企业对工作流的每一步都直接调用昂贵且延迟高昂的前沿API，不仅无法满足实时业务需求，还将直接陷入财务与落地的双重泥潭。 聚焦核心锚点破除零散试点陷阱 (5:00) Bajaj反对业内普遍提倡的“从小做起”策略，认为数百个相互孤立的部门级微型试点只会产生重复建设，且单个项目均缺乏足够的商业回报来论证底层平台与共享基础设施（即“管道层”）的资金投入。这种分散尝试如同持续向角子机投币，累积了成本却无法沉淀平台资产。相比之下，大型企业应当集中资源推进约5个潜在商业价值在1亿美元以上的核心“锚点用例”（Anchor Use Cases）。 以“发电厂效应”为比喻，企业建设发电厂是为了驱动工厂运转，随后厂区内的手机充电则近乎零边际成本。上述医疗企业通过5亿次呼叫的IVR改造这一重大项目，成功论证并建成了企业级控制平面基础设施。在核心管道层落地后，企业在一年内迅速衍生出15个以上各具1000万美元商业价值的AI应用，以及数十个规模较小的应用；试点到上线的周期从数月压缩至数天或数周，同时使引入和评测新模型的时间缩短至24小时以内。 控制平面架构与分级路由优化 (10:27) 为摆脱受制于专有模型API供应商的局面，企业必须自主掌握流量路由和架构控制权。通过部署集成语义路由、预取、语义缓存、策略引擎及GPU自动扩缩容的单一智能网关，该医疗企业在项目启动6个月内实现首个生产部署，并在18个月内将智能体语音系统推广至全美9000家门店，在实现亚两秒响应（如自动识别老年患者用药、校验病历并完成续方）的同时，将年化运营成本从3亿美元报价压降至1500万美元，降幅达95%。 在优化后的调用流中，高达91%的请求直接由确定性规则、语义缓存或本地轻量级开源模型处理，仅有9%的复杂长尾任务会被分流至高成本的前沿模型。此外，网关层通过数据预取与安全合规防护栏（Guardrails）的并行评估，大幅消减了传统串行验证造成的等待时间，既阻断了高昂的无谓调用，又保障了亚两秒的严苛电话延迟指标。 代币归属度量与用例驱动型治理 Bajaj指出，治理机制能直接改变开发者的工程习惯。借鉴Meta的内部管理经验，通过将每一个生成的Token精确归属于具体用户、团队和智能体，并在内部排行榜上进行成本可视化展示，开发者会本能地优化超参数、剪枝任务或优先选用轻量模型。仅通过拦截配置错误、防止失控循环以及提升成本透明度，企业便能普遍消除20%至50%的AI资金浪费。 在安全防御策略上，Bajaj警告切勿陷入“防护栏先行”导致的分析瘫痪。试图预先防御300种假想故障不仅拖延上线，还会构建无用防线。务实的治理框架应坚持“用例先行”，仅针对明确业务场景下的具体风险开发防御规则；同时，所有上线智能体必须配置硬性预算上限和强制终止开关（Kill Switch），一旦发生严重幻觉或无限循环，系统可立即切断进程并将流量降级至确定性规则引擎。 Bajaj总结的企业生成式AI投资回报法则强调：精确计量每个Token归属、按任务复杂度精准路由模型、规模化阶段自建托管开源模型，以及优先资助高价值锚点用例而非碎片化试点。不过，该案例主要立足于调用量极大的特定语音服务场景，低频或非结构化探索性业务能否以同等幅度复现该降本路径仍有待验证。

Detailed notes — zh

## Background 本内容由 YouTube 频道 Ai4 于 2026 年 8 月 17 日发布，演讲者为 TrueFoundry 联合创始人兼 CEO Nikunj Bajaj。页面简介指出，企业内部 Agent 快速上线导致 Token 账单失控且缺乏有效治理，Nikunj 结合服务全美大型医疗零售巨头（如 CVS 和 Siemens Healthineers）的实际生产经验，分享如何打破传统“从小处着手”的误区，通过大锚点用例与控制平面策略帮助企业削减上亿美元的模型与 Token 成本。 ## TL;DR 演讲围绕一家拥有 9,000 家门店的财富 50 强医疗巨头展开，该企业在推进年处理 5 亿通电话的客服 Agent 改造时，面临模型厂商开出高达 3 亿美元报价及 90 秒严重延迟的双重危机。针对业内 95% 试点项目零 ROI 的普遍困境，Nikunj 提出以亿级价值的“大锚点用例”撬动底层通用管道层建设的破局打法，并通过单一控制平面、分层路由、语义缓存、开源模型托管与紧急熔断等五大工程杠杆，将实际落地成本降至 1500 万美元（降幅达 95%），响应速度压进 2 秒。最后，他总结了精准归属 Token、按需路由、掌控模型层及集中重金资助核心标杆场景的企业 AI 降本增效四大核心法则。 ## Atmosphere & style 整场演讲务实、干练且极具实操感染力。Nikunj 开场便以拉斯维加斯赌场输赢和现场举手互动调侃，迅速拉近与受众的距离，并在调侃“90 秒延迟比在小教堂办婚礼还长”的幽默中直击行业痛点。叙事节奏紧凑而直白，毫不拖泥带水，将冷冰冰的架构图与生动的“发电厂与手机充电”隐喻巧妙结合，传递出兼具战略高度与工程硬核底气的务实掌控感。 ## Key takeaways - [00:58](https://www.youtube.com/watch?v=HezMNx2NZT4&t=58s) 某全美财富 50 强医疗零售巨头在改造年处理 5 亿通电话的 IVR 系统时，最初前沿模型 API 报价高达 3 亿美元，最终通过系统性工程架构将成本压缩至 1500 万美元，降幅达 95%。 - [03:19](https://www.youtube.com/watch?v=HezMNx2NZT4&t=199s) 企业在直接调用前沿大模型构建客服原型时遭遇了超过 90 秒的超长交互延迟，而真实的电话语音交互场景要求系统必须在 2 秒以内完成端到端响应。 - [03:19](https://www.youtube.com/watch?v=HezMNx2NZT4&t=199s) MIT 调研显示 95% 的生成式 AI 试点项目没有产生投资回报，Gartner 也预测超 40% 的项目将面临流产，过度依赖分散且低价值的微小试点往往导致巨大浪费。 - [05:00](https://www.youtube.com/watch?v=HezMNx2NZT4&t=300s) 企业应当摒弃传统的“从小处着手”思维，转向“从大处着手”，集中资源攻克能产生过亿美元业务价值的锚点用例，以此支撑起标准化的通用底层工程管道层（Plumbing Layer）。 - [08:21](https://www.youtube.com/watch?v=HezMNx2NZT4&t=501s) 通过围绕核心锚点用例搭建统一平台，后续 15 个以上的生产级 AI Agent 能够在数天或数周内快速交付，每个应用均贡献了超千万美元的业务价值。 - [10:27](https://www.youtube.com/watch?v=HezMNx2NZT4&t=627s) 实现 95% 降本的核心在于建立单一控制平面（Single Control Plane），结合分层路由、语义缓存与预取、开源小模型驱动、预算上限及 GPU 弹性伸缩五大杠杆。 - [10:27](https://www.youtube.com/watch?v=HezMNx2NZT4&t=627s) 在实际生产流量中，通过智能网关分层路由，90% 的简单和高频请求均由缓存与自建开源小模型快速处理，仅有约 9% 的高难长尾请求才会调用昂贵的商业前沿模型。 - [13:19](https://www.youtube.com/watch?v=HezMNx2NZT4&t=799s) 实行精确到 Token 级的成本归因与团队排行榜公示机制，能促使工程师从心理和行为上主动优化参数和数据集，形成自觉控费的工程文化。 - [13:19](https://www.youtube.com/watch?v=HezMNx2NZT4&t=799s) 针对自主运行的 Agent 必须建立严格的预算上限与紧急熔断机制（Kill Switch），在 Agent 陷入死循环或失控时即时熔断并平滑降级至规则引擎兜底。 - [14:43](https://www.youtube.com/watch?v=HezMNx2NZT4&t=883s) 警惕“安全护栏先行（Guardrails First）”的落地陷阱，脱离实际业务凭空构想数以百计的安全漏洞会导致项目难产，正确的路径应由具体核心业务场景驱动针对性的护栏与基础设施演进。 - [15:29](https://www.youtube.com/watch?v=HezMNx2NZT4&t=929s) 企业 AI 规模化降本增效的四大核心法则为：精准归属每一个 Token、智能路由并克制模型升级、规模化时掌控私有化开源模型层，以及集中重金资助核心标杆场景。 ## Detailed walkthrough ### [00:04](https://www.youtube.com/watch?v=HezMNx2NZT4&t=4s) 开场调查：企业 AI 落地面临的成本与回报困境 演讲在一场与现场观众的互动调查中拉开序幕。Nikunj Bajaj 首先抛出两个普遍困扰企业的问题：有多少人正在为高昂的 AI 成本感到头疼？又有多少人发现企业内部虽然做了大量的 AI 原型和试点项目（Pilots），却迟迟看不到任何实质性的业务回报？现场近半数的观众纷纷举手表示共鸣。紧接着，他幽默地调侃起拉斯维加斯现场的氛围，询问有多少人前一天在扑克牌桌上输了钱。Nikunj 笑称，这场演讲的目的不仅要帮大家解决企业在 AI 成本与业务落地上的两大难题，还要帮大家省下足够的资金去扑克桌上“翻本”。 随后，他引入了一个核心标杆案例：一家全美财富 50 强（Fortune 50）的医疗零售巨头，在构建面向外部真实客户的 Agent 业务系统时，曾被模型供应商开出高达 3 亿美元的天价报价，但最终通过一套系统性的工程打法，成功将实际落地成本压缩至 1500 万美元。 --- ### [00:58](https://www.youtube.com/watch?v=HezMNx2NZT4&t=58s) 案例背景：5 亿通电话与 3 亿美元的模型报价危机 在生成式 AI 浪潮爆发之前，这家覆盖全美 9,000 家门店的医疗零售巨头就面临着极为庞大的业务规模——其门店热线每年需要处理大约 5 亿通客户电话。在过去，支撑这套系统的仅仅是传统的 IVR（交互式语音应答）系统，也就是用户熟悉的“咨询业务请按 1，查询信息请按 2”。这套老旧系统的年软件运营成本大约在 200 万至 300 万美元之间。 随着 2023 年大语言模型（LLM）与 Agent 技术的兴起，该企业雄心勃勃地计划对整个电话系统进行彻底的现代化升级，打造全自动化的智能 Agent 客服。然而，当他们向主流模型供应商询价时，对方针对这一个业务场景的一年使用量给出了超过 3 亿美元的惊人报价。即便企业多方比价，最终的净成本依然超过 1 亿美元。对于任何理性的企业管理者而言，这样脱离实际的成本结构都是无法接受的，迫使他们放弃了直接采购前沿模型 API 构建系统的最初设想。 - **业务规模**：全美约 9,000 家门店，年呼叫量约 5 亿通电话。 - **原有成本**：传统按键式 IVR 系统每年软件成本仅 200 万至 300 万美元。 - **最初报价**：模型供应商开出一年的 API 使用费超过 3 亿美元，竞价后仍高达 1 亿多美元。 --- ### [03:19](https://www.youtube.com/watch?v=HezMNx2NZT4&t=199s) 90 秒延迟与行业通病：“从小处着手”的陷阱 高昂的 Token 费用还只是整个问题的一半。当该企业使用前沿模型搭建出第一版原型系统时，发现系统对客户查询的平均响应时间竟超过了 90 秒。Nikunj 幽默地吐槽道：“在拉斯维加斯街角的小教堂里，两个人结个婚都用不了 90 秒。”而真实电话场景对交互延迟的要求极其苛刻，必须在 2 秒以内完成响应。 ```mermaid flowchart LR accTitle: Prototype vs Target Latency accDescription: Comparison between the initial LLM prototype latency of 90 seconds and the production target of under 2 seconds. subgraph Prototype["初期原型方案"] A["客户语音提问"] --> B["全量前沿模型处理"] B --> C["90秒超长延迟 ❌"] end subgraph Target["生产上线指标"] D["客户自然语言需求"] --> E["企业级 Agent 架构"] E --> F["< 2秒极速响应 ✅"] end ``` 这种困境在整个行业中绝非孤例。MIT 的相关调研显示，当前企业构建的生成式 AI 试点项目中，高达 95% 的项目没有产生任何投资回报；Gartner 也预测超过 40% 的生成式 AI 项目最终将被取消。Nikunj 指出，如果企业在 Agent 系统的每个环节都无脑调用昂贵的前沿模型，并且在 5 亿次调用的规模下运行，必然会导致数亿美元的 API 账单。 - **延迟差距**：初期原型响应耗时超过 90 秒，而生产环境要求低于 2 秒。 - **行业数据**：MIT 调查显示 95% 的 GenAI 试点投资回报为零；Gartner 预测超 40% 的 GenAI 项目将面临流产。 --- ### [05:00](https://www.youtube.com/watch?v=HezMNx2NZT4&t=300s) 破局之道：以“大锚点用例”撬动底层基础设施建设 面对 AI 项目普遍失败的现状，Nikunj 提出了一个反直觉的战略建议：**不要盲目遵循“从小处着手”（Start Small）的传统思维**。在传统模式下，企业内部各个部门各自为战，立项了多达数百个分散的自动化试点，但每个小试点的预期收益都极其微弱，根本无法证明投资搭建一套通用底层基础设施（Plumbing Layer）的合理性。这就像在赌场的角子机上不断下小注，虽然单笔看起来很小，但最终获利的永远是开赌场的“庄家”（即模型提供商）。 相反，成功的企业会选择“从大处着手”（Start Big）——在全公司范围内筛选出大约 5 个能够带来巨大业务价值的“锚点用例”（Anchor Use Cases）。例如对于大型企业而言，单个用例的潜在业务价值应达到 1 亿美元以上。只要能跑通其中一个重磅用例，其产生的巨大收益就足以支撑起一整套标准化的底层工程管道层，后续所有长尾场景便可以直接复用该基础设施，无需重复造轮子。 ```mermaid flowchart TD accTitle: Start Small vs Start Big Strategy accDescription: Comparing the fragmented small pilots approach with the anchor use case strategy driving a shared plumbing layer. subgraph AntiPattern["传统模式：分散试错（低回报）"] P1["试点用例 1"] P2["试点用例 2"] P3["试点用例 200..."] P1 & P2 & P3 --> Loss["ROI 极低 / 无法沉淀通用基础设施"] end subgraph Recommended["推荐模式：锚点用例驱动"] Anchor["选择 5 个亿级业务价值锚点用例"] --> Plumbing["搭建统一底层工程管道层 (Plumbing Layer)"] Plumbing --> Scale["后续 15+ 业务用例快速接入并在数天/数周内投产"] end ``` --- ### [07:08](https://www.youtube.com/watch?v=HezMNx2NZT4&t=428s) 落地见效：从 90 秒到 2 秒的生产级 Agent 体验 在与该医疗客户合作仅 6 个月后，原有的传统电话线就被全面改造为全智能的 Agent 交互系统。Nikunj 分享了一个极具代表性的真实场景：一位 90 岁的老奶奶拨通电话，只说了一句“我想续订我的心脏病药”，没有任何复杂的按键或格式化输入。 系统在接收到这一句话后，能够瞬间完成一系列复杂的意图与业务逻辑处理：精准识别出她所指的心脏病药物名称、调取该处方的历史开具记录、核实她是否还拥有有效的续订配额、自动在后台完成下单，并在不到 2 秒的时间内向患者完成语音回复。这套系统不仅在 6 个月内成功投产，更在短短一年半的时间内铺设到了全美所有 9,000 家门店。 --- ### [08:21](https://www.youtube.com/watch?v=HezMNx2NZT4&t=501s) 财务回报与溢出效应：像建发电厂一样搭建 AI 平台 对于财务团队而言，整个项目最具冲击力的是两组数据：原本供应商预估需要 3 亿美元的系统，实际构建与首年运行成本仅为 1500 万美元，成本直降 95%，且后续每年的运营成本稳定在 1500 万美元。 Nikunj 用了一个形象的比喻来解释平台化建设的溢出价值：“你不会为了给手机充电而去专门建造一座发电厂；你建造发电厂是为了驱动整个工厂运转，而整栋大楼里的所有手机顺便都能免费充电。”正是因为围绕这一核心场景建成了坚实的底层管道层，该企业在随后一年左右的时间里，迅速上线了 15 个以上的生产级生成式 AI Agent 应用，每个应用都贡献了超过 1000 万美元的业务价值，同时还有数十个产生数十万至数百万美元价值的长尾场景同步运行。过去需要数月乃至数年才能落地的试点项目，如今只需数天或数周即可投产。更重要的是，核心 IVR 系统的拦截解决率（Containment Metric）持续提升，且系统对接新模型的时间被压缩到 24 小时以内。 --- ### [10:27](https://www.youtube.com/watch?v=HezMNx2NZT4&t=627s) 降低 95% 成本的五大杠杆与统一控制平面 要想打破模型供应商“庄家通吃”的局面，企业必须将控制权掌握在自己手中，即“让自己成为庄家”（Become the House）。实现 95% 成本削减的核心在于建立一个面向 Agent 应用的“单一控制平面”（Single Control Plane），并通过五大关键杠杆进行精细化流量调度： ```mermaid flowchart TD accTitle: Single Control Plane Architecture accDescription: Five levers within a single control plane enabling cost reduction and intelligent query routing. Query["用户输入 Query"] --> ControlPlane["单一控制平面 (Single Control Plane)"] subgraph Levers["五大降本杠杆"] L1["语义缓存与预取 (Caching & Pre-fetching)"] L2["分层路由 (Tiered Routing)"] L3["开源小模型驱动 (Open Source Models)"] L4["预算上限与策略引擎 (Budget Caps & Policy)"] L5["GPU 弹性伸缩 (Autoscaling)"] end ControlPlane --> Levers Levers -->|90% 流量| FastPath["规则命中 / 缓存 / 开源轻量模型 (<2s 响应)"] Levers -->|仅 9% 高难长尾流量| Frontier["商业前沿模型 (Frontier Models)"] ``` Nikunj 详细拆解了五大杠杆的运作机制： 1. **分层路由（Tiered Routing）与开源模型**：对于极其复杂的问题调用商业前沿模型，而对于大量简单、重复性的高频请求，直接路由到自建托管的开源低成本模型。 2. **语义缓存与预取（Caching & Pre-fetching）**：确保模型回答过的问题绝不再重复提问，“最便宜的 Token 永远是那些根本不需要发给模型的 Token”。 3. **策略引擎与预算上限（Policy Engine & Budget Caps）**：在工程层面设置硬性限额，防止 Agent 陷入死循环调用。据统计，仅避免人为配置失误这一项，在很多客户场景中就能直接挽回 20% 到 50% 的非必要开销。 4. **GPU 弹性伸缩（GPU Autoscaling）**：根据实时负载动态调配底层算力。 5. **Token 级成本归因（Token Tagging）**：为每个发出的 Token 打上业务标签与责任人。Nikunj 引用了自己在 Meta 工作时的经历——在内部平台触发训练任务之前，系统会明确弹窗提示“本次训练运行将花费 4,500 美元”，这种清晰的归因机制极大地改变了工程师的使用行为。 最终，在该企业的实际生产运行中，有 90% 的调用完全无需经过昂贵的前沿模型，仅有约 9% 的疑难请求会真正触发前沿模型调用，从而在保障极速响应的同时实现了数亿美元的成本节约。 ### [13:19](https://www.youtube.com/watch?v=HezMNx2NZT4&t=799s) 行为转变、网关级低延迟优化与 Agent 紧急熔断机制 在软件与模型开发中，当成本变得透明且可追踪时，工程师的心理和行为模式会发生根本性转变。作为开发者，当发现每次调试都会产生成本时，自然会反思：“我是否可以先用更小的数据集跑通流程？能否通过缩减超参数或模型参数来控制开销？” 这种逻辑在生成式 AI 时代完全一致：一旦将每个 Token 的开销精确归属到具体团队，并把消耗情况放在排行榜上公示，团队就会主动寻找优化空间，系统整体也会形成由被动监控转向主动节约的行为文化。与此同时，将开源模型部署在企业自有的 GPU 上，也能在规模化后释放出巨大的成本红利。 除了成本之外，企业落地还必须解决延迟挑战。通过在网关层（Gateway Layer）统一调度，企业可以采用多种技术组合来显著降低系统延迟，包括预取数据集、利用语义缓存（Cache）复用历史相似查询，以及并行执行各项安全护栏检查（Parallel Guardrails）。更关键的是，针对自主运行的 AI Agent，企业必须建立严格的预算上限（Budget Limit）和紧急关停机制（Kill Switch）。一旦某个 Agent 出现逻辑死循环或失控行为，系统能够立即“扣动扳机”将其强制下线，并将用户请求平滑路由至预设的规则备用系统（Fallback Rule-based Systems）。 ```mermaid flowchart TD accTitle: AI Gateway 延迟优化与 Agent 熔断控制流 subgraph Gateway["AI 网关控制层 (Gateway Layer)"] Req["应用/用户请求"] --> Cache{"语义缓存命中?"} Cache -- "是" --> Ret["直接返回缓存响应"] Cache -- "否" --> Parallel["并行处理: 数据预取 & Guardrails"] end subgraph Governance["Agent 运行与治理"] Parallel --> Agent["Agent 执行环境"] Agent --> Monitor{"预算/行为监控"} Monitor -- "异常/超额" --> KillSwitch["触发 Kill Switch 熔断"] KillSwitch --> Fallback["降级路由至规则引擎"] Monitor -- "正常" --> LLM["路由至对应开源/商用模型"] end ``` - **行为驱动降本**：通过 Token 归属与团队排行榜机制，激发工程师主动减少不必要的高消耗调用。 - **网关层性能加速**：利用查询缓存、数据预取与并行护栏过滤，在保证合规的同时极大削减推理延迟。 - **Agent 强制熔断机制**：为每个 Agent 设定严格的预算配额与 Kill Switch，防止智能体失控烧毁预算，并无缝切换至基于规则的兜底系统。 --- ### [14:43](https://www.youtube.com/watch?v=HezMNx2NZT4&t=883s) 警惕“安全护栏先行”陷阱：场景驱动的务实落地法 许多企业在推进大模型项目时容易陷入过度防御的误区，即试图在上线任何业务前构建一套“完美且无死角”的安全护栏。然而从服务众多大型受监管行业企业的实际经验来看，“护栏先行（Guardrails First）”往往会导致项目彻底难产、一件产品都无法交付。原因在于，如果脱离实际业务凭空构想，系统可能会面临多达 300 种潜在的崩溃或漏洞模式，团队会陷入不断填补假想漏洞的泥潭中无法自拔。 切忌“本末倒置（Don't put the cart before the horse）”。正确的工程落地路径必须始于具体的业务场景：首先明确核心用例，识别出该场景下真正致命且相关的风险点，有针对性地构建专用安全护栏；随后，再搭建一套具备高度扩展性的基础设施管道（Plumbing Layer），以便未来随着新业务的接入逐步扩展安全策略。 --- ### [15:29](https://www.youtube.com/watch?v=HezMNx2NZT4&t=929s) 企业 AI 降本增效的四大核心法则与总结 如果将整个企业级 AI 的治理与降本战略浓缩为四条核心执行原则，它们分别是： 1. **精准归属每一个 Token（Attribute every token）**：建立细粒度的计量与审计机制，明确每笔调用的团队与业务归属。 2. **智能路由并克制模型升级（Route to the right model, do not upgrade every time）**：根据具体任务难度将请求分发给最具性价比的模型，避免盲目随主流升级至成本昂贵的最强旗舰大模型。 3. **规模化时掌控模型层（Own the model layer whenever it makes sense for your scale）**：当调用规模达到临界点时，积极采用私有化部署的开源模型以获得确定性的成本优势。 4. **重金资助核心标杆场景（Fund the anchor use cases, not just the pilots）**：停止在数百个分散的概念验证（PoC）小项目中浪费资源，集中预算打造能产生数千万至数亿美元业务价值的标杆用例。 TrueFoundry 团队正致力于为全球大型企业构建这一套 AI 控制平面（Control Plane），帮助企业在加速创新与严格风控之间取得最佳平衡。 ## Notable quotes > [03:19](https://www.youtube.com/watch?v=HezMNx2NZT4&t=199s) 在拉斯维加斯街角的小教堂里，两个人结个婚都用不了 90 秒。 — Nikunj Bajaj > [07:08](https://www.youtube.com/watch?v=HezMNx2NZT4&t=428s) 我想续订我的心脏病药 — 电话客户 > [08:21](https://www.youtube.com/watch?v=HezMNx2NZT4&t=501s) 你不会为了给手机充电而去专门建造一座发电厂；你建造发电厂是为了驱动整个工厂运转，而整栋大楼里的所有手机顺便都能免费充电。 — Nikunj Bajaj > [10:27](https://www.youtube.com/watch?v=HezMNx2NZT4&t=627s) 最便宜的 Token 永远是那些根本不需要发给模型的 Token — Nikunj Bajaj > [10:27](https://www.youtube.com/watch?v=HezMNx2NZT4&t=627s) 本次训练运行将花费 4,500 美元 — Meta 内部系统提示 > [13:19](https://www.youtube.com/watch?v=HezMNx2NZT4&t=799s) 我是否可以先用更小的数据集跑通流程？能否通过缩减超参数或模型参数来控制开销？ — Nikunj Bajaj > [14:43](https://www.youtube.com/watch?v=HezMNx2NZT4&t=883s) 护栏先行（Guardrails First）往往会导致项目彻底难产、一件产品都无法交付。 — Nikunj Bajaj > [15:29](https://www.youtube.com/watch?v=HezMNx2NZT4&t=929s) 精准归属每一个 Token（Attribute every token） — Nikunj Bajaj > [15:29](https://www.youtube.com/watch?v=HezMNx2NZT4&t=929s) 智能路由并克制模型升级（Route to the right model, do not upgrade every time） — Nikunj Bajaj > [15:29](https://www.youtube.com/watch?v=HezMNx2NZT4&t=929s) 规模化时掌控模型层（Own the model layer whenever it makes sense for your scale） — Nikunj Bajaj > [15:29](https://www.youtube.com/watch?v=HezMNx2NZT4&t=929s) 重金资助核心标杆场景（Fund the anchor use cases, not just the pilots） — Nikunj Bajaj ## Mentioned Nikunj Bajaj, TrueFoundry, Ai4, CVS, Siemens Healthineers, Meta, MIT, Gartner ## Video description Nikunj Bajaj - Co-founder & CEO, TrueFoundry Your engineering teams are shipping agents faster than anyone can track them. Token bills climb with no clear attribution, and when an agent loops or breaks, there's no way to audit the spend or shut it down. Most enterprises got here following the "start small" advice, which quietly produces hundreds of disconnected pilots, GPU waste, and no shared governance. Nikunj Bajaj, CEO of TrueFoundry, flips the script and makes the case for starting big. Drawing on production deployments at large global enterprise customers (CVS and Siemens Healthineers) that drove tens to hundreds of millions in impact, he shares the concrete plays used to rein in token spend and maximize ROI. [Source](https://www.youtube.com/watch?v=HezMNx2NZT4)

Source quotes

[3:19] Nikunj Bajaj: 在拉斯维加斯街角的小教堂里，两个人结个婚都用不了 90 秒。 [7:08] 电话客户: 我想续订我的心脏病药 [8:21] Nikunj Bajaj: 你不会为了给手机充电而去专门建造一座发电厂；你建造发电厂是为了驱动整个工厂运转，而整栋大楼里的所有手机顺便都能免费充电。 [10:27] Nikunj Bajaj: 最便宜的 Token 永远是那些根本不需要发给模型的 Token [10:27] Meta 内部系统提示: 本次训练运行将花费 4,500 美元 [13:19] Nikunj Bajaj: 我是否可以先用更小的数据集跑通流程？能否通过缩减超参数或模型参数来控制开销？ [14:43] Nikunj Bajaj: 护栏先行（Guardrails First）往往会导致项目彻底难产、一件产品都无法交付。 [15:29] Nikunj Bajaj: 精准归属每一个 Token（Attribute every token） [15:29] Nikunj Bajaj: 智能路由并克制模型升级（Route to the right model, do not upgrade every time） [15:29] Nikunj Bajaj: 规模化时掌控模型层（Own the model layer whenever it makes sense for your scale） [15:29] Nikunj Bajaj: 重金资助核心标杆场景（Fund the anchor use cases, not just the pilots）

Source quotes

[1:35] Nikunj Bajaj: 90 seconds is long enough for people to get married in a Las Vegas chapel down the street [7:08] Nikunj Bajaj: Refill my heart medicine. [9:05] Nikunj Bajaj: You don't build a power plant to charge your mobile phone... You build it to run the factory and then all the phones in the building charge for free. [10:27] Nikunj Bajaj: the cheapest token is obviously the one that you never have to send to a model. [5:00] Nikunj Bajaj: start small [10:27] Nikunj Bajaj: become the house [10:27] Nikunj Bajaj: $4,500 for this run [14:23] Nikunj Bajaj: pull the trigger

Original transcript

[0:04] Okay, we'll start with asking a couple of quick  questions. How many in this room have heard or [0:12] are upset about the overall AI cost being a  problem? All right, good. We have close to 50%. [0:24] How many in this room have also talked about that  there's lots of prototypes, pilots happening with [0:29] AI but no meaningful business impact? How many  people have heard that or facing that problem? [0:35] Okay, another 50%. How many in this room  lost some money at the poker table yesterday? [0:44] All right. So hopefully what is it? [laughter]  So got no money for the poker table, right? [0:53] Um so hopefully by the end of this talk we  would have solved all those three problems. [0:58] Um so we'll talk about how one of our enterprise  customers shipped a real business use case [1:08] which is external facing actual customerf  facing um where they were quoted $300 million [1:15] to build out that use case that they eventually  ended up shipping for about $15 million. Okay, [1:21] this is a healthcare enterprise, Fortune50. I  won't name them, but I am confident that every [1:28] single one of us in this room have at some  point use their services. So, let's jump in. [1:35] Um, before generative AI was such a big  deal, right? uh this healthcare enterprise [1:42] had phone line to their stores and they receive  approximately 500 million phone calls each year. [1:50] So pretty large scale and across the 9,000 stores  for the 500 million phone calls they were spending [1:58] about 2 to3 million in the software cost okay  to run that IVR system they were spending about [2:04] $2 to3 million. Now, of course, this is  the old school IVR system. If you want X, [2:09] press one. If you want Y, press two. Type of the  IVR system, right? And then they were ambitious. [2:16] They wanted to modernize it with LLMs. As 2023  came in, people wanted to use LLMs and build [2:23] agentic systems. Um, they had ambitions  to modernize this entire IVR system. [2:31] They went to the model providers and um  unforgivingly uh they gave a quote of more [2:38] than $300 million. The model provider for one year  to build this one use case gave them a quote of [2:46] north of $300 million. They shopped around. Sure  the cost came down a little bit but was still [2:52] more than $100 million net cost uh that  they would have incurred. Of course, as any [3:00] um reasonable leader, um this number just does  not uh float for anyone, right? So, they decided [3:07] uh to not go with direct model providers and use  this uh use these frontier models to build out [3:14] the overall system. And we'll talk through  how they solve the problem. But turns out [3:19] that actually cost was only half the problem.  Okay. When they built the initial prototype with [3:25] the Frontier models, it took over 90 seconds  to respond to customer queries. 90 seconds. [3:33] Actually, in less than that, in less than that  time frame, a couple blocks down the line, [3:39] people actually get married in Vegas at the  chapels, right? So, the expected time duration was [3:46] u that they want the system to respond in less  than 2 seconds and they were investing 90 seconds. [3:52] Um and turns out that this enterprise was  not the only one facing this challenge. [3:59] MID actually ran a um survey where they they  mentioned that 95% of all of the pilots that [4:05] are being built in genative AI has exactly zero  returns. Um and more than 40% of the overall [4:12] genative AI projects that are being run will  likely get cancelled is predicted by Gartner. [4:17] Right? And I'm sure that all of us sitting here in  the room have faced this in our own organizations [4:22] as well. Right? So this is a systemic problem  that the entire industry is trying to solve. Okay? [4:31] Um and one thing is very clear uh  based on what we saw with this customer [4:36] uh that if you try to use your frontier models  for every single task in an agentic application [4:43] and you do that half a billion times, [4:48] you are looking at hundreds of  millions of dollar worth of API cost [4:52] and you would have no money left,  sir, to go invest in poker, right? [5:00] So there's one other systemic  problem that we want to talk about um [5:05] that gets in the way of building agent  applications that actually deliver business value. [5:11] And this advice is actually counterintuitive to  the to the traditional wisdom that we all know of, [5:17] right? Which is what which is start small, right?  Every team within the entire enterprise comes up [5:23] with their own pilots. What are the workflows that  they all can automate in their own little teams? [5:28] basically right and like that you basically  have 200 pilots running within the enterprise [5:35] none of those pilots as you're seeing  on the left side generate enough returns [5:41] that you can invest into building a plumbing layer  that makes shipping generative AI easier right the [5:48] overall business impact of these pilots is so low  that it just does not make sense to take that bet [5:54] right it's basically like those slot machines in  the casino You know, every little bet feels small, [6:01] but the house is counting on us that  we're going to make enough number of bets [6:05] and they will keep making money.  Guess who is the house here? Um, [6:12] the counterintuitive advice uh that we have  on this and we have seen this as a pattern [6:17] with a lot of our customers is don't start small.  Actually figure out large anchor use cases, maybe [6:23] five of them, okay? and pick a number relative to  your business scale, right? What business value [6:30] is reasonable to your scale of business to justify  building a plumbing layer. The kind of customers [6:36] that we are working with, they typically need  a $100 million or above in the business impact, [6:42] right? So some of our successful customers  chose five such use cases any one of which [6:48] is sufficient to build out the entire plumbing  layer and then everything else actually just [6:53] works on the top of those that plumbing layer  um without needing to reinvent the wheel right [7:00] and once you do that then your meaningful  generative AI journey actually kicks off. [7:08] So about 6 months later um after we  started to work with this customer. Okay. [7:15] Um the phone the same phone line that  was serving the old school IVR system [7:22] then got migrated to a completely  agentic system. A 90-year-old lady [7:30] phone called asked to refill her heart medicine  and that's it. That's the only prompt she sent [7:40] and the agentic system was able to understand  what is this heart medicine she is referring to. [7:45] When was this prescribed? Does she have enough  refills uh that she is able to place an order [7:53] completely place the order and respond  back to the patient in under two seconds. [7:58] And this was launched in production in under 6  months. And then in an overall one and a half year [8:05] time frame, this was live across all of the  9,000 stores in the United States. Um, and [8:14] we'll talk about what are the other implications  that came once the system was lived. [8:21] So first of all, the most important thing, do we  have any people who work in finance in this room? [8:28] Okay, we have a handful of people. This these  are the only two numbers that you care about. uh [8:33] $300 million was the quote uh to build this  system. $15 million is what it costed to [8:39] bring this system to life and now it's running  at a recurring cost of $15 million per year [8:45] and a lot of added benefits.  Let's talk through those. [8:51] So this system went to life um went live  at 95% less cost but then a lot of other [9:01] important benefits came out. Number one, [9:05] it's like you know how they say you don't build  a power plant to charge your mobile phone, [9:09] right? You build it to run the factory and then  all the phones in the building charge for free, [9:15] right? It's the same thing. people built out this  entire plumbing layer based on this one use case [9:22] and then within about a year or so they had more  than 15 generative AI agent AI use cases live in [9:31] production each of which was generating more than  $10 million in business impact and dozens others [9:38] that were generating like a million dollar a few  hundred,000 in business impacts. All those pilots [9:43] that we talked about started going to production  in the order of days or weeks and not months or [9:49] years. Okay. Um and all of this happened while  the business metric continued to improve. Right. [9:57] So what we are seeing is it's not just that the  cost went down the actual containment metric [10:03] for their IVR system improved further. Okay.  Um and they ended up building out a system [10:09] where they can always stay current. So overall  it they ended up taking less than 24 hours to [10:16] catch up to any new models that are relevant  that is relevant in their own ecosystem. [10:20] Right? So they basically got an entire platform  that helps them scale up their agent journey. [10:27] Breaking this down a little bit, where  are the cost savings coming from? [10:35] You know in Vegas there is a rule  that the house always wins, right? [10:41] The house being the model providers will always  win in the world of agent AI, right? You don't [10:48] want to put all your eggs in one basket. So  that's why you become the house. That's the answer [10:55] to how you solve the cost problem meaningfully,  right? Once you're running systems at scale, [11:00] you start owning up at least you start directing  some part of your traffic to open source models [11:06] and then you start doing tiered  routing. A very complex query, [11:10] send it to frontier model all day long.  If these are simple repeatable queries, [11:15] send it to cheaper models and build  it build an automated system to run [11:20] um your agenti applications. Make sure that if the  model has answered a question, you're not asking [11:27] it the same question again. You're caching it,  pre-fetching it. Build the architecture, right? [11:34] And then we had a bunch of other cost  savings that came from budget caps [11:37] policy engine that they ended up building out  uh and then the GP autoscaling uh bits of it. [11:43] By the way, the budget caps in this case is  actually a small percentage of cost savings [11:48] because for them this was the major use case that  they were going after. Now that we have put dozens [11:53] of similar use cases with a bunch of our customers  in production, we are seeing just avoiding [11:58] human errors often result 20 to 50% of the cost  savings. Uh which is not reflected in this slide. [12:07] So we talked about five levers and all of these  five levers are achievable by building what we [12:15] call as a single control plane. One control  plane for your agenti applications. Okay. [12:23] Um, and all of these pointers that we talked  about are basically achievable through this lever. [12:31] So net net what we are seeing with this customer  is on an average nine out of the 10 calls [12:36] actually never reach the model never reach the  frontier model right um because of rules caching [12:44] being able to route queries to small models only  approximately 9% of the overall agentic calls [12:50] hit a frontier model and by the way the cheapest  token is obviously the one that you never have [12:57] to send to a model right so that's that's where  you save the Then every token carries a name. [13:02] Like once you start building out a  system where you have each token tagged, [13:08] we have seen this behavior. I used to be at Meta  before uh we started up and we had a internal [13:13] platform that before I trigger a training run, it  would show me this training run cost you $4,500. [13:19] And as a developer, I would question myself, can  I run this initially with a smaller set of data? [13:25] Can I run this with smaller number of parameters  or maybe hyperparameters and that just put [13:30] the cost in check. The same logic applies in  generative AI. You start tagging a token to a team [13:37] and when they see their name on the leaderboard,  they realize that I can bring the cost down, [13:41] right? So there's a behavioral change that  happens in the system. We talked a little bit [13:47] about how open source models on their own GPUs  like you know return a lot of cost yield. Okay. [13:56] Um then we had mentioned initially that there are  these there's this latency problem that people had [14:02] to solve right. Um and there's a lot of different  techniques uh like pre-fetching the overall [14:09] uh data sets uh being able to reuse some of the  queries by caching running guardrails in parallel. [14:16] All of this can be executed by the gateway  layer. Okay. Um and using your cache. All of this [14:23] can help you save the latency. Every agent that  you build, you put a budget limit on it, right? [14:30] And a kill switch. If an agent goes rogue, just  pull the trigger, kill the agent, route the query [14:37] to some fallback rulebased systems. Okay, every  agent in the organization must have a kill switch. [14:43] So these are some of the learnings. And the last  one that I'll share in terms of learnings is [14:50] organizations that start with guardrails  first are usually not able to ship anything. [14:57] And this is me selling to large enterprises  and regulated industries saying this because [15:04] if you build guardrails first then you have 300  different ways in which your applications can [15:09] break and you will end up trying to fix all of  those things. The right way to do this is start [15:14] with a use case. Figure out what guardrails are  most relevant. Build those guardrails and then [15:19] build a plumbing layer that enables you to extend  to further guardrails. But don't put the card [15:24] before the horse basically right. First figure  out the use case then figure out the guardrails. [15:29] So netnet if you had four things to take out  of this uh presentation. Attribute every token. [15:38] Route it to the right model. Do not upgrade the  model every time. Own the model layer whenever it [15:44] makes sense for your scale. And lastly, fund  the anchor use cases, not just the pilots. [15:53] If you all care to learn more about it, uh we are  at booth number 322 and we would love to chat with [15:58] you. We are team True Foundry uh building  control plane for enterprise AI. Thank you.