Not Diamond Cuts Enterprise AI Agent Costs 40% Via Predictive Routing

Intelligent Model Routing: Frontier Performance Without Frontier Bills With Not Diamond — Ai4

Ai4

Intelligent model routing lowers enterprise inference spending by treating multi-turn prompt selection as a robotics problem that preserves cache state and avoids frontier pricing.

AI-assisted summary of Intelligent Model Routing: Frontier Performance Without Frontier Bills With Not Diamond — original source. Check the original for full context.

Source published: 2026-08-16

Detailed notes

- [00:04](https://www.youtube.com/watch?v=5YdCKw2kAYI&t=4s) Tomás Hernando Kofman introduces Not Diamond and its focus on intelligent model routing to optimize performance and reduce costs for enterprise coding agents. ## Key takeaways - [00:04](https://www.youtube.com/watch?v=5YdCKw2kAYI&t=4s) Enterprise AI inference spend is scaling massively, leading engineering organizations to deplete annual coding budgets in matter of months or weeks. - [02:15](https://www.youtube.com/watch?v=5YdCKw2kAYI&t=135s) Rising costs for frontier models, continuous autonomous execution, and broader developer adoption drive rapidly growing inference expenditures. - [02:15](https://www.youtube.com/watch?v=5YdCKw2kAYI&t=135s) A strong model arbitrage opportunity exists because fast-improving small and open-weight models deliver frontier-equivalent capabilities at lower prices. - [05:05](https://www.youtube.com/watch?v=5YdCKw2kAYI&t=305s) Intelligent routing predictively chooses the optimal model for each prompt, whereas gateways primarily handle connectivity and governance across model APIs. - [05:05](https://www.youtube.com/watch?v=5YdCKw2kAYI&t=305s) Simple or naive routing techniques like heuristics, semantic embeddings, or LLM-based decisions fail to handle long-horizon agent interactions effectively. - [05:05](https://www.youtube.com/watch?v=5YdCKw2kAYI&t=305s) Not Diamond frames model routing as a robotics problem in a long-horizon environment to optimize costs and rewards across multi-turn sessions. - [09:01](https://www.youtube.com/watch?v=5YdCKw2kAYI&t=541s) Key factors influencing agentic costs include model selection, reasoning effort levels, KV cache preservation, and context window compaction. - [09:01](https://www.youtube.com/watch?v=5YdCKw2kAYI&t=541s) Invalidating the KV cache by switching models mid-session can eliminate expected cost savings due to forced full context rewrites. - [12:09](https://www.youtube.com/watch?v=5YdCKw2kAYI&t=729s) Not Diamond operates as a telemetry layer without inspecting prompt payloads, ensuring user code and data remain strictly local and private. - [16:11](https://www.youtube.com/watch?v=5YdCKw2kAYI&t=971s) Applying Jevons' Paradox to AI, cheaper and more efficient inference will increase overall usage across knowledge and background agents over time. - [16:11](https://www.youtube.com/watch?v=5YdCKw2kAYI&t=971s) Model routing prevents single-vendor lock-in, shifts pricing power back to enterprises, and reduces global compute and financial waste.

## Detailed walkthrough ### [00:04](https://www.youtube.com/watch?v=5YdCKw2kAYI&t=4s) Introduction and the Soaring Costs of Enterprise AI

Tomás Hernando Kofman, co-founder and CEO of Not Diamond, opens his presentation by introducing his company's focus: intelligent model routing specifically tailored for coding agents. He defines intelligent model routing as the ability to predictively determine when to deploy specific Large Language Models (LLMs) to maximize accuracy while minimizing costs. As new models launch on a weekly basis with vastly different price points and performance profiles, deciding which model to invoke for any given task becomes a critical operational challenge.

Kofman highlights the deep background of the Not Diamond team, noting that they are serial founders and AI researchers who previously published foundational papers introducing Retrieval-Augmented Generation (RAG) and function calling. Backed by prominent AI scientists and founders, Not Diamond powers auto-routing for major platforms such as OpenRouter and works with large enterprise clients like SAP.

Transitioning to the underlying problem, Kofman emphasizes that enterprise AI inference spending has surged tenfold over the past year and is on track to scale by another 10x. Enterprise engineering organizations are rapidly exhausting their annual coding agent budgets in months or even weeks. Kofman references a real-world example where one of the world's largest banks burned through its entire monthly Claude Code budget in a single afternoon.

- **Background & Credentials:** Team members invented function calling and introduced RAG; investors and partners include major AI founders, OpenRouter, and SAP. - **Inference Cost Surge:** Enterprise inference spend grew 10x year-over-year and is projected to grow another 10x. - **Budget Burn Example:** A major global bank depleted its entire monthly budget for Claude Code in just one afternoon.

---

[02:15](https://www.youtube.com/watch?v=5YdCKw2kAYI&t=135s) Key Trends Driving Inference Spend and Model Arbitrage

Kofman details three key macro trends accelerating inference expenditure across organizations:

1. **Increasing Frontier Costs:** Frontier models (such as "Fable GB 5.6 Soul") continue to grow more expensive as compute demands scale alongside performance gains. 2. **Autonomous Execution Scaling:** Incremental performance improvements allow models to run far longer and execute parallel, autonomous workloads. For instance, when Claude 4.5 Opus was released, spend skyrocketed 10x—not because the token price was 10x higher, but because the model was reliable enough to run autonomously overnight. 3. **Broadening User Adoption:** While AI adoption reaches 80%–95% within engineering orgs, total spend is still disproportionately driven by power users, leaving significant room for further growth as average developers ramp up usage.

At the same time, open-source and smaller models are catching up to frontier capabilities at a fraction of the cost. Kofman points to models like GLM-5.2 (approximating Sonnet or Opus), Kimi k3 (surpassing Opus and nearing Fable), and DeepSeek-V4-Flash as high-performing, cost-effective options. This gap between expensive frontier models and powerful low-cost options creates a durable "model arbitrage" opportunity.

- **Claude 4.5 Opus Spend Impact:** Driven by continuous overnight autonomous runs rather than direct unit price increases. - **High-Performing Budget Models:** GLM-5.2, Kimi k3, and DeepSeek-V4-Flash offer frontier-like capabilities at significantly lower price points.

---

[05:05](https://www.youtube.com/watch?v=5YdCKw2kAYI&t=305s) Defining Intelligent Model Routing vs. Gateways

Intelligent model routing is defined as automatically selecting the optimal model for each individual prompt or request to balance accuracy and expense. Kofman clarifies the distinct difference between a gateway and a router: - **Gateways** (e.g., OpenRouter, LiteLLM, Portkey) govern access, routing protocols, and infrastructure connections across multiple model APIs. - **Intelligent Routers** predictively decide *which* specific model to invoke for each turn or request.

Kofman notes that building an effective router is "deceptively difficult." Early or naive approaches often turn out to be architectural dead ends, including: - Heuristic `if/else` logic - Semantic routing based purely on query embedding space - Using an LLM call to route another LLM call - Training simple complexity classifiers - Cascade routing (trying a small model first, then falling back to a larger model on failure) - Session-based static routing (basing the entire agent trajectory on the first message)

Rather than treating routing as a single classification task, Not Diamond frames it like a **robotics problem**: a long-horizon environment with sparse rewards. The agent state serves as the environment, while the available models and reasoning effort levels form the action space. The router must predict future costs and rewards across 3, 5, 10, or 100 interaction turns before the final reward is realized upon session completion.

- **Gateways vs. Routers:** Gateways handle management/governance; routers make predictive intelligence decisions per request. - **Performance Benchmark:** On TerminalBench, Not Diamond achieved performance equivalent to Claude 4.8 Opus (Extra High reasoning) while delivering a **30% cost reduction**. - **Enterprise Savings:** Fortune 500 engineering teams achieve 20% to 40%+ cost savings using Anthropic models alone (routing between Opus, Sonnet, and Haiku), with savings nearly doubling when open-weight models are integrated.

---

[09:01](https://www.youtube.com/watch?v=5YdCKw2kAYI&t=541s) Cost Drivers in Agentic Environments

Kofman outlines four primary levers influencing cost and performance in agentic applications:

1. **Model Selection:** Defaulting to top-tier frontier models for trivial tasks wastes budget, but selecting an underpowered small model can be equally expensive if it gets stuck in endless loops or requires costly bug fixes later. 2. **Reasoning Effort Levels:** Distinctions between reasoning settings (e.g., Opus Low vs. Medium, or Sonnet High vs. Max) dramatically impact output token volume, which compounds on every subsequent conversation turn. 3. **KV Cache Economics:** Reusing context via Key-Value (KV) caching heavily discounts input costs. Switching models mid-session invalidates the existing cache, forcing a complete cache rewrite that can negate the savings of a cheaper model. 4. **Context Window Expansion & Compaction:** Unmanaged context context growth triggers auto-compaction—an expensive process that consumes entire context windows and degrades model performance as window saturation increases.

Compounding these technical variables is the fact that the underlying ecosystem is a moving target: new models launch weekly, harnesses update, and pricing structures evolve continuously.

- **Key Cost Drivers:** Model choice, reasoning effort configurations, KV cache hit rates, and context window compaction frequency. - **Cache Trade-off:** Switching models mid-session destroys KV cache state, potentially increasing overall spend despite choosing a lower-cost model.

---

[12:09](https://www.youtube.com/watch?v=5YdCKw2kAYI&t=729s) Real-Time Optimization and Architecture Walkthrough

To tackle these challenges, Not Diamond optimizes every turn of an agent session by evaluating model choice, reasoning effort, KV cache status, and task boundaries. For example, when developers keep a single session running indefinitely across distinct tasks, Not Diamond detects task changes (e.g., moving from Task A to Task B) and triggers targeted context compaction. Furthermore, the system incorporates implicit developer feedback—such as when a developer accepts a suggestion or manually overrides a model choice—to continuously adjust its real-time routing recommendations tailored to individual user preferences.

``` +--------------------------------------------------------------------+ | Local Developer Machine | | | | +--------------------+ Anonymized Metadata | | | Agent Harness | ---------------------------------+ | | | (Claude Code, etc) | | | | +--------------------+ v | | ^ +------------------+| | | Model & Reasoning Recommendation | Local Proxy || | +---------------------------------- | (Cache Monitor) || | +------------------+| +----------------------------------------------------------|---------+ | Metadata v +-----------------------+ +------------------+ | Existing Gateway | <---------------------- | Remote Server | | (Anthropic / LiteLLM) | Routed Execution | (Router Engine) | +-----------------------+ +------------------+ | v +-----------------------+ | Model Endpoints / VPC | +-----------------------+ ```

Kofman explains that a key privacy benefit of this architecture is that Not Diamond never sees user prompt payloads, inputs, or outputs. The local proxy extracts only privacy-preserving telemetry metadata—message counts, token counts, local complexity scores, developer feedback indicators, and KV cache states—meaning Not Diamond operates purely as a telemetry layer rather than a data subprocessor.

- **Optimization Levers:** Turn-by-turn routing, KV cache tracking, task boundary detection, and continuous real-time learning from user feedback. - **Privacy Design:** Payloads and code remain entirely local/private; only derived telemetry metadata is transmitted for routing decisions. - **Infrastructure Integrations:** Works alongside existing enterprise gateways or dedicated private VPC inference endpoints optimized for open-weight models.

---

[16:11](https://www.youtube.com/watch?v=5YdCKw2kAYI&t=971s) Future Horizon and Industry Vision

Looking beyond software development, Kofman argues that coding agents are merely the first wave of agentic deployment. Knowledge workers across enterprise functions are rapidly adopting knowledge agents, which will soon be followed by massive deployments of autonomous background agents running continuously without human intervention to manage data pipelines and system monitoring.

Referencing **Jevons' Paradox**—which states that increased efficiency in resource usage leads to an overall increase in consumption rather than a decrease—Kofman predicts that cheaper, more accessible inference will exponentially drive up total AI workload volume over the next 6 to 18 months.

Kofman closes by outlining three key reasons why model routing is essential for the future of AI: 1. **Preventing Monopolization:** It prevents reliance on a single monolithic provider by lowering switching costs to zero and promoting a diverse model ecosystem. 2. **Empowering Consumers:** It shifts pricing power and control away from model producers and AI research labs back to enterprises and individual consumers. 3. **Maximizing Efficiency:** It eliminates environmental, compute, and financial waste across global AI infrastructure.

- **Jevons' Paradox:** Reductions in inference cost will expand overall agent usage across enterprise knowledge and background workloads over the next 6–18 months. - **Closing Contact:** Tomás Hernando Kofman shared his Twitter/X handle and email, inviting attendees to meet outside for further discussion.

## Notable quotes > [00:04](https://www.youtube.com/watch?v=5YdCKw2kAYI&t=4s) Intelligent model routing is the ability to predictively determine when to deploy specific Large Language Models to maximize accuracy while minimizing costs. — Tomás Hernando Kofman

> [00:04](https://www.youtube.com/watch?v=5YdCKw2kAYI&t=4s) Enterprise AI inference spending has surged tenfold over the past year and is on track to scale by another 10x. — Tomás Hernando Kofman

> [00:04](https://www.youtube.com/watch?v=5YdCKw2kAYI&t=4s) One of the world's largest banks burned through its entire monthly Claude Code budget in a single afternoon. — Tomás Hernando Kofman

> [02:15](https://www.youtube.com/watch?v=5YdCKw2kAYI&t=135s) Frontier models continue to grow more expensive as compute demands scale alongside performance gains. — Tomás Hernando Kofman

> [02:15](https://www.youtube.com/watch?v=5YdCKw2kAYI&t=135s) Spend skyrocketed 10x—not because the token price was 10x higher, but because the model was reliable enough to run autonomously overnight. — Tomás Hernando Kofman

> [05:05](https://www.youtube.com/watch?v=5YdCKw2kAYI&t=305s) Gateways govern access, routing protocols, and infrastructure connections across multiple model APIs, while intelligent routers predictively decide which specific model to invoke for each request. — Tomás Hernando Kofman

> [05:05](https://www.youtube.com/watch?v=5YdCKw2kAYI&t=305s) Building an effective router is deceptively difficult. — Tomás Hernando Kofman

> [05:05](https://www.youtube.com/watch?v=5YdCKw2kAYI&t=305s) Not Diamond frames routing like a robotics problem: a long-horizon environment with sparse rewards. — Tomás Hernando Kofman

> [09:01](https://www.youtube.com/watch?v=5YdCKw2kAYI&t=541s) Selecting an underpowered small model can be equally expensive if it gets stuck in endless loops or requires costly bug fixes later. — Tomás Hernando Kofman

> [09:01](https://www.youtube.com/watch?v=5YdCKw2kAYI&t=541s) Switching models mid-session invalidates the existing cache, forcing a complete cache rewrite that can negate the savings of a cheaper model. — Tomás Hernando Kofman

> [12:09](https://www.youtube.com/watch?v=5YdCKw2kAYI&t=729s) Not Diamond never sees user prompt payloads, inputs, or outputs. — Tomás Hernando Kofman

> [16:11](https://www.youtube.com/watch?v=5YdCKw2kAYI&t=971s) Coding agents are merely the first wave of agentic deployment. — Tomás Hernando Kofman

> [16:11](https://www.youtube.com/watch?v=5YdCKw2kAYI&t=971s) Model routing shifts pricing power and control away from model producers and AI research labs back to enterprises and individual consumers. — Tomás Hernando Kofman

## Mentioned Ai4, Tomás Hernando Kofman, Not Diamond, OpenRouter, SAP, Claude Code, Fable GB 5.6 Soul, Claude 4.5 Opus, GLM-5.2, Sonnet, Opus, Kimi k3, DeepSeek-V4-Flash, LiteLLM, Portkey, TerminalBench, Claude 4.8 Opus, Haiku, Anthropic, Jevons' Paradox, Twitter/X

## Video description Tomás Hernando Kofman - CEO, Not Diamond

Enterprises are burning through AI coding agent budgets in months. In this talk, Not Diamond CEO Tomás Hernando Kofman explains how intelligent model routing cuts coding agent costs by 20%+ while maintaining frontier quality, automatically identifying when a prompt needs a powerful model and when a cheaper one is sufficient. He'll cover how routing differs from gateways, why naive approaches backfire, and what model routing means for the future of AI.

[Source](https://www.youtube.com/watch?v=5YdCKw2kAYI)

中文简报

Not Diamond通过预测性路由将企业AI Agent成本削减40% 智能模型路由将多轮提示词选择视为机器人问题，通过保留缓存状态并规避前沿模型定价来降低企业推理开销。 Tomás Hernando Kofman介绍了Not Diamond及其用于优化企业编码Agent性能并降低成本的智能模型路由技术。当前，前沿模型费用上涨、持续自主执行及更广泛的开发者采用导致企业AI推理支出急剧增加，许多工程团队在数月甚至数周内便耗尽年度预算。由于快速发展的小型与开源权重模型能以更低价格提供媲美前沿模型的能力，从而带来了显著的模型套利机会；不同于仅负责API连接与治理的模型网关，智能路由能够针对每个提示词预测性地挑选最佳模型。 Not Diamond 联合创始人兼 CEO Tomás Hernando Kofman 指出，企业 AI 推理支出同比激增 10 倍，导致研发团队在数周或数月内就耗尽全年的编程 Agent 预算。 (0:04) 前沿模型单价上涨、自主执行时间延长以及开发者普及推动了成本飙升；同时，开源和较小模型性能逼近前沿水平，为模型套利创造了持久机会。 (2:15) 智能路由不同于仅负责连接与治理的网关，它将决策视作长期限的类机器人问题，在 TerminalBench 上实现与 Claude 4.8 Opus 相当性能的同时降低了 30% 的成本。 (5:05) Agent 的成本受模型选择、推理努力程度、KV Cache 保留及上下文压缩共同影响；Not Diamond 通过本地代理仅提取遥测元数据进行实时路由，有效保障了用户代码隐私。 (9:01) 基于杰文斯悖论（Jevons' Paradox），推理效率的提升将在未来 6 到 18 个月推动后台自主 Agent 工作负载呈指数级爆发，使模型路由成为平衡成本与打破供应商垄断的关键基础设施。 (16:11)

Reading summary

Tomás Hernando Kofman argues that surging enterprise AI inference costs can be curtailed through intelligent model routing, which dynamically balances frontier and lower-cost models across multi-turn agent workflows without compromising performance or data privacy. Escalating Agent Spend and Model Arbitrage (0:04) Enterprise inference spend has expanded tenfold year-over-year and is projected to grow by another 10x, with engineering teams rapidly exhausting allocations—including one major bank that depleted its monthly Claude Code budget in a single afternoon. Kofman attributes this surge to increasingly expensive frontier compute, expanding developer adoption, and the rise of autonomous workloads, noting that spend increased tenfold following the release of Claude 4.5 Opus due to continuous overnight execution rather than direct token price increases. At the same time, high-performing alternative models such as GLM-5.2, Kimi k3, and DeepSeek-V4-Flash are closing the capability gap with frontier models at substantially lower prices. This divergence creates an opportunity for enterprise "model arbitrage," allowing organizations to capture top-tier performance by substituting lower-cost models wherever feasible. Predictive Routing as a Robotics Problem (5:05) While API gateways govern infrastructure connections and routing protocols, intelligent routers dynamically decide which specific model to deploy for individual prompts. Kofman cautions that naive approaches—such as heuristic rules, embedding-based semantic routing, LLM-driven routing, simple classifiers, and cascade fallbacks—represent architectural dead ends. Instead, Not Diamond frames routing as a robotics problem characterized by long horizons and sparse rewards, predicting costs and performance across dozens or hundreds of conversational turns before a task completes. On the TerminalBench benchmark, this approach reportedly matched the performance of Claude 4.8 Opus with extra-high reasoning while reducing costs by 30%. According to Kofman, Fortune 500 engineering organizations realize 20% to 40%+ savings when routing strictly within Anthropic models (Opus, Sonnet, and Haiku), with cost reductions nearly doubling when open-weight models are integrated into the action space. Managing Reasoning, Caching, and Compaction Levers (9:01) Optimizing coding agents requires navigating interconnected technical variables where naive model switching can backfire. Beyond primary model selection, reasoning effort levels directly dictate output token volume, which compounds context size across all future turns. Furthermore, Key-Value (KV) cache economics heavily discount repeated prompt contexts; switching models mid-session invalidates the existing KV cache, forcing a complete cache rewrite whose expense can negate the savings of a cheaper model. Routers must also monitor context window expansion to trigger targeted context compaction when task boundaries change, preventing unmanaged context growth from degrading model accuracy or forcing expensive automated compactions across full context windows. Metadata-Only Architecture and Workload Expansion (12:09) To preserve enterprise data confidentiality, Not Diamond operates as a local telemetry layer rather than a data subprocessor. A local proxy on the developer machine monitors anonymized metadata—such as message counts, cache status, task boundaries, and developer override signals—transmitting only telemetry to the remote router without exposing raw prompts, code, or outputs. This architecture integrates with existing gateways or dedicated VPC endpoints. Looking ahead, Kofman invokes Jevons' Paradox to forecast that reducing inference costs will accelerate overall AI consumption over the next 6 to 18 months as organizations expand beyond coding agents into knowledge workers and continuous background agents for data pipelines and monitoring. Kofman concludes that intelligent model routing is essential to prevent single-vendor monopolization, return pricing power to enterprises, and reduce global compute waste. However, maintaining effective routing remains an ongoing operational challenge because model releases, agent harnesses, and provider pricing structures evolve continuously.

精炼阅读摘要

Not Diamond 联合创始人兼首席执行官 Tomás Hernando Kofman 提出，企业 AI 推理支出正面临数十倍的爆炸性增长，通过智能模型路由在多模型间进行动态套利，可在维持前沿性能的同时显著削减成本。企业推理成本激增与模型套利机会 (0:04) Kofman 指出，过去一年企业 AI 推理支出增长了 10 倍，并预计将再增长 10 倍。许多工程团队在几周甚至几个月内就耗尽了全年的编程智能体预算，例如某全球大型银行在一个下午就用光了其 Claude Code 的整月预算。推动开销激增的因素包括前沿模型（如 Fable GB 5.6 Soul）本身算力成本上升，以及模型可靠性提高后引发的自主长周期运行。以 Claude 4.5 Opus 为例，其支出暴增 10 倍主要归因于能够整夜自主执行任务，而非单价上涨；同时开发者使用率也在逐步推高总体用量。与高昂的前沿模型相比，开源与小型模型正在以极低成本逼近顶尖水平。例如 GLM-5.2 在表现上接近 Sonnet 或 Opus，Kimi k3 表现超过 Opus 并逼近 Fable，DeepSeek-V4-Flash 也提供了高性价比选择。Kofman 认为，昂贵的前沿模型与低成本高性能选项之间的差距，为企业创造了长期存在的“模型套利”空间。智能路由与网关的区别及长周期决策架构 (5:05) Kofman 明确区分了网关（Gateway）与智能路由器（Intelligent Router）：OpenRouter、LiteLLM 或 Portkey 等网关负责跨 API 的访问管理、基础设施连接与协议治理，而智能路由器则在每次交互中预测性地决定调用哪一个具体模型。他指出，启发式 if/else 规则、基于语义嵌入的简单路由、用大模型调度大模型、级联回退或基于会话首条消息的静态路由等朴素方案，在实际架构中均容易陷入死胡同。 Not Diamond 将路由问题视作具备稀疏奖励的长周期“机器人控制问题”：智能体状态构成环境，可选模型及其推理强度构成动作空间。路由器必须在完成会话并获得最终奖励之前，提前预测未来 3 到 100 轮交互的成本与收益。基准测试显示，在 TerminalBench 上，Not Diamond 取得了等同于高推理设置下 Claude 4.8 Opus 的性能，同时降低了 30% 的成本；财富 500 强工程团队仅在 Anthropic 内部模型间路由即可节省 20% 至 40% 以上的费用，若引入开源权重模型，节省幅度几乎翻倍。智能体环境中的四大成本动因与权衡 (9:01) 在智能体应用中，成本控制受制于四个核心技术变量：首先是模型选择，为简单任务调用顶级模型会浪费预算，但选用能力不足的小模型若陷入死循环或产生代码缺陷，后续修复成本同样高昂；其次是推理强度（Reasoning Effort）设置（如 Opus 的 Low 与 Medium，或 Sonnet 的 High 与 Max），推理 token 的增加会在后续对话轮次中产生复利式的成本累积。第三个关键变量是 KV 缓存（KV Cache）经济学。通过缓存复用上下文可以大幅降低输入成本，但如果在会话中途随意切换模型，会导致已有缓存完全失效，重新写入缓存的开销可能会抵消选用低价模型省下的费用；第四则是上下文窗口的膨胀与压缩，未受控的上下文增长会触发代价高昂的自动压缩，这不仅消耗窗口资源，还会随着饱和度提高而降低模型表现。每周推出的新模型和动态变化的定价结构，进一步加剧了优化的复杂性。实时优化架构与零 Payload 隐私设计 (12:09) 为了在运行中权衡上述动因，Not Diamond 在智能体运行的每一轮交互中评估模型选择、推理强度、KV 缓存状态以及任务边界。当开发者在一个长时间会话中从任务 A 切换到任务 B 时，系统会识别任务边界并触发针对性的上下文压缩；同时，系统还会捕捉开发者的隐式反馈（如接受建议或手动覆盖模型选择），以此实时微调路由策略以适配个体偏好。该方案采用隐私保护架构，核心在于 Not Diamond 不接触用户的提示词载荷、输入或输出代码。本地代理（Local Proxy）仅提取去标识化的遥测元数据，包括消息数量、Token 计数、本地复杂度评分、开发者反馈信号和 KV 缓存状态，并将其传送到远程路由引擎。因此，Not Diamond 仅作为遥测决策层运行，不承担数据二级处理者角色，同时能够无缝接入企业现有的 API 网关或私有 VPC 推理节点。 Kofman 认为编程智能体只是第一波浪潮，未来知识型智能体与后台自主智能体将大规模铺开；依据杰文斯悖论（Jevons' Paradox），推理效率的提升和单价下降将在未来 6 至 18 个月内刺激全行业算力需求呈指数级爆发。他主张模型路由技术不仅有助于降低计算资源的无效浪费，更能消除企业对单一垄断供应商的依赖，将定价主导权重新交还给企业与消费者。

Detailed notes — zh

## Background 该视频由 YouTube 频道 Ai4 于 2026 年 8 月 16 日发布，由 Not Diamond 联合创始人兼首席执行官 Tomás Hernando Kofman 带来主题演讲。页面简介指出，企业在 AI 代码智能体上的预算消耗极快，本场演讲系统阐述了智能模型路由如何在维持前沿模型质量的前提下将代码智能体成本降低 20% 以上，并剖析了路由与传统网关的区别、常见朴素方案的缺陷以及模型路由对 AI 未来格局的影响。 ## TL;DR 演讲首先剖析了模型推理支出指数级暴增的驱动因素，并指出依赖单一前沿模型或采用启发式规则、级联回退等朴素路由方案在实践中均属死胡同。Not Diamond 将长程 Agent 调度视作机器人学问题，在每一轮交互中动态预测最优模型、微调推理强度（Reasoning Effort）并最大化维持 KV 缓存，从而在权威基准评测和真实企业场景中实现 20% 到 40% 的支出节省。此外，演讲介绍了其无数据留存的本地代理隐私架构，并结合杰文斯悖论展望了未来后台自主智能体（Background Agents）的爆发以及智能路由在打破巨头生态锁定中的核心使命。 ## Atmosphere & style 整场演讲节奏紧凑、直截了当且极具工程实战洞察，讲者以资深 AI 研究者与架构师的沉稳自信直击大模型落地最昂贵的成本痛点。叙事风格兼具严谨的技术剖析与宏观的产业透视，在拆解行业直觉误区时犀利透彻，展望自主 Agent 未来时充满前瞻性，带给观众兼具实操指导价值与思想启发的高密度信息体验。 ## Key takeaways - [00:04](https://www.youtube.com/watch?v=5YdCKw2kAYI&t=4s) Not Diamond 聚焦于 Coding Agent 的智能模型路由，旨在通过预测性算法在请求级别动态匹配最适合的 LLM，从而在保证前沿质量的前提下大幅降低成本。 - [01:17](https://www.youtube.com/watch?v=5YdCKw2kAYI&t=77s) 企业 AI 推理成本正经历指数级爆炸增长，主要由前沿模型单价攀升、细微能力提升引发长周期自主运行、以及组织内重度用户消耗激增等三大趋势驱动。 - [03:00](https://www.youtube.com/watch?v=5YdCKw2kAYI&t=180s) 开源与轻量模型（如 DeepSeek Flash、GLM、Kimi 等）性能逼近前沿模型，为企业通过动态分流昂贵模型与高效模型创造了巨大的“模型套利”空间。 - [05:05](https://www.youtube.com/watch?v=5YdCKw2kAYI&t=305s) 智能模型路由与仅负责协议接入和流量代理的 API 网关有本质区别，其核心在于在每个交互轮次实时决策调用哪款模型与推理强度。 - [05:30](https://www.youtube.com/watch?v=5YdCKw2kAYI&t=330s) 基于 If-Else 规则、向量语义路由、LLM 裁判、级联回退及首条消息判断等直觉方案多为死胡同，往往会导致高延迟、二次开销或忽略复杂上下文动态演变。 - [07:08](https://www.youtube.com/watch?v=5YdCKw2kAYI&t=428s) 团队将 Coding Agent 的长程调用视作“机器人学问题”，把模型列表与推理强度作为动作空间，在长视距与稀疏奖励环境中优化长周期累积收益与成本。 - [07:45](https://www.youtube.com/watch?v=5YdCKw2kAYI&t=465s) 在 Terminal Bench 基准与 Fortune 500 头部企业实测中，智能路由在效果对齐顶级模型的前提下，稳定实现了 20% 至 40% 以上的直接支出节省。 - [09:01](https://www.youtube.com/watch?v=5YdCKw2kAYI&t=541s) Agent 成本受模型选型、推理强度（Reasoning Effort）、KV 缓存重写开销及上下文自动压缩等多重因素制约，必须在每轮交互中进行多维动态权衡。 - [13:21](https://www.youtube.com/watch?v=5YdCKw2kAYI&t=801s) 通过捕获开发者在终端交互中的纠偏与确认等隐式反馈信号，路由策略能够在线自适应学习并实现个性化调整。 - [13:43](https://www.youtube.com/watch?v=5YdCKw2kAYI&t=823s) 采用轻量本地代理架构，仅向云端发送 Token 计数、复杂度评分等匿名遥测元数据，不接触任何 Prompt 或代码 Payload，实现零数据留存与合规保障。 - [16:11](https://www.youtube.com/watch?v=5YdCKw2kAYI&t=971s) 未来推理需求将向自主运行的“背景 Agent”倾斜，并触发杰文斯悖论——单次推理越便宜，后台算力消耗的总量反而会呈现数十倍的爆发式增长。 - [17:33](https://www.youtube.com/watch?v=5YdCKw2kAYI&t=1053s) 智能模型路由的终极使命是消除跨模型切换成本，打破单一前沿模型实验室的垄断，将技术话语权与生态控制力重新交还给企业与开发者。 ## Detailed walkthrough ### [00:04](https://www.youtube.com/watch?v=5YdCKw2kAYI&t=4s) 智能模型路由与 Not Diamond 的核心使命在开场介绍中，Not Diamond 的联合创始人兼首席执行官 Tomás Hernando Kofman 阐述了公司聚焦的核心方向——专门针对 Coding Agent（代码智能体）的“智能模型路由”（Intelligent Model Routing）。他将智能模型路由定义为一种预测性机制，即通过算法预判何时该调用哪一款大语言模型（LLM），从而在最大化准确率的同时实现成本的最小化。每周都有不同性能与价格特征的新模型问世，如何精准抉择成为企业落地的关键。 Tomás 进一步介绍了 Not Diamond 团队的技术背景与行业积淀。团队成员均由连续创业者与前沿 AI 研究人员组成，其中不乏发明 Function Calling（函数调用）机制以及率先提出 RAG（检索增强生成）范式的核心论文作者。目前，Not Diamond 得到了顶级 AI 科学家和创始人的注资支持，其智能路由引擎已为包括 OpenRouter（在其平台上驱动 Auto-routing 自动路由功能）以及企业级巨头 SAP 在内的高并发 AI 平台与企业提供底层能力支持。 ```mermaid flowchart LR accTitle: Intelligent Model Routing Overview subgraph Input["用户请求与环境"] REQ["Agent 任务请求"] end subgraph RoutingEngine["Not Diamond 智能路由引擎"] ROUTER["预测性决策算法 (最大化准确率 / 最小化成本)"] end subgraph Models["模型池"] FRONTIER["前沿大模型 (Opus / GPT-5 等)"] CHEAP["轻量 / 开源模型 (Flash / Qwen / GLM 等)"] end REQ --> ROUTER ROUTER -->|"复杂推理 / 关键任务"| FRONTIER ROUTER -->|"简单查询 / 工具调用"| CHEAP ``` --- ### [01:17](https://www.youtube.com/watch?v=5YdCKw2kAYI&t=77s) 推理成本爆炸的驱动因素与模型套利机会谈及行业痛点，Tomás 直言当前企业在模型推理（Inference）上的支出正在经历惊人的指数级增长：全行业的推理成本相比一年前增长了约 10 倍，并正在朝着再翻 10 倍的趋势演进。许多大型企业甚至在短短数周或数月内就彻底耗尽了全年的 AI 代码智能体预算。Tomás 举例提到，全球最大的一家跨国银行在部署 Claude Code 时，“在一个下午之内就烧光了整整一个月的预算”。导致这种推理支出失控式增长的背后有三大核心趋势驱动： 1. **前沿模型本身持续变贵**：只要算力规模与模型性能之间依然存在强相关性，模型厂商推出的旗舰前沿模型单价就会持续攀升。 2. **细微的能力提升引发使用时长成倍扩张**：当模型性能产生微幅跃升时，企业便能让智能体自主并行地执行长周期任务。以 Claude Opus 4.5 发布为例，支出飙升 10 倍并非因为该模型单价上涨了 10 倍，而是因为其可靠性“刚好提升了一点”，使得开发者能够放心让 Agent 整夜自主运转。 3. **AI 普及曲线仍处早期**：即便企业内 Coding Agent 的普及率达到 80% 至 95%，绝大部分 Token 消耗依然由极少数 Power User（重度用户）主导，伴随广泛普通开发者的逐步深入，整体消耗还会持续增加。与此同时，开源模型与高性价比轻量模型正在迅速崛起。GLM-5.2 的表现已逼近 Sonnet 甚至 Opus；Kimi K3 在某些维度上超越了 Opus；而 DeepSeek V4 Flash 等小型模型则展现出了极具竞争力的极致性价比。这为企业创造了巨大的“模型套利（Model Arbitrage）”空间——利用高昂的前沿模型与低成本高效核心模型之间的差距，按需分流。 --- ### [05:05](https://www.youtube.com/watch?v=5YdCKw2kAYI&t=305s) 网关与智能路由的区别及常见“死胡同”方案 Tomás 澄清了智能模型路由与网关（Gateway）的本质区别。网关（例如 OpenRouter、LiteLLM、Portkey 及企业自建的 API 代理网关）主要负责多模型接入的治理、鉴权与流量代理；而智能模型路由则是要在每个请求发出的当下，动态决定“何时使用哪一个模型”。解决模型路由是一个“极具欺骗性（Deceptively Difficult）”的难题。Not Diamond 早在 2023 年就开源了首个在 GPT-4 与 GPT-3.5 之间进行调度的智能路由器。Tomás 指出，行业中许多看似合理的直觉做法，在实践中往往都是死胡同（Dead Ends）： - **基于规则的 If-Else 启发式路由**：规则极其脆弱，难以适应复杂多变的任务场景。 - **基于 Query 向量空间的语义路由（Semantic Routing）**：无法准确衡量复杂代码上下文背后的实际解决难度。 - **使用 LLM 充当裁判进行路由（LLM-as-a-router）**：调用额外模型的延迟与 Token 成本往往直接抵消了路由带来的收益。 - **单独训练复杂度分类器（Complexity Classifier）**：离散分类难以捕捉 Agent 连续执行过程中的动态演变。 - **级联路由（Cascade Routing）**：先尝试小模型、失败后再回退调用大模型，不仅导致延迟翻倍，而且在长链路中极易引发重复计费。 - **基于首条消息的会话级路由（Session-based Routing）**：仅凭用户的第一条 Prompt 决定整条 Trajectory（轨迹）的模型选型，忽略了后续交互步骤中复杂度的大幅波动。 ```mermaid flowchart TD accTitle: Failed Naive Routing Approaches subgraph NaiveApproaches["常见的朴素/失效路由方案"] A["If-Else 启发式规则"] B["向量语义路由 (Semantic)"] C["LLM 充当裁判路由器"] D["复杂度分类模型"] E["级联回退路由 (Cascade)"] F["基于首条 Prompt 会话路由"] end subgraph Problems["引发的致命缺陷"] P1["规则过于僵硬脆弱"] P2["高延迟与二次推理开销"] P3["上下文连续状态丢失"] P4["无法应对长周期稀疏奖励"] end NaiveApproaches -.->|"导致"| Problems ``` --- ### [07:08](https://www.youtube.com/watch?v=5YdCKw2kAYI&t=428s) 机器人学视角：Terminal Bench 与多模型路由实测收益 Not Diamond 团队将代码智能体的长程调用视作一个**机器人学问题（Robotics Problem）**：在具有长视距（Long Horizons）与稀疏奖励（Sparse Rewards）的环境中，Coding Agent 的内部状态构成了运行环境，而可选的模型列表及推理强度级别（Reasoning Effort）则构成了动作空间（Action Space）。路由器必须在特定步骤做出决策，评估当前推荐的模型与思考深度在未来数轮甚至上百轮之后的期望总成本与累积奖励，因为真正的回报往往要等到会话终局才能得到验证。在权威基准评测 Terminal Bench 上，Not Diamond 的路由架构展现了卓越表现： - **同等效果降本**：在性能完全对齐 Claude Opus 4.8 Extra High 的前提下，实现了 **30% 的成本缩减**。 - **财富 500 强企业实测**：在合作的众多 Fortune 500 头部工程团队中，智能模型路由普遍带来了 **20% 到 40% 以上** 的直接支出节省。 - **跨供应商模型混合**：仅在 Anthropic 体系内部（Opus、Sonnet 与 Haiku 之间）切换即可稳定降低 30% 成本；一旦引入开源与开放权重模型，节省幅度几乎翻倍，并在综合性能上超越单一供应商方案。 --- ### [09:01](https://www.youtube.com/watch?v=5YdCKw2kAYI&t=541s) 影响 Agent 成本的核心维度与动态优化控制点深入剖析智能体内部的成本构成，Tomás 强调必须综合考量多维控制杠杆： | 成本维度 | 机制机理与潜在风险 | 优化手段 | |:--- |:--- |:--- | | **模型选型 (Model Choice)** | 盲目默认使用旗舰模型极度浪费；但若在错误节点调用了不合适的小模型，其代码错误、往返修复或陷入死循环（Loop Churning）所累积的消耗甚至比直接用大模型更贵。 | 按轮次动态预测并匹配当前步骤所需的最经济模型。 | | **推理强度 (Reasoning Effort)** | 开发者通常理解 Opus 与 Sonnet 的差异，但往往忽略 Opus Low 与 Medium、或 Sonnet High 与 Max 之间的悬殊开销。思考深度直接决定了思考 Token 的产出数量，并在后续多轮对话中被不断作为上下文反复读取。 | 细粒度控制推理深度与思考预算，避免无意义的过量 Token 输出。 | | **KV 缓存 (KV Cache)** | KV 缓存机制允许大模型厂商以低廉价格复用前序会话历史。如果在长会话中途贸然切换模型，会导致已有缓存完全失效并强制从头重写，反而大幅增加实际开销。 | 权衡缓存重写成本与模型单价差异，最大化维持 KV Cache 利用效率。 | | **上下文噪声与自动压缩** | 会话上下文过载会触发自动上下文压缩（Auto-compaction），不仅因一次性扫描全部超长上下文而极为昂贵，还会导致模型注意力和输出质量出现断崖式衰减。 | 主动识别长会话中的任务边界（Task Boundaries），在任务切换点提前且更高频地主动触发压缩。 | 针对这些多维动态因素，Not Diamond 在 Agent 的每一轮交互（Turn）中实时执行优化调度：不仅动态推荐模型与推理深度，同时监控并最大化 KV 缓存命中率；系统还能智能识别开发者在同一超长会话中切换的业务任务阶段，适时重构上下文；并通过开发者的即时反馈（例如在推荐 Haiku 后开发者是否手动重试或指出缺陷）进行持续在线学习与策略迭代。 ### [13:21](https://www.youtube.com/watch?v=5YdCKw2kAYI&t=801s) 隐式反馈与实时个性化路由当开发者在终端与 Coding Agent 交互时，会产生大量自然的交互信号。如果模型生成的方案偏离预期，开发者会直接指出“这不是我的意思，换这种方式来写”；如果方案正确，开发者则会确认“很好，继续执行”。Not Diamond 能够捕获这些隐式的开发者反馈信号，自动判断此前路由推荐较轻量模型（如 Claude Haiku）的决策是否正确。随后，路由算法会在后台实时更新，针对每位开发者的编码偏好与使用习惯进行个性化自适应调整。 ```mermaid flowchart TD accTitle: Not Diamond 架构与路由流程 subgraph Client["开发者本地机器 (Local Machine)"] Agent["Coding Agent Harness (Claude Code / OpenCode 等)"] Proxy["Not Diamond Local Proxy (提取衍生元数据 & 监控本地缓存)"] Agent --> Proxy end subgraph Server["Not Diamond 云端"] Router["远程路由服务器 (结合序列历史评估复杂度与推理等级)"] end subgraph Execution["模型执行与网关层"] Gateway["现有网关 / API (Anthropic 账户 / LiteLLM 等)"] VPC["专用 VPC 推理端点 (针对开源权重模型定制量化与缓存)"] end Proxy -- "仅发送匿名遥测元数据" --> Router Router -- "返回推荐模型与 Reasoning Effort" --> Proxy Proxy --> Gateway Proxy --> VPC ``` ### [13:43](https://www.youtube.com/watch?v=5YdCKw2kAYI&t=823s) 零数据留存的架构设计与本地代理在技术架构层面，Not Diamond 将自身无缝嵌入在现有的研发技术栈中，且不会接触任何企业敏感数据。开发者本地运行着 Coding Agent 的执行环境（Harness，如 Claude Code、OpenCode 等），并在后台部署一个轻量级的 Local Proxy。该本地代理负责提取去标识化的衍生元数据并监控本地 KV 缓存状态，随后将元数据发送至远程路由服务器；服务器结合当前元数据与任务序列中的历史步骤，实时向本地代理返回最优的模型推荐以及推理努力程度（Reasoning Effort Level）。本地代理收集并上传的元数据严格限制在遥测范畴内，包括： - 消息数量（Message Counts）与 Token 计数（Token Counts） - 本地计算的代码与任务复杂度评分（Complexity Score） - 本地计算的开发者反馈评分（Feedback Score） - 键值缓存（KV Cache）的容量与复用状态一旦本地代理接收到服务器返回的推荐模型和推理等级，便会将请求透明转发给用户原本的执行网关，例如直接调用企业的 Anthropic 账户或内部搭建的 LiteLLM 网关。此外，针对开源权重模型（Open-weight Models），系统还支持专用 VPC 推理端点，深度定制量化策略与缓存机制，使其性能与路由算法实现最大化协同。这种架构最核心的优势在于纯粹的隐私保护：Not Diamond 在全流程中“绝不会看到用户的任何 Prompt 输入、代码载荷（Payloads）或输出内容”。上传的纯遥测数据使得 Not Diamond 在法律和合规层面上“甚至不需要作为数据子处理者（Data Subprocessor）”，从根本上消除了企业对代码外泄和合规审计的顾虑。 ### [16:11](https://www.youtube.com/watch?v=5YdCKw2kAYI&t=971s) 从 Coding Agent 到背景 Agent 与杰文斯悖论 Coding Agent 当前虽然备受瞩目，但在整个 AI 演进中“仅仅是冰山一角”。在软件研发之外，知识工作者的总体规模更为庞大，各行各业的知识型 Agent 正在经历与代码 Agent 相同的爆发轨迹。比人机协作更庞大的计算需求将来自于“背景 Agent”（Background Agents）——这类 Agent 完全无需人类实时交互，它们在系统底层自主通信、7×24 小时持续监控基础设施、自动编排管理复杂的数据流水线。这种无需人工干预的自主运行模式将推动模型推理需求迎来指数级膨胀。这完美契合了经济学中的“杰文斯悖论”（Jevons Paradox）：某项资源的利用效率越高、成本越低，其整体消耗量反而会爆炸式增长。如果市场上出现比当前前沿模型“便宜 10 倍且强大 10 倍”的新模型，业界绝不会削减预算，而是会以数十倍的规模在后台 Agent 中消耗算力。预计在未来 6 到 18 个月内，全球推理算力支出的分配格局将全面向后台自主 Agent 倾斜。 ### [17:33](https://www.youtube.com/watch?v=5YdCKw2kAYI&t=1053s) 模型路由的终极使命与生态解耦建立智能模型路由体系不仅是一项有趣的工程挑战，更是重塑 AI 产业格局的核心路径。创立 Not Diamond 的初衷，是为了打破“全行业依赖单一巨型模型通吃一切”的垄断未来： - **消除锁定与切换成本**：将跨模型调用的切换成本实质上降为零，推动多样化的模型提供商生态繁荣共存。 - **权力向使用者转移**：通过根据任务能力最优分配工作负载，将行业话语权从顶尖前沿模型研发实验室（Model Labs）转移回企业与个人消费者手中。 - **消除资源与算力浪费**：精准匹配任务难度与模型算力，大幅降低全行业在算力消耗、经济成本以及能源消耗上的结构性低效。在演讲尾声，讲者打趣提到屏幕底部的字幕遮挡了 LinkedIn 主页信息，并向与会者提供了 Twitter 及邮箱联络方式。考虑到场内剩余时间有限，讲者建议将问答交流移步至会场外继续探讨，并感谢全场观众的参与。 ## Notable quotes > [01:17](https://www.youtube.com/watch?v=5YdCKw2kAYI&t=77s) 在一个下午之内就烧光了整整一个月的预算 — Tomás Hernando Kofman > [05:05](https://www.youtube.com/watch?v=5YdCKw2kAYI&t=305s) 基于规则的 If-Else 启发式路由：规则极其脆弱，难以适应复杂多变的任务场景 — Tomás Hernando Kofman > [05:05](https://www.youtube.com/watch?v=5YdCKw2kAYI&t=305s) 使用 LLM 充当裁判进行路由（LLM-as-a-router）：调用额外模型的延迟与 Token 成本往往直接抵消了路由带来的收益 — Tomás Hernando Kofman > [07:08](https://www.youtube.com/watch?v=5YdCKw2kAYI&t=428s) 在性能完全对齐 Claude Opus 4.8 Extra High 的前提下，实现了 30% 的成本缩减 — Tomás Hernando Kofman > [07:08](https://www.youtube.com/watch?v=5YdCKw2kAYI&t=428s) 在合作的众多 Fortune 500 头部工程团队中，智能模型路由普遍带来了 20% 到 40% 以上的直接支出节省 — Tomás Hernando Kofman > [07:08](https://www.youtube.com/watch?v=5YdCKw2kAYI&t=428s) 不仅在 Anthropic 体系内部（Opus、Sonnet 与 Haiku 之间）切换即可稳定降低 30% 成本；一旦引入开源与开放权重模型，节省幅度几乎翻倍 — Tomás Hernando Kofman > [13:43](https://www.youtube.com/watch?v=5YdCKw2kAYI&t=823s) 绝不会看到用户的任何 Prompt 输入、代码载荷（Payloads）或输出内容 — Tomás Hernando Kofman > [13:43](https://www.youtube.com/watch?v=5YdCKw2kAYI&t=823s) 甚至不需要作为数据子处理者（Data Subprocessor） — Tomás Hernando Kofman > [16:11](https://www.youtube.com/watch?v=5YdCKw2kAYI&t=971s) 某项资源的利用效率越高、成本越低，其整体消耗量反而会爆炸式增长 — Tomás Hernando Kofman > [17:33](https://www.youtube.com/watch?v=5YdCKw2kAYI&t=1053s) 将跨模型调用的切换成本实质上降为零，推动多样化的模型提供商生态繁荣共存 — Tomás Hernando Kofman > [17:33](https://www.youtube.com/watch?v=5YdCKw2kAYI&t=1053s) 通过根据任务能力最优分配工作负载，将行业话语权从顶尖前沿模型研发实验室（Model Labs）转移回企业与个人消费者手中 — Tomás Hernando Kofman ## Mentioned Tomás Hernando Kofman, Not Diamond, OpenRouter, SAP, Claude Code, Anthropic, Claude Opus 4.5, Claude Opus 4.8, Claude Sonnet, Claude Haiku, GLM-5.2, Kimi K3, DeepSeek V4 Flash, GPT-4, GPT-3.5, LiteLLM, Portkey, OpenCode, Terminal Bench ## Video description Tomás Hernando Kofman - CEO, Not Diamond Enterprises are burning through AI coding agent budgets in months. In this talk, Not Diamond CEO Tomás Hernando Kofman explains how intelligent model routing cuts coding agent costs by 20%+ while maintaining frontier quality, automatically identifying when a prompt needs a powerful model and when a cheaper one is sufficient. He'll cover how routing differs from gateways, why naive approaches backfire, and what model routing means for the future of AI. [Source](https://www.youtube.com/watch?v=5YdCKw2kAYI)

Source quotes

[1:17] Tomás Hernando Kofman: 在一个下午之内就烧光了整整一个月的预算 [5:05] Tomás Hernando Kofman: 基于规则的 If-Else 启发式路由：规则极其脆弱，难以适应复杂多变的任务场景 [5:05] Tomás Hernando Kofman: 使用 LLM 充当裁判进行路由（LLM-as-a-router）：调用额外模型的延迟与 Token 成本往往直接抵消了路由带来的收益 [7:08] Tomás Hernando Kofman: 在性能完全对齐 Claude Opus 4.8 Extra High 的前提下，实现了 30% 的成本缩减 [7:08] Tomás Hernando Kofman: 在合作的众多 Fortune 500 头部工程团队中，智能模型路由普遍带来了 20% 到 40% 以上的直接支出节省 [7:08] Tomás Hernando Kofman: 不仅在 Anthropic 体系内部（Opus、Sonnet 与 Haiku 之间）切换即可稳定降低 30% 成本；一旦引入开源与开放权重模型，节省幅度几乎翻倍 [13:43] Tomás Hernando Kofman: 绝不会看到用户的任何 Prompt 输入、代码载荷（Payloads）或输出内容 [13:43] Tomás Hernando Kofman: 甚至不需要作为数据子处理者（Data Subprocessor） [16:11] Tomás Hernando Kofman: 某项资源的利用效率越高、成本越低，其整体消耗量反而会爆炸式增长 [17:33] Tomás Hernando Kofman: 将跨模型调用的切换成本实质上降为零，推动多样化的模型提供商生态繁荣共存 [17:33] Tomás Hernando Kofman: 通过根据任务能力最优分配工作负载，将行业话语权从顶尖前沿模型研发实验室（Model Labs）转移回企业与个人消费者手中

Source quotes

[0:04] Tomás Hernando Kofman: Intelligent model routing is the ability to predictively determine when to deploy specific Large Language Models to maximize accuracy while minimizing costs. [0:04] Tomás Hernando Kofman: Enterprise AI inference spending has surged tenfold over the past year and is on track to scale by another 10x. [0:04] Tomás Hernando Kofman: One of the world's largest banks burned through its entire monthly Claude Code budget in a single afternoon. [2:15] Tomás Hernando Kofman: Frontier models continue to grow more expensive as compute demands scale alongside performance gains. [2:15] Tomás Hernando Kofman: Spend skyrocketed 10x—not because the token price was 10x higher, but because the model was reliable enough to run autonomously overnight. [5:05] Tomás Hernando Kofman: Gateways govern access, routing protocols, and infrastructure connections across multiple model APIs, while intelligent routers predictively decide which specific model to invoke for each request. [5:05] Tomás Hernando Kofman: Building an effective router is deceptively difficult. [5:05] Tomás Hernando Kofman: Not Diamond frames routing like a robotics problem: a long-horizon environment with sparse rewards. [9:01] Tomás Hernando Kofman: Selecting an underpowered small model can be equally expensive if it gets stuck in endless loops or requires costly bug fixes later. [9:01] Tomás Hernando Kofman: Switching models mid-session invalidates the existing cache, forcing a complete cache rewrite that can negate the savings of a cheaper model. [12:09] Tomás Hernando Kofman: Not Diamond never sees user prompt payloads, inputs, or outputs. [16:11] Tomás Hernando Kofman: Coding agents are merely the first wave of agentic deployment. [16:11] Tomás Hernando Kofman: Model routing shifts pricing power and control away from model producers and AI research labs back to enterprises and individual consumers.

Original transcript

[0:04] Uh yeah, hi everyone. I'm I'm Tomas. I'm co-founder and CEO of Not Diamond [0:09] where we focus on intelligent model routing specifically for coding agents. And uh for those [0:16] who are unfamiliar with the category, intelligent model routing is predictively determining when to [0:23] use which LLM to maximize accuracy and minimize cost. And there are new models coming out every [0:31] week and these vary in performance and price. And uh the the question of when to use which [0:38] is is sort of the the focus of this talk. And before I jump into how we actually do this, [0:45] uh I'll share a little bit about uh my team and and our background where uh all serial founders, [0:53] AI researchers, folks in our team published papers that invented function calling uh [0:59] and also introduced rag. Uh we're backed by some of the greatest AI scientists and founders in the [1:04] world and we work with some of the highest volume AI companies in the world as well uh including [1:09] uh open router where we power uh auto routing uh within the open router platform and SAP. [1:17] Um so before I go into uh you know how to think about intelligent model routing and [1:25] what the benefits are, I want to start with the problem and the problem is that we are [1:33] uh spending an incredibly uh incredibly high amount on on inference. We have about [1:40] 10x the spend that we did a year ago and we're on track to 10x again. And some of you may have seen [1:48] headlines like the ones I've shared here where you have some of the largest enterprises in the world [1:54] burning through their annual coding agent budgets in uh months if not weeks. Uh I know one um one [2:01] of the biggest banks in the world that burned through their uh entire claude code monthly [2:07] budget in a single afternoon. Um and so uh the question then is you know how what do we do about [2:15] this? and and there are three kind of underlying trends that are driving this increase in spend. [2:21] So one is that models are becoming more expensive not less you know fable GB 5.6 soul [2:30] um models keep keep getting uh more and more expensive and I think that's likely to continue [2:37] happening as long as there's a relationship between compute and performance. Um but the other [2:44] kind of major driver of spend is that a very small improvements in model performance and capability [2:53] leads to very large increases in spend because you can then run these models and these agents uh much [3:00] longer and in much more autonomous parallelized ways. So if you think back to in December when [3:07] Claude Opus 4.5 was released, the reason why spend 10xed, it's not because Opus 4.5 is 10x more [3:15] expensive than the previous models. It's because it was just a little bit better. And that little [3:20] bit better meant that we could actually start running it all night long overnight autonomously. [3:26] And then the third trend is that we're still early in the AI adoption curve. [3:30] um even as uh enterprises start to get to 80 90 95% adoption of AI coding agents uh within that [3:40] most spend is still dominated by power users uh whereas on the other hand you have a lot of [3:45] developers who are still still um ramping up in their usage of these tools. So we can expect that [3:53] spend is going to continue uh growing and at the same time uh the cheapest models are becoming more [4:01] powerful and open-source options are increasingly competitive with the frontier. Uh GLM 5.2 uh had a [4:09] huge shakeup I think in terms of its performance approximating sonnet if not opus. Kimmy K3 [4:16] uh surpasses opus and approximates fable. And you also have very powerful small models like DeepSeek [4:24] V4 flash uh which which has uh an incredibly strong price to performance um uh profile. [4:33] So the question then is how do we exploit essentially this model arbitrage opportunity [4:41] um as providers are uh competing with each other on price and performance and uh [4:48] sort of take advantage of this kind of durable growing uh gap between the frontier and and the [4:56] high cost of the frontier and this increasingly powerful lowcost kind of core of of models. [5:05] And that is um really what intelligent model routing is all about. It's about figuring out when [5:10] to use these more powerful frontier models versus when you can make use of a smaller model. And [5:17] uh the way I define intelligent model routing is uh automatically selecting the best model for [5:24] each request to maximize accuracy and minimize cost. And this is uh different from a gateway [5:32] which is used to govern access to many different models. So for example, open router is a gateway, [5:39] light LM is a gateway, port key is a gateway. Many enterprises have gateways uh set up internally. [5:46] But the intelligent model routing piece is really about determining when to use which model. Um [5:53] and and this is a deceptively difficult problem. Uh I've been working on this since 2023. We [6:00] open sourced the first uh intelligent model routing router that was between GB4 and 3.5. [6:06] And uh and there are a lot of uh sort of seemingly good paths that are actually deadend. So [6:13] uh some some of these that I have here are heristic if else routing, [6:18] semantic routing on on the embedding space of the uh of the query. Um using an LLM to do routing. So [6:26] using another model to figure out which model to use. Um training a complexity classifier. [6:32] Uh cascade routing where you start with a small model and you go to a bigger model and [6:36] until you get the right answer. And then session based routing where you use the first message [6:42] in an agent uh trajectory to figure out whether you should use a more powerful or weaker model. [6:48] I'm not going to go into why all of these are not the the architectures that we use or recommend at [6:54] Not Diamond. uh happy to talk to more uh to this after the talk if anyone has uh has questions. But [7:01] these are some of the um some of the approaches or potentially more naive approaches to solving this. [7:08] And the way we uh think about this is almost more like a robotics problem where you have [7:15] um long horizons with sparse rewards and you have uh essentially the environment which is [7:21] your coding agent state and you have an action space which is the the set of models and reasoning [7:27] efforts you can uh select. And so just like a robot needs to select the right action at [7:35] at a given point in time to maximize its reward in the future uh in uh model routing in long horizons [7:44] you're trying to identify if I recommend this model and this reasoning effort at this point [7:49] what are my expected future costs and my future rewards because you often don't actually get [7:55] to the reward until you complete the session and that might be 3, 5, 10, 100 turns later. [8:03] And um and when you do this, right, model routing really works. Um so here uh I have on the screen [8:12] our performance uh on terminal bench where we're essentially uh equivalent to [8:18] uh Opus 4.8 extra high at a 30% cost reduction. And similarly, we see in the Fortune 500 [8:26] engineering teams that we work with uh these kinds of cost savings anywhere from 20 to 40% or more um [8:34] simply by determining when to use which model. And this graph here is is only using anthropic models. [8:41] So we're just rounding between Opus, Sonnet, and Haiku in Cloud Code. [8:46] uh with uh open weight models in the mix, we actually nearly double the cost savings and also [8:54] uh improve performance over over uh single provider routing. Um and and so the way that [9:01] we think about this basically is it's important to start by understanding cost and how cost [9:09] uh develops within uh a coding agent or another any agent setting. And there's there's a few very [9:17] uh kind of dimensions here. The first is of course the model choice. Uh so um a lot of developers [9:25] will just default to the most powerful model for everything which is obviously wasteful. Um but [9:30] using the wrong small model at the wrong time can be just as expensive. Small models can take longer [9:36] to get the same amount of work done. they can make mistakes that have to go back and be fixed later [9:41] or they can get stuck in loops and kind of churn uh endlessly uh incurring a lot of spend. [9:48] The other really important lever here is the reasoning effort which um you know I [9:53] think a lot of people have a intuition for the difference between opus and sonnet. Um but not a [10:00] lot of people have a good sense for the difference between opus low and opus medium or uh sonnet high [10:06] and sonnet max. And these have very big impacts on cost because it directly correlates the number [10:12] of output tokens that the model is producing and then reconsuming on every subsequent turn. [10:18] And then the other very important piece in uh agentic settings and thinking about model [10:24] selection or model switching is the KV cache. So the KV cache is what allows a provider to reuse [10:32] previous messages in the session at a much lower cost. And so if you're very deep in a long session [10:38] and you switch to a cheaper model, you might actually end up spending more money [10:43] because you're rewriting the cash from scratch. So it's important to always understand what is [10:48] the state of the KB cache and how do the cash economics compare to the routing economics. [10:54] And all of this is sort of exacerbated by uh context and noisy context which can [11:00] blow up the context window uh and and trigger autoco compaction which is a a very expensive [11:05] way uh to do compaction both because uh you're consuming or ingesting the entire context window [11:12] in that operation a and also because as you uh maximize context window utilization [11:18] model performance degrades reliably and so uh it's also important to understand when and how [11:24] can we clear uh the context window by compacting earlier and more often? And then of course uh [11:32] this is all kind of challenging enough to solve at a single point in time. [11:37] But the bigger challenge is that it's a moving target and you have new models coming out [11:41] every week, old models getting updated, uh harnesses changing, pricing models evolving, [11:48] and and this is part of the reason why we think it makes sense for a company like ours to exist. [11:54] Um just because enterprises shouldn't not only have to be solving this difficult problem, [11:59] but then constantly resolving it, reinventing the wheel every uh every month as new models come out. [12:09] So the way that we uh approach this is essentially by uh optimizing applying optimizations on each [12:18] turn of a coding agent session and uh the kind of various levers that we look at are [12:25] uh first of all the model and reasoning effort recommendations. Uh we also as I mentioned look [12:30] at the state of the KV cache. we monitor it and we help maximize the utilization of the [12:36] of the KV cache. Uh we also identify task boundaries within long running sessions. [12:43] Uh I'm I'm sure everyone knows at least one developer who has been doing their entire [12:50] uh entire workflow on a single session of cloud code since they got started because they don't [12:55] want to lose what they were doing in the past. uh and that's very very inefficient and expensive [13:01] and so uh identifying when tasks change from you know task A to task B and triggering compaction [13:09] and then uh finally we we also do continuous learning from developer feedback. [13:15] So if we recommend haik coup on a given turn and on the next turn a developer says [13:21] that's not what I meant. Uh do it this way or they say okay great uh keep going. [13:25] We use those implicit developer feedback signals to learn whether it was a good [13:30] or bad decision to recommend IU there. And then we update in real time our routing algorithm in [13:36] a way that's personalized to that developer and their you know usage patterns and preferences. [13:43] And uh and then I'll I'll speak here to the architecture so you have a sense of like you know [13:49] where this actually fits in in the stack. So um do I have a oh great I actually have a pointer here. [13:56] Didn't realize that. So uh on the developer local machine uh you have your coding agent [14:02] harness that could be cloud code uh that could be open code any other harness. And uh and then [14:08] we also install a local proxy that is running in the background and extracting uh anonymized [14:16] derived metadata in a privacy preserving way. um and also monitoring the state of the local cache. [14:22] And it passes this uh metadata through to a remote server which uses the the metadata along [14:29] with uh every previous step in the sequence to return a reasoning uh a model and reasoning effort [14:36] level recommendation to the the local proxy. And this metadata includes uh message counts, token [14:43] counts, uh a locally computed complexity score, a locally computed developer feedback score, and the [14:50] city of the KB cache. And once it receives back the uh model and reasoning effort recommendation, [14:56] it passes this through to uh your existing gateway in the way requests would normally be executed. [15:03] So this could be directly through to your anthropic account. It could be to your internal [15:07] lightn gateway or any other uh way that you're calling models. And in some cases we also make [15:13] available dedicated uh VPC uh inference endpoints for openw weight models uh which can have many [15:21] configurations around quantization and uh cache mechanisms and this allows us to uh recommend [15:28] them in a way that's kind of maximally aligned with the uh the under underlying algorithm. [15:37] Oh, and uh actually before I go on, so one thing that I'll mention about this wi-i which is I think [15:42] pretty cool is that we what this means is that we actually never see your uh payloads or your [15:49] inputs or your outputs. Uh all we receive is this uh derived metadata which means that we're not [15:55] even a data subprocessor. um this is essentially telemetry data and uh and and I think that's [16:04] that's a pretty uh exciting feature of the way that we've approached this problem. Um and [16:11] and then I'll say here, you know, I've been talking throughout this uh this talk [16:17] about coding agents, but coding agents are really just the tip of the iceberg. uh for every engineer [16:25] there are many more knowledge workers and those knowledge workers are starting to ramp up on [16:30] on knowledge agents uh in a similar or towards a similar trend as engineers. Um [16:37] and and then I think even greater than that is going to be background agents. So these are agents [16:43] that are never interacting with humans that are communicating with each other that are [16:47] running continuously monitoring systems uh and uh managing data processes and so I think that the um [16:58] the growth in inference is uh only going to continue. I think some of you may be familiar with [17:04] Jevans paradox which says that the the cheaper a utility becomes the more we spend on it. And so if [17:10] we had a model that was, you know, 10x better and 10x cheaper than the frontier today, uh we would [17:17] be doing a lot more of all of this and especially uh this. And so this is where I expect uh the [17:24] sort of um uh allocation of inference spend to move in the next 6 to 18 months. [17:33] And uh and finally I'll I'll just wrap up with uh why I think this is such an important [17:38] problem to work on it. First of all, it's very interesting. So it's enjoyable to work on it. [17:43] But really the the reason why my co-founders and I started this company is that I think model routing [17:51] moves us away from a future in which everyone is using a single giant model for everything. uh it [17:57] promotes a diverse ecosystem of providers by bringing the switching costs of these models to uh [18:04] essentially to zero and optimally distributing workloads across all these different [18:09] capabilities. Uh it also in that same sense shifts power away from uh producers from the labs [18:16] to the consumers uh both individuals as well as enterprises. And then I also think that [18:23] it it reduces the ecological and uh economic inefficiencies uh sort of within this technology. [18:31] Um so yeah, I think I think that's my last slide. So uh appreciate everyone coming out. [18:38] I uh have up here my Twitter handle and email and covered by my subtitles is my LinkedIn, [18:47] but you can find me on there. Um I have a minute and a half on the clock but in lie [18:51] of taking questions here I think it might be more productive to just meet outside and we [18:56] can chat there. So thank you all very much. Uh really enjoyed uh getting to share with you all.