OpenAI Engineers Warn Cheap Tokens Drive Up Total AI Task Costs

Stop Overpaying for Intelligence | DevDay 2026 — OpenAI

OpenAI

Nominally cheaper models inflate enterprise spending by failing basic tasks and triggering costly human escalations, making cost per completed task the only viable efficiency metric.

AI-assisted summary of Stop Overpaying for Intelligence | DevDay 2026 — original source. Check the original for full context.

Source published:

Key ideas

  • Focusing strictly on list prices per token is a trap; cheaper token models often end up more expensive per completed task due to rambling, retries, and high human escalation costs. (0:13)
  • Pairing lighter models like Luna with high reasoning effort ('Luna-maxing') frequently outperforms older or competing frontier models at a fraction of the operating cost. (7:44)
  • Four primary levers beyond model selection dramatically cut spend: prompt caching (up to 90% discount), programmatic tool calling, reasoning effort tuning, and the Batch API (50% discount). (8:47)
  • OpenAI introduced explicit breakpoints, explicit-only caching mode, and a Diagnostics API to debug cache misses and lock in deterministic hit rates across dynamic workflows. (18:43)

Detailed notes

## Background Published by OpenAI on YouTube on October 7, 2026, this DevDay presentation features AI deployment engineers sharing architectural strategies to slash API expenditures without compromising performance. According to the video description, the session covers measuring cost per completed task, tuning model selection and reasoning effort, and leveraging prompt caching, Programmatic Tool Calling, and the Batch API.

## TL;DR OpenAI deployment engineers Mandeep and Sapto warn that developers frequently fall into the trap of evaluating models based on raw cost per token rather than total cost per completed task, where weaker models can drive up costs through token bloat and human escalations. The panel advocates plotting setups along the Pareto frontier of accuracy versus expense, highlighting practices like 'Luna-maxing' to achieve frontier-level performance at lower tiers. To optimize further, they detail four concrete levers: prompt caching with explicit breakpoints and diagnostics, Programmatic Tool Calling to filter data before LLM ingestion, dynamic reasoning effort tuning, and asynchronous Batch API processing. Teams are encouraged to adopt a disciplined evaluation playbook that isolates single variables against production datasets to maximize efficiency.

## Atmosphere & style The session has the brisk, pragmatic energy of an open-air developer masterclass, blending conversational warmth with rigorous systems engineering advice. Host Arti facilitates a dynamic exchange between Mandeep and Sapto, whose rapport is grounded in relatable real-world engineering headaches—such as personal travel chatbot failures—rather than theoretical vendor pitching. It feels engaging and actionable, inviting the audience to trade superficial pricing comparisons for empirical benchmarking discipline.

## Bulletin - [00:13](https://www.youtube.com/watch?v=_nmlHbSB8kM&t=13s) Focusing strictly on list prices per token is a trap; cheaper token models often end up more expensive per completed task due to rambling, retries, and high human escalation costs. - [07:44](https://www.youtube.com/watch?v=_nmlHbSB8kM&t=464s) Pairing lighter models like Luna with high reasoning effort ('Luna-maxing') frequently outperforms older or competing frontier models at a fraction of the operating cost. - [08:47](https://www.youtube.com/watch?v=_nmlHbSB8kM&t=527s) Four primary levers beyond model selection dramatically cut spend: prompt caching (up to 90% discount), programmatic tool calling, reasoning effort tuning, and the Batch API (50% discount). - [18:43](https://www.youtube.com/watch?v=_nmlHbSB8kM&t=1123s) OpenAI introduced explicit breakpoints, explicit-only caching mode, and a Diagnostics API to debug cache misses and lock in deterministic hit rates across dynamic workflows.

## Key takeaways - [00:13](https://www.youtube.com/watch?v=_nmlHbSB8kM&t=13s) Developers frequently overpay for AI by evaluating raw token list prices rather than measuring true cost per completed task, which includes token volume and costly human fallbacks. - [00:13](https://www.youtube.com/watch?v=_nmlHbSB8kM&t=13s) Customer implementations show capability drives token efficiency: Perplexity achieved 9% higher accuracy at half the cost with 6 Astra, and Notion cut cost per task by 50% transitioning to 5.6 Sol. - [04:47](https://www.youtube.com/watch?v=_nmlHbSB8kM&t=287s) Model selection should start by defining crisp task boundaries and a target accuracy threshold, then finding the lowest-cost configuration along the Pareto frontier. - [07:44](https://www.youtube.com/watch?v=_nmlHbSB8kM&t=464s) 'Luna-maxing'—running lightweight models with boosted reasoning effort—enables deploying Luna for 80% to 90% of developer workflows while only escalating edge cases to frontier models. - [08:47](https://www.youtube.com/watch?v=_nmlHbSB8kM&t=527s) Prompt caching provides up to a 90% discount on input tokens by reusing pre-processed attention states for static prefixes like instructions, documentation, and few-shot examples. - [10:19](https://www.youtube.com/watch?v=_nmlHbSB8kM&t=619s) Programmatic tool calling offloads intermediate API orchestration to lightweight code, preventing raw, noisy data round-trips from bloating the LLM context window. - [10:19](https://www.youtube.com/watch?v=_nmlHbSB8kM&t=619s) Adopting programmatic tool calling allowed Clio to cut prompt tokens by 38% across multi-step document analysis workflows with zero loss in output quality. - [12:55](https://www.youtube.com/watch?v=_nmlHbSB8kM&t=775s) Reasoning effort should be actively tuned independently of output length, dialing compute down for routine tasks and reserving extra reasoning tokens for high-leverage edge cases. - [12:55](https://www.youtube.com/watch?v=_nmlHbSB8kM&t=775s) The OpenAI Batch API delivers a 50% cost discount for latency-tolerant, high-volume workloads processed asynchronously within a 24-hour window. - [16:41](https://www.youtube.com/watch?v=_nmlHbSB8kM&t=1001s) Keeping static prompt content at the prefix is essential for maximizing automatic cache hit rates; Blitzy increased cache reuse from 24% to 90%, slashing total costs by 87%. - [18:43](https://www.youtube.com/watch?v=_nmlHbSB8kM&t=1123s) Explicit breakpoints and explicit-only mode give developers deterministic control to dictate exact cache boundaries before variable user inputs. - [20:02](https://www.youtube.com/watch?v=_nmlHbSB8kM&t=1202s) Modern caching architecture allows developers to modify reasoning effort and update allowed tools mid-thread without invalidating or busting existing cached prefixes. - [20:02](https://www.youtube.com/watch?v=_nmlHbSB8kM&t=1202s) A new web admin dashboard tracks organization-wide cache hit rates, while a Diagnostics API compares query payloads to pinpoint the exact token boundary causing cache misses. - [21:20](https://www.youtube.com/watch?v=_nmlHbSB8kM&t=1280s) Engineering teams should iteratively test single variables against representative evaluation datasets to systematically advance their position along the accuracy-cost Pareto frontier.

## Detailed walkthrough ### [00:13](https://www.youtube.com/watch?v=_nmlHbSB8kM&t=13s) The Metric Trap: Cost per Token vs. Cost per Task

Opening the DevDay outdoor stage session, host Arti welcomes attendees to a panel discussion with two members of OpenAI's AI deployment engineering team—Mandeep and Sapto—focused on how developers can dramatically optimize AI operational costs without sacrificing output quality. Arti identifies the single biggest mistake engineering teams make at the outset: focusing on the wrong metric. Most developers fixate on the raw list prices published on model pricing pages—the upfront cost of input and output tokens. Mandeep, who leads OpenAI's Applied AI Engineering team in San Francisco, explains that this surface-level view is deeply misleading: "A model that is cheap on cost per token basis may not be the cheapest model on cost per task basis."

```mermaid flowchart TD accTitle: Cost per Task Breakdown accDescr: Diagram illustrating how total task cost combines token consumption and human escalation expenses. subgraph Input ["Task Request"] REQ["User Request"] end

subgraph Evaluation ["Model Execution"] CHEAP["Cheaper Token Model"] FRONTIER["Frontier / Right-Sized Model"] end

subgraph Resolution ["Final Task Cost"] FAIL["High Token Usage / Failure"] --> ESC["Human Agent Escalation"] SUCCESS["Token-Efficient Direct Completion"] ESC --> TOTAL_HIGH["Higher Cost per Task (Tokens + Labor)"] SUCCESS --> TOTAL_LOW["Lower Cost per Task"] end

REQ --> CHEAP REQ --> FRONTIER CHEAP --> FAIL FRONTIER --> SUCCESS ```

Mandeep highlights two primary reasons a nominally cheaper model ends up driving overall expenditures higher. First, a less capable model often burns far more tokens—either rambling, generating redundant reasoning steps, or requiring multiple round-trip retries—to reach an acceptable answer. Second, and more costly, a weaker model frequently fails to complete the objective altogether, triggering expensive human fallbacks. Mandeep illustrates this with a personal travel mishap from the previous week. After his flight to New York City was canceled, he attempted to rebook via an airline customer support chatbot. Despite having "a soft spot for customer support chatbots" because his team helps build them, he found the bot incapable of understanding his basic constraint—that he refused to take a red-eye flight. Frustrated, he had to demand a human agent, who finally routed him through Houston in time for his team offsite. While the airline's underlying model might have cost fractions of a cent per token, the true cost of that task was the API tokens plus the expensive human labor required to finish the booking.

Real-world deployments from OpenAI customers underscore the impact of token efficiency and capability over unit token cost: - **Perplexity**: Reported on their research benchmarks that 6 Astra achieved 9% higher accuracy at half the overall cost compared to their previous model, because it required far fewer tokens to resolve queries. - **Notion**: Transitioned internal systems from 5.5 to 5.6 Sol, reporting improved accuracy alongside a 50% reduction in cost per task.

---

[04:47](https://www.youtube.com/watch?v=_nmlHbSB8kM&t=287s) Right-Sizing Models Along the Pareto Frontier

When counseling businesses on model selection, Mandeep emphasizes that teams should never start by picking an arbitrary checkpoint. Instead, engineering must start by crisply specifying the task and defining a concrete accuracy threshold. In the airline chatbot scenario, an organization might establish an explicit service-level target such as resolving 80% of routine incoming traveler inquiries entirely autonomously. With GPT Live paired with 6.0 as a delegate model, for example, the airline could have settled the schedule conflict without ever involving a human representative.

To evaluate which setup best satisfies product requirements, Mandeep introduces the framework of the Pareto frontier. Developers should evaluate model candidates, prompt harnesses, and operational configurations by plotting accuracy against total cost. By testing varying configurations systematically against an evaluation dataset, teams can pinpoint the exact setup that clears their accuracy threshold at the absolute lowest cost per task.

- **Simple and Structured Tasks**: For classification, lightweight routing, or structured field extraction from documents, developers should begin testing smaller, highly efficient models early in the process. - **Complex Reasoning Tasks**: For ambiguous planning, high-stakes synthetic reasoning, or complex tool orchestration, teams should start at the frontier with 6 Astra, mapping downward along the frontier to see if a smaller model can hold the target accuracy bar.

---

[07:44](https://www.youtube.com/watch?v=_nmlHbSB8kM&t=464s) The Power of "Luna-Maxing"

Addressing a question from Arti on what teams should do when a model's fitness isn't obvious upfront, Mandeep brings up a popular engineering practice known as "Luna-maxing." Many enterprise customers assume they must rely on massive frontier models for high-quality developer and production workflows. However, teams are frequently astonished to discover that running a lightweight model like Luna with its reasoning effort set to "high" or "extra high" frequently outperforms older, larger generation models and competing models while maintaining a fraction of their operating cost.

Rather than treating small models merely as rudimentary routers, numerous OpenAI customers now deploy Luna as the primary default engine for 80% to 90% of internal developer coding workflows. They configure their systems to escalate only to larger frontier models like 6 Astra or 6.1 Sol when an edge case genuinely requires deeper reasoning intelligence. Mandeep urges builders not to underestimate smaller models before benchmarking them thoroughly with boosted reasoning effort. Arti adds that the newly announced Decisions API similarly expands options for resolving straightforward decisions efficiently.

---

[08:47](https://www.youtube.com/watch?v=_nmlHbSB8kM&t=527s) Four Levers Beyond the Model: Prompt Caching

Selecting the base model family is only the first step in cost governance. Sapto points out that identical models running the same core task can produce wildly divergent bills depending on how input data is structured, how context is passed, and how output generation is constrained. He outlines four distinct architectural levers developers can pull to curb spend while strictly preserving quality: 1. **Prompt caching** 2. **Programmatic Tool Calling** 3. **Reasoning effort tuning** 4. **Batch or Flex API processing**

Diving into the first lever, Sapto focuses on prompt caching, explaining that the most immediate savings come from eliminating redundant token processing across repeated prompt contexts. In a standard corporate customer support assistant, every single user turn typically re-sends extensive blocks of static knowledge—such as internal policy manuals, standard operating procedures, and company documentation. Instead of running full compute across these shared tokens on every incoming request, prompt caching reuses the pre-processed state of static prefix data across calls, restricting active token costs solely to the unique, novel user text and generated completions.

[10:19](https://www.youtube.com/watch?v=_nmlHbSB8kM&t=619s) Lever 2: Programmatic Tool Calling

Sapto poses a real-world scenario to the audience: imagine an online ordering system where an agent processes incoming orders and needs to check whether there is sufficient stock available to fulfill them. Normally, the agent calls multiple tools, inspects what is missing versus what is arriving, and feeds all intermediate raw data back to the LLM reasoning engine. Sapto asks the crowd whether the model really needs to read, parse, and compare all of those raw tool outputs itself. Seeing heads shake in disagreement across the room, he confirms that it does not—and that this insight forms the core of the second major cost optimization lever: **programmatic tool calling**.

```mermaid flowchart TD accTitle: Programmatic Tool Calling Architecture accDescr: Diagram showing how a lightweight execution program coordinates tools and passes only a packaged report to the LLM.

subgraph LLM_Layer["LLM Decision Engine"] LLM["LLM Agent"] end

subgraph Execution_Layer["Programmatic Layer (Code)"] Program["Lightweight Program / Script"] T1["Inventory Tool"] T2["Shipment Tracking"] T3["Order DB"] end

LLM -->|"Dispatches execution task"| Program Program -->|"Calls"| T1 Program -->|"Calls"| T2 Program -->|"Calls"| T3 T1 -->|"Raw data"| Program T2 -->|"Raw data"| Program T3 -->|"Raw data"| Program Program -->|"Aggregated, packaged report"| LLM ```

Instead of sending every raw intermediary API payload across multiple round-trips into the context window, the model triggers a small program. That script orchestrates the required tool calls, calibrates and combines the responses, and returns a single, concise packaged report back to the LLM for final judgment. This setup avoids bloating the context window with intermediate noise and slashes the number of repetitive model invocations required by multi-step workflows.

To illustrate the real-world financial and performance impact, Sapto highlights **Clio**, an AI-powered legal software company that provides cloud management platforms to law firms: - **38% prompt token reduction:** Clio achieved a 38% decrease in prompt tokens across their multi-step document analysis workflows by adopting programmatic tool calling. - **Zero quality loss:** The savings came purely from changing how the application coordinates data with the model, without degrading output accuracy.

---

[12:55](https://www.youtube.com/watch?v=_nmlHbSB8kM&t=775s) Levers 3 & 4: Reasoning Effort and Batch/Flex Processing

Turning to the third lever, Sapto addresses **reasoning effort**, urging developers to treat it as an active knob to test rather than a static configuration. Crucially, reasoning effort is distinct from generated output length: even a short, concise answer can require deep, token-heavy internal reasoning.

For routine routing or straightforward classification tasks, the baseline recommendation is to dial reasoning effort down to a lower tier and validate whether output quality holds via evals. Extra reasoning tokens should be reserved for difficult, highly differentiated tasks where the added compute genuinely moves the needle. Because model architectures evolve rapidly, developers should systematically test one parameter change at a time rather than guessing.

The fourth lever is the **Batch API** (or Flex processing), which centers on latency tolerance: "How much are you willing to wait?" When processing non-urgent, high-volume workloads such as analyzing thousands of asynchronous documents: - **24-hour turnaround window:** OpenAI Batch API processes requests asynchronously within 24 hours. - **50% discount:** Token costs are slashed by 50% compared to standard synchronous API pricing.

Sapto emphasizes that optimizing AI applications requires moving beyond the simple question of *"Which model family should I use?"* Developers must also ask, *"How much work am I going to make the model do?"* The ultimate goal is not chasing token minimization in a vacuum, but achieving a successful task outcome while balancing low cost and acceptable latency.

---

[15:29](https://www.youtube.com/watch?v=_nmlHbSB8kM&t=929s) The Four Optimization Levers Summarized

Mandeep steps in to synthesize the four core mechanisms discussed so far, underscoring their cumulative impact on production unit economics:

1. **Prompt Caching:** Delivers up to a 90% discount on cached input token costs, making it the single largest potential price reducer in high-frequency applications. 2. **Programmatic Tool Calling:** Offloads data filtering, aggregation, and repetitive execution logic to code, preventing intermediary clutter from entering the model's context window. 3. **Reasoning Effort Tuning:** Allows granular control over model thinking time. Mandeep highlights a critical API milestone introduced with Astra: developers can now dynamically modify the reasoning effort parameter without invalidating or busting the prompt cache. 4. **Batch / Flex API:** Cuts pricing in half (50% off) for latency-tolerant, non-real-time workloads.

---

[16:41](https://www.youtube.com/watch?v=_nmlHbSB8kM&t=1001s) Deep Dive: Maximizing Prompt Cache Hit Rates

Mandeep pivots the conversation back to prompt caching, asking Sapto to unpack common developer mistakes and best practices for maximizing cache hit rates. When Sapto asks the audience how many are already using prompt caching, Mandeep quips, *"It's all of us. It's enabled by default."*

Despite automatic enablement, many teams leave substantial savings on the table because of poor prompt layout. OpenAI's primary design rule is: **keep the static beginning of your prompt strictly consistent.** By placing persistent system prompts, static few-shot examples, and shared tool definitions at the prefix of the prompt, the platform reuses the preprocessed attention states of that shared prefix. Only dynamic, variable user information placed at the end needs to be freshly computed, resulting in immediate drops in both latency and billing.

To demonstrate the upside of intentional cache structuring, Sapto shares results from **Blitzy**, an AI-powered software development platform: - Transitioned from a single structured output call to an iterative tool-calling loop. - **Cache reuse surged from 24% to 90%**. - Across thousands of production queries, Blitzy achieved **8.5× fewer prompt tokens** and **87% lower costs** compared to their previous baseline, while handling **2.2× more context**.

---

[18:43](https://www.youtube.com/watch?v=_nmlHbSB8kM&t=1123s) Explicit Breakpoints and Explicit-Only Mode

Mandeep raises a common real-world hurdle: keeping prompt prefixes identical is easy in theory, but in practice, prompt structures frequently shift across disparate user tasks and dynamic templates.

Sapto explains that earlier model families relied entirely on automatic caching heuristics based on fixed token intervals; developers could align their prompt prefixes, but they had no direct way to dictate boundary cutoffs. With the latest model generations, OpenAI introduced **explicit breakpoints**.

An explicit breakpoint functions as a developer-defined directive telling the system: *"Cache up to this exact point."* For example, in an agentic coding workflow with extensive, persistent system guidelines followed by rapidly changing tasks, an engineer can place an explicit breakpoint directly after the stable instructions and right before the dynamic user input. This guarantees that the core instructions hit the cache every time, regardless of varying input lengths downstream. For teams requiring strict, deterministic billing boundaries, the API also includes an **explicit-only mode**, enabling applications to specify precisely which designated chunks are eligible for caching.

[20:02](https://www.youtube.com/watch?v=_nmlHbSB8kM&t=1202s) Prompt Caching Flexibility and Observability Tools

Selective caching is essential for real-world efficiency because developers should never cache volatile content that changes frequently and will not be reused across requests. Sapto highlights a major advancement in caching architecture: developers now have the ability to dynamically adjust reasoning effort within an ongoing conversation without invalidating the cache prefix. Arti jumps in to underscore an existing, underappreciated feature—developers can also swap or update the tools passed to the model (such as specifying them via `allowed_tools` or other parameter controls) without breaking the cache. Together, these controls grant engineering teams immense architectural flexibility to tweak reasoning depth and toolsets on the fly while preserving high cache hit rates.

OpenAI has paired these caching capabilities with dedicated observability tooling to help teams actively audit and debug their token savings. A dedicated prompt caching dashboard is available on the admin web portal, allowing teams to monitor, measure, and track cache hit rates across all production applications. Additionally, a new Diagnostics API accepts two queries and pinpoints the exact boundary where a cache miss occurred, eliminating the guesswork from prompt restructuring and turning prompt caching into an easily actionable cost-optimization lever.

- **Dynamic reasoning without cache invalidation**: Reasoning effort can be adjusted mid-thread without busting the cached context. - **Cache-safe tool definitions**: Modifying available tools via parameters like allowed tools preserves existing cached prefixes. - **Admin caching dashboard**: Provides visibility into organization-wide cache hit rates across distinct application endpoints. - **Diagnostics API**: Compares two prompt payloads and isolates the precise token position causing a cache miss.

```mermaid flowchart TD accTitle: Diagnostics and Cache Optimization Flow accDescr: Flow showing how the Diagnostics API identifies prompt mismatches to protect cache hits while varying reasoning and tools.

subgraph Inputs ["Prompt Requests"] Q1["Baseline Prompt Prefix"] Q2["Modified Prompt / Follow-up"] end

subgraph CachingEngine ["Cache-Preserving Modifications"] RE["Adjust Reasoning Effort"] TL["Update Allowed Tools"] end

subgraph Observability ["Observability & Debugging"] Diag["Diagnostics API (Detects Miss Point)"] Dash["Admin Dashboard (Tracks Hit Rates)"] end

Q1 --> CachingEngine Q2 --> CachingEngine CachingEngine --> Diag Diag --> Dash ```

---

[21:20](https://www.youtube.com/watch?v=_nmlHbSB8kM&t=1280s) Core Levers and the Practical Evaluation Playbook

Arti summarizes the session's overarching framework for eliminating unnecessary spend: developers must abandon naive token list prices and instead optimize for total "cost per completed task." By locating configurations on the Pareto frontier of accuracy versus cost, teams can balance performance and expense using key tactical levers: prompt caching, Programmatic Tool Calling, dialed-in reasoning effort, and the Batch API.

Turning to practical implementation, Arti asks Mandeep and Sapto for their parting guidance on what developers should immediately do when returning to their codebases. Mandeep outlines a structured, metrics-driven testing workflow:

1. **Establish task boundaries and requirements**: Clearly define the exact task and establish the baseline accuracy threshold required by the business or use case. 2. **Assemble representative evals**: Curate an evaluation dataset that accurately mirrors production usage patterns. 3. **Explore configuration permutations**: Systematically test candidate models across diverse reasoning effort levels, prompt variations, and prompt caching harnesses. 4. **Plot the Pareto frontier**: Graph accuracy against cost to identify the optimal configuration harness that satisfies business accuracy requirements at minimal spend.

Sapto closes with an essential rule of discipline for ongoing iteration: isolate and test a single variable or improvement at a time. Once a concrete gain is verified, lock that setting in and continue to run evaluations continuously as models, prompts, and application requirements evolve.

## Notable quotes > [00:13](https://www.youtube.com/watch?v=_nmlHbSB8kM&t=13s) A model that is cheap on cost per token basis may not be the cheapest model on cost per task basis. — Mandeep

> [00:13](https://www.youtube.com/watch?v=_nmlHbSB8kM&t=13s) a soft spot for customer support chatbots — Mandeep

> [12:55](https://www.youtube.com/watch?v=_nmlHbSB8kM&t=775s) How much are you willing to wait? — Sapto

> [12:55](https://www.youtube.com/watch?v=_nmlHbSB8kM&t=775s) Which model family should I use? — Sapto

> [12:55](https://www.youtube.com/watch?v=_nmlHbSB8kM&t=775s) How much work am I going to make the model do? — Sapto

> [16:41](https://www.youtube.com/watch?v=_nmlHbSB8kM&t=1001s) It's all of us. It's enabled by default. — Mandeep

> [16:41](https://www.youtube.com/watch?v=_nmlHbSB8kM&t=1001s) keep the static beginning of your prompt strictly consistent.

> [18:43](https://www.youtube.com/watch?v=_nmlHbSB8kM&t=1123s) Cache up to this exact point. — Sapto

## Mentioned OpenAI, Arti, Mandeep, Sapto, Perplexity, 6 Astra, Notion, 5.5, 5.6 Sol, GPT Live, 6.0, Luna, 6.1 Sol, Decisions API, Clio, Batch API, Flex API, Blitzy, Diagnostics API

## Video description Cut AI costs while preserving quality. Learn to measure cost per completed task, choose models and reasoning effort, and use prompt caching, Programmatic Tool Calling, and the Batch API.

[Source](https://www.youtube.com/watch?v=_nmlHbSB8kM)

中文简报

OpenAI工程师警告：低廉Token单价反而推高AI总任务成本 名义上更便宜的模型因无法完成基础任务并引发昂贵的人工介入，反而推高了企业支出，这使得每个已完成任务的成本成为衡量效率的唯一可行指标。 OpenAI部署工程师Mandeep与Sapto警告称，开发者往往容易陷入根据原始Token单价而非每个已完成任务的总成本来评估模型的误区，性能较弱的模型可能会因Token膨胀和人工升级介入而推高总体成本。该讨论小组提倡沿着准确率与支出的帕累托前沿来规划配置，并强调了如“Luna-maxing”等在更低层级实现前沿级性能的做法。为了进一步优化，他们详细介绍了四个具体途径：带有显式断点和诊断功能的提示词缓存、在LLM摄入前过滤数据的程序化工具调用、动态推理算力调节，以及异步Batch API处理。他们鼓励团队采用严谨的评估方案，针对生产数据集隔离单一变量，以实现效率最大化。 单纯关注Token标价是一个陷阱；由于回答冗长、重试以及高昂的人工升级成本，Token单价更便宜的模型在每个已完成任务上的最终成本往往更高。 (0:13) 将Luna等较轻量模型与高推理算力相结合（即“Luna-maxing”），往往能以极低的运营成本超越较旧或竞争对手的前沿模型。 (7:44) 除模型选择之外，还有四个主要途径可大幅削减开支：提示词缓存（最高可享90%折扣）、程序化工具调用、推理算力调节以及Batch API（50%折扣）。 (8:47) OpenAI推出了显式断点、仅显式缓存模式以及Diagnostics API，以调试缓存未命中问题，并在动态工作流中锁定确定性的命中率。 (18:43)

Reading summary

OpenAI deployment engineers Mandeep and Sapto argue that engineering teams frequently overspend by prioritizing raw token list prices rather than total cost per completed task. Achieving cost efficiency requires right-sizing models along a Pareto frontier while leveraging architectural levers like prompt caching, programmatic tool orchestration, and batch processing. Reframing Spend: Cost per Token Versus Task (0:13) Engineering teams often make the mistake of evaluating models based solely on upfront per-token pricing. Mandeep explains that lower-priced token models often drive up total expenses because weaker models frequently ramble, require redundant retries, or fail completely. When a model fails, it triggers expensive human intervention, drastically inflating the real cost of the task. For example, an airline customer service bot incapable of understanding a constraint against overnight flights forced Mandeep to escalate to a human agent, shifting the true task expense from fractions of a cent to costly human labor. Optimizing for task efficiency rather than unit price produces measurable savings. OpenAI customer Notion transitioned systems to 5.6 Sol, achieving improved accuracy alongside a 50% reduction in cost per task. Similarly, Perplexity reported that 6 Astra delivered 9% higher accuracy at half the overall cost of their prior model because it resolved queries with significantly fewer tokens. Pareto Right-Sizing and 'Luna-Maxing' (4:47) Rather than arbitrarily selecting high-end checkpoints, Mandeep advises teams to define concrete accuracy thresholds and map configurations along the Pareto frontier of accuracy versus cost. Simple extraction and routing tasks can begin on small models, whereas ambiguous reasoning tasks should start at the frontier—such as 6 Astra—and step down only if accuracy holds. Teams can also extract outsized performance from lightweight models through 'Luna-maxing.' Configuring a smaller model like Luna with its reasoning effort set to 'high' or 'extra high' often outperforms older or larger models at a fraction of their operating cost. Mandeep notes that several enterprise customers now route 80% to 90% of internal coding tasks to Luna by default, reserving escalations to frontier models like 6 Astra or 6.1 Sol strictly for complex edge cases. Context Reduction via Programmatic Tools and Batching (10:19) Model choice alone does not dictate expenses; how context is structured and handled strongly influences total spend. Sapto highlights programmatic tool calling as a mechanism to prevent context window bloat. In traditional agent architectures, raw outputs from multiple tool queries are returned directly to the model, consuming large volumes of tokens. Instead, developers can dispatch a lightweight script that executes the calls, aggregates the data, and returns a single concise report to the model. Legal platform Clio implemented this pattern, achieving a 38% reduction in prompt tokens across multi-step document analysis with zero quality degradation. Sapto also identifies the Batch API as a major lever for non-urgent, high-volume workloads. By accepting an asynchronous turnaround window of up to 24 hours, applications receive a 50% discount compared to standard synchronous API pricing. Cache Optimization, Breakpoints, and Observability (16:41) Prompt caching offers up to a 90% discount on cached input tokens, but realization depends on prompt design. Teams must keep static system prompts, tool definitions, and few-shot examples consistently positioned at the prompt prefix, leaving dynamic user text at the end. Development platform Blitzy transitioned to an iterative tool loop and structured prompt prefixing, boosting cache reuse from 24% to 90%, reducing prompt tokens by 8.5×, and cutting costs by 87%. To handle dynamic prompt layouts, OpenAI introduced explicit breakpoints and explicit-only mode, enabling developers to manually designate exact boundaries where the system must cache. Furthermore, developers can adjust reasoning effort dynamically or alter tool parameters mid-thread without invalidating cached prefixes. Observability tools, including an admin caching dashboard and a Diagnostics API that isolates the exact token causing a cache miss, help teams monitor and debug hit rates. The speakers outline a four-step implementation playbook: define explicit task requirements, assemble representative evaluation datasets, systematically test configurations against the Pareto frontier, and isolate one variable at a time. The unresolved operational challenge for developers is continually maintaining disciplined evaluation workflows as base model capabilities, prompt designs, and application demands shift.

Detailed notes — zh

## Background 该视频由 OpenAI 官方 YouTube 频道于 2026 年 10 月 9 日发布，属于 DevDay 2026 的演讲环节。视频简介指出，本次分享旨在帮助开发者在保持质量的同时削减 AI 成本，核心涵盖衡量每完成任务成本、选择合适模型与推理努力（Reasoning Effort），以及利用 Prompt Caching、编程式工具调用（Programmatic Tool Calling）和 Batch API 等降本手段。 ## TL;DR OpenAI 部署工程团队指出，开发者优化 AI 成本时应告别单一的每 Token 标价误区，转向考核包含人工介入与实际消耗在内的“每任务成本”（Cost per Task）。团队分享了如何利用帕累托曲线在准确率和成本之间找到最优模型配置，并探讨了“Luna Maxing”等高性价比选型实践。随后，演讲重点剖析了四大降本杠杆：Prompt Caching、编程式工具调用、合理控制推理努力（Reasoning Effort）以及 Batch API，并结合 Clio 与 Blitzy 等真实案例展示了如何实现高达近 90% 的成本缩减。最后，讲者介绍了显式断点与缓存诊断 API 等最新工具，并为开发者梳理出一套以代表性评测集为核心的单变量渐进迭代落地指南。 ## Atmosphere & style 整场演讲在室外舞台轻松展开，主讲人与搭档之间默契互动、风趣幽默，穿插着如“航班取消与低效 AI 客服”等亲历趣事，现场笑声与共鸣不断。讲解风格兼具前沿架构的专业严谨与面向工程实战的通俗务实，既有发人深省的现场提问，也有极具说服力的客户降本数据。整体节奏层层递进、干货密集，给观众带来一种清晰破除虚荣指标迷思、重获工程掌控力的踏实感与启发感。 ## Bulletin - [00:58](https://www.youtube.com/watch?v=_nmlHbSB8kM&t=58s) 开发者应将核心评估指标从“每 Token 成本”转向“每任务成本”，避免因低单价模型消耗更多 Token 或引发昂贵的人工介入而反向推高总开销。 - [08:42](https://www.youtube.com/watch?v=_nmlHbSB8kM&t=522s) OpenAI 推出超越模型的四大成本优化杠杆：Prompt Caching（输入缓存最高优惠 90%）、程序化工具调用（PTC）、动态推理努力调节（Reasoning Effort）及非实时 50% 折扣的 Batch API。 - [10:17](https://www.youtube.com/watch?v=_nmlHbSB8kM&t=617s) 程序化工具调用允许将中间数据的聚合与比对下放给轻量代码，法律 AI 平台 Clio 藉此在保持质量的同时将 Prompt Token 消耗直降 38%。 - [18:43](https://www.youtube.com/watch?v=_nmlHbSB8kM&t=1123s) Prompt Caching 迎来重大升级：显式断点与 Explicit-only 模式支持精确锁定缓存，且动态调整 Reasoning Effort 和可用工具集不再会击穿现有缓存，同时官方推出了专用的管理面板与诊断 API。 ## Key takeaways - [00:58](https://www.youtube.com/watch?v=_nmlHbSB8kM&t=58s) 评估 AI 成本的首要误区是仅关注官网的每 Token 标价；真正决定总支出的是包含重试、冗余输出与人工介入在内的“每任务成本”（Cost per Task）。 - [00:58](https://www.youtube.com/watch?v=_nmlHbSB8kM&t=58s) Perplexity 与 Notion 的实测案例表明，迁移至单价可能更高的先进模型（如 6 Astra 或 5.6）反而能通过更高的准确率和更少的 Token 消耗实现每任务成本减半。 - [04:40](https://www.youtube.com/watch?v=_nmlHbSB8kM&t=280s) 模型选型应基于帕累托前沿（Pareto Curve），在明确业务准确率底线的前提下，绘制成本与准确率的平衡曲线以寻找最优配置边界。 - [06:33](https://www.youtube.com/watch?v=_nmlHbSB8kM&t=393s) “Luna Maxing”策略正在兴起：为轻量级模型 Luna 配置高推理努力，可满足 80% 至 90% 的常规开发任务，表现足以媲美甚至超越大型模型。 - [08:42](https://www.youtube.com/watch?v=_nmlHbSB8kM&t=522s) 在架构层面控制成本依赖四大核心杠杆：Prompt Caching、程序化工具调用（Programmatic Tool Calling）、推理努力调节及 Batch API。 - [10:17](https://www.youtube.com/watch?v=_nmlHbSB8kM&t=617s) 程序化工具调用让模型生成或驱动轻量脚本处理多工具返回的原始数据，仅向 LLM 回传精简摘要，有效防止上下文膨胀并降低 API 调用开销。 - [12:55](https://www.youtube.com/watch?v=_nmlHbSB8kM&t=775s) 推理努力（Reasoning Effort）并不等同于回答长度；对于普通路由等简单任务应默认从低推理努力起步，仅对核心难题分配深度思考资源。 - [12:55](https://www.youtube.com/watch?v=_nmlHbSB8kM&t=775s) 对延迟不敏感的处理场景（如批量文档分析），使用 OpenAI Batch API 可在 24 小时处理窗口内直接换取 50% 的 Token 成本减免。 - [15:29](https://www.youtube.com/watch?v=_nmlHbSB8kM&t=929s) Prompt Caching 最高可节省 90% 的输入成本，其设计的黄金法则是保持提示词前缀长期高度一致，以最大化复用预处理结果。 - [15:29](https://www.youtube.com/watch?v=_nmlHbSB8kM&t=929s) 软件开发平台 Blitzy 通过将单次结构化输出改造为循环工具调用，将缓存命中率提升至 90%，在扩大上下文的同时使综合成本降低 87%。 - [18:43](https://www.youtube.com/watch?v=_nmlHbSB8kM&t=1123s) 新一代模型支持显式断点（Explicit Breakpoints）与 Explicit-only 模式，赋予开发者精确指定缓存边界的能力，避免易变数据污染缓存空间。 - [20:02](https://www.youtube.com/watch?v=_nmlHbSB8kM&t=1202s) 运行时动态修改推理努力（Reasoning Effort）或工具白名单（Allowed Tools）现已与缓存前缀解耦，不再会导致已有的 Prompt Cache 失效。 - [20:02](https://www.youtube.com/watch?v=_nmlHbSB8kM&t=1202s) OpenAI 推出了官方 Prompt Caching Dashboard 与缓存诊断 API（Diagnostics API），帮助开发者监控全局命中率并精准定位 Cache Miss 的发生位置。 - [21:21](https://www.youtube.com/watch?v=_nmlHbSB8kM&t=1281s) 工程落地的最佳实践是构建代表性评估数据集，通过单变量控制法逐步验证改动，持续运行 Evals 并锁定坐落在帕累托最优线上的配置方案。 ## Detailed walkthrough 根据 DevDay 2026 的演讲内容，以下是第一部分会议（优化 AI 成本与模型选择策略）的详细内容 walkthrough： ### [00:13](https://www.youtube.com/watch?v=_nmlHbSB8kM&t=13s) 会议开场与主题介绍 Arty 在室外舞台上开启了关于优化 AI 成本的小组讨论，邀请了来自 AI 部署工程团队的 Mandeep 和 Sapto 上台分享。讨论的核心围绕如何正确看待成本指标、如何为特定任务选择合适的模型，以及超越模型本身的各项成本优化杠杆，并结合 OpenAI 真实客户的案例进行剖析。 ### [00:58](https://www.youtube.com/watch?v=_nmlHbSB8kM&t=58s) 纠正首要误区：从“每 Token 成本”转向“每任务成本” Mandeep 指出，开发者在优化成本时常犯的第一个错误是看错了指标。大家往往只盯着官网列出的输入和输出 token 列表价（Cost per Token），但实际上，单价便宜的模型可能会带来更高的总成本。Mandeep 解释了背后的两大原因：一是便宜的模型可能会消耗或产生更多的 token 才能完成任务；二是模型可能无法独立完成任务，中途需要人工介入。因此，企业和开发者真正应该关注的指标是**“每任务成本”（Cost per Task）**。 Mandeep 分享了一个亲身经历的生动故事：上周他去纽约出差遭遇航班取消，在机场排队时无法在网上找到合适的替代航班，于是拨打了航空公司的客服电话。接听的是一个 AI 智能客服，由于无法理解他“需要改签但不想坐红眼航班”的具体诉求，最终不得不转由人工客服介入，人工最终通过休斯顿帮他重新订了航班，让他赶上了团队的 offsite。虽然航空公司使用的 AI 模型本身按 token 计算单价很便宜，但算上人工介入的时间和成本，整个任务的总成本其实非常高。因此，航空公司应该考核的是包含人工时间在内的“每任务成本”。 Arty 随后分享了两个真实客户案例： - **Perplexity**：在研究基准测试中，使用 **6 Astra** 产生了比前代模型高出 9% 的准确率，同时成本减半。虽然 6 Astra 的 token 单价可能更高，但从每任务成本来看，它消耗更少的 token 就能高效搞定任务。 - **Notion**：将模型从 5.5 迁移到 5.6 后，不仅准确率更高，每任务成本也降低了一半。 - 关键指标：应关注“每任务成本”（Cost per Task），而非单一的“每 Token 成本”。 - 核心考量：低单价模型可能因低效、多消耗 token 或导致频繁人工介入，反而推高整体开销。 ### [04:40](https://www.youtube.com/watch?v=_nmlHbSB8kM&t=280s) 如何为任务精准匹配模型与帕累托曲线 在讨论完正确的衡量指标后，Arty 将话题引向了“如何为特定任务选择合适的模型”。Mandeep 建议，首先要**清晰定义任务以及业务或开发者对该任务的准确率期望**（例如航空公司可以设定“希望 80% 的咨询由聊天机器人自主解决”）。明确目标后，通过寻找能够以最佳成本达成该准确率的模型架构和配置来实现平衡。 在此过程中，Mandeep 引入了**“帕累托曲线”（Pareto Curve）**的概念：即通过绘制相互制约的两个变量（准确率与成本），尝试不同的模型、设置和配置，找出能够在满足准确率阈值的同时实现最低成本的最优边界。 - 步骤一：明确任务目标及期望的准确率指标。 - 步骤二：利用帕累托曲线测试并绘制不同模型配置在准确率与成本之间的平衡点。 ### [06:33](https://www.youtube.com/watch?v=_nmlHbSB8kM&t=393s) 模型选择策略与“Luna Maxing”现象 针对如何选择具体的模型系列，Mandeep 介绍了 OpenAI 现有的产品组合：从具备顶级智能的 frontier 模型 **6 Astra**、极具性价比且高效的 **6.1 Sol**，到轻量高效的 **Luna**。 对于简单分类或从文档中提取几个字段的轻量任务，建议尽早测试更高效的小模型；而对于需要高度智能的任务，则从 6 Astra 开始。随后 Arty 提到了社区里流行的术语——**“Luna Maxing”**。Mandeep 解释说，很多客户惊喜地发现，当给 **Luna** 赋予“高”或“超高”的推理努力（Reasoning Effort）时，它的表现往往能击败许多前代甚至竞争对手的大模型。许多客户将 Luna 作为开发者编码的默认首选（80% 至 90% 的工作流都能完美胜任），只有在确实需要更高智能时才切换到其他模型。此外，今天还发布了专门针对简单任务的 Decisions API，让模型家族进一步壮大。 - **Luna Maxing**：开启高推理努力配置的 Luna 可以替代许多大型模型，满足 80%-90% 的日常开发和工作流需求。 - 策略：以 Luna 作为常规主力，仅在必要时调用高级模型。 ### [08:42](https://www.youtube.com/watch?v=_nmlHbSB8kM&t=522s) 超越模型的四大成本优化杠杆 在确认了正确的指标和模型选择策略之后，Sapto 指出，即使在同一模型家族中，成本也会因为处理的信息量和输出需求而大不相同。为了在保持质量的同时控制成本，Sapto 介绍了**四大优化杠杆**： 1. **Prompt Caching（提示词缓存）** 2. Programmatic Tool Calling（程序化工具调用） 3. Reasoning Effort（推理努力控制） 4. Batch 或 Flex API 处理 在深入探讨第一个杠杆——提示词缓存时，Sapto 以公司客服助手为例：当助手处理每个客户时都需要用到相同的公司政策和共享文档，提示词缓存技术能够复用这些共享信息的预处理过程，而模型只需处理提示词中的新信息并生成最新回答。 <ElicitationsGroup message="如需深入了解后续的优化杠杆及具体实现："> <Elicitation label="了解程序化工具调用与提示词缓存的具体运作方式" query="详细讲解程序化工具调用以及提示词缓存的技术原理和最佳实践。"/> <Elicitation label="探索推理努力（Reasoning Effort）与 Batch API 的成本节省策略" query="介绍如何通过调整推理努力和使用 Batch API 进一步降低 AI 应用成本。"/> </ElicitationsGroup> ### [10:17](https://www.youtube.com/watch?v=_nmlHbSB8kM&t=617s) 第二项杠杆：程序化工具调用（Programmatic Tool Calling） 在简要回顾了 Prompt Caching 之后，演讲者引出了降低 AI 成本的第二个核心杠杆——程序化工具调用（Programmatic Tool Calling）。为了让观众身临其境地理解，讲者抛出了一个线上订单履约系统的场景互动：假设有一个智能体（Agent）正在处理海量订单，需要核实是否有足够的库存来完成交付。在这个流程中，Agent 可以连续调用多个工具，汇总订单、检查缺货与即将入库的情况，然后将所有原始结果全部塞回大语言模型（LLM）进行推理。讲者向台下提问："大家真的认为模型必须亲自阅读并对比所有的原始中间结果吗？"现场观众纷纷摇头表示否定。 ```mermaid flowchart TD accTitle: 程序化工具调用架构对比 accDescr: 对比传统多次往返调用与通过小程序打包摘要结果的程序化工具调用流程 subgraph Traditional["传统工具调用流程"] direction TB T_LLM["LLM 推理引擎"] <-->|"多次往返传递全部原始数据"| T_Tools["多个底层工具 / API"] end subgraph PTC["程序化工具调用流程"] direction TB P_LLM["LLM 推理引擎"] -->|"下发轻量级脚本 / 代码"| P_Code["轻量级编排程序"] P_Code <-->|"调用、聚合、对比"| P_Tools["底层工具与数据"] P_Code -->|"仅返回精简打包报告"| P_LLM end ``` 这种普遍的直觉正是程序化工具调用的精髓所在：虽然某些场景确实需要模型进行深度判断，但在大量常规数据比对、组合和校验中，完全没必要把每项工具产生的中间数据都回传给模型。程序化工具调用的机制是让模型生成或驱动一段轻量级程序，由这段代码在后台统筹工具调用、清洗和比对结果，最终仅将精简后的打包报告返回给 LLM。这种架构彻底避免了模型上下文被冗余中间数据撑爆，同时也大幅削减了多步工作流中的 API 调用次数。 - **客户案例（Clio）：** 知名法律 AI 平台 Clio（专注于为律师事务所提供云管理系统）在多步骤文档分析场景中引入了程序化工具调用。在完全没有质量损失的前提下，其 Prompt Token 消耗直降了 38%。 - **核心逻辑：** 仅通过改变应用程序与大模型之间的交互机制，就达成了近四成的成本优化。 --- ### [12:55](https://www.youtube.com/watch?v=_nmlHbSB8kM&t=775s) 第三与第四项杠杆：推理力度（Reasoning Effort）与 Batch API 讲者紧接着拆解了第三项杠杆——推理力度（Reasoning Effort）。大家通常熟悉滑块形式的推理设置，但讲者特别指出一个常见误区：推理力度绝不等同于回答的最终长度（Answer Length），即使是一个极简短的回答，其推导逻辑也可能极其复杂，耗费海量的推理 Token。对于普通的分发路由任务（Routing），最佳实践始终是先从较低的推理力度开始，通过自动化评估（Evals）确认质量是否达标；只有面对那些高难度、强差异化的核心任务时，才值得为模型深思熟虑所消耗的额外推理 Token 买单。 第四项杠杆则是针对延迟不敏感场景的 Batch API（或称 Flex API 处理）。讲者向开发者发问："面对成千上万份待处理的文档，你愿意等待多久？"对于那些不需要实时响应的工作流，OpenAI Batch API 提供最长 24 小时的处理窗口，但其 Token 计费比常规同步请求足足便宜了 50%。 - **应用构建指导原则：** 在为系统选型时，不应只问"我该选哪个模型家族"，还要问"我打算让模型分担多少计算量"。 - **迭代闭环：** 紧跟单个具体任务的输入输出全生命周期，排查重复流程与冗余工具调用，每次仅修改单一变量并跑通 Evals 测试。 - **终极目标：** 优化的本质不是单纯地压低 Token，而是"在可接受的延迟与更低成本下，达成可靠的业务成果"（a successful outcome through lowering cost and acceptable latency）。 --- ### [15:29](https://www.youtube.com/watch?v=_nmlHbSB8kM&t=929s) 四大成本杠杆总结与 Prompt Caching 的深层机制 在 Sapto 讲完四大杠杆后，搭档 Mandeep 进行了精辟的阶段总结，进一步梳理这套控本组合拳： 1. **Prompt Caching：** 官方价格表中缓存输入的费用最高可减免 90%，具备极大的成本杀伤力； 2. **程序化工具调用：** 将数据预处理和上下文对比下放给轻量代码，不把垃圾信息塞进 Context Window； 3. **推理力度调节：** 在 API 中灵活调整，且在最新架构下调整推理力度已不会击穿已有的 Prompt Cache； 4. **Batch / Flex API：** 利用非紧急请求换取半价折扣。 由于 Prompt Caching 能够带来高达 90% 的降本空间，两位讲者决定就其常见误区与优化手法进行深度展开。当 Sapto 询问台下"有多少人在使用 Prompt Caching"时，Mandeep 风趣地补充道："所有人都在用，因为它是默认开启的！"台下顿时响起会心的笑声。 然而，默认开启并不意味着用到了极致。许多开发者并没有充分发挥其潜能。Prompt Caching 的黄金准则在于：**尽可能保证 Prompt 前半部分内容的长期一致性**。只有这样，模型才能复用对共享指令的预处理结果，而后续不同的动态信息则进行全新的单独处理。 - **客户案例（Blitzy）：** AI 软件开发企业 Blitzy 将原有的单次结构化输出模式改造为循环工具调用体系后，其缓存命中复用率从 24% 飙升至 90%。在数千次生产环境调用中，Blitzy 实现了 8.5 倍更少的 Prompt Token 输出量，在上下文处理量提升 2.2 倍的同时，综合成本较以往模型方案下降了 87%。 --- ### [18:43](https://www.youtube.com/watch?v=_nmlHbSB8kM&t=1123s) 显式断点（Explicit Breakpoints）与 Explicit-Only 模式 Mandeep 针对 Prompt Caching 提出了一个现实挑战：对于指令频繁变动或跨查询差异巨大的复杂工作流，开发者又该如何维系缓存命中率？ Sapto 解释了缓存技术的演进历程。在早期模型家族中，系统完全依赖基于固定的 Token 区间自动选取缓存切分点，开发者除了机械地把静态文字往最前面堆叠之外，缺乏主动控制权。但在新一代模型体系中，OpenAI 引入了**显式断点（Explicit Breakpoints）**功能。 用最浅显的话来说，显式断点就是让开发者明确告诉系统："我就要缓存到这一行为止"。例如在长程代码编写场景中，前置的代码规约和系统提示词长期不变，而具体的业务要求则在末尾频繁变动。开发者可以在不变的静态块之后、变动任务之前精准设置断点，从而锁死静态资产的缓存收益。对于追求绝对确定性与极致工程控制的开发者，系统还提供了 **Explicit-only 模式**，允许自由圈定并严格指定哪一段内容才准许进入缓存体系。 ### [20:02](https://www.youtube.com/watch?v=_nmlHbSB8kM&t=1202s) Prompt Caching 的进阶灵活性与全新诊断工具 在构建对话与 Agent 系统时，Prompt Caching（提示词缓存）的精准控制至关重要。开发者最希望避免的情况是将频繁变动且后续不会复用的动态内容写入缓存。针对这一痛点，演讲者指出当前模型系列不仅保留了之前的所有控制项与功能，还引入了一项关键突破：开发者现在可以在持续的多轮对话中随时动态调整模型的推理努力程度（Reasoning Effort），而完全不会破坏（break）现有的上下文缓存。 Aarti 随即补充了一个许多开发者容易忽视的隐藏特性：除了推理努力程度外，开发者在运行时通过参数调整传递给模型的工具集（例如修改 `allowed_tools` 等配置）同样不会导致缓存失效。这意味着开发者既能按需灵活调整每轮对话的思考深度与可用工具，又能始终维持极高的缓存命中率（Cache Hit Rate）。为了支持这一体系，OpenAI 在底层工具链上持续加大投入，并最新发布了两大重磅功能： 1. **Prompt Caching 管理面板（Dashboard）**：集成在后台管理网页中，允许开发者直观查看、衡量和追踪旗下所有应用在生产环境中的实际缓存命中表现。 2. **缓存诊断 API（Diagnostics API）**：开发者传入两条不同的请求 Query，该 API 能够精准比对并定位缓存发生未命中（Cache Miss）的具体位置与断点，从而彻底消除开发者优化 Prompt 结构时的盲目猜测。 ```mermaid flowchart TD accTitle: Prompt Caching 优化与诊断架构 subgraph Conversation["动态对话流 (保持 Cache 命中)"] A["多轮上下文 Prefix"] --> B["动态调整 Reasoning Effort"] A --> C["动态调整 Allowed Tools"] B --> D["Prompt Cache 持续生效"] C --> D end subgraph Observability["可观测性与诊断工具"] E["Admin 仪表盘<br/>(追踪全应用 Cache Hit Rate)"] F["Diagnostics API<br/>(输入对比 Query -> 定位 Miss 断点)"] end D --> Observability ``` - **动态参数解耦**：推理努力程度（Reasoning Effort）与工具白名单（Allowed Tools）均实现与缓存前缀解耦，修改时不会导致 Prefix Cache 失效重新计算。 - **排除易变内容**：强调缓存路径应精细化管理，避免频繁变动、缺乏复用价值的高频变动数据污染缓存空间。 - **可视化与可调试性**：通过官方 Dashboard 监控全局命中率，借助 Diagnostics API 进行细粒度 Diff 诊断，成为业务降本的核心杠杆。 --- ### [21:21](https://www.youtube.com/watch?v=_nmlHbSB8kM&t=1281s) 帕累托前沿与落地行动指南：如何停止为智能多付成本 在演讲收尾阶段，Aarti 对整场核心策略进行了系统性复盘。团队反复强调，衡量大模型性价比的唯一正确指标是“完成单项任务的实际成本”（Cost per completed task），而非单纯盯紧 API 列表上的每百万 Token 标价。在模型选型上，开发者的终极目标是在“成本”与“准确率”之间找到坐落在帕累托最优前沿（Pareto Frontier）上的模型组合。此外，团队归纳了四大核心降本增效杠杆：Prompt Caching（提示词缓存）、Programmatic Tool Calling（编程式工具调用）、Reasoning Effort（推理努力程度微调）以及 Batch API。 面对“听完演讲后明天回工位应该先做什么”的实际落地问题，Mandeep 和 Sapto 给出了一套结构化的工程实施路径。Mandeep 建议首先明确业务核心目标，划定业务能够容忍的准确率底线，并提炼出一套能真实反映生产环境调用特征的“代表性评估任务集”（Representative Task Set）。随后，遍历不同模型规格、推理深度和 Prompt 缓存策略，绘制出属于自己业务的帕累托前沿曲线，从而挑选出满足业务指标且综合成本最低的模型配置线（Model Configuration Harness）。Sapto 则特别叮嘱了工程迭代的心法：“一次只改一个变量（Test one change, one improvement）”，每当验证出一项优化便固化该配置，并随着模型版本与业务需求的发展，坚持常态化运行评估套件（Evals）。 ```mermaid flowchart TD accTitle: AI 成本与准确率优化落地流程 subgraph Phase1["阶段一：目标定义与数据集"] A["明确具体任务<br/>设定业务准确率阈值"] --> B["构建代表性评估任务集<br/>(反映真实使用场景)"] end subgraph Phase2["阶段二：实验与前沿寻优"] B --> C["组合测试不同模型/配置<br/>(Prompt Caching / 推理努力度)"] C --> D["绘制帕累托前沿<br/>(Pareto Frontier: 准确率 vs 成本)"] end subgraph Phase3["阶段三：锁定与敏捷迭代"] D --> E["选定最优 Model Harness"] E --> F["单变量递进改进<br/>(One change at a time)"] F --> G["随系统演进持续运行 Evals"] end ``` - **核心选型法则**：告别单价虚荣指标，牢牢锚定“完成每个有效任务的端到端成本”与帕累托前沿平衡点。 - **四大降本抓手**：Prompt Caching、编程式工具调用（Programmatic Tool Calling）、动态调节推理深度（Reasoning Effort）与非实时批处理（Batch API）。 - **落地方法论**：先定阈值，再建代表性 Eval 集，跨配置扫参绘制成本-精度曲线，最后遵循单变量控制法进行长期敏捷演进。 ## Notable quotes > [00:58](https://www.youtube.com/watch?v=_nmlHbSB8kM&t=58s) 大家往往只盯着官网列出的输入和输出 token 列表价（Cost per Token），但实际上，单价便宜的模型可能会带来更高的总成本。 — Mandeep > [00:58](https://www.youtube.com/watch?v=_nmlHbSB8kM&t=58s) 企业和开发者真正应该关注的指标是“每任务成本”（Cost per Task）。 — Mandeep > [06:33](https://www.youtube.com/watch?v=_nmlHbSB8kM&t=393s) 当给 Luna 赋予“高”或“超高”的推理努力（Reasoning Effort）时，它的表现往往能击败许多前代甚至竞争对手的大模型。 — Mandeep > [10:17](https://www.youtube.com/watch?v=_nmlHbSB8kM&t=617s) 大家真的认为模型必须亲自阅读并对比所有的原始中间结果吗？ — Sapto > [12:55](https://www.youtube.com/watch?v=_nmlHbSB8kM&t=775s) 面对成千上万份待处理的文档，你愿意等待多久？ — Sapto > [12:55](https://www.youtube.com/watch?v=_nmlHbSB8kM&t=775s) 在为系统选型时，不应只问“我该选哪个模型家族”，还要问“我打算让模型分担多少计算量”。 — Sapto > [12:55](https://www.youtube.com/watch?v=_nmlHbSB8kM&t=775s) a successful outcome through lowering cost and acceptable latency — Sapto > [15:29](https://www.youtube.com/watch?v=_nmlHbSB8kM&t=929s) 所有人都在用，因为它是默认开启的！ — Mandeep > [15:29](https://www.youtube.com/watch?v=_nmlHbSB8kM&t=929s) 尽可能保证 Prompt 前半部分内容的长期一致性。 — Sapto > [18:43](https://www.youtube.com/watch?v=_nmlHbSB8kM&t=1123s) 我就要缓存到这一行为止 — Sapto > [21:21](https://www.youtube.com/watch?v=_nmlHbSB8kM&t=1281s) Test one change, one improvement — Sapto ## Mentioned OpenAI, Arty, Mandeep, Sapto, Perplexity, Notion, 6 Astra, 5.5, 5.6, 6.1 Sol, Luna, Decisions API, Clio, Blitzy, Prompt Caching, Programmatic Tool Calling, Batch API, Diagnostics API ## Video description Cut AI costs while preserving quality. Learn to measure cost per completed task, choose models and reasoning effort, and use prompt caching, Programmatic Tool Calling, and the Batch API. [Source](https://www.youtube.com/watch?v=_nmlHbSB8kM)

Source quotes

[0:58] Mandeep: 大家往往只盯着官网列出的输入和输出 token 列表价（Cost per Token），但实际上，单价便宜的模型可能会带来更高的总成本。 [0:58] Mandeep: 企业和开发者真正应该关注的指标是“每任务成本”（Cost per Task）。 [6:33] Mandeep: 当给 Luna 赋予“高”或“超高”的推理努力（Reasoning Effort）时，它的表现往往能击败许多前代甚至竞争对手的大模型。 [10:17] Sapto: 大家真的认为模型必须亲自阅读并对比所有的原始中间结果吗？ [12:55] Sapto: 面对成千上万份待处理的文档，你愿意等待多久？ [12:55] Sapto: 在为系统选型时，不应只问“我该选哪个模型家族”，还要问“我打算让模型分担多少计算量”。 [12:55] Sapto: a successful outcome through lowering cost and acceptable latency [15:29] Mandeep: 所有人都在用，因为它是默认开启的！ [15:29] Sapto: 尽可能保证 Prompt 前半部分内容的长期一致性。 [18:43] Sapto: 我就要缓存到这一行为止 [21:21] Sapto: Test one change, one improvement

Source quotes

[0:13] Mandeep: A model that is cheap on cost per token basis may not be the cheapest model on cost per task basis. [0:13] Mandeep: a soft spot for customer support chatbots [12:55] Sapto: How much are you willing to wait? [12:55] Sapto: Which model family should I use? [12:55] Sapto: How much work am I going to make the model do? [16:41] Mandeep: It's all of us. It's enabled by default. [16:41] keep the static beginning of your prompt strictly consistent. [18:43] Sapto: Cache up to this exact point.

Original transcript

[0:13] Hello everyone, hope you're [0:15] having an amazing DevDay so far. [0:17] On the outdoor stage now we're going [0:19] to have a panel side discussion [0:21] with two people from our AI [0:23] deployment engineering team to talk [0:25] about how to optimize AI costs. [0:28] So we're going to talk about [0:29] things like you know what is [0:30] the right what is the right metric [0:32] to look at even when you think [0:34] of optimizing costs, how do you [0:35] right-size a model for a particular [0:37] task, levers beyond the model itself, [0:39] and throughout we'll be drawing [0:41] on examples from real customers [0:43] that OpenAI works with. [0:45] So with that please welcome [0:46] to the stage Mandeep and Sapto. [0:53] Thank you for being here. [0:55] Thanks for the introduction, Arty. [0:57] Thank you. Of course. [0:58] So let's start with the number one [1:00] mistakes that we see developers make [1:02] which is starting with even the metric. [1:04] You know most people think of cost [1:06] and look at the list price [1:07] of the input and the output [1:09] tokens on our website. [1:10] But talk to us Mandeep about [1:12] what do those numbers miss? [1:14] Yeah, by the way quick [1:16] introduction Mandeep. [1:17] I lead the applied AI engineering team [1:19] based out of here in San Francisco [1:20] and welcome to the DevDay. [1:22] A model that is cheap on cost per [1:25] token basis may not be the cheapest [1:28] model on cost per task basis. [1:30] And there are many reasons for this. [1:33] Two main reasons are a cheaper model [1:36] may consume more tokens or produce [1:37] more tokens to complete the task [1:39] or the model may not complete [1:41] the task at all and a human [1:43] has to intervene to complete the task. [1:45] So the metric that developers [1:48] and businesses should be looking at is [1:51] cost per task not cost per token. [1:54] Got it. [1:55] So cost per task not cost per token. [1:58] Can you give us a few examples [1:59] of what this looks like in practice? [2:02] Yes, I'll give a very personal example. [2:04] Last week I had to travel to New [2:06] York City, had to take a flight. [2:08] Who here has been stranded on [2:10] an airport with a canceled flight? [2:12] And well, I guess that misfortune [2:15] has happened to most of us [2:17] at one point in our life. [2:18] And I look at the flights online [2:21] change, but couldn't find the flight [2:23] that actually matches the schedule. [2:24] I had to call the airline and I'm [2:27] greeted by a customer support chatbot. [2:30] Now I have a soft spot [2:32] for customer support chatbots [2:33] because we're in business of building [2:35] customer support chatbots. [2:37] So I very patiently explained [2:38] to the AI chatbot that I [2:40] wanted to reschedule my flight. [2:42] I did not want a red eye, [2:44] but I could not make it understand. [2:46] So I literally had to say, [2:48] I need to speak with a human. [2:50] So a human had to intervene and complete [2:52] the transaction, answer my query, [2:55] book me in a flight through Houston, [2:56] and I arrived before my team offsite. [2:59] So while the model that the airline [3:02] may have been using may have been [3:03] cheaper on cost per token basis, [3:06] but the cost to actually complete [3:08] the task was the cost of the tokens [3:10] plus the human that had to intervene [3:12] to actually complete the task. [3:14] So the metric that airlines should [3:16] have been looking at is the cost [3:18] per task, which includes the human, [3:20] the time that human had to spend [3:22] to complete the task as well. [3:24] Unfortunately, a very relatable example, [3:26] and I hope the rest of you [3:27] had a smoother journey here to DevDay. [3:29] Yeah, and I'll quote a couple [3:31] of customer examples. [3:33] So perplexity, we're familiar [3:34] with the product, a customer we [3:35] work very closely with. [3:37] On their research benchmark, [3:39] they reported six Astra [3:41] produces 9% more accurate [3:43] results at half the cost. [3:45] So while compared [3:47] with the previous model. [3:49] So while six Astra may be more [3:51] expensive on token per cost basis, [3:54] but on a cost per task basis, [3:57] it's actually much more efficient. [3:58] It uses fewer tokens [3:59] and gets the task done. [4:01] Likewise, Notion, a customer we [4:03] work very closely with, we're [4:04] familiar with the Notion product, [4:06] I use it every day. [4:07] They are very good at using [4:09] the migrating [4:10] to the latest series of models. [4:11] For example, they migrated from 5.5 [4:14] to 5.6 sold, reported better [4:16] accuracy, half the cost per task. [4:19] Makes sense. [4:20] You've mentioned, you [4:21] know, cost per task. [4:21] So is the task really done successfully? [4:24] But then the other thing you called [4:25] out is token efficiency. [4:26] We continue to invest a lot [4:27] in making not just our models [4:29] performant, but also token efficient. [4:31] And that really plays in. [4:33] Speaking of, let's start with, you know, [4:35] we've talked about the right metric [4:37] here, which is cost per task. [4:39] The biggest decision you have to make, [4:41] of course, is which model to pick. [4:42] So how do you think through right [4:44] sizing a model for a particular task? [4:47] Yeah. [4:48] So when we advise developer on business [4:51] on right sizing the model, we recommend [4:53] begin with crisply defining the task [4:56] and what accuracy business [4:58] or a developer expects from that task. [5:01] So from the airline example, it could [5:04] have been we want 80% of inquiries [5:06] to be resolved by the chatbot. [5:08] So if they had used, for example, [5:11] GPT Live with 6.0 as a delegate [5:13] model, they might have been able [5:15] to resolve my inquiry without [5:16] delegating it to a human. [5:18] So begin with define the task [5:20] and then define what is the accuracy [5:22] we expect from the task. [5:25] So the goal here is [5:26] to find the model harness [5:29] and the configuration that can [5:32] achieve that accuracy at [5:34] the best cost possible. [5:36] Here, it'll be good to introduce [5:38] a concept, Pareto curve. [5:40] Many of you may have heard of it. [5:41] It's essentially plotting two [5:43] variables that are dependent [5:45] on each other and you have [5:46] to optimize for one or the other. [5:49] In this case, the two variables that we [5:51] would focus on is accuracy and cost. [5:54] The idea here is try different models, [5:57] try different settings, different [5:59] configurations, different harnesses, [6:01] figure out what is the accuracy [6:03] per task with those configurations [6:05] and figure out which model configuration [6:07] and harness meets that accuracy [6:10] threshold with the best cost possible. [6:12] Got it. [6:13] So you touched on, you know, [6:15] there are some tasks where you might [6:16] know upfront what model to use. [6:18] Like let's say if it is a simple, [6:20] simpler task, you might want [6:21] Luna to let's say route. [6:23] Or if it's a more complex [6:24] example, you might want Astra [6:26] to handle a lot of the planning. [6:27] But what if it is not clear upfront [6:29] what model to use for a task? [6:32] Yes. So while luckily here at OpenAI, we [6:35] have models that are at the Pareto [6:36] frontier for accuracy and cost. [6:39] So starting with six Astra, [6:41] which is our frontier model, very [6:44] high intelligence, 6.1 Sol provides [6:47] a very high level of intelligence, [6:48] very cost efficient per task. [6:50] And then Luna and we've [6:52] heard Luna maxing. [6:54] So to your point, if the task [6:56] is somewhat, many tasks you would [6:59] intuitively know which model to use. [7:01] If it is simple classification, [7:02] or it is extracting a few [7:04] fields out of a document, [7:06] you would know that Luna is probably [7:08] so I would in those cases, [7:09] I would test a smaller, more efficient [7:12] model early on in the process. [7:14] If the task is more [7:16] requires more intelligence, [7:18] I would begin with six Astra. [7:19] And then as I test different models, [7:22] I would plot that Pareto frontier [7:24] and figure out what is that cutoff where [7:27] the model and the configuration meets [7:29] the performance threshold with the best [7:32] cost efficiency per task. [7:34] Makes sense. [7:35] So start with evaluation sets, [7:37] try to find the model that meets [7:39] the accuracy threshold that you want, [7:41] and then try to optimize for costs. [7:43] Yes, exactly. [7:44] Got it. You also mentioned Luna maxing. [7:46] How many people have heard of that term? [7:49] Okay, quite a few people. [7:51] But the overall takeaway there is [7:53] Luna is actually quite capable model. [7:55] And it's found to beat a lot [7:57] of models, like larger models [7:59] from previous generations. [8:00] But yeah, Mandib, do you want to quickly [8:02] tell us about what Luna maxing is? Yes. [8:04] So a lot of customers are [8:06] pleasantly surprised that Luna [8:08] with a high or extra high reasoning [8:11] effort can actually beat many [8:13] bigger models, especially from [8:15] previous generation or competitors [8:16] to a lot of customers offer [8:19] Luna as the primary way [8:21] for their developers to code [8:23] and only default to other models [8:26] when a higher intelligence is needed. [8:28] So we hear that a lot from [8:30] our customers that Luna [8:31] for 80-90% of their workflows, [8:34] it works really well for our customers. [8:36] So don't discount [8:37] Luna and give it a shot. [8:39] It might do more than you [8:40] think it can do. [8:41] And today we also announced Decisions [8:43] API, also for simpler tasks. [8:44] So the family grows. [8:47] So we've talked about the right [8:49] metric, cost per task, [8:50] that you want to optimize for. [8:51] We've talked about how do you [8:53] find the right model family, [8:54] or the model size for the task. [8:56] But that's of course [8:57] just one lever, right? [8:58] Finding the right model. [8:59] So Sapto, can you talk to us [9:01] about beyond the model, what are [9:03] other levers developers can use [9:04] to optimize costs for their application? [9:07] Yeah, great question, Arti. [9:08] Firstly, I would like [9:10] to tell like this, right? [9:13] Same model family can cost [9:15] very different amounts based on [9:17] the amount of information we process, [9:20] amount of information we pass [9:22] it through, and what the output [9:23] that we are seeking from it. [9:25] Now today I'm going to talk about [9:26] four ways, which where you can [9:29] control cost while also kind [9:30] of maintaining the quality. [9:32] So we are going to talk [9:33] about four levers here. [9:35] The first one is prompt caching, then [9:37] we have programmatic tool calling, third [9:39] one is reasoning effort, and the last [9:40] one is batch or flex API processing. [9:43] So let's talk about or dive deeper into [9:45] a first lever, that is prompt caching. [9:48] So a good starting point is [9:50] to take the repeated text [9:51] within your input prompt. [9:53] So let me give a quick example here. [9:56] So suppose you have a company [9:57] support assistant, right? [9:58] And it uses same company policies [10:01] and documents to process every customer. [10:04] Now prompt caching like works on [10:06] the concept where it can reuse [10:08] the pre-processing of those shared [10:11] information, while your model still [10:13] processes any kind of new information [10:15] in the prompt and produces a fresh [10:17] response each and every time. [10:19] So now that we talked about prompt [10:21] caching a little bit, I want [10:22] to go deeper into our second lever. [10:25] But actually this is a very important [10:26] one, so I'm going to ask all [10:27] of you a quick question here. [10:29] So I'm going to give you a quick [10:30] example or a scenario. [10:32] Suppose you have an online [10:33] ordering system, [10:35] and you want your agent, which kind [10:37] of fills or processes a lot of orders, [10:39] and you want your agent to find, [10:41] you know, if there's F stock [10:43] available to fill those orders. [10:45] Now how is it going to work? [10:46] You know, the agent can call [10:48] multiple tools, process the orders, [10:50] find what's missing, what's [10:52] arriving, and then pushes all [10:54] the results back to your LLM, [10:55] which is the reasoning engine. [10:57] Quick question to all of you. [10:58] Do you really think your model [11:00] needs to read and compare all [11:01] of those results by itself? [11:03] What do you all think? [11:05] Also, it's up to the mic, sorry. [11:08] Oh. [11:08] Yeah, what do you all think? [11:10] Like, does your model really [11:11] need to read and compare all [11:13] of those results by itself? [11:16] I'm seeing some shakes. [11:18] Okay. [11:19] Okay, probably not. [11:20] And that's the main [11:23] concept behind our second viewer, [11:26] that is programmatic tool calling. [11:29] So programmatic tool calling is, [11:31] like, for some scenarios, your model, [11:34] you can pass all of those tool [11:36] results back to your model for judgment. [11:38] But for some scenarios, there is [11:41] no need for you to pass those [11:42] tool results back to your model. [11:44] It can be calibrated, combined, [11:46] and then compared as well. [11:48] So programmatic tool calling is [11:49] a concept where your model runs [11:52] a small program, and that small [11:54] program kind of coordinates the tool [11:56] calls and processes the results [11:58] and shares the package report [12:01] back to your model for judgment. [12:03] With programmatic tool calling, [12:05] the thing that you're avoiding here [12:07] is there is no need for you [12:09] to send every piece of intermediary [12:11] data back to your model, [12:13] and you're also reducing the number [12:15] of model calls that your workflow [12:17] is making at every possible step. [12:19] Now, let me give a quick example. [12:22] This is a customer scenario, just [12:23] to show all of you the real-world impact [12:25] of using programmatic tool calling. [12:27] So we have one of our customers, Clio. [12:29] So it's a legal AI-powered company [12:31] which provides cloud management [12:32] software systems to law firms. [12:35] They reported in their multi-step [12:37] document analysis that they were [12:39] able to achieve 38% fewer PROM [12:41] tokens by using programmatic tool [12:43] calling with no loss of quality. [12:46] So this is a very important [12:47] thing to remember. [12:48] That's 38% fewer PROM tokens just [12:51] by changing how the application [12:52] interacts with the model. [12:55] So now that we talked about [12:56] a second lever, programmatic [12:57] tool calling, the third lever, [12:59] which is the reasoning effort, I think [13:01] all of you might already be aware [13:03] of it, but think of reasoning [13:05] effort as a setting to test. [13:07] The reasoning effort, like our main [13:09] recommendation, that reasoning effort is [13:11] very different from the answer length. [13:14] Like even like a short answer [13:15] can require a lot of reasoning [13:16] effort to generate it. [13:18] But one of our major guidance for all [13:20] of you here is whenever for any [13:22] kind of routing task, always start [13:24] with a lower reasoning effort level, [13:26] and then run your evals to see [13:28] whether the quality holds or not. [13:30] Now for any kind of difficult [13:32] or differentiated task, the extra [13:34] tokens is worth paying for. [13:36] But a most important thing to keep [13:38] in mind is it's very important [13:40] to run your evals iteratively [13:42] to help you make that informed decision. [13:45] So we talked about these three levers. [13:48] Let's talk about our last one, [13:50] that is batch or flex API processing. [13:53] Now this comes with a question, like [13:56] how much are you willing to wait? [13:57] So suppose you are processing, you know, [14:00] thousands of documents, and you have [14:02] the capacity to wait a little bit. [14:04] We have OpenAI batch APIs, which [14:06] provides a 24-hour processing window at [14:09] a 50% lower token pricing compared [14:12] to standard synchronous requests. [14:14] Now this is ideal for workflow where [14:17] you do not need an immediate [14:20] response back from the model. [14:22] So now that we talked about these [14:23] four levers, I want to, like, [14:26] give a quick thought to all [14:28] of you guys here, is whenever you [14:30] are building your AI applications, [14:32] and you have this question [14:33] in your mind, like, hey, like, [14:35] which model family should I use? [14:37] You should also ask an additional [14:39] question, that how much work am [14:41] I going to make the model do? [14:43] Our recommendation for all of you [14:45] is follow through one complete [14:46] task within your application, [14:48] see what went in, what [14:51] the output looks like, [14:53] you know, if there are any kind [14:54] of repetitive workflows, or if [14:57] there are any kind of, you know, [15:00] what you can say, like, some, there [15:03] are tool calls, the number of tool [15:04] calls that has been made here as well. [15:06] Now think of it, see what improves [15:09] the results, change one setting, [15:11] and then iterate, run your evals, [15:12] as your application and models [15:14] kind of evolves from there. [15:16] And another thing, another [15:18] important thing to keep in mind is, [15:20] it's not about lowering tokens [15:22] or having simply, like, pure tokens. [15:24] It's about having a successful [15:26] outcome through lowering cost [15:28] and acceptable latency. [15:29] Makes sense. [15:30] Thank you, Sapto. [15:31] That was a lot of different [15:32] levers, so let me quickly summarize [15:34] that for the audience, and we'll [15:35] dive a little deeper. [15:36] First, we talked about prompt caching. [15:39] This ends up being [15:39] really, really important. [15:40] If you even look at our price [15:42] list, you'll see cost of input [15:44] and output tokens, and there's up [15:45] to 90% fewer costs for cached inputs. [15:48] So prompt caching ends up [15:50] having up to 90% impact in terms [15:53] of the cost that you pay, so [15:54] we'll talk about it a little more. [15:56] The second one we talked about is, [15:58] of course, you want to reduce the amount [15:59] of input tokens you give the model. [16:01] Instead of just dumping in a lot [16:02] of information, you can use [16:04] programmatic tool calling to offload [16:07] some of that processing of data [16:09] or context to code and keep [16:11] that out of the model's context window, [16:13] which directly lowers the cost. [16:15] We talked about reasoning effort. [16:17] A lot of you are familiar [16:18] with the slider that we have on Codex. [16:19] On our API, too, you [16:21] can change the reasoning effort. [16:22] And actually now, in the API, [16:24] you can change the reasoning [16:25] effort without breaking the cache, [16:28] which was massive. [16:29] I think we released it with Astra [16:30] a few weeks ago. [16:32] And then lastly, batch or flex, which [16:34] is if a request is not urgent [16:36] and you can delay it, you get [16:38] to pay less for those types of requests. [16:41] Just given how important prompt caching [16:43] is, I think it's worth going deeper. [16:45] So, Septo, can you talk to us [16:46] about, you know, what are the common [16:48] mistakes you see developers [16:49] make in terms of optimizing [16:50] their prompt cache hit rate, [16:52] and any recommendation you might have? [16:54] Yes. So, that's a very interesting [16:56] topic, and I'm really excited [16:57] to talk to you about here. [16:59] But before we do that, I want [17:00] to ask a quick question again. [17:02] How many of you are currently [17:03] using prompt caching? [17:06] It's all of us. [17:07] It's enabled by default. [17:09] Yeah. [17:10] So, as, like, and I'm [17:11] quite excited to see, like, [17:13] a lot of raised hands here. [17:14] Thanks, Mandeep. [17:15] But as Mandeep mentioned, like, [17:17] it's, like, enabled by default. [17:19] And we talk to a lot [17:21] of our customers about prompt caching. [17:23] But we are also seeing that a lot [17:25] of our customers are not utilizing [17:27] it to the fullest extent as possible. [17:29] Now, our number one guidance [17:31] with prompt caching is to keep [17:33] the beginning of your prompt [17:34] consistent whenever possible. [17:36] In that way, prompt caching [17:37] can reuse the preprocessing [17:40] of that shared instruction. [17:41] Well, any kind of newer information [17:43] in your prompt, like, which is not [17:46] in that matching portion, can be [17:49] freshly processed by your model, [17:51] and your model produces a fresher [17:53] response each and every time. [17:55] Now, when this technique is combined [17:58] with our newer model families, [18:00] it significantly reduces cost [18:02] and latency for our customers. [18:03] Let me give a quick example [18:05] of our customer scenario here [18:06] to show you how you can improve [18:08] your cache reuse with prompt caching. [18:10] So we have one of our customers, Blitzy. [18:12] Blitzy is, like, an AI-powered [18:14] software development company. [18:15] And they reported that by moving [18:18] from single-structured output [18:21] call to Luna's tool calling loop, [18:23] they were able to improve [18:24] their cache reuse from 24% to 90%. [18:27] And across their thousands [18:29] of production calls, they also reported [18:32] 8.5x fewer output prompt tokens [18:34] and 87% cheaper cost compared [18:37] to GPD 5.4 Mini, while also [18:40] handling 2.2x more context. [18:43] Wow, okay. [18:44] So you can keep the prompts [18:46] consistent, as it were, but what [18:47] if your prompts change frequently [18:49] across different queries? [18:51] Yeah, great question. [18:52] So as we were saying, like, [18:54] for our earlier model families, [18:56] automatic caching was [18:57] enabled by default. [18:59] We didn't have, like, the system [19:01] was able to choose breakpoints [19:02] based on token intervals. [19:04] We were able to keep the beginning [19:06] of the prompt consistent, [19:08] but there weren't any option for us [19:10] to choose our own boundaries. [19:12] Now, that has completely changed with [19:14] GPD 5.6 families, [19:17] where we have the option [19:19] to add explicit breakpoints. [19:21] So if I want to put it [19:22] in simple words, think of explicit [19:24] breakpoints as, like, I want [19:25] to cache until this point, right? [19:28] With explicit breakpoints, what you [19:30] can do is, if I give you [19:32] a quick example, you have the same [19:34] long-running coding instructions, [19:35] and you have tasks that keep on [19:37] changing every time at the end. [19:39] With explicit breakpoints, you can [19:41] actually put breakpoints right after [19:42] the stable piece and before the task [19:44] that actually keeps on changing. [19:46] In that way, you are actually reusing [19:48] the stable piece of information [19:49] while the model still processes [19:51] the task that keeps on changing. [19:53] Now, if you want more control, [19:55] we have another option that is [19:57] explicit only mode. [19:58] With explicit only mode, you can pick [20:00] and choose which specific portion [20:02] or path that you want to cache. [20:05] And this is very ideal because you [20:06] do not want to cache unnecessary [20:08] content which changes frequently, [20:10] and it won't be reused later. [20:12] Now, as Artie mentioned, [20:14] like, with GPD 6 Astra, [20:16] we keep all of the controls are [20:18] all of the capabilities that you [20:19] had with GPD 5.6 families, [20:21] but with one added capability here. [20:23] With GPD 6 Astra, you now have [20:25] the power to change your reasoning [20:27] effort within an ongoing conversation [20:29] without breaking the cache. [20:31] Thank you, Sapto. [20:32] That just reminded me, the other [20:33] thing we already had that people [20:35] might not be aware of is you [20:37] can already change the tools that you [20:38] give to the model without breaking [20:40] the cache as well by adding them [20:41] in allowed tools or other parameters. [20:43] So we're adding just a lot [20:44] of flexibility in terms of changing [20:46] the reasoning effort and the tools [20:47] while maintaining the cache rate. [20:50] Again, we continue to invest [20:52] so much in this area. [20:53] Two other things to flag. [20:54] We shipped a prompt caching, like, [20:56] dashboard on our admin webpage [20:58] so that you can actually see, [20:59] measure, and track your cache hit [21:01] rate across your applications. [21:03] The other thing we released [21:04] is a diagnostics API. [21:05] So given two different queries, [21:08] we can actually tell you where [21:10] the cache miss is happening so [21:12] that you can start to optimize it. [21:13] But again, reinforcing, [21:15] again, how important that is [21:16] and really an important [21:17] lever for cost optimization. [21:20] Great. [21:21] So we've covered a lot. [21:22] We've talked about the right metric, [21:24] look at cost for completing the task, [21:26] not just the list price. [21:28] We talked about how do you find [21:30] the right model for the task. [21:31] And again, we're very proud to say [21:33] we have models at the Pareto Frontier [21:35] in terms of cost and accuracy. [21:38] We talked about the various levers, [21:40] prompt caching, programmatic [21:41] tool calling, reasoning effort, [21:43] and then batch reflux. [21:45] So with all of that now, [21:46] Mandeep and Sapto, what would you [21:47] like to leave the audience with? [21:49] Like, what should they go home and do? [21:51] Yeah. So the first starting point is [21:54] define the task that you want [21:56] the model to be able to do. [21:58] Identify the accuracy threshold [22:00] that business expects or you [22:02] expect from the use case. [22:04] Second, define a representative [22:06] set of tasks that define how [22:08] the application is going to be used. [22:11] Use, try different models, different [22:13] settings that Sapto mentioned, [22:16] different reasoning effort, [22:18] different prompts to see [22:19] prompt caching, how that works. [22:22] Plot the Pareto Frontier and find [22:24] the model configuration harness [22:26] that works best for the task [22:28] at the accuracy threshold [22:30] that the business expects. [22:32] Yeah. And from my side is test [22:34] one change, one improvement. [22:36] And when you find that improvement, keep [22:37] that setting and iterate, run your evals [22:41] as your application and model evolves. [22:43] Amazing. [22:44] Thank you all. [22:45] And you can come find us throughout [22:46] the day if you have any questions. [22:49] Thank you. [22:50] Thank you. [22:51] Thank you. [22:58] Thank you.