GitHub Cuts Claude Copilot Costs via 94% Prompt Caching Target

GitHub设94% Prompt缓存目标以削减Claude Copilot成本 缓存Token具有10倍成本优势，意味着上下文压缩往往会在财务上适得其反，迫使团队优先考虑Prompt稳定性与混合顾问-执行者模型路由。 本演讲详细介绍了GitHub在兼顾成本效益与开发者心流状态的同时，在其Copilot平台规模化应用Claude所采用的工程模式与优化策略。GitHub通过保持静态系统前缀和稳定多模态缓存亲和性，力求实现94%至96%的Prompt缓存命中率。为平衡顶级智能水平与低延迟，他们引入了诸如“顾问策略”（Advisor strategy，即由Claude Haiku执行任务并在复杂推理时调用Claude Opus）以及用于关键代码审查节点的“橡皮鸭”（Rubber Duck）评审模型等架构模式。最后，团队强调通过系统化的准入流程评估新模型，并优先考虑代码“留存率”而非单纯的采纳指标。 Prompt 缓存是 GitHub Copilot 大规模运营在财务可行性上的核心支柱；在应对海量消息规模时，缓存效率每提升 1% 便能直接节省数百万美元成本。 (0:00) GitHub 将缓存命中率目标定在 94% 至 96%（约 70% 则被视作工程调用或组装 Bug），并总结出避免在前缀中插入动态内容、避免动态加载工具以及在多模型切换中保持缓存亲和性三大准则。 (5:00) 团队破除了“长上下文必然更贵”的误区：过小的上下文窗口需频繁进行历史压缩，而生成摘要所需的输出 Token 费用是输入 Token 的 5 倍，反而会大幅推高开销并严重破坏缓存效率。 (10:00) GitHub 联合 Anthropic 打造了 Advisor 策略与“橡皮鸭”Critic 模式：日常由轻量低成本的 Claude Haiku 执行任务，仅在遇到高难度难题或关键评审节点（规划后、实现后、运行测试前）呼叫 Claude Opus 进行深度推理与指导。 (15:00)

Operating developer AI tools at massive scale requires balancing cost and intelligence through strict prompt-caching discipline, tiered model routing, and measuring real-world code retention over simple usage activity, according to GitHub's Mario Rodriguez and Anthropic's Brad Adams. Prompt Caching Discipline and Hit Targets (5:00) Operating Copilot across billions of messages makes prompt caching economically vital, with Rodriguez noting that a 1% gain in caching efficiency translates to millions of dollars in savings. GitHub targets a 94% to 96% cache hit rate; rates near 70% are treated as configuration bugs. Because cached input tokens cost 10% of the standard rate, invalidating the cache multiplies input costs tenfold across surfaces like VS Code, Copilot CLI, IntelliJ, and mobile. To maintain high hit rates, GitHub enforces strict engineering rules: system prompts must eliminate dynamic content such as UUIDs, tool configurations must avoid dynamic loading to prevent invalidating conversation history, and prompt structures must follow a strict hierarchy from system prompt to tools, conversation history, and the final message. Additionally, GitHub's multimodal harness preserves cache affinity when developers switch between different model families, such as Opus, GPT, or open-source alternatives, and return to an earlier model. Context Compaction Pitfalls and Context Costs (10:00) Rodriguez refutes the assumption that larger context windows are inherently more expensive than smaller ones. In simulated tests comparing context allocations on the same model, smaller context windows required three times more compaction through message summarization. Because summarizing generates output tokens—which cost five times more than input tokens—aggressive compaction drove overall costs higher while repeatedly invalidating prompt caches. Consequently, managing longer raw context windows can prove more cost-effective than continuous summarization cycles. Hybrid Routing with Advisors and Critics (15:00) To combine high-end reasoning with lower operational costs, Anthropic and GitHub developed an "advisor strategy." Claude Haiku acts as the primary executor handling standard tasks; when it encounters a complex barrier, it calls Claude Opus as an advisory tool. Opus returns targeted hints, allowing Haiku to complete execution with lower latency and cost than running Opus exclusively. Rodriguez notes this pattern will be incorporated into the GitHub Copilot CLI. Complementing this execution pattern, GitHub uses an experimental "Rubber Duck" critic model to inject Opus reviews at three touchpoints: after drafting a plan, after complex code generation as a pre-review before human evaluation, and before running long-duration test suites. Rodriguez reports that catching errors during initial planning yields the largest efficiency gains. Evaluating Deployments and Code Survival Rates (22:15) Onboarding new models into the Copilot API (CAPI) involves tuning prompt templates, tool interfaces, and context compaction before running offline benchmarks and internal Microsoft and GitHub dogfooding. Rodriguez cautions against tool proliferation, noting that offering hundreds of tools causes model confusion; GitHub instead scopes specific tool packages to particular scenarios. To measure operational success, GitHub prioritizes outcomes over raw activity. Rather than relying solely on code acceptance rates, teams track the "survival rate"—measuring whether accepted code remains in the codebase or is deleted shortly after. Rodriguez emphasizes that optimizing for isolated acceptance metrics risks masking poor overall user experiences. Rodriguez emphasizes that managing AI developer tools at scale requires aligning infrastructure with developer flow and verified code survival rather than superficial activity metrics, cautioning that rapid weekly shifts in model capabilities demand continuous harness tuning and empirical validation.

GitHub 的 Mario Rodriguez 阐述了在大规模场景下构建 Copilot 的工程策略，核心是通过高命中率的提示词缓存（Prompt Caching）、大小模型协同架构（“顾问”与“橡皮鸭”批评者）以及精细化的 Harness 工程，在保障财务可行性的同时兼顾低延迟与高阶智能。 提示词缓存工程与上下文成本控制 (5:00) Rodriguez 透露，在服务扩展至数十亿乃至数万亿级消息规模时，提示词缓存是维持业务财务可行性的绝对支柱。他将其比作高频交易，缓存效率仅提升 1% 就能直接节省数百万美元；在缓存机制下，输入 Token 的成本仅为标准费率的 10%（产生 10 倍的成本差距）。GitHub 将目标缓存命中率设定在 94% 至 96% 之间，一旦降至 70% 左右，即被内部判定为提示词组装或调用逻辑存在缺陷。为避免缓存频繁失效，团队总结了严格的前缀规范：前缀严禁包含动态内容（此前曾因在系统提示词中写入 UUID 而导致全量缓存失效），且提示词层级必须严格按照“系统提示词（最稳定）→ 工具定义 → 历史会话 → 最新消息”排列；同时需杜绝动态工具加载，并在多模型切换的工作流中维持缓存亲和性（Cache Affinity）。 针对“长上下文必然更加昂贵”的认知误区，Rodriguez 通过对比测试指出事实并非如此。在使用较小上下文窗口时，系统不得不执行高达三倍的消息压缩与摘要（Compaction）；由于生成摘要所需的输出 Token 单价是输入 Token 的 5 倍，过度压缩反而导致昂贵的输出计费飙升，并彻底破坏了提示词缓存，最终使总体费用远高于直接使用长上下文窗口。 双模型协同架构：顾问模式与橡皮鸭机制 (10:00) 为在 Claude Haiku 级别的成本与速度下实现接近 Claude Opus 的推理智能，Anthropic 的 Brad Adams 与 GitHub 联合设计了“顾问策略（Advisor Strategy）”。该架构模拟了资深工程师指导初级工程师的分工：由 Haiku 作为“执行者”处理常规任务，一旦遇到超出能力的复杂问题，便通过调用“Advisor”工具向 Opus 求助；Opus 完成深度推理并回传关键提示后，再交由 Haiku 继续完成落地执行。在针对逻辑难题的实机对比中，独立运行的 Haiku 容易陷入试错停滞，而引入 Opus 顾问的实例则凭借提示迅速解决，该功能将整合进 GitHub Copilot CLI。 此外，GitHub 还推出了一项名为“橡皮鸭（Rubber Duck）”的实验性批评机制，其核心是由 Opus 在三个关键节点充当“批评者”：一是在开发者起草方案后立即审查规划，以在研发最前端拦截缺陷并获取最高效率收益；二是在复杂代码编写完成后提供预审查，节省昂贵的人工 Code Review 成本与 Token 消耗；三是在编写测试用例之后、运行耗时漫长的 CI 测试套件之前介入评估，从而确保开发者始终处于连贯的心流状态中。 模型接入流程与 Harness 工具精简优化 (20:00) 当合作伙伴发布新模型时，GitHub 会通过历经两年半迭代的 Copilot API（CAPI）管线将其标准化接入。Rodriguez 提到，工程团队的核心工作并非调整底层 Agent 循环，而是聚焦在多模态 Harness 内部优化系统提示词、工具接口及上下文压缩策略。在评估阶段，除了 Terminal Benchmark 2 等离线基准测试外，团队重度依赖微软与 GitHub 内部开发者的“Dogfooding”以及线上 A/B 测试，并将数据按周汇总反馈给 Anthropic，以便在 30 天优化周期内微调模型检查点或 API 配置。 在 Harness 的工程调优中，系统围绕“构建提示词与上下文 → 调用模型 → 执行工具 → 结果回填循环”闭环运转。Rodriguez 强调，提供数百个工具反而会导致模型产生决策混淆，因此工程团队根据具体交互界面和业务场景进行专用工具包裁剪，并严格把控执行权限，避免工具冗余降低模型判断精度。 以成果为导向的度量：从采纳率到存活率 (22:15) 在衡量 Copilot 的系统优化效果时，GitHub 强调必须关注最终成果而非表层的交互活跃度。Rodriguez 认为，代码采纳率虽然常被用于衡量效果，但如果开发者采纳生成的代码后不久便将其删除，实际上并没有为研发带来真实价值。 基于这一考量，GitHub 将“代码存活率（Survival Rate）”视为更关键的指标，重点追踪被采纳的代码能否长期留存于代码库中。Rodriguez 警告产品管理者在 AI 时代切勿孤立地追求单点活跃指标，避免因局部指标虚高而最终损害整体用户体验与产品实效。 GitHub 的经验表明，大模型在超大规模工业场景下的落地不仅依赖底层基础模型的智慧，更取决于系统级的工程调度：通过严格的提示词分层锁定缓存红利、利用双模型协同优化推理成本，并以代码存活率等深层业务指标倒逼工程调优，从而在成本、性能与实用价值之间取得平衡。

[0:00] Mario Rodriguez: a little bit messy," "vibrant," and crucially, "dev first, not agent first. [0:00] Mario Rodriguez: Without that, you know, we're not dead, but oh my God. Like the amount of money that we would spend compared to what we do it incredible. [5:00] Mario Rodriguez: without data is really, really hard to make decisions [10:00] Brad Adams: One of the pieces of feedback we got from the Copilot team is they really wanted Opus intelligence but it at haiku level prices. That sounds like a good deal, right? [22:15] Mario Rodriguez: If you ended up accepting it, and then you ended up deleting it after, well, that actually did not accomplish the outcome overall.

[0:00] Thank you. One, one, one. of GitHub, Mario Rodriguez. Hello, hello everyone. It great to be back at Code with Cloud I love developer conferences You know they a little bit messy They vibrant They still dev first not agent first And this is where we're to have fun. We're here to have fun and to learn together. Now, you're here because you want to learn more about the platform. And I'm here to kind of share with you some of the top things that we end up doing to running Copilot and all of our inference on top of this platform. Now, I want to get started with the why. Like, at GitHub, we're very mission-driven. You know, our vision is to kind of empower developers to provide the best tools out there to advance human progress. But when I talk to customers overall, I'm telling them something that they want to achieve. They want to achieve a set of outcomes. And for them, they want to keep their people in flow. They want to keep developers in flow. They want their teams to gain velocity. They got teams to actually achieve more with the people that they have. And they want to do this at scale. And to be able to do it at scale, you have to have efficiency and you also have to have trust And I start with this because almost every single product decision that I make kind of is grounded on these pillars Even when we actually talk about the platform and integrating with our cloud platform I want to make sure that I keep developers in flow. I want to make sure that teams gain that velocity. And I want to make sure that the companies can do that with intelligence and trust. Now, as I think about it forward, And I was thinking really hard, okay, like what do I want to say here? And if I was sitting there, what I would like to understand is, okay, how do you operate, you know, we do billions and billions, probably over a big enough time sense, trillions of messages against the platform. So I wanted to kind of give you the best practices, or maybe not even best practices, because things change so often, almost on a weekly basis. But kind of like the key learnings, the things that are at the bottom of almost all the decisions that we make to be able to integrate with the platform and achieve that skill and that efficiency that is necessary. So I'm going to divide it into three things. Number one is prompt caching. Without that, you know, we're not dead, but oh my God. Like the amount of money that we would spend compared to what we do it incredible right So just 1 efficiency on this means a lot to us It kind of like high frequency trading Just 1% efficiency means millions overall. So I just want to spend a little bit more on how we're doing that. The second thing is we're collaborating with Anthropic on capabilities to make sure that our customers are using the right inference, the right amount of intelligence at the right time. And then one of those is this advisor model. And we have two ways of looking at it, both through an advisor and a critic. So Brad and I will share a little bit more about that too. And then the third one is, okay, every time a new model drops, and if you're integrating with the platform, you know that this happens constantly, and you get a call at 5 a.m. on a Saturday, and it's like, hey, we're launching on Tuesday. So how do we actually go through that the right way? How do we make decisions like what is the default model to keep developers in flow? And how do we make the decisions to actually route to the right intelligence at the right time overall? So I'll start with this. And this is a dashboard. This is not our dashboard. But if you are on top of the platform, I believe Anthropic released this last week, we've got a little [5:00] The preview of it and what allows you to understand is how you're doing, you know, like your cache hit ratios, how many messages you're sending against messages API, right? And I think this is great. This is the first step because without data is really, really hard to make decisions. Really, really hard. So if you haven't checked it out, please do so. Now, the one that I want to spend a little bit more time on, it's ours. And this is just one look. This is not the entirety of all of our dashboards that we have. But it's one where we take a look at deltas between models. In this case, I want you to look mainly at the left-hand side. That's Opus 4.6. And then on the right is 4.7. And then the next column is the delta between them. So when a new model drops, and sometimes we get it in EAP, we go ahead and run a set of benchmarks against it. So think about terminal benchmark 2. And then one of the suite, we have our own ones as well. And then we try to decide, OK, how is the model performing? And then we ship it, and then we pay attention to that data again. And then after a period probably of 30 days we probably done or maybe sometimes even sooner we done with all of the optimizations for that model Now you can see over here kind of the median cash tokens and at the end, the cash rate. For us to operate the service at scale, we need to run above 94, usually 94, 95, 96%. If we operate at 70%, that means usually that we have a bug, believe it or not. Like we're doing something not right, And then we need to change the approach on how we're calling the model, how we're assembling the prompt, how we're doing the end-to-end to keep that developer in flow. So we pay attention a lot to that. The same thing, if I want to actually change the cache rate from, I'm sorry, change the default model from 4.6 to 4.7, I need to understand what's the cache rate difference and what it actually means to me. Again, we make billions and billions of calls overall. So just 1% in that. in this case is 1.3%, means a lot to us. Now, you also have to take into account that from an input perspective, your cash rate is only 10% of the cost, right? So it's a 10x difference in there. So you're constantly invalidating that cash. That is not a very good thing overall because you going to be paying 10x more And then we kind of go across it In this case we did a baseline against 4 on it and then also on Haikyuu. Now, imagine it stays like this for a second, and there's a bunch of red, as you can see in this screen, overall. Then we have a decision to make. We have a decision to make to try to figure out how to get all of those reds into green. And it's not luck. It's a lot of hard work. And I want to make sure you understand that. It's a lot of hard work to get it. You know, maybe from 50% to 70% is not, because you're just kind of easy hanging through it. But from 70% to 80%, from 80% to 90% and 90% and above, there's a lot of hard work that goes into this and engineering. And there's three things that we'll take a look at. Number one is we make sure that there's no, and these are lessons learned. Like, we have made these mistakes before, and that's why I wanted to share it with you. We put no dynamic content in the prefix, right? Like, you need to keep that prefix as static as possible. And as an example, at one moment, we had UUIDs in the actual system prompt. And then we're getting constantly reset. And then that was invalidating the entirety of the cash rate. So remember system then there tools then there the conversation and then there the last message that you sending That kind of the hierarchy So you want to keep that system prompt as stable as humanly possible, no dynamic content on it. Then goes tools. We made a lot of mistakes in tools at times. If you're actually loading tools dynamically and you're changing that tools prefix, then all of a sudden everything, the entire conversation, gets invalidated again. So you have to do work, and you have to make sure you have a lot of regression tests, because you're going to be experimenting a lot with what? Skills and tools. Overall, in your end-to-end, you have to have regression tests to make sure that you're not affecting your tools. And then the third thing, then, overall, is that we need to have that cache affinity. And let me tell you, it's really hard when you're doing a multimodal hardness. So take, for example, Copilot. The customer can be calling, let's say, Opus, then calling a GPT model, and then going to an OSS model, and then coming back into Opus again. And then through all of that, I have to make sure that the next Opus call, that last one that happened, actually has the right cache affinity too. So we do a lot of work when it comes to our multi-modal hardness to make sure that that is guaranteed. [10:00] So the next slide. So here's an example on something that we ran, and also to debunk and miss. So one of the key things that I heard many times, actually when I go to customers and they ask me about integrating with the platform is, hey, does long context mean that it's more expensive? And the answer to us is no. And this is a quick test that we ended up running. So there's a smaller context window, and you could see in there that my average compaction went up three times compared to the largest context window. This was something that we simulated. We kept the same model with the same context window. We just ended up doing more compaction on it and filling it at a rate. Now, the key thing over here, if you remember the math of how many input and output tokens ends up happening, is 5x, right? So for ops, it would be $5, $25 as an example. And in that case, whenever you do compaction, you get 4,000 tokens of output because you have to summarize the message, et cetera, et cetera. So whenever that happens, then you end up usually paying more because you end up doing a lot more compaction which means that your output tokens go through the roof and then your cash rate also gets a little bit more invalidated because of that too Well significantly invalidated because of that too So longer context windows does not mean you're spending more. In fact, what you have to understand is how compaction is being done. And depending on the scenario, you want to manage that for the user appropriately. So at a high level then, instrument cash risk rate have a dashboard that actually shows it to you invest the time to do it they're already shipping one for you so at the very least use that one but go and invest in deltas go and invest in pre the model launch and then post the model launch as well now no other efficiency matters until you see that one again like you want to be driving that from 50 70 90s overall and it will take a lot of hard work a lot of engineering time and then you want to measure per surface we have VS Code, which is one of the dashboards that we show you there. But we also have the Cobalo CLI. We also have our coding agent in the cloud. We also have IntelliJ, and we have Mobile, too. So it's a lot of surfaces that you're going to have to go and understand, either share the harness across them or then tweak individually So you want to have a very good kind of sense of we shipped something we regressed or not We shipped something we improved or not Now, another thing that I was telling you about is, OK, if you've got prompt caching already done, what about making sure that the right intelligence hits the user at the right time? And for that, Anthropic and us have been partnering on this advisor model. And what I want to do is bring Brad Adams from the Anthropik team to talk more about it. Go ahead. Brad. Thank you, Mario. Oh. Thank you, Mario, for being here. It's so great to have a partner like GitHub Copilot. The Copilot team gives such great feedback. We give them very little time to evaluate models before we launch, and they give great, insightful feedback. and it's true on our API features as well. In fact, if you're using the Claude platform, a lot of what you're seeing is thanks to the things Mario and his team are doing on Copilot. And it's actually one of those pieces of feedback that I want to talk about today. One of the pieces of feedback we got from the Copilot team is they really wanted Opus intelligence intelligence but it at haiku level prices that sounds like a good deal right opus intelligence so I don get to write the deals but what I do get to do is build fun API features so I want to talk about the advisor strategy and really the insight on the advisor strategy comes from software development teams We all know if you take a junior engineer and you give them a mentor who's a senior engineer, that junior engineer gets a lot better. Because the senior engineer looks over their shoulders, they review code, they look at design docs together. The senior engineer can make the junior engineer a lot more productive without taking too much time from the senior engineer. And it turns out the same thing is true for models. You can take a junior model like Haiku, who we're listing here as the executor, and give it access to Opus, and you can very conservatively use those Opus tokens. So in this beautiful diagram that Claude created for me, what you see is the executor, Haiku, is able to identify every shape as it comes in, no problem, except for one little weird shape. [15:00] shape. That weird shape, it's beyond what Haiku can do. So Haiku has a tool it calls the advisor, it calls Opus. Opus then, because it's a bigger model, it does more reasoning and it knows that. And what we see from this in our evals is that we get close to Opus level intelligence at much lower prices because we're being very conservative about the tokens that advisor actually sends. And because of the price difference between the two, it works out really well. And what I'm excited about is an integration that we work with with the Copilot team. So let's switch over to the demo machine. Yeah, let's switch over to the demo machine. And what you're seeing is GitHub Copilot. On the left-hand side, it's just GitHub Copilot with Haiku. and on the right hand side you're seeing GitHub Copilot with Haiku plus the new advisor tool hooked up. So I going to hit enter here to give this time to run and there a little kind of brain teaser problem here that it has to solve It has to give it the exact same problem And we see Haiku has taken off whereas on the right side is Consulting Advisor So it going a little bit slowly. So I'm keeping my fingers crossed here that Haiku will, Opus will come back. And so on the left-hand side, Haiku is just spinning away, trying a bunch of things, just like you would see a junior engineer who's very eager, trying lots of things, but hasn't found it yet. On the right side, the Opus Advisor just returned. And because it's Opus, it actually knows this bit of data. And so it's able to bring that back into context. And now we're done with Opus. And everything's still in Haiku. But because it got that little bit of hint, the right-hand side has finished here. So we see the right-hand side finished. the left is still trying to figure out which way is up here. So just that little bit of hint from Opus, very small cost, very small latency hit makes Haiku so much more powerful. So this is an experiment we doing in the GitHub Copilot CLI that we release soon and looking forward to you playing with it Thank you and welcome back Mario Thank you, thank you. I love some Opus Intelligence and Haiku prices. That's kind of what we've got to do. Okay, so what we show you is an advisor model. There's another algorithm that you could actually employ as well as you're tinkering more how to get the right call to the right intelligence. And we call, you know, we're GitHub. We're a little bit weird on some of the things. We call it rubber duck. And what rubber duck is really all about, and here's a demo I'm going to show you in a second, it's really about inserting a critique at the right moments in time. So I have a feature. That feature, you know, it had an issue. And over here, I'm implementing it at the moment. And what I want to highlight, and it's going to come really soon, is the model now is asking for a critique. And you can see how it's asking for a critique from 4.6 and 4.5 opus. Then it receives that critique, and then it's able to actually change the plan prior to implementation, and then continue with that implementation the right way. And then we get that feature, and we deploy it. So that very very quick But really what I wanted to highlight is a different model It a different model that acts less as an advisor and more as a critique on hey, here's what I think you should do. And we insert this in three core places. One of them is after drafting a plan. We have a lot of our users that do a plan first, and then after drafting that plan, they go into execution. We have another one after a complex implementation. So think about it. I just finished all of this. Go and critique. It's kind of like a pre-code review to a degree. And we do it sometimes there because it ends up saving tokens then to wait all the way into an official code review as well. And then the other one is after writing test but before running them. And in certain places where your CI suite with test takes a significant amount of time, then you can see how that at the end gets you to that flow faster. and it keeps that developer in flow. So those are the three places that we're doing it right now. And from my end, what I really have seen work well is kind of at the plan phase. A lot of these systems are getting really, really good at planning. And then if you catch it there, then you end up with the most gains post as well. Now, Robert Dog is already an experimental. [20:00] So if you download the Copalo CLI and you enable that experiments, you're going to see it there. And you could invoke it whenever you want. So you could just say, hey, create a plan on this and consult with Robert Doug. And then you get a quick critique across model families as well. Now, let's continue moving. The third thing that we wanted to talk about is, okay, how do we, at scale, take new models? Like, what is the process that we do? You know, at the very beginning, it was messy for us. But over the last kind of two and a half years, we've done a lot of work to make sure we're very methodical about how we take this model. So you can see Anthropic has a model to try. We onboard that model into what we call CAPI, which is our Copilot API. And then we have an endpoint. And then from that endpoint, three things really need to happen. Number one, as you all know, you have to go ahead and work in the harness and work in your prompt. So for us, we have a multi-modal harness, right? So it's not just one model family. So we have to go over there and make sure that we update the system prompts. We partnered very, very closely with Anthropic on that. We have to then make sure the tool interfaces are right and optimized We then have to and we don do this anymore as much but we have to tweak the agent loop a little bit And then a lot more work goes into context management into compaction, into making sure that we're hitting the right cash hit rates overall. Then from there, we do two things. We run offline benchmarks. And then we also do internal duck-fruiting. Think of that as online benchmarks overall. So a lot of the Microsoft developers use it, a lot of GitHub developers use it. And we have a set of other users that we kind of partner very closely with as well to give us feedback. So we have that offline, and then we have that internal dog feeding. Then from there, we share findings with Anthropic. And this is a presentation. You know, before we used to write a document. Now we write a document and kind of expand it in detail, where we're like, this is everything that we're seeing. And then we go on a loop with them on changes that we need either at the API or even changes on some of those models as well on the checkpoints that we end up getting. And then from there, we kind of do another loop all the way until the release of the model. Now, it's very important two things. Like, yes, we do offline. And offline gives you an indication But I would say it not going to be a tense reality You learn a lot more from your online evals overall and the online experiments after launch than really the offline. Offline just kind of sets a base, and if that base is consistent, you kind of know what to expect out there. And it gives you a pretty good indication of what to expect. But all of the details do not get done through offline. So it takes us usually days or sometimes even weeks to tune everything of that model out through those online experiments. We do a lot of A-B testing, and we're very methodical about that. And we do weekly reporting that we report up to Anthropic and also across the entirety of the teams that we have to make sure that we are tuning up the right model at the right time. So when it comes to optimizing the hardness for us, there are kind of four things. One is there's the build prompt and context. There's calling the model, executing the tool, and then appending the results and pull back and look back. And what I was saying here, the places where I spend the most time with the team is usually in the execute tool and that build prompt and context. Tools are very important to us. The more tools you have then the more confusion ends up happening the more you have to tune So hundreds and hundreds of tools is not a good thing So you want to make sure you tuning by surface and you tuning for the exact scenarios with the right tools in that package So we spend a lot of time in there and optimizing that model. And then the check permissions and then run, we end up, again, in that execution of the tool, you have to do the right things to pass the right context on it and then to read the right results overall. So whenever we want to introduce a new tool, as an example, we spend a lot of time making sure that we're optimizing the hardness both at the model level and at the tool execution level. So what does this mean at a high level? There's two things. You have to go in and have both published benchmark, internal benchmark, but more importantly, dog food with A and B, and make sure that you have online evals set up. And make sure you could trust them with stat 6 as well. Then the second thing is you want to measure outcomes, not activity. When I talked about the tools, like, as an example, acceptance rate on a code line, that's OK. Survival rate is actually an even better metric overall, because if you ended up accepting it, and then you ended up deleting it after, well, that actually did not accomplish the outcome overall. So even though I could have an amazing acceptance rate, if the survival rate, [25:00] ends up being very low, that means that we did not do the right work. So it's very important if you're in product, and product really changes in the era of AI, that you're optimizing for the outcomes and not just the individual rows. Like I said, you end up then optimizing for a metric that then tanks the measurement of your entire product. So then with that, again, we walk through prompt caching, the advisor strategy, and then we talk a little bit about measurement and how we improve the hardness. And then for us, I just want to leave you and say thank you. Really excited again to be with you. Hopefully you learned a couple of things today. Outro Music