Replit Battles Agent Failure Patterns With New ByBench Platform

Replit推出全新ByBench平台应对Agent失效模式 传统软件评估难以适用于自然语言氛围编码（vibe coding），因为AI模型很难在未经核实验证的Agent生成代码基础上构建功能。 视频探讨了评估“氛围编码”（vibe-coding）智能体所面临的独特挑战：用户无需编写测试或底层代码，完全通过自然语言生成整个应用。为解决这一问题，Replit推出了开源端到端评估套件ByBench，通过基于浏览器的自动化评估智能体来模拟真实用户交互。此外，演讲还详细介绍了Replit的“Telescope”闭环系统，该系统能自动聚类生产环境中的故障追踪，通过编码智能体生成修复方案，并将其导入严格的测试流水线。最后，演讲者强调，尽管自动化程度很高，但在应对相互冲突的A/B测试指标以及为非技术用户打造工具时，人类的审美与产品理念依然至关重要。 Replit 阐明传统代码基准测试（如 SWE-bench）已不适用于“Vibe coding”，因为非技术用户通常从零构建应用且不编写测试用例；为此其实施了“离线基准拦截回归 + 在线生产 Trace 聚类与 A/B 测试”的双支柱评测体系。 (3:39) Replit 开源了端到端 Vibe coding 基准测试 ByBench，直接依据自然语言 PRD 从零构建完整 Web 应用，并由自动化 Evaluator Agent 通过浏览器模拟真实交互来评估功能正确性。 (5:55) ByBench 初步测试揭示，前沿闭源模型与开源权重模型在应用生成上存在接近 2 倍的性能差距；同时模型在未经检验的代码之上二次迭代时面临严重的性能惩罚（即“slop on slop”衰退）。 (8:55) Replit 借助内部系统 Telescope 实现了研发闭环：利用语义聚类从每日数百万条 Trace 中自动捕获隐蔽异常，驱动 Coding Agent 自动开 PR 修复，并通过基准测试门禁与 A/B 测试快速上线。 (16:29)

Mikael Carbin, President and Head of AI at Replit, argues that evaluating and improving software agents at scale requires moving beyond static code-patch benchmarks toward continuous, production-integrated evaluation loops measuring end-to-end functional correctness. Limits of Traditional Developer Benchmarks (0:21) Traditional evaluation suites like SWE-bench and HumanEval rely on applying generated code patches to existing repositories with established unit test suites. Carbin asserts this paradigm is poorly suited for 'vibe coding,' where users provide only high-level natural language prompts and expect a functional application without specifying frameworks, architecture, or test suites. Because users start from greenfield, empty repositories, passing a unit test or compiling a patch is insufficient to determine whether an application actually works. Furthermore, because underlying foundation models, system prompts, and tool interfaces change constantly, static evaluations scored by humans cannot keep up with platforms shipping multiple daily agent updates to millions of users. Effective agent evaluation must instead track real-world functional correctness—verifying whether an application fulfills the user's intent—while leveraging continuous execution data from production environments. Agnostic Functional Testing via ByBench (5:55) To address the absence of relevant greenfield benchmarks, Replit introduced ByBench, an open-source evaluation suite built around 20 real-world user traces initiated from Product Requirements Documents (PRDs). Rather than relying on synthetic codebases or slow human reviews, ByBench evaluates generated applications using an automated evaluator agent. This agent inspects the running web application in a browser and executes natural language testing workflows step by step, remaining completely agnostic to the programming language or framework chosen by the model. Early benchmark findings revealed an approximate 2x performance gap between frontier closed-source models and open-weights alternatives. Additionally, ByBench exposed a steep performance drop across models when tasked with extending their own generated codebases—a scenario Carbin termed 'slop on slop.' Without intermediate testing phases, agents struggle to build on top of unverified code, compounding architectural flaws. Semantic Trace Clustering and Telescope (16:29) While ByBench provides offline gating, Replit captures primary engineering signals by analyzing millions of unscripted daily user traces. Because non-deterministic agent outputs generate distinct logs for identical underlying bugs, conventional regex parsing proves ineffective. Replit instead embeds failure summaries and clusters them semantically with language models—retraining cluster definitions nightly—to catch recurring failure modes, such as agents prematurely trying to fix execution environments before system setup finishes. This operational pipeline feeds into 'Telescope,' an internal system where coding agents automatically generate pull requests to fix identified failure clusters using trace and log data. Proposed fixes undergo automated litmus evaluations, where a score drop of 10 points or more triggers immediate rejection. Non-regressive or controversial changes are routed into live A/B tests to measure real-world user impact before deployment. Calibrating Human Taste for Non-Technical Users (19:37) Although Replit automates roughly 90% of its issue discovery and patching workflow with agents in the loop, Carbin emphasizes that human judgment remains crucial for prioritizing failure clusters and resolving conflicting telemetry. In production A/B testing, dashboards frequently show trade-offs rather than obvious wins—such as an agent candidate being 7% slower but 8% cheaper alongside minor sentiment shifts—requiring human product philosophy to dictate which trade-offs to accept. Engineers must also calibrate their judgment to match their user base. While Replit's engineers are deeply technical, their platform is primarily used by non-technical knowledge workers who have never coded. As a result, product intuition and design decisions must frequently counter an engineer's personal preferences to serve users who prioritize working software over conventional engineering norms. Carbin concludes that evaluation must function as an active development engine rather than a static pre-deployment check, though maintaining this pipeline requires navigating ambiguous production trade-offs and bridging the substantial capability divide between proprietary frontier models and open-weights alternatives.

Replit总裁兼AI主管Mikael Carbin与团队成员Michele指出，传统基准测试无法衡量自然语言直接生成应用的“氛围编码”（vibe coding）；为此Replit构建了结合端到端离线评估与海量在线追踪分析的双轨系统，以推动AI代理的每日高频迭代。 氛围编码带来的评估困境与双轨架构 (0:21) Replit作为面向知识工作者的开发平台，其实践的“氛围编码”要求系统仅凭自然语言提示词从零构建完整应用，用户不提供框架、测试用例或代码约束。Carbin认为，诸如SWE-bench或HumanEval等依赖既有代码库补丁和预设单元测试的传统基准，无法评估这种从空代码库起步的系统，因为测试需要衡量的是最终功能正确性，而非代码编译或通过断言。此外，底层基础模型的加速演进、提示词调整及工具定义变动，导致单次人工打分的静态黑盒测试无法适应日常多次发布的需求。 为应对这些挑战，Replit建立了双轨评估体系：第一轨是离线基准测试，作为版本发布的门禁标记，若检测到严重性能倒退则立即中止上线；第二轨是在线评估，利用平台每日从数百万用户运行会话中提取的执行轨迹（traces），结合大规模A/B测试与聚合聚类，为工程团队提供快速、可执行的实时反馈闭环。 端到端基准ByBench与自动化评测机制 (5:55) 为了填补绿地代码生成的评测空白，Replit推出了开源基准ByBench。该基准输入长篇纯英文产品需求文档（PRD），包含从实际使用中整理出的20组真实应用场景，涵盖单次从零构建、基于参考实现扩展、以及在未经校验的代理生成代码上继续扩展（被Carbin戏称为“在残渣上堆残渣”）等多种复杂模式。测试数据显示，前沿闭源模型与开源权重模型在ByBench上存在接近两倍的性能差距，且多数模型在自身生成的代码库上二次开发时表现大幅下滑，表明中间测试环节不可或缺。 由于用户可以选择任意编程语言与框架，ByBench设计了与实现完全解耦的自动化评测智能体。该评估器会在浏览器中直接加载生成的Web应用，按自然语言指令（例如进入后台、特定账号登录、点击开关等）逐步操作界面，一旦步骤受阻即收集失败上下文并打出功能正确性得分，避免了人工评估的高昂成本与缓慢周期。 线上追踪聚类与Telescope自动化闭环 (11:50) 虽然离线测试提供了发布门槛，但Replit的生产环境每天产生数百万条无预设会话。平台监控多维线上指标，包括代理是否陷入死循环、提示词反映出的用户挫败情绪，以及用户是否将应用公开发布或分享等强正向信号。由于A/B测试常出现相互矛盾的信号（如候选版本运行时间增加7%但成本降低8%），团队不依赖僵化的正则日志解析，而是每晚将失败摘要嵌入向量并进行语义聚类，由大语言模型归纳出高频故障模式。 内部研发的“Telescope”技术则将这一流程进一步自动化：一旦语义聚类发现特定问题，编码智能体便结合日志与监控指标自动生成补丁并提交PR。随后补丁须通过PyBench评估，若基准分下降10分以上即判定为破坏性变更而直接驳回；若影响存在争议则转入真实用户A/B测试，明确有效的改动则直接合入上线。据Michele介绍，目前该工程流程中已有约90%的环节由代理辅助完成。 竞态故障案例与非技术用户导向的工程品味 (18:20) Michele分享了一个环境竞态条件的生产实例：在长尾延迟场景下，执行环境的启动耗时略微超出预期，而现代代理因急于解决问题，误将尚未准备好的环境当作故障自行介入排查。由于每次调试轨迹各不相同，传统日志分析难以捕捉，但Telescope通过语义轨迹聚类迅速锁定了该模式，使团队得以在免于A/B测试的情况下快速修复回归问题。 尽管自动化程度极高，Michele与Carbin一致强调人类工程师在“提出假设、排列优先级与确立产品哲学”上的核心地位。Michele特别指出，Replit团队由全技术人员组成，但服务的用户大多从未写过一行代码。因此，工程品味必须针对非技术用户进行校准，在决策优化目标（如追求极低成本还是更长运行）时，往往需要做出与工程师自身使用习惯完全相反的产品选择。 Replit将评估工具开源并提倡行业协作，主张停止在私有评估框架上竞争，而是将端到端评测和线上追踪聚类转化为驱动模型与系统每日迭代的常规引擎。如何持续消除复杂代理在多层堆叠代码上的可靠性衰减，仍依赖于前沿模型能力与深入非技术用户语境的产品品味。

[0:21] Mikael Carbin: Unlike traditional software development tools, vibe coding represents an extreme paradigm: users start with nothing more than a natural language specification of what they want. [0:21] Mikael Carbin: Because Replit's production environment generates millions of execution traces daily, there is an immense amount of "hot fat" (valuable signal) to be extracted directly from real-world usage. [3:39] Mikael Carbin: In vibe coding, users do not write tests and often start from a completely empty codebase. There is no existing repository to patch; everything is built from the ground up. [5:55] Mikael Carbin: slop on slop [8:55] Mikael Carbin: This greenfield evaluation surface represents a major shift from the fixed, predictable environments of SWE-bench. [8:55] Mikael Carbin: most models perform significantly worse when extending their own code [11:50] Mikael Carbin: This is the harsh truth about running an A/B test [16:29] Michele Catasta: 90% of it... now aided by an agent in the loop [18:20] Michele Catasta: really eager to fix problems [19:37] Michele Catasta: an engine that allows you to ship a better agent every single day. [21:47] Michele Catasta: I don't believe in competing on evaluations. [25:12] Michele Catasta: it was more like a survival game just to make sure it was barely functional [25:12] Michele Catasta: knowledge workers who have never written a single line of code.

[0:21] All right. Hi everyone. I'm Mikael Cata [0:24] president and head of AI at Replit. And [0:27] today I'm going to be talking about our [0:29] both evaluating and improving on a daily [0:31] basis Replit agent at scale. [0:34] And as you know, Replit is a back-coding [0:36] platform for knowledge workers. We're [0:38] one of the top players in the space and [0:40] we've been literally battling with this [0:42] problem for the last you know almost 2 [0:44] years. [0:45] The key difference between I would say [0:47] back-coding has a very broad definition. [0:50] It ranges all the way from being used by [0:52] software developers. But in our specific [0:55] case is even more of an extreme [0:56] definition where you start from just a [0:59] natural language specification of what [1:01] the user wants and literally nothing [1:03] else. So the user expects to go from a [1:06] prompt to a working application, but [1:08] they don't tell us what kind of [1:10] framework they want to use. They don't [1:12] write tests. They just expect things to [1:15] work after you know our the agent run. [1:18] So it means that a lot of the [1:20] evaluations and a lot of the ways in [1:22] which we've been building agents in the [1:24] past especially for software developers [1:26] has to fundamentally change. [1:29] Now, [1:30] how do we track internally the fact that [1:33] the agent gets better on a daily basis [1:34] and is actually aligned with all with [1:36] what our users want? [1:38] This is especially challenging because [1:40] if you think about it, models are [1:42] changing almost constantly and if [1:44] anything the pace of evolution is even [1:46] faster than it was a year ago. So we are [1:48] always sit on shifting grounds. Models [1:51] change fast. We have to change our [1:54] prompts, system prompts, user prompts [1:56] very often, you know, to comply to the [1:58] new features that we're building in [1:59] product. Of course, the tools [2:01] definitions are changing as well. We're [2:03] adding more of them, we're simplifying, [2:05] and we're shipping constantly. Every [2:08] single day we have multiple releases of [2:10] wrap agent, which we put in front of, [2:11] you know, millions of users. [2:15] So, my argument today for this talk will [2:17] be that we have to fundamentally rethink [2:19] how we do evaluations. [2:21] On the left side, you can see how evals [2:25] look like for basically, you know, since [2:26] since we've been working on AI. Um they [2:30] you run your evaluation, you they [2:33] produce a single score, and that score [2:35] is produced by a human. And what the [2:37] human decides based on the score is, my [2:39] agent harnesses better than it was [2:40] before or it regress, or my model is [2:43] better than before, or we should be [2:44] doing, you know, more post training. [2:46] That's how how we've been doing this, [2:47] you know, for a for a long time. [2:49] I argue that we should move more and [2:51] more towards a continuous setting of [2:53] evaluations, where especially when you [2:56] actually have something runs in [2:57] production, and in our case it generates [2:59] millions of traces every day, there is a [3:01] lot of hot fat to be extracted from the [3:03] traces themselves. [3:05] So, what we do is we have a combination [3:07] of fine evals, so very similar to sweet [3:10] bench and other benchmarks that are very [3:11] famous in the space, as well as online [3:14] evaluations based on the on the system [3:16] usage. [3:17] What do we want our evals to capture? [3:19] Fundamentally, first of all, they need [3:21] to optimize what our users care about, [3:24] you know, rather than just telling us if [3:26] the agent is editing code correctly. [3:29] They should also highlight immediately [3:30] what is breaking, and they should inform [3:33] us where what we should be shipping next [3:36] in terms of improvement for our agent. [3:39] What goes into this [3:41] box, this black box? So, we're building [3:43] a new system that allows us to do these [3:45] constant improvements. [3:47] The system looks in the following way. [3:49] We have two pillars. On one end, we have [3:53] the standard old-school benchmarks. They [3:55] are usually employed before shipping a [3:59] new version of the agent. Think of them [4:00] sort of as a boolean flag. It's a [4:02] gatekeeper before deciding if you should [4:04] be launching a new version or not. So, [4:06] if there is a major regression on an [4:08] offline benchmark, of course you stop [4:10] your release. If you see no changes or [4:13] positive changes, you know that you can [4:14] go on. [4:15] But the part that is more intriguing and [4:17] is what is helping us to make constant [4:19] progress is the fact that we do a lot of [4:22] baby testing and we also cluster all the [4:25] traces that we obtain in production and [4:27] we try to gain insights from them. So, [4:29] in this case, [4:31] the online pillar actually works after [4:33] you ship a version of the agent and it [4:35] forces you to react as fast as possible. [4:39] And the loop here is, once I have these [4:41] two pillars in place, you reflect on the [4:43] results that both of them are giving [4:44] you, you likely change the code once [4:46] again, run your eval, rinse and repeat. [4:49] And this is the cycle that we go we are [4:51] going through every single day at [4:52] Replit. [4:55] I mentioned before SweetBench and I want [4:58] to give you a quick understanding why [5:00] we work on something, you know, more [5:02] powerful than SweetBench or at least [5:04] like more tailored towards our use case. [5:07] Um something as SweetBench, HumanEval, [5:09] they played a key role in our space, you [5:12] know, to allow us to air climb them and [5:14] make models more powerful. But [5:16] eventually, they all follow the same [5:17] protocol. [5:18] Uh you make sure that the code that is [5:20] generated complies with the user prompt, [5:22] then you apply the patch on your [5:24] repository, you run the test. If the [5:26] tests are passing, you know, you you [5:28] have a higher score on the benchmark. [5:30] This does not reflect what happens in by [5:32] coding. As I was mentioning before, [5:34] users are not writing the tests. They [5:36] often start from a completely empty code [5:38] base. So, there is not a scenario where [5:39] you can just apply patches. You're [5:41] building things from the ground up. [5:43] So, what we need to capture in instead [5:46] is does the app do what the user asked? [5:48] So, there is a functional correctness [5:51] gap between what SweetBench provides and [5:53] what we were working on. [5:55] And today on stage, I'm launching [5:59] ByBench. It's a new public benchmark for [6:02] back-coding end-to-end that we worked on [6:05] at Raptic for several months. I'm going [6:07] to give you all the details after the [6:08] URL and so forth, but let's start from [6:10] the setup of the benchmark. The input is [6:14] just a PRD. It's literally like a long [6:16] prompt that describes how to build an [6:18] application. [6:19] Rather than crafting them from synthetic [6:22] data, we pick 20 real-world traces from, [6:26] you know, usage of Raptic, just plain [6:28] English specifications [6:30] without any implementation constraints. [6:32] Then, we have an harness that builds the [6:35] application end-to-end. So, it goes from [6:38] an empty repo into something functional. [6:40] And then, the key insight is rather than [6:43] stopping the benchmark here and then [6:45] having humans evaluating the output of [6:47] those PRDs, we built automated [6:49] evaluators. This is what allows you to [6:52] go from a benchmark that you run maybe [6:54] on a weekly basis into something that [6:56] you can run every single time you [6:57] literally have a new PR merge in your [6:59] repository. So, of course, as you can [7:01] imagine, we have AI running all the [7:03] evaluations on our behalf. [7:06] The [7:07] the setup is the following. There is one [7:10] core, and the protocol is you have an [7:13] you have a input, you have an [7:15] implementation strategy, and then you [7:17] have a set of our core, sorry, our wired [7:19] evaluations that you run on your app. [7:21] And we came up, you know, with five [7:23] different pairings, but in reality, [7:26] there are a lot more pairings that you [7:27] could come up with yourself. The most [7:29] basic pairing is the input is the PRD, [7:32] and we build the application single shot [7:35] zero to one. [7:36] Then, maybe in order, you know, to go [7:39] from uh less complicated to more [7:41] advanced, we have something called by [7:43] bond ref or like the reference [7:44] implementation where we start from [7:46] something is already working and we [7:48] build a feature on top of that. [7:50] Let's make it more advanced. We have by [7:52] bond vibe where we start from an agent [7:55] on MVP and then we build a new feature [7:57] on top of that. Maybe this is something [8:00] that in the X lingo you would call slop [8:02] on slop. So if we have something that's [8:04] been built by the agent not verified and [8:06] then you had more agent code written on [8:08] top of that and then you run the [8:10] evaluations to see if it's working or [8:11] not. You can keep making this [8:14] arbitrarily more complex. Uh for example [8:16] on in agent four that we launched a [8:18] couple of months ago, we completely hide [8:20] the complexity of doing task [8:22] decomposition and running agents in [8:24] parallel and then merging all the [8:26] patches together. So [8:28] we can also take as an input the PRD, we [8:30] do the task decomposition, we run all of [8:33] that in parallel and then we run the [8:34] valves after that. As you can imagine [8:36] the parallel plus merge scenario is far [8:39] more uh you know, challenging for a [8:41] standard coding agent. [8:43] I give you here some more ideas we [8:44] describing in the paper, you know, but [8:46] you can even start from a buggy [8:47] application, build a feature on top of [8:49] that and see how much your agent [8:50] struggles, you know, to actually make it [8:52] make it work correctly. [8:55] So the the key question or the [8:57] complexity here is how do you actually [8:59] do the grading? And that's what I told [9:01] you before, you know, was where we spent [9:02] most of our effort. It turns out that [9:05] when we're working on our previous [9:06] version of the agent, we put quite a lot [9:08] of effort in building an automated, you [9:10] know, app testing feature which [9:12] literally doesn't know anything about [9:14] the app per se. Cuz if you think about [9:16] it, when it comes to web coding, you we [9:19] don't give any level of [9:21] uh you know, guardrails to our users. [9:23] They can decide the language they want [9:25] to use. We can decide a different [9:26] framework. So the evaluator has to be [9:29] totally agnostic on how the [9:30] implementation looks like. [9:32] And what our evaluator [9:34] evaluator agent does is it reads the [9:36] code base, it then opens a browser and [9:39] points it to the application that our [9:41] agent has built and then step-by-step [9:44] goes through our testing plan. And even [9:46] the testing plan is expressed in natural [9:48] language and the actions are like open [9:51] the admin dashboard and log in with a [9:52] certain account and click on this [9:54] toggle. And if any of those steps fails, [9:57] then you know, we collect all of them [9:58] together and we generate a score. So, we [10:01] go through all these test plan and then [10:03] we decide ultimately if the score is [10:05] good or not. So, this is the key [10:08] complexity. When it comes to SweetBench, [10:11] we always had a fixed surface. We know [10:13] the repositories, we know the task [10:14] hardness, we know exactly how to make [10:16] this work. In ByBench, the surface is [10:18] completely green field and that's why it [10:21] took us several months of work to get [10:23] there. [10:24] Now, as I mentioned, we're announcing it [10:27] today. Here's the QR code. You can find [10:30] the entire benchmark available open [10:31] source on bybench.ai. [10:34] I'll let you to dig into the paper. You [10:36] know, we [10:37] it's going to be presented a couple of [10:38] weeks from today at the conference on AI [10:40] and energetic systems. [10:42] Peter, the you know, lead of the project [10:44] and first author of the paper is sitting [10:46] here. So, after my talk, please come and [10:48] have a chat with him if you want more [10:49] details. [10:50] I'm just going to give us sneak peek of, [10:52] you know, the most notable results [10:53] there. [10:54] First of all, we are noticing almost a [10:57] 2x gap between the frontier models and [10:59] the open weights models. [11:01] And [11:02] the reason why I want you to pay [11:03] attention there is how I think it [11:05] because we know that [11:07] every single model player as every [11:09] single model builder is hit climbing [11:11] specific benchmarks. That's why we [11:13] release this open source. I want the [11:15] entire community to embrace it across [11:17] open weights and closed weights and make [11:19] sure that you know, we make progress [11:21] overall on by coding as well. [11:23] The other one, possibly not very [11:25] surprising is most models do a worse job [11:28] when extending their own code. So, the [11:30] slop on slop or by bomb vibe scenario [11:33] that I was describing to you before is [11:35] by far the most challenging. And that's [11:37] the reason why I always advocate to have [11:39] a testing step in between every time you [11:41] create a new feature, otherwise you keep [11:43] building on shaking foundations and [11:45] eventually your vibe coding application [11:47] is going to fail. [11:50] I'm going to give you back the QR code [11:51] later on. But, you know, let let's move [11:53] to the to the next step. [11:55] We talk about offline eval. I talked to [11:58] you about vibe bench, but as you recall [12:00] there were like two main pillars that we [12:01] care about. And the reason why we also [12:04] care about the online one is because [12:07] there is a great difference in terms of [12:09] volume of signal that we can collect. [12:12] So, on the offline or right now we're [12:14] working roughly 20 applications. The [12:16] reason, by the way, why we made it open [12:18] is because we are more than welcoming [12:21] contributions from the community. So, [12:23] more applications, harder prompts, we [12:25] can keep making this benchmark arbitrary [12:28] harder so that we make our traffic's [12:30] life harder to actually come with over [12:31] time. [12:32] But, [12:33] when it comes to Rabbit Engine instead, [12:35] you know, we're we're collecting [12:37] millions of sessions every day and [12:38] they're very valuable because they [12:40] capture what our users actually do on [12:42] the platform. And they're completely [12:44] unscripted, the agent is always running. [12:46] So, how do we distill useful information [12:49] out of them? [12:50] Well, it turns out that we run a lot of [12:54] AB tests. It's basically our way to keep [12:56] ourselves honest because vibe bench only [12:58] tells us part of the story. [13:01] And I invite every agent builder out [13:03] there to start to invest on their AB [13:04] testing infrastructure as soon as [13:06] possible. It's the best way in which you [13:08] can make steady progress. [13:10] Now, we ask a wide variety of questions [13:13] for ourselves. So, we we built a lot of [13:15] different instrumentation and metric in [13:17] our agent so that for instance, we can [13:19] find out is there a normal span ongoing [13:22] or is the agent running for longer than [13:23] expected? Or we constantly keep track of [13:27] the user sentiment. It's it's a very [13:29] easy thing of to collect because every [13:30] time you receive a prompt, you can do [13:32] sentiment analysis on it and then [13:34] receive the user is getting frustrated [13:36] or not. And last but not least, this is [13:39] very idiosyncratic to Rapid. Our users [13:41] can also publish their application on [13:43] our product. So, it's a very strong [13:46] positive signal if they decide that [13:47] whatever they built is worth to share [13:50] with their colleagues or to put it in [13:51] public in front of everyone. So, all [13:53] these signals clustered together gives [13:56] us something like this. If you have ever [13:58] done an AB test in your life, I'm sure [14:00] this looks familiar. [14:02] And [14:03] it might look familiar especially [14:04] because not everything is either green [14:07] or red. This is the, you know, hard [14:09] harsh truth about running an AB test. [14:12] They never give you a crystal clear [14:14] signal of what you should be doing. So, [14:16] in this case for example, uh the average [14:19] run duration of our agent, you know, [14:20] went up by 7% but conversely is 8% [14:24] cheaper and we saw the fluctuation in [14:26] terms of positive and negative [14:28] sentiment. [14:29] What shall we do in this case? This is [14:31] where human taste and product philosophy [14:34] still plays a key role. You will never [14:37] get or very rarely do we get like a [14:39] crystal clear configuration of the [14:41] results of your AB test. [14:43] And [14:45] to make our life a bit easier on how to [14:47] generate these AB test candidates, what [14:49] we do is we cluster all the traces that [14:52] we receive every single day and night as [14:54] you can imagine and we try to identify [14:57] first of all, which cluster are [15:00] capturing normal nominal behavior of the [15:03] agent. And as you can imagine, there are [15:04] plenty of them because the vast majority [15:06] of times the application is actually [15:08] successful but of course there is a long [15:10] tail of problem that it happens to [15:12] surface. [15:13] In case we find a problematic cluster, [15:15] then what do we do? We embed all the [15:17] failure summaries. [15:19] We [15:21] we cluster them by type so that we find [15:23] out the different failures that are [15:24] happening at a certain point in time. [15:26] Then we pass them through an LLM to [15:28] classify exactly what is going on. And [15:31] the fact that we do this [15:33] not based on regex on logs or like very [15:35] deterministic techniques is what [15:37] actually makes a difference because you [15:39] will be able to cluster things that are [15:41] semantically close to each other even [15:43] though the agent doesn't give you always [15:45] the same type of output. [15:47] And you know, once once we find these [15:50] uh cluster configurations, we are forced [15:52] to retrain them every single night [15:54] because we are running several versions [15:56] of the engine in parallel. We ship many [15:58] versions every single day, so we can't [16:00] live with a fixed cluster configuration. [16:05] And aside, you know, from also helping [16:07] us to identify the failure modes, [16:09] another interesting feature is the fact [16:11] that once we retrain those clusters, we [16:14] can go back and see if after we believe [16:17] we fixed a problem, a certain cluster is [16:19] actually disappeared. So, say you have a [16:21] certain tool failure that is happening [16:23] only a limited amount of time. So, not [16:26] something that will show up in your [16:27] DataDog dashboard, for example, not a [16:29] failure that happens 50% of the times, [16:32] but maybe it was happening 1% of the [16:34] times in certain specific conditions. [16:37] And from here, you will start to [16:38] realize, okay, I shipped my new PR and [16:41] that cluster is disappeared. So, you [16:43] start to have some evidence that you [16:44] mitigated the problem. [16:48] The [16:50] The technology that we built internally [16:53] was called Telescope and it's a fairly [16:56] simple loop, I hope, after I explain [16:57] everything. Uh first of all, we start by [17:00] discovering the problems. And this is [17:02] based on the trace clustering that I was [17:05] talking about before. [17:06] Then, based on that, we create code [17:09] changes. So, it's completely automated. [17:11] We basically cut PRs with a coding agent [17:14] based on the information we got from the [17:16] trace, based on information we collected [17:18] from the logs, from all the different [17:19] different dashboards that we had. [17:21] Then we evaluate if the change we [17:24] generated, first of all, is is is a [17:26] breaking change. So, we rerun by bench. [17:29] By bench is kind of like a litmus test. [17:31] If the score drops by 10 points, we [17:33] decide that the change is bad. [17:35] If it's a controversial change that we [17:38] believe could affect also negatively our [17:40] agent, we run an AB test. So, we put it [17:43] in front of the users and we try to find [17:44] out what are the trade-offs. If it's a [17:47] clear sure shot, then we just ship it in [17:49] production. [17:51] And [17:52] few times when we believe that the [17:54] hypothesis was correct and maybe the PR [17:57] was not perfect, we keep iterating on [17:59] the problem. Maybe maybe we run another [18:01] AB test until we are in a good [18:03] configuration and then we actually ship [18:05] it. [18:06] If you think about it, this is very [18:07] similar to the work you do as an AI [18:09] engineer all the time, but 90% of it is [18:12] now aided by an agent in the loop, you [18:15] know, rather than you having to sweat [18:16] about every single step of the process. [18:20] Let me give you all an example of, you [18:21] know, what we actually experienced, you [18:23] know, once in production. [18:25] Um [18:26] Rabbit Agent immediately starts an [18:28] execution environment the first time you [18:30] submit a prompt. And of course, to set [18:32] up everything, especially with the level [18:34] of complexity we built, requires, you [18:35] know, quite a few seconds. [18:37] We had a degradation in the long tail [18:39] where our, you know, setup time was [18:41] longer than expected and our agent was [18:44] ready to roll before the environment was [18:47] fully set up. [18:48] As you know, agents have become really [18:49] eager to fix problems. So, Rabbit Agent [18:52] went on a tangent and started to try to [18:54] fix the environment, you know, on the [18:56] spot. Um [18:57] it turns out that by the nature of [18:59] agents not being deterministic, every [19:02] single debugging session looked a bit [19:04] different. So, if it tried to find out [19:06] this long tail problem just by grabbing [19:08] the logs, it would have almost not [19:10] appeared. [19:11] But the moment we cluster all the [19:13] traces, you know, with basically with we [19:15] we through the semantic layer, then we [19:17] started to figure out that this problem [19:19] was happening quite often, and we [19:21] immediately able to create a patch and [19:24] fix the issue. In this case, we didn't [19:26] have to run an AB test because it was a [19:28] pure regression case. But in case it was [19:31] a problem that required testing, of [19:32] course, we would have put an additional [19:34] step in between. [19:37] And even though I'm a big fan of [19:40] trying to optimize as much as possible [19:42] the life of an AI engineer with agents, [19:45] I do think that there is a lot of [19:48] intellectual work to be done by humans [19:50] still in the kind of loop that I [19:51] presented to you. [19:53] For example, whenever we find one of [19:55] these clusters that I was talking about [19:57] before, as a team, we always focus on [20:00] formulating an hypothesis on what could [20:02] be possibly go wrong there. Cuz the [20:04] truth is when running in production, [20:06] you're going to get a really a large [20:08] amount of candidates of what you could [20:10] be fixing. So, you need to prioritize [20:12] where you put effort, where you put [20:13] resources. So, if the hypothesis is [20:16] intriguing enough, then we move forward, [20:18] and we don't just let the agent write a [20:21] PR without any kind of supervision. We [20:23] actually try to feed some of our [20:24] understanding of what is wrong. So, we [20:26] give it more guidance. [20:28] And ultimately, if you think about it, [20:31] every single time we decide of working [20:34] on a PR and doing an AB test, we're sort [20:36] of like shaping the hill that we try to [20:38] optimize. Cuz if we only care about, [20:40] say, making the product more affordable, [20:43] we're going to go and try to fix all the [20:45] anomalous spend problems, and then we're [20:46] going to do our best to optimize that as [20:49] much as possible. So, [20:51] all these choices really determine the [20:53] product philosophy of what you want to [20:55] put in front of our users. And then, [20:57] last but not least, as I was showing you [20:59] before on the AB test dashboard, if [21:02] there is not a clear result, ultimately, [21:05] if you decide to launch or not, is still [21:07] a choice that a human does. Often times, [21:10] that's that's on me in the rabbit. And [21:12] again, that determines really what's [21:14] going to be the ultimate product [21:15] experience. [21:18] So, my In closing, what I want you to [21:20] have as a takeaway today is don't think [21:22] of evaluation just as this last check [21:25] before shipping. It shouldn't be just a [21:27] boolean flag, but rather think of this [21:29] as an engine that allows you to ship a [21:31] better agent every single day. [21:34] Thanks, everyone. And [21:36] I'm going to invite Hannah on stage so [21:37] we can have a chat about how we've been [21:38] working together in the last few months [21:40] to go. Thank you. [21:42] >> [applause] [21:47] >> Awesome. So, my name is Hannah. I'm part [21:48] of the Applied AI team at Anthropic, and [21:51] I work with Michele [21:52] on the Replit agent and on everything [21:54] that they build with Claude. So, I want [21:56] to ask you a few questions about what [21:58] you built here. My first question is [22:00] Vibez is something that we've seen a [22:02] couple times. You've used it with us to [22:04] give us feedback on research models that [22:06] we're testing. But, what was was your [22:08] vision from the beginning to make [22:10] something that you were going to open [22:11] source and make available to the [22:13] community? I think a lot of people may [22:15] be trying to build evals like this for [22:16] themselves where they see holes in the [22:18] public benchmarks, but I think this is [22:20] kind of a unique approach, and I'm just [22:22] wondering if you could talk about that a [22:23] little bit. [22:24] Yeah, I've been asked this often, why [22:26] don't you keep this as a private appeal [22:28] of your company? Fundamentally, I [22:30] believe that [22:31] we should try to give back as much as [22:33] possible to the community. And we all [22:35] have to benefit from creating public [22:37] evals. It helps you to make better [22:39] models. It helps us to make a better [22:40] agent. It helps everyone to create [22:42] better products. And I I don't believe [22:45] in competing on evaluations. I come from [22:47] a research background where everything [22:49] should be open. [22:50] So, we'll we'll keep doing this for as [22:52] long as we can in Replit. And And again, [22:54] I I really invite people to collaborate [22:57] on Vibez. I I tried to instill this idea [22:59] in the community several times when I [23:01] was giving talks in the past and at a [23:03] certain point I realized we have to [23:04] build it ourselves rather than just [23:06] trying to invite others to do it. Mhm. [23:08] Well, I'm really excited to see what [23:10] people do with that. Um [23:12] I also want to ask you about the second [23:13] pillar, telescope. This is um you know, [23:17] a pretty sophisticated system that [23:19] you've built. It sounds incredibly [23:20] useful. It sounds like something you've [23:22] tuned quite a lot over time. [23:24] For people in the room who might be [23:26] trying to build something similar like [23:28] this, what lessons did you learn along [23:31] the way? What kind of tips could you [23:32] give them about how they can build [23:34] something useful like this? [23:38] So, first of all, if you try to do this [23:40] even, I don't know, 6 months ago and you [23:42] got discouraged because you didn't get [23:44] like a return on your investment, uh [23:46] definitely try it try it again. The same [23:50] inflection point that you experienced [23:51] with coding agents back say in November [23:53] with Opus 4.5 and and similar frontier [23:56] technology, uh it also reflects to what [23:59] I just talked about now. So, the the [24:01] fact that long context and models that [24:04] are really capable of reasoning on a lot [24:06] of content means that you can [24:08] practically inject an entire trace of [24:10] your agent into Opus and get fairly [24:13] sophisticated feedback about it. Um so, [24:17] you at the same time you should really [24:19] be investing in collecting all the [24:21] signal that you can from your agent and [24:23] as well as you know, the environment [24:25] where you're executing it. So, I know [24:26] that I I had to run fast during the [24:28] talk, you know, but I couldn't go in [24:30] depth on all the different signals that [24:31] we bring into telescope. It is not [24:33] purely [24:35] what's coming from the trace, it's also [24:37] the feedback that we collect in product. [24:39] Like we have a feedback form, so every [24:40] time the agent doesn't behave correctly [24:43] and the user complains, we have the user [24:45] point of view, we have the trace of the [24:47] agent, we have everything we instrument [24:49] in the platform in DataDog. [24:52] Unsurprisingly, all of that together, [24:54] put, you know, in context, helps even [24:56] more to pinpoint where where the issues. [24:58] Yeah, I think that idea of building the [25:00] clusters, like you were saying that a [25:02] single trace, it's very hard to debug [25:04] what might be going wrong there, but [25:05] when you are able to group them all [25:07] together and get these other signals in, [25:09] really reveals a lot more information [25:11] and something actionable. Makes a ton of [25:12] sense. Yeah, it does and I think it also [25:15] makes the life of an agent builder less [25:17] overwhelming because it really is when [25:19] you start to get a lot of usage, the [25:22] amount of feedback you receive is [25:23] actually overwhelming. And it's a good [25:26] It's a good signal. It means that people [25:27] care about what you're doing, but at the [25:29] same time it's hard to prioritize what's [25:31] actually important versus what doesn't [25:33] move the needle. [25:34] What I showed today is also a way for us [25:36] to try to understand, okay, this cluster [25:39] keeps showing on a daily basis and it's [25:41] fairly large and it has a lot of volume, [25:43] so we should actually fix that first of [25:45] all and then there is the long tail of [25:46] problems to fix will always be there [25:48] with agents given the fact that they're [25:49] non-deterministic. Yeah, that really [25:52] makes me think about the third thing I [25:53] wanted to ask you, which is what you [25:54] said about taste. [25:56] Uh, I think this is really interesting [25:57] point um that the taste of the AI [26:00] engineer is very important and could you [26:02] talk about how your team develops that [26:05] taste? Have you always had this taste or [26:08] have you improved your taste over time? [26:10] Like what's the process of building that [26:12] and developing that for people who might [26:13] be trying to build their own taste? [26:16] I think we all brewed it over time [26:18] because you know, when we launched [26:20] these, you know, was more than a year [26:21] and a half ago, agents were way less [26:24] powerful than they are today. [26:25] Uh, maybe not much taste to be applied. [26:27] It was more like a survival game just to [26:29] make sure it was barely functional. And [26:32] now that they're becoming so powerful [26:34] and you have such a wide variety of [26:36] choices you can make, um then you you [26:38] start to develop a taste that should [26:40] really be in tune with your actual user [26:42] base. Like if if you're building an [26:44] agent for software developers, maybe 80% [26:48] of the choices that we make every day in [26:49] the team will be almost the polar [26:51] opposite of what we do. [26:53] And you always want to keep in mind that [26:56] it's being used by personal likeness [26:57] different from you. Especially in our [26:59] case, like everyone at Rabbit is very [27:00] technical and we are giving the product [27:02] to knowledge workers who have never [27:04] written a single line of code. [27:06] So, it's a it's a very good trade-off to [27:08] to learn. [27:09] Awesome. Thank you for sharing that. [27:11] I'm sure people are going to have lots [27:12] of questions for you and for Peter, who [27:14] is one of the authors of Vibrent. So, [27:16] everyone please come find them and thank [27:17] you so much for coming for [27:19] >> Thank you. [27:20] Have a good one.