Braintrust Scraps Complex Agent Frameworks for Durable Eval Harnesses

Five Hard-earned Lessons About Evals With Braintrust — Ai4

Ai4

Complex agent architectures become obsolete with every foundation model upgrade, making durable evaluation harnesses and continuously failing aspirational benchmarks the only lasting engineering asset.

AI-assisted summary of Five Hard-earned Lessons About Evals With Braintrust — original source. Check the original for full context.

Source published: 2025-08-24

Detailed notes

## Background Presented by Ankur Goyal, Founder and CEO of Braintrust, and published on YouTube by the Ai4 channel on August 24, 2025, this talk shares five hard-earned lessons learned from powering thousands of daily LLM evaluations. According to the page description, the session covers engineering data pipelines, writing custom scorers that catch real failures, optimizing context windows, and utilizing continuous improvement tools like Loop.

## TL;DR Ankur Goyal outlines the essential evaluation strategies that high-performing AI teams use to ship reliable, production-grade applications. Mature engineering organizations evaluate new foundation models within 24 hours and maintain 'offensive' aspirational test suites alongside defensive regression tests to unlock transformative capabilities as models improve. The talk emphasizes treating custom scorers as the modern, programmatic version of Product Requirement Documents (PRDs) and managing an agent's context 'token diet' rather than relying on generic tools. Ultimately, engineering teams should focus on building durable, reusable evaluation harnesses instead of over-engineering ephemeral agent frameworks that become obsolete with each model release.

## Atmosphere & style The presentation carries the brisk, authoritative energy of a founder delivering pragmatic wisdom directly from the production trenches. Goyal speaks with punchy conviction, stripping away speculative hype around autonomous agents to focus on the unglamorous, high-leverage reality of data engineering and benchmark testing. The delivery feels intensely practical, fast-paced, and empowering for software engineers and product managers looking to build disciplined LLM workflows.

## Key takeaways - [00:04](https://www.youtube.com/watch?v=KL7I5GHDC58&t=4s) Mature AI teams should be capable of evaluating and shipping support for newly released foundation models within 24 hours. - [00:04](https://www.youtube.com/watch?v=KL7I5GHDC58&t=4s) The feedback loop from production complaints and user bug reports into versioned evaluation datasets must be frictionless, ideally achievable in two clicks. - [02:44](https://www.youtube.com/watch?v=KL7I5GHDC58&t=164s) Evals should not only be used defensively for regression testing, but also offensively through broken 'aspirational evals' that track unsolved, high-value product capabilities. - [02:44](https://www.youtube.com/watch?v=KL7I5GHDC58&t=164s) Aspirational evals signal the exact moment a new model release or retrieval technique unlocks a viable, groundbreaking feature as failing tests turn green. - [04:07](https://www.youtube.com/watch?v=KL7I5GHDC58&t=247s) High-quality evaluation datasets cannot be bought off-the-shelf; they must be actively engineered and continuously reconciled against live user and leadership feedback. - [04:07](https://www.youtube.com/watch?v=KL7I5GHDC58&t=247s) Custom scorers serve as the modern equivalent of Product Requirement Documents (PRDs), allowing PMs to define exact acceptance criteria through prompt or code-based logic. - [06:32](https://www.youtube.com/watch?v=KL7I5GHDC58&t=392s) Context engineering requires managing an LLM's 'token diet,' as tool definitions and retrieved data now constitute the vast majority of context window tokens rather than system prompts. - [06:32](https://www.youtube.com/watch?v=KL7I5GHDC58&t=392s) Generic tool schemas degrade agent performance; tool definitions and return payloads must be customized and prompt-engineered specifically for the agent's task. - [08:15](https://www.youtube.com/watch?v=KL7I5GHDC58&t=495s) Engineering teams should avoid over-indexing on brittle, complex agent frameworks, which often get simplified to basic while loops as underlying models improve. - [08:15](https://www.youtube.com/watch?v=KL7I5GHDC58&t=495s) Investing in a durable, evaluatable system harness ensures agent code can be completely discarded and rewritten without losing testing benchmarks and quality guardrails. - [11:29](https://www.youtube.com/watch?v=KL7I5GHDC58&t=689s) Braintrust developed its AI optimizer 'Loop' by maintaining a year-long failing aspirational eval until a breakthrough model turned the test green, enabling a two-week product rollout.

## Detailed walkthrough ### [00:04](https://www.youtube.com/watch?v=KL7I5GHDC58&t=4s) Introduction and the 24-Hour Model Test

Braintrust is a two-year-old company that collaborates with some of the most advanced AI software teams in the world to solve notoriously difficult problems around evaluation, observability, and cross-functional workflows between technical and non-technical team members. Once organizations are properly equipped to run evaluations, the volume of evals they execute skyrockets, enabling them to build and ship high-quality products at an unprecedented pace. Drawing from these production experiences, there are five major themes and hard-earned lessons that define what mature eval practices look like in practice.

A primary indicator that an organization has mastered evals is its responsiveness to model releases: teams should be capable of confidently evaluating and launching support for a brand-new model within approximately 24 hours. Given how frequently cutting-edge models are released, staying on the capability frontier is critical. With robust evals in place, an engineering team does not have to guess; they can immediately run their test suites, observe where the new model excels or fails, identify specific edge cases to investigate, and make an informed deployment decision in a single day. Another crucial sign of eval maturity is closing the feedback loop between production and development. If an end user files a complaint or encounters a bug, a developer or product manager should be able to capture that exact failure into an eval dataset within two clicks so the failure mode is permanently tracked and never forgotten.

- **The 24-Hour Benchmark**: Teams should possess the automated infrastructure required to benchmark and assess any newly released foundation model within 24 hours. - **Two-Click Production Ingestion**: The feedback loop from production complaints directly into versioned eval datasets must be virtually frictionless.

---

[02:44](https://www.youtube.com/watch?v=KL7I5GHDC58&t=164s) Playing Offense vs. Defense: Unit Tests and Aspirational Evals

Most engineering organizations treat evals purely defensively as unit tests. In a defensive setup, when a developer modifies a system prompt, they run an existing eval suite to check for regressions—confirming, for example, that 75 test cases remain green while three turn red. While defensive unit testing is essential and should always be done, the highest-performing AI companies use evals to play offense.

Offensive evals are deliberately designed to be broken. These "aspirational evals" represent difficult, world-changing use cases that current models or architectures cannot yet solve. Engineering teams keep these failing evals in their repositories indefinitely. Whenever a superior model is released, a new vector database strategy is implemented, or a novel prompting technique is conceived, teams rerun these aspirational suites. The moment those broken tests begin turning green, the team knows they have unlocked a viable, game-changing product feature ready to ship.

```mermaid flowchart TD accTitle: Defensive vs Offensive Eval Workflows subgraph Defensive ["Defensive Evals (Unit Tests)"] D1["Prompt or Code Change"] --> D2["Run Regression Eval Suite"] D2 --> D3{"Pass Threshold?"} D3 -- "Yes" --> D4["Safe to Deploy"] D3 -- "No" --> D5["Fix Regressions"] end

subgraph Offensive ["Offensive Evals (Aspirational)"] O1["Unsolved High-Value Use Case"] --> O2["Continuously Failing Eval Suite"] O3["New Model Drop / Retrieval Tweak"] --> O2 O2 --> O4{"Tests Turning Green?"} O4 -- "Yes" --> O5["Ship Groundbreaking Feature"] O4 -- "No" --> O6["Remain in Backlog for Next Frontier"] end ```

- **Defensive Evals**: Act as regression guardrails (e.g., verifying 75 passing tests remain intact after prompt changes). - **Offensive / Aspirational Evals**: Live permanently broken test cases representing transformative product capabilities that get unlocked as models and architectures mature.

---

[04:07](https://www.youtube.com/watch?v=KL7I5GHDC58&t=247s) Evals as an Engineering Discipline: Datasets and Custom Scorers

Evaluations must be treated as the central engineering problem of AI application development, rather than an afterthought relegated behind complex agentic architectures. High-quality evals depend on two engineered pillars: high-fidelity datasets and customized scoring logic. Datasets cannot simply be bought off-the-shelf or commissioned once from a third party; they must be actively engineered and continuously reconciled against live customer feedback and executive observations. When leadership tests a product and notes discrepancies with real-world behavior, the engineering team must be able to directly map those qualitative observations back to their evaluation data.

Similarly, while off-the-shelf scorers and template evaluation prompts serve as helpful starting points, reaching enterprise-grade quality requires teams to write domain-specific scorers. Custom scorers represent the modern evolution of product management. Instead of authoring static Product Requirement Documents (PRDs) and handing them over the fence to engineers, product managers should define scoring logic directly. With AI coding assistants like Cursor, PMs who cannot code can readily write programmatic scoring functions, and every PM can craft structured prompt-based scorers to define the exact acceptance criteria for what the system should build.

- **Engineered Datasets**: Dynamic, living representations of production realities that cannot be outsourced or statically purchased. - **Scorers as the New PRD**: Product managers define product specifications directly through prompt-based or code-based scoring functions.

---

[06:32](https://www.youtube.com/watch?v=KL7I5GHDC58&t=392s) Context Engineering and the Token Diet

Context engineering is rapidly becoming one of the most critical disciplines in AI development. A practical way to conceptualize context engineering is as managing an LLM's "token diet." The handwritten tokens inside system prompts and the input tokens submitted directly by end users now represent a tiny minority of the total context window an LLM processes. The vast majority of tokens consumed during an execution originate from tool definitions, database retrievals, and tool execution outputs.

Because tool outputs dominate the context window, developers must be just as deliberate and disciplined when engineering tool outputs as they are with system prompts. A common fallacy surrounding emerging protocols like MCP (Model Context Protocol) is the assumption that generic, universally defined tools will function seamlessly across every agent. In practice, both tool definitions and the data payloads returned by tools must be prompt-engineered and tailored specifically to the agent's exact operational task.

- **The Token Diet Analogy**: System prompts and user inputs are a small fraction of the context window; retrieved data and tool outputs constitute the bulk of the model's diet. - **MCP & Tool Optimization**: Generic tool schemas degrade agent performance; tool definitions and return payloads must be customized and prompt-engineered per use case.

---

[08:15](https://www.youtube.com/watch?v=KL7I5GHDC58&t=495s) Building Durable Evaluatable Systems Over Ephemeral Agent Code

Engineering cycles should not be squandered building brittle, over-engineered agent frameworks, state graphs, or complex control flows that teams become emotionally attached to. Foundation models evolve so rapidly that code written today may become obsolete tomorrow. For instance, a leading AI code-generation company systematically *decreased* the total lines of code in its core engine with every subsequent model release—transitioning away from rigid third-party frameworks down to a minimalistic `while` loop because their CTO held zero attachment to custom infrastructure.

Instead of over-indexing on ephemeral agent code, durable engineering effort should be invested into building an "evaluatable system." When teams focus on constructing comprehensive evaluation harnesses—comprising the data pipeline, custom scorers, feedback capture, and orchestration—they create a resilient artifact. If an underlying model update obsoletes the agent logic, the evaluation system remains fully intact to test, optimize, and validate whatever replacement architecture is introduced.

```mermaid flowchart LR accTitle: The Full Evaluatable System Architecture subgraph System ["Evaluatable System Harness"] direction TB subgraph Core ["Agent Execution (Ephemeral)"] A["Input / State"] --> B["Simple While Loop"] B <--> C["Tool Calls & Context Diet"] B --> D["Output"] end subgraph EvalEngine ["Durable Evaluation Layer"] E["Engineered Datasets"] -.-> B D -.-> F["Custom Scorers & Feedback"] F -.-> G["Continuous Optimization / Ingestion"] G -.-> E end end ```

- **Ephemeral Agent Logic**: Complex graphs and frameworks frequently get replaced by simple `while` loops as base models improve. - **Durable Eval Harness**: Investing in the surrounding evaluation box ensures architectures can be completely discarded and rewritten without losing quality benchmarks.

---

[11:29](https://www.youtube.com/watch?v=KL7I5GHDC58&t=689s) Case Study: Building Loop and Automated Continuous Improvement

The culmination of these evaluation principles is embodied in Loop, an AI copilot and optimizer built directly into Braintrust. For roughly a year, Braintrust maintained an aspirational, continuously failing eval designed to test whether LLMs could autonomously optimize prompts, analyze dataset failure patterns, and write their own custom scoring functions. For months across iterative model releases—up to and including GPT-4o—the results were consistently poor, and the evals remained red.

The breakthrough occurred with the release of Claude 4, where evaluation benchmark numbers jumped dramatically. Because the team had maintained their aspirational eval suite for a year, they recognized the capability shift instantly, threw away previous prototype code, and shipped Loop within two weeks as a simple `while` loop augmented with tool calls. Loop automates the entire eval-optimization lifecycle. Testing across current models reveals distinct operational profiles:

Braintrust actively operates Loop in production with a collection of working, fully deployed evals, while concurrently running roughly eight broken, aspirational evals for upcoming capabilities that are monitored daily as frontier models continue to evolve.

## Notable quotes > [00:04](https://www.youtube.com/watch?v=KL7I5GHDC58&t=4s) teams should be capable of confidently evaluating and launching support for a brand-new model within approximately 24 hours. — Ankur Goyal

> [00:04](https://www.youtube.com/watch?v=KL7I5GHDC58&t=4s) If an end user files a complaint or encounters a bug, a developer or product manager should be able to capture that exact failure into an eval dataset within two clicks so the failure mode is permanently tracked and never forgotten. — Ankur Goyal

> [02:44](https://www.youtube.com/watch?v=KL7I5GHDC58&t=164s) Most engineering organizations treat evals purely defensively as unit tests. — Ankur Goyal

> [02:44](https://www.youtube.com/watch?v=KL7I5GHDC58&t=164s) The moment those broken tests begin turning green, the team knows they have unlocked a viable, game-changing product feature ready to ship. — Ankur Goyal

> [04:07](https://www.youtube.com/watch?v=KL7I5GHDC58&t=247s) Evaluations must be treated as the central engineering problem of AI application development, rather than an afterthought relegated behind complex agentic architectures. — Ankur Goyal

> [04:07](https://www.youtube.com/watch?v=KL7I5GHDC58&t=247s) Custom scorers represent the modern evolution of product management. Instead of authoring static Product Requirement Documents (PRDs) and handing them over the fence to engineers, product managers should define scoring logic directly. — Ankur Goyal

> [06:32](https://www.youtube.com/watch?v=KL7I5GHDC58&t=392s) A practical way to conceptualize context engineering is as managing an LLM's 'token diet.' — Ankur Goyal

> [06:32](https://www.youtube.com/watch?v=KL7I5GHDC58&t=392s) The vast majority of tokens consumed during an execution originate from tool definitions, database retrievals, and tool execution outputs. — Ankur Goyal

> [06:32](https://www.youtube.com/watch?v=KL7I5GHDC58&t=392s) In practice, both tool definitions and the data payloads returned by tools must be prompt-engineered and tailored specifically to the agent's exact operational task. — Ankur Goyal

> [08:15](https://www.youtube.com/watch?v=KL7I5GHDC58&t=495s) Engineering cycles should not be squandered building brittle, over-engineered agent frameworks, state graphs, or complex control flows that teams become emotionally attached to. — Ankur Goyal

> [08:15](https://www.youtube.com/watch?v=KL7I5GHDC58&t=495s) When teams focus on constructing comprehensive evaluation harnesses—comprising the data pipeline, custom scorers, feedback capture, and orchestration—they create a resilient artifact. — Ankur Goyal

> [11:29](https://www.youtube.com/watch?v=KL7I5GHDC58&t=689s) Because the team had maintained their aspirational eval suite for a year, they recognized the capability shift instantly, threw away previous prototype code, and shipped Loop within two weeks as a simple while loop augmented with tool calls. — Ankur Goyal

## Mentioned Ankur Goyal, Braintrust, Loop, Notion, Cursor, MCP (Model Context Protocol), Claude 4, GPT-4o, GPT-4.1, GPT-5, Gemini

## Video description Ankur Goyal, Founder & CEO, Braintrust

Abstracts

Are you catching critical issues in your LLM applications before they reach users? In this talk, I’ll share five hard‑earned lessons drawn from powering thousands of daily evals on Braintrust. You’ll discover how to engineer data pipelines and custom scorers that surface real failures, why optimizing your full eval loop outperforms prompt tweaks alone, and how Loop, our AI copilot, automates continuous improvement. I'll share real examples like Notion’s 24‑hour model swaps and demonstrate practical steps to tighten your eval workflows. Join me to learn actionable strategies that ensure your LLM features ship reliably and confidently.

[Source](https://www.youtube.com/watch?v=KL7I5GHDC58)

中文简报

Braintrust弃用复杂Agent框架，转而构建持久评估体系复杂的Agent架构在基础模型每次升级时都会面临淘汰，使得持久的评估测试框架与持续处于失败状态的前瞻性基准成为唯一具备长效价值的工程资产。 Ankur Goyal概述了顶尖AI团队用于交付可靠、生产级应用的核心评估策略。成熟的工程团队能够在24小时内评估新发布的基础模型，并在维护防御性回归测试的同时运行“进攻性”前瞻测试集，以便随着模型性能的提升及时解锁颠覆性能力。演讲强调应将自定义评分器视为产品需求文档（PRD）的现代代码化实现，并精细管理Agent上下文的“Token摄入”，而非依赖通用工具。工程团队最终应当专注于构建持久且可复用的评估测试框架，而不是过度设计那些随模型发布迅速过时的临时Agent框架。成熟的AI团队应当具备在24小时内评估新发布的基础模型并上线支持的能力。 (0:04) 从生产环境中的用户问题与Bug报告到版本化评估数据集的反馈闭环必须保持顺畅，理想情况下仅需两次点击即可完成。 (0:04) 评估（Eval）不仅应用于防御性回归测试，还应通过处于未通过状态的“前瞻性评估”进行进攻性布局，以持续追踪尚未解决的高价值产品能力。 (2:44) 当未通过的测试转为通过（变绿）时，前瞻性评估能精准标明新模型发布或检索技术何时解锁了切实可行且具突破性的功能。 (2:44) 高质量的评估数据集无法直接开箱现成购买，必须主动进行工程化构建，并根据真实用户和管理层的反馈持续核对校准。 (4:07)

Reading summary

Braintrust argues that mature AI development treats evaluation infrastructure—rather than ephemeral agent architectures—as the central engineering discipline, using fast feedback loops and aspirational tests to ship features quickly as foundation models evolve. Rapid Model Benchmarking and Frictionless Feedback (0:04) According to Braintrust, a core metric of evaluation maturity is an engineering team's ability to benchmark and ship support for newly released foundation models within approximately 24 hours. Because frontier models launch frequently, robust test suites allow teams to diagnose capabilities, pinpoint edge cases, and deploy updates in a single day rather than guessing. Balancing Defensive Unit Tests with Offensive Evals (2:44) While standard defensive evaluations function as regression tests—verifying that passing prompts or code changes do not break existing test cases—advanced teams also play offense. Offensive, or aspirational, evals are intentionally designed to remain broken, representing difficult product capabilities that current architectures cannot yet achieve. Teams maintain these failing suites in repositories indefinitely. Whenever a new foundation model, prompting technique, or vector database optimization appears, developers rerun the aspirational tests. The moment these tests turn green, teams can immediately package and ship the unlocked capability as a production feature. Engineered Datasets and Scorers as Modern PRDs (4:07) High-quality evaluation systems require continuous engineering rather than off-the-shelf purchases. Datasets must be dynamic assets reconciled against live customer bug reports—ideally captured from production into eval sets within two clicks—and leadership feedback. Similarly, Braintrust views custom scoring functions as the modern replacement for static Product Requirement Documents (PRDs). Product managers can define acceptance criteria directly through prompt-based scorers or programmatic code written with AI coding assistants like Cursor, ensuring domain-specific evaluation logic matches product intent. Context Engineering and Managing the Token Diet (6:32) System prompts and raw user queries now make up only a tiny fraction of an LLM's context window. Instead, the bulk of a model's 'token diet' consists of database retrievals, tool definitions, and tool execution outputs. Braintrust cautions against assuming emerging protocols like Model Context Protocol (MCP) allow generic, plug-and-play tools. Because tool payloads dominate the context window, tool schemas and returned data must be deliberately prompt-engineered and tailored to each agent's specific operational task to prevent performance degradation. Prioritizing Durable Eval Systems Over Ephemeral Code (8:15) Braintrust advises against over-investing in rigid agent frameworks or complex state graphs, which often become obsolete when newer models arrive. For example, one AI code-generation company systematically reduced its codebase down to a simple while loop with each model upgrade because leadership avoided attachment to custom infrastructure. This pattern informed Braintrust’s development of Loop, an autonomous prompt and scorer optimizer. Braintrust ran an aspirational eval for a year that consistently failed until Claude 4 was released. Upon observing benchmark gains, the team discarded old prototypes and shipped Loop in two weeks as a minimal while loop with tool calls. In comparative internal testing, Claude 4 demonstrated strong tool synthesis, GPT-4.1 trailed slightly, Gemini struggled with reliable tool-calling execution, and evaluations for GPT-5 remain ongoing. Braintrust continues to run Loop in production alongside roughly eight failing aspirational evaluations, which the team monitors daily to detect the next capabilities unlocked by incoming frontier models.

精炼阅读摘要

Braintrust 认为 AI 应用工程的核心在于打造持久耐用的“可评估系统”，而非过度沉溺于瞬态脆弱的 Agent 架构。团队应当建立敏捷的评测闭环，以防御性测试保障基准，并利用长期失败的愿景评估主动捕捉前沿模型突破。防御与进攻：24小时基准与愿景评估 (0:04) 成熟的评估能力体现在对前沿模型发布的极速响应上。Braintrust 指出，具备成熟评测基础设施的团队应能在新基础模型发布约 24 小时内完成基准评测并上线部署决策，同时支持通过“两次点击”将生产环境中的用户报错无缝捕获至评估数据集。这种敏捷闭环消除了部署新模型时的盲目猜测。在功能测试逻辑上，团队需将评估划分为防御性与进攻性两类。防御性评估类似于传统单元测试，用于在修改 Prompt 或代码时排查回归问题；领先团队则会部署故意通不过的“愿景评估”（Aspirational Evals）。这类评估针对当前模型无法解决的高价值难题长期保留在代码库中，一旦新模型演进或检索优化使指标翻绿，团队便能立即验证并交付突破性功能。评估工程支柱：动态数据与评分器规范 (4:07) 评估工作应当被视作 AI 应用开发的核心工程问题。高质量的评测依赖两大支柱：高保真数据集与定制化评分逻辑。数据集不能依赖外部现成采购或一次性委托，而必须作为动态工程资产，持续与生产环境反馈及管理层的实际观察进行校准映射，确保测试用例反映真实业务行为。通用评分模板无法满足企业级质量要求，团队必须编写特定领域的自定义评分器。Braintrust 提出“评分器即新型 PRD”的概念：借助 Cursor 等 AI 编程助手，产品经理无需精通复杂编码，也能直接编写程序化评分函数或结构化的 Prompt 评分器，将验收标准直接转化为可执行的机器评测逻辑，消除研发沟通鸿沟。上下文工程与“Token 饮食”治理 (6:32) 上下文工程的核心在于对大语言模型的“Token 饮食”（Token Diet）实施精细化管理。在实际运行中，人工编写的系统提示词和用户直接输入的 Token 仅占上下文窗口极小比例，绝大部分 Token 消耗来自工具定义、数据库检索数据以及工具执行的返回结果。针对以 MCP（Model Context Protocol）为代表的协议，Braintrust 提醒开发人员警惕“通用工具定义可无缝适配所有 Agent”的误区。若工具返回的载荷未经提示词工程优化和裁剪，会显著稀释上下文质量并拉低模型表现。团队必须对工具模式及其输入输出数据进行针对性的定制与工程优化。投资持久评估系统，摒弃瞬态 Agent (8:15) 基础模型的快速迭代使得复杂的状态图、第三方 Agent 框架和脆弱的控制流极易过时。来源案例指出，某头部 AI 代码生成公司的 CTO 在每次模型能力提升后都会系统性缩减核心代码，最终将复杂的第三方框架精简为纯粹的最小化 while 循环，避免对定制化控制逻辑产生技术依赖。工程团队应当避免将资源耗费在瞬态的 Agent 代码上，而应全力构建包含数据管道、自定义评分器与反馈捕获机制的“可评估系统”（Evaluatable System）。只要外部评测框架具备韧性，即便底层模型演进彻底淘汰了旧有的 Agent 控制架构，团队也能无损迁移并快速验证替代方案。 Loop 实践：以前沿模型突破驱动自动化 (11:29) Braintrust 内部孵化的优化器 Loop 验证了攻防评估体系的有效性。团队曾维护一项长达一年的持续失败愿景评估，用于测试大模型能否自主优化 Prompt 并生成评分器；在经历包括 GPT-4o 在内的多次迭代均未通过后，Claude 4 的发布使评测指标出现质的飞跃，团队随即抛弃旧原型代码，在两周内基于极简 while 循环和工具调用交付了 Loop。实测对比显示：Claude 4 在工具调用与评分器合成上表现优异；GPT-4.1 具备可用性但优化表现稍逊；GPT-5 仍处于持续基准测试中；Gemini 表现积极但在稳定执行工具调用上存在局限。目前 Braintrust 在生产环境除了运行部署评测外，仍并行维护约 8 个针对未来能力的愿景评估，以持续监控前沿边界。 Braintrust 证实了将工程重点由编写瞬态控制流转向构建动态评估体系的价值，但对于 GPT-5 等更新一代模型的全面优化能力，其基准评测仍在进行中，行业如何在大规模复杂调用中持续平衡自动化工具开销与模型能力边界仍待持续检验。

Detailed notes — zh

## Background 该视频由 YouTube 频道 Ai4 于 2025 年 8 月 24 日发布。视频内容为主办方邀请 Braintrust 创始人兼 CEO Ankur Goyal 带来的技术主题演讲，分享基于 Braintrust 平台每天运行数千次评估所总结出的五大宝贵经验，探讨如何构建数据管道、编写定制化评分器以及通过 AI Copilot 实现自动化持续优化。 ## TL;DR 演讲围绕 LLM 应用的评估体系展开，指出成熟的评估体系需具备 24 小时内适配新模型、快速捕获生产环境 Bad Case 以及设置进攻型评估用例三大特征。作者强调团队应将评估本身视为一等公民的工程问题，重点投入高质量数据集与定制化评分器的构建，并将评分器作为下一代产品需求规格说明书。同时，开发者应做好工具层面的上下文工程与“Token 饮食”管理，摆脱对脆弱 Agent 胶水代码的执念，转而构建包含核心逻辑、测试集与反馈闭环的“可评估系统”。最终，结合 Braintrust 自研的自动化优化 Agent“Loop”，展示了利用前沿大模型闭环驱动 LLM 应用持续进化的工程实践。 ## Atmosphere & style 整场演讲风格极其务实、干货密集且直击 AI 工程落地的痛点。讲者 Ankur Goyal 展现出深谙一线实战的冷静与敏锐，以精炼有力的技术洞察打破了业内对复杂 Agent 框架的盲目崇拜。叙事节奏明快，兼具工程实用主义与前瞻性视野，让听众在真实的踩坑案例与清醒的技术反思中感受到构建高可靠 AI 系统的严谨与笃定。 ## Key takeaways - [00:59](https://www.youtube.com/watch?v=KL7I5GHDC58&t=59s) 优秀的评估体系能让团队在基础模型更新后 24 小时内完成评测，并充满信心地决定是否切换或跟进。 - [00:59](https://www.youtube.com/watch?v=KL7I5GHDC58&t=59s) 建立生产环境到评测集的低阻力闭环至关重要，线上用户的真实报错应能在两次点击内捕获并沉淀为测试用例。 - [00:59](https://www.youtube.com/watch?v=KL7I5GHDC58&t=59s) 评估体系应兼具防守与进攻能力：既要通过防守型测试防止代码回归，也要维护常态化报错的雄心型用例以驱动突破性创新。 - [04:07](https://www.youtube.com/watch?v=KL7I5GHDC58&t=247s) 评估必须被当作核心工程问题对待，高质量的评测数据集无法简单外包，必须结合真实业务深度工程化构建。 - [04:07](https://www.youtube.com/watch?v=KL7I5GHDC58&t=247s) 定制化评分器正在成为下一代产品管理的核心形态，产品经理应直接通过代码或自然语言参与定义评分逻辑来作为产品 Spec。 - [06:32](https://www.youtube.com/watch?v=KL7I5GHDC58&t=392s) 在当前的上下文工程中，工具调用及其返回内容占据了绝大部分 Token，必须对其进行严格裁剪与 Prompt 定制以控制模型摄入。 - [08:15](https://www.youtube.com/watch?v=KL7I5GHDC58&t=495s) 复杂的 Agent 胶水代码极易随着底层大模型的升级而被淘汰，高质量的数据集与评分器才是最持久耐用的研发工程资产。 - [09:51](https://www.youtube.com/watch?v=KL7I5GHDC58&t=591s) 开发 AI 应用不应局限于孤立的 Agent，而应将其与评分逻辑、反馈闭环整合成一个整体的“可评估系统”进行全局优化。 - [11:29](https://www.youtube.com/watch?v=KL7I5GHDC58&t=689s) Braintrust 基于新一代前沿模型构建了自动化评估 Agent 工具 Loop，通过轻量级 while 循环结合工具调用来全面自动化 Eval 优化流程。 ## Detailed walkthrough ### [00:04](https://www.youtube.com/watch?v=KL7I5GHDC58&t=4s) 深入评估（Evals）：构建顶尖 AI 应用的基石 Braintrust 成立约两年时间，一直与全球最前沿的 AI 应用开发企业并肩作战，帮助他们攻克极其棘手的技术挑战。在 AI 软件的构建过程中，评估（Evals）、可观测性（Observability）以及打破技术人员与非技术人员之间的协作壁垒，都是极具挑战的核心难题。实践中最令人振奋的现象是：一旦团队真正具备了自动化运行评估的能力，他们执行评估的频率和打造优秀产品的速度都会迎来爆发式的增长。在伴随众多客户探索的过程中，团队积累了大量踩坑经验。本次分享提炼了过去几年中总结出的五大核心经验，旨在帮助开发者和产品团队在模型飞速迭代的浪潮中，建立起科学、可靠的 LLM 评测与交付体系。 ### [00:59](https://www.youtube.com/watch?v=KL7I5GHDC58&t=59s) 良好评估体系的三个衡量标志一个组织是否真正建立了成熟的评估体系，通常有三个非常清晰的判断标准： ```mermaid flowchart TD accTitle: 优秀评估体系的三大支柱 accDescr: 展示了优秀评估体系在模型迭代、用户反馈和前瞻性探索三个维度的表现。 subgraph S1["1. 快速响应能力"] A["新模型发布"] --> B["24 小时内完成评估与决策"] end subgraph S2["2. 闭环反馈机制"] C["生产环境用户吐槽/报错"] --> D["两步点击捕获并沉淀至评测集"] end subgraph S3["3. 评估驱动创新"] E["单元测试型防守评估<br/>(防回归/保持绿灯)"] F["野心型进攻评估<br/>(常态报错/攻克即颠覆)"] end ``` 第一，**能否在 24 小时内完成新模型的评估与适配**。如今基础大模型的更新换代极为频繁，保持在模型能力的最前沿至关重要。当外部发布了新的基座模型时，一个拥有优秀评估体系的团队并不一定会立刻全量切换，但他们绝对有能力在 24 小时内充满信心地给出评测结论：“新模型已经测试完毕，这是评估数据对比，其中有几个特定场景需要我们进一步深挖”，从而极快地完成技术评估。第二，**生产环境到研发测试的反馈链路是否足够顺畅**。当线上真实用户产生抱怨或遭遇 Bad Case 时，系统应当支持在“两次点击以内”将该反馈捕获并保存到评估测试集中。建立高水准评估最难的部分，就是打通生产环境发生的真实情况与开发者/产品经理日常测试之间的闭环。如果捕获和沉淀用户反馈的过程充满阻力，团队就很难建立起贴近实际的高质量评估。第三，**用评估来“打进攻”（Offense）而非单纯“打防守”（Defense）**。 - **防守型评估（Defensive Evals）**：类似于传统的单元测试。例如工程师调整了 Prompt，跑一遍测试集发现“75 个通过、3 个报错”，用来确认自己有没有把现有功能改坏。这是基础且必要的。 - **进攻型/雄心勃勃的评估（Aspirational Evals）**：顶尖团队的代码库里往往运行着一批“常态化处于红灯报错状态”的评估用例。这些用例代表了那些一旦被攻克就足以“改变世界”、带来颠覆性体验的极致场景。随着更强模型发布或团队想出了全新的架构思路，重新运行这些进攻型评估，一旦它们开始变绿，就意味着可以发布突破性的新功能。 ### [04:07](https://www.youtube.com/watch?v=KL7I5GHDC58&t=247s) 将评估视为核心工程问题：高质量数据集与定制评分器构建高水平评估的关键法则在于：**必须将评估本身当作一等公民的工程问题来对待**。评估绝不是项目后期的临时补丁；实际上，最优秀的 AI 团队往往把评估作为首要工程问题来攻坚，而具体的 Agent 架构设计反而退居次席。工程化评估的核心由两部分组成： 1. **工程化构建数据集（Datasets）**：优秀的数据集既不可能凭空获得，也无法简单通过花钱外包采购来一劳永逸地解决问题。它必须由团队结合业务场景深度工程化打造。一个良好的工程文化标志在于，工程和产品团队坚信其评估数据集能真实反映现实世界。当管理层亲自体验产品并提出“这几个场景似乎效果不好”时，团队应该能够迅速将这些体感或客诉与现有的评估数据集进行对齐与复现。 2. **构建定制化评分器（Scorers）**：很多团队倾向于直接套用现成（Off-the-shelf）的通用评分方法或从网上复制 Prompt。虽然 Braintrust 等平台也内置了许多开箱即用的评分工具，但在实际业务中，不亲自构建专属评分器就很难达到极致的产品质量。评分器本质上是“下一代产品管理（Next-generation Product Management）”的新形态。产品经理不应再花费大量时间编写干瘪的传统 PRD（产品需求文档）然后丢给研发，而是应该直接参与定义评分逻辑（Scoring Logic）。即使部分产品经理不会手写代码，也可以在 Cursor 等 AI 编程工具的辅助下完成，或者直接撰写基于自然语言 Prompt 的评估逻辑，将评分器直接作为项目的最终产品规格说明书（Spec）。 ### [06:32](https://www.youtube.com/watch?v=KL7I5GHDC58&t=392s) 上下文工程与“Token 饮食”管理随着大模型应用的发展，“上下文工程（Context Engineering）”正在从一个极客圈内的小众概念迅速走向主流。理解上下文工程有一个非常形象的比喻——**“Token 饮食（Diet of Tokens）”**。 ```mermaid flowchart LR accTitle: LLM 上下文 Token 构成与工具工程化 accDescr: 说明了模型实际处理的 Token 大部分来自工具调用的输出，而非提示词本身。 subgraph Context["LLM 实际处理的 Token 结构"] direction TB A["System Prompt & 用户输入<br/>(极少数比例)"] B["工具定义与工具返回输出<br/>(绝大部分比例)"] end Context --> C["上下文工程核心要求"] C --> D["针对 Agent 目标深度 Prompt 化定制工具定义"] C --> E["精细过滤与调优工具返回的输出格式"] ``` 在当下的复杂 LLM 交互中，开发者手工编写的 System Prompt 以及用户直接输入的文本，在模型实际处理的全部 Token 中已经沦为“极少数”；绝大多数 Token 实际上是由工具调用（Tool Calls）及其返回的数据所占据。因此，开发者必须像精心设计 System Prompt 一样，严谨审视工具吐出的每一个 Token。目前社区在采用诸如 MCP（Model Context Protocol）等协议时存在一个常见误区：误以为定义一个高度通用化的工具就能适配所有 Agent。事实上，通用的定义往往效果不佳，开发者必须针对 Agent 要解决的具体垂直场景，对工具的定义（Tool Definitions）和工具返回的输出结果（Tool Outputs）进行深度的 Prompt 工程与格式裁剪，控制好大模型摄入的“Token 饮食结构”。 ### [08:15](https://www.youtube.com/watch?v=KL7I5GHDC58&t=495s) 研发投资的持久性：拥抱模型颠覆，资产沉淀在评估中在投入实质性的工程研发资源时，团队需要清晰地认识到哪些资产是易逝的，哪些是长久耐用的： - **不要过度迷恋复杂的 Agent 胶水代码**：不要为你手动编写的复杂 For 循环、多层状态图（Graphs）或复杂的 Agent 框架感到自豪。底座大模型每演进一代，就可能瞬间颠覆并抹平上层所有的复杂设计。 - **头部代码生成公司的真实案例**：Braintrust 的一家知名 AI 代码生成（Codegen）客户，其 CTO 在团队早期就树立了“对现有代码零执念”的文化，明确告知团队“明天新模型发布，我们可能就要把今天写的代码全部扔掉”。事实上，随着底层基础模型能力的不断跃升，该产品的核心代码行数不增反降，他们直接丢弃了原先沉重的 Agent 框架，将其大幅重构成了一个极其精简的 `while` 循环。 - **最持久的工程投资是数据集与评分器**：无论底层的 Agent 架构如何推倒重构，团队辛勤构建的高质量评估数据集和业务评分器永远具有持久价值。当团队决定扔掉旧架构尝试全新的技术路径时，正是这些持续沉淀的评估资产在为每一次重构和升级保驾护航。 ### [09:51](https://www.youtube.com/watch?v=KL7I5GHDC58&t=591s) 构建“可评估系统”（Evaluatable System）而非孤立的智能体如果要在工程资源上做出取舍，最值得投入精力的地方绝不是反复死磕单一 Prompt 的微调，而是构建一套强大的反馈闭环（Feedback Loop）、基础设施以及精准的评分逻辑（Scoring Logic）。这引出了贯穿整个演讲的总结性核心观点：在构建 AI 应用时，工程师的目标不应仅仅是写一个 Agent、写一段 `for` 循环或做一个 Chatbot，而是必须将其作为一个**可评估系统（Evaluatable System）**来整体构建。 ```mermaid flowchart LR accTitle: Evaluatable System Architecture subgraph SystemBox ["可评估系统 (Evaluatable System)"] direction TB Agent["核心逻辑 / Agent / Chatbot"] Scorer["评分逻辑与测试集 (Scoring & Datasets)"] Feedback["反馈与数据闭环 (Feedback Loop)"] Agent <--> Scorer Scorer <--> Feedback end Optimizer["优化器 / 外部模型 (如 Loop)"] -->|基于全局信息优化| SystemBox ``` 你所打造的交付物（Artifact）应当是外层那个“可被评估的完整黑盒”，而不仅仅是位于中心的 Agent 本身。唯有把评估指标、数据集和核心逻辑整合成一个系统，当底层的底层大语言模型或技术浪潮发生迭代更替时，你的系统才具备持续进化和存活的能力。Braintrust 曾在其内置的优化器 **Loop** 上做过一组对照实验：如果仅把 System Prompt 喂给大模型让其尝试优化，效果非常有限；但如果把包含评分函数（Scoring Functions）在内的整套可评估系统全部开放给模型进行全局优化，大模型的优化表现会获得显著跃升，因为更全面的上下文为它提供了充足的决策依据。 --- ### [11:29](https://www.youtube.com/watch?v=KL7I5GHDC58&t=689s) 自动化评估与优化：Braintrust 内置 Agent “Loop” 的诞生与实践谈及系统级评估的终极形态，Braintrust 推出了自动化评估与优化工具 **Loop**。在此之前，团队维护了一套评估基准长达一年之久，每逢市面上有新的基础大模型发布，就会拉跑一次测试。但在很长一段时间里（包括 GPT-4o 阶段），测试结果都令人沮丧——当时的大模型在自主分析数据集、自动优化 Prompt 以及编写评分函数方面表现极差。转折点出现在新一代前沿模型（如 Claude 4 系列）问世时，评测指标突然出现了令人极其振奋的跃升。团队果断舍弃了原有的旧代码，仅用了两周时间就快速构建并上线了 Loop。 ```mermaid flowchart TD accTitle: Loop Agent Evaluation and Optimization Process subgraph LoopWorkflow ["Loop 内部运行机制"] Start["接收可评估系统上下文"] --> WhileCheck{"While 循环优化中"} WhileCheck -->|执行 Tool Call| ToolAction["读取数据集 / 调整 Prompt / 编写评分函数"] ToolAction --> EvalRun["运行 Evals 评估效果"] EvalRun -->|反馈得分| WhileCheck WhileCheck -->|达到最优或收敛| Done["输出优化后的系统配置"] end ``` Loop 的本质是一个驻留在 Braintrust 内部、利用 LLM 全面接管并自动化 Eval 流程的 AI Agent。它的底层实现极其轻量优雅，本质上就是一个带有工具调用（Tool Calls）能力的 `while` 循环。在实际落地中，不同模型的表现差异明显： - **Claude 4 系列**：表现尤为出色，能够高质量完成数据洞察与评分函数编写。 - **GPT-4.1 / 后续模型**：表现尚可（OK），对于更新一代模型（如 GPT-5）的效果团队仍在持续观察与评估中。 - **Gemini**：虽然模型态度十分“积极努力”，但在精准的工具调用（Tool Calling）能力上存在短板，导致无法顺利跑通完整流程（期待官方后续修复）。目前，Braintrust 已在产品中内置并开放了多项经过验证、运行良好的 Loop 评估能力，同时内部还有大约 8 个难度更高、尚未完全攻克的实际用例正在接受每日监控与迭代优化。越来越多的企业客户不仅将 Loop 作为提升自身 Eval 效率的利器，更将其视为在实际软件产品中落地 Agent 架构的“参考实现”（Reference Implementation）。 ## Notable quotes > [00:59](https://www.youtube.com/watch?v=KL7I5GHDC58&t=59s) 新模型已经测试完毕，这是评估数据对比，其中有几个特定场景需要我们进一步深挖 — Ankur Goyal > [00:59](https://www.youtube.com/watch?v=KL7I5GHDC58&t=59s) 建立高水准评估最难的部分，就是打通生产环境发生的真实情况与开发者/产品经理日常测试之间的闭环。 — Ankur Goyal > [04:07](https://www.youtube.com/watch?v=KL7I5GHDC58&t=247s) 必须将评估本身当作一等公民的工程问题来对待 — Ankur Goyal > [04:07](https://www.youtube.com/watch?v=KL7I5GHDC58&t=247s) 评分器本质上是“下一代产品管理（Next-generation Product Management）”的新形态。 — Ankur Goyal > [06:32](https://www.youtube.com/watch?v=KL7I5GHDC58&t=392s) 在当下的复杂 LLM 交互中，开发者手工编写的 System Prompt 以及用户直接输入的文本，在模型实际处理的全部 Token 中已经沦为“极少数”；绝大多数 Token 实际上是由工具调用（Tool Calls）及其返回的数据所占据。 — Ankur Goyal > [08:15](https://www.youtube.com/watch?v=KL7I5GHDC58&t=495s) 不要为你手动编写的复杂 For 循环、多层状态图（Graphs）或复杂的 Agent 框架感到自豪。底座大模型每演进一代，就可能瞬间颠覆并抹平上层所有的复杂设计。 — Ankur Goyal > [08:15](https://www.youtube.com/watch?v=KL7I5GHDC58&t=495s) 明天新模型发布，我们可能就要把今天写的代码全部扔掉 — Ankur Goyal > [08:15](https://www.youtube.com/watch?v=KL7I5GHDC58&t=495s) 无论底层的 Agent 架构如何推倒重构，团队辛勤构建的高质量评估数据集和业务评分器永远具有持久价值。 — Ankur Goyal > [09:51](https://www.youtube.com/watch?v=KL7I5GHDC58&t=591s) 在构建 AI 应用时，工程师的目标不应仅仅是写一个 Agent、写一段 for 循环或做一个 Chatbot，而是必须将其作为一个可评估系统（Evaluatable System）来整体构建。 — Ankur Goyal > [09:51](https://www.youtube.com/watch?v=KL7I5GHDC58&t=591s) 你所打造的交付物（Artifact）应当是外层那个“可被评估的完整黑盒”，而不仅仅是位于中心的 Agent 本身。 — Ankur Goyal ## Mentioned Ankur Goyal, Braintrust, Notion, Loop, Cursor, MCP (Model Context Protocol), GPT-4o, Claude 4, GPT-4.1, GPT-5, Gemini ## Video description Ankur Goyal, Founder & CEO, Braintrust Abstracts Are you catching critical issues in your LLM applications before they reach users? In this talk, I’ll share five hard‑earned lessons drawn from powering thousands of daily evals on Braintrust. You’ll discover how to engineer data pipelines and custom scorers that surface real failures, why optimizing your full eval loop outperforms prompt tweaks alone, and how Loop, our AI copilot, automates continuous improvement. I'll share real examples like Notion’s 24‑hour model swaps and demonstrate practical steps to tighten your eval workflows. Join me to learn actionable strategies that ensure your LLM features ship reliably and confidently. [Source](https://www.youtube.com/watch?v=KL7I5GHDC58)

Source quotes

[0:59] Ankur Goyal: 新模型已经测试完毕，这是评估数据对比，其中有几个特定场景需要我们进一步深挖 [0:59] Ankur Goyal: 建立高水准评估最难的部分，就是打通生产环境发生的真实情况与开发者/产品经理日常测试之间的闭环。 [4:07] Ankur Goyal: 必须将评估本身当作一等公民的工程问题来对待 [4:07] Ankur Goyal: 评分器本质上是“下一代产品管理（Next-generation Product Management）”的新形态。 [6:32] Ankur Goyal: 在当下的复杂 LLM 交互中，开发者手工编写的 System Prompt 以及用户直接输入的文本，在模型实际处理的全部 Token 中已经沦为“极少数”；绝大多数 Token 实际上是由工具调用（Tool Calls）及其返回的数据所占据。 [8:15] Ankur Goyal: 不要为你手动编写的复杂 For 循环、多层状态图（Graphs）或复杂的 Agent 框架感到自豪。底座大模型每演进一代，就可能瞬间颠覆并抹平上层所有的复杂设计。 [8:15] Ankur Goyal: 明天新模型发布，我们可能就要把今天写的代码全部扔掉 [8:15] Ankur Goyal: 无论底层的 Agent 架构如何推倒重构，团队辛勤构建的高质量评估数据集和业务评分器永远具有持久价值。 [9:51] Ankur Goyal: 在构建 AI 应用时，工程师的目标不应仅仅是写一个 Agent、写一段 for 循环或做一个 Chatbot，而是必须将其作为一个可评估系统（Evaluatable System）来整体构建。 [9:51] Ankur Goyal: 你所打造的交付物（Artifact）应当是外层那个“可被评估的完整黑盒”，而不仅仅是位于中心的 Agent 本身。

Source quotes

[0:04] Ankur Goyal: teams should be capable of confidently evaluating and launching support for a brand-new model within approximately 24 hours. [0:04] Ankur Goyal: If an end user files a complaint or encounters a bug, a developer or product manager should be able to capture that exact failure into an eval dataset within two clicks so the failure mode is permanently tracked and never forgotten. [2:44] Ankur Goyal: Most engineering organizations treat evals purely defensively as unit tests. [2:44] Ankur Goyal: The moment those broken tests begin turning green, the team knows they have unlocked a viable, game-changing product feature ready to ship. [4:07] Ankur Goyal: Evaluations must be treated as the central engineering problem of AI application development, rather than an afterthought relegated behind complex agentic architectures. [4:07] Ankur Goyal: Custom scorers represent the modern evolution of product management. Instead of authoring static Product Requirement Documents (PRDs) and handing them over the fence to engineers, product managers should define scoring logic directly. [6:32] Ankur Goyal: A practical way to conceptualize context engineering is as managing an LLM's 'token diet.' [6:32] Ankur Goyal: The vast majority of tokens consumed during an execution originate from tool definitions, database retrievals, and tool execution outputs. [6:32] Ankur Goyal: In practice, both tool definitions and the data payloads returned by tools must be prompt-engineered and tailored specifically to the agent's exact operational task. [8:15] Ankur Goyal: Engineering cycles should not be squandered building brittle, over-engineered agent frameworks, state graphs, or complex control flows that teams become emotionally attached to. [8:15] Ankur Goyal: When teams focus on constructing comprehensive evaluation harnesses—comprising the data pipeline, custom scorers, feedback capture, and orchestration—they create a resilient artifact. [11:29] Ankur Goyal: Because the team had maintained their aspirational eval suite for a year, they recognized the capability shift instantly, threw away previous prototype code, and shipped Loop within two weeks as a simple while loop augmented with tool calls.

Original transcript

[0:04] Yeah, we're going to talk about Evals. [0:06] Uh before we do that, uh let's talk a [0:08] little bit about Brain Trust. Um we're [0:10] about a 2-year-old company and we work [0:12] with some of um the most advanced [0:15] companies in the world that build AI [0:17] software. Um and we help them solve uh [0:20] really really hard problems. I I heard a [0:22] little bit of the panel uh talk about [0:23] how hard eval um it's a really tough [0:26] problem. um we help folks solve hard [0:29] problems like evals, um observability, [0:32] bringing technical and non-technical [0:34] users together to solve these problems [0:36] and so on. Um and it's been a lot of [0:38] fun. I think the most interesting thing [0:40] I've seen is just once you enable people [0:43] to run evals, how many evals they'll [0:44] actually run and how quickly they're [0:46] able to build really good products. Um [0:49] now along the way, uh we've learned a [0:51] lot of really challenging things too. [0:52] And so I'm going to talk about five uh [0:54] sort of themes uh that we've picked up [0:57] over the past couple years. [0:59] Um the first is uh when you have evals [1:02] working in your organization um it [1:05] actually becomes pretty clear. Um and [1:07] and I would guess uh uh probably some of [1:10] you have figured this out u most of you [1:12] may be early in the eval journey or [1:13] early about thinking um about this. So [1:15] hopefully this can be helpful as kind of [1:17] a reference point of what's really a [1:19] great place to get to. Um the first [1:22] thing is that if a new model comes out, [1:25] you should be prepared to um launch [1:28] support for it within about 24 hours. [1:30] Now you may decide you don't want to do [1:31] that. Um however, uh you should be able [1:34] to confidently assess that within 24 [1:37] hours. And I like this test because [1:39] nowadays new models are coming out quite [1:40] often. And I think uh for a variety of [1:43] reasons, it's very important to be on [1:45] the cutting edge of model capabilities. [1:47] And if you have really good eval, which [1:49] we'll talk a little bit more about in a [1:51] second, um then uh you'll know because [1:54] your team will say, "Okay, you know, the [1:55] new model came out. Here's what our eval [1:57] say. Here are some things we need to dig [1:59] into further." And you can kind of work [2:00] all of that out really quickly. [2:04] Um, another really good sign of having [2:07] good evals is um, if a user complains [2:10] about something within, let's say, two [2:13] clicks, you're able to capture that [2:15] complaint and save it in a data set so [2:18] that you can email it and you you'll [2:20] never forget about that complaint again. [2:22] I think probably the most challenging [2:24] problem in establishing good eval is [2:26] building a very good feedback loop from [2:29] what's happening in production to what [2:31] developers um and product managers are [2:33] testing dayto-day. And I think if it's [2:35] not frictionless to capture people's [2:37] feedback and save it, then it's very [2:39] difficult to to get to good eval. [2:44] And the last sign of really good evals [2:46] is if you use evals to play offense [2:48] rather than defense. Um, when I say [2:51] that, uh, defensive evals are like unit [2:54] tests. So, let's say, um, Charles, uh, [2:57] changes his prompts and he wants to [2:59] figure out, um, did I break anything? [3:01] Okay, I changed my prompts. I run my [3:03] evals. 75 of them are green, three of [3:05] them are red. It's kind of like a unit [3:06] test. That's great. You should do it. [3:08] However, uh, the best organizations are [3:11] the ones that have EVELs sitting in, uh, [3:13] their org that are constantly broken. Um [3:17] and if they were to get those evals to [3:19] not be broken, they would be able to [3:21] ship products that will change the [3:23] world. Um you saw some of our customers [3:26] uh some of them have built products that [3:28] have started changing the world. I I'm [3:30] telling you from experience, this is how [3:31] they operate. So um in addition to these [3:34] unit tests, you have these aspirational [3:36] evals and aspirational eval represent [3:39] use cases that would be completely [3:41] mind-blowing if you solve them. And as [3:44] new models come out or um someone has a [3:47] really great idea about how to uh maybe [3:49] uh tweak the prompt or use um you know a [3:52] new vector database or whatever it is, [3:54] you can rerun these evals and um when [3:57] they start to work, you know you can [3:58] ship something. I'm actually going to [3:59] talk about our journey with that in in a [4:01] little bit here uh for building our own [4:03] AI products. [4:07] Um a second rule about building great [4:09] evals is that you need to think of them [4:11] as an engineering problem. Um evals are [4:13] not an afterthought. I actually think [4:15] the best teams think about evals as the [4:17] engineering problem and um you know [4:20] whatever your agentic architecture is or [4:22] whatever is a secondary thing. Uh but no [4:24] matter how you think about it, you need [4:26] to think about engineering evals uh very [4:28] intentionally. Um so let's think about [4:31] what that means. Uh the first part is [4:34] building a really good data set. And [4:35] again, a data set is not something that [4:37] gets handed to you for free. It's also [4:39] not something that you can pay someone [4:41] and just commission and have a really [4:42] good data set that will forever tell you [4:44] whether your application will work or [4:46] not. It needs to be something that you [4:47] engineer. Um and I think again a sign of [4:50] a good um eval culture is if your uh [4:54] engineering and product teams believe [4:56] that uh their eval represent reality and [4:59] let's say you're an executive you try [5:01] using the product you say hey um I'm [5:03] hearing this from our customers or I'm [5:05] playing around with it myself and it's [5:07] not really working you should be able to [5:09] reconcile what you're observing or what [5:10] your customers are observing with the [5:12] data sets that your teams are using to [5:14] do evals. [5:17] Um, the other half of the equation is [5:19] scorers. Um, and again, I think a lot of [5:22] people jump to uh using off-the-shelf [5:25] scoring methods, um, copy pasting [5:27] prompts that seem to work. Honestly, [5:30] Brain Trust has a bunch of these things [5:31] bundled in as well. Um, but when I talk [5:33] to people about it, I always encourage [5:35] folks to think about creating their own [5:37] scores. You don't have to do this um, on [5:39] day one. In fact, I think it's good to [5:41] use an existing scorer and then iterate [5:43] it. Um, but unless you build your own [5:45] scores, it's very hard to achieve really [5:47] great quality. Um, and I actually uh [5:50] think that scores are kind of like um [5:52] the next generation form of product [5:55] management. So instead of writing a PRD [5:58] and then handing it to the engineering [5:59] team and saying, "Hey, here's a spec for [6:01] what you should build." Um, you should [6:03] actually encourage product managers to [6:05] think about uh defining scoring logic. [6:07] Um, not every product manager can write [6:10] code, but cursor can help them do that. [6:13] Um, and every product manager can write [6:15] prompts and prompts are really good uh [6:17] for scoring. And so I think um instead [6:20] of spending a lot of time writing PRDS [6:22] uh you might as well just write scorers [6:24] um and then use that as the definition [6:26] of you know the spec of what you're [6:28] actually trying to build in your [6:29] project. [6:32] Um I you know when I first gave this [6:34] talk context engineering was a niche [6:36] term. I think it's recently exploded on [6:38] LinkedIn and X. So actually how many [6:40] people have heard of this term? Yeah. [6:42] So, wow. Okay. I think when I first gave [6:44] it, maybe like one super nerd raised [6:47] their hand. Um uh although I think [6:50] there's probably a lot of super nerds [6:51] here. Anyway, um so context engineering [6:54] is really important and and I have a [6:56] very simple way of thinking about it. [6:57] It's kind of like a diet. So, um, you [7:00] know, the the tokens that you handw [7:02] write in your system prompt and even the [7:04] tokens that your users uh contribute to [7:06] to an LLM interaction have actually [7:09] become a severe minority of the tokens [7:11] that an LLM is crunching on when it's [7:13] trying to help you solve a problem. And [7:16] so, you need to think as deliberately [7:18] about uh the tokens that are getting [7:21] spit out by your tool uh calls as you do [7:24] um your system prompt and user [7:26] definitions. Um, if anyone wants to talk [7:28] about MCP and all that stuff later, I'm [7:30] I'm around and I'm happy to talk about [7:32] it. But I think one of the uh core [7:35] challenges with MCP or maybe one of the [7:37] fallacies and how people uh use it today [7:40] is that um you sort of assume that a [7:43] generically defined highly general ter [7:45] uh tool will work on every agent. But [7:48] the reality is that you actually need to [7:50] prompt engineer tool definitions and [7:53] prompt engineer the output of tools to [7:55] be quite specific to the use cases uh [7:57] that your agent is solving. And so [7:59] whether or not you use MCP, I think it's [8:00] irrelevant. I just think it's very [8:02] important to think carefully about the [8:05] diet of tokens that your LLM is uh [8:08] crunching on and to think very [8:10] specifically about tool definitions and [8:12] tool outputs. [8:15] Um so this uh uh topic is kind of about [8:20] where and why you should spend real [8:22] engineering cycles. Um you should not [8:25] spend real engineering cycles on [8:27] building an agent. Um yes you should be [8:30] building an agent but you should not be [8:32] proud of the characters that someone is [8:34] typing and the you know the for loops [8:36] and graphs and all that stuff that [8:37] you're putting together because a new [8:39] model might come out tomorrow um and [8:41] that might just change everything. Um, [8:44] and so it's not I it's not a great idea [8:47] to build uh agent implementations that [8:50] are guaranteed to last. Um, and I'll [8:53] talk about that a little bit more, but [8:55] um, models can literally change [8:57] everything. One of our customers is one [8:59] of the most popular codegen companies [9:01] out there. Um, and every time a new [9:03] model has come out, the number of lines [9:05] of code of their codegen thing has [9:07] actually decreased. Um, and they went [9:09] from using a framework to using a while [9:11] loop. uh and I think that is really [9:14] really important. the CTO there had zero [9:17] attachment to uh the stuff that the team [9:20] had built and he encouraged the team [9:22] early on um to say hey a new model might [9:24] come out we might have to throw away [9:26] literally everything right away and I [9:28] would encourage you to think about that [9:30] now I I personally think that building [9:32] great data sets and building great [9:33] scores are a more durable investment [9:36] because a new model comes out you [9:38] completely throw away some agentic [9:40] whatever or try something new you [9:42] actually still want to be able to test [9:44] it with the data sets and scores that [9:46] you've created to help you get to that [9:47] point. Of course, you want to keep [9:49] iterating those two, but if you're going [9:51] to spend engineering effort, I would [9:53] spend engineering effort on building a [9:55] great feedback loop and infrastructure [9:56] around that and great uh scoring logic. [10:02] Um, so the last thing I think this is [10:04] kind of um a summary theme uh for all of [10:07] this is when you're building an AI [10:09] system, you're not just building an [10:11] agent or a for loop or a chatbot or [10:14] whatever. You need to think about as [10:16] building an evaluatable system. And so [10:18] the the box that you're building is an [10:20] evaluatable box. It's not the agent or [10:23] the chatbot or whatever in the center. [10:25] It's the whole thing. And if you think [10:26] about building the whole thing um then [10:28] you uh have an artifact that can [10:31] continue to improve and survive as uh [10:34] the underlying model or uh technical [10:36] trends evolve um underneath you. This is [10:40] a really simple example. Um we have a an [10:44] optimizer built into brain trust which [10:46] I'm going to talk about in a second. [10:47] It's called loop. Um and it's actually a [10:49] summary of all of these themes in many [10:51] ways. But we measured loop um uh which [10:55] can take a an optimizable or an agent or [10:58] an optimi an evaluatable agent and [11:00] optimize it. And we gave it just the [11:02] prompt uh like the system prompt. And [11:04] then we gave it the entire evaluatable [11:06] system including the scoring functions [11:08] and let it optimize that. And hopefully [11:10] to no surprise um if you give uh an LLM [11:14] the entire system to optimize it will do [11:17] much better because it just has more [11:18] information. So I think it's really [11:20] really really important to think about [11:22] the thing that you're building as an [11:24] entire evaluatable system. [11:29] Speaking of which, uh we just released a [11:31] system in Brain Trust called Loop. Um we [11:33] had an eval for about a year. Um, and I [11:36] ran it every month as a new model came [11:38] out and was very sad because um, up [11:42] until and including GPT40, [11:44] uh, LLMs were very very bad at [11:47] automatically improving prompts, uh, uh, [11:50] looking at data sets to help them do [11:52] that, write scoring functions and so on. [11:54] Um, and when Claude 4 came out, uh, the [11:58] numbers were incredibly compelling all [12:00] of a sudden. Um, and so we threw away a [12:02] bunch of code, uh, and we built Loop, [12:05] which is now built into Brain Trust, um, [12:07] and we shipped it in about two weeks, [12:08] which was really fun and exciting. Um, [12:10] Loop lives within Brain Trust, and it [12:12] can help you automate the entire eval [12:14] process, um, using LLMs, and we've seen [12:18] that it works especially well with Cloud [12:19] 4. Um, it works, uh, okay, with GPT41. [12:23] The jury is still out on GPT5. Um, and [12:26] Gemini is very exciting and it tries [12:28] super hard, but it uh, you know, for [12:31] some reason it can't really call tools [12:32] very well and so it just doesn't make [12:34] it. Um, I'm hoping that'll get fixed [12:36] soon though. Um, anyway, uh, loop it [12:39] really is a summary of all of these [12:40] themes. Um, it is literally implemented [12:42] as a while loop with tool calls. It's [12:45] very simple. Um, we uh have a bunch of [12:48] evals and loop that work and those are [12:50] exposed uh, very nicely in the product. [12:52] We have uh I think like eight or so [12:54] evals now of use cases that we want to [12:57] expose for loop that don't work yet. Um [13:00] and we're monitoring and trying to [13:01] improve those every day. Um and I would [13:05] encourage you to check it out. I think a [13:06] lot of our customers have started using [13:08] it as kind of like a reference [13:09] implementation of what an agent can be [13:12] like in a product. Um and of course they [13:14] also use it to improve their evals. [13:18] Um so that's it. Uh these are five [13:20] lessons from doing a lot of evals uh [13:23] over the last several years.