Snorkel AI Commits $3M as Agent Benchmarks Fall Behind Capabilities

@Scale

Because AI model capabilities have outpaced measurement for the first time in computing history, evaluating autonomous agents requires replacing static benchmarks with interactive, multi-step environments.

AI-assisted summary of Observability: Role of Evals, Benchmarks & Data in Frontier AI | Alex Ratner from Snorkel AI — original source. Check the original for full context.

Source published: 2026-06-19

Key ideas

Frontier AI development has outpaced evaluation, inverting traditional machine learning into an empirical science where logging alone is useless without rigorous output measurement. (2:37)
Software benchmarks are rapidly shifting along three axes—environment complexity, autonomy horizons, and output complexity—moving past saturated metrics like HumanEval (95%) toward brutal tests like ProgramBench (3% accuracy). (9:35)
In agentic systems, defining the evaluation and training data specification essentially defines the majority of the product specification itself. (13:51)
Snorkel AI has committed approximately $3 million to an open benchmark grants program, supporting around 20 upcoming public benchmarks targeting agent failure modes. (19:05)

Detailed notes

## Background Published on YouTube by the @Scale channel on June 19, 2026, this keynote features Snorkel AI Co-Founder and CEO Alex Ratner addressing the @Scale Conference. The session focuses on overcoming the AI evaluation gap by establishing enduring benchmarks and complex environments to measure agentic models across environment complexity, autonomy horizons, and output complexity.

## TL;DR Alex Ratner argues that model development has rapidly outpaced our ability to measure frontier AI, shifting computer science into an empirical discipline where static, saturated benchmarks like HumanEval fail to provide meaningful signal. To achieve genuine agent observability, evaluation must expand across three compounding vectors: dynamic environment complexity, extended autonomy horizons, and verifiable output complexity. Achieving this requires fundamentally redefining evaluation data from static text prompts into complete, interactive operating environments equipped with rich verifiers and feedback harnesses. Highlighting emerging evaluations like TerminalBench, SlopCodeBench, and a $3 million Snorkel-backed open grant program, Ratner outlines an engineering roadmap for diagnosing and training reliable autonomous agents.

## Atmosphere & style The presentation carries the brisk, authoritative cadence of an experienced systems researcher turned founder, blending academic rigor with pragmatic industry insight. Ratner speaks with earnest clarity and intellectual urgency, using touches of Silicon Valley wit alongside concrete software engineering realities to demystify agent hype. It feels like an insider briefing on frontier AI infrastructure, transitioning seamlessly from theoretical shifts to hands-on pipeline blueprints and an open call to action for the broader research ecosystem.

## Bulletin - [02:37](https://www.youtube.com/watch?v=iX79X61Jonk&t=157s) Frontier AI development has outpaced evaluation, inverting traditional machine learning into an empirical science where logging alone is useless without rigorous output measurement. - [09:35](https://www.youtube.com/watch?v=iX79X61Jonk&t=575s) Software benchmarks are rapidly shifting along three axes—environment complexity, autonomy horizons, and output complexity—moving past saturated metrics like HumanEval (95%) toward brutal tests like ProgramBench (3% accuracy). - [13:51](https://www.youtube.com/watch?v=iX79X61Jonk&t=831s) In agentic systems, defining the evaluation and training data specification essentially defines the majority of the product specification itself. - [19:05](https://www.youtube.com/watch?v=iX79X61Jonk&t=1145s) Snorkel AI has committed approximately $3 million to an open benchmark grants program, supporting around 20 upcoming public benchmarks targeting agent failure modes.

## Key takeaways - [00:05](https://www.youtube.com/watch?v=iX79X61Jonk&t=5s) The evaluation gap is the primary bottleneck in generative AI, as the ability to develop models has outpaced the ability to reliably measure them. - [00:05](https://www.youtube.com/watch?v=iX79X61Jonk&t=5s) Observability requires more than systems logging and tracing; infrastructure telemetry is ineffective without reliable scoring, red teaming, and benchmark rubrics. - [02:37](https://www.youtube.com/watch?v=iX79X61Jonk&t=157s) AI has shifted from theoretical engineering with clean guarantees into an empirical science driven by diagnostic measurement and experimentation. - [05:16](https://www.youtube.com/watch?v=iX79X61Jonk&t=316s) Next-generation benchmarks must evaluate along three critical dimensions: environment complexity, autonomy horizon, and output verifiability. - [09:35](https://www.youtube.com/watch?v=iX79X61Jonk&t=575s) Early coding benchmarks like HumanEval have saturated at around 95%, while new suites like ProgramBench expose frontier models down to a 3% accuracy rate. - [09:35](https://www.youtube.com/watch?v=iX79X61Jonk&t=575s) The compounding surface area of expanding task environments and extended sequence lengths outweighs gains in model sample efficiency. - [12:03](https://www.youtube.com/watch?v=iX79X61Jonk&t=723s) Data in the agentic era must shift from static chatbot prompts to fully interactive environments with file trees, live tools, and APIs. - [12:03](https://www.youtube.com/watch?v=iX79X61Jonk&t=723s) Verifiable deliverables and rich execution harnesses serve a dual purpose as evaluation benchmarks and dense reward signals for reinforcement learning. - [13:51](https://www.youtube.com/watch?v=iX79X61Jonk&t=831s) In agentic product development, the data specification functions as the majority of the actual product specification. - [16:33](https://www.youtube.com/watch?v=iX79X61Jonk&t=993s) Effective benchmark design requires purposefully targeting the specific failure modes and edge cases of frontier agents rather than broad success metrics. - [16:33](https://www.youtube.com/watch?v=iX79X61Jonk&t=993s) SlopCodeBench tests an agent's ability to navigate mid-task specification drift without producing architectural cruft or degraded code. - [16:33](https://www.youtube.com/watch?v=iX79X61Jonk&t=993s) UC Berkeley's Continual Learning Bench measures whether autonomous agents can retain procedural insights and compound efficiency across sequential tasks. - [19:05](https://www.youtube.com/watch?v=iX79X61Jonk&t=1145s) Snorkel AI committed around $3 million to open benchmark grants to fund academic and open-source measurement environments.

## Detailed walkthrough ### [00:05](https://www.youtube.com/watch?v=iX79X61Jonk&t=5s) The AI Evaluation Gap and Agent Observability

Alex Ratner, co-founder and CEO of Snorkel AI, introduces the core mission of his organization: operating as a frontier lab dedicated to creating datasets and environments for evaluating, tuning, and training artificial intelligence models. The central thesis of the presentation is closing the "evaluation gap" in generative AI, which serves as one of the most critical bottlenecks in achieving genuine agent observability.

Ratner contextualizes the team's perspective with over a decade of research originating in academia. The work began at Stanford University alongside co-founder Professor Chris Ré, followed by Ratner's lab at the University of Washington (UW), generating more than 200 peer-reviewed publications focused on data-centric AI and benchmark creation.

Observing autonomous agents is fundamentally a massive systems and scientific challenge. While standard observability involves tracking telemetry, logging tool calls, and monitoring actions across increasingly broad attack surfaces, logging alone is insufficient without reliable evaluation frameworks. Agent traces and outputs must be evaluated through diverse methods: - Spot checks, red-teaming exercises, and user interviews - Formal benchmarks containing concrete tasks, golden reference answers, and automated scoring mechanisms - Rigorous data pipelines capable of verifying whether an agent acts reliably, safely, and correctly

```mermaid flowchart LR accTitle: Agent Observability to Evaluation Pipeline accDescr: High-level overview of telemetry capture flowing through measurement mechanisms to actionable benchmark evaluation. subgraph Capture["Telemetry & Systems"] A["Agent Actions & Tool Use"] --> B["Monitoring & Traces"] end subgraph Measurement["Evaluation Layer"] B --> C["Red Teaming & Spot Checks"] B --> D["Formal Benchmarks"] D --> E["Golden Answers & Rubrics"] end subgraph Outcome["Goal"] E --> F["Safe & Reliable Agent Deployment"] end ```

[02:37](https://www.youtube.com/watch?v=iX79X61Jonk&t=157s) The Flip from Theoretical AI to Empirical Science

Ratner highlights a historic turning point: for arguably the first time in the history of AI and computer science, the ability to develop models has outpaced the ability to measure them. In traditional machine learning, evaluating a classifier was straightforward—collecting 100 ground-truth examples to detect a cat versus a dog (or "hot dog, not hot dog", referencing the famous *Silicon Valley* joke) made grading a simple matter of computing test-set accuracy before optimizing the model.

Today, this dynamic has inverted. Model capabilities are advancing faster than researchers can build high-quality datasets to test them at the frontier. Standard benchmarks suffer from rapid saturation; model cards show scores of 80% to 90%+ in coding or computer use, yet these same agents break down in complex, real-world deployments. Infrastructure logging and trace collection become completely useless if practitioners lack the measurement tools to verify output quality on frontier tasks.

Consequently, AI has evolved from a theoretical engineering discipline governed by clean mathematical theory into an empirical science. Because frontier models and agents have grown far too complex for simple theoretical guarantees, progress now depends on empirical experimentation, reliable measurement, and diagnostic benchmarks that pinpoint exactly where systems succeed or fail.

[05:16](https://www.youtube.com/watch?v=iX79X61Jonk&t=316s) The Three Dimensions of Next-Generation Benchmarks

To address this evaluation gap, Ratner outlines three foundational axes that define the next generation of LLM and agent benchmarks:

```mermaid flowchart TD accTitle: Three Axes of Frontier Agent Benchmarks accDescr: The three key dimensions defining next-generation benchmarks: environment complexity, autonomy horizon, and output complexity. subgraph FrontierSpace["Frontier Evaluation Dimensions"] Env["1. Environment Complexity (Single prompt vs. Full enterprise repo/Slack)"] Horiz["2. Autonomy Horizon (One-shot answer vs. Long-running, dynamic steps)"] Out["3. Output Complexity (Multiple choice vs. Multi-week verified software artifact)"] end ```

1. **Environment Complexity (Input Richness):** Evaluating how dynamic and rich the operating environment is. While early coding tasks relied on isolated paragraph-length problem descriptions, modern software engineering agents must navigate miniature enterprise environments containing complex repositories, tool harnesses, execution environments, and dynamic organizational inputs (such as changing requirements from product managers via Slack).

2. **Autonomy Horizon (Sequence Length and Dynamism):** Measuring how long and independently an agent can operate across extended workflows. Tasks are expanding from single-turn code generation to replicating full application architectures—such as Meta-affiliated research on *ProgramBench*, which demands multi-step execution mirroring weeks of human effort. Real-world projects are never linear; agents must handle evolving constraints, mid-task goal changes, and dynamic error recovery.

3. **Output Complexity (Deliverable Verifiability):** Evaluating sophisticated, open-ended deliverables. Unlike a 101-level programming exercise that is easily validated with one or two unit tests, evaluating an entire generated software product requires comprehensive rubrics, sophisticated verifiers, and multi-layered testing harnesses. Verifiable output measurement is crucial not only for accurate evaluation, but also as the reward signal for reinforcement learning (RL) and post-training alignment.

Even in domains widely considered mature or saturated, such as software engineering, there remains massive, exponentially growing white space across these combined dimensions of sequence length and input/output complexity.

[09:35](https://www.youtube.com/watch?v=iX79X61Jonk&t=575s) The Evaluation Gap: From Saturated Coding Benchmarks to Staff-Level Architecture

The frontier of software engineering evaluations is shifting rapidly from toy problem-solving to architecting entire software systems, leaving a massive amount of unmeasured white space in agent capability. Foundational benchmarks like HumanEval have effectively saturated at roughly 95% performance, offering little remaining discriminative signal. In response, subsequent benchmarks were created to push the boundary: TerminalBench 2.0—which Snorkel AI helped fund and collaborate on—already sees performance hovering around 82%, while the upcoming TerminalBench 3.0 will test significantly harder challenges with an anticipated failure rate exceeding 80%. Other demanding evaluations, such as ProgramBench (developed by a team now partly at Meta), show frontier agents scraping by at a meager 3% accuracy rate. Snorkel is actively working on a new, unannounced benchmark designed to test staff-engineer-level system architecture challenges.

```mermaid flowchart LR accTitle: Evolution of Coding Benchmarks accDescr: Progression of benchmark difficulty from HumanEval to staff-level architecture tasks. subgraph Saturated["Early Benchmarks"] HE["HumanEval (~95% accuracy)"] end subgraph Intermediate["Terminal Agents"] TB2["TerminalBench 2.0 (~82% accuracy)"] TB3["TerminalBench 3.0 (>80% failure rate)"] end subgraph Frontier["Complex Systems"] PB["ProgramBench (~3% accuracy)"] ARCH["Staff-Level Systems (In Development)"] end HE --> TB2 TB2 --> TB3 TB3 --> PB PB --> ARCH ```

Even within well-trodden spaces like code generation, the challenge surface area is compounding along three distinct axes: extending autonomy horizons (longer sequence lengths), expanding input environment complexity, and increasing output sophistication. While training algorithms and foundation models continue to show improved sample efficiency, Ratner argues that the "double exponential" of expanding task complexity and operational surface area will overwhelmingly dominate any gains in model efficiency. To keep pace with this capability frontier, the AI community must produce increasingly realistic, high-fidelity synthetic and real-world evaluation environments rather than relying on static, short-horizon tests.

- **HumanEval:** Approximately saturated at 95% completion rate. - **TerminalBench 2.0:** Performance hovering around 82%. - **TerminalBench 3.0 (Upcoming):** Anticipated failure rate of 80%+. - **ProgramBench:** Currently yields only a 3% accuracy rate across leading models. - **The Core Driving Vector:** A tri-fold increase in autonomy horizons, environment complexity, and output verifiability.

---

[12:03](https://www.youtube.com/watch?v=iX79X61Jonk&t=723s) Redefining Data: The Shift from Chatbot Prompts to Agent Environments

Moving into the agentic era requires fundamentally rethinking what constitutes evaluation and training "data." In the initial chatbot wave of generative AI, evaluation data was straightforward: a static prompt or question paired with an expected text response, scored against reference answers or human preference labels. In the agentic regime, however, these "test questions" transform into complex, multi-modal deliverables. A single evaluation instance is no longer a text snippet; it is an end-to-end task—such as architecting and implementing a production software artifact within an interactive environment.

Consequently, the evaluation payload must provide an entire operating world rather than just an input string. This includes domain-specific tools, file trees, documents, simulated personas, and live databases. Grading whether the agent succeeded cannot rely on simple string matching or basic heuristics; it demands rich intermediate verifiers, execution test harnesses, and nuanced grading rubrics that provide granular feedback not just at task completion, but continuously across hundreds of autonomous steps. These rich environments serve a dual purpose: they act as observational benchmarks to measure frontier models and supply the dense reward signals required for reinforcement learning and post-training.

---

[13:51](https://www.youtube.com/watch?v=iX79X61Jonk&t=831s) The Lifecycle of Agentic Data and Environment Development

Building high-quality agentic evaluations requires an extensive, iterative engineering pipeline. It begins with rigorous scoping and calibration. Ratner highlights a core operating thesis: "the data spec is a majority of the product spec in the agentic world." Defining the tasks an agent should complete, the environments it must navigate, and the explicit standards for a high-quality output essentially defines the agentic product itself. Tasks and environments must then be co-designed in lockstep to mirror actual workflows.

```mermaid flowchart TD accTitle: Agentic Benchmark Development Pipeline accDescr: Step-by-step pipeline for building and calibrating agentic benchmarks. A["1. Scope & Calibrate Data Spec = Product Spec"] --> B["2. Build Environments & Tasks Toolsets, EHRs, Docs, APIs"] B --> C["3. Run Agents & Discover Failures Identify edge cases & calibrate difficulty"] C --> D["4. Construct Rubrics & Verifiers Step-level & outcome rewards"] D --> E["5. Rigorous Review & Versioning Governance, packaging & standard tooling"] ```

Practical examples illustrate the domain-specific demands of this environment construction: - **Legal Deep Research (with Harvey):** Building realistic environments equipped with legal citation databases, case research tools, and statutory search capabilities to evaluate multi-step legal reasoning. - **Clinical Diagnostics:** Supplying agents with simulated patient vignettes, electronic health record (EHR) feeds, lab test histories, and diagnostic exploration tools.

Once tasks and environments are stood up, developers must actively run frontier agents against them to identify unexpected failure modes, calibrate difficulty, and build verifiable rubrics. Every payload component undergoes thorough review to weed out sub-par questions or surface nuanced edge cases that necessitate updating the original data specification. Finally, the suite is versioned and packaged under strict data governance protocols so it can operate as a reliable, reproducible evaluation harness.

---

[16:33](https://www.youtube.com/watch?v=iX79X61Jonk&t=993s) Next-Generation Benchmarks: TerminalBench, SlopCodeBench, and Continual Learning

A critical principle of modern benchmark design is intentionally steering the task distribution toward the specific failure modes of frontier agents, rather than building broad tests that agents can easily ace. To catalyze research, Snorkel AI established an Open Benchmarks Grants program supporting projects that push these boundaries:

- **TerminalBench:** Provides autonomous agents with full terminal execution access. Rather than serving merely as a static scoreboard, it has functioned over the past year as an active guidepost for frontier labs to expose brittle execution paths and failure modes in terminal-based coding agents. - **SlopCodeBench:** Bearing an affectionate "PhD kind of grad school name" (humorously compared to the naming origins of Snorkel during Ratner's Stanford days), this benchmark measures how coding agents handle spec drift. In real software development, requirements shift mid-implementation; SlopCodeBench evaluates an agent's ability to refactor and accommodate changing specifications over long horizons without accumulating architectural "slop code," dead logic, and cruft. - **Continual Learning Bench:** Developed out of UC Berkeley, this suite evaluates an agent's capacity to learn across sequential tasks. Rather than treating every prompt as an isolated execution, it measures whether an agent retains procedural insights and compounds efficiency when moving sequentially from task one to task two to task three.

[19:05](https://www.youtube.com/watch?v=iX79X61Jonk&t=1145s) Snorkel's Benchmark Grants Program and Open Ecosystem Support

Alex Ratner emphasizes a core principle underlying AI progress: models cannot improve from one task to another unless the community can rigorously measure that progress with high-quality data and robust benchmarks. For those whose interest was piqued by the benchmark examples shared during the talk, and who have ideas for benchmarks they want to develop in the open, Snorkel AI has established a dedicated benchmark grants program. Snorkel has already committed roughly $3 million toward this initiative and plans to increase that funding amount in the near future.

Currently, approximately 20 different benchmarks developed under this initiative are slated for announcement in the coming months, after which Snorkel plans to onboard an entirely new batch of projects. Ratner underscores that the team is deeply excited to support a broader proliferation of creative, well-architected benchmarks that live out in the open—spearheaded by academic and open-source teams with direct backing and tooling support from Snorkel AI.

- **Grant Program Commitment:** Approximately $3 million committed to date, with plans to increase funding shortly. - **Pipeline:** Roughly 20 open benchmarks scheduled for public release in the coming months, followed by a new cohort onboarding. - **Target Collaborators:** Academic researchers and open-source developers building public, reproducible benchmark environments.

[19:56](https://www.youtube.com/watch?v=iX79X61Jonk&t=1196s) Closing the Evaluation Gap Across Agentic Frontiers

Concluding his presentation, Ratner reiterates the central thesis of the talk: closing the "evaluation gap" is not only critical for AI safety and utility, but it is also an intellectually rich challenge. The community has barely begun to traverse the full frontier of agent capabilities across the three vital dimensions: 1. **Environment complexity** (the dynamic, rich environments an agent must perceive and navigate) 2. **Autonomy horizon** (how far an agent can plan, execute, and course-correct independently) 3. **Output complexity** (how intricate, structured, and verifiable the final deliverables are)

Ultimately, Ratner stresses that reliable evaluation depends entirely on the underlying data and environments constructed to test these systems. Thanking the audience for their time, he invites further discussion and collaboration to build the next generation of frontier AI evaluations.

## Notable quotes > [02:37](https://www.youtube.com/watch?v=iX79X61Jonk&t=157s) For arguably the first time in the history of AI and computer science, our ability to develop has outpaced our ability to measure. — Alex Ratner

> [02:37](https://www.youtube.com/watch?v=iX79X61Jonk&t=157s) Logging is not observability if you can't evaluate whether the trace is good or bad or where it broke down. — Alex Ratner

> [02:37](https://www.youtube.com/watch?v=iX79X61Jonk&t=157s) AI has flipped from being a theoretical engineering discipline to being an empirical science. — Alex Ratner

> [13:51](https://www.youtube.com/watch?v=iX79X61Jonk&t=831s) The data spec is a majority of the product spec in the agentic world. — Alex Ratner

> [12:03](https://www.youtube.com/watch?v=iX79X61Jonk&t=723s) A test question or an evaluation question is no longer a prompt; it's an entire task environment. — Alex Ratner

> [09:35](https://www.youtube.com/watch?v=iX79X61Jonk&t=575s) The double exponential of task complexity and operational surface area will overwhelmingly dominate any gains in model efficiency. — Alex Ratner

> [00:05](https://www.youtube.com/watch?v=iX79X61Jonk&t=5s) Closing this evaluation gap is one of the most important problems facing the field. — Alex Ratner

> [19:05](https://www.youtube.com/watch?v=iX79X61Jonk&t=1145s) We cannot improve from one task to another unless we can rigorously measure that progress with high-quality data and robust benchmarks. — Alex Ratner

## Mentioned Alex Ratner, Chris Ré, Snorkel AI, Stanford University, University of Washington, Meta, Harvey, UC Berkeley, HumanEval, TerminalBench, TerminalBench 2.0, TerminalBench 3.0, ProgramBench, SlopCodeBench, Continual Learning Bench, @Scale Conference

## Video description The excitement around agentic AI is real — backed by quantitative progress on model cards and genuine leaps in capability. But our ability to measure AI has been outpaced by our ability to develop it, and closing this evaluation gap is one of the most important problems facing the field. More enduring benchmarks are needed to advance the next vectors of capability and chart the path to reliable agents.

In this talk, Snorkel AI Co-Founder and CEO Alex Ratner will share insights from major research and benchmark collaborations on agentic coding and continual learning, along with practical tips from working with global frontier labs and leading academics. He'll focus on three dimensions where today's models most often break down, and where the next generation of benchmarks will need to deliver real signal: environment complexity (how dynamic and rich the operating world is), autonomy horizon (how far an agent can act independently), and output complexity (how sophisticated and verifiable the deliverable is).

Want to learn more about the @Scale Conference? Visit our website here: atscaleconference.com

[Source](https://www.youtube.com/watch?v=iX79X61Jonk)

中文简报

Agent基准落后于模型能力，Snorkel AI投入300万美元设立资助计划在计算历史上AI模型能力首次超越评估手段之际，评估自主Agent需要用交互式、多步骤环境取代静态基准。 Alex Ratner认为，模型开发的速度已大幅超越衡量前沿AI的能力，使计算机科学转变为一门经验学科，而像HumanEval这类静态且已饱和的基准已无法提供有意义的信号。为实现真正的Agent可观测性，评估必须在三个复合维度上进行扩展：动态环境复杂度、更长时程的自主性，以及可验证的输出复杂度。要做到这一点，从根本上需要将评估数据从静态文本提示词重新定义为配备丰富验证器与反馈控制机制的完整交互式运行环境。通过重点介绍TerminalBench、SlopCodeBench等新兴评估标准，以及由Snorkel支持的300万美元开放资助计划，Ratner概述了一份用于诊断和训练可靠自主Agent的工程路线图。前沿AI的发展已超过评估手段，使传统机器学习反转为一门经验科学，在此背景下，若没有严格的输出测量，仅靠日志记录毫无用处。 (2:37) 软件基准正沿着环境复杂度、自主时程和输出复杂度这三个维度快速演进，逐渐淘汰如HumanEval（95%）这类已饱和的指标，转向如ProgramBench（准确率仅3%）等严苛测试。 (9:35) 在Agent系统中，定义评估和训练数据规范，本质上就定义了绝大部分产品本身的规范。 (13:51) Snorkel AI已承诺向一项开放基准资助计划投入约300万美元，以支持针对Agent失效模式的约20个即将推出的公开基准。 (19:05)

Reading summary

Alex Ratner, co-founder and CEO of Snorkel AI, argues that model development has outpaced the ability to measure artificial intelligence, making closing the "evaluation gap" through rich, interactive environments the primary bottleneck for autonomous agent observability and deployment. The Inversion from Theoretical AI to Empirical Science (2:37) Ratner asserts that for the first time in computer science, model capabilities have advanced faster than the datasets required to test them. Traditional machine learning relied on static test-set accuracy against ground-truth labels, but modern frontier models frequently achieve 80% to 90% or higher on standard coding and computer-use benchmarks while breaking down in real-world workflows. Because systems have grown too complex for simple theoretical guarantees, Ratner characterizes modern AI as an empirical science that depends on diagnostic benchmarks capable of locating exact points of failure. Three Dimensions Defining Next-Generation Benchmarks (5:16) To evaluate autonomous agents accurately, Ratner outlines three foundational axes: environment complexity, autonomy horizons, and output complexity. Environment complexity shifts tests from isolated text prompts to rich interactive spaces containing code repositories, tool harnesses, and dynamic inputs like changing requirements. Autonomy horizons measure how long an agent can operate independently over multi-step, non-linear workflows with shifting constraints. Saturation in Early Benchmarks Versus Systems Complexity (9:35) Early software benchmarks have reached practical ceilings; Ratner notes that HumanEval is saturated at roughly 95% accuracy, and TerminalBench 2.0 hovers around 82%. In response, researchers are designing significantly harder suites: TerminalBench 3.0 targets an expected failure rate exceeding 80%, while Meta-affiliated ProgramBench records roughly 3% accuracy across leading models. Snorkel is also developing an unannounced benchmark focused on staff-engineer-level architecture challenges. Ratner argues that the expanding operational surface area and task complexity represent a "double exponential" that dominates gains from foundation model sample efficiency. Data Lifecycle and Open Ecosystem Support (13:51) In an agentic regime, evaluation data transitions from single prompt-response pairs into interactive environments with toolsets, file structures, and simulated personas. Ratner states that "the data spec is a majority of the product spec in the agentic world," requiring a full pipeline of task scoping, agent trial runs, discovery of failure modes, and rubrics designed for step-level and outcome rewards. Concrete implementations include legal research environments built with Harvey and clinical diagnostic setups incorporating patient vignettes and electronic health record feeds. To expand open testing frameworks, Snorkel AI has committed approximately $3 million toward an Open Benchmarks Grants program, with plans to increase funding and release around 20 community-developed benchmarks, including SlopCodeBench for spec drift and UC Berkeley's Continual Learning Bench. Ratner notes that the community has only begun exploring the space across environment complexity, autonomy horizons, and verifiable outputs.

Detailed notes — zh

## Background 该视频由 YouTube 频道 @Scale 于 2026 年 6 月 19 日发布，内容来自 @Scale Conference 的演讲。根据页面简介，Snorkel AI 联合创始人兼 CEO Alex Ratner 探讨了智能体 AI（Agentic AI）时代日益严峻的“评测差距”，并结合与前沿实验室及学术界的合作洞察，剖析了下一代基准测试必须攻克的三大核心维度。 ## TL;DR Alex Ratner 指出，AI 发展史上首次出现了模型能力跃进速度远超人类测量能力的情况，导致诸如 HumanEval 等传统基准迅速饱和。为了建立真正的智能体可观测性，评测必须向三大维度拓展：更动态丰富的环境复杂度、更长序列与适应需求变更的自主视野，以及更难检验评判的输出复杂度。数据本身的定义也从简单的单轮问答演进为“任务-环境-验证体系”的多层级负载。为此，Snorkel AI 投入了 300 万美元的基准资助计划，携手学术界与开源社区共同构建涵盖 Terminal Bench 3.0、Program Bench 及持续学习等方向的全新评测生态。 ## Atmosphere & style 整场演讲呈现出一种严谨、沉稳且极具学术底蕴的工程师气质。Alex Ratner 条理清晰，不讲空泛的概念，而是像在大学研讨会上拆解课题一般，层层推导出当前 AI 评测脱节的深层矛盾。现场氛围专业而务实，既有指出行业盲点时的警醒与严肃，也充满了以经验科学方法和工程管线重新定义 AI 评测边界的笃定与热情。 ## Bulletin - [02:37](https://www.youtube.com/watch?v=iX79X61Jonk&t=157s) AI 发展史上首次出现模型开发速度全面超越评估衡量能力的现象，导致基准快速饱和（如 HumanEval 饱和度达 95%），无法真实反映复杂现实场景中的智能体表现。 - [04:32](https://www.youtube.com/watch?v=iX79X61Jonk&t=272s) 新一代智能体基准评测正从单一问答走向三大核心维度：高动态度的环境复杂度、长步数且具备适应力的自主视野，以及难以验证的高复杂度工程交付物。 - [12:03](https://www.youtube.com/watch?v=iX79X61Jonk&t=723s) 智能体时代的数据定义发生范式转移，测试数据已从静态的“提示词-回答”对演进为包含特定环境、长程任务、验证规则（Rubrics/Verifiers）的复合任务负载，成为微调和强化学习的关键信号源。 - [19:05](https://www.youtube.com/watch?v=iX79X61Jonk&t=1145s) Snorkel AI 设立了约 300 万美元的开源基准资助基金并计划继续扩大规模，目前已有约 20 个覆盖持续学习、代码重构等前沿方向的全新基准正在研发推进。 ## Key takeaways - [01:08](https://www.youtube.com/watch?v=iX79X61Jonk&t=68s) 底层系统遥测、日志记录与轨迹监控只是基础，缺乏高质量的数据驱动基准评测，智能体可观测性便失去了有效指导意义。 - [04:32](https://www.youtube.com/watch?v=iX79X61Jonk&t=272s) AI 正从依赖公式证明的理论科学转变为面对海量参数与复合黑盒系统的“经验科学”，依赖严格的实验与评测驱动。 - [04:32](https://www.youtube.com/watch?v=iX79X61Jonk&t=272s) 评测环境复杂度正从单行文本提示题演进为内嵌真实代码仓库、开发工具链、数据库及多角色协同的拟真业务环境。 - [06:36](https://www.youtube.com/watch?v=iX79X61Jonk&t=396s) 自主视野（Autonomy Horizon）衡量智能体在动态变化、需求漂移的长链路任务中独立稳定运行的步数与能力，已逐步覆盖相当于人类数周工作量的任务跨度。 - [06:36](https://www.youtube.com/watch?v=iX79X61Jonk&t=396s) 智能体交付物的复杂度使得简单单元测试或布尔判断失效，必须依赖细致的评价量表（Rubrics）与综合验证器（Verifiers）来支持评估与强化学习。 - [09:35](https://www.youtube.com/watch?v=iX79X61Jonk&t=575s) 代码赛道评测难度呈指数级攀升：Terminal Bench 3.0 设计失败率超 80%，Program Bench 准确率仅约 3%，更高级别的系统架构基准也已进入研发阶段。 - [09:35](https://www.youtube.com/watch?v=iX79X61Jonk&t=575s) 任务复杂度与应用表面的“双重指数级扩张”远超算法样本效率的微小提升，使构建前沿仿真环境和数据成为关键瓶颈。 - [13:51](https://www.youtube.com/watch?v=iX79X61Jonk&t=831s) 智能体时代的数据规格说明书（Data Spec）在很大程度上直接等同于产品规格说明书，决定了智能体能力范围与交付标准。 - [13:51](https://www.youtube.com/watch?v=iX79X61Jonk&t=831s) 构建专家级评测需要严密的六阶段工程管线：范围界定、任务环境协同构建、实际运行标定、验证器开发、边缘用例专家审核以及版本化治理。 - [16:33](https://www.youtube.com/watch?v=iX79X61Jonk&t=993s) 优秀的基准设计不仅在于单点题目，更在于其“任务混合分布”，必须精准映射智能体的典型失败模式，起到指路标作用。 - [16:33](https://www.youtube.com/watch?v=iX79X61Jonk&t=993s) 诸如 Slop Code Bench（测试需求变更重构）和 Continual Learning Bench（测试连续任务经验沉淀）展现了从单次任务完成度向适应力与“在实践中学习”元能力的评测演进。 - [19:05](https://www.youtube.com/watch?v=iX79X61Jonk&t=1145s) Snorkel AI 通过投入 300 万美元以上的资助计划支持学术界和开源团队，旨在通过开放式合作弥合当前的 AI 评测差距。 ## Detailed walkthrough ### [00:05](https://www.youtube.com/watch?v=iX79X61Jonk&t=5s) 从学术前沿到 Snorkel AI：数据与智能体可观测性 Alex Ratner 是 Snorkel AI 的联合创始人兼 CEO。Snorkel AI 致力于构建数据集与运行环境，用于评估、微调及训练前沿 AI 模型。在这场演讲中，Alex 聚焦于生成式 AI 领域日益严峻的“评测差距”（Evaluation Gap），并深入探讨为何缩小这一差距是提升智能体（Agent）可观测性最核心的挑战。在进入核心技术议题前，Alex 简要回顾了团队十余年的深厚学术积累。Snorkel 团队长期专注于利用技术手段构建、管理与开发高质量数据集及评测环境，以衡量并推动 AI 模型与智能体能力的进化。这项工作最早源自斯坦福大学（Stanford University），团队联合创始人 Chris Ré 目前仍是斯坦福的教授；Alex 本人也曾任教于华盛顿大学（University of Washington）并领导独立实验室，现仍保持关联学者（affiliate）身份。迄今为止，该团队已发表了超过 200 篇同行评审的学术论文。Alex 幽默地表示，如果有听众想深入钻研如何构建高质量数据以实现严谨的评测与智能体可观测性，完全可以顺着他们发表的学术成果这条“兔子洞”一路探寻到底。 - **团队学术渊源**：起源于斯坦福大学 Chris Ré 实验室与华盛顿大学，累计发表超过 200 篇同行评议论文。 - **业务核心定位**：专注于打造支撑 AI 模型及智能体评估、微调与训练的高质量数据体系与真实模拟环境。 --- ### [01:08](https://www.youtube.com/watch?v=iX79X61Jonk&t=68s) 从遥测监控到基准评测：智能体可观测性的核心 Alex 指出，要理解智能体的可观测性，必须走通从底层系统观测到上层基准评测（Benchmarks & Evals）的逻辑链条。观察并理解智能体的行为是一个极为庞大的系统工程与科学命题：最基础的环节在于底层的系统遥测（Telemetry）、运行监控与日志记录，尤其是在智能体所接触的行动范围、调用工具及生成输出越来越复杂的当下，全面记录其交互轨迹至关重要。然而，仅仅把智能体的执行轨迹（Traces）和输出记录下来远远不够。业界需要一整套工具链来分析这些日志，判断智能体是否做出了正确的决策。这类评估涵盖了抽样检查（Spot checks）、红队攻防（Red teaming）、用户访谈，乃至形式化评测（Formal evaluations）——即预设一系列关键任务、标准答案（Golden answers）以及衡量智能体表现的评分规则，这在行业中通常被称为基准测试（Benchmarks）。Alex 强调，建立高质量的数据驱动基准评测，是智能体可观测性中最丰富、最关键的命题。如果没有可靠的评测指标来明确智能体在何处有效、在何处失效，以及何时能保持安全稳定，那么再完善的底层监控也失去了指引意义。 ```mermaid flowchart LR accTitle: 智能体可观测性链路 accDescr: 展示从底层系统监控到高阶基准评测与验证的完整链条 subgraph S1["底层遥测与执行记录"] A["智能体行为与工具调用"] --> B["遥测监控与日志抓取"] end subgraph S2["轨迹分析与评估手段"] B --> C["抽查 / 红队测试 / 访谈"] B --> D["形式化基准评测 (Benchmarks)"] end subgraph S3["落地目标"] D --> E["验证可靠性与安全性"] end ``` --- ### [02:37](https://www.youtube.com/watch?v=iX79X61Jonk&t=157s) 评测差距：AI 发展速度史上首次超越衡量能力 Alex 提出了一个振聋发聩的观点：在 AI 乃至整个计算机科学的发展史上，"我们测量 AI 的能力，有史以来第一次被我们开发 AI 的能力甩在了身后"。他对比了早期的经典机器学习时代：彼时最典型的顶尖任务就是训练一个小模型来做二分类，比如区分猫和狗，或者像美剧《硅谷》（Silicon Valley）里那个经典桥段一样判断“是热狗还是非热狗”（Hot Dog / Not Hot Dog）。在那个阶段，人类可以轻而易举地挑出 100 张具备确定标注的标准答案样本，计算模型在测试集上答对了百分之几，评测闭环即告完成，大家可以一头扎进算法与模型架构的优化中。但今天，局面发生了惊人的反转：模型能力的跃进速度，已经远远超出了人类为前沿能力构建高质量评测数据集与基准测试的速度。这种脱节导致了严重的基准饱和（Benchmark Saturation）现象——模型技术报告（Model Cards）上可能自豪地宣称代码生成或计算机操作（Computer Use）的准确率已达 80% 甚至 90% 以上，但一旦部署到真实复杂场景中，智能体就会频繁碰壁甚至崩溃。Alex 警示道，如果无法在实际应用的最前沿衡量模型的真实表现，那么搭建再多的监控与日志基础设施也是徒劳，所有的可观测性都将沦为空谈。 --- ### [04:32](https://www.youtube.com/watch?v=iX79X61Jonk&t=272s) 新一代基准评测的三大维度：环境复杂度 Alex 强调，AI 正在经历一次深刻的范式转移：它从一门侧重构建理论模型与公式证明的理论科学与工程实践，转变成了一门必须面对超大规模 LLM 与复合智能体的“经验科学”（Empirical Science）。面对参数庞大、行为难以通过极简理论完全解释的黑盒系统，构建严谨的基准评测、开展经验科学实验是当前至关重要的任务。针对新一代基准评测的发展方向，Alex 提炼出了评估智能体与大模型的三个核心维度，其第一维度便是**环境复杂度（Environment Complexity / Input）**。以编码智能体（Coding Agent）为例，简单的输入环境可能只是一道算法挑战题，由一段提示文本构成；而当智能体向真正的“软件工程智能体”（Software Engineering Agent）迈进时，它所面对的环境将演变成一个微缩版公司或真实的代码仓库（Repo）。它必须面对纷繁的代码文件、开发工具、测试框架，甚至要应对多元的角色系统——比如处理产品经理（PM）在 Slack 上发来的消息以及来自协同方的多源输入，并经历漫长的操作链路来最终交付产物。从单行文本问题到高度拟真的复杂业务环境，输入复杂度的跃迁直接决定了评测信号的真实度。 --- ### [06:36](https://www.youtube.com/watch?v=iX79X61Jonk&t=396s) 自主视野与输出复杂度：长程动态任务与验证难题第二维度是**自主视野（Autonomy Horizon）**，即智能体独立完成任务所需的执行步骤长度与动态适应能力。过去基准往往只测试单选题或写几行独立代码；而前沿评测正迅速向长周期任务演进。Alex 提到了学术界同行及 Meta 合作者最新推出的 ProgramBench，该基准要求智能体从零复现一整套程序制品（Program artifact），涉及的大量交互步骤相当于人类工程师数周的工作量。更重要的是，真实工程从来不是笔直平坦的单行道，目标、约束与边界条件随时会发生变动。因此，“自主视野”衡量的是智能体在动态变化、目标漂移的复杂长程链路中，究竟能独立稳定地运行多久。第三维度则是**输出复杂度（Output Complexity）**。当输入环境日益复杂、自主决策链路大幅拉长后，智能体输出的成果也变得极其繁杂且难以裁决。初学者编程练习题可以通过一两个单元测试轻松完成判题，但当智能体花费相当于人类数周的精力交付一整套软件产品时，简单的布尔断言或单测完全无法判断其优劣。行业必须为此构建复杂的评价量表（Rubrics）、规范指南、综合验证器（Verifiers）与全套集成测试套件。这不仅关乎基准评测的严谨性，更是后续开展强化学习（RL）与模型微调的生命线。 - **自主视野（Autonomy Horizon）**：涵盖交互步数的长短（跨越数分钟到人类数周工作量）以及应对需求变更、环境变动的动态鲁棒性。 - **输出复杂度（Output Complexity）**：产出物从单一文本答案演变为复杂工程产物，依赖多维度量表与自动化验证器（Verifiers）进行判别。 --- ### [08:46](https://www.youtube.com/watch?v=iX79X61Jonk&t=526s) 软件工程评测的广阔空白：多维坐标系的演进为了具象化这三个维度，Alex 重新审视了代码与软件工程领域。虽然外界有时会觉得代码评测已经趋于饱和，但 Alex 指出，软件工程领域在评测维度上依然存在着指数级增长的“空白地带”（White space）。为了便于直观理解，Alex 将三维空间进行了降维压缩：横轴为智能体完成任务所需的大致步数与序列长度（即自主视野），纵轴则将环境输入复杂度与输出复杂度合并为统一指标。从早期的简单代码补全和基础单测，到如今长程、高交互且包含大量非结构化工程约束的综合任务，评测的坐标系正在向右上方全面延展。现有的评测体系只是触及了冰山一角，如何在这类高复杂度、长视野的严苛空间中建立可靠的衡量指标，正是下一代智能体可观测性必须解答的核心课题。 ### [09:35](https://www.youtube.com/watch?v=iX79X61Jonk&t=575s) 代码基准评测的快速饱和与下一代评估挑战在代码生成领域，评测任务正在经历从“解决简单算法题”到“构建与架构完整软件系统”的巨大跃迁。Alex Ratner 指出，这一领域目前存在大量的“未探索空白”（white space），现有的评测甚至尚未开始对其进行衡量。早期的代表性基准如 HumanEval 已经被主流模型基本刷爆，性能饱和度接近 95%。为了跟上智能体的发展，Snorkel AI 资助并合作推出了 Terminal Bench 2.0，目前模型的准确率也已被推到了大约 82%；而即将发布的 Terminal Bench 3.0 则大幅拔高了难度，预计智能体的失败率将达到 80% 以上。此外，由部分目前就职于 Meta 的团队推出的 Program Bench，模型准确率甚至仅有 3%。Snorkel 团队目前还在研发一个暂定名、旨在评估“主任/主管工程师（Staff Engineer）级别系统架构问题”的全新基准，其挑战性将极其严苛。 ```mermaid flowchart LR accTitle: 代码评测基准演进历程 accDescr: 展示从简单代码题目到系统架构评测中模型表现与难度的剧烈变化 subgraph Saturated["已接近饱和"] HE["HumanEval (~95% 准确率)"] TB2["Terminal Bench 2.0 (~82% 准确率)"] end subgraph Frontier["前沿与高难度基准"] TB3["Terminal Bench 3.0 (预计失败率 >80%)"] PB["Program Bench (~3% 准确率)"] Arch["架构级基准 (Staff Engineer 水平)"] end HE --> TB2 TB2 --> TB3 TB3 --> PB PB --> Arch ``` 哪怕在代码这样最为成熟的赛道中，评估难度的指数级增长依然十分显著。这种增长由三大驱动力主导：更长的序列与自主性时间跨度（Autonomy Horizon）、更复杂的输入环境，以及需要交付的高复杂度输出。虽然未来的模型架构和训练算法在样本效率（Sample Efficiency）上会持续提升，但 Ratner 强调，任务复杂度与应用表面的“双重指数级扩张”（double exponential）将从根本上压倒样本效率的微小提升。面对智能体能力的爆发式增长，构建能够跟上前沿步伐的真实数据与仿真环境，成为准确衡量并观测智能体的核心前提。 - **关键基准与当前准确率表现**： - **HumanEval**：~95% 饱和，基本失去区分度。 - **Terminal Bench 2.0**：~82% 准确率。 - **Terminal Bench 3.0**：即将发布，设计目标为使智能体面临 80%+ 的失败率。 - **Program Bench**：极高难度基准，目前智能体准确率仅约 3%。 - **Staff Engineer 级别架构基准**：正在开发中，聚焦复杂分布式系统设计与全系统工程权衡。 --- ### [12:03](https://www.youtube.com/watch?v=iX79X61Jonk&t=723s) 数据定义的范式转移：从“提示词-问答”到“任务-环境-验证体系” 进入专家级智能体时代后，对“数据（Data）”本身的理解正在发生颠覆性的重塑。在生成式 AI 的聊天机器人阶段，所谓的数据通常等同于“测试考题”，形式非常简单：一段 Prompt（提示词或提问）、模型给出的 Response（回答），以及人工或自动给出的偏好标签（Preference Label），评估标准仅仅是看回答是否与预期相符。但在专家级智能体体系中，数据不再是一道静态的题目，而演变为了一个多层级的任务负载： ```mermaid flowchart TD accTitle: 智能体评估数据的构成要素 accDescr: 展示专家级智能体评估数据中任务、环境、反馈验证与评测打分的组成结构 subgraph DataPayload["现代智能体评测数据体系 (Data Payload)"] T["复杂交付任务 (如：从零构建复杂软件产物)"] E["领域特定环境 (工具集、代码库、真实数据库、上下文)"] V["验证器与评分规则 (Rubrics, Verifiers, 中途过程反馈)"] end T --> E E --> V V --> Out["最终评估与持续强化学习 (RL) 信号"] ``` 在全新范式下，“考题”变成了一个复杂的端到端任务（例如“构建某个软件产物”）；它必须在一个内嵌了真实工具、长文档、数据库与用户角色的领域特定环境里运行；同时还需要配套极其复杂的评分细则（Rubrics）、自动验证器（Verifiers）以及范例输出。评测系统不仅要在智能体完成数百步操作后的最终状态给出判断，还必须在自主执行链路的过程中持续提供反馈。Ratner 强调，这些数据既是观测与评估智能体表现的尺子，也是后续用于微调（Fine-tuning）和强化学习（RL）的关键信号源。 --- ### [13:51](https://www.youtube.com/watch?v=iX79X61Jonk&t=831s) 构建专家级智能体评测与环境的核心管线构建高质量的智能体评测数据与仿真环境是一项极具工程挑战的管线工作。作为数据领域的从业者，Ratner 提出了一个鲜明观点：“在智能体时代，数据规格说明书（Data Spec）很大程度上就是产品规格说明书（Product Spec）的大头。”定义评估智能体的数据类型、它所必须完成的任务边界、它赖以生存的作业环境以及何为“优质输出”，直接决定了智能体产品的定义。 ```mermaid flowchart LR accTitle: 智能体数据与基准构建管线 accDescr: 涵盖范围校准、环境构建、运行标定、评分机制、审核到版本治理的完整流程 S1["1. 范围界定与校准 (制定 Data Spec)"] --> S2["2. 环境与任务构建 (工具/数据/上下文)"] S2 --> S3["3. 运行智能体标定 (摸底强弱与断点)"] S3 --> S4["4. 构建规则与验证器 (Rubrics & Verifiers)"] S4 --> S5["5. 逐项审核与边缘用例反馈"] S5 --> S6["6. 版本化与数据治理"] S5 -.->|"发现边缘用例反哺"| S1 ``` 开发高质量评估体系需要经历六个关键阶段： 1. **范围界定与校准（Scoping and Calibrating）**：明确智能体能力的上下限与任务边界，这是一个持续迭代的循环过程。 2. **环境与任务同步构建**：环境与任务必须协同设计。例如与 Harvey 合作开源的法律深度调研（Deep Research）评估数据，必须包含法律引文检索、文书数据库等真实工具；若评估临床医疗智能体，则必须配备包含患者病历摘要（Vignette）、电子病历（EHR）和实验室检查指标的虚拟诊疗环境。 3. **运行并标定（Agent Calibration）**：在实际环境中运行前沿模型，摸清智能体的表现盲区与失效断点。 4. **制定评分细则与验证器**：构建能对整个执行过程与复杂产物进行判别、反馈的自动化检验程序和打分标准。 5. **严密的人工与专家审核**：逐一审查数据包的每一个构件。审核不仅为了排除缺陷，更是为了捕获长尾边缘情况（Edge Cases），进而反哺并修改最上游的数据规范。 6. **版本化与治理封装**：将完成的基准进行严格版本控制与合规治理，包装为可复现的标准评测工具。 --- ### [16:33](https://www.youtube.com/watch?v=iX79X61Jonk&t=993s) 前沿基准探索：Terminal Bench、Slop Code Bench 与持续学习 Ratner 介绍了 Snorkel 通过其“开源基准资助计划”（Open Benchmarks Grants Program）支持的几个代表性项目。优秀的基准设计绝不仅仅在于环境与任务本身，更在于其“任务混合分布（Mixture Distribution）”。一个所有模型都能轻松拿到 95% 的基准是毫无价值的，而只针对单一错误模式或单一领域的基准同样意义有限。基准的核心价值在于作为一面镜子和一个指路标（Guidepost），必须精准覆盖当前智能体的各类典型失败模式（Failure Modes），以此引导行业针对性改进。 - **Terminal Bench**：为赋予终端全权限访问的智能体量身打造的高自由度基准，已成为过去一年推动代码智能体能力演进的重要航标，系统覆盖了各种复杂的智能体操作方式与崩溃场景。 - **Slop Code Bench**：由博士生团队发起、命名诙谐但切中要害的基准。它旨在测试“开发过程中需求规格发生变更（Spec Change）”时代码智能体的适应力。在真实软件工程中，开发从不是一条笔直的单行线，需求随时会变；该基准衡量智能体能否在需求变更时优雅重构代码，而不是堆砌大量劣质冗余代码（Slop Code & Cruft）。 - **Continual Learning Bench（持续学习基准）**：由加州大学伯克利分校（UC Berkeley）主导推出。该基准专门测试智能体在执行一连串任务时的能力演进——即智能体在从任务一演进到任务二、任务三的过程中，能否自主沉淀经验并持续自我提升。这代表了智能体性能维度的重要飞跃：不仅要看当下单次任务的完成度，更要评估其“在实践中学习”的元能力。 ### [19:05](https://www.youtube.com/watch?v=iX79X61Jonk&t=1145s) Snorkel 基准资助计划与弥合 AI 评测差距的未来 Alex Ratner 在演讲收尾时再次强调了一个核心原则：从一个任务迁移到另一个任务时，如果我们无法凭借高质量的数据和基准测试来进行精准衡量，技术和模型能力就根本无从谈起改进。为了真正推动这一领域的进展，Snorkel AI 设立了基准资助计划（Benchmark Grants Program）。针对那些对前述三个维度案例感兴趣、且有意愿在开源社区构建开放式基准测试的研究者，Snorkel AI 已经投入了约 300 万美元的资助基金，并计划在近期进一步扩大该资金规模。目前已有大约 20 个涵盖上述方向的全新基准测试正在推进中，预计将在接下来的几个月内陆续对外公布，随后还将启动新一批资助项目的招募。Snorkel AI 致力于全力支持由学术界和开源团队主导的、极具创意且设计精良的开源基准项目。 Alex 总结道，当下的“评测差距”（evaluation gap）不仅是一个亟待解决的瓶颈，更是一个充满探索空间的研究领域。在衡量 Agent 能力的过程中，整个行业在环境复杂度（environment complexity）、自主时间跨度（autonomy horizon）以及输出复杂度（output complexity）等核心维度上才刚刚起步。而要做好高质量评测，最根本的依托始终是构建这些评测时所投入的高质量数据与动态环境。 ```mermaid flowchart TD accTitle: Snorkel 基准生态与评测三大核心维度 accDescr: 展示 Snorkel 基准资助计划与衡量 Agent 能力的三大评测维度的关系 subgraph Fund["Snorkel 基准资助计划 (Benchmark Grants)"] A["已投入 300 万美元 / 即将追加"] --> B["支持开源与学术团队构建开放基准"] B --> C["首批约 20 个基准陆续发布 / 开启新批次"] end subgraph Axes["弥合评测差距的三大核心维度"] D["环境复杂度 (Environment Complexity)"] E["自主时间跨度 (Autonomy Horizon)"] F["输出复杂度 (Output Complexity)"] end C -->|"落地验证与数据驱动"| Axes ``` - **基准资助基金规模**：已承诺投入约 300 万美元，并计划在近期继续追加投入额度。 - **项目落地节奏**：已有约 20 个全新的基准项目将在未来数月内对外发布，后续会持续吸纳和资助新的项目批次。 - **合作生态定位**：重点扶持学术界与开源社区主导的开放式创新，由 Snorkel 提供资源与技术支持。 - **三大核心攻坚维度**：下一代可靠 Agent 的评测必须深挖输入环境复杂度、长程自主规划跨度，以及高难度输出的验证机制。 ## Notable quotes > [02:37](https://www.youtube.com/watch?v=iX79X61Jonk&t=157s) 我们测量 AI 的能力，有史以来第一次被我们开发 AI 的能力甩在了身后 — Alex Ratner > [13:51](https://www.youtube.com/watch?v=iX79X61Jonk&t=831s) 在智能体时代，数据规格说明书（Data Spec）很大程度上就是产品规格说明书（Product Spec）的大头。 — Alex Ratner ## Mentioned Alex Ratner, Chris Ré, Snorkel AI, Stanford University, University of Washington, Meta, Harvey, UC Berkeley, Silicon Valley, HumanEval, Terminal Bench, Terminal Bench 2.0, Terminal Bench 3.0, Program Bench, Slop Code Bench, Continual Learning Bench, Slack ## Video description The excitement around agentic AI is real — backed by quantitative progress on model cards and genuine leaps in capability. But our ability to measure AI has been outpaced by our ability to develop it, and closing this evaluation gap is one of the most important problems facing the field. More enduring benchmarks are needed to advance the next vectors of capability and chart the path to reliable agents. In this talk, Snorkel AI Co-Founder and CEO Alex Ratner will share insights from major research and benchmark collaborations on agentic coding and continual learning, along with practical tips from working with global frontier labs and leading academics. He'll focus on three dimensions where today's models most often break down, and where the next generation of benchmarks will need to deliver real signal: environment complexity (how dynamic and rich the operating world is), autonomy horizon (how far an agent can act independently), and output complexity (how sophisticated and verifiable the deliverable is). Want to learn more about the @Scale Conference? Visit our website here: atscaleconference.com [Source](https://www.youtube.com/watch?v=iX79X61Jonk)

Source quotes

[2:37] Alex Ratner: 我们测量 AI 的能力，有史以来第一次被我们开发 AI 的能力甩在了身后 [13:51] Alex Ratner: 在智能体时代，数据规格说明书（Data Spec）很大程度上就是产品规格说明书（Product Spec）的大头。

Source quotes

[2:37] Alex Ratner: For arguably the first time in the history of AI and computer science, our ability to develop has outpaced our ability to measure. [2:37] Alex Ratner: Logging is not observability if you can't evaluate whether the trace is good or bad or where it broke down. [2:37] Alex Ratner: AI has flipped from being a theoretical engineering discipline to being an empirical science. [13:51] Alex Ratner: The data spec is a majority of the product spec in the agentic world. [12:03] Alex Ratner: A test question or an evaluation question is no longer a prompt; it's an entire task environment. [9:35] Alex Ratner: The double exponential of task complexity and operational surface area will overwhelmingly dominate any gains in model efficiency. [0:05] Alex Ratner: Closing this evaluation gap is one of the most important problems facing the field. [19:05] Alex Ratner: We cannot improve from one task to another unless we can rigorously measure that progress with high-quality data and robust benchmarks.

Original transcript

[0:05] Hey everyone, I'm Alex. I'm one of the [0:06] co-founders and CEO at Snorkel. We're a [0:08] frontier lab for building data sets and [0:11] environments for evaluating and tuning [0:13] and training AI. And today I'm going to [0:15] be talking about closing the evaluation [0:18] gap in a generative AI and how this is [0:20] one of the most central kind of [0:22] challenges in actually improving [0:24] observability of agents. [0:27] So quick 2 seconds on our background as [0:29] a team. We've been working for over a [0:31] decade in [0:32] using technology to build and curate and [0:35] develop data sets and environments for [0:38] measuring and advancing AI models and [0:41] agents. First out of Stanford, one of [0:42] our co-founders Chris Ray is a professor [0:44] there. I had a lab at University of [0:46] Washington. I'm now affiliate status but [0:48] but still a big U Dub fan and through [0:51] over 200 peer-review publications. If [0:53] you're interested in going further down [0:55] the rabbit hole and getting into some of [0:57] the gory details of how you build good [0:58] data for good evaluations for for better [1:01] agent observability. [1:03] Lots of lots of weight routes to go down [1:05] there with the academic work. [1:08] Jumping into the meat though. [1:09] First things first is to you know, talk [1:12] about this first [1:13] logical step from agent observability to [1:17] benchmarks and evaluation. [1:20] Now observing agents is a very large [1:22] systems and and science problem that [1:25] includes you know, the basics of [1:27] actually tracking the telemetry of what [1:29] they do and and monitoring and logging [1:31] the increasingly complex surface area [1:34] of of actions they touch and tools they [1:36] take and outputs they produce. It [1:38] includes a wide range of tools to then [1:41] take in those agent traces or or outputs [1:45] that we pull in through monitoring and [1:46] logging and measure them to see if the [1:48] agent did the right thing or not. This [1:51] is everything from you know, spot checks [1:53] and red teaming and you know, user [1:56] interviews, [1:58] to more formal evaluations where we have [2:00] say [2:02] a set of tasks that we want the agent to [2:04] do and our think think are important [2:06] that the agent can accomplish and some [2:09] golden answers and ways to grade the [2:10] answers performance the agents [2:12] performance. [2:13] And we call those benchmarks. So today [2:15] I'm going to zoom in on that part and [2:17] I'm going to [2:19] you know, try to emphasize how rich and [2:21] critical that part of agentic [2:23] observability is which is building [2:25] better data to lead to better benchmarks [2:28] and better evaluation of whether agents [2:30] are working or not, where they're [2:32] working or not and where they're [2:33] performing reliably and and safely. [2:37] And before diving into the meat, I want [2:40] to know what you know, one more pitch [2:42] for why this is such an exciting problem [2:43] is that [2:44] our ability to measure AI has actually [2:47] been outpaced for arguably the first [2:49] time in the history of AI and and and [2:50] much of computer science by our ability [2:52] to develop it. In other words, if you [2:55] like me have been in AI for a while and [2:56] you remember when it was called machine [2:58] learning and when the state of the art, [3:00] you know, style problem was you know, [3:01] training a small model to detect a cat [3:04] or a dog or hot dog not hot dog if you [3:06] know that Silicon Valley TV show [3:08] episode, [3:10] you know, it was pretty trivial to come [3:11] up with a 100 ground truth examples of [3:13] this is a cat, this is a dog and and [3:16] then you know, it was pretty easy then [3:17] to measure your model of what percent [3:19] did it get right of the test questions [3:21] of the benchmark and then you're off to [3:23] the races of trying to build that model [3:25] that algorithm. [3:26] Today we've had this pretty astounding [3:28] flip where model capabilities are [3:30] advancing faster than we can build good [3:34] data sets and good benchmarks to [3:35] empirically evaluate them at the [3:38] frontier of their capabilities. And you [3:40] can see this showing up in some of the [3:41] saturation of benchmarks where [3:43] you know, you see them all the card and [3:45] you know, in the area you care about [3:46] like coding or computer use the agent is [3:49] you know, 80 90 plus percent but then in [3:51] the field in real-world settings, you [3:53] run into problems. [3:54] Our evaluations, our benchmarks, [3:57] are, you know, lagging behind the things [3:59] that we're starting to use agents for. [4:01] And this is one of the most critical [4:02] gaps to close because you can, you know, [4:04] do all the monitoring, all the logging, [4:06] all the kind of, you know, set up of [4:08] agentic observability infrastructure in [4:09] the world, but if you don't have to wait [4:11] and measure the resulting output to see [4:13] if the agent is performing or not at the [4:16] things you actually care about at the [4:17] frontier of of use cases and [4:19] capabilities you're using the agent for, [4:22] then all that observability is useless. [4:24] So, we're going to double-click now [4:25] into, you know, how that evaluation gap [4:28] gets closed with better benchmarks [4:30] driven by better data. [4:32] So, building benchmarks is really a [4:34] mixture of science and art, and in some [4:36] ways, I won't wax long on this today, [4:38] but it's really about this big shift of [4:41] AI from a somewhat theoretical science [4:44] and an engineering practice of trying to [4:46] build better models, trying to build [4:48] theory that explain them nicely, to an [4:50] empirical science where we have these [4:52] massive, [4:53] you know, LLMs and models and agents [4:56] that we've grown [4:57] that have gone far beyond the complexity [4:59] that can be captured with simple theory, [5:01] and we now need to figure out how to [5:02] measure them, how to measure where [5:03] they're working, where they're not [5:05] working, where they are failing, and how [5:07] we can think about fixing that. [5:09] And building those benchmarks, doing [5:10] that empirical science, is, you know, [5:13] one of the most important things these [5:15] days. [5:16] Before going into how data plays a role [5:18] in that and how you actually build good [5:20] benchmarks, [5:21] I want to give a little bit of a view, a [5:23] very high level, about, you know, what [5:25] we're excited about in terms of where [5:27] the next great benchmarks are going to [5:29] come. [5:30] And we often distill this down to three [5:31] axes for agents and LLMs. One is the [5:35] complexity of the environment or the [5:37] input to the agent. So, if we take the [5:39] example of a coding agent, you know, a [5:41] simple environment or simple input would [5:44] be just a, you know, a tricky coding [5:46] challenge, maybe a paragraph of text. As [5:49] we try to go from coding agents to [5:50] actual software engineering agents and [5:52] beyond, [5:54] an input and an environment that the [5:56] agent operates in might start to look [5:58] like [5:59] an entire miniature company and and code [6:01] base and repo with code files, tools, [6:04] harnesses, and even other personas and [6:07] systems like Slack messages from a PM [6:10] or, you know, input from another [6:12] constituent that the agent needs to take [6:14] into account through a long series of [6:16] actions to build the output. So, again, [6:19] simple environment and input of just [6:22] literally a question like on on you [6:24] know, on a simple coding test to complex [6:27] environments that represent something [6:29] that looks like the real-world settings [6:31] that a coding agent or a legal agent or [6:33] a medical agent would would operate in. [6:36] Axis number two is the autonomy horizon, [6:39] or really the the length and dynamism of [6:42] the steps the agent needs to go through [6:44] to complete the task. [6:46] So, again, you know, the simplest [6:48] example if we stick with coding would be [6:50] a simple multiple-choice question or a [6:52] simple kind of question that requires a [6:54] bit of code that's written. [6:56] Increasingly, we're moving towards, you [6:58] know, bigger and bigger, longer and [7:00] longer tasks. Program bench just came [7:02] out of some some friends of ours in the [7:03] research research space that are also at [7:05] Meta. [7:06] Uh asks agents to actually replicate an [7:09] entire program artifact. [7:12] And these things can take, you know, [7:14] many, many steps, the equivalent of a [7:16] project that might take a human, you [7:18] know, weeks to do. [7:19] Moreover, it's not just about the length [7:22] of the task, it's also about the fact [7:23] that those those steps [7:26] may have changing dynamics. You know, [7:29] most projects are not straight line. The [7:31] goals, the objectives, the constraints [7:32] change while you're actually doing [7:34] something. So, that is what we mean by [7:36] autonomy horizon of how independently [7:39] and dynamically and for what for for how [7:42] long can an agent successfully operate [7:44] as we go from simple kind of quick [7:46] answers to long running autonomous [7:48] tasks. [7:50] And the third axis is the output [7:52] complexity, meaning when you give it [7:54] that input environment that's [7:56] increasingly complex, it takes all of [7:59] these steps to actually, you know, do [8:01] increasingly long tasks. Now agents are [8:04] producing increasingly complex outputs [8:06] that are increasingly tricky to judge. [8:09] In other words, back to that coding [8:10] example, you know, a simple [8:12] multiple-choice question is trivial to [8:13] grade. A simple coding challenge, [8:16] you know, in a 101 course [8:18] can be tested with a, you know, unit [8:20] tester too. [8:21] But when you ask a coding agent to [8:22] actually go and spend, you know, the [8:25] human equivalent of of a couple weeks [8:27] building an entire software product, [8:29] you can't just trivially evaluate [8:31] whether the output was good or not. You [8:33] need to actually build complex rubrics [8:35] and guidelines and verifiers and unit [8:37] tests to figure out if what it did was [8:40] correct. [8:42] And that's critical both for evaluation [8:44] and for reinforcement learning and [8:45] tuning. [8:46] So three main axes where for everything [8:48] that our agents are doing, we think [8:50] there's so much room to go. [8:52] And the first thing that needs to happen [8:54] is we need to be able to measure these [8:55] increasingly complex [8:57] agentic settings. [8:59] So we'll share some examples in a bit. [9:02] But I want to just double click into [9:03] that coding example one more time. [9:06] And note that even in areas that [9:08] sometimes feel like they're saturated [9:09] like software engineering, and that's [9:11] why I picked this example, we think, you [9:12] know, still have exponentially growing [9:14] white space. If you look at coding [9:16] examples, and here on this axis we have [9:19] this is a, you know, condensed view of [9:20] that three-dimensional space I was [9:22] talking about. Here we're collapsing the [9:24] input and the output complexity into one [9:26] axis. And then there's the autonomy [9:28] horizon or the sequence length, the [9:29] number of steps, roughly, that an agent [9:31] would have to take to complete the task. [9:33] If you look at how this has progressed, [9:35] like I was talking about from, say, [9:37] simple coding problems to you know [9:39] building and architecting entire [9:40] software systems. We think there's a ton [9:43] of white space and we're just not even [9:45] measuring it yet, right? So we started [9:46] with tasks that were you know things [9:49] like human eval if you're familiar with [9:50] that benchmark that are you know [9:52] approximately saturated at 95% [9:53] performance. Terminal bench 2.0 came out [9:56] and we helped fund and and collaborate [9:57] that effort. It's now at about 82%. [10:00] Terminal bench 3.0 is upcoming and I [10:03] can't reveal any exact stats but likely [10:05] it'll be you know much much more [10:07] difficult maybe 80% plus failure rate [10:09] for agents. Program bench came out of [10:11] that team I mentioned that's partly at [10:13] meta now. Is at a 3% accuracy rate. Uh [10:16] we're working on something name to be [10:18] finalized but that is looking at you [10:20] know kind of staff engineer level system [10:22] architecture problems. That's going to [10:24] be quite difficult. [10:26] So basically even in the most saturated [10:28] areas like [10:30] like coding [10:31] you know you have this [10:32] exponential increase that's driven by [10:34] the increasing sequence length or [10:36] autonomy horizon, the increasing [10:38] complexity of the input environment and [10:41] the output that needs to be produced. [10:44] And while models and and and algorithms [10:46] to train models will get more sample [10:48] efficient we think that the kind of you [10:50] know [10:51] double exponential of this massively [10:53] expanding surface area and task [10:55] complexity is going to you know very [10:57] significantly dominate the the the [11:00] sample efficiency gains. So [11:02] what does that mean? Going back to the [11:04] focus of the talk there's going to be a [11:06] massively exploding area of things that [11:09] agents can do and that we want them to [11:10] do that we're going to need to measure [11:12] with tools like evaluations and [11:14] benchmarks. And the key to that is going [11:16] to be building realistic data and [11:18] environments that keeps pace with that [11:20] frontier. [11:23] So again lots of things get harder as [11:25] you advance in these axes. You've got to [11:28] build richer environments with tools, [11:31] documents, databases, personas, [11:34] everything that simulates a real-world [11:35] environment more and more. [11:37] You have to build tasks that require [11:39] increasingly long autonomy horizons that [11:42] an agent might, you know, go through [11:43] hundreds of steps to complete rather [11:45] than one or two. [11:46] And you need to be able to not just [11:48] build but grade the increasingly complex [11:51] outputs again that require nuanced [11:53] rubrics or grading keys to even assess [11:56] whether they're correct or not, whether [11:57] the agent was successful or not compared [12:00] to, you know, simpler tasks. [12:03] So, what does this look like? You know, [12:04] we think about this as this kind of big [12:06] leap forward from [12:08] you know, not just the task complexity [12:10] side of what are we asking the agents to [12:12] do, but also the data that powers that, [12:15] right? So, let's now double click into [12:17] the data. What What do we mean when we [12:18] say data? Um we're talking about, you [12:21] know, the data that goes into evaluation [12:22] or benchmarking. Think about data as in [12:25] this context as kind of like test [12:27] questions. In the, you know, initial [12:29] kind of chatbot stage of generative AI, [12:32] this might be, you know, a prompt or a [12:34] question and a response or an answer. [12:37] And maybe a preference label. [12:39] And we would measure and evaluate the [12:41] agent based on, you know, how well its [12:44] response matched the expected response [12:46] to one of those questions. [12:48] In this world of increasingly complex [12:51] input context and environments, [12:53] increasingly long horizons of autonomy, [12:55] increasingly complex outputs, [12:57] this gets a lot more complex. [13:00] Suddenly this this data you have to [13:01] build, these test questions, are not [13:04] just the like a question or a prompt, [13:06] they're a complex task, like [13:08] go build this software artifact. They're [13:11] a complex domain-specific environment [13:13] that the agent needs to operate within [13:15] in order to complete that task or that [13:17] test question. [13:18] And they're complicated rubrics and [13:20] verifiers and example outputs that give [13:22] feedback not just at the end state but [13:25] along the way of how the agent is doing [13:27] against this increasingly complex long [13:29] horizon task. [13:31] So, the same basic idea that data is [13:33] basically a bunch of test questions for [13:35] the AI agent that we can use to measure [13:38] its performance, observe its performance [13:40] in other words, and also tune it or do [13:42] reinforcement learning. [13:44] But, what those test questions look like [13:46] is kind of night and day as we head into [13:47] this expert agentic environment regime. [13:51] So, how do you build that? Well, you [13:53] know, there's a lot of work that goes [13:55] into this pipeline. I'll only skim the [13:56] surface, but just to give a sense of of [13:58] how hard it is to really build good [14:00] quality data and environments in this [14:02] expert agentic regime. [14:04] A lot goes in first of all to scoping [14:06] and calibrating. This is a point I'll [14:08] emphasize here. I'm a little biased as a [14:10] data person, but I would emphasize I [14:12] think most folks would agree to some to [14:14] some degree that the data spec is a [14:17] majority of the product spec in the [14:19] agentic world. In other words, defining [14:21] what types of data you want to evaluate [14:23] the agent on, what types of tasks should [14:26] it be able to complete, what [14:28] types of environments should it be able [14:29] to work in, what does a good output from [14:31] the agent look like is an increasing [14:33] percentage of the product spec for that [14:35] agent. So, scoping and calibrating [14:39] the the benchmark and really that means [14:41] scoping and calibrating the the data [14:44] is is an absolutely critical step and [14:46] it's an ongoing continuous one when you [14:48] are dealing and building these complex [14:50] benchmarks. Second, you have to build [14:53] the environments and the tasks usually [14:55] in tandem. [14:57] So, you know, let's jump examples. If [14:58] you want to build a a legal agent, we [15:00] just open sourced some examples of legal [15:03] deep research [15:04] evaluation data with Harvey, for [15:06] example. You need to, you know, build [15:09] the right environment of context, of [15:11] tools like, you know, legal citation [15:13] research, look up [15:15] that the agent is going to need to use [15:17] to accomplish a task like a deep [15:19] research task for, you know, answering a [15:21] complex legal question. [15:23] Or if you're [15:25] you know, asking [15:26] measuring a clinical agent on its [15:27] ability to diagnose a patient, you need [15:29] to give it the environment that might [15:30] include the patient's vignette, their [15:32] EHR data, their test data, etc. [15:36] So, building the environments and the [15:37] tasks is step one. [15:40] You then need to actually, you know, run [15:42] agents to see where the agents perform [15:44] and don't perform, to calibrate those [15:46] environments and tasks. You then need [15:48] to, you know, build the rubrics and the [15:50] verifiers that give feedback to the [15:52] agent that actually help evaluate [15:54] whether it was correct or not in in [15:57] executing the task in the environment. [15:59] There's a lot of work that goes into [16:00] reviewing every single component of this [16:03] payload. [16:04] Sometimes because it it wasn't built to [16:06] quality, sometimes because it surfaces [16:09] some [16:10] edge case that you want to go back and [16:11] use to edit the scope and the spec of [16:15] the data. [16:16] And then finally, um actually, you know, [16:19] package that up in a way that is [16:20] versioned, that is well governed, and [16:23] that is, you know, well set up to be [16:25] used as a standard evaluation tool. [16:28] So, I'm just going through that to give [16:29] a a quick glimpse of, you know, what [16:31] goes into today development. Now, I'll [16:33] just quickly flash up a couple of [16:35] examples before wrapping up. Some of the [16:37] benchmarks that we've gotten to help [16:39] collaborate with and support to our open [16:41] benchmarks grants program, which I'll [16:42] mention, include things like Terminal [16:44] Bench, which is an extremely exciting, [16:46] you know, very open domain benchmark [16:47] that includes tasks for an agent that [16:50] has full access to the terminal to [16:51] complete, and has actually not just [16:54] allowed us to measure, but actually been [16:55] a guide post for driving a lot of the [16:57] progress of coding agents over the last [16:59] year. [17:00] And covers all kinds of different, you [17:01] know, areas, types of agentic [17:04] operations, and specifically, failure [17:06] modes. [17:07] This is worth noting that, you know, a [17:09] big part of developing good data sets [17:12] for good benchmarks [17:13] is adapting the data distribution to the [17:16] failure modes of common agents. A [17:19] benchmark that every agent out there [17:20] aces and gets 95% on is not that [17:22] interesting. One that only focuses on [17:25] one specific error mode or domain is not [17:27] that interesting either. So, good [17:29] benchmark building is not just about [17:30] building the environment, the task, the [17:32] data components we talked about. It's [17:34] also about building the right mixture of [17:36] tasks that cover the right types of [17:38] domains and the right failure mode areas [17:41] of current agents. Because again, [17:43] benchmarks are not just about measuring. [17:45] They're about guiding towards improving [17:47] agents. [17:48] And so, you want that good mixture. So, [17:50] again, mixture distribution across [17:53] topics, domains, but also failure modes [17:57] of your agent or agents out there is [17:59] critical. [18:02] Another uh cool example, again, I'll go [18:03] through quickly cuz I think we're [18:04] running on time, but it's something [18:05] called Slop Code Bench. This is a great [18:07] PhD kind of grad school name. Never [18:10] meant to be too serious. You know, see [18:11] our our name from back in the Stanford [18:13] day, Snorkel. [18:15] But, this is a really, really cool [18:15] benchmark that's really about measuring [18:17] coding tasks where there's spec change [18:20] during the process of iterating on the [18:21] code. So, this gets to that autonomy [18:23] horizon of most tasks in the real world [18:26] are not straight line. The specs change [18:28] as you're working on a real world coding [18:30] task and measuring how [18:32] that's done without kind of slop code [18:34] and cruft being built up by coding [18:36] agents is a really, really interesting [18:38] new direction and benchmark that we're [18:40] proud to support. [18:42] And one last example, continual [18:43] learning. We were proud to support [18:45] continual learning bench that just [18:46] launched out of Berkeley. And this is [18:48] measuring the ability of agents to [18:51] improve when they go through sequences [18:53] of tasks where there's some learning to [18:55] be had, you know, from task one to two [18:57] to three. Which is another very [18:59] important element of agentic [19:01] performance. Again, not just their [19:02] ability to perform a task, but to learn [19:05] from one task to the other. That again, [19:08] we can't improve unless we can measure [19:10] it with good data and good benchmarks. [19:13] So, I'll wrap here, but I'll just [19:14] briefly note that if [19:16] any of this was interesting, and [19:17] especially if those last three examples [19:19] kind of piqued your interest, and and [19:21] you have ideas of, you know, benchmarks [19:23] that you'd be interested in building out [19:25] in the open, [19:26] we have a benchmark grants program. [19:28] We've committed about 3 million. We're [19:30] going to increase that [19:31] pretty soon. We have about 20 different [19:34] benchmarks like those examples I shared. [19:36] They're going to be going to be [19:36] announced in the coming months, and then [19:39] we're going to onboard a new batch. And [19:41] we're really, really excited to support [19:43] the proliferation of, you know, really [19:46] creative, [19:47] you know, well-designed benchmarks that [19:49] are, you know, out in the open, driven [19:51] by open-source and academic academic [19:53] teams [19:54] with Snorkel support. [19:56] So, with that, hopefully I convinced you [19:57] a little bit that, you know, the [19:59] evaluation gap is critical, it's [20:01] interesting, it's extremely rich with [20:03] all these different axes we have, you [20:06] know, so barely traversed in terms of [20:08] agent capabilities around input [20:10] environment complexity, autonomy [20:12] horizon, and output complexity, and [20:14] that, you know, one of the central [20:15] points of doing evaluation well is the [20:18] data and environments that go into [20:20] building these these evaluations. [20:23] With that, thank you for the time, and [20:25] I'm excited to follow up.