Meta Prevents 74 Production Incidents With Disposable AI Tests

@Scale

The limit of engineering velocity is verification confidence rather than code generation speed, forcing teams to replace permanent test suites with disposable tests filtered aggressively to protect developer trust.

AI-assisted summary of Ship Fast, Break Nothing: How AI Tests What AI Builds | Sophie Zeng & Sreeja Thummala from Meta — original source. Check the original for full context.

Source published: 2026-10-30

Key ideas

Meta designed an AI-native quality gate that deploys ephemeral, throwaway catch tests targeting risky diffs, passing on the parent commit and failing on the diff to isolate regressions without adding permanent test maintenance debt. (2:17)
Providing the test generator with full developer intent and failure hypotheses doubled the bug catch rate from 4.0% to 8.0%, a 40x improvement over diff-blind baselines. (6:53)
To prevent developer notification fatigue, an LLM judge evaluates test failures and discards the vast majority of candidate tests, driving an estimated 70% reduction in engineer triage time. (13:03)
Counterfactual baseline modeling revealed that Meta's system prevented 74 critical SEV incidents and lowered privacy incident growth roughly one-fifth below forecast projections. (17:39)

Detailed notes

## Background Published on YouTube by the @Scale channel on October 30, 2026, this presentation features Meta engineers Sophie Zeng and Sreeja Thummala discussing automated quality assurance. Although no official show notes were provided, the talk outlines how Meta uses AI to build ephemeral test suites capable of matching the velocity of modern AI-generated code.

## TL;DR As AI code generation accelerates while traditional test suites remain slow and backward-looking, Meta restructured its testing architecture around disposable, diff-aware 'catch tests' that pass on parent commits but fail on introducing diffs. Operating across a four-stage pipeline—Flag, Generate, Filter, and Check—the system triages hundreds of thousands of daily commits, using cheap heuristic risk-scoring before invoking expensive test synthesis. To protect developer trust from false alarms and intentional product shifts, an LLM judge aggressively discards over 90% of candidate failures, framing notifications as neutral inquiries rather than definitive bug alerts. This approach reduced engineer triage time by approximately 70% and demonstrably averted 74 critical SEV incidents while driving privacy incident growth roughly 20% below baseline projections.

## Atmosphere & style The presentation is candid, analytical, and grounded in the operational realities of hyperscale engineering, trading generic AI enthusiasm for hard-won architectural lessons. Sophie Zeng and Sreeja Thummala deliver a complementary tag-team talk that balances algorithmic problem-solving with the fragile realities of human behavior, memorably noting that developer trust is won in drops and lost in buckets. Listening feels like attending an insider post-mortem from production trenches, where the ultimate bottleneck in software velocity is revealed not to be code generation speed, but human confidence in automated verification.

## Bulletin - [02:17](https://www.youtube.com/watch?v=Pf10kLLcFR0&t=137s) Meta designed an AI-native quality gate that deploys ephemeral, throwaway catch tests targeting risky diffs, passing on the parent commit and failing on the diff to isolate regressions without adding permanent test maintenance debt. - [06:53](https://www.youtube.com/watch?v=Pf10kLLcFR0&t=413s) Providing the test generator with full developer intent and failure hypotheses doubled the bug catch rate from 4.0% to 8.0%, a 40x improvement over diff-blind baselines. - [13:03](https://www.youtube.com/watch?v=Pf10kLLcFR0&t=783s) To prevent developer notification fatigue, an LLM judge evaluates test failures and discards the vast majority of candidate tests, driving an estimated 70% reduction in engineer triage time. - [17:39](https://www.youtube.com/watch?v=Pf10kLLcFR0&t=1059s) Counterfactual baseline modeling revealed that Meta's system prevented 74 critical SEV incidents and lowered privacy incident growth roughly one-fifth below forecast projections.

## Key takeaways - [00:05](https://www.youtube.com/watch?v=Pf10kLLcFR0&t=5s) The traditional software testing compact has collapsed because code generation costs plummeted while regression verification remained slow, static, and backward-looking. - [00:05](https://www.youtube.com/watch?v=Pf10kLLcFR0&t=5s) A survey of 50,000 developers showed an 80% adoption rate of AI tools against an alarmingly low 29% trust rate in AI-generated code. - [02:17](https://www.youtube.com/watch?v=Pf10kLLcFR0&t=137s) Ephemeral catch tests invert traditional testing economics by generating disposable test harnesses that validate specific commits and are discarded immediately after execution. - [02:17](https://www.youtube.com/watch?v=Pf10kLLcFR0&t=137s) A valid catch test acts as a differential oracle by strictly passing on the parent commit and failing on the child diff to prove causality. - [03:42](https://www.youtube.com/watch?v=Pf10kLLcFR0&t=222s) Meta's pipeline manages over one commit per second across its monorepo via a four-stage process: Flag, Generate, Filter, and Check. - [05:17](https://www.youtube.com/watch?v=Pf10kLLcFR0&t=317s) Diff risk scoring evaluates content discrepancies, blast radius incident history, and codebase topography upfront because scoring compute is far cheaper than test generation. - [06:53](https://www.youtube.com/watch?v=Pf10kLLcFR0&t=413s) Catch tests are synthesized through property-based invariant checks and semantic mutation testing rather than standard unit test generation. - [06:53](https://www.youtube.com/watch?v=Pf10kLLcFR0&t=413s) Incremental context exponentially multiplies catch rates: 0.2% for coincidental, 2.0% for hardening-style LLM generation, 4.0% for diff-aware, and 8.0% for intent-aware generation. - [10:13](https://www.youtube.com/watch?v=Pf10kLLcFR0&t=613s) The majority of raw test failures on risky diffs stem from deliberate behavioral changes by the author rather than actual product defects. - [10:13](https://www.youtube.com/watch?v=Pf10kLLcFR0&t=613s) Human developer trust does not track mathematical expected value; unfiltered false positives cause engineers to quietly disengage from quality gates entirely. - [13:03](https://www.youtube.com/watch?v=Pf10kLLcFR0&t=783s) Meta couples high-volume, low-cost test generation with an aggressive LLM judge filter that discards the vast majority of failures before they reach human reviewers. - [15:21](https://www.youtube.com/watch?v=Pf10kLLcFR0&t=921s) Reframing bug alerts into low-friction verification questions ('This used to be X. Now it's Y. Is that expected?') prevents developer resistance and eases review. - [15:21](https://www.youtube.com/watch?v=Pf10kLLcFR0&t=921s) Measuring prevention requires prioritizing precision over recall and relying on counterfactual modeling against baseline growth to demonstrate business impact. - [17:39](https://www.youtube.com/watch?v=Pf10kLLcFR0&t=1059s) The ultimate bottleneck in software velocity is not how fast AI writes code, but how confidently teams can verify that the generated code is safe.

## Detailed walkthrough ### [00:05](https://www.youtube.com/watch?v=Pf10kLLcFR0&t=5s) The Collapse of the Old Software Testing Compact

For the last half-century, the phrase "the tests passed" served as one of the strongest safety guarantees across the entire software industry, but it has quietly ceased to mean what it once did. Early in engineering careers a decade or two ago, a pre-land test suite routinely took forty minutes to execute: a developer would submit a change request, take a coffee break, and return to discover that three thousand tests had passed—virtually none of which had any bearing on the twelve specific lines of code altered in the change. Yet that process was regarded as working correctly because it reflected a balanced economic deal. Writing software was inherently slow and expensive, so writing test suites to defend that software was equally slow, deliberate, and expensive. Teams hardened code, landed regression suites, and diligently stockpiled defenses against future breakages.

Today, that economic balance has shattered because the cost of generating code has completely collapsed while the verification side stayed frozen. A Stack Overflow survey of roughly 50,000 developers revealed that while 80% were actively leveraging AI tools, only 29% actually trusted the output. Adoption has skyrocketed, but confidence has cratered. As Sreeja Thummala put it: “Landed tests only protect against bugs we already know about; they don't go looking for new ones, so as an industry we are accelerating into an unknown territory with a rear-view mirror.”

- **The Old Pre-land Model**: 40-minute test runs validating 3,000 legacy tests over a 12-line code diff. - **Developer Survey Disconnect**: 80% AI adoption rate vs. 29% trust in AI-generated code output among 50,000 respondents. - **The Core Asymmetry**: Writing code became instantaneous and cheap, while landed verification remained slow, backward-looking, and tied to historical bugs.

---

[02:17](https://www.youtube.com/watch?v=Pf10kLLcFR0&t=137s) Ephemeral Catch Tests: AI-Native Quality for AI Velocity

Faced with this expanding gap between generation speed and verification trust, Meta posed an entirely different foundational question: what if test suites do not wait around to be written, maintained, and hardened over years? Instead of preserving tests indefinitely as technical debt, the system spins up disposable, targeted tests the exact instant a high-risk change appears. Their singular mission is to hunt down bugs introduced by that specific diff. By design, these tests must pass cleanly on the parent commit but fail on the diff commit, proving with statistical rigor that the newly introduced change is the exact culprit that broke functionality. Once their validation job is done, the tests are thrown away entirely.

This disposable paradigm was economically unthinkable before large language models because human engineers could never justify writing throwaway test harnesses for isolated commits. While the tech industry frequently hypes the label "AI-native," the speakers argue that applying AI with surgical precision to testing is what makes massive engineering velocity survivable in real production environments.

```mermaid flowchart TD accTitle: Ephemeral Catch Test Flow accDescr: High-risk diffs trigger disposable AI-generated tests that pass on parent and fail on diff before being discarded.

subgraph Evaluation ["Targeted Evaluation"] D["Risky Diff Submitted"] --> G["Generate Diff-Aware Catch Test"] G --> P["Run against Parent Commit: PASS"] G --> C["Run against Diff Commit: FAIL"] end

subgraph Outcome ["Verification & Cleanup"] P & C --> V{"Proves Diff Caused Failure?"} V -->|"Yes"| B["Flag Bug to Author"] V -->|"Ephemeral"| T["Discard Test (Zero Tech Debt)"] end ```

- **Ephemeral Execution**: Tests are generated on demand for a single commit and discarded immediately, eliminating long-term test suite maintenance and code rot. - **The Differential Oracle**: A valid catch test must evaluate to a green pass on the parent commit and a red failure on the child diff. - **Economic Viability**: LLMs reduce the marginal cost of test synthesis to the point where throwaway test generation becomes cheaper than maintaining brittle suites.

---

[03:42](https://www.youtube.com/watch?v=Pf10kLLcFR0&t=222s) The Four Uncomfortable Questions of AI-Native Quality Gates

Transitioning from a neat theoretical concept to a production-grade AI-native quality gate required the Meta team to confront four difficult operational bottlenecks. The overall architecture follows a four-stage pipeline—Flag, Generate, Filter, and Check—with artificial intelligence embedded in every step rather than tacked onto the end as a single LLM prompt. Sophie Zeng noted: “AI native means to us is AI in every single step, not just one of them.”

The first challenge is triage: Meta's monorepo processes hundreds of thousands of diffs landing daily, representing more than one diff every single second around the clock. Running agentic generation on every diff would burn an untenable amount of expensive GPU compute just to confirm that benign changes are harmless. The second challenge is synthesis design: defining what a bug-catching test looks like and deciding which technical methodology produces it. The third challenge centers on oracle validation: a failing test does not automatically indicate a real product defect. The fourth and most delicate challenge is developer trust: because false positives can never be reduced to absolute zero, surfacing imperfect or noisy signals to already overstretched engineers risks burning team credibility permanently.

```mermaid flowchart LR accTitle: Four-Stage Quality Gate Pipeline accDescr: The Flag, Generate, Filter, and Check stages of Meta's AI quality gate.

A["1. FLAG: Diff Risk Scoring"] --> B["2. GENERATE: Catch Test Synthesis"] B --> C["3. FILTER: True Positive Triage"] C --> D["4. CHECK: Trust & Feedback Delivery"] ```

- **Scale of Changes**: Meta processes hundreds of thousands of diffs per day (over 1 commit per second, 24/7). - **The Core Pipeline Stages**: 1. **Flag**: Score changes upfront to separate benign diffs from risky diffs. 2. **Generate**: Synthesize targeted failing catch tests using contextual techniques. 3. **Filter**: Differentiate genuine bugs from intentional behavioral updates. 4. **Check**: Package and present verified signals without overwhelming engineers.

---

[05:17](https://www.youtube.com/watch?v=Pf10kLLcFR0&t=317s) Flagging Risky Diffs: Cheap Scoring Before Expensive Generation

Because agentic test generation consumes significant compute, Meta enforces a strict rule: scoring must always precede generation. Sophie Zeng explained: “Scoring is cheaper than generation; that's the trick, and getting that order right is the difference between a research demo versus a system you can afford to run in production every day.”

To determine whether a diff warrants deep testing, the scoring system assesses three core dimensions: 1. **Content and Semantic Discrepancy**: The system compares what the diff author wrote in the summary or commit description against what the code modifications actually execute. Discrepancies between stated intent and implemented logic serve as an immediate red flag. 2. **Blast Radius and Incident History**: Changes touching subsystems with a historical record of production outages or high incident frequency receive heightened scrutiny, as certain corners of the codebase have earned suspicion through past failures. 3. **Shape and Topography**: Structural metrics—such as the total lines of code moved, the number of files touched, and the architectural boundaries crossed within the codebase—are evaluated. While shape alone does not point to a specific defect, it identifies where testing resources are worth allocating.

---

[06:53](https://www.youtube.com/watch?v=Pf10kLLcFR0&t=413s) Generating the Failing Test: Invariants, Mutations, and the Context Multiplier

Once a diff is flagged as risky, the system relies on two technical approaches to generate a targeted failing test. The first is property-based checking: taking an established system invariant that held true before the commit—such as an API endpoint strictly requiring an approved authorization policy—and evaluating whether that invariant still holds post-change. If the invariant breaks, it exposes a quiet regression or weakening of existing security controls. The second approach is semantic mutation testing: rather than spraying hundreds of purely mechanical, arbitrary syntax mutations at the codebase, the generator deliberately injects a targeted fault that engineers actively worry about, confirms that no existing test suite catches it, and synthesizes a tailored test to kill that specific mutant.

Three structural shifts separate these catch tests from standard unit test generators: - **Execution Mode**: Moving from permanent suites committed to the repo to ephemeral tests generated dynamically on demand. - **Context Awareness**: Moving from diff-blind generators to engines built exclusively around the exact semantic delta of the diff. - **The Evaluation Oracle**: Conventional generators are optimized to produce green checks (passing tests), whereas catch tests are explicitly rewarded only when they produce a failing test on the diff while passing on the parent commit.

Context proves to be the single most impactful lever governing bug detection rates. When evaluating catch rate performance under increasing levels of context, the gains scale dramatically:

As Sophie Zeng summarized: “Every time we tell the generator a little bit more about the code change, the catch rate roughly doubled.”

---

[10:13](https://www.youtube.com/watch?v=Pf10kLLcFR0&t=613s) The Judgment Problem: Intentional Changes versus True Bugs

Building a model that reliably catches regressions is only half the battle. As Sreeja Thummala revealed, the team soon ran into a surprising wall: even after successfully generating tests that passed on the parent and failed on the diff, the raw output could not be safely shown to engineers. The team realized they were dealing with two distinct problems: catching bugs, and convincing developers that the system actually caught a bug.

```mermaid flowchart TD accTitle: The Test Failure Ambiguity Dilemma accDescr: Identical red test failures arise from both invalid tests, intentional changes, and genuine bugs.

F["Test Fails on Diff & Passes on Parent"] --> S["Produces Identical Red Signal"] S --> C1["Flawed Generation:<br/>Unrealistic state / broken assumption"] S --> C2["Intentional Feature Shift:<br/>Author intended to change behavior"] S --> C3["Genuine Product Defect:<br/>Unintended regression"]

When an ephemeral test fails, that failure stems from one of two fundamentally different causes: 1. **Faulty Generator Logic**: The test asserted an invariant that was never guaranteed, or it constructed an artificial runtime state that production could never encounter. This is an engineering flaw in the generator that can be corrected with better modeling. 2. **Intentional Behavioral Shifts**: The test is syntactically and logically sound, the parent passes, and the diff fails—making it a technically flawless catch—but the author purposefully changed the system behavior.

Because both scenarios produce the exact same red failure signal, code-level analysis alone cannot differentiate an intentional design change from an accidental regression. When testing aggressively against high-risk diffs, the overwhelming majority of raw failures stem from intentional changes the author designed on purpose.

While saving an engineer a day of debugging through a true positive mathematically outweighs the ten minutes wasted investigating a false positive, human psychology does not follow mathematical expected value. As Sreeja Thummala emphasized: “Trust is won in drops and lost in buckets, as they say, and the failure mode for this is quieter than people complaining: it's just that they stop paying attention.” If engineers are exposed to unfiltered, noisy failures, they simply tune out the quality gate entirely.

[13:03](https://www.youtube.com/watch?v=Pf10kLLcFR0&t=783s) The LLM Judge and Aggressive Filtering

Sreeja explains that the team made a crucial decision to hold back deployment when they realized the raw signal was still too noisy. In developer tooling, the cost of crying wolf is permanent disengagement: once an engineer decides an automated system produces noise, they tune it out completely and never look at it again. Rather than pushing an imperfect alert stream into code reviews, the team stepped back to analyze what separates an actual software defect from an intentional code modification. They reframed the problem as an explicit classification decision: could an LLM judge sit between the test generator and the engineer to determine whether a failure reflects an author's deliberate change or an accidental regression?

```mermaid flowchart LR accTitle: Test Generation and LLM Judge Pipeline accDescr: The generator creates candidate tests in high volume, which are filtered by an LLM judge before reaching the engineer. subgraph Generation ["High-Volume Exploration"] A["Code Diff"] --> B["Test Generator"] B --> C["Candidate Test Failures"] end subgraph Judgment ["Confidence Filter"] C --> D["LLM Judge"] D -->|"Discard ~90%+\n(Deliberate changes & noise)"| E["Discarded"] end subgraph HumanReview ["Developer Attention"] D -->|"High-Confidence Defect"| F["Engineer Review"] end ```

The resulting architecture deliberately relies on an ultra-aggressive filter, a design choice Sreeja notes many engineers find completely counterintuitive. Instead of trying to make the generator output pristine tests, Meta lets the generator hypothesize cheaply, broadly, and in massive volume without caring how many tests are wrong. The LLM judge then scrutinizes every generated failure against the diff, estimating the probability of an unintended defect, and ruthlessly throws away the vast majority of generated candidates. Sreeja Thummala summarized the governing philosophy of this design: “The overwhelming majority of the test this system generates never reach an engineer.” By presenting human reviewers with only high-confidence failures, Meta achieved an estimated 70% reduction in engineer bug-triaging time—not by catching fewer bugs, but by eliminating the time spent reviewing false alarms.

- **Aggressive filtering ratio:** Most tests generated by the system are discarded immediately; only signals passing a high-confidence threshold reach human reviewers. - **Triage efficiency:** Achieved an approximate 70% reduction in developer triage time by protecting engineers from investigating non-bugs. - **Architectural philosophy:** Generate widely, fail cheaply, discard almost everything, and protect developer attention at all costs.

---

[15:21](https://www.youtube.com/watch?v=Pf10kLLcFR0&t=921s) Reframing the Developer Ask and Measuring Absence

Beyond filtering, Meta fundamentally altered the way signals are presented to developers. Proclaiming that a pull request contains a bug is an aggressive assertion that naturally triggers resistance and debate. The team replaced accusatory alerts with a low-friction question: *"This used to be X. Now it's Y. Is that expected?"* This framing takes seconds to review, is simple to confirm, and carries zero friction if an engineer chooses to dismiss it. Sreeja pointed out an unexpected insight: inferring diff intent had originally doubled test generation accuracy, but it turned out to be just as essential on the downstream delivery side to make notifications palatable. The exact same missing variable—understanding developer intent—had to be solved at both boundaries of the pipeline.

Sreeja openly acknowledged two fundamental trust dilemmas inherent to prevention tooling that cannot be formally proven. First, Meta measures precision but deliberately ignores recall; measuring recall would require tracking bugs that were discarded and never reached production, representing an accepted blind spot in their evaluation. Second, while individual developers build trust one accurate diff notification at a time, engineering leadership requires aggregate proof of impact. Because platform teams cannot simply point to an incident that never occurred, Meta turned to counterfactual modeling, comparing observed incident rates directly against projected baseline growth. Sreeja Thummala emphasized how fragile developer adoption is: “Once you taught an engineer that your tool is noise, you cannot really unteach it and there is no second launch.”

- **The prompt shift:** Reframed alerts from bug assertions into simple diff comparisons (*"This used to be X. Now it's Y. Is that expected?"*) that take seconds to verify or dismiss. - **Symmetric intent inference:** Developer intent inference doubled generator catch rates on the frontend and proved equally mandatory for output delivery and human acceptance on the backend. - **Precision over recall:** Meta accepts an unmeasured recall metric because measuring recall would require tracking prevented defects that never shipped. - **Counterfactual proof:** Overcame leadership skepticism by benchmarking actual metrics against modeled counterfactual trends rather than raw catch counts.

---

[17:39](https://www.youtube.com/watch?v=Pf10kLLcFR0&t=1059s) 74 Prevented Incidents and the Future of AI Testing

Sophie Zeng took the stage to detail the concrete business impact and address the inherent paradox of prevention metrics: "Nobody can see what didn't happen." Meta's counterfactual tracking demonstrated that the system successfully averted 74 SEV (site reliability/privacy) incidents. When tracking the trendline, privacy incident growth dropped roughly one-fifth lower than Meta's internal baseline forecast without the tool. While acknowledging that counterfactual projections are models rather than raw physical measurements, Sophie emphasized that bending the incident trajectory provided the exact evidence needed to satisfy both grassroots engineers and executive leadership.

Looking forward, Sophie outlined three major transformations already reshaping Meta's testing infrastructure. First, test suites are shifting from permanent, durable codebases to disposable, ephemeral artifacts that are generated for a specific diff and discarded as soon as the pull request is merged. Second, the team is solving the problem of sustaining developer trust across long operational horizons. Third, test generation is moving out of downstream CI review gates and shifting left directly into developer editing environments, transforming the AI from an external code reviewer into an interactive co-author. Sophie Zeng captured the foundational bottleneck of the modern AI software lifecycle: “And whatever you are building, the limit of how fast you can move isn't how fast the model write your code. It's how confident you are that what it wrote is safe.”

- **Incident prevention:** 74 critical SEV incidents prevented from ever reaching production. - **Trendline impact:** Privacy incident growth was bent down roughly one-fifth lower compared to the counterfactual forecast. - **Disposable test paradigm:** Tests are dynamically generated on a per-diff basis and discarded once their verification purpose is complete. - **Shift left to co-authoring:** Moving AI verification from asynchronous pull request review comments directly into IDE-level authoring loops. - **Definition of speed:** Real engineering velocity is governed by verification confidence rather than raw generation throughput.

## Notable quotes > [17:39](https://www.youtube.com/watch?v=Pf10kLLcFR0&t=1059s) The limit of how fast you can move isn't how fast the model write your code. It's how confident you are that what it wrote is safe. — Sophie Zeng

> [05:17](https://www.youtube.com/watch?v=Pf10kLLcFR0&t=317s) Scoring is cheaper than generation; that's the trick, and getting that order right is the difference between a research demo versus a system you can afford to run in production every day. — Sophie Zeng

> [15:21](https://www.youtube.com/watch?v=Pf10kLLcFR0&t=921s) Once you taught an engineer that your tool is noise, you cannot really unteach it and there is no second launch. — Sreeja Thummala

> [00:05](https://www.youtube.com/watch?v=Pf10kLLcFR0&t=5s) Landed tests only protect against bugs we already know about; they don't go looking for new ones, so as an industry we are accelerating into an unknown territory with a rear-view mirror. — Sreeja Thummala

## Mentioned Sophie Zeng, Sreeja Thummala, Meta, Stack Overflow, Scale

[Source](https://www.youtube.com/watch?v=Pf10kLLcFR0)

中文简报

Meta借助一次性AI测试规避74起生产故障工程速度的瓶颈在于验证信心而非代码生成速度，这迫使工程团队淘汰永久测试套件，转而使用经过强力过滤的一次性测试，以维护开发人员的信任。随着AI代码生成日益加速，而传统测试套件依然缓慢且滞后，Meta围绕一次性且能感知diff的“捕捉测试”（catch tests）重构了其测试架构；这类测试在父提交上通过，但在引入变更的diff上失败。该系统通过“标记、生成、过滤、检查”四阶段流水线处理每日数十万次提交，在调用昂贵的测试合成之前先利用低成本启发式风险评分进行分流。为避免误报和故意的产品变动损害开发者的信任，LLM评审器会主动筛除超过90%的候选失败用例，并将通知表述为中立询问而非断定性的缺陷警报。这种方法将工程师的分类排查时间减少了约70%，明显规避了74起严重SEV故障，同时使隐私事件的增长比基准预测降低了约20%。 Meta设计了一种AI原生质量把关机制，针对高风险diff部署临时且用后即弃的捕捉测试；这些测试在父提交上通过、在当前diff上失败以隔离回归问题，同时不会增加长期的测试维护负担。 (2:17) 为测试生成器提供完整的开发者意图和失效假设后，缺陷捕捉率从4.0%翻倍至8.0%，相较于未结合diff的基准提升了40倍。 (6:53) 为防止开发者产生通知疲劳，LLM评审器会对测试失败情况进行评估并筛除绝大多数候选测试，促使工程师的分类排查时间估计减少了70%。 (13:03) 反事实基准建模显示，Meta的该系统规避了74起重大SEV故障，并使隐私事件的增长比预测值降低了约五分之一。 (17:39)

Reading summary

Meta engineers Sophie Zeng and Sreeja Thummala argue that as code synthesis becomes cheap and rapid, verification bottlenecks can only be solved through ephemeral, disposable tests filtered aggressively by an AI judge. The Asymmetry of Modern Software Verification (0:05) Historically, long-running test suites balanced the high cost of manual software authoring. Today, that economic equation has broken down because code generation costs have fallen while verification remains tied to historical regressions. According to a Stack Overflow survey of roughly 50,000 developers, 80% use AI coding tools, but only 29% trust the resulting code. Traditional landed test suites execute legacy tests that do not actively seek out newly introduced bugs, creating an expanding gap between code production speed and deployment confidence. Ephemeral Catch Tests and Contextual Generation (2:17) To prevent permanent test-suite bloat, Meta synthesizes disposable catch tests designed specifically for high-risk code changes. A catch test operates under a differential oracle: it must pass on the parent commit and fail on the diff commit to prove the change broke functionality, after which it is discarded. Running on Meta's monorepo scale of hundreds of thousands of daily diffs requires scoring code diffs for semantic discrepancies, blast radius, and structural shape before initiating compute-heavy generation. Catch efficiency depends heavily on the level of contextual input provided to the generator. In Meta's evaluations, a diff-blind coincidental baseline achieved a 0.2% bug catch rate, whereas standard LLM generation reached 2.0%. Exposing the raw diff increased the rate to 4.0%, and providing full intent awareness—combining the diff, intended outcome, and failure hypotheses—raised the catch rate to 8.0%. Filtering Intentional Changes to Protect Developer Trust (10:13) A failing differential test does not always indicate an unintended regression; authors frequently alter system behaviors on purpose, producing an identical red failure signal. Because noisy alerts lead developers to disengage permanently, Meta deployed an LLM judge between test generation and developer notifications. Rather than constraining the generator to perfect output, the pipeline produces candidates in high volume, and the judge discards the vast majority of non-defects and deliberate modifications. This aggressive filtering reduced developer bug-triaging time by an estimated 70%. Meta also softened the user interface by replacing bug assertions with non-accusatory comparisons ("This used to be X. Now it's Y. Is that expected?"), lowering friction for authors reviewing or dismissing alerts. Measuring Absences Through Counterfactual Impact (15:21) Evaluating automated prevention tools introduces measurement dilemmas because prevented production defects cannot be directly counted. Meta prioritizes precision and accepts an unmeasured recall metric, avoiding tracking discarded potential bugs that never reached production. To justify platform impact to leadership, the team compared operational data against projected counterfactual baselines, finding that the system averted 74 site reliability and privacy incidents and lowered privacy incident growth roughly one-fifth below baseline projections. Meta plans to shift ephemeral test generation from asynchronous pull request reviews directly into developer editing environments as an interactive authoring assistant. Sophie Zeng maintains that sustainable engineering velocity is ultimately bounded by verification confidence rather than model code generation speeds.

精炼阅读摘要

Meta 工程师 Sophie Zeng 与 Sreeja Thummala 指出，生成代码成本骤降与传统验证滞后打破了测试的经济平衡，Meta 因而构建了针对高危改动动态生成、用完即弃的临时捕获测试体系。传统验证失效与临时测试范式 (0:05) 在过去半个世纪中，测试通过曾是软件工程的核心安全保障。但 Stack Overflow 对约 5 万名开发者的调查显示，尽管 80% 的人积极使用 AI 工具，仅有 29% 信任其生成的代码输出。过去耗时 40 分钟运行 3000 个旧测试来验证 12 行代码的合入前模式已无法应对极速膨胀的代码量；既有测试只能防范已知缺陷，无法探索未知漏洞。针对生成速度与验证信任脱节的问题，Meta 提出了“临时捕获测试”（Ephemeral Catch Tests）。该模式不再长期维护作为技术债务的测试套件，而是在高危 diff 提交时即时合成一次性测试。此类测试要求在父 commit 上顺利通过，但在当前 diff commit 上报错失败，以证明代码改动破坏了功能，验证完成后即刻彻底废弃。大语言模型大幅降低了生成边际成本，使合成一次性测试在经济上比长期维护脆弱测试更为合算。四阶段质检与低成本前置风险评分 (3:42) Meta 单体代码库每天处理数十万次 diff，相当于全天每秒合入超过一次。对每个改动运行端到端智能体生成会消耗海量昂贵的 GPU 算力。因此，团队将质检流水线划分为“标记（Flag）、生成（Generate）、过滤（Filter）、检查（Check）”四个阶段，并坚持在生成前先进行低成本评分。评分系统依据三个维度评估改动风险：首先是内容与语义差异，比对作者在说明中声称的意图与实际执行逻辑是否脱节；其次是影响半径与事故历史，对曾发生线上故障或事故频发的子系统提高审查权重；最后是代码改动的形态与拓扑结构，包括变更行数、涉及文件数及跨越的架构边界。通过前置评分，系统得以精准筛选真正值得分配生成算力的高风险改动。上下文驱动的定向失败测试生成 (6:53) 一旦 diff 被标记为高危，系统主要通过两种技术途径生成定向报错测试：一是基于属性的检查，验证合入前原本成立的系统不变量（例如 API 端点严格要求的授权策略）在变更后是否仍成立；二是语义变异测试，针对工程师担忧的故障注入特定缺陷，并合成能精准击杀该变异的测试。与传统追求测试通过的单元测试生成器不同，捕获测试仅在父 commit 通过且在子 diff 报错时才被认定有效。实验表明，上下文是提升缺陷捕获率的关键杠杆。无 diff 上下文的纯随机捕获率仅为 0.2%；常规 LLM 加固生成的捕获率为 2.0%；引入原始 diff 代码后上升至 4.0%；而结合了 diff 代码、预期结果及潜在失败假设的意图感知生成，捕获率进一步翻倍至 8.0%。向生成器提供越丰富的改动意图，捕获效率越呈现成倍提升。 LLM 裁判激进过滤与开发者信任交互 (10:13) 由于代码层面的测试报错既可能源于真正的缺陷，也可能来自作者有意为之的行为变更或生成器本身的逻辑瑕疵，未经筛选的报错无法直接呈现给开发者。在开发者工具中，频繁报虚警会导致工程师彻底放弃关注。为此，Meta 在测试生成器与开发者之间引入 LLM 裁判（LLM Judge），允许前端廉价且大量地发散生成，再由裁判结合 diff 意图无情丢弃绝大多数测试候选。这种极度激进的过滤机制将工程师排查 bug 的时间减少了约 70%。同时，下游交互形式也从指责性的“发现 bug”提示，转变为低阻力的差异确认：“此前为 X，现变为 Y，这是否符合预期？”。团队在评估中明确将精确率置于召回率之上，宁可承受未被追踪的召回盲区，也要全力保护开发者的注意力与信任度。反事实验证成效与未来左移展望 (17:39) 预防性工具的成效通常难以直接用常规指标度量。Meta 利用反事实模型将实际事故发生率与没有该工具时的基线增长预测进行比对，确认该系统成功拦截了 74 起高等级（SEV）线上故障，并将隐私事故的增长轨迹压低了约五分之一。面向未来，Meta 正推动测试体系发生三大转变：测试套件从永久持久化代码转向用完即弃的动态工件；建立长期维持开发者信任的运作机制；以及将测试生成能力从下游 CI 审查流水线全面“左移”至开发者 IDE 编写环境，使 AI 从外部代码审查者转变为实时协同创作者。Sophie Zeng 强调，现代软件交付的极限不再取决于模型写代码的速度，而取决于团队确信代码安全的信心。该系统在实操中仍面临预防类工具固有的度量局限：Meta 选择放弃追踪由于过滤或未捕获而溜走的缺陷召回率，转而完全依赖反事实模型向管理层证明价值。随着测试逻辑进一步从 CI 审查向 IDE 实时协作左移，如何在交互早期平衡算力成本与噪音控制，仍是该体系持续演进的核心课题。

Detailed notes — zh

## Background 该视频由 YouTube 频道 @Scale 于 2026 年 10 月 30 日发布，主讲人为来自 Meta 的 Sophie Zeng 与 Sreeja Thummala。页面未提供具体的简介信息，内容围绕 AI 生成代码背景下自动化测试的重构与信任机制展开。 ## TL;DR 面对 AI 编程带来的边际成本骤降与开发者信任鸿沟，Meta 提出了“即用即弃测试”的全新质量门禁架构。系统首先对高风险代码变更（Diff）进行低成本风险打分，随后生成针对该变更意图、在父提交通过而在当前变更失败的专用测试用例。为了避免误报消耗开发者的信任，平台引入 LLM 裁判进行激进过滤，并以客观提问的形式呈报给工程师，成功在生产环境中阻断了 74 起高危故障与合规事故，将测试从永久维护的技术债转变为按需生成的防御屏障。 ## Atmosphere & style 整场演讲呈现出冷静、硬核且极具反思性的工业级工程叙事风格。两位讲者配合默契，毫不避讳地拆解了团队在误报率和开发者心理学上踩过的坑，将原本枯燥的代码测试升华到了“人类信任数学”的认知层面。台下的听感既有大规模分布式架构落地的扎实严谨，又带有某种打破半个世纪传统测试教条的颠覆感与笃定。 ## Bulletin - [00:05](https://www.youtube.com/watch?v=Pf10kLLcFR0&t=5s) Stack Overflow 对约 5 万名开发者的调研显示，AI 工具采用率已达 80%，但开发者对其输出代码的信任度仅有 29%，突显出代码生成提速与质量验证之间的严重脱节。 - [03:42](https://www.youtube.com/watch?v=Pf10kLLcFR0&t=222s) Meta 提出“即用即弃测试”新范式：针对每天合入数十万次的高风险变更动态生成专用测试，要求在父提交通过而在当前变更失败，验证后立即销毁且不留技术债。 - [06:53](https://www.youtube.com/watch?v=Pf10kLLcFR0&t=413s) 实验表明测试生成的上下文深度直接决定捕获率：巧合捕获率仅 0.2%，引入变更代码提升至 4%，而注入意图与破坏边界推演后捕获率跃升至 8%。 - [13:03](https://www.youtube.com/watch?v=Pf10kLLcFR0&t=783s) Meta 引入二级 LLM 裁判架构对生成用例进行激进过滤，工程师分流 Bug 的耗时锐减约 70%，在生产环境中已成功拦截 74 起严重事故，并将隐私事故增长率压低约 20%。 ## Key takeaways - [00:05](https://www.youtube.com/watch?v=Pf10kLLcFR0&t=5s) 存量测试套件本质上是通过后视镜看世界，仅能防御已知的既往回归缺陷，无法应对 AI 时代低成本代码生成带来的未知新错误。 - [03:42](https://www.youtube.com/watch?v=Pf10kLLcFR0&t=222s) Meta 将 AI 原生质量门禁拆解为 Flag（打分）、Generate（生成）、Filter（过滤）与 Check（校验）四步，先打分再生成的顺序是工业级系统可负担的关键。 - [03:42](https://www.youtube.com/watch?v=Pf10kLLcFR0&t=222s) 高风险变更打分聚焦三要素：变更内容（说明与实际行为的脱节）、爆炸半径（历史故障频度）与变更形态（增减行数及跨文件规模）。 - [06:53](https://www.youtube.com/watch?v=Pf10kLLcFR0&t=413s) 测试生成依赖基于属性和基于变异的两大家族，核心在于生成在父节点通过但在变更节点失败的测试，以此证明缺陷由该代码引入。 - [06:53](https://www.youtube.com/watch?v=Pf10kLLcFR0&t=413s) 测试上下文具备捕获率翻倍效应：从盲目的巧合捕获（0.2%）到代码补丁感知（4%），再到意图感知（8%），每多理解一层意图，捕获率即大幅倍增。 - [10:14](https://www.youtube.com/watch?v=Pf10kLLcFR0&t=614s) 真阳性可节省工程师一天调试时间，假阳性仅浪费 10 分钟，但信任流失遵循“以滴积累、以桶流失”，高频误报会导致工程师彻底忽略系统信号。 - [13:03](https://www.youtube.com/watch?v=Pf10kLLcFR0&t=783s) 测试失败常因业务刻意调整而非逻辑缺陷造成，由于代码本身无法表达完整意图，必须在生成器与开发者之间架设 LLM 裁判进行激进过滤。 - [13:03](https://www.youtube.com/watch?v=Pf10kLLcFR0&t=783s) LLM 裁判采取“宽进严出”原则，大批量廉价试错并丢弃绝大部分无意图或误报用例，仅向人工呈现极高置信度的缺陷信号。 - [15:21](https://www.youtube.com/watch?v=Pf10kLLcFR0&t=921s) 交互模式从定罪式断言（“代码有 Bug”）转变为中立提问（“原为 X 现变为 Y，符合预期吗”），大幅降低开发者的防御心理与验证成本。 - [15:21](https://www.youtube.com/watch?v=Pf10kLLcFR0&t=921s) 系统面临召回率不可观测的单点盲区，团队选择优先保护工程师注意力；面对预防价值难以直接度量的问题，则采用反事实建模向管理层证实因果关系。 - [17:39](https://www.youtube.com/watch?v=Pf10kLLcFR0&t=1059s) 该体系在生产环境中成功拦截 74 起高危故障及隐私事故，通过反事实对比使实际隐私事故增长率比基线低约五分之一。 - [19:22](https://www.youtube.com/watch?v=Pf10kLLcFR0&t=1162s) AI 测试正向三大方向演进：从持久资产走向即抛资产、构建长效自动化信任，以及机制全面左移成为代码编写的“共同作者”。 - [19:22](https://www.youtube.com/watch?v=Pf10kLLcFR0&t=1162s) 软件研发提速的最终上限从来不是大模型生成代码的速度，而是工程师对所生成代码是否安全、能否被信任的信心底线。 ## Detailed walkthrough ### [00:05](https://www.youtube.com/watch?v=Pf10kLLcFR0&t=5s) “测试通过”背后的信任断层与即用即弃测试的新范式在软件开发领域，“测试通过（the test passed）”这句陈述在过去半个世纪中一直是整个行业最坚固的质量保障基石，但如今它早已失去了原有的含义。在传统的研发流程中，工程师提交代码变更后，预合入（pre-land）测试往往需要跑上三四十分钟；工程师甚至可以去喝杯咖啡，回来发现三千多个测试用例全部变绿，即使这些测试没有一个与刚刚修改的那十二行代码真正相关。这种旧契约的逻辑在于：编写代码既缓慢又昂贵，因此编写防护测试同样缓慢昂贵，二者在成本上保持着微妙的平衡。然而，随着大语言模型的普及，写代码的边际成本断崖式下跌。Stack Overflow 对约 50,000 名开发者的调研显示，80% 的人在使用 AI 辅助工具，却仅有 29% 的人信任这些工具生成的代码产物。 Sreeja Thummala 指出：“Adoption went up but trust went down. That gap is the whole reason we are up here because nothing on the other side of equation mode. Landed test only protect against bugs we already know about. They don't go looking for new ones. So as an industry we are accelerating into an unknown territory with a rear view mirror.” 存量测试套件本质上是在通过后视镜看世界，只能防御已知的既往回归缺陷，无法发现未知的新错误。为此，Meta 提出了全新的思路：不再等待人工编写永久性的回归测试套件，而是在高风险代码变更（diff）出现的瞬间，自动生成只为捕捉该变更缺陷而存在的专用测试；这种测试要求在父提交（parent commit）上能够通过、在当前变更上必然失败，以此高置信度地证明缺陷正是由该变更引入的；随后这些测试会被立即销毁，既不需要维护成本，也不会留下技术债。 - **行业调研数据**：Stack Overflow 针对约 50,000 名开发者的调查显示，AI 工具采用率高达 80%，但对输出结果的信任度仅有 29%。 - **核心逻辑反转**：传统测试追求的是长期保留和绿灯通过，而新范式生成的是只为当前变更服务的临时用例，追求在特定变更上的红灯失败，并在执行后即刻抛弃。 --- ### [03:42](https://www.youtube.com/watch?v=Pf10kLLcFR0&t=222s) AI 原生质量门禁与风险代码打分 Sophie Zeng 将系统拆解为四个互相关联的关键步骤：标记（Flag）、生成（Generate）、过滤（Filter）与校验（Check）。要实现这一体系，必须解答四个棘手问题：每天有海量 diff 合入，不可能为每个变更都生成测试，究竟选哪些？选定 diff 之后，能抓到 bug 的测试长什么样？测试失败并不等同于捕获了真 bug，如何确认是真阳性？假阳性永远无法归零，如何在不消耗工程师信任的前提下将带有瑕疵的信号呈现给他们？在 Meta 的规模下，每天合入的代码变更高达数十万次，相当于全天候每秒都有超过一个 diff 合入。由于基于大模型的测试生成并非免费，若对每一次变更都盲目生成测试，将耗费天文数字级的算力，最终却发现绝大部分代码本身并无大碍。因此，系统在生成测试之前必须先进行风险打分（Scoring）。Sophie Zeng 强调，打分成本远低于生成成本，理顺这一先后顺序正是区分“实验室研究原型”与“每天在生产环境中跑得起的工业级系统”的关键所在。 ```mermaid flowchart LR accTitle: AI Native Quality Gate Pipeline accDescr: Pipeline showing Flag, Generate, Filter, and Check stages powered by AI. subgraph SG1["阶段 1：筛选与生成"] A["Flag（风险打分）"] --> B["Generate（定向测试生成）"] end subgraph SG2["阶段 2：校验与交付"] B --> C["Filter（意图过滤）"] C --> D["Check（置信度校验与呈现）"] end ``` - **海量变更规模**：Meta 内部每天合入数十万个 diff，合入速率持续超过 1 diff/秒。 - **风险判定三要素**： - **变更内容（Content）**：比较 diff 描述声称要做的事情与实际代码行为之间是否存在脱节或偏差。 - **爆炸半径（Blast radius）**：评估受影响代码是否具有高频故障历史，定位那些已被历史事故证明脆弱的模块。 - **变更形态（Shape）**：统计代码行数的增减移动、跨越的文件数量以及所涉及的具体代码库属性。 --- ### [06:53](https://www.youtube.com/watch?v=Pf10kLLcFR0&t=413s) 测试生成范式与上下文对捕获率的倍增效应选定高风险变更后，系统主要依赖两类技术家族来构建能够触发失败的测试。第一种是**基于属性的测试（Property-based）**：提取变更前原本成立的不变式（例如某个端点必须强制走审批过的鉴权策略），检测变更后该不变式是否被悄然破坏或削弱；第二种是**基于变异的测试（Mutation-based）**：主动注入团队实际担忧的故障类型，然后生成专门能捕捉该故障的测试。与传统变异测试机械性地抛出数百种变异不同，该系统仅生成少数真正值得关注的变异体，并在当前测试完全遗漏该变异时，编写定向测试将其杀死。这两类方法的成立依赖三大范式转变：执行模式从“永久存留”变为“按需即用即弃”；上下文感知从“盲目独立”变为“针对具体 diff 定制”；神谕判定（Oracle）从“以生成通过测试为荣”彻底转向“以在父节点通过但在变更节点失败为准”。在这之中，上下文信息起到了决定性作用。Sophie Zeng 给出了一组捕获率（Catch Rate）的实验对比： - **巧合捕获（Coincidental catch）**：测试对变更一无所知，全凭运气碰上，基线捕获率为 0.2%。 - **通用加固型 LLM 生成（Hardening LLM generation）**：捕获率达到 2%，虽比基线高出 10 倍，但对具体变更仍是盲目的。 - **变更感知型生成（Diff-aware）**：让生成器看具体的代码补丁内容，捕获率跃升至 4%。 - **意图注入型生成（Intent-aware）**：进一步向生成器输入 diff 的真实意图以及可能破坏边界的推演，捕获率再次翻倍至 8%。 Sophie Zeng 总结道：“Every time we tell the generator a little bit more about the code change, the catch rate roughly doubled. That's what I mean about AI native. In every step, the generator is only as good as what it knows about the diff. And working that out, what to tell it is most of the job.” --- ### [10:14](https://www.youtube.com/watch?v=Pf10kLLcFR0&t=614s) 真阳性陷阱与工程师信任的数学难题即使平台能够精准生成在父提交通过、在当前变更失败的测试，直接将其推给工程师依然行不通。Sreeja Thummala 回归演讲时坦言，团队曾误以为这只是一个“如何构建抓 bug 平台”的技术问题，后来才发现核心是“如何让工程师相信我们抓到的是真 bug”，后者不是模型优化问题，而是人类工程判断力的问题。一个生成的测试变红失败，背后的根因可能完全不同：一方面是生成器自身产生缺陷，例如断言了系统从未承诺过的行为，或构建了生产环境不可能达到的非法状态；但更普遍且棘手的情况是，测试本身完全正确、失败也是真实的，但该变更本质上就是在**故意**调整业务行为。对于这类变更，测试在父提交成功、在当前变更失败，在机械指标上属于完美的“捕获”，但在工程师眼中却是纯粹的干扰噪音。在海量高风险 diff 的激进生成中，绝大部分红灯恰恰属于这种“作者本意如此”的变更行为。代码本身并不包含作者的全部意图，这种信息鸿沟无法靠单纯继续调用生成模型来填补。 Sreeja Thummala 强调了工程师信任的非对称损耗机制：“A true positive saves an engineer maybe about a day of debugging. A false positive cost them 10 minutes. So an expected value, you could argue we could have shipped it and eaten the noise. The arithmetic is on our side. But that's not how humans work. A true positive ends you a little credit. A false positive spends a lot of it. Trust is won in drops and lost in buckets as they say.” - **期望值悖论**：数学期望上，一次真阳性可替工程师节省约整整一天（8 小时）的调试排障时间，一次假阳性仅浪费 10 分钟评估时间，数学收益看似划算，但人性心理无法容忍高频假阳性。 - **信任流失模式**：工程师对系统的信任“以滴积累，以桶流失（won in drops and lost in buckets）”；而最致命的失败模式并不是工程师公开投诉，而是他们开始默默忽略并彻底无视所有的预警信号。 ### [13:03](https://www.youtube.com/watch?v=Pf10kLLcFR0&t=783s) 保护工程师注意力：LLM 裁判的激进过滤架构 Sreeja Thummala 深入剖析了自动化工具上线时最致命的陷阱：一旦测试警报沦为噪声，工程师就会彻底失去对工具的信任。“Once you thought an engineer that your tool is noise, you cannot really unteach it and there is no second launch.”（“一旦你让工程师认定你的工具只是噪声，你就再也无法扭转这种认知，而且绝不会有第二次重新发布的机会。”）Sreeja 直言，团队深知这一点，因此宁可不发布最初的原型，也要先彻底解决区分“真正的缺陷”与“测试误报”的难题。他们反思的核心问题在于：系统需要什么样的信息，才能准确判断测试失败究竟是作者未预料到的缺陷，还是作者故意为之的业务变更？为了破局，团队构建了一套引入大模型作为裁判的二级架构（LLM Judge）。该系统直接嵌入在测试用例生成器（Generator）与工程师之间。LLM 裁判会审视生成的测试用例、代码变更（Diff）以及失败信号，评估该失败属于“非预期缺陷”而非“刻意修改”的概率，只有达到极高置信度的信号才会被推送到人工面前。Sreeja 强调，这种过滤策略极其激进，甚至反直觉到绝大多数生成的测试用例都会被直接丢弃：“Propose hypothesis aggressively, cheaply in volume without worrying whether it's wrong and then let the judge decide which one are worthy of an engineer's attention.”（“以极低成本大胆且大批量地提出假设，无需担心是否出错，然后交由裁判去判定哪些才配得上工程师的注意力。”） ```mermaid flowchart LR accTitle: LLM Judge Architecture accDescr: 架构展示了从代码变更到生成海量测试，再通过 LLM 裁判激进过滤并仅向工程师呈报高置信度缺陷的流程。 subgraph GenStage["生成阶段"] A["代码变更 (Diff)"] --> B["测试生成器 (Generator)"] B --> C["海量假设测试"] end subgraph JudgeStage["裁决过滤阶段"] C --> D["LLM 裁判 (Judge)"] D -->|"丢弃绝大多数误报/无意图用例"| E[("废弃池")] D -->|"极高置信度缺陷信号"| F["高确定性 Bug 警报"] end subgraph DevStage["人工交付阶段"] F --> G["工程师极速分流"] end ``` - **大批量廉价试错**：系统的核心逻辑是“宽进严出”——生成器只管大量生成，绝大部分产物会被无情抛弃，这并不代表生成器失效，反而是其各司其职的表现。 - **保护工程师注意力**：过滤机制使工程师分流 Bug 的耗时锐减了约 70%；这并非因为找到的 Bug 变少了，而是工程师再也不用把时间浪费在根本不是 Bug 的误报上。 --- ### [15:21](https://www.youtube.com/watch?v=Pf10kLLcFR0&t=921s) 从断言到提问与信任的两大盲区除了过滤模型，团队还彻底重构了向工程师呈现信息的方式。以往的系统倾向于直接下定论并指责“你的 Diff 存在 Bug”，这种主观断言往往会立刻激起开发者的反驳与争论。新机制将其精简为一个客观中立的事实比对：“这里原本是 X，现在变成了 Y。这是符合预期的改动吗？”这种提问方式让工程师无需心理防御，只需几秒钟便能确认，即使随手驳回也没有任何负担，真正实现了“快速验证、极易忽略”。 Sreeja 指出，Sophie 此前提到推断 Diff 的业务意图使生成器的捕获率翻倍，而事实证明，意图推断也是让交付端输出具备可用性的唯一要素。“Same missing variable both ends of the pipeline. We just didn't realize it until we hit it twice.”（“流水线两端缺失的原型变量完全相同；只是在我们两次撞南墙之后才意识到这一点。”）与此同时，Sreeja 坦率承认了系统目前无法实证的两个信任盲区： | 信任维度 | 面临的度量挑战与核心取舍 | 解决方案 / 妥协现状 | |:--- |:--- |:--- | | **单点信任（召回率盲区）** | 团队只度量查准率（Precision），无法统计被过滤丢弃的测试中有多少是真实 Bug（Recall）。召回率在此意味着去统计“从未上线过的潜在 Bug”，属于不可观测的盲区。 | 主动接受该盲区作为已知限制，优先保障工程师注意力不受干扰，将其列为后续最关键的待解课题。 | | **全局信任（管理层盲区）** | 工程师通过一个个精准信号建立信任，而领导层需要对整体指标建立信任。但预防性工作（Prevention）无法直接统计“没有发生的事”。 | 引入反事实度量（Counterfactual Measurement），通过对比没有该系统下的历史预测趋势线与实际事故线来证明价值。 | --- ### [17:39](https://www.youtube.com/watch?v=Pf10kLLcFR0&t=1059s) 74 起阻断事故与反事实度量法 Sophie Zeng 接过话茬，向全场公布了这套系统交出的成绩单：系统在生产环境拦截了整整 74 起事故。她特别强调，这些绝非无关痛痒的静态代码规范（Linter）告警，每一例都是真实被遏制在萌芽状态的线上故障（SEV）或隐私合规事故。然而在展示成效时，团队同样遭遇了防御性工程共有的阻力——“没人能亲眼看见未曾发生的事”。面对工程师和管理层合理的怀疑，单纯罗列“捕获了多少个 Bug”已经缺乏说服力，必须采用反事实建模来证实因果关系。Sophie 解释道：“Counterfactuals are arguable. But that is the thing that moved the conversation because it's the difference between we caught some bugs versus the curve bent.”（“反事实预测确实存在争论空间，但正是它推动了对话，因为‘我们抓到了一些 Bug’与‘我们彻底压低了事故曲线’完全是两码事。”） - **74 起严重事故拦截**：每一个数字代表一次被提前消解的高级别线上故障或隐私违规风险。 - **隐私事故曲线显著弯折**：随着拦截系统的运转，实际发生的隐私事故增长率，比原本缺乏该系统介入的模型预测基线低了大约五分之一（约 20%）。 - **度量成为系统内在闭环**：反事实度量不再仅仅是事后向领导层汇报的 PPT 工具，而是作为评估防护网络有效性的关键核心组件。 --- ### [19:22](https://www.youtube.com/watch?v=Pf10kLLcFR0&t=1162s) 未来三大范式转变与“能够被信任的速度” 在演讲尾声，Sophie 总结了 AI 驱动测试正在经历的三大深刻范式演进：首先，**测试正在从“持久资产”变为“即抛资产”（Disposable rather than durable）**，测试用例按单次 Diff 动态生成、高度靶向变更细节，并在完成使命后随时废弃；其次，系统虽然解决了单次信号的信任度，但**如何维持长期、深度的自动化信任**仍是下一阶段的硬仗；最后，**防御机制正在全面“左移”**，从代码合并前的审查关卡（Review gate）直接渗透进编写代码的交互闭环中，系统不再是站在对立面的代码审查者，而是并肩编写代码的“共同作者”（Co-author）。最后，Sophie 呼应了 Sreeja 开场提到的“跑 40 分钟测试套件顺便喝杯咖啡”的痛点，为现代工程交付与 AI 编程的终极关系下了定论：“And whatever you are building, the limit of how fast you can move isn't how fast the model writes your code. It's how confident you are that what it wrote is safe.”（“无论你正在构建什么，决定你能跑多快的上限，从来不是大模型写代码的速度，而是你对它所写内容是否安全的信心。”） - **消除周五上线的恐惧**：真正行之有效的防护体系，能让工程师在周五下午从容部署高风险变更而无需彻夜难眠——这绝非由于他们变得漫不经心，而是因为他们确信后台有一套严密运行的机制会稳稳托底。 - **重新定义研发速率**：缺乏信任的速度并不是真正的速度；如果每一次提交都如同一场不可控的豪赌，开发者就永远不可能真正跟上思维的迭代步伐。 ## Notable quotes > [19:22](https://www.youtube.com/watch?v=Pf10kLLcFR0&t=1162s) And whatever you are building, the limit of how fast you can move isn't how fast the model writes your code. It's how confident you are that what it wrote is safe. — Sophie Zeng > [10:14](https://www.youtube.com/watch?v=Pf10kLLcFR0&t=614s) A true positive ends you a little credit. A false positive spends a lot of it. Trust is won in drops and lost in buckets as they say. — Sreeja Thummala > [00:05](https://www.youtube.com/watch?v=Pf10kLLcFR0&t=5s) Landed test only protect against bugs we already know about. They don't go looking for new ones. So as an industry we are accelerating into an unknown territory with a rear view mirror. — Sreeja Thummala > [13:03](https://www.youtube.com/watch?v=Pf10kLLcFR0&t=783s) Once you thought an engineer that your tool is noise, you cannot really unteach it and there is no second launch. — Sreeja Thummala > [13:03](https://www.youtube.com/watch?v=Pf10kLLcFR0&t=783s) Propose hypothesis aggressively, cheaply in volume without worrying whether it's wrong and then let the judge decide which one are worthy of an engineer's attention. — Sreeja Thummala > [06:53](https://www.youtube.com/watch?v=Pf10kLLcFR0&t=413s) Every time we tell the generator a little bit more about the code change, the catch rate roughly doubled. — Sophie Zeng ## Mentioned Sophie Zeng, Sreeja Thummala, Meta, Stack Overflow, LLM [Source](https://www.youtube.com/watch?v=Pf10kLLcFR0)

Source quotes

[19:22] Sophie Zeng: And whatever you are building, the limit of how fast you can move isn't how fast the model writes your code. It's how confident you are that what it wrote is safe. [10:14] Sreeja Thummala: A true positive ends you a little credit. A false positive spends a lot of it. Trust is won in drops and lost in buckets as they say. [0:05] Sreeja Thummala: Landed test only protect against bugs we already know about. They don't go looking for new ones. So as an industry we are accelerating into an unknown territory with a rear view mirror. [13:03] Sreeja Thummala: Once you thought an engineer that your tool is noise, you cannot really unteach it and there is no second launch. [13:03] Sreeja Thummala: Propose hypothesis aggressively, cheaply in volume without worrying whether it's wrong and then let the judge decide which one are worthy of an engineer's attention. [6:53] Sophie Zeng: Every time we tell the generator a little bit more about the code change, the catch rate roughly doubled.

Source quotes

[17:39] Sophie Zeng: The limit of how fast you can move isn't how fast the model write your code. It's how confident you are that what it wrote is safe. [5:17] Sophie Zeng: Scoring is cheaper than generation; that's the trick, and getting that order right is the difference between a research demo versus a system you can afford to run in production every day. [15:21] Sreeja Thummala: Once you taught an engineer that your tool is noise, you cannot really unteach it and there is no second launch. [0:05] Sreeja Thummala: Landed tests only protect against bugs we already know about; they don't go looking for new ones, so as an industry we are accelerating into an unknown territory with a rear-view mirror.

Original transcript

[0:05] Welcome everybody. Welcome to our talk [0:07] on ship fast break nothing. [0:11] How many of you have shipped a change [0:14] this week that your test suite said was [0:17] fine. Keep that in mind because for the [0:21] last 50 years that sentence the test [0:24] passed has been one of the strongest [0:27] guarantee our industry had and it [0:30] quietly stopped meaning what it used to. [0:33] Early in my career I worked on a service [0:36] where pre-land test suite took about 40 [0:38] minutes. So it send you a change [0:41] request, go get a coffee, come back, [0:44] find out that 3,000 test suites have [0:46] tests have passed. none of which had [0:49] probably nothing to do with the 12 lines [0:51] I actually changed and I just ship it. [0:54] That was the process that was working [0:56] correctly [0:58] and that was also the deal that we made. [1:01] Writing code was slow and expensive. So [1:04] writing test to defend it was slow and [1:06] expensive and those two costs were in [1:09] balance. We hardened, we landed tests, [1:12] we built defenses against tomorrow's [1:14] regressions. [1:15] Some of you who have been doing it for a [1:17] decade or more knows exactly what I'm [1:20] talking about. Some of you who have been [1:22] started in last two years. This probably [1:25] sounds like I'm talking about punch [1:28] cards. [1:30] In recent times, the cost of writing [1:32] code collapsed. [1:35] Stack Overflow surveyed about 50,000 [1:39] developers. [1:40] 80% were using AI tools. 29% trusted the [1:45] output. Sit with it. [1:49] Adoption went up but trust went down. [1:53] That gap is the whole reason we are up [1:55] here [1:57] because nothing on the other side of [1:59] equation mode. Landed test only protect [2:03] against bugs we already know about. [2:06] They don't go looking for new ones. So [2:09] as an industry we are accelerating into [2:12] an unknown territory [2:14] with a rear view mirror. [2:17] So at Meta we asked a different [2:19] question. What if tests don't wait to be [2:22] written? What if the moment a risky [2:26] change appears [2:28] we generate tests whose only job is to [2:31] hunt for the changes bugs. [2:34] test that pass on the parent but fail on [2:37] the dip and prove with really high [2:40] confidence that the dip or the change is [2:43] what broke it and we would throw them [2:47] away. So there is no test speed to [2:49] maintain no technical D. This was [2:54] impossible before LLMs. The economics [2:56] did not work. Now they do. [3:00] We hear a lot of hype about phrase AI [3:02] NATO. What we want to show is what it [3:06] looks like apply with precision to a [3:09] very specific problem with real numbers [3:12] behind it because AI native quality is [3:16] what makes AI native velocity [3:18] survivable. [3:20] For the rest of the talk, Sophie is [3:23] going to walk you through how to [3:24] generate this cash test. I will come [3:28] back and tell you how we earned [3:30] engineering trust and what we still [3:33] can't prove and Sophie will close with [3:36] impact we have measured at Meta and [3:38] where this goes next. Sophie over to you [3:42] on generating cache test. Thanks Sua. [3:46] Before I take you into how we generate a [3:48] catch test, I want to put the whole [3:50] thing on screen once. Susha gave you the [3:53] idea. A test generated for one specific [3:56] change and can be thrown away. That's a [3:58] lovely idea. This is how we build it. AI [4:02] native means to us is AI in every single [4:06] step, not just one of them. And getting [4:09] there means answering four generally [4:11] uncomfortable questions. The first [4:14] question is there are so many divs [4:16] landed at the meta every day. We cannot [4:18] generate a test for all of them. So [4:21] which ones? [4:23] Second, say we have picked a div. What [4:27] does a test that catch bugs even look [4:29] like? What's the requirement? [4:32] Third, we generate a test. It fails. So [4:36] what? A failing test is not necessarily [4:38] a bug. How do we know we are looking at [4:42] a true positive [4:44] enforce? That is probably the most [4:46] challenging one. We are never going to [4:49] get to get false positives to zero. So [4:52] how do we hand a imperfect signal in in [4:56] front of busy engineers without burning [4:58] their trust? [5:00] Flag generate filter check. That's the [5:05] whole quality gate with AI running in [5:07] every step of it. I'll take the first [5:10] two and then SA will come back to tell [5:13] you more about the second two. [5:17] So which divs here is the scale of the [5:20] problem at meta we are talking about [5:23] hundreds of thousands of divs landing [5:26] every single day that's more than one [5:28] div per second around the clock and I [5:31] know some of you do more than that [5:33] because generating tests isn't free if [5:37] we run a gentic test generation on all [5:40] of the stiffs we may end up spending [5:43] extraordinary amount of compute just to [5:46] discover [5:47] The majority of them are actually fine. [5:51] So before we generate tests, we score. [5:55] Scoring is cheaper than generation. [5:58] That's the trick. And that getting that [6:00] order right is the difference between a [6:02] research demo versus a system you can [6:05] afford to run in production every day. [6:09] So what makes a train what makes a div [6:12] look risky? We look at many three [6:15] things. The first one is the content of [6:17] the change. What the diff says it does [6:20] and what the code actually does. If [6:22] there's a difference, that's an [6:24] interesting thing that we should look [6:26] into. Second is the radius of the blast, [6:29] whether it's touching a code that has a [6:32] history of incidents. Some part of the [6:34] codebase have earned our suspicion. And [6:37] then we look at the shape of the change. [6:39] how many lines of code was moved across [6:42] how many files in what code base. None [6:45] of that tells us what the bug is. It [6:48] tells us where is worth looking. [6:53] So we have picked out a div. What do we [6:56] generate? Two families of technology get [6:59] us a failing test. The first one is [7:02] property based. We take a invariance [7:05] that held before the change. say a end [7:08] point that required a approved [7:10] authorization policy and we check [7:13] whether that still hold after the change [7:16] where one doesn't that's our failing [7:18] test and it catches a quiet weakening of [7:22] existing control mutationbased goes the [7:25] other way we inject the fault that we [7:28] actually worry about then generate a [7:32] test that catches it traditional [7:34] mutation testing through hundreds of [7:37] mechanical fault in front of you to seek [7:39] to to pick from. We generate a handful [7:43] that we genuinely care about and keep [7:46] only the ones that nothing currently [7:48] catches. Then we write a test to kill [7:52] that mutant. [7:54] Three things had to happen to make [7:57] either of those work. Execution mode [8:00] tests used to be landed impermanent. [8:04] hours are generated on demand. Context [8:07] awareness. A traditional test doesn't [8:10] know about what the div is about, what [8:13] the change is going to do. Ours is built [8:16] for one specific div, and Oracle, which [8:20] is the deepest of the three. A [8:22] traditional generator is rewarded for [8:25] generating a passing test, the green [8:27] check mark. Ours is rewarded for [8:31] generating a failing test. The test that [8:34] pass on the parent of the commit but [8:37] fails on the commit. [8:40] And that the middle one context turns [8:43] out to matter more than anything else. [8:46] What you are going to see next is four [8:48] bars. Same system increasing amount of [8:52] contacts. [8:54] Watch what happens to the catch rate. [8:58] Our floor is coincidental catch. That is [9:02] a test that knew nothing about the [9:04] change and caught the bug by luck. [9:06] That's 2%. Adding hardening type of LM [9:10] generation 2%. 10 times better but still [9:14] blind to the change. We make it [9:17] differentware. Show the generator what [9:19] the code change is 4%. [9:23] Then we give the generator the intent as [9:26] well. That is what the div was trying to [9:29] do and what could possibly break 8%. [9:33] Forget about the numbers. Here's the [9:35] pattern. Every time we tell the [9:37] generator a little bit more about the [9:40] code change, the catch rate roughly [9:42] doubled. [9:44] That's what I mean about AI native. In [9:46] every step, the generator is only as [9:50] good as what it knows about the diff. [9:53] And working that out, what to tell it is [9:56] most of the job. [9:59] Which bring us to the harder part of the [10:01] pipeline and the challenge. A catch rate [10:05] is worth nothing if engineers don't [10:09] trust it. And SA is going to tell you [10:11] more about it. Thanks OP. So where we [10:14] are in the story, we can take a risky [10:17] dip, generate a test targeting that [10:19] specific change and prove that it passes [10:22] on parent and fails on the diff. And on [10:25] its own, it was not something that we [10:27] could put in front of a single engineer. [10:30] And that surprised us. We'd been [10:32] treating this as one problem, build a [10:35] platform that can catch bugs. But it's [10:38] two. [10:39] First one is can we catch bugs and [10:42] second one is how do we make an engineer [10:45] believe us that we caught a bug. The [10:50] second one turns out not to be a [10:52] modeling problem. It's more of a [10:54] judgment problem. And here's why a [10:58] generated test fails and the failure [11:00] could mean two completely different [11:02] thing. The first one generation itself [11:05] is faulty. The test asserted on [11:07] something that was never a promise [11:10] or it constructed a state a production [11:13] can never reach. That's generator's [11:15] fault and you can fix it by generating [11:17] better tests. But on the other case, [11:22] the one which we are going to talk about [11:24] it has nothing to do with generator [11:26] quality. Sometime the test generated is [11:29] good, the failure is real. [11:32] But the diff changes the behavior on [11:34] purpose. the other meant to do it and by [11:37] our own definition passes on the parent [11:40] and fails on the diff. That's a perfect [11:42] cats. [11:44] Both of these issues produce the same [11:47] identical signal. A red test and this is [11:51] nothing in test that can distinguish [11:54] them. And the volume the volume made it [11:57] much more acute problem. We generate [11:59] aggressively against a risky dip and the [12:02] overwhelming majority of what comes back [12:04] is later of those two things change that [12:07] other meant to make. [12:10] If you put that raw output in front of [12:12] engineers almost everything they saw [12:14] would have been noise. [12:17] The information that suppressed them [12:20] isn't in the code. It's what the author [12:22] was trying to do and you cannot generate [12:25] your way to it. [12:28] A true positive saves an engineer maybe [12:31] about a day of debugging. A false [12:33] positive cost them 10 minutes. So an [12:36] expected value, you could argue we could [12:38] have shipped it and eaten the noise. The [12:41] arithmetic is on our side. But that's [12:44] not how humans work. A true positive [12:47] ends you a little credit. A false [12:50] positive spends a lot of it. [12:53] Trust is worn in drops and lost in [12:55] buckets as they say. And the failure [12:58] mode for this is quieter than people [13:01] complaining. It's just that they stop [13:03] caring. The signal will stay there but [13:06] that no one would read it or act on it. [13:09] Once you thought and engineer that your [13:11] tool is noise, you cannot really unteach [13:14] it and there is no second launch. And we [13:18] knew that and so we didn't ship it. [13:21] We started thinking about what makes the [13:26] failure distinguishable from the test [13:29] failure test issue. [13:33] We asked the question [13:35] what kind of an information should we be [13:39] present should we need to classify the [13:43] test failure into something that other [13:46] meant to do do something the author does [13:49] not meant to do do we model this as a [13:53] judgment decision that could be given to [13:56] the LLM [13:58] the LLM judge is a taken system that [14:03] would now sit between generator [14:06] and the engineer. It looks at the [14:08] generator test change signal and [14:10] estimate the probability that failure is [14:14] an unintended defect rather than a [14:16] deliberate change. Only high confidence [14:20] signals then reach a human. And I want [14:23] to be very clear on how aggressive this [14:25] filter is because it's the part that [14:27] people find very counterintuitive. [14:30] The overwhelming majority of the test [14:32] this system generates never reach an [14:34] engineer. Most what we produce we throw [14:37] it away. This isn't generator failing. [14:41] That's generator doing its job. Propose [14:44] hypothesis aggressively, cheaply in [14:46] volume without worrying whether it's [14:48] wrong and then let the judge decide [14:52] which one are worthy of an engineer's [14:54] attention. That's the architecture. [14:56] generate widely, discard almost [14:58] everything and protect engineers [15:01] attention. [15:03] This effect is pretty wild. We realized [15:08] that about 70% reduction in time for [15:11] engineers triaging our bugs. Not because [15:14] we found fewer bugs, because we stopped [15:16] making them look at things that were [15:18] never bugs in first place. [15:21] And we changed the ask itself. We didn't [15:24] tell engineer your diff has a bug. [15:26] That's a claim. And the claims invite [15:29] argument. We ask in one question. This [15:32] used to be X. Now it's Y. Is that [15:34] expected? Seconds to answer. Nothing to [15:38] lose by dismissing it. Fast to verify, [15:41] easy to dismiss. One thing I find [15:44] genuinely interesting about where we [15:46] landed. Sophie mentioned that inferring [15:49] the intent of a dip doubled our catch [15:51] rate on generator side. Intent inference [15:54] also turned to be the only thing that [15:56] makes the output usable on delivery [15:58] side. Same missing variable both ends of [16:01] the pipeline. We just didn't realize it [16:04] until we hit it twice. [16:07] I want to be straight about two things [16:09] that we can't prove because they are [16:11] both about trust. The first we cannot [16:13] tell you how many real bugs we [16:15] discarded. We measure precision. We [16:18] don't measure recall because recall here [16:20] means counting bugs that never shift. [16:23] We have accepted that blind spot and [16:26] that's a real limitation. It's [16:28] deliberate and it's honestly the most [16:30] interesting problem that we have left to [16:33] solve. Which brings me to the second [16:36] thing because it's the same problem one [16:38] level different. [16:40] Engineers trust the system one signal at [16:43] a time but leadership needs to trust the [16:46] aggregate. [16:49] How? But how do you measure something [16:51] that you cannot observe? Every platform [16:55] team in the industry which does [16:58] prevention hits the same wall. [17:01] So you are asking people to find [17:03] something on page and they're right to [17:05] be skeptical. What broke that help for [17:09] us was counterfactual measurement. We [17:12] can show the trend line would have [17:14] looked like without this framework [17:16] against what it actually was. [17:19] That's the evidence that landed with [17:21] every technical team and leadership we [17:23] have talked to. That's why we treat [17:25] measurement as part of the system rather [17:28] than reporting after that. Which is [17:30] exactly where Sophie will speak about [17:34] next. The results how we measured them [17:37] and what's still ahead. Sophie, thanks [17:39] Suja. So does it work? I bet you are all [17:44] wondering how we measured this and what [17:46] the impact actually was. And I'm excited [17:49] to say that not only have we seen some [17:52] real results, take a look at the [17:54] incident prevented. [17:58] 74. [17:59] These aren't l warnings. Every one of [18:02] them is a self or a incident that did [18:06] not happen. And now I want to be honest [18:09] with you about what this number is and [18:11] what isn't. [18:13] Here is a hard thing about prevention [18:14] work. Nobody can see what didn't happen. [18:18] When we started showing this results [18:21] internally, the first reaction from [18:23] engineers and leadership was reasonable [18:26] doubt. We were asking people to believe [18:29] in a absence. [18:32] So we stopped only counting catches and [18:35] we look at the trend line instead with [18:37] catching running privacy instant growth [18:40] came in about one fix lower than our own [18:44] forecast without it. [18:47] I'll be straight with you. The forecast [18:49] is a model not a measurement. [18:52] Counterfacturers are arguable. But that [18:55] is the thing that moved the conversation [18:58] because it's the difference between we [19:01] caught some bucks versus the curve debt [19:06] which is really two different kinds of [19:08] trust. Engineers had to trust the signal [19:11] resurface on their diffs and leadership [19:14] had to trust the aggregated impact [19:16] number. You need both and they're not [19:19] the same problem. [19:22] So that's what we have proven. There is [19:25] a great deal more ahead and I want to [19:28] close on three shifts that are already [19:30] underway. Tests are becoming disposable [19:33] rather than durable. generated per div [19:36] targeted at a specific code change and [19:39] kept only when they are needed. That one [19:41] is already true. Catching is solving [19:43] enough to trust, but trusting it long [19:46] term is the next challenge. And catching [19:50] is moving left out of the review gate [19:53] into the loop where the code is actually [19:56] written. So the system is no longer a [19:59] reviewer of your code, but a co-author. [20:03] If that last one sounds like the problem [20:05] you want to work on, we'd like to hear [20:07] from you. [20:10] You have heard a lot from Sua and [20:12] myself. And if there is one thing I want [20:15] you to remember today is this. [20:19] Sua started us with a 40 minutes test [20:22] sweet and a coffee break. We have gained [20:25] tremendous speed since then. But speed [20:29] you cannot trust isn't really speed. And [20:32] you can move at the speed of ideas if [20:34] every div is a gamble. [20:37] What that looks like when it works. An [20:40] engineer ships a risky change on a [20:42] Friday afternoon and doesn't lie awake [20:45] about it. Not because they've stopped [20:47] the caring, but because something [20:49] running behind them will catch it. [20:53] And whatever you are building, the limit [20:56] of how fast you can move isn't how fast [20:59] the model write your code. It's how [21:02] confident you are that what erodess is [21:04] safe. [21:06] Thank you.