Graphite Picks Claude Over GPT-5 as AI Review Acceptance Beats Humans
Ai4
As generative coding shifts the software bottleneck to pull requests, Graphite finds Claude 3.7 Sonnet outperforms OpenAI models in production code review by aggressively suppressing false alarms to beat human acceptance rates.
## Background
Published on YouTube by Ai4 on August 28, 2025, this talk features Graphite CTO Greg Foster discussing how the company evaluates LLMs for code comprehension. According to the page description, Foster outlines Graphite's evaluation framework for models like Claude, Gemini, and GPT in real-world code review workflows, highlighting matched comment (recall) and unmatched comment (precision) scores used to build a high-signal reviewer.
## TL;DR
As AI code generation accelerates developer authoring speed, code review and pipeline management have become the central bottleneck in software engineering. Graphite developed Diamond, an AI code reviewer engineered to prioritize precision over noise by requiring 90% to 95% confidence before posting inline comments, achieving an acceptance rate of 52% that surpasses human baselines. Through extensive real-world benchmarks on Braintrust across models like GPT-5, GPT-4o, Gemini Pro, and Claude, Graphite identified Claude 3.7 Sonnet as the top production performer due to its optimal balance of speed, reasoning, and tool use. Looking beyond diff review, Graphite is also expanding outer-loop automation to parse thousands of lines of CI failure logs and generate one-click inline fixes.
## Atmosphere & style
The presentation carries a grounded, engineering-first pragmatism that strips away generative AI hype in favor of hard-won production lessons. Foster speaks with candid, approachable authenticity, punctuating technical discussions of benchmark evals with relatable developer realities like GitHub 'flame wars,' Saturday morning model testing rituals, and early LLMs writing 'high school essays' on pull requests. The tone feels like an earnest, behind-the-scenes engineering post-mortem that balances technical rigor with an appreciation for the daily friction software developers face.
## Key takeaways
- [00:00](https://www.youtube.com/watch?v=WDUxjZic4r4&t=0s) AI code generation is rapidly accelerating the developer inner loop (writing and debugging in IDEs), shifting the primary engineering bottleneck to the outer loop (code review, CI, and deployment).
- [04:06](https://www.youtube.com/watch?v=WDUxjZic4r4&t=246s) Early automated code reviews failed due to narrow context windows and subjective, essay-like summaries, necessitating a shift toward precise, high-confidence inline comments on PR diffs.
- [04:06](https://www.youtube.com/watch?v=WDUxjZic4r4&t=246s) Graphite's AI reviewer, Diamond, has scaled across thousands of organizations like Snowflake and Duolingo, evaluating over 1.6 million PRs and generating more than 500,000 inline comments.
- [07:08](https://www.youtube.com/watch?v=WDUxjZic4r4&t=428s) Modern applied AI development has shifted away from fine-tuning and complex RAG pipelines toward assembling massive evaluation datasets of real-world diffs and human comments.
- [07:08](https://www.youtube.com/watch?v=WDUxjZic4r4&t=428s) To preserve developer trust, Diamond aggressively filters out noise by vetoing comments with 50% confidence and requiring 90% to 95% confidence before publishing.
- [08:40](https://www.youtube.com/watch?v=WDUxjZic4r4&t=520s) Code review utility is benchmarked against human baselines using acceptance rate (whether flagged code is changed pre-merge) and a guardrail metric of keeping negative emoji reactions under 4%.
- [08:40](https://www.youtube.com/watch?v=WDUxjZic4r4&t=520s) Graphite's AI reviewer achieves a 52% acceptance rate, outperforming the historical 45% to 50% human review baseline on the platform.
- [11:11](https://www.youtube.com/watch?v=WDUxjZic4r4&t=671s) Model performance is evaluated across two core metrics: matched comment score (recall on PRs with confirmed bugs) and unmatched comment score (precision and silence on clean PRs).
- [11:11](https://www.youtube.com/watch?v=WDUxjZic4r4&t=671s) Frontier model benchmarks reveal sharp trade-offs: GPT-5 excels at complex bug detection but produces noise on clean PRs, GPT-4o suppresses noise but misses bugs, and Gemini Pro provides cheap 1M token contexts with lower overall composite scores.
- [14:24](https://www.youtube.com/watch?v=WDUxjZic4r4&t=864s) Claude 3.7 Sonnet serves as Graphite's primary production model due to its balance of speed, multi-file codebase reasoning, and low noise relative to models like Sonnet 4.
- [16:51](https://www.youtube.com/watch?v=WDUxjZic4r4&t=1011s) Adopting newly released foundation models requires significant engineering overhead to re-optimize 10 to 20 distinct pipeline prompts to match each architecture's nuances.
- [17:25](https://www.youtube.com/watch?v=WDUxjZic4r4&t=1045s) Graphite is expanding beyond inline reviews to automate CI build debugging by summarizing verbose error logs and generating one-click inline PR fixes.
## Detailed walkthrough
### [00:00](https://www.youtube.com/watch?v=WDUxjZic4r4&t=0s) The AI Code Explosion and the Inner vs. Outer Loop
AI code generation is producing unprecedented volumes of code, fundamentally shifting the engineering bottleneck to the review side. As generative AI accelerates authoring speed, development teams face mounting pressure to validate, integrate, and keep the software delivery pipeline safe. Industry leadership across major tech companies reflects this rapid transition: GitHub Copilot reported over 30% of code being AI-generated, Google reported roughly 30% of internal new code is AI-generated (up from 25%), Dario Amodei of Anthropic aggressively predicted 90% of new software would be AI-generated, and Microsoft CTO Kevin Scott claimed that 100% of code would be written by AI within five years. Regardless of the exact percentage, software development is bifurcated into two distinct operational loops:
```mermaid
flowchart TD
accTitle: Inner Loop vs Outer Loop in Software Development
subgraph InnerLoop["Inner Loop (Shrinking)"]
IDE["IDE / Terminal / Cursor / Claude Code"] --> CodeGen["AI Code Generation & Autocomplete"]
CodeGen --> Debug["Local Building & Debugging"]
end
The inner loop represents the authoring environment—IDEs, terminals, Claude Code, Cursor, or Vim—where developers write, build, and debug code. This loop is collapsing rapidly in duration, shrinking tasks that previously took five to ten hours down to ten minutes or offloading them entirely to background agents while engineers grab lunch. In contrast, the outer loop remains largely untouched by standard code-generation tools. This loop encompasses the collaborative and operational overhead: opening pull requests, drafting descriptions, assigning reviewers, passing CI test suites and linters, conducting thorough reviews for bugs and security vulnerabilities, executing merge strategies (such as feature branching or trunk-based development managed via merge queues), and handling deployment, monitoring, and rollbacks. Graphite focuses on accelerating and safeguarding this outer loop.
- **Copilot & Google Metrics:** Approximately 30% of new code is reported as AI-assisted or generated.
- **Industry Projections:** Leadership estimates range from 90% in the near term (Anthropic) to 100% within five years (Microsoft).
- **Core Thesis:** Accelerating code generation without accelerating code review creates a severe deployment bottleneck.
---
[04:06](https://www.youtube.com/watch?v=WDUxjZic4r4&t=246s) The Evolution of AI Code Review and the Birth of Diamond
Graphite’s exploration of automated code review began years earlier with GPT-3 APIs, but early attempts encountered fundamental technical limitations. First, restrictive context windows of 2,000 to 4,000 tokens limited input diffs to only a few hundred lines of code, making it impossible to evaluate the large, complex PRs where engineers needed review assistance the most. Second, early models produced verbose, subjective feedback: "they would write me like a high school essay about my pull request and they would give me all this like subjective feedback." This added hundreds of lines of prose for engineers to read without providing actionable fixes.
A Saturday morning ritual emerged of re-evaluating new model drops—progressing through GPT-3.5, GPT-4, and the Claude 3.5 and 3.7 Sonnet families. Once models attained wide context windows and sufficient reasoning capabilities, automated code review became practical. Graphite launched **Diamond**, an AI code reviewer engineered to avoid generic summary-level commentary in favor of precise, high-confidence inline comments on specific lines of code.
```mermaid
flowchart LR
accTitle: Diamond Code Review Adoption and Scale
A["Pull Request Diff"] --> B["Diamond AI Reviewer"]
B --> C["1.6M+ PRs Reviewed"]
B --> D["500K+ Inline Comments Left"]
B --> E["Enterprise Deployment<br>(Snowflake, Duolingo)"]
```
Diamond has scaled across thousands of organizations, including Snowflake and Duolingo, processing tens of thousands of PRs daily. This scale amplifies the risk of model drift or regressions, as even a minor prompt tweak can cause hallucinations or noisy feedback that immediately degrades developer trust.
- **Scale to Date:** Over 1.6 million pull requests evaluated and more than 500,000 inline comments generated.
- **Form Factor Shift:** Transitioned away from summary essays to discrete, actionable inline diff feedback.
- **Developer Scrutiny:** Maintaining developer trust requires near-zero tolerance for unhelpful or noisy comments.
---
[07:08](https://www.youtube.com/watch?v=WDUxjZic4r4&t=428s) Prioritizing Signal-to-Noise Ratio and Modern LLM Evals
To maintain developer trust, Diamond operates on an aggressive signal-to-noise filter. Rather than attempting to catch every conceivable edge case or bug across every PR, the system prioritizes precision: if internal confidence in a comment is at 50%, it is vetoed and withheld. The system only posts when confidence reaches 90% to 95%, ensuring that posted feedback is useful, actionable, and directly leads to code revisions.
This operating philosophy reflects how applied AI engineering in 2025 differs from traditional machine learning workflows. Classical ML relied heavily on custom Python feature engineering and hyperparameter tuning, with evaluations treated as an afterthought at the end of the pipeline. In contrast, modern production systems rely on off-the-shelf foundation models from Anthropic and OpenAI. Techniques like fine-tuning have yielded limited improvements, and retrieval-augmented generation (RAG) has become less essential as LLMs improved at native tool calling and file globbing. The central engineering effort has shifted almost entirely to building comprehensive, real-world evaluation suites—assembling datasets of hundreds or thousands of real production diffs paired with human review comments to continuously benchmark and validate review accuracy.
- **High Precision Threshold:** Comments with ~50% confidence are vetoed internally; 90–95% confidence is required to publish.
- **Evaluation-Driven Development:** Focus shifts from model fine-tuning and RAG architectures to massive diff-and-comment evaluation datasets.
- **Objective:** Maximize developer action rate rather than maximizing raw comment volume.
[08:40](https://www.youtube.com/watch?v=WDUxjZic4r4&t=520s) Real-World Evals with Braintrust and the Acceptance Rate Metric
To determine whether updated prompts, refined pipelines, or newly shipped models actually improve code review quality, Graphite built an extensive evaluation harness using Braintrust. Giving a shout-out to the Braintrust team present at the conference, the setup relies on thousands of real-world pull requests gathered from both open-source repositories and Graphite's internal engineering codebases. These test cases span clear bugs, syntax errors, and security vulnerabilities, as well as deceptive "red herring" pull requests containing no actual defects. For clean pull requests, the explicit evaluation target is for the model to "shut up" and leave zero comments rather than hallucinating superficial issues.
```mermaid
flowchart LR
accTitle: PR Evaluation Pipeline
subgraph Input["Test Suite"]
PR["PR Dataset<br/>(Open-source & Internal)"]
end
subgraph Eval["AI Review Evaluation"]
direction TB
Bug["Buggy PRs<br/>(Catch Syntax / Security)"]
Clean["Clean PRs<br/>(Verify Zero Noise)"]
end
subgraph Metrics["Core Benchmarks"]
AR["Acceptance Rate<br/>(Code Changed Pre-Merge)"]
GR["Downvote Guardrail<br/>(< 4% Negative Reactions)"]
end
PR --> Bug
PR --> Clean
Bug --> AR
Clean --> GR
```
To quantify real-world utility, Graphite measures the *acceptance rate*: the probability that an inline comment directly causes the flagged line of code to be modified before the pull request merges. Drawing from observations across hundreds of millions of PRs processed on Graphite's broader platform, human review comments carry a baseline acceptance rate of 45% to 50%—with the remaining balance chalked up to compliments or online "flame wars" where the author simply closes the thread. Graphite's AI reviewer currently outperforms this human baseline at a 52% acceptance rate, while maintaining a guardrail metric of under 4% negative emoji reactions across GitHub and Graphite interfaces.
- **Human baseline acceptance rate:** 45%–50% of human review comments lead to subsequent code updates.
- **Graphite AI acceptance rate:** ~52% of AI comments result in actionable code improvements.
- **Negative feedback guardrail:** Downvote emoji reactions on comments remain below 4%.
---
[11:11](https://www.youtube.com/watch?v=WDUxjZic4r4&t=671s) Evaluating Frontier Models: GPT-5, GPT-4o, o3, and Gemini Pro
Graphite evaluates every major frontier model against two foundational metrics: *matched comments* (recall on pull requests with confirmed bugs) and *unmatched comments* (precision and suppression on clean pull requests). Having tested the newly released GPT-5 over the preceding weekend, the team found it was the strongest model at identifying genuine bugs and executing deep reasoning. However, GPT-5 exhibited noticeable noise by flagging non-issues on clean PRs and ran significantly slower than Anthropic's offerings. While developer workflows in editors like Cursor or Claude Code require near-instant feedback, code review affordances are more forgiving: against industry medians where human reviews take 18 hours (and 3 hours at best), an AI review taking a couple of minutes remains comfortably faster than standard 10- to 15-minute CI pipelines.
OpenAI's earlier offerings and competing frontier models showed distinct trade-offs in the review pipeline. GPT-4o proved adept at remaining quiet on clean pull requests (scoring well on unmatched comments) but struggled to reliably surface actual bugs. The OpenAI o3 reasoning model showed capable logic but was hampered by slow execution, high token expense, and mediocre code comprehension. Meanwhile, Google's Gemini Pro offered an attractive price point and an expansive 1-million-token context window capable of ingesting massive 2,000 to 3,000-line PRs, but suffered from slower response times and lower composite quality on matched and unmatched evaluations.
[14:24](https://www.youtube.com/watch?v=WDUxjZic4r4&t=864s) Claude Sonnet in Production and the Precision-Recall Trade-off
Anthropic's Claude 3.7 Sonnet stands as Graphite's primary production model, chosen strictly for empirical performance rather than vendor loyalty. Modern AI code review extends beyond isolated diff analysis: the reviewer acts as an agent equipped with tool use to search the broader codebase and inspect historical pull requests or previous discussion threads to deduce whether code changes introduce subtle regressions. While Sonnet 4 demonstrated remarkable tool-use capabilities and strong reasoning, it proved considerably noisier than Sonnet 3.7—generating nearly twice as many comments without a proportional increase in quality.
This divergence highlights the core tension in automated code review: balancing high recall (catching every conceivable edge case) against high precision (delivering only high-signal, zero-fluff feedback). While Graphite currently tunes this boundary globally, future iterations aim to let developers configure their own tolerance slider between noisier, exhaustive auditing and conservative, high-signal reviews. In standardized benchmark comparisons, Sonnet 3.7 led the pack, with models like o3, Gemini, and GPT-5 clustering above the 60% quality threshold. However, running identical prompts across all models introduces an inherent benchmark bias, as newly released models like GPT-5 require dedicated prompt engineering and tuning within the eval suite to reach peak performance.
- **Production choice:** Claude 3.7 Sonnet remains the most reliable, accurate, and balanced model for live reviews.
- **Sonnet 4 behavior:** Excelled at agentic codebase navigation but generated nearly 2x the comment volume, increasing noise.
- **Benchmark threshold:** Top-tier models (Sonnet 3.7, o3, Gemini, GPT-5) scored above 60% on combined matched/unmatched evals.
- **Evaluation limitation:** Zero-shot cross-model evaluations run identical prompts, penalizing models that require tailored prompt structures.
[16:51](https://www.youtube.com/watch?v=WDUxjZic4r4&t=1011s) The Prompt Engineering Overhead and Current Model Landscape
Navigating the AI landscape requires significant ongoing effort due to how rapidly foundation models evolve. New frontier models launch roughly every three months, and adapting to them is far from plug-and-play. To accurately evaluate and adopt a newly released model, teams must meticulously revisit 10 to 20 different pipeline prompts and re-optimize them to match the new architecture's strengths. While Graphite maintains this rigorous testing cadence, the engineering overhead is "non-trivial and you have to be kind of thoughtful about running those experiments."
Summarizing the current benchmarks for code review, Claude holds the lead in execution speed and is roughly tied with competing frontier models on deep code understanding. If Claude were off the table, Graphite would likely deploy a hybrid mixture of Gemini and GPT. Looking ahead, the rapid pace of model development is expected to continue shaking up benchmarks, with future foundation models—such as upcoming iterations of Gemini—positioned to potentially "dominate the field later this year."
- **Prompt Maintenance:** Updating pipelines across 10–20 distinct prompts is required every few months to properly evaluate and tune for new model releases.
- **Current Model Standings:** Claude leads in generation speed and ties on code comprehension, with Gemini and GPT serving as the primary alternatives.
```mermaid
flowchart TD
accTitle: CI Failure Resolution Flow
subgraph ManualFlow["Traditional Manual Debugging"]
A["CI Pipeline Fails"] --> B["Scroll through 2,000+ log lines"]
B --> C["Ctrl+F for error line"]
C --> D["Check out branch locally"]
D --> E["Apply fix & push new commit"]
end
subgraph AIFlow["Graphite Automated Flow"]
F["CI Pipeline Fails"] --> G["AI extracts & summarizes error"]
G --> H["AI analyzes diff & generates inline fix"]
H --> I["Developer approves with one click"]
end
```
[17:25](https://www.youtube.com/watch?v=WDUxjZic4r4&t=1045s) Beyond Code Review: Closing the Loop on CI Failures
Graphite is extending its AI tooling beyond inline pull request comments to streamline the broader "outer loop" of software development. A major focus—developed heavily by team member Michael—is addressing the friction around Continuous Integration (CI) build failures. In a standard workflow, a failed build forces an engineer to sift through 2,000 lines of irrelevant console output, scroll all the way to the bottom, and search for the failure point just to realize they introduced a minor bug into their PR.
Graphite uses LLMs to parse those voluminous log files, accurately summarize the root cause, and correlate the error with the PR diff to suggest an inline code fix. Instead of requiring the developer to pull the branch locally, run the test suite, and manually commit a one-line correction, the interface offers a single-button action to apply the update directly. This workflow automates tedious log parsing while keeping the engineer firmly in control of what code gets merged.
- **Automated Failure Summarization:** Filters out thousands of lines of log noise to immediately surface the underlying error.
- **Diff-Aware Inline Fixes:** Generates context-aware fixes directly in the PR interface for one-click resolution without local context switching.
## Notable quotes
> [04:06](https://www.youtube.com/watch?v=WDUxjZic4r4&t=246s) they would write me like a high school essay about my pull request and they would give me all this like subjective feedback. — Greg Foster
> [08:40](https://www.youtube.com/watch?v=WDUxjZic4r4&t=520s) shut up — Greg Foster
> [16:51](https://www.youtube.com/watch?v=WDUxjZic4r4&t=1011s) non-trivial and you have to be kind of thoughtful about running those experiments. — Greg Foster
> [16:51](https://www.youtube.com/watch?v=WDUxjZic4r4&t=1011s) dominate the field later this year. — Greg Foster
## Mentioned
Ai4, Anthropic, Braintrust, Claude 3.5, Claude 3.7 Sonnet, Claude Code, Cursor, Dario Amodei, Diamond, Duolingo, Gemini, Gemini Pro, GitHub Copilot, Google, GPT, GPT-3, GPT-3.5, GPT-4, GPT-4o, GPT-5, Graphite, Greg Foster, Kevin Scott, Michael, Microsoft, OpenAI, OpenAI o3, Snowflake, Sonnet 4, Vim
## Video description
Greg Foster, CTO, Graphite
Abstracts
Not all AI models are created equal, especially when it comes to understanding code. This talk breaks down how we evaluate models for code comprehension at Graphite, focusing on their performance in real-world code review workflows. I’ll explain the two core metrics we rely on: matched comment score (recall) and unmatched comment score (precision), and how these metrics guide us in building a high-signal, low-noise code reviewer. You’ll learn how models like Claude, Gemini, and GPT compare, and what actually makes a model useful in production. If you're building with or evaluating LLMs for software engineering, you’ll leave with a practical framework for assessing model quality.
As generative AI accelerates software authoring, Graphite argues that engineering bottlenecks have shifted to pull request review and operational integration, where automated review tools must prioritize high precision over comment volume to preserve developer trust.
The AI Authoring Surge and the Outer Loop Bottleneck (0:00)
Engineering workflows are dividing into two tracks: a shrinking inner loop of local authoring and an outer loop comprising pull requests, continuous integration (CI), peer review, and deployments. With GitHub Copilot and Google reporting roughly 30% of new code being AI-generated, and executives like Anthropic's Dario Amodei and Microsoft CTO Kevin Scott projecting 90% to 100% within five years, authoring tasks have collapsed from hours to minutes. However, because code generation tools do not resolve operational review overhead, accelerating generation without automating review creates a severe deployment bottleneck.
Precision Thresholds and Real-World Evaluation Frameworks (7:08)
Automated code review evolved past early token limits and essay-like summaries into systems like Graphite's Diamond, which has processed over 1.6 million pull requests across enterprises such as Snowflake and Duolingo. Graphite found that fine-tuning and retrieval-augmented generation offered limited gains compared to off-the-shelf foundation models equipped with real-world evaluation datasets. Because developer trust degrades quickly under noisy feedback, Diamond enforces a strict precision threshold: review comments with roughly 50% internal confidence are withheld, publishing only when confidence reaches 90% to 95%.
Using an evaluation harness built on Braintrust, Graphite tests models on thousands of real-world pull requests, including clean "red herring" changes where the explicit goal is to post zero comments. Real-world utility is measured via acceptance rate—the frequency with which an inline comment causes the author to update code before merging. While human review comments average a 45% to 50% baseline acceptance rate, Graphite reports its AI reviewer achieves 52% while keeping negative emoji reactions under 4%.
Benchmarking Frontier Models Across the Precision-Recall Frontier (11:11)
Empirical evaluations highlight clear operational trade-offs across frontier architectures. In testing, GPT-5 detected genuine bugs and displayed strong reasoning but produced noisy comments on clean pull requests and ran slowly. OpenAI's o3 suffered from high token costs and mediocre code comprehension, while GPT-4o suppressed false alarms well but struggled to catch real defects. Google's Gemini Pro provided competitive pricing and a 1-million-token context window suited for large diffs, though it yielded lower composite quality scores and slower response times.
Graphite uses Anthropic's Claude 3.7 Sonnet in production because it best balances speed, multi-file codebase search, and comment precision. While Sonnet 4 exhibited strong agentic reasoning, it generated nearly twice as many comments without a matching quality gain. Graphite notes that standard zero-shot benchmarks run identical prompts across models, which inherently penalizes newer architectures like GPT-5 until engineering teams spend substantial effort re-tuning pipeline prompts.
Expanding Automation to CI Failure Remediation (17:25)
Beyond diff review, Graphite is extending LLMs to address operational friction in continuous integration pipelines. A standard CI failure often forces engineers to manually comb through 2,000 lines of console output to locate an error, switch branches locally, and push a small fix. By deploying models to ingest verbose logs, summarize the root cause, and correlate errors with pull request diffs, the system generates inline patches that engineers can review and apply with a single click.
Maintaining production-grade AI code review requires recurring prompt engineering across 10 to 20 distinct pipeline stages every few months as frontier architectures update. While Claude currently leads in production deployment, Graphite notes that rapid foundation model iterations could quickly shift cross-model performance benchmarks.
## Background
本内容由 YouTube 频道 Ai4 于 2025 年 8 月 28 日发布。视频为 Graphite 联合创始人兼首席技术官 Greg Foster 的主题演讲，主要探讨在真实代码审查（Code Review）工作流中如何评估大语言模型对代码的理解能力，并分享匹配评论得分与未匹配评论得分等核心评测指标与模型横向对比经验。
## TL;DR
随着 AI 代码生成让开发“内环”急剧提速，代码审查、验证与合并等“外环”环节正面临严重的吞吐瓶颈。Graphite 团队打造了 AI 代码审查器 Diamond，放弃无意义的宏观总结，专注于高置信度（90%-95%）的行内批注，并以代码修改采纳率作为北极星指标（当前达 52%，已超越人类基准）。通过构建包含数千真实 PR 的评测集（Evals），团队对 Claude、GPT-5、Gemini 及 o3 等模型在召回与噪音控制等维度进行了深度实测与对比。此外，Graphite 还将这一能力扩展至 CI 失败自动诊断与一键修复，致力于全方位重塑软件工程的外环闭环。
## Atmosphere & style
演讲展现出极为严谨、务实且兼具前瞻性的工程实战风格。Greg Foster 摒弃了行业中常见的宏大商业宣传，直击开发者日常痛点与生产环境部署大模型的脆弱性，言语间充满硬核工程师的冷静与克制。整场分享节奏清晰、数据详实，通过层层展开的图表与横向对比，带给观众一种在现代 AI 工程前沿“用数据说话、以评测驱动”的扎实信赖感与洞察力。
## Key takeaways
- [00:00](https://www.youtube.com/watch?v=WDUxjZic4r4&t=0s) AI 代码生成的爆发极大压缩了代码编写的开发内环，将工程瓶颈转移到了代码审查、CI 测试与集成的开发外环环节。
- [02:04](https://www.youtube.com/watch?v=WDUxjZic4r4&t=124s) 开发外环（提交 PR、同行审查、合并队列与发布监控）是保障工程质量与系统稳定的核心，当前代码生成工具大多难以触及该痛点。
- [03:53](https://www.youtube.com/watch?v=WDUxjZic4r4&t=233s) Graphite 彻底放弃了宽泛的概括性总结评语，全面押注技术难度更高但实用价值更强的行内精准批注（Inline Comments）。
- [06:04](https://www.youtube.com/watch?v=WDUxjZic4r4&t=364s) 生产级 AI 审查器必须追求极致的信噪比：系统在拥有 90% 到 95% 置信度时才会发送建议，宁缺毋滥以维系开发者信任。
- [06:04](https://www.youtube.com/watch?v=WDUxjZic4r4&t=364s) 现代大模型工程从传统的特征工程和微调转向以评测体系（Evals）为驱动，通过海量真实 Diff 构建的黄金测试集进行回归验证。
- [08:40](https://www.youtube.com/watch?v=WDUxjZic4r4&t=520s) 评测基准同时覆盖缺陷样本（检验找错召回率）与无缺陷/红鲱鱼样本（检验模型克制发声、避免噪音告警的能力）。
- [09:44](https://www.youtube.com/watch?v=WDUxjZic4r4&t=584s) AI 行内评论促成代码修改的采纳率已达到约 52%，超越了人类评审员 45%-50% 的平均基准线，同时将点踩率压至 4% 以下。
- [11:11](https://www.youtube.com/watch?v=WDUxjZic4r4&t=671s) 横向评测中 Claude 3.7 Sonnet 综合信噪比最佳，GPT-5 找错推理强但偏啰嗦嘈杂，Gemini Pro 则在百万 Token 大上下文分析大型 PR 上优势明显。
- [11:11](https://www.youtube.com/watch?v=WDUxjZic4r4&t=671s) 后台代码审查对数分钟的延迟有较高容忍度，现代代码审查更依赖模型通过 Tool Calling 遍历代码库与历史 PR 进行综合判断。
- [17:25](https://www.youtube.com/watch?v=WDUxjZic4r4&t=1045s) AI 能力进一步延伸至 CI 失败自动诊断，通过提炼海量日志错误并结合 Diff 生成一键应用修复方案，打通外环闭环体验。
## Detailed walkthrough
### [00:00](https://www.youtube.com/watch?v=WDUxjZic4r4&t=0s) AI 代码生成浪潮与代码审查的瓶颈
作为 Graphite 的联合创始人兼首席技术官，Greg Foster 开门见山地指出了软件工程正在经历的剧烈范式转移。Graphite 一直致力于打造业内领先的代码审查（Code Review）协作平台，而在 AI 蓬勃发展的当下，这一使命变得前所未有的紧迫。AI 正在以史无前例的速度生成海量代码，这直接将软件交付的压力转移到了审查、验证和集成的环节。如果无法保障审查流程的安全与高效，代码审查必然会沦为整个开发周期的首要瓶颈。要跟上 AI 代码生成（Codegen）所释放的生产力，审查端的速度也必须实现数量级的跃升。
关于“当今究竟有多少代码是由 AI 生成的”这一问题，业界领袖们给出了各种激进的预测与统计数据：
- GitHub Copilot 团队在一两年前便声称已有 30% 或更多的代码由 AI 生成，微软 CEO Satya Nadella 也给出过类似的判断。
- Google 近期披露其内部新代码中约有 30% 是由 AI 生成的，较此前的 25% 进一步上升。
- Anthropic 的 Dario Amodei 曾更大胆地预测，今年新编写的软件中有 90% 将由 AI 生成。
- 微软 CTO Kevin Scott 更是断言，未来 5 年内这一比例将达到 100%。
尽管科技巨头的高管们在商业上倾向于抛出极具颠覆性的宏大口号，但无论实际占比是 10%、30% 还是 90%，大量代码正通过自动补全、AI 辅助或无头 Agent（Headless Agents）自动生成已是不争的事实。
```mermaid
flowchart TD
accTitle: 软件开发内环与外环对比
accDescription: 展示开发者编写代码的内环与代码提交、审查、合并上线的开发外环的区别。
subgraph InnerLoop["开发内环 (Inner Loop) - 正在被 AI 极速压缩"]
A["打开 IDE / 终端 / Cursor / Claude Code"] --> B["代码编写、构建与本地调试"]
B --> C["原本耗时数小时 -> 如今缩减至 10 分钟或后台 Agent 完成"]
end
subgraph OuterLoop["开发外环 (Outer Loop) - 决定工程质量与安全的核心"]
D["提交 Pull Request / 填写说明与标记审查者"] --> E["CI / Linter / 单元测试流水线"]
E --> F["同行代码审查 (安全漏洞/Bug/业务逻辑校验)"]
F --> G["合并策略 (Trunk-based / Merge Queue)"]
G --> H["发布部署、监控、回滚保障"]
end
InnerLoop --> OuterLoop
```
---
### [02:04](https://www.youtube.com/watch?v=WDUxjZic4r4&t=124s) 软件工程的内环与外环 (Inner Loop vs Outer Loop)
面对代码生成的爆发，Greg Foster 将软件工程师的日常工作解构为两个核心闭环：**内环（Inner Loop）**与**外环（Outer Loop）**。
内环是当前所有代码生成工具疯狂卷入的领域。它涵盖了开发者打开 IDE 或终端（无论是使用 Cursor、Claude Code，还是仍然坚持用 Vim 手写），在本地敲出代码、构建并调试直至功能跑通的过程。这个内环正在急剧收缩——过去需要花费 5 到 10 个小时编写的代码变动，如今在工具辅助下可能仅需 10 分钟；甚至开发者只需触发一个后台 Agent，外出吃个午饭回来，代码就已经静静地躺在编辑器里了。
然而，真正能保住工程师饭碗、且目前仍未被 AI 彻底解决的，是**开发外环**。外环始于提交 Pull Request（PR），包括撰写 PR 标题与描述、指派审查人员、等待公司的 CI 流水线运行、代码检查器（Linter）和单元测试的跑通。紧接着是高标准的严谨人工审查，以确保代码中没有引入隐蔽 Bug、业务逻辑缺陷或重大安全漏洞，防止代码合并后搞垮生产环境。此外，后续的合并流程同样极其复杂——无论是采用特性分支（Feature Branches）还是主干开发（Trunk-based Development），团队往往都需要依赖合并队列（Merge Queue）来维持主干的稳定性，再经历灰度部署、系统监控、以及出现异常时的紧急回滚。当前市面上的代码生成工具基本无法触及外环的核心痛点，而 Graphite 的重心正是深耕开发外环，让 PR 的流动、审查、集成与部署更加顺畅。
---
### [03:53](https://www.youtube.com/watch?v=WDUxjZic4r4&t=233s) Diamond 的诞生与早期大模型的局限
Graphite 围绕代码审查流程打造了丰富的产品矩阵，包括 PR 页面和合并队列等，而本次演讲的核心则是其自研的 AI 代码审查器——**Diamond**。Graphite 探索 AI 代码审查并非一朝一夕，早在 GPT-3 时代，团队就开始调用 API 喂入代码差异（Diff），尝试让模型提供审查意见，但在早期版本中遇到了两大致命瓶颈：
1. **上下文窗口狭小**：当时模型的上下文仅有 2,000 到 4,000 个 Token，只能容纳两三百行代码。对于小 PR 尚可应付，但开发者真正渴望 AI 提供审查协助的往往是那些规模庞大、逻辑复杂的改动，而这些改动根本无法塞进上下文窗口中。
2. **生成内容充斥主观废话**：早期模型的推理能力有限，面对代码往往会写出一篇类似“高中作文”的冗长评语，充斥着大量无意义的主观建议。"这就好像我本来正在费力审查 200 行代码，AI 又给我额外写了 200 行毫无建设性的废话让我读。"
```mermaid
flowchart LR
accTitle: Graphite Diamond 的技术演进与设计定位
accDescription: 从早期 GPT-3 试验到基于 Sonnet 的高置信度行内审查演进路线。
subgraph Evolution["模型与产品演进"]
M1["早期 GPT-3 / 3.5<br/>(2k-4k Token 上下文小 / 废话多)"] --> M2["Claude 3.5 / 3.7 Sonnet<br/>(宽上下文 / 强推理能力)"]
M2 --> M3["Diamond 正式上线<br/>(聚焦行内精准反馈)"]
end
subgraph Strategy["审查策略转变"]
S1["废弃宏观总结式评价<br/>(Summary-level)"] --> S2["聚焦行内针对性批注<br/>(Inline Comments)"]
S2 --> S3["高置信度才发言<br/>(90-95% 把握才发声)"]
end
Evolution --> Strategy
```
团队并没有放弃，每当业内发布新模型（从 GPT-3、GPT-3.5、GPT-4 到 Claude 系列），Greg 都会在周六早晨坐到办公桌前，把所有的测试实验重新跑一遍。直到 Claude 3.5 Sonnet 及后续更强大的模型问世，上下文窗口和代码理解能力才真正跨过了实用门槛。
团队于去年 9 月正式启动并推出了 Diamond。在产品设计上，Graphite 做出了一个关键决断：**彻底放弃宽泛的概括性总结评语，全面押注“行内批注（Inline Comments）”**。工程师在审查代码时只愿意阅读清晰、可直接执行的建议，尽管在特定代码行留下高质量精准批注的技术难度要高得多，但唯有这样才能提供真正的价值。
---
### [06:04](https://www.youtube.com/watch?v=WDUxjZic4r4&t=364s) 生产级规模、极高信噪比与现代评测驱动开发
如今 Diamond 已经在大规模生产环境中稳定运行，服务于包括 Snowflake、Duolingo 在内的数千家主流企业。截至目前，Diamond 已累计审查了约 160 万个 PR，留下了超过 50 万条行内批注，且每天仍在处理数以万计的新 PR。
在如此巨大的规模下运行大模型极为惊险。生产级大模型极其脆弱，可能只是对 Prompt 进行了一丝微调，模型就会突然开始胡言乱语、吐出极其劣质的评论。一旦发生这种情况，用户对 AI 的信任度就会瞬间崩塌。开发者群体天生对机器留下的代码反馈持极其挑剔和审视的态度，因此维护信任底线是生死攸关的事。
为此，Graphite 将核心优化目标定为**极致的信噪比（Signal-to-Noise Ratio）**：
- **宁缺毋滥的置信度门槛**：即使系统有 50% 的把握认为某条批注有用，内部策略也会直接一票否决、放弃发送。只有当系统拥有 90% 到 95% 的极高置信度，确信这是一条能带来实际代码修改的有效建议时，才会公开发布。
- **放弃全覆盖，追求高精准**：Graphite 并不苛求抓住每一个 PR 中的所有 Bug（不追求 100% 召回率），而是全力确保发出的每一条批注都切实有用、可操作并促成代码修改（极致的精确率）。
这一目标也彻底改变了团队的工程开发模式。在传统的机器学习（ML）时代，工程师把大量精力耗费在 Python 特征工程和超参数调优上，评测往往只放在最后；而在 2025 年的大模型工程实践中，团队直接使用 Anthropic 或 OpenAI 的前沿基座模型。团队尝试过微调（Fine-tuning），但发现收益有限；自代码大模型具备强大的 Glob 和工具调用（Tool Calling）能力后，传统的 RAG 架构也失去了必要性。团队如今将绝大部分精力投入到**评测体系（Evals）**的建设上——构建包含海量真实 Diff 及其对应人工精选批注的黄金评测集，通过成百上千个评估案例在流水线中自动化跑测，以此驱动模型的持续迭代。
### [08:40](https://www.youtube.com/watch?v=WDUxjZic4r4&t=520s) 基于真实 PR 构建评测基准与测试用例
在模型迭代、提示词（Prompt）更新或上线全新模型的过程中，如何准确判断系统的表现究竟是在变好还是变差？Graphite 团队深度借助了评估平台 Braintrust 来搭建评测体系。为了消除数据敏感性并确保评测贴近实际开发，团队构建了包含数千个真实 Pull Request（PR）的测试集，这些样本部分来自开源代码库，大部分则抽取自内部代码库及工程团队的日常研发流程。
```mermaid
flowchart TD
accTitle: AI代码评审评测工作流
subgraph Dataset["测试集构建 (数千真实 PR)"]
A["开源仓库 PR"] --> C["评测用例库"]
B["内部代码库 PR"] --> C
end
subgraph Scenarios["典型评测场景"]
C --> D["缺陷样本 (语法错误 / 安全漏洞 / 业务 Bug)"]
C --> E["无缺陷样本 (包含误导性红鲱鱼代码)"]
end
subgraph Evaluation["评测目标"]
D --> F["匹配检查: 能否准确找出已知问题"]
E --> G["未匹配检查: 能否保持沉默、不产生噪音"]
end
```
这些测试用例覆盖了多样化的现实场景：
- **存在明确缺陷的 PR**：涵盖常规 Bug、语法错误以及安全层面的漏洞。评测重点在于：当切换新模型、调整 Prompt 或演进上下文管线时，系统是否仍能稳定检出这些已知问题；或者过去漏报但人类评审员能发现的缺陷，新版本能否成功捕获。
- **完全没有问题的正常 PR**：部分代码可能看似存疑（红鲱鱼 / Red Herring），但实际上并无瑕疵。评测要求 AI 在此类场景下必须“保持闭嘴”，坚决不留任何无意义的评论。通过将这两类场景完整固化为 Eval 测试集，团队得以为 AI 代码评审器建立起客观的自动化回归基准。
---
### [09:44](https://www.youtube.com/watch?v=WDUxjZic4r4&t=584s) 核心生产指标：评论采纳率与人类基准
为了量化 AI 在真实生产环境中的表现，Graphite 核心依赖的北极星指标是**采纳率（Acceptance Rate）**。在 Graphite 的定义中，采纳率指当 AI 在 PR 的某行代码上留下行内评论（Inline Comment）后，该行代码在 PR 最终合并（Merge）前发生修改和改进的概率，即该条评论促成实际代码修正的几率。
```mermaid
flowchart LR
accTitle: 代码评审评论采纳率与操作流
A["AI 留下行内评论"] --> B{"作者后续操作"}
B -->|"修改对应行代码并合并"| C["有效采纳 (Acceptance Rate: ~52%)"]
B -->|"关闭评论 / 未作修改"| D["未采纳 (赞赏/争论/误报)"]
B -->|"点踩反对"| E["负面反馈 (Downvote Guardrail: <4%)"]
```
Graphite 平台不仅完成了 160 万次 AI 代码评审，其系统更流转处理了数千万乃至上亿次未使用 AI 评审的常规 PR，这使得团队能够精准统计人类工程师的代码评审基准行为：
- **人类基准线**：人类评审员留下评论后，约有 45% 到 50% 的概率会促成该行代码的修改；其余情况通常是纯粹的夸奖鼓励，或者是作者与评审员展开激烈争论后直接关闭评论。
- **AI 当前表现**：经持续针对性优化，Graphite 的 AI 评论促成代码更新的概率达到了约 52%，已正式超越人类平均基准线。
- **防护护栏指标（Guardrail Metric）**：Graphite 与 GitHub UI 均支持用户对 AI 评论点踩（Down Emoji）。团队将点踩率作为核心防护指标并全力压低，目前该数据已成功控制在 4% 以下。
---
### [11:11](https://www.youtube.com/watch?v=WDUxjZic4r4&t=671s) 评测指标维度与主流大模型实测对比
Graphite 在评估各大基础模型在代码理解与评审上的能力时，主要锚定两个核心维度：**匹配评论得分（Matched Comment Score / 召回率）**——在确实存在缺陷的代码中，模型能否准确指出问题；以及**未匹配评论得分（Unmatched Comment Score / 精确率）**——在本身正确的代码中，模型能否克制发声、不产生虚假告警。
| 模型 | 匹配表现 (召回) | 未匹配表现 (精确/噪音控制) | 响应速度与成本特征 | 综合评价 |
|:--- |:--- |:--- |:--- |:--- |
| **GPT-5** | 极强，推理能力出众，找已知 Bug 最准 | 噪音偏多，容易在无问题 PR 上挑刺 | 偏慢；输出偏啰嗦，实际 Token 成本未达预期优势 | 检出能力极强，但较嘈杂且速度较慢 |
| **GPT-4o** | 较弱，难以稳定检出复杂 Bug | 较好，误报较少 | 响应较快 | 不适合作为主力高精度代码评审器 |
| **o3** | 较好，具备较强推理链 | 表现中规中矩 | 极慢且昂贵，工具调用表现一般 | 推理深入但代码综合理解与速度性价比偏低 |
| **Gemini Pro** | 中等 | 一般 | 具备 1M 超大上下文且价格极具吸引力，但速度较慢 | 大上下文极具诱惑力（适合 2k-3k 行大 PR），但质量未达最佳 |
| **Claude 3.7 Sonnet** | 极高且均衡 | 优秀，噪音低、信噪比极高 | 速度与延迟表现平衡 | 生产环境当前主力首选，综合表现最稳定 |
| **Claude 4 Sonnet** | 极高，工具调用与代码检索极佳 | 较差，评论数量激增近一倍 | 推理强劲，但产生大量边缘评论 | 工具使用强悍，但过于多嘴，信噪比不如 3.7 |
在实测中，针对代码评审特有的工作流，团队发现了各模型的一系列关键特征：
- **响应速度的工程权衡**：与 Cursor 或 Claude Code 等需要 30 秒至 1 分钟内即时反馈的本地交互工具不同，后台代码评审对延迟有一定容忍度。由于工程 CI 流水线通常需要跑 10 到 15 分钟，而行业内人类 Review 的中位数等待时间长达 18 小时（最快也要 3 小时左右），因此 AI 在几分钟内给出结果完全处于可接受区间。
- **GPT-5 的初测体验**：在刚发布的实测中，GPT-5 的匹配找错和推理逻辑极强，但较为缓慢且噪音较高；虽然输入 Token 标价相比 Anthropic 模型便宜 2-3 倍，但由于其输出更为冗长（Wordy），在实际生产调用中的真实成本优势被明显稀释。
- **上下文窗口与工具集成**：Gemini Pro 率先提供了 100 万 Token 的上下文，对于分析 2000 至 3000 行的大型 PR 极具价值。而新一代代码评审早已超越“输入 Diff、输出 Comment”的简单阶段，模型需要利用检索工具遍历整个代码库、翻阅历史 PR 与评论以辅助判断。在这一类 Tool Use 场景下，Anthropic 的 Sonnet 系列展现出了极高的工程检索适配度。
---
### [15:56](https://www.youtube.com/watch?v=WDUxjZic4r4&t=956s) 综合评分结论与提示词适配权衡
根据统一 Benchmark 面板的综合得分统计，Claude 3.7 Sonnet 在匹配与未匹配两项综合指标中均名列前茅，得分超过 60% 的模型均处于可用区间，o3、Gemini 以及 GPT-5 也都展现了不俗的技术潜力。
在产品设计哲学上，工程团队面临着“高召回率（不漏过任何潜在问题但评论繁多）”与“高信噪比（发言极少但条条切中要害）”之间的经典权衡。Graphite 目前在产品底层主动设定这一平衡阈值，但未来更倾向于将这一控制权开放给开发者，允许工程师按需调节评审器倾向于“捕获更多”还是“绝对高信噪比”。
同时需要指出的是，当前横向评测是在保持所有模型**完全相同的 Prompt 与工作流**下执行的。这种评测方式虽然保证了控制变量的公平性，但对新模型并不完全公允。例如像 GPT-5 这样架构与行为发生变化的模型，往往需要针对其特有推理机制重新设计并微调 Prompt 才能激发全部潜能。拥有完备的 Eval 体系使得后续针对各模型单独调优成为可能。
### [16:51](https://www.youtube.com/watch?v=WDUxjZic4r4&t=1011s) 模型迭代速度与评估总结：Claude、Gemini 与 GPT 的角逐
在当今的大模型竞争格局中，几乎每三个月就会有重磅模型发布。这种极快的迭代节奏对工程团队提出了极高的实验要求：每当新模型问世，团队必须极其审慎地重新审视并调优管道中 10 到 20 个不同的 Prompt（提示词），使其适配该模型的最佳表现。这项工作虽然非常繁琐且具有挑战性，但对于保持评测与生产环境的精确度至关重要。
总结 Graphite 的评测与实际体验，Claude 在推理响应速度上处于明显领先地位，而在代码理解能力上与其他顶级模型基本打平。Greg 指出，如果当前无法使用 Claude，他很可能会采用 Gemini 和 GPT 模型的混合方案（Mixture）。同时，他也预测即将发布的下一代 Gemini 模型可能会在今年晚些时候占据主导地位。
- **快速迭代与维护成本**：新模型约每 3 个月发布一次，团队需要持续维护并针对 10-20 个 Prompt 流程运行回测实验。
- **当前模型格局总结**：Claude 在生成速度上占优，代码理解深度与顶尖模型并驾齐驱；Gemini 与 GPT 系列紧随其后，且后续迭代潜力巨大。
---
### [17:25](https://www.youtube.com/watch?v=WDUxjZic4r4&t=1045s) 拓展 AI 在代码评审外的应用：一键诊断与修复 CI 失败
除了在代码行间留下审查评论（Inline Comments）之外，Graphite 正在全力打通软件开发“外循环”（Outer Loop）的闭环体验。团队成员 Michael 专注于解决开发者日常最头疼的问题之一——CI（持续集成）构建失败。在传统工作流中，开发者遇到 CI 报错时，往往需要痛苦地在终端或网页里翻阅 2000 多行无关的冗长日志，一路滑到底部用快捷键搜索报错红线，然后无奈地感叹：“糟了，我的 PR 里有个小 bug（'Oh crap, I have a minor bug in my PR.'）。”
Graphite 利用同样的 AI 基础设施重构了这一流程。系统会自动抓取并提炼 CI 日志中的核心错误，省去开发者手动排查日志的繁琐步骤。更进一步的是，模型会结合当前的 Git Diff（代码差异）直接在行内生成修复建议，并提供“一键采纳更新”（One Button Press）功能。开发者无需在本地重新检出（Checkout）分支、运行测试并手动提交单行修复，就能让代码快速通过 CI 并推进合并，在确保开发者始终拥有掌控权的同时极大提升了研发流转速度。
```mermaid
flowchart TD
accTitle: CI 故障智能诊断与修复流程
subgraph Problem["传统排查痛点"]
A["CI 构建失败"] --> B["手动翻阅 2000+ 行日志"]
B --> C["本地 Checkout 分支修复"]
end
subgraph AISolution["Graphite 智能闭环"]
D["AI 自动提炼核心错误"] --> E["结合 Git Diff 生成修复建议"]
E --> F["界面一键确认并应用补丁"]
end
A -.-> D
F --> G["CI 重新变绿，加速 PR 合并"]
```
- **痛点消除**：告别在数千行无用日志中寻找报错信息的低效过程。
- **极速修复闭环**：通过日志摘要与 Diff 深度分析，直接在界面提供单行修复建议，支持一键合并应用，避免繁琐的本地上下文切换。
## Notable quotes
> [03:53](https://www.youtube.com/watch?v=WDUxjZic4r4&t=233s) 这就好像我本来正在费力审查 200 行代码，AI 又给我额外写了 200 行毫无建设性的废话让我读。 — Greg Foster
> [06:04](https://www.youtube.com/watch?v=WDUxjZic4r4&t=364s) 即使系统有 50% 的把握认为某条批注有用，内部策略也会直接一票否决、放弃发送。只有当系统拥有 90% 到 95% 的极高置信度，确信这是一条能带来实际代码修改的有效建议时，才会公开发布。 — Greg Foster
> [09:44](https://www.youtube.com/watch?v=WDUxjZic4r4&t=584s) 人类评审员留下评论后，约有 45% 到 50% 的概率会促成该行代码的修改；其余情况通常是纯粹的夸奖鼓励，或者是作者与评审员展开激烈争论后直接关闭评论。 — Greg Foster
> [09:44](https://www.youtube.com/watch?v=WDUxjZic4r4&t=584s) 经持续针对性优化，Graphite 的 AI 评论促成代码更新的概率达到了约 52%，已正式超越人类平均基准线。 — Greg Foster
> [11:11](https://www.youtube.com/watch?v=WDUxjZic4r4&t=671s) 与 Cursor 或 Claude Code 等需要 30 秒至 1 分钟内即时反馈的本地交互工具不同，后台代码评审对延迟有一定容忍度。由于工程 CI 流水线通常需要跑 10 到 15 分钟，而行业内人类 Review 的中位数等待时间长达 18 小时（最快也要 3 小时左右），因此 AI 在几分钟内给出结果完全处于可接受区间。 — Greg Foster
> [16:51](https://www.youtube.com/watch?v=WDUxjZic4r4&t=1011s) 如果当前无法使用 Claude，他很可能会采用 Gemini 和 GPT 模型的混合方案（Mixture）。 — Greg Foster
> [16:51](https://www.youtube.com/watch?v=WDUxjZic4r4&t=1011s) 在当今的大模型竞争格局中，几乎每三个月就会有重磅模型发布。这种极快的迭代节奏对工程团队提出了极高的实验要求：每当新模型问世，团队必须极其审慎地重新审视并调优管道中 10 到 20 个不同的 Prompt（提示词），使其适配该模型的最佳表现。 — Greg Foster
> [17:25](https://www.youtube.com/watch?v=WDUxjZic4r4&t=1045s) 糟了，我的 PR 里有个小 bug（'Oh crap, I have a minor bug in my PR.'） — Greg Foster
## Mentioned
Greg Foster, Graphite, Diamond, GitHub Copilot, Satya Nadella, Google, Anthropic, Dario Amodei, Kevin Scott, Microsoft, Cursor, Claude Code, GPT-3, GPT-3.5, GPT-4, GPT-4o, GPT-5, o3, Claude 3.5 Sonnet, Claude 3.7 Sonnet, Claude 4 Sonnet, Gemini Pro, Snowflake, Duolingo, Braintrust, Michael, GitHub
## Video description
Greg Foster, CTO, Graphite
Abstracts
Not all AI models are created equal, especially when it comes to understanding code. This talk breaks down how we evaluate models for code comprehension at Graphite, focusing on their performance in real-world code review workflows. I’ll explain the two core metrics we rely on: matched comment score (recall) and unmatched comment score (precision), and how these metrics guide us in building a high-signal, low-noise code reviewer. You’ll learn how models like Claude, Gemini, and GPT compare, and what actually makes a model useful in production. If you're building with or evaluating LLMs for software engineering, you’ll leave with a practical framework for assessing model quality.
[Source](https://www.youtube.com/watch?v=WDUxjZic4r4)
Source quotes
[3:53] Greg Foster: 这就好像我本来正在费力审查 200 行代码，AI 又给我额外写了 200 行毫无建设性的废话让我读。
[6:04] Greg Foster: 即使系统有 50% 的把握认为某条批注有用，内部策略也会直接一票否决、放弃发送。只有当系统拥有 90% 到 95% 的极高置信度，确信这是一条能带来实际代码修改的有效建议时，才会公开发布。
[9:44] Greg Foster: 人类评审员留下评论后，约有 45% 到 50% 的概率会促成该行代码的修改；其余情况通常是纯粹的夸奖鼓励，或者是作者与评审员展开激烈争论后直接关闭评论。
[9:44] Greg Foster: 经持续针对性优化，Graphite 的 AI 评论促成代码更新的概率达到了约 52%，已正式超越人类平均基准线。
[11:11] Greg Foster: 与 Cursor 或 Claude Code 等需要 30 秒至 1 分钟内即时反馈的本地交互工具不同，后台代码评审对延迟有一定容忍度。由于工程 CI 流水线通常需要跑 10 到 15 分钟，而行业内人类 Review 的中位数等待时间长达 18 小时（最快也要 3 小时左右），因此 AI 在几分钟内给出结果完全处于可接受区间。
[16:51] Greg Foster: 如果当前无法使用 Claude，他很可能会采用 Gemini 和 GPT 模型的混合方案（Mixture）。
[16:51] Greg Foster: 在当今的大模型竞争格局中，几乎每三个月就会有重磅模型发布。这种极快的迭代节奏对工程团队提出了极高的实验要求：每当新模型问世，团队必须极其审慎地重新审视并调优管道中 10 到 20 个不同的 Prompt（提示词），使其适配该模型的最佳表现。
[17:25] Greg Foster: 糟了，我的 PR 里有个小 bug（'Oh crap, I have a minor bug in my PR.'）
Source quotes
[4:06] Greg Foster: they would write me like a high school essay about my pull request and they would give me all this like subjective feedback.
[8:40] Greg Foster: shut up
[16:51] Greg Foster: non-trivial and you have to be kind of thoughtful about running those experiments.
[16:51] Greg Foster: dominate the field later this year.
Original transcript
[0:00] Awesome.
[0:04] So, we can get started. My name is Greg.
[0:06] I'm the co-founder and CTO at Graphite.
[0:09] If you don't know about us, Graphite is
[0:11] the best place to review software. We've
[0:14] been creating the platform for code
[0:17] review. Specifically, also in the age of
[0:19] AI, our general thesis is that AI is
[0:21] creating more code than ever than
[0:24] before. It's creating demand and
[0:26] pressure on the review side of things to
[0:28] make sure we're validating it,
[0:29] integrating it, and keeping the whole
[0:31] process safe. Otherwise, by default,
[0:34] we're bottlenecking on the code review
[0:35] process. And we want to keep up with the
[0:37] pace that's being unlocked with all the
[0:39] AI codegen, we need to make sure the
[0:41] review side of things is moving
[0:42] incredibly fast as well.
[0:44] So before I dive into what we've been
[0:47] doing with the AI code review side of
[0:49] it, I want to ask a question which is
[0:51] what percentage of code do we think is
[0:54] being AI generated today? There's
[0:57] constant attempts to answer this
[0:59] question. I think a year or two ago the
[1:01] co-pilot team from GitHub was claiming
[1:04] that 30% or more was being AI generated.
[1:08] You had Satia Nadell claiming some
[1:10] similar statistics. I think currently uh
[1:13] Google is claiming that about 30% of
[1:15] their new code internally is being AI
[1:18] generated that was up from 25%. Now if
[1:20] you're more bullish and you look at uh
[1:23] what is it Dario of Enthropic who's
[1:25] constantly trying to make really
[1:27] aggressive claims he claimed earlier
[1:29] this year that 90% of software of new
[1:32] code being written would be AI
[1:33] generated. Kevin Scott the CTO of
[1:36] Microsoft was claiming that within 5
[1:38] years it'll be 100%. Now these business
[1:40] leaders are kind of incentivized to make
[1:42] really bold claims and just say that all
[1:44] of software is going to be written by AI
[1:47] and it'll fully take over regardless of
[1:50] where it's landing. It it is true a lot
[1:51] of the new code is being you know
[1:53] autocompleted or written or headlessly
[1:55] created whether it's 10% 30% 90% some
[1:58] amount of it is being generated but when
[2:01] I think about that generation I think
[2:02] about our jobs as software engineers I
[2:04] think about two loops we have on the
[2:06] left side we have the inner loop and on
[2:07] the right side we have the outer loop
[2:10] the inner loop is what all this codegen
[2:12] is kind of obsessing about it's when you
[2:14] open up your IDE or your terminal you
[2:16] break open cloud code or cursor or if
[2:19] you're you know Vim if you're actually
[2:20] still handwriting it and you're creating
[2:22] the code and you're building it and
[2:23] you're debugging it and you're making
[2:24] sure it works. This is the inter loop of
[2:26] software development and I really think
[2:27] it's starting to shrink. The time spent
[2:29] here is going less and less. We used to
[2:31] spend 10 hours, 5 hours writing a code
[2:33] change. Now with the help of some of
[2:35] these tools, you might be doing it in 10
[2:36] minutes. You might be kicking off a
[2:38] background agent and it's going getting
[2:40] lunch and coming back and your code
[2:41] changes sitting there. But the part of
[2:43] the system that is not yet solved that
[2:45] ensures that we still all have good jobs
[2:47] is the outer loop of software
[2:49] development. This is when you put up
[2:50] that poll request and you give it a
[2:52] title and a description and you tag in
[2:54] some reviewers and you look at your
[2:55] company's CI and your llinters and your
[2:57] unit tests. You give it a thoughtful
[2:59] human grade review to make sure there's
[3:01] no bugs issues. You're sharing context
[3:04] learning no major security
[3:05] vulnerabilities. You're asserting hey
[3:07] this is not going to take down
[3:08] production when we merge it in. And then
[3:10] the merge part of the process which is
[3:12] non-trivial. Companies have very
[3:14] different merge strategies. Some people
[3:15] use feature branches. Some companies use
[3:16] trunkbased development. But even in
[3:18] trunkbased development, you might have a
[3:19] merge queue. You might do a lot of work
[3:21] to keep that trunk green. And then you
[3:23] got to deploy it. And then you got to
[3:24] monitor it. And if there's an issue, you
[3:26] got to roll it back, revert it, write
[3:27] your next code change. This is the outer
[3:28] loop of software development. And by and
[3:30] large, cursor, cloud code, all these
[3:32] codegen tools are not meaningfully
[3:34] making a dent here. This is up to us as
[3:36] software engineers to still do a very
[3:38] good job. So at Graphite, we obsess
[3:41] about this outer loop of software
[3:42] development. We want to make it as easy
[3:43] and smooth and fast as possible to keep
[3:46] the PRs moving, to review them, to
[3:48] integrate them, to deploy them out.
[3:53] Now, one thing we've been working hard
[3:55] on is Diamond, which is our AI code
[3:58] reviewer. We build a lot of stuff within
[4:00] the code review process. We have a pull
[4:01] request page. We have a merge cube. But
[4:03] what I'm going to talk about today is
[4:05] Diamond, which is our AI code reviewer.
[4:06] And it's kind of interesting. We were
[4:08] trying to build AI code review for a
[4:10] while. We've been we were experimenting
[4:12] with it for years. Ever since I think
[4:14] about GBT3,
[4:16] you know, with the API, we would feed in
[4:18] diffs and we would ask it, hey, please
[4:21] review this code change. And we see,
[4:22] okay, how good is AI these days at
[4:24] feeding it a code change. And with the
[4:26] early versions, we noticed a couple
[4:27] things. One, you couldn't feed in a
[4:30] large pull request because the context
[4:32] window was like 2,000 or 4,000 tokens.
[4:34] That was only a couple hundred lines of
[4:35] code. And that was okay. Okay, it could
[4:38] help you review smaller code changes,
[4:39] but honestly, the ones I really wanted
[4:41] help reviewing were some of the larger,
[4:42] more complicated ones. And I couldn't
[4:43] fit it into the context window. The
[4:46] other issue we ran into with the early
[4:47] models is they were kind of stupid and
[4:49] they would write me like a high school
[4:51] essay about my pull request and they
[4:53] would give me all this like subjective
[4:54] feedback. I'm like, "Okay, great. You
[4:55] wrote me another 200 lines I have to
[4:57] read while I'm trying to review this
[4:58] first 200 lines." It wasn't helpful. It
[5:00] was attempting it, but it was it wasn't
[5:01] very helpful. And so, we were optimistic
[5:04] though. So every time a new release came
[5:06] out, it'd be like GPT3, GPT 3.5, you
[5:09] know, 4, then you have Sonic come out.
[5:11] Every time one came out, it'd be like
[5:13] Saturday morning. I'd sit down at my
[5:14] desk, I'd be like, "Okay, let's rerun
[5:15] all our experiments. Let's see if this
[5:16] new model is good enough." And honestly,
[5:19] it wasn't until about Sonnet 4 or so or
[5:23] Sonnet 3.7, even 35, Sonnet model
[5:25] started coming out that the token window
[5:27] was wide enough, the model was smart
[5:29] enough that we could start getting some
[5:31] useful feedback on these PRs. And so we
[5:34] ended up building uh diamond. We we came
[5:36] up with this. We started working on last
[5:37] September. We launched it. And what we
[5:40] really focused on was trying to leave
[5:42] inline comments. We also realized that
[5:44] with summary level comments, it was not
[5:46] that interesting because again, people
[5:48] don't really want to read anything more
[5:50] than they have to unless it's something
[5:52] very clear and actionable. And so we're
[5:53] like, okay, interesting. Now, this is a
[5:55] harder problem leaving inline comments,
[5:57] but let's let's attempt to do that.
[5:58] Let's try and only speak up if we have
[6:00] deep confidence that we can call
[6:02] something out. So that's what we built.
[6:04] Now I just want to we're going to talk
[6:06] about how we tested this, how we
[6:07] validated, how we made sure it was high
[6:09] quality because that's kind of the
[6:10] interesting engineering the exploration
[6:11] that we did. But just to put this in
[6:13] context today, Diamond's really widely
[6:16] used. It's used at thousands of major
[6:18] organizations including Snowflake and
[6:19] Dolingo and many more. We've reviewed
[6:23] about 1.6 million PRs. We've left over
[6:26] half a million comments. And every
[6:29] single day, today, tomorrow, we're
[6:30] reviewing tens of thousands of more PRs.
[6:32] So, we're actually shipping this at
[6:34] scale to many thousands of engineers who
[6:36] are experiencing this today, which is
[6:38] scary because it means that we actually
[6:41] need to make sure these comments and
[6:42] these code reviews are accurate and
[6:44] useful and not buggy because I'll tell
[6:46] you, it is very easy to break a
[6:48] production level LM model. You change a
[6:50] prompt ever so slightly, some somehow it
[6:52] starts spewing out really bad comments,
[6:54] flaking on users, and trust tanks. And
[6:57] if we got to do one thing with LMS, we
[6:59] got to keep that trust bar really,
[7:00] really high because developers,
[7:01] rightfully so, are very scrutinizing of
[7:03] anytime AI is leaving feedback or
[7:06] writing on their poll requests.
[7:08] So, what we've tried to zero in on is
[7:11] what we think about is signal to noise
[7:12] ratio. If we're going to leave a
[7:14] comment, we want to be as useful as
[7:16] possible. In fact, I'd actually even if
[7:19] I have like 50% confidence that we can
[7:21] leave a comment that's going to be
[7:22] helpful, we're going to rule it out.
[7:23] We're going to veto it internally. we're
[7:24] going to not post on the user's pull
[7:26] request. We want to have like 90 95%
[7:29] confidence that this is going to be
[7:30] useful, actionable comment and then
[7:31] we'll speak up and we'll post it. I
[7:33] actually don't necessarily want to catch
[7:35] every single bug and every single PR.
[7:37] Rather, what we attempt to optimize is
[7:38] make sure every single comment we leave
[7:40] is useful, actionable, and results in
[7:42] change on that poll request. So it's
[7:45] it's really interesting like in old
[7:46] school ML what we would have done is
[7:48] spent a ton of time working in Python
[7:51] feature engineering tuning
[7:52] hyperparameters
[7:54] and a little bit at the very end we
[7:56] would have done evals but now in 2025
[7:59] the the engineering process looks very
[8:00] different we're mostly working with the
[8:02] major providers so anthropic openai
[8:05] we're taking these off-the-shelf large
[8:07] models we can fine-tune them fine-tune
[8:10] them and we we experimented a little bit
[8:11] with that but honestly we have not found
[8:12] it to be too useful You can build rag
[8:14] systems, but I'll be honest, we have not
[8:16] found rag to be crazy useful ever since
[8:19] got very good at glob and tool calling.
[8:22] And the most of our energy and what
[8:24] we've realized is is very useful is
[8:26] going very hard on evals and creating
[8:28] incredible test data sets of real world
[8:30] in our case it's diffs and then the
[8:33] comments that get left on those diffs
[8:34] and can we create hundreds or even
[8:35] thousands of evaluative cases to then
[8:38] run through our pipelines run through
[8:40] our models if we update our prompts if
[8:42] we ship a new model do we know that it's
[8:44] better or worse so we're big fans of
[8:47] brain trust brain trust is also at this
[8:49] conference so if you see them you see
[8:50] their booth give them a shout out I
[8:51] really love brain trust trust. But what
[8:52] we've gone in here and I made sure the
[8:54] screenshot didn't contain anything
[8:55] sensitive. We we've created thousands of
[8:57] real world pull requests. Some are from
[9:00] open source repositories and many are
[9:02] also from our internal code bases and
[9:04] internal engineering teams. And we're
[9:05] trying to create as many of these test
[9:06] cases as possible. Cases where often
[9:08] there's a bug, maybe a syntax error,
[9:10] maybe security level vulnerability. And
[9:13] what we want to say is, okay, great. If
[9:15] we were to rerun on a new model or
[9:16] change our prompts or evolve our
[9:18] pipeline, can it still find all these
[9:20] issues? Or are there cases where the
[9:22] model didn't used to find an issue? We
[9:23] as humans know there was an issue in it
[9:25] and can we catch it moving forward. And
[9:27] lastly, there's another case of this
[9:28] which is pull requests where there is no
[9:30] issue in which case we want the model to
[9:32] not comment on anything and maybe it's a
[9:34] red herring. Maybe it looks like there
[9:35] might be a problem but we want to shut
[9:36] up. And so we want to encode all these
[9:38] different scenarios into eval test cases
[9:40] that we can then run our AI code
[9:42] reviewer on.
[9:44] So when we're trying to evaluate how
[9:47] good we're doing, how good these
[9:49] different models are, what we've chosen
[9:50] to lean on is what we call acceptance
[9:52] rate. And for acceptance rate, for us,
[9:54] what that means is given an inline
[9:57] comment on a PR, what are the odds that
[9:59] that line of code then gets updated
[10:02] after the comment, but before the poll
[10:04] request gets merged in. So the odds that
[10:06] that comment leads to action on the PR.
[10:09] We studied this very closely because
[10:10] graphically I said, you know, we've
[10:11] reviewed 1.6 6 million pull requests.
[10:13] But broadly, I mean, Graphite processes
[10:16] so many. We have tens, hundreds of
[10:17] millions of PRs passing through our
[10:19] systems that aren't using AI code
[10:20] review, but we can still study the
[10:22] average rates of the comments and what
[10:24] are happening on those different
[10:25] different poll requests. So, we know for
[10:27] a fact that if a human leaves a comment
[10:29] on a PR, about 45 to 50% of the time
[10:32] it'll result in that line of code being
[10:33] changed. The other percent of the time,
[10:35] I don't know exactly. I assume it's
[10:36] either a compliment or a flame war
[10:38] happening where the author just closes
[10:40] it and moves on with their life. But we
[10:41] know that that's about the human
[10:42] baseline. So we try to beat that. And
[10:44] right now, and Michael, my teammate
[10:47] right here, is like working daily to
[10:48] optimize this number further and
[10:49] further. But if we leave a comment with
[10:52] about 52% chance, it'll then result in a
[10:54] that that line of code being up uh
[10:56] updated and improved. Now, we also have
[10:58] UI that lets users down those comments
[11:01] either on Graphite or in the GitHub UI.
[11:02] They can leave a down emoji. And we see
[11:05] that we also try as a guardrail metric
[11:06] to really push that down. And it's right
[11:08] now it's below 4%.
[11:11] But what I want to talk to you about is
[11:13] we've tried all of the models. So one
[11:15] thing that's kind of fun here is which
[11:16] of the models is actually best at doing
[11:17] the AI code review. We study this very
[11:19] closely. Every time a new one comes out
[11:20] we try and run them.
[11:23] So as I was saying before when we say
[11:25] what is good, what is bad, we care about
[11:27] these two things. One matched comments
[11:29] which is cases where we know there
[11:31] should be a comment on the on the pull
[11:33] request. We know that there's a bug. Did
[11:35] the model find it? and unmatched
[11:37] comments, which is cases where we know
[11:38] there should not be a comment on this
[11:40] forecast. There's no valid, you know,
[11:41] issue. It's a perfectly fine PR. Did the
[11:43] model not speak up? Was it quiet in
[11:45] those cases? So, I'm going to run
[11:47] through what we found with some of the
[11:49] latest models. We just ran GB GB5, of
[11:52] course, just came out a few days ago. We
[11:53] we spent the weekend, Michael, we sat
[11:55] down, we ran the numbers on it. We were
[11:57] kind of curious. GB5 is kind of
[11:59] interesting. It was the best at finding
[12:01] the matches. the matches in cases where
[12:03] there was actually an issue. It was the
[12:04] best at finding them and calling them
[12:06] out. It was really good at the
[12:07] reasoning. What we noticed though is it
[12:09] was also kind of noisy. It was more more
[12:11] likely to call out issues on PRs that
[12:13] don't have a meaningful issue. It's also
[12:16] a lot slower we found than the current,
[12:18] you know, anthropic models. And speed is
[12:20] kind of interesting for AI code review.
[12:22] You know, depends use case. If you're
[12:23] running it locally or if you're trying
[12:25] to get quick feedback on your draft,
[12:26] yeah, within 30 seconds, within a
[12:27] minute, yeah, you'd love that. you'd
[12:29] love that feedback back so you can take
[12:30] action on it. At the same time, there is
[12:33] affordances to be a little bit slow
[12:34] here. Compared to cursor, compared to
[12:36] claude code, you know, if it takes five
[12:38] minutes, it's not the end of the world.
[12:39] Your CI already takes 10 to 15 minutes.
[12:42] You're willing to wait in a human
[12:43] review, I'll tell you, takes at best 3
[12:45] hours on median in the industry, it's
[12:47] like 18 hours before a PR gets a human
[12:50] review. So, if we're still talking in
[12:52] the seconds, we're doing pretty well.
[12:53] But this is my observation. GT5 good for
[12:56] matching a little noisy pretty slow.
[12:58] What we also don't have up here, but I
[13:00] know the token costs are very
[13:01] interesting. People were very excited
[13:02] that that GPT5 was maybe 2 to 3x cheaper
[13:05] on some of the token measurements
[13:06] compared to the anthropic models. But we
[13:08] also find that the output is a little
[13:11] bit more expensive, a little bit more
[13:12] wordy, and actually in production, it's
[13:14] not that much cheaper than just running
[13:16] the sauna models. That's what we found
[13:17] so far.
[13:19] Of course, there's GPD40.
[13:21] Uh this at the time before GBD5 was one
[13:24] of the best at unmatched comments, not
[13:27] the matched comments, but honestly it
[13:29] was not a very good code reviewer is not
[13:31] very good at consistently finding all
[13:33] the bugs.
[13:35] GBD3 or sorry GB03 uh pretty good of
[13:39] course the reasoning went pretty far.
[13:41] This was the first reasoning model and
[13:42] really quickly Anthropic came around and
[13:44] dropped their own reasoning model and
[13:45] tool use. We found to be super slow. It
[13:47] was really expensive. It was really slow
[13:49] and it was not the best at understanding
[13:51] code.
[13:53] Gemini Pro was very promising.
[13:56] It has or had the widest context window
[13:58] at a million tokens. I know Anthropic
[13:59] just competitively updated their token
[14:01] window to be a million. That was
[14:03] tempting. That's actually really useful
[14:04] for us because sometimes people are
[14:05] trying to review, you know, 2,000 LPRs,
[14:07] 3,000 LPRs and you actually want to
[14:08] leverage that full wide context window.
[14:11] But it was slow. The price is very
[14:13] tempting. It's got one of the best
[14:14] prices, but it was pretty slow and we
[14:16] did not find it to have the highest
[14:18] quality in mash and unmatched comment
[14:19] scores. And so we ended up not leaning
[14:22] heavily on it. Sonnet 3.7 and Sonnet 4
[14:25] I'm about to talk about this is what we
[14:26] this is honestly what we use a lot in
[14:28] our AI code review and it's not because
[14:29] we have loyalty to anthropic. It's just
[14:31] what we find to be the best in
[14:32] production.
[14:34] Sonnet 40 very strong in the reasoning,
[14:36] really great at the tool use which is
[14:37] important to us because as we evolve AI
[14:39] code review, it's not just diff in and
[14:41] comment out. It's also running a lot of
[14:43] searches under the hood. It's scrapping
[14:45] the codebase. It's following it's it's
[14:46] trying to pull in code and information
[14:48] that's not in the existing PR, but might
[14:51] be relevant to concluding that there's
[14:52] an issue or a bug in it. It could we
[14:54] give it tools for searching the
[14:55] codebase. We give it tools for reading
[14:57] old poll requests and old comments and
[14:59] trying to come to the best possible
[15:00] conclusion. So, Sonnet we found was
[15:02] really good at using the tools. It's
[15:04] interesting though, we actually don't
[15:06] prefer it to 37 in a lot of use cases or
[15:09] even 35. We found it got a lot noisier
[15:11] in in short. You was pretty good at
[15:12] calling stuff out, but it would just
[15:13] leave a ton of comments. It was leaving
[15:15] almost twice as many comments and not
[15:16] always better comments. And this is
[15:18] where we and you know us as engineers,
[15:20] the whole room, you have to be really
[15:21] thoughtful around what are you going
[15:22] for? Do you really care about low noise
[15:25] or do you really care about just
[15:26] catching every single possible case
[15:28] right now you know as as a product we
[15:30] choose to just fine-tune that line and
[15:32] decide what level of uh of recall we
[15:34] want. But I hope honestly one day in the
[15:36] future we'll kind of give that to the
[15:37] developers and say, "Hey, would you want
[15:39] your AI code reviewer to be a little bit
[15:40] noisier but maybe catch more things or a
[15:42] little bit quieter but when it calls
[15:44] something out it's really high high
[15:46] signal." So yeah, so 37, you know, not
[15:49] the noisiest one but one of the most
[15:51] reliable and accurate ones and honestly
[15:53] one of the ones we lean most on in
[15:55] production.
[15:56] So here's the scores just to lay it out
[15:58] in a in a table here. Here's the scores
[16:00] we found across these across both
[16:02] matched and unmatched. And you can see
[16:04] based on this dashboard why we It's
[16:06] interesting. We kind of prefer 37 across
[16:08] all of our eval cases. It's done the
[16:10] best. 03 is not far behind. Anything
[16:12] over 60% is doing pretty well here.
[16:14] Gemini is very good. GPD5, the brand new
[16:18] one, is very good. And if I had to
[16:19] explain what we're thinking about doing
[16:21] moving forward, I'm optimistic we can
[16:23] get really good performance out of GPT5.
[16:26] What we've not done in these slides, so
[16:27] in these slides and these measurements
[16:29] I'm showing, we're running the exact
[16:31] same prompts and the exact same workflow
[16:33] for all the different models. But that
[16:35] is actually that you know that's a
[16:36] little bit flawed and a little bit
[16:37] biased because ideally any specific
[16:39] model such as GT5 probably demands you
[16:42] to go back and rethink all your prompts
[16:43] and tune them perfectly for the model.
[16:45] And it's nice we have eval so we can go
[16:47] about doing that. But it is kind of it
[16:49] is kind of tricky because every time
[16:51] these models are dropping like every 3
[16:53] months and it is you do have to be very
[16:54] careful in your experimentation to go
[16:56] back through 10 20 different pipeline
[16:58] prompts update them to be the best
[17:00] prompts for the model. We're doing it
[17:02] and we do it but but it's non-trivial
[17:04] and you have to be kind of thoughtful
[17:05] about running those experiments.
[17:08] So this is kind of summarizing what I
[17:09] was saying. In general, Claude's winning
[17:11] on the speed. Claude's about tied on the
[17:13] code understanding. If I didn't have
[17:15] Claude, I'd probably be using Mixture
[17:16] Gemini and GPT5. I also suspect Gemini 3
[17:20] is going to drop and that's going to
[17:21] probably dominate the field later this
[17:22] year.
[17:25] To end it out, I just want to comment on
[17:26] some other things we've been leveraging
[17:28] AI for in the code review process beyond
[17:29] just leaving inline comments. We're
[17:32] trying really hard to close the loop of
[17:33] the the loop being that outer loop of
[17:35] software development. So, one of the fun
[17:36] things we've been building and Michael,
[17:37] my teammates's been working hard on is
[17:40] CI failures.
[17:42] Common case is you have a CI error. You
[17:44] scroll through 2,000 logs that you
[17:46] actually don't care about. You scroll to
[17:47] the very bottom. You control F for like
[17:48] the red squiggly line. You find the
[17:50] issue. You're like, "Oh crap, I have a
[17:52] minor bug in my PR." What we do, we use
[17:54] that same these same AI systems. We
[17:56] summarize the issue so you don't have to
[17:57] scroll through your logs. They're very
[17:59] good at summarization and pulling that
[18:00] out and then going further and actually
[18:02] suggesting a fix right there in line.
[18:04] We'll look at the diff. If it's
[18:05] possible, we'll suggest a fix in line
[18:07] and then give you one button press to
[18:08] actually accept a code update to fix it
[18:11] in that line. So you don't have to check
[18:12] out the code, run it locally, integrate
[18:14] the the the oneline change that would
[18:16] fix your failing CI. We're trying to
[18:18] make this whole process really smooth,
[18:20] really fast, but still keeping the
[18:21] developer in control and just giving you
[18:23] the feedback so you can take the fixes
[18:24] and then keep the PR moving quickly.
[18:27] So thank you. That's our talk.
[18:30] [Applause]
