Qodo CEO Sees AI Swarms Ending Human Pull Request Reviews by 2027

Qodo - The Last Human Code Review | DevDay 2026 — OpenAI

OpenAI

Autonomous coding and review agents will eliminate manual pull request reviews by resolving disputes against institutional memory graphs, escalating to engineers only when models reach an impasse.

AI-assisted summary of Qodo - The Last Human Code Review | DevDay 2026 — original source. Check the original for full context.

Source published:

Key ideas

  • Qodo combines background 'sleep-time compute' with mined institutional tribal knowledge to provide real-time adversarial code reviews alongside coding agents like Codex, catching systemic downstream blast radiuses before PR submission. (0:13)
  • The industry's methodology has progressed from prompt engineering in 2023 to structured flow engineering (AlphaCodium) in 2024, arriving at explicit, bounded swarm architectures to prevent wasteful sub-agent delegation. (5:07)
  • In a mature software factory model, coding agents and review agents debate and resolve issues autonomously, inverting human engagement so Codex prompts the human only when context conflicts create an impasse. (7:21)
  • Qodo is shifting the atomic unit of software development from isolated GitHub pull requests to 'Work Package Triage,' allowing teams to review and validate cohesive features spanning 5 to 15 cross-repository PRs. (13:56)

Detailed notes

## Background Published by OpenAI on YouTube on October 7, 2026, this DevDay session highlights how Qodo integrates OpenAI models to review AI-generated code. According to the description, the talk explores an adversarial reviewer that challenges coding agents' assumptions and output while examining the boundaries where model confidence still demands human judgment.

## TL;DR Qodo CEO Itamar Friedman joins Danielle from OpenAI to outline how software development is shifting from manual human pull request reviews toward an automated 'software factory.' By continuously running background 'sleep-time compute' and mining institutional tribal knowledge from chats and repositories into a living 'wisdom base,' Qodo creates dynamic software maps that catch downstream architectural blast radiuses. Rather than relying on simple prompts, the architecture pairs coding agents like Codex with adversarial review agents in bounded swarms that autonomously debate and refine code before involving developers. The session charts the path toward treating multi-repository changes as unified work packages, effectively eliminating routine human code reviews while retaining incident-driven organizational learning.

## Atmosphere & style The session carries the upbeat, conversational energy of an authentic technical fireside chat on an active expo stage, complete with spontaneous banter, inside jokes about 'scripts,' and candid, unvarnished expressions. Danielle and Itamar display an easy, collaborative chemistry rooted in shared history—from Danielle's early days teaching Qodo at Intuit to mutual reflections on the rapid evolution of prompt engineering. The discussion balances ambitious, futuristic predictions about autonomous software factories with grounded, pragmatic engineering realities, creating a forward-looking yet approachable dialogue.

## Bulletin - [00:13](https://www.youtube.com/watch?v=bQ3-pXCpCTg&t=13s) Qodo combines background 'sleep-time compute' with mined institutional tribal knowledge to provide real-time adversarial code reviews alongside coding agents like Codex, catching systemic downstream blast radiuses before PR submission. - [05:07](https://www.youtube.com/watch?v=bQ3-pXCpCTg&t=307s) The industry's methodology has progressed from prompt engineering in 2023 to structured flow engineering (AlphaCodium) in 2024, arriving at explicit, bounded swarm architectures to prevent wasteful sub-agent delegation. - [07:21](https://www.youtube.com/watch?v=bQ3-pXCpCTg&t=441s) In a mature software factory model, coding agents and review agents debate and resolve issues autonomously, inverting human engagement so Codex prompts the human only when context conflicts create an impasse. - [13:56](https://www.youtube.com/watch?v=bQ3-pXCpCTg&t=836s) Qodo is shifting the atomic unit of software development from isolated GitHub pull requests to 'Work Package Triage,' allowing teams to review and validate cohesive features spanning 5 to 15 cross-repository PRs.

## Key takeaways - [00:13](https://www.youtube.com/watch?v=bQ3-pXCpCTg&t=13s) Qodo acts as an adversarial review and software governance layer that injects organizational context into Codex during code generation. - [01:22](https://www.youtube.com/watch?v=bQ3-pXCpCTg&t=82s) Engineering wisdom differs from model intelligence: intelligence reasons over input, while wisdom requires codified lived experience of how systems fail under real-world stresses. - [03:21](https://www.youtube.com/watch?v=bQ3-pXCpCTg&t=201s) Local code diffs can appear syntactically and semantically perfect while still carrying catastrophic downstream blast radiuses in complex microservice architectures. - [03:21](https://www.youtube.com/watch?v=bQ3-pXCpCTg&t=201s) Sleep-time compute continuously scans and updates the wider codebase dependency graph in the background, matching Google-scale repository mapping that humans cannot hold in memory. - [05:07](https://www.youtube.com/watch?v=bQ3-pXCpCTg&t=307s) Dynamic and unbounded autonomous agent delegation remains costly and prone to hallucination; builders must enforce explicit boundaries, goals, and grounded toolsets. - [07:21](https://www.youtube.com/watch?v=bQ3-pXCpCTg&t=441s) By 2027, manual human review of static PR comments will be obsolete, replaced by real-time agent debates that only escalate irreconcilable impasses to developers. - [08:56](https://www.youtube.com/watch?v=bQ3-pXCpCTg&t=536s) Discrepancies between coding agents and review critics stem primarily from context asymmetry; grounding both agents in identical repository evidence leads to autonomous consensus. - [08:56](https://www.youtube.com/watch?v=bQ3-pXCpCTg&t=536s) Production bugs and false consensuses must be fed back into developer memory as post-mortems so the software factory never repeats the exact same architectural mistake. - [11:01](https://www.youtube.com/watch?v=bQ3-pXCpCTg&t=661s) Capturing tribal knowledge requires mining historical discussions across VCS tools, Slack, and Teams to uncover hidden runtime contracts and cross-service dependencies. - [13:56](https://www.youtube.com/watch?v=bQ3-pXCpCTg&t=836s) The traditional single pull request is too narrow an abstraction; complex engineering deliverables naturally span 5 to 15 interconnected PRs that require unified work package validation. - [15:44](https://www.youtube.com/watch?v=bQ3-pXCpCTg&t=944s) Swarm architecture should decompose tasks into granular workloads, pairing specialized agents with right-sized models ranging from fast decision APIs to heavy reasoning engines. - [17:34](https://www.youtube.com/watch?v=bQ3-pXCpCTg&t=1054s) Eliminating manual code reviews must be balanced with mechanisms that ensure junior and senior developers continue learning and growing alongside automated agents.

## Detailed walkthrough ### [00:13](https://www.youtube.com/watch?v=bQ3-pXCpCTg&t=13s) Introduction to Qodo and the Software Governance Layer

Danielle from the developer experience team opens the session by welcoming attendees and passersby at OpenAI DevDay, introducing Itamar Friedman, CEO of Qodo. To kick off the discussion, Danielle asks Itamar to explain what Qodo is, whom they build for, and how the platform positions itself alongside tools like Codex.

```mermaid flowchart TD accTitle: Qodo Context and Review Flow accDescr: Diagram showing tribal knowledge feeding the software map to guide Codex and the code review agent. subgraph ContextEngine["Context & Governance"] TK["Engineering Tribal Knowledge"] --> SM["Dynamic Software Map\n(Background compute)"] SM --> Gov["Governance & Drift Tracking"] end

subgraph AgentLoop["Agent Coding & Review"] SM --> Codex["Coding Agent\n(Codex)"] SM --> Reviewer["Adversarial Review Agent\n(Qodo)"] Codex <-->|"Iterative checks"| Reviewer Reviewer --> PR["Clean Pull Request"] end ```

Itamar explains that customers frequently describe Qodo as "Codex's best friend." Rather than operating merely as a generic linter or post-commit scanner, Qodo functions by mining institutional and engineering tribal knowledge across an organization. It injects that accumulated context directly into Codex and simultaneously uses it to review code in real time while the model is actively writing. The primary goal is to ensure engineers submit a clear, clean pull request from the very start. At a broader organizational scale, Qodo serves as an essential governance layer, maintaining a living software map that tracks and controls architectural drift over time.

[01:22](https://www.youtube.com/watch?v=bQ3-pXCpCTg&t=82s) Moving from Prompt Engineering to Swarms, and Intelligence vs. Wisdom

When Danielle asks how OpenAI models integrate into Qodo's architecture, Itamar highlights three distinct workloads. He details how their technical approach has transitioned from early prompt engineering, through flow engineering, into modern swarm engineering. While coordinating swarms can be difficult, narrowing their scope to targeted use cases—such as building a context-rich software map—yields substantial results through what he terms a "wisdom-based" approach.

Prompted by Danielle to define the distinction between intelligence and wisdom, Itamar outlines a hierarchy running from raw data to information, knowledge, intelligence, and finally wisdom: - **Raw Data to Intelligence**: Intelligence can be analogized to the brain itself—capable of processing and reasoning over inputs. - **The Jump to Wisdom**: Wisdom requires lived experience, akin to an entire physical body interacting with a complex environment over time. - **Codifying Experience**: Senior developers intuitively understand how large software architectures operate and break under real-world stresses. To run an end-to-end "software factory" where humans transition into managing swarms and policies rather than writing every line of code, this experiential wisdom must be explicitly codified and supplied directly to coding and review agents. Danielle reinforces this by noting that organizational wisdom often ends up being far more valuable than the raw code itself.

[03:21](https://www.youtube.com/watch?v=bQ3-pXCpCTg&t=201s) Catching Downstream Blast Radius with Sleep-Time Compute

Danielle asks for a concrete scenario where Qodo detects critical flaws that look entirely complete at first glance. Itamar points out that localized code quality is deceptively easy to get right; an agent or engineer can produce a snippet with zero semantic or syntax errors that nevertheless triggers a downstream catastrophe.

To illustrate, Itamar shares an example from a client that powers millions of small and medium businesses: - The platform offers high flexibility, giving merchant tools near-direct access to underlying databases. - A proposed code modification appeared completely safe when viewed strictly in isolation within the pull request diff. - Because of interconnected dependencies, the change actually created a downstream vulnerability capable of causing a massive site outage.

Human reviewers frequently miss these subtle architectural traps because humans are ill-equipped to scan entire codebases comprehensively in their heads. Itamar compares this to searching: a human might analyze a single web page better than a search engine, but no human can compete with Google when it comes to scanning an entire library. While standard coding agents operate strictly just-in-time, Qodo leverages "sleep time compute" running continually in the background to recalculate the broader software graph and expose systemic downstream risks before deployment.

[05:07](https://www.youtube.com/watch?v=bQ3-pXCpCTg&t=307s) The Evolution: Prompting, Flow Engineering, and Swarm Architecture

Reflecting on the evolution of LLM application design, Itamar and Danielle reminisce about 2023, when "prompt engineer" was treated as a distinct job title focused on back-and-forth question answering and coding competitions. By 2024, Qodo introduced their research paper *AlphaCodium*, demonstrating that structured flow engineering—codifying the actual end-to-end thought process of a human software engineer—far outperformed simple prompt tweaks.

Today, advanced models can generate many internal sub-workflows autonomously, shifting the engineer's role to swarm architecture: - **Explicit Swarm Architecture**: Designers must establish clear boundaries by defining explicit agent goals, toolsets, operational guardrails, and review policies. - **Risks of Autonomous Delegation**: Permitting primary agents to dynamically spawn sub-agents and arbitrarily select tools without strict boundaries remains premature, expensive, and wasteful—a concern shared by OpenAI researchers. - **Trust and Grounding**: For enterprise software systems requiring high trust, every agent output must remain rigorously grounded in real repository evidence and verifiable system data.

[07:21](https://www.youtube.com/watch?v=bQ3-pXCpCTg&t=441s) The Software Factory and the Inversion of Human Prompting

Addressing how engineering teams can verify whether an adversarial reviewer genuinely improves outcomes, Itamar outlines the vision of the modern "software factory." He predicts that by 2027, the traditional practice of a human developer reviewing endless static comments directly on a GitHub pull request page will feel archaic and impose an unacceptable cognitive burden.

Instead, the review agent should continuously provide real-time feedback directly to the coding agent while the code is being drafted. The coding agent and the review agent must first debate, verify, and resolve issues autonomously. Only when they reach an impasse should the problem be escalated to a human developer. Danielle shares an observation from an OpenAI API developer who described this paradigm shift: "I am no longer prompting Codex; Codex is prompting me." In this setup, success is measured by minimizing the number of times the human engineer needs to be interrupted to yield a production-ready, high-quality PR.

[08:56](https://www.youtube.com/watch?v=bQ3-pXCpCTg&t=536s) Resolving Disagreements and Handling Silent Failures

When coding agents and adversarial review agents disagree, Danielle asks what should be surfaced to the human and what context they need to make an informed call. Itamar cites the principle that two rational agents—human or synthetic—provided with the exact same information should naturally converge on the same conclusion. When discrepancies emerge between an authoring agent, an adversarial reviewer, and a human engineer, the divergence is driven almost entirely by discrepancies in context:

```mermaid flowchart TD accTitle: Grounding and Escalation Process accDescr: How discrepancies in context are reconciled through evidence grounding before human escalation. A["Coding Agent Output"] --> C{"Context Alignment\n& Grounding"} B["Adversarial Reviewer"] --> C

C -->|"Sufficient Grounded Evidence"| D["Autonomous Agreement"] C -->|"Irreconcilable Conflict"| E["Escalate to Human Developer"]

E --> F["Human Decision Grounded in Map Data"] F --> G["Merged to Production"] G -.->|"Post-Mortem / Bug Incident"| H["Update Wisdom & Guardrails"] H -.->|"Prevents Making Mistake Twice"| C ```

Drawing on research into generative-critic review networks, Itamar stresses that when both agents are grounded in identical, verifiable facts from the codebase, they almost always reach consensus.

Finally, Danielle raises the danger of false consensus: what happens when the coding agent and the reviewer agent both agree, but are both fundamentally mistaken? Itamar acknowledges that mistakes and production bugs will inevitably slip through ("sorry for my French, it will happen"). The true benchmark of a mature software factory is institutional learning: whenever an incident occurs or a bug is filed against code that passed mutual review, that failure must be fed back into the system to refine the software map and codify new rules so the system never makes the exact same mistake twice.

[11:01](https://www.youtube.com/watch?v=bQ3-pXCpCTg&t=661s) Building the "Wisdom Base" and Codifying Tribal Knowledge

The discussion centers on how software organizations must capture engineering experience. When mistakes are made, they need to be recorded intelligently into what Qodo terms the "wisdom base" or developer memory. Prior to modern AI, capturing that organizational context was exceptionally difficult because tribal knowledge typically lived in the heads of senior engineers—and when those individuals left the company, their context vanished with them. With AI-assisted continuous learning, mistakes can be codified permanently until they are intentionally pruned when no longer relevant.

The interviewer steps "a tiny bit off script" (prompting a lighthearted exchange: "Script? What are you talking about? No script!") to draw a parallel to her previous role at OpenAI, where she managed technical onboarding before joining the developer experience team. Tribal knowledge was an omnipresent operational headache. In response, Qodo's speaker illustrates the problem with a real-world case: a major financial institution in the Southeast managing hundreds of code repositories and countless microservices. Operating as a regulated financial institution striving toward an automated "software factory," they simply cannot afford critical production failures—"they cannot fuck up again, sorry for my French."

Each microservice handles a specific responsibility, but only a handful of long-tenured developers (some of whom have already departed) truly understood what works at scale. Qodo addresses this by pointing its wisdom base backwards: scanning years of discussions across GitHub, GitLab, Bitbucket, Slack, and Microsoft Teams to systematically codify that latent context. This historical analysis uncovers subtle, cross-service dependencies that aren't obvious in the codebase itself, such as data streaming pipelines where runtime schema and message flow define the real operational contract. Qodo maps these hidden relationships into an overarching "software map."

```mermaid flowchart TD accTitle: Tribal Knowledge Ingestion into the Qodo Wisdom Base accDescr: Historical developer communications and code repositories feed into Qodo's extraction pipeline to produce a permanent wisdom base and software map.

subgraph Sources["Historical Sources"] A["Version Control Discussions<br/>(GitHub / GitLab / Bitbucket)"] B["Team Communications<br/>(Slack / Microsoft Teams)"] C["Streaming & Runtime Data Flow"] end

subgraph QodoEngine["Qodo Knowledge Ingestion"] D["Context Mining & Parsing"] E["Cross-Service Dependency Analysis"] end

subgraph KnowledgeArtifacts["Living Memory"] F["Developer Wisdom Base<br/>(Codified Mistakes & Rules)"] G["Software Architecture Map"] end

A --> D B --> D C --> E D --> F E --> G F -.->|"Prune obsolete rules"| F ```

[13:56](https://www.youtube.com/watch?v=bQ3-pXCpCTg&t=836s) Reimagining the Task: From Pull Requests to Work Package Triage

The conversation pivots to the future of coding agents and workflow automation. Today's AI models are acknowledged as "the dumbest that they will ever be," yet agents are increasingly capable of sustained, long-running execution. Because agents will inevitably execute full end-to-end assignments reliably, development teams must rethink what constitutes a "task."

Historically, tooling limitations forced teams to treat a single pull request as the atomic unit of work through the software factory. However, software features are rarely self-contained within one PR; delivering a net-new capability typically spans 5 to 15 interrelated PRs across multiple repositories. Developers cannot effectively orchestrate or review such holistic changes on a standard GitHub pull request page. The speaker shares a preview of an upcoming release:

- **Work Package Triage:** A Qodo capability designed to analyze how multiple discrete PRs connect across repositories, allowing teams to review and validate an entire feature end-to-end as a single unified work package rather than disconnected diffs.

[15:44](https://www.youtube.com/watch?v=bQ3-pXCpCTg&t=944s) Practical Advice for Builders: Designing Agent Swarms

When asked what developers building with OpenAI should try right away, the speaker highlights the emerging potential of decision APIs for fast routing and classification. Drawing on their prior experience building embedding models at a previous company, the speaker emphasizes that the key architectural challenge is not choosing a single model, but orchestrating workloads efficiently across an agentic team.

```mermaid flowchart LR accTitle: Workload-Driven Swarm Architecture accDescr: A development task is decomposed into workloads, each handled by an optimized swarm of specialized agent bots using right-sized models.

Task["Broad Engineering Task"] --> Decomp["Task Decomposition"]

subgraph SwarmDesign["Workload-Specific Swarms"] Decomp --> W1["Workload A"] Decomp --> W2["Workload B"]

W1 --> Agent1["Specialized Bot 1<br/>(Fast / Decision Model)"] W1 --> Agent2["Specialized Bot 2<br/>(Reasoning Model)"]

W2 --> Agent3["Specialized Bot 3<br/>(Context Miner)"] W2 --> Agent4["Specialized Bot 4<br/>(Adversarial Reviewer)"] end

SwarmDesign --> Output["End-to-End Delivery<br/>(Optimized for Cost & Speed)"] ```

Builders are urged to adopt a structured methodology: 1. Deconstruct the overarching task into distinct, granular workloads. 2. Determine the right "swarm" structure for each workload, deploying a set of 2, 3, or 5 specialized agents ("your own bots") tailored to that slice of the problem. 3. Assign the appropriate underlying model to each player on the game board—pairing high-capacity models where complex reasoning is required and lightweight decision models where rapid triage suffices. 4. Continuously optimize model selection for cost efficiency, ensuring that running comprehensive automated reviews at enterprise scale remains economically viable.

[17:34](https://www.youtube.com/watch?v=bQ3-pXCpCTg&t=1054s) Full Circle and Concluding Thoughts

The session concludes on a reflective personal note. The interviewer shares that prior to OpenAI, she worked at Intuit conducting technical training, where Qodo happened to be one of the very first developer products she ever taught. Sharing the stage years later to discuss the evolution of AI-driven code reviews serves as a meaningful full-circle moment.

The speaker closes with a final playful warning to the audience about eliminating manual friction: while teams should strive to "kill your last human code review," they must ensure continuous training and growth mechanisms remain intact so their human developers continue learning alongside the machines.

## Notable quotes > [00:13](https://www.youtube.com/watch?v=bQ3-pXCpCTg&t=13s) Codex's best friend — Itamar Friedman

> [07:21](https://www.youtube.com/watch?v=bQ3-pXCpCTg&t=441s) I am no longer prompting Codex; Codex is prompting me. — Danielle

> [08:56](https://www.youtube.com/watch?v=bQ3-pXCpCTg&t=536s) sorry for my French, it will happen — Itamar Friedman

> [11:01](https://www.youtube.com/watch?v=bQ3-pXCpCTg&t=661s) Script? What are you talking about? No script! — Itamar Friedman

> [11:01](https://www.youtube.com/watch?v=bQ3-pXCpCTg&t=661s) they cannot fuck up again, sorry for my French — Itamar Friedman

> [13:56](https://www.youtube.com/watch?v=bQ3-pXCpCTg&t=836s) the dumbest that they will ever be

> [15:44](https://www.youtube.com/watch?v=bQ3-pXCpCTg&t=944s) your own bots — Itamar Friedman

> [17:34](https://www.youtube.com/watch?v=bQ3-pXCpCTg&t=1054s) kill your last human code review — Itamar Friedman

## Mentioned AlphaCodium, Bitbucket, Codex, Danielle, GitHub, GitLab, Google, Intuit, Itamar Friedman, Microsoft Teams, OpenAI, OpenAI DevDay, Qodo, Slack

## Video description See how Qodo uses OpenAI to review AI-generated code. Explore an adversarial reviewer that challenges coding agents’ assumptions and output, and learn when model confidence still needs human judgment.

[Source](https://www.youtube.com/watch?v=bQ3-pXCpCTg)

中文简报

Qodo CEO预测到2027年AI群集将终结人工PR审查 自主编码与审查智能体将通过结合机构记忆图谱解决争议来消除人工Pull Request审查，仅在模型陷入僵局时才转交工程师处理。 Qodo首席执行官Itamar Friedman与OpenAI的Danielle共同探讨了软件开发如何从人工Pull Request审查转向自动化的“软件工厂”。通过持续运行后台的“睡眠时间计算”，并将来自聊天记录和代码库的机构内隐知识提炼为动态的“智慧库”，Qodo创建了可捕捉下游架构影响范围的动态软件图谱。该架构不再依赖简单的提示词，而是将Codex等编码智能体与对抗性审查智能体配对组成受控群集，在开发者介入前自主辩论并优化代码。本次交流展望了将多代码库变更作为统一工作包处理的路径，旨在有效消除日常人工代码审查，同时保留由事件驱动的组织学习机制。 Qodo将后台“睡眠时间计算”与挖掘出的机构内隐知识相结合，与Codex等编码智能体协同提供实时对抗性代码审查，在提交PR前捕捉系统性下游影响范围。 (0:13) 行业方法论已从2023年的提示工程发展到2024年的结构化流程工程（AlphaCodium），并迈向明确且受控的群集架构，以防止无谓的子智能体任务委派。 (5:07) 在成熟的软件工厂模式中，编码智能体与审查智能体自主辩论并解决问题，从而反转了人工参与模式，使得Codex仅在上下文冲突导致僵局时才向人类发起提示。 (7:21) Qodo正将软件开发的原子单元从孤立的GitHub Pull Request转向“工作包分流”，使团队能够审查和验证跨越5到15个跨代码库PR的完整功能。 (13:56)

Reading summary

At OpenAI DevDay, Qodo CEO Itamar Friedman argued that software development is transitioning into an automated 'software factory,' where adversarial agent swarms and codified organizational wisdom replace manual human code reviews. Codifying Wisdom and Mitigating Downstream Blast Radius (1:22) Friedman distinguished between intelligence and wisdom, asserting that while raw model intelligence can process localized logic, wisdom requires lived experience under real-world operational stress. He noted that localized diffs often pass semantic and syntax checks while masking systemic downstream failures, such as a client case where an isolated change threatened to trigger a massive site outage across small and medium business merchant tools. To address these vulnerabilities, human reviewers—who cannot mentally hold an entire architecture—rely on 'sleep-time compute' that runs continually in the background to maintain an up-to-date software map and expose systemic blast radiuses. Structuring Swarms Over Prompt Tweaks (5:07) Tracing application architecture from 2023 prompt engineering through structured flow engineering—such as Qodo's 2024 AlphaCodium research—Friedman emphasized that builders must now implement explicit swarm architectures. Rather than allowing primary agents to dynamically spawn sub-agents and select arbitrary tools, which he and OpenAI researchers view as expensive, wasteful, and premature, enterprise systems require strictly defined agent boundaries, toolsets, and guardrails grounded in verifiable repository evidence. Adversarial Autonomous Review and Context Alignment (7:21) Friedman predicted that by 2027, manual human reviews on static GitHub pull requests will be obsolete. Instead, authoring agents and adversarial review agents should debate and verify code iteratively in real time, escalating to developers only when context conflicts arise. When agents or humans disagree, Friedman maintained the divergence stems almost entirely from discrepancies in context rather than reasoning flaws. In instances of false consensus where bugs reach production, the failure must be captured systematically so the system refines its guardrails and avoids repeating identical mistakes. Mining Tribal Knowledge and Expanding Beyond Pull Requests (11:01) To prevent institutional knowledge from disappearing as senior developers depart, Qodo scans historical communications across Slack, Microsoft Teams, GitHub, GitLab, and Bitbucket. This process maps latent runtime dependencies, such as streaming pipelines across microservices in heavily regulated financial environments. Looking ahead, Friedman argued that treating a single pull request as the atomic unit of work is obsolete, previewing a 'Work Package Triage' feature that bundles 5 to 15 interrelated PRs across repositories into a unified reviewable package. Architecting Workloads Across Right-Sized Model Swarms (15:44) For teams deploying multi-agent systems, Friedman recommended decomposing overarching tasks into targeted workloads handled by groups of two to five specialized agents. Builders should avoid relying on a single uniform model, instead pairing lightweight decision APIs for routing and triage with high-capacity reasoning models where complex analysis is required, thereby ensuring economic viability at enterprise scale. While Friedman advocated eliminating manual human code reviews, he cautioned that organizations must maintain continuous learning mechanisms so human developers continue building architectural intuition alongside autonomous agents.

Detailed notes — zh

## Background 本内容由 OpenAI 于 2026 年 10 月 9 日在 YouTube 官方频道发布。视频记录了 OpenAI DevDay 2026 期间的主题对谈，重点探讨 Qodo 如何借助 OpenAI 基础模型构建对抗性代码审查机制、挑战编码 Agent 的假设，以及在模型具备高置信度时为何仍需保留人类判断。 ## TL;DR Qodo 首席执行官 Itamar Friedman 与 OpenAI 开发者体验团队的 Danielle 深入剖析了 AI 时代代码审查范式的重构。Qodo 的架构经历了从提示工程、流程工程到群体工程（Swarm Engineering）的演进，核心在于利用“休眠算力”绘制全局软件拓扑图，避免局部代码正确却引发下游故障的隐患。系统通过将团队隐性“部落知识”转化为永久沉淀的智慧库，实现编码 Agent 与对抗性审查 Agent 内部辩论对齐，仅在出现实质分歧时反向提示人类仲裁。此外，Qodo 提出告别以单 PR 为中心的传统模式，转向跨多仓库的端到端“工作包”协同治理，旨在大幅减少人类认知过载并终结重复性的人工代码审查。 ## Atmosphere & style 整场对话兼具前沿技术的硬核拆解与老友重逢般的轻松默契，现场互动活跃而充满笑意。Itamar 的分享直率犀利且带着几分不拘小节的幽默，敢于直面多智能体共同犯错等真实系统工程窘境；Danielle 的控场与追问则精准切中一线开发者的痛点。随着尾声揭示两人早期的职场渊源，整场对谈在展望未来软件工厂的理性架构设计之余，流露出见证行业与彼此共同演进的温情闭环感。 ## Bulletin - [00:13](https://www.youtube.com/watch?v=bQ3-pXCpCTg&t=13s) Qodo 定位为 Codex 的核心治理与审查伙伴，通过利用“休眠算力”（sleep-time compute）持续绘制全量软件拓扑地图，有效预防局部正确但破坏下游依赖的架构漂移。 - [05:07](https://www.youtube.com/watch?v=bQ3-pXCpCTg&t=307s) 研发范式正从 Prompt 与 Flow Engineering 跃迁至 Swarm Engineering（群体工程），主张由专精 Agent 蜂群在后台基于事实证据对抗辩论，分歧时才提示人类介入仲裁。 - [07:21](https://www.youtube.com/watch?v=bQ3-pXCpCTg&t=441s) AI 协作衡量标准发生颠倒：从“人类提示 AI 多少次”转变为“人类被 AI 提问/打扰了多少次”，到 2027 年在 PR 页面逐条阅读 AI 审查评论的工作流将被淘汰。 - [13:56](https://www.youtube.com/watch?v=bQ3-pXCpCTg&t=836s) Qodo 推出“工作包分类分流”（Work Package Triage）功能，打破以单个 PR 为核心的局限，将跨代码库的 5 至 15 个关联 PR 聚合为端到端任务进行整体审查。 ## Key takeaways - [00:13](https://www.youtube.com/watch?v=bQ3-pXCpCTg&t=13s) Qodo 承担工程治理角色，核心职责在于提取团队非正式经验（tribal knowledge）、实时对抗审查代码生成，并绘制全景软件地图以监控架构漂移。 - [02:05](https://www.youtube.com/watch?v=bQ3-pXCpCTg&t=125s) “智能”与“智慧”的本质差异在于“经验”：资深工程师的不可替代性来自真实系统的踩坑历练，软件工厂的前提是将这些经验系统化编码并赋予 Agent。 - [03:21](https://www.youtube.com/watch?v=bQ3-pXCpCTg&t=201s) 代码常出现局部正确但引发全局下游雪崩的问题；Qodo 借助休眠算力（sleep-time compute）预先计算全量调用图谱，解决即时分析导致的盲区。 - [05:07](https://www.youtube.com/watch?v=bQ3-pXCpCTg&t=307s) 从《AlphaCodium》固化人类思考流水线的流程工程演进至今，群体工程（Swarm Engineering）的核心在于精确定义专精 Agent 的边界、护栏与证据链，而非放任其失控派生。 - [07:21](https://www.youtube.com/watch?v=bQ3-pXCpCTg&t=441s) 审查机制前置化：编码 Agent 与审查 Agent 在后台预先对抗辩论，唯有产生实质分歧时才唤醒人类仲裁，大幅降低开发者的认知过载。 - [08:56](https://www.youtube.com/watch?v=bQ3-pXCpCTg&t=536s) 双 Agent 分歧多源于信息切片不一致，需通过“事实锚定”（grounding）严格绑定代码与契约证据来消解冲突。 - [08:56](https://www.youtube.com/watch?v=bQ3-pXCpCTg&t=536s) 面对系统全部误判导致线上事故（incident）的情况，核心诉求不是幻想“永不犯错”，而是将错误沉淀为智慧库闭环，确保“同一错误不犯第二次”。 - [11:01](https://www.youtube.com/watch?v=bQ3-pXCpCTg&t=661s) 通过扫描代码托管平台与聊天工具（Slack/Teams）沉淀的历史沟通记录，Qodo 将即将流失的隐性部落知识转化为永久性的结构化智慧库与跨服务调用拓扑。 - [13:56](https://www.youtube.com/watch?v=bQ3-pXCpCTg&t=836s) 重新定义任务单元：端到端业务交付往往跨越多个仓库、涉及 5 至 15 个 PR，必须从单 PR 审查转向多 PR 聚合的“工作包”（Work Package）统一治理。 - [15:44](https://www.youtube.com/watch?v=bQ3-pXCpCTg&t=944s) 智能体落地工程实践：针对不同细分工作负载组建由 2 到 5 个微型 Agent 构成的蜂群，并根据决策复杂度匹配不同成本梯度的模型以兼顾准确率与 ROI。 - [17:34](https://www.youtube.com/watch?v=bQ3-pXCpCTg&t=1054s) “终结人类代码审查”不等于完全脱离人类：即使日常合并卡点被 AI 接管，依然必须保留对工程师的专业培训以及必要的人机协同闭环。 ## Detailed walkthrough ### [00:13](https://www.youtube.com/watch?v=bQ3-pXCpCTg&t=13s) Qodo 定位：Codex 的最佳拍档与软件地图治理 活动由 OpenAI 开发者体验团队的 Danielle 主持开场。在 DevDay 现场轻松热烈的氛围中，她邀请 Qodo 首席执行官 Itamar Friedman 上台，深入探讨 Qodo 的核心定位与服务对象。Itamar 指出，客户常把 Qodo 形容为 "Codex 的最佳拍档"（Codex's best friend）。在软件研发的生命周期中，Qodo 承担着三大核心职责：首先是从工程团队的非正式经验与潜规则（tribal knowledge，部落知识）中萃取上下文并注入 Codex，同时利用这些知识在 Codex 编码过程中提供实时审查，确保从源头上就能生成干净、清晰的 Pull Request（PR）；其次，Qodo 扮演着工程治理层的角色，负责绘制全量“软件地图”（software map）并监控其随时间的架构漂移（drift），从而为工程团队提供可控性。 谈及 OpenAI 基础模型与 Codex 如何整合进 Qodo 的审查流程，Itamar 解释称 Qodo 针对三个独特的工作负载搭建了定制化系统架构。Qodo 的架构演进经历了三个明确阶段：从最初的提示工程（prompt engineering）迈向流程工程（flow engineering），最终发展到当前的群体工程（swarm engineering）。尽管行业普遍认为智能体蜂群的协作极其困难，但当集中于明确边界的场景时能释放巨大价值。其中一个至关重要的场景，正是利用所谓的“基于智慧”（wisdom-based）机制来构建软件拓扑地图与工程上下文，用于支撑代码生成与审查。 ```mermaid flowchart LR accTitle: Qodo 治理与审查架构流向 accDescr: 展示从团队隐性知识到上下文注入、编码审查与软件地图漂移监控的完整闭环。 subgraph Data["知识沉淀"] TK["工程团队隐性知识<br/>(Tribal Knowledge)"] SM["全局软件地图<br/>(Software Map)"] end subgraph Core["协作引擎"] QD["Qodo 治理与上下文层"] CX["Codex 编码 Agent"] CR["Qodo 审查 Agent"] end subgraph Output["交付物与监控"] PR["高质量干净 PR"] DF["架构漂移监控<br/>(Drift Governance)"] end TK --> QD QD -->|"注入业务上下文"| CX CX <-->|"实时对抗与审查"| CR CR --> PR PR --> SM SM --> DF ``` - **Qodo 的核心职责**：提取团队隐性知识、实时审查 AI 编码过程、绘制架构地图以防软件漂移。 - **技术范式演进路径**：Prompt Engineering（2023） $\to$ Flow Engineering（2024） $\to$ Swarm Engineering（当下）。 --- ### [02:05](https://www.youtube.com/watch?v=bQ3-pXCpCTg&t=125s) “智能”与“智慧”的区别：隐性经验的系统化编码 Danielle 深入追问了“基于智慧”（wisdom-based）的深层含义，以及智能（intelligence）与智慧（wisdom）之间究竟有何本质差异。对于代码审查而言，究竟有哪些组织层面的维度是单凭阅读代码本身无法习得的？Itamar 引用了认知金字塔的递进过程：从底层原始数据（raw data），到信息（information），再到知识（knowledge）和智能（intelligence），而跃迁到“智慧”的关键催化剂在于“经验”（experience）。 Itamar 打了个形象的比方：智能就像是大脑本身，而经验必须依赖一具完整的身体在物理或业务系统中经历历练才能形成。资深软件工程师之所以不可替代，是因为他们拥有通过长年系统踩坑、运维事故和架构演进所沉淀下的真实经验。若要构建端到端的“软件工厂”（software factory），让人类退居幕后专心制定策略与管理智能体蜂群，核心前提就是把这些工程智慧与业务经验沉淀、编码化，并直接赋予编码 Agent 和代码审查 Agent。 --- ### [03:21](https://www.youtube.com/watch?v=bQ3-pXCpCTg&t=201s) 局部正确性 vs 下游隐患：休眠算力构建全局软件图谱 为了让技术价值更具象，Danielle 提到了代码看似毫无纰漏却潜藏重大隐患的情形。Itamar 直言这种情况屡见不鲜，不仅 AI 容易犯，人类工程师也常掉入这一陷阱：一段代码在局部（locally）审视时完全挑不出毛病，语法规范、语义无误；然而一旦置于全局软件系统当中，就会对下游依赖（downstream surface）产生致命的连锁破坏，而大多数编码 Agent 根本无法预见这种后果。 他分享了一个标杆客户的真实案例：该客户为数百万中小企业提供极其灵活的定制化建站与业务搭建服务，灵活性极高，甚至允许部分功能直接与底层数据库交互。因此，该团队大量的代码变更都会触碰核心数据库。如果只从 PR 的局部变更来看，代码逻辑毫无瑕疵；但由于某处修改破坏了未显式关联的下游服务，极易在生产环境引发全盘宕机事故。Qodo 正是凭借后台的“休眠算力”（sleep-time compute）持续运算、构建并更新完整的全局软件调用图谱（software graph），成功阻截了这类故障。Itamar 形象地对比道：这就好比让人类去理解单一网页，人类能胜过搜索引擎；但若要扫描通读并理清一整个庞大图书馆的交叉引用关系，人类根本无法匹敌机器。 - **局部感知与全局感知的矛盾**：编码 Agent 和人类审查者往往受限于上下文局部窗口，难以洞察跨服务、跨数据库的下游雪崩效应。 - **休眠时间算力（Sleep-time Compute）**：Qodo 在后台持续运行分析任务，在代码尚未进入审查环节前就已经计算好全量软件拓扑图，避免仅依靠即时计算（just-in-time）导致的漏判。 --- ### [05:07](https://www.youtube.com/watch?v=bQ3-pXCpCTg&t=307s) 从 AlphaCodium 流程工程到可信群体工程（Swarm Engineering） 谈及工程范式的跃迁，Itamar 回顾了技术迭代的飞速步伐。在 2023 年，“提示词工程师”（prompt engineer）甚至是一个独立的职业头衔，当时大模型只能完成简单的问答交互以及初级的编程竞赛题目。到了 2024 年，Qodo 团队发表了开创性的论文《AlphaCodium》，提出不能再把代码生成当作单轮或多轮的简单一问一答，而是应该构建“流程工程”（flow engineering）——将人类专业开发人员在头脑中构思架构、编写测试、逐步迭代排错的完整思考流水线用代码固化下来。 然而行至当下，基础模型已经具备自行规划并构建工作流的能力。此时最具工程价值的环节，转向了群体工程（swarm engineering）的高级架构设计：开发者需要精确定义不同专精 Agent 的职责边界、目标定位、可用工具链、安全护栏（guardrails）以及治理策略。Itamar 特别提及在本次 DevDay 上 OpenAI 专家的警告：目前若完全放任主 Agent 随意派生子 Agent 并自行探寻工具，极易导致失控与算力浪费，技术仍处于极早期。对于企业级产品而言，其生命线在于“信任”，所有产出必须基于确凿数据并附带证据链（evidence-backed）。因此，当下最理性的工程路径是为模型划定清晰的大任务框架，而不是完全无约束地自生智能体。 --- ### [07:21](https://www.youtube.com/watch?v=bQ3-pXCpCTg&t=441s) 软件工厂的目标：Agent 提示人类，而非人类提示 Agent 针对“如何量化代码审查 Agent 真正改善了工程交付”这一核心问题，Itamar 引入了“软件工厂”的成熟度维度。在最原始的初级阶段，代码审查工具往往只是机械地在 GitHub PR 页面上堆砌告警与建议。Itamar 大胆断言："到 2027 年，在 PR 页面逐条读 AI 评论的做法将显得滑稽可笑"，因为那会带来极其沉重的认知过载。 未来的审查机制必须前置，让审查 Agent 在开发者与编码 Agent 编写代码的同时实时介入并给出反馈。理想的工作流是：编码 Agent 与对抗性审查 Agent（code review agent）在后台先行辩论，达成一致；只有在两者出现实质性分歧时，才打扰人类开发者介入仲裁。衡量系统优劣的关键指标因此发生了颠倒：不再是人类给 AI 提供了多少次 Prompt，而是**人类被 AI 提问/打扰了多少次（how many times the developer was prompted）**。人类被唤醒做决定的次数越少、产出的 PR 质量越高，软件工厂的效能就越高。Danielle 对此深有同感，并转述了一位 OpenAI 内部 API 开发者的原话：“我现在不再去提示 Codex 了，是 Codex 在反过来提示我。” ```mermaid flowchart TD accTitle: 对抗审查与人类仲裁流程 accDescr: 展示编码智能体与审查智能体在证据链支撑下辩论，达成共识则直接生成PR，发生分歧则呼叫人类裁决的逻辑。 subgraph DualAgent["双智能体对抗阶段"] CA["Coding Agent<br/>(生成代码)"] RA["Review Agent<br/>(对抗审查)"] end subgraph ConsensusCheck{"是否达成共识？"} Branch{"两者观点一致？"} end subgraph Resolution["解决路径"] CleanPR["自动生成高质量 Clean PR"] HumanPrompt["Agent 反向提示人类工程师<br/>(基于事实证据请求裁决)"] end CA <-->|"基于事实证据 (Grounding) 辩论"| RA CA --> Branch RA --> Branch Branch -- "是" --> CleanPR Branch -- "否" --> HumanPrompt ``` --- ### [08:56](https://www.youtube.com/watch?v=bQ3-pXCpCTg&t=536s) 分歧裁决与“两者皆错”的兜底保障 探讨深入到异常分支：当编码 Agent 与审查 Agent 发生分歧时，该如何向人类呈现信息？当它们达成共识却双双犯错时，又该如何构建防护网？针对分歧，Itamar 引用了博弈论与决策理论：两个具备理性推理能力的智能体（无论人类或 AI），在拥有完全相同的信息上下文时，理应得出相同的结论。编码 Agent 与审查 Agent 之所以产生冲突，往往是因为彼此掌握的上下文切片不一致。因此解决争端的关键在于“事实锚定”（grounding）——利用生成式对抗/批判网络（critic/generative patterns）的研究机制，将所有的争论点都严格绑定到具体的代码行、接口契约与事实证据上，一旦数据充分对齐，系统内部往往就能自行消解分歧。 随后 Danielle 抛出了更棘手的情况：“如果它们达成了一致，但两者全都错了呢？”面对这一尖锐问题，Itamar 坦诚而幽默地回应道：“这种破事确实会发生（shit happens, pardon my French），而且未来必定还会发生。”当编码 Agent、审查 Agent 乃至一线开发者和人工 Reviewer 全部被蒙蔽，导致缺陷悄悄流入生产环境并引发线上故障（incident）时，终极的系统性诉求绝非妄想“永不犯错”，而是“同样的错误绝不犯第二次”（we don't do a mistake twice）。工程体系必须具备将线上事故、Bug 反馈自动沉淀回业务智慧库的能力，使系统在闭环中持续强化防御。 ### [11:01](https://www.youtube.com/watch?v=bQ3-pXCpCTg&t=661s) 从隐性“部落知识”到永久沉淀的“智慧库” 知识与教训的沉淀是软件工程中最顽固的痛点之一。在 AI 普及之前，团队高度依赖资深工程师口耳相传的“部落知识”（tribal knowledge）。一旦核心员工离职，这些关于系统边界和踩坑经验的记忆便随之蒸发。Qodo 正在构建的不仅是基础知识库，更是一个具备持续学习能力的“智慧库”（wisdom base）。系统会将过往犯过的错误与踩坑细节固化为代码级规范，永久存入开发者的“智慧记忆”中，直到该规范因业务演进而被主动修剪剔除。 主持人在现场幽默地表示要“稍微脱离一下提纲”，嘉宾随即打趣反问：“什么提纲？我们哪有提纲！”主持人提到自己此前在 OpenAI 负责技术入职培训（technical onboarding），后来转岗至开发者体验（DevEx），期间听到最多的痛点就是部落知识流失。针对知识库如何取代部落知识的提问，Qodo 创始人分享了一个大型金融机构的典型案例：该客户拥有数百个代码仓库与庞杂的微服务架构，业务容错率为零。随着经历过系统搭建的老员工离职，没人敢轻易改动跨服务逻辑。Qodo 的智慧库通过向后扫描 GitHub、GitLab、Bitbucket 以及 Slack 和 Microsoft Teams 中沉淀多年的讨论记录，将隐性知识形式化，梳理出服务之间不易被静态代码察觉的调用依赖与数据流拓扑。 ```mermaid flowchart TD accTitle: 部落知识转化为智慧库的流程 accDescr: 展示从非结构化历史沟通记录到结构化软件地图与智慧库的转化过程。 subgraph Sources["多源历史数据挖掘"] VCS["代码托管平台讨论<br/>GitHub / GitLab / Bitbucket"] IM["即时通讯沟通记录<br/>Slack / Microsoft Teams"] end subgraph Mining["语义解析与依赖映射"] Analyze["扫描历史错误与决策脉络"] DataFlow["解析隐式数据流与跨服务连接"] end subgraph WisdomEngine["Qodo 智慧库 (Wisdom Base)"] Map["生成全景软件地图 (Software Map)"] Rules["固化代码规范与避坑规则"] Prune["动态修剪过时知识"] end Sources --> Analyze Analyze --> DataFlow DataFlow --> Map Map --> Rules Rules --> Prune ``` - **金融级容错要求**：金融客户希望推进“软件工厂”式的高效流水线，但受监管与业务性质约束，“绝不能再次搞砸”（"cannot fuck up again, sorry for my French"）。 - **隐式跨服务依赖**：微服务间的调用常经由消息流或流式数据中转，仅凭静态代码难以直接追踪影响面，需要结合历史讨论与实时数据流建立全景软件地图。 --- ### [13:56](https://www.youtube.com/watch?v=bQ3-pXCpCTg&t=836s) 告别 PR 局限：以完整“工作包”重构研发任务 针对“今天的模型将是未来最笨的模型”这一行业共识，开发者需要重新定义何为软件工程中的“任务”（Task）。长期以来，开发者受限于工具与版本控制机制，习惯将一个拉取请求（Pull Request，PR）视作一个标准处理单元。然而，当 AI 智能体具备自主执行长流程任务的能力时，用户真正关心的并非单个 PR 的合并，而是能否端到端交付一项完整的新功能或新业务能力。 这项新能力往往跨越多个仓库，涉及 5 到 15 个相互关联的 PR。传统的 GitHub PR 页面根本无法支持这种全局视角下的上下文沟通与代码审查。为了打破这种割裂，Qodo 即将发布名为“工作包分类分流”（Work Package Triage）的功能，帮助工程团队将分散的关联 PR 聚合为一个完整的“任务整体”，进行统一的上下文理解与代码审查。 ```mermaid flowchart LR accTitle: 从单个 PR 到端到端工作包任务的演进 accDescr: 对比传统单 PR 处理模式与未来工作包协同模式的差异。 subgraph Legacy["传统模式：以单个 PR 为中心"] PR1["PR #1 (单一服务)"] PR2["PR #2 (配置变更)"] PR3["PR #3 (前端适配)"] end subgraph Future["Qodo 演进：工作包分类分流 (Work Package Triage)"] WP["全局功能任务 (Work Package)"] Swarm["多 PR 关联上下文理解"] UnifiedReview["端到端整体代码审查"] end Legacy -.->|"重塑任务边界"| WP WP --> Swarm Swarm --> UnifiedReview ``` --- ### [15:44](https://www.youtube.com/watch?v=bQ3-pXCpCTg&t=944s) 智能体蜂群架构与决策 API 的工程实践 面向在现场使用 OpenAI 技术构建应用的开发者，Qodo 创始人给出了极具操作性的架构设计建议，并特别强调了对决策 API（Decision API）前景的看好。他在创立 Qodo 之前曾在向量嵌入模型公司任职，深谙高效决策模型对于复杂推理链路的重要性。面对复杂的开发负载，开发者不应试图用单一巨型模型解决所有问题，而应构建分工明确的智能体蜂群（Swarm）。 高效落地的关键在于三步走：首先重新拆解任务的工作负载（Workloads）；其次为每个负载定制专属的智能体群（例如 2 到 5 个专注于特定子任务的 Agent）；最后依据每个智能体在“棋盘”上的角色精细匹配不同成本与能力梯度的底层模型。由于审查与交付任务极其频繁，只有兼顾高准确率与极致的成本效益（Cost efficiency），AI 代码审查方案才具备规模化落地的可行性。 - **任务拆解原则**：将宏观目标细分为具体的细分工作负载，避免单个 Agent 上下文过载。 - **智能体蜂群配置**：针对特定工作负载组建由 2 至 5 个微型智能体构成的 Swarm 协同作战。 - **成本与模型分级**：依据决策复杂度分配顶尖推理模型或轻量高速模型，确保高频调用的 ROI。 --- ### [17:34](https://www.youtube.com/watch?v=bQ3-pXCpCTg&t=1054s) 闭环时刻与“终结最后一次人类代码审查” 访谈尾声流露出深厚的个人渊源。主持人动情地分享道，自己在加入 OpenAI 之前曾在 Intuit 负责技术培训，而 Qodo 正是她当年最早进行技术培训的产品之一。如今能够以 OpenAI 员工的身份在 DevDay 舞台上共同探讨代码审查的未来，有一种见证双方成长与行业演进的“圆满闭环时刻”（full circle moment）。 嘉宾最后向全场开发者送上祝福，幽默地鼓励大家“祝大家顺利消灭自己的最后一次人类代码审查”（"Good luck on killing your last human code review"），但同时也认真提醒团队：即便 AI 审查接管了日常合并卡点，依然必须保留对开发人员的持续工程培训以及必要的人机协同闭环（Human in the loop）。 ## Notable quotes > [00:13](https://www.youtube.com/watch?v=bQ3-pXCpCTg&t=13s) Codex 的最佳拍档 — Itamar Friedman > [07:21](https://www.youtube.com/watch?v=bQ3-pXCpCTg&t=441s) 到 2027 年，在 PR 页面逐条读 AI 评论的做法将显得滑稽可笑 — Itamar Friedman > [07:21](https://www.youtube.com/watch?v=bQ3-pXCpCTg&t=441s) 我现在不再去提示 Codex 了，是 Codex 在反过来提示我。 — Danielle > [08:56](https://www.youtube.com/watch?v=bQ3-pXCpCTg&t=536s) 这种破事确实会发生（shit happens, pardon my French），而且未来必定还会发生。 — Itamar Friedman > [08:56](https://www.youtube.com/watch?v=bQ3-pXCpCTg&t=536s) 同样的错误绝不犯第二次（we don't do a mistake twice） — Itamar Friedman > [11:01](https://www.youtube.com/watch?v=bQ3-pXCpCTg&t=661s) 绝不能再次搞砸（cannot fuck up again, sorry for my French） — Itamar Friedman > [13:56](https://www.youtube.com/watch?v=bQ3-pXCpCTg&t=836s) 今天的模型将是未来最笨的模型 — Itamar Friedman > [17:34](https://www.youtube.com/watch?v=bQ3-pXCpCTg&t=1054s) 祝大家顺利消灭自己的最后一次人类代码审查（Good luck on killing your last human code review） — Itamar Friedman ## Mentioned Qodo, OpenAI, Danielle, Itamar Friedman, Codex, AlphaCodium, GitHub, GitLab, Bitbucket, Slack, Microsoft Teams, Intuit, DevDay 2026 ## Video description See how Qodo uses OpenAI to review AI-generated code. Explore an adversarial reviewer that challenges coding agents’ assumptions and output, and learn when model confidence still needs human judgment. [Source](https://www.youtube.com/watch?v=bQ3-pXCpCTg)

Source quotes

[0:13] Itamar Friedman: Codex 的最佳拍档 [7:21] Itamar Friedman: 到 2027 年，在 PR 页面逐条读 AI 评论的做法将显得滑稽可笑 [7:21] Danielle: 我现在不再去提示 Codex 了，是 Codex 在反过来提示我。 [8:56] Itamar Friedman: 这种破事确实会发生（shit happens, pardon my French），而且未来必定还会发生。 [8:56] Itamar Friedman: 同样的错误绝不犯第二次（we don't do a mistake twice） [11:01] Itamar Friedman: 绝不能再次搞砸（cannot fuck up again, sorry for my French） [13:56] Itamar Friedman: 今天的模型将是未来最笨的模型 [17:34] Itamar Friedman: 祝大家顺利消灭自己的最后一次人类代码审查（Good luck on killing your last human code review）

Source quotes

[0:13] Itamar Friedman: Codex's best friend [7:21] Danielle: I am no longer prompting Codex; Codex is prompting me. [8:56] Itamar Friedman: sorry for my French, it will happen [11:01] Itamar Friedman: Script? What are you talking about? No script! [11:01] Itamar Friedman: they cannot fuck up again, sorry for my French [13:56] the dumbest that they will ever be [15:44] Itamar Friedman: your own bots [17:34] Itamar Friedman: kill your last human code review

Original transcript

[0:13] Hello, hi everyone, welcome, welcome. [0:17] My name is Danielle, I work on [0:18] the developer experience team here. [0:20] And today we are going [0:21] to start our interview. [0:23] So for those of you, anyone passing by, [0:25] feel free to come on in, hang out. [0:28] So I'd love to introduce the CEO [0:30] of Qodo, Itamar Friedman. [0:32] Come join me. [0:36] All right, have a seat. [0:41] All right, [0:42] welcome, Itamar. [0:43] Such a beautiful day. [0:45] It is, thank goodness. [0:46] So before we get into [0:48] the technology, can you tell us [0:50] a little bit about what Qodo is [0:52] and who you're building for? [0:54] Sure. So our customers are describing [0:56] Qodo as Codex's best friend. [0:58] Basically, Qodo is collecting [1:00] context from tribal knowledge, [1:02] engineering tribal knowledge, and give [1:04] it into Codex, as well as using [1:06] that to review Codex while it's coding. [1:09] We want to help building a clear, [1:12] clean PR to begin with. [1:14] And eventually, it's a governance [1:16] layer showing the software map [1:18] and how it drifts over time, [1:19] because we need to control it. [1:22] And let's talk for a moment, [1:24] we're here at OpenAI DevDay, [1:25] about how OpenAI models and Codex fit [1:27] into Qodo's review workflow today. [1:30] And what has Qodo built around them? [1:32] Yeah, sure. So as I mentioned, we have three [1:35] workloads, and each one [1:37] of them is quite unique. [1:39] We actually moved from [1:41] flow engineering to, sorry, [1:43] from prompt engineering to flow [1:44] engineering to swarm engineering. [1:45] And I think we heard here [1:46] in the talks that it's quite hard. [1:48] But when you focus to swarm on [1:50] specific, you know, use cases, [1:53] you can get actually get a lot. [1:54] And one of those use cases, as [1:56] I mentioned, is creating that software [1:58] map, the context for proper [2:00] code generation and review. [2:02] We actually call that wisdom-based. [2:05] Okay, so you talk about wisdom-based. [2:08] And can you tell us the difference [2:10] between intelligence [2:11] and wisdom? [2:15] What does a reviewer need [2:17] to know about an engineering [2:18] organization that it really can't [2:21] learn from just the code alone? [2:24] Yeah, sure. [2:24] So I think like, in general, [2:26] we have like, [2:27] raw data, and then there's [2:28] information, and there's [2:30] knowledge, and then intelligence. [2:31] And I think the jump to wisdom [2:33] is about experience. [2:34] I think like, intelligence [2:36] could be like the brain itself, [2:37] if I may, if you allow me. [2:39] And experience could be only with like [2:41] a full system, a full body. [2:43] And I think like as a very [2:45] professional developers in front [2:47] of us, like, what makes us [2:48] unique is that we experience, [2:51] like we have understanding of how [2:53] the software works through experience. [2:55] And that's what you need [2:56] to codify if you want to provide [2:59] that directly to the coding [3:00] agent and code review agent. [3:01] Basically, if you want to build [3:03] a software factory end-to-end [3:06] and let human deal more [3:09] with the policies and managing [3:11] the swarm, you have to codify [3:12] that wisdom, that experience. [3:14] Yeah. So the wisdom actually ends up [3:16] being maybe more valuable in some [3:18] cases than the code itself. [3:20] Yeah. [3:21] Okay. I want to talk about an example [3:23] where code really shines. [3:25] So can you walk us through [3:26] a time when a change maybe [3:28] looked complete until [3:30] the reviewer found a problem? [3:32] Yeah, sure. [3:33] It happens a lot. [3:34] I think also to us as humans, right? [3:37] So you look on the [3:38] code and it looks locally really good. [3:41] There's no semantic issues or so. [3:44] But then when you're thinking [3:46] about the software as a whole, [3:47] you might create a downstream problem. [3:51] That's really, really hard for a lot [3:53] of the coding agent to find. [3:54] So giving an example of one [3:55] of her customers, they're serving [3:57] millions of small and medium businesses [3:59] and letting them build whatever, [4:01] but whatever they're enabling [4:03] their small businesses is very flexible. [4:05] And something that those [4:06] customers can do can almost [4:09] direct access to the databases. [4:10] So a lot of their code changes is [4:13] actually touching databases. [4:16] And again, locally, it might look good. [4:18] But then when [4:20] you consider a downstream [4:23] surface that actually can [4:26] go get a huge outage [4:29] because of change you look locally is [4:32] something that Qodo helps to find. [4:34] And it's really hard [4:35] also for human reviewers. [4:37] It's like almost equivalent of if I [4:39] go and compete with Google understanding [4:41] of a search engine, understanding on [4:43] a page something, I will do better. [4:45] But if I'm competing on a library, [4:47] like scanning the library, I won't. [4:49] And that's exactly where [4:51] it's hard for humans. [4:52] And like the coding agents are just [4:54] running just in time while [4:57] having a workflow that is running [4:59] on the background sleep time [5:01] compute and always calculating [5:03] that software map, software graph. [5:07] And a little earlier, you kind [5:09] of mentioned prompt engineering, [5:13] flow engineering, swarm engineering. [5:15] So can you tell, well, firstly, can [5:17] you maybe like describe what that is [5:19] a little bit and talk about what [5:21] has changed as you've moved from [5:22] those different types of engineering? [5:23] Do you remember those days [5:25] that there was like a profession [5:26] prompt engineering? [5:28] And that's, that's how we [5:31] helped building like [5:34] a simple like question and answering [5:37] that's what what the model were capable. [5:40] And that was [5:41] also what was used in order [5:43] to compete in coding competition. [5:44] We're talking about 2023. [5:46] In 2024, we came out [5:48] with a paper called Alpha Codium [5:51] that what we described there is [5:53] that we shouldn't think about like [5:54] back and forth question answering [5:56] and prompting that rather thinking about [5:58] it as a whole whole flow that you [6:00] should engineer like basically download [6:03] how you work, for example, [6:05] as a developer and engineer, [6:07] engineer that that's flow engineering. [6:10] But then today, the models [6:12] are actually quite capable [6:14] in building their own workflows. [6:16] So what's still [6:19] there and that's very valuable is [6:22] designing the swarm as a whole, [6:24] like having different agents [6:26] and defining exactly what are [6:28] their goals, what are their tools, [6:31] how, what are their guardrails, [6:32] what are their policies and all that. [6:35] That's if you try to let the agent [6:37] like sub, send their own sub agent [6:39] and they're find their own tools. [6:41] I even was in a talk [6:42] that one of the people from OpenAI [6:45] warned about that, that it's [6:46] still very early and could be [6:48] like very wasteful. [6:51] Specifically when you're thinking [6:53] about a product that's [6:54] supposed to bring trust, [6:56] that you need like the output to be [6:59] backed by data and you need [7:02] to design like swarms that are [7:03] like have evidence to their work. [7:06] So that's where I recommend everyone [7:09] going like thinking about like you [7:11] know where where the models are being [7:13] able to complete tasks and you want [7:15] to help them to focus on what [7:17] are what are the tasks, the big [7:19] tasks that they need to do. [7:21] Yeah okay and then [7:22] so moving [7:26] on to the reviewer, [7:28] how how do you actually [7:29] know that the reviewer is [7:32] improving the outcome? [7:34] What are the signs? [7:35] Yeah it's a really good question. [7:38] I'll focus on a software [7:41] factory notion because I think [7:43] like we're all on our journey [7:45] to build our software factory [7:46] but depends where you are. [7:48] If you're really in the early stage [7:49] then probably like your code [7:51] review tool is giving you [7:53] findings on the pull request page. [7:56] I think that by 2027 that will [7:58] look ridiculous to us. [8:00] It's so much mental load so basically [8:03] we need the code review for example [8:04] to give that feedback to the agents [8:06] in the developer while they're coding [8:08] and the first thing is that the coding [8:11] agent and the code review they need [8:12] to agree among themselves and then [8:14] only when they don't agree then they [8:16] bring it to the to the developer [8:18] and and I think the the least [8:21] the the if the developer [8:22] the minimum you can you can measure [8:24] how many times that developer [8:26] was needed to be prompted. [8:27] So instead of the developer [8:29] prompting the agents how like [8:31] the agents are prompting the the human [8:33] and and you ask yourself how many [8:34] times developers would needed to be [8:36] prompt to complete a high quality PR. [8:39] Yeah, no, I actually think [8:41] this is a really interesting point. [8:42] I've heard [8:43] an API developer at OpenAI he said [8:46] to me something along the lines of [8:48] I am no I'm no longer asking [8:51] Codex or prompting Codex [8:53] Codex is prompting me. [8:55] Yeah [8:56] And so coming to this point [8:57] about the coding agent [8:59] and the reviewer agreeing [9:01] when they do disagree with each [9:03] other which inevitably [9:04] will happen What should [9:07] reach a human and what [9:10] is that person going to need [9:12] in order to make a decision? [9:14] Yeah, totally. [9:15] So [9:17] First of all, I think like there's [9:19] a theory saying that two agents [9:21] humans or artificial that have [9:24] the same amount the same information [9:26] and they both are reasonable people uh, [9:30] then they should reach to the same [9:31] conclusion and the idea here is [9:33] I think the reason is that coding [9:35] agent is basically reached [9:38] different conclusion than a [9:40] Adversarial agent is the context [9:42] they have and also the human agent [9:44] the context they have and then [9:46] The real reviewer at the end [9:48] of the pull request that hopefully [9:49] he or she got a clean PR [9:50] the context they have So I think [9:52] what will help to work this out [9:56] Is if everything is grounded [9:58] like everything like there's [10:00] actually papers about it If you're [10:01] if you want to go and read [10:03] about the reviewer critic generative [10:06] like networks that Everything is [10:08] grounded then eventually they they [10:10] should get to agreement should [10:14] Okay [10:15] So [10:16] Maybe they agree with each other [10:19] That's great [10:20] What if they agree? [10:22] But they're both wrong [10:23] What is going to catch [10:25] those specific cases? [10:27] When is a human still [10:29] going to need to step in? [10:31] So Uh should happen uh, [10:33] sorry for my french Uh, [10:35] and it will happen And I think [10:37] what We want to achieve is [10:40] that we don't do a mistake twice [10:42] So we could reach to a point [10:44] where the coding agent agrees [10:46] with the With the review agent [10:48] and then eventually developers agree [10:50] and the reviewer agree And then [10:52] like uh, it goes to production [10:53] But then something happens, right? [10:56] Like there is some incident some [10:58] bug that was opened or or so [11:01] And then that has to be recorded [11:04] in a really smart way Into [11:06] the knowledge base and what [11:07] we call the wisdom base Uh, [11:09] so I think that's the potential It's [11:11] so hard to do pre-ai But actually [11:13] that's a potential post-ai It's [11:15] really hard to uh, those the tribal [11:17] knowledge sometimes it's like Uh [11:21] People accumulate them uh, and then [11:23] maybe they even leave the company [11:25] Right, and and I think like what's [11:27] really interesting like one of the most [11:29] promising thing is that continuous [11:31] learning that You do mistakes, [11:33] but they're codified from now on [11:34] forever until you need to prune it [11:36] And it's not relevant anymore [11:38] and into the wisdom base [11:39] of the developer uh memory [11:43] Um, [11:44] we have a little time i'm gonna go [11:46] a tiny bit off script because [11:47] we've been chatting Um [11:49] script what are you talking about? [11:50] No script no [11:52] Um [11:53] So one of the things that you've [11:55] sort of mentioned here as well is [11:57] you know wisdom tribal knowledge We talk [12:00] about these words all the time um [12:01] I actually used to work on technical [12:03] onboarding at OpenAI before moving [12:05] to developer experience And this was [12:06] something I heard a lot people [12:07] were always talking about tribal [12:09] knowledge and how much of an issue [12:10] it was So can you tell me [12:12] about how the knowledge base [12:15] and how the or how this wisdom? [12:17] Is sort of replacing [12:19] that tribal knowledge and how [12:20] people can leverage that? [12:22] Yeah, of course So one of our clients [12:24] financial institution the big one um [12:27] south east uh, they have like a [12:30] Hundreds of repositories and multiple [12:32] micro services and and there's [12:35] so much uh knowledge about [12:38] What works and what does not [12:40] work at scale and they cannot [12:42] fuck up again sorry for my french [12:43] And and it's it's it's a financial [12:45] institution you can't go wrong [12:47] like they they want to They want [12:49] to progress towards a software factory, [12:52] but they can just can't let [12:53] things go go wrong And uh, [12:56] all of this micro services each one [12:57] is is in charge of something else [12:59] And only those senior developers [13:01] that they had some of them left [13:04] Actually knows what what works [13:06] and what what not So part [13:08] of the giving you an example part [13:10] of the wisdom base is scanning backwards [13:13] years of discussions [13:16] Bit of developers on github get love [13:18] bit bucket whatever on slack teams, [13:21] uh, whatever and and Uh codifying [13:23] that tribal knowledge and bringing it [13:25] back and understanding What is [13:28] the meaning of making a change of one [13:31] service on another another service? [13:33] Sometimes that connection is [13:34] really hard to do because it goes [13:37] through through some streaming [13:38] or Like it's not like literal [13:40] and in the code that the data [13:42] that streams through it Is actually [13:45] defining the actual usage right? [13:47] And that's part of what we're [13:48] trying to do in coda we look [13:49] on that data and try to codify [13:51] that in the software map [13:54] Amazing so [13:56] Let's talk about what coda's plan [13:58] for the future is considering [13:59] the fact that the models today [14:01] Are the dumbest that they will ever be [14:04] Probably yeah, totally [14:06] So I think like we [14:08] talked about the the size [14:09] of the task what is a task? [14:11] I encourage everyone if something [14:13] to take from here that the models [14:15] are just going to be better [14:16] the The model that we sound out [14:18] is just the dumbest and then it's [14:20] we're hearing for a couple of months [14:22] not just now that Agents are being [14:24] able to do a long run Sometimes [14:26] it works sometimes not but let's [14:27] assume they will that means that they [14:29] will be able to complete a full [14:31] task for us And I think we [14:32] need to reimagine what is a task [14:35] for example in software development [14:38] Just by by the tools we're using [14:40] in our capabilities to manage [14:42] with code We define a task like [14:44] as a pull request or or that's [14:46] like the units that that we process [14:48] through through our factory Right, [14:50] and I think that will be reimagined [14:52] who cares about that pr what you [14:54] care about is completing a new [14:55] capability Or a new feature end [14:57] to end now that is not something [15:00] you can or can or want to discuss [15:03] In github pull request page, right? [15:05] It's just one pr there's five [15:07] or 15 other pr's in that repo [15:10] or or another And that as a whole [15:13] is a task So we need to reimagine [15:15] how we do that and shameless [15:17] plug stay tuned in the next few [15:19] days We are going to release what [15:21] we called work package triage [15:23] Basically being able to help you [15:25] understand how all the pr's are [15:27] connected to each other in order [15:29] to complete a task and being [15:30] able to review that as a whole. [15:32] So that's for us the inspirational [15:34] part of reimagining a task [15:36] in order to build a factory [15:38] And I think of all builders here. [15:40] That's what I would point out [15:41] to think about Okay, and then [15:44] I also would love to leave them [15:47] with something that they can try today [15:50] So what should a developer building [15:53] with OpenAI try when they are [15:55] designing this type of reviewer? [15:57] Yeah, sure so I I do [16:00] first of all I am very excited [16:01] about the decision API [16:04] I used to work in a company that [16:08] Build embedding models and so [16:10] on before Qodo and I knew [16:12] that this potential is coming [16:14] actually I think one of the known [16:17] decision models out there are [16:18] based on a model that we developed [16:21] in that company Anyway, I think like [16:23] the the potential of the decision API [16:26] that is out there is in order [16:28] to create like fast a decision [16:31] and in general I do recommend what [16:34] I said is thinking about your task [16:36] breaking the task to workloads [16:39] Thinking what is the swarm that you [16:41] need for each workload that could [16:43] be two three five different uh [16:45] of your own dots that's uh [16:48] that that you need to develop [16:50] and thinking how each one of those [16:53] players uh in in in the game [16:56] board you just uh we just [16:59] developed the designed what kind [17:01] of models they should be using [17:03] and that's that's I think [17:05] like going to be our Our game [17:07] a game board for a while what [17:09] I just described designing the swarm [17:11] according to specific workloads to [17:14] Help our customers and users to finish [17:17] tasks end to end and optimizing [17:20] with the right models to make [17:21] that cost efficient because That's [17:23] why not if it's such a promising [17:26] and helping you complete tasks You [17:28] would want to use it a lot [17:30] and it needs to be cost efficient Yeah, [17:33] okay amazing. [17:34] Um, thank you so much. [17:35] I also like personal note, [17:37] you know I used to work it [17:39] into it before this and um, [17:41] I worked on technical training and Qodo [17:43] was one of the very first products [17:46] that I Did technical training [17:49] for so being here getting to do [17:50] this is like a very full circle [17:52] moment to see you where you've [17:53] come where the company has gone [17:55] Since then since a few years ago [17:56] is um really special so loops [18:01] Human in the loop [18:02] Um amazing. [18:05] Thank you so much for giving [18:06] us a look at the work behind [18:08] the future of the code review. [18:10] We appreciate your time [18:10] Thank you so much. [18:12] So good luck on killing your last [18:14] human code review And don't [18:16] forget that when you do that you [18:18] still need to have like training [18:19] for your developers, right? [18:22] Good luck [18:30] Good luck