OpenAI Releases Codex Security as Frontier AI Slashes Exploit Costs

Codex Security: From Vulnerability to Verified Fix | DevDay 2026 — OpenAI

OpenAI

As automated exploit costs approach zero, organizations must deploy frontier models to find and patch code flaws before matching offensive capabilities proliferate to attackers.

AI-assisted summary of Codex Security: From Vulnerability to Verified Fix | DevDay 2026 — original source. Check the original for full context.

Source published:

Key ideas

  • OpenAI announced Codex Security as an open-source Apache 2.0 CLI and SDK alongside a managed turnkey cloud platform powered by the specialized Daybreak Blue security model. (2:12)
  • During an internal security sprint allocating 25% of product engineers, automated validation and deduplication reduced false positive vulnerability alerts to 0.81% and patch rollbacks to under 1%. (2:12)
  • Frontier models currently maintain a temporary advantage over open-weight models in cyber exploitation, creating an urgent 'defender's window' to scan and remediate codebases before automated offensive capabilities proliferate. (0:22)
  • On the PatchBench evaluation suite, 6 Astra operating at medium reasoning outperformed 5.6 Sol at extra-high reasoning while cutting compute costs by more than half. (16:00)

Detailed notes

## Background Published by OpenAI on YouTube on November 11, 2026, this DevDay presentation demonstrates how to identify and remediate code vulnerabilities using OpenAI's cyber models and open-source tooling. The session draws directly from OpenAI's own internal infrastructure security sprint to explain how teams can scan large codebases, catch bugs in pull requests, and deploy verified automated patches.

## TL;DR OpenAI cyber product engineers Ian and Kyle present the concept of the 'defender's window,' arguing that organizations must proactively leverage frontier AI capabilities to fix vulnerabilities before offensive exploitation costs drop to zero. Drawing on an internal sprint where 25% of OpenAI's product engineering workforce mobilized to build an automated continuous security pipeline, they walk through the architecture of the open-source Codex Security CLI. The talk demonstrates how multi-tiered context, parallelized deep scans, agentic deduplication, and adversarial patch verification enable automated fixes with sub-1% rollback rates, concluding with benchmarks for the Astra model and the introduction of Codex Security Cloud powered by Daybreak Blue.

## Atmosphere & style The presentation strikes an urgent yet pragmatic and technically grounded tone, balancing the existential stakes of shifting cyber economics with candid developer realities. Ian and Kyle maintain a collaborative, fluid rhythm on stage, interspersing serious discussions of vulnerability triage and model efficiency with relatable live-demo moments, such as Kyle joking about his nervous habit of making vocal sound effects while operating the presentation clicker. The session follows an encouraging arc, moving from the overwhelming initial dread of alert fatigue and massive issue backlogs to the reassuring calm of an automated steady-state defense pipeline.

## Bulletin - [02:12](https://www.youtube.com/watch?v=gUQN7juxvVI&t=132s) OpenAI announced Codex Security as an open-source Apache 2.0 CLI and SDK alongside a managed turnkey cloud platform powered by the specialized Daybreak Blue security model. - [02:12](https://www.youtube.com/watch?v=gUQN7juxvVI&t=132s) During an internal security sprint allocating 25% of product engineers, automated validation and deduplication reduced false positive vulnerability alerts to 0.81% and patch rollbacks to under 1%. - [00:22](https://www.youtube.com/watch?v=gUQN7juxvVI&t=22s) Frontier models currently maintain a temporary advantage over open-weight models in cyber exploitation, creating an urgent 'defender's window' to scan and remediate codebases before automated offensive capabilities proliferate. - [16:00](https://www.youtube.com/watch?v=gUQN7juxvVI&t=960s) On the PatchBench evaluation suite, 6 Astra operating at medium reasoning outperformed 5.6 Sol at extra-high reasoning while cutting compute costs by more than half.

## Key takeaways - [00:22](https://www.youtube.com/watch?v=gUQN7juxvVI&t=22s) The marginal cost of software exploitation is approaching zero as frontier AI models uncover dormant vulnerabilities across widely used codebases. - [00:22](https://www.youtube.com/watch?v=gUQN7juxvVI&t=22s) A temporary 'defender's window' exists because frontier models currently lead open-weight models in cyber capability, giving defenders an asymmetric advantage to remediate flaws first. - [02:12](https://www.youtube.com/watch?v=gUQN7juxvVI&t=132s) OpenAI mobilized a company-wide security sprint reallocating a quarter of product engineering to build an automated, continuous 'defense factory.' - [04:21](https://www.youtube.com/watch?v=gUQN7juxvVI&t=261s) Unconstrained agent scanning failed to generate high-fidelity alerts; actionable results required targeted scan scopes combined with shared knowledge bases. - [04:21](https://www.youtube.com/watch?v=gUQN7juxvVI&t=261s) OpenAI decentralized threat modeling by establishing repository-level security.md files to define architectural constraints and threat boundaries. - [07:54](https://www.youtube.com/watch?v=gUQN7juxvVI&t=474s) The codex-security CLI runs parallelized deep scans that iteratively explore code paths and reachability until issue discoveries plateau. - [09:12](https://www.youtube.com/watch?v=gUQN7juxvVI&t=552s) Deduplication became the primary scaling bottleneck, with OpenAI discovering that 36% of all raw scan findings across monorepos were redundant duplicates. - [11:43](https://www.youtube.com/watch?v=gUQN7juxvVI&t=703s) Enrichment pipelines automate triage by scoring issues against organizational criticality rubrics and routing them through codified service ownership rules. - [12:42](https://www.youtube.com/watch?v=gUQN7juxvVI&t=762s) Automating patches should be approached as a gradual dial rather than a switch, relying on validation prompts and ephemeral test environments to build confidence. - [12:42](https://www.youtube.com/watch?v=gUQN7juxvVI&t=762s) An adversarial 'verify fix' agent subjects proposed patches to hostile regression tests prior to entering standard software development lifecycle pipelines. - [16:00](https://www.youtube.com/watch?v=gUQN7juxvVI&t=960s) Benchmark results on PatchBench show model patching speeds and pass rates surpassing OpenAI's internal baseline human engineering average of approximately 50%. - [17:03](https://www.youtube.com/watch?v=gUQN7juxvVI&t=1023s) Organizations adopt a predictable adoption curve that begins with heavy backlog triage and transitions into continuous, automated pull requests catching vulnerabilities on push. - [17:03](https://www.youtube.com/watch?v=gUQN7juxvVI&t=1023s) Codex Security Cloud provides a managed continuous monitoring loop via a Codex plugin, using the Daybreak Blue model to auto-generate pull requests directly to GitHub.

## Detailed walkthrough ### [00:22](https://www.youtube.com/watch?v=gUQN7juxvVI&t=22s) The Defender's Window and the Changing Cyber Landscape

Ian and Kyle introduce their work on the cyber product engineering team at OpenAI. While Kyle’s day-to-day focus centers on developing the open-source Codex Security CLI, Ian frames the broader existential shift currently underway across the software industry. The economics of offensive software exploitation are fundamentally destabilizing: the cost to discover and exploit software vulnerabilities is rapidly approaching zero. Over recent months, frontier AI models have demonstrated the capability to uncover thousands of previously dormant security vulnerabilities buried deep inside critical software systems relied upon daily.

Crucially, frontier models currently maintain a technical lead over open-weight models in cyber capabilities. Ian explains that this asymmetry creates a temporary strategic advantage known as the "defender's window." This brief period provides organizations and open-source maintainers an opportunity to proactively uncover and remediate flaws before those same offensive capabilities proliferate into widespread, unregulated tooling.

Ian makes the team's urgency unmistakable when discussing this temporary operational advantage: “This is a window of opportunity to find and fix weaknesses before the same capabilities become widely available and fall into the hands of attackers.”

- Frontier AI models currently outperform open-weight models in complex cybersecurity identification and remediation tasks. - The shrinking economic cost of vulnerability exploitation necessitates offensive-style automated discovery for defenders before exploit tooling democratizes. - Defensive AI scanning is positioned as essential not only for enterprise infrastructure, but equally for open-source repositories and personal software projects.

[02:12](https://www.youtube.com/watch?v=gUQN7juxvVI&t=132s) Building the Defense Factory: OpenAI's Internal Security Sprint

To validate these security capabilities in a production environment, OpenAI made internal infrastructure security a top company priority. Several months prior, the company mobilized a major internal security sprint, assigning a full quarter of its product engineering workforce exclusively to identifying, verifying, and patching vulnerabilities across their entire internal codebase.

This company-wide push laid the operational groundwork for what OpenAI calls its internal "defense factory." Rather than treating security as an intermittent checklist, the defense factory operates as an automated, continuous loop: scanning repositories, validating theoretical bugs into confirmed issues, routing findings to domain owners, and verifying generated fixes. The system achieved a 0.81% false positive rate through automated deduplication and agentic validation, while patches generated by Codex achieved a production rollback rate of less than 1%. OpenAI translated the lessons, pipelines, and tooling learned from this sprint into the Codex Security product suite, spanning an Apache 2.0 open-source CLI, an SDK, plugins, and the newly announced turnkey Codex Security Cloud.

Ian explains the dedication required to run this operation at scale: “That meant committing the people, the time, the resources to find and fix vulnerabilities as a top priority.”

- 25% of OpenAI's product engineers were reallocated during the dedicated internal security sprint. - Automated validation and deduplication reduced false positive vulnerability alerts down to 0.81%. - Automated code fixes authored with Codex assistance yielded a rollback rate of under 1%. - Codex Security is structured as an open-source CLI, SDK, and IDE/CI plugin, alongside the managed turnkey Codex Security Cloud.

[04:21](https://www.youtube.com/watch?v=gUQN7juxvVI&t=261s) Context is Key: Structuring Scan Instructions and Knowledge Bases

Kyle outlines the five core stages derived from OpenAI's internal sprint: establishing agent context, running targeted scans, enriching findings, generating patches, and embedding automation into the software development lifecycle (SDLC). The CLI is entirely open source under the Apache 2.0 license. In early iterations, pointing an autonomous agent at OpenAI's massive repositories with a blanket instruction to locate all bugs failed to generate actionable results. To produce precise security findings, scans had to be fanned out across distinct subcomponents, with each agent equipped with system context that code alone cannot convey.

To solve this, OpenAI split agent context into two tiers: 1. **Narrow Scan Instructions:** A specific instruction file (`prompt.md`) detailing the exact scope, specific target directories, and precise vulnerability classes for that run. 2. **Shared Knowledge Bases:** Reusable documentation shared across scanners and validators containing architectural diagrams, network boundary rules, and cloud firewall configurations.

To decentralize security governance, OpenAI engineering teams also adopted `security.md` files committed directly alongside code components—analogous to `agents.md` files—empowering individual product owners to define their own threat boundaries and security constraints.

Reflecting on their initial, unconstrained scanning attempts, Kyle remarks: “Pointing an agent scanner at our repositories and saying, find all vulnerabilities, make no mistakes, didn't really cut it.”

```mermaid flowchart TD accTitle: Codex Security Context and Scan Pipeline accDescr: Diagram illustrating how targeted prompts and shared knowledge feed into parallel agent scans and deduplication. subgraph Context["1. Context Layer"] A["Scan Scope & Vulnerability Targets (prompt.md)"] B["Knowledge Base (Architecture & Firewalls)"] C["Component Threat Models (security.md)"] end subgraph Execution["2. Parallel Deep Scan"] D["Multi-Agent Scan Fleet"] E["Iterative Validation Passes"] end subgraph Triage["3. Enrichment & Triage"] F["AI Deduplication Engine"] G["Criticality Rubric & Owner Assignment"] end

A --> D B --> D C --> D D --> E E --> F F --> G ```

[07:54](https://www.youtube.com/watch?v=gUQN7juxvVI&t=474s) Executing Deep Scans in Parallel

Demonstrating the Codex Security CLI on an example repository, Kyle illustrates how context files translate into command-line execution. After inspecting `prompt.md` (which isolates targeted vulnerability classes) and reviewing the architecture documentation in the knowledge base file, the scanner is launched using `codex-security scan deep`.

A deep scan fans out multiple agent processes in parallel. Agents run iterative discovery loops: probing code paths, attempting to validate reachability, checking configurations against the knowledge base, aggregating outputs, and initiating follow-up passes until no additional novel issues are detected or the preconfigured scan limit is reached. A comprehensive codebase scan takes significant compute time, prompting Kyle to fast-forward through the live output while joking with the audience about his nervous habit of making vocal sound effects while operating the presentation clicker.

Describing the underlying mechanics of the command, Kyle states: “The deep scan runs multiple security scans in parallel with agents investigating and validating potential vulnerabilities.”

- Custom CLI arguments pass both ephemeral prompt instructions and persistent knowledge bases into the scanner engine. - Parallelized deep scans re-scan dynamically until findings saturate or hit user-defined execution boundaries.

[09:12](https://www.youtube.com/watch?v=gUQN7juxvVI&t=552s) Tackling the Deduplication Bottleneck and Enriching Findings

Running fleets of hundreds of agent workers across distinct hosts and monorepo components introduced OpenAI's first major scaling bottleneck: finding deduplication. Tens of thousands of raw alerts flowed into the pipeline, many referencing the identical root cause from different entry points. Because assigning dozens of repetitive tickets to engineering teams risks severe fatigue and friction, OpenAI built an agentic deduplication workflow into the CLI to consolidate overlapping issues prior to human triage.

During OpenAI's internal security sprint, deduplication revealed that more than a third of all generated alerts were redundant. Once deduped, the remaining findings entered an enrichment phase tasked with answering two questions: how critical the issue is to OpenAI, and which engineering team owns the code. Criticality was initially defined as any vulnerability granting unauthorized external actors access, though refining this rubric required repeated iteration as unique system edge cases emerged.

Kyle highlights the developer friction that motivated his work on deduplication: “In our internal sprint, we found that across all of the scans, 36% of our findings were duplicates.”

- Hundreds of scanner instances distributed across multiple hosts produced tens of thousands of initial findings. - 36% of all raw scan findings across OpenAI's codebases were identified as duplicates. - Automated deduplication avoids developer alert fatigue, such as engineers receiving dozens of redundant tickets at 1:00 AM for a single bug. - Triage enrichment maps findings against custom criticality rubrics and automatically pinpoints product owners.

[11:43](https://www.youtube.com/watch?v=gUQN7juxvVI&t=703s) Prioritizing Findings and Solving Ownership Bottlenecks

Kyle explains that once scanners generate output, the findings must fit into an organization's internal rubric. Running issues through the Codex Security CLI against that rubric produces a prioritized queue aligned with company priorities—or an indie developer's personal concerns. However, prioritizing is only half the battle; routing the issue to the correct team turned out to be unexpectedly painful. As Kyle emphasizes: “Knowing who owns what in a company that's moving so fast and growing so quickly took actually a large amount of human effort.”

Once OpenAI codified service and code ownership into a formal ruleset, they eliminated friction, enabling automated, accurate assignment of findings with minimal ticket churn.

```mermaid flowchart LR accTitle: Vulnerability Triage and Assignment Workflow accDescr: Pipeline showing raw findings filtered by a custom rubric into a prioritized backlog and routed by ownership rules. subgraph Triage["Triage Stage"] A["Raw Scanner Findings"] --> B["CLI + Custom Rubric"] B --> C["Prioritized Backlog"] end subgraph Routing["Assignment Stage"] C --> D["Codified Ownership Rules"] D --> E["Assigned Engineering Teams"] end ```

[12:42](https://www.youtube.com/watch?v=gUQN7juxvVI&t=762s) From Human-in-the-Loop to Auto-Generated Patches and Adversarial Verification

Addressing findings initially required engineers to manually review assigned tickets, prompt Codex to write a solution, shepherd the pull request through reviews, and deploy it to production. Over time, the team moved toward generating patches automatically. Kyle acknowledges the natural developer hesitation toward automated code modifications, remarking: “Which, that thought process is why we found this process is more of a dial that you turn up, not a switch that you flip on.”

Confidence grew as contextual safeguards matured. Using the CLI's `patch` command, developers can supply a `--validation-prompt-file` flag that guides the agent on spinning up test environments, assessing behavior, and drafting a pull request. Because of rigorous pre-merge validation, the internal deployment achieved an exceptionally low rollback rate of 0.81% across tens of thousands of findings. To guarantee correctness, OpenAI built a `verify fix` command acting as an adversarial agent against the generated patch, subjecting it to hostile tests to confirm the vulnerability is eliminated before it enters the standard software development lifecycle (SDLC).

- **`patch` CLI command**: Consumes vulnerability data and a validation prompt to set up ephemeral dev environments and generate pull requests. - **Rollback reliability**: Achieved a sub-1% rollback rate (specifically 0.81%) on auto-generated remediation patches inside OpenAI. - **Adversarial verification**: The `verify fix` utility pits an adversarial evaluation agent against the proposed fix to prevent incomplete or regressive patches. - **SDLC integration**: The output is designed to seamlessly slot into existing code reviews, QA stages, and release pipelines.

[16:00](https://www.youtube.com/watch?v=gUQN7juxvVI&t=960s) Model Benchmarks: PatchBench and Astra Efficiency

Ian takes over to detail the performance leaps observed in patch quality and compute costs, demonstrating results from PatchBench—an external evaluation benchmark comparing model capabilities using Codex Security's patch skill. In these evaluations, 6 Astra operating at medium reasoning comfortably outstripped 5.6 Sol at extra-high ("X-high") reasoning, while requiring less than half the financial cost per test case.

Human performance at OpenAI hovers around the 50% mark on the benchmark, placing Astra's automated patching abilities well ahead of standard baseline human speeds. Ian notes that engineering continues to work in lockstep with the research team to translate underlying model breakthroughs directly into defender capabilities.

| Model / Baseline | Reasoning Tier | Benchmark Performance | Relative Cost | |:--- |:--- |:--- |:--- | | **5.6 Sol** | Extra-High (X-High) | Lower than Astra Medium | >2x cost per case | | **6 Astra** | Medium | Outperforms 5.6 Sol X-High | <0.5x cost per case | | **Human Engineers (OpenAI)** | N/A | ~50% baseline | High human effort |

[17:03](https://www.youtube.com/watch?v=gUQN7juxvVI&t=1023s) The Defender's Curve, Codex Security Cloud, and Availability

Reflecting on OpenAI's internal rollout curve, Ian outlines the shift from drowning in security alerts to reaching an operational steady state. Early adoption is dominated by an intimidating backlog of deduplication, validation, and remediation. However, as the backlog is eliminated, organizations hit a thin tail where continuous background scans and automated pull requests catch defects before they ever merge to production.

To help teams that do not want to hand-wire the open-source CLI and SDK into custom GitHub Actions or CI/CD pipelines, Ian highlights Codex Security Cloud (first introduced by Sam Altman in the keynote). Codex Security Cloud provides a turnkey, continuous orchestration loop powered out of the box by the security-specialized Daybreak Blue model without complex approval gates. Demonstrating the workflow, Ian installs the Codex Security Cloud plugin inside Codex, connects a GitHub repository, configures scans to trigger on every push, inspects org-wide issue trends, and auto-generates a direct pull request to GitHub.

- **Codex Security CLI / SDK**: Fully open-source and downloadable via NPM for custom CI/CD setups. - **Codex Security Cloud**: Managed turnkey platform available through the Codex plugin marketplace. - **Daybreak Blue**: Dedicated cyber/security-focused model architecture running under the hood to power automated detection and remediation loops.

## Notable quotes > [00:22](https://www.youtube.com/watch?v=gUQN7juxvVI&t=22s) This is a window of opportunity to find and fix weaknesses before the same capabilities become widely available and fall into the hands of attackers. — Ian

> [04:21](https://www.youtube.com/watch?v=gUQN7juxvVI&t=261s) Pointing an agent scanner at our repositories and saying, find all vulnerabilities, make no mistakes, didn't really cut it. — Kyle

## Mentioned Ian, Kyle, OpenAI, Codex Security CLI, Codex Security SDK, Codex Security Cloud, Codex, Sam Altman, Daybreak Blue, PatchBench, 5.6 Sol, 6 Astra, Apache 2.0, GitHub, GitHub Actions, NPM

## Video description Find and fix vulnerabilities with OpenAI’s cyber models and open-source CLI and SDK. Learn to scan large codebases, catch issues in pull requests, and verify fixes, drawing on OpenAI’s own security work.

[Source](https://www.youtube.com/watch?v=gUQN7juxvVI)

Reading summary

OpenAI's cyber product engineering team argues that the collapsing cost of vulnerability discovery necessitates automated, agent-driven defensive pipelines before offensive AI capabilities democratize. Drawing on an internal enterprise security sprint, the team introduced Codex Security—an open-source and cloud framework designed to turn code scanning into verified, automated patches. Capitalizing on the Temporary Defender's Window (0:22) According to OpenAI engineer Ian, the economics of software exploitation are shifting rapidly as frontier AI models uncover thousands of dormant vulnerabilities across critical infrastructure. Because frontier systems currently outperform open-weight models in complex cybersecurity tasks, defenders possess a temporary strategic advantage known as the "defender's window." This window offers organizations and open-source maintainers an opportunity to systematically identify and remediate flaws before identical discovery capabilities become accessible to malicious actors through unregulated, democratized tooling. Context-Driven Scanning and Alert Deduplication (4:21) To operationalize defense, OpenAI mobilized 25% of its product engineering workforce during a dedicated internal sprint. Early attempts to deploy autonomous scanners with blanket instructions to find all vulnerabilities failed to produce actionable outcomes. Kyle noted that precision required structuring scans into distinct tiers: narrow scan instructions (`prompt.md`), shared architectural knowledge bases, and component-level threat boundaries defined in `security.md` files. When distributed agent fleets scanned OpenAI's codebases in parallel, finding deduplication emerged as a primary bottleneck. Raw scans produced tens of thousands of alerts, of which 36% were duplicates of identical root causes. To mitigate alert fatigue, OpenAI introduced an agentic deduplication pipeline and codified code-ownership rules, allowing the system to categorize issues against a criticality rubric and route verified findings directly to relevant product teams. Autonomous Patch Generation and Adversarial Verification (12:42) OpenAI addressed developer hesitation regarding automated code modification by treating remediation as an adjustable dial rather than an instant shift. The CLI's `patch` command provisions ephemeral development environments, evaluates code behavior, and drafts pull requests automatically. To ensure code reliability before entering the software development lifecycle, the pipeline runs a `verify fix` utility that deploys an adversarial agent to subject proposed patches to hostile validation tests. Through structured context and adversarial verification, OpenAI reported achieving a 0.81% false positive alert rate and lowering the production rollback rate of machine-generated patches to under 1% (specifically 0.81%). Model Economics and Continuous Cloud Orchestration (16:00) Benchmarking on PatchBench highlighted significant improvements in remediation efficiency. The 6 Astra model operating at medium reasoning outperformed 5.6 Sol at extra-high reasoning on automated patching tasks, while consuming less than half the financial compute cost per test case. Both configurations surpassed the ~50% baseline accuracy recorded by OpenAI's human engineers. To simplify deployment beyond custom CI/CD pipelines wired via the open-source Apache 2.0 CLI and SDK, OpenAI launched Codex Security Cloud. Powered by the security-specialized Daybreak Blue model, the turnkey platform monitors repositories, runs automated scans on code pushes, and issues pull requests directly to GitHub. OpenAI demonstrated that large-scale codebases can transition from deep vulnerability backlogs to steady-state automated remediation, but the strategy relies on maintaining frontier model performance advantages over open-weight alternatives and overcoming organizational friction around auto-merging machine-generated code.

Source quotes

[0:22] Ian: This is a window of opportunity to find and fix weaknesses before the same capabilities become widely available and fall into the hands of attackers. [4:21] Kyle: Pointing an agent scanner at our repositories and saying, find all vulnerabilities, make no mistakes, didn't really cut it.

Original transcript

[0:22] Hi, everyone. [0:23] I'm Ian. [0:24] I work on cyber product [0:26] engineering here at OpenAI. [0:27] And today, I'm going to share [0:29] a bit about what we're building [0:31] for AI security, how we've [0:33] used it internally, and also [0:35] how you can use it as well. [0:37] Hi, and I'm Kyle. [0:38] I work with Ian on the cyber [0:40] product engineering team. [0:41] And I'm focused on the Codex Security [0:43] CLI, which provides building blocks [0:46] to help secure your software with AI. [0:50] Today, we'll walk through how the CLI [0:53] works, from discovering and validating [0:55] findings to generating patches [0:58] that a developer can review and show [1:01] a demo of our new Codex Security [1:04] Cloud, which is a more turnkey [1:06] solution of the CLI workflows that we [1:09] announced today during the keynote. [1:11] Great. [1:13] So before we get all into [1:14] that, I want to acknowledge the state [1:16] of cybersecurity landscape today. [1:19] We're definitely at an inflection [1:21] point when it comes to security. [1:22] The cost of hacking stuff [1:24] is going to basically zero. [1:27] And over the past few months, [1:29] the latest models have made it [1:31] possible to find thousands of latent [1:32] security vulnerabilities in software [1:35] that people depend on every single day. [1:38] So this matters a lot, [1:40] not just for security people, [1:42] but also really for any developer [1:43] building any type of application. [1:47] And right now, fortunately, [1:49] these frontier models are ahead [1:51] of open-weight models in terms [1:53] of their cyber capabilities. [1:55] So this creates what we [1:56] call the defender's window. [1:58] This is a window of opportunity to find [2:01] and fix weaknesses before the same [2:03] capabilities become widely available [2:05] and fall into the hands of attackers. [2:08] We need to use that lead now. [2:12] So we took this to heart at [2:15] OpenAI, and the way that we met [2:18] the moment was by making the security [2:21] of our systems a company-wide priority. [2:24] That meant committing the people, [2:26] the time, the resources to find [2:29] and fix vulnerabilities as [2:32] a top priority. [2:34] And while this is true for companies, [2:37] it's also true for open-source projects [2:39] and even personal projects as well. [2:42] So at OpenAI, what it looked like [2:44] is we actually had a security sprint [2:46] a couple months ago that had a quarter [2:48] of all of our product engineers [2:50] focused solely on strengthening [2:53] the security of our internal systems. [2:56] This was a huge effort. [2:58] And this was really the starting [2:59] point for building what we call [3:01] internally our defense factory. [3:04] This is a continuous loop [3:05] that scans our systems, finds [3:08] and validates vulnerabilities, [3:10] assigns owners, and verifies fixes. [3:13] So through deduplication [3:15] and validation across thousands [3:17] of findings, we were able to scan [3:19] our code base and focus really [3:21] only on findings that could lead [3:22] to actual exploits with a very [3:25] low false positive rate of 0.81%. [3:27] And of course, we used Codex [3:29] to help create the fixes, [3:32] which it did remarkably well [3:34] with a rollback rate of less than 1%. [3:40] So how did we actually make this happen, [3:42] and how can you use it? [3:43] We took everything that we learned from [3:46] our internal security sprint at OpenAI [3:48] and we built it into Codex Security. [3:52] Codex Security comes in several flavors. [3:54] It's available as an open-source [3:56] plug-in, CLI, and SDK. [3:59] And today, we're launching a major [4:01] update to the cloud version, [4:03] which is a more turnkey solution. [4:06] So Kyle is going to walk through [4:08] some of the more critical [4:10] and interesting problems that we [4:11] ran into during our security sprint [4:13] and show you how we address them. [4:18] Thanks, Ian. [4:21] Thanks. [4:23] So today, like Ian said, I'm going [4:25] to talk about some of the bottlenecks [4:27] that we hit while we were running [4:28] our own internal defense factory [4:30] and how we address them. [4:32] We'll cover five main steps, [4:35] giving agents context, [4:37] running scans, [4:39] enriching the findings, [4:41] generating patches, [4:43] and integrating this into [4:45] your software development lifecycle. [4:48] Each step comes directly [4:51] from how we ran our internal [4:53] security sprint here at OpenAI. [4:56] We'll use the CLI commands together [4:58] as we walk through these steps to help [5:01] us understand how it all comes together. [5:04] The Codex Security CLI was [5:06] a core building block of OpenAI's [5:08] internal defense factory. [5:10] And you can leverage the same [5:12] tools to build your own defense [5:14] factory by using the CLI, [5:16] which is fully open-source [5:18] and under the Apache 2.0 license. [5:21] Let's start with giving [5:22] agents the right context. [5:25] You can imagine we have quite [5:28] a bit of code here at OpenAI [5:29] that we've written over the years. [5:31] Pointing an agent scanner at [5:33] our repositories and saying, [5:35] find all vulnerabilities, make no [5:37] mistakes, didn't really cut it. [5:41] So we needed to fan out [5:43] a large number of scanners [5:46] and focus each scan on [5:47] a specific part of the code base. [5:51] In doing so, we found that it [5:53] was critical to give the agent [5:55] enough context to understand [5:57] what it was looking at. [5:59] What does this part of the system do? [6:01] How does it fit into the hole? [6:03] How is it exposed? [6:05] What protections are already [6:06] in place around it? [6:08] Some of that context, [6:09] I'm sure you all know, lives [6:11] outside of actual code bases. [6:14] So there are two levels at [6:16] which we think about giving [6:18] a security agent context. [6:20] The first is a narrow slice, [6:23] which is a custom scan instruction. [6:26] You can tell the agent which areas [6:28] to investigate, what's in or out [6:31] of scope for this scan, and which [6:33] types of vulnerabilities to focus on. [6:35] This was extremely helpful. [6:38] But we found ourselves repeating [6:40] the same background information over [6:43] and over in those instructions. [6:45] So we pulled that context [6:47] out into a knowledge base [6:49] we could reuse across scans. [6:53] That's the second layer. [6:55] Cross scan shared context [6:57] we call a knowledge base. [6:59] You can take the knowledge base, or you [7:02] can make the knowledge base available [7:04] to both scanners and validators. [7:06] And it includes things [7:08] like architecture docs, [7:10] firewall configuration, etc. [7:13] It helps the agent understand [7:15] the system and the protections [7:17] that aren't visible in source code. [7:21] At OpenAI, we also started [7:24] committing security .md files. [7:26] If you know what an agent's [7:28] .md file, it's a very similar [7:29] thing, but for security. [7:31] We started committing these [7:33] across components in our repo [7:35] so that product owners could [7:37] define their own threat models [7:39] and own their own security guidelines. [7:43] Together, these give the agent [7:45] a broader context, the broader [7:47] context it needs to understand what [7:49] it's looking at, along with specific [7:52] instructions for the scan at hand. [7:54] Let's see that in action [7:56] with a deep scan of an example [7:58] repository using the Codex security CLI. [8:02] So here we go. [8:04] This is our first [8:05] catting our first prompt .md. [8:07] This is the custom scan prompt [8:10] with instructions for what the agent [8:12] could focus on in this specific scan. [8:14] So we're going to do a deep scan. [8:16] We're also going to show [8:17] the knowledge base file here, [8:19] which is the shared context about [8:22] the architecture and the access rules. [8:23] Again, this is an example repository [8:25] we're showing as we go through this. [8:29] So we're going to kick [8:31] off this deep scan now. [8:32] And you can see we passed [8:34] those arguments in to provide [8:35] the context, those files we saw before. [8:37] The deep scan runs multiple [8:39] security scans in parallel with agents [8:42] investigating and validating [8:44] potential vulnerabilities. [8:46] It combines their results [8:47] and keeps running additional passes [8:49] until it can't find any new issues [8:52] or it reaches its configured limit. [8:55] So we're going to fast forward [8:57] a ton because this can take [8:59] a pretty long time, as you can [9:00] imagine scanning a whole code base. [9:01] And a deep scan is very thorough. [9:04] So I'm going to click here. [9:07] They told me I make a lot [9:08] of sound effects when I use the clicker, [9:10] so deal with me. [9:12] So now we have some findings. [9:14] As I mentioned before, we had [9:16] to split the OpenAI code base into [9:19] smaller areas and scan them in parallel. [9:21] That meant spinning up a huge [9:23] fleet of hundreds of agents across [9:25] a bunch of different hosts, scanning [9:28] different parts of the code base. [9:30] The first challenge [9:31] this introduced was deduplication. [9:34] This is actually the part that I worked [9:36] on during our internal security sprint. [9:38] So it's pretty near and dear to my heart. [9:39] And it was the first [9:41] major bottleneck we hit. [9:42] We had tens of thousands [9:44] of findings coming in from different [9:46] scanners across our repositories. [9:49] And before we handed those off [9:51] to developers and asked them to go [9:52] fix them, we need to understand [9:54] which findings corresponded [9:56] to the same underlying issues. [9:58] At that volume, we needed agents [10:01] to help with the deduplication. [10:03] And as a result of that work [10:05] internally, we distilled it into [10:07] what is now the deduplication [10:08] workflow available in the CLI. [10:13] So this is what it [10:14] would look like if you were trying [10:15] to deduplicate the scan results [10:17] that you got from the CLI. [10:19] And so more concretely, [10:21] if multiple findings can [10:23] describe the same vulnerability, [10:25] before you assign that work [10:28] to a developer, you need to sort [10:30] out which findings are duplicates. [10:32] So you don't have someone sending [10:34] you an angry DM at like 1 [10:36] o'clock in the morning asking you, [10:37] why did you assign 32 tickets [10:39] to me instead of just one? [10:41] Because this is all just the same [10:42] vulnerability. [10:43] In our internal sprint, we found [10:46] that across all of the scans, [10:48] 36% of our findings were duplicates. [10:51] Removing those dupes meant less time [10:54] spent investigating the same issue [10:56] and much less work for developers. [10:58] From there, we had [11:01] to classify them using our security [11:03] rubric and prioritize what to fix. [11:06] The next two steps [11:07] in enriching the findings. [11:09] So we had to answer [11:11] two critical questions. [11:13] First is, how important are these [11:16] findings for OpenAI as a company? [11:18] And two, who should [11:20] own the vulnerability? [11:22] To answer the first question, we had [11:25] to define which types of vulnerabilities [11:27] we cared about the most. [11:29] We started with defining critical [11:31] vulnerabilities as ones where [11:33] external actors could gain [11:35] access they shouldn't have. [11:37] We had to iterate on this rubric [11:40] several times as we learned [11:41] more about the vulnerabilities [11:43] the scanners produced and how they [11:45] fit into our internal framework. [11:49] With the rubric, you can run [11:51] your findings through the CLI [11:52] and [11:56] get a prioritized list so you know [11:58] what to work on first, based on [12:00] what your company cares about. [12:04] Or if you're an indie developer, [12:05] what you care about as well. Shout out. [12:07] The next step is figuring out [12:10] who should work on that finding. [12:12] For me, this was actually the most [12:15] surprisingly difficult part [12:17] of our OpenAI internal sprint. [12:19] Knowing who owns what [12:21] in a company that's moving so fast [12:24] and growing so quickly took actually [12:26] a large amount of human effort. [12:29] Once we codified it though, [12:31] we were able to use that codification [12:33] to reliably assign findings [12:35] to the relevant teams with minimal churn [12:38] in tickets being passed back and forth. [12:42] So, with all of these great enriched [12:44] findings that have been assigned, [12:46] the next step is actually fixing them. [12:49] At first, we started with a human [12:52] in the loop, assigning vulnerability [12:54] tickets to the teams and they would [12:57] prompt their Codex to go look at [12:58] the ticket they were assigned and go [12:59] fix the issue and then shepherd it [13:01] through a PR and get it to production. [13:04] As we learned more and more [13:06] and improved our processes, [13:08] we started auto-generating these [13:09] patches on behalf of the developers. [13:12] Some of you may be thinking, [13:14] no way, I do not want auto-generated [13:17] security patches in my system. [13:19] Which, that thought process is [13:21] why we found this process is more [13:23] of a dial that you turn up, [13:26] not a switch that you flip on. [13:28] We had to gain confidence [13:30] in the patches and once we had [13:32] the necessary context in place, [13:34] we were able to do so. [13:36] And the auto-generated patches [13:38] actually became a really valuable [13:40] tool and super useful as we [13:43] scaled this across the tens [13:45] of thousands of findings. [13:46] So, that led us, all [13:48] of that led us to the Codex [13:49] security CLI patch command. [13:52] So, you can run it against [13:54] the finding, tell the agent how [13:56] to spin up the dev environment [13:58] to test it with a special flag [14:01] like validation prompt file. [14:03] And you can put it in a, you [14:04] can, sorry, it can put up [14:06] a PR to fix that vulnerability. [14:09] When we started using this internally, [14:11] we had a less than 1% rollback [14:14] rate on auto-generated patches. [14:16] The 0.81% that Ian mentioned earlier. [14:19] Part of that low rollback rate [14:21] and the reason we could be [14:23] so confident in those auto-generated [14:25] patches was due to how we verified [14:28] the fixes before we merged them. [14:31] We baked that into a verify fix [14:34] command that you can use as a secondary [14:37] check on the patches themselves to make [14:40] sure the finding is actually fixed. [14:41] You can think of this as [14:43] an adversarial agent of sorts [14:44] against the fix that you've created. [14:47] So, this is now [14:50] our generated verified patch. [14:52] From here, we would put up [14:55] a PR and follow the normal SDLC [14:57] in your company to get it merged. [15:01] One of the best and, in my opinion, [15:03] the most important parts of having [15:05] access to these security primitives [15:08] via the Codex security CLI is [15:11] that you can integrate them [15:12] into the way that you work. [15:14] Generating a verified patch [15:16] is a great step towards [15:18] getting a vulnerability fixed. [15:20] That change then needs to go through [15:22] your own team's development process. [15:25] Every organization that I've [15:27] seen does that just a little bit [15:28] differently or very differently. [15:30] And you probably need at least [15:32] something like a code review, [15:33] maybe you have a QA step, [15:35] maybe there's product review, [15:38] or another step before you can [15:40] actually ship the code to production. [15:44] Having the security building [15:46] blocks available through the CLI [15:47] gives you the flexibility in how [15:50] these tools are used and where. [15:53] Once you have a defense [15:54] factory of your own, [15:56] how do you maintain the lead [15:59] within the defender's window? [16:00] And as the models themselves get [16:02] better and better at fixing these [16:04] vulnerabilities, this workflow [16:06] actually gets more and more useful. [16:08] So, Ian, can you show us a little [16:09] bit more about what we're seeing here? [16:11] Yes, sir. [16:12] Thank you, Kyle. [16:16] So, we're seeing a lot of improvements [16:17] in both patch quality and cost. [16:20] So, this is PatchBench. [16:21] It's a third-party eval which compares, [16:24] in this case, 5.6 Sol versus 6 Astra [16:26] using Codex Security's patch skill. [16:29] Obviously, you can see that Astra at [16:31] medium reasoning outperforms 5.6 Sol at [16:34] X-high pretty substantially [16:37] at less than half the cost per case. [16:40] And for reference, humans at [16:43] OpenAI benchmark right about here, 50%. [16:47] So, needless to say, we are working [16:49] very closely with research [16:52] so that as models get better, [16:54] your defense factory gets better. [16:56] And we expect a lot more improvement [16:59] here in the coming days and months. [17:03] So, as we rolled all [17:05] this out internally, we [17:06] ran into all the bottlenecks [17:07] that Kyle has talked about. [17:10] So, things like figuring out [17:11] which findings are duplicates, [17:13] validating the vulnerabilities, [17:15] actually patching them, and so forth. [17:17] This curve is an extremely [17:19] sanitized version of our internal [17:21] security workload over time. [17:24] So, you can see that there's [17:24] this huge spike of effort [17:27] and a large backlog [17:29] that needs to be burned down. [17:31] But, to generalize our experience [17:34] at OpenAI, I'm actually really [17:37] optimistic about the way this curve [17:40] will look for many organizations. [17:42] We eventually arrived at a steady [17:45] state in which there are much fewer [17:47] vulnerabilities that are found [17:49] much more quickly than before [17:52] the use of AI to find vulnerabilities. [17:54] And, you know, [17:56] you work down the backlog, [17:57] you patch the vulnerabilities, [17:59] you get down to that skinny part [18:00] of the tail, where you're catching [18:02] new issues before they merge, [18:03] and you're running periodic scans. [18:07] And I think this skinny [18:09] part of the tail, you know, [18:11] is very important, because you need [18:12] a way to keep that curve down, [18:15] which means that you need some way [18:17] to run these scans continuously [18:19] as the code base changes. [18:21] The CLI and the SDK [18:23] that we've open sourced are [18:25] like the nuts and bolts, right? [18:27] So, you can connect them [18:28] to your repositories and your review [18:30] process, you can string them together [18:32] yourself, you can put them into CI [18:34] CD or GitHub Actions and so forth. [18:37] But, we've also solved this problem [18:39] in a much more turnkey way through [18:41] Codex Security Cloud, which Sam talked [18:44] about in the keynote earlier today. [18:47] So, Codex Security Cloud runs [18:49] this continuous security loop so you [18:51] don't have to orchestrate it yourself. [18:53] You just hook up [18:55] your GitHub repositories. [18:56] And the best part about Codex Security [18:59] Cloud, and really one of the most [19:00] notable things about this launch, [19:01] is that it gives you access [19:03] to our Daybreak class models, [19:05] which are designed specifically [19:07] for security use cases. [19:09] So, that's what's running under [19:10] the hood in Codex Security Cloud. [19:13] So, I can show you a quick [19:14] demo so you know what it looks [19:15] like in action. [19:17] Step one, find Codex Security [19:19] Cloud in the plugin marketplace. [19:23] So, this is within Codex, you [19:25] open plugins, you search for Codex [19:26] Security Cloud and install it. [19:28] Open the plugin. [19:29] Also, remember to hook up [19:30] your GitHub connector. [19:31] So, we'll go [19:32] in and we'll run a new scan. [19:34] I'm going to set up this one [19:35] to run on every single new commit. [19:38] We will hit scan [19:40] and fast forward a bit. [19:43] And you get these results. [19:45] So, this is org-wide view [19:48] of what your security [19:50] pipeline looks like over time, [19:53] the issues that are found and so forth. [19:55] You can dig into these results [19:58] and look at [20:01] some of the specific findings. [20:02] And then optionally, you can [20:04] auto-generate a patch. [20:05] So, this is kind of the full continuous [20:07] loop that we were discussing earlier. [20:10] And it will open a PR [20:12] straight to GitHub. [20:14] So, if you want to run these [20:16] same capabilities that Kyle has shown [20:18] you in a much more automated way, [20:20] use Codex Security Cloud. [20:22] We rebuilt it. [20:23] It's now powered by our Daybreak [20:25] Blue model by default. [20:26] And you don't have to go through [20:27] any access requests or permissions [20:29] and so forth. [20:31] We're really so excited to give [20:33] you these tools that we've been [20:35] using internally at OpenAI. [20:37] So, you can do this in your own [20:39] code base, in your own projects. [20:41] Install Codex Security on NPM. [20:43] It is completely open source. [20:45] Or, you can find Codex Security [20:48] Cloud in the plugin marketplace [20:50] before the end of the week. [20:53] That's all. [20:54] Thanks, everyone, for coming. [20:56] Yeah.