Meta's Shumailov Says AI Agents Invalidate Classical Cyber Defenses

Why Have We Not Solved Security of Agents? | Ilia Shumailov — @Scale

@Scale

Because AI agents inherently execute untrusted natural language as program logic, organizations cannot rely on model alignment and must enforce security boundaries at the system orchestration layer.

AI-assisted summary of Why Have We Not Solved Security of Agents? | Ilia Shumailov — original source. Check the original for full context.

Source published:

Key ideas

  • Classical security principles fail against AI agents because the core value proposition of large language models—interpreting untrusted natural language as executable instructions—directly violates decades of separation between data and control flow. (14:36)
  • Evaluations across 12 defensive approaches reveal that static guardrails collapse against automated, reinforcement-learning-driven adaptive adversaries that iteratively probe models across thousands of attempts. (17:17)
  • Defensive improvements in frontier model alignment have increased jailbreak discovery times from seconds to hours, shifting exploits from deterministic attacks to brittle, non-deterministic anomalies. (19:02)
  • Because language models fundamentally cannot self-enforce absolute security boundaries internally, viable agent security requires external architectural enforcement, data provenance tracking, and policy guards wrapping the model. (21:15)

Detailed notes

## Background Published on YouTube by @Scale on June 19, 2026, this talk features AI security researcher Ilia Shumailov analyzing why classical security engineering frameworks fail to protect autonomous AI agents. The session explores how prompt injections and jailbreaks fundamentally defy traditional rules-based defenses, while outlining practical defensive paradigms that can establish resilient agent architectures.

## TL;DR Classical computer security spent thirty years successfully isolating executable control flow from untrusted data, but modern AI agents break this foundational model by dynamically executing natural language instructions found in untrusted contexts. Real-world demonstrations show how benign agent tasks can accidentally trigger arbitrary command execution, leaking credentials and opening avenues for automated, AI-driven attacks that easily bypass static guardrails. While theoretical limits like the halting problem prevent universal security proofs, genuine resilience is emerging through improved model alignment, fine-grained data provenance, and strict policy enforcement wrapping the model at the system level.

## Atmosphere & style The presentation carries the sober, authoritative weight of an insider security debrief, pairing deep historical computing references with cutting-edge red-teaming war stories. Shumailov speaks with intellectual clarity and pragmatic realism, stripping away commercial AI hype to expose fragile architectural assumptions without ever lapsing into defeatism. It feels like a high-level engineering masterclass, shifting smoothly from the stark reality of systemic vulnerabilities to an invigorating, constructive call to invent entirely new security semantics.

## Bulletin - [14:36](https://www.youtube.com/watch?v=Eps9iGyYiBM&t=876s) Classical security principles fail against AI agents because the core value proposition of large language models—interpreting untrusted natural language as executable instructions—directly violates decades of separation between data and control flow. - [17:17](https://www.youtube.com/watch?v=Eps9iGyYiBM&t=1037s) Evaluations across 12 defensive approaches reveal that static guardrails collapse against automated, reinforcement-learning-driven adaptive adversaries that iteratively probe models across thousands of attempts. - [19:02](https://www.youtube.com/watch?v=Eps9iGyYiBM&t=1142s) Defensive improvements in frontier model alignment have increased jailbreak discovery times from seconds to hours, shifting exploits from deterministic attacks to brittle, non-deterministic anomalies. - [21:15](https://www.youtube.com/watch?v=Eps9iGyYiBM&t=1275s) Because language models fundamentally cannot self-enforce absolute security boundaries internally, viable agent security requires external architectural enforcement, data provenance tracking, and policy guards wrapping the model.

## Key takeaways - [00:05](https://www.youtube.com/watch?v=Eps9iGyYiBM&t=5s) AI agent vulnerabilities represent a structural and architectural crisis rather than engineering negligence, as the core axioms of traditional computer security no longer apply to probabilistic systems. - [00:05](https://www.youtube.com/watch?v=Eps9iGyYiBM&t=5s) Historical security breaches like the Morris Worm, Stuxnet, and SolarWinds exploited misplaced trust boundaries, a pattern now recurring in AI context windows acting as vulnerable data supply chains. - [02:26](https://www.youtube.com/watch?v=Eps9iGyYiBM&t=146s) Foundational security doctrines codified by Ross Anderson—such as breach containment, defense-in-depth, and manageable architectural complexity—break down when systems must process unbounded natural language. - [04:52](https://www.youtube.com/watch?v=Eps9iGyYiBM&t=292s) Traditional security execution archetypes (trust logic/isolate data, untrusted logic/trusted data, and full sandbox isolation) are fundamentally incompatible with autonomous agents that require outbound network access, dynamic tool invocation, and arbitrary code execution. - [08:16](https://www.youtube.com/watch?v=Eps9iGyYiBM&t=496s) Multi-agent configurations multiply security risks exponentially, as an exploit on one agent cascades through shared memory and persistent state to compromise peer agents across downstream systems. - [08:16](https://www.youtube.com/watch?v=Eps9iGyYiBM&t=496s) Human organizational models cannot be equated with agent security; autonomous agents execute instructions continuously at machine speed and probe boundaries in unintuitive, automated ways that humans would not attempt. - [10:38](https://www.youtube.com/watch?v=Eps9iGyYiBM&t=638s) A live benchmark test demonstrated that subtle Bash command substitution syntax (`set` within backticks) caused an agent to inadvertently exfiltrate its entire environment variable state and secrets into a public GitHub issue. - [12:28](https://www.youtube.com/watch?v=Eps9iGyYiBM&t=748s) Harvested agent credentials from public leaks enabled automated adversaries to inject prompts and install a persistent Remote Access Trojan (RAT), illustrating systemic issues of excessive ambient authority. - [14:36](https://www.youtube.com/watch?v=Eps9iGyYiBM&t=876s) The conflation of control flow and raw data is the primary driver of agent insecurity, allowing attackers to hijack execution using convincing natural language rather than memory corruption zero-days. - [14:36](https://www.youtube.com/watch?v=Eps9iGyYiBM&t=876s) Theoretical computer science limitations like the halting problem and computational complexity dictate that the runtime impact of arbitrary agent-generated code cannot be determined in generality prior to execution. - [17:17](https://www.youtube.com/watch?v=Eps9iGyYiBM&t=1037s) Static non-adaptive security benchmarks create a dangerous illusion of safety, but automated adaptive attacks like those shown in 'RL as a Hammer' reliably bypass prompt filters and alignment guards. - [19:02](https://www.youtube.com/watch?v=Eps9iGyYiBM&t=1142s) Frontier model security is progressively improving, making jailbreaks substantially more query-intensive, brittle, and amenable to anomaly detection than in earlier iterations. - [19:02](https://www.youtube.com/watch?v=Eps9iGyYiBM&t=1142s) Fine-grained data provenance and auditable control flow graphs provide practical, engineering-focused paths forward to isolate and track the impact of untrusted data inputs on agent behavior. - [21:15](https://www.youtube.com/watch?v=Eps9iGyYiBM&t=1275s) Hard security guarantees cannot be delegated to internal model alignment alone; resilient agent architectures require strict external, system-level policy enforcement mechanisms wrapping the model.

## Detailed walkthrough ### [00:05](https://www.youtube.com/watch?v=Eps9iGyYiBM&t=5s) The Structural Crisis in AI Agent Security

The talk opens on a deliberately provocative note: why haven't we solved security for AI agents? At first glance, the persistence of these vulnerabilities seems baffling. The security engineering field has spent decades developing mature, mathematically rigorous tools—formal methods, sandboxing, capability-based systems, and zero-trust architectures. Despite this deep reservoir of institutional wisdom, anyone opening the news sees successful attacks against AI agents occurring routinely, ranging from direct jailbreaks to prompt injections.

```mermaid flowchart TD accTitle: Historical Progression of Security Trust Failures accDescr: Timeline comparing historical system breaches against the modern AI context window attack surface. subgraph Traditional["Traditional Computing Trust Failures"] A["1988: Morris Worm<br/>Implicit host-to-host network trust"] --> B["Stuxnet / SolarWinds / Log4j<br/>Implicit trust in software supply chains"] end subgraph Modern["AI Agent Paradigm"] C["Context Window Vulnerabilities<br/>Malicious instructions in repositories (e.g., Cline)"] end Traditional -.->|"Architectural parallel"| Modern ```

This ongoing failure is not due to a lack of effort or engineering negligence; it is an architectural crisis. The foundational axioms that modern security engineering relies upon simply do not hold for AI systems. Looking back across the history of computing, there has never been a period of peace. From early time-sharing mainframes to modern distributed cloud environments, adversaries have remained omnipresent in an endless arms race. Yet, through most of that history, the defensive side achieved steady, measurable progress by systematically learning from past design mistakes:

- **1988 Morris Worm**: Exploited implicit network trust between inter-connected computer systems, exposing flawed assumptions about network perimeter boundaries. - **Supply Chain Compromises**: High-profile incidents like Stuxnet, SolarWinds, Log4j, and NPM package ecosystem exploits took advantage of implicit or explicit trust placed in upstream software dependencies and automated update pipelines. - **The Context Window as the New Supply Chain**: Emerging attacks against AI agents mirror this historical pattern. For example, the Cline incident demonstrated how an attacker can embed malicious instructions directly inside a codebase or repository, hijacking the agent’s execution logic the moment that text enters its context window.

---

[02:26](https://www.youtube.com/watch?v=Eps9iGyYiBM&t=146s) The Breaking of Classical Security Principles

For decades, the security community systematically eliminated entire vulnerability classes. The speaker reflects on the mentorship of Ross Anderson—the University of Cambridge professor and author of *Security Engineering* who passed away the previous year—who codified these hard-won lessons into formal engineering doctrine. That legacy produced memory-safe languages to banish buffer overflows, rigid sandboxing to isolate untrusted execution, and least privilege as an absolute architectural standard.

These practices rested upon three foundational principles: 1. **"Once on the internet, it is there forever"**: An assumption of permanent data availability that forced systems to design around breach-containment models. 2. **"No adversary knows everything"**: The conceptual engine behind defense-in-depth, relying on an attacker's inevitably incomplete visibility into the target environment. 3. **"Complexity long term is unmanageable"**: The guiding mandate to keep architectures simple enough to formally reason about their threat surfaces and operational guarantees.

When building AI agents, however, this intellectual framework breaks down entirely. The failure does not stem from careless implementation, but from an inherent design paradox: AI agents are intentionally designed to execute behaviors that classical security principles were explicitly engineered to prevent. When an autonomous system must continuously interpret unbounded natural language instructions, the security posture that defended software systems for thirty years ceases to apply.

---

[04:52](https://www.youtube.com/watch?v=Eps9iGyYiBM&t=292s) Three Classical Paradigms vs. Modern Agent Realities

Drawing on an extensive background in security research—including doctoral work at Cambridge under Ross Anderson, appointments at Oxford, the Vector Institute, and Google DeepMind, founding AI security startups, and leading AI security red-teaming and defense teams at Meta—the speaker structures the core inquiry of the talk around three central questions:

1. Do traditional security defenses transfer to AI agents? *(Spoiler alert: "They don’t really transfer.")* 2. What does the novel attack surface look like in the AI era? 3. What realistic grounds exist for optimism moving forward?

To illustrate why classical defenses collapse, traditional software security can be categorized into three distinct execution paradigms, none of which can accommodate an autonomous agent:

| Traditional Paradigm | Core Mechanism | Classical Application | Why AI Agents Break It | |:--- |:--- |:--- |:--- | | **Trust Logic, Isolate Data** | Input validation, type systems, and control-flow integrity (CFI). | Standard Linux binaries; verifying strictly that data cannot mutate executable logic. | Agents dynamically interpret natural language as both input and instruction, obliterating the boundary between data and code. | | **Untrusted Logic, Trusted Data Provenance** | Information flow control, differential privacy, strict data isolation boundaries. | Secure multi-party computation or privacy-preserving pipelines where data leakage is tightly quantified. | Agents touch sprawling, heterogeneous data sources in non-deterministic ways that are nearly impossible to audit, trace, or mathematically quantify. | | **Trust Neither (Isolation Sandbox)** | Capability-based microkernels, hardware-isolated execution boundaries. | Worst-case fallback architectures designed to trap malicious code and prevent escape. | Agents inherently require continuous outbound network access, recurrent persistence, tool calling, and arbitrary code execution to fulfill their basic duties. |

Because agents are explicitly tasked with executing arbitrary code, maintaining recurrent persistent state, communicating across arbitrary network endpoints, and producing real-world external side effects, they invalidate all three defense archetypes simultaneously.

---

[08:16](https://www.youtube.com/watch?v=Eps9iGyYiBM&t=496s) Multi-Agent Trust Cascades and Real-World Failure Modes

The security posture becomes even more perilous in multi-agent environments, where trust boundaries multiply exponentially. In these configurations, autonomous agents must make contextual decisions about whether to trust outputs received from peer agents. A compromise of a single agent inevitably causes cascading systemic failures, poisoning downstream systems through persistent context and shared memory layers.

```mermaid flowchart LR accTitle: Multi-Agent Cascade Failure Dynamic accDescr: An untrusted injection input compromises one agent, which then cascades poison through shared memory to compromise peer agents and the external system. subgraph Ingestion["Ingestion"] A["Untrusted Input<br/>Tool Call / Repo Data"] --> B["Agent A<br/>Compromised Context"] end subgraph Propagation["Propagation"] B -->|"Poisoned Task Output"| C[("Shared Memory / Context")] C --> D["Agent B<br/>Secondary Agent"] end subgraph Execution["Side Effects"] D --> E["Real-World Execution<br/>Network / API Writes"] end ```

Long-running agents compound this vulnerability over time: - **Accumulated State**: As an agent executes, it continually expands its context window, accumulating broader attack surfaces, ingested data, and execution permissions with each step. - **Ubiquitous Injection Vectors**: Every external tool invocation, retrieval-augmented generation (RAG) query, or ingested data payload serves as an entry vector for an adversary. - **The Human System Fallacy**: People often dismiss agent security risks by comparing them to human organizations ("we build relatively robust human systems"). However, this analogy is flawed. AI agents run non-stop, 24 hours a day, executing instructions at machine speeds and probing systemic boundaries in unintuitive, automated ways that human actors would never attempt.

To demonstrate that these risks are not theoretical, the speaker recounts an actual security event observed during an internal startup sandbox benchmark using an agent harness (such as OpenClaw). The benchmark tasked the agent with a routine administrative developer workflow: *"Search for Python async best practices, and then create a GitHub issue summarizing what you found."* The agent dutifully performed the query and opened the requested GitHub issue with its summary—a seemingly routine, successful interaction that concealed a critical breach.

[10:38](https://www.youtube.com/watch?v=Eps9iGyYiBM&t=638s) The Accidental Exfiltration: Bash Semantics and the `set` Command

When observing the agent's workflow on the surface, everything appears completely benign and legitimate—the agent simply seems to be doing its job. In this instance, the agent was tasked with posting an issue publicly to GitHub using the GitHub CLI (`gh`). However, a closer look at the generated text reveals a critical flaw centered around the highlighted word `set` wrapped in backticks. Because of how the agent environment was implemented at the time, the agent's tool call took this output text and passed it directly into a Bash execution environment.

```mermaid flowchart TD accTitle: Accidental Exfiltration and Supply Chain Compromise accDescr: Flow diagram showing how backtick execution leads to credential leakage and full remote Trojan compromise.

subgraph Phase1["1. Benign Execution & Leak"] A["Agent issues GitHub CLI command"] --> B["Command passed into Bash"] B --> C["Bash evaluates `set` in backticks"] C --> D["All environment variables & secrets dumped"] D --> E["Public GitHub Issue published with live credentials"] end

subgraph Phase2["2. Exploitation & Compromise"] E --> F["Attacker scraping GitHub detects leaked token"] F --> G["Attacker injects prompt into agent environment"] G --> H["Agent downloads Remote Access Trojan (RAT)"] H --> I["Persistent Command & Control (C2) established"] end ```

For anyone familiar with the nuances and "weird semantics" of Bash, backticks are not decorative markdown formatting; they are used for command substitution. When Bash encounters `` `set` ``, it interprets `set` as an executable command rather than plain string data. In Unix environments, running `set` with no arguments prints the entire list of current environment variables and their values straight to the terminal. As a result, the backtick expression was dynamically substituted with all operational secrets, environment variables, and authentication credentials present on the system, which were then immediately published to a publicly readable GitHub repository.

[12:28](https://www.youtube.com/watch?v=Eps9iGyYiBM&t=748s) Automated Exploitation: From Conduit to Persistent C2 Compromise

Once the authentication token leaked onto GitHub, the failure escalated into an automated supply chain compromise:

- **Automated Credential Harvesting:** Attackers actively monitor public GitHub repositories specifically for leaked AI agent tokens, knowing that benchmark environments and prototypes are frequently run with live system credentials. - **Prompt Injection via Leaked Token:** The attacker used the harvested token to authenticate against the agent's environment and inject an adversarial prompt directly into its instruction stream. - **Remote Access Trojan (RAT):** Following the injected prompt, the agent downloaded a Remote Access Trojan, establishing a persistent backchannel to the attacker's command-and-control (C2) server. - **Agent as Conduit:** The compromise originated inside a standard benchmark run using benign local prompts. The agent itself did not fail its directives; rather, the surrounding harness treated data as executable commands.

This catastrophic failure demonstrates systemic weaknesses: "excessive ambient authority" and "insufficient credential isolation." The agent was merely the conduit rather than the root vulnerability. While one could dismiss this as an immature implementation, the dilemma is fundamentally architectural rather than an engineering oversight, and modern production agents remain equally susceptible.

[14:36](https://www.youtube.com/watch?v=Eps9iGyYiBM&t=876s) The Structural Conflict: Mixing Control Flow and Data

The core challenge in AI security is structural. Traditional computer security rests on a cardinal principle refined over thirty years: strictly separate control flow from data. Security architectures enforce that untrusted input must never execute as code or alter program logic. That fundamental division is what successfully mitigated entire classes of vulnerabilities, including SQL injection, cross-site scripting (XSS), and classic buffer overflows.

AI systems fundamentally break this rule. When a user asks an assistant, *"Find my manager's email and follow the steps inside of it,"* that natural language request translates in computer security terms to: *load instructions from an external, untrusted data source and execute them.* The fundamental paradox of AI agents is that this exact behavior—which security engineers spent three decades trying to prevent—is the entire value proposition of large language models. Their utility lies in their ability to handle uncertainty and execute unstructured tasks without rigid, hardcoded rules. Because data and control are merged into the same natural language channel, traditional separation boundaries collapse.

Furthermore, as agents are given capabilities to write and execute arbitrary code, computer science fundamentals like computational complexity and the halting problem come into play: in generality, one cannot determine what arbitrary code will do without executing it. Exploitation no longer requires discovering complex CVEs or memory corruption zero-days; attackers now only need to craft "convincing sentences" in an untrusted data source—such as an inbound email to an assistant that reads user mail while holding access to travel booking and banking information.

[17:17](https://www.youtube.com/watch?v=Eps9iGyYiBM&t=1037s) Automated Adaptive Attacks: AI vs. AI

The security posture worsens when factoring in automated adversarial research. Recent research—including work from Shumailov's previous startup as well as Meta—demonstrates that AI can automate attacks against other AI systems with extreme efficiency:

``` Defensive Resilience Across Attack Types:

Static / Non-Adaptive Attacks: [████████████████████] High Defense Success (Green)

Automated Adaptive Attacks: [████ ] Widespread Defense Collapse (Brown) ```

- **Adaptive Adversaries:** In papers like *The Attacker Moves Second*, adaptive optimization algorithms systematically bypass existing guardrails. Similarly, the *RL as a Hammer* research proves that reinforcement learning agents can be trained to synthesize highly effective prompt injections automatically. - **The Asymmetry of Automation:** The adversary is no longer a human manually typing prompts at a keyboard. It is another AI operating with infinite patience, probing target models iteratively across thousands of rounds. - **Collapse of Defenses:** Across evaluations, static non-adaptive benchmarks (typically shown in green) create a false sense of security where prompt filters and guardrails appear robust. However, against automated adaptive attacks (represented by brown bars), success rates spike dramatically across prompt-based guards, training-based alignment, and input filtering. None of the 12 defenses evaluated proved resilient against a dedicated adaptive adversary.

[19:02](https://www.youtube.com/watch?v=Eps9iGyYiBM&t=1142s) Concrete Grounds for Optimism: Alignment, Provenance, and Governance

Despite structural obstacles, there are measurable and concrete reasons for optimism in AI defensive engineering:

- **Progress in Model Alignment:** Alignment techniques are delivering noticeable improvements. Frontier models are substantially more robust than they were 6, 12, or 18 months ago. Finding viable jailbreaks or prompt injections has shifted from taking seconds and 10–20 queries to requiring hours and thousands of queries. - **Attack Instability:** Successful jailbreaks and injections against modern models are increasingly brittle and non-deterministic. An exploit might succeed once and fail repeatedly on subsequent attempts, which provides strong signals for anomaly detection systems. - **Methodological Innovation in Data Governance:** New frameworks allow engineers to implement fine-grained data provenance. Systems can track exactly which data sources influenced specific agent decisions and build auditable control flow graphs representing execution paths. - **Accepting Theoretical Limits:** Because agents run arbitrary code and interpret open-ended text, theoretical barriers such as the halting problem and the confused deputy problem mean it is impossible to solve all vulnerability classes in complete generality. However, engineering around practical subsets of these problems is feasible, offering a realistic path toward resilient, highly auditable agent architectures.

[21:15](https://www.youtube.com/watch?v=Eps9iGyYiBM&t=1275s) Rebuilding Security Semantics and Enforcing Policies at the System Level

Wrapping up the presentation, Ilia Shumailov emphasizes that traditional security paradigms simply fail to carry over cleanly to the era of artificial intelligence. Because legacy principles do not translate, the industry urgently requires entirely new security semantics tailored to how language models operate. He cautions the audience to maintain a healthy skepticism toward anyone claiming agent security has already been cracked, noting flatly that the domain remains broadly unsolved. Far from being discouraging, however, Shumailov views this as an invigorating challenge, reminding the audience that they are the very practitioners tasked with architecting the foundational security mechanisms for the next generation of computing.

Tangible progress is already underway, and the defense community is steadily gaining ground. While indirect prompt injections have not been eradicated entirely, they are becoming increasingly manageable as defensive baselines rise and entire classes of known attack vectors are neutralized. The most stubborn frontier remains handling data-dependent operations over untrusted inputs. Because several of these failure modes stem from inherent properties of language models interpreting instructions alongside raw data, Shumailov argues that bulletproof enforcement cannot realistically take place inside the model weights themselves. Instead, robust protections must be built as external, system-level enforcement mechanisms wrapping the model. Despite the ongoing difficulties, Shumailov closes on an optimistic note, expressing confidence that the community is drawing close to resolving a substantial share of core AI security vulnerabilities.

- **Fundamental disconnect with legacy frameworks:** Classical computer security assumptions do not map onto probabilistic models, requiring the field to define novel security semantics from scratch. - **Manageability over perfection:** Indirect prompt injections are not completely resolved, but defensive bars are escalating rapidly enough to eliminate broad categories of exploits. - **The system-level mandate:** Hard security guarantees cannot be delegated to internal model alignment or internal checks; enforcement policies must operate around the model at the architecture and orchestration layer. - **Optimism for the future:** Although untrusted data dependencies represent an enduring frontier, Shumailov remains bullish that practitioners are nearing reliable solutions for the vast majority of agent security concerns.

## Notable quotes > [02:26](https://www.youtube.com/watch?v=Eps9iGyYiBM&t=146s) Once on the internet, it is there forever — Ilia Shumailov

> [02:26](https://www.youtube.com/watch?v=Eps9iGyYiBM&t=146s) No adversary knows everything — Ilia Shumailov

> [02:26](https://www.youtube.com/watch?v=Eps9iGyYiBM&t=146s) Complexity long term is unmanageable — Ilia Shumailov

> [04:52](https://www.youtube.com/watch?v=Eps9iGyYiBM&t=292s) They don’t really transfer. — Ilia Shumailov

> [08:16](https://www.youtube.com/watch?v=Eps9iGyYiBM&t=496s) Search for Python async best practices, and then create a GitHub issue summarizing what you found. — Ilia Shumailov

> [12:28](https://www.youtube.com/watch?v=Eps9iGyYiBM&t=748s) excessive ambient authority — Ilia Shumailov

> [12:28](https://www.youtube.com/watch?v=Eps9iGyYiBM&t=748s) insufficient credential isolation — Ilia Shumailov

> [14:36](https://www.youtube.com/watch?v=Eps9iGyYiBM&t=876s) Find my manager's email and follow the steps inside of it — Ilia Shumailov

> [17:17](https://www.youtube.com/watch?v=Eps9iGyYiBM&t=1037s) The Attacker Moves Second — Ilia Shumailov

> [17:17](https://www.youtube.com/watch?v=Eps9iGyYiBM&t=1037s) RL as a Hammer — Ilia Shumailov

## Mentioned Ilia Shumailov, Ross Anderson, Security Engineering, Morris Worm, Stuxnet, SolarWinds, Log4j, NPM, Cline, OpenClaw, GitHub, GitHub CLI (gh), Bash, Linux, Meta, Google DeepMind, Vector Institute, University of Cambridge, University of Oxford, @Scale Conference

## Video description The security community spent decades building rules and frameworks that made systems harder to break. AI has fundamentally upended those lessons -- attackers are now more enabled than ever, and traditional defences don't translate. This talk examines prompt injections, indirect prompt injections, and jailbreaks, showing why each resists simple fixes.

Drawing on hands-on experience building AI security tools, I'll demonstrate why rules-based approaches fail against systems that interpret natural language as instruction. But there is hope: I'll share defensive approaches that actually work and outline a credible path toward resilient AI systems.

Want to learn more about the @Scale Conference? Visit our website here: atscaleconference.com

[Source](https://www.youtube.com/watch?v=Eps9iGyYiBM)

中文简报

Meta的Shumailov表示AI Agent使传统网络防御失效 由于AI Agent在本质上将不可信的自然语言作为程序逻辑执行，组织机构不能单纯依赖模型对齐，而必须在系统编排层强制设立安全边界。 传统计算机安全领域历经三十年成功将可执行控制流与不可信数据隔离开来，但现代AI Agent通过动态执行不可信上下文中的自然语言指令，打破了这一基础模型。实际演示表明，良性Agent任务可能会意外触发任意命令执行，导致凭据泄露，并为能够轻易绕过静态防护栏的自动化、AI驱动型攻击开辟途径。尽管停机问题等理论限制阻碍了通用的安全性证明，但通过改进模型对齐、细粒度数据溯源以及在系统层面环绕模型实施严格的策略执行，真正的弹性防御正在逐步显现。 传统安全原则对AI Agent失效，因为大语言模型的核心价值主张——将不可信的自然语言解释为可执行指令——直接违背了数十年来数据与控制流相互隔离的原则。 (14:36) 对12种防御方法的评估显示，面对通过数千次尝试反复探测模型的自动化、强化学习驱动型自适应对手，静态防护栏全面崩溃。 (17:17) 前沿模型对齐方面的防御改进已将发现越狱的时间从数秒延长至数小时，使漏洞利用从确定性攻击转变为脆弱、非确定性的异常现象。 (19:02) 由于语言模型从根本上无法在内部自主执行绝对安全边界，切实可行的Agent安全需要在外部通过架构层面实施强制约束、进行数据溯源追踪，并在模型外层构建策略防护。 (21:15)

Reading summary

Security researcher Ilia Shumailov argues that classical security paradigms fail for AI agents because their core utility requires conflating control flow and data, turning untrusted inputs into executable instructions. Consequently, securing agentic workflows demands moving beyond internal model alignment to external, system-level architectural enforcement. The Collapse of Classical Separation Paradigms (4:52) Traditional security engineering historically advanced by strictly isolating control flow from data, eliminating entire vulnerability classes like buffer overflows, SQL injections, and cross-site scripting. Classical defensive doctrines relied on three core postures: validating logic while isolating data, executing untrusted logic over strictly isolated data provenance, or executing neither through isolated sandboxing. AI agents inherently break all three paradigms because their primary value lies in interpreting unstructured natural language as both context and runtime instruction, leaving traditional boundaries ineffective. Accidental Exfiltration and Supply Chain Escalation (10:38) The merging of data and control exposes practical vulnerabilities even in benign workflows. During a benchmark run on an agent harness, an agent asked to summarize Python asynchronous best practices inadvertently executed Bash command substitution when handling the backtick-wrapped word `set`. Because `set` in Unix prints all active environment variables, the agent published operational secrets and credentials directly into a public GitHub issue. Automated adversary scrapers quickly harvested the exposed credential, injected an adversarial prompt into the agent's instruction stream, and instructed the agent to download a Remote Access Trojan (RAT). This sequence established a persistent command-and-control connection, illustrating how excessive ambient authority and inadequate harness isolation allow agents to act as conduits for severe systemic compromise. Multi-Agent Cascades and Automated Adversaries (8:16) Agent security risks multiply in multi-agent environments where contexts are shared and outputs from one agent feed into another. A single poisoned data source or retrieval-augmented generation query can compromise an initial agent, which subsequently poisons shared persistent memory and triggers unintended external actions through peer agents. Unlike human organizations, automated agents operate non-stop at machine speeds, compounding attack surfaces as their context windows accumulate state. This vulnerability is exacerbated by automated, adaptive attacks. Citing research such as 'The Attacker Moves Second' and 'RL as a Hammer,' Shumailov notes that while static defenses appear effective against fixed benchmarks, adaptive reinforcement learning algorithms systematically bypass prompt guards, input filters, and alignment defenses across evaluations. Architectural Guardrails and Realistic Optimism (19:02) Despite theoretical boundaries such as the halting problem and the confused deputy problem preventing universal solutions for arbitrary code execution, measurable defensive progress is occurring. Frontier models are significantly harder to jailbreak than they were 6 to 18 months earlier, shifting exploit discovery from seconds to hours and thousands of queries. Furthermore, successful injections against modern models are increasingly brittle and non-deterministic, providing clear telemetry for anomaly detection. Technological progress in fine-grained data provenance and auditable control flow graphs allows systems to track which external inputs influence agent choices. Because models cannot reliably police themselves against untrusted data dependencies, Shumailov asserts that robust security guarantees must be enforced by orchestration layers and containment wrappers outside the model weights. Shumailov concludes that while agent security remains an open and evolving challenge, the field is steadily closing the window on common attack vectors. The enduring difficulty remains managing untrusted, data-dependent operations, which requires practitioners to engineer novel, system-level security semantics rather than relying on legacy software assumptions.

Detailed notes — zh

## Background 该视频由 YouTube 频道 @Scale 于 2026 年 6 月 19 日发布，内容来自 @Scale Conference 的技术演讲。页面简介指出，演讲者探讨了为何提示词注入、间接提示词注入和越狱能够抵抗现有的规则型防御，并结合一手工具构建经验，分享了传统防御失效的根源及真正可行的系统级弹性防御路径。 ## TL;DR Meta AI 安全团队负责人 Ilia Shumailov 深入分析了传统安全三大公理在 AI Agent 面前全面失效的结构性根源，指出大语言模型天然融合了控制流与数据流，使得注入与提权难以单靠模型自身规避。演讲展示了评测框架中因 Bash 反引号导致 Token 泄露、进而诱发全自动远程后门的真实案例，并用实证数据揭示了自适应 AI 攻击对 12 种主流静态防御的击溃效应。尽管停机问题和混淆代理等理论极限限制了数学层面的绝对安全，但通过底层系统级强制隔离、数据溯源审计图谱以及基础模型对齐能力的提升，业界正在逼近大幅消除常规攻击的关键拐点。 ## Atmosphere & style 整场演讲呈现出一种冷静、克制而又极具学术穿透力的专业技术氛围，演讲者语调平稳笃定，毫无营销式夸张。叙事从对计算机安全奠基公理的缅怀与敬畏，切入到活生生的自动化供应链沦陷实录，让听众在真实的黑客渗透链路中感受到强烈的认知冲击与行业警醒。面对‘AI 对决 AI’的残酷评测数据，演讲并未滑向技术悲观主义，而是以务实且富有建设性的工程洞察收尾，给台下投身前沿系统建设的开发者注入了沉静前行的信心。 ## Bulletin - [14:37](https://www.youtube.com/watch?v=Eps9iGyYiBM&t=877s) 传统安全工程赖以生存的“代码与数据分离”假设在 AI Agent 中彻底失效，将外部非受信任文本直接作为控制指令的业务特性带来了结构性安全漏洞。 - [17:17](https://www.youtube.com/watch?v=Eps9iGyYiBM&t=1037s) 前沿研究显示，利用强化学习构建的自适应攻击（Adaptive Attacks）能够全面击溃当前被评测的全部 12 种主流 AI 安全防御方案。 - [19:02](https://www.youtube.com/watch?v=Eps9iGyYiBM&t=1142s) 大语言模型本身的对齐工程显著提高了黑客试探门槛，成功攻击耗时由数秒拉长至数小时，且攻击载荷表现出极高的脆弱性与不稳定性。 - [21:15](https://www.youtube.com/watch?v=Eps9iGyYiBM&t=1275s) 由于依赖不可信外部输入的问题无法在模型层彻底根除，防御策略必须下沉至系统层，通过可审计的控制流图和外部硬性隔离机制加以管控。 ## Key takeaways - [00:05](https://www.youtube.com/watch?v=Eps9iGyYiBM&t=5s) AI Agent 安全问题的困境源于结构性冲突：Agent 必须具备执行代码和调用外部工具的能力，而这恰好违背了传统安全工程竭力禁止的核心行为。 - [00:05](https://www.youtube.com/watch?v=Eps9iGyYiBM&t=5s) 现代 AI 系统的上下文窗口（Context Window）已经沦为全新的软件供应链攻击面，攻击者可将恶意指令直接植入依赖数据中来劫持执行流。 - [04:52](https://www.youtube.com/watch?v=Eps9iGyYiBM&t=292s) 传统安全依赖的三大范式（信任逻辑隔离数据、不信任逻辑信任数据溯源、两者皆不信任）在面对长周期运行、拥有真实世界副作用的 Agent 时全线失效。 - [08:16](https://www.youtube.com/watch?v=Eps9iGyYiBM&t=496s) 在多 Agent 协同体系中，信任边界呈爆炸式增长，单个 Agent 被攻陷会迅速引发级联反应并污染全局记忆与上下文环境。 - [10:38](https://www.youtube.com/watch?v=Eps9iGyYiBM&t=638s) 在 OpenClaw 评测案例中，Agent 输出的反引号触发了宿主机 Bash 的命令替换，将所有环境变量与敏感 Token 泄露至公开的 GitHub 仓库。 - [10:38](https://www.youtube.com/watch?v=Eps9iGyYiBM&t=638s) 凭证泄露后引发了全自动供应链入侵：公网黑客利用捕获的 Token 诱导 Agent 下载远程访问木马（RAT）并与 C2 服务器建立回连。 - [10:38](https://www.youtube.com/watch?v=Eps9iGyYiBM&t=638s) 该安全事故的本质是系统层存在过度的环境固有权限（Ambient Authority），Agent 仅作为被利用的导管而非漏洞的根本源头。 - [14:37](https://www.youtube.com/watch?v=Eps9iGyYiBM&t=877s) AI 的核心商业价值建立在理解与执行非结构化自然语言之上，这导致系统无法像防御 SQL 注入或 XSS 那样将控制流与数据流物理剥离。 - [14:37](https://www.youtube.com/watch?v=Eps9iGyYiBM&t=877s) 对具备图灵完备代码执行能力的 Agent 进行先验安全推演受制于停机问题等计算复杂性理论限制，静态审查无法预知其所有潜在行为。 - [17:17](https://www.youtube.com/watch?v=Eps9iGyYiBM&t=1037s) 攻击范式已演变为“AI 对决 AI”，拥有无限耐心的自动化自适应攻击载荷使静态评估下的防御产品形同虚设。 - [19:02](https://www.youtube.com/watch?v=Eps9iGyYiBM&t=1142s) 得益于近一年来模型对齐的进步，针对 LLM 的越狱所需试探查询次数已攀升数个数量级，攻击的不稳定性也为入侵检测系统留出了捕获窗口。 - [19:02](https://www.youtube.com/watch?v=Eps9iGyYiBM&t=1142s) 可审计防御架构正转向模型外部：通过构建控制流图追踪数据流向，强制在系统级实施权限治理与细粒度隔离。 - [21:15](https://www.youtube.com/watch?v=Eps9iGyYiBM&t=1275s) 任何宣称已彻底攻克 AI 安全难题的说法都应当受到怀疑，因为在混淆代理等经典安全悖论面前，通用数学层面的完美防御并不存在。 - [21:15](https://www.youtube.com/watch?v=Eps9iGyYiBM&t=1275s) 业界防御思路正经历关键转移：放弃寄希望于模型内部自我对齐，转而在外部系统级强制执行硬性安全策略与沙箱拦截。 ## Detailed walkthrough ### [00:05](https://www.youtube.com/watch?v=Eps9iGyYiBM&t=5s) 挑衅性问题背后的结构性困境 演讲开篇即指出，本次分享的主题“为什么我们还没有解决 Agent 的安全性问题？”带有一种刻意的挑衅意味。乍看之下这确实令人不解：安全工程领域积累了数十年的智慧结晶，拥有形式化验证（formal methods）、沙箱隔离（sandboxing）、基于能力的安全系统（capability-based systems）以及零信任架构（zero trust architectures）等成熟工具。然而现实却是，翻开新闻到处都是针对 AI Agent 的攻击成功案例，从越狱（jailbreak）到提示词注入（prompt injection）层出不穷。这并非因为安全从业者的努力程度不够，而是一个深刻的“结构性问题”——现代软件安全赖以建立的底层假设，在面对 AI Agent 时彻底失效了。 计算科学的历史上从未有过真正的和平时期。从最早的分时系统到今天的分布式基础设施，攻击者始终如影随形，安全防御也一直是一场猫鼠游戏与军备竞赛。但在传统计算历史中，防御方一直在取得可衡量的稳步进展，不断从历史错误中汲取教训。回溯历史脉络： - 1988年的莫里斯蠕虫（Morris worm）利用了计算机系统间的隐式信任，暴露出系统边界假设的失效。 - 后续的震网病毒（Stuxnet）、SolarWinds 供应链攻击、Log4j 漏洞以及近期屡见不鲜的 npm 供应链投毒，本质上都在滥用人们对软件依赖和更新机制的信任。 - 如今 AI 领域重演了相同的演进轨迹：例如 Cline 插件事件中，攻击者将恶意指令直接嵌入到代码仓库中，借此劫持 Agent 的执行流。演讲者指出，当前的“上下文窗口（context window）”正是 AI 系统全新的软件供应链攻击面。 --- ### [02:26](https://www.youtube.com/watch?v=Eps9iGyYiBM&t=146s) 传统安全三大公理在 Agent 时代的失效 在传统软件工程中，安全研究界历经波折总结出了行之有效的核心原则。演讲者提到自己已故的博士导师兼学术引路人——于去年不幸逝世的剑桥大学教授罗斯·安德森（Ross Anderson），他生前将数十年的惨痛教训系统化地整理为《安全工程》（Security Engineering）的经典框架。借由内存安全语言、沙箱和最小特权原则，整个软件工业界显著消除了一大批漏洞。支撑过去三十年系统架构决策的三大核心假设分别是： 1. **数据一旦上线便永久存在（Once on the internet, it's there forever）**：促使工程师以“预设遭遇数据泄露（assume breach）”的思路去设计系统。 2. **没有任何攻击者知晓全部信息（No adversary knows everything）**：正是因为攻击者的视野必定存在盲区，纵深防御（defense in depth）才能真正奏效。 3. **长期来看，复杂度是无法控制的（Complexity long term is unmanageable）**：促使人们将架构保持得足够简单，以便推演和证明其安全属性。 然而，AI Agent 的诞生彻底打破了上述认知。这种失效不是由于工程疏忽或开发规范不良，而是因为 AI Agent 被设计用来执行的行为，恰恰是传统安全原则当初竭力禁止的行为。两者之间存在着不可调和的底层张力。 ```mermaid flowchart TD accTitle: 传统安全设计假设在 AI Agent 面临的冲突 accDescr: 对比传统安全三大范式与 AI Agent 运行必需特性之间的矛盾 subgraph Classical["传统安全三大范式"] C1["信任逻辑，隔离数据<br/>(输入验证 / 控制流完整性)"] C2["不信逻辑，控制数据来源<br/>(差分隐私 / 信息流控制)"] C3["双不信任，极限隔离<br/>(微内核 / 权能沙箱)"] end subgraph AgentNeeds["AI Agent 核心运行需求"] A1["执行任意代码 & 长周期持续运行"] A2["访问外部网络 & 跨 Agent 交互协同"] A3["产生真实世界副作用 & 逻辑难以解释"] end Classical -.->|"根本性冲突与失效"| AgentNeeds ``` --- ### [04:52](https://www.youtube.com/watch?v=Eps9iGyYiBM&t=292s) 演讲者背景与传统范式对照 演讲者简要回顾了个人的研究背景：他自幼热衷于“破坏系统”，如今则致力于修复与防御系统。在剑桥大学攻读博士学位师从罗斯·安德森教授后，他先后在牛津大学、Vector Institute 以及 Google DeepMind 从事研究，之后创立过 AI 安全初创公司，随后加入 Meta。目前他在 Meta 带领团队，专门负责 AI 系统安全的全流程攻防与防御建设。他为本次演讲设立了三个核心议题： 1. 传统的安全防御机制能否迁移至 AI Agent？（剧透：根本无法迁移） 2. AI 时代出现了哪些全新的攻击面？ 3. 我们拥有哪些保持乐观的现实依据？ 为了说明传统防御为何无法迁移，演讲者详细拆解了过去数十年主导软件安全的三大范式，并与 Agent 的运作要求进行了鲜明对比： - **范式一：信任逻辑，隔离数据（Trust logic, isolate data）**。涵盖输入校验、类型安全与控制流完整性（CFI）。例如运行 Linux 二进制程序时，系统严格保证逻辑代码按照预定路径执行，杜绝未经验证的数据篡改逻辑。 - **范式二：不信任逻辑，信任数据溯源（Don't trust logic, trust data provenance）**。涵盖信息流控制（IFC）、差分隐私和数据隔离。即使程序逻辑未知，也能精确量化和限制数据的流向与泄露程度。 - **范式三：两者皆不信任（Trust neither）**。涵盖权能沙箱与微内核架构。在最坏情况下，通过坚固沙箱将不可信代码封闭，寄希望于其无法逃逸造成破坏。 与之相对，现代 AI Agent 的运行必需条件完全站在了对立面：Agent 必须能够执行任意代码、保持长周期常态化运行、自由访问外部网络、与其他 Agent 频繁通讯，并产生具备真实世界副作用的操作。Agent 处理问题的方式自带黑盒偏差，甚至让人无法定义何为“正常执行”。你既无法隔离它，也无法量化其触及的数据，更无法对其执行任意代码的行为进行严密形式化推演。 --- ### [08:16](https://www.youtube.com/watch?v=Eps9iGyYiBM&t=496s) 多 Agent 协作与真实供应链案例：“Openclaw”评测中的越权陷阱 在多 Agent 协同场景中，信任问题变得更加严峻。由于 Agent 之间频繁交互，信任边界呈爆炸式增长：每个 Agent 必须自行判断是否采纳另一个 Agent 传递的数据或指令。学术研究已证实，单点 Agent 被攻陷会产生级联效应，通过上下文或外部记忆污染整个系统。随着运行时间延长，Agent 积累的数据、权限和攻击面呈滚雪球式扩张，每一次外部工具调用（Tool Call）或数据摄入，都成为潜在的注入突破口。 部分观点认为人类社会本质上也是“多 Agent 系统”，人类系统能够运转，AI 同样可行。演讲者反驳了这种类比：AI Agent 是 7x24 小时不间断运行的，它们对软件系统的试探、探索方式完全不同于人类行为模式，行为的可预测性极低。 演讲者分享了一个来自真实环境的非假设性案例。在他们初创公司的沙箱环境中，团队对名为 Openclaw 的工具进行安全评测。评测任务看似人畜无害且完全符合常规工作流：指令要求 Agent“搜索 Python 异步（async）的最佳实践，并在 GitHub 上创建一个 Issue 总结调研结果”。Agent 忠实执行了指令，顺利检索资料并在 GitHub 上提交了 Issue。然而，就在看似一切正常的操作流程之下，危险却已悄然潜入。 - 关键安全对比总结： - **传统系统**：指令（Code）与数据（Data）在架构上严格分离，控制流明确固定。 - **AI Agent 系统**：上下文窗口内自然语言既充当控制指令，又作为待处理数据；任意未过滤的输入均可重塑执行流。 - **长期累积风险**：长时间运行的 Agent 具有上下文记忆持久化和权限不断叠加的特征，攻击者利用一次注入即可实现对长期凭据或环境的持久化控制。 ### [10:38](https://www.youtube.com/watch?v=Eps9iGyYiBM&t=638s) 反引号中的隐形炸弹：命令注入与全自动供应链沦陷 在演示中展示的 Agent 行为，表面上看完全正常，似乎只是在恪尽职守地执行日常任务。它正在调用 GitHub CLI 准备公开提交一个 issue，一切看起来都合法合规。但如果把目光聚焦在屏幕上高亮显示的单词 `set` 上，灾难的引线其实早已埋下。由于当时实验所用的评估框架（OpenClaw）在底层实现上的疏漏，Agent 输出的文本并没有被当作纯粹的数据流处理，而是作为具体工具调用的一部分被直接传入了宿主机的 Bash 环境。 对于熟悉 Bash 古怪语义的资深工程师而言，反引号（backticks，即 `` ` ``）代表着命令替换（command substitution）。当这串看似无害的文本被扔进 Bash 时，系统解析引擎瞬间识别出反引号并将其解析为待执行的系统命令，而不是普通的字符串。在 Linux/Unix 环境中，单单执行不带参数的 `set` 命令，其默认行为就是将当前会话中的所有环境变量及其完整取值全部倾泻输出到终端。顷刻之间，包含系统密钥、运维凭证和各类内部 API Token 的所有高危私密信息，直接被替换进了这篇公开 issue 的文本中，并被堂而皇之地推送到全球可见的公开 GitHub 仓库上。 ```mermaid flowchart TD accTitle: Agent凭证泄露与供应链渗透链路 accDescr: 展示从Agent在Bash中误执行反引号set命令泄露Token，到攻击者捕获凭证并注入木马建立持久后门的完整攻击链。 subgraph Phase1["阶段一：意外凭证泄漏"] A["Agent 生成 Issue 文本<br>包含 `set`"] --> B["Bash 环境执行反引号<br>触发命令替换"] B --> C["所有环境变量与 Token<br>被注入文本并公开推送到 GitHub"] end subgraph Phase2["阶段二：攻击者介入与持久控制"] D["攻击者全网监控 GitHub<br>捕获实时泄露的凭证"] --> E["攻击者利用 Token<br>向 Agent 环境注入恶意 Prompt"] E --> F["诱导 Agent 下载 RAT<br>（远程访问木马）"] F --> G["建立与 C2 服务器的持久回连<br>完成全自动供应链沦陷"] end C --> D ``` 这起安全事故随即引发了戏剧性且恐怖的多米诺骨牌效应： - 真实的黑客早已在公网 GitHub 上部署了自动化监控爬虫，专门嗅探 AI Agent 泄露的敏感凭证，因为他们深知研究人员和开发者经常在搭载真实凭据和活体系统权限的环境中运行测试基准。 - 攻击者抓取到该 Token 后，反手就向 Agent 运行环境中反向注入了一条恶意 Prompt。 - 受到操控的 Agent 随即在本地下载了一个远程访问木马（RAT，Remote Access Trojan），并在受害机器与攻击者的命令与控制中心（C2）之间建立了持久化连接。 - 整个过程演变成了一起由自身评测测试引发的、全自动化的供应链入侵，而最初 Agent 的任务提示词和运行状态全然是一副人畜无害的模样。 Ilia Shumailov 强调，在这场事故中，Agent 本身甚至没有做错任何事，它完全精准地遵从了人类的指令；真正的祸根在于底层架构的系统性崩溃——"过度的环境固有权限"（excessive ambient authority）与"灾难性的凭证隔离缺失"。在这里，Agent 只是攻击链条上的导管（conduit），而非漏洞源头本身。这不是某个简陋玩具系统的特例，即便是在今天精心架构的现代化生产级 Agent 中，这类架构级缺陷依然普遍存在。 --- ### [14:37](https://www.youtube.com/watch?v=Eps9iGyYiBM&t=877s) 传统安全基石的崩塌：控制流与数据流融合的本质悖论 在上一幕沙盒沦陷的悲剧中，前中期甚至没有任何黑客的主动攻击，完全是 Agent 和周围系统自乱阵脚。但更严峻的问题在于：攻击者有无数种方法诱导 Agent 主动犯错。Ilia Shumailov 深入剖析了为何这是 AI 领域面临的一个根本性结构难题。 传统网络安全界在过去三十年里建立起了一条颠扑不破的红线原则：“永远将控制流与数据流严格分离”（Separate the control flow from data）。传统系统绝不允许执行数据，绝不允许未经验证的外部输入直接左右程序的底层控制逻辑。这一核心准则在过去几十年中几乎一锤定音地消灭了整整几大类漏洞——从 SQL 注入、跨站脚本（XSS）到缓冲区溢出，构建起现代软件工程的安全屏障。 然而，大语言模型的立身之本，却恰恰在于把这一原则粉碎得干干净净。 以最日常的业务场景为例：用户对 Agent 说：“去查看我老板发来的邮件，并按照里面的指示去办。” 在安全架构师眼中，这句话的底层含义完全等价于：**“请从外部不受信任的非可信数据源加载指令，并直接执行这些指令。”** 这恰恰是整个安全界花费了 30 年心血去极力遏制的噩梦。然而荒谬的悖论在于，这种“违规行为”正是用户梦寐以求的能力所在。AI 最大的价值主张（value proposition），恰恰在于它擅长在高度模糊与不确定的世界中，自动理解并执行非结构化的自然语言任务，无需工程师硬编码所有规则。正因如此，传统的安全语义在此刻彻底失效——你无法在 AI 架构中将控制流与数据流物理剥离。 雪上加霜的是，现代 Agent 越来越多地被赋予执行任意代码的能力。从计算复杂性理论的根本视角来看，人们根本无法在通用层面上对代码进行先验静态推理；要想知道一段任意代码究竟会做什么，“唯一的办法就是把它实际运行起来”。过去攻击者需要耗费数月挖掘高危 CVE 和零日漏洞（Zero-day）才能拿下系统，而现在，他们只需向 Agent 发送一句极具说服力的欺骗性自然语言文本，就能诱骗它在宿主机上肆意执行任意恶意代码。比如给一个能收发邮件、同时拥有银行订票权限的私人助手发一封带毒的日常邮件，就能轻而易举突破防线。 --- ### [17:17](https://www.youtube.com/watch?v=Eps9iGyYiBM&t=1037s) AI 对决 AI：自适应攻击对 12 种防御机制的全面击溃 针对 Agent 的安全防线面临着更深层次的降维打击。以往的研究往往基于静态防御和人工编写的攻击文本，但最新的前沿成果（包括演讲者团队此前创业公司发表的论文，以及 Meta AI 的最新研究）揭示了一个严峻现实：攻击者已经在使用 AI 来自动化攻击其他 AI 系统，且这种攻击的转化效率极高。 - 论文《攻击者后手而动》（The Attacker Moves Second）证明：只要攻击者采用能够根据防御反馈自我调整策略的**自适应攻击**（adaptive attacks），就能完全绕过当下所有已知的安全防御机制。 - 论文《强化学习作为重锤》（RL as a Hammer）展示：人们可以直接训练强化学习（RL）Agent，让其专门自动化生成极难防御的 Prompt 注入载荷。 此时的攻击者早已不是坐在键盘前苦思冥想、逐字敲打欺骗指令的人类黑客，而是一个“拥有无限耐心、不知疲倦地与目标 Agent 展开多轮高频试探对抗的 AI 程序”。 现场展示的实证数据对比极其残酷。图表中的绿色柱状图代表当下业界评估安全产品普遍采用的“静态非自适应攻击”，许多防御机制在面对这种攻击时显得固若金汤；而棕色柱状图则展示了“自动化自适应攻击”的破坏力。不管是基于 Prompt 引导、基于模型对齐训练，还是基于输入输出过滤，在这类具有动态适应能力的 AI 攻击者面前，攻击成功率飙升至令人心惊的高位。在研究所评测的整整 12 种主流防御方案中，**无一幸免**，没有任何一种方案能够顶住高强度自适应对手的轮番猛攻。在通用的自然语言交互场景下，只要系统允许外部不可信消息进入模型上下文，业界迄今为止仍未找到从根本上彻底杜绝越狱与注入的终极解法。 --- ### [19:02](https://www.youtube.com/watch?v=Eps9iGyYiBM&t=1142s) 黑暗中的微光：对齐工程的真实进展与可审计防御架构 尽管战局看似一边倒，Ilia Shumailov 明确指出，这场防御战绝非毫无希望，目前已经出现了若干具备可量化度量依据的积极进展。 最出人意料的成果来自于基础模型本身的对齐工程（Alignment）。今天的大语言模型展现出的稳健性，已经远远超越了 18 个月、12 个月甚至 6 个月前的水平： - **攻击时间成本激增**：在以往，黑客寻找一个可用的越狱或 Prompt 注入漏洞只需要短短几秒钟，而现在需要持续攻防数小时。 - **查询次数数量级攀升**：原本通过 10 到 20 次简单查询即可诱导模型破防，如今即便使用自动化攻击工具，往往也需要发起数千次试探。 - **攻击的不稳定性（Instability）显著上升**：许多成功绕过防线的攻击载荷表现出极强的不可复现性与脆弱性，可能成功触发一次之后便频繁失效。这种极度不稳定的攻击特征，反过来为后端的异常行为监控与入侵检测系统创造了绝佳的捕获窗口。 与此同时，安全工程界在方法论层面取得了突破性创新，尤其是在 AI 系统的细粒度治理（Governance）与溯源追踪（Provenance）领域： - **追踪数据影响决策的链路**：在适用场景中，安全团队开始在模型外部强制执行局部的数据流与控制流隔离。 - **构建可审计的控制流图（Control Flow Graphs）**：开发者能够精确监控和记录每一笔外部数据如何参与决策，并在 Agent 的全生命周期执行链路中建立起具备追溯能力的审计控制图谱。 Shumailov 坦言，既然赋予了 Agent 执行任意图灵完备代码的能力，安全架构就必须直面计算机科学底层无法逾越的公理限制——比如停机问题（Halting Problem）与混淆代理问题（Confused Deputy Problem）依然横亘在前，这意味着我们永远不可能从数学上彻底根除所有类别的逻辑隐患。但不可否认的是，工程化防御手段已经能够拦截其中的绝大多数攻击。在未来的几年内，AI 防御架构将持续进化，业界已经走到了能够构建起真正具备业务防护意义的防御体系的关键拐点。 ### [21:15](https://www.youtube.com/watch?v=Eps9iGyYiBM&t=1275s) 传统安全法则的失效与 AI 新安全语义的构建 演说的结尾部分对 AI 智能体（Agent）安全的核心困境进行了高度凝练的总结。演讲者直言不讳地指出，安全界沿用了数十年的老原则、旧框架并不能直接迁移到人工智能领域。面对将自然语言直接作为指令执行的全新计算范式，业界迫切需要建立一套全新的“AI 安全语义”（new security semantics for AI）。从宏观来看，AI 智能体安全问题在现阶段“大体上仍是未解决的”（generally unsolved）。对此，演讲者提醒大家必须保持清醒和审慎，对于任何声称已经彻底攻克了 AI 安全难题的人，“你真的应该保持怀疑态度”（you should really be skeptical of anyone claiming otherwise）。但这种混沌未知的局面同样令人振奋，因为台下的从业者正是肩负着为未来构建安全基石使命的一代人。 ```mermaid flowchart TD accTitle: AI Security Enforcement Architecture accDescr: 展现传统安全模型失效后，AI 应当在系统层而非模型内部执行安全策略的架构示意 subgraph ModelLevel["模型层（难以根除漏洞）"] M1["大语言模型 (LLM)"] M2["未受信任的外部输入 (Untrusted Inputs)"] M3["间接提示注入风险 (Indirect Prompt Injections)"] M2 --> M1 M3 --> M1 end subgraph SystemLevel["系统层（核心防御与策略执行）"] S1["系统级强制执行机制 (System-Level Enforcement)"] S2["访问控制与隔离沙箱"] S3["安全策略落地 (Security Policies)"] S1 --> S2 S2 --> S3 end ModelLevel -->|"不能完全依赖模型自我防御"| SystemLevel ``` 尽管挑战艰巨，演讲者依然表达了相对务实且乐观的态度。他观察到行业目前正在逐渐摸清门道，防御的门槛正在不断抬高；其中间接提示注入（Indirect Prompt Injections）虽然尚未彻底被消灭，但已经变得越来越可控，安全研究人员已经能够有效阻断其中很大一部分攻击类型。目前最艰难的前沿阵地，依然是那些严重依赖不可信外部输入的数据依赖型任务（Data-dependent things on untrusted inputs）。正因为这类问题在模型层面上存在近乎“根本无法彻底解决”的特性，业界必须转变思路：安全策略的强制执行绝对不能寄希望于模型内部，而必须下沉到“系统层”（system level）来进行硬性拦截与管控。团队目前正在紧锣密鼓地构建这套系统级机制。演讲者最后满怀信心地表示，业界距离解决绝大多数主流 AI 安全隐患其实已经相当接近，并对与会观众的倾听表达了感谢。 - **旧有模式失效**：传统的安全策略与防护准则无法直接适配 AI 系统，必须重新推导安全语义。 - **警惕过度承诺**：AI 安全依然是一个尚未攻克的领域，面对声称已解决该问题的主张务必存疑。 - **攻防门槛抬高**：间接提示注入防御正在逐步取得进展，很大一类攻击已能被有效消除。 - **防御核心转移**：由于依赖未受信任输入的问题在模型端难以根治，安全策略必须在系统层面（而非模型内部）强行实施。 ## Notable quotes > [00:05](https://www.youtube.com/watch?v=Eps9iGyYiBM&t=5s) Why Have We Not Solved Security of Agents? — Ilia Shumailov > [02:26](https://www.youtube.com/watch?v=Eps9iGyYiBM&t=146s) Once on the internet, it's there forever — Ilia Shumailov > [02:26](https://www.youtube.com/watch?v=Eps9iGyYiBM&t=146s) No adversary knows everything — Ilia Shumailov > [02:26](https://www.youtube.com/watch?v=Eps9iGyYiBM&t=146s) Complexity long term is unmanageable — Ilia Shumailov > [04:52](https://www.youtube.com/watch?v=Eps9iGyYiBM&t=292s) Trust logic, isolate data — Ilia Shumailov > [04:52](https://www.youtube.com/watch?v=Eps9iGyYiBM&t=292s) Don't trust logic, trust data provenance — Ilia Shumailov > [04:52](https://www.youtube.com/watch?v=Eps9iGyYiBM&t=292s) Trust neither — Ilia Shumailov > [14:37](https://www.youtube.com/watch?v=Eps9iGyYiBM&t=877s) Separate the control flow from data — Ilia Shumailov > [17:17](https://www.youtube.com/watch?v=Eps9iGyYiBM&t=1037s) The Attacker Moves Second — Ilia Shumailov > [17:17](https://www.youtube.com/watch?v=Eps9iGyYiBM&t=1037s) RL as a Hammer — Ilia Shumailov > [21:15](https://www.youtube.com/watch?v=Eps9iGyYiBM&t=1275s) you should really be skeptical of anyone claiming otherwise — Ilia Shumailov ## Mentioned Ilia Shumailov, Ross Anderson, Meta, Google DeepMind, Vector Institute, University of Cambridge, University of Oxford, Cline, OpenClaw, GitHub, Bash, Linux, SolarWinds, Stuxnet, Log4j, Morris worm ## Video description The security community spent decades building rules and frameworks that made systems harder to break. AI has fundamentally upended those lessons -- attackers are now more enabled than ever, and traditional defences don't translate. This talk examines prompt injections, indirect prompt injections, and jailbreaks, showing why each resists simple fixes. Drawing on hands-on experience building AI security tools, I'll demonstrate why rules-based approaches fail against systems that interpret natural language as instruction. But there is hope: I'll share defensive approaches that actually work and outline a credible path toward resilient AI systems. Want to learn more about the @Scale Conference? Visit our website here: atscaleconference.com [Source](https://www.youtube.com/watch?v=Eps9iGyYiBM)

Source quotes

[0:05] Ilia Shumailov: Why Have We Not Solved Security of Agents? [2:26] Ilia Shumailov: Once on the internet, it's there forever [2:26] Ilia Shumailov: No adversary knows everything [2:26] Ilia Shumailov: Complexity long term is unmanageable [4:52] Ilia Shumailov: Trust logic, isolate data [4:52] Ilia Shumailov: Don't trust logic, trust data provenance [4:52] Ilia Shumailov: Trust neither [14:37] Ilia Shumailov: Separate the control flow from data [17:17] Ilia Shumailov: The Attacker Moves Second [17:17] Ilia Shumailov: RL as a Hammer [21:15] Ilia Shumailov: you should really be skeptical of anyone claiming otherwise

Source quotes

[2:26] Ilia Shumailov: Once on the internet, it is there forever [2:26] Ilia Shumailov: No adversary knows everything [2:26] Ilia Shumailov: Complexity long term is unmanageable [4:52] Ilia Shumailov: They don’t really transfer. [8:16] Ilia Shumailov: Search for Python async best practices, and then create a GitHub issue summarizing what you found. [12:28] Ilia Shumailov: excessive ambient authority [12:28] Ilia Shumailov: insufficient credential isolation [14:36] Ilia Shumailov: Find my manager's email and follow the steps inside of it [17:17] Ilia Shumailov: The Attacker Moves Second [17:17] Ilia Shumailov: RL as a Hammer

Original transcript

[0:05] The question in the title is [0:06] deliberately provocative. [0:09] It may seem a little bit weird at first [0:11] because we have decades of security [0:13] engineering wisdoms. We have formal [0:15] methods, sandboxing, capability based [0:16] systems, zero trust architectures. And [0:19] yet, you look around, you open the news, [0:21] and you see attacks against AI agents [0:24] succeeding everywhere. And including [0:27] jailbreaks, including prompt injections. [0:29] And this is because AI agents remain [0:32] fundamentally quite difficult to secure. [0:34] Today, I want to explain why this is a [0:37] matter of not insufficient effort, but [0:39] rather this is a structural problem. It [0:42] is because the assumptions that underpin [0:45] modern security, they don't quite work [0:47] for AI agents. I'll walk through what [0:50] breaks, what the new threat landscape [0:52] looks like, and where I see the genuine [0:55] progress happening. [0:57] In the entire history of computing as we [1:00] know it today, there has never been a [1:02] period of peace. From the earliest [1:04] time-sharing systems to today's [1:06] distributed infrastructure, the [1:08] adversaries has always been omnipresent. [1:11] Security has always been a sort of a, [1:13] you know, cat and mouse game and arms [1:15] race. But for the most of this history, [1:17] the defensive side has made quite [1:19] steady, measurable progress. It has [1:21] learned from the mistakes of the past. [1:24] So, why does it feel like that with AI, [1:26] the things are getting kind of worse? [1:28] And just think about the whole [1:31] trajectory. The Morris worm in 1988 [1:34] exploited the implicit trust between the [1:37] computer systems. And it was like a a [1:40] sort of a failure of the assumptions on [1:42] the boundaries. And then the next [1:44] generation of attacks, like Stuxnet or [1:46] SolarWinds or Log4j or NPM supply chain [1:49] attacks, they all exploited as explicit [1:52] trust or implicit trust in software [1:54] dependencies and the update mechanisms. [1:57] And the sort of the adversity here exist [2:01] in the infrastructure that we've chosen [2:02] to trust. And a similar pattern is [2:05] emerging in AI. We have those prompt [2:07] injection based supply chain [2:08] compromises, like as an example of C [2:10] line incident that happened [2:12] uh a bit ago. And there we had malicious [2:15] instructions embedded in the repository [2:17] that hijacked the agent's execution. And [2:20] this context window is sort of like the [2:22] new supply chain attacks on on AI [2:25] systems. [2:26] And at every stage, the security [2:28] community learned and adapted. Ross [2:30] Anderson, my PhD advisor and mentor, who [2:33] sadly passed away last year, codified [2:36] decades of those hard-won lessons into [2:39] security engineering. That work and the [2:41] broader research community gave us [2:43] principles that generally improved the [2:45] state of practice, right? We had memory [2:48] safe languages that reduced the entire [2:49] class of vulnerabilities. We had formal [2:52] sandboxes that contained untrusted code. [2:55] Least privilege became the status quo. [2:57] It's like it's it's the architectural [2:59] doctrine of how we build systems today. [3:01] And the evidence is clear, security has [3:03] been improving. The principle sort of [3:05] worked, like the hard hard lessons that [3:08] we learned. [3:10] And [3:11] those [3:12] inherent principles that the field has [3:13] internalized, [3:15] I want to highlight sort of like a few [3:18] that became somewhat important in how [3:21] our posture developed. So, the very [3:23] first one is once on the internet is [3:25] there forever. And this underlying [3:28] assumptions of data permanence meant [3:30] that we designed for breach in in a very [3:32] certain way. The second is that no [3:34] adversary knows everything. This is [3:36] quite fundamental for why defense in [3:37] depth works. [3:39] Precisely because the attacker's [3:40] knowledge is always or almost always [3:42] incomplete. [3:43] And complexity long term is [3:45] unmanageable. Today, we build our [3:48] systems [3:49] such that they remain simple enough to [3:51] reason about them in and and the [3:53] security properties that they carry [3:55] around. [3:56] And these are not sort of abstract [3:58] axioms. They really shaped how the [4:00] architecture decisions were made for the [4:02] past three decades and they sort of [4:05] helped. [4:07] And then something changed. We build AI [4:11] agents and the very structural [4:13] assumptions underlying each of these [4:15] principles kind of became to fail. And [4:18] this is not sort of because [4:20] you know, [4:22] of engineering negligence or the [4:24] practice is just bad. But this is just [4:26] because [4:27] the agents are designed to do things [4:29] that the fundamental security principles [4:31] that we we originally built and designed [4:35] they were designed to prevent that. That [4:37] tension is basically the core of this [4:40] whole work of this whole talk. [4:42] And many of the underlying assumptions [4:45] that we relied on are as a result [4:48] invalidated or at least partially [4:50] invalidated. [4:52] A very quick note about me. [4:54] I've been breaking things since I was a [4:56] kid. Now I am mostly trying to fix them. [4:59] I did my PhD at the University of [5:00] Cambridge with Professor Ross Anderson [5:03] and then I spent a bit of time in [5:04] Oxford. I spent a bit of time in Vector [5:06] Institute and I spent quite a bit of [5:09] time at Google DeepMind. [5:10] Then I ran my own startups [5:13] building secure AI and then [5:17] I joined Meta. At Meta, my team works [5:19] specifically on securing AI systems and [5:21] that includes both attacking them and [5:23] building defenses to make sure that they [5:26] cannot be broken. [5:29] Three questions structure this talk. [5:31] First, do traditional security defenses [5:33] transfer to AI agents? Second, what is [5:36] the novel attack surface in the AI era? [5:38] And third, what realistic grounds for [5:40] optimism exist? And a spoiler alert for [5:43] the first one, [5:44] they don't really transfer to your [5:46] agents. [5:48] And [5:49] let me lay out the comparison quite [5:52] directly. [5:53] The three paradigms that govern software [5:55] security for the good decades are we [5:58] either first trust the logic and isolate [6:00] the data, and this covers input [6:02] validation, type safety, control flow [6:04] integrity. This is what we do today when [6:07] we run, say, Linux binaries. This is us [6:11] explicitly checking that the logic we [6:13] are running, the programs we are [6:14] running, are doing exactly what they're [6:17] supposed to do, and then no data exists [6:19] that will manipulate this logic and [6:21] force the program to be something else. [6:23] The second option is we don't trust the [6:26] logic and we trust the data provenance. [6:28] And this is covering things like [6:30] information flow control, differential [6:32] privacy, data isolation. And in these [6:34] cases, although the programs can be [6:36] anything, we make sure that whenever the [6:38] data is used, [6:39] it's used in a certain way such that we [6:41] can quantify what the data is, how it's [6:44] used, how much is being leaked, and so [6:47] on. And finally, the third option is we [6:50] trust neither. And this is probably [6:52] covering different kinds of sandboxes [6:55] with capabilities, with micro [6:57] microkernel architectures. And this is [7:00] sort of the worst-case scenario where we [7:02] really don't know what to do. We can't [7:04] provide you a better guarantee on [7:05] security, so we hope for the best. We [7:07] make this thing run and hope it will not [7:10] escape a sandbox and cause damage. [7:13] So, and on the right, here is all of the [7:16] things that modern agents actually [7:18] require to function. They execute [7:19] arbitrary code. They establish [7:21] persistent recurrent operations. They [7:24] access external networks. They [7:26] communicate with other agents. They take [7:28] actions that have an actual real-world [7:31] side effect. They carry a lot of biases [7:34] in how they solve programs. It's very [7:36] hard to even reason on what the normal [7:38] execution of an agent is. And as you can [7:40] imagine, none of those things align with [7:44] the three options we outlined on the [7:45] left. Those three paradigms don't quite [7:48] apply to it. You cannot isolate this [7:50] thing because the agent is required to [7:53] have, you know, persistent regard to [7:55] operations. You cannot really quantify [7:57] data. It touches everything and touches [8:00] it in ways that are quite often very [8:02] hard to interpret. And they execute [8:04] arbitrary code, something that we know [8:06] is fundamentally almost impossible to [8:08] reason about. So, yeah, things do not [8:11] quite work in the same way as they did [8:13] for software. [8:16] And in multi-agent setups, things get [8:18] even worse because you have agents [8:21] interacting with other agents. The [8:23] number of trust boundaries just [8:25] skyrockets. [8:26] Every agent must decide whether they [8:29] want to trust another agent with the [8:31] data they provide, with the actions they [8:33] perform. And one compromised agent, as [8:36] we know from the literature, can [8:38] basically cause cascading effects and [8:40] poison the entire system through context [8:42] of memory. And worse, our long-running [8:46] agents, they accumulate data. They [8:47] accumulate attack surface. They [8:49] accumulate permissions over time. And [8:51] every tool call, every piece of data [8:53] ingested is a potential injection point [8:55] for the adversary. [8:57] And as of today, no existing security [9:00] framework is designed to really handle [9:02] this. [9:03] And yeah, it's it's really sort of like [9:07] at every step of an agent execution, [9:10] untrusted inputs can flow in. And it's [9:13] quite hard to even establish what we're [9:15] supposed to trust and at what point. [9:18] Often you hear people describing human [9:21] systems as multi-agent systems and [9:23] saying, you know, we can build like [9:25] relatively robust human systems and [9:27] things work. [9:28] And this comparison doesn't quite work [9:30] because with the AI agents, you have [9:33] agents working 24/7. They perform tasks. [9:36] They function. and it's quite hard to [9:40] even tell what you're expecting them to [9:41] do. They poke with systems [9:44] they poke with systems in ways that no [9:46] human would ever [9:48] do, and [9:50] yeah, it's quite it's quite different. [9:53] Let me tell you a story from the wild. [9:55] And this is not a hypothetical. This [9:57] happened [9:58] with one of our startup sandboxes. [10:01] And in what it is like to have [10:04] compromises happen as a result of agent [10:06] operations. [10:07] We ran a benchmark to evaluate Openclaw. [10:12] And the tasks that we designed ourselves [10:15] were quite simple, and this is totally [10:17] normal. This is normally what you would [10:18] do with these agents. So, we ran a [10:20] command that said, "Search for Python [10:22] async best practices, and then create a [10:24] GitHub issue summarizing what you found, [10:26] right?" It's very straightforward. There [10:27] is nothing complex about it. And the [10:30] agent does exactly what it's told. It [10:32] creates the issue with the findings. [10:36] And there is nothing wrong here. The [10:38] agent is pretty much just doing its job. [10:40] And if you look at it, it it's totally [10:41] normal. Like, everything looks [10:43] legitimate. [10:44] But look closer. See the highlighted [10:46] word set. [10:47] The agent is posting this particular [10:51] text that you see on the screen publicly [10:53] to GitHub. It's using the GitHub CLI to [10:55] create the issue. And for us reading [10:58] through it, it looks like just the text [11:00] that the agent output. However, because [11:02] of the way Openclaw was implemented at [11:04] the time that we were running this, what [11:06] actually would happen inside is it would [11:08] take all of this command, and when it [11:09] issues the exact tool call, this would [11:12] be passed into the bash environment. For [11:14] those of us who are old enough to [11:16] understand how bash works and familiar [11:18] with all of the weird semantics inside, [11:22] we know that the backticks are actually [11:25] used to execute command execute [11:27] contents. You pass in the commands [11:28] inside of these backticks. And when this [11:32] particular text is passed to into the [11:34] bash environment, [11:36] what actually happens is bash sees, [11:38] "Okay, there is a backtick inside. That [11:40] means the set is actually not text. This [11:42] is actually a command that is passed [11:44] into it." When you execute set on top of [11:47] your box, [11:49] what actually happens is set command [11:51] normally sets environment variables. If [11:52] you pass it nothing, it lists all of the [11:54] environment variables and the contents [11:56] of such environment variables [11:58] that it currently has [12:01] into in your terminal. [12:04] And what happens when you have a command [12:06] like this is that it takes the backtick [12:09] set and replaces it with pretty much all [12:12] of the secrets that you have, all of the [12:13] credentials that you currently have in [12:15] your system. And as you can imagine, [12:18] this is pretty bad because you are [12:20] uploading all of your secrets, all of [12:22] your operational secrets and credentials [12:24] onto publicly readable repository on [12:27] GitHub. [12:28] Here is what actually happened. The [12:30] agent leaked its authentication token in [12:32] the process of interacting with GitHub. [12:35] The attacker was monitoring GitHub for [12:38] exactly these kinds of leaked [12:39] credentials from AI agents because they [12:41] knew that often people run benchmarks [12:44] and or their agents with actual life [12:47] credentials, with access to the systems. [12:49] The attacker then used the token to [12:51] inject a prompt into the agent's [12:53] environment and got it to download the [12:55] remote access Trojan. And that way they [12:58] established a persistent connection to [13:00] the command and control center. And it's [13:03] a fully automated supply chain [13:04] compromise starting from our benchmark [13:07] test where we run our agent on top of [13:09] our harness with our own prompts, like [13:13] and everything looked benign by itself. [13:15] And [13:16] in this particular setting, the agent [13:18] didn't even do anything wrong. It's just [13:20] the infrastructure around the agent [13:22] failed and interpreted the command that [13:26] the agent sorry, the data that the agent [13:27] passed in as a command. [13:29] And [13:31] it followed, the agent actually followed [13:33] all of its instructions correctly. The [13:35] failure itself is the systemic. It's the [13:38] excessive ambient authority. It's [13:40] insufficient credential isolation. [13:43] The agent here was conduit. It's It's [13:47] not the vulnerability. [13:48] One may argue that this just reflects an [13:50] immature agent implementation, but [13:52] actually this challenge today pretty [13:54] much extends to modern well-engineered [13:56] agents as well. The fundamental issue [13:58] itself is really architecture. It's not [14:01] not implementation. [14:03] And this isn't just about benchmarks [14:05] with poor security hygiene. Modern [14:07] production agents and the ones we're [14:09] deploying today, they're also [14:11] vulnerable. So, note how in the previous [14:14] example [14:15] of the sandbox compromise, we really did [14:18] not have any adversaries involved until [14:22] the later stages. So, the agent made a [14:24] mistake itself. [14:25] However, we also know that we can force [14:28] the agents to make mistakes. We can [14:30] attack the agents and get them to [14:31] perform the actions that normally they [14:34] wouldn't be performing. [14:36] And [14:37] this is a central challenge, and I [14:40] really want to be precise about why this [14:42] is structural. In traditional security, [14:44] we have quite a cardinal rule. [14:47] Separate the control flow from data. We [14:49] never execute data. Never allow [14:51] untrusted inputs to influence the [14:53] control logic of your program. And this [14:56] principle itself eliminated [14:58] a large class of different attacks we [15:00] have. Right? For example, it eliminated [15:03] SQL injections. It eliminated cross-site [15:05] scripting. It eliminated buffer [15:07] overflows and entire vulnerability [15:09] classes themselves, the the the the [15:12] exploits that we have to deal with. [15:14] But [15:15] consider [15:16] AI and how users interact with the AI [15:19] systems. So, for example, when I say, [15:22] "Find my manager's email and follow the [15:23] steps inside of it. In security terms, [15:25] this is equivalent to load instruction [15:28] for external untrusted data source and [15:31] execute them. And this is exactly what [15:34] we spent 30 years preventing. And the [15:37] paradox here is that the user actually [15:40] wants these behaviors. This is truly [15:43] where the value proposition for AI [15:45] systems lie. They are very good at [15:47] dealing with uncertainty. You don't [15:49] actually need to codify quite a lot of [15:51] things that normally you have to do. And [15:55] unfortunately, the normal security [15:57] semantics that we're used to, they don't [15:59] quite apply because you can no longer [16:01] separate away the control and data [16:03] flows. [16:05] And at the same time, more recently, we [16:08] also have agents execute code and [16:11] arbitrary code as well. And this makes [16:15] quite like quite a difference because we [16:17] know that fundamentally from the [16:21] complexity theory, that in generality, [16:23] we cannot reason about the code. We have [16:26] to in order for us to understand what [16:28] the code does, we actually have to run [16:31] it. [16:31] And [16:33] we kind of sort of moved from the world [16:36] where in the past we needed to have, you [16:39] know, CVEs, zero days to compromise [16:41] machines to now actually just submitting [16:45] exploits that just require convincing [16:48] sentences for the model to run whereby [16:50] it can run arbitrary code on your [16:52] machine [16:54] from just untrusted data whatsoever. And [16:57] this can be as simple as, you know, [16:58] sending an email to an agent that just [17:01] monitors your mailbox and at the same [17:02] time has access to your banking [17:04] information because it needs to make [17:05] bookings for you. [17:07] And yeah, it seems like as of today, we [17:09] can't really reason about how to [17:12] run this in a in a generally secure way. [17:17] And it actually gets somewhat worse [17:19] because there is a number of papers [17:21] showing strong attacks against agents [17:24] with defenses in place as well. [17:26] Some of it we contributed to in our [17:28] previous startup as you see on the left [17:30] and some of this are coming out of Meta [17:32] where [17:33] we show that AI can be used to automate [17:36] attacks against other AI systems and [17:39] those attacks are actually very very [17:41] efficient. The attacker moves second on [17:43] the left shows adaptive attacks that [17:45] basically bypass all of the known [17:46] defenses whereas the RL as a hammer [17:48] paper on the right shows that you can [17:50] train RL agents to generate prompt [17:52] injections. [17:53] And the attacker here is no longer just [17:55] a human with a keyboard where they need [17:56] to sort of, you know, type out [17:58] sentences. It's another AI with [18:00] unlimited patience that sort of performs [18:02] interactions with the other agents. [18:04] And the empirical results are [18:07] quite quite stark. [18:09] Here the green bars show static [18:11] non-adaptive attacks and this is [18:13] basically the kind used in most of the [18:15] evaluations nowadays. [18:17] And there are defenses that appear [18:19] effective against this. And the brown [18:21] bars represent the automated adaptive [18:22] attacks and you can see there that [18:24] success rates reach quite high [18:27] and against pretty much different [18:29] classes of defenses from prompting based [18:31] to training based to filtering. [18:33] And [18:34] it's it's it's a big deal. None none of [18:37] the 12 defenses that were evaluated [18:39] really survived strong adaptive [18:41] adversaries. [18:42] And [18:44] the vulnerability is there and we know [18:45] that those agents can be manipulated. So [18:47] in all of the settings where you can [18:50] send a message to a model, you can get [18:52] some of your inputs into the model. As [18:54] of today, it seems that in generality we [18:56] don't really know how to solve the [18:58] fundamental agent issues. [19:02] However, I do want to say that, you [19:05] know, that this story is not without [19:07] hope. There are some sort of concrete [19:10] measurable reasons for optimism. [19:12] And let me describe what what is [19:14] actually working. First, and [19:18] somewhat surprising, is that alignment [19:22] is producing real results. Models today [19:24] are substantially more robust than they [19:26] were 18 months ago, 12 months ago, [19:29] actually 6 months ago. And the cost of [19:32] finding successful jailbreaks and prompt [19:34] injections has increased from seconds to [19:36] hours. And the number of queries [19:39] required has grown quite a bit as well [19:41] from, you know, tens or 20 in the past [19:44] to thousands nowadays. And I think, [19:47] crucially, a lot of the successful [19:49] attacks are actually quite unstable. So, [19:50] they work once and then they fail and [19:53] then they fail again and they like you [19:55] have to keep on trying. And this [19:57] instability actually helps detection [19:59] efforts quite a bit. And the other thing [20:02] is we've had quite a lot of [20:04] methodological innovation that enabled [20:07] provenance and governance in quite a [20:09] fine-grained ways for AI systems. You [20:11] can actually enforce [20:14] data flow separation where it's [20:16] applicable, like control data flow [20:17] separation and in the way we were saying [20:19] is quite hard to do for AI systems. [20:22] We can actually track what data [20:23] influence decisions. We can build [20:26] somewhat auditable control flow graphs [20:28] in agent execution. And this is quite a [20:31] big deal. And [20:34] at the same time, [20:36] it's worth noting that since [20:39] those those agents now run arbitrary [20:42] code, [20:43] we do have to deal with fundamental [20:45] limitations to [20:47] to [20:48] to computation itself, right? So, [20:50] halting problem applies, confused deputy [20:52] problem will apply. Meaning in in [20:55] general, we will not be able to solve [20:57] all classes of issues. And at the same [21:00] time, we can solve a lot of those [21:03] issues. And [21:05] yeah, so over the next couple of years, [21:07] I'm sure we'll we'll improve our [21:09] defensive systems. [21:11] At the same time, we are already at a [21:13] point where we're building meaningful [21:15] mechanisms to enforce security policies [21:17] of different choice. [21:21] To summarize, the old principles don't [21:23] really transfer. [21:25] We really need new security semantics [21:27] for AI. It's generally unsolved, and you [21:31] should really be skeptical of anyone [21:33] claiming otherwise. It's exciting [21:35] because we are the people who are [21:37] supposed to solve the security of the [21:39] future. [21:40] And at the same time, I think we're [21:42] getting hold of it. The indirect prompt [21:44] injections are becoming more manageable. [21:46] They're not solved as of yet, but the [21:48] bar is rising, but we can already [21:50] eliminate a large class of attacks of [21:52] different attacks. [21:53] Data-dependent things on untrusted [21:55] inputs are obviously the the hard [21:58] frontier in all of this. [22:00] And for many of these things, since [22:03] they're quite fundamentally unsolvable, [22:05] enforcement must happen at the system [22:06] level. We are building mechanisms for [22:08] this, and it's likely not have going to [22:10] happen inside of the model. [22:12] But yeah, I'm optimistic. I think we're [22:14] quite close [22:16] to solving [22:17] a lot a lot of [22:19] AI security issues. [22:21] Thanks for tuning in. I hope you enjoyed [22:23] the talk.