NATO Official Deploys Basement AI Agents for Threat Forecasting

Agents in the Basement with NATO Support and Procurement Agency — Ai4

Ai4

Autonomous threat scanning requires strict human oversight and continuous trace logging to convert automated intelligence into credible forecasts.

AI-assisted summary of Agents in the Basement with NATO Support and Procurement Agency — original source. Check the original for full context.

Source published:

Detailed notes

## Background Published on YouTube by channel Ai4 on August 20, 2026, this video features a presentation by Sergio Coronado, CIO at the NATO Support and Procurement Agency. According to the page description, the talk explores real-world experience building supervised agentic workflows for sensitive research and threat forecasting using open-source tools, local models, and public sources while keeping human judgment paramount.

## TL;DR Sergio Coronado details his methodology for conducting 24/7 cybersecurity threat forecasting using specialized AI agent teams operating on a local server setup in his Luxembourg basement. The multi-agent architecture employs Searcher, Extractor, Synthesizer, Skeptic, and Evaluator roles under a strict governance harness enforcing data privacy routing and safety 'red lines.' Supported by case studies predicting AI infrastructure attacks and perimeter edge compromises, Coronado demonstrates how automated agentic workflows can accelerate research while preserving absolute human oversight. He concludes with a four-step practical guide for practitioners to build their own supervised local agent systems.

## Atmosphere & style The session carries a pragmatic, authoritative, and deeply practical tone, balancing high-level strategic intelligence with hands-on technical architecture. Coronado presents with calm expertise and transparent humility, underscored by explicit disclaimers separating his personal open-source projects from official NATO policy. The narrative moves cleanly from historical methodology evolution to practical code-level harness design and real-world predictions, instilling a grounded sense of security-conscious innovation.

## Key takeaways - [00:04](https://www.youtube.com/watch?v=38lYJQ502pA&t=4s) Sergio Coronado presents personal work on supervised agentic workflows for threat forecasting, emphasizing that his views do not represent official NATO policy or activities. - [00:04](https://www.youtube.com/watch?v=38lYJQ502pA&t=4s) The ultimate goal of using AI agents in threat research is not to replace human judgment, but to compress time while maintaining strict human ownership over scope and final decisions. - [02:35](https://www.youtube.com/watch?v=38lYJQ502pA&t=155s) Coronado's threat forecasting methodology evolved from a manual PESTLE framework prior to 2021 into an 'always-on' continuous scanning engine running 24/7 on a home server in Luxembourg. - [05:43](https://www.youtube.com/watch?v=38lYJQ502pA&t=343s) The continuous scanning architecture uses specialized roles, including Searchers, Extractors, Synthesizers, a dedicated Skeptic to challenge assumptions, and Evaluators. - [06:47](https://www.youtube.com/watch?v=38lYJQ502pA&t=407s) A governance harness enforces strict operational boundaries, tool permissions, safety red lines, and human checkpoints to manage agent execution. - [06:47](https://www.youtube.com/watch?v=38lYJQ502pA&t=407s) Coronado establishes the core operational rule 'No traces, no trust,' requiring full logging of prompts, tool calls, costs, and reasoning before outputs can be trusted. - [11:27](https://www.youtube.com/watch?v=38lYJQ502pA&t=687s) The system follows a privacy routing model with an approximate 80% local to 20% cloud split to manage sensitive threat intelligence while leveraging cloud reasoning when necessary. - [11:27](https://www.youtube.com/watch?v=38lYJQ502pA&t=687s) Data is categorized into color-coded classification tiers (Green for public data, Amber for sensitive/local processing, and Red for isolated secrets/credentials). - [13:35](https://www.youtube.com/watch?v=38lYJQ502pA&t=815s) A case study on Linux kernel vulnerability forecasting demonstrates predicting 30- to 90-day threat escalation trends while strictly forbidding agents from retrieving or generating exploit code. - [15:26](https://www.youtube.com/watch?v=38lYJQ502pA&t=926s) The agent setup successfully forecast targeted attacks on AI agent infrastructure and mass perimeter edge device exploitation ahead of real-world incidents. - [17:42](https://www.youtube.com/watch?v=38lYJQ502pA&t=1062s) Implementation for home practitioners should follow a four-step path: start with a single workflow, separate agent roles, mandate trace logging, and enforce clear harness red lines. - [17:42](https://www.youtube.com/watch?v=38lYJQ502pA&t=1062s) The presentation closes with the philosophy: 'Put your agents in the basement and keep the job upstairs,' emphasizing human responsibility over automated execution.

## Detailed walkthrough ### [00:04](https://www.youtube.com/watch?v=38lYJQ502pA&t=4s) Introduction and Personal Scope

Sergio Coronado introduces his talk on using supervised agentic workflows to conduct sensitive research, horizon scanning, and forecasting for future cybersecurity threats. He opens with an explicit personal disclaimer: the views, techniques, and projects discussed are strictly his own personal work and do not represent NATO policy, official NATO activities, or his formal responsibilities. He frames his background across three distinct roles—serving as the Chief Information Officer (CIO) at the NATO Support and Procurement Agency (NSPA), running a non-profit organization, and teaching advanced project management as a university professor.

Coronado outlines the session's five main themes: setting boundaries on what AI agents should and should not do, tracing his personal evolution in threat forecasting, breaking down his system architecture, walking through a concrete Linux kernel vulnerability case study, and providing actionable guidance for building a similar system at home. He emphasizes that while AI agents compress the distance between asking questions, gathering evidence, synthesizing findings, and executing decisions, the ultimate goal is not human replacement. Humans maintain total ownership over scope, verification, implementation, and final judgment.

- **Talk focus:** Supervised agentic workflows for threat forecasting and cybersecurity horizon scanning. - **Personal disclaimer:** Personal capacity only; no tie to official NATO policy or activities. - **Presenter background:** CIO at NATO Support and Procurement Agency (NSPA), non-profit founder, and university professor. - **Core philosophy:** "The question is no longer 'Can AI answer the question?' It is 'What is it allowed to do after it answers?'"

---

[02:35](https://www.youtube.com/watch?v=38lYJQ502pA&t=155s) Evolution of a Forecasting Methodology (2021–Present)

Coronado traces his personal approach to threat forecasting across three distinct eras. Prior to 2021, he relied on a structured, highly manual Thread Horizon framework to analyze how critical sectors—such as energy, banking, and healthcare—were exposed to emerging threats. This early method evaluated data across six PESTLE lenses (political, economic, social, technological, legal, and environmental) and drew from open-source material, academic literature, vendor reports, and incident data. Promising signals were vetted through hands-on proofs of concept (PoCs) to test feasibility before establishing 2- to 5-year technology roadmaps.

Between 2021 and 2025, Coronado shifted toward field intelligence and LLM-assisted synthesis. Instead of relying solely on published papers, he conducted direct interviews with critical infrastructure operators, attended vetted industry conferences, and leveraged a trusted network of security researchers. Sensitive intelligence gathered from these interactions was processed using an air-gapped, fine-tuned LLM, moving his workflow from "reading more" to "listening more" while maintaining strict data isolation.

Today, Coronado operates an "always on" continuous scanning engine. While he presents at conferences or travels, AI agents continuously work in his basement server setup back home in Luxembourg. The modern pipeline uses dedicated agents for web search, claim and signal extraction, intelligence synthesis, skeptic-based adversarial challenging, and formal quality/uncertainty evaluation—all terminating at a mandatory human gating step before taking action.

``` [ 2021 & Prior ] [ 2021 - 2025 ] [ Today ] +------------------+ +------------------+ +------------------+ | Manual Framework | | Field Intel & | | Continuous | | PESTLE Lenses | --> | Air-Gapped LLMs | --> | Agent Scanning | | PoC Feasibility | | Trusted Networks | | 24/7 Basement | +------------------+ +------------------+ +------------------+ ```

- **Pre-2021 critical sectors:** Energy, banking, and healthcare evaluated via PESTLE dimensions. - **2021–2025 environment:** Air-gapped fine-tuned LLM used to synthesize sensitive field intelligence and interview notes. - **Present-day location:** Basement home setup in Luxembourg running 24/7 background agent workflows.

---

[05:43](https://www.youtube.com/watch?v=38lYJQ502pA&t=343s) Core Agent Architecture and System Roles

The scanning architecture relies on a clear division of specialized agent roles working in concert to process intelligence. Searcher agents scour the web and security advisories for emerging signals, while Extractor agents parse incoming media—ranging from technical text and CVE bulletins to video transcripts and PoC repositories—into structured claims. Synthesizer agents aggregate these extracted data points, looking for weak or early indicators of widespread vulnerability trends.

Crucially, the system incorporates a dedicated Skeptic agent (or committee of skeptic agents) whose sole purpose is to challenge the gathered findings. The skeptic actively searches for missing evidence, factual inaccuracies, logical contradictions, and exaggerated severity claims. An Evaluator stage then weighs the raw findings against the skeptic's counterarguments, scoring uncertainty and data validity. Only after passing these automated checks does the output reach the human supervisor for final judgment and decision-making.

- **Searchers:** Discover raw security advisories, vulnerability disclosures, and web signals. - **Extractors:** Convert unstructured media (videos, text, PoCs) into standardized claims and data points. - **Synthesizers:** Collate signals to spot early macro trends across sectors. - **Skeptics:** Play advocate, searching for errors, missing context, and contradictions. - **Evaluators & Human Supervisors:** Assess confidence scores and gate final operational decisions.

---

[06:47](https://www.youtube.com/watch?v=38lYJQ502pA&t=407s) Control Plane, Harness Design, and the Trust Rule

The technical execution stack consists of three key layers: public data sources/APIs, model runners, and a strict harness control plane. For runners, Coronado highlights open-source orchestration tools like OpenClaw's Lobster engine, which manages persistent workflows, web searching, API calls, files, and Model Context Protocol (MCP) tool connections. Models run locally first to preserve data privacy, bridging out to public cloud models only when specialized capabilities are needed.

The harness acts as the governance plane, enforcing strict operational contracts, system instructions, tool permissions, token budgets, and evaluation metrics. It continuously monitors execution traces against predefined safety "red lines." If an agent violates a constraint or encounters high uncertainty, a safety trigger immediately halts execution and routes the task to a human checkpoint. Coronado underlines this architecture with a core operational rule: "No traces, no trust." Without complete logging of prompts, tool calls, token costs, and agent reasoning, an LLM's output cannot be trusted or promoted to consequential use.

``` +--------------------------------------------------------------------+ | HARNESS & GOVERNANCE | | - Scope Contracts - Red Line Rules - Human Checkpoints| | - Routing Policies - Token Budgets - Audit Tracing | +--------------------------------------------------------------------+ | v +--------------------------------------------------------------------+ | AGENT RUNNERS | | (e.g., OpenClaw Lobster Engine, MCP Servers, Local/Cloud Models) | +--------------------------------------------------------------------+ ```

- **Open-source orchestration:** Uses tools such as OpenClaw Lobster for persistent workflow execution. - **Harness parameters:** Controls token usage, model routing, safety red lines, and tool permissions. - **Log requirements:** Full tracking of prompts, actions, source claims, costs, and execution failures. - **Golden rule:** "No traces, no trust. If you don't have enough information about what your agents were doing, you cannot trust the output."

---

[11:27](https://www.youtube.com/watch?v=38lYJQ502pA&t=687s) Privacy Routing and Data Classification

Because the system processes sensitive threat intelligence, Coronado uses a strict hybrid execution model designed around privacy and cost control. High-volume, computationally intensive tasks—such as initial web scraping, document summarization, entity extraction, vector embedding, and clustering—are executed entirely on local hardware. Cloud-based models are leveraged selectively (following an approximate 80% local / 20% cloud split) when complex multi-step reasoning, advanced logic, or model diversity is required.

To prevent data leakage, the workflow relies on explicit data labeling akin to military classification frameworks. Information is categorized into three strict color-coded tiers before agents interact with it:

``` [ GREEN ] --> Public Data (Already online, safe for cloud LLMs) [ AMBER ] --> Sensitive Data (Restricted sharing, local processing) [ RED ] --> Secrets & Credentials (Strictly isolated, never sent to APIs) ```

- **Execution split:** Approximately 80% local processing to 20% cloud model requests. - **Green data:** Publicly available advisories and open web data; safe for cloud models. - **Amber data:** Sensitive or restricted intelligence shared under trust networks; limited to local processing. - **Red data:** Hard secrets, credentials, and sensitive private keys; completely forbidden from external transmission.

---

[13:35](https://www.youtube.com/watch?v=38lYJQ502pA&t=815s) Case Study: Linux Kernel Vulnerability Forecasting

To demonstrate the agent framework in practice, Coronado walks through a practical threat scanning scenario focused on predicting Linux kernel vulnerability trends across major enterprise distributions. The goal is to forecast whether specific kernel bugs are likely to escalate into widespread exploits within a 30- to 90-day window, giving security teams lead time to prioritize patching.

Before unleashing the agents, explicit red lines are programmed into the harness: agents may only pull from official, publicly available vendor advisories, and they are strictly forbidden from searching for or generating functional exploit payloads or proof-of-concept code. The pipeline then executes sequentially across five distinct stages:

``` +-----------+ +-----------+ +--------------+ +----------+ +------------+ | Searcher | --> | Triage & | --> | Synthesizer | --> | Skeptic | --> | Human Gate | | (Data) | | Executor | | (30-90d Fcst)| | (Dev) | | (Briefing) | +-----------+ +-----------+ +--------------+ +----------+ +------------+ ```

- **Objective:** Predict Linux kernel vulnerability trends 30 to 90 days out. - **Red Line Constraints:** Use official public advisories only; do not collect or generate functional exploit code. - **Pipeline Flow:** Searcher gathers advisories -> Executor performs triage -> Synthesizer builds forecast -> Skeptic challenges assumptions -> Human approves for daily/weekly briefing.

---

[15:26](https://www.youtube.com/watch?v=38lYJQ502pA&t=926s) Real-World Predictions and Evaluation Results

Coronado presents two real-world threat forecasts generated by his basement setup over the preceding two months, demonstrating the system's accuracy when validated against actual industry incidents:

1. **AI & Agent Infrastructure Attack Surface:** The system detected early signals showing targeted attacks against Google APIs, Vercel deployments, and lightweight open-source agent libraries. The agents synthesized a forecast predicting that agentic infrastructure would become a primary target. The Skeptic challenged the claim, arguing the attack scale was isolated. However, within 30 days, widespread real-world attacks against AI management tools verified the forecast. 2. **Enterprise Perimeter Exploitation:** Agents flagged high-severity vulnerabilities (CVSS scores up to 10.0) being actively exploited across edge devices, including Fortinet VPN routers and Joomla web deployments. The agents forecast imminent mass exploitation of edge perimeter infrastructure. Despite the Skeptic arguing that rapid patch deployment would mitigate the threat, the human-approved forecast proved accurate: by July 17, nearly 90,000 devices were compromised globally in the "FortiBleed" exploitation wave.

- **Forecast 1 (AI Infrastructure):** Targeted attacks on Google APIs and open-source agent components hit within 30 days. - **Forecast 2 (Perimeter Edge):** Mass exploitation of Fortinet and Joomla edge devices predicted; verified by 90,000 FortiBleed compromises by mid-July. - **Maximum severity tracking:** System actively monitors CVSS 10.0 vulnerabilities exploited in the wild.

---

[17:42](https://www.youtube.com/watch?v=38lYJQ502pA&t=1062s) DIY Implementation Steps and Conclusion

Coronado concludes with a four-step roadmap for practitioners looking to build supervised agent workflows on local hardware:

1. **Start with a single workflow:** Avoid overly complex setups initially. Focus on one simple, well-defined task like parsing daily security advisories or triaging incident reports. 2. **Enforce separation of agent roles:** Build dedicated agents for searching, triaging, synthesizing, and challenging rather than relying on a single monolithic prompt. 3. **Mandate full trace logging:** Ensure every source URL, intermediate extraction step, token cost, and tool output is recorded. Remember: no traces, no trust. 4. **Establish clear harness red lines:** Define explicit permissions, budget limits, and mandatory human checkpoints where the system must pause for approval.

He closes the presentation with a final guiding maxim: "Put your agents in the basement and keep the job upstairs"—retaining full human accountability while delegating heavy execution to automated machines.

- **Step 1:** Pick one simple workflow (e.g., security report triage). - **Step 2:** Separate roles (Searcher, Extractor, Synthesizer, Skeptic, Evaluator). - **Step 3:** Require complete execution traces for auditability. - **Step 4:** Define red lines and mandatory human checkpoints. - **Closing motto:** "Put your agents in the basement and keep the job upstairs."

## Notable quotes > [00:04](https://www.youtube.com/watch?v=38lYJQ502pA&t=4s) The question is no longer 'Can AI answer the question?' It is 'What is it allowed to do after it answers?' — Sergio Coronado

> [06:47](https://www.youtube.com/watch?v=38lYJQ502pA&t=407s) No traces, no trust. If you don't have enough information about what your agents were doing, you cannot trust the output. — Sergio Coronado

> [17:42](https://www.youtube.com/watch?v=38lYJQ502pA&t=1062s) Put your agents in the basement and keep the job upstairs — Sergio Coronado

## Mentioned Sergio Coronado, NATO Support and Procurement Agency, NSPA, NATO, OpenClaw, Lobster engine, Model Context Protocol, MCP, Linux, Google, Vercel, Fortinet, Joomla, FortiBleed

## Video description Sergio Coronado - CIO, NATO Support and Procurement Agency

Anyone with a computer can put AI agents to work. This talk shares real experience building supervised agentic workflows for sensitive research and forecasting using open-source tools, local models, and public sources. Covers architecture, privacy routing, harness design, evaluation loops, and keeping human judgement visible when machines do the heavy lifting.

[Source](https://www.youtube.com/watch?v=38lYJQ502pA)

中文简报

北约官员部署地下室AI智能体进行威胁预测 自主威胁扫描需要严格的人工监督与持续的追踪日志记录，以将自动化情报转化为可信预测。 Sergio Coronado 详细介绍了其利用卢森堡地下室本地服务器运行的专业 AI 智能体团队开展 7×24 小时网络安全威胁预测的方法。该多智能体架构在执行数据隐私路由与安全“红线”的严格治理框架下，分工设置了搜索者（Searcher）、提取者（Extractor）、综合者（Synthesizer）、怀疑者（Skeptic）和评估者（Evaluator）等角色。结合预测 AI 基础设施攻击和边界突破的案例研究，Coronado 展示了自动化智能体工作流如何在保持绝对人工监督的同时加速研究。最后，他为从业人员总结了一份构建受监督本地智能体系统的四步实用指南。 Sergio Coronado（以个人身份分享，不代表北约政策）介绍了用于网络安全威胁预测的受监督 Agent 工作流，其核心理念并非取代人类判断，而是在保持严格人类所有权的同时大幅压缩研究与决策时间。 (0:04) 该预测架构从早期的手动 PESTLE 框架演变为在卢森堡家庭服务器上 7x24 小时运行的连续扫描引擎，通过搜索器、提取器、合成器、专属质疑者（Skeptic）与评估器协作运行，并在采取行动前设置强制性人类把关点。 (2:35) 系统通过治理 Harness 执行工具权限、安全红线与“无追踪，不信任（No traces, no trust）”规则，必须完整记录 Prompt、工具调用与推理全流程才可信任其输出。 (6:47) 数据路由采用约 80% 本地与 20% 云端的隐私架构，并划分为绿色（公开数据）、黄色（受限敏感数据/仅限本地处理）和红色（严禁外发的凭据密钥）三级分类标准。 (11:27) 在 Linux 内核漏洞预测等实测中，该系统曾成功提前预警 AI 基础设施攻击及涉及 Fortinet 和 Joomla 的企业边缘设备大规模入侵（如预警后引发的近 9 万台设备受害事件）。 (15:26)

Reading summary

Sergio Coronado, speaking in a personal capacity while serving as CIO at the NATO Support and Procurement Agency, details a framework for continuous cybersecurity threat forecasting using supervised agentic workflows. He argues that AI agents can compress research and data synthesis cycles, but must remain bound by strict governance harnesses, complete audit traces, and mandatory human authorization before decisions take effect. Role Specialization and Adversarial Verification (5:43) Coronado structures his continuous scanning engine around specialized agents rather than a single monolithic prompt. Searcher agents identify incoming vulnerability notices and web signals, Extractor agents standardize unstructured text, video transcripts, and code repositories into structured claims, and Synthesizer agents aggregate signals to detect emerging macro trends. To counter hallucinations and overconfidence, the pipeline integrates a dedicated Skeptic agent—or committee of skeptics—tasked with challenging conclusions, finding missing evidence, and disputing exaggerated severity ratings. An Evaluator stage subsequently weighs the synthesized findings against the skeptic's counterarguments to score overall data validity and uncertainty. Final outputs do not trigger autonomous actions; instead, they terminate at a mandatory human checkpoint where operators exercise final judgment over scope, verification, and implementation. Coronado emphasizes that while agents operate continuously in the background, human supervisors retain ultimate operational accountability. Control Planes, Safety Red Lines, and Auditing (6:47) To govern model behavior, the execution architecture pairs runner engines like OpenClaw's Lobster with a strict harness control plane. The harness enforces operational contracts, tool permissions, Model Context Protocol connections, token budgets, and system instructions. It continuously monitors execution against predefined safety "red lines." For instance, during vulnerability research, agents are strictly forbidden from searching for or generating functional exploit payloads. If an agent breaches a constraint or encounters high uncertainty, the harness immediately halts execution and alerts a human reviewer. Underpinning this control plane is Coronado's rule: "No traces, no trust." The harness mandates complete logging of agent reasoning steps, source URLs, tool outputs, and token costs. Coronado argues that without an exhaustive audit trail demonstrating what models did and where they sourced evidence, outputs cannot be verified or relied upon for consequential operational decisions. Tiered Classification and Hybrid Execution (11:27) Because threat forecasting involves sensitive intelligence, the system employs a hybrid deployment model structured around privacy and cost control. Coronado routes roughly 80% of tasks to local hardware in his Luxembourg basement setup, reserving the remaining 20% for cloud models when advanced multi-step logic or diverse perspectives are required. High-volume workloads—such as scraping, document summarization, entity extraction, vector embedding, and clustering—remain entirely local. To prevent accidental data exposure, intelligence is categorized before processing using a three-tier classification framework. Publicly available advisories and web pages are labeled Green and cleared for cloud processing. Sensitive material received from trusted industry networks or interviews is marked Amber and restricted to local hardware. Hard secrets, credentials, and private keys are labeled Red and strictly prohibited from external transmission. Empirical Threat Forecasting and Incident Validation (15:26) Coronado illustrates his pipeline with a Linux kernel vulnerability scanning workflow designed to forecast whether emerging bugs will escalate into widespread exploits within a 30- to 90-day window. Beyond simulated workflows, he cites two real-world forecasts generated by his basement setup that were later corroborated by major industry incidents. In one instance, the system detected early attacks targeting Google APIs, Vercel deployments, and lightweight agent libraries, predicting that agentic deployment tools would soon face broader compromise. Despite the skeptic agent arguing the initial attack scale was isolated, real-world attacks verified the forecast within 30 days. In another scenario, the pipeline tracked vulnerabilities with CVSS scores up to 10.0 across Fortinet routers and Joomla systems, forecasting imminent perimeter exploitation. Although the skeptic agent argued that enterprise patching would mitigate risk, approximately 90,000 devices were subsequently compromised in the mid-July "FortiBleed" campaign. Coronado advises practitioners to begin on local hardware with a single workflow, clear agent role separation, full trace logging, and strict human checkpoints. However, because his methodology relies on personal infrastructure and trusted researcher networks, questions remain regarding how easily such air-gapped, hybrid agent stacks can scale across standard enterprise environments without compromising governance or data isolation.

精炼阅读摘要

北约支持与采购局（NSPA）首席信息官 Sergio Coronado 以个人身份展示了一套受监督的智能体工作流架构，用于前瞻性网络威胁情报扫描与风险预测；他强调技术的核心不在于替代人类，而在于通过严格的操作边界与人工把关将执行权与决策权有效分离。 个人实践范式演进与地下室自动化 (2:35) Coronado 将其威胁预测方法划分为三个阶段：2021 年之前，他依赖高人工投入的 Thread Horizon 框架，针对能源、银行及医疗等关键基础设施，借助 PESTLE 模型的六个维度结合概念验证（PoC）制定 2 至 5 年的技术路线图；2021 年至 2025 年间，他转向现场情报搜集，通过直接访谈基础设施运营方和安全研究人员，并将敏感信息交由物理隔离（air-gapped）且经过微调的 LLM 处理，实现从“多读”向“多听”的转变。目前，他在卢森堡的家中地下室部署了全天候不间断运行的自动化扫描引擎，利用部署在本地硬件上的智能体集群全天候执行数据检索与分析，所有工作流最终均需经由人工把关才能进入实质操作。 智能体分工机制与对抗性质疑设计 (5:43) 这套扫描架构依赖多智能体的细致分工来协同处理威胁情报。系统由负责发现公开安全公告与网络信号的 Searcher 智能体发起，Extractor 智能体随后将 CVE 公告、PoC 仓库以及技术视频脚本等非结构化媒介解析为标准化的断言与数据点；Synthesizer 智能体在此基础上汇总零散线索，识别潜在的宏观脆弱性趋势。为了防范幻觉与误判，系统引入了核心的 Skeptic（怀疑者）智能体或委员会机制，专门用于挑刺、核实证据缺漏、排查逻辑矛盾并压制夸大的严重性评级；随后 Evaluator 模块会对原始结论与质疑论据进行综合权衡并计算不确定性与置信度得分，最终产出只有在通过这些自动化校错后才会提交给人类监督者做最终裁决。 控制平面治理与严格的红线追踪 (6:47) 底层执行环境采用开源编排工具（例如 OpenClaw 的 Lobster 引擎），利用模型上下文协议（MCP）管理持久化工作流、API 交互与本地或云端模型调度。为了保证系统的可控性，治理层设计了严格的控制平面（Harness），用以配置操作合约、系统提示词、工具权限、Token 预算及评估指标；一旦智能体越过预设的“红线”或面临极高不确定性，安全触发器会即刻中断任务并交接至人工检查点。Coronado 提出了“没有追踪就没有信任”（No traces, no trust）的操作准则，要求对提示词、工具调用、Token 消耗、推导过程及错误日志进行全量记录，否则该模型输出绝不能用于任何高风险的决策场景。 数据分类分级与混合隐私路由 (11:27) 为兼顾计算成本与敏感情报的安全隔离，该系统实施了大约 80% 本地运算与 20% 云端调用的混合执行策略。网页抓取、文本摘要、实体抽取、向量嵌入与聚类等计算量大且格式化的前置任务完全由本地硬件承载，只有在面临极其复杂的多步推理或逻辑验证时才会选择性地交由公有云模型。在数据流动控制上，系统参考军事分类建立了三色分类框架：公开发布的安全公告被标记为“绿色数据”，允许安全上传至云端；来自受信任网络共享的敏感限制性情报标记为“黄色数据”，严格限制在本地计算处理；而涉及密码、凭证与敏感私钥的“红色数据”则被完全物理隔离，严禁向外部 API 传输。 实战预测案例验证与落地步骤 (13:35) 以预测 Linux 内核漏洞在 30 至 90 天内是否会演变为大规模利用的演练为例，系统在 Harness 中固化了红线限制：仅允许从官方渠道提取数据，严禁寻找或生成有效利用载荷与 PoC 代码。在近期两项真实预测验证中，系统首先捕获到针对 Google API、Vercel 部署及轻量化开源智能体库的攻击早期迹象，尽管 Skeptic 质疑其规模较小，但该预测在 30 天内被真实的 AI 工具攻击事件证实；另一次针对包含 CVSS 满分 10.0 的边缘设备漏洞（如 Fortinet VPN 与 Joomla）的分析中，系统成功预警了更大规模的边界渗透风险，7 月中旬全球因“FortiBleed”漏洞被攻破的设备接近 90,000 台。对于个人或团队构建类似系统，Coronado 建议从解析日常公告等单一任务起步，强制落实角色分离、留存全流程审计追踪，并明确系统红线与人工核准机制。 Coronado 总结认为，部署智能体的精髓在于“将智能体置于地下室，把核心决策权留在楼上”，确保自动化工具在明确的边界、细致的分工及详尽的日志留痕下运转，从而在大幅压缩情报处理链路的同时，将责任与最终判断权始终牢牢掌握在人类手中。

Detailed notes — zh

## Background 本内容由 YouTube 频道 Ai4 于 2026 年 8 月 20 日发布，主讲人为北约支持与采购局（NATO Support and Procurement Agency）首席信息官（CIO）Sergio Coronado。视频描述介绍了他如何利用开源工具、本地模型和公共数据源构建受监督的 Agent 工作流，用于敏感领域研究与威胁预测，并涵盖系统架构、隐私路由、控制框架（Harness）设计及人类研判机制。 ## TL;DR 演讲系统梳理了网络安全威胁预测从传统分析框架向 7x24 小时本地常态化 Agent 演化的历程。Coronado 提出了一套包含 Searcher、Extractor、Synthesizer、Skeptic 与 Evaluator 的多角色架构，并强调以 Harness 控制平面划定不可逾越的“红线”。系统采用 80% 本地与 20% 云端的混合路由及红绿灯数据分级机制以保障隐私安全，并在 Linux 内核漏洞、AI 基础设施攻击和企业边界网络设备失陷等实际场景中成功验证了其短期预测能力。核心原则在于机器负责委派执行，人类始终牢牢掌控最终的决策门禁与问责权。 ## Atmosphere & style 整个演讲呈现出极高专业度的技术务实风格，严谨、克制且充满工程落地的说服力。Coronado 虽具备北约高管背景，但开篇即清晰界定个人学术研究边界，语言洗练且直击痛点。他以生动的“地下室 Agent 干脏活重活，地面人类做研判把关”为隐喻，将复杂的安全治理与大模型应用拆解为清晰的工程原则，带给听众一种既冷静审慎又极具实操指导价值的硬核技术分享体验。 ## Key takeaways - [00:04](https://www.youtube.com/watch?v=38lYJQ502pA&t=4s) Sergio Coronado 强调 AI Agent 极大地缩短了从提出问题到采取行动的距离，但核心原则在于“委派执行，人类保留最终问责”，机器必须在严密监督下运行。 - [02:35](https://www.youtube.com/watch?v=38lYJQ502pA&t=155s) 态势感知与威胁预测方法历经三个演进阶段：从 2021 年前依赖 PESTEL 框架和 PoC 的静态分析，到 2021-2025 年的人际现场情报加物理隔离大模型提炼，最终发展为当前 7x24 小时本地常态化运行的 Agent 协作网络。 - [05:43](https://www.youtube.com/watch?v=38lYJQ502pA&t=343s) 高效的 Agent 架构需要明确细分专业角色，包括搜索者（Searcher）、提取者（Extractor）、合成者（Synthesizer）、寻找矛盾与反例的质疑者（Skeptic）以及评估者（Evaluator）。 - [05:43](https://www.youtube.com/watch?v=38lYJQ502pA&t=343s) 治理控制平面（Harness）是系统的中枢，负责划定不可逾越的“红线”（Red Lines）、监控权限与 Token 消耗，并强制记录完整的执行轨迹。 - [05:43](https://www.youtube.com/watch?v=38lYJQ502pA&t=343s) 信任三原则是治理体系的核心基石：无轨迹则无信任（No traces, no trust）、无量化评估则不晋级数据（No evaluations, no promotion）、无人踩刹车则不用于实质决策（No human gates, no consequential use）。 - [11:27](https://www.youtube.com/watch?v=38lYJQ502pA&t=687s) 数据路由采用“本地优先（Local-first）”架构，将 80% 的大吞吐量信息提取、向量嵌入与初审保留在本地设备，仅在需要复杂多步推理或模型多样性验证时，将 20% 的公开/脱敏数据路由至云端。 - [11:27](https://www.youtube.com/watch?v=38lYJQ502pA&t=687s) 敏感数据严格执行红绿灯分级机制（公开绿色、受限敏感黄色、机密核心红色），从数据底层标注源头防止 Agent 在调用工具与网络交互时失控外泄。 - [13:35](https://www.youtube.com/watch?v=38lYJQ502pA&t=815s) 在 Linux 内核态势研判实战中，系统设置严格红线：仅限分析公开官方安全公告，严禁抓取漏洞利用（Exploit）步骤或 PoC 代码，防止 Agent 意外执行高危操作。 - [13:35](https://www.youtube.com/watch?v=38lYJQ502pA&t=815s) 系统成功在前 30 天内准确预测了“AI 与 Agent 基础设施成为首要攻击面”，并提前预警了涉及近 9 万台边界网络设备的 FortiBleed 重大漏洞失陷浪潮。 - [17:49](https://www.youtube.com/watch?v=38lYJQ502pA&t=1069s) 给个人研究者的落地建议包括：从单一工作流起步、坚持 Agent 角色严格分离、强制留存全链路调用轨迹，以及建立清晰的控制平面红线契约。 - [17:49](https://www.youtube.com/watch?v=38lYJQ502pA&t=1069s) 核心理念总结：“把 Agent 留在地下室干重活，把决策与治理留在楼上”，人类必须对 Agent 的产出与行动承担最终的战略决策与责任问责。 ## Detailed walkthrough ### [00:04](https://www.youtube.com/watch?v=38lYJQ502pA&t=4s) 演讲背景与核心原则：监督式 Agent 工作流 北约支持与采购局（NATO Support and Procurement Agency）首席信息官（CIO）Sergio Coronado 发表演讲，分享其利用受监督的 Agent 工作流（Supervised Agents Workflow）进行敏感领域研究与网络安全未来威胁横向扫描（Horizontal Scanning）的实践经验。Coronado 首先作出免责声明：本次演讲仅代表其个人观点与研究成果，不代表北约政策或官方活动，亦与其正式职务无关。除担任北约机构 CIO 外，他还运营一家非营利机构，并在大学教授高级项目管理课程。 ```mermaid flowchart TD accTitle: 受监督 Agent 工作流的核心闭环与人机分工 subgraph S1["执行层：地下室 Agent 运作"] A["多源数据收集 (Search/Extract)"] --> B["信号提炼与合成 (Synthesize)"] B --> C["质疑与反向验证 (Skeptic)"] end subgraph S2["控制与治理层：地面人类决策"] C --> D{"质量评估与门禁 (Evaluation / Red Lines)"} D -->|触发告警或完成初步评估| E["人类最终研判与问责 (Human Gate)"] E --> F["实际行动与战略决策 (Consequential Action)"] end ``` Coronado 强调，演讲的核心绝非宣扬“AI 替代人类”或给出盲目的自动化配方，而是探讨如何有效监督机器工作。AI Agent 极大地缩短了从“提出问题”、“获取证据”、“提炼综合”到“采取行动”之间的距离。但无论技术如何演进，人类始终掌控研究范围、确定性验证、实施步骤以及最终的研判决策。“现在的关键问题不再是‘AI 能否回答这个问题’，而是‘当它给出答案后，它被允许做什么’。” - **个人多重角色**：北约支持与采购局 CIO、非营利机构负责人、大学高级项目管理教授。 - **演讲五大板块**：引言与边界界定、威胁预测方法的演进历程、系统架构与技术栈、实战用例分析、个人复现与落地指南。 - **核心理念**：Agent 负责委派执行（Delegating execution），人类保留最终责任与问责（Keeping accountability）。 --- ### [02:35](https://www.youtube.com/watch?v=38lYJQ502pA&t=155s) 预测方法的演进：从严谨框架到“地下室”常态化 Agent Coronado 梳理了自己在网络安全态势感知与威胁预测方面的三个演进阶段： ```mermaid flowchart LR accTitle: 预测方法演进的三个阶段 subgraph P1["2021 年前：框架与开源扫描"] A1["威胁视界框架 (PESTEL 维度)"] --> A2["关键行业分析 (能源/金融/医疗)"] A2 --> A3["概念验证 (PoC) 与技术路线图"] end subgraph P2["2021–2025 年：现场情报与 LLM 合成"] B1["关键行业人员深度访谈"] --> B2["受信任会议与闭门交流"] B2 --> B3["物理隔离 (Air-gapped) 微调 LLM 提炼"] end subgraph P3["当前：7x24 小时常态化 Agent"] C1["搜索与提取 (Search/Extract)"] --> C2["信号提炼与质疑 (Skeptic)"] C2 --> C3["门禁与人类研判 (Human Gates)"] end ``` 在 2021 年之前，主要依赖严谨的威胁视界框架（Threat Horizon Framework），结合政治、经济、社会、技术、法律与环境（PESTEL）等多维度视角，扫描能源、银行、医疗等关键基础设施部门的开源数据、厂商报告和学术成果，并搭建概念验证（PoC）评估技术在未来 2 至 5 年的落地可行性。 2021 至 2025 年间，方法转向“现场情报”（Field Intelligence）与大语言模型（LLM）的结合。通过与关键行业专家访谈、参与高门槛闭门会议、依托受信任的人际网络获取高敏情报，并在物理隔离（Air-gapped）的微调大模型环境中进行安全合成，实现从“海量阅读”向“倾听与大模型提炼”的转变。 如今，演化为“7×24 小时持续在线的地下室自动化扫描”。当 Coronado 远在美国演讲时，部署在卢森堡家中地下室电脑里的 Agent 正在不间断工作。系统不再追求堆砌冗长报告，而是建立一个能够自主发现弱信号、相互质疑校验、并在关键节点触发人类决策的监督引擎。 - **阶段一（2021 年前）**：严谨的威胁视界分析框架 + PESTEL 评估 + 概念验证（PoC）构建技术演进路线图。 - **阶段二（2021–2025 年）**：一线专家访谈 + 高壁垒信任网络 + 物理隔离环境下的 LLM 归纳提炼。 - **阶段三（当下）**：分布式 Agent 角色分工 + 持续弱信号探测 + 本地化全天候运行。 --- ### [05:43](https://www.youtube.com/watch?v=38lYJQ502pA&t=343s) 架构与控制平面：Agent 角色、Runner 与 Harness 治理 系统的核心在于角色细分、执行器（Runner）与治理框架（Harness）的结合。Agent 团队被划分为搜索者（Searcher）、提取者（Extractor，负责解析多模态视频、文本或 PoC 代码）、合成者（Synthesizer，挖掘弱信号与深层含义）、质疑者（Skeptic，挑战数据有效性、寻找矛盾与遗漏）以及评估者（Evaluator）。 ```mermaid flowchart TD accTitle: 监督式 Agent 架构与控制平面 subgraph Infrastructure["基础设施与运行环境 (Runner)"] R1["数据源 (公共报告/API/Advisory)"] R2["工作流引擎 (如 Lobster / 持久化状态)"] R3["混合模型支持 (本地 Ollama/vLLM + 云端 API)"] R1 --> R2 R2 --> R3 end subgraph HarnessControl["控制平面与治理机制 (Harness)"] H1["红线契约 (Red Lines / 权限控制)"] H2["预算与 Token 消耗审计"] H3["完整执行轨迹跟踪 (Traces: Prompts/Actions)"] H4{"安全触发器 / 评估循环 (Evallooops)"} H1 --> H4 H2 --> H4 H3 --> H4 end subgraph Governance["人类决策 (Human-in-the-Loop)"] G1["人类质量复核与门禁 (Human Gates)"] G2["30-90 天威胁预测输出与行动"] end Infrastructure --> HarnessControl H4 -->|违规/高不确定性/阶段完成| G1 G1 --> G2 ``` 治理平面（Harness）是整个系统的中枢神经。它负责划定 Agent 绝对不能逾越的“红线”（Red Lines），监控调用权限、设置 Token 与计算预算，并强制记录全部执行轨迹（Traces，包括 Prompts、行为、引用的源数据、主张与失败记录）。Coronado 提出了至关重要的信任原则：“无轨迹，不信任”（No traces, no trust）；“缺乏评估，就不能晋级推广数据”；“没有人类门禁，就绝不能做出具有实质后果的决策”。 - **执行器生态**：采用开源持久化工作流引擎（如 Open Cloud 的 Lobster 等）编排 Agent。 - **核心 Agent 角色分工**： - `Searcher`：负责广域威胁情报检索与初步分流（Triage）。 - `Extractor`：结构化提取非结构化文本、多模态视频及 PoC。 - `Synthesizer`：归纳聚合数据，挖掘底层关联与弱信号。 - `Skeptic`：扮演怀疑论者，专门寻找矛盾、反例与缺失证据。 - `Evaluator`：综合计算不确定性与置信度。 - **信任三原则**： - *No traces, no trust*（无轨迹记录则无信任）。 - *No evaluations, no promotion*（无量化评估则不晋级数据）。 - *No human gates, no consequential use*（无人踩刹车则不用于实质决策）。 --- ### [11:27](https://www.youtube.com/watch?v=38lYJQ502pA&t=687s) 敏感数据与混合路由：本地优先与数据分级标签 在处理高敏感网络安全情报时，数据隐私与防泄漏是核心底线。Coronado 采用“本地优先（Local-first）”架构，将 80% 的计算负载保留在家中地下室的本地设备上，仅在极少数情况下（约 20%）将脱敏任务路由至云端。 ```mermaid flowchart TD accTitle: 本地与云端混合路由及数据分级逻辑 A["外部数据与多源情报输入"] --> B{"数据分级标记 (Traffic Light)"} B -->|"🟢 绿色 (公开数据/已发布公告)"| C["允许根据需要调用云端高级模型"] B -->|"🟡 黄色 (受限分享/敏感信息)"| D["强制本地处理 (脱敏后方可评估释放)"] B -->|"🔴 红色 (绝密数据/凭据/私钥)"| E["严禁出域 (仅限本地隔离环境，触发硬红线)"] subgraph Local80["本地计算 (80% 负载)"] D --> F["海量提取与文本摘要"] E --> F F --> G["向量嵌入 (Embedding) & 聚类分析"] G --> H["第一轮核心研判与初步综合"] end subgraph Cloud20["云端计算 (20% 负载)"] C --> I["多步骤复杂深度推理"] H -.->|需模型多样性/高阶校验且合规| I end ``` Coronado 借鉴北约等敏感机构成熟的数据分类标准，建立红绿灯数据分级机制： 1. **绿色标签（Green Data）**：已在公共互联网公开的数据、官方安全通报； 2. **黄色/琥珀色标签（Amber/Sensitive Data）**：由受信任网络定向共享、受访问限制的情报； 3. **红色标签（Red Data）**：包含凭据、私钥、核心机密等绝对禁止外流的信息。 本地模型负责大吞吐量的信息抽取、摘要、向量嵌入（Embedding）与聚类分析。只有当遇到复杂的“多步骤逻辑推理（Multi-step reasoning）”、需要模型多样性验证以避免单一模型偏差时，且在确保数据级别合规的前提下，才会将特定任务分派给更强大的云端前沿模型。 - **负载配比规则**：常态下 80% 本地运算，20% 云端运算。 - **云端调用触发条件**：复杂多步推理、需要交叉验证模型多样性（Diversity of thought）、严格确认属于公开/脱敏数据。 - **数据标记红线**：若不进行底层数据打标，Agent 将会在调用工具和传输数据时失控。 --- ### [13:35](https://www.youtube.com/watch?v=38lYJQ502pA&t=815s) 实战用例与预测验证：Linux 内核与近两个月真实案例 Coronado 详细展示了一个“Linux 内核未来漏洞态势扫描”的落地用例，以及过去两个月中系统成功命中的两项真实预测： ```mermaid flowchart LR accTitle: Linux 内核漏洞态势研判流程 subgraph RedLines["设置防护红线"] RL1["仅限公共官方公告"] RL2["严禁抓取 Exploit/PoC 代码"] end subgraph AgentLoop["Agent 处理流"] S["Searcher: 收集多发行版信号"] --> E["Executor: 关联分析与归纳"] E --> P["输出 30-90 天漏洞趋势预测"] P --> SK["Skeptic: 质疑严重性与证据链"] end subgraph HumanDecision["人类决策"] SK --> H["每周/每日情报简报研判"] end RedLines -.-> AgentLoop ``` 在 Linux 内核案例中，首先设定防护红线：**仅使用公开官方通报，严禁抓取实际漏洞利用（Exploit）步骤或 PoC 代码**，防止 Agent 意外执行高危操作。搜索 Agent 从各大 Linux 发行版抓取漏洞信号，执行 Agent 提炼趋势并给出 30 至 90 天的发生概率，质疑 Agent 随后提出反向抗辩（例如核实漏洞严重性是否被夸大），最终提交人类专家进行简报研判。 近期两项真实案例验证了这套机制的有效性： 1. **AI 与 Agent 基础设施成为首要攻击面**：Agent 系统在捕捉到 Google API 及部分轻量开源组件受损的弱信号后，预测针对 AI 基础设施的攻击将在短期内激增；质疑 Agent 曾提出“影响范围可能有限”的反向观点，但经综合研判后仍确立了高风险评估。该预测在发出后 **30 天内** 得到全面应验。 2. **企业边界网络设备（Perimeter）的大规模利用**：系统监控到 Fortinet VPN 路由器、Joomla 内容管理系统框架及各类外网防火墙暴露出的高危漏洞（CVSS 评分为 10 分满分且已被野外利用）。质疑 Agent 提出“补丁正在推送、仅限罕见配置”的抗辩，但系统经评估后认定风险极高。事实证明，随后爆发的“FortiBleed”漏洞导致近 **90,000 台** 设备被攻破，预测再次命中。 - **用例红线设计示例**：仅限官方 Advisory、禁止触碰 Exploit 代码、规避潜在自主执行风险。 - **实战命中案例 1**：成功预测 AI Agent 基础设施将成为核心受攻击面（30 天内验证）。 - **实战命中案例 2**：成功预测企业边界网络设施失陷浪潮（Fortinet/Joomla 等，涉及近 9 万台设备受 FortiBleed 影响）。 --- ### [17:49](https://www.youtube.com/watch?v=38lYJQ502pA&t=1069s) 本地实践指南与总结 在演讲尾声，Coronado 为希望在个人设备上搭建监督式 Agent 系统的研究人员给出了四点具体实操建议： ```mermaid flowchart TD accTitle: 个人落地四步法 Step1["1. 从单一简单工作流起步 (如特定公告监控/报告分流)"] --> Step2["2. 明确角色分离 (搜索 / 提取 / 合成 / 质疑 / 评估)"] Step2 --> Step3["3. 强制记录运行轨迹 (Traces: 来源/提取逻辑/合成路径)"] Step3 --> Step4["4. 严密构建治理控制平面 (明确 Red Lines 与人类审查节点)"] Step4 --> EndGoal["终极目标: 将 Agent 留在地下室干重活，人类在楼上掌控决策与问责"] ``` 1. **从单一工作流起步（Start with one workflow）**：切忌一开始构建过于庞杂的系统，可从单一安全公告监控或报告自动分选（Triage）做起，避免在环境配置阶段迷失。 2. **严格进行 Agent 角色分离（Separation of roles）**：清晰划分 Searcher、Extractor、Synthesizer、Skeptic 与 Evaluator，并在产出结论前始终保留人类监督介入机制。 3. **强制全链路轨迹留存（Require traces）**：确保系统能清晰输出每条信息的抓取来源、提取路径与合成逻辑。“No traces, no trust”。 4. **构建严密的控制平面与红线契约（Build the harness & red lines）**：以书面形式明确权限、限制边界、人类检查点（Human Checkpoints）以及绝对不可通行的违规行为（No-go）。 Coronado 以一句生动形象的话总结全篇：“把你的 Agent 留在地下室干重活，把真正的决策工作留在楼上——始终由你来对 Agent 的产出承担最终责任。” - **关键建议**：从小处着手、细分 Agent 职能、留存每一步调用 Trace、设立绝不逾越的控制平面红线。 - **结语金句**：“Put your agents in the basement and keep the job upstairs. Keep yourself the accountability of your agents.” ## Notable quotes > [00:04](https://www.youtube.com/watch?v=38lYJQ502pA&t=4s) 现在的关键问题不再是‘AI 能否回答这个问题’，而是‘当它给出答案后，它被允许做什么’。 — Sergio Coronado > [00:04](https://www.youtube.com/watch?v=38lYJQ502pA&t=4s) Agent 负责委派执行（Delegating execution），人类保留最终责任与问责（Keeping accountability）。 — Sergio Coronado > [05:43](https://www.youtube.com/watch?v=38lYJQ502pA&t=343s) 无轨迹，不信任（No traces, no trust） — Sergio Coronado > [05:43](https://www.youtube.com/watch?v=38lYJQ502pA&t=343s) 缺乏评估，就不能晋级推广数据（No evaluations, no promotion） — Sergio Coronado > [05:43](https://www.youtube.com/watch?v=38lYJQ502pA&t=343s) 没有人类门禁，就绝不能做出具有实质后果的决策（No human gates, no consequential use） — Sergio Coronado > [13:35](https://www.youtube.com/watch?v=38lYJQ502pA&t=815s) 仅使用公开官方通报，严禁抓取实际漏洞利用（Exploit）步骤或 PoC 代码 — Sergio Coronado > [17:49](https://www.youtube.com/watch?v=38lYJQ502pA&t=1069s) 从单一工作流起步（Start with one workflow） — Sergio Coronado > [17:49](https://www.youtube.com/watch?v=38lYJQ502pA&t=1069s) 把你的 Agent 留在地下室干重活，把真正的决策工作留在楼上——始终由你来对 Agent 的产出承担最终责任。 — Sergio Coronado > [17:49](https://www.youtube.com/watch?v=38lYJQ502pA&t=1069s) Put your agents in the basement and keep the job upstairs. Keep yourself the accountability of your agents. — Sergio Coronado ## Mentioned Sergio Coronado, 北约支持与采购局 (NATO Support and Procurement Agency), Ai4, PESTEL, Lobster, Open Cloud, Ollama, vLLM, Linux, Google, Fortinet, Joomla, FortiBleed ## Video description Sergio Coronado - CIO, NATO Support and Procurement Agency Anyone with a computer can put AI agents to work. This talk shares real experience building supervised agentic workflows for sensitive research and forecasting using open-source tools, local models, and public sources. Covers architecture, privacy routing, harness design, evaluation loops, and keeping human judgement visible when machines do the heavy lifting. [Source](https://www.youtube.com/watch?v=38lYJQ502pA)

Source quotes

[0:04] Sergio Coronado: 现在的关键问题不再是‘AI 能否回答这个问题’，而是‘当它给出答案后，它被允许做什么’。 [0:04] Sergio Coronado: Agent 负责委派执行（Delegating execution），人类保留最终责任与问责（Keeping accountability）。 [5:43] Sergio Coronado: 无轨迹，不信任（No traces, no trust） [5:43] Sergio Coronado: 缺乏评估，就不能晋级推广数据（No evaluations, no promotion） [5:43] Sergio Coronado: 没有人类门禁，就绝不能做出具有实质后果的决策（No human gates, no consequential use） [13:35] Sergio Coronado: 仅使用公开官方通报，严禁抓取实际漏洞利用（Exploit）步骤或 PoC 代码 [17:49] Sergio Coronado: 从单一工作流起步（Start with one workflow） [17:49] Sergio Coronado: 把你的 Agent 留在地下室干重活，把真正的决策工作留在楼上——始终由你来对 Agent 的产出承担最终责任。 [17:49] Sergio Coronado: Put your agents in the basement and keep the job upstairs. Keep yourself the accountability of your agents.

Source quotes

[0:04] Sergio Coronado: The question is no longer 'Can AI answer the question?' It is 'What is it allowed to do after it answers?' [6:47] Sergio Coronado: No traces, no trust. If you don't have enough information about what your agents were doing, you cannot trust the output. [17:42] Sergio Coronado: Put your agents in the basement and keep the job upstairs

Original transcript

[0:04] I'm going to talk about supervised agents  workflow for sensitive research and forecasting. [0:10] Uh basically is work I do to predict to  forecast to to do horizontal scanning on [0:16] future threats on cyber security and I want to  share with you how how I've been doing this. [0:21] Uh before that a small disclaimer. Uh what I'm  talking here is on my personal views my personal [0:26] work. It does not represent NATO policy or NATO  activities and is not linked to my job. Just to [0:33] make that clear. Uh introduction was already  given uh different activities that I perform [0:39] um as a CIO in the N support and  procurement agency running a a nonprofit [0:44] and also a professor in the university  teaching advanced project management. [0:49] What I'm going to cover is five elements. First  an introduction and what is this talk not about. [0:56] Second, a little bit how I've been evolving  my predictions to using aentic research and [1:01] agents in general. Uh the architecture I  use a use case to give to give more reality [1:07] and some real examples of the use and last but not  least for the ones that are interesting some some [1:12] ideas on how you could do this yourself. So first  a little bit of introduction basically setting [1:19] the scene the expectations. Um what changed for me  is that I used to scan cyber security for for the [1:26] forecasting for the future more and reading and  interviews and using kind of uh synthetizing what [1:31] what data comes to my mind to my hands uh to now  today basically use uh agents that I supervise. [1:39] Uh what did not change is I own the scope I own  the certainty the verifications the implementation [1:45] steps and the final judgment. So what it what is  what it matter today is of course is the speed [1:51] agents compress the distance between questions  evidences synthesities and action. The point is [1:58] not about humans being replaced by agents is about  delegating execution and keeping accountability. [2:08] Now I'm not going to cover in this talk is about  humans being replaced by I or a recipe on how to [2:13] do it in detail that will require more time.  uh blind automation how you can do this and [2:18] and don't look on the output and and and pray uh  but I'm going to talk is about supervised machine [2:24] work the question is no longer can AI answer the  question is what is allowed to do after it answers [2:35] let me take you through my own journey  in three steps first one before 2021 [2:41] I used to do my forecasting with a framework and  of course I use plenty of open source data. Um, [2:48] so six elements I used to have a thread  horizon framework. There are severals out [2:54] there. It's a method on how you process data  to predict critical sectors. I'm talking about [3:00] cyber security. So I'm looking about, you  know, energy sectors, banking, healthcare [3:06] and how they're exposed and prioritized. I look  into different lenss. I look from the political, [3:11] economical, social, technological, legal and  environmental impact and then use a lot of [3:16] open source uh data available you know vendors  reports academic work results from an incidents [3:23] and then when that signal pop up I try to  build a proof of concept I try to see will [3:28] this technology be available in two three four  five years in the future how difficult is to do it [3:33] how feasible is to have it and then with that  produce a road map so very very rigorous framework [3:40] Now between 2021 and 2025 I changed my approach  for more field intelligence and the use of LLMs [3:48] to synthesis. So basically I conducted  interviews with the critical sector players [3:55] used thread reports participate in many vetted  conference when you have the researchers talking [4:00] more open what they found having a trust  network that share with you key insights [4:06] and then put all that together uh in an air gap  system because at this time I'm talking about very [4:11] sensitive data that has been shared with you and  you cannot share out you can you can synthesize it [4:18] and then decide and build your future assessments.  So this is done in our air gap fine-tuning LI. [4:24] Now I move from reading more to listening  more and the use of of LLMs for synthesis. [4:32] Now fast forward to today and now in  an approach of continuous scanning. [4:37] So basically it's an always on operation  where my agents are working as I speak [4:42] Zang year in the US. My agents are working  in the basement back home in Luxembourg. [4:48] I have like these six elements that  I use. I have agents doing search. [4:53] I have agents extracting looking for claims,  looking for data, looking for signals. That [4:58] that's what is interesting in forecasting logo.  Then synthetizing that information. You start [5:03] pulling a lot of data. Uh so you want this to  look for signals. What are the weak signals there? [5:09] You also need agents that challenge what  has been found. Challenge the quality. [5:14] Look for contradictions. Look for missing  data. And then you evaluate all this together. [5:19] You look for uncertainty. You look for the quality  of your finding. You look the challenges were true [5:24] or false. And then you gate elements for me the  human to really take on and take actions on that. [5:31] So the goal is not to have a bigger report  because I have my data. No, is to have this [5:35] supervised engine that while you are doing more  intelligence is producing from you the signals. [5:43] Now let me explain you a  little bit the architecture. [5:47] So it's all about agents. Okay. So I have  different roles of agents. So I have agents that [5:52] are searchers. They're looking for information.  They're looking for different aspects. [5:57] Then I have extractors. They find information but  maybe it's in a particular format. Maybe it's a [6:01] video. Maybe it's a text. Maybe it's a you know a  proof of concept and you want to capture you want [6:05] to extract information out of that and then as  you collect data you start looking for meaning. [6:12] you start synthetizing that data and then you  always challenge you know you have this skeptic [6:19] uh agent or group of agents that's going  to challenge you know what is wrong is this [6:23] this data is is is valid what is the validity of  the data and so on and then you put it together [6:29] you make your evaluations and you see how how  good they are and last but not least it come to [6:34] me to the supervisor to the human to now take a  final judgment are these signals enough to have a [6:39] prediction that something could happen in the near  future. So that is that is what is this all about. [6:47] Now how this basement look  like in terms of the stack. [6:52] So it's about the runner who runs these agents.  It's about the harness. How you make these agents [6:57] behave in a particular way and how you govern  all this infrastructure. So you need sources [7:03] public sources advisory that you can grab reports  APIs that you can pull data from. You need the [7:10] models to operate to to run not only the agents  but to operate on on on the findings on the data. [7:16] I'm talking about sensitive data. So it's  local first and then you use the cloud. [7:22] You will bridge the cloud in a particular cases.  I will explain a little bit more that later on. [7:27] You can have different runners. You can build  your own. You can bring your own angel runner. [7:30] You can use the ones that I listed there. They're  open source. I don't recommend any. Uh I use them [7:35] of course but they are there. You know they're  they're practically available for you to use it. [7:40] And then you have the harness the control  plane. What are your contracts? What are [7:44] your instructions? What are your permissions? What  the aims are allowed to do? What is your budget? [7:48] What are the token consumption that you allow?  What is the evils the valuations that we use? [7:53] And how they will decide to gate to gate  something for the human to do the governance. [7:58] And then you control the scope, you control the  interaction, you control the promotion from local [8:03] to public cloud. And you take  accountability of the output. [8:08] Now a little bit more about  the harness the control plane. [8:12] One of the key things here on on this structure  is the red lines. You want to establish red lines [8:17] when your agency is going to constrain and that is  part of the scope the red lines and then they run. [8:24] So they run within this red lines and then you  look to evidences how they have been running. [8:31] So the traces are very critical in this  control plane and then you're going to check [8:36] are these evidences that these tracers violating  red lines for example. Yes. Then you you have a [8:42] safe trigger. You know you trigger a gate  for a human to intervene and see okay what [8:47] what is happening with the particular  group of agents or the quality itself [8:51] and then you will take actions. you would recover  you you you know trigger or the human do final [8:55] review on that and then you have this in a loop  as these agents are working 24 by7. The key of [9:01] the control plane is that you record you register  your prompts your actions your sources your claims [9:07] your the cost the token utilization and the  failures. And what you see on the right is [9:12] one of the most important elements is the trust  rule. No traces no trust. If you don't have enough [9:20] information about what your agents were doing, you  cannot trust the output. If you have no evaluate, [9:26] if you don't have an evaluation done, how  you going to promote particular information? [9:31] If there are no human gates, how you  get consequential use of this data? [9:38] Now, a little bit more about the runner and the  harness. It's about from one side and you see [9:44] in the middle of this slide example of the  runners and the areas that they're working. [9:49] You have for example persistent workflows. I  mean open cloud has something called lobster [9:54] which is an engine to do your workflows. You can  orchestrate your agents with that for example. [9:59] That's the runner. You're going to give them  tools web search API skills MPC servers files. [10:08] You're going to give them access to LLMs  wherever local as I said on the sensitivity side [10:14] or whatever cloud when you require certain  scenarios to use more powerful models [10:19] you're going to keep the state you're going to  keep logs you're going to keep persistent memory [10:23] state of how this agent has been performing  and operating and then you have your safety [10:30] your approvals your elements that are being  watched and on the right side you see your harness [10:35] your contracts so you're going to define the  scope You're going to define the what the tools [10:41] are are allowed to use which tools and  to what degrees this tool can be allowed. [10:46] You're going to assign the routting policy between  local for privacy for cost control to what goes to [10:52] the cloud. You're going to have a retention how  long you retain these traces. I mean this is data [10:57] that has been generated that has to be in store  and process. And last but not least in safety [11:02] you're going to have the human gates the decisions  that human need to interact. And of course this is [11:08] from the red lines. And this is what is one of  the important elements when people start using [11:13] this these agents you know these runners that  people forget about is what are the rules what are [11:20] the harness you put around what are the red lines  and how you decide to have the human intervening. [11:27] Now I talked about this idea of running so  local and running something on the cloud. [11:33] The point here is I'm talking about sensitive  workloads. Okay. So I want to run a lot locally. [11:39] This is why I have this particular setup of open  source agents running in particular computer at [11:44] home because I want to preserve this data that has  high degrees of privacy and and need to know. So [11:53] I do the high volume processing locally. I do  the summarization, the extraction, the embedding, [11:58] the clustering, the first core of the analysis.  And then only then if I see the need to challenge [12:06] some of this content go to the more advanced  models. Also if I need multi-step reasoning on [12:12] the findings or just to have diversity not all the  models are trained the same. So you need diversity [12:18] but again you have to take a decision on that.  I have a basic rule there. I do 80% local and 20 [12:24] the cloud. But it's just an illustration that  I do more local than pushing to the cloud. [12:30] Why do I want to push it on the  cloud? is an issue of privacy. Okay, [12:33] we know that whatever you put out is  out. So the question is how you decide. [12:40] Something people forget in this environment and  this is very traditional for example in in my [12:45] organization and a sensitive environment in NATO  is that you need to label your data. You need to [12:49] classify your data. So if you don't classify  your data, how you know what is sensitive? [12:54] How you know will trigger a red line? So this  is a very simple approach. You know you have [12:59] green data which will be public is already out  there. You already pull it from the upside. [13:04] You will have sensitive data data that have  been shared with you and it's limited and should [13:09] maybe release. Okay. So you have to to label  possible and then the red one which definitely you [13:16] cannot release it. I mean for example like secrets  okay or credentials. Uh so so if you don't label [13:22] the data your agent is going to be wild with the  data because you know how they know that something [13:27] is red or amber or green. So you need to label  the data. Now let me go quickly to the use cases. [13:35] So here is a this is a very simple case. So this  is about for example looking uh vulnerabilities [13:41] future vulnerabilities or is there is a trend on  vulnerabilities in Linux kernel. So the goal is to [13:47] look into major distributions of Linux and look  whatever vulnerability signals are popping up. [13:53] The prediction is about um you know a forecasting  is is this happening more often and you will need [14:00] guard rails there your red lines. So your red line  is you're going to use only public available data [14:05] uh only official advisory and you're not going to  look for exploit steps or proof of concept because [14:11] maybe your agent is going to use that and operate  within. So you define for example this red line [14:18] and then you could put your agents you know the  searcher the one that pull the data and do a [14:24] triage the executor which is basically the one  really putting together what makes sense out of [14:30] the data you put together and then synthetizing  a report which going to give you a prediction [14:35] between 30 and 90 days based on the signals of  the data and then the skeptic will try to you know [14:42] challenge that that that assessment  that forecasting is anything missing? [14:48] Do we have enough evidences? And then you do your  final evaluation to produce a final forecasting. [14:56] And this is an example how it look like. You  will have evidences what have been observed. [15:00] You will have assumptions. You will have the  prediction that the signal shows you know a trend [15:05] to a particular area. You're going to have this  the skeptic disconfirming elements. For example, [15:11] you know, is the severity really here or should  we put it down? And then the human decision [15:15] where it goes to you to really decide in your  weekly briefing, in your daily brief, it depends [15:19] of your scope. What is your output in your  forecast? For example, in 30 and 90 days. [15:26] So now I'm going to show you real examples,  two examples uh of real data from the last two [15:32] months basically. Uh so one forecast is about  AI and agent infrastructure is now one of the [15:39] primary attack surfaces and that was  been proven by different uh signals [15:44] uh in the Google API has been compromised some of  the veron some of the light elements of some of [15:50] the open sources uh components and attackers  attackers are increasingly targeting that. [15:56] So the prediction is that we see and the agents  see this coming up more and more and more often [16:02] and then we try to to challenge them with this  the confirm try to challenge then like well [16:10] this is not necessarily what's happening the  scale is not as as we see potentially um um [16:16] you know propagating many system and then you take  that and you decide based on this analytic what [16:21] makes sense and then you assess is this happening  okay So this target was hit after 30 days. [16:27] 30 days we saw an increase on attacking basically  um um AI infrastructure. Similarly happened an [16:35] enterprise perimeter exploitation the systems that  are visible from outside of your organization. [16:40] So here are examples that that has been observed  by the agents. For example, for foret um uh VPN [16:48] routers were compromised. Some of the web content  from from Jumla is a web content framework. Um [16:55] some uh defend defense and and you know firewalls  and so on. And the assumption is the this [17:02] the parimeter becomes critical and the monitoring  of infrastructure tools and then the prediction [17:07] is by watching of these many incidents in that  particular domain also with high severity maximum [17:13] severity if you're not familiar is 10 and which  ones are being exploited actively in the wild [17:19] and then of course you have you challenge that  with the skeptic you say well patches are are are [17:24] being deployed mass use uh this is only for a  rare configuration and so on and then you take [17:29] your human decision And then you have your final  outcome and again this one was hit. We know at [17:34] that time u um in June July 17 that already close  to 90,000 devices have been compromised with what [17:42] is called uh 40 bleed. Now how you can do this  at home and I have three minutes so that's good. [17:49] So first start with one workflow. Don't start  with too many things because then you get lost [17:54] just in the setup. Okay. Decide what is  this single workflow. Could be an advisory [17:59] or monitoring a particular domain. Uh could be  report triage. Keep it simple. Do the separation [18:05] of role of agents. You know have agents that  search agents that triage instructor synities [18:10] a skeptic uh evaluator and then push push for  the human action and human supervision before [18:15] you conclude. Number three require traces. You  know prepare your agents to produce output on the [18:21] traces where the source of information has been  taken, how a particular extraction has been done, [18:27] how the synthesis has been produced, no  traces, no trust. Um build the the harness, [18:34] okay, the contracts, the pron, the red  lines and write these red lines clearly, [18:39] you know, the permissions, the restriction,  the human checkpoints and what is a no-go. [18:43] And with this I want to conclude. Put your agents  in the basement and keep the job upstairs. So keep [18:53] keep yourself the accountability of your agents.  And with this I want to thank you. Good wish list.