Meta Replaces Network War Rooms With Hierarchical AI Agents

@Scale

Meta is shifting runtime network operations from 100-engineer manual war rooms to hierarchical AI agent networks that automate incident response and eliminate costly chatter loops.

AI-assisted summary of Lightning Talk: AI-Native Network Operation | YuLing Chen, Meta — original source. Check the original for full context.

Source published: 2026-09-13

Detailed notes

## Background Published by @Scale on YouTube on August 27, 2026, this presentation features Meta Network Production Engineer YuLing Chen introducing Meta's AI4NI @Runtime initiative. According to the video description, the talk covers solution architectures, agentic systems, and production trial findings for running large-scale network infrastructure reliably, performantly, and efficiently using AI agents.

## TL;DR Meta is shifting its runtime network operations from high-stress human 'war rooms' to a hierarchical network of hundreds of thousands of AI agents. To prevent infinite agent chatter and wasted compute, Meta rejects peer-to-peer mesh agent setups in favor of a centralized orchestrator guiding domain experts across data center, backbone, and edge networks. Beyond incident response and root cause analysis via its NetIngram knowledge base, Meta leverages AI agents to optimize routing, adaptive video bitrates, and cross-datacenter GPU cluster load balancing. Critical challenges like non-deterministic LLM behavior and novel incidents are managed using an 'LLM-as-a-Judge' evaluation model and safety guardrails embedded directly into Infrastructure-as-Code pipelines.

## Atmosphere & style The presentation carries an authoritative, pragmatic, and engineering-focused tone, grounded in the high-stakes reality of managing network infrastructure for billions of users. YuLing Chen contrasts the intense stress of multi-hour Sev-1 outages and 100-person war rooms with the structured, methodical promise of AI-native architecture. The narrative builds clear momentum, moving seamlessly from past operational pain points to concrete architectural solutions and real-world production wins.

## Key takeaways - [00:05](https://www.youtube.com/watch?v=WGBLZ4xmWp0&t=5s) Operating Meta's global network leaves zero room for error, as minor engineering or configuration mistakes can trigger multi-hour outages affecting billions of users and causing severe revenue and business losses. - [02:09](https://www.youtube.com/watch?v=WGBLZ4xmWp0&t=129s) Current runtime network operation relies heavily on manual "war rooms" where over 100 engineers act as firefighters to sift through logs and debate root causes under extreme pressure. - [02:09](https://www.youtube.com/watch?v=WGBLZ4xmWp0&t=129s) Meta is transitioning to an AI-native network operation (AI4NI) vision utilizing hundreds of thousands of coordinated AI agents spanning domain experts and individual AI engineers. - [04:37](https://www.youtube.com/watch?v=WGBLZ4xmWp0&t=277s) Early production deployments show that AI workflows match or exceed human performance, detecting severe incidents missed by legacy monitoring tools. - [04:37](https://www.youtube.com/watch?v=WGBLZ4xmWp0&t=277s) Meta is building NetIngram, a centralized network knowledge base and memory system that enables AI agents to continuously self-improve. - [05:47](https://www.youtube.com/watch?v=WGBLZ4xmWp0&t=347s) Meta strongly advises against peer-to-peer mesh agent architectures due to infinite conversation loops, lack of a tiebreaker, and massive wasted GPU compute. - [05:47](https://www.youtube.com/watch?v=WGBLZ4xmWp0&t=347s) Meta standardizes on a hierarchical agent architecture governed by a central AI Cross-Domain Orchestrator directing Data Center, Backbone, and Edge expert agents. - [07:34](https://www.youtube.com/watch?v=WGBLZ4xmWp0&t=454s) Network performance optimization uses a dual strategy: Defense for detecting regressions and Offense for actively finding enhancement opportunities. - [07:34](https://www.youtube.com/watch?v=WGBLZ4xmWp0&t=454s) Proactive AI agents optimize network efficiency by analyzing submarine vs. terrestrial routing, regional adaptive video bitrates, and cross-datacenter GPU cluster loads. - [09:08](https://www.youtube.com/watch?v=WGBLZ4xmWp0&t=548s) To counter AI hallucinations and non-deterministic outputs, Meta employs an "LLM-as-a-Judge" framework that evaluates multiple parallel agent runs against a strict rubric. - [09:08](https://www.youtube.com/watch?v=WGBLZ4xmWp0&t=548s) Novel zero-day incidents are addressed through real-time human feedback loops that dynamically update the agent system's operational memory. - [09:08](https://www.youtube.com/watch?v=WGBLZ4xmWp0&t=548s) Safety and security risks are mitigated by embedding native AI guardrails directly into network data models and Infrastructure-as-Code pipelines.

## Detailed walkthrough ### [00:05](https://www.youtube.com/watch?v=WGBLZ4xmWp0&t=5s) Meta's Network at Scale and the High Stakes of Runtime Incidents

While previous discussions focused on planning, design, and delivery, network operation at runtime presents unique and massive operational challenges. YuLing Chen, a Network Production Engineer at Meta with eight years of experience in network infrastructure, emphasizes that running a network supporting billions of users leaves zero room for error. When reliability incidents occur at this scale, the potential blast radius is massive.

Chen shares several concrete past incidents to illustrate how seemingly minor engineering changes can cascade into severe, multi-hour outages that impact revenue, degrade business metrics, and trigger widespread user frustration across platforms like Instagram: * An engineer pushed a bad change that disrupted data center drop scheduling, causing critical network services to become unavailable and rendering external user-facing websites unreachable. * A network configuration change caused prefix overload and a BGP (Border Gateway Protocol) crash, which ultimately dropped active user traffic. * A severe Sev-1 incident was triggered by regional data center overload, causing cascading failures and site outages that lasted for multiple hours.

Beyond ensuring around-the-clock reliability, runtime network operations must deliver content fast—quantifying latency spikes directly against degraded user experiences—while simultaneously driving down infrastructure costs.

- **Primary Goal:** Enhance network reliability 24/7 while optimizing speed (latency) and infrastructure efficiency. - **Business Impact of Outages:** Meta site down, severe user frustration, operational disruption, and direct revenue loss.

---

[02:09](https://www.youtube.com/watch?v=WGBLZ4xmWp0&t=129s) The "War Room" Reality vs. Autonomous Agentic Systems

Today, network runtime operations remain overwhelmingly manual. When a catastrophic incident occurs—metaphorically described as a "rainy day" where the entire Meta website goes down—Meta's standard response is assembling a high-pressure "war room." Over 100 engineers from various organizations assemble as "firefighters" to manually sift through massive datasets, debate potential root causes over chat, and push mitigations under immense pressure. Chen characterizes this legacy process as incredibly tedious, stressful, and inherently risky for the business.

To transform this, Meta is developing an agentic system composed of specialized AI agents designed to collaborate across the entire incident lifecycle. These agents understand network topology and mirror human engineering expertise across data center, backbone, and edge networks.

- **Human War Room Scale:** 100+ engineers manually parsing logs and datasets under high stress. - **Agent Architecture Vision:** Hundreds of thousands of coordinated AI agents operating simultaneously. - **Agent Taxonomy:** - **Domain Experts:** Maintain broad, cross-domain architectural knowledge. - **Individual AI Engineers:** Specialize in solving targeted, domain-specific technical issues. - **Lifecycle Scope:** Spans detection, root cause analysis (RCA), mitigation, repair, and prevention. - **Autonomy Roadmap:** A gradual multi-year evolution moving from L1/L2 assistive tools up to L3/L4 fully autonomous operations.

---

[04:37](https://www.youtube.com/watch?v=WGBLZ4xmWp0&t=277s) Early Production Wins and Knowledge Infrastructure

Meta is already seeing promising reliability gains from early AI-driven workflows deployed across its data center, backbone, and edge networks. Rather than simply automating manual scripts, these agentic workflows match or exceed human performance in critical scenarios.

Specific early wins highlight the power of these autonomous systems in production: * An internal AI workflow successfully detected severe production incidents that standard, legacy monitoring systems completely failed to discover. * Config/Capacity AI agents scan massive datasets and Meta's internal knowledge base to evaluate egress capacity risks using real-time, map-based visual interfaces. * Meta is building **NetIngram**, a centralized network knowledge base that enables AI agents to continuously self-improve across the network infrastructure organization.

- **Core Capabilities:** Automated incident detection, root cause analysis, and proactive risk evaluation. - **Key System:** **NetIngram** serves as the underlying organizational memory and knowledge graph for continuous agent learning.

---

[05:47](https://www.youtube.com/watch?v=WGBLZ4xmWp0&t=347s) Architecture Principles: Why Mesh Fails and Hierarchy Wins

When designing an agentic architecture, a common debate is whether to build a peer-to-peer mesh agent network or a hierarchical structure. Chen strongly warns against mesh architectures for network operations: "In a mesh network, there's a lack of central decision maker or tiebreaker. To our experience, agents can be very talkative." In practice, mesh agents loop infinitely in conversation, leading to unclear conclusions and massive wasted GPU compute resources.

Instead, Meta has standardized on a **hierarchical agent architecture** governed by parent orchestrators, which has proven effective across nearly all successful production projects.

``` [ AI Cross-Domain Orchestrator ] (Central Brain) │ ┌──────────────────┼──────────────────┐ ▼ ▼ ▼ [Data Center Expert] [Backbone Expert] [Edge Expert] │ │ │ └─[Sub-Agents] └─[Sub-Agents] └─[Sub-Agents] ```

- **Workflow Execution:** Continuous monitoring feeds network KPIs and alerts to the central AI Cross-Domain Orchestrator. The orchestrator dispatches tasks to domain experts (Data Center, Backbone, Edge), which deploy sub-agents to investigate specific telemetry. The orchestrator aggregates the sub-agent evidence, forms a unified conclusion, and generates a clean RCA report for human engineers (with future capability for automated safeguard-protected repair execution). - **GPU Resource Optimization:** Hierarchical controls eliminate agent-to-agent chatter loops and focus compute on actionable diagnostics.

---

[07:34](https://www.youtube.com/watch?v=WGBLZ4xmWp0&t=454s) Optimizing Performance and Driving Infrastructure Efficiency

Beyond reliability, Meta uses hierarchical AI agent networks across two distinct performance vectors: **Defense** (regression detection via logic and self-learning) and **Offense** (discovering performance optimization opportunities via guided learning).

For network efficiency, proactive AI agents scan vast volumes of production traffic data and apply domain-specific recipes across capacity, Edge, and GPU usage. Chen poses several concrete questions these agents actively investigate in production: * *"Is our edge routing and load balancing strategy optimal?"* * *"Are we overusing submarine cables when cheaper terrestrial paths are available?"* * *"Is high video bitrate necessary in markets where low-end devices dominate and HD adoption is low?"* * *"Is GPU utilization balanced across data centers?"*

- **Performance Dual Strategy:** Defense (Performance Regression Detection) vs. Offense (Guided Performance Enhancement). - **Efficiency Vectors:** Submarine cable routing vs. terrestrial path optimization, adaptive video bitrate tuning based on regional device demographics, and cross-datacenter GPU cluster load balancing.

---

[09:08](https://www.youtube.com/watch?v=WGBLZ4xmWp0&t=548s) Overcoming Key Technical Challenges: Hallucinations, Novel Scenarios, and Guardrails

Adopting AI for runtime network operations presents three major technical hurdles, along with Meta's concrete mitigation strategies:

1. **AI Hallucinations and Non-Deterministic Outputs:** Executing the same network function can yield inconsistent results. Meta mitigates this using an **"LLM-as-a-Judge"** framework: running multiple parallel iterations of an agent function against a strict rubric, then using an evaluator LLM to select the single most accurate, compliant output. 2. **Novel Incident Types and Zero-Day Scenarios:** Unknown failure modes are handled via self-improving pipelines that integrate real-time human feedback directly at runtime, expanding the system's operational memory. 3. **Escalating Operational and Security Risks:** As agent autonomy increases, the potential risk to live traffic grows. Meta is developing native AI guardrails built directly into network data models and infrastructure codebases to mitigate privacy, security, and reliability risks before actions touch production hardware.

- **LLM-as-a-Judge:** Uses defined rubrics to evaluate multi-session outputs and eliminate non-deterministic code/configuration execution. - **Safety Integration:** AI guardrails embedded directly into Infrastructure-as-Code (IaC) pipelines. - **Call to Action:** YuLing Chen concludes by inviting engineers to join Meta in building out the future of AI-native network infrastructure (AI4NI) across planning, design, delivery, and runtime operations.

## Notable quotes > [05:47](https://www.youtube.com/watch?v=WGBLZ4xmWp0&t=347s) In a mesh network, there's a lack of central decision maker or tiebreaker. To our experience, agents can be very talkative. — YuLing Chen

## Mentioned YuLing Chen, Meta, Instagram, BGP, NetIngram, LLM-as-a-Judge, AI4NI

[Source](https://www.youtube.com/watch?v=WGBLZ4xmWp0)

中文简报

Meta以分层AI智能体取代网络作战室 Meta正将其运行时网络运维从百名工程师的人工作战室，转向分层AI智能体网络，以实现突发事件响应自动化并消除代价高昂的无效交流循环。 Meta正在将其运行时网络运维从高压的人力“作战室”转向由数十万个AI智能体组成的分层网络。为避免智能体之间出现无限对话并浪费算力，Meta放弃了点对点网状智能体架构，转而采用集中式编排器来指导数据中心、骨干网和边缘网络中的领域专家智能体。除了借助NetIngram知识库进行事件响应和根本原因分析外，Meta还利用AI智能体优化路由、自适应视频码率以及跨数据中心GPU集群的负载均衡。针对LLM非确定性行为与新型突发事件等关键挑战，Meta则通过“LLM-as-a-Judge”评估模型以及直接嵌入在基础设施即代码（Infrastructure-as-Code）流水线中的安全防护机制来进行应对。 Meta 正在将依赖上百名工程师手动排障的高压“作战室”模式，转型为由数十万个协同 AI Agent 组成的 AI 原生网络运维体系（AI4NI），早期试点已成功检出传统监控遗漏的生产严重事故。 (2:09) Meta 明确反对点对点（Mesh）Agent 架构，指出其缺乏决策仲裁者容易导致死循环对话并大量浪费 GPU 算力，转而确立由中央跨域编排器统一调度数据中心、骨干网和边缘专家的分层 Agent 架构。 (5:47) Meta 打造了集中式网络知识库 NetIngram 作为持续自我学习的组织记忆，并采用“防御（退化检测）+ 进攻（主动优化）”的双向策略，优化边缘路由、海底光缆利用率及跨数据中心 GPU 负载平衡。 (7:34) 针对 AI 幻觉和非确定性输出，Meta 采用基于评估标准的“LLM-as-a-Judge”框架筛选最优执行方案，并将原生 AI 安全护栏直接内嵌于基础设施即代码（IaC）模型中以防范现网风险。 (9:08)

Reading summary

Meta Network Production Engineer YuLing Chen argues that operating hyper-scale networks requires replacing high-pressure, manual incident response with hierarchical multi-agent AI systems capable of handling detection, diagnosis, and infrastructure optimization. Replacing High-Pressure War Rooms with Autonomous Agents (2:09) Operating a network serving billions of users allows virtually no margin for error, as minor configuration mistakes—such as BGP crashes or disrupted data center drop scheduling—can cascade into multi-hour outages that impact revenue and degrade user experience across platforms like Instagram. Currently, Meta responds to major outages by assembling high-stress "war rooms" where more than 100 engineers manually parse massive telemetry datasets and debate mitigations under pressure, a process Chen describes as tedious and inherently risky. To transform runtime operations, Meta is shifting toward an agentic architecture designed to deploy hundreds of thousands of coordinated AI agents across the incident lifecycle. The roadmap envisions a multi-year progression from L1/L2 assistive tools to L3/L4 fully autonomous operations. In this model, specialized AI domain experts and individual AI engineers mirror human engineering roles to automate detection, root cause analysis (RCA), mitigation, repair, and prevention. Why Hierarchical Architectures Outperform Peer-to-Peer Mesh Systems (5:47) A critical architectural insight from Meta's production deployments is the failure of peer-to-peer mesh agent networks. Chen reports that mesh setups lack a central decision-maker or tiebreaker, which causes agents to engage in circular, repetitive communication loops that obscure conclusions and waste significant GPU compute resources. Meta instead standardized on a top-down hierarchical model governed by a central AI Cross-Domain Orchestrator. Incoming alerts feed into this orchestrator, which dispatches targeted investigation tasks to Data Center, Backbone, and Edge domain experts. These experts supervise sub-agents that gather specific telemetry, allowing the parent orchestrator to synthesize the evidence into actionable RCA reports. This operational execution is supported by NetIngram, an internal centralized knowledge base and organizational memory that facilitates continuous agent learning. Expanding AI Workflows into Proactive Infrastructure Optimization (7:34) Beyond incident mitigation, Meta applies agentic networks across a dual operational strategy: defensive performance regression detection and offensive, guided performance optimization. Rather than only reacting to alerts, proactive agents continuously evaluate live production data against engineering recipes to drive down infrastructure costs and optimize user latency. In practice, these agents analyze specific efficiency trade-offs. Key investigative vectors include determining whether edge routing strategies are optimal, assessing whether traffic can be shifted from expensive submarine cables to terrestrial paths, tuning video bitrates in regions dominated by lower-end devices with low HD adoption, and rebalancing GPU cluster utilization across geographically distributed data centers. Mitigating Non-Determinism and Live Production Risks (9:08) Deploying autonomous systems on live network infrastructure introduces substantial technical risks, which Meta addresses through three concrete safeguards. To counteract AI hallucinations and non-deterministic behavior—where identical network functions return inconsistent outputs—Meta implements an "LLM-as-a-Judge" evaluation framework. This mechanism runs parallel agent iterations against strict rubrics, using an independent evaluator model to select the most accurate result. For zero-day incidents and unfamiliar failure modes, Meta relies on self-improving pipelines that incorporate runtime human feedback to continuously update agent memory. Finally, to mitigate the operational and security risks inherent in autonomous remediation touching production hardware, Meta embeds automated AI guardrails directly into network data models and Infrastructure-as-Code pipelines. While early implementations demonstrate that specialized AI workflows can detect incidents missed by legacy monitoring, realizing fully autonomous L3/L4 network operations remains contingent on successfully governing agent non-determinism and enforcing strict code-level guardrails on live production traffic.

精炼阅读摘要

Meta 网络生产工程师陈玉玲（YuLing Chen）指出，面对数十亿用户规模带来的严苛运行时容错要求与传统百人应急室（War Room）的人工排障瓶颈，Meta 正全面转向以分层多智能体（Hierarchical Multi-Agent）架构为核心的 AI 原生网络运维体系。该体系依托中央编排器调度领域专家智能体，结合 NetIngram 知识基础设施与 LLM-as-a-Judge 评估机制，在保障可靠性与规避大模型幻觉的同时，协同实现网络性能与基础设施成本的主动优化。运行时故障的高昂代价与传统应急排障瓶颈 (0:05) Meta 支撑全球数十亿用户的网络基础设施对运行时容错率有着近乎为零的要求。陈玉玲指出，网络运维不仅要确保全天候稳定性，还必须保障低延迟并持续压降基础设施成本，任何轻微的工程配置变动都可能引发连锁反应，导致长达数小时的服务中断、收入受损及用户体验恶化。例如在 Meta 以往的生产事故中，工程师推送的错误变更曾破坏数据中心任务调度（Drop Scheduling），导致核心服务不可用及外部网站无法访问；网络配置失误曾引发前缀过载并导致边界网关协议（BGP）崩溃，造成活跃流量中断；而区域性数据中心过载更曾触发严重的 Sev-1 级连锁故障，致使全站服务瘫痪数小时。在应对被称为“暴雨天”的全站瘫痪级重大事故时，Meta 传统的运维模式严重依赖由 100 多名跨部门工程师临时组成的“应急室”（War Room）。这些工程师如同“消防员”一般，在高压环境下通过即时通信频道反复沟通，手动解析海量遥测日志与数据集以推断根因，并紧急推送缓解方案。陈玉玲强调，这种高度依赖人工排障的传统流程极其繁琐且伴随巨大心理压力，随着网络规模扩张，已暴露出不可持续的业务与安全风险。放弃网状通信而采用分层智能体架构的技术逻辑 (5:47) 为改变应急室的高压人工现状，Meta 规划部署协同运作的 AI 智能体体系，覆盖故障检测、根因分析（RCA）、故障缓解、修复到预防的完整生命周期。在系统架构设计上，Meta 明确否定了智能体对等互联的网状（Mesh）架构。陈玉玲警告称，网状架构缺乏中央决策节点和仲裁机制，实践中多智能体极易陷入无休止的“过度交流”死循环，不仅导致无法得出清晰确切的排障结论，还会浪费极其宝贵的 GPU 算力资源。因此，Meta 确立了以父级编排器为核心的分层智能体架构（Hierarchical Agent Architecture）。该架构设立中央大脑“跨领域 AI 编排器”（AI Cross-Domain Orchestrator），由其对接持续监控系统传入的网络 KPI 和警报，并将任务拆解下发给数据中心（Data Center）、骨干网（Backbone）和边缘网（Edge）三大领域的专家智能体。领域专家再调度更细分的子智能体执行具体指标和遥测排查。最终，编排器汇总所有子智能体提交的证据，收敛分歧形成统一结论，为人类工程师生成清晰的根因分析报告，并在未来安全防护机制成熟后执行自动化修复。早期生产验证、NetIngram 基座与自主化路径 (4:37) Meta 的多智能体系统已在其数据中心、骨干网和边缘基础设施中取得初步生产成果。陈玉玲披露，早期的 AI 工作流已经能够匹配甚至超越人类工程表现，例如内部部署的 AI 排障系统成功检测出了传统遗留监控系统完全遗漏的严重生产故障；此外，配置与容量专家智能体能够结合实时地图可视化界面，主动扫描海量数据及内部知识库以评估出向流量（Egress）容量风险。为了使智能体具备持续学习与进化的组织记忆，Meta 正在构建名为 NetIngram 的集中式网络知识库与知识图谱，作为整个网络工程部门的共享认知底座。基于该基础设施，Meta 制定了跨越数年的自主运维演进路线图：从早期的 L1/L2 辅助型建议工具，逐步推进到具备自我闭环排障、修复与预防能力的 L3/L4 级完全自主网络运维。攻防双重优化与针对幻觉和安全风险的治理 (7:34) 在可用性保障之外，Meta 将分层智能体架构应用于网络性能与基础设施成本优化，采取“防御”（Defense）与“进攻”（Offense）双重策略。防御侧依托逻辑规则与自主学习持续检测性能回退；进攻侧则通过引导学习主动挖掘效率提升机会。生产中的智能体正在针对具体工程问题展开常态化核查，例如评估边缘路由与负载均衡策略的最优性、核算是否存在陆缆替代路径以减少昂贵海底光缆的不必要调用、根据区域低端设备普及率动态调优视频码率以避免无意义的高清带宽开销，以及跨数据中心平衡 GPU 集群的计算与网络利用率。针对大模型在实时网络环境中面临的非确定性、新故障模式与操作安全挑战，Meta 提出了针对性的工程防护方案。为了解决大模型幻觉与相同输入产生不同输出的不确定性问题，Meta 引入了“LLM-as-a-Judge”评估框架，通过并行运行多次智能体推理并对照严格的规则基准，由独立的评估大模型选出最准确、最合规的执行方案；针对零日或未知故障模式，系统通过引入运行时的实时人类反馈不断扩充运维记忆；而对于自主操作可能带来的线上风险，Meta 将 AI 安全护栏直接内嵌至网络数据模型和基础设施即代码（IaC）流程中，确保任何自动化指令在作用于真实硬件前均经过严格的安全审计。陈玉玲展示了 AI 原生运维在应对超大规模网络复杂性方面的显著优势，但整个系统向 L3/L4 级全自动化的迈进仍高度依赖严格的工程护栏与人机协同。当前在大模型输出一致性与生产环境实时干预风险之间的权衡表明，在可预见的过渡阶段，智能体系统仍主要扮演高精度辅助诊断与决策支持角色，其自动化线上修复能力与更深层的跨基础设施自主权仍需在实际生产环境中持续验证。

Detailed notes — zh

## Background 本内容由 @Scale 频道于 2026 年 9 月 3 日发布在 YouTube 平台，演讲者为来自 Meta 的网络生产工程团队工程师 YuLing Chen。视频简介指出，演讲聚焦于服务数十亿用户的超大规模生产网络在运行时的严峻挑战，详细阐述了 Meta 如何通过“AI4NI @Runtime”体系从可靠性、性能与效率三个维度构建 AI 智能体，并分享了前沿落地成果、架构设计与应对技术瓶颈的解法。 ## TL;DR 面对人工“作战室”（War Room）高压排障的长周期痛点，Meta 正在打造一套由成千上万个专用 AI 智能体构成的分工体系，覆盖数据中心、骨干网与边缘网络。在系统拓扑上，Meta 摒弃了容易导致无休止争论与 GPU 算力浪费的网状（Mesh）架构，转向以跨域编排器为中枢大脑的层级化智能体架构（Hierarchical Architecture），实现高效的告警检测与 RCA 根因定位。除了在可靠性维度的深耕，该体系还通过攻防机制与全网审计，主动优化传输延迟并排查海底光缆配比与跨数据中心 GPU 算力不均等资源浪费。针对 AI 幻觉、未知故障与权限外溢等核心瓶颈，Meta 落地了基于严格准则的大模型裁判（LLM-as-a-Judge）、人类在环自进化流水线以及原生集成的安全护栏（AI Guardrails）。 ## Atmosphere & style 整场演讲呈现出典型硅谷顶级科技峰会短平快、硬核且直击痛点的技术风格。讲者凭借在 Meta 一线八年的基础设施实战积淀，开篇以真实的严重宕机事故和通宵达旦的人工“灭火”惨状建立起强烈的危机感与共鸣，随后迅速切换至严谨沉稳的架构解构。听感上没有任何虚浮的 AI 概念炒作，而是充满对 Mesh 拓扑弊端、GPU 算力损耗、输出随机性等工程细节的坦率反思与克制务实，最终在向 AI 原生运维演进的号召中展现出极具说服力的前瞻技术张力。 ## Bulletin - [00:05](https://www.youtube.com/watch?v=WGBLZ4xmWp0&t=5s) Meta 生产工程团队正构建由专用 AI 智能体（Specialized AI Agents）构成的 Agentic 系统，旨在从可靠性、性能与效率三维协同攻坚数十亿用户级网络运行时的运维难题。 - [04:37](https://www.youtube.com/watch?v=WGBLZ4xmWp0&t=277s) Meta 明确否定无中心的网状智能体拓扑（Mesh Network），因其易造成智能体过度争辩并虚耗 GPU 算力；转向采用跨域 AI 编排器为中枢的树状层级化架构（Hierarchical Architecture）。 - [04:37](https://www.youtube.com/watch?v=WGBLZ4xmWp0&t=277s) 生产实测中，NIA AI 工作流已成功检测出传统监控遗漏的高危故障，CFGI 智能体则能基于全网拓扑数据前置预警边缘机房容量暴增风险。 - [09:08](https://www.youtube.com/watch?v=WGBLZ4xmWp0&t=548s) 针对 AI 幻觉与随机非确定性，Meta 落地了基于预设评分规则的 LLM-as-a-Judge（大模型裁判）机制，并在底层代码库原生集成 AI 安全护栏以防范高自主权智能体的操作失控风险。 ## Key takeaways - [00:05](https://www.youtube.com/watch?v=WGBLZ4xmWp0&t=5s) 网络基础设施在运行时（Runtime）的核心目标由三大要素构成：全天候极高可靠性、极低延迟传输性能以及持续优化的基础设施成本效率。 - [00:05](https://www.youtube.com/watch?v=WGBLZ4xmWp0&t=5s) 超大规模网络故障的爆炸半径极大，历史严重事故曾因错误变更引发调度崩溃、BGP 路由前缀过载甚至全站数小时宕机，带来巨大的商业信誉与营收损失。 - [02:09](https://www.youtube.com/watch?v=WGBLZ4xmWp0&t=129s) 传统“作战室”（War Room）排障机制高度依赖 100 多名工程师通宵手动比对海量异构日志与监控，流程冗长繁琐且极易因人为误判加剧业务风险。 - [02:09](https://www.youtube.com/watch?v=WGBLZ4xmWp0&t=129s) Meta 拒绝打造单一全能型单体 Agent，而是规划由成千上万个具备网络专业拓扑知识的 AI 智能体协同编排，逐步推动运维从 L1/L2 辅助向 L3/L4 自主运维演进。 - [02:09](https://www.youtube.com/watch?v=WGBLZ4xmWp0&t=129s) 智能体系统在纵向上覆盖数据中心、骨干网与边缘网络，横向上明确划分为掌握全局上下文的“领域专家”与专注于特定网络组件排障深潜的“专职 AI 工程师”。 - [04:37](https://www.youtube.com/watch?v=WGBLZ4xmWp0&t=277s) 去中心化的 Mesh 智能体拓扑会因智能体无休止互相争论（talkative）而空耗宝贵的 GPU 推理算力，且难以收敛得出高确定性的诊断结论。 - [04:37](https://www.youtube.com/watch?v=WGBLZ4xmWp0&t=277s) 层级化架构以跨域 AI 编排器为大脑，负责接收告警、下发诊断调度、汇总领域专家证据链并自动产出结构化 RCA 归因报告与缓解方案。 - [04:37](https://www.youtube.com/watch?v=WGBLZ4xmWp0&t=277s) Meta 正在搭建统一的网络知识库 NetIngram，为底层 AI 智能体的自主微调、拓扑理解与持续自进化提供坚实的数据基座。 - [07:34](https://www.youtube.com/watch?v=WGBLZ4xmWp0&t=454s) 网络性能优化采取“攻防协同”策略：防守端依托金丝雀分析与无监督自学习检测性能回归，进攻端运用引导式强化学习主动探寻毫秒级延迟优化空间。 - [07:34](https://www.youtube.com/watch?v=WGBLZ4xmWp0&t=454s) AI 智能体正全面介入资源效率治理，针对边缘路由全局最优解、昂贵海底光缆过度占用、基于低端机市场的差异化视频码率分发以及跨机房 GPU 算力均衡展开常态化审计。 - [09:08](https://www.youtube.com/watch?v=WGBLZ4xmWp0&t=548s) 面对 AI 幻觉和模型输出的不确定性，系统通过多轮并行采样并引入基于技术准则的 LLM-as-a-Judge 裁判模型，择优输出高严密性的生产指令。 - [09:08](https://www.youtube.com/watch?v=WGBLZ4xmWp0&t=548s) 针对罕见的未知黑天鹅故障，团队正在构建集成人类专家反馈（Human-in-the-Loop）的自优化流水线，以便在实战中敏捷吸收专家处置智慧以更新知识库。 - [09:08](https://www.youtube.com/watch?v=WGBLZ4xmWp0&t=548s) 随着智能体由只读诊断迈向具有写权限的闭环自愈，Meta 将 AI 安全护栏（AI Guardrails）原生嵌入底层数据仓库与基础设施代码库（IaC），确保物理网络变更绝对受控。 ## Detailed walkthrough ### [00:05](https://www.youtube.com/watch?v=WGBLZ4xmWp0&t=5s) 运行时网络运维的严峻挑战：可靠性、性能与效率的三维命题在网络基础设施（Network Infrastructure, NI）的规划、设计与交付阶段之后，系统在生产运行阶段（Runtime）所面临的挑战更为苛刻。Meta 网络生产工程团队（Network Production Engineering）工程师 YuLing Chen 基于在 Meta 基础设施团队八年的一线经验指出，支撑数十亿全球用户的超大规模网络一旦发生可靠性故障，其故障爆炸半径（Blast Radius）往往会迅速蔓延。演讲中列举了数起真实的严重事故：近期曾有工程师推送了一项错误变更（Bad Change），直接破坏了数据中心内部的作业调度机制，导致关键网络服务相继瘫痪、面向用户的站点服务彻底不可访问；在另一次事故中，某项网络配置变更引发了路由前缀过载，导致 BGP 协议崩溃并最终造成大面积丢包；而数年前更是发生过因区域数据中心过载诱发的连锁故障（Cascading Failures），导致长达数小时的全站级严重故障（SEV）。这些事故直接造成全网服务宕机、用户体验受挫，并带来无法忽视的商业信誉受损与营收损失。全天候（24/7）守护网络可靠性是运行时运维的最高准则，但这并非孤立存在的考核目标。系统在保障高可用的同时，还必须确保极低延迟的内容传输速度；一旦传输延迟上升，例如 Instagram 等核心应用的用户将立即感受到明显的卡顿与体验降级，Meta 内部对此建立了精确的量化指标以及时触发干预。不仅如此，网络团队还背负着在可靠、快速交付内容的前提下持续压降基础设施成本的重任。如何借助 AI 技术，让超大规模网络在运行时同时实现“更高可靠、更快速度、更优效率”，成为了网络工程演进的核心命题。 - **高危故障形态**：配置变更引发的 BGP 路由前缀过载崩溃、底层作业调度异常导致的前端服务失联、数据中心过载引发的长达数小时级联中断。 - **业务核心痛点**：全站瘫痪、数十亿用户体验受损、直接商业损失与品牌信誉危机。 - **运行时的核心三角目标**：极高的全天候可靠性（Reliability）、严苛的低延迟传输性能（Performance）、持续优化的基础设施成本效率（Efficiency）。 --- ### [02:09](https://www.youtube.com/watch?v=WGBLZ4xmWp0&t=129s) 传统人工“灭火”困局与分工协作的 AI 智能体体系构想目前网络运行时运维仍然高度依赖人工干预。YuLing Chen 生动地描述了典型的“倒霉日”排障场景：当遭遇重大网络可靠性故障、Meta 全站服务瘫痪时，内部会紧急拉起一个线上“作战室”（War Room）。来自不同组织的 100 多名工程师迅速涌入其中充当“灭火队员”，在巨大的业务恢复时间压力下，通宵达旦地手动翻查海量日志与监控数据集，在聊天群组中反复推导可能的技术根因并激烈讨论应急措施。这种高度依赖人工排障的流程不仅冗长繁琐、令人筋疲力尽，还会因人为误判给业务带来极高风险。为了彻底扭转这种被动局面，Meta 正在构建一套由大量专用 AI 智能体（Specialized AI Agents）构成的 Agentic 系统。这些智能体深刻理解网络拓扑结构，具备与人类资深网络专家对齐的专业知识和技能。团队设想的终极形态绝非单一的“全能型单体 Agent”，而是由成千上万个 AI Agent 协同编排组成的庞大集群。在组织分工上，AI 智能体直接映射人类工程师的团队架构，纵向覆盖数据中心（Data Center）、骨干网（Backbone）和边缘网络（Edge Network）；横向则划分为掌握全局上下文的“领域专家”（Domain Experts）和专注于定向故障排查攻坚的“AI 工程师”（Individual AI Engineers）。从故障全生命周期来看，系统全面覆盖检测、根因分析、缓解恢复与风险预防，预计在未来几年内逐步完成从 L1/L2 辅助运维向 L3/L4 高度自主化运维的演进。 ```mermaid flowchart TD accTitle: AI 智能体分工架构图 accDescr: 展示从领域专家到专项工程师在数据中心、骨干网与边缘网络的协同关系与生命周期能力。 subgraph Roles ["智能体分工矩阵"] DE["领域专家智能体 (Domain Experts) 掌握全局宏观上下文与跨域决策"] IE["专项 AI 工程师 (Individual AI Engineers) 精通特定垂直领域排障与执行"] end subgraph Domains ["网络基础设施覆盖"] DC["数据中心网络 (Data Center)"] BB["骨干网 (Backbone)"] ED["边缘网络 (Edge Network)"] end subgraph Lifecycle ["故障生命周期闭环能力"] DET["检测 (Detection)"] RCA["根因定位 (RCA)"] MIT["缓解修复 (Mitigation & Repair)"] PRV["前置预防 (Prevention)"] end DE --> DC & BB & ED IE --> DC & BB & ED DC & BB & ED --> DET --> RCA --> MIT --> PRV ``` - **传统 War Room 痛点**：100 多名跨团队工程师涌入排障群，人工对比庞杂异构数据集，精神高压且耗时长。 - **智能体定位分层**： - **领域专家（Domain Experts）**：掌握宏观网络知识与跨架构拓扑，负责宏观协调与证据研判。 - **专职 AI 工程师（Individual AI Engineers）**：精通特定网络组件与故障排障，负责细分问题的深潜追踪。 - **演进路线**：从早期的 L1/L2 自动化告警归因辅助，逐步跨越至 L3/L4 级的跨域闭环自愈运维。 --- ### [04:37](https://www.youtube.com/watch?v=WGBLZ4xmWp0&t=277s) 生产实践成效与层级化架构（Hierarchical Architecture）智能体运维在 Meta 生产环境中已展现出显著成效。部署于数据中心、骨干网与边缘网络的工作流，已在告警检测、根因定位（RCA）以及风险评估中发挥实际价值，在某些关键场景下的表现甚至超越了资深人类工程师。例如，NIA AI 工作流成功检测到了传统监控告警系统完全遗漏的高危生产事故；CFGI AI 智能体则能全量扫描 Meta 内部的大规模网络运行数据集与知识库，借助精细的拓扑地图可视化界面，前置预测出口流量暴增对边缘机房容量的冲击风险。此外，Meta 正在统一打造名为 NetIngram 的专属网络知识库系统，为底层智能体的持续自我演化与自优化提供坚实的数据基座。在系统拓扑设计上，团队明确否定了网状智能体网络（Mesh Agent Network）。实践表明，在缺乏中心决策仲裁机制的网状拓扑中，智能体往往表现得极为“话痨”（"talkative"），它们在彼此之间无休止地交互争辩，导致耗费巨额 GPU 推理算力的同时却无法收敛出清晰明确的技术结论。因此，Meta 坚定采用了以父级编排器为中枢的层级化智能体架构（Hierarchical Agent Architecture）。跨域 AI 编排器扮演着整个系统的“大脑”（"brain"），持续监控网络核心 KPI 指标并接收异常告警；一旦出现异动，编排器迅速向下调度数据中心、骨干网及边缘网络的领域专家，各专家再驱动内部子智能体进行并行深度诊断。最后，由中枢编排器汇总全量证据链，产出结构化 RCA 归因报告呈送给人类工程师，未来系统还将结合严格的安全护栏机制，自动化生成并下发执行故障缓解与物理修复脚本。 ```mermaid flowchart TD accTitle: 运行时事件检测与层级化排障架构 accDescr: 告警输入后由中心跨域编排器调度各网络域专家及子智能体，汇总生成RCA报告并联动执行闭环。 subgraph Inputs ["指标采集与触发"] KPI["全网 KPI 实时监控"] --> Alert["异常告警触发 (Alerts)"] end subgraph Central ["中央决策大脑"] Alert --> Orch["跨域 AI 编排器 (Cross-Domain Orchestrator) 评估、指导、证据归纳、决策仲裁"] end subgraph DomainExperts ["领域专家与子智能体执行"] Orch --> D_DC["数据中心领域专家"] Orch --> D_BB["骨干网领域专家"] Orch --> D_ED["边缘网络领域专家"] D_DC --> S_DC["子智能体群 (Sub-Agents)"] D_BB --> S_BB["子智能体群 (Sub-Agents)"] D_ED --> S_ED["子智能体群 (Sub-Agents)"] end subgraph Outputs ["证据聚合与自动化闭环"] S_DC & S_BB & S_ED -.->|归集诊断细节| Orch Orch --> RCA["生成自动化 RCA 根因报告 (交付人类工程师审批)"] RCA --> Action["安全护栏加持的自动化修复方案 (规划修复、下发执行缓解)"] end ``` - **前沿生产战果**： - **NIA AI 工作流**：捕获传统监控完全脱靶的严重隐蔽生产故障。 - **CFGI AI 智能体**：自动化扫描知识库与全网数据，通过地图级可视化评估边缘出口容量激增风险。 - **NetIngram 知识基座**：沉淀网络配置与运维拓扑资产，支撑 Agent 自主微调与持续演进。 - **架构范式反思**：坚决摒弃 Mesh 架构，防止由于 Agent 无休止沟通造成 GPU 算力虚耗与逻辑死锁；采用“中心大脑+领域下潜”的层级化树状模式实现高确定性交付。 --- ### [07:34](https://www.youtube.com/watch?v=WGBLZ4xmWp0&t=454s) 性能与效率维度的双向突围：攻防协同与智能算力调度除了在可靠性维度的深耕，Meta 进一步将层级化 AI 智能体集群推广到网络性能优化与资源效率治理中。在网络性能领域，系统确立了“攻防兼备”的两大核心机制： 1. **防御机制（Defense）**：利用金丝雀分析（Canary Logic）与无监督自学习技术，构建常态化的性能劣化监测体系，第一时间发现端到端延迟畸变或吞吐回退； 2. **进攻机制（Offense）**：借助引导式强化学习（Guided Learning），在日常业务常态流转中主动探寻网络参数与路由策略的优化空间，持续为核心应用换取毫秒级延迟收益。在网络效率与成本优化维度，系统旨在利用 AI 智能体主动挖掘庞大基础设施集群中的浪费隐患。智能体持续巡检全网生产流量，并严格套用涵盖带宽容量、网络服务质量（QoS/Egress）以及 GPU 算力集群效率等维度的领域策略配方（Domain Recipes）。YuLing Chen 抛出了几个智能体正在积极拆解的经典工程问题：当前的边缘路由选路与负载均衡策略是否达到了全局数学最优？在低成本陆地网络通路完好可用的情况下，底层链路调度是否过度占用了成本高昂的海底光缆（Subsea Cables）？在低端智能手机占据绝对主导、高清视频接收率极低的区域市场中，系统是否依然盲目下发了不必要的高码率视频流造成边缘带宽浪费？跨数据中心间的 GPU 算力集群负载与网络互联资源是否维持了绝对均衡？这些由 AI 驱动的常态化自动化审计，为全网压降基础设施成本开辟了崭新的技术空间。 - **性能维度战法**： - **防守端（Defense）**：基于金丝雀逻辑的自学习性能回归检测。 - **进攻端（Offense）**：基于引导式学习的性能增强机会主动探测。 - **效率维度的核心质询**： - 边缘调度与负载均衡策略的全局最优性评估。 - 核心骨干网中昂贵海底光缆与平价备用链路的配比审计。 - 基于终端硬件生态（低端机占主导）与区域 HD 视频需求的内容码率差异化分发策略。 - 分布式训练场景下跨数据中心 GPU 集群算力利用率的平衡与通信利用率调优。 --- ### [09:08](https://www.youtube.com/watch?v=WGBLZ4xmWp0&t=548s) 运行时落地的三大技术瓶颈与原生安全护栏应对将 AI Agent 引入超大规模生产级网络运行时并非坦途，Meta 在推进过程中识别出三大硬核挑战。其首要瓶颈在于“AI 幻觉”（AI Hallucinations）以及模型输出的随机非确定性。正如日常使用大语言模型提问多次往往得到互不相同的答案一样，当命令 AI 智能体执行某一特定网络运行时函数时，输出结果也时常出现不一致甚至随机性漂移。为破解该难题，团队积极探索并落地了“大模型裁判”（LLM-as-a-Judge）评测闭环：针对具体的网络运维函数并行触发多轮模型执行，同时引入一套预设好的严苛技术评测准则（Rubric）；裁判模型严格依据该准则对各轮执行的会话轨迹进行多维度打分和比对，从中择取准确率最高、推理逻辑最严密的输出作为最终生产调用指令。第二个重大挑战在于系统如何应对罕见的未知故障类型或从未遇到过的黑天鹅事件（Never-seen Scenarios）。对此，Meta 正在建设集成运行时人类专家反馈（Human-in-the-Loop）的自我提升流水线，使系统能够在真实排障碰撞中吸收资深工程师的临时处置智慧，实现知识库的高频闭环自更新。第三个挑战则聚焦于“能力跃迁伴生的风险失控”，随着 Agent 权限从只读诊断深入至可写变更，必须防范隐私泄露、鉴权逃逸与可靠性反噬。Meta 正在研发 AI 专用安全护栏（AI Guardrails），并将其原生注入（Natively Integrated）到网络底层数据仓库与基础设施代码库（Infrastructure as Code）之中，确保无论 AI 智能体的自主度多高，所有对物理网络的下发控制都必须在硬性安全防线之内进行。演讲最后，YuLing Chen 总结并号召整个业界积极拥抱由 AI 原生驱动的下一代网络基础设施运维变革。 - **挑战一：AI 幻觉与执行输出非确定性** - **技术解法**：建立基于明确准则（Rubric）的 **LLM-as-a-Judge（大模型裁判机制）**，多轮采样推理并优中选优。 - **挑战二：全新长尾故障与未知黑天鹅场景** - **技术解法**：搭建包含运行时人类反馈（Human Feedback at Runtime）的自进化持续学习流水线。 - **挑战三：高权限智能体的安全性与可靠性失控风险** - **技术解法**：构建原生集成于网络基础设施代码与数据通道的 **AI Guardrails（安全护栏）**，严控数据隐私合规与变更准入安全。 ## Notable quotes > [00:05](https://www.youtube.com/watch?v=WGBLZ4xmWp0&t=5s) 如何借助 AI 技术，让超大规模网络在运行时同时实现“更高可靠、更快速度、更优效率”，成为了网络工程演进的核心命题。 — YuLing Chen > [02:09](https://www.youtube.com/watch?v=WGBLZ4xmWp0&t=129s) 100 多名工程师迅速涌入其中充当“灭火队员”，在巨大的业务恢复时间压力下，通宵达旦地手动翻查海量日志与监控数据集，在聊天群组中反复推导可能的技术根因并激烈讨论应急措施。 — YuLing Chen > [02:09](https://www.youtube.com/watch?v=WGBLZ4xmWp0&t=129s) 团队设想的终极形态绝非单一的“全能型单体 Agent”，而是由成千上万个 AI Agent 协同编排组成的庞大集群。 — YuLing Chen > [04:37](https://www.youtube.com/watch?v=WGBLZ4xmWp0&t=277s) 智能体往往表现得极为“话痨”（talkative），它们在彼此之间无休止地交互争辩，导致耗费巨额 GPU 推理算力的同时却无法收敛出清晰明确的技术结论。 — YuLing Chen > [04:37](https://www.youtube.com/watch?v=WGBLZ4xmWp0&t=277s) 跨域 AI 编排器扮演着整个系统的“大脑”（brain），持续监控网络核心 KPI 指标并接收异常告警。 — YuLing Chen > [07:34](https://www.youtube.com/watch?v=WGBLZ4xmWp0&t=454s) 当前的边缘路由选路与负载均衡策略是否达到了全局数学最优？ — YuLing Chen > [07:34](https://www.youtube.com/watch?v=WGBLZ4xmWp0&t=454s) 在低成本陆地网络通路完好可用的情况下，底层链路调度是否过度占用了成本高昂的海底光缆（Subsea Cables）？ — YuLing Chen > [07:34](https://www.youtube.com/watch?v=WGBLZ4xmWp0&t=454s) 在低端智能手机占据绝对主导、高清视频接收率极低的区域市场中，系统是否依然盲目下发了不必要的高码率视频流造成边缘带宽浪费？ — YuLing Chen > [09:08](https://www.youtube.com/watch?v=WGBLZ4xmWp0&t=548s) 正如日常使用大语言模型提问多次往往得到互不相同的答案一样，当命令 AI 智能体执行某一特定网络运行时函数时，输出结果也时常出现不一致甚至随机性漂移。 — YuLing Chen > [09:08](https://www.youtube.com/watch?v=WGBLZ4xmWp0&t=548s) 无论 AI 智能体的自主度多高，所有对物理网络的下发控制都必须在硬性安全防线之内进行。 — YuLing Chen ## Mentioned Meta, YuLing Chen, @Scale, Instagram, NIA AI, CFGI AI, NetIngram, BGP, GPU ## Video description When operating a large-scale production network serving billions of users, we face daily challenges in running network infrastructure reliably and efficiently. AI4NI @Runtime tackles these challenges across three dimensions: reliability, performance, and efficiency. At Meta, we’ve begun developing AI agents in each area and have seen promising early results. In this talk, we’ll share our solution architecture, example agentic systems, and initial findings from production trials, along with the key technical challenges and our planned directions to address them. Learn more about the @Scale conferences here: https://atscaleconference.com/ [Source](https://www.youtube.com/watch?v=WGBLZ4xmWp0)

Source quotes

[0:05] YuLing Chen: 如何借助 AI 技术，让超大规模网络在运行时同时实现“更高可靠、更快速度、更优效率”，成为了网络工程演进的核心命题。 [2:09] YuLing Chen: 100 多名工程师迅速涌入其中充当“灭火队员”，在巨大的业务恢复时间压力下，通宵达旦地手动翻查海量日志与监控数据集，在聊天群组中反复推导可能的技术根因并激烈讨论应急措施。 [2:09] YuLing Chen: 团队设想的终极形态绝非单一的“全能型单体 Agent”，而是由成千上万个 AI Agent 协同编排组成的庞大集群。 [4:37] YuLing Chen: 智能体往往表现得极为“话痨”（talkative），它们在彼此之间无休止地交互争辩，导致耗费巨额 GPU 推理算力的同时却无法收敛出清晰明确的技术结论。 [4:37] YuLing Chen: 跨域 AI 编排器扮演着整个系统的“大脑”（brain），持续监控网络核心 KPI 指标并接收异常告警。 [7:34] YuLing Chen: 当前的边缘路由选路与负载均衡策略是否达到了全局数学最优？ [7:34] YuLing Chen: 在低成本陆地网络通路完好可用的情况下，底层链路调度是否过度占用了成本高昂的海底光缆（Subsea Cables）？ [7:34] YuLing Chen: 在低端智能手机占据绝对主导、高清视频接收率极低的区域市场中，系统是否依然盲目下发了不必要的高码率视频流造成边缘带宽浪费？ [9:08] YuLing Chen: 正如日常使用大语言模型提问多次往往得到互不相同的答案一样，当命令 AI 智能体执行某一特定网络运行时函数时，输出结果也时常出现不一致甚至随机性漂移。 [9:08] YuLing Chen: 无论 AI 智能体的自主度多高，所有对物理网络的下发控制都必须在硬性安全防线之内进行。

Source quotes

[5:47] YuLing Chen: In a mesh network, there's a lack of central decision maker or tiebreaker. To our experience, agents can be very talkative.

Original transcript

[0:05] So far we have talked about AI for NI [0:08] across the planning, design and delivery [0:10] phases. What about runtime? My name is [0:14] Yuling Chen and I'm a network production [0:16] engineer at Meta. Over my eight years [0:19] working in meta net infra, I've seen [0:21] running network at scale comes with [0:24] significant challenges. Our network [0:27] infrastructure supports billions of [0:29] users. When network reliability [0:31] incidents happen, they can quickly lead [0:34] to a large blast radius. For example, in [0:38] a recent case, an engineer pushed a back [0:41] change that disrupted data center drop [0:44] scheduling. As a result, critical [0:47] network services became unavailable, [0:49] user eag [0:51] websites could not be reached. In [0:54] another incident, a network [0:56] configuration change caused prefix [0:58] overload and a BGP crash, ultimately [1:01] dropping user traffic. A few years ago, [1:05] a set was triggered by regional data [1:07] center overload leading to cascading [1:10] failures and site outages that lasted [1:13] multiple hours. All these network [1:16] incidents can cause metal website down, [1:18] user frustration, business impact, and [1:22] revenue loss. [1:24] As we can see, enhancing network [1:26] reliability around the clock is the [1:28] number one priority for network at run. [1:32] At the same time, we must deliver [1:34] network content fast. When latency [1:37] increases, customers such as our [1:39] Instagram users experience frustration [1:43] and we have clear metrics that quantify [1:45] this degraded experience for us to act [1:48] on. In addition, we also need to drive [1:52] infrastructure cost savings when we [1:55] deliver content reliably and quickly. [1:59] The question is, can we leverage AI to [2:02] operate network infrastructure more [2:04] reliably, faster, and more efficiently? [2:09] As of today, network at one is mostly [2:12] operated manually. Let's take a look at [2:15] network reliability. [2:17] Assuming today is a rainy day, we are [2:20] hitting a severe network reliability [2:22] incident. The entire Meta website is [2:26] down. What happens behind the scene is [2:28] that we create a war room inside meta [2:32] where more than 100 engineers from [2:34] different organizations get into this [2:37] war room as firefighters to mitigate the [2:40] issue. We manually look into multiple [2:44] large number of data sets try to figure [2:47] out the root causes chat about potential [2:50] solutions and under pressure to resolve [2:52] the issues as fast as possible. [2:55] This is a very tedious stressful process [2:59] which also creates risks to business. [3:03] Using AI, we are developing a agendic [3:06] system that is made of many specialized [3:09] AI agents that collaborate across the [3:12] incident life cycle. These agents [3:15] understand network topology and have the [3:18] knowledge and skills that mirror human [3:20] expertise in network reliability. [3:24] We envision not just one agent but [3:27] hundreds of thousands of AI agents [3:29] working together in a coordinated [3:32] fashion to ensure network reliability at [3:35] one time. Organization wise, AI agents [3:39] align with how human engineers are [3:41] structured. These agents spend data [3:44] center backbone and edge networks. We [3:48] categorize those agents as domain [3:50] experts and individual AI engineers [3:53] where domain experts hold broader [3:56] knowledge and individual AI engineers [3:58] excel at resolving domain specific [4:01] issues [4:03] from reliability incident life cycle [4:05] point of view. These agents provide [4:08] capabilities in detection, root cause [4:10] analysis, mitigation, repair and [4:13] prevention with cross domain [4:15] orchestration capabilities on data [4:17] center back and edge networks. Such a [4:20] system requires time to develop. We [4:23] envision a gradual adoption of AI agents [4:26] throughout the incident life cycles. [4:28] They will evolve from L1, L2 to L3, L4 [4:33] autonomous operations over the next few [4:35] years. [4:37] Where are we today? Across meta network [4:40] infra, we are seeing promising [4:42] reliability gains from agendic [4:44] automation. AIdriven agenda workflows [4:47] across data center, badbomb, and edge [4:50] network are now delivering production [4:52] impact in incident detection, root cause [4:55] analysis, and risk evaluation. [4:59] These workflows don't just replace [5:01] manual effort. They often match or even [5:04] exceed human performance in key [5:06] scenarios. For example, NIA AI workflow [5:10] detected severe production incidents [5:13] that standard monitoring systems were [5:15] unable to discover. [5:18] For another example, CFGI AI agents are [5:21] capable of scanning large data sets and [5:24] also knowledge base across meta and [5:26] evaluate egress increase risks to edge [5:29] capacity using impressive visualizations [5:32] on map view. Alongside these individual [5:36] agendic automation effort, we're [5:38] building a network knowledge base called [5:40] net ingram to support AI agent [5:43] self-improvement across net info. [5:47] Okay. Now you may ask what agentic [5:50] architecture should we use? Should it be [5:52] a mesh agent network? The answer is no. [5:56] In a mesh network, there's a lack of [5:58] central decision maker or tiebreaker. To [6:01] our experience, agents can be very [6:04] talkative. A mesh agent network could [6:06] end up with agents talking to each other [6:08] endlessly, leading to unclear [6:11] conclusions and wasted GPU resources. [6:15] Instead, a hierarchical agent [6:17] architecture with parent orchestrators [6:20] has proven effective across almost all [6:22] of our successful projects. The [6:25] orchestrator play a critical role. They [6:27] evaluate, guide, and coordinate sub [6:30] aents to achieve reliability goals. [6:33] Whether that involves automated root [6:35] cause analysis, risk evaluation, impact [6:38] analysis, mitigation, or repair. [6:43] Here is an example architecture for [6:45] network incident detection and rule [6:47] causing. What we do is we continuously [6:50] monitor network KPIs and generate alerts [6:53] and then send to AI cross domain [6:55] orchestrator which is the brain of the [6:57] entire architecture. This AI [7:00] orchestrator communicates with domain [7:02] experts in data center backbone and edge [7:06] which internally use sub aents to look [7:08] into the problems. The orchestrator then [7:11] collect outputs from these domain [7:13] experts, aggregates the evidence, comes [7:15] up with final conclusions and generate [7:18] root cause analysis RCA report to human [7:21] engineers. [7:23] Looking ahead, the system will also [7:25] propose fixed plans and with the right [7:27] SE safeguards automatically execute [7:30] mitigations and repairs. [7:34] So far we have talked a lot about using [7:36] AI in network reliability. What about [7:38] network performance? For network [7:41] performance, we enable a large number of [7:43] AI agents in a hierarchical architecture [7:46] to achieve two capabilities. [7:49] The first is using CAN logic and [7:52] self-learning for performance regression [7:54] detection. We also call it defense. The [7:57] second is using guided learning to [7:59] discover performance optimization [8:01] opportunities. We also call it offense [8:04] or performance enhancements. [8:07] For network efficiency, we are exploring [8:10] the use of AI agents to proactively [8:13] identify efficiency opportunities across [8:15] meta infrastructure. [8:17] The goal is to build AI agents that can [8:21] scan large volumes of meta production [8:23] traffic data and apply guided domain [8:26] specific recipes in areas like capacity, [8:30] EOS, and GPU efficiency. Here are some [8:34] example question these agents could help [8:36] to answer. [8:38] Is our edge routing and low balancing [8:41] strategy optimal? [8:43] Are we overusing sui cables when cheaper [8:46] paths are available? [8:49] Is high video bit rate necessary in [8:52] markets where low-end devices dominate [8:54] and HD abdoption is low? Is GPU [8:58] utilization balanced across data [9:00] centers? All these explorations help us [9:03] to discover opportunities for network [9:06] efficiency. [9:08] Okay, we've learned a lot about [9:11] reliability, performance, and [9:13] efficiency. What challenges have we [9:16] encountered as we adopt AI for NI in [9:19] network runtime? We are facing at least [9:21] three key challenges. The first is AI [9:25] hallucinations. I believe everyone has [9:28] the experience. Ask an AI the same [9:31] question multiple times and you may get [9:33] different answers. Similarly, when we [9:36] ask an AI agent to execute a network [9:39] runtime function, we may see [9:42] inconsistent or random results from time [9:44] to time. What we are exploring is an LLM [9:48] as judge approach. We run multiple [9:51] iterations of AI network functions using [9:53] a defined rubric. Then we ask the LLM [9:57] judge to evaluate each session against [9:59] that rubric and select the most accurate [10:03] output as the result. The second [10:06] challenge is handling novel incident [10:08] types or neverseen scenarios. [10:12] We are exploring self-improving [10:14] pipelines that incorporate human [10:16] feedback at runtime so the system can [10:19] continuously learn and improve. [10:22] The third one is managing risks as AI [10:26] capability increases. For example, we [10:29] need to minimize privacy, security, and [10:32] reliability risks these agents may [10:35] introduce to our production network. We [10:38] are developing AI guard rails and [10:40] integrating them natively into our [10:43] network data and infrastructure code [10:45] bases. This way, we can reduce AI risks [10:49] to our network production systems. [10:52] Okay, this concludes our AI for NI [10:55] session on network planning, design, [10:58] delivery, and runtime. As you've seen, [11:02] there are so many exciting opportunities [11:05] for us to work together. Join us for an [11:08] AI for ANI exploration journey across [11:11] MetaNet Infra. Thanks everyone.