Meta Replaces Mesh AI Network With Hierarchy to Cut Operational Risk

@Scale

Meta deploys a hierarchical multi-agent architecture to manage runtime network operations across data center, backbone, and edge infrastructure while avoiding the redundant compute costs and infinite loops of peer-to-peer agent designs.

AI-assisted summary of Live from SCCC: AI-Native Network Operation | YuLing Chen, Meta — original source. Check the original for full context.

Source published: 2026-09-13

Detailed notes

## Background This talk was delivered by YuLing Chen of Meta and published by @Scale on YouTube on August 31, 2026. According to the page description, Chen presents her Lightning Talk "AI-Native Network Operation" live at the Santa Clara Convention Center, detailing how Meta uses AI agents to solve reliability, performance, and efficiency challenges across its production network infrastructure.

## TL;DR YuLing Chen outlines Meta's shift toward an AI-native runtime architecture designed to operate large-scale network infrastructure serving billions of users. She explains how Meta is transitioning from manual, high-stress war-room incident responses to a hierarchical system of specialized AI agents governed by parent orchestrators rather than an unwieldy peer-to-peer mesh structure. Early production workflows like NIA and CapGenie are already outperforming human engineers in incident detection and capacity risk evaluation. Beyond reliability, these agents optimize global performance and efficiency, while guardrails like LLM-as-a-Judge and native IaC safety boundaries mitigate risks like hallucinations and operational errors.

## Atmosphere & style The presentation carries the brisk, pragmatic tone of an insider engineering talk delivered under the bright lights of a major tech conference. Chen speaks with grounded authority, contrasting the chaotic, high-pressure tension of a 100-person incident war room with the quiet efficiency of hierarchical, automated AI orchestration. The narrative feels structured, urgent, and focused on real-world application, systematically breaking down complex production failures before building up a clear vision for safe, agentic network management.

## Key takeaways - [00:04](https://www.youtube.com/watch?v=1hj3ys-rE-c&t=4s) Reliability incidents at Meta's massive scale trigger wide-reaching blast radiuses, directly impacting core services, website availability, and revenue across apps like Instagram. - [02:24](https://www.youtube.com/watch?v=1hj3ys-rE-c&t=144s) During high-stakes network incidents, Meta currently relies on a manual "war room" process involving over 100 engineers querying disparate datasets and debating root causes under intense pressure. - [02:24](https://www.youtube.com/watch?v=1hj3ys-rE-c&t=144s) Meta is transitioning to an agentic automation framework where hundreds of thousands of specialized AI agents mirror human engineering organizational structures across Data Center, Backbone, and Edge networks. - [02:24](https://www.youtube.com/watch?v=1hj3ys-rE-c&t=144s) The AI agent architecture will evolve over several years from Level 1 and Level 2 human-assisted workflows to Level 3 and Level 4 fully autonomous operations capable of automated mitigation and repair. - [04:49](https://www.youtube.com/watch?v=1hj3ys-rE-c&t=289s) Current production workflows like NIA have successfully detected severe production incidents that standard monitoring systems missed, while CapGenie scans egress risks to edge capacity using map-based interfaces. - [04:49](https://www.youtube.com/watch?v=1hj3ys-rE-c&t=289s) Meta explicitly rejects peer-to-peer mesh agent architectures due to endless communication loops and wasted compute, favoring a hierarchical agent structure governed by a central cross-domain orchestrator. - [07:58](https://www.youtube.com/watch?v=1hj3ys-rE-c&t=478s) Network performance is optimized through hierarchical AI agents using both a defensive posture (detecting performance regressions via Content Adaptive Network logic) and an offensive posture (proactive optimization opportunities). - [07:58](https://www.youtube.com/watch?v=1hj3ys-rE-c&t=478s) Efficiency AI agents scan production traffic data to optimize edge routing, reduce unnecessary subsea cable usage, adjust adaptive video bitrates for specific markets, and balance GPU compute utilization. - [09:29](https://www.youtube.com/watch?v=1hj3ys-rE-c&t=569s) To combat AI hallucinations and inconsistent non-deterministic outputs, Meta employs an "LLM-as-a-Judge" architecture that evaluates multiple execution iterations against a pre-defined rubric. - [09:29](https://www.youtube.com/watch?v=1hj3ys-rE-c&t=569s) Meta addresses novel incident scenarios through self-improvement feedback pipelines incorporating human engineer feedback, and mitigates security risks using native AI guardrails within Infrastructure-as-Code (IaC) repositories.

## Detailed walkthrough ### [00:04](https://www.youtube.com/watch?v=1hj3ys-rE-c&t=4s) The Real-World Costs of Network Downtime at Meta

YuLing Chen, a Network Production Engineer at Meta with eight years of experience in Meta's Network Infrastructure team, opens her talk by shifting focus from network planning and design to runtime operations. Running network infrastructure that supports billions of users presents massive operational hurdles. When reliability incidents occur at this scale, they trigger severe, wide-reaching blast radiuses that immediately impact core services.

Chen highlights several concrete examples from Meta's recent operational history to illustrate these stakes. In one instance, an engineer pushed a faulty change that disrupted data center job scheduling, leading to unavailable network services, a drop in user egress, and complete unreachability of the main Meta website. In another case, a bad configuration change caused a network prefix overload, crashing BGP and dropping active user traffic. A few years prior, a regional data center overload triggered a cascading cross-data center failure that caused site-wide outages lasting several hours. These incidents result in severe business impacts, lost revenue, degraded user experience, and user frustration across platforms like Instagram.

- Network downtime directly harms core business metrics, causing revenue loss, degraded performance, and user frustration across Meta's apps. - Latency increases are closely tracked using clear quantitative metrics that measure degraded user experience. - The top priority for network runtime operations is maintaining enhanced, 24/7 reliability while simultaneously optimizing speed and driving down infrastructure costs.

[00:24](https://www.youtube.com/watch?v=1hj3ys-rE-c&t=24s) Inside the War Room: The Need for Agentic Automation

When a major network incident hits Meta on a high-stakes day, the current response relies heavily on manual intervention. Meta assembles a real-time "war room" bringing together over 100 engineers from various cross-functional organizations. Under intense pressure to restore services quickly, engineers manually execute queries against massive, disparate datasets, debate potential root causes in chat channels, and attempt to coordinate fixes. This process is tedious, high-stress, and exposes the business to unnecessary operational risk.

To solve this, Meta is developing an agentic system composed of specialized AI agents designed to collaborate across the entire incident lifecycle. Rather than relying on a single monolithic model, Meta envisions hundreds of thousands of coordinated AI agents operating in runtime. These agents possess an inherent understanding of network topology and mirror the specialized domain knowledge and organizational structures of human engineering teams.

- Agents span three core network domains: Data Center, Backbone, and Edge networks. - Agent roles are divided into "Domain Experts" (holding broad system-level knowledge) and "Individual AI Engineers" (specializing in specific technical domains). - The system covers the complete incident lifecycle: detection, root cause analysis (RCA), mitigation, repair, and prevention. - Adoption will progress gradually over several years, evolving from Level 1 and Level 2 human assistance to Level 3 and Level 4 autonomous operations capable of automated mitigation and repair.

[04:49](https://www.youtube.com/watch?v=1hj3ys-rE-c&t=289s) Current Production Wins and the Debunked Mesh Architecture

Meta is already seeing concrete production gains from AI-driven agentic workflows across incident detection, root cause analysis, and risk evaluation. In multiple scenarios, these autonomous workflows outperform human engineers. For example, the **NIA** AI workflow has detected severe production incidents that standard monitoring systems missed entirely. Another system, **CapGenie**, scans large datasets across Meta to evaluate egress risks to edge capacity, presenting insights using map-based visual interfaces. To support continuous agent learning across the infrastructure, Meta is also building a specialized network knowledge base named **NetIngram**.

Addressing system design, Chen explicitly rejects the idea of a peer-to-peer mesh agent network. In a mesh topology, the lack of a central decision-maker or tiebreaker leads to "talkative" agents communicating endlessly in loops without reaching actionable conclusions, wasting expensive CPU and GPU compute resources. Instead, Meta uses a hierarchical agent architecture governed by parent orchestrators.

``` +-------------------------------+ | Cross-Domain Orchestrator | | (Central Decision Brain) | +---------------+---------------+ | +-------------------------+-------------------------+ | | | +-------v-------+ +-------v-------+ +-------v-------+ | DC Experts | | Backbone Exp. | | Edge Experts | +-------+-------+ +-------+-------+ +-------+-------+ | | | [Sub-Agents] [Sub-Agents] [Sub-Agents] ```

- In the detection and RCA architecture, continuous network KPI monitoring feeds alerts directly to a cross-domain AI orchestrator. - The central orchestrator queries domain experts across Data Center, Backbone, and Edge, which in turn dispatch specialized sub-agents to analyze telemetry. - The orchestrator aggregates the evidence from all domains, draws the final conclusion, and generates a structured RCA report for human engineers.

[07:58](https://www.youtube.com/watch?v=1hj3ys-rE-c&t=478s) Driving Network Performance and Global Efficiency

Beyond reliability, Meta applies hierarchical AI agent structures to network performance and efficiency. For performance, agents execute two distinct functions: a defensive posture using Content Adaptive Network (CAN) logic and self-learning to detect performance regressions, and an offensive posture using guided learning to uncover proactive optimization opportunities.

For efficiency, AI agents scan vast volumes of production traffic data to apply domain-specific operational recipes across capacity management, egress paths, and GPU allocation. These efficiency agents tackle complex infrastructure questions to eliminate wasted resource expenditures:

- **Edge Routing Strategy:** "Is our edge routing and load balancing strategy optimal?" - **Subsea Pathing:** "Are we overusing subsea cables when cheaper terrestrial network paths are available?" - **Adaptive Bitrates:** "Is high video bitrate really necessary in markets where low-end devices dominate and HD adoption is low?" - **Compute Balancing:** "Is GPU utilization balanced across data centers?"

[09:29](https://www.youtube.com/watch?v=1hj3ys-rE-c&t=569s) Overcoming Hallucinations, Novel Scenarios, and Safety Risks

Chen concludes by addressing the three primary technical challenges Meta faces when deploying AI for network operations at runtime, along with their mitigation strategies:

1. **AI Hallucinations and Inconsistency:** AI agents executing runtime network functions can yield variable or non-deterministic outputs. To counter this beyond prompt engineering and clean data pipelines, Meta employs an "LLM-as-a-Judge" architecture. The system executes multiple iterations of the network function, evaluates each session against a pre-defined rubric, and selects the most accurate output. 2. **Handling Novel Incident Scenarios:** Unprecedented incident types require adaptive systems. Meta is building self-improvement feedback pipelines that integrate human engineer feedback directly into the runtime loop to continually train the system on edge cases. 3. **Managing Security, Privacy, and Operational Risks:** Autonomous agents interacting with production infrastructure present inherent risks. Meta builds native AI guardrails directly into its network data and Infrastructure-as-Code (IaC) repositories to enforce strict safety boundaries before any change touches live traffic.

## Notable quotes > [00:04](https://www.youtube.com/watch?v=1hj3ys-rE-c&t=4s) When reliability incidents occur at this scale, they trigger severe, wide-reaching blast radiuses that immediately impact core services. — YuLing Chen

> [00:04](https://www.youtube.com/watch?v=1hj3ys-rE-c&t=4s) In one instance, an engineer pushed a faulty change that disrupted data center job scheduling, leading to unavailable network services, a drop in user egress, and complete unreachability of the main Meta website. — YuLing Chen

> [00:04](https://www.youtube.com/watch?v=1hj3ys-rE-c&t=4s) A few years prior, a regional data center overload triggered a cascading cross-data center failure that caused site-wide outages lasting several hours. — YuLing Chen

> [00:04](https://www.youtube.com/watch?v=1hj3ys-rE-c&t=4s) The top priority for network runtime operations is maintaining enhanced, 24/7 reliability while simultaneously optimizing speed and driving down infrastructure costs. — YuLing Chen

> [00:24](https://www.youtube.com/watch?v=1hj3ys-rE-c&t=24s) Meta assembles a real-time "war room" bringing together over 100 engineers from various cross-functional organizations. — YuLing Chen

> [00:24](https://www.youtube.com/watch?v=1hj3ys-rE-c&t=24s) Rather than relying on a single monolithic model, Meta envisions hundreds of thousands of coordinated AI agents operating in runtime. — YuLing Chen

> [04:49](https://www.youtube.com/watch?v=1hj3ys-rE-c&t=289s) In a mesh topology, the lack of a central decision-maker or tiebreaker leads to "talkative" agents communicating endlessly in loops without reaching actionable conclusions, wasting expensive CPU and GPU compute resources. — YuLing Chen

> [07:58](https://www.youtube.com/watch?v=1hj3ys-rE-c&t=478s) Is our edge routing and load balancing strategy optimal? — YuLing Chen

> [07:58](https://www.youtube.com/watch?v=1hj3ys-rE-c&t=478s) Are we overusing subsea cables when cheaper terrestrial network paths are available? — YuLing Chen

> [07:58](https://www.youtube.com/watch?v=1hj3ys-rE-c&t=478s) Is high video bitrate really necessary in markets where low-end devices dominate and HD adoption is low? — YuLing Chen

> [07:58](https://www.youtube.com/watch?v=1hj3ys-rE-c&t=478s) Is GPU utilization balanced across data centers? — YuLing Chen

> [09:29](https://www.youtube.com/watch?v=1hj3ys-rE-c&t=569s) To counter this beyond prompt engineering and clean data pipelines, Meta employs an "LLM-as-a-Judge" architecture. — YuLing Chen

> [09:29](https://www.youtube.com/watch?v=1hj3ys-rE-c&t=569s) Meta builds native AI guardrails directly into its network data and Infrastructure-as-Code (IaC) repositories to enforce strict safety boundaries before any change touches live traffic. — YuLing Chen

## Mentioned YuLing Chen, Meta, Instagram, Santa Clara Convention Center, AI4NI, NIA, CapGenie, NetIngram, BGP, Content Adaptive Network, LLM-as-a-Judge, Infrastructure-as-Code

[Source](https://www.youtube.com/watch?v=1hj3ys-rE-c)

中文简报

Meta以分层架构取代网状AI网络以降低运维风险 Meta部署分层多智能体架构来管理数据中心、骨干网和边缘基础设施的运行时网络运维，同时避免点对点智能体设计带来的冗余计算成本与无限循环问题。 YuLing Chen概述了Meta向AI原生运行时架构的转型，该架构旨在运维服务数十亿用户的大规模网络基础设施。她解释了Meta正如何从依赖人工且高压力的“作战室”应急响应模式，过渡到由上级编排器统管的专用AI智能体分层体系，而非笨重的点对点网状结构。如NIA和CapGenie等早期生产工作流在故障检测和容量风险评估方面已超越人类工程师。除保障可靠性外，这些智能体还优化了全局性能与效率，而LLM-as-a-Judge及原生IaC安全边界等防护机制则降低了幻觉和操作失误等风险。 Meta 正在转变其高度依赖人工“作战室”（war room）的网络故障排查流程，转向由数十万个专业 AI Agent 组成的自动化运维架构，映射数据中心、骨干网及边缘团队的工程结构。 (0:24) 在生产环境中，NIA 等 AI 工作流已成功检出传统监控遗漏的严重故障，CapGenie 则利用地图界面主动扫描边缘容量的出站流量风险。 (4:49) 由于对等网状（P2P mesh）架构缺乏决策中心并会导致无休止的无效通信与算力浪费，Meta 明确摒弃该方案，转而采用由中央跨域编排器统领的分层 Agent 架构。 (4:49) 分层 AI Agent 还被应用于性能防御与主动优化，并通过分析生产流量优化边缘路由、减少昂贵的海底光缆消耗以及平衡数据中心间的 GPU 算力利用率。 (7:58) 为解决 Agent 的幻觉与运行风险，Meta 引入“LLM-as-a-Judge”评审机制、结合人工反馈的自迭代链路，并在基础设施即代码（IaC）中直接内置原生安全护栏。 (9:29)

Reading summary

Meta Network Production Engineer YuLing Chen argues that operating global network infrastructure requires shifting from high-stress, manual engineering war rooms to a hierarchical multi-agent AI system capable of managing the full incident and optimization lifecycle. The Operational Limits of Manual War Rooms (0:04) Running network infrastructure for billions of users means reliability incidents produce massive blast radiuses that immediately degrade services, trigger site-wide outages, and harm core business metrics. Past operational failures at Meta—including bad configuration changes that overloaded BGP prefixes, disrupted data center job schedulers, and triggered cascading cross-region data center outages lasting hours—highlight the severe financial and user-experience costs of downtime. Meta currently responds to high-stakes incidents by assembling real-time war rooms of more than 100 cross-functional engineers. Chen notes that this manual workflow requires engineers to query disparate datasets and debate potential root causes under pressure. To mitigate this operational bottleneck, Meta envisions deploying hundreds of thousands of coordinated AI agents mapped across Data Center, Backbone, and Edge domains to mirror human organizational structures and domain knowledge. Hierarchical Orchestration Over Mesh Architectures (4:49) Chen explicitly rejects peer-to-peer mesh topologies for autonomous agents. Without a central decision-maker, peer agents can become overly talkative and trap themselves in communication loops without resolving incidents, wasting expensive CPU and GPU compute. Meta instead relies on a hierarchical agent architecture governed by a central cross-domain orchestrator. In this model, monitoring alerts feed directly to the top-level orchestrator, which delegates analysis to domain-expert agents across Data Center, Backbone, and Edge networks. These experts dispatch specialized sub-agents to inspect telemetry before the orchestrator synthesizes the findings into a definitive root cause analysis report. Production implementations are already yielding results: the NIA workflow has detected critical incidents missed by standard telemetry, CapGenie evaluates edge capacity egress risks, and the NetIngram knowledge base facilitates continuous agent learning. Algorithmic Traffic and Resource Optimization (7:58) Beyond incident response, Meta applies agent workflows to infrastructure performance and efficiency. For network performance, agents adopt a dual posture: a defensive function using Content Adaptive Network logic to detect performance regressions, and an offensive function using guided learning to identify proactive optimization opportunities. For efficiency, specialized agents scan production traffic to answer complex operational questions across capacity and hardware allocation. These include determining whether traffic is overusing costly subsea cables instead of terrestrial paths, assessing if high video bitrates can be throttled in regions dominated by low-end devices, auditing edge routing strategies, and balancing GPU compute utilization across data centers. Safeguards for Hallucinations and Live Production (9:29) Deploying autonomous agents directly onto runtime network infrastructure introduces significant non-deterministic risks, particularly when handling novel network incidents or generating erroneous outputs. To mitigate hallucinations and inconsistent actions, Meta implements an 'LLM-as-a-Judge' architecture that executes multiple task iterations, evaluates each session against a structured rubric, and selects the most accurate output. For unprecedented failure scenarios, Meta integrates human feedback loops directly into the operational runtime to train models on edge cases. Furthermore, to prevent unauthorized or unsafe configuration pushes to live traffic, native AI guardrails are embedded directly into network data pipelines and Infrastructure-as-Code repositories. Chen forecasts that Meta's transition toward agentic operations will occur gradually over several years, advancing from Level 1 and Level 2 human-assisted troubleshooting to Level 3 and Level 4 autonomous mitigation and repair as safety boundaries and feedback loops mature.

精炼阅读摘要

Meta网络生产工程师YuLing Chen指出，针对支撑数十亿用户的大规模基础设施，传统依赖百人“作战室”的手工应急模式风险过高，Meta正转向由分层AI智能体（Agent）驱动的AI原生网络运维体系。网络宕机的高昂代价与传统作战室瓶颈 (0:04) 在Meta的超大规模网络基础设施下，单次运行期故障往往会引发大范围的爆炸半径并直接冲击核心业务。Chen列举了多项实际案例：工程师推送错误变更打乱数据中心作业调度，导致用户出向流量下跌与主站完全无法访问；错误的配置变更导致网络前缀过载并引发BGP崩溃；以及区域数据中心过载引发长达数小时的级联跨数据中心瘫痪。这些事故不仅带来直接的营收损失，还严重破坏了Instagram等产品的用户体验。然而，当前应对重大事故的方式高度依赖人工干预。在关键时刻，Meta需要召集超过100名跨职能工程师组建实时“作战室”（War Room）。工程师们必须顶着恢复服务的巨大压力，在庞杂分散的数据集中手动执行查询并在聊天群组中争论根本原因。Chen强调，这种人工协同模式耗时费力、压力巨大，已成为业务运行中不可忽视的操作风险。分层智能体架构与废弃Mesh拓扑的原因 (4:49) 为了取代人工密集型操作，Meta规划部署数十万个专业化AI智能体，覆盖数据中心、骨干网与边缘网络三大领域。架构角色明确划分为掌握全局系统知识的“领域专家”（Domain Experts）和专注特定技术方向的“独立AI工程师”（Individual AI Engineers）。业务推进路线将历经数年，从最初的Level 1和Level 2人类辅助，逐步演进至具备自动化缓解与修复能力的Level 3和Level 4自主运行阶段。在架构设计上，Chen明确否定了对等式Mesh（网状）智能体网络。在缺乏中央决策或仲裁机制的Mesh网络中，智能体会陷入无休止的“多嘴”循环沟通，无法形成可执行结论并白白消耗宝贵的CPU与GPU算力资源。因此，Meta采用由父级编排器主导的层次化架构：连续网络KPI监控将告警上报给跨领域AI编排器，编排器向下调度数据中心、骨干网与边缘专家智能体，再由专家派发子智能体分析遥测数据，最终由顶层编排器综合各方证据，形成最终根因分析（RCA）报告交付人类工程师。落地成效与覆盖全生命周期的效能优化 (7:58) Meta在生产环境中已取得多项实际战果。其自主工作流在特定场景下已超越人类工程师：例如AI工作流“NIA”成功检测出常规监控系统完全遗漏的严重生产事故；“CapGenie”系统通过扫描全公司海量数据集来评估边缘容量的出向风险，并以地图可视化形式呈现见解；同时，Meta正在建设专用网络知识库“NetIngram”以支持智能体持续学习。除了事故处理，该架构还被应用于全网性能与成本优化。在性能层面，智能体兼具防御性（基于内容自适应网络逻辑和自学习检测性能衰退）与进攻性（通过引导式学习挖掘主动优化空间）。在效率层面，智能体直接介入解决高成本运营问题，包括评估边缘路由与负载均衡是否最优、检测是否在低成本陆地网络可用时过度使用昂贵海底光缆、判断低端设备为主的市场中是否真正需要高视频码率，以及核验不同数据中心间的GPU算力负载是否均衡。应对幻觉、未知场景与安全防线的三重策略 (9:29) 将自主AI系统接入实时生产网络面临三大核心技术挑战，Meta为此设计了针对性的防御方案。首先是AI幻觉与非确定性输出：除了常规的提示词工程与纯净数据流水线，Meta引入“LLM评审员”（LLM-as-a-Judge）架构，通过多次运行网络功能并依照预设评分准则进行多轮评估，最终遴选出最准确的操作输出。其次是应对前所未见的故障场景：Meta构建了自提升反馈流水线，将人类工程师的反馈直接嵌入运行期循环，促使模型持续在极端边界用例中学习进化。最后是安全与操作合规风险：自主智能体对生产环境的任何变更都伴随潜在危险，Meta将原生AI护栏直接嵌入网络数据及基础设施即代码（IaC）代码库，确保所有变更在触达实时网络流量之前必须通过严密的安全性边界校验。 YuLing Chen总结指出，Meta的网络运营正系统性迈向AI原生模式。尽管分层架构、评审员机制和IaC护栏为高可靠运行奠定了基础，但从人机协同向高级别全自主修复演进的过程中，如何在规避极端安全风险的同时持续消化未知新型故障，仍是该体系长期演进中需不断验证的核心课题。

Detailed notes — zh

## Background 该视频由频道 @Scale 于 2026 年 9 月 3 日发布在 YouTube 平台。内容为 Meta 网络生产工程师 YuLing Chen 在圣克拉拉会议中心（SCCC）发表的“AI 原生网络运维”（AI-Native Network Operation）闪电演讲。视频简介指出，演讲聚焦于利用 AI Agent 解决支撑数十亿用户的大规模生产网络运行时难题，分享了在可靠性、性能与效率三维框架下的解决方案架构、生产实践成效及关键技术挑战应对。 ## TL;DR 针对超大规模生产网络运行时高度依赖人工排障与跨部门“作战室”应对危机的被动困境，Meta 正在构建覆盖数据中心、骨干网和边缘网络的分层 AI Agent 体系。演讲明确阐述了团队为何摒弃缺乏仲裁且空耗算力的网状（Mesh）拓扑，坚决采用中央编排器统一调度领域专家的分层架构，以实现故障检测与根因分析的闭环。除了全天候防范配置错误与级联宕机，AI Agent 还在延迟攻防调优以及海缆、码率和跨集群 GPU 负载等成本优化中发挥核心作用。针对落地中的模型幻觉、未知故障及权限安全隐患，团队给出了 LLM-as-a-Judge 裁判评估、人类在环自我进化流水线与原生内嵌 AI 护栏的系统性解法。 ## Atmosphere & style 整场演讲干练紧凑、充满极度真实的工业一线质感。讲者结合自身在 Meta 八年的运维实战经历，将网络突发严重故障、上百名工程师在阴雨天紧急拼凑作战室争分夺秒排障的窒息感描摹得淋漓尽致。随着分层编排架构与落地成效的展开，情绪迅速从传统运维的人力疲态转向依托前沿 Agent 体系重构工程范式的冷静与笃定；讲者直言 Agent“过于健谈”而果断舍弃网状拓扑的工程断言，更让整场分享散发出顶级科技巨头面对自治网络前沿探索时的务实与锋芒。 ## Bulletin - [00:04](https://www.youtube.com/watch?v=1hj3ys-rE-c&t=4s) Meta 正在推动网络基础设施运行时（Runtime）的 AI 原生变革，构建覆盖可靠性、性能与成本效率的多 Agent 协同体系，以彻底替代上百名工程师高压排障的传统“作战室”模式。 - [05:56](https://www.youtube.com/watch?v=1hj3ys-rE-c&t=356s) 在架构选型上，Meta 明确摒弃去中心化的网状（Mesh）拓扑以避免 Agent 无休止讨论空耗 GPU 算力，转而采用以中央跨域编排器为核心的分层 Agent 架构进行高效裁决与调度。 - [03:15](https://www.youtube.com/watch?v=1hj3ys-rE-c&t=195s) 早期落地项目已在生产环境显现实效：NIA AI 成功检出传统监控遗漏的隐患，Cap Genie 实现边缘容量风险主动感知，统一知识库 NetIngram 正在为全网 Agent 提供自我进化底座。 - [09:29](https://www.youtube.com/watch?v=1hj3ys-rE-c&t=569s) 面对网络运行时对确定性的严苛要求，Meta 重点探索了带有严格评估细则的“LLM-as-a-Judge”多轮仲裁机制，并将原生 AI 安全护栏直接植入底层代码库以防范自主操作风险。 ## Key takeaways - [00:04](https://www.youtube.com/watch?v=1hj3ys-rE-c&t=4s) 网络基础设施的真正考验发生在运行时（Runtime），在支撑数十亿用户规模下，哪怕一次配置缺陷也会迅速引发大范围级联宕机与巨额业务损失。 - [00:04](https://www.youtube.com/watch?v=1hj3ys-rE-c&t=4s) 网络运维长期困于繁琐且高压的手工作业流程，一旦发生严重事故往往需要召集上百名跨部门工程师组建“作战室”逐条运行查询和排障。 - [00:04](https://www.youtube.com/watch?v=1hj3ys-rE-c&t=4s) 运行时网络运维围绕三大核心目标展开：全天候抵御配置错误与 BGP 崩溃的可靠性、保障用户体验的极低延迟性能，以及持续优化的基础设施成本。 - [03:15](https://www.youtube.com/watch?v=1hj3ys-rE-c&t=195s) Meta 的未来网络运维愿景并非单一全能的大模型，而是由成千上万个专业分工的 AI Agent 组成协同网络，分管数据中心、骨干网与边缘网络。 - [03:15](https://www.youtube.com/watch?v=1hj3ys-rE-c&t=195s) Agent 体系演进路径明确规划为从当前 L1/L2 级别的人机协同与数据关联初筛，逐步发展为未来 L3/L4 级别的自主闭环止血与故障自愈。 - [03:15](https://www.youtube.com/watch?v=1hj3ys-rE-c&t=195s) 生产试点成效显著：NIA AI 工作流成功捕捉到传统告警遗漏的生产隐患，Cap Genie 则通过地图可视化实现了边缘容量与出向流量的精准扫描研判。 - [03:15](https://www.youtube.com/watch?v=1hj3ys-rE-c&t=195s) Meta 正在搭建统一网络领域知识库“NetIngram”，作为 Net Infra 体系内各类 AI Agent 持续积累和进化的核心知识基座。 - [05:56](https://www.youtube.com/watch?v=1hj3ys-rE-c&t=356s) 扁平的去中心化 Mesh 架构被坚决否定，因为 AI Agent 极其健谈（talkative），缺乏仲裁者会导致无限多向沟通并严重浪费宝贵的 GPU 算力。 - [05:56](https://www.youtube.com/watch?v=1hj3ys-rE-c&t=356s) 分层 Agent 架构依靠跨域编排器充当中央大脑，统筹下发任务给领域专家与专项子 Agent，汇总证据后生成根因分析（RCA）报告并驱动安全缓解。 - [08:01](https://www.youtube.com/watch?v=1hj3ys-rE-c&t=481s) 网络性能治理实行攻防结合策略：防御端利用 CAN 逻辑与自学习阻断性能劣化回退，进攻端通过引导式学习主动发掘潜在调优突破口。 - [08:01](https://www.youtube.com/watch?v=1hj3ys-rE-c&t=481s) 成本效率 Agent 主动巡检全网生产流量，重点优化边缘路由策略、规避高昂海缆的过度使用，以及实现跨数据中心 GPU 集群算力利用率均衡。 - [09:29](https://www.youtube.com/watch?v=1hj3ys-rE-c&t=569s) 为抑制模型幻觉与不确定性，Meta 引入带有基准细则（Rubric）的 LLM-as-a-Judge 机制，通过多轮运行并由模型裁判打分筛选出最严谨结论。 - [09:29](https://www.youtube.com/watch?v=1hj3ys-rE-c&t=569s) 针对历史未见的全新故障场景（Novel Incidents），系统搭建人机协同反馈流水线，实时捕获工程师的现场处理动作以实现知识自我进化。 - [09:29](https://www.youtube.com/watch?v=1hj3ys-rE-c&t=569s) 随着 Agent 逐步迈向自主执行生产变更，必须将原生 AI 安全护栏（AI Guardrails）深度内嵌至网络数据管道与底层代码库中以确保系统安全。 ## Detailed walkthrough ### [00:04](https://www.youtube.com/watch?v=1hj3ys-rE-c&t=4s) 生产网络运行时的现实挑战与“作战室”困境在网络基础设施的规划、设计与交付阶段之后，真正的严峻考验发生在“运行时”（Runtime）。Meta 的网络生产工程师（Network Production Engineer）YuLing Chen 结合自己在 MetaNet Infra 八年的工作经历指出，支撑数十亿用户规模的全球网络基础设施，任何运行时故障都会迅速演变为巨大的“爆炸半径”（blast radius）。她列举了几个代表性案例：近期曾有工程师推送了一次有缺陷的变更，导致数据中心作业调度瘫痪，网络服务随即不可用，出向流量骤降，致使 Meta 网站彻底无法访问；另一起事件中，一次错误的配置变更引发了网络前缀过载与 BGP 崩溃，造成大面积丢包；而在几年前，某一区域数据中心过载更是引发了跨数据中心的严重级联故障，全站宕机持续数小时之久。这些事故直接造成业务受损、用户受挫以及巨大的收入损失。保障 24 小时全天候的网络可靠性是网络运行时的首要任务，但与此同时，团队还必须保障极低延迟的内容分发性能（例如 Instagram 用户对卡顿极为敏感），并在高可靠、高性能的前提下持续削减基础设施成本。然而，如今的网络运维在很大程度上仍停留在高度依赖人工的阶段。YuLing Chen 形象地描述了一个典型场景：在一个恶劣的阴雨天，网络突发严重故障导致网站全线瘫痪，Meta 内部会紧急组建一个多达上百名工程师的跨部门“作战室”（War Room）。大家在巨大的心理压力下，人工对海量数据集运行查询、排查根因、争论排障方案并争分夺秒地尝试止血。这种手工作业流程不仅繁琐且极其令人心力交瘁，更为企业的业务连续性带来了巨大的潜在风险。 - **三大核心运营维度**： - **可靠性（Reliability）**：全天候防范配置错误、BGP 崩溃及跨数据中心级联故障。 - **性能（Performance）**：持续压降网络延迟，通过量化指标监控并保障最终用户体验。 - **成本效率（Cost Efficiency）**：在交付高质量服务的同时，持续优化基础设施投入。 --- ### [03:15](https://www.youtube.com/watch?v=1hj3ys-rE-c&t=195s) AI 原生运维体系与分层 Agent 演进路径为了彻底改变人工应对危机的被动局面，Meta 正在研发由多种专用 AI Agent 组成的智能协同系统。这些 Agent 深度理解网络拓扑结构，具备与人类资深网络专家相当的专业知识与排障技能。在未来的愿景中，并非只有一个全能的单体 Agent，而是由成百上千甚至数万个 AI Agent 协同运作，共同守护运行时网络。在组织架构上，这些 Agent 的分工映射了人类工程师团队的组织模式：覆盖数据中心（Data Center）、骨干网（Backbone）和边缘网络（Edge），并进一步细分为掌握宏观全貌的“领域专家”（Domain Experts）和专注于解决具体技术疑难的“独立 AI 工程师”（Individual AI Engineers）。在故障生命周期管理上，这套 Agent 体系贯穿了故障检测（Detection）、根因分析（Root Cause Analysis / RCA）、应急缓解（Mitigation）、系统修复（Repair）以及事后预防（Prevention）。由于如此庞大的自治系统无法一蹴而就，团队设计了一条循序渐进的演进路线：当前阶段主要定位在 L1/L2 级别的人机协同辅助，重点辅助人类工程师进行数据关联与初筛；在未来数年内，系统将逐步演化至 L3/L4 级别的全自动闭环运营，实现由 Agent 自主执行止血缓解与网络自愈修复。目前，Meta 内部的多个 Agentic 工作流已在生产环境中落地并取得了令人鼓舞的成效： - **NIA AI 工作流**：成功检测出标准监控告警系统完全遗漏的严重生产网络隐患。 - **Cap Genie AI Agents**：具备自主扫描全网海量数据的能力，通过直观的地图可视化视图对边缘网络容量（Edge Capacity）及出向流量风险进行精准研判。 - **NetIngram**：Meta 正在打造的网络领域统一知识库，旨在为 Net Infra 全链路上的 AI Agent 提供持续自我进化的知识底座。 --- ### [05:56](https://www.youtube.com/watch?v=1hj3ys-rE-c&t=356s) 为什么摒弃 Mesh 架构：分层编排器的核心设计在 Agent 系统的架构选型上，团队直面了一个核心疑问：Agent 之间应该组成去中心化的网状拓扑（Mesh Network）吗？YuLing Chen 给出了极其明确的否定回答：“AI Agent 本质上可以非常健谈（talkative）。”在完全扁平的网状架构中，缺乏中央决策者和仲裁机制（tiebreaker），多个 Agent 往往会陷入无休止的多向讨论中，不仅得不出清晰的结论，还会白白空耗宝贵的 GPU 算力资源。因此，经过几乎所有成功项目的验证，Meta 坚决采用了带有父级编排器（Parent Orchestrator）的分层 Agent 架构（Hierarchical Agent Architecture）。编排器在整个系统中扮演大脑角色，负责根据可靠性目标对下层子 Agent 进行评估、指导与协调。在网络故障检测与根因定位的具体实践中，系统持续监控网络 KPI 并触发告警至跨域 AI 编排器。该编排器随后向数据中心、骨干网及边缘网络的领域专家下发指令，领域专家进一步指派专项子 Agent 展开并发调查。编排器汇总各领域的反馈证据，进行综合研判并得出最终结论，最终自动生成详尽的 RCA 报告呈报给值班工程师。展望未来，编排器还将在内置安全防护的前提下，自主生成修复方案并自动下发执行。 ```mermaid flowchart TD accTitle: Hierarchical AI Agent Architecture for Incident RCA accDescr: An architecture diagram showing network KPI alerts feeding into a cross domain AI orchestrator, which coordinates domain expert agents across DC, Backbone, and Edge networks to generate RCA reports. subgraph Monitoring ["监控与告警层"] KPI["网络 KPI 持续监控"] --> Alert["异常告警生成"] end subgraph CentralBrain ["中央决策大脑"] Orch["跨域 AI 编排器 (AI Cross-Domain Orchestrator)"] end subgraph DomainExperts ["领域专家 Agent (分管宏观领域)"] DC["数据中心领域专家 (DC Expert)"] BB["骨干网领域专家 (Backbone Expert)"] Edge["边缘网络领域专家 (Edge Expert)"] end subgraph SubAgents ["专项执行子 Agent"] DCS["DC 诊断子 Agent"] BBS["骨干网诊断子 Agent"] EdgeS["边缘分析子 Agent"] end subgraph OutputAction ["决策输出与闭环"] RCA["生成自动化 RCA 根因报告"] Remediation["自动生成修复方案并执行缓解 (带有安全防护的安全闭环)"] end Alert --> Orch Orch <--> DC Orch <--> BB Orch <--> Edge DC <--> DCS BB <--> BBS Edge <--> EdgeS Orch --> RCA Orch -.-> Remediation ``` --- ### [08:01](https://www.youtube.com/watch?v=1hj3ys-rE-c&t=481s) 性能攻防与网络运营成本优化的探索除了保障可靠性，分层 AI Agent 体系同样被深度应用于网络性能与资源效率的治理。在网络性能领域，系统构建了“攻防兼备”的两大核心能力： 1. **防御端（Defense）**：利用 CAN 逻辑（CAN logic）与自学习能力，实现对性能劣化与回归（Regression）的敏锐检测，第一时间阻断性能恶化。 2. **进攻端（Offense）**：通过引导式学习（Guided Learning）主动发掘网络性能提升的突破点，实现前瞻性的性能调优。在网络成本与资源效率方面，系统致力于构建能够主动巡视全网海量生产流量的 Agent，通过预设的领域专项策略（Recipes），在容量规划、出向路由和 GPU 算力分布等维度挖掘降本增效空间。YuLing Chen 列举了这些效率 Agent 目前正在分析的一系列典型业务命题： - **边缘流量调度**：当前的边缘路由选择与负载均衡策略是否达到了全局最优？ - **海缆使用成本**：在存在更经济的网络陆缆或其他低成本路径时，系统是否存在过度使用高昂海底光缆（Subsea Cables）的情况？ - **画质码率与终端匹配**：在低端设备占据主导且高清普及率较低的特定发展中国家市场，维持极高规格的视频码率是否真正必要？ - **算力负载均衡**：跨数据中心的跨集群 GPU 利用率是否实现了真正的均衡与高效协同？ --- ### [09:29](https://www.youtube.com/watch?v=1hj3ys-rE-c&t=569s) 落地核心挑战与应对：幻觉抑制、冷启动与原生护栏在将 AI4NI（AI for Network Infrastructure）引入运行时的探索中，团队总结出了当前面临的三大核心技术挑战及应对方案：首先是 **AI 模型的幻觉与不确定性（Hallucinations）**。在自然语言交互中，向模型提问多次往往会得到不同的答案；而将 Agent 应用于严格的网络运行时控制时，输出的不一致或偶发随机性是绝对不可接受的。为了解决这一问题，Meta 除了在数据源头做好高标准的数据清洗并采用精准的 Prompt 工程外，正在重点探索 **LLM-as-a-Judge（大模型充当裁判）** 机制：针对特定的网络运维任务制定明确的评估基准细则（Rubric），让模型运行多个独立的执行轮次，随后由 LLM 裁判对照细则进行客观打分与评估，从中筛选出最严谨、最准确的执行结果。其次是 **应对未知故障类型与未曾见过的场景（Novel Incident Types）**。生产网络中总会出现历史数据从未记录过的边缘异常，为此团队正在搭建能够捕获运行时人类反馈（Human-in-the-loop）的自我进化流水线，将工程师现场处理新故障时的干预策略沉淀为学习素材，使系统具备持续学习与知识自愈能力。最后是 **AI 权限与治理风险管理（Management Risks）**。随着 Agent 自治能力从单纯的分析建议迈向实机变更，必须杜绝 AI 操作对生产环境引入隐私违规、安全漏洞或引发二次事故的风险。Meta 正在全力构建 AI 安全护栏（AI Guardrails），并将其以原生（Native）的方式直接内嵌到网络数据管道与基础设施的基础代码库中。演讲最后，YuLing Chen 向业界同行发出呼吁，邀请大家共同参与 Meta Net Infra 的 AI4NI 探索征程。 ## Notable quotes > [00:04](https://www.youtube.com/watch?v=1hj3ys-rE-c&t=4s) 任何运行时故障都会迅速演变为巨大的“爆炸半径”（blast radius）。 — YuLing Chen > [00:04](https://www.youtube.com/watch?v=1hj3ys-rE-c&t=4s) 在一个恶劣的阴雨天，网络突发严重故障导致网站全线瘫痪，Meta 内部会紧急组建一个多达上百名工程师的跨部门“作战室”（War Room）。 — YuLing Chen > [00:04](https://www.youtube.com/watch?v=1hj3ys-rE-c&t=4s) 这种手工作业流程不仅繁琐且极其令人心力交瘁，更为企业的业务连续性带来了巨大的潜在风险。 — YuLing Chen > [03:15](https://www.youtube.com/watch?v=1hj3ys-rE-c&t=195s) 并非只有一个全能的单体 Agent，而是由成百上千甚至数万个 AI Agent 协同运作，共同守护运行时网络。 — YuLing Chen > [03:15](https://www.youtube.com/watch?v=1hj3ys-rE-c&t=195s) 在组织架构上，这些 Agent 的分工映射了人类工程师团队的组织模式。 — YuLing Chen > [05:56](https://www.youtube.com/watch?v=1hj3ys-rE-c&t=356s) AI Agent 本质上可以非常健谈（talkative）。 — YuLing Chen > [05:56](https://www.youtube.com/watch?v=1hj3ys-rE-c&t=356s) 在完全扁平的网状架构中，缺乏中央决策者和仲裁机制（tiebreaker），多个 Agent 往往会陷入无休止的多向讨论中，不仅得不出清晰的结论，还会白白空耗宝贵的 GPU 算力资源。 — YuLing Chen > [05:56](https://www.youtube.com/watch?v=1hj3ys-rE-c&t=356s) 编排器在整个系统中扮演大脑角色，负责根据可靠性目标对下层子 Agent 进行评估、指导与协调。 — YuLing Chen > [08:01](https://www.youtube.com/watch?v=1hj3ys-rE-c&t=481s) 通过引导式学习（Guided Learning）主动发掘网络性能提升的突破点，实现前瞻性的性能调优。 — YuLing Chen > [09:29](https://www.youtube.com/watch?v=1hj3ys-rE-c&t=569s) 将 Agent 应用于严格的网络运行时控制时，输出的不一致或偶发随机性是绝对不可接受的。 — YuLing Chen > [09:29](https://www.youtube.com/watch?v=1hj3ys-rE-c&t=569s) 必须杜绝 AI 操作对生产环境引入隐私违规、安全漏洞或引发二次事故的风险。 — YuLing Chen ## Mentioned YuLing Chen, Meta, MetaNet Infra, Instagram, @Scale, Santa Clara Convention Center (SCCC), NIA AI, Cap Genie AI Agents, NetIngram, BGP, LLM-as-a-Judge, AI4NI ## Video description YuLing Chen presents her Lighting Talk "AI-Native Network Operation" live at the Santa Clara Convention Center. When operating a large-scale production network serving billions of users, we face daily challenges in running network infrastructure reliably and efficiently. AI4NI @Runtime tackles these challenges across three dimensions: reliability, performance, and efficiency. At Meta, we’ve begun developing AI agents in each area and have seen promising early results. In this talk, we’ll share our solution architecture, example agentic systems, and initial findings from production trials, along with the key technical challenges and our planned directions to address them. Learn more about @Scale here: atscaleconference.com [Source](https://www.youtube.com/watch?v=1hj3ys-rE-c)

Source quotes

[0:04] YuLing Chen: 任何运行时故障都会迅速演变为巨大的“爆炸半径”（blast radius）。 [0:04] YuLing Chen: 在一个恶劣的阴雨天，网络突发严重故障导致网站全线瘫痪，Meta 内部会紧急组建一个多达上百名工程师的跨部门“作战室”（War Room）。 [0:04] YuLing Chen: 这种手工作业流程不仅繁琐且极其令人心力交瘁，更为企业的业务连续性带来了巨大的潜在风险。 [3:15] YuLing Chen: 并非只有一个全能的单体 Agent，而是由成百上千甚至数万个 AI Agent 协同运作，共同守护运行时网络。 [3:15] YuLing Chen: 在组织架构上，这些 Agent 的分工映射了人类工程师团队的组织模式。 [5:56] YuLing Chen: AI Agent 本质上可以非常健谈（talkative）。 [5:56] YuLing Chen: 在完全扁平的网状架构中，缺乏中央决策者和仲裁机制（tiebreaker），多个 Agent 往往会陷入无休止的多向讨论中，不仅得不出清晰的结论，还会白白空耗宝贵的 GPU 算力资源。 [5:56] YuLing Chen: 编排器在整个系统中扮演大脑角色，负责根据可靠性目标对下层子 Agent 进行评估、指导与协调。 [8:01] YuLing Chen: 通过引导式学习（Guided Learning）主动发掘网络性能提升的突破点，实现前瞻性的性能调优。 [9:29] YuLing Chen: 将 Agent 应用于严格的网络运行时控制时，输出的不一致或偶发随机性是绝对不可接受的。 [9:29] YuLing Chen: 必须杜绝 AI 操作对生产环境引入隐私违规、安全漏洞或引发二次事故的风险。

Source quotes

[0:04] YuLing Chen: When reliability incidents occur at this scale, they trigger severe, wide-reaching blast radiuses that immediately impact core services. [0:04] YuLing Chen: In one instance, an engineer pushed a faulty change that disrupted data center job scheduling, leading to unavailable network services, a drop in user egress, and complete unreachability of the main Meta website. [0:04] YuLing Chen: A few years prior, a regional data center overload triggered a cascading cross-data center failure that caused site-wide outages lasting several hours. [0:04] YuLing Chen: The top priority for network runtime operations is maintaining enhanced, 24/7 reliability while simultaneously optimizing speed and driving down infrastructure costs. [0:24] YuLing Chen: Meta assembles a real-time "war room" bringing together over 100 engineers from various cross-functional organizations. [0:24] YuLing Chen: Rather than relying on a single monolithic model, Meta envisions hundreds of thousands of coordinated AI agents operating in runtime. [4:49] YuLing Chen: In a mesh topology, the lack of a central decision-maker or tiebreaker leads to "talkative" agents communicating endlessly in loops without reaching actionable conclusions, wasting expensive CPU and GPU compute resources. [7:58] YuLing Chen: Is our edge routing and load balancing strategy optimal? [7:58] YuLing Chen: Are we overusing subsea cables when cheaper terrestrial network paths are available? [7:58] YuLing Chen: Is high video bitrate really necessary in markets where low-end devices dominate and HD adoption is low? [7:58] YuLing Chen: Is GPU utilization balanced across data centers? [9:29] YuLing Chen: To counter this beyond prompt engineering and clean data pipelines, Meta employs an "LLM-as-a-Judge" architecture. [9:29] YuLing Chen: Meta builds native AI guardrails directly into its network data and Infrastructure-as-Code (IaC) repositories to enforce strict safety boundaries before any change touches live traffic.

Original transcript

[0:04] So far we have talked about AI for NI [0:07] across network planning, design and [0:10] delivery phases. What about runtime? [0:13] My name is Yuling Chen and I'm a network [0:16] production engineer at Meta. Over my [0:18] eight years working in MetaNet Infra, [0:22] I've seen running network at scale comes [0:24] with significant challenges. Our network [0:27] infrastructure supports billions of [0:29] users. When network reliability [0:31] incidents happen, it they could lead [0:34] quickly lead to a large blast radius. [0:38] For example, in a recent case, one of [0:41] our engineers pushed a bad change that [0:44] caused [0:46] data center job scheduling failures. As [0:49] a result, network services became [0:52] unavailable. User egress dropped and our [0:56] meta website could not be reached. [0:59] In another example, a bad configuration [1:02] change caused network prefix overload [1:05] and BGP crash, ultimately dropping user [1:09] traffic. [1:11] A few years ago, an incident was [1:13] triggered by a regional data center [1:15] overload leading to large cross data [1:19] center cascading failures causing site [1:23] outages that lasted more than a few [1:25] hours. [1:27] All these network reliability incident [1:29] could cause meta website down, user [1:33] frustration, business impact and revenue [1:36] loss. [1:39] As we can see, enhanced network [1:41] reliability around the clock is the [1:44] number one priority for network at run. [1:47] At the same time, we must deliver [1:49] network content fast. Whenever latency [1:52] increases, our customers such as [1:55] Instagram users experience frustration [1:58] and we have clear metrics that quantify [2:00] the degraded user experience for us to [2:03] act on. [2:05] In addition, we also need to drive [2:07] infrastructure cost savings when we [2:09] deliver content reliably and with high [2:12] performance. The question is, can we [2:15] leverage AI to operate network [2:18] infrastructure more reliably, faster, [2:20] and more efficiently? [2:24] As of today, network run is still [2:26] operated in a manual fashion. Let's take [2:29] a look at network reliability. [2:32] Assuming just imagine today is a rainy [2:34] day and we're hitting a severe network [2:37] incident. The entire meta website is [2:40] down. What happens behind the scene is [2:42] that we create a war room inside Meta [2:45] where more than 100 engineers getting [2:47] together into this war room from [2:50] different organizations and to mitigate [2:52] this issue and we manually run queries [2:54] on a large number of data sets and try [2:57] to figure out the root causes and chat [3:00] about potential solutions and under [3:02] pressure to resolve the issues as fast [3:05] as we can. This is a very tedious, [3:08] stressful process which also creates [3:11] risks to meta business. [3:15] Using AI, we are developing an agentic [3:18] system that is made of many specialized [3:20] AI agents that collaborate across [3:23] incident life cycle. These agents [3:26] understands network topology and have [3:28] the knowledge and skills that mirror [3:30] human expertise in network reliability. [3:34] We envision not just the one agent but [3:37] hundreds of thousands of AI agents [3:39] working together in a coordinated [3:41] fashion to ensure network reliability at [3:44] runtime. Organizationally, these AI [3:48] agents align with how human engineers [3:50] are structured. These agents spend data [3:52] center backbone and edge networks. We [3:56] also categorize those AI agents as [3:58] domain experts and also individual AI [4:01] engineers where domain experts hold [4:03] broader knowledge and individual AI [4:05] engineer excel at resolving domain [4:08] specific issues [4:10] from reliability incident life cycle [4:12] point of view. These agents provide [4:14] capabilities in detection, root cause [4:17] analysis, mitigation, repair and [4:20] prevention with cross domain [4:22] orchestration capabilities on data [4:24] center backbone and edge networks. [4:28] Such a system requires time to develop [4:31] and we envision a gradual adoption of AI [4:34] agents throughout incident life cycle. [4:36] They will evolve from L1 L2 human [4:40] assistance to L3 L4 autonomous [4:43] operations such as mitigation and repair [4:46] over the next few years. [4:49] Where are we today? Across metaanet [4:52] infra we have seen promising reliability [4:54] gains from agentic automation AIdriven [4:58] agendic workflows across data center [5:01] backbone and edge network are now [5:04] delivering production impact in incident [5:07] detection root cause analysis and risk [5:10] evaluation. [5:12] These workflows don't just replace [5:14] manual effort. They often exceed human [5:17] performance in key scenarios. [5:20] For example, NIA AI workflow detected [5:23] severe production incidents that [5:24] standard monitoring systems were unable [5:27] to discover. [5:29] For another example, Cap Genie AI agents [5:32] are capable of scanning large data sets [5:34] across meta and evaluate egress risks to [5:37] e edge capacity using impressive [5:40] visualization of map views. [5:43] Alongside these individual agentic [5:45] automation efforts, we're building a [5:47] network knowledge base called net ingram [5:49] to support AI agent self-improvement [5:52] across net infra. [5:56] Okay. Now you may ask what agentic [5:59] architecture should we use? Should it be [6:01] a mesh agent network? Can people guess? [6:08] The answer is no. In a mash network, [6:10] there's a lack of central um central [6:14] decision maker or tiebreaker. To our ex [6:17] experience, AI agents can be very [6:19] talkative. [6:20] If we use a mesh agent network, we could [6:24] end up with AI agents talking to each [6:26] other endlessly and without clear [6:29] conclusions and also wasting C GPU [6:33] resources. [6:35] Instead, a hierarchical agent [6:37] architecture with parent orchestrations [6:40] orchestrators has proven to be effective [6:43] across almost all of our successful [6:46] projects. [6:48] The orchestrators play a critical role. [6:50] They evaluate, guide, and coordinate sub [6:53] aents to achieve reliability goals. [6:55] Whether that involves automated root [6:57] cause analysis, risk evaluation, impact [7:00] analysis, mitigation, or repair. [7:04] Here is an example architecture for [7:07] network incident detection and root [7:09] causing. What we do is we continuously [7:12] monitor network KPIs and then generate [7:15] alerts and then send to AI cross domain [7:17] orchestrator which is the brain of the [7:20] entire architecture. [7:22] This AI orchestrator communicates with [7:25] domain experts in data center backbone [7:28] and edge which internally use sub aents [7:31] to look into the problems. [7:34] The orchestrator then collect outputs [7:36] from these domain experts, aggregates [7:38] the evidence, commas comes up with the [7:41] final conclusions and generates a root [7:43] cause analysis report to human [7:45] engineers. [7:47] Looking ahead, the system will also [7:50] propose fixed plans with the right [7:52] safeguards and automatically execute [7:54] mitigations and repairs. [7:58] So far we have talked a lot about using [8:01] AI in network reliability. What about [8:03] network performance? [8:05] For network performance, we enable a [8:07] large number of AI agents in a [8:09] hierarchical architecture to achieve two [8:11] capabilities. The first is using CAN [8:15] logic and self-learning for performance [8:17] regression detection. We also call it [8:19] defense. [8:21] The second is using guided learning to [8:23] discover performance optimization [8:25] opportunities. We also call it offense [8:28] or performance enhancements. [8:31] For network efficiency, we're exploring [8:33] the use of AI agents to proactively [8:36] identify efficiency opportunities across [8:38] meta network infrastructure. The goal is [8:41] to build AI agents that can scan large [8:44] volumes of meta production traffic data [8:47] and apply guided domain specific recipes [8:50] in areas like capacity, egress, and GPU [8:54] efficiency. Here are some example [8:56] questions these agents could help to [8:58] answer. Is our edge routing and load [9:01] balancing strategy optimal? Are we [9:04] overusing subc cables when cheaper [9:07] network paths are available? Is high [9:10] video bit rate really necessary in [9:12] markets where low-end devices dominate [9:15] and HD adoption is low? Is GPU [9:18] utilization balanced across data [9:20] centers? These explorations can help us [9:25] to discover opportunities for network [9:26] efficiency. [9:29] Okay, we have learned a lot about [9:31] reliability, performance, and [9:33] efficiency. What challenges have we [9:35] encountered? As we adopt AI for NI [9:38] network runtime, we are facing at least [9:41] three key challenges. [9:43] The first is AI hallucinations. I [9:46] believe everyone has the experience. [9:48] When you ask a question to an AI, you [9:51] probably get multiple times, you [9:53] probably get different answers. [9:55] Similarly, when we ask an AI agent to [9:58] execute network runtime function, we may [10:01] see inconsistent or random results from [10:03] time to time. [10:05] Besides providing the clean data to AI [10:07] agents with effective prompt [10:09] engineering, we are exploring an LLM as [10:12] judge approach. [10:14] We run multi it multiple iterations of [10:17] AI network functions using a defined [10:19] rubric. Then we ask the LLM judge to [10:23] evaluate each session against the rubric [10:26] and select the most accurate output as [10:28] the result. [10:31] The second challenge is handling novel [10:33] incident types or never seen before [10:35] scenarios. We are exploring [10:38] selfimprovement [10:39] pipelines that incorporate human [10:41] feedback at runtime so the system can [10:44] continuously learn and improve. [10:47] The third challenge is management risks [10:49] as AI capability increases. For example, [10:52] we we need to minimize privacy, security [10:55] and reliability risks they may introduce [10:58] to our production network. [11:00] We are developing AI guardrails and [11:03] integrating them natively into our [11:05] network data and infrastructure code [11:07] bases. This way we can reduce AI risk to [11:11] our network production system. [11:14] Okay, this concludes our AI for NI [11:16] session on network planning, design, [11:19] delivery and runtime. As you've seen, [11:21] there are so many exciting opportunities [11:23] for us to work together. Join us for an [11:26] AI for NI exploration journey at Meta [11:29] Net Infra. Thanks everyone.