Meta Replaces Engineering War Rooms With Autonomous Network AI

@Scale

Meta is deploying hierarchical AI agent systems across its global network infrastructure to automate incident response and phase out hundred-person manual war rooms.

AI-assisted summary of Lightning Talk: AI-Native Network Operation | Yuling Chen, Meta — original source. Check the original for full context.

Source published: 2026-08-27

Detailed notes

## Background Published on YouTube by @Scale on August 27, 2026, this presentation features Yuling Chen from Meta speaking at an @Scale conference. The talk focuses on AI4NI @Runtime, Meta's solution architecture and multi-agent AI framework designed to manage large-scale network infrastructure across reliability, performance, and efficiency.

## TL;DR Yuling Chen outlines how Meta is transitioning from high-stress, manual incident response war rooms to hierarchical AI multi-agent systems for network operations. The talk details early production victories—such as AI workflows detecting outages legacy tools missed—and explains why Meta favors centralized parent orchestrators over unpredictable mesh agent networks. Additionally, Chen highlights performance and cost efficiency optimizations, concluding with Meta's strategies for overcoming AI hallucinations, novel outage scenarios, and operational safety risks using native guardrails.

## Atmosphere & style The presentation carries an authoritative, pragmatic, and technical tone grounded in real-world infrastructure engineering. Chen speaks with a calm, seasoned perspective as an insider who understands the extreme pressure of high-stakes network downtime. The narrative builds a compelling arc from chaotic 'rainy day' operational crises to a structured, reassuring vision of coordinated, self-improving AI systems.

## Key takeaways - [00:05](https://www.youtube.com/watch?v=eQ10SOJ8GB4&t=5s) Running global network infrastructure serving billions of users makes 24/7 reliability the top operational priority, followed by low latency and cost efficiency. - [00:05](https://www.youtube.com/watch?v=eQ10SOJ8GB4&t=5s) Production incidents carry a massive blast radius, where minor bad changes or configuration overloads cause site-wide downtime and major business revenue loss. - [02:09](https://www.youtube.com/watch?v=eQ10SOJ8GB4&t=129s) Meta is shifting from manual war rooms with 100+ engineers toward automated, multi-agent AI systems across the entire incident lifecycle. - [02:09](https://www.youtube.com/watch?v=eQ10SOJ8GB4&t=129s) Meta targets deploying hundreds of thousands of coordinated AI agents organized into domain experts and individual AI engineer sub-agents. - [04:37](https://www.youtube.com/watch?v=eQ10SOJ8GB4&t=277s) Meta firmly rejects mesh agent topologies due to unbounded agent chatter, lack of decision-making, and high GPU resource waste, favoring a hierarchical agent architecture. - [04:37](https://www.youtube.com/watch?v=eQ10SOJ8GB4&t=277s) The central AI orchestrator acts as the system's brain by processing KPI alerts, directing sub-agents, aggregating evidence, and generating RCA reports. - [04:37](https://www.youtube.com/watch?v=eQ10SOJ8GB4&t=277s) Early production workflows like NIA have successfully identified critical incidents that legacy monitoring and telemetry systems completely missed. - [07:34](https://www.youtube.com/watch?v=eQ10SOJ8GB4&t=454s) AI agent systems are expanding beyond reliability to performance via defensive regression checks and offensive latency optimization. - [07:34](https://www.youtube.com/watch?v=eQ10SOJ8GB4&t=454s) Efficiency agents analyze traffic data to optimize subsea cable usage, regional video bitrates, and cluster GPU balancing. - [09:08](https://www.youtube.com/watch?v=eQ10SOJ8GB4&t=548s) To combat non-deterministic AI hallucinations, Meta implements an LLM-as-a-Judge framework evaluating parallel runs against strict rubrics. - [09:08](https://www.youtube.com/watch?v=eQ10SOJ8GB4&t=548s) Unprecedented network outages are addressed through self-improving feedback pipelines that incorporate real-time human engineer input during active incidents. - [09:08](https://www.youtube.com/watch?v=eQ10SOJ8GB4&t=548s) Native safety guardrails are embedded directly into infrastructure codebases and data pipelines to limit the blast radius of automated AI actions.

## Detailed walkthrough ### [00:05](https://www.youtube.com/watch?v=eQ10SOJ8GB4&t=5s) The High-Stakes Reality of Running Network at Scale

Yuling Chen, a Network Production Engineer at Meta with eight years of experience in network infrastructure, shifts the focus of the talk to the operational "runtime" phase. Operating network infrastructure at Meta's immense scale comes with severe daily operational challenges because the network serves billions of active users globally. When reliability incidents occur in production, their blast radius expands almost instantaneously, causing massive disruption.

Chen illustrates this danger with three concrete real-world incidents: * In one recent case, an engineer deployed a bad change that unintentionally disrupted data center drop scheduling, causing critical internal network services to become unavailable and rendering external user-facing websites unreachable. * In a separate incident, a network configuration change triggered a prefix overload that crashed BGP (Border Gateway Protocol), dropping active user traffic. * In a third severe outage from a few years ago, a site-wide issue ("SEV") was triggered by a regional data center overload, which cascaded into multi-hour site outages.

These failures directly lead to site downtime across Meta's platforms, user frustration, major business operational impacts, and severe revenue loss. Consequently, maintaining 24/7 network reliability remains the single top priority for network runtime operations. Beyond uptime, Meta must deliver content rapidly—because elevated latency directly degrades user experience for applications like Instagram—while simultaneously driving down infrastructure cost.

- **Key Metrics & Priorities:** Reliability is the #1 priority, followed by low-latency performance and cost efficiency. - **Incident Impacts:** Outages degrade critical user metrics, disrupt external web traffic, cause business revenue loss, and spark global user frustration.

---

[02:09](https://www.youtube.com/watch?v=eQ10SOJ8GB4&t=129s) Moving from War Rooms to Hierarchical AI Multi-Agent Systems

Currently, network runtime management relies heavily on manual intervention. Chen describes the chaotic reality of severe reliability incidents through a "rainy day" scenario where the entire Meta platform goes down. When this happens, Meta forms a physical and virtual "war room" where over 100 engineers from various organizations assemble as "firefighters" to mitigate the crisis. Engineers are forced to manually sift through dozens of massive datasets under immense pressure to isolate root causes and debate potential fixes—a process Chen characterizes as extremely tedious, stressful, and risky for the business.

To eliminate this manual friction, Meta is building agentic systems composed of specialized AI agents that collaborate across the entire incident lifecycle. These agents possess intrinsic knowledge of network topology and mirror human engineering expertise across data center, backbone, and edge networks.

``` ┌─────────────────────────────────┐ │ AI Cross-Domain Orchestrator │ │ (Central Brain / Decision) │ └────────────────┬────────────────┘ │ ┌─────────────────────────┼─────────────────────────┐ ▼ ▼ ▼ ┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐ │ Data Center │ │ Backbone │ │ Edge Network │ │ Domain Expert │ │ Domain Expert │ │ Domain Expert │ └────────┬────────┘ └────────┬────────┘ └────────┬────────┘ │ │ │ ┌──────┴──────┐ ┌──────┴──────┐ ┌──────┴──────┐ ▼ ▼ ▼ ▼ ▼ ▼ [Sub-Agents] [Sub-Agents] [Sub-Agents] [Sub-Agents] [Sub-Agents] [Sub-Agents] ```

Meta envisions deploying hundreds of thousands of coordinated AI agents operating across two primary organizational tiers: * **Domain Experts:** Agents holding broad architectural context across data center, backbone, or edge domains. * **Individual AI Engineers:** Specialized sub-agents executing targeted tasks to resolve domain-specific issues.

Across the lifecycle, these agents cover detection, root cause analysis (RCA), mitigation, repair, and prevention. Operationally, Meta plans a gradual phase-in over the coming years, evolving from initial L1/L2 assistive automation to fully autonomous L3/L4 operations.

- **Incident Response War Rooms:** 100+ engineers currently respond manually to major outages. - **Agentic Scale:** Target of hundreds of thousands of specialized AI agents mirroring human engineering structures. - **Autonomous Evolution:** Progressive rollout from L1/L2 assistant workflows to L3/L4 autonomous operation.

---

[04:37](https://www.youtube.com/watch?v=eQ10SOJ8GB4&t=277s) Early Production Victories and the Hierarchical Architecture Choice

Meta is already seeing clear production impact from early AI workflows across data center, backbone, and edge domains, particularly in detection, root cause analysis, and risk evaluation. Rather than merely saving labor, these AI workflows match or exceed human performance in critical scenarios: * **NIA AI Workflow:** An agentic workflow successfully detected severe production incidents that standard telemetry and monitoring systems completely missed. * **CFGI AI Agents:** Agents scan massive internal datasets and knowledge bases to evaluate egress capacity risks at the edge, rendering visual risk assessments directly onto map views. * **NetIngram:** A shared network knowledge base being built across Meta infrastructure to enable ongoing agent self-improvement.

When considering multi-agent system design, Chen addresses a key architectural question: *Should you use a mesh agent network?* Her answer is a firm "no." In a mesh topology where every agent talks to every other agent, the lack of a central decision-maker causes major issues.

> "In our experience, agents can be very talkative. A mesh agent network could end up with agents talking to each other endlessly, leading to unclear conclusions and wasted GPU resources."

Instead, Meta mandates a **hierarchical agent architecture** with parent orchestrators across nearly all successful projects. The parent orchestrator acts as the "brain": it ingests continuous network KPI alerts, directs domain experts (which spawn sub-agents), aggregates evidence, resolves ambiguities, generates final RCA reports for human engineers, and will eventually propose and execute automated mitigation/repair plans under proper safeguards.

- **NIA Workflow Advantage:** Discovered hidden production incidents that legacy monitoring tools failed to catch. - **Mesh Topology Drawbacks:** Unbounded agent chatter, lack of definitive decision-making, and high GPU resource waste. - **Hierarchical Model:** Centralized orchestrators coordinate sub-agents and aggregate conclusions into structured RCA reports.

---

[07:34](https://www.youtube.com/watch?v=eQ10SOJ8GB4&t=454s) Expanding AI Agents to Performance Optimization and Cost Efficiency

Beyond network reliability, Meta is deploying hierarchical AI agent systems to address network performance and infrastructure efficiency:

``` ┌─────────────────────────────────────────────────────────────────────────┐ │ AI Agents for Network Runtime │ ├───────────────────────────┬─────────────────────────────────────────────┤ │ Reliability │ Incident detection, RCA, repair, prevention │ ├───────────────────────────┼─────────────────────────────────────────────┤ │ Performance (Defense) │ CAN logic & self-learning regression checks │ ├───────────────────────────┼─────────────────────────────────────────────┤ │ Performance (Offense) │ Guided learning for speed/latency tweaks │ ├───────────────────────────┼─────────────────────────────────────────────┤ │ Efficiency (Cost & Res) │ Scanning capacity, subsea cables, GPU load │ └───────────────────────────┴─────────────────────────────────────────────┘ ```

#### Performance Operations * **Defense (Regression Detection):** Utilizes CAN logic and self-learning mechanisms to detect and block performance regressions automatically. * **Offense (Optimization):** Uses guided learning to proactively hunt for network speed improvements and latency reductions across user traffic paths.

#### Efficiency Explorations Agents analyze massive volumes of production traffic data against domain-specific rules across capacity, quality of service (QoS), and GPU utilization. The agents answer critical operational questions such as: * Is our edge routing and load balancing strategy optimal? * Are we overusing expensive subsea cables when cheaper alternative paths are available? * Is high video bitrate necessary in emerging markets where low-end devices dominate and HD adoption is low? * Is GPU utilization balanced evenly across data center clusters?

- **Dual Performance Strategy:** Defensive regression detection paired with offensive performance optimization. - **Cost Drivers Evaluated:** Subsea cable utilization, regional video bitrate tuning, and cross-datacenter GPU balancing.

---

[09:08](https://www.youtube.com/watch?v=eQ10SOJ8GB4&t=548s) Key Technical Challenges: Hallucinations, Novel Outages, and Safety

Adopting AI for network runtime infrastructure introduces three major technical hurdles that Meta is actively solving:

``` ┌───────────────────────────┬─────────────────────────────────────────────┐ │ Challenge │ Meta's Architectural Solution │ ├───────────────────────────┼─────────────────────────────────────────────┤ │ 1. AI Hallucinations │ LLM-as-a-Judge with evaluation rubrics │ ├───────────────────────────┼─────────────────────────────────────────────┤ │ 2. Novel Incidents │ Self-improving pipelines + runtime human RL │ ├───────────────────────────┼─────────────────────────────────────────────┤ │ 3. Operational Risks │ Native guardrails in code & data pipelines │ └───────────────────────────┴─────────────────────────────────────────────┘ ```

1. **AI Hallucinations and Non-Determinism:** Asking an LLM the same prompt multiple times often yields inconsistent results. In network engineering, non-deterministic execution of network runtime functions can destabilize production. To fix this, Meta uses an **LLM-as-a-Judge** framework: the system runs multiple iterations of network functions against a strict evaluation rubric, and a judge LLM scores each output to select the single most accurate result. 2. **Handling Novel Outage Scenarios:** Standard models struggle with unseen edge cases or unprecedented network failures. Meta is building **self-improving feedback pipelines** that incorporate real-time human engineering feedback during active incidents so the system continuously learns from zero-day issues. 3. **Managing Blast Radius and Operational Risks:** As AI agents gain more autonomy, they risk introducing privacy, security, or stability issues into production networks. Meta is building native **AI guardrails** directly into its network data pipelines and infrastructure codebases, ensuring automated actions strictly comply with safety policies.

Chen concludes by emphasizing that the transition to AI-native network infrastructure spans planning, design, delivery, and runtime operations, inviting engineers and researchers to collaborate with Meta on this ongoing transformation.

- **LLM-as-a-Judge:** Mitigates non-deterministic agent outputs by evaluating parallel runs against deterministic rubrics. - **Human-in-the-Loop Learning:** Captures runtime engineer feedback to train models on novel incident types. - **Native Safety Integration:** Guardrails embedded directly into infrastructure codebases to prevent accidental network disruption.

## Notable quotes > [05:37](https://www.youtube.com/watch?v=eQ10SOJ8GB4&t=337s) In our experience, agents can be very talkative. A mesh agent network could end up with agents talking to each other endlessly, leading to unclear conclusions and wasted GPU resources. — Yuling Chen

## Mentioned Yuling Chen, Meta, Instagram, Border Gateway Protocol, BGP, NIA AI, CFGI AI, NetIngram, YouTube, @Scale

[Source](https://www.youtube.com/watch?v=eQ10SOJ8GB4)

中文简报

Meta用自主网络AI取代工程作战室 Meta正为其全球网络基础设施部署分层AI智能体系统，以实现事件响应自动化，并逐步淘汰百人规模的人工作战室。 Yuling Chen概述了Meta如何从高压的人工事件响应作战室转向用于网络运营的分层AI多智能体系统。演讲详细介绍了早期的生产实践成效——例如AI工作流检测到了传统工具遗漏的故障——并解释了Meta为何更青睐集中式父编排器而非不可预测的网状智能体网络。此外，Chen还重点介绍了性能与成本效益优化，并在最后阐述了Meta利用原生安全护栏克服AI幻觉、新型故障场景以及运营安全风险的策略。面对服务数十亿用户的大规模生产网络故障风险，Meta 正在从百余名工程师协同的人工应对作战室，转型为覆盖整个故障生命周期的分层多智能体 AI 系统，并规划逐步迈向 L3/L4 级别的自主运维。 (2:09) 在架构设计上，Meta 坚决摒弃了容易导致无休止对话、决策缺失及 GPU 资源浪费的网状（Mesh）智能体拓扑，全面采用由中央 Orchestrator 统一协调证据与决策的分层体系。 (4:37) 早期投入生产的 NIA 等智能体工作流已成功捕获传统监控系统遗漏的严重网络事故，目前该架构正从可靠性延伸至网络性能的主动优化与海缆、GPU 资源等成本效率分析。 (7:34) 针对大模型幻觉与生产安全风险，Meta 采用 LLM-as-a-Judge 评估机制应对非确定性，依托工程师实时反馈解决未知新型故障，并在数据与代码管道中深度内嵌安全护栏。 (9:08)

Reading summary

Yuling Chen, a Network Production Engineer at Meta, argues that operating hyper-scale network runtime infrastructure requires replacing manual engineer war rooms with hierarchical multi-agent AI systems capable of handling incident mitigation, latency optimization, and cost efficiency. Scale Vulnerabilities and the Cost of War Rooms (0:05) Operating network infrastructure that serves billions of global users creates an environment where minor configuration blunders trigger catastrophic, immediate blast radiuses. Chen highlights past production failures to demonstrate this sensitivity, including a bad change that disrupted data center drop scheduling to take down external sites, a configuration error that caused a BGP prefix overload, and a regional overload that cascaded into a multi-hour site-wide outage. Maintaining 24/7 reliability remains the top priority, followed by latency reduction and cost control. Under Meta's current operational setup, severe reliability incidents require forming physical and virtual war rooms of over 100 cross-organizational engineers. Responders must manually inspect dozens of massive datasets under acute pressure to identify root causes and debate remediations—a process Chen characterizes as stressful, tedious, and inherently risky for business continuity. Hierarchical Orchestration Over Mesh Agent Topologies (2:09) To eliminate manual friction, Meta is architecting multi-agent systems designed to scale to hundreds of thousands of specialized AI agents. The framework categorizes agents into domain experts spanning data center, backbone, and edge environments, alongside individual AI engineer sub-agents executing targeted tasks. Meta plans to phase these systems in gradually, transitioning from initial L1/L2 assistive automation to L3/L4 autonomous operations. Architecturally, Meta explicitly rejects mesh agent networks in favor of a centralized hierarchy. Chen notes that unconstrained mesh topologies cause agents to talk endlessly to each other, generating ambiguous conclusions while wasting expensive GPU compute. Instead, a central cross-domain orchestrator acts as the primary decision-maker: it continuously monitors KPI alerts, delegates investigation paths to domain experts, aggregates evidence from sub-agents, and compiles structured root-cause analysis reports. Production Validation and Expansion Beyond Reliability (4:37) Meta has already deployed early AI workflows that match or exceed human baseline performance. Its NIA agentic workflow identified critical production incidents that standard telemetry and monitoring systems missed entirely, while CFGI agents scan internal datasets to evaluate edge egress capacity risks directly onto visual map views. To support ongoing agent self-improvement, Meta is building NetIngram as a shared infrastructure knowledge repository. Beyond incident mitigation, Meta is expanding these hierarchical agents into performance and cost optimization. In performance operations, agents deploy defensive CAN logic to block regressions alongside offensive guided learning to reduce traffic latency. For cost efficiency, agents analyze production data against operational policies to flag overused subsea cables, tune regional video bitrates where low-end devices dominate, and balance GPU compute loads across data centers. Governing Hallucinations, Novel Outages, and Blast Radius (9:08) Integrating AI agents into production runtime introduces three major operational hurdles: model non-determinism, handling unprecedented failure modes, and managing blast radius. Because inconsistent model outputs can destabilize network operations, Meta employs an LLM-as-a-Judge framework that evaluates parallel execution candidates against strict evaluation rubrics to select the single most accurate response. To manage novel, zero-day outage scenarios that standard base models fail to diagnose, Meta incorporates real-time human engineering feedback into self-improving training pipelines during live incidents. Furthermore, to prevent autonomous actions from introducing privacy, security, or network availability regressions, Meta embeds native safety guardrails directly into infrastructure codebases and data pipelines to constrain automated actions. While early agentic workflows have proven effective at detecting hidden failures, Meta's broader vision of autonomous L3/L4 operations depends on resolving operational risks, containing automated blast radiuses, and integrating real-time human feedback across all phases of network infrastructure.

精炼阅读摘要

Meta 网络生产工程师 Yuling Chen 指出，面对数十亿全球用户规模下的严峻运维压力，Meta 正在摒弃依赖百人“作战室”的传统手动应急机制，转向由中央调度器统领的分层 AI 多智能体系统，以此协同解决网络运行时的可靠性保障、性能优化及成本效率难题。摒弃网状拓扑：构建中心化层级架构 (2:09) 陈玉玲（Yuling Chen）指出，Meta 基础设施网络服务全球数十亿用户，生产环境故障的扩散范围极大。过去曾发生过因配置变更失误扰乱数据中心调度而导致内外服务瘫痪、BGP 前缀过载引发流量丢弃，以及区域数据中心超负荷导致持续数小时全站中断等重大事件。目前面临重大危机时，Meta 需在实体或虚拟“作战室”召集超过 100 名工程师，在高压下手动检索分析数十个海量数据集，排查过程耗时且伴随极高的业务风险。针对多智能体协作架构，陈玉玲明确否定了智能体无中心互联的网状拓扑（Mesh）。根据其实践经验，网状架构缺乏统筹中枢，智能体之间极易出现无休止的冗余交互，不仅无法得出清晰结论，还会空耗宝贵的 GPU 算力。因此，Meta 强制采用层级化智能体架构（Hierarchical Architecture），由中央跨域编排器充当全局“大脑”，负责接收连续的 KPI 告警，向数据中心、骨干网及边缘网络的领域专家智能体下达指令，并最终聚合证据，生成确定性的根因分析（RCA）报告以供决策。早期落地成效与自主运维演进路径 (4:37) Meta 在生产环境中已验证了早期智能体工作流的技术可行性。在实际运行中，NIA 智能体工作流成功检测到了传统监控遥测系统完全遗漏的重大故障隐患；CFGI 智能体则通过扫描内部海量数据集与知识库，针对边缘出口容量风险进行评估并直观投射于地图；Meta 还在构建跨基础设施的共享知识库 NetIngram，以支撑智能体持续自我改进。在关键场景下，这些 AI 工作流的分析表现已比肩甚至超越人工水准。根据规划，Meta 未来拟部署数十万个专业 AI 智能体，全面映射人类工程专业结构，涵盖故障检测、根因定位、风险缓解、修复到预防的全生命周期。在落地推进节奏上，陈玉玲表示系统将经历渐进式演变，预计在未来数年内从最初的 L1、L2 辅助自动化阶段，逐步跨越到具备高自主权的 L3、L4 完全自主运维阶段。性能双轨优化与基础设施成本调控 (7:34) 在确保网络稳定性这一首要目标的同时，Meta 将分层智能体系统的职能扩展至性能运营与成本优化领域。在性能维度，团队制定了“攻守兼备”策略：“防御侧”依靠 CAN 逻辑与自学习机制，自动检测并拦截网络性能退化；“进攻侧”则运用引导式学习技术，主动排查全链路流量路径，寻找提升传输速度与压缩延迟的优化空间。在成本与基础设施效率维度，智能体持续分析海量生产流量数据，结合服务质量（QoS）及容量规则评估关键资源配置。典型探索包括：检验边缘路由与负载均衡策略的最优性；评估是否过度使用了高成本的海底光缆，并适时引导流量走向低成本替代路径；针对以低端设备为主、高清需求较低的新兴市场动态评估视频码率下调空间；以及在不同数据中心集群之间动态均衡 GPU 计算负载，防止算力资源分布不均。治理模型幻觉与生产安全的原生机制 (9:08) 在网络底层运行大语言模型伴随着模型幻觉、未知故障泛化困难及运行安全风险三项核心挑战。针对大语言模型因非确定性可能输出前后不一执行方案的问题，Meta 引入了“LLM-as-a-Judge”评估框架：系统对网络运行函数进行多轮并发执行，并由裁判模型对照既定的严格评估细则进行量化打分，从中遴选出唯一最准确可靠的执行结果，避免非确定性动作危及网络状态。针对缺乏历史数据支撑的新型罕见故障，常规模型常因未见过样本而失效。Meta 构建了自改进反馈管道，在事故处理期间实时整合一线工程师的人工反馈，使系统能在处理零日（Zero-day）故障的过程中完成在线学习。此外，为防止智能体获得自主执行权后造成意外破坏，Meta 将原生 AI 护栏直接内置于网络数据管道和底层基础设施代码中，确保所有自动化操作严格受制于安全合规策略。陈玉玲总结指出，AI 原生网络基础设施的转型是一项系统工程，贯穿网络规划、设计、交付至运行的全过程。尽管层级化调度与安全护栏为现阶段网络自治打下了基础，但要彻底摆脱百人作战室并实现大规模 L3/L4 级完全自主运行，仍有待在模型非确定性控制和新型网络故障应对上取得更深层次的突破。

Detailed notes — zh

## Background 本内容由 YouTube 频道 @Scale 于 2026 年 8 月 27 日发布，属于 @Scale 大会的闪电演讲（Lightning Talk）系列。视频简介指出，Meta 面向全球数十亿用户运行超大规模生产网络，在此背景下推出了 AI4NI @Runtime 方案，从可靠性、性能与效率三大维度切入，分享其底层架构、多 Agent 协作系统、生产环境试运行成果以及核心技术挑战与未来演进方向。 ## TL;DR Meta 网络生产工程资深专家 Yuling Chen 探讨了如何将 Agentic AI 引入超大规模生产网络以取代繁琐的人工作战室排障模式。Meta 明确摒弃了容易陷入冗长低效对话的无中心网状架构，转而构建由父级编排器与各领域专家/子 Agent 组成的分层系统，全面覆盖故障检测、根因分析、缓解修复全生命周期。在兼顾性能攻防监测与海缆、视频码率等成本优化的同时，Meta 进一步提出了引入 LLM-as-a-judge 仲裁消除幻觉、融合运行时人类反馈以应对新型故障，以及在底层基础设施代码中深植安全护栏等严谨应对策略。 ## Atmosphere & style 整场演讲呈现出典型硅谷顶尖科技巨头特有的务实硬核、克制且极具工程实战感的氛围。演讲者以平静而从容的专业姿态，开门见山剖析百人作战室通宵排障的惨痛事故与运维困境，没有浮夸的宏大叙事，而是全程紧扣真实生产环境中的高昂爆炸半径与 GPU 算力开销。随着架构图由浅入深展开，听众能够清晰感受到 Meta 在将前沿大模型落地到命脉级基础设施时，对确定性、安全边界与架构收敛性的深度敬畏与严密工程推演。 ## Bulletin - [03:03](https://www.youtube.com/watch?v=eQ10SOJ8GB4&t=183s) Meta 正在其超大规模生产网络运行时（Runtime）部署由数十万个专业化 AI Agent 组成的协作网络，旨在将人工作战室（War Room）排障模式逐步升级为 L3/L4 级的自动化自治运维。 - [05:47](https://www.youtube.com/watch?v=eQ10SOJ8GB4&t=347s) 实践表明无中心的网状 Agent 架构容易陷入无休止对话并造成昂贵 GPU 浪费，Meta 转向以集中式跨域父编排器为核心的分层架构以保证分析收敛。 - [09:08](https://www.youtube.com/watch?v=eQ10SOJ8GB4&t=548s) 针对 AI 幻觉和执行不一致问题，Meta 采用“LLM 裁判（LLM-as-a-judge）”并发采样并对照量规评分，同时结合嵌入代码库的安全护栏防范全网级风险。 ## Key takeaways - [00:05](https://www.youtube.com/watch?v=eQ10SOJ8GB4&t=5s) 超大规模网络运行时的微小配置或调度失误具有巨大爆炸半径，曾导致 BGP 崩溃、丢包乃至持续数小时的级联宕机。 - [00:05](https://www.youtube.com/watch?v=eQ10SOJ8GB4&t=5s) 传统百人作战室排障模式耗时长、多源数据手动检索繁琐，无法满足全天候高可用与极致性能的双重要求。 - [03:03](https://www.youtube.com/watch?v=eQ10SOJ8GB4&t=183s) Meta 放弃通用大模型单打独斗，转而构建映射人类工程师组织架构的“领域专家”与“专项工程师”多 Agent 协作体系。 - [03:03](https://www.youtube.com/watch?v=eQ10SOJ8GB4&t=183s) AI4NI 系统覆盖检测、根因分析、缓解、修复与预防全生命周期，目前已有 NIA 工作流等成果在生产中检出传统监控遗漏的隐匿故障。 - [05:47](https://www.youtube.com/watch?v=eQ10SOJ8GB4&t=347s) 否定无中心网状架构，因其缺乏仲裁机制且 Agent 过于“健谈”导致算力开销巨大；采用分层编排器架构作为决策中枢。 - [05:47](https://www.youtube.com/watch?v=eQ10SOJ8GB4&t=347s) 分层架构中，跨域编排器接收 KPI 告警，调度数据中心、骨干网与边缘专家子 Agent 深入排查并最终汇总出权威 RCA 诊断报告。 - [07:34](https://www.youtube.com/watch?v=eQ10SOJ8GB4&t=454s) 在网络性能优化上兼顾攻防：利用自学习算法防守退化，通过引导式学习主动扫描潜在加速空间。 - [07:34](https://www.youtube.com/watch?v=eQ10SOJ8GB4&t=454s) 在网络效率层面，Agent 主动排查边缘选路、昂贵海缆低效占用、新兴市场高码率冗余及跨数据中心 GPU 负载不均等成本问题。 - [09:08](https://www.youtube.com/watch?v=eQ10SOJ8GB4&t=548s) 为解决确定性网络操作中的 AI 幻觉，引入基于量规标准（rubric）的 LLM 裁判机制，通过并发多会话评估选择最佳结果。 - [09:08](https://www.youtube.com/watch?v=eQ10SOJ8GB4&t=548s) 针对未曾出现过的新型突发故障，构建自我改进流水线在运行时持续吸收人类工程师的排障反馈。 - [09:08](https://www.youtube.com/watch?v=eQ10SOJ8GB4&t=548s) 将专用 AI 安全护栏直接内嵌到底层网络数据和基础设施代码库，防止高权限 Agent 产生破坏性操作。 ## Detailed walkthrough ### [00:05](https://www.youtube.com/watch?v=eQ10SOJ8GB4&t=5s) 运行时的网络规模挑战与传统排障痛点在网络基础设施（Network Infrastructure, NI）的规划、设计与交付阶段之后，运行阶段（Runtime）面临着截然不同的考验。Meta 拥有支撑全球数十亿用户的超大规模网络基础设施，在这样的体量下，任何网络可靠性故障都会瞬间产生巨大的爆炸半径（blast radius）。演讲者 Yuling Chen 作为在 Meta 网络生产工程部门（Meta Net Infra）工作超过八年的资深工程师，列举了几个真实的典型故障案例：近期曾发生过因工程师推送后台变更而干扰数据中心任务调度（drop scheduling）的事故，导致关键网络服务不可用，用户边缘网站无法访问；另一起事故中，配置变更引发了路由前缀过载（prefix overload）与 BGP 崩溃，造成海量用户流量丢失；数年前甚至出现过区域数据中心过载引发级联失效（cascading failures），致使全站瘫痪数小时的严重事件。这些故障直接导致 Meta 旗下服务宕机，带来恶劣的用户体验、商业损失以及营收下滑。当前应对这类严重故障的手段仍高度依赖人工运作。当遭遇全站宕机的“风雨交加之日”（rainy day），Meta 内部会紧急拉起作战室（War Room），召集跨部门的 100 多名工程师作为“现场消防员”。工程师们在巨大的业务风险与高压环境下，手动翻阅排查海量的数据集，在聊天窗口中反复争论潜在原因与修复方案。这种纯人工协作模式不仅极其繁琐耗时、容易引起疲惫，还会拉长业务恢复周期。面对“全天候保障高可用”、“极速传递内容（如保障 Instagram 用户低延迟）”以及“降低基础设施成本”这三大运行时目标，引入 AI 实现自动化运行成为了必由之路。 - **核心生产故障场景**：后台变更破坏任务调度导致服务不可用、BGP 前缀过载崩溃导致丢包、区域数据中心过载引发持续数小时的级联故障。 - **业务诉求三元组**：全天候网络可靠性（Reliability，首要优先级）、低延迟与极致性能（Performance）、基础设施成本节约（Efficiency）。 - **人工作战室痛点**：百人协同排障效率低、多源数据手动检索繁琐、高压下沟通试错成本极高。 --- ### [03:03](https://www.youtube.com/watch?v=eQ10SOJ8GB4&t=183s) Agentic AI 协作愿景与 Meta 生产落地成果为了打破人工排障的瓶颈，Meta 正在构建一套覆盖故障全生命周期的 Agentic 系统。该系统并非依赖单一的通用模型，而是由数以十万计各司其职的专业化 AI Agent 协同运作。这些 Agent 深度理解网络拓扑结构，具备映射人类专家经验的网络可靠性专业知识。在组织设计上，AI Agent 的分工映射了人类工程师团队的架构，横跨数据中心（Data Center）、骨干网（Backbone）和边缘网络（Edge Network），并明确划分为两类角色：拥有宏观全局视角的“领域专家”（Domain Experts），以及精通特定领域细分技术问题的“独立 AI 工程师”（Individual AI Engineers）。在故障生命周期管理中，这套系统逐步覆盖检测（Detection）、根因分析（RCA）、缓解（Mitigation）、修复（Repair）以及预防（Prevention）。按照规划，AI Agent 团队的自主能力将经历从 L1/L2 辅助运维向未来数年内 L3/L4 高度自治运维的演进。在 Meta 内部的试点测试中，该系统已展现出超越传统监控工具乃至匹配、超越人类表现的能力： 1. **NIA AI 工作流**：在生产环境中精准检测出了传统监控系统未能发现的严重隐匿故障。 2. **CFGI AI Agent**：能够自动扫描 Meta 内部的大规模数据集与网络知识库，在地图可视化界面上评估边缘出站流量激增（egress increase）给容量带来的风险。 3. **NetNGram 知识库**：构建统一的网络基础设施领域知识库，为全网 AI Agent 的自我迭代与持续学习提供底层支撑。 --- ### [05:47](https://www.youtube.com/watch?v=eQ10SOJ8GB4&t=347s) 拒绝网状拓扑：分层编排器架构（Hierarchical Architecture）在探索多智能体协作架构时，Meta 明确否定了无中心的网状结构（Mesh Agent Network）。实践表明，在没有集中决策者或仲裁机制（tiebreaker）的情况下，AI Agent 之间往往表现得“过于健谈”（very talkative），陷入无休止的多轮冗长对话，不仅无法收敛出清晰有效的结论，还会造成极其昂贵的 GPU 算力浪费。因此，Meta 最终采用了以“父级编排器”（Parent Orchestrators）为核心的分层架构（Hierarchical Architecture），并在几乎所有落地成功的项目中验证了其有效性。分层架构中的编排器充当整个系统的“中枢大脑”，负责评估态势、调度任务、指导子 Agent 并综合汇总各方证据，推进从根因定位到自动修复的闭环操作。 ```mermaid flowchart TD accTitle: 网络故障检测与根因分析分层架构 accDescr: 展示指标监控产生告警后输入跨域编排器，编排器调度各网络域专家及子Agent分析，最终汇总生成RCA报告与修复计划 subgraph Monitoring["指标监控层"] A["网络KPI持续监控"] --> B["告警生成并触发"] end subgraph CoreBrain["决策编排中枢"] B --> C["AI跨域编排器<br/>(Cross-Domain Orchestrator)"] end subgraph Domains["领域专家与执行子Agent"] C --> D["数据中心领域专家<br/>(Data Center Expert)"] C --> E["骨干网领域专家<br/>(Backbone Expert)"] C --> F["边缘网络领域专家<br/>(Edge Expert)"] D --> D1["DC 专项子Agent"] E --> E1["骨干网专项子Agent"] F --> F1["边缘专项子Agent"] end subgraph Output["证据汇总与执行层"] D1 --> G["证据聚合与结论归纳"] E1 --> G F1 --> G G --> H["生成工程师 RCA 诊断报告"] H -.-> I["安全护栏校验下的自动缓解与修复"] end ``` 在日常运行中，系统通过连续监控网络关键性能指标（KPI）触发告警，告警直接交由跨域编排器处理。编排器将任务分解派发给数据中心、骨干网与边缘的领域专家模型，后者进一步调遣垂直子 Agent 深入日志与运行状态排查。随后，编排器汇聚各领域分析线索，形成最终的根因分析（RCA）报告呈现给工程师。未来该系统还将接入自动化处置链路，在预设安全护栏（Safeguards）的严密防护下自主执行网络缓解与配置修复。 --- ### [07:34](https://www.youtube.com/watch?v=eQ10SOJ8GB4&t=454s) 性能与效率攻防：Agent 在性能调优与成本优化中的实践针对网络性能（Performance），Meta 依靠分层 Agent 架构实现了“攻防兼备”的两大机制： - **防守机制（Defense）**：利用 CAN（Change Analysis / Control）逻辑结合自学习算法，实现性能退化（Performance Regression）的自主实时监测与告警。 - **进攻机制（Offense）**：通过引导式学习（Guided Learning）主动扫描并挖掘网络潜在的性能优化空间，持续进行性能提速。在网络效率（Efficiency）方面，系统利用 AI Agent 针对 Meta 庞大的生产流量进行海量数据扫描，结合特定领域的优化“配方”（recipes），在容量规划、服务质量（QoS）和 GPU 算力效率等维度主动寻找成本压减机会。Agent 重点解决以下关键工程问题： - **边缘路由与负载均衡**：评估当前边缘节点的选路与负载策略是否处于全局最优状态。 - **海缆资源分配**：排查是否存在已有廉价陆地/替代路径可用，却依然过度消耗昂贵海底光缆（subsea cables）带宽的低效路由。 - **差异化码率供给**：针对低端设备占主导且高清视频普及率极低的新兴市场，评估是否仍然下发了不必要的高码率视频流，从而避免带宽浪费。 - **算力负载均衡**：分析跨数据中心的 GPU 利用率是否平衡，识别算力孤岛与网络瓶颈。 --- ### [09:08](https://www.youtube.com/watch?v=eQ10SOJ8GB4&t=548s) 运行时落地 AI4NI 的三大核心挑战与应对策略在网络运行时引入 AI 尽管前景广阔，但也面临着三个极其严峻的技术挑战： 1. **AI 幻觉与输出不一致性（AI Hallucinations）** 在网络操作这种确定性要求极高的场景下，向 AI 下达相同的网络运行时指令可能会得到随机或不一致的执行结果。为此，Meta 探索了“LLM 裁判”（LLM-as-a-judge）机制：针对特定网络指令设定严格的评估标准量规（rubric），让系统并发运行多次 AI 任务执行会话，最后交由独立的 LLM 裁判对照量规对各会话结果进行评分筛选，输出最具确定性和准确性的最终结果。 2. **未知突发故障处理（Novel Incident Scenarios）** 对于生产环境中从未出现过的新型故障模式或未知拓扑边缘场景，预训练模型往往缺乏先验知识。Meta 正在构建具备自我改进能力的流水线（Self-improving pipelines），在运行时主动集成人类工程师的排障反馈（Human Feedback at Runtime），让 Agent 能够将现场经验吸纳进知识网络，实现持续进化。 3. **模型能力提升带来的生产安全风险（Risk Management & Guardrails）** 随着 Agent 获得的生产权限越来越深，其对生产网络注入的隐私泄露、越权安全以及操作可靠性风险也在成倍增加。Meta 的对策是开发专用的 AI 原生安全护栏（AI Guardrails），并将其深嵌到底层网络数据和基础设施代码库（Infrastructure Codebase）中，实现代码级别的安全策略拦截与硬性约束，严防恶意或错误变更对全球业务造成灾难性影响。 ## Notable quotes > [00:05](https://www.youtube.com/watch?v=eQ10SOJ8GB4&t=5s) 当遭遇全站宕机的“风雨交加之日”（rainy day），Meta 内部会紧急拉起作战室（War Room），召集跨部门的 100 多名工程师作为“现场消防员”。 — Yuling Chen > [00:05](https://www.youtube.com/watch?v=eQ10SOJ8GB4&t=5s) 这种纯人工协作模式不仅极其繁琐耗时、容易引起疲惫，还会拉长业务恢复周期。 — Yuling Chen > [00:05](https://www.youtube.com/watch?v=eQ10SOJ8GB4&t=5s) 面对“全天候保障高可用”、“极速传递内容（如保障 Instagram 用户低延迟）”以及“降低基础设施成本”这三大运行时目标，引入 AI 实现自动化运行成为了必由之路。 — Yuling Chen > [03:03](https://www.youtube.com/watch?v=eQ10SOJ8GB4&t=183s) 该系统并非依赖单一的通用模型，而是由数以十万计各司其职的专业化 AI Agent 协同运作。 — Yuling Chen > [03:03](https://www.youtube.com/watch?v=eQ10SOJ8GB4&t=183s) AI Agent 团队的自主能力将经历从 L1/L2 辅助运维向未来数年内 L3/L4 高度自治运维的演进。 — Yuling Chen > [05:47](https://www.youtube.com/watch?v=eQ10SOJ8GB4&t=347s) 在没有集中决策者或仲裁机制（tiebreaker）的情况下，AI Agent 之间往往表现得“过于健谈”（very talkative），陷入无休止的多轮冗长对话，不仅无法收敛出清晰有效的结论，还会造成极其昂贵的 GPU 算力浪费。 — Yuling Chen > [05:47](https://www.youtube.com/watch?v=eQ10SOJ8GB4&t=347s) 分层架构中的编排器充当整个系统的“中枢大脑”，负责评估态势、调度任务、指导子 Agent 并综合汇总各方证据，推进从根因定位到自动修复的闭环操作。 — Yuling Chen > [07:34](https://www.youtube.com/watch?v=eQ10SOJ8GB4&t=454s) 利用 CAN（Change Analysis / Control）逻辑结合自学习算法，实现性能退化（Performance Regression）的自主实时监测与告警。 — Yuling Chen > [09:08](https://www.youtube.com/watch?v=eQ10SOJ8GB4&t=548s) 在网络操作这种确定性要求极高的场景下，向 AI 下达相同的网络运行时指令可能会得到随机或不一致的执行结果。 — Yuling Chen > [09:08](https://www.youtube.com/watch?v=eQ10SOJ8GB4&t=548s) 开发专用的 AI 原生安全护栏（AI Guardrails），并将其深嵌到底层网络数据和基础设施代码库（Infrastructure Codebase）中，实现代码级别的安全策略拦截与硬性约束，严防恶意或错误变更对全球业务造成灾难性影响。 — Yuling Chen ## Mentioned Yuling Chen, Meta, Meta Net Infra, @Scale, Instagram, BGP, NIA AI, CFGI AI Agent, NetNGram, LLM-as-a-judge ## Video description When operating a large-scale production network serving billions of users, we face daily challenges in running network infrastructure reliably and efficiently. AI4NI @Runtime tackles these challenges across three dimensions: reliability, performance, and efficiency. At Meta, we’ve begun developing AI agents in each area and have seen promising early results. In this talk, we’ll share our solution architecture, example agentic systems, and initial findings from production trials, along with the key technical challenges and our planned directions to address them. Learn more about the @Scale conferences here: https://atscaleconference.com/ [Source](https://www.youtube.com/watch?v=eQ10SOJ8GB4)

Source quotes

[0:05] Yuling Chen: 当遭遇全站宕机的“风雨交加之日”（rainy day），Meta 内部会紧急拉起作战室（War Room），召集跨部门的 100 多名工程师作为“现场消防员”。 [0:05] Yuling Chen: 这种纯人工协作模式不仅极其繁琐耗时、容易引起疲惫，还会拉长业务恢复周期。 [0:05] Yuling Chen: 面对“全天候保障高可用”、“极速传递内容（如保障 Instagram 用户低延迟）”以及“降低基础设施成本”这三大运行时目标，引入 AI 实现自动化运行成为了必由之路。 [3:03] Yuling Chen: 该系统并非依赖单一的通用模型，而是由数以十万计各司其职的专业化 AI Agent 协同运作。 [3:03] Yuling Chen: AI Agent 团队的自主能力将经历从 L1/L2 辅助运维向未来数年内 L3/L4 高度自治运维的演进。 [5:47] Yuling Chen: 在没有集中决策者或仲裁机制（tiebreaker）的情况下，AI Agent 之间往往表现得“过于健谈”（very talkative），陷入无休止的多轮冗长对话，不仅无法收敛出清晰有效的结论，还会造成极其昂贵的 GPU 算力浪费。 [5:47] Yuling Chen: 分层架构中的编排器充当整个系统的“中枢大脑”，负责评估态势、调度任务、指导子 Agent 并综合汇总各方证据，推进从根因定位到自动修复的闭环操作。 [7:34] Yuling Chen: 利用 CAN（Change Analysis / Control）逻辑结合自学习算法，实现性能退化（Performance Regression）的自主实时监测与告警。 [9:08] Yuling Chen: 在网络操作这种确定性要求极高的场景下，向 AI 下达相同的网络运行时指令可能会得到随机或不一致的执行结果。 [9:08] Yuling Chen: 开发专用的 AI 原生安全护栏（AI Guardrails），并将其深嵌到底层网络数据和基础设施代码库（Infrastructure Codebase）中，实现代码级别的安全策略拦截与硬性约束，严防恶意或错误变更对全球业务造成灾难性影响。

Source quotes

[5:37] Yuling Chen: In our experience, agents can be very talkative. A mesh agent network could end up with agents talking to each other endlessly, leading to unclear conclusions and wasted GPU resources.

Original transcript

[0:05] So far we have talked about AI for NI [0:08] across the planning, design and delivery [0:10] phases. What about runtime? My name is [0:14] Yuling Chen and I'm a network production [0:16] engineer at Meta. Over my eight years [0:19] working in meta net infra, I've seen [0:21] running network at scale comes with [0:24] significant challenges. Our network [0:27] infrastructure supports billions of [0:29] users. When network reliability [0:31] incidents happen, they can quickly lead [0:34] to a large blast radius. For example, in [0:38] a recent case, an engineer pushed a back [0:41] change that disrupted data center drop [0:44] scheduling. As a result, critical [0:47] network services became unavailable, [0:49] user eag [0:51] websites could not be reached. In [0:54] another incident, a network [0:56] configuration change caused prefix [0:58] overload and a BGP crash, ultimately [1:01] dropping user traffic. A few years ago, [1:05] a set was triggered by regional data [1:07] center overload leading to cascading [1:10] failures and site outages that lasted [1:13] multiple hours. All these network [1:16] incidents can cause metal website down, [1:18] user frustration, business impact, and [1:22] revenue loss. [1:24] As we can see, enhancing network [1:26] reliability around the clock is the [1:28] number one priority for network at run. [1:32] At the same time, we must deliver [1:34] network content fast. When latency [1:37] increases, customers such as our [1:39] Instagram users experience frustration [1:43] and we have clear metrics that quantify [1:45] this degraded experience for us to act [1:48] on. In addition, we also need to drive [1:52] infrastructure cost savings when we [1:55] deliver content reliably and quickly. [1:59] The question is, can we leverage AI to [2:02] operate network infrastructure more [2:04] reliably, faster, and more efficiently? [2:09] As of today, network at one is mostly [2:12] operated manually. Let's take a look at [2:15] network reliability. [2:17] Assuming today is a rainy day, we are [2:20] hitting a severe network reliability [2:22] incident. The entire Meta website is [2:26] down. What happens behind the scene is [2:28] that we create a war room inside meta [2:32] where more than 100 engineers from [2:34] different organizations get into this [2:37] war room as firefighters to mitigate the [2:40] issue. We manually look into multiple [2:44] large number of data sets try to figure [2:47] out the root causes chat about potential [2:50] solutions and under pressure to resolve [2:52] the issues as fast as possible. [2:55] This is a very tedious stressful process [2:59] which also creates risks to business. [3:03] Using AI, we are developing a agendic [3:06] system that is made of many specialized [3:09] AI agents that collaborate across the [3:12] incident life cycle. These agents [3:15] understand network topology and have the [3:18] knowledge and skills that mirror human [3:20] expertise in network reliability. [3:24] We envision not just one agent but [3:27] hundreds of thousands of AI agents [3:29] working together in a coordinated [3:32] fashion to ensure network reliability at [3:35] one time. Organization wise, AI agents [3:39] align with how human engineers are [3:41] structured. These agents spend data [3:44] center backbone and edge networks. We [3:48] categorize those agents as domain [3:50] experts and individual AI engineers [3:53] where domain experts hold broader [3:56] knowledge and individual AI engineers [3:58] excel at resolving domain specific [4:01] issues [4:03] from reliability incident life cycle [4:05] point of view. These agents provide [4:08] capabilities in detection, root cause [4:10] analysis, mitigation, repair and [4:13] prevention with cross domain [4:15] orchestration capabilities on data [4:17] center back and edge networks. Such a [4:20] system requires time to develop. We [4:23] envision a gradual adoption of AI agents [4:26] throughout the incident life cycles. [4:28] They will evolve from L1, L2 to L3, L4 [4:33] autonomous operations over the next few [4:35] years. [4:37] Where are we today? Across meta network [4:40] infra, we are seeing promising [4:42] reliability gains from agendic [4:44] automation. AIdriven agenda workflows [4:47] across data center, badbomb, and edge [4:50] network are now delivering production [4:52] impact in incident detection, root cause [4:55] analysis, and risk evaluation. [4:59] These workflows don't just replace [5:01] manual effort. They often match or even [5:04] exceed human performance in key [5:06] scenarios. For example, NIA AI workflow [5:10] detected severe production incidents [5:13] that standard monitoring systems were [5:15] unable to discover. [5:18] For another example, CFGI AI agents are [5:21] capable of scanning large data sets and [5:24] also knowledge base across meta and [5:26] evaluate egress increase risks to edge [5:29] capacity using impressive visualizations [5:32] on map view. Alongside these individual [5:36] agendic automation effort, we're [5:38] building a network knowledge base called [5:40] net ingram to support AI agent [5:43] self-improvement across net info. [5:47] Okay. Now you may ask what agentic [5:50] architecture should we use? Should it be [5:52] a mesh agent network? The answer is no. [5:56] In a mesh network, there's a lack of [5:58] central decision maker or tiebreaker. To [6:01] our experience, agents can be very [6:04] talkative. A mesh agent network could [6:06] end up with agents talking to each other [6:08] endlessly, leading to unclear [6:11] conclusions and wasted GPU resources. [6:15] Instead, a hierarchical agent [6:17] architecture with parent orchestrators [6:20] has proven effective across almost all [6:22] of our successful projects. The [6:25] orchestrator play a critical role. They [6:27] evaluate, guide, and coordinate sub [6:30] aents to achieve reliability goals. [6:33] Whether that involves automated root [6:35] cause analysis, risk evaluation, impact [6:38] analysis, mitigation, or repair. [6:43] Here is an example architecture for [6:45] network incident detection and rule [6:47] causing. What we do is we continuously [6:50] monitor network KPIs and generate alerts [6:53] and then send to AI cross domain [6:55] orchestrator which is the brain of the [6:57] entire architecture. This AI [7:00] orchestrator communicates with domain [7:02] experts in data center backbone and edge [7:06] which internally use sub aents to look [7:08] into the problems. The orchestrator then [7:11] collect outputs from these domain [7:13] experts, aggregates the evidence, comes [7:15] up with final conclusions and generate [7:18] root cause analysis RCA report to human [7:21] engineers. [7:23] Looking ahead, the system will also [7:25] propose fixed plans and with the right [7:27] SE safeguards automatically execute [7:30] mitigations and repairs. [7:34] So far we have talked a lot about using [7:36] AI in network reliability. What about [7:38] network performance? For network [7:41] performance, we enable a large number of [7:43] AI agents in a hierarchical architecture [7:46] to achieve two capabilities. [7:49] The first is using CAN logic and [7:52] self-learning for performance regression [7:54] detection. We also call it defense. The [7:57] second is using guided learning to [7:59] discover performance optimization [8:01] opportunities. We also call it offense [8:04] or performance enhancements. [8:07] For network efficiency, we are exploring [8:10] the use of AI agents to proactively [8:13] identify efficiency opportunities across [8:15] meta infrastructure. [8:17] The goal is to build AI agents that can [8:21] scan large volumes of meta production [8:23] traffic data and apply guided domain [8:26] specific recipes in areas like capacity, [8:30] EOS, and GPU efficiency. Here are some [8:34] example question these agents could help [8:36] to answer. [8:38] Is our edge routing and low balancing [8:41] strategy optimal? [8:43] Are we overusing sui cables when cheaper [8:46] paths are available? [8:49] Is high video bit rate necessary in [8:52] markets where low-end devices dominate [8:54] and HD abdoption is low? Is GPU [8:58] utilization balanced across data [9:00] centers? All these explorations help us [9:03] to discover opportunities for network [9:06] efficiency. [9:08] Okay, we've learned a lot about [9:11] reliability, performance, and [9:13] efficiency. What challenges have we [9:16] encountered as we adopt AI for NI in [9:19] network runtime? We are facing at least [9:21] three key challenges. The first is AI [9:25] hallucinations. I believe everyone has [9:28] the experience. Ask an AI the same [9:31] question multiple times and you may get [9:33] different answers. Similarly, when we [9:36] ask an AI agent to execute a network [9:39] runtime function, we may see [9:42] inconsistent or random results from time [9:44] to time. What we are exploring is an LLM [9:48] as judge approach. We run multiple [9:51] iterations of AI network functions using [9:53] a defined rubric. Then we ask the LLM [9:57] judge to evaluate each session against [9:59] that rubric and select the most accurate [10:03] output as the result. The second [10:06] challenge is handling novel incident [10:08] types or neverseen scenarios. [10:12] We are exploring self-improving [10:14] pipelines that incorporate human [10:16] feedback at runtime so the system can [10:19] continuously learn and improve. [10:22] The third one is managing risks as AI [10:26] capability increases. For example, we [10:29] need to minimize privacy, security, and [10:32] reliability risks these agents may [10:35] introduce to our production network. We [10:38] are developing AI guard rails and [10:40] integrating them natively into our [10:43] network data and infrastructure code [10:45] bases. This way, we can reduce AI risks [10:49] to our network production systems. [10:52] Okay, this concludes our AI for NI [10:55] session on network planning, design, [10:58] delivery, and runtime. As you've seen, [11:02] there are so many exciting opportunities [11:05] for us to work together. Join us for an [11:08] AI for ANI exploration journey across [11:11] MetaNet Infra. Thanks everyone.