Meta Deploys AI Agents to Patch Production Bugs and Cut On-Call Drain

@Scale

Autonomous maintenance loops that inject local product context into shared AI agents can resolve live software failures before humans intervene, reclaiming the 50% of engineering bandwidth lost to firefighting.

AI-assisted summary of From On-Call to Always-On | Sumeet Arora from Meta — original source. Check the original for full context.

Source published: 2026-10-30

Key ideas

Meta introduced Autopilot, an agentic engineering platform that converts reactive on-call firefighting into continuous, autonomous software maintenance. (0:05)
Engineering teams at Meta can spend up to 50% of their capacity purely on operational upkeep and bug fixes, starving feature development. (0:05)
To prevent agent drift and hallucination, Autopilot decouples shared agent capabilities from localized product truth encapsulated in standardized 'Product Packs'. (5:43)
Meta evaluates competing agents empirically using the 'Agent Arena' harness, which tests against historical real-world fixes for solve rates, cost, and reliability. (10:24)
Autopilot is deployed in production across Meta, having already landed thousands of code changes autonomously for live bugs and performance regressions. (14:50)

Detailed notes

## Background Published on YouTube by @Scale on October 30, 2026, this talk features Sumeet Arora from Meta discussing how the company automates software maintenance through AI agents. No show notes or page description were provided with the release.

## TL;DR Meta engineering leader Sumeet Arora introduces Autopilot, an agentic platform built to automate the continuous software maintenance workflows that typically consume up to half of an engineering team's capacity. Autopilot bridges both reactive bug tickets and proactive telemetry-driven issue discovery through a unified loop of classification, root-cause investigation, patch generation, and regression verification. To prevent hallucinations and off-target runs across diverse apps like Facebook Feed and Reels, the architecture decouples reusable agent definitions from localized 'Product Packs' that supply repo-specific ground truth. Governed by calibrated security identities and benchmarked against historical fixes in an empirical 'Agent Arena,' the platform has already landed thousands of autonomous production patches, moving human engineers from reactive triage queues to supervisory roles.

## Atmosphere & style The presentation carries the polished, authoritative, and systems-heavy tone of an advanced engineering talk, focusing strictly on production realities rather than speculative AI hype. Sumeet Arora delivers the material with pragmatic confidence, framing the endless treadmill of firefighting as a tractable engineering problem with clear operational trade-offs. The narrative moves deliberately from the acute friction of on-call rotations toward clean architectural solutions, making the viewer feel like an insider observing how hyper-scale engineering infrastructure is quietly being rewired.

## Bulletin - [00:05](https://www.youtube.com/watch?v=DJFcABDi5wE&t=5s) Meta introduced Autopilot, an agentic engineering platform that converts reactive on-call firefighting into continuous, autonomous software maintenance. - [00:05](https://www.youtube.com/watch?v=DJFcABDi5wE&t=5s) Engineering teams at Meta can spend up to 50% of their capacity purely on operational upkeep and bug fixes, starving feature development. - [05:43](https://www.youtube.com/watch?v=DJFcABDi5wE&t=343s) To prevent agent drift and hallucination, Autopilot decouples shared agent capabilities from localized product truth encapsulated in standardized 'Product Packs'. - [10:24](https://www.youtube.com/watch?v=DJFcABDi5wE&t=624s) Meta evaluates competing agents empirically using the 'Agent Arena' harness, which tests against historical real-world fixes for solve rates, cost, and reliability. - [14:50](https://www.youtube.com/watch?v=DJFcABDi5wE&t=890s) Autopilot is deployed in production across Meta, having already landed thousands of code changes autonomously for live bugs and performance regressions.

## Key takeaways - [00:05](https://www.youtube.com/watch?v=DJFcABDi5wE&t=5s) Software maintenance obligations begin immediately after product shipment and scale exponentially with user interactions, often consuming up to 50% of engineering bandwidth. - [00:05](https://www.youtube.com/watch?v=DJFcABDi5wE&t=5s) Maintenance workflows are uniquely well-suited for autonomous agents because problems have bounded scopes, rich telemetry, and safe staging environments with reversible rollbacks. - [03:00](https://www.youtube.com/watch?v=DJFcABDi5wE&t=180s) Maintenance enters through two doors: reactive queues (incidents, bug reports, regressions) and proactive finders that continuously inspect live surfaces and telemetry. - [03:00](https://www.youtube.com/watch?v=DJFcABDi5wE&t=180s) Both reactive and proactive maintenance converge into a single unified lifecycle: classify and triage, investigate, fix and verify, and continuous learning. - [05:43](https://www.youtube.com/watch?v=DJFcABDi5wE&t=343s) Autopilot drives maintenance tasks to one of three concrete outcomes: generating a verified diff patch, escalating with evidence to a human, or closing with no action required. - [05:43](https://www.youtube.com/watch?v=DJFcABDi5wE&t=343s) A central architectural challenge in multi-tenant agent platforms is balancing shared, reusable agent capabilities with local, product-specific truth. - [07:39](https://www.youtube.com/watch?v=DJFcABDi5wE&t=459s) Product Packs package local domain context—including architecture docs, code pointers, triage boundaries, reproduction navigation, and test user credentials—to ground agent execution. - [07:39](https://www.youtube.com/watch?v=DJFcABDi5wE&t=459s) Agent Providers build and maintain vertical-specific agent definitions that can be published to an internal marketplace and combined with any team's Product Pack. - [09:26](https://www.youtube.com/watch?v=DJFcABDi5wE&t=566s) Grounding runs through tight contextual binding ensures agents do not hallucinate, drift down dead ends, or execute unintended actions. - [10:24](https://www.youtube.com/watch?v=DJFcABDi5wE&t=624s) Without empirical evidence, an agent marketplace is merely a 'directory of demos'; the Agent Arena evaluates agents on real historical tasks using solve graders, cost analysis, and reliability benchmarks. - [12:04](https://www.youtube.com/watch?v=DJFcABDi5wE&t=724s) Agent security and permissions must be calibrated between capability and blast radius, utilizing default constrained bot identities alongside scoped custom service users. - [13:02](https://www.youtube.com/watch?v=DJFcABDi5wE&t=782s) Autopilot's platform combines an orchestration state machine, identity controls, evaluation arenas, and a tiered ecosystem ranging from one-click Starter Packs to custom agents. - [14:50](https://www.youtube.com/watch?v=DJFcABDi5wE&t=890s) Deploying autonomous maintenance shifts engineers from manually grinding through ticket queues to supervising agent loops 'on the loop' while thousands of fixes land autonomously.

## Detailed walkthrough ### [00:05](https://www.youtube.com/watch?v=DJFcABDi5wE&t=5s) The Maintenance Problem and the Case for AI Agents

Sumeet Arora introduces the talk "From On-Call to Always-On," presenting an agentic platform developed at Meta that leverages AI agents to handle ongoing product maintenance. The core premise is that shipping a product marks the beginning, not the end, of engineering labor. As soon as code hits production, the maintenance clock starts ticking: teams must continuously monitor for regressions, bugs, and live-site incidents. The traditional engineering workflow follows a demanding cycle of triage, root-cause debugging across affected application components, implementing code repairs, applying patches, and establishing regression guardrails.

At Meta's operational volume, maintenance ceases to be an isolated task and turns into an overwhelming systemic drain. Tiny software defects multiply across billions of interactions, placing non-stop operational pressure on product groups. Arora points out that engineering teams often end up dedicating up to half their working hours purely to firefighting, which severely starves new feature work and roadmap commitments.

Maintenance workflows, however, possess structural characteristics that make them particularly well-suited for autonomous AI agents. Unlike broad, open-ended product ideation, maintenance issues are bounded in scope—confined to specific features or failure modes and backed by rich telemetry, crash logs, and operational evidence. Furthermore, fixes are inherently testable: an agent can stage changes in isolated environments, verify behavioral correctness, roll them back if tests fail, and iteratively attempt alternative solutions. This creates a self-reinforcing feedback loop that enables agents to learn from execution failures without repeating the same mistake.

> “Your maintenance obligation or the maintenance clocks starts as soon as the products get shipped.” — Sumeet Arora

- Teams at Meta scale can spend as much as 50% of their engineering capacity solely addressing operational problems. - Key agent-friendly attributes of maintenance: bounded problem scope, rich telemetry/logs, staging and roll-back reversibility, and closed-loop learning.

---

[03:00](https://www.youtube.com/watch?v=DJFcABDi5wE&t=180s) Reactive vs. Proactive Maintenance

Maintenance tasks arrive at engineering teams through two distinct entry channels: reactive and proactive. The reactive path represents the classic operational model where work begins from an existing incident signal, filed bug, or regression alert in a queue. Here, engineers validate the severity, reconstruct the situational context, route the issue to the correct code owner, investigate the underlying failure, author a fix, verify it, and merge the patch.

Proactive maintenance operates before issues ever reach a human bug tracker. Instead of waiting for users or monitors to report degradation, an autonomous finder agent continuously scans production surfaces. It pulls in inputs such as live surface telemetry (for instance, tracking Facebook Feed load latency) and historic user crash reports, generating evidence-based candidate tasks. This approach uncovers latent bugs, performance bottlenecks, accessibility issues, and reliability risks before they cause widespread user harm.

Despite these differing entry doors, both reactive and proactive flows converge onto a single unified maintenance lifecycle. In both scenarios, issues must be classified, triaged to the correct organizational domain, resolved, verified through automated checks, and fed into a continuous learning loop.

```mermaid flowchart TD accTitle: Unified Maintenance Loop accDescr: Diagram illustrating how reactive and proactive inputs flow through a shared maintenance pipeline to reach three potential outcomes.

subgraph Inputs["Entry Doors"] R["Reactive Queue (Bugs, Incidents, Regressions)"] P["Proactive Finder Agents (Telemetry, Logs, Live Surfaces)"] end

subgraph Engine["Autopilot Platform"] ML["Shared Maintenance Loop (Classify & Triage -> Investigate -> Fix & Verify -> Learn)"] end

subgraph Outcomes["System Outcomes"] O1["Change / Diff Patch"] O2["Escalation to Human"] O3["No Action Required / Close"] end

R --> ML P --> ML ML --> O1 ML --> O2 ML --> O3 ```

- **Reactive domains:** Incident reports, crash spikes, validated functional regressions. - **Proactive domains:** Performance optimization, accessibility standards, reliability tuning, and pre-emptive bug discovery.

---

[05:43](https://www.youtube.com/watch?v=DJFcABDi5wE&t=343s) Autopilot: The Persistent Platform and the Grounded Execution Problem

Meta created "Autopilot" to serve as the unified agentic engine for both reactive and proactive engineering maintenance. Rather than acting as a simple stateless script, Autopilot acts as a persistent automated owner that drives an issue from triage to one of three concrete resolution states: generating a validated diff/change patch, escalating to a human engineer with assembled evidence and specific remediation recommendations, or determining that no action is needed (for example, if the error was transient or resolved by an independent push).

Building this platform across a massive, multi-tenant engineering organization surfaced a core architectural dilemma: shared capability versus local product truth. A single, shared agent cannot possibly understand the idiosyncratic architectural details, nuanced team boundaries, and unique user flows across drastically different surfaces like Facebook Feed, Reels, Threads, or specialized internal AI tooling.

Meta's design solution decouples generic agent capabilities from product-specific context. Reusable agent logic is maintained centrally, but local domain truth is dynamically injected at dispatch time. When shared agent capabilities combine with localized product truth, the agent executes in a grounded, context-aware environment rather than hallucinating or relying on generic debugging heuristics.

> “Shared capability and the local truth which gets attached to that capability leads to a grounded execution.” — Sumeet Arora

---

[07:39](https://www.youtube.com/watch?v=DJFcABDi5wE&t=459s) Product Packs and the Agent Provider Ecosystem

Local product truth is packaged into standardized repositories known as "Product Packs." A Product Pack contains the full operational context required for an autonomous agent to navigate, test, and repair a specific product area without human intervention:

- `knowledge.md` files: Document internal architecture, component ownership, and upstream/downstream service dependencies. - Code pointers: Direct references to critical repositories, modules, and relevant entry points. - Triage rules: Explicit boundary conditions determining whether an incoming ticket belongs to the current team, an adjacent sister team, or an external group. - `navigation.md` files: Step-by-step reproduction instructions, debugging paths, and UI navigation flows. - Test user configurations: Credentials and profile states allowing agents to authenticate into apps and reproduce user-facing behavior safely. - GST config: Team-level project metadata and operational preferences.

On the other side of this architecture are the "Agent Providers"—domain-expert teams that build, maintain, and publish shared agent definitions across verticals like performance optimization, accessibility, and bug fixing. An agent definition bundles systemic instructions, domain plugins, and programmatic skills into an installable capability. These definitions are cataloged in an internal agent marketplace, allowing distributed product teams across Meta to install pre-built domain agents and combine them on demand with their local Product Packs.

> “The shared capability like the agents... cannot know every product very deeply.” — Sumeet Arora

[09:26](https://www.youtube.com/watch?v=DJFcABDi5wE&t=566s) Closing the Loop: Task Context and Grounded Runs

Connecting an agent to real-world software maintenance requires bridging high-level problem reports with execution environments. When a maintenance task arrives, the system first inspects the incoming issue to parse its context—determining, for example, whether it is an interface bug on the Facebook Feed or an intricate performance regression in an underlying subsystem. Once the nature of the issue is clear, the system selects the appropriate agent equipped with the relevant toolset, such as emulators, simulators, and targeted operational instructions.

```mermaid flowchart LR accTitle: Grounded Run Execution Loop accDescr: Diagram showing how task context and local ground truth bind with an agent to produce a grounded run. subgraph Input ["Task Ingestion"] T["Task Reported"] --> C["Extract Task Context (Bug vs. Regression)"] end subgraph Binding ["Context & Agent Binding"] C --> A["Select Specialized Agent"] L["Local Truth & Product Context (Test Users, Navigation)"] --> G["Bind Context to Agent"] A --> G end subgraph Execution ["Action"] G --> R["Grounded Run (No Hallucinations / Drifting)"] end ```

To ensure the run succeeds without straying off course, the system binds local ground truth to the selected agent. This binding injects concrete product details—such as navigation flows, testing accounts, and environment setups—preventing the agent from hallucinating or drifting into dead ends. Sumeet Arora describes the necessity of this tight contextual anchoring, noting: “It leads to a grounded run. So basically it means is like your agent will not go in the wrong directions.” By publishing toolings and shared capabilities once, disparate teams can reuse them across Meta to solve specific product bugs reliably.

[10:24](https://www.youtube.com/watch?v=DJFcABDi5wE&t=624s) Agent Arena: Measuring Quality Instead of Asserting It

Creating an open marketplace for agents introduces a critical evaluation dilemma: when multiple candidate agents claim to handle the same class of bugs, selecting the best one cannot rely on self-reported claims. As Arora warns, “A marketplace of agents as we talked about without evidence is just a directory of demos.” To replace assertions with empirical proof, Meta created the Agent Arena, an evaluation harness designed to run head-to-head comparisons across standardized, repeatable cases.

The Agent Arena leverages a curated "fix suite" drawn from historically resolved real-world tasks and landed fixes. Competing agent variations face identical problems under controlled conditions, producing a comprehensive scorecard evaluating key performance dimensions:

- **Solve Graders:** Measures the exact pass rate and whether the proposed fix accurately resolves the root issue. - **Anti-Cheat Detection:** Monitors agents to ensure they are not taking unauthorized shortcuts or gaming the test harness. - **Cost Analysis:** Tracks resource utilization, API queries, and compute overhead consumed during each run. - **Reliability:** Identifies failure rates, timeouts, and execution stability across recurring test suites.

Autopilot uses this standardized arena both to help engineering teams pick proven tools and to give agent builders the objective diagnostics needed to refine their models.

[12:04](https://www.youtube.com/watch?v=DJFcABDi5wE&t=724s) Calibrating Agent Identity and Blast Radius

Granting autonomous agents access to production systems requires a delicate balance between capability and security. If an agent is granted insufficient privileges, it cannot execute builds, reproduce issues, or draft pull requests; conversely, overly permissive credentials inflate the blast radius, exposing codebases to accidental disruption. Arora summarizes this operational trade-off directly: “If you give too little access to your agents, no work is going to get done. If you give too much access to your agents and they have a higher blast radius, they can go do things which you don't want them to do.”

To manage risk, Autopilot ties execution identities directly to individual capabilities and agent roles:

- **Autopilot Bot (Default):** A constrained baseline identity capable of generating code diffs and running safe checks, intentionally stripped of high-risk operational privileges. - **Custom Service Users / Crewmates:** Tailored identities provisioned with scoped permissions to touch protected data sets or interact with specialized team infrastructure whenever an agent requires elevated operational clearance.

[13:02](https://www.youtube.com/watch?v=DJFcABDi5wE&t=782s) The Autopilot Platform Architecture and Ecosystem

Autopilot brings together reactive inputs—such as crashes, bugs, regressions, and ongoing incidents—with proactive finder agents through a centralized orchestration engine. Operating as a state machine, the Autopilot platform tracks open tasks, injects relevant team-specific product context, and coordinates the execution loop from triage to resolution.

```mermaid flowchart TD accTitle: Autopilot Platform Architecture accDescr: Architecture showing reactive and proactive inputs flowing into the Autopilot state machine and connecting to the agent ecosystem. subgraph Inputs ["Signals & Detection"] R["Reactive Signals (Bugs, Regressions, Incidents)"] F["Proactive Finder Agents (Performance, Anomaly Detectors)"] end subgraph Platform ["Autopilot Core"] SM["Autopilot State Machine (Task State, Context Binding)"] ID["Identity & Governance (Autopilot Bot / Crewmate)"] AR["Agent Arena (Evaluation & Benchmarking)"] end subgraph Ecosystem ["Agent Ecosystem"] SP["Starter Pack (One-Click Baseline Agents)"] CA["Custom Agents (Domain Specialists)"] MP["Marketplace (Shared Cross-Team Agents)"] end Inputs --> SM SM <--> ID SM <--> AR SM --> Ecosystem ```

When engineering teams onboard to Autopilot, they access a tiered ecosystem designed to support varying levels of operational complexity:

- **Starter Pack:** A turn-key collection of pre-configured agents built to address common cross-domain maintenance needs out of the box with one-click setup. - **Custom Specialist Agents:** Bespoke agents built by product teams to navigate unique product quirks, proprietary data schemas, or specialized domain logic. - **Agent Marketplace:** A central registry where teams share, discover, and reuse domain-specific agents developed across the broader company, validated by Agent Arena benchmarks.

[14:50](https://www.youtube.com/watch?v=DJFcABDi5wE&t=890s) Real-World Impact: From Reactive On-Call to Always-On Maintenance

Autopilot has seen wide adoption across Meta, where it is running in production and has already landed thousands of code changes to fix live bugs and performance regressions. Rather than waiting for a human on-call engineer to notice an alert, log in, and begin triaging, the automated loop initiates remediation immediately. When an issue cannot be fully resolved autonomously, the system avoids open-ended failures by escalating to engineers with pre-assembled evidence and concrete, recommended next steps.

This shift fundamentally alters software team dynamics. Engineering teams historically spend roughly 50% of their bandwidth reacting to maintenance backlogs. By turning triage queues into continuous loops that find, fix, verify, and learn in the background, human engineers shift from manually grinding down ticket queues to supervising agent performance "on the loop." Arora highlights this operational milestone: “From a queue you move to a continuous loop of finding, fixing, verifying and learning.”

In his closing remarks, Arora challenges the engineering organization to reconsider the scope of unattended software decay, asking: “Which maintenance problem in your product is still waiting for human to notice it?”

## Notable quotes > [10:24](https://www.youtube.com/watch?v=DJFcABDi5wE&t=624s) A marketplace of agents as we talked about without evidence is just a directory of demos. — Sumeet Arora

> [12:04](https://www.youtube.com/watch?v=DJFcABDi5wE&t=724s) If you give too little access to your agents, no work is going to get done. If you give too much access to your agents and they have a higher blast radius, they can go do things which you don't want them to do. — Sumeet Arora

## Mentioned Sumeet Arora, Meta, Autopilot, Facebook Feed, Reels, Threads, Agent Arena

[Source](https://www.youtube.com/watch?v=DJFcABDi5wE)

中文简报

Meta部署AI智能体修复生产环境Bug并减轻值班负担将本地产品上下文注入共享AI智能体的自主维护闭环，可在人工干预前解决线上软件故障，挽回因紧急排错损失的高达50%的工程带宽。 Meta工程主管Sumeet Arora介绍了Autopilot，这是一个旨在自动化持续软件维护工作流的智能体平台，而此类工作通常会消耗工程团队多达一半的精力。Autopilot通过集分类、根本原因调查、补丁生成和回归验证于一体的统一闭环，打通了被动缺陷工单与主动遥测驱动的问题发现机制。为防止在Facebook Feed和Reels等不同应用中出现幻觉和偏离目标的操作，该架构将可复用的智能体定义与提供特定代码库基准事实的本地化“产品包”（Product Packs）解耦。该平台受精确校准的安全身份管辖，并在实证“Agent Arena”中以历史修复记录为基准进行评估，目前已在生产环境中自主合入了数千个补丁，使人类工程师从被动的分拣队列转向监督性角色。 Meta推出了Autopilot，这是一个将应对式值班排错转变为持续、自主软件维护的智能体工程平台。 (0:05) Meta的工程团队可能要将高达50%的精力纯粹用于运维支持和缺陷修复，从而挤占了功能开发的时间。 (0:05) 为防止智能体发生漂移和幻觉，Autopilot将共享的智能体能力与封装在标准化“产品包”中的本地化产品基准事实进行了解耦。 (5:43) Meta利用“Agent Arena”测试框架对竞争的智能体进行实证评估，通过真实的历史修复案例来检验其解决率、成本和可靠性。 (10:24) Autopilot已在Meta的生产环境中部署，并已针对线上缺陷和性能退化自主合入了数千次代码变更。 (14:50)

Reading summary

Meta developed an agentic maintenance engine called Autopilot to transition engineering teams from reactive on-call duty to continuous, autonomous remediation. According to Sumeet Arora, pairing shared agent capabilities with localized product knowledge enables autonomous systems to safely find, debug, and patch production defects. The Operational Burden of Software Maintenance (0:05) Software maintenance begins immediately after deployment, demanding continuous operational labor to monitor, debug, and patch regressions. Arora reports that engineering teams at Meta scale often spend up to 50% of their working capacity solely firefighting operational issues and live-site incidents, which directly impedes progress on new product roadmaps. Despite this systemic drain, Arora argues that maintenance workflows are structurally suited for autonomous AI agents. Unlike open-ended product ideation, maintenance tasks operate within bounded scopes backed by telemetry and crash logs. Fixes are also testable and reversible in isolated staging environments, creating a closed feedback loop where agents can learn from failed attempts without repeating mistakes. Unifying Reactive Queues and Proactive Discovery (3:00) Autopilot processes work originating from two distinct sources: reactive queues and proactive finder agents. The reactive flow responds to standard operational triggers, including user-reported bugs, incident alerts, and crash spikes. Proactive agents run continuously across live production surfaces, analyzing surface telemetry (such as Facebook Feed load latency) and historical crash logs to uncover latency bottlenecks, accessibility flaws, and reliability defects before human monitors file an issue. Both channels converge into a shared lifecycle within the Autopilot platform. The state machine classifies the problem, triages it to the correct owner, investigates root causes, and attempts resolution. Each run terminates in one of three concrete states: generating a validated code diff, escalating to a human engineer with compiled diagnostics and suggested next steps, or closing the task when no operational action is warranted. Grounding Shared Agents with Local Product Packs (5:43) Deploying autonomous agents across heterogeneous surfaces—such as Facebook Feed, Reels, Threads, or internal tooling—revealed an architectural tension between shared capabilities and local context. A generic agent lack awareness of nuanced team boundaries or unique user flows, leading to hallucinations or drift. Meta resolves this by decoupling reusable agent logic from local product truths. Under this architecture, centralized Agent Providers supply domain-specific agent definitions containing instructions and tooling for areas like accessibility or performance. Product teams supply "Product Packs" consisting of architectural documentation (`knowledge.md`), code pointers, triage routing rules, step-by-step reproduction flows (`navigation.md`), test user accounts, and project metadata (GST configs). Dynamically binding these Product Packs to shared agents at runtime produces grounded runs anchored in the target product's reality. Empirical Benchmarking and Scoped Execution Governance (10:24) To evaluate competing agent definitions across the internal marketplace, Meta created the Agent Arena. Arora notes that without empirical evidence, an agent marketplace degrades into a "directory of demos." The Arena subjects agents to identical conditions against a curated suite of historically landed fixes, scoring each variant on pass rate, anti-cheat compliance, API and compute overhead, and execution reliability. To balance operational autonomy against security risks, Autopilot restricts the execution blast radius through distinct identity tiers. By default, the Autopilot Bot operates with baseline permissions sufficient to author code diffs and run test checks. For sensitive datasets or specialized infrastructure, teams provision scoped service identities, known as Crewmates, granting agents only the elevated access necessary to complete specific workflows. Autopilot is deployed in production at Meta, where it has landed thousands of diffs to repair live bugs and performance regressions without initial human intervention. While the platform shifts human engineers into supervisory roles overseeing automated loops, its success remains dependent on teams maintaining accurate Product Packs, defining strict identity boundaries, and identifying which latent operational defects are still awaiting detection.

精炼阅读摘要

Meta 技术总监 Sumeet Arora 指出，在超大规模系统下软件维护占用了工程团队高达 50% 的工作时间；为此 Meta 构建了名为 Autopilot 的智能体平台，通过将通用 Agent 能力与封装本地上下文的 Product Pack 相结合，推动运维从被动人工轮值（On-Call）转向持续自主闭环（Always-On）。软件维护负担与 AI Agent 的结构契合 (0:05) Sumeet Arora 强调，软件交付生产环境并非工程劳动的终点，而是维护时钟的起点。在 Meta 的体量下，数十亿次交互会迅速放大细微缺陷，工程团队常常被迫将多达 50% 的工时投入到故障排查、根因定位、代码修补以及防退化（Regression）等日常救火任务中，严重挤占了新功能研发资源。相比目标发散的产品创新，软件维护流程在结构上与自主 AI Agent 具有天然契合性。维护任务的问题边界相对清晰，并依托丰富的遥测指标、崩溃日志与运行凭证。更重要的是，代码修复具备高度可验证性与可逆性：Agent 可以在隔离环境中试运行补丁并验证行为，若测试失败即可安全回滚并迭代替代方案，形成闭环学习机制以避免重复错误。响应式与主动式维护的统一生命周期 (3:00) 软件维护主要经由两条路径进入工程视野：响应式（Reactive）与主动式（Proactive）。响应式处理经典事故告警、工单缺陷与功能退化，需验证严重度、重构上下文并派发修复；主动式维护则利用自主寻找型 Agent（Finder Agents）持续扫描生产表面，分析 Facebook Feed 加载延迟等实时遥测与历史崩溃数据，在故障影响扩散前预先发现性能瓶颈、无障碍缺陷及可靠性隐患。 Autopilot 将这两类不同来源的输入整合至统一的状态机维护闭环中。无论任务由告警触发还是由探针发现，均经过统一的分类派发、调查修复与自动验证，最终收敛至三类明确结果：生成经测试验证的代码补丁（Diff）、携带完整证据链与建议升级转交人工工程师，或在确认属于瞬态波动时判定无需操作并直接结案。架构解耦：通用 Agent 结合本地事实 (5:43) 跨越 Facebook Feed、Reels、Threads 等多业务线的大规模工程体系，使单一通用 Agent 无法掌握所有产品的内部架构与专有逻辑。Meta 的架构方案将通用 Agent 能力与本地产品事实（Local Product Truth）彻底解耦：中央专家团队作为 Agent Providers，负责在内部市场开发和发布专注于性能优化或缺陷修复的通用技能；具体的业务上下文则被打包进各团队独立维护的“Product Pack”中。 Product Pack 包含 Agent 自主运行所需的基础凭据：记录架构与上下游依赖的 `knowledge.md`、代码入口指针、团队分工规则、复现步骤文件 `navigation.md`、用于真实模拟环境的测试账号以及配置元数据。任务派发时，系统将本地事实动态绑定至选定的 Agent 上，实现受控的“锚定运行”（Grounded Run），避免模型脱离实际边界或产生幻觉调试路径。实证评测与爆炸半径：Arena与权限治理 (10:24) Arora 警示称，缺乏实证数据的 Agent 市场只是“Demo 目录”。为了替代主观宣称并科学评估能力，Meta 建立了基准测试平台 Agent Arena。该平台依托真实历史已解决缺陷构建测试套件，在严格受控条件下横向评测候选 Agent，指标覆盖根因解决率（Solve Graders）、防作弊检测（防止绕过测试直接伪造结果）、算力与 API 消耗（Cost Analysis）以及超时与执行稳定性（Reliability）。在执行权限与安全治理方面，系统需要平衡自主操作能力与潜在系统破坏风险。过低的权限会导致任务停滞，而过高的权限则会放大故障爆炸半径（Blast Radius）。Autopilot 采取了分级身份策略：默认采用低权限的 Autopilot Bot 生成代码补丁与执行只读检查；仅在需要访问受保护数据集或特定基础设施时，才按需调度具备明确权限范围的定制服务账号（Crewmates）。目前 Autopilot 已在 Meta 投入生产运行，累计自主合入了数千次修复线上缺陷与性能退化的代码变更。随着大量维护任务从人工排查工单转变为后台全天候自动化循环，工程师的工作重心正逐步转向监督与微调 Agent 闭环；而如何在庞大复杂的生产系统中持续挖掘尚未被人类察觉的隐性系统衰退，依然是该平台演进面临的长期课题。

Source quotes

[10:24] Sumeet Arora: A marketplace of agents as we talked about without evidence is just a directory of demos. [12:04] Sumeet Arora: If you give too little access to your agents, no work is going to get done. If you give too much access to your agents and they have a higher blast radius, they can go do things which you don't want them to do.

Original transcript

[0:05] Hi everyone. Today I will be presenting [0:06] on topic from on call to always on. In [0:09] this presentation we'll talk about the [0:12] agentic platform we built that lets AI [0:14] agents to do product maintenance for [0:16] you. [0:17] Okay. Now let's talk about the [0:18] maintenance problem. The product shipped [0:21] but the work did not. What does that [0:23] means? Your maintenance obligation or [0:25] the maintenance clocks starts as soon as [0:27] the products get shipped. So you have to [0:29] continuously monitor for bugs, [0:31] production incidents or any regressions [0:33] that can happen within your product. And [0:36] once you get a task or a production [0:38] incident, you go and debug and find [0:41] where those regressions or the [0:43] production incidents happened, what part [0:45] of the app is impacted and everything. [0:48] And then once you know, you come up with [0:50] a fix, you fix it, you repair it, and [0:53] then you update your app or product and [0:55] you apply the patch. And once you're [0:57] done with it, you want to defend it so [0:59] that you make sure it doesn't happen [1:01] again. And you put in some cartridges. [1:05] At scale, maintenance becomes a system [1:07] problem. What does that mean? Let's say [1:09] you have a scale of meta and you have [1:11] these small failures and then small [1:13] failures into the enormous scale of meta [1:15] leads to continuous pressure on [1:17] engineering teams and the teams may [1:19] spend as much as 50% of their time just [1:22] fixing problems and this takes away [1:24] their time from the product road map and [1:26] the road map they want to deliver on and [1:27] the new features. [1:30] Now let's look deeper into why [1:32] maintenance is a good fit for agents or [1:35] AI agents in this case. So the product [1:38] the task or the regression or the bug is [1:41] usually bounded. So you know it's [1:42] happening or it's impacting certain part [1:44] of your app or your product. And then [1:47] you have telemetry logs and all the [1:49] evidence you need to understand what's [1:52] going on. [1:54] It's usually testable. You can stage it. [1:56] You can test it whether it's working or [1:58] not. You can even reverse it. If things [2:00] are not working, you can reverse it and [2:02] then you can make it try another thing [2:04] and make it work and then test it again. [2:06] So you can stage it, reverse it, stage [2:08] it, reverse it and all that. And it [2:10] becomes an easy problem for agents as [2:12] well to do that. And the finally you can [2:14] have a feedback loop. You can learn from [2:16] your mistakes. You can understand what [2:18] didn't work and then you make sure you [2:19] don't make the same mistake twice. [2:23] Okay. So in today's agenda, what will we [2:25] cover? So we'll cover different types of [2:27] maintenance. We'll talk about reactive [2:29] maintenance and proactive maintenance [2:31] and their entry points. And then we'll [2:33] talk about why product context in the [2:35] agent ecosystem is necessary to be able [2:37] to build a platform that can solve [2:39] product maintenance at scale. And [2:41] finally we'll come go to the evaluation [2:44] of these agents and the control these [2:45] agents can have and finally the results [2:47] from building such platform. My name is [2:50] Sumit Aurora. I am a software engineer [2:52] at Meta and I have been looking at the [2:54] product maintenance problem for last 6 [2:56] months. Now let's dive deep. [2:59] Now let's talk about the two kinds of [3:00] maintenance we just talked about. So [3:02] work enters through two doors. [3:05] The first one we call as a reactive [3:07] maintenance. It's like an existing bug [3:09] or some bug is filed or some regression [3:11] happened or some production incident [3:12] happened. So it's an existing signal and [3:14] then you go and investigate and restore [3:16] and understand where things are going [3:18] wrong and then you fix it. The other [3:21] type of product maintenance you might be [3:23] doing or may not be doing is called [3:26] proactive maintenance. It's about [3:27] finding the performance opportunities, [3:29] performance optimization opport [3:31] opportunities or finding the bugs even [3:33] before they are filed. So it's very [3:35] proactive in nature. And in this case of [3:37] the AI agents, the system gets up and [3:39] discover it. Then you find it, [3:42] prioritize the things you think are [3:43] important for the end users and then you [3:46] keep going on this loop and keep [3:48] improving your agents to be able to do [3:50] these things. [3:52] Okay. Now let's dive deeper into the [3:54] reactive maintenance. As I said, [3:56] reactive maintenance begins with a [3:57] queue. But what does that queue look [3:59] like? So, first you validate is it a [4:01] real regression? Is it a real incident? [4:03] And then we reconstruct the context to [4:05] understand who is the right owner and [4:07] find the right owner for it. And the [4:09] right owner is going to go investigate [4:11] and understand what what's going on [4:13] here. And then they will be able to come [4:15] up with the fix and restore it and then [4:17] verify it and then apply the patch. So [4:20] that is the typical reactive maintenance [4:21] Q or begins with or look like. [4:25] Okay, now let's look at the other side [4:27] of the coin which is proactive [4:28] maintenance. It's even starts before the [4:30] queue. You can have a finder agent. As [4:32] you can see in the center, they take in [4:34] some inputs. They have live surfaces [4:36] like okay, I want to look at Facebook [4:38] feed. You have quality signals. You may [4:40] have like telemetry around how long it's [4:42] taking to load a Facebook feed. You may [4:44] have user sports reports which are the [4:46] past bugs or things like that has been [4:47] reported by user and you feed it to the [4:50] finder agents and then they come up with [4:51] a task or an evidence-based candidate [4:53] task on what are the things that it can [4:55] be done. In short, you discover product [4:58] issues and opportunities for [5:00] optimizations and they can be across [5:02] different domains. You can have bugs, [5:04] regressions, accessibility, performance [5:07] and reliability. [5:09] So we talked about two maintenance cues [5:12] or systems but they have like two entry [5:14] points but you can have one maintenance [5:15] system for them. One is reactive and [5:17] proactive and they share the maintenance [5:19] loop. You classify the problems as they [5:22] come in either from reactive or [5:23] proactive. Then you triage you [5:25] understand where it should live what [5:26] team it should live on and then that [5:29] team or that particular engineer can go [5:31] fix it verify it and learn from it. And [5:34] similarly in auto the agentic platform [5:36] we build do the same thing and have a [5:38] shared maintenance loop for both [5:40] reactive and proactive maintenance. [5:43] Introducing autopilot. So I have been [5:45] talking about the agentic platform that [5:47] lets AI agents to do product maintenance [5:49] for you. That is what it sits in the [5:51] middle. You have reactive maintenance. [5:53] You have your things which went wrong. [5:55] You have proactive stuff finding things [5:58] that can be improved. Then you have a [5:59] persistent owner in between which is the [6:01] autopilot system which is the agentic [6:03] system that owns it and then produces [6:06] one of three outcomes or maybe more. One [6:08] of the outcome can be changed. It [6:09] produces a fix. Hey by the way this is [6:11] the fix and how to fix this thing. It [6:12] may say escalation [6:14] in case of escalation it will say hey by [6:16] the way human I need your help. I'm not [6:18] able to do these steps because of XYZ's [6:20] reasons. But it has evidence and it has [6:22] recommendation for you the things that [6:24] can be done. Or it may say no action is [6:27] needed. the things are already resolved [6:29] or maybe was transient it it can be [6:32] closed. So one system called autopilot [6:34] for both reactive and proactive product [6:37] maintenance. [6:39] So when you're about to build such a [6:41] platforms that can do the product [6:43] maintenance at scale. So you can imagine [6:46] a lot of team wants to use it. So it's [6:48] very distributed in that way. The first [6:51] hard problem you are going to face in [6:53] that scenario is like shared capability [6:55] needs a local product growth. What does [6:57] that means? It means the reusable agents [6:59] like you can build the agents, you can [7:01] have everything but it cannot know every [7:03] product very deeply. It doesn't know [7:04] what I need to do for Facebook feed. It [7:06] doesn't know what I need to do for reals [7:08] or a very different product can be [7:10] threads or a very different product for [7:11] AI. So it can have instructions and [7:14] everything it can follow but it doesn't [7:15] know every product very deeply. So in [7:18] the design response to be able to solve [7:20] that problem at scale was to keep the [7:23] capability shared like the agents. Okay, [7:25] let's share across teams but attach the [7:28] product context at the dispatch time. So [7:31] shared capability and the local truth [7:33] which gets attached to that capability [7:35] leads to a grounded execution. [7:39] Now let's look deeper into the local [7:42] product truth I talked about. We call it [7:44] a product pack. What does it can [7:46] contain? It contains knowledge MD files [7:49] which is basically architecture [7:50] ownership and dependencies for your [7:52] team. It has code pointers. It has [7:55] triage rules like how do whether this [7:58] task particularly sits on my team or a [7:59] sister team or maybe a different team [8:01] altogether that can have navigation MD [8:04] which is like how the agents can go and [8:06] reproduce these bugs and it has all the [8:08] debugging paths and everything. [8:11] Basically you also want to give them [8:12] some test user config because the agents [8:14] need to log in as test users and [8:15] understand what's going on in your [8:17] product. And lastly it has GST config [8:20] which is basically a team projects [8:22] metadata. so that it understands your [8:24] team better. So the thing is load this [8:26] right context whenever the decision in [8:28] case this agents need it. [8:32] Now let's look into the other side of [8:34] the coin which is the agent definition [8:35] or the shared capability we just talked [8:37] about. So you have two parts to it. You [8:40] have agent providers. The agent [8:41] providers are the owners of these [8:44] expertise or domains. Let's say you have [8:46] performance domain, you have bugs domain [8:48] and they own the improvement loop of [8:50] these shared capabilities. And the [8:52] second thing is you have agent [8:53] definition itself. What does an agent [8:54] means? It means instructions. It means [8:56] plugins. It means skills that agent has. [8:59] And you combine these it leads to the [9:01] shared capability. [9:04] Okay. So we as we said these shared [9:06] capabilities need to be attached to the [9:08] local truth and they need to be able to [9:10] be installable by the other teams. So [9:12] you publish these agents or share [9:14] capability for other teams to install. [9:16] You have agent providers as we s talked [9:18] about. to build and publish these agents [9:20] and then you have a marketplace which [9:22] can be cataloged of these agents [9:24] together they can be installed and [9:26] discovered by other teams. So you [9:29] publish once and reuse across the teams. [9:34] Now let's put all these things together. [9:36] So how does this loop works? So you get [9:39] a task, you can extract the context from [9:42] that task. You understand what's going [9:43] on in that particular task. Is it a bug [9:46] on Facebook feed? Is it a performance [9:48] regression on XYZ? So you understand the [9:51] context. You select the right agent. You [9:53] understand okay this is a pug probably I [9:55] need to have emulators simulators and [9:57] all that stuff in instructions. Maybe I [9:59] need to select the right agent and then [10:02] you bind the local truth with that agent [10:05] and combine it together. It leads to a [10:07] grounded run. So basically it means is [10:09] like your agent will not go in the wrong [10:11] directions. It knows what kind of the [10:13] context behind this task is what I have [10:15] the agent I have selected and what is [10:17] the product context like test users and [10:19] navigation and all we talked about is [10:20] this there for this particular ground. [10:24] Okay. So we talked about the hard [10:25] problem one or the challenge you face [10:27] when you're going to build an agentic [10:28] platform for product maintenance. Now [10:30] let's go into the hard problem two. We [10:32] talked about agents shared capability a [10:34] lot but the agent quality must be [10:37] measured or that shared capability must [10:39] be measured not asserted. What does that [10:41] mean? A marketplace of agents as we [10:43] talked about without evidence is just a [10:45] directory of demos. How do I know if [10:47] agent A is better? Let's say there is [10:50] agent A for bugs, agent B for bugs, [10:52] agent C for bugs. So there are a lot of [10:53] agents which can take care of bugs for [10:55] me. How do I know which one is better? [10:57] So we our design response our solution [11:00] to that was to come up with something [11:02] called agent arena where you can compare [11:04] different variables on repeatable cases. [11:07] So same cases you get a scorecard and [11:09] you go improve it [11:12] in details. What does agent arena look [11:15] like? So agent arena makes the [11:17] comparisons controlled. You have fix [11:19] suite which is nothing but real close [11:21] task and landed fixes like what worked [11:24] and what didn't work. And then you take [11:27] agents A and B and you compare them [11:29] head-to-head in this arena. And finally [11:31] you get a scorecard which is like how [11:33] many problems are they able to solve [11:35] which is sol graders think see that are [11:38] these agents cheating or not we do an [11:41] anti-che in them what's the cost of [11:43] these runs are are these agents reliable [11:45] do they fail often so get this whole [11:47] scorecard and you can use this scorecard [11:49] to either understand what agent will be [11:51] useful for my team or what will not be [11:54] so autopilot provides the arena surface [11:56] for both teams to understand the [11:57] performance of agent as well as the [11:58] agent providers who can go and improve [12:01] their agents as they go out. Okay. Now [12:04] let's look into another hard problem. [12:07] The right agent needs the right [12:09] identity. So if you give too little [12:11] access to your agents, no work is going [12:14] to get done. If you give too much access [12:16] to your agents and they have a higher [12:18] blast radius, they can go do things [12:20] which you don't want them to do. So you [12:23] need to understand what kind of identity [12:26] or access these agents can have in order [12:28] to solve this. The design response we [12:30] came up with was select an execution [12:33] identity for each capability or the [12:34] agents we talked about. So we provide a [12:37] default one which is autopilot bot which [12:39] can probably produce changes and has [12:40] some permissions and others but it [12:42] doesn't have too many permissions so [12:43] that it goes do the stuff you don't want [12:46] it to do but if there is some need for [12:48] your team you can change the execution [12:50] identity at the agent and come up with a [12:52] custom service user or crewmate which [12:54] has some some explicit permissions based [12:56] on your team or your data sets and can [12:59] work rightfully for you. [13:02] Okay. So let's put all things together. [13:05] I introduced the autopilot earlier. I [13:07] talked about is an agentic platform for [13:09] product maintenance. Now let's look at [13:11] the architecture. So you have your [13:13] reactive signals, you have reactive [13:15] bugs, reactions, reactions and incidents [13:17] going on. You have your finder agents [13:20] which are going to go find a stuff and [13:22] in the middle you have the autopilot [13:23] platform which is a state machine which [13:25] has product context for that particular [13:27] team if it's installed on that team and [13:29] has a state of task and everything. So [13:31] it maintains the whole loop for you and [13:34] you know when you onboard such a loop or [13:36] such a smart flow on your team what do [13:38] you get? We get an agent ecosystem to be [13:40] able to solve the problems for you. The [13:42] agent ecosystem consists of three parts. [13:44] One first one is starter pack. So [13:47] whenever you onboard this system you [13:48] want it to be easy onboarding. You want [13:50] it to be one click. So we give you a [13:52] starter pack of things we think are [13:53] useful across domains for your team and [13:56] you can install these things across your [13:58] team and they will work out of the box [14:01] for you but that may not be always the [14:03] case because onboarding cannot be [14:06] catered to every team. It may work for [14:07] some, it may not work for them. So in [14:09] order to be able to address that [14:11] problem, you can also have custom agents [14:13] which are product specialists for your [14:15] specific team and then you can install [14:17] those. If you don't want to go and work [14:20] by yourself, starter pack is not making [14:21] sense for you. As we have said, there's [14:23] a marketplace of agents. You can go and [14:26] either publish your agents or see what [14:28] other agents are published and reuse [14:29] them across your teams. So the agent [14:32] arena compare the variants in the [14:35] autopilot system. You have identity at [14:37] the each execution level and this is how [14:39] the whole system is working together for [14:41] reactive and proactive maintenance. [14:43] autopilot assist in between gives you an [14:45] agent ecosystem to be able to do the [14:46] product maintenance for you. [14:50] Now let let us look at the results. So [14:52] this has been built already the system [14:54] and it has been adopted across many team [14:57] across Mecca and they're already finding [14:59] value in this and has already landed [15:01] thousands of production changes for them [15:04] to be able to fix the regressions or the [15:06] bugs and other things we talked about. [15:08] So the system is already in production [15:10] and it practice across meta. [15:13] Okay. Now what happens magically when [15:16] the this loop works things becomes [15:18] interesting. So your maintenance begins [15:20] even sooner. So you don't need your on [15:22] call or on call person to go wait and [15:26] take an action on that. The agent is [15:28] there. The platform is there. It's going [15:29] to take an action for you. So the [15:31] maintenance begins much sooner. [15:33] the places where agents are not able to [15:35] understand what to do next. It comes [15:37] with an escalation to human. It comes [15:39] with an evidence and recommended steps. [15:41] Hey, by the way, I I think this is why [15:43] where things are going wrong. Here are [15:46] my recommended steps. These are the [15:48] things you can go and fix them. [15:51] In case of proactive maintenance, [15:53] opportunities are found even before they [15:54] become incidents. You can have [15:55] performance opportunities. You can even [15:57] fix bugs before they go to the end user. [16:01] So and the team role changes remember we [16:03] talked about it takes almost 50% of the [16:05] time the teams may spend on the product [16:07] maintenance. So instead of them [16:10] themselves training the cues the team [16:12] just supervises and it becomes [16:15] on the loop human experience where they [16:17] they're able to go and manage these [16:19] agents in the platform and it takes care [16:21] of the product maintenance problem for [16:23] them [16:25] from on call to onways on. Now coming [16:27] back to the topic what does it means? So [16:29] it means from on call to always on leads [16:32] to a queue to a continuous loop. From a [16:35] queue you move to a continuous loop of [16:37] finding, fixing, verifying and learning. [16:42] So the system provides this continuity. [16:44] The specialist provide the capability as [16:46] we talked about the product provides the [16:48] context. The arena provides the quality [16:51] evidence and the control plane or the [16:53] identity we talked about provides the [16:54] governors and hence together the system [16:58] forms from on call to always on. So I'm [17:01] going to leave the audience with a [17:04] closing question. Which maintenance [17:06] problem in your product is still waiting [17:08] for human to notice it? Thank you.