Meta Engineers Stop Outage After Control Plane Deadlock Breaks Web Tier

控制面死锁致Web层中断，Meta工程师化解宕机危机 Meta利用应急裸机工具绕过容器调度器与持久层之间的循环依赖，成功恢复了冻结的区域Web层。 当 Meta 的 Agents Manager Service 中一次存在缺陷的特性开关更新在某区域触发错误的容器清除时，容器化的 Zelos 持久层遭到破坏，并导致区域 Twine 控制面陷入自托管死锁而冻结。Web Foundation 团队凭借常规风暴演练形成的肌肉记忆，迅速执行了区域流量切除，在 20 分钟内彻底消除了对用户端的影响。为了在调度器无法运行的情况下打破死锁，工程师部署了底层 Bell Jar Toolkit 手动修改宿主机节点上的任务规范，并引导启动了恢复容器。事件发生后，Meta 通过将 Zelos 提升为对等基础设施运行、改进宿主机映射日志以及推行强制性应急演练，对其基础设施进行了加固。 Meta 的 Web Foundation 团队通过应急工具快速化解了一起严重的区域性基础设施冻结故障，成功避免了全站级长时间瘫痪。 (0:05) 故障根源在于为应对 AI 工作负载引入的 Agents Manager Service 在重启后误报容器状态，导致调度器清理了底层的 Zelos 持久化层容器，进而陷入循环死锁。 (7:27) 团队凭借风暴演练（storm drills）积累的操作经验迅速排空该区域流量，在收到初始报警后 15 到 20 分钟内便彻底消除了面向用户的影响。 (9:52) 工程师利用底层 Bell Jar Toolkit 绕过挂起的控制平面，在宿主机上手动拉起移除故障 feature flag 的恢复调度器，并在重新夺回控制锁后顺利交接回标准容器。 (11:31) 事故后，Meta 将 Zelos 提升为与 Twine 调度器同级的核心基础设施以消除循环依赖，并推行持续的 Bell Jar 自动化测试与定期人工恢复演练。 (15:53)

Meta Web Foundation engineers Phil Lopreiato and Rahul Iyengar detail how a circular dependency in the Twine container orchestration stack caused a regional control plane freeze, which was mitigated through emergency low-level bootstrapping tools and structural architectural changes. Scale-Driven Topology and Circular Dependencies (4:37) To absorb capacity surges driven by AI training and inference, Meta added scheduler shards to its Twine orchestration framework. This shift created an M × N connectivity mesh and traffic bursts between schedulers and host agents. To streamline connectivity into an M + N model, Meta introduced the Agents Manager Service. Both the schedulers and the Agents Manager relied on Zelos—an in-house distributed store providing the ZooKeeper API atop Delos—for strong consistency guarantees. However, Twine was self-hosted in a hierarchical tree where regional scheduler shards were managed by a Regional Meta Scheduler, while the underlying Zelos ensembles were hosted as containerized workloads within Twine shard 0. When an update introduced the Agents Manager Service behind a feature flag, the service restarted and prematurely returned an empty task state before reading from Zelos. Twine's scheduler interpreted the empty response as an absence of containers and purged all active workloads in shard 0. This wiped out the containerized Zelos ensemble, freezing the Regional Meta Scheduler in a control plane deadlock that prevented standard automated remediation. Blast Radius Containment and Out-of-Band Bootstrapping (9:52) Because the update was staged to a single region behind a feature flag, the failure blast radius was strictly localized. Applying procedures practiced during routine storm drills, the response team drained all incoming traffic from the affected data center to healthy regions, eliminating user-facing impact within 15 to 20 minutes of the first alert. Because the control plane was frozen, operators could not rely on standard deployment tools to revert the feature flag. Responders turned to the Bell Jar Toolkit—developed after Meta's October 4, 2021 site outage—to bootstrap raw recovery containers on host machines. Using operational logging tools, engineers identified former host nodes, verified local binary integrity, and bypassed tool limitations by manually modifying the host job specification to strip the flag. Once launched, the Bell Jar recovery schedulers claimed the control lock, restarted downstream tasks, handed the primary control lock back to standard managed containers, and allowed operators to safely undrain regional traffic. Structural Architectural Hardening and Drill Automation (15:53) To prevent recursive bootstrapping deadlocks, Meta elevated Zelos (ZooKeeper) from a regular Twine-managed workload into an independent peer infrastructure layer that operates directly alongside regional schedulers. Rahul Iyengar noted that although this change added capacity and operational overhead for the Zelos storage team, it eliminated circular startup dependencies during regional outages. Meta also revamped historical host-to-task mapping logs to eliminate visibility blind spots during control plane failures. To ensure emergency tooling would function during novel incidents, Meta introduced automated Bell Jar test suites to continuously detect software regressions. The organization instituted mandatory monthly or quarterly hands-on drills so that operators maintain operational familiarity rather than touching manual tools for the first time during a catastrophic failure. Iyengar reported that this modernized recovery tooling was subsequently deployed to recover core infrastructure from scratch during two separate post-incident events. The engineers observed that rapid infrastructure growth from AI adoption continually increases architectural complexity while reducing human full-stack context. They concluded that surviving severe control plane deadlocks ultimately requires preserving basic manual operational capabilities—the ability to rebuild systems directly on bare hardware—paired with foundational safety practices like canary deployments and routine regional traffic drains.

Meta 工程师 Phil Lopreiato 与 Rahul Iyengar 阐述了核心容器调度平台 Twine 在一次单区域严重瘫痪事故中的故障成因与恢复过程，指出系统自托管与隐式依赖会在极端情况下锁死控制面；该事故表明，在 AI 负载推高系统复杂度的背景下，团队不仅需要金丝雀发布与快速排流能力，更依赖与高层软件解耦的底层应急工具及常态化实操演练。 架构扩张与自托管循环依赖死锁 (4:37) Meta 核心容器编排系统 Twine 采用分片集群架构，由调度器（Schedulers）、分配器（Allocators）、资源代理（Resource Brokers）和主机代理（Agents）组成，单个调度器分片管理 10 万至 100 万个任务。随着 AI 推理与训练负载扩张，集群物理机数量及单机任务密度激增，导致调度器分片与主机代理之间的 M×N 通信产生剧烈流量突增。为此，Meta 引入了代理管理服务（Agents Manager Service）将其转化为 M+N 架构，并依赖基于 Delos 构建且提供强一致性保障的 Zelos（兼容 ZooKeeper API）存储系统维持调度器与代理管理服务的状态。 然而，Twine 采用层级树状自托管架构，区域调度器分片受区域元调度器管理，而关键的 Zelos 集群并未直接部署于裸机，而是作为容器化作业运行在 Twine 区域调度器的 0 号分片（shard 0）中。事故发生时，通过特性开关（feature flag）发布的代理管理服务发生重启，在尚未从 Zelos 读取最新状态前，便向区域调度器返回了“无容器运行”的空响应。区域调度器误触发标准清理逻辑，终止并销毁了包括 Zelos 集群在内的所有活动容器。由于元调度器与代理管理服务均强依赖 Zelos 保持一致性，持久化层的丢失导致控制面瞬间冻结并陷入死锁——工程师既无法通过自动化发布系统推送修复，代理管理服务也无法重新拉起被清除的容器。 流量隔离与 Bell Jar 应急恢复 (9:52) 监控显示特定区域的主 Web 作业运行状态从健康绿骤变为故障粉色，警报迅速触发。得益于 Meta 长期推行的灾难恢复机制（可追溯至 2012 年飓风桑迪）及每周常规的“风暴演练”（Storm Drills），响应团队在确认故障局限于单一区域后，迅速执行区域排流（drain），在最初警报触发后的 15 至 20 分钟内将入站流量完全切换至其他未受影响的数据中心，迅速消除了面向用户的实际影响。 在标准部署流水线因控制面死锁而完全不可用的情况下，团队启用了在 2021 年 10 月 4 日全站大停机后开发的“Bell Jar”应急恢复工具包。该工具允许绕过高层软件依赖直接在裸硬件层面运行极简应急服务。工程师通过仍正常工作的历史日志系统定位此前运行调度器任务的物理节点，核实本地磁盘上保留的执行二进制文件，并针对工具包仅支持追加参数而无法删除参数的限制，在主机上手动修改作业配置以剔除有缺陷的特性开关，最终直接在宿主机上成功拉起应急调度容器。应急调度器在夺得权威控制锁后重新唤醒下游作业，随后工程师通过 TTY 更新全局回滚开关并切换回标准托管容器，顺利完成了区域恢复。 解耦底层依赖与消除能见度盲区 (15:53) Rahul Iyengar 指出，为防止类似死锁重演，Meta 对基础设施控制面进行了关键架构重构。最核心的决定是将 Zelos（ZooKeeper）从普通托管应用服务提升为关键基础设施层级，使其作为与 Twine 区域调度器同级的对等组件独立运行，而不再作为 Twine 管理的下属容器。虽然这一转变给 Zelos 运维团队带来了更重的主机容量与生命周期管理负担，但彻底切断了区域恢复时出现的自举循环依赖。 此外，事故处置过程中工程师曾因无法精准确定哪些物理机承载了核心控制任务而面临可见性盲区，部分依赖人工经验甚至运气进行排查。为此，团队重构了历史主机到任务的映射日志系统，确保在未来的控制面黑屏状态下，运维人员依然拥有确定性的追踪与定位能力。 自动化演练与高复杂度下的工程底线 (17:11) 为了防止应急工具在非演练期因代码迭代而无声失效，Meta 将 Bell Jar 工具包接入隔离的自动化测试框架中进行持续回归验证。在人员训练层面，Meta 确立了强制性的月度或季度实战演练制度。Rahul 强调，应急恢复工具绝不能在真正的全网重大故障中才首次被操作人员接触，团队必须通过定期演练形成肌肉记忆。在此次事件之后，Meta 已在另外两次真实的重大基础设施故障中借助该套升级后的工具链从零重建核心服务，验证了该机制的实战价值。 Rahul 总结认为，随着 AI 基础设施以极高速度扩张，系统的全栈认知壁垒不可避免地加深，导致故障排查与自动化修复更加棘手。在此背景下，工程师必须在高度自动化之外保留最原始的底层手动干预能力；而分阶段的金丝雀发布、细粒度的区域流量隔离以及制度化的灾备演练，构成了系统在面对不可预测故障时最关键的工程安全网。 本报告展现了大规模分布式系统在过度依赖自托管自动化时可能遭遇的脆弱性。Meta 的处置经验表明，抵御级联故障的核心不仅在于优化上层算法，更在于预先打破控制面的循环依赖，并始终维持一套与主系统解耦的应急工具和具备底层手工操作能力的工程师团队。

## Background 本视频由技术频道 @Scale 于 2026 年 6 月 30 日发布在 YouTube 平台。主讲人是来自 Meta 的基础设施工程师 Phil Lopreiato 与 Rahul Iyengar，内容聚焦于 Meta 在 2025 年 3 月如何通过应急工具与完备的容灾预案（DR），在几分钟内遏制住一起险些酿成全网长时间瘫痪的重大区域控制面死锁事故，并现场热拉起整个数据中心控制平面。 ## TL;DR 本次分享详细复盘了 Meta 容器调度平台 Twine 因 AI 业务暴增进行架构改造时所引发的严重事故：新上线的 Agents Manager 服务在重启后因未读出数据便响应查询，导致调度器误杀包含底层 Zelos 强一致存储在内的所有运行容器，使区域控制平面因自托管循环依赖而彻底死锁。依托每周常态化的‘风暴演练’（Storm Drills），团队在 20 分钟内完成受灾区域的全量流量排空（Region Drain），彻底封死外部用户影响。面对常规部署通道尽失的绝境，工程师使用 2021 年大断网后孵化的 Belljar 带外应急工具箱，在物理机上复用残留二进制文件、手动剔除致祸的 Feature Flag 并夺回系统控制权，随后通过架构重构将 Zelos 提升为调度器同级组件，彻底斩断了控制面循环死锁链条。 ## Atmosphere & style 分享洋溢着老练 SRE 人员特有的自嘲幽默与硬核工程坦诚，叙事张力十足，如同在听一场扣人心弦的基础设施排障悬疑剧。两位讲者配合默契，开场由周五下班小聚前夕甘特图骤然‘全盘泛粉’的窒息瞬间切入，瞬间唤起 2021 年全网大瘫痪的深层心理创伤；中段生动描绘了拿着‘削尖的木棍’在控制面废墟中带外抢险的焦灼与惊险；最后在化险为夷、群发表情包奔赴延迟的 Happy Hour 之中释怀，既让听众领略到生产事故现场的心跳加速，又展现出高度成熟的容灾体系在绝境中所赋予工程师的沉着底气。 ## Bulletin - [07:27](https://www.youtube.com/watch?v=-0v6UUPLvF8&t=447s) Meta 基础设施因新上线的 Agents Manager Service 在重启后未完成 Zelos 数据读取即返回空结果，触发 Shard 0 调度器误杀包括区域元调度器底座在内的所有容器，导致单区域控制平面陷入死锁瘫痪。 - [09:52](https://www.youtube.com/watch?v=-0v6UUPLvF8&t=592s) 得益于每周常态化的“风暴演练”（Storm Drills），现场团队在 15 至 20 分钟内果断执行区域切流（Region Drain），将受影响数据中心流量全部无缝调度至其他区域，完全消除了外部用户侧感知。 - [11:43](https://www.youtube.com/watch?v=-0v6UUPLvF8&t=703s) 在常规发布管道彻底死锁的情况下，工程师利用 2021 年大停机后开发的 Belljar 应急工具箱，带外定位原宿主机并修改 Job Spec 剔除致祸 Feature Flag，成功重新拉起恢复调度器与下游任务。 - [15:53](https://www.youtube.com/watch?v=-0v6UUPLvF8&t=953s) 针对根本隐患，Meta 彻底重构了控制面依赖架构，将 Zelos 提升为与 Twine 调度器同级对等的核心一等公民组件，终结了以普通作业运行带来的自托管循环死锁。 ## Key takeaways - [04:37](https://www.youtube.com/watch?v=-0v6UUPLvF8&t=277s) AI 算力爆发导致宿主机容器密度剧增，Scheduler 与 Agent 的 M×N 全连接网格带来网络流量洪峰，推动了 Agents Manager 抽象层的引入。 - [07:27](https://www.youtube.com/watch?v=-0v6UUPLvF8&t=447s) Twine 的自托管（Self-Hosted）树状架构埋下了脆弱隐患：元数据存储 Zelos 以普通作业运行在 Shard 0 上，导致控制面存在致命的循环依赖路径。 - [07:27](https://www.youtube.com/watch?v=-0v6UUPLvF8&t=447s) 新服务因重启未完成读恢复即对外响应，返回“无容器运行”假象，触发调度器的防御性销毁逻辑，强行抹除了区域元调度器的 Zelos 依赖并锁死控制平面。 - [09:12](https://www.youtube.com/watch?v=-0v6UUPLvF8&t=552s) 常规部署通道因控制面调度系统本身停摆而彻底失效，使得工程师即便定位出问题也无法通过常规手段下发 Feature Flag 回滚。 - [09:52](https://www.youtube.com/watch?v=-0v6UUPLvF8&t=592s) 基于日常常态化演习建立的肌肉记忆，团队通过区域切流（Region Drain）在 20 分钟内平滑隔离故障，将爆炸半径严格锁死在单一故障域内。 - [10:48](https://www.youtube.com/watch?v=-0v6UUPLvF8&t=648s) 严格执行特性开关（Feature Flag）门禁与单区域、单分片灰度发布，是避免单点代码缺陷演变为全球性级联瘫痪的关键工程防线。 - [11:43](https://www.youtube.com/watch?v=-0v6UUPLvF8&t=703s) 极端故障下必须具备“带外（out-of-band）应急工具”储备，Belljar 工具箱让团队能在最小依赖下直接干预单机与容器，绕过上层失能的调度系统。 - [11:43](https://www.youtube.com/watch?v=-0v6UUPLvF8&t=703s) 排障中发现原物理机本地仍缓存着所需二进制文件，避免了在停机高压下跨网拉取历史版本或线上热修（hotfix）带来的巨大时间风险。 - [14:14](https://www.youtube.com/watch?v=-0v6UUPLvF8&t=854s) 系统恢复采用 Belljar 应急容器接管控制权、正规部署管道回滚配置、标准容器交接再逐步回切流量（Undrain）的严谨双轨闭环流程。 - [15:53](https://www.youtube.com/watch?v=-0v6UUPLvF8&t=953s) Meta 将 Zelos 提升为与 Twine 调度器对等的系统一等公民组件，彻底斩断了普通服务托管导致的架构循环死锁。 - [18:08](https://www.youtube.com/watch?v=-0v6UUPLvF8&t=1088s) “所谓运气就是充分准备遇上了契机”：监控基础设施的存活与只需恢复少数关键作业的控制面设计，为系统的级联自愈创造了前提。 - [19:18](https://www.youtube.com/watch?v=-0v6UUPLvF8&t=1158s) 在 AI 辅助生成代码与系统复杂度爆发的时代，人类对系统上下文的理解被稀释，团队必须依靠完备的工具链与底线灾备机制而非侥幸来应对黑天鹅事件。 ## Detailed walkthrough ### [00:05](https://www.youtube.com/watch?v=-0v6UUPLvF8&t=5s) 下班前夕的致命警报：甘特图全面飘红 演讲由 Meta 基础设施工程师 Phil Lopreiato 与 Rahul Iyengar 共同开场。Phil 描绘了一个所有运维与 SRE 人员都不陌生的场景：一天的工作结束了，大家正收拾东西准备离开办公室去参加 Happy Hour（下班小聚）。然而就在此时，警报毫无预兆地响起。办公区工位旁边的监控仪表板瞬间被密密麻麻的错误飘红填满，值班手机因接连触发的最高级别 P0/致命严重警报响个不停。显而易见，某个底层核心系统彻底坏了，必须立刻启动应急响应。 大家把正要走出大门的团队成员拉了回来，直接临时征用了同事 Paul 的会议室作为作战指挥室（War Room）。当他们打开系统监控时，眼前的一幕令人震惊。在 Meta 内部，有一个资深工程师看了都会倒吸一口凉气的甘特图（Gantt Chart）：横轴代表时间，每一条水平线代表单作业（Job）中一个正在运行的任务（Task）。绿色代表健康运行，粉色则代表异常终止或失效。而此时屏幕上显示的，是该数据中心区域的核心主 Web 作业（main web job）——整排整排的绿色在极短时间内骤然断裂，整齐划一地变成了一片粉色，整个作业彻底陷入停滞。 ```mermaid flowchart LR accTitle: Web 任务状态突变甘特图示意 accDescr: 展示单个 Web 任务从绿色的健康运行状态骤变为粉色的完全停止状态 subgraph Timeline["时间轴"] direction LR A["正常阶段: 任务平稳运行<br/>（绿色 / Healthy）"] -->|突发故障告警| B["异常阶段: 任务全面停摆<br/>（粉色 / Stopped）"] end ``` 看到整个区域的 Web 任务瞬间归零，Phil 的脑海立刻闪回到了 2021 年 10 月 4 日 Meta 那场震惊全球的长达数小时的全网大瘫痪。他清楚地记得那一天故障发生时自己身在何处、在干什么，以及随后长达数日痛苦的排查与恢复过程。眼前这个主要 Web 作业在单个区域彻底归零的景象，让他不可避免地产生了一种最深沉的恐惧：2021 年的那场灾难性噩梦可能要重演了，全员必须做好打一场旷日持久恶仗的准备。 ### [01:28](https://www.youtube.com/watch?v=-0v6UUPLvF8&t=88s) Web Foundation 团队与危机处理哲学 Phil 与 Rahul 来自 Meta 的 Web Foundation（Web 基础设施）团队。该团队负责拥有并维护支撑整个 Meta 核心业务流量的主 Web 层（main web tier），因此每当发生影响全站的大型事故时，他们通常都是第一响应者。“如果网站挂了，我们大概率就在现场，并且每次都能讲出一堆惊心动魄的故事。”Phil 解释道，本次分享的主题正是发生在前一年春季的一起严重事故。他们将详述事故的根因、底层基础设施面临的压力、为何没有演变成全局灾难（得益于长达十余年的容灾投资与 2021 年大故障后的整改体系），以及从中吸取的宝贵教训。 Phil 提出了应对复杂故障的核心设计哲学：**在高度复杂的突发事故中，你必须做好所有现有工具和既有假设全都已经失效的心理准备**。 - 系统可能会存在未知的隐式依赖； - 用于管理和修复这些系统的常规网络与访问通道可能已经断开； - 在 2021 年的大事故中，甚至连带外通信（out-of-band）网络通道都挂掉了。这意味着团队必须随时准备“预期未知的意外”，靠创造性的方案绝地求生。 然而，从容应对不是凭空而来的，平时的演练才能建立信心与肌肉记忆。Meta 自 2012 年飓风桑迪（Hurricane Sandy）以来就建立了完善的区域级容灾机制（DR Program），其中最核心的实践是定期的**“风暴演练”（Storm Drills）**：主动将整个数据中心区域的全部生产流量安全排空（Drain），维持一段时间，再平稳切回。这种在刀尖上跳舞的常态化演习，让团队在面临高压实战时有了深厚的技术底气与操作经验。 为了生动描绘大规模分布式系统底层的排障现实，Phil 引用了 James Mickens 在经典文章《The Night Watch》中的一段名言： > “当你调试一个分布式系统或操作系统内核时，你是在以得克萨斯风格行事。你召集几个面色冷峻、坚毅沉稳、亲眼见过事物覆灭的人；带上一些简陋原始的工具——比如指南针、帆布背包和一根削尖的木棍；然后径直走向荒野，去主动寻找麻烦。” 除了“削尖的木棍”，Meta 团队在历次事故后打造了强大的专业工具，其中最具代表性的是 2021 年大停机后孵化的 **Bell Jar 工具包**。Bell Jar 允许工程师在仅保留绝对最小依赖的前提下，对底层基础设施执行极底层的控制和单机/小集群拉起操作。结合成熟的 DR 排水调度能力，这套武器库为他们在悬崖边缘挽救系统提供了可能。 ### [04:37](https://www.youtube.com/watch?v=-0v6UUPLvF8&t=277s) 容器调度巨兽 Twine 与 AI 带来的集群扩展瓶颈 随后，Rahul 接过话筒，深入剖析了本次事故所涉及的底层架构。这起事故的中心是 Meta 内部的容器编排调度平台——**Twine**（Meta 曾就此在 OSDI 发表过顶会论文并在工程博客深度拆解）。为了理清本次故障链条，Rahul 重点介绍了 Twine 的核心架构组件及其分片（Sharded）机制： - **调度器（Scheduler）**：负责编排协调容器端到端的完整生命周期。调度器是分片部署的，每个调度器分片管理约 10 万到 100 万个容器任务。 - **分配器（Allocator）**：负责核心调度决策，回答“每个容器具体应该放置在哪台物理宿主机上”的问题。 - **资源代理（Resource Broker）**：向分配器提供可用的物理资源容量。 - **主机代理（Agent）**：运行在每台宿主机上的单机守护进程，负责本机容器的具体执行与宿主机生命周期管理。 ```mermaid flowchart TD accTitle: Twine 核心组件及其与 Agent Manager 的关系 accDescr: 展示 Scheduler、Allocator、Resource Broker 与宿主机 Agent 之间的交互架构，以及引入 Agent Manager 后的解耦关系 subgraph TwineControlPlane["Twine 控制平面 (Control Plane)"] RB["Resource Broker<br/>(资源容量管理)"] --> Allocator["Allocator<br/>(容器放置决策)"] Scheduler["Schedulers (多分片)<br/>(每分片管理 10w~100w 任务)"] end subgraph AgentCoordination["代理协调层"] AMS["Agents Manager Service<br/>(聚合与解耦层，强一致视图)"] end subgraph HostFleet["集群宿主机层"] A1["Host Agent 1"] A2["Host Agent 2"] An["Host Agent N"] end Scheduler --> AMS AMS --> A1 AMS --> A2 AMS --> An ``` 本次事故的焦点集中在 **Scheduler 与 Agent 之间的交互** 上。近年来，随着人工智能（AI）训练与推理业务的爆炸式增长，Meta 的机房算力规模急剧膨胀，集群物理机数量猛增。与此同时，AI 架构下的单机运行模式往往要求单台宿主机上承载密度极高的容器任务。这意味着宿主机 Agent 的数量以及每个 Agent 上的任务量成倍增加，导致 Scheduler 与众多 Host Agent 之间的通信产生了极其庞大的网络扇出流量（Fan-out Traffic）。为了应付这个规模，集群需要切出更多的 Scheduler 分片，最终在分片调度器与单机 Agent 之间演变成了一个极其臃肿的 $M \times N$ 全连接网格，频繁引发不可控的突发网络流量洪峰。 为了解决这一可扩展性瓶颈，Twine 团队引入了一个全新的中间抽象层——**Agents Manager Service**。它的主要作用有两个： 1. 为所有调度器分片提供一个统一、强一致的单机 Agent 状态视图； 2. 架构拓扑从恶劣的 $M \times N$ 网格直接降维为 $M + N$ 连接，大幅降低了集群控制平面的网络流量压力。 在持久化存储方面，Scheduler 和 Agents Manager 均严重依赖 **Zelos**。Zelos 是 Meta 基于内部无依赖/低依赖表格存储引擎 **Delos** 构建的系统，对外提供与 Apache ZooKeeper 完全兼容的 API。Scheduler 与 Agents Manager 都是严格要求强一致性（Strong Consistency）的系统，它们将所有关键状态写入 Zelos，且**一旦 Zelos 不可用，这两者都将完全无法工作**。 ### [07:27](https://www.youtube.com/watch?v=-0v6UUPLvF8&t=447s) 自托管树状依赖与导致区域冻结的“清空容器”Bug Twine 架构中还有一个至关重要的设计特征：**Twine 是自托管（Self-Hosted）的**。这意味着，Twine 调度器本身也是作为容器运行的，由上一层或另一分片的 Twine 调度器来管理，形成了一个自顶向下的树状控制体系： - 顶层是管理全球的根元调度器（Root Meta Scheduler）； - 区域层级由区域元调度器（Regional Meta Scheduler）管理该数据中心内的所有普通调度器分片； - 每个区域内部运行着多个普通分片调度器（Shard 1, Shard 2 等）。 **但是，致命的循环依赖由此埋下：** 负责提供强一致元数据存储的 **Zelos 实例** 并不是由 Twine 团队独立运维的外部系统，而是以普通作业的形式，直接托管在数据中心区域调度器的 **0 号分片（Shard 0）** 之上！ ```mermaid flowchart TD accTitle: Twine 调度器与 Zelos 自托管依赖树 accDescr: 展示 Root Meta Scheduler、Regional Meta Scheduler、Shard 0 调度器以及运行在其上的 Zelos 之间的依赖关系与循环脆弱点 Root["Root Meta Scheduler<br/>(全球根元调度器)"] --> RMS["Regional Meta Scheduler<br/>(区域元调度器)"] RMS --> S0["Shard 0 Scheduler<br/>(区域 0 号分片)"] RMS --> SN["Shard N Scheduler<br/>(区域其他普通分片)"] S0 -.->|以普通 Job 形式托管运行| Zelos["Zelos 集群 (ZooKeeper API)<br/>(存储强一致元数据)"] Zelos ==>|强一致存储依赖<br/>不可用即死锁| RMS Zelos ==>|强一致存储依赖| S0 ``` 随后，Rahul 揭开了那场险些酿成巨祸的致命 Bug 的具体触发链条： 1. **服务上线与重启**：Twine 团队正在该区域灰度推广上线 Agents Manager Service。上线后不久，该服务进程发生了重启。 2. **读空返回**：重新拉起后的 Agents Manager Service 触发了一个严重代码逻辑缺陷——**它在尚未完成从后端持久层 Zelos 读取恢复已有数据的情况下，就立刻开始响应外部调度器的查询**。 3. **“无容器运行”的假象**：此时，负责管理该区域底层作业的 0 号分片调度器（Shard 0）前来轮询：“请问当前有哪些容器在正常运行？”Agents Manager 脱口而出：“没有任何容器存在。” 4. **触发毁灭性清理**：基于“目前没有任何容器运行，但我本地期望有许多容器在跑”的错误状态判定，0 号分片调度器做出了符合其逻辑但极其致命的决断：它判定需要从头冷启动这些缺失的任务。而在重新启动新容器前，调度器的标准流程会忠实地执行清理逻辑——**强行停止并销毁（Stop & Destroy）当前所知的所有现有“脏/旧”容器**。 5. **核心大脑被物理抹除**：运行在 0 号分片之上的关键服务被直接就地摧毁，其中首当其冲被强行掐死的，恰恰是 **区域元调度器（Regional Meta Scheduler）所依赖的 Zelos 存储集群**！ 失去了底层的 Zelos 强一致存储，区域元调度器瞬间因无法读写元数据而彻底停摆、完全冻结；与此同时，本身包含致命缺陷的 0 号分片调度器陷入半死不活的混乱态。由于负责管理和治愈这片区域的“大脑”已经由于元数据崩溃而死锁，团队陷入了无法通过常规手段下发任何配置变更或修复补丁的绝境。控制平面彻底停摆，多米诺骨牌开始向全区业务猛烈倾倒。 ### [09:12](https://www.youtube.com/watch?v=-0v6UUPLvF8&t=552s) 陷入僵局：调度系统陷入死锁与“瘫痪”状态 在故障发生的核心时刻，整个控制平面的调度链路彻底锁死。负责调度“调度器”本身的元调度器已经完全冻结，这意味着所有常规的配置下发与更新路径彻底失效。更为致命的是，伴随着区域调度器 Zelos 集群（Zelos ensemble）的停止，代理管理服务（Agents Manager Service）的 Zelos 集群也同步停止，导致代理管理服务自身陷入瘫痪，再也无法拉起任何容器。 系统瞬间陷入了“停滞在水中”（dead in the water）的绝对僵局：所有由第 0 分片（zeroth shard）管理的容器全部停止运行，而系统又因“调度调度器的调度器”已死，根本无法通过常规管道回滚引发问题的 Feature Flag。 ```mermaid flowchart TD accTitle: 调度系统控制平面循环死锁示意图 accDescr: 展示了因 Feature Flag 引入依赖死锁，导致调度器与其底层 Zelos 依赖互相卡死、无法自愈的级联故障过程。 subgraph FailureLoop["控制平面死循环瘫痪"] A["下发 Feature Flag 新版本"] --> B["引发循环依赖"] B --> C["Zelos 集群停止运行"] C --> D["代理管理服务 / 调度器冻结"] D --> E["所有第 0 分片容器全部停止"] E -.->|"常规更新通道瘫痪<br/>无法下发回滚"| B end ``` ### [09:52](https://www.youtube.com/watch?v=-0v6UUPLvF8&t=592s) 隔离爆炸半径：果断切流（Region Drain）与 20 分钟止血 在排障现场，团队迅速拉入更多专家成立应对小组，紧盯着日志和监控图表。幸运的是，监控表明故障被严格局限在单一数据中心区域内。这为团队提供了明确的止血手段：执行区域切流（Region Drain）。得益于 Meta 内部每周进行的灾备演练，团队对 Drain 操作有着极高的信心与熟练度。 切流指令下达后，涌向受影响区域的用户请求逐渐平稳归零，所有线上应用和产品流量全被无缝调度至其他数据中心。通过这一操作，用户侧感受到的实际影响在 15 到 20 分钟内便被彻底消除，“希望外界所能感知的就只有这些”。 - **演练保障**：常态化、每周必做的 Region Drain 演习使得团队敢于在第一时间采取激进切流手段。 - **用户感知消除**：事故发生后 15–20 分钟内完成全量引流，保障了外部站点的整体稳定性。 ### [10:48](https://www.youtube.com/watch?v=-0v6UUPLvF8&t=648s) 追根溯源：灰度发布与 Feature Flag 的防护机制 前端流量切除后，团队终于可以沉下心剖析根本原因。日志分析显示，下游任务是被调度器主动关停的。经过深入排查，工程师们理清了那条复杂的循环依赖链，并迅速将其与刚发布的一个版本变更关联起来——该变更在 Feature Flag 保护下启用了新的依赖关系。 这一架构实践拯救了局势：因为变更被置于 Feature Flag 之后，并且严格执行了单区域分阶段灰度（staged rollout），故障的“爆炸半径”（blast radius）被严密锁死在单一区域内，没有蔓延至全球其他区域。但尴尬的是：由于控制面调度系统本身停摆，工程师无法通过常规的部署管道去把这个 Flag 翻转回来。 - **核心工程准则**：“永远要给你的变更加上开关门禁（always gate your changes）”。 - **故障域控制**：分阶段灰度发布确保了未经充分验证的变更仅影响预设的故障域（即单个 Region）。 ### [11:43](https://www.youtube.com/watch?v=-0v6UUPLvF8&t=703s) 绝境求生：启用 Belljar 应急工具箱制作恢复容器 面对无法“自我更新”的调度器，团队不得不拿出专门应对极端情况的恢复工具箱——Belljar，通过带外（out-of-band）方式强行拉起应急恢复容器。整个现场修复分为三个环环相扣的步骤： 1. **寻找历史宿主机**：为了最快地恢复状态，首要任务是找到此前运行故障调度器的物理机。虽然预先设想的低依赖定位路径突发小故障，但好在当时受影响任务体量不大，且切流后内部运维日志系统依然存活，团队通过内部工具成功从日志中手动复原了任务的分布机器。 2. **确认宿主机可执行文件存留**：幸运之神再次眷顾，在找到机器后，原本运行的二进制文件（binary）依然缓存在磁盘上。如果故障持续更久被清理机制删掉，团队就必须经历拉取旧版本甚至线上紧急热修（hotfix）并分发的漫长风险期。 3. **构造纯净指令并修改 Job Spec**：团队利用本地现有二进制文件，构造不带致祸 Feature Flag 的启动命令。虽然 Belljar 工具内置了更改任务启动 Flag 的功能，但初期仅支持追加（append）参数；团队尝试传入第二个相反参数来抵消，发现容易引发不可预知的解析问题，最终通过更稳妥却更繁琐的手动编辑 Job Spec 方式彻底剔除了该 Flag。 ```mermaid flowchart LR accTitle: Belljar 应急带外恢复工作流 accDescr: 展示了通过 Belljar 恢复容器定位机器、复用本地二进制文件、修改 Spec 剔除坏 Flag 并夺权恢复的步骤。 subgraph Stage1["1. 机器与环境确认"] A1["内部日志比对"] --> A2["定位原宿主机"] A2 --> A3["确认磁盘保留二进制文件"] end subgraph Stage2["2. 绕过常规调度启动"] B1["编辑 Job Spec 剔除 Flag"] --> B2["Belljar 拉起恢复容器"] end subgraph Stage3["3. 恢复控制权"] C1["恢复容器获取全局锁"] --> C2["拉起下游停止的服务"] end Stage1 --> Stage2 --> Stage3 ``` 恢复调度器成功拉起后，它们顺利获取了系统的控制锁（control lock），识别出大量处于停止状态的下游任务，并依序将其启动。监控系统原本一片通红粉碎的甘特图，在短时间内迅速翻转为令人心安的大片深绿色。 ### [14:14](https://www.youtube.com/watch?v=-0v6UUPLvF8&t=854s) 双轨交接与平稳回切：解除切流状态 尽管甘特图恢复了代表健康的深绿，但在将外部流量重新导入该区域前，系统还不能直接交付使用。此时团队执行了规范的交接与收尾三步法： 1. **正规通道回滚配置**：恢复调度器正常工作后，团队回到正规部署通道，通过 Meta Meta Scheduler 执行标准 TTY 更新，从全系统配置层面彻底清除残留的 Feature Flag。 2. **容器交接与控制权释放**：通过标准通道在相同宿主机上拉起第二组正规托管的容器。在确认正规容器准备就绪后，手动停止 Belljar 恢复容器；此时正规容器自动接管控制锁，正式接管系统的主导运行权。 3. **逐步引流回切（Undrain）**：在确认所有关键后台服务在调度权交接后运转如常后，团队严格依照成熟的“区域恢复引流手册”（region undrain runbook）启动引流，外部请求平稳回流，数据中心彻底恢复全天候正常运转。 紧张的气氛至此终于散去。讲台上展示出 Phil 当时疲惫但放松的照片——大家终于可以长舒一口气，在群里发发表情包、开开玩笑，放松紧绷的神经，准备奔向那场因事故而延后的周五快乐时光（Happy Hour）。 ### [15:53](https://www.youtube.com/watch?v=-0v6UUPLvF8&t=953s) 复盘与长效改进：拥抱架构重构、去除侥幸 放松过后，团队迅速投入到严肃的复盘中，针对暴露出的人工与工具短板推进深度改进： - **核心架构升级：将 Zelos 提升为 Twine 核心对等组件**：原本 Zelos 是作为普通服务（regular service）运行在调度系统之上的，这导致了致命的循环死锁。团队作出了重大架构权衡：将 Zelos 提升为 Twine 调度生态体系的一等公民核心组件。每个 Zelos 集群（ensemble）直接作为对应调度器的同级对等项（peer）进行调度，而非托管在其上的普通服务。此改动虽给 Zelos 团队带来了巨大的运维与容量管理挑战，但彻底斩断了控制面死锁的可能。 - **消除对“运气”的依赖**：针对事件中无法直观查询“哪台物理机正运行哪个具体调度任务”的问题，团队大幅强化了底层历史机器任务映射日志（historical logging），确保再也不用靠“运气和磁盘残留”来碰运气。 - **常态化演练与自动化验证**：仅有应急工具还不够，团队新增了大量自动化 Belljar 测试，并强制推行月度或季度的人工恢复演习，确保值班工程师在真实的黑天鹅事件中不会是“第一次使用恢复工具”。 - **演练成效验证**：在此次事故之后，Meta 内部又遭遇了两次必须依赖 Belljar 工具箱从零冷启动核心基础设施的险情，这套在实战中淬炼出的灾备体系全部成功化解了危机。随着未来更大规模 AI 基础设施所带来的系统复杂度激增，这种在设计上拥抱不可控、时刻准备“从灰烬中重构控制面”的灾备思想显得至关重要。 ### [18:08](https://www.youtube.com/watch?v=-0v6UUPLvF8&t=1088s) 事故复盘：工程规范、容灾工具与“准备好的运气” 随着技术变更的节奏不断加快，人类对这些复杂变更上下文的理解与认知却在相对减弱，这导致一旦系统发生故障，局势往往会变得极其棘手。在高度自动化的现代基础设施面前，工程师有时依然必须懂得如何回归本源，就像老一辈那样“拿着铁锹和提灯”（"the old-fashioned way with a shovel and a lamp"），在黑暗中手动排除险情。 回顾这次化险为夷的事故处置过程，团队之所以能够避免灾难性的大范围停机，主要得益于三个关键支柱： 首先是严谨的**工程规范（Good engineering）**。团队在功能发布中严格落地了特性开关（Feature flags）机制，并遵循既定流程，仅将改动灰度推送到单个 Region（区域）中的单个 Shard（分片）。这种克制的分阶段上线规范，使得故障爆发后团队能够以极快的速度顺藤摸瓜，精准锁定破坏系统的具体代码提交。 其次是**日常容灾与自动化工具的沉淀**。在经历 2021 年那场席卷全公司的重大断网事故后，Meta 持续投入建设灾难恢复（Disaster Recovery, DR）体系，开发了大量应急工具。在本次事件中，团队能够极为轻松、常规地将流量从故障 Region 迅速排空（drain traffic away），并利用平日储备的应急恢复工具现场拉起控制面。这些工具之所以能在火烧眉毛的关键时刻发挥作用，核心在于团队不仅构建了它们，更在常态化演练中反复验证、磨砺过它们的可靠性。 最后则包含了一点点“运气”。整个基础设施控制面并不需要全量重启，仅仅救活“少数几个关键作业（a handful of jobs）”，整个系统就能产生级联修复效应，而且绝大部分监控基础设施在故障期间始终保持存活。但正如格言所说，“所谓运气，就是充分准备遇上了契机”（"luck is where preparation meets opportunity"）。 ```mermaid flowchart TD accTitle: 三大核心防线支撑系统快速恢复 accDescr: 结构展示防范重大停机的三大支柱：工程规范、容灾工具链与监控自愈 subgraph ENG["1. 规范工程实践"] A["Feature Flags 控制"] B["单 Region 单 Shard 灰度"] A --> C["快速精准定位变更"] B --> C end subgraph TOOL["2. 容灾建设与工具"] D["2021 年重大事故后反思"] E["一键化 Region 流量排空"] F["实战化演练的恢复工具"] D --> E D --> F end subgraph LUCK["3. 监控与自愈契机"] G["核心监控工具未中断"] H["仅需恢复少数几个核心 Job"] G --> I["系统快速自愈联动"] H --> I end C & E & F & I --> J["成功遏制区域雪崩，避免全网瘫痪"] ``` - **灰度隔离机制**：故障严格限制在单个 Region 的特定 Shard 内，未在初始阶段扩散至多地域。 - **流量排空成熟度**：Region 级别的流量抽干已实现平滑化与常规化，为底层紧急抢修争取了隔离环境与时间。 - **关键依赖极小化**：控制平面具备弹性设计，只需恢复极少量的核心 Job 即可撬动整个系统的全流程自愈。 --- ### [19:18](https://www.youtube.com/watch?v=-0v6UUPLvF8&t=1158s) 面对 AI 时代的思考：你的系统是在碰运气还是有备而来？ 演讲在此处引申出了一个面向整个软件工程行业的宏观警示：随着人工智能（AI）技术的迅猛崛起，整个行业的技术迭代与变更速度正在呈现爆发式增长。 在海量代码由模型辅助生成、系统规模爆炸式膨胀的当下，系统变更的数量级急剧攀升，然而人类对这些变更背后细节与上下文的真实理解深度却在被动稀释。当不可避免的系统性崩溃再次来临时，工程师们所面临的未知与混乱将远超以往。 演讲者在结尾向台下所有的架构师与运维工程师提出了一个发人深省的问题：“面对这种失控的复杂度和变革步伐，你是在纯粹依赖虚无飘渺的运气，还是真正拥有能驾驭这一切、化险为夷的完备工具？”（"Are you dependent on luck, or do you have the tooling you need to make this work?"）以此强调了在 AI 时代，系统可观测性、受控灰度机制与底层硬核应急工具链不可替代的基石地位。 ## Notable quotes > [01:28](https://www.youtube.com/watch?v=-0v6UUPLvF8&t=88s) 如果网站挂了，我们大概率就在现场，并且每次都能讲出一堆惊心动魄的故事。 — Phil Lopreiato > [01:28](https://www.youtube.com/watch?v=-0v6UUPLvF8&t=88s) 在高度复杂的突发事故中，你必须做好所有现有工具和既有假设全都已经失效的心理准备。 — Phil Lopreiato > [01:28](https://www.youtube.com/watch?v=-0v6UUPLvF8&t=88s) 当你调试一个分布式系统或操作系统内核时，你是在以得克萨斯风格行事。你召集几个面色冷峻、坚毅沉稳、亲眼见过事物覆灭的人；带上一些简陋原始的工具——比如指南针、帆布背包和一根削尖的木棍；然后径直走向荒野，去主动寻找麻烦。 — James Mickens > [09:52](https://www.youtube.com/watch?v=-0v6UUPLvF8&t=592s) 希望外界所能感知的就只有这些 > [10:48](https://www.youtube.com/watch?v=-0v6UUPLvF8&t=648s) 永远要给你的变更加上开关门禁（always gate your changes） > [18:08](https://www.youtube.com/watch?v=-0v6UUPLvF8&t=1088s) the old-fashioned way with a shovel and a lamp > [18:08](https://www.youtube.com/watch?v=-0v6UUPLvF8&t=1088s) luck is where preparation meets opportunity > [19:18](https://www.youtube.com/watch?v=-0v6UUPLvF8&t=1158s) 面对这种失控的复杂度和变革步伐，你是在纯粹依赖虚无飘渺的运气，还是真正拥有能驾驭这一切、化险为夷的完备工具？（Are you dependent on luck, or do you have the tooling you need to make this work?） ## Mentioned Meta, Phil Lopreiato, Rahul Iyengar, James Mickens, Twine, Agents Manager Service, Zelos, Delos, Apache ZooKeeper, Bell Jar (Belljar), OSDI, Hurricane Sandy ## Video description In March 2025, Meta avoided an major outage that would have resulted in prolonged sitewide unavailability. Instead, we were able to quickly contain the issue in a matter of minutes and avoid the worst case scenario. This is a story of how we re-bootstrapped the infrastructure control plane in an entire region, on the fly, using emergency tooling. It's a showcase of how to put DR-preparedness and calm incident response into practice. And it's a pretty cool story. Want to dive deeper into the content? Read the featured article here: https://atscaleconference.com/almost-fail-tell-stop-the-world/ Learn more about the @Scale conferences here: atscaleconference.com [Source](https://www.youtube.com/watch?v=-0v6UUPLvF8)

[1:28] Phil Lopreiato: 如果网站挂了，我们大概率就在现场，并且每次都能讲出一堆惊心动魄的故事。 [1:28] Phil Lopreiato: 在高度复杂的突发事故中，你必须做好所有现有工具和既有假设全都已经失效的心理准备。 [1:28] James Mickens: 当你调试一个分布式系统或操作系统内核时，你是在以得克萨斯风格行事。你召集几个面色冷峻、坚毅沉稳、亲眼见过事物覆灭的人；带上一些简陋原始的工具——比如指南针、帆布背包和一根削尖的木棍；然后径直走向荒野，去主动寻找麻烦。 [9:52] 希望外界所能感知的就只有这些 [10:48] 永远要给你的变更加上开关门禁（always gate your changes） [18:08] the old-fashioned way with a shovel and a lamp [18:08] luck is where preparation meets opportunity [19:18] 面对这种失控的复杂度和变革步伐，你是在纯粹依赖虚无飘渺的运气，还是真正拥有能驾驭这一切、化险为夷的完备工具？（Are you dependent on luck, or do you have the tooling you need to make this work?）

[2:16] James Mickens: When you debug a distributed system or an OS kernel, you do it Texas style. You gather some mean, stoic people, people who have seen things die, and you get some primitive tools, like a compass and a rucksack and a stick that's pointed on one end, and you walk into the wilderness and look for trouble. [7:27] Agents Manager Service: No containers exist [17:11] Rahul Iyengar: Sometimes you still need to know how to do things the old-fashioned way with a shovel and a lamp. [17:11] Rahul Iyengar: where preparation meets opportunity.

[0:05] Hi everybody. My name is Philip Prado, [0:06] here with Rahul Iyengar, and we're going [0:08] to tell a story about how Meta almost [0:10] had a very bad day. Put yourself in this [0:12] situation. The work day is over. The [0:14] team is starting to pack up to leave the [0:16] office. We're going to head for happy [0:18] hour soon. But all of a sudden, an alert [0:20] fires. [0:21] Dashboards in the desk area light up [0:23] with errors. [0:24] On-call phones start ringing due to [0:26] critical alerts firing at [0:28] Something is broken, we now see, and [0:30] it's time to mobilize our response. [0:33] We round the team back up from their way [0:34] out the door. We commandeer a conference [0:36] room. Thank you, Paul, for lending us [0:38] yours. And we start to investigate. And [0:40] what we see is striking. [0:43] If you've worked on Meta services [0:44] before, this Gantt chart should scare [0:46] you. If you haven't, each horizontal [0:49] line here represents a running task on a [0:51] single job. Green means the task is [0:53] running healthy. Pink means it's not. [0:56] The job here is a main web job in the [0:58] region, and this transition to pink [1:00] means it just stopped. [1:04] My mind jumps back immediately to [1:06] October 4th, 2021, when Meta had a major [1:09] outage. [1:10] I so vividly remember this day, exactly [1:13] when and where I was when it started, [1:15] and the very long day investigating and [1:16] working on recovery that followed. [1:19] Seeing an entire web job come to a [1:20] complete halt made me fear for the [1:22] worst. [1:23] That we are going to have a repeat of [1:24] this day, and we braced ourselves for [1:26] another similarly long one. [1:28] Now is a good time to more properly [1:30] introduce ourselves. I'm Phil, here with [1:32] Rahul, and we're on Meta's web [1:34] foundation team. Web foundation owns and [1:36] operates Meta's main web tier, and we're [1:37] also the primary incident responders for [1:39] major site-wide incidents. [1:41] If the site breaks, chances are we're [1:43] there, and we have a story to tell about [1:45] what happened. [1:47] For the rest of our time together, Rahul [1:48] and I will take you through the [1:49] specifics of an incident that happened [1:51] last spring. We'll discuss first how we [1:53] got here, the root cause of the [1:54] incident, and the stresses on our [1:56] infrastructure that led us to this [1:57] situation, [1:58] how we recovered, and how we were able [2:00] to prevent this incident from cascading [2:01] out of control due to our decades-long [2:03] investment in disaster recovery, as well [2:05] as the follow-ups that were put into [2:06] place after the 2021 outage. And then [2:09] finally, we'll talk about the lessons we [2:11] learned and how we're continuing to make [2:12] our infrastructure even more resilient [2:14] going forwards. [2:16] But first, a design philosophy. [2:18] During complex incidents, we need to [2:20] expect our tools and assumptions are all [2:22] broken somehow. [2:23] Systems might have unexpected [2:25] dependencies. [2:26] The normal access path to manage those [2:27] systems might also not work. [2:30] During the 2021 incident, even the [2:32] out-of-band access path was down. [2:34] We'll have to expect the unexpected here [2:36] and be able to apply creative [2:37] problem-solving to get ourselves out of [2:39] whatever situation we're in. [2:41] However, practice builds confidence and [2:44] competence. [2:45] We are well prepared to regional issues [2:47] thanks to our recurring storm drills, [2:49] where we take all of the traffic out of [2:50] our region safely, wait a little bit, [2:52] and then bring it back safely. [2:54] This experience gives us the confidence [2:56] and the know-how to perform these [2:58] operations during other stressful [3:00] situations. A well-trodden path here [3:02] minimizes the risk of using those tools, [3:05] and the more paths we've tested, the [3:06] more operations we have at our disposal [3:09] during the next incident. [3:11] But sometimes, we often find ourselves [3:13] off the beaten path as well. [3:15] To quote James Mickens in The Night [3:17] Watch, "When you debug a distributed [3:19] system or an OS kernel, you do it Texas [3:21] style. You gather some mean, stoic [3:23] people, people who have seen things die, [3:26] and you get some primitive tools, like a [3:27] compass and a rucksack and a stick [3:29] that's pointed on one end, and you walk [3:31] into the wilderness and you look for [3:32] trouble." [3:33] This paper, I think, perfectly [3:35] characterizes the experience of [3:36] debugging distributed systems at scale, [3:38] and I highly recommend giving it a read. [3:40] Our primary plan, and sometimes even the [3:42] backup plan, don't always work. But with [3:44] the right people, tools, skills, and [3:47] preparation, we can solve almost [3:49] anything. [3:52] In addition to our pointy [3:54] here are some of the tools we'll have at [3:55] our disposal. Some of them, like the [3:57] Bell Jar toolkit, were built in the wake [3:59] of the 2021 outage. [4:01] These tools let us perform low-level [4:02] operations against our infrastructure [4:04] and allow us to run services with the [4:06] absolute minimum set of dependencies. [4:08] Others, like our disaster recovery [4:10] program, date back to Hurricane Sandy in [4:12] 2012. And our long history of ensuring [4:14] the site can run safely in the absence [4:16] of an entire region, and by frequently [4:18] running drills to confirm the state, [4:20] gives us more flexibility when [4:22] mitigating other incidents. [4:24] Both Bell Jar and DR have existing [4:26] engineering blog posts or at scale [4:28] talks, and I recommend giving them a [4:29] look to learn more. [4:31] Now, Rahul will take us through some of [4:32] the details about Twine, our container [4:34] orchestration infra, and how this [4:36] incident came to be. [4:37] >> Thank you, Phil. Now, to understand this [4:39] incident, there's a few components that [4:40] you need to understand. The first of [4:42] these is Twine, which is our container [4:44] orchestration framework. We have [4:46] published multiple blog posts about this [4:48] on the Facebook and Meta Engineering [4:49] blogs as well as the research blog. And [4:52] we have a paper that we have at OSDI [4:54] about this if you'd like much more [4:55] detail than I'm going to go into in this [4:57] talk. [4:59] The key parts of Twine that are relevant [5:01] for this incident are mostly on this [5:02] slide. There are many, many systems [5:05] involved in it, but the key things to [5:07] understand here are almost all of them [5:08] are sharded. We have schedulers, whose [5:11] job it is to coordinate the end-to-end [5:12] task life cycle. We Each of these [5:15] manages about order of 100,000 to order [5:17] of a million tasks. We have the [5:19] allocators, which answer the question of [5:21] where do I place each container? [5:23] Resource broker, which gives allocator [5:25] the capacity to run these containers on, [5:28] as well as the agents, which manage the [5:30] on-host life cycle of each container. [5:32] We do have a blog post about this that [5:34] goes into much more detail if you're [5:35] interested in that. Again, if you look [5:37] for Twine, you'll you will find it. [5:40] The interaction of interest for this [5:42] incident is the one between the [5:45] scheduler and the agent, [5:47] which [5:48] we will go into in a little bit more [5:49] detail in a bit. [5:51] Now, back to present times in this [5:52] incident, what was the thing that [5:53] happened? [5:56] The key problem we were trying to solve [5:57] is that there's a lot of fan out traffic [5:59] between the schedulers and the agents. [6:02] We have multiple shards of the scheduler [6:03] as I mentioned before. We run one agent [6:05] per host and our fleet's been growing. A [6:08] lot of this is to solve issues that come [6:10] with the advent of AI. We've had a [6:12] capacity boom. There's a lot more [6:14] machines we're running in the fleet. The [6:15] architecture of inference and training [6:17] jobs also means that we run a lot more [6:19] tasks per per host, which means that we [6:22] end up with a lot more fan out traffic. [6:24] We need a lot more shards of the [6:25] scheduler to handle this larger scale. [6:28] And so, we end up with this very odd [6:31] burst of traffic that we'd really like [6:33] to avoid. [6:34] To do this, we introduced something [6:35] called the agents manager service. The [6:37] primary value proposition of this is [6:40] that all the schedulers have a [6:41] consistent view of the agents. It has a [6:43] secondary benefit in that it turns what [6:45] was previously an M by N mesh of [6:47] connectivity into an M plus N mesh of [6:50] connectivity, which is much more [6:51] manageable in terms of the amount of [6:53] traffic. [6:56] In terms of persistence, both the [6:58] scheduler and agents manager service [7:00] store all of their data in Zelos. Zelos [7:03] effectively implements the Zookeeper API [7:05] on top of Delos, which is our in-house [7:07] low dependency table store. We do have a [7:10] talk at @Scale about both Zelos and [7:13] Delos. If you're interested in it, you [7:14] can get more details there as well. But, [7:16] the key thing here is that both of these [7:18] are systems that expect strong [7:19] consistency. They store data in Zelos [7:21] and they will not function if Zelos [7:24] isn't functioning properly. [7:27] The other bit of information that's [7:28] important here is the general deployment [7:31] of our schedulers. Twine is self-hosted [7:34] and so every shard of the scheduler is [7:36] managed by another shard of the [7:38] scheduler. We sort of have a tree of [7:39] them. There are multiple shards per [7:41] region, each of which is managed by the [7:43] regional meta scheduler, which is then [7:45] managed by a root meta scheduler which [7:47] handles the entire world. Zelos, [7:49] however, as you'll notice here in gray, [7:51] is not managed by the Twine team. It is [7:55] run as a regular job under the region [7:57] schedulers. In this case, the zeroth [7:59] shard. [8:00] Now, back to the bug. What actually [8:02] happened during this incident? [8:04] The trigger for this incident was that [8:05] we rolled out the use of agents manager [8:07] service. [8:09] And one of the things that happened [8:11] right after that is that agents manager [8:12] service restarted. When it restarted [8:15] without reading data from Zelos, it [8:17] immediately responded to the scheduler. [8:19] So, when the scheduler asked, "Hey, [8:21] which containers are running?" it [8:23] responded, "No containers exist." [8:25] At this point, the scheduler concludes [8:27] that it needs to start all the [8:28] containers, which is reasonable given [8:30] the information that it was given. [8:32] And as it is going to dutifully do, it [8:35] runs all of its cleanup logic, including [8:37] stopping and destroying all containers [8:40] before it starts them back up again. [8:44] In this case, the bug hit the zeroth [8:47] shard of the region, which stopped the [8:50] Zelos ensemble for the meta region [8:52] scheduler. And as I've told you earlier, [8:54] this means that the meta region [8:56] scheduler stops functioning because it [8:59] needs Zelos to work due to its strong [9:01] consistency guarantees. [9:03] Now, at this point, the meta region [9:05] scheduler is frozen, the zeroth shard [9:07] scheduler has the bug in it, and we [9:10] can't do anything about that bug because [9:12] the scheduler that schedules the [9:13] scheduler is frozen. At this point, we [9:15] basically say that all standard update [9:17] paths cease to work. [9:20] The other key thing is that along with [9:23] the regional schedulers Zelos ensemble, [9:25] the agents manager services Zelos [9:27] ensemble was also stopped, which meant [9:29] that agents manager service itself was [9:31] frozen and could no longer start [9:33] containers back up, putting us in a sort [9:35] of dead in the water state here. [9:38] So, at this point, the state of the [9:39] world is that all containers managed by [9:41] the zeroth shard are stopped. [9:44] And we cannot revert the flag because [9:46] the scheduler's scheduler is frozen. [9:49] What do we do here? Phil will tell you [9:51] more. [9:52] >> Now, let's take a step back into the [9:54] moment. We have the team investigating. [9:56] We're escalating to more people. We're [9:58] bringing them into the room as we need. [9:59] We're looking at graphs and logs. We're [10:01] trying to figure out what's going on. [10:03] The first thing we see is that the issue [10:04] seems contained to just one region. [10:07] This is fortunate. It means we have a [10:08] way to contain the damage. You drain the [10:11] region. [10:12] Draining is an operation that we are [10:14] confident taking due to all of our [10:15] frequent drills. [10:17] We know how to drain a region. We do it [10:18] every week, and we know it's safe. [10:21] After we issue the drain, requests [10:23] flowing into the affected region slowly [10:24] come to a stop, and then all the apps [10:27] and products are served by other data [10:28] centers. [10:29] This technique lets us resolve the [10:31] user-facing impact of the incident [10:33] within the first 15 to 20 minutes, which [10:35] is really great. And hopefully, that's [10:37] all you saw externally. [10:41] Now that the site was fine for the most [10:42] part, we could focus on understanding [10:44] what happened and how to fix it [10:46] properly. [10:48] From reading the logs, we could see that [10:50] the tasks were stopped by the scheduler. [10:52] The team does a lot of investigation, [10:54] and they're able to untangle the [10:55] dependency cycle that Rahul talked about [10:57] earlier. [10:58] We were then able to correlate this [10:59] issue to a release, which enabled the [11:01] new dependency behind a feature flag. [11:04] This is an excellent practice, and it [11:06] really saved us here because we could [11:08] now revert the flag and undo the change. [11:11] The lesson to take away is always gate [11:12] your changes. [11:14] This release had also only progressed [11:16] through one region, and we knew it [11:17] wouldn't progress farther once we [11:18] stopped it. This is another great [11:20] practice. By doing a staged rollout, it [11:23] means we can limit the blast radius of [11:24] an unaffected change to a single region, [11:27] which is already in assumed fault domain [11:29] in the system. [11:31] However, the scheduler itself is broken [11:32] here, so we can't update ourselves out [11:34] of it like we've discussed. [11:37] So, we now need to take out our recovery [11:40] toolkit and think about a way to [11:41] creatively solve this problem. [11:43] We know we need to update the scheduler [11:45] to remove this flag and revert the [11:46] change, but we can't use the scheduler [11:48] itself. [11:50] We can use our Bel Jar toolkit to start [11:52] recovery containers. The first thing we [11:54] have to do is find the host that used to [11:57] be running the scheduler, but now are no [11:59] longer. Recovery is easiest here if we [12:01] can find the hosts that used to be [12:03] running this because they might still [12:04] have some state on the hosts that we can [12:06] use to then re-bootstrap things. [12:09] Here, we ended up needing to use some [12:11] internal tools to look at our logs to [12:13] find where the tasks used to be running [12:15] since our expected low dependency way of [12:17] doing this didn't quite work the way [12:18] that we needed to. [12:20] Thankfully, the jobs in question were [12:21] small and our internal tools were still [12:24] running post drain, so we were able to [12:26] reconstruct them manually. [12:28] The next step is to make sure that the [12:29] binary we need to run is present on the [12:31] hosts. [12:33] After we found the machines, they still [12:34] had the binary that they used to be [12:36] running there. [12:38] This is another case of luck where if [12:40] more time had elapsed along the way or [12:42] if the breakage were different or if we [12:44] couldn't find the hosts, it might not [12:45] have already been there and we need to [12:47] then fetch an old version or even worse, [12:49] hotfix something forward and then [12:51] distribute it. However, none of that was [12:53] needed and we had the binary we needed [12:55] already present. [12:58] Step three is to start a recovery [12:59] container. [13:00] We could then construct the command to [13:02] run that binary but without the flag [13:04] that added in the new dependency. [13:07] In this case, we were able to just reuse [13:09] everything already on the host and [13:11] simply delete the flag from the spec and [13:13] that would put us back to where we [13:14] wanted to be. [13:16] The Bel Jar toolkit even has a way to [13:18] change flags running in a job. [13:20] However, it only supported append [13:22] initially. [13:23] We then tried to pass a second flag to [13:25] negate the value of the first, but that [13:26] got kind of more complicated and didn't [13:28] quite work as well. [13:29] Ultimately, we did the more complicated [13:31] thing of editing the job spec to get set [13:33] the flags to the right value in the [13:34] first place, [13:36] which was more complicated, but did [13:37] ultimately work. And then, the [13:39] schedulers could start up, and then all [13:41] the downstream tasks could start running [13:43] again. [13:45] Here, [13:46] we now see that the web jobs are back up [13:49] and running to be green. [13:50] After the recovery schedulers started [13:51] up, they were able to take the control [13:53] lock for the system, see that they had a [13:56] bunch of downstream jobs that needed to [13:57] be started, and then start them [13:59] normally. [14:00] Which then results in this happy-looking [14:02] Gantt chart, where we go from pink, [14:04] which is the original state of stopped, [14:06] back into dark green, which is running [14:08] and healthy. [14:09] But, these jobs aren't yet serving [14:10] traffic because we drained it away. [14:14] That is the next step, fully restoring [14:16] operation of the site. [14:19] Once again, it'll be a three-step [14:20] process. [14:21] Now, we can actually do our flag revert [14:24] the normal way, and do a TTY update on [14:26] the meta meta scheduler to remove the [14:28] flag from all downstream systems that [14:30] might still have it lingering around [14:31] somewhere. [14:33] This will start up a second set of [14:34] containers on those same hosts, but this [14:36] way fully managed the normal way. [14:38] But, they're not yet taking the lock to [14:40] be authoritative yet. [14:41] Once we've verified that the second real [14:43] containers are running, we can then stop [14:46] the recovery containers, which will let [14:48] the real ones take over the lock, and [14:49] then take over normal operation. [14:52] After a little bit more time to verify [14:53] all the other services are still running [14:55] normally after the scheduler handoff, [14:57] we can kick off our well-trodden region [14:59] undrain runbook to bring traffic back [15:01] into the region, and this fully restores [15:03] normal operation everywhere. [15:06] Now, [15:07] we can take a deep breath. [15:09] Here's a real picture of me from that [15:10] day at about this point in time. Here, [15:13] the jobs have been recovered, and we're [15:14] just waiting out recovery. [15:17] Now is maybe the time to take a deep [15:18] breath and relax a little bit after a [15:20] stressful kind of time before that while [15:21] dealing with the recovery. Maybe make [15:23] some memes, make some jokes, [15:25] and lighten the mood a bit. The team [15:27] wants to get back off to that happy hour [15:29] we were heading towards originally. So, [15:30] we want to make sure we can, you know, [15:31] get out of there once everything's [15:32] recovered fully. [15:34] However, before we do that and before [15:36] too long elapses, it's important to take [15:38] a moment to look back and to think of [15:40] the follow-ups that we want to make sure [15:41] we get done, [15:43] what architectural changes we want to [15:44] make, and what tooling improvements we [15:46] wish we had, or areas that didn't quite [15:48] work that well. Or who would take us [15:50] through those follow-ups and what we did [15:52] about them next. [15:53] >> Thanks, Phil. Now, to prevent incidents [15:55] like this from being as bad as they are [15:57] in the future, we had to make a few [15:58] changes. The first one is a set of [16:01] architectural fixes we've made. We know [16:03] that the AI train is not stopping, and [16:05] we know that more complexity is coming. [16:07] So, we had to improve the core of the [16:08] infrastructure. [16:11] As you'll remember earlier, we talked [16:12] about the various shards of the [16:14] scheduler and the meta region schedulers [16:16] that we run, [16:17] and the fact that Zela's is run as a [16:19] regular service. [16:21] This is one of the biggest changes we [16:22] made. We promoted Zela's to being a [16:24] critical part of the Twine ecosystem. [16:27] This means that we schedule each Zela's [16:30] ensemble as a peer of the scheduler that [16:32] uses it, rather than being run as a [16:34] regular service. This does have pretty [16:36] substantial operational and capacity [16:38] management implications for the Zela's [16:40] team, but overall, it is the right [16:43] trade-off for our infrastructure. [16:45] The second bit [16:47] is to make sure that all of our tools [16:48] work exactly when we need them to. [16:51] There were some bits of missing glue [16:52] during this incident where we struggled [16:54] to get a good understanding of which [16:56] hosts were running which specific tasks [16:58] in order to be able to use the tooling [17:00] to bring up the recovery versions. [17:02] We improved historical logging of this [17:04] so that we're substantially less [17:06] dependent on luck. [17:08] We like luck, but we don't like [17:09] depending on it here. [17:11] Another follow-up that we had is more [17:13] drills. There are two angles that we [17:15] took here. The first one is to improve [17:17] the drills themselves by adding a bunch [17:20] of Bel Jar testing. We do have a blog [17:22] post about this and you can read more [17:23] about it online. [17:25] We also added automatic runs of these [17:27] Bel Jar tests to ensure that the [17:29] recovery tooling works correctly. [17:31] And on top of this, we added either [17:33] quarterly or monthly drills for the [17:36] humans to operate these tools and make [17:38] sure that the recovery tooling is [17:40] something that they're familiar with and [17:41] not something they're using for the [17:42] first time during an incident. [17:45] The key thing for us to keep in mind is [17:47] incidents like this will happen again. [17:49] Since the incident we're talking about [17:50] today, we've had two more cases where [17:53] we've had to use this recovery tooling [17:55] to bring up some core infrastructure [17:57] from scratch. [17:59] And the last thing, [18:01] the winds of change are coming for us. [18:03] Be prepared for AI and the deluge of [18:04] changes that it brings, but more [18:06] importantly, [18:08] the pace of change is going up while the [18:10] context of human understanding of these [18:12] changes going down [18:14] means that [18:15] everything is a lot trickier when it [18:17] does break. And sometimes you still need [18:20] to know how to do things the [18:21] old-fashioned way with a shovel and a [18:23] lamp. [18:25] Looking back at this incident, there's a [18:27] few things that worked out pretty well [18:28] for us. The first is good engineering. [18:30] We had feature flags for this feature. [18:32] We rolled it out to one shard of one [18:34] region and we had a defined process for [18:37] this roll out. So, it was fairly easy [18:39] for us to figure out exactly what change [18:41] it was that broke things. [18:43] The second is as follow-ups to our large [18:45] incident in 2021 as well as our disaster [18:48] recovery program, we built a lot of [18:50] tooling that makes things easier. We had [18:52] a way to drain traffic away from a [18:54] region fairly trivially. We had the [18:56] recovery tooling that we used in this [18:58] case. [18:59] And we know that both of those work [19:01] because we used them and exercised them [19:03] regularly. [19:04] The last thing is a little bit of luck. [19:06] We only needed a handful of jobs to [19:08] recover for the whole system to recover [19:11] and most of our monitoring tooling still [19:13] worked. [19:14] As they say, luck is where preparation [19:16] meets opportunity. [19:18] In this context, I want everyone to [19:19] think again about what this means in the [19:21] pace of with the pace of change coming [19:24] to our industry with the advent of AI. [19:27] There's substantially more changes [19:28] happening, and people know a lot less [19:30] about it. [19:31] Are you dependent on luck, or do you [19:33] have the tooling you need to make this [19:34] work? [19:36] Thank you.