Meta Deploys AI Agents to Automate Network Capacity Planning

@Scale

Meta is deploying AI agents to triage traffic anomalies, fix data inconsistencies, and log human decision traces so network capacity planning continuously improves from past outcomes rather than relying on static long-range forecasts.

AI-assisted summary of Live from the SCCC: The Plan That Learns | Ben Vallis, Meta — original source. Check the original for full context.

Source published: 2026-09-13

Detailed notes

## Background Published on YouTube by @Scale on August 31, 2026, this page features Meta's Ben Vallis delivering a lightning talk titled "The Plan That Learns" at the Santa Clara Convention Center. According to the page description, the presentation focuses on applying AI agents to high-stakes network capacity planning decisions—specifically acting inside narrow buy windows, standardizing data across systems, and generating costed menus of timing-sensitive levers.

## TL;DR Ben Vallis argues that network capacity planning is inherently asymmetrical, where underforecasting creates catastrophic platform bottlenecks while overforecasting is relatively benign. To avoid getting cornered by unforecastable black swan events and correlated demand surges, Meta uses AI agents across three key operational pillars: triaging hundreds of noisy telemetry anomalies down to actionable signals, elevating data accuracy past 80% through an evaluation and enrichment loop, and modeling complex hardware and software mitigation levers in hours rather than months. Ultimately, the system captures human overrides as structured decision traces, allowing institutional judgment to compound over multi-year horizons into a continuously learning feedback loop.

## Atmosphere & style The presentation carries the pragmatic, grounded tone of an experienced systems engineer confronting the unpredictable nature of massive-scale infrastructure. Vallis speaks with candid humility about the certainty of being wrong, blending strategic chess analogies with high-stakes operational realities. The dynamic feels authoritative yet accessible, channeling an analytical focus that transforms complex network trade-offs into an engaging discussion on AI-driven resilience.

## Key takeaways - [00:04](https://www.youtube.com/watch?v=3IteDwOdNyk&t=4s) Capacity planners must commit network resources years in advance while accepting that long-range demand forecasts are guaranteed to be wrong. - [00:04](https://www.youtube.com/watch?v=3IteDwOdNyk&t=4s) Forecast errors are asymmetrical in tech infrastructure: overbuilding capacity is a minor issue, but underforecasting bottlenecks the entire service platform. - [00:04](https://www.youtube.com/watch?v=3IteDwOdNyk&t=4s) Massive upside demand shocks are driven by spontaneous concurrency events (e.g., global news like the Queen's passing) rather than predictable calendar events like New Year's Eve. - [00:04](https://www.youtube.com/watch?v=3IteDwOdNyk&t=4s) The core strategy of modern network planning is positioning capacity to withstand a wide range of forecast failures without getting backed into a corner. - [03:40](https://www.youtube.com/watch?v=3IteDwOdNyk&t=220s) Real-world supply chain constraints prevent companies from simply over-provisioning hardware to match maximum demand estimates. - [03:40](https://www.youtube.com/watch?v=3IteDwOdNyk&t=220s) AI agents reduce signal triage time from weeks down to minutes by correlating telemetry, parsing news, and traversing codebases to isolate true anomalies. - [03:40](https://www.youtube.com/watch?v=3IteDwOdNyk&t=220s) Improving AI data trustability from 30% to over 80% requires an iterative measure-enrich-regrade loop and single sources of truth. - [03:40](https://www.youtube.com/watch?v=3IteDwOdNyk&t=220s) AI compresses the evaluation of costed hardware and software mitigation levers from months down to hours. - [08:06](https://www.youtube.com/watch?v=3IteDwOdNyk&t=486s) Human engineering overrides of AI recommendations are treated as valuable unencoded judgment rather than compliance errors. - [08:06](https://www.youtube.com/watch?v=3IteDwOdNyk&t=486s) Structured decision traces log human reasoning alongside downstream outcomes to build an empirical memory of long-term trade-offs. - [08:06](https://www.youtube.com/watch?v=3IteDwOdNyk&t=486s) Capturing decision histories transforms capacity planning into a self-improving system where institutional knowledge compounds over time.

## Detailed walkthrough ### [00:04](https://www.youtube.com/watch?v=3IteDwOdNyk&t=4s) The Asymmetry of Capacity Planning and Surprises

Ben Vallis opens his talk by sharing a fundamental reality of his job: waking up every single morning knowing he is guaranteed to be wrong. As someone who has spent the better part of a decade positioning network infrastructure at Meta against unexpected surges, he views this continuous uncertainty as an unavoidable occupational hazard. Capacity planners must commit resources and make decisions today to intercept demand levels and unforeseen events years into the future. Over time, he has grown comfortable with this constant margin of error by coming to appreciate the profound asymmetry hidden within forecast graphs—while demand curves look symmetrical, the real-world consequences are anything but.

Tracing a line on the upside of a forecast and overbuilding capacity is a relatively harmless mistake in a high-growth technology business; it simply means the infrastructure was deployed a bit early. However, the inverse scenario is catastrophic: underforecasting demand leaves the network underbuilt, creating severe bottlenecks across the entire business. Consequently, planning teams are intensely sensitive to upside demand shocks. These upside spikes are driven by concurrency—tens or hundreds of millions of users hopping online simultaneously to perform the exact same action. While predictable calendar events like New Year's Eve can be planned for, true black swan events, such as the global reaction to the passing of Queen Elizabeth II across the UK, Canada, and Australia, create massive, un-forecastable traffic spikes. Realizing that predicting geopolitical events is a "fool's errand," Vallis flipped the core problem on its head: rather than chasing impossible forecasting accuracy, the real challenge is positioning network capacity so robustly that it covers the largest possible range of wrongness without ever getting backed into a corner or sacrificing user quality.

- **Consequences of Misestimate**: Overforecasting leads to early capacity (low impact in high-growth environments); underforecasting bottlenecks the entire service platform. - **Concurrency Drivers**: Spontaneous mega-events (e.g., global news like the Queen's passing) versus predictable spikes (e.g., New Year's Eve messaging/video calls). - **Core Philosophy**: Avoid getting "cornered" in a chess-like capacity game; design systems to remain robust against the widest spectrum of forecast failures.

---

[03:40](https://www.youtube.com/watch?v=3IteDwOdNyk&t=220s) Three AI Levers: Acting in Time, Fixing Data, and Augmenting Decisions

Addressing a common counter-proposal—simply buying maximum hardware to match the top line of the uncertainty cone—Vallis notes that real-world supply chain constraints make it impossible to get precisely what is needed, where it is needed, or when it is needed. Capacity management is fundamentally a constrained optimization problem. To navigate this, Meta relies on AI across three core functional areas: acting in time, addressing unglamorous data foundations, and augmenting human engineering judgment.

First, AI enables teams to act within critical time windows. Meta's network observes hundreds of novel traffic anomalies weekly, ranging from benign cat video virality and routine codebase updates to internet routing shifts or random noise. Historically, manually triaging each dot on the metric graph to determine root cause and distinguish structural shifts from ephemeral noise took engineers weeks or months, often causing them to miss key action windows. AI agents excel at traversing massive codebases, cross-referencing global news, and correlating disparate telemetry, successfully whittling down hundreds of noisy signals into a manageable shortlist of actionable events. Second, AI acts as a forcing function to fix broken data foundations. Initially, asking different models basic questions like "how much capacity do I have?" yielded three conflicting answers depending on prompt phrasing or time of day. To eliminate these shaky foundations, Meta established a standardized evaluation suite. Early accuracy hovered around 30% ($\pm$ a few hundred basis points across models). Through an iterative "measure-enrich-regrade" loop—adding context, clarifying definitions, and establishing single sources of truth—they rapidly boosted model trustability past 80%. Third, AI dramatically accelerates decision-making when selecting from complex mitigation levers (e.g., pulling supply from warehouses, rerouting hardware in transit, shaping demand via software, or relaxing SLOs), compressing analysis time from weeks or months down to hours.

---

[08:06](https://www.youtube.com/watch?v=3IteDwOdNyk&t=486s) The Plan That Learns: Decision Traces and Compounding Judgment

While generating automated menus of network actions is a major milestone, Vallis emphasizes that the real frontier is moving from a good system to a great system through "The Plan That Learns." In daily operations, human engineers often deviate from model recommendations when choosing the next network move. Rather than seeing this variance as a flaw or compliance issue, Vallis views human deviation as a valuable "gift" representing unencoded judgment—decades of institutional knowledge and intuitive trade-offs that engineers apply on the fly.

Historically, this human reasoning was discarded or buried in unorganized documentation after a decision was executed. To capture it, Meta is implementing structured "decision traces." This framework logs every operational decision alongside its underlying reasoning and tracks the actual downstream outcome over subsequent months and years. By analyzing these decision traces at scale across counterfactuals and repeated scenarios, the platform builds an empirical memory. Future recommendation engines can then explicitly inform planners: "70% of the time when a similar situation occurred, Option B yielded the optimal outcome." By capturing unencoded human intuition and measuring long-term results, capacity planning transforms into a continuously learning system where engineering judgment compounds over time instead of resetting with every new planner.

- **Unencoded Judgment**: Human overrides of AI recommendations are captured as high-value training inputs rather than errors. - **Decision Traces**: Historical choices are linked to real-world outcomes over multi-year horizons to systematically evaluate past trade-offs. - **Long-term Vision**: Network capacity decisions transition from static static forecasts to a dynamic, self-improving feedback loop.

## Notable quotes > [00:00](https://www.youtube.com/watch?v=3IteDwOdNyk&t=0s) The Plan That Learns — Ben Vallis

> capacity is committed years ahead, every long-range forecast is guaranteed to be wrong, and the costliest surprises are correlated demand shocks that defeat traditional risk-pooling.

> not to chase a more accurate forecast, but to position capacity so it keeps our options open no matter how demand breaks.

> acting inside narrow buy windows (triaging signals continuously to surface the handful of truly novel sites, weeks down to minutes), trusting data across systems that disagree on even basic definitions (a measure-enrich-regrade loop), and choosing among coupled, timing-sensitive levers (a ranked, costed menu of moves with the reasoning attached).

> agents that remember every decision and how it turned out, so human judgment compounds over time instead of resetting with each planner.

> [03:40](https://www.youtube.com/watch?v=3IteDwOdNyk&t=220s) how much capacity do I have? — Ben Vallis

> [08:06](https://www.youtube.com/watch?v=3IteDwOdNyk&t=486s) 70% of the time when a similar situation occurred, Option B yielded the optimal outcome. — Ben Vallis

## Mentioned Ben Vallis, Meta, Santa Clara Convention Center, @Scale, YouTube, Queen Elizabeth II, UK, Canada, Australia

[Source](https://www.youtube.com/watch?v=3IteDwOdNyk)

中文简报

Meta部署AI智能体实现网络容量规划自动化 Meta正在部署AI智能体来排查流量异常、修复数据不一致，并记录人类决策轨迹，从而使网络容量规划能够基于以往结果持续改进，而非依赖静态的长期预测。 Ben Vallis认为，网络容量规划本质上具有不对称性，预测不足会导致灾难性的平台瓶颈，而预测过量带来的影响则相对温和。为了避免因无法预测的黑天鹅事件及关联的需求暴增而陷入困境，Meta在三大核心运营支柱中采用了AI智能体：将数百个嘈杂的遥测异常甄别筛选为可采取行动的有效信号，通过评估与丰富闭环将数据准确率提升至80%以上，并在数小时而非数月内对复杂的软硬件缓解杠杆进行建模。最终，该系统将人工干预记录为结构化的决策轨迹，使机构判断力能够在数年跨度中持续沉淀，形成一个不断学习的反馈闭环。容量规划具有严重的不对称性，在高速增长业务中过度建设影响较小，而低估需求则会导致整个服务平台面临灾难性瓶颈，因此核心策略是合理布局容量以抵御最大范围的预测失效。 (0:04) 受供应链物理限制无法单纯靠超量采购硬件来应对峰值，Meta 借助 AI Agent 跨代码库、新闻和遥测数据排查异常，将信号分流时间从数周缩减至数分钟。 (3:40) 通过“测量-丰富-重新评分”的循环优化并建立单一数据源，Meta 将模型在基础容量查询上的可信度从最初的约 30% 提升至 80% 以上。 (3:40) AI 将软硬件缓解措施的成本与风险评估周期从数月压缩至数小时，能够快速生成带推理依据的分级应对方案。 (3:40) Meta 构建了“会学习的计划”，通过记录工程师背离模型建议的决策轨迹及后续真实结果，让未编码的人工经验沉淀为系统经验，实现工程判断力的长期复利。 (8:06)

Reading summary

Ben Vallis argues that because capacity planning suffers from an inherent asymmetry—where underforecasting creates catastrophic bottlenecks while overforecasting merely deploys capacity early—teams must focus on structural resilience rather than impossible forecast accuracy. To manage physical supply constraints, Meta deploys AI to triage operational anomalies, standardize fragmented data, and capture human overrides in multi-year decision traces that build compounding institutional memory. Asymmetric Forecasting Risks and Concurrency Shocks (0:04) Vallis explains that capacity planners inevitably work with erroneous projections because infrastructure investments must be committed years in advance. However, the consequences of forecast errors are fundamentally asymmetric: overbuilding infrastructure is relatively benign in a high-growth environment, whereas underforecasting risks critical, platform-wide bottlenecks. This exposure makes planning intensely sensitive to upside demand shocks driven by sudden user concurrency. While predictable calendar peaks like New Year's Eve can be modeled, spontaneous geopolitical and cultural events—such as the surge in traffic across the UK, Canada, and Australia following the death of Queen Elizabeth II—are impossible to forecast. Recognizing that predicting global events is futile, Vallis reorients capacity strategy from pursuing narrow forecast precision to positioning network capacity so robustly that it accommodates a wide margin of error without degrading user experience or backing operations into a corner. Accelerating Triage and Repairing Data Baselines (3:40) Because real-world supply chain constraints prevent organizations from simply purchasing maximum hardware to cover the entire uncertainty cone, capacity management operates as a constrained optimization problem. Meta applies AI across three primary operational levers to manage these constraints: anomaly triage, data foundational repair, and mitigation modeling. First, AI agents compress anomaly triage from weeks or months down to minutes by correlating code updates, news events, and telemetry across hundreds of weekly traffic spikes to isolate true signals from background noise. Second, deploying AI served as a forcing function to fix broken operational data. Initial tests showed that queries regarding available capacity yielded contradictory answers, achieving only around 30% accuracy across standard evaluation suites. Through an iterative loop of adding context, clarifying definitions, and establishing single sources of truth, Meta raised model trustability above 80%. Third, automated systems model hardware and software trade-offs—such as rerouting in-transit hardware or adjusting service-level objectives—compressing mitigation planning from months to hours. Institutional Memory Through Decision Traces (8:06) Although automated action menus streamline response options, Vallis identifies human engineering overrides as the primary catalyst for advancing system intelligence. When engineers reject algorithmic recommendations, they apply unencoded institutional knowledge and intuitive trade-offs that models lack. Rather than penalizing this variance as non-compliance, Meta logs each override and its underlying rationale as structured 'decision traces.' By linking these decision traces to actual downstream outcomes over subsequent months and years, the system builds an empirical record of real-world trade-offs and counterfactuals. This continuous feedback loop allows future recommendation engines to inform planners of historical success rates under analogous conditions, converting ephemeral engineering intuition into a durable, compounding institutional memory. Vallis concludes that long-term capacity resilience requires shifting from static forecasting to dynamic, self-improving feedback loops that systematically encode human operational judgment. However, the system's ongoing reliability depends on maintaining rigorous multi-year outcome tracking and verifying that historical decision patterns remain valid as network architectures and external demand dynamics evolve.

精炼阅读摘要

Meta 容量规划专家 Ben Vallis 认为，网络容量预测面临严重的后果不对称性，追求完全精准预测突发事件并不现实；团队应当将规划视为受供应链约束的优化问题，借助 AI 应对时效与数据质量瓶颈，并通过“决策追踪”（Decision Traces）将工程师未编码的实战经验转化为可自我进化的系统。预测不对称性与容错布局 (0:04) Ben Vallis 指出，容量规划工程师必须在当下做出资源承诺并拍板决策，以承接数年之后不可预知的流量水平和突发状况，因此每天都要面对预测必然出错的职业现实。在高速增长的技术业务中，预测曲线在图表上看似对称，但真实后果极度不对称：若高估需求而导致网络超前建设，仅意味着硬件部署稍早，商业影响相对轻微；但若低估需求导致网络建设滞后，则会引发波及整个业务平台的严重瓶颈。因此，规划团队对正向需求激增具有极高的敏感度。这类激增往往由海量用户同时在线执行相同操作的极高并发引起。尽管跨年夜等日历事件属于可提前规划的常规峰值，但诸如英国女王伊丽莎白二世去世在英国、加拿大与澳大利亚引发的突发反应等黑天鹅事件，会带来完全无法预先建模的庞大流量。Vallis 认为预测地缘政治事件是徒劳之举，因此 Meta 转变了解题重心：不再追求不可能实现的预测精度，而是建立极具鲁棒性的容量布局，使其能包容最大范围的预测失误，避免在容量博弈中陷入被动局面或牺牲用户体验质量。 AI 赋能时效与数据基座治理 (3:40) 针对直接按不确定性预测上限大量采购硬件的建议，Vallis 指出真实的供应链限制使得团队无法在所需的时间与地点获取全部所需资源，容量管理本质上是一个带约束条件的优化问题。为此，Meta 首先利用 AI 智能体在关键时间窗口内快速行动。其网络每周都会监测到数百起新型流量异常，成因涵盖猫咪视频走红、常规代码变更到互联网路由切换等各类别。以往工程师人工定位根本原因并区分结构性转变与短暂噪音需要数周甚至数月，极易错失干预窗口；AI 智能体通过遍历大型代码库、关联全球新闻并交叉比对跨系统遥测数据，能迅速将庞杂噪音收窄为可操作的事件短清单。其次，AI 成为倒逼修复脆弱数据底座的关键动力。初期，不同模型对“当前拥有多少容量”等基础提问会给出相互矛盾的回答，标准评估套件测得的准确率仅在 30% 左右（不同模型间浮动数百个基点）。Meta 通过“测量-丰富-重新评判”（measure-enrich-regrade）的迭代闭环，持续补充上下文环境、明晰概念定义并确立单一事实来源，成功将模型可信度提升至 80% 以上。决策追踪与工程经验复利 (8:06) 在应对手段的选择上，AI 将评估从仓库调拨物资、调整在途硬件路由、利用软件塑造需求或放宽服务等级目标（SLO）等手段的分析周期，从数周或数月大幅压缩至数小时，自动输出附带成本与风险排序的决策菜单。然而，Vallis 强调更高维度的突破在于实现“会学习的规划”（The Plan That Learns）。在实际运维中，人类工程师经常会偏离模型给出的推荐路径；Vallis 并不将这种人为偏离视为系统缺陷或合规问题，而是将其视作宝贵的“未编码经验”——它代表着工程师现场调用的数十年组织机构知识与直觉权衡。以往这些人类决策逻辑在执行完毕后往往被丢弃或散落在文档中。Meta 正在推行结构化的“决策追踪”（decision traces）框架，记录每项操作决策及其背后的推演理由，并在随后的数月乃至数年中持续跟踪现实世界的下游结果。通过跨反事实情景和重复场景的大规模分析，系统得以构建实证记忆，未来能基于历史概率为规划人员提供指导（例如提示“以往类似情况下，选择方案 B 有 70% 概率产生最优结果”）。这种机制使容量规划不再随着人员更替而重置认知，实现了工程经验的长期复利积累。 Vallis 的核心论断在于，容量规划必须从依赖静态预测与易流失的人工直觉，转向结合 AI 自动化决策与实证经验闭环的自进化系统；该体系的成效取决于能否长期忠实记录每一次决策动因，并在跨越数年的真实业务周期中客观量化下游结果。

Detailed notes — zh

## Background 本内容由 YouTube 频道 @Scale 于 2026 年 9 月 3 日发布，记录了 Meta 的 Ben Vallis 在圣克拉拉会议中心 @Scale 大会上发表的主题为《The Plan That Learns》（能学习的规划）的闪电演讲。视频简介指出，网络容量规划风险极高且长周期预测注定会出错，演讲核心聚焦于如何利用 AI 智能体协助人类在关键窗口期内做出艰难权衡，从而在需求波动中始终保留回旋余地。 ## TL;DR Meta 网络容量规划团队转变了执拗于“预测精准度”的传统思路，将重心调整为“如何在明知预测会错的情况下覆盖最大范围的错误”。面对硬件短缺和突发并发流量，团队引入 AI 智能体落地了三大支柱：在狭窄采购窗口内将海量流量异动收敛至极少数关键问题、通过治理“单一事实来源”将模型推理准确率从 30% 提升至 80% 以上，以及在数小时内生成带成本与风险权衡的行动菜单。其终极前沿是构建“能学习的规划”，借助“决策轨迹”记录并回溯人类工程师的未编码经验与反事实结果，让组织判断力持续复合沉淀。 ## Atmosphere & style 整场演讲务实、坦诚且极具工程实战底色，Ben Vallis 以举重若轻的幽默感直面“预测每天都会犯错”的职业宿命，瞬间拉近了与技术听众的距离。叙事节奏明快紧凑，从认清现实困境、拆解约束优化，一路层层推进至 AI 智能体决策与前沿的自我学习闭环，兼具冷静的理性反思与对技术突破的克制兴奋。整场分享散发着一线基础设施老兵深思熟虑后的从容气度，让听众在短短十分钟内清晰感受到工程智慧如何与 AI 深度融合。 ## Bulletin - [00:04](https://www.youtube.com/watch?v=3IteDwOdNyk&t=4s) Meta 将网络容量规划的战略重心从追求绝对精准预测，转向利用 AI“覆盖最大范围的错误”，以避免在供应链与需求突发冲击下陷入无退路的绝境。 - [04:00](https://www.youtube.com/watch?v=3IteDwOdNyk&t=240s) 面对每周数百个未知的异常流量模式，AI 智能体通过串联代码变更、新闻热点与多系统指标，将原本数周的人工根因排查收敛至分钟级。 - [05:32](https://www.youtube.com/watch?v=3IteDwOdNyk&t=332s) 容量模型最初在基础可用容量问答上的基线准确率仅约 30%，通过确立单一事实来源（Single Source of Truth）与闭环治理，试点推理准确率已跃升至 80% 以上。 - [07:07](https://www.youtube.com/watch?v=3IteDwOdNyk&t=427s) Meta 探索“能学习的规划”（The Plan That Learns），利用决策轨迹（decision traces）追踪并吸收工程师偏离模型建议背后的未编码判断力，将多维调度权衡从数月压缩至数小时。 ## Key takeaways - [00:04](https://www.youtube.com/watch?v=3IteDwOdNyk&t=4s) 容量规划存在天然的时间不对称风险：真金白银的部署承诺必须提前数年敲定，而任何长期需求预测注定会犯错。 - [00:04](https://www.youtube.com/watch?v=3IteDwOdNyk&t=4s) 预测误差的后果极度不对称：超量建设仅是基础设施提前就位，而建设不足则会导致业务被网络瓶颈卡死，造成不可承受的代价。 - [00:04](https://www.youtube.com/watch?v=3IteDwOdNyk&t=4s) 导致流量海啸的突发地缘政治或全球重大事件无法被精确预判，试图推演所有黑天鹅事件纯属徒劳，规划核心应转向保持战略回旋余地。 - [02:09](https://www.youtube.com/watch?v=3IteDwOdNyk&t=129s) 容量管理不是简单开支票拉满采购，而是在硬件短缺与交付延迟等现实边界下的“约束优化问题”，必须精打细算挑准发力点。 - [04:00](https://www.youtube.com/watch?v=3IteDwOdNyk&t=240s) AI 赋能规划的第一支柱是争取时间窗口，通过高速跨代码库、指标与外部新闻分析，在分钟级内甄别流量异动是偶发噪音还是结构性转变。 - [05:32](https://www.youtube.com/watch?v=3IteDwOdNyk&t=332s) AI 落地倒逼底层枯燥的数据治理，解决了系统各方定义不一致导致模型对基础容量问题给出不同答案的数据信任危机。 - [05:32](https://www.youtube.com/watch?v=3IteDwOdNyk&t=332s) 通过构建自动化评测套件并确立单一事实来源，团队在试点项目中将模型的逻辑推理准确率从 30% 基线迅速拉升至 80% 以上。 - [07:07](https://www.youtube.com/watch?v=3IteDwOdNyk&t=427s) AI 智能体将跨越物理硬件调配、流量整形与放宽 SLO 等复杂多维度的权衡决策周期，从过去的数周或数月压缩至数小时级别。 - [07:07](https://www.youtube.com/watch?v=3IteDwOdNyk&t=427s) 工程师偏离 AI 推荐方案并非算法失效，而是反映出人类脑海中多年沉淀、尚未被系统结构化捕捉的“未编码判断力”。 - [07:07](https://www.youtube.com/watch?v=3IteDwOdNyk&t=427s) 通过引入决策轨迹机制建立长期闭环跟踪，系统能够回溯工程师决策在现实中的最终结果，将个人隐性经验转化为组织智慧。 - [09:02](https://www.youtube.com/watch?v=3IteDwOdNyk&t=542s) 在自学习规划中，精准建模与评估反事实情况（即未选择方案的实际潜在结果）是目前落地过程中最核心的技术挑战。 - [09:34](https://www.youtube.com/watch?v=3IteDwOdNyk&t=574s) AI 的终极价值在于赋予团队卓越的容量定位与敏捷决策能力，在面对不可预测的需求突变时始终守住不被逼入绝境的底线。 ## Detailed walkthrough ### [00:04](https://www.youtube.com/watch?v=3IteDwOdNyk&t=4s) 容量规划的不对称性：为何“注定犯错”不可怕，可怕的是被逼入绝境在 Meta 负责网络容量规划，最痛苦也最无可奈何的现实就是每天清晨醒来，你心里都非常清楚：自己的预测今天又要犯错了。正如 Ben Vallis 所言，这并非能力问题，而是这项工作的“固有职业风险”。容量规划工程师必须提前数年去推演未来的需求走势、预判各色可能发生或不会发生的重大事件；但与此同时，真金白银的硬件投入与容量部署承诺，却必须在今天就白纸黑字地敲定，以迎头赶上面向未来的那个交汇点。长年累月的历练让 Ben 逐渐习惯并接受了这种“注定会错”的日常，而让他心态转变的关键，在于看透了容量预测图表背后极度不对称的现实后果。表面上看，围绕基准预测波动的上下预测区间是一条漂亮的对称曲线，但其实际引发的业务后果却完全不对称： - **向上偏离（乐观预测并超量建设）：** 如果规划团队画了一条偏高的需求上限并据此采购建设，最终实际需求不及预期，在 Meta 这样保持高速增长的业务体量下根本算不上灾难，充其量只是“基础设施建早了一点”，多出来的容量很快就会被后续增长消化吸收。 - **向下偏离（保守预测且建设不足）：** 反之，如果预估过低导致建设不足，实际需求却强势突破上限，整个业务就会瞬间被网络容量卡死，形成致命瓶颈。这种极度不对称的代价结构，使得容量团队对所有“向上超预期的冲击”（upside surprises）抱有近乎严苛的敏感度。而引发这种暴增的核心诱因，归结起来就是一个词——“高并发”（concurrency）。当数千万甚至数亿用户在同一瞬间涌入网络、做着同一件事时，流量就会形成海啸。像跨年夜（New Year's Eve）这类节点并不属于真正的意外，因为团队早有预见，知道届时会有海量用户集中发送祝福消息、分享视频和拨打音视频通话；真正的考验是那些无法预测的突发重大事件——例如英国女王伊丽莎白二世逝世（the Queen's passing），那一瞬间，英国、加拿大、澳大利亚乃至全球数十个国家的用户几乎在同一秒内全部涌向网络，拉出巨大的流量尖峰。面对这种冲击，Ben 直言试图去精确预测此类地缘政治或全球突发事件纯属“白费力气”（fool's errand）。因此，他决定将整个思考框架彻底颠倒过来：团队的工作重心不再执拗于“怎样才能预测得更准”（尽管这依然是一个崇高且值得追求的目标），而是转变为“既然明知预测一定会错，我们该如何覆盖最大范围的错误？”用象棋来打比方，其终极目标就是“永远不要被逼入绝境”，绝不让自己陷入退无可退、丧失所有回旋余地，最终不得不以牺牲用户体验为代价的被动局面。 ### [02:09](https://www.youtube.com/watch?v=3IteDwOdNyk&t=129s) 约束优化破局：以 AI 覆盖最大错误范围与摆脱供应链困境在 Meta 负责抵御突发意外、布局网络容量近十年的 Ben 经常面对一种看似直截了当的建议：既然目标是覆盖最广泛的可能结果，直接把不确定性锥体（cone of uncertainty）的顶格数值拉满，按照最高需求全面采购建设不就万事大吉了吗？这种做法看似成立，在实际工程与现实商业运作中却寸步难行。任何有供应链打交道经验的人都会明白，世界上根本不存在“在想要的时间、地点，百分之百拿到想要的全部资源”这种好事。现实中的容量管理不是单纯地开支票，而是一个复杂的“约束优化问题”（constrained optimization problem）。面对硬件短缺、交付延迟和地域限制，团队必须精打细算、挑准关键发力点（pick your spots）。正是在这种极其考验精细权衡的约束优化场景中，AI 智能体开始展现出巨大的早期应用价值。 Ben 将 AI 赋能网络容量布局的核心突破归结为三个环环相扣的维度： 1. **争取时间窗口（Acting in time）：** 从无休止的流量异动中剥离噪音，让团队能在狭窄的采购与部署周期内及时响应。 2. **直面枯燥的数据底座（Dealing with data）：** 倒逼系统清洗并对齐底层基础定义，构建可被验证的模型信任曲线。 3. **增强人类判断（Augmenting judgment）：** 从纷繁复杂的调度手段中迅速给出带权衡视角的行动清单，并演进为能够自我迭代的“能学习的规划”。 ```mermaid flowchart TD accTitle: AI-Driven Capacity Decision Framework accDescr: Diagram showing the flow from network noise to AI-driven triage, trusted data grounding, and decision trace learning. subgraph Sense["感知阶段：捕捉信号"] direction TB Raw["海量网络流量异动 （每周数百个模式）"] --> Triage{"AI Agent 分流排查 （跨代码库/新闻/指标）"} Triage --> Filtered["极少数关键异动 （分钟级收敛）"] end subgraph Trust["底座阶段：数据信任与评测"] direction TB GroundTruth["确立单一事实来源 （Single Source of Truth）"] --> Eval["评估与丰富上下文 （Measure-Enrich-Regrade）"] Eval --> HighAccuracy["推理准确率提升 （30% 跃升至 80%+）"] end subgraph Decide["行动阶段：决策演进"] direction TB Moves["行动备选方案 （仓库调拨/路由重选/SLO放宽）"] --> Menu["成本与风险排序菜单 （数月压缩至数小时）"] Menu --> Human["工程师人类判断 （未编码经验）"] Human --> Trace["决策轨迹记录 （Decision Traces 回溯学习）"] end Filtered --> GroundTruth HighAccuracy --> Moves ``` ### [04:00](https://www.youtube.com/watch?v=3IteDwOdNyk&t=240s) 支柱一：在时间窗口内行动，从海量噪音中过滤出真实信号在 Meta 庞大的基础设施运作中，每周都会涌现出数百个此前从未见过的全新流量模式。摆在工程师面前的首要难题，是必须迅速界定：这个异动究竟是业务增长或架构调整带来的“结构性转变”（structural），还是转瞬即逝的“偶发噪音”（ephemeral）？是否真需要团队投入资源去应对？造成流量波动的源头往往五花八门： - 可能是某个特定地区的新用户群体迅速崛起，或者是一张爆火的猫咪图片/视频在短时间内疯传； - 可能是庞大代码库中某次微小的代码变更引发了连锁反应； - 可能是公网骨干路由策略的动态重选，甚至可能纯粹是无规律的随机抖动。过去，人工逐一排查图表上散落的每一个数据点极其耗时耗力，往往需要数周甚至数月才能真正锁定根本原因。这种漫长的调查周期极具杀伤力，往往导致团队彻底错失宝贵的行动与采购窗口。这恰恰是一个“天然契合智能体形态的问题”（agentically shaped problem）。AI 展现出了强大的跨领域排查能力：它不仅能够高速穿透与理解 Meta 庞大的内部代码库，还能无缝比对全网新闻突发事件，并串联起分散在组织各处互不相通的底层监控指标。AI 在极短时间内将每周数百个杂乱无章的流量异动点筛减过滤，收敛为少数几个真正关键、需要深入追踪的实质性问题。正是这种秒级到分钟级的过滤能力，使得团队终于能够“在窗口期内及时采取行动”。 ### [05:32](https://www.youtube.com/watch?v=3IteDwOdNyk&t=332s) 支柱二：直面枯燥的数据底座，让模型评测走上信任曲线将 AI 引入网络容量规划的第二重价值，是它成了一台强有力的“倒逼机器”，迫使团队必须正视业务中那些繁琐且毫无光环的数据治理难题。在刚尝试将 AI 应用于容量领域时，团队遭遇了尴尬的现实：哪怕只是向系统手头的不同模型抛出一个基础到不能再基础的问题——“我们当前究竟有多少可用容量？”，根据提示词提问方式的微调、甚至是提问时间的不同，模型竟然能给出三种截然不同的答案。毫无疑问，绝不可能在一个摇摇欲坠的数据底座上搭建起值得信赖的智能体工作流。但这还不是最核心的痛点。更深层的挑战在于：团队能否真正信任模型本身的推理能力？它们的逻辑输出是否稳定一致？ - **基线评测：** 团队搭建了一套标准的自动化评测套件（eval suite），准备了大量已知确定答案的基础设施问题，分发给手头各家模型进行答题。首轮评测结果相当残酷：各模型的平均准确率仅仅徘徊在 30% 左右（上下浮动数百个基点）。 - **优化循环：** 面对这种表现，团队展开了持续的迭代治理：注入丰富的业务上下文、完善数据字典与资产描述，并严格确立“单一事实来源”（Single Source of Truth），坚决杜绝过去多个系统各自为政、各执一词的混乱状态。 - **飞跃上升：** 经过“丰富上下文-重新测量-持续复盘”的闭环，系统在极短时间内迅速沿着“可信度曲线”攀升。在最初的试点项目中，模型问答与推理的准确率迅速飙升至 80% 以上，并且这一数字目前仍在稳步上扬。 ### [07:07](https://www.youtube.com/watch?v=3IteDwOdNyk&t=427s) 支柱三与终极演进：从辅助决策到沉淀人类判断的“能学习的规划” 当团队终于能够及时捕捉信号，并且拥有了可信的数据推理基底后，第三个核心难题迎面而来：既然必须采取行动，到底该选择哪一种手段？面对迫在眉睫的容量压力，工程团队手头往往有多种杠杆可供调度，但每一种都牵一发而动全身： - **物理硬件调配：** 紧急从后备仓库调取存量硬件，或者强行截停改道正在运输途中的硬件物资； - **软件层面调度：** 在应用层实施流量需求整形（demand shaping），或是阶段性放宽某些非核心业务的服务等级目标（SLO）。评估这些措施的收益、潜在风险与经济成本极其复杂。这绝非单纯跑一个仿真数学模型就能搞定，以往往往需要多组工程师跨部门反复协作讨论，耗费数周乃至数月才能达成一致。如今借助 AI 智能体，团队已经成功将这套多维度权衡决策时间压缩到了“数小时级别”，而且给出方案的精准度与可行性极高。然而，在 Ben 看来，能在一张成本与风险明确的菜单上列出排序推荐固然“很酷很不错”，但距离“真正卓越”仍有一步之遥。这引出了本次演讲的核心主旨——“能学习的规划”（The Plan That Learns）。在真实的业务运作中，Ben 观察到一个非常普遍的现象：一线工程师往往不会百分之百照搬 AI 给出的最优推荐动作。许多人可能会将这种偏离视为算法失效，但 Ben 却将其视作“一份天赐的礼物”： - 工程师绕开模型推荐，反映出的并非抵触，而是他们脑海中沉淀多年、尚未被现有模型结构化捕捉的**“未编码判断力”（unencoded judgment）**。 - 在传统的运作模式下，这种极其宝贵的人类隐性经验每次决策后都会被悄然浪费，最多零散记录在几篇离散的文档中，无法复用与积累。为了将这种人类直觉沉淀为系统能力，团队正在全面引入“决策轨迹”（decision traces）机制。对于工程师在推荐菜单上做出的每一个选择，系统都会建立完整的跟踪记录，并在接下来的数月乃至数年间全程回溯追踪：观察当初那个决策在现实中最终如何展开？它到底是对是错？如果最终证明工程师是正确的，系统便能从中逆向汲取经验，让全团队的专业判断力随着时间不断复合沉淀，实现整个规划系统的自我进化。 ### [09:02](https://www.youtube.com/watch?v=3IteDwOdNyk&t=542s) 自我学习的规划机制与反事实评估在探讨如何让规划系统具备持续学习能力时，Ben Vallis 阐明了其核心运作闭环：系统在每次决策后都会持续追踪“我们最终得到了什么结果”。随着类似决策场景不断重复、样本数量不断累积，系统未来再次面对同一类选项清单时，就能够给出具备概率依据的推荐，例如直接提示规划人员：“你知道吗，在我以往遇到类似情况的记录中，有 70% 的概率选择‘B 门’（door B）才是正确的决断。”这种基于历史决策轨迹的复盘机制，让原本散落的个人经验得以转化为系统化的组织智慧。不过，Ben Vallis 坦言目前这项工作仍处于“极早期的开局阶段”（very early innings）。要真正把这套机制落地极其棘手，最大的技术挑战在于如何精准评估复杂的反事实情况（counterfactuals）——即当系统选择了某种策略后，难以直接观察“如果当初选了另一种方案会发生什么”。尽管挑战重重，这一方向展现出的潜力依然令人兴奋，他也热情邀请现场对此感兴趣的同行在活动后的欢乐时光（happy hour）深入交流探讨。 ```mermaid flowchart TD accTitle: 规划系统的自学习与反馈闭环 accDescr: 展示容量决策从生成选项到追踪结果、积累样本，最终通过反事实分析反哺未来决策的完整循环。 subgraph S1["决策制定"] A["评估当前情境与候选清单"] --> B["选择执行策略 （如：选项 B）"] end subgraph S2["结果评估与反哺"] B --> C["追踪实际落地效果与交付结果"] C --> D["反事实评估与样本库沉淀"] D --> E["输出概率化推荐经验 （如：70% 场景下选项 B 最优）"] E -.-> A end ``` - **经验沉淀与概率推荐**：通过反复对比决策输入与实际交付结果，AI 能够在相似场景中给出历史成功率建议（例如指出 70% 的历史情况下选择某选项最优）。 - **核心难点**：反事实推理（Counterfactuals）的建模难度极高，目前整体技术探索仍处于极早期阶段。 ### [09:34](https://www.youtube.com/watch?v=3IteDwOdNyk&t=574s) 拥抱预测的不确定性：用 AI 优化容量定位在演讲的尾声，Ben Vallis 将所有技术讨论拉回到了容量规划的现实本质。他坦承自己每天早晨醒来时都深知“预测注定会出错”，并风趣地表示自己不可能成为一个未卜先知的“地缘政治预测家”（geopolitical forecaster），现实中即便竭尽全力，长期预测也绝不可能做到百分之百准确。因此，容量规划的核心从来不是追求虚幻的绝对精准预测，而是面对不确定性时如何做好容灾与资源定位。他总结强调，AI 带来的真正价值在于赋予团队更强大的应变能力：“借助 AI，我们在容量定位（positioning）上做得越来越好，而且我们的决策速度每天都在变得更快。”这不仅守住了不被错误预测逼入死角的底线，更让网络基础设施在面对任何需求突变时都能保有主动权。 - **规划理念重塑**：承认长周期预测必然存在误差，放弃对完美预测的执念，不再试图扮演地缘政治预言家。 - **AI 的核心落脚点**：重心转移到容量的弹性定位与决策敏捷度上，以更快的流转速度和保留选项的策略化解供应链与需求冲击。 ## Notable quotes > [00:04](https://www.youtube.com/watch?v=3IteDwOdNyk&t=4s) 既然明知预测一定会错，我们该如何覆盖最大范围的错误？ — Ben Vallis > [00:04](https://www.youtube.com/watch?v=3IteDwOdNyk&t=4s) 永远不要被逼入绝境 — Ben Vallis > [02:09](https://www.youtube.com/watch?v=3IteDwOdNyk&t=129s) 在想要的时间、地点，百分之百拿到想要的全部资源 — Ben Vallis > [04:00](https://www.youtube.com/watch?v=3IteDwOdNyk&t=240s) 天然契合智能体形态的问题 — Ben Vallis > [04:00](https://www.youtube.com/watch?v=3IteDwOdNyk&t=240s) 在窗口期内及时采取行动 — Ben Vallis > [05:32](https://www.youtube.com/watch?v=3IteDwOdNyk&t=332s) 我们当前究竟有多少可用容量？ — Ben Vallis > [05:32](https://www.youtube.com/watch?v=3IteDwOdNyk&t=332s) 单一事实来源 — Ben Vallis > [07:07](https://www.youtube.com/watch?v=3IteDwOdNyk&t=427s) 一份天赐的礼物 — Ben Vallis > [07:07](https://www.youtube.com/watch?v=3IteDwOdNyk&t=427s) 未编码判断力 — Ben Vallis > [09:02](https://www.youtube.com/watch?v=3IteDwOdNyk&t=542s) 你知道吗，在我以往遇到类似情况的记录中，有 70% 的概率选择‘B 门’才是正确的决断。 — Ben Vallis > [09:34](https://www.youtube.com/watch?v=3IteDwOdNyk&t=574s) 借助 AI，我们在容量定位上做得越来越好，而且我们的决策速度每天都在变得更快。 — Ben Vallis ## Mentioned Ben Vallis, Meta, Santa Clara Convention Center (SCCC), @Scale, 伊丽莎白二世 (Queen Elizabeth II) ## Video description Ben Vallis from Meta presents his lighting talk on "The Plan That Learns" live at the Santa Clara Convention Center. Network capacity planning is high-stakes and unforgiving: capacity is committed years ahead, every long-range forecast is guaranteed to be wrong, and the costliest surprises are correlated demand shocks that defeat traditional risk-pooling. Much of the pipeline is already automated — but the decisions that take real judgment have stayed manual, and those are the ones that decide whether a wrong forecast corners you. This talk is about applying AI to exactly those decisions — not to chase a more accurate forecast, but to position capacity so it keeps our options open no matter how demand breaks. We show how agents are beginning to crack the three that matter most: acting inside narrow buy windows (triaging signals continuously to surface the handful of truly novel sites, weeks down to minutes), trusting data across systems that disagree on even basic definitions (a measure-enrich-regrade loop), and choosing among coupled, timing-sensitive levers (a ranked, costed menu of moves with the reasoning attached). The real frontier is a plan that learns — agents that remember every decision and how it turned out, so human judgment compounds over time instead of resetting with each planner. Learn more about @Scale here: atscaleconference.com [Source](https://www.youtube.com/watch?v=3IteDwOdNyk)

Source quotes

[0:04] Ben Vallis: 既然明知预测一定会错，我们该如何覆盖最大范围的错误？ [0:04] Ben Vallis: 永远不要被逼入绝境 [2:09] Ben Vallis: 在想要的时间、地点，百分之百拿到想要的全部资源 [4:00] Ben Vallis: 天然契合智能体形态的问题 [4:00] Ben Vallis: 在窗口期内及时采取行动 [5:32] Ben Vallis: 我们当前究竟有多少可用容量？ [5:32] Ben Vallis: 单一事实来源 [7:07] Ben Vallis: 一份天赐的礼物 [7:07] Ben Vallis: 未编码判断力 [9:02] Ben Vallis: 你知道吗，在我以往遇到类似情况的记录中，有 70% 的概率选择‘B 门’才是正确的决断。 [9:34] Ben Vallis: 借助 AI，我们在容量定位上做得越来越好，而且我们的决策速度每天都在变得更快。

Source quotes

[0:00] Ben Vallis: The Plan That Learns capacity is committed years ahead, every long-range forecast is guaranteed to be wrong, and the costliest surprises are correlated demand shocks that defeat traditional risk-pooling. not to chase a more accurate forecast, but to position capacity so it keeps our options open no matter how demand breaks. acting inside narrow buy windows (triaging signals continuously to surface the handful of truly novel sites, weeks down to minutes), trusting data across systems that disagree on even basic definitions (a measure-enrich-regrade loop), and choosing among coupled, timing-sensitive levers (a ranked, costed menu of moves with the reasoning attached). agents that remember every decision and how it turned out, so human judgment compounds over time instead of resetting with each planner. [3:40] Ben Vallis: how much capacity do I have? [8:06] Ben Vallis: 70% of the time when a similar situation occurred, Option B yielded the optimal outcome.

Original transcript

[0:04] Let's talk about the plan that learns. [0:08] So, one of the hardest things about my [0:10] job is that I wake up every morning and [0:12] I know I'm going to be wrong. And some [0:14] of it just like comes with the [0:15] territory. It's an occupational hazard, [0:17] if you would. And think about it for a [0:19] second. Like, I have to think many, many [0:21] years out into the future about our [0:22] demand levels, about various events that [0:25] could or could not happen. But I need to [0:27] make decisions today and make [0:28] commitments today with our capacity to [0:30] intercept that future. [0:33] And over the years, I've gotten a little [0:35] bit more comfortable with being wrong on [0:37] this and knowing I'm going to wake up [0:39] and be wrong on it. And part of it is by [0:41] understanding the inherent asymmetry in [0:43] the graph behind me here. Now, it looks [0:45] very symmetric, but in reality, the [0:47] consequences are highly asymmetric. [0:48] Like, think about it with me for a [0:50] second. Imagine I trace a line on the [0:52] upside, put in a high demand figure, I [0:54] build capacity to that, and it comes in [0:56] a little bit below. Well, in a high [0:59] growth business like ours, like that's [1:00] not really a problem. It just means [1:02] we're a bit early. But the converse [1:04] isn't true, right? If I put in a low [1:06] forecast, don't build enough stuff, [1:07] demand comes in above it, that's a real [1:09] problem. We can bottleneck the business. [1:12] And so in practice, what this means is [1:13] it makes us very sensitive to upside [1:15] surprises. [1:18] So what are some of those surprises? [1:19] what what gets us to the upside. And in [1:22] a word, it's just concurrency. It's [1:24] hundreds of tens of millions of users or [1:27] hundreds of millions of users all [1:28] getting online at the same time doing [1:31] the same thing. Now, you might say, [1:33] well, hey, you know, about New Year's E [1:35] that's not really a surprise. And you're [1:36] right. It's not not a surprise. I know [1:38] it's going to happen there more or less. [1:39] I know lots of people are going to send [1:40] messages and share videos and call each [1:43] other. What I'm talking about are real [1:45] surprises. A good example of this might [1:47] be the Queen's passing where like all of [1:49] the UK and Canada and Australia and a [1:51] bunch of other countries all get online [1:53] at the same time and do the same thing. [1:57] And over the years I've realized it's [1:59] actually kind of a fool's errand to try [2:00] and even predict geopolitical events [2:02] like that. It's just not a thing. Um and [2:05] so what I've learned to do is to flip [2:06] the question on its head. [2:09] Instead of saying, "Hey, how do I get [2:12] better at predicting?" Even though I [2:13] think that's a very noble goal and we do [2:15] pursue it, I say instead given that I [2:18] know I'm going to be wrong, how do I [2:19] cover as much wrongness as I possibly [2:21] can? So differently, it's like how can I [2:24] position our capacity in a way that I [2:26] can be robust to the largest range of [2:28] outcomes that could come our way? And [2:31] the goal with all of this, of course, is [2:32] I just like never want to get cornered. [2:33] It's like a a chess game. I never want [2:35] to get backed into a spot where I [2:38] foreclose on options or I'm forced to do [2:40] something where I have to reduce quality [2:42] for our users. Just quickly, by way of [2:44] background, I'm Ben Val. I've been [2:46] positioning our network against [2:47] surprises for the better part of 10 [2:49] years. [2:50] At this point, you might say, you know [2:52] what, Ben, I've got a great solution for [2:54] you. Right? If you are trying to [2:56] position yourself against the widest [2:58] range of outcomes, I've got a great [3:00] solution. go back a couple slides and [3:02] look at the top end of that cone of [3:04] uncertainty you showed me. Just do the [3:05] thing, right? Put in a high forecast, [3:08] buy a bunch of stuff. And I'm kind of [3:10] with you, that's actually a valid [3:12] strategy, except for the fact it doesn't [3:14] really work in practice. Any of you who [3:16] in this room who have ever dealt with a [3:18] supply chain issue, for example, will [3:20] know that you never get exactly what you [3:22] want, where you want, or when you want [3:24] it. And so really, this boils down to a [3:26] constrained optimization program. [3:29] it it forces you to kind of pick your [3:31] spots. And it's in that constrained [3:34] optimization problem, that picking of [3:36] spots, where I'm really starting to see [3:38] the early benefits of AI. [3:40] And so that really forms the balance of [3:42] my talk here. I'm going to talk about [3:43] the three ways that AI is helping us to [3:45] position our capacity better. First way [3:47] is around acting in time. Second, it's [3:50] forcing us to deal with some of the [3:51] unglamorous parts of our workflows [3:53] related to data. And third, it's [3:55] starting to augment our judgment. So let [3:57] me take these in turn. [4:00] First, [4:02] AI is helping us to position our [4:04] capacity in a way such that we can act [4:06] in time. Well, what do I mean by that? [4:08] Well, think about it from our [4:10] perspective. Every single week we're [4:12] seeing hundreds of novel traffic [4:13] patterns emerge. And these can be for [4:16] any number of reasons, but it really [4:18] boils down to or essentially is I need [4:20] to figure out is this structural or is [4:22] it ephemeral? Do I actually need to do [4:23] something about this or is it just [4:25] noise? And the reasons can be varied. It [4:27] can be as simple as, hey, there's some [4:29] new user cohort showing up and it's a [4:32] really popular cat photo or video. Um, [4:34] that's a fun one, right? It could be a [4:37] code change, right? We have to traverse [4:39] a very large codebase. It could be a [4:41] routing change on the internet. It could [4:43] just be random. But the point is by hand [4:45] each one of these takes uh many many [4:48] weeks and sometimes even longer to [4:50] really investigate and understand. And [4:52] you can kind of see this on the graph, [4:53] all the dots out there. All these things [4:55] take forever to like really nail down [4:57] what is the root cause and this can [4:59] cause us to miss the important windows [5:01] we have to act within. With AI, it's a [5:04] little bit different. This is a [5:06] perfectly agentically shaped problem. AI [5:08] is great at traversing our co codebase. [5:10] It's great at understanding news events [5:12] or looking at disperate metrics all [5:14] across our organization. And in [5:16] practice, what it does is it whittleles [5:17] down all that noise, all those dots, [5:20] right? those hundreds of weekly traffic [5:21] patterns into a manageable number that [5:24] we can go track down and figure out what [5:26] we need to do. And by virtue of doing [5:28] this, it enables us to act in time. [5:32] The second way it's helping us position [5:34] our capacity or second way AI is helping [5:36] us to position our capacity, it's a [5:38] forcing function for us to deal with [5:39] some of the unlamorous parts of our [5:41] business uh or part especially around [5:43] data. So when I first got started [5:45] applying AI to uh the capacity planning [5:48] space, when we routed questions as [5:51] simple as how much capacity do I have to [5:53] various models that we have at our [5:55] disposal, you might have gotten three [5:57] different answers depending on how you [5:59] phrase the question or what time of day [6:02] even that you asked. And of course, this [6:04] is like an obvious problem. You can't [6:06] have trustable agentic workflows sitting [6:08] on top of shaky data foundations. So [6:10] it's been very motivational for us to go [6:12] fix this up. [6:14] But the real problem in my mind actually [6:16] transcends this one and is the [6:18] following. Can I trust the reasoning [6:20] capability of the models themselves? Are [6:22] they consistent? So we set out to answer [6:24] that and this is a fairly standard eval [6:26] suite. We went and set up a bunch of [6:28] questions for which we knew the answers [6:30] to. We forked it out to the various [6:32] models at our disposal. Said how well [6:35] can you answer it? Unfortunately, we [6:36] found out not so great. At least at the [6:38] beginning it was about 30% accuracy plus [6:40] or minus a few hundred basis points [6:42] depending on the model. And so what we [6:44] did then is we enriched, we added [6:46] context, we added descriptions, we [6:48] marked a single source of truth, not [6:50] multiple single sources of truth. We [6:52] rememeasured, rinsed and repeated. And [6:55] what we were able to do very quickly was [6:57] ascend this trustability curve. In our [6:59] early pilot, we were able to get this [7:00] north of 80% in short order. And even [7:03] now it's going higher. [7:07] Third way AI is helping us to position [7:09] our capacity better. It's helping us to [7:10] pick the right move. So a couple slides [7:12] ago I talked about acting in time. But I [7:15] didn't really say well what action do we [7:16] take? Well you can imagine when we [7:18] decide we need to do something. There's [7:20] any number of things we could go after. [7:23] You could pull supply out of a [7:24] warehouse. You could reroute existing [7:26] supply that's on its way. You can do [7:29] more software based things around [7:30] shaping demand or relaxing service level [7:32] objectives. But the point is with all of [7:35] these it takes time to understand the [7:37] benefits, risks and costs. This isn't [7:39] even strictly a modeling problem though [7:41] that is a line share of it. Right? It it [7:44] takes a lot of effort by hand across our [7:47] a variety of engineers to figure out [7:48] which one of these is the right choice [7:50] to make and it can take weeks if not [7:52] months. With AI, we've been gotten able [7:55] or brought this down to like hours. It's [7:57] like it's a pretty big win and it's [7:59] pretty accurate, too. And while this is [8:02] really cool and I think it's pretty [8:03] good, in my mind, it's not yet great. [8:06] But here's how we're going to make it [8:08] great. And that's the plan that learns. [8:09] And this is the part I'm really excited [8:11] about. It's the title of my talk. What [8:13] do I mean by that? Well, in practice, [8:15] what what do we see? We see engineers [8:17] don't always follow the model [8:18] recommendations. We don't always follow [8:19] precisely what AI says to pick as the [8:22] next right move. And I view this as a [8:25] gift. It's not a problem. What I what I [8:27] view it as is unencoded judgment. It's [8:29] the judgment that our engineers have [8:31] accumulated over many many years which [8:34] they are applying to the problem in [8:35] front of them. And today we're actually [8:37] kind of throwing it away in a sense [8:39] every time we make a decision, right? [8:42] It's not being recorded necessarily [8:44] maybe in a doc here and there, but it's [8:45] there's no structured way of recording [8:47] this. But what if you could? And that's [8:49] the concept that we're introducing here. [8:51] We're implementing decision traces. [8:53] We're looking at, hey, for every [8:54] decision that you make on that menu of [8:56] options, I want to trace over the [8:58] following months and years, how it [9:00] panned out. Was it the right decision? [9:02] What did we get? And with enough repeats [9:05] of this, with enough samples, we can get [9:07] to a spot where that same menu of [9:09] options in the future is going to show [9:11] up and say, you know what, 70% of the [9:13] time when I saw a similar situation, [9:15] door B was the right one to choose. So, [9:19] this is very early, very early innings [9:21] on this. It's a really tricky problem to [9:23] do it right. Uh especially if you think [9:24] about all the counterfactuals that are [9:26] involved. Uh but if you are interested [9:28] in learning more or talking with me [9:29] about it, uh find me at happy hour. I'm [9:31] happy to chat more. [9:34] So let me bring this all back home. I [9:36] still wake up in the morning. I I know [9:38] I'm going to be wrong. I'm not going to [9:39] be a geopolitical forecaster, right? I'm [9:42] not going to be right on the forecasting [9:44] even if I try. [9:46] But with AI, we're getting better at [9:49] positioning and we're getting faster [9:51] every day. Thank you.