Meta Builds 100,000-Strand Fiber Grid for New Albany AI Center
@Scale
Meta scaled its New Albany AI cluster connectivity a hundred-fold by swapping traditional telecom conduits for deep civil utility trenches and custom high-density fiber cables.
## Background
Published on YouTube by @Scale on August 27, 2026, this presentation features Fabrice Ouandji and Sebastian Gault from Meta. The talk addresses the immense connectivity complexity introduced by Project Prometheus, detailing how Meta tackled high fiber count requirements, dynamic location demands, and tight delivery schedules across its data center cluster.
## TL;DR
To support the massive connectivity requirements of Project Prometheus in New Albany, Ohio, Meta scaled its fiber capacity from hundreds of strands to over 100,000 strands per path. The team achieved this by pioneering high-density cables (including a 1,728-strand cable with AFL), shifting from narrow telecom trenches to massive civil engineering excavations with Ohio Valley Excavating, and innovating field jetting techniques by cascading machines inside underground vaults. The project highlights how scaling AI infrastructure requires cross-industry collaboration, flexible execution, and modernizing municipal permitting workflows.
## Atmosphere & style
The presentation carries an energetic, pragmatic, and collaborative tone, blending high-stakes engineering problem-solving with grounded field realities. Speakers Fabrice Ouandji and Sebastian Gault deliver a dynamic tag-team talk, framing daunting physical hurdles—like massive trenches and unexpected cable friction—with transparency and dynamic optimism. Watching the presentation feels like an insider look into pioneering heavy infrastructure, where real-world trial and error meets hyper-scale ambition.
## Key takeaways
- [00:05](https://www.youtube.com/watch?v=XJk7SJzKHuo&t=5s) Project Prometheus required Meta to scale from hundreds of fiber strands per path up to over 100,000 strands to connect data centers in New Albany, Ohio.
- [02:46](https://www.youtube.com/watch?v=XJk7SJzKHuo&t=166s) Meta held a Kaizen jetting trial in Asheville, North Carolina, evaluating cable manufacturers on a 3,700-foot test track to choose the best high-density optical fiber.
- [02:46](https://www.youtube.com/watch?v=XJk7SJzKHuo&t=166s) The team selected a standard 15 mm 864-fiber cable from AFL over micro-cables for Phase 1 to ensure structural rigidity and prevent kinking during complex field handling.
- [02:46](https://www.youtube.com/watch?v=XJk7SJzKHuo&t=166s) For Phase 2, Meta co-developed a 1,728-fiber cable that doubled strand capacity within a 16.4 mm diameter, using only 1.4 times more space than the 864-fiber cable.
- [02:46](https://www.youtube.com/watch?v=XJk7SJzKHuo&t=166s) Meta is installing 2.1 million fiber-kilometers within a 5 km radius in New Albany, which is equivalent to three round trips to the moon.
- [05:39](https://www.youtube.com/watch?v=XJk7SJzKHuo&t=339s) The scale of the build required excavating trenches up to 9 feet wide and 20 feet deep, forcing Meta to partner with heavy civil engineering firms like Ohio Valley Excavating.
- [05:39](https://www.youtube.com/watch?v=XJk7SJzKHuo&t=339s) Simultaneous megaproject construction by tech competitors and Intel created heavy local traffic, forcing tight coordination with municipal authorities for lane closures.
- [05:39](https://www.youtube.com/watch?v=XJk7SJzKHuo&t=339s) Meta used 5-foot-diameter under-highway micro-tunneling and deployed massive 10 ft × 12 ft precast concrete vaults to manage the massive conduit banks.
- [12:18](https://www.youtube.com/watch?v=XJk7SJzKHuo&t=738s) To overcome high cable friction, Meta placed jetting machines directly inside the 10x12 underground vaults and expanded to a series of seven cascaded units.
- [12:18](https://www.youtube.com/watch?v=XJk7SJzKHuo&t=738s) Because Phase 2's 1,728-fiber cable was twice as heavy, the team had to reduce jetting machine push distances from 2,500 feet to 1,800 feet.
- [12:18](https://www.youtube.com/watch?v=XJk7SJzKHuo&t=738s) A continuous subducted vault design eliminated massive slack coils inside manholes, allowing operations to reduce surface road closures to small 100–200 foot work zones.
- [16:12](https://www.youtube.com/watch?v=XJk7SJzKHuo&t=972s) Key takeaways emphasize the need to adapt when field reality disproves theoretical models, maintain strong human connections across municipal and industry lines, and adopt non-traditional engineering practices.
- [18:33](https://www.youtube.com/watch?v=XJk7SJzKHuo&t=1113s) Meta modernized its permitting and build workflows by shifting away from traditional telecom carriers and working directly with city officials and heavy civil partners.
Fabrice Ouandji, a Partner Manager within Meta's Network Investment team, opens the talk with a provocative question: "What if you were asked to do something that has never been done before?" He frames the challenge around Project Prometheus—Meta's massive AI training cluster located in New Albany, Ohio—which required a completely reimagined fiber architecture. Pre-2020, Meta’s network team sourced around 10 to 12 fiber strands per path from traditional carriers like Zayo or Lumen to connect data centers and Points of Presence (POPs). Between 2020 and 2024, as bandwidth demands grew, Meta began opening up right-of-ways themselves, laying conduits with a primary cable housing 432 to 864 strands and reserving a second conduit for growth.
Prometheus shattered this incremental model entirely. Instead of hundreds of strands, Ouandji's team was tasked with delivering over 100,000 strands per path between data centers—a nearly hundred-fold increase over pre-Prometheus baselines. To tackle this massive scale, the session outlines four core areas: fiber cable innovations, heavy civil engineering, complex fiber placement and jetting operations, and the key lessons learned for future AI infrastructure builds.
- **Pre-2020 Baseline:** 10–12 fiber strands per path, sourced from carriers like Zayo and Lumen.
- **2020–2024 Baseline:** Self-built fiber routes with 1–3 conduits, carrying 432–864 strands per path.
- **Prometheus Scale:** Over 100,000 fiber strands per path connecting data center buildings.
[02:46](https://www.youtube.com/watch?v=XJk7SJzKHuo&t=166s) Fiber Innovations and the Asheville Kaizen Trial
Sebastian "Seb" Gault, Network Implementation Engineer at Meta and lead PM for Project Prometheus, steps in to explain the fiber developments required to support this unprecedented strand density. To select the optimal 864-strand cable for the initial phase, Meta hosted a "Kaizen"—a continuous improvement workshop—at a conduit manufacturer's facility in Asheville, North Carolina. Meta invited three of its top cable manufacturing partners to participate in a head-to-head jetting trial along a 3,700-foot test track.
Meta ultimately selected an 864-fiber cable from AFL measuring 15 mm in diameter. Though micro-cables were considered, the team intentionally chose a standard, more rigid cable construction. Because Phase 1 required jetting 140 individual 864-fiber cables per path, thin micro-cables posed a significant risk of kinking or structural damage if stepped on during the intensive field installation. Looking toward Phase 2 where conduit space became increasingly premium, Meta collaborated with AFL to develop a high-density 1,728-fiber cable. This new design packed double the fibers into a package measuring just 16.4 mm in diameter—only 1.4 times larger than the 864 cable—allowing Meta to double its fiber density within the exact same conduit footprint.
- **Asheville Kaizen:** Jetting competition on a 3,700 ft test track among 3 top cable suppliers.
- **Phase 1 Cable:** AFL standard 864-strand cable (15 mm diameter); chosen over micro-cable for rigidity and kink resistance across 140 cable runs per path.
- **Phase 2 Cable:** High-density 1,728-strand cable (16.4 mm diameter), doubling fiber count with only a 1.4x increase in cable size.
- **Local Fiber Density:** Meta is installing 2.1 million fiber-kilometers within a single 5 km radius in New Albany, Ohio—equivalent to "three times to the moon and back."
[05:39](https://www.youtube.com/watch?v=XJk7SJzKHuo&t=339s) Heavy Civil Engineering and Mass Trenching
Ouandji returns to detail the massive civil engineering shift needed to accommodate the conduit banks for over 100,000 strands. Traditional telecom deployments rely on narrow micro-trenches roughly 6 inches wide and 3 to 4 feet deep. For Prometheus, placing 140 separate cables required a trench up to 9 feet wide and 20 feet deep in certain sections—large enough to "accommodate three SUVs stacked on top of each other." This scale forced Meta to look beyond traditional telecom contractors and partner with heavy civil engineering firms experienced in deep excavations for storm sewers, water mains, and power utilities, ultimately contracting Ohio Valley Excavating (OVE), a division of the Congruex Group.
The civil build presented severe execution challenges, primarily driven by dynamic AI capacity requirements versus the "one-shot" nature of heavy road construction. Permitting authorities would not allow repeated road closures on the same thoroughfares, forcing Meta to fully engineer entire right-of-ways within a 5-to-10 kilometer radius upfront. Compounding the complexity was extreme local traffic congestion; New Albany was simultaneously experiencing construction from hyperscalers like Meta, Microsoft, Google, and Amazon, third-party developers like Vantage and QTS, and Intel's multi-billion-dollar semiconductor foundry. With 5,000 to 10,000 construction workers traveling these corridors daily, Meta had to closely coordinate lane closures with city officials and regional stakeholders.
- **Trench Dimensions:** Standard telecom (6 in wide × 3–4 ft deep) vs. Prometheus (9 ft wide × up to 20 ft deep).
- **Civil Partner:** Ohio Valley Excavating (OVE) / Congruex Group.
- **Regional Traffic Pressure:** 5,000 to 10,000 daily trade workers navigating shared roads during simultaneous megaproject builds.
- **Under-Highway Micro-tunneling:** 5-foot-diameter bore pipe allowing operators to sit inside specialized machinery within the tube to manually clear soil.
- **Vault Sizing:** Replaced standard 3 ft × 4 ft handholes with massive 10 ft × 12 ft precast concrete vaults—described as "almost as big as a studio apartment."
[12:18](https://www.youtube.com/watch?v=XJk7SJzKHuo&t=738s) High-Density Cable Jetting and Vault Placement
Gault takes over to detail the field execution of jetting 140 cables during Phase 1 and 76 cables during Phase 2. To optimize pushing distance and equipment placement, Meta brought together AFL, jetting manufacturer Fremco (FLT), and conduit manufacturer Dura-Line (M-Tel) for collaborative whiteboard sessions and field trials. While initial theoretical models suggested spacing jetting machines 3,800 feet apart, field tests immediately failed because the heavy 864 cable created too much friction for machines to pick up pushing force at that distance.
To solve this, the engineering team dropped jetting machines directly into the 10x12 foot underground vaults and cascaded seven machines in series rather than the planned three to five. Placing machines inside the vaults eliminated the need for massive slack coils inside manholes—avoiding "140 slack coils in the manhole" which Gault noted would be "messy" and difficult to maintain during an outage—and reduced surface road footprints to temporary closures of just 100 to 200 feet. Phase 2 presented new physics challenges: although the 1,728-fiber cable fit into the conduit, its double-weight construction forced the team to reduce jetting machine spacing from 2,500 feet down to 1,800 feet, requiring additional inline pushing stations to meet strict project deadlines and light up GPUs on time.
- **Jetting Machine Cascading:** Expanded from 3–5 planned machines to 7 cascaded units operating directly inside underground vaults.
- **Phase 1 Execution:** Jetted 140 individual 864-strand cables between core data center facilities.
- **Phase 2 Distance Adjustments:** Reduced push distances from 2,500 ft to 1,800 ft to handle the doubled weight of 1,728-strand cables.
- **Subducted Vault Design:** Eliminating slack loops enabled continuous subducting through vaults, minimizing local traffic disruption to small 100–200 ft work zones.
[16:12](https://www.youtube.com/watch?v=XJk7SJzKHuo&t=972s) Key Takeaways and Lessons Learned
Ouandji closes the presentation by synthesizing three core lessons from executing a first-of-its-kind optical build:
1. **"Measure twice, but do not be afraid to cut."** While extensive upfront modeling, Kaizens, and engineering route designs are mandatory for complex builds, teams must remain flexible. When field conditions disproved theoretical models—such as weight limitations during cable jetting—the team pivoted immediately rather than delaying execution.
2. **"AI is about human connections."** Building large-scale physical infrastructure relies heavily on clear communication across internal data center teams, external hyperscaler competitors, municipal permitting authorities, and heavy civil contractors to manage road closures and utility conflicts.
3. **Embrace new ways of working.** Delivering a hundred-fold expansion in fiber density requires breaking out of traditional telecom silos, adopting cross-industry civil engineering methodologies, and establishing unconventional vendor partnerships.
<ElicitationsGroup message="Explore related infrastructure topics:">
<Elicitation label="Summarize Part 2 of this presentation" query="Summarize Part 2 of the Meta Fiber Networks Innovations for AI presentation."/>
<Elicitation label="Compare 864 vs 1728 fiber density specs" query="Compare the technical specifications and conduit efficiency of 864 vs 1728 strand fiber cables."/>
<ElicitationsGroup/>
[18:33](https://www.youtube.com/watch?v=XJk7SJzKHuo&t=1113s) Modernizing Permitting and Building Engineering Partnerships
Building out the massive infrastructure required for AI at scale forced Meta to completely re-evaluate their traditional operational models. Delivering a duck bank of this magnitude meant moving past conventional telecom infrastructure carriers and forging direct partnerships with specialized engineering firms. Working closely with local municipal leaders and city officials on the wall proved vital for streamlining complex permitting processes, allowing both sides to collaborate dynamically rather than sticking to slow, legacy workflows. Embracing this new operational paradigm was an absolute necessity for keeping the project moving on compressed timelines and successfully deploying the massive volume of fiber strands needed across the cluster.
Looking forward, this project sets the blueprint for how Meta intends to tackle future infrastructure scale. While the next challenge might not specifically involve digging a giant duct bank, the team fully expects to encounter equally complex, unprecedented engineering roadblocks as AI workloads continue to expand exponentially.
* **Engineering Shift:** Transitioned from traditional telecom carriers to specialized engineering firms better suited for ultra-high-density duct bank builds.
* **Municipal Collaboration:** Partnered directly with city permitting authorities to enable rapid, joint problem-solving during construction.
* **Core Philosophy:** Positioned the initiative as a template for Meta’s ongoing commitment to taking on complex, large-scale infrastructure challenges.
## Notable quotes
> [00:05](https://www.youtube.com/watch?v=XJk7SJzKHuo&t=5s) What if you were asked to do something that has never been done before? — Fabrice Ouandji
> [02:46](https://www.youtube.com/watch?v=XJk7SJzKHuo&t=166s) three times to the moon and back — Sebastian Gault
> [05:39](https://www.youtube.com/watch?v=XJk7SJzKHuo&t=339s) accommodate three SUVs stacked on top of each other — Fabrice Ouandji
> [12:18](https://www.youtube.com/watch?v=XJk7SJzKHuo&t=738s) 140 slack coils in the manhole — Sebastian Gault
> [12:18](https://www.youtube.com/watch?v=XJk7SJzKHuo&t=738s) messy — Sebastian Gault
> [16:12](https://www.youtube.com/watch?v=XJk7SJzKHuo&t=972s) Measure twice, but do not be afraid to cut. — Fabrice Ouandji
> [16:12](https://www.youtube.com/watch?v=XJk7SJzKHuo&t=972s) AI is about human connections. — Fabrice Ouandji
To support Project Prometheus—Meta's AI training cluster in New Albany, Ohio—the company departed from standard telecom models to construct optical routes carrying over 100,000 fiber strands per path. Fabrice Ouandji and Sebastian Gault describe how delivering this hundred-fold capacity increase required custom high-density cabling, heavy civil excavation, and re-engineered underground jetting operations.
Scaling Demands and AI-Driven Fiber Density (0:05)
Before 2020, Meta sourced optical connectivity by leasing roughly 10 to 12 fiber strands per path from traditional carriers such as Zayo and Lumen to interconnect data centers and Points of Presence. As bandwidth requirements rose between 2020 and 2024, the company began self-building routes using one to three conduits housing 432 to 864 strands per path. Project Prometheus broke this incremental growth model by demanding over 100,000 fiber strands per path between data center buildings.
According to Ouandji and Gault, supporting the AI cluster required installing 2.1 million fiber-kilometers within a single 5-kilometer radius in New Albany, an aggregate length they likened to three round trips to the moon. Meeting this volume forced Meta's network investment and implementation teams to discard standard telecom playbooks in favor of custom physical infrastructure.
Cable Innovation from Kaizen to Custom Densification (2:46)
To identify cabling capable of handling these high strand counts, Meta conducted a Kaizen trial on a 3,700-foot test track in Asheville, North Carolina, pitting three cable manufacturers against one another. For Phase 1, Meta selected AFL's standard 864-fiber cable measuring 15 millimeters in diameter. Although micro-cables were evaluated, the engineering team opted for a more rigid standard jacket because Phase 1 required pulling 140 separate cables per path, where thinner micro-cables risked kinking or structural damage if trodden on during site operations.
Anticipating conduit bottlenecks in Phase 2, Meta worked with AFL to engineer a 1,728-fiber high-density cable. The resulting design packed twice the fiber count into a 16.4-millimeter-diameter package—only 1.4 times larger than the 864-strand cable—allowing the team to double pathway capacity without expanding the physical conduit bank footprint.
Heavy Civil Engineering and Multi-Project Coordination (5:39)
Accommodating 140 cable conduits required moving away from standard telecom micro-trenches (typically 6 inches wide and 3 to 4 feet deep) to heavy utility cuts up to 9 feet wide and 20 feet deep. To execute trenches large enough to hold three stacked SUVs, Meta bypassed traditional telecom contractors and partnered with heavy civil firm Ohio Valley Excavating (OVE), a division of Congruex Group. The build also required installing 5-foot-diameter micro-tunneling bore pipes under highways—where operators manually cleared soil from inside the pipe machinery—and replacing standard 3-by-4-foot handholes with precast concrete vaults measuring 10 by 12 feet.
Execution was constrained by strict local permitting rules that barred repeated road closures on the same corridors, forcing Meta to design entire rights-of-way within a 5-to-10-kilometer radius in a single pass. Civil operations faced extreme road congestion, as New Albany simultaneously hosted 5,000 to 10,000 daily construction workers supporting parallel builds by Intel, Microsoft, Google, Amazon, Vantage, and QTS.
Underground Jetting Mechanics and Vault Adaptation (12:18)
Placing 140 cables during Phase 1 and 76 cables during Phase 2 required collaborative planning among Meta, AFL, Fremco (FLT), and Dura-Line (M-Tel). Theoretical modeling initially recommended spacing cable-jetting machines 3,800 feet apart, but field tests quickly failed because the weight of the 864-strand cable generated excessive friction in the conduit.
To overcome this, engineers relocated the jetting equipment directly inside the 10-by-12-foot underground vaults and cascaded seven machines in series, up from the planned three to five. Operating inline inside the vaults eliminated the need to store 140 cable slack loops inside each manhole and limited active surface road closures to small footprints of 100 to 200 feet. In Phase 2, the added weight of the 1,728-strand cable introduced further friction limits, requiring Meta to reduce jetting machine spacing from 2,500 feet down to 1,800 feet to maintain pushing speeds and meet cluster power-on deadlines.
Ouandji and Gault argue that scaling physical AI infrastructure requires telecom teams to discard functional silos, pivot quickly when field physics challenge theoretical models, and coordinate closely with civil contractors and local permitting authorities. Meta views the cross-disciplinary engineering partnerships established during Project Prometheus as the operational framework for managing future large-scale computing expansions.
## Background
该视频由 YouTube 频道 @Scale 于 2026 年 9 月 3 日发布。视频简介指出，Meta 的 Prometheus AI 训练集群在连接层面引入了巨大复杂性，团队面临前所未有的超高光纤芯数需求、高度动态的建设选址以及极为紧迫的交付工期，本次演讲深入揭秘了为满足多座数据中心互联需求而开展的一系列工程创新。
## TL;DR
Meta 网络投资团队的 Fabrice Ouandji 与网络部署工程师 Sebastian Gault 详细复盘了俄亥俄州新奥尔巴尼 Prometheus 超大规模 AI 集群背后的光纤互联工程。随着单路径光纤需求从早期的十多芯暴增至十万芯以上，团队跳出传统电信工程模式，定制了 AFL 864 芯和 1,728 芯超高密度光缆，并引入市政级重型挖掘承包商开挖足以纵向容纳三辆 SUV 的超级地下管廊与单身公寓般巨大的检查井。面对同区域内微软、谷歌、亚马逊及英特尔工地扎堆引发的严重交通冲突与动态选址难题，团队在高速公路下实施大口径钻孔，并通过现场试错打破白板理论，将气吹机间距从 3,800 英尺大幅压缩并采用井内级联直吹，最终成功交付了连接数座数据中心的超级光纤骨干网。
## Atmosphere & style
整场演讲充满硬核工程叙事的务实与张力，两位主讲人以清晰自信、富有感染力的技术语言复盘极限工程挑战。Fabrice 侧重宏观战略与土木奇迹，用诸如“叠放三辆 SUV 的深坑”、“大如单身公寓的人孔井”等形象比喻展现颠覆性规模；Sebastian 则以一线工程师的细腻与敏锐拆解材料选型与气吹动力学碰撞。从白板计算被现场重重打脸的困窘，到跨界重构施工范式并如期点亮的释然，通篇洋溢着硅谷巨头深入泥泞工地、以敢啃硬骨头的敏捷实干精神打破传统物理极限的自豪感与共鸣。
## Bulletin
- [00:05](https://www.youtube.com/watch?v=XJk7SJzKHuo&t=5s) Meta 为其位于俄亥俄州新奥尔巴尼的 Prometheus AI 训练集群重构了光纤骨干网，数据中心间互联光纤需求激增至单路径超 100,000 芯，规模达到传统自建网络的百倍。
- [02:46](https://www.youtube.com/watch?v=XJk7SJzKHuo&t=166s) Meta 与 AFL 联合定制了外径仅 16.4 毫米的 1,728 芯超高密度光缆，在半径 5 公里的狭小区域内敷设了高达 210 万光纤公里，总长相当于往返月球三次。
- [05:39](https://www.youtube.com/watch?v=XJk7SJzKHuo&t=339s) 为了承载上百根光缆，Meta 跳出传统弱电外包圈子，引入重型市政土木承包商开挖宽 9 英尺、深达 20 英尺的超级管廊，并建造了 10×12 英尺如同单身公寓大小的地下光缆井。
- [12:18](https://www.youtube.com/watch?v=XJk7SJzKHuo&t=738s) 面对 1,728 芯光缆翻倍的自重挑战，工程团队推翻纯理论设计，将级联气吹机间距从预估的 3,800 英尺大幅缩短至 1,800 英尺，并直接在井下级联直吹以取消盘留。
## Key takeaways
- [00:05](https://www.youtube.com/watch?v=XJk7SJzKHuo&t=5s) 通信架构发生历史性范式转移：Meta 从早期的运营商租芯（约 12 芯），到 2020 年后自建 432-864 芯，最终在 Prometheus AI 集群跨越式飙升至单路径超 100,000 芯。
- [02:46](https://www.youtube.com/watch?v=XJk7SJzKHuo&t=166s) 通过 3,700 英尺的 Kaizen 改善实测，团队放弃易折损的微缆，选用外径 15 毫米的 864 芯标准光缆以保证施工抗弯折刚性。
- [02:46](https://www.youtube.com/watch?v=XJk7SJzKHuo&t=166s) 为突破管道截面瓶颈，Meta 联合 AFL 研发外径仅 16.4 毫米的 1,728 芯定制光缆，截面仅增加 40% 即可使纤芯密度翻倍。
- [05:39](https://www.youtube.com/watch?v=XJk7SJzKHuo&t=339s) 传统电信沟槽（宽 6 英寸、深 3-4 英尺）无法承载 AI 规模，Prometheus 升级为宽 9 英尺、深近 20 英尺的超级管廊，体量足可垂直叠放三辆 SUV。
- [05:39](https://www.youtube.com/watch?v=XJk7SJzKHuo&t=339s) Meta 跨界联合市政级重工业承包商 Congruex/OVE，将下水道与高压电缆等重型土建经验引入电信管廊施工。
- [07:44](https://www.youtube.com/watch?v=XJk7SJzKHuo&t=464s) 面对未来数据中心选址动态变化的难题，工程团队实施 5 至 10 公里全域地下管线摸底，推行“只挖一次”的全面管网规划以避免重复破路。
- [07:44](https://www.youtube.com/watch?v=XJk7SJzKHuo&t=464s) 应对该片区多达 5,000 至 10,000 名建筑工人的交通风暴，Meta 与谷歌、微软、亚马逊及英特尔代工厂等多家科技巨头建立了高密度交通维持协同机制。
- [07:44](https://www.youtube.com/watch?v=XJk7SJzKHuo&t=464s) 针对繁忙高速公路节点，实施外径达 5 英尺的超大口径地下定向钻孔，工人甚至可直接驾驶机械进入管道内部掏挖土石。
- [11:21](https://www.youtube.com/watch?v=XJk7SJzKHuo&t=681s) 地下人孔井尺寸由传统的 3×4 英尺剧增至 10×12 英尺，如同单身公寓大小的地下空间为未来检修与扩展打下了物理底座。
- [12:18](https://www.youtube.com/watch?v=XJk7SJzKHuo&t=738s) 气吹光缆实测打破理论白板模型：光缆自重导致最初设定的 3,800 英尺推力失败，第一阶段被迫将气吹机间距压缩至 2,400-2,500 英尺并增至 7 台级联。
- [12:18](https://www.youtube.com/watch?v=XJk7SJzKHuo&t=738s) 团队推行井下级联直吹且彻底取消光缆盘留（No Slack），不仅解决了井下堆叠 140 圈光缆的排障隐患，还将地面封路范围缩减至 100-200 英尺。
- [15:18](https://www.youtube.com/watch?v=XJk7SJzKHuo&t=918s) 第二阶段敷设 76 根超大芯数 1,728 芯光缆时，因线缆重量与外径翻倍，气吹机间距遭遇二次缩紧，被迫进一步收缩至 1,800 英尺。
- [16:12](https://www.youtube.com/watch?v=XJk7SJzKHuo&t=972s) 工程核心启示在于“测量两次但绝不害怕下刀”——以实测与快速动工揭示物理瓶颈，并依靠现场敏捷度快速迭代修正。
- [16:12](https://www.youtube.com/watch?v=XJk7SJzKHuo&t=972s) AI 基础设施的核心本质是“人与人的连接”——通过跨越政企与同行边界的开诚布公协商，才得以化解复杂的路权冲突与施工危机。
## Detailed walkthrough
### [00:05](https://www.youtube.com/watch?v=XJk7SJzKHuo&t=5s) 时代巨变：从数十芯采购到十万芯自建的挑战
如果有人交给你一项前所未有的任务，要求你去解决公司近年来最核心的重大战略难题，你会作何感想？Meta 网络投资团队的合作伙伴经理法布里斯·旺吉（Fabrice Ouandji）以此为引言，拉开了这场关于基础设施技术革新的序幕。旺吉所在的团队主要负责为 Meta 内部网络的数据中心与网络接入点（PoP）之间采购或建设光纤连接。而他和同事塞巴斯蒂安·高尔特（Sebastian Gault）此行的目的，是揭秘位于俄亥俄州新奥尔巴尼（New Albany）的 Prometheus AI 训练集群背后，那些为了打破物理连接极限所做出的工程创新。
回溯 Meta 的光纤发展史，在 2020 年之前（大约 2018 年前后），团队接到的任务不过是每条路径交付 10 到 12 芯光纤。彼时只要有新数据中心拔地而起，团队只需向 Zayo 或 Lumen 这样的传统电信运营商租用或采购少量纤芯即可应付需求。然而从 2020 年开始到 2024 年，流量需求呈爆发式增长，促使 Meta 转型为自己动手修网：工人们沿着道路路权（Right-of-Way）开挖路面，埋下两三根管道，并在其中一根管道中穿入一条容纳 432 芯至 864 芯的光缆，另一根管道则留作未来扩展备用。长久以来，团队对这种传统部署模式驾轻就熟。但 Prometheus 这一超大规模 AI 集群的出现，彻底将原有的网络架构逻辑颠覆——业务方提出的新需求是：数据中心之间每条路径必须提供超过 100,000 芯光纤。这个数字几乎是 Prometheus 诞生前部署规模的 100 倍。
```mermaid
flowchart LR
accTitle: 数据中心间光纤部署规模的历史演进
accDescr: 展示从2018年租用12芯，到2020至2024年自建864芯，再到Prometheus阶段自建超10万芯的演进过程。
subgraph Phase1["2018年（前2020时代）"]
A["每条路径 10-12 芯<br/>（向 Zayo / Lumen 等运营商租购）"]
end
subgraph Phase2["2020-2024年（自建光纤时代）"]
B["自建少量管道<br/>（每路径部署 432-864 芯光缆）"]
end
subgraph Phase3["Prometheus 时代"]
C["AI 训练集群互联<br/>（每路径超 100,000 芯，规模飙升100倍）"]
end
A --> B --> C
```
- **历史容量对比**：2018 年单路径约 12 芯；2020—2024 年自建单路径 432 至 864 芯；Prometheus 阶段跃升至单路径 100,000+ 芯。
- **建设范式转移**：从“依赖外部运营商采购网络”逐步过渡到“自建传统电信管道”，最终走向“重度土木工程级别的超级集群管道集群”。
---
### [02:46](https://www.youtube.com/watch?v=XJk7SJzKHuo&t=166s) 光纤创新与“改善”（Kaizen）：挑选与定制高密度光缆
Prometheus 项目的首席项目经理、网络部署工程师塞巴斯蒂安·高尔特接过了话筒，详细介绍了光纤层面的技术选型与创新。面对前所未有的工程量，Meta 团队在北卡罗来纳州阿什维尔（Asheville）的一处导管制造工厂组织了一场光纤“改善”（Kaizen，源自日语，意为持续改进）测试。为了保障后续成百上千根光缆的气吹敷设（Jetting）能够顺畅推进，团队邀请了三家最核心的光缆制造商同台竞技，各自拿出底牌产品，在长达 3,700 英尺的真实测试跑道上展开了一场公平公开的“同台对比”气吹试验。
最终拔得头筹的是 AFL 生产的标准 864 芯光缆，其外径为 15 毫米。高尔特特别解释了团队为何断然舍弃更细的微缆（Micro Cable）而选择标准光缆：关键在于刚性（Rigidity）与抗折弯性能。在单条路径上，工程团队需要连续气吹多达 140 根 864 芯光缆，在这种高密度的极端施工现场，微缆极易发生死折（Kinking），或者在混乱中被工人无意踩踏致损，而标准光缆的结构强度则能有效避免这类致命隐患。然而，受限于地下管道空间有限，团队随后又与 AFL 联合研发了面向二期部署的杀手级产品——外径仅为 16.4 毫米的 1,728 芯高密度光缆。它的截面仅比 864 芯光缆大 1.4 倍，却能在相同的地下管道空间内直接实现纤芯容量翻倍，赋予了网络极大的弹性与可扩展性。
这套基础设施的铺设体量惊人：仅仅在新奥尔巴尼周围半径 5 公里的狭小区域内，Meta 敷设的光纤总长度就达到了 210 万光纤公里（Fiber Kilometers）。高尔特感慨地比喻道：“如果你仔细想想，这个长度相当于从地球去往月球并返回整整三次”，而在如此紧凑的物理空间内完成这种级别的部署，在通信工程历史上堪称绝无仅有。
- **测试规模**：在阿什维尔工厂的 3,700 英尺测试轨道上进行 1:1 实际气吹测试。
- **一期核心线缆**：AFL 864 芯标准光缆，外径 15 毫米，强调抗弯折与施工耐用性。
- **二期定制创新**：AFL 1,728 芯超密光缆，外径仅 16.4 毫米，在相同管径下将纤芯密度提升 100%。
- **区域总里程**：在半径 5 公里范围内铺设 210 万光纤公里，总长足以往返月球 3 趟。
---
### [05:39](https://www.youtube.com/watch?v=XJk7SJzKHuo&t=339s) 土木工程突破：挖掘“能叠放三辆SUV”的超级管廊
当光缆方案敲定后，真正的物理瓶颈落在了地下管廊（Conduit Bank）的土木工程上。法布里斯·旺吉重新接过讲解，指出要想将传统容量扩大 100 倍，就必须将重型土木工程技术跨界引入电信基础设施领域。在传统电信建设中，工人们开挖的沟槽通常只有 6 英寸（约 15 厘米）宽、3 到 4 英尺（约 0.9 至 1.2 米）深，里面塞入一到三根导管即可完事。然而，Prometheus 项目为了在单条路径上容纳由 140 多根光缆组成的 10 万芯庞大网络，导管数量随之剧增，沟槽尺寸也发生了质的飞跃。
在新奥尔巴尼的工地上，Meta 开挖的沟槽宽度达到了惊人的 9 英尺（约 2.7 米），部分路段的开挖深度甚至逼近 20 英尺（约 6.1 米）。旺吉形象地描述其横截面：“这已经大到足够把三辆 SUV 汽车垂直叠放在里面了。”面对这样超常规的开挖体量，传统电信布线承包商根本无能为力。Meta 果断跳出原有的供应商圈子，寻找专门从事市政重型土木建设的合作伙伴——那些平日里负责开挖下水道主干管、市政供水管网、高压电缆管廊以及大型雨水防洪排涝系统的专业团队。他们最终选中了 Congruex 集团旗下的 OVE（Ohio Valley Excavating，俄亥俄河谷挖掘公司），正是凭借这批重型土方工程专家的机械与施工经验，才在城市地底啃下了这条巨大的超级管廊。
- **传统电信沟槽尺寸**：宽约 6 英寸，深约 3 至 4 英尺。
- **Prometheus 超级沟槽尺寸**：宽达 9 英尺，深达 20 英尺（体积相当于叠放三辆全尺寸 SUV）。
- **施工跨界合作**：放弃传统弱电施工队，引入市政级重工业重型挖掘承包商 Congruex / OVE。
---
### [07:44](https://www.youtube.com/watch?v=XJk7SJzKHuo&t=464s) 动态需求、施工冲突与地下穿越工程
如此庞大的土木工程在现实推进中遭遇了两大核心难题。第一个棘手挑战在于“高度动态的数据中心选址需求”与“只许成功一次的工程开挖”之间的尖锐对立。Meta 打造 Prometheus 集群的目标是让多座物理数据中心如同单台超级计算机一样协同运作，但在工程破土动工之初，未来所有数据中心落户的确切位置与容量需求根本没有完全锁定。而一条 9 英尺宽的超级沟槽在施工时会彻底吃掉整整一条机动车道，市政交通与审批部门绝不可能允许 Meta 把路挖开、铺好、封闭路面后，过几个月因为新站点的敲定又重新把同一条马路挖开一次。这种重复开挖在成本上无法承受，在市政许可审批上更是天方夜谭。
为了在需求未定的大局下破局，工程团队采取了“一次性全面勘测规划”策略：在整个新奥尔巴尼 5 到 10 公里半径的巨大扇区内，逐一摸底该区域内的每一条市政道路。团队深度调取现有地下管线（水、电、燃气、通信）的分布情况，精密计算出在哪些路段可以直接借用路权边侧的草坪进行开挖敷设，在哪些路段必须申请封路许可、把管廊深埋在机动车道下方并随后整体修复路面。
第二个巨大挑战则是极其恶劣的“施工交通维持”（Maintenance of Traffic）。当时的建设现场堪称全球算力基建最密集的风暴眼：在这 5 到 10 公里半径的方圆之内，不仅 Meta 在疯狂动工，微软（Microsoft）、谷歌（Google）、亚马逊（Amazon）也都在建设或扩建数据中心；Vantage 和 QTS 等大型托管数据中心服务商同样在日夜赶工；更夸张的是，英特尔（Intel）还在该区域东北角投资数百亿美元兴建半导体晶圆代工厂（Foundry）。整个片区每天有高达 5,000 至 10,000 名各行业建筑施工工人驾车穿行于有限的道路网络中。Meta 一边要在高峰期封闭机动车道进行深坑挖掘，一边要确保自家数据中心工人、友商超级数据中心工人以及英特尔项目的通勤通畅。这倒逼团队必须与内部数据中心团队、各大科技巨头、总承包商以及市政许可监管机构建立极高密度的日常协同与沟通机制，最终通过精确到小时的交通引导克服了封路危机。
最后，旺吉展示了一项令人叹为观止的穿越工程奇迹。在遭遇无法开挖的繁忙高速公路节点时，团队实施了超大口径水平定向钻孔（Bore），在高速路地下打通了一条直径达 5 英尺（约 1.5 米）的超级管道，以便为已知和未来的算力需求尽可能多地塞入导管。这条管道的直径巨大到机器操作员可以直接端坐在专用的掘进机械内部，人机一体直接钻入地下管道深处进行内部掘土与清渣作业，其工程规模彻底超越了人们对传统通信布线的固有认知。
- **区域施工并发密度**：涵盖 Meta、微软、谷歌、亚马逊、Vantage、QTS 数据中心以及英特尔数百亿美元晶圆代工厂。
- **高峰期交通压力**：每天多达 5,000 至 10,000 名建筑技术工人（Trades）挤占道路网络。
- **高速路下穿奇迹**：完成了外径达 5 英尺的大型地下穿越管道，施工挖掘工人可直接驾驶机器进入管道内部进行土石掏挖与清理。
### [11:21](https://www.youtube.com/watch?v=XJk7SJzKHuo&t=681s) 超大规模土木工程：如同单身公寓般的巨型地下光缆井
在跨越公路等关键障碍物时，团队面临着严苛的现实限制——在高速公路下方顶管钻孔施工往往“只有一次机会”（"we knew we only had one shot to be able to bore under a highway"），必须尽可能密集地塞入最大容量的管道。施工现场挖出的巨型管沟规模极其惊人，在很多工段甚至宽大得犹如陨石坑一般。而在管道路由的关键交汇处，团队部署的地下光缆井（Vaults/Manholes）更是彻底颠覆了传统电信工程的标准。
平常人们在马路上开车压过的常规手孔井通常只有 3 英尺乘 4 英尺（约 0.9 米 × 1.2 米），而为了容纳 Prometheus 项目庞大的互联管道与光缆接入，Meta 建造的地下人孔井尺寸达到了惊人的 10 英尺乘 12 英尺（约 3 米 × 3.7 米）。Fabrice 打趣地形容这些地下井“几乎和单身公寓一样大”（"almost as big as a studio apartment"）。这些巨大的地下空间作为网络检修与接入的枢纽，使工程师后续作业时无需再次开挖路面破坏地表。正是这种前所未见的重度土木工程，才为 Prometheus 的非标光纤骨干网打下了坚实的物理底座。
- **标准手孔井尺寸对比**：普通电信手孔约为 3 ft × 4 ft，而 Prometheus 项目专用巨型井体达 10 ft × 12 ft。
- **施工策略**：高速路下钻孔要求一次性容纳未来所有管孔需求，杜绝重复破路。
---
### [12:18](https://www.youtube.com/watch?v=XJk7SJzKHuo&t=738s) 气吹光缆的试错与突破：从理论设计到现场重塑
当庞大的地下管道网络落成后，重担交到了负责光纤敷设与气吹（Fiber Jetting）的 Sebastian Gault 手中。团队的使命是跨数据中心交付数百根高芯数光缆：第一阶段需要铺设 140 根光缆，第二阶段则是 76 根超大芯数光缆。在这项前无古人的工程中，团队唯有依靠不断试错。为此，Sebastian 组织协调了光缆制造商 AFL、气吹设备商以及管道制造商 MTEL，展开全流程的白板推演、联合设计与现场实测。
在最初的白板推演中，团队根据设计计算制定了详尽方案，计划将气吹机间隔设定为 3,800 英尺（约 1,158 米）。然而实地测试证明，理论计划在现实面前迅速碰壁——光缆自重远超预期，当气吹推至 3,800 英尺后，下一台机器根本无法顺利接力牵引。团队被迫大幅缩短气吹机之间的间距，原本预计只需要 3 到 5 台气吹机，最终不得不增加至 7 台级联作业。
```mermaid
flowchart TD
accTitle: Fiber Jetting Workflow Comparison
accDescr: 比较第一阶段与第二阶段气吹光缆间距与部署策略的变化过程
subgraph Phase1["第一阶段 (Phase 1)"]
direction TB
A1["140 根 864 芯光缆"] --> B1["气吹机间距：缩短至 2,400-2,500 ft"]
B1 --> C1["级联气吹机直置于井内 / 无盘留 (No Slack)"]
end
subgraph Phase2["第二阶段 (Phase 2)"]
direction TB
A2["76 根 1,728 芯光缆 (重量翻倍)"] --> B2["原有 2,500 ft 间距失效"]
B2 --> C2["二次缩短至 1,800 ft 间距并增设级联机台"]
end
Phase1 --> Phase2
```
与此同时，团队做出了一个极为大胆的技术决断：在井内采用“级联直吹”（Cascaded Jetting），彻底取消井内光缆盘留（Slack coils）。如果在同一个检查井里堆积 140 圈多余光缆，现场将杂乱无章，一旦遭遇断纤故障更会成为排障噩梦。团队直接把气吹机降入巨型井下，让光缆直接全管贯通敷设，不仅彻底规避了井下复杂的线缆管理问题，还将地面封路范围从整条车道压缩到了仅 100 至 200 英尺，将对纽奥尔巴尼市（City of New Albany）公共交通的干扰降到了最低。
- **设备与厂商协同**：联合 AFL（光缆）、气吹设备厂及 MTEL（管道）进行多方联调。
- **气吹机数量激增**：从设计预估的 3-5 台最终增加到现场实际部署的 7 台级联。
- **路面交通影响优化**：利用下井施工将封路区间控制在 100-200 英尺以内，避免整条车道长期瘫痪。
---
### [15:18](https://www.youtube.com/watch?v=XJk7SJzKHuo&t=918s) 第二阶段挑战升级：应对翻倍重量的 1,728 芯超重光缆
进入第二阶段后，团队需要在三座数据中心之间的线路上敷设 76 根 1,728 芯超高密度光缆。尽管第一阶段积累了宝贵经验，但新的物理瓶颈再度出现。团队最初沿用了第一阶段成功验证的 2,400 至 2,500 英尺气吹间距，并同样将气吹机级联下井，但推进过程很快受阻。
原因在于 1,728 芯光缆的纤芯数量比 864 芯翻了一倍，光缆本身的重量与外径也近乎翻倍。面对沉重无比的超高芯数缆线，原有的推力与气压参数无法支撑 2,500 英尺的跨度。现场团队不得不迅速重新评估气吹动力学模型，最终将每台气吹机的部署间距进一步缩短至 1,800 英尺（约 548 米）。间距的再次收紧意味着线路上必须增设更多气吹机台与施工节点，团队以极强的现场敏捷度动态调整了施工方案，最终确保了这批关键互联光缆如期点亮。
- **第一阶段规格**：在两座数据中心之间气吹 140 根 864 芯光缆。
- **第二阶段规格**：在三座数据中心互联线路上气吹 76 根 1,728 芯超重光缆。
- **间距二次压缩**：受缆重制约，气吹机间距从 2,400-2,500 ft 被迫缩短到 1,800 ft。
---
### [16:12](https://www.youtube.com/watch?v=XJk7SJzKHuo&t=972s) 三大核心启示：工程敏捷性、人际连接与工作范式革新
在总结这场空前工程的成败得失时，Fabrice 提炼出了三条至关重要的核心经验：
首先是**“测量两次，但绝不要害怕下刀切割”**（"measure twice but do not be afraid to cut"）。面对前所未有的复杂工程，严密的前期规划至关重要——团队通过 Kaizen 改善机制严选光缆厂商，进行反复气吹实测，耗费大量精力精细规划路由。然而，现实往往偏离计划，真正的关键在于“敢于开始”。如果因害怕偏离原案而止步不前，就无法发现真实地质与物理环境中的瓶颈。唯有先动工、在实地碰壁后再快速修正（灵活缩短气吹间距、调整设备部署），才能在极限工期内突围。
其次是**“AI 的底层是关于人与人的连接”**（"AI is about human connections"）。Prometheus 绝非单纯的技术或算力堆砌，它牵涉到极度复杂的多方协作：从负责征地的数据中心土地拓展团队，到负责土建的现场施工团队；从邻近区域同样在施工作业的其他超大规模云服务商（Hyperscalers），到地方市政的规划审批部门。当路由设计产生冲突、进度发生挤压、乃至地下管道出现意外破损时，化解危机的终极手段从来不是冰冷的指令，而是现场工程师与各方负责人坐在一起开诚布公地沟通协商。
最后是**“勇于拥抱全新的工作方式”**（"embracing new ways of working"）。为了完成这场前所未有的工程突破，Meta 彻底打破了传统通信运营商的基础设施建设惯例。他们引入了高规格的专业土木工程集团来主导大型地下管道包（Duck Bank）的建设，并与纽奥尔巴尼市政府建立了深度互信的政企协作机制，以解决新型非标工程的合规与许可问题。Fabrice 强调，面对人工智能时代的算力需求，解决高难度工程挑战将是 Meta 的常态，这种跨越行业边界的合作范式将成为未来持续突破技术边界的核心动力。
- **启示一：敏捷规划**——详尽规划与大胆执行并重，根据现场实测数据敏捷调整工程参数。
- **启示二：多元协同**——打通内外部生态，包括土地拓展、数据中心施工、友商超大规模厂商（Hyperscalers）及地方市政机构。
- **启示三：跨界伙伴**——突破传统电信外包模式，联合顶级重型土建企业与市政部门共建创新工程流程。
## Notable quotes
> [02:46](https://www.youtube.com/watch?v=XJk7SJzKHuo&t=166s) 如果你仔细想想，这个长度相当于从地球去往月球并返回整整三次 — 塞巴斯蒂安·高尔特（Sebastian Gault）
> [05:39](https://www.youtube.com/watch?v=XJk7SJzKHuo&t=339s) 这已经大到足够把三辆 SUV 汽车垂直叠放在里面了。 — 法布里斯·旺吉（Fabrice Ouandji）
> [11:21](https://www.youtube.com/watch?v=XJk7SJzKHuo&t=681s) we knew we only had one shot to be able to bore under a highway — 法布里斯·旺吉（Fabrice Ouandji）
> [11:21](https://www.youtube.com/watch?v=XJk7SJzKHuo&t=681s) almost as big as a studio apartment — 法布里斯·旺吉（Fabrice Ouandji）
> [16:12](https://www.youtube.com/watch?v=XJk7SJzKHuo&t=972s) measure twice but do not be afraid to cut — 法布里斯·旺吉（Fabrice Ouandji）
> [16:12](https://www.youtube.com/watch?v=XJk7SJzKHuo&t=972s) AI is about human connections — 法布里斯·旺吉（Fabrice Ouandji）
> [16:12](https://www.youtube.com/watch?v=XJk7SJzKHuo&t=972s) embracing new ways of working — 法布里斯·旺吉（Fabrice Ouandji）
## Mentioned
Meta, 法布里斯·旺吉（Fabrice Ouandji）, 塞巴斯蒂安·高尔特（Sebastian Gault）, Prometheus, 新奥尔巴尼（New Albany）, Zayo, Lumen, AFL, MTEL, Congruex, OVE（Ohio Valley Excavating）, 微缆（Micro Cable）, 气吹（Jetting）, 改善（Kaizen）, 微软（Microsoft）, 谷歌（Google）, 亚马逊（Amazon）, Vantage, QTS, 英特尔（Intel）
## Video description
Prometheus introduced significant complexity for connectivity. We were faced with significantly higher fiber count requirements, highly dynamic demand location, and compressed timelines for delivery.
Watch now to find out the innovations required to meet the demand across multiple data centers in the cluster.
Learn more about the @Scale conferences here: https://atscaleconference.com/
[Source](https://www.youtube.com/watch?v=XJk7SJzKHuo)
Source quotes
[2:46] 塞巴斯蒂安·高尔特（Sebastian Gault）: 如果你仔细想想，这个长度相当于从地球去往月球并返回整整三次
[5:39] 法布里斯·旺吉（Fabrice Ouandji）: 这已经大到足够把三辆 SUV 汽车垂直叠放在里面了。
[11:21] 法布里斯·旺吉（Fabrice Ouandji）: we knew we only had one shot to be able to bore under a highway
[11:21] 法布里斯·旺吉（Fabrice Ouandji）: almost as big as a studio apartment
[16:12] 法布里斯·旺吉（Fabrice Ouandji）: measure twice but do not be afraid to cut
[16:12] 法布里斯·旺吉（Fabrice Ouandji）: AI is about human connections
[16:12] 法布里斯·旺吉（Fabrice Ouandji）: embracing new ways of working
Source quotes
[0:05] Fabrice Ouandji: What if you were asked to do something that has never been done before?
[2:46] Sebastian Gault: three times to the moon and back
[5:39] Fabrice Ouandji: accommodate three SUVs stacked on top of each other
[12:18] Sebastian Gault: 140 slack coils in the manhole
[12:18] Sebastian Gault: messy
[16:12] Fabrice Ouandji: Measure twice, but do not be afraid to cut.
[16:12] Fabrice Ouandji: AI is about human connections.
Original transcript
[0:05] What if you were asked to do something
[0:07] that has never been done before?
[0:10] Imagine being tapped to work on solving
[0:13] complex problem in support of the
[0:16] company's biggest priority in years.
[0:20] Hi, my name is Fabris Wanji. I'm a
[0:22] partner manager within the network
[0:24] investment team. My team is responsible
[0:27] for sourcing or building cyber
[0:29] connectivity between data centers and
[0:32] point of presence pops within the meta
[0:34] network. Today my colleague Seb and I
[0:38] want to talk to you about some of the
[0:40] innovations that was required to support
[0:43] the Prometheus AI training cluster in
[0:46] Alab Ohio.
[0:49] Now before we talk about Prometheus,
[0:51] it'll be interesting to go back to kind
[0:54] of talk about where we started from. And
[0:57] if I take you back to probably pre2020,
[1:00] so about 2018, my team was asked to
[1:04] deliver about 10 strands, 12 strands per
[1:09] path, right? So if a new data center was
[1:11] coming up, we had to deliver a couple of
[1:13] fiber paths. Each one of them will have
[1:15] about 12 strands. The way we solved it
[1:17] will be to go and source that fiber from
[1:20] a carrier like Z or Lumen and that was
[1:23] that was good enough. Starting about
[1:25] 2020 leading up to almost 2024 we
[1:28] started seeing an increase in demand and
[1:30] that led us to start building fiber
[1:32] networks. So we go ahead and open up
[1:35] roads in the rideway, place a couple of
[1:38] conduits and place one cable in one of
[1:40] those conduits with the second one being
[1:42] reserved for future growth. And that
[1:44] cable would typically carry anywhere
[1:46] from 432 strands to about 864 strands.
[1:50] And we were pretty good at that. We're
[1:52] doing that. We could deliver that. We
[1:53] knew how to do that. Prometheus flipped
[1:56] that model completely upside down. With
[2:00] Prometheus, we were asked to deliver
[2:02] over a 100,000 strands per path between
[2:05] data centers. Let me repeat that. Over a
[2:08] 100,000 strands per path between data
[2:11] center. If you're tracking that's almost
[2:13] a hund times what we were doing prior to
[2:16] Prometheus.
[2:18] So with that in mind the rest of the
[2:21] conversation will go into how we went
[2:23] about solving that problem. Uh first my
[2:26] my colleague Seb will talk about the
[2:27] fiber innovations that was required. I
[2:30] would then talk you through some of the
[2:31] civil engineering work that we carried
[2:33] out. Then Seb will come back and take
[2:36] you through the fiber placement process
[2:38] and again some of the innovation that
[2:39] went into that. and we'll close with
[2:41] some lessons learned and as we look to
[2:43] the future. With that, I'll turn it over
[2:45] to Seb.
[2:46] >> Thank you for brief
[2:49] G. I'm a network implementation engineer
[2:51] at Meta and I was the lead PM for
[2:53] project Prometheus. I'm going to walk
[2:56] you through the fiber innovation that
[2:57] we've encountered and that we've done
[2:59] throughout this project. Also, the fiber
[3:02] kaizen that we held at our conduit
[3:04] manufacturers facility in Asheville,
[3:06] North Carolina.
[3:08] Kaizen being a Japanese word for
[3:11] continuous improvement. We did this to
[3:13] find the best 864 cable for this
[3:16] deployment. Critical to ensure a smooth
[3:20] build and a smooth jetting operation.
[3:23] We invited three of our closest and best
[3:28] cable manufacturers to to a jetting
[3:31] trial.
[3:32] We've asked them to bring what they
[3:35] think, what they believe is the best
[3:37] cable for this deployment. We jet it on
[3:40] a 3,000
[3:42] 700 ft test track in applesto apples
[3:48] comparison.
[3:50] We chose the 864 cable from AFL. It's a
[3:55] standard 864 15 mm in diameter. Why did
[4:00] we choose this and not the micro cable?
[4:02] We chose the standard because it's more
[4:04] rigid. It's not prone to kink. You have
[4:07] to you have to remember we're jetting
[4:10] 140 864 cables per path.
[4:14] If you do this, there's a high risk of
[4:17] micro cables cking, somebody stepping on
[4:20] it. We didn't want that to happen, so we
[4:23] went with the standard.
[4:25] taken you to our fiber innovation what
[4:28] we have then developed with AFL. We
[4:31] needed something for our phase 2
[4:33] deployment because conduit space is
[4:36] limited. We built a lot and Fris is
[4:39] going to take you through that on the
[4:40] next slides. But we developed a 1728
[4:45] fiber cable with a diameter of 16.4
[4:50] mm that will fit in the same conduit.
[4:53] It's only 1.4 4 times bigger than the
[4:57] 864. So it gives us a lot more
[5:00] flexibility. It gives us future
[5:02] scalability with doubling the fiber in
[5:05] the fiber count in the same conduit
[5:09] space.
[5:12] In a 5 kmter radius in New Albany, Ohio,
[5:16] we're going to be placing 2.1 million
[5:19] fiber kilometers.
[5:21] That is and if you think about that is
[5:24] that is three times to the moon and back
[5:27] which is crazy for something in a 5 km
[5:31] radius that nobody has ever done before.
[5:34] I'm handing back over to Fris and he's
[5:36] going to run you through the civil
[5:38] engineering.
[5:39] >> Thank you Seb. You just heard about the
[5:41] choice of the cable that was required
[5:43] for this project. Let's talk about the
[5:46] conduit bank that we had to build to
[5:48] actually accommodate the cable. And
[5:50] again, if you hearken back to where we
[5:52] started, we're trying to deliver 100
[5:54] times the capacity that we typically
[5:56] deploy. How do you do that? You really
[5:58] bring civil engineering expertise to the
[6:01] world of telecom infrastructure. And let
[6:03] me explain that, right? So in a typical
[6:06] telecom infrastructure deployment, as I
[6:08] mentioned, we would go, we typically
[6:10] place one to three conduits. We place
[6:13] one cable in one of those conduits and
[6:16] we have about 864 count to be able to
[6:20] have a 100,000 strands and place say 140
[6:24] cables as as Seb mentioned you need that
[6:28] many more conduits and that requires a
[6:30] much larger trench right so we go from a
[6:34] trench that's about 6 in wide and about
[6:36] 3 to 4 feet deep in our traditional
[6:39] telecom deployment to a trench that was
[6:42] in this project 9 ft wide by almost 20
[6:46] ft deep in some cases. Let me repeat
[6:48] that. 9 ft wide by 20 ft deep. What does
[6:51] that mean in practice? That's big enough
[6:53] to accommodate three SUVs stacked on top
[6:56] of each other. And to be able to build
[6:58] that kind of trench, you need a
[7:00] different type of partner. And that's
[7:01] what we did here. Right? So, this
[7:03] required really a lot of expertise in
[7:07] civil engineering. the type of builds
[7:09] that you know that are involved in
[7:12] deploying sewer lines, deploying water
[7:14] lines, electricity lines, right? And all
[7:17] these storm systems, these huge projects
[7:20] that require excavation. And we went
[7:23] outside of a traditional partners and
[7:24] find a company that had exactly that. Of
[7:27] course, it helps that their former name
[7:29] was Ohio Valley Excavating OVE, which is
[7:33] part of the Congruex group of companies.
[7:36] So we worked with congrux and obbe to
[7:38] build these type of trenches and as we
[7:41] did that we had to overcome a few
[7:44] challenges right and the first one was
[7:46] really around the just the position of
[7:48] the dynamic demand versus the oneshot
[7:52] approach to building and what do I mean
[7:54] by that we wanted to connect multiple
[7:56] data centers to act as one that's what
[7:58] we call a cluster the problem is that we
[8:01] did not always know where all the data
[8:03] centers would be when we started
[8:05] building right And because of the size
[8:06] of a trench, we really only had one shot
[8:10] of it, right? It was very unlikely that
[8:12] the city would let us take a lane of
[8:14] traffic, which is really the 9 ft that
[8:15] we need. Open it up, close it, enable
[8:18] one site, and come back later if the man
[8:21] was confirmed on the same road for a
[8:23] different site, and open that up again.
[8:24] It would have been too expensive, and
[8:26] frankly, permitting wouldn't have been
[8:28] possible. So we were kind of confronted
[8:30] with that problem of build starting to
[8:32] build without fully knowing exactly
[8:34] where the demand was going to be and how
[8:36] much of it was going to be because each
[8:38] site comes with a different requirement
[8:40] in terms of the number of strands. Our
[8:42] solution was to really do two things.
[8:44] First is take a wide swath within a 5 to
[8:48] 10 kilometer radius in New Albany.
[8:50] Engineer each of those roads and what
[8:52] that means is engineer each of the roads
[8:55] in that in that radius. And what that
[8:57] means is getting a full understanding of
[8:59] the existing utilities but then also
[9:02] being able to understand where we could
[9:04] go and deploy a duck bank within the
[9:06] rightway. In some cases it would be
[9:08] existing right away in the grass. In
[9:11] other cases it meant actually applying
[9:13] for permits to go and open up the road
[9:15] and put our duck bank underneath the
[9:18] road and then do some restoration. So
[9:20] that was kind of the engineering part of
[9:21] it. Once we started constructing, we
[9:24] also had then to overcome challenges
[9:26] related to maintenance of traffic. And
[9:29] let me set the stage for you. That 5 to
[9:31] 10 kilometer radius I talked about
[9:33] today. We have Microsoft, you have Meta,
[9:36] you have Google and you have Amazon who
[9:39] have existing facilities or are in the
[9:42] process of developing facilities. You
[9:44] also have data center providers like
[9:46] Vantage and QTS who are also working on
[9:48] developing facility. That mean a lot of
[9:50] construction. As if is that is not
[9:53] enough. You also have Intel who has a
[9:55] multi-billion dollar deploy development
[9:58] to create a foundry in a northeast
[10:00] corner of the of that radius. So
[10:03] altogether you're probably talking
[10:04] upwards of 5,000 to 10,000 straits who
[10:08] are trying to traverse these roads every
[10:11] day to support these different different
[10:12] construction projects as we are going in
[10:15] shutting down traffic lanes to be able
[10:17] to do our construction. So major
[10:20] challenge right. So it required a lot of
[10:22] coordination with internal stakeholders
[10:24] like our data center teams that also
[10:26] needed their own trades into the sites
[10:29] but also external stakeholders the
[10:31] different hyperscalers their contractors
[10:34] the city and the permitting authorities
[10:36] that had strict requirements around
[10:38] permitting. So we were through
[10:40] collaboration and really a lot of
[10:42] communication able to overcome some of
[10:44] those challenges.
[10:46] Now let's talk about again the the
[10:48] engineering marvel here and we want to
[10:50] kind of put that in perspective. What
[10:52] you see on the left is a bore that we
[10:55] had to do. We had to bore under a
[10:57] highway and pack as many conduits as
[11:00] possible both to serve known demand and
[11:03] unknown demands. And that bore that you
[11:06] see the diameter of that big pipe is
[11:09] about 5 ft. What that meant is the
[11:11] operator that was responsible for
[11:14] digging actually sat in a machine and he
[11:16] was actually able to go inside that tube
[11:20] to be able to remove the ground. That's
[11:21] how wide and that big that trench was.
[11:24] And again, all of that was done because
[11:25] we knew we only had one shot to be able
[11:28] to bore under a highway. We had to pack
[11:30] as many count as possible. What you see
[11:33] on the right is what we call vaults.
[11:35] Think about manholes and hand holes that
[11:37] you typically drive over. Those are
[11:39] access point to connect into our network
[11:41] so that you don't have to go again and
[11:43] open the trench. A typical handhole is
[11:46] probably three feet by four feet. Here
[11:48] we're talking about 10 by 12 ft. Super
[11:51] big, almost as big as a studio
[11:53] apartment. And again, the trenching
[11:56] that's required to deploy some of these
[11:59] really resemble a crow a crater at
[12:01] times. So this is the type of civil
[12:03] engineering that was required to really
[12:06] support this non-standard built from
[12:09] Prometheus. Now that that's done, I'm
[12:11] going to turn it back over to Seb that
[12:14] will take you through placing the fiber
[12:15] once you have these conduits in place.
[12:18] Over to you Seb. Thank you for as you
[12:21] now heard from Fris about all the civil
[12:23] engineering and all the issues and risk
[12:25] we ran into. I'm going to talk you
[12:28] through our fiber jetting and the fiber
[12:30] placement that we had to overcome to
[12:32] deliver 140 cables on phase one and 76
[12:37] cables on phase 2. The main point here
[12:40] was it was a trial and error. So we
[12:42] trial trial trial. I coordinated a field
[12:46] trial and a design trial with AFL our
[12:49] cable manufacturer FLT our jetting
[12:52] manufacturer and and our conduit
[12:54] manufacturer MTEL. We had a whiteboard
[12:57] session where we ran through where are
[12:59] we going to place our jetting machines.
[13:02] We came up with a really good plan, but
[13:05] the plan was called it was a plan, a
[13:07] triant plan, most likely going to fail
[13:11] in the field. And our first test showed
[13:14] that this was exactly what happened. We
[13:17] had our jetting machine set
[13:20] 3,800 ft apart because this is what the
[13:22] design showed.
[13:24] The issue we ran into that the that the
[13:27] cable was too heavy and we weren't able
[13:29] to pick it back up after 3,800 ft. So,
[13:32] we had to move jetting machines closer
[13:34] together which then caused that we had
[13:36] to use more jetting machines. We were
[13:39] planning with three to five. In the end,
[13:41] we used seven.
[13:43] What we also did is we cascaded jetting
[13:46] machines because we didn't want to use
[13:48] slack. Could you imagine having 140
[13:51] slack coils in the manhole? Messy. in a
[13:55] case of an outage, not really nice to
[13:57] work with. So, we placed the jetting
[14:00] machines into the vault
[14:03] to jet it through. On phase one, as I
[14:07] already mentioned, we jetted 140 864
[14:10] cables between two of our data centers.
[14:13] Most of the risks involved, how many
[14:15] cables are we going to jet per day to
[14:17] meet our timeline? We had a deadline
[14:20] that we had to meet because otherwise we
[14:22] couldn't turn up the data centers and
[14:24] GPUs would be hanging in the data
[14:26] centers without any light on. Monitoring
[14:28] of traffic with the city. So we had to
[14:31] work closely with the city of New Albany
[14:34] to how many lanes can we shut down? Can
[14:37] we shut down a full lane as you saw from
[14:39] Freeze or can we only shut down 100 feet
[14:43] 200 feet? Because we dropped the jetting
[14:46] machines into the manholes that gave us
[14:50] the opportunity to only only bury off
[14:54] 100 ft, maybe 200 in most cases and not
[14:57] impact traffic too much.
[15:00] Then the next one was also the cable
[15:02] management inside the vault. It's like
[15:04] how are we going to do this? What if
[15:06] there are slack holes? We finally came
[15:09] to the conclusion that we don't need
[15:10] slack. So, we didn't have to worry about
[15:13] the cable management inside the vault as
[15:15] it was fully subducted through. In phase
[15:18] two, we jetted 76 1728 fiber cables per
[15:22] route between three data centers.
[15:26] Different challenges that we didn't
[15:28] think of. We placed the jetting machines
[15:30] the same 24 2500 feet like we did with
[15:33] the 864.
[15:35] But that didn't that didn't do it. We
[15:37] jetted the first we cascaded again
[15:40] dropped them into the manhole. Dropped
[15:42] the jetting machines into the manholes.
[15:45] But that cable now has double the
[15:47] fibers. So it's almost double as heavy.
[15:51] So this had us re-evaluate how we need
[15:54] to jet the cables. So, we had to bring
[15:56] the cables in, the jetting machines in
[15:59] and do 1,800 ft. This also added more
[16:02] jetting machines on these other routes
[16:04] that we jetted these cables. As you've
[16:07] heard all about this, I'm going to hand
[16:09] it back off to Fris for the key
[16:12] takeaways. Thank you, Seb. So, what do
[16:15] you learn when you do something that has
[16:17] never been done before? In our case, it
[16:20] came in three takeaways. The first one
[16:23] which really talks to planning is
[16:25] measure twice but do not be afraid to
[16:27] cut. Seb talked about the planning that
[16:30] went into selecting a cable
[16:32] manufacturing through the Kaizen. Talked
[16:35] about the trials that we did for fiber
[16:37] placement. We talked about engineering
[16:39] the routes so that we would know where
[16:41] to go build. All of that was required
[16:43] because again it's a first of its kind.
[16:45] It's complex. You need to do a lot of
[16:47] planning. But what we learned along the
[16:49] way is that it was equally important to
[16:51] get started right and fully
[16:54] understanding that things might not go
[16:55] according to plan. So we started placing
[16:57] cable we hit some roadblocks as said
[17:00] talked about and then we adjust it and
[17:02] that's really the learning measure
[17:03] measure measure but don't be afraid to
[17:05] cut and get started. The important thing
[17:07] is really to be adaptable and flexible
[17:09] according to what you observe in the
[17:11] field. So that's the first takeaway. The
[17:14] second takeaway which might be
[17:15] surprising is that at the end of the day
[17:17] AI is about human connections. Right? So
[17:20] this project required a lot of
[17:22] coordination, a lot of engagement with
[17:24] multiple stakeholders, a data center
[17:26] teams who was working on that was
[17:28] working on acquiring the site. The other
[17:31] part of our data center team that was
[17:32] working on construction of the site,
[17:35] some of the other hyperscalers who are
[17:37] actually having doing construction in
[17:39] the area. the permitting authority so we
[17:42] can resolve conflict in our design. That
[17:44] required constant constant engagement
[17:47] and collaboration and that was
[17:50] specifically import particularly
[17:51] important when conflicts arose whether
[17:53] it's about resolving schedule issues or
[17:56] you know repairing something that had
[17:58] been de damaged. What we learned is that
[18:00] ultimately it came down to people talk,
[18:03] human beings talking to each other and
[18:06] really collaborating to resolve
[18:07] problems. And again, at the end of the
[18:09] day, it's about human connections.
[18:11] Solving problem is about human
[18:12] connections. AI is about human
[18:15] connections and people engaging with
[18:17] each other. And the third takeaway,
[18:20] which applies to this project, but
[18:21] definitely applies to everything we do
[18:22] at Meta, is about embracing new ways of
[18:26] working. In our case, we had to seek out
[18:29] new types of partnerships which frankly
[18:31] we don't typically do. The civil
[18:33] engineering firm that is very different
[18:36] from the traditional telecom
[18:37] infrastructure carriers that we work
[18:39] with that was required for the success
[18:41] of building that massive duck bank and
[18:44] building a partnership with the city on
[18:46] the wall so we can work collaboratively
[18:49] around permitting for the for these type
[18:51] of builds. that was required. That was a
[18:54] new way of working that we had to
[18:56] embrace again for the success of the
[18:57] project. Ultimately, all of these helped
[19:00] us be successful, helped us build this
[19:03] huge duck bank and deliver those fiber
[19:05] strengths. Those type of projects we
[19:07] expect to continue to happen in meta a
[19:10] meta. It might not be building a huge
[19:12] duck bank, but it will be something
[19:13] complex. So that's what meta does. Solve
[19:15] complex problems. If that's the type of
[19:18] project that appeals to you, let us
[19:20] know. With that, I thank you for your
[19:22] time.
