Dropbox Beats AWS on Storage Margins via Custom Rust Hardware

Ryan Peterman

Dropbox beat Amazon S3 unit economics by tailoring bare-metal drives to its exact sync workload, but surviving capacity crises required keeping AWS as an emergency overflow valve.

AI-assisted summary of AWS to Dropbox: The Largest Ever Data Migration In History | James Cowling — original source. Check the original for full context.

Source published: 2026-07-29

Key ideas

Dropbox migrated away from Amazon S3 to achieve strategic independence, control its destiny, and capture immense financial savings before its IPO. (0:00)
By adopting early shingled magnetic recording (SMR) hard drives and bypassing the operating system filesystem, Dropbox achieved better unit economics than AWS S3. (2:01)
Memory pressure on early Go-based storage nodes triggered Out-of-Memory events that risked cascading congestion collapse, prompting a rewrite in pre-1.0 Rust. (6:01)
Dropbox engineered 'Trampoline,' an automated safety valve that burst-dumped excess write traffic to Amazon S3 during capacity crunches while keeping user data durability absolute. (12:54)

Detailed notes

## Background Published on YouTube by Ryan Peterman on July 29, 2026, this video features a clip from a wider conversation on Peterman's podcast with James Cowling, former top senior engineer at Dropbox and current CTO at Convex. According to the show description, the discussion focuses on technical details behind Dropbox's landmark infrastructure projects alongside career insights from scaling massive distributed systems.

## TL;DR James Cowling details the engineering and business strategy behind Dropbox's historic migration away from Amazon S3 onto its bespoke storage engine, Magic Pocket. To beat AWS's unit economics prior to going public, Dropbox deeply optimized hardware spend, harnessed shingled magnetic recording (SMR) disks, and bypassed standard operating system layers. Addressing critical operational bottlenecks, the team moved language runtimes from Python and Go to pre-1.0 Rust to prevent memory-induced congestion collapse. Faced with physical bare-metal realities—including ruined delivery trucks that erased weeks of capacity—Dropbox instituted rigorous Failure Mode and Effects Analysis (FMEA) alongside 'Trampoline,' an architectural safety valve routing burst writes back to S3 to guarantee absolute data durability.

## Atmosphere & style The conversation carries an authentic, insider tone characterized by technical depth, pragmatic candor, and shared infrastructure camaraderie. Cowling balances retrospective humor with the gravitas of managing mission-critical bare-metal hardware, openly recalling the stress of 24/7 on-call firefights and sudden capacity crises. Listening in feels like sitting beside a veteran distributed systems architect reflecting without corporate spin on the high-stakes trade-offs of building at world-class scale.

## Bulletin - [00:00](https://www.youtube.com/watch?v=Ar84Ow4l2XE&t=0s) Dropbox migrated away from Amazon S3 to achieve strategic independence, control its destiny, and capture immense financial savings before its IPO. - [02:01](https://www.youtube.com/watch?v=Ar84Ow4l2XE&t=121s) By adopting early shingled magnetic recording (SMR) hard drives and bypassing the operating system filesystem, Dropbox achieved better unit economics than AWS S3. - [06:01](https://www.youtube.com/watch?v=Ar84Ow4l2XE&t=361s) Memory pressure on early Go-based storage nodes triggered Out-of-Memory events that risked cascading congestion collapse, prompting a rewrite in pre-1.0 Rust. - [12:54](https://www.youtube.com/watch?v=Ar84Ow4l2XE&t=774s) Dropbox engineered 'Trampoline,' an automated safety valve that burst-dumped excess write traffic to Amazon S3 during capacity crunches while keeping user data durability absolute.

## Key takeaways - [00:00](https://www.youtube.com/watch?v=Ar84Ow4l2XE&t=0s) James Cowling discussed replacing S3 with Dropbox founder Drew Houston as early as 2010 before joining the company in 2012. - [00:00](https://www.youtube.com/watch?v=Ar84Ow4l2XE&t=0s) Building a custom storage engine acted as a recruitment flywheel, attracting top-tier engineering talent who later redesigned client sync protocols and distributed databases. - [02:01](https://www.youtube.com/watch?v=Ar84Ow4l2XE&t=121s) While Amazon offered deeply discounted pricing due to Dropbox's massive scale, S3's general-purpose multi-tenant architecture could not match the bespoke efficiency of Dropbox's internal stack. - [03:27](https://www.youtube.com/watch?v=Ar84Ow4l2XE&t=207s) Dropbox separated storage into an access-efficient temporary cluster for fast reads and writes, paired with a massive cold storage cluster that took batch writes outside the synchronous path. - [03:27](https://www.youtube.com/watch?v=Ar84Ow4l2XE&t=207s) Hardware teams measured power down to rack circuit breakers, packing maximum drive density within strict amp budgets. - [06:01](https://www.youtube.com/watch?v=Ar84Ow4l2XE&t=361s) An initial storage prototype written in Python lacked the memory controls and concurrency performance required for production storage. - [06:01](https://www.youtube.com/watch?v=Ar84Ow4l2XE&t=361s) An early Go implementation suffered from unpredictable garbage collection pauses and memory usage on disk-heavy bare metal nodes. - [06:01](https://www.youtube.com/watch?v=Ar84Ow4l2XE&t=361s) Storage node OOM restarts were misread by cluster coordinators as dead disk drives, causing a sevenfold data reconstruction amplification that threatened cascading congestion collapse. - [05:58](https://www.youtube.com/watch?v=Ar84Ow4l2XE&t=358s) Project DiscoTech utilized low-level drive interfaces like Zone-Based Commands (ZBC) via draft specifications from manufacturers to interact directly with raw disk platters. - [05:58](https://www.youtube.com/watch?v=Ar84Ow4l2XE&t=358s) The DiscoTech architecture rebalanced capital expenditures so that the vast majority of hardware spending went toward raw disks rather than RAM, compute, network, or power infrastructure. - [10:16](https://www.youtube.com/watch?v=Ar84Ow4l2XE&t=616s) Managing physical on-premises hardware required manual human labor to swap out memory sticks, unlike elastic cloud environments where resources can be provisioned in seconds. - [10:16](https://www.youtube.com/watch?v=Ar84Ow4l2XE&t=616s) Two separate delivery trucks carrying racks of server hardware crashed in the same week, wiping out planned capacity expansions for six weeks during a critical period. - [11:44](https://www.youtube.com/watch?v=Ar84Ow4l2XE&t=704s) FMEA (Failure Mode and Effects Analysis) pre-mortems were used to map every conceivable failure mode and engineer software-level mitigations against catastrophic risks. - [12:54](https://www.youtube.com/watch?v=Ar84Ow4l2XE&t=774s) Dropbox maintained an absolute, non-negotiable standard for user data durability and refused to lower replication factors during high-stress traffic events. - [12:54](https://www.youtube.com/watch?v=Ar84Ow4l2XE&t=774s) The Trampoline system routed overflow writes directly to Amazon S3 when internal bare-metal clusters neared maximum capacity, enabling high utilization without risking data loss.

## Detailed walkthrough ### [00:00](https://www.youtube.com/watch?v=Ar84Ow4l2XE&t=0s) The Strategic and Cultural Motivations for Leaving S3

Ryan Peterman opens by asking about one of the most prominent projects James Cowling worked on at Dropbox: the large-scale migration away from Amazon S3. Cowling notes that the desire to leave AWS had existed within the company long before he arrived in 2012; he had even discussed it with Dropbox founder Drew Houston around 2010. Strategically, Dropbox wanted to "control the destiny of the company." At that time, before it repositioned itself around broader team collaboration, Dropbox's core market was strictly file sync and share, making full ownership of the underlying storage engine essential to its business model.

Beyond strategy, building an in-house storage engine unlocked massive financial efficiencies prior to Dropbox's IPO. Cowling stresses that taking on an engineering challenge of this magnitude pays cultural dividends across an organization by drawing top-tier talent who thrive on hard problems. Once the storage foundation stabilized, those same senior engineers cycled into other complex layers of the company: teammate Jamie redesigned the desktop client sync protocol, while Cowling moved on to distributed databases and metadata file systems. However, Cowling cautions that such infrastructure comes with heavy ongoing obligations, noting:

> “You know, it's like having a baby and then you have to—you've got to raise the baby. You can't just build a system like this and be that's it, we're done.”

- Cowling spoke with Drew Houston about replacing S3 as early as 2010 and joined Dropbox in 2012. - The project delivered immense cost reductions that proved critical in setting up the company's financial profile before going public. - Complex engineering challenges functioned as a recruitment flywheel, attracting engineers who solved foundational storage and then went on to overhaul client sync protocols and distributed databases.

---

[02:01](https://www.youtube.com/watch?v=Ar84Ow4l2XE&t=121s) Outperforming AWS with SMR Disks and Workload Tuning

When Peterman asks whether AWS tried to negotiate and retain Dropbox's business with steep discounts, Cowling admits with a laugh that he is finally free to speak on it. For a long period, Amazon was unaware of Dropbox's plans, but data center industry chatter picked up once Dropbox began purchasing massive amounts of physical floor space. Dropbox was already securing extremely favorable, deeply discounted contract pricing far below retail sticker prices on S3 due to their scale. Despite that, Amazon could not match Dropbox's target unit economics because the bespoke internal infrastructure actually proved more efficient than S3 itself.

Dropbox gained an efficiency advantage because S3 must remain a general-purpose product designed for arbitrary multi-tenant traffic, whereas Dropbox could tailor hardware and software to its exact usage profiles. Dropbox became an early pioneer in deploying shingled magnetic recording (SMR) hard drives at massive scale. Even so, Cowling offers a strong note of caution for modern startups:

> “I wouldn't recommend another company do this right now. Uh but I think at the time it it certainly made sense for us as a company.”

```mermaid flowchart TD accTitle: Generic Cloud Storage vs Tailored In-House Architecture accDescr: Comparison showing S3 serving broad arbitrary workloads versus Dropbox's tailored hardware and software stack.

subgraph AWS ["Amazon S3 General Architecture"] A["Arbitrary Multi-Tenant Workloads"] --> B["Standard Drive Provisioning"] B --> C["Compromised General-Purpose Trade-offs"] end

subgraph DBX ["Dropbox Bespoke Storage Engine"] D["Specific File Sync Workloads"] --> E["SMR High-Density Disks"] E --> F["Hardware & Power Optimized Per Rack"] end ```

---

[03:27](https://www.youtube.com/watch?v=Ar84Ow4l2XE&t=207s) Exploiting Access Patterns and Physical Rack Constraints

Cowling breaks down how deep visibility into internal traffic allowed Dropbox to optimize where and how data lived. When users upload files to Dropbox, access patterns are highly predictable: a burst of immediate activity follows the initial write—either from generating image previews or immediate sharing—followed by an exponential decay in access frequency. Dropbox leveraged this decay curve by splitting storage into two tiers: an access-efficient temporary cluster that traded storage density for fast, cheap reads and writes, paired with a massive, colder secondary storage system. The colder cluster took bulk writes in the background using dense, highly efficient layout algorithms; because it lived outside the synchronous write path, an outage on that cluster did not impact user-facing uploads.

These micro-optimizations extended all the way to physical hardware and electrical budgeting. Engineering teams measured power down to the rack's power distribution unit (PDU) circuit breakers, attempting to pack maximum drive density into a specific amp budget. Cowling recalls that during periods where hardware batches experienced higher-than-average drive failure rates, the resulting background re-replication traffic pushed drives so hard that data center operators warned him the racks were running dangerously hot.

- Early file lifecycle: rapid initial read burst for sync and previews, followed by steady cold-tier decay. - Two-tiered storage design: an ingress buffer prioritized for read/write performance feeding an ultra-dense, batch-written cold store off the critical upload path. - Hardware-level management: tuning disk counts directly to the amp limits of top-of-rack PDUs across a multi-exabyte footprint comprising millions of spinning hard drives.

---

[06:01](https://www.youtube.com/watch?v=Ar84Ow4l2XE&t=361s) Language Evolution and the Threat of Congestion Collapse

Peterman brings up Dropbox's migration across programming languages, noting the eventual adoption of Rust over memory-related issues. Cowling reveals that the very first prototype was actually written in Python. While Python held up surprisingly well for pure I/O-bound operations, its weak memory controls, concurrency limitations, and refactoring friction disqualified it for production storage where correctness was paramount. Dropbox then rewrote the storage tier in Go well before Go reached 1.0 general availability (GA). Go served well for networking proxies and concurrent server handling, but on disk-heavy bare metal nodes, Go’s runtime and garbage collection introduced unacceptable memory unpredictability.

At massive scale—tracking hardware cost down to individual pie slices for sheet metal and screws—nodes constantly ran on tight memory margins. If a storage node experienced an Out of Memory (OOM) event and restarted, the cluster orchestrator interpreted the missing node as a hard drive failure and initiated data re-replication. Cowling recalls a high-traffic incident where an album release (such as De La Soul) saturated nodes hosting the popular blocks, causing them to OOM under load. Attempting to rebuild the seemingly lost replicas from surviving nodes turned a single incoming traffic spike into a sevenfold data reconstruction avalanche, threatening the system with cascading congestion collapse. To gain exact control over memory and eliminate this failure mode, Dropbox chose Magic Pocket and rebuilt the core storage node layer in pre-1.0 Rust, pairing the rewrite with custom disk-level primitives that bypassed the operating system filesystem altogether. Cowling highlights the fragility of large systems under stress:

> “Congestion collapse is when workload to a system crosses a threshold where it all kind of collapses. So can you design against congestion collapse is really the hardest part or one of the hardest parts of Magic Pocket, which is the name of the storage system.”

- Language progression: started with an initial Python prototype, moved to early pre-GA Go for the initial storage engine, and ultimately migrated node software to pre-GA Rust. - OOM cascading failure loop: heavy user traffic triggered memory spikes on Go nodes $\rightarrow$ nodes restarted $\rightarrow$ the coordinator treated restarts as dead drives $\rightarrow$ Reed-Solomon or replica reconstructions turned one stream of requests into a 7x load amplification across peer nodes. - Direct disk access: the move to Rust coincided with completely discarding traditional Linux filesystems on the data drives to speak directly to raw drive heads.

[09:28](https://www.youtube.com/watch?v=Ar84Ow4l2XE&t=568s) Project DiscoTech and Optimizing Hardware Spend

To squeeze maximum efficiency out of their custom storage hardware, the team adopted low-level drive interfaces like ZBC (Zone-Based Commands), which allowed them to bypass standard drive abstractions and communicate directly with the disk platters. Dropbox partnered closely with disk drive manufacturers, who provided draft specifications for cutting-edge drive models before they hit the general market. Operating directly against those draft specs, Dropbox wrote software to control the drives directly. This entire initiative was packaged under the internal project name "DiscoTech," short for the Disk Technology project.

The architectural overhaul solved stability problems by ending cascading congestion collapse and radically improving fleet reliability. Crucially, it completely restructured Dropbox's capital expenditure pie chart. Previously, non-storage overhead consumed huge portions of the budget, but DiscoTech rebalanced the ledger so capital went almost entirely toward raw disk media rather than secondary supporting components. As James Cowling explained the financial objective: “And what we wanted to do is get that pie chart to be almost all the money spent on disks and as little money spent on RAM, on compute, on network, on power, on sheet metal, etc.”

- **ZBC (Zone-Based Commands):** Dropbox relied on early instruction sets and draft specifications supplied directly by disk manufacturers to exert low-level control over drive mechanics. - **Project DiscoTech:** The overarching disk technology initiative that stabilized the fleet and eliminated congestion collapse. - **Hardware cost shift:** The goal was to maximize the share of hardware spend going into storage density (disks) while minimizing expenditures on compute, RAM, networking gear, facility power, and server sheet metal.

---

[10:16](https://www.youtube.com/watch?v=Ar84Ow4l2XE&t=616s) The Reality of Physical Hardware and the Two-Truck Crash

When Ryan Peterman asks whether the catastrophic cascading failure prompted a postmortem, Cowling laughs, noting that no formal postmortem was needed because the team felt the failure in real time while being paged around the clock. The episode serves as a quintessential argument in favor of the public cloud. In cloud environments, resizing an instance or allocating RAM takes seconds; with custom on-premises bare metal, hardware constraints are physical and unforgiving. As Cowling reflected: “Imagine you've spent hundreds of millions of dollars on a on a storage system and it's out in production and it's on physical hardware.” If your fleet runs low on RAM, fixing it requires paying technicians to manually unrack chassis and physically swap out memory sticks across thousands of servers.

These physical constraints became starkly evident during one of the most stressful operational episodes on Magic Pocket. In a single week, by bizarre coincidence, two separate delivery trucks transporting full racks of Dropbox server hardware crashed en route. While the drivers were fortunately uninjured, the payload was wrecked, abruptly wiping out anticipated capacity expansions not just for two weeks, but for roughly six weeks. Unlike a personal laptop running out of disk space, this capacity squeeze affected a million production drives. Customers could not be turned away, nor could their files be deleted, forcing the infrastructure team into high-stakes capacity maneuvering to keep the platform alive.

- **On-premises vs. cloud elasticity:** Physical fleets require physical human labor for every hardware upgrade; you cannot click a button to provision RAM across hundreds of millions of dollars of hardware. - **The logistics disaster:** Two separate transit trucks carrying racks of servers crashed in the same week, destroying shipments and causing a six-week deficit in planned server capacity. - **Customer invariants:** Production capacity had to be maintained under all circumstances because dropping customer writes or pruning customer data was entirely out of the question.

---

[11:44](https://www.youtube.com/watch?v=Ar84Ow4l2XE&t=704s) FMEA Threat Modeling and the Thrill of the Firefight

Navigating supply chain shocks and capacity crunches forced the team to build comprehensive defensive buffers into their infrastructure planning. Systems had to be architected from first principles to prevent memory spikes and make congestion collapse mathematically impossible. To systematically evaluate catastrophic edge cases, Dropbox instituted a disciplined engineering exercise known as FMEA (Failure Mode and Effects Analysis).

The FMEA process functioned as an exhaustive pre-mortem. Engineers maintained an expansive spreadsheet listing every imaginable failure mode, rating the severity of each scenario from localized hardware faults all the way to catastrophic existential threats, such as a full-scale data center fire that could result in human casualties. Identifying every vulnerability in advance allowed the engineering team to design automated mitigations directly into the platform. While Cowling admits that decades of on-call duty have curbed any fondness for midnight pages, he embraces the sheer technical challenge of bare-metal crises: “I can't say I like getting paged because I've spent my whole life on call, but I do like that rubber hits the road stuff.”

- **FMEA (Failure Mode and Effects Analysis):** Structured pre-mortem threat modeling where teams map every possible failure, assess existential and physical hazards, and design systemic software defenses around them. - **Engineered headroom:** Incorporating strict buffer margins to safeguard the platform against black swan supply chain disruptions and traffic surges. - **The on-call mindset:** Solving novel, high-stakes infrastructure disasters where downstream dependencies run out and no external vendor can step in to rescue you.

---

[12:59](https://www.youtube.com/watch?v=Ar84Ow4l2XE&t=779s) Non-Negotiable Durability and the Trampoline Escape Hatch to S3

Peterman draws a parallel to his time working on infrastructure at Instagram, where teams implemented configurable "DEFCON knobs" to gracefully degrade performance during emergencies—such as temporarily dropping storage replication factors and accepting heightened data-loss risk to keep services upright. Peterman asked: “When I worked on infrastructure at Instagram, we had this concept of DEFCON knobs, which are basically these configs that you could flip that would gracefully degrade your system where you can still operate it, but you know, maybe in the case of Dropbox, you store less replicas.”

Cowling makes it clear that Dropbox took an entirely different stance toward durability: “So, I mean, we just had an just absolute zero non-negotiable like there was no room for negotiation on on users' durability.” While Dropbox did use operational knobs to shed background CPU tasks, throttle memory-heavy jobs, or disable synthetic test traffic during surges, user data safety could never be compromised.

To solve severe capacity shortfalls without degrading durability, Dropbox created a hybrid safety mechanism called "Trampoline." Because Dropbox retained its network connections to AWS, the team configured Magic Pocket to use Amazon S3 as an automated pressure-relief valve. If bare-metal capacity neared a dangerous threshold, Trampoline automatically routed write streams over to S3. This escape hatch allowed Dropbox to run its internal data centers far closer to 100% disk utilization, confident that they could burst-dump 30 petabytes onto S3 during an emergency and seamlessly migrate it back to internal bare-metal drives once new hardware racks arrived.

```mermaid flowchart TD accTitle: Trampoline Storage Architecture accDescr: Diagram illustrating how Magic Pocket routes data locally until capacity thresholds trigger Trampoline to overflow to AWS S3. subgraph Ingest["Client Ingestion"] Writes["Incoming User Writes"] end subgraph LocalDC["Dropbox On-Premises Infrastructure"] MagicPocket["Magic Pocket Engine"] DiscoTech["DiscoTech Disk Fleet"] end subgraph PublicCloud["AWS Cloud Fallback"] S3["Amazon S3 Storage"] end

- **Durability as an absolute invariant:** Dropbox refused to lower replication counts or compromise data durability under any system stress level. - **Operational load shedding:** Non-critical background processes, memory-intensive routines, and synthetic test suites were throttled via knobs during load spikes. - **Project Trampoline:** An automated safety valve that overflowed emergency write volumes (such as 30 PB bursts) to AWS S3 when local disks neared capacity limits. - **Capital efficiency dividend:** Treating S3 as an elastic overflow tank enabled Dropbox to push internal drive utilization significantly closer to maximum thresholds during standard day-to-day operations.

## Mentioned Amazon S3, Amazon Web Services (AWS), Convex, De La Soul, DiscoTech, Drew Houston, Dropbox, Failure Mode and Effects Analysis (FMEA), Go, Instagram, James Cowling, Magic Pocket, Python, Reed-Solomon, Rust, Ryan Peterman, Shingled Magnetic Recording (SMR), Trampoline, Zone-Based Commands (ZBC)

## Video description This is a clip from a longer conversation with James Cowling who is the CTO at Convex and was previously the most senior engineer at Dropbox. We discussed technical details of his past projects and career advice given where AI is today.

Where to watch: • YouTube - https://youtu.be/3XkmNSuHFmY • Spotify - https://open.spotify.com/episode/1Xt2SfdlM8huFQERnsknqP?si=qU-6R9T4QTiwYlpMeppmzg • Apple Podcasts - https://podcasts.apple.com/us/podcast/the-peterman-pod/id1777363835 • Transcript - https://www.developing.dev/p/dropboxs-former-most-senior-eng-building

[Source](https://www.youtube.com/watch?v=Ar84Ow4l2XE)

中文简报

Dropbox通过自研Rust硬件在存储利润率上超越AWS Dropbox通过针对其具体的同步工作负载定制裸金属硬盘，在单位经济效益上超越了Amazon S3，但要在容量危机中幸存，仍需保留AWS作为紧急分流阀。 James Cowling详细阐述了Dropbox从Amazon S3历史性迁移至其定制存储引擎Magic Pocket背后的工程与商业策略。为了在上市前超越AWS的单位经济效益，Dropbox深度优化了硬件支出，利用了叠瓦式磁记录（SMR）磁盘，并绕过了标准操作系统层。为解决关键的运维瓶颈，团队将语言运行时从Python和Go迁移至1.0版本之前的Rust，以防止内存引发的拥塞崩溃。面对裸金属的物理现实问题——包括因货运卡车事故导致数周容量储备受损——Dropbox引入了严格的失效模式与影响分析（FMEA），并开发了架构安全阀“Trampoline”，将突发写入流量路由回S3，以确保绝对的数据持久性。 Dropbox迁出Amazon S3是为了实现战略自主、掌握自身命运，并在IPO前斩获巨额财务节省。 (0:00) 通过采用早期的叠瓦式磁记录（SMR）硬盘并绕过操作系统文件系统，Dropbox实现了优于AWS S3的单位经济效益。 (2:01) 早期基于Go语言的存储节点面临内存压力，引发内存溢出（OOM）事件并面临级联拥塞崩溃的风险，促使团队使用1.0版本之前的Rust进行了重写。 (6:01) Dropbox设计了名为“Trampoline”的自动化安全阀，在容量告急期间将突发的超额写入流量倾倒至Amazon S3，同时绝对保障用户数据的持久性。 (12:54)

Reading summary

James Cowling details how Dropbox migrated away from Amazon S3 to build its custom storage engine, Magic Pocket, achieving cost efficiencies prior to its IPO while navigating bare-metal hardware and reliability challenges. Strategic Incentives and Bespoke Workload Economics (0:00) Dropbox had considered moving off Amazon S3 as early as 2010 to control its core file sync and share destiny, executing the shift prior to its IPO. The company obtained unit economics that AWS could not match, even though Dropbox already had heavily discounted contracts below retail sticker rates. Because S3 had to cater to arbitrary multi-tenant workloads, Dropbox could extract far higher efficiencies by engineering hardware and software tailored directly to its specific traffic patterns. However, Cowling notes he would not recommend modern startups replicate this migration today. Storage Tiering and Low-Level Hardware Optimization (3:27) To exploit user access patterns—a burst of read activity followed by an exponential decay into cold data—Dropbox implemented a two-tiered architecture. An ingress buffer managed initial high-throughput reads and writes, while a dense, colder secondary cluster processed bulk writes in the background off the synchronous upload path, preventing user-facing outages if the cold tier failed. Under Project DiscoTech, the team bypassed operating system filesystems and standard drive layers, using Zone-Based Commands (ZBC) based on draft specifications from drive manufacturers to communicate directly with shingled magnetic recording (SMR) drive heads. Infrastructure teams balanced hardware counts to the amp limits of top-of-rack power distribution units (PDUs), realigning capital expenditure so budgets funded raw storage media rather than secondary components like RAM, networking, compute, or sheet metal. Eliminating Memory Cascades via Rust and FMEA (6:01) Dropbox iterated its storage engine through three languages, beginning with Python and then moving to pre-GA Go. Go functioned well for network proxies, but its runtime and garbage collector created memory unpredictability on bare-metal storage nodes operating with tight margins. A high-demand event—such as a De La Soul album release—caused nodes to run out of memory (OOM) and restart. Because the cluster coordinator interpreted restarts as disk failures, it initiated rebuilds that amplified traffic sevenfold, creating a risk of cascading congestion collapse. To establish strict memory control, Dropbox rewrote the storage nodes in pre-1.0 Rust. To systematically safeguard the fleet, the team employed Failure Mode and Effects Analysis (FMEA), mapping and modeling everything from drive faults to catastrophic data center fires to build automated software defenses directly into the platform. Physical Constraints and the Trampoline Safety Valve (10:16) Unlike public cloud infrastructure where resources scale on demand, bare-metal operations face rigid physical constraints. Cowling describes an incident where two separate transit trucks carrying server racks crashed in the same week, destroying shipments and causing a six-week deficit in planned capacity expansion across a footprint of a million drives. Because Dropbox treated data durability as non-negotiable—refusing to drop replication factors even during emergencies—the team created "Trampoline." This mechanism used existing network links to AWS S3 as an automated overflow escape hatch. If local bare-metal capacity approached dangerous limits, Magic Pocket routed excess write traffic (such as 30-petabyte bursts) to S3, later draining it back to local disks once new hardware was racked. This buffer enabled Dropbox to safely run internal data center utilization closer to maximum capacity. While the migration yielded structural financial advantages and built engineering talent that drove company-wide innovations, operating custom bare-metal infrastructure introduced continuous operational responsibilities and severe supply chain risks that demand dedicated software escape hatches.

精炼阅读摘要

前 Dropbox 工程师 James Cowling 阐述了公司将其数艾字节（EB）级数据从 Amazon S3 迁往自研存储系统 Magic Pocket 的深层逻辑；他指出，针对单一文件同步负载的高度定制化软硬件使 Dropbox 获得了公共云无法比拟的单位经济优势，但也使团队背负了极具挑战性的物理约束与长期系统运维责任。战略自主与定制化存储的单位经济学 (0:00) Cowling 透露，早在 2010 年他便与创始人 Drew Houston 探讨过摆脱 S3 的构想，并于 2012 年加入公司推进这一计划。当时 Dropbox 的核心业务完全集中在文件同步与共享上，自主掌控底层存储系统对于在首次公开募股（IPO）前确立财务健康度和掌握商业命运至关重要。尽管由于规模庞大，Dropbox 当时已拿到远低于 S3 公开标价的合同折扣，但 AWS 作为通用型多租户平台，无法针对特定工作负载实施极限优化，其单位成本依然无法满足 Dropbox 的财务预期。为重构资本支出结构，Dropbox 发起了代号为 DiscoTech 的磁盘技术项目，在大规模裸机环境中率先采用叠瓦式磁记录（SMR）硬盘，并基于硬件厂商的早期草案规范编写底层软件，使用分区基础命令（ZBC）直接操控磁盘盘片。该项目彻底改变了采购成本的构成比例：团队将绝大多数硬件开销集中在纯磁盘介质上，最大程度压缩了内存、计算、网络设备、配电以及机箱钣金和螺丝等次要配套资产的支出占比。访问衰减分层与机柜级物理极限控制 (3:27) 通过对内部流量模式的深入分析，Dropbox 发现用户上传的数据呈现出高度一致的生命周期规律：在刚写入时由于生成图片缩略图和初期共享，会经历短暂的高频读取高峰，随后访问频率迅速呈指数级衰减。工程团队利用这一特性构建了双层架构：前端设立专注于读写吞吐的临时接入缓冲集群，后端对接以高密度算法离线批量写入的二级冷存储集群。由于冷存储脱离了同步写入的关键路径，即便发生后台中断也不会影响用户的正常上传流程。这种微观优化直接延伸至物理数据中心设施层面。团队针对机架顶部配电单元（PDU）断路器的安培负荷极限精确计算硬盘装载密度，在涉及数百万块机械硬盘的庞大体系中榨取电力与空间上限。Cowling 回忆，当某些批次的硬件遭遇高于平均水平的故障率时，系统后台自动触发的数据重建流量会导致硬盘高负荷运转，机房运营人员甚至曾发出机架发热量过高、面临危险物理过热的警告。编程语言演进与根除拥塞崩溃雪崩 (6:01) Magic Pocket 核心存储节点的软件实现经历了从 Python、Go 到 Rust 的重大演进。最初的 Python 原型虽然在纯 I/O 操作中表现良好，但其弱内存控制与重构阻力难以胜任生产环境；团队随后在 Go 达到 1.0 正式版之前便重构了存储层，虽然 Go 擅长网络代理，但其运行时的垃圾回收机制在裸机紧凑的内存环境中带来了致命的不确定性。在一次音乐专辑（如 De La Soul）发布引发的超高并发流量中，承载热门数据块的 Go 节点因内存溢出（OOM）相继重启。由于集群协调调度器将节点 OOM 重启错误判定为物理硬盘损坏，系统随即启动跨节点冗余重建。单次外部流量洪峰瞬间被放大了 7 倍的内部重建负载，导致更多健康节点因资源耗尽而接连崩溃，使整个系统陷入灾难性的“拥塞崩溃”。为了杜绝内存暴涨并实现绝对的系统稳定性，Dropbox 最终在 Rust 达到 1.0 前再次重写了存储节点层，同时抛弃了传统 Linux 操作系统文件系统，直接与裸盘磁头交互以获得精确的底层控制权。物理供应链冲击与S3跳板应急机制 (12:59) 与公有云秒级调整规格不同，私有裸机架构面临着残酷的物理与物流现实。Cowling 讲述了一次严峻的容量危机：在短短一周内，两辆满载 Dropbox 服务器机架的运输卡车相继发生车祸，导致既定规划被直接延误了约六周。在数百万块硬盘组成的生产环境中，面对不可预测的物理供应链黑天鹅事件，团队既不能停止接收用户文件，也无法点击按钮直接在线扩充内存，只能通过失效模式与影响分析（FMEA）系统性推演从局部硬件损坏到数据中心火灾等各类极端灾害。在应对极端压力时，Dropbox 明确将用户数据的持久性列为不可违背的绝对底线，拒绝采用类似降低副本数等损害数据安全性的降级旋钮。为了在不牺牲持久性的前提下渡过容量危机，团队开发了名为 Trampoline（跳板）的安全机制：当本地裸机容量逼近临界危险阈值时，系统会自动将写入流量无缝溢流并倾倒至 AWS S3（单次应急规模可达 30 PB），待后续硬件机架到货上架后再在后台回迁。这一设计不仅为突发事件提供了减压阀，还使 Dropbox 能够将日常自建机房的磁盘利用率极限推高至接近 100%。 Cowling 强调，尽管 Magic Pocket 为 Dropbox 带来了极佳的单位经济效益并成为吸引顶尖工程人才的文化飞轮，但构建并长期抚育这样一套复杂的裸机存储系统需要付出高昂的组织与运维代价。基于当今的技术与商业环境，他明确表示不建议现代创业公司模仿这种自建底层存储架构的做法。

Original transcript

[0:00] one of the big projects you worked on at [0:01] Dropbox was my migrating away from S3. [0:04] >> Yes. Yes. [0:05] >> So, um you know, why why did Dropbox [0:08] migrate away from S3? [0:10] >> Yeah, that was a desire of the company [0:11] for a long time. You know, from even [0:13] before I was there. So, I started at [0:15] Dropbox in 20 uh 2012. [0:18] Um and I spoke to to Drew, the Dropbox [0:20] founder, I think in 2010 about this [0:22] project. And so, I think there was a [0:24] desire to um control the destiny of the [0:28] company from a strategic perspective. at [0:30] the time, it was before Dropbox kind of [0:32] reshaped itself as being more about [0:34] collaboration, you know, but at the time [0:36] it was a file sync and share category. [0:37] That was that was the that was the that [0:39] was the market sector, you know? And so, [0:41] and owning the file system was really [0:43] valuable to the company. [0:45] Ultimately, we saved a huge amount of [0:46] money. I mean, we we And this is before [0:49] the public company went public. We [0:51] really drove massive cost efficiencies [0:54] through the project. Um but it was hard, [0:56] you know, in a way that I think it'd be [0:58] very difficult to to emulate without a [1:00] huge investment. [1:01] And I do think that there is um [1:05] there is a benefit to an organization [1:07] from having hard problems to solve. [1:10] Because if you have a a company with [1:12] extremely hard technical challenges, you [1:14] can [1:15] attract engineers who like working on [1:17] those hard technical problems. And when [1:19] they've solved those problems, they [1:21] cycle off and work on different parts of [1:23] the system. So, you know, after we all [1:25] worked we had a such a great team. It [1:27] was a very very small engineering team. [1:29] And after we kind of shipped the the [1:31] storage system reliably, we all went off [1:33] and, you know, Jamie went and and [1:36] redesigned the sync protocol, the [1:38] desktop client, and I worked on the on [1:40] the file system and the distributed [1:41] databases. And so, yeah, there's there's [1:43] value to a business to have um that [1:46] level of technical investment, but it is [1:48] a [1:49] you know, it's like it's like having a [1:50] baby and then you have to you've got to [1:53] raise the baby. You can't just build a [1:54] system like this and be that's it, we're [1:56] done. [1:57] You own it and you have to keep [1:59] investing in it. [2:01] >> Did um [2:03] did S3 do any counter negotiation before [2:05] you set out to leave them? Did they say [2:07] like, "Oh, you know, we'll cut you a [2:08] deal if you stay." [2:09] >> [laughter] [2:11] >> I guess I'm allowed to talk about this [2:12] now. It was a long time ago. Um [2:15] yeah, for the longest time there I don't [2:16] think they were they were particularly [2:17] aware that this was happening. [2:19] Um but you know, the the data center [2:21] folks talk and uh certainly it was [2:25] noticed that Dropbox was buying up a lot [2:27] of data center space. So, yeah, we had [2:29] we had um obviously at at the scales [2:32] that we were at, I mean, we were [2:34] negotiating very good rates with Amazon. [2:36] You know, we weren't paying sticker [2:37] price, we're paying very very very good [2:39] discounted rates. Um but yeah, at a [2:42] certain point they weren't able to meet [2:43] our cost efficiency because it it's when [2:45] we launched the system, it really was [2:48] more efficient than S3. [2:50] And that's for a variety of reasons. One [2:51] was that we were using kind of new [2:53] experimental disks called shingled [2:55] magnetic recording. We were the first [2:57] ones I think to use these disks at [2:58] scale. [2:59] Um and two, we had a very tight [3:01] understanding of our workloads. So, we [3:03] were able to design the system [3:04] specifically optimized for our [3:06] workloads, whereas S3 has to design the [3:08] system for everybody. So, you know, [3:11] it's it got to the point where, you [3:12] know, Amazon would not have been able to [3:13] offer us a more competitive deal because [3:16] we had a more efficient system. [3:18] Um [3:20] I wouldn't recommend another company do [3:21] this right now. [3:23] Uh but I think at the time it it [3:24] certainly made sense for us as a [3:25] company. [3:27] >> Can you give an example of a tight [3:29] understanding of your workloads leading [3:31] to like something you could do that S3 [3:34] couldn't? [3:35] >> Yeah, absolutely. So, for example, I [3:36] know [3:38] that um well, I'll try not to leak any [3:40] confidential data, right? But when you [3:42] when you upload a file to Dropbox, there [3:44] is a pattern of access, right? Um so, [3:48] typically people um access the file very [3:50] quickly shortly afterwards um because [3:53] you're sharing it with someone, or maybe [3:55] your Dropbox is processing that file to [3:58] generate an image preview, and then it [3:59] decays at a certain rate. And so we [4:02] understand in general the average block [4:04] size, and we understand also the access [4:08] pattern. So we could do things like at a [4:10] certain point we had these two clusters. [4:14] One was One was [4:15] designed for kind of temporary storage. [4:19] That was kind of storage inefficient, [4:21] but access efficient. So it was very [4:24] cheap to read and write to, but it was [4:26] inefficient to store. And data would get [4:28] written to there first, and then in the [4:30] background it would get moved in bulk to [4:33] this colder storage system. [4:36] And this colder storage system was far [4:37] more static. [4:39] And so it was able to have kind of more [4:40] efficient algorithms and and and be [4:43] written to in bulk. And if it went down [4:45] for rights, that was no problems because [4:47] it wasn't in the live path. And so we're [4:49] able to trade off again, like systems is [4:51] all about trade-offs. So we're able to [4:53] trade off the the live data write path [4:55] from the long-term read path. [4:57] That's one of many examples where [5:00] knowing the size of your data, where [5:02] it's accessed from, how frequently it [5:03] gets accessed, how how long it takes to [5:05] delete that data, you can really tune a [5:08] system [5:09] to your workload. If you know, stuff [5:11] like looking at [5:13] um [5:14] even things down to knowing how much [5:16] power to put in a rack. You know, you [5:18] have a rack of hardware there's a power [5:21] distribution unit, a PDU, at the top of [5:23] that rack. It has a circuit breaker [5:24] which can handle a certain number of [5:26] amps. [5:27] We would have to figure out, you know, [5:29] how many amps are required for that rack [5:32] based on access patterns. And there were [5:34] times where we got it slightly wrong. [5:35] There was times where [5:37] we got to maybe a bad batch of hardware, [5:39] and disks were failing too frequently. [5:41] And as a result we were re-replicating [5:42] the data [5:44] more regularly, and I was getting you [5:45] know, messages from the data center team [5:46] saying, "Hey, [5:48] we're running the racks really hot right [5:49] now, you know?" [laughter] So, that's [5:51] the level of optimizations you can make [5:53] when you get to that scale. Um but [5:55] again, that's multi-exabyte scale, you [5:58] know, [5:59] million hard drive scale. [6:01] >> Yeah, when I was reading about this [6:02] migration, I saw somewhere in the [6:04] migration [6:05] you initially started with Go, and then [6:09] the racks or something that you're [6:10] running on the hardware itself was [6:12] ooming too much or [6:14] >> Yeah. [6:14] >> requesting too much memory, and then you [6:16] migrated to Rust. [6:18] >> So, initially, the prototype was in [6:20] Python, if you can believe that. [6:22] And to be fair, Python is actually [6:23] pretty efficient for IO. I think people [6:25] give Python a a bad rap for IO-bound [6:28] workloads. It's pretty good at IO. Um [6:31] but obviously, um not great for [6:33] concurrency, not great for memory [6:34] management, and um very hard to [6:36] refactor. [6:38] And it was critical the system was [6:39] correct. And so, we migrated everything [6:42] to Go, and we built most of the storage [6:44] system in Go. This was before Go was in [6:47] general availability. I think this is [6:48] before Go was GA. Um [6:50] and we built the system in Go. Go is a [6:52] great language for concurrency, a great [6:54] language for proxies. You know, it's a [6:56] really well designed for like servers [6:58] that moved out of one place to another [7:00] place. [7:01] At a certain point though, [7:03] we would have, you know, let's just pick [7:05] a number. Let's say a million. Let's say [7:06] we have a million nodes in the system. [7:08] And every node has um [7:12] some amount of memory, some amount of [7:13] disk, some amount of sheet metal in the [7:16] chassis, right? And we would itemize all [7:18] these things. You'd have a pie chart of [7:21] of how much money is spent on all the [7:23] things. And it would still have stuff [7:24] like, yeah, sheet metal and screws and [7:26] stuff. And so, you're trying to optimize [7:28] the storage, and a big problem for us [7:32] was the amount of memory um [7:34] these nodes were using. Not just the [7:36] memory they were using, but the [7:37] unpredictability of it. Uh with, you [7:40] know, with Go having a runtime and and [7:44] you know, in a storage system an out of [7:46] memory error is pretty bad. [7:48] Because if a node runs out of memory and [7:49] restarts, that looks like a disk [7:51] failure. [7:52] So that looks a lot like a disk has [7:54] failed and has to be re-replicated. And [7:56] so a batch of nodes OOMing [7:59] can lead to cascading failures [8:01] throughout the system. [8:02] Because maybe [8:04] um [8:05] I remember a time it was a band, it [8:07] might have been De La Soul. I can't [8:08] remember there was a band released an [8:09] album on Dropbox. [8:11] So there was a big spike in load [8:14] um to a to a [8:16] a few files. So it was very, very high [8:19] um bandwidth and it caused them the [8:21] machines to OOM. So that those machines [8:23] uh OOMed, those disks OOMed. Um and so [8:26] as a result, no problems. The system [8:28] went to try to recover that data from a [8:30] whole bunch of other replicas, right? [8:33] Now all of a sudden you've taken one [8:35] amount one fire hose worth of load [8:37] coming in and you've turned this into [8:39] seven fire hoses worth of load coming [8:41] cuz now you have to do a more expensive [8:42] reconstruction operation, right? So now [8:44] you've seven Xed the load. And these are [8:46] the kind of um cyclical kind of [8:48] behaviors that can lead to something [8:50] called congestion collapse. Congestion [8:51] collapse is when [8:53] workload to a system crosses a threshold [8:55] where it all kind of collapses. [8:58] So can you design against congestion [9:00] collapse is really the hardest part or [9:02] one of the hardest parts of Magic [9:03] Pocket, which is the name of the storage [9:04] system. [9:05] Um so ultimately we switched to Rust for [9:10] the storage nodes themselves. And this [9:12] again, this is before Rust was in GA, so [9:14] that was a bit of a risky move. Uh [9:16] but we rewrote um [9:18] the switch to Rust coincided with [9:20] getting rid of the file system entirely [9:22] on the disks and directly addressing the [9:25] the disk disk heads. So there was a [9:27] there's an instruction set, I think it's [9:28] called ZBC, zone based block control, [9:31] something like that. There's there's a [9:32] there's an instruction set for accessing [9:34] disks that we were using. [9:36] The disk manufacturers gave us the draft [9:38] specs of these new disks and we were [9:40] operating off the draft specs and [9:42] directly controlling the the disks. And [9:44] so, [9:45] all that product was tied up um together [9:47] um [9:49] into a product called DiscoTech. [9:50] It was the disk technology project. Um [9:53] and ultimately, if you look at that pie [9:55] chart, one congestion collapse stopped [9:57] and reliability improved. But if you [9:59] looked at the pie chart of where all the [10:00] money was going, it really shifted to be [10:03] almost all disks. And what we wanted to [10:05] do is get that pie chart to be [10:08] almost all the money spent on disks and [10:10] as little money spent on RAM, on [10:12] compute, on network, on power, on sheet [10:14] metal, etc. [10:16] >> The cascading failure you mentioned, was [10:18] there So, that that happened and then [10:20] there was a postmortem. [10:21] >> I don't think we needed a postmortem. I [10:23] think we we knew as it was happening. [10:25] >> [laughter] [10:26] >> I think when I was getting paged in the [10:28] middle of the night on these things, I [10:29] would you know you'd be pretty pretty [10:31] evident. And so, look, this is the [10:33] Again, this is an argument in favor of [10:35] the cloud. All right. Imagine you've [10:37] spent [10:38] hundreds of millions of dollars on a on [10:40] a storage system and it's out in [10:42] production and it's on physical [10:44] hardware. [10:45] You can't just go and put new memory [10:48] chips in every one of them. I mean, you [10:50] can, but you have to pay people to come [10:52] in and swap them out, you know? And so, [10:54] that's a tricky place. And so, as a And [10:57] this is what This is what I love about [10:58] industry. You know, cuz in like some of [11:01] the more um [11:03] challenging moments on magic pocket were [11:05] stuff like In one week, [11:08] just a weird coincidence, two trucks [11:10] crashed that were delivering servers. [11:12] So, there's two trucks showing up to [11:13] deliver racks and they both crashed. I [11:15] don't know, you know, the drivers were [11:16] okay. So, we lost capacity for 2 weeks. [11:19] Uh so So, we lost capacity for for more [11:22] than 2 weeks for for for probably 6 [11:24] weeks. [11:25] Um [11:26] How What do you do? [11:28] What happens, you know, is is the [11:29] equivalent of your disk filling up on [11:31] your laptop, except it's you know, it's [11:32] it's a million disks. [11:34] And you can't tell the customers to go [11:36] away. You can't delete their files. So, [11:38] a lot of a lot of tricky [11:41] a lot of tricky capacity work. [11:44] And ultimately what that what that led [11:45] to was trying to build in [11:49] all these protections against the [11:50] unknowns. You know, making sure we had [11:53] the right amount of buffer planned out [11:55] for anything bad that could happen. [11:57] Making sure we we design the system so [11:59] they couldn't be congestion collapsed so [12:01] they wouldn't be memory spikes. [12:03] Um and [12:05] you know, at one point there's a there's [12:06] a process called I think it's called [12:07] FMEA, [12:09] which is like a threat modeling process [12:11] where you had a big spreadsheet and you [12:12] kind of write down every bad thing that [12:15] could possibly happen and then, you [12:16] know, all the how bad it would be if it [12:18] happened. [12:20] Existential risk [12:22] to someone die, you know, if there's a [12:23] fire in the data center. All these kind [12:24] of things get put into the spreadsheet. [12:26] And you kind of do a bit of a almost [12:28] pre-mortem kind of a work to figure out [12:31] all the all the potential failure modes [12:33] and then design around them. And I [12:35] that's I love that work. I I really um [12:38] I don't know. I do like the firefight. I [12:40] I like the [12:42] I don't I can't say I like getting paged [12:44] because [12:45] I've spent my whole life on call, [12:47] but I do like that rubber hits the road [12:50] stuff. I like that [12:52] wow, there's congestion collapse and and [12:54] there's no one that can help you. And so [12:56] you've got to [12:58] think through this problem. [12:59] >> When I worked on infrastructure at [13:01] Instagram, we had this concept of DEFCON [13:03] knobs, which are basically these configs [13:06] that you could flip that would [13:08] gracefully degrade your system where you [13:11] can still operate it, but you know, [13:13] maybe in the case of Dropbox, you store [13:15] less replicas. So, you take on a [13:18] temporary increase of [13:21] risk in losing data because you need to. [13:24] Did you have something like that and did [13:25] you flip it in that type of case? [13:27] >> Never when it came to durability. So, I [13:30] mean, we just had an just absolute zero [13:33] non-negotiable like there was no room [13:35] for negotiation on on users' durability. [13:37] And so, um [13:39] we had those knobs for background [13:41] processes for CPU and and memory, for [13:42] example. So, if there was like a spike [13:44] in load, you could turn off background [13:46] processes and and turn off um you know, [13:49] um the the test load, for example, on [13:51] the system. Eventually, we we built this [13:53] system called [13:55] trampoline. [13:57] And what trampoline did was um which [14:00] actually was a it it saved us a ton of [14:01] money, right? When if we ever got too [14:04] close to the threshold, we would just [14:06] start writing data to S3. [14:08] Cuz it's right there, right? So, it's um [14:11] you can run your capacity way closer to [14:14] the edge if you're willing under [14:16] worst-case scenarios just to just to [14:18] dump 30 petabytes on S3 and then move it [14:21] back when it's done. And now, it did it [14:23] didn't happen very often. We would do it [14:24] to test it. We would do that just to [14:26] make sure the system worked. [14:27] But, um yeah, being able to have a [14:30] escape hatch for worst-case scenario was [14:32] really nice. [14:34] >> I see. So, S3 is kind of um [14:36] it's like elastic storage. [14:37] >> Exactly.