FoundationDB Simulates Clusters on Single Threads to Stop Data Loss

"Testing Distributed Systems w/ Deterministic Simulation" by Will Wilson — Strange Loop Conference

Strange Loop Conference

Distributed systems cannot guarantee transactional reliability unless developers eliminate real-world hardware entropy and validate production logic inside an accelerated, deterministic simulator.

AI-assisted summary of "Testing Distributed Systems w/ Deterministic Simulation" by Will Wilson — original source. Check the original for full context.

Source published:

Key ideas

  • FoundationDB spent its first two years building a completely deterministic simulation before writing the actual database engine, ensuring all concurrency and network interactions were testable inside a single operating system process. (4:03)
  • The team created Flow, an actor-model C++ syntactic extension and precompiler, to avoid callback hell and provide deterministic, single-threaded pseudo-concurrency without hardware thread-race conditions. (7:16)
  • FoundationDB runs 5 to 10 million simulation runs nightly—the equivalent of trillions of real-world CPU hours—by accelerating failure frequencies and skipping idle simulated time. (28:26)
  • Physical validation on the 'Sinkhole' power-cycling cluster uncovered a power-safety bug in Apache ZooKeeper and a missing fsync bug in Linux's apt package manager, while finding zero power-safety bugs in FoundationDB itself. (33:02)

Detailed notes

## Background Presented by Will Wilson of FoundationDB, this talk was published by the Strange Loop Conference on YouTube on September 20, 2014. According to the description, debugging concurrent distributed systems in noisy network environments is nearly impossible through purely analytical or empirical means, making deterministic simulation—where hardware components like disks and networks are swapped for fully transparent software contracts—a powerful method for indefinitely reproducing and verifying complex failure modes.

## TL;DR Will Wilson outlines FoundationDB's architectural choice to build a fully deterministic simulation framework before implementing the physical database itself, ensuring transactional ACID guarantees could be reliably tested. To overcome thread-scheduling randomness while preserving C++ performance, the team built Flow, an actor-model language extension compiling asynchronous routines into single-threaded state machines. By replacing network and disk interfaces with deterministic models, the engine compresses years of operational time into minutes, injecting synthetic disasters, correlated cascading failures via the Hurst exponent, and randomized contract permutations using a `buggify` macro. While deterministic simulation drastically accelerates bug discovery across millions of nightly runs, it forces engineers to rely on deterministic `printf` logging and complementary physical chaos testing via a dedicated hardware cluster nicknamed 'Sinkhole.'

## Atmosphere & style Wilson's presentation is dryly humorous, deeply practical, and grounded in the shared trauma of distributed systems debugging, opening with the quip that debugging distributed systems is only narrowly preferable to sticking a fork in one's eye. His narrative style balances academic depth—such as applying 1950s hydrology statistics to disk crash correlations—with candid confessions about the absurdity of spending years writing a simulator for a nonexistent database. The delivery feels refreshingly honest and battle-hardened, giving the audience an authentic look at the sheer engineering discipline, developer friction, and burned-out hardware required to guarantee database correctness.

## Bulletin - [04:03](https://www.youtube.com/watch?v=4fFDFbi3toc&t=243s) FoundationDB spent its first two years building a completely deterministic simulation before writing the actual database engine, ensuring all concurrency and network interactions were testable inside a single operating system process. - [07:16](https://www.youtube.com/watch?v=4fFDFbi3toc&t=436s) The team created Flow, an actor-model C++ syntactic extension and precompiler, to avoid callback hell and provide deterministic, single-threaded pseudo-concurrency without hardware thread-race conditions. - [28:26](https://www.youtube.com/watch?v=4fFDFbi3toc&t=1706s) FoundationDB runs 5 to 10 million simulation runs nightly—the equivalent of trillions of real-world CPU hours—by accelerating failure frequencies and skipping idle simulated time. - [33:02](https://www.youtube.com/watch?v=4fFDFbi3toc&t=1982s) Physical validation on the 'Sinkhole' power-cycling cluster uncovered a power-safety bug in Apache ZooKeeper and a missing fsync bug in Linux's apt package manager, while finding zero power-safety bugs in FoundationDB itself.

## Key takeaways - [00:01](https://www.youtube.com/watch?v=4fFDFbi3toc&t=1s) Debugging distributed systems is notoriously difficult because uncontrolled environmental entropy—such as delayed packets or reordered transmissions—destroys test repeatability. - [04:03](https://www.youtube.com/watch?v=4fFDFbi3toc&t=243s) ACID transactional guarantees cannot fail safely in production, prompting the decision to simulate the distributed environment before implementing database operations against real hardware. - [07:16](https://www.youtube.com/watch?v=4fFDFbi3toc&t=436s) To achieve single-threaded determinism without thread race non-determinism, FoundationDB developed Flow, an actor-model concurrency precompiler for C++. - [12:00](https://www.youtube.com/watch?v=4fFDFbi3toc&t=720s) Hardware communication interfaces for disks and networks were replaced with deterministic software implementations that programmatically inject random connection drops and latency. - [13:50](https://www.youtube.com/watch?v=4fFDFbi3toc&t=830s) Absolute determinism requires deriving all system randomness from a controllable, seeded PRNG; unseeded time lookups or free disk space queries completely break repeatability. - [13:50](https://www.youtube.com/watch?v=4fFDFbi3toc&t=830s) To verify reproducibility, approximately 1% of all simulation tests are run twice with identical seeds to ensure their final PRNG states match perfectly. - [15:59](https://www.youtube.com/watch?v=4fFDFbi3toc&t=959s) Workload tests, such as mutating a closed ring of keys and values, check invariants to catch atomicity and isolation violations without requiring the system to crash. - [19:11](https://www.youtube.com/watch?v=4fFDFbi3toc&t=1151s) Simulations subject nodes to gradual degradation, whole-datacenter dropouts, and human error such as atomic IP address swaps or data file exchanges. - [20:42](https://www.youtube.com/watch?v=4fFDFbi3toc&t=1242s) Because real customers possess vastly more CPU hours than a developer lab, simulation testing must achieve a far higher failure density per CPU hour to discover bugs first. - [22:04](https://www.youtube.com/watch?v=4fFDFbi3toc&t=1324s) Time acceleration compresses testing cycles by forcing disks to fail every few minutes and advancing virtual clocks instantaneously across idle periods. - [23:04](https://www.youtube.com/watch?v=4fFDFbi3toc&t=1384s) The buggify macro forces the exploration of subtle edge cases by probabilistically triggering alternate, contract-compliant code paths such as reversed packet order or suppressed timeouts. - [24:51](https://www.youtube.com/watch?v=4fFDFbi3toc&t=1491s) Real-world infrastructure disasters follow high Hurst exponent distributions with fat tails, requiring simulations to model correlated, cascading hardware failures rather than naive Gaussian independence. - [29:25](https://www.youtube.com/watch?v=4fFDFbi3toc&t=1765s) Because interactive debuggers struggle with single-process simulated clusters and callback-heavy code, developers rely on surgical printf logging that behaves identically across repeated deterministic runs. - [30:37](https://www.youtube.com/watch?v=4fFDFbi3toc&t=1837s) The 'Sinkhole' chaos cluster physically stress-tests server motherboards and SSDs using programmable power supplies to catch flaws where simulator models deviate from physical reality. - [33:02](https://www.youtube.com/watch?v=4fFDFbi3toc&t=1982s) ZooKeeper was completely replaced with a native Flow implementation of Multi-Paxos because external third-party software could not participate in deterministic simulation. - [38:16](https://www.youtube.com/watch?v=4fFDFbi3toc&t=2296s) While production chaos testing stresses physical hardware and firmware idiosyncrasies, deterministic simulation provides adversarial control down to the microsecond with zero irreproducible Heisenbugs.

## Detailed walkthrough ### [00:01](https://www.youtube.com/watch?v=4fFDFbi3toc&t=1s) The Fundamental Problem of Debugging Distributed Systems

Will Wilson introduces himself as an engineer at FoundationDB, a company building a scalable, fault-tolerant database with ACID transactions. Rather than discussing the database itself, he focuses on simulation testing and why it can dramatically simplify an engineer's life. He opens with a universally relatable observation about engineering pain: “debugging distributed systems is preferable to sticking a fork in your eye, but not that preferable it's actually like it's actually surprisingly close.” While people often blame the sheer complexity of distributed architectures for this misery, Wilson argues that complexity is merely hard; it is not the true root cause of why distributed systems are uniquely impossible to debug.

```mermaid flowchart LR accTitle: Asymptomatic Bug Manifestation accDescr: Server on the right sends packets that get delayed or reordered by network anomalies, triggering catastrophic failure on the server on the left.

subgraph NetCondition["Network Chaos (Entropy)"] direction TB Delay["Time Warp / Lost Packet"] Reorder["Out-of-Order Delivery"] end

SR["Server (Right)"] -->|"Packet(s)"| NetCondition NetCondition -->|"Retry or Delayed Delivery"| SL["Server (Left)"] SL -->|"Unhandled Edge Case"| Boom["Explosion / Data Loss"] ```

To illustrate the actual culprit, Wilson presents a simple scenario involving two nodes: a server on the left and a server on the right. The server on the right sends a non-idempotent packet—meaning receiving it twice produces a fundamentally different outcome than receiving it once. The server on the left contains an asymptomatic bug: it fails to verify how many times the packet has arrived. Because networks usually behave, the bug remains dormant. However, the moment a real-world network hiccup occurs—such as a misconfigured switch or a router routing traffic on a bizarre detour through China—the original packet vanishes into a temporary time warp. The system initiates a retry as intended, but then the delayed packet unexpectedly emerges. The server processes the duplicate, causing the system to crash, corrupt data, or explode.

When the developer attempts to reproduce and debug the failure, they restart the environment, but the system runs flawlessly. Because the triggering condition was an extraordinarily transient network anomaly requiring microsecond-precise timings, the developer cannot reproduce the failure locally. Worse, even if a putative fix is written, there is no way to verify that it actually solved the underlying issue. Wilson reinforces this with a second example: the right-hand server sends packet `A` followed by packet `B`. Although the protocol specification does not guarantee order of arrival, developers naturally write linear code (`send(A)` then `send(B)`) rather than rolling dice to randomize transmission order. As a result, the receiving server implicitly assumes `A` will always arrive before `B`. When packet `A` gets delayed and `B` arrives first, the system breaks in an unrepeatable manner.

Wilson summarizes the underlying cause: “the surface level problem is that we have no repeatability are debugging the real problem is that the messy dirty Universe has intruded on our beautiful pristine land of pure functions right like the execution history of this system is not a pure function of its inputs there's a source of entropy a source of Randomness which you do not control which is the network.” This intrusion of uncontrolled entropy applies equally to threads, disks, and operating systems.

[04:03](https://www.youtube.com/watch?v=4fFDFbi3toc&t=243s) The FoundationDB Philosophy: Simulation Before Implementation

For FoundationDB, building a distributed transactional database made uncontrolled non-determinism an existential issue. Wilson highlights that ACID transactional guarantees are not emergent behaviors that occur by happenstance; there are exponentially more ways to build them incorrectly than correctly. More terrifyingly, there are countless ways to implement them so that they appear correct during normal execution but fail completely the instant an unpredicted environmental failure occurs. Furthermore, unlike a stateless web server or caching layer where operators can casually nuke a malfunctioning virtual machine and spin up a replacement, database users are never philosophical about losing their data.

Consequently, the founding team made a radical architectural decision: if they could not solve the problem of deterministic testability, they should not build a database at all. Instead of writing a database engine from day one, they spent the first couple of years building a 100% deterministic simulation of a database. Wilson acknowledges how absurd this sounds: “for actually a couple years there was no database there was just a simulation that probably sounds totally insane I promise you it's more insane than you think it is.” Only after the simulated architecture was exhaustively debugged did they connect it to real physical disks and physical network interfaces.

Wilson outlines the three massive software engineering challenges required to simulate not just a single process, but an entire distributed cluster of interacting processes, networks, disks, and operating systems within a single physical operating system process:

1. **Single-threaded pseudo-concurrency**: Simulating dozens of concurrent virtual nodes within one single physical OS process without introducing non-deterministic execution schedules from hardware threads. 2. **Simulated environmental interfaces**: Providing deterministic software replacements for all communication interfaces (networks, disks, timers, and clocks) so that every drop of randomness in the system is completely under developer control. 3. **Deterministic process logic**: Ensuring that the internal logic of the simulated processes themselves remains strictly deterministic.

[07:16](https://www.youtube.com/watch?v=4fFDFbi3toc&t=436s) Flow: Actor-Model Concurrency for C++

To achieve single-threaded pseudo-concurrency, FoundationDB ruled out multi-threading due to scheduling non-determinism and dismissed coroutines due to practical implementation issues at the time. This left callback-driven architectures. However, Wilson explains that FoundationDB required C++ for raw execution speed, creating a major dilemma: “callbacks plus C++ is like a nightmarish hellscape um was not something that we wanted to deal with what we needed were higher level Primitives um some abstractions that we could actually reason about that we could actually write and actually read.”

```mermaid flowchart TD accTitle: Flow Compilation Pipeline accDescr: Flow source with actor keywords compiles to standard C++ callbacks and state machines.

A["Flow Source (.actor.cpp)<br/>actor keyword, async/wait, apparent plain return"] -->|"Flow Precompiler"| B["Standard C++ Code (.actor.g.h /.g.cpp)<br/>Generates class, constructor, returns Future&lt;T&gt;"] B -->|"Standard C++ Compiler"| C["Executable with Deterministic<br/>Event-Loop Runtime"] ```

To bridge this gap, the team created **Flow**, an abstraction that sits halfway between a library and a programming language. Flow is a custom syntactic extension and pre-compiler targeting C++ that allows engineers to write asynchronous code using actor-model concurrency while compiling it down into single-threaded, callback-driven state machines.

Wilson walks through the simplest conceivable Flow actor: an asynchronous addition routine (`asyncAdd`). The code accepts a `Future<float>` and a plain `float`, waits for the future value to resolve, and returns their sum. He highlights two syntax peculiarities that differentiate Flow from vanilla C++: - The `actor` keyword flags the block to the Flow compiler so it knows to transform the function into an actor class and state machine. Passing this keyword into a standard C++ compiler would produce a syntax error. - The declared return type is `Future<float>`, yet the body contains what looks like a synchronous return of a standard `float`. The function body never runs directly as written; it acts declaratively. Flow synthesizes a wrapper function that immediately returns an unfulfilled `Future`, which only completes when the underlying asynchronous operations finish.

Behind the scenes, the Flow precompiler strips the original body, adds `const` modifiers to parameters, instantiates a newly generated actor class, passes the parameters to its constructor, and ties the lifecycle into the single-threaded event loop.

- Flow acts as a precompiler targeting standard C++. - The actor's return type in Flow represents the eventual asynchronous completion (`Future<T>`), allowing concise, linear syntax for asynchronous operations. - All actor scheduling runs sequentially within a single thread, eliminating thread-race non-determinism while preserving readable, asynchronous control flow.

[09:42](https://www.youtube.com/watch?v=4fFDFbi3toc&t=582s) Under the Hood of Flow and the Ring Benchmark

When compiling Flow code, the generated actor class returns immediately, and the actual implementation of the actor is broken up across various methods within the class. Every time the code encounters an `await` statement—where it must wait for an asynchronous process to terminate—the current function in the class ends. It sets a callback on that future pointing to the next part of the actor, and then yields immediately to a centralized scheduler so the rest of the program can continue running. At the very end of this process, the actor fulfills the future it was holding with its promise, destroys itself, and cleans up. This entire asynchronous flow runs efficiently within a single thread.

To demonstrate the speed of this approach, Will Wilson shares performance numbers from the "Ring Benchmark," originally designed by Erlang creator Joe Armstrong. In this benchmark, $N$ actors are arranged in a ring, and they pass a message to one another all the way around the ring $M$ times. Wilson warns the audience to take these benchmark numbers with a "gigantic grain of salt" for two reasons: first, the non-Flow comparison numbers were pulled from the internet and run by other teams, and second, the benchmark itself is highly simplistic.

As actors become more complex and incorporate more asynchronous logic in their bodies, Flow's relative performance advantage decreases. This limitation is what has driven the team to begin writing Flow 2 to achieve even greater speeds.

> "I hope Joe Armstrong's not here, is he? It's kind of a dumb benchmark... like I mean it doesn't actually model like what real distributed actor systems do very much."

- **The Ring Benchmark:** A test where $N$ actors pass a message around a ring $M$ times to measure message-passing latency and overhead. - **Flow Actor Compilation:** Translates asynchronous `await` statements into a series of callbacks and state transitions, yielding execution back to a single-threaded scheduler. - **Flow 2:** An ongoing development effort to maintain high performance even as actor complexity and internal asynchrony scale up.

---

[12:00](https://www.youtube.com/watch?v=4fFDFbi3toc&t=720s) Simulating Hardware and Injecting Network Chaos

Because FoundationDB was already designed as a multi-platform program, it featured clean, abstracted interfaces for interacting with disks and networks. This architectural separation made it incredibly easy to drop in simulated implementations right alongside the real Unix and Windows networking code. In the simulation environment, attempting to establish a network connection does not open a real socket; instead, it initializes a C++ class containing internal state and buffers. To simulate network latency, the simulator waits for a short period on the sending side, waits again on the receiving side, copies the bytes, and finally notifies the receiver that data is available.

```mermaid flowchart TD accTitle: Simulated Network Connection with Latency and Fault Injection accDescr: A flowchart showing how a simulated network connection processes data, applies latency, and randomly injects faults.

Start["Send Data"] --> LatencySend["Simulate Sender Latency"] LatencySend --> RollLoss{"Roll Random Close?"}

RollLoss -->|Unlucky / True| InjectFault["Inject Fault: Close Connection / Return Error"] RollLoss -->|Lucky / False| LatencyRecv["Simulate Receiver Latency"]

LatencyRecv --> CopyBytes["Copy Bytes to Buffer"] CopyBytes --> Notify["Notify Receiver: Bytes Available"] ```

The true power of this simulated network lies in its ability to inject faults. Within the simulated network's `read` method, the code makes a call to a function called `roll_random_close`. Every time a process attempts to read from the simulated network, the system rolls a pseudo-random number. If the run is "unlucky," subsequent random rolls determine a specific failure mode: the peer might abruptly close the connection, the local side might close it, or the connection might fail entirely with an error. This chaotic environment is designed to aggressively flush out any latent assumptions about network reliability.

> "This is like how we try to smoke out any code in our system that might have been assuming the network was reliable, which is a very foolish thing to assume..."

---

[13:50](https://www.youtube.com/watch?v=4fFDFbi3toc&t=830s) The Elusive Quest for Absolute Determinism

Achieving complete determinism sounds straightforward in theory, but non-determinism creeps into software in incredibly subtle ways. The most obvious source is random number generation, such as the random numbers used to calculate exponential backoff. To keep this deterministic, all randomness must flow from a pseudo-random number generator (PRNG) that the developers control and seed. The seed of this PRNG then becomes a direct input to the program, ensuring that the exact same seed always produces the exact same execution history.

However, randomness is not the only culprit. Simply checking the system time inside an `if` statement or querying the operating system for free disk space instantly introduces non-determinism, causing two different runs of the same program to diverge. Despite rigorous training, programmers still make mistakes and introduce non-deterministic behavior. To combat this, FoundationDB runs approximately 1% of its simulation tests twice using the exact same inputs. At the end of the dual runs, the engine compares the state of the PRNGs; if the next numbers they would have generated do not match perfectly, a determinism failure is flagged and investigated.

> "Our programmers are trained to recognize this stuff and to try and avoid it, and we still screw it up all the time — really all the time..."

- **Sources of Non-Determinism:** System time checks, free disk space queries, unseeded random number generators, and local OS resource checks. - **PRNG Control:** All simulation randomness is derived from a single, seeded pseudo-random number generator, making the seed a critical input for test replication. - **Double-Run Verification:** A testing safety net where 1% of simulations are run twice with identical inputs to verify that the final internal states match exactly.

---

[15:59](https://www.youtube.com/watch?v=4fFDFbi3toc&t=959s) Test Files, Cycle Tests, and the "Scary" Swizzle Flag

To find bugs, the simulation framework utilizes test files that declare two opposing forces: a workload that the system is trying to achieve, and a set of simulated disasters designed to prevent it. A prime example of a workload is the "cycle test." In this test, a ring of keys and values is inserted into the database, where each value points to the next key in the loop. The system then executes concurrent transactions at a rate of 1,000 per second. Each transaction mutates the keys and values but must preserve the integrity of the overall ring. This setup provides a simple, elegant invariant to check at the end of a run: if ACID properties are violated via an atomicity or isolation failure, the ring will be broken, shortened, or lengthened.

```mermaid flowchart TD accTitle: Cycle Test Invariant Ring accDescr: A diagram showing a ring of keys and values in a database being concurrently mutated by transactions while maintaining the ring structure.

subgraph Database ["Database State (The Ring)"] K1["Key 1"] -->|points to| K2["Key 2"] K2 -->|points to| K3["Key 3"] K3 -->|points to| K1 end

Tx1["Transaction 1 (Mutates Keys)"] -.->|Must Preserve Ring| Database Tx2["Transaction 2 (Mutates Keys)"] -.->|Must Preserve Ring| Database ```

While the cycle test runs, the simulation file unleashes a battery of faults. It performs "random clogging," which pseudo-randomly stops simulated network connections from sending or receiving packets. This test also utilizes a "swizzle" flag—a variation of clogging where a subset of network connections is stopped on a rolling basis and then brought back online in reverse order.

Additionally, the test features "attrition," which simulates power failures by killing up to 10 machines over the course of a 30-second test while ensuring at least three remain online. Most deviously, the test triggers a database configuration change mid-run, forcing the cluster to execute Multi-Paxos coordination changes while machines are rebooting and network connections are being clogged and swizzled.

> "For reasons that we totally don't understand, this is better at finding bugs than normal clogging — it's kind of scary."

- **Cycle Test Invariant:** A database workload where concurrent transactions mutate a closed loop of key-value pairs; any violation of ACID properties breaks the loop, making transactional bugs easy to detect without a system crash. - **Swizzling:** An advanced network clogging pattern that pauses connections on a rolling basis and restores them in reverse order, which has proven highly effective at exposing distributed coordination bugs. - **Attrition and Reconfiguration:** Simulates physical machine power-offs and reboots alongside database configuration changes, forcing the system to run Multi-Paxos consensus under extreme network and hardware stress.

[19:11](https://www.youtube.com/watch?v=4fFDFbi3toc&t=1151s) Torturing the Database with Extreme Failure Modes

FoundationDB subjects its database to highly aggressive, simulated failure scenarios while simultaneously pushing a heavy workload of 1,000 transactions per second. The goal is to expose deep architectural flaws, such as bugs in their Paxos implementation, before the software ever reaches production. Rather than simply shutting down machines, the simulation engine can transition nodes into a state of gradual degradation. For example, the engine can inject a rule where all future system calls on a specific machine have a 1% chance of returning an error. This allows the team to verify if the cluster is intelligent enough to detect the failing node, quarantine it, and destroy it before it can infect the rest of the system.

The simulation framework also tests graceful shutdowns and leverages its awareness of network and physical topology. Because FoundationDB understands locality—such as rack configurations and WAN links—the simulation can mimic the catastrophic loss of an entire datacenter to verify that data remains durable. Additionally, the team simulates what they jokingly refer to as "dumb sysadmins," a phenomenon they dryly note "totally don't exist in the real world." To test resilience against human error, the simulation can atomically swap the IP addresses of two active machines, or instantly copy and swap the data files of two computers to see if any data loss occurs.

Will Wilson describes the joy of designing these brutal scenarios: “...one of the most fun parts of my job is thinking up new and creative ways of torturing our database and then implementing that in simulation...”

- **Gradual Degradation Simulation:** System calls are configured with a 1% failure rate to test the cluster's ability to quarantine failing nodes. - **Topology-Aware Failures:** The simulation can wipe out entire simulated datacenters to test WAN durability. - **Human Error Simulation:** The framework atomically swaps IP addresses or hard drive data files between active machines.

[20:42](https://www.youtube.com/watch?v=4fFDFbi3toc&t=1242s) The CPU Hour Race Against the Real World

The fundamental challenge of relying on simulation is the sheer scale of the state space. While a cluster of test machines diligently runs simulations to explore possible inputs and failure modes, the real-world deployment scale is vastly larger. Customers run distributed systems on highly imperfect infrastructure, and they do so at a scale that dwarfs any internal test lab. The engineering team must find a way to explore this massive state space and uncover bugs faster than their entire customer base combined.

To stay ahead, the testing framework must achieve a bug-discovery rate per CPU hour that is many orders of magnitude higher than that of the real world. If the customer base has access to far more CPU hours, the simulation must compensate by being incredibly dense with failures and edge cases, turning what would be rare, multi-year real-world events into routine, minute-by-minute occurrences.

Will Wilson highlights the harsh reality of shipping software to customers: “...your customers are running your distributed systems on buggy crashy failey grossly underprovisioned totally improperly configured Hardware and they're doing insane things to it by the way and PS if it breaks it's your fault.”

```mermaid flowchart TD accTitle: Simulation vs. Real World Bug Discovery accDescr: Comparison of how simulation compresses time and increases failure density to find bugs faster than customers in the real world. subgraph RealWorld ["The Real World"] A["Millions of Customer CPUs"] -->|Slow Failure Rate: Disks fail every 2-3 years| B["Bugs Found Over Years"] end subgraph SimWorld ["Deterministic Simulation"] C["Internal Test Cluster"] -->|High Failure Rate: Disks fail every 2-3 minutes| D["Time Acceleration & Bugification"] D -->|Fast Failure Rate| E["Bugs Found in Hours"] end ```

- **Scale Discrepancy:** Customers possess orders of magnitude more computing power than the developer's test lab. - **The Goal:** Achieve a significantly higher bug-detection rate per CPU hour than the real world through artificial acceleration.

[22:04](https://www.youtube.com/watch?v=4fFDFbi3toc&t=1324s) Accelerating Failures and Speeding Up Time

To compress years of real-world operations into hours of testing, FoundationDB employs two primary strategies: accelerating the frequency of failures and speeding up simulated time. In a production environment, a hard drive might fail once every two or three years under heavy load. In the simulation, the engine forces disks to fail every two or three minutes. This ensures that the code responsible for handling disk failures is executed and tested far more frequently per CPU hour than it ever would be in a standard deployment.

The second strategy is "cheating" by accelerating time itself. Distributed systems often spend significant periods idling, waiting for timeouts, or recovering from failures. When the simulated cluster enters a quiescent state where it is simply waiting for a hardcoded number of seconds to pass, the simulation engine skips the idle physical time. It instantly advances the simulated clock, allowing many simulated seconds to pass in a fraction of a real-world second, limited only by the processing power of a single CPU core.

Will Wilson explains the philosophy behind this acceleration: “...in the real world how often does a disc fail like every two or three years if it's under a heavy load or something we make them fail like every two or three minutes...”

- **Compressed Failure Cycles:** Disk failures are accelerated from a multi-year real-world cadence to a 2-to-3-minute simulation cadence. - **Time Skipping:** Idle and quiescent states are bypassed by instantly advancing the simulated clock.

[23:04](https://www.youtube.com/watch?v=4fFDFbi3toc&t=1384s) Bugification: Exposing Asymptomatic Bugs

Even with accelerated failures, some bugs remain asymptomatic because they require highly specific, rare conditions to manifest. To combat this, FoundationDB uses an in-house technique called "bugification." This process involves intentionally and randomly altering the behavior of the code during simulation to ensure that interfaces are robust and do not rely on implicit, narrow assumptions.

Bugification is implemented via a `buggify` macro. In simulation, this macro evaluates to true with a small probability, using a hash of the line number to maintain determinism. For example, a developer might write code that occasionally reverses the order of sent packets or deliberately suppresses a timeout. By forcing the system to handle these unusual but contract-compliant behaviors, the team ensures that the system can survive extreme edge cases.

Will Wilson describes the counterintuitive nature of this technique: “...the idea here is you do just that thing which I said programmers never do which is sometimes randomly change what your code does...”

```mermaid flowchart TD accTitle: Bugification Macro Logic accDescr: Flowchart showing how the buggify macro randomly alters code execution paths during simulation to test edge cases. A["Code Execution Path"] --> B{"Is Simulation & buggify active?"} B -->|Yes (Small Probability)| C["Altered Behavior (e.g., Swap packet order, suppress timeout)"] B -->|No| D["Standard Behavior (Contract-compliant path)"] C --> E["Verify System Resilience"] D --> E ```

- **The `buggify` Macro:** A compiler macro that randomly activates alternative, valid execution paths during simulation. - **Interface Testing:** Forces systems to handle out-of-order packets and missing timeouts to prevent implicit dependencies.

[24:51](https://www.youtube.com/watch?v=4fFDFbi3toc&t=1491s) The Hurst Exponent and Cascading Failures

To explain why naive testing fails to capture real-world disaster scenarios, Will Wilson introduces the concept of the Hurst exponent. First described by British civil servant Harold Edwin Hurst in his 1951 paper *"Long-Term Storage Capacity of Reservoirs"* (published in the *Transactions of the American Society of Civil Engineers*), the exponent was developed while studying the Nile River's dams. Previous engineers assumed that annual rainfall events were independent, uncorrelated trials, leading them to design reservoirs for "100-year" floods and droughts. However, Hurst observed that these extreme events were actually occurring every 15 years.

The discrepancy exists because real-world events are highly correlated; the probability of rain on Wednesday is higher if it rained on Tuesday. This lack of independence yields stable Paretian distributions with fatter tails than a standard Gaussian distribution. In a Gaussian distribution (where the Hurst exponent is 0.5 and the alpha parameter is 2), extreme "high-sigma" events are virtually impossible. When the Hurst exponent exceeds 0.5, the fatter tails mean these extreme events occur far more frequently.

In distributed systems, hardware failures are similarly correlated. If a hard drive fails, it is often due to a bad manufacturing batch, poor maintenance, or environmental factors like high humidity, meaning other drives in the same rack are highly likely to fail soon after. While physical testing of cascading, correlated failures is incredibly difficult, FoundationDB's simulation framework easily manipulates the Hurst exponent to generate highly correlated, catastrophic failure cascades.

Will Wilson emphasizes the non-random nature of infrastructure failures: “...the problem is that multiple cascading failures which a non-naive approach to statistics tells us are much more likely than a simple naive Gaussian would predict are among the very very hardest things to test in real world physical testing...”

- **Harold Edwin Hurst (1951):** Discovered that natural events are correlated, invalidating the independent-trial assumptions used in reservoir engineering. - **Fatter Tails:** A Hurst exponent greater than 0.5 results in stable Paretian distributions where extreme events occur much more frequently than a Gaussian distribution predicts. - **Correlated Hardware Failures:** Simulation allows the team to artificially increase the correlation of failures to mimic real-world cascading disasters.

[28:26](https://www.youtube.com/watch?v=4fFDFbi3toc&t=1706s) Trillions of Hours and the Curse of Callback Debugging

By combining time acceleration, aggressive failure injection, and bugification, FoundationDB achieves an extraordinary testing scale. The team runs between 5 and 10 million simulation runs every single night on their testing cluster. Each run simulates an entire cluster operating for anywhere from five minutes to two hours. Over years of continuous testing, they estimate they have executed the equivalent of trillions of real-world CPU hours.

However, this powerful testing methodology introduces a major drawback: it completely breaks traditional debugging. Stepping through highly concurrent, asynchronous code that is heavily reliant on callbacks is already notoriously difficult for developers. Attempting to use a traditional debugger to step through a deterministic simulation of callback-infused code is significantly worse, creating what Wilson describes as "callback infected monstrosities" that defy standard debugging techniques.

Will Wilson laments the developer experience in this environment: “...you all already know that stepping through code that is laced and infused with callbacks is horrible if you were to take a traditional debugging approach to a deterministic simulation of your call back infused code that'll be much worse.”

- **Testing Scale:** 5 to 10 million simulation runs executed nightly, representing trillions of equivalent real-world CPU hours over the project's lifespan. - **Simulation Duration:** Each run simulates between 5 minutes and 2 hours of cluster lifetime. - **The Debugging Tradeoff:** Traditional step-through debugging is rendered highly impractical due to the complex, callback-heavy nature of the simulated asynchronous code.

[29:25](https://www.youtube.com/watch?v=4fFDFbi3toc&t=1765s) Debugging in Simulation: The Surprising Power of Deterministic `printf`

Debugging distributed systems inside an attached debugger like GDB is notoriously difficult. When running a simulated cluster, multiple simulated processes coexist within the exact same operating system process you are attached to, alongside all the simulation harnesses busily convincing those nodes that they are secretly communicating with one another. While a few exceptional engineers on the FoundationDB team can somehow step two or three levels down the call stack in such an environment, the majority cannot. Consequently, developers are forced to rely heavily on `printf` logging.

```mermaid flowchart TD accTitle: Deterministic Printf Debugging Cycle accDescr: Flowchart illustrating how deterministic simulation enables targeted printf debugging without log spam.

subgraph Iteration["Debugging Workflow"] A["Reproduce Bug with Fixed Seed"] --> B["Add Precise, Conditional printf Statements"] B --> C["Re-run Simulation with Identical Input"] C --> D["Observe Exact Event Order at Exact Timestamps"] D --> E["Narrow Scope and Eliminate Bug"] end ```

While relying on print statements sounds primitive, it functions remarkably well under deterministic simulation. Because rerunning a run with the exact same inputs reproduces identical events in the exact same order at the exact same virtual times, developers can write surgical, non-spammy log statements tied to specific known preconditions. Wilson explains:

> “What you're left with is basically print F uh which sounds pretty horrible and barbaric uh and I'll be honest it is um but it's actually not that bad when you have deterministic simulation and you know that if you rerun the simulation with the exact same inputs the exact same set of stuff will happen in the exact same order at the exact same times.”

This repeatability enables engineers to incrementally drive their logging deeper into the code path, corner the elusive bug, fix it, and verify that it no longer occurs under that exact seed.

---

[30:37](https://www.youtube.com/watch?v=4fFDFbi3toc&t=1837s) The Limits of Simulation and the "Sinkhole" Chaos Cluster

Deterministic simulation has an inherent blind spot: the simulation can simply be wrong. Wilson highlights two distinct failure modes for a simulation framework: 1. **Insufficient brutality:** The simulation may not model bizarre real-world network phenomena that occur in the wild. 2. **Flawed contract assumptions:** The developers might misinterpret operating system semantics (such as subtle differences in what `fsync` guarantees across platforms) or hit an unexpected OS-level bug.

To catch discrepancies between simulated behavior and actual physical hardware, the team constructed an on-premise hardware validation cluster nicknamed "Sinkhole."

```mermaid flowchart LR accTitle: Architecture of the Sinkhole Testing Cluster accDescr: Hardware components of the Sinkhole chaos-testing cluster.

subgraph Cluster["Sinkhole Hardware Cluster"] direction TB MB["Server Motherboards<br>(Real Hardware w/ ECC RAM)"] SW["Network Switches<br>(Connected via Cat 5)"] PDU["Programmable Network<br>Power Supplies"] SSD["Host SSDs"] end

PDU -.->|"Randomized Hard Power Cycling"| MB PDU -.->|"Randomized Hard Power Cycling"| SW MB --- SSD ```

Sinkhole consists of small real-server motherboards equipped with ECC memory, wired through network switches with Cat 5 cables. Both the switches and the motherboards are plugged into programmable, networked power distribution units. The team programs these power supplies to relentlessly cycle power on and off all night while the database executes transactions under heavy load.

This brutal physical testing revealed practical operational headaches: - Network power distribution units burn out frequently under constant cycling: “Network power supplies suck they burn out after like two days it's a giant pain in the ass then we have to buy more we've run through hundreds of these things.” - Rapid power cycling rapidly degrades and destroys solid-state drives (SSDs). - Despite burning through hardware, the Sinkhole cluster has never uncovered a power-safety bug in FoundationDB itself, providing strong validation for the simulator's efficacy.

---

[33:02](https://www.youtube.com/watch?v=4fFDFbi3toc&t=1982s) Finding Bugs in Upstream Software: ZooKeeper and `apt`

While Sinkhole did not catch bugs in FoundationDB's own engine, it exposed critical bugs in third-party software that the team could not test inside their deterministic simulator. Any software written outside of their custom C++ actor framework (Flow) cannot participate in simulated scheduling.

- **Replacing Apache ZooKeeper:** Early FoundationDB builds used ZooKeeper for coordination state. A predecessor to the Sinkhole cluster uncovered a power-safety bug in ZooKeeper. Although the upstream team resolved the issue after FoundationDB filed a bug report, the scare was severe enough that FoundationDB completely ripped ZooKeeper out of their codebase. They wrote their own implementation of Paxos natively in Flow, allowing coordination to run directly within the simulation framework (with the added bonus of eliminating Java deployment friction). - **The Missing `fsync` in Package Managers:** In another Sinkhole trial, a node dropped offline and refused to restart after a power cut. The team investigated, puzzled, since the database was proven to be power-safe. They discovered the transactional database files had all been durably synced to disk, but the configuration files were corrupted or missing entirely. The operating system's package manager (`apt`) had omitted calling `fsync` when installing the package files. Upon restarting, the database process detected existing data files without a valid configuration file, assumed an inconsistent state, and safely panicked.

---

[34:34](https://www.youtube.com/watch?v=4fFDFbi3toc&t=2074s) Open Challenges: Developer Ergonomics and Test Suite Evolution

Even with deterministic simulation, exploring the combinatorial state space of distributed failures remains a massive undertaking. Wilson outlines several ongoing challenges and operational friction points:

- **Mutation Testing Teams:** Wilson notes they would love to dedicate engineers strictly to injecting subtle bugs into the codebase to observe whether the simulation caught them, treating failures of detection as actionable bugs against the simulation framework itself. - **Workflow Latency and Cycle Times:** The testing cycle places significant cognitive friction on engineers:

> “If you look at the workflow of one of us it looks like you make some changes to the program you recompile and then you run like 10,000 simulations to see if you did something obviously dumb and then you hope that if you did something subtly dumb you'll get caught that night when we run 10 million that sucks running 10,000 simulations takes like a minute or two or I don't know a while um you lose your train of thought it totally breaks your flow.”

- **Dual-Framework Mutation Resistance:** Because long-lived test suites can inadvertently allow surviving bugs to adapt around specific test heuristics, the team has considered building two distinct simulation frameworks based on different design principles. Developers would iterate against the fast primary framework day-to-day, while a second, independent framework would stress release candidates across tens of millions of runs—analogous to multi-drug protocols used to prevent antibiotic resistance. - **The Misery of Real Hardware:** Despite recognizing the value of physical chaos testing, interacting with physical hardware remains universally hated by developers. A colleague often jokes that all programming is enjoyable except for the code that interacts with networks, disks, or power supplies—yet that work is all he does now.

---

[38:16](https://www.youtube.com/watch?v=4fFDFbi3toc&t=2296s) Q&A: Simulation versus Production Chaos Testing (Chaos Monkey)

An audience member compares FoundationDB’s approach to Netflix's Chaos Army and asks about testing directly in production. Wilson clarifies that all simulation tests run actual, production-ready code. The only code path omitted from simulation is the thin outer boundary that interfaces with physical disks and the network interface cards.

Wilson contrasts the two paradigms: - **Simulation's Advantage (Omnipotent Control):** In a simulator, developers hold godlike, deterministic control over the entire virtual universe. They can inject calculatedly evil, worst-case network partitions and manipulate timing down to the microsecond precisely when Paxos is in its most vulnerable state. Real-world chaos testing rarely achieves this degree of targeted adversarial precision. - **Production Chaos Testing's Advantage (Hardware Fidelity):** Real-world chaos testing directly stresses the idiosyncratic firmware and physical failure modes of commercial disks, cables, and network cards in use across customer sites.

Wilson concludes by acknowledging that both approaches offer unique advantages, though FoundationDB leans into deterministic simulation because of the unmatched precision and reproducibility it affords when tackling subtle distributed concurrency bugs.

## Notable quotes > [20:42](https://www.youtube.com/watch?v=4fFDFbi3toc&t=1242s) your customers are running your distributed systems on buggy crashy failey grossly underprovisioned totally improperly configured Hardware and they're doing insane things to it by the way and PS if it breaks it's your fault. — Will Wilson

## Mentioned Will Wilson, FoundationDB, Flow, C++, Joe Armstrong, Harold Edwin Hurst, Paxos, Multi-Paxos, Apache ZooKeeper, GDB, Netflix, Chaos Monkey, Sinkhole, apt

## Video description Debugging highly concurrent distributed systems in a noisy network environment is an exceptionally challenging endeavor. On the one hand, evaluating all possible orders in which program events can occur is a task ill-suited to human cognition, rendering a pure analytic understanding of the control flow of such a system beyond the reach of any individual programmer. On the other hand, a more “empirical” approach to the task is also fraught with difficulty, as the dependence of severe bugs on precise timings or transient network conditions makes every part of the debugging cycle – from bug replication to verification of a fix – a Sisyphean labor bordering on the impossible.

One approach which has been developed to ameliorate this situation is that of deterministic simulation, wherein the hardware components of the system – including hard disks, network links, and the machines themselves – are replaced in testing with software which fulfills the contracts of those systems, but whose state is completely transparent to the developer. This enables the simulation of a wide diversity of failure modes including network failures, disk failures or space exhaustion, unexpected machine shutdown or reboot, IP address changes, and even entire datacenter failures. Moreover, once a particular pattern of failures has been identified which uncovers a bug, the determinism property of the simulation means that the exact same series of events can be replayed an indefinite number of times, greatly facilitating the debugging process, and providing confidence when a bug has been fixed.

Attendees of this talk will gain an understanding of the benefits, drawbacks, and tradeoffs involved in implementing a deterministic simulation framework, with frequent reference both to theory and to real-world engineering experience gleaned from applying this method to a complex distributed system. Attendees will also learn about language features which aid in the development of such a framework.

Will Wilson FoundationDB

Will Wilson works on the engineering team at FoundationDB (https://foundationdb.com). Will started his career in biotechnology, leading a successful R&D effort in spinal cord injury diagnostics, currently undergoing commercialization by a company he co-founded. Since then, Will has worked in a variety of technical and business roles at data science and data virtualization startups. Will has a degree in math and philosophy from Yale.

[Source](https://www.youtube.com/watch?v=4fFDFbi3toc)

中文简报

FoundationDB通过单线程集群模拟杜绝数据丢失 除非开发人员消除现实硬件的熵增，并在加速的确定性模拟器中验证生产逻辑，否则分布式系统无法保证事务可靠性。 Will Wilson概述了FoundationDB在实现物理数据库本体之前，先构建一套完全确定性模拟框架的架构抉择，以确保事务的ACID保证能够得到可靠测试。为了消除线程调度的随机性并兼顾C++的高性能，团队构建了Flow——一种基于Actor模型的语言扩展，可将异步例程编译为单线程状态机。通过用确定性模型替换网络与磁盘接口，该引擎能将数年的运行时间压缩至数分钟，注入合成灾难、利用赫斯特指数（Hurst exponent）模拟关联级联故障，并通过“buggify”宏进行随机契约排列。尽管确定性模拟在每晚数百万次运行中极大地加速了缺陷发现，但也迫使工程师依赖确定性的“printf”日志记录，并借助名为“Sinkhole”的专用硬件集群进行补充性的物理混沌测试。 FoundationDB在编写实际数据库引擎前耗费了最初两年构建一套完全确定性的模拟系统，以确保所有并发和网络交互均可在单个操作系统进程内测试。 (4:03) 团队创建了基于Actor模型的C++语法扩展与预编译器Flow，以避免回调地狱，并在没有硬件线程竞态条件的情况下提供确定性、单线程的伪并发。 (7:16) FoundationDB通过加快故障发生频率并跳过模拟的空闲时间，每晚运行500万至1000万次模拟——相当于数万亿小时的现实CPU运行时间。 (28:26) 在名为“Sinkhole”的断电循环集群上进行的物理验证发现了Apache ZooKeeper中的一处断电安全漏洞以及Linux apt包管理器中一处缺失fsync的漏洞，而在FoundationDB自身中未发现任何断电安全漏洞。 (33:02)

Reading summary

FoundationDB engineer Will Wilson argues that the core obstacle to debugging distributed systems is uncontrolled environmental entropy rather than mere architectural complexity. To guarantee transactional ACID properties, the team spent its initial years building a 100% deterministic simulation of the database, hardware, and network before deploying to real physical infrastructure. The Root Cause: Uncontrolled Entropy (0:01) Wilson asserts that distributed systems are notoriously difficult to debug because the external physical environment introduces uncontrolled randomness—such as network delays, dropped packets, and unsynchronized clocks—into execution histories. Consequently, systems lack repeatable execution paths. In normal environments, subtle bugs remain dormant because networks typically perform well; however, rare network anomalies (such as delayed duplicate packets or out-of-order delivery) can trigger unhandled edge cases that crash nodes or corrupt data. Because these conditions depend on microsecond-level timing, developers cannot easily reproduce them locally or verify that a bug fix works. Because ACID transactional guarantees have exponentially more ways to be built incorrectly than correctly, FoundationDB treated non-determinism as an existential threat. Concluding that stateless failure strategies like replacing malfunctioning instances are unacceptable for persistent storage, the team chose not to write a database engine immediately. Instead, they spent their first two years building a single-process, fully deterministic simulation of an entire distributed cluster. Architectural Foundations: Flow and Simulated Environments (7:16) To achieve single-threaded pseudo-concurrency without non-deterministic operating system thread scheduling, FoundationDB created Flow, an actor-model extension and precompiler for C++. Flow translates asynchronous functions into single-threaded, callback-driven state machines managed by a centralized scheduler. Every asynchronous wait point yields execution back to the scheduler, preserving readable, sequential code flow while entirely eliminating thread-race non-determinism. The team also implemented deterministic software replacements for platform communication interfaces, replacing physical sockets, disks, timers, and clocks with internal C++ classes and memory buffers. All simulated entropy originates from a single pseudo-random number generator (PRNG) controlled by an explicit seed. Because subtle operations like checking the operating system clock or free disk space can introduce non-determinism, FoundationDB runs approximately 1% of all simulation tests twice using the exact same seed; if the final internal PRNG states diverge, the framework flags an immediate determinism failure. Compressing Trillions of Hours via Accelerated Chaos (20:42) Because customers deploy distributed systems on vast, imperfect infrastructure that dwarfs internal test labs, the testing harness must achieve a bug-discovery rate per CPU hour far higher than real-world deployments. FoundationDB accelerates testing by forcing disk failures to occur every two to three minutes instead of every two to three years, and by advancing simulated clocks instantaneously whenever nodes enter idle or waiting states. Executing 5 to 10 million simulation runs per night, the team estimates they have accumulated trillions of equivalent real-world CPU hours. Simulations combine heavy workloads with destructive fault injection. In the "cycle test," concurrent transactions mutate a ring of key-value pairs at 1,000 transactions per second while connections are clogged, swizzled in reverse order, or reconfigured mid-flight under Multi-Paxos consensus. To uncover dormant bugs, the system uses a `buggify` macro that probabilistically alters code paths within interface contracts. Furthermore, the framework models correlated failures by adjusting the Hurst exponent ($H > 0.5$), reproducing the heavy-tailed cascading failure distributions observed in real-world infrastructure that naive Gaussian models underestimate. Deterministic Debugging and Physical Testing Limits (29:25) While simulation breaks traditional step-through debuggers due to the complexity of nested asynchronous callbacks across dozens of virtual nodes, it makes `printf` logging exceptionally effective. Because rerunning a failed test with an identical seed reproduces the exact sequence of events at identical virtual timestamps, developers can iteratively place targeted log statements to isolate the fault. However, deterministic simulation remains blind to flawed assumptions about underlying operating system semantics, such as subtle differences in `fsync` guarantees. To bridge this gap, the team built "Sinkhole," a physical cluster of server motherboards and switches plugged into programmable power units that cycle power continuously under load. While Sinkhole rapidly burned out power supplies and solid-state drives, it validated the simulator by finding zero power-safety bugs in FoundationDB itself. It did, however, expose critical third-party bugs: an uncovered power-safety defect in Apache ZooKeeper prompted FoundationDB to rewrite consensus natively in Flow, and an omitted `fsync` in the operating system package manager (`apt`) was found to corrupt configuration files during reboots. While deterministic simulation provides unmatched precision for reproducing complex concurrency edge cases, Wilson highlights ongoing operational challenges, including workflow interruptions from long test runs and the risk that bugs adapt around established testing heuristics. Physical chaos testing remains necessary to expose low-level operating system discrepancies, but deterministic simulation serves as FoundationDB's primary defense against distributed state corruption.

精炼阅读摘要

FoundationDB 工程师 Will Wilson 认为，分布式系统难以调试的根源并非单纯的架构复杂度，而是不受控的外部环境熵（如网络、硬件与调度的不确定性）打破了纯函数式的因果规律。为此，团队确立了在编写实际数据库引擎前先构建 100% 确定性模拟器的理念，通过全受控的执行历史彻底解决分布式并发缺陷不可复现的痛点。 环境熵入侵与模拟优先架构 (0:01) 分布式系统在现实中往往因为偶发且短暂的网络抖动（如数据包乱序、重发延时或非幂等处理失误）诱发致命崩溃。由于开发人员在本地重启后无法复现微秒级的临界时序，常规的调试与修复验证极度困难。Wilson 指出，ACID 事务的正确性极其脆弱，错误实现数不胜数且常在正常测试中表现正常，但用户绝不容忍数据丢失。因此，团队在项目前两年完全没有编写物理数据库，而是全力开发了一套能在单个操作系统进程内运行整个分布式集群的确定性模拟系统。该体系立足于三大支柱：单线程伪并发、软件虚拟化的环境通信接口（网络、磁盘、时钟），以及完全确定性的进程内部逻辑。 Flow 语言与单线程伪并发调度 (7:16) 为了在单个操作系统进程中运行数十个虚拟节点并彻底消除多线程硬件调度的非确定性，团队摒弃了常规的多线程与当时尚不成熟的协程方案。然而，直接使用 C++ 编写回调函数会导致代码演变为难以阅读和维护的复杂状态机。为此，FoundationDB 开发了名为 Flow 的语法扩展预编译器。Flow 结合了 Actor 模型与 C++ 的执行效率，工程师可采用类似线性同步调用的语法编写异步逻辑，随后由编译器自动将其转换为基于回调的单线程状态机，并由集中式事件循环统一调度。在针对消息传递延迟的“环形基准测试”（Ring Benchmark）中，Flow 展现了显著的调度效率，但 Wilson 同时也坦承随着 Actor 内部异步逻辑复杂度上升其性能优势会有所稀释，团队因此启动了 Flow 2 的研发以应对扩展挑战。 混沌故障注入与不变量检验 (12:00) 模拟环境中的网络与存储均采用轻量级 C++ 对象替代真实系统调用。在读取网络数据时，系统通过受控的伪随机数生成器（PRNG）调用随机闭包函数，人为制造连接中断、延时或单向丢弃，以强行暴露代码中对网络可靠性的隐式假设。为防止代码偷偷调用真实系统时间或磁盘余量引入不确定性，测试框架会将约 1% 的模拟测试执行两次，并在结束时比对 PRNG 最终状态以核验完全一致。在验证层面，系统引入了对抗性的“循环测试”（Cycle Test）：数据库以每秒 1000 笔事务的高并发速率持续修改一组形成闭环的键值对，若 ACID 的原子性或隔离性遭到破坏，闭环结构必然变异，从而实现无需系统崩溃即可自动检出的不变量断言；在此期间，测试框架还会同步施加网络反转阻滞（Swizzle）、节点断电重启以及 Multi-Paxos 动态集群重配等多重压力。 时间压缩、Bugification 与长尾灾难 (20:42) 面对数以万计客户 CPU 年限与有限测试算力之间的巨大差距，模拟框架必须在单位 CPU 工时内爆发式提升故障密度。团队将真实世界中两到三年才发生一次的磁盘物理损坏，压缩至模拟器中每两到三分钟发生一次；遇到节点等待超时的空闲期，模拟时钟会瞬间跳跃，以此在一台物理机上实现超越物理时间的超高密度测试。此外，代码中嵌入了基于行号哈希的 `buggify` 宏，在模拟环境中以极小概率激活非标准但合规的极端边缘逻辑（如刻意抑制超时或反转包序）。Wilson 引用了 Harold Edwin Hurst 于 1951 年在尼罗河水坝研究中提出的赫斯特指数（Hurst Exponent），指出真实世界的物理故障从来不是高斯分布下的独立事件，而是具有强相关性的肥尾分布；通过在模拟中调高该指数，系统能够高效复现成批发生的级联硬件灾难。 物理验证机群与模拟调试的局限 (30:37) 依靠时间跳跃与故障加速，团队每晚可在测试集群上运行 500 万至 1000 万次模拟，累计覆盖了相当于数万亿真实 CPU 工时的执行历程。然而，高度嵌套的回调状态机导致传统 GDB 单步调试彻底失效，开发者转而依赖确定性模拟下输入种子完全一致的特性，通过精准、条件化的 `printf` 逐步缩小排错范围。同时，针对模拟可能误判操作系统底层语义（如跨平台的 `fsync` 契约差异）的短板，团队搭建了名为“Sinkhole”的真实硬件混沌机群，利用可编程电源分配单元对主板和交换机进行彻夜断电测试。尽管烧毁了大量电源插座和固态硬盘，Sinkhole 从未在数据库自身逻辑中检出断电安全性缺陷，反倒逼出了第三方软件的隐患：早期依赖的 ZooKeeper 因断电漏洞被团队彻底移除并重写为纯 Flow 实现的 Paxos，操作系统的软件包管理器（`apt`）也曾因安装缺少 `fsync` 导致配置文件在断电后损坏。 Will Wilson 总结指出，相比于针对实际物理硬件的生产环境混沌工程（如 Chaos Monkey），确定性模拟虽需前期投入数年基建且牺牲了单步调试的人机工效，但它赋予了开发者对并发时序绝对精准的掌控力与百分之百的可复现性。模拟测试的核心瓶颈在于模拟模型本身是否存在盲区，因此结合真实硬件层面的断电混沌测试，是确保全分布式系统达成企业级可靠性的关键闭环。

Detailed notes — zh

## Background 该视频由 Strange Loop Conference 官方频道于 2014 年 9 月 20 日在 YouTube 发布，主讲人为来自 FoundationDB 的 Will Wilson。根据页面简介，该演讲探讨了在充满网络噪音的环境中调试高度并发分布式系统的极端困境，并详细介绍了通过确定性仿真（Deterministic Simulation）替代物理硬件与网络以实现精确复现缺陷、验证修复方案的工程实践与设计权衡。 ## TL;DR Will Wilson 深入剖析了分布式系统因不受控的网络与时序随机性而极难调试的根本痛点，并分享了 FoundationDB 在尚未编写正式数据库前先耗时数年打造确定性仿真框架的逆向研发哲学。为了在单线程中兼顾并发表达力与完全确定性，团队自研了基于 Actor 模型的 C++ 扩展语言 Flow，将网络、磁盘及环境状态全部收拢至伪随机数种子控制之下。通过环形事务不变量、网络阻塞交织（Swizzling）、代码变异（Bugification）以及基于赫斯特指数的重尾级联故障建模，测试框架能以数十万倍于现实的时速挖掘极端并发缺陷。尽管仿真导致传统单步断点调试失效并拉长了日常研发反馈闭环，团队仍辅以物理混沌测试集群 Sinkhole 校验真实掉电边界，最终建立起一套具备全知控制力的高可靠分布式系统工程方法论。 ## Atmosphere & style 整场演讲充满硅谷硬核系统黑客式的自嘲幽默与极度理性的技术坦诚，开篇便将排查分布式 Bug 戏谑为“拿叉子扎眼睛”。Will Wilson 的表达兼具严密的数学哲学思辨与实战中的痛苦反思，既能引经据典拆解水利工程统计学中的赫斯特指数，又能绘声绘色地吐槽在机房里连续烧毁上百个 PDU 电源的狼狈遭遇。现场氛围在揭秘极端故障注入的狂放构想与坦承单步调试报废的无奈妥协之间自然流转，让听众在阵阵欢笑中切身感受到顶尖分布式数据库团队对抗物理世界混乱熵增的敬畏与执着。 ## Bulletin - [04:03](https://www.youtube.com/watch?v=4fFDFbi3toc&t=243s) FoundationDB 采取逆向开发哲学，在真正编写数据库之前先耗费数年打造全确定性仿真系统，在单一物理进程内模拟整个分布式集群及磁盘与网络交互，以此攻克分布式系统不可复现的调试难题。 - [07:16](https://www.youtube.com/watch?v=4fFDFbi3toc&t=436s) 为了在 C++ 中避免多线程带来的非确定性熵，团队自研了 Flow 语言扩展，将高层 Actor 异步语法在编译期重写为纯单线程、回调驱动的状态机执行引擎。 - [24:52](https://www.youtube.com/watch?v=4fFDFbi3toc&t=1492s) 测试框架通过虚拟时间加速、Bugification 变异机制以及赫斯特指数（Hurst exponent）建模多节点级联故障，每晚跑 500 万至 1000 万次模拟，累计完成相当于真实世界数万亿 CPU 小时的测试。 - [30:37](https://www.youtube.com/watch?v=4fFDFbi3toc&t=1837s) 为弥补纯软件模拟对底层硬件与 OS 契约的盲区，团队搭建了由可编程网络电源通宵高频断电的物理集群 Sinkhole；其自身代码未测出缺陷，却抓出了 ZooKeeper 的掉电安全性 Bug 并促使团队彻底自研 Paxos。 ## Key takeaways - [00:01](https://www.youtube.com/watch?v=4fFDFbi3toc&t=1s) 分布式系统调试困难的深层根源并非单纯的业务复杂性，而是未受控的外部物理熵（不可预测的网络时延、丢包和乱序）打破了函数确定性，使时序相关的潜在 Bug 难以在本地复现和验证。 - [04:03](https://www.youtube.com/watch?v=4fFDFbi3toc&t=243s) FoundationDB 坚持“先写确定性仿真，再写数据库”的开发路线，通过在单个物理进程内模拟大规模虚拟节点与环境交互，在动工构建核心存储引擎前就彻底解决可复现性问题。 - [07:16](https://www.youtube.com/watch?v=4fFDFbi3toc&t=436s) 为杜绝多线程操作系统调度引入的非确定性，团队开发了 Flow 语言扩展作为 C++ 的语法超集，将 Actor 模型的高级并发抽象在编译期静态展开为单线程事件循环回调。 - [12:07](https://www.youtube.com/watch?v=4fFDFbi3toc&t=727s) 确定性模拟网络在内存中实现跨平台接口替换，利用显式指定种子的伪随机数生成器（PRNG）模拟延迟、单向静默断开与数据损毁等恶劣故障。 - [13:52](https://www.youtube.com/watch?v=4fFDFbi3toc&t=832s) 任何探查系统当前壁钟时间或物理磁盘剩余空间的调用都会破坏确定性；系统通过对 1% 的模拟任务进行全等双跑并比对 PRNG 最终状态来捕获确定性漏洞。 - [16:01](https://www.youtube.com/watch?v=4fFDFbi3toc&t=961s) 测试用例通过极简的不变量进行验证，例如在每秒 1000 次事务与网络交织（Swizzle）、断电损耗等混沌注入下运行环形键值测试，只需在测试尾声检查环拓扑即可直接证伪 ACID 特性。 - [20:42](https://www.youtube.com/watch?v=4fFDFbi3toc&t=1242s) 为了抗衡海量客户硬件与不可预测操作带来的故障探索速度，模拟器利用虚拟时间加速跨越静止等待，并将现实中数年一次的硬件故障压缩至每两到三分钟发生一次，大幅提升单 CPU 小时挖 Bug 的密度。 - [23:06](https://www.youtube.com/watch?v=4fFDFbi3toc&t=1386s) 团队使用 Bugification 宏在仿真中对数据包顺序、超时事件等合法行为进行随机变异，迫使代码暴露其隐性假设，防范接口在狭窄的标准实现子集下形成脆弱依赖。 - [24:52](https://www.youtube.com/watch?v=4fFDFbi3toc&t=1492s) 现实中硬件故障并非独立的随机试验，而是具备重尾特征的帕累托分布；通过调节赫斯特指数（Hurst exponent），模拟系统能精准还原真实世界中高概率发生的级联关联崩溃。 - [28:58](https://www.youtube.com/watch?v=4fFDFbi3toc&t=1738s) 全模拟虽然让断点调试器因跨节点的单线程回调交织而失效，却让基于重放的精细化 printf 战术成为极其高效的调试利器，排查完补丁后可直接通过完全重放验证问题已被修复。 - [30:37](https://www.youtube.com/watch?v=4fFDFbi3toc&t=1837s) 为了兜底模拟器对 OS 语义理解偏差或未构造极端硬件故障的盲区，FoundationDB 打造了 Sinkhole 物理硬件集群，通过脚本控制可编程 PDU 高频断电重启服务器与交换机。 - [33:00](https://www.youtube.com/watch?v=4fFDFbi3toc&t=1980s) Sinkhole 物理断电测试揪出了 ZooKeeper 的掉电安全性缺陷以及 apt 安装未 fsync 导致配置丢失的问题，这促使团队彻底剔除第三方依赖，基于 Flow 完全从零自研 Paxos 并将其纳入模拟框架管理。 - [38:18](https://www.youtube.com/watch?v=4fFDFbi3toc&t=2298s) 确定性模拟相比于在生产环境跑 Netflix Chaos Monkey 的核心优势在于拥有“全知全能的上帝视角”，能够在 Paxos 等协议的最脆弱时序窗口实施精准计算后的定向打击。 ## Detailed walkthrough ### [00:01](https://www.youtube.com/watch?v=4fFDFbi3toc&t=1s) 分布式系统调试的真正痛点：不受控的外部熵 Will Wilson 开场幽默地指出，尽管所有人都承认调试分布式系统比“拿叉子扎自己的眼睛”稍微强一点，但两者之间的差距其实出奇地小。人们常常把分布式系统难调试归咎于“系统太复杂”，但复杂性并不是真正致命的根源。为了阐明真相，Will 举了两个极简的反例： 在第一个场景中，右侧服务器向左侧服务器发送一个非幂等（non-idempotent）数据包。左侧服务器存在一个潜在 Bug——它没有检查数据包被接收的次数。然而只要网络正常，这个 Bug 永远处于无症状状态。某天，因为交换机配置错误或路由器发疯将包乱甩，这个包迷失在网络的“时间黑洞”中；右侧服务器随后触发了分布式系统常见且期望的超时重传；紧接着，失踪的第一个数据包突然重新冒出并被接收。由于非幂等且缺乏校验，系统瞬间崩溃、数据丢失。当你满头大汗试图在本地复现时，重新跑一遍系统却表现完美；再跑一次依然正常。因为引发崩溃的底层网络抖动极其短暂罕见，极度依赖毫秒甚至微秒级的时序。这种 Bug 无法在本地重现，却注定会在客户生产环境再次爆发；更折磨人的是，即使你写了一段补丁，你也根本无法验证它是否真正修复了问题。 ```mermaid flowchart LR accTitle: 分布式系统中罕见网络时序导致 Bug 爆发的过程 accDescr: 展示非幂等数据包因网络延迟重传后旧包再度出现引发系统崩溃的流程 subgraph Sender["发送端 (Server Right)"] direction TB P1["发送非幂等包 A (初始)"] Retry["触发超时重传包 A2"] end subgraph Network["不可控网络环境"] direction TB Delay["数据包 A 滞留于时间黑洞"] Reappear["数据包 A 突然延迟送达"] end subgraph Receiver["接收端 (Server Left，含无校验Bug)"] direction TB ExecRetry["先处理重传包 A2 (系统正常)"] Crash["再接收迟到包 A (状态损坏，系统崩溃)"] end P1 --> Delay P1 -.->|"超时"| Retry Retry --> ExecRetry Delay --> Reappear Reappear --> Crash ``` 在第二个场景中，右侧服务器先后发送数据包 A 和 B。虽然协议契约并没有规定顺序，但程序员写代码时自然而然会写成 `send(A); send(B);`，没有人会在代码里扔硬币决定发包顺序。这导致左侧服务器长期在巧合下工作，潜意识地假定“A 永远在 B 之前到达”。一旦网络发生乱序，B 先到而 A 后到，整个世界再次崩溃。 Will 指出，表象问题是调试缺乏可重复性，而深层问题在于：我们干净、纯粹的函数世界被混乱肮脏的现实宇宙入侵了。系统的执行历史不再是其输入的纯函数，网络成为了一个完全无法控制的随机性来源（熵）。同样的道理不仅存在于网络，把“网络”替换为“线程”或“磁盘”，并发 Bug 的逻辑完全一致。 - 调试难复现的根源：“The real problem is that the messy dirty Universe has intruded on our beautiful pristine land of pure functions right like the execution history of this system is not a pure function of its inputs there's a source of entropy a source of Randomness which you do not control which is the network.” —— Will Wilson - 数据一致性的代价：对于一般的 Web 服务器或缓存，挂了可以轻松地销毁 VM 重启；但用户对底层数据极其敏感，“人们对自己的数据可没有那么豁达”。 --- ### [04:03](https://www.youtube.com/watch?v=4fFDFbi3toc&t=243s) 逆向开发哲学：先做确定性仿真，再做真实数据库 FoundationDB 面临的核心挑战在于：他们要做的是一个提供 ACID 强事务保证的分布式数据库。ACID 绝不可能凭空意外产生，做错的方式远比做对的方式多得多；更危险的是，一个系统在常规流程下能跑通，但一旦遭遇意外环境就瞬间崩塌的概率，远大于它在异常下依然保持正确并持续稳定的概率。团队因此得出结论：如果不能彻底解决调试和测试的可重复性难题，根本就不该动手去造数据库。 团队采取的策略堪称疯狂：在最初的整整几年里，FoundationDB 甚至没有碰真正的数据库网络和磁盘交互，而是先研发了一套全确定性的仿真系统（Deterministic Simulation）。在这套仿真环境被调试到无可挑剔、穷尽所有异常之后，团队才加上真正的网络与磁盘通信，将它蜕变为一个真实的生产数据库。 这项工程的疯狂之处在于仿真范围：系统不仅要在单个物理进程内模拟单个节点，而是要在一个物理进程中模拟出一整个由大量相互通信的虚拟进程构成的网络，同时完整模拟这些进程与外部环境（磁盘、操作系统、网络拓扑）的所有交互。 实现这一架构需要克服三大理论与工程挑战： 1. **单线程伪并发（Single-threaded pseudo concurrency）**：系统在概念上必须模拟许多并发事件和它们自以为在进行的网络通信，但绝对不能使用真正的操作系统线程或多进程并发。因为真并发会重新引入线程调度带来的非确定性熵，破坏确定性。 2. **模拟所有外部交互**：所有进程与宇宙打交道的方式（磁盘 I/O、网络传输、时钟等）都必须由软件模拟实现，将环境中的所有随机性都转化为测试框架人为、受控注入的随机性。 3. **被模拟代码本身的确定性**：进程自身的业务逻辑必须是完全确定性的，不能调用未受控的外部系统状态。 - 研发初期的激进决策：“The thing that that the company did was we didn't write a database we started by writing a simulation of a database a totally deterministic simulation and then once we'd exhaustively debugged that we were like okay now we can write a database which is just that plus talking on networks and talking to discs for real.” —— Will Wilson --- ### [07:16](https://www.youtube.com/watch?v=4fFDFbi3toc&t=436s) 打破 C++ 回调地狱：Flow 语言扩展的诞生 在攻关“单线程伪并发”时，团队陷入了艰难的技术选型困境。因为确定性要求，多线程方案被彻底排除；协程在当时存在诸多局限；剩下的唯一底层手段只有“事件回调（Callbacks）”。另一方面，为了追求极致的数据库吞吐与低延迟性能，系统必须使用 C++ 编写。 然而，“C++ 加上纯回调函数无异于人间地狱（a nightmarish hellscape）”，代码的可读性和逻辑推理性会彻底粉碎。团队需要的是高级抽象原语，既能让开发者像写顺序代码一样思考并发，又能在底层编译为单线程、纯回调驱动的状态机。 为此，FoundationDB 创造了 **Flow**。Flow 介于库和新语言之间，是 C++ 的语法超集（附带一个目标代码为 C++ 的预处理编译器）。Flow 引入了基于 Actor 模型的并发语法糖，但物理执行完全由单线程调度器和回调支撑。 ```mermaid flowchart TD accTitle: Flow 编译与执行模型 accDescr: 展示 Flow 源代码通过预编译生成纯 C++ 回调状态机并在单线程调度器中执行的过程 subgraph FlowSource["Flow 源代码 (.actor.cpp)"] ActorCode["actor 语法糖定义<br/>高层并发逻辑 (wait / future)"] end subgraph FlowCompiler["Flow 预处理器 / 编译器"] CodeGen["静态分析并展开状态机<br/>生成对应辅助类与回调函数"] end subgraph OutputCpp["标准 C++ 源码"] direction TB FuncEntry["入口函数: 立即返回 Future&lt;T&gt;"] StateClass["生成的 Actor 状态机类<br/>封装状态参数与单线程回调逻辑"] end subgraph Execution["单线程执行引擎"] EventLoop["单线程事件循环 (Event Loop) 调度回调"] end ActorCode --> FlowCompiler FlowCompiler --> CodeGen CodeGen --> FuncEntry CodeGen --> StateClass FuncEntry -.->|"实例化"| StateClass StateClass --> EventLoop ``` Will 现场展示了一个极简的 Flow 示例——一个异步加法器（Asynchronous Adder）： ```cpp ACTOR Future<float> asyncAdd(Future<float> f, float b) { float a = wait(f); return a + b; } ``` 针对这段代码，Will 指出了两个初看极不寻常的特征： 1. **非标准关键字 `ACTOR`**：这并非标准 C++，直接传给标准编译器会报错。它向 Flow 编译器标明这是一个需要被展开为 Actor 状态机的函数定义。 2. **表面上的“类型不匹配”**：签名声明返回 `Future<float>`，但函数体内部却直接写了 `return a + b;`（一个普通的 `float`）。这是因为该函数体并不是最终运行的代码，而是一种声明式描述。Flow 编译器会将其重写为一个崭新的外部函数和一个内部状态机类；当调用 `asyncAdd` 时，外部函数会立刻返回一个 `Future<float>`，该 Future 会在整个异步等待链条执行完成、求值结束的一瞬间就绪（Ready）。 在底层展开的代码中，Flow 会重写原函数，给输入参数加上 `const` 修饰符，并将参数打包传入自动生成的内部状态机类构造函数中，以单线程事件回调的机制完成异步驱动。 - 对底层回调方案的评价：“Callbacks plus C++ is like a nightmarish hellscape.” —— Will Wilson ### [09:42](https://www.youtube.com/watch?v=4fFDFbi3toc&t=582s) Flow 编译器实现与基准测试的局限 Will Wilson 详细解释了 Flow 编译器在幕后所做的工作。当一段 Actor 代码被编译时，编译器会完全自动生成一个 C++ 类，且该类的调用会立即返回。Actor 的主体逻辑被拆解并分散在该类的多个方法中：每当遇到等待异步操作结束的 `wait` 语句时，当前方法就会在此处终止，并在对应的 Future 上设置一个回调函数。该回调函数直接指向 Actor 的下一个逻辑片段。随后，它会立即向一个中央调度器（centralized scheduler）让出控制权（yield），以便程序的其他部分能够继续推进。当整个 Actor 的生命周期走到尽头时，它会利用手中持有的 Promise 去履行（fulfill）最开始对外暴露的那个 Future，最后执行对象的自我清理和销毁。 这种设计的优势在于，程序员可以采用完全直观的 Actor 语法编写逻辑，而所有操作都在单线程中执行，同时保持极高的异步并发性与运行效率。 ```mermaid flowchart TD accTitle: Flow 编译与回调流转图 accDescr: 展示 Flow 编译器如何将 Actor 方法拆分并通过中央调度器在单线程中执行异步流转 subgraph Compiler["Flow 编译与生成"] A["Actor 源代码"] -->|"编译生成"| B["C++ 包装类"] B -->|"初始化并立即返回"| C["Future / Promise 对"] end subgraph Runtime["单线程调度执行"] D["Actor 第一段方法"] -->|"遇到 wait"| E["在 Future 上注册回调"] E -->|"Yield 让出执行权"| F["中央调度器"] F -->|"触发回调"| G["Actor 后续方法"] G -->|"完成执行"| H["履行 Promise 并销毁自身"] end C -.-> D ``` Will 随后展示了系统的性能数据，但他坦言大家应当“对其持巨大保留态度”（take with a gigantic grain of salt）。这项测试采用了由前一位演讲者 Joe Armstrong 发明的“环形基准测试”（Ring Benchmark）：创建 $N$ 个 Actor 排成一个闭合环，在它们之间依次传递消息并循环 $M$ 次。关于为何不应盲目相信这些对比数据，Will 幽默地环顾会场并解释了两点： > “希望 Joe Armstrong 不在这里，这其实算是一个挺傻的基准测试。” —— Will Wilson 首先，除 Flow 之外的其他测试数据均来自网络上其他人针对各自系统运行的公开结果，虽然假设对方已经尽力优化，但实际运行环境无法考证；其次，环形测试过于简陋，根本无法真实反映分布式 Actor 系统的实际负载。在实践中，当 Actor 内部逻辑越来越复杂、引入更多异步调用时，Flow 的相对性能优势会出现大幅下降。这也是团队着手开发 Flow 2 的主要动力，希望藉此进一步压榨运行效率。 - 环形基准测试逻辑：$N$ 个 Actor 组成环形链路，消息在节点间顺次传递，总共循环传递 $M$ 次。 - 架构现实差异：简单的基准测试无法模拟现实中 Actor 高度异步与复杂状态变更的场景。 --- ### [12:07](https://www.youtube.com/watch?v=4fFDFbi3toc&t=727s) 确定性模拟的构件：网络抽象与随机故障注入 脱离了定制语言编译器后，Will 转向构建确定性模拟框架的核心要素。由于 FoundationDB 在设计之初就是一个跨平台系统，底层与磁盘、网络等硬件交互的接口早已抽象完毕。因此，团队不需要重构系统，只需在原有的 Unix 网络实现与 Windows 网络实现之外，再平行插入一套“模拟网络”（simulated network）代码。 在模拟网络环境下，建立网络连接不会产生实际的套接字调用，而是实例化一个维护内部状态和缓冲区的 C++ 类。当节点发送数据时，该类会在发送端插入延迟模拟传输耗时，随后在接收端模拟排队延迟，再完成字节的内存拷贝，最后通知接收端有数据可读。 在模拟的 `read` 方法中，团队埋入了一个名为 `roll random cloth` 的故障注入机制，用来逼出潜在的代码缺陷。每次尝试从模拟网络读取数据时，系统都会抛掷随机数。如果遭遇“倒霉”的随机结果，系统会继续连环掷骰子触发各种恶劣场景：或是对端直接挂断，或是本端主动断开，亦或是连接发生未知错误并向读取者抛出异常。Will 直言： > “通过这种机制，我们试图排查系统里所有假设网络是完全可靠的代码，而做这种假设是一件非常愚蠢的事。” —— Will Wilson - 跨平台抽象：利用现成的平台接口分离层，无缝接入模拟驱动。 - 模拟网络机制：纯内存缓冲区实现，精准模拟发送侧延迟、接收侧延迟以及数据搬运。 - 故障注入类型：随机连接中断、单向静默断开、底层读取错误等非确定性网络事故。 --- ### [13:52](https://www.youtube.com/watch?v=4fFDFbi3toc&t=832s) 确定性的达成：伪随机控制与 1% 重复运行检验 要实现整套模拟框架，最后一道也是最具挑战性的关卡是“确定性”（determinism）。表面上看，让程序具备确定性似乎理所当然，因为开发者通常不会故意编写非确定性的代码，但这种隐患在日常工程中无处不在。最普遍的例子是随机数——例如指数退避（exponential backoff）算法中使用的随机抖动。若要保证确定性，必须确保系统内所有随机数都源自一个开发者完全可控、显式指定种子的伪随机数生成器（PRNG），而该种子本身则作为整个测试程序的初始输入参数。 更为隐蔽的陷阱存在于系统对环境状态的嗅探中。代码中如果出现类似“在 `if` 条件中检查系统当前时间”或者“查询当前磁盘剩余空间”的操作，确定性就会瞬间被粉碎；即使两次运行的输入完全一致，系统的执行轨迹也会彻底分道扬镳。Will 强调，尽管团队内的工程师都接受过严格训练以识别并规避这些代码习惯，但在庞大的工程中依然会屡屡失手。 为了捕获确定性破坏（determinism failure），FoundationDB 引入了一道防线：在所有模拟测试中，抽取大约 1% 的测试任务执行两次。这两次运行输入完全相同的种子和外部事件序列。在运行结束的最后一刻，系统会去比对伪随机数生成器预备生成的“下一个随机数”。如果两个副本输出的值不完全匹配，就证明两次执行在某个中间步骤的分支判断上产生了分歧，进而触发确定性告警并由开发人员追踪定位。虽然这无法穷尽所有确定性漏洞，但确保了违背确定性的逻辑能在早期被揪出。 - 时间与环境非确定性：禁止读取真实的壁钟时间（wall-clock time）、实时磁盘容量或操作系统硬件计数器。 - 种子驱动：将所有非确定性源头收拢至单一 PRNG 种子，使其构成可重放的程序输入。 - 1% 双跑校验机制：在完全相同的参数下跑两遍，运行结束时校验内部 PRNG 状态是否一致。 --- ### [16:01](https://www.youtube.com/watch?v=4fFDFbi3toc&t=961s) 测试用例编排：环形事务不变量与混沌故障场景 当 Actor 运行时、模拟网络与确定性环境三者合一后，系统就获得了在纯软件环境中完全确定性地模拟整个分布式集群交互的能力。团队通过声明式测试文件来挖掘 Bug。一份测试文件核心包含两类要素：系统要达成的目标（负载），以及试图阻止系统达成目标的各类干扰（故障注入）。 在负载方面，测试声明了一个“环形测试”（cycle test）。系统首先在数据库中写入一个闭环的键值对链表（例如 Value 1 指向 Key 2，以此类推），然后以每秒 1000 个事务的并发速率施加负载。每个并发事务都会对数据库中的键值进行修改，但受制于一个全局约束——事务在整体逻辑上必须保持环的完整性，允许改变节点顺序或位置，但链环不能断裂。 ```mermaid flowchart LR accTitle: 环形测试不变量校验逻辑 accDescr: 展示环形键值链路在受到并发事务突变并经历故障注入后，最终通过链环完好性验证 ACID 的过程 A["K1 -> V1 (指向 K2)"] --> B["K2 -> V2 (指向 K3)"] B --> C["K3 -> V3 (指向 K1)"] subgraph Transactions["并发事务变异 (1000 TPS)"] T["原子性 / 隔离性突变"] end A -.-> T B -.-> T C -.-> T T -->|"保持环结构"| PASS["验证通过: ACID 未被破坏"] T -->|"断环 / 伸长 / 缩短"| FAIL["报错: 出现原子性破损或隔离级别冲突"] ``` 这一设计的精妙之处在于提供了一个极易验证的不变量（invariant）：如果在测试结束后环发生断裂、变长或变短，就直接证明 ACID 遭到了破坏——要么是事务部分写入导致原子性（Atomicity）破损，要么是事务并发覆盖破坏了隔离性（Isolation）。在这个测试中，由于后续施加了极度严苛的故障注入，团队甚至只预期能成功提交 1% 的事务。 而测试文件中用于破坏系统达成目标的干扰手段则极为残暴，主要包含三类： 1. **随机阻塞（Clogging）与交织（Swizzling）**：系统会伪随机且确定性地阻断部分模拟网络连接，使数据包无法收发。测试中启用了名为“Swizzle”的增强模式。Will 承认： > “交织（Swizzling）基本上可以看作是阻塞（Clogging）的升级版。它会选取一部分网络连接按滚动方式截断，随后以相反的顺序恢复它们；尽管我们完全不理解背后的深层原因，但它在挖 Bug 方面就是比普通阻塞有效得多，这其实挺让人害怕的。” —— Will Wilson 2. **硬件损耗（Attrition）**：在整个 30 秒的测试周期内，系统会随机切断机器电源并在后续恢复，期间总计杀死 10 台机器，任何时刻仅保留 3 台处于开机状态，借此检验集群在极端断电场景下的安全性。 3. **协同变更（Coordination Change）**：这是整个配置中最具破坏性的操作。系统会在混乱流水线中动态触发数据库配置变更，强迫集群在面临网络链路被大面积交错切断、机器接二连三断电重启的极端恶劣环境下，强行执行 Multi-Paxos 共识协议来完成协调节点的交接。 - 环形不变量设计：通过简单的环拓扑检查，无需复杂的外部比对即可在测试尾声直接证伪 ACID 特性。 - 事务预期：在严重干扰下，即便 99% 的事务失败或回滚，只要通过的 1% 满足一致性即算系统正确。 - 复合故障场景：网络阻塞颠倒（Swizzle）、大规模断电重启（Attrition）与一致性协议迁移（Multi-Paxos reconfiguration）在毫秒级时序内并发交织。 ### [19:11](https://www.youtube.com/watch?v=4fFDFbi3toc&t=1151s) 模拟极端故障场景与“愚蠢系统管理员” 在每秒处理上千次事务的高并发压力下，FoundationDB 的仿真框架正在不断寻找 Paxos 协议实现中的潜在漏洞。Will Wilson 提到，系统所做的测试远不止常规的断网或关机，他们还会注入极其恶劣的渐进式故障。例如，仿真环境会把某些机器置于一种“缓慢死亡”的状态——所有后续系统调用（system calls）都有大约 1% 的概率返回错误。通过这种方式，测试团队可以验证集群是否足够智能，能否及时检测出异常节点、对其进行隔离并在其影响扩散到整个集群之前将其销毁，同时也以此检验系统的优雅停机（graceful shutdown）等特性。 此外，FoundationDB 本身具备“位置感知”（locality）的概念，能够识别机架拓扑结构以及广域网（WAN）链路的位置，而仿真环境完全继承并理解这些信息。这让团队可以轻松模拟“瞬间摧毁整座数据中心”的灾难级场景，以检验数据是否真正具备持久性和高可用性。Will Wilson 还幽默地表示，他们还会模拟现实中“绝不可能存在”的“愚蠢系统管理员”（dumb sysadmins）： > “Like one of the most fun parts of my job is thinking up new and creative ways of torturing our database and then implementing that in simulation.”（我工作中最有趣的部分之一，就是想出各种新颖且极具创意的方法来折磨我们的数据库，然后在仿真中实现它们。） 在这些模拟管理员失误的场景中，仿真框架能够： - 瞬间原子性地互换两台物理机器的 IP 地址，观察集群路由和连接状态如何应对。 - 瞬间将两台计算机上的数据文件互相拷贝覆盖，或者直接在两者之间对换硬盘，检验这种混乱操作是否会导致数据损坏或丢失。 --- ### [20:42](https://www.youtube.com/watch?v=4fFDFbi3toc&t=1242s) 核心挑战：如何比客户更快地遍历状态空间？ 尽管仿真框架极其强大，但确定性仿真测试面临一个本质上的严峻挑战：测试实验室中的计算资源是有限的，测试机群在努力探索所有可能输入与故障模式组成的状态空间；与此同时，真实世界中的客户也在探索同样的空间。Will Wilson 形象地描述了客户的运行环境——他们往往在充满 Bug、容易崩溃、严重超配/欠配（underprovisioned）且配置完全错误的硬件上运行分布式系统，并且在上面执行各种匪夷所思的疯狂操作。最关键的是，一旦系统出现问题，“顺便说一句，如果系统坏了，那就是你的错”。 由于客户的数量及其拥有的机器规模通常比研发实验室多出数个数量级，开发者面临的核心命题是：**如何确保自己在测试环境中遍历错误空间的速度，远快于真实客户遇到这些错误的速度？** > “Another way of thinking about this is you need to find more bugs per CPU hour than the real world by many orders of magnitude, which is hard.”（换个角度来想，你必须比现实世界每 CPU 小时多发现数个数量级的 Bug，这极其困难。） ```mermaid flowchart TD accTitle: 仿真测试与生产环境探索故障空间的对比 accDescr: 比较测试实验室有限资源与生产环境海量客户硬件在探索故障空间上的差异 subgraph LAB["研发测试实验室"] L_MACH["有限的测试集群"] --> L_EXPLORE["确定性仿真探索"] L_EXPLORE --> L_GOAL["目标：提升每 CPU 小时发现 Bug 概率"] end subgraph PROD["生产真实世界"] C_USERS["海量客户群体"] --> C_HARDWARE["错误/欠配/配置混乱的硬件"] C_HARDWARE --> C_CHAOS["执行极端、不可预测的操作"] C_CHAOS --> C_EXPLORE["极速遍历故障空间"] end L_GOAL -.->|"必须大幅领先以提前捕获 Bug"| C_EXPLORE ``` 为了战胜客户环境在规模上的压倒性优势，FoundationDB 采用了多种提升 Bug 触发密度的工程策略： - **大幅提高故障频率**：现实世界中，重负载下的硬盘可能两到三年才损坏一次；而在仿真中，系统让硬盘每两到三分钟就损坏一次。这意味着测试环境单位 CPU 小时内经历的磁盘故障数量比真实世界高出几个数量级，从而能更快暴露磁盘故障处理代码中的缺陷。 - **虚拟时间加速（Speed up time）**：分布式集群在许多时候处于静止等待状态，或者在从故障中恢复时需要等待固定的硬编码超时秒数。在单核计算性能允许的情况下，仿真引擎会直接快进虚拟时钟，让一秒真实时间内流逝大量的“仿真秒”，极大压缩无效等待时间。 --- ### [23:06](https://www.youtube.com/watch?v=4fFDFbi3toc&t=1386s) 变异测试技术：Bugification 除了加快时钟和提升硬件故障率，FoundationDB 采用的另一项关键技术是团队内部称为 **“Bugification”**（Bug 化）的机制。Will Wilson 回顾了早先提到的案例：许多分布式 Bug 属于“无症状 Bug”，即两端代码在契约理解上从一开始就存在分歧，但必须等待某种极其罕见的物理条件才会显现。为了不必被动等待极端罕见的极端事件发生，团队打破了“程序员从不故意改变代码逻辑”的常规，在仿真中主动对代码行为注入随机变异。 `bugif` 本质上是一个宏定义，其工作逻辑是：当系统处于仿真运行状态时，以一个很小的随机概率触发非标准行为（内部结合行号哈希等机制）。通过这种手段，系统可以强制接口暴露出更多潜在合法的边缘行为，防止组件之间的依赖建立在“某个狭窄的常规实现子集”之上。 代码变异的典型应用场景包括： - **颠倒数据包发送顺序**：若常规逻辑是依次发送数据包 A 和数据包 B，宏判定命中后会写成 `if (bugif) { send(B); send(A); }`，以验证接收方是否严格遵循乱序到达的协议规范，而非暗中假设了先后顺序。 - **不完全发送**：在某次网络写入就绪时，故意保留一部分数据不发，仅发送缓冲区中的片段。 - **永久丢弃超时事件**：在判定为 Bugification 运行时，完全抑制超时信号的发送，检验对端是否错误地将超时作为必须到达的控制流依赖。 --- ### [24:52](https://www.youtube.com/watch?v=4fFDFbi3toc&t=1492s) 赫斯特指数与级联故障建模 Will Wilson 引入了他极为热衷的一个数学与工程概念——**赫斯特指数（Hurst exponent）**。该概念最早发表于 1951 年《美国土木工程师学会会刊》（*Transactions of the American Society of Civil Engineers*）中由 Harold Edwin Hurst 撰写的论文《水库的长期蓄水能力》（*Long-term storage capacity of reservoirs*）。作为一名被派往埃及研究尼罗河大坝与水库的英国公职人员，Hurst 发现上一代工程师的设计原则表面上看起来很合理：水库的设计指标是足以抵御“百年一遇的洪水”与“百年一遇的干旱”。然而在实际运行中，这种所谓的“百年一遇”灾害大约每 15 年就会发生一次。 问题根源在于，早期的工程师（以及 2008 年金融危机前的许多金融从业者）错误地假设每天“下雨”或“不下雨”是完全独立的随机试验，事件之间互不相关。但在现实中，若周二下雨，周三下雨的概率显然高于普通基准概率——事件绝非相互独立。高斯正态分布基于独立试验假设，其尾部衰减极快；而赫斯特指数正是用来量化这种“非独立性”的工具。当赫斯特指数大于 0.5 时，系统服从具有“重尾”（heavy tails）特征的稳定帕累托分布（stable Paretian distributions，其参数 $\alpha$ 正好是赫斯特指数的倒数，高斯分布对应 $\alpha=2$ 且赫斯特指数为 0.5）。重尾分布意味着现实中遭遇极高 $\sigma$（标准差）极端事件的概率，远超朴素正态分布的预期。 > “The answer is hardware failures are not independent random events. Okay? You all know this in your guts, right? Like if a hard drive fails in a rack, what is the first thing you do? Oh my god, check every other hard drive in that rack!”（答案是：硬件故障并不是独立的随机事件。大家打心底里都知道这一点，对吧？比如机架里有一块硬盘坏了，你的第一反应是什么？天哪，赶紧检查那个机架上的所有其他硬盘！） 一块硬盘故障，可能意味着整批硬件质量有瑕疵、机房维护操作不当或者数据中心湿度异常。多节点级联故障在统计学上极为常见，但在物理硬件测试中极难复现与构造。而在确定性仿真环境中，FoundationDB 可以自由调节赫斯特指数，人为制造大量高密度的多节点级联关联崩溃，以此测试系统在真实极端环境下的生存韧性。 - **庞大的测试规模**：结合时间加速、故障注入与 Bugification，FoundationDB 估算他们已经执行了相当于真实世界**数万亿 CPU 小时**的测试。 - **日常回归吞吐量**：利用测试集群，团队每晚运行 **500 万到 1000 万次** 独立的仿真测试。 - **场景跨度**：每次仿真测试模拟整个分布式集群在 5 分钟到 1 到 2 个小时内的完整运作过程，该机制已稳定运行多年。 --- ### [28:58](https://www.youtube.com/watch?v=4fFDFbi3toc&t=1738s) 仿真带来的代价：回调地狱与调试灾难 在确定性仿真带来巨大测试收益的同时，Will Wilson 话锋一转，指出这也带来了一个极其棘手的负面代价：“我们彻底毁掉了调试”（we've totally broken debugging）。 在传统的异步编程模型中，代码中往往充斥着层层嵌套的事件回调函数（callbacks），在单机环境下单步调试这种代码本就极其痛苦。如果在一个完全由事件驱动、内部全由回调交织而成的确定性仿真系统里，用传统断点调试器去排查由多个“充斥着回调的怪物节点”组成的分布式死锁或状态错误，其复杂度与认知负荷将呈指数级爆炸。常规单步调试工具面对这种高度交织的事件仿真基本处于失效状态，必须寻求全新的架构与语言层支持来解决这一困局。 ### [29:25](https://www.youtube.com/watch?v=4fFDFbi3toc&t=1765s) 调试困境与基于确定性模拟的“printf 战术” 当所有节点与模拟器逻辑全都塞进同一个进程时，传统的交互式调试变得极其反人类。不仅有多个相互协作的节点在同一个挂载的进程里共存，外层还包裹着整套竭力欺骗节点“它们正通过网络向彼此通信”的模拟框架代码。在调用栈里逐帧单步跟踪极为艰难，Will Wilson 坦言团队里虽然有两三位神级工程师能硬生生在栈里向下深潜两三层单步单步排查并让全组叹为观止，但对其余普通工程师而言根本不可行。 最终剩下的调试利器退化到了看似最原始的手段——打日志（printf）。Will Wilson 直白地承认： > “So what you're left with is basically printf, which sounds pretty horrible and barbaric, and I'll be honest it is, but it's actually not that bad when you have deterministic simulation.” 在具备确定性模拟的前提下，printf 的威力被彻底释放。只要用完全相同的输入重新跑模拟，所有事情就会在完全相同的时刻、以完全相同的顺序发生。工程师不需要打印海量垃圾日志，而是可以针对“已知下次必定触发”的精确前置条件添加针对性日志，一层层将 printf 推进到代码深处，像步步紧逼一样把 Bug 逼入死角并彻底剿灭。修完之后再次回放，还能立即验证 Bug 是否真的不再重现。 --- ### [30:37](https://www.youtube.com/watch?v=4fFDFbi3toc&t=1837s) 模拟框架的盲区与残酷的物理集群 “Sinkhole” 确定性模拟虽然强大，但依然存在一种终极噩梦：模拟本身是错误的。这通常来自两个方向： 1. **残酷程度不够**：现实世界中存在某种极其诡异的网络或硬件故障模式，但模拟器从未构造过这种模式。 2. **对操作系统契约理解有误或 OS 本身有 Bug**：比如团队自以为理解了 `fsync` 的语义，但实际行为存在出入；或者跨平台移植时，同名系统调用在不同 OS 上的底层保证存在微妙差异。 为了捕获这类超出模拟器认知的真实漏洞，FoundationDB 打造了一个专门的后备物理测试集群——**Sinkhole**。该集群由多块配备 ECC 内存的小型服务器主板组成，彼此通过五类线（CAT 5）和交换机互联。最致命的设计在于：交换机和所有主板全部插在可编程网络电源（PDU）上，由脚本控制整夜对整套集群疯狂切断和恢复电源。 ```mermaid flowchart TD accTitle: Sinkhole 物理混沌测试集群架构 accDescr: 展示可编程电源控制主板与交换机断电重启以测试掉电安全性的硬件拓扑架构 subgraph PowerCtrl["自动化控制层"] PDU["可编程网络电源<br/>(PDU / 脚本轮流断电)"] end subgraph HardwareCluster["Sinkhole 硬件环境"] SW["网络交换机<br/>(模拟物理网络断联)"] MB1["真实服务器主板 1<br/>(ECC RAM + SSD)"] MB2["真实服务器主板 2<br/>(ECC RAM + SSD)"] MB3["真实服务器主板 3<br/>(ECC RAM + SSD)"] end subgraph StackUnderPower["被测对象"] FDB["FoundationDB 数据库进程<br/>(验证掉电安全性)"] end PDU -.->|"高频断电 / 烧毁风险"| SW PDU -.->|"高频断电 / 极损 SSD"| MB1 PDU -.->|"高频断电 / 极损 SSD"| MB2 PDU -.->|"高频断电 / 极损 SSD"| MB3 SW --- MB1 SW --- MB2 SW --- MB3 MB1 & MB2 & MB3 --- FDB ``` 尽管硬件损耗极其惨烈——市面上的可编程电源质量堪忧，连续开关两天就会报废烧毁，团队为此用坏了上百个电源，高速断电对 SSD 也是极具破坏性的损耗——但在 FoundationDB 数据库自身的代码上，Sinkhole 从未炸出过任何 Bug。这反向证明了模拟器在覆盖系统边界上做得相当出色。 - **硬件配置**：小型真实服务器主板、ECC 内存、千兆网线与交换机，全部由可编程网络 PDU 供电。 - **运行方式**：数据库跑负载的同时，全自动脚本通宵交替切断/接通主板与交换机电源。 - **硬件战损**：网络可编程电源几天就烧掉一批，损耗了上百台；SSD 在高频掉电下故障率极高。 - **测试战果**：在 FoundationDB 自身代码中从未发现过掉电安全 Bug。 --- ### [33:00](https://www.youtube.com/watch?v=4fFDFbi3toc&t=1980s) 第三方依赖的惨痛教训：弃用 ZooKeeper 与 APT 的“fsync 惨案” 尽管自身代码抗住了掉电考验，但 Sinkhole 却无情揪出了第三方软件的缺陷。非 Flow 语言编写、无法进入确定性模拟的外部代码，构成了系统的致命盲区。 在 FoundationDB 早期版本中，团队曾引入 ZooKeeper 来维护集群协调状态。在 Sinkhole 的早期雏形测试中，他们直接抓到了 ZooKeeper 的一个严重掉电安全性 Bug。虽然上报后 Apache 官方修复了该问题，但这段惊魂经历让团队彻底丧失了对外部依赖的安全感。Will Wilson 提到，这一创伤促使他们做出激进决定：直接把 ZooKeeper 彻底剥离出技术栈，完全基于 Flow 从零编写了一套自研 Paxos 实现，将其完全置于确定性模拟的保护伞之下；顺带还省去了部署和维护 Java 运行时的巨大麻烦。 另一次荒谬的事故发生在包管理工具上。团队在 Sinkhole 的多台机器上安装数据库并启动运行，接着对集群进行疯狂断电。重启后，其中一个进程竟然无法拉起。排查后发现，底层数据文件由于正确调用了持久化刷盘与日志，完好无损地躺在磁盘上；然而数据库的配置文件却彻底损坏或丢失了。罪魁祸首竟然是 Debian/Ubuntu 的包管理器 `apt` 在写入配置时未调用 `fsync`。进程重启后判定“没有配置文件却有一堆数据文件”，误以为状态错乱而主动 Panic。 --- ### [34:31](https://www.youtube.com/watch?v=4fFDFbi3toc&t=2071s) 人机关系的异化与打破研发“心流”的十万次模拟 在技术分享的收尾阶段，Will Wilson 探讨了分布式测试在面对天文数字般的故障状态空间时，工程团队面临的深层困惑与未来方向。他幽默地感慨技术开发似乎进入了某种“人机关系的新阶段”： > “I feel like we've entered like a new phase in the relationship between man and machine, because if you look at the workflow of one of us, it looks like: you make some changes to the program, you recompile, and then you run like 10,000 simulations to see if you did something obviously dumb, and then you hope that if you did something subtly dumb, you'll get caught that night when we run 10 million.” 每次写完几行代码，跑 10,000 次模拟需要等上一两分钟甚至更久。这彻底打破了程序员沉浸式的 Coding 状态（Flow state）。很多人之所以喜欢解释型语言，就是为了省去哪怕几秒钟的编译时间，而 FoundationDB 却把单次调试循环拉长到了数分钟之久。Will Wilson 指出，增加更多硬件资源也许能显著压缩等待时长，从而反常识地大幅拯救工程师的日常生产力。 此外，为了防范模拟器自身的“盲点”引发类似抗药性的现象，团队还构想过以下手段： - **专职 Bug 注入工程师**：雇佣专人向代码库故意植入隐秘 Bug，测试它们能否穿透模拟框架；若能穿透，则视为对模拟器框架本身的 Bug 报告。 - **双重模拟框架**：建立两套原理各异的模拟器。一套用于日常敏捷开发回归，另一套专门在发布前夜运行上千万次模拟测试，杜绝单一模拟器模式下的“抗药性”。 - **加码真实硬件测试**：虽然面对硬件极其痛苦（有同事戏言“编程除了跟网络、磁盘和电源打交道的部分很好玩，其余全都很痛苦”，结果现在整天全在摆弄硬件和被烧毁的电源），但适度增加像 Sinkhole 这样的真实物理混沌注入，依然是弥补纯软件模拟盲区不可或缺的防线。 --- ### [38:18](https://www.youtube.com/watch?v=4fFDFbi3toc&t=2298s) 现场问答：模拟器对抗 Netflix Chaos Monkey 与确定性的“全知全能” 在演讲后的问答环节，有观众提问：这种测试方式与 Netflix 的混沌工程（Chaos Army / Chaos Monkey）有何不同？是否考虑过直接在生产环境中制造混乱？ Will Wilson 首先重申了架构基础：模拟测试运行的正是纯粹的**生产环境核心代码**，唯一的区别是剥离了真正对接物理磁盘与物理网络的底层系统调用胶水层。在比较确定性模拟与生产环境混沌测试的优劣时，Will Wilson 深刻剖析了模拟测试独有的上帝视角： > “The benefit of simulation is you can do many, many more different types of things, and you can generally be far more brutal and far more like calculatedly evil. Like in a real production environment, you don't have like omnipotent evil Godlike power to like stop and start network connections in the most inconvenient possible way, or to like do what you know is like the hard case for Paxos and like deliberately do that to try and screw with it.” 在真实生产环境搞破坏，系统无法拥有这种“全知全能的邪恶力量”，去在最刁钻的时间点精准掐断网络，或者专门针对 Paxos 的极限 Corner Case 实施定向破坏。然而真实物理测试的优势在于能发现软件模型未曾预料的硬件异常，生产客户所使用的特定磁盘、网卡也可能存在未知的固件隐患，这恰恰是软件模拟无法感知的。因此两种方案各有千秋，兼收并蓄才是最稳健的体系。会议在对 Hurst 统计学指数相关问题的简短交谈与全场致意中落下帷幕。 ## Notable quotes > [24:52](https://www.youtube.com/watch?v=4fFDFbi3toc&t=1492s) The answer is hardware failures are not independent random events. Okay? You all know this in your guts, right? Like if a hard drive fails in a rack, what is the first thing you do? Oh my god, check every other hard drive in that rack! — Will Wilson > [20:42](https://www.youtube.com/watch?v=4fFDFbi3toc&t=1242s) Another way of thinking about this is you need to find more bugs per CPU hour than the real world by many orders of magnitude, which is hard. — Will Wilson > [07:16](https://www.youtube.com/watch?v=4fFDFbi3toc&t=436s) Callbacks plus C++ is like a nightmarish hellscape. — Will Wilson ## Mentioned Will Wilson, FoundationDB, Flow, Joe Armstrong, Multi-Paxos, Paxos, Harold Edwin Hurst, Sinkhole, Apache ZooKeeper, Debian, Ubuntu, APT, Netflix, Chaos Monkey ## Video description Debugging highly concurrent distributed systems in a noisy network environment is an exceptionally challenging endeavor. On the one hand, evaluating all possible orders in which program events can occur is a task ill-suited to human cognition, rendering a pure analytic understanding of the control flow of such a system beyond the reach of any individual programmer. On the other hand, a more “empirical” approach to the task is also fraught with difficulty, as the dependence of severe bugs on precise timings or transient network conditions makes every part of the debugging cycle – from bug replication to verification of a fix – a Sisyphean labor bordering on the impossible. One approach which has been developed to ameliorate this situation is that of deterministic simulation, wherein the hardware components of the system – including hard disks, network links, and the machines themselves – are replaced in testing with software which fulfills the contracts of those systems, but whose state is completely transparent to the developer. This enables the simulation of a wide diversity of failure modes including network failures, disk failures or space exhaustion, unexpected machine shutdown or reboot, IP address changes, and even entire datacenter failures. Moreover, once a particular pattern of failures has been identified which uncovers a bug, the determinism property of the simulation means that the exact same series of events can be replayed an indefinite number of times, greatly facilitating the debugging process, and providing confidence when a bug has been fixed. Attendees of this talk will gain an understanding of the benefits, drawbacks, and tradeoffs involved in implementing a deterministic simulation framework, with frequent reference both to theory and to real-world engineering experience gleaned from applying this method to a complex distributed system. Attendees will also learn about language features which aid in the development of such a framework. Will Wilson FoundationDB Will Wilson works on the engineering team at FoundationDB (https://foundationdb.com). Will started his career in biotechnology, leading a successful R&D effort in spinal cord injury diagnostics, currently undergoing commercialization by a company he co-founded. Since then, Will has worked in a variety of technical and business roles at data science and data virtualization startups. Will has a degree in math and philosophy from Yale. [Source](https://www.youtube.com/watch?v=4fFDFbi3toc)

Source quotes

[24:52] Will Wilson: The answer is hardware failures are not independent random events. Okay? You all know this in your guts, right? Like if a hard drive fails in a rack, what is the first thing you do? Oh my god, check every other hard drive in that rack! [20:42] Will Wilson: Another way of thinking about this is you need to find more bugs per CPU hour than the real world by many orders of magnitude, which is hard. [7:16] Will Wilson: Callbacks plus C++ is like a nightmarish hellscape.

Source quotes

[20:42] Will Wilson: your customers are running your distributed systems on buggy crashy failey grossly underprovisioned totally improperly configured Hardware and they're doing insane things to it by the way and PS if it breaks it's your fault.

Original transcript

[0:01] all right good morning everybody uh so [0:04] my name is Will I work at Foundation DB [0:07] if you haven't heard of us we make a [0:08] scalable and fault tolerant database [0:10] with acid transactions I'm not here to [0:12] talk about that today though uh instead [0:14] I'm going to talk about simulation [0:16] testing and why it might make your life [0:19] easier so I want to start with an [0:21] observation which I hope everybody here [0:23] agrees with which is that debugging [0:26] distributed systems is preferable to [0:28] sticking a fork in your eye [0:31] uh but not that preferable it's actually [0:35] like it's actually surprisingly close um [0:37] and people will tell you a lot of [0:38] reasons why this is true uh one of the [0:41] most common ones I hear is it's because [0:42] they're complicated and I'll I'll grant [0:45] that it's true they are complicated and [0:47] debugging complicated stuff is harder [0:50] than debugging simple stuff but I want [0:53] to propose that this is not the real [0:55] reason why they're really hard to debug [0:58] um so to to look at the real reason [1:00] let's look at a super simple example so [1:02] we've got a server on the left here and [1:04] a server on the right here and the [1:06] server on the right is going to send a [1:08] packet to the server on the left and [1:10] this is a non item potent packet that [1:13] means if it gets received twice it has a [1:15] different effect than if it gets [1:17] received once and the server on the left [1:18] has a bug in it which is it doesn't [1:20] check how many times the packet's been [1:22] received but you know this bug is never [1:24] manifested it's asymptomatic because you [1:27] know networks always work right so [1:29] server on the right sends a packet and [1:31] it gets lost in a Time Warp because NE [1:33] networks are mysterious you know [1:35] somebody misconfigured a switch some [1:37] berserk router sent it to China um and [1:41] sooner or later like a retry happens [1:43] like you hope a distributed system would [1:44] do and then this packet emerges from the [1:47] time warp and everything catches fire [1:49] and explodes you lose your data [1:52] everything's a disaster and you're like [1:53] oh goody I get to debug this now so you [1:56] set everything up again and you run your [1:58] system again and and this time it works [2:02] fine and you're like okay I got to get [2:04] to crash again so I can debug it so you [2:06] set up again and it works fine and it [2:09] may just keep working fine because this [2:11] random Network this network condition [2:13] here was super transient and super rare [2:15] and depending on ridiculously precise [2:17] timings um you can never get it to show [2:19] up again it is going to show up again [2:22] probably on a client site right um but [2:26] you can't make it happen you can't track [2:28] down the bug and even worse once you [2:30] think you fixed the bug you can't verify [2:33] that you've actually fixed it so one [2:35] more example uh just to build our [2:37] intuition here right same setup server [2:39] on the left server on the right server [2:41] on the right is going to send two [2:42] packets it sends them like in order [2:46] packet a and then packet B it's not in [2:48] its contract that it has to send them in [2:50] that order it's just like we're normal [2:52] human beings right so when we program [2:54] sending two packets we type like send a [2:56] then send b we don't type you know roll [2:59] a die % chance send a send b then 50% [3:01] chance send b send a that's not what [3:03] humans do uh so it just always happens [3:06] to send them a and then B well [3:08] unfortunately the server on the left [3:10] thought that that was the contract [3:12] thought that that always did happen well [3:14] you see where this is going now a gets [3:16] lost in a Time Warp B shows up first [3:19] then a then the world [3:21] explodes uh good luck every debugging [3:24] this so the real so the the mean the [3:27] surface level problem is that we have no [3:29] repeatability are debugging the real [3:31] problem is that the messy dirty Universe [3:34] has intruded on our beautiful pristine [3:37] land of pure functions right like the [3:40] execution history of this system is not [3:43] a pure function of its inputs there's a [3:45] source of entropy a source of Randomness [3:47] which you do not control which is the [3:49] network um this problem is especially [3:52] apparent in distributed systems but you [3:54] can get the same kinds of bugs just [3:56] replace all the words uh with like [3:59] thread or discs or you name it um so so [4:03] this is the problem and this was like a [4:06] super bad problem for us because we make [4:09] a distribute database that tries to [4:10] provide a meaningful guarantee to its [4:12] users um acid is not a property of a [4:15] system that emerges by accident right [4:19] there there are a lot of ways of getting [4:21] that wrong a lot more ways of getting it [4:22] wrong than getting it right and what's [4:25] really scary is that there are many more [4:27] ways of getting it right and then have [4:29] having it fail as soon as something [4:31] unexpected happens than there are of [4:33] getting it right and having it stick and [4:35] be right the second problem is that we [4:37] make a database like if we just made a [4:39] web server or a cache or something we'd [4:41] be like whatever dude nuke the VM start [4:43] a new one turns out people aren't like [4:46] philosophical like that about their data [4:48] um I don't know why they get really [4:50] upset when you tell them to do that so [4:53] we basically decid that if we couldn't [4:54] solve this problem we shouldn't even [4:55] bother trying to build a database um so [4:58] we did solve the problem so the thing [5:00] that that the company did was we didn't [5:03] write a database we started by writing a [5:05] simulation of a database a totally [5:08] deterministic simulation and then once [5:10] we'd exhaustively debugged that we were [5:13] like okay now we can write a database [5:14] which is just that plus talking on [5:16] networks and talking to discs for real [5:18] um for actually a couple years there was [5:21] no database there was just a simulation [5:23] that probably sounds totally insane I [5:25] promise you it's more insane than you [5:27] think it is um we're talking about [5:29] simulating not just a process but an [5:32] entire network of communicating [5:35] processes plus all the interactions they [5:38] have with their environment with diss [5:39] with os's with networks all within a [5:42] single virtual process or a single [5:45] physical process many virtual processes [5:48] um that sucks it's really hard to do it [5:50] turns out to create some real software [5:52] engineering challenges so let me run [5:54] through those quickly sort of what they [5:56] are in theory and then I'll talk about [5:58] how we address them and how you might [5:59] address them um the first is you need [6:02] something that I'm going to call [6:03] single-threaded pseudo concurrency it's [6:05] a bit of a mouthful so let me unpack [6:07] that this the I mean like conceptually [6:11] right like you're trying to simulate [6:12] many many different things all happening [6:13] at the same time plus all the Machinery [6:16] around them that's making them think [6:17] they're talking to each other uh so [6:19] you're going to need something that [6:20] looks a little bit like concurrency it [6:22] can't actually be concurrent though [6:24] because if it were actually concurrent [6:26] you would now have a source of [6:27] non-determinism which we just like [6:29] sought to extra Pate right um like I [6:31] just said like all these problems happen [6:33] again with threads so we can't do it [6:35] with threads we can't do it in more than [6:37] one process we have to do it in one [6:39] process it's really easy in some [6:41] languages it was really hard in the one [6:43] we picked um the next thing you need is [6:47] a simulated implementation of all the [6:49] ways in which your processes can talk to [6:51] the rest of the Universe I think it's [6:52] pretty clear why you need that basically [6:54] all that Randomness all that entropy [6:56] needs to be Randomness and entropy that [6:57] you put there on purpose and then the [6:59] last thing you need sounds kind of dumb [7:02] but the processes you're simulating [7:03] themselves need to actually be [7:05] deterministic and you know if you get [7:07] one and two right three is usually not [7:09] that bad but there's some ways of [7:10] shooting yourself in the foot here so I [7:12] I'll talk about those um right so so the [7:16] way we tackled one was maybe a little [7:18] bit insane uh we knew that we couldn't [7:21] do this with threads for the reasons [7:23] I've already said co- routines have [7:25] their own issues that basically left [7:28] callbacks and we also knew that we [7:31] wanted to write in C++ because you know [7:33] it's fast so you know the problem here [7:37] this is a little bit of a dilemma CU [7:38] callbacks plus C++ is like a nightmarish [7:42] hellscape um was not something that we [7:44] wanted to deal with what we needed were [7:46] higher level Primitives um some [7:49] abstractions that we could actually [7:51] reason about that we could actually [7:53] write and actually read so the way we [7:56] did this was we created something called [7:58] flow which is sort of halfway between a [8:01] library and a new language um it's a [8:03] syntactic extension to C++ which lets [8:06] you use actor model concurrency um with [8:09] the physical implementation totally [8:12] single-threaded entirely in callbacks um [8:16] that's done via a sort of scare quotes [8:18] compilation step um you could call it a [8:20] pre-processing step I don't know it's a [8:22] compiler who's targeted C++ let's say [8:24] that um so let's you write things like [8:27] this this is like the simplest possible [8:28] actor you could right I guess it's like [8:31] an asynchronous Adder it takes a future [8:33] float and a float it waits for the [8:35] future to become ready it Returns the [8:37] sum uh there's two things you should [8:39] immediately notice about this the first [8:40] is that this actor keyword is not C++ [8:42] you feed that into your compiler it'll [8:44] yell at you um this tells the flow [8:46] compiler that what follows is an actor [8:48] definition and it should treat it as an [8:50] actor and generate an actor from this uh [8:53] the second thing you might notice here [8:54] is that this return type is wrong right [8:57] it's declared to return a future of [8:59] float but this sure looks like it's just [9:00] returning a plain ordinary float that's [9:02] because this function body is not going [9:04] to appear anywhere in your program it's [9:07] almost like declarative you're telling [9:08] the actor compiler what you want to have [9:10] happen what's actually going to happen [9:12] is it's going to generate uh a new [9:14] function and a new class and the [9:17] function is immediately going to return [9:19] a future which will become ready as soon [9:21] as the actor is done executing and as [9:23] soon as everything asynchronous is [9:25] finished happening um here's a tiny tiny [9:28] amount of the general code that it [9:30] produces so the async ad function gets [9:33] redefined or defined for the first time [9:36] it gets constant modifier stuck on its [9:37] parameters those parameters get passed [9:39] into the Constructor of a class which is [9:42] totally generated um that class returns [9:46] immediately um and the implementation of [9:49] the actor gets broken up across methods [9:51] in the class so every time there was one [9:54] of these like weight statements where [9:56] we're waiting for some asynchronous [9:57] process to terminate that means a [9:59] function in the class is going to end [10:02] and it's going to set a call back on [10:04] that future and the call back is going [10:06] to point to the next part of the actor [10:07] and then it's going to yield immediately [10:09] to a centralized scheduler so that the [10:11] rest of your program can continue [10:13] running um at the very end here's some [10:17] call back junk it generated uh at the [10:19] very end it's going to it's going to [10:21] fulfill that future that you were [10:23] holding with the promise that it holds [10:24] on to and then it destroys itself and [10:26] cleans itself up and blah blah blah um [10:29] and yeah so it's pretty nice you get to [10:31] write actors you know you get to write [10:33] this but it all happens in one thread [10:35] and it can still be super [10:37] asynchronous uh and it's also pretty [10:39] fast so here are some numbers that you [10:42] should take with a gigantic grain of [10:45] salt um this is a benchmark called The [10:48] Ring Benchmark it was invented by that [10:51] guy who just uh who just gave the [10:53] previous talk Joe Armstrong uh basically [10:56] you get a ring of actors n of them and [10:58] they're all lined up in a big ring and [10:59] they pass a message one to the other and [11:01] it goes all the way around the ring and [11:02] you do that M times and these were some [11:05] numbers so uh two reasons why you [11:08] basically shouldn't believe this one is [11:11] that the numbers that weren't flow were [11:12] numbers that other people ran that we [11:14] found on the internet and we think they [11:16] probably did a good job running their [11:17] own systems but we have no idea um the [11:20] second reason is that it's kind of a [11:22] simplistic Benchmark I hope Jo [11:24] Armstrong's not here is he it's kind of [11:26] a dumb Benchmark [11:28] like [11:30] like I mean it doesn't I mean it's an [11:32] okay Benchmark but it doesn't actually [11:34] model like what real distributed actor [11:36] systems do very much um and in fact we [11:41] found that as you make actors more and [11:42] more complicated and have like way more [11:44] asynchronicity in the body of the actors [11:47] flow's relative Advantage decreases [11:49] considerably um although we don't have [11:51] nice numbers from other people to [11:53] compare that against so that's one [11:55] reason why we're currently runting flow [11:56] 2 uh to try and try and make that a [11:58] little faster [11:59] but anyway um assuming you're not using [12:02] C++ you don't have to do anything near [12:04] that crazy but we did um so the next [12:07] thing conversely is something that might [12:09] be hard for most of you but was super [12:10] easy for us so we already had a [12:13] multi-platform program so we had nice [12:15] like abstracted out interfaces for [12:18] talking to diss talking to networks etc [12:20] etc and so we could just drop in a new [12:22] implementation right next to those so [12:24] you've got like your Unix Network code [12:26] and your Windows networking code and so [12:27] we added some simulated networking code [12:30] um that was not too bad and so if it's a [12:32] simulated Network then like when you try [12:34] and make a network connection instead of [12:36] actually making a network connection it [12:37] initializes a C++ class which has got [12:40] some internal State and some buffers and [12:42] like if somebody sends to it then it's [12:44] going to wait for a little while to [12:47] simulate latency on the setting sending [12:49] side it's going to wait for a little [12:50] while to simulate latency on the [12:51] receiving side then it's going to copy [12:53] some bites around and then it's going to [12:55] make the receiver know that like there's [12:57] some Bites available um [12:59] so one kind of cool thing here in the [13:01] read method and I'm going to talk a lot [13:02] more about this later is there's this [13:05] call here to roll random cloth and that [13:08] gets into sort of the fun part of this [13:09] talk uh which is how we try to turn up [13:12] bugs so this is just a taste basically [13:16] every time you try and read from the [13:17] simulated Network there's a chance that [13:19] something terrible will happen so we [13:21] roll a random number and if you get [13:24] unlucky then we roll some more random [13:26] numbers and then there's a chance that [13:28] your peer is just just going to close [13:29] the connection there's a chance that [13:30] you're just going to close the [13:31] connection there's a chance that you [13:33] don't know what happened and the [13:34] connection just fails and you receive an [13:35] error when you try and read um so this [13:38] is like how we try to smoke out any code [13:41] in our system that might have been [13:42] assuming the network was reliable which [13:45] is a very foolish thing to assume um but [13:47] this is just a start it gets a lot more [13:50] fun so then the last thing you need is [13:52] determinism uh this sounds brain dead [13:53] easy right like like we don't like [13:56] usually try and write our processes to [13:57] be non-deterministic never Le it shows [13:59] up all the time so the obvious example [14:02] is if you ever have a random number in [14:05] your program so for instance in [14:07] exponential back off we use random [14:08] numbers uh then you're going to have to [14:11] make sure that that random number comes [14:13] from a pseudo random number generator [14:15] which you control which you seeded and [14:18] then the seed of that random number [14:20] generator becomes part of the input of [14:21] the program right so so that's fine okay [14:25] it's a little weird but we can do that [14:28] have you ever [14:29] checked the time in an if [14:32] statement if so your program just became [14:35] non-deterministic if you ever checked [14:37] how much dis space you have free like [14:40] again like non-deterministic now like [14:42] running it two different times uh could [14:44] produce totally different execution [14:47] histories um our programmers are trained [14:51] to recognize this stuff and to try and [14:54] avoid it and we still screw it up all [14:56] the time uh really all the time and [14:59] that's why we do one last thing which is [15:03] some small percentage of our simulation [15:05] runs I think right now about 1% we [15:07] actually run them twice and then we [15:10] check uh one run them twice with the [15:11] exact same inputs mind you so the exact [15:13] same things happen in the exact same [15:15] sequence and at the very end we check [15:17] those random number generators I [15:18] mentioned and see what the next random [15:20] number it would have generated would [15:22] have been and if those don't match [15:24] exactly we know there was a determinism [15:26] failure and we go track that down so [15:29] this is obviously not an exhaustive [15:30] check you can break determinism and this [15:32] will still work um but it guarantees [15:34] that you'll find those problems sooner [15:37] rather than later um okay so you get [15:40] three things that behave like that and [15:42] you roll them together not necessarily [15:44] in the way that we did again you may not [15:46] be using C++ in which cases probably a [15:48] lot easier um and you have something [15:51] which can take a network of interacting [15:54] processes uh and simulate them together [15:56] in a totally totally deterministic way [15:59] so uh that's where the fun [16:01] begins um because the fun part is now we [16:04] need to find bugs and so the way we do [16:06] that is is is with these test files and [16:10] these test files declare two types of [16:13] things um one is a set of stuff that the [16:16] system is going to try and Achieve [16:18] that's this up here and one the rest is [16:20] a set of stuff that's going to try to [16:22] prevent it from achieving that so in [16:24] this case we've declared that we're [16:25] going to try and do a cycle test a cycle [16:28] test test is where we insert a ring of [16:32] keys and values into the database so [16:34] they're each pointing to the next one [16:35] like value one's pointing to key2 so on [16:38] and then we begin executing transactions [16:40] concurrently uh in this case a thousand [16:42] per second and they're all going to run [16:43] at the same time and each transaction is [16:46] going to mutate the keys and values in [16:49] the database and it's going to do that [16:51] subject to a constraint which is that [16:53] each transaction taken as a whole [16:55] preserves the ring it may reorder stuff [16:58] may move stuff around but the ring as a [17:00] whole will stay intact um so this is [17:03] super cool because it gives us an [17:05] invariant that we can check at the EAS [17:06] at the end that's super easy to tell [17:08] whether acid was violated right so like [17:11] if there was an atomicity failure one of [17:12] those transactions would partially [17:14] execute and the ring would be broken or [17:16] if some of these transactions clobbered [17:18] each other and there was a lack of [17:19] isolation again the ring would break or [17:21] it get shorter or get longer or [17:22] something so that's just like a cool a [17:25] cool like easy way of telling if there [17:26] was a failure um that is not of like you [17:29] know it crashed kind um and then the [17:32] last thing you'll note in that cycle [17:33] definition is that we're only expecting [17:35] to complete 1% of these and that's [17:38] because of the rest of the file uh so so [17:41] the next thing here is we're going to do [17:42] some random clogging that's where we [17:44] take these simulated network connections [17:46] like the ones I showed you and at random [17:48] or pseudo randomly deterministically [17:51] next time it'll happen the exact same [17:52] way if you run with the same inputs [17:54] we're just going to stop them and [17:55] prevent them from sending or receiving [17:56] packets try and flush out things of the [17:59] sort that you saw in that very first [18:01] slide um this clogging has the Swizzle [18:04] flag set swizzling is sort of like [18:07] clogging Plus+ I guess it's uh where you [18:10] take a a subset of the network [18:12] connections and you stop them like on a [18:13] rolling basis and then you bring them [18:16] back up in reverse order and for reasons [18:19] that we totally don't understand this is [18:21] better at finding bugs than normal [18:23] clogging uh it's kind of [18:25] scary so the next thing the next thing [18:28] we do here is is called is called [18:30] attrition in this test attrition is just [18:33] powering stuff off sometimes we power [18:35] stuff back on in this case we do we're [18:37] going to kill 10 machines total over the [18:38] course of the test and we're going to [18:40] leave three on at any given time so [18:42] hopefully if there's some power safety [18:44] problem that'll turn it up uh the last [18:45] one here is actually by far the most [18:47] evil even though it looks kind of [18:49] innocuous sometime in those 30 seconds [18:51] we're going to change the configuration [18:53] of the [18:54] database and uh this particular [18:57] configuration change is is designed to [18:59] provoke a coordination change which [19:02] means that the cluster is going to have [19:03] to execute multi paos and it's going to [19:06] be doing that while network connections [19:08] are randomly clogged while machines are [19:09] going down and rebooting left and right [19:11] and while it's trying to get a thousand [19:13] transactions done per second so you know [19:15] if there's a bug in our paxos hopefully [19:17] we'll find it um yeah and we do a crap [19:20] ton of other stuff too uh so this one's [19:23] like really nasty instead of just [19:25] killing machines we sometimes set them [19:27] in a state [19:29] where like sort of trying to simulate [19:31] them gradually failing all future system [19:34] calls have like a 1% chance of returning [19:36] an error so then the cluster we get to [19:39] find out like if it's smart about like [19:41] detecting that and like quarantining it [19:43] and destroying it before it infects [19:45] everything else um it tests graceful [19:47] shutdown tests a lot of good stuff like [19:49] that um we also have like a uh our [19:53] database has a concept of locality you [19:55] know it can be aware of your rack [19:56] topology or whatever and of where your [19:58] Wan links are are and so the simulation [20:00] knows that too uh so we can simulate [20:02] just you know killing an entire data [20:03] center see if that was really durable uh [20:06] we also simulate Dum CIS admins which [20:09] totally don't exist in the real world um [20:12] so one thing we can do is just instantly [20:14] automically swap the IP addresses of two [20:17] machines and see what happens or we can [20:20] instantly copy the data files from two [20:22] computers onto each other and just swap [20:25] them just switch their hard drives and [20:27] see if that results in any data loss [20:29] uh so that's tons of fun this is just a [20:31] random sample uh like one of the most [20:33] fun parts of my job is thinking up new [20:35] and creative ways of torturing our [20:37] database and then implementing that in [20:39] simulation uh and it's Oodles of fun so [20:42] uh now we get to the basic problem with [20:45] this whole approach which is you have [20:49] some set of machines out there which are [20:51] diligently chugging through simulations [20:54] exploring the space of all possible [20:56] inputs and the space of all possible [20:58] failure modes [20:59] which as we learned just a second ago is [21:00] very large but still you're exploring it [21:03] and you're hoping that you find the bugs [21:04] um so the interesting fact is you have [21:08] customers hopefully and your customers [21:11] are running your distributed systems on [21:14] buggy crashy failey grossly [21:18] underprovision [21:19] totally improperly configured Hardware [21:22] um and they're doing insane things to it [21:25] by the way and PS if it breaks it's your [21:27] fault um [21:29] so they are also exploring the space of [21:31] all possible inputs and failure modes [21:33] and hopefully you have many orders of [21:35] magnitude more customers than you have [21:38] machines in your test lab and so the [21:41] question is how on Earth can you [21:43] possibly hope to explore that space more [21:45] quickly than your customers do um [21:48] another way of thinking about this is [21:49] you need to find more bugs per CPU hour [21:52] than the real world uh by many orders of [21:54] magnitude which is hard so there's a [21:57] couple ways we try to do that [21:59] um I'm interested to hear what you all [22:00] think about this since it's definitely [22:02] an ongoing problem one way we do it is [22:05] just we make things fail a lot more [22:06] often right so like you saw that our [22:08] connections drop like pretty [22:10] regularly um I don't know in the real [22:13] world how often does a disc fail like [22:16] every two or three years if it's under a [22:17] heavy load or something we make them [22:19] fail like every two or three minutes so [22:21] that means hopefully if there's a bug in [22:23] our like dis failure handling code we [22:26] have many more dis failures per CPU hour [22:28] than the real world gets and so we find [22:29] it faster than our customers would um [22:33] another thing we can do which is kind of [22:34] cheating but that's the name of the game [22:36] here is uh is speed up time right so [22:40] there's a lot of time when the cluster [22:41] is just sort of sitting around being [22:42] quiescent or it's trying to recover from [22:45] something and there's some hard number [22:47] of seconds it's waiting for something to [22:48] happen uh when we're in that kind of a [22:51] state we can just speed up time and make [22:53] many more Sim seconds pass per real [22:56] world second than we did before so long [22:58] as we have the computing power on a [23:00] single core um so that's another [23:04] trick um okay so another thing we do is [23:06] called bug ification that's that's a an [23:09] in-house term I'm sure everybody else [23:10] does this but calls it something else so [23:13] the so the intuition here is if you [23:16] remember those first examples I gave you [23:18] they were both B the basic problem there [23:20] was that there was an asymptomatic bug [23:23] right the computer on the left actually [23:25] didn't understand the contract of the [23:27] computer on the right it was always a [23:29] bug it just took something exceedingly [23:31] rare to make it show up well if you have [23:34] deterministic simulation and you're [23:36] exploring that space of all possible [23:37] failures you know you hope that it will [23:40] show up eventually but wouldn't it be [23:41] nice if it you didn't have to wait for [23:43] something really weird to happen for it [23:44] to show up and so the idea here is you [23:47] do just that thing which I said [23:49] programmers never do which is sometimes [23:51] randomly change what your code does um [23:56] so in the first example you might have [23:58] written something and bugy is just a [24:00] macro right it expands to like do this [24:02] if you're in simulation with probability [24:04] something small um and we compute a hash [24:07] on the line number and stuff anyway [24:10] um right so you would say something like [24:12] if bugy send packet B and then send [24:14] packet a just to make sure that any [24:16] consumer of that interface really [24:18] understood what that interface did to [24:21] make sure that the like space of things [24:23] that the interface did was not like a [24:25] very narrow Subspace of the set of [24:29] things that it could do um so here are [24:31] just two more examples from our code you [24:33] know sometimes don't send everything [24:34] you've got ready to send here's one [24:36] where if we're buggying we just never [24:38] send a timeout you know because like [24:41] timeout might never show up it would [24:43] really suck if the other person on the [24:45] other end of here were counting on that [24:46] timeout for some reason and you know we [24:48] do this in other ways too um okay so the [24:52] last thing relates to one of my favorite [24:54] topics it's called the heurst exponent [24:57] uh which was first described in the 1951 [25:00] edition of the transactions of the [25:01] American Society of civil engineers [25:03] which I own a copy of because I'm a huge [25:05] nerd and it was described in this [25:08] article long-term storage capacity of [25:10] reservoirs by Harris Hurst so Hurst was [25:15] a British civil servant who was sent to [25:19] Egypt to study the dams and reservoirs [25:22] on the [25:23] Nile and he uh noticed that the previous [25:28] generation of Engineers had designed [25:30] these dams and reservoirs so they [25:32] wouldn't explode unless there was a [25:34] 100-year flood and they wouldn't run dry [25:36] unless there was a 100-year drought [25:38] seems pretty sensible um so then Hurst [25:41] noticed that the 100-year flood and the [25:43] 100-year drought were happening about [25:44] once every 15 [25:46] years and that was interesting and what [25:49] he figured out was that the previous [25:52] generation of Engineers and we're all [25:53] going to Snicker at this but don't [25:55] because people in finance did this up [25:57] until 2008 [25:59] uh had [26:00] assumed that every rain or not rain [26:04] event was an independent [26:06] trial and they were totally uncorrelated [26:08] right but of course this is a total lie [26:11] this is not real um in the real world [26:13] the probability of rain on Wednesday [26:15] given rain on Tuesday is slightly higher [26:18] than the probability of rain on [26:19] Wednesday you know full stop um they're [26:23] not independent at all and the heurst [26:25] exponent is a way of quantifying that [26:27] lack of Independence [26:28] so you know when you have independent [26:30] trials like that what you get is a [26:31] gaussian probability distribution which [26:33] falls off very very quickly in the tals [26:36] um when you take into account the fact [26:38] that the Hurst exponent could be not one [26:40] half you get a broader class of [26:43] distributions called stable parition [26:46] distributions um and the HST exponent is [26:48] the exact reciprocal of the alpha [26:50] parameter of these guys so Alpha of two [26:53] corresponds to a herst exponent of2 [26:55] which is a gaussian but if the H [26:57] exponent gets big bigger than a half [26:59] then all of a sudden you have something [27:01] that's not a gaussian that has way more [27:03] space in the Tails um that means that [27:06] high Sigma events or events that naive [27:09] Theory would predict or high Sigma are [27:11] going to happen way way way more often [27:13] okay what on Earth does that have to do [27:15] with distributed systems the answer is [27:19] Hardware failures are not independent [27:22] random events okay you all know this in [27:25] your guts right like if a hard drive [27:27] fails in a rack what is the first thing [27:29] you do oh my God check every other hard [27:32] drive in that rack um it could have been [27:34] in a bad batch there could have been [27:36] some bad maintenance there there could [27:37] be a humidity problem in the data center [27:40] like everybody knows this intuitively [27:42] even if they don't actually do what it [27:45] implies all the time um the problem is [27:49] that multiple cascading failures which a [27:52] non-naive approach to statistics tells [27:55] us are much more likely than a simple [27:57] naive gas would predict are among the [28:00] very very hardest things to test in real [28:03] world physical testing like getting a [28:05] whole bunch of Hardware all set up to [28:07] like fail in like some randomly totally [28:10] terrible together way is challenging but [28:13] in simulation you can do it really [28:15] easily and we do and we absolutely [28:17] manipulate the Hearst exponent and try [28:20] and simulate as many of these multiple [28:21] cascading failures as we can because we [28:23] know that they're going to happen in the [28:24] real world a lot um so you put that all [28:27] together [28:29] we estimate we've run the equivalent of [28:31] trillions of real world CPU hours worth [28:33] of tests we can run 5 to 10 million [28:36] simulation runs per night on our cluster [28:39] um each one of those simulates an entire [28:41] cluster behaving anywhere between I call [28:45] it five minutes and an hour or two um [28:47] depending on the test and then plus all [28:50] that bug ification and speeding up time [28:52] and everything else and we've been doing [28:54] this for years uh so we've we've done a [28:55] lot of testing so there's some bad news [28:58] though which is that we've totally [29:00] broken debugging um so you all already [29:04] know that stepping through code that is [29:08] laced and infused with callbacks is [29:11] horrible um if you were to take a [29:13] traditional debugging approach to a [29:15] deterministic simulation of your call [29:18] back infused code that'll be much worse [29:20] so you're going to have multiple sort of [29:22] call back infected monstrosities all [29:25] interacting coexisting in the same [29:27] process that you're attached to [29:28] plus all the simulation code around them [29:30] that's trying to convince them that [29:31] they're secretly talking to each other [29:33] um it's insanely challenging we do have [29:37] people who are really good at debugging [29:39] who are able to like step through like [29:41] two or three levels at the stack doing [29:43] this and like we're all in awe uh the [29:45] rest of us cannot um so what you're left [29:48] with is basically print F uh which [29:53] sounds pretty horrible and barbaric uh [29:57] and I'll be honest it is [29:59] um but it's actually not that bad when [30:02] you have deterministic simulation and [30:04] you know that if you reun the simulation [30:06] with the exact same inputs the exact [30:09] same set of stuff will happen in the [30:10] exact same order at the exact same times [30:13] that lets you put in non-spammy printfs [30:16] that depend on conditions that you know [30:17] will be fulfilled next time it lets you [30:20] gradually like work your printfs deeper [30:22] into something kind of you know Corner [30:24] the bug and then kill it and then you [30:26] know once you've killed it it lets you [30:28] know that the bug didn't repeat although [30:31] there's a subtlety there that if I have [30:32] time I'll get to maybe in questions um [30:37] okay so that's cool but there's still a [30:40] nightmare case which is that our [30:42] simulation is [30:44] wrong uh there's at least two ways that [30:48] a simulation can be wrong one is it's [30:51] not brutal enough there is some pattern [30:55] of failures which can happen in the the [30:58] real world and which does not happen in [31:00] our simulation and you know we don't [31:02] think that's true but who the hell knows [31:05] um networks are weird scary things uh so [31:10] that's one problem another problem is we [31:12] may just misunderstand the contracts of [31:15] the operating [31:16] systems right like fsync might mean [31:19] something different than we thought it [31:20] did I actually have a funny story about [31:22] that um not now though uh or you know [31:26] you're porting to a new platform and [31:27] like some some system call that you [31:29] thought you know is given the same name [31:30] confusingly enough actually has a [31:32] slightly different slightly different [31:34] guarantee um that could be a really big [31:38] problem or the OS has a bug right that [31:40] could be a big problem too you know the [31:42] simulation is only as good as our [31:44] understanding of what the operating [31:45] systems and the hardware actually do so [31:48] to solve both well to try and catch both [31:51] of these kinds of problems with [31:53] simulation we have this backup cluster [31:55] which we call syho so syho is um a bunch [31:59] of little little server motherboards [32:01] these are like real server they're kind [32:02] of cute but they're they're real server [32:04] motherboards with ECC RAM and the whole [32:06] whole shebang um and they're all hooked [32:09] up with like CAT 5 uh and through a [32:11] bunch of switches and both the switches [32:13] and the motherboards are all connected [32:15] to programmable Network Power Supplies [32:19] right oh yeah you see where I'm going so [32:22] then we just programmed these things to [32:23] turn on and off all night while our [32:26] database is running um [32:29] in the hopes that you know we'll find [32:30] something and what we've found is that [32:33] Network Power Supplies suck they burn [32:35] out after like two days it's a giant [32:37] pain in the ass then we have to buy more [32:39] we've run through hundreds of these [32:41] things um also ssds don't like it when [32:44] you toggle their power on and off really [32:46] fast that's a little bit more expensive [32:49] um we're really proud to say though that [32:52] we've actually never found a bug in our [32:53] database doing this uh which makes us [32:55] think that simulation is is doing a [32:57] pretty good job [32:58] I said I never found a bug in our [33:00] database doing this um we have found [33:02] bugs in other people's [33:03] software um so this is the other thing [33:06] right that simulation can't test and [33:07] that's a real danger so software that we [33:11] didn't write software that wasn't [33:12] written in flow uh but that's part of [33:14] our stack we cannot test in simulation [33:18] so early early versions of foundation DB [33:20] used zookeeper for uh maintaining [33:22] coordination State we found a power [33:25] safety bug in Zookeeper with a precursor [33:27] to sinkhole um we reported it they fixed [33:29] it that was awesome but that experience [33:31] was so traumatic and kind of scary that [33:34] we ripped it out of our stack and wrote [33:36] Our Own PA house that's written in flow [33:38] it runs under simulation so like now [33:41] we're like a little more chill about [33:43] that also deploying Java is a pain [33:46] um this one's kind of funny so we once [33:50] did a sinkhole run we installed our [33:52] database on all these computers started [33:55] them all up and running and then just [33:56] like started killing the power left and [33:57] right and one of the processes didn't [33:58] come back up and we were like what that [34:01] can't happen our database is power safe [34:03] so we dug into it a little bit and we [34:05] found out that the data files had all [34:08] been durably synced and logged to disk [34:10] they were right there the configuration [34:13] files for the database were like [34:14] non-existent or corrupt or something [34:16] because apt had not called fsync uh so [34:20] then the database process came back up [34:21] it was like oh no configuration file I [34:23] must be a new process oh wait there's [34:25] data here oh no and it panicked [34:28] um so that was that was fun okay I've [34:31] talked for a long time let me let me [34:33] wrap up [34:34] here uh despite all that we're doing you [34:36] know that space of all possible failures [34:38] is very very very large and so we have [34:39] to explore systematically but there's [34:42] all kinds of ways in which we fail to do [34:43] that because we're humans so here are [34:45] just a few of the things that are on our [34:46] mind uh that we worry about a lot um one [34:51] is you know we'd probably feel a lot [34:52] better if we had the resources to have a [34:54] couple employees whose job was just to [34:55] introduce bugs um and see if they make [34:58] it through simulation and then if it [35:01] does then that's a bug report against [35:03] the simulation framework um another [35:06] thing is more Hardware so like this is [35:08] kind of weird I feel like we've entered [35:10] like a new like phase and the [35:12] relationship between man and machine uh [35:15] because if you look at the workflow of [35:16] one of us it looks like you make some [35:18] changes to the program you recompile and [35:20] then you run like 10,000 simulations to [35:22] see if you did something obviously dumb [35:24] and then you hope that if you did [35:25] something subtly dumb you'll get caught [35:27] that night when we run 10 million that [35:29] sucks running 10,000 simulations takes [35:32] like a minute or two or I don't know a [35:34] while um you lose your train of thought [35:37] it totally breaks your flow like this is [35:40] why one reason why people use [35:41] interpreted languages right is so you [35:43] can even get rid of that compilation [35:44] step we've just massively increased [35:47] cycle time uh of debugging or of coding [35:51] uh and that's terrible so if we added [35:53] more Hardware that might like weirdly [35:56] like increase the productivity of all [35:57] our [36:12] [Music] [36:57] it's a nightmare so we thinking of a [37:00] bunch of different ways of trying to fix [37:01] this one is have two simulation [37:05] Frameworks that operate on different [37:08] principles and you know use one of them [37:10] for like day-to-day debugging and like [37:12] development and bounce against it all [37:13] the time and the other one only use it [37:16] when we're about to do a release and run [37:18] millions and millions of simulations on [37:20] that one and see if any bugs slip [37:21] through the first one and this is sort [37:23] of like how we're trying to defeat [37:24] antibiotic resistance right that's sort [37:26] of the inspiration [37:28] that would be a lot of work though so [37:29] we're trying to think of other ways to [37:31] do this let me know if you have any [37:32] ideas um then the last thing is just [37:35] more real world testing more stuff like [37:37] sinkhole we know we should do this but [37:40] we hate it because Hardware is terrible [37:43] I have a cooworker who jokes that like [37:45] every part of programming is fun except [37:47] the parts that interact with networks [37:49] discs or power supplies and like that's [37:51] now all he does um you [37:55] know yeah like yeah [37:58] goes through a million programmable [37:59] power supplies a week it's terrible the [38:02] idea of having that times 10 gives me [38:04] nightmares but we all know that it's [38:06] good for us and like you know we're at [38:09] the point where having a little bit more [38:11] of that would probably give us a really [38:12] good return um okay so that's my talk uh [38:16] does anybody have any [38:18] questions yeah so what you're doing here [38:22] awesome kind of reminds Mee Netflix [38:26] army I a at [38:30] thatp [38:32] consider yeah [38:38] this yeah so um there are definitely [38:41] pros and cons um to both approaches I [38:43] will I will stop you one second here all [38:45] this simulation is run on our production [38:47] code if I didn't make that totally clear [38:49] so the processes huh the production enir [38:52] yes that's right so the only part of our [38:54] actual production code that's not tested [38:55] under simulation is the part actually [38:58] does the non simulate talking to discs [38:59] and networks which we obviously can't [39:01] simulate although we've thought about [39:03] how we could [39:04] um right so so right so the benefit of [39:07] simulation is you can do many many more [39:09] different types of things and you can [39:12] generally be far more brutal and far [39:14] more like calculatedly evil like in a [39:17] real production environment you don't [39:19] have like omnipotent evil Godlike power [39:22] to like stop and start network [39:24] connections in the most inconvenient [39:25] possible way or to like do what you know [39:29] is like the hard case for paxos and like [39:31] deliberately do that to try and screw [39:33] with it right um on the plus side like [39:36] doing it for real in your for real [39:37] production environment maybe teaches you [39:39] something about Hardware that you didn't [39:40] know right so sinkhole is hopefully I [39:44] really hope not anything like what any [39:46] of our customers run in production um [39:49] maybe there's something wrong with all [39:50] those discs and and network cards that [39:52] our customers run in production that we [39:53] don't know about so there's advantages [39:55] to both approaches we should probably we [39:57] could probably do a lot more of that [39:59] yeah what do you think would think of [40:01] the [40:03] as uh I mean Hurst was a statistician [40:06] not an environmentalist or a development [40:08] Economist but I I don't know we got [40:13] we're done all right sorry uh I'll talk [40:15] to you after [40:18] thanks