TigerStyle! (Or How To Design Safer Systems in Less Time) by Joran Dirk Greef
TigerBeetle
AI-assisted summary of TigerStyle! (Or How To Design Safer Systems in Less Time) by Joran Dirk Greef — original source. Check the original for full context.
Source published:
Key ideas
- TigerStyle compresses traditional 10-year distributed database engineering lifecycles down to 4–5 years by pairing deliberate, upstream design restraint with dense runtime assertions and deterministic simulation. (0:00)
- Software development velocity should be modeled across total lifecycle cost of ownership; typing speed only yields up to a 2x improvement, whereas solving problems in the design phase can yield 10x to 1,000x downstream returns. (5:00)
- TigerBeetle eliminates post-startup dynamic memory allocation entirely, statically preallocating all message buffers and storage pipelines to bound edge cases, manage backpressure, and prevent out-of-memory crashes. (32:15)
- Deploying a deterministic in-memory simulator that injects network partitions and aggressive storage corruptions uncovered and resolved 30 complex distributed bugs within its first three weeks of operation. (45:00)
Detailed notes
## Background Published by the TigerBeetle channel on YouTube on April 26, 2023, this video presents the final talk from the Systems Distributed '23 conference. The presentation, delivered by Joran Dirk Greef, introduces the TigerStyle engineering methodology and invites viewers to join the community discussion on the project's Slack workspace.
## TL;DR Joran Dirk Greef details TigerStyle, an engineering philosophy developed while building TigerBeetle that compresses the timeline for constructing mission-critical distributed systems without cutting corners. The methodology relies on three core tenets: evaluating the full development lifecycle to eliminate code early during design, asserting hard bounds and invariants via extensive tripwire assertions, and leveraging deterministic simulation testing to accelerate verification. By utilizing static memory allocation, fail-fast crash semantics, and Zig's compile-time safety, teams can eliminate technical debt and avoid production corruption. Greef demonstrates these concepts using TigerBeetle's deterministic simulator, VAPOR, as well as an interactive retro-themed browser game that visualizes Viewstamped Replication under extreme network partitions and storage corruption.
## Atmosphere & style The presentation is energetic, passionate, and charismatic, striking an engaging balance between deep systems engineering rigor and playful pop-culture metaphors ranging from Tron and The Lord of the Rings to classic chiptune video games. Greef speaks with evangelical conviction and warmth, candidly celebrating his engineering team and early collaborators while recounting grueling debugging mysteries. The live demonstrations shift the mood from reflective architectural philosophy to high-adrenaline excitement as terminal logs and retro WASM simulations withstand relentless fault injection, concluding the conference on an inspiring, community-minded note.
## Bulletin - [00:00](https://www.youtube.com/watch?v=w3WYdYyjek4&t=0s) TigerStyle compresses traditional 10-year distributed database engineering lifecycles down to 4–5 years by pairing deliberate, upstream design restraint with dense runtime assertions and deterministic simulation. - [05:00](https://www.youtube.com/watch?v=w3WYdYyjek4&t=300s) Software development velocity should be modeled across total lifecycle cost of ownership; typing speed only yields up to a 2x improvement, whereas solving problems in the design phase can yield 10x to 1,000x downstream returns. - [32:15](https://www.youtube.com/watch?v=w3WYdYyjek4&t=1935s) TigerBeetle eliminates post-startup dynamic memory allocation entirely, statically preallocating all message buffers and storage pipelines to bound edge cases, manage backpressure, and prevent out-of-memory crashes. - [45:00](https://www.youtube.com/watch?v=w3WYdYyjek4&t=2700s) Deploying a deterministic in-memory simulator that injects network partitions and aggressive storage corruptions uncovered and resolved 30 complex distributed bugs within its first three weeks of operation.
## Key takeaways - [00:00](https://www.youtube.com/watch?v=w3WYdYyjek4&t=0s) The TigerStyle methodology rests on three core pillars: seeing the whole picture across the lifecycle, laying dense tripwire assertions, and leveraging deterministic simulation as a time machine. - [05:00](https://www.youtube.com/watch?v=w3WYdYyjek4&t=300s) Typing faster offers marginal speedups; the only order-of-magnitude engineering leverage lies in the design phase, where architectural breakthroughs can eliminate code entirely before it is written. - [10:00](https://www.youtube.com/watch?v=w3WYdYyjek4&t=600s) Software teams must resist rushing into implementation; TigerBeetle's deliberate 3.5-month initial incubation period yielded an estimated 1,000x return on execution velocity. - [12:38](https://www.youtube.com/watch?v=w3WYdYyjek4&t=758s) Testing complex distributed systems historically takes decades because verification is bounded by wall-clock time; breaking this bottleneck requires deterministic simulation. - [15:00](https://www.youtube.com/watch?v=w3WYdYyjek4&t=900s) Building mission-critical systems as 'painters' rather than 'sculptors' avoids bloat and performance regressions by composing from four fundamental colors—Network, Disk, Memory, and CPU—governed by latency and bandwidth. - [20:00](https://www.youtube.com/watch?v=w3WYdYyjek4&t=1200s) Systems must explicitly define mechanical fault models upfront across all hardware layers and aggressively minimize surface area by limiting stateful components and network ports. - [22:31](https://www.youtube.com/watch?v=w3WYdYyjek4&t=1351s) Enforcing a zero-technical-debt policy prevents compounding context-switching penalties, while biweekly walk-and-talk sessions help teams invalidate flawed ideas before implementation. - [25:07](https://www.youtube.com/watch?v=w3WYdYyjek4&t=1507s) Functions must serve as self-verifying correctness boundaries that validate entry preconditions, parameter relationships, and postconditions, immediately shutting down upon invariant violations. - [28:13](https://www.youtube.com/watch?v=w3WYdYyjek4&t=1693s) While typical programmers work within positive space, distributed systems environments act like hackers operating in negative space, requiring developers to defensively assert against unanticipated states. - [30:00](https://www.youtube.com/watch?v=w3WYdYyjek4&t=1800s) Loops should never be unbounded; enforcing explicit upper and lower bounds prevents deadlocks, runaway compute usage, and unverified minimum work. - [37:28](https://www.youtube.com/watch?v=w3WYdYyjek4&t=2248s) Operating errors that can be gracefully handled must be strictly separated from programming bugs, which represent undefined logic territory that must trigger an immediate fail-stop crash. - [40:00](https://www.youtube.com/watch?v=w3WYdYyjek4&t=2400s) In distributed replicated systems, crashing quickly on invariant failure preserves safety and high availability, whereas limping along risks silent and catastrophic data corruption. - [40:00](https://www.youtube.com/watch?v=w3WYdYyjek4&t=2400s) Zig provides an ideal foundation for memory-intensive systems programming through default-checked arithmetic and compile-time execution ('comptime') for verifying structural invariants. - [50:00](https://www.youtube.com/watch?v=w3WYdYyjek4&t=3000s) The VAPOR deterministic simulator validates strict serializability and consensus correctness under extreme fault models, including simultaneous 30% storage read and write corruptions. - [52:45](https://www.youtube.com/watch?v=w3WYdYyjek4&t=3165s) Compiling TigerBeetle's real production codebase into WebAssembly demonstrates how deterministic simulation can be run entirely in the browser to visualize consensus under network partitions and storage faults.
## Detailed walkthrough ### [00:00](https://www.youtube.com/watch?v=w3WYdYyjek4&t=0s) The Magic Trick: Velocity Through Style
Joran Dirk Greef introduces what he calls a "magic trick" for tackling the most complex software challenges conceivable, such as consensus protocols, distributed databases, operating systems, self-driving vehicle software, or mission-critical medical devices. Typically, these systems require a decade or more of painstaking engineering. Greef argues that applying this particular style allows an engineering team to accomplish that same build in four to five years without cutting corners: they work hard, but with vastly greater velocity. The resulting architecture is safer, faster, simpler to operate, and robust enough to allow engineers to sleep soundly at night.
```mermaid flowchart TD accTitle: The Three Core Pillars of TigerStyle accDescr: Diagram showing how TigerStyle combines seeing the whole picture, checking assertions, and time machine testing to compress system build time. subgraph TS["TigerStyle Methodology"] P1["1. Sees the Whole Picture<br>(Design & Lifecycle TCO)"] P2["2. Checks Yourself Before You Wreck Yourself<br>(Assertions & Preconditions)"] P3["3. Gives You a Time Machine<br>(Deterministic Simulation Testing)"] end P1 --> OUT["Safer, Rock-Solid Systems in 50% of the Time"] P2 --> OUT P3 --> OUT ```
This methodology emerged directly from the creation of TigerBeetle, an open-source, dedicated distributed financial accounting database engineered specifically for mission-critical safety and high throughput. Greef notes that he learned this development philosophy from pioneers such as Gerard Holzmann (author of the Spin model checker and NASA/JPL's coding rules) and James Cowling (known for consensus and distributed storage work). Because this philosophy feels unfamiliar and counterintuitive to standard industry practices, observers often label it "crazy." Greef's explicit goal for the talk is to shift how developers think about programming so that building systems once considered unreachable comes within grasp.
Greef connects this engineering style to recent breakthroughs in testing, referencing Will Wilson's presentation on deterministic simulation testing at Antithesis. To explain how TigerBeetle went from an idea to a running production database and fostered a community—giving credit to early supporters and collaborators including Hardcore, Zig SHOWTIME, Amplify, Coil, Adrian, and co-founder Don, who originally coined the name TigerBeetle—Greef breaks the methodology down into three mutually accelerating concepts: 1. Seeing the whole picture 2. Checking yourself before you wreck yourself 3. Giving yourself a time machine
As Greef puts it when describing the transformative impact of the methodology: “And the way that we developed Tiger Beetle is something that gets our team excited. Because when you see these systems run the way they do, it's hard to describe. It's like magic.”
- TigerBeetle was created less than three years prior to the talk as a mission-critical distributed ledger for financial accounting. - Don, present in the audience, is credited as the co-founder who created the name TigerBeetle alongside Greef. - The overarching promise of TigerStyle is that it "buys you time" by pulling future development velocity into the present.
---
[05:00](https://www.youtube.com/watch?v=w3WYdYyjek4&t=300s) Seeing the Whole Picture and the Total Cost of Ownership
Greef begins with the premise that the audience already possesses what he terms "the blessing of a hard problem"—a high-value, non-trivial engineering challenge that justifies serious effort. When developers start such projects, they frequently succumb to the appeal of popularity, adopting languages and ecosystems simply because they are trending or because they want a resume boost. While engineers often mock the logical fallacy of appealing to authority, they routinely select stacks like Rust, npm dependencies, or GraphQL merely because everyone else uses them. Greef clarifies that developers should choose tools on quality rather than quantity: Rust, for instance, should be chosen for its borrow checker and memory safety guarantees, not because of industry hype.
```mermaid flowchart LR accTitle: System Development Lifecycle Cost Equation accDescr: Horizontal progression through the phases of engineering showing how poor upstream design amplifies downstream cost. D["Design Phase"] --> C["Coding Phase"] C --> T["Testing Phase"] T --> P["Production Operations & Incidents"] ```
TigerStyle deliberately disrupts standard instincts by forcing developers to practice strict restraint. Developers are required to pause, deliberate, and delay coding. Even when coding finally starts, the process is deliberately slower and more methodical. Greef compares this mindset to a disciplined martial artist: by practicing restraint upstream, overall delivery accelerates. The fundamental error of modern software engineering is reducing system creation entirely down to the coding phase, ignoring design, testing, and day-to-day operational realities. TigerStyle instead views the software lifecycle through total cost of ownership, expressing development time and cost as an interrelated function of design, coding, testing, and production incidents:
$$\text{Time / Cost} = f(\text{Design}, \text{Coding}, \text{Testing}, \text{Production Incidents})$$
Trying to speed up software delivery by typing faster yields at most a 2x improvement, which Greef dismisses as marginal and immaterial when tackling distributed systems. True order-of-magnitude leverage exists exclusively in the design phase. A high-leverage design makes it possible to eliminate code entirely before it is ever written. Conversely, rushing out of design into coding creates catastrophic downstream costs: a flaw caught in production inflicts real customer damage and requires months of painful coordination, patch development, and deployment planning to remediate.
Highlighting how engineering restraint generates speed, Greef emphasizes: “And if we move too early out of the design phase and into coding testing production, a bad design will become more and more expensive and take longer and longer to fix.”
- Merely increasing typing speed or working more hours caps productivity gains at roughly 2x, which fails to move the needle on hard problems. - Thoughtful architectural design can completely design away entire categories of code, sparing the team from having to write, test, and maintain them. - Decisions made at the engineering level (such as choosing static memory allocation) directly impact business survival, operational costs, and product differentiation. - Bugs that escape into production inflict severe compounding costs, turning minor architectural oversights into multi-month mitigation efforts.
[10:00](https://www.youtube.com/watch?v=w3WYdYyjek4&t=600s) Deferring Code, the Ent Mentality, and the 1,000x Return on Design
Engineers often struggle with the tension between cold objectivity and the subjective notion of elegance. Invoking Edsger W. Dijkstra, Joran Dirk Greef explains that simple, elegant architectures do not just look neat on paper; they execute faster and fail far less often. He urges developers to be like the Ents from *The Lord of the Rings* when heading to war: deliberate, unhurried, and a force to be reckoned with. The central discipline is to deliberately postpone writing code until the high-level design rests on solid footing. If the back-of-the-envelope calculations do not add up or an engineer's intuition signals that something is amiss, they must pause rather than rushing ahead. Instead of settling for an early local maximum, engineers need to act like Frodo and Sam marching toward Mordor—locating the absolute steepest, fiercest challenge, the global maximum on the slopes of Mount Doom, and designing directly for that worst-case reality.
```mermaid flowchart TD accTitle: Iterative Design Cost vs Production Cost accDescr: Diagram illustrating how cheap conceptual iteration prevents expensive late-stage rewrites.
subgraph Ideation["Design Phase (Cheap Iteration)"] A["Sketch System Ideas"] --> B{"Numbers Match & Gut Check?"} B -- "Unresolved Issues" --> C["Discard & Re-sketch"] C --> A end
subgraph Implementation["Downstream Delivery"] B -- "Slopes of Global Maxima" --> D["Write Production Code"] D --> E["Rigorous Testing & Validation"] end ```
In TigerBeetle's case, this patient restraint was partly an accident of timing. Greef notes that the project had a three-and-a-half-month incubation period from March to July 2020 before any coding began, allowing critical concepts to simmer. Without that forced gestation, TigerBeetle's most fundamental architectural choices would have been missed entirely. Spending an extra month on design across a two-year timeline amounts to a negligible 4% of the total project schedule, but that minor delay reaps exponential rewards. Redoing sketches on paper or batting concepts around with teammates costs almost nothing, whereas discovering a structural flaw after implementation forces devastating rewrites. For TigerBeetle, Greef calculates that this initial 4% time investment delivered an astounding return on execution velocity.
- Edsger W. Dijkstra on the practical power of elegance: “The simple and elegant systems tend to be easier and faster to design and get right, more efficient in execution and much more reliable.” - Greef on resisting the impulse to start typing prematurely: “So I think the discipline is to defer coding, to wait, until you have the design roughly right.” - On the massive payoff of the upfront simmering period: “And over the course of, say, like a two-year project, that extra month is only 4%. When you compare it with the coding and testing phases, yet that 4% is going to give you like a 10x impact on those phases or 20x or 1000x for TigerBeetle was literally 1000x.”
---
[12:38](https://www.youtube.com/watch?v=w3WYdYyjek4&t=758s) The Reality of Testing Complex Systems and the Need for a Time Machine
While simple programs can be tested in a timeframe roughly equal to their development time, complex storage and distributed systems belong to an entirely different league where testing dominates the lifecycle. Historically, proving database correctness takes decades—an ongoing saga exemplified by PostgreSQL working through storage edge cases like "FsyncGate" three decades in. Industry gold standards like Kyle Kingsbury's Jepsen tool are indispensable for finding distributed anomalies, but they are bounded by wall-clock time: if a distributed race condition takes seven continuous days of load to manifest, verifying the bug and verifying its eventual fix each require a full week of runtime.
To escape this timeline trap, distributed systems engineers require a conceptual "time machine" that can manipulate time deterministically. If an engineering team spends two years building a high-performance consensus engine and storage layer like TigerBeetle, industry precedent dictates an evaluation window of at least five to ten years to weed out catastrophic edge cases. Accelerating that verification phase down to a single year represents a true 10x "silver bullet." Navigating this trade-off requires zooming out to appreciate how intentional slowness in early design enables unprecedented downstream speed.
- Joran Dirk Greef on what truly drains engineering resources in infrastructure: “Testing I think this is my opinion, but I think testing is what really dominates the development of complex systems most of all and the reason is that for simple systems people say you know well testing time is is roughly equivalent to the coding time... but for complex systems testing can take longer, sometimes decades.” - On the inherent operational cost of real-time fuzzing and testing frameworks: “Postgres took 30 years and we're still solving FsyncGate, you know, in 2020. So for example, like a tool like Jepsen is terrific at testing distributed databases. But even with Jepsen, if you want to find a bug that takes a week to manifest, you have to run Jepsen for a week.” - Greef on compressing the multi-year testing lifecycle: “If it took two years to build TigerBeetle, what if we could test it in a year? Instead of four, that would be a silver bullet. Because normally to test something like TigerBeetle is 10 years.”
---
[15:00](https://www.youtube.com/watch?v=w3WYdYyjek4&t=900s) Bio-Digital Jazz, Painters vs. Sculptors, and the Four Colors of Systems
Great software extends beyond textbook engineering; it combines empirical rigor with creative human artistry. Greef invokes his favorite fictional hacker, Kevin Flynn from *Tron*, who celebrated this synthesis as "bio-digital jazz"—the interplay of a rigid, tight logical beat with a soulful, organic groove. Deciding to build software as art forces teams to confront their fundamental creative mode, a dynamic captured by record producer Daniel Lanois's taxonomy of musicians: sculptors versus painters. Sculptors, such as Bob Marley carving an album out of a massive wall of sound, begin with a heavy block of material and chip away until only the essential reggae cadence remains. Painters, such as The Beatles composing in the studio with nothing more than blank paper or odd percussion items, build their canvas layer by layer from nothing.
Modern software engineering leans heavily toward sculpting: teams drag in enormous third-party frameworks and external dependencies, hacking them down to shape their product. For TigerBeetle, however, grafting outside dependencies would have ruined safety, ballooned testing times, and degraded throughput; creating something as singular as "Strawberry Fields Forever" required painting out of thin air. In the systems painting palette, there are only four fundamental colors—Network, Disk, Memory, and CPU—governed by just two shared textures: latency and bandwidth. By running back-of-the-envelope calculations across those four dimensions during design sketches, engineers can verify physical feasibility and avoid latency spikes long before committing code, and even take the time to solve the "second hardest problem" in computer science: naming things well.
- Joran Dirk Greef on the dual nature of programming: “If you believe that software is at least engineering and also more than engineering, that there's a precision and an art to software, that there's hard data, cold logic, and also what people bring to the table, human experience, creativity, information, inspiration.” - Greef on TigerBeetle's deliberate refusal to rely on off-the-shelf codebases: “However, we didn't build TigerBeetle like that because we zoomed out and we saw that the dependencies were going to slow down testing. They were going to introduce safety and performance risk. So we realized it was actually just going to be faster to paint TigerBeetle.” - Defining the low-level systems canvas: “And for systems, I think these are all the colors you need right, this is it, like just remember this: Network, disk, memory, CPU... because the two textures that all these paints share is latency and bandwidth.” - Greef's parting rule of thumb for the drafting phase: “You want to try and sketch as many different solutions as you can in the design phase because sketches are cheap, rewrites are expensive.”
[20:00](https://www.youtube.com/watch?v=w3WYdYyjek4&t=1200s) Mechanical Fault Models and Reducing Surface Area
Good naming is not merely an aesthetic concern; having your nouns and verbs precisely right proves that you understand the system and possess a clear mental model. Humans have an inherent fault model—we make mistakes, and clear naming reduces miscommunication. Machines, likewise, operate under specific mechanical fault models that must be explicitly defined from the outset, spanning the network, storage, memory, and CPU. For a database like TigerBeetle, memory allocation failure cannot simply result in a panic—it must be safely handled. Other systems operating at massive hyperscale must reckon with ECC memory limitations, CPU bit-flips, and Byzantine fault tolerance. As Joran Dirk Greef emphasizes, “The big point is just be explicit upfront about what your system should tolerate.”
```mermaid flowchart TD accTitle: Fault Model and Surface Area Reduction accDescr: Mapping explicit mechanical fault models to architectural surface area reduction strategies.
subgraph Faults["Mechanical Fault Models"] direction TB F1["Storage Faults: Bit rot & disk slowness"] F2["Memory Faults: Allocation failures"] F3["CPU & Network: Scale & Byzantine faults"] end
subgraph Reductions["Architecture & Surface Area Reductions"] direction TB R1["Keep components stateless & off disk"] R2["Handle allocation safely without panic"] R3["Consolidate into single-port processes"] end
F1 --> R1 F2 --> R2 F3 --> R3 ```
Once those fault models are established, the architecture can be engineered to minimize exposure to faults in the first place. If the storage fault model anticipates bit rot or disk latency spikes, the design should deliberately keep as many components off the disk as possible, because stateful systems are notoriously hard to build. Beyond storage, total surface area must be ruthlessly trimmed. A system with two hundred components overwhelms human comprehension, whereas a system with three is manageable and vastly reduces unexpected interactions. Joran notes that this principle extends straight down to networking: rather than opening multiple listening ports across a single process for different roles, consolidating down to a single port dramatically simplifies deployment, reduces operational exposure, and boosts efficiency.
- Defining fault models explicitly covers network, storage, memory, and CPU failure modes. - In database engines like TigerBeetle, memory allocation failures must be safely caught and handled rather than triggering a crash or panic. - Minimizing stateful components limits bit rot and storage bottleneck exposure. - Limiting the raw count of components avoids cognitive overload and unpredictable emergent interactions. - Single-port process architectures minimize open networking attack surface and streamline operations.
---
[22:31](https://www.youtube.com/watch?v=w3WYdYyjek4&t=1351s) Zero Technical Debt and Walk-and-Talk Design
Trimming surface area does not mean cutting corners. TigerBeetle maintains a strict zero-technical-debt policy. Joran jokes about who holds the line on the team—“I don't know which one of us is Gandalf. It's probably Adrian who wears the Gandalf hat.”—explaining that if an engineer discovers a better implementation while coding, they drop the current draft and do it the right way immediately. Deferring cleanup is a trap because the mental overhead of reloading context into your head later is massive, and no one can guarantee they will ever be granted the time to return and fix it. Doing it right the first time maintains forward momentum and keeps team morale high because everyone knows the foundational work is rock solid.
To break through stubborn design bottlenecks, the team relies on biweekly "walk and talk" sessions. Pacing triggers creative problem-solving in ways that sitting at a desk cannot. Joran paces the Cape Town boardroom for an hour on Bluetooth headphones with cameras off, while his colleague Fed walks the office halls and steps onto the balcony until his connection cuts out. They alternate pitching solutions, hunting for fatal counterexamples, and tearing down assumptions. The ultimate victory of a walk and talk is not merely solving a thorny problem, but realizing the entire problem can be engineered out of existence before a single line of code is written.
- TigerBeetle enforces a zero-technical-debt policy: if a superior approach is found during implementation, the team switches to it immediately. - Context switching and reloading complex system mental models later is treated as an unaffordable hidden cost. - High architectural standards create a positive second-order effect by motivating the engineering team and maintaining momentum. - Walk-and-talk sessions occur every two weeks with cameras off over Bluetooth headsets while engineers pace rooms or offices. - The primary objective in design walks is attempting to invalidate each other's ideas until the problem itself can potentially be designed away entirely.
---
[25:07](https://www.youtube.com/watch?v=w3WYdYyjek4&t=1507s) Tripwire Assertions and Self-Verifying Functions
Shifting from system design into actual coding, Joran warns that adopting TigerStyle will initially slow engineers down, but it does so deliberately to accelerate all subsequent testing and review phases while guaranteeing high software quality. Regardless of how exhaustive code review is, subtle edge cases inevitably slip through—resembling the unexpected glitch that allows a player to jump straight through the ceiling in Super Mario. When testing and review reach their limits, the final line of defense is ensuring the software adheres to a strict binary guarantee: it must either operate cleanly within verified safety parameters, or immediately shut down the instant an invariant is violated.
Borrowing from Gerard Holzmann’s NASA Power of 10 rules for safety-critical code, TigerBeetle relies heavily on runtime assertions. Joran views assertions as defensive tripwires scattered deliberately throughout the codebase: “I love them because you're laying tripwires everywhere. You think, ah, someone is going to try and do this. I'm going to lay a tripwire there.” Functions should not exist solely for code abstraction; their primary duty is contributing to correctness. Every function checks its arguments upon entry, validates how those parameters relate to each other and global state, and asserts invariants over its return values upon exit. This makes the entire codebase continuously self-verifying under real workloads.
```mermaid flowchart LR accTitle: Self-Verifying Function Pattern accDescr: Logic flow of a function acting as a correctness boundary with entry assertions, execution, exit assertions, and fail-stop handling.
In["Input Arguments"] --> Pre["Assert Preconditions<br/>& State Relations"] Pre -->|"Invariant Violated"| Crash["Immediate Shutdown<br/>(Fail-Stop)"] Pre -->|"Valid"| Body["Execute Core Function Logic"] Body --> Post["Assert Postconditions<br/>& Return Values"] Post -->|"Invariant Violated"| Crash Post -->|"Valid"| Out["Return Verified Result"] ```
- Software must make a hard reliability guarantee: execute 100% safely within expected parameters or shut down instantly. - Assertion practices are inspired directly by Gerard Holzmann's NASA Power of 10 rules for safety-critical systems. - Function arguments, parameter relationships, and return values are rigorously validated via entry and exit assertions. - The fundamental purpose of procedural abstraction is reframed around enforcing program correctness. - Codebases saturated with tripwire assertions become self-verifying engines that catch subtle logic bugs early.
---
[28:13](https://www.youtube.com/watch?v=w3WYdYyjek4&t=1693s) The Negative Space: Programmers vs. Hackers
Assertions enforce what an engineer expects to happen, but that highlights a critical division between ordinary programmers and hackers. Most programmers dwell exclusively in positive space: they want their code to work, operate with partially formed mental models, and assume that passing the compiler equates to correctness. Joran forcefully pushes back against this mindset: “I'll say it again, if it compiles, you've just begun.” In systems programming, having code compile is merely table stakes.
Hackers, by contrast, thrive entirely in the negative space. They do not enter through the front door, nor do they honor the compiler's assumptions; they actively attack edge cases and exploit scenarios the original author never conceived—much like the Mario runner finding a roof exploit that Nintendo engineers never imagined. When designing distributed systems, the operational environment itself acts as a relentless adversary. The network, disk bit rot, and unpredictable node crashes behave exactly like hackers, constantly probing the negative space of your assumptions. Surviving that environment requires adopting the adversarial mindset before deploying code.
- Standard programmers work in positive space, frequently assuming that compiling code implies logical correctness. - In TigerStyle and resilient systems programming, successful compilation marks only the starting line of engineering. - Hackers operate in negative space, searching for unhandled permutations and logic gaps that lie outside the author's expectations. - Distributed systems environments behave essentially like adversarial hackers, relentlessly exploiting unhandled edge cases, disk faults, and network partitions.
[30:00](https://www.youtube.com/watch?v=w3WYdYyjek4&t=1800s) Thinking Like a Hacker and Bounding Loops
The production environment is inherently complex and adversarial, constantly finding novel ways to stress and break software. Developers must not merely plan for what they expect, but cultivate an adversarial mindset and ask what they do *not* expect. By determining what would be genuinely surprising or anomalous, an engineer can discover where the boundaries of the system truly lie and assert them. If an invariant is drawn too tightly, an assertion will trip, prompting the team to correct their mental model and discover the actual shape of the problem space.
A prime example is the ubiquitous unbounded `while (true)` loop used in data parsing. Greef asks the audience for a show of hands on who has stopped to consider the expected runtime of a `while (true)` loop they authored, noting that developers rarely interrogate whether a loop might run for an hour or spin infinitely, burning compute. Instead of unbounded loops, systems should rely on finite loops bounded by fixed minimum and maximum iterations. Setting an upper bound prevents runaway execution that can deadlock production systems, while establishing a lower bound ensures that the loop actually executes the required minimum work. As Joran Dirk Greef emphasizes: “So think like a hacker, you have to assert the lower bound.”
[32:15](https://www.youtube.com/watch?v=w3WYdYyjek4&t=1935s) No Limits, 90s Dance Hits, and Static Memory Allocation
Greef laments that modern software architecture often mirrors the 1990s dance-pop anthem by 2 Unlimited: *No Limit*. When software is built without rigid boundaries, it ceases to be a comprehensible black box with defined edges and turns into an amorphous, "wobbly blob." Hardware resources—memory, disk, network, CPU, and concurrency—are inherently finite. Making these constraints explicit in code is essential to handling edge cases, designing effective backpressure, and managing load spikes, a principle central to Chapter 8 ("Handling Overload") of Google's *Site Reliability Engineering* book.
```mermaid flowchart TD accTitle: Error Taxonomy and Handling Strategy accDescr: Contrasting operational errors that require graceful handling with programming bugs that require assertion crashes.
subgraph OperationalErrors["Operational Errors (Expected Runtime Conditions)"] A["Resource Depletion / High Load"] --> B["Apply Backpressure"] B --> C["Reject New Connections / Shed Load"] C --> D["Keep Server Operating"] end
subgraph ProgrammingBugs["Programming Errors (Bugs / Invariant Violations)"] E["Violated Invariant / Loop Bound Exceeded"] --> F["Trigger Assertion Failure"] F --> G["Crash Process Immediately"] G --> H["Prevent Silent State Corruption"] end ```
Drawing directly from NASA's *Power of 10* rules for safety-critical systems, TigerBeetle enforces hard limits across its entire architecture: - TigerBeetle performs zero dynamic memory allocation after the startup phase. - All memory requirements are statically calculated up front, covering every message buffer permutation required by the consensus protocol. - The storage engine's memory allocations for compaction pipelines and concurrent disk/network tasks are fixed at initialization. - Every resource is strictly bound, making the system predictable and vastly simpler to reason about under stress.
As Joran Dirk Greef states: “A system has limits and you may as well make those limits explicit so that your design can handle all the edge cases.”
[35:00](https://www.youtube.com/watch?v=w3WYdYyjek4&t=2100s) Test Velocity and the Sherlock Holmes Cache Bug
While upfront sizing makes the initial implementation phase slower, the payoff is a lean and highly performant codebase. Because static sizing forces developers to account for every byte, the resulting software avoids memory waste and can end up ten times more memory-efficient than dynamically allocated alternatives. Once those structural limits are defined in code, developers can assert against them continuously, turning assertions into an immense force multiplier for testing velocity. TigerBeetle runs thousands of assertions actively in production builds, providing immediate feedback whenever reality diverges from system invariants.
Greef shares a concrete example involving cache validation—one of the classic hard problems in computer science. In TigerBeetle, when a block of data is read from the in-memory cache, the system periodically reads the corresponding block from disk to assert that the cached data perfectly matches persistent storage without subtle cache-divergence issues. During benchmarking on an unusual workload, this cache assertion fired: - The team immediately knew a critical invariant had failed, despite having no explicit unit test coverage for that specific workload sequence. - Team member Jamie spent eight intensive hours tracking down the root cause, documenting the investigation in a blog post that Greef describes as reading like a Sherlock Holmes mystery novel. - Without that production-active assertion, the flaw likely would have slipped into production undetected for months or years before causing catastrophic data loss or corruption.
Reflecting on the power of this pattern, Joran Dirk Greef observes: “And that's because without this assertion, the bug might have taken months or years just to catch with catastrophic consequences.”
[37:28](https://www.youtube.com/watch?v=w3WYdYyjek4&t=2248s) Language Defaults and the Flaw in Go's Assertion Philosophy
Despite their proven utility in catching catastrophic bugs early, assertions are widely neglected in mainstream codebases. Greef attributes this directly to programming language ergonomics and design philosophies, invoking computer scientist Barbara Liskov's observation that communicating new ways of thinking to programmers requires giving them new languages to think in. Too many existing languages either omit assertions entirely or disable them by default in production or optimized release builds—precisely where bugs possess the greatest potential to inflict catastrophic damage.
Greef highlights the Go programming language to illustrate this design failure. While TigerBeetle utilizes Go for operational tooling and respects many of its qualities, the official Go FAQ explicitly defends the absence of language-level assertions by asserting that programmers use them as an improper crutch to bypass disciplined error handling. Greef dismantles this perspective on multiple fronts: - Assertions are not syntactic conveniences for error handling; they are mission-critical verifications of programmatic correctness. - The fact that some developers misuse assertions does not justify stripping the tool from the entire ecosystem. - Most fundamentally, the Go stance conflates operational errors with programming bugs.
Operational errors are runtime conditions that software must handle gracefully—for example, if a server experiences memory pressure or fails an allocation, it should stop accepting new network connections and shed load rather than panicking globally. Conversely, programming errors are bugs that violate fundamental invariants, rendering the program's internal state untrustworthy. As Joran Dirk Greef puts it: “And most of all, the answer it conflates operating errors, which are errors that you can handle, with programming errors or bugs, which are errors you can't.”
[40:00](https://www.youtube.com/watch?v=w3WYdYyjek4&t=2400s) Assertions, Crash-Only Safety, and Checked Arithmetic in Zig
Joran Dirk Greef tackles a classic developer objection regarding assertions and system failures: if a program can detect a programming error, why not catch it and handle it gracefully? His answer is that the premise is fundamentally flawed. Joran explains that “by definition, bugs can’t be predicted.” If a bug could be anticipated and handled cleanly, it should simply be fixed in the code from the outset. Once an invariant fails, the application has entered “undefined logic territory,” and attempting to limp along risks catastrophic data corruption.
```mermaid flowchart TD accTitle: Assertion Failure and Crash Recovery accDescr: Flow diagram showing how assertion failures trigger safe crashes and rapid recovery in distributed databases.
subgraph BugEncounter["Bug Encountered"] A["Programming Error Detected"] --> B{"Try to Handle Gracefully?"} B -->|"No: Undefined Territory"| C["Assert Fails / Immediate Crash"] B -.->|"Anti-pattern"| Bad["Data Corruption & Latent Bugs"] end
subgraph ClusterRecovery["Distributed Recovery"] C --> D["Fault-Tolerant Cluster"] D --> E["Restart Clean Process"] E --> F["Higher Availability & Visibility"] end ```
Addressing skeptics who claim mission-critical databases cannot afford to crash, Joran argues that correctness must trump raw uptime. In high-throughput financial systems processing tens of thousands of transactions per second, failing fast is the only safe choice. Joran emphasizes: “The question is not if you shut down, but look, how quickly can we? Because, you know, if you're going to shut down manually or automatically, that could be the difference between your business staying in business or going out of business.” In distributed systems running multiple replicas, a crash does not sacrifice high availability; rather, restarting an errant process purges the bad state and exposes latent bugs so they can be fixed.
Turning to language choice, Joran praises Zig as the ideal systems language for building database engines because databases are fundamentally exercises in memory manipulation. Unlike languages that disable safety checks or omit assertions by default, Zig actively enforces safety primitives. Andrew Kelley and the Zig compiler lay tripwires throughout the generated code, turning integer overflows into immediate assertion failures with clear stack traces. Joran argues that checked arithmetic enabled by default in safe builds should be the mandatory standard across all programming languages, calling any alternative “madness.” Furthermore, Zig allows developers to run arbitrary logic at compile time (`comptime`), verifying structural integrity and invariants before tests or production environments ever run.
- Assertions catch bad code and logic bugs, not runtime operational errors like network disconnects. - In distributed databases, fault tolerance allows processes to crash safely and reboot without compromising availability. - Zig enforces checked arithmetic by default in safe build modes; developers can selectively opt out with explicit block annotations where 100% test coverage exists. - Zig's compile-time evaluation (`comptime`) executes real code—not macro expansions—allowing design invariants to be verified during compilation.
---
[45:00](https://www.youtube.com/watch?v=w3WYdYyjek4&t=2700s) Assertions as Living Documentation and the TigerBeetle Simulator
Beyond preventing bugs, assertions serve as a powerful teaching mechanism and architectural blueprint. As engineering teams scale, assertions define both the positive space (the intended logic) and the negative space (states that should never exist). Joran reflects on his own career, sharing that he has been coding for 28 years but only truly grasped the power of assertions over the past seven. To bridge that gap, TigerStyle leverages a deterministic simulation testing framework—a virtual "time machine" capable of advancing time at high speeds.
This simulator models the physical environment of a TigerBeetle cluster entirely in memory. It subjects nodes to extreme, adversarial conditions: injecting simulated network latencies, partitioning nodes, and corrupting storage across all disk replicas simultaneously. Throughout these simulated fault injections, the framework continuously verifies strict serializability and asserts that no data corruption occurs. Joran highlights the sheer productivity gains unlocked by deterministic testing: “When we first switched on our simulator, we found and fixed 30 bugs, like crazy distributed bugs, in just three weeks. Every day was just adrenaline.”
Joran shifts into a live walkthrough of `simulator.zig` from the TigerBeetle repository. He points to how Zig constructs the distributed database cluster at compile time by assembling pluggable, modular components into an in-memory replica. While joking that viewing text-based terminal logs might not be as visually flashy as a Super Mario demo, he stresses that deterministic execution seeded by a single pseudorandom number allows developers to reproduce and diagnose intricate, distributed race conditions within seconds.
```mermaid flowchart LR accTitle: TigerBeetle In-Memory Simulator Architecture accDescr: Diagram illustrating how the TigerBeetle simulator passes pluggable components to build an in-memory replica cluster driven by a random seed.
subgraph Config["Deterministic Seed & Config"] Seed["Random Seed"] Time["Virtual Time Source"] end
subgraph Assembly["Compile-Time Assembly (comptime)"] SM["State Machine Logic"] MB["Simulated Message Bus"] Storage["Faulty Disk Backend"] end
subgraph Cluster["In-Process Simulation"] Replica["In-Memory Replicas"] Check{"Assert Invariants & Strict Serializability"} end
Config --> Assembly Assembly --> Replica Replica --> Check ```
- High-density assertions capture the negative space of a program, explicitly documenting invariant violations directly inside the codebase. - The TigerBeetle simulator models network faults, storage delays, and concurrent disk corruption across all replicas up to theoretical limits. - The simulator discovered 30 complex distributed bugs in its first three weeks of operation, saving an estimated hundreds of engineering person-years. - In `simulator.zig`, components like the state machine, message bus, storage backend, and time source are supplied as pluggable compile-time parameters to construct the replica cluster.
[50:00](https://www.youtube.com/watch?v=w3WYdYyjek4&t=3000s) Deterministic Simulation Testing: The VAPOR Simulator
Joran Dirk Greef describes how TigerBeetle designs and controls its network simulation environment. Instead of relying on real-world randomness, the simulator explicitly configures latency distributions, packet drops, network partitions, and asymmetric latencies and bandwidth between nodes where one replica talks slower to another than in reverse. However, TigerBeetle goes far beyond standard network faults: “This is very, I don't know of other systems that really do storage fault corruption on the read and write paths.” While other systems rarely inject storage corruption during active read and write operations, TigerBeetle actively corrupts disk operations at high rates: "With TigerBeetle we just like to corrupt like 30% of the time, like everything... If you're going to read something, 30% of them corrupt it. Write something? Let's just write it to the wrong place. It's just more fun, right?"
```mermaid flowchart TD accTitle: TigerBeetle VAPOR Simulation Pipeline accDescr: Deterministic fault injection simulator testing client requests and replica state transitions against strict serializability.
subgraph Inputs ["Deterministic Seed"] S["Seed & Config"] C["27 Clients"] end
subgraph Faults ["Injected Faults"] NF["Network Faults<br/>(34% message replays, asymmetric latency, drops)"] SF["Storage Faults<br/>(3% read corruptions, 3% write corruptions)"] end
subgraph Cluster ["3 Replicas (VSR)"] R0["Replica 0"] R1["Replica 1"] R2["Replica 2 (Leader)"] end
subgraph Oracle ["Global Checker"] O["Global Omniscient Oracle<br/>(Strict Serializability, Liveness, Consensus Checks)"] end
S --> NF S --> SF C -->|"Requests"| Cluster NF -.-> Cluster SF -.-> Cluster Cluster -->|"State Transitions"| O ```
To demonstrate this in action, Greef boots up TigerBeetle's simulation tool, named the VAPOR (Viewstamped Operation Replicator), paying homage to the WOPR war simulator from the movie *WarGames*. Running from a deterministic random seed, the simulator spins up a logical TigerBeetle database of three replicas subjected to requests from 27 simulated clients. For this specific run, the simulator replays network messages 34% of the time to heavily test idempotency, while corrupting 3% of reads and 3% of writes. As it executes, the terminal visualizes Replica 1 transitioning across states, which the simulator immediately verifies against strict serializability. Because the simulator possesses omniscient global visibility over all in-flight client requests, it knows exactly what the valid state of the entire universe must be. When Replica 0 experiences issues, the cluster elects Replica 2 as the new primary, advances state, and completes the simulation successfully without consensus violations, deadlocks, or liveness failures.
- **VAPOR Name Origin:** Named after the Viewstamped Operation Replicator, referencing the WOPR supercomputer from the film *WarGames*. - **Cluster Parameters:** 1 deterministic seed driving 3 replica nodes and 27 client workloads. - **Fault Probabilities in Run:** 34% network message replay rate, 3% storage read corruptions, and 3% storage write corruptions. - **Verification Capabilities:** Real-time checking of strict serializability, consensus violations, and liveness/deadlock detection.
---
[52:45](https://www.youtube.com/watch?v=w3WYdYyjek4&t=3165s) Leveling Up: Compiling TigerBeetle to WASM for Browser Games
Greef transitions from the terminal-based simulator to a skunkworks project created with Joey and Fabio, whom Greef met in Milan in April of the previous year. Taking advantage of the development velocity unlocked by TigerStyle, the team decided to turn deterministic simulation testing into an interactive retro video game. Because TigerBeetle is written in Zig, the entire distributed database engine compiles directly to WebAssembly (WASM), allowing real TigerBeetle production code to execute inside a web browser.
Inspired by retro gaming cartridges, the team created three levels: **City Breeze**, **Red Desert**, and **Radioactive**. Greef loads the first level, City Breeze, which represents the utopian condition assumed by typical distributed systems code where nothing ever fails: no crashes, zero network faults, and perfect conditions. As upbeat chiptune synth music blares through the speakers, Greef emphasizes the role of sound in engineering: “You need to have a playlist called Sega Tracks or something, it really like motivates the team.” The animated beetles descend to the field and rapidly complete their Viewstamped Replication rounds under pristine conditions. The test passes effortlessly, prompting the audience to cheer and demand they level up.
---
[55:00](https://www.youtube.com/watch?v=w3WYdYyjek4&t=3300s) Red Desert: Network Partitions and Mexican Standoffs
The presentation transitions to the second level, **Red Desert**, which introduces an ominous change in soundtrack and a severe jump in difficulty. This level models hostile network dynamics akin to Jepsen testing: dropped packets, partitioned nodes, and asymmetric delays. Two beetles vie for leadership in what Greef calls a "Mexican standoff," triggering leader elections and deterministic view changes when the current primary crashes. The simulator UI allows the player to select units in real time, like an RTS game such as *StarCraft*, while watching individual nodes transition between preparing, backing up, and acknowledging operations.
Greef points out one of the beetles on screen, comparing it to Captain America as it assumes leadership, before watching the primary crash and another beetle recognize itself as the next rightful leader. Greef highlights a structural advantage of their consensus protocol: “VSR is a lot cooler I think because you can predict the next leader so we exploit that entirely.” Unlike consensus protocols that select leaders randomly or endure split votes, Viewstamped Replication features deterministic primary succession. Despite severe network partitions—faults that Kyle Kingsbury (Aphyr) would love—TigerBeetle completes all state transitions without crashing or violating consistency.
---
[57:38](https://www.youtube.com/watch?v=w3WYdYyjek4&t=3458s) Level 3: Radioactive Disk Corruption and Cosmic Rays
Rather than stopping at network testing, Greef launches the third and most grueling level: **Radioactive**. The level introduces simulated radioactive green slime and cosmic ray zaps that directly corrupt storage writes and reads while operations are in flight. As the music intensifies, bright visual lightning strikes represent cosmic rays hitting disks across the replica cluster.
The cluster sustains intense read/write disk corruptions across 49 consecutive operations while executing in-browser via WASM. Despite repeated storage errors, TigerBeetle detects the corrupted blocks, fetches repaired data from peers, and completes the entire test run without data loss or panics. Greef credits Joey, Fabio (who built the browser game engine), and Bia from Brazil (who created the pixel art animations) for building the visualizer.
---
[01:00:00](https://www.youtube.com/watch?v=w3WYdYyjek4&t=3600s) Closing Systems Distributed '23: Paying It Forward
Greef concludes the talk and brings the Systems Distributed '23 conference to a close, expressing gratitude to the attendees, the Cape Town venue, and the international speaker lineup. He shares an anecdote about inviting Will: “I emailed him to invite him and he replied he said, ‘Yeah, I've booked my flight,’ and I had barely given him the dates or anything.”
Greef thanks speakers who traveled from across the globe to attend: - Will and other speakers traveling from the United States and Europe - Tiamat and Terapai traveling from India - Imre traveling from Hungary - Lindsay traveling from Salt Lake City
Greef extends special praise to his TigerBeetle co-founder Phil (Ful), who organized the logistics and operations to make the conference run smoothly, as well as the wider TigerBeetle team. In closing, Greef reflects on the community ethos behind both TigerBeetle and the conference, noting that speakers traveled entirely at their own expense to share their knowledge with the local ecosystem: “So I guess that's the final lesson of TigerStyle: pay it forward and have fun as you build systems distributed.”
## Notable quotes > [15:00](https://www.youtube.com/watch?v=w3WYdYyjek4&t=900s) You want to try and sketch as many different solutions as you can in the design phase because sketches are cheap, rewrites are expensive. — Joran Dirk Greef
> [32:15](https://www.youtube.com/watch?v=w3WYdYyjek4&t=1935s) A system has limits and you may as well make those limits explicit so that your design can handle all the edge cases. — Joran Dirk Greef
> [28:13](https://www.youtube.com/watch?v=w3WYdYyjek4&t=1693s) If it compiles, you've just begun. — Joran Dirk Greef
> [10:00](https://www.youtube.com/watch?v=w3WYdYyjek4&t=600s) I think the discipline is to defer coding, to wait, until you have the design roughly right. — Joran Dirk Greef
> [20:00](https://www.youtube.com/watch?v=w3WYdYyjek4&t=1200s) The big point is just be explicit upfront about what your system should tolerate. — Joran Dirk Greef
## Mentioned Joran Dirk Greef, TigerBeetle, Gerard Holzmann, Spin, NASA JPL, James Cowling, Will Wilson, Antithesis, Hardcore, Zig SHOWTIME, Amplify, Coil, Adrian, Don, Rust, GraphQL, Edsger W. Dijkstra, The Lord of the Rings, PostgreSQL, Kyle Kingsbury, Jepsen, Tron, Kevin Flynn, Daniel Lanois, Bob Marley, The Beatles, Fed, Super Mario, Google, 2 Unlimited, Jamie, Barbara Liskov, Go, Zig, Andrew Kelley, Joey, Fabio, WebAssembly, StarCraft, Bia, Phil, Tiamat, Terapai, Imre, Lindsay, Systems Distributed '23
## Video description Our final talk from Systems Distributed '23: https://systemsdistributed.com.
Join the chat at slack.tigerbeetle.com/invite!
[Source](https://www.youtube.com/watch?v=w3WYdYyjek4)