FoundationDB Alum Wilson Launches Startup to Automate Test Creation
The Linux Foundation
Because developers cannot mentally trace the complex failure states they fail to code around, software engineering must transition from manual test authoring to autonomous machine generation.
AI-assisted summary of Autonomous Testing and the Future of Software Development - Will Wilson — original source. Check the original for full context.
Source published:
Key ideas
- While industry analysts estimate testing accounts for roughly 30% of software development effort, real-world friction—including triage, downstream library mitigations, and outage fallout—pushes the true cost past 50% of developer time, squandering trillions globally. (4:52)
- The software industry conflates automated test execution via CI/CD with automated testing, masking the reality that test creation remains an entirely manual, painstakingly human bottleneck ripe for disruption. (14:05)
- Branch coverage yields a false sense of security in distributed systems, failing to catch critical concurrency and timing defects across an execution history search space bounded mathematically by 256 raised to the power of input bytes. (10:44)
- Autonomous testing leverages cloud elasticity and embarrassing parallelism to compress 'bug latency'—the time between writing a defect and detecting it—from months to seconds, dramatically reducing remediation costs. (27:17)
Detailed notes
## Background Published on YouTube by The Linux Foundation on December 20, 2018, this presentation features former FoundationDB engineer Will Wilson discussing the trajectory of software verification. No show notes or description were provided on the original video page.
## TL;DR Will Wilson argues that software engineering is still in its primitive infancy, where flawed, fragile testing practices currently squander over half of developer productivity. He proposes shifting from traditional automated test execution to 'autonomous testing'—systems that automatically synthesize and explore test cases, inspired by FoundationDB's deterministic simulation methods. Despite the immense challenges of searching vast execution state spaces and designing proper test oracles, Wilson asserts that autonomous testing dramatically compresses bug latency and catches catastrophic edge cases humans cannot fathom. He concludes by announcing a new venture co-founded with Dave Scherer to make autonomous testing turnkey and universally adoptable across the industry.
## Atmosphere & style The presentation carries a dryly humorous, conversational, and intellectually ambitious tone, kicking off with comedic running gags about historical technology charts before diving into rigorous technical critique. Wilson blends developer commiseration with genuine visionary optimism, treating audience members like fellow pioneers navigating the early frontiers of computing. The session feels both grounded and inspiring, moving effortlessly from the day-to-day misery of flaky CI runs into an energizing manifesto on radically transforming how software is verified.
## Bulletin - [04:52](https://www.youtube.com/watch?v=fFSPwJFXVlw&t=292s) While industry analysts estimate testing accounts for roughly 30% of software development effort, real-world friction—including triage, downstream library mitigations, and outage fallout—pushes the true cost past 50% of developer time, squandering trillions globally. - [14:05](https://www.youtube.com/watch?v=fFSPwJFXVlw&t=845s) The software industry conflates automated test execution via CI/CD with automated testing, masking the reality that test creation remains an entirely manual, painstakingly human bottleneck ripe for disruption. - [10:44](https://www.youtube.com/watch?v=fFSPwJFXVlw&t=644s) Branch coverage yields a false sense of security in distributed systems, failing to catch critical concurrency and timing defects across an execution history search space bounded mathematically by 256 raised to the power of input bytes. - [27:17](https://www.youtube.com/watch?v=fFSPwJFXVlw&t=1637s) Autonomous testing leverages cloud elasticity and embarrassing parallelism to compress 'bug latency'—the time between writing a defect and detecting it—from months to seconds, dramatically reducing remediation costs.
## Key takeaways - [00:00](https://www.youtube.com/watch?v=fFSPwJFXVlw&t=0s) Moore's law does not demonstrate computing exceptionalism; steep exponential gains reflect early primitive baselines, mirroring historical curves seen in textile spinning, steam engines, and synthetic diamonds. - [00:00](https://www.youtube.com/watch?v=fFSPwJFXVlw&t=0s) Because software engineering has not yet plateaued along a traditional logistic curve, present-day developers have the opportunity to drive radical transformations across foundational practices. - [03:18](https://www.youtube.com/watch?v=fFSPwJFXVlw&t=198s) The most fertile ground for transformative engineering breakthroughs lies where daily workflows are most painful, making software testing the prime target. - [04:52](https://www.youtube.com/watch?v=fFSPwJFXVlw&t=292s) Real-world testing overhead routinely exceeds 50% of engineering time once developers account for writing suites, debugging regressions, working around upstream library bugs, and firefighting outages. - [07:44](https://www.youtube.com/watch?v=fFSPwJFXVlw&t=464s) Non-deterministic, flaky test suites induce alert fatigue and cynicism, training teams to dismiss failures until catastrophic defects slip into production. - [07:44](https://www.youtube.com/watch?v=fFSPwJFXVlw&t=464s) Fragile tests over-specify incidental implementation details such as animation durations or tick rates rather than meaningful system invariants, causing tests to fail open or be disabled. - [10:44](https://www.youtube.com/watch?v=fFSPwJFXVlw&t=644s) A fundamental human cognitive paradox limits testing: if developers possessed the foresight to anticipate every edge case and concurrency interleaving required for complete test suites, they would write bug-free code initially. - [10:44](https://www.youtube.com/watch?v=fFSPwJFXVlw&t=644s) High branch coverage cannot guarantee reliability in distributed systems or Turing-complete software because it fails to capture process execution ordering and asynchronous timing bugs. - [14:05](https://www.youtube.com/watch?v=fFSPwJFXVlw&t=845s) Autonomous testing fundamentally redefines quality assurance by automating the generation of test cases rather than merely automating the execution of human-scripted tests. - [16:13](https://www.youtube.com/watch?v=fFSPwJFXVlw&t=973s) Dismissing automated testing via the Turing halting problem is invalid; software verification tools do not need to be infallible oracles, but merely need to outperform human debugging capability. - [19:54](https://www.youtube.com/watch?v=fFSPwJFXVlw&t=1194s) Finding software defects requires navigating an astronomically vast state space of 256 raised to the power of received input bytes, demanding intelligent generative systems over naive random fuzzing. - [21:07](https://www.youtube.com/watch?v=fFSPwJFXVlw&t=1267s) Autonomous simulation testing historically required systems to be architected from day one as deterministic pure functions fed by a pseudo-random seed, presenting severe hurdles for legacy codebases. - [23:46](https://www.youtube.com/watch?v=fFSPwJFXVlw&t=1426s) Autonomous testing systems require sophisticated test oracles capable of validating complex transactional guarantees and system invariants beyond simple crashes or inline assertions. - [24:54](https://www.youtube.com/watch?v=fFSPwJFXVlw&t=1494s) Machine-driven test generation and human-authored tests are orthogonal and complementary: humans verify expected workflows, while algorithmic testing probes unintuitive edge cases. - [27:17](https://www.youtube.com/watch?v=fFSPwJFXVlw&t=1637s) The cost and cognitive burden of fixing bugs scale exponentially with time, making the embarrassingly parallel execution of autonomous testing vital for keeping bug latency near zero. - [29:49](https://www.youtube.com/watch?v=fFSPwJFXVlw&t=1789s) Autonomous testing represents a potential epochal leap on par with high-level compilers, poised to relegate manual test authoring to specialized, rare edge cases.
## Detailed walkthrough ### [00:00](https://www.youtube.com/watch?v=fFSPwJFXVlw&t=0s) Moore’s Law and the Illusion of Computing Exceptionalism
Will Wilson introduces himself by noting that he worked at FoundationDB back in “the ancient mists of time” and is now working on something different, setting the stage for a talk on the future of software development. As someone tasked with giving a big-picture talk at a tech conference, he jokes that he is contractually obligated to begin with a slide depicting Moore’s law. Wilson advances to a slide displaying an exponential curve running up and to the right, labeling the axes as years and clock speed, only to double-check his notes in mock confusion. The graph is actually tracking historical productivity in weaving and spinning. He tries a second slide featuring a steep logarithmic scale, joking that this one must surely be Moore’s law, only to discover it illustrates steam engine efficiency over time. On his third attempt, showing a graph spanning the exact decades during which Moore’s law operated, the audience is left guessing before Wilson reveals it plots the industrial synthesis of diamonds.
```mermaid flowchart TD accTitle: Technology Growth Trajectory accDescr: Diagram contrasting early primitive exponential growth with the mature logistic curve flattening.
subgraph EarlyStage ["Early Historical Stage"] A["Primitive Baseline Technology"] -->|"Rapid iteration & low-hanging fruit"| B["Exponential Growth Phase"] end
subgraph MatureStage ["Mature Plateau Stage"] B -->|"Physical & practical limits"| C["Logistic Curve Flattening"] C --> D["Diminishing Marginal Returns"] end ```
Wilson uses this comedic sequence of historical charts to challenge a core piece of mythology common among software engineers: the idea that computing is uniquely exceptional in human history. Technologists often point to Moore’s law and ask where else humanity has ever witnessed exponential gains in baseline hardware capabilities. Wilson's point is that such explosive progress is found across nearly every technological domain in its infancy. As Wilson explains: “The reason is that exponential growth is really really easy when what you have is really really primitive and so the fact that we observe this Moore's Law relationship maybe still is actually not telling us that software is special it's telling us that software is early right it's telling us that we're at the beginning that we haven't trailed off yet into a logistic curve which is what usually happens in technology and I actually find this fact to be incredibly empowering right it means it means we we get the chance to be pioneers too it means those guys in the 60s and the 70s didn't have all the fun it means there's a lot more to do and I think one implication of this is it also means that we should be looking for the possibilities of radical transformations in software engineering right things that will completely change how what it means to be a software engineer what it means to write programs.”
- Moore’s law curves are mirrored historically in spinning and weaving output, steam engine thermal efficiency, and synthetic diamond production. - Exponential gains reflect primitive early baselines rather than something intrinsically unique to silicon or code. - Because computing has not yet flattened out into a standard logistic curve, present-day developers remain pioneers on a par with early practitioners from the 1960s and 1970s.
---
[03:18](https://www.youtube.com/watch?v=fFSPwJFXVlw&t=198s) The Opportunity for Radical Transformation in Software Development
Being so early in the broader timeline of software engineering carries two massive implications. First, substantial low-hanging fruit still exists across the engineering landscape. Second, the vast majority of all software that human civilization will ever write has not been written yet. Any architectural, tooling, or process improvements established today will pay dividends over an extraordinarily long future time horizon. These factors justify taking a hard, foundational look at how software is constructed. Wilson notes that while there are many candidate fields for radical transformation—such as category theory yielding revolutionary programming languages, or drastic demographic and economic shifts altering who writes software—he chooses to bypass those topics entirely to focus squarely on testing.
To uncover where a radical transformation is most likely to happen, Wilson suggests a practical heuristic: identify where current engineering practices are the most painful, grueling, and inefficient. Software testing fits this description perfectly. As Wilson puts it: “And if you want to look for the potential for radical transformations one way of doing that one pretty good heuristic is to look at where things are currently the most absolutely awful and terrible and painstaking and brutal and I think testing kind of qualifies right.”
---
[04:52](https://www.youtube.com/watch?v=fFSPwJFXVlw&t=292s) The Real Economic Toll of Testing and the FoundationDB Perspective
Industry analyst firms like Gartner or Forrester have published reports estimating that roughly 30% of all software development effort is consumed by testing and quality assurance. Wilson argues that this figure is an absurd underestimate once developers tally the full reality of their workdays. A true accounting includes writing tests, fixing the bugs caught by tests, triaging the bugs that tests missed, spending days mitigating defects in underlying third-party libraries and layers written by “those jerks” who refused to fix them, and cleaning up after production outages and customer-facing fallout. Wilson estimates that for himself and the average engineer, testing-related concerns eat up at least 50% of developer time.
```mermaid flowchart TD accTitle: Real Scope of Testing Overhead accDescr: Flowchart illustrating the full compounding costs of testing and software defects.
subgraph DirectCost ["Direct QA Activities"] A["Authoring Tests"] B["Fixing Test-Found Bugs"] end
subgraph IndirectCost ["Mitigation & Remediation"] C["Debugging Production Bugs (Missed by Tests)"] D["Compensating for Downstream Vendor & Layer Bugs"] E["Triage, Outages, and Economic Damage"] end
DirectCost --> F["Visible Effort: ~30% (Analyst Estimates)"] IndirectCost --> G["Total Real Effort: >= 50% Developer Time"] ```
Scaled across millions of software engineers worldwide, this friction represents trillions of dollars in squandered productivity. Wilson emphasizes that addressing this drag is the motivation for speaking at a FoundationDB gathering. Unlike the broader software ecosystem, FoundationDB had an unusually tranquil relationship with testing, where verifying software was not an all-consuming ordeal that drained developer morale. Rather than repeating familiar FoundationDB war stories—which colleagues like Evan and Ben had already presented—Wilson explains that his goal is to recontextualize FoundationDB's methods within a wider framework of testing philosophies. As Wilson asserts: “If you could take that and you could eliminate it or make it much more efficient much smaller fraction of what we do that would be a radical transformation of the software industry.”
- Analyst estimates place testing and QA at roughly 30% of development effort, but real-world overhead routinely exceeds 50%. - Compounding costs include downstream defect mitigation, undetected production defects, post-outage firefighting, and economic disruption. - Globally, developer time lost to testing friction amounts to trillions of dollars of potential output. - FoundationDB decoupled testing from developer misery, treating verification as an asset rather than a resource drain.
---
[07:44](https://www.youtube.com/watch?v=fFSPwJFXVlw&t=464s) The Plagues of Modern Testing: Non-Determinism and Fragility
Wilson breaks down why the status quo of software testing is miserable, starting with the rampant issue of non-deterministic, flaky tests. The moment a test harness interacts with an external service or operates in an unisolated, noisy hardware environment, it begins failing intermittently—once every 100 runs, or once every 10,000 runs. This behavior destroys developer confidence: when a flaky test turns red, engineers cannot tell if their code broke or if the test is misbehaving. Developers fall into the dangerous habit of dismissing failures as known flakiness until an actual catastrophic bug slips through. While the FoundationDB engineering team made heroic efforts to claw back determinism via deterministic simulation, Wilson points out that the rest of the industry remains trapped in the dark ages, suffering from flaky test suites daily.
Beyond flakiness, Wilson highlights test fragility, where test suites inadvertently depend on arbitrary implementation details rather than meaningful invariants. He illustrates this using a 2D platformer game tested via recorded user keystrokes: holding the right arrow for 6.7 seconds, jumping for 3.2 seconds, and firing a weapon at an enemy. The test functions only until the slightest change occurs—such as tweaking a platform position, altering the physics tick rate, modifying input polling, or running the suite on a faster processor. The exact same vulnerability plagues UI testing and distributed systems verification, where minor early timing divergences derail the rest of the sequence into checking irrelevancies. As Wilson explains: “Another reason testing is terrible is that your tests are very very often fragile what do I mean by that I mean that your test comes to rely on properties of your system that are incidental in some way that are not the ones that you thought you were testing.” This persistent fragility breeds cynicism, leading teams to ignore test warnings—a pattern highlighted by a recent major outage post-mortem.
- Non-deterministic test suites undermine debugging confidence, creating alert fatigue where teams normalize and ignore failing runs. - FoundationDB avoided this via deterministic simulation, but most engineering teams lack deterministic execution environments. - Fragile tests over-specify incidental execution details (such as hardcoded sleep times, input durations, or strict timeline ordering) rather than system invariants. - Fragility appears across domains, from game physics and front-end UIs to distributed network traces, where small initial divergences cause tests to wander away from their intended assertions.
[10:44](https://www.youtube.com/watch?v=fFSPwJFXVlw&t=644s) The Trap of False Confidence and the Limits of Human Foresight
To illustrate the danger of fragile test suites, Will Wilson shares an anecdote about a giant, unnamed company where a massive system outage occurred. In a cruel twist of irony, the exact code check-in that introduced the catastrophic bug also disabled the very test designed to catch it. The developer had disabled the test because it was notoriously fragile, routinely breaking with every single check-in. This highlights a highly insidious failure mode: tests can "fail open," continuing to pass after a code change but doing so for entirely different, unintended reasons. This creates a dangerous illusion of security, leaving teams with a false sense of confidence.
Even if developers manage to write tests that are neither flaky nor fragile, they are still highly unlikely to test the right scenarios. Wilson argues there is an analytical proof for this limitation. Humans write tests because our brains are remarkably bad at tracing every possible branch of control flow, anticipating how asynchronous threads might interleave, or predicting where and how a distributed algorithm might fail. However, this exact cognitive limitation means we are also incapable of writing the very tests needed to cover these complex scenarios.
Because of this paradox, traditional tests are useful for catching regressions but almost entirely useless for uncovering "unknown unknowns." Attempting to solve this with branch coverage metrics is a dead end. Even if a distributed system achieves 100% branch coverage, it remains vulnerable to timing bugs—such as whether a piece of code executes on Process A before or after Process B—which branch coverage completely misses. Similarly, achieving 100% branch coverage on a Turing-complete system, like a Python interpreter, still leaves an infinite mountain of untested, complex behaviors.
* **The Outage Anecdote:** A developer at a massive tech company disabled a critical test because it broke on every check-in, directly leading to a giant system outage. * **The Human Cognitive Limit:** “If we were smart enough or had the right kind of brains to write all the tests we needed to write, then we would have just written the code correctly in the first place.” * **The Coverage Illusion:** 100% branch coverage cannot guarantee the absence of bugs in distributed systems or Turing-complete software.
[14:05](https://www.youtube.com/watch?v=fFSPwJFXVlw&t=845s) Redefining "Automated" Testing: The Shift to Autonomous Testing
The root of these testing failures is that what the industry calls "automated testing" is actually entirely manual. The only part we have automated is the absolute simplest step: running the tests via CI/CD tools like Jenkins. The tests themselves are still painstakingly, manually constructed by human beings. Wilson jokes that if any other industry employed armies of highly paid, intelligent professionals to perform such rote, repetitive, and predictable tasks without automation, venture capitalists would instantly declare it ripe for disruption. Yet, this remains the standard state of software engineering.
The true breakthrough at FoundationDB was not just their deterministic simulation engine, but a fundamental shift in philosophy. Instead of asking how to write individual tests for a new feature, the engineering team focused on building systems that could continuously generate new and interesting tests. Wilson defines this paradigm—the automated creation of tests, rather than just their automated execution—as "autonomous testing."
```mermaid flowchart TD accTitle: Manual vs Autonomous Testing Workflows accDescr: A comparison showing how traditional automated testing still relies on manual test creation, whereas autonomous testing automates both creation and execution. subgraph Traditional ["Traditional 'Automated' Testing"] A["Human Engineer"] -->|"Painstakingly writes"| B["Static Test Cases"] B -->|"Executed by"| C["CI/CD (e.g., Jenkins)"] end subgraph Autonomous ["Autonomous Testing"] D["Human Engineer"] -->|"Designs"| E["Test-Generating System"] E -->|"Continuously generates"| F["Infinite Dynamic Tests"] F -->|"Executed by"| G["Deterministic Simulator"] end ```
* **The Mindset Shift:** “Whenever we had a new piece of functionality, we didn't say, 'How can I write some tests to cover this?' It was more like, 'How can I write a system that will forever be generating new and interesting tests that will exercise this?'” * **The Definition of Autonomous Testing:** The automated creation of tests, distinguishing it from the mere automated execution of human-written tests.
[16:13](https://www.youtube.com/watch?v=fFSPwJFXVlw&t=973s) Glimmerings of Autonomy and the Turing Objection
While FoundationDB's approach was comprehensive, primitive forms of autonomous testing exist across the industry if one knows where to look. Fuzzing is a classic example: instead of manually writing unit tests for a network parser, developers use a fuzzer to feed millions of random byte strings into the program to see if it crashes. Similarly, functional programming communities (particularly Haskell developers) pioneered property-based testing. In this approach, developers define a specification of a data structure's contract and guarantees, and the computer is tasked with finding counterexamples. This technique was used extensively to test the FoundationDB document layer, which Apple recently open-sourced.
Despite the clear benefits of autonomous testing, its widespread adoption is slow. When asking why the entire industry hasn't adopted these methods, Wilson notes he is bound by NDAs from describing just how bad things get when companies don't use them. He frequently encounters a weak, pseudo-intellectual objection: that the Turing halting theorem mathematically proves software can never test software.
Wilson dismisses this argument as a trivial impossibility result, comparing it to developers who claim the CAP theorem makes it impossible to build a good distributed database. He points out that computers do not need to be perfect Turing Oracles to debug software; they simply need to outperform humans.
* **Fuzzing as Autonomy:** A primitive but highly effective form of autonomous testing used to secure parsers against untrusted network inputs. * **Property-Based Testing:** A method where computers generate counterexamples to a defined specification, heavily utilized in testing FoundationDB's document layer. * **The Turing Objection:** “Humans debug software; humans are not mystical Turing Oracles—if humans can do it, then so can computers.”
[19:54](https://www.youtube.com/watch?v=fFSPwJFXVlw&t=1194s) The True Challenge: Navigating an Inconceivable State Space
While the Turing halting theorem is a poor excuse for avoiding autonomous testing, it is closely linked to a very real and incredibly difficult engineering challenge: the sheer scale of the state space. When trying to find a bug, a testing system is implicitly searching through the space of all possible execution histories.
The size of this search space is mathematically defined as 256 raised to the power of the number of bytes of input the program has ever received. This is an inconceivably vast space. A naive testing tool could wander randomly through a tiny, insignificant corner of this space for the entire lifetime of the universe and never find a single bug. The core challenge of autonomous testing is designing systems smart enough to navigate this infinite expanse and locate the highly specific, interesting execution paths where bugs actually hide in a very short amount of time.
* **The State Space Formula:** The size of the execution history space is $256^{N}$, where $N$ is the number of bytes of input the program has received. * **The Scale of the Problem:** “The size of that space is 256 raised to the power of the number of bytes of input your program has ever received—that is a mind-bogglingly large space, that is inconceivably vast.”
[021:07] The Pitfalls of Autonomous Testing: Tuning and Architectural Form
Tuning an autonomous testing system is an excruciatingly delicate task. At FoundationDB, an immense amount of effort went into balancing parameters—a responsibility largely shouldered by co-founder Evan. Wilson points out that getting bugs to surface requires carefully "buggifying" components and tweaking random number generators so that processes crash frequently enough to expose rare edge cases, but not so often that test runs can never make forward progress. This calibration requires an engineer with deep architectural understanding. If done improperly, the worst possible failure occurs: the tests report green, giving a false sense of security. As Wilson stresses: “And if you get it wrong the failure mode is your tests say everything's good that's a really really scary failure mode right you need some kind of side channel to tell you that no actually everything is not good hopefully that side channel is not your customers.”
Beyond the difficulty of tuning, a far steeper barrier to adoption is that nearly all autonomous testing methods require the target software to be built in a very specific architectural form. FoundationDB’s deterministic simulation works by corralling a massive, stateful, hairy distributed system into acting like a pure function fed by a single pseudo-random seed. That was only feasible because the team possessed tremendous foresight from day one. Wilson laments that most developers in the industry simply cannot do this: they are maintaining legacy codebases—what he quips is "some ginormous java monstrosity from the 1990s"—or face commercial pressure where they cannot afford to spend a full year building a custom simulator before writing their first line of product code. Until techniques can be retrofitted onto existing systems, autonomous testing will remain out of reach for most.
```mermaid flowchart TD accTitle: FoundationDB Deterministic Simulation Model accDescr: Diagram showing how a random seed is fed into a deterministic engine wrapping a distributed system to produce repeatable test outcomes.
subgraph Inputs ["Test Generation"] Seed["Random Seed"] Knobs["Tuned Failure Rates<br/>(Crash frequency, latency, drops)"] end
subgraph Core ["Deterministic Simulation Box"] Wrapper["FoundationDB Simulation Engine<br/>(Pure function abstraction)"] Workload["Workloads & State Machines"] Wrapper --- Workload end
subgraph Results ["Test Oracle Verification"] Asserts["Invariants & ACID Checks"] Green["Clean Run / Progress"] BugFound["Deterministic Failure Repro"] end
Seed --> Wrapper Knobs --> Wrapper Workload --> Asserts Asserts --> Green Asserts --> BugFound ```
[023:46] Test Oracles: Knowing When a Bug Has Occurred
Autonomous tests can execute millions of operations, but they are utterly useless if they cannot detect when an error has actually happened. Wilson classifies this as the problem of "test oracles." In simple software domains, this is straightforward: a crash, an out-of-memory error, or a triggered assertion immediately signals a bug. However, in more nuanced application domains, failure modes are subtle. In a graphical interface, for example, a bug might mean a border rendered five pixels too far to the right, or an entire window unexpectedly turned purple.
FoundationDB ran into this exact dilemma with database internals. High-level transactional guarantees and ACID invariants were far too complicated to capture with simple inline assertions. To verify correctness, the engineering team had to build dedicated testing workloads that initialized specific database invariants, ran randomized concurrent operations, and then exhaustively checked whether the invariant guarantees held intact at the end of the run.
[024:54] Why Use Autonomous Testing: Economics, Orthogonality, and Elasticity
Despite these formidable hurdles, Wilson argues that software teams should aggressively pursue autonomous testing. The primary argument is raw economics: computers are dramatically cheaper than human software engineers, especially those in the San Francisco Bay Area. Using machines to generate and run test variations allows a team to perform exponentially more testing than human-written scripts ever could.
Crucially, autonomous testing is not intended to replace handcrafted unit or integration suites. Humans and computers bring complementary perspectives to problem-solving. Human-written tests target common architectural expectations and workflows, whereas computers act as "total aliens" that easily wander into the dark, bizarre edge cases humans struggle to imagine. Pairing both produces a uniquely robust synthesis. Moreover, cloud infrastructure makes machine testing elastic and instantaneous:
- **Cost:** Compute instances cost pennies compared to the salaries of Bay Area software engineers. - **Complementary coverage:** Humans test expected code paths; machines explore counter-intuitive edge cases. - **Zero onboarding latency:** Unlike hiring, training, and feeding human developers, cloud instances can be booted and terminated on Amazon EC2 on demand.
[027:17] Compressing Bug Latency and Embarrassing Parallelism
The most compelling, yet subtle, argument for autonomous testing is reducing "bug latency"—the elapsed time between the introduction of a defect and its discovery. Wilson explains that the difficulty and cost of fixing a bug grow exponentially with the time that passes after it is written. If a pre-commit hook flags an issue immediately, the author still holds the entire mental model in their head and has the exact diff right in front of them, making the fix painless.
Wilson contrasts this ideal scenario with an operational nightmare: a developer introduces a subtle bug that slips past standard tests and ships into several releases; months later that developer leaves the company, and years down the road an on-call engineer receives an emergency midnight call from their largest production customer. Diagnosing whether a bug even exists in that scenario requires orders of magnitude more effort than fixing it on day one. Autonomous testing solves this because it is embarrassingly parallel: teams can burn a million CPU seconds across thousands of distributed cores in just a matter of seconds, driving discovery latency down to near zero. Wilson highlights the transformative effect: “It's going to repay you in money and productivity and frustration and everything so fast it's going to change your life it's going to change it's it's gonna qualitatively change the kinds of things your team can do.”
[029:49] The Future Vision: Building the Next Epochal Leap
Reflecting on the overarching problem, Wilson asks what it will take to make autonomous testing the universal standard for all software development. He compares the potential shift to the invention of the compiler—a monumental advancement after which writing raw machine code became restricted only to rare, exotic edge cases.
To bridge this gap and eliminate the friction of adopting simulation and fuzzing techniques, Wilson reveals that he and Dave Scherer have co-founded a new company. Their stated mission is to solve the adoption challenges, make autonomous testing turnkey for every developer, and revolutionize software reliability across the entire industry. As Wilson enthusiastically puts it: “Dave Scherer over there and I have recently started a company and our goal is to try and figure out the answer to this question and then thereby making autonomous testing super easy thereby making it universal thereby radically transforming the software industry thereby ushering in a new golden age of peace and prosperity for all mankind.” Although the venture is in its earliest stages, they have established a concrete technical roadmap to realize this vision and are looking to connect with interested engineers and collaborators.
[31:21](https://www.youtube.com/watch?v=fFSPwJFXVlw&t=1881s) Call for Collaborators, Early Customers, and Closing Remarks
Will Wilson concludes his presentation by extending an open invitation to anyone in the audience looking to collaborate or try out their approach. Specifically, he explains that they are actively searching for two categories of people: those interested in joining the team to work alongside them, and potential early customers facing particularly daunting quality assurance challenges. Elaborating on what kind of partner would be an ideal fit, Wilson describes them as “people who have really scary QA problems that they feel like they don't have a handle on and they think some of these techniques could maybe help because we have a prototype that might kind of work.”
Beyond formal business relationships or recruiting, Wilson expresses a genuine desire to connect with fellow enthusiasts in the broader software engineering community. He points out that deterministic simulation and autonomous testing remain relatively obscure concepts across the software industry, remarking: “I'm also just interested in talking to anybody who's into this stuff because this is still a really small niche world and not a lot of people think about this stuff and so if you do I would love to meet you.” Realizing that his session has run past its scheduled limit, he announces that he will not take questions from the stage, thanks the attendees, and concludes the talk to applause.
- Wilson is seeking both talent to join the initiative and early prospective customers dealing with critical, unmanageable QA issues. - The team has developed a working prototype testing these autonomous, simulation-based techniques. - Wilson highlights how niche this corner of testing and systems engineering currently is, encouraging anyone passionate about the subject to reach out.
## Notable quotes > [10:44](https://www.youtube.com/watch?v=fFSPwJFXVlw&t=644s) If we were smart enough or had the right kind of brains to write all the tests we needed to write, then we would have just written the code correctly in the first place. — Will Wilson
> [16:13](https://www.youtube.com/watch?v=fFSPwJFXVlw&t=973s) Humans debug software; humans are not mystical Turing Oracles—if humans can do it, then so can computers. — Will Wilson
## Mentioned Will Wilson, FoundationDB, Moore's law, Gartner, Forrester, Jenkins, Haskell, Apple, Evan, Ben, Dave Scherer, Amazon EC2
[Source](https://www.youtube.com/watch?v=fFSPwJFXVlw)