Debugging with the Scientific Method - Stuart Halloway
ClojureTV
AI-assisted summary of Debugging with the Scientific Method - Stuart Halloway — original source. Check the original for full context.
Source published:
Key ideas
- Software debugging accounts for an estimated 50% or more of development costs, yet it is often approached through reactive tool usage rather than systematic hypothesis testing. (0:00)
- Unlike academic science, software debugging is purely deductive and insulated from sociopolitical biases because the system itself provides the existing grand theory. (13:59)
- Effective troubleshooting relies on proportional reduction to bisect the search space and strictly changing only one variable per experiment. (25:02)
- Failing to document hypotheses and observations forces engineers to hold complex state spaces in memory, turning debugging into an unnecessarily difficult meta-problem. (33:59)
- Clojure's immutable data structures and code-as-data model make it uniquely well-suited for automating scientific debugging via state-shrinking algorithms. (44:57)
Detailed notes
## Background Published on YouTube by ClojureTV on November 17, 2015, this video captures a presentation by Stuart Halloway, founder and President of Cognitect, Clojure committer, and Datomic developer. According to the description, Halloway—co-author of 'Programming Clojure' and a regular speaker across industry events—explores the methodology and mindset of effective troubleshooting.
## TL;DR In this talk, Stuart Halloway argues that debugging software is fundamentally more scientific than professional science itself, advocating for hypothesis-driven inquiry over reactive, tool-directed tinkering. He breaks down the troubleshooting cycle into crisp problem statements, proportional search space reduction, and isolated single-variable experiments that prune away non-essential tooling. Drawing on real-world examples from Clojure and Datomic, Halloway highlights critical JVM realities—such as memory pressure masquerading as bizarre unrelated exceptions—and insists that documenting hypotheses and observations in writing is vital to avoiding cognitive overload. Finally, he urges the community to leverage Clojure's immutable, code-as-data architecture to automate diagnostic search algorithms like delta debugging and state shrinking.
## Atmosphere & style The presentation is witty, authoritative, and playfully cynical, delivered with the dry, self-deprecating humor of a battle-tested systems veteran. Halloway balances rigorous philosophy—drawing on Kuhn, Popper, and modus tollens—with pop-culture nods to 'House, M.D.', 'Ocean's Eleven', and plush Cthulhu toys, keeping the mood grounded and delightfully irreverent. Watching it feels like getting a candid masterclass from a pragmatic senior architect who demystifies the engineering ego, urging developers to replace chaotic trial-and-error with cool-headed discipline so they can finish work at five and enjoy a glass of wine.
## Bulletin - [00:00](https://www.youtube.com/watch?v=FihU5JxmnBg&t=0s) Software debugging accounts for an estimated 50% or more of development costs, yet it is often approached through reactive tool usage rather than systematic hypothesis testing. - [13:59](https://www.youtube.com/watch?v=FihU5JxmnBg&t=839s) Unlike academic science, software debugging is purely deductive and insulated from sociopolitical biases because the system itself provides the existing grand theory. - [25:02](https://www.youtube.com/watch?v=FihU5JxmnBg&t=1502s) Effective troubleshooting relies on proportional reduction to bisect the search space and strictly changing only one variable per experiment. - [33:59](https://www.youtube.com/watch?v=FihU5JxmnBg&t=2039s) Failing to document hypotheses and observations forces engineers to hold complex state spaces in memory, turning debugging into an unnecessarily difficult meta-problem. - [44:57](https://www.youtube.com/watch?v=FihU5JxmnBg&t=2697s) Clojure's immutable data structures and code-as-data model make it uniquely well-suited for automating scientific debugging via state-shrinking algorithms.
## Key takeaways - [00:00](https://www.youtube.com/watch?v=FihU5JxmnBg&t=0s) Debugging drives over half of software development costs, making disciplined approaches far more valuable than ad-hoc tinkering. - [05:59](https://www.youtube.com/watch?v=FihU5JxmnBg&t=359s) Tool-directed development is an anti-pattern; developers must have a hypothesis and a plan before reaching for diagnostic tooling. - [11:47](https://www.youtube.com/watch?v=FihU5JxmnBg&t=707s) In debugging, falsification through modus tollens is an ideal outcome because eliminating a candidate hypothesis definitively narrows the search space. - [13:59](https://www.youtube.com/watch?v=FihU5JxmnBg&t=839s) Software debugging is more strictly scientific than academic science because it operates apolitically on concrete deductive behavior rather than social consensus. - [17:22](https://www.youtube.com/watch?v=FihU5JxmnBg&t=1042s) An actual cause is the difference between the observed reality and the closest counterfactual world where the failure does not occur; a fix is merely the concluding experiment proving it. - [23:00](https://www.youtube.com/watch?v=FihU5JxmnBg&t=1380s) Problem statements must explicitly define the steps taken, what was expected, and what actually occurred, completely eliminating vague pronouns like 'it didn't work'. - [25:02](https://www.youtube.com/watch?v=FihU5JxmnBg&t=1502s) Hypotheses should target proportional reduction of the candidate failure space rather than jumping immediately to highly specific leaf-node guesses. - [28:56](https://www.youtube.com/watch?v=FihU5JxmnBg&t=1736s) Reproduction cases should aggressively strip out non-essential tools and dependencies like test frameworks, editor plugins, and build tooling to isolate the core failure. - [31:05](https://www.youtube.com/watch?v=FihU5JxmnBg&t=1865s) Unexplained system outputs and warnings cannot be dismissed as unrelated without first understanding them, as they frequently point to classpath collisions or underlying defects. - [33:59](https://www.youtube.com/watch?v=FihU5JxmnBg&t=2039s) Not writing down hypotheses, justifications, and observations is an act of hubris that forces the mind to solve a difficult meta-tracking problem instead of the bug itself. - [35:51](https://www.youtube.com/watch?v=FihU5JxmnBg&t=2151s) The observed failure—such as an exception—is merely where the system noticed an invalid state, not necessarily the location or nature of the defect. - [38:13](https://www.youtube.com/watch?v=FihU5JxmnBg&t=2293s) Memory pressure and garbage collection on the JVM act as an 'anti-Lupus', triggering erratic thread scheduling, surfacing rare race conditions, and masquerading as unrelated errors. - [40:07](https://www.youtube.com/watch?v=FihU5JxmnBg&t=2407s) Debugging documentation should function like a short, dense specification rather than an expansive tutorial so that it can be completely read when addressing unknown unknowns. - [41:34](https://www.youtube.com/watch?v=FihU5JxmnBg&t=2494s) Diagnostic claims like 'it fails in production but works in dev' must be treated with skepticism and verified via minimal scripts, as engineering miscommunications often lead teams astray. - [44:57](https://www.youtube.com/watch?v=FihU5JxmnBg&t=2697s) Debugging is fundamentally a search problem over program states that can be automated through difference-shrinking algorithms like those found in property-based testing. - [47:55](https://www.youtube.com/watch?v=FihU5JxmnBg&t=2875s) Explaining a proposed test to an inanimate object before touching code forces deliberate reasoning and curbs reactive trial-and-error impulses.
## Detailed walkthrough ### [00:00](https://www.youtube.com/watch?v=FihU5JxmnBg&t=0s) Clarifications and Rationale for Debugging
Stuart Halloway begins by clarifying a couple of points from the introduction. He jokes that Alex Miller just drew a chain of causality suggesting Halloway is responsible for Strange Loop, so he is excited and eagerly waiting for his checks to arrive in the mail. He also playfully dismisses the implication that working at Cognitect offers endless chances to watch debugging in the product team, joking that Datomic has “really no bugs at all.” Turning serious, Halloway explains that in the Clojure community, driven by values instilled by Rich Hickey, every major talk should begin with a strong rationale.
His rationale for focusing on debugging has several parts:
1. **The Cost of Bugs:** Tracking down bugs is estimated to account for 50% or more of the total cost of software development. While developers in certain environments might manage to hand things off and run away, anyone who owns a system over time quickly realizes how real this cost is. 2. **Appearing Disproportionately Smart:** Debugging is a remarkably straightforward activity, yet people perceive it as exceedingly difficult. Halloway confesses he is constantly intimidated by the intelligence and specialized accomplishments of people in the audience. Being able to look bright without burning extra mental energy is a great trick: “this is not hard to do but people think it's hard to do and so you can look disproportionately smart by doing it.” 3. **Connecting Debugging to Clojure's Design Philosophy:** Understanding how Halloway and, by proxy, Rich Hickey think about debugging offers insight into the aesthetic, evolution, and processes behind Clojure itself. While people hold varied opinions on the onboarding experience, documentation, or exception handling, adopting a scientific approach inherently shapes the tools and techniques you choose. Halloway notes that exploring this deeper connection is a great topic for the evening's "office hours," which he jokes will take place at whichever local bars stay open the latest.
[03:37](https://www.youtube.com/watch?v=FihU5JxmnBg&t=217s) Why Stuart Halloway?
Halloway next addresses why he is qualified to speak on debugging, citing his deep operational history and practical motivations:
- **Clojure Committer and Screener:** Clojure does not follow the standard GitHub project workflow. Halloway reviews tickets, reads code, and has pushed every commit to Clojure since the 1.0 release. He has evaluated countless patches that entered the language—and far more that were rejected—learning to distinguish real bugs from perceived ones. - **Datomic Support Backstop:** In his role supporting Datomic, Halloway often acts as the place of last resort for the thorniest issues. Though support shifts rotate, he regularly serves as the backstop when hard problems arise, giving him extensive experience observing defects in Datomic itself as well as diagnosing tricky bugs within customer architectures. - **Gray Hair as an Appeal to Authority:** He playfully cites his graying hair as proof of experience, noting it was entirely dark when he began his career: “I earned this gray hair; I've been doing this for a long time and my hair was not gray at all when I first started debugging.” (He admits an alternate theory exists: it might just come from raising children.) - **Constructive Laziness:** Emphasizing programmer laziness, he explains that he hates working needlessly hard. His goal is to follow a systematic approach that allows him to close his laptop at 5:00 PM, enjoy dinner, have a glass of wine, watch whatever violent show HBO is airing, and go to bed.
[05:59](https://www.youtube.com/watch?v=FihU5JxmnBg&t=359s) Debugging in Clojure and the Trap of Tool-First Thinking
Turning to the Clojure ecosystem specifically, Halloway mentions a Reddit thread shared by his colleague Carin Meier. The post captured the widespread frustration of newcomers struggling with Clojure error messages. Beginners often complain about cryptic stack traces, confusion when dealing with dependencies or Leiningen plugins, or type-related errors that they feel would have been trivial to diagnose in Java. While Halloway's own experience with Clojure debugging has been positive rather than painful, he recognizes that many developers find it fraught and hopes to share principles to make the process smoother.
```mermaid flowchart TD accTitle: Tool-Directed vs. Scientific Debugging accDescr: Comparison of reactive tool-led tinkering versus hypothesis-driven debugging.
subgraph Reactive["Reactive Tinkering (Anti-Pattern)"] R1["Hit Bug / Cryptic Error"] --> R2["Reach for Tools Blindly"] R2 --> R3["Mutate Code Without a Plan"] R3 --> R4["Loss of Initial State & Context"] end
subgraph Scientific["Scientific Approach"] S1["Observe Issue"] --> S2["Formulate Hypothesis"] S2 --> S3["Select Tool to Test Hypothesis"] S3 --> S4["Know Direction Before Acting"] end ```
After asking how many people in the room had built tooling for Clojure—and acknowledging the impressive ecosystem of editors, debuggers, and visualizers—Halloway clarifies that his talk is deliberately not about tools:
- **Tools Overwhelm the Discourse:** A standard Google search for debugging reveals that well over half of all available resources are exclusively about tooling. Providing another tool tutorial would not deliver keynote-level value. - **Tools Do Not Provide Direction:** Developers must control their tools, not be driven by them. As Halloway quips, “there is no TDD out there that stands for Tool directed development; we wield the tools, they don't wield us.” - **The Need for a Plan:** Quoting Yogi Berra, Halloway reminds the audience: “if you don't know where you're going you might wind up someplace else.” When assisting customers in support, the first impulse is often wanting to travel back in time to the moment right before they started blindly changing things. While being a proactive self-starter is valuable, taking action without a deliberate plan typically obscures the original problem and leads developers astray.
[09:46](https://www.youtube.com/watch?v=FihU5JxmnBg&t=586s) The Scientific Method, Rainbow Graphics, and Etymology
Stuart Halloway admits that prior to preparing this talk, he had not read up on what people had to say specifically about the scientific method since earning his science degree in college. Naturally, he turned to Google to investigate why one should care about it. What he discovered was that according to the internet, the scientific method is represented entirely by "rainbows." He jokes that all the Edward Tufte information design purists must be pulling their hair out wondering what the meaningless color palettes are supposed to convey. Consequently, Halloway decided to strip all the colors out and build his own graphic using OmniGraffle—admitting self-deprecatingly that this also happened to represent the absolute limit of his diagramming skills.
```mermaid flowchart TD accTitle: Scientific Method for Debugging accDescr: The cycle of debugging through failure, hypothesis, experiment, observation, refinement, and theory. F["Failure<br/>(Omission of expected action)"] --> H["Hypothesis<br/>(Starting explanation)"] H --> E["Experiment<br/>(Test or trial)"] E --> O["Observation<br/>(Active acquisition of info)"] O -->|"Falsified (Modus Tollens)"| FH["Hypothesis Dead<br/>(Kill possibility)"] O -->|"Supported"| R["Refinement<br/>(Subtract unwanted elements)"] R --> H R -->|"Validated by predictions"| T["Theory<br/>(True given current info)"] ```
Leaning into Clojure/conj tradition, Halloway notes that he will define every term using the etymology dictionary. He advises future conference proposal submitters that including unfamiliar words and referencing etymology in their abstracts is clearly something highly valued in the Clojure community. He proceeds to lay out the scientific method as applied to debugging:
- **Failure**: A lack of success, or more precisely, an omission of an expected action (expecting one outcome and observing another). - **Hypothesis**: An explanation made without complete evidence, functioning as a starting point for further investigation. - **Experiment**: A test or a trial designed to probe that hypothesis. - **Observation**: The active acquisition of information from a primary source.
[11:47](https://www.youtube.com/watch?v=FihU5JxmnBg&t=707s) Falsification, Refinement, and the Danger of Emotional Attachment
When an experiment concludes, one of two positive outcomes occurs. The first is falsification—a deductive process governed by *modus tollens*. If a hypothesis implies that observation $O$ should not happen, but running the experiment reveals $O$, the hypothesis is decisively dead. While this may be disappointing to someone seeking research grant money in conventional science, Halloway stresses that in debugging, eliminating a candidate hypothesis is fantastic news because it narrows the problem space.
The second outcome is that the experiment supports the hypothesis, but the answer remains insufficiently granular to pinpoint the root cause. This requires refinement: the subtractive process of removing impurities or unwanted elements to shrink the conceptual size of the problem and hone in on the actual issue. Over time, repeated refinement yields a theory—a hypothesis that has been validated by predictions. Crucially, a theory is never absolute truth; it is simply true given all available information.
Halloway cautions against the dangerous cognitive trap of emotional attachment to pet theories: > “I have 99 pieces of evidence in favor of this is where the bug is and I have one little piece of evidence that Flatout contradicts it and I don't understand that piece of evidence and so well 99's pretty good I'm just going to keep going right if you have that one little piece of evidence that says you're dead you're dead right you can't and and we get attached to our theories that's the other thing you get emotional about it you're like you know what I really like that theory and look it had 99 good pieces of evidence in favor of it you can't do that”
[13:59](https://www.youtube.com/watch?v=FihU5JxmnBg&t=839s) Why Debugging Is More Scientific Than Science
Researching the topic led Halloway into the broader philosophical debate surrounding the scientific method. Drawing on his graduate school studies of Thomas Kuhn's *The Structure of Scientific Revolutions*, he cynically summarizes the book's premise: science is only slightly better than politics or religion because you wait for people with obsolete ideas to die of old age instead of actively murdering them. Kuhn demonstrated that science is fundamentally a messy social process driven by human biases, academic politics, and shifting moral debates (such as the controversy during Halloway's academic years over *The Bell Curve* and attempts to misuse science to rank human intelligence).
Remarkably, software debugging is completely insulated from this sociopolitical baggage. Halloway argues that if the scientific method is the gold standard of inquiry, debugging actually resembles true science far better than professional science does: > “So unfortunately the scientific method may not be all that great for science but it absolutely rocks for debugging.”
- **Deductive vs. Inductive**: Debugging does not require formulating an inductive, sweeping grand theory like gravity or biological evolution by turning a crank. In software, the grand theory already exists—it is the running software system itself—and debugging is purely deductive proof regarding concrete behavior. - **Apolitical Nature**: Unlike academia or politics, debugging does not trigger moral outrage, bureaucratic power struggles, or postmodern academic debates claiming reality is subjective. Everyone involved simply wants the bug found and resolved.
[17:22](https://www.youtube.com/watch?v=FihU5JxmnBg&t=1042s) Causes, Actual Causes, and the Fix
Translating scientific terminology into concrete troubleshooting terms, Halloway substitutes the word "cause" for "hypothesis." Formally, a cause is an event preceding an effect without which that effect would not have occurred. However, plain causes are too broad to be actionable. For example, "the universe exists" is technically an antecedent cause of every bug, but knowing that does not help a developer write a patch.
To be effective, an engineer must isolate the *actual cause*: the difference between the actual world and the closest possible counterfactual world in which the failure does not occur. While philosophers dive deep into the formal semantics of "closeness," intuitively, if five environmental variables contribute to a failure and isolating one specific variable stops the problem while the other four do not, that single variable represents the actual cause.
| Term | Scientific Parallel | Practical Definition in Software | |:--- |:--- |:--- | | **Failure** | Anomaly | Omission of an expected action; receiving unexpected behavior | | **Cause** | Hypothesis | An event preceding an effect without which the effect does not happen | | **Actual Cause** | Refined Hypothesis | Difference between the actual world and the closest possible world where the bug does not occur | | **Fix** | Final Experiment | The concluding experiment validating that the actual cause has been neutralized |
This progression leads to Halloway's definition of a resolution: a fix is simply the final experiment in the debugging chain. The initial experiment is reproducing the failure, intermediate experiments test various hypotheses, and the fix is the terminal trial proving the defect is eliminated. Halloway emphasizes that if you have applied a code change but haven't run an experiment demonstrating that the issue is genuinely gone, you have not actually fixed it.
[19:07](https://www.youtube.com/watch?v=FihU5JxmnBg&t=1147s) A Stack Overflow Case Study and the Power of the Scientific Method
Stuart Halloway emphasizes that even if an engineer is strongly suspicious about the root cause of an issue, the bug is not actually identified until it is fixed. To demonstrate how to apply the scientific method in a deliberately slapdash way, he brings up a real-world, bite-sized question sourced from Stack Overflow: “why is this partial not working?” In the snippet, a developer defines `partial-join` as `(partial (clojure.string/join ",")...)`, calls `(partial-join ["foo" "bar"])`, and is met with an intimidating runtime error: `ClassCastException: java.lang.String cannot be cast to clojure.lang.IFn`. Because the error includes an obscure generated numeric identifier, developers often assume it is impossible to understand.
```mermaid flowchart TD accTitle: REPL Bottom-up Evaluation accDescr: Step-by-step evaluation of the problematic Clojure form at the REPL revealing the type error. subgraph S1["Inner Form Evaluation"] A["(clojure.string/join \",\")"] -->|evaluates to| B["\",\" (String)"] end subgraph S2["Outer Form Substitution"] B --> C["(partial \",\")"] C --> D["Attempt to use String as IFn"] D -->|throws| E["ClassCastException"] end ```
Halloway runs through the common software engineering alternatives to debugging methodically: teams could build hyper-specific error messages that anticipate every conceivable mistake, provide extensive documentation, turn to syntax and paren highlighting, use interactive debuggers, rely on static typing, validate with schemas, or simply stare intently at the snippet until the solution surfaces. While every tool has a place—except merely staring at code—he argues that the scientific approach trumps them all. As Halloway insists:
> “The scientific method is better sauce than these other things because it's more general and because it doesn't require anything to exist in the world except your brain before we go in.”
Testing the hypothesis slapdash, Halloway notes there are only three plausible points of failure: `join` did not behave as expected, `partial` did not behave as expected, or `def` did not behave as expected. Adopting a simple heuristic for small code problems—evaluating forms bottom-up from the inside out at the REPL—he inspects the innermost expression: `(clojure.string/join ",")`. Evaluating it returns the string `","`. Since the separator was passed without a collection, `join` simply yielded the separator string. Substituting that back into `(partial ",")` reveals the bug immediately: Clojure is being asked to partially apply a string as if it were a function. By evaluating just one inner expression, the mystery unravels completely.
[23:00](https://www.youtube.com/watch?v=FihU5JxmnBg&t=1380s) Rigor, Precise Problem Statements, and Eliminating Pronouns
Reflecting on the Stack Overflow example, Halloway explains that “weak science is stronger than strong tools.” The exercise involved a vague problem statement, poor hypotheses, experiments lacking explicit goals, and minimal domain knowledge. Yet it succeeded easily because the problem was trivial. To solve genuinely difficult real-world bugs, developers must execute every step of the scientific method with far greater rigor: formulating crisp problem statements, choosing efficient hypotheses, conducting reproducible experiments, gathering actionable observations, and faithfully writing things down.
The most critical upgrade begins with how an issue is reported. The absolute antithesis of an effective problem statement is the ubiquitous phrase: “it didn't work.” That phrase conceals all relevant technical reality behind vague pronouns. A rigorous problem statement requires three explicit components: 1. The exact steps taken. 2. What was expected to happen. 3. What actually happened.
Halloway jokes that this single slide alone is easily worth a hundred dollars, urging attendees to send him checks. Simply forcing oneself to verbalize these three details often sparks an instant realization of the root cause—much like realizing one stepped out of a rolling car because they forgot to put it in neutral. Clear communication dispels the fog long before reaching for heavy machinery.
[25:02](https://www.youtube.com/watch?v=FihU5JxmnBg&t=1502s) Efficient Hypotheses and Proportional Reduction
After briefly troubleshooting a frozen Keynote slide deck during his presentation, Halloway moves to the formulation of hypotheses. He brings up the Malloy twins (played by Casey Affleck and Scott Caan) from the movie *Ocean’s Eleven* to illustrate poor deductive reasoning. In the film, while playing Twenty Questions, Affleck's character asks: "Am I alive? Yes. Am I a person? Yes. Evel Knievel! Damn it!" It mirrors how a five-year-old plays the game by immediately guessing a hyper-specific leaf node like "a tall oak tree" right after establishing the broad category.
```mermaid flowchart LR accTitle: Hypothesis Search Space Reduction accDescr: Progressively narrowing down the bug search space through proportional reduction. A["Total Possibility Space (100%)"] -->|Step 1: Carve space| B["Candidate Space (~50%)"] B -->|Step 2: Proportional cut| C["Candidate Space (~25%)"] C -->|Step 3: Rapid convergence| D["Root Cause Identified"] ```
A productive hypothesis, by contrast, seeks to bisect the search space. While often labeled "divide and conquer," computer science theoreticians frequently call this technique "decrease and conquer" because only one branch is pursued. Halloway dislikes that label because linear reductions (decreasing by one element each round) are painfully slow. Instead, the essential property is proportional reduction:
> “And the important characteristic here is proportional reduction: I want to take the space of where the problem is and I want to reduce it by ideally half, but as you know if you've done any algorithm analysis, right, if I can get rid of 10% of the possibilities on every step I'm going to have the answer really quickly.”
Formulating questions that cut the problem space in half demands domain knowledge. Beginners lacking domain experience must proceed with extreme caution on their initial cuts so they do not inadvertently eliminate the true cause from the universe of possibilities.
At the same time, engineers already possess natural intuitions about where bugs realistically live in an application stack: - Application-level business logic is overwhelmingly the most common culprit. - Underlying frameworks and language libraries account for a smaller fraction of genuine defects. - In all the years of triaging Clojure bugs, Clojure maintainers have had to pass very few issues down to the JVM or operating system, and they have never once had to escalate a ticket down to physics. - Because proportional reduction compounds so rapidly, an engineer does not have to be an infallible expert; eliminating even modest slices of the possibility space on each step converges on the bug remarkably fast.
[028:56] Designing Good Experiments and Pruning the Repro Case
An experiment must be small, reproducible, and strictly hypothesis-driven. Stuart Halloway cautions that simply trying things at random is not experimenting: real experiments require an idea of what is happening, accompanied by a test specifically designed to provide information that refines that idea. Crucially, developers must change only one variable at a time. If two things are changed at once, any resulting difference forces the investigator to step backward to untangle which change produced the impact, meaning no real information was gained.
```mermaid flowchart TD accTitle: Scientific Experiment Loop accDescr: Cycle of hypothesis formation, minimal isolated testing, observation, and refinement. subgraph HYP ["Hypothesis Stage"] H["Formulate Hypothesis"] P["Prune Non-Essential Tools<br/>(Cursive, Leiningen, Midje, etc.)"] end subgraph EXP ["Experimentation"] E["Change Exactly ONE Thing"] R["Reproduce Minimally"] end subgraph OUT ["Evaluation"] O["Observe All Outputs & Correlate"] F["Refine Mental Model"] end H --> P P --> E E --> R R --> O O --> F F -->|Next iteration| H ```
To illustrate what belongs in an experimental setup, Halloway poses a trick quiz to the audience: when creating a reproduction case for an application bug to hand off to a teammate, which of the following tools should not be included? The list includes `clojure.test`, Cursive, Prismatic Schema, Midje, Pipethrough, nREPL, Leiningen, Leiningen plugins, `core.typed`, and `test.generative`. The answer is that none of them belong in the repro case. Unless the hypothesis explicitly states that a component fails only when running under Cursive inside `clojure.test`, every single one of those dependencies should be stripped out. Removing unrelated tooling not only simplifies the problem space, but it also spares tool maintainers like Colin Fleming from receiving irrelevant bug reports.
- “Experiments are small and when you're making changes to a system you change one thing at a time right because if you change two things now you just have to go back to figure out what the impact of that was.” — Stuart Halloway - “It's incredibly important to remove things that do not contribute to your hypothesis statement and it's a freebie right I was saying earlier that you have to have this like mental model of the universe to allow you to narrow down the things that aren't it right this is a freebie right in bug these things are not it so start by taking them out make a really tiny thing that shows the problem.” — Stuart Halloway
---
[031:05] Observations, Tools, and the "Unrelated" Logback Warning
Making sound observations requires understanding every single output emitted by a system. If a system produces several familiar outputs alongside an unfamiliar one that appears unrelated, the investigator must screech to a halt: without understanding it, there is no way to know whether it connects to the failure. Halloway emphasizes that failures correlate strongly with recent edits: the bug is almost always in the last five lines of code written. Tools such as debuggers, structured logging, and metrics are vital because they turn opaque black boxes into white boxes, producing more observable outputs. While Google searches on debugging focus more than 50% of their results on these observational tools alone, tools represent only one small piece of the broader scientific discipline.
To demonstrate the risk of ignoring strange outputs, Halloway shares an incident from his own work the previous week. While working on a Clojure application, he encountered a cryptic failure in a subsystem, accompanied by a seemingly unrelated Logback configuration warning in the logs. Although tempted to dismiss it as a Java logging quirk, he decided to stop and clean up the Logback issue first. That investigation revealed that a build error had copied two different versions of the same library (`foo.bar` 2.1 and `foo.bar` 2.2) into the `lib` directory. Because the JVM had both versions on the classpath, it loaded an overlapping, non-identical set of classes from both JARs, creating subtle incompatibilities that triggered the cryptic subsystem errors. Addressing the seemingly unrelated output immediately solved the real failure.
- “If your system adds outputs right if your system has four or five outputs that you understand and then it has one that's unrelated to your current problem and you don't understand it breaks Screech right if you don't understand it how do you know if it's related to your problem or not so you need to understand the output from your system and you need to be suspicious of correlations.” — Stuart Halloway
---
[033:59] Write Things Down: The Mastermind Analogy and the Deadly Sin of Hubris
Halloway identifies the single most crucial recommendation of the entire presentation: write things down. Programmers should explicitly document the problem statement, every hypothesis, what the proposed experiment will prove, the justification for running that experiment before touching the system, and all resulting observations.
To demonstrate why, Halloway references the board game Mastermind. In Mastermind, colored pegs at the bottom represent a hidden code, pegs across the rows track past guesses, and small red and white pegs provide feedback on correctness. Debugging without writing things down is equivalent to playing Mastermind blindfolded or covering all previous guesses and trying to hold the entire history and state space in memory. Failing to document the process is not sloth; it is pure hubris. Trying to keep the entire state space of experiments in one's head forces the engineer to solve a meta-problem harder than the debugging task itself just to show off.
- “Write things down this is the single most important piece of advice I'm going to give in this talk write things down.” — Stuart Halloway - “It's amazingly arrogant not to write things down because basically what you're saying is I'm such a badass at solving problems that I'm going to solve the meta problem of keeping track of everything in my head just to show off when that's actually probably a harder problem than the debugging that you're doing and people do this all the time and it's staggering.” — Stuart Halloway
---
[035:51] Software Advice: Choosing Abstractions and Separating Failures from Defects
Turning to advice specific to software development, Halloway offers two core principles:
1. **Avoid C when possible:** Reviewing historical debugging literature reveals countless pitfalls tied to low-level memory management and pointers. Unless a specific domain strictly prevents using high-level environments like Java, Clojure, or Python, developers should not default to C in 2015, as it operates at the wrong level of abstraction. 2. **The failure is not the defect:** Programmers frequently confuse the manifest failure—such as an exception—with the underlying defect. An exception marks where the system noticed an invalid condition, not necessarily where the bug originated.
Halloway illustrates this with an incident encountered on a Datomic deployment. The system suffered from a large query, high CPU utilization, and an `IllegalStateException` thrown out of HornetQ. The naive conclusion would be to assume HornetQ was broken. However, HornetQ had been part of Datomic since version 0.0, and in Halloway's extensive experience with its development team, an issue traced below Datomic into HornetQ had only happened once—and even then, the root cause lay deeper down inside Netty's SSL handling. The Datomic team knew HornetQ was rock-solid, making an internal HornetQ defect an extremely poor hypothesis to pursue.
- “Boy you shouldn't reach for that as the default tool in 2015 in most domains.” — Stuart Halloway - “The failure is not the defect right the failure is not the defect and the way this comes up the most often in software is assuming that the exception has anything to do with the actual problem I mean it does have something to do with it that's how you know right but that it's in some way directly translatable into the problem.” — Stuart Halloway
[38:13](https://www.youtube.com/watch?v=FihU5JxmnBg&t=2293s) The Anti-Lupus of the JVM: Garbage Collection
Stuart Halloway introduces a key philosopher for debugging: Gregory House from the television series *House, M.D.* On that show, a running catchphrase is that “it’s never lupus.” In Clojure and Java programming on the JVM, however, developers face the exact opposite phenomenon: an anti-Lupus. As Halloway argues, “the actual bug is always garbage collection just as just as in House it's never lupus.” It is the culprit that it constantly turns out to be, even though it never looks like garbage collection at the outset.
```mermaid flowchart TD accTitle: How Out Of Memory Cascades Into Obscure Bugs accDescr: Diagram showing how low memory conditions trigger erratic GC scheduling, expose rare race conditions, and surface as misleading exceptions. subgraph GC["Memory Pressure"] A["Low Memory Condition"] --> B["GC Thread Runs Constantly"] end subgraph Effects["System Disruption"] B --> C["Radical Thread Scheduling Shift"] C --> D["Race Conditions Exposed in 2,000 Seconds"] A --> E["Allocation Failures"] E --> F["Cascading Secondary Failures"] end subgraph Output["Symptom"] D --> G["Bizarre Unrelated Exceptions"] F --> G end ```
Halloway outlines several concrete reasons why memory problems are uniquely insidious:
- **Untested code paths**: Most applications are never deliberately designed or tested to induce `OutOfMemoryError` conditions, leaving the resulting recovery paths as uncharted territory. - **Ubiquity**: An out-of-memory error can strike anywhere; there is no designated critical section of code where you can guarantee memory will never run out. - **Chameleon exceptions**: When an allocation fails deep in the stack, cascading secondary failures report radically different, misleading exceptions to the caller. - **Distorted thread scheduling**: Approaching the memory ceiling causes the garbage collection thread to run continuously, radically disrupting standard thread scheduling and exposing rare concurrency race conditions: “all those race conditions that would normally take your system 2,000 years to expose by accident all of a sudden it takes them 2,000 seconds to expose by accident.” - **Cascading breakdowns**: Out-of-memory issues snowball across components, requiring developers to maintain constant suspicion of memory health in any garbage-collected runtime.
---
[40:07](https://www.youtube.com/watch?v=FihU5JxmnBg&t=2407s) Specs as Debugging Docs and Re-evaluating the Clojure Bug
Halloway turns to documentation strategy, offering the blunt advice to read the entire manual. Because a bug begins as an unknown unknown, developers do not know where to look in advance. Consequently, ideal documentation for debugging differs fundamentally from tutorial documentation. Halloway summarizes the distinction: “good docs for debuggers are specs.” While learning materials can span a thousand pages and repeat themselves to ease comprehension, debugging documentation needs to be short, dense, and non-repetitive—ideally around 50 pages—so a developer can realistically read all of it.
Returning to the earlier string-joining bug with `partial`, Halloway illustrates how reading documentation immediately resolves misunderstandings. Looking up the docstring for `clojure.string/join` reveals it has two arities. The single-argument arity simply takes a collection and returns a string concatenation of its elements, rather than taking a separator and returning a partially applied function. A developer using a single step of experimentation or a single step of reading the documentation would have caught the problem instantly.
---
[41:34](https://www.youtube.com/watch?v=FihU5JxmnBg&t=2494s) The Datomic Mystery: "Everybody Lies"
Halloway applies these principles to a real-world, complex Datomic bug that surfaced with high CPU utilization and an `IllegalStateException` on HornetQ. The investigation team was provided a crucial clue: the failure occurred on Cassandra in production, but never on H2 in local development. Halloway challenges the audience to scrutinize that claim scientifically, noting that “it happens on Cassandra in production but not on H2 in development” is an amorphous claim lacking a concrete subject.
To turn this vague notion into an actionable experiment, they eliminated application complexity by writing a tiny, 10-to-15-line standalone reproduction script (`test.Java`) using a Datomic peer that executed the offending query in a tight loop. When run against Cassandra with production-sized data, the failure reproduced immediately—and turned out to be an out-of-memory error because the query demanded more memory than the JVM possessed. Suspicious of why Cassandra would hog so much more memory than H2, they resisted theorizing ahead of the evidence and ran the exact same tiny script against H2 storage. The reproduction failed identically on H2.
This result demonstrates Dr. House’s second famous adage: “everybody lies.” Halloway clarifies that in software engineering, this is rarely malicious deception:
> “And by the way never attribute to malice what you can contribute to accidental miscommunication. This is not lying in the sense of covering up your drug use which is always what it is on House right this is this is lying as in saying a statement which turns out not to be true.”
He jokes about how hard it is to teach his own children that not every mistaken statement is a lie. Had the engineers sat around a conference table trading blame—Datomic pointing at Cassandra drivers and Cassandra devs pointing back—they would have wasted days debating a premise that was completely false. Writing a minimal reproducer mechanically eliminated application code, third-party libraries, and baseless assumptions.
---
[44:57](https://www.youtube.com/watch?v=FihU5JxmnBg&t=2697s) Debugging as a Search Operation: Automating with Shrinking
Halloway presents his final recommendation: recognizing that debugging is fundamentally an algorithmic search operation over possible states. Since software engineers specialize in writing search algorithms, they should expand their tooling beyond passive inspectors and breakpoints. Instead, developers should write programs that automate the scientific method by generating hypotheses and running experiments programmatically. A basic example already in widespread use is `git bisect`, which cuts the revision history in half repeatedly to locate the commit that introduced a failure.
Halloway issues a direct call to action to the Clojure community to study Andreas Zeller’s book *Why Programs Fail*, highlighting chapters 5 through 7 and chapters 11 through 14. These chapters outline algorithms that model the complete state space of failing program executions against passing executions and isolate defects by shrinking the differences. Halloway points out that the Clojure community is uniquely qualified for this challenge:
> “Do we know how to do shrinking in the Clojure world? We're pretty good at that right we have um test.check and test.generative there are algorithms for this.”
While Zeller's book demonstrates these concepts using Python, Halloway challenges attendees to implement them in Clojure for upcoming conferences like ClojureWest or the next Clojure/conj. Capturing and comparing state spaces across program runs is notoriously hard in mutable, object-oriented codebases, but Clojure's emphasis on immutable data structures and explicit, minimal state makes it an ideal platform to automate scientific debugging.
[47:21](https://www.youtube.com/watch?v=FihU5JxmnBg&t=2841s) Clojure, Code as Data, and the Core Debugging Checklist
Reflecting on automated debugging techniques, Stuart Halloway notes that if such automated approaches can be implemented for a Python program, they ought to be substantially easier to build in Clojure. Because Clojure treats code as data and passes everything around as data, developers in the Clojure ecosystem are uniquely well-positioned to implement and take advantage of these sophisticated diagnostic algorithms. As Halloway states: “And oh by the way, having a language that models code as data and passes everything around as data, we are uniquely in a good position to implement these kinds of algorithms.”
Looking back over the technical rules of thumb covered throughout the presentation, Halloway recaps the core software-specific tenets of debugging: - Work at a high level of abstraction. - Remember the fault is not the defect. - Remember that garbage collection (GC) did it. - Read the manual twice. - Don’t trust, because people lie. - Reproduce the problem. - Go out and automate things.
Recognizing that a six-bullet list might be too much for an audience to retain after an exhausting conference day, Halloway strips the discipline down to its most fundamental scientific essence.
---
[47:55](https://www.youtube.com/watch?v=FihU5JxmnBg&t=2875s) Scientific Discipline: Well-Founded Choices, Plushy Cthulhu, and Writing It Down
To make the methodology as simple and actionable as possible, Halloway distills debugging into three foundational habits: knowing where you are going, making well-founded choices, and writing down your reasoning before acting. Invoking Yogi Berra, he points out that if you know which direction you are heading, you stand a much better chance of arriving there. The biggest trap for novice and veteran engineers alike is reacting frantically under pressure. When a system breaks and deadlines loom, developers instinctively panic and declare that they will just try a random tweak. Halloway urges engineers to immediately pause and challenge that impulse: “Stop and ask yourself, why should I try that?”
```mermaid flowchart TD accTitle: The Scientific Debugging Decision Flow accDescr: A flow illustrating the shift from haphazard trial-and-error to hypothesis-driven debugging. subgraph Reactive["The Pressure Trap"] A["Bug Occurs / System Fails"] --> B["'I'm just going to try this'"] end subgraph Scientific["The Scientific Approach"] B -.->|"Pause and Challenge"| C["Formulate a Hypothesis"] C --> D["Explain to Plushy Cthulhu"] D --> E["Justify Action in Writing"] E --> F["Focused, Directed Cruise to Solution"] end ```
To break the trial-and-error cycle, Halloway suggests explaining the problem to an inanimate object, such as a plushy Cthulhu, rather than immediately pulling in another person. Joking about a glitch in his own talk, he suggests telling the toy: “Plushy Cthulhu, I have this idea about what’s going wrong and why my Keynote presentation didn’t do what it was supposed to do during the middle of the talk; maybe you and I can sit down afterwards and I’m going to talk you through—I’m going to give you a hypothesis.” The cognitive effort of formulating that hypothesis and recording it in writing transforms troubleshooting entirely. As Halloway summarizes: “These those two steps—making good choices and writing your steps down—are going to turn a haphazard random walk around into a directed, focused cruise to an easy solution and an early night relaxing at the bar with your friends.”
- **Know where you are going:** Heading in a deliberate direction dramatically increases the likelihood of reaching a solution. - **Explain the hypothesis:** Talking through an assumption—even to a desk toy like a plushy Cthulhu—forces logical rigor. - **Justify actions in writing:** Writing down the rationale before modifying code stops random trial-and-error and ensures focused, reproducible progress.
## Notable quotes > [13:59](https://www.youtube.com/watch?v=FihU5JxmnBg&t=839s) the scientific method may not be all that great for science but it absolutely rocks for debugging. — Stuart Halloway
> [05:59](https://www.youtube.com/watch?v=FihU5JxmnBg&t=359s) there is no TDD out there that stands for Tool directed development; we wield the tools, they don't wield us. — Stuart Halloway
> [40:07](https://www.youtube.com/watch?v=FihU5JxmnBg&t=2407s) good docs for debuggers are specs. — Stuart Halloway
## Mentioned Stuart Halloway, Alex Miller, Rich Hickey, Carin Meier, Colin Fleming, Andreas Zeller, Edward Tufte, Thomas Kuhn, Yogi Berra, Cognitect, Clojure, Datomic, Cassandra, H2, HornetQ, Netty, Leiningen, Cursive, Midje, Pipethrough, nREPL, Prismatic Schema, Logback, Keynote, OmniGraffle, The Structure of Scientific Revolutions, Why Programs Fail, House, M.D., Ocean's Eleven, Mastermind, Strange Loop, Clojure/conj, ClojureWest
## Video description About the speaker: Stuart Halloway (@stuarthalloway) is a founder and President of Cognitect (formerly Relevance). He is a Clojure committer, and a developer of the Datomic database. Stuart has spoken at a variety of industry events, including Strange Loop, Clojure/conj, EuroClojure, ClojureWest, SpeakerConf, QCon, GOTO, OSCON, RailsConf, RubyConf, JavaOne, and NFJS. Stuart has written a number of books and technical articles. Of these, he is most proud of Programming Clojure.
[Source](https://www.youtube.com/watch?v=FihU5JxmnBg)