Claude
As AI tools compress infrastructure development from years to days, verification replaces construction as software's primary bottleneck, forcing engineering teams to constrain agents with formal state machines rather than human code reviews.
AI-assisted summary of How Datadog built a universal machine tool for Claude Code — original source. Check the original for full context.
Source published:
Key ideas
- Within four months, 90% of Datadog's roughly 3,000 engineers adopted AI coding tools for production tasks, with Claude Code driving approximately two-thirds of that total volume. (7:44)
- Datadog demonstrated that a single engineer paired with Claude Code could construct Helix, a Kafka-compatible distributed streaming prototype, in just three days, whereas its 2024 predecessor Courier took human engineers a full year. (7:44)
- To solve the sprawl of fragile, unmaintainable verification scripts created during autonomous agent sessions, Datadog developed Temper—a universal machine tool framework that models systems as declarative state blueprints rather than procedural CRUD code. (15:00)
- Temper isolates arbitrary code execution into sandboxed WebAssembly (WASM) modules and enforces a four-tier verification cascade spanning automata algebra, model checking, schedule simulation, and property testing. (25:00)
Detailed notes
## Background Published on YouTube by the Claude channel on May 6, 2026, this presentation features Sesh Nalla, Vice President of Engineering at Datadog. The talk explains how Datadog addressed the problem of sprawling, unmaintainable one-off verification scripts after 90% of its engineering organization adopted AI coding tools, introducing a constrained framework called Temper to build reusable, compounding tools for Claude Code.
## TL;DR Tracing Datadog's journey from hand-crafted distributed systems like Courier to building the Kafka-compatible Helix engine in just days with Claude Code, Sesh Nalla highlights how agentic construction shifted engineering bottlenecks downstream into verification, human coordination, and tool sprawl. To transition from mere mechanization to true industrialization, Datadog created Temper, a constrained framework acting as a software "machine tool." Rather than generating arbitrary procedural code, Claude Code outputs declarative blueprints compiled into verifiable formal state transitions and sandboxed WASM side effects. Backed by a multi-tiered verification cascade, Temper establishes secure, reusable control planes that let autonomous agents safely power long-running "dark factories" and directed system evolution.
## Atmosphere & style The talk carries the grounded, battle-tested pragmatism of a veteran infrastructure leader who has spent years on midnight on-call rotations, elevated by vivid historical analogies to the Industrial Revolution. Sesh Nalla speaks with an easygoing, wry candor, making dry jokes about brutalist slide formatting and engineers experiencing the "seven stages of grief" as daily coding morphs into high-altitude agent management. Watching the talk feels like an intellectual recalibration, shifting the audience's mindset away from the dizzying chaos of throwaway agent scripts toward a structured, deliberate vision of software as an adaptable, cultivated organism.
## Bulletin - [07:44](https://www.youtube.com/watch?v=EdmuYPBt_EM&t=464s) Within four months, 90% of Datadog's roughly 3,000 engineers adopted AI coding tools for production tasks, with Claude Code driving approximately two-thirds of that total volume. - [07:44](https://www.youtube.com/watch?v=EdmuYPBt_EM&t=464s) Datadog demonstrated that a single engineer paired with Claude Code could construct Helix, a Kafka-compatible distributed streaming prototype, in just three days, whereas its 2024 predecessor Courier took human engineers a full year. - [15:00](https://www.youtube.com/watch?v=EdmuYPBt_EM&t=900s) To solve the sprawl of fragile, unmaintainable verification scripts created during autonomous agent sessions, Datadog developed Temper—a universal machine tool framework that models systems as declarative state blueprints rather than procedural CRUD code. - [25:00](https://www.youtube.com/watch?v=EdmuYPBt_EM&t=1500s) Temper isolates arbitrary code execution into sandboxed WebAssembly (WASM) modules and enforces a four-tier verification cascade spanning automata algebra, model checking, schedule simulation, and property testing.
## Key takeaways - [00:00](https://www.youtube.com/watch?v=EdmuYPBt_EM&t=0s) Software engineering is shifting from mechanization (delegating tasks to models) to industrialization, where execution becomes standardized, repeatable, and governed by specialized machine tools. - [00:00](https://www.youtube.com/watch?v=EdmuYPBt_EM&t=0s) The advent of frontier models like Claude Opus 4.5 in late 2025 marked an inflection point where engineers began trusting agents with ambiguous, system-scoped architectural tasks. - [02:22](https://www.youtube.com/watch?v=EdmuYPBt_EM&t=142s) Autonomous evolutionary code generation, demonstrated in Datadog's Bits Evolve project, is fundamentally constrained by the fidelity of the verification harness and telemetry it adapts within. - [07:44](https://www.youtube.com/watch?v=EdmuYPBt_EM&t=464s) Agentic coding shifts the primary engineering bottleneck downstream from raw construction velocity to validation, human coordination, and accumulating operational production mileage. - [10:00](https://www.youtube.com/watch?v=EdmuYPBt_EM&t=600s) Because existing software toolchains are designed around human cognitive limits, engineers face disorientation as their daily role shifts from writing code to setting constraints and managing agents. - [10:00](https://www.youtube.com/watch?v=EdmuYPBt_EM&t=600s) Long-running autonomous agent sessions spanning 20 to 28 hours produce fragmented, ad-hoc glue scripts that create false progress unless captured by durable, compounding frameworks. - [15:00](https://www.youtube.com/watch?v=EdmuYPBt_EM&t=900s) Temper abstracts systems into declarative blueprints defining explicit states, permissible transitions, and invariants, rather than letting LLMs emit raw, sprawling procedural application code. - [15:00](https://www.youtube.com/watch?v=EdmuYPBt_EM&t=900s) In a 'dark factory' paradigm, agents execute continuous background maintenance and tool synthesis without human intervention, requiring human engineers to act primarily as factory architects. - [15:00](https://www.youtube.com/watch?v=EdmuYPBt_EM&t=900s) Unlike standard CRUD applications that scatter control rules across routes and databases, Temper borrows from Erlang/OTP by elevating operational state machines into first-class, verifiable transition tables. - [20:00](https://www.youtube.com/watch?v=EdmuYPBt_EM&t=1200s) External compilers verify agent-drafted blueprints and hot-reload state transition tables directly into the runtime without triggering traditional CI deployment cycles. - [25:00](https://www.youtube.com/watch?v=EdmuYPBt_EM&t=1500s) Executing untrusted agent side effects inside WebAssembly (WASM) sandboxes prevents external logic from becoming an uncontrolled backdoor into system state. - [25:00](https://www.youtube.com/watch?v=EdmuYPBt_EM&t=1500s) Verification is the central bottleneck of agentic development, requiring a multi-layered Swiss cheese model ranging from automata algebra to randomized property testing. - [27:30](https://www.youtube.com/watch?v=EdmuYPBt_EM&t=1650s) Autonomous agents dramatically lower the labor cost of formal methods, allowing high-assurance engineering practices historically reserved for aviation and finance to be applied across general software development. - [27:30](https://www.youtube.com/watch?v=EdmuYPBt_EM&t=1650s) Composing systems from small, verifiable machine tools transforms monolithic architectures into living codebases capable of directed evolution based on real-time production traffic patterns.
## Detailed walkthrough ### [00:00](https://www.youtube.com/watch?v=EdmuYPBt_EM&t=0s) Machine Tools and the Shifting Curve of Ambition
Taking the stage at 4:00 PM, Sesh Nalla, Vice President of Engineering at Datadog, begins by checking the room's energy and asking for a show of hands from anyone familiar with classical manufacturing machine tools—the jigs, fixtures, gauges, and milling cutters displayed on his introductory slide. Joking that he was expecting zero hands raised anyway, Nalla explains that these physical artifacts from the Industrial Revolution were the breakthrough mechanisms that unlocked massive scale through interchangeability, standardization, and precision. In manufacturing, these specialized tools enabled repeatable production of components that fit together into complex machines such as aircraft, nuclear reactors, engines, and lunar landing modules. For Nalla, machine tools serve as the core metaphor for the engineering discipline Datadog is establishing around Claude Code.
Reflecting on his personal journey over the previous 18 months, Nalla describes an exponential shift in how software engineers scope their ambition when working with frontier models. Throughout much of 2025, model utility was confined to narrow boundaries: writing small local changes, producing glue code, and spinning up throwaway prototypes that served primarily as learning exercises. However, the trajectory bent sharply upward in late 2025 with the arrival of advanced reasoning models like Claude Opus 4.5. Nalla observed that this inflection point fundamentally altered his relationship with the technology: “And then around late 2025, I think you all have seen this and noticed it, when the slope started to change, like this exponential, I started trusting Claude with larger and more ambiguous system-scoped work.”
[02:22](https://www.youtube.com/watch?v=EdmuYPBt_EM&t=142s) Courier and Bits Evolve: From Manual Rigor to Code Cultivation
To contextualize Datadog's current tooling philosophy, Nalla traces the lineage of systems development across four sequential projects, beginning in 2024 with Courier—a distributed queuing system built completely by hand before the advent of agentic coding workflows. In classical systems engineering, building a robust distributed architecture requires far more than assembling parts; it demands rigorous formal modeling and simulation to ensure that component interactions remain observable, testable, and verifiable. The team applied heavy verification where mistakes were expensive or hard to reverse, establishing a baseline of engineering rigor designed to prevent catastrophic failures from ever reaching production.
```mermaid flowchart TD accTitle: The Lineage of Systems and Evolving Bottlenecks accDescr: Diagram illustrating the progression from Courier to Bits Evolve, Helix, and Industrialization at Datadog.
subgraph S1["Classical Engineering (2024)"] A["Courier (Distributed Queue)"] B["Bottleneck: Manual Construction<br/>(1 year to build by hand)"] A --> B end
subgraph S2["Autonomous Adaptation (2025)"] C["Bits Evolve (Optimization Harness)"] D["Bottleneck: Feedback Loop & Harness<br/>(Natural selection of variants)"] C --> D end
subgraph S3["Agentic Construction (Late 2025)"] E["Helix (Kafka-Compatible Streaming)"] F["Bottleneck: Human Coordination & Mileage<br/>(3 days to build, hard to ship)"] E --> F end
subgraph S4["Industrialization"] G["Machine Tools (Temper Framework)"] H["Outcome: Repeatable, Verifiable Scale"] G --> H end
B --> C D --> E F --> G ```
By September 2025, Datadog experimented with closed-loop evolutionary optimization in a project called Bits Evolve, inspired by DeepMind's AlphaEvolve. Instead of manual refactoring, Bits Evolve deployed a council of large and small models to generate variants of performance-critical code blocks and hot functions. These candidates were then subjected to an automated verification cascade combining performance benchmarks, unit and integration tests, and production observability data to decide which mutations survived. Nalla found this to be an eye-opening paradigm shift, realizing that software could be cultivated much like living organisms adapting through natural selection. However, the project revealed a fundamental dependency on verification infrastructure, as Nalla explains: “The insight was that, this kind of evolution is only as good as the environment it adapts within.”
- **Courier (2024):** A distributed queuing system built entirely by human engineers using formal modeling, simulation, and manual verification to protect high-risk architectural paths. - **Bits Evolve (September 2025):** An automated harness using a multi-model council to propose mutations to hot functions, where survival was governed strictly by test cascades and telemetry. - **Key Realization:** If benchmarks are flawed or telemetry is weak, model-driven optimization produces shallow or counterproductive code evolutions.
[05:14](https://www.youtube.com/watch?v=EdmuYPBt_EM&t=314s) Helix and the Limit of Unconstrained Autonomous Evolution
Armed with Claude Opus 4.5 and rising confidence, Nalla progressively tested Claude on larger, more demanding subsystems, absorbing many failed experiments along the way before achieving reliable breakthroughs. The team eventually asked an audacious question: could a single human engineer, paired with Claude Code, rebuild an infrastructure service as massive as Apache Kafka from scratch? Applying the rigorous verification cascade developed during Courier, Claude Code took over the vast majority of the mechanical construction. In a matter of days, the experiment yielded Helix—a fully functional, Kafka-compatible distributed streaming system.
While generating Helix was a stunning technical feat, getting it ready for production deployment hit an immediate obstacle: building production "mileage" and battle-tested operational confidence. Datadog initially attempted to solve this by applying the Bits Evolve framework to whole components of Helix. That attempt quickly stalled because the architectural surface area was simply too vast; even with a strict verification cascade, the autonomous loop generated edge cases requiring constant human interpretation, interactive steering, and multi-turn debugging. Stepping back, the team dialed their ambition to a narrower, more constrained problem: Datadog's live metrics aggregation servers. By narrowing the scope, they enabled Claude to optimize materialization logic dynamically on a per-customer basis, enforcing a "proof-carrying path" around every proposed change so human reviewers would not need to inspect every candidate variant manually.
- **Helix Development:** A complete Kafka-compatible streaming engine generated in just a few days by a single human working alongside Claude Code. - **The Mileage Problem:** Generating functional code was fast, but running it reliably under real-world production constraints exposed operational gaps. - **Scope Adjustment:** Moving from full-component autonomous evolution to targeted, per-customer metric materialization optimizations backed by formal proofs.
[07:44](https://www.youtube.com/watch?v=EdmuYPBt_EM&t=464s) Shifting Bottlenecks: Moving from Mechanization to Industrialization
Looking across Courier, Bits Evolve, and Helix, Nalla highlights a recurring historical pattern: solving one engineering bottleneck immediately pushes the constraint downstream. With Courier in 2024, the primary constraint was raw construction speed; careful human design and manual implementation took an entire year. By late 2025, Helix proved that agentic construction could compress a year's worth of engineering into three days. That leap moved the bottleneck directly into the feedback and evaluation loop, where the quality of the verification harness determined survival. Once agents could reliably generate entire systems, the bottleneck shifted once again to human coordination—specifically, the friction of reviewing, validating, and safely shipping agent-generated software using tools that had been designed strictly for human workflows.
Citing Dario Amodei's keynote remarks on Amdahl's law, Nalla frames this challenge as the necessary leap from mere mechanization to true industrialization. Mechanization simply means delegating labor to agents, whereas industrialization establishes repeatable, verifiable, and controllable processes. As Datadog experienced explosive internal adoption—reaching 90% of engineers using AI coding tools for production work within four months, with Claude Code accounting for two-thirds of that volume—the sprawling creation of unmaintainable, one-off verification scripts proved that human-centric coordination mechanisms could not keep pace. Nalla argues that modern agentic software development requires dedicated "machine tools" to govern this burgeoning complexity: “Mechanization means agents are doing more of the work now, and industrialization, if I were to borrow the metaphor, means work becomes repeatable, verifiable, controllable and scalable.”
- **Construction Speed Comparison:** Courier took 12 months for human engineers to construct; Helix achieved functional Kafka compatibility via Claude Code in approximately 3 days. - **Adoption at Datadog:** Over the span of four months, 90% of Datadog engineers integrated AI coding tools into production workflows. - **Tooling Share:** Claude Code drove roughly two-thirds (66%) of all agentic coding activity across Datadog's engineering organization. - **The Core Dilemma:** When construction becomes instantaneous, human coordination, validation, and production safety tooling become the overriding operational bottleneck.
[10:00](https://www.youtube.com/watch?v=EdmuYPBt_EM&t=600s) The Shift from Writing Code to Steering Agents and the Tool Sprawl Problem
At Datadog, AI coding tools have rapidly integrated into day-to-day engineering workflows, with roughly 3,000 engineers adopting them for production tasks and Claude Code powering at least two-thirds of that total volume. Much of the early exploration—such as building the Helix streaming prototype—relied on a single human engineer steering one or more interactive agent sessions. Work spans a two-dimensional map defined by generation complexity on one axis and verification ambiguity on the other. This dynamic has introduced highly personalized workflows across the software development lifecycle (SDLC) because individual engineers can suddenly output vastly more than was previously possible.
Historically, developer "flow" meant a tight, direct feedback loop between human intent and handwritten code: understanding a problem, writing the code, testing, reviewing, deploying, and operating it. Agents upend that traditional abstraction. Sesh Nalla quips that while he made peace long ago with not writing code because he transitioned into management, many software engineers are still navigating their own "seven stages of grief" as looking at daily code fades away. The engineer's craft is shifting away from typing lines of code toward defining constraints: framing the problem, provisioning the right tools, establishing success criteria, and creating robust failure-detection mechanisms. Sesh Nalla observes:
> “All of this is powerful, it's like everyone's promoted three levels up into management chain, which they weren't signed up for.”
Because this leap happens against gravity, it can feel disorienting and dizzying for engineers unaccustomed to operating at such high altitudes of abstraction.
```mermaid flowchart TD accTitle: The Changing Engineering Abstraction accDescr: Transition from manual engineering flow to agent orchestration. subgraph Traditional["Traditional Engineering Flow"] direction LR A["Problem Understanding"] --> B["Write Code"] B --> C["Test & Review"] C --> D["Ship & Operate"] end subgraph Agentic["Agent-Orchestrated Model"] direction LR E["Human: Define Outcomes & Constraints"] --> F["Cloud Code / Agents"] F --> G["Parallel Execution Sessions"] G --> H["Automated Verification"] end ```
Until recently, the human organization served as the factory itself. Development tooling was strictly optimized around human attention spans, personal judgment, and operational memory stored inside engineers' heads. With the advent of Claude Code and models like Claude Opus 4.5, a single engineer can orchestrate four, five, or even ten parallel sessions. Yet existing toolchains remain "human-shaped," while agents operate two orders of magnitude faster. As a consequence, the individual human has become an awkward manual bridge between rapid agent execution and sluggish, human-oriented systems.
When autonomous, cloud-managed background agents take over long-running, judgment-bearing tasks spanning hours or days (benchmarking runs of 20 to 28 hours), they inevitably improvise their own localized tooling, glue scripts, and conventions. This causes a confusing blur between transient intermediate artifacts and actual product code. Because these ad-hoc tools exist only within the context of a single session or markdown scratchpad, they do not compound across the organization. Sesh Nalla highlights the need to break out of this trap by introducing a durable, structural framework—a universal "machine tool."
- Datadog's AI adoption reached approximately 3,000 engineers using AI tools on production code, with Claude Code accounting for at least two-thirds of that footprint. - Operational memory increasingly risks getting marooned in individual developers' heads and isolated markdown files. - Agent sessions running 20 to 28 hours demonstrate that local tool invention produces high velocity, but much of it represents fragile, false progress that cannot be easily shared or maintained.
---
[15:00](https://www.youtube.com/watch?v=EdmuYPBt_EM&t=900s) Temper: The Machine Tool for the "Dark Factory"
To safely build and run mission-critical systems like massive databases, autonomous agents require the software equivalent of an industrial machine tool. Datadog developed Temper around this concept. Rather than letting agents improvise disconnected, throwaway scripts for every task, Temper requires them to output precise, declarative domain specifications. Sesh Nalla compares this to a jig or computer-aided CNC machine used to manufacture aircraft components: instead of hand-filing screw threads from scratch every time, an operator supplies rigorous specifications to a machine tool that guarantees deterministic, repeatable execution. With Temper, an agent drafts a precise description, iterates until the workflow functions, and leaves behind a standardized, reusable, and checkable tool that compounds into a coherent software factory.
```mermaid flowchart LR accTitle: Temper as a Repeatable Machine Tool accDescr: Agent intent translated via Temper into standardized factory outputs. A["Agent Intent & Problem Domain"] --> B["Temper (Constrained Framework / Machine Tool)"] B --> C["Repeatable Tooling"] B --> D["Control Planes & APIs"] B --> E["Verified Dark Factory Execution"] ```
This setup enables what Simon Willison has popularized as the "dark factory"—an automated environment where virtual machines and agents run continuously without humans on the factory floor, allowing teams to "turn off the lights." The engineer's responsibility pivots to architecting the factory itself: establishing constraints, framing desired outcomes, and wiring verification loops so autonomous processes can run stably for hours, days, or weeks.
Datadog applied this dark-factory architecture directly to Helix, an internal Kafka-like streaming service that handles some of the company's heaviest production workloads. Remarkably, a working prototype of Helix was spun up in roughly a week and shadowed against production traffic, revealing potential cost efficiencies 2x to 5x cheaper than existing solutions. However, graduating a system from a prototype to a dependable production engine requires accumulating real operational mileage. Datadog established an automated synthetic factory that exercises Helix using Temper across three distinct layers:
1. **Agent Control Plane:** Coordinates cloud-managed agent fleets, tracking their execution lifecycles, work queues, roles, and interactive sessions. 2. **Tool-Building Engine:** Enables agents to generate compact, standardized Temper applications that bridge SDLC workflows across Git, CI, and deployment pipelines. 3. **Helix Control API:** Acts as the lifecycle and control interface wrapped around the Helix data plane to systematically drive and verify synthetic workloads.
This implementation uncovered an unexpected realization: Temper is fundamentally broader than mere agent orchestration infrastructure. Sesh Nalla reflects:
> “The surprise was it started to feel more general than agent infrastructure. A lot of software, if you squint closely, is just control logic around database, APIs around state, policies around mutation, lifecycle transitions.”
When evaluating why this differs from simply asking Claude Code to generate a standard TypeScript or Python CRUD application, the difference lies in state architecture. While standard CRUD applications scatter control logic across routes, database constraints, background workers, service layers, and documentation, Temper brings the operational state machine front and center. Drawing inspiration from battle-tested platforms like Erlang/OTP and the actor model, Temper makes the underlying state machine completely explicit, verifiable, and safe for autonomous agents to drive.
- **Dark Factory Metaphor:** Borrowed from manufacturing via Simon Willison, describing software environments where agents operate reliably without active human supervision. - **Helix Performance and Savings:** Built within a week, Helix showed potential 2x to 5x cost improvements when shadowing Datadog's production Kafka workloads. - **Explicit vs. Implicit State:** While typical CRUD architectures bury operational lifecycle rules inside procedural code and separate database constraints, Temper defines lifecycle transitions and mutation policies as first-class, inspectable state machines.
[20:00](https://www.youtube.com/watch?v=EdmuYPBt_EM&t=1200s) Temper Internals: Blueprints, Formal State Transitions, and Hot Reloading
Sesh Nalla dives into the architecture of Temper by comparing it to durable execution engines like Temporal and battle-tested actor environments like the Erlang BEAM runtime. Rather than having an agent emit raw, messy application code, Temper asks the model to operate inside an operational domain and generate a structured blueprint through iterative plan mode. This declarative artifact answers fundamental questions: what states exist, which transitions are legal, who is permitted to trigger them, what invariants must hold, and how tool failures are handled. A specialized external compiler verifies this blueprint and hot-deploys it directly into the execution environment without needing traditional CI cycles. Sesh surveys the crowd to see who is familiar with the Erlang BEAM, noting that running high-assurance systems where an agent can hot-deploy code mirrors how hardened platforms have historically functioned.
```mermaid flowchart TD accTitle: Temper Build and Runtime Architecture accDescr: Shows how an agent blueprint is compiled into a transition table and executed through policy gates and side effects. subgraph Build["Build Path"] A["Agent (Claude Code)"] -->|"Iterative plan mode"| B["Blueprint (States, Actions, Triggers)"] B --> C["Compiler / Verifier"] end subgraph Runtime["Runtime Execution"] C -->|"Hot deploy"| D["Transition Table (Data-like Spec)"] D --> E{"Policy Gates"} E --> F["WASM Modules / Side Effects"] F -->|"Callbacks"| D end ```
The runtime interface behaves like a standard CRUD API, but underneath, the critical abstraction has been elevated away from procedural code. Temper transforms the blueprint into formal state transitions. Sesh jokes about his slide’s “completely brutalistic design” lacking syntax highlighting, but emphasizes the clarity of an agent-written spec for something like a helix rollout. In that system, an action like `start rolling` is only legal when the deployment status is `planned`. Once triggered, it moves to `rolling`, kicks off a non-idempotent side effect—such as patching a Kubernetes StatefulSet—and waits for an asynchronous callback to mark progress or fail. Because control logic is turned into a data-like transition table rather than being buried in spaghetti code or improvised service methods, agents can inspect it, reason about it, and make targeted modifications safely.
- The agent produces a structured blueprint rather than arbitrary application code, elevating the level of abstraction. - Compilation and formal verification take place entirely outside the LLM, similar to how Rust code is handed off to the Rust compiler. - Formal state transitions represent the core application as a precise transition system, making it far easier for both humans and agents to reason about. - Agents can modify individual states or rollback paths and hot reload them without risking regressions or skipping safety checks.
> “Formal here doesn’t mean every possible property is proved. It isn’t theoretical. It means the basic shape of the application is represented as a precise transition system.” — Sesh Nalla
> “It’s important to note that the agent is not generating arbitrary application code directly. We have raised the abstraction.” — Sesh Nalla
---
[25:00](https://www.youtube.com/watch?v=EdmuYPBt_EM&t=1500s) Policy Gates, WASM Sandboxing, and Multi-Level Verification
Because Temper's state transition tables are structured purely as data, the runtime evaluates state transitions and policy decisions side by side. Teams can configure strict role boundaries between agents: operator agents running in an autonomous "dark factory" setup might only be permitted a narrow set of execution tasks, while builder agents are explicitly forbidden from executing production changes. Operators can enforce invariants deterministically, answering questions such as whether a completed rollout can ever be re-triggered or whether a failed tool execution is allowed to continue down a rollout path. When side effects are required, Temper isolates them into small, typed operations. To run arbitrary execution logic without turning the state machine into an uncontrolled backdoor, that logic is sandboxed into WebAssembly (WASM) modules, severely restricting the surface area where LLM-generated code can misbehave and streamlining debugging.
```mermaid flowchart LR accTitle: Swiss Cheese Verification Model accDescr: Four levels of verification checks ranging from algebra to property tests. L1["Level 1: Automata Algebra"] --> L2["Level 2: Reachable State Graph"] L2 --> L3["Level 3: Fault Injection & Schedules"] L3 --> L4["Level 4: Randomized Property Tests"] ```
The final core pillar of Temper is the verifier, which Sesh identifies as the primary industry bottleneck for autonomous software development. Acting as a strict gatekeeper before transition tables can be loaded live into the runtime, the verifier operates on a multi-tiered "Swiss cheese" model where Claude evaluates how many tiers are necessary based on compiler feedback:
1. **Level 1 (Algebra Check):** Validates the mathematical and structural algebra of the state automata. 2. **Level 2 (Model Checking):** Exhaustively explores the reachable state graph to guarantee no path terminates in an invalid or dangerous state. 3. **Level 3 (Schedule Simulation & Fault Injection):** Injects faults, manipulates timing schedules, and ensures system invariants hold under adverse conditions. 4. **Level 4 (Property Testing):** Executes randomized action sequences to uncover subtle regressions.
- Verification gaps compound positively over time: every bug or edge case discovered in production or simulation is encoded directly into the model and test suite, echoing Boris Cherny's observations on compounding test suites. - Confining arbitrary generated code strictly to WASM modules prevents unchecked side effects from polluting the state machine. - Policy evaluation happens in tandem with state transition logic, ensuring fine-grained access control across different specialized agent roles.
> “Verification is basically the bottleneck right now, for pretty much everything.” — Sesh Nalla
> “FX are deliberately small typed operations here in Temper. Keeping them small prevents the state machine from becoming a backdoor for arbitrary application behavior.” — Sesh Nalla
---
[27:30](https://www.youtube.com/watch?v=EdmuYPBt_EM&t=1650s) Machine Tools for Software: From Dark Factories to Directed Evolution
Reflecting on years of operating mission-critical infrastructure at Datadog and getting woken up thousands of times in the middle of the night, Sesh stresses that humans cannot maintain mental models of thousands of lines of tangled, AI-generated code. Whether running financial systems or cloud infrastructure, applications must remain decomposable into concise, readable artifacts that humans can inspect and reason about. Returning to the historical analogy of the Industrial Revolution, Sesh observes that the industrial breakthrough came not from an artisan building one magnificent machine, but from inventing machine tools that produced standardized, interchangeable, and inspectable parts. Building scalable, production-ready databases and critical software with autonomous agents requires that exact same discipline.
Sesh concludes with an alternate vision of the future. While the concept of a fully automated "dark factory" represents automated efficiency, the term itself feels cold and sad. Instead, when software is constructed out of small, verifiable, and interchangeable components, the codebase begins to behave like a living organism that engineers can grow, cultivate, and guide through directed evolution. Rather than static architectures, systems can adapt dynamically based on real-world usage patterns—such as automatically mutating Kafka into an optimized FIFO queue or a lightweight buffer based on customer traffic. Sesh thanks the audience and wraps up the presentation.
- Mission-critical systems must remain small, explicit, and human-readable; throwing thousands of lines of untangled generated code at an LLM for review is an unreliable strategy. - High-assurance disciplines from aviation and financial engineering have used formal state transitions for decades, but the human labor cost was historically prohibitive for general software engineering. - AI agents drastically shift the economics of formal methods, making extreme rigor and composability cost-effective for general application development.
> “The win was not that one artisan could build a brilliant machine, the win was that a machine built with machine tools made parts composable and inspectable and replaceable that we could build larger machines.” — Sesh Nalla
> “However, the cost of such rigor with humans was too high for general software until now. Agents are changing their calculus.” — Sesh Nalla
> “So the whole software built this way can feel like an organism, that we can grow, cultivate, and evolve through feedback, selection, and adaptation.” — Sesh Nalla
## Notable quotes > [07:44](https://www.youtube.com/watch?v=EdmuYPBt_EM&t=464s) Mechanization means agents are doing more of the work now, and industrialization, if I were to borrow the metaphor, means work becomes repeatable, verifiable, controllable and scalable. — Sesh Nalla
> [15:00](https://www.youtube.com/watch?v=EdmuYPBt_EM&t=900s) A lot of software, if you squint closely, is just control logic around database, APIs around state, policies around mutation, lifecycle transitions. — Sesh Nalla
> [25:00](https://www.youtube.com/watch?v=EdmuYPBt_EM&t=1500s) Verification is basically the bottleneck right now, for pretty much everything. — Sesh Nalla
> [27:30](https://www.youtube.com/watch?v=EdmuYPBt_EM&t=1650s) The win was not that one artisan could build a brilliant machine, the win was that a machine built with machine tools made parts composable and inspectable and replaceable that we could build larger machines. — Sesh Nalla
## Mentioned Datadog, Claude Code, Sesh Nalla, Temper, Courier, Bits Evolve, Helix, Apache Kafka, Claude Opus 4.5, AlphaEvolve, DeepMind, Dario Amodei, Simon Willison, Boris Cherny, Temporal, Erlang/OTP, Erlang BEAM, WebAssembly (WASM), Kubernetes
## Video description At Datadog, 90% of engineers adopted AI coding tools for production work in the last four months, with Claude Code driving two-thirds of that usage. As sessions grew more ambitious, the reusable tools they produced — for verification, debugging, orchestration — sprawled into unmaintainable one-offs. Sesh Nalla, VP of Engineering, shares how Datadog built Temper: a constrained framework that produces secure, reusable tools that compound across sessions and teams instead.
[Source](https://www.youtube.com/watch?v=EdmuYPBt_EM)