Datadog Curbs Claude Code Sprawl With State-Machine Framework

How Datadog built a universal machine tool for Claude Code — Claude

Claude

As AI tools compress infrastructure development from years to days, verification replaces construction as software's primary bottleneck, forcing engineering teams to constrain agents with formal state machines rather than human code reviews.

AI-assisted summary of How Datadog built a universal machine tool for Claude Code — original source. Check the original for full context.

Source published:

Key ideas

  • Within four months, 90% of Datadog's roughly 3,000 engineers adopted AI coding tools for production tasks, with Claude Code driving approximately two-thirds of that total volume. (7:44)
  • Datadog demonstrated that a single engineer paired with Claude Code could construct Helix, a Kafka-compatible distributed streaming prototype, in just three days, whereas its 2024 predecessor Courier took human engineers a full year. (7:44)
  • To solve the sprawl of fragile, unmaintainable verification scripts created during autonomous agent sessions, Datadog developed Temper—a universal machine tool framework that models systems as declarative state blueprints rather than procedural CRUD code. (15:00)
  • Temper isolates arbitrary code execution into sandboxed WebAssembly (WASM) modules and enforces a four-tier verification cascade spanning automata algebra, model checking, schedule simulation, and property testing. (25:00)

Detailed notes

## Background Published on YouTube by the Claude channel on May 6, 2026, this presentation features Sesh Nalla, Vice President of Engineering at Datadog. The talk explains how Datadog addressed the problem of sprawling, unmaintainable one-off verification scripts after 90% of its engineering organization adopted AI coding tools, introducing a constrained framework called Temper to build reusable, compounding tools for Claude Code.

## TL;DR Tracing Datadog's journey from hand-crafted distributed systems like Courier to building the Kafka-compatible Helix engine in just days with Claude Code, Sesh Nalla highlights how agentic construction shifted engineering bottlenecks downstream into verification, human coordination, and tool sprawl. To transition from mere mechanization to true industrialization, Datadog created Temper, a constrained framework acting as a software "machine tool." Rather than generating arbitrary procedural code, Claude Code outputs declarative blueprints compiled into verifiable formal state transitions and sandboxed WASM side effects. Backed by a multi-tiered verification cascade, Temper establishes secure, reusable control planes that let autonomous agents safely power long-running "dark factories" and directed system evolution.

## Atmosphere & style The talk carries the grounded, battle-tested pragmatism of a veteran infrastructure leader who has spent years on midnight on-call rotations, elevated by vivid historical analogies to the Industrial Revolution. Sesh Nalla speaks with an easygoing, wry candor, making dry jokes about brutalist slide formatting and engineers experiencing the "seven stages of grief" as daily coding morphs into high-altitude agent management. Watching the talk feels like an intellectual recalibration, shifting the audience's mindset away from the dizzying chaos of throwaway agent scripts toward a structured, deliberate vision of software as an adaptable, cultivated organism.

## Bulletin - [07:44](https://www.youtube.com/watch?v=EdmuYPBt_EM&t=464s) Within four months, 90% of Datadog's roughly 3,000 engineers adopted AI coding tools for production tasks, with Claude Code driving approximately two-thirds of that total volume. - [07:44](https://www.youtube.com/watch?v=EdmuYPBt_EM&t=464s) Datadog demonstrated that a single engineer paired with Claude Code could construct Helix, a Kafka-compatible distributed streaming prototype, in just three days, whereas its 2024 predecessor Courier took human engineers a full year. - [15:00](https://www.youtube.com/watch?v=EdmuYPBt_EM&t=900s) To solve the sprawl of fragile, unmaintainable verification scripts created during autonomous agent sessions, Datadog developed Temper—a universal machine tool framework that models systems as declarative state blueprints rather than procedural CRUD code. - [25:00](https://www.youtube.com/watch?v=EdmuYPBt_EM&t=1500s) Temper isolates arbitrary code execution into sandboxed WebAssembly (WASM) modules and enforces a four-tier verification cascade spanning automata algebra, model checking, schedule simulation, and property testing.

## Key takeaways - [00:00](https://www.youtube.com/watch?v=EdmuYPBt_EM&t=0s) Software engineering is shifting from mechanization (delegating tasks to models) to industrialization, where execution becomes standardized, repeatable, and governed by specialized machine tools. - [00:00](https://www.youtube.com/watch?v=EdmuYPBt_EM&t=0s) The advent of frontier models like Claude Opus 4.5 in late 2025 marked an inflection point where engineers began trusting agents with ambiguous, system-scoped architectural tasks. - [02:22](https://www.youtube.com/watch?v=EdmuYPBt_EM&t=142s) Autonomous evolutionary code generation, demonstrated in Datadog's Bits Evolve project, is fundamentally constrained by the fidelity of the verification harness and telemetry it adapts within. - [07:44](https://www.youtube.com/watch?v=EdmuYPBt_EM&t=464s) Agentic coding shifts the primary engineering bottleneck downstream from raw construction velocity to validation, human coordination, and accumulating operational production mileage. - [10:00](https://www.youtube.com/watch?v=EdmuYPBt_EM&t=600s) Because existing software toolchains are designed around human cognitive limits, engineers face disorientation as their daily role shifts from writing code to setting constraints and managing agents. - [10:00](https://www.youtube.com/watch?v=EdmuYPBt_EM&t=600s) Long-running autonomous agent sessions spanning 20 to 28 hours produce fragmented, ad-hoc glue scripts that create false progress unless captured by durable, compounding frameworks. - [15:00](https://www.youtube.com/watch?v=EdmuYPBt_EM&t=900s) Temper abstracts systems into declarative blueprints defining explicit states, permissible transitions, and invariants, rather than letting LLMs emit raw, sprawling procedural application code. - [15:00](https://www.youtube.com/watch?v=EdmuYPBt_EM&t=900s) In a 'dark factory' paradigm, agents execute continuous background maintenance and tool synthesis without human intervention, requiring human engineers to act primarily as factory architects. - [15:00](https://www.youtube.com/watch?v=EdmuYPBt_EM&t=900s) Unlike standard CRUD applications that scatter control rules across routes and databases, Temper borrows from Erlang/OTP by elevating operational state machines into first-class, verifiable transition tables. - [20:00](https://www.youtube.com/watch?v=EdmuYPBt_EM&t=1200s) External compilers verify agent-drafted blueprints and hot-reload state transition tables directly into the runtime without triggering traditional CI deployment cycles. - [25:00](https://www.youtube.com/watch?v=EdmuYPBt_EM&t=1500s) Executing untrusted agent side effects inside WebAssembly (WASM) sandboxes prevents external logic from becoming an uncontrolled backdoor into system state. - [25:00](https://www.youtube.com/watch?v=EdmuYPBt_EM&t=1500s) Verification is the central bottleneck of agentic development, requiring a multi-layered Swiss cheese model ranging from automata algebra to randomized property testing. - [27:30](https://www.youtube.com/watch?v=EdmuYPBt_EM&t=1650s) Autonomous agents dramatically lower the labor cost of formal methods, allowing high-assurance engineering practices historically reserved for aviation and finance to be applied across general software development. - [27:30](https://www.youtube.com/watch?v=EdmuYPBt_EM&t=1650s) Composing systems from small, verifiable machine tools transforms monolithic architectures into living codebases capable of directed evolution based on real-time production traffic patterns.

## Detailed walkthrough ### [00:00](https://www.youtube.com/watch?v=EdmuYPBt_EM&t=0s) Machine Tools and the Shifting Curve of Ambition

Taking the stage at 4:00 PM, Sesh Nalla, Vice President of Engineering at Datadog, begins by checking the room's energy and asking for a show of hands from anyone familiar with classical manufacturing machine tools—the jigs, fixtures, gauges, and milling cutters displayed on his introductory slide. Joking that he was expecting zero hands raised anyway, Nalla explains that these physical artifacts from the Industrial Revolution were the breakthrough mechanisms that unlocked massive scale through interchangeability, standardization, and precision. In manufacturing, these specialized tools enabled repeatable production of components that fit together into complex machines such as aircraft, nuclear reactors, engines, and lunar landing modules. For Nalla, machine tools serve as the core metaphor for the engineering discipline Datadog is establishing around Claude Code.

Reflecting on his personal journey over the previous 18 months, Nalla describes an exponential shift in how software engineers scope their ambition when working with frontier models. Throughout much of 2025, model utility was confined to narrow boundaries: writing small local changes, producing glue code, and spinning up throwaway prototypes that served primarily as learning exercises. However, the trajectory bent sharply upward in late 2025 with the arrival of advanced reasoning models like Claude Opus 4.5. Nalla observed that this inflection point fundamentally altered his relationship with the technology: “And then around late 2025, I think you all have seen this and noticed it, when the slope started to change, like this exponential, I started trusting Claude with larger and more ambiguous system-scoped work.”

[02:22](https://www.youtube.com/watch?v=EdmuYPBt_EM&t=142s) Courier and Bits Evolve: From Manual Rigor to Code Cultivation

To contextualize Datadog's current tooling philosophy, Nalla traces the lineage of systems development across four sequential projects, beginning in 2024 with Courier—a distributed queuing system built completely by hand before the advent of agentic coding workflows. In classical systems engineering, building a robust distributed architecture requires far more than assembling parts; it demands rigorous formal modeling and simulation to ensure that component interactions remain observable, testable, and verifiable. The team applied heavy verification where mistakes were expensive or hard to reverse, establishing a baseline of engineering rigor designed to prevent catastrophic failures from ever reaching production.

```mermaid flowchart TD accTitle: The Lineage of Systems and Evolving Bottlenecks accDescr: Diagram illustrating the progression from Courier to Bits Evolve, Helix, and Industrialization at Datadog.

subgraph S1["Classical Engineering (2024)"] A["Courier (Distributed Queue)"] B["Bottleneck: Manual Construction<br/>(1 year to build by hand)"] A --> B end

subgraph S2["Autonomous Adaptation (2025)"] C["Bits Evolve (Optimization Harness)"] D["Bottleneck: Feedback Loop & Harness<br/>(Natural selection of variants)"] C --> D end

subgraph S3["Agentic Construction (Late 2025)"] E["Helix (Kafka-Compatible Streaming)"] F["Bottleneck: Human Coordination & Mileage<br/>(3 days to build, hard to ship)"] E --> F end

subgraph S4["Industrialization"] G["Machine Tools (Temper Framework)"] H["Outcome: Repeatable, Verifiable Scale"] G --> H end

B --> C D --> E F --> G ```

By September 2025, Datadog experimented with closed-loop evolutionary optimization in a project called Bits Evolve, inspired by DeepMind's AlphaEvolve. Instead of manual refactoring, Bits Evolve deployed a council of large and small models to generate variants of performance-critical code blocks and hot functions. These candidates were then subjected to an automated verification cascade combining performance benchmarks, unit and integration tests, and production observability data to decide which mutations survived. Nalla found this to be an eye-opening paradigm shift, realizing that software could be cultivated much like living organisms adapting through natural selection. However, the project revealed a fundamental dependency on verification infrastructure, as Nalla explains: “The insight was that, this kind of evolution is only as good as the environment it adapts within.”

- **Courier (2024):** A distributed queuing system built entirely by human engineers using formal modeling, simulation, and manual verification to protect high-risk architectural paths. - **Bits Evolve (September 2025):** An automated harness using a multi-model council to propose mutations to hot functions, where survival was governed strictly by test cascades and telemetry. - **Key Realization:** If benchmarks are flawed or telemetry is weak, model-driven optimization produces shallow or counterproductive code evolutions.

[05:14](https://www.youtube.com/watch?v=EdmuYPBt_EM&t=314s) Helix and the Limit of Unconstrained Autonomous Evolution

Armed with Claude Opus 4.5 and rising confidence, Nalla progressively tested Claude on larger, more demanding subsystems, absorbing many failed experiments along the way before achieving reliable breakthroughs. The team eventually asked an audacious question: could a single human engineer, paired with Claude Code, rebuild an infrastructure service as massive as Apache Kafka from scratch? Applying the rigorous verification cascade developed during Courier, Claude Code took over the vast majority of the mechanical construction. In a matter of days, the experiment yielded Helix—a fully functional, Kafka-compatible distributed streaming system.

While generating Helix was a stunning technical feat, getting it ready for production deployment hit an immediate obstacle: building production "mileage" and battle-tested operational confidence. Datadog initially attempted to solve this by applying the Bits Evolve framework to whole components of Helix. That attempt quickly stalled because the architectural surface area was simply too vast; even with a strict verification cascade, the autonomous loop generated edge cases requiring constant human interpretation, interactive steering, and multi-turn debugging. Stepping back, the team dialed their ambition to a narrower, more constrained problem: Datadog's live metrics aggregation servers. By narrowing the scope, they enabled Claude to optimize materialization logic dynamically on a per-customer basis, enforcing a "proof-carrying path" around every proposed change so human reviewers would not need to inspect every candidate variant manually.

- **Helix Development:** A complete Kafka-compatible streaming engine generated in just a few days by a single human working alongside Claude Code. - **The Mileage Problem:** Generating functional code was fast, but running it reliably under real-world production constraints exposed operational gaps. - **Scope Adjustment:** Moving from full-component autonomous evolution to targeted, per-customer metric materialization optimizations backed by formal proofs.

[07:44](https://www.youtube.com/watch?v=EdmuYPBt_EM&t=464s) Shifting Bottlenecks: Moving from Mechanization to Industrialization

Looking across Courier, Bits Evolve, and Helix, Nalla highlights a recurring historical pattern: solving one engineering bottleneck immediately pushes the constraint downstream. With Courier in 2024, the primary constraint was raw construction speed; careful human design and manual implementation took an entire year. By late 2025, Helix proved that agentic construction could compress a year's worth of engineering into three days. That leap moved the bottleneck directly into the feedback and evaluation loop, where the quality of the verification harness determined survival. Once agents could reliably generate entire systems, the bottleneck shifted once again to human coordination—specifically, the friction of reviewing, validating, and safely shipping agent-generated software using tools that had been designed strictly for human workflows.

Citing Dario Amodei's keynote remarks on Amdahl's law, Nalla frames this challenge as the necessary leap from mere mechanization to true industrialization. Mechanization simply means delegating labor to agents, whereas industrialization establishes repeatable, verifiable, and controllable processes. As Datadog experienced explosive internal adoption—reaching 90% of engineers using AI coding tools for production work within four months, with Claude Code accounting for two-thirds of that volume—the sprawling creation of unmaintainable, one-off verification scripts proved that human-centric coordination mechanisms could not keep pace. Nalla argues that modern agentic software development requires dedicated "machine tools" to govern this burgeoning complexity: “Mechanization means agents are doing more of the work now, and industrialization, if I were to borrow the metaphor, means work becomes repeatable, verifiable, controllable and scalable.”

- **Construction Speed Comparison:** Courier took 12 months for human engineers to construct; Helix achieved functional Kafka compatibility via Claude Code in approximately 3 days. - **Adoption at Datadog:** Over the span of four months, 90% of Datadog engineers integrated AI coding tools into production workflows. - **Tooling Share:** Claude Code drove roughly two-thirds (66%) of all agentic coding activity across Datadog's engineering organization. - **The Core Dilemma:** When construction becomes instantaneous, human coordination, validation, and production safety tooling become the overriding operational bottleneck.

[10:00](https://www.youtube.com/watch?v=EdmuYPBt_EM&t=600s) The Shift from Writing Code to Steering Agents and the Tool Sprawl Problem

At Datadog, AI coding tools have rapidly integrated into day-to-day engineering workflows, with roughly 3,000 engineers adopting them for production tasks and Claude Code powering at least two-thirds of that total volume. Much of the early exploration—such as building the Helix streaming prototype—relied on a single human engineer steering one or more interactive agent sessions. Work spans a two-dimensional map defined by generation complexity on one axis and verification ambiguity on the other. This dynamic has introduced highly personalized workflows across the software development lifecycle (SDLC) because individual engineers can suddenly output vastly more than was previously possible.

Historically, developer "flow" meant a tight, direct feedback loop between human intent and handwritten code: understanding a problem, writing the code, testing, reviewing, deploying, and operating it. Agents upend that traditional abstraction. Sesh Nalla quips that while he made peace long ago with not writing code because he transitioned into management, many software engineers are still navigating their own "seven stages of grief" as looking at daily code fades away. The engineer's craft is shifting away from typing lines of code toward defining constraints: framing the problem, provisioning the right tools, establishing success criteria, and creating robust failure-detection mechanisms. Sesh Nalla observes:

> “All of this is powerful, it's like everyone's promoted three levels up into management chain, which they weren't signed up for.”

Because this leap happens against gravity, it can feel disorienting and dizzying for engineers unaccustomed to operating at such high altitudes of abstraction.

```mermaid flowchart TD accTitle: The Changing Engineering Abstraction accDescr: Transition from manual engineering flow to agent orchestration. subgraph Traditional["Traditional Engineering Flow"] direction LR A["Problem Understanding"] --> B["Write Code"] B --> C["Test & Review"] C --> D["Ship & Operate"] end subgraph Agentic["Agent-Orchestrated Model"] direction LR E["Human: Define Outcomes & Constraints"] --> F["Cloud Code / Agents"] F --> G["Parallel Execution Sessions"] G --> H["Automated Verification"] end ```

Until recently, the human organization served as the factory itself. Development tooling was strictly optimized around human attention spans, personal judgment, and operational memory stored inside engineers' heads. With the advent of Claude Code and models like Claude Opus 4.5, a single engineer can orchestrate four, five, or even ten parallel sessions. Yet existing toolchains remain "human-shaped," while agents operate two orders of magnitude faster. As a consequence, the individual human has become an awkward manual bridge between rapid agent execution and sluggish, human-oriented systems.

When autonomous, cloud-managed background agents take over long-running, judgment-bearing tasks spanning hours or days (benchmarking runs of 20 to 28 hours), they inevitably improvise their own localized tooling, glue scripts, and conventions. This causes a confusing blur between transient intermediate artifacts and actual product code. Because these ad-hoc tools exist only within the context of a single session or markdown scratchpad, they do not compound across the organization. Sesh Nalla highlights the need to break out of this trap by introducing a durable, structural framework—a universal "machine tool."

- Datadog's AI adoption reached approximately 3,000 engineers using AI tools on production code, with Claude Code accounting for at least two-thirds of that footprint. - Operational memory increasingly risks getting marooned in individual developers' heads and isolated markdown files. - Agent sessions running 20 to 28 hours demonstrate that local tool invention produces high velocity, but much of it represents fragile, false progress that cannot be easily shared or maintained.

---

[15:00](https://www.youtube.com/watch?v=EdmuYPBt_EM&t=900s) Temper: The Machine Tool for the "Dark Factory"

To safely build and run mission-critical systems like massive databases, autonomous agents require the software equivalent of an industrial machine tool. Datadog developed Temper around this concept. Rather than letting agents improvise disconnected, throwaway scripts for every task, Temper requires them to output precise, declarative domain specifications. Sesh Nalla compares this to a jig or computer-aided CNC machine used to manufacture aircraft components: instead of hand-filing screw threads from scratch every time, an operator supplies rigorous specifications to a machine tool that guarantees deterministic, repeatable execution. With Temper, an agent drafts a precise description, iterates until the workflow functions, and leaves behind a standardized, reusable, and checkable tool that compounds into a coherent software factory.

```mermaid flowchart LR accTitle: Temper as a Repeatable Machine Tool accDescr: Agent intent translated via Temper into standardized factory outputs. A["Agent Intent & Problem Domain"] --> B["Temper (Constrained Framework / Machine Tool)"] B --> C["Repeatable Tooling"] B --> D["Control Planes & APIs"] B --> E["Verified Dark Factory Execution"] ```

This setup enables what Simon Willison has popularized as the "dark factory"—an automated environment where virtual machines and agents run continuously without humans on the factory floor, allowing teams to "turn off the lights." The engineer's responsibility pivots to architecting the factory itself: establishing constraints, framing desired outcomes, and wiring verification loops so autonomous processes can run stably for hours, days, or weeks.

Datadog applied this dark-factory architecture directly to Helix, an internal Kafka-like streaming service that handles some of the company's heaviest production workloads. Remarkably, a working prototype of Helix was spun up in roughly a week and shadowed against production traffic, revealing potential cost efficiencies 2x to 5x cheaper than existing solutions. However, graduating a system from a prototype to a dependable production engine requires accumulating real operational mileage. Datadog established an automated synthetic factory that exercises Helix using Temper across three distinct layers:

1. **Agent Control Plane:** Coordinates cloud-managed agent fleets, tracking their execution lifecycles, work queues, roles, and interactive sessions. 2. **Tool-Building Engine:** Enables agents to generate compact, standardized Temper applications that bridge SDLC workflows across Git, CI, and deployment pipelines. 3. **Helix Control API:** Acts as the lifecycle and control interface wrapped around the Helix data plane to systematically drive and verify synthetic workloads.

This implementation uncovered an unexpected realization: Temper is fundamentally broader than mere agent orchestration infrastructure. Sesh Nalla reflects:

> “The surprise was it started to feel more general than agent infrastructure. A lot of software, if you squint closely, is just control logic around database, APIs around state, policies around mutation, lifecycle transitions.”

When evaluating why this differs from simply asking Claude Code to generate a standard TypeScript or Python CRUD application, the difference lies in state architecture. While standard CRUD applications scatter control logic across routes, database constraints, background workers, service layers, and documentation, Temper brings the operational state machine front and center. Drawing inspiration from battle-tested platforms like Erlang/OTP and the actor model, Temper makes the underlying state machine completely explicit, verifiable, and safe for autonomous agents to drive.

- **Dark Factory Metaphor:** Borrowed from manufacturing via Simon Willison, describing software environments where agents operate reliably without active human supervision. - **Helix Performance and Savings:** Built within a week, Helix showed potential 2x to 5x cost improvements when shadowing Datadog's production Kafka workloads. - **Explicit vs. Implicit State:** While typical CRUD architectures bury operational lifecycle rules inside procedural code and separate database constraints, Temper defines lifecycle transitions and mutation policies as first-class, inspectable state machines.

[20:00](https://www.youtube.com/watch?v=EdmuYPBt_EM&t=1200s) Temper Internals: Blueprints, Formal State Transitions, and Hot Reloading

Sesh Nalla dives into the architecture of Temper by comparing it to durable execution engines like Temporal and battle-tested actor environments like the Erlang BEAM runtime. Rather than having an agent emit raw, messy application code, Temper asks the model to operate inside an operational domain and generate a structured blueprint through iterative plan mode. This declarative artifact answers fundamental questions: what states exist, which transitions are legal, who is permitted to trigger them, what invariants must hold, and how tool failures are handled. A specialized external compiler verifies this blueprint and hot-deploys it directly into the execution environment without needing traditional CI cycles. Sesh surveys the crowd to see who is familiar with the Erlang BEAM, noting that running high-assurance systems where an agent can hot-deploy code mirrors how hardened platforms have historically functioned.

```mermaid flowchart TD accTitle: Temper Build and Runtime Architecture accDescr: Shows how an agent blueprint is compiled into a transition table and executed through policy gates and side effects. subgraph Build["Build Path"] A["Agent (Claude Code)"] -->|"Iterative plan mode"| B["Blueprint (States, Actions, Triggers)"] B --> C["Compiler / Verifier"] end subgraph Runtime["Runtime Execution"] C -->|"Hot deploy"| D["Transition Table (Data-like Spec)"] D --> E{"Policy Gates"} E --> F["WASM Modules / Side Effects"] F -->|"Callbacks"| D end ```

The runtime interface behaves like a standard CRUD API, but underneath, the critical abstraction has been elevated away from procedural code. Temper transforms the blueprint into formal state transitions. Sesh jokes about his slide’s “completely brutalistic design” lacking syntax highlighting, but emphasizes the clarity of an agent-written spec for something like a helix rollout. In that system, an action like `start rolling` is only legal when the deployment status is `planned`. Once triggered, it moves to `rolling`, kicks off a non-idempotent side effect—such as patching a Kubernetes StatefulSet—and waits for an asynchronous callback to mark progress or fail. Because control logic is turned into a data-like transition table rather than being buried in spaghetti code or improvised service methods, agents can inspect it, reason about it, and make targeted modifications safely.

- The agent produces a structured blueprint rather than arbitrary application code, elevating the level of abstraction. - Compilation and formal verification take place entirely outside the LLM, similar to how Rust code is handed off to the Rust compiler. - Formal state transitions represent the core application as a precise transition system, making it far easier for both humans and agents to reason about. - Agents can modify individual states or rollback paths and hot reload them without risking regressions or skipping safety checks.

> “Formal here doesn’t mean every possible property is proved. It isn’t theoretical. It means the basic shape of the application is represented as a precise transition system.” — Sesh Nalla

> “It’s important to note that the agent is not generating arbitrary application code directly. We have raised the abstraction.” — Sesh Nalla

---

[25:00](https://www.youtube.com/watch?v=EdmuYPBt_EM&t=1500s) Policy Gates, WASM Sandboxing, and Multi-Level Verification

Because Temper's state transition tables are structured purely as data, the runtime evaluates state transitions and policy decisions side by side. Teams can configure strict role boundaries between agents: operator agents running in an autonomous "dark factory" setup might only be permitted a narrow set of execution tasks, while builder agents are explicitly forbidden from executing production changes. Operators can enforce invariants deterministically, answering questions such as whether a completed rollout can ever be re-triggered or whether a failed tool execution is allowed to continue down a rollout path. When side effects are required, Temper isolates them into small, typed operations. To run arbitrary execution logic without turning the state machine into an uncontrolled backdoor, that logic is sandboxed into WebAssembly (WASM) modules, severely restricting the surface area where LLM-generated code can misbehave and streamlining debugging.

```mermaid flowchart LR accTitle: Swiss Cheese Verification Model accDescr: Four levels of verification checks ranging from algebra to property tests. L1["Level 1: Automata Algebra"] --> L2["Level 2: Reachable State Graph"] L2 --> L3["Level 3: Fault Injection & Schedules"] L3 --> L4["Level 4: Randomized Property Tests"] ```

The final core pillar of Temper is the verifier, which Sesh identifies as the primary industry bottleneck for autonomous software development. Acting as a strict gatekeeper before transition tables can be loaded live into the runtime, the verifier operates on a multi-tiered "Swiss cheese" model where Claude evaluates how many tiers are necessary based on compiler feedback:

1. **Level 1 (Algebra Check):** Validates the mathematical and structural algebra of the state automata. 2. **Level 2 (Model Checking):** Exhaustively explores the reachable state graph to guarantee no path terminates in an invalid or dangerous state. 3. **Level 3 (Schedule Simulation & Fault Injection):** Injects faults, manipulates timing schedules, and ensures system invariants hold under adverse conditions. 4. **Level 4 (Property Testing):** Executes randomized action sequences to uncover subtle regressions.

- Verification gaps compound positively over time: every bug or edge case discovered in production or simulation is encoded directly into the model and test suite, echoing Boris Cherny's observations on compounding test suites. - Confining arbitrary generated code strictly to WASM modules prevents unchecked side effects from polluting the state machine. - Policy evaluation happens in tandem with state transition logic, ensuring fine-grained access control across different specialized agent roles.

> “Verification is basically the bottleneck right now, for pretty much everything.” — Sesh Nalla

> “FX are deliberately small typed operations here in Temper. Keeping them small prevents the state machine from becoming a backdoor for arbitrary application behavior.” — Sesh Nalla

---

[27:30](https://www.youtube.com/watch?v=EdmuYPBt_EM&t=1650s) Machine Tools for Software: From Dark Factories to Directed Evolution

Reflecting on years of operating mission-critical infrastructure at Datadog and getting woken up thousands of times in the middle of the night, Sesh stresses that humans cannot maintain mental models of thousands of lines of tangled, AI-generated code. Whether running financial systems or cloud infrastructure, applications must remain decomposable into concise, readable artifacts that humans can inspect and reason about. Returning to the historical analogy of the Industrial Revolution, Sesh observes that the industrial breakthrough came not from an artisan building one magnificent machine, but from inventing machine tools that produced standardized, interchangeable, and inspectable parts. Building scalable, production-ready databases and critical software with autonomous agents requires that exact same discipline.

Sesh concludes with an alternate vision of the future. While the concept of a fully automated "dark factory" represents automated efficiency, the term itself feels cold and sad. Instead, when software is constructed out of small, verifiable, and interchangeable components, the codebase begins to behave like a living organism that engineers can grow, cultivate, and guide through directed evolution. Rather than static architectures, systems can adapt dynamically based on real-world usage patterns—such as automatically mutating Kafka into an optimized FIFO queue or a lightweight buffer based on customer traffic. Sesh thanks the audience and wraps up the presentation.

- Mission-critical systems must remain small, explicit, and human-readable; throwing thousands of lines of untangled generated code at an LLM for review is an unreliable strategy. - High-assurance disciplines from aviation and financial engineering have used formal state transitions for decades, but the human labor cost was historically prohibitive for general software engineering. - AI agents drastically shift the economics of formal methods, making extreme rigor and composability cost-effective for general application development.

> “The win was not that one artisan could build a brilliant machine, the win was that a machine built with machine tools made parts composable and inspectable and replaceable that we could build larger machines.” — Sesh Nalla

> “However, the cost of such rigor with humans was too high for general software until now. Agents are changing their calculus.” — Sesh Nalla

> “So the whole software built this way can feel like an organism, that we can grow, cultivate, and evolve through feedback, selection, and adaptation.” — Sesh Nalla

## Notable quotes > [07:44](https://www.youtube.com/watch?v=EdmuYPBt_EM&t=464s) Mechanization means agents are doing more of the work now, and industrialization, if I were to borrow the metaphor, means work becomes repeatable, verifiable, controllable and scalable. — Sesh Nalla

> [15:00](https://www.youtube.com/watch?v=EdmuYPBt_EM&t=900s) A lot of software, if you squint closely, is just control logic around database, APIs around state, policies around mutation, lifecycle transitions. — Sesh Nalla

> [25:00](https://www.youtube.com/watch?v=EdmuYPBt_EM&t=1500s) Verification is basically the bottleneck right now, for pretty much everything. — Sesh Nalla

> [27:30](https://www.youtube.com/watch?v=EdmuYPBt_EM&t=1650s) The win was not that one artisan could build a brilliant machine, the win was that a machine built with machine tools made parts composable and inspectable and replaceable that we could build larger machines. — Sesh Nalla

## Mentioned Datadog, Claude Code, Sesh Nalla, Temper, Courier, Bits Evolve, Helix, Apache Kafka, Claude Opus 4.5, AlphaEvolve, DeepMind, Dario Amodei, Simon Willison, Boris Cherny, Temporal, Erlang/OTP, Erlang BEAM, WebAssembly (WASM), Kubernetes

## Video description At Datadog, 90% of engineers adopted AI coding tools for production work in the last four months, with Claude Code driving two-thirds of that usage. As sessions grew more ambitious, the reusable tools they produced — for verification, debugging, orchestration — sprawled into unmaintainable one-offs. Sesh Nalla, VP of Engineering, shares how Datadog built Temper: a constrained framework that produces secure, reusable tools that compound across sessions and teams instead.

[Source](https://www.youtube.com/watch?v=EdmuYPBt_EM)

Reading summary

Sesh Nalla, Vice President of Engineering at Datadog, argues that as AI agents compress software construction from months to days, the primary bottleneck shifts to verification, coordination, and tooling. To move beyond fragile, ad-hoc automation, Datadog developed Temper, a declarative framework functioning as an industrial "machine tool" to ensure agent-generated systems remain deterministic, verifiable, and safe for production. The Evolution from Manual Rigor to Agentic Construction (2:22) Tracing Datadog's engineering lineage, Nalla contrasts Courier—a distributed queuing system built entirely by human engineers over 12 months in 2024 using formal modeling and manual verification—with subsequent evolutionary experiments. In September 2025, Datadog launched Bits Evolve, using a multi-model council to mutate hot code paths guided by test cascades and telemetry. The experiment showed that automated adaptation succeeds only when the verification harness and environment are robust. Shifting Bottlenecks and Tool Sprawl in Production (5:14) By late 2025, using models like Claude Opus 4.5, a single engineer generated Helix—a Kafka-compatible streaming engine—in approximately three days. While construction time collapsed, getting Helix production-ready revealed a severe "mileage" gap: unconstrained agent evolution produced operational edge cases that demanded heavy human steering. Within four months, Datadog saw 90% of its engineers (about 3,000 people) adopt AI coding tools, with Claude Code driving roughly two-thirds of that volume. However, individual sessions running up to 28 hours created brittle, ad-hoc scripts and isolated markdown notes, turning the human engineer into an awkward manual bridge between rapid agent generation and human-oriented delivery pipelines. Temper: Turning Control Logic into Formal State Machines (15:00) To eliminate throwaway scripting, Datadog designed Temper as a "machine tool" for agent development. Instead of emitting arbitrary application code, Claude Code generates a structured, declarative blueprint defining states, legal transitions, and invariant rules through iterative planning. Inspired by Erlang/OTP, the actor model, and durable execution engines like Temporal, Temper hoists control logic out of procedural routes and database tables into an explicit, inspectable state transition table. This table is compiled, verified outside the model, and hot-deployed directly into the execution runtime without traditional continuous integration cycles. Multi-Level Verification and Sandboxed Execution (25:00) Because Temper structures state transitions as pure data, it evaluates transition rules and policy gates side by side, allowing strict role separations such as barring builder agents from executing production changes. To prevent arbitrary code from compromising the state machine, Temper isolates side effects into small, typed operations sandboxed in WebAssembly (WASM) modules. Safety is enforced by a four-tiered "Swiss cheese" verifier evaluating automata algebra, reachable state graphs via model checking, schedule fault injection, and randomized property testing, transforming discovered production bugs into compounding test suites. Nalla concludes that while formal state methods were historically too labor-intensive for general software development, frontier agents shift this economic calculus. Adopting machine-tool discipline allows organizations to move past opaque codebases and cold, lights-out "dark factories" toward modular architectures that engineers can reliably inspect, cultivate, and guide through directed evolution.

Source quotes

[7:44] Sesh Nalla: Mechanization means agents are doing more of the work now, and industrialization, if I were to borrow the metaphor, means work becomes repeatable, verifiable, controllable and scalable. [15:00] Sesh Nalla: A lot of software, if you squint closely, is just control logic around database, APIs around state, policies around mutation, lifecycle transitions. [25:00] Sesh Nalla: Verification is basically the bottleneck right now, for pretty much everything. [27:30] Sesh Nalla: The win was not that one artisan could build a brilliant machine, the win was that a machine built with machine tools made parts composable and inspectable and replaceable that we could build larger machines.

Original transcript

[0:00] . Thank you. please welcome to the stage VP of engineering of data dog Sesh Nala Thank you. I know it's 4:00 PM. How's the energy going right now so far? How are you like in the conference? Great? Alright, let's see some blood flowing, so a quick show of hands. if you have heard of machine tools before. What's on the slide? Have you used any? Thank you. I was expecting zero hands anyways, so that's the talk anyways. So these are tools like jigs, fixtures, gauges, and melts that you see in manufacturing. to produce precise and repeatable machine parts, the kind that, you know, you assemble them into larger machines like engines, aircrafts, nuclear reactors, lunar landing modules that we saw this morning, right? So machine tools were a breakthrough during the industrialization period that enabled the scale due to the interchangeability with standardization and precision. So that was the inspiration for my talk and what we built at Datadog, and I will share with you how all this fits into building software with Cloud Code. So what you're seeing on the slide there is My view of the last 18 months, I think there is a different view of this graph at multiple sessions today. These are non-scientific, so don't please... consider that my eval is purely personal, but is the case of ambition with the models. For most of 2025, The models were useful to me, but within very narrow boundaries, like local changes, small functions. Peace. glue code, throw away prototypes for me to learn a lot, kind of learned a lot through them. And then around late 2025, I think you all have seen this and noticed it, when the slope started to change, like this exponential, I started trusting Clark with larger and more ambiguous system-scoped work. Thank you. Before I talk about machine tools, I want to go through the lineage of some ideas that led me here. So this is 2024. It's a write-up. We were building a distributed queuing system called Courier from scratch. All of this was before agents, all by hand. Can you believe it? Like still software. was built by Hand Dan. The hard part was, for any distributed system that is human-built or agent-built, It's not just building the parts, but it's building the pieces. and making the interaction between them observable, testable, and verifiable. So we were rigorous with formal modeling and simulation, so you see various techniques in this post. All of this is classical systems work. where you identify the pods, where mistakes would be expensive or hot rivers, and you raise the rigor for those parts that you don't want them to slip to production. Thank you. The next idea was around September 2025, We called it Bits Evolve. It's a closed-loop evolutionary optimization harness, inspired by Alpha Evolve from DeepMind, The idea is that you let parts of code improve themselves on a narrow, controlled hardness. I think we have seen some announcements today at the keynote about dreaming and various loops. So this was us trying in September 2025. It's an ensemble or a council of models, big and small, and they generate variants of code, whatever you designate, you want them to improve. In our case, it's hot functions, blocks of code. And then you have a casket that you see on the right-hand side of the screen with benchmarks, tests, and production observability. decide what survives. It's like natural selection. So this was the first glimpse for me that parts of software maybe could be cultivated like living organisms, like plants or like microbes, and growing through variation with feedback and adaptation. Thank you. [5:00] - The insight was that, This kind of evolution is only as good as the environment it adapts within. If you have bad benchmarks, they produce bad evolutions. Weak observability, your optimizations are shallow. Thank you. So then during this period of leap, or the exponential that started, we were building whole distributed systems with cloud code. I mentioned the Opus 4.5 inflection point where I raised my ambition. It wasn't like sudden, like I didn't wake up one day and then started trusting cloud with whole systems or even my databases, we have a lot of them. It happened progressively. Like I tried harder tasks than larger subsystems, failed a lot, learned a lot through a lot of experiments. and then started seeing successes around this period. So what we decided, To be more ambitious, "Okay, could we build something as big as Kafka?" Show of hands if you've heard of Kafka. OK, more hands now. So Kafka is a streaming service. So we kind of attempted, OK, can we build this from scratch? It's like repeating the same... playbook that we had for courier it's a queuing system pretty close right same rigor so you will see the Cask it on the right-hand side. of the slide. But Cloud Code doing most of the construction with one human Building it. In a few days, to our disbelief, we had a full functional CAFTA-compatible system working. And we called it helix. So the source code, methodology, and the details are all linked. in this post, feel free to check them out later. But taking Helix to production now requires building mileage. And that has been challenging for us. Amen. So the next natural move for us was, I spoke about Bits Evolve earlier, Could bits evolve parts of Helix? the same way it evolved hard functions. The ambition was too big, right? From small functions to like, can the whole component evolve? Can it provide the mileage we needed so we could get to production? The answer was not quite. The surface area was too large. Even with the verification casket that I showed you in the previous slide, quite rigorous, There were too many places where the human had to interpret and correct It's too multi-turn, too interactive. So we're like, OK, can we look for a narrower surface? Can we dial back the ambition a little bit? So this post was about that. So we chose... or metrics aggregation server. We are data dogs, so we have lots of metrics. Could we improve the materialization logic, life, not offline like what we did with Bitsowall before. Can we optimize them per customer? with a proof carrying path around the change. So, A human doesn't have to review every candidate that is being generated. So if you look at the flow across these four projects, If you observe the pattern, each project exposed the bottleneck for the next. There were many talks and... comments today about the bottleneck being moved. So with Courier, the bottleneck was construction, with humans building systems through careful design, modeling and verification. It took us one year to build Courier. And then if you compare it, it took like three days. to build Kafka. In like what, 12 months? So with bits evolved, the bottleneck moved. to the feedback loop. Like the model produces variation, but the hardness decides what survives. And with Helix, the Kafka system that we built, the bottleneck moved again, where agents could build large parts of the system, so we have seen it, But then the human have to coordinate to ship the work to production, through tools and mechanisms built for humans. So that's the Amdell's law that Dario was talking about earlier today. So that's the jump we are making, if you look at this slide at the center, from mechanization to industrialization. So mechanization means agents are doing more of the work now. An industrialization, if I were to borrow the metaphor, And that's why I kind of introduced machine tools. means work becomes repeatable, verifiable, controllable and scalable. So the idea of a machine tool is I know it sounds cool, but... Why do we need them now for agent build software? aren't these for just lunar? modules to land. Because Thank you. of the complexity and the ambiguity growing, right? So each time we are trying to like Increase our ambition levels started from targeted changes on existing systems. In the last four months, about 90% of Datadog use [10:00] AI coding tools for production code That's roughly 3,000 engineers. and CloudCode drove at least two thirds of that. Thank you. And most of the work, as I described, about Helix was still single human driving, like one engineer steering one or more agent sessions. And the work was moving across this map. more complex to generate on one axis, and more ambiguous to verify on the other axis. Here are a few concrete examples. I'm not going to enumerate all of these. but feel free to take a picture if useful. And if any of these resonate with you, I'm happy to chat about more afterward or later on. But the main point I want to underline here is that These are generating personalized flows in our software development lifecycle. Because one human could do a lot, a lot more than what they were used to do before. Thank you. So for engineers, in the software delivery lifecycle I was talking about, if I were to use the word flow, It used to mean direct relationship between intent and code. You understood the problem, you wrote the code, you tested it, you reviewed it, you shipped it, you operated it, you repeated it again and over and over again. But with agents, That abstraction level is changing rapidly. I don't know. I haven't seen Cord. I mean, I haven't seen code in a while. I've made my peace with it because I've been a manager for a while, but many people like they're still trying to go through their seven stages of grief. of not seeing code on a day-to-day basis, right? So you're no longer writing the code. You're shaping the work. You're deciding what the agent should see. We saw the keynotes today, like outcomes, right? what tools it should have, what success means. how failure should be detected. All of this is powerful, it's like everyone's promoted three levels up. into management chain, which they weren't signed up for. engineers, right? Because it's a huge leap, it is also disorienting. It's like you're pushed against gravity very fast. it can feel sickening, right? We haven't acclimated to this altitude of working. specifically engineers. who love looking at God. Thank you. So before this jump in model capabilities, The human team was the factory. Tools were designed around human attention, our judgment, and the operational memory of what's actually happening in production. is in this world. human organization, our collective brains, our minds. Connecting back to courier, like I said, only 12 months ago, this was the world we were in. And only four months ago, it continued and started to change. And that's the inflection point with Cloud Code and Opus 4.5. We are starting to see one lead human, coordinate multiple interactive sessions. I've seen like many screenshots of like parallel sessions, Cranking out stuff. It's disorienting for me to personally watch. three four five agents working on different parts i heard jared today saying he's doing 10 things at a time Like the stuff she was showing was only 10% of his whatever time being spent. So these tools were still human shaped. The agents work two orders of magnitude faster, and this tool chain isn't built for their speed. So what happened? The human became the bridge between agent execution and the human-shaped systems. Thank you. And now all this operational knowledge like you wake up at night, something broken, it's just in that person's head and probably in some markdown files that agents worked on. and in between them. Thank you. So that further amplified with cloud managed agents. So agents are doing a lot more background work, Right? It's compute. and they start taking judgment-bearing roles. Meaning they're not just following your instructions anymore. They're making their own decisions. and they're running longer for hours. sometimes overnight, sometimes for days. I don't know, the longest task someone like... Benchmark 20 hours, 28 hours. So they construct their own tools in these sessions. But I don't call it. And the mismatch is still there. Like each agent invents its own tools, its own glue, its own conventions, And that system becomes really hard to share and operate. You can see the blur. between what the agent sessions produced as like intermediate tools, and what is your product doing, like your code. You get a lot of output fast. A lot of it is useful, but some of it can look like false progress and most of the tool construction that only makes sense in that local session. And you start to see that blur. So this is where I felt we need something. more structural. [15:00] If agents are going to build, and operate large parts of our systems of our databases which are mission critical, they need the equivalent of this machine tool concept. that I'm trying to introduce. So temper. as a machine tool. The idea is, instead of agent inventing disconnected tools for every local need, it produces precise specifications of the intent and problem domain. It is a machine tool in the same sense that a JIG or a CNC machine, have you seen some of those that computer aided? hot machines where you give them specifications of what your screw threading needs to be, what this needs to be, it's extremely repeatable, you can run them and you can build aircrafts and things like that with them, right? So in this case, the agent does not improvise the final mechanism each time, it produces a precise description and iterates with temper, or a temporal-like mechanism, to make something work first, and then later turn that into something repeatable, checkable, and reusable. So you could actually build a software factory around your code base. Thank you. So that's the concept of a dark factory. Simon Wilson of SimonWillison.net Pretty amazing blog. I think he is one of the most influential AI voices right now in teaching how to work with agents and build software. has been popularizing this phrase called dark factory. I think it's a pretty good encapsulation of a software process where the agents keep working without the humans on the virtual factory floor. You can turn off the lights. So the human role now becomes like designing the factory and the constraints and the outcomes and the verification loop. So this thing can run for hours and days and weeks. producing what you wanted it to produce. So something like Tempor can fill in this role of a machine tool to build such factories. Let's look at a dark factory concretely with Helix as a target. what I shared about Helix. It's a Kafka Lake streaming service. Thank you. It is probably one of the five expensive services we are running at Datadog, like Kafka. So we have been shadowing our production workloads with Helix. In some cases, we actually believe it can be materially cheaper than our current production solution. Can you believe it? It took like a week. to build it, and we started shadowing it. And we saw like 2 to 5x opportunities that it can be cheaper. But getting from this promising Stay it. to production still takes to work on the mileage, the system needs to earn. that it can run and multiple people can operate it, not just the person who built it. Thank you. So we created a bunch of synthetic workloads that models our production ships And we constructed a factory. So a software factory for Helix uses Tempor in three distinct ways. First, as an agent control plan, two cloud-managed agents, where the sessions, roles, work, queues, and operational lifecycle are more precisely managed, Second, as a way for agents to build their own tools with small temper apps, Bridging. the SDLC tooling, like Git CI deployment, And third, as a Helix Control API, the interface and the life cycle surface around the helix data plan to exercise this workload. So that was a surprise to me. Thank you. The surprise was... It started to feel more general than agent infrastructure. A lot of software, if you squint closely, is just control logic around database, APIs around state, policies around mutation, Life cycle transitions. Okay. integrations with external systems. So Tempur could be this universal in a sense that it can be applied to any software that has the shape I described. Thank you. Before I go deeper into how temper works, You might be wondering at this point, Why is this different from asking Cloud Code to build a CRUD app, like in TypeScript? or Python. Clark can do that very well. We have seen like lots of PRs and lots of code going. However, in normal CRUD apps, the control logic is spread across routes, database constraints, service code, background jobs and documentation. It may all have good tests and coverage, but the operational model which is generally a state machine shaped, is mostly implicit in the code base. So the idea is Tempor makes that state machine explicit. This isn't particularly novel. We have this with runtimes like Erlang. Erling OTP for decades, actor-runner [20:00] times and more recently with workflow engines and durable execution run times like temporal they all have like popularized this precise runtime so you can run invincible applications. Thank you. So let's look at some temporal internals. to understand. Thank you. on the build path for TEMPR, It asks for this operational domain I'm describing about the model, what you want in the form of a blueprint. Like what are the states? which traditions are legal. Who can request them? What effects are allowed? what invariants must hold. What happens if a tool call fails? Right? So the agent will often iterate and arrive at this blueprint over like multiple turns think of it like it's its own plan mode to arrive at, instead of arriving at a markdown file of some description, it could arrive at this precise declarative artifacts. And then you have a compiler equivalent, like Temple, verify it. and it can hard deploy into a runtime. There are other runtimes who does this, Erlang Beam does that if you have heard of it. Anybody heard of Orlang B? There you go. Thank you. So the run path feels the same as any other CRUD shaped API. you wouldn't notice the difference. It's important to note that the agent is not generating arbitrary application code directly. We have raised the abstraction. It's generating a structured description that compiles into our end-time shift. The compilation step is outside the LLM. It's the same like you write Rust code and you give it to Rust compiler, right? So Tempo turns this blueprint, into something called formal state transitions. It's very common. functional programming and actor runtimes if you have used them. This is the most important technical detail. Thank you. Formal here doesn't mean every possible property is proved. It isn't theoretical. It means the basic shape of the application is represented as a precise transition system. And when you have that, it is a much better reasoning for both humans, and agents than arbitrary code. Pardon my completely brutalistic design of this slide with no. syntax highlighting, but that's an agent-written spec. For a helix rollout, that spec looks like this. states, actions, and triggers. I actually got an idea this morning when I saw the announcement on cloud managed triggers Maybe this could be just a pass-through, you could declare a trigger here and then have Claude run them. But the idea is you define your states, actions, and triggers, or Asian does it, Claude does it, And in this case, like the deployment description start rolling is only valid when planned or entity moves to rolling and that triggers a side effect like going and patching a Kubernetes stateful set. which is not idempotent. and then a callback that comes back, whether to mark the progress or fail. What Tempor does, so that artifact is declarative, so what Tempor does, it takes this and generates that spec into this transition table. A concept. So, where the critical control logic is data-like. It is not just Spaghetti Um... imperatively encoded in code, it's just data lake that's interchangeable uncheckable, and it is not hidden in any improvised chain of service methods. This is easier for agents to work with. And they can change this dynamically. with safety. That's the promise, right? I have seen people like writing Erling scripts and hard deploying into Beam, I think that's a pretty creative way of using run times that are extremely hardened, over higher assurance systems for agents to work with. So in this case, if a rollout needs a new state or a rollback path, The agent can make a targeted spec change. and it can hot reload it. So the iteration speed is even faster. You don't have to go through CI and then deploy and everything. It's like when you're leaving agents overnight, I think they can come up with some pretty good progress. overnight. without compromising on safety PR reviews and code reviews and things like that. Thank you. And we have policy gates. Because the transition table is data-like, Thank you. Tempo evaluates the state decisions and the policy decisions together such as like who can mutate a deployment rollout? which actions can an operator agent, like the team of agents that are working on the dark factory, you can say these operator agents can only do so much. or these actions are forbidden [25:00] for builder agent. And You can ask or you can specify independently as a human Can a completed rollout be rolled again? Or can a failed tool result continue on its rollout path, etc.? Thank you. And then we have the side effects. An effect system. Also very popular and typed Runtimes like TypeScript. FX are deliberately small typed operations here in Tempur. Keeping them small prevents the state machine from becoming a backdoor for auditory application behavior. But if you need arbitrary application behavior, you could package them as WASM modules, Familiar with wasn't? Come on. OK, a few hands. Thank you. So this is where arbitrary code generated by an LLM can leave. It's a very narrow place. So what that means is, it makes troubleshooting easier. And the last building block. for Tampur is the verifier. Thank you. Verification is basically the bottleneck right now, for pretty much everything. We have seen so many announcements and discussions around it. So in Tempor, the verifier is a gate before the transition table is loaded into the runtime. So that's what allows it. to say this is safe, you can put this and load it live. And it is multiple levels, like a Swiss cheese pattern. Not all levels need to find everything exhaustively. The agent will, Claude is generally very good at, making judgment calls. Do I need to run all levels? or just some levels which seem like it uses its compiler output. Like level one checks the algebra of the automation, Level 2 model checks the reachable state graph. Can any path reach a bad state? And level 3 runs schedules, injects falls, do invariants survive? timing, failure conditions and et cetera. and level four uses property tests. which I highly encourage that we have to start learning, this mechanism of randomized testing with action sequences. All of this isn't exhaustive on day one. Every discovery gap compounds the verifier I also heard Boris mentioning the compounding effect of like, you find tests and then you keep fixing the gaps. A missing condition revealed in production or simulation can get added to the model or the test suite. And it gets compounds. Thank you. So where is all this going? I don't know. I mean, I can't predict much... about where this could go. But the idea is if each artifact of a temper app or a bundle is concise, very few lines, fits in your head, I spend a lot of time running machine critical infrastructure for Datadog, Woken up at night like thousands of times. Past. three or four years. you won't be able to keep your complex machine critical logic in your head when you want to operate it. For a complex business domain like banking or financial systems, you still should be able to encode your business logic this way, Something a human can read. If the generated artifact is thousands of lines of tangled code, we are back to where And I don't know entirely yet that an LLM can just completely review and then sign off on it. Thank you. If it is few small artifacts with explicit roles, when both humans and agents can modify the system without disturbing everything around it, I still feel like this is just good systems engineering anyways, and high assurance software like aviation and financial systems have been built this way for decades. However, the cost of such rigour with humans was too high for general software until now. Agents are changing their calculus. Going back to my manufacturing metaphor, The win was not that one artisan could build a brilliant machine, The win was that a machine built with machine tools made parts composable and inspectable and replaceable that we could build larger machines. And my claim is that for software, Agent build software, we need such kind of rigor for us to scale from here on if you were to really build databases and put them in production. I learned where I started. If agents can build software autonomously inside factories, with such discipline and rigor, Maybe, maybe, we don't need to stop at dark factories. I don't know, there's a word dark in there. It kind of sounds sad. So the whole software built this way can feel like an organism. that we can grow, cultivate, and evolve through feedback, selection, and adaptation. Thank you. And it looks like, I don't know, agriculture... or a directed evolution where you choose the direction in which your software needs to go You can choose Kafka to be a queue because there are more customers using it as a FIFO queue versus, no, we need more buffers. We don't need all these brokers. [30:00] Or maybe just dreaming, right? We have dreams now. And Clark. That's all I have. Thank you so much for listening. Bye.