LangChain Automates Agent Code Fixes as Chase Targets Self-Healing AI

How to Build Agents That Improve Over Time: Model, Harness, Context | Interrupt London — LangChain

LangChain

Executable evaluation suites must replace traditional product requirement documents because language models fail through semantic breakdowns rather than syntax crashes, requiring teams to turn execution traces into regression benchmarks that update model weights, code harnesses, and context memory.

AI-assisted summary of How to Build Agents That Improve Over Time: Model, Harness, Context | Interrupt London — original source. Check the original for full context.

Source published:

Key ideas

  • Harrison Chase deconstructs an agent into three core components—model, harness, and context—arguing that systematic self-improvement can occur across each layer independently. (0:00)
  • LangChain introduced dedicated infrastructure including the smithtune CLI for model fine-tuning and LangSmith Engine, an autonomous Deep Agent that clusters runtime failures and generates pull requests. (3:22)
  • To handle semantic failures that bypass standard logs, LangChain created SmithDB and AI Search to query agent execution trajectories using free-form semantic questions. (10:44)
  • Chase positions executable eval suites as the new PRD for AI applications, emphasizing that grading external environment mutations is often far more reliable than assessing free-form text output. (12:02)
  • LangChain outlines an autonomous 'North Star' improvement flywheel where agents flag their own errors, generate regression test cases, and propose self-healing code patches. (19:28)

Detailed notes

## Background This talk was delivered by Harrison Chase, Co-Founder and CEO of LangChain, and published on YouTube by LangChain on October 27, 2026, from their Interrupt London conference. According to the page description, Chase explains how agents can improve across three architectural pillars—model, harness, and context—using traces and evaluations as a common foundation, while exploring tools like LangSmith Engine, smithtune, and OpenWiki.

## TL;DR Chase deconstructs agents into three core components: the model (reasoning engine), the harness (execution code, sandbox, and control flow), and the context (dynamic knowledge and memory). He explains how each layer can be upgraded over time, ranging from fine-tuning and reinforcement learning for model weights, to automated pull requests and refactoring via LangSmith Engine for the harness, to dynamic knowledge bases like OpenWiki for context. Throughout, he argues that high-fidelity traces and rigorous evaluations serve as the shared foundation for diagnosing semantic failures, running red teaming probes, and preventing regressions. Finally, he outlines an improvement flywheel that transitions from human-driven debugging to a North Star vision where agents autonomously detect their own errors, generate eval cases, and patch their own code and context.

## Atmosphere & style The presentation carries the pragmatic, grounded energy of a technical conference keynote tailored directly for AI builders and system architects. Chase speaks with clear, rapid-fire engineering conviction, deliberately tempering industry hype by acknowledging that fully automated agent self-improvement remains an aspirational North Star rather than an immediate reality. Watching the talk feels like peering into an active developer workshop—analytical and structured, yet energized by real-world case studies, architectural diagrams, and the tangible frontier of agent observability.

## Bulletin - [00:00](https://www.youtube.com/watch?v=xy_7SzwsIjM&t=0s) Harrison Chase deconstructs an agent into three core components—model, harness, and context—arguing that systematic self-improvement can occur across each layer independently. - [03:22](https://www.youtube.com/watch?v=xy_7SzwsIjM&t=202s) LangChain introduced dedicated infrastructure including the smithtune CLI for model fine-tuning and LangSmith Engine, an autonomous Deep Agent that clusters runtime failures and generates pull requests. - [10:44](https://www.youtube.com/watch?v=xy_7SzwsIjM&t=644s) To handle semantic failures that bypass standard logs, LangChain created SmithDB and AI Search to query agent execution trajectories using free-form semantic questions. - [12:02](https://www.youtube.com/watch?v=xy_7SzwsIjM&t=722s) Chase positions executable eval suites as the new PRD for AI applications, emphasizing that grading external environment mutations is often far more reliable than assessing free-form text output. - [19:28](https://www.youtube.com/watch?v=xy_7SzwsIjM&t=1168s) LangChain outlines an autonomous 'North Star' improvement flywheel where agents flag their own errors, generate regression test cases, and propose self-healing code patches.

## Key takeaways - [00:00](https://www.youtube.com/watch?v=xy_7SzwsIjM&t=0s) An agent consists of three distinct architectural pillars: the model (reasoning engine), the harness (control flow, sandbox, tools, execution), and the context (text, files, and injected knowledge). - [00:00](https://www.youtube.com/watch?v=xy_7SzwsIjM&t=0s) Agent improvement workflows span three maturity tiers: manual AI engineer debugging, human-in-the-loop automated remediation, and fully autonomous self-updating agents. - [03:22](https://www.youtube.com/watch?v=xy_7SzwsIjM&t=202s) Model weight updates via supervised fine-tuning (SFT) suit narrower tasks like extraction or classification, while reinforcement learning (RL) offers a higher ceiling for complex reasoning despite lacking mature managed tooling. - [05:52](https://www.youtube.com/watch?v=xy_7SzwsIjM&t=352s) LangSmith Engine operates as a Deep Agent within a sandbox to monitor traces, cluster recurring runtime bugs, draft code patches, and submit pull requests for human review. - [07:13](https://www.youtube.com/watch?v=xy_7SzwsIjM&t=433s) Context is the most adaptable improvement layer because it relies on plain text or markdown and can update dynamically across individual users, projects, or entire organizations. - [08:42](https://www.youtube.com/watch?v=xy_7SzwsIjM&t=522s) OpenWiki provides an open-source, self-updating knowledge base for coding agents by analyzing execution traces where agents get stuck and feeding solutions back into indexed docs. - [09:23](https://www.youtube.com/watch?v=xy_7SzwsIjM&t=563s) High-fidelity execution observability relies on a strict structural hierarchy: runs (individual invocations), traces (task sequences), threads (conversations), and trajectories (standardized thread views). - [10:44](https://www.youtube.com/watch?v=xy_7SzwsIjM&t=644s) Because agents fail through semantic and conversational reasoning breakdowns rather than syntax errors, traditional logging databases are inadequate, prompting the creation of SmithDB and AI Search. - [12:02](https://www.youtube.com/watch?v=xy_7SzwsIjM&t=722s) Evals are replacing PRDs as living specifications because static requirements cannot capture the long tail of edge cases and off-topic user inputs encountered in production. - [13:38](https://www.youtube.com/watch?v=xy_7SzwsIjM&t=818s) Evaluating agent outputs by inspecting side effects—such as database state mutations and tool call sequences—is significantly more reliable than judging generative text responses. - [14:57](https://www.youtube.com/watch?v=xy_7SzwsIjM&t=897s) Running comprehensive eval suites requires isolated simulation environments and mock sandboxes to prevent destructive mutations on production databases or external third-party services like Salesforce. - [16:01](https://www.youtube.com/watch?v=xy_7SzwsIjM&t=961s) Because universal correctness judges do not exist, automated red teaming relies on consistency probing across paraphrased inputs to identify semantic divergences where at least one response is guaranteed incorrect. - [18:46](https://www.youtube.com/watch?v=xy_7SzwsIjM&t=1126s) The continuous improvement flywheel converts failure traces from human review or online judges into permanent regression evals before deploying updates to model weights, harness code, or context memory. - [19:28](https://www.youtube.com/watch?v=xy_7SzwsIjM&t=1168s) The industry's ultimate North Star vision is an autonomous loop where an agent observes its own failures, creates its own eval data points, and generates verified fixes without human intervention.

## Detailed walkthrough ### [00:00](https://www.youtube.com/watch?v=xy_7SzwsIjM&t=0s) Defining an Agent: Model, Harness, and Context

Harrison Chase opens the session by addressing what he describes as a guiding vision for LangChain and the broader AI industry: building agents that improve over time. While acknowledging that achieving this in a completely automated fashion remains a long-term goal rather than an immediate reality, he stresses that you cannot improve a system without first being precise about its architecture. As Harrison puts it, “Because you can't improve it if you don't define what it is.” To establish that baseline, he breaks an agent down into three fundamental components:

```mermaid flowchart TD accTitle: The Three Core Components of an Agent accDescr: Diagram showing how Context feeds into the Harness, which wraps and directs the Model.

subgraph AgentStructure["Core Agent Anatomy"] C["Context<br/>(Markdown, Files, Knowledge, Tools)"] H["Harness<br/>(Control Loop, Sandbox, Execution, Memory)"] M["Model<br/>(Reasoning Engine / Workhorse LLM)"] end

C -->|"Informs and bounds"| H H -->|"Manages prompts and actions"| M M -->|"Outputs decisions and tool calls"| H ```

First is the model, the core powerhouse and reasoning engine of the agent. Builders must choose an appropriate workhorse model based on performance traits like speed, reasoning depth, or context window length, while reserving smaller peripheral models for secondary tasks like guardrails and evaluations. Second is the harness, which encompasses the surrounding software system—such as Deep Agents, Claude Code, or Codex—that makes the model practically useful. While the model handles reasoning and decision-making, the harness governs control: it decides what the model sees, executes tool actions, observes and verifies outcomes, and persists state to disk or databases. LangChain's own Deep Agents framework exemplifies this layer by packaging an execution environment (a sandbox with a file system), context management utilities (skills, memory, summarization), and mechanisms for human-in-the-loop oversight and subagent delegation. Third is context, which refers primarily to the text, markdown, files, tools, and background knowledge injected into the system.

When considering how an agent improves over time, enhancements can occur across any of these three pillars—model, harness, or context. Furthermore, the operational loop driving those improvements spans three maturity levels: - **Manual (Human-driven):** An AI engineer inspects performance data, diagnoses problems, and manually implements updates, which reflects how most engineering teams operate today. - **Human-in-the-loop:** An automated background process analyzes system behavior and proposes concrete fixes, but requires a human engineer to review and approve changes before deployment. - **Fully Automated:** The agentic system monitors itself, updates its components, and deploys fixes autonomously without human intervention—a state Harrison characterizes as the industry's “North Star or the Holy Grail,” while conceding, “I don't think we're there right now by any stretch of the imagination.”

---

[03:22](https://www.youtube.com/watch?v=xy_7SzwsIjM&t=202s) Updating the Model: Supervised Fine-Tuning vs. Reinforcement Learning

Improving the model layer requires updating model weights, either for the core agent workhorse or for peripheral classifiers, guardrails, and evaluation models. Because adjusting model weights is computationally demanding, capital-intensive, and time-consuming, Harrison notes that model updates are almost always human-initiated rather than run continuously in the background on a whim. In practice, teams leverage two distinct methodologies: supervised fine-tuning (SFT) and reinforcement learning (RL).

Supervised fine-tuning works well for narrower, simpler tasks such as extraction, classification, or domain-specific summarization. In an SFT workflow, developers pull down execution traces, filter for high-performing trajectories where the agent succeeded, format the data, and train the model on those ideal behaviors. To streamline this path, LangChain launched LangSmith Fine-Tuning, centered around a dedicated CLI tool called `smithtune`. The CLI pulls logged trajectories directly from LangSmith, filters and processes them into standardized datasets, and hands them off to managed model provider APIs such as Fireworks or Baseten to produce an optimized fine-tuned checkpoint.

For complex, dynamic tasks—particularly training the primary decision-making model running the agent—some teams bypass SFT and reach for reinforcement learning. Harrison points out that RL has a significantly higher ceiling than SFT, but comes with steep technical hurdles:

> “RL is a lot more complicated than supervised fine-tuning. There's no managed offering for RL that's really that great.”

Unlike SFT, which simply fits on static logged trajectories, RL requires defining environment tasks and executing live rollouts. The agent attempts the task, is scored in real time against an objective function, backpropagates rewards to update model weights, and repeatedly runs subsequent rollouts to climb the policy gradient.

---

[05:52](https://www.youtube.com/watch?v=xy_7SzwsIjM&t=352s) Updating the Harness: How LangSmith Engine Automates Agent Refactoring

The harness consists of application code, control flow logic, prompts, and tool interfaces. Historically, updating the harness follows a straightforward manual loop: an AI engineer reviews production traces, diagnoses failure patterns, and edits the codebase or system prompts accordingly. Because altering harness logic or core instructions impacts all users universally, teams generally require human oversight before any change ships to production.

To move this process from manual maintenance toward automated human-in-the-loop refactoring, LangChain developed LangSmith Engine. Engine is an autonomous background agent designed to audit agent systems: it monitors traces, clusters recurring runtime failures into discrete issue groups, and formulates actionable remediations. When Engine identifies a fix, it generates code patches, opens a pull request, and adds regression tests and evaluation data points to verify the correction.

Structurally, LangSmith Engine is itself built as a Deep Agent operating inside a secure sandbox. It is equipped with the LangSmith CLI, specialized prompts, custom domain skills, and task-specific subagents that allow it to browse traces, evaluate regressions, and prepare pull requests for developer review.

---

[07:13](https://www.youtube.com/watch?v=xy_7SzwsIjM&t=433s) Updating Context and Memory: Hex, Profound, and OpenWiki

Context is the easiest and most flexible layer of an agent to update because it consists primarily of plain text, structured markdown, and retrieved documentation. Consequently, context updates do not have to happen strictly at the global agent level; they can evolve dynamically across individual users, projects, or entire organizations, forming persistent layers of memory.

Harrison highlights Hex and Profound as real-world products executing this strategy effectively: - **Hex:** In their data analytics platform, Hex maintains a unified context layer as a core value proposition. To keep this context accurate, Hex runs an automated Review Agent in the background that analyzes user interactions, flags missing semantic definitions or context gaps in datasets, and proposes context updates to improve future queries. - **Profound:** Operating in marketing AI, Profound synthesizes interaction data on multiple planes. At the organizational level, background jobs compile data into persistent "Brand Records," while at the individual level, the system curates "Personal Records" derived from direct conversations between users and their AI Marketer.

Within LangChain's own ecosystem, the team explores context evolution through OpenWiki, an open-source project designed to provide coding agents with a living, self-updating knowledge base. OpenWiki initializes its knowledge base by indexing a software repository's codebase. From there, integrations with coding agents like Claude Code or Codex monitor operational traces to identify conceptual areas where the coding agent repeatedly gets stuck or fails, feeding those insights back into the wiki so the agent has sharper context on future tasks.

Harrison concludes by highlighting the common denominator linking all three improvement loops—model, harness, and context:

> “We think there's two things that are common, traces and evals. And we think these are the core foundation of this improvement loop, and it's where we spend a lot of time.”

[09:23](https://www.youtube.com/watch?v=xy_7SzwsIjM&t=563s) The Hierarchy of Traces: Runs, Traces, Threads, and Trajectories

Harrison Chase acknowledges that the memory improvement loop varies dramatically depending on the specific vertical or use case. The most difficult challenge in that cycle is prompting-heavy: determining what information to record, what to discard, and how to format and store it. Because domain-specific prompting must often be handled by application builders, LangChain focuses its efforts on the shared underlying foundation: traces and evals.

```mermaid flowchart TD accTitle: Structural Hierarchy of Agent Observability Data accDescr: Diagram showing individual runs aggregating into a trace, traces forming a thread, and trajectories providing a standardized view. subgraph Execution["Execution Granularity"] R["Run<br>(Single model or tool invocation)"] T["Trace<br>(Sequence of runs in one task)"] TH["Thread<br>(Multi-turn conversation session)"] end subgraph Analysis["Standardized Observability"] TJ["Trajectory<br>(Standardized view over a thread)"] end R -->|Aggregates into| T T -->|Combines into| TH TH -->|Viewed as| TJ ```

To establish clarity around observability, Chase delineates the core terminology LangChain uses to dissect agent execution: - **Run**: A single discrete step, such as an individual model call or a single tool invocation. - **Trace**: A chronological sequence of runs generated when an agent is triggered to complete an action. - **Thread**: The overarching conversational container formed when a user exchanges multiple messages back and forth with an agent over time. - **Trajectory**: A standardized view mapped over a thread, capturing every single event in a uniform schema.

Regardless of whether a developer plans to fine-tune underlying model weights or leverage a coding agent to debug harness logic, having high-fidelity execution records is non-negotiable. As Chase emphasizes: “The important thing is that for all of these, you have the exact inputs, the exact outputs, and the exact sequence of events.”

---

[10:44](https://www.youtube.com/watch?v=xy_7SzwsIjM&t=644s) Semantic Failures, SmithDB, and AI Search

Traces represent an entirely new class of application data because LLM agents fail in ways that traditional software logs fail to capture. Instead of throwing standard system crashes or syntax exceptions, agents succumb to semantic failures and reasoning breakdowns that exist solely within the conversational context. Because standard logging databases are poorly equipped to query or triage these context-bound failures, LangChain created SmithDB, a database custom-built for agent observability—specifically engineered to ingest, store, query, and visualize agent interactions.

To illustrate how querying must evolve for semantic data, Chase highlights the newly launched AI Search (AI filter) feature. Rather than relying on rigid keyword matches or SQL filters, developers supply free-form semantic questions describing the failure or behavior they want to inspect. Under the hood, the system evaluates and scores each trajectory individually to determine whether it matches the criteria, allowing teams to isolate subtle reasoning bugs across massive datasets.

---

[12:02](https://www.youtube.com/watch?v=xy_7SzwsIjM&t=722s) Evals as the New PRD: Defining What “Good” Looks Like

Once developers gain visibility into traces, the next critical requirement is evaluation. Chase adopts an increasingly popular industry maxim: “First, they're the new PRD.” Whereas conventional software relies on static Product Requirements Documents to dictate behavior, AI systems encounter an endless long tail of ambiguous edge cases. Consequently, executable eval suites become the most accurate, living specification of what an agent should and should not do.

Chase characterizes evals as the essential guardrails that keep an agent in the safe zone throughout continuous iteration loops. However, building reliable evaluations remains exceptionally difficult: - **No silver bullets**: While platforms provide the infrastructure to run evaluations and visualize metrics, the datasets and evaluators must remain strictly domain-specific. - **Realistic inputs**: Anticipating the wild variety of user queries is notoriously hard; real users routinely veer completely off-topic or ask tangential questions that designers never anticipated during prompt design. - **Subject-matter expertise**: Specifying boundaries and desired agent behaviors requires close guidance from domain experts rather than purely engineering assumptions.

---

[13:38](https://www.youtube.com/watch?v=xy_7SzwsIjM&t=818s) Three Methods for Grading Agent Output

When defining what constitutes a good response, Chase breaks down the three primary evaluation approaches teams use in production:

| Evaluation Strategy | Definition & Scope | Primary Challenges & Use Cases | |:--- |:--- |:--- | | **Exact Match** | Enforcing that the model outputs a specific string, number, or verbatim phrase. | Fragile and unworkable for complex, multi-step agent interactions with variable wording. | | **Criteria-Based (Semantic)** | Verifying positive or negative criteria (e.g., must mention topic X, must never say Y). | Evaluates the substance of the response without over-constraining the phrasing. | | **Environment State / Tool Sequence** | Inspecting the end-state mutation of the database/environment or the exact sequence of tool calls. | Essential for agents performing real work (e.g., writing to databases, altering files) where final text output matters less than correct side effects. |

Chase notes that evaluating external mutations is often superior to judging text: “And so you don't want to judge its final answer. You want to judge how it interacts with its environment, or more specifically, what the end state of the environment is.”

---

[14:57](https://www.youtube.com/watch?v=xy_7SzwsIjM&t=897s) The Simulation Environment Dilemma

Executing dynamic agent evaluations creates a major infrastructure dilemma: agents require an environment to act upon, but running large test suites against live production backends is unacceptable—especially when agents execute write operations or mutate state. Teams are forced to architect dedicated simulation environments, using mock databases or ephemeral staging systems where agents can trigger tools freely.

This challenge compounds when agents interact across external third-party software ecosystems. While mocking an internally developed database is relatively straightforward, orchestrating reliable sandboxes for third-party platforms like Salesforce presents a significant operational hurdle because developers do not control those systems. Developing robust, isolated simulation environments remains one of the largest ongoing investments for teams shipping real-world agents.

---

[16:01](https://www.youtube.com/watch?v=xy_7SzwsIjM&t=961s) Proactive Discovery and Red Teaming Without Universal Judges

Teams traditionally uncover agent failures in two reactive ways: manually paying domain experts to stress-test the system, or scouring production traces after errors occur. The former is slow and prohibitively expensive; the latter carries severe downsides when rare failures damage user trust before teams even detect them. To address this gap, LangChain introduced automated red teaming to uncover vulnerabilities proactively before deployment.

Automated red teaming faces a core philosophical dilemma: judging correctness without a universal source of truth. As Chase observes: “The challenge here is how do we know if it's correct or not? So what I mean by this is in order to judge whether an answer is correct or not is actually not trivial at all. That requires a ton of domain knowledge. If you could do that reliably as part of the agent loop, then you just do it as part of the agent loop and run it and then judge it.”

Because universal correctness judges do not exist, LangChain employs specialized algorithmic probing strategies: - **Consistency Probing Across Paraphrased Inputs**: The red teaming engine sends two semantically identical queries phrased in slightly different ways. If the agent generates semantically divergent answers, the engine flags the inconsistency, knowing with certainty that at least one response must be incorrect. - **Targeted Category Reports**: In addition to factual accuracy, the system probes along distinct vectors such as security exploits and fairness, generating comprehensive red teaming reports on demand. - **Simulation Coupling**: Because probing requires running the agent end-to-end against live tools, proactive red teaming directly integrates with the simulated environments to prevent unwanted side effects.

[18:46](https://www.youtube.com/watch?v=xy_7SzwsIjM&t=1126s) The Improvement Flywheel: From Feedback to Eval Data Points

Harrison Chase highlights that building realistic simulation environments is an active, demanding area of study, noting that the LangChain labs team in particular is currently dedicating substantial research effort to solving it. Stepping back to the broader picture, he lays out the abstract lifecycle required to make any agent continuously get better at its job. As Harrison explains: “So at a high level, this is like the really high level abstract kind of like flywheel that you want to set up in order to improve your agent over time.”

```mermaid flowchart TD accTitle: The Agent Continuous Improvement Flywheel accDescr: Cycle showing how running an agent leads to feedback, trace inspection, eval creation, and targeted updates across model, harness, or context.

subgraph Execution["1. Execution & Evaluation"] Run["Run Agent"] --> Feed["Get Feedback<br/>(Human or Online Judge)"] Feed --> Trace["Inspect Trace Data"] end

subgraph Iteration["2. Eval & Update Loop"] Trace --> AddEval["Create Eval Data Point<br/>(Prevent Regressions)"] AddEval --> Update["Suggest Fixes / Updates"] end

Update -->|"Update Model Weights"| Run Update -->|"Update Harness Code"| Run Update -->|"Update Context / Memory"| Run ```

The cycle begins by running the agent in production or testing and collecting direct feedback on its performance, whether that signal originates from an explicit human review or an automated online judge. Once a failure or suboptimal response is flagged, developers must dig into the execution trace to pinpoint precisely what went wrong internally. That captured interaction cannot simply be fixed in isolation; it must be converted into a permanent evaluation benchmark to ensure future iterations actually fix the flaw without triggering regressions on previously solved scenarios. Finally, improvements are proposed and deployed across one of the agent's core layers: updating the model weights, modifying the harness code, or refining the context and memory.

- **Simulation research:** The LangChain labs team is actively researching how to build robust, synthetic simulation environments for evaluation. - **Feedback channels:** Quality signals can come from human review or automated online judges observing live runs. - **Regression prevention:** Converting problematic traces into concrete eval benchmarks ensures new agent updates do not break existing capabilities. - **Three levers for updates:** Fixes flow back into model weights (via fine-tuning/RL), harness code (application logic and tool interfaces), or context (memory stores and dynamic system knowledge).

---

[19:28](https://www.youtube.com/watch?v=xy_7SzwsIjM&t=1168s) The Autonomous Improvement Loop: A North Star Vision

Looking toward the frontier of agent engineering, Harrison presents what he considers the ultimate evolution of this flywheel: shifting human operators out of the maintenance loop entirely so the system can refine itself. As Harrison puts it: “The really kind of future version of this is where all of this is done by the agent itself.” In this self-healing setup, the agent actively detects when it makes an error, automatically drafts an evaluation case capturing that failure mode, and generates its own targeted remedy across its context, harness code, or underlying weights.

Harrison notes that LangChain has already begun exploring and implementing pieces of this autonomous workflow inside LangSmith Engine, though he cautions that fully realizing it remains an ambitious long-term goal. As Harrison emphasizes: “This is very much a North Star vision.” The end-to-end loop sounds clean and straightforward at an abstract level, but deploying it reliably introduces immense practical nuance around verification, safety, and stability. He invites teams grappling with these exact architecture problems to collaborate with LangChain, reiterating that solving the intricate details of agent self-improvement is one of the engineering challenges they are most excited about pursuing before closing out the talk.

- **Self-directed evaluation:** An agent monitors its own output, identifies failures, and automatically creates eval data points without manual intervention. - **Autonomous remediation:** The system formulates its own bug fixes or configuration patches across context memory, code harnesses, or fine-tuned model checkpoints. - **LangSmith Engine roadmap:** Autonomous agent repair forms the foundational direction and North Star vision for LangSmith Engine development.

## Notable quotes > [03:22](https://www.youtube.com/watch?v=xy_7SzwsIjM&t=202s) RL is a lot more complicated than supervised fine-tuning. There's no managed offering for RL that's really that great. — Harrison Chase

## Mentioned Harrison Chase, LangChain, LangSmith, LangSmith Engine, LangSmith Fine-Tuning, smithtune, SmithDB, OpenWiki, Deep Agents, Claude Code, Codex, Fireworks, Baseten, Hex, Profound, Salesforce

## Video description Harrison Chase, Co-Founder and CEO of LangChain, splits an agent into three parts, the model, the harness, and the context, and explains how each one can improve over time: the model's weights through supervised fine-tuning with LangSmith Fine-Tuning and its smithtune CLI, or through RL for more complex tasks; the harness through LangSmith Engine, a deep agent that runs over traces, clusters issues, and proposes fixes with a human in the loop; and the context through memory, where he points to Hex and Profound as companies he thinks do this well and to LangChain's open source OpenWiki project. He argues that traces and evals are the common foundation of all three loops, which is why LangChain built SmithDB and the new LangSmith AI Search for trace data, then covers what makes evals hard (realistic inputs, defining what good looks like, simulation environments) and how red teaming in LangSmith Engine looks for agent mistakes proactively. He closes with what he calls a North Star vision: an agent that notices its own mistakes, creates its own eval data points, and suggests its own fixes.

This session was captured at Interrupt London, the agent conference by LangChain.

Chapters: 0:00 What is an agent, and how do you improve it? 3:22 When should you fine-tune the model behind your agent? 5:52 How LangSmith Engine proposes fixes to your harness 7:13 Why context is the easiest part of an agent to improve 8:42 How OpenWiki gives coding agents a brain that updates 9:11 Why traces and evals are the foundation of improvement 10:44 Why LangChain built SmithDB and AI Search for traces 12:02 Why evals are the new PRD 12:50 Why is building evals so hard? 14:57 How to run evals without hitting production systems 16:01 How red teaming finds agent mistakes ahead of time 16:54 How do you judge correctness without a universal judge? 18:54 The flywheel that improves your agent over time 19:28 What if the agent ran its own improvement loop?

Resources: → LangSmith Engine: https://www.langchain.com/langsmith/engine → LangSmith Fine-Tuning (smithtune): https://docs.langchain.com/langsmith/smithtune → OpenWiki on GitHub: https://github.com/langchain-ai/openwiki → LangSmith: https://www.langchain.com/langsmith → Interrupt, the agent conference by LangChain: https://interrupt.langchain.com

[Source](https://www.youtube.com/watch?v=xy_7SzwsIjM)

Reading summary

LangChain CEO Harrison Chase argues that engineering agents that improve over time requires decoupling systems into model, harness, and context layers, all anchored by continuous trace observability and rigorous evaluation suites. The Tripartite Agent Architecture (0:00) Chase defines an agent through three constituent components: the model, the harness, and the context. The model acts as the core reasoning engine, while the harness encompasses the surrounding application code, execution sandbox, and control loops that govern what the model sees and execute its tool calls. The context layer supplies dynamic knowledge, markdown files, and domain data. System improvements can occur across any of these layers and currently transition through three maturity tiers: manual engineer intervention, automated human-in-the-loop proposals, and fully autonomous self-improvement. Layer-Specific Levers for Iterative Upgrades (3:22) Updating an agent requires distinct techniques depending on the targeted architectural layer. For the model layer, teams utilize supervised fine-tuning (SFT) for narrow tasks via tools like `smithtune`, or turn to reinforcement learning (RL) for complex reasoning despite the lack of managed infrastructure and the need for live rollouts. At the harness level, code changes affect all users universally, prompting LangChain to build LangSmith Engine—a sandboxed agent that clusters runtime failures, drafts code patches, and opens pull requests with regression tests. Context remains the most adaptable layer, allowing dynamic memory updates at individual or organizational levels, as demonstrated by Hex's automated Review Agent and Profound's persistent Brand Records. Trace Observability for Semantic Failures (9:23) Because agents suffer from conversational and reasoning breakdowns rather than conventional code syntax crashes, traditional software logs fail to isolate agent bugs. LangChain structures observability data into a strict hierarchy: individual runs aggregate into traces, multi-turn conversations form threads, and trajectories provide a standardized view across entire threads. To store and triage these semantic interactions, LangChain developed SmithDB alongside semantic search filters, enabling developers to query reasoning failures using natural language questions rather than rigid SQL filters. Evaluation Strategies and Simulation Sandboxes (12:02) Chase characterizes evaluation suites as the new Product Requirements Document (PRD), establishing living specifications for agent behavior in production. Because exact-match grading fails against variable phrasing, engineering teams increasingly grade agents via semantic criteria checks or by evaluating external environment mutations and tool call sequences. However, verifying environment mutations introduces infrastructure bottlenecks: testing agent write operations requires dedicated simulation sandboxes, which become difficult to manage when agents interact with external third-party software like Salesforce. To uncover flaws without universal correctness judges, LangChain uses automated red teaming techniques, such as probing for inconsistencies across paraphrased inputs. Chase frames autonomous agent self-healing—where systems detect operational failures, synthesize evaluation data points, and deploy remediations to their own weights, harnesses, or context without human oversight—as an ambitious long-term industry North Star that currently faces steep practical hurdles in safety, verification, and environment simulation.

Source quotes

[3:22] Harrison Chase: RL is a lot more complicated than supervised fine-tuning. There's no managed offering for RL that's really that great.

Original transcript

[0:00] Cool. [0:07] For this session, I want to talk about something that we think a lot about [0:11] at LangChain, and [0:12] that is definitely part of the future of where we think the industry and where [0:16] we are going, [0:17] which is building agents that improve over time. [0:21] And so this is a North Star that, while doing this in a fully automated way, may not be anytime soon. [0:28] This is a North Star that guides a lot of how we think about things at LangChain. [0:32] And so as I talk about this, I want to set the stage with talking about [0:35] just like what is an agent [0:37] and how do we think about improving it? [0:39] Because you can't improve it if you don't define what it is. [0:42] And so we think there are three parts to building an agent. [0:45] First is a model. [0:46] And so this is choosing the powerhouse of the agent. [0:51] And there's a bunch of different models that you can choose from. [0:54] And some are good at different things. [0:55] Some have long context windows. [0:56] Some are really fast. [0:58] But choosing one to be the core workhorse of an agent. [1:01] And then the other thing I'd say is there's other models at the periphery. [1:04] There's other models for guardrails and evaluations and things like that that [1:08] might be part of the harness, but not the core workload. [1:12] But anyways, at the core of an agent is a model. [1:14] Then there is the harness. [1:16] So you could use Deep Agents, Claude Code, Codex. [1:19] This is the harness of the model. [1:20] What is a harness? [1:21] This is the system around the model that makes it really useful. [1:24] So the model does all the reasoning and decision making. [1:27] But the harness is responsible for deciding what the model sees, for actually [1:31] taking the actions of what the model decides to do, [1:35] for observing and verifying results, for persisting things onto disk or into databases, [1:41] and is really related to all the control around what happens in this loop. [1:46] We've built Deep Agents as an example harness for what this can look like. [1:50] It encodes a bunch of best practices for what these harnesses should do. [1:55] So it has an execution environment, which is where it runs. [1:58] So this includes a file system and sandbox. [2:00] It does a bunch of off-the-shelf context management. [2:03] So it deals with skills, memory, summarization. [2:06] And then it has human in the loop as well as subagents for different ways for [2:10] interacting with humans and other agents, respectively. [2:13] And then the third part of an agent is context. [2:17] And so this is markdown. [2:19] This can be files. [2:21] This can be knowledge that's broad. [2:22] It can be tools that it uses. [2:26] But when I say context here, I mostly [2:28] refer to kind of like text and other sources of knowledge [2:31] that is brought. [2:32] And these three things, we think, make up what an agent is. [2:35] It's got the model, you wrap it in a harness, [2:37] and you connect it to context. [2:39] And so when we talk about agents improving over time, [2:43] they can improve across any of those dimensions. [2:45] So you can improve the model, you can improve the harness, [2:47] or you can improve the context. [2:50] And they can improve in a bunch of different ways as well. [2:52] So they can improve manually. [2:54] So they can be human driven. [2:55] So you can have a human AI engineer go in and kind of [2:57] like drive the cycle. [2:59] And that's a lot of what AI engineers do today. [3:03] They can be human in the loop. [3:04] You can have some background process that is improving [3:07] this agent over time. [3:08] But you have a human approving things. [3:11] Or they can be fully automated. [3:12] And again, this is maybe the North Star or the Holy Grail. [3:15] And I don't think we're there right now by any stretch [3:18] of the imagination. [3:18] But this is where we want to work towards. [3:20] And so I want to talk about these three different dimensions. [3:22] I want to talk about improving the model first. [3:25] So models are generally improved by updating the weights of the model. [3:30] Agents use different models, so you want to choose which ones to update. [3:33] And so maybe you update the weights of the core model that's driving the agent, [3:37] or maybe you update the weights of a guardrail or an eval check or something like that. [3:43] This is almost always human-initiated. [3:45] So the way that you update the models, and I'll touch on that in a second, [3:48] but the way that you update the models [3:49] is generally pretty time and capital intensive. [3:52] And so you almost always want to have a human going in [3:54] and preparing and deciding to do this. [3:56] It would be pretty costly to just do this continuously [3:58] in the background or on a whim. [4:01] And there's two different methods [4:02] that we see for doing this. [4:04] One is called supervised fine-tuning, [4:06] and the other is RL, or reinforcement learning. [4:08] Talking very briefly about supervised fine-tuning, [4:11] at a high level, you take a bunch of traces. [4:14] You filter to ones that you want the agent to learn on. [4:16] And so these are generally ones where [4:18] it does well. [4:20] And then you basically train the model on that. [4:23] And this is a really good option for simpler tasks. [4:26] So we see people doing this a lot for classification or extraction or [4:29] summarization tasks, where [4:31] it's pretty narrowly defined. [4:34] And so one of the things that we launched recently is LangSmith Fine-Tuning [4:38] in the form of smithtune. [4:39] smithtune is a CLI. [4:41] It interacts with LangSmith. [4:43] It pulls down trajectories. [4:44] It filters them. [4:46] It processes them into the right data format. [4:48] We then hand off to Fireworks or Baseten [4:51] or another one of these model providers that have, [4:54] they actually have a bunch of off-the-shelf supervised [4:57] fine-tuning, basically, APIs where you can just pass in data and get back a model. [5:01] And so we plug into that and then get back an optimized model. [5:05] And so we see people doing this, again, for kind of like simpler tasks. [5:09] For more complex tasks, [5:10] specifically like that core model that's driving the agent, [5:13] we see a few people reaching for RL. [5:15] So RL is a lot more complicated than supervised fine-tuning. [5:18] There's no managed offering for RL that's really that great. [5:23] So there are a bunch of managed offerings for SFT. [5:25] There's none for RL. [5:27] You can generally do more things with RL. [5:30] And the way that RL works is a little bit different. [5:33] Rather than just giving it trajectories, [5:34] you give it a bunch of tasks. [5:36] And you do rollouts over those tasks. [5:39] You basically run the agent over it. [5:40] You then score it in real time, backprop it, [5:43] update the agent or the model that's part of the agent, [5:46] and then do more rollouts. [5:47] And so a little bit different than SFT, [5:49] a lot more complicated, but higher ceiling. [5:52] The next thing you can update is the harness. [5:54] The harness, how do you update it? [5:56] Well, it itself is basically code and logic there. [5:59] And so you'd update the code, or you'd update the prompts, [6:01] or you'd update the tools. [6:03] The general process for this at a high level is pretty simple. [6:06] You look at a bunch of traces. [6:08] You identify issues. [6:09] You then make changes based on those traces. [6:13] You'll probably want a human in the loop for this, [6:15] because this will be changing core code of the agent, [6:18] or core instructions of the agent that will usually [6:20] propagate to all users. [6:22] And so you probably want some human in the loop [6:24] to review these things. [6:28] But you can start to automate it. [6:29] And so LangSmith Engine, and Ben talked [6:30] about LangSmith Engine earlier, LangSmith Engine [6:33] is our attempt to do this. [6:35] So this is an agent running in the background. [6:38] It basically runs over traces, monitors them, [6:41] clusters them into basically different issues, [6:44] and then tries to propose fixes, put up a PR, add tests, [6:48] add eval data points, things like that. [6:52] Engine itself is nothing more than a deep agent in a sandbox [6:55] with LangSmith CLI and some specialized subagents, [6:59] some specialized skills, and some specialized prompts. [7:02] And so this general process we think [7:04] is an example of starting to automate, [7:06] but not going fully there. [7:07] Still human in the loop, very important part, [7:09] but starting to automate some of this improvement of the harness. [7:13] And then lastly, talking about improving the context, [7:15] this is the easiest to update. [7:16] This is just text. [7:17] And so it's very easy to update this. [7:18] And as a result, you can update this at different levels. [7:21] So rather than just updating this at purely the core agent [7:25] level, you can start to update this at the user or org level. [7:28] And you can start to have memory at those different dimensions [7:31] that evolves over time. [7:33] And so similar to how you'd update a harness, [7:36] the process is basically look at a bunch of traces, [7:40] identify issues, and then make changes. [7:41] And again, you're making changes to the text, [7:43] so it's a lot easier to do. [7:45] There's a few companies that I think do this well. [7:48] One I want to call out is Hex. [7:49] And so Hex is a data analytics agent. [7:51] Part of their value prop [7:53] is this unified context layer. [7:55] This unified context layer improves over time. [7:57] That's a core part of the value prop. [7:59] How do they do it? [8:00] They basically do something similar to Engine. [8:01] They run an agent in the background. [8:04] So the Review Agent analyzes interactions for you [8:06] and finds context gaps and suggests fixes. [8:09] And again, it's specifically for the agent [8:10] that they're building. [8:11] But they basically have this background process, [8:13] looking at traces, identifying things, [8:15] and then proposing fixes. [8:18] Profound is another example of a company that I think does this well. [8:21] And they actually do this at the user level as well. [8:23] So you can see that they constantly synthesize data into Brand Records. [8:27] That's maybe like a company-level thing. [8:28] But then it also collects Personal Records informed by interactions with your AI Marketer. [8:33] So as you're interacting with the agent, they have a process in the background [8:36] that will create this basically personal memory. [8:39] So these are examples of two companies that I think are doing a really good job of this. [8:42] We're experimenting with things in this vein. [8:45] We don't have anything in the platform, [8:47] but we do have an open source project called OpenWiki. [8:50] OpenWiki is basically this idea of creating this knowledge [8:53] or this brain for an agent that can be updated. [8:56] So this is... [8:57] This is built for coding. [8:59] It started... [9:00] It's populated by looking at your coding repo. [9:02] But we actually have integrations with your coding agents, [9:04] so we can look at Codex or Claude Code traces [9:07] and see what it struggles with, and then bring it back [9:09] into OpenWiki. [9:11] So what's common across all of these three things? [9:14] We think there's two things that are common, traces and evals. [9:17] And we think these are the core foundation of this improvement loop, [9:21] and it's where we spend a lot of time. [9:23] This improvement loop may look different for different use cases and different verticals. [9:27] The hardest part of the improvement loop is oftentimes choosing what to remember [9:31] and what not to remember and how to remember. [9:34] And a lot of that is in the prompting, and so that's very domain-specific. [9:37] But we think there are other ways that we can help, namely with traces and evals. [9:41] And so this is where we're putting a bunch of effort. [9:44] So traces, what are traces? [9:46] We think there's a few different aspects of traces. [9:50] So there's a few different terms that we use when talking about traces. [9:53] First is run. [9:54] This is like a single invocation of a model or a single invocation of a tool. [9:58] It's basically a single step. [9:59] You then add up a bunch of those runs and you get a trace. [10:02] And this is a sequence of those steps. You combine a bunch of those traces and [10:05] you get a thread. And so the way to think about this is if I message an agent, it [10:09] goes and does a bunch of work. There's a bunch of runs happening in the background. [10:13] That itself is one trace. [10:15] When I start having a conversation with it, I create this thread. [10:18] And then trajectory, as we talked about in the keynote, trajectories [10:21] are a view over this thread in a standardized format. [10:25] The important thing is that for all of these, [10:27] you have the exact inputs, the exact outputs, [10:29] and the exact sequence of events. [10:31] And this is really needed, [10:32] whether you're going to be fine-tuning the model [10:34] or trying to debug it with a coding agent and then update the code. [10:38] Being able to see exactly what goes in, exactly what comes out, [10:41] and the exact sequence is really important. [10:44] We think traces are a new type of data. [10:46] That's why we built SmithDB, which is a database of our own [10:51] that we built to deal with this new type of data. [10:54] We think there's new types of failures here. [10:56] The failures here are semantic failures. [10:58] They're logic failures. [11:00] There are a bunch of issues that pop up that don't show up in traditional logs. [11:04] They show up in the conversational context. [11:06] And so how you deal with those and how you triage those needs to change. [11:11] And likewise, how you interact with the data, how you query it, also needs to change. [11:17] And all of that is why we built SmithDB, which is our database that we [11:22] purpose-built for agent observability. [11:25] We think a bunch about how to structure this database to be the best database [11:30] for ingesting, [11:32] querying, storing, visualizing data. [11:36] The example of this that we launched today [11:38] is this AI filter, this AI Search, [11:40] where you type in, again, a question [11:42] that you want to filter on. [11:43] This is a semantic question. [11:45] And under the hood, we score each trajectory individually [11:49] according to whether it meets that filter or not. [11:52] And so this is an example of how we [11:53] think about building this database that [11:55] is optimized for this new type of data. [11:59] So you can see what's going on. [12:01] You can query what's going on. [12:02] But next comes evals. [12:04] So evals are really important for doing this loop [12:07] for a few reasons. [12:07] First, they're the new PRD. [12:09] And so this is a saying that I really [12:10] like, where you might have a product requirements [12:13] document describing what the product should do. [12:16] In AI, there's so many edge cases [12:19] that the best way to describe that oftentimes [12:21] is in the form of evals. [12:23] So evals are where you define what good looks like. [12:27] And so good can mean you respond in a certain way [12:30] to certain queries. It can mean you don't respond to certain queries. Defining the limits, [12:35] defining what good looks like is a really important part of evals and a really important [12:38] part of defining how you want to improve. And so this is kind of like the guardrails [12:43] that are going to keep you in the safe zone as you iterate in this improvement loop. [12:50] Building evals is really, really hard. This is one of the main things that we work with [12:53] a bunch of folks on. There's not really a silver bullet for evals out there. [12:58] Sure, we can provide infrastructure for running evals and visualizing [13:02] eval results. [13:03] But evals need to be domain-specific, both the datasets [13:05] that they run on and the evaluators that are used. [13:10] And so we still think building evals is really hard. [13:12] There's different parts of this. [13:14] So first, you need to come up with realistic inputs. [13:17] And so this can often be hard. [13:18] You might not know what people will even ask your agent. [13:21] Sometimes they'll go completely off topic. [13:22] Sometimes they'll go orthogonally or close to it. [13:26] But you may not be able to predict all these cases up front. [13:28] And so even being able to just enumerate what realistic inputs are [13:32] is not as easy as [13:33] it sounds. And this is why tracing is so important. You need to be able to define [13:37] what a good answer looks like. [13:38] There's different ways that we've seen people do this, maybe calling out [13:42] kind of like three off the top of my head. One, you can define what the expected output is, [13:47] a certain number that it should mention, an exact kind of like phrase that it should use. [13:53] This is kind of called an exact match. [13:55] This gets complicated for more complex agents, [13:58] because it's usually hard to do that. [14:00] And so what we see people do is then defining criteria. [14:03] Basically, this agent should have part of the answer [14:07] that mentions this. [14:08] This answer should not say this. [14:09] It could be positive or negative criteria. [14:12] These criteria could act on either the final answer [14:15] itself or the environment that the agent's running in. [14:19] So oftentimes, these agents are interacting [14:21] with the environment. [14:22] They might be changing databases in certain ways. [14:24] They might be manipulating files in certain ways. [14:26] And so you don't want to judge its final answer. [14:28] You want to judge how it interacts with its environment, [14:31] or more specifically, what the end state of the environment is. [14:34] And the third thing that the criteria can judge [14:36] is actually the exact sequence of tool calls as well. [14:38] And so depending on your use case, [14:40] you may just care that it ends with the right database state. [14:44] Or you might care about the sequence of tool calls. [14:47] And so you can write criteria for either of those. [14:49] But defining what good looks like, this is oftentimes [14:51] one of the hardest things to do and one of the things [14:53] that's best done by domain experts [14:55] or subject matter experts. [14:57] And then the last thing that I want to call out [14:59] is the simulation environment. [15:02] So when you run evals, you need an environment to run them in. [15:05] If your agent naturally interacts with production systems, [15:09] you probably don't want it hitting production systems as you run a bunch of evals, [15:13] especially if those are doing write operations or actually changing the database. [15:18] And so one of the things that we see people doing [15:21] in order to be able to run these evals [15:23] is coming up with these simulation environments [15:25] where they can basically interact freely with either [15:27] like a mock database or a dev environment that they spin up, [15:31] basically run these end-to-end evals that actually do interact [15:35] again with this simulated environment and run that way. [15:40] This can be tricky. [15:41] Your agent may interact with a bunch of different services. [15:43] You may or may not have kind of like dev or staging things [15:46] for those. [15:47] If the agent's interacting with a system that you create, [15:50] that might be easier. [15:50] If it's interacting with Salesforce, what do you do there? [15:54] That's not a system that you kind of easily own. [15:57] And so we also spend a lot of time [15:59] thinking about what these environments should look like. [16:01] And then on top of that, we also spend a lot of time [16:03] thinking about how do you find agent mistakes. [16:06] So again, there's different ways to do this. [16:09] You can look through how people actually use your agent [16:11] and look for places where it fails. [16:13] You can have domain experts think about how they would use [16:16] the agent, try it out, and then see when it messes up. [16:20] The domain expert one is kind of manual and costly. [16:23] The looking at production traces one, [16:26] this basically relies on the instances having already [16:29] happened. [16:30] And sometimes something might happen so infrequently [16:33] that by the time you catch it, it's too late. [16:36] And you still might want to quantify that as an error, [16:38] but it doesn't do you as good as if you'd known about it [16:42] ahead of time. [16:42] And that's really the idea behind red teaming, [16:44] which we launched recently. [16:45] The idea behind red teaming is to help [16:47] find these issues proactively. [16:49] And so there's a bunch of different ways [16:51] that we think about doing this. [16:53] There's a bunch of different dimensions that we look at. [16:54] So one is just the overall correctness of the response. [16:58] Can we find issues where the agent is just not [17:02] correct in how it responds? [17:04] The challenge here is how do we know if it's correct or not? [17:08] So what I mean by this is in order to judge whether an answer [17:13] is correct or not is actually not trivial at all. [17:15] That requires a ton of domain knowledge. [17:17] If you could do that reliably as part of the agent loop, [17:20] then you just do it as part of the agent loop and run it [17:22] and then judge it. [17:23] And then if it says it's incorrect, [17:25] just keep on running it. [17:25] And so having this universal judge for correctness [17:28] is one of the things that makes this hard to do. [17:33] We haven't solved that. [17:34] We don't have a universal judge for correctness. [17:36] But what we do is we have a few different algorithms [17:39] that probe the agent in clever ways [17:42] so that we do know what the correct answer should be. [17:45] So for example, one of the algorithms that we have [17:48] is we basically send two very similar queries, [17:51] at least semantically similar queries, to the agent. [17:55] And if it responds in semantically different ways, [17:58] we know one of them is wrong. [18:00] And then we can investigate which one is more likely to be [18:02] right or more likely to be wrong. [18:03] And so by doing that, we can ground basically the answer in [18:07] what we know the right answer to be. [18:09] It's not something that you could do in the loop, because [18:10] you're probably not going to do that as the agent's running. [18:13] That would be too crazy and too expensive. [18:14] But it provides this kind of reference point. [18:17] There's other types of issues. [18:19] There's types of issues that are related to security, that [18:21] are related to kind of like fairness. [18:22] In the long run, what we want to have is basically a menu of these different [18:28] issues that you can check for, and that they can generate these basically reports, these [18:34] red teaming reports on the fly. One of the things that we're also marrying with this, [18:38] I mentioned kind of like one of the issues with running evals is this simulated [18:42] environment. [18:43] You'll need that for this because this does run the agent. And so we're also doing a lot [18:46] of work on basically creating these simulated environments. [18:51] And our labs team in particular [18:52] is doing a lot of research there. [18:54] So at a high level, this is like the really high level abstract [18:57] kind of like flywheel that you want to set up [19:00] in order to improve your agent over time. [19:02] You want to run your agent. [19:03] You want to get feedback on how it does, [19:06] either by a human or by an online judge. [19:09] You then want to go in. [19:10] You want to see what happened inside of that trace [19:12] to get that feedback. [19:13] You want to add that as an eval data point [19:16] so that as you try to improve, you can make sure [19:18] that you're actually improving on that [19:19] and not regressing against previous ones. [19:22] And then you want to suggest changes to the agent, [19:24] either update to the model weight, update to the harness code, [19:26] or update to the context. [19:28] The really kind of future version of this [19:30] is where all of this is done by the agent itself. [19:34] And so the agent notices that it made a mistake. [19:37] The agent goes, creates an eval data point of its own, [19:39] suggests its own changes, [19:41] and it basically generates its own fixes, [19:44] again, either to the context or the harness code or the model weights. [19:47] We're starting to think about what this looks like in Engine. [19:49] This is very much a North Star vision. [19:51] There's a ton of nuance in this loop as well. [19:54] Again, this loop is very high level, so there's a ton of nuance there. [19:58] We spend a ton of time thinking about the nuance. [20:01] If you're thinking about how to do this and want to dive deep into the nuance with us, [20:04] please talk to us about this loop. [20:06] We get really excited by this. [20:09] So that's all I got for this talk. [20:11] So thank you guys for the time.