LangChain
Executable evaluation suites must replace traditional product requirement documents because language models fail through semantic breakdowns rather than syntax crashes, requiring teams to turn execution traces into regression benchmarks that update model weights, code harnesses, and context memory.
AI-assisted summary of How to Build Agents That Improve Over Time: Model, Harness, Context | Interrupt London — original source. Check the original for full context.
Source published:
Key ideas
- Harrison Chase deconstructs an agent into three core components—model, harness, and context—arguing that systematic self-improvement can occur across each layer independently. (0:00)
- LangChain introduced dedicated infrastructure including the smithtune CLI for model fine-tuning and LangSmith Engine, an autonomous Deep Agent that clusters runtime failures and generates pull requests. (3:22)
- To handle semantic failures that bypass standard logs, LangChain created SmithDB and AI Search to query agent execution trajectories using free-form semantic questions. (10:44)
- Chase positions executable eval suites as the new PRD for AI applications, emphasizing that grading external environment mutations is often far more reliable than assessing free-form text output. (12:02)
- LangChain outlines an autonomous 'North Star' improvement flywheel where agents flag their own errors, generate regression test cases, and propose self-healing code patches. (19:28)
Detailed notes
## Background This talk was delivered by Harrison Chase, Co-Founder and CEO of LangChain, and published on YouTube by LangChain on October 27, 2026, from their Interrupt London conference. According to the page description, Chase explains how agents can improve across three architectural pillars—model, harness, and context—using traces and evaluations as a common foundation, while exploring tools like LangSmith Engine, smithtune, and OpenWiki.
## TL;DR Chase deconstructs agents into three core components: the model (reasoning engine), the harness (execution code, sandbox, and control flow), and the context (dynamic knowledge and memory). He explains how each layer can be upgraded over time, ranging from fine-tuning and reinforcement learning for model weights, to automated pull requests and refactoring via LangSmith Engine for the harness, to dynamic knowledge bases like OpenWiki for context. Throughout, he argues that high-fidelity traces and rigorous evaluations serve as the shared foundation for diagnosing semantic failures, running red teaming probes, and preventing regressions. Finally, he outlines an improvement flywheel that transitions from human-driven debugging to a North Star vision where agents autonomously detect their own errors, generate eval cases, and patch their own code and context.
## Atmosphere & style The presentation carries the pragmatic, grounded energy of a technical conference keynote tailored directly for AI builders and system architects. Chase speaks with clear, rapid-fire engineering conviction, deliberately tempering industry hype by acknowledging that fully automated agent self-improvement remains an aspirational North Star rather than an immediate reality. Watching the talk feels like peering into an active developer workshop—analytical and structured, yet energized by real-world case studies, architectural diagrams, and the tangible frontier of agent observability.
## Bulletin - [00:00](https://www.youtube.com/watch?v=xy_7SzwsIjM&t=0s) Harrison Chase deconstructs an agent into three core components—model, harness, and context—arguing that systematic self-improvement can occur across each layer independently. - [03:22](https://www.youtube.com/watch?v=xy_7SzwsIjM&t=202s) LangChain introduced dedicated infrastructure including the smithtune CLI for model fine-tuning and LangSmith Engine, an autonomous Deep Agent that clusters runtime failures and generates pull requests. - [10:44](https://www.youtube.com/watch?v=xy_7SzwsIjM&t=644s) To handle semantic failures that bypass standard logs, LangChain created SmithDB and AI Search to query agent execution trajectories using free-form semantic questions. - [12:02](https://www.youtube.com/watch?v=xy_7SzwsIjM&t=722s) Chase positions executable eval suites as the new PRD for AI applications, emphasizing that grading external environment mutations is often far more reliable than assessing free-form text output. - [19:28](https://www.youtube.com/watch?v=xy_7SzwsIjM&t=1168s) LangChain outlines an autonomous 'North Star' improvement flywheel where agents flag their own errors, generate regression test cases, and propose self-healing code patches.
## Key takeaways - [00:00](https://www.youtube.com/watch?v=xy_7SzwsIjM&t=0s) An agent consists of three distinct architectural pillars: the model (reasoning engine), the harness (control flow, sandbox, tools, execution), and the context (text, files, and injected knowledge). - [00:00](https://www.youtube.com/watch?v=xy_7SzwsIjM&t=0s) Agent improvement workflows span three maturity tiers: manual AI engineer debugging, human-in-the-loop automated remediation, and fully autonomous self-updating agents. - [03:22](https://www.youtube.com/watch?v=xy_7SzwsIjM&t=202s) Model weight updates via supervised fine-tuning (SFT) suit narrower tasks like extraction or classification, while reinforcement learning (RL) offers a higher ceiling for complex reasoning despite lacking mature managed tooling. - [05:52](https://www.youtube.com/watch?v=xy_7SzwsIjM&t=352s) LangSmith Engine operates as a Deep Agent within a sandbox to monitor traces, cluster recurring runtime bugs, draft code patches, and submit pull requests for human review. - [07:13](https://www.youtube.com/watch?v=xy_7SzwsIjM&t=433s) Context is the most adaptable improvement layer because it relies on plain text or markdown and can update dynamically across individual users, projects, or entire organizations. - [08:42](https://www.youtube.com/watch?v=xy_7SzwsIjM&t=522s) OpenWiki provides an open-source, self-updating knowledge base for coding agents by analyzing execution traces where agents get stuck and feeding solutions back into indexed docs. - [09:23](https://www.youtube.com/watch?v=xy_7SzwsIjM&t=563s) High-fidelity execution observability relies on a strict structural hierarchy: runs (individual invocations), traces (task sequences), threads (conversations), and trajectories (standardized thread views). - [10:44](https://www.youtube.com/watch?v=xy_7SzwsIjM&t=644s) Because agents fail through semantic and conversational reasoning breakdowns rather than syntax errors, traditional logging databases are inadequate, prompting the creation of SmithDB and AI Search. - [12:02](https://www.youtube.com/watch?v=xy_7SzwsIjM&t=722s) Evals are replacing PRDs as living specifications because static requirements cannot capture the long tail of edge cases and off-topic user inputs encountered in production. - [13:38](https://www.youtube.com/watch?v=xy_7SzwsIjM&t=818s) Evaluating agent outputs by inspecting side effects—such as database state mutations and tool call sequences—is significantly more reliable than judging generative text responses. - [14:57](https://www.youtube.com/watch?v=xy_7SzwsIjM&t=897s) Running comprehensive eval suites requires isolated simulation environments and mock sandboxes to prevent destructive mutations on production databases or external third-party services like Salesforce. - [16:01](https://www.youtube.com/watch?v=xy_7SzwsIjM&t=961s) Because universal correctness judges do not exist, automated red teaming relies on consistency probing across paraphrased inputs to identify semantic divergences where at least one response is guaranteed incorrect. - [18:46](https://www.youtube.com/watch?v=xy_7SzwsIjM&t=1126s) The continuous improvement flywheel converts failure traces from human review or online judges into permanent regression evals before deploying updates to model weights, harness code, or context memory. - [19:28](https://www.youtube.com/watch?v=xy_7SzwsIjM&t=1168s) The industry's ultimate North Star vision is an autonomous loop where an agent observes its own failures, creates its own eval data points, and generates verified fixes without human intervention.
## Detailed walkthrough ### [00:00](https://www.youtube.com/watch?v=xy_7SzwsIjM&t=0s) Defining an Agent: Model, Harness, and Context
Harrison Chase opens the session by addressing what he describes as a guiding vision for LangChain and the broader AI industry: building agents that improve over time. While acknowledging that achieving this in a completely automated fashion remains a long-term goal rather than an immediate reality, he stresses that you cannot improve a system without first being precise about its architecture. As Harrison puts it, “Because you can't improve it if you don't define what it is.” To establish that baseline, he breaks an agent down into three fundamental components:
```mermaid flowchart TD accTitle: The Three Core Components of an Agent accDescr: Diagram showing how Context feeds into the Harness, which wraps and directs the Model.
subgraph AgentStructure["Core Agent Anatomy"] C["Context<br/>(Markdown, Files, Knowledge, Tools)"] H["Harness<br/>(Control Loop, Sandbox, Execution, Memory)"] M["Model<br/>(Reasoning Engine / Workhorse LLM)"] end
C -->|"Informs and bounds"| H H -->|"Manages prompts and actions"| M M -->|"Outputs decisions and tool calls"| H ```
First is the model, the core powerhouse and reasoning engine of the agent. Builders must choose an appropriate workhorse model based on performance traits like speed, reasoning depth, or context window length, while reserving smaller peripheral models for secondary tasks like guardrails and evaluations. Second is the harness, which encompasses the surrounding software system—such as Deep Agents, Claude Code, or Codex—that makes the model practically useful. While the model handles reasoning and decision-making, the harness governs control: it decides what the model sees, executes tool actions, observes and verifies outcomes, and persists state to disk or databases. LangChain's own Deep Agents framework exemplifies this layer by packaging an execution environment (a sandbox with a file system), context management utilities (skills, memory, summarization), and mechanisms for human-in-the-loop oversight and subagent delegation. Third is context, which refers primarily to the text, markdown, files, tools, and background knowledge injected into the system.
When considering how an agent improves over time, enhancements can occur across any of these three pillars—model, harness, or context. Furthermore, the operational loop driving those improvements spans three maturity levels: - **Manual (Human-driven):** An AI engineer inspects performance data, diagnoses problems, and manually implements updates, which reflects how most engineering teams operate today. - **Human-in-the-loop:** An automated background process analyzes system behavior and proposes concrete fixes, but requires a human engineer to review and approve changes before deployment. - **Fully Automated:** The agentic system monitors itself, updates its components, and deploys fixes autonomously without human intervention—a state Harrison characterizes as the industry's “North Star or the Holy Grail,” while conceding, “I don't think we're there right now by any stretch of the imagination.”
---
[03:22](https://www.youtube.com/watch?v=xy_7SzwsIjM&t=202s) Updating the Model: Supervised Fine-Tuning vs. Reinforcement Learning
Improving the model layer requires updating model weights, either for the core agent workhorse or for peripheral classifiers, guardrails, and evaluation models. Because adjusting model weights is computationally demanding, capital-intensive, and time-consuming, Harrison notes that model updates are almost always human-initiated rather than run continuously in the background on a whim. In practice, teams leverage two distinct methodologies: supervised fine-tuning (SFT) and reinforcement learning (RL).
Supervised fine-tuning works well for narrower, simpler tasks such as extraction, classification, or domain-specific summarization. In an SFT workflow, developers pull down execution traces, filter for high-performing trajectories where the agent succeeded, format the data, and train the model on those ideal behaviors. To streamline this path, LangChain launched LangSmith Fine-Tuning, centered around a dedicated CLI tool called `smithtune`. The CLI pulls logged trajectories directly from LangSmith, filters and processes them into standardized datasets, and hands them off to managed model provider APIs such as Fireworks or Baseten to produce an optimized fine-tuned checkpoint.
For complex, dynamic tasks—particularly training the primary decision-making model running the agent—some teams bypass SFT and reach for reinforcement learning. Harrison points out that RL has a significantly higher ceiling than SFT, but comes with steep technical hurdles:
> “RL is a lot more complicated than supervised fine-tuning. There's no managed offering for RL that's really that great.”
Unlike SFT, which simply fits on static logged trajectories, RL requires defining environment tasks and executing live rollouts. The agent attempts the task, is scored in real time against an objective function, backpropagates rewards to update model weights, and repeatedly runs subsequent rollouts to climb the policy gradient.
---
[05:52](https://www.youtube.com/watch?v=xy_7SzwsIjM&t=352s) Updating the Harness: How LangSmith Engine Automates Agent Refactoring
The harness consists of application code, control flow logic, prompts, and tool interfaces. Historically, updating the harness follows a straightforward manual loop: an AI engineer reviews production traces, diagnoses failure patterns, and edits the codebase or system prompts accordingly. Because altering harness logic or core instructions impacts all users universally, teams generally require human oversight before any change ships to production.
To move this process from manual maintenance toward automated human-in-the-loop refactoring, LangChain developed LangSmith Engine. Engine is an autonomous background agent designed to audit agent systems: it monitors traces, clusters recurring runtime failures into discrete issue groups, and formulates actionable remediations. When Engine identifies a fix, it generates code patches, opens a pull request, and adds regression tests and evaluation data points to verify the correction.
Structurally, LangSmith Engine is itself built as a Deep Agent operating inside a secure sandbox. It is equipped with the LangSmith CLI, specialized prompts, custom domain skills, and task-specific subagents that allow it to browse traces, evaluate regressions, and prepare pull requests for developer review.
---
[07:13](https://www.youtube.com/watch?v=xy_7SzwsIjM&t=433s) Updating Context and Memory: Hex, Profound, and OpenWiki
Context is the easiest and most flexible layer of an agent to update because it consists primarily of plain text, structured markdown, and retrieved documentation. Consequently, context updates do not have to happen strictly at the global agent level; they can evolve dynamically across individual users, projects, or entire organizations, forming persistent layers of memory.
Harrison highlights Hex and Profound as real-world products executing this strategy effectively: - **Hex:** In their data analytics platform, Hex maintains a unified context layer as a core value proposition. To keep this context accurate, Hex runs an automated Review Agent in the background that analyzes user interactions, flags missing semantic definitions or context gaps in datasets, and proposes context updates to improve future queries. - **Profound:** Operating in marketing AI, Profound synthesizes interaction data on multiple planes. At the organizational level, background jobs compile data into persistent "Brand Records," while at the individual level, the system curates "Personal Records" derived from direct conversations between users and their AI Marketer.
Within LangChain's own ecosystem, the team explores context evolution through OpenWiki, an open-source project designed to provide coding agents with a living, self-updating knowledge base. OpenWiki initializes its knowledge base by indexing a software repository's codebase. From there, integrations with coding agents like Claude Code or Codex monitor operational traces to identify conceptual areas where the coding agent repeatedly gets stuck or fails, feeding those insights back into the wiki so the agent has sharper context on future tasks.
Harrison concludes by highlighting the common denominator linking all three improvement loops—model, harness, and context:
> “We think there's two things that are common, traces and evals. And we think these are the core foundation of this improvement loop, and it's where we spend a lot of time.”
[09:23](https://www.youtube.com/watch?v=xy_7SzwsIjM&t=563s) The Hierarchy of Traces: Runs, Traces, Threads, and Trajectories
Harrison Chase acknowledges that the memory improvement loop varies dramatically depending on the specific vertical or use case. The most difficult challenge in that cycle is prompting-heavy: determining what information to record, what to discard, and how to format and store it. Because domain-specific prompting must often be handled by application builders, LangChain focuses its efforts on the shared underlying foundation: traces and evals.
```mermaid flowchart TD accTitle: Structural Hierarchy of Agent Observability Data accDescr: Diagram showing individual runs aggregating into a trace, traces forming a thread, and trajectories providing a standardized view. subgraph Execution["Execution Granularity"] R["Run<br>(Single model or tool invocation)"] T["Trace<br>(Sequence of runs in one task)"] TH["Thread<br>(Multi-turn conversation session)"] end subgraph Analysis["Standardized Observability"] TJ["Trajectory<br>(Standardized view over a thread)"] end R -->|Aggregates into| T T -->|Combines into| TH TH -->|Viewed as| TJ ```
To establish clarity around observability, Chase delineates the core terminology LangChain uses to dissect agent execution: - **Run**: A single discrete step, such as an individual model call or a single tool invocation. - **Trace**: A chronological sequence of runs generated when an agent is triggered to complete an action. - **Thread**: The overarching conversational container formed when a user exchanges multiple messages back and forth with an agent over time. - **Trajectory**: A standardized view mapped over a thread, capturing every single event in a uniform schema.
Regardless of whether a developer plans to fine-tune underlying model weights or leverage a coding agent to debug harness logic, having high-fidelity execution records is non-negotiable. As Chase emphasizes: “The important thing is that for all of these, you have the exact inputs, the exact outputs, and the exact sequence of events.”
---
[10:44](https://www.youtube.com/watch?v=xy_7SzwsIjM&t=644s) Semantic Failures, SmithDB, and AI Search
Traces represent an entirely new class of application data because LLM agents fail in ways that traditional software logs fail to capture. Instead of throwing standard system crashes or syntax exceptions, agents succumb to semantic failures and reasoning breakdowns that exist solely within the conversational context. Because standard logging databases are poorly equipped to query or triage these context-bound failures, LangChain created SmithDB, a database custom-built for agent observability—specifically engineered to ingest, store, query, and visualize agent interactions.
To illustrate how querying must evolve for semantic data, Chase highlights the newly launched AI Search (AI filter) feature. Rather than relying on rigid keyword matches or SQL filters, developers supply free-form semantic questions describing the failure or behavior they want to inspect. Under the hood, the system evaluates and scores each trajectory individually to determine whether it matches the criteria, allowing teams to isolate subtle reasoning bugs across massive datasets.
---
[12:02](https://www.youtube.com/watch?v=xy_7SzwsIjM&t=722s) Evals as the New PRD: Defining What “Good” Looks Like
Once developers gain visibility into traces, the next critical requirement is evaluation. Chase adopts an increasingly popular industry maxim: “First, they're the new PRD.” Whereas conventional software relies on static Product Requirements Documents to dictate behavior, AI systems encounter an endless long tail of ambiguous edge cases. Consequently, executable eval suites become the most accurate, living specification of what an agent should and should not do.
Chase characterizes evals as the essential guardrails that keep an agent in the safe zone throughout continuous iteration loops. However, building reliable evaluations remains exceptionally difficult: - **No silver bullets**: While platforms provide the infrastructure to run evaluations and visualize metrics, the datasets and evaluators must remain strictly domain-specific. - **Realistic inputs**: Anticipating the wild variety of user queries is notoriously hard; real users routinely veer completely off-topic or ask tangential questions that designers never anticipated during prompt design. - **Subject-matter expertise**: Specifying boundaries and desired agent behaviors requires close guidance from domain experts rather than purely engineering assumptions.
---
[13:38](https://www.youtube.com/watch?v=xy_7SzwsIjM&t=818s) Three Methods for Grading Agent Output
When defining what constitutes a good response, Chase breaks down the three primary evaluation approaches teams use in production:
| Evaluation Strategy | Definition & Scope | Primary Challenges & Use Cases | |:--- |:--- |:--- | | **Exact Match** | Enforcing that the model outputs a specific string, number, or verbatim phrase. | Fragile and unworkable for complex, multi-step agent interactions with variable wording. | | **Criteria-Based (Semantic)** | Verifying positive or negative criteria (e.g., must mention topic X, must never say Y). | Evaluates the substance of the response without over-constraining the phrasing. | | **Environment State / Tool Sequence** | Inspecting the end-state mutation of the database/environment or the exact sequence of tool calls. | Essential for agents performing real work (e.g., writing to databases, altering files) where final text output matters less than correct side effects. |
Chase notes that evaluating external mutations is often superior to judging text: “And so you don't want to judge its final answer. You want to judge how it interacts with its environment, or more specifically, what the end state of the environment is.”
---
[14:57](https://www.youtube.com/watch?v=xy_7SzwsIjM&t=897s) The Simulation Environment Dilemma
Executing dynamic agent evaluations creates a major infrastructure dilemma: agents require an environment to act upon, but running large test suites against live production backends is unacceptable—especially when agents execute write operations or mutate state. Teams are forced to architect dedicated simulation environments, using mock databases or ephemeral staging systems where agents can trigger tools freely.
This challenge compounds when agents interact across external third-party software ecosystems. While mocking an internally developed database is relatively straightforward, orchestrating reliable sandboxes for third-party platforms like Salesforce presents a significant operational hurdle because developers do not control those systems. Developing robust, isolated simulation environments remains one of the largest ongoing investments for teams shipping real-world agents.
---
[16:01](https://www.youtube.com/watch?v=xy_7SzwsIjM&t=961s) Proactive Discovery and Red Teaming Without Universal Judges
Teams traditionally uncover agent failures in two reactive ways: manually paying domain experts to stress-test the system, or scouring production traces after errors occur. The former is slow and prohibitively expensive; the latter carries severe downsides when rare failures damage user trust before teams even detect them. To address this gap, LangChain introduced automated red teaming to uncover vulnerabilities proactively before deployment.
Automated red teaming faces a core philosophical dilemma: judging correctness without a universal source of truth. As Chase observes: “The challenge here is how do we know if it's correct or not? So what I mean by this is in order to judge whether an answer is correct or not is actually not trivial at all. That requires a ton of domain knowledge. If you could do that reliably as part of the agent loop, then you just do it as part of the agent loop and run it and then judge it.”
Because universal correctness judges do not exist, LangChain employs specialized algorithmic probing strategies: - **Consistency Probing Across Paraphrased Inputs**: The red teaming engine sends two semantically identical queries phrased in slightly different ways. If the agent generates semantically divergent answers, the engine flags the inconsistency, knowing with certainty that at least one response must be incorrect. - **Targeted Category Reports**: In addition to factual accuracy, the system probes along distinct vectors such as security exploits and fairness, generating comprehensive red teaming reports on demand. - **Simulation Coupling**: Because probing requires running the agent end-to-end against live tools, proactive red teaming directly integrates with the simulated environments to prevent unwanted side effects.
[18:46](https://www.youtube.com/watch?v=xy_7SzwsIjM&t=1126s) The Improvement Flywheel: From Feedback to Eval Data Points
Harrison Chase highlights that building realistic simulation environments is an active, demanding area of study, noting that the LangChain labs team in particular is currently dedicating substantial research effort to solving it. Stepping back to the broader picture, he lays out the abstract lifecycle required to make any agent continuously get better at its job. As Harrison explains: “So at a high level, this is like the really high level abstract kind of like flywheel that you want to set up in order to improve your agent over time.”
```mermaid flowchart TD accTitle: The Agent Continuous Improvement Flywheel accDescr: Cycle showing how running an agent leads to feedback, trace inspection, eval creation, and targeted updates across model, harness, or context.
subgraph Execution["1. Execution & Evaluation"] Run["Run Agent"] --> Feed["Get Feedback<br/>(Human or Online Judge)"] Feed --> Trace["Inspect Trace Data"] end
subgraph Iteration["2. Eval & Update Loop"] Trace --> AddEval["Create Eval Data Point<br/>(Prevent Regressions)"] AddEval --> Update["Suggest Fixes / Updates"] end
Update -->|"Update Model Weights"| Run Update -->|"Update Harness Code"| Run Update -->|"Update Context / Memory"| Run ```
The cycle begins by running the agent in production or testing and collecting direct feedback on its performance, whether that signal originates from an explicit human review or an automated online judge. Once a failure or suboptimal response is flagged, developers must dig into the execution trace to pinpoint precisely what went wrong internally. That captured interaction cannot simply be fixed in isolation; it must be converted into a permanent evaluation benchmark to ensure future iterations actually fix the flaw without triggering regressions on previously solved scenarios. Finally, improvements are proposed and deployed across one of the agent's core layers: updating the model weights, modifying the harness code, or refining the context and memory.
- **Simulation research:** The LangChain labs team is actively researching how to build robust, synthetic simulation environments for evaluation. - **Feedback channels:** Quality signals can come from human review or automated online judges observing live runs. - **Regression prevention:** Converting problematic traces into concrete eval benchmarks ensures new agent updates do not break existing capabilities. - **Three levers for updates:** Fixes flow back into model weights (via fine-tuning/RL), harness code (application logic and tool interfaces), or context (memory stores and dynamic system knowledge).
---
[19:28](https://www.youtube.com/watch?v=xy_7SzwsIjM&t=1168s) The Autonomous Improvement Loop: A North Star Vision
Looking toward the frontier of agent engineering, Harrison presents what he considers the ultimate evolution of this flywheel: shifting human operators out of the maintenance loop entirely so the system can refine itself. As Harrison puts it: “The really kind of future version of this is where all of this is done by the agent itself.” In this self-healing setup, the agent actively detects when it makes an error, automatically drafts an evaluation case capturing that failure mode, and generates its own targeted remedy across its context, harness code, or underlying weights.
Harrison notes that LangChain has already begun exploring and implementing pieces of this autonomous workflow inside LangSmith Engine, though he cautions that fully realizing it remains an ambitious long-term goal. As Harrison emphasizes: “This is very much a North Star vision.” The end-to-end loop sounds clean and straightforward at an abstract level, but deploying it reliably introduces immense practical nuance around verification, safety, and stability. He invites teams grappling with these exact architecture problems to collaborate with LangChain, reiterating that solving the intricate details of agent self-improvement is one of the engineering challenges they are most excited about pursuing before closing out the talk.
- **Self-directed evaluation:** An agent monitors its own output, identifies failures, and automatically creates eval data points without manual intervention. - **Autonomous remediation:** The system formulates its own bug fixes or configuration patches across context memory, code harnesses, or fine-tuned model checkpoints. - **LangSmith Engine roadmap:** Autonomous agent repair forms the foundational direction and North Star vision for LangSmith Engine development.
## Notable quotes > [03:22](https://www.youtube.com/watch?v=xy_7SzwsIjM&t=202s) RL is a lot more complicated than supervised fine-tuning. There's no managed offering for RL that's really that great. — Harrison Chase
## Mentioned Harrison Chase, LangChain, LangSmith, LangSmith Engine, LangSmith Fine-Tuning, smithtune, SmithDB, OpenWiki, Deep Agents, Claude Code, Codex, Fireworks, Baseten, Hex, Profound, Salesforce
## Video description Harrison Chase, Co-Founder and CEO of LangChain, splits an agent into three parts, the model, the harness, and the context, and explains how each one can improve over time: the model's weights through supervised fine-tuning with LangSmith Fine-Tuning and its smithtune CLI, or through RL for more complex tasks; the harness through LangSmith Engine, a deep agent that runs over traces, clusters issues, and proposes fixes with a human in the loop; and the context through memory, where he points to Hex and Profound as companies he thinks do this well and to LangChain's open source OpenWiki project. He argues that traces and evals are the common foundation of all three loops, which is why LangChain built SmithDB and the new LangSmith AI Search for trace data, then covers what makes evals hard (realistic inputs, defining what good looks like, simulation environments) and how red teaming in LangSmith Engine looks for agent mistakes proactively. He closes with what he calls a North Star vision: an agent that notices its own mistakes, creates its own eval data points, and suggests its own fixes.
This session was captured at Interrupt London, the agent conference by LangChain.
Chapters: 0:00 What is an agent, and how do you improve it? 3:22 When should you fine-tune the model behind your agent? 5:52 How LangSmith Engine proposes fixes to your harness 7:13 Why context is the easiest part of an agent to improve 8:42 How OpenWiki gives coding agents a brain that updates 9:11 Why traces and evals are the foundation of improvement 10:44 Why LangChain built SmithDB and AI Search for traces 12:02 Why evals are the new PRD 12:50 Why is building evals so hard? 14:57 How to run evals without hitting production systems 16:01 How red teaming finds agent mistakes ahead of time 16:54 How do you judge correctness without a universal judge? 18:54 The flywheel that improves your agent over time 19:28 What if the agent ran its own improvement loop?
Resources: → LangSmith Engine: https://www.langchain.com/langsmith/engine → LangSmith Fine-Tuning (smithtune): https://docs.langchain.com/langsmith/smithtune → OpenWiki on GitHub: https://github.com/langchain-ai/openwiki → LangSmith: https://www.langchain.com/langsmith → Interrupt, the agent conference by LangChain: https://interrupt.langchain.com
[Source](https://www.youtube.com/watch?v=xy_7SzwsIjM)