LangChain's Chase Urges Custom AI Harnesses for Niche Tasks
Sequoia Capital
Building a custom agent harness is necessary when tasks diverge from standard model training distributions, though core components like file-editing primitives must still align with provider-specific reinforcement learning patterns.
## Background
This content was published on YouTube by Sequoia Capital on August 13, 2026, featuring Harrison Chase, co-founder and CEO of LangChain. Based on the page description, the session focuses on the agent harness—the orchestration loop that delivers context to a model—and addresses when to use off-the-shelf harnesses versus building custom ones, alongside insights on evals, observability, and data flywheels.
## TL;DR
Harrison Chase breaks down AI agents into three core components: the model, the context, and the harness that orchestrates their interaction. He explains that while general execution loops suffice for standard, in-distribution tasks, out-of-distribution applications and regulated domains require custom harnesses or explicit cognitive architectures. To continuously improve these systems, Chase emphasizes building private benchmarks with frameworks like Harbor, utilizing deep trace observability to identify context failures, and closing the data flywheel by leveraging autonomous tools like LangSmith Engine to turn production traces into automated fixes across harness logic, prompts, and models.
## Atmosphere & style
The presentation carries an authoritative, pragmatic, and highly technical tone tailored for an audience of AI builders and engineering leaders. Chase delivers a structured, grounded key-note style talk that strips away hype, focusing instead on real-world architectural tradeoffs and practical implementation details. Listening to the session feels like a deep dive into modern AI software engineering, balancing clear conceptual frameworks with concrete engineering lessons, live product demos, and candid insights into model behavior.
## Key takeaways
- [00:58](https://www.youtube.com/watch?v=HI2q3ci3Iuc&t=58s) An AI agent consists of three fundamental pillars: the model, the context, and the harness that orchestrates execution.
- [02:12](https://www.youtube.com/watch?v=HI2q3ci3Iuc&t=132s) A harness's primary job is delivering the right context to the model at the exact right point in time.
- [03:25](https://www.youtube.com/watch?v=HI2q3ci3Iuc&t=205s) Developers can customize standard agent execution loops using middleware constructs, hooks, and plugins without rewriting core logic.
- [05:47](https://www.youtube.com/watch?v=HI2q3ci3Iuc&t=347s) Explicit cognitive architectures remain valuable for domain-specific tasks where strict guidance, predictability, and safety gates are required.
- [07:03](https://www.youtube.com/watch?v=HI2q3ci3Iuc&t=423s) Building a custom harness is necessary for out-of-distribution tasks, but in-distribution sub-tasks like file editing should stay aligned with native model training.
- [09:39](https://www.youtube.com/watch?v=HI2q3ci3Iuc&t=579s) Private evals define what 'good' performance means within an organization and form the benchmark for continuous improvement.
- [11:04](https://www.youtube.com/watch?v=HI2q3ci3Iuc&t=664s) The open-source Harbor framework structures eval tasks using three components: an environment, a verifier, and an instruction prompt.
- [13:20](https://www.youtube.com/watch?v=HI2q3ci3Iuc&t=800s) Most agent failures stem from bad context rather than insufficient model reasoning capacity, making deep observability critical.
- [14:24](https://www.youtube.com/watch?v=HI2q3ci3Iuc&t=864s) A continuous learning data flywheel relies on capturing execution traces, curating them, and running iterative experiments.
- [15:16](https://www.youtube.com/watch?v=HI2q3ci3Iuc&t=916s) Implicit UX feedback and fine-tuned small language models as evaluators reduce the cost of trace evaluation compared to using top-tier frontier models.
- [16:20](https://www.youtube.com/watch?v=HI2q3ci3Iuc&t=980s) Production trace data can be leveraged to update all three components: the harness, the model weights, or the context.
- [17:28](https://www.youtube.com/watch?v=HI2q3ci3Iuc&t=1048s) LangSmith Engine uses production traces to automatically curate failure patterns, issue reports, and propose code updates to the harness.
- [20:44](https://www.youtube.com/watch?v=HI2q3ci3Iuc&t=1244s) Harnesses are converging around off-the-shelf standards for general tasks, but diverging into custom architectures for out-of-distribution enterprise applications.
## Detailed walkthrough
### [00:00](https://www.youtube.com/watch?v=HI2q3ci3Iuc&t=0s) Introduction and the Three Core Components of an Agent
The session opens with an introduction highlighting how the landscape of building AI agents has rapidly matured since the early days of GPT-3 in 2022. While early language models functioned primarily as sophisticated text auto-completers, the focus has shifted entirely toward building harnesses around them so they can act as true virtual collaborators. Harrison Chase, co-founder and CEO of LangChain, takes the stage to frame what it means to "own your intelligence" in the modern AI ecosystem. He breaks an agent down into three fundamental pillars:
* **The Harness:** The orchestration framework that runs the execution loop, managing how the model interacts with tools and data.
* **The Model:** The underlying language model (LLM). Chase notes the importance of remaining model-agnostic to prevent vendor lock-in and retain the flexibility to swap in the best-performing open-weight or proprietary models (such as those provided by Fireworks) as they emerge.
* **The Context:** The dynamic and persistent state—including long-term memory, semantic knowledge bases, and conversation history—that personalizes and guides the agent throughout its task execution.
[00:02](https://www.youtube.com/watch?v=HI2q3ci3Iuc&t=2s) What a Harness Does and Customizing the Core Loop
At its core, a harness's primary responsibility is to bring the right context to the model at the exact right point in time. While the public often views agents through the lens of a deceptively simple architecture—an LLM running in a loop making tool calls and receiving environment observations—real-world implementation requires extensive customization. Chase contrasts LangChain (a minimal, foundational harness) with "Deep Agents" (LangChain's general-purpose, model-agnostic harness inspired by systems like Claude Code).
To tailor this central execution loop to specific domains without re-architecting the system from scratch, developers use middleware constructs, hooks, and plugins. These entry points allow developers to intercept and modify execution at specific lifecycle events:
* **Pre-invocation & Pre-model Hooks:** Executing custom logic or checks right before the model is called.
* **Model Call Wrapping:** Modifying prompts, enforcing guardrails, or manipulating token limits dynamically.
* **Tool Call Wrapping & Interception:** Modifying tool arguments, handling execution errors, or offloading oversized outputs.
* **Middleware Capabilities:** Adding sandboxed code execution, dedicated file system access, specialized sub-agent delegation, and automated context summarization (e.g., trimming or summarizing context right before a model call if token limits are approached).
[05:47](https://www.youtube.com/watch?v=HI2q3ci3Iuc&t=347s) Cognitive Architectures vs. General Loops
In the early years of agent development (2023–2024), weaker model capabilities forced developers to build highly rigid, bespoke "cognitive architectures." These hardcoded flowcharts forced the model through explicit steps—such as generating sub-questions, fanning out search tasks in parallel (for deep research), or moving through predefined validation gates (for code review bots).
While modern, highly capable LLMs allow developers to start with a generalized loop, explicit cognitive architectures remain valuable for domain-specific tasks where strict guidance is required. Chase advises teams to start with a general harness to achieve fast time-to-value, only adding constrained gates, checks, and structured paths as they identify edge cases in their specific use cases.
[07:03](https://www.youtube.com/watch?v=HI2q3ci3Iuc&t=423s) In-Distribution vs. Out-of-Distribution Customization
When deciding whether to use an off-the-shelf harness (like Claude Code, Claude Agent SDK, or OpenAI's Codex) versus building a custom one, the key factor is whether the task is **in-distribution** or **out-of-distribution** relative to how the underlying models were trained.
If a task is far out-of-distribution (for example, specialized legal AI workflows), a custom harness becomes necessary. However, developers must be careful to keep *in-distribution sub-tasks* as close to the model's native training as possible:
* **The File-Editing Example:** Models like OpenAI and Anthropic's Claude have been fine-tuned using Reinforcement Learning (RL) on specific file-editing formats and patterns. A Claude model edits files differently than an OpenAI model because each was RL-trained on its own harness patterns.
* **Model Profiles in Deep Agents:** To avoid breaking these learned behaviors, Deep Agents uses "model profiles." Even within a fully customized, out-of-distribution legal harness, the system dynamically switches its low-level file-editing tools to match the specific model in use, preserving in-distribution performance.
[09:39](https://www.youtube.com/watch?v=HI2q3ci3Iuc&t=579s) Evals, Observability, and the Harbor Framework
To continuously improve agents, developers need robust evaluation and observability frameworks. Quoting insights from AI practitioner Satya, Chase emphasizes three core tenets of owning organizational intelligence:
1. *Create private evals*, because an evaluation suite explicitly defines what "good" performance looks like inside an organization.
2. *Retain ownership* over memory, execution traces, feedback, and institutional decisions.
3. *Build a continuous learning loop*—a "hill-climbing machine" that allows AI investments to compound in value over time.
To structure these benchmarks, the industry is coalescing around **Harbor**, an open-source evaluation runner developed by the creators of Terminal Bench 2 and widely used for coding benchmarks like Frontier Bench.
A Harbor evaluation task consists of three main elements:
* **Environment:** A sandboxed container (defined via Dockerfile) where long-running, stateful agents interact with system tools and files.
* **Verifier (Test Script):** The scoring mechanism, which can range from standard unit tests and code execution to LLM-as-a-judge or sub-agent evaluators.
* **Instruction (`instruction.md`):** The exact system prompt and task instructions fed into the agent.
These benchmark runs generate detailed evaluation telemetry—tracked in platforms like LangSmith—allowing teams to compare different models, harnesses, and reasoning levels not just on raw accuracy, but also on latency and token cost.
[13:20](https://www.youtube.com/watch?v=HI2q3ci3Iuc&t=800s) Observability and Trajectories
Observability is often underrated, yet it is essential for debugging failing agents. When an agent fails, it is usually caused by one of two factors: the model lacks the reasoning capacity, or the context provided to the model was flawed. In practice, bad context is the primary root cause of agent failures.
Effective observability provides clear visibility into how context accumulates step-by-step across execution trajectories—the full sequence of human messages, internal tool invocations, and model responses:
* **Trajectory Tracking:** Capturing the complete message history and tool inputs/outputs as the agent executes its multi-step loops.
* **Developer-Friendly Views:** Displaying agent trajectories in clean UI formats (similar to the Claude Code interface) that collapse repetitive intermediate tool calls while preserving full trace data for deep-dive debugging.
[14:24](https://www.youtube.com/watch?v=HI2q3ci3Iuc&t=864s) Observability and the Agent Data Flywheel
While high-level performance metrics are a start, they are insufficient for truly debugging an agent. Full execution traces are critical because they allow developers to click directly into specific operations and examine exactly what is happening inside the model. This degree of observability, combined with rigorous evaluation benchmarks, provides the foundation for setting up a data flywheel—a process that compounds an agent's intelligence over time as real users interact with the system and generate feedback.
At Swyx's AI Engineering Fair, a team member presented a simple high-level recipe for continuously improving agents: build the agent, run it to collect execution traces, curate that trace data, and run experiments on the curated datasets. Though conceptually straightforward, executing this flywheel involves significant operational complexity.
- **Data Flywheel Recipe:** Build agent $\rightarrow$ Collect execution traces $\rightarrow$ Curate trace data $\rightarrow$ Run experiments.
- **Trace Observability:** Deep visibility into intermediate steps is required to diagnose why an agent failed or produced suboptimal context.
[15:16](https://www.youtube.com/watch?v=HI2q3ci3Iuc&t=916s) Collecting Feedback: Clever UX and Online Evaluators
A crucial aspect of curating trace data is acquiring reliable feedback, either from the environment or through synthetic mechanisms. Explicit feedback from real users (such as clicking thumbs up or thumbs down) is rare because very few people actually use those buttons. However, clever user interface (UX) design can implicitly extract valuable user feedback without requiring deliberate rating actions.
Complementing implicit user signals, developers can deploy "online evaluators" that run continuously over incoming execution traces. Running top-tier models like Claude 3.5 Opus over every trace to act as an LLM-as-a-judge rapidly becomes cost-prohibitive. To solve this, developers can fine-tune small language models (SLMs) for trace evaluation—an approach demonstrated in experiments with Harvey that drastically cut evaluation costs—or rely on off-the-shelf models with custom prompting, or simple programmatic code checks for basic assertions.
- **Environmental Feedback:** Users rarely click explicit thumbs-up/thumbs-down icons; thoughtful UX design is needed to infer feedback implicitly.
- **Synthetic Feedback:** Online evaluators judge traces continuously in production.
- **Cost Reduction:** Running frontier models like Opus as judges across all traces racks up massive bills; fine-tuned SLMs, custom-prompted models, or code-based assertions provide cost-effective alternatives.
[16:20](https://www.youtube.com/watch?v=HI2q3ci3Iuc&t=980s) Updating the System: Harness, Model, and Context
The data gathered and curated through production traces can be leveraged to update any of the three fundamental components of an agent system:
1. **The Harness:** Improved through harness engineering (modifying loops, middleware, and tools).
2. **The Model:** Improved through targeted fine-tuning on curated trace datasets.
3. **The Context:** Improved by implementing better memory systems and context management.
While other approaches focus heavily on model fine-tuning from traces, LangChain concentrates primarily on automating the harness engineering loop to systematically iterate on agent architectures.
- **Three Upgrade Targets:** Traces inform updates to the harness (orchestration logic), the model (weights), or the context (retrieval and memory).
- **Automation Goal:** Because manual curation and harness iteration are time-consuming and complex, automating the trace-to-harness loop is a central priority.
[17:28](https://www.youtube.com/watch?v=HI2q3ci3Iuc&t=1048s) Demo: Automating Improvement with LangSmith Engine
LangChain introduced **LangSmith Engine**, an autonomous coding agent designed to sit directly on top of production traces and automate the curation and debugging workflow. Under the hood, Engine operates as an agent equipped with access to the LangSmith CLI, allowing it to filter execution traces by user feedback, error states, and other metadata.
During a live demonstration, Chase showcased how Engine continuously scans trace streams to identify recurring error patterns and automatically generates "issue boards." Each issue includes a detailed description of the problem, direct links to supporting evidence within the trace logs, and proposed code edits. Engine automatically generates updates across the system—modifying context templates, updating system instructions, and writing code patches directly for the agent harness itself.
- **LangSmith Engine:** An autonomous agent that monitors production traces via the LangSmith CLI to discover failure patterns.
- **Automated Issue Curation:** Creates issue boards containing problem descriptions, supporting trace links, and recommended fixes.
- **Multi-Layer Fixes:** Generates pull requests or code edits that modify context instructions, prompts, and harness logic.
[19:23](https://www.youtube.com/watch?v=HI2q3ci3Iuc&t=1163s) Q&A: Dogfooding Engine and the "Codex-ification" Sprint
When asked if LangChain runs Engine on Engine itself, Chase confirmed that they dogfood the system internally. Engine connects directly to Slack to send self-diagnostic reports and performance updates about its own trace data. To evaluate Engine's capabilities systematically, the team built **IssueBench**, a benchmark formatted using the Harbor standard.
By evaluating different agent harnesses and models on IssueBench, the team uncovered a key behavioral insight: OpenAI's Codex harness performed exceptionally well because it aggressively wrote small, custom scripts on the fly to query and process trace data. Recognizing this advantage, the LangChain team executed a rapid engineering sprint—termed the "codex-ification" of Engine—to integrate dynamic script-writing capabilities into Engine's core harness.
> "Codex was doing a really interesting thing, where it would write itself a bunch of small scripts to run against these traces... so we did a sprint to do what we call the 'codex-ification' of Engine and basically take that learning and bring it into the core Engine harness."
- **Dogfooding:** Engine analyzes its own traces and reports diagnostic findings to the engineering team via Slack.
- **IssueBench:** A specialized benchmark based on the Harbor format used to test Engine across various models and harnesses.
- **Codex-ification Sprint:** After discovering that Codex outperformed other setups by generating dynamic execution scripts, LangChain updated Engine's harness to adopt the same strategy.
[20:44](https://www.youtube.com/watch?v=HI2q3ci3Iuc&t=1244s) Q&A: Will Agent Harnesses Converge or Diverge?
Addressing whether agent harnesses will eventually converge into a single standardized pattern or diverge into domain-specific implementations, Chase noted that both forces are currently at play. For general-purpose tasks and getting started, off-the-shelf harnesses (such as Deep Agents, Codex, or Claude Code) have become good enough for most standard use cases. However, as applications move further out-of-distribution (OOD), the demand for custom harnesses grows significantly.
Custom cognitive architectures remain essential in highly regulated domains like financial services, where predictable, deterministic control is strictly required. Showplaying an off-the-shelf, highly autonomous agent to financial institutions often frightens stakeholders who prefer tightly constrained workflows. At the same time, subtle model-level differences force harness adaptation; for example, OpenAI and Anthropic have settled on distinctly different file-editing mechanisms. If model providers begin specializing heavily in specific domains—such as biology—harnesses will naturally diverge to optimize for those specialized agentic workflows.
> "The more out of distribution you get, the more you're going to want to customize the harness. It's a scale... At the extreme end, you might want to build a complete cognitive architecture... for predictability and control."
- **Convergence at the Baseline:** Off-the-shelf harnesses work well for standard coding and general-purpose tasks due to built-in support for sandboxes, file systems, and sub-agents.
- **Divergence for OOD & Enterprise:** Highly out-of-distribution tasks and risk-averse industries (e.g., financial services) require custom harnesses for strict safety, predictability, and control.
- **Model-Specific Differences:** AI labs employ differing file-editing primitives, forcing harnesses to accommodate specific model quirks or domain specializations.
## Notable quotes
> [19:23](https://www.youtube.com/watch?v=HI2q3ci3Iuc&t=1163s) Codex was doing a really interesting thing, where it would write itself a bunch of small scripts to run against these traces... so we did a sprint to do what we call the 'codex-ification' of Engine and basically take that learning and bring it into the core Engine harness. — Harrison Chase
> [20:44](https://www.youtube.com/watch?v=HI2q3ci3Iuc&t=1244s) The more out of distribution you get, the more you're going to want to customize the harness. It's a scale... At the extreme end, you might want to build a complete cognitive architecture... for predictability and control. — Harrison Chase
## Mentioned
Harrison Chase, LangChain, Sequoia Capital, YouTube, GPT-3, Fireworks, Deep Agents, Claude Code, Claude Agent SDK, OpenAI, Anthropic, Claude, Satya, Harbor, Terminal Bench 2, Frontier Bench, LangSmith, Swyx, AI Engineering Fair, Claude 3.5 Opus, Harvey, LangSmith Engine, IssueBench, Codex
Harrison Chase argues that building effective AI agents requires separating the system into harness, model, and context, advising teams to start with generalized execution loops and reserve custom harnesses for tasks that diverge significantly from baseline model training.
Deconstructing Agents and Customizing Execution Loops (0:02)
Chase defines modern AI agents around three core components: the orchestration harness that runs the execution loop, the swappable model, and the context containing memory and system state. Whereas developers in 2023–2024 relied on rigid cognitive architectures—hardcoded flowcharts like parallel fan-out search or fixed validation gates to compensate for weaker model reasoning—modern high-capability LLMs allow teams to begin with general-purpose loops.
Rather than rebuilding architectures from scratch, teams can tailor foundational harnesses through middleware, lifecycle hooks, and tool interception. Pre-model checks, dynamic token limit adjustments, sandboxed code execution, and context summarization enable developers to retain a fast time-to-value with general loops while progressively inserting domain-specific constraints as edge cases arise.
Customizing for Out-of-Distribution Tasks and Models (7:03)
The choice between using an off-the-shelf harness (such as Claude Code or OpenAI Codex) and engineering a custom solution depends on whether a task is in-distribution or out-of-distribution (OOD) relative to model pre-training. Standard coding tasks operate well within existing harnesses, but specialized domains like legal analysis or risk-averse environments like financial services require custom harnesses to ensure deterministic control and institutional compliance.
However, Chase cautions that even within custom OOD harnesses, sub-tasks must respect how specific models were trained with reinforcement learning. Because Anthropic and OpenAI models are fine-tuned on distinct file-editing conventions, LangChain's Deep Agents framework utilizes dynamic "model profiles" to swap low-level tool calling patterns to match each model's native habits without degrading performance.
Evaluating Trajectories and Context Failures via Harbor (9:39)
Chase notes that when agents fail, the primary root cause is typically defective context rather than model reasoning deficits. Maintaining organizational ownership of AI therefore requires private evaluation suites, trace telemetry, and continuous feedback loops. The industry is standardizing evaluation runners on Harbor—an open-source framework developed by the creators of Terminal Bench 2—which isolates tasks across sandboxed Docker environments, verifier scripts (such as unit tests or LLM-as-a-judge), and explicit prompt instructions.
Detailed observability across entire execution trajectories—capturing sequences of human inputs, tool calls, and model outputs—is critical to isolating where context degraded. Clean trajectory views in tools like LangSmith allow teams to collapse repetitive intermediate tool calls while retaining granular telemetry to measure accuracy against token costs and latency.
Automating the Flywheel with Trace-Level Engineering (17:28)
Continuous agent improvement relies on an operational flywheel: deploying the agent, gathering execution traces, curating the trace data, and testing updates across the harness, model weights, or context systems. Because end users rarely use explicit thumbs-up or thumbs-down buttons, teams collect feedback through clever UI design and online evaluators. To avoid the high cost of running frontier models as judges, developers use custom code assertions or fine-tuned small language models (SLMs), as demonstrated in cost-reduction experiments with Harvey.
To automate this cycle, LangChain built LangSmith Engine, an autonomous agent that monitors CLI trace streams, identifies error patterns, compiles issue boards, and drafts pull requests directly modifying harness code. By evaluating harnesses on IssueBench, LangChain observed that OpenAI's Codex harness achieved high performance by writing dynamic, on-the-fly execution scripts, prompting a "codex-ification" engineering sprint to adopt the same dynamic scripting pattern into Engine.
Chase concludes that while general-purpose tasks will continue to converge around standard off-the-shelf harnesses, domain specialization and regulatory demands will drive divergence toward custom harnesses. The central design challenge remains building flexible orchestration layers that can absorb proprietary model quirks while automating harness iteration through production trace data.
## Background
该视频由红杉资本（Sequoia Capital）于 2026 年 8 月 18 日发布在 YouTube 平台。视频记录了 LangChain 联合创始人兼 CEO Harrison Chase 在“Own Your Intelligence”活动上的主题演讲与问答，探讨智能体 Harness（运行框架与编排循环）的核心机制、何时采用开箱即用方案与何时自研、分布内外的权衡，以及如何通过私有评测（Evals）、可观测性与数据飞轮实现系统的持续迭代。
## TL;DR
Harrison Chase 指出智能体由 Harness、模型与上下文三大支柱构成，其本质是在循环中按需向模型精准输送上下文并调度工具，系统的差异化关键在于利用中间件与钩子实现的编排细节。在架构选型上，团队应根据业务偏离模型训练的程度决定是否自研 Harness，但对文件编辑等分布内原子操作应贴合各模型的原生训练习惯。此外，绝大多数智能体失效源于上下文缺陷而非模型能力不足，因此构建私有基准测试（如 Harbor 体系）与深度链路可观测性是企业定义质量的根本。最后，LangChain 提出了通过前端隐式反馈与在线评估转动数据飞轮的路径，并演示了利用 LangSmith Engine 自动分析 Trace、定位问题并生成 Harness 修复补丁的自举迭代模式。
## Atmosphere & style
整场分享呈现出一种极度务实、专注落地排障的硬核工程氛围，红杉主持人的开场引言带着对 LLM 从“自动补全”演进到自主智能体的历史纵深感。Harrison Chase 表达清晰、条理分明，用不加修饰的行业洞察（如直言“现实中没人会去点赞点踩”）打破虚假繁荣，直戳可观测性与上下文管理的真实痛点。从展示 LangSmith Engine 自动修 Bug 到分享内部“吃狗粮”与 Codex 较量的细节，互动兼具极客探索的技术兴奋感与探讨行业未来分化趋势时的理性与克制，让观众如同身临顶尖 AI 架构师的实战复盘现场。
## Bulletin
- [00:58](https://www.youtube.com/watch?v=HI2q3ci3Iuc&t=58s) 智能体系统由 Harness（运行编排）、模型（Model）和上下文（Context）三大支柱构成；企业掌控自身智能的关键在于保持无缝切换模型的中立能力，将私有数据与轨迹沉淀为上下文壁垒，并借助自研或定制 Harness 在精准时间向模型输送正确信息。
- [07:03](https://www.youtube.com/watch?v=HI2q3ci3Iuc&t=423s) 是否自研 Harness 取决于任务是否处于训练分布内（In-distribution vs. Out-of-distribution）：业务逻辑越偏离通用预训练分布越需要自研架构，但在文件编辑等细粒度原子操作上，应通过模型画像（Model Profiles）紧密贴合各主流模型的原生强化学习微调范式。
- [13:04](https://www.youtube.com/watch?v=HI2q3ci3Iuc&t=784s) 在智能体排错与运行调优中，上下文缺陷导致的失败远多于基础模型能力缺失；建立私有评测集（Evals）并辅以端到端上下文可观测性，是界定组织内部“何为优秀”并防止系统能力回归的生命线。
- [16:51](https://www.youtube.com/watch?v=HI2q3ci3Iuc&t=1011s) LangChain 演示了自动化数据飞轮产品 LangSmith Engine：该 Coding Agent 可直接挂载在生产 Trace 流上自主挖掘异常模式，并自动生成修复建议以及针对 Harness 底层工程的代码补丁。
## Key takeaways
- [00:58](https://www.youtube.com/watch?v=HI2q3ci3Iuc&t=58s) 掌控智能的三大支柱包括负责编排调度的 Harness、作为推理底座的 Model，以及承载核心壁垒的 Context；掌控模型不一定指自研预训练，更体现为拥有低成本无缝切换前沿模型以规避供应商锁定的自由度。
- [02:12](https://www.youtube.com/watch?v=HI2q3ci3Iuc&t=132s) 智能体架构的本质是“大语言模型置身于循环之中持续调用外部工具”，Harness 最核心的工程职能便是在最精准的时间点将最恰当的上下文编排输送至模型推理窗口。
- [03:25](https://www.youtube.com/watch?v=HI2q3ci3Iuc&t=205s) 通过类似 Web 框架的中间件（Middleware）与钩子机制，Harness 能够在保留极简通用循环的同时，在调用前后实现自动化长文本摘要（Summarization）、海量日志上下文卸载（Context Offloading）、隔离沙箱挂载及子智能体任务派发。
- [05:47](https://www.youtube.com/watch?v=HI2q3ci3Iuc&t=347s) 认知架构的选择应当遵循实用演进路径：业务起步阶段优先采用通用循环 Harness 以最小化交付周期；当业务收敛到高严谨度、低容错阶段时，再针对性引入硬编码状态机与确定性门禁（Gates）。
- [07:03](https://www.youtube.com/watch?v=HI2q3ci3Iuc&t=423s) 前沿模型实验室（如 OpenAI 与 Anthropic）在文件编辑等基础工具调用上具有截然不同的强化学习微调习惯；即便在定制 Harness 中，也应通过 Model Profile 兼容原厂调用接口，避免自造协议破坏模型原生的微调收益。
- [09:39](https://www.youtube.com/watch?v=HI2q3ci3Iuc&t=579s) 私有评测集（Evals）定义了组织内部对于输出质量的标准与边界，企业需结合运行追踪（Traces）与机构化上下文打造持续学习的“爬山机”，量化衡量系统每一次迭代的实际成效。
- [11:10](https://www.youtube.com/watch?v=HI2q3ci3Iuc&t=670s) 现代智能体评测体系（如 Harbor 规范）将测试任务标准化为隔离沙箱环境、Prompt 指令、验证器（Verifier）与基准解四个组件，评测维度必须将准确率、端到端延迟与 Token 经济成本综合权衡。
- [13:04](https://www.youtube.com/watch?v=HI2q3ci3Iuc&t=784s) 智能体可观测性被严重低估：当多步推理崩溃脱轨时，归因往往非模型推理不足，而是上下文窗口中的输入信息存在污染或遗漏；具备全链路追踪与上下文演进透视能力是定位问题的核心前提。
- [14:34](https://www.youtube.com/watch?v=HI2q3ci3Iuc&t=874s) 持续进化的智能体数据飞轮依赖高质量反馈输入：产品交互（UX）应通过允许用户修改与撤销来隐式收集偏好，同时利用低成本微调小模型（SLM）或确定性逻辑替代昂贵的大模型在线评估。
- [16:51](https://www.youtube.com/watch?v=HI2q3ci3Iuc&t=1011s) LangSmith Engine 实现了利用 Agent 自动化闭环数据飞轮：通过命令行检索并清洗生产 Trace，定位共性系统问题，最终直接输出用于 Prompt 优化或 Harness 代码修改的修复补丁。
- [19:19](https://www.youtube.com/watch?v=HI2q3ci3Iuc&t=1159s) 团队通过 IssueBench 评测集对比不同 Harness 表现，发现 Codex 在处理复杂数据时具备自主编写临时执行脚本的优势，进而将这种“Codex 化（Codex-ification）”行为成功吸纳固化至自身 Harness 架构中。
- [20:44](https://www.youtube.com/watch?v=HI2q3ci3Iuc&t=1244s) Harness 生态未来不会简单趋于单一收敛：金融等严监管业务对确定性与控制力的苛求使得定制认知架构不可或缺，加之模型厂商的技术路线分歧及未来垂直行业（如生物医药）专用模型的涌现，将驱动 Harness 持续走向多元分化。
## Detailed walkthrough
### [00:00](https://www.youtube.com/watch?v=HI2q3ci3Iuc&t=0s) 开场引言：从自动补全到智能体 Harness 的演进
活动开场由红杉资本的主持人向现场观众抛出议题：当前许多团队都在积极思考是否应当自研专用的 Harness（智能体支架/运行框架）。主持人热情引介了 LangChain 联合创始人兼 CEO Harrison Chase，并回忆道，自己早在 2022 年 GPT-3 时代便在 Twitter 上注意到了 Harrison。当时大多数人还将大语言模型视为单纯的文本“自动补全（auto-complete）”工具，而 Harrison 是最早探索如何围绕模型搭建一整套 Harness、使其蜕变具备自主行动能力的虚拟协作者与智能体（Agent）的先驱之一。
从 2022 年至今，AI 智能体生态经历了爆发式增长。主持人感慨 Harrison 本人在智能体架构、Harness 设计以及评估（Eval）体系上的认知也在持续进化。本次分享 Harrison 将围绕 Harness 架构与模型评估两个核心维度展开，先进行约 15 分钟的主题演讲，随后留出 15 分钟与参会者进行深度问答交流。
### [00:58](https://www.youtube.com/watch?v=HI2q3ci3Iuc&t=58s) 掌控智能的三大支柱：模型、上下文与 Harness
Harrison 登台后直奔主题，指出在探讨“掌控自身智能（Owning Your Own Intelligence）”的大背景下，行业焦点通常落在智能体（Agent）上。LangChain 认为，一个完整的智能体由三个核心支柱紧密啮合而成：负责编排的 **Harness**、提供推理底座的 **Model（模型）**，以及注入关键信息的 **Context（上下文）**。若企业希望真正掌控智能主权，通常意味着必须对这三个部分都拥有掌控力。
```mermaid
flowchart TD
accTitle: 智能体的三大核心组成要素
accDescr: 展现了智能体由 Harness、模型和上下文三部分共同构成，其中 Harness 负责核心的编排逻辑。
subgraph Agent["智能体核心架构"]
H["Harness<br/>运行时与编排调度"]
M["Model<br/>模型底座与切换能力"]
C["Context<br/>记忆、知识与对话历史"]
end
H -->|按需动态注入| M
C -->|提供背景事实| H
M -->|生成调用与决策| H
```
关于模型层面，Harrison 强调掌控模型并不一定意味着要自己从头预训练，Fireworks AI 的 Lin 此前分享了开源权重模型带来的掌控感；更现实的路径在于具备“无缝切换模型”的能力。这类似于过去云计算时代提倡的“云中立（Cloud-Agnostic）”，能够随时根据性价比与任务表现切换到业界最新、最强的底座模型，规避供应商锁定。至于上下文，则是企业最具壁垒的护城河，包括内部语义知识、长期记忆以及历史对话轨迹，它们赋予智能体独特的个性化指导与执行边界。而本次演讲的核心，正是连接两者的关键调度中枢——Harness。
### [02:12](https://www.youtube.com/watch?v=HI2q3ci3Iuc&t=132s) Harness 的核心机制：循环调用与上下文编排
Harrison 抛出了核心问题：Harness 的本职工作到底是什么？简而言之，“Harness 的核心任务就是在最精准的时间点，将最恰当的上下文输送给模型”。它负责打理所有静态与动态的上下文数据，在推理窗口内容积有限的情况下，把关键信息精准投递进上下文窗口（Context Window），收集模型的推理产出，并根据反馈触发后续动作。
智能体在执行任务时需要与各类外部系统发生交互。每一次系统交互、API 调用或代码执行，都会生成全新的上下文碎片，而这些碎片又必须被有选择地反馈回运行流中。剥离所有复杂的行业术语，当业界谈论智能体时，其本质架构非常单纯：“就是一个大模型置身于一个循环之中，不断进行工具调用（an LLM running in a loop calling tools）”。
```mermaid
flowchart LR
accTitle: 智能体基础循环架构
accDescr: 描绘了用户请求进入后，LLM 决策生成工具调用，系统执行工具并将观察结果回传 LLM 的核心循环。
Req["外部请求输入"] --> Loop
subgraph Loop["核心执行循环"]
direction LR
LLM["LLM 推理决策"]
Tool["工具执行调用"]
Obs["获取环境观察结果"]
LLM -->|发出工具调用| Tool
Tool -->|返回执行结果| Obs
Obs -->|写回上下文| LLM
end
```
- **极简通用的核心范式**：请求进入系统 $\rightarrow$ LLM 进行推理并生成工具调用 $\rightarrow$ 系统执行工具 $\rightarrow$ 将观察结果（Observation）作为新上下文回传 $\rightarrow$ LLM 决定下一步操作或完成任务。
- **差异化的落脚点**：尽管所有智能体底层都遵循上述简单循环，但优秀与平庸系统的分野，完全取决于 Harness 在循环各个节点所做的细节编排与控制。
### [03:25](https://www.youtube.com/watch?v=HI2q3ci3Iuc&t=205s) 借助中间件定制 Harness：沙箱、子智能体与上下文卸载
为了让简单的通用循环适应复杂的工程环境，Harness 必须提供灵活的定制接口。Harrison 介绍了 LangChain 自研的轻量极简基础框架，以及在其之上演进出的 Deep Agents。Deep Agents 可以视作 LangChain 团队打造的“模型中立且更通用的 Claude Code 替代方案”——它不仅支持连接真实文件系统，还内建了技能库与子智能体编排机制。
这种高度定制化并未破坏核心循环，而是通过类似现代 Web 框架的“中间件（Middleware）”或“钩子（Hooks/Plugins）”机制来实现。开发者可以在几个关键切入点插入控制逻辑：
1. **智能体触发前（Before Agent Invocation）**：执行前置身份验证、环境初始化或安全拦截。
2. **模型调用前后（Wrap Model Calls）**：在上下文填入 Prompt 之前进行长度审查；若历史对话过长，立刻触发自动化上下文摘要（Summarization），压缩冗余数据后再送入模型。
3. **工具调用前后（Wrap Tool Calls）**：当遇到产生巨量文本或日志的工具时，通过“上下文卸载（Context Offloading）”机制将大块输出转存至外置存储或沙箱环境，仅将摘要或引用句柄暴露给模型，避免上下文窗口被无意义的日志冲垮。
借助中间件体系，系统在保持底层循环结构不变的前提下，能够自然挂载隔离沙箱、读写文件系统、派发任务给子智能体（Sub-agents），并精细控制长短期记忆，从而将通用架构打磨为高度适配垂直业务的专用执行引擎。
### [05:47](https://www.youtube.com/watch?v=HI2q3ci3Iuc&t=347s) 认知架构的退潮与保留：何时需要硬编码流程
回顾 2023 年至 2024 年初的智能体开发历程，Harrison 指出当时社区普遍热衷于设计极其复杂的定制化“认知架构（Cognitive Architectures）”。原因在于当时的底层模型能力有限，无法稳定地在开放循环中自主完成多步复杂推理；工程师不得不设计诸如 Deep Research 的子问题扇出与汇总图谱，或是代码审查机器人的固定状态机流转。
伴随前沿模型推理能力的跃迁，大部分硬编码流程已经被收敛进了带有中间件的通用 Harness 循环中。然而，对于某些业务逻辑极其严苛、容错率极低的场景，带有固定门禁（Gates）和确定性校验的认知架构仍然具有不可替代的价值。对此，Harrison 给出了务实的落地建议：
- **起步阶段**：优先选用通用 Harness 循环。这种方式能以最低的工程代价快速验证方案，最大化缩短业务价值交付周期（Time to Value）。
- **优化阶段**：当系统需要向极致的准确率收敛时，再针对具体的薄弱环节逐步引入确定的控制节点、校验逻辑和人工确认门禁。
### [07:03](https://www.youtube.com/watch?v=HI2q3ci3Iuc&t=423s) 自研还是开箱即用：分布内外的权衡与文件编辑案例
对于团队究竟应该“自研 Harness”还是“使用成熟开箱即用套件（Off-the-shelf）”，Harrison 给出了判断准则：**关键取决于任务场景与模型底层训练分布的重合度（In-distribution vs. Out-of-distribution）**。
像 Anthropic 的 Claude Code / Claude Agent SDK 或 OpenAI 的 Codex 体系，都是与各自专属模型深度绑定的开箱即用 Harness。如果业务诉求刚好落在模型原生训练的目标任务区间内（高度在分布内），直接采用官方 Harness 往往效果最顶尖；一旦任务开始脱离主流场景并进入垂直长尾领域，自研与深度微调 Harness 便是必经之路。
```mermaid
flowchart TD
accTitle: 场景分布性与 Harness 选择决策
accDescr: 根据业务任务以及子工具是分布内还是分布外，决定如何构建定制 Harness 与适配模型行为。
Task{"任务所处场景分布"}
Task -->|"高度在分布内 (如标准通用编码)"| OffShelf["直接使用官方开箱即用 Harness<br/>(如 Claude Code / Codex)"]
Task -->|"处于分布外 (如法律 AI 业务流)"| SubCheck{"具体子工具有无分布内能力？"}
SubCheck -->|"纯分布外逻辑"| CustomHarness["自建专有 Harness 编排与业务中间件"]
SubCheck -->|"底层包含基础操作 (如文件编辑)"| Profile["采用定制 Harness + 模型专属画像 (Model Profile)<br/>将底层工具调用对齐模型训练习惯"]
```
Harrison 以法律科技（Legal AI）为例展开论述。从宏观业务角度看，处理复杂的法律事务属于典型的模型训练“分布外”任务，因此团队必须打造一套自研 Harness 来约束法律逻辑。但在该业务链条中，诸如“文件局部编辑（File Editing）”这样的子操作，却又属于主流前沿模型经历了海量强化学习（RL）训练的“分布内”能力。
值得玩味的是，各家基础模型厂商在强化学习微调时，对文件编辑工具的调用范式存在微妙但关键的分歧。例如 OpenAI 与 Anthropic 的模型在各自官方 Harness 中编辑文件的方式并不相同，这也导致两家模型对特定的编辑指令协议存在天然的亲和度与性能依赖。如果开发者在自研 Harness 中自造一套全新的文件编辑工具协议，很可能会打破模型原生的条件反射。
针对这一工程痛点，LangChain 在 Deep Agents 中引入了**模型画像（Model Profiles）**概念：即便整体处于高度定制的业务 Harness 之中，系统也会针对具体绑定的底层模型，自动呈现该模型在强化学习训练期间所习惯的文件编辑工具定义与上下文交互接口，从而最大化榨取前沿模型的原生微调收益。
### [08:49](https://www.youtube.com/watch?v=HI2q3ci3Iuc&t=529s) 在分布内与分布外之间权衡：以文件编辑为例
定制化智能体（Agent） Harness 的核心原则在于：当任务偏离基础模型的能力分布（Out-of-Distribution, OOD）时，开发者应当对整体 Harness 进行深度定制；但对于那些模型在训练期间就已经熟练掌握的分布内（In-Distribution, ID）小组件，则应尽可能让实现紧贴模型层本身。以代码生成中的“文件编辑（Edit File）”功能为例，不同的基础模型在预训练时所习惯的文件修改交互范式截然不同——有的模型习惯于输出全量文件覆盖，有的擅长输出 diff 或特定占位符标记。因此，一个设计良好的 Harness 应当根据底层所调用的具体模型，灵活切换其内部的文件编辑实现，而不是用一套生硬统一的抽象强行套在所有模型之上。
- **分布外逻辑定制化**：当业务场景或流程编排与通用预训练分布差异巨大时，由上层 Harness 承载针对性的调度控制。
- **分布内逻辑原生化**：在细粒度原子操作（如编辑代码、运行特定命令）上，保持与所选模型的训练数据范式一致，最大化激发基础模型的能力。
---
### [09:06](https://www.youtube.com/watch?v=HI2q3ci3Iuc&t=546s) 评测驱动组织认知：为什么 Evals 与可观测性至关重要
在针对模型、Harness 或上下文（Context）这三大支柱进行调优时，开发者必须看清系统内部究竟在发生什么，并具备量化评估的能力。哈里森·蔡斯（Harrison Chase）引用了微软 CEO 萨提亚·纳德拉（Satya Nadella）两周前在推特上发表的三句精辟论断：第一，“打造你自己的私有评测集，因为 Eval 定义了组织内部何为优秀（eval defines what good looks like inside the organization）”；第二，“牢牢掌握企业自身的记忆、运行追踪（Traces）、反馈、决策以及机构化上下文”；第三，“构建你自己的持续学习闭环‘爬山机’（Hill Climbing Machine），让 AI 投资为企业持续复利增值”。这三点道破了可观测性、基准测试与持续优化闭环对构建专有智能壁垒的核心价值。
对于任何构建关键业务智能体的企业来说，搭建专业 Benchmark 都是必经之路。正如演讲嘉宾 Gabe 提到为法律垂类构建基准一样，团队不仅需要利用 Benchmark 建立防线、防止能力回归，更需要借助它持续迭代“爬山”——无论是改进调度逻辑还是微调基准模型。
- **定义评价标准**：私有 Eval 决定了企业内部对模型输出质量的共识与边界。
- **资产留存沉淀**：Traces、用户反馈与系统决策不仅是调试副产物，更是企业独有的机构化知识资产。
- **持续爬山机制**：通过可重复的基准测试，持续衡量 Harness 规则调整与模型微调带来的增量价值。
---
### [11:10](https://www.youtube.com/watch?v=HI2q3ci3Iuc&t=670s) 解构 Harbor：现代智能体评估基准的标准实现
在定义任务基准时，Harbor 正在成为业界通用的开源 Eval 运行器。它由知名终端评测基准 Terminal Bench 2 的创造者打造，并被广泛应用于 Frontier Bench 等前沿编程测评中。Harbor 的运行模型极为清晰：它调度一个由 Harness 和模型组成的智能体，针对包含大量任务的数据集并行跑测。
```mermaid
flowchart TD
accTitle: Harbor 评测任务执行流程
accDescr: 展示 Harbor 沙箱中包含的 Environment、Solution、Test 和 Instruction 结构
subgraph HarborTask["Harbor 评估任务包"]
Env["environment / Dockerfile<br/>(隔离容器沙箱环境)"]
Instr["instruction.md<br/>(给 Agent 的初始提示词)"]
Test["test / verifier<br/>(验证脚本: 单元测试 / Agent-as-a-Judge)"]
Sol["solution<br/>(黄金基准解 / 健全性检查)"]
end
AgentRunner["Agent 运行实例<br/>(Harness + Model)"] -->|"在沙箱中交互执行"| Env
Instr -->|"注入任务目标"| AgentRunner
AgentRunner -->|"生成运行结果"| Test
Sol -.->|"基准对齐校验"| Test
Test -->|"输出综合评分"| Report["评测报告与指标追踪"]
```
每个 Harbor 任务都被封装成一个独立的标准目录结构：
- **`environment`**：通过 Dockerfile 声明沙箱环境。长周期、多状态交互的智能体必须运行在独立隔离的环境中，以支持高并发横向扩展。
- **`instruction.md`**：提供给智能体的初始 Prompt，明确任务目标与边界。
- **`test`**：评测的核心验证器（Verifier）。它可以执行一段测试代码、跑单元测试用例，也可以调用另一个大模型乃至由更高级的智能体担任裁判（Agent-as-a-Judge）进行打分。
- **`solution`**：预先准备的“黄金标准解（Golden Solution）”，主要用于验证测试脚本本身是否存在逻辑缺陷（Sanity Check）。
在 LangSmith 与 Harbor 的集成视图中，每次实验不仅会聚合准确率奖励分（Reward Function），还会全量统计每次执行的端到端延迟（Latency）与 Token 消耗成本。在工程落地时，单纯追求高准确率往往不够，延迟与经济成本必须综合纳入权衡。
---
### [13:04](https://www.youtube.com/watch?v=HI2q3ci3Iuc&t=784s) 可观测性的盲区：上下文缺陷往往大于模型能力缺失
谈及可观测性，哈里森直言它听起来非常基础，但在智能体领域实际上被“严重低估（really underrated）”。当智能体在多步执行中崩溃或脱轨时，直接诱因固然是某一次具体的 LLM 调用失效，但根本原因通常只有两个：要么模型推理能力不足，要么上下文窗口中接收的信息存在缺陷。而在实际排错中，“绝大多数问题都出在后者”。
因此，能够穿透查看整个模型上下文窗口在运行过程中的动态演变，对于智能体调试至关重要——上下文是如何一层层累积起来的？中间调用了哪些工具？经历了哪些中间步骤？在展示 LangSmith 的界面时，哈里森介绍了两种呈现视角：一种类似于 Claude Code 风格的轨迹化展示（Trajectory），折叠了冗长复杂的工具调用（如折叠显示“7 次工具调用”），方便工程师按人类对话轨迹快速扫视；另一种则是全链路 Trace 视图，能够下钻到每一次底层 LLM 调用的全部原始 Payload、Token 明细与上下文快照，确保对异常归因无所遁形。
---
### [14:39](https://www.youtube.com/watch?v=HI2q3ci3Iuc&t=879s) 智能体数据飞轮：从链路追踪到持续自进化
在 AI 工程大会上，LangChain 团队提出了一套智能体持续改进的飞轮范式。从顶层看其流程极为简洁：构建智能体 $\rightarrow$ 部署运行并采集 Traces $\rightarrow$ 筛选清洗轨迹数据（Data Curation） $\rightarrow$ 在高价值数据集上跑实验迭代。
```mermaid
flowchart LR
accTitle: 智能体持续学习数据飞轮
accDescr: 从生产 Trace 到数据清洗，再反馈到 Harness、模型微调与上下文记忆的自演进闭环
subgraph Collection["运行与数据采集"]
AgentExec["Agent 生产运行"] -->|"全量 Trace"| TraceDB["Trace 存储与捕获"]
end
subgraph Curation["数据治理与反馈校验"]
TraceDB --> Filter["数据筛选 / 过滤"]
Feedback1["用户交互 UX 隐式反馈"] -.-> Filter
Feedback2["在线评估器 / SLM 裁判"] -.-> Filter
end
subgraph Optimization["系统全方位迭代"]
Filter --> Exp["离线基准测试与实验"]
Exp -->|"Harness 工程"| HEngine["编排调度 / 中间件改进"]
Exp -->|"轨迹微调"| MFinetune["模型层微调 (Fine-tuning)"]
Exp -->|"长期记忆维护"| ContextMem["上下文与提示工程"]
end
HEngine --> AgentExec
MFinetune --> AgentExec
ContextMem --> AgentExec
```
这个飞轮若要真正转动起来，关键在于获取高质量的反馈信号（Feedback）：
1. **环境与 UX 隐式反馈**：哈里森指出，智能体产品设计中最被轻视的环节是前端交互设计（UX Design）。“现实中根本没人会认真点赞或点踩”，但如果产品交互设计得足够巧妙（例如允许用户微调中间计划、修改输出内容或触发撤销），就能在自然交互中捕捉到极其宝贵的偏好与纠错数据。
2. **合成反馈与轻量化在线评估（Online Evaluators）**：直接在后台调用大参数量模型（如 Claude 3 Opus）对进来的每一条生产 Trace 做 LLM-as-a-Judge 审查，成本极其昂贵，账单会瞬间失控。业界可行的方式是通过定制 Prompt 使用廉价模型，或者将大模型裁判的能力蒸馏微调到专用的小语言模型（SLM）中，以此实现低成本、毫秒级的全天候自动打分；针对逻辑边界清晰的场景，甚至可直接使用确定性代码逻辑进行判定。
3. **系统三元组定向迭代**：筛选提炼出的核心失败案例（Hard Cases），可针对性地作用于智能体的三个不同层级——通过 Harness Engineering 调整路由策略与重试中间件；通过微调（Fine-tuning）强化底层模型的特定工具调用稳定性；或是通过改进 Memory 机制来补全上下文召回短板。
### [17:22](https://www.youtube.com/watch?v=HI2q3ci3Iuc&t=1042s) LangSmith Engine 演示：用 Agent 自动化数据飞轮
针对“人工审查 Trace 既棘手又耗时”的痛点，Harrison 现场演示了团队过去几个月探索的成果——**LangSmith Engine**。它的核心定位是一个直接架设在海量 Trace（调用追踪）之上的 Agent，专门负责自动化执行前述的数据飞轮工作：自动清洗整理 Trace、运行实验，并针对 Context（上下文）、模型或 Harness（执行底座）提出具体的修复建议，而 LangChain 现阶段则重点聚焦在 Harness 工程的优化上。
在现场屏幕上，Harrison 展示了 LangSmith 中不断涌入的 Trace 流，以及名为“Engine”的独立标签页。在底层，Engine 是一个具备 LangSmith CLI（命令行工具）调用权限的 Coding Agent。用户可以通过 CLI 对 Trace 进行带有反馈（Feedback）等维度的深度过滤；开发团队为 Engine 配备了详尽的 Prompt，并设计了数个专门辅助其探索数据的 Sub-agents（子智能体）。Engine 在后台自主漫游、聚类并定位共性问题后，会自动生成类似看板的“Issue Boards”。Harrison 随机点开一个生成的 Issue：卡片中不仅有详细的故障描述，还附带了直达原始 Trace 的链接作为佐证。更关键的是，Engine 直接给出了修改方案——不仅能够更新 Prompt 中的上下文和指令，甚至能自动生成需要合并进 Harness 的代码补丁。Harrison 总结道，这套流程生动展现了团队对自动化闭环的追求：“运行 Agent，获取 Trace，发现规律，实施修复”（"Run agent, get traces, see patterns, fix."）。
```mermaid
flowchart TD
accTitle: LangSmith Engine 自动化飞轮架构
accDescr: 展示 LangSmith Engine 如何通过 CLI 和子智能体分析 Trace，最终自动输出 Issue 及修复补丁。
subgraph TraceLayer["数据观测层"]
T["生产环境 Trace 流"]
CLI["LangSmith CLI 接口<br/>(支持标签与反馈过滤)"]
T --> CLI
end
subgraph EngineCore["LangSmith Engine (Coding Agent)"]
P["主 Prompt 指令"]
SA["探索与模式识别 Sub-agents"]
CLI --> SA
P --> SA
end
subgraph OutputLayer["自动化产出看板 (Issue Board)"]
ISSUE["生成 Issue<br/>(挂载溯源证据 Trace)"]
FIX1["更新 Context 与 Prompt 指令"]
FIX2["输出 Harness 核心代码补丁"]
SA --> ISSUE
ISSUE --> FIX1
ISSUE --> FIX2
end
```
- **底层机制**：Engine 本质是一个 Coding Agent，通过 LangSmith CLI 检索、过滤包含用户反馈的 Trace 数据。
- **协同架构**：依托完备的提示词与多 Sub-agent 机制在海量 Trace 中自主挖掘模式。
- **交付成果**：自动生成问题看板（Issue Board），提供追溯证据链接，并直接给出上下文微调建议及 Harness 底层代码修复。
---
### [19:19](https://www.youtube.com/watch?v=HI2q3ci3Iuc&t=1159s) 问答环节：用 Engine 递归评估自身与 Engine 的“Codex 化”
在问答开场，现场观众敏锐地指出：既然 Engine 本质上也是一个接收 Prompt 并在数据上搜索的 Agent，团队是否尝试过“用 Engine 来跑 Engine 本身”（递归评估/吃狗粮）？Harrison 笑着确认，他们已经在这么做了：Engine 已经接入了 Slack，并且每天都会在频道里自动推送关于它自身运行状况的诊断报告，团队正是借此完成内部狗粮测试（Dogfooding）。
为了持续验证和演进 Engine，团队还基于 Harbor 规范构建了一个名为 **IssueBench** 的基准评测集，用以横向对比不同模型与不同 Harness 在该任务上的表现。Harrison 强调，拥有独立 Benchmark 的最大价值在于可以把同一任务丢给多个业界 Harness，直观观察各家的优劣势。例如几周前，他们将自己的 Deep Agents、Codex 以及 Claude Code 放在 IssueBench 上同台竞技。测试中他们捕捉到了 Codex 一个非常亮眼且激进的行为模式：面对复杂的 Trace 数据，Codex 会自主编写大量精简的小脚本来批量检索与清洗数据，这一策略让它的表现格外优异。LangChain 团队受到启发后立刻发起了一轮冲刺，对 Engine 进行了所谓的“Codex 化”（"Codex-ification"），将这种动态自写脚本的技术直接反哺并固化进 Engine 自身的核心 Harness 中。Harrison 直言，这正是评测集最迷人的地方：让不同架构互相碰撞，再把涌现出的最佳实践抽离融合回自己的核心 Harness 之中。
- **自举迭代（Dogfooding）**：Engine 接入企业 Slack，定期输出分析自身的自省报告。
- **基准评测集 IssueBench**：采用 Harbor 统一规范，实现模型与 Harness 的解耦评测。
- **Codex 化（Codex-ification）**：复刻 Codex 遇到复杂数据处理时“自写自跑临时脚本”的行为范式，显著提升了 Engine 的 Harness 能力。
---
### [20:44](https://www.youtube.com/watch?v=HI2q3ci3Iuc&t=1244s) 问答环节：Agent Harness 会趋同收敛还是分化发展？
观众随后提出了一个行业普遍关心的核心问题：业界的 Harness 最终是会收敛为一套大一统的标准方案，还是会因各家企业业务场景的不同而走向高度分化？Harrison 坦承这个问题自己和 Factory 的 Eno 也深入讨论过，最老实的回答是“目前谁也无法确定”（"I don't know is the honest answer"），但可以从几个客观发展维度来观察：
一方面，通用开箱即用型（Off-the-shelf）Harness（如 Deep Agents、Codex 或 Claude Code）已经足够成熟，足以应对冷启动或大多数标准化任务。这得益于底层大模型能力的飞跃，以及行业在核心模式上达成的认知共识——例如赋予 Agent 文件系统权限、Sub-agents 拆分机制等。但另一方面，任务越偏离常规分布（Out-of-distribution），定制 Harness 的需求就越强烈。这不仅仅取决于任务逻辑本身，还深受“确定性与控制力”（Predictability and Control）诉求的影响。Harrison 分享了一个生动的案例：当他们把自主性极高的 Deep Agents 展示给金融服务领域的客户时，对方的第一反应往往是恐慌：“等等，这对我们来说太危险了！我们需要的是定制的认知架构（Cognitive Architecture），让我们能牢牢把控每一步。”因此，市场上必然存在一条光谱：一端是高度自由通用的开源 Harness，另一端是步步可控的定制认知架构，而中间层则由 Hook 和中间件（Middleware）予以衔接。
谈及模型厂商本身的演进，Harrison 指出了一种耐人寻味的分歧现象：虽然 OpenAI 和 Anthropic 的模型在编程代码领域都极度强悍，但两家在底层设计文件编辑（File Editing）机制时却选择了截然不同的实现路径，且各自在 Benchmark 上证明自己的方法“绝对优于对方”。目前这种微小的技术分歧主要体现在代码编辑等细节上，但设想未来如果某家模型实验室全面押注生物计算（Bio）等全新纵深领域，其匹配的专用 Harness 势必会与通用编程 Harness 大相径庭，从而引发 Harness 的大幅分化。Harrison 总结道，正因为整个 Agent 领域瞬息万变且路径未定，建立严谨的评测（Evals）和可观测性（Observability）才显得至关重要，只有用数据持续衡量，才能在不断分化与演进的技术浪潮中看清最优路径。
- **控制力与自由度的权衡**：金融等严监管行业宁愿牺牲自主性也要选择定制认知架构，拒绝“过于不可控”的黑盒 Agent。
- **头部厂商的技术分歧**：在基础的代码文件修改机制上，OpenAI 与 Anthropic 的工程范式已呈现出各执一词的分化。
- **垂类领域的潜在爆发**：未来若出现针对生物医药等特殊领域的专用基座模型，将彻底打破单一通用 Harness 的收敛假设。
## Notable quotes
> [02:12](https://www.youtube.com/watch?v=HI2q3ci3Iuc&t=132s) Harness 的核心任务就是在最精准的时间点，将最恰当的上下文输送给模型 — Harrison Chase
> [02:12](https://www.youtube.com/watch?v=HI2q3ci3Iuc&t=132s) 就是一个大模型置身于一个循环之中，不断进行工具调用（an LLM running in a loop calling tools） — Harrison Chase
> [09:06](https://www.youtube.com/watch?v=HI2q3ci3Iuc&t=546s) 打造你自己的私有评测集，因为 Eval 定义了组织内部何为优秀（eval defines what good looks like inside the organization） — Satya Nadella
> [09:06](https://www.youtube.com/watch?v=HI2q3ci3Iuc&t=546s) 牢牢掌握企业自身的记忆、运行追踪（Traces）、反馈、决策以及机构化上下文 — Satya Nadella
> [09:06](https://www.youtube.com/watch?v=HI2q3ci3Iuc&t=546s) 构建你自己的持续学习闭环‘爬山机’（Hill Climbing Machine），让 AI 投资为企业持续复利增值 — Satya Nadella
> [13:04](https://www.youtube.com/watch?v=HI2q3ci3Iuc&t=784s) really underrated — Harrison Chase
> [14:39](https://www.youtube.com/watch?v=HI2q3ci3Iuc&t=879s) 现实中根本没人会认真点赞或点踩 — Harrison Chase
> [17:22](https://www.youtube.com/watch?v=HI2q3ci3Iuc&t=1042s) Run agent, get traces, see patterns, fix. — Harrison Chase
> [20:44](https://www.youtube.com/watch?v=HI2q3ci3Iuc&t=1244s) I don't know is the honest answer — Harrison Chase
> [20:44](https://www.youtube.com/watch?v=HI2q3ci3Iuc&t=1244s) 等等，这对我们来说太危险了！我们需要的是定制的认知架构（Cognitive Architecture），让我们能牢牢把控每一步。 — Harrison Chase
## Mentioned
Harrison Chase, LangChain, Sequoia Capital, Satya Nadella, Lin, Fireworks AI, Gabe, Eno, Factory, Deep Agents, Claude Code, Claude Agent SDK, Anthropic, OpenAI, Codex, Harbor, Terminal Bench 2, Frontier Bench, LangSmith, LangSmith Engine, LangSmith CLI, IssueBench, Slack, Docker
## Video description
An agent is a harness orchestrating a model and context. If you want to own your intelligence, you probably want to own all three. LangChain co-founder and CEO Harrison Chase focuses on the harness: the loop that brings the right context to the model at the right time, and how to customize it with middleware, hooks, and sub-agents.
At Sequoia Capital's Own Your IntelligenceI event, Harrison explains when an off-the-shelf harness is enough and when to build your own (the more out-of-distribution your task, the more customization you'll want) and why even custom harnesses should keep in-distribution pieces, like file editing, close to how each model was trained. He also covers building benchmarks, why observability is underrated for debugging agents, and the data flywheel that turns production traces into harness, model, and context improvements.
00:00 Introduction
00:58 The three parts of an agent: harness, model, context
02:12 What a harness actually does
03:25 Customizing the core loop with middleware
04:41 Sandboxes, file systems, sub-agents, summarization
05:47 Cognitive architectures — and when you still need them
07:03 Build your own harness or use off the shelf?
08:24 In-distribution vs. out-of-distribution: the file-editing example
09:39 Why evals define what "good" means in an organization
11:04 Harbor: what an eval task actually looks like
12:11 Comparing harnesses and models on accuracy, latency, and cost
13:20 Why observability is underrated — it's usually the context
14:34 The data flywheel: traces → curation → experiments
15:42 Getting feedback through UX design and online evaluators
16:51 Demo: LangSmith Engine
19:23 Q&A: Running Engine on Engine, and "codex-ification"
20:44 Q&A: Will harnesses converge or diverge?
[Source](https://www.youtube.com/watch?v=HI2q3ci3Iuc)
Source quotes
[2:12] Harrison Chase: Harness 的核心任务就是在最精准的时间点，将最恰当的上下文输送给模型
[2:12] Harrison Chase: 就是一个大模型置身于一个循环之中，不断进行工具调用（an LLM running in a loop calling tools）
[9:06] Satya Nadella: 打造你自己的私有评测集，因为 Eval 定义了组织内部何为优秀（eval defines what good looks like inside the organization）
[9:06] Satya Nadella: 牢牢掌握企业自身的记忆、运行追踪（Traces）、反馈、决策以及机构化上下文
[9:06] Satya Nadella: 构建你自己的持续学习闭环‘爬山机’（Hill Climbing Machine），让 AI 投资为企业持续复利增值
[13:04] Harrison Chase: really underrated
[14:39] Harrison Chase: 现实中根本没人会认真点赞或点踩
[17:22] Harrison Chase: Run agent, get traces, see patterns, fix.
[20:44] Harrison Chase: I don't know is the honest answer
[20:44] Harrison Chase: 等等，这对我们来说太危险了！我们需要的是定制的认知架构（Cognitive Architecture），让我们能牢牢把控每一步。
Source quotes
[19:23] Harrison Chase: Codex was doing a really interesting thing, where it would write itself a bunch of small scripts to run against these traces... so we did a sprint to do what we call the 'codex-ification' of Engine and basically take that learning and bring it into the core Engine harness.
[20:44] Harrison Chase: The more out of distribution you get, the more you're going to want to customize the harness. It's a scale... At the extreme end, you might want to build a complete cognitive architecture... for predictability and control.
Original transcript
[0:00] Harnesses. I think this is a very
[0:02] important topic. A lot of you are
[0:03] thinking through building your own
[0:04] harnesses right now. Um, I'm very
[0:06] excited to introduce Harrison. I first
[0:09] noticed Harrison on Twitter in 2022,
[0:12] back in the GPT-3 era. And Harrison was
[0:15] one of the first people thinking about,
[0:17] "Okay, we have these models. How can we
[0:19] build an entire harness around them so
[0:21] that they're not just um auto-complete
[0:24] uh tasks, but that they start acting as
[0:27] virtual collaborators or agents?" Um and
[0:30] Harrison, like the ecosystem has grown
[0:32] so much since 2022, and I've seen you
[0:35] you also grow a lot in terms of how you
[0:36] think about building agents, building
[0:38] harnesses, how to eval them, etc. Um so,
[0:41] I'm very excited to have you talk today.
[0:43] I think the talk's going to be both
[0:44] about harnesses and evals. And then
[0:46] format again will be 15 minutes or so of
[0:48] presentation content, uh 15 minutes of
[0:51] Q&A. Thanks for joining us, Harrison.
[0:57] >> Cool. Um my name's Harrison, co-founder
[1:00] CEO of LangChain. I want to talk about
[1:03] evals and harnesses in the context of
[1:05] kind of owning your own intelligence.
[1:09] So, when we talk about intelligence,
[1:10] we're normally talking about agents.
[1:12] What exactly makes up an agent? At
[1:14] LangChain, we think there's kind of like
[1:15] three main parts. There's a harness that
[1:17] orchestrates a model and some context.
[1:19] And if you're talking about owning your
[1:20] intelligence in general, you probably
[1:22] want to own all three parts of these.
[1:24] And so, owning the model, I'm not going
[1:26] to talk too much about. Lin was here
[1:27] from Fireworks and talking about open
[1:29] weight models and owning that. Uh big
[1:31] part of this is also the ability to
[1:32] switch models. Uh
[1:34] there used to be this concept of kind of
[1:36] being like cloud agnostic and being able
[1:37] to switch clouds uh back in the day.
[1:40] Same thing exists, but for models. You
[1:41] want to be able to switch to avoid
[1:42] lock-in, but also to just use the best
[1:44] model when it's available. Context, you
[1:46] want to own all the the context that
[1:49] your agent uses. Whether that is memory,
[1:52] uh whether that is semantic knowledge,
[1:54] um whether that is previous
[1:55] conversations. You can these can help
[1:56] personalize and guide the agent as it
[1:58] goes along. And then the last bit is the
[2:00] harness, and that's what I really want
[2:01] to focus on.
[2:02] So, how how do you how do you really own
[2:04] your harness? What does that even mean?
[2:06] What's the main job of a harness? The
[2:07] main job of a harness is to bring
[2:08] context to the model at the right point
[2:11] in time.
[2:12] And so, it does all the orchestration
[2:13] around the the fixed context, the the
[2:16] dynamic context. It brings it into the
[2:18] context window of the model, shows it
[2:20] something, gets some response, and then
[2:22] does something with that.
[2:24] And so, agents need to do all these
[2:27] different things in order to accomplish
[2:28] their jobs. There's a ton of
[2:30] domain-specific stuff that they need to
[2:31] do as well, but they need to interact
[2:33] with external systems. These external
[2:34] systems, when you interact with them,
[2:36] they emit more context that can get fed
[2:37] back into the agent into the loop. And
[2:39] so, the harness is the thing that really
[2:40] orchestrates all of this together.
[2:43] Agents at [snorts] their their kind of
[2:45] like simplest, when everyone talks about
[2:47] agents, what they really talk about is
[2:48] just an LLM running in a loop calling
[2:50] tools.
[2:51] Um and uh this this is a really simple,
[2:53] but really general architecture. Some
[2:55] request comes in, the the LLM makes some
[2:59] uh generation. That generation may
[3:01] include a tool to call. If it does, you
[3:03] invoke those tools, and you pass that
[3:05] observation back to the LLM. And and
[3:06] this is this is the core architecture
[3:08] behind pretty much every agent out there
[3:10] today.
[3:12] But they're all different in like
[3:13] slightly different ways. And so, on the
[3:15] left here, this is kind of like uh the
[3:16] the the base core kind of like loop. But
[3:19] there's a bunch of different things that
[3:21] you can do in your particular harness at
[3:24] different stages.
[3:25] And so, this is uh over here, this is
[3:28] So, so we built LangChain, which is a
[3:29] really really base minimal harness, and
[3:31] that's LangChain over here. And then
[3:33] this is Deep Agents. Deep Agents is kind
[3:34] of like our model-agnostic and and more
[3:37] general-purpose version of of Quad Code.
[3:39] And so, it does more things. It connects
[3:40] to file systems. Uh it has skills. It
[3:43] has sub-agents.
[3:44] It's built on top of this really simple
[3:46] harness, but we customize it by using
[3:48] these uh these levers over here. So, you
[3:50] can run particular code snippets before
[3:53] the agent's invoked, before each model
[3:55] call. You can kind of like wrap these
[3:57] model calls, you can wrap the tool
[3:59] calls, and you can and you can customize
[4:01] this core uh simple loop in a lot of
[4:04] really powerful ways just by just by
[4:07] using kind of like small what we call
[4:09] kind of like middleware constructs.
[4:10] There's other ways to customize the
[4:12] harness as well, but this is kind of
[4:14] emerged as uh there's a concept of hooks
[4:16] and plugins in a lot of the coding
[4:17] agents as well, and that's essentially
[4:19] what they're doing. They're taking this
[4:20] base loop that's running, and they're
[4:22] adding little hooks or plugins at
[4:23] various points to let you customize it.
[4:26] And so, a lot of the stuff that you can
[4:28] do when you can customize, this is all
[4:30] done by that concept of middleware, by
[4:32] just modifying that core loop. So, the
[4:34] agent's still running in a loop, it's
[4:36] still doing that same simple
[4:37] architecture, but through that you can
[4:39] give it access to a sandbox, you can
[4:41] give it access to a file system, you can
[4:43] give it access to sub-agents, you can
[4:45] give it access to to memory, you can
[4:47] have summarizations. So, summarization,
[4:49] if if if we go back to this thing,
[4:51] summarization would come in before the
[4:52] model. Before the model's invoked, you
[4:54] check if the context is too long, and
[4:55] then you summarize it. And so, that that
[4:57] you can add into this core loop through
[4:59] this concept of middleware, same with
[5:01] context offloading,
[5:03] um which which is a way of basically
[5:04] taking large tool calls and dumping
[5:06] them. That kind of wraps the tool call.
[5:08] And so, the point is there's there's
[5:10] this really simple kind of like general
[5:12] architecture of an agent. All of these
[5:14] more advanced agent harnesses are
[5:16] basically doing that loop, but adding in
[5:18] a bunch of stuff while still running the
[5:20] core loop. And so, as you think about
[5:21] kind of like building or customizing
[5:24] your own harness,
[5:26] these are the different places that you
[5:27] can insert things into. You can add your
[5:29] own summarization step, you can add your
[5:31] own handling of particular tool calls,
[5:33] and that's one way that you can
[5:34] customize kind of like the agent to your
[5:37] particular domain and the harness to
[5:39] your particular domain.
[5:41] The other way that you can customize the
[5:42] harnesses that the agent runs in is by
[5:44] having a more explicit kind of like
[5:46] cognitive
[5:47] So, this used to be the way that a lot
[5:50] of people would build agents in kind of
[5:51] 2023, 2024 because the models weren't
[5:54] good enough to run in a loop. And so, in
[5:56] order to get it to do particular things,
[5:58] you would have these very bespoke
[6:00] cognitive architectures. And so, this
[6:02] one over here is
[6:04] for
[6:05] a deep research example
[6:07] where it would generate some some sub
[6:09] questions, fan them out, and then go and
[6:11] execute them. And then this one over
[6:12] here is for a code review bot. And you
[6:14] can see that there's these very kind of
[6:15] like the bespoke steps.
[6:17] A lot of this has gone into the harness
[6:21] now. And by the harness, I mean it's
[6:22] still this core loop. These might be
[6:24] added as particular kind of like
[6:25] modifications to that core loop. But,
[6:28] for a lot of really particular kind of
[6:30] like flows, we do see people still using
[6:34] cognitive architectures like these to
[6:35] really guide it in particular ways. One
[6:37] thing that we recommend to people is to
[6:39] start with a general harness.
[6:41] That's the easiest to get started. It's
[6:42] going to be quickest to time to value.
[6:44] And then as you kind of narrow in on the
[6:47] use case that you want to be excellent
[6:48] at, you can start to add more of these
[6:50] kind of like gates and checks around it
[6:52] to to to guide it into particular ways.
[6:55] One question that we get a lot is when
[6:58] to kind of think about building your own
[7:00] harness versus using an off-the-shelf
[7:01] harness. Um a lot of the off-the-shelf
[7:04] harnesses are uh work with particular
[7:06] models. So, the off-the-shelf harnesses
[7:09] include things like Claude Code or
[7:11] Claude Agent SDK, which works with
[7:12] Anthropic models, Codex, which works
[7:14] with OpenAI models. I think this is a
[7:16] big open question in the industry. My
[7:19] answer generally is the more in
[7:21] distribution you are of what the models
[7:23] are trained on, then the better the
[7:25] off-the-shelf harness will be. As soon
[7:27] as you start to move further and further
[7:28] out of distribution, then the then
[7:31] you'll probably want to tune your
[7:32] harness in some way.
[7:34] There's different ways to tune the
[7:35] harness as well. So, the models may be
[7:37] in distribution on particular things
[7:40] that you are doing on an out of
[7:41] distribution task. So, what I mean by
[7:43] that is if you think about something
[7:46] like like legal AI, which Gabe just
[7:48] talked about, and I think he mentioned
[7:49] how they have their own harness,
[7:51] there are things that are in legal AI
[7:53] that are still in distribution of the
[7:56] the the main models. So, for example,
[7:58] editing files is something that the main
[8:01] models have all been RL'd on. And
[8:03] they've actually all been RL'd in very
[8:04] particular ways. So, OpenAI and and
[8:07] Claude models edit files in in different
[8:10] ways in their harnesses, and as a
[8:11] result, their models are actually best
[8:13] at editing files in different ways.
[8:16] Now, the the models themselves are out
[8:18] of distribution on this larger task of
[8:20] legal AI, but they're in distribution on
[8:22] this task of editing files. So, if you
[8:24] think about building a harness that
[8:26] works there, you'll probably want a
[8:28] custom harness, but you'll want it to
[8:30] use the edit file tool that is in
[8:32] distribution for
[8:35] the the the model that you're using. So,
[8:37] one of the things we do in Deep Agents,
[8:39] for example, so Deep Agents is our
[8:40] customizable harness, we actually have
[8:42] this concept of model profiles, where
[8:44] for things that are in distribution of
[8:46] models like editing files, we basically
[8:49] switch between different edit file
[8:51] implementations depending on which
[8:52] model's being used. And so, I think
[8:54] that's an example of customizing the
[8:56] overall harness when it's out of
[8:57] distribution for a task, but keeping
[8:59] smaller in distribution parts
[9:02] as close to the model layer as as
[9:04] possible.
[9:06] The second big part of what I want to
[9:08] talk about is evals and observability.
[9:10] And so, I think as you're experimenting
[9:11] with all parts of an agent, whether it's
[9:14] the model or the harness or the context,
[9:16] you're going to want to know what's
[9:17] going on inside of this system, and
[9:20] you're going to want to be able to
[9:21] evaluate it. And so, these are useful
[9:23] tools that you can use, again, not just
[9:25] for custom harnesses, but also for
[9:27] custom models.
[9:29] So, there was a great Twitter article
[9:30] that Satya wrote uh
[9:33] 2 weeks ago,
[9:34] um where he talked about a lot of these
[9:35] concepts. And there's three quotes in
[9:37] particular that kind of stood out for
[9:38] me. One, create your private evals
[9:40] because eval defines what good looks
[9:42] like inside the organization. Two,
[9:45] retain ownership of your organization's
[9:47] memory, traces, feedback, though that
[9:49] bold is mine, decisions, and
[9:51] institutional context. And then three,
[9:53] you create your own continuous learning
[9:54] loop hill climbing machine that will
[9:56] allow your AI investments to compound
[9:58] the value of your firm. And so I think
[10:00] these speak to the importance of evals
[10:03] and observability and the learning loop
[10:05] that they power in in really owning your
[10:07] intelligence and compounding it. So how
[10:10] exactly do they do that?
[10:11] So evals. Gabe was here talking about
[10:14] how they built benchmarks for the legal
[10:17] domain. I think every company what when
[10:19] they're building a mission-critical
[10:20] agent, they will build benchmarks for
[10:22] that agent.
[10:23] Um you can use it to define and catch
[10:25] regressions or you can hill climb on
[10:27] that benchmark. Again, either by
[10:28] adjusting the harness or adjusting the
[10:30] model.
[10:31] Things that we see becoming the industry
[10:33] standard for defining these benchmarks
[10:35] is Harbor. Harbor is an open-source eval
[10:37] runner. It's uh created by the makers of
[10:40] Terminal Bench 2, which is one of the
[10:42] industry standard benchmarks for
[10:43] benchmarking coding agents, and it's
[10:45] become pretty popular for a variety of
[10:46] domains.
[10:48] What it lets you do, so this is so this
[10:49] is Frontier Bench, which is another uh
[10:51] coding benchmark. You get this nice
[10:53] benchmark and you can compare different
[10:56] agent harnesses, different models,
[10:58] different reasoning efforts, and you can
[11:00] get this nice benchmark and you can see
[11:01] how all these different harnesses and
[11:03] all these different models do on your
[11:04] task. And so having a benchmark for your
[11:06] task will become really really important
[11:08] when you're trying to define it.
[11:10] What exactly is Harbor? It's pretty
[11:12] simple. At a high level, it consists of
[11:15] agent that you you run an agent against
[11:17] the data set. A data set has a bunch of
[11:19] different tasks. Generally, they're run
[11:21] in sandboxes because they are a lot of
[11:24] these different tasks and you might want
[11:25] to parallelize them. And as we talk as I
[11:28] talk about in a little bit, each task
[11:30] has its own kind of like environment. So
[11:31] this is what a Harbor task looks like.
[11:33] So, on the right, you can see that it
[11:35] has an environment. This is where you
[11:36] define the environment that the agent
[11:38] runs in. A lot of these longer running,
[11:39] more stateful agents need to interact
[11:41] with their environment. And so, you
[11:43] basically spin up a sandbox, give it its
[11:45] own environment that's defined in a
[11:46] Dockerfile, and run it there.
[11:48] There's then uh a solution, which is
[11:50] basically this this is uh a kind of like
[11:52] golden solution that you use to sanity
[11:54] check it, so it's not that interesting.
[11:55] Test is more interesting. This is
[11:56] basically the verifier for the the agent
[11:59] run.
[12:01] The test scripts can do anything. They
[12:02] can run code, they can run unit tests,
[12:05] they can run another LLM as a judge,
[12:06] they can run an agent as a judge. You
[12:08] basically define how the agent is scored
[12:10] in this test. And then instruction.md is
[12:12] the prompt that the agent is given. And
[12:15] that's kind of like the core of Harbor.
[12:17] You define these tasks, which are
[12:19] bundled up things that can be run in a
[12:20] sandbox, and then you run a bunch of
[12:21] them against agents. And agents again
[12:24] consist of models and harnesses, and you
[12:25] score how how they do.
[12:29] When you do all of that, what do you
[12:30] get? You get some nice results that you
[12:31] can compare. So, this is LangSmith, the
[12:33] the platform that we build for evals and
[12:35] observability. And so, you can see here
[12:37] a bunch of different experiments. Uh we
[12:39] have a great integration with Harbor.
[12:40] You can see the the feedback scores. In
[12:42] this case, it's a single reward
[12:44] function. You can also track latency and
[12:45] tokens. So, when you're benchmarking
[12:47] agents, you probably don't just care
[12:49] about accuracy. You also probably care
[12:51] about latency and and and cost. And so,
[12:53] you'll want to track all of those.
[12:55] And then for a particular experiment
[12:57] that you run, these are these would be
[12:58] the different tasks that are in a in a
[13:00] in a Harbor data set.
[13:04] Talking about a little bit about
[13:05] observability. Uh observability sounds
[13:07] basic, but I think it's really important
[13:09] and really underrated for agents,
[13:10] actually. So, when agents mess up, they
[13:12] mess up because an LLM call goes wrong.
[13:13] Why might it go wrong? It might go wrong
[13:15] for one of two reasons. One, the model
[13:17] is not good enough. Two, the context
[13:18] that the LLM received isn't good enough.
[13:20] And so, I actually think it's the the
[13:21] second one that more often than not
[13:23] causes issues. And so, having really
[13:25] good observability into what is going
[13:27] into the context window of the the model
[13:30] and then how that context is
[13:32] accumulated, what steps were run, what
[13:34] tools were run, how does that context
[13:36] get there? All of that is really
[13:37] important for debugging your agent when
[13:40] it when it goes wrong.
[13:43] So this is one view of observability
[13:45] that we have. This is intended to be a
[13:46] more kind of like user-friendly view
[13:49] where we actually represent it. This is
[13:50] similar to what you might see in kind of
[13:52] like Claude code. We we kind of like
[13:54] hide some of the tool calls so you can
[13:55] see seven tool calls up there.
[13:57] And so we try to make it really easy to
[13:59] kind of like skim through this. Most
[14:01] agent paths these days come in the form
[14:03] of trajectories. Trajectories are
[14:05] basically you can think of them as the
[14:06] list of messages that you see kind of
[14:08] like Claude code running. So when you
[14:10] run Claude code or another agent, you
[14:12] type in a human message, it then makes a
[14:14] bunch of tool calls. Those are all
[14:15] messages under the hood and then it
[14:17] responds and then you type in another
[14:18] human message. That's kind of like this
[14:20] message trajectory that is becoming more
[14:22] and more of a central part of of these
[14:24] agents. But that's not enough to fully
[14:26] debug it and so we also have this full
[14:28] kind of like trace and you can click
[14:30] into particular things and see exactly
[14:32] what goes on inside the model. And this
[14:34] type of observability is pretty
[14:35] important for knowing what's going on.
[14:39] Evals and observability really let you
[14:40] set up this data flywheel and compound
[14:43] the intelligence as you as you start to
[14:46] use the agent, as your users start to
[14:48] use the agent and you start to get
[14:49] feedback.
[14:50] So this is this is a slide that one of
[14:53] our team members presented at Swyx's AI
[14:56] engineering fair actually around a
[14:58] recipe for continuously improving
[15:00] agents.
[15:01] At a high-level it's really simple. You
[15:03] build an agent, you start running it,
[15:04] you collect lots of traces, you then
[15:06] curate the trace data, and then you run
[15:08] experiments on on that data that you
[15:10] create. And so it's really simple,
[15:12] but of course there's a lot of
[15:13] complexity under the hood. So so one
[15:16] thing that's really important for this
[15:17] is is feedback. Getting feedback either
[15:20] from the environment or from synthetic
[15:22] source. So from the environment, one
[15:24] thing that I think is really
[15:25] underestimated in agent design is
[15:26] actually UX design of how you present
[15:28] the agent to your users. If you present
[15:30] it in a really intelligent way, you can
[15:31] actually end up getting a lot of
[15:33] feedback from them. They may not click
[15:34] thumbs up or thumbs down explicitly. No
[15:36] one really doing that. But if you if you
[15:39] design the UX in a clever way, you can
[15:40] get some of that feedback. The other
[15:42] thing you can do is you can start to get
[15:45] synthetic feedback. So you can run what
[15:47] we call kind of like online evaluators
[15:49] over these traces to judge things. So
[15:52] Gabe was talking about an experiment
[15:53] that we did with Harvey where we we
[15:55] significantly reduced the cost of some
[15:58] of these LLM as a judge type thing. So
[16:00] if you imagine running Opus over every
[16:02] single trace that comes into your
[16:04] system, that's going to rack up a big
[16:05] bill. And so you want a really cheap and
[16:07] fast way of doing this. So we've
[16:08] fine-tuned some SLMs for actually doing
[16:10] this, but you can of course use
[16:11] off-the-shelf models with custom
[16:13] prompting to do it. Or you can just use
[16:14] code if some of the things that you want
[16:16] to test are are simple enough.
[16:20] Um so this is the the the full a part of
[16:22] it. Um the curating the trace data,
[16:25] feedback is a big part there. And then
[16:27] the other thing is is when you use that
[16:29] data to update what happens, you can
[16:32] update any part of the agent with this
[16:34] with this kind of like system. Um so you
[16:36] can update the harness by doing harness
[16:39] engineering. You can update the model by
[16:41] do a by doing fine-tuning on that. You
[16:42] can update the context by doing memory.
[16:45] Um and and so the part that we are that
[16:47] we think most about at LangChain is the
[16:50] harness engineering part of that. And so
[16:51] I want to show a really quick demo of of
[16:53] one of the things that we added to help
[16:54] with that. But I I think Trajectory is
[16:57] talking next on some fine-tuning that
[16:59] can be done. Uh and and so it's a very
[17:01] similar process where you run the agent,
[17:02] get some traces, use that data in some
[17:05] way to improve the system. What is the
[17:06] system? It's these three pieces. Any of
[17:08] these can be updated in some way.
[17:11] Um and so yeah, this is the full
[17:13] end-to-end um
[17:15] uh uh flow that you might want to do.
[17:18] One of the things that we think about is
[17:20] how can you automate this as much as
[17:22] possible because this is tricky and
[17:24] takes a lot of time.
[17:25] Um and so that's one of the things that
[17:27] we've been thinking about for the past
[17:28] few months. I want to do a quick demo of
[17:30] what we call LangSmith Engine, uh which
[17:33] is basically an agent that sits on top
[17:35] of your traces and does all this work.
[17:37] So if we look at what that work was, you
[17:39] know, you've got these traces, it's it
[17:41] the work from there is curating the
[17:43] traces and running some experiments,
[17:45] suggesting fixes to one of the three
[17:46] things. And as I mentioned, we mostly
[17:47] focus on the harness engineering bit. Um
[17:50] so uh in the demo I want to show uh what
[17:54] this looks like and how it represents
[17:55] that. So hopefully this will work. If
[17:58] not, it's not that big of a deal.
[17:59] Perfect. Okay. So this is LangSmith.
[18:01] This is a bunch of traces we have coming
[18:03] in. Um we have this tab called engine
[18:05] over here. Um this is an agent. It runs
[18:08] in the background. It creates what we
[18:10] call kind of like issue boards. So this
[18:11] is the part of curating data. It will
[18:13] look for It will It will It will
[18:14] basically under the hood is a coding
[18:16] agent that has access to our LangSmith
[18:17] CLI. The LangSmith CLI, you can filter
[18:20] traces for feedback and other things
[18:22] like that. So we give it a nice big
[18:24] prompt and some sub-agents that help it
[18:25] basically go out and explore this data
[18:27] and identify issues and see what common
[18:29] things are. And then it will create
[18:31] these issues right here. And so here um
[18:34] it's created an issue. It gives a
[18:35] description of it. It has It links to
[18:37] the traces so I can go see some
[18:39] supporting evidence. And then down here,
[18:41] I guess this is very simple changes to
[18:43] the prompts. Um but here it's updating
[18:45] part of uh of the context in this case.
[18:49] Um here it's also updating some
[18:50] instructions. Um and we can see here
[18:53] that it's adding uh some code to go into
[18:55] the harness as well. And so this is uh
[18:58] something we launched in the past few uh
[19:00] months and I think speaks to this data
[19:03] flywheel, which again is a very simple
[19:05] thing. Run agent, get traces,
[19:08] see patterns, fix. Um and and this is
[19:11] our attempt at automating it. Um, that's
[19:15] all I've got. Happy to take any
[19:16] questions on harnesses or evals.
[19:19] >> Um, this is great, by the way. Really
[19:21] appreciate the whole whole presentation.
[19:23] Uh, Engine itself is an agent, right?
[19:26] That's given a prompt and go search over
[19:28] things. Uh, have you run Engine on
[19:30] Engine?
[19:31] >> We have it running, yeah. So, we got
[19:32] slight So, Engine also hooks up to Slack
[19:34] and sends it kind of like reports about
[19:36] itself. Um, and yeah, that's how we
[19:38] that's how we dogfood it. Yeah.
[19:41] We also uh we also created uh
[19:44] uh what we call kind of like issue bench
[19:45] for Engine, which again is like a harbor
[19:47] a harbor formatted uh
[19:49] uh
[19:51] benchmark, basically, that we're
[19:52] constantly benchmarking different models
[19:54] and different harnesses on.
[19:56] Um, and so, it's
[19:58] I think Gabe talked about this a little
[19:59] bit, but one of the benefits of having a
[20:00] benchmark is you can you can benchmark
[20:02] it on a bunch of different harnesses and
[20:04] see what they're good and bad at. So, we
[20:06] we uh I think a few weeks ago we ran our
[20:09] our own kind of like deep agents and
[20:10] then Codex and then Claude Code on this.
[20:13] And we saw that Codex was doing a really
[20:15] interesting thing, where it would write
[20:16] itself a bunch of small scripts to run
[20:19] against these traces, and it was doing
[20:20] that really aggressively and actually
[20:21] allowing it to perform really well. So,
[20:23] we we did a sprint to do what we call
[20:25] the kind of like codis- codexification
[20:28] of Engine and basically take that
[20:29] learning and and bring it into kind of
[20:31] like the the core Engine harness. And
[20:33] so, I think that's another uh a benefit
[20:35] of having a benchmark is you can just
[20:36] run a bunch of different things on it
[20:38] and see how they actually perform and
[20:39] then bring those things back into your
[20:41] core kind of like agent harness.
[20:45] >> Super cool talk. Um, to which extent do
[20:48] you think that like harnesses will
[20:50] converge into one thing and users will
[20:52] be educated to do that and the models
[20:54] will be best for that versus
[20:55] diversifying here, every company has
[20:57] their own
[20:58] uh way of doing things uh optimized for
[21:01] them?
[21:02] >> Yeah. Yeah, really good question, one
[21:03] that we think a lot about. And um, I I
[21:05] chatted with Eno from Factory, who also
[21:08] kind of like thinks a lot about this.
[21:10] Um, and and I think there's some stuff
[21:11] we talked about this.
[21:12] Um,
[21:13] I I think
[21:15] uh
[21:17] I think there's
[21:19] I I think the model I I don't know is
[21:22] the is the honest answer. I think uh
[21:24] some things that I've seen is that the
[21:26] general purpose harnesses have gotten
[21:29] good enough to work for a lot of basic
[21:32] tasks at least when you're getting
[21:33] started. So, I would recommend getting
[21:35] started with like an off-the-shelf
[21:36] harness, whether it's Deep Agents or
[21:38] Codex or Cloud Code or something like
[21:39] that. Because I think the models are now
[21:41] good enough and the things that we've
[21:42] learned about what these makes these
[21:44] models good, access to file systems,
[21:46] sub-agents, things like that, those are
[21:48] those those are kind of like good
[21:49] enough. Um, I think we often see that
[21:54] the more out of distribution you get,
[21:56] the more you're going to want to
[21:57] customize the harness. And it's a scale,
[21:59] right? So, like it uh it it it it at the
[22:03] extreme end of a scale, you might want
[22:05] to build a complete kind of like
[22:06] cognitive architecture that that is
[22:08] really focused on things. A A reason
[22:10] Another reason you might want to do
[22:11] that, by the way, is kind of like for
[22:12] predictability and control. And so, we
[22:13] have a lot of customers in financial
[22:15] services where they want where they need
[22:16] kind of like predictability. And so,
[22:18] they we show them something like Deep
[22:20] Agents and they're like, "Woah, woah,
[22:22] woah. That's way too like scary an agent
[22:24] for us. We want like more of this kind
[22:25] of like custom cognitive architecture
[22:27] where we can really control things." Um,
[22:29] but but then on the other end, uh you
[22:31] know, you could just use an
[22:32] off-the-shelf harness and there's things
[22:33] in the middle like hooks or middleware
[22:35] that you can use to kind of So, so it's
[22:36] a spectrum as well. The more out of
[22:38] distribution you get, the more custom
[22:39] harness you're going to want to have.
[22:41] Um, and then there's other like weird
[22:43] things where like again, like um, I I I
[22:45] think uh both OpenAI and Anthropic are
[22:48] getting really good at coding, but
[22:49] they've landed on different ways to kind
[22:51] of like edit files um that are like, you
[22:53] know,
[22:54] that are like actually pretty different.
[22:57] Um, you know, I I think they have some
[22:59] benchmark and I think he was uh he he
[23:01] thought that one way was just better
[23:03] than the other way just like strictly
[23:04] superior
[23:06] and and so that's like like
[23:08] so I think the model labs will kind of
[23:10] converge and that they all seem to be
[23:11] kind of like really good at coding the
[23:13] harnesses will converge to to kind of
[23:15] like being really good at coding if they
[23:16] keep on going down that path but they're
[23:18] at the same time there are these like
[23:19] really small differences and I don't
[23:21] really know how to explain those either
[23:22] and and
[23:24] right now I think those show up most
[23:25] concretely in small things but you could
[23:28] imagine what what if one lab really goes
[23:30] down kind of like bio and those
[23:33] harnesses become really good at kind of
[23:34] like bio agent things then then then the
[23:37] harnesses themselves start to diverge
[23:39] and so
[23:40] I I I don't know is the answer to the
[23:42] fast moving space that's why evals and
[23:43] observability are important and I think
[23:45] and and I think we
[23:47] to measure all of that.
[23:49] Cool. Awesome. Thank you guys.
[23:52] >> [applause]
