LangChain Launches Self-Testing Engine v2 as Base Models Commoditize

Interrupt NYC: Opening Keynote — LangChain

LangChain

As foundation model capabilities commoditize, software differentiation shifts entirely to teams that build domain-specific harnesses and compound proprietary trace data through automated evaluation loops.

AI-assisted summary of Interrupt NYC: Opening Keynote — original source. Check the original for full context.

Source published:

Key ideas

  • LangChain framed 'owning your intelligence' as the core competitive imperative for AI builders, shifting focus from commoditized foundation models to proprietary execution harnesses, compounding eval loops, and enterprise governance. (1:42)
  • LangChain launched Managed Deep Agents featuring built-in enterprise authentication primitives (Connections, Identity, Channels), user-level memory, and native Parallel web search MCP integration. (12:42)
  • The company introduced LangSmith Trajectories alongside the smithtune CLI, enabling teams to normalize complex multi-step agent traces and pipeline them directly into compute providers like Fireworks or Baseten for targeted fine-tuning. (19:50)
  • LangSmith Engine v2 added proactive fix verification and automated pre-launch red teaming on preview deployment branches, having already scanned over 70 million production traces and flagged more than 21,000 issues. (26:03)

Detailed notes

## Background Published on YouTube by LangChain on October 5, 2026, this keynote captures the opening session of Interrupt NYC, LangChain's flagship agent conference. In the presentation, LangChain Co-Founder and CEO Harrison Chase outlines the concept of 'owning your intelligence' and walks through major updates across the LangSmith platform, including Managed Deep Agents, trajectory-based evals, fine-tuning tools, and LangSmith Engine v2.

## TL;DR Harrison Chase explains how the AI landscape transitioned from raw prototypes to domain-specific agents embedded directly in the application layer, driven by enterprise case studies like Rogo, Harvey, and JPMorgan Chase. He defines the three pillars required to own enterprise intelligence: an open and controllable harness, compounding evaluation loops, and rigorous governance. Walking through the three tiers of the LangSmith platform—runtime, observability/evals, and intelligence—Chase introduces Managed Deep Agents with enterprise auth and memory, decision model routing via the LLM Gateway, and LangSmith Trajectories for cleaner debugging. He concludes with announcements for fine-tuning via the smithtune CLI, LangSmith Custom Apps, and LangSmith Engine v2, which autonomously red teams agents and verifies its own code fixes on preview deployments.

## Atmosphere & style The presentation carries an energetic, forward-looking conference keynote vibe, with Chase speaking authoritatively and enthusiastically as he frames industry shifts across the past three years. The delivery blends high-level architectural philosophy with concrete product mechanics, moving methodically through framework diagrams and technical feature breakdowns. It feels like an ambitious practitioner-focused briefing, balancing enterprise infrastructure realities with palpable excitement about autonomous agent engineering.

## Bulletin - [01:42](https://www.youtube.com/watch?v=950byF7njfw&t=102s) LangChain framed 'owning your intelligence' as the core competitive imperative for AI builders, shifting focus from commoditized foundation models to proprietary execution harnesses, compounding eval loops, and enterprise governance. - [12:42](https://www.youtube.com/watch?v=950byF7njfw&t=762s) LangChain launched Managed Deep Agents featuring built-in enterprise authentication primitives (Connections, Identity, Channels), user-level memory, and native Parallel web search MCP integration. - [19:50](https://www.youtube.com/watch?v=950byF7njfw&t=1190s) The company introduced LangSmith Trajectories alongside the smithtune CLI, enabling teams to normalize complex multi-step agent traces and pipeline them directly into compute providers like Fireworks or Baseten for targeted fine-tuning. - [26:03](https://www.youtube.com/watch?v=950byF7njfw&t=1563s) LangSmith Engine v2 added proactive fix verification and automated pre-launch red teaming on preview deployment branches, having already scanned over 70 million production traces and flagged more than 21,000 issues.

## Key takeaways - [00:06](https://www.youtube.com/watch?v=950byF7njfw&t=6s) The AI industry has evolved rapidly from early 2023 prototypes into autonomous, domain-specific agents fundamentally reshaping the software application layer. - [01:42](https://www.youtube.com/watch?v=950byF7njfw&t=102s) Leading production implementations like Rogo, Harvey, and JPMorgan Chase's 'Jarvis' illustrate the shift toward building proprietary domain logic, routing, and scaffolding around base models. - [01:42](https://www.youtube.com/watch?v=950byF7njfw&t=102s) Escalating token costs, high-performing open-weight models, and commoditized base model intelligence are driving enterprise development teams to own their intelligence. - [04:25](https://www.youtube.com/watch?v=950byF7njfw&t=265s) Owning your intelligence rests on three core pillars: an open and model-neutral execution harness, a compounding evaluation loop, and robust enterprise governance. - [07:22](https://www.youtube.com/watch?v=950byF7njfw&t=442s) High-performing organizations structure agent efforts across three concentric rings: central platform engineers, hybrid agent engineers, and subject matter experts establishing domain ground truth. - [09:38](https://www.youtube.com/watch?v=950byF7njfw&t=578s) LangSmith's platform is organized into three distinct architectural tiers: cloud runtime infrastructure, agent-native observability and evals, and an overarching intelligence layer. - [12:42](https://www.youtube.com/watch?v=950byF7njfw&t=762s) Managed Deep Agents bundles the Deep Agents harness with managed cloud infrastructure, introducing Connections, Identity, and Channels to solve complex enterprise auth boundaries. - [15:51](https://www.youtube.com/watch?v=950byF7njfw&t=951s) Specialized decision models like Jev and SemIf offer rapid deterministic scoring for real-time guardrails, in-loop evaluation, and cognitive graph routing without generation latency. - [18:00](https://www.youtube.com/watch?v=950byF7njfw&t=1080s) The LangSmith LLM Gateway provides unified routing, message format normalization, enterprise rate limiting, cost auditing, and automated provider fallbacks. - [19:50](https://www.youtube.com/watch?v=950byF7njfw&t=1190s) LangSmith Trajectories abstracts multi-turn execution traces into standard conversational streams powered by SmithDB to simplify debugging and human annotation. - [22:57](https://www.youtube.com/watch?v=950byF7njfw&t=1377s) The smithtune CLI bridges production observability and post-training by exporting filtered trace datasets directly into model fine-tuning providers like Fireworks and Baseten. - [24:44](https://www.youtube.com/watch?v=950byF7njfw&t=1484s) LangSmith Custom Apps enables users to prompt internal coding agents to generate bespoke web interfaces on top of tracing data with inherited enterprise RBAC and ABAC controls. - [26:03](https://www.youtube.com/watch?v=950byF7njfw&t=1563s) LangSmith Engine v2 validates its own code patches by spinning up preview branches on LangSmith Deployment to verify bug fixes before presenting findings to engineers. - [26:03](https://www.youtube.com/watch?v=950byF7njfw&t=1563s) Engine v2 introduces automated agent red teaming to autonomously probe pre-launch deployments for prompt leaks and bad outputs while bootstrapping golden evaluation datasets. - [26:03](https://www.youtube.com/watch?v=950byF7njfw&t=1563s) LangSmith Engine has processed over 70 million production traces, detected more than 21,000 issues, and is expanding to self-hosted environments via BYOK support in v17.

## Detailed walkthrough ### [00:06](https://www.youtube.com/watch?v=950byF7njfw&t=6s) Welcome to Interrupt NYC: From Prototypes to the Application Layer

Opening the keynote with palpable energy, Harrison Chase welcomes the crowd to Interrupt NYC, remarking on the excitement of bringing LangChain's flagship agent conference to New York for the first time after holding it in San Francisco for the previous two years. Setting the stage for a jam-packed full-day schedule of announcements, fireside chats, and practitioner presentations, he pauses to contextualize how quickly the artificial intelligence landscape has transformed over the prior three years. While LangChain was founded just prior to the late-2022 generative AI boom, the industry has transitioned through clear generational phases at breakneck speed.

Recapping that progression, Chase explains that 2023 was primarily characterized by proof-of-concept prototypes. By 2024, development teams had begun shipping simple applications to production, though these early systems still largely relied on isolated, single LLM API calls. Exploration into multi-step agents expanded throughout 2025, culminating in the reality of the current landscape: autonomous agents reshaping the core application layer across entire verticals. Rather than serving as superficial "thin wrappers" around third-party models, production agents are now engineered with substantial software infrastructure to deliver defensible, differentiated experiences.

As Harrison Chase observed regarding the shift from generic wrappers to specialized production systems: “And really this year, we've started seeing that agents have transformed the application layer.”

- **Conference Background**: First New York City edition of Interrupt following two consecutive years in San Francisco. - **Industry Timeline**: - **2023**: Rapid prototyping and experimental proofs-of-concept. - **2024**: Production deployments of straightforward, single-LLM-call applications. - **2025**: Broad exploration into autonomous agents and multi-step architectures. - **2026**: Domain-specific agents fundamentally redefining the modern software application layer.

---

[01:42](https://www.youtube.com/watch?v=950byF7njfw&t=102s) Domain-Specific Agents and Owning Your Intelligence

To demonstrate how leading organizations construct value around foundation models, Chase highlights three prominent production case studies featured at the conference. First, fintech startup Rogo built a specialized model-routing harness to power an institutional-grade financial agent. Second, legal tech leader Harvey leveraged targeted model fine-tuning to maximize accuracy and procedural adherence across specialized legal workflows. Third, JPMorgan Chase engineered a deeply customized internal agent harness to power their proprietary "Jarvis" assembly line for enterprise operations.

```mermaid flowchart TD accTitle: Owning Your Intelligence Framework accDescr: Diagram illustrating how domain-specific context, routing, and fine-tuning wrap around foundation models. subgraph Scaffolding["Differentiated Domain Harness"] R["Intelligent Model Routing\n(e.g., Rogo)"] FT["Domain Fine-Tuning\n(e.g., Harvey)"] H["Custom Scaffolding & Context\n(e.g., JPMorgan Jarvis)"] end

subgraph Core["Base Intelligence Layer"] M["Open & Proprietary Foundation Models"] end

R --> M FT --> M H --> M ```

These architectures demonstrate what the industry now describes as "owning your intelligence"—the deliberate strategy of building domain-specific logic, data pipelines, and execution scaffolding directly in and around base models. Chase identifies three principal market pressures driving this movement: 1. **Escalating Token Expenditures**: As enterprise adoption scales into heavy production volumes, managing and auditing token consumption has become essential for preserving unit economics and proving business return on investment. 2. **Rise of Capable Open-Weight Models**: High-performing open models offer cost-efficient, self-hostable alternatives that allow teams to post-train and specialize weights for their exact domain tasks. 3. **Commoditization of Raw Model Intelligence**: Foundation model outputs are increasingly commoditized; real differentiation now stems from proprietary context injection, private data stores, and tailored business rules embedded within the application harness.

Reflecting on why differentiation has shifted away from foundation models alone, Harrison Chase stated: “And then the last thing that I think has become pretty clear over the past year is that the models themselves are becoming somewhat commoditized, and it's all of this stuff around the model, in particular the data, whether that is actual kind of like context you're providing or some of the domain-specific paths that you're baking into the harness.”

---

[04:25](https://www.youtube.com/watch?v=950byF7njfw&t=265s) The Three Core Pillars of Owning Intelligence

Connecting this philosophy back to LangChain’s founding, Chase recounts that while the project began a month prior to the release of ChatGPT, widespread adoption surged when engineers saw conversational AI and immediately wanted that same capability tailored to their specific industries and enterprise data. A core premise of LangChain has always been decoupling the application logic from the underlying model provider, giving builders complete model optionality.

Chase outlines the three fundamental pillars required to truly own intelligence in an agentic architecture: 1. **An Open and Controllable Harness**: The execution harness must remain strictly model-neutral, serving as both an offensive hedge (enabling instant migration to cutting-edge models from OpenAI, Anthropic, or specialized providers like TypeSafe’s Jev decision models) and a defensive safeguard against vendor lock-in or sudden API price increases. Furthermore, the harness must support granular control over context injection, whether expressed as explicit directed cyclic graphs via LangGraph or as extensible middleware within agentic frameworks like Deep Agents. 2. **A Compounding Evaluation Loop**: Launching an agent into production is merely the starting line. Builders must continually capture interaction data, trace edge-case failures, and compound the agent's capability over time through manual code and prompt adjustments or automated mechanisms like persistent memory and prompt optimization. 3. **Robust Governance**: Particularly critical for internal enterprise deployments, governance ensures organizations retain complete visibility into agent inventories, control compute spend, and enforce strict, role-based authentication and authorization across enterprise data sources.

Emphasizing the competitive advantage of domain-specific evaluation, Harrison Chase argued: “If you can define what better and what best looks like for your domain better than anyone else, you will be able to build the best agent for that domain.”

---

[07:22](https://www.youtube.com/watch?v=950byF7njfw&t=442s) Organizing People and Process for Enterprise Agents

Executing on these pillars requires restructuring how enterprise teams coordinate their workforce and design their software lifecycles. Chase details how successful organizations are structuring their talent into three distinct concentric rings: - **Ring 1: Core Platform Engineers**: Central infrastructure teams that deliver reusable tooling, operationalize governance, provide deployment rails, and maintain the compounding evaluation platform utilized across the business. - **Ring 2: Agent Engineers**: A newly emerged professional cohort that Chase describes as an interdisciplinary hybrid of data scientists, software engineers, and machine learning practitioners who consume platform tools to assemble, ship, and refine production agents. - **Ring 3: Subject Matter Experts (SMEs)**: Non-developer domain authorities situated at the business edge—such as legal analysts, traders, or clinical specialists—who must be equipped with intuitive interfaces to review agent behavior and establish what constitutes ground truth.

```mermaid flowchart LR accTitle: The Iterative Agent Lifecycle Loop accDescr: The continuous cycle of building, testing, deploying, and monitoring governed agents. subgraph Lifecycle["Iterative Agent Development Loop"] B["Build"] --> T["Test"] T --> D["Deploy"] D --> M["Monitor & Trace"] M -->|"Extract Learnings & Edge Cases"| B end ```

Alongside this organizational blueprint, organizations are embracing an inherently iterative development lifecycle. Because first-generation agent deployments inevitably fail on novel edge cases, high-performing engineering teams organize their workflow around a continuous feedback loop: build, test, deploy, monitor, and directly ingest production tracing learnings back into prompt and graph refinements—all executed within a compliant governance framework.

Describing this emerging engineering persona, Harrison Chase noted: “So, this is a great term to describe the weird hybrid mix of data scientist and engineer and machine learning engineer that makes up the people who are driving these agents.”

---

[09:20](https://www.youtube.com/watch?v=950byF7njfw&t=560s) Technology: Runtime Frameworks and Model Harnesses

Transitioning from organization and workflow into technical architecture, Chase introduces the foundational layers of the LangChain product suite designed to support this paradigm. The technological foundation begins with the runtime layer—the origin of LangChain’s open-source libraries, and the continuous focus of modern orchestration frameworks like LangGraph and Deep Agents.

These runtimes provide the programmatic scaffolding required to manage complex state, handle asynchronous tool execution, coordinate multi-agent loops, and enforce domain constraints before execution ever touches an evaluation or fine-tuning pipeline.

Summarizing LangChain's technical focus, Harrison Chase explained: “And when we think about the technology, and the technology that we help build and provide, there's really like two core parts to it: so one is the runtime.”

[09:38](https://www.youtube.com/watch?v=950byF7njfw&t=578s) LangSmith's Three Layers: Runtime, Observability/Evals, and Intelligence

Harrison Chase frames the architectural scope of LangSmith by breaking it down into two historical pillars and a newly added third layer. An agent cannot simply run locally on an engineer's laptop; it must run reliably in the cloud to serve an entire organization or external customers, which introduces a distinct deployment and runtime challenge. Alongside that runtime sits observability and evaluation, which Harrison describes as the bedrock of the compounding learning loop: being able to trace agent behavior and systematically evaluate whether actions were successful.

On top of runtime and observability, LangChain has recently layered an intelligence tier. This intelligence layer sits across the platform to help teams actively utilize trace data and speed up the agent development lifecycle. Harrison explains: “So now when we think about LangSmith, our commercial platform, we think about it serving the runtime aspect, serving the observability and eval aspect, and then building in intelligence across the platform to really help people utilize these components and speed up that process, that agent development life cycle that I talked about earlier.”

```mermaid flowchart TD accTitle: LangSmith Three-Tier Architecture accDescr: Diagram showing the three tiers of LangSmith: Intelligence Layer on top, supported by Runtime Infrastructure and Observability and Evals. subgraph L3["Intelligence Layer"] IL["Intelligence Across Platform (Engine v2, Red Teaming, Fix Proposals)"] end subgraph L2["Observability & Evals"] OE["LangSmith Tracing, Trajectories, Evals & Compounding Loops"] end subgraph L1["Runtime & Infrastructure"] RT["LangSmith Deployment, Sandboxes, LLM Gateway, Tool Servers"] end L1 --> OE OE --> IL ```

[10:38](https://www.youtube.com/watch?v=950byF7njfw&t=638s) Deconstructing the Agent Runtime: Logic, Harness, and Infrastructure

Harrison unpacks the runtime architecture of an agent into three distinct ingredients: business logic, the harness, and the underlying cloud infrastructure. Business logic encompasses instructions, tools, skills, and hooks—the specialized domain knowledge that developers supply. Skills required for an agent operating in finance differ drastically from those needed in a legal environment, making business logic the bespoke element contributed by each team.

The harness acts as the wiring layer, connecting the base model with this domain context so the model sees the right information at the exact right moment. Harrison points out that developers can either build custom, tightly controlled cognitive architectures using LangGraph or leverage general-purpose harnesses. To support the latter pattern, LangChain built Deep Agents, an open-source off-the-shelf harness launched roughly a year prior. Deep Agents borrows concepts from modern coding agents, featuring a dedicated execution environment, file system, sandbox, and context compaction mechanisms tailored for domain-specific applications. Finally, running in production requires cloud infrastructure: durable server execution, secure tool execution, code execution sandboxes, and reliable model access.

- **LangSmith Deployment:** Manages remote, durable agent runtime execution in the cloud. - **LangSmith Sandboxes (Beta):** Provides secure environments for running arbitrary agent-generated code. - **LangSmith LLM Gateway:** Handles unified access, routing, and governance across foundation models. - **Tool Servers:** Under active development to streamline secure, remote tool connectivity.

[12:42](https://www.youtube.com/watch?v=950byF7njfw&t=762s) Managed Deep Agents: Auth Primitives, User-Level Memory, and Parallel Search

To spare engineering teams from having to manually assemble disparate infrastructure components, LangChain introduced Managed Deep Agents, combining the Deep Agents harness and managed cloud infrastructure into a unified platform. Developers need only supply their business logic to launch a functional agent. Harrison introduces several major enhancements included in the latest Managed Deep Agents release, starting with enterprise authentication and permissions—an area he singles out as one of the hardest hurdles when launching internal corporate agents.

When an internal agent accesses tools like Notion or Jira, it must navigate complex identity boundaries: should it act under a fixed service identity or inherit the personal credentials of the calling user? Managed Deep Agents resolves this by introducing three core auth primitives: 1. **Connections:** Defines how end users authenticate to tool servers when invoking tools. 2. **Identity:** Distinguishes whether the agent acts as user-permissioned or service-permissioned as a whole. 3. **Channels:** Manages user context and credential attribution across communication surfaces such as shared Slack channels, ensuring runs are credited to the specific coworker sending the message.

Beyond authentication, the new release upgrades memory architecture and tool access: - **User-Level Memory:** Complements existing agent-level memory by tracking traits, context, and preferences specific to individual end users, enabling deep personalization across repeated interactions. - **Built-In Web Search via Parallel:** Agents can access real-time web search natively through Parallel's MCP without developers having to provision a separate third-party API key; the search integration runs directly against the user's LangSmith API key.

Harrison highlights how design partners use this setup: “We've been working with a number of early design partners on this, and we've seen really good results, especially for just getting an agent up and running and having it be like this company harness, and that's how we think about Managed Deep Agents, is this company harness that makes it really easy to get started building and sharing agents inside an org...”

```mermaid flowchart LR accTitle: Managed Deep Agents Workflow accDescr: Flow diagram illustrating how user business logic combines with Managed Deep Agents primitives into production runtime. BL["Business Logic (Instructions, Skills, Tools)"] --> MDA["Managed Deep Agents"] subgraph MDA_Feat["Managed Deep Agents Primitives"] AUTH["Auth Primitives (Connections, Identity, Channels)"] MEM["User-Level Memory"] PWS["Parallel Web Search MCP"] end MDA_Feat --> MDA MDA --> PROD["Production Agent Deployment"] ```

[15:51](https://www.youtube.com/watch?v=950byF7njfw&t=951s) The Rise of Decision Models: Jev and SemIf

Turning to model architectures, Harrison spotlights decision models, highlighting the launch of Jev over the preceding weekend. Unlike generative models, a decision model cannot generate arbitrary natural language text. Instead, given an input state and a discrete list of questions or options, it evaluates and outputs scores, boolean flags, or categorical choices. Harrison emphasizes that despite their narrower scope, decision models represent a critical shift in agent system design. While frontier models like Claude, Codex, and Deep Agents run primary multi-step loops, fast decision models can handle micro-decisions and deterministic routing.

Harrison details three primary use cases where decision models excel: - **In-Loop and Post-Hoc Evals:** Scoring an agent's intermediate actions in real time as tasks execute, allowing systems to flag or intercept bad outputs before they compound. - **Low-Latency Guardrails:** Filtering or vetting incoming user prompts almost instantaneously without incurring the multi-second latency penalty typical of standard frontier model calls. - **Model and Skill Routing:** Determining which specialized sub-agent, toolset, or cognitive graph branch to activate within LangGraph architectures.

Both Jev and SemIf—a prominent open-source decision model—are accessible directly through the LangSmith LLM Gateway, with Harrison predicting that specialized decision models will see rapid proliferation.

[18:00](https://www.youtube.com/watch?v=950byF7njfw&t=1080s) LangSmith LLM Gateway: Normalization and Traffic Governance

Harrison closes his walkthrough of the runtime tier by examining the LangSmith LLM Gateway, which entered beta roughly one to two months prior. The gateway serves as a unified routing, cost control, and governance hub across all internal and customer-facing LLM workloads. It standardizes model inputs and outputs across OpenAI and Anthropic message formats, while still offering a transparent pass-through mode so engineering teams can adopt vendor-specific features the moment APIs update.

The gateway also provides out-of-the-box integrations with developer-facing coding agents like Codex, Claude Code, and Cursor. Harrison argues that centralizing this traffic is vital for enterprise oversight: “Again, we think you should be routing all of your traffic, for governance and cost control reasons. We think you should be routing all of it through...”

[18:44](https://www.youtube.com/watch?v=950byF7njfw&t=1124s) Routing Every Model Call Through the LangSmith LLM Gateway

Harrison Chase transitions to the governance and infrastructure required when handling model providers across an enterprise. Because an LLM gateway must integrate with every AI application across an organization, LangChain focuses on three core operational pillars: cost controls, rate limiting, and fallbacks. Cost controls provide visibility into spending and allow teams to enforce granular budget limits across different teams, users, and API keys. Rate limiting ensures that a single runaway application cannot throttle shared quotas or dominate usage. Finally, provider fallbacks allow applications to gracefully switch to an alternative LLM provider when encountering upstream errors.

Looking ahead, Chase previews more stateful fallback behavior designed to eliminate wasted downtime:

> “So if you notice that requests are failing and Anthropic might be down, you don't even try them for the future. You just switch over to OpenAI for the next 10 minutes or 30 minutes, and make it really configurable in that way.”

He summarizes the ultimate objective for the gateway layer:

> “And so we want to make LangSmith LLM Gateway the most kind of like secure and reliable way to monitor and govern and fall back on different LLM providers when you're using them.”

[19:50](https://www.youtube.com/watch?v=950byF7njfw&t=1190s) Introducing LangSmith Trajectories

Moving from runtime systems to observability and evals, Chase emphasizes that traces serve as the fundamental system of record that powers the compounding intelligence loop. Historically, LangChain talked about observability using three main primitives: - **Run:** A single LLM invocation. - **Trace:** A single invocation of an entire agent. - **Thread:** A series of traces representing a multi-turn user interaction.

Over the past year, agent patterns have converged: whether builders are running coding agents or first-party assistants, the systems fundamentally run an LLM in a loop and compile an evolving sequence of messages. Because complex traces with thousands of tool calls become overwhelming to parse in traditional tree views, LangChain is introducing **LangSmith Trajectories**—a standardized format and UI representation for agent executions.

Trajectories offer three major benefits: - **Faster, intuitive debugging:** Instead of wading through massive raw traces, the UI condenses tool calls and iterations into a conversational stream that looks and feels like using a chat or coding agent. Users can scan sequences quickly and expand individual tool or LLM calls on demand. - **Simpler human annotation:** Because the full interaction history is cleanly rendered, reviewers and domain experts can leave feedback and evaluate steps far more easily. - **Standardized ingestion for post-training:** LangSmith parses messy trace formats from diverse SDKs and agent frameworks into a consistent trajectory schema, making production data immediately ready for fine-tuning.

```mermaid flowchart LR accTitle: Evolution from Traces to Trajectories accDescr: Diagram showing raw SDK agent traces condensed and normalized into LangSmith Trajectories.

subgraph Ingestion["Trace Collection"] A["Agent SDK Runs"] B["Coding Agent Runs"] end

subgraph Normalization["LangSmith Processing"] C["SmithDB Storage"] D["LangSmith Trajectories"] end

subgraph Applications["Downstream Uses"] E["Clean Debugging & Annotation"] F["Model Fine-Tuning"] end

A --> C B --> C C --> D D --> E D --> F ```

[22:57](https://www.youtube.com/watch?v=950byF7njfw&t=1377s) Turning Trajectories into Models with smithtune

Fine-tuning breaks down into two core ingredients: the training data and the training/inference compute infrastructure. To bridge the gap between production agent logs and specialized models, LangChain announces **LangSmith Fine-Tuning** alongside the new **`smithtune` CLI**.

Using `smithtune`, engineers can query and filter relevant traces directly out of LangSmith, pre-process them into fine-tune-ready trajectories, and send them directly into training platforms such as Fireworks or Baseten. Once training finishes, the models are evaluated and output as optimized, task-specific models. Chase underscores that this workflow unlocks fine-tuning across multiple use cases:

> “And so we think this is a really powerful way to get started fine-tuning, whether it's on agents or classification tasks or guardrails or anything like that.”

Underneath these trajectories is **SmithDB**, an in-house database introduced at a previous Interrupt conference. Built specifically to handle agent workloads, SmithDB dramatically accelerates ingesting, storing, and querying high-volume message histories, making rich trajectory visualizations possible without lag.

```mermaid flowchart LR accTitle: smithtune Fine-Tuning Workflow accDescr: End-to-end pipeline from LangSmith traces to an optimized model via smithtune.

A["LangSmith Traces<br/>(SmithDB)"] --> B["smithtune CLI<br/>(Filter & Preprocess)"] B --> C["Training Compute<br/>(Fireworks / Baseten)"] C --> D["LangSmith Evals"] D --> E["Optimized Task Model"] ```

[24:44](https://www.youtube.com/watch?v=950byF7njfw&t=1484s) Building Custom Interfaces with LangSmith Custom Apps

To solve the challenge of presenting internal data to non-technical stakeholders and domain annotators, LangChain announces **LangSmith Custom Apps**:

> “So, LangSmith Custom Apps you can basically think of as Lovable, but for your data that's already on LangSmith.”

Users can chat with an internal coding agent that builds bespoke web applications on the fly on top of their LangSmith data. LangChain open-sourced its own design system so the coding agent can construct polished, native-looking UIs out of the box. Crucially for enterprise governance, these generated apps automatically inherit the user's workspace boundaries, role-based access controls (RBAC), and attribute-based access controls (ABAC), providing a secure mechanism to bring human-in-the-loop reviewers directly into evaluation workflows.

[26:03](https://www.youtube.com/watch?v=950byF7njfw&t=1563s) Proactive Fixes and Red Teaming with LangSmith Engine v2

Shifting to the intelligence layer, Chase revisits **LangSmith Engine**, an autonomous system designed to act like a background AI engineer sitting directly on top of tracing projects. Rather than relying entirely on manual human triage, Engine continuously scans traces, clusters emerging failure modes on an automated issue board, writes candidate code patches, and generates regression datasets and evaluators.

Building on six months of deployment experience, LangChain is introducing **LangSmith Engine v2**, which adds two major automated capabilities:

1. **Proactive Fix Verification:** Deeply integrated with LangSmith Deployment, Engine no longer just guesses at code fixes. When it discovers a bug pattern in traces, it spins up a preview deployment branch to reproduce and confirm the failure. It then writes a fix, spins up a second preview branch, executes tests against it to verify the bug is resolved, and delivers an issue report complete with end-to-end evidence proving both the defect and the resolution. 2. **Automated Agent Red Teaming:** Previously, Engine could only catch errors that had already occurred in production traces. Through its deployment integration, Engine can now autonomously simulate adversarial inputs, generate failure hypotheses, and probe the agent deployment before launch: - Identifies prompt leakage or unauthorized learning. - Catches off-brand, hallucinated, or incorrect outputs. - Automatically bootstraps golden evaluation datasets, eliminating the historically painful manual effort required to generate test suites before shipping.

Chase highlights the real-world scale and efficiency milestones achieved by the Engine team:

> “Engine has already scanned over 70 million traces and detected over 21,000 issues.”

He stresses that speeding up the iteration loop is the single most critical factor in building reliable agents. To ensure enterprise availability, Chase announces that **Engine is now available for self-hosted environments using bring-your-own-key (BYOK)**, rolling out in self-hosted version v17 within a week or two.

- **Engine scale:** Over 70,000,000 traces scanned and 21,000+ distinct issues surfaced across fast-growing AI engineering organizations. - **Self-hosted availability:** Shipping in LangSmith self-hosted v17 with full BYOK support.

[30:52](https://www.youtube.com/watch?v=950byF7njfw&t=1852s) The Full Stack for Owning Your Intelligence and Looking Ahead

Harrison Chase wraps up the opening keynote by tying together the architectural layers LangChain has developed across the platform. At the foundation is the runtime layer: open-source harnesses paired with deployments, where managed agent runtimes are increasingly emerging as the most reliable path for teams pushing complex agent systems into production. Sitting directly above that is observability and evaluation, which LangChain has re-engineered to be distinctly agent-native. By focusing on multi-step trajectories rather than flat single-turn traces, allowing teams to construct tailor-made interfaces using Custom Apps, and enabling developers to feed production interaction data directly into fine-tuning workflows, teams can continuously adapt underlying models to their actual operational workloads.

```mermaid flowchart TD accTitle: LangChain Three-Layer Architecture accDescr: Architecture diagram showing the Intelligence Layer at top, Observability and Evals in the middle, and Runtime at the foundation.

subgraph Intelligence["Intelligence Layer"] Engine["LangSmith Engine<br/>• Trace scanning<br/>• Automated red teaming<br/>• Automated engineering tasks"] end

subgraph Observability["Observability & Evals"] Obs["LangSmith Platform<br/>• Agent trajectories<br/>• Custom Apps<br/>• Model tuning (smithtune)"] end

subgraph Runtime["Runtime Layer"] Run["Managed Agent Runtimes & Deployments<br/>• Open-source frameworks (LangGraph, Deep Agents)<br/>• Production managed execution"] end

Engine --> Observability Observability --> Runtime ```

At the top of the stack sits the intelligence layer driven by LangSmith Engine. Harrison emphasizes that automated trace scanning and adversarial red teaming represent merely the starting point for what LangChain envisions this layer handling on behalf of builders. As Harrison Chase puts it: “And then lastly, we have this intelligence layer that sits above, and we want to make Engine do a lot of the things that agent engineers and AI engineers are doing now.” The ultimate objective across all these layers is empowering organizations to take control of their systems and build deep, domain-specific capabilities—regardless of whether those agents power external user-facing products or automate and modernize complex internal workflows. Harrison concludes by thanking the crowd for attending Interrupt in New York City and invites the event's master of ceremonies, Jake, onto the stage to kick off the rest of the conference program.

- **Managed Agent Runtimes**: Bridging open source and deployment infrastructure to provide production-grade foundations for agent architectures. - **Agent-Native Observability and Evals**: Centered around structured trajectory analysis, bespoke visibility via Custom Apps, and continuous fine-tuning pipelines. - **The Intelligence Layer**: Positioning LangSmith Engine to automate tasks traditionally handled manually by AI and agent engineers. - **Owning Intelligence**: Giving organizations the harness, data loops, and tooling needed to build proprietary domain intelligence into both internal workflows and outward-facing products.

## Notable quotes > [04:25](https://www.youtube.com/watch?v=950byF7njfw&t=265s) If you can define what better and what best looks like for your domain better than anyone else, you will be able to build the best agent for that domain. — Harrison Chase

## Mentioned Harrison Chase, LangChain, LangSmith, LangGraph, Deep Agents, Managed Deep Agents, LangSmith Engine, LangSmith LLM Gateway, LangSmith Trajectories, LangSmith Fine-Tuning, LangSmith Custom Apps, SmithDB, smithtune, Rogo, Harvey, JPMorgan Chase, Parallel, Jev, SemIf, Fireworks, Baseten, OpenAI, Anthropic, Claude, Codex, Cursor, Lovable, Interrupt NYC

## Video description Harrison Chase, Co-Founder and CEO of LangChain, uses examples from Rogo, Harvey, and JPMorgan Chase to explain owning your intelligence, which means building domain-specific pieces in and around the model, and the three pillars it takes: an open, model-neutral harness such as LangGraph or Deep Agents, a loop that compounds what you learn from how people use your agents, and governance for internal agents.

He then walks through the LangSmith platform layer by layer: a new Managed Deep Agents release with auth primitives, user-level memory, and built-in web search via Parallel, decision models like Jev and SemIf through the LangSmith LLM Gateway, and three launches for observability and evals: LangSmith Trajectories, LangSmith Fine-Tuning with the smithtune CLI, and LangSmith Custom Apps. He closes with LangSmith Engine v2, which adds red teaming and tests its own fixes on LangSmith Deployment, and shares that Engine has scanned over 70 million traces and detected over 21,000 issues.

This session was captured at Interrupt New York City, the agent conference by LangChain.

LangChain products in this talk: LangSmith, LangSmith Engine, Managed Deep Agents, Deep Agents, LangGraph, LangSmith Deployment, LangSmith Sandboxes, LangSmith LLM Gateway, LangSmith Trajectories, LangSmith Fine-Tuning, LangSmith Custom Apps, SmithDB

Speakers: Harrison Chase, Co-Founder and CEO, LangChain

Chapters: 0:00 How agents went from prototypes to the application layer 2:43 Why teams want to own their intelligence now 4:25 What it takes to own your intelligence 7:22 How to organize your people and process around agents 9:20 What LangChain builds for runtime, evals, and intelligence 12:42 How Managed Deep Agents handles auth, memory, and web search 15:51 What a decision model like Jev is good for 18:00 Routing every model call through the LangSmith LLM Gateway 19:50 Why trajectories make agent debugging faster 22:57 Turning trajectories into optimized models with smithtune 24:44 Building custom UIs on your LangSmith data 26:03 How LangSmith Engine clusters issues and proposes fixes 28:04 How Engine v2 tests its fixes and red teams your agent 30:45 What LangChain wants Engine to take on next

Resources: → LangSmith Engine: https://www.langchain.com/langsmith/engine → Managed Deep Agents: https://docs.langchain.com/langsmith/python/managed-deep-agents-overview → LangSmith Fine-Tuning (smithtune): https://docs.langchain.com/langsmith/smithtune → LangSmith: https://www.langchain.com/langsmith → Interrupt, the agent conference by LangChain: https://interrupt.langchain.com

[Source](https://www.youtube.com/watch?v=950byF7njfw)

Reading summary

At Interrupt NYC, LangChain CEO Harrison Chase argued that raw foundation models are commoditizing, making domain-specific harnesses, compounding evaluation loops, and proprietary data pipelines the primary sources of defensibility for enterprise AI agents. Shift toward Owning Domain Intelligence (1:42) Chase outlined an industry evolution from experimental prototypes in 2023 to multi-step autonomous agents transforming enterprise application layers by 2026. Because base model outputs are increasingly commoditized and token expenses continue to rise, organizations are prioritizing 'owning their intelligence' by building custom execution scaffolding, context routing, and fine-tuning around base models. Examples include Rogo's financial model-routing harness, Harvey's specialized legal model fine-tuning, and JPMorgan Chase's internal 'Jarvis' assembly line. Runtime Scaffolding, Decision Models, and Gateways (10:38) Agent runtimes require three interconnected layers: bespoke business logic, execution harnesses, and underlying cloud infrastructure. To streamline enterprise deployment, LangChain introduced Managed Deep Agents, which bundles the open-source Deep Agents harness with managed cloud infrastructure, enterprise authentication primitives (Connections, Identity, Channels), user-level memory, and native web search via Parallel's MCP. Chase also highlighted the emergence of specialized decision models like Jev and SemIf, which output discrete scores or categorical choices rather than text to handle low-latency guardrails, routing, and in-loop evaluations. Managing these workloads enterprise-wide relies on centralized routing through the LangSmith LLM Gateway, which enforces cost controls, rate limits, and cross-provider fallbacks. Trajectory-Based Observability and Custom Interfaces (19:50) Because complex agent traces can involve thousands of tool calls, LangChain introduced LangSmith Trajectories to normalize messy execution logs into clean, conversational message streams. Stored on the specialized SmithDB database, these trajectories simplify debugging and human annotation while providing standardized datasets for post-training. The new `smithtune` CLI allows engineering teams to filter production traces directly from LangSmith and route them into compute providers like Fireworks or Baseten for model fine-tuning. For non-technical subject matter experts evaluating outputs, LangSmith Custom Apps enables users to prompt a coding agent to build internal web tools that inherit existing role-based and attribute-based access controls. Automated Remediation with LangSmith Engine v2 (26:03) To accelerate the continuous evaluation cycle, LangSmith Engine v2 acts as an autonomous background engineering layer across collected trace data. According to Chase, Engine has scanned over 70 million traces and surfaced more than 21,000 issues. The v2 release introduces proactive fix verification—spinning up preview environments to reproduce a bug, writing a code patch, and verifying the fix against a second deployment branch before delivering an issue report. Additionally, Engine v2 performs automated red teaming by simulating adversarial inputs to catch prompt leaks, hallucinations, and off-brand outputs, automatically creating golden evaluation datasets prior to production deployment. Chase positioned LangChain's three-tier stack—runtime harnesses, trajectory-driven observability, and autonomous intelligence—as the blueprint for enterprise teams to maintain independent, domain-adapted agent architectures, noting that Engine v2 will become available for self-hosted enterprise environments with bring-your-own-key support in v17.

Source quotes

[4:25] Harrison Chase: If you can define what better and what best looks like for your domain better than anyone else, you will be able to build the best agent for that domain.

Original transcript

[0:06] Welcome to Interrupt, everyone. [0:08] Isn't this awesome? This is crazy. [0:10] Yeah, [0:13] this is the first time we've done this in New York. [0:16] So, we've done this in SF the past two years, but we're bringing it here for [0:20] a full day. We've got an awesome lineup. [0:23] We'll be announcing a lot of awesome stuff, and we've got a ton of great fireside [0:27] chats with folks in the field, [0:29] and then a lot of great talks from people that are building agents and practitioners. [0:33] So, a lot to hopefully learn from. [0:35] So, I want to talk a little bit about the state of AI now and where [0:39] we think it's going. But before diving into that, I just want to do a brief [0:43] kind of like tour over the past few years, because it seems like things move [0:47] so fast, but it's only like 3 years ago was when all of this kind of like really [0:52] kicked off. Actually, a little bit before that was when LangChain launched, [0:56] but in 2023, we were building prototypes. [0:59] In 2024, we were shipping some of these simple applications to production, [1:02] but they were still like single LLM calls. [1:05] 2025, we started exploring agents. [1:08] And really this year, we've started seeing that agents have transformed the [1:12] application layer. And so, a lot of the talks that you'll hear from today are [1:16] companies that are doing exactly that. [1:18] They're taking these LLMs and they're really building them into differentiated [1:22] experiences. So it's not just the model that they're providing to their customers. [1:27] It's everything around the model. [1:29] And so we're seeing that these domain-specific agents, they're not just wrappers. [1:33] They're providing real value to folks. [1:35] And people are having to build a lot of things around the model to properly [1:39] take advantage of that. And so, calling out a few. [1:42] So Rogo, we'll do a fireside chat with them later today. [1:44] They had a great blog where they wrote about how they built a specialized harness [1:48] with model routing to provide a domain-specific agent for finance. [1:52] Harvey, who we heard from at some previous Interrupts, has been talking a lot [1:56] about how they've fine-tuned models for specialized tasks to get the best performance [2:00] on legal domain. And then JPMorgan Chase, we'll hear from them later today. [2:04] A little bit of a different use case, internal use case, but customizing [2:08] the harness around the model to provide a lot of internal advantages for the [2:12] agents that they're building as part of their Jarvis assembly line. [2:16] So, all of these are examples of companies taking the model, building a [2:22] lot of stuff around it to make it truly theirs, and the phrase that [2:26] has been going around to really exemplify all of this is owning your intelligence. [2:31] When we talk about owning your intelligence, we talk about all of these examples. [2:34] Basically building a lot of domain-specific things in and around the models [2:39] to make it truly best for the domain that you're trying to tackle, and really [2:43] owning your intelligence. And we think there's a few reasons why this has become [2:46] a phrase that's top of mind for so many people these days. [2:50] First, token costs are rising a lot. [2:53] People are spending more and more on AI, and so as they do so, they want to make [2:57] sure that they're getting the proper ROI, and being able to control and monitor [3:01] token costs is really important. And so owning your intelligence in that regard [3:05] is a key factor. Second, we see open models becoming more and more popular. [3:10] So this provides not only a really cheap alternative in order to own the token [3:14] costs more, but it also lets you customize things more. [3:17] And so as these base open-weight models get better and better, you can start [3:21] to post-train them. You can start to use them in specialized ways, and that [3:25] just makes it easier than ever to really own your intelligence. [3:28] And then the last thing that I think has become pretty clear over the past [3:32] year is that the models themselves are becoming somewhat commoditized, and it's [3:36] all of this stuff around the model, in particular the data, whether that is actual [3:40] kind of like context you're providing or some of the domain-specific paths [3:45] that you're baking into the harness. [3:47] This is becoming the differentiator to provide a lot of value for folks. [3:50] And so because of all of these factors, we're seeing people want to own their [3:53] intelligence more and more. [3:55] And that's how we think about our mission as a company. [3:57] It's always been this way. [3:58] LangChain got started a month before ChatGPT, but really took off after, when people [4:04] saw that came out and they were like, "I want that, but for my industry, for [4:08] my vertical." And then a big value prop of LangChain has always been [4:12] kind of like model optionality. [4:14] So from the beginning, people were building these domain-specific things that [4:18] were naturally kind of like decoupled from the model. [4:21] And so this mission of kind of like owning intelligence has been something [4:25] we've thought a lot about, and we think there's three core pillars to this. [4:30] And this is what we help folks with. [4:32] So the first core pillar is open and controllable harness. [4:36] And so there's two kind of like key parts to this. [4:40] One, we think the harness needs to be model neutral. [4:43] So you shouldn't be coupled to a particular model. [4:45] This is both offensive and defensive. [4:47] When the best new model comes out, whether it's from OpenAI or Anthropic or [4:52] TypeSafe with some of their Jev models, you want to be able to switch your [4:55] harness to that as quickly as possible. [4:57] And so your harness needs to be able to adapt to the best new model. [4:59] It's also defensive. You don't want to get locked into a single model provider [5:03] so that they can raise rates and things like that. [5:05] The other part of this open and controllable harness is again what we talked [5:09] about in terms of making these agents like really domain-specific. [5:12] All of that is about bringing specialized context in particular ways to the [5:17] model, to bring the right context at the right point in time, and [5:21] you need to really control what the harness does, and what that control looks [5:24] like could look different for different scenarios. [5:26] So we've had LangGraph for a while, which is an open source framework for really [5:29] creating these directed graphs, and that's a really controllable way of bringing [5:33] this harness. We also have Deep Agents now, and I'll talk about that later, but [5:37] that's another kind of like open source model harness. [5:40] We have a concept of middleware in there, which is a different way of injecting [5:45] like decisions to make and things to do into this core agentic loop. [5:49] And so regardless of what it is, you need a way to control really [5:53] in real detail what's going into the model. [5:57] The second core pillar that we think makes up owning your intelligence is compounding, [6:01] specifically compounding the intelligence that you get from when people use [6:05] your system. So when you launch an agent, that is far from having a successful [6:09] agent. You need to learn from how people use the agent, what the agent does, [6:14] see where it makes mistakes, and being able to compound that agent's intelligence, [6:19] whether manually, by seeing errors and updating the code or updating the prompt, [6:23] or more automatically, whether that's with memory or prompt optimization things. [6:28] This compounding loop is really important to build into your systems and [6:31] to own. If you can define what better and what best looks like for your domain [6:37] better than anyone else, you will be able to build the best agent for that [6:41] domain. And so having eval... [6:48] ...intelligence is a really good way to build these domain-specific agents. [6:52] And then lastly, governed. [6:54] And so this is really crucial for internal agents. [6:56] We're seeing a ton of companies transform how they do internal operations. [7:00] But when you have a lot of agents running around, you need to make sure that [7:02] they're properly governed. [7:03] You need to make sure that you know how many exist, that you can control costs, [7:07] that you can handle auth reliably. [7:09] So different people accessing it only have access to what they should have [7:13] access to. And so we see that governance is becoming an increasingly big topic [7:17] around owning your intelligence. [7:18] And so this is how we think about the three core pillars of owning intelligence. [7:22] And we think there's a bunch of different ways that you can get there. [7:25] And so calling three out at a high level: people, process, and technology. [7:30] And we're going to talk most about technology during the rest of the keynote, [7:33] but I want to touch really briefly on people and process, because [7:36] those are also needed to transform your organization into an organization that [7:40] can own its intelligence. So touching really briefly on people, this is roughly [7:44] how we see a lot of organizations being built. [7:46] There's a ring one of like core platform engineers. [7:49] These provide a lot of tools for other people to build agents. [7:53] When we talk about governance, when we talk about this kind of like eval [7:57] compounding loop, these provide the platform that other agent building teams [8:02] internally can use to build and govern and compound their agents. [8:06] Outside of that, we see agent engineers. [8:08] So, this is a great term to describe the weird hybrid mix of data scientist [8:12] and engineer and machine learning engineer that makes up the people who are [8:15] driving these agents. And it's a new skill set, and we see a lot of people from [8:19] different disciplines coming in to be agent engineers, consuming things from [8:23] the core platform team. And then lastly, there are a ton of subject matter [8:27] experts inside and outside the company who are often the best people to decide [8:31] what good looks like for an agent. [8:32] And so having a platform that can touch those non-developers at the edges is [8:37] super important. And so people, and kind of like setting up your organization [8:41] to have this type of model, is what we see people doing. [8:46] We also see people either implicitly or explicitly adopting a kind of like [8:51] new process for building agents. [8:52] As I mentioned before, it's very iterative. [8:54] You first launch an agent, it's not going to be good. [8:57] You need to iterate. And so there's this life cycle, this loop that people [9:00] are adopting in order to really get agents to be good enough. [9:04] Roughly speaking, it's build, test, deploy, monitor, and then crucially, when [9:07] you take those monitor learnings and you bring them back into the build step. [9:12] And of course, you want to do all of this in a governed way. [9:14] And we'll come back to this a little bit later, but people are doing this [9:17] type of process all the time to improve their agents. [9:20] And then I want to talk about the technology. [9:22] And when we think about the technology, and the technology that we help build [9:25] and provide, there's really like two core parts to it. [9:28] So one is the runtime. And this is how we got started, with the open source [9:31] LangChain libraries, and continue to do open source things with LangGraph [9:34] and Deep Agents. And then of course we want to build agents that don't just [9:38] run locally on your laptop but run in the cloud and can serve everyone [9:43] inside your org, or the customers that you're building for. [9:46] And so there's a deployment aspect to this runtime as well. [9:50] And then there's this observability and eval part, which is really crucial for [9:53] the compounding layer. And so being able to see what agents are doing and measure [9:57] how good they are behaving is really important. [10:00] So these are the two kind of like core pillars of things that we build. [10:04] And recently we've also focused on kind of like a third part, which is an intelligence [10:07] layer that sits above them. [10:09] So now when we think about LangSmith, our commercial platform, we think about [10:12] it serving the runtime aspect, serving the observability and eval aspect, and [10:17] then building in intelligence across the platform to really help people utilize [10:22] these components and speed up that process, that agent development life [10:26] cycle that I talked about earlier. [10:27] And so, for the next few minutes, I want to walk through these kind of [10:31] like three core parts of our platform and talk about some of the cool things [10:35] that we're doing and where we think the future's going. [10:38] So, I want to start with runtime. [10:40] And so, what does the runtime for kind of like an agent look like? [10:43] When we think of an agent, we generally think of there being three components. [10:46] There's the business logic. [10:48] So, this is things like instructions, tools, skills, hooks. [10:51] These are things that you guys have to provide when you're building your domain-specific [10:54] agents. What skills for a financial agent might look like are [10:58] very different than what skills for a legal agent might look like. [11:01] And so this is the business logic that everyone out here will [11:05] provide to the agents. Then there's the harness. [11:08] And so the harness is what takes the model, takes a lot of this domain-specific [11:11] context, and basically wires it all up so that the model can see the [11:16] right context at the right point in time. [11:17] This can be a very kind of like custom harness that you might build in LangGraph, [11:23] or it can be a more kind of like general purpose harness, which we're seeing [11:26] more and more of. And so that's why we built Deep Agents. [11:28] And so Deep Agents is our newest open source library. [11:31] We launched it about a year ago. [11:33] It's an off-the-shelf agent harness. [11:35] It learns a lot from what coding agents do. [11:38] So it has an execution environment with a file system and sandbox. [11:41] It does a lot of context management in ways that coding agents do it, with compaction, [11:46] but it's built so that you can build these domain-specific agents. [11:51] And then lastly, there's the infrastructure. [11:53] And so if you take this agent and it's running locally on your laptop, great. [11:56] You want to bring it to the cloud. [11:57] There's all of these components that you're going to need to add in so that [12:00] it can run remotely. You'll want a runtime in a server that has durable [12:04] execution. You'll want some way to access tools securely. [12:07] You'll want some way to run this code that it writes. [12:10] And we see a lot of agents writing code even if they're not coding agents. [12:13] You'll need to access models. [12:14] And so there's all this infrastructure that goes into it as well. [12:18] And so these are a lot of the components that we've built into LangSmith in [12:20] pieces. So we have LangSmith Deployment, that's really around the agent runtime. [12:25] We have LangSmith Sandboxes for running code, in beta. [12:29] We have an LLM Gateway in LangSmith, and I'll talk about that, but that's [12:32] for access to models. And then a tool server is something that we're working [12:35] on for easy access to tools. [12:38] And so there's these different components that make up the infrastructure. [12:42] And one of the things that we launched in beta at last Interrupt, but I want [12:46] to spotlight real quick, is Managed Deep Agents. [12:48] So Managed Deep Agents basically takes this harness and the managed infrastructure [12:53] and combines it into one. And so now when you're building an agent, you just [12:57] provide your business logic, and then you get the harness and managed infrastructure [13:01] together. And so that's a lot easier to get started with than having to go [13:04] business logic to harness to assembling all these three or four or five different [13:08] pieces of infrastructure. And so we're really, really excited about the [13:12] Managed Deep Agents platform as a way to accelerate time to value and speed [13:16] to kind of like production for a lot of these agents. [13:20] There's a few things that we're announcing today as part of a new [13:24] Managed Deep Agents release that we think will help even more with this. [13:27] So first, one of the hardest parts of building these, especially internal [13:31] kind of like company agents, is auth, and making sure that the right people [13:35] can access and use the agent, but also that when they use the agent, if the agent [13:39] connects to Notion or Jira, it's connecting either on behalf of the user or [13:44] as the agent's kind of like fixed set of identity. And what the right [13:48] answer is might depend, and so we're introducing a lot of different primitives [13:52] in Managed Deep Agents to facilitate this type of connections. [13:56] So first is connections. These are basically tools, when you have a tool [14:01] server, which is how Managed Deep Agents connects to tools. [14:04] You can specify how you want your end users to auth to those connections. [14:08] So we're using auth there. Related to that, we have this idea of identity [14:12] for agents. This just defines the agent to be either kind of like user permissioned [14:17] or service permissioned on a whole. [14:19] And then third is channels. [14:21] So a big part of building these agents is exposing them to people. [14:24] Slack is a massive one. When you have agents in Slack, how do you deal with [14:27] auth in like a shared channel? [14:29] If I message it in a shared public channel and my coworker messages it, [14:33] whose auth is used for which runs? [14:35] A lot of that we're baking into off-the-shelf channels in Managed Deep Agents [14:39] to hopefully make the auth problem a lot easier. [14:42] Second, we're adding more things around memory into Managed Deep Agents. [14:46] And so, Managed Deep Agents has always had a concept of agent-level memory, [14:50] but we're now adding in user-level memory. [14:52] So this means that it can remember things specific to the individual users [14:55] that it interacts with. This is really powerful for customizing agents. [15:01] And then third, we're adding in built-in tools. [15:03] So one of the most common things that agents need to do is search the web. [15:08] And so we're adding in a built-in tool via Parallel so that agents can access [15:12] the web. And so now you no longer have to go get a separate API key for that. [15:15] You can just use your LangSmith API key, use it with Managed Deep Agents, and get [15:19] access to the Parallel web search MCP. [15:23] And so putting this all together, Managed Deep Agents, we think, is the easiest [15:26] way to go from the business logic to an agent that's actually running in production. [15:31] We've been working with a number of early design partners on this, and we've [15:35] seen really good results, especially for just getting an agent up and running [15:38] and having it be like this company harness, and that's how we think about [15:42] Managed Deep Agents, is this company harness that makes it really easy to get started [15:46] building and sharing agents inside an org... [15:51] ...that I want to spotlight is decision models. And so over the weekend I, as [15:55] I'm sure many of you did, played around with Jev. [15:58] So it launched over the weekend. [15:59] It's a really interesting new type of model. [16:01] And so what exactly is Jev? [16:03] It's a decision model. What does that mean? [16:05] It basically means that it makes decisions. [16:07] You send it a state. You send it a list of questions. [16:09] It can then score those questions. [16:11] And there's different things that you can have it score on. [16:13] So it can either be like a boolean or an actual score or a categorical choice, [16:17] but it basically makes decisions. [16:18] It can't generate text, but it can make decisions. [16:21] And so even though it's more limited, we think this is really interesting. [16:24] We think there are a bunch of different types of models and a bunch of different [16:27] types of agent harnesses that will be built. [16:29] So we think absolutely there will be these frontier models that are driving [16:33] these really complex harnesses, and that's what you're going to use in like [16:36] Claude Code and Codex and Deep Agents. [16:38] These take advantage of these really frontier models. [16:41] But there's also a ton of other decisions that need to be made. [16:45] And that's where we think a decision model can be really helpful. [16:47] And maybe these are small things that actually go along with the frontier model, [16:51] or maybe they empower the LangGraph cognitive architectures and [16:56] routing between different things. [16:57] And so calling out a few of the things that we think these decision models [17:00] are really useful. One is eval. [17:03] So either running after the fact or in the loop. [17:05] If you have an agent doing work, if you can score it quickly according to some [17:09] criteria as it's running, that's really powerful, because then you can catch [17:12] mistakes in real time. Two, guardrails. [17:14] So flagging user input really fast. [17:17] So I was talking to some people yesterday, and we were talking about guardrails. [17:20] One of the downsides of guardrails is that it always adds latency. [17:22] If now we can add them in a really low latency way, that's really cool and [17:26] powerful. And so we're excited about it for that. [17:27] And then lastly, model routing as well. [17:29] Choosing which agents or which models or which skills or tools to load. [17:33] We think things like Jev will be really, really useful for that. [17:36] So you can access Jev through the LangSmith LLM Gateway. [17:40] You can also access an open source decision model that we're hosting there [17:44] called SemIf. There's been a massive explosion in these open source decision [17:49] models since Jev launched. [17:50] We think this is a new type of model that there will absolutely be a lot of [17:54] options for, and we're really, really excited to power and host a lot [17:58] of these through our gateway. [18:00] Speaking really quickly of our gateway, this is the last thing in the runtime [18:03] that I want to spotlight. So, we launched LangSmith LLM Gateway in beta about [18:07] a month or two ago. There's two main things that I want to highlight. [18:11] One, we try to make it so that the gateway can be used in all of the LLM applications [18:15] that you're building, whether they are customer-facing or internal. [18:19] So, specifically, we normalize everything through OpenAI and actually [18:24] Anthropic message format. So you can access it there. [18:26] But we also allow straight pass-through in case there are new features that [18:29] come out in these APIs and you just want to take advantage of. [18:32] And then we also have a bunch of coding agent integrations, [18:35] so Codex and Claude Code and Cursor. [18:38] Again, we think you should be routing all of your traffic, for governance [18:41] and cost control reasons. We think you should be routing all of it through [18:44] a single LLM gateway. And so that means it needs to integrate with every application [18:47] that's using AI. And so we want to make it really easy to plug into all [18:52] of those. And then the main kind of like features that we're focusing on: [18:56] one is cost controls, having cost visibility, but also setting spend limits [19:00] for different teams, different users, different API keys. [19:03] Two is rate limiting. So making sure that there's not a single application [19:06] that's really throttling or taking up a lot of usage there. [19:09] And then third is fallbacks. [19:11] So you can fall back to different model providers when you get an error. [19:15] And then one of the things that we'll be launching soon is actually more stateful [19:18] fallbacks. So if you notice that requests are failing and Anthropic [19:23] might be down, you don't even try them for the future. [19:25] You just switch over to OpenAI for the next 10 minutes or 30 minutes, and make [19:29] it really configurable in that way. [19:31] And so we want to make LangSmith LLM Gateway the most kind of like secure and [19:35] reliable way to monitor and govern and fall back on different LLM providers [19:41] when you're using them. So that's everything that we've got going on in the [19:45] runtime space. I now want to talk about observability and eval. [19:50] This is a space that we think is crucial to really power that compounding loop, [19:55] and where we've done a ton of stuff over the past year, but especially [19:59] over the past month. So, we think traces are really the core of this loop. [20:06] When you think about this loop, this process that I mentioned, we think traces [20:09] being the system of record and showing what the agent does at all steps are [20:13] really, really important to be able to power this kind of like learning loop. [20:17] When we've talked about traces previously, we've talked about basically runs, [20:22] traces, and threads. And so a run is like a single LLM call. [20:26] A trace is like a single invocation of an agent. [20:29] And then a thread is a series of traces, basically. [20:32] So when you have a multi-turn interaction. [20:35] So these are all really useful primitives, but it's become pretty clear over [20:38] the past year or so that there's a more useful standard for how to represent [20:45] what agents do. And this is what we're calling trajectories. [20:48] And so a lot of the agents these days, whether they're coding agents or [20:52] first-party agents, run in a similar way where they basically run an LLM in a loop [20:56] and compile this list of messages. [20:58] This list of messages is a standard format that can be used in a lot of really [21:03] interesting ways. And so today, one of the things that we're excited to launch [21:06] is LangSmith Trajectories, a way to standardly interact with these trajectories [21:11] and use them in a bunch of places. [21:13] So let's take a look. [21:49] So what do trajectories do? [21:50] We think there's two main benefits to them. [21:53] One, they make debugging a lot faster. [21:55] So if you've used LangSmith before and you've had a really complex trace, ones [21:59] that have thousands of kind of like iterations, you might have noticed that [22:02] it's pretty hard to parse through all of that. [22:04] So the LangSmith Trajectories in the UI have a much more user-friendly interface [22:08] for viewing what's going on. [22:10] They'll look basically as if you're using like a coding agent or a chat agent. [22:14] We basically condense and display tool calls in the same way that those [22:18] do. So you can easily kind of like scan through them. [22:21] One big benefit of this is simpler annotation. [22:24] So now that you can see what's going on, you can leave feedback on them much [22:27] more easily. A third big benefit that we think is this actually makes it much [22:32] easier for post-training models. [22:34] So when you post-train models, you commonly have these trajectories that [22:38] are in this kind of like standard format. [22:40] And so we do a lot of work in the back end to parse these really complicated [22:45] kind of like trace formats that come in from all these different [22:50] SDKs and different coding agents and expose them in a standard trajectory format. [22:54] This can be used really, really easily in fine-tuning. [22:57] When we talk about fine-tuning, there's generally two pieces. [23:00] There's the data, and this will come from a lot of the traces and from a lot [23:03] of the trajectories, and then there's the inference and training infrastructure. [23:07] And so we want to make it as easy as possible to bring these two things together, [23:10] which is why we're also announcing LangSmith Fine-Tuning today. [23:14] So what does this mean? So you have all your data in LangSmith. [23:18] These are all the traces, which now we're porting into this trajectory format, [23:22] which is a much more condensed and standard representation of what's going [23:26] on. You can then use the smithtune CLI, which is a CLI that we're launching [23:31] today to basically select traces, filter down to relevant ones, pre-process [23:36] those traces in a way that makes them even more fine-tune ready, then pass [23:40] them into Fireworks or Baseten, which is the training and inference infrastructure [23:45] to do the actual training. [23:46] We can then evaluate the models from there, and as the output, [23:49] you get an optimized model. [23:51] And so we think this is a really powerful way to get started fine-tuning, whether [23:55] it's on agents or classification tasks or guardrails or anything like that. [23:59] We want to help you all take the most advantage of all the data that's in LangSmith. [24:03] And so we think trajectories power a lot of this. [24:07] Now, where we're storing these trajectories is SmithDB. [24:10] We launched this at Interrupt a few months ago. [24:12] This is a database that we built in house, and it's really good for agent workloads. [24:16] It's a lot faster to ingest and store and query a lot of these workloads. [24:20] We've made trajectories show up really nicely in them. [24:24] So for example, I was talking about what they might look like earlier. [24:27] This is a screenshot. You can see that we condense a lot of the long tool [24:30] calls that are going on, but you can click into those, expand those. [24:33] You can click into an LLM call and jump directly into that. [24:36] So we try to make it look as nice as possible, but we know there's a ton of [24:40] different ways to actually visualize the data that you're working with. [24:44] And so that's why today we're also launching LangSmith Custom Apps. [24:48] Let's watch a video on that. [25:16] Hey. [25:31] So, LangSmith Custom Apps you can basically think of as Lovable, but for your [25:36] data that's already on LangSmith. [25:38] So, you can chat with it. It will create a completely custom app. [25:41] We actually open sourced our design system so that the coding agent [25:45] that we use under the hood can leverage it, and it comes built in with all the [25:49] permissions for sharing with your team and respecting the workspace or RBAC [25:53] or ABAC boundaries that you've set up. [25:55] And so we think this is a really powerful way to bring humans into the loop [25:58] and inspect data. [26:00] So that's what we're launching for observability and eval. [26:03] The last thing I want to talk about is intelligence. [26:06] So I mentioned earlier that there is this process, this life cycle for building [26:10] agents. You build an agent, you then test it, deploy it, monitor. [26:13] One of the things that we launched previously at Interrupt is LangSmith Engine. [26:18] So LangSmith Engine is an engine that sits on top of this whole loop. [26:22] Specifically, it sits on top of your traces in a tracing project. [26:25] It runs in the background, and it basically does all the things that an AI engineer [26:30] would do. It looks at issues. [26:32] It tries to cluster them. It creates this like issue board. [26:34] It tries to fix it with code. [26:35] It tries to add an evaluator, add things to a dataset. [26:38] And so this is something we've been working on for the past six months or [26:43] so, and we've been working on a lot of improvements, and so today we're packaging [26:46] up a lot of those and announcing Engine v2. [26:49] So let's take a quick look at a video, and then I'm going to walk through a [26:52] few of the exciting things in here after that. [28:04] So I want to call out a few of the things in there. [28:07] First, I want to call out that Engine can now test its fixes proactively. [28:10] So what exactly does this mean? [28:12] So this integrates really nicely with LangSmith Deployment, which we talked about [28:15] in the runtime. So specifically, Engine now will spot issues in traces. [28:20] It will spin up a preview branch on your deployment. [28:24] It will confirm that that issue is in fact an issue. [28:27] It will then make a fix, spin up a preview branch for that fix, run it against [28:32] that to confirm that it's fixed, and then basically present it as an issue that [28:36] you can go in and see all that evidence that that issue actually was real and [28:40] that the fix now fixes it. [28:42] So again, this integrates right now with LangSmith Deployment, and that's how [28:46] we do it there. Another thing that I want to call out is red teaming. [28:50] So previously, Engine only identified issues that existed in traces that people [28:54] were sending to it, that were happening in the wild. [28:57] With red teaming, Engine can now simulate different issues and then see what [29:02] happens. So specifically, Engine will, and again this integrates with LangSmith Deployment [29:06] for now, which is how we know how to call it. [29:08] So LangSmith Engine will basically have access to the deployment. [29:12] It will spin up different issues, different hypotheses, run these against the [29:17] deployment, identify which ones produce bad results, where bad could [29:22] be it learned something that it shouldn't have. [29:25] It produces a bad result. It is off-brand in some way. And then it will [29:30] flag any issues to you. So, this is really good for identifying places where [29:35] your agent fails. [29:37] It's also really good for creating an initial eval dataset. [29:40] So, we've heard for a while that creating evals is really hard and really painful. [29:44] And so, this helps automate a lot of that. [29:46] This can absolutely be used before you even ship your agent into production [29:49] and before those issues even occur. [29:52] Another thing I want to call out is the performance improvements that the team [29:56] has made. So, we've worked really hard on making it both better at finding [30:00] issues, better at fixing issues, and then also making it a lot cheaper as well. [30:04] And so, a lot of these are packaged up into Engine v2, which is live as of [30:08] now. Engine has already scanned over 70 million traces and detected over 21,000 [30:14] issues. It's used by a lot of the fastest growing AI companies out there to [30:19] really speed up that iteration cycle. [30:21] That iteration cycle is everything when you're building agents. [30:23] And so anything we can do to help speed that up, we will absolutely do. [30:27] And the last thing I want to announce is that Engine is now also available [30:31] in self-hosted with bring your own key. [30:34] And so this will be rolling out in V17 of self-hosted, which should be out in [30:38] a week or two. So we're excited to bring this to even more of the customers [30:42] that we work with. [30:45] So this is everything that we've launched and how we think about the technology [30:48] that we build. We think there's this runtime, and we think about this as the [30:52] open source but also deployments, and more and more we're seeing that these [30:56] managed agent runtimes are becoming really powerful for going to production. [31:00] We have observability and eval. [31:02] We think a lot about how we can make this more agent native with trajectories, [31:05] how we can make this more customizable with Custom Apps, and how we can help [31:08] you use this data to tune models to better serve your workloads. [31:13] And then lastly, we have this intelligence layer that sits above, and we want [31:16] to make Engine do a lot of the things that agent engineers and AI engineers [31:20] are doing now. And so the scanning of traces and the red teaming are [31:24] just the start. All of this we want to build to help you guys build domain-specific [31:30] agents and really own your intelligence. [31:32] Again, whether that is agents that you're building and putting in products [31:35] or agents that you're building to transform the internal operations of your [31:39] company. And so you'll hear from a lot of speakers and panelists later today [31:43] about different ways to do that, different techniques, different examples of [31:47] what that looks like. We're really excited to be doing Interrupt in New York. [31:51] We're really thankful for all of you guys to be joining us. [31:54] I think we're going to have a fantastic day, and so I'm going to step off for [31:58] a little bit, and I'd like to welcome up Jake, who will be our MC for the day. [32:02] Thank you guys all for coming.