LangChain Engineer Urges Shift to Citizen Evaluators in Enterprise AI

LangChain工程师呼吁企业AI转向“公民评估员”模式 当非技术专家被迫编写智能体代码，而非使用结构化评估工具将领域知识转化为机器可执行的标准时，企业AI部署便会失败。 Joel Johnson 概述了在复杂企业环境中扩展智能体 AI 的实用框架，强调组织瓶颈往往源于人员和流程，而非单纯的技术。他提出了智能体开发生命周期（ADLC）与三环利益相关者模型，展示了技术开发者与领域专家之间应如何协作。Johnson 主张不应强求非技术领域专家成为“公民开发者”，而应将重心转向“公民评估员”，赋能业务专家通过直接和间接评估工作流来定义质量。结合 Chime 的财务助手智能体以及 LangChain 的 GTM Slack 智能体等实际案例，该演讲展示了持续评估和直观抽象如何将领域知识转化为可投入生产的 AI 系统。 企业扩展智能体 AI 的组织瓶颈主要源于人员和流程，需依托智能体开发生命周期（ADLC）及三环模型促进技术开发者与领域专家的协作。 主张将非技术专家的重心从成为“公民开发者”转向“公民评估员”，赋能其通过评估工作流将领域知识转化为机器可执行的质量标准。 通过 Chime 财务助手和 LangChain GTM Slack 智能体等实际案例，证明持续评估与直观抽象是将领域知识转化为生产级 AI 系统的关键。

LangChain deployed engineer Joel Johnson argues that successful enterprise agent adoption requires shifting focus from 'citizen developers' to 'citizen evaluators' within a structured operating model. Under this framework, engineering teams maintain infrastructure and runtime logic while non-technical domain experts guide and govern agent behavior through evaluation abstractions. Structuring Enterprise AI Through the Three-Ring Model (0:32) Johnson describes enterprise agent deployment as an iterative Agent Development Lifecycle (ADLC) consisting of build, test, deploy, and evaluate/monitor loops governed by IT and information security policies. Moving agents from local prototypes to production, however, is typically bottlenecked by organizational alignment rather than model mechanics. To resolve this friction, Johnson introduces the Agentic Operating Model, which divides organizational participants into three stakeholder tiers: Ring 1 (platform infrastructure engineers), Ring 2 (AI engineers developing runtime code and integrations), and Ring 3 (non-technical subject matter experts and business stakeholders). Because agents operate across broad, non-parametric surfaces such as prompt engineering and tool definitions, production readiness depends on establishing clear collaboration channels across all three rings. Replacing Citizen Developers with Citizen Evaluators (8:01) Drawing on military leadership principles regarding capability alignment, Johnson cautions against pushing non-technical domain experts to become 'citizen developers.' Assigning domain specialists to build production agent code risks platform stability and results in mutual confusion, as software engineers lack task context while subject matter experts (SMEs) lack technical engineering skills. Johnson argues enterprises should instead cultivate the 'citizen evaluator.' In this setup, SMEs are not asked to build systems, but rather to define what constitutes successful performance. Organizations must provide intuitive abstractions, such as visual flowcharts and decision trees, that translate natural language domain expectations into executable code assertions, evaluation rubrics, and LLM-as-a-judge criteria. Gating Compliance and Alignment: Chime's Financial Assistant (10:07) Evaluations serve as an artifact of organizational consensus, demonstrating that business experts and technical builders agree on acceptable agent behaviors while shielding enterprises from legal liability and wasted token expenses. Johnson notes that evals remain essential regardless of model trajectory: they provide structured guardrails if model capability plateaus, and act as superalignment mechanisms if capabilities scale rapidly. This dynamic is illustrated by Chime's deployment of 'Jade,' an automated financial assistant operating under strict regulatory constraints. Rather than providing business staff with no-code agent builders, Chime had SMEs define domain taxonomies using flowcharts. Technical teams used LangSmith and Giza to bootstrap concrete output datasets for SME review, converting business preferences into programmatic evaluation suites that doubled as regulatory compliance sign-offs. Capturing Direct and Indirect Production Feedback (11:52) To maintain reliable agent performance without placing administrative burdens on users, Johnson differentiates between direct and indirect evaluation methods. Direct evaluations rely on explicit criteria, transforming SME rules into automated test harnesses or LLM-as-a-judge assertions to relieve AI engineers from chasing edge cases individually. Indirect evaluations leverage telemetry captured during natural human-in-the-loop workflows. Within LangChain's internal Go-To-Market Slack agent, account executives routinely edit or overwrite draft responses prior to sending. The system automatically records these edits as implicit negative evaluations paired with target corrections, while specialized evaluators monitor user dwell times and post-generation behavioral patterns to identify probable errors without requiring explicit feedback clicks. Johnson concludes that enterprise agent development is fundamentally an exercise in translating unscalable human domain expertise into scalable machine workflows, cautioning that organizations cannot shortcut deep task understanding. While no-code builder tools are widespread, Johnson identifies a critical industry deficit in evaluation tools designed specifically for non-technical domain experts, leaving business-side evaluators under-supported.

LangChain部署工程师Joel Johnson提出，企业规模化落地AI智能体的核心瓶颈在于组织协同与流程，而非单纯的模型技术；企业应当建立结合合规治理的智能体开发生命周期（ADLC），并将非技术业务专家的定位从“平民开发者”转向“平民评估者”，通过评估体系将领域专业知识转化为可执行的系统规范。 双重框架：生命周期与三层利益相关者 (0:32) Johnson指出，企业构建生产级智能体需要依托两个核心认知模型：智能体开发生命周期（ADLC）与智能体运营模型（Agentic Operating Model）。ADLC借鉴传统软件开发范式，由构建（Build）、测试（Test）、部署（Deploy）以及评估与监控（Evaluate & Monitor）四个环节构成持续改进闭环，并全程受制于IT与信息安全（InfoSec）的统一治理。与传统机器学习仅需标注数据或编写PyTorch代码不同，智能体拥有广阔的非参数化改进空间（如提示词工程与技能定义），能够让团队将初始原型打磨至生产可用水平。 在组织协同上，智能体运营模型划分出三个同心利益相关者圈层：维护底层基础设施与部署通道的平台团队（Ring 1）、编写智能体逻辑与运行时的AI工程师（Ring 2），以及深谙业务细微规则的领域专家与业务人员（Ring 3）。Johnson认为，技术本身的运作并不构成最大障碍，人与流程的脱节才是阻碍项目迈向生产就绪状态的主要症结。 角色重塑：从平民开发者转向平民评估者 (8:01) 针对工程师缺乏业务上下文、业务专家缺乏工程能力的组织断层，Johnson援引美国海军陆战队“依官兵能力合理指派任务”的原则，主张企业应停止推动非技术人员转型为“平民开发者”（Citizen Developer）。他认为由财务等非技术业务人员直接构建智能体极易破坏系统稳定性并引发安全隐患；相反，企业应将业务专家转型为“平民评估者”（Citizen Evaluator），通过抽象工具帮助他们将业务语言转化为代码断言、评分标准（rubrics）以及大模型评审（LLM-as-a-judge）规范。 在战略层面，Johnson认为评估（evals）是连通三大圈层的关键桥梁。无论AI模型未来能力是停滞不前还是走向全面超智能，评估均不可或缺：前者需要评估来搭建防护栏与支架以弥补模型短板；后者则需要评估来确保系统的安全性与对齐。完善的评估套件不仅是验证智能体是否安全合规、避免法律风险与Token浪费的防护闸门，更是证明业务专家与技术工程师就理想行为达成一致的组织对齐凭证。 双模评估实操：Chime案例与交互遥测 (10:07) 在企业落地实践中，金融科技公司Chime在构建财务助手“Jade”时采用了以评估为中心的路径。面对严格的金融合规约束，Chime并未向业务人员提供无代码智能体开发工具，而是让领域专家使用流程图和决策树定义领域分类规范；工程师则利用Giza与LangSmith引导生成示例数据集，让业务专家对具体输出进行判定，最终将反馈转化为兼具评估与合规签批功能的测试套件。 此外，Johnson结合LangChain内部嵌入Slack的销售（GTM）智能体，将评估划分为两种运作模式：直接评估（Direct Evaluation）依靠专家显式定义的规则与代码断言，解脱工程师追踪琐碎边缘案例的负担；间接评估（Indirect Evaluation）则通过工作流中的隐式交互遥测实现。例如，当销售人员修改或覆盖智能体生成的草稿时，该操作会被系统自动捕获为负向评价与修正目标；系统还可通过分析用户停留时间等交互行为的感知错误评估器，在无需用户手动点赞点踩的情况下持续积累真实反馈数据。 赋能业务圈层与人类专业知识的抽象化 (15:52) Johnson强调，当前AI行业虽充斥着各类无代码智能体开发工具，但专供非技术专家使用的评估工具却存在严重供给真空，使得业务圈层（Ring 3）成为目前企业AI战略中投资最匮乏的部分。为高效提取业务专家的反馈，企业必须在其日常办公环境中提供直观、无侵入的图形化界面（如流程图与结构图），让业务专家能够顺畅表达正常与异常的业务边界，而非逼迫其进入复杂的工程开发环境。 从更深层次的技术本质来看，构建智能体实质上是对人类专业知识的抽象过程。Johnson指出，人类的领域经验无法通过常规手段实现机器级别的规模化扩展，而智能体正是将这些难以扩展的专业经验转化为自动化、可规模化机器流程的产物。企业构建智能体的前提是对具体工作任务建立扎实透彻的认知，这一梳理与抽象过程不存在任何投机捷径。 Johnson总结认为，企业成功落地智能体是一项需要将平台运维、算法工程与领域评估各置其位的跨学科组织工程。然而，行业目前在面向业务专家的评估工具链建设上仍显滞后，企业若要摆脱脆弱的原型阶段并建立可靠的生产级系统，关键在于持续补齐对业务评估者的工具赋能与流程协同。

## Background 本视频由 Ai4 于 YouTube 平台发布，记录了 LangChain 驻场部署工程师 Joel Johnson 在大会上的主题演讲。页面简介指出，随着 Agentic AI 显著拓宽了组织内能够影响系统成效的人员范围，本场分享结合 LangChain 的智能体开发生命周期（ADLC）与智能体运营模型，重点探讨如何协调人员与流程以支撑企业级智能体项目的成功落地。 ## TL;DR 演讲系统拆解了企业级智能体开发生命周期（ADLC）与三大协同圈层（平台工程师、AI 工程师及业务专家）的运营模型。讲者犀利指出企业应当摒弃盲目鼓吹“平民开发者”的误区，转向赋能“平民评估者”，以评估（Eval）作为弥合业务直觉与机器执行鸿沟的核心抓手。通过 Chime 的金融助手 Jade 与 LangChain 内部 GTM Agent 等实战案例，演讲深入阐述了直接评估与间接评估的构建路径，强调智能体的本质是将人类深度的业务专长转化为可无限扩展的自动化能力。 ## Atmosphere & style 整场演讲节奏明快、生动干练，讲者巧妙融合了斯坦福学术背景与海军陆战队服役经历，以风趣自嘲与充满力量感的军旅隐喻牢牢抓住观众注意力。叙事风格直击痛点、金句频出，既有对“公民开发者”迷思的辛辣解构，又有充满启发性的落地心智模型拆解，使原本枯燥的企业级架构议题变得极具感染力与互动性。 ## Key takeaways - [00:04](https://www.youtube.com/watch?v=-r94uLZB-6s&t=4s) 企业级 Agent 的落地成功不仅依赖技术本身，更取决于人与流程的协同；LangChain 提出 ADLC（Agent 开发生命周期）与智能体运营模型作为核心心智框架。 - [01:54](https://www.youtube.com/watch?v=-r94uLZB-6s&t=114s) 智能体运营模型将组织划分为三大核心圈层：第一圈层平台工程师（底层基建）、第二圈层 AI 工程师（智能体架构与模型编排）与第三圈层业务专家（领域知识与场景需求）。 - [04:02](https://www.youtube.com/watch?v=-r94uLZB-6s&t=242s) 评估（Eval）占据 ADLC 的半壁江山，不仅贯穿离线回归测试与线上实时监控，更是连接平台团队、AI 工程师与业务专家的核心纽带。 - [07:11](https://www.youtube.com/watch?v=-r94uLZB-6s&t=431s) 评估套件本质上是组织协同的“试金石与金丝雀”，反映了组织能否跨越“业务专家的领域自然语言”到“机器可执行代码”之间的转化鸿沟。 - [07:11](https://www.youtube.com/watch?v=-r94uLZB-6s&t=431s) 企业在落地 Agent 时普遍存在角色错配，即让缺乏业务认知的工程师编写业务逻辑规则，让精通业务的业务专家去写代码，导致分工严重失调。 - [09:31](https://www.youtube.com/watch?v=-r94uLZB-6s&t=571s) 企业应停止盲目迷信“平民开发者（Citizen Developer）”，转而聚焦赋能“平民评估者（Citizen Evaluator）”，帮助业务专家准确表达评估标准。 - [10:07](https://www.youtube.com/watch?v=-r94uLZB-6s&t=607s) Chime 开发 Jade 金融助手案例表明，通过展示具体样本引导业务专家反馈，远比强推无代码构建器让业务人员凭空搭建 Agent 更为有效且利于合规。 - [11:52](https://www.youtube.com/watch?v=-r94uLZB-6s&t=712s) 将人在回路（HITL）无缝嵌入 Slack 等原生工作流，不仅能极大降低使用摩擦，还能沉淀海量真实的修正行为数据用于模型迭代反哺。 - [13:16](https://www.youtube.com/watch?v=-r94uLZB-6s&t=796s) 评估体系应由“直接评估”与“间接评估”双轮驱动：前者依靠工具将业务专家的抱怨转化为评分规则（Rubric），后者通过交互遥测（UX Telemetry）无感捕获用户纠错信号。 - [17:58](https://www.youtube.com/watch?v=-r94uLZB-6s&t=1078s) 构建 Agent 的技术本质是将人类个体无法直接规模化的领域专业认知，抽象封装为机器级别的自动化流程；Agent 是彻底理解任务后获得的终极奖励，没有捷径可走。 ## Detailed walkthrough ### [00:04](https://www.youtube.com/watch?v=-r94uLZB-6s&t=4s) 个人背景与 LangChain 的企业级 Agent 定位 Joel Johnson 作为 LangChain 的驻场部署工程师（Deployed Engineer）登台演讲。他自嘲道，在加入 LangChain 之前，他在斯坦福 AI 实验室（Stanford AI Lab）从事应用 AI 与机器学习研究，拿到了计算机科学硕士和 MBA 学位——“不过大家千万别因为这个（MBA）就对我抱有偏见”。在此之前，他还曾在美国海军陆战队负责密码情报工作，因此对如何在“令人窒息的官僚主义机构”中落地技术工作有着极其深刻的体会，这恰恰成了他在企业环境中探索 Agentic AI 的天然经验衔接。 在介绍 LangChain 时，他不仅提到了广为人知的开源项目矩阵——包括经典的 LangChain 开源库、LangGraph 以及 Deep Agents，还重点介绍了公司的商业化产品 LangSmith。LangSmith 的核心定位是帮助企业跨越早期的原型构建阶段，真正打造出稳健的生产级智能体，并在此过程中为企业级客户扮演长期战略技术顾问的角色。面对短短 20 分钟的演讲时间，Joel 直言今天的分享略带一点“标题党”，因为单次演讲不可能涵盖企业级 Agent 的所有宏大命题，所以他将聚光灯集中在两个核心心智模型上：Agent 开发生命周期（Agent Development Lifecycle, ADLC）与智能体运营模型（Agentic Operating Model）。 ```mermaid flowchart TD accTitle: Agent 开发生命周期循环 subgraph DevCycle ["Agent Development Lifecycle (ADLC)"] direction LR B["Build 构建"] --> T["Test 测试"] T --> D["Deploy 部署"] D --> M["Monitor & Eval 监控与评估"] M --> B end Gov["Governance 组织治理与风控"] -.-> DevCycle ``` ### [01:54](https://www.youtube.com/watch?v=-r94uLZB-6s&t=114s) ADLC 与智能体运营模型（Agentic Operating Model） Agent 开发生命周期（ADLC）与传统的软件开发生命周期（SDLC）一脉相承，本质上是一个用于持续构建与迭代智能体的闭环系统。在构建（Build）阶段，开发者通常借助开源框架在本地（如 `localhost:8080`）完成原型；但开发者往往不敢拿自己的职业声誉打赌这个原型能否在生产环境稳定运行，因此必须引入测试（Test）阶段以摸清系统的脆弱点；确认边界后，再通过类似 LangChain 部署基础设施将 Agent 发布到真实世界（Deploy），并通过实时评估（Monitor & Eval）捕捉线上故障，开启下一轮迭代。而这一切技术流转，都在信息安全（InfoSec）与 IT 部门所要求的治理（Governance）保护伞下运行。 然而，单纯的技术流程不足以确保项目成功。技术从不是孤立发生的，阻碍技术人员高效产出的往往是周围的“人与流程”。在传统的机器学习时代，开发者的参与方式非常狭窄，要么做数据标注，要么写 PyTorch；但在如今的 Agentic 时代，改进 Agent 的表面积被极大拓宽——撰写一段 Prompt、补充一个技能（Skill）等大量非参数化手段，就能把一个体验糟糕透顶的 Agent 提升至接近生产可用。因此，LangChain 提出了智能体运营模型，明确界定了协同构建 Agent 的三大核心利益相关方圈层： - **第一圈层（Ring 1 - 平台工程师）**：负责底层基础设施建设与平台级运维。 - **第二圈层（Ring 2 - AI 工程师）**：负责核心智能体架构、工作流编排与模型逻辑。 - **第三圈层（Ring 3 - 非技术构建者/业务专家）**：拥有实际业务场景知识的领域专家与最终使用者。 ### [04:02](https://www.youtube.com/watch?v=-r94uLZB-6s&t=242s) 为何评估（Eval）是贯穿一切的核心抓手 在探讨企业落地时，评估（Eval）被选为最关键的切入案例，原因主要有三点： 1. **占据 ADLC 的半壁江山**：评估直接覆盖了 ADLC 四大步骤中的两步（离线回归测试与线上实时监控），没有可执行落地动作的“可观测性”是毫无价值的。 2. **连接三大运营圈层**：从平台团队、AI 工程师到非技术业务方，每一个圈层都对“Agent 应该如何工作”持有独特的评价标准与期望。 3. **技术发展的刚性需求**：不论未来 AI 模型的智力水平是出现增长放缓（需要通过 Eval 不断兜底修正来使其可用），还是迅速演化到“天网（Skynet）”级别的超级智能（需要 Eval 来实现超级对齐与行为约束），Eval 都是不可或缺的防线。 评估的形态跨越了从粗放经验到严密工程的整个光谱：低维度如所谓的“拍桌子测试”（Desk Test）——将 Agent 部署给坐在开敞式工位隔壁的同事，听到对方因使用体验极度崩溃而怒拍桌子，这在广义上也算一种评估；高维度则如 Terminal-Bench 等需要行业专家协同制定的标准化、高度结构化的正式基准测试。 - **性能度量（Performance Measurement）**：作为基础工具，量化 Agent 在各类任务中的准确率与执行效果。 - **改进的载体（Substrate for Improvement）**：为微调与后训练（Post-training）提供数据挖掘支持，精准筛选出高价值的执行轨迹（Traces）。 - **准入把关机制（Gating Mechanism）**：为合规与风控部门提供防线，“Token 消耗固然昂贵，但谁也不想让 Agent 的最终代价变成‘Token 账单外加一张法院传票’”。 ### [07:11](https://www.youtube.com/watch?v=-r94uLZB-6s&t=431s) 作为组织协同“金丝雀”的评估与角色错配困境 在更高阶的抽象视角下，评估套件（Eval Suite）本质上是“组织对齐的试金石与金丝雀”。它绝非一段仅仅跑在 Trace 上的测试代码，而是考验组织能否将正确的人聚在同一个房间里、能否就期望与非期望的行为达成共识，以及能否把业务专家（SME）的大白话顺畅翻译成可执行的评估代码。 ```mermaid flowchart LR accTitle: 业务认知向代码转化的评估鸿沟 SME["业务专家 SME<br/>（精通业务 / 缺乏工程能力）"] -- "SME English<br/>（领域语言）" --> Bridge{"组织对齐与翻译<br/>High Eval-yrian"} Bridge -- "Executable Evals<br/>（可执行评估代码）" --> Eng["AI / 平台工程师<br/>（精通技术 / 缺乏业务直觉）"] ``` 在实际企业运作中，最致命的顽疾正是分工错配。Joel 引用海军陆战队的领导力法则警告道：“知己知部，因能授任（Know your command and employ them in accordance to their capabilities）——你绝不能让炊事兵端着枪去冲锋碉堡。”但现实中的企业恰恰陷入了这种荒谬怪圈： - **技术工程师（圈层 1 与 2）**：精通代码与架构，却对具体的终端业务逻辑一头雾水，却被迫编写业务判断规则。 - **非技术业务人员（圈层 3）**：对业务痛点与正确操作了如指掌，却完全不知道如何把这些直觉落地为技术指标。 - **双方的错位共识**：双方都在困惑为什么自己被逼着去做对方份内的工作。 当今行业过度鼓吹所谓的“公民开发者（Citizen Developer）”，将构建工具的压力一股脑推向前台业务端，但“构建（Building）”与真正的“软件工程开发（Developing）”存在本质鸿沟。Joel 对此给出了 LangChain 风格鲜明的犀利观点（Hot Take）：企业应当停止对“公民开发者”概念的盲目迷信，正如信息安全团队所担心的那样，企业绝不需要让财务部的普通员工去从头开发底层的系统组件，而是需要理顺专业分工。 ### [09:31](https://www.youtube.com/watch?v=-r94uLZB-6s&t=571s) 从“平民开发者”转向“平民评估者”：企业面临的真实脱节 企业在推进 Agent 落地时最常见的失败模式，并不是缺乏构建工具，而是陷入了一种错误的执念——生怕非技术人员“自己动手胡乱搭建 Agent，结果搞垮了整套核心系统”。Joel Johnson 直言，企业当前最该聚焦的绝非所谓的平民开发者（Citizen Developer），而是“平民评估者”（Citizen Evaluator）。在服务众多企业级客户的过程中，他发现最普遍的断层在于：业务人员脑海中往往非常清楚“什么才是好的结果”，却完全无法将其清晰阐述并转化为机器可执行、且能针对 Trace（调用追踪）进行自动校验的评估指标。 与其把全部精力耗费在把构建权限硬推给第三环（Ring 3）的非技术业务人员，企业更应该思考的是：如何构建合适的工具和抽象层，帮助这些业务专家把对业务优劣的直觉准确表达出来？只有当业务人员能够定义出具体的验收标准，工程团队才有可能引导系统持续、稳定地向“好”演进。 ```mermaid flowchart LR accTitle: 平民评估者的双向循环机制 subgraph Ring3["第三环：业务专家 (SMEs)"] A["业务认知：何为优秀结果"] --> B["业务语言表达：分类法/图表"] end subgraph Ring12["第一、二环：平台与 AI 工程师"] C["脚手架与自动化评测工具"] --> D["可执行 Evaluator / 自动化流水线"] end B -->|"转化与对齐"| C D -->|"持续反馈与迭代"| A ``` --- ### [10:07](https://www.youtube.com/watch?v=-r94uLZB-6s&t=607s) Chime 的 Jade 案例：用分类法与样本引导填补“业务到评估”的转化鸿沟 金融科技公司 Chime 打造名为“Jade”的金融助手 Agent 的经历，是这一理念的最佳佐证。涉足财务助理或金融建议类 Agent 时，合规与风控部门通常会“吓出一身冷汗”，唯恐稍有不慎引发法律诉讼。Chime 的工程团队并没有直接丢给业务人员一个无代码（No-Code）Agent 构建器，让他们自己去糊一个合规 Agent，而是坐下来询问：“你们眼中的问题到底是什么？你们希望这个 Agent 究竟达成什么目标？” 业务领域的专家（SME）天然更习惯使用流程图和架构图来表达逻辑。工程师团队据此利用 Giskard 和 LangSmith 快速搭建了一批冷启动引导数据集（Bootstrap Datasets），并将直观的样本直接展示给业务专家。Joel 强调，人类大脑面对具体事物进行“反应式反馈”，远比凭空凭想象去构思一个抽象系统要容易得多。团队通过展示真实样例确认“这是否符合你们的预期”，逐步收敛并确立了真正的评估标准（Evals）。这一举措不仅让评估机制真实反映了合规团队的偏好，更直接作为合规上线前的检查清单（Check-off），使 Jade 得以大幅提前交付生产。这一案例表明： - 第三环的非技术人员核心需求不是“去写代码构建”，而是“准确表达业务逻辑”。 - 工程师的核心职责是搭建桥梁，将业务专家的主观认知转化为 LLM-as-a-judge、评分细则（Rubric）、断言代码（Assertion）或定制评估器。 - 弥合业务思维到机器可执行代码之间的“转化鸿沟”，是 Agent 能够真正持续迭代改良的前提。 --- ### [11:52](https://www.youtube.com/watch?v=-r94uLZB-6s&t=712s) LangChain 内部的 GTM Agent：将人在回路无缝融入自然工作流 在 LangChain 内部，被广泛高频使用的典范是他们自研的 GTM（Go-To-Market，市场拓展）Agent。Joel 调侃道，虽然自己都不敢细算这个 Agent 究竟撬动了多少净新增 ARR（年度经常性收入），但其内部热度极高，销售代表（AE）们极为重度依赖它。这款 Agent 之所以能够爆发式普及，根本原因在于它“直接驻留在工作发生的原点”——日常协作的 Slack 之中。 当 Agent 输出有误时，团队成员不需要跳转到复杂的后台界面去排错，而是直接在对话流中进行纠正。这种将人在回路（Human-in-the-loop）无缝嵌入自然工作习惯的交互方式，既不突兀也不打断思路，就像平时与实习生或同事协作交流一样流畅。正是这种极低的上手摩擦力与激活门槛，迅速激发了全员的使用黏性，进而积累了海量的修正数据并精准注入 Trace 日志。Joel 提醒道，低门槛能催化初期采纳，但用户绝不会长期忍受一个持续输出劣质结果的工具；唯有借助这些埋点修正数据不断反哺优化模型，才能使 Agent 的效能长期维持。 --- ### [13:16](https://www.youtube.com/watch?v=-r94uLZB-6s&t=796s) 直接评估与间接评估：从“宣泄变准则”到行为遥测分析 为了系统化梳理评估体系，Joel 将评估抽象概括为直接评估与间接评估两大支柱。二者相辅相成，共同解决了传统评估中“强行让用户打大拇指点赞点踩如同拔牙般艰难”的行业痛点： | 评估类型 | 核心驱动源 | 实现机制与典型场景 | 核心价值与工程挑战 | |:--- |:--- |:--- |:--- | | **直接评估 (Direct Evaluation)** | 业务专家（SMEs）主动定义 | 借助评估工程技能（Eval Engineering Skills）或编码助手，将业务叙述转化为代码断言或在线评估规则。 | 核心挑战在于“把一段情绪化的抱怨转化为客观的评分标准”（From a rant to a rubric）。让业务自主评估，解放底层工程师。 | | **间接评估 (Indirect Evaluation)** | 终端用户的交互遥测数据 (UX Telemetry) | 追踪自然工作流中的行为，例如用户在 Slack 中直接编辑覆盖 Agent 给出的草稿、交互停顿时间（Dwell Time）等。 | 将交互行为转化为隐式评估标签。结合自定义微调的感知错误评估器（Perceived Error Evaluator），无需显式点赞即可洞察模型缺陷。 | 直接评估的核心是“工具赋能”。AI 工程师的精力应当聚焦于系统运行时（Runtime）与持久执行（Durable Execution）等深层技术架构，无暇应对海量特定场景下的偶发业务错误。因此必须提供迭代工具，帮助业务专家把“解释变成断言，宣泄变成准则”。而间接评估则应当像产品体验分析（UX Analytics）一样运作，利用高频日常使用中沉淀的纠错动作，在不打扰用户的前提下源源不断捕获评测信号。 --- ### [15:48](https://www.youtube.com/watch?v=-r94uLZB-6s&t=948s) 技能与任务对齐：释放第三环（Ring 3）的业务潜能 企业构建 Agent 体系的最佳实践，是打通从第三环（非技术业务人员）流向第一环（核心平台与基础设施团队）及第二环（AI 应用工程师）的信息通路。这种流动依赖于业务专家最习惯的交互形态——图表、业务流程图以及无感融入的工作节点。如果整个链路简单顺畅，业务专家能够毫不费力地表达什么是好、什么是坏，企业在“评估这场硬仗中就已经赢了大半”。 Joel 借用《指环王》的梗风趣地谈到：“这可不是中土世界，并不存在能统御一切的至尊魔戒（It's not one ring to rule them all）。”在各司其职的模型中： - 第一环和第二环工程师负责搭建基础设施、攻坚底层工程挑战。 - 第三环业务人员聚焦深耕业务领域逻辑，负责评估好坏。 - 业界虽然充斥着各种无代码 Agent 构建平台，但第三环在“评估工具链”上的建设却长期严重缺乏投入。 想要摆脱平民开发者浪潮带来的权责混淆，必须参考 LangChain 提出的“Agent 运营模型”（Agentic Operating Model），将多利益相关方复杂的协作过程精准映射到 Agent 开发生命周期（ADLC）之中。通过跨学科视角的角色与技能重新对齐，才能确保所有人都在做自己最擅长的事情。 --- ### [17:58](https://www.youtube.com/watch?v=-r94uLZB-6s&t=1078s) 抽象领域专长：Agent 是深度理解任务后的终极奖励 在谈到 LangChain 与 LangSmith 的产品矩阵如何提供全链路支撑、统一治理标准后，Joel 引用业界洞见总结了 Agent 的技术本质：开发 Agent 的根本目的，是将人类个体那些无法直接机器规模化扩展的专业业务认知（Subject Matter Expertise），抽象并封装为能够以机器级别无限扩展的自动化流程。 “你必须将 Agent 视作彻底吃透一项具体任务后，在终点获得的奖赏。这件事情绝对没有任何捷径可走。”无法清晰解构业务过程，就不可能构建出合格的 Agent。在演讲的尾声，Joel 鼓励大家以更加开放、创新的方式去构建评估逻辑，并向与会者推介了 LangChain 全新推出的认证 Agent 工程师（Certified Agent Engineering）培训体系，呼吁更多工程师和业务骨干共同补齐行业在 Agent 评估与落地工程中的关键拼图。 ### [19:17](https://www.youtube.com/watch?v=-r94uLZB-6s&t=1157s) 学习资源推荐与大会闭幕致谢 Joel 在演讲结尾向与会者推荐了相关的学习与实践资源。他表示，LangChain 提供了海量的配套资源，旨在帮助大家更好地构建并深入理解智能体开发生命周期（ADLC, Agent Development Lifecycle）以及更广泛层面的 Agent 体系。他指着屏幕打趣道，虽然幻灯片上当前展示的这个项目写着“立减 50%”（get 50% off）并不是免费的，但官方依然为开发者准备了大量免费资源可供参考学习。 在查看会议计时时，Joel 幽默地发现自己已经严重超时，笑着向台下致歉：“我刚才没注意到计时器，噢，我严重超时了，实在不好意思大家！”他诚挚地感谢了现场观众的时间与倾听，并表示希望这门干货满满的分享能以充满活力的方式为各位的 Ai4 大会之行画上圆满句号。最后，他热情地邀请大家保持联系，表示自己演讲结束后会在会场后方停留交流，并期待在当晚的 Afterparty 上与大家进一步畅聊。 - **核心资源支持**：提供围绕 ADLC（智能体开发生命周期）与 Agent 落地的一系列深度指南与实操资料。 - **互动交流**：讲者在会后留步于会场后方进行答疑交流，并邀约参会者在 Ai4 晚间聚会（Afterparty）继续深入探讨。 ## Notable quotes > [00:04](https://www.youtube.com/watch?v=-r94uLZB-6s&t=4s) 不过大家千万别因为这个（MBA）就对我抱有偏见。 — Joel Johnson > [04:02](https://www.youtube.com/watch?v=-r94uLZB-6s&t=242s) Token 消耗固然昂贵，但谁也不想让 Agent 的最终代价变成‘Token 账单外加一张法院传票’。 — Joel Johnson > [07:11](https://www.youtube.com/watch?v=-r94uLZB-6s&t=431s) 知己知部，因能授任（Know your command and employ them in accordance to their capabilities）——你绝不能让炊事兵端着枪去冲锋碉堡。 — Joel Johnson > [09:31](https://www.youtube.com/watch?v=-r94uLZB-6s&t=571s) 企业当前最该聚焦的绝非所谓的平民开发者（Citizen Developer），而是“平民评估者”（Citizen Evaluator）。 — Joel Johnson > [10:07](https://www.youtube.com/watch?v=-r94uLZB-6s&t=607s) 你们眼中的问题到底是什么？你们希望这个 Agent 究竟达成什么目标？ — Joel Johnson > [13:16](https://www.youtube.com/watch?v=-r94uLZB-6s&t=796s) 从一段情绪化的抱怨转化为客观的评分标准（From a rant to a rubric）。 — Joel Johnson > [15:48](https://www.youtube.com/watch?v=-r94uLZB-6s&t=948s) 这可不是中土世界，并不存在能统御一切的至尊魔戒（It's not one ring to rule them all）。 — Joel Johnson > [17:58](https://www.youtube.com/watch?v=-r94uLZB-6s&t=1078s) 你必须将 Agent 视作彻底吃透一项具体任务后，在终点获得的奖赏。这件事情绝对没有任何捷径可走。 — Joel Johnson > [19:17](https://www.youtube.com/watch?v=-r94uLZB-6s&t=1157s) 我刚才没注意到计时器，噢，我严重超时了，实在不好意思大家！ — Joel Johnson ## Mentioned Joel Johnson, LangChain, LangSmith, LangGraph, Deep Agents, Stanford AI Lab, Ai4, Chime, Jade, Giskard, Slack, Terminal-Bench, 美国海军陆战队 ## Video description Joel Johnson - Deployed Engineer, LangChain Agentic AI has dramatically expanded the scope of who can meaningfully influence the success and performance of AI systems within your organization, which presents both challenge and opportunity. In this session, Joel Johnson (Deployed Engineer; MS Computer Science (AI), MBA -- Stanford University) will highlight critical lessons learned through LangChain's Agent Development Lifecycle and the newly released Agentic Operating Model. This talk goes beyond just the tech and offers practical insights on how to align people and processes to complement, not conflict with, your enterprise's Agentic initiatives. [Source](https://www.youtube.com/watch?v=-r94uLZB-6s)

[0:04] Joel Johnson: 不过大家千万别因为这个（MBA）就对我抱有偏见。 [4:02] Joel Johnson: Token 消耗固然昂贵，但谁也不想让 Agent 的最终代价变成‘Token 账单外加一张法院传票’。 [7:11] Joel Johnson: 知己知部，因能授任（Know your command and employ them in accordance to their capabilities）——你绝不能让炊事兵端着枪去冲锋碉堡。 [9:31] Joel Johnson: 企业当前最该聚焦的绝非所谓的平民开发者（Citizen Developer），而是“平民评估者”（Citizen Evaluator）。 [10:07] Joel Johnson: 你们眼中的问题到底是什么？你们希望这个 Agent 究竟达成什么目标？ [13:16] Joel Johnson: 从一段情绪化的抱怨转化为客观的评分标准（From a rant to a rubric）。 [15:48] Joel Johnson: 这可不是中土世界，并不存在能统御一切的至尊魔戒（It's not one ring to rule them all）。 [17:58] Joel Johnson: 你必须将 Agent 视作彻底吃透一项具体任务后，在终点获得的奖赏。这件事情绝对没有任何捷径可走。 [19:17] Joel Johnson: 我刚才没注意到计时器，噢，我严重超时了，实在不好意思大家！

[0:04] Joel Johnson: please don't hold that against me. [8:01] Joel Johnson: know your command and employ them in accordance to their capabilities [8:01] Joel Johnson: Gerald from accounting building an agent and breaking my entire system [4:05] Joel Johnson: You don't want the cost of your agent to be tokens and a lawsuit

[0:04] Hi everybody, my name is Joel Johnson. I'm  a deployed engineer here at Langchain. Um, [0:08] before Langchain, I was doing applied AI  and ML research at the Stanford AI lab where [0:13] I got my masters in computer science and also  my MBA. But please don't hold that against me. [0:17] Um, before that I was in the Marine Corps where  I did cryptologic intelligence. So very familiar [0:21] with what it's like to do technical work in a  very uh crushingly bureaucratic organization. [0:26] So I feel like that's a natural segue to talk  about agentic AI in the enterprise of all places. [0:32] So let's get started. Um maybe more importantly  we should be talking about who is lang chain [0:37] and lang chain is you know we're the purveyor of  such great smash hits like the actual lane chain [0:42] oss library link graph and deep agents but we're  also a provider of a commercial offering called [0:48] link smmith which is specifically designed to help  you get past that early build stage and actually [0:53] create better agents. We help companies small and  large build very very robust production agents and [0:59] also serve as long-term strategic technical  advisers along their agentic AI journey. Now [1:05] I must admit I have done a little bit of clickbait  here in the sense that uh we only have 20 minutes [1:11] together and we could not possibly talk about the  full extent of what agents in the enterprise are. [1:16] So, we're really just going to cover a small part  of agents in the enterprise. And specifically, [1:21] we're going to be talking about the agent  development life cycle and the agentic [1:24] operating model, which for us are the mental  models that we're using to better understand how [1:28] to build agents faster. And specifically, how to  build agents faster under the context of being in [1:34] an enterprise where it's not just the technology,  but it's also the people and the processes that [1:38] surround that technology that determine whether  something gets done. We'll finish this off by [1:42] talking about eval specifically as a case study  for us to understand how agents in the enterprise [1:49] actually work in practical reality. So that's our  plan. We're going to go small to understand big. [1:54] And let's go ahead and get started. So the agent  development life cycle is very very similar to [1:59] the software development life cycle. It's the  understanding that there is some kind of loop [2:03] that you should be setting up for you to be  able to effectively build and improve agents. [2:08] It starts of course with build which is  what our open source frameworks are for. [2:11] And after you've built something that's running  on localhost 880 um you're sitting there thinking [2:16] well gosh I don't necessarily know if I have if  I want to stake my professional reputation on [2:20] whether this thing works in production. So  you do a little bit of testing and you say [2:23] okay well now I have a better understanding  of where this breaks. Um maybe we do want to [2:26] deploy it. So you get it out into the world. You  use uh maybe our lang deployments infrastructure [2:31] and then you start finding some evaluations to to  look and to monitor it in real time. And as you [2:37] find what's going wrong, you start the cycle all  over again. And then ideally for the infosc and [2:41] IT people in here, it's done under an umbrella  of governance. And that's exactly what we do. [2:46] So the agentic operating model, this is our  understanding and acceptance that the ADLC [2:53] that technology development doesn't happen on  its own. I imagine that at least one actually [2:59] most of you here if you're technical, if you're  a developer have probably thought in some part [3:03] of your career that uh tech like the techn is  fine but it's the people and the processes that [3:08] are getting in the way of you being able to do  your job as effective as you can. I'm seeing [3:11] lots of people smile. I can assume that that's the  case. So this is our recognition that with agents [3:16] especially where there are so many more surface  areas for an agent to be improved like I come from [3:21] a classical machine learning background where  it's like you either labeling data or you can [3:25] write pietorch that's the only two ways to be  useful but now like you can just write a prompt [3:30] you can write a skill there's tons of these  non-parametric methods that you can use to take an [3:33] agent from absolute for lack of better terms crap  and make it actually somewhat production ready. [3:38] So, this is our recognition that there are  multiple stakeholder groups that touch an [3:43] agent and that you need to think about them when  building your agents. Just out of curiosity, um, [3:48] if it's kind of an eye test, I know, but  ring three for the non-technical builders, [3:52] how many of us here fall into that category?  We're like nontechnical, we're non-developers. [3:58] Okay, cool. So, now I know who to pander to. It's  definitely the people in rings one and wing two. [4:02] So, with that in mind, let's talk a little  bit about why we're going to be covering [4:05] Neval specifically. Well, first of all, it's  because they're two of the four steps of ADLC. [4:11] They're important. You need them for offline  regression testing. You need them for online [4:14] monitoring. They're they matter. Like they're  a fundamental part of actually improving your [4:18] agent. Observability without any actionability  is completely useless. Next, they cover all three [4:25] rings of the agentic operating model. So, every  single stakeholder that we talked about, platform, [4:29] AI engineers, non-technical stakeholders, they all  have some particular opinion about how an agent [4:33] should work. And lastly, I'm not too proud to  admit it's because they're hot. Everybody wants to [4:39] know about eval. They're interesting. They're uh  controversial. It's very it it makes for a great [4:44] presentation topic that allows us all to leave  this stage knowing a little bit more about evals [4:49] and make it slightly easier for you to justify  your expense report to Las Vegas. So, this is [4:54] just to kind of highlight this whole situation  in this very fastmoving space. No matter what [5:00] happens with with a with aentic AI, we're going  to need evals. Whether they like models have like [5:06] this logistic sort of fall off and they remain to  be kind of stupid and then you need evals to sort [5:11] of prop them up to make your agent helpful. You're  going to need that to actually productionize them [5:15] or it goes like full Skynet and is actively trying  to harm us. You need evals for super alignment. [5:20] You need something to say, is this agent actually  acting in my best interest and doing what I'm [5:25] expecting it to do. So across all circumstances,  whether it's production um or or early prototyping [5:31] or doofus and Skynet here, you're going to need  evals to some degree. So it's very important that [5:36] we come together and understand what these  things are and why they're important. Now [5:41] what are eval? Evals fundamentally are a measure  of agent performance. That's it. They come in a [5:47] variety of different levels of abstraction um  and also complexity. Uh the the basic one that [5:52] I like to term is the uh the desk test where  it's like let's say that you've deployed an [5:57] agent to somebody who sits next to you in like  an open concept tech office and you hear them [6:01] smacking the desk because it's so frustrating to  use your agent. That is technically an eval in [6:05] a way. Um this can also be something as highly  complex as something like terminal bench where [6:10] it's a very very structured highly organized  requires lots of experts to come together and [6:15] agree on it uh in a formal evaluation. Now this  is going to get more progressively abstract. [6:22] They're also a substrate for improvement, right?  Like if you have evaluations, you have the ability [6:25] to actually understand where your agent is falling  short and actually where you can improve it. [6:30] This is also for those of y'all who are interested  in post- trainining. This is another way to do it [6:34] because essentially these evaluations allow you  to identify and do the data mining to surface [6:39] interesting traces and interesting information  that you can use uh for post- training. [6:43] Now it's also a gating mechanism. Um, a lot of the  times if you're trying to build or put something [6:48] into production, you're going to have somebody  on the compliance side of the house saying, [6:51] "Hold on there, Tiger. Like, we have to understand  like how this thing is going to work and how we [6:55] can evaluate it to make sure that it's not  doing something bad." Tokens are expensive. [6:59] You don't want the cost of your agent to be tokens  and a lawsuit. So, gating mechanisms can be very, [7:03] very important here. And lastly, and maybe most  abstractly, eval is a canary in the coal mine. [7:11] And what do I mean by that? An eval suite is an  artifact of organizational alignment. It is not [7:19] just a set of code that you run against traces. It  is specifically saying, can you as an organization [7:25] get the right people into the same room?  Do you know who's relevant for this agent? [7:30] Can you agree on what desirable and undesirable  agent behavior looks like? It's a process issue. [7:35] And can you actually go from taking the  information and the way that a subject [7:39] matter expert thinks about right and wrong and  turn that into actual executable substrate. [7:44] In this particular case I am trying to dub some  interesting terms here so you can keep it in the [7:48] head. Can you convert SM English into evalu for  those of you who are watching Game of Thrones [7:53] high eval anybody? All right nice. Okay cool.  Um so this is the reality. This is the reality. [8:01] In the Marine Corps, we have a leadership  principle that's known as know your command and [8:04] employ them in accordance to their capabilities.  You do not want the cook charging the bunker. [8:09] You want somebody else who's trained to do that  to handle that job. But the problem is we are very [8:14] very commonly seeing a failure to assign the tasks  who are the most capable of accomplishing them. [8:19] This is what we're seeing. So in a bit more of a  clean diagram, this is what we're seeing. We have [8:25] engineers in platform ring one and two who are  they have the techn technical expertise they [8:30] don't know how to necessarily execute the end task  if this is like a subject matter expertise kind of [8:35] agent we have the non-technical users who know and  execute that end task directly and understand it [8:41] intimately familiar and I know we're all starved  for you know connection and common ground these [8:45] days but this is not the common ground that we  want they're both just confused why they're being [8:50] asked to do the other person's job I believe  this is a an artifact of of of the world that [8:56] we live in where we've pushed build out from the  infra out from the platform and out to the front. [9:03] There's a lot of people talking about the citizen  builder or the citizen developer, but I encourage [9:07] us all to think that there is a difference between  building and developing. They're functionally [9:12] different things. So, at Lang Chain, we have a bit  of a culture of hot takes. And this is my hot take [9:19] is that we need to focus a lot less on the citizen  developer. I I know there's at least one infosc [9:25] person in here who's like, "Yes, 100%. I totally  agree. I do not need like Gerald from accounting [9:31] building an agent and breaking my entire system."  We need to instead focus on the citizen evaluator. [9:37] This is the most common breakdown that I see  across all of my enterprise clients is that [9:42] there is an inability for people to articulate  what good looks like in their head into something [9:46] that is executable and can be evaluated against a  trace. So the citizen evaluator, what that means [9:52] is all of that effort that we're putting into  pushing building out into the third ring into [9:56] the nontechnical users. What if we instead put all  of that effort to build tooling and abstractions [10:01] that help them articulate what good looked like so  you can consistently drive towards what good is? [10:07] So what does this look like in practice? I see my  formatting has been messed up slightly here, but [10:13] let's talk about Chime. So they built this  agent called Jade. And Jade is a financial [10:17] assistant agent. And I know there's somebody in  here who's like sweating thinking about like, [10:20] "Oh, that sounds like really scary.  Financial assistance, financial advice, [10:23] like that's an opportunity again to get sued."  Um, and they also recognize that too. So, [10:28] what they started off was they didn't give like  a no code agent builder to their users and say, [10:33] "Hey, like here you go, like make a compliance  agent." They said, "Okay, what do you guys see [10:38] as the problem here? What do you guys trying to  act? What are you actually trying to accomplish [10:42] with this agent?" So, they started off with  a taxonomy. I mean subject matter experts [10:46] love flowcharts, love diagrams. So they started  off exactly with that. So the engineers in [10:52] response, they use the tool Gizard and also  Langmith, you know, like slight product plug there [10:58] um to bootstrap an example and actually show this  to them. It's a lot easier for you to react to [11:04] something than it is for you to drum up an idea of  what it could be in your head. So they created all [11:08] of these bootstrap data sets to show them, is this  sort of in line what you have? is this sort of in [11:12] line with what you're expecting? And they used  that to narrow in on what an actual eval looked [11:17] like and specifically create evals that reflected  their preferences, which also allowed them to [11:22] serve this as a basically a compliance check off  and actually push this agent out in production [11:26] a lot faster. So what's the lesson here? It's that  the ring three nontechnical people, they did not [11:32] need to build. They needed to articulate. They  needed their engineers to help them go from [11:37] I have this idea of what good looks like to  the actual code that gets executed whether it [11:41] was an LM as a judge, a rubric, an assertion, a  code evaluator, whatever it may be. That is the [11:46] biggest translational translational gap. And  without that, you cannot improve your agents. [11:52] Let's talk about another example. homegrown  inside lang chain. Go to market agent is a very [11:58] very highly adopted agent that we use within  uh within link chain. It I think I don't even [12:04] want to know how much total net new ARR this agent  touches but I use it extensively. My AEES love it. [12:10] Whatever these numbers are like they're they're  way higher than that. Now people love go to [12:15] market agent because it's easy to use. It's  right at the point of work. It lives in Slack [12:20] where we work. So when something comes through  that is wrong it's very easy to remediate. Now [12:27] this is extremely important. It is natural. It is  easy to use. And it's specifically it is natural [12:32] and easy to use because the corrections that are  being done and that the telemetry that's captured [12:36] from that is done at the natural points of human  in the loop. It's not obtrusive. It's not getting [12:40] in the way. It's doing exactly the way same way  you would work with an intern or work with your [12:44] with your with your co-orker. It was very very  natural to use. And that low activation energy [12:49] is what really drove adoption. So the low barrier  of entry and the low friction is what actually [12:55] allowed us to catalyze and seed enough information  to see where is this thing going wrong. [13:00] But it is the fact that we had this and these  corrections injected into the trace that allowed [13:04] us to actually improve the agent and sustain it  because people will use the agent if it's easy to [13:10] use in the beginning, but they're absolutely not  going to keep using it if it continues to be bad. [13:16] So this is sort of the way that I'm going to  distill what I just told you guys into the most [13:20] pity way I I possibly can. There are two different  abstractions you need to think about here. There's [13:24] direct evaluation and indirect evaluation. Direct  evaluation ismemes defining the eval directly. [13:31] They oftentimes need some tools. They need some  help to specifically go from that SM English to [13:36] that eval. Um, one one thing that we've worked  on recently is we released a eval engineering [13:41] skill that you can use with a coding agent  that can either develop hardware tasks if [13:45] that's uh what you're thinking for eval which is  not a ton of people these days but if you are that [13:49] sophisticated and hats off to you or if you're  using lang development online eval as well but [13:54] it is an ex is an understanding that this is an  iterative process that requires tooling just like [13:58] building does this is what they need help with.  You need to go from explanation to assertion, [14:03] like a rant to a rubric. Like that's that's  what the the current translational challenge is, [14:08] especially within larger organizations. And this  is an iterative process. The way that you win here [14:13] is by enabling those citizen evaluators to be  able to evaluate on their own to not need an [14:18] extensive amount of technical support from  the engineering because they have they have [14:22] things to build. AI engineers have to focus on  runtime. They have to focus on durable execution. [14:25] They can't just focus on very bespoke esoteric  failures of an agent. Now for indirect evaluation, [14:33] this is your telemetry from the agent UX. This  is exactly what I was talking about those inline [14:37] corrections with the go to market agent in  Slack. Every single time somebody edits the [14:41] response that's generated by go to market agent  and pastes over it a different answer. That is an [14:46] eval that is saying this did a bad job. I replace  it with this. It's very difficult. I mean anybody [14:52] here has probably known you get it's like pulling  teeth trying to get somebody to give you a thumbs [14:55] up or thumbs down for just some feedback in your  in your trace data. This is how you win that is by [15:01] actively injecting these correction points at the  natural human in the loop. Now these are generated [15:07] from adoption and usage. It's better to have more  data and also it has to align with the natural [15:13] human in the loop ergonomics as I said before.  So you can kind of think about this actually [15:17] as UX analytics. I think that we should also  generally be thinking about eval beyond just uh [15:23] somebody explicitly stating it. I mean, one thing  that we've done internally at Langchain is we've [15:27] customtuned a perceived error evaluator that just  basically looks at the most common responses or [15:33] sort of dwell times in response to an agent um and  and just allows us to better understand um is this [15:39] does this seem wrong from a standpoint that's not  just somebody saying this is wrong. So, I really [15:43] encourage everybody just to be a little bit more  creative about how they're understanding evals. [15:48] So, what's the the prime takeaway here? The best  practice is that you want to facilitate the flow [15:52] of information out of ring 3, out of those  non-technical users back to your core platform [15:58] and info team by using natural abstractions  like diagrams, flowcharts, things that are [16:04] familiar tome at their natural points of work.  I my subtitles are cutting it off unfortunately, [16:09] but natural points of work. It should not be an  it should not be obtrusive. If it's easy to adopt, [16:14] easy to use and you can also make it easier for  them to articulate specifically what is good and [16:19] what is bad. You have won the eval game. Now  this is what we're trying to go for. Ring one, [16:25] ring two, they build the tooling, they do engineer  stuff. Ring three, they do non-engineer stuff and [16:30] together they execute tasks that are suited for  them and abstractions that are natural to them. [16:34] That is exactly what we are driving to. But in  the era of the citizen developer, it can sometimes [16:39] feel very unclear who should do what. This is  our recommendation skill and task alignment. [16:46] So I want to take a pause here and just say again  this dynamic this this this idea of the citizen [16:53] evaluator this is only really feasible  to understand because I was able to use [16:57] the agentic operating model which is a lovely  PDF that I highly encourage everybody to read. [17:02] um and allowed me to transpose that understanding  of this multistakeholder mult like complicated [17:07] process environment over the ADLC. This is the  the secret sauce, right? This is the ability to [17:16] think about agents as an interdisciplinary  discipline. That's a bit redundant there. [17:21] Um but specifically try to align and realign  people to the jobs that they were meant to do. So [17:27] I know I talked a lot about ring three today, but  it isn't Middle Earth. It's not one ring to rule [17:31] them all. Everybody should be represented, but  Ring 3 is currently the most underinvested part [17:36] of this entire model and specifically in the area  of evaluations. Tons of no code agent builders [17:43] out there, of course, we have one at Lanith,  of course, but this is the area where they're [17:47] highly highly underinvested and I encourage  you guys to think about how you can give the [17:52] non-technical stakeholders the tools they need  to evaluate better. Now I have to earn my expense [17:59] report and talk a little bit about what lang chain  and languid are doing about it. U I'm not going to [18:03] spend a lot of time here because it's it's it's  somewhat explanatory. We recognize that all these [18:09] three rings need support and we have different  product offerings to specifically support that [18:13] and support the information traversal out of ring  three back into ring two and ring one and then [18:17] also help governance and control the development.  So everything is standardized. That's that's all. [18:25] It wouldn't be some kind of tech talk without,  you know, blatantly pandering to some kind of like [18:30] uh you know, tech royalty. Um and in this  case, this is I think this is a really [18:34] great understanding of sort of the challenge that  we're all facing right now. We have to remember [18:38] that these agents are fundamentally abstracting  somebody's subject matter expertise. In a way [18:44] at the very core of what we are doing is we are  taking something that does not scale to h to [18:48] machine levels somebody's expertise understanding  and and knowledge of a task and turn that into [18:55] actually scalable machine scalable processes.  You have to think about an agent at the [19:01] as a reward at the other side of truly  understanding a task. You cannot shortcut this. [19:06] So if you want to also get better at understanding  these tasks again another another pitch here [19:11] um you can get lane chain certified we have a new  um certified agent engineering um uh curriculum. [19:17] We have a ton of free resources um this one is not  free as indicated by the get 50% off um but we do [19:23] have a ton of resources specifically to helping  you guys build and better understand the ADLC [19:29] uh and agents more broadly speaking. So I think I  am probably out of time. I didn't see the timer, [19:34] but oh, I'm way over time. Yep. Sorry, guys.  Um, but thank you for your time and attention. [19:38] Hopefully, this was a relatively energetic  way to end your your AI4 conference. Um, [19:44] I will see you guys at the afterparty. Stay  in touch. I'll be in the back. [applause]