Monte Carlo Deploys Agent Fleets to Fix Production Software Errors

Monte Carlo 部署 Agent 集群修复生产软件错误 自动化强化循环无法优化 AI Agent，除非其建立在涵盖数据管道、系统性能、推理轨迹和输出的统一可观测性之上。 Barr Moses 解释说，只要企业建立起由信任基础设施驱动的 Agent 强化循环，具备自我改进能力的 AI Agent 就已经是生产现实，而非科幻概念。团队不能仅仅依赖基础 LLM 的更新，而必须用专门的 Agent 集群取代人工维护瓶颈，自动检测异常、排查根本原因（例如上游管道故障）并提交修复 PR。她概述了内聚可观测性的四大关键支柱——上下文、性能、行为和输出，并强调跨这些支柱的统一端到端可见性对于构建可靠的自主系统至关重要。 具备自我优化能力的 AI Agent 如今正通过专门的工程基础设施在生产环境中实际运行，它既不是遥不可及的概念，也不是基础模型更新即可直接解锁的即用功能。 (1:33) 将 AI Agent 从试点演示推进到真实生产环境中，往往会暴露出高昂的维护开销、跑题行为以及对持续人工看护的严重依赖。 (0:08) 构建可靠且可自我改进的 Agent 系统需要三大基础要素：蓝图（规划）、劳动力（运维执行）以及坚实的数据信号基础。 (2:01) Agent 强化循环通过连接追踪链路、评估得分和拉取请求（PR）等分散的遥测数据流，将实际生产生态系统本身转变为一个持续的强化学习环境。 (4:10) Agent 强化循环分四步运行：摄取集中式遥测数据、检测全局异常、收集运维 PR 上下文，以及自动分析根本原因以发起修复拉取请求。 (4:50)

Barr Moses, CEO of Monte Carlo, contends that moving AI agents from fragile pilots to dependable enterprise scale requires an automated reinforcement loop embedded directly in production, replacing manual troubleshooting with specialized maintenance agents and unified observability. Operationalizing Production as a Reinforcement Loop (4:10) While AI agents often appear highly capable in isolated demos, Moses observes that deploying them into enterprise production frequently results in high maintenance overhead, off-topic behavior, and persistent engineering babysitting. Rather than treating self-improvement as a passive side effect of foundational model updates, she argues that self-optimization must be engineered into operational workflows through a structured reinforcement loop. In this framework, the live production system serves as the reinforcement environment. The loop centralizes telemetry, execution traces, evaluation scores, and pull requests to detect global anomalies—such as an agent delivering contradictory outputs to users while passing isolated evaluations due to conflicting database sources. Once an anomaly is confirmed against repository telemetry, the system automatically analyzes root causes and generates corrective pull requests, establishing an incrementally improved performance baseline across each cycle. Replacing Engineering Bottlenecks with Specialized Fleets (7:19) Maintaining continuous feedback loops manually may work for a handful of agents, but Moses asserts that human engineering bandwidth becomes an unsustainable bottleneck when managing hundreds or thousands of agents. To address this, Monte Carlo advocates deploying dedicated fleets of specialized AI agents designed specifically to maintain and monitor other agents. Within this architecture, an automated Monitoring Agent sets dynamic performance baselines, resolving the ambiguity teams face regarding what constitutes normal agent behavior. When an incident occurs, a Troubleshooting Agent scans code logic and execution telemetry to isolate the failing lines and author a corrective pull request. Human engineers remain strictly in a governance role, intervening only at the final stage to review and approve the proposed code patch before deployment. Addressing Data-Layer Failures Beyond Agent Code (9:53) Moses illustrates the reinforcement workflow through an enterprise case study involving a deployed "Order Assistant" that incorrectly notified eligible customers that their product return windows had lapsed. Rather than an internal logic error or prompt failure, the troubleshooting agent identified that an upstream dbt pipeline had failed, leaving the assistant to evaluate stale data tables. After cross-referencing repository data to verify that no duplicate fix was underway, the automated maintenance system drafted a pull request to restore the broken data table. Crucially, it also attached a data freshness monitor to preempt identical data-staleness defects in the future. Moses emphasizes that agent malfunctions routinely originate in underlying data pipelines rather than foundational models, making end-to-end data pipeline awareness essential for automated remediation. The Four Pillars of Cohesive Observability (12:28) Isolating telemetry into fragmented silos—such as evaluating traces, eval scores, or code repositories independently—risks causing systems to learn incorrect patterns and generate flawed patches. Moses maintains that enterprise reliability across complex environments with diverse LLMs and dozens of context systems requires observability structured across four pillars: Context, Performance, Behavior, and Output. Context monitors whether underlying enterprise data is fresh and accurate. Performance tracks latency and cost thresholds, noting that multi-minute response times destroy user trust while unchecked runs can generate catastrophic cloud expenses. Behavior audits reasoning trails to catch models that assertively hallucinate citations or skip logical steps. Finally, Output verifies that customer-facing responses comply with business guidelines, preventing inappropriate answers or inadvertent endorsements of competitors. Moses concludes that enterprise trust in autonomous agents relies on combining live production feedback loops, autonomous maintenance fleets, and unified multi-pillar observability. However, achieving this level of automation requires organizations to accept AI-generated code fixes and maintain comprehensive telemetry across disparate enterprise data pipelines.

Monte Carlo 联合创始人兼首席执行官 Barr Moses 指出，当前企业 AI 智能体在生产环境中面临维护成本高昂且频繁偏离预期的现实落差；通过在生产环境构建由智能体机队驱动的强化闭环与统一可观测性，才能真正实现可靠的自我优化与规模化信任。 生产落地脱节与自我优化的基础设施定位 (0:08) Moses 表示，许多企业在概念验证或演示阶段常误以为 AI 智能体无所不能，但将其推向生产环境后，往往需要人工进行持续看护，并面临严重的跑题和高昂的维护成本。这一差距导致许多企业长期停滞在试点阶段，无法实现数百个智能体产品的规模化部署。她强调，自我优化并非等待基础模型厂商更新即可自动开启的开关，而是需要深度嵌入工程工作流的专用基础设施来实现。 以 Monte Carlo 自身的转型为例，该公司早前已转变为代码全部由 AI 生成、全员构建智能体的“AI 优先”模式，并据此提炼出构建自我改进系统不可或缺的三大支柱：定义方向与规则的蓝图（Blueprint）、将蓝图落地的执行劳动力（Labor），以及由数据和运行信号构成的上下文底座（Foundation）。 基于生产遥测的智能体强化闭环 (4:10) 在传统工程实践中，执行追踪、评估打分和 Pull Request（PR）等遥测数据通常彼此割裂，一旦系统出错，工程师必须手动翻查日志并跨团队排查修补。Moses 提出了“智能体强化闭环”（Agent's Reinforcement Loop）概念，借鉴强化学习的逻辑，将企业的实际生产环境直接转化为改进环境，从而打破孤岛化运维。 该闭环包含四个核心步骤：首先集中摄取遥测指标、Trace、Eval 打分及 PR 数据；其次识别全局异常，捕捉在单次测试中看似合格但在全局维度暴露出的系统性缺陷；接着结合代码仓库与 PR 遥测验证该问题是否真实存在或已在修复中；最后自动分析根因并生成补丁 PR。补丁一经部署，便会立即确立新的性能基线，在生产循环中逐步提升系统能力。 以专门化智能体机队破解人工维护瓶颈 (7:19) 当企业管理一到两个智能体时，手动运维尚可应付，但若扩展到成百上千个智能体，软件工程师的人力便会成为扩展的关键瓶颈。Moses 提出利用专门的 AI 智能体机队替代人工排错，主要涵盖负责建立动态健康基线的监控智能体（Monitoring Agent），以及在事件触发后自动扫描代码行与遥测记录以定位根因的故障排查智能体（Troubleshooting Agent）。 以生产案例“Order Assistant”为例，该智能体曾误判具有退货资格的客户已过退货期限。排查智能体介入后发现，模型逻辑与提示词并无缺陷，实际根因是上游 dbt 数据流水线故障导致智能体读取了过期数据表。排查智能体验证仓库后自动生成了修复数据表的 PR，并附带了数据时效性监控器，而人类工程师仅需在最终环节执行代码审查与合并批准，实现了人机协作的闭环。 全链路可观测性的四大核心支柱 (12:28) Moses 告诫企业切勿采取碎片化的监控方式；孤立地审视评估分或执行追踪可能导致智能体学习错误模式，从而提交错误的修复并强化不良行为。特别是在涉及超过 30 种不同上下文系统、多样化框架以及多模型供应商的异构企业环境中，建立跨越四层维度的端到端可观测性至关重要。 这四大支柱具体包括：确保底层数据鲜活准确的“上下文”（Context）；防范用户等待过长或单次运行产生 60,000 美元极端开销的“性能”（Performance）；审查推理路径并防范 LLM 自信造假甚至虚构引用的“行为”（Behavior）；以及最终拦截不当表述、篇幅过长或意外推荐竞品等商业失误的“输出”（Output）。只有统筹四者，才能防止信任断裂。 Moses 总结认为，摆脱 AI 部署焦虑并实现规模化可信交付的路径，取决于三项要素的结合：以真实生产环境为驱动的改进蓝图、全天候运行的自主维护智能体机队，以及覆盖输入、上下文与输出的统一可观测底座。这一体系旨在将人类工程师从无休止的基础排障中解脱出来，使自动化自我修复成为可审计、可验证的确定性流程。

## Background 该视频由 Ai4 于 2026 年 8 月 20 日发布在 YouTube 平台上。内容记录了 Monte Carlo 首席执行官兼联合创始人 Barr Moses 的主题演讲，核心探讨了如何利用企业现有的信任与可观测性基础设施，构建驱动 AI Agent 自主纠错与持续迭代的自优化强化循环。 ## TL;DR 演讲围绕企业如何在真实生产环境中落地具备自我优化能力的 AI Agent 展开。Barr Moses 提出了自优化系统的三大核心支柱：以生产环境为运行底座的“强化循环”演进蓝图、负责自动化监控与故障排查的 7×24 小时“Agent 舰队”，以及贯穿端到端全链路的可观测性基石。通过订单助手的真实溯源案例，她深入剖析了在上下文、性能、行为和输出四大维度建立全局感知的重要性，强调唯有打破数据与代码的孤岛并建立信任底座，才能避免错误模式的自我强化，实现安全可控的自主闭环演进。 ## Atmosphere & style 整个演讲节奏明快、务实干练且直击痛点，Barr Moses 以极具行业共鸣的“演示神话与运维现实”迷因破题，迅速拉近了与工程师及管理者的距离。她的叙事兼具严密的工程逻辑与前瞻性的系统架构视角，从揭示生产环境的割裂困境逐步推演至清晰的闭环解法。听众在过程中能清晰感受到一种从面对海量 Agent 运维失控时的焦虑，转变为依托自动化信任基础设施实现确定性演进的踏实与从容。 ## Key takeaways - [00:08](https://www.youtube.com/watch?v=ZcJDRonHPl4&t=8s) AI Agent 在 Demo 演示阶段看似无所不能，但部署到生产环境后往往暴露出需要时刻人工看护、频繁偏离主题以及维护成本极高三大困境。 - [00:08](https://www.youtube.com/watch?v=ZcJDRonHPl4&t=8s) 实现生产级自优化 Agent 绝非依赖基础模型升级的“魔法开关”，而是必须构建深度融入开发流程的底层工程基础设施。 - [02:36](https://www.youtube.com/watch?v=ZcJDRonHPl4&t=156s) 构建自演进基础设施依赖三大支柱：指引演进方向的蓝图（强化循环）、替代重复劳动的执行单元（Agent 舰队）以及数据基石（端到端全栈可观测性）。 - [04:10](https://www.youtube.com/watch?v=ZcJDRonHPl4&t=250s) Monte Carlo 颠覆传统模拟环境限制，直接将真实生产环境作为强化学习的运行底座，构建包含数据摄取、全局检测、上下文验证与自动提交修复 PR 的强化闭环。 - [07:19](https://www.youtube.com/watch?v=ZcJDRonHPl4&t=439s) 面对规模化 Agent 集群，企业应部署监控 Agent 与排障 Agent 等专用舰队，将人工介入后移至最终 PR 审核环节（Human-in-the-loop），打破人力排查瓶颈。 - [09:53](https://www.youtube.com/watch?v=ZcJDRonHPl4&t=593s) 生产环境中的 Agent 故障往往不仅源于模型或 Prompt 自身，上游数据转换流水线故障或底层数据滞后（Stale Data）同样会导致严重的业务决策失误。 - [12:28](https://www.youtube.com/watch?v=ZcJDRonHPl4&t=748s) 缺乏一体化可观测性会导致系统在割裂的错误信号中持续放大错误逻辑（Compounding bad behavior），使自优化演进走向歧途。 - [12:28](https://www.youtube.com/watch?v=ZcJDRonHPl4&t=748s) 建立企业级 Agent 信任体系必须覆盖四大核心维度：数据新鲜度与准确性（Context）、响应延迟与单次成本（Performance）、决策推理路径（Behavior）以及业务规范与安全性（Output）。 - [17:07](https://www.youtube.com/watch?v=ZcJDRonHPl4&t=1027s) 现代企业普遍混合使用多模型、多框架及数十种上下文存储系统，必须依靠自动化手段跨越异构环境抓取全链路信号以筑牢信任底座。 - [18:56](https://www.youtube.com/watch?v=ZcJDRonHPl4&t=1136s) 基于真实生产数据的演进蓝图、7×24 小时不间断运行的 Agent 舰队与全局可观测性底座构成的闭环架构，是让团队能够充满信心地在生产中发布 AI Agent 的核心保障。 ## Detailed walkthrough ### [00:08](https://www.youtube.com/watch?v=ZcJDRonHPl4&t=8s) 走出“演示神话”：构建生产级自优化 Agent Monte Carlo 的首席执行官兼联合创始人 Barr Moses 在演讲开场时强调，公司的核心使命是赋能全球企业采用值得信赖的 AI（trusted AI），其客户包括美国航空（American Airlines）、思科（Cisco）等知名企业。在谈及当下的行业热点时，Moses 直言，让 AI Agent 实现“自我优化”（self-optimize）在许多人看来宛如科幻小说，因为大部分团队目前仍处于探索的早期阶段。她生动地引用了一个广为流传的行业迷因：在 Demo 演示或试点（pilot）阶段，Agent 看起来无所不能，宛如通晓一切的神器；可一旦真正部署到生产环境，团队就会陷入现实的泥潭——"Oh, wait, hold on. First of all, I need to babysit this thing. Second, it goes off topic all the time. Third, it's a lot of work to maintain these."（“等等，首先我得像保姆一样时刻盯着它；其次，它总是偏离主题；第三，维护这些东西简直是一场噩梦。”） 针对这一痛点，Moses 邀请观众共同设想一个不再停留在试点阶段、而是有大量 Agent 和 AI 产品在生产环境中规模化稳定运行的世界。在这个世界里，人们总梦想着某种“魔法棒”，只要睡一觉醒来，随着底层基础模型能力的提升，Agent 就会自动升级变强。但 Moses 指出，现实中并不存在这种一按就能生效的开关，真正的自我改进并非魔法，而是必须深度融入开发流程的底层工程基础设施。Monte Carlo 已经在生产环境中实现了真正具备自我改进能力的 Agent，这不仅不是科幻，而且是已经落地的现实。 ```mermaid flowchart LR accTitle: Demo vs Production Agent Gap subgraph Demo["试点与演示阶段 (Demo / Pilot)"] D1["表现完美"] --> D2["无所不能的假象"] end subgraph Prod["生产落地困境 (Production Reality)"] P1["需要人工时刻看护 (Babysit)"] P2["频繁偏离主题 (Off-topic)"] P3["高昂的维护成本 (Maintenance)"] end Demo -.->|"部署到真实环境"| Prod ``` --- ### [02:36](https://www.youtube.com/watch?v=ZcJDRonHPl4&t=156s) 自优化基础设施的三大支柱与 Monte Carlo 的 AI 实践 要让 Agent 在生产中实现自我改进，必须构建起完整的基础设施。Moses 将这套基础设施解构为三大核心支柱： 1. **蓝图（Blueprint / Map）**：明确系统的行进方向、运行规则与演进逻辑； 2. **劳动力（Labor）**：负责执行具体工作、将蓝图变为现实的执行单元； 3. **基石（Foundation）**：为整个系统提供驱动力的数据底座与信号来源。 基于这一体系，Monte Carlo 借鉴强化学习（Reinforcement Learning）的机制提出了“强化循环”（Reinforcement Loop）作为演进蓝图；在劳动力层面，Moses 提出不应依赖繁琐的人工，而应组建一支由 Agent 构成的“Agent 舰队”（Fleet of Agents）协同工作；而在底层基石层面，则依托 Monte Carlo 深耕多年的端到端全栈可观测性（End-to-End Observability）。Moses 分享道，Monte Carlo 自身在年初已经转型为 100% AI-First 的企业——公司所有的代码均由 AI 生成，全员都在构建和交付 Agent，正是这种深度的内部实践，让他们摸清了实现生产级自演进系统的关键路径。 --- ### [04:10](https://www.youtube.com/watch?v=ZcJDRonHPl4&t=250s) 强化循环（Reinforcement Loop）：以生产环境为演进底座 Moses 指出，当前生产环境中的常见痛点是：遥测数据（telemetry）、调用链路（traces）、评估测试（evals）等各种信号处于彼此割裂的孤岛中，缺乏统一脉络。即使工程师发现指标异常，也极难定位问题根源并确定修复方案，往往需要几组工程师分别排查 Trace、评估 Eval 得分、分析故障并构思补丁，流程极其耗时繁琐。为此，Monte Carlo 提出了“强化循环”（Reinforcement Loop）模型。 与传统强化学习需要单独搭建模拟环境不同，该机制的颠覆性在于直接将“真实生产环境”作为强化学习的运行环境。该循环包含四个关键环节： ```mermaid flowchart TD accTitle: The Agent Reinforcement Loop subgraph Loop["Agent 强化循环 (Reinforcement Loop)"] A["1. 数据摄取 (Ingestion)<br/>统一整合 Traces、Evals、PRs 遥测"] --> B["2. 异常检测 (Detection)<br/>全局多维比对，识别隐性偏离"] B --> C["3. 上下文验证 (Operational Context)<br/>结合 PR 与变更确认未修复真问题"] C --> D["4. 故障分析与修复 (Troubleshoot & Issue PR)<br/>生成修复 PR 并建立全新基准线"] D -->|"持续迭代部署"| A end ``` - **数据摄取（Ingestion）**：在系统顶层将分散的 Traces、Eval 得分、代码变更（PRs）与操作遥测统一汇聚到一处。 - **全局检测（Detection）**：从宏观视角捕捉单点看似正常、综合来看却存在缺陷的异常。例如，一个 Agent 面对两个不同用户给出了相互矛盾的答案，从单一 Eval 来看因分别命中了不同数据表而判定合格，但在全局关联分析下，这种不一致性便会立刻暴露。 - **上下文验证（Operational Context）**：结合历史 PR 与系统上下文，确认该缺陷是当前真实存在的活跃问题，而非已被其他变更修复的偶发陈旧现象。 - **分析、排障与修复提案（Troubleshoot & Issue PR）**：自动定位代码或数据问题并提交 PR 修复。一旦合并生效，次日早晨该 Agent 就拥有了全新的性能基准线（Baseline），并在循环往复中实现自主迭代演进。 --- ### [07:19](https://www.youtube.com/watch?v=ZcJDRonHPl4&t=439s) 劳动力重构：以 Agent 舰队打破人工排查瓶颈 当企业拥有 100 个甚至 10 万个 Agent 时，纯靠人工维护强化循环是完全无法扩展的。Moses 强调，必须使用 Agent 组成的专业化舰队来替代重复性脑力劳动，从而在不增加团队人头编制（Headcount）的前提下实现产能的大幅扩展。 Monte Carlo 在实践中部署了多种专用 Agent，其中包括： - **监控 Agent（Monitoring Agent）**：企业面临的头号难题往往是“不知道该监控 Agent 的什么指标”，也不明确“良好的状态”具体长什么样。监控 Agent 能够扫描所有遥测指标与 Eval 数据，自动建立合理的监控基线。 - **排障 Agent（Troubleshooting Agent）**：在检测到故障告警后，自主检索数千行代码与系统遥测，精准定位根因。 在整体流程设计上，Moses 主张将传统以工程师为中心的模式（人工排查、人工走查代码、人工写 PR）转变为 Agent 主导的流水线。前四个阶段（识别故障、配置监控、故障排查、问题分流）完全交由 Agent 自动化处理，而人类只需作为最后一道防线介入——在最终合并修复 PR 前进行代码审查与确认（Human in the loop），在确保安全可控的同时极大加快系统迭代效率。 --- ### [09:53](https://www.youtube.com/watch?v=ZcJDRonHPl4&t=593s) 实战案例：Order Assistant 故障溯源与自主修复 为了证明自优化系统已在数十家客户的生产环境中真实落地，Moses 现场拆解了一个订单处理助手（Order Assistant）的真实案例。系统中出现了一个业务故障：符合退货资格的真实用户在有效申请窗口期内发起了退货请求，却被 Agent 错误地拒绝并告知“已超过退货期限”。 ```mermaid flowchart LR accTitle: Order Assistant Incident Resolution Flow subgraph Diagnosis["自主排查与验证"] F["识别故障：合法退货被拒"] --> T["排障 Agent 介入"] T --> L["确认 Agent 业务逻辑无误"] T --> U["定位上游 dbt 作业失败导致数据陈旧"] U --> V["关联比对 PR，确认故障未被修复"] end subgraph Fix["自动修复与人工确认"] V --> PR["Agent 自动提交 PR：恢复表结构 + 增加数据新鲜度监控"] PR --> H["工程师一键 Approve 合并上线"] end ``` 排障 Agent 介入后迅速展开了全链路溯源： - 首先排查了 Agent 自身的代码与 Prompt 逻辑，确认 Agent 本身没有任何设计缺陷； - 随后沿着链路向上追溯，发现真正根因出在上游的 dbt 数据转换作业失败，导致 Agent 读取到的底层数据库存在严重的数据滞后（Stale Data）； - Agent 进一步扫描了代码库中所有相关的 Pull Requests，确认当前没有正在处理该问题的在途补丁，验证了该故障的有效性； - 最终，排障 Agent 自动生成了完整的解决方案：提交 PR 修复数据流水线、恢复损坏的数据表，并主动添加了“数据新鲜度监控”（Freshness Monitor）以防未来再次发生同类故障。人类工程师只需在界面上审查变更并点击批准（Approve），系统即完成了自主闭环修复。 Moses 借此总结道，生产环境中的 Agent 故障根因非常多元，可能源于模型自身、上游数据流水线、亦或是底层基础设施，仅盯住模型单点永远无法解决全链路系统问题。 --- ### [12:28](https://www.youtube.com/watch?v=ZcJDRonHPl4&t=748s) 信任基石：端到端可观测性与四大核心维度 Moses 强调，若缺乏端到端一体化可观测性（Cohesive Observability）这一基石，上述所有自动化都无从谈起。在割裂的视角下观察系统极易得出错误推论：单纯看 Telemetry 无法看清全局，单纯看 Eval 会漏掉意料之外的边缘问题，而忽视 PR 和代码变更则会丢失关键上下文。最危险的后果并非单纯漏掉 Bug，而是让系统在孤立的错误信号中强化了错误的修复模式（compounding bad behavior），使 Agent 演进走向歧途。 为了建立真正的 Agent 信任体系，企业必须在以下四个核心维度上建立一体化可观测能力： | 核心维度 | 关键考量指标与常见风险 | 生产场景警示与痛点 | |:--- |:--- |:--- | | **上下文 (Context)** | 数据新鲜度 (Freshness)、准确性、准时性、AI 数据就绪度 (AI-ready) | 依赖陈旧过时的数据直接导致下游 Agent 产生逻辑荒谬的业务决策。 | | **性能 (Performance)** | 响应延迟 (Latency)、调用单价与总体成本 (Cost) | 响应若耗时长达 4 分钟，用户早已流失；单次运行若耗资高达 6 万美元（$60k），系统将无法在生产中规模化运转。 | | **行为 (Behavior)** | 决策推理路径 (Reasoning Traces)、决策链连贯性 | 传统报表顶多是单纯的数据错误，但 Agent 却擅长“极其自信地胡说八道”（wrong with confidence），甚至带上完全错误的引用源与用户争辩。 | | **输出 (Output)** | 响应相关性、长度适配度、业务安全边界 | 前三步即便全对，最终输出仍可能严重违背商业利益，例如在电商零售场景中，Agent 竟然公然在回复中向客户推荐竞品。 | --- ### [17:07](https://www.youtube.com/watch?v=ZcJDRonHPl4&t=1027s) 异构企业环境下的全链路可观测性落地 在演讲这一部分的尾声，Moses 展示了企业实际技术栈的复杂版图。现代企业为了避免“将所有鸡蛋放在同一个篮子里”（not wanting to put all of their eggs in one basket），普遍构建在高度异构（Heterogeneous）的基础设施之上： - 混合使用来自不同供应商的底层模型以控制成本； - 采用多种 Agent 框架以保持业务敏捷度与灵活性； - 往往接入超过 30 种不同的上下文与数据存储系统（Context Systems）； - 底层深度集成 GitLab、GitHub 等多种代码管理与协作平台。 Moses 指出，尽管底层技术栈错综复杂且分布广泛，但关于 Agent 信任的四大法则始终不变：只要上游数据陈旧，Agent 就会给出错误的推荐；只要跳过了预设的推理步骤，执行就会产生偏离；只要输出违背了业务规范，就会破坏用户信任。因此，企业必须借助自动化的技术手段跨越这些异构平台抓取全链路信号，建立统一的信任基石，从而为 Agent 的自主进化提供安全、稳固的运行保障。 ### [18:56](https://www.youtube.com/watch?v=ZcJDRonHPl4&t=1136s) 端到端可见性与构建可信 Agent 系统的三大支柱 Barr Moses 强调，围绕 AI Agent 建立端到端（end-to-end）的可见性以及跨所有集成的全方位信号感知，正是让整个强化与自优化循环（Reinforcement Loop）发挥出最大威力的关键所在。她邀请现场观众畅想一个理想的未来图景：企业与团队能够在生产环境中充满信心地发布 AI Agent。在这样的世界里，开发者和管理者每天清晨醒来时，能够确信自己交付给用户和同事的产品是绝对可靠且值得信赖的。 要真正实现这一愿景并驾驭自优化机制，底层依赖于三大核心要素构成的系统框架： ```mermaid flowchart TD accTitle: 可信自优化 Agent 系统的三大支柱 accDescr: 展示生产环境蓝图、24/7 Agent 舰队与全局可观测性底座构成的闭环架构 subgraph Pillars ["可信 Agent 系统的三大核心要素"] direction TB A["1. 生产环境蓝图 (Blueprint)<br/>基于真实生产数据建立并驱动强化循环"] B["2. 全天候 Agent 舰队 (Agentic Fleet)<br/>7×24 小时自主运行且无需休整的智能体集群"] C["3. 统一可观测性底座 (Observability Foundation)<br/>汇聚端到端全局信号，提供底层信任支撑与置信度"] end C --> A A --> B B -->|"运行反馈与状态信号"| C ``` - **生产环境蓝图（Blueprint）：** 充分利用真实生产环境的数据与上下文来构建闭环的强化反馈循环，并能依据系统反馈即时采取优化行动。 - **全天候运行的 Agent 舰队（Agentic Fleet）：** 打造能够实现 7×24 小时不间断自主工作、无需休息的企业级智能体集群。 - **一体化可观测性底座（Cohesive Observability Foundation）：** 为整个自优化流程提供驱动力的底座支撑，通过全面捕获与分析所有可能的运行信号，赋予团队充分的掌控力与置信度。 在演讲尾声，Barr Moses 向与会者表达了感谢，并表示非常乐意与有兴趣深入探索的企业同行交流，展示这些机制在实际生产工作环境中的运行样例，最后真诚祝愿大家在各自的 AI 探索之旅中取得成功。 ## Notable quotes > [00:08](https://www.youtube.com/watch?v=ZcJDRonHPl4&t=8s) Oh, wait, hold on. First of all, I need to babysit this thing. Second, it goes off topic all the time. Third, it's a lot of work to maintain these.（等等，首先我得像保姆一样时刻盯着它；其次，它总是偏离主题；第三，维护这些东西简直是一场噩梦。） — Barr Moses > The idea that AI agents could improve on their own — detect their own failures, fix them, and get better without a human operator — sounds like science fiction. It isn't. — Barr Moses > Get the foundation wrong, and the loop just optimizes your agent around a broken system. — Barr Moses > The framework that tells you whether an agent is trustworthy in production today — a single view across context, performance, behavior, and outputs — turns out to be exactly what a self-improving system needs to run on. — Barr Moses > [00:08](https://www.youtube.com/watch?v=ZcJDRonHPl4&t=8s) 现实中并不存在这种一按就能生效的开关，真正的自我改进并非魔法，而是必须深度融入开发流程的底层工程基础设施。 — Barr Moses > [09:53](https://www.youtube.com/watch?v=ZcJDRonHPl4&t=593s) 生产环境中的 Agent 故障根因非常多元，可能源于模型自身、上游数据流水线、亦或是底层基础设施，仅盯住模型单点永远无法解决全链路系统问题。 — Barr Moses > [12:28](https://www.youtube.com/watch?v=ZcJDRonHPl4&t=748s) wrong with confidence（极其自信地胡说八道，甚至带上完全错误的引用源与用户争辩） — Barr Moses > [12:28](https://www.youtube.com/watch?v=ZcJDRonHPl4&t=748s) compounding bad behavior（让系统在孤立的错误信号中强化了错误的修复模式，使 Agent 演进走向歧途） — Barr Moses > [17:07](https://www.youtube.com/watch?v=ZcJDRonHPl4&t=1027s) not wanting to put all of their eggs in one basket（避免将所有鸡蛋放在同一个篮子里） — Barr Moses > [17:07](https://www.youtube.com/watch?v=ZcJDRonHPl4&t=1027s) 只要上游数据陈旧，Agent 就会给出错误的推荐；只要跳过了预设的推理步骤，执行就会产生偏离；只要输出违背了业务规范，就会破坏用户信任。 — Barr Moses ## Mentioned Monte Carlo, Monte Carlo AI, Barr Moses, Ai4, American Airlines（美国航空）, Cisco（思科）, dbt, GitLab, GitHub, Order Assistant ## Video description Barr Moses - CEO & Co-founder, Monte Carlo AI Self-Optimization: The reinforcement loop and the future of agent trust The idea that AI agents could improve on their own — detect their own failures, fix them, and get better without a human operator — sounds like science fiction. It isn't. We're closer to that self-optimizing future than most people think, and what it takes to get there might surprise you because it's something we've already built. Trust infrastructure. The framework that tells you whether an agent is trustworthy in production today — a single view across context, performance, behavior, and outputs — turns out to be exactly what a self-improving system needs to run on. In this session, I'll show you how the two connect: how the reinforcement loop that drives self-optimization depends on that trust infrastructure underneath it to stay reliable, cycle after cycle. Get the foundation wrong, and the loop just optimizes your agent around a broken system. Drawing on patterns from billion-dollar enterprises running real agents in production, you'll leave understanding why the path to autonomous, self-improving agents runs straight through the trust foundation you're building right now. [Source](https://www.youtube.com/watch?v=ZcJDRonHPl4)

[0:08] Barr Moses: Oh, wait, hold on. First of all, I need to babysit this thing. Second, it goes off topic all the time. Third, it's a lot of work to maintain these.（等等，首先我得像保姆一样时刻盯着它；其次，它总是偏离主题；第三，维护这些东西简直是一场噩梦。） Barr Moses: The idea that AI agents could improve on their own — detect their own failures, fix them, and get better without a human operator — sounds like science fiction. It isn't. Barr Moses: Get the foundation wrong, and the loop just optimizes your agent around a broken system. Barr Moses: The framework that tells you whether an agent is trustworthy in production today — a single view across context, performance, behavior, and outputs — turns out to be exactly what a self-improving system needs to run on. [0:08] Barr Moses: 现实中并不存在这种一按就能生效的开关，真正的自我改进并非魔法，而是必须深度融入开发流程的底层工程基础设施。 [9:53] Barr Moses: 生产环境中的 Agent 故障根因非常多元，可能源于模型自身、上游数据流水线、亦或是底层基础设施，仅盯住模型单点永远无法解决全链路系统问题。 [12:28] Barr Moses: wrong with confidence（极其自信地胡说八道，甚至带上完全错误的引用源与用户争辩） [12:28] Barr Moses: compounding bad behavior（让系统在孤立的错误信号中强化了错误的修复模式，使 Agent 演进走向歧途） [17:07] Barr Moses: not wanting to put all of their eggs in one basket（避免将所有鸡蛋放在同一个篮子里） [17:07] Barr Moses: 只要上游数据陈旧，Agent 就会给出错误的推荐；只要跳过了预设的推理步骤，执行就会产生偏离；只要输出违背了业务规范，就会破坏用户信任。

[0:08] Barr Moses: help global enterprises adopt trusted AI systems [1:33] Barr Moses: 100% AI-first [4:10] Barr Moses: agent's reinforcement loop [7:19] Barr Moses: aren't actually sure what to monitor in their agent [9:53] Barr Moses: Order Assistant [12:28] Barr Moses: learn the wrong patterns, ultimately submitting the wrong fixes and reinforcing bad behavior [12:28] Barr Moses: If an agent takes four minutes to reply to a user... the agent might as well just not give an answer at all [12:28] Barr Moses: agents are wrong with confidence

[0:08] My name is Bar Moses. I'm the CEO and co-founder  of Monte Carlo. Monte Carlo's mission is to enable [0:15] the world's enterprises to adopt trusted AI. We're  very fortunate to work with amazing companies, [0:22] many of which I've met here today like American  Airlines, Cisco in it, and many others. And you [0:29] know, I I was sort of looking at the topic today  and kind of speaking with folks and I think this [0:34] idea of sort of what does it mean to have agents  that self optimize feels a little bit like science [0:40] fiction right now. A lot of folks are sort of  earlier in the journey. Perhaps you resonate with [0:46] this meme. I I don't know if anyone here, show of  hands, feels like their agent in demo or agent in [0:52] pilot can do everything under the sun, but then  you shift it to production and you're like, "Oh, [0:57] wait, hold on. First of all, I need to babysit  this thing. Second, it goes off topic all the [1:03] time. Third, it's a lot of work to maintain  these." And so what I'd like to do for this [1:09] talk today is to invite you to imagine a world  where we actually have lots of agents and lot [1:17] of AI products in production. We're not just in  pilot mode. We're scaling those up and we can [1:25] trust that AI and those agents. Let's talk about  what that world might look like. In that world, I [1:33] think there's sort of a dream that these self that  these agents will self-improve that I wake up in [1:38] the morning and suddenly there's some magic wand  that I can wave that makes agents better. Well, [1:45] I'm here to tell you that first of all, folks  think that it's a switch that you turn on that [1:51] one day you'll have hundreds of thousands of agent  in production, the model, a better model will land [1:56] and boom, you'll get a better agent. Well, I'm  here to share that we actually have self-improving [2:02] agents today in production. It's very exciting.  It's not science fiction. It's here and real. And [2:09] it's not a switch. It's actually infrastructure  that you have to build into the process. And I I'd [2:15] like to share with you all what was the journey  that we took to build this infrastructure to [2:20] actually have self-improving agents in production  because I've built the conviction from that that [2:26] that's not science fiction. Many of us are earlier  in the journey and still figuring out what does [2:31] that look like. But actually that reality is  here and now. So what does that infrastructure [2:36] actually include? There's three different  components to it. The first is a blueprint, [2:41] a map if you will. Where are we going with this?  What are the rules of the road? What do we need [2:46] to do? The second is the labor. Someone has to  execute the work to make it happen. Someone needs [2:53] to take the blueprint and make it real. Who does  that? The third is you need to have a foundation. [2:59] It has to drive off of some sort of data or some  some sort of signals that help us drive this. So [3:06] at Monte Carlo, we actually sort of coined this  term, a reinforcement loop. It drives inspiration [3:12] from the concept of reinforcement learning. And  I'll explain what the reinforcement loop on the [3:17] labor front. I'm going to make the argument  that you could do this with a fleet of agents. [3:23] Today a lot of this is done very manually but  we are actually seeing seeing that it's possible [3:27] to do this with agents. And then finally the  foundation which is where myself and Monte Carlo [3:33] have spent the last several years is cohesive  endto-end observability for your agentic stack [3:40] that includes your agents but not only. We'll  talk about what that means. And so what we found [3:46] in sort of our journey and Monte Carlo started a  few years ago and earlier this year Monte Carlo [3:51] became 100% AI first and that means that all of  our code is 100% AI generated and every single [3:59] person at the company is building and shipping  agents and throughout that work we've been able [4:03] to see what does it take to actually go through  this journey. So let's double click into these. [4:10] The first is the the blueprint the reinforcement  loop. The reality is that when you have agents [4:15] in productions, you have different signals.  You might have traces, you might have evalued [4:26] information about what's happening with your  agent, but nothing actually connects them. So [4:31] even if you're able to have it in disparate  places, there's no cohesive line. And worse, [4:38] even if you know that something is wrong in  one of them, it's nearly impossible to make the [4:44] connection between what went wrong to what should  I fix. That process is very manual. You have [4:51] engineers that are sifting through traces, sifting  through eval scores, trying to understand what [4:56] went wrong, and then a different set of engineers  are trying to figure out, rack their brain around [5:01] what exactly broke. And then a different set of  engineers trying to figure out what is the fix. [5:08] What we have defined is sort of an  agent's reinforcement loop. You see [5:12] the reinforcement loop. This is our proposal for  what a reinforcement loop should look like. Now, [5:17] many of you probably are familiar with the concept  of reinforcement learning. The idea here with the [5:23] reinforcement loop is that you're able to actually  get a reinforcement learning shaped improvement [5:28] process for your agents but without creating the  environment. meaning your production becomes the [5:36] environment. That's a very important distinction  and has never been done before. So what are the [5:42] different process or what's the blueprint of this  reinforcement loop? The first is to actually start [5:46] with the ingestion on the right hand side at  the top. Take all of that telemetry, the traces, [5:51] the eval, the pull requests that you have and  bring them together in one place. Then you can [5:57] do the second step which is detection. actually  understand what might look normal at first, [6:03] but when you take a zoom out and look at all of  those together suddenly looks wrong. Meaning, [6:09] I'll give you a practical example. An agent can  give two different answers to two different users. [6:14] It passes the eval test because it's actually  anchored on the same data but in two different [6:20] tables. However, when you zoom out and look  at eval traces and and your pull requests, [6:25] you realize that it doesn't make sense to give  two different responses to a user that something [6:30] is off with the agent. The next step here is how  do we take the operational context to make sure [6:37] that this is a real issue that hasn't already been  fixed. That's where you can take you can use your [6:42] your pull requests and all the telemetry around  that to to confirm that. And then finally you [6:49] do all of this work to analyze, troubleshoot and  issue a pull request to actually fix the issue. [6:56] Now that means that tomorrow morning the agent  had took that fix and has implemented that and [7:02] now this fix becomes the new baseline for agents.  And in that way by iterating on this loop again [7:08] and again we're actually able to introduce this  new baseline that's self-improving. And all of [7:14] this is using your production as an envir  environment for the reinforcement loop. So [7:19] a very powerful concept. However, how does this  work when you have 100 agents or 100,000? Maybe [7:27] you can do this manually for one, maybe two.  What does this actually look like? And this is [7:32] where we start talking about the labor. And my  argument is that it's very hard to do this with [7:38] um with engineers and you actually need to build  a fleet of agents. So at Monte Carlo, we've built [7:43] this fleet of agents to help execute on that. I'll  give you just a couple examples. The first is a [7:49] monitoring agent. The monitoring agent actually  looks at all of the telemetry, the eval traces, [7:55] and actually identifies what are the things  that you need to monitor. The surprising thing [8:00] is that the number one problem the folks have  is they're not actually sure what to monitor in [8:05] their agent. They're not actually sure what good  looks like. And so using AI or using an agent, [8:10] we can have a starting place to at least identify  that. Another example here is a troubleshooting [8:15] agent. Once there's an incident detected, we can  use an agent to sift through all of the code, [8:22] the lines of codes, the telemetry, and actually  identify and pin pin pinpoint their root cause. [8:28] And so using agents, we're actually uh able to  do to increase our labor significantly without [8:35] actually increasing headcount, but rather using  these agents to do the work on our behalf. And [8:41] this has been, you know, an important point in  being able to scale the reinforcement loop because [8:47] a reinforcement loop doesn't work if you don't  have each of those components agentically built. [8:55] This is just an example of what it looks like when  you have humans as the as a sort of a bottleneck [9:01] if you will uh and how you're able to fasttrack  that with agents. So on the left hand side you'll [9:06] see a lot of the things that are done are done  in a very manual way where you have engineers [9:11] identifying the issue, engineers walking  through the code and eventually actually [9:16] writing the pull request. And so you could design  this entire process with human in the center. [9:22] I would argue that we want to be on the right  hand side where humans are intervening at the [9:27] last stage meaning the first four steps can  be identified identifying the issue setting up [9:34] the monitoring troubleshooting triaging and then  ultimately before we ship the pull request before [9:40] we ship the fix we have a human in the loop I  still think that that's a very important component [9:46] um uh given given where we're at in with our  AI journey and So, I'm going to walk through [9:53] um this is a little bit hard to see from afar, but  I wanted to show what this example looks like. Um [10:00] because often the number one question that I get  is, is this actually possible? Is this real? And [10:05] the answer is yes, it is real. It's in production  uh with dozens of customers at this point. And I [10:10] want to show it to you with a real example here.  So, what we have here is an agent that's called [10:15] order assistant where it's basically processing  orders as we go. And you can see that there's an [10:21] identified failure in which customers are unable  to process uh returns that are eligible. And so [10:29] there's actually sort of something that uh a user  wants a return. It's within the valid window and [10:35] they got told that they were too late by the  agent. Now in this um view what you can see is [10:42] that the troubleshooting agent kicked into gear  and started looking through what is the issue. [10:48] The first thing that the troubleshooting agent  identified is that the logic was totally fine. [10:53] The agent itself was absolutely uh fine in this  instance. However, due to an upstream DBT failure, [11:01] there was an error where the the data that the  agent was using was stale. So, first of all, [11:07] what's our learning here? Incidents can happen as  a result of the agent the model or incidents can [11:14] happen as a result of the upstream data or it can  result as the underlying infrastructure. So the [11:20] root cause for what breaks is actually varied  and not only concentrated in the agent itself. [11:28] The next place here you can see that the agent is  verifying that there's actually an issue and the [11:33] way that it's verifying it is by running through  all the pull requests that are somewhat related to [11:38] this. it's finding that there's one issue that's  related to it, but it's not actually the fix. So, [11:44] you're the agent is verifying that this is indeed  a problem. And then finally, this is the fix. The [11:50] agent is proposing um to restore the table to  ship a pull request that fixes it and to add a [11:59] freshness monitor to make sure that the underlying  data is always accurate and correct. the human [12:06] is left in this um uh process to hit approve  to review the PR as needed and to ultimately [12:13] uh ship this fix. So this is a real example from  a production environment um where you can see this [12:20] sort of semi-autonomous or autonomous workflow for  your agents and your data. Now the none of this [12:28] would be possible without the foundation of what  I would call cohesive or endto-end observability. [12:34] What we're finding is this, you know, this is  sort of the the last stop on the train, if you [12:39] will. What we're finding is that it's actually  very important to be thinking a systems view to [12:44] thinking through your AI estate. And why is that?  Because if you look at each incident on its own, [12:51] actually, you're going to be making the wrong  conclusions. What do I mean by that? If you're [12:56] only looking at telemetry, you might be  finding things that are that are off, [13:01] but don't tell the full story. Excuse  me. Um, if you're only looking at eval, [13:06] you might not be catching an issue that you  were unaware of [clears throat] and you're if [13:11] you're not looking at your code or pull request,  you're missing important information. And so the [13:16] actually the really worst case scenario here is  not necessarily that you're missing an issue. [13:21] The worst case scenario that we've seen here is  that your agent is learning the wrong patterns, [13:28] ultimately submitting the wrong fixes and  reinforcing bad behavior. So you actually have bad [13:33] behavior that compounds itself on the agent and it  sort of loses the point here. [clears throat] The [13:42] way the the trick or sort of the way that we've  seen companies avoid this is by not look at those [13:47] issues in silo. So while the the instinct might be  oh we just have evals that's sufficient or oh we [13:54] just have traces that should be good enough  that's actually where the core mistake is. [13:59] We have found that you have to look at all four  of these core areas to truly trust an agent. The [14:05] first one is the context and my bet is that most  of you in the audience are thinking through that [14:10] area in particular and making sure that your  data is actually AI ready is the most important [14:17] uh sort of part that we've spent the last couple  of decades on if you will. So my bet is the large [14:22] majority of folks are still in this stage. And in  that, you know, in on the on the context piece, [14:27] we want to make sure there's sort of no surprises  here. There's sort of the bread and butter, [14:31] making sure that the data is fresh,  it's on time, that the values are right, [14:36] etc. But actually doing that across all of  the data that your agents have access to [14:41] and doing that at scale becomes a lot harder if  you're not taking an agentic approach to that. [14:47] The second important part around agents is looking  at their performance and again correlating that [14:52] with those other parts. What do I mean by that? If  an agent takes four minutes to reply to a user, I [14:58] don't know about you, but I don't hang around four  minutes waiting for for an answer. The agent might [15:02] as well just not give an answer at all in that  instance. That makes your user lose trust. If your [15:09] agent can't give the right answer at the right  time, you will lose your end users. Similarly, [15:16] if one agent run costs 60k to run it, that's  probably not something that you want to be doing [15:22] at scale or at least you want to be aware of that.  So looking at latency costs and other indications [15:27] of performance becomes really important. The third  aspect is around behavior. You know, I always sort [15:33] of joke for folks who've been here um uh you know,  in this space for a while or come from data, I [15:39] always find that you know, dashboards were always  wrong. Like they were just this just happened. You [15:43] like data was wrong all the time. But agents are  wrong with confidence. I don't know if it happens [15:49] to you, but I find myself arguing with Claude  or Gemini like, "No, you're wrong. I don't think [15:54] that's true." But they will argue back with you  with confidence with totally, you know, absolutely [15:59] wrong answers. And the funny thing is that often  times they will add sources and it kind of makes [16:05] you feel better about yourself because you know  the source, but it could be completely wrong. [16:09] And so the problem is that we want to make sure  that these agents are making the right decisions [16:14] because separate from dashboards and data agents  actually have a reasoning and make decisions and [16:20] so we need to make sure that we understand the  behavior of an agent and can trace the series [16:24] of decisions that they make and then finally  we want to look at the output. So one all you [16:31] know all these sort of steps context performance  and behavior can be perfect but still the answer [16:36] that an agent can give might be inappropriate for  your particular question might leave the end user [16:41] frustrated with a too long or too short you know  work with some companies where the agent actually [16:46] recommends their competitor uh and the response  is an example I don't know about you I don't want [16:51] to be recommended you know as a retail company  you probably don't want your competitor to be [16:55] recommended and so actually also looking at the  outputs and connecting that to all to the first [17:00] three is critical and you know I will say that in  order to sort of set set sort of all all four of [17:07] these again um it's actually quite complicated  to do that for an enterprise environment that's [17:13] typically heterogeneous so when I speak with folks  they'll typically tell me you know I have multiple [17:18] agent platforms multiple context platforms etc and  the truth is what I'm seeing is that environments [17:26] are very heterogeneous and so we're actually  building in different places. However, across [17:33] your environment, these four things stay true. The  agent will recom will make a bad recommendation if [17:40] it's using data that's stale. The agent will make  a bad recommendation if it's skipping the first [17:45] five steps of what it was supposed to do. And  the agent will make a bad recommendation if it's [17:51] um giving an answer that's four pages long and  inappropriate for your end user. And so these [17:56] sort of are the core sort of foundation to making  this entire thing run um all at once. I will also [18:03] say that over time we've developed new ways to  collect this information. So we can use different [18:09] technologies to automate this. This is not sort of  human manual input. On this next slide you can see [18:16] a variety of logos. This is what a most common  environment looks like for an enterprise. very [18:23] heterogeneous variety of models. I am seeing a  lot of enterprises not wanting to put all of their [18:30] eggs in one basket and actually want to be using  different models for cost reasons, different agent [18:36] frameworks, for flexibility reasons and a lot  of different context systems. So, it's typically [18:43] it's not uncommon to have over 30 um different  context systems for an enterprise. And at the [18:50] bottom there's also different integrations like um  you know GitLab and GitHub Word. There's a lot of [18:56] information around your agents. And so my argument  today is that actually having endto-end visibility [19:02] and signals across all of these integrations is  what helps make the the reinforcement loop the [19:09] most powerful. And so to summarize to recap what  we've seen sort of again sort of imagine a world [19:18] where you you and your organization is actually  shipping agents in production and you can trust [19:23] them. You wake up in the morning and you know  that what you're providing is actually a trusted [19:28] product that both you and your peers can use. And  the core to that includes three different things. [19:34] One is a blueprint and actually you can use  your production environment to create this [19:40] reinforcement loop and to act on it. The second  is an agentic fleet that's working for you 24/7, [19:48] doesn't need to take a break. And the third is the  observability or sort of cohesive observability [19:54] foundation that powers the entire process and  gives you confidence that you're working off [19:59] of all of the signals that you could. So with  that, I will say thank you so much and I hope [20:05] uh if you're interested in learning any more, feel  free to reach out to me. Happy to show you some [20:10] um some working environments with what this  looks like. And uh good luck on your AI journey.