TraceRoot Automates AI Agent Bug Fixes From Live Production Traces

TraceRoot - open-source observability and self-improving layer for AI agents. With TraceRoot.AI — Ai4

Ai4

Converting live production failures directly into sandboxed regression tests and automated pull requests prevents isolated prompt fixes from triggering silent regressions across agent workflows.

AI-assisted summary of TraceRoot - open-source observability and self-improving layer for AI agents. With TraceRoot.AI — original source. Check the original for full context.

Source published:

Detailed notes

## Background Published on YouTube by Ai4 on August 15, 2026, this presentation features Xinwei He, Founder and CEO of TraceRoot.AI, detailing an open-source observability and self-improving layer for AI agents. The talk outlines how TraceRoot catches silent production failures using LLM-as-a-judge detectors and automatically generates hotfix PRs via sandboxed debugging agents.

## TL;DR While AI agents frequently perform well in demos, real-world production inputs often trigger silent hallucinations, broken tool calls, and unexpected regressions. TraceRoot closes this gap by implementing an automated Karpathy-style feedback loop that converts failing production traces into golden evaluation datasets. When live LLM-as-a-judge detectors identify an anomaly, the platform boots an isolated Daytona sandbox at the matching Git commit, uses a coding agent to pinpoint the root cause, and opens a GitHub fix PR. TraceRoot also exposes an agent-native CLI that allows coding assistants like Claude Code to inspect telemetry and diagnose multi-trace failure patterns autonomously.

## Atmosphere & style The presentation carries the pragmatic, solution-oriented energy of a technical conference demo tailored for developers dealing with production agent fragility. Xinwei He maintains a clear, confident, and developer-first delivery, shifting effortlessly between architectural diagrams and live sandbox runs. Watching the demo feels both practical and futuristic, offering immediate relief to on-call engineering pain by replacing stressful manual triage with zero-touch autonomous remediation.

## Key takeaways - [00:04](https://www.youtube.com/watch?v=f1sGCFM5-bA&t=4s) AI agents frequently succeed in sales demos and controlled settings but fail silently in production due to subtle variations in real-world user prompts. - [00:04](https://www.youtube.com/watch?v=f1sGCFM5-bA&t=4s) Standard offline evals only output benchmark scores and create isolated silos without providing actionable workflows for fixing code or prompt regressions. - [00:04](https://www.youtube.com/watch?v=f1sGCFM5-bA&t=4s) Applying prompt fixes for isolated production edge cases carries a high risk of silently breaking other critical agent workflows. - [00:04](https://www.youtube.com/watch?v=f1sGCFM5-bA&t=4s) TraceRoot creates a self-improving data flywheel inspired by Andrej Karpathy by streaming failing production traces directly into golden datasets for regression testing. - [01:53](https://www.youtube.com/watch?v=f1sGCFM5-bA&t=113s) The platform offers one-line instrumentation across frameworks like Claude Agent SDK, LangChain, Pydantic AI, as well as headless CLI tools like Claude Code and Codex. - [01:53](https://www.youtube.com/watch?v=f1sGCFM5-bA&t=113s) Real-time LLM-as-a-judge detectors continuously monitor live execution traces to flag anomalies such as hallucinations or broken execution chains. - [01:53](https://www.youtube.com/watch?v=f1sGCFM5-bA&t=113s) When a failure occurs, TraceRoot automatically provisions an isolated Daytona sandbox loaded with the production trace and the exact Git commit SHA. - [01:53](https://www.youtube.com/watch?v=f1sGCFM5-bA&t=113s) A cloud-based coding agent investigates the codebase, correlates errors with recent commit history, and opens an automated hotfix PR on GitHub. - [05:18](https://www.youtube.com/watch?v=f1sGCFM5-bA&t=318s) A live demonstration shows an injected bug in an open-tool agent script being flagged via span telemetry and triaged autonomously using the open-source PiMono coding agent. - [07:56](https://www.youtube.com/watch?v=f1sGCFM5-bA&t=476s) TraceRoot includes an npm-distributed CLI specifically designed for consumption by coding agents like Claude Code to query telemetry, aggregate failure patterns, and synthesize fixes. - [07:56](https://www.youtube.com/watch?v=f1sGCFM5-bA&t=476s) The entire TraceRoot ecosystem, encompassing SDKs, detectors, and the agentic CLI, is fully open-source with multi-runtime support.

## Detailed walkthrough ### [00:04](https://www.youtube.com/watch?v=f1sGCFM5-bA&t=4s) The Production Agent Dilemma and Closing the Karpathy Loop

Building and maintaining AI agents in production remains a notoriously fragile endeavor. While prototype agents routinely perform flawlessly during sales demos and controlled testing, they often unravel once exposed to real-world users who prompt them with subtle variations in phrasing. In production, an agent confronted with an unexpected request might trigger the wrong tool, repeatedly loop through malformed prompts, and ultimately return completely erroneous output. Standard industry practices rely on offline agent evaluations to catch these regressions, but offline evals merely spit out benchmark numbers without providing a clear, actionable workflow for engineering teams to remedy the underlying issues.

The core challenge lies in bridging the divide between local development and live production failures. When developers employ coding agents like Claude Code or OpenAI Codex connected to observability tooling, the agent may propose a fix—such as tweaking a prompt to address an isolated edge case. However, because prompt engineering is inherently non-deterministic, merging that patch risks triggering silent regressions across dozens of other workflows. TraceRoot aims to close this cycle into an autonomous feedback engine inspired by Andrej Karpathy’s data flywheel: streaming actual failing production traces directly into offline evaluation suites, converting real-world edge cases into golden regression tests, and validating code modifications before deployment.

```mermaid flowchart LR accTitle: TraceRoot Self-Improving Agent Loop subgraph Production ["Production Environment"] A["Agent Execution"] --> B["Traces & Tool Calls"] B --> C["LLM-as-a-Judge Detectors"] end subgraph SelfHealing ["Automated Remediation"] C -->|"Anomaly Detected"| D["Daytona Sandbox"] D --> E["Coding Agent Debugger"] E --> F["Automated Git Hotfix PR"] end subgraph Verification ["Evaluation & Guardrails"] F --> G["Update Golden Dataset"] G --> H["Offline Regresion Eval"] H -->|"Verified Safe"| A end ```

- **The Demo Trap:** Agents frequently achieve 100% demo success but break when production inputs deviate slightly from expected prompt structures. - **Evaluation Silos:** Traditional evaluation frameworks produce disconnected benchmark scores rather than feeding directly into real-time code and prompt remediation pipelines. - **Non-Deterministic Fixes:** A prompt modification targeted at a specific production failure can silently degrade performance on other critical workflows without continuous cross-dataset evaluation.

[01:53](https://www.youtube.com/watch?v=f1sGCFM5-bA&t=113s) Architectural Pillars: Instrumentation, Daytona Sandboxes, and Git-Aware PRs

TraceRoot tackles agent reliability through a modular four-tier architecture starting with zero-friction instrumentation. Designed as a one-line setup, it hooks natively into frameworks like the Claude Agent SDK, LangChain, and Pydantic AI. Recognizing a common pattern in San Francisco startups where developers encapsulate terminal tools like the Claude Code CLI or Codex CLI inside sandboxes for rapid scaffolding, TraceRoot ensures complete observability across both framework-based and CLI-driven agent implementations.

The platform distinguishes itself through its active monitoring, alerting, and autonomous remediation pipeline. When real-time LLM-as-a-judge detectors spot an anomaly—such as an 80% likelihood of hallucination or a broken execution chain—the system does far more than post a warning to Slack or email. It immediately boots an isolated sandbox using Daytona, downloads the failing production trace alongside the exact Git commit SHA running in production, and launches a cloud coding agent to diagnose the root cause. This reproduces the precise workflow of a human on-call engineer, allowing the agent to trace failures back to specific recent commits—noting, for example, that "three days ago your intern pushed a PR" that triggered the bug—and publish a verified hotfix PR directly to GitHub.

```mermaid flowchart TD accTitle: TraceRoot Cloud Debugging and Auto-Fix Pipeline subgraph Detect ["1. Detection & Ingestion"] T["Failing Production Trace"] --> D["Detector Flags Anomaly"] end subgraph Sandbox ["2. Sandbox Orchestration"] D --> SB["Daytona Sandbox Initialized"] GC["Git Clone at Production Commit SHA"] --> SB T --> SB end subgraph Remediation ["3. Autonomous Root-Cause Analysis"] SB --> CA["Open-Source Coding Agent"] CA --> AN["Commit History Analysis & Patch"] AN --> PR["GitHub Hotfix PR Opened"] end ```

- **Universal One-Line Setup:** Seamlessly captures execution traces from high-level frameworks (Claude Agent SDK, LangChain, Pydantic AI) as well as headless agent CLIs. - **Git-Aware Sandboxing:** Spins up ephemeral Daytona environments configured with the production commit hash and the exact trace payloads to simulate live troubleshooting. - **Automated Root-Cause Forensics:** Bypasses generic error codes like 502 Bad Gateway to pinpoint the specific pull request, commit, or prompt change that introduced the logic flaw. - **Golden Dataset Synchronization:** Exports problematic production traces into local golden datasets to verify that suggested fixes improve aggregate pass rates without regressions. - **AI-Native CLI:** Exposes full platform functionality through terminal tooling, enabling users—and their autonomous coding agents—to generate dashboards, manage API keys, and configure alert policies programmatically.

[05:18](https://www.youtube.com/watch?v=f1sGCFM5-bA&t=318s) Live Demo: Tracing Failures and Autonomous Cloud Debugging

To demonstrate the system in real time, a live test run is initiated using an intentional bug placed inside version 3 of an open-tool agent script. While the agent executes in the background, the TraceRoot UI displays standard trace telemetry: hierarchical span trees, granular input and output payloads, token consumption metrics, and sequential execution timelines. As the agent encounters the injected defect, the tracing interface immediately flags the anomalous spans with detailed error states.

Behind the scenes, the error triggers TraceRoot's automated remediation workflow. The platform runs multiple LLM-as-a-judge assertions against the live traces and pipes the resulting error logs directly into an open-source coding agent running inside an isolated cloud sandbox. Rather than forcing human engineers out of bed to decipher telemetry charts, the sandbox agent ingests the trace context, inspects the codebase, correlates the failing execution path with recent commit activity, and drafts a proposed hotfix pull request.

- **Trace Deep Dives:** Users can inspect individual LLM-as-a-judge evaluations, tracing the exact prompt-response pairs that tripped algorithmic thresholds. - **PiMono Integration:** The demonstration features PiMono, an open-source autonomous coding agent, serving as the sandbox debugger to inspect source files and propose code changes. - **Zero-Touch Triage:** The entire diagnostic sequence—from detector tripwire to sandbox spinning, code inspection, and hotfix authoring—runs autonomously without human engineer intervention.

[07:56](https://www.youtube.com/watch?v=f1sGCFM5-bA&t=476s) Agentic CLI in Action and the Open-Source Vision

TraceRoot provides a dedicated command-line tool, distributed via npm, complete with built-in autocompletion. Developers can interactively execute commands like `traces list` to review incoming production telemetry or `traces get` to unpack structured span trees and export them directly into their local file systems. However, the CLI is explicitly architected to be operated by coding agents rather than humans. By allowing an agent like Claude Code to invoke the CLI autonomously, developers can ask high-level operational questions in plain text, prompting the agent to query the API, inspect recent traces, and synthesize failure patterns across multiple executions.

During the demonstration, Claude Code is directed to query recent telemetry through the CLI and diagnose the latest issues. Within moments, the tool identifies two distinct failures across the last six agent executions, parses the trace anomalies, summarizes the technical root causes, and presents a concrete hotfix recommendation with proposed next steps.

```mermaid flowchart LR accTitle: Agentic CLI Interaction Flow subgraph UserPrompt ["Developer Command"] P["'Use CLI to summarize recent patterns'"] end subgraph AgentExecution ["Claude Code / Local Agent"] P --> C["Invoke TraceRoot CLI"] C --> TL["Run 'traces list' & 'traces get'"] end subgraph Output ["Automated Synthesis"] TL --> AN["Analyze 2 Failures Across Last 6 Calls"] AN --> R["Generate Root Cause & Hotfix Plan"] end ```

- **Open-Source Footprint:** The entire suite—including the instrumentation SDKs, the detection engine, and the agentic CLI—is fully open-sourced on GitHub. - **Broad Agent Ecosystem Support:** Supports emerging agentic runtimes including PiMono, Claude Code, OpenAI Codex, and managed agent frameworks, with a commitment to support new runtimes within days of release. - **Community Engagement:** Attendees are invited to visit Booth K816, join the project Discord, explore the GitHub repository, or schedule 15- to 30-minute 1-on-1 architecture and onboarding sessions.

## Notable quotes > What happens after your AI agent ships — when it silently hallucinates, misfires a tool call, or drifts, and you only hear about it from an angry user? — Xinwei He

> Detectors act as an LLM-as-judge over your live traces to catch failures automatically, then an agentic debugger spins up a sandbox with your production source, pinpoints the exact failing line, cross-references your GitHub history, and opens a fix PR. — Xinwei He

> You'll leave knowing how to catch silent agent failures before your users do — and how to go from a failing trace to a proposed fix without playing detective. — Xinwei He

> [00:04](https://www.youtube.com/watch?v=f1sGCFM5-bA&t=4s) While prototype agents routinely perform flawlessly during sales demos and controlled testing, they often unravel once exposed to real-world users who prompt them with subtle variations in phrasing. — Xinwei He

> [00:04](https://www.youtube.com/watch?v=f1sGCFM5-bA&t=4s) Standard industry practices rely on offline agent evaluations to catch these regressions, but offline evals merely spit out benchmark numbers without providing a clear, actionable workflow for engineering teams to remedy the underlying issues. — Xinwei He

> [00:04](https://www.youtube.com/watch?v=f1sGCFM5-bA&t=4s) When developers employ coding agents like Claude Code or OpenAI Codex connected to observability tooling, the agent may propose a fix—such as tweaking a prompt to address an isolated edge case. However, because prompt engineering is inherently non-deterministic, merging that patch risks triggering silent regressions across dozens of other workflows. — Xinwei He

> [01:53](https://www.youtube.com/watch?v=f1sGCFM5-bA&t=113s) three days ago your intern pushed a PR — Xinwei He

> [01:53](https://www.youtube.com/watch?v=f1sGCFM5-bA&t=113s) When real-time LLM-as-a-judge detectors spot an anomaly—such as an 80% likelihood of hallucination or a broken execution chain—the system does far more than post a warning to Slack or email. — Xinwei He

> [05:18](https://www.youtube.com/watch?v=f1sGCFM5-bA&t=318s) Rather than forcing human engineers out of bed to decipher telemetry charts, the sandbox agent ingests the trace context, inspects the codebase, correlates the failing execution path with recent commit activity, and drafts a proposed hotfix pull request. — Xinwei He

> [05:18](https://www.youtube.com/watch?v=f1sGCFM5-bA&t=318s) The entire diagnostic sequence—from detector tripwire to sandbox spinning, code inspection, and hotfix authoring—runs autonomously without human engineer intervention. — Xinwei He

> [07:56](https://www.youtube.com/watch?v=f1sGCFM5-bA&t=476s) By allowing an agent like Claude Code to invoke the CLI autonomously, developers can ask high-level operational questions in plain text, prompting the agent to query the API, inspect recent traces, and synthesize failure patterns across multiple executions. — Xinwei He

## Mentioned TraceRoot, TraceRoot.AI, Xinwei He, Ai4, Claude Agent SDK, Vercel AI SDK, Claude Code, OpenAI Codex, Andrej Karpathy, Daytona, LangChain, Pydantic AI, GitHub, PiMono, Slack, Discord, npm

## Video description Xinwei He - Founder and CEO, TraceRoot.AI

What happens after your AI agent ships — when it silently hallucinates, misfires a tool call, or drifts, and you only hear about it from an angry user? In this talk I'll show how to turn observability into a self-improving loop: Detectors act as an LLM-as-judge over your live traces to catch failures automatically, then an agentic debugger spins up a sandbox with your production source, pinpoints the exact failing line, cross-references your GitHub history, and opens a fix PR. I'll demo this end-to-end on real traces, across frameworks like Claude Agent SDK, and the Vercel AI SDK and coding tools like Claude Code and Codex. You'll leave knowing how to catch silent agent failures before your users do — and how to go from a failing trace to a proposed fix without playing detective.

[Source](https://www.youtube.com/watch?v=f1sGCFM5-bA)

中文简报

TraceRoot基于生产追踪数据实现AI Agent故障自动修复 将实际生产故障直接转换为沙盒回归测试和自动化PR，可防止单一提示词修复引发Agent工作流的隐性回归。 尽管AI Agent在演示中通常表现优异，但现实中的生产输入经常会触发隐性幻觉、工具调用失败以及意外回归。TraceRoot通过构建受Karpathy启发的自动化反馈闭环来弥合这一差距，将失败的生产追踪数据直接转化为黄金评估数据集。当在线LLM-as-a-judge检测器发现异常时，平台会在对应的Git提交记录上启动一个隔离的Daytona沙盒，利用编码Agent查明根本原因并提交GitHub修复PR。此外，TraceRoot还提供了Agent原生的CLI，使Claude Code等编码助手能够自主检查遥测数据并诊断多追踪链路的故障模式。 AI Agent 在 Demo 中表现优异但在生产环境中极易因微小的提示词差异而失效；TraceRoot 借鉴 Andrej Karpathy 的数据飞轮理念，将生产环境中的失败 Trace 直接导入离线黄金测试集，解决孤立提示词修改易引发隐性回归的难题。 (0:04) 平台提供针对 Claude Agent SDK、LangChain、Pydantic AI 以及 Claude Code 和 Codex 等终端 CLI 工具的一行代码接入方案，利用实时的 LLM-as-a-judge 检测器对幻觉和执行链断裂等异常进行持续监控。 (1:53) 当监测到执行异常时，TraceRoot 会自动启动配置了对应生产 Commit SHA 和 Trace 数据的 Daytona 隔离沙箱，调度开源编码 Agent 追溯导致 Bug 的代码提交并直接向 GitHub 提交热修复 PR。 (5:18) TraceRoot 推出了专为编码 Agent 设计的开源 CLI 工具，现场演示了 Claude Code 如何直接调用该 CLI 快速分析近期的执行异常、排查根因并自动生成修复建议。 (7:56)

Reading summary

TraceRoot proposes an open-source observability and self-improving architecture designed to resolve the fragility of AI agents in production. The platform captures live execution failures and channels them into isolated sandboxes where autonomous coding agents diagnose root causes and draft verified code patches. Production Agent Fragility and the Evaluation Gap (0:04) According to the presentation, prototype AI agents often succeed during controlled sales demonstrations but fail under production conditions due to minor phrasing variations, incorrect tool selection, and recursive prompt loops. Standard industry workflows rely on offline evaluation benchmarks that output aggregate metrics without providing actionable paths for code remediation. Furthermore, because prompt engineering is non-deterministic, isolated patches deployed to resolve a specific edge case risk triggering silent regressions across unrelated workflows. TraceRoot conceptualizes this problem through Andrej Karpathy's data flywheel concept, arguing that production trace failures must feed directly into continuous offline evaluation suites rather than remaining siloed telemetry. Git-Aware Sandboxing and Automated Forensics (1:53) TraceRoot's architecture begins with zero-friction instrumentation compatible with frameworks like LangChain, Pydantic AI, and the Claude Agent SDK, as well as CLI-based agent wrappers. When real-time LLM-as-a-judge detectors spot an anomaly—such as an 80% probability of hallucination or a broken execution sequence—the platform bypasses passive Slack or email notifications. Instead, it launches an ephemeral cloud sandbox via Daytona, configured with the failing trace and the precise Git commit SHA running in production. An open-source coding agent conducts root-cause analysis by correlating the failure directly with recent commit history—such as a specific pull request merged days prior—bypassing generic error codes to author an automated hotfix pull request on GitHub. Live Verification and Golden Dataset Integration (5:18) In a live test involving an intentional bug in an open-tool agent script, TraceRoot displayed hierarchical span trees, execution payloads, token usage, and timeline metrics before triggering its automated remediation workflow. An open-source coding agent named PiMono was deployed inside the Daytona sandbox to inspect source files, correlate the failure with recent commits, and draft a pull request without human intervention. To prevent regression cascades across subsequent releases, TraceRoot synchronizes problematic production traces directly into local golden evaluation datasets. Proposed code modifications are executed against these historical test suites to ensure that fixing an isolated edge case elevates overall pass rates rather than degrading performance on existing workflows. Agent-Operated CLI and Open Ecosystem (7:56) TraceRoot distributes a command-line interface via npm featuring commands like `traces list` and `traces get`. While human developers can use it, the tool is explicitly structured for programmatic use by coding agents like Claude Code. In a demonstration, an agent queried through the CLI autonomously diagnosed two failures out of six recent executions, synthesized the technical root causes, and presented a structured remediation plan. The entire platform—spanning instrumentation SDKs, anomaly detectors, and the agentic CLI—is fully open-sourced on GitHub, with committed support for emerging runtimes and autonomous coding tools like PiMono and OpenAI Codex. TraceRoot claims that automating the sequence from trace anomaly detection to sandboxed PR generation provides a path toward self-improving agents, though its operational success relies on the diagnostic reliability of autonomous coding agents and the accuracy of LLM-as-a-judge evaluations.

精炼阅读摘要

TraceRoot.AI 针对生产环境中 AI 智能体易因输入细微变化而失效的问题，推出了一套开源可观测性与自主修复层，通过借鉴 Andrej Karpathy 的数据飞轮理念，将线上失败追踪直接转化为离线回归测试并驱动自动代码修复。 生产环境失效与传统评估局限 (0:04) 在演示和受控测试中表现完美的 AI 智能体，进入生产环境后常因用户提示语的细微差异而失效，表现为调用错误工具、陷入异常提示循环或输出错误结果。行业通常依赖离线评估来捕捉回归，但离线评估仅能输出基准跑分，无法为工程团队提供可落地的修复流程。若开发者仅针对个别边界用例调整提示词，由于提示工程固有的非确定性，合入补丁容易在其他工作流中引发隐性回归。 沙箱重现与全自动 Git 热修复 (1:53) TraceRoot 采用四层架构，提供面向 Claude Agent SDK、LangChain 和 Pydantic AI 等框架以及无头终端 CLI 的一行代码埋点方案。当系统内的实时大模型裁判（LLM-as-a-judge）检测到异常（例如 80% 的幻觉概率或执行链断裂）时，平台不会只发送报警通知，而是自动在 Daytona 中启动隔离沙箱，拉取线上故障追踪及对应的生产环境 Git Commit SHA。随后，云端编码智能体接管排查流程，分析近期提交记录定位引入缺陷的具体代码或提示词变更，并直接向 GitHub 提交验证过的热修复 PR。 黄金数据集闭环与零干预排障验证 (5:18) 系统通过将线上捕获的异常追踪同步至本地黄金数据集，确保生成的修复方案在合并前通过回归评估，避免破坏整体通过率。在现场演示中，演练团队向开源工具智能体代码中注入了缺陷，TraceRoot 追踪界面在展示层次化跨度树、输入输出负载及 Token 消耗的同时即刻捕获异常状态。开源编码智能体 PiMono 随即在云端沙箱中完成代码审查，并在无需人工工程师介入的情况下完成了从警报触发、上下文分析到草拟热修复 PR 的全流程闭环。 面向智能体调用的 CLI 与开源生态 (7:56) 除常规控制台外，TraceRoot 提供了通过 npm 分发的命令行工具，且专门针对编码智能体的调用进行了适配。开发者可使用 Claude Code 等工具以自然语言下发排查指令，由智能体自主执行 traces list 和 traces get 等命令拉取并聚合分析近期故障模式。演示中，Claude Code 在短时间内解析了最近 6 次执行中的 2 起故障，给出了技术根因总结与修复建议。该项目的埋点 SDK、检测引擎与 CLI 工具已全面在 GitHub 开源，并支持与多种智能体运行时对接。 TraceRoot 试图将生产环境故障排查与修复流程彻底自动化，但提示词工程本身的非确定性以及自动生成热修复代码在复杂业务场景下的可靠性，仍依赖黄金测试集的覆盖完备度与实际落地验证。

Detailed notes — zh

## Background 该视频由频道 Ai4 于 2026 年 8 月 15 日发布在 YouTube 平台。内容为 TraceRoot.AI 创始人兼 CEO 何欣伟（Xinwei He）的技术演讲，主要介绍了如何通过开源可观测性与沙盒调试层，构建一套捕获 AI Agent 生产异常并自动生成修复代码的自改进闭环系统。 ## TL;DR 演讲聚焦于 AI Agent 在生产环境中面临的静默幻觉与非确定性工具调用错误等核心痛点。TraceRoot 通过一行代码插桩、LLM-as-a-Judge 实时检测、Daytona 沙盒自动分析以及黄金数据集评估，打通了生产 Trace 反哺线下基准测试的数据飞轮。现场演示展示了当线上出现故障时，沙盒中的云端 Coding Agent 能自动关联 GitHub 提交历史并输出修复 PR。此外，项目全面开源了 AI 原生 CLI 工具，支持与 Claude Code 等编码工具深度联动，实现全自动化的排错与回归验证。 ## Atmosphere & style 演讲节奏明快紧凑，风格极具硬核工程师导向与前瞻性。主讲人以开发者深有共鸣的“线上翻车”痛点切入，结合行云流水的实时界面与终端 CLI 演示，生动展示了 sandboxed agent 自动调试与修复代码的自动化流程。整体氛围既专业务实又充满极客感，直观传递出 AI 原生可观测性工具带来的开发范式变革。 ## Key takeaways - [00:04](https://www.youtube.com/watch?v=f1sGCFM5-bA&t=4s) AI Agent 在投产时面临 Demo 表现完美但线上易因边缘输入出现工具误调或幻觉的难题，且传统离线评估难以闭环反哺生产。 - [00:04](https://www.youtube.com/watch?v=f1sGCFM5-bA&t=4s) 单纯依赖本地 Coding Agent 修复非确定性 Prompt 极易引发其他场景的逻辑回归（Regression），缺乏可靠验证手段。 - [00:04](https://www.youtube.com/watch?v=f1sGCFM5-bA&t=4s) TraceRoot 借鉴闭环数据飞轮理念，直接将生产环境真实 Trace 转化为评估数据集，构建从捕获到验证的 Agent 自改进闭环。 - [01:53](https://www.youtube.com/watch?v=f1sGCFM5-bA&t=113s) 系统支持一行代码插桩，原生适配 Claude Agent SDK、LangChain、Pydantic 以及在沙盒运行的 Claude Code/Codex CLI。 - [01:53](https://www.youtube.com/watch?v=f1sGCFM5-bA&t=113s) 结合 LLM-as-a-Judge 实时异常检测，系统在报警时会自动拉起 Daytona 隔离沙盒并同步报错 Trace 与生产对应 Commit 源码。 - [01:53](https://www.youtube.com/watch?v=f1sGCFM5-bA&t=113s) 沙盒中运行的云端 Coding Agent 能够精准定位引发 Bug 的具体代码行与历史提交，并自动生成 Hotfix PR 修复方案。 - [01:53](https://www.youtube.com/watch?v=f1sGCFM5-bA&t=113s) 支持通过 CLI 将异常 Trace 快速归入黄金数据集（Golden Dataset），在 CI/本地环境中重跑评估以彻底杜绝回归风险。 - [01:53](https://www.youtube.com/watch?v=f1sGCFM5-bA&t=113s) 具备 AI 原生 CLI 与 MCP 支持，Web 端操作均可被上层编码 Agent 以工具调用的方式自主执行。 - [05:18](https://www.youtube.com/watch?v=f1sGCFM5-bA&t=318s) 可观测性平台支持细粒度 Span 追踪、Token 消耗下钻以及多维度独立的 LLM 实时判定指标。 - [05:18](https://www.youtube.com/watch?v=f1sGCFM5-bA&t=318s) 全流程支持无人值守自动化诊断，无需工程师半夜手动排查即可自动生成详尽的根因分析报告与修复补丁。 - [07:56](https://www.youtube.com/watch?v=f1sGCFM5-bA&t=476s) TraceRoot CLI 可直接作为 Claude Code 等 Agent 的下游工具，支持通过自然语言提问自动抓取指标并归纳错误模式。 - [07:56](https://www.youtube.com/watch?v=f1sGCFM5-bA&t=476s) TraceRoot 核心技术栈全面开源，支持自托管部署并深度兼容 Pi-Mono、Claude Code、Codex 等生态组件。 ## Detailed walkthrough ### [00:04](https://www.youtube.com/watch?v=f1sGCFM5-bA&t=4s) AI Agent 投产痛点与闭环自改进架构 构建 AI Agent 并将其维护在生产环境中是一项极其困难的任务。几乎所有人做出的 AI Agent 在 Demo 演示阶段都表现得完美无瑕，但一旦部署到生产环境，真实用户开始提出各种稍微偏离预期的问题时，Agent 就可能频繁调用错误的工具或生成偏离的 Prompt，最终产生完全错误的输出。当前行业的标准应对方案主要是运行离线的 Agent 评估（Offline Eval）并生成一系列分数，但如何最高效地利用这些评估分数去反哺系统，行业内尚无清晰的路径。即使开发者使用本地编码 Agent（如 Claude Code、Codex）连接可观测性工具来修复 Bug 并提出 PR，如果这个 PR 只是修改了非确定性的 Prompt，开发者依然无法确保该修改不会在其他海量场景中引发回归（Regression）。 为了解决线上监控与线下评估割裂的难题，TraceRoot 借鉴了类似 Andrej Karpathy 提出的闭环数据飞轮理念：直接将生产环境中的真实运行轨迹（Traces）转化为评估数据集，用来更新线下基准测试以验证修复有效性，从而彻底打通 Agent 的自改进闭环。 ```mermaid flowchart TD accTitle: TraceRoot 闭环自改进流程 subgraph S1["1. 生产运行与捕获"] A["生产环境 Agent 运行"] --> B["一行代码插桩 / 实时 Trace 捕获"] end subgraph S2["2. 异常检测与自动定位"] B --> C["LLM-as-a-Judge 实时检测"] C --> D["启动 Daytona 沙盒"] D --> E["拉取生产 Commit 源码与错误 Trace"] end subgraph S3["3. 自动修复与评估闭环"] E --> F["云端 Coding Agent 生成修复 PR / Hotfix"] F --> G["Trace 转化为黄金数据集并运行 CLI 离线评估"] G --> H["验证无回归后合并上线"] end ``` ### [01:53](https://www.youtube.com/watch?v=f1sGCFM5-bA&t=113s) TraceRoot 四大核心能力：从一行插桩到沙盒自动修复 TraceRoot 的核心能力主要由四个环节构成，覆盖了从接入、监控、排错到回归验证的完整生命周期： 1. **一行代码插桩（One-line Instrumentation）**：除了基础的可观测性指标采集，TraceRoot 原生支持各类主流 Agent 框架的一行接入，包括 Claude Agent SDK、LangChain、Pydantic 等。针对旧金山很多团队为了便利直接在沙盒中运行 Claude Code CLI 或 Codex CLI 的开发模式，TraceRoot 也提供了深度适配支持。 2. **监控、告警与自动修复（Monitoring, Auto-fix & Alerts）**：这是 TraceRoot 最具创新性的护城河功能。当线上 LLM-as-a-Judge 检测到异常（例如有 80% 概率存在幻觉、逻辑错误或工具调用失败）时，系统不仅会在 Slack 或邮件中发送报警，还会即时在后台启动一个隔离沙盒（当前基于 Daytona 等沙盒环境）。沙盒会自动拉取报错的完整 Trace，并结合 GitHub 仓库中对应生产环境的确切 Commit ID，完整模拟工程师排查生产 Bug 的全过程，交由云端 Coding Agent 进行代码分析并直接生成 Hotfix PR。系统给出的告警不再只是冷冰冰的网关报错，而是明确指出类似于“三天前实习生提交的代码导致了某处逻辑回归”，并附带修复方案。 3. **评估与黄金数据集更新（Eval & Dataset Update）**：针对非确定性 Prompt 修复难以验证有效性的问题，开发者可以通过 CLI 将生产中捕获的问题 Trace 直接归入黄金数据集（Golden Dataset），并在本地或 CI 中针对该数据集重新运行评估，确保分数提升且没有任何旧用例发生回归，真正闭环验证修复。 4. **AI 原生 CLI 与生态工具（AI-Native Features）**：TraceRoot 提供了全功能的 CLI 工具与 MCP（Model Context Protocol）支持。用户在 Web 界面上能执行的操作（如创建仪表盘、生成项目、新建 API Key、配置告警等），均可通过 CLI 由上层 Coding Agent 自主调用执行。 ### [05:18](https://www.youtube.com/watch?v=f1sGCFM5-bA&t=318s) 现场演示：实时 Traces 捕获与沙盒 Agent 自动化诊断 在现场实操演示中，主讲人展示了一个标准的 Agent 可观测性界面，并通过触发包含故意植入 Bug 的多工具调用 Agent（Open Tool Agent，带有故意写错的 main version 3 代码）来展示系统的实时响应能力。 - **Span 详情与指标下钻**：在 Trace 详情页中，开发者可以清晰查看每一次调用的 Span 分层、输入输出内容、Token 消耗量以及时间线。 - **实时错误捕获与多维度检测**：当运行产生异常时，内置的 LLM-as-a-Judge 检测器会立即被激活（演示用例中配置了 4 项独立的 Judge 检查），并针对错误 Span 展开判定。 - **沙盒 Coding Agent 诊断分析**：触发告警后，系统会将问题 Trace 的所有上下文作为 Prompt 输入并导入云端沙盒中运行的开源编码 Agent（如 Pi-Mono）。该 Agent 会在沙盒中对报错原因展开多维度排查，定位出错的具体代码行与历史 Commit 关联，并输出详细的错误分析和建议修复方案（Hotfix）。 - **无人值守的自动化体验**：整个排错与代码分析流程完全自动化，无需工程师半夜被告警唤醒人工介入；开发者早晨上班时便能直接查看已经生成好的分析报告与修复补丁。 ### [07:56](https://www.youtube.com/watch?v=f1sGCFM5-bA&t=476s) AI 原生 CLI 工具演示与开源生态展望 在终端演示环节，主讲人展示了 TraceRoot 的 CLI 工具。该 CLI 支持通过 npm 安装，具备自动补全能力，不仅支持 `traces list`（列出所有 Trace）、`traces get`（获取指定 Trace 结构）以及将 Trace 下载到本地文件系统等基础指令，更核心的用法是作为 Coding Agent（如 Claude Code）的“下游工具”。 开发者无需手动记忆并敲击 CLI 命令，只需向 Claude Code 提问："使用 CLI 获取最近的异常发现，并总结出错误模式"，Claude Code 即可自主多次调用 TraceRoot CLI 抓取必要的数据指标、拉取最近失败调用的分析报告，并在终端直接归纳出问题规律与修复建议。 - **全面开源**：TraceRoot 的整个技术栈（包括 AI 原生 CLI、SDK、插桩工具等）均已开源，全面支持 Pi-Mono、Claude Code、Codex、Claude Agent SDK 等生态组件。 - **后续验证与支持**：TraceRoot 支持直接将线上 Trace 同步至本地离线测试集以完成回归验证。主讲人现场公布了展位信息（Booth K816），并邀请开发者通过 GitHub、Twitter、官网或 Discord 体验自托管部署与交流。 ## Notable quotes > [00:04](https://www.youtube.com/watch?v=f1sGCFM5-bA&t=4s) 构建 AI Agent 并将其维护在生产环境中是一项极其困难的任务。几乎所有人做出的 AI Agent 在 Demo 演示阶段都表现得完美无瑕，但一旦部署到生产环境，真实用户开始提出各种稍微偏离预期的问题时，Agent 就可能频繁调用错误的工具或生成偏离的 Prompt，最终产生完全错误的输出。 — Xinwei He > [00:04](https://www.youtube.com/watch?v=f1sGCFM5-bA&t=4s) 即使开发者使用本地编码 Agent（如 Claude Code、Codex）连接可观测性工具来修复 Bug 并提出 PR，如果这个 PR 只是修改了非确定性的 Prompt，开发者依然无法确保该修改不会在其他海量场景中引发回归（Regression）。 — Xinwei He > [00:04](https://www.youtube.com/watch?v=f1sGCFM5-bA&t=4s) TraceRoot 借鉴了类似 Andrej Karpathy 提出的闭环数据飞轮理念：直接将生产环境中的真实运行轨迹（Traces）转化为评估数据集，用来更新线下基准测试以验证修复有效性，从而彻底打通 Agent 的自改进闭环。 — Xinwei He > [01:53](https://www.youtube.com/watch?v=f1sGCFM5-bA&t=113s) 沙盒会自动拉取报错的完整 Trace，并结合 GitHub 仓库中对应生产环境的确切 Commit ID，完整模拟工程师排查生产 Bug 的全过程，交由云端 Coding Agent 进行代码分析并直接生成 Hotfix PR。 — Xinwei He > [01:53](https://www.youtube.com/watch?v=f1sGCFM5-bA&t=113s) 系统给出的告警不再只是冷冰冰的网关报错，而是明确指出类似于“三天前实习生提交的代码导致了某处逻辑回归”，并附带修复方案。 — Xinwei He > [01:53](https://www.youtube.com/watch?v=f1sGCFM5-bA&t=113s) 针对非确定性 Prompt 修复难以验证有效性的问题，开发者可以通过 CLI 将生产中捕获的问题 Trace 直接归入黄金数据集（Golden Dataset），并在本地或 CI 中针对该数据集重新运行评估，确保分数提升且没有任何旧用例发生回归，真正闭环验证修复。 — Xinwei He > [01:53](https://www.youtube.com/watch?v=f1sGCFM5-bA&t=113s) 用户在 Web 界面上能执行的操作（如创建仪表盘、生成项目、新建 API Key、配置告警等），均可通过 CLI 由上层 Coding Agent 自主调用执行。 — Xinwei He > [05:18](https://www.youtube.com/watch?v=f1sGCFM5-bA&t=318s) 整个排错与代码分析流程完全自动化，无需工程师半夜被告警唤醒人工介入；开发者早晨上班时便能直接查看已经生成好的分析报告与修复补丁。 — Xinwei He > [07:56](https://www.youtube.com/watch?v=f1sGCFM5-bA&t=476s) 开发者无需手动记忆并敲击 CLI 命令，只需向 Claude Code 提问："使用 CLI 获取最近的异常发现，并总结出错误模式"，Claude Code 即可自主多次调用 TraceRoot CLI 抓取必要的数据指标、拉取最近失败调用的分析报告，并在终端直接归纳出问题规律与修复建议。 — Xinwei He ## Mentioned Xinwei He, TraceRoot.AI, TraceRoot, Andrej Karpathy, Claude Agent SDK, Vercel AI SDK, LangChain, Pydantic, Claude Code, Codex, Daytona, Slack, GitHub, Pi-Mono, MCP (Model Context Protocol), npm, Discord, Twitter ## Video description Xinwei He - Founder and CEO, TraceRoot.AI What happens after your AI agent ships — when it silently hallucinates, misfires a tool call, or drifts, and you only hear about it from an angry user? In this talk I'll show how to turn observability into a self-improving loop: Detectors act as an LLM-as-judge over your live traces to catch failures automatically, then an agentic debugger spins up a sandbox with your production source, pinpoints the exact failing line, cross-references your GitHub history, and opens a fix PR. I'll demo this end-to-end on real traces, across frameworks like Claude Agent SDK, and the Vercel AI SDK and coding tools like Claude Code and Codex. You'll leave knowing how to catch silent agent failures before your users do — and how to go from a failing trace to a proposed fix without playing detective. [Source](https://www.youtube.com/watch?v=f1sGCFM5-bA)

Source quotes

[0:04] Xinwei He: 构建 AI Agent 并将其维护在生产环境中是一项极其困难的任务。几乎所有人做出的 AI Agent 在 Demo 演示阶段都表现得完美无瑕，但一旦部署到生产环境，真实用户开始提出各种稍微偏离预期的问题时，Agent 就可能频繁调用错误的工具或生成偏离的 Prompt，最终产生完全错误的输出。 [0:04] Xinwei He: 即使开发者使用本地编码 Agent（如 Claude Code、Codex）连接可观测性工具来修复 Bug 并提出 PR，如果这个 PR 只是修改了非确定性的 Prompt，开发者依然无法确保该修改不会在其他海量场景中引发回归（Regression）。 [0:04] Xinwei He: TraceRoot 借鉴了类似 Andrej Karpathy 提出的闭环数据飞轮理念：直接将生产环境中的真实运行轨迹（Traces）转化为评估数据集，用来更新线下基准测试以验证修复有效性，从而彻底打通 Agent 的自改进闭环。 [1:53] Xinwei He: 沙盒会自动拉取报错的完整 Trace，并结合 GitHub 仓库中对应生产环境的确切 Commit ID，完整模拟工程师排查生产 Bug 的全过程，交由云端 Coding Agent 进行代码分析并直接生成 Hotfix PR。 [1:53] Xinwei He: 系统给出的告警不再只是冷冰冰的网关报错，而是明确指出类似于“三天前实习生提交的代码导致了某处逻辑回归”，并附带修复方案。 [1:53] Xinwei He: 针对非确定性 Prompt 修复难以验证有效性的问题，开发者可以通过 CLI 将生产中捕获的问题 Trace 直接归入黄金数据集（Golden Dataset），并在本地或 CI 中针对该数据集重新运行评估，确保分数提升且没有任何旧用例发生回归，真正闭环验证修复。 [1:53] Xinwei He: 用户在 Web 界面上能执行的操作（如创建仪表盘、生成项目、新建 API Key、配置告警等），均可通过 CLI 由上层 Coding Agent 自主调用执行。 [5:18] Xinwei He: 整个排错与代码分析流程完全自动化，无需工程师半夜被告警唤醒人工介入；开发者早晨上班时便能直接查看已经生成好的分析报告与修复补丁。 [7:56] Xinwei He: 开发者无需手动记忆并敲击 CLI 命令，只需向 Claude Code 提问："使用 CLI 获取最近的异常发现，并总结出错误模式"，Claude Code 即可自主多次调用 TraceRoot CLI 抓取必要的数据指标、拉取最近失败调用的分析报告，并在终端直接归纳出问题规律与修复建议。

Source quotes

Xinwei He: What happens after your AI agent ships — when it silently hallucinates, misfires a tool call, or drifts, and you only hear about it from an angry user? Xinwei He: Detectors act as an LLM-as-judge over your live traces to catch failures automatically, then an agentic debugger spins up a sandbox with your production source, pinpoints the exact failing line, cross-references your GitHub history, and opens a fix PR. Xinwei He: You'll leave knowing how to catch silent agent failures before your users do — and how to go from a failing trace to a proposed fix without playing detective. [0:04] Xinwei He: While prototype agents routinely perform flawlessly during sales demos and controlled testing, they often unravel once exposed to real-world users who prompt them with subtle variations in phrasing. [0:04] Xinwei He: Standard industry practices rely on offline agent evaluations to catch these regressions, but offline evals merely spit out benchmark numbers without providing a clear, actionable workflow for engineering teams to remedy the underlying issues. [0:04] Xinwei He: When developers employ coding agents like Claude Code or OpenAI Codex connected to observability tooling, the agent may propose a fix—such as tweaking a prompt to address an isolated edge case. However, because prompt engineering is inherently non-deterministic, merging that patch risks triggering silent regressions across dozens of other workflows. [1:53] Xinwei He: three days ago your intern pushed a PR [1:53] Xinwei He: When real-time LLM-as-a-judge detectors spot an anomaly—such as an 80% likelihood of hallucination or a broken execution chain—the system does far more than post a warning to Slack or email. [5:18] Xinwei He: Rather than forcing human engineers out of bed to decipher telemetry charts, the sandbox agent ingests the trace context, inspects the codebase, correlates the failing execution path with recent commit activity, and drafts a proposed hotfix pull request. [5:18] Xinwei He: The entire diagnostic sequence—from detector tripwire to sandbox spinning, code inspection, and hotfix authoring—runs autonomously without human engineer intervention. [7:56] Xinwei He: By allowing an agent like Claude Code to invoke the CLI autonomously, developers can ask high-level operational questions in plain text, prompting the agent to query the API, inspect recent traces, and synthesize failure patterns across multiple executions.

Original transcript

[0:04] uh we are building the open source  self-improving layer for AI agents. Uh [0:10] so building AI agents and maintaining it in  production is a very difficult job. Uh so I think [0:17] everybody in this room are building some sort of  AI agents and then it works perfectly in the demo. [0:23] But if it after you ship it to production the  customer start asking slightly different question [0:29] then your agents will it might cause is  the cause the wrong tool or cause the [0:34] wrong uh prompt several times and then end up in  generating something that is completely wrong. [0:39] So how do you deal with that? like one I guess the  industry standard right now is that we could just [0:45] run offline agent eval right but is able to  like generate a bunch of scores but how do [0:51] you use these scores in the most efficient manner  that part is uh not clear and then let's say you [0:57] are using some type of a local coding agent like  co codeex I would imagine that's what everybody [1:03] are using and you do that it connect to your  observability tools and it fixes some bugs [1:08] it suggest a PR but how do you Oh, if you can just  merge that PR let's say it's a prompt change such [1:15] that it's able to solve this exact problem because  it's uh not deterministic right and then uh [1:22] how do you make sure that it doesn't regresses  in lots of other cases so all these things are [1:27] difficult and and most importantly you do not have  a very good way to loop all these things together [1:33] like online evaluation uh compared to the offline  agent eval so that's what we hope that we can [1:39] close it's like a loop idea in the sense of  Android party where you are able to like using [1:46] the production traces as data set and update  your offline to verify everything is good. Um [1:53] so this is the first step basically first is  the uh instrumentation. So we do like this is [1:59] bread and butter of course but we also support  like the oneline setup for any agent frameworks. [2:04] Let's say are you are using like cloudy SDK  launch chain or pedantic those are pretty typical [2:10] these days I as I talk to more people in San  Francisco somebody just for the convenience [2:15] of it they are suggesting that they have a  sandbox and they literally run cloud code CLI [2:19] or like codec cli that's another option uh  is less customizable but either way we are [2:24] able to have like full support for that with one  line instrumentation which I will show very soon [2:30] uh the next is the monitoring autofig and  alerts I I think this is where I would say the [2:37] our innovation comes because I don't see a lot of  other competitors make do this and do this very [2:42] well. Uh so what it does is of course the online  I as a judge that's uh the basic stuff but once [2:49] we detect there is a problem let's say we detects  like 80% uh chance that there is a hallucination [2:55] then we just don't we don't just generate a score  and put it there we literally alert you either [3:00] in email or slack whichever twice and then the  coolest part I feel like is this we will literally [3:07] uh spin up a sandbox uh it's uh right now we are  using Daytona but there are many other vendors [3:11] in the space we We have the sandbox we download  the whole trees that we feel like have a problem. [3:16] Let's say it's hallucinates or it has error or it  has logical error doesn't matter. We download that [3:21] and then we will download like do gflow of your  actual repo after you connected to our platform [3:27] with the exact commit ID that is running in  production. So that basically what it does is [3:32] simulate the how the engineer debug any sort  of like AI problem in production. we do that [3:38] uh run like a cloud cloud coding agent um for run  and then we will actually send you an alert with a [3:45] fix. So he's not not going to say oh you have like  a f01 like gateway error you come with a fix as to [3:52] maybe like three days ago your intern push a PR  and it causes certain certain regression and now [3:57] here's a fix sometimes it gave you like a directly  give you a github PR but obviously you don't need [4:03] to merge it you can verify the third is eval and  data set update so let's say we suggest there is a [4:10] there's a PR like like I just mentioned that  how are you sure is good right like maybe just [4:14] a prompt It's not deterministic. So you are not  sure if it's good or not good. Then with our CLI [4:21] you are able to use whatever trees you are running  in production put it in your golden data set [4:26] and then just run eval on that then you are sure  that okay the score doesn't regress or sometimes [4:32] even getting a little bit better uh ideally  then you verify as a fix. So that basically [4:36] close the loop. The last is the uh AI native type  of feature. I know some lots of competitor also [4:42] have the CLI they have the MCP but what we are  trying to do is make it slightly more powerful [4:48] meaning that our CLI is able to like access do  everything you you can do in UI let's say you [4:54] click a button to generate a dashboard or click  a generate a new project create a new API key or [4:59] let's say you set up a new alert we are able to do  everything through the CLI of course with scarios [5:05] and this feature is a little bit in a beta stage  but this This is something that we will invest a [5:11] lot in and making sure it really works. And now is  the interesting part. I will do a quick live demo. [5:18] Okay. So here you can see this is a pretty  basic like agent observability tool setup [5:23] where everybody like all the competitor has uh  I'm going to trigger some LM agent runs right now. [5:30] It takes a while. So I'll just in the meanwhile  I just introduce other part of the feature here. [5:34] I just ask it to run like open tool agent and I  put actually I put a bug in main version three. [5:41] So let's wait. Uh yeah in the meanwhile I can show  the what a product looks like. Let's see you click [5:47] into one of the traces. H it shows uh okay so all  the like like different spam span tabs and then [5:53] you can see the input output call the token usage  and uh I think everything the timeline this is of [6:00] course it's a basic feature should be supporting  a detector uh wait actually I think yeah you can [6:06] see that we get some agent traces generated on the  fly right now and we got errors here right so this [6:15] is what I trying to say that the detector which is  mostly like IM Mr. Judge but on top of that it's [6:20] like a running cloud coding agent in a sandbox.  This will be triggered soon not right now but will [6:26] be SW and this is something after it finishes will  look look like. So for this uh agent trace you can [6:32] see there are four as a judge checks. Uh this is  basic you can click into it and see the the traces [6:39] just for the as judge. But the most interesting  part is the of course is the uh coding agent in [6:44] the cloud part right. So you click alert you can  see all the agent learnings are piped as the input [6:51] as a prompt in this our coding agent. So to be  specific we are running this uh open source agent [6:56] called pimono which is not exactly cloud code but  it's like open source version of that uh and then [7:01] give you a bunch of analysis as to like sometimes  give you a PR hix this time maybe it doesn't but [7:06] you have all the analysis showing here uh  the alert is follow up in here. Uh let's see. [7:14] Okay, it's not triggered right now. It may be wait  for a while, but is uh this is what it's supposed [7:18] to look like after it's triggered. So it gives you  tells you like which exact LMS judge is trigger [7:24] along with uh the uh the problematic uh traces and  also give you a summary. So this is the part where [7:31] I mentioned it was running in production  and then find a problem and then submit a [7:36] a hot fix. So all of this is automatic, right?  So no no engineers is involved in this step. [7:41] So let's say you got a problem your engineers  are not alerted like you you're all sleeping [7:45] at least something can be triggered and when  they wake up can take a look not just an alert [7:50] but also a hot fix. Next I want to show the CLI  that I uh keep bragging about. So here it is [7:56] with our CLI is called like download  this thing in APM. It has the auto remix [8:02] available here. Hopefully you guys can see that.  And then uh that's a basic stuff which is like [8:07] trees traces list showing the thing that we just  showing in the UI and also something like uh [8:13] uh traces get and then it shows the structure.  Of course you can also download everything in the [8:18] local file system. Uh but of course this is very  hard to hard to like track right hard to remember. [8:24] So literally the the best way to use this is  literally using your coding agent like cloud code [8:28] ask you it to use the CLI right you don't need to  use it you use the CLI to do whatever let's say [8:34] uh let me just ask a question uh use CLI to get  the most recent findings summarize a pattern uh [8:49] okay uh hope this uh get serviced soon so  the whole idea is that the uh coding agent [8:55] is able to use CLI several times uh as many  times as necessary to get all the important [9:00] information to help you do what you uh got  to do. Um I think this takes a while. So um [9:06] I'm just going to continue talking about  some of the uh other features. Of course, [9:10] we have the basic dashboard as well and the  setup for the detectors is provided in here. [9:16] Uh you guys can play with it uh as you want. uh  and the next step is like updating the offline [9:21] and data set for that part because of the  time is limited. Uh I wouldn't call that uh [9:26] so jumps to the last page. So everything you  just see all these like CLI like AI native uh [9:34] the ICK and instrumentation all this is open  source. So just scan this and then you can see my [9:40] open source. If you want to give me a star I would  really love that. uh anybody who is uh building co [9:46] AI agents let's say like pimono we recently  support that cloud codeex cloud educate uh cloud [9:52] manage agents everything you can think of uh if  you don't have it we can support in today's uh so [9:58] I think that should be about it let's just wait  for the CLI to finish I want to show you guys [10:03] the alert got just got triggered so this part is  good the CLI uh not here sorry give me a second [10:10] yeah you see there is a two finding in the last  six course which is uh actually triggered by me [10:15] and then it gave you analysis as to what's the  problem what is uh how fix what do you plan to [10:20] do next so you can see it works uh we I have  a little booth uh at booth uh I think K 816 [10:30] and uh from my GitHub you're able to  see all my information like uh Twitter [10:35] or like my landing page or you can join  the discord please reach out to me if you [10:39] think uh you're interested in a solution is all  self- served to play with it. But if you need any [10:44] type of support, uh I'm open for like a 50 minute,  30 minute call. Uh thank you for listening. Yes.