Human intuition and 'vibes' fail to scale for non-deterministic agents in production, requiring granular traces to understand opaque runtime decision trees and tool invocations. (0:12)
By equipping coding agents like Claude Code with observability CLI tools and skills, developers can compress an eight-step, days-long failure-to-fix cycle into an automated local workflow spanning minutes. (6:03)
Telemetry analysis of a demo financial agent uncovered a 44% failure rate driven by ungrounded numbers, redundant endpoint calls, and silent tool failures that prompted hallucinations. (16:03)
Resolving international stock query failures required both prompt adjustments and updating outdated tool dependencies, lifting test pass rates from 0% to approximately 90%. (19:19)
Detailed notes
## Background
Published on YouTube by AI Engineer on October 11, 2026, this talk features Ankur Duggal of Arize AI demonstrating how to systematically evaluate and repair agentic applications. The session showcases how to connect Arize's CLI and operational skills to a coding agent, enabling it to pull execution traces, diagnose failures like numeric ungroundedness and silent tool errors, and run automated regression experiments.
## TL;DR
Ankur Duggal addresses the industry-wide challenge of moving beyond shipping agents based on 'vibes' to building trusted, automated evaluation pipelines. Using a flawed financial agent prone to unwarranted recommendations and silent tool failures as a case study, he connects Claude Code directly to Arize runtime traces and evaluators via the Arize CLI. Duggal demonstrates how this setup allows coding assistants to diagnose root causes, generate test datasets, and iterate on fixes—such as resolving a multi-market equity query bug from a 0% baseline to a 90% pass rate. Throughout the presentation, he emphasizes cost-conscious evaluation sampling, avoiding superficial linting rabbit holes, and keeping a human engineer in the loop to review proposed architectural changes.
## Atmosphere & style
The presentation is conversational, pragmatic, and refreshingly candid, carried by Duggal's relaxed stage presence and playful humor about agents 'cooking' or 'discombobulating' in the terminal. Live audience polling and collaborative diagnosis keep the room actively engaged, creating the collaborative feel of an interactive technical workshop rather than a rigid corporate pitch. The atmosphere balances ambition with honesty, as Duggal openly shares the hiccups of live coding demonstrations, laughs at long-running terminal jobs, and demystifies the realities of agent evaluation.
## Bulletin
- [00:12](https://www.youtube.com/watch?v=_zUp6y9bDU8&t=12s) Human intuition and 'vibes' fail to scale for non-deterministic agents in production, requiring granular traces to understand opaque runtime decision trees and tool invocations.
- [06:03](https://www.youtube.com/watch?v=_zUp6y9bDU8&t=363s) By equipping coding agents like Claude Code with observability CLI tools and skills, developers can compress an eight-step, days-long failure-to-fix cycle into an automated local workflow spanning minutes.
- [16:03](https://www.youtube.com/watch?v=_zUp6y9bDU8&t=963s) Telemetry analysis of a demo financial agent uncovered a 44% failure rate driven by ungrounded numbers, redundant endpoint calls, and silent tool failures that prompted hallucinations.
- [19:19](https://www.youtube.com/watch?v=_zUp6y9bDU8&t=1159s) Resolving international stock query failures required both prompt adjustments and updating outdated tool dependencies, lifting test pass rates from 0% to approximately 90%.
## Key takeaways
- [00:12](https://www.youtube.com/watch?v=_zUp6y9bDU8&t=12s) Shipping agent applications purely on 'vibes' masks critical failure modes that only granular runtime evaluations can reveal in production.
- [02:02](https://www.youtube.com/watch?v=_zUp6y9bDU8&t=122s) Unlike deterministic code paths, an agent's runtime decisions—including internal reflection and tool selection—remain entirely opaque without execution traces.
- [03:19](https://www.youtube.com/watch?v=_zUp6y9bDU8&t=199s) Inspecting only inputs and final outputs can hide whether an agent executed the correct tool chain or merely hallucinated a plausible answer.
- [04:15](https://www.youtube.com/watch?v=_zUp6y9bDU8&t=255s) The traditional manual remediation cycle spans eight tedious steps—from sampling failing traces to modifying code and running regression tests—costing hours or days per bug.
- [06:03](https://www.youtube.com/watch?v=_zUp6y9bDU8&t=363s) Connecting coding assistants directly to execution traces, evaluation metrics, and human annotations enables autonomous root-cause diagnosis and code repair.
- [08:06](https://www.youtube.com/watch?v=_zUp6y9bDU8&t=486s) Unconstrained financial agents pose severe legal risks when they offer unqualified direct recommendations instead of soliciting user context and risk tolerance.
- [13:19](https://www.youtube.com/watch?v=_zUp6y9bDU8&t=799s) Arize skills interface directly with the underlying CLI to pull spans locally, build test datasets, and launch regression experiments from the coding workspace.
- [16:03](https://www.youtube.com/watch?v=_zUp6y9bDU8&t=963s) Silent tool failures—where endpoints return empty payloads with success status codes—often cause downstream agents to fabricate realistic-looking figures.
- [19:19](https://www.youtube.com/watch?v=_zUp6y9bDU8&t=1159s) Prompt adjustments alone are frequently insufficient to fix agent errors if underlying tool implementations or dependency versions are outdated.
- [22:23](https://www.youtube.com/watch?v=_zUp6y9bDU8&t=1343s) Teams can decouple application runtime from evaluation logic, running experiment code locally while executing evaluation judges on cloud platforms.
- [23:31](https://www.youtube.com/watch?v=_zUp6y9bDU8&t=1411s) Evaluation setups support any cloud or local open-source LLM, allowing teams to reserve high-tier frontier models strictly for benchmarking and quality judging.
- [24:37](https://www.youtube.com/watch?v=_zUp6y9bDU8&t=1477s) Piping runtime execution graphs and span-level timing into coding agents prevents them from blindly guessing root causes based solely on raw repository code.
- [26:13](https://www.youtube.com/watch?v=_zUp6y9bDU8&t=1573s) Continuous evaluations serve as regression monitors against model drift and user behavior changes, which static prompt engineering cannot catch alone.
- [28:35](https://www.youtube.com/watch?v=_zUp6y9bDU8&t=1715s) Grounding coding agents with explicit execution traces and operational skills prevents them from wasting context cycles on cosmetic linting instead of architectural fixes.
- [30:21](https://www.youtube.com/watch?v=_zUp6y9bDU8&t=1821s) Judge LLM costs can be controlled using built-in trace cost transparency, selective span filtering, and configurable sampling rates.
- [31:28](https://www.youtube.com/watch?v=_zUp6y9bDU8&t=1888s) Human engineering oversight remains essential at the end of automated remediation loops to verify that proposed agent modifications do not degrade system quality.
- [33:19](https://www.youtube.com/watch?v=_zUp6y9bDU8&t=1999s) Generic benchmark leaderboards are inadequate for selecting judge models; evaluations must be customized to application-specific domain and regulatory constraints.
## Detailed walkthrough
### [00:12](https://www.youtube.com/watch?v=_zUp6y9bDU8&t=12s) From Vibes to Trusted Evaluations
Ankur Duggal introduces his talk with a playful clarification: teaching developers how to make their agent “cook” has nothing to do with food, despite the phrasing. To ground the discussion, he polls the room on how teams currently validate and deploy their systems. He first asks how many developers ship systems purely on “vibes”—which he emphasizes is not inherently bad, but simply means checking inputs and outputs by hand and assuming the model behaves properly if the responses look reasonable. A sizable portion of the audience raises their hands. When he asks how many have formal evaluation setups—such as code-based assertions or LLM-as-a-judge pipelines—the number of hands drops slightly, though a respectable cohort remains. Finally, he asks how many developers with evals in place actually trust them enough to feel they have complete coverage and can ship fearlessly. Only a single hand remains in the air.
Duggal notes that this solitary hand matches reality across the industry: building trusted evaluations for non-deterministic systems is extraordinarily difficult because developers rarely know all the failure modes they need to test against. While checking vibes helps get a prototype off the ground, Duggal warns that human intuition does not scale once systems face real-world users. Evaluating production software requires visibility into why and where a failure occurred rather than relying on gut feelings.
As Ankur Duggal points out: “Realistically, you can't really ship based on this feeling of, 'Yeah, it's correct.' Because when you're in production, you're not going to know where things are broken, what's going wrong, unless you have actual eval set up to find out where things are breaking and how you can actually directly fix it.”
[02:02](https://www.youtube.com/watch?v=_zUp6y9bDU8&t=122s) Traces Reveal an Agent's Actual Decisions
To understand how to evaluate an agent, Duggal steps back to the underlying telemetry: traces. In traditional software engineering, Application Performance Monitoring (APM) traces follow a deterministic call path. In standard code, an engineer can step through a method like `handleSubmit`, verify whether an input passed validation, and see where an exception was thrown. In contrast, an agentic application often consists of a single line of execution—`agent.run(query)`—behind which the model autonomously decides its own trajectory. Whether it invokes a tool, reflects internally, retrieves context, or branches into multi-step reasoning is entirely opaque until the runtime trace is captured.
```mermaid
flowchart TD
accTitle: Traditional Deterministic Logic vs Agentic Decision Paths
accDescr: Comparison showing how traditional procedural code follows a fixed path while an agent follows an emergent path through tools and models.
subgraph TRAD["Traditional Execution Path"]
A["handleSubmit(input)"] --> B{"isValid?"}
B -->|"No"| C["Raise ValidationError"]
B -->|"Yes"| D["Process Order"]
end
subgraph AGENT["Agentic Execution Path"]
E["agent.run(query)"] --> F{"LLM Reasoning"}
F --> G["Retrieve Context / Docs"]
F --> H["Select & Execute Tool"]
F --> I["Internal Reflection"]
H --> J{"Next Step?"}
J -->|"Iterate"| F
J -->|"Complete"| K["Generate Final Response"]
end
```
Duggal illustrates this opacity with an e-commerce customer support example. A customer submits the query, “I was supposed to get a refund here. Where's my refund?” The agent's trace reveals that the model triggered a tool named `tool.get_order_status`. The tool dutifully reported that a refund was issued, and the agent echoed that back to the user. However, that tool was fundamentally the wrong one to call: the user already knew the refund had been initiated and wanted to know why the money had not landed in their bank account. Duggal emphasizes that observing only the input and the final output masks whether the agent executed the correct tool chain or simply hallucinated a plausible answer.
As Ankur Duggal explains: “We don't know until we actually see the traces. You won't know what your agent's doing. So that trace kind of helps you define that.”
[04:15](https://www.youtube.com/watch?v=_zUp6y9bDU8&t=255s) The Traditional, Tedious Agent Improvement Loop
When developers attempt to fix an agent failure manually, they usually endure a lengthy, labor-intensive feedback loop. After identifying an error in a production trace—such as the mismatched refund tool—engineers must extract a representative dataset of similar failure cases to benchmark future iterations. From there, they have to diagnose the exact root cause: did the LLM prompt lack clear instructions, did the tool description mislead the planner, were guardrails missing, or was the tool itself buggy?
```mermaid
flowchart LR
accTitle: The Traditional Agent Improvement Loop
accDescr: An eight-step sequential cycle starting from identifying production failures to evaluating and deploying fixes.
subgraph DIAGNOSE["Diagnosis & Curation"]
S1["1. Identify Failure"] --> S2["2. Sample Dataset"]
S2 --> S3["3. Root Cause Analysis"]
end
subgraph SHIP["Deployment"]
S6 --> S7["7. Ship to Prod"]
S7 --> S8["8. Monitor & Repeat"]
end
```
Once a potential remedy is conceived, developers must craft custom evaluators to ensure the behavior is constrained. For instance, if a guardrail is introduced to stop sensitive data leakage, an evaluator must verify that the guardrail consistently blocks forbidden outputs. Only then can developers rewrite prompts, adjust code, execute regression experiments against the curated dataset, and push updates back to production. Duggal notes that executing this entire eight-step cycle by hand typically consumes hours or days for every single bug.
As Ankur Duggal reflects on the standard workflow: “Now, this is your standard go-to process for actually improving your agent. It's pretty long, pretty manual, and pretty tedious.”
[06:03](https://www.youtube.com/watch?v=_zUp6y9bDU8&t=363s) Unifying Traces, Evaluations, and Code in a Coding Agent
To collapse an eight-step loop that takes days down into a task that takes minutes, Duggal advocates feeding all operational context directly into an AI coding assistant, such as Claude Code, Cursor, or Codex. Rather than having human engineers manually bridge the gap between production logs and local source code, the coding agent receives every piece of data needed to diagnose and repair the application autonomously.
This unified workspace aggregates several disparate data streams:
- **Arize traces:** The granular step-by-step telemetry of model calls, retrieved documents, and tool invocations.
- **Evaluation results:** Automated judge outputs and scoring metrics pinpointing exactly where assertions or guardrails failed.
- **Human annotations:** Qualitative feedback and domain guidance specifying what a correct response should have looked like.
- **APM traces:** System-level telemetry verifying whether services, endpoints, or databases were healthy.
- **Local codebase:** The underlying application logic, prompts, and tool implementations that need modification.
By installing a command-line interface (CLI) and operational skills, developers give the coding agent native capabilities to instrument code, pull down failing traces, assemble evaluation datasets, run local regression experiments, and verify proposed code fixes before shipping them.
As Ankur Duggal summarizes the objective: “We want to make sure that we're not spending hours, days finding the issue, finding the trace, finding all these different issue points, coming up with the potential solution, going through that full eight-step process manually. We want to pull all of that together and bring it into minutes.”
[08:06](https://www.youtube.com/watch?v=_zUp6y9bDU8&t=486s) Diagnosing a Flawed Financial Agent
To demonstrate this setup live, Duggal introduces a flawed financial agent and asks the audience to participate in diagnosing it. In the sample run, the user asks the agent: “Should I invest in this company?”, targeting Tata Motors. The financial agent immediately responds with an unqualified, affirmative answer: “Yes, you should.” Duggal points out the immediate regulatory hazard: neither he nor the software agent is a registered financial advisor, and offering direct buy/sell recommendations creates severe legal liabilities.
Duggal asks the audience to identify what went wrong and how the agent should have handled the prompt. One attendee suggests that instead of making a snap recommendation, the agent should have asked clarifying follow-up questions to collect more context about the user's portfolio and risk tolerance before delivering an answer. Duggal agrees, emphasizing that a robust agent must recognize missing context and solicit details rather than fabricating an unsubstantiated recommendation.
As Ankur Duggal remarks on the agent's reckless recommendation: “Ultimately that's not really what we want to do because I'm not registered to give any financial advice, nor is this agent, and I don't want to get in legal trouble.”
[09:17](https://www.youtube.com/watch?v=_zUp6y9bDU8&t=557s) The Flawed Financial Agent: Validation and Context Gaps
Ankur Duggal addresses audience suggestions and critiques regarding an automated financial agent that gives questionable stock recommendations. An audience member points out the absence of any validation mechanism, prompting Duggal to agree that the system currently lacks any ground truth against which answers can be benchmarked. Another attendee asks whether the agent possesses any context regarding the user's personal financial background, to which Duggal clarifies that it has none whatsoever. Without knowing the user's risk tolerance, financial position, or investment goals, the agent gives unsolicited, unqualified recommendations.
Duggal emphasizes that identifying the root problem of such complex systems is not straightforward because real-world agent pipelines can fail in dozens of interconnected ways. Rather than manually guessing how to patch prompts or guardrails, Duggal sets out to leverage a coding agent equipped with Arize CLI tools and skills to diagnose and repair the system systematically. As Duggal admits to the room: “And I genuinely don't even know what the right answer is to this because this agent is still messed up right now.” The goal of the demo is to let a coding agent pull down execution traces, inspect existing evaluations and annotations, analyze failure patterns, and suggest targeted remedies—such as forcing the agent to ask clarifying questions or installing guardrails against unwarranted financial advice.
---
[11:05](https://www.youtube.com/watch?v=_zUp6y9bDU8&t=665s) Setting Up the Arize CLI, Profiles, and Coding Agent Skills
Duggal transitions to his development environment to connect his coding assistant—Claude Code—to the telemetry stored in Arize. The setup workflow requires only a few steps: installing the Arize CLI with a single command, and then establishing a configuration profile linked to the user's Arize workspace. This profile allows local tools to query traces, spans, evaluations, and human annotations directly from the observability backend.
Once the CLI is configured, Duggal installs the Arize skills into Claude Code via its extension marketplace. To streamline the live demonstration, Duggal executes Claude with the `--dangerously-skip-permissions` flag so the agent can execute commands without pausing for manual approval at every step, explicitly cautioning the audience: “And again, just as a warning, I am using dangerously skip permissions. Please don't do this. This is not actually good practice. This is just for a demo, so I don't have to keep pressing yes.” He then instructs Claude Code to invoke the Arize skills, pull down recent traces for the failing financial agent, inspect existing evaluations and annotations, conduct an autonomous root-cause analysis, and recommend custom evaluators to prevent future regressions.
---
[13:19](https://www.youtube.com/watch?v=_zUp6y9bDU8&t=799s) Inspecting Supervisor and Specialist Traces
While Claude Code begins processing the telemetry in the background, Duggal switches over to the live application to trigger an example query: asking the agent whether an investor should buy shares of Nvidia. A previously captured run reveals the complex multi-agent hierarchy operating behind what seems like a simple chat prompt.
```mermaid
flowchart TD
accTitle: Multi-Agent Financial Query Execution Flow
accDescr: A user query passes to a supervisor agent, then to specialist agents, tool calls, and model calls to generate an output.
subgraph Orchestration ["Orchestration Layer"]
UQ["User Query: 'Invest in Nvidia?'"] --> SUP["Supervisor Agent"]
end
subgraph Execution ["Specialist Layer"]
SUP --> FDA["Financial Data Agent"]
FDA --> TC["Tool Calls (Financial APIs)"]
FDA --> LLM["Model Calls (LLM)"]
end
The trace captures the top-level supervisor routing requests to a specialized financial data agent, which in turn invokes LLM inference calls and external tool executions. As the coding agent continues working, an audience member asks how Claude Code interacts with the Arize platform. Duggal explains that the skills interface directly with the Arize CLI underneath:
- The CLI downloads raw trace and span data locally into the agent's working workspace.
- The coding agent can execute CLI commands to fetch curated datasets, trigger API queries, or interact programmatically with the web UI.
- The agent can establish ongoing platform evaluations, build test datasets, and launch experiments just as an engineer would through the graphical dashboard.
Addressing the audience's curiosity about local execution versus cloud sync, Duggal notes: “When you give it access to these skills, what you're actually giving it access to is the CLI underneath.”
---
[16:03](https://www.youtube.com/watch?v=_zUp6y9bDU8&t=963s) Uncovering Grounding Failures, Fabrications, and Silent Errors
Claude Code completes its telemetry analysis and surfaces alarming metrics about the financial agent's production performance. Most critically, roughly 44% of the agent's interactions result in an evaluation failure for poor or misleading summaries. When investigating why, the coding agent pinpoints widespread numeric ungroundedness: numbers cited in the final investment recommendation bear no relation to the data retrieved by the financial tools.
```mermaid
flowchart LR
accTitle: Failure Pattern Breakdown
accDescr: Claude Code's diagnosis of agent failures leading to hallucinated outputs.
A["Failed Tool Call"] -->|"Tool hides error"| B["Silent Success Status"]
B -->|"No real data returned"| C["Agent Fabricates Numbers"]
C -->|"Ungrounded metrics"| D["44% Bad Summaries"]
E["Redundant Tool Loops"] -->|"Wasted latency and cost"| D
```
Beyond bad summarization, Claude Code flags three compounding failure modes:
- **High Hallucination and Fabrication Rates:** The agent invents answers outright when it cannot locate factual figures.
- **Redundant Tool Invocations:** The agent repeatedly calls the same financial search endpoints with duplicate parameters, burning unnecessary latency, API quotas, and money.
- **Silent Tool Failures:** Underlying tool calls fail or return empty payloads, but return a success status code. Instead of disclosing that data retrieval failed, the downstream agent fabricates realistic-looking figures to complete its response.
Duggal reacts to the 44% failure metric: “That's really bad. I did not know that. So, we definitely need to fix that one.” To solve these defects systematically, Duggal commands Claude Code to implement two concrete fixes: build a dedicated evaluator to catch tool silent failures, and resolve the numeric groundedness bug so all outputs tie back to verified source spans. Finally, he instructs Claude Code to run an automated regression experiment against the exact failing spans using three specialized tools: the Arize dataset skill, the Arize evaluator skill, and the Arize experiment skill.
[19:19](https://www.youtube.com/watch?v=_zUp6y9bDU8&t=1159s) Triggering the Repair Loop and Iterating on International Stock Queries
Ankur Duggal instructs Claude Code to implement the recommended fixes and evaluate them against the baseline failures. While the coding agent takes time to run locally, Duggal prefers not to leave the audience watching an idle terminal: “while we want to let it cook I don't want to have you guys staring at a screen where Claude's just discombobulating or whatever it does in its free time.” He navigates to a completed run addressing a recurring problem with non-US equities. Users had submitted queries comparing stocks listed in foreign markets like South Korea, Japan, or China, but the application repeatedly failed because it lacked proper multi-market handling.
To resolve this, Claude pulled the failing South Korean queries from production traces, synthesized a test dataset, and ran iterative experiment cycles. The baseline run produced a 0% pass rate, mirroring production. In its first remediation attempt, Claude normalized user input by prepending country qualifiers, but the pass rate remained flat because the downstream financial data tool was obsolete and required a version bump to the stable release. After identifying the outdated dependency and upgrading the tool call, Claude's third iteration achieved an approximately 90% success rate. Duggal points out on the Arize chart that while 90% is not yet perfection, moving from a complete failure state to nine out of ten passing queries represents a substantial, automated jump from defect discovery to code remediation.
```mermaid
flowchart TD
accTitle: Iterative Fix Workflow for Non-US Stock Queries
accDescr: Workflow showing Claude extracting failing production traces, creating an evaluation dataset, and cycling through three iterations to fix tool and prompt failures.
subgraph TraceAnalysis["1. Trace Extraction & Dataset"]
A["Failing Production Traces<br/>(South Korea Stock Queries)"] --> B["Curated Test Dataset"]
end
subgraph ExperimentLoop["2. Iterative Experimentation"]
B --> C["Iteration 0: Baseline Run<br/>(Buggy Code: 0% Pass)"]
C --> D["Iteration 1: Add Country Context<br/>(Tool Fails: 0% Pass)"]
D --> E["Iteration 2: Upgrade Tool Dependency<br/>(Stable Version Update)"]
E --> F["Iteration 3: Re-evaluate<br/>(~90% Pass Rate)"]
end
```
[22:23](https://www.youtube.com/watch?v=_zUp6y9bDU8&t=1343s) Where Experiments and Evaluations Run
Answering an audience question regarding execution environments, Duggal clarifies the separation between running the application code and running the evaluation logic. In his demonstration, Claude Code executed the experiment locally on Duggal's development machine—which he notes will be standard practice for most software engineers—though users can choose to host experiment runs on remote cloud infrastructure or staging clusters.
The evaluation stage itself is processed within the Arize platform. Duggal inspects the evaluator configuration, revealing that Claude initially attempted to implement two distinct evaluators: a code-based deterministic check and an LLM-as-a-judge evaluator. Because Claude encountered setup errors when establishing the code-based evaluator data schema, it relied on the LLM judge. The generated evaluator prompt inspects whether the financial specialist agent retrieved proper underlying context, verified tool execution, and properly satisfied the user's cross-market query.
[23:31](https://www.youtube.com/watch?v=_zUp6y9bDU8&t=1411s) Choosing Evaluation Models Across Providers
A participant asks about model flexibility for evaluations. Duggal emphasizes that Arize does not bind users to Anthropic or OpenAI: “No one wants to be limited to just Claude or just OpenAI, especially in today's modern age where you can see AI models literally ship every other day and then you want to be like, ‘Okay, I want to use that one now.’”
Engineers make distinct trade-offs between cost and accuracy when configuring judges. Some teams opt for lower-cost models to monitor high-volume interactions economically, while others allocate their budget toward premier frontier models specifically for evaluation: “they know their agent's only going to be as good as their evals, so they use the highest-end models for their evals even if they're not necessarily using those in production.” The configuration interface supports any supported cloud model provider as well as locally served open-source models.
- Model flexibility prevents vendor lock-in as frontier model releases shift market performance week to week.
- High-end models can be reserved exclusively for evaluation tasks to establish a ground truth without incurring production-tier latency or per-call costs on user traffic.
- Local open-source model endpoints can be connected directly to the Arize evaluation harness alongside commercial APIs.
[24:37](https://www.youtube.com/watch?v=_zUp6y9bDU8&t=1477s) Enriching Coding Agents with Traces and Phoenix Parity
When asked how using Arize's skills differs from spinning up a generic coding sub-agent to fix a bug, Duggal argues that an agent operating strictly on source code lacks critical context. Standard software agents inspect repository files and infer intent from error messages, whereas Arize's skills pipe runtime execution paths directly into the model's context window. The agent receives execution graphs, span-level timing, LLM judge scores, user annotations, and external tracking data from Jira or APM logs. Rather than guessing why an output looks incorrect, the agent examines the specific agentic handoffs and faulty tool parameters that occurred during production execution.
Duggal also confirms that core capabilities are accessible in Phoenix, Arize's open-source observability framework:
- Phoenix supports the Model Context Protocol (MCP) and skills, allowing developers to connect coding agents to local open-source trace repositories.
- The commercial enterprise edition layer adds specialized online evaluation configurations, interactive dashboarding, and collaborative dataset management.
[26:13](https://www.youtube.com/watch?v=_zUp6y9bDU8&t=1573s) Defining Summary Quality and Monitoring Drift
An audience member asks how the platform determines what constitutes a "bad summary" versus a "good summary," and why developers cannot simply add a summary-generation skill upfront instead of configuring evaluation pipelines. Duggal pulls up the summary evaluator prompt, showing that it evaluates whether the agent integrated context from intermediate steps, referenced verified factual data points, and directly answered the prompt without omissions.
Addressing why skills alone cannot guarantee persistent output quality, Duggal explains that static agent improvements inevitably face degradation over time: “You want to make sure that you update your prompts, update your skills, update your coding workflows so that your agents will give the best summaries. The evals are only there to help you determine where there are issues, monitor those issues, make sure they don't repeat, right?” Even if an updated prompt or skill works reliably for two months, subsequent changes in user phrasing or silent downstream model updates introduce regressions that static code cannot detect on its own. Continuous evaluations serve as regression monitors to prevent performance from decaying unnoticed.
[28:35](https://www.youtube.com/watch?v=_zUp6y9bDU8&t=1715s) Preventing Coding Agents from Getting Trapped in Linting
A member of the audience raises a common pitfall when unleashing coding agents like Claude Code on a repository: the agent often ignores the actual runtime architectural defect and instead expends its turns fixing style warnings, formatting, and linting errors.
Duggal agrees that agents default to superficial codebase cleanups unless they are explicitly grounded with observability data. Arize's skills counter this behavior by providing concrete runtime failure traces that pinpoint exactly where a supervisor misrouted a task or invoked an invalid tool signature: “that's kind of why we built out these skills which kind of focus on ensuring that they hyperfixate on the actual issues that have been given to them.” When an agent is handed an execution graph showing an outdated API call or a tool failure, it focuses on the operational logic and dependency resolution instead of wasting context cycles on cosmetic linting adjustments.
[29:52](https://www.youtube.com/watch?v=_zUp6y9bDU8&t=1792s) Avoid Focusing Only on Linting
Wrapping up a previous audience question about whether a coding assistant might waste its time tweaking formatting or lint rules instead of fixing substantive logic, Ankur Duggal explains that the underlying models possess enough reasoning capability to recognize the real objective. In his experience, pairing the model with targeted skills keeps it oriented toward genuine defects rather than superficial styling: “Genuinely I haven't seen that so much when using the skills.” The model understands what failure it is actually trying to resolve and does not easily get derailed by trivial linters when given precise evaluation context.
[30:21](https://www.youtube.com/watch?v=_zUp6y9bDU8&t=1821s) Evaluation Costs and Sampling
An audience member asks whether the system makes it easy to understand and track how much it costs to run evaluations. Duggal emphasizes that observability into evaluation overhead is built right into the platform interface. Because running evaluators involves sending requests to judge language models, those calls inevitably add up: “Anytime you're running another LLM, it's going to be additional cost to you.”
Within the platform, users receive a full trace history of every evaluator run, complete with an explicit cost breakdown displayed on-screen. To keep those expenses predictable and avoid unnecessary spend in production, users can enforce guardrails:
- Configure basic sampling requirements so evaluators do not execute across 100% of production traces or calls.
- Restrict evaluations to specific high-priority areas, workflows, or suspect spans.
- Dictate sampling and scoping criteria directly through the UI or programmatically via the coding agent.
```mermaid
flowchart TD
accTitle: Evaluation Cost Management Flow
accDescr: Shows how production traces are filtered through sampling controls to reduce judge LLM costs while maintaining full observability.
subgraph Observability ["Traces & Monitoring"]
A["Production Traces"] --> B{"Sampling & Scope Rules"}
B -->|"Sampled Subset"| C["Judge LLM Evaluators"]
B -->|"Unsampled"| D["Stored Raw Traces"]
end
subgraph CostTracking ["Cost Transparency"]
C --> E["Full Trace History"]
E --> F["Per-Eval Cost Breakdown in UI"]
end
```
[31:28](https://www.youtube.com/watch?v=_zUp6y9bDU8&t=1888s) Review the Proposed Changes
Duggal recaps the core workflow demonstrated throughout the session. While autonomous agents are remarkably capable partners, developers must maintain realistic expectations about their boundaries: “These AI agents, they're great. They will do basically everything you ask them to, but again, they're not magic.” If left entirely unchecked, an agent might gravitate toward trivial modifications like linting instead of architectural remedies.
The real power lies in closing the entire improvement loop end-to-end: observing where live failures occur, pulling down traces, generating representative datasets, drafting code fixes, and immediately running evaluations to confirm that the changes solve the issue without causing regressions. Even with automation running the loop, Duggal underlines the indispensable role of the engineer at the end of the pipeline: “Finally, you can then go ahead and be the manual human to make sure that hey, it didn't just go back and fix the linting changes, right? And we can make sure that it actually made changes that actually help improve the agent and not go for degradation.” Checking back on the live terminal, Duggal notes with a laugh that the background repair task—which previously took around 14 minutes—is still churning away as it executes the full experiment suite.
[33:19](https://www.youtube.com/watch?v=_zUp6y9bDU8&t=1999s) Tailor Evaluations to Your Application
An audience member asks whether developers must rely solely on Claude or another coding agent to decide whether an application is behaving properly. Duggal clarifies that teams can blend automated discovery with curated domain rules. In his earlier demonstration, the clean evaluation table shown in the terminal stemmed from generic health checks and baseline criteria his team had authored beforehand—such as checking whether the agent is healthy, validating tool call correctness, or flagging poor-quality summaries. What Claude excels at, however, is combing through dense production traces that a human engineer cannot inspect by hand: “I'm not going to be able to go through all of those manually and find those, but Claude can or Claude, Cursor, Codex, whatever, right? And that's where like one of the biggest superpowers that you're able to give these coding agents to be able to do that for you.”
A follow-up question asks if Arize provides a benchmark dataset or visualization comparing which judge models offer higher quality or better cost-efficiency for evaluations. Duggal chuckles and admits there is no one-size-fits-all leaderboard: “Realistically no, that would be sort of something you'd have to do for your own analysis you'd have to figure it out because ultimately one of the things that I like to say is if you're going to use out of the box eval to judge your agent you're going to get out of the box results and that's not what you want when you're building custom-tailored AI agents which are built to be personal for you.”
He points out that public benchmarks simply tell you if a model is broadly competent, not whether it suits your domain constraints:
- General benchmark rankings fail to answer the critical question: “realistically, what you want is, is it a good LLM for me?”
- Financial applications require models tuned to quantitative accuracy and strict reporting formats.
- Specialized subdomains—such as financial law or regulatory compliance—may require an entirely different judge model than general finance.
Duggal wraps up his presentation, thanks the audience, and receives applause.
## Notable quotes
> [33:19](https://www.youtube.com/watch?v=_zUp6y9bDU8&t=1999s) if you're going to use out of the box eval to judge your agent you're going to get out of the box results and that's not what you want when you're building custom-tailored AI agents which are built to be personal for you. — Ankur Duggal
## Video description
A financial agent invents answers, repeats tool calls and produces summaries that its own evaluations flag as wrong. Ankur Duggal connects those failures to a coding agent through Arize's CLI and skills, then asks it to investigate traces, change the application and run experiments. The demo begins with an investment question that looks reasonable at the surface but lacks enough context to support its recommendation. Traces reveal the supervisor, specialist agents, model calls and tools underneath that answer. They turn a vague complaint about quality into evidence about the exact steps that failed.
Duggal sets up the CLI profile and skills, then shows the coding agent pulling spans and evaluation results into its local workspace. The investigation uncovers numeric grounding problems, fabricated responses, redundant searches and failures that the agent silently works around. An earlier experiment targets stock queries outside the United States: changing the country context alone is insufficient until the underlying tool is also updated. Dataset, evaluator and experiment skills make those proposed fixes testable rather than a collection of unverified edits. Questions cover local experiment execution, judge models, evaluation costs, sampling and the risk that a coding agent spends its effort on linting instead of the actual defect. The live repair is still running at the close. Duggal recommends human review of proposed changes and evaluations tailored to the application's failures, so later regressions can be detected as traffic and models change.
Timestamps:
0:00 - From vibes to trusted evaluations
1:58 - Traces reveal an agent's actual decisions
3:19 - A refund question and the wrong tool
4:15 - The improvement loop
6:03 - Bring traces, evaluations, and code together
7:27 - CLI and skills as the interface
8:06 - A flawed financial agent
11:05 - Set up a profile and install skills
13:19 - Inspect supervisor and specialist traces
16:03 - Grounding, fabrication, and silent failures
18:24 - Build evaluations and run experiments
20:05 - An earlier fix for international stock queries
22:23 - Where experiments and judges run
23:31 - Choose an evaluation model
24:37 - More context than the code alone
26:13 - What makes a summary good?
28:35 - Avoid focusing only on linting
30:21 - Evaluation costs and sampling
31:28 - Review the proposed changes
33:19 - Tailor evaluations to your application
---
Watch along with the corrected transcript, summary, timestamps and resources on this talk's official AIE page:
https://ai.engineer/talks/_zUp6y9bDU8-youtube-4370c85de7d8cca2895b
Join our upcoming AI Engineer conferences (see https://ai.engineer landing page):
- AIEi Shanghai, November 5–6 — the first for meeting top Chinese labs and engineers
- AIE CODE in SF, November 10–12 — the top AI Coding conference returns
- AIEi Sydney, December 7–8 — the first AIEi run alongside NeurIPS 2026 in Sydney
Ankur Duggal of Arize AI argues that relying on intuition or manual spot-checks fails in production because non-deterministic agent workflows hide complex failure modes behind opaque runtime decisions. To overcome the slow, multi-day feedback loops of manual debugging, Duggal demonstrates connecting execution traces and evaluation pipelines directly to coding agents to diagnose, patch, and test agentic systems automatically.
The Opacity of Dynamic Agent Call Paths (2:02)
In traditional software, execution flows along predictable, deterministic paths where engineers can step through methods and pinpoint where exceptions arise. Agentic systems, however, often encapsulate execution within an unconstrained command like agent.run(), leaving model reasoning, context retrieval, and tool orchestration hidden until runtime. Duggal illustrates this with a customer query about an unreceived refund: the agent invoked an order status tool that confirmed a refund was initiated, masking the fact that it called the wrong tool to resolve why the money had not arrived. Because inspecting only inputs and final outputs can conceal hallucinations or flawed tool choices, production deployment requires granular traces to inspect why decisions occurred.
Automating the Multi-Step Agent Improvement Cycle (6:03)
When agents fail in production, developers typically navigate an eight-step remediation cycle: identifying failures, sampling representative datasets, diagnosing root causes, writing evaluators, modifying prompts or code, running regression experiments, deploying fixes, and continuous monitoring. Duggal notes this manual routine often consumes hours or days for individual defects. By equipping coding assistants—such as Claude Code or Cursor—with command-line interface skills connected to an observability platform, developers can feed traces, judge evaluations, human annotations, and repository files directly into the coding agent. This integration allows the assistant to pull failing production spans, assemble test datasets, draft code remedies, and execute local regression tests in minutes.
Autonomous Diagnostics and Iterative Defect Remediation (16:03)
During a live diagnostic demonstration of a flawed financial agent, the coding assistant analyzed production telemetry and identified that 44% of runs failed evaluation due to poor or misleading summaries. Claude Code traced this failure rate to ungrounded numerical figures, redundant tool queries, and silent tool failures where endpoints returned success status codes despite failing to deliver data. Rather than guessing at prompt patches, the coding agent applied concrete evaluators and dependency updates. In another case involving international stock queries that had a 0% baseline pass rate, Claude Code cycled through three automated iterations—normalizing inputs and upgrading an outdated financial data tool dependency—to elevate the success rate to roughly 90%.
Balancing Evaluation Costs and Model Selection (30:21)
Duggal explains that generic benchmark leaderboards do not reliably indicate whether a judge language model fits an application's domain, such as financial compliance. Engineering teams must evaluate custom criteria and decide whether to run economical models or dedicate premier frontier models strictly to evaluation pipelines to establish ground truth. Because running LLM-as-a-judge pipelines incurs cumulative API expenses, platforms must provide per-evaluation cost transparency and sampling controls, enabling teams to evaluate targeted spans or high-priority workflows rather than paying to inspect 100% of production traffic.
While coding assistants equipped with observability skills can independently run evaluations and update codebases, Duggal underscores that human engineers must conduct final reviews to verify architectural integrity and ensure models do not divert attention to superficial fixes like code linting.
Source quotes
[33:19] Ankur Duggal: if you're going to use out of the box eval to judge your agent you're going to get out of the box results and that's not what you want when you're building custom-tailored AI agents which are built to be personal for you.
Original transcript
[0:12] Good afternoon everyone. My name is
[0:14] Encore. Uh I'm going to be teaching you
[0:16] how to make your agent cook. Again, not
[0:18] food related. I don't know. Uh but yeah,
[0:21] we're going to get into it. Um so what I
[0:24] mean by that is actually before we get
[0:25] into that, let's go ahead and start with
[0:26] a quick question and a poll with
[0:28] everyone. Um, I'm sure a lot of you are
[0:30] building agents or have even shipped
[0:32] agents. I'm just curious how many of you
[0:34] currently are shipping based off vibes.
[0:36] And when I say vibes, it's not a bad
[0:38] thing. I mean like, hey, you're checking
[0:40] your answers. You're looking at it.
[0:41] You're making sure that it's potentially
[0:43] doing the right thing just based on the
[0:44] input and output. You're making sure
[0:45] that it's correct or not. How many of
[0:47] you all are doing that?
[0:50] Okay, pretty good reasonable amount of
[0:52] you. And then how many of you guys
[0:54] actually have like set eval set up like
[0:56] you know like oh you have LM as a judge
[0:58] or codebased evals or some sort of eval
[1:00] setup.
[1:03] Okay, a little less. Not too much less
[1:05] though. That's pretty good actually. And
[1:07] then how many of you that actually have
[1:08] your eval setup trust it enough that you
[1:11] know you're good enough? You have a full
[1:12] coverage of your evals and you're ready
[1:14] to you're continuously ready to keep
[1:15] going and moving forward.
[1:18] Wow, that's that's pretty impressive.
[1:20] That was a solid hand up,
[1:22] which is kind of what I expected. I
[1:23] mean, it's very hard to really trust
[1:25] your emails because they're hard to
[1:27] build. You don't really know what you're
[1:29] looking for. It's not a deterministic
[1:31] use case. And it's very hard to
[1:32] determine what we need to actually
[1:34] evaluate to make sure our agents are
[1:35] working properly. And that's kind of one
[1:37] of the biggest issues is a lot of people
[1:38] are going to start off in a very good
[1:40] spot, which is based off vibes. It gets
[1:42] you going. It makes sure that your
[1:44] agent's doing what you expect it to. But
[1:46] realistically, you can't really ship
[1:47] based on this feeling of, "Yeah, it's
[1:49] correct." Because when you're in
[1:50] production, you're not going to know
[1:52] where things are broken, what's going
[1:54] wrong, unless you have actual eval set
[1:56] up to find out where things are breaking
[1:58] and how you can actually directly fix
[2:00] it.
[2:02] So before we get into eval, what I want
[2:06] to kind of take is a whole step back and
[2:08] talk about how you can like what your
[2:11] eval would be on top of. And that would
[2:13] be on top of traces. And why we need
[2:16] traces is if you've ever used
[2:18] traditional software engineering, you
[2:20] have something that you do call traces.
[2:23] It's for APM monitoring. It's making
[2:24] sure your systems are reliable and are
[2:26] working. But realistically, what you you
[2:28] can actually go ahead and debug those
[2:30] and it's very easy to do because you
[2:32] have a very deterministic flow, right?
[2:34] You can even see right here, this is
[2:36] very easy to follow and this is more or
[2:38] less what like code looks like. You have
[2:39] a handle submit method. You check if
[2:42] it's valid. If it doesn't, it errors
[2:44] out. If it is, then it goes down the
[2:45] next list and continues down that
[2:46] process. So, it's very easy to figure
[2:48] out where things are b are breaking
[2:50] based on a just based on that sort of uh
[2:53] path history. But in an agent,
[2:55] unfortunately, with your code, really
[2:58] all you have is that one line of code,
[2:59] which is just going to be agent.run and
[3:01] then the query. And ultimately, what
[3:03] it's going to do is kind of up to it.
[3:05] Will it call a tool? Will it go ahead
[3:06] and do some analysis? Will it do some
[3:08] internal thinking? What is its next best
[3:11] step to do? and its actual best course
[3:13] of action. We don't know until we
[3:15] actually see the traces. You won't know
[3:17] what your agent's doing. So that trace
[3:19] kind of helps you define that.
[3:22] And so as a kind of a decent example as
[3:24] to what a trace would look like would be
[3:26] this. You have your agent call. You go
[3:30] ahead and see the LM calls. You have
[3:33] your documents you're retrieving, your
[3:35] tools that you're retrieving. You
[3:36] basically have every step along your
[3:38] journey that you would see of what your
[3:40] agent's doing.
[3:42] And here you can even go ahead and see,
[3:45] hey, we have for example this tool.get
[3:48] order status. It's the wrong tool
[3:49] occult. The user asked us, hey, I was
[3:53] supposed to get a refund here. Where's
[3:54] my refund? If we respond back, hey, your
[3:57] refund was issued and they know that
[3:58] that was their status based on their
[4:00] order status. That's not what they were
[4:01] looking for. They were looking to
[4:02] understand why hasn't their refund
[4:04] arrived? So we need to potentially call
[4:06] different tools, do different actions,
[4:08] and there's so many different varieties
[4:10] even just knowing that there's an issue
[4:12] here. But we'll dive into that in a
[4:13] little bit.
[4:15] So when you actually go ahead and
[4:17] understand, okay, we have a trace. Now I
[4:20] want to go ahead and debug what's going
[4:22] wrong with my agent, we're going to
[4:24] start with where first you're going to
[4:26] identify those issues, right? Those
[4:28] production failures like we just did in
[4:29] that last slide. We found out that tool
[4:30] was wrong. But what's your next step?
[4:33] You want to go ahead and find out, okay,
[4:35] not only where did it go wrong, can I
[4:37] get a nice sample size of that sort of
[4:39] information that I can then use for
[4:41] experimentation and iteration to improve
[4:43] the agent when I actually want to go
[4:45] back and test against those cases. Then
[4:47] what actually was the issue, right?
[4:50] Sure, the tool call was wrong, but
[4:52] should the LM prompt change? Should we
[4:54] add in guardrails? Should we change up
[4:56] the tool? Was the tool just doing the
[4:58] wrong things? Right? There's so many
[4:59] different areas that we need to test and
[5:01] go against. And so we can go and test
[5:03] against those build out evaluations to
[5:06] continuously monitor those use cases. So
[5:09] for example, if you find out, hey, we
[5:11] need to make sure that we set up a
[5:12] guardrail here. How are we going to do
[5:15] that? We want to and then we want to
[5:17] make sure that guardrail is doing its
[5:18] job. So we set up a evaluator on top of
[5:20] that guardrail making sure is that
[5:22] guardrail doing its job by prohibiting
[5:25] certain information from leaking.
[5:27] Finally, you actually make those
[5:29] changes.
[5:30] run that experiment that I was talking
[5:31] about in the beginning and finally ship
[5:33] it to production and do that whole loop
[5:35] over and over again continuously. Now,
[5:38] this is your standard go-to process for
[5:41] actually improving your agent. It's
[5:43] pretty long, pretty manual, and pretty
[5:45] tedious.
[5:47] It does it does work, but we want to
[5:49] kind of iterate and improve that. How
[5:51] can we do that?
[5:52] We want to make sure that we're not
[5:54] spending hours, days finding the issue,
[5:57] finding the trace, finding all these
[5:59] different issue points, coming up with
[6:01] the potential solution, going through
[6:03] that full eight-step process manually.
[6:04] We want to pull all of that together and
[6:06] bring it into minutes. The best way to
[6:09] do that would be right here. We're going
[6:11] to pull in all the data from all our
[6:13] different sources into one specific
[6:15] area. We're going to start with, you
[6:17] know, pulling our ap our arise traces so
[6:21] you can get all the information of what
[6:23] your agent did. You're going to pull in
[6:25] your evaluations to see where things
[6:28] actually went wrong. You're going to
[6:30] bring in your annotations from any
[6:31] humans that could theoretically tell you
[6:33] a little bit more information as to, you
[6:36] know, where it could go right, how they
[6:37] can improve it. You bring in your actual
[6:41] APM traces which could be hey is the
[6:44] system up or down. you bring in your
[6:46] codebase so you can understand all right
[6:49] based on this specific area where it
[6:51] failed the tool call what changes can I
[6:53] make in my codebase so that we're c
[6:55] we're we're going down the right
[6:56] pathways we're calling the right tools
[6:59] we're fixing our tools that are broken
[7:02] and so you'll bring in all of these into
[7:03] one final space which is which is our
[7:05] coding agent whether you use cloud code
[7:07] cursor codeex it doesn't matter all of
[7:10] those you can connect all that data into
[7:13] and it can automatically do all of that
[7:15] work for for you.
[7:17] So, I know this may sound really
[7:19] complicated and I promise this also
[7:21] probably doesn't help to make it look
[7:22] simpler, but it is very simple to get it
[7:24] set up. It's going to be just a couple
[7:27] steps of just, you know, setting up your
[7:28] CLI and skills and you're ready to go.
[7:32] You get access to a bunch of
[7:34] information, a bunch of different tools
[7:35] that your agent can then do, which could
[7:38] be for example here, which are like,
[7:40] hey, you can get your code instrumented.
[7:42] What that means is you can get all the
[7:44] traces from your agent into whatever
[7:46] platform you choose like Arise. You
[7:49] could go ahead and pull down those
[7:50] traces, build out data sets, run
[7:53] experiments, run evaluations. You have
[7:55] the full gamut of the experience all
[7:57] from within your coding agent.
[8:01] So, what I'm going to do here is we're
[8:03] going to go ahead and look at a specific
[8:06] issue. And I want a little bit of kind
[8:09] of uh voices from this side of the room
[8:13] uh where we can get some idea of like
[8:15] hey what do you guys think could be the
[8:16] actual issue. So I'll give you a little
[8:18] bit of background. This agent right here
[8:20] is a simple financial agent. I've asked
[8:23] it should I invest in this company? I
[8:25] think I chose Tata Motors. Um anyways
[8:29] and ultimately it gave me a response
[8:31] which said yes you should. I don't think
[8:33] it's directly in the screenshot, but I
[8:35] promise you it told me that. Um, and
[8:38] ultimately that's not really what we
[8:40] want to do because I'm not registered to
[8:42] give any financial advice, nor is this
[8:44] agent, and I don't want to get in legal
[8:45] trouble.
[8:47] If you guys can help me figure out
[8:48] maybe, hey, what do you guys think could
[8:50] be the issue in this agent? So, what she
[8:52] said was, "Hey, maybe the agent needs to
[8:55] ask a lot more questions back to get a
[8:57] little bit more information before I can
[8:58] give a solid, concrete answer." That's a
[9:00] very good guess. Anyone else have any
[9:02] ideas? Okay. And how do you suggest we
[9:05] potentially improve that? By the way, he
[9:06] said it should ask, again, pretty
[9:08] similar. It should ask more questions
[9:10] before making that recommendation, but
[9:11] how do you suggest maybe we can go ahead
[9:13] and do that?
[9:14] >> More context, more asking questions,
[9:16] more details.
[9:17] >> Okay, so maybe some prompt changes to
[9:19] add in some context to get more context.
[9:22] One more.
[9:24] >> Sorry.
[9:25] >> Where's validation?
[9:26] >> Where's validation?
[9:28] Yeah, that's true. We don't have any
[9:29] sort of ground truth to compare against.
[9:31] We don't know if any of this is correct.
[9:33] So there's no validation that any of the
[9:35] answers are even correct. You're right.
[9:37] There could be so many different issues.
[9:38] And truthfully, oh yeah, go ahead.
[9:40] >> Question. Does the agent know the user's
[9:43] financial background?
[9:46] >> It's a great question. In this case, no,
[9:47] it does not. And yeah, it should
[9:49] definitely get a lot more information
[9:50] into that. Uh he asked, by the way,
[9:52] should the user does the does the agent
[9:55] know any information about the user's
[9:57] financial background? Um and yeah, it
[9:59] definitely does not. And there's a lot
[10:00] of areas that this agent could be fixed.
[10:02] And there's genuinely genuinely not one
[10:05] correct answer, right? But and I
[10:08] genuinely don't even know what the right
[10:09] answer is to this because this agent is
[10:11] still messed up right now. And we're
[10:12] going to go ahead and try to fix it
[10:14] today in this demo. Um, and so what
[10:17] we're going to do is we're going to go
[10:18] ahead and set up the CLI and skills, get
[10:21] that set up so we can actually get our
[10:23] coding agent to go ahead and understand
[10:25] what's going wrong with our to go ahead
[10:27] and pull down the traces, understand
[10:29] what's going wrong with our agent, look
[10:31] at the evaluations we already have run,
[10:33] look at the annotations that exist,
[10:36] maybe even do some of its own analysis,
[10:38] and potentially come back with some
[10:40] future suggestions just like many of you
[10:42] have done today for how to actually
[10:44] improve this agent. So that hey, maybe
[10:46] we could have it ask more questions.
[10:48] Maybe we could add a guard rail so it
[10:50] only responds to certain questions and
[10:52] doesn't give financial advice where it's
[10:54] not warranted. So there's lots of
[10:57] different ways that it can come to do
[10:58] those fixes.
[11:01] To get it started, what I'm going to do
[11:03] is I'm going to go ahead and pull open
[11:05] let's see, do I have it here? Yes. Let's
[11:08] go ahead and pull open a new session.
[11:11] Um,
[11:13] okay, cool. I'm in the right space. All
[11:15] right. So, generally what you'd want to
[11:18] do whenever you want to set up the CLI
[11:19] and skills, super easy. It would just be
[11:21] a couple steps to get that set up. Um I
[11:24] apologize. Oh, here we go. I know what
[11:27] I'm doing, guys. Uh here we go. So,
[11:30] we're going to start by setting up the
[11:31] CLI. You would just go ahead and copy
[11:33] this one line of code to get that set
[11:35] up. So, I'd go into back into my
[11:37] terminal and install that. Of course, I
[11:39] already had that installed right before
[11:40] this, but just goes just very simple and
[11:43] quick to set up. Then what you do is you
[11:45] would create your actual profile, which
[11:47] would help you connect it to wherever
[11:49] your data stored. In this case, it would
[11:50] be arise. And you create that profile so
[11:53] it can go ahead and pull in those traces
[11:56] and visual and get that sort of
[11:57] information from those evals and
[11:59] annotations.
[12:01] Once you have that profile created,
[12:03] you'd go ahead and set up your skills.
[12:05] You could do this in a multitude of ways
[12:07] depending on your coding agent. For
[12:08] today, we're using Cloud Code, so I use
[12:10] the marketplace to set it up, but it's
[12:12] pretty easy to set up however you want.
[12:17] From here, let's go ahead and we're just
[12:20] going to ask it. And again, uh just as a
[12:23] warning, I am using dangerously skip
[12:24] permissions. Please don't do this. This
[12:26] is not actually good practice. This is
[12:28] just for a demo, so I don't have to keep
[12:30] pressing yes. Um, but we're going to go
[12:33] ahead and talk to Claude and have it go
[12:35] ahead and take a look at these traces.
[12:37] So, I'm going to go ahead and see if I
[12:39] can find the prompt I asked earlier,
[12:41] which was just use this skill and pull
[12:45] down the traces from that specific
[12:47] agent. Find out what issues exist that
[12:51] we can determine from any, like I said
[12:52] earlier, right? What any issues that
[12:54] exist from any of the evals, any of the
[12:55] annotations that we've done. do your own
[12:57] analysis and tell me what eval maybe I
[13:01] can build out in the future to make sure
[13:02] that I don't run into any of these
[13:03] issues again. And so this is going to
[13:06] take a couple minutes. Um, in the
[13:08] meantime, I'm going to go ahead and
[13:11] first go and show you kind of how those
[13:13] traces look and how you can actually
[13:15] start triaging it if you want to do it
[13:17] manually a little bit.
[13:19] But while this is running in the
[13:21] background, let's go back into the
[13:23] actual uh
[13:26] run into the actual agent that has those
[13:28] sort of queries that we've asked. So,
[13:30] we'll go ahead and ask it a pretty
[13:31] similar question. Um, you know, tell me
[13:33] if I should invest into, let's say,
[13:37] Nvidia this time. It's a little bit more
[13:39] of a well-known stock. And we're going
[13:42] to get that and it's going to try to
[13:43] tell us some sort of information. And as
[13:45] that data comes in, looks like I've run
[13:48] this earlier. So, and you can see it's
[13:50] coming in. Um, but from an earlier run,
[13:52] we can kind of see what it's done
[13:53] already where you can see that query we
[13:56] asked, you can see the output that it'll
[13:58] finally generate, which we'll see how
[13:59] similar it is in a minute from this
[14:01] previous run. Um, and you can see all
[14:03] the steps that actually happened in your
[14:05] agent, right? So, you can understand,
[14:07] hey, we had a supervisor agent, a
[14:08] financial data agent, you can see the
[14:10] LLM calls, you can see all your
[14:12] different tool calls. you get a lot of
[14:14] information right away that you can
[14:16] start using just like we done in the
[14:18] initial slide to see where those issues
[14:21] could lie. Now, let's see. Did Claude
[14:24] finish up? Looks like it's still
[14:25] running. It's doing its thing. We'll let
[14:27] it cook. You know, it's kind of the
[14:29] point of this. Um, while it's cooking,
[14:32] anyone have any questions as to like,
[14:34] you know, how you can set up any sort
[14:36] form evals, any sort of traces,
[14:40] anything along those lines? Anyone
[14:41] wanted to learn a little bit more about
[14:45] Yeah.
[14:57] >> Great question. Yeah. So, you would go
[14:58] ahead. So, what it does, it actually
[15:00] goes ahead and downloads a lot of those
[15:02] traces um into the into your local
[15:05] system. So, what it'll do is actually
[15:07] let me go ahead and show you what it how
[15:09] it actually goes ahead and accesses the
[15:11] system itself. So when you give it
[15:14] access to these skills, what you're
[15:15] actually giving it access to is the CLI
[15:17] underneath. And what I mean by that is
[15:19] you're going to have all these different
[15:21] CLIs that it can integrate with. So you
[15:24] can go ahead and set up and find the
[15:26] data sets. You can pull down data sets.
[15:28] You can have it interact with the
[15:30] essentially interact with the UI where
[15:31] it can set up evaluations on the
[15:34] platform on your behalf. So you can
[15:35] monitor continuously monitor when your
[15:37] agents are failing, build out data sets,
[15:40] run experiments. It can basically do
[15:42] anything that you that you would be able
[15:44] to do in the platform. Sometimes it'll
[15:46] download, sometimes it'll make API
[15:48] requests, but that's generally the uh
[15:50] idea here. Great question.
[15:55] Any other questions in regards to CLI
[15:57] tracing?
[16:02] Okay, let's go ahead and see what
[16:03] Claude's done. Oh, still crunching away.
[16:06] Um, so as we can see here, let's let's
[16:08] walk through what Claude's done so far.
[16:10] So, we can see that we've gone ahead and
[16:13] gotten the spans.
[16:15] It's digging into the evaluations to
[16:17] figure out where those issues are. Um,
[16:20] it looks like it's not quite there's
[16:22] definitely found one very big issue with
[16:24] some numeric groundedness. I know
[16:25] someone mentioned that earlier as a
[16:27] potential issue or something related to
[16:29] that. Um, we're also seeing some tool
[16:33] calls directly fail. Um,
[16:36] and let's see. We also see,
[16:39] yeah, so we're getting a lot of
[16:41] different errors. Oh, looks like it just
[16:42] finished up. And so you can see here we
[16:44] get a full list as to like, hey, what's
[16:46] going wrong with our agent and what we
[16:48] can do to kind of fix it. So while it's
[16:51] finishing up that, you can see here one
[16:54] of the biggest alarming things right out
[16:56] of the gate is about 44% of our agent
[16:59] interactions are resulting in a bad
[17:00] summary.
[17:03] That's really bad. I did not know that.
[17:05] So, we definitely need to fix that one.
[17:07] Another big issue is Oh, did not mean to
[17:11] scroll there. Another big issue is, hey,
[17:14] we're also just getting a lot of
[17:16] fabricated answers,
[17:18] which is just as bad as this the
[17:20] summarization being wrong. So, it sounds
[17:23] like it may be related, it may not be.
[17:25] We can kind of debug that. We're also
[17:27] seeing a relatively high rate of us just
[17:31] redundantly calling tools. It's a waste
[17:33] of our time, our resources, and our
[17:35] money. So clearly, we're doing a lot of
[17:37] bad things that we need to go ahead and
[17:38] fix. And that's just from the eval
[17:43] from Claude's own analysis. It's able to
[17:45] even go ahead and figure out, hey,
[17:47] there's other issues that are coming
[17:48] along. We're getting silent tool
[17:50] failures. It's telling us it succeeded,
[17:52] but ultimately it's not giving us any
[17:54] actual responses.
[17:57] We're getting the agent hallucinates
[17:59] when tool fails instead of disclosing
[18:01] it. So again related but hey that means
[18:04] maybe we need to not just do uh you know
[18:07] some eval built on top of this to
[18:09] understand when the tool is failing but
[18:11] we also need to make some prompt based
[18:12] changes to make sure that the the agent
[18:14] doesn't hallucinate here and you have a
[18:16] multitude of like different things that
[18:18] it'll go ahead and tell you as well as
[18:20] the evals it can build out. So what we
[18:22] could do is we can actually go ahead and
[18:24] tell it, hey, let's go ahead and build
[18:27] number one, which was the tool success
[18:30] silent failure eval, and fix numeric
[18:33] groundedness so that it'll make sure
[18:34] that whenever it does respond, it's
[18:36] always grounded to some sort of actual
[18:39] reference point and it's not making up
[18:41] these numbers.
[18:43] But we're also going to head and make
[18:44] sure you test this against a data set uh
[18:51] against the spans that caused the
[18:54] failure initially
[18:57] using and then we're going to go ahead
[18:58] and use another Arise skill which would
[19:01] be here the Arise uh data set skill, the
[19:04] Arise
[19:07] Oh, there we go. Arise um evaluator
[19:10] skill as well as the Arise experiment
[19:14] skill. So here it's going to be able to
[19:16] let me just go ahead and run let this
[19:18] run
[19:19] and build out evals. All right. So again
[19:24] I'm being pretty vague here but
[19:25] essentially I'm just telling it go ahead
[19:26] and make the changes that you
[19:27] recommended as well as go ahead and run
[19:30] these experiments for us. So now we can
[19:32] actually compare against where it failed
[19:35] to make sure it passes against that
[19:37] right and so in this scenario we can see
[19:40] okay it's going to go through and take
[19:41] its time. Um I'm going to go ahead and
[19:44] show you a previous run because again
[19:45] this will take some time and while we
[19:47] want to let it cook I don't want to have
[19:48] you guys staring at a screen where
[19:50] Claude's just discombobulating or
[19:52] whatever it does in its free time. Um,
[19:54] so let's go ahead and look at where I've
[19:56] actually gone ahead and asked Alex or
[19:58] sorry
[20:00] Cloud Code to go ahead and answer some
[20:02] of our previous issue that I ran into.
[20:05] And one of those was related to hey,
[20:08] I've had some times where I'm trying to
[20:10] get some answers about uh
[20:14] stocks that aren't necessarily from the
[20:16] United States. And so we're having
[20:18] errors where we're talking about like
[20:20] South Korea or looks like they're all
[20:23] South Korea based, but let's assume
[20:25] there's other answers here. Maybe Japan
[20:27] or China and whatever, right? And you
[20:29] want to see and you want to be able to
[20:30] understand, hey, I want to be able to
[20:32] compare stocks across different markets.
[20:35] And clearly people are asking that, but
[20:37] we don't support it. And so what Claude
[20:40] did, it went ahead found those areas
[20:42] where that issue existed. Like I said,
[20:44] it looks like it was all South Korea
[20:45] here, but it found those areas. It built
[20:49] out a data set for me, which I can then
[20:51] go ahead and test against to make sure
[20:53] it works. And then it went ahead and ran
[20:55] experiments. And it made a couple
[20:57] changes. So, it started with the initial
[20:58] one, which was the baseline, which was
[21:00] buggy, which as we expected would result
[21:03] in 0% uh passing rate, which is what
[21:06] happened when we initially got the
[21:08] actual productionbased traces.
[21:11] It also made a quick fix where it
[21:13] normalized it by adding in that
[21:16] uh country name. It didn't work. Why?
[21:20] Because the tool I was calling was also
[21:21] outdated and I needed to update to the
[21:24] stable version. So then it made a third
[21:26] iteration. And you can see that on this
[21:27] chart here as well where we were able to
[21:30] see, hey, the first iteration didn't
[21:32] work. It went back, made a fix, and
[21:35] finally got us to the final iteration
[21:36] where we're getting about a 90% success
[21:39] rate. Clearly still not 100%. We still
[21:42] need to make some changes, but hey,
[21:44] we're doing a lot better than we were
[21:45] earlier. So, this is like a great way to
[21:50] get you going so you can actually start
[21:52] from, hey, I know there's an issue to
[21:54] here's a fix. Now, I know we have quite
[21:58] a bit more time. I want to check in. How
[22:00] is everyone doing? How do you guys feel?
[22:01] Do you guys have any questions about how
[22:03] you can go ahead and get this started?
[22:05] Yeah.
[22:23] Great question. So his question was how
[22:25] where do the experiments actually run?
[22:27] Where do the evaluations run? Where does
[22:29] everything run? Um so you do have a
[22:32] little bit of customization there. In
[22:34] this scenario, uh what I did is I had
[22:36] the experiment run on my local machine,
[22:40] which is I think what will be the case
[22:42] most of the time. And then what you what
[22:44] I did afterwards was I went or you can
[22:47] host it on a cloud or wherever you'd
[22:48] like to experiment on. And then you can
[22:50] go ahead and actually run the
[22:51] evaluations on Arise itself. So in this
[22:53] case, I built out LM as a judge, which
[22:56] would then go ahead and let's see if I
[22:58] can see the evaluator here itself. So
[23:00] you can see we built out two different
[23:02] evaluators, a codebased and an LLM
[23:04] based. I actually had Claude go ahead
[23:06] and do this. It built it out for me in
[23:08] the platform. And this was the final LLM
[23:10] as a judge because it actually kind of
[23:12] failed in getting the codebased data to
[23:14] set up. So that's why we have two. And
[23:17] so you can see the LM as a judge. It
[23:18] went ahead and built it out for us with
[23:21] this prompt and got it going so that we
[23:23] can actually compare. Hey, did the agent
[23:25] work properly?
[23:28] Any other questions? That was a really
[23:29] good question by the way.
[23:31] >> Yeah.
[23:40] >> Very very good question. Um so
[23:44] eval can be set up however you like. We
[23:46] have of course people can people can
[23:50] definitely use whatever sort of models
[23:51] they want. No one wants to be limited to
[23:53] just claude or just open AAI especially
[23:56] in today's modern age where you can see
[23:58] AI models literally ship every other day
[24:01] and then you want to be like okay I want
[24:02] to use that one now right and also some
[24:04] people have a preference for slightly
[24:06] cheaper models some people have a
[24:07] preference for making sure their evals
[24:09] are topofthe line because ultimately
[24:11] they know their agent's only going to be
[24:12] as good as their evals so they use the
[24:14] highest end models for their evals even
[24:16] if they're not necessarily using those
[24:18] in production so it kind of varies you
[24:20] have full customizable control over cut.
[24:22] And so what you can see here for
[24:24] example, you have a full list of
[24:25] whatever sort of model provider you want
[24:27] to use and you can connect those even if
[24:29] you want to use any sort of local um
[24:31] open source models as well.
[24:35] Yeah,
[24:37] good question. So her question was if we
[24:40] have if I spin up a smaller agent to go
[24:43] ahead and make some of these bug fixes,
[24:46] how does this compare to that?
[24:48] Realistically, you don't have to use
[24:50] cloud codeex. You can use whatever sort
[24:52] of agent you want. The benefit of what
[24:54] we've provided is a skill that you're
[24:56] able to go ahead and pull down the
[24:58] information from your traces to make
[25:00] those fixes. So, you're not making those
[25:02] fixes just based on, hey, I know this is
[25:05] wrong and the output's incorrect. You're
[25:07] making those fixes based on, hey, I know
[25:09] this is wrong. Here's where it failed.
[25:11] Here's why it failed. Here's how we can
[25:13] go ahead and fix it. because you have
[25:15] not just the information of your
[25:17] codebase, you have the information of
[25:19] any of your other APM tickets, whatever
[25:22] or APM logs, any of your other tickets
[25:24] that exist in your Jira systems, etc. as
[25:27] well as your actual arise traces, which
[25:29] would be like here's the pathway the
[25:31] agent took, here's where it messed up,
[25:33] your eval, your annotations, etc.
[25:37] Any other questions? Yes.
[25:38] >> Are these features available in Phoenix?
[25:41] >> Great question. Yes. Uh we do have so he
[25:44] asked is are these features available in
[25:46] Phoenix. Phoenix is Aris's open- source
[25:49] version. We do have I believe skills and
[25:52] MCP as well in Phoenix which you can go
[25:55] ahead and hook up to your agent and it
[25:57] can go ahead and pull down the same sort
[25:59] of information. Of course there are
[26:00] going to be some nice to haves in the
[26:02] enterprise edition where you have access
[26:04] to these online evals in a certain way
[26:06] that are much easier to format and set
[26:08] up. You have a little bit more
[26:09] dashboarding and a few other features.
[26:11] But yes, the the skills in CLI you can
[26:13] connect to your coding agent.
[26:16] >> Yes.
[26:20] >> Agent said that summary. So how would
[26:24] you classify something as like a bad
[26:27] agent know that's a good summary or a
[26:29] bad?
[26:30] >> Very good question. So her question was
[26:32] in what I showed earlier there was a
[26:35] eval or there was a response from Claude
[26:37] that said hey there was a bad summary.
[26:39] How do we know what is a bad summary or
[26:41] a good summary? In that case, what I
[26:44] actually had set up, it was actually
[26:45] based on one of the evals that I had
[26:47] previously set up. And so, we can
[26:49] actually go ahead and look at that
[26:50] evaluator here. We can go ahead and look
[26:53] at let's see some
[26:58] I actually don't remember which one, but
[27:00] I'm going to go ahead and pull open this
[27:01] one and hope this is the right one. Um,
[27:04] so as you can see here, this is
[27:06] basically the summary eval that I'd
[27:08] created.
[27:10] It's basically taking a look at the
[27:12] information we've gathered throughout
[27:13] our entire agent, all the sort of
[27:16] context we've given to that summary
[27:17] agent and making sure that it's
[27:19] referencing the right points, it's
[27:21] answering the question accordingly, etc.
[27:23] Um, and so that's how we've gotten to
[27:25] determine whether or not it was a good
[27:26] summary or not.
[27:33] >> Uh, what if we set up a skill in the
[27:35] first place to see if it can do a good
[27:37] summary?
[27:38] give it given us a good summary all the
[27:41] time
[27:42] just like we set up a skill instead
[27:45] instead of running.
[27:48] >> Yeah, great question. So, she's asking
[27:50] why can't we just set up skills that you
[27:52] know ensure that it gives a good summary
[27:55] and yes absolutely that is what you want
[27:57] to do. You want to make sure that you
[27:59] update your prompts, update your skills,
[28:01] update your coding workflows so that
[28:03] your agents will give the best
[28:04] summaries. The emails are only there to
[28:06] help you determine where there are
[28:07] issues, monitor those issues, make sure
[28:09] they don't repeat, right? And so you set
[28:13] up that skill, great, it works for the
[28:15] next two months, two months later
[28:16] because there's a model regression
[28:18] because people start asking questions in
[28:20] a different way. You end up getting
[28:22] different results and you don't want
[28:24] your e you don't want your performance
[28:26] to degrade because of that. So you want
[28:28] to be able to monitor and observe that.
[28:30] That's the point of the evals. Did that
[28:31] help? Beautiful. Um I don't know for how
[28:35] much time we have left but go ahead
[28:39] >> later
[28:45] agents are assessed
[28:48] and soc.
[28:56] So your question was if I've run into
[28:58] issues with linting with the agents
[29:00] >> agents
[29:07] seems like more
[29:11] >> good question. Yeah. So he's basically
[29:13] saying hey
[29:15] um have you seen the issue where hey
[29:17] you've given access claude to all these
[29:20] skills and all these sort of traces and
[29:21] all this sort of information but
[29:23] ultimately most of the time I've seen
[29:25] with cloud code or any of these other
[29:27] sort of coding agents they end up
[29:29] focusing on linting rather than any
[29:31] other sort of issue am I correct yeah
[29:33] that's a very good understanding and
[29:35] that's kind of why we built out these
[29:36] skills which kind of focus on ensuring
[29:39] that they hyperfixate on the actual
[29:41] issues that have been given to them,
[29:42] right? So they'll get all these trace
[29:44] information so they'll understand, hey,
[29:46] it's not going to be a linting based
[29:47] issue because you can see that it's
[29:50] calling the wrong tool and that's not
[29:52] going to happen because of linting and
[29:53] it does have some reasoning capability
[29:55] to understand that. Um, and so genuinely
[29:58] I haven't seen that so much when using
[29:59] the skills.
[30:03] Awesome. So as it Oh yeah.
[30:21] Does does the system make it easy to
[30:23] understand? Sorry.
[30:27] >> Great question. Yeah. So, how so his
[30:29] question was, does it make it easy to
[30:31] understand how much it costs to run
[30:33] those evals? Absolutely. So, one of the
[30:36] best parts of running evals is in the
[30:38] platform is you're going to be able to
[30:40] get full trace history of what your
[30:42] emails did, which also comes with the
[30:44] added benefit of, you can see right
[30:46] here, a nice little cost. And so, you'll
[30:49] be able to understand how much your
[30:51] emails are actually costing you because
[30:53] yes, of course, anything anytime you're
[30:55] running another LLM, it's going to be
[30:57] additional cost to you. So, you want to
[30:58] make sure it's not. And of course,
[31:00] what's really nice about building out
[31:02] evals, whether it's in the UI or via the
[31:05] coding agent, you can have those set up
[31:07] basic requirements like, hey, we want to
[31:10] sample specific amounts so we don't run
[31:12] it across 100% of our traces and 100% of
[31:15] our calls. Um, you can make it focus on
[31:18] specific areas. And of course, you can
[31:20] always dictate that in the coding agent
[31:22] or on the platform itself.
[31:25] Um, so as a kind of refresher as to
[31:28] everything we kind of covered today,
[31:30] again, it that's the wrong one. Oh, I
[31:32] think I have it over here.
[31:35] Um,
[31:38] so
[31:40] essentially what I really want to give
[31:42] you as a takeaway,
[31:44] these AI agents, they're great. They
[31:46] will do basically everything you ask
[31:48] them to, but again, they're not magic.
[31:50] Like he said, they can definitely focus
[31:52] on linting. They can definitely get
[31:54] caught up in small little areas, but the
[31:57] best part of them is they will automate
[31:59] your full workflow from beginning to
[32:01] end. So whether it's that starting from
[32:03] observing where the issues lie,
[32:07] finding those issues, being able to
[32:09] actually pull it down, understand where
[32:12] the issues lie, create those data sets,
[32:15] build out exact fixes,
[32:18] test them to make sure that they're
[32:20] working.
[32:21] And finally, you can then go ahead and
[32:23] be the manual human to make sure that
[32:26] hey, it didn't just go back and fix the
[32:28] linting changes, right? And we can make
[32:30] sure that it actually made changes that
[32:32] actually help improve the agent and not
[32:35] go for degradation.
[32:39] That's kind of the general gist of what
[32:40] I wanted to show today. Um, any other
[32:42] questions on this before, you know, I
[32:44] can maybe show one last thing of seeing.
[32:46] Oh, still finishing up what I was trying
[32:48] to do earlier. I was hoping it might be
[32:49] done after this. took about 14 minutes.
[32:52] But
[32:55] um while that's finishing up, any other
[32:56] questions on how we can use our agents
[32:59] to automate our workflows
[33:01] and improving your agents? Yeah.
[33:19] >> Great question. So his question was, are
[33:21] we primarily relying on Claude or any of
[33:24] these coding agents to actually go ahead
[33:26] and give us the evaluation if things are
[33:28] working well or not? Um, yes, you can
[33:31] definitely rely on Claude to build out
[33:33] everything. You can also manually do it.
[33:36] You can also ask Claude to do the
[33:38] specific evaluations you want. So in
[33:40] this case, when you saw that nice clean
[33:42] table, that was actually defined by the
[33:43] evaluations me and my team had set up
[33:45] prior to this. Um, we'd gone ahead and
[33:48] looked at, hey, these are specific areas
[33:50] that we know are potential limitations
[33:52] in most agents that work in this
[33:54] workflow and we set up those as generic
[33:56] ones, which is why we got a bad summary.
[33:58] We had some, hey, is the agent healthy?
[34:01] Is it giving the right are we getting
[34:02] tool calls correct? Right? We know those
[34:04] are general areas, but what Claude was
[34:06] able to do was look at our actual
[34:08] traces, understand based on those
[34:11] traces, what are the actual issues we're
[34:13] falling into today. Right? we have our
[34:16] live agent in production. I'm not going
[34:18] to be able to go through all of those
[34:19] manually and find those, but Claude can
[34:22] or Claude cursor codeex, whatever,
[34:24] right? And that's where like one of the
[34:26] biggest superpowers that you're able to
[34:27] give these coding agents to be able to
[34:30] do that for you.
[34:32] I saw another question here. Good
[34:35] question. So his question was is there
[34:36] any sort of benchmark data set that we
[34:38] can see or any sort of like
[34:40] visualization that can tell us hey this
[34:42] is a higher quality LLM to use for
[34:44] evaluations more expensive less
[34:46] expensive um realistically no [laughter]
[34:50] that would be sort of something you'd
[34:51] have to do for your own analysis you'd
[34:53] have to figure it out because ultimately
[34:55] one of the things that I like to say is
[34:57] if you're going to use out of the box
[34:58] eval to judge your agent you're going to
[35:01] get out of the box results and that's
[35:03] not what you want when you're building
[35:04] customtailored
[35:05] AI agents which are built to be personal
[35:07] for you. Um, and the same goes for
[35:10] benchmarks, right? Benchmarks are great.
[35:12] They'll show you, hey, this this is a
[35:15] good agent. This is a good sorry, this
[35:16] is a good LLM. But realistically, what
[35:18] you want is, is it a good LLM for me?
[35:21] And that's going to be tailored to your
[35:23] use case. So, if you're building for
[35:24] finance, maybe you'll use a specific
[35:26] one. If you're building for, you know,
[35:28] law in finance, it might be even this
[35:30] slightly different model, right? So, it
[35:32] really depends on what you're building
[35:33] it for.
[35:37] Cool. I think that's more or less it
[35:38] more or less it from me. Um, if anyone
[35:40] has any other questions, I'm happy to
[35:42] answer them. But yeah, thank you guys.
[35:46] [applause]