Arize Deploys Alex to Automate Engineers Out of LLM Observability

LLM Observability, Evaluation, Experimentation Platform — Dat Ngo, Arize — AI Engineer

AI Engineer

Because static code cannot audit nondeterministic systems, telemetry must serve as the primary audit layer while autonomous agents generate and run evaluations directly from execution traces.

AI-assisted summary of LLM Observability, Evaluation, Experimentation Platform — Dat Ngo, Arize — original source. Check the original for full context.

Source published:

Key ideas

  • Code does not audit non-deterministic agents—telemetry does, requiring OpenTelemetry-native auto-instrumentation to trace dynamic execution paths and stateful sessions. (0:00)
  • Agent evaluation spans five distinct signal flavors (LLM-as-a-judge, human feedback, golden datasets, deterministic checks, and business metrics) across four granular scopes from single spans to full conversational sessions. (5:00)
  • Arize aims to eliminate manual dashboard interaction and evaluation writing through Alex, an autonomous AI layer that monitors traces, flags anomalies, and creates evals dynamically. (12:50)
  • The platform offers Arize Phoenix as an open-source, single-container deployment requiring no Kubernetes, alongside the enterprise Arize AX for large-scale production workloads. (15:15)

Detailed notes

## Background Published by the AI Engineer channel on YouTube on June 7, 2026, this talk features Dat Ngo, an AI architect at Arize AI. The presentation addresses the challenge of debugging non-deterministic AI agents, explaining how telemetry acts as an audit layer and walking through evaluation signals, scoping, Arize Phoenix, and automated observability with Arize's Alex system.

## TL;DR Dat Ngo argues that modern AI architecture is fundamentally software reimagined, requiring telemetry rather than traditional code to audit non-deterministic agents and test harnesses. He outlines an OpenTelemetry-native approach that captures traces, spans, trajectories, and multi-turn sessions across complex graph execution paths. The talk details five key evaluation signals—LLM-as-a-judge, human feedback, golden datasets, deterministic checks, and business metrics—and explains how to scope them from single spans to full user sessions. Finally, Dat demonstrates how curated trace datasets power experimentation and previews Arize's internal agent, Alex, which aims to automate developers out of manual evaluation loops.

## Atmosphere & style The presentation is energetic, pragmatic, and delivered with casual developer humor, featuring jokes about burning through hundreds of billions of tokens, 'vibing it out,' and forgetting to switch off dark mode for stage demos. Dat strips away AI mysticism to speak directly to seasoned builders as peers, establishing an authentic, technical camaraderie. The session feels fast-paced and grounded, transforming messy non-deterministic agent debugging into an actionable, disciplined systems engineering challenge.

## Bulletin - [00:00](https://www.youtube.com/watch?v=JsCCrBF7F1g&t=0s) Code does not audit non-deterministic agents—telemetry does, requiring OpenTelemetry-native auto-instrumentation to trace dynamic execution paths and stateful sessions. - [05:00](https://www.youtube.com/watch?v=JsCCrBF7F1g&t=300s) Agent evaluation spans five distinct signal flavors (LLM-as-a-judge, human feedback, golden datasets, deterministic checks, and business metrics) across four granular scopes from single spans to full conversational sessions. - [12:50](https://www.youtube.com/watch?v=JsCCrBF7F1g&t=770s) Arize aims to eliminate manual dashboard interaction and evaluation writing through Alex, an autonomous AI layer that monitors traces, flags anomalies, and creates evals dynamically. - [15:15](https://www.youtube.com/watch?v=JsCCrBF7F1g&t=915s) The platform offers Arize Phoenix as an open-source, single-container deployment requiring no Kubernetes, alongside the enterprise Arize AX for large-scale production workloads.

## Key takeaways - [00:00](https://www.youtube.com/watch?v=JsCCrBF7F1g&t=0s) Modern AI engineering reduces to standard software discipline across three foundational pillars: observability, evaluations, and experimentation. - [00:00](https://www.youtube.com/watch?v=JsCCrBF7F1g&t=0s) Traditional static code and unit tests cannot audit non-deterministic agentic workflows; runtime telemetry serves as the true audit layer. - [00:00](https://www.youtube.com/watch?v=JsCCrBF7F1g&t=0s) OpenTelemetry-native auto-instrumentation captures execution traces and multi-turn stateful sessions using a single line of code. - [05:00](https://www.youtube.com/watch?v=JsCCrBF7F1g&t=300s) Macro distributional graph views are necessary to identify traffic divergence across branches, uncover latency bottlenecks, and diagnose dependency-ordering hallucinations. - [05:00](https://www.youtube.com/watch?v=JsCCrBF7F1g&t=300s) Signal extraction requires five complementary evaluation types: LLM-as-a-judge, human feedback, expert golden datasets, deterministic code checks, and bottom-line business KPIs. - [05:00](https://www.youtube.com/watch?v=JsCCrBF7F1g&t=300s) LLM observability platforms must bridge two personas: AI engineers building programmatic pipelines and non-technical domain experts writing evals via UI templates. - [10:00](https://www.youtube.com/watch?v=JsCCrBF7F1g&t=600s) Evaluation must be scoped intentionally across single spans, multi-spans, full trajectories, and multi-turn sessions to balance holistic insight against evaluation inference costs. - [11:47](https://www.youtube.com/watch?v=JsCCrBF7F1g&t=707s) Observability data creates a continuous feedback loop by allowing engineers to filter failed production traces directly into benchmark datasets for iterative prompt and model experimentation. - [12:50](https://www.youtube.com/watch?v=JsCCrBF7F1g&t=770s) Arize exposes CLI tools, skills, and MCP definitions designed for seamless integration with coding agents like Claude Code and Codex. - [12:50](https://www.youtube.com/watch?v=JsCCrBF7F1g&t=770s) Arize's ultimate architectural goal is automating engineers out of the observability loop using Alex, an AI system that plans tasks, diagnoses latency spikes, and authors test suites autonomously. - [15:15](https://www.youtube.com/watch?v=JsCCrBF7F1g&t=915s) Arize Phoenix provides a lightweight, open-source tracing option deployable as a single container without Kubernetes, while Arize AX supports high-scale enterprise infrastructure.

## Detailed walkthrough ### [00:00](https://www.youtube.com/watch?v=JsCCrBF7F1g&t=0s) Reimagining Software: Telemetry as the Audit Layer for Agents

Dat Ngo introduces himself as an AI architect at Arize AI, explaining that his day-to-day work centers on observability, evaluation, and experimentation across the world’s largest enterprises. Looking back on the scale of enterprise deployments over the previous year, Dat quips that between the prior OpenAI DevDay and the present, he personally burned through “somewhere between 100 billion and 1 trillion tokens last year.” While he jokes about liking to “vibe it out,” he stresses that the unique vantage point of Arize AI reveals the deep architecture patterns, common pains, and remediation strategies used by leading teams. Surveying the audience on who has built, deployed, or productionized agents and test harnesses, he remarks that the room is clearly “pretty cracked.”

Demystifying the hype around artificial intelligence, Dat argues that modern AI architecture is fundamentally familiar: “The AI space, it really just feels like software reimagined.” Stripping away the feeling of magic, it reduces to standard engineering discipline across three major pillars:

1. **Observability**: Asking what is actually happening inside an agent or harness during execution. 2. **Evaluations (Evals)**: Extracting reliable signal from systems to judge behavior and quality. 3. **Experimentation**: Navigating non-deterministic updates where fixing one bug often produces two or three unexpected regressions.

Dat underscores that Arize is built entirely OpenTelemetry-first (OTel-native). Rather than forcing teams to adopt custom proprietary logging wrappers, developers can drop in a standard auto-instrumenter using a single line of code. That single line intercepts execution within the framework or SDK, automatically capturing OpenTelemetry traces and spans. Emphasizing why this architectural foundation is mandatory for non-deterministic software, Dat points out: “Because now we know that code doesn’t audit agents or harnesses. It’s actually the telemetry that does that.”

Beyond raw trace-and-span execution trees, modern observability demands aggregate scopes like sessions and runs. Pointing to the recent Anthropic paper *Managed Agents*, Dat notes that sessions capture stateful, back-and-forth multi-turn interactions. In enterprise production environments, stakeholders frequently do not want to inspect low-level tool calls for every interaction; instead, they need to evaluate the entire conversational session to verify whether the customer was satisfied and had their problem fully resolved.

```mermaid flowchart TD accTitle: OpenTelemetry Agent Instrumentation Pipeline accDescr: An agent runtime uses a one-line auto-instrumenter to emit traces and spans into Arize for session and trajectory analysis.

subgraph Runtime["Agent Runtime Environment"] A["Agent / Harness Execution"] -->|"1-line auto-instrumenter"| B["OpenTelemetry (OTel) Collector"] end

subgraph Observability["Arize Observability Layer"] B --> C["Traces & Spans\n(Low-level execution record)"] B --> D["Sessions & State\n(Multi-turn interaction context)"] C --> E["Distributional Views &\nTrajectory Evals"] D --> F["User Satisfaction &\nResolution Metrics"] end ```

[05:00](https://www.youtube.com/watch?v=JsCCrBF7F1g&t=300s) Graph Trajectories and the Five Flavors of Evaluation Signal

Switching his interface into light mode so the audience can see clearly, Dat shows Arize AX to illustrate how to inspect complex, non-deterministic agents across large-scale traffic distributions. Rather than analyzing single isolated execution runs, enterprise teams need a distributional graph view showing every potential path, branch, and loop their agent can traverse. This macro view answers critical architectural questions: what percentage of total production traffic diverts down Branch A versus Branch B, and which specific component within an uncommon branch is responsible for sudden latency spikes?

Dat highlights trajectory evals as the key mechanism for debugging agent behavior. For instance, an agent might perform exceptionally well along one execution path, but on an alternative path, signal quality plummets. Using graph telemetry, developers can diagnose subtle root causes that traditional unit tests miss entirely: “These two components are actually out of order. I did B before A, and actually B has a dependency on A.” Telemetry exposes where the underlying model hallucinated the execution sequence or inverted dependencies, indicating that additional prompt guardrails or context are needed to enforce execution order.

Turning to evaluations, Dat insists that observability is only step one; engineers must systematically capture signal. He categorizes signal into five concrete flavors:

- **LLM-as-a-Judge**: Running automated model-based evaluations. While conceptualized simply, running robust model evaluations in production becomes deeply complex. - **Human Feedback**: Gathering direct telemetry from end users and internal reviewers, providing ground truth on product utility for technical leads and product managers alike. - **Golden Datasets**: High-trust benchmark datasets curated by domain experts. Teams run LLM-as-a-judge prompts across golden datasets to continuously tune and align the judge model to approximate trusted human domain scoring. - **Deterministic Logic Checks**: Inexpensive, non-LLM, code-level validators. For example, validating whether a paragraph-to-JSON transformation produces valid JSON matching a strict schema with non-null required fields, saving unnecessary model inference costs. - **Business Metrics**: Ground-level Key Performance Indicators (KPIs) mapped directly to corporate utility—specifically asking whether the system makes money, saves money, or saves time.

Dat notes that scaling AI products requires uniting two distinct personas: technical AI engineers who excel at systems engineering, harnesses, and programmatic pipelines, and non-technical subject matter experts (SMEs) or product managers who understand the desired user experience and domain nuances. Tooling must support both worlds: technical users can attach and trigger evals programmatically via code, while non-technical domain experts can select models, configure pre-built templates, or write custom judge prompts through a dedicated user interface without touching code.

[10:00](https://www.youtube.com/watch?v=JsCCrBF7F1g&t=600s) Granularity and Scope in LLM Evaluations

When building an application, agent, or harness, system components may execute deterministically or non-deterministically. To understand whether these components function properly, evaluation cannot just be treated as a monolith; teams must decide the exact scope and depth at which their evals run. At the most granular level is the single span eval, which simply examines the input and output pairs of an isolated component, such as an individual LLM call. Moving up one layer, multi-span evals come into play when an evaluation depends on data distributed across multiple components—for instance, evaluating how effectively multiple agents pass contextual data back and forth to one another.

```mermaid flowchart TD accTitle: Evaluation Scopes Hierarchy accDescr: Diagram illustrating the four scopes of evaluation from granular span to session level.

subgraph Granular ["Component Level"] S1["Single Span Eval<br/>(One LLM call input/output)"] S2["Multi-Span Eval<br/>(Cross-agent data passing)"] end

subgraph Holistic ["System & Journey Level"] S3["Trajectory Eval<br/>(Execution order & business steps)"] S4["Session Level Eval<br/>(Full conversation & user state machine)"] end

S1 --> S2 S2 --> S3 S3 --> S4 ```

Beyond isolated components, teams must monitor broader behavior across an execution run through trajectory evals and session-level evals. A trajectory eval inspects the full sequence of spans to determine whether the agent executed calls in the proper sequence to achieve the target business process. Zooming out further, session-level evals inspect the full state machine of a user interaction across time, answering holistic questions like whether a customer expressed frustration or whether all of their inquiries were resolved. However, teams should avoid running exhaustive evaluations across every conceivable metric because doing so quickly becomes cost-prohibitive. As Dat Ngo explains:

> “You want to see like, hey, what are the minimal set of evals I can get away with to understand signal of like, is my application working as intended because there's a cost associated with this stuff, right?”

- **Single Span Eval**: Inspects the input and output of a single component or LLM call. - **Multi-Span Eval**: Pulls data across distinct interacting components (e.g., verifying context exchange between cooperating agents). - **Trajectory Eval**: Evaluates all spans across an execution path to ensure steps followed the intended workflow. - **Session-Level Eval**: Evaluates overall conversation dynamics, state transitions, customer sentiment, and task completion.

---

[11:47](https://www.youtube.com/watch?v=JsCCrBF7F1g&t=707s) Driving Experimentation and Iteration from Datasets

Once observability data exists, it directly feeds into the experimentation and improvement workflow. Teams that capture traces can filter for problematic signals, identify missing information or failed logic, and curate those traces directly into clean evaluation datasets. For teams that do not yet have production traces, datasets can be uploaded directly as raw tables of input-output pairs.

With structured datasets in hand, developers can run iterative experiments either through the Arize UI or programmatically. Experimentation fundamentally revolves around testing modifications across core application levers: swapping foundation models, refining system prompts, updating orchestration logic, or adjusting parameter configurations. Tracking these experiments against golden datasets allows teams to measure whether proposed updates genuinely improve agent behavior or introduce regressions before deploying them to production.

---

[12:50](https://www.youtube.com/watch?v=JsCCrBF7F1g&t=770s) Automating Developers Out of the Observability Loop with Alex

Observability tooling is experiencing a major shift away from manual interaction. Most developers do not want to spend their working hours clicking around dashboards, flipping switches, or manually writing evaluation criteria. Because software authoring is compressing and becoming heavily agent-driven, Arize exposes its primitives directly through a command-line interface, skills, and tool definitions compatible with coding assistants such as Claude Code and Codex.

```mermaid flowchart LR accTitle: Automated Observability Flywheel accDescr: Flowchart showing how coding agents and Alex interact with traces and evaluation generation.

subgraph DevEnvironment ["Developer Environment"] Agent["Coding Agent<br/>(Claude Code / Codex / CLI)"] end

subgraph ArizePlatform ["Arize Platform & Tracing Engine"] Alex["Alex AI System"] Traces["Application Traces<br/>& Latency Hooks"] GenEval["Dynamic Eval Creation"] end

Agent <-->|"Queries / Invokes"| Alex Traces -->|"Telemetry Data"| Alex Alex -->|"Flags Latency & Errors"| GenEval ```

To eliminate the manual overhead entirely, Arize built an internal AI agent called Alex. Developers or external coding agents can directly task Alex with inspecting system health simply by asking if it spots issues in the application. Because the platform has direct hooks into trace telemetry, Alex autonomously plans and executes analytical tasks, identifying anomalies such as elevated latency clusters or error spikes. Dat Ngo envisions a self-driving observability flywheel where engineers no longer manually curate their tests:

> “Our ultimate goal as a company is actually to automate you out of this process.”

> “We are very heavy believers that you shouldn't even have to choose your evals.”

- **Autonomous Evaluation**: Alex evaluates trace history dynamically to detect regressions and author new test suites on the fly without human prompting. - **Coding Agent Interoperability**: Telemetry hooks, tools, and primitives can be invoked straight from CLI tools and environments like Claude Code. - **Trace-Driven Diagnostics**: Alex autonomously scans trace feeds to surface bottlenecks, latency spikes, and error patterns.

---

[15:15](https://www.youtube.com/watch?v=JsCCrBF7F1g&t=915s) Open Source Phoenix vs. Enterprise Arize AX

Arize serves two distinct deployment environments depending on organizational scale and architectural needs. For engineers looking for a lightweight, developer-first solution, Arize Phoenix is offered completely open source. Phoenix runs as a self-contained, single-container deployment that can be run entirely on a local machine without needing a complex Kubernetes cluster.

For large-scale enterprise deployments, the company provides Arize AX, designed to support high-throughput, mission-critical infrastructure for enterprise customers like Uber, Booking.com, and Reddit. Dat Ngo wraps up the session with a playful self-critique regarding Arize's heavy preference for dark mode across its user interfaces, noting that they should probably remember to switch to light mode occasionally for demos, before inviting developers to connect and explore the ecosystem.

## Notable quotes > [00:00](https://www.youtube.com/watch?v=JsCCrBF7F1g&t=0s) Because now we know that code doesn’t audit agents or harnesses. It’s actually the telemetry that does that. — Dat Ngo

> [12:50](https://www.youtube.com/watch?v=JsCCrBF7F1g&t=770s) We are very heavy believers that you shouldn't even have to choose your evals. — Dat Ngo

## Mentioned Dat Ngo, Arize AI, OpenAI DevDay, OpenTelemetry, Anthropic, Managed Agents, Arize AX, Alex, Claude Code, Codex, Arize Phoenix, Kubernetes, Uber, Booking.com, Reddit

## Video description Your agent called tool B before tool A, and B has a dependency on A. You did not catch it because nothing in your code audits agents. The telemetry does. Dat from Arize AI walks through what observability actually means when the system you are debugging is nondeterministic and the execution path changes with every run.

The talk covers the five flavors of eval signal (LLM as judge, human feedback, golden datasets, deterministic checks, business metrics), what scope to run them at (single span, multispan, trajectory, session), and where this is heading. Arize Phoenix is open source, runs as a single container, no Kubernetes required. The enterprise product adds an AI layer called Alex that scans traces, surfaces high latency and errors, and creates evals automatically. The stated goal: automate you out of the observability loop entirely.

Speaker info: - https://www.linkedin.com/in/datdarylngo/ - https://x.com/dat_attacked

---

Watch along with the corrected transcript, summary, timestamps and resources on this talk's official AIE page: https://ai.engineer/talks/JsCCrBF7F1g-llm-observability-evaluation-experimentation-platform-dat-ngo-arize-ai

Join our upcoming AI Engineer conferences (see https://ai.engineer landing page):

- AIEi Shanghai, November 5–6 — the first for meeting top Chinese labs and engineers - AIE CODE in SF, November 10–12 — the top AI Coding conference returns - AIEi Sydney, December 7–8 — the first AIEi run alongside NeurIPS 2026 in Sydney

---

[Source](https://www.youtube.com/watch?v=JsCCrBF7F1g)

Reading summary

Arize AI architect Dat Ngo argues that non-deterministic AI agents must be audited through telemetry rather than static code, requiring open-standard observability, multi-tiered evaluations, and automated feedback loops. OpenTelemetry as Foundation for Agent Telemetry (0:00) Dat Ngo frames AI development as standard software engineering reimagined around three pillars: observability, evaluations, and experimentation. Arize implements an OpenTelemetry-native (OTel) architecture, allowing engineers to capture traces and spans using a single-line auto-instrumenter instead of custom proprietary wrappers. Telemetry serves as the primary audit layer for non-deterministic software, capturing execution trees that static code analysis cannot audit. Beyond low-level spans, modern observability demands aggregate scopes such as multi-turn conversational sessions. Pointing to Anthropic's Managed Agents paper, Ngo emphasizes that enterprise stakeholders rarely need to inspect every low-level tool call. Instead, evaluating full sessions reveals state transitions and confirms whether user problems were fully resolved. Distributional Graph Trajectories and Evaluation Signals (5:00) Debugging complex agents requires a distributional graph view rather than isolated trace inspection. This macro perspective maps all possible paths, branches, and loops across production traffic, identifying latency bottlenecks and sequence failures, such as dependencies executing out of order. Trajectory evaluations surface where underlying models hallucinate execution order, informing where prompt guardrails are required. Ngo identifies five distinct evaluation signals: LLM-as-a-judge automated evaluations, human feedback from users and reviewers, expert-curated golden datasets, deterministic logic checks (such as verifying JSON schemas without model costs), and corporate business KPIs. Tooling must support two groups: technical engineers running programmatic pipelines and non-technical domain experts configuring judge prompts via visual interfaces. Granular Scopes and Trace-Driven Experimentation (10:00) Because exhaustive evaluation across every metric is cost-prohibitive, teams must constrain evaluations to the minimum viable scope needed to detect regressions. Ngo defines four evaluation granularities: single-span evaluations for isolated component input-output pairs, multi-span evaluations for cross-agent contextual data exchanges, trajectory evaluations across complete execution paths, and session-level evaluations for conversational sentiment and task completion. Observability directly feeds the experimentation cycle. Developers can filter production traces for failures or edge cases and curate them into structured datasets. These golden datasets allow teams to run iterative experiments—such as swapping foundation models, altering system prompts, or tuning orchestration logic—to verify improvements before deploying updates to production. Autonomous Observability and Flexible Deployment (12:50) Arize aims to remove developers from manual testing loops through Alex, an internal AI agent that interfaces directly with trace telemetry. Alex autonomously analyzes system health, flags latency clusters, detects error spikes, and writes new evaluation suites dynamically. arize also exposes CLI primitives and tool definitions compatible with coding assistants such as Claude Code and Codex. Deployment options are split by scale: Arize Phoenix offers a lightweight, single-container open-source deployment that runs locally without a Kubernetes cluster. For mission-critical, high-throughput enterprise infrastructure, Arize AX supports enterprise customers such as Uber, Booking.com, and Reddit. Ngo concludes that the future of LLM observability lies in automating engineers out of manual evaluation authoring, though teams must carefully balance evaluation scope against inference costs to keep non-deterministic agent testing practical.

Source quotes

[0:00] Dat Ngo: Because now we know that code doesn’t audit agents or harnesses. It’s actually the telemetry that does that. [12:50] Dat Ngo: We are very heavy believers that you shouldn't even have to choose your evals.

Original transcript

[0:00] Thank you. So really nice to meet you all. My name is Dat. I work at Arise AI, so I'll talk a little bit about what that is. A little bit about me and kind of what I want to share today is... I work very deeply in the space. I'm an AI architect. I work with a lot of the largest enterprises across the world to talk about, we work on things like observability, evaluation, experimentation, but really it's just How do you make AI work, right? So I do spend a lot of tokens, um, in this space. So this was last OpenAI Dev Day. I think I made it to probably somewhere between 100 billion and 1 trillion tokens last year. I do like to vibe it out. But I do know the space really, really well. We work with some of the world's largest companies and enterprises. So we get to see their transformation into this space. And really what I wanted to share today was like, what do I see in the industry? So I think we have a very unique vantage point, being the company that we are. So we get to see what every team is building, how they're building it, What are the biggest pains that they face? And really how they're trying to fix those things. So if I had to really distill down what we do in kind of a nutshell, It's really these three things Maybe by show of hands, like who's built agents, who's building agents, who's productionized agents, Who's building harnesses? and who has no idea what a harness is. Okay, we're all pretty cracked, so that's good. You know, I think it's really funny is that, like, the AI space, it really just feels like software reimagined. It's really the same set of patterns, just maybe a different flavor coming out. And it's really it feels like magic, but it's not magic and It's all just engineering. Really what we're gonna cover today is just three things. The first one is observability, which answers the question of like, What's happening in the thing that I've built? And what does that look like, whether it's a harness or an agent? Then we'll go into evals. Eviles is just simply, How do I drive signal from my systems in some form or fashion, right? And then as we talk about how we make improvements in this new non-deterministic world, you'll come to find out that when you make, you know, what you perceive as a fix, and you fix the thing that you thought you fixed, you might have actually produced like two or three regressions that you didn't really know about. And so, like in our world, we talk about observability, right? Everything that we do, something that we're super proud about is OTEL is a really strong pattern for those in the engineering space, but everything we do is through open telemetry. So it doesn't really matter you know, what particular type of you know, harness agent model setup that you have. The good news is that, you know, being hotel first, We're really prepped for a lot of these use cases, whether it's an auto instrumenter, if you know what that is, but basically you add one line of code. That one line of code will basically see what's happening around in that particular framework or SDK. create open telemetry traces and spans, and produce these views kind of here. So if you've ever seen a trace or a span, It's basically the audit record of what did my agent do? Because now we know that code doesn't audit agents or harnesses. It's actually the telemetry that does that. So traces is a big fundamental part of observability. Now, there are many other different parts of observability that you should be thinking out outside of traces and spans. You can think about sessions too. So I don't know if any of you read the anthropic paper, Managed Agents, that came out two days ago. Pretty awesome read. But Sessions is another one about like state. So what are the back and forth maybe conversations? What are the back and forth states that are happening between You can think runs. For a lot of people in the enterprise, they may want to end up running I realize this isn't the easiest to see, so let me change over to light mode. A lot of folks, sorry about that. Yeah. A lot of folks would like to understand like, hey, What is the back and forth conversations? So that will be something like a session. So it's like, hey, what are those back and forth conversations that are being had? Great. Now, people like to eval those. So in the enterprise, you'll see a lot of folks being like, I don't really care the deep level, the agent did this tool call, that tool call. They may not care about that as much as like, hey, Was the end user satisfied? Were all their questions kind of answered? Now, one unique thing, I'll change this over to light mode too. One unique thing about what we do here at Arise, this is Arise AX. [5:00] - Is that sometimes when you think about your agent, as a non-deterministic call, right? You want to be able to see, hey, what did my agent do, for example. There's different paths that your agent could take, so different branches. But what if you wanted to like look over all instantiations? of your agent and get a more distributional view of what's happening. These are kind of like views into like the distribution of your agent. So what are all the possible paths and branches, also loops. It allows you to answer questions like, what percentage of my traffic goes down one branch versus another. Was there a particular component in that particular branch that we took that caused a significant amount of latency? When we start to talk about agents or different pads, you may think about trajectory evals. So trajectory could be like, hey, I went down this one path and everything was really good. But for some reason when I go down this path, the evals, or the signal that I'm collecting, is dropping. Why? What's the root cause? oh, the root cause issue was that These two components are actually out of order. I did B before A, and actually B has a dependency on A. So actually, it turns out the way my LLM decided to call these things was mismatched, right? We need to put some context in there to say, hey, actually, before you do this, you need to do that. So there's many different views into observability. Of course, things like analytics aren't dead either. So what a lot of the folks in the enterprise end up looking at. They just want to build views on what they're what their agents look like in real time. And so being able to customize and build those views out Super, super helpful. And that's observability in a nutshell. It's like, can I see all the different layers? We have many, many different types of layers here. Um, You know, the next thing is like, okay, so I have observability. That's kind of step one. It's the same thing that happened in software, right? Now you have to determine signal. And so signal comes in actually many various forms as well. The way I like to break it down is kind of these five flavors of signal. I think everyone here in the room has heard of LLM as a judge. You know, it may seem like a simple concept, but in actuality It could actually get quite complex, and we'll kind of go through all of that. Now you can't forget about your humans. When you think about humans, whether it's the end users using your product, it's extremely valuable signal. So whether you're a product manager or someone technical or not technical, you do care about the signal. We've all heard of golden data sets. They're extremely valuable because it's If the third column here represents quality, You know, you trust the person who labeled this data because they know the domain. Then you'll run techniques like, hey, I'm going to run my LLM as a judge on some golden data set so that you can tune your LLM as a judge. You basically say, hey, can I get my LLM? to approximate this thing or this person or this data set that I trust. And then of course we're all thinking about costs as well. So when we think about costs, you don't always have to use an LLM call or even humans. Determinism is super nice, so think about logic or deterministic based evals. If I go from paragraph to JSON payload Does this JSON, is it a valid JSON? Does it have this schema? Does it have, you know, these fields that are non-null? And then of course, we're all building these things. for one of three purposes, I think. So the business metrics you care about are either some form of, like, how do I make more money? How do I save money? or how do I save time? Right? And so what you'll notice, as you start to build really good AI products, you'll start to have two types of personas that kind of end up coming together. So obviously you have your technical users, right? These are your AI engineers, your developers of the world. These people are extremely good at building and automating things. They're good at frameworking, But then you have folks who are maybe less technical, but they understand what the AI experience should be, right? These are like the subject matter experts, the product managers of the world. These folks end up You know, you want to relegate kind of the work of like, hey, this is how the prompt engineering should go. Here's the evals that I care about. Because you want people who can code coding, and you want people who know the domain to work in that domain. And so in our world, what that looks like is something like this. We allow folks to be able to run evals at... This is a non-technical way. Of course if you are technical you can attach your evals and run them kind of programmatically if you want. But in our world, we want to be able to say, hey, I want to be able to allow a user to be able to select their model, be able to run some out of the box template or customize Some eval here. - And when we talk about [10:00] - Complexity on the eval side, right? Imagine for a second that you have built some application, some agent, some harness, That harness has got components in it. They may be called deterministically or non-deterministically. Whatever. So evals can be run on, you can think, a single kind of component. We call that a spend kind of eval. Let me kind of come here. So the scope would be one single input and output. I'll pull up like a more complex view of this, but... Oops, let me close this. But you can think of the simple span input and output as, hey, I want to look at the input and output of one part of an LLM call. So most people understand that, and that's really, really simple. Now, um... We also have multi-span evals, so you can think of that as like, hey, In order to run the eval that I want, it actually requires data across many different components in the system. So if I wanna say, hey, how well are agents passing data back and forth to each other, Well, it turns out I need the data from every single agent and how they pass data. So that's a multi-span eval and it allows you to run more complexity. If you want to look over all of the spans in total, that's something like a trajectory eval. Did we call things in the right trajectory to finish the business process? And then there's that session level eval, right? It's like zooming out and saying, "Hey, what does the state machine Like if I want to evaluate that state machine of Hey, let me turn this light mode. Hey, in this conversation was the user ever frustrated? Did we answer all of their questions? So think of that as I want to evaluate the state machine. So as you're thinking about evals, it's not generally-- it's also like, hey, what flavor of evals do we want to run? But at what scope and depth? So you can get very granular, and then you can also zoom out. And just because you can't eval something doesn't mean you always should. It's not this exhaustive thing. You want to see like, hey, what are the minimal set of evals I can get away with to understand signal of like, is my application working as intended because there's a cost associated with this stuff, right? So TLDR, That's observability and evals in a nutshell. We'll talk about experimentation and improvement. So not everyone starts with traces. If you do start with traces, you can take them and do really cool things like, say like, "Hey, show me where some signal is you know, whatever, bad, where am I missing stuff, right? Then you can find those things, collect them up into a data set, Also, if you don't have traces, you can just upload a data set outright, input output pairs. And then from here, you can do things like grab that data set. for example, which is just rows and columns of data, then you can start to run experiments. Experiments can be changes. So as you think about how do I make things better, for my agent or harness, it's generally changes. Changes to prompts, changes to models, changes to orchestration, changes to configurations. Think that way. We allow folks to be able to test these things in a UI or programmatically. But, you know, you know, One thing I always like to share with our customers is like, where is the space going? What we quickly realized at Arise here is that like, most people don't want to live in dashboards or buttons or manual things. And we very much recognize this. As you think about where the future is going, software will compress, it's going to be easier to build, easier to customize. So everything I just showed you was just like the nice manual way to see it. But everything we've done, we've allowed you to be able to do this through your coding agent. We realized people are very comfortable with their cloud code and codexes. So we expose all the primitives via the CLI and a set of tools and skills. So that's kind of opinionated. And then there's also an AI system built into all of this too, meaning Cloud Code, your AI system can end up calling our system. So that you can do things like, if you don't want to just Figure out these things on your own We believe a lot of this stuff can be automated. So I can go here and ask Alex. Obviously, Cloud Code or something outside of the system can call Alex. And you can just simply say, hey, do you see-- any issues with my application. MR PRICE: Yeah. And because we have all the data, because we have the hooks and everything else, You know Alex will go in and plan and run these these tasks and So our ultimate goal as a company is actually to automate you out of this process. observability, evals, experimentation, and improvement. We think the whole flywheel is very much automatable, meaning it's not magic again, but it should feel like magic. So our main goal is like one day you work with Arise, and you pull the ecosystem down, And then it just works. We are very heavy believers that [15:00] You shouldn't even have to choose your evals. Thank you. An AI should have context of like, hey, here's the traces. Here's what's happening. Let me create evals on the fly and think about them for you. Or like, hey, something has changed. I know I need a new eval. But you'll notice Alex is already getting to work with like hey What's happening here? It looks like we have some high latency. We have, you know, some errors detected, things like that. Um, So in a nutshell, this is kind of where we're going and what we're after. And so as we think about this world, We actually have two products out today, so for more of the engineering first folks, we have Arise Phoenix, which is open source. The really nice thing about Phoenix is it's single... single container. You can deploy it locally, doesn't require a Kubernetes layer. And then for our largest enterprises, they use Arise AX. which is kind of generally reserved for some of the largest enterprises between Uber and booking and Reddit. You guys couldn't tell we love dark mode, so we probably should try to go light mode in some stuff. But yeah, in a nutshell, that's kind of what we do and who we are. If you guys want to chat about anything past that, super excited. Yeah, thank you very much for your time today. Thank you. *music*