ElevenLabs Pushes Live Diagnostics to Cut Voice Agent Errors
Ai4
Deploying voice agents to live traffic with quantitative benchmarks and human review resolves performance bottlenecks that synthetic testing and endless prompt tinkering fail to catch.
## Background
Published by the Ai4 channel on YouTube on August 21, 2026, this conference presentation features Amanda Milberg, an Enterprise Solution Engineer at ElevenLabs. According to the description, the session provides a practical framework for moving past aimless tinkering in agent development, guiding teams on how to know when an agent is "done enough" and why human review remains indispensable over full automation.
## TL;DR
Milberg argues that successful voice agents rely on structured human instruction and continuous iteration rather than black-box AI "magic." She breaks down the primary obstacles in agent development—unconstrained autonomy, shifting business realities, and elusive failure evaluation—and presents practical remedies anchored by retrieval and strict baseline metrics. To structure improvement, she introduces the "Define, Deploy, Diagnose" (DDD) framework, demonstrating how canary deployments and diagnostic tools like ElevenLabs Architect allow teams to review actionable configuration diffs. The session culminates with an ElevenLabs voice agent delivering the final takeaway: sustainable agent design is an iterative feedback loop with humans firmly in control.
## Atmosphere & style
The presentation is pragmatic, engineering-grounded, and refreshingly candid, stripping away AI conference hype in favor of operational reality. Milberg speaks with the direct, approachable demeanor of a forward-deployed engineer who has personally navigated messy production edge cases. An unexpected on-stage moment—when her cell phone rings live because a deployed test agent routes calls directly to her—injects authentic, relatable humor that reinforces her battle-tested credibility.
## Key takeaways
- [00:04](https://www.youtube.com/watch?v=PE6XNvO3tNA&t=4s) True business value from voice agents comes from deliberate human instruction and operational guidance rather than standalone AI capabilities.
- [00:04](https://www.youtube.com/watch?v=PE6XNvO3tNA&t=4s) Voice agent architecture requires synchronizing LLM reasoning, knowledge bases, and tools with speech orchestration layers like Voice Activity Detection, STT, TTS, and turn-taking.
- [05:23](https://www.youtube.com/watch?v=PE6XNvO3tNA&t=323s) Production voice agents face three core hurdles: balancing autonomy against control, tracking shifting real-world business policies, and diagnosing conversational failure root causes.
- [08:15](https://www.youtube.com/watch?v=PE6XNvO3tNA&t=495s) Autonomy should be scoped methodically by collaborating with internal teams to convert tacit institutional knowledge into concrete Standard Operating Procedures (SOPs).
- [08:15](https://www.youtube.com/watch?v=PE6XNvO3tNA&t=495s) Dynamic enterprise information should be integrated via live retrieval and external APIs rather than hardcoded into static system prompts.
- [08:15](https://www.youtube.com/watch?v=PE6XNvO3tNA&t=495s) Baseline performance metrics must be established before modifying prompts to avoid making unverified changes that cause regressions.
- [11:50](https://www.youtube.com/watch?v=PE6XNvO3tNA&t=710s) The Define, Deploy, Diagnose (DDD) framework establishes a continuous cycle to replace vibes-based agent evaluation with structured iteration.
- [11:50](https://www.youtube.com/watch?v=PE6XNvO3tNA&t=710s) Setting explicit quantitative targets upfront prevents endless tinkering by defining exactly when an agent is 'done enough'.
- [11:50](https://www.youtube.com/watch?v=PE6XNvO3tNA&t=710s) Routing a small portion of live traffic (such as 10% for 2–4 weeks) surfaces authentic conversational edge cases that synthetic testing suites fail to capture.
- [15:42](https://www.youtube.com/watch?v=PE6XNvO3tNA&t=942s) Automated diagnostic tools like ElevenLabs Architect evaluate call transcripts and agent configurations via natural language queries to pinpoint latency and quality bottlenecks.
- [15:42](https://www.youtube.com/watch?v=PE6XNvO3tNA&t=942s) Diagnostic workflows should incorporate Git-style diff interfaces so human operators can inspect, refine, or reject automated recommendations before deployment.
- [17:24](https://www.youtube.com/watch?v=PE6XNvO3tNA&t=1044s) Full automation is not the final objective; ongoing human review is necessary to verify that agent iterations genuinely improve performance rather than simply running continuously.
## Detailed walkthrough
### [00:04](https://www.youtube.com/watch?v=PE6XNvO3tNA&t=4s) Introduction: The Role of Humans in Agent Building
Amanda Milberg introduces herself as a Solutions Engineer on the forward-deployed engineering team at ElevenLabs. Acknowledging the conference theme where AI often feels like "magic," she argues that true business value from voice agents does not come from magic, but from the human beings who instruct and guide the agent. ElevenLabs is a research-first AI company recognized for frontier models across speech, music, and audio, and provides an enterprise platform called Eleven Agents to help organizations build, test, deploy, and monitor voice agents in production.
Milberg defines a voice agent as an AI system that conducts live spoken conversations with a person by understanding what they say and responding in natural speech to complete a task. To contrast this with legacy interactive voice response (IVR) phone trees, she presents the example of a customer calling to reschedule a delivery: the voice agent listens to the request, queries the live delivery database in real time, offers reschedule windows, and confirms the new appointment—all without human agent intervention. Technically, this combines a traditional LLM agent architecture (system prompt, agentic tools, knowledge bases/SOPs) with ElevenLabs' speech engine orchestration (speech-to-text, text-to-speech, voice activity detection, and turn taking).
[05:23](https://www.youtube.com/watch?v=PE6XNvO3tNA&t=323s) The Three Core Challenges in Building Agents
Building production-ready voice agents presents unique difficulties, which Milberg groups into three primary friction points:
1. **Agency cuts both ways:** The ability for an agent to reason autonomously makes conversations fluid and adaptable, but that same autonomy allows it to stray off-script, invoke tools unexpectedly, or resolve customer issues in unintended ways. Teams must deliberately balance autonomy against step-by-step restrictions.
2. **Reality keeps moving:** Enterprise policies, documentation, and internal workflows shift continuously. Agents risk operating on outdated guidance unless memory structures and underlying data integrations are continuously kept fresh.
3. **Evaluation is hard:** While identifying an outright failure is straightforward, pinpointing the root cause is difficult. Failures can stem from conflicting system prompt instructions, stale knowledge retrieval, or an inappropriate vocal tone that frustrates the user despite technically accurate answers.
[08:15](https://www.youtube.com/watch?v=PE6XNvO3tNA&t=495s) Three Practical Mitigation Strategies
To counter these core challenges, Milberg provides three practical countermeasures:
- **Scope autonomy deliberately:** Treat agent onboarding like human onboarding. Instead of expecting a "two-day immersive onboarding program in Florida," collaborate directly with internal enablement and contact center teams to translate tacit operational knowledge into clear Standard Operating Procedures (SOPs) delineating allowed and prohibited behaviors.
- **Prioritize retrieval over memorization:** Avoid hardcoding company knowledge directly into prompt text. Ground the agent in live database queries and connected knowledge bases, while maintaining a scheduled cadence outside the prompt to purge stale source documentation.
- **Establish baseline metrics before changing prompts:** Pushing prompt fixes without a benchmark makes it impossible to know whether changes represent progress or "one step forward and two steps back." Quantitative metrics must be tracked continuously over time.
[11:50](https://www.youtube.com/watch?v=PE6XNvO3tNA&t=710s) The "Define, Deploy, Diagnose" (DDD) Framework
To give teams an actionable workflow, Milberg introduces the "Triple D" framework: **Define**, **Deploy**, and **Diagnose**.
```mermaid
flowchart LR
accTitle: Define Deploy Diagnose Iteration Loop
subgraph Loop["The Iteration Loop"]
direction LR
D1["1. DEFINE<br/>• North-star metrics<br/>• Guardrails & SOPs<br/>• Reporting setup"]
D2["2. DEPLOY<br/>• Low-risk live traffic (e.g. 10%)<br/>• Run 2–4 weeks<br/>• Surface real edge cases"]
D3["3. DIAGNOSE<br/>• Analyze failures<br/>• Expand agent scope<br/>• Update success criteria"]
D1 --> D2
D2 --> D3
D3 --> D1
end
```
- **Define:** Explicitly establish quantitative success criteria upfront to prevent "vibes-based" evaluation. Rather than vague goals like "make the agent better at support," set clear targets: *"Reduce 70% of Tier-1 support tickets with less than 5% negative sentiment detected across 500 live calls."*
- **Deploy:** Deploy early to capture real conversational edge cases that synthetic test suites miss. A standard deployment strategy routes 10% of production traffic to the voice agent for 2 to 4 weeks—statistically significant for data collection while limiting blast radius for minor bugs.
- **Diagnose:** Analyze what worked and what failed to set the criteria for the next iteration loop. This might involve expanding agent autonomy (such as granting refund processing capabilities instead of pure routing) or shifting optimization targets (such as optimizing for first-contact resolution rather than basic deflection).
During the discussion on live deployments, Milberg's phone rings on stage—revealing that the live production transfer number for an agent currently being tested is routed to her personal cell phone.
[15:42](https://www.youtube.com/watch?v=PE6XNvO3tNA&t=942s) ElevenLabs Architect: Automated Diagnostics with Human-in-the-Loop
ElevenLabs built this iterative diagnostic workflow directly into their platform through a feature called **Architect**. Instead of manually parsing thousands of call transcripts, teams can query Architect in natural language (e.g., *"Tell me why my agent is slow"*).
Architect evaluates the entire agent JSON configuration alongside workspace conversation logs to pinpoint latency bottlenecks and surface recommendations, such as switching to a lighter LLM or adjusting the text-to-speech model. Changes can be inspected through a Git-like diff interface where human operators approve, modify, or reject recommendations before deploying updates into production.
[17:24](https://www.youtube.com/watch?v=PE6XNvO3tNA&t=1044s) Summary Delivered by an ElevenLabs Voice Agent
To conclude the presentation, Milberg has an ElevenLabs voice agent deliver the four main takeaways:
1. **Define success criteria:** Vague goals create endless iteration; know what "done" looks like to prevent aimless tinkering.
2. **Run the Define-Deploy-Diagnose loop:** Iteration is an ongoing cycle feeding diagnostic data back into future deployments.
3. **Use self-improving tooling:** Use tools like Architect to cluster failure modes across conversations and triage with evidence rather than guesswork.
4. **Keep humans in the loop:** Automation provides scale, but human review confirms whether an update genuinely improved performance.
> [08:15](https://www.youtube.com/watch?v=PE6XNvO3tNA&t=495s) one step forward and two steps back — Amanda Milberg
> [11:50](https://www.youtube.com/watch?v=PE6XNvO3tNA&t=710s) Reduce 70% of Tier-1 support tickets with less than 5% negative sentiment detected across 500 live calls. — Amanda Milberg
> [15:42](https://www.youtube.com/watch?v=PE6XNvO3tNA&t=942s) Tell me why my agent is slow — Amanda Milberg
## Video description
Amanda Milberg - Enterprise Solution Engineer, ElevenLabs
Agent building is a constant process of iteration, which raises two questions most teams don't have good answers to: how do you know when to stop, and how do you know what to improve next? In this talk, we'll share practical ways to identify what actually needs improving in an agent and how to prioritize those changes, plus techniques for building self-improving agents that can surface their own weak points. We'll also make the case for why full automation isn't the end goal, and where human review is what actually tells you the iteration is working, not just running. Attendees will leave with a framework for knowing when their agent is "done enough" and a concrete process for deciding what to fix next. If you've ever wondered whether you're improving your agent or just tinkering with it, this talk will give you the answer.
Amanda Milberg of ElevenLabs argues that enterprise voice agents deliver real business value not through autonomous AI breakthroughs, but through continuous human instruction, structured operational guardrails, and disciplined iteration.
Voice Agent Architecture and Human Direction (0:04)
A voice agent conducts live, bidirectional spoken conversations to complete tasks, distinguishing itself from legacy interactive voice response (IVR) phone trees through dynamic execution—such as querying a live database in real time to reschedule a delivery appointment without human intervention. Technically, this system integrates an LLM reasoning core (system prompts, agentic tools, knowledge bases, and standard operating procedures) with speech orchestration covering speech-to-text, text-to-speech, voice activity detection, and turn-taking.
Milberg notes that despite perceptions of AI as self-contained or magical, voice agents depend directly on human operators for direction. Without deliberate human configuration and monitoring, systems lack the context needed to navigate organizational workflows and handle complex customer interactions properly.
Core Friction Points in Production Deployments (5:23)
Building production-grade voice agents introduces three primary challenges. First, autonomous reasoning introduces unpredictability: the flexibility that enables natural dialogue also allows agents to veer off-script, invoke tools inappropriately, or resolve inquiries in unapproved ways. Teams must balance open-ended autonomy against strict operational constraints.
Second, enterprise realities shift constantly as policies, documentation, and workflows update, threatening to leave agents relying on stale data. Third, failure evaluation is complex; pinpointing whether an error stems from conflicting prompt instructions, outdated knowledge retrieval, or an inappropriate vocal tone that frustrates callers requires deeper analysis than merely detecting that a call went poorly.
Mitigation Through the Define, Deploy, Diagnose Loop (8:15)
To counter these operational risks, Milberg recommends treating agent onboarding like employee enablement: translating tacit knowledge into explicit Standard Operating Procedures (SOPs) and prioritizing live database retrieval over hardcoded prompt memorization. Prompt changes should never occur without established quantitative baselines to prevent untracked regressions.
Teams apply these safeguards through the 'Define, Deploy, Diagnose' (DDD) framework. Operators first define concrete, non-subjective metrics (such as deflecting 70% of Tier-1 support tickets with less than 5% negative sentiment across 500 calls). Next, they deploy early to a low-risk slice of live traffic—typically routing 10% of calls over two to four weeks—to capture real-world edge cases. Finally, operators diagnose failure modes to recalibrate success metrics or gradually expand permissions, such as enabling direct refund processing.
Automated Diagnostics and Human Verification (15:42)
To accelerate root-cause analysis across thousands of call transcripts, ElevenLabs developed Architect, an internal diagnostic tool. Operators query the tool in natural language to identify performance issues, such as high latency. Architect inspects the agent's JSON configuration and conversation history to recommend specific adjustments, such as switching to a lighter LLM or altering the speech synthesis model.
Rather than applying automatic updates, the system presents changes through a Git-like visual diff interface. Human operators inspect, alter, or reject the proposed modifications before pushing them to production, ensuring that diagnostic automation assists analysis without removing human accountability.
Milberg concludes that sustainable voice agent performance relies on treating deployment as an ongoing learning cycle rather than a single release event. While automated tooling can pinpoint failure patterns across large call volumes, human operators remain essential for setting boundaries, defining success, and verifying that configuration updates actually improve customer outcomes.
[5:23] Amanda Milberg: Agency cuts both ways
[5:23] Amanda Milberg: Reality keeps moving
[5:23] Amanda Milberg: Evaluation is hard
[8:15] Amanda Milberg: Scope autonomy deliberately
[8:15] Amanda Milberg: two-day immersive onboarding program in Florida
[8:15] Amanda Milberg: Prioritize retrieval over memorization
[8:15] Amanda Milberg: Establish baseline metrics before changing prompts
[8:15] Amanda Milberg: one step forward and two steps back
[11:50] Amanda Milberg: Reduce 70% of Tier-1 support tickets with less than 5% negative sentiment detected across 500 live calls.
[15:42] Amanda Milberg: Tell me why my agent is slow
Original transcript
[0:04] And now at the stage we have Amanda presenting
to you the loop, not the launch. Building agents
[0:12] that get better with you. Put your
hands together for her. [applause]
[0:20] Thank you everyone. And obviously I had to
test the audio. I mean we're an audio company.
[0:25] What kind of demo would it be today if I didn't
play some voices for you? Um, but thanks for
[0:31] having me. Um, my name is Amanda Milberg. I'm a
solutions engineer at 11 Labs. And today I'm going
[0:38] to talk about building agents that get better with
you. And I think that this is a really interesting
[0:45] topic and a theme that we even heard in the
keynote this morning. AI often feels like magic,
[0:52] right? And in a lot of ways, it truly is. But I
want to make the case that building a successful
[0:58] agent, one that delivers genuine business value,
comes down to the people that instruct it what
[1:05] to do. So, what are we going to talk about today?
Well, I'm going to start by walking you through a
[1:12] quick background of what 11 Labs is if you haven't
heard of us before. Um, and then kind of settle on
[1:18] a definition of voice agents. Then we're going
to talk about why agent building is hard, right?
[1:24] I'm sure we've all built agents. What are the
common challenges that we see and then how can
[1:28] we mitigate those challenges? Then I'm going
to walk through a framework that we use with
[1:33] our customers internally to build effective
agents. How do we define them correctly and
[1:38] then launch them into production successfully. Um
focusing as I mentioned about how the people are
[1:44] the key factor here of making this work. And then
I'll talk a little bit about new features that we
[1:50] actually have in the product um in 11 labs for
building these self-improving agent loops while
[1:56] keeping the human in the loop as part of that.
And then finally we'll end with a little wrap-up
[2:02] and if we have any questions um I'll be able to
answer them live or after. So to start with a
[2:10] little background about what is 11 labs. Um we
are a research first AI company most known for
[2:17] building frontier models across speech, music and
audio and we deliver them in enterprise platforms
[2:25] so folks and companies like you see on the screen
can build real world workflows. And we're most
[2:33] known for having the most humanlike voices on the
market. And like I mentioned having a platform
[2:39] called 11 agents um where we help companies build,
test, deploy and monitor agents in production.
[2:47] And what I do as part of our forward deployed
engineering team here is help those c customers
[2:53] build their first of hopefully many voice agents
that they launch in the production. So it's
[2:58] safe to say I've seen firsthand what iteration
tactics work and which ones don't. And that's
[3:03] the information that I'm going to share with you
today. And so now that you know a little bit about
[3:08] what 11 Labs is and what 11 agents are, let's talk
about what is a voice agent. So I'm going to say
[3:17] with confidence that I'm not the first person that
has talked about agents today at this conference
[3:23] and I probably won't be the last, but I always
find it helpful to start with the definition
[3:28] uh that we can all settle to make sure that we're
agreeing on the same term here when we talk about
[3:32] a voice agent. out. I'm defining that right now as
a voice agent is an AI system that conducts live
[3:39] spoken conversation with a person, understanding
what they say and then responding in natural
[3:45] speech to complete a task. And let's illustrate
this point with an example. So, let's imagine a
[3:52] customer calls in um to reschedule a delivery and
instead of a phone tree or traditional IVR system,
[4:00] that voice agent will listen to their request,
maybe it checks the delivery system in real time,
[4:06] um offers available windows to reschedule that
delivery and then confirms the new date without
[4:13] potentially even a human being a part of that
call workflow. And so to get into kind of like
[4:20] the technical details of what this looks
like is if you've built an agent before,
[4:24] you're probably familiar with this concept
of you have an LLM, you have a system prompt,
[4:30] um you have tools that you give that LLM access
to to become agentic or to take action, maybe
[4:35] you have knowledge bases or SOPs that you put into
this agent. So we take that agent architecture and
[4:41] we unite it with what we call our speech engine.
And a speech engine has a it's an orchestration
[4:47] of various models like text to speech, speech to
text, voice activity detection, turn taking. And
[4:53] putting those two concepts together, we we get
something really powerful, right? Because us as
[4:58] humans, we like to communicate in a modality
that we prefer, voice, right? That's how I'm
[5:03] communicating to you right now. I'm not giving
this presentation chatbot style. And so when
[5:09] we have the ability of of doing something like
that, we it really makes things super powerful,
[5:15] right? And so now that we understand what a voice
agent is, let's take this example of building a
[5:23] customer success agent and talk about why are
building agents hard? And so I've boiled this
[5:31] down to to three things that we'll go through. Um
but you know there's many more but this is kind
[5:37] of just what what I constantly encounter as as the
as the tough thing. So the first concept is agency
[5:45] cuts both ways right when we build an agentic
workflow we give this agent the ability to reason
[5:51] and act on its own. And that's the exact value of
why we like agents right we're using this agentic
[5:58] approach over this rigid call structure. It sounds
more natural. The agent is able to pivot. It can
[6:06] reason effectively and giving its knowledge come
up with an answer. But that same autonomy means
[6:13] that it can also decide to do things that maybe we
didn't intend for it to do. It can go off script
[6:19] mid-con conversation. It can unexpectedly call
a tool that we didn't want it to. It could maybe
[6:24] solve a problem in a way that is technically
right, but not in the way that we imagined.
[6:29] And so part of building this agent is deciding how
much room we want to give it and then when do we
[6:35] want to restrict it down to a more concrete
step-by-step action. The second thing that
[6:41] I notice is that reality keeps moving. I mean
think of all of us in our day-to-day jobs today.
[6:46] How much we have to just keep up with policies
are changing documentation is changing. And so
[6:52] we must think about how we can design our agents
and its underlying data sources. That way, we're
[6:58] connecting to the most live and up-to-date data.
Our agent is only as powerful as the information
[7:05] that we give it. And so, we want to think about
how we can easily add memories to our agent or
[7:10] make sure it's constantly being updated with the
most up-to-date and relevant information. And then
[7:17] the last thing that I find of why building
agents really hard, especially voice agent,
[7:23] is that evaluation is hard. Right at the
surface, it's easy to spot a bad answer,
[7:31] but we it's harder to triage what exactly what
went what went wrong in that conversation. So,
[7:38] I'll give you some examples. Did the agent
misspeak based on conflicting guidance in
[7:42] the prompt? Did the agent pull stale data to
answer a question where it then provided the
[7:49] wrong guidance? Did the agent simply deliver
the message correctly but in a tone that wasn't
[7:57] really right that made the user frustrated? And so
these are all reasons why I find sometimes agent
[8:04] building is hard. Um and like I said only some of
them but I wanted to call out these as the ones
[8:10] that I see coming up the most often. Um but luck
luckily we've developed some general strategies
[8:15] to to mitigate uh these challenges. So the first
is scope the autonomy deliberately. This is the
[8:25] integral part of kind of why I named this session.
It starts with with you, right? This is where you
[8:30] come in. You have to design your agent to act like
you would in a conversation or on a call. You have
[8:38] to train them just like you train your internal
employees. And what this often means is pra in
[8:44] practice is sitting down with your enablement team
with your call center agents um with folks that
[8:51] are doing this day-to-day and defining a clear
standard operating procedure or SOPs of what
[8:56] this agent can do and how what this agent should
behave and then what the agent cannot do, right?
[9:03] And luckily, you know, training these agents isn't
the same as what we're used to of like a two-day
[9:08] immersive onboarding program in Florida
or something. But if we have these SOPs,
[9:13] we can give that information to the agent and we
can make scope that autonomy, deliberately to say
[9:18] you can do this and you can't do that. The second
thing that we should keep in mind um to work
[9:25] through these challenges is prioritize retrieval
over memorization. So connect your agents to live
[9:32] data sources instead of having that source of
truth baked into the prompt. A prompt is a great
[9:39] way to give guidance to the agent on how it should
perform, what it should do and shouldn't do, but
[9:45] you need to find a way that your agent is always
up to date with the latest information, right?
[9:50] The agent is only as good as the information that
you supply it. And so this includes things like
[9:56] connecting to databases, connecting to live
knowledge basis sources, but it also conclues
[10:03] having a structure of how do I refresh that
knowledgebased cadence in general? How do I
[10:08] get rid of the stale data that is even outside of
the agent design itself? And then if you do need
[10:14] to make these prompt changes, how do you do so in
a way that you keep the human in the loop and make
[10:20] it really easy for your business team to change,
update that prompt, test to see if it worked,
[10:26] etc. And then lastly, when we're talking about
how evaluation is hard, I think what makes it
[10:34] even harder is when you don't have a metric to
test against. So let's say we've identified the
[10:40] fix. Now, we're going to push a fix, but we need a
baseline metric to prove that it actually worked,
[10:47] not just that something changed. And this means
that you need to define the key metrics that
[10:53] you care about in this agent build. And track
them over time. You need that ground truth to
[11:01] isolate whether this change was one step forward
and not one step forward and two steps back.
[11:09] And so three problems, three solutions, but I hope
you notice the common theme that kind of unites
[11:17] all of these is that you have to be a part of this
definition. And I want you to almost think about
[11:25] any challenging business problem that you faced,
right? Was it easier to face that challenge when
[11:31] you had a definition or an metric that you were
optimizing for in the beginning? Right? I I feel
[11:37] like this isn't new to agent design in general.
This is why we have product documents, scope,
[11:42] etc. So now that we understand kind of this common
problem, um let's talk about the framework that I
[11:50] use to put this in practice. So I wanted to
kind of make it something easy to remember.
[11:56] And so I came up with like triple D, the agent
version of not Diners, Drive-ins, and Dives,
[12:01] but define, deploy, and diagnose. Right? So the
first thing that I want you to do is define your
[12:08] success criteria and specifically define what does
good look like. If you have a northstar metric
[12:16] you're optimizing for it's a forcing function to
align everything else your guard rails your data
[12:22] sources how you scope the autonomy to achieving
that and this criteria should be specific. Okay,
[12:30] so bad criteria, make the agent better at
handling support tickets. Good criteria,
[12:37] reduce 70% of tier one tickets. Make sure
that those tickets have less than 5% negative
[12:45] sentiment detected in the call for 500 live
calls scenario type of thing. And then once
[12:54] you've aligned on that criteria, make sure you
have reporting in place so that it's really easy
[13:00] to track that criteria. In general, you want
to have something reporting that's robust,
[13:07] not just a gut check. The less vibes based, as we
say, the definition of your success criteria is,
[13:14] the easier to make sure that you're not building
to a moving target. And so I generally feel like
[13:19] if you take one thing away from this conversation,
it's that you should align on what you want to
[13:25] build and how you build it before you even start
building it. Now the second thing I want to say
[13:31] is is don't be afraid to deploy early. Obviously
I want to make sure that you have a robust test
[13:37] harness that's testing edge criteria that you know
and that you are aware of. But I also want to tell
[13:44] you that live conversations, especially in the
voice AI space, are going to bring edge cases
[13:48] that you never even thought existed. It's part of
the fun. And so I want you to define a deployment
[13:54] plan that gives you enough signal to validate that
success criteria. Okay? So a deployment plan might
[14:01] look something like route 10% of traffic to an 11
agent as an example for 2 to four weeks. Right?
[14:09] That's large enough traffic to be statistically
significant with a small enough population where
[14:16] if you have some small glitches along the way,
which you will, um, it won't cost you much,
[14:21] right? This deployment is meant to be a learning
experience. It's meant to be iterative. And then
[14:27] the last phase is diagnose. See what worked in
that B1 deployment and then see what didn't. Those
[14:35] learnings define your updated criteria that you'll
want to bring in and reloop to the next phase of
[14:41] iteration. This might mean letting your agent
handle more of the workflow. So instead of maybe
[14:47] just being that intelligent router, we're now
processing refunds via tools. This may be changing
[14:53] your optimization metric from something like
optimizing for resolution instead of deflection.
[15:00] Whatever it is, diagnose doesn't just necessarily
in that phase, you're not just finding out what's
[15:06] broken, but you're defining what good looks like
for the next loop. And so this is the loop that
[15:12] we use to set our customers up for success.
Someone is calling me. Sorry. [laughter] Um,
[15:21] it's actually pretty funny. I'm we're testing an
agent right now in production and the transfer
[15:26] number is my number just to see that it works.
So I get calls like every day from this customer.
[15:30] Cool. [gasps] So this is the loop that we use to
set our customers up for success. I'm not lying.
[15:36] This is what I use. Um and so whether you're
building in 11 agents or not, um I encourage
[15:42] you to use this framework. But if you are building
in 11 agents, I have some good news for you. So
[15:49] we introduced this concept called architect. Um,
and essentially what architect is is it has all
[15:56] of these concepts built into the platform.
So for example, I can just ask architect,
[16:01] sorry it's a little small, tell me why my agent
is slow and architect has awareness of this entire
[16:08] agent JSON config to provide me recommendations
specific to my use case. So in my example, it says
[16:16] maybe try changing the text to speech model, maybe
tweaking the LLM to something a little smaller,
[16:21] less heavyweight. And the nice thing is is that
I can work in real time human in the loop to
[16:27] tell architect to make these changes. Um, in my
case, you know, change the LLM to a smaller LLM.
[16:33] Let's see how this works. Um, change some other
parameters. And architect will actually propose
[16:38] that fix and let me confirm or deny it. And
once I approve it, once everything looks good,
[16:47] there we go. Um, I can kind of review the
changes in this gitlike version, select improve,
[16:53] and continue to iterate. And what's nice about
architect is that you can use it to analyze your
[16:58] entire agent configuration. You can use it to zoom
into specific conversation points. And so at the
[17:05] end of the day, um, what architect can do is it
can help surface these weak points at scale. So
[17:11] I can only look through so many conversations at
once. Architects can analyze every conversation in
[17:15] my workspace and help define a solution. So with
that um we can go up we can go and quickly wrap up
[17:24] but I thought as part of the fun I won't wrap up.
I'll let my little agent pal wrap us up for us.
[17:30] So let's hear [clears throat] Tom four things I
want you to include. One, define success criteria
[17:39] for your goods. Debate goals create endless
depression. If you don't know what good looks
[17:44] like, well, that don't know when you're done.
It'll just keep tinkering. Two, run the define,
[17:50] deploy, diagnose. This isn't a one-time note.
It's a cycle. You define your criteria. You
[17:55] deploy with the right guard connections.
You diagnose what worked, what didn't,
[18:00] and then that diagnosis being straight back into
the shepherd of time. Three, use self reviewing
[18:06] tools. Let architect define big twins faster.
You don't have to read every transfer by hand to
[18:12] figure out where a is going down. Let the tooling
cost to failures and you win the likely cause. So
[18:17] you're triaging with evidence instead of action.
And four, keep your humility. Review is what
[18:23] actually tells you you're improvement, not just
running. Automation is succeeding scale with the
[18:28] human review step that confirms the fix actually
worked. That's the whole talk in Coral Banks.
[18:33] Define it on the loop. Use your tools, control
the automation, and place it down all at once.
[18:41] Hey, did I do a good job or what? [laughter]
So, that's the end of of the presentation. Um,
[18:48] that was obviously one of our voice
agents speaking speaking for us. But,
[18:52] um, if this sounds if this sounds interesting,
[18:55] if this sounds something like you want to
talk further about, you have some use cases,
[18:59] come talk to us. I'll be around. We're at booth
507 and we would love to chat with you. Thank you.
