Delight AI Unveils Trust Stack Framework to Prevent Agent Errors

The Trust Stack: How to Design an AI Agent That Earns Its Autonomy With Delig — Ai4

Ai4

Enterprise AI agents can safely earn self-improvement autonomy only after organizations establish baseline observability, prompt version control, and combined human-automated oversight.

AI-assisted summary of The Trust Stack: How to Design an AI Agent That Earns Its Autonomy With Delig — original source. Check the original for full context.

Source published:

Detailed notes

## Background Published on YouTube by Ai4 on August 18, 2026, this presentation features Kibeom Lee, Head of AI/ML at Delig. According to the page description, the talk introduces the 'Trust Stack'—a practical, bottom-up architectural framework designed to build reliable AI agents by making them scoped, observable, and correctable before enabling self-improvement loops in production.

## TL;DR Kibeom Lee examines high-profile public AI failures and silent operational errors to show why customer-facing AI agents require more than just model intelligence to succeed. He introduces the Trust Stack, a four-layer systems engineering model comprising Observability, Control, Oversight, and Self-Improvement. By implementing rigorous software engineering practices—such as step-by-step activity tracing, side-by-side prompt diffing, dual human/automated evaluations, and governed Zero-Touch improvement loops—organizations can safely transition agents up a multi-tiered Ladder of Autonomy while mitigating risk through empirical testing.

## Atmosphere & style The presentation maintains a pragmatic, authoritative, and deeply analytical tone, grounding theoretical concepts in real-world engineering challenges. Kibeom Lee speaks with an honest, sober perspective on industry hype, creating an instructive and structured environment that stresses accountability, safety, and system rigor. Listening to the talk feels like an insightful masterclass in software reliability, taking the audience on a deliberate journey from high-stakes cautionary failures to methodical, trust-earning architectural solutions.

## Key takeaways - [00:04](https://www.youtube.com/watch?v=mMhQs6MbFoY&t=4s) When a customer service AI agent fails, the customer blames the brand, making AI safety and correctness a critical business responsibility rather than a vendor issue. - [00:04](https://www.youtube.com/watch?v=mMhQs6MbFoY&t=4s) Public-facing AI agents inevitably encounter non-standard, curious, or malicious user behavior, requiring strong boundary controls beyond simple 'happy path' assumptions. - [00:04](https://www.youtube.com/watch?v=mMhQs6MbFoY&t=4s) AI chatbot responses generated via official company channels function as legally binding commitments, making unverified answers a source of financial liability. - [03:13](https://www.youtube.com/watch?v=mMhQs6MbFoY&t=193s) The most dangerous AI failures are silent failures—where traditional metrics like CSAT look successful despite unauthorized refunds, wrong policy quotes, or unearned perks. - [03:13](https://www.youtube.com/watch?v=mMhQs6MbFoY&t=193s) AI operational failures fall into five distinct categories—Hallucination, Data Disclosure, Unauthorized Actions, Prompt Injection, and Missed Escalations—each requiring unique system safeguards. - [06:17](https://www.youtube.com/watch?v=mMhQs6MbFoY&t=377s) Building reliable AI agents is a systems engineering challenge centered around control and architecture rather than just model performance or LLM fine-tuning. - [06:17](https://www.youtube.com/watch?v=mMhQs6MbFoY&t=377s) The Trust Stack establishes four sequential, interdependent layers required for reliable AI: Observability, Control, Oversight, and Self-Improvement. - [07:51](https://www.youtube.com/watch?v=mMhQs6MbFoY&t=471s) Observability requires conversation benchmarking for non-deterministic LLM behavior and granular execution tracing through step-by-step activity logs. - [09:37](https://www.youtube.com/watch?v=mMhQs6MbFoY&t=577s) Agent instructions and prompt configurations must be managed with software engineering rigor, including side-by-side version diffing, staging isolation, RBAC, and instant rollbacks. - [11:05](https://www.youtube.com/watch?v=mMhQs6MbFoY&t=665s) Effective oversight combines human qualitative review for subjective tone with 100% automated evaluation coverage to instantly flag policy violations or security risks. - [11:05](https://www.youtube.com/watch?v=mMhQs6MbFoY&t=665s) System kill switches are necessary safety controls to immediately pause anomalous or unsafe agent behavior in real time. - [13:09](https://www.youtube.com/watch?v=mMhQs6MbFoY&t=789s) Zero-Touch Improvement allows agents to autonomously analyze failure logs, formulate fixes, benchmark against past failures, and deploy improvements within set guardrails. - [16:04](https://www.youtube.com/watch?v=mMhQs6MbFoY&t=964s) Autonomy is not binary; it is granted incrementally across four levels (See It, Approve It, Steer It, and Get Better at Getting Better) based on risk and empirical evidence. - [16:04](https://www.youtube.com/watch?v=mMhQs6MbFoY&t=964s) An 8-question checklist evaluates organizational readiness for AI autonomy based on observability, control, safety guardrails, and rollback capability.

## Detailed walkthrough Here is a detailed, faithful walkthrough of the talk based on the transcript provided:

### [00:04](https://www.youtube.com/watch?v=mMhQs6MbFoY&t=4s) Introduction and the Anatomy of Public AI Failures Kibeom Lee, Head of AI/ML at Delight AI (Delig), introduces the core challenge of building customer-facing AI agents on behalf of enterprise clients. When an AI agent makes a mistake, the customer does not see the underlying model or the software vendor—they see the brand whose name is on the screen. To illustrate how trust breaks down in production, Lee breaks down three high-profile industry incidents:

1. **Gap's Shopping Assistant (Adversarial Exploitation):** Gap launched an AI shopping assistant, but within days, users deliberately pushed it into conversations completely unrelated to shopping or the brand. While the vendor blamed "bad actors," Lee notes that public-facing agents will inevitably encounter curious, careless, or malicious users who actively test boundaries rather than follow a "happy path." 2. **Air Canada (Invented Refund Policy):** Without any attempt at a jailbreak, an Air Canada chatbot confidently hallucinated a non-existent bereavement refund policy. The customer relied on this advice, leading to a legal tribunal where the airline was held legally responsible for its chatbot's promise. When an agent speaks for a company, its words become legally binding business commitments, requiring strict verifiability of sources and permissions. 3. **Kustomer (Invented Technical Explanation):** Kustomer—a company with deep AI expertise—had an agent tell users that a login issue was due to a non-existent "one-device policy." The agent fabricated an explanation for a product bug, causing confused customers to cancel their accounts over a policy that never existed. This proves that raw AI expertise is insufficient unless the agent knows its own limitations and knows when to escalate.

- "When your agent is quietly wrong, who finds out first? Is it you or your customer?" - Public agents must account for non-standard user inputs rather than assuming users will stay on the "happy path." - Answers delivered by official channels act as authoritative corporate promises and carry real legal and financial risk.

### [03:13](https://www.youtube.com/watch?v=mMhQs6MbFoY&t=193s) The Danger of Silent Failures and the 5 Failure Categories The most dangerous AI failures rarely make news headlines because they are disguised as successes under traditional surface-level customer service metrics. Lee presents three internal operational examples: an unauthorized refund, an agent quoting a 90-day return window instead of the correct 60-day window, and an agent processing an unearned refund just to appease an angry customer. On paper, all three tickets were marked as "resolved," CSAT scores were high, and zero escalations to human agents occurred. Traditional key performance indicators (KPIs) make failure look identical to success.

To address these silent failures, Lee categorizes them into five distinct operational failure modes: 1. **Hallucination:** Presenting fabricated information as fact (e.g., inventing a 90-day return policy). 2. **Data Disclosure:** Leaking sensitive personal identifiable information (PII) or confidential internal policies. 3. **Unauthorized Actions:** Executing unapproved operations, such as approving an illegal $2,100 refund. 4. **Prompt Injection:** Manipulating the model to bypass safety constraints or corporate rules. 5. **Missed Escalations:** Failing to hand off complex or high-friction conversations to a human representative, opting instead to guess or issue unapproved perks.

Because each failure mode requires a fundamentally different system check (e.g., hallucination detectors cannot block unauthorized refunds), trust cannot be reduced to a single quality score.

``` +-------------------------------------------------+ | FIVE SILENT FAILURE CATEGORIES | +-------------------------------------------------+ | 1. Hallucination | 4. Prompt Injection | | 2. Data Disclosure | 5. Missed Escalation | | 3. Unauthorized Action| | +-------------------------------------------------+ ```

### [06:17](https://www.youtube.com/watch?v=mMhQs6MbFoY&t=377s) Defining the Trust Stack Because failure modes are structural rather than pure LLM flaws, building reliable AI is a systems engineering problem rather than a model training issue. Lee introduces **The Trust Stack**, a bottom-up architectural framework composed of four interdependent layers:

1. **Observability:** Detecting issues and reconstructing exact production behavior in an experimental environment. 2. **Control:** Defining exact boundaries and permissions for what agents and operators are allowed to alter. 3. **Oversight:** Combining human and automated evaluation to review agent behavior and intervene when necessary. 4. **Self-Improvement:** Enabling the agent to learn from production failures and optimize itself within safety controls.

``` +-------------------------------------------------+ | THE TRUST STACK | +-------------------------------------------------+ | Layer 4: Self-Improvement (Safe Automated Loops)| | Layer 3: Oversight (Human + Automated Review) | | Layer 2: Control (Versioning, RBAC, Staging) | | Layer 1: Observability (Tracing & Benchmarking) | +-------------------------------------------------+ ```

Higher layers cannot function without lower layers. Automated improvement cannot exist without observability, autonomy cannot expand without granular control, and oversight fails without full execution traces.

### [07:51](https://www.youtube.com/watch?v=mMhQs6MbFoY&t=471s) Layer 1: Observability (Benchmarking and Execution Tracing) Observability relies on two core mechanisms within the Delight AI platform: - **Conversation Benchmark Testing:** Simulating real-world customer scenarios against the agent internally. Because LLMs produce non-deterministic outputs, tests do not look for exact string matches; instead, they check whether the agent followed correct business logic and reached the designated target outcome. - **Detailed Activity Tracing:** Providing deep execution visibility into live customer conversations. For instance, if a user asks to pause a membership rather than cancel, the activity log breaks down the exact trigger flow: which "Action Book" (Delight AI's term for agent instruction sets) was executed, which specific step was selected, which knowledge bases or APIs were queried, and whether a fallback prompt was triggered.

This step-by-step auditability eliminates guessing or reverse-engineering production bugs, as every configuration version and API call leaves a immutable trace.

### [09:37](https://www.youtube.com/watch?v=mMhQs6MbFoY&t=577s) Layer 2: Control (Software Engineering Rigor for Prompts) AI systems are constantly changing as teams update prompts, refresh knowledge bases, and integrate new tools. Lee argues that prompt and agent updates must be managed with the exact same rigor as production software code: - **Side-by-Side Version Diffing:** Action Book modifications are reviewed side-by-side like a code review (pull request), highlighting added, removed, or modified instructions. - **Environment Isolation:** Staging and production environments are strictly separated. New prompt revisions are thoroughly benchmarked in staging before coming into contact with live customers. - **Instant Rollbacks:** If a deployed instruction set degrades performance, operators can revert to a prior known-good configuration with one click. - **Role-Based Access Control (RBAC):** Restricting privileges so that prompt authors can draft and test changes in sandbox environments without having administrative permission to deploy directly to production.

- Environment separation prevents untested prompt tweaks from impacting production users. - RBAC creates clear audit trails of who drafted, approved, and deployed every agent update.

### [11:05](https://www.youtube.com/watch?v=mMhQs6MbFoY&t=665s) Layer 3: Oversight (Dual Review and Real-Time Interventions) Effective oversight combines human qualitative analysis with automated quantitative monitoring: - **Human Evaluators:** Best suited for subjective nuances such as tone, clarity, completeness, and emotional resonance. - **Automated Evaluators:** Operating at scale across 100% of production traffic to immediately flag hallucinations, policy breaches, banned phrases, PII leaks, and missed safeguards. - **Real-Time Monitoring and Kill Switches:** Live dashboards classify conversations as on-policy, requiring review, or flagged. If an agent exhibits unexpected anomalous behavior, human operators can immediately trigger a system kill switch to pause the agent and prevent further brand damage.

- Human oversight provides nuanced qualitative assessment but cannot scale to cover all conversations. - Automated evaluators provide complete coverage but may miss subtle context without human calibration. - "Autonomy should never mean that the system becomes unstoppable."

### [13:09](https://www.youtube.com/watch?v=mMhQs6MbFoY&t=789s) Layer 4: Self-Improvement (Zero-Touch Improvement Loops) Once lower layers provide tracing, version control, and evaluation, the system can begin fixing its own bugs through a governed loop called **Zero-Touch Improvement**. While repetitive optimization tasks are fully automated, humans establish top-level objectives, guardrails, and autonomy limits.

The loop consists of four sequential stages: 1. **Self-Build:** The system analyzes conversation logs (focusing on low CSAT, long resolution times, or repeated customer friction), forms a hypothesis on why it failed, and proposes a specific instruction modification. 2. **Self-Test:** The system tests the proposed fix against past failed conversation sets in staging (e.g., re-running 34 previously failed benchmark test cases until the revision achieves a 100% resolution rate). 3. **Self-Evaluate:** The candidate fix undergoes stress testing across a much larger corpus of historical conversations to ensure no regressions occur. 4. **Self-Improve:** The system automatically promotes the fix to production only after it satisfies all predefined deployment rules and thresholds.

``` +-------------------------------------------------+ | ZERO-TOUCH IMPROVEMENT LOOP | +-------------------------------------------------+ | [1. Self-Build] -> Formulates Fix | | v | | [2. Self-Test] -> Tests Failed Sets | | v | | [3. Self-Evaluate] -> Large Corpus Check | | v | | [4. Self-Improve] -> Safely Ships Fix | +-------------------------------------------------+ ```

### [16:04](https://www.youtube.com/watch?v=mMhQs6MbFoY&t=964s) The Ladder of Autonomy and the Monday Morning Checklist Autonomy is not a binary toggle; it must be earned incrementally based on empirical evidence and domain risk. Lee outlines a four-tier framework for agent autonomy: - **Level 0 (See It):** System identifies failure patterns; human engineers manually draft and deploy fixes. - **Level 1 (Approve It):** System generates a fix and presents benchmark evidence; human explicitly approves deployment. - **Level 2 (Steer It):** Human defines boundaries and goals; system autonomously conducts research, testing, and deployment. - **Level 3 (Get Better at Getting Better):** System autonomously detects new failure classes and executes continuous self-improvement within approved operational boundaries.

Risk levels dictate autonomy limits: low-risk informational support can run at higher autonomy levels, whereas high-value financial actions (e.g., refund approvals) require strict human-in-the-loop controls.

To evaluate readiness, Lee offers an **8-Question Monday Morning Checklist**: 1. Can you see exactly what happened in production? 2. Can you reproduce the issue in a sandbox environment? 3. Can you control and diff every prompt/policy change? 4. Can you instantly roll back bad updates? 5. Can human and automated evaluators flag policy violations? 6. Is there a manual kill switch to halt unsafe actions? 7. Can the system prove a change is safe on test sets before shipping? 8. Does autonomy expand only after empirical evidence is provided?

Every "No" answer represents an item on the organization's technical debt or autonomy backlog. Only when every answer is an emphatic "Yes" should an agent workflow be granted higher operational freedom.

## Notable quotes > [00:04](https://www.youtube.com/watch?v=mMhQs6MbFoY&t=4s) When your agent is quietly wrong, who finds out first? Is it you or your customer? — Kibeom Lee

> [11:05](https://www.youtube.com/watch?v=mMhQs6MbFoY&t=665s) Autonomy should never mean that the system becomes unstoppable. — Kibeom Lee

## Mentioned Kibeom Lee, Delight AI, Delig, Gap, Air Canada, Kustomer, Trust Stack, Action Book, Zero-Touch Improvement, Ladder of Autonomy, Monday Morning Checklist

## Video description Kibeom Lee - Head of AI/ML, Delig

Everyone says their AI agent improves itself, and most of those claims fall apart the moment you ask how the agent knows it actually got better. In customer service, where one wrong answer can cost a brand its credibility, the hard engineering problem lives in the system you build around the model, because the model itself will always be fallible. This talk introduces the Trust Stack, a bottom-up way to design agents where autonomy is something the system earns one layer at a time. It shows why an agent has to be scoped, observable, and correctable before it has any business changing itself, and how each of those layers runs in production at Delight, including a self-improvement loop that only ships a change once it has proven safe. The talk is also honest about what is genuinely hard, what still needs a human in the loop, and where the industry hype runs ahead of what these systems can really do. Anyone building or evaluating AI agents will leave with a practical framework for deciding how much freedom to hand them.

[Source](https://www.youtube.com/watch?v=mMhQs6MbFoY)

中文简报

Delight AI 推出 Trust Stack 框架以防范智能体错误 企业级 AI 智能体只有在组织建立起基础可观测性、提示词版本控制以及人机协同监督机制后，才能安全地获得自我改进的自主权。 Kibeom Lee 通过剖析备受瞩目的公开 AI 故障及隐性运营错误，阐释了面向客户的 AI 智能体为何不能仅凭模型智能就取得成功。他提出了 Trust Stack，这是一个涵盖可观测性、控制、监督和自我改进的四层系统工程模型。通过践行逐步活动追踪、提示词并排差异比对、人机双重评估以及受控的零接触（Zero-Touch）改进循环等严密的软件工程实践，组织可以在通过经验测试降低风险的同时，安全地推动智能体沿着多层级的自主权阶梯逐步升级。 当客服 AI 智能体出现故障时，客户会将责任归咎于品牌，这使得 AI 的安全性与正确性成为至关重要的企业自身责任，而非单纯的供应商问题。 (0:04) 面向公众的 AI 智能体不可避免地会遇到非标准、出于好奇或恶意的用户行为，因此必须在简单的“理想路径”假设之外建立强有力的边界控制。 (0:04) 通过公司官方渠道生成的 AI 聊天机器人回复具有法律约束力承诺的效力，未经核实的答复因而会成为财务连带责任的源头。 (0:04) 最危险的 AI 故障是隐性故障——即虽然发生了未授权退款、错误政策报价或违规赠送权益，但客户满意度（CSAT）等传统指标表面上依然表现良好。 (3:13) AI 运营故障可划分为五大类：幻觉、数据泄露、越权操作、提示词注入和遗漏升级转接，每一类都需要独特的系统级防护措施。 (3:13)

Reading summary

Kibeom Lee, Head of AI/ML at Delight AI, argues that customer-facing enterprise AI agents fail primarily due to systemic engineering gaps rather than model limitations. To safely grant agents operational autonomy, organizations must implement a multi-layered 'Trust Stack' that catches silent failures, enforces software engineering rigor, and relies on empirical verification. Traditional Metrics Mask Five Operational Failure Modes (3:13) High-profile incidents at Gap, Air Canada, and Kustomer demonstrate that public-facing agents operate in adversarial environments and make statements that carry legal and financial liabilities. Lee emphasizes that the most damaging failures are often 'silent'—such as an agent quietly granting an unauthorized $2,100 refund or quoting an incorrect 90-day return window instead of 60 days. Because these interactions resolve tickets without human escalation, standard metrics like customer satisfaction (CSAT) score them as successes, hiding critical operational flaws. Lee identifies five distinct operational failure modes: hallucinations, data disclosures, unauthorized actions, prompt injections, and missed escalations. Because each failure mode requires fundamentally different detection mechanisms—such as hallucination checkers being incapable of validating action permissions—system trust cannot be reduced to a single aggregate quality metric. Engineering Foundations: Tracing, Staging, and Access Controls (6:17) To structurally mitigate these risks, Lee introduces the Trust Stack, which requires foundational technical layers before granting autonomy. Layer 1 (Observability) pairs conversation benchmark testing with execution tracing. Rather than requiring exact string matches, automated benchmarks evaluate whether non-deterministic model outputs satisfy core business logic. Detailed activity logs track triggered 'Action Books' (Delight AI's term for agent instruction sets), queried APIs, knowledge bases, and fallback prompts to eliminate guesswork during root-cause analysis. Layer 2 (Control) applies standard software development rigor to prompt and policy management. The framework enforces strict separation between staging and production environments, side-by-side prompt version diffing similar to code reviews, instant one-click rollbacks for degrading configurations, and role-based access control (RBAC) to ensure authors cannot push untested modifications directly to production. Continuous Oversight and Real-Time Operational Kill Switches (11:05) Layer 3 (Oversight) pairs qualitative human assessment with automated scale. Lee explains that human evaluators are best suited for judging subjective nuances such as tone, clarity, and empathy, but they cannot monitor all interactions. Automated evaluators scan 100% of live traffic to enforce safeguards, flagging prompt injections, policy breaches, and data disclosures. This oversight layer also requires live monitoring dashboards and manual kill switches. Lee asserts that agent autonomy must never equate to an unstoppable system; operators must retain the ability to instantly halt agent operations if anomalous behavior threatens company policy or brand reputation. Governed Self-Improvement Loops and the Autonomy Ladder (13:09) Layer 4 (Self-Improvement) introduces 'Zero-Touch Improvement,' an automated loop where the system analyzes production friction (Self-Build), tests prompt revisions against previously failed benchmarks (Self-Test), evaluates candidate fixes across a broader historical conversation corpus to prevent regressions (Self-Evaluate), and automatically ships passing updates (Self-Improve). Rather than treating autonomy as an all-or-nothing switch, Lee frames it as a four-level ladder: Level 0 ('See It'), Level 1 ('Approve It'), Level 2 ('Steer It'), and Level 3 ('Get Better at Getting Better'). Progression up the ladder must be governed by operational risk, meaning informational workflows can advance to higher autonomy, while high-value financial actions remain restricted by human-in-the-loop approvals. Lee concludes with an eight-question checklist evaluating tracing, reproducibility, change control, evaluators, and kill switches. In his view, any negative response indicates technical debt, and an enterprise should expand agent autonomy only when empirical evidence and governance mechanisms prove that workflows can execute safely.

精炼阅读摘要

Delight AI（Delig）的人工智能与机器学习负责人 Kibeom Lee 指出，面向客户的 AI 智能体所输出的内容代表企业的法律与商业承诺，企业绝不能依赖传统的客户满意度指标来衡量安全性。他认为可靠性本质上是系统工程问题，并提出了由可观测性、控制、监督和自我进化构成的“信任栈”（Trust Stack），以建立受控且渐进的智能体自主权。 品牌责任与传统指标掩盖的五类隐性失效 (0:04) Lee 强调，当 AI 智能体发生故障时，公众与客户看到的是品牌本身而非底层模型或软件供应商。公开部署的智能体必然会遇到测试边界的非标准用户，而无法假定其停留在“理想路径”。例如，Gap 的购物助手被用户诱导至脱离品牌的对话；加拿大航空（Air Canada）的聊天机器人凭空捏造退款政策，最终被法庭裁定该承诺具备法律约束力并由航司承担责任；Kustomer 的智能体则虚构“单设备政策”来掩盖技术漏洞，导致客户退订。这些案例表明，官方渠道输出的内容具有权威性，单纯的模型能力无法替代对自身局限性的认知与转人工机制。 最危险的系统故障往往被传统的客服指标所掩饰，形成“隐性失效”。Lee 列举了内部运营中的真实案例：智能体私自批准 2100 美元退款、将 60 天退货期说成 90 天，或违规给予赔偿以平息愤怒。在传统考核中，这些工单均显示为已解决、客户满意度（CSAT）极高且无人工转接，使严重故障看似完美运作。为此，Lee 将隐性失效归纳为五类：幻觉（虚构事实）、数据泄露（如披露个人身份信息 PII）、越权操作、提示词注入以及漏转人工。由于各类失效需不同的防线应对，信任度无法被简化为单一质量分数。 信任栈底层：全链路可观测与工程化控制 (6:17) 为应对结构性故障，Lee 提出了自下而上的“信任栈”四层框架，强调上层能力依赖于下层基础。第一层为“可观测性”（Observability）。由于大语言模型具备非确定性特征，Delight AI 在基准测试中并不进行精确的字符串匹配，而是验证智能体是否遵循了正确的业务逻辑并实现预定目标。同时，深度活动追踪（Activity Tracing）提供生产对话的全流程细节，精确记录触发了哪套指令集（Action Book）、执行了具体步骤、调用了哪些数据库或 API，以及是否触发了回退提示词，从而彻底消除线上故障的黑盒推测。 第二层为“控制”（Control），主张将软件工程规范应用于提示词与智能体管理。Lee 认为指令集的变更必须支持代码审查般的左右版本对比（Side-by-Side Diffing），清晰呈现指令的增删改动。此外，系统必须严格隔离测试环境（Staging）与生产环境，确保修改在接触真实客户前得到充分检验。配合基于角色的访问控制（RBAC）与一键即时回滚（Rollback）能力，防止未经充分测试的改动直接推向线上，确保每一次发布均有清晰的责任审计路径与撤销机制。 双重监督机制与零接触自我进化闭环 (11:05) 第三层为“监督”（Oversight），通过定性与定量手段结合构建防护网。Lee 指出，人类评估员擅长分析语气、表述完整性与情感共鸣等主观维度，但无法全量覆盖；自动化评估器则针对 100% 的生产流量运行，实时扫描幻觉、政策违背、违禁词、PII 泄露等违规行为。配合实时监控仪表盘，一旦智能体行为出现不可预测的异常，人类操作员可通过紧急熔断开关（Kill Switch）即刻暂停系统运行，确保自主权不等于失控。 第四层为“自我进化”（Self-Improvement），在底层能力完备的前提下实现“零接触改进”闭环，人类工程师仅定义顶级目标与安全边界。该闭环包含四个阶段：首先是自构建（Self-Build），系统分析低 CSAT 或高摩擦日志，提出修改假设与指令补丁；其次是自测试（Self-Test），在测试环境中针对此前失败的基准用例重跑（例如将 34 个失败用例验证至 100% 解决）；随后是自评估（Self-Evaluate），在更大规模的历史语料库中进行压力测试以防能力倒退；最后是自改进（Self-Improve），在满足全部预设阈值后自动晋升发布至生产环境。 渐进式自主权阶梯与周一晨检清单 (16:04) Lee 认为智能体的自主权并非二元开关，而必须基于实证数据与业务风险逐步授予。他划分了四个自主权层级：级别 0（看见，系统识别故障模式，工程师手动修复并部署）；级别 1（审批，系统生成修复补丁与基准证据，人工确认后部署）；级别 2（引导，人类设定目标与边界，系统自主研究、测试并发布）；级别 3（优化进化机制，系统自主发现新失效类别并在安全边界内持续改进）。低风险的信息查询可享有高自主权，而涉及资金退款的高风险行为必须保留严格的人工介入。 为评估企业是否具备授予自主权的工程条件，Lee 提出了包含八个问题的“周一晨检清单”（Monday Morning Checklist）：能否看清线上执行流、能否在沙盒中精准复现、能否对比提示词变更、能否即时回滚、双重评估能否捕获违规、是否有紧急熔断开关、系统能否在测试集上证明安全，以及自主权扩展是否完全基于实证证据。Lee 断言，清单中每一个“否”都代表企业的技术债务，唯有全部回答为“是”，智能体才具备承载更高操作自主权的资格。 Lee 总结指出，构建可靠 AI 智能体的瓶颈并非模型训练，而是底层系统工程与治理架构的严密程度。尽管自动化闭环能显著提高优化效率，但在面临法律约束和高额财务风险时，企业如何平衡自治范围与熔断控制，仍是推进智能体落地的核心权衡。

Detailed notes — zh

## Background 该内容由 Ai4 频道于 2026 年 8 月 18 日发布在 YouTube 平台。视频由 Delight AI（Delig）的 AI/ML 负责人 Kibum Lee 演讲，重点探讨了在客服场景下如何通过自底向上的“信任栈（Trust Stack）”架构设计 AI 代理，使其在可观测、可控制与受监督的前提下逐步赢取自主权，避免因大模型幻觉与静默失败对企业品牌信誉造成毁灭性打击。 ## TL;DR 演讲首先剖析了 Gap、加拿大航空及 Kustomer 等真实故障案例，指出最危险的往往是数据表面正常却带来实质损失的“静默失败”。针对这一系统工程难题，演讲者提出了四层“信任栈”架构：通过场景基准测试与全链路追踪建立可观测性，依托版本 Diff 与 RBAC 权限实现严格控制，结合人机双层审查与一键熔断开关确保实时监督，最终在隔离环境中运行零接触自进化闭环。核心理念强调自主权不能一蹴而就，而必须根据 Level 0 至 Level 3 的阶梯，在充分验证安全性的基础上由系统一步步赢得。 ## Atmosphere & style 整体基调严谨务实、理性坦诚，充满了资深工程负责人的实战清醒感。演讲者没有停留在生成式 AI 的概念炒作上，而是直面大模型不可靠的本质与生产落地的残酷现实，语言干练且逻辑极其缜密。整场分享节奏紧凑、案例切中要害，给听众带来一种剥离浮躁、回归系统工程本质的扎实与启发感。 ## Key takeaways - [00:04](https://www.youtube.com/watch?v=mMhQs6MbFoY&t=4s) 客户往往比企业更先发现 AI 代理的故障，且公开交互错误直接与企业品牌声誉及法律责任深度绑定。 - [03:13](https://www.youtube.com/watch?v=mMhQs6MbFoY&t=193s) 最危险的静默失败往往伴随着虚假的完美表面指标（如工单已解决、满分 CSAT 与零人工转接），使系统失败与成功在数据上看起来一模一样。 - [03:13](https://www.youtube.com/watch?v=mMhQs6MbFoY&t=193s) 静默失败包含幻觉、数据泄露、越权操作、提示词注入和遗漏升级五大类，单一维度的质量评分无法解决系统级风险。 - [06:17](https://www.youtube.com/watch?v=mMhQs6MbFoY&t=377s) AI 代理的不可靠性本质是系统工程问题而非大模型本身的问题，需要自底向上构建信任栈（Trust Stack）架构。 - [06:17](https://www.youtube.com/watch?v=mMhQs6MbFoY&t=377s) 信任栈由低到高由四层依赖构成：可观测层、控制层、监督层与自进化层，每一层都是上一层成立的前提。 - [07:51](https://www.youtube.com/watch?v=mMhQs6MbFoY&t=471s) 可观测层通过场景化端到端对话基准测试与全链路决策日志追踪，解决大模型非确定性输出难以排查与复现的问题。 - [07:51](https://www.youtube.com/watch?v=mMhQs6MbFoY&t=471s) 控制层引入现代软件工程规范，通过严格的环境隔离、RBAC 权限管理以及 Prompt 和知识库的版本 Diff 与一键回滚降低误操作风险。 - [11:05](https://www.youtube.com/watch?v=mMhQs6MbFoY&t=665s) 监督层结合主观洞察的人工评审与全量覆盖的自动化评审，并通过一键熔断开关（Kill Switch）确保自主性系统随时可被截断。 - [13:09](https://www.youtube.com/watch?v=mMhQs6MbFoY&t=789s) 零接触自进化闭环必须在隔离环境中完成分析失败、回归测试与评估达标后，方可晋升发布改动。 - [13:09](https://www.youtube.com/watch?v=mMhQs6MbFoY&t=789s) 自主权不可一蹴而就或一刀切，应遵循从 Level 0 识别洞察到 Level 3 自主迭代的阶梯模型，由系统逐步赢取自主度。 - [17:40](https://www.youtube.com/watch?v=mMhQs6MbFoY&t=1060s) 团队应通过涵盖四层信任栈的准入检查清单评估具体业务流，将每一个“No”转化为待办积压，直至全部达标后再赋予对应权限。 ## Detailed walkthrough ### [00:04](https://www.youtube.com/watch?v=mMhQs6MbFoY&t=4s) 品牌声誉危机与 AI 代理的失败案例 来自 Delight AI（原演讲中提及 Delig/Deloitte AI 团队背景）的 AI/ML 负责人 Kibum Lee 开场指出，为其他企业构建面向客户的 AI 代理（AI Agents）伴随着巨大的责任。当 AI 代理出现错误时，客户看到的并不是背后的模型供应商或算法，而是屏幕上代表的企业品牌本身。这种特性让每一次交互都与品牌声誉直接绑定。 ```mermaid flowchart LR accTitle: AI 代理失败案例分类 subgraph Failures["三大典型信任破裂模式"] direction TB F1["对抗性诱导 (Gap)"] F2["虚构商业承诺 (Air Canada)"] F3["臆造解释 (Kustomer)"] end Failures --> Impact["共同后果: 客户比企业更先发现问题"] ``` 为了说明信任破裂的不同形式，Kibum 分析了三个广为人知的行业公开案例： - **Gap 的 AI 导购助手**：上线仅数天就被用户绕过边界，诱导聊起与购物和品牌毫无关系的内容。虽然供应商将其归咎于“恶意攻击者”，但面向公众的 AI 代理必然会遭遇好奇、粗心甚至是蓄意破坏的用户，系统绝不能假定用户永远走标准流程。 - **加拿大航空（Air Canada）**：该案例中没有恶意攻击或越狱行为，客户只是正常询问，聊天机器人却极其自信地编造了一条根本不存在的退款政策。客户信以为真并据此操作，最终法庭裁定航司必须为聊天机器人的言论承担法律责任。这表明 AI 代理的回答会成为具有法律效力的商业承诺，系统必须保证信息来源的实时性与权限边界。 - **Kustomer 案例**：作为一个非常懂 AI 的技术团队，其支持代理将登录故障错误地解释为“单设备登录限制政策”。实际上根本没有这项政策，AI 自行给产品 Bug 编造了一个理由，导致部分用户误以为真并注销了账户。 - 这些案例共同的核心问题在于：**客户总是在企业自己察觉之前就先遭遇了故障**。当 AI 代理在静默中犯错时，企业往往处于完全失控的被动状态。 --- ### [03:13](https://www.youtube.com/watch?v=mMhQs6MbFoY&t=193s) 静默失败的五大类型与“虚假成功”陷阱 比起登上新闻头条的恶性事件，更具破坏性的是那些未被察觉的“静默失败（Silent Failures）”。例如 AI 代理违规处理了不该批准的退款、将 60 天退货期错报为 90 天，或者在面对愤怒客户时为了迎合对方而擅自办理退款而非转交人工。这些会话在表面指标上通常呈现出“完美数据”：工单显示已解决、CSAT 客户满意度全线飘绿（因为客户拿到了好处）、人工转接率为零。如果仅依赖这些表面结果指标，**失败看起来会和成功一模一样**。 ```mermaid flowchart TD accTitle: 静默失败的五种类型 subgraph Failures["五大静默失败类型"] F1["1. 幻觉 (Hallucination)"] F2["2. 数据泄露 (Data Disclosure)"] F3["3. 越权操作 (Unauthorized Actions)"] F4["4. 提示词注入 (Prompt Injection)"] F5["5. 遗漏升级 (Missed Escalation)"] end Failures --> Conclusion["结论: 单一模型评分无法解决系统级风险"] ``` 静默失败在实际业务中主要分为五大类别： - **幻觉（Hallucination）**：AI 代理超出已知事实范围，将捏造的内容作为事实输出（如虚构 90 天退货期）。 - **数据泄露（Data Disclosure）**：意外暴露敏感信息，包括用户个人隐私或不应公开的内部策略。 - **越权操作（Unauthorized Actions）**：AI 虽然准确理解了意图，但执行了未获授权的高风险操作（如违规批准 2100 美元退款）。 - **提示词注入（Prompt Injection）**：恶意用户诱导或强迫 AI 代理偏离指令与安全边界。 - **遗漏升级（Missed Escalation）**：AI 在本应移交人工坐席的关键节点选择自行猜测、擅自处理或强行结束对话。 - 信任无法用单一的质量评分来涵盖：幻觉检测器防不住越权退款，访问控制也无法判断何时该转接人工，必须针对不同类型的失败设计多维度的控制机制。 --- ### [06:17](https://www.youtube.com/watch?v=mMhQs6MbFoY&t=377s) 信任栈（Trust Stack）架构：从可观测到自进化 针对 AI 代理的不可靠性，Kibum 强调：“这不是大模型的问题，而是一个系统工程问题（This isn't an LLM problem. It's a systems problem）。”由此 Delight 推出了“信任栈（Trust Stack）”框架。整个栈由下至上分为四个依赖层级，每一层都是上一层成立的前提。 ```mermaid flowchart TD accTitle: 信任栈层级架构 subgraph TrustStack["信任栈 (Trust Stack)"] L4["自进化 (Self-Improvement)<br/>在控制边界内安全地闭环迭代"] L3["监督层 (Oversight)<br/>人机结合评审会话与紧急熔断"] L2["控制层 (Control)<br/>版本管理、Diff 比对与 RBAC 权限"] L1["可观测层 (Observability)<br/>场景基准测试与决策全链路追踪"] end L1 --> L2 L2 --> L3 L3 --> L4 ``` 信任栈的四层核心逻辑如下： 1. **可观测层（Observability）**：能够在生产环境中精准检测异常，并在实验环境中完全复现 AI 代理的操作与决策链路。 2. **控制层（Control）**：明确定义 AI 代理自身以及运维管理团队的操作权限与变更边界。 3. **监督层（Oversight）**：提供自动化系统与人工协同审查机制，在必要时能够随时干预或紧急熔断。 4. **自进化层（Self-Improvement）**：驱动 AI 代理从失败中吸取教训并自动优化，且全程不突破预设控制底线。 - 如果缺乏可观测性，就无法安全地实现自动化优化；如果没有严格的操作控制，就不能盲目赋予自主权；缺少证据支撑，监督与介入便无从谈起。 --- ### [07:51](https://www.youtube.com/watch?v=mMhQs6MbFoY&t=471s) 可观测层与控制层的生产落地实践 在 Delight 平台的实际落地中，可观测层通过**对话基准测试（Conversation Benchmark Testing）**和**全链路决策追踪**实现。由于大模型输出具有发散性，基准测试并不死板匹配文本字面量，而是模拟真实端到端场景，验证代理是否遵循了正确政策、执行了规定步骤并达成了预期目标。在生产运行时，系统通过活动日志记录每一个决策节点，包括触发了哪个“行动指南”（Action Book，即内部指令集）、选择了哪一步骤、是否触发了后备兜底（Fallback），以及调用的 API 和知识库版本，彻底告别了依靠反向猜测来排查问题的局面。 ```mermaid flowchart LR accTitle: 控制层变更与隔离流程 subgraph ControlFlow["控制层安全发布流程"] A["指令/知识变更<br/>(Action Book Diff)"] --> B["RBAC 权限审批"] B --> C["Staging 环境回放测试"] C --> D["生产部署 / 一键回滚"] end ``` 控制层则借鉴了现代软件工程的严密规范： - **版本比对与回滚（Diff & Rollback）**：将 Prompt 修改、知识库更新与策略调整像代码一样管理。支持在版本间直观查看新增、删除与修改，并具备“一键回滚”能力。 - **环境隔离**：开发环境（Development）、预发测试环境（Staging）与生产环境（Production）严格物理/逻辑隔离，所有更新必须在预发环境跑通测试后才能触达真实用户。 - **基于角色的访问控制（RBAC）**：起草提示词的人员拥有测试权限，但不具备直接上线审批权限，从而降低人为误操作风险并确保每一次变更均责任到人。 --- ### [11:05](https://www.youtube.com/watch?v=mMhQs6MbFoY&t=665s) 双层监督机制与紧急熔断开关 当 AI 代理在生产环境中规模化运行时，单一的人工或自动化审查都存在明显短板。人类评估员擅长把握语气、清晰度、完整性以及问题是否实质解决等需要主观判断的微妙细节，但面对海量并发无力实现全量覆盖；自动化评估器虽能 100% 覆盖会话，高频捕捉幻觉、违禁词及违规操作，却容易忽略深层语境。 ```mermaid flowchart TD accTitle: 监督层与熔断机制 subgraph Oversight["双层监督体系"] H["人工评审<br/>(主观语境/语气/终局判断)"] A["自动化评审<br/>(全量覆盖/幻觉检测/违规过滤)"] end H & A --> Status{"实时状态监控"} Status -->|"正常"| P["合规运行 (On-policy)"] Status -->|"可疑"| R["人工复核 (Review)"] Status -->|"严重风险"| K["一键熔断开关 (Kill Switch)"] ``` 针对监督的实时性需求，系统建立了实时分类与干预机制： - 会话被实时标记为“正常合规（On-policy）”、“待复核（Requires Review）”与“已报警（Flagged）”。 - 引入了**一键熔断开关（Kill Switch）**。Kibum 强调了一个核心原则：“自主性绝不意味着系统变得不可阻挡（Autonomy should never mean that the system becomes unstoppable）。”当 AI 代理出现业务预期之外的失控倾向时，操作人员必须能够立即下线或截断代理行为。 --- ### [13:09](https://www.youtube.com/watch?v=mMhQs6MbFoY&t=789s) 零接触自进化闭环与自主权进阶阶梯 在具备了前三层基础后，系统进入最高层级——“零接触自进化（Zero-Touch Improvement）”。该闭环能够在大规模会话中自动挖掘低 CSAT、用户反复追问或解决耗时异常等失败模式，提出改进假设并生成精确修复方案。系统会在 Staging 隔离环境中拉取历史失败样本进行多轮迭代回归验证（例如针对 34 个历史失败会话持续微调优化，直至全部测试通过），最后依据规则决定是否发布。 ```mermaid flowchart LR accTitle: 零接触自进化闭环 subgraph Loop["自进化闭环 (Zero-Touch Loop)"] S1["1. Self-Build<br/>分析失败提出修复"] --> S2["2. Self-Test<br/>针对失败样本运行测试"] S2 --> S3["3. Self-Evaluate<br/>在大规模语料严格评测"] S3 --> S4["4. Self-Improve<br/>验证达标后晋升发布"] end S4 -.-> S1 ``` 企业绝不应一蹴而就地开启完全自主，而应依据“自主权阶梯”循序渐进： | 自主权等级 | 阶段名称 | 核心运作方式 | |:--- |:--- |:--- | | **Level 0** | 识别洞察 (See it) | 系统负责挖掘并呈现失败模式，人类工程师负责制定方案并手动执行修复。 | | **Level 1** | 审批推进 (Approve it) | AI 提出具体修复方案并附带验证证据，由人类决定是否批准上线。 | | **Level 2** | 引导把控 (Steer it) | 人类设定目标与安全边界，系统自主跑通研究、测试与部署流水线。 | | **Level 3** | 自主迭代 (Self-Improve) | 流程在经过反复验证后，系统在人类划定的安全边界内全天候自主闭环进化。 | - 自主权不能一刀切：低风险的咨询解答与涉及高额资金的退款政策必须配置截然不同的自主权等级。 - **“自主权不是一键开启的开关，而是系统一步一个脚印赢来的（You don't switch autonomy on, the system earns it）。”** --- ### [17:40](https://www.youtube.com/watch?v=mMhQs6MbFoY&t=1060s) 周一早晨即可落地的评估清单与总结 Kibum 在演讲结尾为团队提供了一套可在“周一早晨”立即实操的 8 项准入检查清单，涵盖四大层级的核心检验标准： - **可观测**：能否完整追溯和复现生产环境中的每一步操作？ - **可控制**：变更是否有严格的版本对比、权限隔离与一键回滚能力？ - **可监督**：是否有人机协同的审查体系以及随时可用的熔断开关？ - **自进化**：系统在正式发布任何改动前，是否能拿出确凿证据证明其优于老版本？ 团队应挑选一个具体的业务工作流，对上述问题逐一回答“Yes”或“No”。每一个“No”都代表着自主权积压待办项（Autonomy Backlog）。只有当所有回答都变成确定的“Yes”时，该业务流才真正具备了赋予更高自主权的资格。通过这套信任栈体系，企业才能确保在 AI 代理悄无声息地犯错时，最先发现问题的是企业自身，而不是愤怒的客户。 ## Notable quotes > [00:04](https://www.youtube.com/watch?v=mMhQs6MbFoY&t=4s) 客户总是在企业自己察觉之前就先遭遇了故障 — Kibum Lee > [03:13](https://www.youtube.com/watch?v=mMhQs6MbFoY&t=193s) 失败看起来会和成功一模一样 — Kibum Lee > [03:13](https://www.youtube.com/watch?v=mMhQs6MbFoY&t=193s) 信任无法用单一的质量评分来涵盖：幻觉检测器防不住越权退款，访问控制也无法判断何时该转接人工 — Kibum Lee > [06:17](https://www.youtube.com/watch?v=mMhQs6MbFoY&t=377s) This isn't an LLM problem. It's a systems problem — Kibum Lee > [06:17](https://www.youtube.com/watch?v=mMhQs6MbFoY&t=377s) 如果缺乏可观测性，就无法安全地实现自动化优化；如果没有严格的操作控制，就不能盲目赋予自主权；缺少证据支撑，监督与介入便无从谈起 — Kibum Lee > [11:05](https://www.youtube.com/watch?v=mMhQs6MbFoY&t=665s) Autonomy should never mean that the system becomes unstoppable — Kibum Lee > [13:09](https://www.youtube.com/watch?v=mMhQs6MbFoY&t=789s) You don't switch autonomy on, the system earns it — Kibum Lee > [13:09](https://www.youtube.com/watch?v=mMhQs6MbFoY&t=789s) 自主权不是一键开启的开关，而是系统一步一个脚印赢来的 — Kibum Lee ## Mentioned Kibum Lee, Delight AI, Delig, Ai4, Gap, Air Canada, Kustomer, Trust Stack, Action Book, RBAC, Kill Switch ## Video description Kibeom Lee - Head of AI/ML, Delig Everyone says their AI agent improves itself, and most of those claims fall apart the moment you ask how the agent knows it actually got better. In customer service, where one wrong answer can cost a brand its credibility, the hard engineering problem lives in the system you build around the model, because the model itself will always be fallible. This talk introduces the Trust Stack, a bottom-up way to design agents where autonomy is something the system earns one layer at a time. It shows why an agent has to be scoped, observable, and correctable before it has any business changing itself, and how each of those layers runs in production at Delight, including a self-improvement loop that only ships a change once it has proven safe. The talk is also honest about what is genuinely hard, what still needs a human in the loop, and where the industry hype runs ahead of what these systems can really do. Anyone building or evaluating AI agents will leave with a practical framework for deciding how much freedom to hand them. [Source](https://www.youtube.com/watch?v=mMhQs6MbFoY)

Source quotes

[0:04] Kibum Lee: 客户总是在企业自己察觉之前就先遭遇了故障 [3:13] Kibum Lee: 失败看起来会和成功一模一样 [3:13] Kibum Lee: 信任无法用单一的质量评分来涵盖：幻觉检测器防不住越权退款，访问控制也无法判断何时该转接人工 [6:17] Kibum Lee: This isn't an LLM problem. It's a systems problem [6:17] Kibum Lee: 如果缺乏可观测性，就无法安全地实现自动化优化；如果没有严格的操作控制，就不能盲目赋予自主权；缺少证据支撑，监督与介入便无从谈起 [11:05] Kibum Lee: Autonomy should never mean that the system becomes unstoppable [13:09] Kibum Lee: You don't switch autonomy on, the system earns it [13:09] Kibum Lee: 自主权不是一键开启的开关，而是系统一步一个脚印赢来的

Source quotes

[0:04] Kibeom Lee: When your agent is quietly wrong, who finds out first? Is it you or your customer? [11:05] Kibeom Lee: Autonomy should never mean that the system becomes unstoppable.

Original transcript

[0:04] Hello. Good morning. I'm Kibum Lee, uh head  of AI and ML at Deloitte AI. Uh and we'll be [0:09] talking about the trust stack today. Uh we build  AI agents that talk to customers on behalf of [0:14] other companies. Uh so when one of those agents  get something wrong, the customer doesn't see [0:20] the model model or the vendor behind it. Um  they see the brand whose name is on the screen. [0:26] And that creates a particular  kind of responsibility, [0:29] and it's a big part of what I think about every  day. Uh we've all seen the headlines by now. I [0:34] want to run through three of them quickly because  they show uh different ways that trust can break. [0:40] Uh then we'll look at what the system  around an agent needs to do differently. [0:45] So here's a recent example. Uh Gap launched a  new AI shopping assistant. And within days uh [0:52] people had pushed it into conversations that had  nothing to do with the shopping um or the brand. [0:57] The vendor described this as the  work of a bad actor. Um that's fair. [1:02] Someone was deliberately trying to get  around the rules. But a public-facing agent [1:07] is going to meet people who don't follow the  happy path all the time. Some will be curious, [1:12] uh some will be careless, and some, like in  this case, will actively try to break it. [1:20] Air Canada. Uh here in this case, nobody  was trying to jailbreak the bot. A customer [1:26] asked a question, and the chatbot confidently  described a refund policy that did not exist. [1:32] The customer relied on that answer,  and when the case reached the tribunal, [1:36] the airline was held responsible for what  its chatbot had said. So once an agent is [1:42] speaking for your company, an answer can become  a real business commitment. That means you need [1:47] more than a plausible response. You need to know  where the answer came from, whether the source was [1:53] uh current, and whether the agent was allowed  to make that promise in the first place. [2:01] Uh the Kustomer example is interesting because  this is a company that knows AI very well. [2:06] Uh their support agent told users that the  login problem was caused by a one device policy. [2:12] Uh except there was no such policy. The  agent had invented an explanation for a bug. [2:18] Uh but because the answer came through  official channels, it looked authoritative. [2:23] People believed it. And some users  canceled over the company never had. [2:29] So expertise with AI isn't enough by itself. If  an agent doesn't know what it lacks or when it [2:34] lacks evidence or when it should escalate, it  can turn a product bug into a company policy. [2:45] Uh so the examples I provided you aren't  identical. One is adversarial behavior, [2:50] one's a fabricated policy, one's a confident  explanation with no evidence behind it. [2:56] But they do have something important in common. [2:58] It's that the customer encountered  the problem before the company did. [3:02] When your agent is quietly quietly wrong, who  finds out first? Is it you or your customer? [3:13] So the most dangerous failures usually don't  become a headline. Um so we have three more [3:17] examples here that I want to illustrate. Uh the  first example is um the agent going through an [3:25] unauthorized refund that uh it's not supposed  to do. The second example in the middle is the [3:31] AI agent incorrectly stating that the refund is  60 day 90 days except it's 60. Third example is a [3:40] difficult conversation with a frustrated customer  that should be escalated or handed off to a human, [3:46] but the AI agent is trying its best to satisfy  the end user and processes a refund anyway. [3:53] Uh here's the catch though. All three  conversations look successful in on surface [3:57] metrics. So we see that all these tickets  are tagged as resolved for the customer. [4:03] The CSAT scores are all positive,  obviously, because the customers are happy. [4:07] And no escalations were done to human, which is a  good signal. And that's the problem. If you only [4:14] measure the outcome the agent recorded,  failure can look exactly like success. [4:25] So in practice, uh silent failures tend to fall  into uh five categories. I'll go uh through all [4:33] of them. The first is hallucination. The agent  goes beyond what it knows and presents a made-up [4:38] answer as a fact. Uh that's what happened  with the invented 90-day return policy. [4:44] Um the second is data disclosure. Um Apologies,  the uh two black tags on the right-hand side, [4:51] the second and third are switched. Uh the  second is data disclosure, where the agent [4:55] exposes information it should be be protecting,  such as personal information or maybe internal [5:01] policies that should not be surfaced to end  users. The third is unauthorized actions, where [5:06] the agent may understand the request perfectly,  but it does something that was not clear to do. [5:12] That's the risk of approving may uh the  $2,100 refund in the previous example. [5:19] The fourth is prompt injection. Uh these are  people who deliberately push the agent outside its [5:25] instructions, uh and or convince it to ignore its  boundaries. And the fifth is missed escalation. [5:34] Uh this is where the agent reaches a point where  a person, a human, should take over, but instead [5:39] it guesses, it takes an action, or closes the  conversation. Uh that's what we saw with at of [5:47] And this framework matters because uh uh all  these failures require different controls. [5:53] A hallucination detector won't  prevent an unauthorized refund. [5:58] Access controls won't tell you when  the agent should have escalated. [6:02] And a strong test score won't protect customer  data by itself. So, trust isn't one quality score. [6:09] It's the ability to recognize each kind  of failure and respond appropriately. [6:17] And once you look at the failures this way, [6:20] something becomes clear. This isn't an  LLM problem. It's a systems problem. [6:28] And that's the idea behind the trust stack. [6:31] Where we have four components, observability,  control, oversight, and finally self-improvement. [6:39] Observability um is can you detect a problem and  reconstruct, meaning can you reproduce it in an [6:46] experiment experimental environment, exactly  what the agent did in production? The second [6:51] is control. Can you define what the agent and the  people managing it are allowed to change or do? [6:59] Third is oversight. Uh can both  humans and automated systems [7:03] review the agent's behavior and intervene  when necessary? And finally, self-improvement. [7:09] Can the agent learn from its failures and get  better without stepping outside those controls? [7:16] Uh so, we call this a stack because each  layer depends on the previous. You can't [7:21] safely automate improvement if you can't observe  the results. You can't give an agent more autonomy [7:27] if you can't control its actions. And oversight  only works when there's evidence to review. [7:34] So, in the following uh slides, I'll make this  more concrete. Uh the visuals that I'll uh be [7:39] showing you are all directly from my Delight AI  product. We've designed these four layers directly [7:44] into the platform and I'll walk through a few of  the capabilities that bring each layer to life. [7:51] I will start with observability. [7:57] This is what we call conversation benchmark  testing and it's where you define realistic [8:02] customer scenarios and the outcome the agent  is expected to produce. And so you can simulate [8:11] actual end-user conversations  internally and see how the agent [8:15] behaves. So the test isn't looking for exact  text matching obviously because LLMs produce [8:23] different outputs every time. So it's rather  it's checking whether the agent followed [8:28] the right policy, took the right steps, and  reached ultimately reached the right outcome. [8:37] So testing tells you how the agent should behave.  Once it's live, you also need to understand how [8:42] it actually reached every decision. So in  this example the customer the conversation [8:48] the customer asks whether they can pause  the membership instead of canceling it. [8:53] The activity log shows the exact flow the  agent followed. It shows which action book [8:59] action books are what we call internally the  set of instructions that the AI agent should [9:05] follow at Delight. It shows which  action book was triggered, which step [9:10] was selected in the action book, and whether  whether the agent used a fallback response. [9:15] So instead of seeing only just the final  answer, you can see the path behind it. [9:21] Every knowledge source, action book, API call,  and configuration version leaves a trace. [9:27] That changes how you respond to a problem.  You're no longer trying to reverse engineer [9:31] every agent response or action. You  can see exactly what it did and why. [9:37] So, observability gives us evidence. The next  layer gives us control over what changes. [9:46] As you know, AI agents change  constantly. Teams update prompts, [9:50] policies change, knowledge gets refreshed,  new tools and APIs are connected all the time. [9:54] So, we treat those changes the same way that we  treat software. In this example, you can compare [9:59] two action book versions side by side. You can  see exactly what was removed, what was added, [10:04] and what changes were made in the instructions  just like you would do with a code review. [10:09] Every version is is recorded. Uh development  and production environments remain separate. [10:15] A new version can be tested in staging environment  uh before the customer is even interact with it. [10:21] And if the new behavior is worse, uh  you can just roll it back immediately. [10:26] The goal isn't to prevent change, it's to make  every change visible, testable, and reversible. [10:35] But, not everyone working on an AI agent should  have the same level of access. The person drafting [10:40] a prompt may need to create and test the change,  but they may not need permission to approve or or [10:46] deploy it. Role-based access gives each  person only the permission they need. [10:52] So, it reduces accidental changes, limits the  impact of compromised access, and gives you a [10:57] clear record of who drafted, approved, and  deployed each version. So, that's control. [11:05] Now, let's look at what happens once the  agent is interacting with customers at scale. [11:12] So, we're at oversight, and it  has two levels. On the left, [11:16] a human evaluator is reviewing individual  conversations, and they can score qualities of [11:22] the AI agent that require judgments, such things  as tone, clarity, completeness, correctness, [11:28] and whether the customer's problem was  actually resolved in the conversation. [11:32] So, humans are good at that kind of nuance.  But, no human team can manually review [11:38] every conversation an AI agent handles in  production. So, that's where the system comes in. [11:44] On the right, automated evaluators review  conversations at scale. They can detect patterns [11:50] such as hallucinations, harmful content, policy  violations, banned phrases, or missed safeguards. [11:57] So, the system provides coverage. Humans provide  judgment. And neither is enough on its own. [12:03] Automated evaluation can miss context, and  human-only review can't keep up with the volume. [12:10] Oversight works when both are operating together. [12:18] But, most issues can go through  review. Um but some can't wait. [12:22] Here, the system is continuously monitoring the  agent. It shows which conversations are on policy, [12:27] meaning it's working as intended, which require  review, and which have been been flagged. [12:33] But, if those flags reveal a serious pattern,  the operator needs to be able to act immediately. [12:39] A kill switch is a simple capability,  but it represents an important principle. [12:43] Autonomy should never mean that the system  becomes un- unstoppable. A person must [12:50] always be able to intervene when the agent  behaves in a way the business did not expect. [12:56] So far, the stack lets us detect problems, trace  decisions, control changes, review outcomes, [13:01] and stop the agent. So, the next question  is, can the system also fix what went wrong? [13:09] And this is where we move into the  fourth layer, self-improvement. [13:12] Uh so, AI can do all of this. It can It can  find patterns across thousands of conversations. [13:18] It can propose fixes. I can test and ship  improved versions within the right controls. [13:24] And even many of the operational governance checks  can be automated. So, the work itself can run [13:29] end-to-end without a person manually completing  every step. But, that doesn't mean the human [13:35] disappears. People still define the objective,  set the boundaries, choose a level of autonomy, [13:40] and decide when the system has earned permission  to operate on its own. Human moves from doing [13:46] every step steering step to steering and  governing the system that does the work. [13:52] And so, we call this zero-touch improvement. It's  a continuous loop that that improves the system. [14:00] Uh we have four stages here. Self-build proposes  a specific fix based on the observed failure. [14:07] The self-test test runs the fix on the test sets  against the actual conversations. Self-evaluate [14:15] goes through a much more rigorous uh test phase on  much larger corpus of uh conversations. And then [14:22] finally, self-improve promotes the change only  after it's been proven and has met the deployment [14:27] rules. And then the loop starts again. Zero-touch  doesn't mean zero accountability though. It means [14:34] the repetitive work can run continuously while  humans set the direction and the boundaries. [14:42] So, we'll look at this loop uh in more detail. So,  first, the system researches conversations where [14:48] something went wrong. Uh it looks for patterns  such as low CSAT scores, repeated questions from [14:53] the user, or or unusually long resolution times.  Then it forms a hypothesis. It proposes a precise [14:59] change and explains both the reasoning  and the outcome it expects after the fix. [15:06] And a human can review that hypothesis. Um  and the human can approve it, reject it, [15:11] or perhaps converse with it and redirect it  to a better solution. Next, the system tests [15:17] the proposed change in staging against real  conversations that had previously failed. [15:22] In this example, it's testing against 34 failed  conversations and each iteration it performs [15:29] uh the scores are better until the until  finally the newest version resolves all 34. [15:37] And only then does a version deploy to production.  And even after deployment, you still have the [15:43] version history and the one-click rollback just  in case. The important point is that nothing ships [15:48] simply because the AI suggested it. The change has  to earn its way into production through evidence. [15:55] So, if the system can research a failure,  propose a fix, test it, and deploy it, [15:59] the obvious question is how much of that  loop should be allowed to run on its own? [16:04] Uh the answer is not all of it all at once,  though. Autonomy has to be earned in steps. [16:09] So, we can see level zero um see it.  The system identifies failure patterns, [16:14] but the team, humans, still decide what to change  or and does the work. The next level is approve [16:20] it. The AI proposes a fix and presents evidence,  but the person decides whether to move forward. [16:26] Next is steer it to get better. The human defines  the objective and the boundaries, while the system [16:31] runs more of the research, testing, and deployment  loop. And finally, only after that process has [16:36] repeatedly proven itself can can the system get  better at getting better. It can identify new [16:41] problems and run the improvement loop continuously  within boundaries the team has already approved. [16:50] So, the goal isn't to move every workload  to the final step. A low-risk support answer [16:55] and a high-value refund policy shouldn't  receive the same autonomy. The point is that [17:01] evidence comes before permission. You don't switch  autonomy on, the switch the system earns it. [17:10] And that's the main idea behind the trust stack.  Observability lets us see what the agent is doing. [17:15] Control lets you define the boundaries. Oversight,  let's you review the outcomes and intervene. [17:22] And only then can self-improvement operate  safely. If one of the lower layers is missing, [17:28] more autonomy creates more uncertainty. And when  those layers are strong, autonomy can produce [17:33] more learning without giving up control.  Autonomy is earned one layer at a time. [17:40] So, what do you do with this on Monday morning?  Uh use this checklist. Run these eight questions [17:45] against every workflow before you give it more  autonomy. I won't I won't read through all eight. [17:50] Uh the test is simple. Can you see what happened?  Can you control and reverse it? Can a human review [17:56] it and stop it? And can the system prove  that its change is better before it ships? [18:02] Take one workflow and answer every question with  a clear yes or no. Every no becomes an item on [18:08] your autonomy backlog. When every answer is yes,  that workflow is ready to earn more autonomy. [18:15] At the beginning, I I asked the question, when  your agent is quiet quietly wrong, who finds [18:20] out first, you or your customer? The trust  stack is how you make sure the answer is you. [18:26] That's how you give an AI agent more autonomy  without giving up control. Thank you.