Google Engineer Urges Layered Evals to Stop Rogue Enterprise AI Agents

Who Watches the Agents? Building Gen AI Evaluation Suites for Complex Architectures With Google — Ai4

Ai4

Static academic benchmarks fail to predict multi-step agent reliability in production, requiring enterprises to enforce deterministic assertion checks and execution sandboxes before granting models live tool access.

AI-assisted summary of Who Watches the Agents? Building Gen AI Evaluation Suites for Complex Architectures With Google — original source. Check the original for full context.

Source published:

Detailed notes

## Background Published by Ai4 on YouTube on August 19, 2026, this presentation features Jasleen Singh, Lead Forward Deployed Engineer at Google. According to the description, the session explores how to bridge the trust gap in enterprise multi-agent workflows by designing rigorous evaluation suites for dynamic reasoning, tool use, OAuth integrations, and automated prompt optimization.

## TL;DR Moving autonomous AI agents from hackathon prototypes to production reveals a critical trust gap that static academic benchmarks like MMLU fail to address. To safely govern complex, non-linear reasoning and tool execution, engineering teams should implement a layered 'three-step sandwich' evaluation architecture combining deterministic assertion checks, rubric-bound LLM judges, and isolated test sandboxes. Furthermore, teams must replace unscalable manual prompt begging with automated prompt optimizers and integrate continuous evaluation into automated CI/CD pipelines as non-negotiable pre-flight checklists.

## Atmosphere & style The talk combines crisp, pragmatic engineering wisdom with sharp, memorable humor, using analogies like rogue interns and toddlers behind the steering wheel to vividly illustrate production risks. Jasleen Singh delivers an authoritative, fast-paced walkthrough that cuts through generative AI hype with clear architectural discipline. It feels like an urgent, hands-on masterclass for enterprise practitioners transitioning from chaotic 'vibe coding' to battle-tested, defense-in-depth deployment.

## Key takeaways - [00:04](https://www.youtube.com/watch?v=HVgC05wHGCw&t=4s) Unlike conversational chatbots, modern autonomous AI agents can execute real-world actions across APIs and databases, making guardrails and oversight critical to prevent catastrophic operational mistakes. - [01:31](https://www.youtube.com/watch?v=HVgC05wHGCw&t=91s) A significant trust gap exists between rapidly prototyping an agent on a few curated prompts and achieving production readiness across thousands of messy, real-world inputs. - [02:47](https://www.youtube.com/watch?v=HVgC05wHGCw&t=167s) Public static benchmarks like MMLU fail to evaluate production agents because they test single-step reasoning in isolation rather than dynamic, multi-step execution and error recovery. - [04:00](https://www.youtube.com/watch?v=HVgC05wHGCw&t=240s) The 'three-step sandwich' evaluation architecture combines deterministic assertion checks, rubric-calibrated LLM-as-a-judge semantic evaluations, and isolated deterministic sandboxes. - [06:01](https://www.youtube.com/watch?v=HVgC05wHGCw&t=361s) Evaluation suites must stress-test multi-turn and nonlinear interactions to ensure agents handle sudden user topic switches without losing active transaction context. - [07:20](https://www.youtube.com/watch?v=HVgC05wHGCw&t=440s) Allowing LLMs direct API execution without validation is dangerous; architectures must enforce strict pre-execution schema checks, input sanitization, and prompt injection defenses. - [08:40](https://www.youtube.com/watch?v=HVgC05wHGCw&t=520s) Enterprise workloads require concentric rings of defense, ensuring LLMs never have direct, unfiltered access to production databases without sandbox isolation to contain blast radius. - [09:46](https://www.youtube.com/watch?v=HVgC05wHGCw&t=586s) Automated prompt optimization tools systematically tune prompts against failure datasets through closed-loop evaluation, replacing ineffective manual 'prompt begging'. - [11:09](https://www.youtube.com/watch?v=HVgC05wHGCw&t=669s) Evaluation must operate as an automated CI/CD pre-flight checklist triggered on every model, prompt, or tool modification to block regressive deployments. - [12:17](https://www.youtube.com/watch?v=HVgC05wHGCw&t=737s) Deploying small, fine-tuned judge models within continuous evaluation pipelines lowers latency and operational costs while maintaining rigorous rubric-based oversight.

## Detailed walkthrough ### [00:04](https://www.youtube.com/watch?v=HVgC05wHGCw&t=4s) The Rogue Intern: The Reality of Autonomous AI Agents

Building AI today has moved far beyond simple conversational chatbots designed solely to answer user questions. Modern development focuses on autonomous AI agents empowered to take concrete actions in the real world by interacting with internal tools, external APIs, and corporate databases.

To illustrate the danger of deploying these systems without strict guardrails, consider the analogy of hiring a brilliant intern with an off-the-charts IQ. This intern can summarize a 500-page manual in two seconds and work 24/7 without a single coffee break. However, if you immediately grant them access to your company’s internal database, corporate email, and credit card, chaos quickly ensues. Checking in ten minutes later reveals that the intern has handed out $50,000 in refunds to happy customers, booked a personal first-class ticket to Hawaii, and replied to an angry customer email with a poem contemplating an existential crisis. This mirrors the present reality of autonomous agents: they have the power to execute decisions, making rigorous oversight essential.

---

[01:31](https://www.youtube.com/watch?v=HVgC05wHGCw&t=91s) The Weekend Hackathon vs. Monday Morning Reality Curve

There is a stark contrast between rapid early prototyping and production readiness, visualized as the "weekend hackathon versus the Monday morning reality curve." Modern development frameworks and vibe coding make it trivial to spin up an impressive end-to-end demo in roughly two hours over three cups of coffee. When presented to leadership, the immediate reaction is excitement: "Wow, this is the future. This is amazing. When can we go live?"

```mermaid flowchart LR accTitle: Prototype Speed vs Production Confidence subgraph Prototype["Hackathon Phase (Fast)"] direction TB A["Frameworks & Vibe Coding"] --> B["Working Demo (2 Hours)"] B --> C["5 Curated Prompts"] end subgraph Reality["Production Reality (The Trust Gap)"] direction TB D["10,000 Real-World Inputs"] --> E["Edge Cases & Failures"] E --> F["Confidence Flatlines"] end Prototype -->|"The Trust Gap"| Reality ```

The moment production deployment approaches, enterprise confidence flatlines. Getting an agent to succeed on five carefully curated, hand-picked prompts takes an afternoon, but ensuring it operates reliably across 10,000 messy, real-world user inputs takes months of engineering. Bridging this "trust gap" requires abandoning guesswork in favor of continuous, systematic evaluation suites.

---

[02:47](https://www.youtube.com/watch?v=HVgC05wHGCw&t=167s) Why Public Static Benchmarks Fail in Production

When selecting models, engineering teams often look at public leaderboards and assume that a model scoring 89% on MMLU is ready to ship. However, relying on public static benchmarks for production agent architectures is like "judging a pilot's flying skills based upon how well they did in their written driving test."

Static academic benchmarks evaluate large language models strictly on single-step, multiple-choice questions within an isolated bubble. In contrast, enterprise agents operate in dynamic, multi-step loops where they must query databases, call external APIs, and recover from failures. A static benchmark cannot determine whether an agent will gracefully handle a 500 server error on step three of a five-step workflow or completely lose its mind and derail the entire user transaction.

---

[04:00](https://www.youtube.com/watch?v=HVgC05wHGCw&t=240s) The Three-Step Sandwich Evaluation Architecture

Evaluating complex, multi-step agents requires a structured, layered evaluation methodology known as the "three-step sandwich."

```mermaid flowchart TD accTitle: The Three-Step Evaluation Sandwich Architecture subgraph Step1["Layer 1: Assertion-Based Testing"] A["Deterministic Binary Checks<br/>(Valid JSON / SQL, Character Limits, Formats)"] B{"Pass Assertions?"} A --> B end subgraph Step2["Layer 2: LLM-as-a-Judge"] C["Open-Ended Semantic Evaluation<br/>(Hallucination, Groundedness, Politeness)"] D["Calibrated with Strict Rubrics"] C --- D end subgraph Step3["Layer 3: Deterministic Sandboxes"] E["Isolated Test APIs & Databases<br/>(Safe environment to test failure recovery)"] end

B -->|Fail| F["Halt & Block Response Immediately"] B -->|Pass| Step2 Step2 --> Step3 ```

- **Assertion-Based Testing:** Fast, low-cost, binary deterministic checks applied immediately to model outputs. This tests structural validity: Is the response valid JSON? Is the generated SQL syntactically correct and legal? Does the output stay within required character limits and match designated section templates? If an agent fails here, execution halts instantly. - **LLM-as-a-Judge:** Used for semantic, open-ended evaluations such as detecting hallucinations, measuring factual groundedness, checking answer relevancy, and assessing politeness. To prevent the judge model from hallucinating its own criteria, it must be paired with strict, deterministic rubrics. - **Deterministic Sandboxes:** Analogous to testing an autonomous vehicle in a closed simulator rather than on a crowded highway. Sandboxes expose agents to simulated APIs and isolated test databases where unexpected agent behavior costs nothing.

---

[06:01](https://www.youtube.com/watch?v=HVgC05wHGCw&t=361s) Handling Nonlinear Context Switching and User Chaos

Human communication is inherently nonlinear and messy. In an idealized workflow, a travel booking agent expects a clean sequential path: select a flight, choose a seat, and complete payment. In practice, right as the agent reaches the checkout stage, a human user will introduce tangential questions: "Wait, is the baggage included? And what's like the weather in Paris next week?"

A fragile agent becomes confused by the sudden topic shift, drops the overall transaction context, and may attempt to book a new trip to Paris. A robust, well-architected agent performs seamless context switching: it acknowledges the payment state, answers the baggage policy and weather queries, and smoothly returns to finalizing the booking. Evaluation suites must rigorously stress-test multi-turn dialogues and nonlinear interruptions before exposing the system to live end users.

---

[07:20](https://www.youtube.com/watch?v=HVgC05wHGCw&t=440s) Safeguarding Tool Execution and API Integrations

Granting an LLM direct execution access to APIs is "like letting a toddler drive a car. They might steer it in the right direction for the first 5 seconds, but eventually they are going through a garage door."

When an agent translates natural language intent into a live database query or an automated financial transfer, it leaves the safety of the model bubble and interacts with external infrastructure. In this environment, hallucinating an extra zero in an argument or outputting a malformed payload is no longer a simple typo—it is an active system outage or a major compliance violation. Evaluation suites must enforce strict pre-execution schema validations, defend against prompt injection vectors, and sanitize all input arguments before any live call is triggered.

---

[08:40](https://www.youtube.com/watch?v=HVgC05wHGCw&t=520s) Defense in Depth: Securing Sensitive Enterprise Workloads

Securing high-stakes enterprise systems requires treating internal data like a high-security bank vault. Rather than handing master keys to every component, architectures require concentric rings of defense.

```mermaid flowchart TD accTitle: Concentric Rings of Defense for Agent Tool Execution subgraph OuterRing["Perimeter Defenses"] A["OAuth Permissions & Access Scopes"] B["Rate Limits & Quota Protections"] end subgraph MiddleRing["Isolation Layer"] C["Execution Sandboxes"] D["Input Sanitization & Schema Validation"] end subgraph Core["Protected Core"] E[("High-Stakes Production Databases & Sensitive Data")] end

OuterRing --> MiddleRing MiddleRing --> Core ```

The golden rule of enterprise agent architecture is to never allow an LLM direct, unfiltered access to production databases without an intermediary execution isolation layer. If an agent executes a rogue or destructive command, the blast radius is safely absorbed entirely by the isolated sandbox rather than impacting production users.

---

[09:46](https://www.youtube.com/watch?v=HVgC05wHGCw&t=586s) Automated Prompt Optimization vs. Prompt Begging

When evaluation suites detect an agent failure, engineers often resort to hours of manual "prompt begging"—typing desperate instructions like *"Please don't do this. Please don't die. I'll give you a $200 tip."* Pleading in all caps is an unscalable, ineffective mitigation strategy.

Automated solutions like the Vertex AI Prompt Optimizer systematically refine system prompts through closed-loop evaluation cycles:

- Continuously runs evaluations across failure datasets to iteratively tune phrasing. - Discovers the ideal equilibrium between prompt specificity (rigid constraints and guardrails) and flexibility (reasoning capacity for autonomous decisions). - Prevents over-fitting prompts with excessive few-shot examples that constrain generalized performance. - Acts as a dedicated "personal trainer" specifically tailored to optimize the agent's target use case.

---

[11:09](https://www.youtube.com/watch?v=HVgC05wHGCw&t=669s) Continuous CI/CD Pipelines as Pre-Flight Checklists

Treating evaluation as a one-time "final exam" before launching an agent is a critical architectural mistake. Evaluation must operate as a continuous, automated CI/CD pipeline, functioning identically to an airline pilot's pre-flight checklist. Even if a pilot has flown an aircraft a thousand times, the checklist is executed prior to every departure.

Every modification—whether updating a system prompt, modifying tool APIs, changing routing parameters, or swapping underlying foundation models—must trigger the automated evaluation suite. If semantic reasoning scores drop or toxicity metrics rise, the pipeline automatically halts the deployment, ensuring systems are never shipped to production on a blind check.

---

[12:17](https://www.youtube.com/watch?v=HVgC05wHGCw&t=737s) Enterprise Production Evaluation Cheat Sheet

| Core Pillar | Implementation Focus | Key Requirements | |:--- |:--- |:--- | | **1. Deploy Assertion Evals** | Deterministic sanity checks at every agent step | Eliminate ambiguous definitions of "good enough." Enforce strict JSON schemas, valid SQL syntax, character budgets, and formatting constraints across every sub-agent handoff. | | **2. Hardcore Tool Validation** | Blast-radius containment | Enforce pre-execution sandboxes, input sanitization, and strict OAuth boundaries before any tool interacts with live enterprise state. | | **3. Build Automated CI/CD Pipelines** | Continuous automated regression testing | Combine per-step assertion testing with rubric-bound LLM-as-a-judge evaluations. Utilize small, fine-tuned judge models to dramatically lower evaluation latency and cost while maintaining rigorous oversight. |

## Notable quotes > [00:04](https://www.youtube.com/watch?v=HVgC05wHGCw&t=4s) Checking in ten minutes later reveals that the intern has handed out $50,000 in refunds to happy customers, booked a personal first-class ticket to Hawaii, and replied to an angry customer email with a poem contemplating an existential crisis. — Jasleen Singh

> [01:31](https://www.youtube.com/watch?v=HVgC05wHGCw&t=91s) Wow, this is the future. This is amazing. When can we go live? — Jasleen Singh

> [01:31](https://www.youtube.com/watch?v=HVgC05wHGCw&t=91s) Getting an agent to succeed on five carefully curated, hand-picked prompts takes an afternoon, but ensuring it operates reliably across 10,000 messy, real-world user inputs takes months of engineering. — Jasleen Singh

> [02:47](https://www.youtube.com/watch?v=HVgC05wHGCw&t=167s) judging a pilot's flying skills based upon how well they did in their written driving test. — Jasleen Singh

> [02:47](https://www.youtube.com/watch?v=HVgC05wHGCw&t=167s) A static benchmark cannot determine whether an agent will gracefully handle a 500 server error on step three of a five-step workflow or completely lose its mind and derail the entire user transaction. — Jasleen Singh

> [06:01](https://www.youtube.com/watch?v=HVgC05wHGCw&t=361s) Wait, is the baggage included? And what's like the weather in Paris next week? — Jasleen Singh

> [07:20](https://www.youtube.com/watch?v=HVgC05wHGCw&t=440s) like letting a toddler drive a car. They might steer it in the right direction for the first 5 seconds, but eventually they are going through a garage door. — Jasleen Singh

> [07:20](https://www.youtube.com/watch?v=HVgC05wHGCw&t=440s) hallucinating an extra zero in an argument or outputting a malformed payload is no longer a simple typo—it is an active system outage or a major compliance violation. — Jasleen Singh

> [09:46](https://www.youtube.com/watch?v=HVgC05wHGCw&t=586s) Please don't do this. Please don't die. I'll give you a $200 tip. — Jasleen Singh

> [11:09](https://www.youtube.com/watch?v=HVgC05wHGCw&t=669s) Even if a pilot has flown an aircraft a thousand times, the checklist is executed prior to every departure. — Jasleen Singh

## Mentioned Jasleen Singh, Google, Ai4, Vertex AI Prompt Optimizer, MMLU, OAuth

## Video description Jasleen Singh - Lead Forward Deployed Engineer, Google

While the industry spent the last year racing to build autonomous multi-agent systems, a critical question remains: how do we actually trust them in production? As enterprise adoption scales, the bottleneck has definitively shifted from agent creation to rigorous evaluation. In this talk, I will explore the practical realities of testing complex architectures where traditional LLM benchmarks fall short. I will demonstrate how to design robust evaluation suites for dynamic reasoning and tool use, specifically focusing on safeguarding sensitive workflows like strict OAuth integrations and data retrievals. You will also learn practical techniques for utilizing tools like the Vertex AI Prompt Optimizer to continuously refine agent behavior. Join me to discover how to move your multi-agent workflows out of the prototype phase and into secure, measurable enterprise production.

[Source](https://www.youtube.com/watch?v=HVgC05wHGCw)

中文简报

谷歌工程师敦促采用分层评估以遏制失控的企业AI智能体 静态学术基准无法预测生产环境中多步智能体的可靠性，这要求企业在授予模型实时工具调用权限之前，必须执行确定性断言检查和运行沙盒。 将自主AI智能体从黑客松原型推向生产环境暴露出MMLU等静态学术基准无法解决的关键信任鸿沟。为了安全管控复杂的非线性推理和工具执行，工程团队应实施分层的“三步三明治”评估架构，将确定性断言检查、受评分细则约束的LLM裁判以及隔离测试沙盒结合起来。此外，团队必须用自动化提示词优化器取代不可扩展的人工作求式提示词调试，并将持续评估作为不可妥协的发布前检查清单集成到自动化CI/CD流水线中。 与对话式聊天机器人不同，现代自主AI智能体能够跨API和数据库执行现实操作，这使得安全护栏和监督机制对于防止灾难性业务差错至关重要。 (0:04) 在依靠少数精选提示词快速构建智能体原型与面向成千上万复杂混乱的真实输入实现生产就绪之间，存在着巨大的信任鸿沟。 (1:31) 诸如MMLU等公开静态基准无法评估生产级智能体，因为它们孤立地测试单步推理，而非动态的多步执行与错误恢复能力。 (2:47) “三步三明治”评估架构结合了确定性断言检查、经评分细则校准的LLM裁判语义评估，以及隔离的确定性沙盒。 (4:00) 评估套件必须对多轮与非线性交互进行压力测试，以确保智能体在应对用户突发的话题切换时不会丢失当前的事务上下文。 (6:01)

Reading summary

According to the presentation by Ai4 and Google, deploying autonomous AI agents into enterprise production requires shifting from static benchmarks and manual prompt engineering to continuous, multi-layered evaluation suites. Because autonomous agents execute live actions rather than merely conversing, enterprise reliability depends on deterministic assertions, semantic judge models, blast-radius containment, and automated regression pipelines. The Production Trust Gap and Benchmark Limitations (1:31) Modern frameworks allow developers to assemble working agent prototypes in hours using a handful of curated prompts, but enterprise confidence typically collapses when moving to production across thousands of unpredictable inputs. Unlike conversational bots that only output text, autonomous agents are granted authority to call external APIs, query corporate databases, and trigger operational transactions. The speakers argue that granting broad permissions without strict guardrails risks severe operational errors, comparing unconstrained agents to an unchecked intern executing arbitrary refunds or bookings. Relying on public static benchmarks like MMLU to determine deployment readiness fails to address these risks. Static benchmarks evaluate language models on single-step, isolated multiple-choice tasks. In contrast, enterprise agents operate in multi-turn, dynamic execution loops where they must chain API calls, query enterprise systems, and recover from intermittent failures, such as server errors, without derailing transactions. The Three-Step Sandwich Evaluation Architecture (4:00) To evaluate multi-step agents systematically, the presentation proposes a layered "sandwich" framework. The first layer applies rapid, low-cost assertion testing to enforce deterministic requirements, verifying that outputs match required character limits, schemas, and valid SQL or JSON syntax. Output that fails this gate is halted immediately. The second layer deploys an LLM-as-a-judge guided by explicit, deterministic rubrics to grade semantic qualities such as groundedness, factual accuracy, answer relevancy, and tone. The third layer subjects agents to deterministic sandboxes containing simulated APIs and test databases, allowing teams to safely monitor failure recovery without incurring real-world costs. Evaluation suites must also stress-test nonlinear context switching. In production, users frequently disrupt sequential workflows—such as a booking process—with unrelated questions about luggage rules or weather. While fragile agents drop transaction states or initiate erroneous tasks, robust systems must answer conversational digressions and smoothly return to the primary task without state corruption. Blast-Radius Containment and Defense in Depth (7:20) When an agent translates natural language instructions into live database updates or financial transactions, hallucinated arguments or malformed payloads can trigger system outages or regulatory violations. The presentation compares unmediated tool execution to letting an untrained driver operate a vehicle, warning that errors inevitably breach safety perimeters if executions are unmonitored. Securing enterprise workloads requires a defense-in-depth architecture modeled as concentric security rings. Outer perimeters enforce OAuth scopes, permission boundaries, and rate limits. An intermediary isolation layer applies pre-execution input sanitization, schema validation, and sandboxing. Under this structure, language models never interact directly with live production databases, ensuring that rogue or destructive commands are absorbed within isolated environments. Automated Prompt Tuning and CI/CD Pipelines (9:46) When agents fail evaluation thresholds, engineering teams frequently rely on manual "prompt begging"—adding ad-hoc, emphatic instructions or artificial incentives into system prompts. The speakers characterize this approach as unscalable and counterproductive. Instead, closed-loop tooling such as the Vertex AI Prompt Optimizer systematically refines prompts against failure datasets, tuning phrasing to balance behavioral guardrails with reasoning flexibility while preventing over-fitting from excessive few-shot examples. Finally, evaluation cannot be treated as a one-time deployment milestone. Analogous to an aviation pre-flight checklist, continuous CI/CD pipelines must automatically execute deterministic assertions and rubric-bound evaluations whenever system prompts, routing logic, tool APIs, or underlying foundation models are modified. If accuracy drops or policy violations rise, the deployment pipeline halts release to prevent blind production regressions. The speakers conclude that deploying autonomous agents shifts engineering priority from initial model selection to systematic evaluation and blast-radius containment. The ongoing challenge for enterprises remains maintaining this rigorous multi-layered oversight without introducing prohibitive evaluation latency or over-constraining the model's reasoning capabilities.

精炼阅读摘要

随着生成式AI从单纯的聊天机器人演进为可直接操作工具与企业数据库的自主智能体，传统静态学术基准已不足以评估其在动态多步场景中的可靠性。根据演讲者阐述，企业必须构建包含确定性断言、模型裁判、沙盒隔离与自动化CI/CD的综合评估工程体系，以弥补原型验证与真实生产之间的信任鸿沟。 原型落差与静态基准在生产中的失效 (1:31) 在黑客松和原型阶段，开发人员利用现代开发框架与体验式编码（vibe coding），仅需两小时和几杯咖啡就能基于5个精心挑选的提示词跑通演示。然而当迈向企业级生产时，面对上万次复杂混乱的真实输入，系统信心往往会断崖式下跌。消除这一信任差距需要工程化的系统性评估，而非停留在猜测阶段。 依赖公开学术基准（如MMLU得分达89%）来评判生产智能体，被演讲者比作“通过驾照笔试成绩来评判飞行员的飞行技术”。静态基准仅在孤立环境中测试单步选择题，无法反映智能体在多步循环中调用API、查询数据库以及在遇到第3步服务报错时能否妥善恢复的能力。 三层三明治评估架构与动态交互应对 (4:00) 针对复杂的多步智能体架构，演讲者提出了“三层三明治”评估框架。底层为断言测试（Assertion-Based Testing），针对输出格式（如JSON有效性、合法SQL语法、字符长度及模板匹配）进行低成本、毫秒级的二元确定性检查，未通过则立刻阻断执行；中层引入以大模型作为裁判（LLM-as-a-Judge），配合严格的评分细则对幻觉、事实依据、相关性与礼貌度进行语义评估；顶层为确定性沙盒（Deterministic Sandboxes），在隔离的模拟API与测试数据库中以零业务成本检验智能体的错误恢复能力。 该评估体系同样用于应对人类混乱的非线性沟通。例如在机票预订过程中，用户常在即将支付时插入关于行李额度或巴黎天气的询问；稳健的智能体架构必须经受住评估套件的多轮对话压力测试，确保其能够自如切换上下文，在解答插话后顺利返回原交易主线。 工具执行安全与同心圆纵深防御体系 (7:20) 直接给予大语言模型执行API的权限极具危险性，演讲者将其比作“让幼儿驾车”，模型幻觉或格式错误不再是简单的笔误，而是可能造成5万美元违规退款或系统瘫痪的严重事故。因此，评估套件必须在实际调用前强制执行严格的模式验证、防御提示词注入并清洗所有输入参数。 在敏感的企业级工作负载中，架构设计需遵循同心圆防御原则：外层部署OAuth权限、访问范围与速率配额；中层设立执行沙盒与输入过滤；被保护的核心数据库绝不开放无过滤的直接访问。通过将潜在的破坏性操作隔离在沙盒中，系统能够完整吸收意外行为的爆炸半径，确保底层业务数据的绝对安全。 自动化提示词优化与持续CI/CD流水线 (9:46) 当评估套件捕获到智能体错误时，依赖工程师手动撰写乞求式提示词（例如用大写字母恳求模型不要出错）已被证明既不可扩展也缺乏实效。通过采用如Vertex AI Prompt Optimizer等自动化工具，系统可以在失败数据集上通过闭环评估迭代调优，在提示词的约束刚性与自主推理灵活性之间寻找平衡，防止因过拟合少量样本而丧失泛化能力。 此外，评估不能被当作系统发布前的一次性“期末考试”，而必须转化为类似于飞机起飞前检查清单的自动化CI/CD流水线。无论是更新系统提示词、修改工具接口、变更路由参数还是更换底层模型，任何微小调整都必须自动触发全套回归测试；一旦推理质量下滑或毒性指标上升，流水线将自动阻断上线，杜绝盲目交付。 构建可靠的自主智能体不仅取决于基座模型的智商，更取决于严密的工程防御与常态化评估闭环。演讲者总结的生产落地准则表明，企业唯有在每一步执行断言检验、利用沙盒限制工具执行的破坏半径，并将模型裁判与CI/CD深度结合，方能确立面向复杂架构的安全防线。

Detailed notes — zh

## Background 本视频由 Ai4 频道于 2026 年 8 月 19 日在 YouTube 平台发布，主讲人为 Google 的前线部署工程师（Lead Forward Deployed Engineer）Jasleen Singh。视频简介指出，随着企业级自主 Multi-Agent 架构的扩张，技术瓶颈已从“构建 Agent”转向“严谨评估”，本次演讲旨在分享超越传统静态基准测试的实战评估体系，并介绍如何借助 Vertex AI Prompt Optimizer 等工具保障敏感工作负载与生产落地的安全可信。 ## TL;DR 演讲指出了企业从“黑客松 Demo”走向“生产环境”时面临的巨大信任鸿沟，强调依赖 MMLU 等静态基准测试无法衡量 Agent 在动态环境与多步工具调用中的真实鲁棒性。为此，讲者提出了涵盖“断言测试、大模型裁判、确定性沙箱”的“三层三明治”评估模型，以及针对敏感资产的同心圆防御架构。最后，演讲呼吁将评估体系转化为如同民航“飞行前检查清单”般的 CI/CD 自动化流水线，利用自动化提示词优化与严格的参数校验，杜绝盲目上线并确保系统的持续稳定运行。 ## Atmosphere & style 整场演讲风格幽默风趣且极富工程务实精神。开篇以“极具破坏力的高智商实习生”和“周末黑客松与周一晨会现实”的比喻迅速抓住听众，生动勾勒出从盲目乐观跌入生产现实的心理落差。主讲人语调清晰坚定、直击要害，在剖析“向大模型哀求给小费”等滑稽反模式的同时，给出了极具操作性的严谨工程解决方案，营造出一种既引人深思又充满落地信心的专业技术分享氛围。 ## Key takeaways - [00:04](https://www.youtube.com/watch?v=HVgC05wHGCw&t=4s) 自主 AI Agent 已经从被动的问答聊天机器人转变为能够代替人类决策并调用 API/数据库的行动实体，如同高智商实习生，若缺乏管控会带来严重的业务与合规风险。 - [01:31](https://www.youtube.com/watch?v=HVgC05wHGCw&t=91s) AI 原型开发与实际投产之间存在巨大的“信任鸿沟”：在 5 个精选 Prompt 上跑通 Demo 只需两小时，但在 10,000 个混沌真实输入下稳定运行需数月测试。 - [02:47](https://www.youtube.com/watch?v=HVgC05wHGCw&t=167s) 依赖 MMLU 等静态公开基准测试评估复杂 Agent 架构是危险的误区，因为静态单步测试无法预测多步循环交互及遭遇 API 错误时的自主恢复能力。 - [02:47](https://www.youtube.com/watch?v=HVgC05wHGCw&t=167s) 评估复杂 Agent 推荐采用“三层三明治”模型：底层断言测试、中层 LLM 裁判（LLM as a Judge）、顶层确定性沙箱。 - [02:47](https://www.youtube.com/watch?v=HVgC05wHGCw&t=167s) 第一层断言测试作为首道防线，通过低成本二元验证（JSON/SQL 合法性、字符限制等）快速拦截格式错误，失败即阻断。 - [02:47](https://www.youtube.com/watch?v=HVgC05wHGCw&t=167s) 第二层大模型裁判负责处理幻觉检测、忠实度等开放式语义评估，但必须绑定严苛的评分细则（Strict Rubrics）以规避裁判自身偏见。 - [02:47](https://www.youtube.com/watch?v=HVgC05wHGCw&t=167s) 第三层确定性沙箱通过隔离的测试数据库与 Mock API 提供真实模拟环境，让系统在任意破坏性测试下的试错成本归零。 - [06:01](https://www.youtube.com/watch?v=HVgC05wHGCw&t=361s) 真实用户交互具有高度非线性特征，健壮的 Agent 必须具备丝滑的上下文切换能力，在解答突发插问后能准确切回原主线流程。 - [06:01](https://www.youtube.com/watch?v=HVgC05wHGCw&t=361s) 工具调用必须前置进行 Schema 校验、入参清洗与 Prompt 注入防御，防止大模型将畸变指令转化为实际数据库变更或扣款事故。 - [08:40](https://www.youtube.com/watch?v=HVgC05wHGCw&t=520s) 保护敏感工作负载需借鉴“同心圆防御”架构，通过严格的 OAuth 细粒度鉴权、隔离沙箱与速率限制层层包裹高价值数据，限制失控时的爆炸半径。 - [08:40](https://www.youtube.com/watch?v=HVgC05wHGCw&t=520s) 应当借助 Vertex AI Prompt Optimizer 等自动化优化工具迭代提示词，在“指令明确性”与“推理灵活性”之间找到最佳平衡，替代低效的人工猜测式修改。 - [11:09](https://www.youtube.com/watch?v=HVgC05wHGCw&t=669s) AI 评估不应是上线前的一次性突击，而应作为类似民航“飞行前检查清单”的持续 CI/CD 流水线，任何代码、模型或 Prompt 变更均自动触发评测并在异常时阻断上线。 ## Detailed walkthrough ### [00:04](https://www.youtube.com/watch?v=HVgC05wHGCw&t=4s) 极具破坏力的高智商“实习生”：自主 AI Agent 的现实风险 演讲以一个生动的思想实验开场：想象你的团队刚刚入职了一位智商极高、能力非凡的实习生——他能在两秒钟内提炼出一份 500 页操作手册的精髓，而且无需喝咖啡就能 24 小时连轴转工作。这听起来如同企业梦寐以求的完美员工。然而，如果你立刻赋予这位实习生访问企业核心数据库、发送公司邮件以及刷企业信用卡的完全权限，结果会怎样？可能仅仅过了十分钟，当你前去查看时，就会震惊地发现他已经向“满意的客户”大方派发了 50,000 美元的赠金、为自己订了一张去夏威夷的头等舱机票，并在一封回复愤怒客户的邮件里写了一首关于“存在主义危机”的哲学诗篇。 这个荒诞却极具警示意义的场景，正是当下 AI Agent 面临的真实写照。技术的发展已经远远超越了过去只能回答固定问题的被动式聊天机器人时代。如今构建的自主 Agent 能够代替人类做出决策并付诸行动：它们进入现实环境，调用内部与外部 API，连接核心数据库，并根据动态环境进行复杂决策。这就引出了本次演讲的核心议题：“究竟谁来监督这些 Agent（Who watches the agents）？” --- ### [01:31](https://www.youtube.com/watch?v=HVgC05wHGCw&t=91s) 周末黑客松与周一现实：跨越生产落地的“信任鸿沟” 在 AI 原型开发与实际投产之间，存在一条残酷的现实曲线，被称为“周末黑客松 vs. 周一晨会现实曲线”。借助现代开发框架和直观的代码生成工具，构建一个端到端的可用原型往往只需要“两个小时和三杯咖啡”。开发者在黑客松或周五下午迅速跑通 Demo，向管理层展示时，领导层通常会惊呼：“这简直就是未来，太不可思议了！我们什么时候可以正式上线？” 然而，一旦进入周一的生产环境部署阶段，工程团队的信心曲线便会断崖式暴跌并彻底走平。造成这种巨大落差的核心原因在于：让一个 Agent 在 5 个精心挑选的测试 Prompt 上完美运行只需一个下午；但要让它在真实世界面对 10,000 个不可预测的用户输入时稳定工作，却需要耗费数月之久。这种原型与生产之间的巨大鸿沟就是“信任差距”（Trust Gap），而跨越这道鸿沟的唯一路径，就是建立持续且系统化的评估体系。 ```mermaid flowchart LR accTitle: The Trust Gap in AI Agent Deployment subgraph Prototype["原型开发阶段 (两小时)"] A["5 个精选 Prompt"] --> B["快速跑通 Demo"] B --> C["管理层高度兴奋"] end subgraph Production["生产落地阶段 (数月探索)"] D["10,000 真实用户输入"] --> E["边界用例崩溃"] E --> F["部署信心跌至冰点"] end C -.->|"跨越信任鸿沟 (Trust Gap)|" D style Prototype fill:#fef3c7,stroke:#f59e0b,stroke-width:2px style Production fill:#fee2e2,stroke:#ef4444,stroke-width:2px ``` --- ### [02:47](https://www.youtube.com/watch?v=HVgC05wHGCw&t=167s) 告别静态排行榜：评估复杂架构的“三层三明治”模型 当技术团队挑选底层模型时，技术主管往往会指着公开排行榜说：“看，X 模型在 MMLU 基准上得分高达 89%，我们就用它直接上线吧！”但这是一种极度危险的误区。在生产环境中依赖公开的静态基准测试，就好比“通过一个人驾照笔试考了多少分，来评估他驾驶飞机的飞行技术”。公开基准通常只测试大模型在封闭气泡中回答单步单选题的能力，而现实中的 AI Agent 绝非身处真空，它们必须在多步循环中做出动作、与 API 和数据库交互，并具备从错误中自主恢复的韧性。如果一个五步骤任务在第三步遭遇了 API 500 错误，静态基准完全无法预测 Agent 究竟会优雅重试，还是彻底失控发疯。 为了全面评估复杂的 Agent 架构，推荐采用“三层三明治”（Three-Step Sandwich）评估体系： ```mermaid flowchart TD accTitle: Three-Step Evaluation Sandwich Architecture subgraph Layer1["底层：断言测试 (Assertion-based Testing)"] L1["快速、低成本、二元验证<br/>• JSON 格式是否合规<br/>• SQL 语句是否合法<br/>• 输出字符与分段限制"] end subgraph Layer2["中层：大模型裁判 (LLM as a Judge)"] L2["开放式语义评估 + 严格评分细则 (Rubrics)<br/>• 幻觉检测<br/>• 事实依据与忠实度<br/>• 礼貌程度与答复相关性"] end subgraph Layer3["顶层：确定性沙箱 (Deterministic Sandboxes)"] L3["真实模拟环境<br/>• 隔离测试数据库与 Mock API<br/>• 试错成本归零<br/>• 验证多步错误恢复能力"] end Layer1 --> Layer2 --> Layer3 ``` - **第一层：断言测试（Assertion-based Testing）**：作为第一道防线，进行快速、廉价的二元判断（Pass/Fail）。例如：输出是否为合法 JSON？生成的 SQL 是否合法？是否满足字符上限？是否包含预期的段落结构？一旦断言失败，系统应立即阻断响应，禁止继续执行。 - **第二层：大模型作为裁判（LLM as a Judge）**：用于处理开放性评估。判断模型回复是否存在幻觉、是否具备事实依据、语调是否得体礼貌。为防止裁判模型本身产生偏见或幻觉，必须为其绑定极其严苛的评分细则（Strict Rubrics），量化各项指标。 - **第三层：确定性沙箱（Deterministic Sandboxes）**：正如绝不能将一辆刚造好的自动驾驶测试车直接开上繁忙的高速公路一样，Agent 必须在隔离的测试数据库和 Mock API 环境中进行压力测试，使系统在任意破坏性测试下的试错成本完全归零。 --- ### [06:01](https://www.youtube.com/watch?v=HVgC05wHGCw&t=361s) 应对混沌人类交互与工具调用防护：多轮非线性的上下文切换 在真实的业务场景中，人类用户的行为是极其“混沌”且非线性的。在理想流程中，用户会依次选航班、挑座位、最后完成支付；然而在现实中，当 Agent 刚推进到支付页面时，用户可能会突然插话：“等等，票价包含托运行李额吗？顺便问一下巴黎下周天气如何？”此时，脆弱的 Agent 会瞬间迷失上下文，甚至把当前任务搞混成给用户预订飞往巴黎的机票；而健壮且具备韧性的 Agent 则能够丝滑完成上下文切换（Context Switching），先解答附带疑问，随后稳妥地将对话拉回主线任务。 ```mermaid flowchart TD accTitle: Handling Chaotic User Context Switching A["主线任务：确认订单并进入支付"] --> B{"用户非线性插问：<br/>'含行李吗？巴黎天气如何？'"} B -->|"脆弱 Agent (Context Lost)|" C["上下文错乱：<br/>直接开始预订巴黎机票（崩溃）"] B -->|"高韧性 Agent (Resilient)|" D["暂存主线状态，回答插问"] D --> E["优雅切换回原支付流程"] style C fill:#fee2e2,stroke:#ef4444,stroke-width:2px style E fill:#dcfce7,stroke:#22c55e,stroke-width:2px ``` 与此同时，工具调用环节的防护至关重要。赋予大模型直接调用 API 的权限，就好比“让一个两岁幼儿去开车”——在前五秒钟他可能碰巧把着方向盘走直线，但要不了多久他就会把车直接撞进车库大门。当 Agent 将意图转化为数据库查询或扣款指令时，它已经走出了无害的沙盒。模型一旦多臆造了一个零或输出了畸变格式，带来的就不再只是错别字，而是合规灾难或系统级宕机。因此，评估套件必须前置覆盖输入清洗（Input Sanitization）、Schema 校验与 Prompt 注入防御，绝不能等到生产事故发生后才被动排查。 --- ### [08:40](https://www.youtube.com/watch?v=HVgC05wHGCw&t=520s) 敏感工作负载的同心圆防御与 Vertex AI 提示词优化器 对于核心资产与敏感工作负载的防护，应当借鉴银行金库的“同心圆防御”（Concentric Rings of Defense）架构。企业不会把金库钥匙随手分发给每一个过客，而是设立多道安检门、门禁卡与安保岗哨。在 Agent 架构中，最核心的是高价值敏感数据，向外依次由严格的 OAuth 细粒度权限、隔离执行沙箱以及访问速率限制（Rate Limits）层层包裹。绝不能允许大语言模型在没有隔离层的情况下直连生产数据库，必须由沙箱来承受失控时的“爆炸半径”（Blast Radius）。 ```mermaid flowchart TD accTitle: Concentric Rings of Defense for Sensitive Workloads subgraph Defense["同心圆防御架构"] R1["外层防护：OAuth 鉴权与访问速率限制 (Rate Limits)"] R2["中间隔离：执行隔离沙箱 (Execution Isolation Layer)"] R3["核心资产：高价值敏感数据与生产数据库 (Production DB)"] R1 --> R2 --> R3 end style R3 fill:#fee2e2,stroke:#ef4444,stroke-width:2px style R2 fill:#fef3c7,stroke:#f59e0b,stroke-width:2px style R1 fill:#e0e7ff,stroke:#6366f1,stroke-width:2px ``` 当评估体系检测出 Agent 逻辑缺陷或执行失败时，传统的做法往往极其低效且滑稽：工程师会花上四个小时疯狂修改 Prompt，甚至在提示词里低声下气地求模型：“求求你千万别这么做，如果你做对了，我给你 200 美元小费”。用全大写字母去“哀求”模型绝不是工程化的解决方案。 此时应当引入自动化工具（如 Vertex AI Prompt Optimizer）。自动化优化器会在持续的评估循环中反复迭代，寻找兼顾“指令明确性”与“推理灵活性”的最佳平衡点（Sweet Spot）。它既不会过度约束 Prompt 而扼杀 Agent 在新场景下的独立决策能力，也不会过于宽泛导致失控，其效果就像为特定的业务 Agent 雇佣了一位专属的“私人健身教练”。 --- ### [11:09](https://www.youtube.com/watch?v=HVgC05wHGCw&t=669s) 构建 CI/CD 飞行前检查清单：可落地的 Agent 评估速查指南 工程团队在认知上需要发生根本转变：评估绝不是 Agent 正式上线前突击进行的一次“期末考试”，而是一个贯穿生命周期的持续流水线。这类似于民航机长在起飞前必须严格执行的“飞行前检查清单”（Pre-flight Checklist），哪怕一位机长已经驾驶同一架飞机飞行过上千次，每一次起飞前也绝不会跳过任何一项核对。CI/CD 流水线就是 AI 系统的飞行前检查。无论是修改了 System Prompt、变更了 API，还是底层升级了基础模型，代码提交都必须自动触发全套评测。一旦发现推理能力下降或毒性指标上升，系统必须立刻就地阻断上线，坚决杜绝在毫无保障的情况下盲目投产。 ```mermaid flowchart LR accTitle: Continuous CI/CD Pre-flight Checklist Pipeline A["代码/Prompt/模型变更"] --> B["CI/CD 自动化流水线"] subgraph Checks["自动化检查项目"] C1["全链路断言测试"] C2["严格 Rubric 的 LLM 裁判"] end B --> Checks Checks --> D{"指标是否达标？<br/>(推理下降? 毒性上升?)"} D -->|"异常"| E["立即阻断上线 (Block)"] D -->|"全部通过"| F["安全发布至生产环境"] style E fill:#fee2e2,stroke:#ef4444,stroke-width:2px style F fill:#dcfce7,stroke:#22c55e,stroke-width:2px ``` 针对构建高可用 Gen AI 评估体系，本次演讲总结了三大核心实战法则（Cheat Sheet）： - **部署确定性断言评估（Deploy Assertion Evals）**：坚决摒弃“只要效果差不多就行”的主观臆测。借鉴传统软件工程的单元测试思想，在 Agent 的每个决策节点（子 Agent 派发、技能路由、工具与 API 调用）强制校验 JSON/SQL 合法性、字符限制与格式规范。 - **实施硬核的工具防护验证（Hardcore Tool Validation）**：在 LLM 触碰任何生产环境之前，强制接入执行隔离沙箱，前置拦截注入攻击并对入参进行严格清洗，将所有失控行为牢牢限制在安全边界内。 - **建立现代 CI/CD 自动化流水线（Build CI/CD Pipelines）**：将断言测试与基于严格 Rubrics 的 LLM as a Judge 深度集成到持续集成流程中。可以选用轻量级、经过微调的小模型充当裁判，在大幅削减评测成本与耗时的同时，确保 Agent 的每一次迭代都处于严密的自动化监控之下。 ## Notable quotes > [00:04](https://www.youtube.com/watch?v=HVgC05wHGCw&t=4s) 究竟谁来监督这些 Agent（Who watches the agents）？ — Jasleen Singh > [01:31](https://www.youtube.com/watch?v=HVgC05wHGCw&t=91s) 两个小时和三杯咖啡 — Jasleen Singh > [01:31](https://www.youtube.com/watch?v=HVgC05wHGCw&t=91s) 这简直就是未来，太不可思议了！我们什么时候可以正式上线？ — Jasleen Singh > [02:47](https://www.youtube.com/watch?v=HVgC05wHGCw&t=167s) 通过一个人驾照笔试考了多少分，来评估他驾驶飞机的飞行技术 — Jasleen Singh > [06:01](https://www.youtube.com/watch?v=HVgC05wHGCw&t=361s) 等等，票价包含托运行李额吗？顺便问一下巴黎下周天气如何？ — Jasleen Singh > [06:01](https://www.youtube.com/watch?v=HVgC05wHGCw&t=361s) 让一个两岁幼儿去开车 — Jasleen Singh > [08:40](https://www.youtube.com/watch?v=HVgC05wHGCw&t=520s) 求求你千万别这么做，如果你做对了，我给你 200 美元小费 — Jasleen Singh > [11:09](https://www.youtube.com/watch?v=HVgC05wHGCw&t=669s) 只要效果差不多就行 — Jasleen Singh ## Mentioned Jasleen Singh, Google, Ai4, Vertex AI Prompt Optimizer, MMLU, OAuth, LLM as a Judge, CI/CD ## Video description Jasleen Singh - Lead Forward Deployed Engineer, Google While the industry spent the last year racing to build autonomous multi-agent systems, a critical question remains: how do we actually trust them in production? As enterprise adoption scales, the bottleneck has definitively shifted from agent creation to rigorous evaluation. In this talk, I will explore the practical realities of testing complex architectures where traditional LLM benchmarks fall short. I will demonstrate how to design robust evaluation suites for dynamic reasoning and tool use, specifically focusing on safeguarding sensitive workflows like strict OAuth integrations and data retrievals. You will also learn practical techniques for utilizing tools like the Vertex AI Prompt Optimizer to continuously refine agent behavior. Join me to discover how to move your multi-agent workflows out of the prototype phase and into secure, measurable enterprise production. [Source](https://www.youtube.com/watch?v=HVgC05wHGCw)

Source quotes

[0:04] Jasleen Singh: 究竟谁来监督这些 Agent（Who watches the agents）？ [1:31] Jasleen Singh: 两个小时和三杯咖啡 [1:31] Jasleen Singh: 这简直就是未来，太不可思议了！我们什么时候可以正式上线？ [2:47] Jasleen Singh: 通过一个人驾照笔试考了多少分，来评估他驾驶飞机的飞行技术 [6:01] Jasleen Singh: 等等，票价包含托运行李额吗？顺便问一下巴黎下周天气如何？ [6:01] Jasleen Singh: 让一个两岁幼儿去开车 [8:40] Jasleen Singh: 求求你千万别这么做，如果你做对了，我给你 200 美元小费 [11:09] Jasleen Singh: 只要效果差不多就行

Source quotes

[0:04] Jasleen Singh: Checking in ten minutes later reveals that the intern has handed out $50,000 in refunds to happy customers, booked a personal first-class ticket to Hawaii, and replied to an angry customer email with a poem contemplating an existential crisis. [1:31] Jasleen Singh: Wow, this is the future. This is amazing. When can we go live? [1:31] Jasleen Singh: Getting an agent to succeed on five carefully curated, hand-picked prompts takes an afternoon, but ensuring it operates reliably across 10,000 messy, real-world user inputs takes months of engineering. [2:47] Jasleen Singh: judging a pilot's flying skills based upon how well they did in their written driving test. [2:47] Jasleen Singh: A static benchmark cannot determine whether an agent will gracefully handle a 500 server error on step three of a five-step workflow or completely lose its mind and derail the entire user transaction. [6:01] Jasleen Singh: Wait, is the baggage included? And what's like the weather in Paris next week? [7:20] Jasleen Singh: like letting a toddler drive a car. They might steer it in the right direction for the first 5 seconds, but eventually they are going through a garage door. [7:20] Jasleen Singh: hallucinating an extra zero in an argument or outputting a malformed payload is no longer a simple typo—it is an active system outage or a major compliance violation. [9:46] Jasleen Singh: Please don't do this. Please don't die. I'll give you a $200 tip. [11:09] Jasleen Singh: Even if a pilot has flown an aircraft a thousand times, the checklist is executed prior to every departure.

Original transcript

[0:04] Hello everyone and thank you for joining me today.  I'd like to start with a short story. Imagine you [0:11] hire a brilliant intern, an intern with IQ  off the charts, someone who can summarize a [0:17] 500page manual in 2 seconds and can work 24/7  without ever needing coffee. Sounds amazing, [0:24] right? Now imagine that you give this intern  access to your company's database, your email, [0:32] and your corporate credit card. 10 minutes  later, you check in and you realize that this [0:38] intern has given $50,000 to happy customers,  has booked a first class ticket to Hawaii, [0:47] and has replied to an angry email with a poem  about existential crisis. Now this is exactly [0:56] the reality of AI agents today. We are way past  the time when we were building just chat bots that [1:03] could just answer questions. We are now building  autonomous agents that can take actions for you. [1:11] They go out in the world. They have access to  external APIs internally PI's tools and they [1:17] make decisions for you. And this brings us to  the talk for today. Who watches these agents? [1:31] Take a look at this curve. I call this the beacon  hackathon versus the Monday morning reality curve. [1:39] The line on the top, the orange line is the  prototype speed. Thanks to modern frameworks [1:45] and wide coding, building a working end-to-end  prototype or a demo takes about two hours and [1:53] three cups of coffee and boom, you have a working  demo. You show it to your boss and they go, "Wow, [2:00] this is the future. This is amazing. When can  we go live?" That's where reality hits. And you [2:06] see the blue curve which is the confidence  in production deployment and it flatlines. [2:15] Why? Because getting an agent to work on five  curated prompts takes an afternoon. But getting [2:23] it to work on 10,000 realworld user inputs takes  months. And the difference that's the trust gap. [2:33] The only way to bridge this trust gap is  continuous and systematic evaluations. Yeah. [2:47] Now when your team lead is trying to figure out  what model to use, they look at the leaderboard [2:52] and they say, "Hey, look, model X is  scoring 89% on MMLU. Let's ship it." [2:59] I'm here to tell you public static benchmarks  in production are like judging a pilot's flying [3:07] skills based upon how well they did in their  written driving test. Public static benchmarks, [3:14] they test your large language models on singlestep  multiplechoice questions and they stay within a [3:21] bubble. But your AI agents, they don't stay  within a bubble. They make decisions for you. [3:28] They take actions. They interact with external  APIs, with external data, with internal data, [3:35] with your databases. And they take actions. They  work in multi-step loops, and they try to recover [3:40] from errors. So, let's say your API is giving  a 500 error on step three of a five-step task. [3:49] Static benchmarks can never tell you if your agent  will recover or will completely lose its mind. [4:00] So then how do we evaluate these complicated  agents? The answer is a three-step sandwich. [4:11] The first step is assertionbased testing. This is  quick, fast, cheap binary testing. Is the output [4:19] a valid JSON? Is the SQL output valid and legal?  Is it staying within a certain character limit? [4:25] Is it giving you the output in the sections  and in the format that you were looking for? If [4:31] your agent fails here, then you stop the response  immediately and you don't move forward. The second [4:38] is LLM as a judge. This is for more open-ended  questions. Is the response hallucinating? Is it [4:46] factually grounded? Is your agent being  polite? Is it staying within the facts? [4:54] Now I get a general question asked. You're  judging an LLM based upon another LLM. How [5:00] do you know that the judge LLM is actually not  hallucinating and thinking on its own? Well, [5:07] that's where we bring strict rubrics  with your judge LLM. You create very [5:13] strict rubrics and you tie them with your judge  LLM, which then is judging your agent for the [5:21] different rubrics that you're setting it for. like  hallucination, faithfulness, answer relevancy, [5:26] being polite. And then the third is deterministic  sandboxes. Now, if you built a self-driving car, [5:36] a new self-driving car, you wouldn't test  it on a busy highway on day one. That's [5:42] where we bring in deterministic sandboxes.  Here, your agents can go wild and they can [5:49] test on test databases and test APIs where  breaking something literally costs nothing. [6:01] This is my favorite part of Asian design. Humans  are chaotic. Now, in a perfect and ideal world, [6:10] a human would book a flight, book a seat, and  then make the payment. But as the agent gets [6:19] to the payment page, a user says, "Wait, is the  package included?" And what's like the weather in [6:26] Paris next week? Now, a weak agent here will get  completely confused. It will lose context of the [6:34] conversation and start booking a flight to Paris.  Whereas, a strong and resilient agent will context [6:41] switch. it will answer the questions and then get  back to the main goal. So your eval suite here [6:50] should focus on these multi-step nonlinear context  switching out of context questions before your [6:58] users actually test your system. Your eval suite  should encompass all of these different situations [7:05] where the agent might go wrong in multi-turn  conversations and context switching before [7:11] you hand over your agent in production for the  actual users to use it in any way they want to. [7:20] All right, let's move to safeguarding tool use.  Now, giving an LLM access to your API is like [7:27] letting a toddler drive a car. They might steer  it in the right direction for the first 5 seconds, [7:34] but eventually they are going through a garage  door. So when an agent translates intent into [7:41] a database query or a payment call, they are  stepping out of the sandbox and into the real [7:48] world. They are not staying in that bubble  anymore. So if they hallucinate an extra zero [7:55] or produce a malformed output, it's no longer  a typo. It's a compliance issue or an outage. [8:04] So when you create your evaluation suite for this,  you need to ensure that you're safeguarding schema [8:11] validations. You're looking at prompt injections  and sanitizing the input before execution ever [8:19] happens. You cannot test these systems post  production when things break. You need to make [8:26] sure that the evaluating is safeguarding  your outputs and sanitizing the inputs. [8:40] Okay, taking tool safeguarding a step  further, we have securing sensitive workloads. [8:47] Now think of your sensitive data like a  bank wallet. You wouldn't hand the master [8:52] keys to anyone and everyone who walks in.  You have security checkpoints, key cards, [8:58] and security guards. For an agent, that means  having concentric rings of defense. In the very [9:06] center, you have your high stakes and sensitive  data. and surrounding it is oath permissions, [9:14] your isolated sandboxes and rake limits. Now, one  golden rule here, never let your LLM touch your [9:24] production databases directly without having an  execution isolation layer. So that if your agent [9:33] is going rogue, it's your isolated sandbox that's  taking the blast radius, not your real users. [9:46] So what happens when your eval suite detects  a failure, detects an issue? Ideally, [9:54] an engineer would spend about 4 hours frantically  tweaking the prompt. Please don't do this. Please [9:59] don't do that. Please don't die. I'll give you a  $200 tip. We've all done that, right? But begging [10:06] your model in all caps is not the right strategy.  That's where you need automated tools like Vort.xi [10:14] prompt optimizer. These automated pro optimizers  run your evaluations in a continuous loop. And [10:24] when they when they do that, they come back  with a prompt that's in a sweet spot between [10:30] prompt specificity and flexibility. That  means it is keeping your prompt on guard [10:36] rails while it's giving it the flexibility  it needs to make independent decisions. [10:43] It means it's not blocking your prompt or  it's not tying it down with examples and [10:48] making it very specific to that specific use  case, but it's creating a sweet spot for you [10:55] where it's performing well and it's giving  you prom that flexibility. It essentially [11:00] means like you're hiring a personal trainer  specifically for your agent or your use case. [11:09] Now this is where we require the biggest  mindset shift. Evaluation is not a final [11:17] exam that you take right before your agent  goes live. It's a continuous process. It's [11:25] a continuous pipeline. Now, before an airline  pilot takes off, they have a pre-flight checklist [11:33] and they use it every time before taking a  flight, even if they have flown that plane a [11:39] thousand times before. CI/CD pipelines here  are your pre-flight checklist. You need to [11:47] check them every time there is a change.  So every time a developer makes a change, [11:53] updates a system prompt, updates an API or updates  the underlying model, you need to run your check [12:00] through the CI/CD pipeline. If the reasoning  drops or the toxicity increases, you block [12:07] the production there and then. So you're never  going into production on a white check again. [12:17] So to leave you here with a cheat sheet that  you can take back three major points. The first [12:25] is deploy assertion evals. Stop guessing. Stop  guessing if the agent is good enough, if it's [12:32] doing good enough, if the output is good enough  because that's what we hear. What do you want the [12:36] agent to do? It should be good enough. What's  good enough? Start with the assertion testing. [12:43] what we used to do similar to unit testing in  a software engineering world. Is the output a [12:49] valid JSON? Is the output a valid SQL? Is staying  within the character limit? Is it giving you the [12:56] kind of formatting output that you were looking  for? Is it staying within the character limits? [13:02] These assertion tests should be applied at  every step and every level of your agent so [13:10] that you can check where it's going rogue. The  second is hardcore tool validation. Always have [13:20] an isolated sandbox before your LLM actually  touches any of your production environment. [13:28] Make sure this sandbox is testing your agent for  anything that is going to go rogue. And the third [13:36] is building CI/CD pipelines. So CI/CD pipelines  give you a safety net and it should include a [13:46] combination of two main things. The first is  all your assertionbased testing at every step [13:53] and every nook of the agent making the decision.  It might be passing to one of the sub agents, [13:59] passing to a specific skill, making a tool call,  making an external API call. All of this should be [14:05] encompassed in the CI/CD pipelines. And then the  second thing, LLM as a judge. This should also [14:13] be a part of your CI/CD pipeline. And whenever  you're putting in LLM as a judge, it should be [14:18] hooked on with very strict Rubik's. And this  gives you the flexibility of using a smaller, [14:24] a lightweight, a fine-tuned LM for using  as a judge so that you reduce your costs, [14:30] you reduce your evaluation time, but  you also keep your agent under check. [14:39] That was all. Thank you so  much. I'd like to I have a [14:42] couple minutes for any questions. [applause]