Vanguard Adopts Actuarial Risk Models to Scale GenAI Across 800 Teams

TAMING STOCHASTICITY: Evaluations & Observability at Scale With Vanguard — Ai4

Ai4

Enterprises cannot eliminate large language model stochasticity and must replace manual testing with simulation-based actuarial sampling to govern production risk without scaling human reviewer headcount linearly.

AI-assisted summary of TAMING STOCHASTICITY: Evaluations & Observability at Scale With Vanguard — original source. Check the original for full context.

Source published:

Detailed notes

## Background Published by Ai4 on YouTube on August 15, 2026, this presentation features Vanguard's Ritesh Shah (Chief Architect - AI/ML/GenAI) and Piyush Saxena (Director AIML, Advice, Wealth Management & AI CoE). According to the video description, the session explores how enterprises can move past manual evaluations and spreadsheets toward an integrated architecture combining persona-driven testing, custom evaluation judges, human-in-the-loop workflows, and continuous production observability.

## TL;DR Vanguard outlines its comprehensive framework for governing non-deterministic generative AI across more than 800 engineering teams by establishing formal risk budgets rather than striving for impossible zero-risk determinism. The strategy replaces manual user acceptance testing with tens of thousands of synthetic persona simulations assessed by a two-tier hierarchy of global Principal Judges and domain-specific Application Judges. Through a continuous subject matter expert (SME) feedback loop and a 2x2 confusion matrix, Vanguard applies actuarial sampling to cut human oversight down to a fraction of traffic while maintaining statistical confidence. The platform integrates directly into CI/CD pipelines and live production observability, enforcing drift detection, incident escalation, tiered kill switches, and token-cost optimization.

## Atmosphere & style The presentation is pragmatic, articulate, and deeply technical, delivered with a grounded confidence built on real production scars and enterprise scale. Saxena and Shah establish an engaging, collaborative rapport with the audience, using humor, quick pop quizzes, and practical analogies to demystify complex statistical and architectural concepts. It feels less like theoretical AI evangelism and more like an authoritative, battle-tested engineering blueprint that balances strict regulatory governance with the operational need for speed.

## Key takeaways - [00:04](https://www.youtube.com/watch?v=Cq6CuBTwikw&t=4s) Generative AI systems cannot have randomness completely eliminated; organizations must focus on taming stochasticity by finding the optimal balance between safety guardrails and utility. - [03:09](https://www.youtube.com/watch?v=Cq6CuBTwikw&t=189s) Traditional manual User Acceptance Testing (UAT) fails for open-ended LLMs, requiring automated large-scale simulations with synthetic personas to establish actuarial-grade statistical confidence. - [06:44](https://www.youtube.com/watch?v=Cq6CuBTwikw&t=404s) Evaluation utilizes a two-tier judge structure: enterprise-wide Principal Judges enforce baseline standards like toxicity globally, while Application-Specific Judges assess domain-specific tasks. - [12:05](https://www.youtube.com/watch?v=Cq6CuBTwikw&t=725s) Isolated non-deterministic model events should not trigger alarms; operational escalation and kill switches must be tied to aggregated issues breaching explicit business risk budgets. - [14:55](https://www.youtube.com/watch?v=Cq6CuBTwikw&t=895s) Mapping SME decisions against AI judge scores in a confusion matrix separates agent defects from judge calibration needs, building a proprietary golden benchmark dataset. - [18:05](https://www.youtube.com/watch?v=Cq6CuBTwikw&t=1085s) Applying actuarial sampling to judge-approved interactions allows enterprises to review only a fraction of compliant traffic while maintaining equivalent residual risk to 100% human oversight. - [20:10](https://www.youtube.com/watch?v=Cq6CuBTwikw&t=1210s) Judge calibration follows Occam's razor, using prompt optimization for standard contexts and transitioning to task-oriented Small Language Models (SLMs) trained via RLHF for complex rubrics. - [21:58](https://www.youtube.com/watch?v=Cq6CuBTwikw&t=1318s) Vanguard scaled its enablement architecture from local developer Python libraries into a centralized AWS evaluation platform providing release go/no-go gates based on residual risk. - [24:44](https://www.youtube.com/watch?v=Cq6CuBTwikw&t=1484s) The long-term vision integrates agentic CI/CD pipelines that run parallelized pre-production evaluations and automatically provision post-production observability with identical metrics. - [27:50](https://www.youtube.com/watch?v=Cq6CuBTwikw&t=1670s) AI judges cannot be treated as static unit tests; they represent living software collateral that must be owned, versioned, and continuously maintained across the SDLC as user behavior evolves. - [33:14](https://www.youtube.com/watch?v=Cq6CuBTwikw&t=1994s) Evaluation infrastructure costs are optimized by combining multi-input multi-output (MIMO) judging prompts with task-specific 50–60B parameter models rather than expensive frontier LLMs. - [37:01](https://www.youtube.com/watch?v=Cq6CuBTwikw&t=2221s) Risk budgets must be calibrated according to deployment context, maintaining strict tolerances for customer-facing applications while prioritizing utility for internal employee tools. - [38:15](https://www.youtube.com/watch?v=Cq6CuBTwikw&t=2295s) Overly conservative governance that paralyzes AI deployment creates strategic competitive vulnerability, where the cost of inaction can exceed potential regulatory penalties.

## Detailed walkthrough ### [00:04](https://www.youtube.com/watch?v=Cq6CuBTwikw&t=4s) Introduction and the Genesis of Vanguard's GenAI Journey

Piyush Saxena, Head of the AI Center of Excellence at Vanguard, and Ritesh Shah, Chief Architect for AI/ML and Generative AI, introduce their session on "Taming Stochasticity: Evaluations & Observability at Scale." The title deliberately puts *TAMING STOCHASTICITY* front and center because, as Saxena emphasizes, organizations cannot completely eliminate randomness from generative systems: "We can tame stochasticity. We cannot weed it out of a system that is generative in nature." Reflecting on Vanguard's journey over the past year and a half of deploying large language models across roughly 800 developer teams, Saxena frames their learnings with good humor: they share both successes and lessons "cuz we don't fail."

```mermaid flowchart TD accTitle: Vanguard AI Evaluation and Governance Lifecycle subgraph PreProd["Pre-Production Evaluation"] P["Customer Data"] --> Persona["Synthetic Personas"] Persona --> Sim["Large-Scale Simulations (10,000s)"] Sim --> Judges["Evaluator Judges (LLMs / SLMs)"] Judges --> EvalCat{"Judge Hierarchy"} EvalCat --> PJ["Principal Judges\n(Enterprise-Wide: Toxicity, Risk)"] EvalCat --> AJ["App-Specific Judges\n(Domain-Specific: Text-to-SQL)"] end

subgraph Feedback["Calibration & Human-in-the-Loop"] SME["Vanguard SMEs / Ethos"] --> Calib["Judge Calibration & Self-Healing"] Calib -.-> Judges end

subgraph Prod["Production Observability"] Deploy["Production Deployment"] --> Obs["Telemetry & Event Logging"] Obs --> Delta{"Drift Analysis\n(Pre-Prod vs Prod KPIs)"} Delta --> Issues["Issues & Incidents"] Issues --> Alerts["Business Unit Alerts"] end

Judges --> Deploy ```

The genesis of Vanguard's evaluation framework stems from a critical interaction with senior leadership. As a major financial institution operating in a heavily regulated industry, trust and company ethos are non-negotiable. During an early GenAI pilot review, an executive pointedly asked: *"How do I know that this application does not hallucinate? How is it safe?"* The development team responded by heavily prompt-engineering guardrails to prevent harmful responses. However, when returning for the second round of executive testing, the feedback swung to the opposite extreme: the model had become unhelpful, refusing nearly every query with boilerplate disclaimers. This tension defined Vanguard's primary challenge: establishing the precise trade-off between risk/harmfulness and helpfulness.

- **Scale**: Over 800 development teams building GenAI applications across the enterprise. - **Foundational Pillars**: Testing, Production Observability, Oversight, and Self-Healing. - **The Core Dilemma**: Finding the optimal balance between safety constraints (minimizing risk/hallucinations) and utility (providing helpful, complete answers).

---

[03:09](https://www.youtube.com/watch?v=Cq6CuBTwikw&t=189s) The Stochastic Reality and Simulation-Based Persona Testing

To demonstrate the stochastic challenge of LLMs, Saxena quizzes the audience with a simple fill-in-the-blank prompt: *"What is the capital of Peru?"* While the room unanimously answers "Lima," Saxena points out that depending on conversational context and training distribution, an LLM might reasonably describe Lima as the "gastronomical capital of South America" or reference numerous other attributes. Because generative outputs vary slightly each time a query is posed, the central problem is defining the threshold where acceptable stylistic variation crosses into an incorrect or improper answer.

Traditional User Acceptance Testing (UAT)—typically involving 20 to 100 human testers clicking through bounded, deterministic software—fails when applied to open-ended LLMs capable of generating infinite responses. Human manual review cannot generate the volume required to validate statistical confidence. Vanguard solves this by replacing manual ad-hoc testing with automated simulations:

- **Synthetic Personas**: Running tens of thousands of automated simulations against the application using synthetic clients derived directly from real customer interaction data, ensuring realistic behavioral patterns ("garbage in, garbage out" still applies). - **Actuarial-Grade Confidence Intervals**: Expanding test volume from a few hundred manual UAT cycles to tens of thousands of automated runs enables statistical claims on macro metrics (risk and helpfulness) down to tight margins, such as "99.5% ± 0.05%". - **Development Velocity**: Automated synthetic test suites allow developers to adjust prompts or add features and immediately re-evaluate the system without waiting for human testing bottlenecks.

---

[06:44](https://www.youtube.com/watch?v=Cq6CuBTwikw&t=404s) Two-Tiered Judges: Principal vs. Application-Specific

Reviewing the outputs of tens of thousands of synthetic customer interactions—which include complex multi-turn dialogs and underlying tool/API execution telemetry—requires automated evaluation. Vanguard employs a mixture of Large Language Models (LLMs) and task-oriented Small Language Models (SLMs) configured as automated judges split into two distinct tiers:

1. **Principal Judges**: Enterprise-wide standards that must remain consistent across every team. For universal metrics like toxicity or brand safety, an organization with 100+ dev teams cannot permit 100 differing definitions. Principal judges enforce these unified guardrails globally. 2. **Application-Specific Judges**: Custom evaluators tailored to individual domain needs. For instance, an application translating natural language into SQL queries requires specialized judges designed specifically to score text-to-SQL syntactical accuracy and query efficiency.

---

[08:29](https://www.youtube.com/watch?v=Cq6CuBTwikw&t=509s) Self-Healing, Calibration, and Human-in-the-Loop Oversight

Automated judges are themselves machine learning models and are never 100% accurate out of the box. Furthermore, frontier foundation models lack Vanguard's proprietary organizational methodology, institutional ethos, and domain expertise. Vanguard closes this gap by integrating Subject Matter Experts (SMEs) directly into a Human-in-the-Loop (HITL) calibration loop.

SME feedback is continuously fed back into the pipeline to fine-tune and align both the operational agents and the evaluator judges. Over a year of iterative production data, Vanguard observed a steady upward alignment between automated judge ratings and human SME scores. This continuous calibration prevents evaluation decay as customer interaction styles, vocabulary, and underlying application requirements evolve over time.

---

[10:47](https://www.youtube.com/watch?v=Cq6CuBTwikw&t=647s) Production Observability, Drift Detection, and Incident Escalation

Pre-production testing establishes baseline confidence, but the exact same Key Performance Indicators (KPIs)—such as toxicity, completeness, and helpfulness scores—must be measured continuously in live production. The mathematical delta between pre-production evaluation benchmarks and production performance metrics defines system **drift**.

Ritesh Shah details how production telemetry is operationalized into actionable governance:

- **Event-to-Alert Pipeline**: Every live interaction emits structured telemetry events. - **Issue & Incident Classification**: Events failing defined acceptance thresholds are flagged as issues and escalated to operational incidents. - **Automated Routing**: Incidents trigger automated alerts routed directly to the responsible business unit and engineering team, notifying them immediately when an AI system deviates from expected behavior.

[12:05](https://www.youtube.com/watch?v=Cq6CuBTwikw&t=725s) AI Observability, Risk Budgets, and Kill Switches

AI observability requires a structured operational pipeline to separate regular behavior from critical system failures. Events generated by generative models are naturally non-deterministic and can frequently look like anomalies without actually signifying broken behavior. For this reason, teams must never raise critical alarms on isolated events. Instead, observability frameworks aggregate events into issues, and only when a collection of issues breaches an explicitly defined threshold are they escalated into actionable incidents. These incidents are then categorized by severity—whether an incident is benign, problematic, or critical—to determine the operational protocol, such as triggering a Major Incident Management (MIM) call or paging on-call engineers via PagerDuty.

```mermaid flowchart LR accTitle: AI Incident Escalation and Kill Switch Architecture subgraph EventStream["Telemetry Stream"] E["Raw AI Model Events"] --> Agg["Event Aggregation"] end subgraph Evaluation["Risk & Triage Layer"] Agg --> I["Aggregated Issues"] I --> T{"Breaches Risk Budget?"} end subgraph Response["Mitigation & Action"] T -- No --> Log["Log & Monitor"] T -- Yes --> Inc["Declare Incident"] Inc --> KS{"Kill Switch Execution"} KS --> H["Route to Human Reviewer"] KS --> A["Trigger Alert / PagerDuty"] KS --> F["Return Graceful Error Response"] end ```

Establishing this pipeline requires defining a formal "risk budget." Because AI and machine learning models are fundamentally generative and probabilistic, zero risk is impossible; the enterprise must accept some level of residual business risk. Without a concrete risk budget negotiated directly with business and risk partners, all monitoring thresholds remain arbitrary. When an evaluation dimension with low risk tolerance (such as toxicity) exceeds the acceptable threshold, systems must engage an automated kill switch. Kill switch mechanisms can be tiered: triggering an urgent alert for human intervention, pausing live responses to route incoming requests to human experts for manual review, or terminating execution to fulfill the prompt with a graceful, safe error message.

- **Threshold Definition**: Thresholds cannot be chosen in a vacuum; they must be a direct derivative of the enterprise's explicit business risk budget. - **Act on Incidents, Not Events**: Isolated model quirks are normal; operational responses (PagerDuty, MIM bridges) should only be provoked when aggregated issues exceed acceptable limits. - **Kill Switch Modalities**: - *Alert Trigger*: Pings a human operator while system execution continues. - *Human-in-the-Loop Interception*: Routes in-flight requests to human reviewers before delivering output. - *System Shutdown / Fail-Safe*: Halts execution entirely and serves an automated fallback error response.

---

[14:55](https://www.youtube.com/watch?v=Cq6CuBTwikw&t=895s) The SME Confusion Matrix and Model Moats

To make the most efficient use of Subject Matter Experts (SMEs), their daily operational interaction with the system must be streamlined to binary decisions: "yes, I agree" or "no, I disagree" with an AI judge's assessment. Mapping these expert validations against AI judge evaluations creates a classical 2x2 confusion matrix that drives both system improvements and governance decisions:

| Evaluation Outcome | AI Judge Flagged | AI Judge Approved | Impact on System | |:--- |:--- |:--- |:--- | | **SME Agrees (Issue Exists)** | **True Positive (TP)**<br>Judge flagged an issue; SME agreed it is an error. | — | **Agent Improvement Opportunity**<br>Identifies real flaws in the underlying generative application to fix. | | **SME Agrees (No Issue)** | — | **True Negative (TN)**<br>Judge cleared output; SME confirmed it is valid. | **System Health Benchmark**<br>High volume confirms the system and judges are performing as intended. | | **SME Overrules (No Issue)** | **False Positive (FP)**<br>Judge flagged output; SME confirmed it was benign. | — | **Business Overhead**<br>Judge is overly conservative, wasting human reviewer review cycles. | | **SME Overrules (Issue Exists)** | — | **False Negative (FN)**<br>Judge cleared output; SME caught an undetected error. | **Real Business Risk**<br>Unseen, unflagged failures slipping silently into production. |

False positives and false negatives represent type I and type II errors, which are inherently inversely proportional. When false positives and false negatives accumulate, they indicate that the AI judges themselves require calibration because they are either acting too conservatively or letting critical business risks slip through unflagged. Conversely, true positives represent legitimate application errors, highlighting exact areas where the generative agent needs tuning. Over time, the accumulated dataset of SME decisions forms a proprietary golden benchmark dataset—the true operational moat for the enterprise—ensuring that whenever underlying applications or judges are updated, they can be rigorously regression-tested with high statistical confidence.

---

[18:05](https://www.youtube.com/watch?v=Cq6CuBTwikw&t=1085s) Actuarial Sampling and Turning Down the Oversight Dial

Enterprises deploying large-scale generative AI cannot scale human oversight linearly. If a system handles 100,000 interactions per day, reviewing every transaction manually is operationally impossible, as "there is not an army of SMEs that is available for us to hire." Enterprises must move away from brute-force 100% human oversight toward statistical sampling by applying principles from actuarial science.

```mermaid flowchart TD accTitle: Statistical Sampling and Human Oversight Reduction Flow subgraph Traffic["100,000 Daily Interactions"] In["Model Invocations"] --> J["AI Judge Assessment"] end subgraph JudgeRouting["Judge Decision Routing"] J -- "Flagged as Defect" --> FullRev["100% Mandatory Review<br>(True & False Positives)"] J -- "Passed as Compliant" --> SamplePool["Candidate Pool for Sampling<br>(True & False Negatives)"] end subgraph ActuarialSampling["Actuarial Calibration"] SamplePool --> Math["Actuarial Risk Model<br>(Evaluates KPIs & Risk Budget)"] Math --> StatRev["Statistical Sample Review<br>(e.g., 20% - 30% Dial)"] StatRev --> RiskEquiv["Achieves Equivalent Residual Risk to 100% Review"] end ```

By analyzing the confusion matrix KPIs, organizations can selectively sample traffic. The left column of the matrix (interactions flagged by judges as True Positives or False Positives) is always routed to reviewers. However, the right column (interactions approved by the judge as compliant) contains True Negatives and hidden False Negatives. Actuarial modeling allows teams to calculate the exact sample size needed from this approved pool, mathematically proving that reviewing a fraction—such as 20% or 30%—assumes the exact same level of residual risk as reviewing 100% of interactions. This approach keeps operational risk strictly bounded within the business risk budget while drastically dialing down manual oversight costs.

- **The Scale Dilemma**: High-throughput systems (100k+ interactions/day) render exhaustive human review unfeasible. - **Targeted Triage**: Flagged outputs receive full human review, while judge-approved streams are routed to actuarial sampling pools. - **Actuarial Risk Equivalence**: Statistical calculations prove that partial sampling can match the risk profile of total review, allowing teams to safely scale without linear SME hiring.

---

[20:10](https://www.youtube.com/watch?v=Cq6CuBTwikw&t=1210s) Judge Calibration: Prompt Optimization vs. Task-Oriented SLMs

Maintaining trustworthy evaluation requires a dual-track self-healing loop: improving the underlying generative application while continually calibrating the AI judges. When judges produce false positives or false negatives, organizations have two main paths for remediation depending on the complexity of the evaluation task:

```mermaid flowchart LR accTitle: Dual-Track Judge Calibration and Self-Healing subgraph SMELoop["SME Feedback Loop"] SME["SME Agree/Disagree Decisions"] --> Data["Curated Benchmark Dataset"] end subgraph CalibrationPath["Judge Remediation Paths"] Data --> Decision{"Prompt Complexity / Context Size"} Decision -- "Standard Context" --> Occam["Occam's Razor: Prompt Optimization<br>(Automated Prompt Tuning & Benchmark Re-run)"] Decision -- "Substantial / Large Context" --> SLM["Task-Oriented Small Language Models (SLMs)<br>(RLHF Pipelines via SME Data)"] end subgraph ClosedLoop["Production Impact"] Occam --> Calibrated["Calibrated AI Judges"] SLM --> Calibrated Calibrated --> ContVal["Continuous Validation & Regression Testing"] end ```

Following the principle of Occam's razor, the default approach uses prompt-based LLM judges. When these judges drift, automated prompt optimization and prompt augmentation mechanisms refine the judging prompt and immediately validate it against the SME benchmark dataset. However, as business rubrics become increasingly complex, judging prompts can grow excessively large and unwieldy. In these cases, teams transition to task-oriented Small Language Models (SLMs), fine-tuning them via Reinforcement Learning with Human Feedback (RLHF) pipelines powered by the exact same thumbs-up/thumbs-down SME decisions. This shifts the organizational mindset away from the traditional static concept of "we ran these tests, we are green" to an active operational posture: "we run these tests, we are confident, and we will continue to run these tests" to govern stochastic model behavior over time.

---

[21:58](https://www.youtube.com/watch?v=Cq6CuBTwikw&t=1318s) The Vanguard Enablement Architecture: From Python Library to AWS Platform

To turn AI evaluation into a shared, standardized capability across the organization, Vanguard evolved its enablement strategy through a multi-stage product roadmap designed to meet engineering teams where they are:

```mermaid flowchart TD accTitle: Vanguard Evaluation and Observability Platform Architecture subgraph CorePlatform["Custom AWS AI Evaluation Platform"] UI["Web Interface / Portal"] --> Core["Orchestration & Simulation Engine"]

subgraph Config["Configuration Modules"] P["Persona Management"] J["Bring Your Own Judges (BYOJ)"] GT["Ground Truth Ingestion"] end

Config --> Core Core --> Sim["Multi-Scenario Simulation"] Sim --> Reports["Multi-Dimensional Evaluation Report"] end

subgraph Governance["Production Risk Governance"] Reports --> Gate{"Go / No-Go Gate<br>(Residual Risk vs. Risk Budget)"} Gate -- Approved --> Prod["Production Deployment"] Gate -- Rejected --> Fix["Iterate on Agent / Judges"] end ```

Vanguard began with a lightweight Python library for developers to run local evaluations early in the development lifecycle. To support the entire enterprise, this was expanded into a centralized, custom web-based evaluation platform hosted on AWS. Engineering and product teams onboard their agents into the platform, configure custom personas and judges, supply ground-truth data, and schedule automated simulation runs. The platform generates comprehensive evaluation reports across multiple risk and quality dimensions. These reports serve as the objective go/no-go gate for release decisions, verifying that residual risk remains within acceptable limits. As the platform matured over several years, its user base broadened significantly beyond application developers to include product owners, divisional risk managers, and enterprise compliance stakeholders across the entire risk ecosystem.

[24:44](https://www.youtube.com/watch?v=Cq6CuBTwikw&t=1484s) Unified Interfaces and the Vision for CI/CD Agentic Evaluation

The architecture behind Vanguard's evaluation framework is designed around a single core definition of personas, simulations, and expected outputs to prevent fragmentation across enterprise risk, engineering, and business groups. Because these distinct groups cannot have conflicting definitions of how an application performs, the platform provides diverse entry points—Command Line Interfaces (CLIs), user interfaces (UIs), and agents—all tied to the exact same backend. As development modernizes with tools like Cloud Code, developers no longer evaluate systems by clicking through a UI; instead, automated pipelines take over before deployment.

The long-term vision is an automated CI/CD pipeline powered by specialized, parallelized agents. Early-stage setup agents configure personas and judge rubrics, while downstream evaluation agents run concurrent test suites across business logic, adversarial prompts, trusted AI standards, and regulatory compliance. If the aggregate results pass approval thresholds, the deployment proceeds to production. Crucially, evaluation does not terminate at release: the CI/CD pipeline automatically provisions post-production observability, importing the same risk rubrics and judges so that metrics remain identical across pre-prod and production environments while configuring operational alerts and triggers.

```mermaid flowchart TD accTitle: CI/CD Agentic AI Lifecycle Architecture subgraph PreProd["Pre-Production CI/CD Pipeline"] A["CI/CD Trigger / Desktop Cloud Code"] --> B["Setup Agents (Personas & Judges)"] B --> C1["Business Evaluation Agent"] B --> C2["Adversarial Testing Agent"] B --> C3["Trusted AI & Regulatory Agent"] C1 --> D{"Approval Gateway"} C2 --> D C3 --> D end

subgraph Production["Production Operations"] D -->|Approved| E["Deploy to Production"] E --> F["Automated Observability Setup"] F --> G["Import Judges (Pre/Post Metric Parity)"] F --> H["Triggers & Alerting Engine"] end ```

[27:50](https://www.youtube.com/watch?v=Cq6CuBTwikw&t=1670s) Redefining Risk, Stochasticity, and the AI Development Mindset

The transition to generative AI requires software organizations to redefine fundamental operating assumptions:

- **Establishing a Risk Budget**: Development cannot work backwards to set target accuracy or simplicity metrics without an explicit risk budget. Because LLM outputs are context-dependent—"The capital of Peru can be Lima. It can be many other things based on context"—teams cannot rely on small-scale User Acceptance Testing (UAT) and must instead apply statistically significant, actuarial science-based methodologies. - **Inherent vs. Residual Risk**: Teams must evaluate both the inherent risk of a raw foundation model and the residual risk remaining after applying guardrails, prompt optimizations, and jury judges. - **Design-Time Governance**: Risk budgets, performance targets, and self-healing loops must be codified at design time prior to production approval rather than treated as post-launch afterthoughts. - **Taming Stochasticity**: Generative AI systems are probabilistic by nature; "We can tame stochasticity. We cannot weed it out." If a use case demands absolute determinism, teams should instead deploy traditional machine learning or deterministic rule systems. - **Judges as Owned Collateral**: Judge scores cannot be treated like static unit tests. Because customer behavior evolves dynamically—as seen across internet communities prompting retail chatbots (like Chipotle) to execute complex math or generate Python code—judges (whether prompt-based or Small Language Models) are living software collateral that developers must own, version, and maintain across the entire software development lifecycle (SDLC).

[31:29](https://www.youtube.com/watch?v=Cq6CuBTwikw&t=1889s) Q&A: Multi-Tiered Observability and Defense-in-Depth

Responding to an audience question regarding runtime guardrail architectures, the speakers outline Vanguard's multi-layered defense model. The first line of defense consists of automated LLM judges executing in the observability platform to monitor incoming production interactions and track aggregate violation volumes.

Because automated LLM judges inevitably produce false positives and false negatives, Vanguard employs a second line of defense composed of human subject-matter experts (SMEs). These experts audit flagged interactions, generating calibrated ground-truth data that feeds back into self-healing loops—specifically prompt optimization, retrieval augmentation pipelines, and automated fine-tuning/RLHF (Reinforcement Learning from Human Feedback) workflows.

[33:14](https://www.youtube.com/watch?v=Cq6CuBTwikw&t=1994s) Q&A: Cost Optimization, Multi-Output Judges, and Tokenomics

Addressing the infrastructure costs associated with high-volume LLM evaluations, Vanguard combines model right-sizing, mathematical optimization, and multi-cloud routing:

- **Benchmarking & Operational Metrics**: Judges are benchmarked based on strict accuracy, sensitivity, and specificity thresholds, with a general preference for open-source architectures to minimize operational latency. - **MIMO (Multiple-Input Multiple-Output) Evaluations**: Rather than invoking separate judges for every dimension, Vanguard uses multi-task scoring prompts. For instance, pairing toxicity detection with token length or related safety attributes in a single pass reduces three separate LLM API calls down to one, leveraging mathematical correlation between outputs. - **Task-Specific SLMs vs. Frontier Models**: High-parameter models (such as 500-billion-parameter open models or trillion-parameter frontier models like Claude Opus) are excessively expensive for routine judging tasks. A model evaluating toxicity "doesn't need to know who won the World Series in '87." Vanguard's Enterprise AI and Research teams deploy specialized in-house models (approx. 50–60 billion parameters) tailored specifically to evaluation tasks. - **Infrastructure Cost Levers**: 1. *AWS Native Bedrock*: Initial managed service adoption. 2. *Self-Hosted Open-Source Models*: Direct hosting within Vanguard's cloud infrastructure to eliminate per-token managed service premiums. 3. *Proprietary In-House Models*: Training internal custom models for domain-specific evaluations. 4. *Multi-Cloud & Hyperscaler Arbitrage*: Distributing high-volume evaluation batches across multiple cloud providers to balance throughput against provider pricing models. 5. *Direct Provider Contracts*: Future pricing leverage negotiated via specialized enterprise agreements with frontier model providers such as Anthropic.

[37:01](https://www.youtube.com/watch?v=Cq6CuBTwikw&t=2221s) Q&A: Stakeholder Alignment and Nuanced Risk Appetites

When asked how the technical team secures enterprise buy-in for non-zero failure tolerances, the speakers emphasize working directly with senior business stakeholders and product owners. The term "risk budget" itself originated from a senior product owner collaborating with the AI team.

Risk appetite cannot be uniform across an enterprise; it must be calibrated strictly according to the potential harm of a specific failure mode:

- **Paid / Tiered Customer Segments**: Require strict, low-tolerance risk budgets and higher-redundancy judge architectures. - **Public-Facing Chatbots**: Require broad safety, adversarial, and brand reputation guardrails due to open-ended user exploitation. - **Internal Crew-Facing Applications**: Systems serving internal employees (e.g., Vanguard's 5,000+ crew members) operate under different risk profiles, where task assistance and productivity outweigh external brand exposure risks.

[38:15](https://www.youtube.com/watch?v=Cq6CuBTwikw&t=2295s) Balancing Inaction Risk Against AI Governance and Session Wrap-up

Addressing the delicate balance between risk management and business agility, the speakers emphasize the dangerous hidden cost of excessive caution. While organizations naturally worry about regulatory fines, safety violations, or compliance penalties when rolling out AI systems, freezing deployment out of fear carries its own severe consequences. Paralyzing progress and choosing to "don't do anything and not let the system move forward" creates a massive strategic vulnerability where a company risks losing the market entirely to faster competitors.

Organizations need to calibrate their mindset to weigh both sides of the risk equation. While taking calculated risks with production AI might risk potential penalties or remediation costs, being left behind due to total inaction can end up costing significantly more than any regulatory fine. Following this pragmatic perspective on AI governance and risk tolerance, the session concludes with closing acknowledgments as the speakers thank the audience to warm applause.

- **The Cost of Inaction**: Overly conservative governance that blocks system deployment risks forfeiting market leadership and relevance, often carrying a far steeper business penalty than manageable regulatory or operational risks. - **Risk Mindset Shift**: Enterprise AI governance must weigh potential fines against the broader financial and competitive losses caused by organizational paralysis.

## Notable quotes > [00:04](https://www.youtube.com/watch?v=Cq6CuBTwikw&t=4s) We can tame stochasticity. We cannot weed it out of a system that is generative in nature. — Piyush Saxena

> [00:04](https://www.youtube.com/watch?v=Cq6CuBTwikw&t=4s) cuz we don't fail. — Piyush Saxena

> [00:04](https://www.youtube.com/watch?v=Cq6CuBTwikw&t=4s) How do I know that this application does not hallucinate? How is it safe? — Senior Leadership

> [03:09](https://www.youtube.com/watch?v=Cq6CuBTwikw&t=189s) What is the capital of Peru? — Piyush Saxena

> [18:05](https://www.youtube.com/watch?v=Cq6CuBTwikw&t=1085s) there is not an army of SMEs that is available for us to hire. — Piyush Saxena

> [20:10](https://www.youtube.com/watch?v=Cq6CuBTwikw&t=1210s) we ran these tests, we are green — Ritesh Shah

> [20:10](https://www.youtube.com/watch?v=Cq6CuBTwikw&t=1210s) we run these tests, we are confident, and we will continue to run these tests — Ritesh Shah

> [27:50](https://www.youtube.com/watch?v=Cq6CuBTwikw&t=1670s) The capital of Peru can be Lima. It can be many other things based on context — Piyush Saxena

> [27:50](https://www.youtube.com/watch?v=Cq6CuBTwikw&t=1670s) We can tame stochasticity. We cannot weed it out. — Piyush Saxena

> [33:14](https://www.youtube.com/watch?v=Cq6CuBTwikw&t=1994s) doesn't need to know who won the World Series in '87. — Ritesh Shah

> [38:15](https://www.youtube.com/watch?v=Cq6CuBTwikw&t=2295s) don't do anything and not let the system move forward — Piyush Saxena

## Mentioned Vanguard, Piyush Saxena, Ritesh Shah, Ai4, AWS, AWS Bedrock, Anthropic, Claude Opus, PagerDuty, Chipotle, Cloud Code, Python, SQL

## Video description Ritesh Shah - Chief Architect - AI/ML/GenAI, Vanguard Piyush Saxena - Director AIML, Advice, Wealth Management & AI CoE, Vanguard

Your AI Shipped. Now What?

Your AI passed the tests. The demo impressed stakeholders. The pilot delivered promising results. But what happens when real users arrive and your model behaves differently than expected? How do you know it is delivering consistent, reliable outcomes across different user groups and scenarios? And when performance drifts, safety issues emerge, or evaluations produce conflicting signals, do you have the visibility and infrastructure to understand why?

Join us at AI4 to explore how leading enterprises are moving beyond manual evaluations, spreadsheets, and intuition to establish a rigorous, scalable approach for measuring and governing AI systems throughout their lifecycle. In this session, we will demonstrate how modern AI evaluation frameworks help organizations build confidence in production AI by combining persona-driven testing, customized evaluation methodologies, human oversight, and continuous monitoring into a single, integrated process.

Learn how Persona Management enables teams to evaluate models against realistic user archetypes rather than relying on a single “average” user, uncovering biases, gaps, and performance variations before they impact customers. See how Bring Your Own Judges (BYOJ) allows organizations to define success on their own terms using domain-specific rubrics, custom scorers, fine-tuned classifiers, or external LLM judges that align with business objectives. Discover how a Jury of Evaluators creates stronger, more trustworthy assessments by aggregating multiple independent perspectives, reducing the risk of relying on a single evaluation metric or model.

We will also showcase how Human-in-the-Loop workflows enable expert reviewers to focus on high-risk and low-confidence outcomes, while continuously improving automated evaluation mechanisms. In addition, attendees will learn how structured adversarial testing and red teaming can uncover vulnerabilities such as prompt injection attacks, jailbreak attempts, and edge-case failures before deploying solutions at scale.

A key highlight of the session will be showcasing our enablement architecture, illustrating how evaluation, governance, and observability capabilities are embedded across the entire AI delivery lifecycle - from experimentation and development to deployment and production operations. Attendees will see how a unified architecture enables continuous validation, monitoring, and improvement of AI systems, providing actionable insights when model behavior changes and ensuring organizations maintain trust, compliance, and performance over time.

Whether you are building AI-powered applications, deploying agents, or scaling generative AI across the enterprise, this session will provide practical strategies and architectural patterns to help you transform AI evaluation from an afterthought into a competitive advantage. Stop relying on guesswork. Start building AI systems you can measure, trust, and confidently scale.

[Source](https://www.youtube.com/watch?v=Cq6CuBTwikw)

中文简报

先锋集团采用精算风险模型在800个团队中规模化应用生成式AI 企业无法消除大语言模型的随机性，必须用基于模拟的精算抽样取代人工测试，从而在不线性增加人工审核员数量的前提下管控生产环境风险。 先锋集团（Vanguard）阐述了其在800多个工程团队中治理非确定性生成式AI的综合框架，该框架通过建立正式的风险预算，而非追求不可能实现的零风险确定性。该策略通过数万次由全局首席裁判（Principal Judges）和特定领域应用裁判（Application Judges）构成的两级层级体系评估的合成角色模拟，取代了人工用户验收测试。借助持续的领域专家（SME）反馈闭环与2x2混淆矩阵，先锋集团应用精算抽样，在保持统计置信度的同时将人工监督缩减至仅占总流量的一小部分。该平台直接集成到CI/CD流水线和生产环境实时可观测性中，严格执行漂移检测、事件升级、分级紧急停机开关及Token成本优化。 生成式AI系统的随机性无法完全消除；机构必须专注于在安全护栏与实用性之间找到最佳平衡，以驾驭这种随机性。 (0:04) 传统的人工用户验收测试（UAT）对于开放式大语言模型已不再适用，需要通过合成角色的自动化大规模模拟来建立精算级的统计置信度。 (3:09) 评估采用两级裁判架构：企业级首席裁判在全局范围内强制执行诸如毒性等基线标准，而特定领域应用裁判则负责评估专有任务。 (6:44) 孤立的非确定性模型事件不应触发告警；运维事件升级和紧急停机开关必须与突破明确业务风险预算的聚合问题挂钩。 (12:05) 在混淆矩阵中将领域专家（SME）决策与AI裁判评分进行对照映射，能够区分智能体缺陷与裁判校准需求，从而构建专有的黄金基准数据集。 (14:55)

Reading summary

Piyush Saxena and Ritesh Shah of Vanguard argue that because organizations cannot completely eliminate randomness from generative AI, enterprise governance must focus on taming stochasticity through actuarial-grade synthetic simulations, hierarchical automated judges, explicit business risk budgets, and continuous human calibration. High-Volume Synthetic Personas Replace Manual Testing (3:09) Traditional user acceptance testing with small groups of human testers fails to validate non-deterministic language models capable of generating infinite variations of responses. To establish statistical confidence without human bottlenecks across roughly 800 developer teams, Vanguard replaces manual ad-hoc testing with tens of thousands of automated simulations executed by synthetic client personas derived directly from real customer interaction data. Saxena explains that expanding evaluation volume from a few hundred manual runs to tens of thousands of synthetic interactions allows teams to make actuarial-grade statistical claims. This enables the enterprise to benchmark macro metrics, such as helpfulness and risk, down to tight margins such as 99.5% ± 0.05%, while allowing developers to iterate and re-evaluate prompts immediately. Hierarchical Judges and SME-Driven Calibration Loops (6:44) To evaluate multi-turn dialogs and API telemetry at scale, Vanguard structures automated Large and Small Language Model evaluators into two tiers. Principal Judges enforce unified, enterprise-wide guardrails for universal standards like toxicity and brand safety, preventing hundreds of teams from inventing conflicting definitions. Below them, Application-Specific Judges evaluate domain-tailored requirements, such as text-to-SQL syntactical accuracy and query performance. Because automated judges can drift and frontier models lack Vanguard's institutional ethos, subject matter experts (SMEs) are integrated into a continuous Human-in-the-Loop calibration workflow. SME agreements and disagreements with judge decisions form a 2x2 confusion matrix that separates true agent defects from judge false alarms. This accumulated feedback generates a proprietary benchmark dataset used to self-heal judges via prompt optimization or fine-tune task-oriented Small Language Models (SLMs) through reinforcement learning. Operationalizing Production Telemetry Through Risk Budgets (12:05) Production observability continuously tracks the mathematical delta between pre-production benchmarks and live KPIs, identifying model drift. Shah emphasizes that because generative models naturally produce non-deterministic variations, operational monitoring must not raise alarms on isolated model events. Instead, the observability platform aggregates events into issues, escalating them to actionable incidents (such as PagerDuty alerts or incident response calls) only when aggregated issues breach pre-set thresholds. These operational thresholds must be derived directly from an explicit business "risk budget" negotiated with senior business stakeholders. Risk tolerance is tailored by application context, demanding lower failure tolerance for public-facing or paid-tier applications than for internal crew-facing productivity tools. When an application breaches its risk budget, automated kill switches execute interventions, such as alerting operators, routing in-flight queries to human reviewers, or returning graceful fallback errors. Actuarial Sampling and Infrastructure Cost Control (18:05) Handling high-volume production (such as 100,000 interactions daily) without linear SME hiring requires moving away from exhaustive human oversight. Vanguard applies actuarial modeling to the evaluation confusion matrix: while 100% of interactions flagged by judges as defects undergo human review, judge-approved compliant outputs are routed to a statistical sampling pool. Actuarial formulas prove that reviewing a fraction—such as 20% to 30%—assumes the exact same residual risk as a 100% manual review. To control the infrastructure costs of running continuous evaluations, Vanguard avoids relying on trillion-parameter frontier models for routine evaluation tasks. Instead, teams deploy multiple-input multiple-output (MIMO) scoring prompts to evaluate correlated metrics like toxicity and length in a single API pass, while relying on self-hosted, domain-specific 50–60 billion parameter models and multi-cloud arbitrage to minimize per-token operational costs. The speakers caution that while managing model safety and regulatory risk is non-negotiable, enterprise governance must actively weigh the cost of inaction. Paralyzing AI deployments out of fear of non-zero risk can lead to strategic failure and lost market share, which frequently imposes a much steeper commercial penalty than manageable compliance risks.

精炼阅读摘要

先锋领航集团（Vanguard）人工智能团队指出，生成式系统固有的随机性无法被彻底根除，企业必须从追求绝对确定性转向“驯服随机性”；通过引入精算级置信区间、合成画像仿真、双层模型裁判体系、量化风险预算与生产可观测性闭环，可在全集团规模化部署生成式 AI 应用。 合成画像模拟与精算级置信区间 (3:09) 面对大语言模型（LLM）输出的开放性与非确定性，传统由几十名人工测试员点击界面的用户验收测试（UAT）已无法满足统计置信要求。例如在回答“秘鲁首都是哪里”等简单问题时，模型每次给出的语境和表述均可能动态变化，企业难以依靠人工经验划定风格演变与实质错误的边界。为此，先锋领航基于真实客户交互数据构建“合成数字画像”（Synthetic Personas），通过自动化引擎并发执行数万次模拟会话。 这种自动化高通量仿真使企业能够对宏观指标（如风险度、有用性）建立精算级的统计推断，将度量精度收敛至紧密区间（如 99.5% ± 0.05%）。在大规模合成测试套件的支持下，开发人员调整 Prompt 或增改功能后可立即重新验证系统，消除了传统人工测试瓶颈并加快了发布节奏。 双层模型裁判与专家在环动态校准 (6:44) 为自动化审核数万次模拟产生的高维多轮对话及底层 API 调用数据，先锋领航采用由大语言模型与任务型小语言模型（SLM）组成的双层裁判架构。第一层为全集团统一执行的“首要裁判”（Principal Judges），负责在 800 多个开发团队间强制落实毒性与品牌安全等通用底线标准；第二层为“专属裁判”（Application-Specific Judges），针对垂直领域（如自然语言转 SQL 的语法准确率与查询效率）进行定制化评估。 由于模型裁判本身也存在误报与漏报，且通用模型缺乏特定金融机构的业务直觉，先锋领航将业务领域专家（SME）纳入人在回路（HITL）校准机制。专家通过对裁判裁定进行简单的“赞同/反对”二元判定构建 2x2 混淆矩阵：真阳性暴露出底层生成应用的缺陷，误报与漏报则提示裁判自身存在过度保守或漏网风险。长期积累的专家决策沉淀为专有的黄金基准数据集。在纠偏路径上，团队遵循奥卡姆剃刀原则，优先优化裁判 Prompt；当业务规则过于复杂导致 Prompt 冗长时，则通过基于人类反馈的强化学习（RLHF）微调任务型小模型。 业务风险预算、生产漂移与分级熔断 (12:05) 系统在预发布阶段确立的关键绩效指标（KPI，包括毒性、完整度与有用性）必须在生产环境中被完全一致地持续监测，预发布基线与生产指标间的数学差值即定义为系统“漂移”（Drift）。在生产可观测性设计中，系统并不对孤立的模型异常事件直接拉响最高警报，而是将非确定性事件聚合为“问题”（Issues），唯有当问题累积突破预设阈值时才升级为可操作的运维事件（Incidents）。 Ritesh Shah 强调，监控阈值绝非凭空设定，而必须脱胎于与业务及风控伙伴协同敲定的“风险预算”。针对公开发布的聊天机器人、付费核心客户以及先锋领航内部 5000 多名员工应用，各场景的风险容忍度截然不同。一旦低容忍维度（如毒性）突破风险预算，系统将自动触发分级熔断机制：第一级向人工操作员发送报警；第二级暂停自主响应并将流转中的请求拦截至人工审核；第三级则直接终止执行并返回安全的兜底错误信息。 精算抽样降本与平台化工程落地 (18:05) 在日均处理 10 万次以上交互的企业级规模下，依赖全量人工审核在运营上无法扩展。先锋领航引入精算学抽样原理：对模型裁判标记为存在缺陷的交互执行 100% 专家复核，而对裁判放行的合规交互池，则依据精算风险模型计算出统计抽样比例（如 20% 至 30%）。数学证明显示，在该抽样比例下系统承担的剩余风险与 100% 全量人工复核相当，从而在受控的风险预算内显著压降了人工审核成本。 在基础设施层面，先锋领航已从早期的 Python 本地开发库演进为部署在 AWS 上的集中化评估平台，供工程团队、产品负责人及合规人员协同配置画像、自选裁判与基准数据。为降低高通量评估带来的算力支出，团队采用单次调用评估多项维度的“多输入多输出”（MIMO）Prompt，并自主托管约 500 亿至 600 亿参数的专用评测小模型，以取代昂贵的万亿参数通用模型。其长期架构目标是在 CI/CD 流水线中引入并行智能体完成对抗测试与合规审查，使模型裁判成为贯穿整个软件开发生命周期、持续版本化迭代的核心资产。 先锋领航的实践表明，生成式 AI 的治理核心在于将评估从传统的静态验收转变为对概率分布与剩余风险的动态运营。演讲者提醒，在严格防范合规与安全漏洞的同时，企业切忌因过度保守而陷入决策瘫痪；在竞争激烈的市场环境下，完全不作为所导致的市场地位丧失，其长期商业代价往往远高于可控的监管罚款与试错成本。

Detailed notes — zh

## Background 该内容由 Ai4 于 2026 年 8 月 15 日发布在 YouTube 平台上，记录了先锋领航集团（Vanguard）AI/ML 首席架构师 Ritesh Shah 与 AI 卓越中心总监 Piyush Saxena 的深度分享。视频介绍了领先企业如何告别传统手动测试与电子表格，通过用户画像管理、自定义裁判矩阵、人机协同（HITL）及全生命周期可观测性架构，建立严谨可扩展的生成式 AI 评估与治理体系。 ## TL;DR 演讲围绕测试、生产可观测性、监督与自我修复四大支柱，系统阐释了金融巨头 Vanguard 驯服生成式 AI 随机性的实战框架。团队通过基于真实数据的大规模合成画像模拟与双层 AI 裁判矩阵（通用原则与应用专用），替代传统样本量不足的 UAT 测试，并运用精算学模型将专家人工监督抽样率安全调低。在生产运维中，系统构建了从底层事件、聚合问题到业务事故的递进响应与熔断机制，并依托融合多智能体的 CI/CD 质量门禁，将设计态的风险预算与运行态的持续漂移监控无缝拉通，最终在技术风险与停滞不前的商业危机之间达成理性平衡。 ## Atmosphere & style 整场演讲洋溢着金融工程特有的严谨务实与工业级落地的从容底气，两位讲者配合默契、风趣生动。通过“秘鲁首都是哪里”的现场互动、向高管汇报时的戏剧性反转，以及“点餐机器人解数学题”等生动隐喻，讲者将枯燥的精算学风控与复杂的模型治理拆解得引人入胜。现场互动频繁且充满心领神会的共鸣，传递出一种在高度不确定的生成式浪潮中用严密工程体系构建确定性护城河的专业力量感。 ## Key takeaways - [00:04](https://www.youtube.com/watch?v=Cq6CuBTwikw&t=4s) 生成式 AI 本质具备概率与随机特性，企业无法彻底根除随机性，必须通过严密的评估体系将其驯服在可控范围内。 - [01:19](https://www.youtube.com/watch?v=Cq6CuBTwikw&t=79s) 强监管环境下单纯通过严苛 Prompt 封堵风险会导致模型失去实用性，必须在风险控制与帮助性之间寻求精准平衡。 - [04:07](https://www.youtube.com/watch?v=Cq6CuBTwikw&t=247s) 大模型无限自由度的输入使得传统小规模人工 UAT 失效，需引入基于真实数据衍生的合成用户画像（Synthetic Personas）进行数万次大规模自动化模拟以获得精算级置信度。 - [06:44](https://www.youtube.com/watch?v=Cq6CuBTwikw&t=404s) AI 裁判体系应划分为通用原则裁判（负责统一毒性与合规红线）和应用专用裁判（校验特定垂直业务逻辑），兼顾底线与差异化需求。 - [08:29](https://www.youtube.com/watch?v=Cq6CuBTwikw&t=509s) 前沿基础模型不具备企业专有业务理解，必须通过人在回路（HITL）引入领域专家（SME）的持续反馈来校准 AI 裁判并沉淀核心金标基准数据集。 - [12:05](https://www.youtube.com/watch?v=Cq6CuBTwikw&t=725s) 生产可观测性遵循“底层事件 → 聚合问题 → 业务事故”的分级流转逻辑，单点波动不应滥发告警，只有突破预设风险预算时才触发分级响应与熔断机制。 - [18:05](https://www.youtube.com/watch?v=Cq6CuBTwikw&t=1085s) 通过精算学统计建模与混淆矩阵历史表现，可以在严格控制业务残留风险的前提下，将全量人工审核安全调低至 20%~30% 的抽样监督比例。 - [20:10](https://www.youtube.com/watch?v=Cq6CuBTwikw&t=1210s) 当裁判产生误判时遵循奥卡姆剃刀原则：优先进行 Prompt 级自动化优化与增强，遇到长上下文与高延迟瓶颈时再转向专用小语言模型（SLM）及 RLHF 微调。 - [21:58](https://www.youtube.com/watch?v=Cq6CuBTwikw&t=1318s) Vanguard 评估平台从轻量 Python SDK 演进至 AWS 集中式 Web 平台，服务对象从底层算法工程师扩展至产品负责人及风控合规团队。 - [24:44](https://www.youtube.com/watch?v=Cq6CuBTwikw&t=1484s) 面向未来的架构在 CI/CD 中集成配置、对抗测试与合规审计等专用智能体集群，并在投产后自动同步发布前后的裁判标准以实现全链路对齐。 - [31:45](https://www.youtube.com/watch?v=Cq6CuBTwikw&t=1905s) 通过 MIMO 多指标单次推理聚合、采用 50B~60B 任务专用小模型替代万亿参数大模型，以及从托管服务逐步迁移至自建算力来大幅优化评估 Token 成本。 - [37:01](https://www.youtube.com/watch?v=Cq6CuBTwikw&t=2221s) 风险预算不能一刀切，应由技术与产品团队依据高净值客户、公开交互、内部员工等不同场景发生故障时的实际破坏力差异化制定。 - [38:15](https://www.youtube.com/watch?v=Cq6CuBTwikw&t=2295s) 企业在评估 AI 风险时不仅要衡量技术落地带来的合规罚款或偶发错误，更要正视停滞不前、不作为可能导致彻底丧失市场份额与竞争力的更大危机。 ## Detailed walkthrough ### [00:04](https://www.youtube.com/watch?v=Cq6CuBTwikw&t=4s) 驯服随机性：Vanguard 的 GenAI 规模化评估与可观测性之旅 来自先锋领航集团（Vanguard）的 AI 卓越中心（AI CoE）负责人 Piyush Saxena 与 AI/ML 及 GenAI 首席架构师 Ritesh Shah 共同开场，向大家分享企业在规模化落地大语言模型（LLM）过程中的评估与可观测性实践。演讲主题中的“TAMING STOCHASTICITY”（驯服随机性）之所以全部采用大写，是因为团队清醒地认识到：在本质上具备生成特性的系统里，随机性是“只能被驯服，而无法彻底根除”的。 Piyush 回顾了 Vanguard 大约一年多前启动 LLM 应用建设以来的旅程。他幽默地表示团队要分享所有成功以及“所有的经验与学习，因为我们从不言失败（we don't fail）”。整场分享将围绕测试（Testing）、生产可观测性（Production Observability）、监督（Oversight）、自我修复（Self-healing）四大核心支柱展开，并逐步深入到 Vanguard 当前运行的架构细节与关键认知。 ```mermaid flowchart TD accTitle: Vanguard GenAI 评估与可观测性核心支柱 subgraph S1["预生产验证阶段 (Pre-Production)"] A["大规模用户画像模拟<br/>Synthetic Persona Simulations"] --> B["双层 AI 裁判评测<br/>Principal & App-Specific Judges"] B --> C["领域专家校准与回路<br/>SME HITL Calibration"] end subgraph S2["生产运行与治理阶段 (Production Operations)"] D["生产遥测与行为监控<br/>Telemetry & Events"] --> E["漂移与异常检测<br/>Issues & Incidents"] E --> F["业务告警与自我修复<br/>Alerts & Self-Healing"] end C -. "基线指标对齐".-> D ``` --- ### [01:19](https://www.youtube.com/watch?v=Cq6CuBTwikw&t=79s) 监管高压下的破局：在“安全性”与“实用性”之间寻找平衡 时间倒回一年半前，Vanguard 全球 800 多个研发团队开始尝试构建第一批生成式 AI 应用。作为一家管理巨额资产的金融机构，Vanguard 处于极高要求的强监管行业中，客户信任与合规伦理是不可逾越的底线。在一次向高管汇报演示时，一位高级领导直接抛出了灵魂质问：“我怎么知道这个应用不会产生幻觉？它到底安不安全？” 研发团队领命后迅速采取防御措施，通过大量严苛的 Prompt 提示词工程，试图彻底封堵模型输出有害内容的任何可能性。然而，基于全网公开数据训练出的通用大模型在此类强限制下迅速走向了另一个极端。在第二轮汇报中，高管们的反馈变成了：“这东西一点用都没有，无论问什么它都回答‘无法回答’。”这次经历成为了 Vanguard 探索评估体系的起点——团队必须在“风险控制（Risk/Harmfulness）”与“帮助性（Helpfulness）”之间找到精准的平衡点。 为了向现场观众形象地解释生成式模型的“随机性”，Piyush 做了一个互动提问：“秘鲁的首都是哪里？”全场异口同声回答“利马”。Piyush 随即反问：“为什么是利马？为什么不是南美美食之都？或者利马享誉全球的其他二十个标签？”他指出，大模型的输出严重依赖于上下文和对话历史；即便针对同一个问题连续提问 10 次，模型给出的答案都会存在微小变动。评估的核心就在于识别哪些变异是在可接受的语义范围内，哪些变异已经越界退化成了错误或有害的回答。 --- ### [04:07](https://www.youtube.com/watch?v=Cq6CuBTwikw&t=247s) 大规模模拟与用户画像（Persona）：用精算级置信度颠覆传统 UAT 在传统软件和经典机器学习时代，用户验收测试（UAT）通常组织 20 到 100 名内部用户对系统进行操作打分。传统系统的功能范围有严格边界且具备确定性，UAT 完全足以覆盖。但在大模型面对无限自由度输入的场景下，仅靠有限的人工测试根本无法穷尽潜在问题。 为了解决这一痛点，Vanguard 引入了基于真实客户数据衍生出的“合成客户画像（Synthetic Client Personas）”，通过自动化管道发起数万次多轮模拟交互。Piyush 强调“输入劣质数据就会输出劣质结果（Garbage in, garbage out）”，因此虚拟画像必须精确映射实际用户的交互习惯与行为模式。 大规模自动模拟不仅保障了研发迭代的速度（开发人员修改 Prompt 或增加功能后无需等待缓慢的手工测试），更带来了量化风险的能力。传统的数百次人工测试无法建立足够收敛的置信区间，难以通过严格的上线审批委员会；而数万次的大规模模拟能够让工程团队像精算师一样提供高精度的量化论据——例如向委员会证明该应用的安全指标达到“99.5% ± 0.05% 的置信水平”，用确定性的统计数据为业务部署建立真正的信心。 - **传统测试局限**：人工 UAT 样本量通常只有数百到上千次，无法为大模型的无限输入空间提供严密的置信区间。 - **用户画像衍生**：合成画像直接源于企业沉淀的客户历史交互数据，确保仿真压力测试贴近真实业务场景。 - **精算级置信度**：通过数万次自动化模拟，输出类似 `99.5% ± 0.05%` 的量化风险评估报告。 --- ### [06:44](https://www.youtube.com/watch?v=Cq6CuBTwikw&t=404s) 评估裁判体系：通用原则裁判（Principal Judges）与应用专用裁判（Application-Specific Judges） 当系统每天产生数万次涵盖多轮对话、API 接口调用、工具检索（Tool Calling）的复杂交互遥测数据时，人工逐条审核变得完全不切实际。Vanguard 引入了大语言模型与面向特定任务的小语言模型（SLM）构成的“AI 裁判（Judges）”评测矩阵，并将其清晰划分为两大维度： ```mermaid flowchart LR accTitle: AI 裁判体系分类架构 subgraph Principal["通用原则裁判 (Principal Judges)"] P1["全企业统一毒性标准 (Toxicity)"] P2["基础安全性与合规红线 (Safety & Redlines)"] end subgraph Specific["应用专用裁判 (Application-Specific Judges)"] S1["自然语言转 SQL (NL2SQL) 效率与准确率"] S2["垂直业务逻辑与工具调用链校验"] end ``` 第一类是**通用原则裁判（Principal Judges）**，负责把控跨业务的通用底线。例如“毒性（Toxicity）”或“有害性”的定义必须在全集团保持绝对统一，不能 100 个研发团队各自维护 100 种毒性判定标准。第二类是**应用专用裁判（Application-Specific Judges）**，专门针对特定业务特征进行深度校验。例如一个负责将客户自然语言转换为 SQL 查询的应用，就需要专门的裁判来评估 SQL 转换的语法有效性、执行效率以及安全性。这种分层机制既保证了企业级红线的统一，又兼顾了各个业务用例的差异化特征。 --- ### [08:29](https://www.youtube.com/watch?v=Cq6CuBTwikw&t=509s) 人在回路（HITL）与自我修复：校准 AI 裁判并融入先锋领航专属资产 AI 应用自我修复的基石在于结合客户交互数据与内部领域专家（SME）的反馈。Piyush 明确指出，市面上任何前沿基础大模型（Frontier Models）都不可能天然理解 Vanguard 的投资方法论与企业价值观，这些专有知识才是企业的“核心秘方（Special Sauce）”。 同时，AI 裁判本身也是机器学习模型，不可能达到 100% 的准确率（最多达到 99.99%），且随着客户交互习惯和业务环境的演变，裁判的标准也会发生偏移。因此，必须将领域专家的持续评估作为“人在回路（Human-in-the-Loop, HITL）”反馈流引入评测闭环，用来动态校准 AI 裁判。 在长达一年多的实践中，Vanguard 持续对比 AI 裁判的打分与业务专家人工打分的一致性，推动裁判模型与专家判断逐步收敛。人在回路不仅是模型演进的催化剂，更是应对未来客户行为变迁、保障系统长期稳定表现的防波堤。 --- ### [10:47](https://www.youtube.com/watch?v=Cq6CuBTwikw&t=647s) 生产环境可观测性：从事件（Events）到告警（Alerts）的漂移监控 完成了预生产环境的模拟测试与裁判校准之后，系统将进入生产可观测性阶段。上线评估的核心逻辑非常明确：在生产环境中持续监测与预生产阶段完全一致的宏观 KPI（例如毒性评分、回答完整性等）。预生产基准与生产实际表现之间的差值（Delta），即构成了系统性能的“漂移（Drift）”。 首席架构师 Ritesh Shah 进一步阐述了生产运维的流转逻辑。在实际运行中，AI 系统无时无刻不在生成底层运行事件（Events）；一旦这些事件偏离了既定指标要求，就会被归集为潜在问题（Issues）；当问题达到一定阈值或表现出系统性风险时，便会升级为需要重点跟进的事故（Incidents）；最终触发运维与业务告警（Alerts），精准通知对应的业务单元介入处置，确保线上 AI 系统的行为始终处于受控状态。 ### [12:05](https://www.youtube.com/watch?v=Cq6CuBTwikw&t=725s) AI 可观测性：从事件（Event）到事故（Incident）与风险预算 Ritesh Shah 和 Piyush Saxena 详细阐述了 Vanguard 内部针对 AI 系统建立的可观测性与响应框架。在生产环境中，系统产生的异常并非每个都要拉响警报。整个治理链路遵循“底层事件（Event） $\rightarrow$ 聚合问题（Issue） $\rightarrow$ 业务事故（Incident）”的递进逻辑。一旦定性为 Incident，系统会引入人工评分，将其划分为“可接受”、“糟糕”或“极其恶劣”，并据此采取分级响应措施。 ```mermaid flowchart TD accTitle: AI Observability and Incident Escalation Lifecycle subgraph Observability ["可观测性与异常分级"] E["底层事件 (Event)<br/>随机波动/单点异常"] --> ISS["聚合问题 (Issue)<br/>模式分析与指标偏离"] ISS --> INC{"是否突破<br/>风险预算?"} end subgraph Actions ["事故响应与熔断机制"] INC --"否"--> MON["持续静默监控 / 记录"] INC --"是"--> SEV["升级为事故 (Incident)<br/>触发 MIM / PagerDuty"] SEV --> KS["熔断机制 (Kill Switch)"] KS --> K1["触发告警，人工介入响应"] KS --> K2["实时路由至专家审核后返回"] KS --> K3["自动阻断并返回降级错误信息"] end ``` Piyush 向现场观众互动发问，询问在场有多少人属于或领导着应用开发团队，现场大部分人都举起了手。紧接着他抛出了核心问题：“**你们当中有谁制定了风险预算（Risk Budget）？**”他强调，面对生成式与 AI/ML 模型的随机本质（Stochasticity），企业必须明确并接受一定的“残留风险（Residual Risk）”。如果业务层面没有界定清晰的风险承受底线，所有的监控阈值都只是拍脑袋的任意数字： - **Event 不是红牌**：单次 Event 只是生成式模型的固有波动或离群点，绝不能频繁触发全局告警； - **聚合后对照风险预算**：只有当多维度 Event 累积突破预设的风险阈值时，才升级为 Incident，进而触发重大事故响应（MIM 会议呼叫）或 PagerDuty 传呼； - **分维度的熔断开关（Kill Switch）**：以最典型的“毒性（Toxicity）”维度为例，如果该维度的风险容忍度极低，系统必须配备一键切断的熔断开关。 Ritesh 补充道，熔断开关的设计形态因场景而异： 1. **告警响应型**：系统产生告警，由人工后台介入处理； 2. **流量拦截与审核型**：实时请求被拦截并路由至人工专家，专家审核确认后再将响应放行回传； 3. **模型自动熔断型**：由前置分类模型或风控拦截能力直接阻断请求，向前台返回预置的降级错误提示，从根本上阻断潜在风险扩散。 --- ### [14:55](https://www.youtube.com/watch?v=Cq6CuBTwikw&t=895s) SME 与混淆矩阵：让人类专家决策驱动裁判校准与模型自愈 作为技术赋能（Enablement）团队，Piyush 指出其核心使命是“最高效地利用领域专家（SME）的时间”。在设计人在回路（HITL）流程时，SME 不需要填写繁冗的表单，**其核心输入应当精简到极致——只有“同意（Agree）”或“不同意（Disagree）”**。而后台工程与算法框架则负责将这些看似简单的二元决策映射为系统自愈与裁判校准的量化动力。 ```mermaid flowchart LR accTitle: SME Confusion Matrix and Feedback Actions subgraph ConfusionMatrix ["SME 混淆矩阵判定"] direction TB TP["True Positive (真正例)<br/>AI 裁判报险 + SME 同意"] FP["False Positive (假正例)<br/>AI 裁判报险 + SME 否决"] FN["False Negative (假负例)<br/>AI 裁判放行 + SME 报险"] TN["True Negative (真负例)<br/>AI 裁判放行 + SME 认可"] end subgraph Optimization ["驱动系统迭代"] TP --> AG_OPT["智能体优化 (Agent Improvement)<br/>暴露应用自身缺陷"] FP --> JC["裁判校准 (Judge Calibration)<br/>降低裁判过度保守倾向"] FN --> JC FN --> BR["控制业务风险 (Business Risk)"] TN --> BM["沉淀金标基准数据集 (Benchmark Data)"] end ``` Piyush 用经典的数据科学混淆矩阵（Confusion Matrix）拆解了 SME 决策的深层含义： - **真负例（True Negative）**：位于右下角，代表系统判定正常且 SME 也认可。这是团队最希望高度密集的区域，表明系统运行平稳，AI 裁判与人类专家判定高度一致。 - **真正例（True Positive）**：AI 裁判判定存在异常，且 SME 也同意该判定。这暴露了实际应用/智能体（Agent）的表现缺陷，是**智能体自身迭代改进的直接切入点**。 - **假负例（False Negative，Type II 错误）**：AI 裁判放行漏判，但 SME 认为存在问题。这是企业面临的**真实业务风险（Business Risk）与隐性盲区**。 - **假正例（False Positive，Type I 错误）**：AI 裁判过于保守报险，但被 SME 否决。这代表**业务管理开销（Business Overhead）**与不必要的人工审查浪费。 Piyush 强调，假正例与假负例呈现反比权衡关系。矩阵的非对角线区域（FP 与 FN）直接驱动 **AI 裁判校准（Judge Calibration）**，而对角线上的正例与负例则不断扩充为高质量的基准数据集（Benchmark Dataset）。“*在每个人都在构建 GenAI 应用的今天，你的真正护城河是什么？SME 的标注数据就是护城河。*” 随着基准数据集的不断扩大，团队对模型上线与版本变更的信心也呈指数级上升。 --- ### [18:05](https://www.youtube.com/watch?v=Cq6CuBTwikw&t=1085s) 规模化人类监督：利用精算学方法动态调低审查比例 在企业级规模下，生成式 AI 系统每天可能产生超过 100,000 次交互。如果要求人类专家对每一次交互进行全量监督，即便 SME 只是简单点击“同意/不同意”，所需的人力成本也是天文数字，“*世上根本没有一整个集团军的 SME 供我们招聘*”。因此，如何将人类监督的旋钮从 100% 安全地调低，成为 AI 规模化落地的生死线。 ```mermaid flowchart TD accTitle: Actuarial Science Sampling and Dialing Down Oversight INT["每日 100,000+ 次模型交互"] --> JUDGE{"AI 裁判判定"} JUDGE --"判定有风险 (TP / FP)"--> FLAG["100% 标记拦截<br/>强制进入人工复核流"] JUDGE --"判定正常 (TN / FN)"--> SAMPL["精算学统计抽样池 (Sampling Pool)"] SAMPL --> ACT["精算学风险建模<br/>依据混淆矩阵历史 KPI 计算置信区间"] ACT --> DIAL["动态调节监督旋钮 (Oversight Dial)<br/>抽样率降至 20% ~ 30%"] DIAL --> RSK["在风险预算内证明：<br/>20% 抽样监督与 100% 全审具备等价残差风险控制力"] ``` Vanguard 引入了**精算学（Actuarial Sciences）**的统计方法来解决该问题： - 在混淆矩阵中，左侧被 AI 裁判标记为有风险的样本（TP 与 FP）必须 100% 拦截并进入审查流程； - 右侧被 AI 裁判判定为安全的区域（TN 与 FN）则构成可供抽样的样本池； - 借助精算建模与混淆矩阵中的历史 KPI，团队能够以严密的数学证明推导出：**在给定的风险预算约束下，采取 20% 或 30% 的抽样监督比例所承担的业务残留风险，与采取 100% 全量监督在统计学上是等价的**。 这种由数据与精算模型驱动的降维监督，使得企业在严格控制风险敞口的同时，实现了业务规模化吞吐。 --- ### [20:10](https://www.youtube.com/watch?v=Cq6CuBTwikw&t=1210s) 裁判校准路径与范式转变：Prompt 优化与 SLM 的 RLHF 当 AI 裁判出现误判（FP/FN）时，团队遵循“奥卡姆剃刀（Occam's Razor）”原则进行分层治理与系统自愈： 1. **Prompt 级裁判优化**：优先使用基于 Prompt 的裁判。系统内置自动化机制，利用 SME 的反馈数据自动执行 Prompt 增强（Augmentation）与优化（Optimization），并在基准数据集上重新运行以输出最新评估指标； 2. **小语言模型（SLM）与 RLHF**：随着业务逻辑复杂度提升，评测 Prompt 的上下文长度往往会急剧膨胀，导致推理成本与延迟飙升。此时，团队转向使用面向特定任务的专用小语言模型（SLM），并将 SME 提供的点赞/点踩、同意/不同意等二元数据无缝接入强化学习流程（RLHF），实现小模型的持续微调。 Piyush 总结道，行业必须经历一次根本性的认知范式转移： > “我们必须从过去传统的‘*我们跑完了这些测试，系统绿灯，可以发布了*’，转变为‘*我们跑完了这些测试，系统具备置信度，但我们仍将在生产中持续运行这些测试，以确保应用在随机性扰动下始终符合预期*’。” --- ### [21:58](https://www.youtube.com/watch?v=Cq6CuBTwikw&t=1318s) Vanguard 评估平台架构演进：从 Python 库到 AWS 自研企业级平台 Ritesh Shah 揭示了 Vanguard 评估体系在工程实现上的演进路径。为了最大程度降低业务团队的接入摩擦，赋能团队采取了“**在业务团队所在的地方与他们汇合（Meet the teams where they are）**”的产品策略： 1. **第一阶段：Python 核心 SDK / 库**：初期封装为一个轻量级 Python 库，供算法工程师与开发人员在本地或开发环境中进行早期的快速单元测试与原型评测； 2. **第二阶段：AWS 集中式自定义评估平台**：随着需求深化，系统升级为运行在 AWS 上的端到端企业级 Web 平台，供全公司各业务产品团队在投产前进行标准化的评估与合规验收。 ```mermaid flowchart TD accTitle: Vanguard AI Evaluation Platform Architecture on AWS subgraph UI ["用户交互层 (Web Component)"] WEB["统一 Web 交互界面<br/>(面向 Dev / PO / 风控专家)"] end subgraph Core ["平台核心编排与能力引擎 (AWS Engine)"] PJ["裁判配置引擎 (Judge Setup)"] PM["角色管理 (Persona Setup)"] SIM["基准真值模拟器 (Ground Truth Simulations)"] EVAL["多维度评估调度与流水线编排"] end subgraph Output ["产出与决策门禁"] REP["多维度系统表现评估报告"] GOV{"投产决策判定门禁<br/>(对照业务残差风险预算)"} PROD["正式放行至生产环境 (Production)"] end WEB --> PJ & PM PJ & PM --> EVAL SIM --> EVAL EVAL --> REP REP --> GOV GOV --"通过"--> PROD ``` #### 平台核心功能与全生命周期工作流 - **裁判与角色设定**：产品团队可在界面上灵活配置自定义 AI 裁判（Judges）以及用于模拟不同用户画像的行为角色（Personas）； - **基准真值仿真**：基于业务提供的 Ground Truth 数据集自动执行批处理模拟演练； - **多维度评估报告**：系统自动生成跨多维度的详尽评估报告，全方位量化 Agent 或 AI 系统的性能、安全性与可靠性； - **投产决策门禁**：业务团队每次更新应用时均可触发定时评估任务，输出的报告直接作为该应用是否准许上线的关键依据。 Piyush 补充指出，随着平台在过去数年间的演进，其**服务客群（Customer Base）已经从最初单一的底层开发者，拓展至产品负责人（Product Owner）、部门风控团队（Divisional Risk）以及合规生态圈**，构建起了横跨技术与治理的统一风险管理底座。 ### [24:44](https://www.youtube.com/watch?v=Cq6CuBTwikw&t=1484s) 统一评估标准与基于智能体的 CI/CD 愿景 在企业级 AI 系统的落地过程中，各职能部门（开发团队、业务部门、企业风险管理部门等）往往各自为战。Ritesh Shah 与 Piyush Saxena 强调，所有这些团队绝不能对“应用的实际表现”持有彼此割裂、互相冲突的衡量标准。系统设计必须从底层提供统一的服务支撑：无论是持续集成与持续交付（CI/CD）流水线、命令行界面（CLI）、智能体（Agents）还是图形用户界面（UI），底层都必须依托同一套后端架构，对“什么是用户画像（Persona）”、“什么是仿真（Simulation）”以及“预期输出应当具备何种特征”拥有完全一致的核心定义。 对于一线软件开发者而言，现代化的开发工作流早已借助 Cloud Code 等工具深度融合进代码编辑器，日常测试根本不会通过点击 Web UI 来完成。面向未来的核心愿景，是在 CI/CD 流水线中深度集成一系列高度专业化的智能体： - **前置配置智能体（Setup Agents）**：自动拉取并初始化评估所需的测试画像、配置专属裁判（Judges）及基准数据集。 - **并行评估智能体集群（Parallel Evaluation Agents）**：在代码合并前并行发起多维度的全方位评估，包括业务逻辑评估、对抗性红队测试（Adversarial Evaluation）、企业内部定义的“可信 AI（Trusted AI）”指标，以及合规与监管审计测试。 - **质量门禁与生产发布（Quality Gates）**：流水线汇总各项评估指标，若符合准出标准并获得系统审批，则自动推进至生产部署环节。 - **上线后可观测性对齐（Post-deployment Observability Setup）**：部署上线绝非终点。CI/CD 流水线会立刻自动配置生产环境的可观测性体系，直接导入开发阶段使用的同一批裁判标准，确保“发布前（Pre-prod）”与“发布后（Post-prod）”的风险衡量标尺完全对齐，并自动挂载告警和触发器。 ```mermaid flowchart TD accTitle: CI/CD Pipeline Agentic Evaluation Architecture accDescr: Pipeline showing setup agents, parallel evaluation agents, deployment gates, and post-production observability sync. subgraph Setup ["前置准备阶段"] A["代码变更 / CI 触发"] --> B["配置智能体 (Setup Agent)"] B --> C["注入 Personas & Judges"] end subgraph Eval ["并行评估阶段 (Parallel Agents)"] C --> D1["业务逻辑评估"] C --> D2["对抗与安全测试"] C --> D3["可信 AI & 合规审计"] end subgraph Deploy ["质量门禁与投产"] D1 & D2 & D3 --> E{"评估门禁审查"} E -->|"通过"| F["部署至生产环境"] E -->|"未通过"| G["阻断构建并反馈开发者"] end subgraph Ops ["生产运行与可观测性"] F --> H["自动配置可观测体系"] H --> I["无缝同步裁判标准 (Pre/Post-prod)"] H --> J["激活动态告警与触发器"] end ``` --- ### [27:42](https://www.youtube.com/watch?v=Cq6CuBTwikw&t=1662s) 核心认知转变：重塑风险预算与驯服随机性 进入核心总结阶段，讲者梳理了企业领导层与工程团队必须建立的三项关键认知跃迁： 1. **重新定义“风险”与确立“风险预算（Risk Budget）”**： 行业内充斥着对模型“准确率（Accuracy）”或“简洁性”的宽泛讨论，但缺乏脱离业务语境的绝对数字。团队必须从精算学（Actuarial Science）中汲取经验，首先锚定用例的“风险预算”，再倒推模型达成业务价值所需的确定性阈值。大语言模型本质具有上下文概率特征——“秘鲁的首都可以是利马，但也可能因特定上下文而产生不同表述”。面对大模型系统，传统软件工程中小规模的业务验收测试（UAT）已经失效，必须引入具备统计显著性的大样本评估。同时，要清晰区分“固有风险（Inherent Risk）”与在层层防御和护栏约束后愿意承担的“残留风险（Residual Risk）”。所有这些风控与自愈决策，必须在“设计阶段（Design Time）”前置敲定，而非上线后的亡羊补牢。 2. **驯服随机性（Taming Stochasticity）而非根除它**： "生成式 AI 系统本质上是具有随机性的。我们能够驯服随机性，但无法彻底消除它。"如果业务场景完全无法容忍任何发散，理应选择传统经典机器学习或确定性规则系统，后者在特定领域依然极为高效稳定。 3. **裁判评分并非静态的“单元测试”**： 切忌将 LLM Judge 的打分视作传统代码的单元测试断言。裁判评分会随着应用本身的代码迭代而动态演变；更关键的是，即便应用程序纹丝不动，**用户的交互模式也会随时间剧烈漂移**。Reddit 上充斥着各种模因恶搞（例如跑到 Chipotle 点餐机器人前让它解数学题或编写 Python 代码），一旦外部交互行为发生裂变，评估分布必须能够随之动态调整。 4. **将裁判工具纳入软件开发生命周期（SDLC）**： 基于 Prompt 的判定规则或轻量级小语言模型（SLMs）绝对不能当作外部黑盒黑产。它们是开发团队核心资产库的一部分，必须接受同等级别的版本控制、代码审查、测试和发布流程，并要求团队全面适配这种基于风险概率的新型交付范式。 --- ### [31:45](https://www.youtube.com/watch?v=Cq6CuBTwikw&t=1905s) 现场问答：多层防御自愈机制与评估成本优化（Tokenomics） 在演讲结束后的 Q&A 环节，观众针对生产落地中的真实痛点展开提问，两位讲者分别从工程实践与成本效益拆解了防御与降本架构： 针对“生产运行中如何及时发现异常并完成闭环”的提问，Piyush 阐明了双层防御与自愈体系： - **第一道防线（自动化实时拦截）**：所有在线交互均旁路挂载 LLM Judges 实时监控，接入统一可观测平台，对异常内容与违规流量进行即时拦截和标记。 - **第二道防线（领域专家介入与回流）**：由于自动化裁判不可避免地存在假阳性（误报）与假阴性（漏报），被标记的高风险或低置信度内容会被推送给业务领域专家（SMEs）进行二次复核。复核清洗后的数据成为自愈闭环的关键燃料，反哺给提示词自动优化管线、数据增强管线或对齐微调（Chef pipelines）。 针对“多维度裁判并行运行时昂贵的模型调用与 Token 成本”问题，讲者给出了多维度的优化组合拳： - **评估指标聚合（MIMO 取代 SISO）**：避免为每个细微维度单独派发一次模型调用。借鉴数学上的多输入多输出（MIMO）理念，输出之间本身具备高度关联性。若已知文本的 Token 长度分布并需要检测毒性（Toxicity），完全可以将毒性与相关的语义偏离维度合并为单次 Prompt 判定，将三次独立的评估合并为单次推理。 - **放弃大模型依赖，拥抱高任务针对性小模型**：评估裁判无需通晓“谁赢得了 1987 年的世界大赛”，它只需要精通毒性识别或实体提取。Vanguard 企业 AI 与研究团队自研构建了参数量约在 50B 至 60B 的任务专用模型，用极高的吞吐与极低的推理开销，替代了参数量动辄上万亿的 Claude Opus 或庞大的 500B 开源模型，且兼顾了任务准确率、敏感度（Sensitivity）与特异度（Specificity）。 - **云基础设施阶梯化降本**： 1. 初始阶段：利用 AWS 原生 Bedrock 服务快速验证； 2. 进阶阶段：在自有实例（Self-hosted Vanguard 算力）上自建部署经过基准测试的开源模型，显著削减托管加价成本； 3. 自研专用轻量模型（Custom Task-Specific Models）； 4. 跨多云（Multi-hyperscaler）分流评估压力，防止单一云提供商的算力配额瓶颈导致评估延迟堆积； 5. 探索与主流模型供应商（如 Anthropic）签订企业专属批量吞吐协议。 --- ### [37:01](https://www.youtube.com/watch?v=Cq6CuBTwikw&t=2221s) 现场问答：如何与业务方共同制定“风险预算” 面对现场关于“如何量化并敲定风险预算”的追问，讲者坦言这个过程充满业务博弈，但必须由技术团队与产品负责人（Product Owners）从业务顶层紧密共创。“风险预算”这一概念正是由 Vanguard 内部的一位资深产品总监提出并确立的。 风险偏好（Risk Appetite）绝不能在整个企业内实行“一刀切”： - **高敏感场景（付费高净值客户群体）**：涉及实际资产运作与核心决策的高端客户服务，容错空间极小，风险预算必须极度严苛。 - **全网开放的浅层对话交互（公开 Chatbot）**：防御重点在于合规红线与品牌形象隔离，风险边界主要集中在防越狱与防恶意攻击。 - **内部员工赋能工具（Crew-facing Applications）**：例如供 Vanguard 内部 5,000 名员工使用的支持系统，由于使用者具备先验专业辨识能力，系统的容错与尝试空间相对宽松。 制定风险预算的核心锚点，在于拆解该场景“一旦发生故障时所能造成的实际最大伤害（Actual Harm）”，并基于潜在破坏力，理性分配和量化从设计端到投产端的容错上限。 ### [38:15](https://www.youtube.com/watch?v=Cq6CuBTwikw&t=2295s) 不作为的风险与企业 AI 的权衡之道 在讨论企业采纳 AI 系统的决策时，演讲者深刻剖析了风险管理中的常见误区——即许多组织往往过度关注技术落地可能带来的惩罚或失误，却忽略了停滞不前所蕴含的更大危机。演讲者指出，如果企业因为恐惧风险而选择“什么都不做，完全不让系统往前推进”，这种消极保守本身就是一种巨大的风险。因为在瞬息万变的技术与商业浪潮中，不采取任何行动可能会导致企业彻底丧失市场份额与核心竞争力。 企业需要树立一种全新的思维模式来量化与权衡这两类风险：“如果我承担了这项风险，我可能会面临某种程度的罚款或违规成本；但如果我不让系统向前迈进，我最终付出的代价可能远比那些罚款或风险带来的损失惨重得多。”在激烈的市场竞争中，被时代淘汰的隐形成本往往是致命的。现场对这一清醒而现实的认知报以心领神会的轻笑与赞同回应（"Yes"、"Correct"）。在结束这一关键洞见的解答后，演讲者向全场观众致意，祝愿大家在大会中享受充实的一天，整场演讲在全场热烈的掌声中圆满落幕。 ```mermaid flowchart TD accTitle: 推进与停滞的风险权衡分析 subgraph Action["积极推进 AI 系统"] RiskA["潜在风险：合规罚款、偶发错误"] GainA["商业回报：抢占市场先机、规模化赋能"] end subgraph Inaction["停滞不前 / 纯防御姿态"] RiskB["致命危机：错失市场、丧失核心竞争力"] CostB["最终代价：远超合规罚款与试错成本"] end Action -->|"通过评估与观测框架控制"| Control["可控风险 (Calculated Risk)"] Inaction -->|"不作为导致落后"| Loss["不可逆的商业出局"] ``` - **不作为的致命代价**：过度防守往往导致不可逆的商业失误，正如演讲者强调的，"you may lose the market completely by not taking any action"（你可能会因为毫无作为而彻底失去市场）。 - **风险思维的重塑**：合规与技术治理团队不仅要评估“上线 AI 带来的违规罚款或事故损失”，更要量化“阻碍系统演进导致被市场淘汰的隐性巨额成本”，在两者之间寻找动态平衡。 ## Notable quotes > [00:04](https://www.youtube.com/watch?v=Cq6CuBTwikw&t=4s) TAMING STOCHASTICITY — Piyush Saxena > [00:04](https://www.youtube.com/watch?v=Cq6CuBTwikw&t=4s) we don't fail — Piyush Saxena > [04:07](https://www.youtube.com/watch?v=Cq6CuBTwikw&t=247s) Garbage in, garbage out — Piyush Saxena > [12:05](https://www.youtube.com/watch?v=Cq6CuBTwikw&t=725s) 你们当中有谁制定了风险预算（Risk Budget）？ — Piyush Saxena > [14:55](https://www.youtube.com/watch?v=Cq6CuBTwikw&t=895s) 在每个人都在构建 GenAI 应用的今天，你的真正护城河是什么？SME 的标注数据就是护城河。 — Piyush Saxena > [18:05](https://www.youtube.com/watch?v=Cq6CuBTwikw&t=1085s) 世上根本没有一整个集团军的 SME 供我们招聘 — Piyush Saxena > [20:10](https://www.youtube.com/watch?v=Cq6CuBTwikw&t=1210s) 我们必须从过去传统的‘我们跑完了这些测试，系统绿灯，可以发布了’，转变为‘我们跑完了这些测试，系统具备置信度，但我们仍将在生产中持续运行这些测试，以确保应用在随机性扰动下始终符合预期’。 — Piyush Saxena > [21:58](https://www.youtube.com/watch?v=Cq6CuBTwikw&t=1318s) 在业务团队所在的地方与他们汇合（Meet the teams where they are） — Ritesh Shah > [27:42](https://www.youtube.com/watch?v=Cq6CuBTwikw&t=1662s) 生成式 AI 系统本质上是具有随机性的。我们能够驯服随机性，但无法彻底消除它。 — Ritesh Shah > [38:15](https://www.youtube.com/watch?v=Cq6CuBTwikw&t=2295s) you may lose the market completely by not taking any action — Piyush Saxena ## Mentioned Vanguard, Ai4, Piyush Saxena, Ritesh Shah, AWS, Amazon Bedrock, PagerDuty, Anthropic, Claude Opus, Cloud Code, Reddit, Chipotle, Python ## Video description Ritesh Shah - Chief Architect - AI/ML/GenAI, Vanguard Piyush Saxena - Director AIML, Advice, Wealth Management & AI CoE, Vanguard Your AI Shipped. Now What? Your AI passed the tests. The demo impressed stakeholders. The pilot delivered promising results. But what happens when real users arrive and your model behaves differently than expected? How do you know it is delivering consistent, reliable outcomes across different user groups and scenarios? And when performance drifts, safety issues emerge, or evaluations produce conflicting signals, do you have the visibility and infrastructure to understand why? Join us at AI4 to explore how leading enterprises are moving beyond manual evaluations, spreadsheets, and intuition to establish a rigorous, scalable approach for measuring and governing AI systems throughout their lifecycle. In this session, we will demonstrate how modern AI evaluation frameworks help organizations build confidence in production AI by combining persona-driven testing, customized evaluation methodologies, human oversight, and continuous monitoring into a single, integrated process. Learn how Persona Management enables teams to evaluate models against realistic user archetypes rather than relying on a single “average” user, uncovering biases, gaps, and performance variations before they impact customers. See how Bring Your Own Judges (BYOJ) allows organizations to define success on their own terms using domain-specific rubrics, custom scorers, fine-tuned classifiers, or external LLM judges that align with business objectives. Discover how a Jury of Evaluators creates stronger, more trustworthy assessments by aggregating multiple independent perspectives, reducing the risk of relying on a single evaluation metric or model. We will also showcase how Human-in-the-Loop workflows enable expert reviewers to focus on high-risk and low-confidence outcomes, while continuously improving automated evaluation mechanisms. In addition, attendees will learn how structured adversarial testing and red teaming can uncover vulnerabilities such as prompt injection attacks, jailbreak attempts, and edge-case failures before deploying solutions at scale. A key highlight of the session will be showcasing our enablement architecture, illustrating how evaluation, governance, and observability capabilities are embedded across the entire AI delivery lifecycle - from experimentation and development to deployment and production operations. Attendees will see how a unified architecture enables continuous validation, monitoring, and improvement of AI systems, providing actionable insights when model behavior changes and ensuring organizations maintain trust, compliance, and performance over time. Whether you are building AI-powered applications, deploying agents, or scaling generative AI across the enterprise, this session will provide practical strategies and architectural patterns to help you transform AI evaluation from an afterthought into a competitive advantage. Stop relying on guesswork. Start building AI systems you can measure, trust, and confidently scale. [Source](https://www.youtube.com/watch?v=Cq6CuBTwikw)

Source quotes

[0:04] Piyush Saxena: TAMING STOCHASTICITY [0:04] Piyush Saxena: we don't fail [4:07] Piyush Saxena: Garbage in, garbage out [12:05] Piyush Saxena: 你们当中有谁制定了风险预算（Risk Budget）？ [14:55] Piyush Saxena: 在每个人都在构建 GenAI 应用的今天，你的真正护城河是什么？SME 的标注数据就是护城河。 [18:05] Piyush Saxena: 世上根本没有一整个集团军的 SME 供我们招聘 [20:10] Piyush Saxena: 我们必须从过去传统的‘我们跑完了这些测试，系统绿灯，可以发布了’，转变为‘我们跑完了这些测试，系统具备置信度，但我们仍将在生产中持续运行这些测试，以确保应用在随机性扰动下始终符合预期’。 [21:58] Ritesh Shah: 在业务团队所在的地方与他们汇合（Meet the teams where they are） [27:42] Ritesh Shah: 生成式 AI 系统本质上是具有随机性的。我们能够驯服随机性，但无法彻底消除它。 [38:15] Piyush Saxena: you may lose the market completely by not taking any action

Source quotes

[0:04] Piyush Saxena: We can tame stochasticity. We cannot weed it out of a system that is generative in nature. [0:04] Piyush Saxena: cuz we don't fail. [0:04] Senior Leadership: How do I know that this application does not hallucinate? How is it safe? [3:09] Piyush Saxena: What is the capital of Peru? [18:05] Piyush Saxena: there is not an army of SMEs that is available for us to hire. [20:10] Ritesh Shah: we ran these tests, we are green [20:10] Ritesh Shah: we run these tests, we are confident, and we will continue to run these tests [27:50] Piyush Saxena: The capital of Peru can be Lima. It can be many other things based on context [27:50] Piyush Saxena: We can tame stochasticity. We cannot weed it out. [33:14] Ritesh Shah: doesn't need to know who won the World Series in '87. [38:15] Piyush Saxena: don't do anything and not let the system move forward

Original transcript

[0:04] Thank you all for making the time. Uh  my name is P Sakena. I'm the head of [0:08] uh AI center of excellence at Vanguard. And  today I'm joined by ret chief architect for [0:15] a IML and Gen AI. So thank you for making  the time and our session taming stoasticity [0:21] evaluations and observability at scale.  Um I believe there have been a quite a few [0:26] presentations around that and you'll see taming  stoasticity is like in all caps. The reason is [0:33] we we can tame stoasticity. We cannot weed it  out of a system that is generative in nature. [0:41] So we our talk overview we want to share our  journey like many of you uh we started building [0:47] large language models enabled applications just  about uh over an year ago and during that process [0:54] during our journey everything all our successes  and all of our learnings cuz we don't fail. Uh [1:01] we'd love to share that with you and how how it  all started. So we'll go over the broad pillars, [1:06] testing, production, observability, oversight,  self-healing, and then we'll have a deep dive [1:11] into what does the system actually look like  today and then share a few key takeaways. [1:19] So before we go into this is the 10,000  foot view. Before we go into this, [1:24] uh if I rewind about an year and a half ago,  we had our first gen applications that were [1:30] being produced across the 800 dev teams we have  at Vanguard. Uh Vanguard is a is a financial [1:36] institution. It's a highly regulated industry. Our  trusts, our ethos are absolutely quintessential. [1:43] Um so when so during one of the conversations with  our senior leadership, we were looking at one of [1:49] the applications and there was an executive that  asked us how do I know that this application does [1:55] not hallucinate? How is it safe? So dev teams get  together, we go back to work and uh we prompt it [2:02] as much as possible to block it from giving any  harmful answers. Again, we talking about large [2:07] language models. They're trained on the entirety  of the information that exists on the internet. [2:12] We come back for a second round and then uh the  narrative changes too. It's not very helpful. [2:18] Every time I ask a question, it says it cannot  answer. So that was the the genesis of our entire [2:24] journey. How do we find the appropriate balance  between what we call risk and harmfulness? So [2:32] looking at the the 10,000 foot view, what does the  system actually entail? This is very similar to [2:37] things that have happened for quite some time in  IML and uh software engineering. There's testing, [2:43] there is observability to make sure the things  that we believe the KPIs that we wanted to see [2:49] that that gave us the confidence pre-production  are observed in production. And then self-healing [2:54] it's it's a newer concept reinforcement learning  has been there quite for quite some time very well [2:59] studied very well used within traditional IML  so we all familiar with this we would have been [3:04] familiar with this like four years ago 5 years  ago maybe even 10 years ago but once we talk [3:09] about genative AI there is a nuance um because  generative models are stoastic in nature for [3:16] for the sake of some audience engagement uh fill  in the blank what is the capital of Peru Anyone? [3:26] I hear Lima across the board. Uh,  why is it Lima? Why is it not the [3:31] gastronomical capital of South America?  Are 20 other things Lima is famous for? [3:39] Welcome large language models. A lot of it depends  on context. What conversation did we have with the [3:44] model before this question was posed? So in gen  in general sense we expect the answers that an [3:51] application provide uh to be relatively close. So  if you ask the same question 10 times you're going [3:56] to get slight variations and that's where the  nuance comes in. Some variations are acceptable [4:01] and at some point in time those variations  become an improper answer. So let's start with [4:07] testing. What is how is testing more nuance when  we talk about large language models? traditional [4:16] traditional applications uh we would do user  acceptance testing where we would have maybe [4:21] um I don't know 20 to 50 to even 100 users  hating an application those applications were [4:26] bounded in their scope they were deterministic in  nature so UATS actually made a lot of sense even [4:32] today they do nothing replaces a good gut feel  but when we talk about models that can answer [4:36] absolutely anything that is challenging there's  there is a limit to how much can be tested with [4:43] humans testing the application and that is  where the concept of simulations becomes very [4:47] very important. We can run tens of thousands  of simulations with synthetic customers on [4:54] these applications. Garbage in garbage out still  remains true. That's the concept of personas. The [5:00] synthetic client that is going to interact with  an application should resemble or should interact [5:06] with the application like the actual customer  would. So that is a very key component. How do [5:12] we get personas? We all have uh data. We have  customer data. The personas that we that we [5:18] curate should be derivatives of that customer  data to best represent how an actual customer [5:24] would interact with the application. Um the other  part of simulations is as I sh as I talked about [5:31] we we care about risk and helpfulness. Those  are macrome metrics. There's many other metrics [5:36] underneath that. Uh for instance toxicity  could be a metric uh within risk. When we [5:42] do user acceptance testing, we can only do maybe  hundreds to maybe a thousand tests. That is not [5:49] enough to create tight confidence intervals  when you're trying to get an application uh [5:55] a gen application to your approval committee  wherever that might be. And I'm I'm confident [6:00] all of us have had that experience and that  fund. That is where those confidence intervals [6:06] become very important. The larger the larger the  volume we can make claims like actuaries do 99.5% [6:13] plus minus 0.05%. That is what actual confidence  looks like once it's quantified. So that's where [6:24] simulations become important. We can run tens of  thousands of them. Uh it also doesn't impede uh uh [6:29] development velocity. If we develop a feature,  we change a prompt, now it has to be retested. [6:34] we have to continually do UATS. That is where  these automated simulations become very powerful. [6:44] Next part, now we have tens of thousands  of uh customer interactions, synthetic [6:49] customers interacting with the application.  Who is actually going to sit and review those [6:54] uh interactions? Some of those interactions could  be just conversational. Some may actually tie into [7:02] um traceability. The gentleman before me was  talking about what does the telemetry for an [7:06] LLM application actually look like? What  are the APIs it's calling? What are the [7:09] tools it's accessing? So all of those become  a critical component and evaluating that [7:14] takes a good amount of time. That is where large  language models and more recently small language [7:20] models SLMs task oriented SLMs have become  really powerful. When we talk about judges [7:29] uh we think of it in two dimensions. principal  judges. The definition of toxicity, I'll go [7:34] back to toxicity again. That has to be uniform  for an enterprise. If we have a 100 dev teams, [7:40] we cannot have 100 definitions of toxicity. But  at the same time, those judges and evaluations [7:46] only go so far. Every application is going to have  some degree of application specific nuance. What [7:54] does that actually mean? Um let's say there's  an application that leverages natural language [7:59] again customer prompt and create SQL. There  are very specialized judges that are needed [8:04] to evaluate the efficiency of natural language  to SQL transformation. A lot of work has already [8:09] happened. The technology already exists but  that becomes an application specific judge [8:14] that that application builds out very very  specific to their use case. So those become [8:22] our two broad categories of judges. principal  judges and application uh specific judges. [8:29] Uh finally the the implicit component uh we  all talk about uh applica AI applications. [8:37] We want AI to be self-healing. We want  the applications to leverage customer [8:42] interaction data and SME feedback. A lot of our  organizations have subject matter experts that [8:48] have more information as a collective than  available anywhere else. No, for instance, [8:53] no frontier model is going to be able to represent  Vanguard methodology or Vanguard ethos. That is [8:59] very specific to Vanguard. Uh that is our special  sauce. So how do we make sure that that gets [9:06] reflected back into not only the agents of your  building but also the judges because judges need [9:11] calibration. Judges at the end of the day whether  they are small language models or prompt based [9:17] uh large language models are still ML models.  There is no there's no ML model that I'm aware [9:22] of. Maybe there are some exceptions that'll  give you 100% accuracy. There'll be 99.992939s, [9:29] but there's nothing that's 100% accurate. And over  a period of time, the definitions, how customers [9:35] engage with applications actually changes. So  having that judge feedback is absolutely critical [9:41] in the pipeline. And that's where our subject  matter experts uh come into play. We'll dig [9:46] deeper into that a level deeper once we go into  uh what does this mean environment look like. [9:55] So what so why do all of this? Uh  everything that we've talked about [9:59] is pre-production. We calibrate our judges  and the the x-axis has been removed on purp [10:06] uh on purpose. What we want to see is the judges  that we are curating no matter what kind they are. [10:11] We want them to see more and more aligned with  how the subject matter experts or human in the [10:16] loop actually is rating and interaction. And over  our experience with doing this for over an year, [10:22] we have seen a gradual improvement. All the  crux of that story is human in the loop is a [10:29] critical need for these systems not just  to improve but also for us to make sure [10:34] that they continue to perform as expected  once customer expectations change. how they [10:40] inter the interaction patterns change  and the applications themselves change. [10:47] Now comes our second uh topic production  observability. So everything that we do [10:52] pre-production is to gain confidence on the  application that we applications that we want [10:56] to deploy. When we go to production, we want  to measure the same KPIs. What was my toxicity [11:02] or completeness uh rating pre-production? That is  what we want to observe and measure in production. [11:09] the delta between them essentially defines what  drift is. So if you want to add a little bit of [11:16] color and talk a little bit about how we do it.  Yeah. Um adding color to what P said, right? Many [11:21] of you have built systems over many many years and  [snorts] when you put a system into production, [11:29] you may want to observe how it's performing  on different domains, different capabilities. [11:36] The way we think about it is hey every system  is generating events. Those events then get [11:43] converted into issues potentially if they don't  meet the requirements and or it becomes incidents [11:51] that we have to pay close attention to and  those incidents start creating alerts over [11:57] time notifying appropriate business unit that  your AI system is not functioning as desired. [12:05] So that's the process we use internally and those  incidents are also then rated by humans whether [12:13] the incident is okay for us whether it's bad or  whether it's extremely bad to take appropriate [12:20] action so that's the framework in which we think  about AI observability of events events generating [12:28] issues issues may get into incident and then  how do you manage those incidents Do you wish [12:35] you want to add any real examples? Absolutely.  And I have a takeaway. Show of hands. How many [12:40] of us are part of dev teams or lead dev  teams and are building applications? Well, [12:46] quite a substantial amount. I'd say the  majority. Does anyone have a risk budget? Today [12:56] we're talking about IML models. At the end  of the day, there has to be some residual [13:01] risk that a business has to uh accept. That's the  conversations we are in. And I see my peer we're [13:10] working on it on something uh together right  now. We need to define what is the appropriate [13:16] level of business risk and only then all  of these actually make sense otherwise our [13:21] thresholds are arbitrary. So as Ret mentioned  just to reiterate events should not raise any [13:29] red flags. Events can be anomalies because again  these are A IML models and generative in nature. [13:36] Once we look at a collection of events that is  when we have to decide what is our actual risk [13:40] threshold at what level are we comfortable in  escalating this to an actual incident. So act [13:47] not on events but on incidents and an incident  drive whether it's going to be a a mim call a [13:53] pager duty or is it something severe maybe  there's a dimension within evaluation again [13:59] I keep coming back to toxicity because it's the  easiest one maybe that is a dimension where the [14:04] where the threshold or the risk acceptance is  low to actually shut the system off but that [14:09] kill switch is absolutely necessary but all of  that is a derivative of what is your risk budget [14:15] Right. Can I add one thing? Um on the kill switch,  right? Kill switches are of different types. You [14:23] may want to trigger an alert and a human comes in  and responds. Another way to handle kill switches, [14:30] the request gets routed to a human, they  review and the response goes back. or it's [14:36] done through some models or other capabilities  where the request that was sent in is fulfilled [14:43] with an error message and your response is killed  based on the risk threshold that Push mentioned. [14:55] Now we come to one of my favorite parts. Uh I'm  a old school data scientist by training so this [15:01] part really excites me. So what does an expert  actually do? I'm also we also create I'm also [15:08] an enablement function. So we have to make sure  that this means that we have we are making the [15:13] best use of their time. So at the end of the  day as meek should only say two things yes I [15:17] agree yes I disagree. But how does that decision  translate into self-healing or even the amount of [15:26] oversight that is needed? That is what we're  going to go through. But the SME's decision [15:32] should either be to accept something they agree or  they disagree with it. But what does it actually [15:37] mean in the grand scheme of things? Our favorite  thing, the confusion matrix. Uh the bottom right, [15:43] those are true negatives. We love those.  The that's what we want. Heavily heavily [15:47] populated. That means the system is performing  as it should. The SME is in alignment with [15:52] the judges that we currently have. Uh when we  look at the other parts of it, true positive, [15:58] this would be an instance where a AI judge flags  something and it's me actually agrees with it. Uh [16:06] the off diagonal, the false negative and false  positive. Uh false negatives. This is this is [16:11] the real business risk places where something  gets something that should be flagged by an AI [16:17] uh judge uh sorry and yeah an AI judge is not  but dism actually flags it. That's the business [16:23] risk. That's the risk we that's the unseen and  uh the bottom left the false positive this is [16:29] business overhead. Uh how much how many times is  a judge being too conservative with this mis are [16:35] actually overturning the verdict. Anything jump  out about the false negatives and false positives [16:44] all things equal they're inversely  proportional. That's type one and type two [16:47] error. So it really leads to what can  we do with all of this information? [16:52] So second okay next one is more interesting. Uh  so when we have false negatives and we have false [17:01] positive this calls for judge calibration. That  means our AI judge isn't doing what it's supposed [17:05] to. It's either being overly conservative or  it is letting risk go unfl flagged. The true [17:12] positive piece the top right of the quadrant those  are opportunities for the actual agent to improve. [17:19] So this is how a yes or no from a sme can actually  be directed to actual impact. Self-healing doesn't [17:26] happen automatically. So there are mechanisms  in place for it. We'll discuss a little [17:30] uh briefly. Uh if we have judges that are  prompt based, there's prompt augmentation, [17:35] prompt optimization. Uh if we have judges that  are small language models, then there is RLHF. [17:41] But all of this me data is what actually drives  that. So when we think about we all everyone [17:47] is building geni applications. is what is our  actual mode that's me data is the it serves as [17:52] a benchmark data set anytime any changes are  made to an application or a judge the larger [17:58] the data set gets the more confidence we can  have in how the application actually performs [18:05] one more interesting uh so this was uh  observability uh the other interesting component [18:11] here is oversight a lot of businesses are  requiring a substantial amount of human oversight [18:19] over these interactions. How do you do that when  you have 100,000 interactions in a day? How many [18:24] SME is is that going to take to review those  100,000 interactions? And even if they're saying [18:29] yes, I agree, I disagree. That was rhetorical.  The answer is too many. So, how do we actually [18:38] move ahead? like how can we how can we ensure  that our risk stays within our risk budget but [18:45] yet we can turn down that oversight dial. Um  if you look at if if you look at the matrix [18:52] uh the confusion matrix right now the left uh  column the true positives and false positives [19:00] those will always be flagged because that's  what a judge does it flags things incorrectly [19:04] or correctly. The right hand part is the part that  can be sampled from. So those are the parts that [19:10] the judge is actually saying everything is okay.  In some cases it actually is true negatives and [19:15] uh in false negatives that is a business incurred  risk. So this is where the same data can be [19:20] transformed into actuarial sciences where we can  actually make claims such as given a risk budget [19:27] the risk that we are assuming with 100% oversight  is the same risk that we would assume if we if our [19:33] oversight was 20% 30%. All that 20 30% all of  that depends on those KPIs. What are what are [19:40] those numbers in a confusion matrix? So that has  been our journey and how and it's a critical step [19:48] in order for these applications to scale based  on the interactions. We cannot there is not an [19:53] army of SME that is available for us to hire.  So it is absolutely critical for us to find and [20:01] decide on things like what is the risk budget and  how do we actually turn that oversight dial down. [20:10] So we talked about uh judges calibrating the  judges two ways. So the false positives and [20:15] false negatives that is a judge getting things  wrong. What do we actually do with it? Um Aam's [20:21] razor if we can get a judge with a prompt that  is the path we go. In that case the prompt can [20:27] actually be optimized um augmented. There is  automated mechanisms for it that will actually [20:32] optimize the prompt for you, rerun it against  the benchmark and give you new metrics for those [20:37] uh judges. However, more recently we find  ourselves in uh places where the prompt size [20:44] for those judges is substantially [clears  throat] sorry substantially large. That is [20:50] where small language models task oriented  small language models come into play. [20:54] And the same data with thumbs up, thumbs down,  agree, disagree that feeds the prompt based [21:00] optimization also feeds reinforcement learning  with human feedback pipelines for those SLMs. [21:10] And that's that's the crux of it. Self-healing  is always like implicit. We want uh testing. We [21:16] want observability and self-healing  usually like falls to the wayside. [21:20] It is absolutely critical especially with  genai applications and applications that [21:26] leverage stochastic models and it has to  be driven on both sides. The application [21:31] itself the a whether we using agents or multiple  agents that has to improve but judges have to be [21:38] continuously calibrated as well. So slight  change in paradigm for we ran these tests, [21:44] we are good to go. We are green versus we  run these tests, we are confident we'll [21:50] continue to run these tests to make sure  that our application performs as it should. [21:58] Now for the fun part. Okay, let's have an  implementation type. Now that you have learned [22:03] different phases that we take any AI system  through, [snorts] we approach this problem [22:10] with multiple angles. We wanted to meet the teams  that are building these solutions where they are. [22:17] So we started with implementing a solution that is  like a Python library that any developer can use [22:25] to do any early testing and then we graduated  it from hey developer testing it to the next [22:33] level of can we build a custom application that  all these product teams can use and leverage to [22:41] run their evaluations before [snorts] they go to  production. P if you can pull up the next slide. [22:47] Um this is a very very highlevel diagram of what  that application looks like. Um it's built and [22:55] running on AWS. It's a custom application. It  has a web component to it where the user or [23:03] the team comes to a web interface and performs  certain actions behind the scenes. It provides [23:11] the capability of you to set up your judges.  It provides you set ability to set up personas [23:19] and it also provides you ability to run your  evaluations. In the centerpiece you will see [23:26] different capabilities that are run as part  of this product that we have. Um just like I [23:33] said it can run simulations on the ground  truth that you provide. Apart from that, [23:39] the system will generate you a report across many  dimensions and tell you how your AI system or [23:46] agent is performing. This is all orchestrated for  the team. Team onboards their agent or AI system [23:54] into this application or a product [snorts] and  then they go and schedule uh their evaluation runs [24:01] as they update their application. The report  that is generated is finally used to make a [24:08] decision whether this application is supposed  to go to production or not based on the risk [24:14] of that application and the residual risk. Um a  decision is made um using that. Anything else you [24:22] want to add? One more thing. Uh you mentioned we  started with a Python library over the year over [24:28] the years like it's been a couple of years. Uh the  customer base has changed. Our original customer [24:33] base was our developers who are actually building  the applications. But when we think about the [24:39] risk ecosystem, there are developers, there are  product owners, there is divisional risk, there [24:44] is enterprise risk. All all of these groups cannot  have differing definitions of how an application [24:50] performs. So the application has to serve all of  these groups. That's why as shared uh there is [24:56] CI/CD, there's CLIs, there is agents, there is a  UI all of them with the same backend with the same [25:02] def core definition of what is a persona, what  is a simulation, what does an outcome or sorry, [25:08] what does an output look like. So that's where  a lot of the diversification and what the system [25:14] design drives for us really come into life. Now  you may be wondering where are we going from here, [25:19] right? Do we always have all these personas always  use a UI and come to a UI when we are modernizing [25:28] ourselves with tools like cloud code etc. And  many of you are software developers, right? You [25:35] don't do this kind of testing through a UI. You  typically have a CI/CD pipeline that does this [25:42] kind of testing before you go to production. P if  you can go to the next slide. This is the this is [25:49] what our vision is. This is what we are building  towards. We are building a series of agents in [25:56] our which will be used by our CI/CD pipeline.  Each agent has a different function. Some agents [26:04] are going to go do the setup for you, set up your  personas, set up your judges, etc. But then there [26:12] are agents that are going to run in parallel  to perform evaluations of different types. You [26:18] would have a business evaluation performed,  adversarial evaluation. We have a category [26:24] internally called trusted AI and regulatory and  compliance testing. All of this is being planned [26:32] actively to be part of our CI/CD pipeline and at  the end of the pipeline the results are taken put [26:40] in a system where they are evaluated and if  the system is approved then the next phase of [26:47] deploying it to production would would continue  on. And after you deploy to production, the job [26:53] is not done yet. Like we showcased, evaluation  happens before production. After production, [27:00] the CI/CD pipeline will set up your observability.  Import your judges so that your risk metrics are [27:07] the same pre-prod and postp prod. And it would do  other observability needs like setting up alerts, [27:13] triggers, etc. The vision is hey how do we build  an agentic solution that is used by CI/CD but [27:22] also developers from their desktop can use cloud  code and interact with this system before going [27:29] to production. So that's the whole overall  arching vision of ACICD integration that we [27:36] are working towards. The journey is not complete.  We are working through many of these components [27:42] actively but that's where we are p we can take to  the next right what are our key takeaways right so [27:50] you said all of this uh what are the key things  that we really think that is something that you [27:57] can take to your teams your leaders uh to really  make meaningful progress there first we have [28:03] to redefine risk the definition of risk has to  change I I talked about the concept of risk budget [28:09] Unless there is a risk budget, you cannot  really work your way backwards and really [28:14] identify what is the degree of you'll see the word  accuracy thrown out a lot or simplicity. What is [28:20] the accuracy that is needed for an application to  drive business value? All of that stems from what [28:25] is a risk budget and risk in this scenario again  we work with large language models. The capital [28:31] of Peru can be Lima. It can be many other things  based on context. We have to look at statistically [28:38] significant numbers. We can we can do u we can  we cannot live with small scale UATs anymore. [28:45] That's where what is actual risk has to be  redefined and really a lot of it is not new. [28:51] It's a derivative from actural sciences.  Uh the other key piece is all of this can [28:56] happen at design time. None of this has to be an  afterthought. What is the risk budget? How should [29:01] an application perform? How will self-healing  happen? All of these can be decisions that are [29:06] made at design time before whatever approvals are  needed for an application to go into production. [29:13] I would I would also add right think of risk  from two perspectives. What is the inherent [29:18] risk of your system? And then after applying  all these techniques, what is your residual [29:24] risk that you are willing to either live with  or have enough safeguards built into your system [29:34] and then finally uh self-healing reinforcement.  Um, geni systems I will go back to geni systems [29:42] are stoastic. We can tame stochasticity. We cannot  weed it out. If you want to weed stoastic city out [29:48] then we should use traditional IML models or  other deterministic systems that actually in [29:52] many situations perform very very well. Um,  we have to move from the mindset of looking [29:58] at judge scores as unit tests. Judge scores  will evolve. They will evolve based on how [30:03] the application changes. Even if the application  stays the same, how customers interact with an [30:08] application that can change over a period of time.  Reddit is full of many many memes. People going to [30:13] Chipotle or other uh chat bots and having it do  math or create Python code. So once that happens, [30:20] then you start seeing differing application  uh usage patterns. Um and one of the final key [30:27] takeaways like when we talk about these  judges uh whether they are prompt based [30:30] judges whether these are small language models  these cannot be treated as external these are [30:36] the collateral a dev team actually owns manages  and they have to go through the through the same [30:42] uh software development life cycle like  everything else does I'll add one more [30:46] thing right because you mentioned software  development life cycle I think you also [30:51] have to adapt to this new way of working than  just indicating to your leaders that a typical [31:00] software development life cycle is how we build  AI. You have to adapt to the risk mindset which [31:06] you may have not done as part of your typical  software development. That's very important [31:14] and I think that wraps us up. Uh we're open to any [31:17] questions. Any questions?  Uh we are here. [applause] [31:29] Cool. We'll wait on the side  if there are any. Go ahead. [31:45] Sorry. [31:50] How do you do that? [32:01] Absolutely. That's the first line of defense.  Every so all of the interactions actually have [32:07] LLM judges running. We have an observability  platform. So we know what is the volume of [32:12] things that are getting flagged and that  is the first line of defense. Now as as [32:16] I shared that can have false positives  and false negatives. So there's a second [32:20] line of defense with subject matter  experts that review everything that [32:23] is flagged and that essentially provides the  self-healing the calibration that data goes [32:28] back and feeds into either prompt optimization  augmentation uh pipelines or chef pipelines. [32:37] So [32:43] concept [32:54] in cost of application [32:59] might be [33:03] looked at. [33:14] I'll give you the what we are actually doing and  then Resh can tell us what steps we're taking of [33:20] tokconomics and optimizing all of that. But that's  a [clears throat] great point. We cannot just we [33:24] don't randomly pick a judge. We pick a specific  we try to stay open source on that because of [33:31] latency concerns as well. And then those judges  are actually benchmarked. So a judge uh an LLM [33:37] at the end of the day for a specific task will  have an accuracy. It will have a sensitivity [33:42] and specificity. So those are the metrics we look  at from an operational perspective. How well does [33:48] this just perform? So that's one thing. The other  thing is uh we don't need a judge for every single [33:53] dimension that we want to monitor. Multiple input  multiple output is better than single input single [34:00] output. Outputs can learn from correlations  from each other. Very old concept in math. [34:04] So if you have a judge that is checking toxicity  and you have the token length which in most cases [34:10] we do you can actually pair up other similar  concepts as toxicity with the same. So in essence [34:16] instead of running three evaluations you're  running a single one. But how do we actually [34:21] what what steps do we take with our hyperscalers  can answer from a hyperscaler perspective we use [34:28] multiple options here right? First we started  with we are AWS native. So right now we are using [34:34] bedrock service. Second we are looking into how  to use some of these open-source models directly [34:42] hosted by Vanguard rather than hosting it on  on bedrock or using a bedrock because that is a [34:48] higher price compared to hosting it ourselves. And  third, some of the evaluations we are looking into [34:54] building our own models which would be slightly  more cheaper than hosting an open-source. So [35:01] those are like three levers and then we are also  looking at other hyperscalers. How do we scale? [35:07] Because it's not just about the cost getting  so many evaluations done in a timely manner is [35:13] more important also. So how do we run evaluations  on not not just one cloud provider but multiple [35:20] cloud providers. So that's also being investigated  in parallel. Hope that helped with a question on [35:27] cost's point. [cough] We [clears throat] do have  um custom models at Vanguard that are built by our [35:36] uh enterprise uh AI and research team. Those  are maybe like 50 60 billion parameter models. [35:42] Hopefully I'm not bastardizing it. compared  to an opus model that might be trillions of [35:46] parameters or an open-source uh model like llama  which is still a 500 uh billion parameter model. [35:53] So being task specific is act is a is actually  saves a lot of tokens because the model that [36:01] has to detect toxicity doesn't need to  know who won the world series in ' 87. [36:07] So basically custom model [36:12] custom model hosting it on your own instances and  then third is multi- multiple hyperscalers because [36:20] you can do that. Fourth option maybe that can come  up in future is you can have a special contract [36:27] with the model provider example anthropic that  may give you more pricing leverage than directly [36:35] using it without a special contract. So that  could be a lever that could be used in future. [36:44] Any other questions? [37:01] will not get me in trouble. Uh [clears  throat] it's a it's a discussion we have [37:08] with our stakeholders in business like people  who are who are our product owners. We work very [37:13] closely with them. That is where the discussion  starts. And I'm not saying like we we've done it [37:19] all through but that at least it's acknowledged  that yes we need to have this concept of a risk [37:24] budget. It was actually coined by uh one of the  senior product uh owners that that we work with [37:34] in the process again uh joke jokes apart you  cannot have the same risk appetite for every [37:40] single use case. There are some applications  uh that you may be they may be paid segments [37:46] of customers. The risk appetite there will be  different than a chatbot that's available to [37:50] everyone versus applications that are internal  crewf facing at a company like Vanguard. That [37:57] could still be like 5,000 uh Vanguard crew members  actually using the application. So it's nuanced. [38:03] It's based on the actual use uh case and what is  the actual harm that could come out when things [38:09] go wrong. And also if you think of risk budget  right you may want to think about what if you [38:15] don't do anything and not let the system move  forward. There's a risk in doing that because [38:21] you may lose the market completely by not taking  any action. So assigning that kind of mindset of [38:29] hey if I take this risk I may have this kind  of penalties but if I don't let the system [38:34] move forward I may be paying lot more than the  fines or um other aspects of taking that risk. [38:47] Yes. [snorts] [38:50] Correct. [38:56] Cool. If enjoy your day. Thank you all. [applause]