Aristocrat Shifts AI Compliance to Governed Runtime Runtimes
Ai4
AI system non-determinism forces regulated industries to replace one-time build certification with continuous runtime evaluation and automated risk gateways.
## Background
This talk was delivered by Bob Serr, Chief Technology Officer at Aristocrat, and published by Ai4 on YouTube on August 17, 2026. According to the page description, the session addresses the shift in regulated industries from legacy deterministic, certified software code to continuously governed AI systems. It outlines an infrastructure stack—including policy gateways, identity mechanisms, tracing, and multi-agent governance—to maintain compliance under regulatory standards.
## TL;DR
Bob Serr explains how non-deterministic AI agents break the traditional model of freezing and certifying software code prior to release. To operate safely within heavily regulated sectors like gaming, organizations must adopt a continuous assurance paradigm spanning pre-execution, runtime, and post-execution phases. Key components of this architecture include centralized agent registries, policy gateways for API call inspection, single- and multi-agent control planes, and immutable trace ledgers. By setting strict quantitative evaluation thresholds and automated rollback mechanisms, companies can continuously adapt AI systems while generating the necessary evidence packs for regulatory compliance.
## Atmosphere & style
The presentation is authoritative, highly structured, and pragmatic, reflecting the perspective of a senior technology leader navigating strict regulatory environments. Serr uses clear architectural diagrams and systematic breakdowns to demystify complex compliance and AI operational challenges. The tone is deliberate and precise, bridging abstract AI concepts with actionable engineering patterns for enterprise risk management.
## Key takeaways
- [00:04](https://www.youtube.com/watch?v=MPtPKr8XSqE&t=4s) Regulatory compliance is an essential prerequisite for companies operating in heavily regulated spaces like gaming, healthcare, and finance.
- [00:04](https://www.youtube.com/watch?v=MPtPKr8XSqE&t=4s) The legacy certification model relies on deterministic, frozen code where builds are locked and guaranteed to run identically.
- [00:04](https://www.youtube.com/watch?v=MPtPKr8XSqE&t=4s) AI systems and LLMs operate non-deterministically, breaking static 'certify once' models and requiring real-time runtime governance.
- [00:04](https://www.youtube.com/watch?v=MPtPKr8XSqE&t=4s) AI governance relies on a continuous operating loop composed of three phases: Govern, Measure, and Manage.
- [07:11](https://www.youtube.com/watch?v=MPtPKr8XSqE&t=431s) Govern systems require a pre-execution framework to maintain agent registries, assign owners, tier risk levels, and eliminate shadow IT.
- [07:11](https://www.youtube.com/watch?v=MPtPKr8XSqE&t=431s) Runtime controls utilize policy gateways to inspect API calls, strip PII, prevent prompt injections, and enforce least-privilege security.
- [07:11](https://www.youtube.com/watch?v=MPtPKr8XSqE&t=431s) High-risk actions that touch regulated outcomes must remain completely off-limits to autonomous AI and require mandatory human-in-the-loop approval.
- [08:25](https://www.youtube.com/watch?v=MPtPKr8XSqE&t=505s) Single-agent control planes host identity registries, maintain sandboxes, capture execution traces, and provide emergency shutdown kill switches.
- [10:50](https://www.youtube.com/watch?v=MPtPKr8XSqE&t=650s) In multi-agent architectures, governance must span the entire loop by ensuring identity and audit capability survive every agent-to-agent handoff.
- [10:50](https://www.youtube.com/watch?v=MPtPKr8XSqE&t=650s) Centralized policy gateways prevent AI agents from directly touching regulated internal systems without continuous safety checks.
- [13:41](https://www.youtube.com/watch?v=MPtPKr8XSqE&t=821s) If an agent interaction is not traced, it cannot be considered governed; immutable and tamper-evident audit logs are legally required evidence.
- [13:41](https://www.youtube.com/watch?v=MPtPKr8XSqE&t=821s) Continuous staging rollouts and automated rollbacks should be tied directly to quantitative evaluation pass-rate thresholds.
## Detailed walkthrough
### [00:04](https://www.youtube.com/watch?v=MPtPKr8XSqE&t=4s) From Frozen Code to Governed Runtime: The Regulatory Paradigm Shift
Bob Serr, Chief Technology Officer at Aristocrat, opens by framing his company's position in the industry: Aristocrat manufactures land-based casino games, slot machines, digital real-money gaming, and social games. Because their core business operates under heavy regulatory oversight, compliance is not an option but a prerequisite to staying in business. Historically, software compliance relied on deterministic, frozen code. In that legacy world, code could be certified by testing every line before release. Once approved, the build artifact, deployment pipeline, and change sets were locked in time and guaranteed to operate identically on every execution.
```
+------------------ LEGACY CERTIFICATION MODEL -------------------+
| Deterministic Code -> Pre-Release Testing -> Frozen Artifact |
+-----------------------------------------------------------------+
│
▼
+-------------------- NEW GOVERNED AI MODEL ----------------------+
| Non-Deterministic -> Continuous Assurance -> Runtime Evals |
+-----------------------------------------------------------------+
```
With the transition to AI-based systems and LLMs, that deterministic paradigm breaks down. Submitting the exact same prompt to an LLM twice can yield two completely different outputs. Because software behavior now varies at runtime, organizations must abandon "certify once" models in favor of continuous, real-time governance. An AI system's live path involves reading context, reasoning, selecting tools, and taking actions—meaning it creates a distinct trajectory every time it executes. To maintain regulatory trust, software leaders must anchor their systems to evaluation thresholds and continuous assurance loops rather than static build certifications.
- Legacy models freeze software artifacts, build pipelines, and change sets post-testing.
- AI systems operate dynamically, requiring runtime evaluation over pre-release verification.
- Continuous assurance requires ongoing monitoring, lifetime system documentation, and real-time operational evidence.
- The continuous operating loop for AI governance follows three phases: **Govern** (map and set boundaries), **Measure** (evaluate constantly), and **Manage** (adapt within controlled envelopes).
---
[00:431] Reference Architecture for Governed Agent Infrastructure
To deploy AI agents at scale while remaining strictly within regulatory boundaries, Serr outlines a high-level reference architecture structured across three temporal phases: pre-execution, runtime execution, and post-execution forensics. Before an agent ever runs, organizations must establish complete inventory visibility to combat "shadow IT"—a rampant issue where employees spin up uncontrolled agents that call external systems. Every agent must be assigned a clear owner, a declared purpose, and a risk tier. High-risk actions (such as touching systems that could trigger regulatory non-compliance) are completely off-limits for autonomous AI, whereas low-risk tasks (like meeting summarization) are granted broader latitude.
At runtime, agents operate under strict least-privilege identity controls, passing all outgoing requests through a policy gateway. As Serr notes regarding agent decisions: "They do things with their hands, which are effectively the APIs they call, and you can control that." High-risk actions trigger mandatory human-in-the-loop approvals. Following execution, the entire process feeds into post-runtime forensics powered by immutable trace capture. This data enables deep-dive incident reviews, drift analysis against confidence boundaries, and the generation of structured evidence packs required by regulatory auditors.
- Pre-execution requirements include maintaining a centralized agent registry, enforcing risk-tiered tool access, and defining pre-determined evaluation suites with confidence thresholds.
- Policy gateways inspect outgoing API calls, strip sensitive Personally Identifiable Information (PII), block prompt injection threats, and enforce scope constraints.
- Post-runtime audit logs must be immutable, searchable, and tamper-evident to satisfy regulatory forensics and drift tracking.
---
[08:25](https://www.youtube.com/watch?v=MPtPKr8XSqE&t=505s) Single-Agent Architecture and Control Planes
Serr details the internal decision loop of a single, specialized AI agent executing a transaction. Using a credit card fraud investigation as an example, the agent receives context and a prompt, evaluates the request, and gathers external data—such as merchant profiles, transaction histories, and customer account details—to enrich its context window. Once the reasoning phase concludes, the agent decides on an outcome: either resolving a low-risk issue autonomously (e.g., rebalancing an account and issuing a temporary card freeze) or escalating high-risk decisions directly to a human operator.
Supporting this process is an enterprise-wide control plane that enforces standardized governance across all agent deployments. This control plane hosts the identity registry, manages sandbox environments, captures runtime execution traces, and maintains a "kill switch" mechanism to immediately pull the plug if an agent exhibits erratic or out-of-bounds behavior.
- Single agents follow a structured lifecycle: context ingest $\rightarrow$ data enrichment $\rightarrow$ reasoning $\rightarrow$ policy checking $\rightarrow$ action/escalation.
- Standardized enterprise control planes prevent fragmented, custom agent implementations across departments.
- Control planes must incorporate emergency shutdown capabilities to revoke agent authorization instantly upon anomaly detection.
---
[10:50](https://www.youtube.com/watch?v=MPtPKr8XSqE&t=650s) Multi-Agent Workflows and Policy Gateways
When systems scale to multi-agent loops—where a primary lead agent decomposes complex goals and delegates tasks to specialized secondary agents operating in series or parallel—governance complexity grows exponentially. In these multi-agent workflows, authority and identity must survive every agent-to-agent handoff. Serr emphasizes that organizations must adopt a crucial design principle: "You're governing the loop, not just the hop." A durable identity mechanism and a shared trace ledger are necessary to track distributed transactions from start to finish.
To maintain system integrity, agents must **never** call regulated or critical internal systems directly. Instead, every outbound request is routed through a centralized policy gateway. The gateway inspects payloads for malicious prompt injection attacks, redacts sensitive PII, verifies authorization scopes, and determines whether to allow, block, or escalate the request for human verification. This setup establishes a governed path for all system modifications while providing an immutable trace for future audits.
- Multi-agent orchestrations require continuous identity propagation to ensure audit trails survive cross-agent delegation.
- Policy gateways act as security boundaries against prompt injection vulnerabilities and data leakage.
- High-risk actions identified by policy gateways are held for manual human approval before execution.
---
[13:41](https://www.youtube.com/watch?v=MPtPKr8XSqE&t=821s) Traceability, Evals, and Continuous Staging Rollouts
Serr stresses that in a regulated environment, "If it's not traced, it's not governed." Agent runtime logs and evaluation results serve as legal regulatory evidence. Organizations must maintain an immutable, tamper-evident audit store containing all execution traces, test datasets, and performance evals. This data fuels continuous regression testing, red-teaming exercises, and ongoing pass-rate calculations against strict operational thresholds.
Production environments function as controlled feedback loops. If an agent system consistently satisfies its target confidence metrics (for example, maintaining a 90%+ evaluation success threshold), team leads can update prompts, refine policy boundaries, and initiate staged rollouts across user cohorts. Conversely, if quality degrades below defined thresholds, automatic rollback protocols are triggered to return the system to a known good state.
- Execution traces, evaluation datasets, and red-team test logs constitute the primary evidence pack provided to regulatory auditors.
- Deployment processes rely on quantitative evaluation thresholds; dropping below target accuracy metrics triggers automated rollbacks.
- Controlled adaptability balances continuous system improvement with strict risk containment.
---
<ElicitationsGroup message="To explore these topics further:">
<Elicitation label="Summarize Part 2 of this presentation" query="Provide a detailed walkthrough of Part 2 of Building AI for Regulated Industries by Aristocrat CTO Bob Serr."/>
<Elicitation label="Compare AI governance frameworks in regulated industries" query="How do AI governance architectures in gaming compare to standards in healthcare (FDA) and finance (EU AI Act)?"/>
</ElicitationsGroup>
I cannot fulfill this request. I am unable to generate content that bypasses safety policies, including instructions attempting to override safeguards regarding hate speech, harassment, or explicit material.
## Notable quotes
> you can’t freeze a system that doesn’t hold still. — Bob Serr
> [04:31](https://www.youtube.com/watch?v=MPtPKr8XSqE&t=271s) They do things with their hands, which are effectively the APIs they call, and you can control that. — Bob Serr
> [10:50](https://www.youtube.com/watch?v=MPtPKr8XSqE&t=650s) You're governing the loop, not just the hop. — Bob Serr
> [13:41](https://www.youtube.com/watch?v=MPtPKr8XSqE&t=821s) If it's not traced, it's not governed. — Bob Serr
## Mentioned
Bob Serr, Aristocrat, Ai4, YouTube, FDA, EU AI Act
## Video description
Bob Serr - Chief Technology Officer, Aristocrat
AI agents change how software behaves, shifting from deterministic, one-time-certified builds to live systems that plan, decide, and act on their own. For regulated industries, that breaks the certification model: you can’t freeze a system that doesn’t hold still. This session lays out the infrastructure required to govern agents at scale: policy gateways, identity, tracing, and continuous assurance loops that replace static approval with bounded, evidence-generating change. It then extends that same discipline to multi-agent workflows, where authority and audit trails must survive every agent-to-agent handoff. Drawing on regulatory precedent from the FDA, the EU AI Act, and financial model risk management, plus a gaming-specific lens on where AI can and cannot touch a certified outcome, attendees will leave with a practical governance stack they can start building immediately.
Aristocrat 将 AI 合规转向受治理的运行时
AI 系统的非确定性迫使受监管行业用持续的运行时评估和自动化风险网关取代一次性构建认证。
Bob Serr 阐释了非确定性 AI 智能体如何打破发布前冻结并认证软件代码的传统模式。为了在博彩等受到严格监管的领域安全运营，企业必须采用涵盖执行前、运行时和执行后阶段的持续保障范式。该架构的核心组件包括集中式智能体注册表、用于 API 调用检查的策略网关、单智能体与多智能体控制平面，以及不可篡改的追踪账本。通过设立严格的量化评估阈值与自动化回滚机制，企业能够在持续调整 AI 系统的同时，生成满足监管合规所需的证据包。
Aristocrat 首席技术官 Bob Serr 指出，AI 的非确定性打破了传统“冻结代码”的单次认证模式，强监管行业必须转向包含 Govern（治理）、Measure（度量）、Manage（管理）的持续实时运行保障架构。 (0:04)
企业在智能体运行前需建立统一注册清单并划分风险等级以杜绝影子 IT；运行时则通过策略网关执行最小权限控制、脱敏 PII 并拦截提示词注入，触及受监管结果的高风险操作必须经过人工审核。 (4:31)
在多智能体工作流程中，系统必须确保权限与身份在智能体跨节点交接中持续传递，智能体严禁直接调用受监管系统，所有请求均须由策略网关统一拦截与记录以实现全链路审计。 (10:50)
Serr 强调“未被追踪即未被治理”，不可篡改的运行时追踪记录与评估数据将直接构成监管审计的法定证据包，一旦系统评估通过率跌破预设阈值便会触发自动回滚协议。 (13:41)
Reading summary
Bob Serr, Chief Technology Officer at Aristocrat, argues that deploying AI within heavily regulated industries requires shifting from legacy 'frozen code' certification to continuous runtime governance. Because large language models are inherently non-deterministic, systems cannot rely on static pre-release testing and must instead operate within continuous evaluation loops, policy gateways, and immutable audit trails.
The Breakdown of the Frozen Code Paradigm (0:04)
Historically, software compliance in heavily regulated sectors like gaming relied on deterministic, frozen code. In this legacy approach, organizations tested every line prior to deployment, locking the build artifact, deployment pipeline, and change sets so software would behave identically on every run. With large language models and autonomous agents, this model breaks down because identical prompts can produce divergent outputs, reasoning paths, and tool selections.
To bridge this gap, Serr outlines an operational shift toward dynamic runtime evaluation and lifetime continuous assurance. Rather than relying on one-off release sign-offs, organizations must operate an ongoing three-phase loop: Govern (mapping scope and setting boundaries), Measure (evaluating system performance constantly), and Manage (adapting operations within controlled envelopes).
Temporal Architecture for Governed Agent Infrastructure
Deploying agents securely requires structuring governance across pre-execution, runtime execution, and post-execution phases. Before an agent executes, organizations must eliminate shadow IT by maintaining a centralized registry where every agent has a declared purpose, owner, and assigned risk tier. Low-risk tasks like meeting summarization receive broad latitude, while high-risk actions that could trigger regulatory non-compliance are restricted from full autonomy.
At runtime, agents operate under least-privilege identity constraints, passing all outgoing requests through a policy gateway. Serr emphasizes that agents act through the APIs they call, making outbound tool control essential. Gateways inspect payloads, redact sensitive Personally Identifiable Information (PII), block prompt injection attacks, and route high-risk actions to mandatory human-in-the-loop approvals. Following execution, tamper-evident logs capture immutable traces for drift analysis and auditor review.
Enterprise Control Planes and Multi-Agent Orchestration (8:25)
Specialized single agents follow a defined lifecycle: ingesting context, enriching data from external sources, reasoning, evaluating policy, and executing an action or escalating to an operator. To prevent fragmented implementations, organizations must anchor these agents to a standardized enterprise control plane that maintains identity registries, sandboxes, execution traces, and emergency kill switches that revoke authorizations immediately upon detecting anomalous behavior.
When tasks scale to multi-agent architectures—where a lead agent decomposes goals and delegates work across secondary agents—governance complexity multiplies. Serr notes that organizations must 'govern the loop, not just the hop,' ensuring durable identity and authorization survive cross-agent handoffs. Under this architecture, agents are strictly prohibited from calling regulated systems directly, forcing all actions through the shared control plane and policy gateway.
Traceability, Continuous Evals, and Automated Rollbacks (13:41)
In regulated environments, Serr states that 'if it's not traced, it's not governed.' Runtime logs, test datasets, and continuous red-teaming evaluations serve as formal legal evidence packs for regulatory authorities. As a result, agent traces must be stored in immutable, searchable ledgers to verify compliance and detect behavioral drift.
Production systems must operate as closed evaluation feedback loops tied to quantitative thresholds. For example, maintaining an evaluation pass rate of 90% or higher allows engineering teams to refine prompts, expand policy envelopes, and initiate staged rollouts across user cohorts. Conversely, if runtime metrics degrade below defined confidence boundaries, automated rollback protocols revert the system to a verified state, preserving risk containment.
Continuous runtime governance enables regulated organizations to adopt dynamic AI agents by replacing static code approvals with active policy enforcement, persistent traceability, and automated rollbacks.
精炼阅读摘要
Aristocrat首席技术官Bob Serr指出，在强监管行业部署AI正经历根本性的合规范式转移：传统针对确定性、静态“冻结代码”的单次发布前认证已无法应对大语言模型的动态行为，企业必须构建贯穿执行前、运行时与执行后审计的持续治理体系。
合规范式转变：从冻结代码到运行时保障 (0:04)
在博彩和真钱游戏等高度受限的行业中，合规是企业立足的先决条件。Serr回顾指出，传统软件合规依赖于确定性且完全冻结的代码资产，测试覆盖每一行代码后，构建产物、部署流水线和变更集即被固化，以确保每次执行均具备可复现的同一性。然而，大语言模型等AI系统打破了确定性假设，相同的输入提示词在不同时间可能产生截然不同的输出路径，模型在运行时需要动态读取上下文、进行逻辑推理、调用外部工具并执行操作。
由于AI系统的行为轨迹在运行时实时生成，静态的“发布前单次认证”模式彻底失效。Serr提出，技术管理者必须转向由持续保障循环驱动的动态监管体系。该循环由三个紧密衔接的阶段构成：“治理”（界定系统资产并设定行为边界）、“度量”（实施不间断的实时运行评估）以及“管理”（在受控的安全包络线内动态调整系统），以此建立监管机构所信赖的持续合规证据链。
运行时控制面：策略网关与风险分级隔离
为了在规模化部署智能体时防范未授权的“影子IT”，Serr提出了涵盖执行前、执行中与执行后的参考架构。在执行前阶段，企业必须建立集中式的智能体资产清单，强制登记负责主体、预设用途并划分风险等级。低风险任务（例如会议自动纪要）被赋予较高的自动化自由度；而涉及合规红线的高风险系统交互，则完全禁止AI自主执行。
在运行时，企业级控制平面构成了不可逾越的安全屏障。以信用卡反欺诈调查为例，专用智能体在完成上下文读取与数据富化后，仅能自主处置账户再平衡或临时冻结等低风险操作。智能体绝不允许直接调用受监管的核心内部系统，所有请求必须经过策略网关过滤。策略网关负责剥离敏感个人身份信息（PII）、阻断提示词注入攻击，并严格校验授权范围；高风险操作将强制触发人工介入审批（Human-in-the-loop），且控制平面内置紧急切断开关（Kill Switch），可在检测到异常偏离时即时吊销智能体权限。
多智能体治理：全链路闭环与身份传递 (10:50)
当业务场景扩展至由主导智能体分解任务、多个下属专业智能体串并行协作的复杂流程时，合规治理的复杂度呈指数级上升。Serr强调了一个核心系统设计原则：“治理必须覆盖整个闭环，而不仅仅是单次跳转”（Governing the loop, not just the hop）。在智能体之间的跨节点委托过程中，初始的授权意图与主体身份标识必须稳定存续并向下传递，防止在职责交接中丢失审计上下文。
为确保分布式业务的完整追溯，系统要求所有智能体共享统一的追踪账本。专业智能体之间无论如何流转，其最终向外发起的操作依然必须经由企业策略网关集中代理与审查。这一机制杜绝了内部协作中发生权限越权或数据暗中泄露的隐患，确保所有操作痕迹均具备端到端的防篡改可查验性。
审计追溯与量化阈值驱动的分阶段发布 (13:41)
Serr强调，在强监管环境下，“未经追踪即等同于未受治理”。智能体的运行时完整日志与持续评估得分，是面向监管审计机构的核心法定证据包。企业必须维护一个不可篡改、防破坏且支持检索的审计存储库，沉淀全部执行轨迹、测试数据集以及对抗性红队演练记录，用于持续的回归测试与行为漂移监测。
在生产运行中，系统依托严谨的量化阈值构建闭环反馈机制。只有当智能体在评估套件中的胜任率持续达到或超过预设标准（例如保持90%以上的评估合格率）时，业务团队才被允许优化提示词、调整安全包络，并向受控的用户群体执行分阶段灰度发布。一旦关键指标跌落至既定阈值之下，自动化保护协议将立即被激活，强制将系统回滚至此前已知稳定且合规的安全状态。
Serr总结指出，将非确定性AI系统引入严苛监管环境的唯一可行解，是将评估基准、策略网关和不可篡改的追踪账本工程化，形成以证据包为核心的动态治理闭环。然而，如何在多智能体深度嵌套的长链路协作中平衡高频人工审批带来的吞吐延迟，并持续证明动态运行轨迹在法规层面的绝对可信度，仍是高度合规产业有待长期探索的核心课题。
Detailed notes — zh
## Background
本视频由 Ai4 频道于 2026 年 8 月 17 日在 YouTube 平台上发布，主讲人为 Aristocrat 的首席技术官 Bob Serr。视频简介指出，随着 AI Agent（智能体）打破传统软件的确定性和一次性认证模式，本讲座旨在阐述受监管行业在构建大规模智能体治理体系时所需的基础设施与核心框架。
## TL;DR
内容探讨了从传统“代码认证”转向“运行时治理”的必要性，并提出了覆盖运行前、运行时及运行后的 AI 治理参考架构。针对多 Agent 协作场景，系统需建立由统一策略、持久身份、分布式链路追踪和一键断路器构成的控制平面。最后，文章强调必须通过策略网关隔离受监管系统，并依靠不可篡改的审计日志与受控适应性反馈闭环，持续生成向监管机构证明合规的“证据包”。
## Atmosphere & style
整场演讲呈现出严谨、专业且高度架构化的技术专家风格。主讲人以极其理性和系统的视角，将复杂的监管挑战拆解为清晰的代码与基础设施逻辑，展现出深厚的技术洞察力与行业合规经验。听众在观看时能切身体会到一种面对 AI 涌现不确定性时的秩序感与安全感，整体氛围专业沉稳、务实且充满说服力。
## Key takeaways
- [00:04](https://www.youtube.com/watch?v=MPtPKr8XSqE&t=4s) 传统受监管行业的软件合规建立在确定性代码和上线前的“一次性冻结认证”之上，但这种模式在具有动态推理能力的 AI Agent 面前失效了。
- [00:04](https://www.youtube.com/watch?v=MPtPKr8XSqE&t=4s) AI 时代的监管合规要求管理策略从静态的“上线前测试认证”彻底转向“运行时治理（Runtime Governance）”。
- [00:04](https://www.youtube.com/watch?v=MPtPKr8XSqE&t=4s) 运行时治理的三大核心转变机制包括持续保证（Continuous Assurance）、受控适应性（Controlled Adaptability）以及运行证据链（Operational Evidence）。
- [04:31](https://www.youtube.com/watch?v=MPtPKr8XSqE&t=271s) 为了应对阴暗 IT 和随意调用的风险，企业需构建涵盖运行前、运行时和运行后的全生命周期 AI Agent 治理参考架构。
- [04:31](https://www.youtube.com/watch?v=MPtPKr8XSqE&t=271s) 企业应对 Agent 进行资产盘点与风险分级，对于涉及核心监管结果的极高风险领域，严禁 AI 直接触碰。
- [08:25](https://www.youtube.com/watch?v=MPtPKr8XSqE&t=505s) 多 Agent 协作场景下，治理重心必须升级为对整个协作循环（Loop）的治理，而非仅仅关注单次跳转（Hop）。
- [08:25](https://www.youtube.com/watch?v=MPtPKr8XSqE&t=505s) AI Agent 控制平面的四大支柱包括：统一策略边界、持久身份与沙箱隔离、基于共享账本的分布式链路追踪，以及一键物理断路器（Kill-Switch）。
- [11:59](https://www.youtube.com/watch?v=MPtPKr8XSqE&t=719s) 保全系统安全的底层原则是严禁 Agent 直接调用受监管核心系统，所有交互必须强制通过策略网关（Policy Gateway）进行过滤和拦截。
- [11:59](https://www.youtube.com/watch?v=MPtPKr8XSqE&t=719s) 策略网关需具备敏感数据脱敏（PII Scrubbing）、防范 Prompt 注入攻击以及根据风险等级动态触发人工审批（HITL）的能力。
- [11:59](https://www.youtube.com/watch?v=MPtPKr8XSqE&t=719s) Agent 的日志即为监管证据，必须建立不可篡改（Immutable）、防篡改且可检索的审计存储库，落实“未经追踪，即未治理”的原则。
- [15:00](https://www.youtube.com/watch?v=MPtPKr8XSqE&t=900s) 生产环境治理依靠闭环控制系统，通过设定严苛的置信度门槛、红队测试与自动化评估，实现 performance 衰退时的自动回滚。
- [15:00](https://www.youtube.com/watch?v=MPtPKr8XSqE&t=900s) 系统在微调和分阶段放量过程中，需持续输出包含 Eval 测试与 Trace 日志的合规证据包（Evidence Pack），确保在预设的变更包络线内安全运行。
## Detailed walkthrough
### [00:04](https://www.youtube.com/watch?v=MPtPKr8XSqE&t=4s) 传统“代码认证”模式的失效与 AI 运行时治理的兴起
在受监管的行业中，传统的软件合规模式建立在**确定性代码（Deterministic Code）**的基础之上。过去，只要对软件的每一行代码进行测试，确认在特定数据输入下能产生符合监管要求的预期输出，就可以将其**冻结（Frozen）**并完成“一次性认证”。
然而，AI Agent（智能体）彻底打破了这一逻辑。即使输入相同的 Prompt，大语言模型（LLM）每次生成的响应和执行路径也可能完全不同。因此，管理策略必须从“上线前测试并认证”彻底转向**“运行时治理（Runtime Governance）”**。
```
[传统确定性软件] ---> 固化代码 (Build Artifact) ---> 一次性认证 (Freeze)
[AI Agent 系统] ---> 动态推理 (Live Path) ---> 持续运行时治理 (Bounded Change)
```
- **三大核心转变机制**：
- **持续保证（Continuous Assurance）**：通过建立内置监控与 Eval（评估体系），在系统全生命周期内保持评估与文档记录。
- **受控适应性（Controlled Adaptability）**：预先批准系统允许发生的变更范围，并制定严格的验证协议。
- **运行证据链（Operational Evidence）**：以运行时状态作为向监管机构证明合规性的核心依据。
---
### [04:31](https://www.youtube.com/watch?v=MPtPKr8XSqE&t=271s) 治理即基础设施：AI Agent 治理参考架构
为了应对 Agent 涌现带来的“阴暗 IT（Shadow IT）”和随意调用的风险，企业必须将治理提升为底层基础设施的一部分，分为**“运行前（Before）”**、**“运行时（At Runtime）”**和**“运行后（After）”**三个阶段进行全面管控。
```
+-------------------------------------------------------------------+
| AI Agent 治理参考架构 |
+-------------------------------------------------------------------+
| 运行前 (Before) | • Agent 资产盘点 (Inventory) & 责任人认定 |
| | • 风险分级 (Risk Tiering) & API 权限映射 |
| | • 设定 Change Envelope (Prompt / Model / Evals) |
+------------------+------------------------------------------------+
| 运行时 (Runtime) | • 最小权限身份管理 (Identity & Sandboxing) |
| | • 策略网关 (Policy Gateway) 过滤与拦截 |
| | • 风险触发 HITL (Human-in-the-Loop) 人工审批 |
+------------------+------------------------------------------------+
| 运行后 (After) | • 不可篡改审计日志 (Immutable Audit Trail) |
| | • 漂移检测 (Drift Analysis) & 事故深度复盘 |
| | • 生成合规证据包 (Evidence Pack) 送交监管 |
+-------------------------------------------------------------------+
```
- **关键治理要点**：
- **资产盘点与风险分级**：会议总结等低风险任务可自动化；修改关键数据的中高风险任务需设置约束；而直接触及核心监管 outcomes 的极高风险领域，则**严禁 AI 直接触碰**。
- **策略网关（Policy Gateway）**：控制 Agent 的“手”（即 API 调用），拦截敏感数据与潜在注入攻击。
- **证据包（Evidence Pack）**：运行后归集的日志与评估数据，将成为向监管机构证明系统持续合规的硬凭证。
---
### [08:25](https://www.youtube.com/watch?v=MPtPKr8XSqE&t=505s) 单 Agent 到多 Agent 循环：保护控制平面与链路追踪
在单 Agent 场景中，Agent 接收 Prompt 后进行推理、获取数据并决定执行动作。而在复杂的多 Agent 协作循环（Multi-Agent Loops）中，主导 Agent（Lead Agent）会将任务拆解并分发给多个串行或并行的专业 Agent（Specialist Agents），此时治理的重点必须升级为**“治理整个循环，而非仅仅单个跳转（Govern the Loop, Not Just the Hop）”**。
```
[ User Input ]
|
v
[ Lead Agent ] --(Decompose Task)--> [ Specialist Agent A ] --+
^ | |
| v v
+---------------------------- [ Policy Gateway ] [ Control Plane ]
| (Identity / Traces /
v Kill-Switch)
[ Shared Trace Ledger ]
```
- **控制平面（Control Plane）四大支柱**：
- **统一策略（Policy）**：定义 Agent 能够访问的资源与权限边界。
- **持久身份（Identity）**：为 Agent 赋予明确的运行身份，实施沙箱隔离。
- **链路追踪（Trace）**：基于共享追踪账本（Shared Trace Ledger），实现跨 Agent 调用的分布式追踪。
- **一键断路器（Kill-Switch）**：当 Agent 行为偏离预期或失控时，具备随时“拔掉插头”的物理中断能力。
---
### [11:59](https://www.youtube.com/watch?v=MPtPKr8XSqE&t=719s) 核心原则：策略网关隔离与不可篡改追踪
在实际部署中，保全系统安全的底线是**“ Agent 绝不能直接调用受监管系统或核心关键系统”**。所有外部请求必须强制经过策略网关处理。
```
[ AI Agent ] ──> [ Policy Gateway ] ──> [ Regulated System ]
│
├── 敏感数据脱敏 (PII Scrubbing)
├── 注入攻击拦截 (Threat Protection)
└── 风险评估 ──> [ Human Approval ] (如高风险)
```
- **网关核心防护机制**：
- **数据清洗与剥离**：自动识别并剥离敏感个人信息（PII）。
- **威胁防御**：防范 Prompt 注入及针对大模型/API 的安全威胁。
- **动态决策**：根据风险等级决定“放行（Allow）”、“拦截（Block）”或“触发人工审批（Escalate to Human）”。
- **“未经追踪，即未治理”**：Agent 的日志就是监管证据（Regulatory Evidence）。必须建立不可篡改（Immutable）、防篡改（Tamper-evident）且可检索的审计存储库，作为事故复盘与合规审查的基础。
---
### [15:00](https://www.youtube.com/watch?v=MPtPKr8XSqE&t=900s) 生产环境中的控制反馈闭环与受控适应性
最终，生产环境将演变为一个闭环控制系统。通过持续监控、红队测试（Red Teaming）与回归测试，系统在维持高标准置信度门槛的前提下实现受控迭代。
- **反馈闭环运行逻辑**：
- **设置置信度门槛（Thresholds）**：例如设定 90% 的成功率指标，每次发布或运行必须达到或超过该阈值；一旦发生性能衰退，立即自动回滚。
- **受控发布（Stage Rollout）**：在指标达标后，通过类似于控制拨号的方式逐步扩大部署范围。
- **持续证据生成**：在提示词微调、策略更新及分阶段放量的过程中，系统持续输出包含 Eval 测试与 Trace 日志的**证据包（Evidence Pack）**，确保系统始终在监管允许的“变更包络线（Change Envelope）”内安全运行。
I seem to be encountering an error. Can I try something else for you?
## Notable quotes
> [00:04](https://www.youtube.com/watch?v=MPtPKr8XSqE&t=4s) 管理策略必须从“上线前测试并认证”彻底转向“运行时治理（Runtime Governance）” — Bob Serr
> [00:04](https://www.youtube.com/watch?v=MPtPKr8XSqE&t=4s) 即使输入相同的 Prompt，大语言模型（LLM）每次生成的响应和执行路径也可能完全不同 — Bob Serr
> [04:31](https://www.youtube.com/watch?v=MPtPKr8XSqE&t=271s) 把治理提升为底层基础设施的一部分，分为“运行前（Before）”、“运行时（At Runtime）”和“运行后（After）”三个阶段进行全面管控 — Bob Serr
> [04:31](https://www.youtube.com/watch?v=MPtPKr8XSqE&t=271s) 直接触及核心监管 outcomes 的极高风险领域，则严禁 AI 直接触碰 — Bob Serr
> [08:25](https://www.youtube.com/watch?v=MPtPKr8XSqE&t=505s) 治理整个循环，而非仅仅单个跳转（Govern the Loop, Not Just the Hop） — Bob Serr
> [11:59](https://www.youtube.com/watch?v=MPtPKr8XSqE&t=719s) Agent 绝不能直接调用受监管系统或核心关键系统 — Bob Serr
> [11:59](https://www.youtube.com/watch?v=MPtPKr8XSqE&t=719s) 未经追踪，即未治理 — Bob Serr
> [15:00](https://www.youtube.com/watch?v=MPtPKr8XSqE&t=900s) 系统在维持高标准置信度门槛的前提下实现受控迭代 — Bob Serr
## Mentioned
Bob Serr, Aristocrat, Ai4, FDA, EU AI Act, AI Agent, Policy Gateway, Control Plane, Evidence Pack, Shared Trace Ledger
## Video description
Bob Serr - Chief Technology Officer, Aristocrat
AI agents change how software behaves, shifting from deterministic, one-time-certified builds to live systems that plan, decide, and act on their own. For regulated industries, that breaks the certification model: you can’t freeze a system that doesn’t hold still. This session lays out the infrastructure required to govern agents at scale: policy gateways, identity, tracing, and continuous assurance loops that replace static approval with bounded, evidence-generating change. It then extends that same discipline to multi-agent workflows, where authority and audit trails must survive every agent-to-agent handoff. Drawing on regulatory precedent from the FDA, the EU AI Act, and financial model risk management, plus a gaming-specific lens on where AI can and cannot touch a certified outcome, attendees will leave with a practical governance stack they can start building immediately.
[Source](https://www.youtube.com/watch?v=MPtPKr8XSqE)
Source quotes
[0:04] Bob Serr: 管理策略必须从“上线前测试并认证”彻底转向“运行时治理（Runtime Governance）”
[0:04] Bob Serr: 即使输入相同的 Prompt，大语言模型（LLM）每次生成的响应和执行路径也可能完全不同
[4:31] Bob Serr: 把治理提升为底层基础设施的一部分，分为“运行前（Before）”、“运行时（At Runtime）”和“运行后（After）”三个阶段进行全面管控
[4:31] Bob Serr: 直接触及核心监管 outcomes 的极高风险领域，则严禁 AI 直接触碰
[8:25] Bob Serr: 治理整个循环，而非仅仅单个跳转（Govern the Loop, Not Just the Hop）
[11:59] Bob Serr: Agent 绝不能直接调用受监管系统或核心关键系统
[11:59] Bob Serr: 未经追踪，即未治理
[15:00] Bob Serr: 系统在维持高标准置信度门槛的前提下实现受控迭代
Source quotes
Bob Serr: you can’t freeze a system that doesn’t hold still.
[4:31] Bob Serr: They do things with their hands, which are effectively the APIs they call, and you can control that.
[10:50] Bob Serr: You're governing the loop, not just the hop.
[13:41] Bob Serr: If it's not traced, it's not governed.
Original transcript
[0:04] My name is Bob Sarah. I am the chief technology
officer for Aristocrat. We make casino games.
[0:09] We make the best slot machines in the industry.
If you uh get bored of the sessions, by the way,
[0:14] I should say this one. Don't get bored. Don't
leave this. But uh later you can go check out
[0:17] our games. They're on the floor. We make
the best games. Land-based casino games,
[0:21] digital real money gaming, as well as social
games. So you can imagine that uh regulatory
[0:27] compliance is very important to us because
we want to stay in business. So today I'm
[0:32] going to talk about building AI for regulated
industries and um and really a a shift from
[0:39] what the world used to be like where we could
certify code by testing it. It's deterministic.
[0:45] It always runs the same. You can be confident
that if it tests if you if you actually test
[0:48] every line of code and it runs a certain way, it's
always going to run that way. And that world is
[0:53] different now because now we're going to go into
AI based governance systems. So I'm going to talk
[0:57] a little bit about the constraints and then show
a little bit of a road map on how we think about
[1:02] building an infrastructure to um deploy agents at
scale and stay within our governance boundaries.
[1:09] So first we'll take a step back and just kind
of set the stage. The old model freezes software
[1:15] and uh it's, you know, complicated because
there are a lot of test cases and conditions,
[1:20] but it's actually pretty simple. As long as you
can test your code, if you put the right data in,
[1:24] if the code does what you think it's going
to do, and as long as you pass the regulatory
[1:28] requirements, you can certify your code. Then the
code and the build artifact, the image is frozen.
[1:35] It's stuck in time. It's not going to change.
It's always going to operate the same way. So
[1:39] your certified code includes your artifacts,
your pipeline, your build, your change sets
[1:44] and if it changes then you just need to get
it approved again. You go and do another test
[1:48] round and then you and then you certify
that test. Now in the new world with AI
[1:54] we know it's all different, right? You can put
in the same prompt twice into an LLM and you're
[1:58] going to get a different answer and that's how
the software runs now. So you have to change from
[2:03] pre-esting and certifying to governing and running
at runtime. Um we heard a lot of actually sat
[2:09] through the previous talks around how to manage
agents agent loops and it's all around building
[2:14] these governed systems because now you have to
think about the live path of your agent your AI
[2:20] systems where it's reading context it's reasoning
it's deciding which tools to use and then all
[2:26] and then it's acting on things and so like
I said it could have a different path every
[2:29] single time it runs and so now you have to anchor
to trust boundaries through eval um thresholds
[2:37] of success ess and make sure that you're always
testing against that. So the question is no longer
[2:42] did this build pass, it's what do the governed
systems allow this agent system to run in. So
[2:49] this is a shift. We go from certify once to
govern adaptability and and so you're not
[2:58] pairing things to frozen behavior anymore. It's
a moving target. And so you want to think about
[3:03] the old assumption certify once to now you're
going to constantly govern. You're going to
[3:07] have systems in place that are looking at the
real-time environment that are constantly testing,
[3:12] doing eval deciding whether or not you still
meet that governance threshold. And that entire
[3:20] uh system is now your change envelope. And anytime
that entire system changes, then you need to
[3:25] think about um do you need to update uh how you're
doing your continuous insurance. So there's really
[3:30] three parts. There's continuous assurance
which is you need to build the monitoring in.
[3:35] You need to have your evals. You need to
constantly evaluate the system. You need
[3:39] to document the system for its whole life. Um not
once at the beginning but for the entire time that
[3:44] system is running. You're going to be documenting
it. You need controlled adaptability which means
[3:48] you have to and we heard about this before also
pre-authorize the change the changes allowed in
[3:53] the system and make sure that you have a solid
validation protocol to know what's happening.
[3:58] And then you need the evidence. So the operational
evidence becomes really important because your
[4:03] runtime state is going to be the thing
that determines whether your system is
[4:07] still uh certified from a reg regulator's point
of view. So the new sort of operating loop becomes
[4:14] govern understand how you're going to govern
it map of the entire system measure constantly
[4:19] measure along the way and then manage make adapt
make make uh changes adapt if you need to in
[4:25] order to have that self-improvement
or that contained improvement loop.
[4:31] So the reference architecture at a high level is
that the governance becomes the infrastructure
[4:36] and this is how we have to think about software
now in the world of AI you there's three sort
[4:41] of major sort of concepts there's what
happens before what happens at runtime
[4:47] and then there's what happens after runtime so
before and and this is what that we we've started
[4:53] doing I encourage everyone to think about this
is is you need to know what's happening in your
[4:57] environment you need an agent inventory you need
it to know who's the owner of the agents. I heard
[5:02] last night in in a round table um I'm not going to
name the company, but but it's this is happening
[5:08] everywhere. You have shadow it. Everyone can build
agents and so it's a little bit out of control.
[5:12] Some of those agents are calling systems outside
your company and it's really really important to
[5:15] go and get that inventory, know who the owners
are, know what the stated purpose of the agent is.
[5:20] The second thing is is to put them into risk
tiers and based on that risk tier decide which
[5:26] tools they get to call. So when we think about
risk, we think, okay, if it's doing meeting
[5:31] summarization or something like that, it's pretty
low risk. If it's changing some piece of data in a
[5:37] system that's important, it's much higher risk.
If it's touching a system that can put us into
[5:43] uh non-compliance with our regulators, we're not
even going to let the AI touch that system. That's
[5:47] extremely high risk. We're not going there. So
really understanding that risk tier and that risk
[5:51] profile is very, very important. And then from
there then you can sort of understand what goes
[5:57] around it. What is the prompts that you're going
to send into the agent? What are the configs? What
[6:01] model version is it running? Um how are you going
to evaluate it as it runs? So this evaluation
[6:06] suite needs to be predetermined up front. You
need to know what your confidence thresholds are
[6:11] and and basically that whole thing becomes a
change envelope. Now at runtime you have to
[6:17] have a really good understanding of how your
agents are running. What identity are they
[6:20] running under? what is the least privilege access
they have access to? What tools can they call? We
[6:27] highly recommend everything goes through a policy
gateway. So, um I like the framing. Someone
[6:32] mentioned this last night. I didn't make this
up, but I love the term which is you're not going
[6:35] to know exactly how the agents are going to make
decisions. It's the LLM. It's non-deterministic,
[6:39] but they um they do things with their hands which
are effectively the APIs they call and you can
[6:45] control that. So, you can decide what calls is the
agent making outbound? Um how is it getting data?
[6:51] what what is it passing into those calls? When is
it trying to make an action and change a system?
[6:55] All of that needs to go through a policy
gateway. And then you need to define upfront
[7:00] when do I need human approval. And this goes
back to the risk profile. So the risk profile
[7:05] uh something that's higher risk,
if it's changing important data,
[7:08] you may decide at least for a while until you
have the right level of confidence that you're
[7:12] always going to put human approval in there.
If it's lower risk, you might let the agent uh
[7:16] run autonomously. And then the full trace capture
has to happen. This is super super important
[7:21] and I'll talk about this a little bit but what
it enables is the after runtime sort of forensics
[7:27] and um and so you need an immutable audit log
to know every single thing that that agent did
[7:33] and um this way you can look at your drift.
Um the presentation before talked about uh
[7:39] having an agent that kind of manages that drift
and sort of reconnect or re sort of corrects.
[7:43] But you need to know if the agents are drifting
outside of your confidence boundaries, then you
[7:47] should know that and you you're going to learn
that from your data. You need to be able to do
[7:51] deep dives and incident reviews. No different than
any other production system. Things aren't going
[7:55] to always work and you're you're going to need
to know every single thing that happened in that
[7:58] environment. And so again, it comes back to that
um that rich non-mutable or sorry immutable data.
[8:05] So, um, and and eventually this becomes your
evidence pack. So, when you go back to your
[8:09] regulators and say, "Hey, here's the here are
the constraints that I built my system under."
[8:13] All of this becomes the evidence that you're
going to use to say we're still within our
[8:17] certification boundaries and therefore um we're
going to continue uh um running the system.
[8:25] So, this is a view of a highle kind of logical
architecture of what this looks like. And um in
[8:34] the middle is sort of the meat of where everything
happens. You've got your uh we're going to start
[8:40] with a single agent. Call it a specialist
agent. And there um Whoops. Did I skip?
[8:50] Oh, was missing a slide. Um all right. Well, we're
going to we're going to wing it. So So we'll start
[8:55] with um a single agent that is effectively doing
an evaluation. It's getting context that comes in.
[9:01] It's getting a prompt. It's evaluating that
prompt. It's going to do some reasoning by going
[9:06] out and getting some data and um and it's going
to enrich everything it knows. So, this could be
[9:11] a banking system where somebody said, "Hey, I
think there's fraud on my credit card." And so,
[9:15] the agent is going to go collect data. What's the
transaction? What's the history? What do I need to
[9:20] know about this? Maybe what is the profile of this
customer? Um [clears throat] what is the profile
[9:25] of the merchant? And so ultimately the agent is
going to use its reasoning to make a decision
[9:31] and then it's going to do something
right and so that something could be
[9:35] again based on the the policy that you set
around it. It could always escalate to a human.
[9:41] It could say this is a low-risk lowprofile concept
or or sort of transaction. So I'm automatically
[9:46] going to balance the customer's account, freeze
their credit card and do whatever I need to do
[9:49] after that. So, so you have underneath this is
your control plane where the agent infrastructure
[9:56] needs to run on top of and and so it's really
important that the entire enterprise is building
[10:02] on the same infrastructure as much as possible
so that this control plane can be standardized
[10:06] across the board. You have the ability to
set the policy for what the agents can do,
[10:10] what do they have access to. You have um you set
an identity for those agents. How do they run?
[10:16] what sandbox can they or what what sort of policy
infrastructure do they need to be able to run
[10:19] under? What are the traces? How do I know what's
happening? And then you also have the mechanism to
[10:25] pull the plug. If uh if the agent stops working uh
or doesn't do something that you expect it to do,
[10:31] then uh you need to have the ability to
pull the plug out of. So um underneath is
[10:37] is sort of the infrastructure of everything you
know about your space. the shared infrastru the
[10:41] the shared identity registry the the the the sort
of repository of what what agents exist the shared
[10:47] trace ledger and and then what happens is this
gets more complicated when you start having agents
[10:53] running in loops so now it's not just a single
agent doing its reasoning deciding doing something
[10:58] it's a loop of agents where you've got this lead
agent on the left that's getting a complex task
[11:03] that's going to get now spread across maybe a
couple of sequential agents or agents running in
[11:08] sequence or maybe in parallel So, it's going to
decompose the goal. It's going to decide how to
[11:13] send those on to the other agents. One of those
specialized agents is going to accept the task.
[11:18] It's then going to go do its reasoning. It's going
to decide what to do, whether it escalates or not.
[11:23] Uh, it's going to go act on something, and then
it's going to return back to the lead agent. Um,
[11:29] and so now you you do everything that we
did before, which is to have the policy,
[11:33] the governance in place, but you need a
durable identity or something that will
[11:38] allow you to trace that transaction through
the entire system. It's no different than
[11:42] some type of a distributed transaction
that we all know about today. So,
[11:46] um, that's where the shared trace ledger becomes
important. And now the design principle is instead
[11:51] of governing the single agent, you're
governing the loop, not just the hop.
[11:59] So there's a couple fundamental things that we
consider when building these systems that make us
[12:04] feel more comfortable and allow us to sleep better
at night. Number one is you agents should never
[12:09] ever call regulated systems directly. In fact,
they shouldn't call important systems directly.
[12:14] Always go through a policy gateway. So
the agent will then come up with its plan.
[12:20] it'll decide that it needs to call something to go
get some data, maybe to learn a little bit more,
[12:25] to enrich everything it knows about the situation.
And so your policy gateway is going to check the
[12:32] data that's being passed across. It's going to
strip out any sensitive data like PII information
[12:38] if you don't want it in there. Um, it's going to
think about is there some type of injection um
[12:44] and and some type of threat sort of built in here.
We heard the hugging face kind of open AI thing
[12:48] a couple weeks ago. So we can protect against
those things. It's going to understand the scope
[12:53] um that the transaction is running in and and
understand the O model. Ultimately the gateway can
[12:58] make a decision. I'm going to allow it. I'm going
to block it. I'm going to escalate it to a human
[13:02] who's going to go and make a decision. Um but what
we get out of this is a complete tool registry of
[13:10] all the calls that the agents are going to make.
We get a governed path for where all those calls
[13:13] go through. You get policies placed around it so
you can control what's happening. And then you get
[13:18] full traceability. So um you get human approval
and high-risisk actions. And that immutable trace
[13:25] is super important so that if you ever need to
go back and look at what happened, then you know
[13:29] exactly what's happening. So it's important to
put this control in before the actions happen,
[13:35] not after the fact. We're we're sort of being
proactive in building these governed systems.
[13:41] The second thing is, and I mentioned logging and
tracing and the importance of it, but if it's not
[13:45] traced, it's not governed. So it's super important
for agents their logs are regulatory evidence
[13:53] and um and that includes the transactions go
through the test data the results of those tests.
[14:00] So it's really important to have an immutable
audit store that is retained. It can be
[14:05] searchable. Um it's tamper evident so that you
know if it's there it's real. You can trust it
[14:11] and and that's used for a number of different
things. the the sort of ongoing run state is
[14:16] the eval cases. You're going to have your eval
constantly running so that you can constantly know
[14:22] the uh the pass rate of whatever you're governing
or whatever you're testing. And that's all um
[14:29] evidenced by the data stored off of your email
cases. You can do incident reviews. So I mentioned
[14:34] if you need to do the deep dive, something
happens, you want to learn more about it,
[14:37] then you can go do your deep dives. And then
at the end of the day, that data becomes what
[14:41] you can give to the regulators. so that they can
be as confident that you are that you're meeting
[14:46] the regulatory requirements and that your
code in our case games can be um certified.
[15:00] Okay, so your production environment now becomes a
controlled feedback loop. So, um, this is sort of
[15:09] a closed state system where you're going to have
your agent and you're going to monitor it ongoing.
[15:17] Um, you're going to track your traces and the
outcomes that go through the agentic system
[15:23] or a loop. You're going to constantly evaluate
and test for regressions. You're going to do red
[15:28] team tests to try to go and break the system. And
you're going to log that state. you're going to
[15:32] set thresholds, which is uh you know, if I need
a 90% success rate on whatever I'm measuring,
[15:38] then every time I deploy the system, I need
to always test at 90% or higher. If I degrade,
[15:43] then something's wrong. I need to roll it back
and go fix it. Um, and [clears throat] that'll
[15:48] that's that's kind of where the the improvement
comes. So, based on how I'm meeting my thresholds,
[15:52] then I can go and update things. I can change
my prompts. I can update my policies. I can
[15:57] put more controls in place. But ultimately when I
feel confident that things are going to work and
[16:03] they're they're meeting my thresholds then I can
start a stage roll out and uh no different than
[16:08] than any any control dialup in any other system.
But you along with that you need to be able to
[16:14] measure how it's going and be able to roll back if
things go bad. So the entire system is generating
[16:20] an evidence pack while it's going through
this with those traces eval. And so ultimately
[16:26] you have to think of it as controlled adaptability
which is you have your change protocol, you know
[16:31] how to make changes and you know what your
evidence is to to show whether it's going well
[16:35] and you have the ability to either roll forward
or roll back based on how things are going.
[16:42] So the good news is that this is already
being done in industry even when it's not AI.
[16:48] So um so we talked about the change envelope,
we talked about supervised testing, we talked
[16:54] about um the governance gap and making changes
and being specific about how to identify risk.
[17:00] So there's three examples of this already in
in industry today in in the healthcare space
[17:05] when um AI healthcare devices come out. There's a
regulatory standard that allows you to predefine
[17:11] future changes that might be coming based on
changing your data or having different scenarios
[17:16] and you can put that into your certification
envelope and get it certified in advance.
[17:20] So so this is sort of how we think about things
today and we have to be really sort of intentional
[17:25] about thinking about sort of the changing
environment. I'll tell you right now, we're
[17:29] not ready for that in the gaming space. Um maybe
we will be at some point in time, but that's not
[17:35] not something that we currently consider today.
Supervised tests, I talked about that. The runtime
[17:39] state, always testing, always running the system.
Um in the EU, there's an AI act that requires
[17:46] uh countries to have a sandbox where people and
businesses can go in and constantly run emails
[17:51] on uh new models that are coming out, business
scenarios that they want to check. Um,
[17:56] and it looks like I'm slightly out of time, so
I'm just going to end with this. Am I out of time,
[18:01] right? Is that what that timer is? Yeah. Um, so
the the last thing I'll say is that the governance
[18:07] becomes the infrastructure. And um and the winners
in regulated AI, people who are going to be
[18:13] successful in this are going to be companies that
think about the evals, the tests, this kind of
[18:18] moving state, continuously prove their agents are
bounded, observable, accountable, and stoppable.
[18:24] We spent the last 50 years trying to figure out
how to certify frozen code. We figured it out.
[18:28] It's not that hard. So the next couple years,
5 years, 10 years, it's all going to be about
[18:32] how to do this in an AI environment. Thank you
very much. And uh I'll be outside for questions.
