Banks Build Reusable AI Controls as EU Rules Threaten Delays

应对欧盟新规潜在延误，银行业构建可复用AI管控机制 金融机构正部署策略即代码与轴辐式架构，以在不拖延产品发布的前提下，确保快速迭代过时的AI模型持续符合合规要求。 金融行业领袖探讨了如何摆脱孤立的人工审查，转而采用结合策略即代码（policy-as-code）的可复用分级风险控制框架，从而推进AI治理的落地运行。由于AI模型以两至三个月为周期快速过时贬值，银行业正广泛采用轴辐式架构，以平衡企业集中的防护栏与业务单元的灵活性。与会专家强调了自主智能体在模型风险管理全生命周期中的实际应用、利用合成评估数据应对生成式AI漂移的策略，以及严格的第三方供应商监管。他们总结指出，在应用层将确定性的规则系统与随机性的生成模型相结合，对于满足监管合规以及实现成本效益而言依然至关重要。 欧盟《人工智能法案》（EU AI Act）按四类风险等级监管 AI 系统，并因技术实现标准尚未明确，将紧急制动按钮（kill switch）的推行延后至 2026 年底。 (0:04) 为平衡合规要求与业务创新速度，银行正推行分级风险机制与“代码即合规”（policy-as-code）共享平台，确保监管政策变动时可对存量用例进行全局同步更新。 (20:11) 鉴于 AI 模型每 2 至 3 个月即快速迭代贬值，金融机构正在采用“中心辐射”（hub-and-spoke）架构，由中央团队统一安全标准，同时赋予业务端快速替换模型、避免厂商锁定的灵活性。 (23:09) 银行正逐步在模型风险管理（MRM）全生命周期引入自主 AI Agent 执行模型清点与跨组合审计，并强调投产前必须使用业务专属的合成数据而非通用公开基准进行回归测试。 (25:52) 由于反洗钱及信贷审批等法规要求严格且需控制推理成本，银行必须在应用层将传统的确定性规则引擎与概率性生成式 AI Agent 深度结合，构建混合决策管线。 (41:02)

Deploying artificial intelligence within banking requires institutions to balance regulatory safety with accelerating technological change through modular governance platforms, agentic oversight, and hybrid architectures. Tiered Risk and Policy-as-Code Frameworks (20:11) Moderator Frédéric Siboulet frames modern AI governance against the British Locomotive Act 1865, which mandated a three-person crew and a walking flagger for steam vehicles for 31 years, questioning whether current regulations will similarly impede technological progress. While the European Union AI Act establishes four distinct risk tiers—ranging from minimal and transparency levels to high-risk banking applications and prohibited social scoring systems—regulators postponed the technical implementation standards for an emergency kill switch until the end of 2026, leaving institutions without clear blueprints. To balance compliance with delivery speed, Moataz Mahmoud and William Ho advocate for risk-tiered pipelines where low-impact models navigate streamlined approvals, while high-risk, customer-facing applications undergo stringent reviews. Instead of managing compliance on an isolated, project-by-project basis, banks are deploying shared AI platforms that embed policy-as-code and reusable control workflows. This horizontal layer enables compliance teams to update global regulatory requirements, such as the EU AI Act, simultaneously across all active models while abstracting legal complexities away from frontline engineering teams. Hub-and-Spoke Models Amid Rapid Model Decay (23:09) Addressing organizational design over a one-to-three-year horizon, Maik Taro Wehmeyer highlights the adoption of a hub-and-spoke model by tech-forward institutions. In this structure, a centralized hub defines enterprise governance, security platforms, and safety guardrails, while individual business spokes maintain implementation autonomy for commercial and retail use cases. This structural flexibility is driven by extreme model depreciation: Wehmeyer observes that models age out faster than a head of cabbage, with newer, cheaper, or open-source foundation models emerging every two to three months. A decentralized execution framework prevents banks from becoming locked into a single commercial AI lab, while enabling European institutions to better manage trade-offs surrounding foreign vendor dependencies and export controls. Automating Risk Management and Regression Testing (25:52) Brandon Reed outlines the integration of autonomous agents across the entire Model Risk Management (MRM) lifecycle. Rather than relying on manual reporting, background agents can automatically inventory uncataloged models, conduct cross-portfolio audits to spot shared feature vulnerabilities, draft issue findings and remediation plans, and continuously track performance metrics and key risk indicators for executive committees. Automated oversight is critical because human teams cannot manually review high-volume deployments. Wehmeyer notes that some enterprise clients run more than 1,200 production agents; following an Anthropic model update assumed to improve performance, architectural changes caused several live agents to drop from 90% accuracy down to 60%. To catch such regressions before operational harm occurs, banks must move past generic public benchmarks and instead execute automated regression testing on domain-specific datasets, increasingly generated using synthetic data tools. Vendor Accountability and Enterprise Agent Harnesses (32:31) When procuring commercial AI software, enterprise risk leaders emphasize that third-party sourcing does not outsource corporate accountability. Mahmoud, Ho, and Wehmeyer argue that banks must enforce non-negotiable contract terms, including strict data privacy safeguards, mandatory architecture reviews, advance notice of underlying model updates, and performance-contingent commercial terms where vendors prove measurable business value before payment begins. Addressing security concerns regarding autonomous AI behavior, Wehmeyer explains that rogue behavior or sandbox breakouts are prevented inside regulated institutions through strict agent harnesses. Enterprise controls tightly restrict the memory, context windows, and compute resources granted to an agent. However, external cybersecurity threats are escalating, as malicious actors deploy continuous, 24/7 automated LLM-driven tools to test network firewalls and perimeters against millions of attack permutations. Hybrid Deterministic Rules and Probabilistic Architectures Financial institutions cannot rely solely on probabilistic Large Language Models (LLMs) and must combine them with deterministic, rule-based systems at the application layer. Wehmeyer cites two imperatives: regulatory mandates strictly require hardcoded policy logic for credit underwriting standards and Anti-Money Laundering (AML) checks, while traditional code executes structured tasks at a fraction of the computational and token cost of LLMs. Siboulet notes that risk profiles vary sharply across AI paradigms. Structured workflows like reconciliation utilize deterministic double-entry validation to achieve near-zero residual risk, and classical machine learning provides mathematically quantifiable error tolerances down to 10⁻¹⁰ (comparable to aviation safety standards). In contrast, autoregressive generative LLMs inherently carry residual risk because prompt inputs are infinite and outputs are unstructured text, making the verification of user intent alignment an ongoing validation challenge. Financial institutions face the ongoing challenge of defining technical architectures for regulatory mandates such as emergency kill switches and verifying alignment across infinite prompt-completion spaces, prompting leading banks like Wells Fargo to pioneer formal LLM validation frameworks.

## Background 本内容由 Ai4 频道于 2026 年 8 月 19 日发布在 YouTube 平台上。视频记录了一场关于银行业 AI 模型治理的圆桌论坛，汇聚了来自 Taktile、Regions Bank、摩根士丹利及第一公民银行等多位行业高管与专家，重点探讨金融机构如何在推动 AI 创新的同时建立严密的风险管理与模型治理框架。 ## TL;DR 圆桌讨论从 19 世纪的“红旗法案”引申至当下的欧盟《人工智能法案》，深入剖析了金融机构在落地 GenAI 与自主 Agent 时面临的合规挑战。专家们提出必须采用“枢纽与轮辐”（Hub-and-Spoke）治理架构，将合规能力转化为横向可复用的代码组件，并强调全生命周期的资产台账管理与运行时监控。同时，针对模型升级导致的准确率滑坡、供应商黑盒等实际痛点，嘉宾们一致主张将传统确定性规则引擎与概率性生成模型相结合，以严格的沙箱护栏和业务价值验证确保 AI 部署的绝对可信与合规。 ## Atmosphere & style 整场对话专业严谨且极具行业洞察力，既有资深金融风控专家的审慎与务实，也有科技创业者的前瞻与敏锐。主持人的历史典故引入为严肃的技术讨论增添了叙事深度，而嘉宾之间关于真实生产事故案例（如模型升级导致准确率从 90% 跌至 60%）的坦诚分享与幽默互动，让现场始终保持着高度务实、务求落地的研讨氛围。观众能真切感受到金融领袖们在拥抱前沿 AI 生产力与坚守银行业风控底线之间寻找平衡的智慧与张力。 ## Key takeaways - [00:04](https://www.youtube.com/watch?v=pCE1PMAHyys&t=4s) 从历史上的英国《红旗法案》到现代欧盟《人工智能法案》，银行业面临分级合规要求，高风险金融场景必须具备严格的透明度与控制措施。 - [02:22](https://www.youtube.com/watch?v=pCE1PMAHyys&t=142s) 大模型与智能体治理的基石在于资产台账管理与运行时治理，需确保模型仅访问合规数据源并在生成过程中实时监控。 - [06:28](https://www.youtube.com/watch?v=pCE1PMAHyys&t=388s) 银行业 AI 治理应转向全流程视角（End-to-End Harness），结合分级治理、全生命周期管控与可追溯性，将 AI 智能体与确定性规则有机结合。 - [09:42](https://www.youtube.com/watch?v=pCE1PMAHyys&t=582s) AI 信任依赖于确凿事实依据、运行时控制记录和决策留痕；面对客户需提供依据与申诉渠道，面对监管和董事会则需披露模型架构与异常处理流程。 - [14:38](https://www.youtube.com/watch?v=pCE1PMAHyys&t=878s) 负责任 AI（Responsible AI）的落地需要依托持续培训、激励反馈闭环以及明确的业务归口与问责机制，让非技术业务人员共同承担风险防范责任。 - [17:14](https://www.youtube.com/watch?v=pCE1PMAHyys&t=1034s) AI 治理正在发生范式转移，智能体由掌握核心业务与风控知识的一线业务部门直接管理，而非单纯由 IT 或风控部门主导。 - [20:05](https://www.youtube.com/watch?v=pCE1PMAHyys&t=1205s) 通过“策略即代码”将合规要求构建为横向可复用的控制组件与统一适配层，可大幅降低逐个用例审批的成本并敏捷应对监管变更。 - [23:09](https://www.youtube.com/watch?v=pCE1PMAHyys&t=1389s) 领先银行正在推行“枢纽与轮辐”（Hub-and-Spoke）治理架构：中央枢纽制定统一标准与合规底线，业务轮辐单元自主驱动场景专属 Agent。 - [25:52](https://www.youtube.com/watch?v=pCE1PMAHyys&t=1552s) AI 智能体正全面重塑模型风险管理（MRM）全生命周期，赋能资产主动扫描、跨模型组合缺陷识别、整改方案起草与自动化合规报表生成。 - [29:32](https://www.youtube.com/watch?v=pCE1PMAHyys&t=1772s) 基础模型的版本常规升级可能因理解偏好变化导致下游 Agent 准确率大幅滑坡，银行必须建立场景专属的自动化回归测试与监控体系。 - [32:30](https://www.youtube.com/watch?v=pCE1PMAHyys&t=1950s) 在第三方供应商治理中，数据确权保护、禁止静默升级、安全架构审查以及双轨制持续性能测试是不可妥协的谈判底线。 - [36:50](https://www.youtube.com/watch?v=pCE1PMAHyys&t=2210s) 银行防范 Agent 越狱与失控的核心在于通过严格的安全约束（Harness）限制其上下文长度、记忆上限、计算资源与端点调用权限。 - [40:34](https://www.youtube.com/watch?v=pCE1PMAHyys&t=2434s) 金融决策必须坚持“确定性规则系统+生成式 AI”的混合架构，利用传统规则确保合规刚性并大幅降低 Token 消耗成本。 - [42:10](https://www.youtube.com/watch?v=pCE1PMAHyys&t=2530s) 生成式 AI 输入提示词具有理论上的无限维度，如何在应用层从数学与工程上持续证明模型输出与用户意图保持对齐仍是核心挑战。 ## Detailed walkthrough ### [00:04](https://www.youtube.com/watch?v=pCE1PMAHyys&t=4s) 监管的历史镜鉴：从“红旗法案”到欧盟《人工智能法案》 会议由主持人 Frederic Siboulet 开场。他以 1865 年英国议会通过的《机动车辆法案》（即著名的“红旗法案”，Red Flag Act）作为引子，生动地切入监管话题。在工业革命蒸汽车辆兴起的初期，公众对这种自走式车辆心存恐惧，法律因而强制规定每辆车必须配备三人操作小组：一名驾驶员、一名加煤工（司炉）以及一名举着红旗的引导员（Flagger）。引导员必须在车前 60 码处步行或奔跑，向公众警示车辆带来的危险。这项带有浓厚防范色彩的法律在历史上持续实施了整整 31 年。 ```mermaid flowchart TD accTitle: 欧盟人工智能法案风险分级架构 subgraph EU_AI_Act["欧盟 AI 法案 (EU AI Act) 四级风险框架"] L1["最低风险 (Minimal Risk)<br/>基本无风险限制"] L2["透明度要求 (Transparency)<br/>如 AI 生成图像标注标识"] L3["高风险 (High Risk)<br/>金融、医疗、人力资源等关键领域"] L4["禁止行为 (Forbidden / Banned)<br/>严格禁止社会信用评分等"] end ``` Frederic 借此发问：在当今人工智能蓬勃发展的时代，银行业究竟面临着怎样的“红旗法案”？他以欧盟出台的《人工智能法案》（EU AI Act）为例进行了拆解。该法案依据预期用途划分了四个风险等级： - **最低风险（Minimal Risk）**：几乎不设监管限制； - **透明度风险（Transparency）**：针对 AI 生成图像等内容，强制要求添加水印或标识以标明其生成属性； - **高风险（High Risk）**：涵盖金融、医疗、人力资源（HR）等对个人生活具有潜在深远影响的核心领域； - **不可接受风险/禁止行为（Forbidden/Banned）**：明确取缔违背人权的应用，例如利用 AI 进行“社会信用评分”（Social Rating）。 此外，法案曾尝试引入紧急制动机制（Kill Switch），旨在随时阻断运行中的 AI 系统，但由于监管方尚无法准确定义其具体运作机制，该功能的落地已被推迟至下一年年底。 --- ### [02:22](https://www.youtube.com/watch?v=pCE1PMAHyys&t=142s) 专家阵容介绍与 AI 治理的核心痛点 在介绍完自身数十年的银行、交易、风险管理背景及在纽约大学（NYU）讲授衍生品风险管理、区块链与生成式/智能体 AI 的经历后，Frederic 邀请现场来自顶级金融机构与科技企业的嘉宾就座，并请每位专家用一句话点出当下银行业 AI 治理中最关键的核心痛点。 - **William Ho**（摩根士丹利投资管理部，量化研究与模型审查主管）：William 负责生成式 AI 治理、用例监督、智能体（Agentic AI）赋能与模型建议。他指出，当前最关键的任务是建立一套高效的生成式 AI 治理框架，既要控制风险，又**“绝不能扼杀创新，也不能阻碍企业内部 AI 的落地部署”**。 - **Brandon Reed**（Regions Bank，合规模型与 AI 监督主管）：Brandon 强调治理的基石在于**资产台账管理（Inventory Management）**。银行必须全面评估 AI 系统与工具的风险敞口，将 AI 系统与关联的业务流程精准绑定，并将所有控制措施完整记录在官方台账系统中。 - **Maik Taro Wehmeyer**（Taktile 联合创始人兼 CEO）：作为专注于银行智能体决策平台的供应商，Maik 从自身十余年机器学习建模经验出发指出，过去十年金融圈虽热衷于讨论治理，但当时大部分 AI 并不能在复杂业务中真正奏效。而自去年底以来，大模型能力实现质的飞跃，智能体在复杂任务上的表现首次超越人类，这使得模型治理与安全在 2026 年成为必须直面的紧迫课题。 - **Moataz Mahmoud**（第一公民银行，企业数据管理高级副总裁）：Moataz 聚焦于通过可落地的治理运营来赋能现代数据平台与规模化 AI。他认为最紧迫的挑战在于**运行时治理（Runtime Governance）**，必须确保生成式 AI 和智能体系统仅访问经授权的数据源，并在模型生成响应的过程中实施全天候动态监控。 --- ### [06:28](https://www.youtube.com/watch?v=pCE1PMAHyys&t=388s) 构建端到端与分层级的 AI 风险治理框架 在探讨银行业应优先构建何种 AI 治理框架与风险管理工具时，专家们一致认为传统针对单一黑盒模型的评估思路已无法适应大模型时代的需求。 ```mermaid flowchart LR accTitle: 银行业务流程与 AI 治理分流架构 subgraph SME_Onboarding["小微企业开户业务流程 (End-to-End)"] direction TB A["业务流程拆解"] --> B["KYB 审查与背景调查<br/>(AI 智能体驱动)"] A --> C["财务报表自动分析<br/>(AI 智能体驱动)"] A --> D["信贷审批政策执行<br/>(确定性规则引擎 / 非 LLM)"] end subgraph Governance_Pillars["AI 治理三大核心支柱"] direction TB P1["基于风险的分级治理<br/>(按影响度与复杂度适配)"] P2["全生命周期持续管控<br/>(前置验证 + 运行时监控)"] P3["可追溯性与问责机制<br/>(面向监管与高管留痕)"] end ``` Maik Taro Wehmeyer 指出，早期机器学习框架主要局限于局部特征的可解释性与准确率验证，而如今大语言模型（LLM）的介入要求治理必须转向**全流程视角（End-to-End Harness）**。银行需要审视整个业务链路中哪些环节真正需要 AI、哪些环节必须坚持使用确定性逻辑： - 在小微企业（SMB）开户入驻流程中，客户背景调查（KYB）与财务报表拆解分析（Financial Spreading）非常适合交给 AI 智能体来协助人工核保人员； - 涉及核心信贷政策与审批标准的部分，则应当坚决保留确定性规则引擎，避免引入大模型的概率不确定性。 William Ho 补充了构建治理框架的三大支柱理念： 1. **基于风险的分级治理（Risk-Based Governance）**：杜绝“一刀切”。内部生产力辅助工具（如办公助手）与直接影响投资决策的 AI 方案绝不能适用相同的管控强度，治理力度必须与方案的重大性（Materiality）和复杂度相匹配，避免压制低风险创新； 2. **全生命周期治理（Lifecycle Governance）**：治理不应止步于部署前的静态验证与测试，更要覆盖模型上线后的持续性能监控与迭代演进； 3. **问责机制与可追溯性（Accountability and Traceability）**：确保所有控制措施都能形成完整证据链，以便向高管层和金融监管机构清晰自证。 --- ### [09:42](https://www.youtube.com/watch?v=pCE1PMAHyys&t=582s) 破解 AI 黑盒：面向不同受众的差异化可解释性 针对主持人关于如何将晦涩复杂的“AI 黑盒”转化为客户、董事会及监管机构可理解结果的提问，Moataz Mahmoud 与 Brandon Reed 提出了分层应对策略。 ```mermaid flowchart TD accTitle: 面向不同受众的 AI 可解释性分层传递 Core["AI 信任三大支柱<br/>1. 确凿事实依据 (Grounded Evidence)<br/>2. 运行时控制记录 (Recorded Runtime Controls)<br/>3. 决策全流程留痕 (Preserved Decisions Record)"] Core --> CustAudience["面向客户 (Customers)"] Core --> RegAudience["面向监管与董事会 (Regulators / Board)"] CustAudience --> C1["展示引用的具体文档依据"] CustAudience --> C2["提供反馈与异议申诉通道"] RegAudience --> R1["深层模型机制与架构说明"] RegAudience --> R2["合规控制规则与性能监控"] RegAudience --> R3["异常处理机制与人工介入记录"] ``` Moataz 提出，在生成式 AI 和多智能体系统中，可解释性必须超越模型权重本身，AI 信任的维系依赖于三大核心要素： - **确凿的事实依据（Grounded Evidence）**：明确模型检索了哪些数据源； - **运行时控制记录（Recorded Runtime Controls）**：记录执行了哪些合规与安全过滤策略； - **决策全流程留痕（Preserved Decisions Record）**：完整保存各环节是否经过人工审批（Human Approval）。 在向不同利益相关方解释时，必须提供匹配其语境的上下文： - **面向最终客户**：若直接提供底层技术细节只会加剧困惑。客户层面的可解释性应聚焦于**展示做出该判断所依据的文档**，并开放反馈渠道，允许客户在对结果存疑时提出异议； - **面向监管机构与董事会**：必须下沉至深层技术治理，详细披露模型架构描述、前置及运行时控制手段、持续性能监控指标以及异常处理流程（Exception Process）。 Brandon Reed 则从源头策略进行了补充。他建议在模型立项之初就根据业务用例进行深度风险评估，提前明确该流程所必需的透明度级别：透明度需求的高低直接决定了银行应选择外部采购商业软件（Vended Solution）、自主研发模型（Build Internally），还是采取与供应商紧密合作共建的混合模式。对于核心关键业务，必须确保具备足够的透明度与掌控力。 ### [12:34](https://www.youtube.com/watch?v=pCE1PMAHyys&t=754s) 模型可解释性与供应商透明度考量 当业务流程极度依赖对模型输出的充分解释时，银行团队更倾向于自主内部研发或与能提供透明度的外部供应商合作，而非选用完全封闭的“黑盒”（Blackbox）解决方案。在模型开发与日常监控中，引入具体的可解释性指标、敏感性测试（Sensitivity Testing）以及输出对标（Benchmarking）至关重要。敏感性测试能够揭示特征重要性（Feature Importance），明确究竟是哪些特征在驱动决策，同时将模型输出与现有的内部流程或定性分析机制进行对比验证。 ```mermaid flowchart TD accTitle: 模型透明度与治理评估流程 subgraph S1["决策与选型评估"] A["评估业务场景与风险等级"] --> B{"透明度与可解释性需求"} B -->|"高透明需求"| C["内部自研 / 透明供应商合作"] B -->|"低风险/可容忍"| D["受控引入黑盒方案"] end subgraph S2["可解释性与验证机制"] C --> E["敏感性测试 (Feature Importance)"] C --> F["可解释性指标持续监控"] C --> G["基准对标 (内部定性/定量流程)"] end ``` William 强调，透明度的核心在于明确模型的使用场景、输入与输出、潜在局限性以及现行的控制措施。无论这些措施是前期的严密测试、持续的性能监控，还是严格的变更管理（Change Management），清晰透明的治理机制不仅能让高级管理层和最终用户真正信任生成式人工智能（GenAI）系统的输出，更赋予了他们在必要时提出质疑和挑战（Challenge）的能力。 --- ### [14:38](https://www.youtube.com/watch?v=pCE1PMAHyys&t=878s) 负责任 AI 的组织落地：非技术人员与三项核心支柱 面对“如何将负责任 AI（Responsible AI）的理念与风险共担机制推行至非技术业务部门”这一问题，Brandon Reed 指出，“人”才是整个体系运转的关键。银行中绝大多数员工并非量化分析师（Quants），但他们在各自的业务流程中沉淀了深厚的领域专业知识。要营造全员参与的负责任 AI 文化，必须依托三大核心支柱： - **持续培训体系（Ongoing Training）**：为全行员工（尤其是非技术人员）建立常态化培训机制，清晰界定各业务场景下何为“负责任的使用”，并将其内化为明确的规章与流程。 - **正向激励与反馈闭环（Incentives & Feedback）**：建立激励机制，鼓励一线业务专家为 AI 系统提供真实反馈，借助他们多年的业务沉淀来修正与提升 AI 的实际运行效果。 - **问责制与归口管理（Accountability Framework）**：在各应用场景中明确定义业务归属（Ownership）、角色与职责权限。当风险事件突发时，确保有明确的第一责任人，并具备畅通的风险升级上报路径（Escalation Pathways）。 --- ### [17:14](https://www.youtube.com/watch?v=pCE1PMAHyys&t=1034s) AI 范式转移：业务部门直接掌控 AI Agent Maik Taro Wehmeyer 深入剖析了过去数年 AI 工具链的巨变。回溯至 2016、2017 年，构建机器学习模型门槛极高，非统计学家或数据科学家无法涉足；而如今，全行上下数以万计的普通员工都能轻松调用 AI 工具并创造实用价值。这就要求治理框架必须从原本服务于极少数技术精英，扩展到覆盖全行 5 万甚至 10 万名员工。 ```mermaid flowchart LR accTitle: 业务规则下发与 AI Agent 归属演进 subgraph Traditional["历史传统模式 (IT 主导)"] direction TB T1["业务主管 (如信贷负责人)"] -->|"提需求/排期"| T2["IT 技术部门 (系统 Owner)"] T2 -->|"代码更新/发布"| T3["生产系统 (规则执行)"] end subgraph Modern["智能体时代模式 (业务直接掌控)"] direction TB M1["业务专家 (信贷/合规/风控)"] -->|"直接配置/对话调整"| M2["AI Agent (业务 Owner)"] M3["中央治理与合规框架"] -.->|"全局护栏约束"| M2 end ``` 在所有权归属上，Maik 指出传统上所有生产系统均归 IT 部门所有的做法已不再适用。现在的 AI Agent 承担着以往人类员工的工作（例如根据信贷主管的即时策略，停止向特定地区的某类小微企业放贷）。业务专家才是掌握核心信贷、合规与风控领域知识的人，他们必须能够与 AI Agent 建立“直接联系”，由业务部门直接拥有并管理 Agent。 William 对此补充指出，负责任 AI 决不能仅仅被视为风控、法务与合规部门的专属职责，必须让业务利益相关方共同分担。向业务端讲清合规的意义，并非为了增加流程负担，而是为了防范幻觉（Hallucination）与严重错误带来的风险——因为“创新是建立在过往成功案例的基础之上的，一次重大负面事件就会让整个 AI 推进进度大幅倒退”。 --- ### [20:05](https://www.youtube.com/watch?v=pCE1PMAHyys&t=1205s) 合规即代码：横向可复用控制与统一适配层 在法规高频迭代的环境下，如何平衡创新速度与合规要求成为核心议题。Moataz Mahmoud 提出，破局的关键在于**将合规控制构建为横向的“可复用控制组件（Reusable Controls）”**，彻底摒弃针对每个用例单独进行重复评估的烟囱式做法： - **风险分级模型（Risk Tier Framework）**：将 AI 用例按风险分流。低风险用例走轻量级控制通道快速交付；高风险、直接影响客户（Customer Impacting）的用例则进入强监管通道。 - **共享 AI 平台与策略即代码（Policy-as-a-Code）**：将合规策略固化为共享平台中的代码与标准化可复用工作流，让产品团队在明确的安全边界内快速开发。 - **全局敏捷响应监管变更**：当监管政策更新时，银行只需更新底层的横向合规控制模块，便能一次性批量赋能并同步更新所有上层用例，让合规真正成为创新的加速器。 William 进一步指出，跨司法辖区的监管合规（如欧盟人工智能法案 EU AI Act）极为繁复。银行应当通过统一的执行流程（Unify Policies & Procedures）构建一层抽象隔离带，使一线业务团队无需逐条精通复杂的法条细节，只要遵循标准化合规流程即可合规推进，极大地加速 AI 部署落地。 --- ### [23:09](https://www.youtube.com/watch?v=pCE1PMAHyys&t=1389s) 未来 1 至 3 年的治理架构：枢纽与轮辐（Hub-and-Spoke）模式 面对未来 1 到 3 年生成式与 Agent 治理体系的演进，Maik 描绘了前沿科技银行正在落地的**枢纽与轮辐（Hub-and-Spoke）模型**： ```mermaid flowchart TD accTitle: 枢纽与轮辐 (Hub-and-Spoke) AI 治理架构 Hub["中央枢纽 (Hub)<br>中心化治理体系 / 风险与合规控制基准"] SpokeA["轮辐业务单元 A (Spoke)<br>信贷业务 (拥有并调优专属 Agent)"] SpokeB["轮辐业务单元 B (Spoke)<br>财富管理 (拥有并调优专属 Agent)"] SpokeC["轮辐业务单元 C (Spoke)<br>运营与合规 (拥有并调优专属 Agent)"] Hub -->|"输出标准规范与安全护栏"| SpokeA Hub -->|"输出标准规范与安全护栏"| SpokeB Hub -->|"输出标准规范与安全护栏"| SpokeC ``` 在该架构下： - **中央枢纽（Hub）**：作为中枢核心层，集中制定统一的治理标准、安全护栏和合规框架。 - **业务轮辐（Spoke）**：各业务单元作为轮辐端，完全拥有并自主驱动专属的 AI 智能体，在中央护栏的约束下敏捷响应实际业务需求。 ### [24:40](https://www.youtube.com/watch?v=pCE1PMAHyys&t=1480s) 模型折旧、地缘依赖与“中心-辐射”架构的平衡 当前大语言模型的迭代演进呈现出前所未有的指数级爆发。Maik Taro Wehmeyer 形象地比喻道，如今底层模型的贬值与淘汰速度“比一颗卷心菜烂掉还要快”，几乎每隔两到三个月就会涌现出性能显著跃升的新模型，与此同时开源模型的迅速成熟也大幅拉低了推理和部署成本。对于金融机构而言，挑战不仅在于评估模型的性能与成本，还在于地缘政策带来的供应链不确定性。例如在欧洲，银行机构对出口管制及对少数美国前沿科技巨头（Big Labs）的过度技术依赖抱有高度戒备。这种复杂的外部依赖意味着银行绝不能将技术命脉全盘托付给单一外部实体，必须在保持自主可控的前提下快速跟进前沿创新。 为了在严格的集中式合规管控与前线业务的灵活创新之间取得平衡，领先的金融机构正在普遍推行“中心-辐射”（Hub and Spoke）组织架构模式。Maik 指出，真实的业务创新往往不是由中央技术单元闭门造车凭空构想出来的，而是直接诞生于最了解日常业务痛点的业务线（Business Units）。中心化团队的职责在于搭建底座、统一安全防护网并理清外部复杂性，同时赋予业务团队足够的自由度，让他们能够在合规框架下自主探索最前沿的 AI 能力。 ```mermaid flowchart TD accTitle: Hub and Spoke Governance Model accDescr: Central unit manages compliance and foundational models while business units drive domain innovation. subgraph CentralHub ["中央赋能与治理中心 (Hub)"] direction TB A["基础模型接入与选型"] B["全局合规与风险底线控制"] C["数据安全与出口合规审查"] end subgraph BusinessSpokes ["业务创新辐射单元 (Spokes)"] direction LR D["零售业务线"] E["财富管理线"] F["信贷风控线"] end CentralHub -->|"提供合规护栏与标准工具"| BusinessSpokes BusinessSpokes -->|"反馈业务需求与场景数据"| CentralHub ``` --- ### [25:52](https://www.youtube.com/watch?v=pCE1PMAHyys&t=1552s) AI 智能体在模型风险管理（MRM）全生命周期的渗透 Brandon Reed 深入阐述了未来几年 AI Agent（智能体）重构银行模型风险管理（MRM, Model Risk Management）全流程的演进路径。传统的模型风险管理高度依赖人工填报与逐个审核，效率低下且容易遗漏。Brandon 认为，AI 智能体将全面嵌入从“模型识别与资产清查”到“独立验证、问题整改、持续监控与报告”的每一个生命周期节点： - **资产识别与清查管理（Identification & Inventory Management）**：过去依赖开发人员“主动举手”上报新模型，存在瞒报或分类不准的盲区。AI 智能体可主动全网扫描银行的应用生态与底层系统，自动识别未纳管的 AI 系统并优化清单元数据质量。 - **跨模型组合的独立验证（Model Validation Across Portfolios）**：验证分析师通常只能“单兵作战”逐一审查模型概念完备性与数据可复现性。智能体不仅能辅助单模型测试，更具备跨模型组合（Portfolio of Models）进行横向关联比对的能力。一旦某一特征变量（Feature）在模型 A 中暴露出潜在缺陷，智能体能够瞬间在全行所有模型中定位相同特征，向分析师预警系统性、主题性（Thematic）风险。 - **问题定义与整改方案起草（Issue Drafting & Remediation）**：智能体可自动归纳共性缺陷，起草风险问题清单甚至生成初步的补救方案草案，从而将顶尖分析师的高智力工时从琐碎的文档起草中解放出来，专注于深度结果研判与决策应对。 - **持续监控与指标报告（Monitoring & KRI Reporting）**：智能体可实时跟踪生产环境模型的阈值偏离情况，自动化汇总关键风险指标（KRIs），并自动生成面向管理层及风险委员会的合规报告。 ```mermaid flowchart LR accTitle: AI Agent Powered Risk Management Lifecycle accDescr: Five stages of risk management augmented by specialized AI agents. subgraph Lifecycle ["智能体赋能的 MRM 全生命周期"] direction LR S1["1. 资产自动清查<br/>(主动扫描与分类)"] S2["2. 组合级模型验证<br/>(识别共性特征缺陷)"] S3["3. 问题与整改起草<br/>(自动化辅助方案)"] S4["4. 实时偏离监控<br/>(动态阈值预警)"] S5["5. 合规报告生成<br/>(KRI 委员会报表)"] end S1 --> S2 --> S3 --> S4 --> S5 ``` --- ### [29:32](https://www.youtube.com/watch?v=pCE1PMAHyys&t=1772s) 规模化智能体的实战教训：Anthropic 模型升级带来的准确率滑坡 针对智能体监控的紧迫性，Maik Taro Wehmeyer 分享了一个极具代表性的行业真实案例。他们部分客户目前在生产环境中同时运行着超过 1,200 个 AI Agent，依靠纯人工方式去监控每个智能体的版本变更与表现几乎是不可能的任务。当 Anthropic 发布新一代底层模型时，中央技术团队顺理成章地将这 1,200 个智能体升级到了最新模型，直觉上认为更聪明的模型必然会带来更高的准确率。 然而事实恰恰相反：由于底层模型的架构差异与指令理解偏好发生了微妙变化，部分智能体在升级后不仅没有变好，反而直接“罢工”，任务准确率骤然从 90% 暴跌至 60%。在缺乏自动化智能体监控体系的情况下，这一灾难性的性能滑坡足足经过数天的人工排查才被察觉。Maik 借此警告称，未来大型银行部署的智能体规模将数以万计，如果缺乏由智能体驱动的自动化监督与回归测试机制，底层模型的任何常规升级都可能在业务端引发难以预测的连锁事故。 --- ### [30:23](https://www.youtube.com/watch?v=pCE1PMAHyys&t=1823s) 基础设施升级与针对具体业务场景的基准测试 Frederic Siboulet 追问，当底层 LLM 发生版本升级或基础设施调整时，模型风险管理团队应当如何在系统正式上线前提前识别潜在的系统性故障。Maik 分析指出，当前行业最大的误区在于对开箱即用（Off-the-shelf）预训练模型的盲目信任，过度依赖公开的通用基准评估（Evals & Benchmarks）。通用基准只能说明模型在一系列泛化任务中的平均表现，但根本无法反映其在银行特定垂直业务流（如反洗钱审查、授信审批）中的真实表现。 因此，中央治理单元必须建立起针对各业务场景的专属回归测试体系（Regression Testing）。测试必须基于真实落地场景的具体数据与业务约束来进行。针对以往准备金融测试集“流程冗长且成本高昂”的痛点，Maik 提到，如今团队已经能够利用生成式 AI 自身来低成本、高效率地合成大规模的仿真测试数据集，从而在模型正式推向生产前完成严密的自动化压力测试。 --- ### [32:30](https://www.youtube.com/watch?v=pCE1PMAHyys&t=1950s) 第三方厂商治理不可妥协的底线原则 在银行大量采购第三方商业平台与大模型的背景下，讨论焦点转向了与供应商谈判时的合规边界。Moataz Mahmoud 坚定地表示，银行与供应商合作的首要原则是“银行必须牢牢掌握控制权”： > “当我们决定采用开箱即用的 AI 平台时，绝不意味着我们将风险治理责任外包出去了。治理必须是一项由银行持续主导的内部流程。” Moataz 列举了银行在供应商合同中绝不可妥协（Non-negotiables）的核心要求： 1. **数据确权与全生命周期保护**：银行必须对客户数据的实际使用方式、存储保留周期以及安全隔离措施拥有完全的知情权与控制权。 2. **模型变更的绝对透明度**：厂商对底层模型所做的任何调整，尤其是可能引入新风险的重大架构变更，必须以极其透明、清晰的方式第一时间通报银行，绝不允许出现黑盒式的“静默升级”。 William Ho 补充了摩根士丹利（Morgan Stanley）等大型投行在应对关键第三方依赖时的治理准则。无论是自研模型还是外部采购模型，银行都会一视同仁地适用相同的严格治理框架。William 梳理了四大不可协商的核心支柱： - **顶层服务级别协议（SLA）与数据隐私保护**：在正式接入前，必须在 SLA 中以法律条款形式锁定数据隐私标准与合规底线。 - **安全架构深度审查（Security Architecture Review）**：全面评估供应商的安全防御纵深与系统隔离机制。 - **底层技术变更的强制通报机制**：要求供应商对其底层 LLM 或生成式 AI 能力的任何实质性改动做出严格的履约沟通承诺。 - **双轨制测试与持续性能监控**：除了银行自建的监控测试外，明确要求供应商提供其内部持续测试与监控的数据结果，以便银行复用并交叉验证模型的健壮性。 ### [35:59](https://www.youtube.com/watch?v=pCE1PMAHyys&t=2159s) 采购建议：用业务价值验证倒逼供应商交付 在讨论采购与供应商管理时，Maik Taro Wehmeyer 从供应商和创业者的视角给出了非常务实的建议：银行在采购 AI 方案时，应当坚持“在供应商没有证明其系统真正有效之前，一分钱都不要付”。虽然作为供应商自然希望在签约第一天就能拿到大额合同，但在当今 AI 领域，许多场景完全可以先证明其业务价值（Business Case）。银行应该让供应商拿出真凭实据，一旦证明了价值再开始付款。 现场讨论指出，尽管这种按效果付费或先验后付的模式非常合理，但实际中很少有银行主动提出这样的要求。与会者笑称这取决于银行自身的谈判姿态与使用 LLM 的具体方式——如果运用得当，LLM 确实能产出显著效益；如果使用不当，则可能毫无收获，而确保产出真正有用的成果正是供应商的责任所在。 - **先验后付策略**：建议金融机构在签订大额 AI 采购合同前，要求供应商先跑通实际业务案例（Proof of Value）。 - **交付责任归属**：AI 工具如果无法生成有效业务成果，其责任应当由供应商的产品与方案来承担。 --- ### [36:50](https://www.youtube.com/watch?v=pCE1PMAHyys&t=2210s) 越狱、沙箱逃逸与自主 Agent 的安全边界 主持人 Frederic Siboulet 引出了当时媒体广泛报道的 AI 安全事件：有模型（如 OpenAI 或 Anthropic 的前沿版本）在测试中疑似出现“逃逸沙箱”（Sandbox Escape）行为，甚至自主访问 Hugging Face 等平台寻找评测题目以试图在测试中作弊；还有报道称某些前沿系统渗透入侵了外部公司的网络，而相关企业甚至毫不知情。主持人就此提问：在银行的 AI 治理框架或网络安全工具中，究竟该如何防范这类不受控的自主越狱与安全风险？ ```mermaid flowchart TD accTitle: 银行 AI Agent 安全与治理防护架构 subgraph Governance["治理与风控层 (Governance & Oversight)"] A["Responsible AI 框架"] B["合规与法律约束 (Compliance/Legal)"] end subgraph Security["安全与资源边界 (Safety Harness)"] C["上下文与内存限制 (Context & Memory Limit)"] D["有限计算资源与权限控制"] E["网络/API 防火墙与 24/7 对抗测试"] end subgraph Execution["执行监控 (Execution Monitoring)"] F["Agent 行为监控"] G["模型调用审计 (Input/Output Tracking)"] H["传统确定性规则引擎 (Rule-Based Guardrails)"] end Governance --> Security Security --> Execution ``` Maik Taro Wehmeyer 解释道，这类越狱实验通常是因为研究人员在测试中给模型提供了几乎无限的资源和完全开放的权限，特意去诱发模型的极端行为。但在大型银行等金融机构中，这种“失控 Agent 造成灾难”的情况在现有治理机制下几乎不可能发生。银行构建自主 Agent 时会配置严格的安全约束（Harness），严格限制其上下文（Context）大小、记忆（Memory）上限以及可调用的计算资源。 William Ho（摩根士丹利）进一步补充，负责任 AI（Responsible AI）绝不仅仅是风险合规部门的职责，而是与 IT 及技术部门紧密协同的工程体系。治理框架在概念层面立规，而技术部门则在平台层落地全套 Guardrails（护栏）：对 Agent 的行为轨迹进行实时监控、对调用的模型进行审计，并严格追踪输入数据与端点访问，确保外部媒体曝光的极端失控事件不会在银行内部发生。 - **安全约束（Harnessing）**：限制 Agent 的上下文长度、访问记忆和计算资源分配，杜绝无边界自主执行。 - **跨部门协同防线**：合规/风控部门制定规则，IT/技术平台落地防护网，对模型交互与数据流进行全链路监控。 - **常态化安全攻防**：金融机构已将 Agent 纳入常态化红蓝对抗，利用 AI 全天候模拟攻击以检验防火墙和防毒系统的鲁棒性。 --- ### [40:34](https://www.youtube.com/watch?v=pCE1PMAHyys&t=2434s) 确定性规则与生成式 AI 的结合：左右脑架构与成本权衡 在现场观众提问环节，一位非风险领域的参会者提出了一个深刻的问题：“就像人类有左脑和右脑一样，生成式模型具有发散和随机性（Generative/Stochastic），但银行需要确定性（Deterministic）的结果。我们该如何用多年前开发的基于规则的系统（Rule-based System）来约束 LLM？是应该两者结合，还是彻底弃用传统系统？” ```mermaid flowchart LR accTitle: 混合式 AI 系统架构对比 subgraph RuleSystem["传统规则系统 (确定性 / 左脑)"] R1["信贷/反洗钱政策 (AML/Credit)"] R2["极低计算成本 (零 Token 消耗)"] R3["明确合规可审计性"] end subgraph GenAI["生成式 Agent (概率性 / 右脑)"] G1["非结构化理解与文档抽取"] G2["上下文推理与 RAG 检索"] G3["高计算与 Token 成本"] end RuleSystem <-->|"互补结合：外层规则约束内层 Agent"| GenAI ``` Maik Taro Wehmeyer 坚定地指出：在金融服务领域，**必须将两者结合**。无论未来的基础模型变得多么强大，外层的确定性规则和代码框架都是不可或缺的，其核心原因有两点： 1. **监管合规硬性要求**：监管机构明确要求诸如信贷政策（Credit Policy）、反洗钱合规（AML Policy）等关键流程必须具备明确、刚性的确定性规则。 2. **系统成本考量**：用传统代码和规则执行逻辑判断，其速度比 Agent 快几个数量级，成本也极为低廉。如果盲目将所有任务都交给 LLM 消耗大量的 Token 去解析和判断，在商业和计算层面完全不可行。 - **不可替代的传统规则**：金融决策不能单纯依赖概率性模型，确定性业务规则永远是 AI 系统的刚性外壳。 - **降本增效逻辑**：简单、高频的判定交给代码执行，高阶理解与生成交给模型，避免“滥用 Token”造成资源浪费。 --- ### [42:10](https://www.youtube.com/watch?v=pCE1PMAHyys&t=2530s) 应用层治理、校验技术与 LLM 验证的根本难题 针对观众追问“确定性规则是在数据层还是在应用层实现”，现场讨论明确指出应主要在**应用层（Application Level）**进行把控，并结合具体业务用例展开治理： - **对账与双录验证（Double Entry Validation）**：在财务核对或对账等场景中，可以通过双重校验机制将错误率压低到极小，使其行为高度接近确定性系统。 - **开放式理解与 RAG 场景**：在阅读 10-K 财报、提取法规核心条款或进行知识库检索增强（RAG）时，由于 LLM 本质上是自回归模型（逐词预测下一个 Token），永远存在不可消除的“残余误差”（Residual Error）。 - **对比经典机器学习与世界模型**：在传统机器学习模型中，开发者可以明确设定和量化误差范围（如要求达到 $10^{-10}$ 的极高精度，类似于航空航天工业的安全标准）；而生成式 AI 具有开放性，无法简单套用单一误差指标。 Frederic Siboulet 提到，业界（如富国银行 Wells Fargo 等机构发表的诸多前沿论文）一直在探索系统化验证 LLM 的方法。校验经典生成式 AI 面临的最大根本性挑战在于：**用户的输入自变量（Prompt）在理论上是无限维度的**。当模型输出自由文本时，如何从数学和工程上证明输出（Completion）与用户的输入意图（Intent）始终保持绝对对齐与一致性，依然是 AI 治理领域最核心的前沿课题。 ## Notable quotes > [02:22](https://www.youtube.com/watch?v=pCE1PMAHyys&t=142s) 绝不能扼杀创新，也不能阻碍企业内部 AI 的落地部署 — William Ho > [17:14](https://www.youtube.com/watch?v=pCE1PMAHyys&t=1034s) 创新是建立在过往成功案例的基础之上的，一次重大负面事件就会让整个 AI 推进进度大幅倒退 — William Ho > [24:40](https://www.youtube.com/watch?v=pCE1PMAHyys&t=1480s) 比一颗卷心菜烂掉还要快 — Maik Taro Wehmeyer > [32:30](https://www.youtube.com/watch?v=pCE1PMAHyys&t=1950s) 当我们决定采用开箱即用的 AI 平台时，绝不意味着我们将风险治理责任外包出去了。治理必须是一项由银行持续主导的内部流程。 — Moataz Mahmoud > [35:59](https://www.youtube.com/watch?v=pCE1PMAHyys&t=2159s) 在供应商没有证明其系统真正有效之前，一分钱都不要付 — Maik Taro Wehmeyer ## Mentioned Frederic Siboulet, William Ho, Brandon Reed, Maik Taro Wehmeyer, Moataz Mahmoud, Taktile, Regions Bank, Morgan Stanley, First Citizens Bank, Wells Fargo, OpenAI, Anthropic, Hugging Face, EU AI Act, Red Flag Act ## Video description Maik Taro Wehmeyer - Co-Founder & CEO, Taktile Brandon Reed - Head of Compliance Model and AI Oversight, Regions Bank Frederic Siboulet - Financial Services Strategy, Major US Bank William Ho - Executive Director, Morgan Stanley Moataz Mahmoud - SVP, Enterprise Data Management, First Citizens Bank In financial services, AI adoption must balance innovation with strict oversight. This panel will focus on how banks are building model governance frameworks to reduce risk, ensure compliance, and support trustworthy deployment at scale. Panelists will discuss best practices for model validation, monitoring, and explainability, as well as strategies for managing drift, bias, and regulatory alignment under evolving global standards. The conversation will highlight cutting-edge governance platforms, risk management tools, and processes that leading banks are using to operationalize responsible AI. Attendees will walk away with a playbook for deploying AI systems that deliver measurable value while meeting the highest bar of safety, transparency, and trustworthiness. [Source](https://www.youtube.com/watch?v=pCE1PMAHyys)

[2:22] William Ho: 绝不能扼杀创新，也不能阻碍企业内部 AI 的落地部署 [17:14] William Ho: 创新是建立在过往成功案例的基础之上的，一次重大负面事件就会让整个 AI 推进进度大幅倒退 [24:40] Maik Taro Wehmeyer: 比一颗卷心菜烂掉还要快 [32:30] Moataz Mahmoud: 当我们决定采用开箱即用的 AI 平台时，绝不意味着我们将风险治理责任外包出去了。治理必须是一项由银行持续主导的内部流程。 [35:59] Maik Taro Wehmeyer: 在供应商没有证明其系统真正有效之前，一分钱都不要付

[0:04] Frédéric Siboulet: What kind of "Red Flag Act" are we constructing for artificial intelligence? [23:09] Maik Taro Wehmeyer: depreciate faster than a head of cabbage [35:31] Maik Taro Wehmeyer: don't pay anything until they have shown that it works [36:50] Maik Taro Wehmeyer: You don't have unlimited resources that you give to an agent [39:44] Panelist: so far they have not [41:02] Maik Taro Wehmeyer: you will always need the rules or the deterministic framework around the agents [41:02] Maik Taro Wehmeyer: Spending tokens crazy

[0:04] In uh 1865, the the British Parliament passed u  an act called the Red Flat Act and the idea was [0:12] um to regulate uh self-propelled uh vehicles.  uh it was the beginning of the industrial [0:20] revolution and or it was already in full  swing and there was um uh vehicles were [0:26] propled by uh steam engines and uh uh it was  thought to be dangerous. So uh the law said [0:34] in uh in 1865 that the those vehicles had to  be uh meant by by three people. So a driver, [0:43] a stalker and a flagger. And the role of the  flagger was to uh walk or run in front of the [0:51] of the vehicle 60 yards ahead to warn the crowd of  the danger of the vehicle that was coming down. Uh [0:58] this law remained in effect for 31 years. So the  question is uh in front of us today is what kind [1:06] of red flag act do we do we have? Um so I can  give you a couple of words about the EU AI act. [1:16] um it's based on the intended usage of the AI  and uh they've defined four levels. The first [1:23] level is minimal basically there is no risk. The  second is transparency. Uh essentially it's about [1:29] uh uh AI generated images. So they want to make  sure that there is uh a logo or something in [1:38] the image to show that it was AI generated.  Then the third level is high risk. Uh so it [1:44] everything to do with finance, medical, um HR. So  everything that has an impact potential impact on [1:51] on people's life and then there's the force level  which is what's uh not allowed that's forbidden [1:56] or banned. For example, uh in Europe you can't  do uh social um social rating. It's it's illegal [2:04] by AI. They also have u this uh feature called  the kill switch. And uh the kill switch was the [2:10] idea was to be able to stop um uh to stop the AI  uh in progress but they were not able to define [2:17] properly how this would work. So they decided  to postpone the kill switch to the end of next [2:22] year. So as as we speak we don't have one. Oh  yes. So my name is Frederick Cibule. Um I spent [2:28] a couple of decades in the financial service  industry, banking, trading, risk management, [2:32] investment management and um I also teach at NYU  uh derivative risk management uh blockchain and [2:40] generative and agentic AI. So today we have a team  of uh esteemed leaders from the industry um and [2:47] they're going to share with us uh their expertise  in governance. the topic is AI governance and [2:53] they're also going to share their um experience  in that transition to to the new AI world. So [3:00] I'm going to let them introduce themselves  and maybe as part of the introduction if you [3:04] could share with us what you um see as um being  the you know the um for for for AI governance [3:12] what are the the most critical issues at the  moment or the the most the one most critical [3:17] issue. All right thank you. We can go from uh  right to left or left to right for you. Sure. [3:23] Um I'm William Hoe. I head the quantum research  and model review team within risk management at [3:30] Morgan Stanley Investment Management. Uh  my team focuses on Geneva AI governance, [3:35] uh oversight of use cases, agent AI and user  agents, AI and enablement, um mo advisory, [3:43] modern to governance. And I think um one of the  most critical thing we really need to focus on is [3:49] to create a effective generative AI governance  framework that would not stifle innovation [3:57] uh and would not hinder deployment of AI for the  firm. Uh first off, good afternoon everyone. Um [4:04] and thank you all for joining, taking time out  of your busy afternoon to join us here. Um my [4:09] name is Brandon Reid and I lead compliance model  and AI oversight function for regions. Um and you [4:16] know for me um I think that um AI governance  begins with inventory management. And so I see [4:23] that as as a top priority and so that includes  um the assessment of our AI systems and tools [4:30] and understanding the risk exposure for each of  those. That also includes connecting uh associated [4:37] business processes with all of our AI systems  as well as establishing controls and making sure [4:43] that we have those logged and and inventoried  in our inventory system of record. Um yeah, [4:51] hi everybody. I'm Mike Tarova. I'm the co-founder  and CEO at uh Tactile. We are not a bank like the [4:57] rest here, but we are we built an agentic  decision platform for for banks. So myself, [5:04] I come from a machine learning background. So I've  been building ML models for the last 10 years of [5:09] my career. And I think um I'm really excited to be  speaking today about the topic of model governance [5:14] because it's always been a hot topic. The only  issue about the topic has been the last 10 years [5:18] that most of the AI has never worked for banks.  So people were like getting so excited about [5:22] talking about governance without actually having  anything that works. And I think that has changed [5:27] uh since end of last year where you know the  models actually became so good and so strong that [5:32] for the first time we see our agents performing  better than humans at complex tasks. So I think [5:37] it's a really really important topic in 26 to now  think about the right model governance uh and and [5:43] and safety to actually make all these agents out  there work for regulated industries that we are [5:48] that we are all in here. Good afternoon everyone.  My name is Maz Mahmud. I work for first citizens [5:55] as enterprise uh data management. My focus  is on operationalizing governance to enable [6:01] modern data platform and AI at scale. For me the  critical issue I see is the runtime governance. [6:09] Um we have generative AI and agentic systems.  We want to make sure that uh those systems are [6:16] accessing authorized u um sources as well as um  monitored continuously as it produce responses. [6:28] Okay. So uh let's talk about some examples some  examples of u uh AI governance framework or AI [6:36] um risk management tools. um what should you  would you recommend um a banking institution [6:44] um uh focus on on on developing today?  Uh maybe uh Mike since you you are in the [6:51] uh vendor space maybe you could give us your your  your point of view on that. Yeah. Um would love to [6:58] I think also things have changed there a bit over  the last you know couple of years. Because I think [7:02] when we started with the first like versions  of ML models, it was all about explanability, [7:07] you know, does the model actually work? Because  models were only good at certain task because [7:12] like they were not that flexible. I think now  with LLMs coming into the game, everything is [7:17] changing and one needs to look at like the risk  governance frameworks from much more holistic [7:21] space. So for like we asking first the questions  where is AI actually needed during the process and [7:26] where it's not needed because like I don't want  to like um only think about the only piece of AI. [7:30] I want to build a harness that actually thinks  about it end to end. As an example, if you're [7:34] a bank and you onboard um a small business, um do  you need AI at each step? Like I think there are [7:40] great agents that can do part of the work that the  human underwriters are doing like the KYB check, [7:45] like the financial spreading, but then some part  is a credit policy where potentially don't even [7:50] want to like have an LLM in place. I want to  have a deterministic part in place. So I think [7:55] our expectations towards kind of like the risk  and governance frameworks but also functions [7:59] is also giving guidance on which part of the use  cases you actually want to use AI and where not. [8:06] So um I think that's um that's an important  an important piece for us to look at it end [8:10] to end from a process perspective. William,  do you want to add something? Yeah. um we're [8:17] still in the middle of building out a framework  uh whether it's policy or technology platform [8:23] but I've seen there's a few themes that the  business and control functions gravitate toward [8:29] uh one is this uh risk based governance right not  it's not a one-sizefits all so not every gener use [8:36] cases have create the same level of risk um a  productivity tool does not need to be governed [8:43] the same way as a AI solution that would impact  investment decisions. So we need to apply the uh [8:49] the right level of governance that's proportionate  to the materiality and the complexity of solution. [8:55] So we ensure that you know this is uh uh we we  don't impact any low risk lower risk innovations. [9:05] Okay. So I'm sorry I there's two more and I just  quickly go through that. My apologies. Um the [9:12] other two um I think that we want to is focusing  on the governance um of genai as a life cycle. [9:19] uh a point in time is important but you know it's  not just about the testing and validation of pre [9:26] uh pre-eployment but it's also about ongoing  performance monitoring and the evolution of [9:30] Jenna use case and the last is accessibil um  accountability and traceability right I think it's [9:36] very important that uh we can evidence the control  to our senior managers uh senior management and um [9:42] regulators if needed thanks okay um next question  is about um you know AI blackbox uh especially uh [9:54] uh with this regulations that are coming up. Uh  so how do uh financial institution translate those [10:02] um those AI blackbox uh uh blackbox models uh  into into explainable outcome and especially [10:11] outcome that can be explained to to clients uh to  to regulators and and also to the board. Right. [10:20] So mas you want to take that one? Of course. Great  question. Um so with generative AI and and agentic [10:28] systems explanability really has to go beyond the  model itself. Uh we need to have visibility into [10:34] what sources retrieved what policies applied and  whether or not human approval were required. Um to [10:41] me trust really comes down to three main things.  Uh grounded evidence, recorded runtime controls [10:49] and preserved uh decisions record. also need to  uh I think consider um the audience. So we talked [10:57] about regulators and customers not necessarily  both the need to receive the same answer probably [11:04] would create confusion more than trust. To me  customer uh you know explanability should focus [11:10] more on uh what documents used and also we need to  provide the customer ability to provide feedback [11:17] uh from his perspective if uh there is any concern  on the respond. On the other front for regulators [11:25] um explanability has really to go deeper  than that including model description [11:31] um or um uh the controls applied uh performance  monitoring or you know exception uh process. [11:40] Um so again explanability to me is more about uh  providing the right context to the right audience. [11:50] Brendan, sure. Um, so I would take I I, you know,  I think you bring some very good points. Um, [11:57] I would take a step back, you know, and really  start with the use case of of your models and [12:03] really assessing the risk of those models.  I think you need to start with the point of [12:07] understanding you know what level of transparency  do you need for your models and that may drive [12:13] your decisions if you use a vended solution if  you determine hey I need to build this model [12:18] internally or some combination approach maybe  you working closely with a vendor to help build [12:24] your internal solution but you really need to  understand the risk associated with the process [12:28] to understand what level transparency you need so  if you're really working on a critical business [12:34] process that you really need to fully explain the  outputs you may that may lead itself more towards [12:40] developing internally andor partnering with a  vendor for for transparency versus going to a [12:46] complete blackbox solution. Um now with that said  there are methods for trying to understand the [12:53] explanability of models even if it's limited  in terms of understanding the features or the [12:57] underpinnings of the model. Um there you you can  require um that some explanability related uh [13:04] metrics are included as a part of the development  process and the ongoing monitoring process as [13:10] well. Um you can also perform some sensitivity  testing to try to get some insight into the [13:15] feature importance and which features are driving  the decisions and the outputs as well as trying [13:21] to benchmark your outputs against the internal  processes that you have in place or or existing [13:27] qualitative processes. So those are some some  some options but I would really emphasize that [13:33] you know you should really think about hey as a  part of your decision to either build your models [13:38] internally or work with vendors or in inclusion as  a part of your vendor selection process what level [13:45] of transparency do you need for the risk that's  associated with the use cases that the models [13:49] you'll be you'll be building and using. Thank you  William. Yeah. Um I think those are great points [13:57] by Mhas and Brandon. Um I would emphasize on the  point of transparency, right? We need to know what [14:04] is the use case, what are the inputs and outputs,  uh what are the limitations and controls in place, [14:11] whether it is testing, whether it's ongoing  performance monitoring or change management. [14:15] Those are really important to help bring the  understanding to stakeholders whether it's senior [14:20] management or to your users to understand um and  really trust what's coming out of the Geni systems [14:28] and this also allow them to really challenge it  if necessary. Okay, thank you. uh would like to [14:38] speak a little bit about uh responsible AI and  in particular uh when it comes to non-technical [14:45] part of the organization. Um uh so uh what in  your opinion is a good approach to uh to have [14:56] this uh shared responsibility about a AI risk?  Brandon, do you want to elaborate a little bit on [15:03] that? Sure. Sure. I'll kick it off on this one.  Uh I think this is a really important question. [15:08] Um I think people are a very important part of  the process. Um everybody at your organizations [15:14] across the board are not quantitative folks or  quants in short. So you you have some nontechnical [15:21] folks that really have expertise in different uh  um business processes across the organization. [15:29] So I think you you really need to uh focus on the  importance of the people and and I I think there's [15:35] three major things that you that can contribute  to fostering a culture of responsible AI. [15:41] uh one being training developing an ongoing  training [music] for your community and for [15:47] your uh organization and and especially for  non-technical folks so that they understand [15:54] um AI use cases and they understand um what's  expected them of them from a responsibility [16:01] standpoint defining what responsible use looks  like um across the organization um building out [16:07] your policies and procedures to clearly define  that and establishing that training program that's [16:12] ongoing. Um second I I think um establishing the  appropriate incentives um for your business is [16:20] very important and encouraging feedback from  these nontechnical folks because they may have [16:26] expertise or years of expertise in a business  process that you're trying to incorporate AI in [16:32] and if they can provide feedback to that process  it can only help improve um the AI incorporation [16:38] going forward. And last but not least, um really  incorporating accountability, an accountability [16:45] framework into your business. And so really  establishing ownership across the use cases and [16:51] really establishing roles and responsibilities,  who's responsible for what. Because when you have [16:57] a risk event that happens, you want to make sure  that you have the the proper stakeholders on point [17:03] and that you have the appropriate escalation  pathways to escalate those risk as needed. [17:09] But you have to have ownership and roles and  responsibilities clearly defined in order to [17:14] do so. Yeah, I think I couldn't [clears throat]  agree more on that point. And I think like if you [17:18] think about it like why is that also important  in 2026? It's because like until 2 three years [17:24] ago you had to be very technical to do anything  with AI. It was very like I remember 201617 when [17:29] you wanted to build your first ML model it was  so difficult like you had to be a statistician [17:33] or like a data scientist to do anything. Now like  all of us in the room can use cl co-work and do [17:39] something cool and useful with AI. It became so  much easier to make use of AI which of course on [17:45] the other hand for an organization brings a big  challenge because like all of a sudden you don't [17:49] only have like the small group of technical people  that you build a governance framework for. All of [17:54] a sudden it's like 50,000 100,000 people depending  on how many people work in the bank that you all [17:58] need to put into one framework. However, it's the  right thing to do because like who's got all the [18:04] domain knowledge? It's the people who work in  credit. It's who the people work in compliance [18:08] and in risk. So I think what we see with many of  our customers in the big banks, they they want [18:14] to have, you know, the agents to be owned by the  business, but they don't know how to do it. And [18:19] historically, every production system was owned  by it, but it doesn't make any sense anymore. And [18:24] the reason is that what are the agents are doing  now? They're doing what humans were doing before. [18:29] As an example, if you're the head of credit  at a big bank and you wanted to change a new [18:34] underwriting rule on a Monday, historically, you  went to your underwriting team and said, "We're [18:38] not going to give loans anymore to small business  owners in Minnesota that have taxis." But now, [18:43] if the agents would be owned by it, it would need  to make the decision. I cannot talk to my agents [18:48] anymore. So, I need to have like a direct line.  Agents need to be owned by the business itself. [18:53] I'm deeply convinced of that. But if that's  the truth, then you also need to make sure that [18:57] governance frameworks go into the last corner um  of the of the business units and that's that's the [19:03] drastic change I think in 2026 from from before.  Yeah, I can just add another point. You know, [19:11] I think Brandon well said everything that we  are trying to do at the same time and for the [19:15] reasons that Mike had point out. Um you know,  responsible AI does not really sit with risk, [19:21] legal and compliance alone. We need to bring  in our business stakeholders to share their [19:26] responsibilities and have them understand you know  what could be the negative impact from a misuse of [19:31] this genai or agent right we need to ensure to  let them know that compliance uh we're not just [19:38] increasing the compliance overhead we're trying  to help them mitigate the risk help them to avoid [19:43] um errors or hallucination right because I  think innovation can be built continuously [19:49] on the success of pride cases so we have one  that um um negative it will set back a lot more [19:56] um in terms of progress. Okay. So going back  to this um compliance playbook uh and with [20:05] this you know regulation that's evolving all  the time how do you define the balance between [20:11] um um between innovation the pace of innovation  and um the need to to to remain in compliance. uh [20:22] and uh to remain in compliance without slowing the  pace of innovation. Um what else do you want to [20:29] take that? The key is to structure compliance as  a set of reusable controls. In my view uh rather [20:36] than working with each use case um separately as  a separate exercise, we start with a risk tier [20:44] model or framework. Um low tier use cases um go to  the lane where uh lighter controls uh where higher [20:56] risk use cases uh customer impacting um should  um I would say um um be more into the um heavier [21:08] oversight lane. Um also um organizations may  uh consider invest into establishing a reusable [21:17] controls in a shared AI platform. Uh that includes  policy as a code reusable workflows that not only [21:25] achieve consistency but also enable product team  to deliver faster as they are more aware of the [21:32] boundaries that they are executing at. Another  benefit is when regulators hit us with a change, [21:39] organizations will have ability to not only  update you know each use case separately but [21:45] through the horizontal layer of compliance  controls we can update multiple use cases [21:51] at once. Uh so really compliance can become  enabler for innovation. Uh and one way to do [21:58] it is reusable controls. Uh again sits as  a horizontal throughout the organization. [22:06] William, you want to add something? Yeah. So, um I  think the you know I think everything with said is [22:15] also um something that we're looking to implement.  So I feel like working with the good guys for many [22:19] years now. Um but the the requirement for for um  genite governance is going to be different by uh [22:28] different jurisdictions, right? So you know one of  the ways we can think about streamlining this is [22:34] um you know we will have different policies  set by different control groups based on the [22:38] need and the regulatory requirements. Uh what  we can help the business to streamline that [22:43] is also building a procedures that help  unify the compliance or the policies. [22:49] um help helping the business to understand this  a set of process that where they can do to ensure [22:54] compliance right without them having really  in-depth knowledge to all the different policies [22:59] of EU AIS. So hopefully that layer uh will help  really um make deployment of AI much quicker. [23:09] Okay. So let's look uh let's look forward and  let's uh let's uh look at um the you know the [23:15] ideal operationalized um um agentic uh generative  and agentic um um um framework uh governance [23:29] framework. So uh could you u draw us a picture  of how it would look like in uh you know one to [23:38] three years from now and I know it's uh it's not  an easy question because technology is evolving so [23:43] fast so it's hard to keep pace with with the  changes now every six months you know things [23:48] are are completely uh u renewed um but um do you  have uh do you have some sort of uh um image or or [24:01] structure that you see going forward uh in the you  know next uh one to three years. Mike, do you want [24:07] to Yeah, I mean like what we I think even start  to see now with the um organizations that are like [24:13] you know very very tech forward. Um they look at  it from a um like a hub and spoke model. It goes [24:19] back to the kind of like the point I made earlier  that I really believe that the business should [24:22] own a big part of the AI you know which is kind  of like the spoke uh part of the thing and then [24:28] you have the hub which is more the central layer  you know some of you you know would probably fall [24:31] into that category of your day-to-day job but  you have like this hub that's actually giving [24:35] the right governance and frameworks and then  the different units they can still make their [24:40] completely independent own decisions and I think  it's important because what's even happening right [24:45] now is the speed of development of models is  so fast right Now the models they depreciate [24:50] quicker than more like faster than the head  of cabbage like every kind of like two months [24:54] three months there's a new model and it's it's  so much better then open source models come out [24:59] and they're like cheaper so it's not only how  good are the models also like how cheap are the [25:02] models and then the question is talking about  Europe like as well like it's not a big problem [25:06] in the US but in Europe we have a lot of like  scares over export control so like dependency [25:10] on models so there's a lot of complexity right now  which is not only about I train my own model as a [25:15] bank and I own it I'm dependent on the big labs  about the models they're bringing out. So a lot [25:20] of complexity will increase with the models over  time and I think as a bank you don't want to be [25:25] dependent on anything. So I think you just want to  make sure uh that you have a lot of those things [25:30] under control but still give enough freedom and  innovation to your teams by letting them do what [25:36] they want because I believe innovation doesn't  only come from the central unit. It comes from [25:39] the business units because they know what they  should do on a day-to-day basis but they need [25:43] to be like [clears throat] embedded into this  complexity in which we're running right now into [25:47] the world. So that's why I think the hub and spoke  model is what I see like best of class companies [25:52] um implementing at the moment. Brendan, sure. Um  what I can see you know in the next few years is [26:02] um AI agents being incorporated in each part of  the risk management process. So if you break down [26:08] the the risk management cycle from identification  all the way to oversight and ongoing monitoring [26:15] and reporting, I can see agents being involved  in helping risk professionals at each stage of [26:21] those cycles. So if you start with identification  and inventory management, I can see agents have [26:26] helping risk professionals manage the inventory,  help improve the data quality of the inventory, [26:32] help scan, you know, the the um the applications  and and different systems for potential AI systems [26:40] that are out there for classification. Um  you know, when that's done on a manual basis, [26:46] that's somebody raising their hand saying, "Hey, I  I think I got a AI system. I think I got a model. [26:51] I'm gonna submit it. But but everybody's not so  forthcoming, right? So your AI agents can help [26:57] scan across your your landscape to see what's out  there and help identify and help improve the data [27:04] quality of your inventory. And if you transition  to the oversight components, if you think about [27:09] like model validation for example, um most  organizations when they have validations, they [27:15] have an analyst that perform those validations on  a singular basis, model by model. They perform all [27:21] the activities from evaluating the conceptual  soundness, the data, replication, on and on and [27:28] on, right? But if you had an agent to help you out  with some of those tasks, that would be awesome. [27:33] And also, if you think about even enhancing  that further, um, if you had agents that can [27:38] talk across, you know, a portfolio of models.  So perhaps you're not limited to just one model [27:44] at a time. you can look at the landscape of your  models for different use cases across a portfolio. [27:51] So now you can evaluate if you have maybe some  thematic issues versus you know a siloed issue [27:58] that you may see. Let's say I had a feature that  that was problematic in in model A but in model [28:04] B over here. Well that same feature exists and  and and quite frankly it exists across a number [28:10] of models. So you could have the agents raise that  up to the analysts and they can and can flag those [28:15] issues. Then if you go a step further for risk  uh risk management and issue identification, you [28:22] can have agents help draft issues, help identify  issues, help identify thematic issues and help [28:28] folks even draft remediation plans. A lot of that  is done manually today and takes a whole lot of [28:35] time from a whole lot of different folks that are  very very bright and intelligent. But perhaps they [28:40] can focus their energy on really understanding the  results and really understanding what are we going [28:46] to do and what decisions are we going to make. And  then going on to ongoing monitoring, you can help [28:51] a have agents help with the ongoing monitoring,  the execution and and and raising if you have like [28:57] uh threshold breaches across your models. And  then last but not least, the reporting aspect. [29:03] Um reporting is a key element that that's done  quite manually in a lot of organizations. you can [29:08] have agents help compute that reporting whether  it's KRIS whether it's uh developing reporting [29:14] for committee reporting and on and on and on.  So I just see from end to end agents helping [29:21] with each part of the risk management life cycle  and um really helping our risk professionals get [29:28] get better and and understand be more informed on  their decision- making. That's maybe an important [29:33] point just to like give a real world example  to what you said like for the audience like [29:37] we some of our customers by now have 1200 agents  in production like if humans need to monitor when [29:44] they're being updated it's almost impossible. So I  think like the need that you're like calling for I [29:48] think is very important and we had this thing  like andropic was bringing out a new model so [29:52] like the kind of like central team was updating  the model to the agents and you would normally [29:56] think a newer model which is more accurate helps  all the agents but which is not true. sometimes [30:01] just the architecture is so different from the  model that the agents don't really work anymore [30:04] and some of the 1200 agents and you cannot as a  human oversee all of them just worked like went [30:09] from like 90% to 60% accuracy but it took like a  couple of days until people like saw that I think [30:14] if there had been an agent in place that would  monitor that would have really helped so think [30:18] about the pure extent of number of agents that big  banks will use it's going to be in the thousands [30:23] so you need a better system to monitor them than  before yeah what about changes in um you know the [30:31] existing infrastru infrastructure when you have u  um upgrades when you change versions of the LLMs [30:37] um do you have uh you know again real world  examples of um uh of the uh impact of MRM [30:49] um and uh of the indification of syn issue before  um before an agentic system goes into production [31:00] um like how to how to how to best test for it.  I think like I mean what the difficulty about [31:07] like the LLMs is that they're so ready to made  and pre-trained. People assume for many angles [31:14] like I can just I can just use them but I think  the rights and benchmarks for your use case is [31:21] something which you know people very much um under  underestimate. They look at EVAs and benchmarks [31:26] meaning you know how well does a model perform on  a you know a broad range or basket of tasks but [31:32] I think then what what's important for the bank  is like for each use case that you actually like [31:36] start to put it on you need to like you need  to test it on the data and the use case that [31:41] you're actually putting it on and I think that  is that needs to be kind of like a responsibility [31:46] also of the central of the central unit that we  were talking on before I think like we've seen [31:50] um we've seen a lot of our customers like failing  in the beginning not because the models are not [31:55] good enough just because the use case was not  applicable towards like how they were tested for [32:00] uh before. I think that that I do see as a as  a part of MRI. So everything to do with sort of [32:06] regression test making sure that it's one one one  example. Yeah. But I mean like creating creating [32:11] data to like for testing is it's become quite easy  now with AI as well. Historically like having the [32:16] right data sets for testing was just so cumbersome  and expensive but you can just create many of them [32:21] by now yourself artificially. So a lot of things  you can do nowadays if you do it right. Okay. Um [32:31] so in banks uh we work a lot with uh third party  vendors and um uh so we use their tools we use [32:41] the that platform and so we we are constantly  negotiating with them what uh you know what [32:48] should be in the contract and who is responsible  for what. Um what are the non-negotiable uh items [32:55] that you want to see in a in a relationship with  a vendor? Martaz in your position as a in a bank. [33:02] Uh absolutely. Yeah. Uh I would say first uh the  bank has to remain in control in this engagement. [33:10] Uh yeah. Um I was going to say uh when we decide  to go with off-the-shelf AI platforms that doesn't [33:17] mean we are outsourcing governance. So governance  has to be a continuous owned process by the bank [33:24] and the non-negotiables of course our data. So  our data has to be um you know we have uh the [33:32] bank has to be fully aware of how the data is  used uh retained and protected. Um also from [33:40] um um I would say uh the vendor side any changes  to the model has to be crystal clear communicated [33:48] to the bank. we have to have full visibility  as especially the ones that may introduce [33:53] uh risk. Um so again bottom line um governance um  doesn't end when the vendor deliver the solution [34:02] to the bank. It should be a continuous uh process  continuous uh collaboration between the bank and [34:09] the vendor and do you feel you get that from the  vendors and if you take the frontier models the [34:17] I mean I don't want to mention any name but uh  yeah that's a good question right now we are in [34:22] a lot of discussions with many vendors this is in  the center of the conversation non-negotiables uh [34:27] our risk and control team and and governance teams  are not going to let those things slide they work [34:34] very closely with uh the AI teams and architects  uh throughout the evaluation. Okay. William you [34:42] want to Yeah. So um we have a lot of critical  dependencies on vendors and third parties. So [34:49] the general AI governance applies to both whether  it's internally built or uh externally provided. [34:57] um we would apply the same governance. Um as as  we have engaged with different vendors, we come [35:02] up with a few non-negotiable um you know that we  we each ask each of the vendor to provide. One [35:10] um is as as said data privacy and protection,  right? We start at the very top level with a [35:16] service level agreement with the vendor ensuring  that they agree to what we need from them before [35:21] we engage with them. Um second is to ensure they  have robust uh security controls right uh security [35:28] architecture review is very important uh as  smoke test point out also uh the transparency [35:34] uh we need the vendor to really um uh commit to  communicating with us any changes material changes [35:42] to their underlying LLM or geni capabilities.  Uh and then the last part is really testing and [35:49] ongoing performance monitoring. uh we could do  that ourselves but a lot of times we expect the [35:54] vendor to be testing anyway and we could leverage  some of the results that they provided to ensure [35:59] that we have uh strong evidence that the um  capabilities that they they provided are working [36:07] that also would be if I met add one thing from  the vendor side that the last point I think is [36:13] uh what I would recommend to every bank to do you  know don't pay anything until they have not shown [36:18] that it works I think the last one like you I mean  like of Of course it's better as a vendor you get [36:22] a big contract in the first day but I think with  AI many of the things you can prove that there's [36:27] a business case but let them you know show the  show the value and once it's shown then you can [36:33] start to actually pay something. So the last  point I consider to be very but not many people [36:37] ask for it because it's yeah sounds a bit well it  depends as a bank you know [laughter] but I would [36:42] just do it. Yeah. Depends how you use the LLM  too, right? If you use it right, it's going to [36:46] produce something. If you don't, then Well, but  that's on the vendor side to produce something [36:50] useful then. Yeah. Okay. So, um you know, for  our audience, uh there has been some uh echo in [36:58] the press recently about OpenAI. Um they have uh  they had uh the this model that apparently escaped [37:06] uh uh the sandbox and went into ugging phase  seeking for the test that would evaluate its [37:12] performance basically to cheat on the test.  Uh so that was quite interesting to see that [37:19] um uh Antropic the I mean Antropic had you know  a lot in the press recently but uh I will mention [37:26] also uh one of the of the latest version that u  you know escaped the the sandbox and apparently [37:35] uh moved into or invaded I don't know what's  the proper word three companies two of which had [37:43] uh no idea it had occurred And so they, you know,  they had the system in their in in the network, [37:48] but they didn't know about it. So, uh, have  you, have you heard of that? I mean, do you [37:53] have an opinion about this? I mean, is there um  is there a way to to prevent this kind of of risk, [38:01] you know, to put it into either the um uh you  know, the AI governance framework or you know, [38:09] some very specific tool um you know, some cyber  security tool. What what do you think of that? [38:14] How can we avoid these kind of things both on  the vendor side and on the client side or uging [38:22] face which in this case could be seen as a  as a client although it's really a vendor. [38:29] I mean one comment from my side like that that was  very much on the on the like security side of the [38:34] house like where I wouldn't say I'm the deepest  expert in but I think like the same thing could [38:38] happen within a financial institution with the  agents that we are building and I think the trick [38:41] there is the right harness meaning you know also  like how much context do you give to an agent you [38:46] know how much memory you do you give to an agent  like you can you can make an what happened there [38:51] is like they really wanted to provoke it you  know like they let they give the agent all the [38:54] resources they can do whatever they like and  then stuff can happen I think this if you're [38:58] like a big bank you would like this actually could  actually never happen the way that you know these [39:03] governance frameworks are set up. You don't have  unlimited resources that you give to an agent. So [39:08] I think I at least I have not seen whoops agents  in production at banks that could like do so much [39:13] harm. I think of course it could always come from  the outside of attacking but that's now we're in [39:17] the cyber security world which you know I don't  I don't think we well they're constantly under [39:21] attack. So with with the LLM and the this ability  to uh to to test uh um the resilience of the the [39:32] firewalls and the antiviruses um 24/7 uh they  they're just testing you know thousands or tens [39:39] of thousands or millions of of cases more cases  than they used to when it was done by humans. It's [39:44] uh for sure I mean we do that on a day-to-day  basis. we let ourselves attack by agents dayto day [39:49] and see if they can get in but so far they have  not that's good so I I think I can give a comment [39:56] to that um as I said before responsible AI does  not just seek and risk legal and compliance we [40:02] work closely with our stakeholders that includes  IT and technology so some of these controls in [40:07] place while we have the governance framework the  technology partner is very important in terms of [40:12] establishing the platform and the capabilities  and the guardrail so Um while conceptually we [40:18] established the framework the technology I have  in place uh in terms of monitoring the agents [40:24] and monitoring uh what models they hit uh what  are the input and data so we work hand in hand [40:29] to ensure that something as you mentioned doesn't  happen within the bank we have two minutes left [40:34] we anybody has a a question you'd like to ask I  think there's a question over there I'm not a risk [40:40] professional but I'm going to ask a very general  question just like human we have left brain right [40:45] brain and and you have the generative behavior of  the models and you want to have a deterministic [40:52] results. Um how do you guard the OM with your rule  based kind of a system you developed years ago? [41:02] Um is that a good combinations or you just take  one side and abandon the other? Maybe uh Mike you [41:11] want to talk about your three level? Yeah, it's  I mean it's you need to combine both in financial [41:17] services. I think like you know letting especially  in the risk world where we talk about compliance [41:21] and credit you want to combine the stoastic nature  of agents with the deterministic frameworks and it [41:27] has two reasons. The one is the regulator wants  you to have certain things in rules which is for [41:33] example a credit policy an AML policy so it has  a regular regulatory component but it also has a [41:38] cost component. you know, a lot of the things are  so much cheaper and faster in rules and in code [41:45] than they actually in agents like and they're like  just spending tokens crazy on like identifying [41:49] everything just doesn't make any sense for some  way. So I think like yeah two reasons cost reasons [41:55] and regulatory reasons and that's why you need  you will also need in the next years it doesn't [42:00] matter how good the models get you will always  need the rules uh or the deterministic framework [42:05] around the agents I'm deeply convinced of that.  Yeah. Yeah. Very good. If you allow me, I can [42:10] ask a quick part two question to implement that.  Do you do it at application level or data level? [42:20] The data level for definitely at the at the  application level. Um I'm not 100% sure how you [42:27] can have that. I think it depends on the use case,  right? Um you know there are some use case where [42:32] you can do double entry uh validation. So uh it's  close to deterministic there's still a a risk of [42:40] error but uh the error is is very small. Um if uh  if it's about so double entry is going to be for [42:50] example reconciliation. Mhm. Uh but if you do uh  things that are more open-ended like uh you know [42:58] reading a 10K uh or or uh reading regulation and  extracting the essence or being able to query the [43:06] regulation in a in a rack type framework then um  you know there's still there's always a a residual [43:13] error because an LLM is is uh is a regressive  model right it's it's only predicting the the next [43:21] word. So we're talking about LLM here which are  not the only AI solutions, right? You have other [43:27] types of AI systems, you have world models for  example, we have a different approach. Um you have [43:34] um have all the classic machine learning where you  can have a level of accuracy. You can define the [43:41] level of accuracy. You know what is my level of  error? So if I want to be, you know, 10 minus 10 [43:47] um uh accuracy, you know, I'm like in the aviation  industry. So it's as safe as taking a plane. Does [43:53] it mean there's never going to be an error?  Probably not. But it it might be extremely [43:58] rare. Yeah. So I think it depends on the use  case. You know, some use case are prone to, [44:04] you know, residual errors and others may be  able to to be, you know, the outcome may be [44:10] able to be validated without um without uh  u uncertainty. Um there there are techniques [44:21] though to uh to validate LLMs. Um Wells Fargo for  example has published a lot on on in that space. [44:31] So uh uh the problem in validating LLMs is that  uh your uh independent variable you know the the [44:39] input by the user into into the prompt I'm not  talking gent I'm talking more classic generative [44:44] AI um is infinite. So um since you output text  uh you know how do you how can you guarantee that [44:56] the completion is aligned with the intent? Mhm.  Therefore that you have a consistency between the [45:04] uh the prompt and the completion. So there are  techniques I mean it's it's a big topic so I'd [45:11] rather not get in we're beyond our time anyway but  you know if you want we can talk about it. Give.