Mastercard, MetLife AI Leaders Urge Technical Validation Over Governance Committees

万事达卡与大都会人寿AI负责人呼吁：重视技术验证而非治理委员会 若缺乏嵌入式红队测试流程与因果风险测试，企业AI治理便难以奏效，未经验证的模型也将转化为企业产品责任风险。 小组讨论探讨了将AI治理从抽象的高层原则转变为可复现、可操作的工程工作流的紧迫性。与会专家强调，治理应当直接嵌入现有的产品设计流程与企业风险管理中，从而作为生产力倍增器和投资回报率驱动因素，而非成为企业的瓶颈。对话还涵盖了用于评估非确定性及多智能体AI系统的技术框架、通过红队测试验证黑盒第三方供应商的策略，以及通过降低产品责任风险和加快创新来衡量治理投资回报率的方法。 AI 治理已成为平衡创新竞争与维系客户信任的当务之急，目前的核心挑战在于高层伦理原则与日常可审计的工程工作流之间存在执行鸿沟。 (0:04) 许多企业因仅拥有书面政策或委员会而陷入“治理假象”，在向概率型及代理型（Agentic）AI 转型的过程中，领导者必须从被动规避风险转向主动进行系统架构设计。 (6:21) 有效的治理应直接嵌入现有产品开发流程与企业风险管理中，早期引入伦理审查可形成治理飞轮，通过标准化的绿色通道加速产品交付并降低缺陷率与法律风险。 (12:08) 管理第三方供应商及多 Agent 系统需要超越传统的合规清单，依托动态红队测试、因果推断以及对单元逻辑、多 Agent 交互和系统涌现行为的多层次验证。 (26:23)

AI governance is shifting from abstract ethical principles and isolated oversight committees to embedded engineering pipelines and risk-proportionate technical validation. Panelists argue that integrating ethical constraints into daily development and enterprise risk management mitigates catastrophic liabilities while accelerating commercial delivery in probabilistic and agentic systems. Closing the Ethical Implementation Gap (2:48) Panelists identify a persistent disconnect between executive ethical statements and everyday technical execution. Cansu Canca critiques organizations that rely on written policies and governance boards without operational mechanisms, warning that companies remain unprepared until public scandals emerge. Canca argues that defining fairness is fundamentally a philosophical task that must precede algorithmic optimization, noting that unaddressed proxy variables and automated systems frequently cause mundane but severe harms, such as denying insurance coverage or misreading healthcare risk indicators. Rajesh Ramaswamy emphasizes this execution gap in high-stakes environments like options market-making and novel drug discovery at Sail Biomedicines, where the cost of model error is near infinite. Ramaswamy contends that high-level principles are ineffective unless translated into repeatable, auditable engineering workflows. JoAnn Stonier observes that while principles establish a vital common language across legal, technical, and business domains, standalone governance councils create friction unless governance is anchored directly in corporate values and operational processes. Embedding Governance Pipelines into Product Lifecycles (18:41) Rather than relying on isolated oversight bodies that impede progress, the panel advocates integrating governance checks directly into product development and risk management workflows. Canca outlines multi-stage pipelines where internal builds and vendor procurements pass through structured checkpoints to evaluate explainability, algorithmic bias, safety constraints, and sector risk. Upskilling internal engineering teams to utilize verification tools creates verifiable audit trails and aligns risk-benefit evaluations with commercial profitability. Stonier illustrates how this integration generates a self-reinforcing governance flywheel. At Mastercard, clear baseline protocols established standardized "green paths" that accelerate deployments for pre-approved vendors and repeatable processes, allowing governance teams to focus on edge cases. Deployments involving sensitive data or novel technologies are flagged as "code purple" for intensive review. Factoring constraints into product design from inception prevents expensive late-stage redesigns, transforming governance into an efficiency multiplier that builds long-term brand trust. Technical Validation for Non-Deterministic and Agentic AI (28:11) Traditional software validation relies on deterministic logic that fails when applied to probabilistic systems. Stonier notes that corporate leaders accustomed to predictable outcomes must learn to navigate probabilistic and agentic architectures. Ramaswamy emphasizes that multi-agent systems require a three-tiered testing structure: unit-level evaluations of individual agent prompts and bounds, interaction testing to detect unintended emergent behaviors across dynamic handoffs, and system-level simulations. These simulations must quantitatively measure both the frequency and severity of failure modes. The panel cautions that third-party vendor management cannot rely on procurement checklists or compliance questionnaires, which provide false reassurance without verifying actual safety. Technical oversight requires active adversarial red teaming, stress testing, and continuous runtime monitoring. Stonier adds that as autonomous agents begin acting on behalf of organizations, enterprises must establish explicit behavioral codes of conduct and operational guardrails comparable to human workforce policies. Quantifying Governance Value and Reducing Product Liability (41:02) The panel challenges the view that governance is merely an administrative cost, presenting concrete frameworks for measuring its commercial return. Ramaswamy advocates an engineering approach to return on investment, suggesting organizations establish a foundational governance nucleus for routine AI tasks and apply causal inference testing to run counterfactual analyses—measuring system performance with protective guardrails against unmitigated baseline states. Canca and Stonier highlight that mature governance accelerates innovation by streamlining approval cycles, reducing manual review headcount, and providing competitive intelligence through pre-deployment audits. Tracking competitor litigation and market backlash allows proactive organizations to quantify the legal and reputational liabilities they successfully avoid. Stonier frames responsible AI as a defect-reduction discipline that directly protects customer retention, feature delivery speed, and market share by insulating the enterprise against future product liability. Funda Gunes concludes that effective AI governance requires rigorous testing protocols, dedicated financial investment, organization-wide AI literacy, and interdisciplinary collaboration. The panel emphasizes that sustainable accountability depends both on internal practitioners proactively questioning systemic failure modes and on external consumers demanding transparency, prompting organizations to embed safety directly into the architectural foundation of their AI systems.

在Ai4研讨会上，Funda Gunes、Rajesh Ramaswamy、Cansu Canca与JoAnn Stonier等专家指出，人工智能治理已不再是可选的合规负担，而是企业规避灾难性责任、保障公众信任与实现持续创新的核心业务能力；企业亟需将宏观伦理原则转化为嵌入日常研发与风险控制流程的可审计操作机制。 破除治理幻象：落地可审计工程 (2:48) Cansu Canca指出，许多企业误以为拥有书面伦理政策或专门委员会便算建立了AI治理，但若缺乏底层操作机制，往往直至公共丑闻爆发才暴露风险；她强调界定“何为公正与正确”本质上属于哲学任务，必须先于算法优化开展，而算法偏见对现实生活的冲击往往体现在保险拒保、血氧读数偏差等日常场景中。Geoffrey Hinton关于AI安全行动滞后的预警同样揭示了行业紧迫性。 Rajesh Ramaswamy结合金融市场期权做市与Sail Biomedicines的新型CAR-T细胞药物研发等高风险领域经验表明，在决策失败代价“近乎无限”的行业中，治理的核心阻碍在于如何跨越“执行鸿沟”，把抽象崇高的道德共识转化为技术团队日常可执行、可重复且可审计的工程操作规范。 嵌入研发流程：构建飞轮机制 (12:08) JoAnn Stonier与Ramaswamy一致认为，设立孤立的AI治理委员会容易引发跨部门摩擦并延误产品上线，成功的治理必须直接嵌入现有的产品研发管线与企业风险管理工作流。Stonier以Mastercard的实践阐述了治理的“飞轮效应”：组织在前期确立规范后，研发团队即可开展合规自评；通过设立“绿色通道”（Green Path）加速常规创新与预审供应商交付，而对采用前沿架构或敏感数据的项目打上“紫色代码”（Code Purple）标签实施重点审查。 Canca进一步提出了分阶段的治理管线模型，主张对内部跨职能团队赋能评估工具，使自主研发与第三方采购均历经偏见检测、可解释性审查与安全门禁，以此建立完备的审计轨迹，彻底打破“伦理与商业盈利相对立”的认知误区。 应对非确定性：验证第三方与智能体 (26:23) 伴随系统从确定性逻辑转向具有概率性与自主性的生成式与Agentic AI，传统的问卷调查与静态法律清单已无法证明系统的实际安全性。多位专家指出，外部供应商交付的往往是黑盒模型，企业内部若缺乏具备深层追问能力的工程人才，采购防线将形同虚设；企业必须主动开展实证压力测试、对抗性红队测试以及实时运行监控。 Ramaswamy提出，针对复杂的多智能体架构，技术验证必须覆盖三个层次：单体Agent的提示词与安全边界测试、多Agent间的动态交互与通信协议评估，以及系统级的应急模拟测试，通过量化故障发生频率与危害严重程度来设定边界。Stonier同时强调，面向能够自主执行操作的AI代理，企业必须为其确立类似于人类员工守则的行为规范与权限边界。 量化商业回报：缺陷控制与因果推断 (35:48) 针对治理投资回报率（ROI）的衡量难题，Stonier指出，AI治理本质上是降低产品责任风险的质量控制手段，通过消除模型缺陷率直接赋能市场份额提升、用户满意度与客户留存；治理体系必须深度锚定企业核心价值，例如Mastercard将安全性视为不可动摇的底线，一旦网络安全失信便会导致商业模式瓦解。 Canca提出，治理能够通过精简审批流程与减少后期人工复核来直接加快创新交付速度，同时对自研与外购系统的主动审查还能提供关键的竞品洞察，帮助企业规避因滞后诉讼与声誉受损带来的巨大潜在损失。Ramaswamy则建议保持工程化务实思维，企业无需初期搭建过于庞杂的架构，可先围绕传统AI建立治理核心，进而利用因果推断与反事实分析（Counterfactual Testing）来精准测算治理干预相较于未受控基线所实现的损失减少与业务增益。 Funda Gunes在总结中指出，有效推进AI治理需要企业建立严格的测试机制、落实专项预算、提升组织内部AI素养并确保跨学科利益方参与决策；同时，企业内部的主动质询与外部社区、消费者对透明度的持续诉求，将成为推动技术开发者在架构底层践行安全与问责的核心动力。

## Background 本内容由 Ai4 于 2025 年 8 月 29 日发布在 YouTube 平台上。该圆桌论坛汇集了来自万事达卡（Mastercard）、东北大学（Northeastern University）、大都会人寿（MetLife）以及 Sail Biomedicines 的多位业界与学界专家，围绕人工智能系统中的问责制展开讨论，深入剖析创作者、使用者与监管机构之间的责任分配，以及这一机制如何长远影响社会信任、技术创新与公司治理。 ## TL;DR 圆桌讨论开门见山地指出，AI 治理早已不再是可有可无的合规选项，而是关乎企业生存与产品质量的核心工程实践。来自金融、生物医药与哲学伦理领域的专家们一致认为，治理的关键不是罗列空洞崇高的口号，而是将其转化为嵌入现有产品生命周期的日常工作流与量化测试标准。面对生成式 AI 与多智能体系统（Agentic AI）带来的概率性风险与涌现行为，企业需通过前置的安全设计与“缺陷消减”机制化解第三方黑盒及供应链隐患。最终，健全的 AI 治理不仅能显著规避法律合规风险，更能直接转化为推动创新速度、用户信任与商业长期回报的“正向乘数”。 ## Atmosphere & style 整场圆桌讨论节奏紧凑、求真务实，褪去了许多行业峰会常见的空洞概念炒作，充满了一线工程构建与深度伦理思辨的硬核碰撞。主持人的热情引导与嘉宾们兼具宏观架构与实战落地的剖析形成良性互补，既有对“虚假合规”现象的尖锐针砭，也有以患者生命与万亿交易为尺度的敬畏感。倾听时能深切感受到传统确定性软件思维被彻底打破后的紧迫感，以及嘉宾们试图跨越学科孤岛、在不确定性中为企业与社会铸造安全护栏的坚定责任心。 ## Key takeaways - [00:04](https://www.youtube.com/watch?v=8C7bAlXjRcg&t=4s) AI 治理已不再是可有可无的选项，而是企业的必修课，其核心目的并非应付合规，而是在赢得客户信任的前提下建立可持续的创新能力。 - [03:38](https://www.youtube.com/watch?v=8C7bAlXjRcg&t=218s) 一线开发者面临的最大痛点并非缺少宏大崇高的伦理原则，而是如何将这些抽象原则转化为日常可重复、可审计的工程操作规范。 - [06:21](https://www.youtube.com/watch?v=8C7bAlXjRcg&t=381s) 许多企业在治理上存在“虚假繁荣”，虽设立了伦理委员会但缺乏实际落地机制；现实中的算法风险往往隐蔽而关乎生死，需要从哲学层面清晰界定公平与价值目标。 - [09:08](https://www.youtube.com/watch?v=8C7bAlXjRcg&t=548s) 人类社会正在从确定性计算向概率性决策转型，传统规避风险的领导力模型面临挑战，负责任 AI 与伦理把控成为了推动安全创新的底层支柱。 - [12:47](https://www.youtube.com/watch?v=8C7bAlXjRcg&t=767s) AI 治理的本质是系统化开发、部署、监控和评估系统的手段，必须与企业营收增长或成本控制等实际商业价值直接挂钩。 - [15:37](https://www.youtube.com/watch?v=8C7bAlXjRcg&t=937s) 治理推行成功的关键在于将其无缝嵌入既有的产品研发与风险控制流程中，避免脱离业务实际而“为了治理而治理”。 - [18:27](https://www.youtube.com/watch?v=8C7bAlXjRcg&t=1107s) 通过“游戏化闯关流水线”赋能一线团队自研和外购 AI 的合规归档，打破负责任 AI 与商业成功对立的误区，商业成功本身就是负责任战略的核心考量。 - [22:04](https://www.youtube.com/watch?v=8C7bAlXjRcg&t=1324s) 将行业环境与 AI 工具风险交叉评估，使合规意识融入组织基因，治理能产生正向乘数效应并带来更长期的产品营收生命周期。 - [24:00](https://www.youtube.com/watch?v=8C7bAlXjRcg&t=1440s) 治理体系运转后会产生飞轮效应，可通过分级机制让常规合规项目走绿色快速通道，而外部第三方供应商往往因缺乏统一标准成为治理落地的最大堵点。 - [26:23](https://www.youtube.com/watch?v=8C7bAlXjRcg&t=1583s) 第三方 AI 系统的黑盒构造要求企业实质性投资专业技术人才进行红队测试与持续监控，避免问责流于“打勾式合规”。 - [28:11](https://www.youtube.com/watch?v=8C7bAlXjRcg&t=1691s) 传统软件验证无法直接套用于具有概率性和动态性的 AI，多智能体系统（Agentic AI）更需要通过大规模仿真测试智能体交互与系统级涌现行为。 - [32:14](https://www.youtube.com/watch?v=8C7bAlXjRcg&t=1934s) 在热议生成式 AI 和智能体 AI 的同时，不可忽视企业内部大量影响现实但尚未得到妥善治理的传统预测性 AI 系统。 - [34:11](https://www.youtube.com/watch?v=8C7bAlXjRcg&t=2051s) 破解跨部门对齐阻力的关键是将治理原则锚定在组织赖以生存的核心价值观上（如金融领域的安全反欺诈），并为 AI 智能体设定明确的行为守则。 - [38:12](https://www.youtube.com/watch?v=8C7bAlXjRcg&t=2292s) 从工程角度量化治理 ROI 可以利用边界条件分析与因果推断测试，先建立小而精的“治理核心”，再根据业务逐步升级扩展。 - [42:21](https://www.youtube.com/watch?v=8C7bAlXjRcg&t=2541s) AI 治理在本质上是数字产品时代消减产品缺陷、管理产品责任的品控手段，能通过降低诉讼争议与提升产品稳定性直接创造商业价值。 - [43:48](https://www.youtube.com/watch?v=8C7bAlXjRcg&t=2628s) 推动可信赖 AI 需要全员参与，员工应在内部主动追问潜在危害，公众与消费者也应积极要求透明度，以市场反馈倒逼“设计即安全”。 ## Detailed walkthrough ### [00:04](https://www.youtube.com/watch?v=8C7bAlXjRcg&t=4s) 为什么 AI 治理与问责制不再是可选项 主持人 Funda Gunes 以极大的热情开启了本场圆桌讨论，直言这关乎当今人工智能领域最核心的议题之一。她回顾了自己的跨界职业生涯：从一名统计学博士、机器学习算法工程师，到运用自主构建系统解决复杂业务难题的数据科学家，再到在大型复杂企业中担任负责任 AI 与 AI 治理团队的负责人。正是这种涵盖算法研发、学术科研、教学以及一线治理的复合经历，让她深刻意识到——“AI 治理早已不再是一个可有可无的选项，而是企业的必修课（Must-have）”，且这一需求已变得极为紧迫。 当下关于 AI 问责制的讨论绝非虚无缥缈的哲学清谈，而是每一个身处技术研发与业务落地一线的从业者每天都在面对的现实抉择。在实际操作中，企业普遍面临一系列严峻挑战： - 如何在加速技术创新的同时，平衡负责任的使用准则与风险评估？ - 如何在庞大的组织架构中建立跨部门的 AI 治理共识与协同机制？ - 当模型给出了无法向监管机构或终端客户合理解释的决策结果时，企业该如何应对？ - 一旦系统在生产环境中发生故障，团队能否迅速定位根因并在最短时间内完成修复？ Funda 强调，这场讨论的底线远不仅是打勾应付合规检查或规避法律诉讼，其终极目标是在赢得客户信任的前提下，打造具备长期竞争力和可持续性的创新能力，确保 AI 真正兑现其赋能商业与社会的承诺，而非沦为企业的重大负债。 ```mermaid flowchart TD accTitle: AI Governance Core Spectrum subgraph SG1["从设计到落地的核心治理维度"] A["算法与研发实践<br/>技术构建与可解释性"] --> D["AI 治理与问责体系"] B["伦理定义与规范<br/>界定公平与价值目标"] --> D C["组织与业务协同<br/>跨职能落地与风险管控"] --> D end D --> E["实现可持续创新与公信力"] ``` --- ### [03:38](https://www.youtube.com/watch?v=8C7bAlXjRcg&t=218s) 开发者视角：将崇高原则转化为日常可落地的行动规范 Sail Biomedicines 的人工智能高级副总裁 Rajesh Ramaswamy 作为系统构建者分享了自己的实战心得。他将自己的日常工作概括为“带领团队利用 AI 与计算科学创造实际价值”。Rajesh 曾在两大容错率极低的领域深耕： 1. **高频金融交易领域**：曾参与打造开创性的期权做市交易平台，为全球金融市场提供流动性； 2. **医疗健康与生物医药领域**：过去八年致力于重构医疗健康价值链，过去四年则专注于攻克全新药物研发的底层规律，旨在为亟待救治的患者带来全新的突破性疗法。 在金融与医疗这种高风险行业中，“决策失误的代价几乎趋近于无穷大”，因此构建高度可靠、值得信赖且完全符合预期指标的 AI 系统是生命线所在。作为一线技术构建者，Rajesh 指出当前 AI 问责与治理面临的最大痛点并非缺少原则： > “业内充斥着各种崇高宏大的原则，但核心难点在于：我们如何将这些宏大的原则转化为日复一日、可重复、可审计的具体行动，真正在一线业务中带来改变？” 大家在口头上很容易就“系统应该做什么、不应该做什么”达成共识，但在面对错综复杂的现实技术环境以及日常紧迫的商业 KPI 压力时，如何把抽象原则拆解为工程规范和操作流程，依然是摆在开发者面前最严峻的考验。 --- ### [06:21](https://www.youtube.com/watch?v=8C7bAlXjRcg&t=381s) 哲学与伦理视角：从微观决策到治理体系的“虚假繁荣” 东北大学负责任 AI 实践总监、AI Ethics Lab 创始人 Cansu Canca 从道德与政治哲学家的视角切入。在转向 AI 之前，她的前十年职业生涯主要聚焦于医学伦理与公共卫生伦理领域，处理的是诸如疫情期间疫苗分配优先级、社会封控以及工作场所重新开放等生死攸关的硬性伦理抉择。 2016 年，Cansu 敏锐地察觉到越来越多的生死决策正在交由算法执行，因而全面转向 AI 伦理研究。她指出，媒体和公众往往热衷于讨论《机器战警》或《终结者》式极具科幻色彩的极端假想，但现实中算法带来的伦理风险往往极其隐蔽且“枯燥”，却同样关乎生死： - 算法因为错误的代理变量（Proxies）导致保险系统直接拒绝为病患报销； - 医疗监测系统因技术缺陷无法准确识别特定人群血液中的氧饱和度，导致患者错失最佳救治时机； - 算法未能正确识别病患的真实健康需求而分配了错误的医疗资源。 针对行业现状，Cansu 表达了尖锐的反思。她提到，当天上午 Geoffrey Hinton 刚刚在大会上警示“人类在 AI 安全问题上已经行动滞后”，但反观分论坛现场稀疏的听众，这种反差恰恰揭示了行业面临的根本挑战——认知脱节与虚假繁荣。 > “很多企业在被问到‘是否建立了 AI 治理体系’时都会自信满满地回答‘是的’——因为他们制定了合规政策、成立了伦理委员会、写下了漂亮的原则宣言。但只要你追问一句‘你们究竟是如何将这些原则落地的’，几乎没有人能给出实质答案。大家都自认为做好了准备，直到一场真正的舆论与伦理丑闻爆发。” Cansu 强调，“什么是正确的做法”本质上是一个不可回避的哲学问题。如果企业在研发之初未能清晰界定何为“公平”，那么即便在工程层面对模型进行再多的算法调优与指标优化，最终也只是在盲目优化一个毫无意义的目标。 --- ### [09:08](https://www.youtube.com/watch?v=8C7bAlXjRcg&t=548s) 组织与设计视角：从确定性系统向概率性社会的治理转型 万事达卡（Mastercard）数据与 AI 会士（Fellow）JoAnn Stonier 从大型企业管理与系统设计的宏观视角分享了实践路径。作为万事达卡的首任首席隐私官（Chief Privacy Officer）和首任首席数据官（Chief Data Officer），JoAnn 亲身主导并实现了从**数据治理**、**AI 治理**再到如今**智能体治理（Agentic Governance）**的落地演进。 JoAnn 针对前述观点提出了互补的视角：治理原则绝非空中楼阁或“虚无软件（Vaporware）”。抽象原则的价值在于为不同业务职能团队提供对话的通用基准，让法律、合规、研发及产品等各业务部门能够从各自的专业领域切入，重新理解治理的边界。例如，面对当下的技术变革，企业法务团队甚至需要重新学习“在 AI 时代如何做一名合格的律师”。 结合自己在卡耐基梅隆大学（Carnegie Mellon）和普瑞特艺术学院（Pratt Institute）的教学经历以及设计学背景，JoAnn 从系统设计的维度阐述了当前社会面临的深层范式转移： - **认知范式重塑**：人类社会正在从传统的“确定性计算社会（Deterministic Society）”向“概率性决策社会（Probabilistic Society）”快速跨越； - **生成式 AI 与 Agentic AI 叠加**：智能体系统的普及要求企业内部的所有决策者重新审视产品与业务流程的设计逻辑； - **传统领导力的转型阵痛**：传统企业高管的晋升路径建立在“规避风险、保持稳健业绩、向华尔街与财报交付确定性收益”的模式之上，他们天然缺乏在全新设计范式下驾驭不确定性的经验；如今企业被要求全员借助全新的颠覆性工具加速创新，这种冲突使得“负责任”与“伦理把控”不再仅仅是约束机制，更是推动企业安全转型不可或缺的底层支柱。 ### [11:49](https://www.youtube.com/watch?v=8C7bAlXjRcg&t=709s) 跨学科学习与落地转型的迫切需求 在治理圆桌讨论中，各方必须打破专业壁垒。JoAnn Stonier 指出，当前正处于一个需要跨学科相互培训、鼓励与支持的时刻，所有围坐在治理桌旁的人都必须抱着开放的心态去主动学习。主持人随后接话强调，许多企业和机构并非不想做好 AI 治理，而是根本不知道从何下手。因此，行业专家有责任去主导设计、落实以及推广最佳实践。接着主持人向 Sail Biomedicines 的高级副总裁兼人工智能负责人 Rajesh Ramaswamy 发问：如何将那些宏观的原则切实转化为具体 AI 应用场景（Use Case）中可执行的洞察。 ### [12:47](https://www.youtube.com/watch?v=8C7bAlXjRcg&t=767s) 重新定义 AI 治理：从虚幻概念到生物医药的落地实践 Rajesh Ramaswamy 赞同企业的痛点在于“不知如何做”，并直言“治理”（Governance）通常是一个非常虚无缥缈（fluffy）的概念，不同的人对其有着截然不同的理解。在他看来，企业的第一步是建立一套明确的内部定义。Rajesh 给出了自己的核心定义：AI 治理是一种系统化地开发、部署、监控和评估 AI 系统的手段，其根本目的是让系统持续提供更优的产出结果，而这一结果应与最初决定立项研发该系统的商业/科研初衷紧密契合。 为了证明这一理念的实际价值，Rajesh 分享了 Sail Biomedicines 在新药研发中的成功落地案例。他们通过系统化地执行这一治理框架，正在推进一种全新的 CAR-T 细胞疗法进入临床阶段。该项目通过对人体内特定细胞进行系统性的瞬时重编程，使细胞能够主动对抗疾病，促使机体从患病状态恢复到健康状态。Rajesh 强调，治理不能停留于空洞的理论，必须直接与企业营收或成本控制等真实价值挂钩；如果无法证明治理举措如何为顶线收入（Top Line）或底线利润（Bottom Line）赋能，这种治理就无法维持下去。 - **AI 治理的闭环链条**：系统化开发（Develop） $\rightarrow$ 部署（Deploy） $\rightarrow$ 监控（Monitor） $\rightarrow$ 评估（Evaluate）。 - **底线商业逻辑**：Rajesh 直言这绝非什么高深莫测的高精尖科技，本质上就是最基础的“商业 101”（Business 101）常识。 ### [15:37](https://www.youtube.com/watch?v=8C7bAlXjRcg&t=937s) 将治理嵌入现有产品研发与风险控制流程 JoAnn Stonier 接着指出，企业推行治理之所以能取得最大成功，关键在于将其无缝对接到既有的成熟流程中，而不是另起炉灶。AI 治理必须自然融入产品开发流程、创新流程以及风险管控流程之中。虽然许多企业为了应对生成式 AI（GenAI）或《欧盟人工智能法案》（EU AI Act）等新规设立了各种 AI 委员会，但这些委员会大多才运作了不到一年，很容易沦为浮于表面的重复建设。 JoAnn 以万事达卡（Mastercard）的实践为例说明：早在 2016 年左右，他们就向全公司发布了数据原则（Data Principles），用于指导新产品与解决方案的设计。由于这些原则自始至终深植于产品开发生命周期，随着技术演进而不断迭代，才避免了“为了治理而治理”的脱节现象。她强调，企业本来就拥有固定的产品构建路径与风险合规框架，治理必须成为调整这双重杠杆的自然延伸，唯有如此，治理才能在业务一线真正生根发芽并发挥实效。 ```mermaid flowchart TD accTitle: AI 治理融入既有企业流程模型 accDescr: 展示 AI 治理并非独立存在，而是深度嵌入既有的产品研发流程与风险合规体系中。 subgraph Existing["既有成熟业务体系"] A["产品与解决方案研发"] B["风险管理与合规评估"] end subgraph Governance["负责任 AI 治理层"] C["数据与 AI 原则规范"] D["全生命周期监控与迭代"] end subgraph Outcomes["业务成果转化"] E["可审计的合规记录"] F["业务增长与商业价值"] end A <--> C B <--> C C --> D D --> E D --> F ``` ### [18:27](https://www.youtube.com/watch?v=8C7bAlXjRcg&t=1107s) 游戏化流水线：让负责任 AI 与商业成功双赢 针对主持人关于如何将理论框架与企业风险偏好、核心价值观相结合的提问，东北大学负责任 AI 实践主任 Cansu Canca 阐述了具体的实操方法。她指出，“负责任 AI”（Responsible AI）是一个包含 AI 伦理、AI 治理、AI 安全以及 AI 战略的总体概念。将负责任 AI 落地时，绝不能仅停留在“把各方拉到一张桌子前开会发表意见”的表面形式，因为这种沟通无法明确权责分配，更无法高效推进工作。 Cansu 介绍了她在咨询企业时采用的“游戏化流水线”机制。通过整合企业既有流程的最佳部分，构建一套统一的准入流水线：无论是自研还是外购（Procurement）的 AI 系统，都必须像闯关游戏一样逐级完成不同关卡的审核与验证，并形成完整归档。为了不增加冗长繁琐的官僚委员会审批，他们更倾向于赋能一线团队，提升现有员工的工具使用能力与文档记录能力。 ``` 【负责任 AI 流水线（闯关机制）】 自研 / 外购 AI 需求 │ ▼ [ 第 1 关: 风险与变量评估 ] ── 权衡偏见、可解释性与商业目标 │ ▼ [ 第 2 关: 赋能团队工具落地 ] ── 一线技能提升、轻量化操作 │ ▼ [ 第 3 关: 完整文档化归档 ] ── 决策证据留存、备查合规底稿 │ ▼ 安全上线与业务部署 ``` Cansu 特别澄清了一个常见的行业误区——负责任 AI 或 AI 伦理往往被错误地置于商业成功的对立面。她从伦理学家的角度明确表示，这种对立毫无道理：“如果你这家企业连生存都做不到，那你宣称自己有多好对我来说毫无意义。我希望你们尽可能有影响力、尽可能成功，这样当你们行善时，才能真正切实地造福世界。”因此，商业成功本身就是负责任 AI 战略考量中最核心的变量之一；企业若不主动显式地进行伦理与风险决策，就必然会隐式地承担这些后果，没有任何组织能够逃避责任。 ### [22:04](https://www.youtube.com/watch?v=8C7bAlXjRcg&t=1324s) 结合领域与工具的风险评估，以及治理对业务的乘数效应 Cansu Canca 指出，商业领域的风险与国防或医疗健康截然不同。在高校环境中的优势在于能够汇集各领域的顶尖专家，共同对利益相关者与具体行业进行全面评估，厘清该领域的风险图景。同时，他们还会分析所用工具本身的风险（无论是生成式 AI 还是传统的预测型 AI），将行业与工具两维度的风险等级进行综合交叉，从而精准制定相应的缓解策略与应对方案。 Rajesh Ramaswamy 对此表示高度赞同，并强调 AI 治理绝不应该、也完全不需要与商业激励机制处于对立状态（"orthogonal to business incentives"）。对于一线构建者（builders）而言，如果治理体系搭建得当并真正融入组织基因，它反而会成为一个“正向乘数”（positive multiplier）。开发团队从一开始就将各种潜在变量与参数纳入考量，不仅能打造出更智能的产品，还能加快交付速度。更重要的是，从长期价值创造来看，推出值得信赖的产品（trustworthy products）能够为企业带来更长期的营收生命周期，因为这些产品的信誉“其价值重于黄金”（"worth their weight in gold"）。 ```mermaid flowchart TD accTitle: AI治理的正向飞轮效应 subgraph ValueCreation["价值与信任闭环"] A["多维风险评估<br>(行业环境 × AI工具特性)"] --> B["合规与前置考量<br>融入组织基因"] B --> C["更智能、更快速的产品交付"] C --> D["值得信赖的高质产品"] D --> E["长期稳健营收与品牌文化"] E --> B end ``` --- ### [24:00](https://www.youtube.com/watch?v=8C7bAlXjRcg&t=1440s) 治理的飞轮效应、分级审批机制与供应链/第三方供应商挑战 JoAnn Stonier 补充道，治理能够塑造企业的文化与品牌。尽管建立治理体系的初期阶段往往进展缓慢，但一旦运转起来，就会产生显著的“飞轮效应”（flywheel effect）。在万事达卡（Mastercard）以及她以会士（Fellow）身份合作过的其他机构中，当合规与风险意识深植于团队 DNA 后，团队在提交方案给治理委员会评审之前就已经完成了自我审查。对于常规场景或已获预批的流程与供应商，项目可以直接走“绿色通道”（green paths）快速推进，从而显著提高创新与产品拓展的效率。 然而，一旦涉及全新、前沿的技术探索，使用高敏感度数据，或是引入全新的流程与供应商，审批等级就会迅速攀升至“红色代码”甚至更严苛的“紫色代码”（"code purple"）。Stonier 特别指出了万事达卡面临的严峻挑战：作为一家网络型组织，业务必须通过银行、合作伙伴和第三方供应商等外部机构来执行。受监管的银行之间标准一致，但对接外部技术供应商时，由于对方往往缺乏同等的治理投入与标准，常导致合作受阻。每当有供应商在大会上急于建立合作时，她都会提醒对方：“小心你所期望的（be careful what you wish for）”，因为在获得准入之前，必须通过极其严苛的安全、隐私和数据合规审查。 - **快速通道（Green Path）**：针对预先批准的流程和合规供应商，消除重复审查摩擦，加速产品落地。 - **高风险分级（Purple Code）**：针对涉及极度敏感数据、前沿试验性技术或全新流程的项目，执行最高规格审查。 - **第三方与供应商屏障**：由于缺乏统一的技术与合规标准，外部生态接入成为企业治理落地中最易造成拥堵的环节。 --- ### [26:23](https://www.youtube.com/watch?v=8C7bAlXjRcg&t=1583s) 供应商黑盒困境、测试验证缺口与开发者激励 Funda Gunes 接着指出，供应商管理确实是最困难的环节之一，因为企业往往难以知晓第三方系统的内部黑盒构造。要真正理解其风险与局限，必须进行深入的测试与验证。然而现实痛点在于，许多机构内部缺乏具备相应技术能力的人员去提出关键质询，往往仅由法规合规团队或第三方管理团队进行常规问询，导致 AI 问责流于“打勾式合规”（check-the-box compliance）。 Gunes 强调，要突破这一局限，企业必须进行实质性投资，招募具备红队测试（red teaming）、压力测试和持续监控能力的专业技术人才，且这些人才必须有足够的底气和安全感去挑战既有方案。但在巨大的市场竞争压力与高管催促产品快速上线的“抢跑”冲动下，开发者面临着艰难的权衡。她进而向 Rajesh Ramaswamy 发问：在快速上市的狂潮中，该如何激励开发者真正重视系统风险与潜在危害？ --- ### [28:11](https://www.youtube.com/watch?v=8C7bAlXjRcg&t=1691s) 传统软件与 AI 验证的本质差异：从单体测试到多智能体系统（Agentic AI）治理前沿 Rajesh Ramaswamy 认为，激励机制的关键在于通过可量化的方式证明治理框架能够直接创造商业价值。他指出，构建一个系统并不难，但如果无法验证系统是否真正按预期运行，这个系统就毫无价值。当前技术层面的核心痛点在于，人们普遍习惯于将 AI 视为传统软件系统的延伸。然而，传统软件系统是确定性的（deterministic），具有单一且僵化的控制逻辑；而 AI 系统则是底层具有概率性（probabilistic）、动态且可能进入全新未知状态的系统。因此，现有的软件验证与确认（V&V）框架必须进行根本性的重构与扩展。 ```mermaid flowchart LR accTitle: 传统软件验证与AI多智能体验证对比 subgraph Trad["传统软件验证"] direction TB T1["确定性逻辑"] --> T2["单点代码单元测试"] T2 --> T3["固定功能验证"] end subgraph AIVal["AI与多智能体验证"] direction TB A1["概率性与动态行为"] --> A2["单 Agent 行为与边界测试"] A2 --> A3["多 Agent 交互与涌现行为测试"] A3 --> A4["仿真模拟与灾难性风险指标监控"] end ``` Ramaswamy 与 Stonier 深入探讨了多智能体系统（Agentic AI）带来的全新挑战与治理范式升级： - **单智能体与交互测试**：在多智能体架构中，不仅需要对单个 Agent 的能力边界进行测试，还必须测试 Agent 之间的交互行为，因为交互往往会引发非线性的复杂后果。 - **系统级涌现行为（Emergent Behaviors）**：多个智能体协同可能产生不可预知的全局涌现行为，必须在系统层级通过大规模仿真（simulation）来摸清系统的完整运行区间（operating regime）。 - **安全边界与灾难性指标**：必须明确量化安全边界指标，例如灾难性后果发生的“频率与严重程度”（frequency and severity），确保在技术和组织双重框架下具备事后纠偏与修复的能力。 - **智能体治理的延伸需求**：JoAnn Stonier 最后补充指出，当前业界许多机构刚刚踏上 Agentic AI 的征程，但现有针对传统 AI 的治理工具和方法论在面对智能体系统时已经显得力不从心，行业必须将治理思维与防护框架推向更深、更前沿的维度。 ### [32:14](https://www.youtube.com/watch?v=8C7bAlXjRcg&t=1934s) 传统 AI 与新兴 AI 的治理现实：莫忽视未解决的基础系统 随着技术演进与创新节奏越来越快，治理必须深度嵌入到整个组织的核心流程之中，成为企业运营不可或缺的组成部分。然而，在大家热火朝天地讨论智能体 AI（Agentic AI）和生成式 AI（GenAI）的同时，业界往往忽略了一个关键事实：当前大量运行在企业内部、直接影响日常生活的传统预测性 AI 系统，其实并没有得到妥善治理。大型企业通常只在少数团队中进行零星测试，很少能在整个组织层面建立统一的 AI 治理流水线，对所有研发、部署和采购的 AI 系统进行同等严格的评估。 在传统 AI 领域，业界其实已经非常清楚应该如何做好治理与负责任 AI，这并非源于知识或技术手段的匮乏；而对于生成式 AI，虽然已有较为成熟的治理路径，但仍存在许多未解的研究问题，需要与高校等科研机构紧密合作；至于智能体 AI，行业更是面临着海量的开放性难题。因此，绝不能因为追逐最新技术浪潮，就忽视了那些正在幕后持续影响现实生活的传统预测系统。 ```mermaid flowchart TD accTitle: AI 治理成熟度与技术层级对应图 subgraph S1["传统预测性 AI"] A["治理方法高度成熟"] --> B["企业落地覆盖率严重不足"] end subgraph S2["生成式 AI (GenAI)"] C["具备一定治理规范"] --> D["存在未解前沿课题 / 需校企协同"] end subgraph S3["智能体 AI (Agentic AI)"] E["前沿探索阶段"] --> F["面临大量开放性理论与工程挑战"] end ``` ### [34:11](https://www.youtube.com/watch?v=8C7bAlXjRcg&t=2051s) 复杂组织中的对齐难题：以核心价值观破解治理阻力 不同用例对治理与测试的严格程度需求并不相同。并非所有场景都需要全套繁琐的验证流程，企业可以通过初始评估来明确风险等级：对于主要用于提升生产力和内部效率的场景，流程可以相对轻量；而对于可能造成深远伤害的高风险决策系统，则必须执行极其严格的测试与验证。然而在面对众多利益相关者时，AI 治理往往因为跨部门意见不一而变得极其缓慢且令人沮丧。 针对复杂组织中的对齐难题，长期在金融服务等强合规行业深耕的实践者指出，打破僵局的关键在于挖掘企业真正关心的核心利益与存在价值： * **锚定组织的终极诉求**：合规负担沉重的行业（如医疗和金融）之所以对治理高度重视，是因为它们输不起结果的准确性与安全性。以 Mastercard 为例，安全与反欺诈是其立足之本——如果用户在刷卡、手机支付或输入卡号时对网络安全性产生怀疑，整个业务大厦就会瞬间坍塌。 * **将价值观转化为原则与流程**：原则必须源于企业赖以生存的核心价值观。治理不能搞一刀切的说教，而是要找到不同利益相关者的利益契合点，将其转化为治理的目标与动因。 * **为智能体建立“行为守则”**：随着向智能体时代演进，企业必须像对待人类员工一样，为 AI Agent 设定明确的行为边界和职业准则。 * **高层支持与基层流程相结合**：自上而下的高层赞助（Sponsorship）不可或缺，但治理规则最终必须深度融入执行层日常操作的组织 DNA 之中，才能真正落地运转。 ### [38:12](https://www.youtube.com/watch?v=8C7bAlXjRcg&t=2292s) AI 治理的投资回报率：工程化边界、因果推断与最小可行核心 在推动 AI 治理时，高管层往往会要求看到具体的投资回报率（ROI）指标。从硬核工程思维的角度来看，构建安全系统的第一步是明确系统的边界条件：摸清系统能做什么、不能做什么，以及发生潜在负面结果的概率与严重程度。通过预先设置护栏和安全防线，可以在极端违规情况发生时提供兜底保护。 证明治理价值的另一条有效路径是引入因果推断（Causal Inference）测试：通过对比“未采取治理措施可能导致的后果”与“实施治理后的现状”，将治理带来的风险规避与系统稳定性直接量化为实实在在的商业价值。企业无需一开始就追求一步到位、极其华丽的治理框架，而是应当先建立一个能够应对日常传统数据与 AI 任务的“治理核心（Nucleus）”，使其在当下产生价值，随后再根据业务发展需要逐步升级以适应多智能体等新兴环境。 ### [41:02](https://www.youtube.com/watch?v=8C7bAlXjRcg&t=2462s) 衡量 ROI 的量化视角：流程精简提速与竞品合规风险洞察 衡量 AI 治理投资回报率还可以从运营效率和行业竞争两个切实的商业维度进行量化评估： * **加速创新与精简决策流程**：优秀的治理机制不仅不会阻碍创新，反而能大幅提升创新速度。聪明的企业会量化流程改善前后的对比数据——原本需要多少人参与层层复核，而在治理机制理顺后，审批和部署流程变得多么精简和敏捷。 * **洞察竞品盲区与合规风险溢价**：在对特定产品或采购系统进行负责任 AI 评估时，企业不仅能确认自身产品的安全性与合规性，更能借此看清竞争对手在哪些关键环节缺乏防范。虽然诉讼和监管处罚往往具有滞后性，但通过观察竞品未来可能陷入的合规泥潭，企业可以清晰地量化自身提前进行治理评估所规避的潜在损失与竞争优势。 ### [42:21](https://www.youtube.com/watch?v=8C7bAlXjRcg&t=2541s) 将 AI 治理视作产品责任与缺陷消减的量化机制 讨论从量化 AI 风险的维度展开。衡量 AI 风险及其治理成效的关键，在于观察客户与消费者信任度的流失、市场波动幅度以及实际发生的诉讼赔偿，这些都为治理提供了直接的量化抓手。随后，与会者提出了一种从“产品责任”（Product Liability）切入的思考范式：AI 治理本质上是在数字产品时代消减产品缺陷（reducing defects）的新型手段。在传统的实体制造业中（例如混凝土或硬件制造），控制产品瑕疵与安全缺陷的标准十分明确；而在 AI 与软件产品开发中，治理机制承担的正是相同的品控与风控角色。 从投资回报与商业价值的角度来看，治理不仅可以提升研发效率与交付概念的落地质量，更能有效阻断潜在的法律纠纷与合规处罚。衡量治理成效的指标与衡量常规新产品的商业指标完全相通，涵盖以下维度： - **信任与风险控制**：客户信任度评级、潜在法律诉讼数量的下降以及合规争议事件的减少。 - **市场与业务表现**：市场份额（Market Share）、客户满意度（CSAT）、用户留存率（Retention Rate）以及新功能上线的顺畅度。 - **交付与产品质量**：系统缺陷率的降低、模型输出的有效性与稳定性。 ```mermaid flowchart LR accTitle: AI 治理作为产品责任管理模型 subgraph GovernanceInput["治理机制介入"] A["设计与测试规范"] B["缺陷消减流程"] end subgraph ProductValue["产品与责任优化"] C["降低产品责任风险<br/>(减少诉讼/合规问题)"] D["提升交付质量与效能"] end subgraph BusinessMetrics["业务量化指标"] E["客户信任与留存"] F["市场份额与满意度"] end A --> C B --> C A --> D B --> D C --> E D --> F ``` --- ### [43:48](https://www.youtube.com/watch?v=8C7bAlXjRcg&t=2628s) 总结与呼吁：全员参与构建负责任、可信赖的 AI 生态 在尝试进入现场观众问答环节时，主持人确认因会议时间已到而无法接入提问，随即对整场圆桌讨论进行了全面总结。治理工作虽然复杂且充满挑战，但只要通过精心设计的系统测试、合理的预算与资源倾斜、提升全员 AI 素养（AI Literacy）以及确保跨职能团队共同参与，就能在企业内部建立起健康且高效的治理体系。 主持人特别强调了个体与社区在推动负责任 AI 发展中的能动性，并提出以下核心呼吁： - **组织内部的批判性审视**：组织内的每位成员都能对正在研发的 AI 系统产生积极影响。员工应主动提出尖锐的关键问题，追问“该系统可能带来哪些潜在危害”，从源头防范风险。 - **公众与社区层面的问责诉求**：不仅在企业内部，在日常作为消费者与本地社区成员使用 AI 产品时，也应主动向技术提供方要求透明度、责任感与问责机制（Accountability）。 - **设计即安全（Safety by Design）**：正是来自市场与公众的明确要求与反馈，才能真正倒逼并激励组织将责任感、问责制与安全性深度嵌入到底层架构设计之中（正如 Rajesh Ramaswamy 所强调的设计理念）。 圆桌在向全体嘉宾精彩观点的致谢以及对现场观众的感谢声中圆满结束。 ## Notable quotes > [00:04](https://www.youtube.com/watch?v=8C7bAlXjRcg&t=4s) AI 治理早已不再是一个可有可无的选项，而是企业的必修课（Must-have） — Funda Gunes > [03:38](https://www.youtube.com/watch?v=8C7bAlXjRcg&t=218s) 业内充斥着各种崇高宏大的原则，但核心难点在于：我们如何将这些宏大的原则转化为日复一日、可重复、可审计的具体行动，真正在一线业务中带来改变？ — Rajesh Ramaswamy > [06:21](https://www.youtube.com/watch?v=8C7bAlXjRcg&t=381s) 很多企业在被问到‘是否建立了 AI 治理体系’时都会自信满满地回答‘是的’——因为他们制定了合规政策、成立了伦理委员会、写下了漂亮的原则宣言。但只要你追问一句‘你们究竟是如何将这些原则落地的’，几乎没有人能给出实质答案。大家都自认为做好了准备，直到一场真正的舆论与伦理丑闻爆发。 — Cansu Canca > [18:27](https://www.youtube.com/watch?v=8C7bAlXjRcg&t=1107s) 如果你这家企业连生存都做不到，那你宣称自己有多好对我来说毫无意义。我希望你们尽可能有影响力、尽可能成功，这样当你们行善时，才能真正切实地造福世界。 — Cansu Canca > [22:04](https://www.youtube.com/watch?v=8C7bAlXjRcg&t=1324s) orthogonal to business incentives — Rajesh Ramaswamy > [22:04](https://www.youtube.com/watch?v=8C7bAlXjRcg&t=1324s) worth their weight in gold — Rajesh Ramaswamy > [24:00](https://www.youtube.com/watch?v=8C7bAlXjRcg&t=1440s) be careful what you wish for — JoAnn Stonier > [24:00](https://www.youtube.com/watch?v=8C7bAlXjRcg&t=1440s) code purple — JoAnn Stonier ## Mentioned JoAnn Stonier, Cansu Canca, Funda Gunes, Rajesh Ramaswamy, Mastercard, Northeastern University, AI Ethics Lab, MetLife, Sail Biomedicines, Carnegie Mellon University, Pratt Institute, Geoffrey Hinton, EU AI Act, CAR-T ## Video description JoAnn Stonier, Fellow of Data & AI, Mastercard Cansu Canca, Director of Responsible AI Practice, Northeastern University Funda Gunes, Former Head of Responsible AI, MetLife Rajesh Ramaswamy, Senior Vice President, Head of Artificial Intelligence, Sail Biomedicines Abstracts This session will explore the critical issue of accountability in AI systems—who is responsible for AI outcomes, and how should responsibility be distributed among creators, users, and regulators? We will gather business leaders, builders of AI systems, and regulatory experts to discuss the distribution of responsibility in AI development, deployment, and usage. Panelists will engage in a society-level discussion about the long-term impacts of how we assign responsibility and the current incentive structures driving responsible AI development. The conversation will also cover how these decisions will shape innovation, governance, and public trust in AI systems moving forward. [Source](https://www.youtube.com/watch?v=8C7bAlXjRcg)

[0:04] Funda Gunes: AI 治理早已不再是一个可有可无的选项，而是企业的必修课（Must-have） [3:38] Rajesh Ramaswamy: 业内充斥着各种崇高宏大的原则，但核心难点在于：我们如何将这些宏大的原则转化为日复一日、可重复、可审计的具体行动，真正在一线业务中带来改变？ [6:21] Cansu Canca: 很多企业在被问到‘是否建立了 AI 治理体系’时都会自信满满地回答‘是的’——因为他们制定了合规政策、成立了伦理委员会、写下了漂亮的原则宣言。但只要你追问一句‘你们究竟是如何将这些原则落地的’，几乎没有人能给出实质答案。大家都自认为做好了准备，直到一场真正的舆论与伦理丑闻爆发。 [18:27] Cansu Canca: 如果你这家企业连生存都做不到，那你宣称自己有多好对我来说毫无意义。我希望你们尽可能有影响力、尽可能成功，这样当你们行善时，才能真正切实地造福世界。 [22:04] Rajesh Ramaswamy: orthogonal to business incentives [22:04] Rajesh Ramaswamy: worth their weight in gold [24:00] JoAnn Stonier: be careful what you wish for [24:00] JoAnn Stonier: code purple

[6:21] Cansu Canca: When I ask you do you have AI governance in place, you will say yes... you have a policy, you have a board, and you have principles. And when I ask you how you operationalize them... you will not give any answers, but you will still say that you are ready until—as always with ethics—until there's a scandal.

[0:04] Thank you. Good afternoon everyone. I am [0:07] very excited to uh lead this panel which [0:10] I believe will be one of the most [0:11] critical conversations in AI today. Uh [0:16] tell me first let me tell you why am I [0:19] so excited and passionate about AI [0:21] accountability and responsibility. Uh my [0:24] background is in statistics. I'm a PhD [0:26] statistician. I started my career as a [0:29] machine learning developer. Then I was a [0:33] data scientist where I used the systems [0:35] that I built to solve complex problems. [0:38] Then I had an opportunity to work as an [0:41] AI governance leader at a complex [0:43] organization for hands-on AI governance [0:47] uh work. Um I'm very excited uh about [0:51] this uh panel because with all my [0:56] background I really now fully understand [0:59] that AI governance is no longer a [1:02] choice. It's a musthave and and my [1:06] experience in developing, researching, [1:09] teaching and governing um helped me [1:13] understand that the need is uh really [1:16] urgent. Um so this is this isn't just a [1:20] philosophical debate today. it is um [1:24] your I'm sure many of you uh are asking [1:27] questions uh at your current jobs making [1:31] decisions on AI development and [1:33] deployment right now you're perhaps [1:36] asking how do I balance uh innovation [1:39] with [1:41] sorry how do I below uh balance [1:44] innovation with um [1:47] um uh with uh responsible use and risk [1:52] assessment and uh how do I create [1:56] organizational alignment on AI [1:58] governance? What happens when my systems [2:02] makes decisions that I cannot explain to [2:04] regulators or my customers or what if my [2:08] system fails? Can I identify why it [2:12] fails and and how quickly I can solve [2:15] that problem? Um stakes couldn't be [2:18] higher. Uh we are not just talking about [2:22] compliance checkboxes or avoiding [2:24] lawsuits though those matter. Um we are [2:28] talking about creating sustainable and [2:31] competitive innovation [2:34] while building customer trust and and [2:37] and making sure AI delivers the promise [2:41] it has not just being uh rather than [2:45] just being a liability. So we have the [2:48] perfect panel today with us today to [2:51] under uh to answer those questions. We [2:55] have Rajesh who builds ML systems that [2:58] powers uh business decisions. Jan Su [3:01] Janja uh she does uh translates AI [3:07] ethics into frameworks for Fortune 500 [3:11] uh companies through her consultancy. [3:14] And we have John Stonier. She builds AI [3:18] data governance and AI governance [3:19] systems in complex uh regulatory [3:22] organizations. With that uh I want to [3:25] start with Rajes. Uh could you please uh [3:29] introduce yourself and and each of you [3:31] could you tell one uh challenge one [3:35] accountability challenge that you see [3:37] today? [3:38] >> Yeah sure. Um thanks a lot for having [3:40] me. It's fantastic to be here uh part of [3:42] a really fantastic panel. Um let me just [3:46] you know give you a brief overview why [3:47] the hell I'm here in the first place. So [3:50] I build [3:52] um and lead organizations to create [3:54] value with AI and computation. That's [3:58] essentially what I do. That's my day in [3:59] and day out. Um I've had the privilege [4:02] of doing that in two high stakes [4:04] industries. One in one finance and one [4:07] being healthcare. In finance, I had the [4:10] privilege of contributing to a [4:12] pioneering uh options marketmaking [4:15] platform to provide liquidity to global [4:17] markets. Uh over the last eight years or [4:20] so, I'm trying to help re-imagine key [4:23] pieces of the healthcare value chain to [4:25] be able to create untapped value. Um and [4:28] specifically over the last four years I [4:32] am helping figure out a way to create [4:36] new medicines by cracking the rules of [4:38] how we can create entirely novel [4:40] medicines to bring to market for [4:43] patients in need. So there's one common [4:46] theme across all these things is to be [4:48] able to build reliable AI systems that [4:51] are trustworthy and actually you know [4:54] fulfill the target requirement that you [4:57] set out you know when you started [4:59] designing it and especially the stakes [5:01] cannot get higher in industries like [5:04] finance and healthcare you know the cost [5:06] of a bad outcome is almost close to [5:09] infinity so it's extremely important to [5:12] build systems that are reliable and I'm [5:14] going to talk about you know my [5:16] perspectives as a builder how to build [5:18] and like you know some of the some of [5:20] the thoughts I have and perspectives I [5:22] have on how to build you know reliable [5:24] AI systems. One of the biggest [5:26] challenges I see as a builder is when it [5:30] comes to accountability and [5:31] responsibility and governance there are [5:33] like lots of lofty principles. [5:36] uh the key is how do we turn those lofty [5:39] principles into repeatable auditable [5:42] actions day in and day out you know that [5:45] can actually change the stuff on the [5:48] ground I think for me that's the that's [5:50] the big challenge because there are [5:51] lofty principles that exist we can all [5:53] kind of like agree on what a system [5:56] should and should not do but how do you [5:58] kind of like translate it to everyday [5:59] real world actions especially when you [6:02] take into account real world complexity [6:04] as well as you know everyday business [6:06] challenges and business pressures. [6:08] >> Very very good point Rajes. You know [6:11] because those are really use case [6:13] specific and principles are meaningful [6:16] for that uh that case that you are [6:19] applying Jansu [6:21] >> same question. Okay. Background and [6:23] >> just background and one challenge that [6:25] you see. [6:26] >> Okay. Um my background I I'm a [6:29] philosopher. I'm a moral and political [6:30] philosopher and um I have spent the [6:33] first decade of my life making life and [6:36] death decisions in medical ethics and uh [6:38] public health ethics. So when you [6:39] imagine all the decisions we did during [6:41] pandemic for example who should get what [6:43] like the vaccines closing and all of [6:45] these questions closing the society [6:47] opening up the workplace workplaces and [6:49] so on. All of these questions were the [6:51] types of questions that I was dealing [6:52] with in bioeththics and health [6:54] technologies. [6:55] Um then in 2016 I started working on [7:00] specifically AI and ethics because I [7:03] realized that we are increasingly making [7:05] life and death decisions through AI and [7:07] they are not necessarily the most [7:09] glorious, you know, Robocop and [7:10] Terminator stories. They are very [7:12] mundane questions where your insurance [7:14] just doesn't cover you for the wrong [7:16] reasons or you don't get to have the [7:18] health care that you should get because [7:20] the system decided that you they they [7:22] the system couldn't um read your oxygen [7:26] level the oxygen level in your blood [7:27] pressure or they they decided the system [7:30] couldn't um identify your health needs [7:33] because of the wrong proxies. very very [7:36] boring in in the you know in the media [7:39] look but very much life and death uh [7:42] questions. So I started AI ethics lab [7:44] that was one of the first initiatives [7:46] that brought the research that happens [7:49] in philosophy and AI to the industry and [7:53] oftentimes you would still see this um [7:56] today philosophy is neglected although [7:58] we had a panel just two panels ago from [8:01] those who were working in philosophy. uh [8:03] when we say AI ethics when we say what [8:05] is the right thing to do that is an that [8:07] is necessarily and unavoidably a [8:09] philosophical question without defining [8:11] that without defining what is fair you [8:13] can spend all your time trying to [8:14] operationalize and optimize it you don't [8:17] know what you're optimizing so when you [8:19] ask me what is the challenge well the [8:21] challenge is look around and how full [8:24] this room is and how important AI ethics [8:27] is and how Hinton this morning said a we [8:30] were late for AI safety And again look [8:33] around how empty this room is. [8:36] I think that is the challenge because I [8:38] can assure you having worked with so [8:40] many companies when I ask you do you [8:42] have AI governance in place you will say [8:44] yes when you will mean that you have a [8:47] policy you have a board and you have [8:49] principles and when I ask you how you [8:51] operationalize them going back to [8:53] Rajes's point you will not give any [8:55] answers but you will still say that you [8:57] are ready until [8:59] you know as always with ethics until [9:02] there's a scandal so that's the [9:03] challenge [9:05] >> thank Thank you, [9:06] >> Joan. [9:08] >> Hi. Um, [9:11] so interesting to be up here. So, hi [9:13] everyone. I'm Joanne Stoner. Um, [9:16] currently a fellow of data and AI for [9:18] Mastercard. Um, my journey is very [9:21] different than my fellow panelists. Um, [9:24] I come to this time having been [9:28] Mastercard's first chief privacy officer [9:31] and after that the first chief data [9:33] officer. and I have operationalized [9:37] data governance, AI governance and now [9:41] agentic governance. [9:43] Um, so I don't think it's vaporware. [9:46] It's not just principles. I think [9:48] principles are important for people to [9:50] engage with so that there's a [9:53] commonality when you have the different [9:56] disciplines across an organization [9:58] digging into what governance means from [10:02] their particular lane. So when you have [10:06] your legal teams at this point having to [10:09] relearn how to be an attorney in this [10:13] moment in time, I think it is important. [10:16] So I I come from a very different [10:18] perspective. My um I'm also a teacher. I [10:22] teach at Carnegie Melon and at Pratt [10:23] Institute. I have a design degree. Uh so [10:26] I think from a design perspective and [10:29] how we have to retach all of us how to [10:32] be designers in this moment of [10:35] inflection as our world is getting [10:39] redesigned with both generative AI but [10:42] agentic AI. Right? We're moving from [10:45] kind of a deterministic computerized [10:48] society to a probabilistic society. And [10:51] so I I do think that this discussion is [10:53] very very important. But I'm very very [10:55] optimistic that when you can get [10:57] everybody engaged in the conversation [11:00] and I think many organizations can do [11:02] it. I don't think it's because they [11:04] don't want to. I think they don't know [11:06] how to do it. Um, and I also get to be [11:09] an adviser to a group called the [11:11] Cantellis Group where we're working with [11:13] organizations who are trying to adapt to [11:15] this moment because I think we are in a [11:18] moment in society where we have all [11:20] these exciting tools and yet um, most of [11:23] the leaders of organizations haven't had [11:26] this new design moment and that's not [11:28] their natural path to leadership. they [11:31] have been in a moment where they've been [11:33] asked to deliver very consistent [11:35] results, be riskadverse, um, right? [11:38] Report to Wall Street, report on [11:40] earnings, right? And now we're asking [11:42] everybody to innovate, but innovate with [11:44] all these new tools. And so I think [11:45] responsibility, which is what the panel [11:48] is about. And I think ethics are also a [11:49] piece of that. um we're actually in a [11:52] moment where we have to retrain each [11:54] other and help and encourage each other [11:56] from all the different disciplines that [11:58] gather around the governance table to [12:01] participate in this moment but everybody [12:04] has to be willing to learn. So I'll stop [12:06] there. [12:06] >> Thank you. I think it's a really [12:08] important point like organizations and [12:11] companies it's not that they don't want [12:13] to do it and they don't know how to do [12:15] it and it's all of our responsibility [12:17] and experts in this area to kind of lead [12:21] uh these designs and implementations and [12:23] best practices. So, Roesh uh you [12:27] mentioned that you know it is really [12:30] important to translate those principles [12:33] to actionable insights at a at a use [12:37] case level for a [12:40] uh AI system. Tell us uh give us a [12:43] concrete example. What does it mean? How [12:45] do you make that translation? [12:47] >> Yeah. So, I I do agree with Joan. I [12:50] think organizations want to actually do [12:53] it and I think the principles exist but [12:55] where they kind of like fall short is [12:57] they just do not know how to do it right [12:59] and and I think also I think one of the [13:02] stuff it also starts from governance is [13:05] a fluffy thing I think many people have [13:07] different definitions right and I think [13:09] I think the first step whether it's [13:11] right or wrong is to first come up with [13:13] the definition as a as an organization [13:15] what do you really mean when you talk [13:17] about AI governance right What is it? [13:20] And I give you my definition. It may not [13:22] be everybody's definition. The way I [13:24] look at it is is the means through which [13:27] you can systematically develop, deploy, [13:31] monitor and evaluate AI systems so that [13:35] it continuously provides you with [13:38] improved outcomes. outcomes as defined [13:42] by why did you you know go on a journey [13:45] to develop and deploy this system in the [13:47] first place right so it's as simple as [13:50] that and I think one of the stuff that [13:52] we have done me being as part of sale [13:55] biio medicines where I'm leading all the [13:57] AI initiatives to be towards being able [14:00] to create entirely new medicines we have [14:03] actually implemented this systematically [14:05] and one of the the fruit the fruits that [14:08] we are bearing because of This is [14:10] actually we have come up with a entirely [14:12] new you know medicine called that is [14:15] that we are currently kind of [14:16] progressing towards the clinic. Uh it's [14:19] a carti program where we systematically [14:23] you know transiently reprogram certain [14:26] cells in the human body so that the [14:28] cells in the human body can actually [14:30] fight the disease and get your body out [14:32] from a state of disease to a state of [14:34] health. Right? So I think I'm just [14:36] providing evidence to some of the stuff [14:38] that Joanne mentioned and also some of [14:40] the some of the stuff that Konu has [14:41] mentioned right I think it's a it's a [14:44] fluffy concept and I think it's [14:46] important to first define what you [14:48] really mean and then actually go about [14:50] executing it and to really connect it [14:53] with real world value. I think one of [14:55] the other challenges that that you know [14:57] organizations face is if it is too [15:00] theoretical, too abstract and it's not [15:02] connected to kind of the end business, [15:04] you know, where the kind of money comes [15:05] from. How do you kind of like show that [15:07] what you are doing and what you have [15:09] implemented is actually adding to the [15:11] top line or adding to the bottom line? [15:15] It's a it's a bit difficult for that [15:16] thing to be sustainable. I I don't think [15:19] these things are like rocket science. [15:20] It's basically business 101 to be [15:22] honest. But there are a lot of different [15:24] components that go together to be able [15:27] to make these things work and it's very [15:29] much possible. Uh you just need to do it [15:32] in a systematic way to translate kind of [15:34] a fluffy concept into actually a [15:36] rigorous framework. [15:37] >> But I think it's the word governance but [15:40] I think where organizations have the [15:43] most success is [15:45] connecting governance to processes that [15:47] already exist. Right? So connect it to [15:50] your product development process. Right? [15:52] governance should be part of your [15:54] innovation process. It should also be [15:56] part of your risk process, right? Um, [15:58] organizations that spin up these [16:00] councils that Hansu mentioned, right, [16:03] have found and and let's also be fair to [16:06] people. These councils are about at most [16:09] a year old, right, for AI and Gen AI. [16:12] No, they are. I mean, like we've had [16:14] governance councils, but if you're [16:16] talking about this moment in time, but [16:18] that by the time we've gotten [16:20] everybody's involved, they're trying to [16:22] govern AI, maybe we've had data [16:26] governance, maybe we've had other types [16:28] of product development, but now we've [16:30] tried to, you know, layer on something [16:32] new because now we have the EU Act or [16:34] something else. instead of thinking well [16:37] maybe I should be thinking about this as [16:39] an extra layer of what I need to be [16:41] governing which is product development [16:43] and risk management which are like the [16:45] two levers that most organizations have [16:48] to adjust to anyway I mean most [16:50] organizations are already trying to [16:53] build products and solutions and [16:55] understand the risk and compliance [16:57] requirements right and if you can [16:58] connect that into your governance [17:00] practices which is not so easy I mean [17:02] because we have risk management [17:03] frameworks right everybody has processes [17:05] we dedicated to our processes, right? [17:07] But if you can connect all of that up, [17:10] then you can have governance that begins [17:12] to really matter and you can begin to [17:15] get people to get their brains wired [17:17] into thinking, "Oh, wait a minute." So, [17:20] if I can help use these frameworks in [17:22] ways that actually connect to the [17:24] issues, I can make this matter. And so [17:27] while I get that like we've had data [17:29] principles since like I don't know 2016 [17:32] I think we announced our data principles [17:34] right for the whole organization which [17:36] help with designing new products and [17:38] solutions and then we had to you know [17:40] evolve them but because of that it was [17:42] always part of our product development [17:45] process it wasn't like just governance [17:47] for governance sake it has to be [17:49] embedded into something that matters so [17:51] I completely agree Raj it's like if you [17:53] don't do that then it doesn't like hang [17:55] off of anything that it's just something [17:57] separate, right? And that's where when I [18:00] think it gets fuzzy. [18:01] >> Yeah. John, so you advise a lot of uh [18:04] companies and the the problem usually is [18:07] that they have those theoretical [18:09] frameworks but they don't work for the [18:12] organization. So you do that translation [18:15] what does it uh mean and could you tell [18:18] us you know how does it related to the [18:21] risk appetite of the company their [18:23] principles and values and what that [18:26] process looks like? [18:27] >> Yeah. Um so I think just also following [18:31] from what Joanna and Raj said um I think [18:33] the most important thing is to make sure [18:35] that everything that we discuss in [18:36] responsible AI or AI ethics or AI [18:39] governance by the way I mean I guess the [18:41] way that we use it responsible AI is a [18:43] good way of including AI ethics, AI [18:46] governance, AI safety and AI strategy. [18:48] So all of these fall under the AI [18:50] governance in sorry responsible AI in [18:52] some form and um just following from uh [18:56] the comments I think one of the most [18:58] important things is that it has to be [19:00] very very concrete. Oftenimes we hear [19:02] again the principles but they are not [19:04] translated into operations or we have um [19:07] the the even the language that we use we [19:09] we usually say have pe having everyone [19:12] on the at the table having everyone come [19:14] together but exactly in which form [19:16] because just getting opinions voiced is [19:19] not sufficient in making sure that you [19:21] are assigning the right responsibilities [19:24] and roles and actually getting things [19:26] done in an efficient manner. Um so when [19:29] we are engaging so I I don't think I [19:32] said what I do right now I said what I [19:33] did but I am at northeastern university [19:35] now and I built a responsive practice at [19:37] the university that consults different [19:39] companies just like AI ethics lab um [19:41] that I have on the side. So what we do [19:44] is to basically look at the organization [19:47] any organization that we are working [19:49] with. Um Joan as you said looking at the [19:52] existing structures because you do not [19:54] want to add yet another full layer of [19:56] process and make it redundant make it uh [19:59] repetitive and make sure that we take [20:03] it's almost like designing a game [20:05] really. You're taking the best parts and [20:07] the most relevant parts of each process [20:09] and you put them in a new pipeline where [20:12] every time you have in-house AI [20:14] development or procurement an AI system [20:17] comes through this pipeline and they get [20:20] clearance from different uh again if you [20:22] think of it like a game they uh complete [20:25] different levels and go to the next [20:27] stage and by the end of it you can [20:29] always look back at the documentation [20:32] and who has approved what and don't [20:34] think of approving as necessarily um [20:37] like this very boring again board [20:39] structure. We often do this by assi [20:41] upskilling certain people in the teams [20:43] to be able to use the the tools to [20:46] employ them and then document them. And [20:48] by the end of it, when and if there is a [20:50] question that arises regarding the the [20:53] bias of this product or the [20:55] explanability of this product or how did [20:57] you decide on whom to target, you have [21:00] everything um already documented and [21:02] structured and thought through. And the [21:05] important thing to realize is that if [21:06] you don't do this explicitly, you are [21:08] definitely doing this implicitly. So you [21:10] are never off the hook. What we do is we [21:12] make sure that when whatever decisions [21:14] that you are making, you are making them [21:17] by understanding the risks and benefits [21:19] and the relevant variables and one of [21:22] the relevant variables of course is um [21:24] the business success. I think a lot of [21:27] the times responsible AI or AI ethics is [21:29] pitted against business success which I [21:32] can tell you as an ethicist makes [21:34] absolutely no sense because if you don't [21:35] exist how good you are is kind of [21:38] irrelevant for me. I want you to be as [21:40] impactful as successful as possible so [21:42] that when you do good it actually [21:43] impacts the world. So business success [21:46] is very important variable for us as we [21:48] implement AI ethics and AI strategy. Um [21:52] and just one more word about the [21:54] specificities of the uh of course you [21:56] have different companies structures and [21:59] sizes and the sectors [22:01] the things that you have to pay [22:03] attention to in the entertainment [22:04] business is very different from defense [22:07] or healthcare. So what we do is being at [22:10] the university is great for this. I have [22:12] all the experts that I need and I bring [22:16] the right expertise to assess the [22:19] stakeholders to assess the sector and [22:20] really um lay out the risk landscape of [22:24] that sector and the risk landscape of [22:27] that tool you know Gen AI or old school [22:31] predictive AI and then figure out the [22:35] combination of these two risk levels to [22:38] assign exactly how the to deal with the [22:41] mitigation and the strate strategies [22:42] around it. [22:43] >> Yeah. And just to add to it, right, I [22:45] think the AI governance and all these [22:47] things I think should not be and need [22:50] not be like orthogonal to business [22:52] incentives at all, right? I think [22:54] absolutely not whatsoever. Sometimes [22:56] there is an assumption it's a bit [22:58] orthogonal. Uh but even I can give you [23:00] an example like builders like me, right? [23:02] If if it's set up right and if it's part [23:05] of the fabric of the organization, it is [23:07] actually a positive multiplier in terms [23:10] of like what you can build and how [23:11] quickly you can build because you can [23:14] build products now that are smarter [23:16] because from the get-go you accounted [23:18] for a lot of different things instead of [23:19] accounting for like certain variables [23:21] and parameters that you did not think [23:23] about, right? So you can ship products [23:26] faster and actually smarter. And the [23:29] other thing that also like adds to [23:30] long-term value creation is if you ship [23:34] trustworthy products, you are going to [23:36] have a revenue over a longer time frame, [23:39] right? So there is there is a clear [23:41] benefit even from that perspective [23:42] because trustworthy products, you know, [23:45] are are worth their weight in gold. [23:47] >> So these are couple of dimensions where [23:49] I think you know there is no reason why [23:52] all these frameworks need to be like [23:55] orthogonal. They can be they can very [23:57] well be aligned and should be aligned [24:00] >> and they build your culture, they build [24:02] your brand and once you know in in a [24:04] weird way governance sometimes can be [24:06] slow to get started but once it gets [24:08] started it has a flywheel effect or at [24:10] least it did at Mastercard um and it it [24:12] has with other organizations that I've [24:14] worked with as a fellow whereby to your [24:17] point Raj like once it's in the DNA of [24:20] different teams they know how to do [24:22] their own checking rather before they [24:24] get to a council and it's, you know, and [24:27] anytime you're doing something that's [24:29] completely net new and on cutting edge [24:31] for your organization or it's using [24:34] extremely sensitive data or it's using a [24:36] new process or a new vendor or you're [24:38] doing something that is completely [24:40] cutting edge for your organization or [24:42] your industry. Okay, that's going to be [24:44] kind of a code red, a code purple. We we [24:46] go beyond red, we go to purple. But when [24:49] you know it's going to be probably green [24:50] pathed, you can move so much faster [24:53] because it's things and parts of [24:55] processes that have already been [24:56] approved, vendors that have been [24:58] preapproved, right? So it begins to be a [25:01] faster innovation path, right? So there [25:03] are green paths that the organization [25:05] can move through at a much quicker pace [25:09] because of governance, right? So because [25:11] the risks have been addressed, because [25:13] the issues have been addressed and you [25:14] already know those and it makes for a [25:16] smarter organization that allows for [25:19] greater efficiency. It allows for [25:21] product extensions. It allows for new [25:24] innovation especially when those [25:26] innovations are happening now. They need [25:29] to happen at a faster pace. One of the [25:31] other things that I think is also going [25:33] to be like one that is for us one of the [25:36] issues we have to counter every day is [25:38] we're a network, right? which means we [25:40] execute through other organizations. [25:43] That is often times one of the harder [25:45] parts of our execution, right? It's one [25:48] thing when we're going through our banks [25:49] because we're all regulated entities, so [25:51] we're all at a certain level, but when [25:54] we're not or our vendors and we have to [25:56] put them through kind of the same [25:58] frameworks, that's where it sometimes [26:00] gets harder and it gums up the works, [26:02] right? Because we're not seeing the same [26:04] level of dedication. And so, you know, [26:07] people come up to me at conferences like [26:08] this and they want to be a vendor and [26:10] I'm like, be careful what you wish for. [26:12] You've got to go through an awful lot of [26:14] security, privacy, data requirements [26:17] before we're going to let you in because [26:19] of the speed requirement now of what [26:21] we've done with governance. [26:23] >> Yeah. Yeah. Your good point. uh vendor [26:25] management is one of the most difficult [26:28] because you really don't know in that [26:30] what is in that system and you really [26:33] need testing and validation and one of [26:36] the challenges is that organizations [26:39] don't really have technical people to [26:42] test and validate and ask really [26:44] important questions to understand risks [26:47] and limitations of that questions. [26:50] uh because when a vendor comes there's a [26:53] compliance team and asks certain [26:56] questions and and and and the third [26:59] party management team. So I feel like [27:02] one of the missing gaps in AI [27:05] accountability is really having [27:08] testing and validation, red teaming, [27:12] stress testing, continuously monitoring, [27:15] really understanding what the risks and [27:18] limitations are. Otherwise, it never [27:21] goes beyond check the box compliance [27:23] exercise. And that requires an [27:26] investment, right? investment bringing [27:29] those people who have skills to be able [27:31] to do that and who are secure enough to [27:34] challenge and ask questions uh because [27:37] there's an incredible rush to market and [27:41] very big competition. [27:43] So developers are h do have a big [27:46] responsibility reje and how how can we [27:49] incentivize them? What is what is in it [27:53] for them uh when there is this huge rush [27:56] to market uh you know from the [28:00] executives and organizations how can we [28:03] build that culture uh Joan is talking [28:06] about really deeply caring about systems [28:08] and thinking about potential harms. [28:11] Yeah, I think it's it's a it's actually [28:13] to be honest pretty simple if it is if [28:16] you can really demonstrate like setting [28:18] up a framework that adds to like [28:21] business value and really in a [28:22] measurable way demonstrate it. I think [28:26] it it kind of like works by itself to be [28:28] very honest. I think that the challenge [28:31] is and also again from a principal [28:33] perspective you can build anything you [28:34] want right but then if you cannot really [28:37] prove that you know the system that you [28:40] set out to build is doing what you want [28:41] it to build I mean it's totally useless [28:43] all of us can do all sorts of random [28:45] things but then to be able to actually [28:47] build a system that does what you set [28:49] out to do is actually an extremely [28:51] important thing and that's where the [28:53] point that you brought up regarding [28:55] having validation and testing frameworks [28:57] become really important when it comes [28:59] comes to AI systems, there is a gap to [29:02] be honest in a technological [29:03] perspective, right? Because we think [29:05] about largely AI systems as an extension [29:08] of software systems, right? validation [29:11] of software systems which tend to be [29:13] deterministic have like one kind of like [29:16] single control logic so to speak that's [29:18] like rigid is not applicable to like AI [29:21] systems which are like dynamic [29:23] underlying or like probabilistic system [29:25] so to speak that can go into like [29:27] totally new territories. So the the [29:30] existing kind of like software [29:33] verification validation frameworks need [29:35] to be extended dramatically right you [29:38] need to think about how do you kind of [29:40] test a single unit so if you talk about [29:41] multi- aent systems you need to talk [29:43] about testing an agent then you need to [29:46] start testing how agents interact [29:48] because interactions can lead to kind of [29:50] like non-trivial like outcomes that you [29:52] may not expect. How do you actually like [29:54] test the system at a system level [29:56] because a multiple like agents that [29:58] interact can actually have total like [30:01] totally new emergent behaviors? The [30:04] other stuff that you need to test for is [30:06] like all the safety characteristics. [30:08] What's the boundary condition and what's [30:10] the boundary that you want to stay [30:11] within and actually define metrics that [30:14] you can use to track. For example, [30:16] what's the frequency and severity of [30:18] like catastrophic outcomes that can [30:20] happen, right? And you need to test them [30:22] from a simulation perspective. Really [30:24] like understand the entire operating [30:27] regime of the system that you really [30:28] built because then you know what it can [30:31] do, where it can go wrong, how it can go [30:33] wrong. And once you actually [30:35] characterize all those things, if at all [30:37] things go wrong, you at least have now a [30:39] framework, a technological framework to [30:41] correct. And hopefully if you have a [30:44] really good governance framework, you [30:45] also have the organizational framework [30:48] to actually fix the stuff. Yeah. [30:49] >> Right. you never want to go there [30:51] because you want to kind of like you [30:53] know design a system to kind of like [30:54] avoid to stay within the boundary [30:56] conditions that you want the system to [30:58] operate but I think having a [31:00] technological framework plus kind of you [31:02] know like an organizational framework [31:04] also like helps you like iterate to kind [31:06] of develop robust systems so to speak [31:08] >> yeah it's it's really critical to have [31:10] those systems in place to design and uh [31:15] test uh and validate those systems to [31:18] really build confidence join And I do [31:20] have a question. Do you have something [31:21] to follow up on? [31:22] >> Well, only I was on a panel earlier [31:24] today where we were talking about [31:25] agentic AI and I think everything Raj is [31:27] saying is true except agents make it [31:29] even more more difficult [31:31] >> and I think that there's um additional [31:34] safeguards and additional governance and [31:36] additional [31:38] um uh frameworks that are going to need [31:41] to be built for the creation of agents. [31:43] And so what we're talking about kind of [31:46] brings us right to the edge of the [31:48] development of agents and we have so [31:51] many organizations now that are [31:52] beginning their journey on agentic AI [31:54] and and the creation of agents and I [31:57] think even the governance models we have [31:59] in place now need to be developed [32:01] further right because of those [32:04] challenges and so Raja as you were [32:05] speaking I was just thinking like some [32:07] of those techniques don't work for [32:09] agents right and so we have to go [32:11] further with our governance and further [32:13] with our thinking [32:14] and further with our values, right? So, [32:16] it's just like every time we think we [32:18] even know what we need to do, even if [32:20] we're not doing it, right? It's just it [32:22] the world keeps evolving on us. And so, [32:26] governance is going to become all the [32:27] more important, which is why it has to [32:29] be embedded throughout the organization. [32:31] It has to be part and parcel of core [32:34] processes, right? because the innovation [32:37] moment is happening so faster and [32:39] tighter tighter innovation um you know [32:42] iteration uh moments. So sorry it's just [32:45] kind of the [32:45] >> just add one thing. Um I agree with [32:48] everything that you said um you both [32:50] said but I want to um also draw our [32:53] attention that even though it is of [32:54] course exciting to talk about agentic AI [32:56] or gen AI a lot of our current uh [33:00] predictive AI traditional AI systems are [33:02] not well governed and we don't like [33:05] again I am saying this with ex having [33:08] experience working in with multiple [33:11] large companies yes we do some testing [33:14] in some teams but it is very rare for [33:16] large organizations to have the AI [33:19] governance this AI pipeline embedded [33:22] throughout the whole organization. So [33:24] every AI system that they develop, [33:26] deploy, procure go through the same [33:29] rigor of assessment. So I want and [33:32] knowing that most of the systems that [33:34] affect our lives today without us [33:36] knowing are the traditional AI systems. [33:39] I really don't want us to lose sight [33:40] that we are talking about governing or [33:43] doing responsible AI in the traditional [33:45] AI where we know how to do very very [33:48] well most of the time we it's not about [33:51] lack of knowledge where we we need to do [33:54] AI response AI for generative AI where [33:57] we know how to do quite well but there [33:59] are a lot of open research questions in [34:01] which case again you have all these [34:02] universities at your disposal to have [34:04] collaborate with and then the agentic AI [34:07] question where we have a lot of open [34:09] questions. So [34:11] >> yeah, very good point John. So and also [34:13] uh to add on you know doing it through [34:16] testing and validating maybe we don't [34:18] need to do that for all use cases right [34:20] we do that like initial assessments to [34:23] understand their uh risk levels if it's [34:26] used for more you know productivity and [34:29] efficiency gain maybe it doesn't need [34:31] that testing but for more highstake [34:33] decisions that can potentially create [34:35] far harm uh that's really necessary uh [34:38] Joanne you build governance systems that [34:42] works in complex organizations like [34:45] John. So I'm working with a lot of [34:47] clients and one of the trouble is [34:50] organizational alignment because when we [34:53] talk about AI governance there is large [34:55] number of stakeholders on the table and [34:58] and it is incredibly slow and and [35:02] frustrating. So what is the secret there [35:06] >> you know and I feel and so please [35:08] understand like my my fire lane has been [35:10] financial services for a really long [35:12] part of my career right and so [35:15] industries that have heavy compliance [35:17] burdens like healthcare and financial [35:19] services I hope you see more governance [35:22] than not right mostly because we are [35:25] vested in the outcomes being accurate [35:28] right compared to maybe you mentioned [35:30] entertainment I don't know a lot of [35:31] people in entertainment so Maybe maybe [35:33] it's a little looser, right? And maybe [35:35] we haven't done the things on our way in [35:38] this digital age, right? We've been [35:41] maybe a little bit more lazy fair. Shall [35:43] I use that phrase? How's that? Okay. So, [35:46] because I think you do you do think see [35:48] things that I have not experienced, but [35:50] I do get a chance um as a fellow to work [35:53] with other organizations, right? And [35:55] what I find is that you need to figure [35:58] out what the organization cares about, [36:00] right? What is in there? And I'm going [36:02] to be talking to some marketers about [36:05] this um tomorrow. And I do think that [36:08] we're at a moment where employees, CEOs, [36:13] customers are at a point where so what [36:15] matters in what's delivered is the hook [36:19] that has to be found that then you [36:22] translate for governance. So the why [36:25] you're doing it, right? Is it for [36:28] customer service? So at Mastercard, it's [36:29] easy, right? We care about security. [36:31] Security has to be dropped one. If you [36:34] don't think the network is secure when [36:36] you tap, swipe, key in those numbers, [36:39] right? All the things. Use your phone. [36:42] If you don't think it's secure, you're [36:43] not going to use your Mastercard. You're [36:44] just not, right? The minute that goes, [36:46] right? That that's it. Game over. If you [36:49] think constant fraud is going to happen, [36:51] game over. So, there's certain [36:53] hierarchical needs in every [36:56] organization, you've got to tap into [36:58] that. Those become the organization's [37:01] values. Now, I know you said principles [37:03] don't matter, but the principles got to [37:04] flow from the values, right? Because the [37:06] values have to be why is the [37:08] organization in existence? Why should it [37:10] continue to be in existence? So, that's [37:12] where we always start, right? The [37:14] conversation because that has to be how [37:17] you get everybody, all the stakeholders [37:20] engaged. Now, it's not the same [37:21] conversation for everybody, right? But [37:23] that's usually because usually there's a [37:25] point in pride of what an organization [37:28] delivers why it is exists right that and [37:32] I think in the next generation when we [37:35] get to Agentic we have to create agents [37:37] that have that same kind of code of [37:39] conduct if you will what what are the [37:42] boundaries of agents and how they're [37:43] delivering in the same way that an [37:45] employee today a person delivers how are [37:48] we using agents in the same way so I [37:50] think those are some of the ways that [37:51] you get people to care and then you [37:54] connect it to the processes and programs [37:57] that are part of the DNA of an [37:59] organization. Those are the things that [38:01] I've seen that work the best and you can [38:03] start you you always need sponsorship at [38:06] the top but you've got to work with the [38:09] processes that work for the people who [38:11] are doing it. [38:12] >> Yeah, that brings my next question. Uh [38:14] it's about the return of investment on [38:17] AI governance. What are some really good [38:20] metrics to keep track on to prove those [38:24] systems are actually helping creating [38:26] innovative and sustainable systems [38:33] >> I think [38:36] >> so we can get more executive [38:37] >> I think one of the I I think to be [38:39] honest I think when let's I think let's [38:41] maybe narrow it down right let's talk [38:43] about you want to build a system that's [38:45] that's that's safe right that is one of [38:47] the stuff that you want to do you don't [38:49] want you know you have a boundary [38:50] condition you don't want you know the [38:52] system to go outside so and I think in [38:54] the validation framework that I just [38:56] said you need to know like really know [38:58] everything about what's the probability [39:01] and the severity with which there is you [39:04] know what is the smallest probability [39:05] and the severity you know of any [39:08] potential you know bad outcome that you [39:10] at least start from there you need to at [39:12] least understand where things can go [39:14] wrong right again this is not rocket [39:16] science it's like a pure [39:18] hardcore kind of engineering mentality, [39:20] right? You just need to know what your [39:22] system can do and what it cannot do and [39:24] you then want to figure out a way to [39:26] protect yourself and put guard rails [39:28] when in you know the unlikely event that [39:31] something can be breached. So I think [39:33] you kind of start from there and I think [39:36] over a period of time when you start to [39:38] kind of like measure you can actually [39:40] now do like you know causal inference [39:42] testing right you can actually say what [39:44] if you had not done XY Z you know what [39:47] would have happened versus you know [39:49] because of you doing all these things [39:51] what is kind of the state of reality [39:53] what's the status quo at present and [39:55] then you can actually start to kind of [39:57] like convert that from you know a [39:59] tangible value to be honest so I think [40:02] It's again all these things that I'm [40:03] mentioning sometimes is I'm like a [40:05] broken record because this is all you [40:07] know solid engineering framework but [40:09] again like all like like many things [40:11] that the many people on the panel have [40:13] said is that there are a lot of [40:14] different components that come into play [40:17] that makes these things a bit [40:18] challenging but I think if one kind of [40:20] takes a step back and actually thinks [40:22] about what to do and how to do it's it's [40:24] actually very doable and many [40:26] organizations have done it and I agree [40:29] that you know governance needs to be [40:30] kind of like upgrade ated in kind of the [40:32] multi- aent world but at least like [40:35] having a nucleus of governance that can [40:37] actually like you know help you know [40:39] tackle the mundane you know data times [40:42] the traditional AI stuff you already can [40:44] start with the nucleus to actually like [40:45] expand and upgrade so it doesn't mean [40:47] that you always have to have the shiny [40:49] governance framework to get go I think [40:52] have something that kind of works and [40:53] adds value today and then you can [40:55] actually upgrade it based on the needs [40:57] that you that you really have you know [40:59] from a business perspective [41:01] >> over time. [41:02] >> Exactly. Yeah. [41:02] >> Some of the I think um concrete ways of [41:05] thinking about the return of investment [41:07] is how first of all if you do it well [41:10] you actually see that the it the [41:12] innovation gets faster. So smart [41:15] companies smart organizations pay [41:17] attention to what was their process how [41:20] many people used to be involved in [41:22] doublechecking and now how fast and how [41:24] leaner their process become. I think [41:26] that's one way of quantifying the return [41:28] of investment and the other way which is [41:30] um something that some of my clients [41:33] said and I was uh first I heard I was a [41:35] little surprised but then of course it [41:36] makes sense whenever we work so we do [41:38] response by meaning we do AI governance [41:40] but also we um a lot of the work that we [41:43] do is clients bring us um a particular [41:46] product that they are working on or they [41:47] just procured and say is this is this [41:50] safe is this responsible can we deploy [41:51] it like a very product specific thing [41:54] and one of the things that we heard [41:56] repeatedly is that once we finish our [41:58] assessment that also gives them an [42:00] assessment of what their competitors are [42:02] not doing. So the smart organizations [42:06] pay attention to their competitors and [42:08] see where they are ending up and [42:10] oftenimes that's where you can quantify [42:13] how much trouble that your competitors [42:15] are getting into eventually. It might [42:17] not be immediately apparent because well [42:19] the lawsuits don't come like the day off [42:21] but the risk how much um both trust you [42:25] know the client trust and customer trust [42:27] that you are losing and the fluctuation [42:29] happens and the actual litigation that [42:32] happens gives you a way of quantifying [42:34] as well [42:35] >> and then another way to think about it [42:36] is kind of from a product liability [42:39] perspective. So governance is kind of [42:41] the new way of um reducing defects in [42:44] your products. And so um there's an an [42:48] alternative school of thoughts. I agree [42:49] with everything that's been said. So you [42:51] can absolutely look at it from an [42:52] efficiency perspective. You can look at [42:54] it from an kind of return on the product [42:56] development side, but there's also the [42:58] issues of product liability and how you [43:00] are reducing the defects in your [43:03] products. um and that and that you are [43:05] actually then improving the actual [43:08] concept of delivery and so but how are [43:11] you measuring that? So you can measure [43:13] it in trust, you can measure it in [43:14] effectiveness, you can measure it in [43:16] future problems, lawsuits and how it [43:20] performs in the marketplace. So your [43:21] market share, customer satisfaction, [43:24] right? Um retention of of of customers, [43:27] um features, new features that you're [43:29] rolling out. So all the metrics that you [43:31] would normally measure in any new [43:33] product or solution um again depends on [43:36] what you do. So it's harder, you know, [43:37] it's easier to kind of do it when you [43:39] have our concrete manufacturer, but all [43:41] of those things that's actually what [43:43] you're doing with governance. It's a way [43:45] of reducing product liability. [43:48] >> Great. Do we have time to take one [43:49] audience question? [43:51] >> We don't. Okay. Well, um well, thank you [43:55] so much. This was really a great panel [43:57] and some of the key point uh key [44:00] takeaways that we have governance is is [44:03] complex. It can be challenging. However, [44:07] there are things to do it uh well and [44:09] better designing well tests uh really [44:12] budgeting and and focusing on um you [44:16] know things like AI literacy and and [44:19] having the right people in the room and [44:22] and and and my personal uh view is that [44:25] we can all have an impact on the AI [44:28] systems our organizations are developing [44:31] and ask critical questions ask you know [44:34] what could be potential harm arm with [44:36] that system and and also not only at [44:39] your organizations. I think there is way [44:41] to contribute with your local [44:43] communities uh too. Whenever you use a [44:47] product just uh ask questions and demand [44:51] responsibility and accountability uh [44:54] from those organizations they are [44:55] offering those products because those [44:57] are the kind of things that will make [44:59] and motivate organizations pay more [45:01] attention uh to develop uh products that [45:05] are kind of where the responsibility and [45:08] accountability and safety is really [45:10] embedded in the design like Rajes is [45:13] talking about. Again, thank you so much [45:15] and thank you to our panelists for their [45:17] uh wonderful point of views and joining [45:20] today.