Mastercard, F5 Tie AI Governance to Faster Enterprise Deals

Mastercard 与 F5 将 AI 治理与加速企业交易挂钩 将分层风险分流与透明度融入现有采购渠道，可将 AI 治理从运营瓶颈转变为加速企业交易周期的商业资产。 资深数据、法律及隐私负责人阐述了在不阻碍业务创新的情况下推行 AI 治理的实用策略，倡导采用可直接融入现有合规与企业架构工作流的分层风险准入模型。讨论涉及了核心部署障碍，包括区分 AI、隐私和安全之间各自独立的“正交”领域、应对本地与云端治理的不同预期，以及为开放式生成式 AI 系统建立评估基准。与会专家强调伦理必须植根于持久的企业价值观而非短暂的炒作周期，并展示了稳健的治理如何作为积极的商业赋能因素发挥作用——通过加快采购周期、避免代价高昂的返工并建立竞争信任，实现从“只会说‘不’的部门”到“深谙业务的‘知悉’部门”的转变。 万事达卡（Mastercard）阐述了有效 AI 治理的五大支柱：定制化政策框架、嵌入现有流程的低摩擦准入、基于影响的分级审查（约80%的非人身影响案例走轻量通道）、标准化缓解方案及通过 MLOps/LLMOps 实现的持续可观测性。 (2:38) 雅诗兰黛（The Estée Lauder Companies）通过跨职能风险委员会与月度企业 AI 工作组推动治理落地，并结合每年全员合规培训与针对具体工具的提示词及边界培训，满足欧盟《人工智能法案》（EU AI Act）的 AI 素养要求。 (6:05) F5 指出 B2B 环境中 AI 透明度能显著加速交易周期，并提出了按部署架构分层的模型治理三原则：本地私有部署赋予客户完全控制权，云服务则由供应商统筹并提供更新预警与审查窗口。 (10:55) 赛诺菲（Sanofi）强调在自动化高风险决策中需建立明确告知与申诉机制，并通过发掘内部具备多元学科背景的员工组建伦理倡导者网络，同时借助结构化偏差追踪来妥善处理全球政策与各地区法律之间的冲突。 (25:55) 专家小组提出将合规职能从阻碍创新的“NO 办公室”转变为赋能业务的“KNOW 办公室”，通过缩短采购接入时间、数据科学团队获取数据的速度以及加速 B2B 合同谈判等非财务指标来量化治理成效。 (40:54)

Enterprise AI governance functions most effectively when integrated directly into existing operational workflows and treated as a commercial enabler rather than an administrative roadblock, according to data and privacy executives from Mastercard, F5, Sanofi, and Estée Lauder. They argue that scalable oversight requires impact-based risk tiering, clearly demarcated compliance boundaries, and continuous alignment with core corporate values. Streamlining Governance Through Impact-Based Risk Triage (2:38) Andrew Reiskind (Mastercard) and Bridget McIlveen (Estée Lauder) emphasize that effective governance avoids creating redundant administrative workflows, instead embedding risk checks directly into existing product, legal, and procurement channels. Mastercard routes approximately 8% of its AI use cases through an expedited review track because they touch internal operational systems rather than directly affecting individual humans. McIlveen similarly advocates for high-, medium-, and low-risk tiers aligned with frameworks like the European Union AI Act, allowing lower-risk internal tools to proceed through standard channels while reserving specialized multi-disciplinary reviews for high-stakes applications. To avoid stalling operational momentum, McIlveen advises organizations against waiting for specialized AI governance platforms to reach the market. Instead, compliance teams should leverage existing enterprise software already deployed for vendor risk assessments, data mapping, and asset inventories to establish an immediate baseline for cataloging models and data pipelines. Defining Orthogonal Mandates Across Risk Functions (34:37) As AI initiatives scale across business units, peer risk functions frequently suffer from boundary confusion over domain ownership. Tom Johnson (F5) frames organizational compliance as an 'orthonormal basis'—collaborative yet distinct. In this framework, AI governance specifically evaluates whether an algorithmic model is fit and appropriate for a designated purpose; privacy governs what underlying data is ingested, focusing on personal data protections; and cybersecurity defends models and data pipelines against unauthorized disclosure, tampering, or exfiltration. Operationalizing these boundaries requires cross-functional coordination. Estée Lauder conducts high-risk evaluations through joint reviews that unite privacy, cybersecurity, legal counsel, and enterprise architecture, supplemented by monthly cross-departmental task forces spanning HR, R&D, and finance. Xavier Pouts (Sanofi) highlights the value of recruiting an internal network of multidisciplinary 'ethics allies' alongside regional privacy officers to identify localized legal conflicts and feed documented regional deviations back into global policy baselines. Aligning Transparency with Deployment Architectures and Speed (9:04) Transparency directly influences commercial outcomes, particularly in enterprise B2B environments where clear data boundaries accelerate customer procurement cycles. F5 governs customer trust through three core deployment principles: providing full visibility into models, training datasets, and testing parameters; granting customers absolute operational control over software running on private on-premises infrastructure; and maintaining provider control over cloud-hosted platforms while providing clients with advance notifications and review windows prior to deploying model updates. Acceptable model release cadences depend heavily on sector-specific constraints. Johnson observes that while retail anti-fraud models can be refreshed bi-weekly to counter rapidly shifting attack vectors, financial institutions operating under rigid Model Risk Management standards reject frequent, unvetted model churn. Reiskind and Johnson urge compliance teams to reposition themselves as the 'office of KNOW,' tracking governance efficacy through operational velocity metrics including enterprise procurement intake speed, dataset access turnaround times for data scientists, and reduced contract negotiation cycles. Addressing Generative Uncertainty and Quantifying Governance ROI (28:30) Unlike classical machine learning models that evaluate structured data against established statistical baselines, generative AI introduces qualitative ambiguity regarding when an open-ended output is safe enough for production. Reiskind notes that current industry reliance on question-and-answer benchmarks and subjective user feedback loops leaves testing cohorts and safety thresholds poorly defined. To mitigate deployment risks, Mastercard equips engineering teams with standardized supervisory toolkits, placing independent secondary monitoring models and continuous meta-monitoring systems over generative outputs to identify drift and bias. Demonstrating the value of governance requires measuring both process maturity and financial return on investment. Pouts argues that maturity is demonstrated when an organization adheres to compliance standards consistently across all deployments while shortening intake cycles. Financial returns are tracked through hard cost avoidance—such as mitigated regulatory fines, lower outside counsel expenses, and eliminated development rework—as well as the 'delta of trust,' in which transparent and ethical AI practices directly win competitive enterprise procurement bids. While structural intake workflows and multi-layered monitoring establish operational control, panelists emphasize that quantitative evaluation standards for generative AI remain immature across the industry. Consequently, enterprises must benchmark evaluation methods with industry peers and anchor high-stakes decisions in enduring corporate principles, accepting that regional legal deviations require flexible operational feedback loops rather than rigid, static mandates.

来自Mastercard、F5、Sanofi以及The Estée Lauder Companies的合规与数据主管指出，企业AI治理的核心在于通过分级风险评估与复用既有流程消除组织阻力，将合规职能从传统的业务卡点转化为加速交易与建立信任的赋能杠杆。 治理架构支柱与轻量化风险分流 (2:38) 根据Mastercard首席数据官Andrew Reiskind的经验，AI治理没有所谓的捷径，而是依赖严谨的基础合规架构，并在六年实践中持续通过“测试与学习”演进。有效的治理框架涵盖量身定制的政策、无摩擦接入、基于影响的分级分流、标准化缓解手册以及持续的MLOps与LLMOps可观测性。Mastercard发现企业约80%的AI用例并不直接影响个人，而是涉及后台运营和系统流程，这意味着绝大多数项目可以通过更轻量、更快捷的通道完成审查，使合规资源集中于信贷审批或自动化招聘等高风险领域。 Estée Lauder首席隐私官Bridget McIlveen同样指出，企业应借鉴EU AI Act的高、中、低风险分级逻辑，避免用同等严苛的标准拖慢所有采购流程。针对落地工具，她建议企业切勿坐等市场上出现专门的AI治理软件，而应直接依托现有的供应商评估、数据资产盘点及隐私审查工具，以极低的预算和组织摩擦快速搭建起资产登记与合规管线。 跨职能协同、素养培育与正交合规边界 (6:05) McIlveen强调，治理不能仅局限于IT或数据科学孤岛，Estée Lauder通过跨职能风险委员会（囊括隐私、网络安全、法律与企业架构）协同把关高风险项目，并通过每月一次的企业AI专班促进业务部门间复用资产。为落实EU AI Act对AI素养的要求，企业将伦理培训纳入年度必修课，并针对具体工具提供提示词规范和操作边界指导。Sanofi隐私与AI伦理官Xavier Pouts提出建立“伦理盟友网络”，发掘数字化团队中具备哲学等多元背景的员工，并借助熟悉当地法规的区域隐私官处理全球标准作业程序（SOP）落地时的地域冲突，通过记录偏差反哺全局基准。 F5的Tom Johnson指出，跨职能协同必须厘清安全、隐私与AI治理的界限。他以数学上的“正交基”（orthonormal basis）作比喻，强调三者虽需协作但各自独立：AI治理判断模型在特定场景下是否“合用且恰当”；隐私关注底层数据是否涉及个人信息及法律保护；安全则防范未授权泄露和外发，安全团队需要知晓模型部署，但无权判定算法适用性。明确边界有助于防止职能部门因职责模糊而陷入过度干预或推诿。 透明度分级与B2B商业转化交付 (9:04) Johnson表示，对于企业级B2B服务商而言，透明度直接决定了交易速度（deal velocity）；当买方能够清晰评估AI的数据边界与安全参数时，采购和安全审查周期将显著缩短。为此，企业内部需在工程、产品与法务之间就“模型”与“智能体（Agent）”等术语建立统一定义。在交付模式上，本地部署（On-Premises）与云端托管SaaS需采取截然不同的治理逻辑：本地客户要求对运行环境拥有绝对控制权，拒绝无通知替换模型或使用其私有数据回流训练；而在云端托管模式下，服务商因算力成本与架构限制必须自主掌控模型集群。 为平衡此矛盾，F5确立了三项原则：全面向客户披露模型及其训练与测试数据集；对于本地部署，完全由客户自主决定是否以及何时在本地运行模型；对于托管云服务，由F5统一部署维护，但上线新版本前必须向客户提前发送预警并保留评估窗口。Pouts则补充，透明度应视受众理解能力分级呈现，在涉及自动化招聘或晋升评估等高风险场景时，除明确告知受众外，还必须配备申诉机制以构建长效信任。 生成式AI评测瓶颈与金融业偏差监管 (19:33) Reiskind指出传统机器学习依靠结构化数据与成熟统计指标即可完成模型监控，而生成式AI开放式的生成特性导致量化基准缺失。企业普遍采用问答对（QA testing）及人工点赞/点踩反馈，但目前行业仍无法在客观上界定测试样本的覆盖广度，也缺乏“何种程度才算足够好”的投产标准。面对NIST等标准机构评估框架的逐步演进，业内领军团队多通过同行交流互相对标风险阈值。 在金融及科技金融领域，面对更严苛的模型风险管理（MRM）要求，频繁更换模型并不被监管接受。Reiskind的团队为数据科学家配备了标准化工具箱，包括QA基准压测、建立覆盖在生成输出之上的二级监督模型，以及对监控模型本身实施元监控（Meta-Monitoring），以持续防范模型漂移与偏差。治理团队还需评估宏观社会影响，防止向公共机构输出的宏观经济洞察误导政府公共资源在不同群体间的分配。 治理成效量化：从拒绝部门转向KNOW办公室 (36:58) Pouts提出以“成熟度”和“投资回报率（ROI）”来衡量治理成效。成熟度体现于组织能够“毫无例外、百分之百”地执行治理流程并提高审核可预测性；在财务层面，治理带来的硬性节省涵盖规避监管罚款、减少诉讼开支、压缩外聘律师账单以及消除非合规开发带来的返工损失。在收益层面，透明度带来的“信任增量”能直接帮助企业在竞标中胜出。Johnson指出，法务合规不应成为阻碍业务的“Say NO的部门”，而应打造成提供专业知识与赋能的“KNOW办公室”。 F5提出三项非资金量化指标来衡量治理的赋能效率：一是采购准入耗时，确保新增的AI审查不拉长企业采购周期；二是数据科学团队在清晰规则下获取训练与验证数据的速度；三是B2B交易速度，检验标准化AI合同条款及附录能否快速化解客户顾虑，从而避免拉锯式的红线修改。 该圆桌讨论达成共识，AI治理并非一次性的制度建设，而是随着全球技术与监管迭代的动态演进。尽管生成式AI的量化评价标准尚未统一，但专家一致认为，企业应当以核心伦理价值观作为锚点，借助现有的基础设施、分级评估和跨职能协同，在防范深层风险的同时最大化释放商业价值。

## Background 本内容由频道 Ai4 于 2025 年 8 月 28 日在 YouTube 平台发布。视频记录了一场关于构建有效 AI 治理框架的圆桌论坛，万事达卡、雅诗兰黛、F5 和赛诺菲的高管及专家共同探讨了如何建立合规、透明且负责任的 AI 治理架构以管理风险并减缓偏见。 ## TL;DR 来自金融、美妆、安全及制药跨国企业的四位专家，系统拆解了企业落地 AI 治理的核心支柱与实战挑战。讨论涵盖了政策接入、分级风险评估、AI 全生命周期可观测性，以及面对 B2B 客户和终端消费者时的差异化透明度策略。针对生成式 AI 带来的评测难题与全球合规分歧，嘉宾们强调应将 AI 治理与网络安全、数据隐私等正交维度清晰解耦，通过复用既有流程打消业务阻力。最终，治理部门应从以往习惯说“不”的合规卡点，转型为以提升采购、研发与商务签约效率为核心的业务赋能者。 ## Atmosphere & style 整场研讨氛围专业、务实且不失幽默，四位深耕一线的法务与数据高管打破了枯燥的条文说教，分享了大量接地气的实战心得。主持人开场以身处拉斯维加斯仍有高上座率的调侃迅速破冰，随后专家们通过“正交基”、“去毛伊岛度假”等生动比喻让抽象的合规概念变得通俗易懂。讨论节奏紧凑连贯，多方视角的碰撞既展现了强监管行业面对技术爆发时的严谨审慎，又传递出以开放协作赋能业务创新的积极底色。 ## Key takeaways - [02:38](https://www.youtube.com/watch?v=ALozv_rtWVM&t=158s) 建立企业级 AI 治理体系并无神秘捷径，核心在于围绕量身定制的政策、复用现有接收入口、分级风险评估以及标准化缓解与 MLOps/LLMOps 可观测性这四大支柱稳步推进。 - [02:38](https://www.youtube.com/watch?v=ALozv_rtWVM&t=158s) 风险评估的首要原则是判断用例是否对自然人产生直接影响；实践表明约 80% 的业务用例对个人无实质影响，轻量级审核能够大幅优化合规资源配置。 - [06:05](https://www.youtube.com/watch?v=ALozv_rtWVM&t=365s) 跨职能利益相关方协同与全员 AI 素养（AI Literacy）是治理落地的关键保障，需通过高风险用例联合评审与常态化工作组打破业务孤岛，并为员工提供针对具体工具的局限性与合规边界培训。 - [09:03](https://www.youtube.com/watch?v=ALozv_rtWVM&t=543s) 在 B2B 场景中，高透明度能显著提升商务签约与采购流转速度（Deal Velocity）；但需区分部署形态，在本地部署中赋予客户绝对掌控权，在云端托管服务中建立提前预警与评审窗口。 - [13:36](https://www.youtube.com/watch?v=ALozv_rtWVM&t=816s) 面向不同受众的透明度策略需差异化设计：专业机构沟通侧重技术与数据来源，面向终端个人用户则需极简透明，在自动化决策（如 HR 招聘晋升）中还必须配套人工复核与申诉通道。 - [16:03](https://www.youtube.com/watch?v=ALozv_rtWVM&t=963s) 模型更新节奏与行业监管要求存在天然张力：零售电商偏好高频迭代的短生命周期防欺诈模型，而金融银行等强监管领域受制于模型风险管理（MRM）无法承受频繁变更，技术供应商的核心职责在于充分告知赋能而非越俎代庖判定合规。 - [19:33](https://www.youtube.com/watch?v=ALozv_rtWVM&t=1173s) 针对生成式 AI（GenAI）应用，企业应搭建多层次自动化测试与双层监控（Monitor the Monitor）工具箱，赋能一线数据科学团队在上线伊始即自带防偏见与审计能力。 - [22:09](https://www.youtube.com/watch?v=ALozv_rtWVM&t=1329s) AI 伦理原则需自顶向下与公司价值观及 NIST AI RMF 等权威框架对齐，并进一步细化为涵盖训练数据代表性、决策审批权限及第三方供应商风险管理（TPRM）的具体操作指南。 - [25:55](https://www.youtube.com/watch?v=ALozv_rtWVM&t=1555s) 跨国企业推行治理应采取“全球统筹、局部调优”策略，充分发掘兼具技术与人文视角的内部盟友，并将区域合规偏离（Deviation）反哺为检视全局合规架构的参考基准。 - [28:30](https://www.youtube.com/watch?v=ALozv_rtWVM&t=1710s) 传统基于统计离群值的可观测性方法在 GenAI 时代失效，因缺乏公认的量化基准与数学评测工具，企业与行业正普遍面临“到底何时才算足够好”的评测与基线定义困境。 - [31:15](https://www.youtube.com/watch?v=ALozv_rtWVM&t=1875s) 推进 AI 治理面临变革管理、把握适度平衡与缺乏统一工具三大瓶颈，最佳落地破局思路是“借力打力”，将 AI 审查深度嵌入既有的法律、隐私及企业架构评审流程。 - [32:45](https://www.youtube.com/watch?v=ALozv_rtWVM&t=1965s) 根据风险级别（高/中/低）建立分级治理审查机制，低风险走常规通道，高风险执行深度审查，并优先盘活现有合规资产盘点系统以避免前期沉重的工具采购成本。 - [34:35](https://www.youtube.com/watch?v=ALozv_rtWVM&t=2075s) AI 治理、网络安全与数据隐私是三个维度正交的合规支柱：安全聚焦未授权共享与攻击防护，隐私聚焦个人数据合法性，AI 治理则专属评估模型在具体业务场景下的适用性与合理性。 - [37:12](https://www.youtube.com/watch?v=ALozv_rtWVM&t=2232s) AI 治理成效可通过成熟度模型（100%按标准执行、流转敏捷）与多维投资回报率（硬性规避罚单与助力大单中标，软性提升信任溢价与留存）进行量化度量。 - [40:54](https://www.youtube.com/watch?v=ALozv_rtWVM&t=2454s) 合规团队必须从总是说“不”的阻碍者转型为“知晓并赋能之所”，通过采购流转速度、数据科学家取数速度及商务成单速度等效率指标来衡量治理的最终成功。 - [43:24](https://www.youtube.com/watch?v=ALozv_rtWVM&t=2604s) 尽管外部 AI 技术与法律监管日新月异，但在面对充满不确定性的治理决策时，始终回归并坚守企业自身底层的核心伦理与价值观才是最具确定性的导航原则。 ## Detailed walkthrough ### [00:04](https://www.youtube.com/watch?v=ALozv_rtWVM&t=4s) 嘉宾介绍与 AI 治理框架讨论背景 论坛在拉斯维加斯会议的最后一天拉开帷幕。主持人、万事达卡（Mastercard）首席数据官 Andrew Reiskind 开场幽默地表示，在大会第三天兼最后一天、且身处拉斯维加斯的情况下，现场能有如此高的出席率，着实让他感到“惊喜与欣慰”。随后，四位拥有深厚数据、法务与伦理背景的行业专家依次进行了自我介绍： ```mermaid flowchart LR accTitle: AI治理专家小组成员及其业务背景 subgraph Panelists["AI 治理研讨专家组成员"] direction TB AR["Andrew Reiskind<br/>万事达卡 (Mastercard)<br/>首席数据官"] TJ["Tom Johnson<br/>F5<br/>杰出工程师兼法务高级总监"] XP["Xavier Pouts<br/>赛诺菲 (Sanofi)<br/>隐私与 AI 伦理官"] BM["Bridget McIlveen<br/>雅诗兰黛 (Estée Lauder)<br/>首席隐私官"] end ``` - **Andrew Reiskind（万事达卡，首席数据官）**：除负责传统的数据治理、数据管理及数据质量工作外，还统领数据战略（确保业务部门理解如何利用数据和 AI 开发产品）以及万事达卡的 AI 治理团队。 - **Tom Johnson（F5，杰出工程师兼法务高级总监）**：领导产品法务（Product Counsel）团队，担任涵盖隐私和 AI 的全方位数据领域主题专家（Data SME）。他介绍 F5 是一家 B2B 应用交付与安全平台企业，致力于帮助客户安全、可靠且大规模地交付 Web 和移动应用程序及 AI 工作负载。 - **Xavier Pouts（赛诺菲，隐私与 AI 伦理官）**：来自加拿大蒙特利尔，拥有深厚的法律顾问、数据隐私和网络安全背景，目前在财富 500 强跨国制药企业赛诺菲负责隐私合规与 AI 伦理事务。 - **Bridget McIlveen（雅诗兰黛集团，首席隐私官）**：雅诗兰黛作为全球顶尖的护肤、彩妆、护发及香水制造与营销巨头，业务涵盖广泛的 B2C 及部分 B2B 场景。作为首席隐私官，她深度参与集团整体的 AI 治理工作。 --- ### [02:38](https://www.youtube.com/watch?v=ALozv_rtWVM&t=158s) 构建有效 AI 治理框架的核心支柱 Andrew Reiskind 率先抛出核心议题，拆解了一个行之有效的 AI 治理项目应具备的基础模块。他直言，建立 AI 治理体系“并没有什么秘密配方或神奇魔法（no secret sauce, no magic）”，本质上是大量扎实的工作，并且完全可以借鉴合规项目诞生以来的既有经验。万事达卡早在约六年前就启动了相关体系的搭建，但面对《欧盟人工智能法案》（EU AI Act）以及生成式 AI（GenAI）的爆发，治理框架必须保持动态演进，遵循“边测试边学习（test and learn）”的迭代思维。 ```mermaid flowchart TD accTitle: 企业 AI 治理核心闭环流程 subgraph Intake["1. 需求与接收入口"] P["制定业务场景政策<br/>(Policy)"] --> I["复用既有流程接入<br/>(Intake Process)"] end subgraph Assessment["2. 评估与标准化处置"] I --> RA{"风险评估<br/>(Risk Assessment)"} RA -- "80% 无个人影响" --> LR["轻量化快速审核"] RA -- "高风险 / 影响个人" --> SM["标准化风险缓解措施<br/>(Standard Mitigation)"] end subgraph Monitoring["3. 持续观测与运维"] SM --> OBS["AI 全生命周期可观测性<br/>(MLOps / LLMOps)"] LR --> OBS end ``` Andrew 将整个治理体系拆解为四大关键环节： 1. **政策制定（Policy）**：必须结合企业具体的 AI 应用场景进行量身定制。评估企业究竟面临欧盟界定的高风险场景（如信贷审批、人力资源招聘），还是仅涉及低风险的内部客服工具。 2. **业务接入机制（Intake Process）**：实现政策落地的第一步是建立接收入口。切忌为业务团队凭空增加全新负担，而应直接复用企业已有的产品开发流程或法务审批入口，提高运营效率。 3. **分级风险评估（Risk Assessment）**：万事达卡在评估中最核心的首要问题是——“该应用是否对自然人个体产生实际影响，抑或仅仅作用于底层系统与运营流程”。实践表明，大约 80% 的业务用例完全不会对个人产生直接影响，识别出这一点后即可采用轻量级审查，极大节约合规资源。 4. **标准化缓解措施与 AI 可观测性（Standard Mitigation & AI Observability）**：针对存在实质风险的项目，提前制定标准化的缓解举措，业务团队甚至可以在提报前主动对照执行。在系统上线后，必须将 AI 可观测性深度融入 MLOps 和 LLMOps 运维体系（特别是在采用第三方供应商方案时），确保模型始终在设定的治理参数边界内运行。 --- ### [06:05](https://www.youtube.com/watch?v=ALozv_rtWVM&t=365s) 跨职能利益相关方协同与全员 AI 素养培训 针对治理框架的补充，来自美妆巨头雅诗兰黛的 Bridget McIlveen 指出，除了政策、风险评估与监控之外，另外两个不可或缺的关键支柱是“利益相关方协同（Stakeholder Engagement）”与“全员教育（Education）”。AI 绝不再局限于 IT 或数据科学团队的单一孤岛，而是高度跨职能的体系工程。 雅诗兰黛在实践中通过两套机制推进跨部门协同： - **高风险用例联合评审**：在接入与风险评估环节，针对高风险用例，拉通网络安全、企业架构（Enterprise Architecture）、数据隐私与法务团队展开多维度的综合审查。 - **常态化 AI 工作组（AI Task Force）**：汇集人力资源（HR）、研发（R&D）、财务等各业务线代表，每月召开一次例会，共同同步各部门的 AI 在研项目、沉淀实战经验，打破部门壁垒并实现企业级经验复用。 在教育与意识普及方面，雅诗兰黛将 AI 合规纳入全员年度必修认证培训计划。同时，为严格响应《欧盟人工智能法案》关于“AI 素养（AI Literacy）”的法定要求，集团要求每一款推向员工的 AI 工具都必须配备专门的操作培训，清晰告知员工“该工具的设计初衷与适用场景”，并明确提示其潜在局限与使用边界，杜绝误用滥用。 --- ### [09:03](https://www.youtube.com/watch?v=ALozv_rtWVM&t=543s) B2B 视角下的 AI 透明度与术语共识 Andrew Reiskind 随后将话题引向治理体系中的另一项核心原则——“透明度（Transparency）”。无论面向终端消费者还是企业级客户，透明度都是不可逾越的底线。他将话筒交给了身处 B2B 基础设施领域的 Tom Johnson。 Tom 从企业级服务商的角度剖析，透明度对于 B2B 企业而言最直接的商业回报就是“交易推进速度（Deal Velocity）”的提升。在企业级采购与法律谈判中，如果服务商能够清晰、透明地展示其技术架构与合规边界，将大幅缩短客户的安全合规审查周期。 要实现透明度，首先必须在企业内部筑牢术语基础。Tom 强调，法务、安全、知识产权（IP）团队必须与研发和产品团队坐在一起达成底层共识，精准定义“究竟什么是模型（What is a model）”、“什么是智能体（What is an agent）”，并据此确立统一的技术运营原则。此外，由于 F5 既提供部署在客户本地基础设施上的产品（On-prem products），又提供基于云端的托管服务（Cloud services），这两种部署形态在落地透明度承诺与履行合规要求时，必须采取差异化的治理策略与技术路径。 ### [10:55](https://www.youtube.com/watch?v=ALozv_rtWVM&t=655s) F5 针对模型控制与透明度的三大治理原则 Tom Johnson 从 B2B 基础设施与云服务的实战经验切入，阐述了客户在本地部署（On-prem）与云端 SaaS 场景下截然不同的诉求。当企业客户选择本地部署时，他们追求的是对运行环境的“绝对控制权”——绝不容许供应商在背地里擅自更换底层模型，不希望任何人都能随意下载模型，并且对自身数据被用来重新训练或生成新模型极其敏感。然而，在云服务体系中，F5 自身必须维持一定程度的统筹控制权。为了平衡双方诉求并明确产品与服务的硬性要求，F5 确立了三项核心治理原则。 ```mermaid flowchart TD accTitle: F5 AI 模型治理三大核心原则 subgraph P1["原则一：全流程透明知情"] A["明确告知客户：模型类型 / 训练与测试数据 / 嵌入的具体服务"] end subgraph P2["原则二：本地基础设施客户主控"] B["客户自主决定模型是否及何时在本地环境中执行与更新"] end subgraph P3["原则三：云端基础设施供应商管控"] C["供应商统筹有限算力资源<br>提前预警并提供评审期后再推向生产"] end P1 --> P2 P1 --> P3 ``` 第一项原则是**透明知情**：客户必须清楚获知所使用的具体模型、模型的生成方式、用于训练和测试的数据来源，以及这些模型具体嵌入到了哪些业务服务中。第二项原则是**本地主权**：客户对其自有基础设施上运行的模型拥有完全的决定权，可自主选择是否以及何时在本地环境中执行该模型。第三项原则则是**云端管控**：F5 必须掌管自身云端基础设施上运行的模型。毕竟算力和硬件资源是有限的，无法无限制地为每一个客户托管其专属的任意模型；因此，F5 的做法是通过提前发布告警通知，为客户预留明确的评审周期，并在模型更新推向生产环境之前向客户充分披露变更。 - **原则一（全面透明）**：向客户披露模型架构、生成流程、训练/测试数据集及落地服务。 - **原则二（本地控制）**：客户对本地环境（On-prem）拥有绝对自由度，决定模型的部署与执行时机。 - **原则三（云端治理）**：供应商管控云端算力与模型版本，设定预警与评审窗口机制以保证平滑过渡。 --- ### [13:36](https://www.youtube.com/watch?v=ALozv_rtWVM&t=816s) 医疗与跨行业视角：基于受众与企业文化的动态透明度 Xavier Pouts 随后从医疗健康及非纯 B2B 的广阔视角补充了对透明度的理解。他强调，建立治理机制的第一步是“认清你是谁，以及你在和谁对话”。透明度并不是千篇一律的技术参数堆砌，而是必须与企业的核心价值观高度契合。面对医院网络、专业机构与面对普通患者或消费者时，沟通策略必须截然不同：在面对技术同行或高度专业的 AI 工具时，固然可以深入技术细节，但即便对方是某一领域的专家，也未必能完全理解晦涩的模型内部机制，因此仍需适当降低专业门槛；而一旦面向普通个体，信息传递就必须做到极致的“极简且透明”（crystal clear）。 对于个人用户，企业必须清晰标识出他们正在与 AI 进行交互，并根据所在国家和地区的法律法规履行通知或获取授权的义务，同时 Xavier 坚信“主动获得用户知情同意（Consent）始终优于单纯的告知”。此外，在人力资源、招聘筛选或决定晋升等涉及自动化决策（Automated Decision-Making）的高敏感场景中，除了必须公示 AI 的介入外，更关键的是要配套建立完善的**申诉机制（Appeal Mechanism）**，允许员工提出“我认为我受到了不公正对待”的异议。这种透明与纠错机制从长远来看能够极大地巩固内外部信任，换来更高的员工忠诚度与企业声誉。 - **受众适配**：针对 B2B 专业客户可深入技术逻辑，面向终端个人用户则需通俗易懂、直奔主题。 - **交互告知与同意**：明确标识 AI 交互界面，倡导以“主动同意”为最高准则。 - **自动化决策与申诉权**：在人才选拔、晋升等 HR 决策链路中引入 AI 时，必须保留人工复核与申诉通道。 --- ### [16:03](https://www.youtube.com/watch?v=ALozv_rtWVM&t=963s) 风险评估实践：模型生命周期与行业节奏的天然冲突 回归到风险评估的实操层面，Tom Johnson 结合 NIST 人工智能风险管理框架（NIST AI RMF）阐述了模型全生命周期的治理逻辑。模型的生命周期从最初的应用构想、训练数据选取、特定算法训练、严格测试，一直延伸到最终的集成部署与业务运行。然而在 B2B 落地中最大的痛点在于：一个模型在某种业务场景下堪称完美，但在另一种客户场景下可能完全不适用。 Tom 以反欺诈模型为例：在零售电商或餐饮等行业，为了阻断新型攻击，安全团队往往会训练针对特定攻击手法的“短生命周期模型”，这些模型可能每两周就需要迭代更新一次，在快速平息攻击后即告淘汰，零售客户对此类高频更新欣然接受；但在银行和金融机构眼中，“两周一更”无异于一场灾难，金融监管合规要求严苛的模型风险管理（MRM）流程，根本无法接受如此高频的模型变动。因此在商务谈判阶段，双方往往需要就产品发布的节奏与周期展开深入协商。Tom 明确指出，供应商在风险管理中的核心职责是**充分告知与赋能**，而不是越俎代庖替客户评估业务风险，更不能提供合规法律意见；供应商的职责是提供详尽的材料与技术依据，由客户自行决断该模型是否符合其行业监管要求。 ```mermaid flowchart LR accTitle: 短生命周期模型在不同行业的适配冲突 A["攻击爆发 / 欺诈威胁"] --> B["训练短生命周期模型"] B --> C{"行业与客户场景"} C -->|"零售 / 餐饮客户"| D["支持高频更新 (每两周迭代)<br>迅速拦截并淘汰"] C -->|"银行 / 金融机构"| E["强监管与模型风险管理 (MRM)<br>无法接受频繁变更，需严格评审"] ``` --- ### [19:33](https://www.youtube.com/watch?v=ALozv_rtWVM&t=1173s) 金融级 AI 治理：群体偏见分析、社会影响与生成式 AI 工具箱 Andrew Reiskind 顺着金融场景展开深入剖析。对于银行及金融科技企业而言，两周一换模型的做法会让模型风险管理部门感到“极度不安”。金融行业的风险评估核心在于精准识别风险并设计针对性的缓解措施（Mitigation）。评估的首要问题是“该模型是否会直接影响到具体个人”；一旦涉及个人利益，团队就必须启动严格的偏见与歧视性分析，由数据科学家出具量化分析报告。Andrew 强调偏见分析不能“一刀切”，例如欺诈检测本身就需要捕捉异常行为特征模式，而这种模式判定如果机械套用到营销模型上就会产生不当歧视。 除了个人层面的微观影响，AI 治理还必须上升到**宏观社会影响（Societal Impacts）**维度。例如在为政府部门提供决策洞察时，必须防止数据偏差导致公共资源向特定群体倾斜或失衡。面对席卷而来的生成式 AI（Gen AI）浪潮，Andrew 的团队没有陷入被动应对，而是为内部数据科学家打造了一套标准化的风险缓解工具包（Standardized Toolkit）。 - **多层次测试与监控机制**：在部署大语言模型等生成式应用时，要求数据科学家生成问答测试对（QA pairs）并进行基准测试。 - **模型套模监控（Monitoring the Monitor）**：在业务模型之上架设第二层独立的监控模型来把控输出质量，同时再部署专门的监控机制去追踪监控模型自身的有效性。 - **标准化交付**：将上述自动化工具包直接分发给一线数据科学家，确保任何 Gen AI 应用在上线之初就自带标准化的防护与审计能力。 ### [22:09](https://www.youtube.com/watch?v=ALozv_rtWVM&t=1329s) 确立伦理原则与落地治理框架 风险评估的核心绝不仅仅停留在“识别评估”本身，更在于如何切实进行风险缓解。在监管合规层面，企业既要从严审视欧盟《人工智能法案》（EU AI Act）以及纽约市关于自动化就业决策工具等法规，更要在道德层面践行“善待客户”（do right by your customer）的黄金法则。对于像雅诗兰黛（The Estée Lauder Companies）这样高度面向消费者的企业，首席隐私官 Bridget McIlveen 指出，引入伦理考量的第一步是在全公司范围内明确负责任和合规使用 AI 的核心原则。这些原则必须涵盖模型从开发、部署到持续监控的全生命周期，形成书面文档并向全体员工深度传达，进而转化为具体的行动准则。 雅诗兰黛在多年前就制定了负责任使用 AI 的原则，其底层既紧密贴合企业的核心价值观与品牌特质，同时也对齐了主流的 NIST AI RMF（人工智能风险管理框架），重点聚焦透明度（Transparency）、公平性与偏见防范（Fairness/Bias）以及问责制（Accountability）。在确立高层原则后，团队进一步将其细化为具体的标准与控制措施：在公平性方面，严格要求训练与评估数据的准确性与代表性；在问责制方面，明晰组织内部对 AI 使用的决策审批权限；在透明度方面，将其融入第三方供应商风险管理（TPRM），要求服务商明确披露其提供的方案中到底运用了哪些 AI 技术。对于高风险用例，由跨职能专家团队共同进行伦理审查，利用多元视角补足单一团队的认知盲区。 - **原则到落地的层次架构**：高层道德原则与公司价值观绑定 $\to$ 细化为具体标准、指南与控制措施 $\to$ 嵌入日常研发部署与供应商风险审查。 - **核心审查维度**：数据代表性与准确度（公平性）、AI 决策归属权（问责制）、外部采购模型披露（透明度）。 ```mermaid flowchart TD accTitle: AI伦理原则落地与治理闭环 subgraph S1["核心伦理原则 (参考 NIST AI RMF)"] A["公司核心价值观与伦理底线"] B["透明度 / 公平性与无偏见 / 问责制"] A --> B end subgraph S2["控制落地与操作指引"] C["明确数据准确性与代表性规范"] D["明确内部 AI 决策审批权"] E["供应商 AI 风险评估与披露"] B --> C B --> D B --> E end subgraph S3["跨职能审查闭环"] F["高风险用例提交审查"] G["跨学科团队多元视角评审"] H["风险缓解措施实施与监控"] C & D & E --> F F --> G G --> H end ``` --- ### [25:55](https://www.youtube.com/watch?v=ALozv_rtWVM&t=1555s) 发掘内部盟友与应对全球合规差异 赛诺菲（Sanofi）隐私与 AI 伦理官 Xavier Pouts 认为，经典的 NIST 框架要素——“识别、保护、检测、响应、恢复”（Identify, Protect, Detect, Respond, Recover）完全可以且应当映射到 AI 伦理实践中。面对庞大组织中的治理诉求，Xavier 强调要积极发掘身边的“内部盟友”。大型企业内部往往隐藏着跨领域的复合型人才，例如某些身处数字化业务部门的员工可能本科主修哲学，这类人才既深刻理解业务运转与企业文化，又具备深厚的伦理思辨能力。将他们纳入企业伦理网络或赋予“伦理支柱”（Pillars）角色，能够直接在业务团队内部推动治理要求的生根发芽；同时，分布在各国的本地隐私专家也是重要盟友，他们深谙底层数据流转与本国法律红线。 在跨国企业的实际推行过程中，标准化的最佳路径是“全球统筹、局部调优”：首先建立集团统一的全局性标准作业程序（SOP）与伦理准则，随后向全球各个区域与具体国家推广。在下沉落地时，难免会有区域团队因当地严苛或特殊的法律法规而提出异议，声明“我们在本地无法这样执行”。此时治理团队应当规范地记录该区域的合规偏离（Deviation），并反向倒逼其他区域乃至集团总部进行复核反思：“我们在其他国家这么做真的万无一失吗？为什么该国会认定此举不可行？”通过这种持续记录、比对与回溯的闭环，逐步构建起坚实稳固且适应全球复杂监管的治理网络。 - **跨学科内部人才赋能**：将懂技术又具备哲学/人文背景的员工发展为去中心化的伦理大使。 - **本地化合规偏离管理**：针对各司法管辖区（Geo）记录合规例外，并将局部限制反哺为全局风险审视的参考基准。 --- ### [28:30](https://www.youtube.com/watch?v=ALozv_rtWVM&t=1710s) 生成式 AI 的评测困境：“何时才算足够好？” 传统 AI 时代与生成式 AI 时代在治理复杂度上存在断层式的差异。万事达卡（Mastercard）首席数据官 Andrew Reiskind 指出，在应对传统结构化数据的 AI 模型时，团队习惯于使用严谨的统计学方法，通过量化指标和简单的统计离群值（Outliers）即可迅速定位缺陷并实现清晰的模型可观测性（Observability）。然而，以大语言模型为代表的生成式 AI 彻底打破了这一确定性。目前业内普遍采用问答对测试（QA pairs）以及人工点赞/点踩（Thumbs Up / Thumbs Down）进行反馈评估，但根本性问题在于行业缺乏统一标准：“一百万用户的测试样本足够吗？测试人群是否足够多元化？数学家们至今尚未攻克生成式 AI 的量化评测难题。” 由于缺乏公认的标准，行业普遍陷入了“到底何时才算足够好（When is good good enough?）”的困惑。即便是像美国国家标准与技术研究院（NIST）这样全球顶尖的权威标准制定机构，其现阶段的行动计划也尚未给出完全成熟的量化度量工具。这导致企业的治理与技术团队不得不自行构建基准（Baselines）和内部工具集，并频繁向行业同行“取经”，通过跨企业的交流互鉴去了解彼此的评测方法与实践底线。Andrew 直言，正因为“正确答案会随着技术演进而不断变化”，如何定义合理有效的测试基线成为了当前 AI 治理中最棘手的动态挑战之一。 - **传统 AI vs. 生成式 AI 评测差异**：前者依赖结构化统计指标与离群值分析，后者依赖问答对、主观评分等缺乏数学确定性的质性手段。 - **行业基准缺失的现状**：官方标准机构尚未推出成熟的定量判定体系，企业高度依赖自主摸索与同行经验交流。 --- ### [31:15](https://www.youtube.com/watch?v=ALozv_rtWVM&t=1875s) 推进治理变革的三大瓶颈与破局思路 雅诗兰黛的 Bridget McIlveen 将企业在落实 AI 治理体系过程中面临的挑战归纳为三大核心瓶颈：变革管理（Change Management）、把握平衡（Striking the Right Balance / 避免“煮沸整片海洋”般好大喜功）以及工具化支持（Tooling）。其中，变革管理是最容易导致治理受挫的关卡。治理体系最忌讳被业务部门视为拖慢创新的“拦路虎”（Blocker）。为了化解这一阻力，推进治理的关键技巧在于“借力打力”，紧紧依附于企业内现有的成熟机制，而不是另起炉灶重新发明轮子。 企业内部通常已经拥有完备的法律评估、隐私合规审查或企业架构评审（Enterprise Architecture Review）流程。将 AI 评估要点嵌入这些既有轨道中，不仅能大幅减轻推进阻力，还能有效整合资源。此外，变革管理必须依赖持续深入的员工赋能与教育培训。治理团队在推出新的 AI 项目接入流程（Intake Process）时，必须向业务端清晰证明其带来的实际商业价值，而非单纯增加审批负担，让业务人员真切体会到合规接入能够帮助他们防范潜在技术翻车风险、保护品牌声誉，从而自发、主动地融入治理体系中。 - **推进 AI 治理的三大关键挑战**： 1. **变革管理**：避免成为业务阻塞点，打破部门推诿与抵触情绪。 2. **适度平衡**：明确优先级，防止过度治理导致资源耗尽（Don't boil the ocean）。 3. **工具化赋能**：缺乏统一且好用的自动化治理与监控工具支撑。 - **借道现有流程**：将 AI 审查深度融入法律、隐私保护及企业架构（EA）既有通道。 ### [32:45](https://www.youtube.com/watch?v=ALozv_rtWVM&t=1965s) AI治理的平衡分级与工具复用挑战 在探讨AI治理体系落地的难点时，布里奇特·麦基尔文（Bridget McIlveen）详细分析了另外两大核心挑战：在监管力度上“把握平衡”以及“治理工具的选型与部署”。面对日新月异的技术与法规，企业往往容易陷入一种“一刀切”的治理误区，试图用同一种审查标准去框定所有AI应用。对此，布里奇特指出，在风险评估与审查流程中，必须建立科学的风险分级机制（Tiering）。企业可以借鉴《欧盟人工智能法案》（EU AI Act）等监管框架，根据风险程度将AI用例划分为高、中、低三个等级；低风险项目可直接走标准采购流程，而高风险项目则需要经过增强型治理与深度风险评估。这种分级分类管理能有效精简审批链路，避免治理成为业务创新的阻碍。 针对工具选择的挑战，布里奇特建议企业优先盘活并复用现有的工具与系统，而不是一味等待昂贵的全新专用软件。无论是进行供应商风险评估，还是梳理数据、AI模型与系统资产清单，将AI治理嵌入现有的基础设施远比重新申请巨额预算采购全套新工具更具可行性。随着市场上针对AI治理的成熟工具不断涌现，先依托现有平台打好基础是目前最高效的落地路径。 - **分级治理机制**：参照《欧盟人工智能法案》等权威框架，将AI项目划分为高、中、低风险等级，低风险走常规采购，高风险执行深度审查。 - **工具链策略**：优先复用现有合规与资产管理工具（如数据盘点、供应商审查系统），降低初期落地门槛与预算压力。 --- ### [34:35](https://www.youtube.com/watch?v=ALozv_rtWVM&t=2075s) 正交的合规维度：厘清AI治理、隐私与安全的职责边界 围绕组织内部协同，汤姆·约翰逊（Tom Johnson）强调，将AI治理嵌入现有流程固然重要，但首要任务是对内部的合规与风险利益相关者进行深入的认知培训。在实际工作中，许多职能部门由于职责模糊而陷入困惑甚至恐慌——安全团队往往认为自己必须全权负责AI治理，而隐私团队则对自身是否需要全面接管AI深感焦虑。为了理顺各方分工，团队必须帮助不同职能部门明确其核心审查诉求。 汤姆用数学上的“正交基”（Orthonormal Basis）打了一个极其生动的比喻，指出AI治理、数据隐私与网络安全是三个彼此独立、维度正交的合规支柱，各自关注完全不同的核心问题： ```mermaid flowchart TD accTitle: 独立而协同的三大合规维度 subgraph GovernanceDimensions["合规三大独立支柱（正交体系）"] direction LR AI["AI治理职能<br/>• 核心审查：模型在特定业务场景下的适用性与合理性"] Sec["网络安全职能<br/>• 核心审查：防止未经授权的信息共享、泄露与攻击"] Priv["数据隐私职能<br/>• 核心审查：数据来源合法性、是否含个人信息及保护充分性"] end AI --- Sec Sec --- Priv Priv --- AI ``` 虽然安全团队需要知晓模型何时被部署，但他们并不负责判断该模型在业务逻辑与伦理上是否“适用”；隐私团队核心聚焦于输入/输出数据中是否包含个人信息（PII）以及是否有充分保护；而AI治理则专注于模型本身的适用性与可靠性。三者职责清晰界定、协同作战，才能避免多头管理或职责真空。 --- ### [37:12](https://www.youtube.com/watch?v=ALozv_rtWVM&t=2232s) 如何衡量治理成效：成熟度模型与硬/软投资回报率（ROI） 针对“如何证明AI治理项目真正发挥了实效”这一核心问题，格扎维埃·普茨（Xavier Pouts）直言：“如果无法衡量，它就等于不存在（If you cannot measure it, it doesn't exist）。”他提出可以从“治理成熟度”与“投资回报率（ROI）”两个维度来量化治理成效： ```mermaid flowchart TD accTitle: AI治理成效衡量体系 Root["AI治理成效评估"] --> M["维度一：成熟度指标 (Maturity)"] Root --> R["维度二：投资回报率 (ROI)"] subgraph MaturityMetrics["成熟度评估"] M1["执行一致性：100%按预定治理标准执行"] M2["响应敏捷度：业务流转加速、错误重工减少"] end M --> M1 M --> M2 subgraph ROIMetrics["ROI 量化评估"] R1["硬性财务收益 (Hard ROI)<br/>• 规避监管罚款与诉讼赔偿<br/>• 减少外部法律顾问咨询开支<br/>• 凭借高透明度赢得大额商业标案 (RFP)"] R2["软性战略收益 (Soft ROI)<br/>• 客户/患者信任度溢价提升转化率<br/>• 保护品牌声誉降低公关风险<br/>• 提升员工保留率与内部协作满意度"] end R --> R1 R --> R2 ``` 第一种评估方式是**成熟度模型（Maturity Level）**。它不单纯关注企业遗留了多少风险，而是衡量企业是否能做到“百分之百、毫无例外地在所有场景下做正确的事”。随着治理能力的成熟，团队会发现审批流转更加迅速，内部重工显著减少，各项合规动作能够100%兑现预期承诺。 第二种评估方式则是**投资回报率（ROI）**。尽管为治理量化财务价值存在一定难度，但可以通过多项硬性与软性指标来向董事会证明其商业价值： - **硬性成本节约与直接创收**：计算因完善治理而避免的监管罚单、诉讼开销、高昂的外部法律顾问费用，以及因流程优化而省下的员工返工工时；更关键的是，如果在大型招投标（RFP）中因为AI透明度和合规治理表现优于竞品而赢得大单，就应当将合同收益的一定比例归功于合规治理（格扎维埃幽默地调侃道：“拿着这笔战绩，你完全可以申请去毛伊岛度个假”）。 - **软性收益与信任溢价**：客户、患者及商业伙伴因为信任企业负责任的AI实践而选择留在平台，这种“信任增量（Delta of Trust）”可以直接转化为长期的客户生命周期价值，同时带来品牌商誉保护和员工留存率的提升。 --- ### [40:54](https://www.youtube.com/watch?v=ALozv_rtWVM&t=2454s) 从“拒绝之所”到“赋能部门”：通过业务效率衡量治理成功 安德鲁·赖斯金德（Andrew Reiskind）与汤姆·约翰逊进一步补充指出，AI治理不应被视作纯粹挑刺的合规监管部门，而必须转型为业务的“赋能部门（Enabling Function）”。安德鲁分享道，两年前曾有同事私下戏称法务合规团队是“总是说‘不’的拒绝之所（The Office of NO）”；为了打破这一刻板印象，团队索性将邮件签名改成了“知晓并赋能之所（The Office of KNOW）”，用专业知识为业务指路，而不是简单粗暴地下达禁令。 在日常业务中，治理团队的成效与业务效率直接挂钩，具体体现在三个维度的运转速度上： | 评估维度 | 核心衡量标准 | 业务价值体现 | |:--- |:--- |:--- | | **采购流转速度 (Procurement Speed)** | AI安全与合规评估是否拉长了软硬件采购周期 | 合规流程无缝嵌入，不成为引入新生产力工具的瓶颈 | | **数据获取速度 (Data Science Access)** | 数据科学家能否更合规、更快捷地拿到建模所需数据 | 政策清晰明确，减少审批扯皮，加速模型研发迭代 | | **商务签约速度 (Deal Velocity)** | AI附加条款（AI Addendum）是否能迅速被B2B客户接受 | 协议条款清晰透明，减少反复谈判摩擦，加速商务成单 | 当销售团队、数据科学团队和业务部门在推进AI项目后由衷感叹“天哪，这比我想象的要顺畅容易得多（Holy cow, that was way easier than I thought it was going to be）”时，就是AI治理取得最大成功的时刻。 --- ### [43:24](https://www.youtube.com/watch?v=ALozv_rtWVM&t=2604s) 闭幕结语：以不变的核心原则应对演进中的AI监管 在讨论的尾声，各位嘉宾总结了自己的核心观点，为构建长期有效的AI治理体系提供了战略建议： 布里奇特·麦基尔文总结道，AI治理是一场持续演进的马拉松旅程，随着全球法律法规与行业标准的不断出台，企业必须保持敏捷，让治理机制与技术和监管生态同步迭代。 格扎维埃·普茨则提醒大家关注“变与不变”的辩证关系：尽管社会舆论和大众对AI的认知变化极其迅猛，但企业的核心价值观与道德原则（Core Principles）是稳固且深厚的。面对瞬息万变、充满不确定性的治理难题时，回归并坚守企业的底层原则，往往能够为组织指明最正确的方向。 汤姆·约翰逊最后表示，AI治理之所以令人着迷，正因为每个具体模型的应用都需要融合技术、法律、安全、伦理等多维视角共同审视。一个优秀的治理体系不仅能防范未知风险，更能汇聚跨部门的智慧，为企业在AI时代的稳健前行保驾护航。 ### [44:49](https://www.youtube.com/watch?v=ALozv_rtWVM&t=2689s) 倡导深度跨界协作与圆桌论坛闭幕致辞 在讨论的尾声，嘉宾特别强调了不同领域、尤其是来自强监管行业（Highly Regulated Industries）从业者交流观点的重要性。发言者热情呼吁现场同行积极建立联系并开展跨界合作：“大家一定要主动站出来自我介绍、相互认识（Make yourself known, introduce yourself），让我们坐下来深入交流，把关于 AI 治理的对话持续进行下去。”在复杂且迅速变化的合规与技术环境下，正是这类深耕于严格监管场景下的实践经验与碰撞，才能为构建切实可行的治理框架带来宝贵启发。 主持人在最后对这一观点表达了由衷赞同（"I wholeheartedly agree with that last sentiment"）。为把时间留给参会者展开面对面的交流与自由探讨，主持人宣布本场圆桌研讨正式圆满结束，动员大家在会场内外相互交流心得，并祝愿各位充分享受峰会接下来的其他精彩议程。全场随即在热烈的掌声中落下帷幕。 - **推动行业交流**：嘉宾特别鼓励来自金融、医疗健康及相关强监管领域的专业人士主动对接、分享观点，打破组织壁垒以共同推进负责任的 AI 落地。 - **自由探讨与闭幕**：主持人顺应嘉宾的呼吁，宣布会议结束，鼓励与会者将研讨内容转化为线下的即时交流与合作网络。 ## Notable quotes > [02:38](https://www.youtube.com/watch?v=ALozv_rtWVM&t=158s) 并没有什么秘密配方或神奇魔法（no secret sauce, no magic） — Andrew Reiskind > [13:36](https://www.youtube.com/watch?v=ALozv_rtWVM&t=816s) 认清你是谁，以及你在和谁对话 — Xavier Pouts > [13:36](https://www.youtube.com/watch?v=ALozv_rtWVM&t=816s) 主动获得用户知情同意（Consent）始终优于单纯的告知 — Xavier Pouts > [22:09](https://www.youtube.com/watch?v=ALozv_rtWVM&t=1329s) 善待客户（do right by your customer） — Bridget McIlveen > [25:55](https://www.youtube.com/watch?v=ALozv_rtWVM&t=1555s) 我们在其他国家这么做真的万无一失吗？为什么该国会认定此举不可行？ — Xavier Pouts > [28:30](https://www.youtube.com/watch?v=ALozv_rtWVM&t=1710s) 到底何时才算足够好（When is good good enough?） — Andrew Reiskind > [28:30](https://www.youtube.com/watch?v=ALozv_rtWVM&t=1710s) 正确答案会随着技术演进而不断变化 — Andrew Reiskind > [31:15](https://www.youtube.com/watch?v=ALozv_rtWVM&t=1875s) 避免“煮沸整片海洋”般好大喜功（Don't boil the ocean） — Bridget McIlveen > [37:12](https://www.youtube.com/watch?v=ALozv_rtWVM&t=2232s) 如果无法衡量，它就等于不存在（If you cannot measure it, it doesn't exist） — Xavier Pouts > [37:12](https://www.youtube.com/watch?v=ALozv_rtWVM&t=2232s) 拿着这笔战绩，你完全可以申请去毛伊岛度个假 — Xavier Pouts > [40:54](https://www.youtube.com/watch?v=ALozv_rtWVM&t=2454s) 总是说‘不’的拒绝之所（The Office of NO） — Andrew Reiskind > [40:54](https://www.youtube.com/watch?v=ALozv_rtWVM&t=2454s) 知晓并赋能之所（The Office of KNOW） — Andrew Reiskind > [40:54](https://www.youtube.com/watch?v=ALozv_rtWVM&t=2454s) 天哪，这比我想象的要顺畅容易得多（Holy cow, that was way easier than I thought it was going to be） — Andrew Reiskind > [44:49](https://www.youtube.com/watch?v=ALozv_rtWVM&t=2689s) 大家一定要主动站出来自我介绍、相互认识（Make yourself known, introduce yourself），让我们坐下来深入交流，把关于 AI 治理的对话持续进行下去。 — 嘉宾 > [44:49](https://www.youtube.com/watch?v=ALozv_rtWVM&t=2689s) I wholeheartedly agree with that last sentiment — Andrew Reiskind ## Mentioned Andrew Reiskind, Tom Johnson, Xavier Pouts, Bridget McIlveen, Mastercard（万事达卡）, F5, Sanofi（赛诺菲）, The Estée Lauder Companies（雅诗兰黛集团）, EU AI Act（欧盟人工智能法案）, NIST AI RMF（美国国家标准与技术研究院人工智能风险管理框架）, MLOps, LLMOps, TPRM（第三方供应商风险管理）, MRM（模型风险管理） ## Video description Andrew Reiskind, Chief Data Officer, Mastercard Bridget McIlveen, Chief Privacy Officer, The Estee Lauder Companies Inc. Tom Johnson, Senior Director, Legal, Product Attorney | Distinguished Engineer, F5 Xavier Pouts, Privacy and AI Ethics Officer, Sanofi Abstracts This session will focus on the importance of establishing robust AI governance frameworks to ensure compliance, transparency, and accountability in AI systems. Panelists will discuss best practices for creating governance structures that manage risk, mitigate bias, and align AI deployment with ethical standards. Learn how to implement processes for monitoring AI models, ensuring data privacy, and maintaining regulatory compliance across various sectors. This discussion will provide actionable strategies for organizations looking to build or strengthen their AI governance and compliance programs. [Source](https://www.youtube.com/watch?v=ALozv_rtWVM)

[2:38] Andrew Reiskind: 并没有什么秘密配方或神奇魔法（no secret sauce, no magic） [13:36] Xavier Pouts: 认清你是谁，以及你在和谁对话 [13:36] Xavier Pouts: 主动获得用户知情同意（Consent）始终优于单纯的告知 [22:09] Bridget McIlveen: 善待客户（do right by your customer） [25:55] Xavier Pouts: 我们在其他国家这么做真的万无一失吗？为什么该国会认定此举不可行？ [28:30] Andrew Reiskind: 到底何时才算足够好（When is good good enough?） [28:30] Andrew Reiskind: 正确答案会随着技术演进而不断变化 [31:15] Bridget McIlveen: 避免“煮沸整片海洋”般好大喜功（Don't boil the ocean） [37:12] Xavier Pouts: 如果无法衡量，它就等于不存在（If you cannot measure it, it doesn't exist） [37:12] Xavier Pouts: 拿着这笔战绩，你完全可以申请去毛伊岛度个假 [40:54] Andrew Reiskind: 总是说‘不’的拒绝之所（The Office of NO） [40:54] Andrew Reiskind: 知晓并赋能之所（The Office of KNOW） [40:54] Andrew Reiskind: 天哪，这比我想象的要顺畅容易得多（Holy cow, that was way easier than I thought it was going to be） [44:49] 嘉宾: 大家一定要主动站出来自我介绍、相互认识（Make yourself known, introduce yourself），让我们坐下来深入交流，把关于 AI 治理的对话持续进行下去。 [44:49] Andrew Reiskind: I wholeheartedly agree with that last sentiment

[2:38] Andrew Reiskind: there's no secret sauce here. There's no magic. There's just a lot of hard work. [9:04] Tom Johnson: our biggest reward for successful transparency is deal velocity. [13:21] Xavier Pouts: Hey, I don't think that I was treated fairly for this and that reason. [22:09] Xavier Pouts: do right by your customer [28:30] Andrew Reiskind: When Is Good Good Enough? [36:58] Xavier Pouts: If you cannot measure it, it doesn't exist. [36:58] Xavier Pouts: 100% of the time all the time without exception. [36:58] Xavier Pouts: external counsel bills that are so small and easy to pay [40:54] Tom Johnson: the office of KNOW [44:49] Panelists: make yourself known, make yourself... introduce yourself, and let's talk and continue the conversation.

[0:04] Good morning everybody and I'm [0:06] pleasantly surprised by the level of [0:08] attendance given it's the third day of a [0:09] conference, the last day of it and it is [0:11] Vegas. Um so I'm Andrew Ricekin. I'm the [0:15] chief data officer for Mastercard. So um [0:19] what I will I will play the moderator [0:21] for the panel. Um but I hope it's uh [0:24] interactive and engaging for you. So um [0:28] I will first start off with uh letting [0:31] everybody introduce themselves and then [0:33] we'll get going to substance in pretty [0:35] short order. So as chief data officer at [0:38] Mastercard, I have your standard data [0:40] governance, data management, data [0:41] quality functions, but I also have data [0:44] strategy which is responsible for [0:46] ensuring that the business knows how to [0:48] develop uh products leveraging data and [0:51] AI. And then most relevant to this [0:53] conversation today, I lead up a AI [0:56] governance team. And so Tom, [0:58] >> I'm Tom. Uh I'm from F5. I am a [1:01] distinguished engineer uh and senior [1:04] director of legal at F5. I lead our [1:07] product council team. Um I operate as a [1:12] data SMEI, all things data, which [1:14] includes our privacy as well as our AI. [1:18] Um F5 just so everybody knows a little [1:22] bit about what it is. It provides a [1:24] application delivery and security [1:27] platform. Um what that does is it [1:30] enables customers to be able to uh [1:33] provide their web and mobile apps [1:36] securely um including their AI workloads [1:40] securely, reliably and at scale. So [1:43] we're B2B. [1:45] >> Thank you Xavier. [1:46] >> Hi Xavier. I'm the privacy and AI ethics [1:50] officer at Seni, which is a Fortune 500 [1:53] pharma company. Um, and uh, I've worked [1:57] in privacy, cyber security, and uh, [2:01] other connects uh, fields in the past as [2:04] a legal counsel and I'm from Montreal. [2:08] >> And my name is Bridget Meline. I'm the [2:10] chief privacy officer at the Estee [2:12] Lauder Companies. The Esteee Lauder [2:14] Companies is one of the world's largest [2:17] manufacturer, seller and marketer of [2:20] skincare, makeup, hair care, and [2:22] fragrance products. So, we have uh B to [2:25] C uh as well as some some B2B and in my [2:30] role of chief privacy officer, I'm [2:33] involved as well in AI governance [2:35] matters. So, happy to talk about that [2:37] today. [2:38] >> Great. Thank you. So I'm going to get us [2:40] kicked off with first just uh laying out [2:44] what are the parts of an effective AI [2:46] governance program and then we can do [2:47] deep dives as we go through this [2:49] conversation this morning. But I would [2:52] start w with simply saying there's no [2:54] secret sauce here. There's no magic. [2:56] There's just a lot of hard work. But [2:58] there's also hard work that's just based [3:00] upon what have been compliance programs [3:02] have been since the genesis of [3:05] compliance programs. We've stood up ours [3:07] about six years ago, but it's [3:09] continuously evolving and as we get to [3:12] the EUA act, it evolves. As we get to [3:14] Gen AI evol, it evolves and so you [3:17] always have to consider whatever you're [3:18] going to build, it will evolve and it's [3:20] okay. You will test and learn throughout [3:22] that entire process. So, first of all, [3:24] you need a policy. The policy should be [3:27] built on where are your uses of AI. You [3:31] really do need to personalize it to your [3:33] uses. Are you worried about high-risisk [3:35] use cases in the EU such as credit or [3:38] HR? Um, or are you really worried about [3:41] lower risk use cases just internal [3:43] customer service as an example? So, I [3:46] would start from uh what are the [3:48] policies that you need in order to make [3:49] that work, especially in light of how's [3:51] your policy framework structured in the [3:54] organization and then what you're going [3:56] to need is how do you operationalize it? [3:58] Well, that usually starts with an intake [3:59] process. So for your organization, what [4:02] is the best intake process? Everybody [4:04] has pre-existing product development [4:06] processes, legal intake processes. Go [4:09] leverage those. Don't create new work [4:12] for your stakeholders to make it [4:14] efficient for your stakeholders. Just [4:16] leverage that. Then as in with most [4:18] compliance programs, you do want a risk [4:21] assessment process to then come as the [4:23] next step. And again, what are the risks [4:25] for your organization will determine [4:27] what your risk assessment process looks [4:29] like. Honestly, the the way we do it is [4:31] first question is always is there an [4:33] actual impact to an individual or is it [4:35] just about a system or an operational [4:37] process and the one thing we've [4:39] discovered is about 80% of the use cases [4:43] don't impact an individual. So it [4:44] actually makes life much easier that you [4:46] are providing much lighter review based [4:49] upon that risk assessment. So risk [4:51] assessment then needs to to to come [4:53] next. Next what you should have are what [4:56] are your standard mitigation processes [4:58] as against the the ones where you do [5:00] find risk. I would tell you having [5:02] standard mitigation processes will make [5:04] it a much more efficient process. Your [5:07] users will also understand oh those are [5:09] the standard proc things that I need to [5:10] do for mitigation. I will do them a [5:12] priority and not even wait till you tell [5:14] me to go do them. Next that you you will [5:17] need is AI observation. Now, that should [5:21] be part and parcel of the ML ops and LLM [5:23] ops that your people are doing, but it's [5:26] not always. It's not always how they [5:27] think about it, especially if they're [5:28] using vendors, but you need to make sure [5:31] you have that observability so that you [5:33] understand that the AI is performing and [5:35] what uh in line with whatever parameters [5:38] you set up as part of the governance [5:39] program. So, those 10 those are the [5:41] basic pieces of the program. There's a [5:45] bit of devil in the detail for your [5:47] individual company, but you have to work [5:49] that out. I I would just say happy to [5:51] give more specifics as we go along, but [5:53] that is something that is unique to the [5:55] company. So, I'm going to hand it off to [5:57] somebody who's in a very different [5:58] industry than us in financial services. [6:01] Um, and so Bridget, can you give a sense [6:03] from your perspective of what else [6:04] should be in there? [6:05] >> Sure. So I agree with everything that [6:08] you said policies and risk assessment [6:11] monitoring for example but two other key [6:14] components that shouldn't be missed are [6:16] stakeholder engagement and education. So [6:20] as we all know in this room AI is very [6:23] crossunctional. It no longer exists in [6:25] the silo of IT or on a data science team [6:28] for example. So stakeholder engagement [6:30] involving the right people in your [6:32] organization is really critical. So [6:35] we've done this in two different ways at [6:37] Estee Lauder. One is in connection with [6:39] the intake and risk assessment process [6:42] that you mentioned. As part of that [6:44] process, we've brought together a broad [6:46] group of stakeholders that manage [6:48] different risks. And so whether that's [6:51] cyber security, enterprise architecture, [6:54] privacy, legal, as part of that risk [6:57] assessment review uh for high-risisk use [7:00] cases, we always make sure that we bring [7:01] together that crossf functional team of [7:04] different stakeholders to get their [7:06] input on AI use cases. Another way that [7:09] we've brought together different [7:11] stakeholders and are engaging different [7:13] stakeholders in AI is through our AI [7:15] task force. In this task force, we have [7:17] representatives from each of the [7:19] different business functions within the [7:21] company. So HR, R&D, finance and we come [7:25] together approximately once a month to [7:28] discuss the projects that each function [7:30] are undertaking that involve AI and [7:33] really share learnings, help the rest of [7:36] the company who we may not work with on [7:38] a day-to-day basis understand how those [7:41] different functions are using AI and [7:43] what we can really leverage across the [7:45] the business um and learn from each [7:48] other in terms of what's working for our [7:51] deployment, education, etc. And as it [7:55] relates to education specifically, [7:58] um in addition to that crossf functional [8:01] uh group that we we bring together every [8:04] month, we have a lot of education and [8:06] and awareness initiatives specifically [8:08] for the broad employee base. So, as you [8:11] mentioned already, Andrew, you know, [8:13] compliance programs have been around for [8:15] a long time. Education as part of [8:17] compliance programs have been around for [8:18] a long time. So we've since added AI as [8:21] one of the annual sort of training [8:24] programs that the company does and that [8:26] every employee will need to uh certify [8:29] to each year. And then in addition um [8:33] and in compliance with the EUAI act [8:35] where we have literacy obligations now [8:38] making sure that every tool that gets [8:41] rolled out has a training aspect to it [8:44] as well. Making sure that employees know [8:46] this is what the tool is designed for. [8:48] So you should use it for this purpose [8:50] and also here are some limitations that [8:52] you really need to be aware of. So I [8:55] think from a compliance program from a [8:58] AI governance program stakeholder [9:00] engagement and education are really key [9:03] >> makes perfect sense. So now to layer on [9:06] to this one of the pieces I didn't [9:08] mention but it is important is as you [9:10] deploy it there just there needs to be [9:12] transparency. I mean that is just [9:14] inherent in a lot of what is required [9:16] from a consumerf facing p p p p p p p p [9:17] p p p p p p p p p p p p p p p p p p p p [9:17] p p p p p p p p p p p p perspective, but [9:18] also from your cor corporate customers [9:20] as well. So Tom, I'm going to hand it [9:22] off to you. From that transparency [9:23] perspective, what would you wind up [9:25] doing? How do you work it through as [9:27] part of your governance program? [9:29] >> So I mentioned ear F5 is a B2B company. [9:34] So, our biggest [9:37] um [9:39] reward for successful transparency is [9:43] deal velocity. [9:45] The way that that works though is first [9:49] establishing a foundation of things like [9:53] what is a model? coming to a consensus [9:56] with our engineers, our product team uh [10:00] as well as other stakeholders like [10:01] security and IP to say hey this is what [10:05] we are going to define as a model this [10:08] is what we are going to define as an [10:10] agent and now what are the principles by [10:15] which we are going to operate. [10:18] Now for F5 being an application delivery [10:22] uh and security platform, we provide [10:25] both products and services. So the [10:28] products when you use the term products, [10:31] I'm talking about something that's going [10:33] to run on the customer's infrastructure [10:35] on prem. Um but we also provide [10:38] services, right, which are on the cloud. [10:41] And those two scenarios require applying [10:47] uh transparency differently and [10:50] committing to certain requirements in [10:52] those uh two different paradigms. So for [10:55] example uh when a customer is wanting to [10:59] run something on prem that means they [11:02] want absolute control over what is going [11:04] on. They do not want their models [11:07] switched out from under them. They do [11:09] not want you to make available uh models [11:13] that anybody can just download. They're [11:16] very sensitive about their data being [11:18] used to process or generate new models. [11:21] Whereas in the services we need F5 needs [11:24] to be able to have more control. So we [11:27] have established three principles by [11:30] which we have est that that we used in [11:33] order to define the requirements that we [11:36] have on our products and services. The [11:38] first one is customers are always made [11:42] aware of the models we use, how they're [11:46] generated, the data that we use to both [11:49] train and test that model, and what [11:53] services those models are implemented [11:55] in. The second principle is that [11:58] customers are in control of the models [12:01] that run on their infrastructure. [12:04] And that makes sense probably given what [12:06] I already just just said with regard to [12:08] customers wanting control over those uh [12:11] where where they have deployed on prem. [12:14] Now [12:16] the third principle is that F5 has to [12:19] control the the models that we run on [12:22] our infrastructure. [12:24] Whilst we have a very large [12:26] infrastructure to support many many [12:29] enterprise clients, enterprise [12:30] customers, there's still some finite [12:33] amount of compute and resources [12:35] available. We cannot host every single [12:39] model for every single uh customer that [12:42] they might want. So part of that [12:44] transparency is we will disclose to you [12:48] what models we're using. We will we will [12:50] give you a certain amount of time. We'll [12:53] give you an alert and we'll give you a [12:55] certain defined amount of time to review [12:58] that model. And then ultimately if it is [13:01] going to be a model that runs on your [13:03] infrastructure, you choose if and when [13:04] that is going to be executed in your [13:07] environment versus if it's our services [13:10] that we're leveraging, we will make it [13:12] clear to the customer that they are that [13:15] when that model is going to be changed [13:17] and pushed into production, [13:19] >> right? That make sense? [13:21] >> Yep. Makes perfect sense, Tom. So that [13:24] that's a view from that B2B world. I'm [13:26] curious for a broader and different [13:28] perspective, Xavier, if you can give one [13:30] in something that's not B2B to C or B2B [13:33] to healthcare provider. So a kind of [13:35] different context here. [13:36] >> Yeah, absolutely. Uh first I would say [13:39] that the most important thing is to [13:41] remember who you are and who you're [13:43] talking to. So you have corporate [13:46] values. Uh you need to align your [13:48] transparency with these. uh who are you [13:51] talking to? Are you talking about um a [13:54] hospital network uh patient individual? [13:58] So align your message with who you're [14:01] talking to. So if we're talking about [14:05] B2B, uh you can be more technical for [14:08] sure, but if you're in a very green, [14:12] highly uh specialized [14:16] uh AI tool uh world, don't think that [14:19] you're saying things that people will [14:22] understand necessarily. If they are uh [14:25] even very specialized in a certain uh [14:28] area, they might not understand. So make [14:30] sure to tone it down a bit. But in a [14:32] nutshell, you can be technical. Uh when [14:34] it comes to uh human beings, [14:37] individuals, well, be as simple and [14:41] crystal clear as possible. Identify [14:43] whenever they're interacting with AI. [14:46] Make sure that they either consent or [14:48] notify depending on the country you're [14:50] in, but I would argue that consent is [14:53] always better. And and finally, if [14:55] you're using a tool to make an automated [14:58] decision such as talent acquisition, HR, [15:02] deciding who will get the next promotion [15:04] based on data in your HR system, well, [15:08] make sure that you make it crystal clear [15:10] that you're using AI again, but also [15:13] think about a mechanism for people to [15:16] appeal and to say, "Hey, I don't think [15:19] that I was treated fairly for this and [15:22] that reason." and further down the line [15:25] it will pay off a lot because people [15:28] will be more loyal and trust your your [15:31] company more internally and externally. [15:34] So always keep that in mind and [15:39] I I realize that it might be simple but [15:41] remember who you are like do the right [15:44] thing align with your corporate culture. [15:48] >> Thank you. I I I think it just [15:50] highlights that transparency is very [15:52] contextual for your business and your [15:54] business may have multiple lines of [15:56] business. So, it's just something to [15:57] consider how you how you think about [16:00] that. I'm going to take a step back to [16:02] something I'd mentioned before about [16:03] risk assessment processes. So, I was [16:05] saying they are specific to each of us [16:07] and what our our risks are. So, Tom, I'm [16:09] going to hand it off to you given you [16:11] were just discussing a lot of where you [16:13] are in the B2B space. where how and [16:15] where do you look at risk assessment as [16:17] part of your AI governance process? [16:19] >> It's a really good question. So um we [16:22] have certainly principles and [16:25] requirements internally that are just [16:27] going to apply right we are going to [16:28] expect a certain standard um with regard [16:32] to the data that we process with regard [16:34] to the models that we train and their [16:36] efficacy. Um, [16:38] but if if you if you think about and I I [16:42] think probably many people in this room [16:44] have looked at the NIST RMF, right? And [16:48] and the the model um model life cycle. [16:52] Um [16:53] that model life cycle starts at [16:55] envisioning how a model is going to be [16:57] used, what data is going to be used to [17:00] train that model, training that model [17:02] given particular algorithms, testing [17:05] against that model and then deploying [17:07] that model. And the last B part is using [17:10] the model in the application and for us [17:12] is B2B. The issue is is that a model may [17:16] be absolutely perfect for one customer's [17:20] use and the same model in the same type [17:23] of service might not be appropriate for [17:26] another customer. Let me give you an [17:27] example. [17:30] We might have a model that is uh used to [17:34] detect fraud. So a request comes in and [17:37] we have a model that says oh this is [17:38] fraud or not. Well, you could have very [17:43] short-lived models that are very good at [17:46] identifying a particular type of [17:48] fraudulent attack [17:51] and it might only be useful for a very [17:53] short amount of time because once you've [17:55] mitigated that attack, it goes away. Uh, [17:58] and so, um, being able to implement [18:02] models that are short-lived are often [18:05] just fine for retails, retail customers [18:10] or for restaurants and things like that. [18:12] But when we're talking about banks and [18:15] financial institutions, they are not [18:17] cool with saying, "Oh, you want to [18:19] update the model every two weeks? [18:21] Totally awesome. Yeah, we're totally [18:22] down with that." And in fact we a lot of [18:24] times in the negotiation phase have to [18:27] talk about what we can do as far as the [18:31] cadence at which we will release our [18:34] products and services. So for us the [18:37] goal of risk management is informative. [18:41] It is our responsibility to inform our [18:45] customers about what we're doing and how [18:49] we're doing it. [18:51] But we make it clear that we are not in [18:53] a position to estimate or evaluate what [18:58] our customers risks always are. And [19:00] we're certainly not in a position to [19:02] give them legal advice as to whether or [19:04] not it would be appropriate for their [19:05] regulators. Um, and so we we we make it [19:09] clear to them that our job from our [19:11] point of view is to inform you and give [19:14] you everything that you need in order to [19:16] be successful using this product or [19:19] service that leverages a model for your [19:22] regulators as they apply and it is up to [19:25] you to decide whether or not that [19:27] service and how the models are used in [19:29] that service are appropriate for you. [19:33] So I I I will pick up from that and [19:35] saying yeah this is where it's showing [19:37] industries do differ. So as Tom pointed [19:39] out about financial services changing a [19:42] model every two weeks that won't work [19:44] with most model risk management uh [19:46] functions within most banks instead [19:48] they'd be like they they would be a [19:50] little bit I I think disturbed to put it [19:52] nicely. Um so instead from a financial [19:54] services and uh financial technology [19:57] company perspective our view on risk [20:00] assessment is really to identify what [20:02] the risks are and if the risks need [20:04] mitigation what those mitigation should [20:06] be. So as an example I pointed out does [20:09] it impact an individual is one of the [20:11] biggest questions that we have and if it [20:13] impacts an individual then we have to [20:15] start determining is there bias as [20:17] against certain populations for that [20:20] model and then we have to start doing [20:21] analyses on that and we have the data [20:24] scientists start providing reporting as [20:27] to where that those impacts may be and [20:29] there may be good reasons for for those [20:31] biases. The easy example is fraud has a [20:34] very distinct pattern versus a marketing [20:37] model. You don't necessarily want those [20:39] same patterns happening in a marketing [20:40] model. And so it's not necessarily a [20:43] cookie cutter piece. But there are [20:45] standardized processes you wind up [20:47] implementing to do bias analysis with [20:49] regard to that. The other pieces to risk [20:51] assessment you tend that we also look at [20:54] is are there societal impacts to what [20:56] you're doing. Are you going to impact [20:58] not just an individual? So I I you know [21:01] I used to be a privacy lawyer. So we [21:02] used to think of everything about an [21:03] individual but now in the day and age of [21:05] AI you also look at the risk to society. [21:08] Are you changing something to society? [21:11] So for example we provide insights to [21:13] governments. We don't want the [21:14] governments deploying resources to [21:16] populations as a whole that might be [21:19] biased in certain ways. And so we have [21:21] to take a look at that as a risk as [21:23] well. Banks are our customers. We want [21:26] to understand are there potential risks [21:27] we're introducing to them because [21:29] honestly as Tom's pointing out in the [21:31] B2B context they will ask us those [21:33] questions they expect that we've already [21:35] addressed them also defining your risks [21:38] define your mitigations and so for [21:40] example most of us are now deploying Gen [21:43] AI and so my team has developed a [21:45] standardized toolkit to say here's your [21:48] here's your standardized risk mitigation [21:50] for Gen AI will that standardized [21:52] toolkit change yes but right now where [21:54] we are Most most of us are generate your [21:57] QA pairs, go test your QA pairs, put a [22:00] second monitoring model on top of it, [22:02] monitor the monitoring model. All those [22:04] standardized tools we now deploy [22:06] straight out to the data scientist. So [22:08] when they do the deployment, they know [22:09] what that risk mitigation should be. So [22:12] our risk assessment isn't just about [22:14] that assessment piece, but it really is [22:16] about how do you mitigate any of those [22:17] risks that need to get mitigated. And I [22:19] would say from our perspective, we are [22:21] very much looking at it from that EU [22:23] lens of how make sure we are going to be [22:25] in compliance with the EU, New York City [22:27] law. There are lots of legal [22:28] considerations, but there's also this [22:30] ethical perspective that um Xavier put [22:33] it put it nicely is do right by your [22:35] customer. It's almost like follow the [22:37] golden rule, I think, is the way to say [22:39] it. And so I'm going to hand it off to [22:41] Bridget because she isn't a very [22:43] consumerf facing brand as well. Is what [22:46] are those ethical considerations? How do [22:48] you weave those into the AI governance [22:50] process? [22:51] >> Yeah. So I think you know approaching [22:54] the ethical considerations in AI the the [22:57] very first step that an organization [22:59] needs to take is really defining the [23:02] principles that you're going to going to [23:04] follow with respect to the responsible [23:06] and ethical use of AI in the development [23:09] the deployment and then the monitoring [23:12] process. And these principles need to be [23:15] well defined, written down. They need to [23:18] be communicated to the business and [23:20] those not only involved in AI governance [23:23] but throughout the organization. And [23:25] then they need to be operationalized. [23:27] And so at Estee Lauder, we defined a few [23:30] years ago our principles for the [23:32] responsible use of AI and really as [23:36] Xavier mentioned already grounded these [23:39] in the core values of the company. So [23:42] they're tweaked to our business, they're [23:43] tweaked to our um particular values, but [23:47] they're really based on common [23:49] principles that you'll see in the NIST [23:52] um AI RMF framework as well. So [23:54] including concepts such as transparency, [23:57] which we've discussed, fairness or bias, [24:00] which Andrew just mentioned, as well as [24:03] accountability. And once you have sort [24:05] of the high level principles defined [24:07] then uh you can work on you know [24:10] documenting and defining additional [24:12] guidelines and other standards and [24:15] controls to address those principles and [24:18] really that will become your your [24:20] guiding light going forward. So as it [24:23] relates to fairness for example, making [24:26] sure you have defined requirements on [24:28] making sure that data accur is accurate [24:31] and representative as you think about [24:34] concepts of accountability. Who has the [24:36] right decisionmaking power within your [24:38] organization related to AI and AI use? [24:42] And then when you think about [24:44] transparency, how do you um get [24:47] information from your vendors that are [24:49] providing you with services about what [24:51] AI they are using and incorporating that [24:54] into your vendor risk management [24:56] process. So really defining those [24:59] principles and guidelines and then [25:01] operationalizing them within the [25:03] different areas of the business is [25:05] really the um place to start. And then [25:08] similar to what you mentioned uh Andrew [25:11] as part of the risk assessment process [25:13] making sure that those um ethical [25:16] considerations are being checked in in [25:19] some way and with that crossf functional [25:22] group of stakeholders that I mentioned [25:23] earlier that we bring together to review [25:26] those risk assessments uh that are of [25:29] particularly high risk. They can all [25:31] bring different perspectives and I think [25:33] that having a group of people looking at [25:37] AI use cases with a different viewpoint [25:40] is also really critical in assessing [25:43] sort of those ethical considerations [25:45] with AI. So Xavier I know you know [25:48] you're in a ethics officer role [25:50] specifically so you have I'm sure [25:52] additional considerations to add. [25:55] >> Yeah. Uh thank you. Um I agree with [25:58] everything you said and uh if you think [26:00] NIST basically like the the basic step [26:04] like you identify, you protect, you [26:06] detect, you respond, you recover that [26:08] should be applied to AI ethics in my [26:11] opinion like if you look at a problem [26:14] with those steps in your mind, you will [26:17] do the right thing over the long run. So [26:20] um [26:22] looking for people within your [26:24] organization in order to make sure that [26:26] you you do the right thing. That's [26:28] that's a very good uh point. So for [26:32] those who are in big organizations, you [26:34] would be surprised about the talent that [26:37] you have internally. You might have [26:38] someone that works in the digital realm [26:42] of your company, but they also have uh [26:45] an undergrad in philosophy. uh you can [26:48] look everywhere and you will find people [26:50] that have great skills and if you can [26:53] leverage that you will have people that [26:55] already know your company they're [26:58] embedded in your culture the principles [27:00] or we call it pillar we call them [27:02] pillars and they will be able to [27:04] implement within their own team a lot of [27:07] things that you roll out so having those [27:10] allies some kind of a ethical AI network [27:13] helps a lot uh if you have uh dedicated [27:17] ated group to privacy in different [27:19] countries. Those people understand data [27:21] flow. They understand the the the law in [27:25] their own jurisdiction. So those are [27:27] powerful allies as well. Um [27:32] a good way to to to to to roll out [27:35] initiatives is to start with your [27:37] principle. Then you will have your [27:39] guidelines or your your rules, your SOPs [27:42] that are global and then you roll them [27:44] out to your uh to your different geos or [27:47] different countries and at that stage [27:49] you might have some people that will [27:51] raise their hand and say I cannot do [27:52] this here uh usually because of [27:55] regulation uh limitation. Then you [27:58] record the deviation for that geo and [28:01] you make sure that [28:04] you loop back to your other geo and say [28:06] are you sure you're doing the right [28:07] thing cuz in this country they say you [28:09] cannot do this or that. Once you have [28:11] documented everything [28:13] you are in a good spot you're you're [28:17] getting there. Uh but it's a lot of [28:19] work. So trying to find allies within [28:21] your own organization and look at the [28:24] skills of people you will be surprised. [28:26] There's a lot of people that can help [28:28] you a lot. [28:30] >> That's good to know. Um, always good to [28:33] lean on others makes the many hands make [28:35] light work. I I would agree with that [28:36] which sort of gets to the next point [28:39] that I will raise and it's one of the [28:40] topics in the title of this is what are [28:42] the challenges that we're facing in AI [28:45] governance and I'll I'll kick this off [28:47] and then hand it over to Bridget. And I [28:50] would say from our perspective is the [28:53] technology keeps changing and the the [28:54] changes keep introducing new things to [28:56] consider and when you start getting new [28:58] things to consider you sit there and go [29:01] well wait what are the standards for [29:03] considering it and so the biggest [29:05] current example that I think many of us [29:07] are facing is generative AI and the [29:10] question with generative AI is when is [29:12] good good enough? We all test the [29:14] generative AI. I see somebody laughing [29:17] in the audience because they experienced [29:18] this themselves is we're all testing it [29:20] with QA pairs. Um is the main way I [29:23] think most of us are testing it. Users [29:25] thumbs thumbs ups thumb down. But are a [29:28] million users enough? Are they diverse [29:30] enough? I I think as to where you're [29:32] going, Xavier, um we don't know what is [29:35] the standard for for good is not [29:37] defined. How do you test for good is not [29:40] defined. Many of us are used to I will [29:43] call it traditional AI uh of using [29:46] structured data and we're all used to [29:48] simple stats and simple stats could give [29:50] you answers as to where are the outliers [29:53] and so we would know the outliers of the [29:55] places to check to see is there a [29:57] problem and is it good enough and you [29:58] could do model observability in a very [30:01] quantitative way with generative AI we [30:04] don't have that quantitative way down [30:06] yet the mathematicians have not exactly [30:08] cracked that uh problem yet and so we're [30:11] all working to say okay we have to [30:14] create our own baseline we have to [30:15] create our our own tool sets and so I [30:18] think many of us originally were looking [30:20] to groups like NIST a lot of people here [30:22] are referencing NIST as a standards body [30:24] here in the United States because they [30:27] are one of the preeminent standards [30:28] bodies globally and people adapt their [30:30] standards but they haven't gotten there [30:32] yet and I think even with the new action [30:34] plan that's not where they're [30:36] necessarily focusing so for many of us [30:38] we're looking to our peers to say how [30:40] are you guys doing it? How are you [30:42] testing this out? And I go to all of my [30:44] peers all the time and ask them. And so [30:47] kind of where Xavier was just going [30:48] before is I look outside the company for [30:51] those friends to help solve those kinds [30:53] of problems to say what are the [30:54] standards you guys are using? What are [30:56] the testing methodologies you're using? [30:58] Because I don't know what I'm doing is [31:00] good enough. I don't know that that's [31:01] the right answer. And I think the right [31:03] answer will keep changing. So I would [31:05] say that for us is one of the biggest [31:07] challenge that that we keep trying to [31:08] tackle and keep trying to to get on top [31:11] of. But I'll hand it off for Bridget for [31:13] challenges you guys are seeing. [31:15] >> So I think uh in implementing our AI [31:18] governance program there are three key [31:21] challenges that come to mind. So one [31:23] being change management, two being um [31:27] striking the right balance how not to [31:29] sort of boil the ocean and the third [31:31] being tooling. So with change management [31:34] I think like implementing any new [31:37] compliance or risk related program [31:40] there's challenges is not becoming a [31:43] blocker and you know not slowing down [31:46] the business and I think that in [31:48] addressing these challenges one of the [31:50] key things to do is latch on to existing [31:53] processes that exist and this is similar [31:55] I think to what Xavier mentioned about [31:57] you know finding your allies find those [32:00] processes within your company that are [32:02] doing similar things whether that's a um [32:06] legal assessment process, a privacy [32:09] review process, an enterprise [32:11] architecture review. Um making sure you [32:14] lean on these people and these processes [32:18] as much as you can instead of [32:19] reinventing the wheel. I also think in [32:23] terms of change management, as I [32:25] mentioned earlier, education is key in [32:28] the sense that you need to show the [32:30] value that you're bringing to the [32:32] business. If you're implementing a new [32:34] AI intake process as part of your [32:37] governance, well, what's the incentive [32:39] for people to to use this process? How [32:42] can you show them that by participating [32:45] in a new intake and review procedure in [32:48] addition to the many other intakes that [32:50] they have to do that they're going to [32:51] get value out of this and that can be by [32:55] helping them find the right AI solution [32:57] for the problem that they're trying to [32:59] solve. The second challenge that I [33:02] mentioned was striking the right [33:04] balance. And I think that in this area, [33:07] as Andrew said, that is constantly [33:09] evolving and changing, there is um I [33:13] think um a tendency to govern it all in [33:16] the exact same way. And part of what [33:20] needs to happen as part of your risk [33:23] assessment and review process is making [33:25] sure you have a right tiering, right? So [33:27] which of the use cases for AI or [33:31] projects that involve AI can go forward [33:34] with standard procurement procedures and [33:37] which ones need that um enhanced [33:40] governance risk assessment reviews to be [33:43] done. So picking a high, medium and low [33:46] using the EUAI act for an example or [33:49] tailoring that to your organization will [33:51] really help to streamline the process [33:53] and help you overcome that challenge. [33:56] And then the third challenge is tooling. [34:00] And similarly, I think that, you know, [34:03] latching on to existing tools that you [34:05] have to help with your AI governance um [34:08] will be easier than maybe getting [34:11] funding and and a whole lot of uh new [34:14] tools um to help with this, whether [34:16] that's for your vendor reviews or for [34:19] your inventory of data or AI or uh [34:23] systems and and assets as we await new [34:27] tools out there on the market to help us [34:29] all with AI. AI governance. [34:31] >> Yeah, I'm I'm waiting not so patiently. [34:34] I would just say that for [34:35] >> sorry to so Tom what you said about [34:38] education um is so interesting and and [34:42] poignant when you know you do want to [34:44] latch on to existing processes. Um one [34:47] of the things though that I noticed is [34:50] these existing processes are already you [34:52] know other compliance slash uh risk uh [34:57] stakeholders and they need education [35:01] with regard to what is the difference [35:04] between an AI governance program and the [35:07] privacy program and security. Security [35:10] people start thinking oh we're [35:11] responsible for AI. privacy people start [35:13] freaking out about oh well maybe we need [35:16] to govern AI and so having training [35:20] uh just on with your peers right the [35:24] other people your other stakeholders [35:25] that you're working with all the time [35:27] helping them understand what you're [35:30] looking for so specifically [35:32] uh what I usually do is is say hey look [35:36] when we're talking about AI governance [35:38] we're talking about whether or not a [35:40] model is appropriate to be used in a [35:44] given situation. [35:46] Security would be concerned with whether [35:49] or not information is going to be uh [35:53] shared or disclosed when it wasn't [35:55] intended to be. So privacy is going to [35:59] be concerned about what data is being [36:01] used. Is it personal information or not? [36:04] And then is it adequately protected? [36:06] Sure, the security team might want to [36:08] know when a model is going to be used, [36:11] but they are not there to determine [36:14] whether or not the model is appropriate [36:17] to be used for a given use. So I [36:20] actually find that yes absolutely you [36:23] have to latch uh on to existing [36:26] processes but I think it's also super [36:29] important to make it clear to your peers [36:34] that this is orthogonal right to put it [36:37] mathematically this is sort of like an [36:38] orthonormal basis of of compliance where [36:41] you have uh security you have uh privacy [36:45] you have AI and they are all independent [36:48] and they are all looking for different [36:50] things. Um, but it is definitely [36:53] important to work together to make sure [36:55] you're on the same page. [36:58] >> Perfect. Well said. So, Xavier, I'm [37:00] going to hand it off to you to build off [37:02] of what Tom said to say, well, how do [37:04] you know it worked? How do you know it [37:06] was an effective program? I mean, [37:07] everything Tom's saying is true. So, now [37:10] you do that. Does it work? [37:12] >> That that's a very good point. Like, if [37:15] you cannot measure it, it doesn't exist, [37:17] right? Um, so there's two things you can [37:21] do. One, my favorite is the the level of [37:24] maturity. So it's not about the level of [37:28] risk that you still have within your [37:30] company when it comes to AI. It's not [37:32] about mitigation per se. It's about are [37:35] you sure you're doing the right thing [37:38] 100% of the time all the time without [37:41] exception. So, as you get better and [37:44] better at AI governance, you'll realize [37:47] that, wow, we're becoming more [37:49] efficient. Wow, we're moving faster. Oh, [37:53] we've done the exact thing we said we [37:55] would 100% of the time last year. Like, [37:58] so that's the level maturity. So, that's [38:00] one way to evaluate your uh how [38:03] effective your governance model is. The [38:06] second one is uh a bit cold maybe but [38:10] it's very efficient. It's looking at [38:13] your ROI. Um and I know some people will [38:16] say that it's hard to put a dollar value [38:18] on a governance but uh if you look at [38:22] the number of dollar you saved uh [38:25] whether in fines or legal action uh [38:28] external council bills that are so small [38:31] and easy to pay. Um [38:34] uh if you look at the the the number of [38:37] hours that your employees had to redo [38:40] because of uh bad governance. If you [38:43] look at the increased efficiency as I [38:45] said uh you'll be able to put a dollar [38:48] value on all of this and go to the board [38:51] and say hey here's how much uh we [38:54] generated by doing proper governance. So [38:57] that's one way to do it. Uh you can also [39:00] look at uh the the revenues you [39:03] generated in increasing trust of your [39:05] your patients or customers or business [39:08] partners for example. Uh so if people [39:12] decide to go with you and not your [39:14] competitor because they trust you more, [39:16] there's a dollar value to that. And if [39:19] you can calculate in some way the delta [39:22] of trust before your governance uh [39:26] program and after well you can present a [39:29] dollar value as well there. Um, now [39:31] there's also soft ROIs. Uh, so [39:34] reputation damages that are decreasing, [39:37] uh, the increase of retention of your [39:39] employees, like there there's there's [39:40] multiple ways to to to do it. But in [39:43] order to accomplish all of that or any [39:46] of that, whether maturity or ROI, you [39:49] need to define your goals. What are my [39:51] goals when it comes to governance? Have [39:53] I achieved them? Uh, what was my [39:56] baseline before achieving anything? and [40:00] make sure that if you can you give a [40:02] dollar value to those uh objectives. So [40:06] for example [40:08] um we won this big RFP because we [40:12] outscored our competitors when it came [40:14] to proper governance [40:17] um appropriate transparency when it [40:19] comes to AI. So maybe you should take a [40:23] percentage of that contract you won and [40:26] use it to calculate how much privacy is [40:29] doing great and you can also ask for a [40:33] proper vacation in Maui because you [40:35] deserve it. [40:37] Um, no, but kidding aside, yeah, [40:39] objective, try to measure, try to put a [40:41] dollar value to everything you [40:43] accomplish because this will resonate [40:46] more with um [40:49] seale people than just a list of tackled [40:53] objectives. [40:54] >> Yeah, I would just layer on, but it's a [40:57] little bit more not necessarily ROI, but [40:59] versus the branding is when you're [41:01] called in front of the client to explain [41:04] what the product is and how it works. [41:06] Well, it's a great value ad that you are [41:08] an enabling function. You're not a [41:10] compliance function. And so I would [41:11] suggest wherever you can get away doing [41:13] that, do that for yourselves to say, I'm [41:15] an enabling function, not just a [41:17] compliance function. It's not just about [41:18] the law. It's about those client [41:20] relationships. As Tom was pointing out [41:22] before, we uh uh I got an interesting I [41:25] was on a call, this was probably two [41:27] years ago or so, and somebody said some [41:30] flippant comment of like, "Oh, well, you [41:32] know, you guys are the office of no." [41:34] Yeah. [41:34] >> And so we it changed that. So now our [41:36] team has on all of our emails our [41:38] signature is the office of no K n. Um [41:42] because the way we measure efficiency or [41:45] effectiveness I should say is sort of [41:48] three ways is one do people get access [41:53] faster to the things that they are [41:55] looking to get access to right and then [41:57] that's in the sort of the procurement [41:58] sense. Are we extending the procurement [42:01] time by adding our assessments to it? [42:06] Another though is are our data [42:08] scientists getting access to data [42:10] faster? Are our policies actually making [42:13] their job easier to get where they need [42:17] to go? And then third is deal velocity, [42:22] right? And maybe this might be one of [42:23] those B2B things again, but but [42:27] if you are constantly having to [42:30] negotiate agreements and renegotiate [42:33] your AI terms over and over and over [42:36] again, [42:38] nobody is happy about that, right? And [42:40] so another measurement of efficacy would [42:44] be, you know, is your AI addendum, if [42:47] you have one, is it raising more [42:49] questions than it's answering? and and [42:52] and and how readily are people willing [42:55] to accept that. Right? So, I think those [42:57] are sort of three different ways to [42:59] think about your it it is not [43:01] necessarily a dollar amount of ROI, but [43:05] from your uh but but from your sales [43:08] team, from your data science team, and [43:10] from everybody else in the company, they [43:12] very much notice and appreciate. Holy [43:15] cow, that was way easier than I thought [43:17] it was going to be. very we we all all [43:21] hope we're in that position, Tom. Yeah, [43:22] we we we all do. So, we're in the final [43:25] uh throws of this in in the last minute. [43:28] So, I'm just going to let the panelists [43:29] give one last closing remark before we [43:32] we say uh goodbye to everybody here. So, [43:34] Bridget, you want to start and we'll go [43:36] into order? [43:37] >> Well, I think um we we covered a lot [43:40] today and this is a journey. So uh you [43:44] know keep keep moving things forward and [43:47] as the laws change the standards change [43:50] just be ready to evolve uh evolve your [43:53] program um with it [43:57] >> um I would say that society evolves [43:59] really fast uh and the perception on AI [44:02] is changing but your principles [44:05] shouldn't like they don't evolve that [44:06] fast and your corporation's culture is [44:09] not evolving as fast as uh the sentiment [44:12] around AI in a society. So whenever [44:15] you're facing a governance problem, [44:17] think about your core principles. They [44:19] will often guide you in the right [44:20] direction. [44:23] Um those are all those are all [44:26] wonderful. Um [44:29] I find this field absolutely fascinating [44:32] because of all of the perspectives [44:35] for which everybody has to view the use [44:38] of a particular model. So as we leave or [44:43] depart um you know if there are people [44:46] that are interested in discussing their [44:49] their viewpoints particularly highly [44:51] regulated industries right like these [44:53] are the type of people that love [44:55] collaborating with make yourself known [44:57] make yourself you know introduce [44:59] yourself uh and and let's let's talk and [45:02] continue the conversation I [45:04] wholeheartedly agree with that last [45:06] sentiment so I will let people go and go [45:08] have the conversations amongst [45:10] themselves elves and enjoy the rest of [45:12] the sessions. Thank you everyone. [45:14] [Applause]