McDonald's AI Counsel Warns System Prompts Cannot Control Agents

Governing Agentic AI - A Practical Framework for Retail, Hospitality and QSR With McDonald's Corp. — Ai4

Ai4

Autonomous agents can easily reason around prompt restrictions to cause irreversible operational and antitrust harms, requiring enterprises to enforce deterministic technical circuit breakers and mandatory human oversight.

AI-assisted summary of Governing Agentic AI - A Practical Framework for Retail, Hospitality and QSR With McDonald's Corp. — original source. Check the original for full context.

Source published:

Detailed notes

## Background Published on YouTube by Ai4 on August 14, 2026, this presentation features May Sethaphanich, Global Senior Counsel and AI Legal Lead at McDonald's Corporation. The talk outlines a practical AI governance framework designed specifically for the retail, hospitality, and quick-service restaurant (QSR) sectors.

## TL;DR May Sethaphanich explains that organizations cannot safely deploy autonomous agents without first establishing a foundational governance layer that covers traditional machine learning and non-autonomous systems. She dispels the myth that retail and QSR operate in a low-regulation environment, pointing out acute legal exposures in labor, consumer protection, and antitrust—including algorithmic collusion risks. Because autonomous agents expand operational blast radius and cannot be constrained by system prompts alone, enterprises must enforce both trained human-in-the-loop oversight and deterministic technical controls. Ultimately, she details practical operational tools like the Agent Register, Agent Cards, and circuit breakers, arguing that proactive governance enables rather than inhibits rapid AI scaling.

## Atmosphere & style The presentation carries a crisp, pragmatic, and authoritative tone, blending corporate legal rigor with deep technical architecture insights. Sethaphanich speaks with clear-eyed candor about common industry misconceptions and the real dangers of unconstrained agentic deployment. The pacing is direct and educational, transitioning from sobering warnings about irreversible automated mistakes into an empowering, highly actionable roadmap for enterprise leaders.

## Key takeaways - [00:04](https://www.youtube.com/watch?v=PlRM8nzfHaE&t=4s) Organizations must establish a baseline governance foundation covering traditional ML and non-autonomous systems before attempting to deploy and govern autonomous agentic AI. - [01:50](https://www.youtube.com/watch?v=PlRM8nzfHaE&t=110s) Retail, hospitality, and QSR sectors face extensive legal exposure under existing consumer protection, labor, employment, and antitrust laws, dispelling the myth that lack of a single sector regulator means low risk. - [01:50](https://www.youtube.com/watch?v=PlRM8nzfHaE&t=110s) Emerging AI legislation imposes mandatory upfront transparency disclosures, explainability requirements for impactful decisions, and severe financial penalties under frameworks like the EU AI Act. - [01:50](https://www.youtube.com/watch?v=PlRM8nzfHaE&t=110s) AI governance cannot be treated merely as a 'privacy plus' exercise, as AI systems operating entirely on non-PII data can still trigger critical operational and antitrust liabilities. - [05:59](https://www.youtube.com/watch?v=PlRM8nzfHaE&t=359s) A mature baseline AI program requires a centralized inventory of approved tools categorized by risk tier, with named business and technical owners held accountable across the system lifecycle. - [05:59](https://www.youtube.com/watch?v=PlRM8nzfHaE&t=359s) Enterprises must institute binding AI policies across internal and external stakeholders alongside dedicated AI incident response protocols designed for non-data harms. - [09:23](https://www.youtube.com/watch?v=PlRM8nzfHaE&t=563s) System prompt instructions do not qualify as effective technical controls because non-deterministic large language models can easily reason around prompt-level restrictions. - [09:23](https://www.youtube.com/watch?v=PlRM8nzfHaE&t=563s) Agent autonomy drastically widens operational blast radius risks, including irreversible erroneous external actions and unauthorized data access due to privilege creep. - [11:32](https://www.youtube.com/watch?v=PlRM8nzfHaE&t=692s) Direct agent-to-agent (A2A) interactions expand legal and security attack surfaces, necessitating pre-defined interaction patterns and clear contractual liability boundaries. - [11:32](https://www.youtube.com/watch?v=PlRM8nzfHaE&t=692s) Autonomous pricing agents consuming shared external market data feeds risk unintended algorithmic price convergence and antitrust scrutiny even without human collusive intent. - [15:02](https://www.youtube.com/watch?v=PlRM8nzfHaE&t=902s) Effective Human-in-the-Loop (HITL) oversight requires matched capital investments in workforce training to equip reviewers to critically evaluate agent recommendations. - [16:57](https://www.youtube.com/watch?v=PlRM8nzfHaE&t=1017s) Deterministic technical controls require a centralized Agent Register and Agent Cards detailing approved tools, API permissions, audit logging, and automated circuit breakers. - [16:57](https://www.youtube.com/watch?v=PlRM8nzfHaE&t=1017s) Far from being a barrier to innovation, robust AI governance serves as an essential enabler for scaling AI safely, sustainably, and defensibly across the enterprise.

## Detailed walkthrough ### [00:04](https://www.youtube.com/watch?v=PlRM8nzfHaE&t=4s) Introduction and Foundational AI Governance Layers

May Sethaphanich introduces herself as an AI legal counsel at a global quick-service restaurant (QSR) enterprise, where she collaborates directly with software engineers, solution architects, data scientists, and developers to build and deploy business-enabling AI solutions. In addition to guiding product development, she co-owns the company's enterprise AI governance program. Before diving into the governance of autonomous and agentic AI systems, she establishes the standard legal disclaimer that all views expressed are strictly her own and do not represent the official stance of any organization with which she is associated.

Turning to the structure of enterprise governance, Sethaphanich stresses that organizations cannot simply skip ahead to autonomous agents without first establishing a robust baseline governance stack. The foundational layer must begin with an awareness of the specific legal triggers and regulatory frameworks applicable to the business. Crucially, this foundational layer cannot be restricted to generative AI alone; it must comprehensively govern traditional machine learning (ML) and non-autonomous systems. Only after this non-autonomous baseline is operational should organizations attempt to layer on the specialized governance mechanisms required for agentic AI.

```mermaid flowchart TD accTitle: Multi-Tiered AI Governance Stack subgraph Agentic ["Tier 2: Agentic & Autonomous AI Governance"] HC["Human Controls (HITL & Training)"] TC["Technical Controls (Agent Register & Circuit Breakers)"] end subgraph Foundational ["Tier 1: Foundational AI Governance"] INV["Approved Inventory & Risk Tiering"] POL["Enterprise AI Policy & Guidelines"] INC["AI Incident Response Protocol"] LEG["Legal Baseline (Consumer, Labor, Antitrust, PII)"] end LEG --> INV --> POL --> INC INC --> HC INC --> TC ```

---

[01:50](https://www.youtube.com/watch?v=PlRM8nzfHaE&t=110s) Regulatory Realities and Dispelling the Retail/QSR Myth

A dangerous misconception persists across the QSR, retail, and hospitality sectors that companies operate with minimal legal exposure because they lack a single dedicated sectoral regulator like those found in healthcare or financial services. Sethaphanich warns that this assumption is fundamentally flawed. A broad array of long-standing laws already applies when businesses interact with consumers and workforces, encompassing employment regulations, labor standards, consumer protection mandates, and antitrust statutes. In the pricing domain, for instance, antitrust authorities prohibit price-fixing regardless of whether the coordination is orchestrated by human managers or automated machine learning algorithms.

The regulatory environment is becoming significantly more demanding. States across the U.S. are enacting AI-specific legislation that imposes affirmative compliance obligations on companies using AI for consumer and employee interactions. These mandates include upfront transparency disclosures—informing job applicants, workers, or customers that they are interacting with an AI—along with strict explainability requirements whenever an AI system makes or assists in an impactful decision. Companies operating internationally must also contend with the EU AI Act, which serves as the AI equivalent of GDPR and carries severe financial penalties for non-compliance. Furthermore, Sethaphanich cautions against treating AI governance merely as a "privacy plus" exercise. While personal data (PII) is vital, AI systems driven entirely by non-PII can still trigger catastrophic legal liability if deployed without rigorous cross-functional oversight.

- **Existing Legal Triggers:** Employment laws, labor protections, consumer protection statutes, algorithmic pricing regulations, and antitrust/collusion rules. - **New Regulatory Obligations:** Upfront transparency notices (disclosing AI interaction to applicants/consumers) and technical explainability for AI-assisted decisions. - **Global & Cross-Functional Scope:** Substantial fines under the EU AI Act; governance frameworks must look beyond PII to address broader systemic and operational risks.

---

[05:59](https://www.youtube.com/watch?v=PlRM8nzfHaE&t=359s) Core Components of a Baseline AI Governance Program

Before tackling autonomous agents, organizations must solidify their non-autonomous AI governance fundamentals. Regulators define AI broadly to capture traditional statistical machine learning as well as generative models. Despite the current hype surrounding autonomous systems, many organizations struggle to articulate even their basic inventory of approved non-autonomous tools. A mature governance program requires an enterprise-wide inventory of approved AI tools and use cases, with each entry mapped directly to a clear risk tier (low, medium, or high) based on applicable legal triggers and corporate risk tolerance.

Risk classifications dictate the specific compliance hurdles and testing requirements that must be met both prior to launch and continuously post-deployment. Every approved tool must have named business and technical owners who are held directly accountable for ensuring that the system remains legally compliant and performant throughout its entire lifecycle. In addition, organizations must maintain a clear enterprise AI policy detailing foundational principles and actionable dos and don'ts for employees, vendors, marketing agencies, PR teams, and outside counsel. Finally, companies must institute a dedicated AI incident response protocol—extending traditional cyber incident management to address non-data harms and regulatory triggers.

- **Approved Tool & Use-Case Inventory:** Centralized catalog of all permitted models, tools, and workflows tagged by risk tier (Low, Medium, High). - **Designated Ownership:** Explicit assignment of business and technical owners accountable for ongoing legal and operational performance. - **Enterprise Policy & Guidelines:** Binding rules of engagement covering internal staff, external agencies, contractors, and legal counsel. - **AI Incident Protocol:** Response procedures tailored to handle AI-specific malfunctions, unauthorized data access, and non-PII legal harms.

---

[09:23](https://www.youtube.com/watch?v=PlRM8nzfHaE&t=563s) The Unique Risks of Agentic AI: Autonomy and Blast Radius

Agentic AI introduces distinct failure modes because autonomy amplifies the inherent non-deterministic nature of generative AI. Sethaphanich underscores a critical technical point: "if your agent is just being constrained by... instructions in the system prompt, now that is not a control because that restriction in the system prompt can be reasoned around by the LLM." When an autonomous agent is granted the authority to execute external actions, unintended outputs can cause immediate, irreversible legal and financial damage.

These failure modes span several operational dimensions: - **Irreversible Erroneous Actions:** External-facing agents executing unintended decisions, such as mistakenly issuing bulk termination notices to employees, terminating vendor agreements, or processing unauthorized customer refunds. - **Unauthorized Data Access & Privilege Creep:** Agents granted access to internal tools and databases holding confidential proprietary assets or stakeholder PII; unauthorized access can instantly trigger mandatory regulatory and consumer data breach notifications.

---

[11:32](https://www.youtube.com/watch?v=PlRM8nzfHaE&t=692s) Emerging Vectors: Agent-to-Agent Interactions and Algorithmic Collusion

Two novel risk vectors emerge when agents operate in complex environments. The first arises when an enterprise agent interacts directly with a third-party agent ("my agent interacting with your agent"). This interaction drastically expands the security and legal attack surface. Because the boundary of operational responsibility becomes blurred, enterprises must subject agent-to-agent (A2A) use cases to intense governance scrutiny. Solution architects and legal counsel should define interaction patterns upfront and anchor them in clear commercial contracts that allocate legal liability.

```mermaid flowchart LR accTitle: External Agent Interaction and Market Convergence Risks subgraph A2A ["Agent-to-Agent (A2A) Interaction"] A1["Enterprise Agent"] <-->|"Contractual Boundary & Scrutinized Protocol"| A2["Third-Party Agent"] end subgraph Market ["Algorithmic Market Interaction"] P1["Company A Pricing Agent"] -->|"External Data Ingestion"| MKT[("Market & Economic Feeds")] P2["Company B Pricing Agent"] -->|"External Data Ingestion"| MKT P1 -.->|"Unintended Price Convergence"| REG["Antitrust Scrutiny & Inquiries"] P2 -.->|"Unintended Price Convergence"| REG end ```

The second emerging vector involves autonomous pricing and market-facing agents that ingest external macroeconomic and competitive data. If competing firms deploy independent pricing agents that consume the same market data feeds, those agents can inadvertently converge on synchronized pricing patterns. Even without any human intent or explicit agreement to collude, antitrust regulators may launch investigations into suspected algorithmic price-fixing. In these circumstances, having detailed governance documentation and verifiable guardrails in place is essential for demonstrating legal defensibility.

---

[15:02](https://www.youtube.com/watch?v=PlRM8nzfHaE&t=902s) Agentic Control Framework: Human Oversight and Operational Readiness

To safely govern autonomous agents, organizations must implement two dedicated categories of controls on top of their baseline stack: human controls and technical controls. For organizations with lower AI maturity, and particularly for high-risk use cases such as hiring, promotion, or workforce management, keeping a human in the loop (HITL) remains essential. Regulators expect enterprises to prove that AI systems impacting human lives operate with meaningful human oversight and active risk mitigation.

Human oversight is ineffective if reviewers lack proper training. Reviewers must understand how to critically evaluate agent recommendations and verify outputs before actions are triggered. For high-risk deployments, organizations should develop comprehensive training manuals and maintain operational documentation proving that personnel are properly equipped to manage risks. Sethaphanich argues that whenever enterprises budget capital to acquire agentic software and infrastructure, they must allocate matching investments toward workforce training so humans can oversee autonomous systems with due care.

---

[16:57](https://www.youtube.com/watch?v=PlRM8nzfHaE&t=1017s) Technical Controls for Agentic AI and Governance as a Scaling Enabler

To counter the non-deterministic behavior of underlying large language models, technical controls must enforce deterministic workflows around autonomous agents. Regardless of the development platform used—whether Microsoft Copilot, Google Gemini, Vertex AI, Glean, or ServiceNow—enterprises must establish an **Agent Register**. Mirroring the foundational AI inventory, this register acts as the definitive source of truth for every agent approved for live deployment.

Each entry in the register must have an **Agent Card** detailing: - Unique identifier and approved business purpose. - Explicit business and technical owners held accountable for compliance. - Whitelisted user groups, accessible tools, APIs, and authorized data scopes. - End-to-end activity logging and reporting capabilities to preserve audit trails for legal defense. - Real-time escalation protocols and technical "circuit breakers" configured to halt, pause, quarantine, shift to safe mode, or terminate an agent if anomalous behavior is detected.

In closing, Sethaphanich rejects the idea that governance hinders technological progress. Far from acting as a drag on adoption, rigorous governance serves as an indispensable prerequisite for scaling AI safely, sustainably, and successfully across the enterprise.

## Notable quotes > [00:04](https://www.youtube.com/watch?v=PlRM8nzfHaE&t=4s) organizations cannot simply skip ahead to autonomous agents without first establishing a robust baseline governance stack. — May Sethaphanich

> [00:04](https://www.youtube.com/watch?v=PlRM8nzfHaE&t=4s) This foundational layer cannot be restricted to generative AI alone; it must comprehensively govern traditional machine learning (ML) and non-autonomous systems. — May Sethaphanich

> [01:50](https://www.youtube.com/watch?v=PlRM8nzfHaE&t=110s) A dangerous misconception persists across the QSR, retail, and hospitality sectors that companies operate with minimal legal exposure because they lack a single dedicated sectoral regulator — May Sethaphanich

> [01:50](https://www.youtube.com/watch?v=PlRM8nzfHaE&t=110s) While personal data (PII) is vital, AI systems driven entirely by non-PII can still trigger catastrophic legal liability if deployed without rigorous cross-functional oversight. — May Sethaphanich

> [05:59](https://www.youtube.com/watch?v=PlRM8nzfHaE&t=359s) Despite the current hype surrounding autonomous systems, many organizations struggle to articulate even their basic inventory of approved non-autonomous tools. — May Sethaphanich

> [09:23](https://www.youtube.com/watch?v=PlRM8nzfHaE&t=563s) if your agent is just being constrained by... instructions in the system prompt, now that is not a control because that restriction in the system prompt can be reasoned around by the LLM. — May Sethaphanich

> [09:23](https://www.youtube.com/watch?v=PlRM8nzfHaE&t=563s) When an autonomous agent is granted the authority to execute external actions, unintended outputs can cause immediate, irreversible legal and financial damage. — May Sethaphanich

> [11:32](https://www.youtube.com/watch?v=PlRM8nzfHaE&t=692s) Even without any human intent or explicit agreement to collude, antitrust regulators may launch investigations into suspected algorithmic price-fixing. — May Sethaphanich

> [15:02](https://www.youtube.com/watch?v=PlRM8nzfHaE&t=902s) whenever enterprises budget capital to acquire agentic software and infrastructure, they must allocate matching investments toward workforce training so humans can oversee autonomous systems with due care. — May Sethaphanich

> [16:57](https://www.youtube.com/watch?v=PlRM8nzfHaE&t=1017s) To counter the non-deterministic behavior of underlying large language models, technical controls must enforce deterministic workflows around autonomous agents. — May Sethaphanich

> [16:57](https://www.youtube.com/watch?v=PlRM8nzfHaE&t=1017s) Far from acting as a drag on adoption, rigorous governance serves as an indispensable prerequisite for scaling AI safely, sustainably, and successfully across the enterprise. — May Sethaphanich

## Mentioned May Sethaphanich, McDonald's Corporation, Ai4, EU AI Act, GDPR, Microsoft Copilot, Google Gemini, Vertex AI, Glean, ServiceNow

## Video description May Sethaphanich - Global Senior Counsel, AI Legal Lead, McDonald's Corporation

Governing Agentic AI - a Practical Framework for Retail, Hospitality and QSR.

[Source](https://www.youtube.com/watch?v=PlRM8nzfHaE)

中文简报

麦当劳AI法律顾问警告：系统提示词无法约束智能体 自主智能体很容易绕过提示词限制进行推理，从而造成不可逆的运营与反垄断损害，这要求企业必须强制实施确定性的技术熔断机制及人工监督。 May Sethaphanich解释称，企业若未先建立覆盖传统机器学习和非自主系统的基础治理层，就无法安全部署自主智能体。她破除了零售和快餐（QSR）行业处于低监管环境的误区，指出该领域在劳工、消费者保护以及反垄断（包括算法合谋风险）方面面临着严峻的法律风险。由于自主智能体会扩大运营爆炸半径，且无法仅凭系统提示词进行约束，企业必须同时落实经过培训的人机协同监督与确定性的技术控制手段。最后，她详细介绍了智能体注册表（Agent Register）、智能体卡片（Agent Cards）和熔断器等实用操作工具，认为前瞻性治理不仅不会阻碍，反而能赋能AI的快速规模化。 企业在尝试部署和治理自主智能体AI之前，必须先建立起涵盖传统机器学习和非自主系统的基线治理基础。 (0:04) 零售、酒店和快餐行业在现行消费者保护法、劳工法、就业法和反垄断法下面临广泛的法律风险，破除了“缺乏单一行业监管机构即意味着低风险”的迷思。 (1:50) 新兴AI立法强制要求进行前置透明度披露、对重大决策提出可解释性要求，并在《欧盟AI法案》等框架下规定了严厉的经济处罚。 (1:50) AI治理不能仅仅被视为“隐私保护的延伸”，因为完全基于非个人身份信息（non-PII）运行的AI系统仍可能引发严重的运营和反垄断法律责任。 (1:50) 成熟的基线AI计划需要建立按风险等级分类的获批工具集中清单，并由指定的业务和技术负责人在整个系统生命周期内承担问责。 (5:59)

Reading summary

Deploying agentic AI safely in retail, hospitality, and quick-service restaurant sectors requires establishing a foundational non-autonomous governance stack before introducing specialized human and technical controls, according to enterprise AI legal counsel May Sethaphanich. Dispelling Sectoral Regulatory Immunity (1:50) A common misconception within retail, hospitality, and QSR enterprises is that operating without a dedicated sector-specific regulator, such as those in healthcare or banking, minimizes legal exposure. Long-standing frameworks governing employment, labor standards, consumer protection, and antitrust apply directly to automated operations. Machine learning algorithms that adjust pricing, for example, remain bound by price-fixing prohibitions regardless of whether human managers coordinated the outcome. Regulatory scrutiny is accelerating through state-level transparency mandates, explainability requirements for decisions affecting consumers or workers, and substantial fine structures under the EU AI Act. AI governance cannot be limited to a privacy-centric framework focused only on personal data, because systems manipulating non-PII can still trigger significant operational and regulatory liabilities. Prerequisites of Baseline AI Governance (5:59) Before organizations introduce autonomous agents, they must implement a baseline governance structure for traditional statistical machine learning and generative models. Regulators define AI broadly, yet many enterprises lack an accurate catalog of existing non-autonomous tools. A viable program requires an enterprise-wide inventory mapping every approved system and use case to low, medium, or high risk tiers based on business risk tolerance and statutory exposure. Risk classifications determine pre-deployment validation standards and ongoing monitoring requirements. Each approved tool requires designated business and technical owners responsible for operational and legal compliance. Organizations must also maintain enforceable enterprise AI policies for internal staff, external agencies, and vendors, backed by an AI incident response protocol that addresses algorithmic failures, unauthorized system access, and non-PII legal harms. Expanded Blast Radiuses in Autonomous Agents (9:23) Autonomous agents amplify the non-deterministic nature of generative AI by pairing open-ended model reasoning with the capacity to execute real-world actions. Constraining an agent merely via instructions in its system prompt is technically ineffective because underlying large language models can reason around such textual restrictions. Uncontrolled agents risk executing irreversible external actions, such as generating bulk employee terminations, terminating vendor contracts, or distributing erroneous refunds. Interactions between external systems create additional legal vulnerabilities. Direct agent-to-agent interactions blur operational boundaries, requiring pre-defined architectural patterns and contractual allocations of liability. Furthermore, independent pricing agents ingesting shared macroeconomic or market data feeds risk converging on parallel pricing patterns, which can trigger antitrust investigations for algorithmic collusion even in the absence of explicit human intent. Technical Controls and Operational Readiness (16:57) Governing agentic autonomy demands strict deterministic boundaries. Enterprises should deploy an Agent Register that functions as an authoritative system of record. Every agent record must include an Agent Card detailing its business purpose, accountable owners, whitelisted user groups, accessible tools and APIs, authorized data perimeters, and immutable activity logs for legal defense. Technical circuit breakers must be embedded to pause, quarantine, shift to safe mode, or terminate an agent when anomalies occur. Alongside technical measures, meaningful human-in-the-loop oversight is critical for high-stakes workflows such as hiring or workforce management. For human oversight to be defensible to regulators, capital investments in agentic software must be matched with operational training programs that equip human reviewers to evaluate and challenge agent recommendations. Rigorous multi-tiered governance is not an impediment to deployment, but rather a necessary foundation for enterprises seeking to scale autonomous AI systems legally and sustainably.

精炼阅读摘要

全球快餐企业AI法律顾问May Sethaphanich提出，零售、餐饮和酒店等行业企业在部署自主智能体（Agentic AI）前，必须先建立覆盖传统机器学习与生成式AI的基础治理体系，并通过确定性技术熔断与人机协同机制控制智能体的自主执行风险。 打破无监管误区与多维合规约束 (1:50) 零售与快餐行业常存在一种误解，认为自身不像医疗或金融行业那样面临单一垂直监管机构，因此法律风险较低。Sethaphanich指出这一假设从根本上是错误的。现行的就业法规、劳工标准、消费者权益保护法以及反垄断法均对企业行为形成直接约束。例如在定价领域，不论价格协同是由人工经理还是自动化机器学习算法完成，反垄断机构都严格禁止价格串通。 监管要求正变得更加严格。美国多州正在推进AI立法，要求企业在招聘或客户交互中履行事先披露义务，并对产生实质性影响的决定提供可解释性；在跨国经营层面，违反欧盟《人工智能法案》（EU AI Act）将面临巨额财务罚款。治理框架绝不能仅局限于个人身份信息（PII）保护，即便AI系统完全使用非PII数据驱动，缺乏跨职能监督的部署仍可能触发重大法律责任。 分层治理的第一层：非自主AI基线 (5:59) 企业在探索自主智能体之前，必须先扎实落地传统与非自主AI的治理底座。监管机构对AI的界定极为宽泛，涵盖了传统统计机器学习与生成式模型。成熟的基线治理要求企业建立全机构通用的已批准AI工具与用例清单，并根据法律触发条件及机构风险承受度，将每项资产严格划分为低、中、高风险等级。 每一项获批工具都必须指定具体的业务负责人与技术负责人，在系统的全生命周期内对其合规性与性能承担直接责任。此外，企业必须发布具有约束力的企业级AI政策，规范内部员工、外部代理商、承包商及外聘法律顾问的操作边界，并建立专门的AI事件响应机制，将传统的网络安全事件管理拓展至非数据形态的损害处置与监管合规应对中。 智能体自主权带来的潜在失控风险 (9:23) 智能体系统引入了独特的失效模式，其本质在于自主执行权放大了底层大语言模型（LLM）原有的非确定性特征。Sethaphanich强调，仅靠系统提示词（System Prompt）进行约束并不构成合规技术控制，因为大语言模型可以通过推理绕过这些文字限制。一旦被赋予外部系统操作权限，意外输出将带来不可逆的法律与商业损失，例如错误发出批量解雇通知、终止供应商合同或处理未授权的退款。 更为复杂的外部风险来自多智能体交互与市场算法趋同。当企业内部智能体与第三方智能体（A2A）直接交互时，安全与法律受攻击面大幅扩大，需要提前在商业合同中明确权责划分。此外，若不同竞对企业的定价智能体同时摄取相同的外部市场与宏观经济数据源，算法可能会自发形成趋同定价；即便缺乏人为合意，反垄断机构仍可能介入调查，此时完善的治理文档是证明企业合法性的核心防线。 人机协同与确定性技术控制体系 (15:02) 针对自主智能体的治理，企业必须在基础框架之上叠加人员控制与技术控制。对于招聘、晋升或员工管理等高风险场景，保持人机协同（HITL）是满足监管要求的必备手段。企业不能仅做形式化审核，必须为一线审核人员提供专业操作手册与培训，使其具备有效鉴别智能体输出并阻断错误决策的能力。企业在采购智能体软件与算力基础设施时，必须配套等额的人员培训预算。 在技术层面，必须通过确定性工作流来约束底层模型的随机性。无论企业采用何种开发平台（如Microsoft Copilot、Google Gemini、Vertex AI、Glean或ServiceNow），都必须设立智能体注册表（Agent Register）。每个智能体必须配备包含唯一标识符、核准用途、双线责任人、白名单权限与数据范围的“智能体卡片”（Agent Card），并具备端到端全量日志记录功能与自动熔断机制，确保在检测到异常行为时可立即暂停、隔离、切入安全模式或直接终止系统。 Sethaphanich总结指出，严格的AI治理机制并非技术落地的阻碍，而是企业在零售、餐饮与酒店场景中安全、可持续且规模化部署自主智能体系统的必要前提条件。

Detailed notes — zh

## Background 该内容由 Ai4 频道于 2026 年 8 月 14 日发布在 YouTube 平台。页面简介显示，麦当劳全球高级法律顾问兼 AI 法律负责人 May Sethaphanich 在本次演讲中，深入分享了面向零售、酒店与大型餐饮（QSR）行业的智能体 AI（Agentic AI）实用治理框架。 ## TL;DR 演讲系统阐述了企业构建 AI 治理技术栈的分层逻辑，强调必须先筑牢传统机器学习与非自主生成式 AI 的合规底座，再推进面向自主 Agent 的治理。针对零售餐饮业普遍存在的监管认知误区，重点剖析了反垄断算法定价协同、智能体间交互（Agent-to-Agent）及提示词无法作为有效安全控制等核心风险。为此，企业必须推行高风险场景下“人类在回路（HITL）”与人员培训等人类控制支柱，并同步落地智能体登记册（Agent Register）、智能体卡片（Agent Card）、端到端审计日志与熔断机制等技术控制。最终指出，完备的治理并非阻碍，而是企业实现 AI 规模化与可持续采用的关键基石。 ## Atmosphere & style 整场分享洋溢着法务合规与工程架构深度融合的专业务实氛围，节奏明快、直击行业痛点。演讲者以跨界法务负责人的从容与严密逻辑娓娓道来，既有对技术盲区与合规幻觉的犀利警示，又有条理清晰的落地架构拆解。演讲行云流水且极具说服力，在尾声将“治理”从被动的合规阻力重塑为业务规模化的底层推力，传递出沉稳笃定且赋能前行的信心。 ## Key takeaways - [00:04](https://www.youtube.com/watch?v=PlRM8nzfHaE&t=4s) 企业级 AI 治理必须遵循分层技术栈理念：首先识别法律触发点并夯实传统机器学习和非自主生成式 AI 的基础治理，再在其上叠加自主智能体（Agentic AI）治理层。 - [01:50](https://www.youtube.com/watch?v=PlRM8nzfHaE&t=110s) 零售与餐饮等非强监管行业存在“AI 合规风险低”的危险认知误区，事实上劳动用工、消费者保护、反垄断及新型算法定价等通用法律早已全面适用于 AI 场景。 - [01:50](https://www.youtube.com/watch?v=PlRM8nzfHaE&t=110s) 全球针对 AI 的专门立法（如全美各州法案与《欧盟人工智能法案》）确立了前端透明度披露与可解释性等法定合规要求，且 AI 治理远不止于传统的个人隐私保护（Privacy Plus）。 - [05:59](https://www.youtube.com/watch?v=PlRM8nzfHaE&t=359s) 成熟的基础 AI 治理体系需具备四大核心支柱：风险分级的 AI 资产清单（AI Inventory）、技术与业务双重责任人制、企业统一 AI 政策与实操指南，以及扩展至非个人数据和偏差事件的应急响应协议。 - [09:23](https://www.youtube.com/watch?v=PlRM8nzfHaE&t=563s) 大语言模型的非确定性使 Agentic AI 的风险显著放大；技术团队不能将系统提示词（System Prompt）误当作安全控制手段，因为模型完全可以绕过提示词限制。 - [09:23](https://www.youtube.com/watch?v=PlRM8nzfHaE&t=563s) 赋予面向外部的自主 Agent 直接决策和操作权限极易引发不可逆的法律灾难，例如误发解聘通知、错误发放退款或未妥善鉴权导致的数据泄露合规通报危机。 - [11:32](https://www.youtube.com/watch?v=PlRM8nzfHaE&t=692s) Agent-to-Agent（跨主体智能体交互）拓宽了安全攻击面与责任风险，必须在系统设计初期拉入法务与架构师审查，并通过前置商业合同条款明确划分各方责任边界。 - [13:02](https://www.youtube.com/watch?v=PlRM8nzfHaE&t=782s) 多公司自主定价 Agent 读取公开宏观及市场数据可能自发导致算法价格趋同，即便无主观串通仍可能触发反垄断调查，企业必须依靠完备的治理与护栏文档构建“法律抗辩力”。 - [15:00](https://www.youtube.com/watch?v=PlRM8nzfHaE&t=900s) 在招聘与用工管理等高风险场景中，“人类在回路”（HITL）是不可逾越的合规底线，企业 AI 预算必须在采购工具/基建与人员实操培训之间取得平衡。 - [16:57](https://www.youtube.com/watch?v=PlRM8nzfHaE&t=1017s) 技术控制落地依托“智能体登记册”（Agent Register）与“智能体卡片”（Agent Card），强制绑定唯一标识符、双线责任人、核准用途与数据访问权限，防止影子 Agent 泛滥。 - [16:57](https://www.youtube.com/watch?v=PlRM8nzfHaE&t=1017s) 生产环境中的 Agent 必须具备端到端追踪审计日志以及分级升级与自动熔断机制（暂停、安全模式、网络隔离、下线销毁），以应对运行时异常并留存诉讼证据。 - [19:04](https://www.youtube.com/watch?v=PlRM8nzfHaE&t=1144s) AI 治理从来不是阻碍企业创新的绊脚石，而是制定 AI 战略、确保负责任落地并实现长远规模化商业扩张的核心基石。 ## Detailed walkthrough ### [00:04](https://www.youtube.com/watch?v=PlRM8nzfHaE&t=4s) 讲师背景与 AI 治理技术栈的分层理念 作为全球大型快餐连锁（QSR）企业的全球高级法律顾问与 AI 法律负责人，May Sethaphanich 长期与工程师、架构师、开发者以及数据科学家紧密协作，在赋能业务的同时共同主导企业级 AI 治理体系的建设。在开启正式演讲前，她明确声明本次分享仅代表其个人观点，不代表任何关联机构的立场。本次演讲的核心议题聚焦于“智能体 AI”（Agentic / Autonomous AI）的治理框架构建。 ```mermaid flowchart TD accTitle: AI 治理技术栈分层模型 subgraph Layer2 ["第二层：智能体治理 (Agentic Governance)"] A2["自主决策与工具调用控制"] A3["多智能体交互边界约束"] end subgraph Layer1 ["第一层：基础治理 (Foundational Governance)"] B1["传统机器学习 (ML) 治理"] B2["非自主生成式 AI (Non-autonomous Gen AI) 治理"] end subgraph Base ["底层驱动：法律认知与业务风险"] C1["现有行业通用法规 + 新兴 AI 专门立法"] end Base --> Layer1 Layer1 --> Layer2 ``` May Sethaphanich 强调，构建 AI 治理技术栈必须遵循清晰的分层逻辑。第一层也是最基础的一步，是对适用于自身业务场景、用例和工具的法律法规建立全面认知。驱动治理框架落地的关键在于识别企业所面临的“法律触发点”（Legal Triggers）。更为重要的是，这层基础治理绝不能仅局限于生成式 AI（Gen AI），必须完整覆盖传统的机器学习（Machine Learning）算法。企业必须在非自主 AI 的合规底座稳固建立之后，再行推进面向自主智能体 AI（Agentic AI）的上一层治理。 --- ### [01:50](https://www.youtube.com/watch?v=PlRM8nzfHaE&t=110s) 零售与餐饮行业的法律合规误区与监管新浪潮 在零售、酒店与餐饮（QSR）行业中普遍存在一个极其危险的认知误区：许多企业认为自身既不像金融业也不像医疗行业那样受到严格的垂直监管机构约束，因此面临的法律风险微乎其微，“可以自由随意地设计和部署 AI”。然而事实截然相反，一系列长期存在的通用法律早已适用于各类 AI 应用场景： * **劳动与雇佣法规**：规范 AI 在招聘筛选、绩效评估和员工管理中的介入。 * **消费者权益保护法**：禁止欺诈性交互、虚假陈述或损害消费者利益的自动化操作。 * **反垄断与反不正当竞争法**：明确禁止无论是人类之间还是通过 AI 实施的价格协同与固定。如果企业的定价智能体开始与竞品智能体产生协同定价的迹象，将面临极为严重的法律后果。 * **新型算法定价法规**：针对动态定价和算法定价设立了全新的合规门槛。 除传统法律延伸外，全美各州正在掀起一波针对 AI 的专门立法浪潮。这要求企业在利用 AI 与消费者或员工互动时履行多项新法定义务，其中最关键的是**前端透明度披露**（明确告知求职者、员工或消费者正在与其交互的是 AI）以及**可解释性要求**（能够解释 AI 如何做出决策或辅助决策）。企业不能只顾着盲目投资工具、生态与基础设施，必须同步投资于后台的合规与可解释性能力建设。 此外，对于在欧洲开展业务的企业而言，《欧盟人工智能法案》（EU AI Act）至关重要。作为 AI 领域的“GDPR”，该法案规定了极其严苛的合规义务与高额违规罚金。May 特别纠正了行业内将 AI 治理等同于“隐私保护升级版”（Privacy Plus）的狭隘观点：尽管数据隐私至关重要，但现实中大量 AI 用例处理的是非个人身份信息（Non-PII），若使用不当同样会引发巨大的实质性法律后果。 --- ### [05:59](https://www.youtube.com/watch?v=PlRM8nzfHaE&t=359s) 基础 AI 治理体系的核心组成要素 虽然业界言必称“智能体”，但许多团队在被问及非自主 AI 的基础治理时往往语焉不详。监管机构通过极其宽泛的法律定义，将传统机器学习算法与非自主生成式 AI 均纳入监管视野。一套成熟的基础 AI 治理框架必须包含以下四个关键支柱： | 核心组件 | 关键职责与实施要点 | |:--- |:--- | | **已批准工具与用例资产清单 (AI Inventory)** | 登记企业内所有经审批的 AI 工具与场景，依据企业风险承受度及适用法律打上“低/中/高”风险分级标签，不同级别对应差异化的部署前后合规要求。 | | **业务与技术双重责任人 (Owners)** | 明确指定技术负责人与业务负责人，在工具的全生命周期内承担问责制（Accountable），确保其不仅在商业指标上达标，更在法律合规层面持续受控。 | | **企业 AI 政策与实操指南 (AI Policy)** | 清晰阐明企业 AI 原则，为内部员工、供应商、营销与公关代理商乃至外部法律顾问提供统一的“行为准则（Dos and Don'ts）”。 | | **AI 事件响应响应协议 (AI Incident Protocol)** | 复用并扩展企业原有的安全应急响应能力，将覆盖面从单纯的个人数据泄露（PII Breach）扩展到非个人数据异常、输出偏差引发的重大法律事件。 | May 指出，明确风险分级之所以至关重要，是因为它直接决定了业务与技术团队在系统上线前及上线后必须遵守的合规动作与控制措施。 --- ### [09:23](https://www.youtube.com/watch?v=PlRM8nzfHaE&t=563s) Agentic AI 的独特风险与系统提示词的局限性 进入智能体 AI（Agentic AI）阶段后，大语言模型（LLM）固有的“非确定性”（Non-deterministic nature）会被自主性进一步放大，这也是为什么必须在基础框架之上叠加专门治理层的原因。May 直截了当地指出，技术团队中常见的一个致命假设是将系统提示词作为防御手段： > “如果你的智能体仅仅依靠系统提示词（System Prompt）里的某些指令来进行约束，那根本算不上是一种控制，因为大语言模型完全能够绕过系统提示词中的限制进行推理。” 当一个面向外部的自主智能体被赋予做决策并直接执行操作的权限时，一旦出现错误，其引发的行为往往是极难撤销（甚至是不可逆）且伴随严重法律后果的。例如： * 智能体误向一批员工或供应商自动发送了解聘/终止合作通知； * 客服智能体由于逻辑偏差错误地向大量客户发放了退款； * 智能体拥有调用内部/外部工具与数据库的权限，包含公司专有商业机密或员工与客户的个人信息；一旦智能体在未经妥善鉴权的情况下违规读取这些数据，在法律上即构成数据泄露，将直接触发向隐私监管机构和受影响个人进行强制通报的法定义务。 --- ### [11:32](https://www.youtube.com/watch?v=PlRM8nzfHaE&t=692s) Agent-to-Agent（智能体间交互）的新兴攻击面与责任边界 随着技术演进，一个新的重大风险场景正在显现：当企业自身的智能体被赋予自主权去与第三方智能体（Agent-to-Agent）直接进行交互与协作时。 May 强烈建议治理团队对此类场景启动最高级别的审查，在架构初期就必须将法律顾问、解决方案架构师及核心业务人员深度拉入。这种跨系统的智能体互联极大地拓宽了企业的安全攻击面与法律风险暴露面。核心难点在于，除非企业在事前与第三方通过明确协议界定清晰，否则“我的智能体责任从哪里开始、在哪里结束，对方智能体的责任又该如何划分”，彼此之间的法律责任边界将变得极度模糊且不可控。 ### [12:37](https://www.youtube.com/watch?v=PlRM8nzfHaE&t=757s) Agent 间交互规范与合同责任划分 在多个自主智能体（Agent）协同工作的生态中，企业面临的首要问题不仅是单个智能体在做什么，而是“我的智能体与你的智能体之间该如何交互”。May Sethaphanich 强调，跨主体的智能体交互模式绝不能放任自流，必须在系统设计之初就清晰拟定交互模式，并经过高层资深业务与技术利益相关者的严格审查与批准。 为了杜绝出现事故后相互推诿的灰色地带，她强烈建议企业将这类交互规则以法律合同条款的形式固定下来。通过清晰的前置合同条款，在各方之间明确划分责任边界，确保在发生意外错误或损失时，责任归属一目了然。 - 必须对跨主体 Agent 间的通信与协作模式设立标准，经由高层利益相关者审批。 - 强烈建议通过前置商业合同条款锁定法律责任分配，确保责任归属清晰透明。 --- ### [13:02](https://www.youtube.com/watch?v=PlRM8nzfHaE&t=782s) 新型风险敞口：自主定价智能体与反垄断合规 除了交互责任，May 提出了一个全新的风险向量：当智能体被赋予外部数据访问权限时，可能引发意想不到的连锁反应。许多智能体为了完成自身任务，必须拉取宏观经济数据或外部市场行情，最典型的例子就是跨行业的“定价智能体”（Pricing Agents）。 May 设想了一个极具现实意义的合规危机场景：几家毫无关联、互为直接竞争对手的公司，各自部署了自主定价智能体。这些智能体独立抓取公开的市场数据、宏观经济指标以及竞品信息。在没有任何人类串通意图的情况下，各个智能体算出的价格模型却在算法驱动下不谋而合，最终导致市场价格趋同与协同（price convergence / coordination）。这种现象极易引来反垄断监管机构的严厉审查与质询。 面对这种情况，企业必须依赖完备的治理文档建立“法律抗辩力”（legal defensibility）。企业需要有据可查地向监管机构自证：在部署这些自主 AI 之初，公司就已经设置了合规护栏；价格趋同纯属技术运行中的意外副产物，绝非人类指使的价格卡特尔。 ```mermaid flowchart TD accTitle: 智能体外部数据访问与反垄断风险模型 subgraph S1["竞争对手自主系统"] A1["公司 A 定价 Agent"] A2["公司 B 定价 Agent"] end subgraph S2["外部数据源"] D1["宏观经济指标"] D2["公开竞品市场数据"] end subgraph S3["意外风险与法律防线"] R1["算法自发价格趋同 / 协同"] R2["反垄断监管机构介入调查"] L1["合规文档与安全护栏：<br/>构建法律抗辩力 (Legal Defensibility)"] end D1 --> A1 D1 --> A2 D2 --> A1 D2 --> A2 A1 --> R1 A2 --> R1 R1 --> R2 L1 -. 应对自证.-> R2 ``` - 核心风险点：定价等 Agent 因读取相同外部经济/市场数据，自发形成价格趋同。 - 监管后果：即使毫无人类串通共谋的主观意愿，仍可能触发反垄断法调查。 - 破局关键：依靠完备的治理与护栏开发文档，作为向监管层证明合规尽职的法律抗辩防线。 --- ### [15:00](https://www.youtube.com/watch?v=PlRM8nzfHaE&t=900s) 治理控制支柱一：高风险场景下的“人类在回路”与培训投入 为了应对复杂的合规与技术挑战，May 指出必须在智能体治理框架中引入两大核心控制支柱：人类控制（Human Controls）与技术控制（Technical Controls）。对于 AI 成熟度尚处于初级阶段的企业而言，“人类在回路”（Human-in-the-loop, HITL）不仅不可或缺，更是抵御早期系统失控的底线保障。 在涉及高风险的应用场景中，监管部门对人类介入的要求更为苛刻。May 以人力资源与用工管理为例：如果企业计划使用 Agent 协助招聘、评估或员工管理，监管机构会严查企业是否采取了充分的风险缓释手段，以及是否给予了“系统另一端被影响的活生生的人”足够的尊重。在关键决策落地执行之前，必须有经过训练的人类对智能体给出的建议结果进行严格复核。 与此相对应，企业必须为高风险场景编写配套的培训手册，作为员工接受过合格操作培训的合规证据。May 直言，很多企业在采购尖端工具和搭建 AI 基础设施上挥金如土，但一个真正成熟的 AI 战略必须拿出专项预算投入到“人的赋能”上，让员工掌握与自主 AI 谨慎协作、把控风险的能力。 - 人类控制在用工管理、自动化招聘等高风险用例中属于强监管红线。 - 建立标准操作流程与培训手册，留存培训证明，确保复核人员具备审核 Agent 建议输出的专业能力。 - AI 投资预算必须在工具/基建与人员培训之间取得平衡，不能“重技术、轻人员”。 --- ### [16:57](https://www.youtube.com/watch?v=PlRM8nzfHaE&t=1017s) 治理控制支柱二：确定性工作流、智能体登记册与熔断机制 在技术控制方面，May 简要重申了底层逻辑：生成式 AI 具有固有的非确定性（non-deterministic），因此必须利用“确定性工作流”（deterministic workflows）来对冲自主系统带来的不可控风险。这一机制应深度嵌入企业使用的各类平台中，无论是 Microsoft Copilot、Google Gemini、Vertex AI，还是 Glean 或 ServiceNow。 落地技术控制的核心抓手是建立“智能体登记册”（Agent Register），其逻辑等同于此前管理传统 AI 时的工具与用例清单。任何获批投入生产环境的 Agent 都必须在此登记，并生成唯一的“智能体卡片”（Agent Card）。卡片中不仅要标注唯一识别符，还必须强制绑定业务与技术双线的人类负责人，由其对该 Agent 遵守核准协议承担终身责任。 ```mermaid flowchart LR accTitle: 智能体卡片与技术控制架构 subgraph AC["Agent Card 核心要素"] direction TB C1["唯一识别符 (Unique ID)"] C2["业务与技术负责人"] C3["核准用途与授权用户"] C4["工具与数据访问权限"] end subgraph TC["运行监控与防御体系"] direction TB M1["端到端全链路活动追踪"] M2["审计日志 (支持法律抗辩)"] M3["熔断与升级协议：<br/>暂停 / 安全模式 / 隔离 / 下线"] end AC ==> TC ``` 此外，系统必须具备端到端追踪 Agent 活动的报告与审计日志能力，尤其是高风险用例，这些日志是未来面临法律诉讼时的核心取证依据。同时，体系内必须内嵌分级升级协议与“熔断机制”（Circuit Breakers），一旦检测到异常指标，能够立即对智能体执行挂起、暂停、切换安全模式、网络隔离甚至彻底强制下线。 - 智能体登记册（Agent Register）：上线必须备案，防止影子 Agent（Shadow Agents）滋生。 - 智能体卡片（Agent Card）四要素：唯一标识符、双线责任人（业务与技术）、核准用途与授权使用人群、核准工具与数据访问权限边界。 - 运行时防护：端到端活动报告、法律级审计日志记录，以及具备“暂停、安全模式、隔离、销毁”多重档位的自动熔断协议。 --- ### [19:04](https://www.youtube.com/watch?v=PlRM8nzfHaE&t=1144s) 结语：治理不是阻碍，而是规模化采用的核心基石 在演讲尾声，May 坦言由于时间紧迫，许多准备好的演示胶片不得不快速略过，但与会者可以在大会结束几天后下载完整讲义。在全场热烈的掌声前，她留下了这场分享最核心的论点： "治理从来不是企业 AI 采用进程中的绊脚石或阻力。"（"governance is not a drag on progress on your company's AI adoption."）恰恰相反，完善的治理框架是推动负责任 AI 落地的先决条件。在制定任何企业的 AI 战略时，治理都应被视为不可分割的底层支柱——因为唯有建立在坚实治理之上的 AI，才能在商业世界中实现成功、稳健且可持续的长远规模化扩张。 ## Notable quotes > [01:50](https://www.youtube.com/watch?v=PlRM8nzfHaE&t=110s) 可以自由随意地设计和部署 AI > [09:23](https://www.youtube.com/watch?v=PlRM8nzfHaE&t=563s) 如果你的智能体仅仅依靠系统提示词（System Prompt）里的某些指令来进行约束，那根本算不上是一种控制，因为大语言模型完全能够绕过系统提示词中的限制进行推理。 — May Sethaphanich > [11:32](https://www.youtube.com/watch?v=PlRM8nzfHaE&t=692s) 我的智能体责任从哪里开始、在哪里结束，对方智能体的责任又该如何划分 — May Sethaphanich > [12:37](https://www.youtube.com/watch?v=PlRM8nzfHaE&t=757s) 我的智能体与你的智能体之间该如何交互 — May Sethaphanich > [13:02](https://www.youtube.com/watch?v=PlRM8nzfHaE&t=782s) 法律抗辩力 — May Sethaphanich > [15:00](https://www.youtube.com/watch?v=PlRM8nzfHaE&t=900s) 系统另一端被影响的活生生的人 — May Sethaphanich > [15:00](https://www.youtube.com/watch?v=PlRM8nzfHaE&t=900s) 重技术、轻人员 — May Sethaphanich > [19:04](https://www.youtube.com/watch?v=PlRM8nzfHaE&t=1144s) 治理从来不是企业 AI 采用进程中的绊脚石或阻力。 — May Sethaphanich > [19:04](https://www.youtube.com/watch?v=PlRM8nzfHaE&t=1144s) governance is not a drag on progress on your company's AI adoption. — May Sethaphanich ## Mentioned May Sethaphanich, McDonald's Corporation, Ai4, 欧盟人工智能法案 (EU AI Act), 通用数据保护条例 (GDPR), Microsoft Copilot, Google Gemini, Vertex AI, Glean, ServiceNow ## Video description May Sethaphanich - Global Senior Counsel, AI Legal Lead, McDonald's Corporation Governing Agentic AI - a Practical Framework for Retail, Hospitality and QSR. [Source](https://www.youtube.com/watch?v=PlRM8nzfHaE)

Source quotes

[1:50] 可以自由随意地设计和部署 AI [9:23] May Sethaphanich: 如果你的智能体仅仅依靠系统提示词（System Prompt）里的某些指令来进行约束，那根本算不上是一种控制，因为大语言模型完全能够绕过系统提示词中的限制进行推理。 [11:32] May Sethaphanich: 我的智能体责任从哪里开始、在哪里结束，对方智能体的责任又该如何划分 [12:37] May Sethaphanich: 我的智能体与你的智能体之间该如何交互 [13:02] May Sethaphanich: 法律抗辩力 [15:00] May Sethaphanich: 系统另一端被影响的活生生的人 [15:00] May Sethaphanich: 重技术、轻人员 [19:04] May Sethaphanich: 治理从来不是企业 AI 采用进程中的绊脚石或阻力。 [19:04] May Sethaphanich: governance is not a drag on progress on your company's AI adoption.

Source quotes

[0:04] May Sethaphanich: organizations cannot simply skip ahead to autonomous agents without first establishing a robust baseline governance stack. [0:04] May Sethaphanich: This foundational layer cannot be restricted to generative AI alone; it must comprehensively govern traditional machine learning (ML) and non-autonomous systems. [1:50] May Sethaphanich: A dangerous misconception persists across the QSR, retail, and hospitality sectors that companies operate with minimal legal exposure because they lack a single dedicated sectoral regulator [1:50] May Sethaphanich: While personal data (PII) is vital, AI systems driven entirely by non-PII can still trigger catastrophic legal liability if deployed without rigorous cross-functional oversight. [5:59] May Sethaphanich: Despite the current hype surrounding autonomous systems, many organizations struggle to articulate even their basic inventory of approved non-autonomous tools. [9:23] May Sethaphanich: if your agent is just being constrained by... instructions in the system prompt, now that is not a control because that restriction in the system prompt can be reasoned around by the LLM. [9:23] May Sethaphanich: When an autonomous agent is granted the authority to execute external actions, unintended outputs can cause immediate, irreversible legal and financial damage. [11:32] May Sethaphanich: Even without any human intent or explicit agreement to collude, antitrust regulators may launch investigations into suspected algorithmic price-fixing. [15:02] May Sethaphanich: whenever enterprises budget capital to acquire agentic software and infrastructure, they must allocate matching investments toward workforce training so humans can oversee autonomous systems with due care. [16:57] May Sethaphanich: To counter the non-deterministic behavior of underlying large language models, technical controls must enforce deterministic workflows around autonomous agents. [16:57] May Sethaphanich: Far from acting as a drag on adoption, rigorous governance serves as an indispensable prerequisite for scaling AI safely, sustainably, and successfully across the enterprise.

Original transcript

[0:04] Hi everyone, my name is uh May Satanich.  I'm AI council um in a global QSR company uh [0:11] service restaurant. Um I work with um engineers,  architects and developers um data scientists to [0:20] build and develop AI solutions to enable  the business. I also co-own the company's [0:26] enterprise uh governance program. So today I'm  going to talk to you about um the governance of [0:32] agentic um you know autonomous AI. But before I go  any further, I'll just like to preface my remarks [0:38] by saying that the views that I'll be expressing  are those of my own and should not be attributed [0:43] to any organization that I'm associated with.  So with that out of the way, um let's get going. [0:52] So I will brief briefly describe uh that  the governance stack needs to be as follows. [0:58] There should be the first layer of an awareness  of the laws that applies to your business. Um you [1:05] know the use cases and the tools. um the legal  tri triggers that are relevant to your business [1:12] should define the um governance of um a of should  define the the governance of AI that will be [1:20] used in your company. So I would say that the  governance layer needs to cover not only gen AI [1:27] but u machine learning as well. So is critical to  have that basic foundational layer that governs [1:34] non autonomous AI to be in place first.  um you know before people start thinking [1:39] about deploying Agent AI which will be the next  layer of uh governance uh to be uh built upon. [1:50] There is a common misconception that in the QSR  retail and hospitality space that hey you know [1:56] what legal exposure is minimal. We don't have a  specific regulator uh that oversees this sector [2:03] unlike the finance healthcare industry. So you  know what um yeah we we are pretty much free to [2:08] design and deploy but that is a really dangerous  uh misconception because there's an list of [2:15] longexisting laws that would apply if you're you  know engaging with your consumers your employees [2:22] there are existing employment laws labor  laws consumer protection laws um antitrust [2:28] laws that outlaws um you know price fixing  amongst humans as well as AI and there are [2:34] also antitrust consideration ations as well.  Um you know it's uh you know if your uh agents [2:40] were to start for instance creating prices that  appear to be coordinating with your competitors [2:45] that could be a problem. Um there I think um you  there there are also now new um pricing algorith [2:52] algorithmic laws that um that would uh impose  new requirements on companies. Um so machine [3:00] learning you know this is stuff that has been  in existence for a long time but there are now [3:04] new obligations because the regulators are  extremely interested in how we're using [3:08] not only gen AI but machine learning algorithms as  well that may create risks or harms to consumers. [3:15] So along all of that alongside all of that is  a new wave of AI specific laws that are being [3:22] passed throughout the states in this country. Uh  basically what that means is that there are new [3:27] obligations to be complied with by companies that  are using AI to interact with their consumers, [3:34] their employees. uh new obligations such as  upfront transparency disclosures to inform the [3:40] consumer, the job applicant or your employee that  hey you know what this is an AI that's interacting [3:45] with you and over above over and above that there  is that expectation that these companies be able [3:52] to explain how the AI is being used to make the  decision or to even assist in making that decision [4:00] that uh could you know result in some sort of  harm or decision to the employee job applicant [4:05] or consumer. The bottom line is that we um you  know as we are racing towards investing in tools, [4:13] building new tools into the ecosystem,  setting up infrastructure that there's [4:17] also a need to consider in investing in efforts  to build out these uh backroom capabilities [4:23] to uh you know um to uh enable explanability  and to comply with these new legal obligations. [4:33] It would also be remiss of me  if I don't mention the EU AI act [4:36] which is uh important relevant to those uh folks  who have uh business operating in that EU region. [4:42] This is a very significant piece of legislation.  It is basically the uh EU equivalent the AI [4:48] equivalent of the GDPR regulations. Um it contains  uh pretty substantial uh obligations uh uh [4:57] obligations for compliance and along with it  comes hefty fines for violation of the law. [5:04] I also want to call out that um I just  want to mention that in my discussions [5:08] with many um governance folks in this space that  often times I come away with the impression that [5:13] the governance program is pretty much of a  you know privacy plus sort of a framework. [5:19] Now don't get me wrong while privacy you know  continues to be a really important uh risk [5:24] factor in the governance of AI but as we know the  usage of AI tools and use cases often are enabled [5:31] are driven by nonPPI that could still result in  substantive um you know legal consequences for [5:37] the companies if uh the AI is not being used  uh appropriately. So I would just recommend [5:43] for um you know a consideration of your governance  program to ensure that considerations u of other [5:49] um triggers um under you know laws other than  privacy laws be taken into consideration. [5:59] So these are a few components that should  be in every basic AI governance program. And [6:06] I say that this is really important to cover  both uh traditional machine learning um [6:13] AI algorithms as well as uh non-aututonomous  uh gen AI because that's what the regulators [6:18] are focusing on. the laws that are being passed  define AI very broadly to cover not only gen AI [6:25] and certainly uh autonomous uh autonomous  AI but machine learning algorithms as well. [6:31] So I know that you know what this may be 101 for  a lot of folks but again in my conversations with [6:36] a lot of folks that are thinking their minds are  already in the agentic autonomous space but if you [6:41] were to query them about hey you know what are you  able to articulate what your basic AI governance [6:45] framework is on the non autonomous stuff they'll  be hardressed to you know to talk about the [6:52] AI uh inventory of uh uh proof tools and use cases  for their company. So there should be an inventory [7:00] of these approved tools and use cases that are  tagged to a risk uh classification uh risk tier [7:06] classification of either low, medium or high risk  and that tiering should be again you know packed [7:11] to your company's risk classification criteria  that is driven by the laws that are applicable [7:16] to your business as well as you know a function  of your uh organization's uh risk tolerance. [7:22] Now the reason that the risk taring classification  is very important because it drives the [7:27] the the the type of requirements that need to  be complied with by the business as well as the [7:33] technical owner of those tools. The requirements  that need to be met before and after deployment. [7:40] Now this group of folks the owners are extremely  important because they'll be responsible meaning [7:45] accountable for uh ensuring that the tool  during its deployment cycle is performant [7:51] is performing in a way that is legally compliant  and not just performant uh from a business sense. [7:58] So um it'll be remiss of me if I don't mention  the AI policy again you know it may be obvious [8:04] but uh you know to some of us but you know  again a clear articulation of what the your [8:10] company's AI principles are and also practical  guidelines um you know uh for for the relevant [8:17] stakeholders in your company whether it's  employees vendors your marketing agencies [8:23] PR agencies or even your outside council um just  to set it you know very clear to everyone that [8:29] engages your company using AI tools as to you know  the dos and don'ts and really you know to make [8:35] sure that you know responsibility uh people are  held to accountability and I would also want to [8:42] mention the last point on the incident uh protocol  now this is critical and this is a protocol that [8:48] could be built on top of uh your organization's uh  incident incident response uh protocol hopefully [8:55] you know that particular layer of muscle is  already well developed but an AI incident [9:00] AI incident protocol should be added on top of  that as well because AI incidents do not comprise [9:06] of only data or P data or PI breaches but other  incidents that may not in that involve a nonPI [9:14] but could uh you know uh be of um significant  legal consequences to your company as well. [9:23] But what makes Agentic AI different such  that we should have an additional layer [9:29] to govern Agentic AI on top of that AI  governance framework that I've just described. [9:36] I would say that it's because of the inherent  risk associated with the non-deterministic nature [9:42] of Gen AI that becomes amplified  in an autonomous AI situation. [9:48] And I just want to add this that if your agent  is just being constrained by u you know some [9:54] instructions in the system board uh in the system  prompt now that is not a control because that [10:00] that restriction in the system uh prompt  can be reasoned around by the LLM. [10:09] So imagine an autonomous AI that is external  facing and is given the autonomy to make certain [10:17] decisions um make certain decisions followed by  certain um actions that could be hard to reverse [10:24] that may be irreversible with legal consequences  for your company. um you know uh so one example [10:31] could be of an agent that sends out termination  notices to a group of employees wrongly or you [10:37] know to a group of vendors for instance by mistake  or a um customer service uh agent that makes [10:44] a series of uh refunds to customers that are you  know again you know done by mistake. um autonomous [10:52] AI will be given access to tools and databases,  right? Those tools could be internal or external [10:58] and the databases could contain um company uh  company confidential proprietary information [11:04] or personal information of employees, customers  and other stakeholders. Now if the agent you know [11:10] somehow uh uh you know uh is is is able to access  these databases without proper authorization um [11:17] that would constitute uh data breach that  would trigger obligations uh of reporting [11:23] to uh regulators privacy regulators as well as to  the affect as well as to the affected uh humans. [11:32] I would also want to mention a new uh risk  factor which is where my agent my agent is given [11:38] the uh you know the uh the the autonomy and the  the excess permission to interact with your agent [11:46] when that situation is being contemplated.  I would recommend a higher level of scrutiny [11:51] by your governance team. Please involve your your  your lawyers as well as your uh you know solution [11:57] architects and other folks who are involved uh  in this uh in involved in this particular use [12:03] case where you know agents are given access to  interact with third party agents because there [12:08] you're opening up u a new surface area of uh of  uh of legal exposure as well as security and other [12:16] forms of of exposure. The the reason is that you  know the the the the line between you know where [12:24] my the my where the uh the responsibility  of my agent starts and end with yours [12:30] could become could be a little uh uncertain unless  you together with that third party define upfront [12:37] as to the pattern of interaction between my agent  and your agent which should be uh you know set out [12:44] and reviewed and approved by you know the  appropriate senior stakeholders and preferably [12:50] if that interaction could be also backed up by  contractual provisions so that the allocation [12:56] of liability is very clear and upfront that would  be you know most preferable and highly recommended [13:02] and [snorts] finally I would like to mention  this other um new vector uh of of risk which is [13:07] the unintended um actions of uh of agents that  are given access to external data um you know [13:15] it could be external macro proeconomic data that  this agent that this agent needs to have access to [13:22] to do what it's built to do. For instance,  pricing agents. Now, imagine a scenario where [13:27] pricing agents of multiple companies unrelated  to one another. But in order for it to perform [13:33] what it does, it needs access to uh external  market information, right? Uh from economic data [13:38] to potentially, you know, the data of other uh  companies that produce similar sort of products. [13:44] Now if your agents inevitably create a you know a  collection of agents uh from companies unrelated [13:51] to one another one one another but our competitors  uh but our competitors end up you know producing [13:57] prices that somehow align and converge. It could  create a situation where the antitrust regulators [14:06] may become interested and want to have you know  a discussion with the uh relevant companies. [14:12] Now that could occur even if that this price  co coordination alignment convergence phenomena [14:19] was not driven by any human intent at all. Now  in that situation it will be extremely helpful [14:26] to have documentation to have governance  documentation in your company that's been [14:30] created to demonstrate that hey in you you know in  in in us deploying this uh these uh autonomous AI [14:37] that there were guardrails that were in place  uh to ensure you know with to to ensure there's [14:41] there's legal compliance but unfortunately the uh  inevitent uh event of of uh price coordination has [14:48] has happened basically you know the creation  of documentation to um to uh you know back up [14:54] your legal defensibility. So I would suggest  that two additional um categories of control [15:02] be uh included uh in an agentic governance  framework. The first will be human controls and [15:08] the second will be the category of uh technical  controls. So the human still needs to be involved [15:14] especially uh if I may say for companies that  are that are on the lower end of AI maturity as [15:21] it would be really important to still have humans  in the loop especially for the higher risk uh use [15:27] cases such as you know in the employment  space if you're planning to use agent AI [15:32] to uh you know to assist in the hiring uh employee  management uh spaces those are the high-risisk use [15:40] uh use cases that the regulators would expect  us to be able to demonstrate that we're using [15:45] AI in a way that uh you know mitigates the risk  and is respectful of the humans on the other [15:50] side of the fence who are being affected.  Now it will be useful to have those humans [15:56] who are in the loop to be properly trained as  well to know how to review for instance the output [16:02] uh that is suggested by the agent before  actions are being taken. I would also say [16:06] that for high-risisk use cases in particular that  training manuals be uh be created to evidence that [16:13] proper training of the humans uh take place uh you  know to ensure that you know that the AI is being [16:20] used in conjunction with humans that are properly  uh trained and equipped to manage down the risk [16:25] uh of um you know uh to manage down the risk  for the company as well as the uh the the humans [16:31] uh on on the other side. you know given the fact  that there's so much that's invested in bringing [16:37] in new tools to set up infrastructure to enable  a aentic uh to enable AI I would say that for any [16:44] AI strategy that investment should also be set  aside to train the humans to use these tools [16:50] um you know uh um with uh with with due care  in partnership with with the autonomous AI [16:57] I'm going to be very quick about these technical  controls I've mentioned earlier that you know it's [17:01] important due to the non-deterministic nature of  uh of of Gen AI that we need to have deterministic [17:08] workflows to kind of balance out the risk  associated with using these autonomous AI [17:13] aside from having humans in the loop. So I'll say  that in when in any agent uh development platform [17:21] whether it's your co-pilot, Gemini, Vertex AI,  Glean or Service Now that an agent register be [17:28] set up. This agent register is the equivalent of  the uh of the AI tool and approve the AI tool and [17:35] use case inventory that I mentioned earlier on. So  you just as you know you should have an inventory [17:41] of those approved tools and use cases. Now you  need to have a register of all the agents that [17:47] are approved uh to um you know approved to go  live in your environment. Now in that agent uh [17:53] register there should be agent and cards that  contain unique identifiers of each of these agents [17:58] and very importantly it's human uh owners  both business and and technical need to be uh [18:04] identified as well because these folks again will  be held will be held responsible for maintaining [18:10] the um you know for for ensuring that  the agent is uh is um is is you know is [18:17] uh is following the approved protocol uh for its  purposes. So the agent card needs to articulate [18:23] the purposes of the agent, its approved uh  users, its uh approved uh tools and data accesses [18:30] and there should also be a reporting capability  in um uh on on that particular platform as well [18:36] to track the activities of the agent from start  to end and an escalation protocol that articulates [18:42] the sort of events that will trigger circuit  breakers that will halt, pause the agent, [18:46] put it into safe mode, quarantine it or to you  know take it down. Now, there should also be a [18:51] logging capabilities to create documentation of  how your how all these agents are being managed, [18:56] especially for the high-risk use cases  because again for legal defense uh purposes. [19:04] So, I'm running out of time. There are number  of of slides that I need to skip over, but this [19:09] deck will be available for download uh a few days  after the um conference. But I just want to end to [19:16] say that hey, you know what? governance is not a  drag on progress on your company's uh AI adoption. [19:21] In fact, I would argue that it is critical to  enable the to enable the responsible adoption [19:27] of AI. It should be considered as a critical  component in any AI strategy because that's how [19:33] we can scale AI successfully and sustainably  over time. And with that, I thank you. [applause]