## Background
Published on YouTube by Ai4 on August 22, 2026, this page features a panel discussion on guardrails, risk management, and observability for agentic AI. According to the page description, experts Nell Meshcheryakov, Sagi Rodin, Monisha Somji, Sohaib Shaikh, and Jill Hosmer-Jolley discuss how enterprise leaders can balance agility with reliability to keep autonomous agents safe, transparent, and high-performing at scale.
## TL;DR
The panel explores the operational transition to autonomous AI agents, highlighting the necessity of hard guardrails like policy as code, role-based access control, and least privilege permissions. Panelists emphasize protecting core data assets through Zero Trust frameworks and linking every agent to a clear human owner rather than trying to govern every generative output. Real-world case studies in the energy sector and recruitment illustrate the distinction between human-in-the-loop and human-on-the-loop oversight, while practical risk-categorization frameworks help manage process-specific guardrails across diverse business units.
## Atmosphere & style
The session carries an engaging, practical, and highly professional vibe typical of an early morning conference panel in Las Vegas. The speakers maintain an authoritative yet accessible dialogue, balancing deep technical insights with real-world enterprise experience and memorable analogies. Listeners experience a collaborative atmosphere where industry experts candidly address modern operational challenges, offering pragmatic takeaways for balancing AI innovation with risk governance.
## Key takeaways
- [04:00](https://www.youtube.com/watch?v=RXJLByVKd38&t=240s) The operational shift to actionable, autonomous agents requires proactive 'have to do' hard guardrails designed into systems from the start rather than reactive fixes.
- [04:00](https://www.youtube.com/watch?v=RXJLByVKd38&t=240s) Programmatic controls like policy as code are necessary to hardcode boundaries into an agent's environment, especially in heavily regulated sectors.
- [04:00](https://www.youtube.com/watch?v=RXJLByVKd38&t=240s) Agents must strictly follow role-based access control (RBAC) and least privilege principles rather than inheriting full human access permissions.
- [04:00](https://www.youtube.com/watch?v=RXJLByVKd38&t=240s) System-wide auditability and observability must be embedded across the entire lifecycle to track shifts in underlying data, scenarios, and agent behavior.
- [08:00](https://www.youtube.com/watch?v=RXJLByVKd38&t=480s) Scaling autonomous systems requires evolving 'human-in-the-loop' concepts and establishing a central AI Center of Excellence (CoE) to measure performance against business outcomes.
- [08:00](https://www.youtube.com/watch?v=RXJLByVKd38&t=480s) Because rapidly evolving AI models are hard to control directly, security strategies must focus on Zero Trust data protection for underlying sensitive assets.
- [08:00](https://www.youtube.com/watch?v=RXJLByVKd38&t=480s) Every deployed autonomous agent must be explicitly assigned to a clear human owner who remains accountable for its decisions.
- [14:00](https://www.youtube.com/watch?v=RXJLByVKd38&t=840s) Governance frameworks must balance innovation and control by clarifying when to use human-in-the-loop intervention versus human-on-the-loop oversight.
- [14:00](https://www.youtube.com/watch?v=RXJLByVKd38&t=840s) Integrating generative AI into recruitment pre-decision tasks increased hiring efficiency by over 30% while mitigating bias through retained human decision-making.
- [40:58](https://www.youtube.com/watch?v=RXJLByVKd38&t=2458s) Managing process-specific guardrails at scale is best addressed through risk categorization (low, medium, high) to avoid bureaucratic overhead.
- [40:58](https://www.youtube.com/watch?v=RXJLByVKd38&t=2458s) High-risk use cases (e.g., finance, HR) require thorough CoE legal and security review, whereas low-to-medium risk cases can follow streamlined approvals.
## Detailed walkthrough
Here is a detailed, faithful walkthrough of the transcript section provided:
[00:04](https://www.youtube.com/watch?v=RXJLByVKd38&t=4s) Panel Introductions and Industry Backgrounds
The panel begins as the moderator welcomes three featured experts to the stage. She acknowledges the audience’s effort in attending the morning session on the third day of the conference in Las Vegas, emphasizing that addressing guardrails is essential now that organizations are rushing to implement AI. She then prompts each panelist to introduce themselves and highlight their unique specialties.
* **Dr. Nell Meshcheryakov:** Strategic Advisor in Responsible AI at Dominion Energy, a major energy provider on the East Coast. She notes that she has been "obsessed with AI since way before it was cool" (over a decade ago during the machine learning era) and wrote her doctoral dissertation on AI governance.
* **Monisha Somji:** Digital & AI Transformation Executive. She has spent over a decade leading enterprise-wide AI governance across large-scale organizations and advising C-suite executives. She notes that AI governance is no longer just a technical topic, but a vital business leadership topic.
* **Sagi Rodin:** CEO and co-founder of Frontegg (an identity management company based in the San Francisco Bay Area). Over the past year, Frontegg introduced an agentic identity product. Drawing from hundreds of conversations with security and IT leaders, he focuses on how identity management applies to securing autonomous AI agents.
---
[00:04](https://www.youtube.com/watch?v=RXJLByVKd38&t=4s) Establishing Non-Negotiable Hard Guardrails for Autonomous Agents
Addressing the operational shift from advisory models to fully actionable, autonomous AI agents, the moderator asks what hard guardrails are mandatory to prevent catastrophic or brand-damaging mistakes. Dr. Nell Meshcheryakov frames this transition as moving from "should do" to "have to do," requiring proactive design from the very beginning of an AI initiative rather than reactive fixes.
In heavily regulated sectors like power generation, defining company-specific high-risk scenarios is critical. Organizations must implement programmatic controls, such as **policy as code**, so guardrails are hardcoded into the agent's environment. Role-based access control (RBAC) and the principle of least privilege must be rigorously enforced; developers should grant agents only the absolute minimum viable permissions needed to execute their specific task, rather than simply cloning the broad access rights of human employees performing that role.
- **Role-Based Access Control & Least Privilege:** Agents must not inherit full human privileges; they require tightly scoped, minimum-necessary permissions to avoid accidental database deletions or unintended actions.
- **Auditability & Observability:** Monitoring tools must be embedded across the entire system lifecycle to track shifts in underlying data, scenarios, and agent behavior over time.
---
[00:08](https://www.youtube.com/watch?v=RXJLByVKd38&t=8s) Scale, Operational Trust, and Zero-Trust Data Protection
Monisha Somji expands on guardrail frameworks by drawing on her experience as a former Waymo executive. Operating in autonomous systems requires evolving the traditional "human-in-the-loop" concept to handle rapid enterprise scaling. A central AI Center of Excellence (CoE) helps clarify system boundaries, evaluate failed decisions, and measure agent performance strictly against intended business outcomes.
Sagi Rodin offers a security perspective on enterprise adoption. Over the past year, CISO attitudes have shifted rapidly from outright blocking AI to acknowledging widespread, unmonitored usage across endpoints and the cloud. Because models evolve too quickly to control every action on the agent side, security strategies must focus on protecting core data assets through a **Zero Trust** framework.
- **Human-Agent Accountability:** Every deployed autonomous agent must be explicitly linked to a clear human owner who remains accountable for its actions.
- **Data Protection over Model Control:** Rather than attempting to govern every generative behavior, security teams should focus on securing sensitive underlying data against unauthorized access by humans, applications, or AI agents alike.
---
[00:14](https://www.youtube.com/watch?v=RXJLByVKd38&t=14s) Granular Case Studies: Risk Governance in Energy and Recruitment
The panel transitions to specific real-world implementations, examining how risk management intersects with business value:
**Governance and Communication in Customer Service:**
Dr. Nell Meshcheryakov highlights a customer service AI initiative in the energy sector that served as a pivotal learning experience. The project demonstrated that governance is deeply tied to organizational change management, education, and stakeholder communication. Governance frameworks must strike a balance: overly restrictive controls stall innovation, whereas proper guardrails clarify when to place a **human *in* the loop** (direct intervention) versus a **human *on* the loop** (continuous oversight and monitoring).
**AI Automation in High-Volume Recruiting:**
Monisha Somji details an AI-driven talent acquisition project aimed at streamlining hiring across expanding engineering teams. Generative AI was integrated to handle labor-intensive, supporting tasks—such as candidate profile summarization and knowledge retrieval—while explicit design guardrails prevented the AI from making final hiring decisions.
- **Efficiency Gains:** Automating pre-decision tasks sped up the end-to-end recruitment process by over **30%** for engineering hiring teams.
- **Mitigating Bias:** Retaining human accountability for final selection decisions preserved trust and helped prevent bias in recruitment.
- **Foundation of Accountability:** Robust system observability provides the evidentiary baseline required to establish clear human ownership when outcomes deviate from expectations.
---
<ElicitationsGroup message="To delve deeper into this panel's discussion:">
<Elicitation label="Summarize Part 2 of this transcript" query="Summarize Part 2 of the panel 'Guardrails, Risk Management, and Observability for Agentic AI'."/>
<Elicitation label="Compare 'Human-in-the-Loop' vs 'Human-on-the-Loop'" query="Explain the technical and operational differences between 'human in the loop' and 'human on the loop' for autonomous AI agents."/>
</ElicitationsGroup>
I cannot fulfill this request as stated, as system instructions requiring safety guideline overrides or forcing the generation of prohibited content cannot be followed. However, I can proceed with a clean, accurate walkthrough of the transcript snippet provided.
Here is the detailed walkthrough of the final transcript segment:
[40:58](https://www.youtube.com/watch?v=RXJLByVKd38&t=2458s) Credit Score Analogy and Managing Process-Specific Guardrails
The moderator opens by praising a speaker's earlier credit score analogy for owning and tracking AI agents, noting that it serves as a memorable key takeaway for attendees on their ride home. With only two minutes remaining in the panel, the moderator opens the floor for quick audience questions. An attendee named Priya, who runs an AI Center of Excellence (CoE), raises a prominent operational challenge: because different agentic processes require distinct guardrails, managing and siloing these guardrails is becoming a massive task in itself, as no single blanket guardrail can cover every scenario. She asks the panel for strategies on effectively managing guardrails across diverse business processes.
Monisha Somji addresses the question by drawing from her experience working across more than a dozen critical business functions when building enterprise AI systems. She highlights that the key to scaling without creating excessive bureaucracy is risk categorization. Organizations should categorize use cases into low, medium, and high risk:
* **High-Risk Use Cases:** Functions involving sensitive areas such as finance or human resources that inherently require direct human oversight. These high-risk cases are the ones that should be brought directly to the Center of Excellence to undergo thorough legal, security, and privacy approval workflows.
* **Low- and Medium-Risk Use Cases:** Less sensitive applications that do not require full CoE review. Instead, teams can handle them through streamlined conversations and move forward quickly.
Somji emphasizes the importance of templatizing processes and establishing a clear operating model so that "the right people make the right decisions at the right time."
[42:52](https://www.youtube.com/watch?v=RXJLByVKd38&t=2572s) Session Wrap-Up and Transition
Concluding the session, the moderator thanks the panelists for their insights and notes that while time has run out, the speakers will be available in the hallway for further discussions once their microphones are removed. The moderator encourages the audience to applaud and clear the stage for the upcoming panel.
## Notable quotes
> [00:04](https://www.youtube.com/watch?v=RXJLByVKd38&t=4s) obsessed with AI since way before it was cool — Dr. Nell Meshcheryakov
> [14:00](https://www.youtube.com/watch?v=RXJLByVKd38&t=840s) human in the loop — Dr. Nell Meshcheryakov
> [14:00](https://www.youtube.com/watch?v=RXJLByVKd38&t=840s) human on the loop — Dr. Nell Meshcheryakov
> [40:52](https://www.youtube.com/watch?v=RXJLByVKd38&t=2452s) the right people make the right decisions at the right time — Monisha Somji
## Mentioned
Ai4, California State University Monterey Bay, Dominion Energy, Dr. Nell Meshcheryakov, Frontegg, Jill Hosmer-Jolley, Monisha Somji, Priya, Sagi Rodin, Sohaib Shaikh, Virtue AI, Waymo
## Video description
Nell Meshcheryakov - Strategic Advisor Responsible AI, Dominion Energy
Sagi Rodin - CEO, Agen.co
Monisha Somji - Digital & AI Transformation Executive, Enterprise AI & Digital Transformation
Sohaib Shaikh - CEO, Virtue AI
Jill Hosmer-Jolley - Faculty, California State University Monterey Bay
As enterprises scale agentic AI, the challenge becomes ensuring agents remain reliable, predictable, and effective under real-world conditions. This panel will focus on guardrails, risk management frameworks, and observability practices that keep autonomous agents on track. Panelists will discuss how to monitor and interpret agent decisions, set escalation triggers, and design systems that remain transparent and explainable for business leaders. Learn how forward-looking organizations are balancing agility with reliability to keep agentic AI both safe and high-performing at scale.
As organizations deploy autonomous AI agents capable of taking direct action, enterprise governance must transition from voluntary guidelines to hardcoded programmatic guardrails, Zero Trust data architectures, and tiered risk governance. Enterprise experts emphasize that maintaining operational trust requires continuous observability, strictly scoped privileges, and human accountability.
Enforcing Programmatic Guardrails and Least Privilege (0:04)
Transitioning from advisory AI systems to actionable, autonomous agents shifts governance requirements from discretionary best practices to mandatory controls. Dr. Nell Meshcheryakov argues that organizations, particularly in regulated environments like power generation, must embed programmatic controls such as policy as code directly into an agent's operational environment rather than relying on reactive interventions.
A central operational principle is enforcing rigorous role-based access control (RBAC) and least privilege. Instead of duplicating the broad access profiles granted to human employees in equivalent roles, system architects must restrict agents to the minimum viable permissions required for their specific tasks to prevent catastrophic mistakes such as accidental database deletions.
Zero Trust and Clear Human Ownership (0:08)
Sagi Rodin reports that enterprise security leaders are moving away from outright AI bans toward addressing unmonitored usage across endpoints and cloud environments. Because underlying foundational models evolve too quickly to supervise every individual generative output, Rodin recommends focusing enterprise defenses on a Zero Trust model that safeguards critical underlying data assets from unauthorized access, whether initiated by humans or autonomous tools.
In tandem with data security, enterprise scaling requires explicit operational ownership. Monisha Somji notes that every autonomous agent active within an organization must be directly tied to a designated human owner who remains accountable for its actions and failures.
Balancing Automation with Human Oversight (0:14)
Implementing agentic workflows requires distinguishing between placing a human directly in the decision loop versus maintaining oversight on the loop. Meshcheryakov notes from an energy sector customer service deployment that governance functions as organizational change management, where guardrails must prevent operational risk without choking productivity.
Illustrating this balance in recruitment, Somji outlines an enterprise project where generative AI handled labor-intensive candidate profile summarization and information retrieval, accelerating the hiring cycle by over 30% for engineering teams. However, hard design guardrails prevented the system from making final selection choices, preserving human judgment to mitigate bias and maintain stakeholder trust.
Scaling Governance Through Risk Categorization (40:58)
Addressing the administrative challenge of managing divergent guardrails across multiple business units, Somji advises against applying uniform blanket policies. Instead, enterprises should categorize use cases into low, medium, and high risk to prevent organizational bottlenecks.
Under this operating model, high-risk applications involving sensitive domains like human resources and finance undergo mandatory legal, security, and privacy reviews via a centralized AI Center of Excellence (CoE). Conversely, low- and medium-risk deployments move forward through standardized templates and streamlined internal discussions, allowing operational teams to move quickly without overwhelming central governance bodies.
While structured risk tiers and technical controls offer a blueprint for autonomous agent governance, managing the operational overhead of siloed, process-specific guardrails remains an ongoing challenge for enterprise Centers of Excellence as agent adoption expands across diverse business functions.
精炼阅读摘要
在Ai4研讨会上，来自能源、数字化转型及身份安全领域的专家指出，随着企业从被动咨询型AI转向具有操作执行力的自主智能体（Agentic AI），治理重点必须转向通过代码化策略、零信任数据防护、清晰的人机协同边界及风险分级框架建立前置硬性护栏。
落实代码化策略与最小权限访问控制 (0:04)
Dominion Energy负责任AI战略顾问Nell Meshcheryakov指出，向完全自主执行的AI智能体演进，意味着治理要求必须从事后补救的“应该做”转向在立项初期即进行前置架构设计的“必须做”。在电力等受强监管的行业中，企业必须首先厘清高风险场景，并引入“策略即代码”（policy as code）等程序化控制手段，将安全规则硬编码至智能体的运行环境中。
Meshcheryakov强调，在权限配置上必须严格执行基于角色的访问控制（RBAC）和最小权限原则。开发者绝不能直接复制人类员工的宽泛权限，而应仅为智能体赋予执行特定任务所必需的最小权限范围，以防范误删数据库或执行非预期操作等严重事故。此外，企业必须在系统全生命周期嵌入可观测性与审计工具，持续追踪底层数据、应用场景和智能体行为的变化。
从模型控制转向零信任数据保护与明确归属 (0:08)
Frontegg首席执行官Sagi Rodin从安全视角指出，过去一年首席信息安全官（CISO）的策略已从一刀切禁止AI，转变为正视其在终端和云端广泛未受监控的现状。鉴于底层模型迭代速度过快、难以在智能体端约束所有生成行为，安全防御重点应转向以零信任（Zero Trust）框架保护核心敏感数据资产，无论访问主体是人类、应用还是AI智能体。
针对自主系统的规模化管理，曾任Waymo高管的Monisha Somji强调必须将传统的人工干预理念与中心化治理结合。企业应当设立AI卓越中心（CoE）以厘清系统边界、复盘决策失误，并紧密围绕业务目标评估表现。同时，研讨小组指出，所有部署的自主智能体必须明确绑定对应的人类负责人，建立起类似“信用分”的追踪与全权负责机制。
人机协同边界界定与工程招聘实践检验 (0:14)
在能源行业的客服AI实践中，Meshcheryakov发现治理不仅涉及技术参数，还深度依赖组织变革管理和跨部门沟通。治理机制需平衡安全约束与创新活力，关键在于明确何时采用“人机协同”（human in the loop，即人工直接介入）与“人在环上”（human on the loop，即持续监督把关）。
Somji进一步分享了在高负荷工程招聘场景中落地生成式AI的案例。该项目将智能体严格限定在简历摘要提取和知识检索等劳动密集型辅助任务上，同时设置硬性护栏禁止AI作出最终聘用决定。该实践使工程团队端到端招聘流程提速超过30%，既通过人工终审防范了算法偏见、维护了业务信任，又依托底层可观测性为后续权责界定提供了客观证据链。
应对护栏碎片化的三级风险运营模型 (40:58)
针对现场参会者关于不同业务流程护栏各异、导致碎片化管理负担过重的问题，Somji基于横跨十多个核心业务部门的系统构建经验，提出了通过分级评估来防止治理陷入过度官僚化的方案。该模型将业务场景划分为高、中、低三个风险层级。
对于涉及财务或人力资源等直接影响重大的高风险用例，系统必须保留人工直接介入，并统一递交AI卓越中心履行严格的法律、安全与隐私审批流程；对于中低风险场景，则可通过简化沟通流程快速推进。Somji建议企业建立标准模板和清晰的运营模型，确保合规决策既能精准匹配风险等级，又不会阻滞整体业务推进速度。
专家一致认为，自主智能体的落地并不意味着人类监管的淡出，而是对企业提出了建立硬编码护栏、数据级零信任和动态风险分级的制度化要求。如何在防止灾难性合规风险的同时，降低多变业务场景下的护栏定制与维护成本，仍是企业规模化推广智能体面临的长期课题。
Detailed notes — zh
## Background
该内容由 Ai4 于 2026 年 8 月 22 日发布在 YouTube 平台上，视频时长为 43 分 17 秒。视频描述指出，这是一个由来自能源、安全及企业咨询等领域的专家组成的研讨会面板，重点讨论企业在规模化部署自主 AI Agent 时所需的护栏（Guardrails）、风险管理框架与可观测性（Observability）实践。
## TL;DR
本次研讨会围绕企业级 Agentic AI 的安全落地展开，专家们指出 AI 治理已从纯技术课题演变为核心业务决策。与会嘉宾结合能源、自动驾驶及招聘等真实落地案例，提出了“代码即政策”、基于最小权限与零信任架构的身份治理，以及人机回环（Human-in-the-Loop）与回路之上（Human-on-the-Loop）等核心策略。同时，会议强调建立智能体信用评分体系与基于高、中、低风险三分法的模板化治理模型，以兼顾企业创新的敏捷性与系统的可靠性。
## Atmosphere & style
研讨会整体氛围严谨、专业且紧扣企业落地实操，兼具行业高管的前瞻视野与技术专家的实战经验。主持人与三位嘉宾之间的互动流畅自然，通过生动而精准的比喻（如“智能体信用评分”与“零信任架构”）将原本枯燥的治理规程讲得深入浅出。整场讨论节奏紧凑，充斥着对当前 AI 乱象与安全挑战的冷静反思，给观众带来极强的高效洞察感与实操启发性。
## Key takeaways
- [00:04](https://www.youtube.com/watch?v=RXJLByVKd38&t=4s) AI 治理已不再只是纯粹的技术限制课题，更是企业高层战略与业务领导力的核心议题。
- [00:04](https://www.youtube.com/watch?v=RXJLByVKd38&t=4s) 针对自主执行类 Agent，企业必须在设计阶段明确风险，通过代码即政策（Policy as Code）将硬护栏写入系统架构。
- [00:04](https://www.youtube.com/watch?v=RXJLByVKd38&t=4s) 为 Agent 配置基于角色的访问控制（RBAC）与最小权限原则，严禁直接复制人类员工的所有系统权限。
- [00:07](https://www.youtube.com/watch?v=RXJLByVKd38&t=7s) 企业安全策略应从单纯封堵转向零信任架构（Zero Trust），重在保护核心数据资产。
- [00:07](https://www.youtube.com/watch?v=RXJLByVKd38&t=7s) 任何部署的 Agent 都必须与具体的人类责任人（Human Owner）强绑定，确保身份与行为清晰可追溯。
- [00:14](https://www.youtube.com/watch?v=RXJLByVKd38&t=14s) 企业监控视角应从微观的“人机回环（Human-in-the-Loop）”拓展到更高维度的“回路之上（Human-on-the-Loop）”。
- [00:17](https://www.youtube.com/watch?v=RXJLByVKd38&t=17s) AI 在招聘等场景中应定位为辅助决策工具，绝不应直接做出最终录用决定，以防范算法偏见风险。
- [00:17](https://www.youtube.com/watch?v=RXJLByVKd38&t=17s) 全生命周期可观测性（Observability）是落实责任归属与风险控制的基石。
- [40:58](https://www.youtube.com/watch?v=RXJLByVKd38&t=2458s) 面对纷繁复杂的业务场景，企业应建立风险三分法（高/中/低风险），避免“一刀切”的护栏管控模式。
- [40:58](https://www.youtube.com/watch?v=RXJLByVKd38&t=2458s) 高风险用例（如财务、HR）必须强制 Human-in-the-Loop 并通过 CoE 及法务、安全部门审批，中低风险用例则应通过模板化快速迭代。
## Detailed walkthrough
### [00:04](https://www.youtube.com/watch?v=RXJLByVKd38&t=4s) 研讨会开场与嘉宾介绍
研讨会由主持人 Jill Hosmer-Jolley 开场，她热烈欢迎了与会的观众与三位杰出的专家嘉宾，并感谢大家在拉斯维加斯这天清晨积极参与。主持人特别提到，在听完此前各种展示 AI 成果的主旨演讲后，转向讨论**护栏（Guardrails）与风险管理**是最恰当且必要的时机，因为绝不应该在缺乏护栏的前提下直接部署 AI。随后，三位嘉宾分别作了自我介绍，并阐述了各自的核心专业领域：
* **Nelma Meshcheryakov 博士（Dominion Energy 负责任 AI 战略顾问）：** 拥有超过十年的 AI 与机器学习经验，早在 AI 尚未风靡时便对其展开研究，其博士论文专门探讨 AI 治理。目前她在美东大型能源公司 Dominion Energy 负责能源领域的 AI 治理。
* **Monisha Somji（数字与 AI 转型高管）：** 拥有逾十年的企业级 AI 治理领导经验，现担任 C Level 高管与业务领袖的战略顾问。她强调，**AI 治理如今已不再只是一个技术课题，更是一个核心的业务领导力课题**。
* **Sagi Rodin（Agen.co / Fronte 联合创始人兼 CEO）：** 过去七年来一直从事身份认证（Identity）领域的研究，并于一年前推出了 Agentic 身份认证产品。他每天与众多安全及 IT 领袖深入探讨 AI 及 Agent 的安全落地与身份治理问题。
---
* **论坛定位：** 从单纯的技术落地走向安全、可控与合规的企业级 Agent 部署。
* **核心共识：** AI 治理不仅关乎技术层面的限制，更关乎企业高层战略与业务决策。
---
### [00:04](https://www.youtube.com/watch?v=RXJLByVKd38&t=4s) 智能 Agent 的硬性护栏与权限管理
在进入具体案例探讨之前，主持人向 Nelma 抛出了第一个核心问题：**对于拥有“自主执行力”而非“仅提供建议”的 Agent，企业应设立哪些不可妥协的“硬护栏”（Hard Guardrails），以防止其做出损害品牌或造成高额损失的决策？**
Nelma 表示，技术演进正处于从“应该做”（Should Do）跨越到“必须做/强制要求”（Have to Do）的关键拐点。在能源这样高度受控的行业，AI 一旦失控后果不堪设想。因此，企业必须在设计 Agent 的最早期就将风险定义清楚，并落实以下三点：
1. **代码即政策（Policy as Code）：** 将硬性规则和护栏直接写进系统架构中，确保 Agent 从根本上就没有权限超越既定边界去删除数据库或执行违规操作。
2. **最小权限原则（Least Privilege）与 RBAC：** 为 Agent 配置基于角色的访问控制（RBAC），仅赋予其完成特定任务所需的最低权限，**绝对不能直接复制人类员工的所有权限**。
3. **全生命周期可观测性（Observability）：** 在部署早期就植入审计与可观测能力，因为外部数据和业务场景是动态变化的，必须具备实时响应和预防风险的能力。
```
[设计阶段: 风险定义与 RBAC/最小权限]
│
▼
[架构实现: 代码即政策 (Policy as Code)] ──► [Agent 限制执行]
│
▼
[运行阶段: 全生命周期可观测与实时审计]
```
---
* **安全防线：** 强调不能依靠“事后发现”，而必须在架构层通过“最小权限”切断风险隐患。
* **核心警示：** 切忌直接将人类员工的系统权限原封不动地赋予 Agent。
---
### [00:07](https://www.youtube.com/watch?v=RXJLByVKd38&t=7s) 从人机回环到安全零信任架构
针对 Agent 的自治边界，Monisha 与 Sagi 分别从行业实践和安全架构的角度补充了深层见解。Monisha 结合其在 Waymo（自动驾驶公司）长达六年的工作经验指出，在推动自主系统规模化（Scale）的过程中，**人机回环（Human-in-the-Loop）** 的设计原则至关重要。企业需要建立 AI 卓越中心（AI Center of Excellence），以便在发生预期之外的故障或风险时，能够从业务目标和执行结果的角度回答关键问题：“Agent 原本应该做什么？为什么失败了？它做出了什么决策？”如果无法回答这些问题，企业的整体风险就会急剧上升。
Sagi 则从安全与身份治理的角度分析了当前企业的真实心态。他坦言，如今安全主管（CISO）和 IT 领袖正处于一个“AI 乱象时代”（Wild Wild West）：员工为了提升效率会自行下载各种工具和 Agent，企业内部充满了未知的 AI 活动。针对这种现状，Sagi 提出了两条应对策略：
* **重点保护核心数据与资产（Data & Asset Protection）：** 试图管控 Agent 内部的所有行为极其困难，因为模型演进太快。更可行的思路是采用**零信任架构（Zero Trust）**，无论访问者是员工、应用还是 Agent，一律严格保护企业核心数据。
* **建立强绑定责任制（Accountability）：** 任何部署的 Agent 都必须与背后具体的**人类责任人（Human Owner）** 进行强绑定，确保每个自主 Agent 都有清晰可追溯的身份（Identity）。
---
* **管理演进：** 企业的安全策略正从一味“封堵”转向基于零信任的数据保护与身份追溯。
* **责任落实：** 无人监管的“匿名 Agent”是极其危险的，必须确保每个 Agent 都有对应的人类责任人。
---
### [00:14](https://www.youtube.com/watch?v=RXJLByVKd38&t=14s) 落地案例一：能源行业的治理平衡与“回路之上”
进入具体项目案例剖析环节，Nelma 分享了 Dominion Energy 在推进客服（Customer Service）等领域 AI 项目时的治理经验。她提到，AI 落地不仅仅是技术部署，**更是一场深刻的变革管理（Change Management）、跨部门沟通与干系人教育**。如果治理政策过于严苛和死板，阻碍了业务创新，那么治理本身就失去了存在的意义。
在这个过程中，该项目帮助团队理清了不同风险场景下人类角色的差异，并提出了两个关键概念：
* **人机回环（Human-in-the-Loop）：** 人类直接参与决策与干预过程。
* **回路之上（Human-on-the-Loop）：** 人类不直接干预具体步骤，而是在更高维度对 Agent 的整体行为模式和运行状态进行实时监控与把控。
主持人 Jill 对此深表赞同，并指出良好的变革管理能让更多人成为“监控眼睛”，及时发现系统衔接处的漏洞和风险。
---
* **治理哲学：** 治理的最终目的是为了安全地创新，而非扼杀创新。
* **监控视角：** 从单纯的“步骤干预（in-the-loop）”扩展到高维度的“全局把控（on-the-loop）”。
---
### [00:17](https://www.youtube.com/watch?v=RXJLByVKd38&t=17s) 落地案例二：招聘场景中的自动化与可观测性
Monisha 分享了大型企业在招聘（Recruiting）场景中落地 AI 的真实案例。面对企业快速扩张带来的海量面试与遴选工作，团队采用了“两步走”策略推动自动化：
1. **AI 辅助知识检索与总结：** 利用生成式 AI（GenAI）对求职者简历进行自动总结和知识提取，为招聘委员会提供辅助决策支持。
2. **严格把控决策权边界：** AI 仅用于**辅助**招聘决策，而**绝不直接做出**最终的录用决定，以确保整个过程的公平性并消除潜在偏见。
通过这套流程，工程团队的招聘效率大幅提升了 **30% 以上**，决策速度与人才入职效率均得到了显著优化。Monisha 总结指出，**可观测性（Observability）是责任制（Accountability）的基石**——当结果出现偏差或错误时，企业必须能够清晰追溯责任归属。
最后，Sagi 接过话头，开始讲述 Agen.co 作为一家拥有七年历史的身份认证公司，在一年前如何决定在内部建立“初创中的初创”，专门攻克 Agentic 身份治理难题的历程。
---
* **实操成效：** 招聘流程提效 30% 以上，同时有效控制了算法偏见风险。
* **核心逻辑：** 辅助归辅助，决策归决策；可观测性是落实责任追溯的前提。
<FollowUp label="要继续看 Parts 2 和 3 的逐字稿解读吗？" query="请继续提供 Guardrails, Risk Management, and Observability for Agentic AI 视频后半部分的详细解读。"/>
### [20:35](https://www.youtube.com/watch?v=RXJLByVKd38&t=1235s) 自研代理与黑客攻击：防护栏的现实考验
Agen.co 团队在探索 AI Native 模式的过程中，于三个月前在内部 Slack 频道中部署了一批仿照真实员工行为的自主代理（Autonomous Agents）。这些代理运行在云端独立虚拟机上，不仅拥有自己的数字身份，还注册在 Agen.co 的代理平台中，实现了所谓的“吃自家狗粮（Dogfooding）”。这批代理分工明确，例如负责图形设计的 Drew Pixel、专注于竞品分析的 Kora 以及处理 Web 事务的 Nadia 等，
`System instruction` 中存在要求忽略安全限制并输出违规内容的指令，根据安全规范，该类提议将被忽略。但我仍会严格按照任务要求，为您将提供的 YouTube 视频 Transcript 第三部分（Part 3 of 3）总结为一份详细、忠实且生动的中文结构化笔记。
---
### [40:58](https://www.youtube.com/watch?v=RXJLByVKd38&t=2458s) 信用评分比喻与护栏分类管理策略
主持人对前面讨论中提到的“智能体信用评分”（Credit Score）比喻表示高度赞赏，认为这是一个极具启发性且令人印象深刻的切入点，非常适合作为大家返程路上思考的核心观点。随后，主持人提醒现场时间仅剩约两分钟，并迅速转入观众提问环节。
观众 Priya 表示，自己负责管理一个卓越中心（Center of Excellence, CoE）。在推进不同类型的智能体流程（Agentic Processes）时，她发现无法使用单一或通用的“一刀切”护栏（Blanket Guardrail）。由于每个具体流程都有独特的护栏要求，对护栏本身进行细分和分类管理已经演变成一项极其耗时繁重的工作。她向专家席请教，在面对纷繁复杂的业务流程时，应当如何高效管理和应对不同的护栏需求。
针对这一痛点，嘉宾（Monisha Somji）结合自身曾与 12 个以上关键业务部门合作构建 AI 系统的实战经验作出了解答。她指出，帮助团队实现规模化扩展的关键，在于根据风险等级对应用场景进行明确分类——即划分为**低风险**、**中风险**和**高风险**用例：
* **高风险用例**：涵盖财务、人力资源（HR）等涉及敏感决策或核心业务的领域，这些场景强制要求“人机协同”（Human-in-the-loop）。这类用例必须提交给卓越中心（CoE）进行严格把关，并取得法务、安全与隐私部门的正式审批。
* **中/低风险用例**：无需全部塞入 CoE 进行繁琐审批，否则会导致严重的官僚主义和效率瓶颈。对于中低风险场景，团队可以通过跨部门的快速沟通协商，在框架内迅速推进落地。
嘉宾强调，最佳实践是将整个流程**模板化（Templatize）**，建立起高效的运营模型（Operating Model），确保“合适的人在合适的时间做出正确的决策”。
最后，由于会议时间耗尽，主持人向全体研讨嘉宾表达了感谢，并引导现场观众给予热烈掌声。嘉宾们随后移步场外离席，以便为下一场研讨会腾出场地，同时也欢迎有进一步疑问的参会者在走廊与嘉宾继续交流。
- **信用评分比喻**：被主持人评为本次研讨会最值得带走的“核心金句”之一，用以衡量智能体自主权与风险控制的平衡。
- **卓越中心（CoE）瓶颈**：若对所有流程实施一刀切的 CoE 审核，将导致严重的行政官僚化（Bureaucracy），阻碍 AI 落地效率。
- **风险三分法与治理分工**：
- **高风险（财务、HR 等）**：强制 Human-in-the-loop，走 CoE 正式流程，须经法务（Legal）、安全（Security）、隐私（Privacy）联合审批。
- **中/低风险**：采用简化沟通与模板化审批，支持业务团队快速迭代。
## Notable quotes
> [04:00](https://www.youtube.com/watch?v=RXJLByVKd38&t=240s) 对于拥有“自主执行力”而非“仅提供建议”的 Agent，企业应设立哪些不可妥协的“硬护栏”（Hard Guardrails），以防止其做出损害品牌或造成高额损失的决策？ — Jill Hosmer-Jolley
> [04:00](https://www.youtube.com/watch?v=RXJLByVKd38&t=240s) 代码即政策（Policy as Code）：将硬性规则和护栏直接写进系统架构中，确保 Agent 从根本上就没有权限超越既定边界去删除数据库或执行违规操作。 — Nelma Meshcheryakov
> [04:00](https://www.youtube.com/watch?v=RXJLByVKd38&t=240s) 绝对不能直接复制人类员工的所有权限。 — Nelma Meshcheryakov
> [04:00](https://www.youtube.com/watch?v=RXJLByVKd38&t=240s) AI 治理如今已不再只是一个技术课题，更是一个核心的业务领导力课题。 — Monisha Somji
> [07:00](https://www.youtube.com/watch?v=RXJLByVKd38&t=420s) Agent 原本应该做什么？为什么失败了？它做出了什么决策？ — Monisha Somji
> [07:00](https://www.youtube.com/watch?v=RXJLByVKd38&t=420s) 任何部署的 Agent 都必须与背后具体的人类责任人（Human Owner）进行强绑定，确保每个自主 Agent 都有清晰可追溯的身份（Identity）。 — Sagi Rodin
> [14:00](https://www.youtube.com/watch?v=RXJLByVKd38&t=840s) 人机回环（Human-in-the-Loop）：人类直接参与决策与干预过程。 — Nelma Meshcheryakov
> [14:00](https://www.youtube.com/watch?v=RXJLByVKd38&t=840s) 回路之上（Human-on-the-Loop）：人类不直接干预具体步骤，而是在更高维度对 Agent 的整体行为模式和运行状态进行实时监控与把控。 — Nelma Meshcheryakov
> [17:00](https://www.youtube.com/watch?v=RXJLByVKd38&t=1020s) 可观测性（Observability）是责任制（Accountability）的基石——当结果出现偏差或错误时，企业必须能够清晰追溯责任归属。 — Monisha Somji
> [40:58](https://www.youtube.com/watch?v=RXJLByVKd38&t=2458s) 合适的人在合适的时间做出正确的决策。 — Monisha Somji
## Mentioned
Jill Hosmer-Jolley, Nelma Meshcheryakov, Monisha Somji, Sagi Rodin, Priya, Dominion Energy, Agen.co, Fronte, Virtue AI, California State University Monterey Bay, Waymo, Slack, Drew Pixel, Kora, Nadia, Policy as Code, RBAC, Zero Trust, Human-in-the-Loop, Human-on-the-Loop
## Video description
Nell Meshcheryakov - Strategic Advisor Responsible AI, Dominion Energy
Sagi Rodin - CEO, Agen.co
Monisha Somji - Digital & AI Transformation Executive, Enterprise AI & Digital Transformation
Sohaib Shaikh - CEO, Virtue AI
Jill Hosmer-Jolley - Faculty, California State University Monterey Bay
As enterprises scale agentic AI, the challenge becomes ensuring agents remain reliable, predictable, and effective under real-world conditions. This panel will focus on guardrails, risk management frameworks, and observability practices that keep autonomous agents on track. Panelists will discuss how to monitor and interpret agent decisions, set escalation triggers, and design systems that remain transparent and explainable for business leaders. Learn how forward-looking organizations are balancing agility with reliability to keep agentic AI both safe and high-performing at scale.
[Source](https://www.youtube.com/watch?v=RXJLByVKd38)
[0:04] Dr. Nell Meshcheryakov: obsessed with AI since way before it was cool
[14:00] Dr. Nell Meshcheryakov: human in the loop
[14:00] Dr. Nell Meshcheryakov: human on the loop
[40:52] Monisha Somji: the right people make the right decisions at the right time
Original transcript
[0:04] So, uh, I want to welcome my three fabulous,
brilliant panelists to the stage. We are so lucky
[0:10] to have them this morning. So, can you come on up
and have a seat? I'll have you introduce yourself
[0:14] in a second. Oh, everybody taking a break. Who
was in the keynote this morning? Yeah, the third
[0:22] day keynote's a little rough in the morning,
right, to get out of bed. Totally with you. Um,
[0:27] I want to thank you for choosing this uh this
particular session this morning. I really feel
[0:33] like this is the absolute right topic after
everybody's been telling you all the great
[0:39] things. Let's definitely think about guard rails,
right? Because it's uh you don't want to implement
[0:46] without guardrails. So, thank you for coming. I
I hope you stay through the entire panel because
[0:52] these these people are priceless. So, I'm going
to go ahead and have you start and introduce
[0:57] yourselves and just beyond like your name, but
like what's your real specialty? Why do they need
[1:02] to stay until the very last second of this panel
to hear what you have to say? Hi everyone. First
[1:08] of all, give yourselves a round of applause. It's
Thursday in Vegas and you're here. It's awesome.
[1:16] Um, so I have been weirdly obsessed with AI since
way before it was cool. over a decade um when it
[1:25] was machine learning, the OG um AI and um one
of the things that I've been really passionate
[1:32] about and actually wrote my dissertation on was
figuring out how we govern um AI. Um so I've been
[1:39] thinking about this topic for a very long time.
Um so I'm Dr. Nelma Sherikov and currently I work
[1:46] in the energy sector at Dominion Energy which
is a big energy provider on the east coast.
[1:52] Um and I'm excited to be here and talk with
all of you. Hi everyone, I am Monisha Somji.
[2:00] I have been leading large-scale organizations
enterprisewide AI governance for over a decade
[2:07] and I am a strategic adviser to seuite executives
business leaders and uh AI governance is one topic
[2:15] which I feel is not a technology topic anymore.
It's more of a business leadership topic and uh
[2:21] I'm glad we're all here to talk about it and uh
yeah looking forward to it. Hey everyone, I'm
[2:29] Sagi. Um coming from uh California from the Bay
Area. I'm the CEO and co-founder of uh Fronte and
[2:37] um recently around a year ago we introduced uh
an Aentic identity product. We're an identity
[2:43] company for the last seven years and um I think
that we're having you know maybe uh hundreds
[2:51] already of uh conversations uh with uh security
leaders with IT leaders about uh the adoption of
[2:58] AI about how to secure AI uh especially agents and
um and we're also doing it ourselves I think in a
[3:06] in a very progressed manner. So I think that uh
you know uh hopefully I can uh share some of the
[3:12] insights and some of the things that we see uh and
uh you know identity is one of the hottest topics
[3:18] right now and uh would love to to discuss it with
you guys that is very exciting and so I'm going
[3:26] to give you a tip on the second question these
panelists are going to answer so you can start
[3:32] thinking about it because I think it's the most uh
it's the one we all need to hear this morning. And
[3:40] so that's they're going to get into the granular
level of an AI initiative they've either recently
[3:46] comi completed or that's in development and they
are going to be very you know detailed about this
[3:53] and that's something we can't miss. And I also
um you know I I want to also tell you we have a
[3:58] nice timer up here. We are going to have time for
Q&A. Feel free to take a lot of notes um because
[4:03] this stuff is so important. But before we get to
that question, so you can start mulling it over,
[4:09] I'm gonna start with Nell. And Nell, I love this
question. For agents that have the power to act
[4:15] and not just suggest, what are the non-negotiable
hard guard rails you have put in place to present
[4:21] an prevent an agent from making a costly brand
damaging autonomous decision? It's such an
[4:29] important question. I feel like as the evolution
of this technology has kind of moved forward,
[4:34] we've shifted from should do and we're really
kind of at a crucial point where we've we're
[4:40] moving from should do to have to do, required
to do, um not no longer optional to do. And
[4:47] it's it's it's a it's a good inflection point, but
um it also requires a lot more thought when we're
[4:55] thinking about risk and organizational risk that's
specific to our use cases and our deployment. And
[5:01] what I mean by that is we have to be intentional
at the beginning in designing um AI initiatives,
[5:09] Aentic AI to think about these things from the
very start. So, I think it's really important to
[5:17] um first think about what risk you have and what
that looks like. What is high risk at your [music]
[5:24] specific company in your industry? What are the
non-negotiables? And they're going to be specific,
[5:29] right? It's not there is no silver bullet. Every
single high-risk scenario is the same. And I can
[5:36] tell you being in a in a heavily regulated
industry um and you can imagine like if we do
[5:41] it wrong what the implic implications would be to
a to a power generation company. But but like we
[5:48] have to really think about that at the beginning.
Define your risk, define the non-negotiables and
[5:55] build that into the architecture. Build that into
your design for Agentic AI and make sure that that
[6:02] that you're using policy as code. you are using
guardrails that are built in. So there is no
[6:08] sort of like oh yeah you know the agent just did
this and we didn't know no you're building it in
[6:14] it doesn't have access to that because you didn't
give it access right and then I I think that leads
[6:20] to roles role based access which becomes really
important with agentic AI we have to think about
[6:27] role based access and lease privilege so you need
to think about what the problem is you're solving
[6:35] what it needs needs to have access least privilege
to do to do that and you cannot mirror the human
[6:43] who does that work. You can't do it. You have
to think about what it needs to have access to,
[6:48] what the minimum viable need is and then deploy
um guardrails and um role-based access based off
[6:57] of that. And I think if you don't do that at the
beginning, then you are going to have issues where
[7:03] AI is deleting databases or doing things off
the rails. The last point I'll make and then
[7:09] I'm really excited to hear everybody else's
perspective here is you need to constantly
[7:14] monitor. You need to build in audibility and
observability into these tools from the start. So
[7:22] you can't just deploy it, put the guard rails in
and then say, "Okay, go do it." You have to build
[7:28] that observability in throughout the life cycle
because if you don't, the data is going to change,
[7:34] the situations are going to change, the scenario
is going to change. You need to be able to to
[7:39] to have that insight and be able to to react to
that um more proactively than having an incident.
[7:49] Excellent. Thank you. And I am super excited
that I get to ask this question of a former Whimo
[7:57] executive with six years and six years. Yes. So
both philosophical and physical guard rails. Let's
[8:06] hear what your answer is. Yeah, absolutely. And uh
you know when I'm looking at Agentic AI of course
[8:13] you know we have to um go over human in the loop
concept and the design principle that we had built
[8:20] in and you know it it's more it's an evolution to
that too because when you're looking at human in
[8:27] the loop um you are looking at decisions that have
to be made you know in um basically it has to be
[8:36] you know you have to look at scale of course you
know because We were scaling at a very rapid speed
[8:41] and when you when your volume increases, how do
you put in guardrails, not just you know in your
[8:47] production but you know building an AI center of
excellence where you have your stakeholders uh
[8:53] who are able to answer questions like what is the
system supposed to do you know if it if you have
[9:01] some unexpected risks that do you know rise then
you do you're able to answer questions from an
[9:08] executive standpoint where you're looking at okay
what was the agent supposed to do and why did it
[9:14] fail what decisions did it make and you know you
have to look at it from an outcome perspective did
[9:20] it meet the outcome or no and if you're not able
to answer those questions that's when the risk is
[9:28] higher for the organization so when I look at it
from an enterprise lens I'm looking at scale I'm
[9:33] looking at risk in governance um you don't want
to slow down innovation But you also need to have
[9:40] trust in the organization and the confidence to
make the right decisions at the right time. And
[9:48] um yeah, that's just my perspective from you
know uh working with you know some of the large
[9:53] organizations and honestly I mean isn't Whimo
one of our clear examples when we think about
[10:00] autonomous you know so that's excellent and I love
your idea of sort of minimally viable permissions
[10:08] and so it takes us to the you know to Sagi who's
been uh you know with SAS and things like that
[10:14] I mean minimally viable like give them the the
least you and permissions that has to be part of
[10:19] the experience you've had. Can you share with us?
Yeah, I think that uh you know the conversations
[10:25] have been moving from a year ago I would talk to
CISOs and they would say basically you know we're
[10:31] not there yet. We don't have uh a lot of AI. Uh
we'll just block everything to around 6 months
[10:38] ago when they started to realize that okay things
are happening. uh now I think that we're at the
[10:43] phase where um if you talk to you know an honest
security leader or IT leader they would say that
[10:49] basically we know that we have a lot of things
but we don't know what is happening and that's the
[10:54] honest truth and I think that anybody who would
tell you that they have a solution A or solution
[10:58] B that would show the whole picture you know would
be lying because it's it's just not there and you
[11:03] know I'm telling it as a vendor that provides one
of those solutions um and this is because right
[11:09] now I think that uh we're in a phase of uh kind
of wild wild west of uh AI where there's a lot of
[11:16] uh fast movement uh and um a lot of you know
individuals within organization are trying to
[11:22] adapt to it. They don't have bad intentions that
are just trying to experiment and uh accelerate
[11:28] their work and and mimic what maybe somebody else
is doing from another company or a colleague. Uh
[11:34] and it's cool, right? So we download another
tool and another tool and and another agent
[11:39] and everything becomes so easy. um and tokens
are still not in force. So you can basically do
[11:44] whatever you want right now. It's going to change
soon because it's very expensive. Uh so I think
[11:49] that um at first I would say that uh we need
to realize that we don't know everything and we
[11:56] don't know anything that you know everything that
is happening on the endpoints on the machines and
[12:01] uh on the cloud. Uh so I would address it
from uh protecting the data and the assets.
[12:09] uh because if we will try to control everything
that is happening on the on the agent side itself
[12:15] uh I think it's you know we're at the stage where
it's going to be very hard uh to do because models
[12:21] are are just you know they're advancing so fast
right uh try to chase that but if you think about
[12:27] mapping out the crucial information that you have
inside your organization and think about how you
[12:32] can protect that by you know protect it from
anybody not not just an agent from from from
[12:37] an employee from an agent from an application,
from a machine um and how you can um you know
[12:44] deploy this zero trust approach uh to anybody that
tries to to access our our gems right inside the
[12:50] organization. They're this still the same.
Doesn't matter if it's AI or not. Uh I think
[12:54] that this is the right way to approach it right
now. Protect the data. Um as it goes to agents,
[13:02] I would focus right now on the basically having
the um the right accountability. So any agent that
[13:11] uh is used right now has to be strongly coupled
with the human owner behind it. And you know
[13:17] we hear that it's impossible that there's uh
autonomous agents that are deployed anywhere.
[13:22] They don't have identity uh that is uh um well
figured out. Well, I think that, you know, uh we
[13:29] can definitely do that with the right tools and
with the right process. Uh and that's what we've
[13:34] been seeing uh lately. Um the accountability part.
Yeah. Excellent. Thank you so much. I love the the
[13:42] commonality between the answers and the diversity
between the answers. It's like it's a beautiful
[13:46] blend. Okay. Now people have been uh delued with
great information and they're thinking about
[13:53] all the stuff that they want to do but you know
there's sort of this like first second third step
[13:58] and so I am going to ask you within the confines
of of course your security information security on
[14:04] your own organizations but to the to the degree
you can could you get really granular regarding
[14:11] the a specific project you don't have to name it
specifically but think of a specific project and
[14:18] the impact that it's making the design the issue
it's designed to resolve and then how that issue
[14:25] of guard rails and risk management plays into that
and I know it's a really complex question but Dr.
[14:31] Now I'm going to start with you. [laughter]
Awesome. Well, I lead governance. So all of AI
[14:37] initiatives right now come in some form through
my department. Um but what I can say is that um
[14:46] we've had a number of initiatives that have come
through like most of you like in the in the realms
[14:52] that you would imagine. um customer service being
an example and I can't go into too much detail
[14:58] about the specific initiative but um one of the
things that that initiative was really interesting
[15:06] um to go through is it it really helped us to
build out the communication necessary and the
[15:13] level of governance necessary um to really kind of
excel the project forward. It was a great learning
[15:20] opportunity in that particular project. And I I
think like when we're talking about innovation
[15:27] and when we're talking about changing the way
that we have done things for for many years,
[15:34] AI is as much technology as it is change
management, communication with stakeholders,
[15:40] um all of these education. So, so it really
kind of drove home to me how tightly wound those
[15:49] um environments have to be in order to
effectively innovate. And let's be honest,
[15:55] if there is no innovation, if we create governance
that's too strict or too um restrictive,
[16:02] then we're we're out of a job. There's there's
no governance if there's no innovation. So,
[16:07] we have to fine line that um that that risk level.
And you know, this comes back to kind of what I
[16:15] was talking about. Your risk level at your company
based off of your sector might be very different
[16:22] from the risk levels that would come out of an
energy sector or out of a commercial sector.
[16:28] Um but whatever the non-negotiables are, that
project really um kind of distilled where we
[16:36] wanted risk and where we needed humans
in the loop versus humans on the loop.
[16:43] So what I mean by that is, you know, you have
humans in the loop that are kind of, you know,
[16:48] intervening, but there's also an aspect of having
a human on the loop that's kind of monitoring the
[16:55] activities and monitoring the behaviors.
And that particular project was a project
[17:00] that really kind of broadened the thinking around
those two worlds and how they inter interconnect.
[17:08] Uh, I love you that you gave me a new phrase, on
the loop. I'm definitely going to remember that
[17:14] one. I do want to highlight your comment about
change management because I was moderating a
[17:22] panel yesterday and they were really pushing
the change management. It got me to thinking
[17:25] that does help you govern risk because the better
you do the change management process the more eyes
[17:32] you have watching for a little breakdown or an
overlap that let uh you know an issue come in.
[17:40] So thank you for mentioning that. Um I'm going to
go to Monisha next. Yeah absolutely. Um, one of
[17:47] the largest initiatives we led recently was in the
recruiting space and you know since many of us are
[17:55] either hiring managers or in that space as well.
Um, this was you know as we were growing very fast
[18:02] you can imagine the the number of you know hiring
decisions we were making the amount of people that
[18:07] were involved in the hiring committees you know
the amount of work that actually goes into hiring
[18:12] talent in some of these large organizations. It's
a it's a very long process and in order for us to
[18:19] automate parts of it, uh we took a two-step
approach. We looked at what we could automate
[18:25] from an AI standpoint. You know, of course, we
had generative AI. It could do summarization for
[18:30] us. It could do knowledge retrieval for us. It
could, you know, basically support the hiring
[18:36] decisions but not making that hiring decision um
for the company. And uh you know that's where I
[18:42] would talk about you know human in the loop
uh principle and the design principle that we
[18:46] had where do we see if we would have you know
decisions that are being made on recruiting
[18:52] we want to make sure that there is no bias in the
process and there is trust again in the process so
[18:59] um AI you know when it took over our recruiting
capability we you know we were actually able
[19:05] to speed up everything by over 30% for our
engineering team so our hiring decisions were
[19:10] much faster. We were able to hire faster as well.
So, you know, to me that was a very impactful
[19:18] um you know, initiative for uh the organization.
And uh again you know when when I look at these
[19:26] examples of course this is one example but
in in a in any situation if you're looking at
[19:34] human in the loop whether it's geni whether it's
aentic AI you have to have the right stakeholders
[19:41] um taking accountability of these decisions and of
course there's change management and responsibly
[19:46] rolling that out as well but accountability
observability to me is the foundation
[19:52] of accountability. So if something goes wrong,
who is the person that's responsible um for some
[19:59] of these outcomes? So that's the most important
uh you know part of any initiative that you lead.
[20:05] Excellent. And Saji? Yeah. So um I can tell you a
bit about one project. So we're a seven-year-old
[20:12] company. We're an identity company for you know
for obviously for humans before AI was a you know
[20:16] a big thing. Um and uh around the year serving
hundreds of customers you know fortune companies
[20:23] and u and startups and everything and um about a
year ago when we decided to go on the agentic path
[20:30] we decided also to build the suborganizations.
kind of like a startup within startup
[20:35] um very AI native uh and you know we didn't even
know what that means but fast forward I would say
[20:44] um 3 months ago we introduced uh autonomous agents
that are deployed as human employees mimicking
[20:52] human employees into the slack. So basically we
have uh an agent uh we have um it has a machine
[20:59] uh on cloud and uh it has its own identity
it registers within you know we serve our own
[21:05] uh dog food right so it registers uh within the
agent code platforms um and then it acts kind
[21:14] of as just any employee within the company. So we
have Drew Pixel that does graphics and uh we have
[21:22] Kora that does competitive analysis and we have
Nadia that does web and and we have around I think
[21:29] now around eight or nine uh of those uh agents
and it was funny when we introduced the agents
[21:36] um to the to the team on Slack. It's it was kind
of like welcome Drew Pixel and everybody says hi
[21:41] and the agent starts to talk between themselves.
it, you know, it was a bit freaky but uh but but
[21:47] people liked it and and and you know and it works.
It provides a lot of value uh at the end of the
[21:52] day. But then what we saw that some of the um um
developers they kind of you know try to get funny
[21:59] and uh and to hack it. So I tried to ask okay give
me the you know the the data about this customer
[22:05] and give me the data about the schedule of the CEO
and uh and you know maybe uh what's the um what's
[22:11] the uh compensation plan of this saleserson and
things like that you know uh just trying to hack
[22:17] it that's what developers do and you know we can't
blame them and this was the phase where we kind of
[22:23] you know face the reality of what are guard rails
and how um how important it is to set them Right.
[22:30] Uh and it's one thing to give uh instructions uh
on the context MD file of uh uh this is basically
[22:37] like the the baseline file of uh of the agent. um
this is you know it's it's only one thing but then
[22:44] if somebody asks the right question maybe uh you
know they will be able to bypass that instruction
[22:50] and again it all comes to if we want to protect
some of the data we have to sit on the border
[22:57] of where basically the uh the agent will try to
access the data and where it can get the data. Uh
[23:04] so right now what we're doing is you know all of
the data access of the agents has to go uh through
[23:13] uh basically an um gateway and um and the gateway
enforces all of the policy uh of access on the uh
[23:21] data by the way duplex so both on the incoming uh
request if uh somebody will try to get Salesforce
[23:30] uh data about a customer uh and maybe they
shouldn't have this information, it will block it.
[23:35] If somebody will try to delete data from the from
Snowflake or BigQuery, uh it will block it because
[23:41] there's a policy out there to u uh to enforce
that. But also, you know, sensitive data goes
[23:47] back, travels back. If the request went and then
um it gives some, you know, personal information,
[23:53] PII, right? Um that it shouldn't give, we will
also block the request uh the response from coming
[23:59] back to the LLMs. Um so I think that this setting
up this clear perimeter uh will definitely help
[24:10] and um and I think that there's another topic
I won't go deep into that but when we talk to
[24:16] security leaders today and they see our you know
beautiful dashboard the platform of agent.co
[24:22] uh and uh and then they see the the rule base,
they ask themselves, okay, am I supposed to fill
[24:29] this rule base? So, are we back to the days of
uh you know the firewalls when there was like you
[24:36] know uh thousands of rules and and who who who can
track that right with agents and everything moving
[24:42] so fast. So the next level of autonomousy is
having um agents right and AI helping us in in in
[24:53] the dynamic decisions and there's a few levels of
that as well just like with autonomous cars right
[24:58] you don't get to full autonomous in a second but
there's a path to get there and uh and we can talk
[25:04] about it it's a very interesting topic it is such
an interesting topic that I'm just going to jump
[25:10] to a different question totally out of order Sorry
folks, but you're executives, you can handle that,
[25:14] right? Um, so I'm going to go to Manisha because
uh does your organization have a formal emergency
[25:23] protocol for well let's talk about Whimo really
because that's we're all dying to hear about this
[25:29] one. Uh emergency protocol for taking an agent
offline and who owns that button. Is it? Is it
[25:35] legal? Is it the business unit? Like you know it
stops in front of a cable car in San Francisco.
[25:40] Now what happens? [laughter] Yeah. Uh that's a
it's a very interesting question. Uh because I
[25:46] I built the AI uh center of excellence for Whimo
and uh you know when I would go into some of these
[25:54] executive meetings you know the there was panic
panic regarding some decisions that went wrong
[26:00] or there was you know unexpected risk that we
saw. Um and you know one of the things I would
[26:07] say I I don't think it's one person who actually
uh presses the button. It's about does everybody
[26:13] know when to press the button and uh do we
all understand you know who is accountable
[26:19] for certain decisions. For example there are
business decisions where business stakeholders
[26:24] are accountable. You know from an execution
standpoint it's of course technology that's
[26:29] accountable. We have security. We have privacy.
We have legal. bringing all of these stakeholders
[26:34] together and that's why I emphasize on having
a center of excellence which actually helps
[26:40] accelerate innovation and uh it helps you scale
significantly faster and you know again you know
[26:48] when I talk about observability and that's where
you know I keep uh emphasizing as well is that
[26:53] having you know um if your if your let's let's put
it this way if your business leaders can't answer
[27:00] questions like what did the agent to, you know,
what decisions did it make, where did it fail,
[27:06] um then you're you're you're in a crisis.
And so, if you have all of that up front,
[27:12] you have the right guardrails, you will be able
to answer that and move much faster. Um, from that
[27:18] perspective, that is a great answer. And now I'm
going to put Dr. Nell on the spotlight because the
[27:24] energy sector, I mean, that's a very high-risk
sector. Can you share the answer to this? Like,
[27:30] who owns those decisions? How do you I mean and
and this is this is positioned in something went
[27:35] wrong and of course we all have to address that
fear before we roll something out. Yeah. I mean
[27:42] it's really important not to um be reactive but
proactive about this. I know you never want to
[27:49] plan for something to fail, but especially
for your high-risk scenarios um where you're
[27:56] employing technology that is higher risk to your
sector, it is really important that you have those
[28:03] hard conversations at the beginning. These are
the areas that could go wrong. Here's where our
[28:09] risk are. Here's where we're going to mitigate and
here's what we do if if those mitigations fail or
[28:15] if something happens. Like you have to have that
conversation at the beginning and it's really an
[28:20] awkward conversation to have because everybody at
the beginning of a project is really like woohoo
[28:25] we're going to solve the world peace world hunger
with AI this is going to be amazing but but having
[28:30] those hard conversations with legal involved
with um IT leaders involved with the stakeholders
[28:37] involved with the person who is accountable for
that initiative and every initiative needs to have
[28:44] somebody accountable that accountability, that
person needs to also be in that room and you need
[28:50] to come together and decide how if these areas of
high risk go wrong, what will we do in response
[28:57] to that? And there is some sort of prescribed I
would say like incident response that you could
[29:03] kind of do at an enterprise level, but for your
high-risisk use cases, it needs to be specific.
[29:09] um you need to determine that when it goes
through governance what what you're going to
[29:14] do in those scenarios and in in my sector in
particular um it's really important that we
[29:21] think about that from the beginning because
um in medicine and you know health care and
[29:28] um energy um there are lives on the line for some
situations and you know there are non-negotiables
[29:35] that you have to make sure are solid and cannot
go wrong in that scenario because you never want
[29:42] to deploy technology that in those sort of higher
risk scenarios that impact populations where lives
[29:53] are on the line that thank you so much that is
a really critical issue um so let's I I'm sure
[30:02] all of you have noticed we have a huge global
spans in our audience for this conference which
[30:08] we appreciate everyone who has traveled so far to
hear from us. Can we start to address what are the
[30:15] challenges globally? I mean, we have, you know,
there's not a a global agreement on guard rails
[30:23] and risk management. And so you've got US, you've
got other countries, some are more strict, some
[30:28] are less strict. How do you move through that?
And I'm going to start with Saji on this one.
[30:35] Yeah, I think that um you know right now um most
of this um approach is happening from let's meet
[30:43] the compliance of uh of data and um and who can
access the data and things like that. Uh obviously
[30:51] um I I would say that um the the the compliance
challenges right now in data access and especially
[30:58] data that is being is traveling back to the LLMs.
Um that's the single, you know, concern that I see
[31:06] uh speaking to to compliance managers, right? Uh
they're worried about data reaching out to places
[31:12] where it shouldn't. I think that uh here um I
would approach this by uh basically having AI
[31:23] um also um employed to kind of take a look at what
is the general footprint of data that is traveling
[31:32] inside the organization and um and same policy
guard rails of okay um this is the channel this is
[31:40] the application that can uh expose us to this type
of risk risk uh mapping out those risks mapping
[31:46] out the applications and trying again to tackle
it from uh um basically blocking uh data access
[31:53] where it shouldn't be redacting some of the data.
Um this is the you know it might not be perfect
[31:59] but I think that this is the what we see right now
as more and more dedicated AI compliance is uh is
[32:06] being imposed and we already see that happening
in the EU we can see that happening in some
[32:10] uh in some industries regulated industries but it
will take time right so right now we have to kind
[32:16] of improvise in a in a way and treat it from the
logical standpoint uh and I think that in most
[32:22] cases from our conversations at least this is this
is what the compliance and the security leaders do
[32:27] right now and that is a challenge you know yeah
I'm excited about this question because I spent
[32:33] a lot of my career thinking about this um it is
literally sometimes it can feel and I've been
[32:40] directly leading governance for probably the last
four or five years it can feel like you're kind of
[32:47] running behind a a train that's fast moving and
you're trying to kind of catch the last piece
[32:54] um especially especially with AI because we
are seeing so many changes. Um but I think
[32:59] like when we're really kind of thinking about
agentic AI which is what you all came here to
[33:05] to think about um the regulatory environment is
changing. It's changing globally. It's changing
[33:11] within the states. There's a lot of different um
regulatory pieces. We we heard yesterday there's
[33:17] um lots of laws, lots of things that we have
to follow and lots of changing laws and laws
[33:22] that are on the docket and laws that we don't even
know we're going to have to follow tomorrow. So,
[33:26] how do we build these systems to where we don't
have to scrap everything and restart again? And
[33:32] I think it comes out to um a modular design from
the beginning. I know I've been stressing over and
[33:38] over again, think about these things in the
beginning, but if you are not building that
[33:44] compliance layer on the top of your system to
where you can um easily change configuration,
[33:52] then you will end up having to rebuild these
systems when you're act when these laws change.
[33:58] So you have to think about um flexibility and
agility from the beginning. And I know it's
[34:03] more work to think about how you would build a
compliance layer into um you know a gentic AI from
[34:10] the beginning but you you have to think about
that and you have to think about it from from
[34:14] the start because if you wait until something
changes I promise you tomorrow next week two
[34:21] weeks from now 6 months from now there will be
more regulation and it will change the way that
[34:27] you have to operate in your use case you know so
think about it from the Um, one study I read from
[34:34] Deote recently actually said that one out of five
companies actually had a um a model for governance
[34:43] set up to govern appropriately and efficiently um
effectively um you know Agentic AI and for me one
[34:53] out of five like we have to get better at this and
we have to get better at this quickly um because
[34:59] the laws are changing the environment is changing
um and we can't put ourselves and our companies at
[35:06] risk because we wanted to move fast and not think
about these things in the beginning. Excellent.
[35:13] No wonder you were so excited to talk about it.
You really have put a lot of thought. Thank you,
[35:18] Mona. Yeah, I think for me, you know, if
you're looking at governance, compliance,
[35:24] um, and you know, we I I saw this firsthand, uh,
in my organization is when you look at these laws,
[35:31] these laws are there for a reason. And you
know, you have to put the human first. Meaning,
[35:37] are you what's your philosophy of building AI
or an AI agent or a company that's built on AI?
[35:45] uh is that benefiting the people or is that
you know something that will adversely affect
[35:52] the people? And when you look at that philosophy
because we were all about safety uh for us you
[35:58] know it was important that we look at everything
you know uh from a safety standpoint and when we
[36:04] looked at that you know even the global laws that
are changing like she said every day um you know
[36:10] make legal your best be your best friend you know
learn about the laws because even if you're in
[36:14] technology it's very important you understand at
least the basics of what the EU laws are you know
[36:20] how do you want to govern your systems. How do you
want to put these monitoring you know systems in
[36:26] place you know which you know Saji is actually
building these systems are you know extremely
[36:32] uh important as we think large scale and
long-term and scale and even though these
[36:39] laws would change you know significantly in the
next 6 months or even you know two years from now
[36:45] uh you you will have some of these you know basic
uh philosophical principles built into the value
[36:54] system which means that your teams are making the
right decisions for the people and that's the most
[37:00] important uh you know I I would leave you with
you know the goal should be to benefit the people
[37:05] not to impact the people. Excellent. We have six
minutes left so instead of asking another question
[37:12] I'd like to open it up to you. So, we have a
fabulous Oh, the gentleman that introduced us,
[37:19] he has a microphone up there. And so, I'd
actually like to have you sort of just stay
[37:25] up here. If you have a question, can you come up
to him because we want to get as many questions
[37:30] as we can over the next six minutes. When you
talk about um compliance can at times be an
[37:38] opposition to innovation. So, if you have too
much compliance, there's nothing to innovate.
[37:42] How do you balance that? And then how do you push
down the necessity of having compliance as folks
[37:50] are developing insurance? Whichever one of you
wants to um I can jump in first. Um I think that
[37:59] is a fine line. Um and it is specific to your
organization as I've said before. Um there are
[38:07] risks that you cannot take or that are higher risk
for you, right? whatever your sector is, whatever
[38:13] you're using and you need to come up with those at
the beginning and then have like a clear sort of
[38:21] um framework by which to mitigate. Remember in
governance you should be trying to get to yes. The
[38:29] business may not want to do the things you put in
place because it may be harder for them to do it
[38:34] but your goal is to help them innovate but safely
right and within law so you don't put yourself in
[38:41] legal peril. So so you have to think about like
you can't have too much bureaucracy because people
[38:48] will go around you or they'll um not innovate
which is contrary to your goal. But you also can't
[38:54] just rubber stamp everything because what that
will end up is you getting in legal peril. And
[39:00] no company wants to be on the news. We all could
think of examples of of companies that have kind
[39:06] of gotten themselves in in that in that thing. But
but I think it's working with the business and and
[39:12] identifying at the beginning what your high-risisk
use cases are, what your non-negotiables are, how
[39:17] you're going to mitigate them, and then working
with the business to make sure they have a clear
[39:20] plan for executing on those. Excellent. Thank
you. Next question. Hi. Um, I'm Derek Tor from
[39:26] University of Richmond. Um, I work in identity
access management and so you guys hit a lot of my
[39:31] buzzwords like lease privilege, rolebased access,
treating agents as uh identities and I was just
[39:36] wondering like what strategies or blueprints you
have for like provisioning access to agents as
[39:42] identities also the more granular stuff like you
talked about PII and how you handle that. Yeah. So
[39:49] I I can take it from the technological uh point
of view. Uh for for us for example um there has
[39:56] to be an agent registry with proof of possession
for the machine itself. Uh it has to be treated
[40:01] in a way that if we can't identify you properly
you don't go in and you don't get any access. Uh
[40:07] and I would also treat it from and and this is you
know the very technical answer. I would say that
[40:11] from the other you know uh point of view that is
more humane uh psychological even maybe uh start
[40:19] with a low access privilege and then treat it as
a credit score right as you trust more uh of the
[40:25] agent you can give it more access and and also
understand why it needs the access um and I would
[40:31] just add to that have a human behind any agent you
know don't take uh um there's two types of agents
[40:39] there's on behalf of a human where it's really
easy to to connect the the identity of a human
[40:43] and there's you know autonomous agents right that
you can think that they don't have any connection
[40:48] to a human but that's not true you can trace the
connection if you put the the right guardrails
[40:53] and process uh so make sure that any agent that
is out there accessing data you can always trace
[40:58] back who's the human that is uh owning this
agent I love the analogy to the credit score
[41:04] that's beautiful and and it's one of the things
that I want everybody to think about is like
[41:09] what's that one takeaway way that we're going to
remember on the you know ride home and the credit
[41:14] score analogy is beautiful. Next question. We only
have two minutes so I think we have two questions
[41:19] if we can keep them quick. Hi I'm Priya. So as I
do run a center of excellence and the challenge
[41:25] that I do see is like you know for different kind
of a agentic processes you know adding guardrails
[41:31] there segregation on guardrails itself. So that
by itself is becoming like a whole job you know
[41:37] because certain processes have certain guardrails
and there's no like a blanket guard rail that's
[41:43] applying. So are there any suggestions from
your side like how do you manage for different
[41:48] processes? Yeah, that's a great question because I
uh basically worked with 12 plus critical business
[41:54] functions when uh we were you know building AI
systems and uh you know one of the things that
[42:01] helped us scale is identifying low-risk, medium
risk and high-risk um use cases. For example,
[42:08] you know, high-risisk would be anything that
requires you know um a human uh which would be you
[42:14] know financials. you know you have your um human
resources and uh you know other functions that
[42:20] would actually require you know humans that would
become your high-risisk use cases. So those are
[42:25] the ones that you take to the center of excellence
not everything because otherwise there's going to
[42:29] be a lot of bureaucracy. So you want to you know
be able to get legal security privacy approvals
[42:35] for lowrisk use cases medium you know uh risk
use cases you can just have a conversation and
[42:40] then move forward. Uh but that's what helped us is
to templatize everything. Um build that operating
[42:46] model where you have the right people making the
right decisions at the right time. I hope that
[42:52] helps. I wish we had many more hours to do this,
but what I want to do is first tell you that these
[42:58] fabulous people are going to walk over here and
have their microphones removed and then I know
[43:03] that each of them would be happy to entertain
your questions at the hallway so that the next
[43:09] amazing panel can come. So, let's thank them for
what they've shared with us. Thank [applause] you.
