H&R Block, Tech Execs Say Firms Bear Full Liability for AI Agent Flaws

H&R Block与科技高管表示：企业须对AI智能体缺陷承担全责 部署自主AI的企业必须承担所有法律及客户质保责任，因为非确定性多智能体系统在投产前无法得到穷尽验证，这在数百万笔自动化交易中极大地放大了品牌风险敞口。 该小组探讨了从对话型语言模型向能够主动使用工具的智能体AI系统的转变，以及由此在网络安全、金融和企业IT领域引发的运营挑战。发言嘉宾强调，企业责任不可转嫁给自主算法，这意味着部署机构必须为其品牌承诺负责，并承担智能体出错的全部责任。由于前沿模型的非确定性使得发布前的穷尽验证难以实现，工程团队必须大力投入红队测试、零信任权限边界和混合架构。最终，讨论指出，建立严格的治理框架、可审计的决策日志以及法定的人工申诉权，对于安全提升智能体的规模化应用速度至关重要。 算力与模型能力的指数级增长与人类有限的认知带宽形成鲜明对比，推动大语言模型通过 Anthropic 的 MCP 等工具调用协议实现从“对话”到“执行”的 Agentic AI 跨越。 (3:44) 去中心化智能体的激增带来了类似“影子 IT”的安全隐患，促使企业必须为智能体构建基于密码学机器身份、持续验证和运行时权限边界的“零信任”安全架构。 (8:22) 前沿多智能体系统交互的状态空间极其庞大，使得部署前彻底验证变得在计算上不可行，系统往往不得不依赖发布后的遥测和实际运行来发现隐患。 (12:42) 在法律责任层面，部署企业必须对智能体的错误承担全部最终责任，无法将问责推诿给底层自动化子组件，诸如税务审计保证等消费者服务承诺依然必须严格兑现。 (16:09) 由于 AI 训练数据基于不完美的人类行为，系统设计重心已从追求零缺陷模型转向构建健全的故障隔离、多智能体共识审查及可追溯的审计日志机制。 (20:22)

As artificial intelligence transitions from conversational prompts to autonomous task execution, enterprise leaders argue that operational reliability, systemic risk governance, and legal accountability must be solved before agentic deployments can safely scale. Agency, Tool Use, and the Non-Deterministic Shift (3:44) Aditya Thadani distinguishes agency from autonomy, observing that large language models have moved from conversational text generation to direct action. Rather than possessing genuine human-like autonomy, modern systems exhibit agency by invoking tools to drive interactions toward predefined outcomes. Beth Linker points out that integration protocols, such as Anthropic's Model Context Protocol (MCP), allow models to execute tasks directly in external software environments rather than merely generating text predictions. Enterprise Asset Sprawl and Zero Trust Architecture (8:22) Bask Iyer compares the tracking of autonomous agents to enterprise IT struggles with unmonitored hardware and shadow staffing, warning that decentralized agent creation threatens to produce an untracked workforce. To counter this, Linker argues that network-perimeter security models are obsolete and advocates applying Zero Trust principles to agentic ecosystems through continuous verification of machine identities, explicit capability constraints, and runtime permission boundaries. Norman Sadeh notes that compounding risk emerges when multiple verified agents interact dynamically, creating emergent states that defy exhaustive pre-deployment testing. Modern models have parameter scales that render formal state verification computationally intractable, forcing organizations to rely heavily on post-release telemetry and error logs. Scale Differential and Unavoidable Enterprise Liability (18:29) Deploying agentic systems dramatically widens corporate exposure. Thadani contrasts H&R Block's traditional workforce of roughly 50,000 tax advisors—each managing a few thousand interactions annually—with a centralized AI assistant processing millions of transactions in its first year. Under this compression, behavioral drift or errors scale instantly across millions of customer touchpoints. Addressing liability, Iyer and Thadani emphasize that commercial enterprises cannot deflect blame onto autonomous software components. When deploying AI tax software, H&R Block provides the same legal protections—including audit guarantees against the IRS—as it does for work completed by human preparers. Iyer adds that executive management and boards, rather than technical scapegoats like CISOs, must absorb the ultimate accountability for automated failures. Governance Frameworks and Scalable Post-Decision Audits (23:28) Panelists highlight the shortfall of existing risk-mitigation toolkits. Sadeh points to the four core dimensions of the NIST AI Risk Management Framework 1.0 (Govern, Map, Measure, Manage), but acknowledges that techniques like red teaming, differential privacy, and algorithmic fairness adjustments remain imperfect. While supporting comprehensive data privacy statutes such as the EU's GDPR, Sadeh cautions that the EU AI Act advanced too quickly before lawmakers understood the underlying technology, expressing a preference for agile baseline standards. To balance consumer protection against the impracticality of placing a human in every automated micro-decision, Sadeh points to existing legal rights to contest automated profiling under the GDPR and CCPA. Rather than mandating humans in every runtime loop, organizations must preserve explainable logs and traceable decision paths so that human reviews can be conducted upon appeal. Testing Bottlenecks and Upstream Model Drift (27:00) Engineering workflows are increasingly dominated by evaluation pipelines rather than core model design. Thadani explains that coupling non-deterministic models with unpredictable human users forces teams to spend significantly more time building behavioral simulations and red-teaming suites. Linker observes that many software teams have not mastered automated testing for deterministic code, making probabilistic systems even harder to evaluate. This fragility is compounded by commercial foundation models that update upstream without public change logs, creating diagnostic blind spots when agent behavior regresses unexpectedly. The panel concludes that defining clear legal liability, transparent audit trails, and strict operational boundaries is not intended to stall adoption, but rather to give enterprise engineers the regulatory and organizational confidence needed to deploy autonomous agents at scale.

Ai4研讨会上，专家指出智能体AI（Agentic AI）正推动系统从被动生成转向主动执行，但这同时放大了非确定性风险与品牌暴露面；企业必须承担最终法律责任，并通过零信任架构与动态测试建立稳固护栏。 从会话走向执行的技术拐点与架构重塑 (0:04) 智能体AI的核心演进在于从单纯的语言交互跨变到实际的任务执行。Norman Sadeh结合其在卡内基梅隆大学四十年的研究经历指出，AI代理系统的本质始终是任务委托；当前的质变源于人类认知能力的恒定与计算智能指数级增长之间的巨大落差。Aditya Thadani明确界定了“能动性”（Agency）与“自主性”（Autonomy）的边界，认为大语言模型已具备使用工具并可靠完成业务目标的能动性，但并未获得人类级别的真正自主，其价值在于将互动从被动应答推进为朝向预设目标的系统化主动执行。 工具调用协议的普及是实现该跨越的技术基石。Beth Linker以Finite State保护联网设备的实践为例，指出Anthropic的Model Context Protocol（MCP）等标准框架使模型能够突破文本预测的限制，直接与软件环境、外部API及开发者工具交互。但这种对外部运行环境的直接操纵打破了传统边界，使企业不得不面对“昨日可行的流程今日可能失效”的非确定性行为模式。 资产无序蔓延与智能体时代的零信任架构 (8:22) 在企业落地层面，Bask Iyer结合前CIO经验指出，去中心化的智能体部署极易诱发严重的“影子劳动力”危机，类似于企业早期无法统计服务器与员工编制的混乱局面；IT管理者往往缺乏全面的资产台账，不知晓内部到底运行着哪些智能体及其背后的权限归属。针对这一盲区，Beth Linker主张淘汰过时的网络边界防御，将“零信任”（Zero Trust）安全范式全面移植至智能体生态，对智能体实施严格的机器身份核验、运行时权限边界划分以及显式能力限制。 与此同时，数字信任的建立呈现出高度的不对称性。Aditya Thadani强调，在高风险的财税与财务规划领域，用户信任需要依托多年可靠的交互和法律层面的正式担保逐步积累。Linker警示称，这种耗费数年时间建立的信誉可能因一次灾难性事故在瞬间瓦解，因此企业绝不能依赖静态防线，必须对智能体的凭证流转、委派权限与运行行为执行持续的加密验证。 多智能体复合失效与非确定性测试瓶颈 (12:42) 随着多个经过单独验证的智能体在一个工作流中组合，不可预测的“涌现行为”（Emergent Behavior）成为系统性风险。Norman Sadeh回顾道，早期商用操作系统即便是带着50000个已知软件缺陷上线，社会依然能通过补丁机制消化容忍；然而以OpenAI最新前沿模型为代表的现代AI，其动态状态空间在计算层面上极为庞大，根本无法在部署前完成穷尽式的形式化验证，产业界不得不高度依赖发布后的遥测数据与线上错误反馈来排查系统隐患。 这种非确定性特征直接颠覆了传统的软件测试范式。Aditya Thadani透露，工程团队如今在评估框架和模拟管道上投入的时间已显著超过设计核心AI解决方案本身。由于开放接口下的终端用户输入不可预测，开发者必须构建庞大的对抗性红队测试与提示词注入防御模拟；为此，企业必须采用混合架构，将非确定性的生成式模型与传统的确定性传统软件、统计学AI以及人工审查相互协同，以相互制衡各自的缺陷。 品牌担保责任不可转嫁与大规模故障半径 (16:09) 在法律责任与商业信誉方面，与会专家一致认为责任无法向算法推诿。Bask Iyer指出，当全自动系统危害用户权益时，最终的问责对象只能是企业、首席执行官与董事会。Aditya Thadani以H&R Block的实践说明，传统模式下50000名人类税务专业人员每人每年仅处理数千次交互，而单一AI税务助手在首年即可处理数百万笔交易，故障扩散的“爆炸半径”被成倍放大；但无论底层是由人类还是算法驱动，企业必须提供一致的法律担保，全面兜底包含防国税局审计在内的保障承诺。 针对系统不可避免的失误，Beth Linker指出公众往往存在假定人类永不犯错的认知双标，实际上重大网络安全事故频发往往源于人类分析师被海量噪音淹没。由于AI系统是基于存在缺陷的人类行为数据训练的，其出错具有必然性。因此，企业的职责在于确立清晰的风险承受度，并通过人机协同工作流以及“多智能体共识架构”（由次级智能体比对既有先例来复核主要智能体的输出）构建严密的容错兜底机制。 制度制衡、审计日志与事后抗辩权救济 (23:28) 针对政府与法律监管的介入，Norman Sadeh提及了NIST人工智能风险管理框架（AI RMF 1.0）包含的治理（Govern）、映射（Map）、度量（Measure）与管理（Manage）四维架构，但他坦言红队测试、算法公平性校准及差分隐私等技术手段目前均无法彻底消除智能体缺陷。Sadeh评价欧盟通用数据保护条例（GDPR）的体系明显优于美国分散的立法现状，但批评《欧盟人工智能法案》在立法者充分理解技术之前走得“过快过远”，并呼吁采取能促使企业持续提升技术底线而又不扼杀创新的敏捷型轻度监管。 在涉及个人信贷审批、人寿保险及医疗报销等重大权益被算法拒绝的侵权场景下，由于复杂多智能体系统的数据流转极难溯源确定近因，事前的全流程人工介入又受制于商业扩展性，Sadeh强调必须借鉴GDPR和加州消费者隐私法案（CCPA）中的法定救济途径。系统必须强制保留透明可追溯的审计日志与可解释的决策推导链条，以保障公民在权益受损时能够有效行使“拒绝纯自动化决策的权利”，从而在事后申诉环节实现实质性的人工复核。 Bask Iyer在总结中强调，及早确立清晰的法律归责标准与信任护栏并非为了阻碍AI的发展进程，而是为工程研发人员划定合规边界；唯有厘清潜在的法律与伦理风险，消除技术落地中的责任模糊地带，企业才能在确保安全受控的前提下全力释放智能体AI的创新速度。

- [00:04](https://www.youtube.com/watch?v=zFYgjFtoZ5M&t=4s) 人类生物学脑力增长平缓与计算机算力指数级跃迁之间的剪刀差，迫使人类不断将复杂任务委托给 AI 智能体，但这也放大了因意图误解带来的系统性失控风险。 ## Key takeaways - [03:44](https://www.youtube.com/watch?v=zFYgjFtoZ5M&t=224s) 大语言模型已从单纯基于提示词的“被动清谈”跨越到通过工具调用（如 MCP 协议）执行操作的“主动实干”，但也引入了非确定性并扩大了系统攻击面。 - [08:22](https://www.youtube.com/watch?v=zFYgjFtoZ5M&t=502s) 企业在智能体落地上面临“影子智能体”的资产治理盲区，而商业信任极其脆弱，建立需要漫长的交互积累，但一次重大失误即可使其瞬间瓦解。 - [11:45](https://www.youtube.com/watch?v=zFYgjFtoZ5M&t=705s) 针对智能体的零信任机制（Zero Trust for Agents）成为必然要求，需持续严格验证智能体的身份、代表权边界以及细粒度操作权限。 - [11:45](https://www.youtube.com/watch?v=zFYgjFtoZ5M&t=705s) 多智能体相互嵌套与动态交互放大了不可预测性，在极端复杂的生态中要提供绝对的系统确定性与安全保障几乎是不可能的。 - [17:05](https://www.youtube.com/watch?v=zFYgjFtoZ5M&t=1025s) 面向消费者的核心业务中，AI 智能体处理单点错误具有瞬时扩散的巨大“爆炸半径”，企业无法以底层使用了 AI 为由规避最终的担保与法律责任。 - [22:40](https://www.youtube.com/watch?v=zFYgjFtoZ5M&t=1360s) 传统“人在回路（HITL）”模式面临人类审核机器输出能力的瓶颈，未来或将演化为由智能体相互审查并达成共识的新型协作模式。 - [23:28](https://www.youtube.com/watch?v=zFYgjFtoZ5M&t=1408s) NIST AI 风险管理框架虽提供了治理路径，但现有红队测试、算法公平性与差分隐私等技术手段仍存在局限，行业更需要敏捷的轻度监管以平衡创新与合规。 - [27:00](https://www.youtube.com/watch?v=zFYgjFtoZ5M&t=1620s) 生成式 AI 的非确定性使得开发者构建测试框架的成本远超业务开发本身，必须将 Agent 嵌入传统确定性软件与判别式 AI 的混合生态中以增强鲁棒性。 - [32:47](https://www.youtube.com/watch?v=zFYgjFtoZ5M&t=1967s) 开放 API 与 MCP 协议加速系统互联的同时，也催生了恶意提示词注入、自主破坏操作等全新的攻击载体，且风险蔓延速度极快。 - [34:00](https://www.youtube.com/watch?v=zFYgjFtoZ5M&t=2040s) 多智能体通信推论引发的隐性损害极难归因，法律法规必须强制要求企业留存系统架构文档与测试记录，以保障责任链条有据可查与用户的申诉权。 - [41:10](https://www.youtube.com/watch?v=zFYgjFtoZ5M&t=2470s) 前置解决信任构建与法律责任划分并不是阻碍创新，而是为开发者设立清晰的护栏，从而从根本上释放并加速 AI 技术的工程化落地速度。 ## Detailed walkthrough ### [00:04](https://www.youtube.com/watch?v=zFYgjFtoZ5M&t=4s) 开场与定义探讨：Agentic AI 是旧瓶装新酒还是范式转移？ 研讨会主持人 Bask Iyer 轻松开场并抛出一句风趣的改编名言：“人非圣贤，孰能无过；但若想制造真正的彻底混乱，还得靠 Agentic AI（智能体 AI）。”他在开场白中坦言，自己在准备阶段也不得不重新查阅资料来理解智能体 AI 的确切内涵——在普遍认知中，智能体似乎就是脱离人类干预、自主做出决策与判断的系统。然而这引发了一个根本性追问：自动化系统早已在代替人类做决策，Agentic AI 究竟是一个全新的技术范式，还是给既有自动化概念包装的新名词？ 卡耐基梅隆大学（Carnegie Mellon University）教授、隐私工程项目联合创始人 Norman Sadeh 首先对此做出了回应。Sadeh 拥有横跨学术界、政府与创业领域的深厚背景，他曾创办网络安全公司 Wombat Security Technologies，并在 2018 年以 2.5 亿美元被 Proofpoint 收购。回顾技术演进，Sadeh 直言自己从事 AI 智能体研究已有整整 40 年之久，概念本身并不新鲜：40 年前他专注于基于 AI 的供应链与生产排程调度，20 年前投身于自动化金融交易技术，而在过去 15 年里，他的团队致力于开发自动阅读隐私条款并配置权限的“AI 隐私助手”，以及通过主动提醒（nudge）防范人为疏漏的“AI 网络安全助手”。 Sadeh 深刻指出，智能体之所以在当下再次爆发巨大吸引力，根本矛盾在于人类大脑的认知与计算能力在过去 50 年间并未显著增长，而计算机与 AI 的算力却呈现指数级跃迁。这种算力不对称诱使人类不断将复杂任务“委托（delegate）”给 AI。任务委托的程度越深，理论上生产效率越高；但潜藏的巨大风险在于，智能体极有可能误解人类的意图、做出违背初衷的决策，从而引发灾难性的失控场景。 - **40年演进路径**：AI 排程调度（40年前） $\rightarrow$ 自动化高频交易（20年前） $\rightarrow$ 自动化隐私与安全助手（15年前至今）。 - **核心驱动逻辑**：人类生物学脑力停滞 vs 计算机算力指数爆炸，导致任务委托（Delegation）成为必然趋势。 - **潜在隐患**：随着自主委托加深，语义误解与非预期动作可能放大系统性风险。 ```mermaid flowchart LR accTitle: 人类与AI能力演进与委托机制 subgraph Gap["能力剪刀差"] H["人类生物脑力<br/>(增长平缓)"] C["计算机与AI算力<br/>(指数级跃升)"] end subgraph Delegation["智能体核心机制"] D["任务委托<br/>(Delegation)"] end subgraph Outcomes["双向结果"] O1["生产力与效能跃升"] O2["理解偏差与失控风险"] end H --> D C --> D D --> O1 D --> O2 ``` --- ### [03:44](https://www.youtube.com/watch?v=zFYgjFtoZ5M&t=224s) 从“清谈”到“实干”：智能体的 Agency、自主性与工具调用 H&R Block 人工智能平台副总裁 Aditya Thadani 从企业服务运营与业务重塑的视角，拆解了智能体系统的两大核心维度：“能动性（Agency）”与“自主性（Autonomy）”。在 Thadani 看来，过去一年技术领域发生的最关键质变，是大语言模型（LLM）完成了从“只会说话（talking）”到“真正做事（doing）”的跨越。这种“实干”能力为企业创造了巨大的商业价值，但如何确保其在交付过程中保持高可靠性、一致性与可预测性，依然是企业级应用面临的巨大痛点。 谈及“自主性”，Thadani 认为公众和业界的定义往往过于宽泛甚至存在虚高。现阶段智能体系统展现出的自主性其实十分有限——其飞跃主要体现为：交互模式从原本被动等待用户输入并做反应（reactive response），演化为能够主动推进对话交互、奔着既定目标结果去执行（goal-driven）。然而，要达到人类语境下真正的全局自主决策，现有的 AI 系统还有很长的路要走。 Finite State 首席产品官 Beth Linker 则从互联设备安全与产品安全的角度补充了关键的技术维度。她指出，大模型从根据提示词给出“最佳猜测”到能够“调用外部工具（Tool Use）”，是行业的分水岭。特别是随着 Anthropic 推出 MCP（Model Context Protocol，模型上下文协议）等标准化工具协议，智能体的能力边界正在以令人震惊的速度扩张。但硬币的反面是，随着不确定性（non-deterministic）的引入，传统的软件评估体系正面临失效风险——系统昨天表现正常，今天可能就会以全新的方式运行；同时，黑客与攻击者也在利用全新的攻击面渗透企业工作环境。 | 核心维度 | 传统 LLM / 聊天机器人 | 现代 Agentic AI 系统 | |:--- |:--- |:--- | | **行动模式** | 清谈响应（Talking）：被动回答 Prompt | 实干执行（Doing）：通过工具调用执行操作 | | **交互逻辑** | 被动式单轮/多轮问答（Reactive） | 目标驱动型主动推进（Goal-Driven Pursuit） | | **连接机制** | 封闭上下文、受限文本输出 | MCP（Model Context Protocol）等协议对接工具与 API | | **行为特征** | 确定性相对可控、依赖预训练知识 | 非确定性（Non-deterministic）、动态可变且攻击面扩大 | --- ### [08:22](https://www.youtube.com/watch?v=zFYgjFtoZ5M&t=502s) 影子智能体危机与信任构建：企业级落地的高风险博弈 Bask Iyer 结合自己多年担任大型企业 CIO 的实战经验，抛出了企业管理中的现实困境。在过去，IT 部门甚至连“公司究竟雇佣了多少人、有没有给离职员工发工资”都需要部署大量繁杂系统才能查清，他调侃这就像经典情景喜剧《宋飞正传》（Seinfeld）中乔治·康斯坦扎（George Costanza）假装在某家公司上班却无人察觉的荒诞桥段。如今，这一治理盲区在智能体时代进一步恶化：企业内部谁在部署智能体、智能体潜藏在哪些角落、系统之间如何互操作，CIO 甚至连一份基础的“智能体资产清单（Inventory）”都没有。在这种“影子智能体”遍地的环境下，企业如何建立起对 AI 的信任？ 对此，Aditya Thadani 结合 H&R Block 的税务与金融咨询业务给出了答案。由于税务咨询直接涉及客户切身财产安全，信任构成了其商业模式的绝对基石。客户对品牌的信任原本建立在坐在桌子对面的专业顾问、背后的交付系统以及服务担保条款之上，这种信任是通过一次又一次的可靠交互逐步累积起来的。Thadani 认为，AI 智能体要想真正融入商业核心链条，没有捷径可走，必须遵守完全相同的规则——通过“交互复交互（interaction after interaction）”的稳定表现来赢取信任，正如 20 年前在线支付从饱受质疑走向全社会普及一样，整个技术与监管生态系统需要协同进化。 Beth Linker 对此深表赞同，并特别指出了商业信任的致命弱点：**“信任极其脆弱（Trust is fragile）。”** 企业可能历经数年甚至数十年的合规运营与优质服务才建立起用户的忠诚度，但只要智能体在一个极端恶劣的场景下出现重大失误或安全漏洞，累积的商业信任就会在瞬间蒸发殆尽。在企业治理框架尚未完全适应这种非确定性系统的当下，信任的建立与防护正变得空前艰巨。 ### [11:45](https://www.youtube.com/watch?v=zFYgjFtoZ5M&t=705s) 智能体时代的“零信任”架构与多智能体交互的不可控性 在传统 IT 运维与安全管理中，追踪企业内部资产——究竟有多少员工、多少台服务器、多少终端设备——向来都是不断变动的动态目标。Beth Linker 指出，如今企业内部“拥有多少个 AI 智能体（Agent）”同样正在演变成一个迅速膨胀的移动靶。回顾网络安全架构的演进历程，行业从早期“只要处于企业边界内即可默认信任”的传统边界防御模型，一路走向了现代的“零信任”（Zero Trust）安全范式。面对智能体爆发的未来，业界迫切需要一套针对智能体的零信任机制（Zero Trust for Agents），必须以清晰明确的方式界定：系统如何精确识别智能体身份？如何验证智能体是否真正代表其声称代表的主体？以及如何严格定义与规范它们“被允许”和“不被允许”执行的具体操作边界。 ```mermaid flowchart TD accTitle: 从传统安全到智能体零信任架构演进 subgraph Traditional["传统网络安全模型"] A["边界防御模型<br/>(默认信任内部网络)"] --> B["零信任架构 (Zero Trust)<br/>(持续验证人与设备身份)"] end subgraph AgenticEra["智能体生态演进"] B --> C["智能体零信任机制<br/>(Zero Trust for Agents)"] C --> D1["身份识别与代表权认证"] C --> D2["细粒度操作权限白名单"] C --> D3["多智能体交互动态监控"] end ``` 针对多智能体协作场景，Bask Iyer 向卡耐基梅隆大学的 Norman Sadeh 教授抛出了一个尖锐的问题：如果将两个各自经过严密测试、被判定为“良好”的智能体组合在一起，它们是否会在交互过程中产生“1 + 1 变成恶意或不可信”的负协同效应？Norman Sadeh 对此给出了肯定答复，并指出这种软件工程层面的系统集成风险由来已久：两个原本在各自测试标准下完全达标的组件，在特定业务场景中拼接后，极可能因角色定位与预期的细微偏差而偏离原有行为轨迹。在引入大语言模型与 AI 之后，这种不可预测性被急剧放大。甚至在根本层面上，业界至今都无法对什么是“好的 AI”给出一个无懈可击的标准定义。 Norman Sadeh 进一步回顾道，软件行业带病部署的历史长达数十年——二十多年前，微软发布新版操作系统时常常夹带多达五万多个已知 Bug，但大众与企业依然照常采纳。然而当今 AI 技术的复杂性已经发生质的跃迁，尤其是大量软件在彼此之上层层嵌套，系统面临的不再是两个智能体之间的对话，而是数以百计乃至千计的智能体在多层生态中复杂交织。在这种极度复杂的交互环境下，想要提供绝对的系统确定性与安全保障“完全是不可能的（it's impossible）”。面对像新一代前沿大模型这样的颠覆性发布，业界在初期往往充斥着各种主观评测，但实际上没有人能做到穷尽式全量测试，行业最终不得不依赖海量终端用户在使用过程中集体暴露并反馈潜在问题。 - **传统边界消亡**：安全机制必须从单纯防范外部实体，转变为对每一个调用的 AI 智能体实施细粒度的身份与权限校验。 - **软件缺陷的历史常态**：回顾 20 多年前 Windows 操作系统携带 50,000+ 个 Bug 交付的历史，AI 系统的不可预测性使得传统测试用例无法全面覆盖多智能体间的交互行为。 --- ### [15:22](https://www.youtube.com/watch?v=zFYgjFtoZ5M&t=922s) 信任建立、动态更新的产品与监管挑战 面对零信任与敏捷创新之间的张力，Bask Iyer 结合自身担任首席信息官（CIO）的职业经历表示，当 AI 系统在业务现场发生偏差时，技术负责人必须承担最终的问责后果。因此，企业既需要引入零信任机制设立底线约束，又不能忽视人类社会传统的“信任积累”逻辑——即从低度信任起步，通过系统在长时间尺度内的稳定表现逐步赢得信任。在强监管行业中，零信任架构看似在早期会增加摩擦、拖慢进度，但长远来看却可能是企业在满足合规要求的前提下安全加速业务创新的必要基石。 Norman Sadeh 随即补充指出，当前技术普及面临的核心困境之一在于普通公众与消费者的技术认知门槛有限，缺乏对 AI 产品可信度做出精准技术判断的专业能力。尽管品牌声誉（Brand Reputation）在消费决策中举足轻重，但现代软件的交付模式已经发生了根本变化。Sadeh 以智能汽车举例：“如今我的汽车通过每周的 OTA 软件空中升级，几乎每周都变成了一辆全新的车。”上周值得信任的品牌与软件版本，在本周推送新模型或新算法后是否依然可靠，普通消费者根本无力自行完成技术验证。这表明行业不能单凭品牌信任进行自我调节，必须依赖行业最佳实践与监管框架的建立，协助终端用户分担技术验证的成本。 --- ### [17:05](https://www.youtube.com/watch?v=zFYgjFtoZ5M&t=1025s) 品牌声誉、法律责任与爆炸半径（Blast Radius） 在探讨企业级落地的法律责任与责任归属时，Bask Iyer 提出了一个极其切中实际的商业场景：当纳税人选择专业报税服务时，客户的核心诉求是确保合规、避免遭到美国国税局（IRS）的审计，客户绝不接受企业在出现税务差错时以“抱歉，这是后台 AI 智能体搞砸了”为由推卸责任。H&R Block 的人工智能平台副总裁 Aditya Thadani 对此明确回应，在面向消费者的核心业务中，底层具体采用了何种算法技术对客户而言是“重要但属于内部实现的细节”。企业在推出面向客户的 AI 报税助手时，必须向用户提供与传统人工服务完全一致的保障承诺与法律背书。例如，H&R Block 长期推行的“安心保障计划”（Peace of Mind warranty，涵盖针对 IRS 审计的保障支持）在 AI 介入后依然无缝适用，企业绝不能因采用自动化智能体而稀释自身对客户的最终责任。 ```mermaid flowchart LR accTitle: 人工顾问与 AI 智能体风险特征与爆炸半径对比 subgraph HumanAdvisors["人类税务顾问 (约 50,000 人)"] H1["单人年均服务数千次"] --> H2["错误分散且局限"] H2 --> H3["风险爆炸半径小"] end subgraph AIAssistant["AI 报税智能体"] A1["单系统年均处理数百万笔交易"] --> A2["系统性缺陷瞬时扩散"] A2 --> A3["风险爆炸半径巨大"] end ``` Aditya Thadani 强调，系统设计者与架构师肩负的核心风险特征（Risk Profile）正在经历数量级上的剧变。过去，企业依托约 50,000 名线下专业顾问拓展市场，每位顾问一年内接待的客户交互次数不过几千次，个体失误的波及面相对孤立；然而，当单一 AI 智能体助手上线后，其在推广首年便直接处理了数以百万计的交易交互。这意味着一旦 AI 系统的核心逻辑或提示词工程出现系统性偏差，其潜在的错误扩散速度与危害程度将呈指数级放大。因此，研发团队在工程架构层面必须确保 AI 的输出严格对齐企业的核心价值观与品牌调性（Brand Voice）。 针对管理层在安全事故中将首席信息安全官（CISO）当作替罪羊的行业沉疴，Finite State 首席产品官 Beth Linker 提出了深刻的反思。她幽默地指出，人们在批评 AI 智能体犯错时，往往会下意识地将其与“从不犯错的人类”进行对照——“但等等，事实根本不是这样。”在过往大量高影响力的重大网络安全与数据泄露事件中，关键威胁情报其实早已记录在系统内，但由于海量信息过载，人类分析师根本无法凭肉眼及时捕获。当前引入 AI 智能体本质上是用“可能犯错的技术”去辅助“同样会犯错的人类”，且许多智能体本身就是基于存在瑕疵的人类历史行为数据训练而成的。 | 评估维度 | 传统人工决策模式 | AI 智能体协同模式 | |:--- |:--- |:--- | | **错误根源** | 疲劳、信息过载、盲区遗漏 | 幻觉、训练数据偏差、非预期交互 | | **交互规模** | 单人每年数千次交互 | 单一系统每年数百万次交互 | | **故障影响** | 局部、分散、可单点止损 | 系统性、瞬时扩散、爆炸半径大 | | **应对策略** | 流程复核、多级人工审批 | 护栏机制（Guardrails）、自动化熔断、细粒度风险隔离 | Beth Linker 总结指出，行业不应陷入“万一 AI 犯错怎么办”的被动恐惧，而应转向主动建立系统化的缓解与防御机制（Mitigations），并对潜在风险的“爆炸半径”（Blast Radius）进行精确度量与限制。作为软件工具与技术提供方，核心职责在于提供完备的安全工具链与合规培训；而对于采用 AI 的企业管理层而言，必须明确界定自身的风险承受底线（Risk Tolerance），在全生命周期内构建起安全运营该系统的实战能力。 ### [22:40](https://www.youtube.com/watch?v=zFYgjFtoZ5M&t=1360s) 人在回路（HITL）的局限与多智能体共识机制 在探讨 AI 决策的审查机制时，Beth Linker 指出，虽然人们常常强调“人在回路”（Human-in-the-loop, HITL），并在 AI 出现疏漏时责怪人类审核者“你的工作本该是抓住这个错误”，但现实情况是，人类并不天然擅长审查 AI 输出。审核和纠错复杂的机器输出是一项绝大多数人尚未建立起来的新型技能。未来的演进方向充满了未知与看点：究竟是继续依赖人类去逐项审核 AI 的输出，还是转向由“智能体审查其他智能体”（Agents reviewing agents）并尝试达成某种共识机制？利用多智能体协作与交叉验证来建立共识，在技术上已有成熟的先例可循，整个行业必将对传统的审核模式进行系统性革新。 ```mermaid flowchart TD accTitle: AI 输出审查与共识演进路径 subgraph Traditional["传统审核模式"] A["AI / Agent 输出"] --> B["人类审核员 (HITL)"] B --> C["输出放行 / 驳回"] end subgraph Agentic["新型智能体共识模式"] D["初级 Agent 动作"] --> E["监督/评估 Agent"] E --> F["多智能体共识机制"] F --> G["最终执行与合规确认"] end ``` ### [23:28](https://www.youtube.com/watch?v=zFYgjFtoZ5M&t=1408s) NIST 风险管理框架与 AI 治理的技术局限 卡耐基梅隆大学（CMU）的 Norman Sadeh 教授切入核心议题，提出组织必须建立通用的风险评估框架，以便在产品和服务上线前系统化地剖析潜在风险。以美国国家标准与技术研究院（NIST）推出的 AI 风险管理框架（AI RMF）为例，该框架虽然并非尽善尽美，但从四个维度提供了清晰的思考路径： - **治理（Govern）**：确立组织内部的角色职责、流程规范与执行政策。 - **映射（Map）**：识别引入新技术或新服务可能暴露的各种风险类型。 - **度量（Measure）**：量化并评估识别出的各类风险严重程度。 - **管理（Manage）**：落实针对性举措以减轻和控制风险敞口。 Norman Sadeh 分享了自己在 CMU 讲授 AI 治理课程时的真实教学体会。他每学期一开始就会坦率地对学生说：“我会尽我所能教给你们当今已知的一切，但我希望你们明白，我手里其实没有任何完美的解决方案。”在课堂实践中，学生们运用现存框架分析大量真实产品并寻找缓解措施，却不得不面对一个令人沮丧的真相——现有的防御与治理手段往往存在严重短板： - **红队测试（Red Teaming）**：虽能模拟对抗攻击，但难以覆盖所有不可预测的边界情况。 - **算法公平性技术（Algorithmic Fairness）**：在数学定义与复杂现实之间难以取得真正平衡。 - **差分隐私（Differential Privacy）**：在保护数据隐私的同时往往会牺牲模型效用。 Sadeh 强调，我们正处于一个前所未有的复杂环境中，既不能扼杀创新，也不能放任自流。在监管取向上，他高度肯定欧洲的通用数据保护条例（GDPR），但认为《欧盟人工智能法案》（EU AI Act）有些操之过急、步子迈得太大；行业需要的是“轻度监管”（Light Regulations），以敏捷方式促使企业随着技术的进步不断拔高治理门槛。 ### [26:10](https://www.youtube.com/watch?v=zFYgjFtoZ5M&t=1570s) 繁文缛节对创新的扼杀与智能体代理潜力 Bask Iyer 对过度监管的副作用表示赞同，并生动地吐槽了欧洲的数字合规现状：每次在欧洲用手机上网，光是点击那几百个 Cookie 授权弹窗就让人不胜其烦，“有那工夫，我都宁愿直接手动去柜台订票了”。他呼吁绝不能让律师和层层条款绑架 AI，否则将严重阻碍创新，毕竟人们迫切希望把繁琐无聊的日常事务全权委托给智能体。Norman Sadeh 随即插话回应了“Cookie 困境”，指出满屏的弹窗并非 GDPR 的本意，未来完全可以开发专门的 AI 智能体代表用户在后台自动处理这些隐私偏好设置。 ### [27:00](https://www.youtube.com/watch?v=zFYgjFtoZ5M&t=1620s) 构建置信度：非确定性环境下的测试挑战与生态协作 H&R Block 的 Aditya Thadani 从工程与工业落地角度指出，团队当前在构建测试工具上所花费的时间，甚至远远超过了设计解决方案本身的时间。这种现象背后的深层原因在于： - **随机系统与开放环境叠加**：非确定性（Stochastic / Nondeterministic）系统运行在一个完全不可预测的动态环境中，用户的交互方式千奇百怪、无法预设。 - **置信度（Confidence）的构建困难**：除了用户对系统的信任，更核心的是开发者与构建者自身对系统安全边界的信心。 - **传统测试手段彻底失效**：常规的软件测试框架完全无法应对生成式 AI 的发散性与恶意滥用，必须耗费大量精力搭建红队模拟与多场景动态评测工具。 Aditya 强调，绝不能孤立地看待 AI 系统。Agent 必须嵌入在一个由“老派传统软件（Old-school software）”、“传统判别式 AI”以及“人类”共同组成的综合生态中，通过系统架构让各组件相互弥补短板，才能打造出真正具备鲁棒性的应用。 ### [29:25](https://www.youtube.com/watch?v=zFYgjFtoZ5M&t=1765s) 炒作还是质变？Agentic AI 的规模化拐点与极端尾部风险 面对 Bask Iyer 提出的尖锐问题——“我们是否小题大做？AI 的概念几十年前就存在，我们现在描述的流程传统系统都在做，现在到底有什么根本性不同？”——Norman Sadeh 与 Beth Linker 给出了坚定的回应：当前的技术演进发生了根本性的质变。 虽然智能体（Agent）的基础概念存在已久，但“Agentic”（具备自主代理能力）作为新范式，其颠覆性体现在三大维度： 1. **部署规模与技术威力激增**：过去 Agent 仅局限于小众利基领域（Niche），而如今随着大模型与算力投资的爆发，Agent 已经能自主浏览网页、自动注册账户、直接调用信用卡进行支付交易。 2. **决策权的不可逆让渡**：人类大脑的算力进化极其缓慢，远落后于计算机算力的指数级增长。无论是在防空驱逐舰上自动拦截导弹的近防自主武器系统，还是企业试图用 Agent 替代软件工程师、在酒店用 AI 替代前台礼宾部，人类正面临前所未有的诱惑将决策权全面移交。 3. **渐近线灾难（Asymptotic Scenarios）的极端风险**：虽然客服电话偶有卡顿尚可容忍，但当具备物理世界操作权或金融权力的自主 Agent 在复杂网络中产生连锁失误时，其后果将是极其严峻且具有毁灭性的。 Beth Linker 补充道，尽管科技行业每隔几年就会将旧概念包装翻新，生成式 AI 面临的许多威胁与当年互联网普及初期如出一辙，但生成式 Agent 具备的爆炸性传播力与自主行动力，使得其潜在危害的扩散速度和影响深度远超以往任何技术周期。 ### [32:47](https://www.youtube.com/watch?v=zFYgjFtoZ5M&t=1967s) API、MCP 与智能体攻击面的激增 Beth Linker 指出，随着 API 的广泛普及以及类似模型上下文协议（Model Context Protocol，MCP）等机制的出现，开发者能够在极短的时间内以惊人的速度将系统各个接口打通。然而，这种极高的开放性也带来了前所未有的安全隐患——攻击者正以过去难以想象的方式利用普通用户。很多用户天真地认为系统完全安全，直到他们突然发现，自己为了配置编程智能体（Coding Agent）而从网上随手下载的一套规则配置文件，里面竟然夹带了恶意指令，正在疯狂删除本地的所有文件。 智能体生态中正在涌现海量全新的攻击载体（Attack Vectors）。这类风险最致命的特征在于其“0 到 100”的失控蔓延速度，这与传统的人为失误有着本质区别。因此，无论从安全合规、监管要求还是企业内部治理的角度来看，保持高度谨慎都至关重要。企业不可能因噎废食停止采用 AI，但绝不能抱有侥幸心理，认为“什么治理都不做也能应付过去”。 ```mermaid flowchart TD accTitle: 智能体连接生态中的新型安全风险传导 subgraph S1["开放连接层"] A["开放 API 与 MCP 协议"] --> B["外部规则 / 提示词配置"] end subgraph S2["执行层风险"] B --> C["恶意指令注入 / 隐蔽攻击"] C --> D["智能体自主执行破坏操作<br/>(例如：批量删除本地文件)"] end subgraph S3["治理需求"] D --> E["内部治理机制"] D --> F["安全与监管合规"] end ``` --- ### [34:00](https://www.youtube.com/watch?v=zFYgjFtoZ5M&t=2040s) 侵权责任的困境：合理人标准与黑盒决策追踪 问答环节开启，现场观众抛出了一个关于法律侵权责任（Tort Liability）的核心问题：传统侵权法的核心基石在于“合理人标准”（Reasonable Person Standard）——如果在特定情境下行为符合合理人假定，通常无需承担责任；然而在现实中，当涉及特斯拉等自动驾驶汽车引发的事故时，公众与司法系统似乎并不遵循这一标准，而是直接判决企业承担责任。那么在未来的智能体时代，法律责任认定将走向何方？ 主持人兼嘉宾 Bask Iyer 直言，自己虽然不是律师，但逻辑非常朴素：如果一辆自动驾驶汽车把他带到了错误的危险地点，他绝对不会因为那是“软件智能体操作的”就放过平台，而是必须追究企业的全部责任。尽管人类司机犯错极多，他平日里也乐于乘坐自动驾驶汽车（既感觉更安全，又省去了和司机寒暄的社交负担），但一旦真正遭遇事故，他的本能反应就是打电话给车企 CEO 或董事会厉声质问：“这到底是怎么回事？” 卡耐基梅隆大学的 Norman Sadeh 教授补充道，相比于车祸这种显而易见、容易定责的灾难性事故，多智能体协同场景下的侵权责任其实更加隐蔽且难以追溯： - **智能体间隐蔽通信**：智能体之间会自主交换关于个人的大量数据，并在背后形成各种不可见的推论。 - **高风险社会后果**：这些算法推论可能直接导致某人被拒贷、被拒绝提供健康保险或人寿保险，甚至失去购买房产的机会。 - **因果归因极难**：受害者很难将这类损害明确倒查回某一个特定的底层模型或算法缺陷上。 - **强制要求最小化文档与测试**：不能指望商业公司出于单纯的善意去主动透明化系统。必须出台法规，强制要求企业提供最小限度的系统架构文档记录与前置测试，确保责任链条有据可查。当前的法律体系在应对此类隐性损害时已经严重滞后。 --- ### [36:56](https://www.youtube.com/watch?v=zFYgjFtoZ5M&t=2216s) 人机协同（Human-in-the-Loop）治理与法定义务 针对观众关于“是否应立法强制在最先进的 Agent 系统中引入人机协同（Human-in-the-Loop）”的提问，H&R Block 的 AI 平台副总裁 Aditya Thadani 给出了构建者维度的见解。他指出，负责任的系统首先源自负责任的设计，但仅仅依赖开发者的道德自律显然是不够的。人类所有的社会机构与国家治理体系，核心都建立在制衡（Checks and Balances）之上：既要鼓励大家做正确的事，也必须有人在外部监督与核实。AI 治理目前正处于快速演进的摸索期，各方必须在动态实践中找到平衡点。 Norman Sadeh 教授结合现有数据保护法给出了法律维度的澄清与补充： - **数据访问与更正权**：无论是欧盟的 GDPR，还是美国加州的 CCPA 等各州隐私法律，都赋予了个体访问自身数据并审查错误的权利。 - **反抗自动化决策权（Right to Object）**：GDPR 明确规定，当个人因自动化决策遭遇严重不利影响（如被拒发房贷或医保拒保）时，有权申请人类介入进行复核。 - **可解释性与日志留存**：为了满足法定的人类复核机制，AI 智能体系统必须保留完整的决策追踪日志，具备解释其推理过程的能力，甚至能够与用户展开交互对话。面对庞大的服务规模，全流程部署真人监管并不现实，但在关键争端环节提供审计踪迹与申诉出口是不可逾越的底线。 --- ### [39:40](https://www.youtube.com/watch?v=zFYgjFtoZ5M&t=2380s) 经典 AI 与生成式 AI：非确定性带来的验证困境 观众随后提到了软件工程底层的核心痛点：经典规则驱动的 AI 智能体输出具有确定性，而基于大语言模型的生成式 AI 智能体本质上是非确定性的（Non-deterministic），目前业界极度缺乏评估和验证非确定性输出的测试框架。 Finite State 的首席产品官 Beth Linker 对此深表赞同，并揭示了一个更为残酷的现实工程现状：在传统软件开发领域，很多工程师甚至连确定性代码的自动化测试套件都未能完备搭建，面对生成式 AI 的非确定性测试更是束手无策。 她分享了自己团队经历的切身之痛：团队曾基于某款商业商业化闭源大模型开发了一项核心功能，最初一周表现完美无瑕；然而到了第二周，团队所有人都反馈“系统好像变钝了、不好用了”。更让人沮丧的是，由于商业大模型底层发生的静默更新或飘移不可探知，工程团队根本无法查清到底是什么因素在何时被改变，更找不到定位问题的抓手。这种调试维度的失控只会越来越严峻，迫使行业必须在评估、观测与度量手段上寻求突破。 --- ### [41:10](https://www.youtube.com/watch?v=zFYgjFtoZ5M&t=2470s) 结语：通过前置解决信任与责任释放创新速度 在全场总结环节，主持人对嘉宾与观众的深度探讨表达了感谢。他强调，智能体 AI 的爆发是一个令人心潮澎湃的伟大时代，讨论安全风险与责任界定绝不是为了给行业泼冷水，连他自己都跃跃欲试想要投身其中打造新产品。 他重申的核心观点在于：**只有尽早解决信任（Trust）与法律责任（Liability）的核心痛点，为技术落地划定清晰的安全护栏，才能真正消除开发者的后顾之忧，让工程师全速推进，从而加速整个行业的创新步伐，而不是拖慢发展的节奏。** ## Notable quotes > [00:04](https://www.youtube.com/watch?v=zFYgjFtoZ5M&t=4s) 人非圣贤，孰能无过；但若想制造真正的彻底混乱，还得靠 Agentic AI（智能体 AI）。 — Bask Iyer > [08:22](https://www.youtube.com/watch?v=zFYgjFtoZ5M&t=502s) 信任极其脆弱（Trust is fragile）。 — Beth Linker > [11:45](https://www.youtube.com/watch?v=zFYgjFtoZ5M&t=705s) 完全是不可能的（it's impossible）。 — Norman Sadeh > [15:22](https://www.youtube.com/watch?v=zFYgjFtoZ5M&t=922s) 如今我的汽车通过每周的 OTA 软件空中升级，几乎每周都变成了一辆全新的车。 — Norman Sadeh > [17:05](https://www.youtube.com/watch?v=zFYgjFtoZ5M&t=1025s) 但等等，事实根本不是这样。 — Beth Linker > [23:28](https://www.youtube.com/watch?v=zFYgjFtoZ5M&t=1408s) 我会尽我所能教给你们当今已知的一切，但我希望你们明白，我手里其实没有任何完美的解决方案。 — Norman Sadeh > [26:10](https://www.youtube.com/watch?v=zFYgjFtoZ5M&t=1570s) 有那工夫，我都宁愿直接手动去柜台订票了 — Bask Iyer > [34:00](https://www.youtube.com/watch?v=zFYgjFtoZ5M&t=2040s) 这到底是怎么回事？ — Bask Iyer > [39:40](https://www.youtube.com/watch?v=zFYgjFtoZ5M&t=2380s) 系统好像变钝了、不好用了 — Beth Linker > [41:10](https://www.youtube.com/watch?v=zFYgjFtoZ5M&t=2470s) 只有尽早解决信任（Trust）与法律责任（Liability）的核心痛点，为技术落地划定清晰的安全护栏，才能真正消除开发者的后顾之忧，让工程师全速推进，从而加速整个行业的创新步伐，而不是拖慢发展的节奏。 — Bask Iyer ## Mentioned Bask Iyer, Norman Sadeh, Beth Linker, Aditya Thadani, Carnegie Mellon University (CMU), Wombat Security Technologies, Proofpoint, H&R Block, Finite State, Anthropic, Model Context Protocol (MCP), 《宋飞正传》(Seinfeld), 乔治·康斯坦扎 (George Costanza), 微软 (Microsoft), 美国国税局 (IRS), NIST (美国国家标准与技术研究院), AI 风险管理框架 (AI RMF), 通用数据保护条例 (GDPR), 欧盟人工智能法案 (EU AI Act), 特斯拉 (Tesla), 加利福尼亚州消费者隐私法案 (CCPA) ## Video description Bask Iyer, CEO, BaskMind.com Beth Linker, Chief Product Officer, Finite State Norman Sadeh, Professor/Director, Carnegie Mellon University Aditya Thadani, VP - Artificial Intelligence Platforms, H&R Block Abstracts This session will examine how autonomous AI systems are transforming high-stakes decision-making, reshaping human-machine trust relationships, and raising urgent questions around legal liability and ethical accountability. [Source](https://www.youtube.com/watch?v=zFYgjFtoZ5M)

[0:04] Bask Iyer: 人非圣贤，孰能无过；但若想制造真正的彻底混乱，还得靠 Agentic AI（智能体 AI）。 [8:22] Beth Linker: 信任极其脆弱（Trust is fragile）。 [11:45] Norman Sadeh: 完全是不可能的（it's impossible）。 [15:22] Norman Sadeh: 如今我的汽车通过每周的 OTA 软件空中升级，几乎每周都变成了一辆全新的车。 [17:05] Beth Linker: 但等等，事实根本不是这样。 [23:28] Norman Sadeh: 我会尽我所能教给你们当今已知的一切，但我希望你们明白，我手里其实没有任何完美的解决方案。 [26:10] Bask Iyer: 有那工夫，我都宁愿直接手动去柜台订票了 [34:00] Bask Iyer: 这到底是怎么回事？ [39:40] Beth Linker: 系统好像变钝了、不好用了 [41:10] Bask Iyer: 只有尽早解决信任（Trust）与法律责任（Liability）的核心痛点，为技术落地划定清晰的安全护栏，才能真正消除开发者的后顾之忧，让工程师全速推进，从而加速整个行业的创新步伐，而不是拖慢发展的节奏。

[0:04] Bask Iyer: to err is human, it takes agentic AI to create real chaos [3:44] Aditya Thadani: talking to doing [3:44] Beth Linker: the way that it worked yesterday may not be the way that it works today. [16:09] Norman Sadeh: today my car is a different car nearly every week as new software gets downloaded. [16:09] Bask Iyer: Can you get away by saying, 'Sorry, I used an agent'? [20:22] Beth Linker: Oh, wait. That's not right. [20:22] Beth Linker: Oh, the AI made a mistake [23:28] Norman Sadeh: a little too far, a little too fast [27:00] Aditya Thadani: can say anything [37:34] Norman Sadeh: right to object to automated decision making [37:34] Aditya Thadani: do the right thing [39:40] Beth Linker: A lot of developers don't even have good automated tests for their deterministic code, never mind generative AI.

[0:04] Welcome everyone. Please see that be be [0:06] relaxed. It's okay. Well, I'm Basa here. [0:09] Thank you for coming. We have a fabulous [0:11] panel. So, I'm not going to talk much [0:13] except they say it's a it's human to [0:16] error to errors but human. But it takes [0:19] really agentic AI to create chaos. So, [0:21] we're going to have a lot of chaos with [0:22] the agents we had. Um, but I want to let [0:26] the panelists introduce themselves, but [0:29] I want to start off with what I thought [0:31] agentic AI was. I had to read up on it. [0:34] I thought I knew and you know um agents [0:38] making decisions autonomously [0:40] and you know making calls with without [0:43] humans. Uh my first question is I'm [0:46] going to start with Norm and go in that [0:48] order. Haven't we haven't we been doing [0:50] this? Are there is it a new term? But [0:53] automatan was pretty much doing making [0:55] decisions about humans right so what do [0:56] you think about it please introduce [0:58] yourself and then [0:59] >> okay [0:59] >> perspective [1:00] >> and I'll try to be brief hopefully this [1:02] is working so I'm Norman said I'm a [1:03] professor at Carnegie Melon it's a [1:05] pleasure to be here thank you for having [1:06] me on this on this panel uh what do I do [1:09] I uh work in um AI and human interaction [1:14] uh cyber security privacy and AI policy [1:18] also co-ounded and co-direct the privacy [1:20] engineering program at Carnegie Melon [1:22] which is considered to be the top [1:25] program in this space and I've done a [1:27] few other things including starting [1:29] companies. I was the founding CEO of [1:31] Forbat Security Technology, a company [1:33] that I sold to Proof Point in 2018 for a [1:36] quarter billion dollars. Uh that company [1:39] uh is helping tens of millions of [1:41] employees around the world stay secure [1:43] and I've done a few other things such as [1:45] working for government at some point in [1:47] time. to get to your to your question. [1:49] So, uh I've been around for a long time [1:51] and in fact I've been working on AI [1:53] agents for about 40 years. Uh and so uh [1:58] this is not exactly a new concept but [2:00] obviously what we have seen most [2:02] recently is an acceleration in the [2:04] capabilities of AI that has opened the [2:06] door to all sorts of different things. [2:08] So 40 years ago I was working on AI [2:12] based scheduling supply chain [2:13] management. 20 years ago, I was working [2:16] on automated trading technologies. Over [2:18] the past 15 years, I've been working on [2:21] AI privacy assistance on Christianity [2:23] that reads privacy policies [2:25] automatically, helps you configure your [2:26] privacy settings. AI cyber security [2:29] assistance that help answer your cyber [2:32] security questions, nudge you to make [2:35] the right decisions, pay attention to [2:37] things that you would naturally tend to [2:39] overlook. So to not take over the whole [2:43] panel here, um I think that the appeal [2:45] of agents has to do with the fact that [2:50] unfortunately [2:51] uh brain power has not increased that [2:54] dramatically over the past 50 years. But [2:56] the power of computers has and will [2:58] continue to and the same applies to AI. [3:01] And so increasingly it's very tempting [3:03] to say let's delegate these tasks to AI. [3:06] And so AI agents are all about [3:08] delegating different tasks to AI. And [3:11] the more you delegate, presumably the [3:13] more productive you are, the happier you [3:15] are. Except that obviously occasionally [3:17] these agents may not fully understand [3:19] what you would like to do. They might [3:21] misinterpret things. They might make [3:23] things that you wish they didn't make. [3:25] And uh obviously it's very easy to come [3:27] up with all sorts of horror scenarios [3:29] which I'm sure we're going to be talking [3:30] about. So I'm going to stop here and [3:32] give a chance to my other panelists [3:34] perhaps to introduce themselves and [3:35] >> I mean you guys also introduce yourself [3:38] and then you give your perspective on [3:40] >> agentic AI as well. So we have an [3:42] variety of definitions here. [3:44] >> Absolutely. Uh hi Adite Tardani with uh [3:47] HR block VP of AI platforms and I work [3:50] primarily right now on um uh building AI [3:54] systems uh really maybe more exploring [3:56] building AI systems to reimagine the [4:00] experience the services we offer but [4:02] also the the service operations that [4:05] goes into delivering [4:07] um you know when I think about uh agents [4:10] and agent existence there are kind of [4:13] two things that come to mind that we [4:16] hear about a lot the the agency aspect [4:19] of it and the autonomy you know the [4:21] agencies in some ways I think over the [4:24] last year or so we made the transition [4:28] where LLM went from talking to doing [4:31] right that's to my mind the the key [4:34] element of agency and that's you know [4:36] obviously has a lot of value to business [4:39] and we kind of look at how do we do that [4:42] reliably consistently predict ably [4:45] something that [4:47] u maybe [4:49] uh you could say we struggled with with [4:51] LLMs generally speaking right the [4:53] reliability the consistency [4:55] but the the autonomy part to my mind is [4:58] is a bit trickier from a definitional [5:01] perspective [5:03] um you know we even in people within [5:07] organizations we talk about autonomy in [5:09] such different tones and the to my mind [5:12] the limited indication of autonomy I see [5:15] right now in agentic systems is to the [5:18] extent that we are designing systems [5:23] uh that have made the leap from reacting [5:27] and responding to a user to actually [5:31] driving to an outcome pursuing an [5:33] objective of its own right so to the [5:36] extent that if you think about a person [5:39] talking to an agentic system it's gone [5:40] from just responding to what the user [5:43] says to actually driving the [5:45] conversation, the interaction to some [5:48] objective and that to my mind is a [5:51] limited I mean but autonomy especially [5:53] in humans when we talk about it is such [5:55] a [5:58] broad concept that I I struggle to see [6:01] true autonomy in these systems yet. [6:04] >> Okay. [6:05] >> Hey everybody, Beth Linker. I'm the [6:07] chief product officer at Finite State. [6:11] We're a cyber security company helping [6:14] connected device manufacturers keep [6:16] their products secure and we are using [6:20] AI to do that and what we're seeing as [6:24] we as we build AI agents as we work with [6:28] organizations who are adopting AI agents [6:32] in their software development process in [6:34] their product security process [6:37] is that we're we're at a really [6:40] interesting in place and the shift from [6:44] LLMs that could take a prompt and give [6:49] you it's their best guess at what you [6:51] wanted in response to LLMs that can use [6:55] tools. So as Adita just said the [6:57] difference between talking and doing and [7:00] doing with tools in particular with the [7:03] roll out of things like MCP model [7:05] context protocol from anthropic [7:09] we're opening up what's possible at a a [7:12] really astonishing rate and in doing so [7:17] we're going to find out some of the [7:20] limitations of these things the hard [7:23] way. Um, as people discover [7:27] new and innovative ways to [7:32] attack these new tools that are that are [7:34] operating within all of our environments [7:39] and, you know, all of our our [7:42] workplaces. [7:43] And so it's a it's a really I think [7:47] exciting time to be to be working with [7:49] agents to be figuring out how to use [7:52] agents responsibly [7:54] and how to measure the behavior of [7:58] agents in a world where everything is [8:02] becoming less predictable, less [8:04] deterministic [8:06] and you know the way that it worked [8:07] yesterday may not be the way that it [8:09] works today. And so all of our [8:12] frameworks for technology are going to [8:15] have to stretch a little bit as as some [8:18] of these things become more variable. [8:22] >> So let's get a debate into that at some [8:24] stage. But um let's get into trust which [8:26] is the topic of what we're talking [8:28] about. We're struggling to define [8:30] agentic AI uh because some of it has [8:33] already shown in other computer systems [8:34] but each of us have a interesting [8:36] definition. Uh I'm an old um enterprise [8:40] IT person and we struggled to even find [8:43] out how many people work for the [8:44] company. It sounds ridiculous but we had [8:46] to put so many systems to find out who [8:48] work for us. Are we paying the wrong [8:50] people etc. And we finally somewhat got [8:53] a hold of it. Reminds me of a Seinfeld [8:55] episode where George will show up to [8:57] some office and people thought he worked [8:59] there and nobody knew. So my concern is [9:02] first of all how do you do trust in this [9:04] kind of environment? Secondly, even as a [9:06] CIO, I don't know where my agents are. I [9:09] don't know who's deploying agents. Uh, [9:12] and I don't even have an inventory. And [9:14] then interoperability of agents because [9:17] I I believe it's a democratic process [9:19] and every every person here is going to [9:21] deploy agents. And so I'm looking at it [9:24] as how do you handle it and how do you [9:26] build trust. So first question is and [9:29] I'm going to put it to the H&R Block [9:31] company because your whole business is [9:32] on trust. So Adita, how do you how do [9:36] you build trust within the AI systems? [9:41] >> So you're absolutely right. Trust is [9:43] absolutely central to our business. We [9:44] are in the advice business, taxes and [9:46] financial services. And when you kind of [9:50] think about from a consumer perspective, [9:53] they have come to trust the brand. They [9:56] have come to trust the advisor that's [9:58] sitting across the desk from them. They [10:00] are implicitly trusting the systems that [10:05] we all use in service delivery, but they [10:08] also kind of trust the warranties that [10:11] come with that service, right? So the [10:14] trust is kind of made up of all of these [10:16] elements and in many ways is [10:20] earned through repeated interactions, [10:22] you know, interaction after interaction. [10:24] And and to my mind in many ways if AI [10:30] systems are going to enter this equation [10:32] in a meaningful way they have to play by [10:35] the same rule of earning that trust [10:38] right interaction after interaction. U [10:42] so I I think people are going to trust [10:46] systems the way they have always trusted [10:48] systems. You know the the example I kind [10:51] of think about is um you you know we all [10:54] take online payments for granted now [10:57] wasn't the case 20 years ago right a [11:00] whole ecosystem had to evolve to build [11:02] that trust right we have to get to the [11:06] same place if we're going to realize [11:07] value from this tech uh it we have to [11:11] get to the same point from from an [11:13] ecosystem perspective [11:16] >> it looks like you're itching to go how [11:18] would you build trust I was so I I was [11:20] nodding in agreement and I I think the [11:21] thing about trust is that it's fragile. [11:25] So you can work really hard to build and [11:29] maintain your customers trust as a [11:31] business and all it takes is one really [11:36] bad situation to make that completely [11:39] evaporate. And you know I was also [11:42] thinking Basque as you were talking [11:45] about you know the challenges of you [11:47] know it traditionally how many people do [11:49] I have many servers do I have how many [11:52] devices do I have and all of those have [11:54] always been moving targets and how many [11:57] agents do I have is is going to be a [12:00] moving target too and if you look at how [12:03] we've evolved from you know the old days [12:06] when you had a network and everything [12:08] was inside the perimeter and you could [12:10] trust the stuff inside but not the stuff [12:12] outside to now when you've got things [12:14] like zero trust. I mean, we're going to [12:17] need a zero trust for agents. We're [12:19] going to need really crisp ways to [12:23] articulate [12:25] how do we identify agents? How do we [12:28] know that they represent who they claim [12:30] to represent? And how do we [12:34] articulate what it is that they are or [12:37] are not allowed to do? And I think that [12:39] a lot of that is is still developing. [12:42] >> Norma, I'm going to twist it. You're [12:43] used to working with smart people. So, [12:44] I'm going to ask you a tougher question. [12:47] So, it's not just one agent. It's like [12:49] an agent interacting with another agent. [12:51] So, you could have could you have two [12:52] good agents which oneplus 1 becomes evil [12:56] or or not trustworthy? [12:58] >> Well, I'm sure that you can, right? Uh [13:01] and so, uh in a way, the problem that [13:03] you're describing here is is not new, [13:06] right? Uh you can have two quote unquote [13:10] good pieces of software in the sense [13:12] that each one of them has been tested to [13:14] meet certain types of criteria and then [13:17] you put them together and hoping that [13:19] together going to do something except [13:20] that in truth they're not quite fitting [13:23] the profile that and the role that [13:25] they're supposed to play within that [13:26] scenario and all of a sudden two good [13:28] pieces of software perhaps with AI in [13:30] this case are no longer going to behave [13:34] as expected. The truth is could you even [13:37] define what a good AI is? I think that [13:41] this is very challenging. So perhaps [13:43] going back to the original question here [13:45] uh you know can we really trust these [13:47] products? Ideally would like to be able [13:48] to we've been living in a world where uh [13:51] we've placed our trust in a number of [13:53] different products. Uh even before any [13:56] AI found its way into products uh [13:58] software as we know has been deployed [14:00] with bugs. I mean uh there was a time 20 [14:03] plus years ago when uh Microsoft was [14:05] known to release new versions of its [14:07] operating system with 50,000 plus bugs [14:10] uh in it and yet we would all adopt the [14:13] technology. What's challenging today is [14:15] that with AI these products these [14:18] technologies are becoming increasingly [14:19] complex and so testing that technology [14:22] and testing all these interactions in [14:24] also an ecosystem where increasingly [14:26] software builds on top of other [14:28] software. It's not even just two agents. [14:30] It's going to be a multitude of agents [14:32] interacting with one another. So, how [14:34] can you provide any sort of guarantees [14:36] within this environment? And the quick [14:37] answer is it's it's impossible, right? [14:39] So, just a few days ago, Chad GPT5 was [14:43] released, right? Uh and different people [14:46] started to write articles about whether [14:48] or not it's good, whether it's better, [14:50] this and that. In truth, nobody has any [14:52] clue. You cannot fully test this thing. [14:54] It might look very good for a while. uh [14:57] increasingly we're going to rely on all [14:59] these users of the system collectively [15:02] hopefully reporting potential problems [15:04] and then when something bad happens [15:05] hopefully that's going to be picked up [15:07] very early hopefully some of these [15:09] things will be picked up before the [15:10] product gets released that would be the [15:12] better scenario can we have regulations [15:14] to help us or far can these regulations [15:16] go I assume we're going to be talking [15:17] about those things so don't want to pre [15:19] preempt future your questions and I [15:20] realize I've answered more than your [15:22] questions [15:22] >> no no good but I'm landing in somewhere [15:24] between where bet is and you you are [15:27] because as a as a chief information [15:29] officer or a technology guy, I'm still [15:32] going to be held accountable if when [15:33] these things do something. So, I do want [15:35] something like a zero trust, but I like [15:37] the idea of building trust. That's how [15:40] we build trust with humans, right? We we [15:42] start off with no trust or or you know, [15:45] a little trust, but then you have to [15:47] prove that uh you're worth the trust [15:49] over a long period of time. Maybe you [15:51] need a combination of both is what I I [15:53] think in when I look at critical cases [15:56] zero trust is probably a not a bad idea. [15:58] It may slow down progress. Uh but maybe [16:02] in a regulated environment you can go [16:03] you can speed up process if you can if [16:05] you have the zero trust. I don't know. [16:06] I'm landing somewhere in the middle. [16:09] >> Could I potentially respond to this? So [16:11] I I think one of the challenges that [16:13] we're facing today is that people are [16:15] not sophisticated enough to necessarily [16:17] be able to make good judgments when it [16:19] comes to deciding what they can trust [16:21] and what they cannot trust. So yes, [16:23] brand reputation is a big deal and we [16:24] want to continue and people will clearly [16:27] continue to build up their brand. But in [16:30] truth, you know, today my car is a [16:33] different car nearly, you know, every [16:34] week as new software gets downloaded. [16:37] And so perhaps it was a grand brand last [16:39] week. still trusted this week and what [16:42] do we need to help us as everyday [16:44] consumers to somehow be able to trust [16:48] these products? Uh clearly we're not [16:50] able ourselves to do all the checking [16:52] that needs to be done. And so what [16:55] potential regulation or best practices [16:57] would we want to see being adopted over [17:00] time to make that task easier for people [17:02] to enable people to continue to trust [17:04] these technologies? [17:05] >> You want to go to the next one which is [17:07] getting a little bit harder now. Okay. [17:09] If um let's talk about both brand and [17:12] liability, okay? So I come to you and do [17:15] my taxes. I don't want to get audited. I [17:18] don't want to hear that you're using AI [17:19] and AI screwed up or agent screwed up. I [17:21] don't want to know about your technology [17:22] now. I just want to make sure it works. [17:26] So how would you who who's liable? How [17:29] do you handle this? Can you get away by [17:31] saying, "Sorry, Basque. I used an [17:33] agent." And then you're going to get put [17:35] into prison. [17:37] No, I I think you're spot on in many [17:39] ways when when consumers are engaging [17:42] with brands, [17:44] um all of these details in some ways are [17:50] important but immaterial, right? Because [17:53] when you think about it, um so we [17:56] deployed an AI assistant for helping [18:00] people who are trying to do using our [18:02] software to do their own taxes, right? [18:05] when we deploy it, we offer the same [18:08] warranties to a user that uses the AI [18:10] assistant that they used to get the day [18:13] before the assistant was there, right? [18:15] So, as as a company, when we deploy [18:17] these systems, we essentially have to [18:20] offer the same warranties. This, you [18:22] know, we have a pro peace of mind that [18:25] protects you against an IRS audit, [18:28] right? [18:29] So those all those things have to stay [18:31] in place and we as designers and [18:34] builders have to put together systems [18:37] that kind of support that warranty right [18:41] that's in some ways how we are trying to [18:44] earn that trust right um now part of [18:48] that also is that uh the the risk [18:52] profile as designers is changing a lot [18:55] right we go to market through 50,000 or [18:59] so people advisor who are sewing our [19:04] clients doing taxes every year and if [19:07] you look at each one of them they may [19:08] have a few thousand interactions client [19:10] interactions in a year each one of them [19:13] you deploy an AI assistant it'll you [19:16] know in early year it had millions of [19:20] transactions in a year right so suddenly [19:23] you are actually creating a totally [19:26] different kind of risk profile because [19:29] uh as much as it is AI from a client's [19:32] perspective, it is it has to align with [19:35] our values with the brand voice. And I [19:39] think the onus to to your point, the [19:41] onus is on designers and builders to [19:43] make sure that these systems represent [19:46] those values and align with that with [19:49] with what we want to stand for. Beth, [19:52] you work with CISOs quite a bit, I would [19:54] assume, or you know, they've been made [19:57] the scapegoat for a period of time, [19:59] right? And something happens, cyber [20:01] security, ransomware, you said, "Oh, the [20:02] CESO didn't do his job or whatever." I [20:05] don't think somehow when you're actually [20:07] using agents to run your business to [20:08] increase your profitability, [20:10] productivity, etc., I have a feeling [20:12] that management cannot have that copout [20:14] to just put it on a seesaw or I mean, he [20:17] he or she is accountable. So, what is [20:19] your perspective? who's who's [20:21] accountable? [20:22] >> So, it's it's interesting to me that [20:24] when when we talk about [20:27] AI agents making mistakes, we we often [20:32] either implicitly or explicitly hold [20:35] them up in comparison to humans who [20:37] never make mistakes. Oh, wait. Oh, wait. [20:41] That's that's not right. Um, you know, [20:44] and it's it's interesting because, [20:48] you know, if you look at a lot of [20:51] security mistakes, a lot of security [20:54] breaches, [20:56] in a lot of really high-profile cases, [20:59] the information was there, but the [21:02] people couldn't see it because it was [21:04] buried in so much other information. [21:08] And so, you know, we're talking about [21:11] bringing AI agent technology, [21:15] which can make mistakes, to augment [21:18] humans who also sometimes make mistakes. [21:21] And a lot of those agents are trained [21:24] off of human behavior, which isn't [21:27] perfect. And so [21:31] I think that we have to think about it [21:34] less in terms of what if this makes a [21:37] mistake and more in terms of what [21:40] mitigations can you put in place for the [21:45] inevitable mistakes that will occur. So, [21:47] Adida, you know, you were saying that [21:49] the the number of interactions your AI [21:52] assistant can have is so many orders of [21:54] magnitude larger than what one person [21:58] can do. And so, sort of the blast radius [22:01] if there's something incorrect in an AI [22:05] assistant like that could be a lot [22:07] bigger. So, you know, thinking about how [22:10] do you how do you mitigate some of that? [22:12] How do you measure it? And then at the [22:15] end of the day, you know, as a as a [22:17] software provider, we're making tools. [22:20] We can train people. We can give them [22:23] the ability to use those properly, but [22:26] at the end of the day, they've got to [22:29] figure out how to operate it safely [22:32] within within their risk tolerances. And [22:34] I think um you know, one of the things [22:37] I've seen a lot of people I've worked [22:38] with who've gone, "Oh, the AI made a [22:40] mistake." And it's like, "Yeah, but your [22:41] job was to catch it." Um, so having [22:44] having humans in the loop and making [22:47] sure that they're set up to be [22:50] successful and I think um, you know, [22:53] reviewing AI output is not necessarily a [22:56] thing that most of us are are good at. [23:00] You know, it's not a a skill that we've [23:02] all had a chance to build. So, it's [23:05] going to be really interesting to see [23:07] how how that evolves and, you know, [23:09] whether it's humans reviewing AI output [23:12] or agents reviewing other agents and [23:15] trying to get to some kind of consensus, [23:17] which again is a thing that we've got a [23:19] lot of established uh precedent on. I [23:23] think that, you know, we're going to [23:24] have to evolve how we do all that. [23:28] Well, I I think we're we're dancing [23:29] around uh some some core issues here and [23:32] so perhaps it's time to mention them. Uh [23:35] and uh I would start with with uh you [23:38] know the need to have some kind of a a [23:41] common framework to help organizations [23:44] whatever their products uh [23:46] systematically analyze uh the potential [23:50] risk that are associated with these [23:51] products these services. There are [23:53] frameworks that have been developed uh [23:55] for instance by NIST. They're not [23:56] perfect but they're certainly doing a a [23:59] reasonable job at identifying the many [24:01] different dimensions along which you [24:03] want to think. For instance uh the the [24:05] the NIST AI risk management framework [24:07] revolves around four different [24:09] dimensions. There's a governance [24:10] dimension which has to do with roles and [24:13] processes and policies that you want to [24:15] have. There is also three there are [24:18] three other dimensions. It's called map, [24:20] manage, and measure. And that has to do [24:22] with identifying the different types of [24:24] risk that you might potentially open [24:27] yourself up to with your new product or [24:29] your new service. Understanding what [24:31] those risks are, measuring them, and [24:34] trying to mitigate them. Now, I would [24:36] like to add that uh so I've been [24:38] teaching this now for a few years. I [24:40] teach AI governance at at Carnegie [24:42] Melon. And one of the first things I [24:44] tell my students is I'm going to do my [24:46] best to teach you everything we know [24:47] today or at least a good amount of what [24:49] we know. But I want you to realize that [24:50] I don't don't have any good solutions. [24:52] And so I've had my students that's part [24:54] of the the work that they do in the the [24:56] class. I've had them analyze a number of [24:58] different products using these [24:59] frameworks and look for potential [25:01] mitigations. And the sad truth is that [25:04] there are very very few mitigations that [25:07] are really going to enable you to say, [25:09] "Oh, wow. We got really this. We really [25:11] got this right." You've got concepts [25:13] like red teaming. You've got algorithmic [25:16] approaches to dealing with fairness. Uh [25:18] you've got differential privacy. I mean, [25:20] you've got a whole series of of [25:21] techniques that I I teach. And each one [25:24] of these techniques, as it turns out, [25:25] fall short. So, I think one of the [25:27] challenges that we need to realize is [25:29] that we've moved into a really really [25:31] complex environment where we clearly [25:34] don't want to stop innovation. And I'm [25:36] not a very strong advocate for very [25:38] strong regulation in this space. I've [25:40] got a very different view about privacy. [25:42] I think GDPR is great. Not perfect, but [25:44] I think it's much better than what we [25:46] have in most places here in in the US. [25:49] The EU AI act is a different story in [25:51] that regard. I think they went a little [25:53] too far, a little too fast, not knowing [25:55] what they want to do. So, we want to [25:56] move towards trying to understand best [25:58] practices, coming up with light [26:00] regulations that force people to [26:02] continuously raise the bar as we get [26:04] better techniques, better methodologies [26:06] to deal with these many different [26:08] threats that we got to deal with. I have [26:10] to agree with you. I think I get [26:12] frustrated when I go to Europe even to [26:13] use the internet, you know, by the time [26:15] you click on the 100 cookies and stuff [26:17] on your mobile phone. Might as well go [26:19] and and book the thing manually, right? [26:21] So, we don't want to do that. That that [26:24] is I mean, we have to use technology to [26:25] do it smarter and not just have lawyers, [26:28] you know, say acceptance for everyone of [26:30] the AI. So, hope hope we don't let [26:32] regulations um stifle innovation because [26:35] it is fun. It seems like cool. I mean [26:37] there's a lot of mundane things I don't [26:39] want to do that I would love an agent to [26:41] do. How do you do it carefully and how [26:43] do you do it? Uh so I think that's a [26:44] good balance to have in the perspective. [26:47] >> Right. [26:48] >> At some point I'd like to respond to [26:49] that cookie issue but I uh the quick [26:51] answer is that GDPR is not responsible [26:53] for that and you can actually develop [26:56] agents that will deal with that for you. [26:58] That's the quick answer. [27:00] No, I I think it's SPA in many ways um [27:05] at least you know in my experience we [27:08] are spending more time building tools to [27:13] test the solution than the time that [27:15] goes into designing the solution itself. [27:18] Right? And in large part because you [27:20] know you kind of have this combination [27:22] of a stochastic system nondeterministic [27:25] system in a in a dynamic unpredictable [27:29] environment people can say anything when [27:31] they talk to it right uh when you put [27:34] that combination together uh building [27:37] confidence and I think to my mind you [27:40] know you talked about trusts [27:42] before I think about consumers trusting [27:46] a system I think about us the builders, [27:51] people working on it, having confidence [27:52] in the system and what does it take to [27:54] have that and with these systems you [27:58] know the way we kind of tested that is [28:00] no longer sufficient like it's actually [28:03] far from sufficient right we have far [28:04] outstripped that so now we are really [28:07] spending a lot of time on building tools [28:09] and frameworks for you know beat red [28:12] teaming and simulating malicious use but [28:14] also sort of just [28:17] uh just the diverse use cases of how [28:20] people will engage and interact with the [28:22] technology is so unpredictable in some [28:24] ways because you know people surprise [28:27] you right you especially when you put [28:31] something as open in it as it is in [28:34] front of people how they engage and how [28:36] they interact is can be really [28:38] surprising. So but you know this is this [28:41] is really a big part of what we are [28:43] learning in some ways is how to uh how [28:48] to uh build solutions and I I think I [28:51] would kind of go back to the idea of we [28:54] shouldn't think about these AI systems [28:56] in isolation they are always going to [28:59] live in an ecosystem where there is sort [29:02] of the old school software the old [29:04] school AI people all actually [29:10] meaningful parts of this ecosystem and [29:13] we have to kind of design it in a way so [29:15] that things are kind of compensating for [29:17] each other's gaps and and I think that's [29:20] kind of how we build these robust [29:23] systems. So, we have five more minutes [29:25] before Q&A. I'm going to ask a quick [29:28] round. As you guys were talking, I'm [29:30] thinking, are we just making a bigger [29:31] deal than it really is? It is. Is it [29:33] just we've been doing testing systems, [29:36] processes? Is this AI now? We're trying [29:39] to invent something that doesn't exist [29:41] because everything we described is stuff [29:43] we've done. And by the way, AI itself, [29:45] Norm, you reminded me, I did my thesis [29:47] in AI a long time ago. So, it's not a [29:49] new term. We just made it up in the last [29:51] few years. So are we just creating [29:54] should we just follow some of the [29:55] guidelines that we've done before or is [29:56] there something fundamentally different [29:58] now? [29:58] >> I'll start with you a little bit. [30:01] >> I think there is something fundamentally [30:03] different even though I made the case [30:04] that agents have been around that [30:06] concept has been around. The term [30:07] agentic by by the way is a fairly new [30:10] term. I was not using that until a few [30:12] years for for many years. But um what is [30:16] new I think is the environment where [30:18] we're deploying these agents, the scale [30:20] at which we're deploying these agents, [30:22] right? So agents were really niche for [30:24] many many years. Uh and AI was still [30:27] very niche for for many years until [30:29] maybe 15 15 years ago. What has changed [30:32] now is the power of the technology, the [30:35] the the the the investments that are [30:37] being made in this technology and how [30:40] quickly this is likely to change [30:42] everything. If you think about these [30:44] agents, they're going to they are today [30:46] going on the web. Uh they can create [30:48] accounts. They can, you know, use your [30:51] credit card. Um they are doing all sorts [30:54] of things right on your behalf. And [30:56] that's the consumer side of things, [30:57] right? Think about the military, right? [30:59] I mean, we we've deployed autonomous uh [31:02] technology for defense. You know, you [31:04] look at these amazing guns that you've [31:06] got on frigates uh to defend against [31:09] incoming missiles. that's fully [31:11] automated automated uh right and and uh [31:15] I think the temptation would be to [31:16] automate more and more just because as I [31:18] mentioned at the beginning brain power [31:20] is not really improving uh at the same [31:23] speed as as computers and so it's really [31:25] tempting to delegate more and more and [31:27] so when you look at the power of this [31:28] technology what is capable of doing how [31:31] tempting it is to rely more and more on [31:32] that replacing all your employees [31:34] perhaps with with agents at some point [31:36] which is clearly very tempting and we're [31:38] seeing some of that in software [31:39] development in all sorts of other [31:41] industries you call if you're staying [31:43] here at the MGM Grand and you call the [31:46] concierge or whatever it is you fall on [31:48] an AI today. It can be a little [31:50] frustrating because that technology is [31:51] not perfect. So I think that's not going [31:53] to be the most horrific scenario. It can [31:55] survive with that but there are clearly [31:57] scenarios I think where where the the [31:59] consequences could be extremely [32:00] extremely drastic. So I don't think that [32:02] we're uh you know making this a bigger [32:05] deal than it is. It is a huge deal with [32:07] potentially asytoic scenarios that are [32:09] absolutely horrific. [32:12] >> Yeah, I think um you know it's a it's a [32:14] great question [32:16] and some extent you know in this [32:19] industry everything old is new again [32:21] every couple of years and so there are [32:24] definitely a lot of things where you [32:26] know the threats associated with AI are [32:30] the same ones that we had when we put [32:32] everything on the internet. But I think [32:34] also to professor Sat's point, the the [32:39] sort of explosive nature of the reach [32:42] that generative AI agents have because [32:47] of because of APIs and things like model [32:50] context protocol where you can really [32:54] just open things up at a a really rapid [32:57] clip is creating [33:00] new and [33:03] previously unmatched Imagine ways of [33:06] taking advantage of users who think [33:10] that, you know, everything is okay until [33:13] they discover that, you know, the set of [33:16] rules that they uh downloaded from the [33:18] internet for their coding agent [33:21] apparently had a bunch of stuff in it [33:22] that wasn't supposed to be there and now [33:24] it's deleting all their files. And you [33:27] know, there are a lot of lot of new [33:28] attack vectors in this stuff. And I [33:31] think that the the speed at which some [33:33] of these can go from 0 to 100 is is [33:37] different from human error. So it's the [33:40] same but also and you I think that [33:43] proceeding with caution from a a safety [33:46] perspective, from a regulatory [33:48] perspective, from an internal governance [33:50] perspective makes a lot of sense. Like [33:52] you can't stop, but you also can't say, [33:55] "Yeah, it'll be fine. We don't we don't [33:57] really need anything for governance cuz [33:59] that's not going to work. [34:00] >> Okay. So, we have about 7 minutes for [34:02] questions. If anybody has a question, [34:04] just wait for a mic or or if you have to [34:07] speak very loud. [34:10] >> Um question about tort liability. [34:14] Okay, maybe you can hear me better now. [34:16] I have a question about tort liability. [34:18] uh and as you mentioned that humans are [34:21] not perfect and tort liability is based [34:22] on the idea of a reasonable person that [34:24] you're not liable if you act as a [34:26] reasonable person under those [34:28] circumstances and yet when we talk about [34:31] electronic um you know self-driving cars [34:34] and the Tesla and the recent cases where [34:37] they've been found liable we've not held [34:39] that same standard at least doesn't seem [34:41] that way I was curious to see where you [34:43] see a liability going forward and where [34:45] that standard may [34:49] I'm not a lawyer, but the the way I look [34:52] at it is if uh if the uh self-driving [34:56] card puts me in a wrong place or puts me [34:59] in a dangerous situation, I'm going to [35:01] hold the the the company accountable. [35:04] I'm I'm not going to give it a pass [35:06] because it's a software agent. At the [35:08] same time, it is true that humans make a [35:10] lot of mistakes. Sometimes I like [35:11] driving this autonomous car. It seems to [35:13] be safer to me and I don't have to have [35:15] a conversation. So that's a call that [35:18] you would make, right? So it seems safer [35:20] to me somehow. But I would hold not [35:23] knowing what the law is, I'm probably [35:25] going to say I'm going to call the CEO [35:27] of the board and say, "What the hell is [35:29] going on?" That's my call. So I'm not a [35:32] lawyer either. Uh but uh and leaving [35:35] aside the the the car crashes and and [35:37] the other uh really uh horrible uh [35:42] things that AI could be responsible for [35:45] in some form or another. I'd like to [35:46] point out there are lots of other [35:48] scenarios where tort might be more [35:50] difficult uh to prove uh today. Uh just [35:54] think about communication that will be [35:56] taking place between agents. Think about [35:59] information that be that would be shared [36:01] by these agents about different people. [36:03] The inferences that might be made based [36:05] on that information. Uh people might get [36:07] denied health insurance, uh life [36:10] insurance, uh mortgages and all sorts of [36:13] other things and tracing that back to a [36:16] particular technology will prove very [36:17] very difficult. This is one reason why I [36:20] think there is a need for people to more [36:23] carefully document their products and [36:25] people will not do this out of their own [36:27] benevolence. I think companies will need [36:30] to be told that there there is an [36:31] expectation of minimal documentation, [36:33] minimal testing so that you can trace [36:36] this back to someone uh because those [36:38] consequences even though they're not as [36:40] bad as directly killing someone in a car [36:43] crash can still be extremely dramatic. [36:45] And I believe even though I'm not a [36:46] lawyer that our laws right now are not [36:49] necessarily sufficient when it comes to [36:50] that [36:52] there's another question there. [36:56] >> Yeah. Thank you. So quick question in [36:58] this way. U I know putting laws like [37:02] what uh EU AI act has created learning [37:06] and all kinds of everything disclosures [37:08] versus completely freedom of not doing [37:10] it. Is there a between place that where [37:14] human in the loop is mandated even with [37:17] the most sophisticated agentic system if [37:20] humans are put in in a certain place not [37:23] just by company's own doing or their own [37:26] risk tolerance but as a mandate what do [37:28] you think about that kind of a way to [37:30] govern this and where would that go I [37:32] just wanted to get perspective [37:34] >> does anybody you have a point of view [37:35] adita or [37:37] >> you know [37:39] maybe a bit of a contrarian point of [37:41] view uh you know as a as a designer and [37:44] builder I couldn't think of [37:47] how do you get [37:49] uh responsible AI or responsible system [37:51] of any kind is to build responsibly [37:56] right I think what you're asking is is [37:58] that enough can we trust people to just [38:00] do the right thing and no I think that [38:03] you're right in the end uh social [38:07] systems institutions governmental [38:09] systems are all designed around checks [38:11] and balances, right? [38:13] Yes, to the right thing and yes, [38:15] somebody is verifying that you're doing [38:17] the right thing. I suspect that we are [38:20] all going to navigate and find that [38:23] equilibrium in this space. It's still [38:24] new and evolving very fast and we'll [38:27] struggle to keep up with it, but we'll [38:29] find it. I suppose I hope [38:33] >> you [38:34] >> so um I'm not sure that your [38:36] interpretation of the EU AI act is [38:38] entirely accurate. uh but going back to [38:42] some of the scenarios I was mentioning [38:43] earlier for instance being denied health [38:45] insurance or a mortgage for instance [38:48] there is something in the EU privacy law [38:50] GDPR and that we also have here in many [38:53] different states like California CCPA [38:56] and it's the right for people to access [38:57] their data and review it for potential [39:00] errors. GDPR also has a right to object [39:03] to automated decision making, which is [39:05] basically if I get denied my mortgage or [39:08] my health insurance, I have the right to [39:10] request that a human being reviews the [39:13] the decision in case there is a mistake. [39:15] And I I think that's that's something [39:16] that we need to have. That means that [39:18] these AI systems will have to maintain [39:20] again some traces for what they're [39:22] doing, be able to explain their [39:24] reasoning and engage potentially even in [39:26] dialogue with the user. uh because at [39:28] scale we may not be able to have a human [39:30] being every time there's a decision [39:31] that's made that you want to potentially [39:32] object to. [39:34] >> We have probably one more question [39:38] behind you. [39:40] Would [39:41] you agree that one of the challenges in [39:44] validating the outputs of classical AI [39:47] agents versus generative AI agents would [39:51] be the non-deterministic nature of the [39:53] outputs in the generative AI as opposed [39:57] to the more deterministic outputs of the [40:01] classical AI and that we don't have [40:03] enough frameworks or models for testing [40:07] and validating um this uh this scenario [40:12] that that we're coming on with the [40:13] undeterministic part of it. [40:16] >> Yeah, bet. Do you want to? [40:17] >> Yeah, I mean and I think that's a a [40:19] great point. And if you look at software [40:22] development, [40:24] a lot of developers don't even have good [40:27] automated tests for their deterministic [40:29] code, never mind generative AI. So, you [40:33] know, there is there is still a lot of [40:35] maturing to do around this. And, you [40:38] know, I've seen it with my own teams [40:39] where we've built something using, [40:44] you know, a a commercial LLM and one [40:48] week it was working great and then the [40:49] next week everybody said, "Hey, you [40:51] know, this this doesn't seem to be [40:53] working as well." And we had really [40:56] nowhere to go for, well, what changed [40:59] and why? And that is probably only going [41:02] to get harder as we go. So we're going [41:04] to have to get creative about how we [41:07] measure it and and manage it. [41:10] >> All right, I think we had about the [41:12] time. So let me just uh thank the [41:14] panelists for fabulous discussion and [41:16] the audience for participating. Uh my [41:18] takeaway is uh we definitely this is [41:21] exciting time. I don't want to put a [41:23] downer on it. It's a fabulous thing. [41:24] Even I feel like I want to get back in [41:26] and do things. So if we tackle the trust [41:30] and liability issues earlier then we can [41:33] actually let the developers go faster [41:34] and and speed up innovation rather than [41:36] slow down. So thank you very much for [41:38] the panelists and thank you for the [41:39] crowd. Appreciate it. Thank you. [41:42] [Applause]