AI Project Failure Rates Surge To 42% As Enterprise Pilots Stall

企业试点陷入停滞，AI项目失败率飙升至42% 当企业领导层侧重模型工程而非变革管理与聚焦业务工作流时，生成式AI项目往往难以带来财务回报。 尽管生成式AI在企业中得到广泛采用，但专家组数据揭示了一个严峻的ROI（投资回报率）悖论：80%的企业报告其未对财务底线产生实质影响，导致项目放弃率急剧飙升。为弥合这一差距，专家组强调，AI规模化应用需要严格的组织变革管理、针对性的人员战略（例如“3T”框架以及从“构建者”转向“编排者”），并将应用范围收窄至高价值的快速见效领域。展望未来，业界领袖警告称，多步骤智能体工作流面临严重的复合错误率问题，严格的治理、人工监督和跨部门协同管理因而不可或缺。 企业部署 GenAI 面临严峻的 ROI 悖论：虽然 80% 的企业声称已实现规模化应用，但同样有 80% 表示未见显著财务回报，且 AI 项目放弃率在 2024 年底激增至 42%。 (0:04) 成功规模化应用 GenAI 本质上是组织变革管理挑战而非纯技术问题，相比试图解决宽泛场景，聚焦在特定客服支持等立竿见影的细分场景更能确立业务价值。 (1:18) 大量概念验证（PoC）难以迈向生产规模，核心瓶颈在于跨数十个数据源时的检索质量（RAG）和安全权限管理；当前企业级应用最切实的定位是协助人类分担高负荷工作的“AI 协作者”，而非全自主 Agent。 (19:10) 技术用工需求正从昂贵的基础模型“构建者”转向平台“编排者”与“增强型专业人员”，企业管理者正通过“培训、转岗或淘汰（3 Ts）”框架与晋升机制强制推动业务端采纳。 (25:06) 多步骤 Agent 存在显著的误差累积效应，若单步执行准确率为 99%，在 50 步复杂业务流中的总体失败率将逼近 40%，因此设定严格护栏和逐步可见性对 Agent 治理至关重要。 (39:38)

Despite widespread enterprise adoption of generative AI, organizations face rising abandonment rates and minimal bottom-line impact. Industry panelists argue that overcoming this scaling impasse requires strict change management, narrow domain-specific deployments, and structured guardrails against compounding agent errors. The Enterprise Adoption and ROI Disconnect (0:04) Enterprise deployment of generative artificial intelligence is marked by a sharp contradiction between deployment claims and realized returns. Citing a McKinsey study, moderator Dan Costa notes that while 80% of companies report deploying GenAI at scale, an identical 80% report no significant bottom-line financial impact. Furthermore, data from an S&P Global survey of 1,000 enterprises indicates that project abandonment rates increased from 17% in 2023 to 42% by late 2024. Panelists draw comparisons between current market messaging and past technology cycles. Jeff Evernham recalls how early vendors of object-oriented databases falsely claimed direct C++ database access would make database administrators obsolete. Today, vendors similarly pitch generative AI as an effortless, out-of-the-box solution, downplaying the operational friction, data integration complexities, and lack of predictability inherent in enterprise rollouts. Organizational Change Management and Workforce Realignment (1:18) Somya Kapoor argues that scaling AI is an organizational change management challenge rather than a purely technical barrier. When executive leadership establishes explicit adoption mandates, companies achieve substantial efficiency gains, such as 2,000 hours saved per month in individual departments. Paul Farnsworth notes that without visible executive sponsorship and departmental champions, corporate initiatives stall. He adds that according to Dice research, 40% of new U.S. tech job postings now require AI skills, spanning three tiers: Builders, Orchestrators, and Enhanced Pros. Farnsworth cautions leadership against falling into the high-cost 'builder trap' of engineering bespoke platforms instead of using orchestrators to integrate existing tools. To compel internal adoption, organizations are employing both incentives and formal operational mandates. Evernham points to Amazon's Ring division, which requires staff to demonstrate AI-driven efficiency gains to qualify for promotions. Howard Miller outlines an aggressive corporate workforce framework termed the '3 Ts'—Train existing employees on AI workflows, Transfer personnel to roles less vulnerable to technological disruption, or Terminate staff who cannot adapt. Miller and Costa observe that human resources departments remain unprepared for managing digital agent workforces alongside human employees. Targeted Deployments and Domain-Specific Architectures (5:42) The panel emphasizes that early enterprise success stems from executing narrowly scoped quick wins rather than broad, speculative projects. Kapoor illustrates this through a customer support deployment where deploying AI alongside human agents enabled the department to reduce staff from 150 to 90 over two years by reallocating workers to other units. In higher education, Miller highlights how UCLA Anderson deployed an interactive Harvard Business School management case study in 30 days; by using a six-page system prompt, every student engaged in a 15-step iterative coaching dialogue with an AI persona. Panelists advocate for domain-focused micro-models trained on internal enterprise data combined with reinforcement learning, rather than broad general intelligence models. Farnsworth cites Ultrasound AI—a startup predicting precise birth dates to prevent preterm complications—as proof that models targeting a single, narrow clinical outcome consistently deliver high accuracy without the overhead and unpredictability of general-purpose LLMs. Technical Scaling Hurdles and Compounding Agent Errors Bridging the gap between initial proofs of concept and production remains a significant obstacle. Evernham notes that while retrieval-augmented generation (RAG) works reliably on 100 to 1,000 documents, retrieval accuracy drops drastically and permission controls become unmanageable when scaled to millions of documents across 20 to 30 disparate source systems. Consequently, Kapoor positions the current enterprise capability at the 'co-worker' stage—where AI assists humans, such as expanding support ticket quality audits from a 5% human sample to 100% automated coverage—while cautioning that fully autonomous agents remain unready for core operations. Looking ahead, Evernham predicts a wave of near-term enterprise agent project failures caused by compounding error rates. Even with advanced models operating at a 99% single-step accuracy rate, executing a sequential 50-step autonomous workflow compounds to an overall failure rate of approximately 40% ((0.99)^50 ≈ 60.5% success). To prevent system failure, panelists recommend narrow task scoping, step-by-step visibility, cross-functional oversight, and automated evaluation frameworks. High-leverage AI tooling is driving unprecedented organizational leverage, as evidenced by lean startups achieving hundreds of millions in revenue with minimal headcounts. However, unresolved challenges in multi-step agent reliability, systemic data governance, and the integration of digital agents into traditional human resource structures will determine whether mainstream enterprises realize sustainable financial returns.

Dan Costa 与行业专家小组指出，尽管生成式人工智能在企业中被广泛试验，但多数企业仍深陷“高采用率与零财务回报”的矛盾之中；成功实现规模化落地的核心并非追求底层通用技术，而在于由管理层自上而下推进组织变革、严格聚焦单点业务场景，并建立应对智能体复合错误风险的治理机制。 回报率悖论与PoC向生产落地的断层 (0:04) 麦肯锡的研究显示，多达80%的受访企业声称已在大规模使用生成式AI，但同样有80%的企业表示这些部署未对公司财务底线产生显著影响。同时，标准普尔全球（S&P Global）对1000家企业的调研揭示，主动放弃绝大多数AI项目的企业比例从2023年的17%急剧攀升至2024年底的42%。Jeff Evernham在现场调查中发现，虽然许多企业已拥有投产方案，但几乎没有一家能够宣称实现了全公司范围内的真正规模化部署。 Evernham指出，从概念验证（PoC）走向企业级规模化存在严重的工程断层。在处理100至1000份文档的小型试点中，检索增强生成（RAG）和定制模型表现良好；然而，当数据规模扩展至跨越20到30个异构系统的数百万份文档时，数据检索质量下降（即“垃圾进，垃圾出”）以及复杂的权限与访问安全控制便会成为核心阻碍。过度轻信厂商“即开即用”宣传的企业往往严重低估了实际业务流的工程摩擦。 组织变革管理与自上而下的制度驱动 (1:18) TheLoops创始人Somya Kapoor与Paul Farnsworth强调，AI规模化受阻本质上是组织变革管理问题，而非纯粹的技术缺陷。企业若缺乏高层设立的明确业务目标与跨部门协调，AI项目必然陷入停滞。Kapoor结合其实践指出，在CFO自上而下的效率指标驱动下，结构化的变革推进每月可为单个部门节省多达2000小时工时，并带来显著的人力效能提升与全职人力重新调配。 在推动企业落地的策略上，管理层正从温和引导转向制度化考核。Evernham提到，亚马逊Ring业务部已将证明使用AI提升运营效率设为员工晋升的前置条件；Howard Miller则提出了大型组织适用的“3T人才评估框架”，即系统评估员工应当接受AI技能培训（Train）、转岗至不易受AI冲击的岗位（Transfer，如清淤等线下操作），还是在无法适应时予以优化淘汰（Terminate）。此外，Kapoor建议通过设立单点冠军、游戏化激励以及在全员大会上展示成果来化解员工焦虑。 场景收敛与人机协同取代完全自主 (5:42) Kapoor回顾指出，过去企业试图用AI“解决所有问题”的宏大设想屡屡失败，过去一年真正见效的策略是将范围收敛至单点业务以实现快速成效。例如某客户支持团队历时两年引入AI，将人工坐席从150人精简至90人，其余人员被重新分配到其他业务部门。加州大学洛杉矶分校（UCLA）在30个工作日内推出两款代理，其中一款基于哈佛商学院案例的6页提示词构建，实现了让全班学生与AI员工进行15轮深度模拟互动与个性化辅导。 对于智能体（AI Agents）的成熟度，Kapoor将其划分为副驾驶（Co-pilots）、协同工作者（Co-workers）和完全自主代理（Autonomous Agents）三个阶段。专家们一致认为，当下完全自主代理尚不具备企业级交付能力，当前的核心落地点应是协同工作者。例如在客服质检中，人工团队通常仅能抽检5%的工单，而AI协同工作者可覆盖100%全量审核，使人类主管的工作重心转向精准培训与员工赋能。 人才技能分层与多步智能体的复合错误 根据Dice对美国科技用工市场的研究，40%的新增招聘岗位已要求具备AI技能。Farnsworth将人才需求划分为三类：研发核心架构的高成本“构建者”（Builders）、使用第三方平台进行集成的“编排者”（Orchestrators），以及利用工具提升自身产出的“增强型专业人员”（Enhanced Pros）。他警告IT管理者切勿陷入耗资数百万自研底座的“构建者陷阱”，而应将预算和人员重点布局在编排者与专业人员上。与此同时，Cursor与Windsurf等工具展示了极高的人均创收能力，正在重塑企业的规模定义。 展望未来12个月，Evernham预警多步自主智能体项目将面临大量搁置。他以单步99%准确率的模型计算指出，当一个自主智能体执行50个连续步骤时，单步1%的错误率会产生复合效应，导致整个工作流的失败率飙升至约40%（0.99的50次方约为60.5%成功率）。为降低风险，企业必须缩小智能体应用范畴，设立严密的步骤级监控机制、确定性护栏规则以及“LLM-as-a-judge”评估体系。 专家小组的共识表明，企业生成式AI应用正在脱离泛化的概念验证，转向注重投资回报率、组织问责与工程纪律的理性阶段；然而，传统人力资源与管理架构尚未准备好应对未来数字智能体规模匹敌人类员工的组织形态，如何在控制多步复合错误率的同时重构人机协作边界，仍是悬而未决的长期挑战。

## Background 本视频由 Ai4 频道于 2025 年 8 月 24 日发布在 YouTube 平台。视频记录了一场汇聚 TheLoops AI、Dice、Worth Media、UCLA 安德森管理学院和 Sinequa 高管的圆桌论坛，围绕企业在推进生成式 AI 和 AI Agent 规模化落地过程中面临的基础设施架构、利益相关方协同、系统集成、人才变革与治理策略展开深入探讨。 ## TL;DR 论坛直面当前 80% 企业应用生成式 AI 却未见底线利润提升、42% 项目遭放弃的严峻现实，指出成功的核心不在于盲目全速推进或“把大海煮沸”，而在于依靠自上而下的变革管理与高度聚焦的“快速见效”切入点。嘉宾们深度剖析了企业级 RAG、领域专用微模型及自主智能体的现实边界，并结合 UCLA 教学模拟及客服重组等案例，说明当前 AI 正处于从副驾驶走向协作者（Co-worker）的过渡阶段。针对多步骤 Agent 流程，专家指出单步 1% 的微小瑕疵经 50 步复合会使整体失败率飙升至 40%，因此严格的治理护栏不可或缺。最后，讨论延伸至组织人效重塑与人才分类，提出应对变革的“3T”法则以及管理者未来必须直面考核硅基智能体的生存警钟。 ## Atmosphere & style 整场研讨会充溢着清醒直率的务实主义与心照不宣的幽默风趣，彻底打破了对 AI 奇迹的盲目鼓吹。现场两次即兴举手表决直观揭示了“试水者众多，而规模化落地者寥寥”的行业落差，引得台上台下会心苦笑。从探讨“Agent 之间是否需要团建”和“完成工作掉糖果的自动出糖机”，到严肃推导长链条智能体的复合溃败率与残酷的“淘汰”现实，嘉宾间的交锋既张弛有度又充满迫切的危机感，真实还原了企业高管身处技术剧变十字路口时的警醒与决断。 ## Key takeaways - [00:04](https://www.youtube.com/watch?v=iO0UoXsejn4&t=4s) 企业级 AI 陷入落地困境：麦肯锡数据显示 80% 企业应用了 GenAI 但无显著盈利成效，标普全球更表明放弃大部分 AI 项目的企业比例激增至 42%。 - [01:18](https://www.youtube.com/watch?v=iO0UoXsejn4&t=78s) 技术本身不足以实现规模化，决定成败的核心在于自上而下的变革管理（Change Management）与 CFO 维度的硬性业务考核指标。 - [05:42](https://www.youtube.com/watch?v=iO0UoXsejn4&t=342s) 落地切忌“把大海煮沸”（Do not boil the ocean），企业应将目标锁定在特定、可快速见效（Quick Win）的高价值场景以赢取信任。 - [10:57](https://www.youtube.com/watch?v=iO0UoXsejn4&t=657s) 全自主 Agentic AI 落地受阻：目前 Agent 缺乏常识与心智，需要大量底层护栏与人工维护，尚无法实现完全端到端闭环交付。 - [14:18](https://www.youtube.com/watch?v=iO0UoXsejn4&t=858s) RAG（检索增强生成）成为解决幻觉与内部知识落地的核心标准路径，通过企业私有知识库注入实现精准溯源与事实约束。 - [17:25](https://www.youtube.com/watch?v=iO0UoXsejn4&t=1045s) 概念验证（PoC）难以直接扩展到生产环境：当数据量激增至数百万文档并接入多源系统时，检索噪音与权限安全隔离问题呈非线性放大。 - [20:06](https://www.youtube.com/watch?v=iO0UoXsejn4&t=1206s) GenAI 规模化落地极少，企业推进应避免“从 0 飙到 60”，而应采取“从 0 到 20”的渐进式验证策略并借力工业级成熟平台。 - [23:19](https://www.youtube.com/watch?v=iO0UoXsejn4&t=1399s) 智能体演进分为副驾驶（Co-pilot）、协作者（Co-worker）与自主智能体（Autonomous agent）三阶段，当前主流处于人机协同的协作者阶段。 - [25:06](https://www.youtube.com/watch?v=iO0UoXsejn4&t=1506s) 企业 AI 人才分为构建者、编排者与增强型专业人员三类；管理层需警惕工程师陷入“为做某事而自造底层框架”的过度研发陷阱。 - [32:42](https://www.youtube.com/watch?v=iO0UoXsejn4&t=1962s) 极高人效企业颠覆传统组织规模，Cursor、Glean 等小团队撬动高额 ARR，重新定义了依靠极少人力撬动成倍增长的“规模化”内涵。 - [36:20](https://www.youtube.com/watch?v=iO0UoXsejn4&t=2180s) 企业面对变革采取“Train（培训）、Transfer（转岗）、Terminate（淘汰）”3T 法则，甚至未来可能设立“AR（Agent 资源部）”来调度数字员工。 - [40:02](https://www.youtube.com/watch?v=iO0UoXsejn4&t=2402s) 智能体面临长链路的“复合错误率陷阱”：即使单步错误率仅 1%，在 50 步连续任务中复合失败率将飙升至约 40%，必须依赖严密治理与护栏机制。 - [42:55](https://www.youtube.com/watch?v=iO0UoXsejn4&t=2575s) 管理者面临新型混编考核体系，不仅要管理人类下属更需监督考核 AI Agent，缺乏动手实践与 Agent 管理能力的领导者将加速被淘汰。 ## Detailed walkthrough ### [00:04](https://www.youtube.com/watch?v=iO0UoXsejn4&t=4s) 研讨会开场：企业级 AI 繁荣表象下的落地困境与数据反差 会议由 Worth Media 主编 Dan Costa 主持开场。他直截了当地指出，尽管整个峰会洋溢着对人工智能（AI）的极高热情，但现实商业环境中的数据却敲响了警钟。Dan Costa 引用了当天《纽约时报》（New York Times）关于企业级 AI 的报道，并列出了两组引人深思的核心数据：其一是麦肯锡（McKinsey & Company）的最新研究显示，受访企业中每 10 家就有 8 家（80%）声称已经在业务中大规模应用了生成式 AI（Generative AI），然而同样比例的企业却表示，AI 对其企业利润底线（Bottom Line）“尚未产生任何显著影响”；其二是标普全球（S&P Global）对 1,000 家公司进行的调研，数据显示截至 2024 年底，彻底放弃大部分 AI 项目的企业比例已经从 2023 年的 17% 飙升到了 42%。 这两项事实表明，在企业内部规模化扩展 AI 面临着巨大挑战。Dan Costa 借此向台上的嘉宾们抛出核心问题：企业究竟该采取怎样的策略去切入和推进 AI，才能跨越技术落地的鸿沟？ - 麦肯锡报告：80% 的企业大规模应用生成式 AI，但 80% 表示对底线盈利“无显著成效”。 - 标普全球数据：2024 年底企业放弃多数 AI 项目的比例激增至 42%（2023 年仅为 17%）。 ### [01:18](https://www.youtube.com/watch?v=iO0UoXsejn4&t=78s) 组织变革管理与“自上而下”的高管意志 TheLoops AI 联合创始人兼 CEO Somya Kapoor 首先分享了她的实战洞察。TheLoops 最初作为一家面向企业提供客户体验（CX）与服务智能解决方案的初创公司，后来被工业 AI 解决方案巨头 IFS 正式收购。Somya 强调，技术本身绝不足以把企业带到成功规模化的彼岸；过去两年里，市面上所有的组织都在盲目试验各种新技术，但决定成败的根本分水岭在于**配套的变革管理流程（Change Management Process）**。现在各类初创公司和云基础设施超大规模厂商（Hyperscalers）都在高谈阔论 AI 能做什么，但如果企业领导层没有明确由业务成果驱动的心态，也没有自上而下推动落地，项目就注定搁浅。 在 Somya 看来，那些真正跑通的成功案例，往往有高层甚至 CFO 的明确考核指令，能够拿出诸如“每月节省 2,000 小时工时”或“削减特定全职人力（FTE）”这类硬性业务指标。随后其他嘉宾也高度赞同，指出企业推进 AI 必须具备“高层定调”（Tone at the top），组织必须在制度层面上将其确立为核心战略。同时，最佳实践是在各个业务部门内寻找愿意担当先锋的领头人（Champions），用局部的早期成功向其他团队证明价值，从而激发全员拥抱变革的主动性。 ### [03:29](https://www.youtube.com/watch?v=iO0UoXsejn4&t=209s) 历史镜像与认知冲突：IT 与业务部门边界的消融 拥有 35 年工程与 IT 管理经验的资深从业者（历任 CIO 和 CTO）向台下发起互动，询问听众属于 IT 侧还是业务侧。他回忆道，当前大模型引发的狂热像极了当年面向对象数据库（Object-Oriented Databases）问世时的场景——当时供应商纷纷宣称这项技术将颠覆企业架构，“今后你们再也不需要数据库管理员（DBA）了，一切就像变魔术一样，只要找几个 C++ 程序员就能把数据搞定”。然而几十年过去，事实证明工程平台的承诺往往言过其实，最终决胜的关键依然是极其繁复的落地实施与变革管理。 现场嘉宾对此展开了进一步剖析： 1. **角色界限模糊化**：有嘉宾指出，现在的 AI 工具已经强大到让业务人员无需事事依赖 IT 部门，传统 IT 与传统业务职能之间的界线正在迅速缩小。 2. **技术的复杂性与不可预测性**：供应商常常对客户粉饰太平，宣称“这很简单，开箱即用”。但事实上生成式 AI 具有底层不可预测性（Unpredictability），其失效和报错的机制与传统确定性软件截然不同，极易引发混乱。 3. **咨询业成为当前赢家**：Dan Costa 顺势幽默地补充道，《纽约时报》文章揭示的一个现实是，目前在这场淘金热中最赚钱的反而是管理咨询顾问，因为大家都在花大钱请他们来帮企业弥合业务与技术之间的巨大鸿沟。 ```mermaid flowchart LR accTitle: 传统技术落地与当前 AI 变革管理的对比 subgraph S1["传统交付模式 (过往技术承诺)"] direction TB A1["工程供应商宣传<br/>'开箱即用/无需专业人员'"] --> B1["业务团队依赖 IT 落地"] B1 --> C1["落地受挫 / 预期落空"] end subgraph S2["有效扩展模式 (现代 AI 落地实践)"] direction TB A2["高层定调与 CFO 业务考核"] --> B2["变革管理 + 业务IT边界融合"] B2 --> C2["高度聚焦单一场景取得快赢"] end ``` ### [05:42](https://www.youtube.com/watch?v=iO0UoXsejn4&t=342s) 拒绝“把大海煮沸”：从客服坐席优化到商学院角色扮演模拟 在探讨最近 12 个月取得成功的具体案例时，Somya Kapoor 提出了企业必须恪守的核心准则：**切忌试图用 AI 解决一切问题（Do not boil the ocean）**。回顾三年前，很多企业刚接触 AI 时甚至幻想用它来“解决世界饥饿问题”（Solve world hunger），各部门列出长串的宏大愿景，销售人员也过度承诺，结果由于期望过高而大面积失败。TheLoops 总结出的成功法则就是将目标锁定在一个“快速见效（Quick Win）”的具体业务场景上，以此换取组织内部各利益相关方的信任与支持。 她分享了一个令人瞩目的客服案例：某客户的支持部门负责人在两年时间内，借助引入的 AI Agent，将部门内的人类客服坐席（Human Agents）从 150 人精简到了 90 人。这位负责人自嘲为“Agent 终结者”（Bottometer / Agent Terminator），立志实现全流程自动化。值得注意的是，这并不是简单的裁员，而是将富余的人力重新调配到其他更高价值的业务线中，实现了显著的人效飞跃。 UCLA 安德森管理学院 CIO Howard Miller 对“快速见效”策略表示完全认同，透露他们首批上线的两个 AI Agent 从立项到交付仅耗时 30 个工作日。他以学术界的实际场景为例，讲述了一堂广受欢迎的哈佛商学院案例课（Harvard Business Case）的变革： - **原有教学模式**：课程围绕一段职场冲突展开模拟——一名员工业绩极其出众、为公司创造了海量收入，但人际关系极差、团队无人愿意与其共事，学生需模拟扮演经理与员工的谈话。由于时间有限，一堂课通常只能让 4 到 6 名学生上台演示，其余全班同学只能坐在台下旁观。 - **AI 赋能方案**：任课教授撰写了一份长达 6 页的系统提示词（6-page system prompt），成功构建出具备高度真实情绪与性格特征的 AI 模拟对象，让全班所有学生都能同时进行深度的 1 对 1 角色扮演实战模拟。 ### [10:12](https://www.youtube.com/watch?v=iO0UoXsejn4&t=612s) UCLA 管理沟通模拟 Agent 的成功实践 Howard Miller 继续分享了加州大学洛杉矶分校（UCLA）安德森管理学院在教学场景中落地 AI Agent 的成功案例。此前，学生们在模拟给员工提供反馈的练习中，大多数人只能在一旁旁观，无法切身体验模拟演练的实际价值。为了解决这一痛点，学院推出了一款专用的 AI Agent。 该课程虽然只有短短两周时间，但每个学生都得以扮演管理者的角色，通过约 15 个步骤的迭代流程，与模拟员工的 Agent 进行深入沟通与辅导演练。整个测试过程中，系统表现极其稳健，没有被任何学生“玩坏”（break the agent）。演练结束后，系统还会为每位学生生成高度个性化、针对其沟通与辅导表现的详细反馈。在短短两周内，该 Agent 产生的对话量和访问点击量，就已经达到了此前运行了约 9 个月的另一款 Agent 总量的一半左右，展现出了极高的学生参与度和亲和力。 - **教学形式转变**：从过去的少数人演练、多数人旁观，转变为每位学生都能独立参与的全员高频实战。 - **演练流程与反馈**：包含约 15 步的交互闭环，并在最后提供细致的个性化教练反馈（individualized & personalized feedback）。 - **数据指标**：两周内的使用量和对话总数达到了以往运行 9 个月 Agent 历史数据的一半。 --- ### [10:57](https://www.youtube.com/watch?v=iO0UoXsejn4&t=657s) 生成式 AI 与 Agentic AI 的现实边界：数字分身与销售场景挫折 Paul Farnsworth（Dice 总裁）对 Howard 的观点表示赞同，并深入剖析了企业在生成式 AI 与真正自主的 Agentic AI 之间的差异。目前 Dice 最成功的生成式 AI 应用主要集中在“研究、检索数据与对话交互”层面。产品和市场团队通过构建数字分身（Digital Twins）和用户画像（Personas），极大地加速了前期调研并激发了团队的高质量思考。 然而，在谈及“不那么顺利”的尝试时，Paul 指出让 AI 完全自主执行任务（Agentic AI）在落地中遇到了不少现实阻碍： ```mermaid flowchart TD accTitle: 知识赋能型 AI 与自主 Agentic AI 对比 subgraph KnowledgeRetrieval["当前成熟方案：知识检索赋能"] A["人类员工"] --> B["企业内部知识检索 / 数字分身"] B --> C["辅助决策与思考加速"] end subgraph AgenticAI["探索中受阻：全自主 Agentic AI"] D["自主 AI Agent (如 SDR 销售外呼)"] --> E["需像人类一样购买席位 License"] E --> F["缺乏常识需大量人工介入 (Care & Feeding)"] F --> G["后置工作流护栏重，难以直接端到端闭环"] end ``` 软件供应商往往将 Agentic AI 视同人类员工并收取“席位许可费”（Seat License），企业也必须像培训员工一样去训练它。但目前的 Agent 还不具备人类成熟的心智，需要大量底层工作流护栏（Guardrails）以及持续的人工维护与干预（Care and Feeding）。企业理想中“晚上给 Agent 指派任务、第二天早上直接验收成果”的状态尚未到来。Dice 在销售端的 SDR（业务开发代表）早期调研和销售触达上进行了 Agent 自动化尝试，但目前的表现依然达不到预期。 --- ### [12:47](https://www.youtube.com/watch?v=iO0UoXsejn4&t=767s) Sinequa 的企业级 RAG 方案与 PoC 内部“反抗”轶事 Jeff Evernham（Sinequa 首席产品官）分享了 Sinequa 从企业级搜索转型至大语言模型（LLM）、检索增强生成（RAG）和助手 Agent 的经验。他提到一个典型的客户 PoC（概念验证）案例：该客户深度使用 OpenAI 技术并在内部部署了大量工具，但核心痛点在于这些聊天机器人无法基于企业内部内容进行智能、准确的回答。 Sinequa 为该客户搭建了基于企业私有知识库的检索增强方案。随后，由于该客户还必须按照流程测试另一家供应商的竞品 PoC，便暂时关闭了 Sinequa 的系统并要求员工换用新工具。结果引发了内部员工的强烈不满与“反抗”（internal revolt）。员工抱怨道：“我已经离不开这个系统来完成日常工作了！”员工之所以对 Sinequa 的系统产生极高黏性，是因为它能迅速且准确地解答多层次、极其复杂的工程技术难题。 --- ### [14:18](https://www.youtube.com/watch?v=iO0UoXsejn4&t=858s) 为什么 RAG 是企业级知识落地的核心路径 在主持人 Dan Costa 确认现场绝大多数观众都已知晓 RAG 概念后，Jeff 深入阐述了 RAG 为何能成为解决大模型“幻觉”和企业内部知识鸿沟的标准解法。 ```mermaid flowchart LR accTitle: 企业级 RAG 技术核心流程 subgraph Retrieval["1. 检索阶段"] Q["用户提问"] --> S["企业内部搜索引擎 / 知识库"] S --> D["检索出高相关性事实文档"] end subgraph Generation["2. 提示词注入与生成阶段"] D --> P["系统 Prompt: 仅依据所给内容回答"] Q --> P P --> LLM["LLM (ChatGPT / Gemini / Claude)"] LLM --> Out["精准业务回答 + 可溯源引用 (Citations)"] end ``` 通用 LLM 是基于公开互联网数据训练的，虽然通晓世界知识，但对企业的内部业务架构和私有文档一无所知。RAG 通过以下机制彻底改变了这一现状： 1. **精准事实注入**：在向模型提问前，先在企业内部检索出与主题相关的真实数据并注入提示词（Prompt），指令大模型仅依据提供的数据作答。 2. **极大抑制幻觉**：将模型的注意力牢牢锁定在检索出的上下文范围内，避免模型自由发挥或捏造事实。 3. **可追溯的引用来源（Citations）**：生成的每一条结论都可以明确关联到企业内部的具体文档出处，这是纯模型训练或 Fine-tuning 难以达到的效果。 --- ### [16:14](https://www.youtube.com/watch?v=iO0UoXsejn4&t=974s) 领域专用微模型（Micro LLM）与不同技术路线的博弈 针对 Jeff 强调的 RAG 路线，Howard Miller 提出了学院采取的另一种路径：完全基于自有数据在内部进行模型训练或限定，确保系统在不向外探索未知互联网数据的前提下保证高置信度与零外部幻觉，稳步建立起可信赖的引用与输出。 Somya Kapoor（TheLoops AI CEO）补充道，当前业内存在多种架构路线。除了通用模型挂载 RAG，许多企业选择构建领域专用模型（Domain Specific Models）或微模型（Micro LLMs）。这些轻量化模型专注于企业自身的垂直数据，不依赖外部庞大的通用 LLM 能力，再结合强化学习（RL）模型持续学习用户输入行为进行微调与迭代。各种路线都在不同细分场景中取得了成功，关键在于明确业务目标并选择适配的解法。 --- ### [17:25](https://www.youtube.com/watch?v=iO0UoXsejn4&t=1045s) 突破 PoC 陷阱：从 Ultrasound AI 看小规模测试到规模化落地的挑战 主持人 Dan Costa 抛出了企业落地的核心痛点：许多 PoC 在有限数据集上表现完美，获得了管理层的预算支持，但一旦全面铺开并试图规模化（Scale up）时却往往遭遇失败。 Paul Farnsworth 以他担任董事的医疗初创公司 **Ultrasound AI** 为例进行了解析。该公司的核心产品是通过算法更精准地预测婴儿出生日期，从而有效预防早产风险。该项目之所以能顺利跨越工程障碍，主要得益于两点： - **模型复杂度向上抽象**：随着底层基础模型能力的不断商品化（Commoditized），团队无需在底层通用工程上耗费过多精力，而是将重心上移至业务结果本身。 - **极度精准的目标与窄数据集**：该模型并不追求做全能的 AGI，而是专注于针对特定预测目标的极小、极精准的高质量数据集，因此准确率极高。 最后，Jeff Evernham 总结了 PoC 在从小规模扩展到大规模企业级生产环境时必然面临的技术断层： - **数据量与系统复杂度的非线性激增**：在 100 到 1,000 份文档的小数据集下，检索容错率很高；但当接入 20 到 30 个异构业务系统、面对数百万份文档时，“垃圾进，垃圾出”（Garbage in, garbage out）的问题会被急剧放大。 - **权限与安全隔离**：大规模扩展必须面对多源系统的数据权限控制与合规治理，这是简单 PoC 阶段难以暴露的关键难题。 ### [20:06](https://www.youtube.com/watch?v=iO0UoXsejn4&t=1206s) 规模化部署生成式 AI 的现实困境与策略 Jeff Evernham 指出，企业在部署生成式 AI 时必须根据场景选择方案。如果某些用例只需要处理一千份文档，那么采用轻量级、简单的方案即可；但如果目标是大规模推广（Scale up），背后就必须有工业级（Industrial strength）的基础设施支撑。他建议企业领导者深入了解底层逻辑，若最终目标是规模化落地，与其完全自己从头研发，不如积极引入成熟的基础设施。Jeff 还分享了前一天分论坛上的真实场景：当演讲者询问台下“有多少人已经在组织内部署了生成式 AI 解决方案”时，大约有 30 到 35 个人举手；但紧接着问“有多少人实现了规模化部署”时，所有举起的手全部放下了。 在主持人的提议下，Jeff 当场对在座听众进行了同样的互动调查：询问有多少人在生产环境中有正在运行的生成式 AI 系统，接着再问有多少人认为自己的系统达到了“规模化部署”。现场依然只有寥寥几只手保持举起。这一现象清晰地反映了当前的行业采纳现状——市场炒作远超实际落地水平，生成式 AI 的规模化部署难度极大。随后有嘉宾补充强调，规模化必须是企业量力而行的审慎决策。大家常说要“从 0 迈向 60”，但在实际操作中，最好先尝试“从 0 到 20”，采取渐进、可衡量的节奏稳步推进，切忌盲目全速前进，否则很难复制在低速实验阶段取得的成功。 ```mermaid flowchart TD accTitle: 落地 GenAI 的渐进式规模化路径 subgraph S1["实验与验证阶段"] A["小规模场景（如 1000 份文档）"] --> B["从 0 到 20 验证业务价值与可行性"] end subgraph S2["规模化扩展阶段"] C["工业级基础设施支持"] --> D["从 20 迈向 60 引入成熟厂商平台"] end B --> C ``` - **炒作与现实差距**：大多数企业均处于早期试点或生产小规模运行阶段，真正实现全组织规模化（At scale）部署的企业极少。 - **渐进推进策略**：避免一上来就期望全速狂飙（0 到 60），应采取分阶段验证（如 0 到 20）的节奏，降低试错与落地风险。 --- ### [22:01](https://www.youtube.com/watch?v=iO0UoXsejn4&t=1321s) 自研与采购平台工具的权衡取舍 Somya Kapoor 分析了过去两到三年的行业演变。前几年整个行业基本处于自研实验阶段，这种尝试非常有价值，因为它让团队亲身体会到这项技术的运作机制以及它为何具备颠覆性。然而，当前市场已经涌现出大量提供成熟平台的厂商与初创公司，这些平台内置了完善的安全层、模型集成以及横向扩展能力。作为 IT 负责人或部门主管，应该积极评估这些外部平台，借力现有成熟生态，因为其他客户也在这些平台上共同经历并解决了规模化扩展的痛点。 Somya 总结道，在企业内部搞自研 Demo 相对容易，但要将自研方案扩展到企业级规模则极为艰难。因此，企业在内部完成价值验证后，若想真正落地端到端方案，应该积极考察各类平台工具与初创企业所提供的方案，从而更高效地拓展业务覆盖面。 --- ### [23:19](https://www.youtube.com/watch?v=iO0UoXsejn4&t=1399s) 智能体（Agentic AI）的三阶段演进：从副驾驶到自主智能体 针对主持人提出关于“智能体时代（Agentic age）企业盲目贪多、同时运行过多 Agent 的挑战”，Somya Kapoor 提出了明确的演进范式划分：**副驾驶（Co-pilot）**、**协作者（Co-worker）** 与 **自主智能体（Autonomous agent）**。她直言，任何宣称目前已经实现完全自主智能体（Autonomous agent）的说法都是不切实际的，因为目前企业既缺乏足够精准的数据支撑，也没有适宜的环境。 ```mermaid flowchart LR accTitle: 智能体（Agentic AI）三阶段演进模型 Stage1["1. 副驾驶 (Co-pilot)<br>提示词交互/辅助建议"] --> Stage2["2. 协作者 (Co-worker)<br>当前阶段: 人机协作/全量质检"] Stage2 --> Stage3["3. 自主智能体 (Autonomous Agent)<br>未来 1-2 年: 复杂自主决策"] ``` 目前市场上如 Cursor 等工具在辅助编程方面表现出色，但在企业端仍无法做到全自动直接上线部署；在法律场景下也以辅助为主。当前智能体正处于“协作者”阶段：它辅助人类工作并在互动中持续学习进化，预计在未来 1 到 2 年内才会逐步发展为更高程度的自主智能体。Somya 举了客户支持质检（QA）的例子：过去人工只能抽检 5% 的工单，而如今在智能体协作者的帮助下可以做到 100% 全量质检，这使得团队能够把精力转向针对性的员工辅导和赋能，而不是把时间浪费在翻找抽样工单上。她强调，在复杂的企业级场景下，真正的自主智能体时代尚未到来。 --- ### [25:06](https://www.youtube.com/watch?v=iO0UoXsejn4&t=1506s) AI 人才画像的三大分类与避免过度自研框架 Dice 总裁 Paul Farnsworth 首先幽默地澄清，Dice 是一家技术人才招聘平台而非博彩网站（在拉斯维加斯参会容易引起误解）。他分享了 Dice 针对招聘市场的最新统计数据：目前全美 40% 的新增招聘职位都要求具备某种 AI 技能。Dice 将这些 AI 人才细分为三大核心群体： 1. **构建者（Builders）**：底层模型与核心架构开发人员，常能拿到顶薪甚至数亿美元开价的顶尖人才（Paul 开玩笑称若现场有开出这种待遇的雇主他随时可去入职）。 2. **编排者（Orchestrators）**：熟练利用供应商与厂商工具将各组件连接、集成落地的技术专家。 3. **增强型专业人员（Enhanced Pros）**：在日常岗位中熟练使用 Cursor 等 AI 工具来成倍提升个人生产力的业务或技术人员。 ```mermaid flowchart TD accTitle: 企业 AI 人才三大阵营 subgraph Talent["企业 AI 人才三层结构"] T1["构建者 (Builders)<br>底层模型研发/顶级稀缺人才"] T2["编排者 (Orchestrators)<br>善用厂商工具/系统集成与编排"] T3["增强型专业人员 (Enhanced Pros)<br>借助 Cursor 等工具提升本职效能"] end ``` 结合自己多年领导工程团队的经验，Paul 提出了一个深刻的行业警示：很多工程师极易陷入“为了做某件事而去造一个用来做这件事的框架（Build the thing that builds the thing）”的陷阱。这种自造框架的做法往往消耗了巨额预算和漫长周期，最终却无法交付真实的业务价值。对于大型 IT 组织而言，管理层必须看清团队构成——明确自己究竟拥有多少构建者、编排者还是增强型专业人员，从而决定把时间和预算投向何处，切忌在缺乏商业回报的底层过度研发中浪费资源。 - **市场数据**：全美 40% 的新增岗位已将 AI 技能列为硬性或优先要求。 - **管理反思**：警惕团队热衷于自研框架（Framework）而忽视实际业务成效的倾向。 --- ### [27:32](https://www.youtube.com/watch?v=iO0UoXsejn4&t=1652s) Agent 协作实践与组织变革管理中的“回头客”挑战 Howard Miller 结合自身机构的探索指出，团队的目标是实现可扩展的 AI 智能体，但目前仍处于早期阶段。他谈到目前已经在运行一个 Agent 调用另一个 Agent 的基础链路，虽然这并不算最终的“圣杯”，但依然遵循了“从小处着手（Start small）”的原则——先确保 Agent 之间能够稳定交互并产出预期结果，再逐步向外扩展。现场嘉宾打趣询问这两个 Agent 之间是否需要搞搞“团建”，Howard 笑称它们之间是非常事务性（Transactional）、非人类的协作关系。 针对如何推动组织内部的落地与文化变革，Jeff Evernham 探讨了员工在面对新技术时的心理障碍。组织内总有一部分人因为未知而产生恐惧心理，管理者不能仅靠自上而下的行政命令，更需要发掘和培养内部的“推广先锋（Champions）”去主动示范具体使用案例、鼓励探索并建立安全感。Jeff 强调，衡量推广成功与否的核心是“用户是否会再次使用（Do they come back?）”；只要打造出来的工具能切实减轻工作负担、创造实际业务价值，员工自然会主动将其融入日常工作流程中。 ### [30:08](https://www.youtube.com/watch?v=iO0UoXsejn4&t=1808s) 变革管理的阵痛：全员技术素养与弥合数字鸿沟 面对员工普遍担忧的“AI 是否会让我失业”这一敏感话题，嘉宾们展开了关于企业变革管理的深度讨论。在大企业工作多年的管理者指出，规模化背后的真相往往是“隐藏的效率”（hidden efficiency at scale）——流程变得更简单、更快速，但不可避免地会导致某些职能被取代。过去企业习惯对所有新员工进行财务知识培训，而如今大家极力主张全员都应接受技术培训。企业需要做的是识别出那些能够拥抱并适应这种变革的员工，并加大对这一群体的投入。 与会者提到，AI 的快速推进必然会加剧“数字鸿沟”（digital divide），因为很多人习惯于固守舒适区。为了化解阻力，许多团队并没有将 AI 宣传为“岗位替代者”，而是将其包装为“工作助手”。通过寻找内部推动者（champions）并为员工个性化定制 AI 场景，能极大提升接受度。例如，为员工推出自主构建个人智能体（Personal Agent）的功能：当员工发现只需将 Agent 指向他们每天都要查阅的某个特定网站，便能直接向 AI 提问并获取精准信息时，其“顿悟时刻”（light bulb moment）便会降临。只要让员工直观看到技术为个人带来的实际价值，变革落地就会顺畅得多。 ```mermaid flowchart TD accTitle: AI 变革管理与个人智能体驱动路径 subgraph MindsetChange["理念与角色重塑"] A["传统认知：岗位替代恐慌"] --> B["新定位：AI 个人助手 / 赋能工具"] C["全员技术素养培训"] --> D["识别并扶持内部变革倡导者 (Champions)"] end subgraph PersonalAgentAdoption["落地实施：个人智能体"] D --> E["定制个人智能体 (Personal Agents)"] E --> F["挂载专属工作流/常访站点"] F --> G["激发顿悟时刻 (Light Bulb Moment) 发现个人价值"] end ``` --- ### [32:42](https://www.youtube.com/watch?v=iO0UoXsejn4&t=1962s) 颠覆性数据与下一代工作方式：从极简人员规模到高额营收 技术落地的推进速度远超大多数人的想象。嘉宾引用了风险投资与科技圈当下极具前瞻性的讨论：亿万富翁群体和 Sam Altman 都在公开探讨，有生之年必将见证“仅凭 10 人甚至单人运作便达成 10 亿美元营收的公司”。市场上已经涌现出一批极高人效的明星企业： - **Cursor**：仅有 150 名员工，年度经常性收入（ARR）已达到 3 亿美元； - **Windsurf (Codeium)**：以 200 名员工达成 7500 万美元规模； - **Glean**：以 600 名员工实现 3 亿美元 ARR； - **Harvey**：作为法律软件领域的颠覆者，成立仅三年便直奔 5 亿美元营收目标。 现场还分享了一个生动的家庭趣事：一名 10 岁的女儿在被要求阅读一本书后，仅用了 10 分钟便交出了一份完美的书本摘要。母亲追问之下，孩子坦白是用 ChatGPT 抓取了书本内容并生成的提示词（Prompt）。这引发了对下一代工作方式的深思：过去习惯的考核问题（如“读完并写摘要”）在 AI 时代已经失效；如果改为提问“第 42 页哪一段对你触动最深”，AI 就无法简单代劳。作为领导者，不能仅停留在观望或指出 AI 缺陷的层面，而必须亲自下场使用。落地的优秀企业往往通过“游戏化竞赛”（gamification）、在全员大会上表彰创新者等方式来推动转型，因为正如硅谷户外广告牌所写：“我们不会取代你，但使用我们的竞争对手会取代你。” --- ### [36:20](https://www.youtube.com/watch?v=iO0UoXsejn4&t=2180s) 变革的硬核手段：“3T”法则与规模化新定义 当温和的引导无法满足节奏时，部分企业开始采取更强硬的管理手段。例如亚马逊（Amazon）旗下的 Ring 部门出台了晋升硬性指标：员工必须证明自己如何利用 AI 提升了业务运营效率；对于管理层，这一考核标准更为严苛，必须证明如何用 AI 做到“降本增效（doing more with less）”才能获得升职。 在大型传统企业中，则存在更为残酷的“3T”管理法则： - **Train（培训）**：通过技术培训使员工掌握新技能并转型； - **Transfer（转岗）**：将无法转型的员工调配到目前不会被 AI 颠覆的传统物理岗位（例如河道清淤部门）； - **Terminate（解雇/淘汰）**：在既无法培训也无法转岗的情况下，必须做出人员置换的艰难抉择。 这重新定义了企业维度的“规模化（Scale）”：规模化不再仅仅意味着企业员工数量和组织架构的庞大扩张，而是指用极少的人员（如 10 人）去撬动成倍增长的客户群与营收。 ```mermaid flowchart LR accTitle: 应对 AI 冲击的员工管理 3T 法则 A["组织内员工技能评估"] --> B{"转型适配判断"} B -->|"有学习意愿与潜力"| C["Train (培训提升)"] B -->|"无法适配但有其他需求"| D["Transfer (转岗至不易被替代部门)"] B -->|"无法培训亦无法调岗"| E["Terminate (人才汰换)"] ``` --- ### [38:02](https://www.youtube.com/watch?v=iO0UoXsejn4&t=2282s) AI 倡议的发起阵地与“AR 部门”的雏形 关于 AI 创新在组织内部究竟应该由谁主导（IT 部门、业务部门还是 HR/财务），嘉宾们探讨了不同路径的利弊。现实中通常存在两条主线：一条是技术与工程领袖驱动的创新，他们从外部汲取最新工具与架构带回组织内部；另一条是业务部门由外向内的采购驱动，因急于获取数据洞察或降本增效而被各类软件供应商介入，这种方式往往带来更痛苦的组织磨合与混乱。 值得注意的是，传统的 HR 部门尚未完全意识到一个即将到来的现实：企业内部运行的 **AI 智能体（Agent）数量未来可能会等同甚至超过人类员工的总数**。这引发了一个前瞻性的管理难题——未来究竟由谁来管理、调度与考核这些数字员工？现场幽默地提议，企业或许应该设立专门的 **“AR 部门”（Agent Resources，智能体资源部）** 来取代或补充传统的 HR（Human Resources）职能。 在讨论的尾声，主持人提醒对谈还剩约 5 分钟，并邀请各位嘉宾对下一年度的企业级 AI 落地格局做出预测，同时鼓励将未尽的“AI 治理（Governance）”议题融入各自的总结展望之中。 ### [40:02](https://www.youtube.com/watch?v=iO0UoXsejn4&t=2402s) 未来 12 个月的展望与智能体（Agent）的复合错误率陷阱 在讨论的尾声，主持人将焦点引向未来，询问嘉宾们对未来 12 个月企业内部 AI 落地趋势的研判。Jeff Evernham 指出，生成式 AI 无疑是一场颠覆性的技术变革，将深刻重塑企业界乃至个人生活的各个方面，但真正充满不确定性的是落地的“时间表”。他直言当前市场充满了狂热的营销炒作，几乎所有人都在空谈“智能体”（AI Agents）。值得玩味的是，整场长达 40 分钟的圆桌讨论中，大家甚至没有刻意去给智能体下一个固定的技术定义，因为业界每个人心中都有不同的标准。在他看来，智能体的本质在于具备自主行动能力并能独立做出决策，但从实际业务来看，目前能真正实现这一点的场景依然寥寥无几。 ```mermaid flowchart TD accTitle: 智能体多步骤执行下的错误率复合放大效应 subgraph S1["单步执行表现"] A["单步极低错误率 (1%)<br/>表面成功率高达 99%"] end subgraph S2["长链路智能体流程"] B["连续执行 50 个操作步骤<br/>(0.99 的 50 次方)"] end subgraph S3["最终业务结果"] C["整体失败率飙升至约 40%<br/>项目面临被搁置放弃的风险"] end A --> B --> C ``` 随着企业蜂拥尝试部署智能体，底层大语言模型（LLM）固有的错误率很可能导致大量项目以失败告终，进一步推高已被放弃的 AI 项目统计数据。Jeff Evernham 强调，失败的主因往往不是技术过早，而是业务目标过宽或企图一步登天；相比试图包揽一切的宏大智能体，专注狭窄领域的专用智能体成功率要高得多。为了证明长链条任务的致命弱点，他算了一笔令人警醒的数学账：即使某些领先模型（如近期宣传的评测结果）声称错误率低至 1%（即 99% 的准确率），一旦放入一个需要执行 50 个连续步骤的复杂智能体工作流中，这 1% 的微小瑕疵经多轮复合后，将使整个任务的失败率飙升到 40% 左右，在商业应用中这是“极其糟糕且不可接受的”（abysmal）。因此，未来一到两年的核心重心必须落在治理（Governance）与护栏（Guardrails）机制上，通过细化步骤可见性、规则度量指标以及“LLM as a Judge”（大模型作为裁判）来分步验证每一步输出。 - **炒作与现实差距**：市场都在谈论具备自主行动与决策能力的智能体，但生产环境中真正落地并交付稳定成果的案例依然极其罕见。 - **范围界定法则**：窄领域、高聚焦的垂直智能体远比试图面面俱到的大型复合型智能体更容易成功。 - **复合错误率陷阱**：在 50 步的长流程任务中，哪怕单步只有 1% 的失误率，复利效应也会让整个智能体执行出现近 40% 的灾难性溃败。 - **工程与治理手段**：必须建立多层护栏与精细化可观测性，结合规则指标与大模型裁判，对智能体的每一个执行中间态进行分步评估。 --- ### [42:55](https://www.youtube.com/watch?v=iO0UoXsejn4&t=2575s) 快速收尾：从自动出糖机到人机混编的组织重塑 在最后的 60 秒极速观点陈述环节，嘉宾们以幽默与前瞻并存的视角描绘了接下来的演进轨迹。一位嘉宾打趣道，未来一年我们或许会在电脑旁看到实体“糖果分发机”——当人类把活干得漂亮时，AI 会主动掉落糖果来奖励人类，引得全场发笑，同僚也随之打趣“希望掉出来的只是纯糖分”。回归到严肃的治理层面，嘉宾建议企业务必抓紧夯实底层数据基础，并在跨职能部门间快速组建有贡献意愿的先锋工作组（Task Force）。现场举手表决也生动揭示了现状：声称引入生成式 AI 的企业很多，但真正做到规模化部署（at scale）的却屈指可数；预计再过一年，业界对于 Agentic AI 的定义将更加清晰统一，落地进程也会向前推进，但绝不会发生脱离现实的一蹴而就。 Somya Kapoor 则给出了更为激进且紧迫的组织变革预判。她指出，未来 12 个月内企业组织的规模必然会进一步缩减，岗位职责说明书（Job Description）将被彻底打破，员工将被迫承担远超传统界限的工作内容。对于各层级的管理者而言，一场剧变即将来临：领导者不仅要考核人类下属的绩效，更将被要求直接管理与评估 AI 智能体（AI Agents）的工作成果，形成新型的人机协同与监督机制。她最后发出警告：那些至今仍拒绝动手实验 AI、搞不懂 AI 管理逻辑的各层级管理者，在接下来的商业优胜劣汰中将会极快被市场淘汰。在这一充满警示意味的高潮中，主持人宣布论坛圆满结束，并引导全场为嘉宾的精彩分享热烈鼓掌。 - **数据与组织治理**：企业推进 AI 的前提是保障数据质量，并由跨业务条线的自发拥护者组成专项行动小组，推动试点走向规模化。 - **组织结构精简**：企业人员编制将进一步收缩，个体职责范围随着工具赋能而大幅泛化扩展。 - **管理范式升级**：管理者即将面对“人类团队与 AI 智能体”混编的考核体系，同时管理碳基员工与硅基智能体。 - **管理者的生存危机**：缺乏 AI 实践经验、不具备智能体监督能力的管理人员，将在未来一年内面临加速被裁汰的职场风险。 ## Notable quotes > [05:42](https://www.youtube.com/watch?v=iO0UoXsejn4&t=342s) 切忌试图用 AI 解决一切问题（Do not boil the ocean）。 — Somya Kapoor > [12:47](https://www.youtube.com/watch?v=iO0UoXsejn4&t=767s) 我已经离不开这个系统来完成日常工作了！ — Jeff Evernham > [20:06](https://www.youtube.com/watch?v=iO0UoXsejn4&t=1206s) 大家常说要‘从 0 迈向 60’，但在实际操作中，最好先尝试‘从 0 到 20’，采取渐进、可衡量的节奏稳步推进。 > [22:01](https://www.youtube.com/watch?v=iO0UoXsejn4&t=1321s) 在企业内部搞自研 Demo 相对容易，但要将自研方案扩展到企业级规模则极为艰难。 — Somya Kapoor > [25:06](https://www.youtube.com/watch?v=iO0UoXsejn4&t=1506s) 为了做某件事而去造一个用来做这件事的框架（Build the thing that builds the thing）。 — Paul Farnsworth > [27:32](https://www.youtube.com/watch?v=iO0UoXsejn4&t=1652s) 衡量推广成功与否的核心是‘用户是否会再次使用（Do they come back?）’ — Jeff Evernham > [32:42](https://www.youtube.com/watch?v=iO0UoXsejn4&t=1962s) 我们不会取代你，但使用我们的竞争对手会取代你。 > [40:02](https://www.youtube.com/watch?v=iO0UoXsejn4&t=2402s) 哪怕单步只有 1% 的失误率，复利效应也会让整个智能体执行出现近 40% 的灾难性溃败。 — Jeff Evernham > [42:55](https://www.youtube.com/watch?v=iO0UoXsejn4&t=2575s) 那些至今仍拒绝动手实验 AI、搞不懂 AI 管理逻辑的各层级管理者，在接下来的商业优胜劣汰中将会极快被市场淘汰。 — Somya Kapoor ## Mentioned Dan Costa, Somya Kapoor, Paul Farnsworth, Howard Miller, Jeff Evernham, TheLoops AI, Worth Media, Dice, UCLA Anderson School of Management, Sinequa, IFS, Ultrasound AI, Cursor, Windsurf (Codeium), Glean, Harvey, Amazon, Ring, OpenAI, ChatGPT, Gemini, Claude, McKinsey & Company, S&P Global, New York Times, Sam Altman ## Video description Somya Kapoor, CEO & Co-Founder, TheLoops AI Paul Farnsworth, President, Dice Dan Costa, Editor In Chief, Worth Media Howard Miller, CIO, UCLA Anderson School of Management, UCLA Jeff Evernham, Chief Product Officer, Sinequa Abstracts Enterprise leaders are under pressure to turn GenAI and AI agents into scalable, secure, and high-impact tools across their organizations. In this panel, CAIOs, CIOs, and IT executives will share how they are leading the charge—architecting the infrastructure, aligning stakeholders, and deploying AI agents across key business functions. Topics will include integration with legacy systems, managing data and compute resources, driving adoption, and balancing innovation with governance. Whether you're scaling early pilots or building a long-term AI strategy, this session offers real-world lessons from the leaders making it happen. [Source](https://www.youtube.com/watch?v=iO0UoXsejn4)

[5:42] Somya Kapoor: 切忌试图用 AI 解决一切问题（Do not boil the ocean）。 [12:47] Jeff Evernham: 我已经离不开这个系统来完成日常工作了！ [20:06] 大家常说要‘从 0 迈向 60’，但在实际操作中，最好先尝试‘从 0 到 20’，采取渐进、可衡量的节奏稳步推进。 [22:01] Somya Kapoor: 在企业内部搞自研 Demo 相对容易，但要将自研方案扩展到企业级规模则极为艰难。 [25:06] Paul Farnsworth: 为了做某件事而去造一个用来做这件事的框架（Build the thing that builds the thing）。 [27:32] Jeff Evernham: 衡量推广成功与否的核心是‘用户是否会再次使用（Do they come back?）’ [32:42] 我们不会取代你，但使用我们的竞争对手会取代你。 [40:02] Jeff Evernham: 哪怕单步只有 1% 的失误率，复利效应也会让整个智能体执行出现近 40% 的灾难性溃败。 [42:55] Somya Kapoor: 那些至今仍拒绝动手实验 AI、搞不懂 AI 管理逻辑的各层级管理者，在接下来的商业优胜劣汰中将会极快被市场淘汰。

[5:42] Somya Kapoor: solve world hunger with AI [1:18] Paul Farnsworth: tone at the top [0:04] Dan Costa: no significant bottom-line impact [25:06] Paul Farnsworth: Too much building is very expensive and probably won't get you what you need [33:38] Somya Kapoor: We're not here to displace people, but your competitors using us are going to. [36:20] Howard Miller: 3 Ts [36:20] Howard Miller: Train, Transfer, or Terminate [37:48] Dan Costa: AR (Agent Resources) department

[0:04] Thank you all for coming in today. Um, I [0:06] hadn't planned on an introduction. Um, [0:08] but I opened up the New York Times [0:09] today. I didn't open it up. I opened up [0:10] the page and there's a big story about [0:13] business enterprise AI and I wanted to [0:15] start by just table settings. There's a [0:17] lot of enthusiasm for AI obviously at [0:19] the conference. Um, but there are two [0:21] facts that came out in the New York [0:22] Times if you can call them facts in the [0:23] New York Times. Um a recent uh Mckenzian [0:26] company study said that eight and 10 [0:28] companies have reported using generative [0:30] AI on a massive scale. Um at the same [0:34] time, the same percentage said they've [0:36] seen quote no significant bottomline [0:38] impact from AI. [0:41] Another data point from S&P Global says [0:43] the percentage of companies abandoning [0:45] most of their AI projects has soared to [0:49] 42% at the end of 2024. That's up from [0:52] 17% in 2023. and this is a a thousand [0:55] companies that they surveyed. Um so [0:58] clearly scaling enterprise AI can be a [1:01] challenge. Um hopefully we'll provide [1:04] some insight into why that is some [1:06] solutions. Um but maybe we start off by [1:08] just um telling the audience a little [1:10] bit about your particular approach to [1:11] enterprise AI when and you know sort of [1:14] how you approach it and Som let's start [1:16] with you. [1:18] >> Hello. Hello. Perfect. Uh hi everyone. [1:21] So, so to that point uh you know um [1:24] enterprise I did offer AI solution. The [1:27] loops offered AI solution to a multitude [1:30] of enterprises when we were a startup. [1:32] We actually got bought by IFS [1:34] um an industrial AI solution and in that [1:38] in that scenario there were I I [1:41] fundamentally believe that it's just not [1:43] technology that's going to get you there [1:45] when you're scaling. Everybody to your [1:48] point was right was experimenting using [1:50] AI technology in the last 2 years. What [1:53] truly it helps to get to scale is a [1:56] change management process associated [1:58] with just the technology right uh I do [2:01] know numerous startups hyperscalers are [2:03] coming and saying yes AI can do this AI [2:05] can do that but if the mindset of the [2:08] organization and what they need to drive [2:10] from a business outcome standpoint the [2:12] leadership that's doing that doesn't put [2:14] that in place that's not going to [2:16] happen. So that's what we saw where the [2:18] successes really happened was just not [2:20] the technology coming in but the [2:23] leadership driving that change [2:24] management downstream through the whole [2:27] process of getting the adoption was the [2:29] true differentiator. They were reporting [2:32] numbers like 2,000 hours saved per month [2:35] uh to FTE and headcount savings and it [2:38] was a mandate forced as you know from a [2:40] CFO as well. So that's where we truly [2:43] saw the benefits and I'm more than happy [2:44] to relay those stories out but you know [2:47] uh we can share more up here. [2:50] >> Waiting for my delay here. There we go. [2:52] Um very similar story I think if you [2:54] don't have tone at the top if you don't [2:57] institutionally decide that this is what [2:59] you want to do as a business. You're [3:01] never going to be able to do that. And [3:02] even with that said that still doesn't [3:04] mean that the change manage compon [3:06] component of it isn't hard. Right? So [3:09] you need somebody who's going to be a [3:11] leader. We've almost taken the approach [3:12] that you find leaders in each of the [3:15] areas that want to help you champion. [3:17] We've also found that when you have [3:19] someone who's had success, they can show [3:22] people that success and that gets people [3:24] excited about wanting to do more. [3:28] >> Jeeoff, [3:29] >> um I'll I'll just start uh with a little [3:31] bit of background. So I'm actually 35 [3:34] year veteran in in in um engineering and [3:38] IT management having been a CIO and a [3:41] CTO. Uh and this is I don't know might [3:44] be worth a quick poll. How many people [3:46] are here on the IT side of the of the [3:49] organization and how many people are on [3:50] the business side? Okay. I remember when [3:54] objectoriented databases were going to [3:56] transform our enterprise architecture [3:59] and companies came in and said you won't [4:01] need DBAs anymore. It's going to be [4:04] magic. You just get C++ programmers to [4:06] get your data out. And um it is exactly [4:09] what it is. It's change management. Um [4:12] if you're on the business side, find the [4:14] IT person that put their hands up [4:16] because they're the ones you want to [4:17] talk to. uh the engineering um platform [4:21] vendors will promise a lot right now and [4:24] it's back to the results that we're [4:25] seeing. It's really about the [4:27] implementation. So if you've been [4:28] through this before um it feels very [4:31] similar to some of these other [4:33] engineering promises. Although what I'd [4:35] say to that real quick is that I think [4:37] there's becoming a more blurred line [4:38] about what used to be traditional IT and [4:41] what used to be traditional business. [4:43] And I think again AI is going to be a [4:45] way that's going to narrow that gap [4:47] because the tools are so good at this [4:48] point are going to become even better [4:50] that business folks won't need the IT [4:52] folks to be able to do some of the [4:54] things that they used to have to rely on [4:56] them for. [4:57] >> Agree. [5:00] >> I guess I would just add to that maybe [5:02] some bad news. It's complicated and it's [5:06] confusing and the vendors will tell you, [5:09] many of the vendors will tell you it's [5:11] not. It's simple. it's magic. We can [5:13] just turn it on and it will work. Um, [5:17] and that's not the case. There's a it's [5:19] it's hard to understand this new [5:21] technology, what it can do well, where [5:23] it breaks. It's, you know, it's kind of [5:26] uh unpredictable in ways that other [5:28] technology is a lot more understandable. [5:31] So, I think that's adding to the [5:32] confusion. So, I would I would agree. [5:33] It's change management. It's not it's [5:36] not fundamentally different. Um but it [5:39] does have some nuances to it that I [5:40] think are underappreciated. [5:42] >> Yeah. I think one of the takeaways from [5:44] the time story is that one of the pe one [5:46] of the sectors that is making money are [5:47] the consultants who are trying to make [5:50] sense of this and connect these divides. [5:52] Um, so let's take a moment and um and [5:54] offer some specifics in terms of uh and [5:57] I just want you to share either [5:58] something you've seen in the last 12 [6:00] months that's worked really well that [6:02] was surprising or something that uh you [6:04] saw attempt to be deployed that failed [6:06] spectacularly and then why it failed or [6:09] succeeded. Um and we can go in any [6:11] order. Whoever's got a a good example [6:12] can can jump in first. [6:16] >> Uh I I you know I'm just going to add to [6:18] what everybody's kind of saying. So the [6:20] loops offered um you know a CX uh [6:23] revolution uh capability. I can say that [6:26] now uh since we've sold it. So we had [6:28] our customer buy us out because they [6:30] loved our technology so much which was [6:32] IFS. But I think uh I my my my examples [6:36] are very preview within the CX space and [6:38] if you're seeing agentic solutions out [6:40] there and there's tons of vendors in [6:42] that space all of them are agentic in [6:44] nature. Uh what truly differentiated us [6:47] in kind of showing the value of the [6:49] platform that we had truly built out was [6:52] very fact of a customer success stories [6:55] people who have done the change [6:56] management and how quickly we were able [6:59] to come in and show customer what they [7:02] were driving down. Now the thing is [7:04] initial discussions when we had three [7:06] years ago everybody wanted to solve [7:08] world hunger with AI right uh you got in [7:11] the organization either talking to the [7:13] CIO or to the head of department [7:15] depending on where the IT deployments [7:17] were happening we were predominantly [7:19] selling to tech sector so it was very [7:21] much with the IT department um and you [7:24] know they had a list of scenarios really [7:28] kind of hounding the organization down [7:30] to say let's focus on one quick win and [7:34] that quick win will help you get a buy [7:37] in from everybody else because three [7:39] years ago AI was just magic, right? You [7:43] would come in, everybody needed to [7:44] experiment with it. Whoever got in [7:46] first, whichever salesperson could [7:48] convince the customer got their vendors [7:50] in and then soon enough they realized it [7:52] didn't work out because the expectations [7:54] were too high. So focus was the key [7:57] aspect that led to successes for us in [8:00] the last 12 months. Uh second, showing [8:02] customer stories that got those wins and [8:05] were transformation leaders. We picked [8:07] up, you know, few folks that were truly [8:09] thinking differently with AI and were [8:12] able to bring the changes. So I have a [8:14] customer within support that went from [8:18] 150 agents to 90 agents in 2 years and [8:23] human agents. [8:24] >> Human agents. [8:25] >> Now we have to now we have to separate. [8:27] >> Yeah, that's right. That's right. You're [8:28] right. human agents uh to 90 human [8:33] agents along with AI agents in the [8:37] environment and saw tremendous value. [8:39] She calls herself a bottometer. She's [8:41] like if I can become an agent [8:43] terminator, I would literally get [8:45] everybody automated. But it was a [8:47] mindset shift. Now that's truly not uh [8:50] you know saying we're doing reduction in [8:51] headcount. That was repurposing those [8:53] headcounts for other areas. So what was [8:56] a quick win within CX that she was able [8:58] to show that apart from chatbots's [9:00] deployments was were these elements and [9:03] I think finding those business [9:05] processes, those use cases and working [9:07] hand in hand with your organization to [9:10] show a quick win gets you there. If [9:12] you're going to boil the ocean, you'll [9:14] never get anywhere and you'll just set [9:15] everybody up for failure. [9:19] >> Yeah, I agree 100% with the quick wins. [9:21] In fact, the first two agents that we [9:23] launched were 30 business days from [9:25] start to finish. And it was those quick [9:27] wins that enabled us to get some instant [9:30] credibility. The use case that we've had [9:32] that's been the most wildly successful [9:34] and again I work in academia and most of [9:36] the use cases we've had have been [9:38] faculty student focused was I had a [9:41] faculty member create a six-page system [9:43] prompt. And in this class, it's a [9:45] Harvard business case class where a [9:48] student does a simulation. So one person [9:52] would be the manager, the other would be [9:54] the employee. The employees done wildly [9:56] well for the corporation and has brought [9:58] in tons of revenue, but is sort of a [10:00] people problem. Nobody likes working [10:02] with this person. So in the class, what [10:04] would happen is that students would get [10:05] up in the front and about four of them, [10:06] maybe six of them if we were lucky, [10:08] would role play. and then everybody else [10:10] would sit in the back and they would [10:12] just have to watch and they wouldn't get [10:14] the value of doing the simulation. So [10:16] now with this agent we launched and this [10:18] class was only two weeks long. Every [10:20] student got the ability to go through [10:22] the simulation as the manager giving the [10:24] employee that feedback. It was about a [10:27] 15-step iterative process. We didn't [10:29] find any way to break the agent. And at [10:31] the end they got that individualized and [10:34] personalized feedback on how well they [10:36] did in coaching that scenario. And [10:38] again, I can tell you within two weeks, [10:40] the number of conversations, the number [10:42] of hits on that agent were about half as [10:45] much as an agent that we had had running [10:47] for about nine months prior to that. So [10:49] again, it was wildly successful with [10:50] students. It was individualized, it was [10:53] personalized, and it was approachable. [10:57] >> Yeah, I'll um I'll double down on that. [10:59] Um because the I think the most [11:01] successful current use um really of [11:03] generative AI wouldn't say it's agentic [11:05] AI right now for us um is something [11:08] similar which is research retrieval data [11:10] and conversation. So um the teams have [11:13] built um digital twins and personas [11:15] particularly on the product and [11:16] marketing side and that's been [11:18] particularly effective at accelerating [11:20] some of that research and just getting [11:22] better thinking out of the team. So [11:24] that's been um particularly good. Um you [11:26] asked for an example of maybe something [11:28] that didn't go well. Yeah, I was hoping. [11:29] >> Yeah. Um, so we have we have tried um [11:33] the the agentic agent and I think [11:36] there's a difference between enabling a [11:38] human agent with knowledge retrieval and [11:41] all that information and trying to make [11:43] the computer do it itself. Um, and you [11:45] know that's been an interesting lesson. [11:47] So first of all um the our software [11:51] vendors want to treat agentic AI like [11:54] humans and license it like that. So you [11:56] need a seat license for the computer. [11:58] Um, and then you need to train the [12:00] edgent AI like a human, but it's they're [12:03] not quite human yet. So the guard rails [12:06] still require some workflow at the back. [12:08] And I, you know, I see the future where [12:11] there is enough enablement in that [12:13] technology to handle it. Um, but it's [12:16] not really ready yet. It just requires [12:18] too much care and feeding to to run on [12:20] its own. The dream would be that you set [12:22] it off on a task and you come back in [12:23] the morning and it's done and and it [12:25] tells you we have our experiences and [12:27] it's not quite there yet. The specific [12:29] use case by the way if you're like what [12:31] is he what is he using it for um was on [12:33] the sales side particularly SDR sales uh [12:37] um uh research and that sales action [12:40] early um that we you know we found it to [12:44] be a little lacking so far. [12:47] So I'll give a a very recent success but [12:50] I guess before that I want to give a [12:52] little bit of context. So I work for a [12:54] software company that started in [12:56] enterprise search and then we moved into [12:59] with LLMs into rag and assistance and [13:02] agent technology. So we help provide [13:04] assistance and agents that run in your [13:07] enterprise on your content. And we [13:10] recently worked with a company did a a [13:12] PC. They were working very extensively [13:14] with open AI had deployed a lot of uh [13:17] technology internally. But where they [13:19] were struggling was in getting the uh [13:23] chat bots to be able to talk [13:24] intelligently about their internal [13:27] content. And that's the the common [13:29] struggle with the LLMs. So we did a PC. [13:31] That's what we do to help get that in [13:33] place. And they actually had to do [13:36] another PC with another company. And so [13:39] they did our PC first and when they shut [13:42] it off to switch over to the other PC [13:44] and said now you got to use the other [13:46] technology there was an internal revolt [13:48] like what I I I need this to get my job [13:51] done every day now. They had gotten such [13:53] good results they could ask very [13:56] detailed engineering questions [13:57] multilayer complex things and were able [14:01] to get back answers that were really [14:04] well informed and in very short order. [14:06] Uh so it became an easy win for us [14:10] because it was just it it had the the [14:12] adoption was not a problem because [14:14] people saw the value immediately. This [14:16] is going to help me get my job done. [14:18] >> Let me just get a measure of the room. [14:19] How many people in the room knows know [14:21] what rag is? Pretty much everyone. Okay. [14:24] So we don't need to define that. They [14:25] they may not have known it on Monday, [14:27] but I think by now [14:28] >> it's late in the conference, but maybe I [14:30] mean can you just talk a little bit [14:32] about that approach because it's, you [14:33] know, and and how it sort of grounds AI [14:35] and provides sort of more deliverable [14:37] results pretty regularly. [14:39] >> Sure. Uh so as a quick overview, we have [14:42] these LLMs. They're trained on world [14:44] data on the internet. They can talk [14:46] about that stuff pretty well most of the [14:48] time, but they don't know anything about [14:50] your internal business, your internal [14:52] repositories. They know your company [14:54] exists because they've seen you on the [14:56] on the website and so they know that or [14:59] uh you know a news article maybe but [15:00] they don't know your internal content so [15:02] they can't speak to it. There are [15:04] various ways that we're tried. We looked [15:06] at fine-tuning and doing all of that but [15:07] rag is the way to do it. And what that [15:10] does is first we go out we do a search [15:12] and we pull back the information from [15:14] your internal content that's relevant to [15:16] the topic. We put that information into [15:19] the prompt and we say, "Hey, LLM, [15:22] ChatPT, Gemini, Claude, whatever you're [15:25] using." Hey, read this information and [15:28] then answer my question based not on [15:31] your training data, but answer it just [15:34] based on the information I'm giving you [15:36] right now. And when you do that, you've [15:38] done several things. First of all, [15:39] you've informed it on real, accurate, [15:41] factual data within your enterprise, [15:43] which it didn't have access to before. [15:46] Secondly, you've greatly reduced the [15:48] chances of hallucination because you're [15:50] focusing it on this content and this [15:52] content only. And third, you get [15:55] citations because it can come back now [15:57] and it's traceable to tell you where it [15:59] got that knowledge. And those three [16:01] things don't come any other way. So, RAG [16:04] is really the it's not a great acronym, [16:06] but it's what everybody's using to use [16:09] these LLMs on internal content. [16:14] See what interests me about that is I [16:15] think we've taken the exact opposite [16:17] approach. Okay? [16:18] >> Because the only thing that we've done [16:19] is train internally on the data that we [16:22] have. And so we get the confidence [16:24] because it's not hallucinating and go to [16:26] find out what it might know somewhere [16:28] else about it. But it only knows what [16:30] we're giving it right now. So that [16:32] doesn't mean that we couldn't or [16:34] shouldn't expand out to that point where [16:36] it's going to search the universe for [16:37] what it may know. But intentionally [16:39] giving it the data that's just yours and [16:41] showing people that it's not going to [16:43] hallucinate and being able to get those [16:45] citations back has been part of our [16:47] initial success. [16:49] Yeah, there's there's other approaches [16:51] of building domain specific models or [16:53] micro LLMs now they call it which are [16:55] very much trained on the enterprise data [16:58] that doesn't leverage the whole LLM [17:00] capability externally and based on the [17:02] use case you can kind of cater and that [17:04] when you do it with reinforcement RL [17:06] models is now learning the behavior of [17:09] the input to to you know expand so there [17:11] there is a lot of multitude of [17:14] approaches but I think it looks like [17:15] each one of them are showing success in [17:17] some way or the [17:18] I said it was complicated. You got to [17:20] know you got to know what you're trying [17:22] to accomplish and what the right [17:24] solution is. [17:25] >> So what and on our prep call, one of the [17:27] things we everyone seem to agree on is [17:29] that there there are plenty of PC's that [17:31] work with limited data sets which [17:33] generally get the green light for [17:34] management. Then they get rolled out and [17:36] money gets thrown behind them. They try [17:38] and scale up. That's usually where they [17:40] tend to fail. Um how do we get past [17:43] that? What what's is there a playbook to [17:44] follow? I mean how do how do we overcome [17:46] that? [17:47] Yeah. So I'm also on a board of a um [17:50] medical startup. Um the company is [17:54] Ultrasound AI and essentially it's [17:57] building models that um give an a more [18:00] accurate birth date for a baby [18:02] particularly to avoid pre-term issues [18:03] and other and other things there. But um [18:07] two things in that startup have happened [18:09] in in very short order. One is the [18:11] constant abstraction of of complexity up [18:14] into more commoditized models means that [18:17] um a lot of the core engineering that [18:19] had to get done to start the business [18:20] has been sort of moved up the stack. So [18:23] they're now concentrating more on the [18:24] outcomes than the engineering bit. Um [18:26] but but more than that um that is an [18:30] example where the training data is very [18:32] narrow and specific to the to the [18:35] result. It's not trying to do the [18:37] everything for everyone or this AGI [18:40] which I know we're all treading gently [18:43] towards. Um but because it's so narrow [18:45] and because it's so focused with the [18:47] goal of that um prediction, it's it's uh [18:51] the models are very very good. So I [18:53] think it's back to what are you trying [18:55] to do? Um what is the precision of the [18:58] problem that you're trying to solve? [19:00] That's an easy example because it's very [19:01] precise. um you know as you start to [19:03] broaden it though of course it gets a [19:05] little messier. [19:07] >> Anyone else want to comment on the size [19:09] of the models? [19:10] Well, in terms of scaling for success, I [19:13] guess um from what we've seen, we see [19:17] every company we work with and probably [19:18] everybody in this room, your [19:20] organization somewhere is running a PC, [19:22] probably doing some form of of rag or [19:26] customized model. And that tends to work [19:30] really well at small scale. But things [19:34] that don't necessarily show up at low [19:37] volume start to become real challenges [19:40] at larger volume. As you start to bring [19:42] in multiple sources, you have to start [19:44] to manage security. In the case of rag, [19:47] you need to make sure that you're [19:48] retrieving the right information because [19:50] garbage in, garbage out. So when you're [19:52] working with, you know, a 100 documents, [19:54] a thousand documents, you can get away [19:56] with a lot of things that you can't get [19:59] away with when you're up to, you know, [20:01] 20 to 30 different sources source [20:03] systems and millions of documents, which [20:06] is not every use case. You may only have [20:08] a use case for a thousand documents, in [20:10] which case run with the simpler version. [20:13] But if you're going to scale it up, [20:15] you've got to have a little more [20:16] industrial strength behind it. And I [20:18] guess my my suggestion there would be [20:21] get educated, know what's going on, and [20:24] if you're trying, if that's your end [20:26] goal is the scale to maybe not do it [20:29] yourself, but to make sure you get the [20:31] the infrastructure necessary to get [20:33] there. And I don't remember the name of [20:35] the session that I was in yesterday, but [20:37] it was about this full. And the the [20:40] gentleman asked, "How many of you have [20:41] deployed generative AI solution in your [20:45] organization?" And I saw about maybe 30 [20:47] 35 hands go up and then he said how many [20:49] of you have it deployed at scale and [20:52] every single hand went down. So partly [20:56] >> try that with this crowd. [20:57] >> What's that? [20:57] >> Want to try that with this crowd? [20:59] >> Sure. How many have a generative AI [21:02] system deployed right now that you are [21:04] using in production? [21:07] And how many of you would say that's [21:08] deployed at scale? [21:11] Few hands stayed up. Good job. And I [21:13] think that's indicative of the state of [21:15] adoption. It's not just as an indicative [21:18] of, you know, where projects have [21:20] failed. We're still, you know, the hype [21:22] is ahead of the reality here, but it's [21:24] it's difficult to scale up and that's [21:27] that's evidence. [21:30] >> Go for it. [21:30] >> No, I was just going to say I think [21:32] that's that's a great reminder to folks [21:35] that scaling becomes your own [21:36] organizational decision. And as you [21:38] said, it was complicated and I think [21:41] it's a reminder to go slow and to [21:43] validate and to plan that out really [21:46] appropriately to go from 0 to 60. We all [21:48] talk about going from 0 to 60. You know, [21:50] maybe go from 0 to 20 first. Take it [21:53] incrementally. Take it measured. Don't [21:55] expect it to go full boore and be as [21:57] successful as you were at a lower speed. [22:01] I think I'll I'll I'll add to that and [22:03] give a recommendation because I the last [22:05] two three years have been all [22:06] experimental which is great because [22:08] people need to build solutions in house [22:10] to understand you know how this [22:12] technology work why is it differentiated [22:14] why is it different so now we're getting [22:16] to a point that there are enough vendors [22:18] in the market uh that do have platforms [22:21] that can help you scale they are [22:23] building platforms I don't want to give [22:25] any names out here but you know they are [22:28] and I think assessing that as IT [22:30] leaders, head of departments to see how [22:33] they're talking about each element, [22:35] security layer, inbuilt, you know, um [22:38] models, uh ability for you to build a [22:40] lot more in scale would really help you [22:43] be part of a network because there are [22:45] other people using those platforms too [22:47] and going through the sales scale [22:49] momentum like you might building [22:52] in-house is easier but scaling that [22:54] solution is way way difficult. It is [22:57] good to experiment internally to see the [22:59] value of what you've built. But if you [23:02] really want point out solutions, there [23:04] are multitude of platforms, startups, [23:06] hyperscalers, everybody's offering [23:09] something or the other. Uh probably not [23:11] hyperscalers but you know startups are [23:13] definitely and you know seeing what [23:15] they're doing will help you expand the [23:17] footprint as well. [23:19] >> So we're definitely uh at the beginning [23:21] of the agentic age. Um I think all of [23:24] you have experimented with some levels [23:25] of agentic AI deployments. Um one of the [23:28] things we talked about is that one of [23:29] the mistakes that is often made is you [23:31] try and do too much agentic role too [23:32] many you have too many agents running [23:34] out there at the same time. Um can you [23:36] talk a little bit about the challenge [23:37] and the temptation to try and move fast [23:39] in that area [23:42] >> uh methodology that we faced we created [23:45] a paradigm saying it's a co-pilot [23:47] co-orker and an autonomous agent. If [23:49] anybody's saying autonomous agent, I [23:51] don't think they're there as yet, right? [23:54] Um just because you don't have the right [23:56] data, you don't have the right [23:57] environment. Um and you know there are [24:00] few things that are coming and saying [24:02] okay we can do this. I'm seeing few in [24:04] the legal side there you know multitude [24:07] of people cursor you know being one of [24:10] them from a coding standpoint but I [24:11] can't take cursor's code and put it in [24:13] full point deployment. It's not going to [24:15] do that. So simple uh examples yes but I [24:20] think agentic is in a very coworker [24:22] state right can I can it help and assist [24:24] me and is it learning and evolving is [24:27] that coworker going to be autonomous [24:29] agent in the next one or two years [24:32] probably so we've kind of seen it in a [24:34] QA scenario for support where you know [24:38] you could only do 5% of QAing of your [24:40] tickets now you can do 100% of those so [24:43] what does that do that actually enables [24:45] you to do a lot more coaching and [24:47] enablement with your team than just [24:49] spending time or looking up sample [24:51] tickets to do that. So that's where I [24:53] think the co-worker approach will [24:55] eventually lead to an autonomous agent. [24:57] It's not there. Uh but I think if [25:00] someone is saying autonomous agent, it's [25:02] not for complex enterprise scenarios as [25:04] yet. [25:06] >> Yeah. Um, so for those that don't know [25:08] DICE, uh, we are a tech platform, tech [25:12] tech talent platform, not a gambling [25:14] website, which is confusing when you're [25:17] in Vegas. Um, over the last year or so, [25:21] uh, we, you know, we look at a a a lot [25:23] of job postings and where talent skills [25:26] are. Um, 40% of new job postings in the [25:30] US require some sort of AI skill. So, [25:32] we've done some work to try and narrow [25:34] that down to help everybody and and kind [25:36] of come up with three big buckets. Um, [25:38] and they are builders. They're the [25:40] people who uh get the $und00 million [25:42] offers. By the way, if anyone's here, [25:44] I'm available. [25:46] >> You're for maybe even for less. [25:48] >> I mean, probably a million less, but you [25:50] know, around in that ballpark. The [25:53] builders. Um, then you have uh [25:55] essentially the orchestrators. They're [25:57] the people who are using the vendor [25:59] tools to to bring all this together. Um [26:02] and then the bottom not the bottom the [26:05] next layer is the enhanced pros. They're [26:07] the people who are using cursor for [26:09] example in their current roles but they [26:11] are really driving efficiency in their [26:13] in their roles. The the thing that I [26:15] have also experienced in my career uh [26:17] leading engineering teams is that uh you [26:20] can get um sucked into the uh the the [26:24] trap of we're going to rather than build [26:26] the thing we're going to build the thing [26:29] that builds the thing. it's the [26:31] framework to build the other thing and I [26:34] think that's where you've got to be [26:35] clear with the teams uh because it is a [26:38] new exciting area that you're not just [26:40] going to end up building a framework [26:42] without actually delivering real [26:44] business value and there's a lot of risk [26:46] and peril in that particularly in bigger [26:48] IT organizations I've seen a lot of [26:50] investment wasted without the real [26:53] result at the back there so if you're in [26:56] a team um uh and you're driving change [26:59] or you're imple implementing I I think [27:01] looking at your team and saying do I [27:03] have builders do I have enhanced pros or [27:06] do I have orchestrators and [27:08] understanding where you want to spend [27:10] the time and your dollars will probably [27:12] help get the result that you're after [27:14] too much building is very expensive and [27:16] probably won't get you what you need [27:19] >> did did they highlight any tools that [27:21] they're asking you in those job [27:23] descriptions that you need to have [27:25] >> um you'll be delighted to know we have [27:26] an entire report on that so see me after [27:29] I'll give you the QR code. [27:32] >> Yeah, I'll just interject real quick. [27:34] For us, we want to get to AI agents. We [27:37] want to be able to scale. We're not yet. [27:40] I can actually say that I'm doing [27:41] agentic AI because I have an agent that [27:44] actually calls out to another agent. And [27:47] you know, I'm not I'm proud to say that, [27:49] but yet I understand it's not the holy [27:51] grail. It's not exactly where we want to [27:53] be. But again, it goes back to that [27:56] start small. Make sure that one agent [27:58] can successfully talk to another agent, [28:01] get the outcome that you're looking for, [28:03] and then go build from there. [28:04] >> Do they like each other? Your agents [28:06] have had to do team building to make [28:08] sure they've been on. Okay. [28:09] >> Very transactional relationship. [28:11] >> They're not human. They like each other. [28:12] >> Okay. [28:13] >> So, let's let's stay with um so we've [28:15] talked about needing sort of executive [28:17] direction from the top. Obviously, some [28:19] kind of funding expertise which may or [28:21] may not exist inside the organization. [28:23] Um but let's say you get through all [28:25] those steps and then you introduce these [28:27] technologies to an enterprise. Then [28:29] there's the people. Um some people are [28:31] being moved around. Some people need to [28:33] be retrained. How do you how do you how [28:35] do you push adoption through an [28:37] organization that you know many of the [28:39] people may not even understand the the [28:41] the core technology? Um any stories [28:44] there successes or failures? [28:49] There's a lot there. There's the you've [28:51] got the change management, you've got [28:52] the education challenges, which I would [28:54] say are probably pretty similar to to [28:56] other things before. Um the the big [29:01] challenge I think here is there's always [29:04] going to be a group of people who are [29:06] kind of afraid of it. They don't really [29:09] know how to approach it. And you've [29:12] really got to encourage just use and [29:15] experimentation with it. um and get them [29:18] that kind of that comfort level and that [29:20] familiarity with it. Uh and that's not [29:23] necessarily an easy thing to do. We find [29:26] that having uh not only the support from [29:29] the top, but having kind of like you [29:32] know the champions the classic idea of [29:33] hey here's somebody who knows how to use [29:35] this and they will come along and and [29:37] you know ask you h how are how are you [29:40] using it? how do you apply it and [29:41] encourage some specific examples? But um [29:46] then you have to deal with the do they [29:48] come back question and sometimes they do [29:50] and sometimes they don't. So, I don't [29:52] have a magic bullet answer for that [29:54] question, but it is something that that [29:56] needs focus and the if you've really got [30:00] something that's creating its own value, [30:03] they will come back to it because they [30:04] realize, hey, this is an easier way or a [30:06] better way of doing my job and then you [30:08] might have to answer the question like, [30:09] am I going to lose my job? So, that's a [30:11] whole another topic. [30:13] >> Yeah. And again, it this is this is [30:17] change. It's classic change. Um, you [30:20] know, my experience, I've worked for [30:21] some businesses that are uh large and [30:24] honestly the answer unfortunately is [30:26] hidden efficiency at scale and no one [30:29] really knows how the magic's done. [30:30] That's the answer. It it is easier to [30:33] do. It's faster. Um, you displace um [30:36] certain functions as a result of that. [30:38] And then um you know a little bit to [30:40] what you said which is when I started my [30:43] career um everyone wanted to train you [30:45] on the finance side of things. say it [30:47] were insistent that you knew what [30:49] finance did. And I I've actually argued [30:51] for a long time that we should be [30:52] getting to a point where everyone's [30:54] trained on on technology and it because [30:56] it's becoming so important to what we [30:59] do. And I think at that point, you just [31:01] find the people who are going to be more [31:03] able to to make that change and double [31:06] down on that group. [31:13] say we're worried about creating a [31:15] digital divide and I think naturally [31:17] we're just going to because to your [31:19] point people don't always want to adopt [31:22] to whatever the technology is. They're [31:24] just comfortable doing things the way [31:26] that they've always done them. Right. [31:28] And so for us there there's a couple of [31:30] things. We're not looking to replace [31:32] jobs. We're trying to figure out how to [31:34] use the AI as an assistant to do the job [31:38] and to actually market it that way. [31:41] Right? That's one way to get around uh [31:43] the digital divide. Again, for us, it's [31:46] just having the technology out there, [31:47] finding the champions, finding the [31:50] people who want to use it, and [31:52] personalizing what the AI can do for [31:55] somebody has been probably the most [31:57] powerful message we've been able to get [31:59] across so far. So, we've recently rolled [32:01] out a function where people can create [32:03] their own personal agents. And if you [32:06] can actually show somebody that, oh, you [32:08] always go to this website and you're [32:10] looking at this website for information [32:12] and I can give you an agent that you can [32:13] just point at that website and then you [32:16] can go ask the AI against it. The light [32:18] bulb goes on for them. They actually get [32:21] to see that the what's in it for them. [32:23] Now, that's hard to do. We're talking [32:25] about at scale. You can't necessarily [32:27] get to everybody, although I'm going to [32:28] try. But I think when you can get that [32:31] message across to people that there is [32:32] something in it for them and they see [32:34] how it can help them, whether it's [32:36] personally or professionally, you'll be [32:38] able to have some success. [32:42] So I think I'm going to just throw a [32:43] little bit on thoughtprovoking [32:46] numbers. [32:48] Um, it's happening faster than you [32:51] think, right? There is a bet going on [32:53] with the billionaires group who's going [32:55] to create the first billiondoll revenue [32:58] company with just 10 employees. And Sam [33:01] Ultimate has kind of put it on LinkedIn [33:02] as well. He said, who's going to create [33:04] one person company hitting a billion [33:07] dollar in revenue? It's going to happen [33:09] in this lifetime. Curser is a 150 [33:12] employee company hitting 300 million in [33:15] ARR. [33:16] Windsurf was at 75 million with 200 [33:21] employees. [33:23] So it is happening. If we if yes there [33:27] are challenges for existing systems or [33:30] people that are working with existing [33:31] environments with the thought process [33:33] and everything that are not jumping on [33:35] it. We have solid examples of companies [33:38] in the market. Glean is at 300 million [33:42] ARR with just 600 employee company. So [33:45] there are examples. Harvey is a legal [33:48] software company. I don't know their [33:51] revenue, but they're going to be hitting [33:53] to 500 million in just three years of [33:56] their starting. So there is disruption [33:58] happening. It is upon all of us as [34:02] leaders. This is a time where it's going [34:04] to change. Behaviors need to change. [34:06] Leaders need to adapt more. They need to [34:08] get more and more technical in using [34:10] this technology. Um because you know uh [34:13] I do know I asked my 10-year-old [34:15] daughter and I told her to go read a [34:17] book and she gave me a summary. Guess [34:18] what? In 10 minutes [34:20] >> and I said you did not write this. She's [34:23] like no I did. She's getting trained on [34:25] GPD, right? She knows how to use chat [34:28] GPD. [34:28] >> But she wrote the prompt. [34:30] >> She did. She went online. She searched [34:33] for the book and she's like, "Can you [34:34] give me a summary?" And she just copied [34:36] it. And I'm like, "Oh my god, what a [34:38] monster I'm creating." But anyhow, you [34:41] know, she's smart enough to use [34:42] technology. So that's the generation [34:44] that's coming that is using and [34:47] adapting. So there are there is as [34:50] leaders, it's no longer saying go use a [34:52] technology or tell me the limitations. I [34:55] think we all have to embody it. We all [34:57] have to use it to help people downstream [35:00] understand its value and create that [35:02] change management process. So areas [35:05] where it has helped and I've seen [35:06] customers do it successfully, they're [35:08] doing gamification. They're giving [35:10] rewards for people that are building [35:11] these solutions. Um they're showing it [35:14] in all hands beating. They're [35:15] appreciating that change management [35:17] process, those disruptors that are [35:19] coming in and saying this is the new way [35:21] of doing it because the old way is not. [35:24] If you don't latch on to it, your [35:26] competitor will. I saw a board in [35:28] Silicon Valley while driving. I'm not [35:30] going to give the name of the company. [35:31] He said, "We're not here to displace [35:33] people, but people your competitor using [35:36] us are going to, right? So, if you don't [35:39] latch on to it, your competitors are [35:41] going to come on to it and take your [35:43] business. So, you know, get on with it." [35:47] >> But here's the great part of what you [35:48] just said, right? Because you asked her [35:50] to go read the book and give you a [35:51] summary. And it's no longer the right [35:53] question to ask, right? That's that's [35:56] where the change is in all of this, [35:57] right? [35:58] we've come a certain way and we've [36:00] always asked it the same way. If you [36:02] asked her to look at page 42 of the book [36:04] and tell you what passage on that page [36:06] was most meaningful to her, that would [36:08] have been harder to find. And that's [36:10] again where I think the methodology here [36:12] is going to make a difference. [36:14] >> Absolutely. I asked her to narrate it [36:15] and she couldn't. So that's why I know [36:17] she used GPD. [36:20] >> Just one final uh the other option if [36:22] you really want adoption is you can [36:25] mandate it. Not something I'd recommend, [36:27] but Amazon's ringing division. Now, in [36:30] order to get promoted, you have to [36:32] demonstrate where you have used AI to [36:36] increase efficiency within your [36:38] operations. And if you're a manager, I [36:41] forget what the exact criteria is, but [36:42] it's even it's even higher than that. [36:44] You have to use AI and show how you've u [36:47] you know done more with less basically [36:50] in order to get promoted. So, uh there's [36:52] that option. [36:53] >> I'll give you a harder version. having [36:55] worked in big hard companies. Uh get a [36:58] spreadsheet and um it's really comes [37:00] down to three T's. Train, transfer, or [37:03] terminate. Get your team to go down say [37:06] who you can transfer to a different [37:07] role. By the way, there was an article [37:09] that said that uh river dredging wasn't [37:11] going to get disrupted by AI. So if you [37:13] have a rigid river dredging uh [37:15] department, you could move them there. [37:17] Um can you train them? Is it possible to [37:19] train and move? And then you're going to [37:22] have to make the hard choice which is if [37:24] that's not possible um it's time for [37:26] talent change. [37:28] >> Yeah. It's an interesting it's funny in [37:30] the title of this panel and maybe you [37:32] guys felt differently when you saw it [37:33] but my assumption was that it was [37:34] scaling across an enterprise. So you [37:36] introduced the technology it scales [37:38] across the enterprise. The enterprise [37:40] grows and that's not necessarily the [37:42] definition of scale. You can scale with [37:44] a consumer base. You can scale your [37:45] revenue with 10 employees and maybe [37:49] less. and and that that is also scaling [37:52] and uh you know I think our assumption [37:54] of what it is to scale something sort of [37:57] needs to change. [37:59] That's my two cents. So um but as you [38:02] introduce these things into the [38:03] organization obviously it seems like the [38:05] audience we split them into IT people [38:08] and business people. Um but there's also [38:10] the HR department which is you would [38:12] think in charge of aligning skills for [38:14] agents or people to particular tasks. Um [38:17] there's the finance department that's [38:18] going to do the costbenefit analysis. [38:19] Like where does where do these [38:21] initiatives live inside an organization? [38:24] Have you seen mistakes made in terms of [38:26] where they come from if they're purely [38:27] IT driven or purely management driven? [38:32] >> I'll go first. Um back to the war [38:34] stories. So I mean there's two parts [38:36] here really. There is the [38:37] >> these like awful spreadsheets of like [38:39] you know [38:39] >> I know uh it's the it's innovation from [38:41] engineering and technology leaders like [38:43] you who are here learning about things [38:46] and trying to take techniques and tools [38:48] back to organizations. Um and then it is [38:52] uh the business side of enterprise where [38:55] they're looking for something. they're [38:56] looking for data insights or they're [38:58] looking for efficiency or they're [39:00] looking for a better way to serve a [39:01] customer and the vendors come in and [39:04] that disruption can be a little more [39:05] painful in my experience and I think it [39:07] it is better the the engineering tools [39:10] are better um but those are really the [39:12] the the core paths here I don't think um [39:16] that the HR department has completely [39:19] plugged together yet that we're going to [39:20] have as many or maybe more agents in our [39:24] organizations as we are humans And who [39:26] manages that in the future is is a [39:29] question mark. [39:31] >> You have to create the AR department. [39:33] >> The AR department [39:34] >> resources. [39:34] >> There we go. [39:36] >> So, I think we've got about 5 minutes [39:38] left. Um, so I'm going to let each of [39:40] you sort of predict where we're going to [39:41] be at AI for next year. Um, what will be [39:45] the state of enterprise adoption, how it [39:47] will be shaking out. Um, I did not get [39:49] to my questions on governance. So, if [39:51] you can slip in a little commentary on [39:54] governance into into your conclusion. [39:55] extra points, but no pressure. Um, but [39:58] yeah, so what are we going to how are [39:59] we, you know, we see the challenges, we [40:01] see we've talked about what you need [40:02] inside organizations and we obviously [40:04] everybody here understands the potential [40:06] of what's possible. Um, but what are we [40:09] going to see in 12 months? [40:12] So I think we are I I think the trend is [40:15] clear. I mean this is this is going to [40:18] be a revolutionary technology that's [40:21] going to impact almost all aspects of [40:24] our lives but certainly corporate [40:25] America. I think the timetable is maybe [40:28] more of the question. My personal view [40:32] is I think there's so much hype pushing [40:35] this. Everybody's talking about agents. [40:36] I love the fact that we made it through [40:38] a 40-minute panel discussion. We didn't [40:40] try to define what an agent is. Um, and [40:43] that's because that's good because [40:44] everybody has a different definition. [40:46] Uh, I don't really care about the [40:48] definition as long as when I'm having a [40:50] conversation with someone, we're talking [40:51] on the same terms. But we know that it's [40:54] going to be something that does [40:56] something on its own. It can take [40:57] action, can make some decisions, do [40:59] those sorts of things. And as you've [41:01] heard from the panel here, we're not [41:03] seeing much of that yet. I think there's [41:06] a strong push to that and people are [41:08] going to try to deploy them. And I think [41:11] the error rates that exist in these LLMs [41:14] is going to result in more failures. So [41:17] those stats of abandoned AI projects, if [41:19] we start to filter in more agentic [41:21] projects, I think we're going to see a [41:23] lot of efforts shelved. [41:26] That might be because the technologies a [41:28] little bit early. It might be because [41:30] you're trying to do something too hard. [41:32] It might be because you're taking a [41:34] bigger scope because a narrow focused [41:37] agent is going to be much more [41:38] successful than a broad agent that tries [41:40] to accomplish many different things. So [41:42] there are techniques that can be used to [41:44] do this and that's where the the [41:46] governance comes in. So how do you get [41:48] this to work? I think we're going to see [41:50] a lot of focus on trying to really [41:54] narrow down what an agent's going to do [41:57] to try to put guard rails on it to try [42:00] to make sure that we are observing the [42:01] results and that everything that it's [42:03] doing is visible so that it can be [42:05] evaluated either with kind of a a [42:07] rules-based metric or with an LLM as [42:10] judge. You're going to have to do that [42:11] incrementally throughout the step of the [42:13] of the process. Um, Grock 5 just claimed [42:18] pretty much the lowest [42:20] uh, error rate of any of the models. [42:22] They said when it's at its best, it's [42:26] wrong 1% of the time, which sounds [42:29] pretty good. You know, 99% success rate. [42:31] Well, if you have an agent that has 50 [42:33] steps, [42:35] that 1% compounds to a chance that it's [42:37] going to fail 40% of the time. And [42:41] that's abysmal. And I don't think 50 [42:43] steps is a lot in terms of where agents [42:45] are going. So I think we're going to see [42:47] some failures, but when the guard rails [42:50] and the governance catches up to it, [42:53] we're going to see that stuff deploy in [42:54] the next year or two. [42:55] >> Right. Rapid fire. 60 seconds a piece. [42:57] >> Sorry, I took too long. [42:58] >> I'll I'll go really quick. I think we're [43:00] going to see physical candy dispensers [43:01] on the side of computers, which the AI [43:04] can reward us with sweets when we do a [43:06] good job. [43:08] >> Hopefully, it's just sugar. And I can't [43:10] top that. For governance, [43:13] make sure you have good data. Try to get [43:16] a task force together of people [43:18] crossunctionally within your businesses [43:20] who want to be able to contribute. It [43:23] was interesting when people raised their [43:24] hands how many people said they had Gen [43:26] AI versus how many had them at scale. I [43:28] would expect that number to be a little [43:30] different next year. And I think we'll [43:32] probably have a better definition that [43:34] we can all agree on next year on Agentic [43:37] AI. And I think we'll be further along, [43:38] but not that far. [43:41] >> I think I'm going to be more towards the [43:43] other side. We're going to see smaller [43:45] organizations. [43:46] That's definitely going to happen. We're [43:48] going to see job descriptions change. [43:50] You're going to do more than what [43:52] typically your job description required [43:54] you to do. And you as managers, you'll [43:57] be not only asked to look at human [44:00] agents and performance, you'll be asked [44:01] to look at AI agents as well in the next [44:05] 12 months. So people who are not [44:07] experimenting with AI, don't know what [44:09] it means from manager different levels [44:11] are going to see them out of jobs pretty [44:13] fast. It's a great place to leave it. [44:15] Let's give him a big round of applause. [44:16] It's a great panel. Thank you so much [44:18] for coming.