AI Leaders Frame Workflow Redesign Over Model Upgrades for GenAI ROI

AI领袖：实现生成式AI的ROI靠重构工作流而非升级模型 企业生成式AI未能为大多数组织带来实质性回报，原因在于企业试图将现代模型直接套用于传统人工流程，而非重组工作流并拓展智能体工具集。 针对多数生成式AI项目因将新技术强加于传统流程而失败的现状，专家小组倡导“10/20/70法则”，优先推进工作流重构和人员能力提升。行业专家分享了实际应用案例，例如将供应商变更审查时间从数小时压缩至40秒、自动化信用评估叙述每年节省25,000小时，以及在不到三个月内完成包含2,800个任务的代码迁移。关键技术讨论强调保持人机协同（human-in-the-loop）监管、拓展智能体工具而非过度依赖提示词、在医疗保健等高风险领域强制采用图约束执行路径，以及依靠主动可观测性调试多智能体系统。 约80%的企业生成式AI项目带来的最终ROI为零，因为企业是将新技术强加于传统人工流程，而非重新设计工作流。 (0:04) 生成式AI的成功落地依托于10/20/70框架：10%取决于模型能力，20%取决于底层基础设施，70%取决于工作流重构与人员能力提升。 (0:04) 智能体模型仅占解决方案的10%至20%，而数据治理、安全保障以及与老旧系统的集成则占据了其余的70%至80%。 (6:08) 在医疗诊断中，语义AI智能体与OCR通过在数分钟内（而非数小时）评估付款方保险规则，加速了癌症患者的从订单到收款周期。 (6:08) 现场设备故障排除利用二维码向操作人员提供基于手册的诊断指导，同时后台智能体自动组织维修步骤并预订零部件。 (6:08)

Enterprise generative AI initiatives frequently fail when organizations attempt to bolt new models onto legacy human processes; sustainable returns require restructuring workflows, bounding agent autonomy, and focusing development on operational redesign. Workflow Redesign and Outcome-Driven Architecture (0:04) Opening the panel, the moderator stated that approximately 80% of enterprise generative AI projects deliver zero bottom-line ROI because organizations apply advanced models directly to legacy procedures. Value realization depends on a 10/20/70 framework: 10% model capabilities, 20% infrastructure, and 70% workflow redesign and human capability enhancement. The panelists emphasized that the core agentic model accounts for only 10% to 20% of an overall solution, with the remaining 70% to 80% consumed by data governance, security, and integration. Bryan Smith advised against having engineers meticulously document standard operating procedures or analyze screen recordings, arguing that doing so merely replicates existing human inefficiencies. Instead, organizations should provide raw input data and explicitly define the target outcome, allowing models to generate a clean first draft and uncover more efficient processing pathways. Compressing Operational Cycles in Regulated Fields (11:23) Panelists cited significant efficiency gains by deploying agents to automate structured drafting and data extraction while retaining human sign-off. At Farm Credit Services of America, Nayan Nair reported that automating regulatory credit narratives across roughly 12,000 annual farm loans saved 24,000 to 25,000 hours per year, reducing analyst drafting time from several hours to minutes. To build institutional trust, the team evaluated the system by running historical loan data side-by-side against past human-written narratives. Similarly, Bryan Smith described an Expedient workflow handling 2,000 to 3,000 annual vendor change notifications, where agents compress a two-hour preparation process into 40 seconds, leaving senior engineers responsible only for final authorization. In healthcare diagnostics, Glenn Fung detailed how Exact Sciences uses OCR and semantic agents to standardize legacy faxes and PDFs, evaluate payer insurance rules, and process cancer patient cases in minutes rather than hours to accelerate post-surgical treatment decisions. In industrial operations, Gregory Hodgkinson noted that Prolifics deployed QR-code-activated agents that diagnose logging equipment in the field and autonomously query maintenance histories to stage repair bay schedules and pre-order parts. Bounded Graphs, Tooling, and Observability (36:28) In high-stakes or time-sensitive settings, speakers argued against open-ended agent autonomy. Glenn Fung explained that healthcare applications require pre-approved, graph-restricted workflow paths analogous to robotics controls. If an edge case requires a deviation from the defined graph, the agent must trigger a real-time escalation to a human specialist rather than attempting unguided problem-solving. The panel cautioned that resolving agent failures through longer, more elaborate prompt engineering often degrades model performance. The superior alternative is expanding the agent's external tool set to give it functional execution capabilities. To monitor these architectures, Bryan Smith highlighted the necessity of full-stack, active observability that functions like a DVR recording, enabling operators to replay multi-agent reasoning steps, trace errors, and programmatically trigger interventions when operational boundaries are exceeded. Transforming Enterprise Roles and Software Delivery (15:30) The panel characterized agent adoption as an operational paradigm shift comparable to the transition to Agile methodologies, forcing organizations to re-evaluate quality assurance, testing speeds, and human requirements vetting. At Expedient, customer-facing roles were augmented by pre-meeting agents that compile stakeholder briefs and post-meeting agents that analyze recordings to generate individual feedback and manager coaching agendas. Gregory Hodgkinson illustrated how agent-assisted delivery models scale using a Prolifics software factory case study. Tasked with migrating 2,800 Talend ETL jobs to PySpark on Databricks within a three-month window, the organization deployed a small expert engineering team alongside an iterative agent pipeline. The system launched with an initial automated efficiency of roughly 40% and was tuned after every sprint based on real-time developer feedback, completing the entire 2,800-job migration ahead of deadline. Realizing value from collaborative AI requires organizations to move away from unconstrained autonomous agents and complex prompt engineering, focusing instead on bounded workflow graphs, expanded tool integrations, outcome-oriented process redesign, and mandatory human checkpoints.

在 Ai4 研讨会上，来自技术、金融、医疗诊断和基础设施领域的多位高管指出，约 80% 的企业生成式 AI 项目未能产生实质商业回报，其核心原因在于企业将先进模型生硬嫁接到陈旧流程中；企业必须采用“10/20/70 法则”，将 70% 的精力投入到工作流重构和人员赋能上，通过有效的人机协同释放业务价值。 价值重心的转移：10/20/70 投资法则 (0:04) Expedient 首席执行官 Bryan Smith 与多位参会专家指出，多数企业在推进生成式 AI 时收效甚微，关键在于陷入了技术错配：将 21 世纪的工具强行绑定到老旧的人工流程上。研讨会为此提出了 10/20/70 的落地框架，即 10% 取决于模型本身的能力，20% 取决于底层基础设施，而最为核心的 70% 则在于工作流的彻底重构与组织能力的协同升级。与会专家强调，在实际系统工程构建中，Agent 算法模型往往只占整体解决方案的 10% 至 20%，其余 70% 至 80% 的工作量普遍消耗在数据治理、系统安全性以及与传统遗留系统的深度集成上。 高复杂度场景突破与人机互信构建 (11:23) 在各类纵深行业中，人机协同工作流正在将传统以小时计算的复杂任务压缩至秒级，而人工监督与历史验证是建立信任的关键。在农业金融领域，Farm Credit Services of America 的 Narayanan Nair 介绍，针对每年约 12,000 份信贷分析报告（credit narratives），通过引入生成式 AI 进行数据聚合与起草，机构每年节省了 24,000 至 25,000 小时，分析人员单份报告的审核验证时间从原本的 30 分钟至 4 小时缩减为数秒到数分钟；项目团队在试点期通过将历史贷款数据输入模型并与信贷员此前撰写的内容并排比对偏差，迅速打消了业务团队的顾虑。在技术基础设施领域，Expedient 每年面临 2,000 至 3,000 起供应商变更通知，原本每起事件需要工程师花费 2 小时排查系统并草拟工单，现由 Agent 在 40 秒内完成全流程准备，但资深工程师依然享有最终的审核批准权。在医疗诊断方面，Exact Sciences 的 Glenn Fung 说明，通过语义 OCR 与保险规则引擎结合，系统可快速标准化处理传真或 PDF 格式的病理报告，将原本耗时数小时的先行授权审核缩短至数分钟，直接为切除肿瘤后亟待评估化疗方案的癌症患者赢得了临床决策时间；而在林业重型机械采伐现场，Prolifics 的 Gregory Hodgkinson 指出，操作员扫码即可调取 Agent 依托技术手册进行现场排障，后台 Agent 还能基于历史维修记录自动预测零配件并预订工位维修计划。 架构约束：工具扩充、图限制与回放观测 (36:28) 针对 Agent 陷入困境或无法独立决策的边缘案例，技术专家们总结了明确的干预原则与架构纪律。在模型调优层面，专家指出当 Agent 表现不佳时，开发者往往倾向于增加更长、更繁琐的 Prompt 指令，但过度提示（Over-prompting）实际上会降低其工作效能；更优路径是为其赋予更多具象化的“工具（Tools）”，扩展其实际执行能力。在强合规与高风险场景中，Glenn Fung 强调医疗领域的 Agent 绝不能采用开放式的自由探索模式，而必须遵循严格的“有界工作流图（Graph-Restricted Workflows）”，如同工业机器人一般仅允许在预先批准的路径内运行，一旦遇到路径外任务，必须立即实时升级并转交给人类专业人员介入裁决。此外，Bryan Smith 强调企业必须部署超越被动遥测的全栈可观测体系，系统需要像 DVR 录像机一样支持单步决策倒带回放，以便精确定位多智能体架构中的推理断点，并且观测系统需要具备主动干预能力，能在触碰边界时动态接管或交接给人类。 范式转换：结果导向设计与软件工厂重构 (40:02) 在业务流程设计方法上，Bryan Smith 提出了反直觉的洞见：与其让工程师花费大量时间精确梳理现有的标准作业程序（SOP）或分析屏幕操作录像——这往往只会机械复刻人类原本低效的习惯——不如直接定义期望的交付成果并输入原始数据，让 AI 自主探索并生成初稿，这通常能发掘出更为高效的全新路径。Gregory Hodgkinson 结合 Prolifics 为大型零售客户实施的“软件工厂”案例印证了这一模式：该项目需在 3 个月内将 2,800 个 Talend ETL 作业迁移至 Databricks 平台的 PySpark 环境，团队未采用传统的人工辅助编码方式，而是由少量专家团队结合 Agent 流水线作业，初始冲刺自动化率达到约 40%，并在后续每个冲刺周期根据真实反馈持续迭代 Agent 版本，最终在 3 个月内全量交付。与此同时，Agent 对组织各角色的重塑正在显现，从 Expedient 销售代表在会前自动接收背景简报、会后接受自动化教练评估与每周 1 对 1 辅导提纲，到软件工程体系经历类似敏捷转型的剧烈调整，协作模式正在全面重塑。 当前人机协同工作流已在多个垂直行业验证了百倍级的提效潜力，但当 AI 生成需求和产出代码的速度全面超越人类传统审查与测试的吞吐上限时，如何重构全生命周期的质检标准与人工审核机制，仍是企业在深化 Agent 应用时面临的核心瓶颈与未决课题。

## Background 本视频由 Ai4 频道于 2026 年 8 月 22 日发布于 YouTube 平台，记录了一场时长约 42 分钟的专家研讨会。视频聚集了来自医疗、金融、基础设施和 IT 咨询领域的五位行业高管，共同探讨生成式 AI 智能体（Agent）如何与人类员工深度协同，通过重构工作流、克服落地阻力以及衡量生产力提升，在企业日常运营中释放真正的商业价值。 ## TL;DR 研讨会直击企业级 AI 项目高达 80% 缺乏 Bottomline ROI 的痛点，指出成功的关键在于遵循将 70% 资源投入到工作流重构与员工能力提升的“10-20-70 法则”。各领域专家分享了利用 AI Agent 实现医疗理赔审核、农牧业信贷报告撰写、重型机械运维以及代码迁移等复杂工作流自动化的实战案例，大幅缩短了业务耗时并创造了显著的经济效益。专家们一致强调，AI 的引入促使人类角色从繁重的数据搜集者转型为“AI 编排者与决策者”，企业必须通过“人类在环”（Human-in-the-Loop）架构设计、主动式可观测性以及分层的员工赋能策略，建立安全防护网并消除员工对 AI 置换岗位的恐慌。 ## Atmosphere & style 整场研讨会节奏紧凑、气氛干货满满且充满务实的探讨氛围。与会嘉宾摒弃了空洞的技术炒作，以极具说服力的硬核数据和亲身实操案例展开对话，既有对技术潜力的热切期待，也有对医疗刚性约束、伦理责任和组织变革阻力的冷静审视。嘉宾之间的互动默契而深入，展现出行业先行者在面对 AI 变革潮水时的深邃思考与理性从容。 ## Key takeaways - [00:04](https://www.youtube.com/watch?v=6wbVAZAPyqo&t=4s) 当前企业级生成式 AI 项目中高达 80% 未能实现净利润投资回报（Bottomline ROI），主因在于企业常将新技术强行嵌入传统人工流程，而非重构工作流。 - [00:04](https://www.youtube.com/watch?v=6wbVAZAPyqo&t=4s) 企业要释放 AI 实际价值应遵循 10-20-70 法则：10% 投资于模型，20% 投资于底层架构，70% 投资于工作流重构与员工能力提升。 - [02:13](https://www.youtube.com/watch?v=6wbVAZAPyqo&t=133s) Agentic AI 与人类协同的复杂场景已在金融授信、重型装备运维、癌症复杂诊断及交互式蓝图生成等多个领域落地应用。 - [06:08](https://www.youtube.com/watch?v=6wbVAZAPyqo&t=368s) 智能体系统通常仅占整套解决方案的 10% 到 20%，剩余 80% 到 90% 的成功关键在于系统集成、数据治理、安全与合规。 - [11:24](https://www.youtube.com/watch?v=6wbVAZAPyqo&t=684s) 引入人机协同（Human-in-the-Loop）能实现数量级上的提效，例如变更通知处理时间由 2 小时缩短至 40 秒，信贷文本自动化起草每年可节省约 25,000 小时。 - [17:52](https://www.youtube.com/watch?v=6wbVAZAPyqo&t=1072s) 在人机协同新模式下，人类员工的角色正在从繁重的“数据搜集与初稿撰写者”演变为“AI 编排者（AI orchestrators）”与“业务决策者”。 - [15:30](https://www.youtube.com/watch?v=6wbVAZAPyqo&t=930s) AI 对软件工程与数据科学的冲击类似于敏捷开发变革，虽然通用模型使开发与 Prompt 门槛降低，但传统 ML 与领域专业知识仍具备不可替代的商业价值。 - [20:17](https://www.youtube.com/watch?v=6wbVAZAPyqo&t=1217s) 多位专家以硬核数据证实人机协同的 ROI：每年节省 2.4 万至 6 万小时、用 660 美元 Token 成本在 8 周内交付数人年工程量、以及通过解决医疗拒赔产生 2000 万美元潜在价值。 - [21:36](https://www.youtube.com/watch?v=6wbVAZAPyqo&t=1296s) 即便底层大模型发展进入平台期，企业仍可通过发挥人类独创性、在自有数据上精调小模型以及推动技术普及来保持核心竞争力。 - [25:42](https://www.youtube.com/watch?v=6wbVAZAPyqo&t=1542s) 化解员工对“被 AI 替代”的恐慌需要建立 AI 识读（AI literacy）培训，并明确“AI 永远无法替代责任担当（Accountability）”，人类始终是最终把关人。 - [31:03](https://www.youtube.com/watch?v=6wbVAZAPyqo&t=1863s) 推行企业 AI 赋能可采用三阶落地框架：从“个人生活趣题”建立兴趣，到“岗位流程痛点”攻克业务，再到“跨部门 Cohort 学习小组”构建内部生态。 - [31:03](https://www.youtube.com/watch?v=6wbVAZAPyqo&t=1863s) 企业设立专门的内部 AI 工程部门，负责公民开发托管、安全框架搭建以及定制化培训课程编制。 - [34:00](https://www.youtube.com/watch?v=6wbVAZAPyqo&t=2040s) “人类在环（HITL）”架构包含生成校验与异常救助两种机制；当智能体陷入死循环或逻辑卡死时，安全兜底机制（failsafe）会主动将控制权转交人工。 - [36:28](https://www.youtube.com/watch?v=6wbVAZAPyqo&t=2188s) 盲目增加 Prompt 长度会导致智能体性能衰减，拓展工具库（Tools）并建立支持逐帧回放决策轨迹的全栈可观测性（Observability）才是提升智能体能力的关键。 - [38:00](https://www.youtube.com/watch?v=6wbVAZAPyqo&t=2280s) 在医疗等高风险高时效场景中，应采用确定性的图谱工作流（Defined Graph Workflows）严格限制智能体边界，并结合主动式可观测性实时触发人工干预。 - [40:00](https://www.youtube.com/watch?v=6wbVAZAPyqo&t=2400s) 业务流程重构切忌照搬旧有的 SOP 或屏幕录像；更高效做法是定义目标产出与数据源，由 AI 起草后结合专家反馈进行敏捷迭代，以流水线方式实现大规模业务迁移。 ## Detailed walkthrough ### [00:04](https://www.youtube.com/watch?v=6wbVAZAPyqo&t=4s) AI 项目的 ROI 困境与 10-20-70 法则 研讨会主持人开场便直击行业痛点：在当前的企业级生成式 AI 项目中，高达 **80%** 的项目未能为企业带来任何净利润投资回报（Bottomline ROI）。造成这一现象的核心原因，在于许多企业只是机械地将 21 世纪的先进技术强行“套用”在传统的人工业务流程之上。 要彻底释放 AI 的实际价值，企业必须遵循 **10-20-70 法则**： * **10%** 投资于模型本身（Model）； * **20%** 投资于底层架构（Infrastructure）； * **70%** 投资于**工作流重构与员工能力提升**（Workflow Redesign & Human Capability）。 本次研讨会的重点正是深入探讨这关键的 70%。五位来自不同领域的行业专家随后依次进行了自我介绍： ``` +-----------------------------------------------------------------+ | 专家研讨会嘉宾 (Panelists) | +-----------------------------------------------------------------+ | Brian Smith | Expedient CEO (智能基础设施与私有云) | | Narayanan Nair | Farm Credit Services of America 高级VP兼CDO | | Gregory Hodgkinson | Prolifics CTO (AI 实施与数字化转型) | | Glenn Fung | Exact Sciences AI 开发与策略副总裁 | +-----------------------------------------------------------------+ ``` --- ### [00:45](https://www.youtube.com/watch?v=6wbVAZAPyqo&t=45s) 嘉宾自我介绍与企业背景 嘉宾们分享了各自的公司定位及其在 AI 转型中的视角： * **Brian Smith**：Expedient 公司 CEO。Expedient 是一家垂直一体化的智能基础设施公司，业务涵盖数据中心、私有云以及完整的 AI 技术栈。 * **Narayanan Nair（Nay）**：Farm Credit Services of America 的高级副总裁兼首席数据官（CDO）。他已在该职位任职超过 6 年，该机构主要为跨越 8 个州的农场主和牧场主提供金融资本支持。 * **Gregory Hodgkinson**：Prolifics 公司 CTO 兼工程负责人。Prolifics 是一家拥有 48 年历史的 AI 实施服务商，长期致力于协助客户进行数字化转型。他强调，未来的业务模式必然是 Agent（AI 智能体）与人类协同作战。 * **Glenn Fung**：Exact Sciences 的 AI 副总裁（该公司为结直肠癌筛查产品 Cologuard 的生产商）。他提到公司近期已被雅培（Abbott）收购，现归属于雅培癌症诊断部门（Abbott Cancer Diagnostics）。 --- ### [02:13](https://www.youtube.com/watch?v=6wbVAZAPyqo&t=133s) 复杂人机协同工作流的探索 主持人引导嘉宾探讨各自领域中最复杂的“人机协同”（Human + Agent）工作流： * **金融贷款自动化（Farm Credit Services of America）**：Narayanan Nair 提到，农牧业贷款流程涉及多个业务部门和功能区域，极其复杂。他们面临的挑战是如何利用 AI 工具链实现跨业务垂直领域的自动化，使整个授信与审批流程既高效又安全。 * **林业设备维护与 AI 智能遗嘱（Prolifics）**：Gregory Hodgkinson 举了两个极具代表性的案例： 1. **重型林业设备保障**：在森林采伐现场，重型大型机械的资本开支极高。他们利用 AI 确保这些机械设备的运行时间（Uptime）最大化。 2. **“AI 律师”撰写遗嘱**：虽然在技术层面并不算最复杂，但在伦理和心理层面却极具挑战——用户是否愿意让 AI 参与并辅助撰写自己的遗嘱？ * **癌症诊断 Order-to-Cash 流程（Exact Sciences）**：Glenn Fung 指出，公司除了提供大众筛查外，还提供指导癌症患者后续治疗（如肿瘤切除后是否需要化疗）的复杂诊断。由于病情危急，从下单到收款（Order-to-Cash）的整个流程都要求极高的时效性，以便医生能迅速做出决策。这需要将 AI 深度嵌入原本高度依赖人工的业务流中。 * **交互式语音蓝图生成器（Expedient）**：Brian Smith 透露，许多企业深感 AI 项目无从下手。为此 Expedient 部署了“Blueprint Agent”交互式语音智能体，通过与用户对话梳理需求，甚至自动向其他利益相关者派发语音 Agent 进行调研访谈，最终生成完整的业务评估报告、CFO 财务审查报告以及 15 到 30 页的技术实施方案。 --- ### [06:08](https://www.youtube.com/watch?v=6wbVAZAPyqo&t=368s) 深入技术底层：Agent 规则引擎与林业现场维修 嘉宾们进一步拆解了各自 Agent 工作流的具体运作机制： #### 1. 医疗诊断中的 OCR 与规则解析 (Exact Sciences) 在医疗行业中，许多病历、数字病理报告仍通过传真或 PDF 传输。Agent 系统首先通过高精度的语义 OCR 技术提取关键数据（如肿瘤大小、单侧/双侧特征等）。随后，Agent 会自动抓取并解析保险公司网站上的复杂赔付规则，将其转化为计算机可执行的逻辑，自动评估并给出审核建议，将原本需要数小时的规则核对缩短至数分钟。 ``` +------------------+ +-------------------+ +------------------+ | 医疗病历/传真/PDF | --> | 语义 OCR Agent | --> | 提取肿瘤特征与 | | ( unstructured) | | ( 提取关键病理数据)| | 病理指标 | +------------------+ +-------------------+ +------------------+ | v +------------------+ +-------------------+ +------------------+ | 保险赔付建议结果 | <-- | 规则评估 Agent | <-- | 自动抓取并解析 | | ( 耗时由小时变分钟)| | ( 匹配保险审核逻辑)| | 保留规章 (PDF) | +------------------+ +-------------------+ +------------------+ ``` #### 2. 林业重型机械的智能运维 (Prolifics) 林业采伐涉及伐木机（Feller）、打枝机（Harvester）和集材机（Forwarder）。当机械出现故障时，现场工人只需扫描设备上的二维码即可唤醒 Agent： * **第一步：现场排查**：Agent 读取手册与知识库，指导工人进行基础故障排除； * **第二步：维修决策与调度**：若故障严重，Agent 自动评估车间空闲排期； * **第三步：备件匹配与调度**：Agent 基于历史维修数据预测所需零部件，并完成全套供应链与车间调度的后台对接。 > Gregory Hodgkinson 强调：“Agent 本身可能只占整套解决方案的 10% 到 20%，剩下的 80% 到 90% 全在于系统集成、数据治理、安全与合规。” --- ### [11:24](https://www.youtube.com/watch?v=6wbVAZAPyqo&t=684s) 极速缩短耗时与人工复核（Human-in-the-Loop） 在极大地提升效率的同时，如何保持人控与安全是重中之重： * **变更管理流程（Expedient）**：Expedient 每年处理 2000 至 3000 份来自供应商的系统变更通知。以往工程师分析影响、开具工单、编写客户通知单，每份耗时约 2 小时。如今在 Agent 调用各类系统工具协作下，**流程时间从 2 小时缩短至仅 40 秒**。不过，最终的审阅、批准与授权依然必须由资深工程师完成（Human-in-the-Loop）。如果工程师提出了修改意见，Agent 还会进行自我迭代学习。 * **信贷文本生成（Farm Credit Services of America）**：在农户授信过程中，分析师需要撰写大量的合规信贷报告（Credit Narratives）。该机构每年需要撰写约 12,000 份报告，每份耗时 30 分钟到 4 小时不等。使用生成式 AI 提取多源数据并自动起草报告后，**每年可节省 24,000 至 25,000 小时**。 针对分析师对 AI 的信任问题，Narayanan Nair 分享了他们的变革管理做法： - 将分析师早期就纳入开发流程中，建立信任感； - 跑历史数据测试，将 AI 生成的新文本与分析师过去亲手写的旧文本进行差异对比（Variance Check），让分析师直观看到 AI 的准确度，极大地增强了他们使用新工具的信心； - 分析师的复核时间从几小时大幅降至数秒到一分钟。 --- ### [15:30](https://www.youtube.com/watch?v=6wbVAZAPyqo&t=930s) AI 带来的组织架构与岗位变革 Agent 的引入正在深刻地改变企业内部的组织架构与岗位职责： * **前端销售岗位的重塑（Expedient）**：销售人员的工作模式发生了根本性变化。会议开始前，Agent 自动搜集客户公司背景、高管信息及行业动态并生成会议预备材料；会议结束后，独立的 Coaching Agent 会分析通话记录，为销售员提供针对性的改进建议，并向管理者提供一对一辅导的关注重点。 * **IT 与软件工程岗位的全盘再思考（Prolifics）**：Gregory Hodgkinson 认为，AI 对软件工程带来的冲击丝毫不亚于当年的**敏捷开发（Agile）变革**。从软件工程师、测试工程师到需求分析师，每一个 IT 岗位的职责都在被重新定义。当代码编写速度超越了测试速度，甚至 AI 生成的需求文档数量多到人类难以逐一审核时，团队必须全面重构软件开发流程与人员角色。 ### [17:52](https://www.youtube.com/watch?v=6wbVAZAPyqo&t=1072s) 核心角色的演变：从繁重的数据搜集者到 AI 编排者与业务决策者 在人类与 AI 协同工作的生命周期（例如 AI 辅助开发生命周期 AI DLC 和 AI 测试生命周期 AI TLC）中，人类员工的角色正经历深远的转变。Narayanan Nair 以信贷分析师为例阐述了这种变化：过去，分析师需要耗费大量精力登录各个系统搜集数据点，并手动撰写分析报告；而在 Agent 辅助的新模式下，AI Agent 会自动搜集数据并生成叙述性草案。分析师的角色进而转型为“AI 编排者”（AI orchestrators）与“数据验证者”。通过将重复性劳动交给 AI，分析师得以腾出宝贵的时间去建立客户关系、深入进行风险评估以及从事需要人类主观判断的高价值活动。 在软件工程与数据科学领域，角色演变同样显著。Glenn Fung 指出，现在的软件开发几乎不再需要从零手写代码，传统 ML 工程师和数据科学家的工作重心也发生了转移。模型训练本身已变成一种通用化商品，工程师更多地是在调用和提示（prompt）既有的大模型。然而，Glenn 特别提醒，传统机器学习（Traditional ML）依然具备深厚的商业价值潜力，这需要多年积累的领域知识以及对数据的深入理解，绝不能为了赶时髦（"get in the latest wagon"）而盲目弃用这些坚实的技术基础。 - **AI 生命周期框架**：AI DLC（AI 辅助开发生命周期）与 AI TLC（AI 辅助测试生命周期）重新定义了人机协同分工。 - **信贷分析师角色转型**：从“数据搜集与初稿撰写”转向“AI 编排、叙述验证、客户关系维护与风险决策”。 - **数据科学与工程变迁**：模型训练日趋商品化，Prompt 工程成为主流，但传统 ML 依然保留了不可替代的深度价值。 --- ### [20:17](https://www.youtube.com/watch?v=6wbVAZAPyqo&t=1217s) 闪电问答：用硬核数据证明人机协作模型的商业回报 座谈进入快速闪电问答环节，各位嘉宾摒弃冗长的背景铺垫，直接抛出最能证明人类与 Agent 协同模型成效的“冷酷数据”（cold single numbers）： - **Bryan Smith（Expedient）**：**60,000 小时**。在其 400 人的公司内部，今年通过部署 AI Agent 预计可节省 60,000 个工作小时，相当于整体生产力提升约 8%。 - **Narayanan Nair（Farm Credit Services of America）**：**24,000 小时**。通过实现叙述性报告与数据搜集的自动化，预计可为团队节省 24,000 小时。 - **Gregory Hodgkinson（Prolifics）**：**660 美元与数人年产出**。公司内部自动化团队仅消耗了价值 660 美元的 API Token，就在短短 8 周内开发并交付了原本需要数人年（person years）才能完成的内部新软件系统。这一数据深刻彰显了新时代开发模式的剧烈变革。 - **Glenn Fung（Exact Sciences）**：**2,000 万美元**。在医疗与保险理赔场景中，通过引入 AI 加速医疗拒绝件（denials）的重新审批流程，潜在商业价值高达 2,000 万美元。 ``` +-----------------------------------------------------------------+ | 闪电问答：AI Agent 协同带来的核心效益数据 | +-----------------------------------------------------------------+ | Expedient (Bryan) | 60,000 小时/年 (400人团队，生产力提升 8%) | | Farm Credit (Narayanan)| 24,000 小时/年 (自动化叙述报告与数据搜集) | | Prolifics (Gregory) | $660 Token 成本 -> 8周内交付数人年级软件 | | Exact Sciences (Glenn)| $2,000 万美元潜在收益 (加速拒赔件二次审批) | +-----------------------------------------------------------------+ ``` --- ### [21:36](https://www.youtube.com/watch?v=6wbVAZAPyqo&t=1296s) 假想实验：若模型发展停滞，未来的核心竞争力何在？ 主持人提出一个假想命题：如果未来 12 个月大语言模型的发展进入平台期（plateau），技术不再突破，明年夏天企业将依靠什么立足？嘉宾们一致认为，即使底层技术停止更新，现有的技术遗产和人类特质依然留有巨大的释放空间。 Gregory Hodgkinson 强调“人类的技术独创性（Human ingenuity）”是决定性的胜负手。AI 极大缩短了“从想法产生到软件发布”的时间周期，但接下来的核心挑战在于“如何挑选真正能产生商业价值的构想进行投资”。Glenn Fung 则指出，企业将迈向“模型训练的成熟期”，重点从盲目追求巨型通用模型，转向利用自有数据精调（fine-tune）成本更低、体积更小的专属模型。他打了个生动的比方：现阶段我们就像是在用“工程起重机给面包抹花生酱”，完全属于大材小用。此外，Glenn 还拿自己用了 3 年的旧手机举例——虽然孩子们总是嘲笑他，但这部手机完全能满足他的日常需求；同理，现有的 AI 技术已经足够强大，企业无需时刻追求最新最贵的模型。 Narayanan Nair 补充道，在未来的 12 到 24 个月内，人类交互（Human interaction）依然不可或缺。由于生成式 AI 的本质是概率性的（probabilistic）而非确定性的（deterministic），目前尚无任何企业能像自动驾驶汽车那样实现 100% 的无人化运行。Bryan Smith 则认为，模型发展的暂停反而会带来“技术普及率（adoption）的爆发”。由于过去技术迭代速度过快，大量员工和企业因感到被边缘化和恐慌而选择观望；一旦技术步调放缓，这些企业和员工将有充分的时间赶上脚步，奠定扎实的架构根基。 - **人类独创性与价值筛选**：AI 解决了“建造速度”问题，人类必须解决“建什么才能创造真正价值”的选择题。 - **小模型与自有数据精调**：摆脱对超大模型的依赖，转向低成本、特定任务的高效小模型。 - **概率性 vs 确定性**：GenAI 的概率特征决定了人类必须在接下来的 1-2 年内持续参与闭环。 - **消除技术恐慌**：发展的喘息期有助于消除员工的边际焦虑，推动观望企业大规模落地。 --- ### [25:42](https://www.youtube.com/watch?v=6wbVAZAPyqo&t=1542s) 克服员工怀疑论：将被威胁者转化为 AI 编排者 员工的怀疑与排斥往往是企业 AI 转型中“默默杀伤力巨大”的隐形障碍。研究表明，仅有约 7% 的员工会主动拥抱转型变革。面对员工对“被 AI 替代”的恐慌，嘉宾们分享了各自消除排斥、提高全员 AI 普及率的实践战略。 Narayanan Nair 透露，他们将 AI 视为一项整体组织战略而非单纯的技术项目。为弥合技术能力与员工心理之间的鸿沟，他们与内部创新团队合作，设立了涵盖线下课程、视频教学、Show and Tell 示范展示以及定制化辅导的 AI 识读（AI literacy）培训体系，直接展示 AI 如何让员工的工作变得轻松。Gregory Hodgkinson 则分享了 Prolifics 的“转型主题月”经验：在 ChatGPT 发布初期，他们举办了“GenAI 6月”活动，全公司停下脚步专门探讨 AI 对各岗位的重构；次年又举办了“Agentic AI 9月”活动。更重要的是，Gregory 向员工明确指出了一个核心真理："There's one thing AI will never be able to take and that is accountability."（AI 永远无法替代的是责任担当）。员工必须作为责任主体，凭自己的经验去审查 AI 系统的漏洞与瑕疵，因为“AI 总是能以极具说服力的方式犯错”。 Glenn Fung 提醒道，AI 转型本质上是业务转型（business transformation），必须让业务人员参与到共同构建（co-building）的过程中，让成果成为业务团队自身的胜利。以 Exact Sciences 为例，一年前全公司 7,000 名员工中仅有 200 人在防火墙内使用 ChatGPT；而在加强沟通与培训后，活跃用户激增，CEO 甚至找上门来要求出台专门的管理策略，因为公司已经采购了超过 3,000 个账户许可。 ``` +-----------------------------------------------------------------+ | 企业克服 AI 排斥与普及梯次推进战略 | +-----------------------------------------------------------------+ | 1. 认知重构：明确“AI 无法承担责任（Accountability）”，人类是最终把关人 | | 2. 沉浸式主题月：如“GenAI 六月”、“Agentic AI 九月”全员动手实操 | | 3. 业务共建：让业务部门参与工具开发，共享效率提升的红利 | | 4. 分层分级教育：建立制度化 AI 识读课程（AI Literacy Training） | +-----------------------------------------------------------------+ ``` --- ### [31:03](https://www.youtube.com/watch?v=6wbVAZAPyqo&t=1863s) Expedient 的三阶员工赋能框架与 AI 工程部门实践 Bryan Smith 详细拆解了 Expedient 在推动员工转变过程中摸索出的“三阶落地法”。首先，安全团队部署了 AI 控制平台，提供基于角色的访问控制（RBAC）、数据权限隔离与算力集成，构建了可自动部署的安全环境。在底层安全保障确立后，公司通过三个阶段引导员工： 1. **个人生活激情项目（Personal Passion Projects）**：鼓励员工先为自己的个人生活打造 AI 工具（例如健身追踪器、拍摄冰箱内部食材自动推荐晚餐食谱等），以此激发对 AI 的兴趣与灵感。 2. **聚焦业务痛点（Workplace Pain Points）**：引导员工结合自身领域专长，在日常工作检查中找出最令自己厌烦的痛点，并给予空间自由去尝试和构建解决方案。 3. **AI 专项开发队列（AI Development Cohorts）**：参照领导力培训模式，每期组织 15 名跨部门员工组成学习小组，进行深度技术与思维方式的培训。 针对现场观众关于“企业内部 AI 协同网络”的提问，Bryan 解释称，公司专门设立了一个面向内部服务（非面向客户）的 **AI 工程部门（AI Engineering Department）**。该部门承担三大职责： - **公民开发管理**：长期维护和监管组织内由普通员工（Citizen Developers）开发的各类应用。 - **安全框架搭建**：占其三分之一的工作量，负责构建全公司统一使用的底层安全框架与开发工具。 - **定制化教育与课程设计**：占其三分之一的工作量，针对每一期 Cohort 编排培训课程。在组建小队时，故意将来自销售、财务、客户支持等不同部门但面临相邻痛点的员工组合在一起，直观展示各业务环节之间意想不到的互联互通。 - **三阶推进体系**：个人生活趣题（建兴趣） -> 岗位流程痛点（攻业务） -> 15人 Cohort 深入培训（建生态）。 - **AI 工程部门三大职能**：公民开发资产托管（1/3）、安全开发框架建造（1/3）、跨部门定制化教育培训（1/3）。 --- ### [34:00](https://www.youtube.com/watch?v=6wbVAZAPyqo&t=2040s) 人类在环（Human-in-the-Loop）架构设计与失败安全机制 当谈及如何对 Agentic AI 的决策过程进行分层控制、并在界面中可视化展示时，Glenn Fung 指出，“人类在环”（Human-in-the-Loop）在传统机器学习中并不新鲜。预训练模型通常能开箱即用解决 90% 的问题，但要搞定剩下的 10%，全靠高质量的人类反馈闭环。虽然这需要大量沉重的软件工程搭建（heavy engineering），但随着 API 化和模块化技术的发展，平台团队已经可以开发出通用的反馈捕获插件（plug-and-play feedback APIs）。 Gregory Hodgkinson 则将“人类在环”进一步拆解为两种截然不同的应用场景： 1. **生成校验型（Verification & Correction）**：AI 生成内容或决策草案，人类在界面中进行实时校验、修改并提供反馈，这些反馈数据随即被用于后续的模型微调与训练。 2. **异常救助型（Failsafe & Course Correction）**：当 Agent 系统在执行复杂任务时陷入死循环（going in loops）、逻辑卡死或无法得出有效计划时，安全兜底机制（failsafe mechanism）必须能够自动捕捉到这一状态，并将控制权无缝移交给人类专家进行干预和修正。 - **90/10 规则**：通用模型解决前 90% 的基础工作，人类在环捕捉最后 10% 的边缘案例并持续反哺模型。 - **HITL 双重机制**： - *校验闭环*：针对 AI 的输出结果进行实时人类审核与打标。 - *救助机制*：监控 Agent 的运行状态，在陷入死循环或逻辑阻塞时触发触发器（trigger）引入人工干预。 ### [36:28](https://www.youtube.com/watch?v=6wbVAZAPyqo&t=2188s) 人机回环：工具拓展与全栈可观测性 在智能体（Agent）遇到无法自行解决的问题时，人类介入（Human-in-the-loop）构成了系统的关键安全网。这种介入分为两种模式：一种是在关键业务流中进行**实时干预与决策**，另一种是在事后进行**诊断与追责**，探究智能体为何会出现异常行为。许多人试图通过不断追加 Prompt 来修正智能体，但 Prompt 越长，智能体反而越不敏感、效果越差；更优的解法是为智能体提供更多可调用的工具（Tools），赋予其多元化的履职手段。 为了支撑高效的人类介入，系统必须具备全栈的**可观测性（Observability）**。理想的系统应当像 DVR 录像机一样，允许维护者逐帧回放（Rewind）智能体在多层协作过程中的每一步决策细节，从而精准定位失效环节并进行针对性微调。管理多智能体系统，本质上与管理人类员工无异。 - **Prompt 陷阱**：盲目增加提示词长度会导致智能体性能衰减，丰富工具库才是提升智能体能力的核心。 - **可观测性 DVR 模式**：支持对智能体多层推演与决策轨迹进行回放与追溯，解决输出结果异常时难以归因的痛点。 --- ### [38:00](https://www.youtube.com/watch?v=6wbVAZAPyqo&t=2280s) 医疗场景的刚性约束与主动式可观测性 针对医疗等对时间紧迫性要求极高的特殊行业，智能体系统的架构设计需要采取更加严谨的态度。与完全自由探索的智能体模式不同，医疗系统应当偏向于**确定性的图谱工作流（Defined Graph Workflows）**，严格限定智能体的行动边界（Zero Tolerance）。这类似于早期工业机器人的轨迹规划——只允许智能体在既定的最佳路径上运行。一旦遇到必须偏离预定路线的复杂状况，智能体必须实时向人类发出请求：“我知道我应该留在原轨道上，但要解决此问题必须采取此变通方案，是否批准？”这种机制能最大限度降低医疗风险。 ``` +-------------------+ 偏离预警 +-------------------+ | 确定性流程图谱 | ----------------> | 人类专家实时 | | (Defined Graph) | <---------------- | 介入与决策授权 | +-------------------+ 授权指令 +-------------------+ ``` 同时，可观测性在 Agentic 架构中不应再是传统的“被动监控”（如仅向 Ops 团队发送日志），而必须升级为**主动干预系统（Active System）**。系统应当能实时理解智能体的当前状态，并在触发风险阈值时主动中断或转接给人类处理。 - **确定性图谱**：限制智能体的自由度，确保其行为约束在预先验证的安全区间内。 - **主动式可观测性**：将监控与实时干预机制深度融入智能体工作流，而非事后被动分析 telemetry 数据。 --- ### [40:00](https://www.youtube.com/watch?v=6wbVAZAPyqo&t=2400s) 流程建模与敏捷迭代：从专家反馈到软件工厂 关于如何对原有业务流程进行建模与优化，嘉宾们分享了各自的实战经验。Farm Credit Services of America 在构建信用评估智能体时，全程引入了 3 名资深信用分析师（Credit Analysts）进行闭环验证。系统通过持续吸收这些一线专家的真实反馈，不断修正 Prompt 和输出偏差，直至差异率缩减至可接受的阈值内，才最终推向生产环境。 针对业务流程的重构，团队得出了一个反直觉的结论：**不要试图去照搬现有的标准作业程序（SOP）或屏幕录像**。过于细致的步骤指令往往只会完美重现旧流程中的低效环节。更高效的做法是：明确定义期望的目标产出以及可用的数据源，先由 AI 生成第一版草案，再在此基础上进行迭代。 Prolifics 展示了一个大型零售客户的真实案例：客户需要在 3 个月内将 2,800 个 Talend ETL 数据作业迁移至 PySpark / Databricks。Prolifics 组建了一支由顶级专家构成的精干团队，并配备了专属的智能体流水线（Agentic Pipeline）： ``` +------------------------+ 迭代反馈 +------------------------+ | 专家团队 (Experts) | <----------------> | 智能体流水线 (Agents) | +------------------------+ +------------------------+ | | v v [首期 Sprint 提效 40%] --------------------> [每期升级 Agent 版本] | v [不足3个月完成 2,800 项迁移] ``` - **数据迁移规模**：3 个月内完成 2,800 个 Talend ETL 作业向 PySpark Databricks 的迁移。 - **渐进式效率提升**：首期 Sprint 即实现 40% 的效率提升，随后基于专家反馈按 Sprint 持续更新智能体版本，最终提前完成任务。 - **会议总结**：论坛在掌声中结束，嘉宾们移步会场外继续与参会者进行深度交流。 ## Notable quotes > [25:42](https://www.youtube.com/watch?v=6wbVAZAPyqo&t=1542s) There's one thing AI will never be able to take and that is accountability. — Gregory Hodgkinson ## Mentioned Mark Green, Narayanan Nair, Gregory Hodgkinson, Bryan Smith, Glenn Fung, Holy Family University, Farm Credit Services of America, Prolifics, Expedient, Exact Sciences Corp, Ai4, Cologuard, Abbott, Abbott Cancer Diagnostics, Blueprint Agent, ChatGPT, Talend, PySpark, Databricks ## Video description Mark Green - VP for Institutional Effectiveness, Technology and Innovation, Holy Family University Narayanan Nair - SVP & Chief Data Officer, Farm Credit Services of America Gregory Hodgkinson - CTO, R&D and Innovation, Prolifics Bryan Smith - CEO, Expedient Glenn Fung - VP of AI Development and Strategy, Exact Sciences Corp The real power of agentic AI emerges when humans and machines collaborate effectively. This panel explores use cases where AI agents work side-by-side with employees, handling repetitive tasks, surfacing insights, and enabling faster decision-making, while humans focus on creativity, strategy, and customer relationships. Speakers will highlight practical examples from industries like healthcare, finance, and logistics, showing how agentic AI is being woven into day-to-day operations. Attendees will learn strategies for redesigning workflows, overcoming resistance to adoption, and measuring productivity gains from this new model of human-machine collaboration. [Source](https://www.youtube.com/watch?v=6wbVAZAPyqo)

[25:42] Gregory Hodgkinson: There's one thing AI will never be able to take and that is accountability.

[0:00] The real power of agentic AI emerges when humans and machines collaborate effectively. [0:04] Moderator: 80% of enterprise generative AI projects deliver zero bottom-line ROI. [0:04] Moderator: 10% model capabilities, 20% underlying infrastructure, and 70% workflow redesign and human capability enhancement. [6:08] The agentic model itself represents roughly 10% to 20% of the overall solution; the remaining 70% to 80% consists of data governance, security, and legacy system integration. [36:28] Over-prompting degrades agent performance; expanding tool sets is the preferred path to improve capabilities. [36:28] Bryan Smith: Operators need the ability to "rewind back" and replay the sequence of decisions made across multi-layered agent architectures. [37:56] Glenn Fung: Healthcare applications require a bias toward well-defined workflow graphs. [40:02] Bryan Smith: Instead of having engineers meticulously map out every existing standard operating procedure (SOP)... it is much more effective to simply define the desired outcome and provide the raw input data.

[0:04] There's nothing quite like 3:00 energy on  a Wednesday in Las Vegas conference. So, [0:08] I'm glad everybody's excited to be here and we  have an excellent panel. Today, we're going to be [0:13] looking about 80% of enterprise genai projects  yield zero bottomline ROI. Why? Because we're [0:21] bolting on 21st century tech onto human processes.  To unlock the actual value, we must apply the 10 [0:28] 2070 rule. 10% model, 20% infrastructure, and 74%  workflow redesign and human capability. Today, [0:37] today we're going to go into that 70%. But  first, I'm going to ask the expert panel up [0:41] here to introduce themselves and tell us how  we're going to get there. Appreciate it. So, [0:45] I'm Brian Smith. I'm the CEO with Expedium. We're  a intelligent infrastructure company that's a [0:50] vertically integrated that from data centers  to private cloud all the way through a full AI [0:55] stack. Good afternoon. Uh my name is Nay. I'm the  senior VP and chief data officer for Farm Credit [1:03] Services of America. Uh our organization provides  financial capital to farmers and ranchers across [1:10] eight different states. I've been in this role for  over six years now. All right. Hey there. So it's [1:16] Greg Hartinson. I'm CTO and head of engineering  at Prolific. We are an AI implementation company. [1:21] Uh 48 years old. Uh the company that is not  me. Um and uh in that we've been doing digital [1:27] transformation for our clients but uh we've really  felt the move towards transformation in this [1:34] latest AI journey and uh along with I think most  of you we believe the future of our business and [1:40] your business is agents and people working side  by side to deliver value and there's a number of [1:47] ways we're going to discuss today for you as to  how we making that happen. Yeah. Uh my name is [1:54] Glenn Fong. I'm the VP of AI at um Exact Sciences,  the maker Cologard. You probably seen the ads. Um [2:02] recently we were acquired. We joined the Abot  family. So we're now Albot cancer diagnostic [2:07] diagnostics diagnostics, sorry. And very glad to  be here with you guys. So thank you gentlemen. So [2:13] why don't we get right into and and talk about  some practical applications about how we're [2:17] building Agentic AI. What is the most complex  human plus agent workflow you're trying to crack [2:23] right now? Let's not get into the weeds right now.  We'll get to that later. Uh Nine, why don't you [2:28] kick us off? Um again, so uh ours is a lending  um company which means we provide uh financial [2:35] capital to farmers and ranchers. So the lending  process itself uh it is complex depending on the [2:41] different business units and the functional areas  uh that we support. So an opportunity that we see [2:48] is with the different business verticals, how do  you automate and leverage artificial intelligence [2:54] and toolkits to make that process flow much much  automation and safer. Thank you Greg it. Yeah. So [3:01] we do a lots in the I guess the big category of  business operations optimization. So efficiency. [3:08] Um there's any number of back office stories I  could tell you and they're all in HR and finance. [3:14] Um, and they're all interesting, but I guess I'll  I'll touch on two very interesting ones. I'll just [3:18] mention them for now. Uh, the one is, as far as  you can imagine from back office, which is out [3:24] in the in the forests. Um, we've got a client of  ours that is in forestry and, [clears throat] uh, [3:30] there's very capital inensive machines used in  that process and we're using AI to help them, uh, [3:35] essentially ensure the uptime of those machines is  as high as they can keep it. And then the second [3:40] one is a we call the AI lawyer. It's really an AI  experience that helps you write your will. Now, [3:45] the reason that's not technically complex,  but it's a bit of a touchy subject. I mean, [3:49] do you firstly do you want AI? What's the role of  AI in helping somebody write their will? Um, but [3:55] those are the two that are probably most complex  for different reasons. Thanks. And you want to [4:00] for us um one use case that I want to bring up is  like we we are a diagnostic um testing company. So [4:07] we do a lot of um screening where we go for the  general population in Colorado but we also do [4:13] tests that help people navigate patients of cancer  whether they already have cancer what is the next [4:18] step do you need chemotherapy after the tumor has  been removed or not and as you might imagine there [4:25] is a lot of urgency to get the whole process order  to cash fast because it's a better experience for [4:31] making decision for the physician to get those  test you know quick into the into the patients. [4:37] That's that's our goal to to you know to minimize  the the damage the cancer and beat cancer with [4:43] these patients and these workflows um there is a  lot of opportunities for automation and also is [4:50] very embedded into manual processes with people.  So I think that's a very good one to to bring [4:55] up to this to the team of this uh of this panel.  Yeah. Thank you. Brian, you want to add anything? [5:00] Sure. the the one that we started with early  was, you know, something we labeled blueprint [5:05] and the reason was we found it was really hard  for people to identify how to start and so they [5:11] would have an idea of what outcome they wanted  but they didn't know the steps or who to have [5:14] necessarily involved. So the agent is actually  a interactive voice agent so that you know we [5:19] can dispatch a blueprint agent to anyone in the  company or also to clients looking at things and [5:25] they could describe the outcome they're trying to  achieve in in a voice conversation and then it'll [5:29] ask lots of different questions from you know  who are the stakeholders you know what are the [5:33] data sources what are the likely blockers and then  from that it'll actually dispatch voice agents out [5:38] to those other stakeholders and collect all the  different interviews so that you actually hear [5:42] have all the different stakeholders in a process  you actually heard and uh it brings back and [5:48] says well you were trying to solve this problem  but instead you really need to solve this other [5:51] thing and then it can uh build a like a one-page  review for the business of what's the purpose it [5:58] can give a financial review that you would give  to like a CFO but then give anything from a 15 to [6:02] 30 paid build plan that you could hand to a coding  agent to build a first draft it's great all great [6:08] examples I really appreciate you guys sharing  could we go under the hood now and maybe walk us [6:12] through whether the workflow that you went through  or another specific work flow that comes to mind. [6:17] What does the agent do where the does the human  plant their feet and what is the actual strategic [6:23] payoff from your perspective? Glenn, you want to  start? Yeah, I can start. So, it's um it's a truly [6:32] workflow integrated for different parts. So far  uh you first you can imagine that a lot of the [6:37] information that we get from from these clinical  records digital pathologies um reports they come [6:45] in believe it or not still in our healthcare  system they come in faxes or PDF so we have [6:51] a lot of OCR to do there. So that's kind of the  first and OCR is not simple because some of these [6:58] medical records have different ways to mention  different things that happen. For example, you're [7:02] looking you're looking for tumor size in a medical  report and characteristics of tumors. Sometimes [7:08] there are unilateral, bilateral, and there are  many ways to say this. So, so there is a semantic [7:12] component to this, right? That that agents can  really help. But after that, if you think about [7:16] it, when you want priorization for the insurance  to pay this so we can move faster, right? Um, and [7:22] making sure that you know that all the processes  kind of kick off and order to cash all everything [7:27] goes away. uh there is a set of rules but those  rules are in PDF in a website of the payer. So you [7:34] need to go I mean the the way that people does it  right now is people who really has been doing this [7:40] for a long time and they go and get the rules and  they know where the rules are. So we're doing that [7:44] automatically now. So we kind of translated those  rules into a way that an agent can interpret them [7:49] and then we build an engine that can take all the  information from these other agents that doing OCR [7:55] give it to um a set of rules uh and an agent can  evaluate those rules and make a recommendation. So [8:01] something that might take hours not takes minutes  and again all of this to benefit the the patient [8:07] uh at the end. Excellent. Excellent example of  setting guard rails and rules for the AI agent. [8:12] Greg, you had an interesting one when we talked  about the woodland forestry. Do you want to just [8:17] share that? Yes. So, I mean, it's it's it's an  interesting one because it's so simple. I mean, [8:21] we all understand the concept of forestry. You're  growing trees and you cut the trees down and [8:27] essentially you're u shipping them, turn them into  lumber. Um for them again, the business problem is [8:34] that they are very specialized machines involved  in doing that exercise. So they have uh harvesters [8:41] that are harvesting the trees. Well, fellas fell  the tree. Harvesters that actually move over the [8:47] fel tree and start stripping it down. Um they  have a forwarding unit that comes up and lifts [8:53] up all these logs and carries them on somewhere  for a truck to start getting ready to transport [8:59] to the lumberyard or to the sawmills. Uh and  again, you'd think where's the opportunity for [9:05] Gentig to transform a a business process like  that? So again, the RO R ROI tells the story, [9:11] which is they're very capital intensive machines.  They're very specialized. How do you keep them [9:16] running at as high an uptime as possible and  that means you need to be able to help the person [9:23] uh using the machine do some basic troubleshooting  of that machine if there's an issue. Um but it [9:29] then means you need to coordinate a process which  is determining is it serious enough an issue that [9:34] we need to send it back to the workshop. um  they run their own workshops so they control [9:38] the workshops but um do we need to send to the  workshop is there time available in the workshop [9:44] um do we have the parts required because we now  understand using AI uh and similar issues that [9:51] have happened what parts are required to do the  job we think needs to be done um and again that's [9:56] just a standard business process so how does  Agentic make that different so what we have is [10:02] an Aentic solution at the forefront they scan  a cure QR code on their fella or their picker [10:08] or their harvester and that plugs them into an  agent which is an interface that will be driven [10:15] by knowledge. There's lots of documentation on  these machines, a lot of manuals and they can [10:21] do basic troubleshooting on you know in situ using  an agent helping them out. Um but also that agent [10:28] can help them make a determination on whether this  machine actually needs to be serviced. And again, [10:34] the whole process from then on with some agents  making decisions at the right time is all agent. [10:39] So everything I described earlier, the uh  the uh the place checking in the workshop, [10:44] see if there's availability, it's all done by the  agent with tools used in the background. Um the [10:51] um checking using AI to say based on the kind  of issue we're seeing, have we done jobs like [10:57] this previously? What kind of parts did we need?  And again the whole thing strung together with [11:02] agents but I think this will be a common theme  you know an agent's about 10% of the solution 20% [11:07] of the solution the rest is integration the rest  is you know the data the governance the security [11:14] um and again that's what we specialize in is  the the end to-end solution that's required [11:18] to build those kinds of solutions. Thank you.  Thank you. Now, Nion and Brian, you both could [11:24] uh really brag and boast about taking processes  that took hours and squashing them down to seconds [11:29] or minutes from there. Um, so Brian, I'm going to  start with you. When you do something like that, [11:35] how do humans maintain peer review, oversight,  and safety over those kind of decisions? And if [11:40] give a little bit of context about that workflow.  Yeah, so the the workflow that you and I discussed [11:44] is uh really about change management and change  controls. So in our business, we get between two [11:50] and three thousand vendor notifications a year of  things that are going to require a change that you [11:56] know we need to have a maintenance window. We  need to notify clients. We need to understand [11:59] the impact. And every one of those generally took  our engineers about two hours to put together and [12:04] write the communication uh provide it to clients,  open the tickets and there were peer reviews [12:10] everything that was that was part of that. And so  very similar to what uh you described that there's [12:16] a number of different tools that the agent has  access to and it can access the different systems, [12:20] understand the impact, see how previous things  were written and it's able to take that process [12:25] from about 2 hours down to 40 seconds uh is kind  of our flow time and uh but then there's still [12:31] a definite human in the middle that we haven't  changed our peer review uh review process where [12:36] it generates the change management. It cues  up all the communication, but then a senior [12:41] engineer still reviews everything, uh, approves  it, authorizes it, provides feedback, and if [12:45] there's feedback, then it's going to self-improve  into the process as well. Thank you. I know Ryan, [12:51] you you had a similar success story. Um, in  that story, can you highlight where does human [12:56] judgment step in to validate that draft that  that you develop? Um, what what we have done [13:02] is we have something called uh credit narratives.  uh even though we've developed different agents, [13:07] the one thing that we've done is as we  grant loan um to our farmers and ranchers, [13:14] our credit analysts will have to write something  called narratives uh which is something from a [13:19] regulatory perspective they have to go through  this. Um so for not every loan requires it, [13:24] but we were running some math on it and then we  run about 12,000 narratives and each of those [13:30] takes about 30 minutes to 4 hours. So there was a  tremendous opportunity to automate the writing of [13:37] the narratives which is getting the data from  different data points bringing it together [13:42] uh uh and automating that task plus also getting  information uh based on the data that we have and [13:48] using generative AI to generate those narratives.  Um so when you think about those we were literally [13:55] saving on a on an average if you take about two  hours per narratives uh we were saving about [14:01] 24,000 25,000 hours per year. So this is the this  is the one that we've implemented in production as [14:08] a pilot. Our analysts are actually using it seeing  the value and we're going to go for a full product [14:14] implementation in November to get it. But one of  the things that that we've done is there is still [14:20] a human in the loop at the end of the narratives  where the analyst will just review it. But that [14:26] review time takes like a matter of seconds to  maybe a minute compared to like 3 or 4 hours. [14:32] So there is that in the picture. Also the other  side is we've been bringing the analysts along [14:37] through the journey. So there's a lot of uh back  to back to the panel said there's a lot of change [14:43] management process that has been involved in  bringing them as a part of the journey. So they [14:48] are they know that there's a lot of trust that  they needs to build in. Uh the other thing that [14:53] we did is the narratives that we've been writing  and generating. Uh we went and looked at the old [15:00] narratives that was created by these analysts. So  we basically look at our historical data, ran a [15:07] scenario, ran our models through it, generated the  narratives, provided some sort of a variance that [15:13] the analyst can look at it. So now they're looking  at the new narratives, they're looking at the [15:18] existing narratives that they themselves wrote and  compare that variance that gave them a boost of [15:24] confidence in trusting and practicing the process  that we've implemented. Thank you. So let's shift [15:30] to the organization chart. Name just one role in  your business that has fundamentally evolved or [15:38] looks unrecognizable because of agents. Uh Brian,  why don't you start off? Sure. Ours would be uh [15:47] the front end of the sales process. Uh and that's  really because we have a number of different [15:51] agents that surround a salesperson where their day  just looks different that you know if they have a [15:56] meeting scheduled on their calendar you know the  agent automatically does the meeting prep for them [16:00] and puts it together you know from background on  the on the company the individuals industry and [16:05] then shares that with all the expedio employees or  partners that are attending the meeting but then [16:10] post meeting it's also their separate agents that  are doing individual coaching for the calls to say [16:16] how they did you know what what they could have  done better and then uh helping provide coaching [16:21] to their leadership for weekly uh meetings and  one-on- ones of areas that they should focus [16:26] on based on different meetings and things that uh  really happened. It's really interesting example. [16:30] Thank you. Greg, what's your role? Uh I mean at  Prolific again we're focused as much on our own [16:38] company and transforming what we're doing with  agents as we are on our customers. So I spoke [16:42] about some business operations examples earlier.  Um there's not [clears throat] a single role, [16:47] so don't ask me to choose one. There's not  a single role on the IT side that has not [16:51] changed. I mean everything has changed. And what  we're finding is this this moment is every bit as [16:59] big if not much bigger from a process point of  view. Let's talk software process as agile was. [17:05] I mean agile agile came out and everybody had to  rethink the way they were doing development. They [17:09] had to rethink the roles. They had to rethought  the ceremonies. Everything fundamentally changed [17:13] at that point and we're at the same point now.  I mean we're having to rethink what does it mean [17:18] to be a software engineer, a test engineer. Uh  when you've got um you know code that's changing [17:26] faster than you can test. Sometimes you don't  even have requirements. What are requirements [17:29] mean in this new day and age? Um who's writing  the requirements? Who's vetting the requirements? [17:34] Sometimes you're almost using AI to write more  requirements than someone can humanly vet, let [17:39] alone test. So it's it's been change across the  board and our my job our job of the company is to [17:48] um figure out how to get the best out of the teams  in this new day and age. So we practice in terms [17:52] of AI DLC, AI assisted development life cycle, AI  TLC testing life cycle and then again how do we [17:59] use agents paired up with the roles across those  life cycles um to get the best out of a team as a [18:04] unit with humans and agents working side by side.  Ryan, what's your role? Um I would go back to the [18:11] credit analyst example that I said. Um in in this  particular scenario, the analysts used to gather [18:19] data. They would go into different systems,  look at the data, bring in those data points. [18:23] They would also gather information to write it.  Now their role will eventually be changed more [18:30] into how do you utilize the agent and using the  agent and how do you orchestrate it and then start [18:37] working on uh so the narrative part is already  written. So just validating it and the time that [18:42] they are actually saving they can use it basically  to build relationship with the customer or looking [18:48] at it from a risk evaluation perspective or doing  some judgmental activity. So that's where we see [18:54] it going. Glenn, I would say I I I was thinking  about software engineering because for me it's the [19:00] obvious one that is changing all over. You almost  well different different people thinks different [19:06] but you almost don't have to write code. You don't  need to. Um there's a but I know there's another [19:13] uh discussion about that. Um but I will say I mean  I've been doing AI for 25 years. So I've been know [19:19] in AI from machine learning data science and I  will say uh ML engineer ML scientist has changed [19:27] that means these roles have changed like training  models is a rare commodity now so you don't train [19:32] models you use models that are already trained  and when you do is prompt mostly although I will [19:38] dare to say that um a lot of the value that I'm  bringing to me my company right now even when [19:42] I'm a super project don't get me wrong I also  achieved with all this technology that we're [19:47] trampling on because we just going to get in  the latest wagon, right? Um, so traditional ML [19:52] has a lot of potential still to bring a lot of  value and that requires having, you know, the [19:58] uh the expertise and understand data to your data  and go and train these models. Uh, I mean that's a [20:06] that's a something that you need to develop with  years and and still can bring a lot of value, [20:11] but that's changing more and more. Absolutely.  Absolutely. Great examples. Uh quick lightning [20:17] round, no contacts, no no leadup, one cold number.  What proves the collaborative model? Uh for us, [20:26] it would be 60,000. You know, that's the number  of hours that that we'll save this year with the [20:31] agents in a 400 person company. So roughly an 8%  improvement. Ryan, you want to go? Right. Um, I [20:39] would probably say the 24,000 hours that we would  probably save as a part of implementing it. Uh, [20:46] the narratives automation. Greg. All right.  660 which is um our internal automation team [20:53] came up $660 of tokens they'd spent um at what  they calculated delivering several person years [21:02] worth of uh software internally within the  company, new systems in 8 weeks. And I mean, [21:08] if anything just makes you sit back and think this  is just different to uh anything that we've had to [21:14] deal with in the past, it's that kind of stat.  Gwen, a couple of things. I mean, um but I had [21:21] to pick one. Um I will say the ability to get  denials approved much faster and the potential [21:28] there is around $20 million for us right now. So,  excellent. Excellent. If we look 12 months out, [21:36] break neck models stop developing, we stay  here. What does this look like next summer? If [21:42] the models completely plateau, what still stands?  Uh, Greg, why don't you start us off on this one? [21:50] The models plateau, where do we stand? I mean, I  think the thing that's going to change that's not [21:54] going to change is human ingenuity. um because AI  and these machine learning models and now Gentic [22:02] uh and we're all kind of at the beginning of this  journey still even though we're a couple of years [22:06] into it but the human ingenuity is what's going to  really make this deliver value for us and I think [22:11] what we're seeing in our organization is um you  know the time from having an idea to to building [22:18] releasing that idea has shrunk down dramatically  and what we're seeing is a shift towards producing [22:25] and actually realizing value because um although  AI helps us build things quicker um the big change [22:32] we need is how do we focus our efforts to make  sure we're building things that are delivering [22:37] value quicker. So I think that is going to be one  of the areas that I think all of us are focused [22:42] on right now is how do we get e uh value out of  the the time and effort we're spending into AI. [22:48] Uh human ingenuity is going to unlock the ideas.  Uh it's it's choosing those right ideas to then [22:53] back and invest in and at prolificics that's what  we help our customers focus their time on. Glenn [23:00] you want to go next? Um yeah I will say a couple  of things. So um the first one is our ability [23:07] maturity again to train smaller models um so and  use our own data. So I think we're all in this [23:15] journey um of fine-tune our models. So we have a  we have again architectures that are way cheaper [23:21] uh to to run um that we can fine-tune with our  data specifically for certain tasks. So it's [23:28] not going to be right now I think we're using  a material to put butter in the in the bread [23:32] right we don't need these huge models and we want  even bigger ones but I think what we have we have [23:37] enough there. Um yeah the the other thing is  that um I think is like we um we love having [23:44] the latest technology. For example, I I have a  phone that is like 3 years old and my kids make [23:49] a photo of me, but I think it does what I need  and whatever I need to to get going, right? So, [23:55] we don't have to have the latest phone all the  time. So, a little bit of analogy. I think it's [23:59] great to have more powerful models, but they are  going to be also more expensive. So, I think there [24:03] is a lot of room still of stuff that we can do  with the technology that we have today. Really [24:08] appreciate if you can share that wise wisdom with  my daughter about not having new phones. Nine. [24:14] Um I I would say something that would continue  to stick around is that human interaction [24:21] uh utilizing the models that we create.  Uh it's because if you think about it [24:26] uh the models are getting better every day. But um  in in at least in our organization there's nothing [24:32] that we have implemented that is 100% um like a  self-driving car type of thing. uh it's because [24:41] most of the generative AI solutions that that gets  created um it is it is you know like it is really [24:49] probabilistic not really deterministic so how do  you it's it's a challenging problem that everybody [24:54] has so what I think is that human interaction will  at least stay there for the next 12 to 24 months [25:02] something like that and Brian can big us home  yeah I think my is is more from the human side [25:07] I think you would significantly increase adoption  because I think there's a whole section of people [25:12] that are somewhat on the sidelines because things  are changing so fast that it's very intimidating [25:17] because they can't keep up with the change.  So, they would have time to kind of catch up [25:20] to that. And then there's also companies that are  somewhat sitting on the sidelines waiting to see [25:25] what is going to be you that the platform what's  going to be the foundation that they can really [25:29] build on versus the next thing's going to be the  next hot sexy thing a few months later. So I think [25:35] that if you had that pause, you could increase  a lot of adoption. Appreciate that segue as our [25:42] next question is employee skepticism is often a  silent killer. Maybe it's just my organization, [25:47] but we all have those people that are skeptical  about adopting AI. It shows that about research [25:54] shows that at least 7% of workforce actively owns  the change of transformation. Um, how do we turn [26:01] our threatened employees into AI orchestrators?  Um, Nyan, why don't you kick us off? Um, again, [26:09] this is a journey that that we went through. So,  this was not necessarily a technology initiative. [26:16] Uh, it was it was an overall organizational  initiative. So, one of the things what we've [26:20] seen is we've seen early adopters where they're  really into AI. Then we've seen some of the [26:27] skepticism around uh what is happening to my  job if we implement that right I mean so that's [26:32] a problem that that's a that's a challenge that we  all have to overcome and then uh I we believe what [26:38] we've seen is uh not some of the employees were  not able to clearly understand the value that's [26:45] getting generated using AI. So we partner with  the rest of the organization innovation teams [26:51] and other areas and then put a structure behind  how do you scale up our organization from a AI [26:58] literacy perspective. So we've got so many classes  uh in-person classes videos do a show and tell [27:06] depending on the particular business functions  they are in. We went in and we did a show and [27:11] tell and specific coaching lessons that kind of  bridge the gap between uh the capabilities that [27:16] the AI can actually do to make your job looks do  more easier. Craig, you want to I don't think so. [27:24] We found that we had to do something. It couldn't  just be um you know training our staff. It had to [27:30] be something more transformational. So I remember  in the early days uh just a couple of months after [27:36] chat GPT came out we had a June of of generative  AI and the entire company focused on how we were [27:45] going to transform ourselves for a month and  what the implications were on us and on each of [27:50] our jobs and on how they would change our roles  but actually so educated people and gotten them [27:55] hands-on and letting them build as well and that  was followed up by September of Aentic AI Okay. Uh [28:02] a year later uh we went through a similar kind of  transformation exercise. So it was again educating [28:07] people but more importantly what we've done is  made sure uh the clear thing that I help our teams [28:14] understand is there's one thing AI will never  be able to take and that is accountability. So [28:20] um you need to look at yourself as you're  still going to be the person that's going to be [28:24] accountable to deliver what we're delivering for  our clients. you need to make sure that you've got [28:30] the AI toolkit that helps you do that delivering,  but again, you're still accountable because the [28:35] AI can never be accountable. So, I think that  has made it clearer for them what their role [28:41] is. And to be accountable, you need to understand  that AI well enough to again reason about whether [28:48] it's doing what it needs to do properly or  not. Um, you need to bring your experience [28:54] um to do that because otherwise we've all seen  AI. AI does everything very in a very compelling [28:59] way. Uh everything's believable, everything looks  correct, but it's really when you put it in front [29:04] of an expert that they will find the gaps and the  flaws. And that is really what the team needs to [29:10] bring in. They need to have that experience. They  need to be able to bring in that mindset to use [29:16] the AI to accelerate what they do, but they're  still accountable. Glenn, you would add to that? [29:22] Uh yes. I I think one one thing that is kind  of a cliche but I think it's true and it's a [29:27] good reminder is like um you know the AI is not  only a technological transformation it's also a [29:35] business transformation I would remind people of  that because people think that we and we talked [29:39] about this before right like you had the solution  and that's not the easiest part integration and [29:44] and assimulation for the workforce so um one thing  that I will say is bring them along like you are [29:51] not building this from them. We are building it  together. So this is something that I couldn't [29:56] build it with without your knowledge. So you're  the business person, right? So we're of a team [30:01] and doing this with you is you are part of this.  So it's a success for you too. Your life is going [30:06] to get easier. You're going to be be able to be  more productive. So bring it along, right? And the [30:10] other part is education. Um I totally agree. So  people we have been very intentional about u kind [30:18] of design education for different different parts  of the organization and and we're seeing how the [30:23] adoption has been more and more. An interesting  an interesting fact is when I joined the company [30:28] a year ago we had like I don't know uh we are 7  7,000 people company and we only have let's say [30:36] 200 people using uh charg. So we make sure that  was safe that they was inside the firewalls and [30:43] then we started promoting educating people and  now my CEO told me hey we're spending too much [30:48] like 3,000 licenses we have to have a strategy.  So again so that tells you right like people gets [30:53] on board and people start using it but it takes  a while and it takes good communication education [30:59] kind of bringing it to people closer to them.  Yeah. Brian how about you? Yeah I think from our [31:03] side I definitely see multiple different layers  that we had to do. So first was really about how [31:08] we could put the foundation in in the company from  a technology standpoint that we felt comfortable. [31:12] So implementing our AI control platform gave  them a place where they could develop that [31:18] had all the role-based access controls. The  administrators provided access to the data [31:22] that people were allowed to have access to. It had  the compute pieces associated with it. So when you [31:27] built something it could self-deploy. So having  those foundational pieces that we as a security [31:33] uh teams felt good with you then went to how can  we we now work with the employees and from there [31:39] there were three really steps that we found  and we had to switch things around and this [31:44] is what we found worked best was first challenge  everybody to build something for their personal [31:49] life and we have everything from fitness trackers  to people that took pictures of their uh inside [31:55] their refrigerator to get an idea for what what  to make for dinner because what they had. So, [32:00] you know, making personal passion projects kind of  got the bug. And then second uh to that was really [32:07] aligning them with their area of expertise like  you're talking about showing them the value they [32:12] have as the human in the process and you know what  is something that annoys you? what's a pain point [32:18] in your current role and identifying that through  our check-in process and and communication and [32:24] then giving them the freedom to build and play to  try that. And then the more formal in that third [32:31] layer is similar to how we've done leadership  development cohorts in the business. We're now [32:36] doing AI development courses inside the business  where you have a group of 15 employees that [32:41] you're going through at a time uh with a deeper  understanding of what our platforms do, you know, [32:47] why it works, how to think of it, the mindset,  and now you have peers that that you can talk to [32:51] and work alongside with along the way. Thank you.  I'll repeat the question. Yeah. So, she just asked [32:58] for more explanation about the AI peer employee uh  learning network in your company. Yeah. So I added [33:03] a new department that is AI engineering that's  internal for our own purpose not for our clients [33:09] and they really have three functions. So you know  one function is owning long-term anything that's [33:15] citizen developed inside the organization. Then  a third of their job is building the frameworks [33:20] that get used for the organization so that we can  safely deploy safely use our own tooling. And then [33:26] a third is around education. And so they're  building that uh training and the curriculum [33:33] that goes with it. And it a lot of it is tailored  to who we have in the cohort at the time because [33:39] we'll try to arrange the cohorts where you have  mixture across different departments but people [33:44] that are focused on similar painpoint. And  so you may have representation from sales, [33:50] finance, client support that are all uh  trying to solve things that are adjacent [33:55] to one another. and we can actually show them how  they're interconnected that they didn't realize. [34:00] Great question. Thank you. So the question here is  about how humans stratify the agentic decision-m [34:06] and make sure that there's an infrastructure or  some kind of interface where the decisions can [34:10] be seen. Um any go ahead I can I can pick one  quickly here. So um this is not new right like [34:19] um human in the loop with machine learning and AI  has been there for a while and what I tell people [34:24] is usually that there is is very useful because  the models especially now with these pre-trained [34:31] models they can give you sometimes 90% out of the  box and the last 10% you know it's like having a [34:37] Corolla versus a Lamborghini that one is very  costly but it comes cheaper if you capture the [34:44] the feedback back or the users and then you can  improve your not only your process but eventually [34:48] your models talking about smaller models that  you can fine-tune and train right but the point [34:53] of that is that that infrastructure that you need  was never easy so it's a lot of heavy engineering [34:58] in order to put those in there uh that has been  changing because obviously there is a lot of need [35:03] for humans in the middle of these um you know  workflows for agents and stuff like that so I [35:09] think we're getting better at that from the  technology point of view and there is a lot [35:13] of functions and companies that offer for it. For  example, I have a platform team and some of the [35:17] things that they develop is something that is  pluggable and reusable for capturing feedback [35:21] for applications, right? It doesn't work 100%  of the times because depending on the way the [35:25] application is hosted or the integrations that we  need, but it's something that we have that always [35:30] can can you know can go to that and plug it in  as an API in any part of development to capture [35:38] uh the feedback and improve the process. Anybody  want to add to that? Well, as I say, the um so [35:43] there's there's two places we see humans in the  loop in the systems we're building. Um you're [35:48] either bringing a human in because something's  being generated and you want them to verify [35:53] um and check and correct. So obviously you need  give them a mechanism to give that feedback in the [35:59] moment and then that gets sent through to further  um development training efforts. Uh but then [36:05] there's also human in the loop when the AI gets  stuck. So that's also another category of human [36:10] in the loop which is you've got a a system that um  normally 90% of the time makes the right decision [36:17] 90% of the time can find the answer and then come  up with a plan to course correct. Uh but you need [36:23] to have a failsafe mechanism that sees well the  agent's getting stuck. It's going in loops. It's [36:28] not able to figure out what it needs to figure  out. So we need to bring a human in either in [36:34] real time because it's a important process  and then they need to step in and and make a [36:38] decision or correct uh or after the fact and then  it's more of a again well diagnosing why did why [36:45] is the agent behaving in the way it's behaving. Um  typically the answer is more often adding tools to [36:53] the agent more often than changing the prompts. Um  cuz you're only going to get so far with prompting [36:59] an agent. Uh and agents, the more you put prompts  into it, and the longer the prompts get, the less [37:04] effective they become. So the better solution  is adding tools that mean they have more ways of [37:11] doing the job that they need to do. Um and again,  those are the two places we see humans stepping in [37:17] the loop. Yeah. One little thing I would add is  uh really focusing on whatever agent deployment [37:23] you do. Make sure that you have observability  through the entire stack so that we have it set [37:28] up so that you can provide feedback on any of the  the agents, but you can literally watch in as a [37:34] DVR the replay of all the different decisions the  agent make because when you think of multi- layers [37:40] of agents and you see the outcome, it's hard to  go back and diagnose where the problem was if [37:45] you can't have the observability and rewind back  to all the different decisions along the way to [37:51] um tweak and edit. So, it's very similar. You need  to manage it similar to how you'd manage a person. [37:56] So the question is how would it be pertain to  health care if specifically so specific to the [38:02] time crunch of healthcare not the privacy of data.  Yeah. Yeah. So time crunch agents correcting them [38:07] in real time. Well that in healthcare I think  you should be biased for having well-defined [38:12] workflows instead of orchestration workflow. So  you have imagine that you have a graph that tells [38:17] you all the things that can happen to resolve the  problem that you're having for that patient and [38:22] you try these agents to stick to that as much as  possible. I mean you can have zero tolerance and [38:27] you say you only going to be operating in this  space in this world. There's nothing else that [38:31] you can do but this actions you know goes back to  robotics in the old days. That's why we we have a [38:36] graph and then robots say from going from A to  B you can use this path but nothing else right [38:42] when now with agency we can tell go to from A to  B and figure it out right I don't think I mean for [38:48] healthcare that's a little bit more risky and I  think and hybrid is more like how do you make sure [38:53] that the agents stay close to that and if they  have to debiate they call a human right away and [38:59] say hey I know they have to stay in this lane but  the only way to solve this problem is this you're [39:03] okay with that right in real time and doing  it. But that will minimize the number of those [39:07] things that can happen, right? No, not specific  to healthcare. And I'm guessing Brian, this might [39:12] be something you have in your your system, but  um observability sounds like a passive thing. [39:18] Uh and typically it has been. I mean, when you're  observing applications, it's it's a passive thing. [39:23] Your telemetry, maybe it goes to an ops team and  maybe that, you know, they start oper operating on [39:28] those things. But in the world of Aentic, it needs  to be much more active. So the the observability [39:33] the outputs that are coming need to have either  in your aentic workflow system you've designed [39:40] in actually understanding what's happening and  then acting on that to to intervene when you [39:46] need to intervene or kick something to a a person  when they need to uh step in. So again it's not a [39:52] observability in this day and age is not a passive  thing. It's very much part of the active system um [39:57] energetic. So the question is how was a model of  the people and humans that were doing the process [40:03] was it the best person or was it an average of  processes and then what does the iterative process [40:08] for optimization look like? I can start. Um so  in the particular example that we had uh we've [40:15] had three credit analysts. They do the job on a  day-to-day basis. They were a part of this uh from [40:20] start to end. the problems that we've generated  or we created that generated the output initially [40:26] that was not the right outcome. So we continue to  keep that refined by the input that we've got from [40:33] the credit analyst. So since they were actually  doing the job they would provide that honest [40:37] and candid feedback. Uh but we tend to iterate  on those uh created the variance at the end uh [40:43] that was within that acceptable threshold and then  finally rolled that into production. 90 seconds. [40:50] Did anybody want to add anything? Yeah, we learned  something maybe a little counterintuitive because [40:55] normally engineers wanted to lay out all the  different steps of what you do and we found it [41:00] to be a lot more efficient to define the outcome  you're trying to achieve all the data that you [41:05] have and then do the first draft from that  instead of giving additional instruction and [41:11] then going in with standard operating procedures  and screen recordings of the people and different [41:16] layers. uh because often you ended up recreating  the process that was in place, not necessarily [41:22] the most efficient process. So I'll give another  example. So prolific, one of the other things we [41:28] specialize in is creating an AI powered software  factory to do a job at scale for a company. Um so [41:33] I'm going to give you an example which is we had  a a retail client of ours say they needed to move [41:38] 2,800 talent ETL jobs to Pispark data bricks in  3 months. Now, you can get a bunch of developers [41:45] in front of clawed code and try and do it. It  would be highly risky. I'm sure it's going to [41:49] accelerate you, but you're doing a lot of the same  thing over and over again. And what we do is we we [41:55] create a software factory team, very small team,  the big experts. We build an agentic pipeline [42:01] that works with them. And to the question of uh  do you you know, how good do you get it before [42:06] you release it, we created something that would  give them about 40% efficiency from the get-go [42:11] in the first few sprints. And then every sprint  we released new versions of the agents based on [42:16] how they were performing and it was continuously  improvements until the end of the project. Um then [42:21] the good the good news was we got it done in  less than 3 months. Um but again it's it's a [42:26] continuous thing. You're tweaking you're tweaking  the system based on feedback so you can get higher [42:30] and higher level of efficiency. Well thank you  gentlemen. This was an amazing panel. I appreciate [42:35] you sharing the wisdom and would you guys mind  meeting outside if anybody has questions and for [42:39] you guys following up? Sounds great. Thank you.  Can you give a round of applause to our panelist?