Health Executives Say Clinical AI Risks Failure Without EHR Integration

医疗高管称临床AI若缺乏EHR整合恐面临失败 在Ai4大会上，医院管理者指出，临床AI工具若不能直接整合到现有EHR工作流程中并在报销环节保留医生责任，将难以被真正采纳。 本次专题讨论探讨了临床AI在各医疗系统中的实际落地应用，涵盖从NLP驱动的疾病识别与就诊前自动摘要，到专业儿科影像以及数字病理热力图等场景。与会专家探讨了关键运营策略，包括通过影子部署确立基准真相、采用混合AI架构消除LLM幻觉，以及实现深度双向EHR集成以避免临床医生出现点击疲劳。专家们最后强调，AI的成功落地必须平衡医生工作负荷、坚持人在回路的问责机制，并使工具契合临床指南与医疗报销框架。 多家医疗系统正在慢病初筛、胃肠病就诊前总结、儿科影像及前列腺病理等场景落地 AI，将其作为临床专家的“第二双眼睛”，在辅助决策的同时保护医师的自主权。 (2:32) 各机构广泛采用“影子部署”（Shadow Deployments）在后台并行评估模型效果，这不仅规避了患者安全风险，也暴露出医疗系统长期缺乏工作流程基线测量标准的痛点。 (12:44) 针对独立大模型处理复杂病历时在时序逻辑和医学计算上的不足，行业正转向融合规则护栏的混合架构，同时积极应对效率提升反致工作负荷加重的“杰文斯悖论”与法律责任归属问题。 (16:53) 临床落地必须坚持深度集成电子病历（EHR）以规避点击疲劳；相比全自动系统，紧密贴合临床指南且绑定医生工作量的辅助型 AI 在当前医保报销体系下更具落地可行性。 (40:12)

Clinical artificial intelligence achieves practical utility as a deterministic, assistive tool integrated into electronic health record workflows rather than an autonomous replacement for clinical judgment. Panelists across health systems and biopharma emphasize that real-world deployment depends on establishing pre-deployment workflow baselines, resolving the liability burden placed on physicians, and aligning with existing fee-for-service reimbursement structures. Targeted Subspecialty Workflows and Clinical Augmentation (2:32) Health systems are targeting narrow operational bottlenecks with customized AI applications rather than broad, generalized models. Alexander Fedotov details AstraZeneca's EHR-integrated natural language processing platform that flags undiagnosed Chronic Kidney Disease and COPD, expanding into oncology workflows to identify missed genetic testing opportunities in metastatic breast and non-small cell lung cancer. This targeting addresses substantial care gaps, such as the fact that 90% of U.S. adults with CKD are unaware of their condition. Similarly, Senthil Kumaran describes MNGI Digestive Health piloting pre-visit summarization for Inflammatory Bowel Disease to synthesize labs, pathology, and records previously compiled manually by registered nurses. In pediatric and pathological subspecialties, standard off-the-shelf models are frequently inapplicable. Zachary Taylor reports that Cincinnati Children's develops on-premises models because pediatric cases constitute less than 10% of general radiology data, creating distinct needs for hand-bone age estimation, liver fibrosis detection, and cardiac volume segmentation for transplant donor matching. At UTMB, Peter McCaffrey implements the Ibex Prostate tool to provide algorithmic overread heat-maps that guard against reader fatigue in high-volume pathology, alongside AI referral triage to prioritize specialty queues in Epic according to clinician-defined acuity logic. Architectural Guardrails Beyond Standalone Generative Models (16:53) Panelists caution that unconstrained Large Language Models struggle with raw electronic health records, particularly concerning temporal event sequencing and precise mathematical calculations. Fedotov explains that AstraZeneca addresses this by adopting hybrid AI architectures, combining generative NLP capabilities with deterministic, rule-based algorithmic guardrails. Arturo Loaiza-Bonilla notes that while matching patients to clinical trials has become commoditized, the operational challenge lies in deploying neuro-symbolic AI that delivers deterministic, hallucination-free outputs at the point of care. To ensure model interpretability, Taylor advocates for bottom-up model designs that quantify uncertainty at the subtask level, giving clinicians dynamic confidence metrics instead of overwhelming them with unstructured data. In digital pathology, McCaffrey points out that AI validation cannot rely on simple binary outputs. Slide analysis contends with physical and optical variables—including Hematoxylin and Eosin staining variations, glass thickness, scanner hardware, and inter-pathologist annotation discrepancies—requiring spatial evaluation metrics such as Dice similarity coefficients over bounding boxes. Physician Liability, Jevons' Paradox, and Click Fatigue Operationalizing AI introduces institutional tensions regarding clinician workload and legal responsibility. McCaffrey highlights a structural paradox: health system efficiency teams expect clinicians to absorb 10% more patient visits when tools promise efficiency, while compliance departments demand that clinicians independently verify every AI output because legal liability remains entirely on the provider. McCaffrey and Taylor argue that treating AI like an untrusted assistant whose work must always be audited makes clinical practice unsustainable, comparing the situation to aviation, where pilots are not personally liable for internal mechanical components. Adoption also hinges on software interface design. Kumaran notes that although ambient AI reduces after-hours administrative 'pajama time,' introducing standalone, unintegrated applications fails because physicians already experience acute click fatigue. MNGI's strategy prioritizes purchasing validated vendor solutions and engineering bidirectional EHR pipelines across multiple hospital systems, reserving unintegrated standalone tools strictly for limited pilot environments led by clinical champions. Adoption Frameworks, Reimbursement Constraints, and Bias Mitigation (40:12) Fedotov outlines AstraZeneca's three-pillar framework for AI adoption: alignment with clinical practice guidelines, explicit clinician decision ownership, and financial viability. Assistive tools currently hold an economic advantage over autonomous systems because health insurance reimbursement models require an identified physician work component and personal liability; autonomous AI currently lacks a direct billing mechanism. Loaiza-Bonilla notes that without this physician work component, payer coverage remains unobtainable. To safeguard clinical safety and validate impact, institutions deploy models silently in 'shadow' environments alongside regular workflows. Kumaran reports MNGI ran a two-month dual-screen shadow pilot to refine pre-visit summarization against human baselines. McCaffrey observes that AI did not cause measurement challenges in healthcare, but rather exposed that health systems had long operated without established baseline metrics. To mitigate algorithmic bias—such as during imaging rollouts like the Swedish MASAI mammography trial, which reduced radiologist screening reading volume by 44% to 49%—Taylor emphasizes transparent training demographics and ongoing performance monitoring across disparate scanner hardware and patient populations. While clinical AI decision-support tools demonstrate measurable operational and diagnostic efficiencies, their long-term viability depends on structural reforms beyond algorithmic accuracy. Sustainable implementation requires health systems to establish baseline workflow metrics, deeply integrate tools within primary EHRs to prevent click fatigue, and resolve liability frameworks that currently hold individual clinicians accountable for auditing machine-generated recommendations.

在 Ai4 主办的研讨会上，临床专家与医疗技术管理者指出，临床决策支持 AI 的定位是增强而非取代医生判断；要跨越技术指标与临床现实之间的鸿沟，关键在于构建结合确定性规则的混合模型、深入嵌入现有工作流，并理顺执业法律责任与医保报销机制。 专科临床落地与混合智能架构转向 (2:32) 多所医疗系统正将 AI 工具部署于具体专科流程以解决诊疗痛点：AstraZeneca 借助嵌入 EHR 的 NLP 模型筛查未确诊的慢性肾病（在美国约 90% 的成年患者未意识到患病）与慢阻肺患者；MNGI Digestive Health 针对炎症性肠病门诊推行病历摘要生成；Cincinnati Children's 克服儿科仅占放射影像数据不足 10% 的限制，开发了专用的心脏体积与器官分割模型；UTMB 则利用 Ibex Prostate 算法热图防止高通量前列腺切片阅片疲劳。面对大语言模型（LLM）直接处理原始病历时在时间序列推理与精确医学计算上的缺陷，Alexander Fedotov 指出行业正加速转向将自然语言理解与规则算法结合的混合架构，以确保临床输出具备完全确定性且杜绝幻觉。 EHR 深度集成与静默部署基准测量 (12:44) 针对临床实施阻力，Senthil Kumaran 强调医生普遍承受严重的点击疲劳，脱离主系统的独立软件极难推广，MNGI 因此坚持将采购的成熟方案通过自建双向数据管道直接融入主 EHR 系统。在正式上线前，医疗系统普遍采用“静默部署”（Shadow Deployments）进行盲测校验；例如 MNGI 展开为期两个月的双屏对照试验，由医生比对人工摘要与算法输出，在不危及患者安全的前提下优化模型。Peter McCaffrey 进一步指出，AI 并未凭空制造测量危机，而是揭示了医疗机构长期缺乏工作流基准指标的客观现实；若机构无法明确没有 AI 时的流程基准，就无法客观量化技术部署前后的真实影响。 执业责任归属困境与辅助型支付闭环 在工作量减负方面，Peter McCaffrey 警示当前管理层面临杰文斯悖论：运营部门因 AI 带来 10% 的效率提升而要求医生增加 10% 的接诊负荷，但合规部门却因法律责任全在医生个人而要求其反复核查算法结果。他以航空飞行员无需直接为起落架机械故障承担个人刑责为例，呼吁行业必须对 AI 法律责任进行再分配，否则持续审查将加剧行医负担。在商业化层面，Alexander Fedotov 与 Arturo Loaiza-Bonilla 说明，现有医保报销体制必须关联持证医生的实际“工作组件”（Work Component）；因此，明确由医生作为最终责任主体的辅助型决策工具能够适配既有计费代码，而全自主系统则缺乏直接报销途径。Senthil Kumaran 补充道，环境声学 AI 虽然减少了医生夜间补写病历的“睡衣时间”（Pajama Time），但是否真能缩小诊疗差距仍有待观察。 细粒度病理验证与影像算法偏差治理 (38:37) 在数字病理验证中，Peter McCaffrey 解释真实临床切片受到苏木精-伊红（H&E）染色、玻璃厚度及不同扫描仪光学特性的多重物理干扰，模型验证无法简化为二元诊断是非题，必须采用 Dice 相似系数对空间边界框和图像瓦片进行细粒度定量评估。而在医学影像领域，Arturo Loaiza-Bonilla 引用了涵盖逾 8 万名受试者的瑞典 MASAI 随机临床试验结果，表明 AI 辅助乳腺钼靶筛查将放射科医生的阅片负荷降低了约 44% 至 49%，释放了操作活检的时间。Zachary Taylor 针对算法偏差提出治理方案，主张模型底层设计应拆解子任务并输出动态置信度指标，同时在正式推广前必须全面公开训练数据人群分布，跨不同厂商硬件与采集协议开展前瞻性验证以规避偏差输入。 总体而言，研讨专家指出医疗 AI 的主战场已由算法指标竞争转向临床信任建立与流程适配。尽管大规模试验已证实决策支持工具可显著分担阅片压力，但在如何厘清临床法律连带责任、验证环境 AI 对患者终点预后的实际改善，以及确保异构医疗数据下的模型稳健性等方面，行业仍面临诸多未决课题。

## Background 本视频由 Ai4 于 2026 年 8 月 21 日发布在 YouTube 平台上。视频记录了一场关于临床决策支持中 AI 应用的专家圆桌讨论，探讨领先医疗机构如何将 AI 融入放射学、病理学及床旁决策，并在确保医生自主权与临床判断力为核心的前提下，应对 EHR 集成、误报管理、医患信任及监管挑战。 ## TL;DR 来自学术医疗中心、专科诊所与制药企业的专家们分享了 AI 在慢病筛查、儿科影像定制、数字病理质控及门诊预访摘要中的实际落地案例。讨论深入剖析了医疗系统在规模化部署中面临的核心阻碍，包括临床基线指标缺失、影子部署验证、大语言模型幻觉以及一线医生面临的“文书增效与兜底责任”双重挤压。专家们一致强调，临床决策支持必须坚守“专家在环”与 EMR 深度集成原则，通过神经符号等混合架构与辅助型 AI 定位，构建多智能体协同生态以切实赋能临床判断而非取代医生。 ## Atmosphere & style 整场讨论务实清醒、专业严谨且富有临床一线的实战幽默感。主持人与嘉宾们以亲身经历轻松解构“AI 取代放射科医生”等过往行业炒作，但在触及医生法律责任归属、系统性警报疲劳与医疗测量盲区等体制难题时，对话展现出高度深刻与犀利的批判反思。整体节奏明快、观点交锋热烈，充满跨界协作与解决真问题的专业求实氛围。 ## Key takeaways - [00:00](https://www.youtube.com/watch?v=F8GG9gCmw1g&t=0s) 临床决策 AI 的核心定位是“赋能与增强”而非“取代医生”，其关键价值在于充当第二双眼睛，协助识别照护缺口并抵御阅片疲劳。 - [02:32](https://www.youtube.com/watch?v=F8GG9gCmw1g&t=152s) 多模态 AI 与 EHR 深度结合可高效筛查慢性肾脏病（CKD）及肿瘤基因检测缺失等长期存在的漏诊与诊断不足痛点。 - [06:34](https://www.youtube.com/watch?v=F8GG9gCmw1g&t=394s) 儿科等小众专科在通用成人商业模型中表现欠佳，依赖本地化自研模型能够针对专属临床痛点进行定制化攻关。 - [11:00](https://www.youtube.com/watch?v=F8GG9gCmw1g&t=660s) 设立首席医疗 AI 官（CMAIO）至关重要，该角色能充当翻译官，架起数据科学技术指标与一线临床认知之间的沟通桥梁。 - [12:43](https://www.youtube.com/watch?v=F8GG9gCmw1g&t=763s) 在正式投产前推行影子部署（Shadow Deployment），是暴露真实世界数据盲区、捕获离群值并构建微调数据集的核心手段。 - [15:29](https://www.youtube.com/watch?v=F8GG9gCmw1g&t=929s) AI 部署遇到的评估阻力并非技术凭空产生，而是无情揭露了医疗系统长期以来缺乏业务基线与量化指标的测量盲区。 - [16:53](https://www.youtube.com/watch?v=F8GG9gCmw1g&t=1013s) 大语言模型在医疗时序和精确数值计算上存在软肋，融合确定性规则的神经符号（Neuro-symbolic）混合 AI 架构是根除幻觉的关键路径。 - [20:16](https://www.youtube.com/watch?v=F8GG9gCmw1g&t=1216s) 既要求医生提升看诊效率又要求其为 AI 黑盒输出承担无限核对责任的制度是不可持续的，必须在系统与法规层面重构责任归属。 - [23:45](https://www.youtube.com/watch?v=F8GG9gCmw1g&t=1425s) 人机交互设计需警惕即时结论弹窗诱发的自动化偏差，通过延迟呈现或双向切换机制维持医生的独立临床判断训练。 - [32:41](https://www.youtube.com/watch?v=F8GG9gCmw1g&t=1961s) 医生应当扮演类似“空中交通管制员”的高阶裁决角色，超出临床精力边界的底层海量数据验证必须由系统承担兜底责任。 - [36:37](https://www.youtube.com/watch?v=F8GG9gCmw1g&t=2197s) 临床 AI 落地必须坚持“EMR 优先”原则，任何需要跳出日常系统或增加额外操作的独立工具都会因“点击疲劳”导致采纳失败。 - [40:12](https://www.youtube.com/watch?v=F8GG9gCmw1g&t=2412s) 医疗 AI 商业化与采纳取决于三大支柱：临床指南契合度、明确的决策权责归属以及清晰的医保报销路径。 - [40:12](https://www.youtube.com/watch?v=F8GG9gCmw1g&t=2412s) 在现行计费体系下，将医生保留在闭环中的辅助型 AI 比自主型 AI 更容易获得医保与商保的审批报销。 - [43:26](https://www.youtube.com/watch?v=F8GG9gCmw1g&t=2606s) 未来临床决策支持体系不会依赖单一全能的庞大单体模型，而是转向协同多源异构数据的多智能体（Agents）生态框架。 ## Detailed walkthrough ### [00:00](https://www.youtube.com/watch?v=F8GG9gCmw1g&t=0s) 圆桌开场与嘉宾背景介绍 主持人 Arturo Loaiza-Bonilla 首先开场致辞。作为一名肿瘤内科医生，他不仅担任宾夕法尼亚州拥有 16 家医院的 St. Luke's University Health Network 的血液肿瘤科主任，同时也是 Massive Bio 的联合创始人兼首席医疗 AI 官（CMAIO），致力于利用 AI 实现肿瘤临床试验的精准匹配。他幽默地调侃自己今天坐着主持还是站着主持都行，随后点明本次讨论的核心主旨：通过各位专家带来的实际落地案例，揭开 AI 在临床实践中的神秘面纱，解答临床医生最核心的关切——如何以最佳方式落地 AI 技术，赋能医生决策而非“取代医生”。 ```mermaid flowchart TD accTitle: 临床 AI 专家圆桌背景分布 subgraph S1["临床与学术医疗体系"] A["Arturo Loaiza-Bonilla<br/>St. Luke's / Massive Bio<br/>(肿瘤学 / 试验匹配)"] B["Peter McCaffrey<br/>UTMB / UT System<br/>(病理学 / 数字化转型)"] D["Zachary Taylor<br/>Cincinnati Children's<br/>(儿科影像 AI)"] end subgraph S2["专科诊所与制药企业"] C["Senthil Kumaran<br/>MNGI Digestive Health<br/>(消化专科 / 流程治理)"] E["Alexander Fedotov<br/>AstraZeneca<br/>(多模态 AI / 慢病识别)"] end ``` 随后各位嘉宾依次进行了自我介绍： - **Peter McCaffrey**：病理学背景，担任德克萨斯大学医学分部（UTMB）的首席数字与 AI 官，并兼任德州大学系统（UT System）AI 协作组的主席。 - **Senthil Kumaran**：明尼苏达消化内科（MNGI Digestive Health）的首席信息官（CIO）。该专科机构在明尼阿波利斯和圣保罗双子城及周边拥有 10 家日间手术中心（ASC）、11 家诊所和 160 多名经过严格胃肠专科培训的医师，是全美规模最大的私立消化专科诊所系统。 - **Zachary Taylor**：来自辛辛那提儿童医院医学中心（Cincinnati Children's Hospital Medical Center）的人工智能与影像研究中心，专注于研发、训练并落地临床影像 AI 模型。 - **Alexander Fedotov**：阿斯利康（AstraZeneca）企业 AI 负责人。他拥有遗传学博士背景及数据科学经验，目前主导多模态 AI 和自然语言处理（NLP）技术在大规模风险患者识别场景中的落地与系统合作。 --- ### [02:32](https://www.youtube.com/watch?v=F8GG9gCmw1g&t=152s) 利用多模态 AI 挖掘未确诊慢病与肿瘤基因检测缺口 Alexander Fedotov 首先分享了阿斯利康在临床决策支持与慢病管理方面的核心应用。在切入病理和放射影像之前，他重点剖析了如何解决慢性肾脏病（CKD）和慢性阻塞性肺疾病（COPD）等慢性病“漏诊与诊断不足”的行业痛点。在全美范围内，高达 90% 的 CKD 成年患者对自己患病毫不知情，如果单靠初级保健医生（PCP）或专科医生手动逐份翻阅病历去筛选这些潜在患者，在现实中几乎是不可能的。 为了应对这一挑战，他们与技术供应商合作，开发能够原生嵌入电子健康档案（EHR/EMR）的 AI 临床推理系统。该系统能够同时解析结构化数据与非结构化临床笔记，自动将高风险患者标记出来并推送给接诊医生，提示其进行精准转诊、开具针对性的实验室检查或调整治疗路径。Alexander 强调，这种从文本与多维病历中寻找潜在风险的逻辑同样被复用到肿瘤临床流程中，例如针对转移性乳腺癌和非小细胞肺癌（NSCLC）患者，筛查其是否遗漏了必要的基因检测（Genetic Testing），进而推动其进入规范化的诊疗路径。 --- ### [04:46](https://www.youtube.com/watch?v=F8GG9gCmw1g&t=286s) 消化专科中的门诊预访摘要与人机协作 Senthil Kumaran 针对专科诊疗中的治理评估和实际落地进行了分享。面对 Arturo 称其为“胃肠科医生”的口误，Senthil 幽默地澄清道：“我一看到血就会晕过去，虽然在医疗行业工作了 25 年，但我绝不直接参与医疗救治，所以你做结肠镜可千万别找我排班。” 回归正题，Senthil 介绍了他们在炎症性肠病（IBD）亚专科门诊中试点的“预访摘要”（Pre-visit Summarization）AI 工具： ```mermaid flowchart LR accTitle: IBD 预访摘要生成与临床把关流程 subgraph Input["原始数据收集"] A["病理 / 检验报告"] B["血液化验结果"] C["患者历史病历"] end subgraph Process["摘要与校验"] D["AI 临床摘要工具<br/>(自动提取综合信息)"] E["专科护士 (RN)<br/>(Human-in-the-loop 校验)"] end subgraph Output["赋能临床"] F["结构化摘要文档"] G["主诊医生查看<br/>(识别遗漏的照护缺口)"] end Input --> D D --> E E --> F F --> G ``` 在以往流程中，IBD 专科护士（RN）在患者就诊前需要耗费大量时间手工阅读并综合分析过往的所有病理报告、化验单、血液检查以及病史，手动生成就诊摘要。引入 AI 工具后，系统能够自动化完成大部分繁重的提炼工作。Senthil 强调，团队始终贯彻“人机协同”（Human-in-the-loop）的原则，由专科护士把关弥补 AI 的不足。这款工具不仅大幅解放了护士的生产力，更充当了“第二双眼睛”，有效识别出医生或照护团队偶尔遗漏的照护缺口（Care gaps）。 --- ### [06:34](https://www.youtube.com/watch?v=F8GG9gCmw1g&t=394s) 儿科专科影像 AI 的定制化研发与落地 Zachary Taylor 结合辛辛那提儿童医院的实践，阐述了本地化（On-prem）自研模型在儿科特定场景下的独特优势。他指出，由于儿科病例仅占全部放射学影像数据的不足 10%，市面上绝大多数现成的商业模型（如器官分割）都是针对成人解剖结构训练的，在儿童身上表现欠佳。拥有本地研发部署能力使他们能够根据医生提出的特定痛点，针对儿科专属的“低垂果实”进行定制化攻关。 Zachary 列举了一系列正在落地与开发的实际应用： - **心脏移植全心脏体积分割**：精准计算供体与受体的心脏体积，为心脏移植手术选择最匹配的器官供体。 - **骨龄自动评估**：对儿童手部 X 光片骨骼发育情况进行自动化测定。 - **数字病理延伸探索**：包括嗜酸性粒细胞（Eosinophil）自动计数，以及肝纤维化（Liver Fibrosis）检测等模型。 --- ### [08:21](https://www.youtube.com/watch?v=F8GG9gCmw1g&t=501s) 数字病理抗疲劳质控与门诊转诊智能分诊 Peter McCaffrey 分别从病理影像与非影像运营两个维度分享了 UTMB 的落地案例： - **数字病理影像工具（Ibex Prostate）**：大约一年半前，UTMB 部署了用于前列腺穿刺活检切片的辅助复核工具。在临床病理诊断中，寻找极微小的肿瘤病灶（Microfoci of neoplasia）极其消耗精力。病理医生在阅片到第 200 个核心组织块时，极易因疲劳导致注意力衰退。通过该工具，医生一键即可生成覆盖组织切片的“过读热力图”（Overread Heatmap），精准警示潜在病灶，形成一道对抗阅片疲劳衰减的工程化防护栏。 - **门诊转诊智能排队（Referral Triage）**：UTMB 作为大型安全网医院（Safety Net Hospital），覆盖范围极广，面临严重的专科转诊积压问题。以往肾脏科、风湿免疫科、结肠镜检查、感染科和内分泌科的转诊预约仅由排班技师按照“先到先得”（FIFO）的机械顺序处理。现在，各专科医生团队制定出明确的高/低危分级逻辑，AI 将这套临床规则映射到患者病历中，自动在 Epic 电子病历系统中打上急危重度标签，成功将原本无序的等待队列重构成优先级队列（Priority Queue），有效防止高危患者在漫长的转诊等待中“掉出安全网”。 在讨论尾声，主持人 Arturo 总结道，无论是肿瘤、IBD 还是慢病筛查，这些工具都在深度交织并改变临床实践。然而，“创新的推进必须建立在信任的基础之上”（"Innovation moves at the speed of trust"）。随着 AI 工具逐渐渗透到异常预警、病情恶化预测及脓毒症（Sepsis）告警等核心环节，如何在防止误诊、避免医生技能退化（De-skilling）的同时，管理好临床各方对于 AI 预警的心理预期，成为了亟待深入探讨的关键议题。 ### [11:00](https://www.youtube.com/watch?v=F8GG9gCmw1g&t=660s) 沟通鸿沟、可解释性与首席医疗 AI 官的必要性 在探讨临床决策支持工具如何落地时，Zachary Taylor 指出，确保医生及所有一线医护人员理解 AI 技术的边界至关重要——他们必须清楚该工具能做什么、不能做什么，以及何时该相信其建议、何时必须对 AI 生成的结果打上“星号”保持审慎。这一挑战本质上是双向的：研发人员与数据科学家在设计输出时，必须意识到最终阅读这些内容的是临床一线人员，因此呈现形式必须让非数据科学专业人员一目了然；与此同时，医院也必须对医生展开针对性教育，普及工具的具体能力与局限。 主持人兼肿瘤学家 Arturo Loaiza-Bonilla 对此深有共鸣。他坦言自己跨界于临床与技术两端，深知大部分临床同事对核心技术概念毫无认知，比如根本不知道 F1 分数（F1 score）代表什么、何为数据过拟合（overfitting），或是训练模型到底使用了何种数据集。因此他极力主张，当今的医疗机构除了设立首席信息安全官（CISO）之外，更应设立**首席医疗 AI 官（Chief Medical AI Officer, CMAIO）**。这一角色能够作为“翻译官”，将晦涩的数据科学概念转化为临床语境下有意义的语言，既能打消医生对“被取代”的恐慌，又能指导团队高效利用工具。 - **双向对齐要求**：数据科学家需降低技术黑话门槛，提供直观易懂的可视化/结构化输出；临床端需普及模型能力范畴与潜在陷阱。 - **关键认知断层**：临床一线人员普遍缺乏对评估指标（如准确率、召回率、F1 score）及训练偏差/过拟合的理解。 - **组织架构创新**：设立 CMAIO 职能，架起技术开发与临床落地之间的桥梁。 --- ### [12:43](https://www.youtube.com/watch?v=F8GG9gCmw1g&t=763s) 影子部署（Shadow Deployment）与“放射科被淘汰”的炒作泡沫 关于临床上线前的新兴验证策略——影子部署（Shadow Deployment，即在系统侧台平行静默运行而不直接干预决策），Senthil Kumaran 分享了 MNGI Digestive Health 的就诊前摘要生成工具（previsit summarization tool）实战案例。他们开展了将近两个月的影子部署：医生在双屏环境下工作，一边照常自行准备患者就诊前摘要，另一边观察 AI 同步生成的摘要。团队全程追踪 AI 在处理哪些病历时速度更快、哪些场景会出现偏差，并将这些差异反哺给算法进行迭代优化。 ```mermaid flowchart TD accTitle: 医疗 AI 影子部署双轨验证流程 subgraph S1["临床工作站（双屏验证）"] A["临床医生自主编写摘要"] B["AI 影子系统同步生成摘要"] end subgraph S2["性能评估与优化"] C["对比分析：准确率 / 遗漏 / 异常值"] D["回传边缘案例与微调数据集"] end subgraph S3["最终决策"] E["临床专家在环审核与决策"] F["正式切入生产环境 (Production)"] end A --> C B --> C C --> D D --> B C --> E E --> F ``` Kumaran 提醒医生群体，任何 AI 工具都不可能做到 100% 绝对不出错。他特意回顾了 10 年前类似 AI 峰会上的极端预言——当时业界有知名学者断言“未来 5 年内放射科医生将被完全取代，千万别让孩子去学放射学”，就好比早年预言“10 年内所有人都会开上全自动驾驶（FSD）汽车”。然而现实是，今天的放射科医生依然开着燃油车（gas-guzzlers）去医院按时上班。AI 工具本质上只是“第二双眼睛”，系统可以提供赋能，但最终决策与系统训练必须依靠临床“专家在环”（experts in the loop）；在未经数月充分验证前，绝不可草率将其投入生产环境。 Zachary Taylor 补充道，提前规划影子部署对于捕获离群值（outliers）和特殊患者亚群具有无可替代的价值。实际运行中常常会暴露诸如医学影像扫描质量问题，或是训练数据未曾覆盖的特定人群特征。将这些真实世界中的盲点收集起来构建更高质量的微调（fine-tuning）数据集，才是影子部署的核心收益。 --- ### [15:29](https://www.youtube.com/watch?v=F8GG9gCmw1g&t=929s) 医疗机构的衡量困境：AI 揭露了长期存在的指标缺失 Peter McCaffrey 从数字化与医疗运营全局出发，深入剖析了规模化部署中的阻力。他指出，医疗组织内部极为复杂，往往在“概念验证（PoC）- 规模化上线 - 回撤修改”之间反复摇摆。在技术评估层面上，检验模型输出与真实标准（ground truth）的吻合度，无论是部署前还是部署后的定期审计，在方法论上并不颠覆过往的质控经验；真正的挑战在于**需要同时评估的模型数量与业务场景规模是史无前例的**。 更为根本的症结在于业务指标衡量的失灵。McCaffrey 直言不讳地指出：“AI 并没有凭空制造测量难题，它只是无情地揭露了医疗系统长期以来一直存在的测量盲区（measurement problem）。”如果医院管理者被问及“在没有引入 AI 之前，工作流 A 的具体量化影响是什么”，多数人根本给不出明确答案。既然连基线（baseline）和预期的干预效果都无法清晰量化，那么评判 AI 工具的落地成败与投资回报率自然也就无从谈起，这直接导致了落地部署中的进退失据与警报疲劳（alert fatigue）。 --- ### [16:53](https://www.youtube.com/watch?v=F8GG9gCmw1g&t=1013s) 幻觉、时间序列与神经符号（Neuro-symbolic）混合架构 针对医疗 AI 领域常出现的“治愈癌症、解决一切问题”等过度炒作，Alexander Fedotov 从大药企及数据科学视角分享了如何让预期回归理性。他强调，建立临床信任的关键在于追求卓越的同时，坦诚公开当前技术的短板。以大语言模型（LLMs）为例，行业早期对其满怀激情，但当尝试把完整的患者电子健康档案（EHR）喂给大模型时，其软肋迅速暴露： - **时间序列理解困难**：处理复杂的病程发展、用药先后顺序等时序数据时常常混乱； - **精准数值计算能力欠缺**：涉及药物剂量、临床评分等数学推导时不可靠； - **幻觉风险**：在缺乏确定性规则约束时容易给出看似合理却错误的断言。 Fedotov 指出，当前最具前景的方向是转向**混合 AI 架构（Hybrid AI Models）**，将 LLMs 的语言理解能力与确定性的基于规则的算法（rule-based algorithms）相融合，从而在数据科学家与临床医生之间达成稳固的互信基础。 Arturo Loaiza-Bonilla 也以 Massive Bio 在肿瘤临床试验匹配中的经验佐证了这一点。他表示，临床试验匹配如今正逐步商品化，真正的核心壁垒在于部署架构中引入**神经符号 AI（neuro-symbolic AI）**。该方法能实现完全确定性的输出（deterministic output），在患者就诊的即时场景下坚决杜绝模型幻觉，确保推荐方案精准符合试验赞助方与入排标准的要求。Loaiza-Bonilla 调侃道，如果 AI 真能根除癌症并彻底取代他的医生工作，他“乐意被淘汰”并全职投入 AI 创业；但现实是，我们必须正视杰文斯悖论（Jevons paradox）——工具效率的提升往往不仅没有减少工作量，反而让医生更深地卷入到海量患者的救治与交互中。 --- ### [20:16](https://www.youtube.com/watch?v=F8GG9gCmw1g&t=1216s) 责任归属的双重夹击：让临床医生承担系统兜底是不可持续的 针对环境智能语音听写（ambient scribes）及门诊临床应用，Peter McCaffrey 剖析了一线医生当前面临的制度性困局。他强调，要使医疗 AI 具备可持续性，监管与责任认定机制的成熟必须走在前面。当前医院管理层往往对一线医生施加相互矛盾的“双重夹逼”： | 组织诉求来源 | 对医生的施压机制 | 带来的实际后果 | |:--- |:--- |:--- | | **运营与效率部门** | “AI 帮你的文书工作提升了 10% 的效率，为什么你没有多接诊 10% 的患者？” | 诊疗负荷与看诊指标直接增加 | | **合规与法务部门** | “AI 的生成结果可能会出错，你必须二次核对、三次核对，最终签字责任全由你个人承担。” | 凭空增加了一项繁重的“模型校验（Validation）”工作 | McCaffrey 尖锐地批评道，这种做法不仅剥夺了技术带来的效率红利，还把“验证黑盒输出”变成了一线医生被迫承担的额外新工种，“这种制度设计让临床工作完全无法维系（impossible）”。 他以航空领域作为经典类比：飞行员固然是飞机的驾驶操作者，但当起落架内部的电子线路或元器件发生故障时，法律和系统决不会让飞行员承担内部硬件设计的责任。医疗系统同样必须界定临床医生的责任边界，将部分系统性缺陷的法律责任（liability）转移至算法系统与基础设施层面。这种顶层制度的重构并非单家医院自身所能解决，必须依赖全行业乃至监管政策层面的系统性革新。 ### [21:36](https://www.youtube.com/watch?v=F8GG9gCmw1g&t=1296s) 责任归属、验证哲学与“睡衣时间”迷思 嘉宾们深入探讨了临床 AI 工具落地时面临的责任转移与验证逻辑重塑。从实际操作层面看，如果 AI 被定义为一款节省时间的工具，但医生却不得不反复核对（double-check, triple-check）它的输出，那它就变成了“一个不可靠的秘书，你得时刻去检查他的工作”，反而增加了认知负荷。为了建立信任，应当将复杂的临床任务拆解成更小的颗粒度，在每个具体任务上逐步积累信任，并明确界定不同任务层级上的法律与临床责任归属（"where does the buck stop"）。 随着各类工具的性能日益增强，现场也抛出了一个核心拷问：AI 真的让患者获益了吗？目前在环境感知 AI（Ambient AI，即环境智能听录与笔录工具）领域，它确实大幅减少了医生的“睡衣时间”（pajama time——指医生下班回家后，在晚餐甚至深夜时间不得不继续补写病历或做门诊前预习摘要的时间）。然而，这些工具是否真正赋能了患者？是否弥合了医疗照护的缺口（care gaps）？是否改善了医保与健康计划的支付效益？这些根本问题仍处于探索阶段。 ```mermaid flowchart TD accTitle: 临床 AI 实施挑战与人机交互设计 accDescription: 展示从责任归属、人机交互时机到可追溯性与监管落地的全流程逻辑。 subgraph ProblemSpace["当前痛点与验证"] A["重复核对负担<br/>(不可靠的秘书效应)"] --> B["睡衣时间 (Pajama Time)<br/>文档减负明确"] B -.-> C["患者实际获益/照护缺口<br/>(仍待验证)"] end subgraph HCIDesign["人机交互 (HCI) 设计"] D["即时呈现 AI 结论"] -->|诱发| E["自动化偏差 (Automation Bias)"] F["延迟呈现 / 双向切换 (Toggle)"] -->|促进| G["医生独立判断 + AI 辅助验证"] end subgraph SolutionSpace["可解释性与监管落地"] H["非结构化/结构化数据结合"] --> I["概率型 + 确定性模型"] I --> J["源头追溯 (Traceability)<br/>(临床笔记/指南/计算规则)"] J --> K["满足监管要求与临床信任"] end ``` --- ### [23:45](https://www.youtube.com/watch?v=F8GG9gCmw1g&t=1425s) 人机交互设计、自动化偏差与医生主导地位 从科学与临床证据的角度来看，目前的 AI 系统绝对无法替代临床医生的专业判断。在整套医疗决策体系中，人类社会必须确保临床医生始终处于决策的核心主导地位（"keeping clinicians in the loop... at the helm of all the decision-making processes"）。 人机交互（HCI）界面的具体呈现方式，会对人类在决策过程中的参与度产生实质影响： - **即时呈现的风险**：如果在医生做出判断的瞬间同时弹出算法结果，极易诱发“自动化偏差”（automation bias），导致医生在后续病例中产生被动依赖； - **延迟呈现与双向切换**：如果适度延迟 AI 建议的输出，或者设计允许医生在自身独立判断与 AI 输出之间来回切换（toggle back and forth）的交互机制，反而能够强化临床思维训练，达到理想的人机协同状态。 --- ### [25:36](https://www.youtube.com/watch?v=F8GG9gCmw1g&t=1536s) 市场过度饱和、投资回报率（ROI）与监管困境 面对市场上铺天盖地的 Ambient AI 供应商，专家直言该领域已经极度拥挤（"unless you're creating quantum computing on the side, we have enough"），行业真正需要的是能切实融入工作流并带来明确 ROI 的创新。虽然“睡衣时间”是一个可量化的减负指标（管理者甚至会强制有严重加班记录的医生使用转录工具），但在 Medicare 等医保报销额度固定、“盘子并未变大”的现实下，医院采购决策高度依赖明确的财务与成效量规。目前除了类似 OpenEvidence 那种依靠赞助或药企支持的商业模式外，多数工具仍难以证明其落地投资回报。 与此同时，监管框架与临床责任的矛盾愈发尖锐。例如某些平台（如部分处方生成或辅助决策工具）已经触及开具处方等核心环节。在实际操作中，用户往往像面对 iPhone 系统更新一样不加阅读地点击“同意/免责声明”，但一旦出现误诊或失误，面临法律诉讼和患者质询的依然是线下的临床机构与医生。如何在监管层面界定这些涉及生死决策（life-death decisions）的算法工具，是行业必须直面的关键课题。 --- ### [27:51](https://www.youtube.com/watch?v=F8GG9gCmw1g&t=1671s) 可追溯性架构与多模态数据的决策支持 针对临床决策支持的可解释性（Explainability）与可追溯性（Traceability），不同医学专科存在路径差异： - **EHR 与文本推理的可追溯实现**：相比放射和病理影像的端到端黑盒特性，基于电子病历（EHR）的临床推理更容易通过架构设计实现追溯。通过结合处理非结构化与结构化数据，并将概率型模型（Probabilistic AI）与确定性模型（Deterministic AI）相融合，AI 可以将其给出的每项建议精确链接到具体的病历记录段落、临床指南条目或特定的生理指标计算公式，从而达到监管机构认可的安全可用标准。 - **影像学与注意力盲区（Inattentive Blindness）**：放射学中著名的“大猩猩实验”表明，83% 的放射科医生在阅片时可能会因非注意盲视而漏掉肺部 CT 中的大猩猩图像。AI 固然可以作为“第二双眼睛”防止遗漏，但防范 AI 本身的假阴性同样依赖于治理委员会（Governance Council）对技术边界的清醒把控。随着数字病理学（Digital Pathology）的普及，行业正借鉴放射科数十年的经验，重新构建数字病理的基础设施与质控标准。 - **专科数据民主化与长文档摘要**：在胃肠（GI）等专科实践中，医生面对的长达 100 多页的复杂病理报告，往往只有不到 10 分钟的时间进行查阅与决策。借鉴 Perplexity 或 OpenEvidence 这种具备明确信源标注、低幻觉的检索与摘要架构，AI 能够对海量报告进行结构化索引、导航与提炼，帮助应对偏远地区专科医生短缺、炎症性肠病（IBD）等患者预约排队动辄 2 至 4 个月导致的病情延误危机。 ### [32:41](https://www.youtube.com/watch?v=F8GG9gCmw1g&t=1961s) 临床判断力、信息过载与系统责任归属 针对关乎生死抉择的临床场景，嘉宾们强调引入 AI 工具必须配以强有力的治理框架，确保其核心作用是“增强而非取代临床判断”。Peter McCaffrey 阐述了自己的核心理念：医疗的理想状态应当是将医生的主要时间分配给“临床判断”，而不是消耗在文书等管理杂务（clerical overhead）上。过去在“有意义使用”（Meaningful Use）等政策推行下，医生往往要先完成海量的事实调查、数据合成、汇总与文档记录，最后才能分出一点点精力来进行真正的临床判断。尽管 AI 能够通过提供可解释的来源与推理依据来减轻负担，但当前临床面临的另一个巨大挑战是——需要判断的事物正呈指数级爆发式增长（如层出不穷的化验、多模态检查、甚至是患者在家使用的 Butterfly iQ 等便携式掌上超声设备）。 如果医疗系统不划定明确的判断边界，医生将陷入一种极具讽刺意味的困境：虽然文书负担因政策改善稍微减少了一点，但随后却迎来了 10 倍于以往的双重、三重事实核查与验证负担。Peter McCaffrey 指出，决策空间必须设立边界，超出该边界的任务必须由系统承担责任（liability）。面对庞大的业务量与患者需求，很多工作不可避免地要交由系统执行；医生应当扮演“空中交通管制员”（air traffic control）的角色去审查与提出意见，而不是被迫去深挖底层八层架构是否存在潜在漏洞并为此承担无限责任。 ```mermaid flowchart TD accTitle: 临床决策边界与责任分配模型 subgraph ClinicianScope["医生核心职责 (空中交通管制员)"] A["高阶临床判断 (Judgment)"] B["方案审查与综合裁决 (Opine)"] end subgraph Boundary["明确决策边界 (Decision Boundary)"] C{"任务是否超出判断边界？"} end subgraph SystemScope["系统与 AI 责任边界 (Liable Entity)"] D["海量数据初步合成与分析"] E["底层多层数据验证与追溯"] F["系统层承担相应法律/运行责任"] end A --> C C -- "常规高阶诊疗" --> B C -- "超出医生精力负荷/底层技术验证" --> SystemScope ``` --- ### [34:41](https://www.youtube.com/watch?v=F8GG9gCmw1g&t=2081s) 多模态黑盒挑战与不确定性度量 多模态（Multimodality）技术的普及正在加速改变临床 AI 模型，算法开始将生物学指标、病理数据、基因组学与影像学深度结合。然而，这种端到端的复杂黑盒常常让人难以洞悉内部机理。Arturo Loaiza-Bonilla 提出了一个深刻的疑问：“面对我们自己都无法完全理解的事物，我们该如何进行临床判断？”他举例说明，在肺癌诊疗中，按照常规生物标志物指南和随机对照试验（RCT）证据，某种疗法可能已被证明对该群体无效；但某些前沿的影像组学（radiomics）模型却能识别出血管排列细微特征等特定模式，并断言“唯独这个患者能产生响应”。临床医生面临着究竟是遵循传统循证医学还是冒险追求个体化（n-of-1）极致疗法的两难选择。 Zachary Taylor 对此表示赞同，指出临床决策支持是一项极为庞大且终极的任务，由无数个细分子任务构成，而目前许多子任务的模型解释性（explainability）和可解释性（interpretability）依然严重不足。要解决这个问题，必须采取“自底向上”（bottom-up）的构建方式，开发能够量化“不确定性”（uncertainty）和在不同子决策环节提供“置信度等级”（confidence levels）的新评估指标，从而在面对无限涌入的医疗数据时，精准理清模型究竟提取了哪些深层见解。 --- ### [36:37](https://www.youtube.com/watch?v=F8GG9gCmw1g&t=2197s) 自研还是外采：EMR 深度集成与点击疲劳 讨论转向具体的落地实践——在“自研还是外采”（Buy vs. Build）以及“EMR 优先还是独立工具”（EMR First vs. Standalone）之间如何权衡。Senthil Kumaran 直言，如果不选择“EMR 优先”，在实际落地中必然会沦为笑柄。临床医生早已深受“点击疲劳”（click fatigue）的困扰，任何需要跳出日常系统、增加额外点击的独立工具都会导致临床采纳率断崖式下跌。 在多机构协同的实际运作中，整合的复杂性尤为突出。Senthil Kumaran 解释道，其机构的医生经常需要往返穿梭于多家不同的医院，各医院所采用的 EMR 系统各不相同，技术团队每天都需要投入大量精力去处理多套系统之间数据的拉取（pull）与推送（push），背负着极重的系统集成包袱。因此在策略上： - **外采定制优于从零自研**：倾向于引入已有成熟成果的外部供应商解决方案，进行定制后再集成到核心 EMR 中； - **试点与推广阶段分步走**：在概念验证或小规模试点阶段，可能依靠少数几位积极的带头医生（physician champions）使用独立工具跑通流程；但若要在整个临床体系全面推广，深度的 EMR 原生集成是不可逾越的前提条件。 --- ### [38:37](https://www.youtube.com/watch?v=F8GG9gCmw1g&t=2317s) 数字病理学的多中心异质性与全切片验证 在病理 AI 落地方面，全切片图像（WSI, Whole Slide Imaging）面临着严峻的数据泛化挑战。Arturo Loaiza-Bonilla 指出，即使算法在 Memorial Sloan Kettering 或 MD Anderson 等顶级癌症中心数以亿计的切片上完成了训练，但由于不同机构使用的切片扫描相机不同、苏木精-伊红（H&E）染色工艺存在差异，模型往往无法直接平移适配到本地的患者人群中。 Peter McCaffrey 从病理学与数字 AI 的角度剖析了这一验证过程。与外界认为“AI 只需直接吐出一个诊断、医生判断是否同意”的刻板印象不同，真正的数字病理验证极为严密细致： - **可量化的指标体系**：必须明确算法的检测框（bounding box）、切片图块（tiles）分布，并利用 Dice 系数（Dice coefficient）等硬性数学指标进行多维度性能分析； - **多重变量的层层叠加**：数字病理不仅存在类似放射科的扫描仪硬件差异，还叠加了制片技术差异、玻璃载玻片物理差异，以及不同病理学家在缺乏绝对基准（ground truth）时的标注差异；因此在跨网络部署前，必须建立明确的金标准病例集，严格验证染色与图像采集工艺对模型识别率的影响。 --- ### [40:12](https://www.youtube.com/watch?v=F8GG9gCmw1g&t=2412s) 临床采纳三大支柱与辅助型 AI 的医保报销优势 Alexander Fedotov 结合在阿斯利康（AstraZeneca）及跨机构转化的实践经验，总结了评估与引入医疗 AI 解决方案的“三大核心支柱”： | 评估支柱 | 核心要求与落地逻辑 | |:--- |:--- | | **临床指南契合度**<br>*(Guideline Alignment)* | 解决方案必须紧密贴合现行的临床指南框架。指南是医生构建诊疗决策的核心骨架，算法若脱离该骨架将极难获得认可。 | | **决策权责归属**<br>*(Clear Ownership)* | 必须在临床工作流中明确最终医疗决策的责任人。“没有明确的责任人，就无法真正融入工作流”（No owner, no integration）。 | | **医保报销机制**<br>*(Reimbursement)* | 方案必须具备明确的支付途径。当前医疗支付体系高度依赖医生的直接参与，缺乏报销机制的产品无法在临床持久生存。 | Alexander Fedotov 与 Arturo Loaiza-Bonilla 进一步指出，业内存在一个反直觉却至关重要的现象：在当前的医保报销环境下，**辅助型 AI 工具（Assistive AI）的报销情况远好于自主型 AI（Autonomous AI）**。这是因为现行计费体系完全绑定在医生实际付出的医疗劳动组件（work component）和承担的法律责任之上；由于医生必须在场并承担最终责任、向患者解释检查必要性，这种将人保留在闭环中的辅助型工具更容易通过医保和商保的审批报销。 --- ### [42:03](https://www.youtube.com/watch?v=F8GG9gCmw1g&t=2523s) 算法偏倚、数据多样性与大规模筛查临床试验 在谈及影像 AI 的算法偏倚（algorithm bias）时，Arturo Loaiza-Bonilla 引用了具有里程碑意义的 MASAI 随机对照临床试验（涉及 80,000 名患者的乳腺钼靶筛查研究）。该研究表明，AI 辅助筛查能够优化放射科医生约 40%~49% 的读片时间，从而实现仅需单人读片即可维持高质量，释放出的时间可让医生执行更多活检等实质性医疗操作。 Zachary Taylor 针对影像算法偏倚的控制策略进行了深入阐述： - **全面审查训练数据源**：明确训练数据的来源与构成是防控偏倚的第一步，警惕“垃圾进，垃圾出”（Garbage in, garbage out）效应； - **全生命周期监控与前置验证**：即使训练数据经过高度清洗与筛选，上线后依然必须部署严格的动态监控系统； - **跨人群与跨设备的大规模验证研究**：必须在不同的人口统计学特征（demographics）和不同品牌/型号的影像扫描仪（scanners）之间进行全面的泛化性测试，持续提升训练数据集的多样性，确保算法在多源异构环境下的稳定性和公平性。 ### [43:26](https://www.youtube.com/watch?v=F8GG9gCmw1g&t=2606s) 总结与展望：打破单体架构，共建多智能体临床决策支持生态 圆桌讨论在倒计时仅剩数秒的紧张氛围中迎来尾声。主持人感叹时间过得飞快，并结合全场专家的分享对 AI 在临床决策支持（Clinical Decision Support）中的未来发展方向进行了精炼总结。他指出，从今天分享的所有案例与实践经验中可以得出一个明确的结论：医疗 AI 的落地绝不可能由单一主体孤军奋战完成，行业内也不会存在一个能够包揽一切的“单一庞大系统（Monolith）”。就像我们无法指望依靠一个通用的超大模型来解决所有医疗问题一样，真正的突破依赖于构建能够调动和协同多数据源的控制框架（Harnesses）与智能体（Agents）。 主持人强调，要想真正做好临床决策支持，必须依靠整个行业的通力合作，汇聚各方力量去搭建和完善更健全的体系架构。最后，他向所有在场嘉宾与观众表达了感谢，并期待这次圆桌只是未来持续深入探讨与跨界合作的起点之一。 - **去中心化与多智能体协同**：未来的医疗 AI 不应寄希望于单一全能大模型，而是通过搭建 Harness 框架与 AI Agent，融合来自不同临床系统的多源异构数据。 - **跨生态协作构筑新框架**：完善临床决策支持体系需要医疗机构、技术开发者与科研人员“共同携手建立更优框架（bring the better frameworks）”。 ## Notable quotes > [04:46](https://www.youtube.com/watch?v=F8GG9gCmw1g&t=286s) 我一看到血就会晕过去，虽然在医疗行业工作了 25 年，但我绝不直接参与医疗救治，所以你做结肠镜可千万别找我排班。 — Senthil Kumaran > [08:21](https://www.youtube.com/watch?v=F8GG9gCmw1g&t=501s) Innovation moves at the speed of trust（创新的推进必须建立在信任的基础之上）。 — Arturo Loaiza-Bonilla > [15:29](https://www.youtube.com/watch?v=F8GG9gCmw1g&t=929s) AI 并没有凭空制造测量难题，它只是无情地揭露了医疗系统长期以来一直存在的测量盲区（measurement problem）。 — Peter McCaffrey > [20:16](https://www.youtube.com/watch?v=F8GG9gCmw1g&t=1216s) 这种制度设计让临床工作完全无法维系（impossible）。 — Peter McCaffrey > [21:36](https://www.youtube.com/watch?v=F8GG9gCmw1g&t=1296s) 它就变成了一个不可靠的秘书，你得时刻去检查他的工作，反而增加了认知负荷。 > [23:45](https://www.youtube.com/watch?v=F8GG9gCmw1g&t=1425s) keeping clinicians in the loop... at the helm of all the decision-making processes（确保临床医生始终处于所有决策流程的核心主导地位）。 > [25:36](https://www.youtube.com/watch?v=F8GG9gCmw1g&t=1536s) Unless you're creating quantum computing on the side, we have enough（除非你顺手造出了量子计算，否则市面上的工具已经足够多了）。 > [34:41](https://www.youtube.com/watch?v=F8GG9gCmw1g&t=2081s) 面对我们自己都无法完全理解的事物，我们该如何进行临床判断？ — Arturo Loaiza-Bonilla > [40:12](https://www.youtube.com/watch?v=F8GG9gCmw1g&t=2412s) No owner, no integration（没有明确的责任人，就无法真正融入工作流）。 — Alexander Fedotov > [42:03](https://www.youtube.com/watch?v=F8GG9gCmw1g&t=2523s) Garbage in, garbage out（垃圾进，垃圾出）。 — Zachary Taylor ## Mentioned Arturo Loaiza-Bonilla, Peter McCaffrey, Senthil Kumaran, Zachary Taylor, Alexander Fedotov, Massive Bio, St. Luke's University Health Network, The University of Texas Medical Branch (UTMB), UT System, MNGI Digestive Health, Cincinnati Children's Hospital Medical Center, AstraZeneca (阿斯利康), Epic, Ibex Prostate, OpenEvidence, Perplexity, Butterfly iQ, MASAI 试验 (MASAI Study), Medicare ## Video description Peter McCaffrey - Chief Digital and AI Officer, The University of Texas Medical Branch Senthil Kumaran - CIO, MNGI Digestive Health Arturo Loaiza-Bonilla - Co-Founder & Chief Medical AI Officer (CMAIO), Massive Bio Zachary Taylor - Imaging AI Core Researcher, Cincinnati Children's Hospital Medical Center Alexander Fedotov - Director, AI to Transform Care, Enterprise AI, AstraZeneca AI is becoming a powerful second set of eyes for clinicians, helping flag anomalies, predict patient deterioration, and surface evidence-based recommendations at the point of care. This panel will explore how leading hospitals are embedding AI into radiology, pathology, and bedside decision-making, while ensuring that clinical judgment remains central. Panelists will discuss integration with EHRs, managing false positives, clinician trust, and the regulatory frameworks that shape adoption. Learn how AI is moving from pilot tools to reliable partners in the clinical workflow—improving patient outcomes without eroding physician autonomy. [Source](https://www.youtube.com/watch?v=F8GG9gCmw1g)

[4:46] Senthil Kumaran: 我一看到血就会晕过去，虽然在医疗行业工作了 25 年，但我绝不直接参与医疗救治，所以你做结肠镜可千万别找我排班。 [8:21] Arturo Loaiza-Bonilla: Innovation moves at the speed of trust（创新的推进必须建立在信任的基础之上）。 [15:29] Peter McCaffrey: AI 并没有凭空制造测量难题，它只是无情地揭露了医疗系统长期以来一直存在的测量盲区（measurement problem）。 [20:16] Peter McCaffrey: 这种制度设计让临床工作完全无法维系（impossible）。 [21:36] 它就变成了一个不可靠的秘书，你得时刻去检查他的工作，反而增加了认知负荷。 [23:45] keeping clinicians in the loop... at the helm of all the decision-making processes（确保临床医生始终处于所有决策流程的核心主导地位）。 [25:36] Unless you're creating quantum computing on the side, we have enough（除非你顺手造出了量子计算，否则市面上的工具已经足够多了）。 [34:41] Arturo Loaiza-Bonilla: 面对我们自己都无法完全理解的事物，我们该如何进行临床判断？ [40:12] Alexander Fedotov: No owner, no integration（没有明确的责任人，就无法真正融入工作流）。 [42:03] Zachary Taylor: Garbage in, garbage out（垃圾进，垃圾出）。

[12:44] Peter McCaffrey: AI didn't create a measurement problem... it revealed that we have had a measurement problem for a very long time. If someone were to say, 'describe the impact of workflow A without AI,' it's not clear in many cases.

[0:00] All [0:04] right, nice to see everyone. I'm Arturo Loisa  Bonia, medical oncologist. I serve as the chief [0:09] of hematology oncology at St. Luke's University  Health Network, uh, which is a network of 16 [0:13] hospitals in Pennsylvania and also the co-founder  and medical AI officer of a company called Massive [0:17] Bio. We use AI for clinical trial matching in  oncology. Um, and I'm going to be moderating [0:22] the panel today. So, I don't know if I should  be sitting down or standing. I don't I don't [0:26] care. I've been sitting down today. So I can  actually put this here so I can have my notes [0:29] easily enough. Uh but as we started today we  uh the purpose of this uh panel is uh kind of [0:36] demystify the application of AI in the clinical  practice with you know very good use cases that [0:41] our panelists are going to bring to the table  here. So um this is becoming really important [0:46] for us clinicians. Uh how can we really implement  this in the best possible way without replacing [0:52] us? You know the question here very very much  well put. Uh, so I'm gonna just go through the [0:56] the panel and just each one introduce themselves.  Um, so Peter, do you want to start? Hey everyone, [1:01] my name is Peter McAffrey. Um, I'm a pathologist  by training. I'm our chief digital and AI officer [1:06] at the University of Texas Medical Branch and  I chair our AI collaborative across the broader [1:10] UT system as well in Texas. Hi everybody. Uh, my  name is Cintel Kumaran. I'm the chief information [1:18] officer for Minnesota gastroenterology. It's a  a GI practice in uh Twin Cities, Minneapolis, [1:25] St. Paul and and around. We have 10 ASC's uh  11 clinics uh 160 plus uh GI trained physicians [1:33] and a we have the largest private GI practice in  the nation. Yeah. Uh and my role is uh yeah doing [1:41] digital transformation. So we'll talk about it.  My name is Zachary Taylor. Uh Zachary P. Taylor. [1:47] I'm from uh Cincinnati Children's Hospital Medical  Center. I'm a uh imaging AI researcher with the uh [1:54] center for artificial intelligence uh and imaging  research center. Um we work to make different [1:59] models and uh train and deploy uh different  solutions for physicians to use in the clinic. Hi, [2:06] my name is uh Alex Peratov. I'm a director  of enterprise AI at Astroenica. I'm a PhD [2:12] trained geneticist and um data scientist. And  in my current role, I'm essentially scaling um [2:18] multimodel AI and NLP models um for risk patient  identification in partnerships with technology [2:24] vendors and healthcare system. Um as a disclaimer  view on my own. Yeah, perfect. Um so just to get [2:32] us started, the the first thing we wanted to do  is just kind of look at a use case of clinical [2:37] AI in your workflows and things you're doing or  you found like a really opportunity to optimize [2:42] uh you know something using AI. So uh just to  a starter so Alex you want to tell us you know [2:48] I you know a painoint that you have been looking  into that may help us in clinical decision support [2:52] or operations. Sure sounds good. So I know we're  going to talk a lot about pathology and radiology. [2:58] So before we get into this beautiful world of I  guess images and slides and so on um I'll um give [3:05] an example of initiative that we're focusing on  which is actually um uh identifying underdiagnosed [3:11] or undiagnosed um patients with uh chronic  uh disease such as uh chronic kidney disease [3:16] and uh chronic obstructive pulmonary disease.  And so the I guess the challenge here is that [3:22] um we know that in the US um at least in the  US um individuals for example living with CKD [3:28] 90% of adults are actually not aware of that  condition that they have this condition and [3:33] um um essentially remain undiagnosed and so  forcing um primary care providers or specialists [3:40] to you know to go through charts and surface  this uh individuals is obviously impossible. [3:47] So we partnering with technology vendors who are  able of uh developing uh AI clinical reasoning [3:52] essentially systems and platforms that can be  natively integrated into health records and um as [3:58] a result of of that will essentially surface these  individuals for physicians to either you know do [4:04] proper referral order additional diagnostic  um testing or um adjust the care treatment [4:10] pathways. And so uh this way we're essentially  improving you know both patient outcomes um and [4:15] so on. And um I'll also add that this approach  uh you know this idea of identifying um patients [4:23] from unstructured and structured uh nodes uh could  also be implemented for some oncology workflows, [4:29] diagnostic workflows for metastatic breast cancer  in non small cell and cancer patients who for [4:34] example are not properly um u tested like genetic  testing is not properly done. And so we can [4:41] utilize the same logic to essentially put them  on uh additional testing pathways. So that's [4:46] essentially our use case. Perfect. Um and so  central you mentioned that you're driving the [4:51] governance and assessment of all these tools. What  are the those major points that you as you do have [4:56] uh and you're at gastronologist as well. So kind  of like give us that point of that notologist but [5:00] you know working in the GI space. Yeah. Yeah. I I  work in GI with with the MDs. I mean moment I see [5:08] blood I pass out. So I I've been in healthcare for  25 years but uh I stay out of uh the actual care [5:16] delivery. So I'm so in my colonoscopy I cannot  call you and schedule it. Okay sorry. Yeah uh [5:22] but uh to answer your question um you we have the  previsit summarizations which we piloted for our [5:29] uh IBD uh subsp specialty group within our GI. Uh  so the first things we looked at is obviously is [5:37] it accurate enough to what the RNs are creating  uh for the folks who do not know what IBD is the [5:45] the inflammatory bow disease uh and our RN spend  quite a bit of time before the patients come in to [5:54] synthesize all the path lab reports blood work  patient history and everything and create the [5:59] summary doc for the uh physicians to look at it.  So now the tool does a lot of its job what the [6:06] RNs would do but still there is a lot of clinical  efficacy and other questions that comes around. Uh [6:13] so we still have the experts in the loop to fill  in the gap. Uh so the tool is a tool obviously [6:19] the expert in the loop is pretty critical. uh but  it definitely in uh acts as a second pair of eyes [6:26] uh in identifying care gaps which sometimes uh  physicians miss or the care delivery folks miss [6:33] too. So that's that's a beauty of these tools. So  Zachary uh you work in imaging so are those use [6:40] cases that you're seeing can you know help us to  optimize things? Yeah. Yeah. So honestly the major [6:44] benefit that we have as a hospital being able to  do things on prem is we kind of get to make things [6:49] bespoke to our kind of uh you know institutional  kind of expectations or or uh you know uh whims [6:57] I guess. Um so you know we have doctors that come  forth with uh different ideas for tools that would [7:02] help them. Um so we kind of uh reach you know  the lowest hanging fruit that is also specific [7:08] to pediatrics. uh us being a children's hospital  um there's so many different you know amazing work [7:14] that's been done with like organ segmentation and  all these other um models that are more oriented [7:19] towards adults. So a lot of the time we've been  kind of picking and choosing our bowels of you [7:24] know these uh subcategory of of tasks that haven't  really been explored with pediatrics due to [7:30] uh pediatrics having a much lower I believe less  than 10% or so of uh total radiology data um is [7:38] uh pediatrics. So uh having access to that uh as  a children's hospital is a major benefit and uh so [7:44] a lot of our uh tasks are kind of like the organ  segmentation for like uh various different tasks [7:50] as well as uh total cardiac volume segmentation  for selecting uh different organ donors for [7:56] heart transplantation. Um bone age estimate  uh estimating um for you know different hand [8:03] uh bones as well. Uh and it's really just been  interesting to see as well as developing out our [8:08] digital pathology branch um seeing new things be  brought into the fold such as uh eosinapl counters [8:15] um things for detecting liver fibrosis things  like that. Yeah. And so speaking of pathology [8:21] so Peter can you just tell us more about you  know work you doing there and pain points [8:25] and stuff. Yeah. So I'll give uh I'll give a  pathology use case and a non-pathology use case. [8:29] One of the things I am happy with that we've done  in the past I don't know maybe a year and a half [8:34] ago we rolled this out was through a digital  pathology enterprise. So um a tool called Ibex [8:40] prostate. So basically it's an overread tool for  prostate cores which you know again a pathologist [8:46] can diagnose small microphosi of neoplasia in a  prostate core but when it's your like 200th core [8:52] you just don't you're not as good doing that. So  a good systems engineering question would be to [8:55] say well inevitably how do you buttress the decay  in that and so you kind of you click a button and [9:00] you'll get this overread heat map on the core  that's like could this be neopre down to very [9:04] small microfosi which is just a a guardrail you  know against like that attrition so that's a kind [9:09] of a core you know process enhancer and the sort  of non-iming one that I think has been nice is you [9:14] know we we have a large catchment area of patients  we're often a safety net hospital we want to make [9:19] sure people don't fall through the cracks as with  every hospital access Access. Access is a big [9:23] question. Referral cues are long of course. So now  when people sit in a referral queue for nefrology, [9:29] rheumatology, colonoscopy, ID, endocrine so far  at UTMB, we have their specialty group define [9:35] a rubric of acuity. So what would you consider a  high acuity nefro referral versus a nonacuity neo [9:41] referral? You as a specialist, what's your logic?  And it will project that logic on the context of [9:45] the chart and write back to Epic like high acuity,  low acuity, next step. Because the person doing [9:50] the booking is not the doctor, they're an access  technician. And so at least to be able to say that [9:53] the FIFO Q now becomes a priority queue at least  and and does again no no decision was being made [9:59] before then. So putting some place of visibility  and decision- making to kind of prevent people [10:04] falling into the the cracks essentially. Yeah.  No, that makes sense. And um you know as a medical [10:08] oncologist I use all those things together.  So I'm a GI oncologist also. So I was like I [10:12] see patients with IBD who get colon cancer. So all  this is extremely relevant to me. So uh so tons of [10:19] questions for myself here. uh but but I always  think that innovation as they say move at the [10:23] speed of trust right and uh even in friends and  become everything becoming much easier and cheaper [10:30] uh the cause we're having is to you know do the  you know wrong diagnosis or getting the things [10:35] that we're missing or not missing because we have  uh a skill erosion or the skilling because of all [10:40] these things happening in the space uh so so when  an AI tool flags something within your workflows [10:47] um or predicts patient iteration or it's a sepsis  uh you know algorithm or whichever algorithm we're [10:54] talking about um how can we make sure that we're  managing the right expectations from not only [11:00] the clinicians but also the patients that we're  serving so and I'm just going to ask Zachary about [11:04] it now yeah honestly I think it's it's a hugely  important piece of the puzzle right to make sure [11:10] that the physicians and any of the healthcare  workers that are interacting or interfacing [11:14] with this technology understand you know what it  can and can't do the limitations as as well as [11:20] um you know when certain things should be trusted,  when things should you know be uh kind of given an [11:26] asterisk of okay this is you know AI generated  uh information that I'm dealing with here. Um, [11:32] so I think a lot of that is it's kind of a a  dualpronged issue of you know the developers [11:37] or the uh you know data scientists or or whoever's  giving the uh uh the output of the AI tool should [11:45] keep in mind that they're humans reading the  input and that you know it should be readable [11:51] in a way that you know a non you know data science  person should be able to understand uh the output [11:57] and also you know physician education as far as  what can a campus tool do. Yeah. See, because I, [12:03] you know, as I live in both worlds, I know clearly  that most of my staff have zero understanding of [12:09] certain concepts that are so essential like  they don't understand F1 score means or what [12:14] overfitting means or what the data says we're  using. uh and that's so important to have always I [12:18] always believe that be beyond a chief security you  know information officer it should be a chief AI [12:24] medical AI officer in every institution this time  and age because they they need to translate it in [12:28] a much more meaningful way that understands those  concepts and translates it to all the folks around [12:33] them uh and that way we prevent you know you know  replacement and also optimize people to be much [12:38] more efficient using these tools uh so um what  do you can tell us about like shadow deployment [12:44] in and when we're trying to test because shadow  deployment is one of those things that's emerging [12:48] these days where we're trying to make sure  that before we fully implement it we kind of [12:53] run it on the side and and but there's very few  places that actually know how to do it right so [12:58] I would love to hear who actually has done it  right to hear that yeah so we when we did this [13:03] uh previsit summarization tool so we did a  shadow deployment for almost a couple of months [13:08] uh where we had our clinicians still prepare the  previsit summary so you'll have dual screens where [13:14] the AI is preparing this this so you look at it  and then see what's working what's not working [13:20] what patients it picks up faster so we have all of  it tracked um and then we fed the algorithm back [13:28] so that it can get better at it uh in predicting  things and I've always told our physicians that [13:36] this is not going to be 100% foolproof isn't it  so it's it's it's always going to make mistakes uh [13:42] and the classic example is 10 years ago the same  AI world conference A4 somebody said radiology [13:50] will be completely obsolate in the next 5 years  I don't know how many of you listen to that [13:55] isn't like don't make your kids study radiology so  it's all going to be was this morning here he was [14:00] talking about yeah so the keynote then it uh all  all of all of us will be driving the FSD vehicles [14:08] in 10 years isn't it uh so today the radiologists  are driving their gaslars to the work So everybody [14:15] knows this uh so some predictions all of these  things will will not be 100% true. So the tools [14:22] are tools there are second set of eyes. So it's  making some empowerment for you but ultimately [14:29] it's the experts in the loop who have to make  these decisions and train the tool like b make it [14:35] better. Uh so that's what we have done. So if you  do don't put put it into production uh till it's [14:41] like ready is what I always say like keep keep it  in the pilot mode for a few months uh before you [14:47] feel comfortable with your main workforce. Yeah,  I'd also add on to that and say, you know, even [14:52] if it's uh just for identification of different  outliers, different subsets. Um I think planning [14:58] your your shadow deployment in advance is very  important to uh kind of keep in mind that hey, [15:04] you know, there will be some issues. Maybe it's a  scanning issue. Maybe it's um you know, a subset [15:09] of the population that we didn't account for in  the training data. Um and you know that's just [15:14] more information to feed back into the system and  um you know make a better fine-tuning data set you [15:19] know uh and bring things back into the fold right  so um I think that's definitely something to keep [15:24] in mind and I totally agree so Peter any comments  yeah so uh I mean organizations are complex always [15:34] and there's always some level of P scaling into  deployment and retracting to some sense back to [15:41] PC I we're reactive to an ever evolving uh world  around us instead of constraints. I think there's [15:48] a there's an extent to which all you can ever  really do right is test the model according to [15:53] its outputs and ground truth before deployment  and after deployment periodically. There's not [15:58] a fundamentally different paradigm really than  would apply to other things we've done or people [16:03] that we've managed. However, the scale at which  we are being confronted with the need to do this [16:08] is itself kind of unprecedented. So you may have  to be in a world where you can do many of these at [16:14] one time and you can compare outcomes. I would say  the you know this is going to be kind of obvious [16:18] but if you can't measure what it's supposed to do  as an impact there's no basis to evaluate anything [16:25] and what I have observed a lot in a typical  healthier enterprise our own as an example is [16:30] uh AI didn't create a measurement problem it but  it revealed that we have had a measurement problem [16:34] for a very long time and if someone were to say  like describe the impact of workflow A without AI [16:40] it's like not clear in many cases and I think that  is the root cause of a lot of the challenges of [16:45] growing and retract acting and being responsive.  Yeah. Yeah. Know it was about deployability, [16:50] right? Because it's so important to prevent  like you know alert fatigue and all the things [16:53] that happening. So uh so Alex from you have an  interesting perspective because we you come from [16:58] different parts of the aisle. Uh so um what have  you seen uh in terms of expectations like say oh [17:05] this is going to cure cancer is going to solve  everything and how is that hype being actually [17:10] tempered down with reality? Yeah. Yeah. So it's  interesting. I think um if we approach the you [17:15] know the trust issue from the data science  perspective I think the you know the most [17:20] important part is actually we have to always  be striving for essentially perfection and [17:25] being transparent about where we're not uh you  know as good just yet. So I think um from the [17:31] perspective for example large large language  models right um I mean we were certainly very [17:36] excited when they came into the scene and uh at  the same time a bit disappointed when we realized [17:42] that you know when they being fed essentially  the totality of patient health records they [17:47] struggled with some parts they struggle with uh  you know sequences uh temporal sequences they [17:52] struggle with um calculations and so on and so  um being transparent that we have these issues [17:59] and essentially moving on to improve them through  for example um this new exciting development with [18:05] hybrid AI models where we have you know LLMs and  also integrate uh some rule-based uh algorithms [18:13] into the you know the over the final model um I  think this allows us to essentially achieve this [18:18] uh mutual understanding of you know where the  field is going from the data science perspective [18:22] and the clinician perspective as well. you know  indeed. Um so and from and from my angle you [18:28] know I I I found massive bio because clinical  trial matching was a big problem now it's fully [18:33] commoditized and know about is about deployment  and showing that actually the what the the output [18:38] of the neuros symbolic AI is fully deterministic  and it doesn't have any hallucinations because [18:43] the patient is in front of me and I need to help  them out to get to a clinical trial. Uh so and we [18:47] work with a lot of sponsors so want to make sure  that we keep our lights on. Uh but beyond that [18:52] um major concern that I think Sentil was already  raising it based of actually Jeff Hinton saying 10 [18:59] years ago that radiologist will be fully replaced  so stop studying you don't need to do anything [19:03] like that and this morning they were just arguing  more about like if they came to work or not uh [19:07] that was an interesting um you know thing to be um  but now yeah that all those folks actually leaving [19:13] I saw or I saw the news today that actually  they're creating a new company so uh the Mr. [19:18] how became the the lead person now but I'm just  diverting here um a major concern of AI is that [19:24] it's going to replace clinicians and and that  was the question earlier um how do we design and [19:30] govern you know clinical decision support systems  that first of all not only well they can replace [19:37] whatever it is I'm I'm okay to be replaced because  I don't want cancer to happen ever again to anyone [19:41] so if I'm fully replaced I'm completely happy to  pursue my AI entrepreneurship uh but if um if we [19:47] are going to develop these systems, they need to  be used by the end users, which are clinicians and [19:53] and how do we create models that not only are you  know trusting, but we make them, you know, usable [20:00] for those folks beyond alert fatigue, but actually  something that is going to help them um and create [20:06] this Javon's paradox of actually getting them  more busy taking care of patients. Uh so, [20:11] um I'm going to start with pathology. So, Peter,  so many thoughts on on this. So one thing one big [20:16] thing I would say is [clears throat] you know any  area of in any area of clinical deployment I would [20:23] speak for the ambient customers for example  in the primary care setting we have to make [20:29] a regulatory maturity happen in the field that  makes the position tenable to be a provider like [20:36] I think if you go to providers and you say hey  here's a tool it'll make you 10% more efficient [20:41] and then all of your business efficiency people  say why aren't you seeing 10% more patience, [20:45] but at the same time your like compliance team  is like, "Oh, but by the way, it might be wrong. [20:50] Double check it. Triple check it." Like the  buck stops with you. Then you've just pinned the [20:55] profession in both an expectation to use tools and  a net new job to validate them. It is impossible. [21:03] And I think if if we can't move liability onto a  system, think of like, you know, in aviation like [21:09] there the pilot is the driver. Yes. But the pilot  isn't responsible for the internal electronics of [21:15] the landing gear. There's got to be some place  where your risk ends. Otherwise, I don't see [21:21] a world where things can actually feasibly scale  that gives a physician like a place to exist that [21:26] anyone would ever be able to do or willing to do.  That's probably the the biggest biggest challenge [21:32] and it has to come No health system can solve that  in in of themselves. It has to come from like I [21:36] think an overall philosophy change in the way we  approach validation. No, I I couldn't agree more. [21:42] Honestly, I think that is the conversation that  isn't being had enough right now is kind of that [21:47] kind of transition of liability. Um, you know,  where does the buck stop? And I mean, just from a [21:53] practical uh standpoint, like you said, if it's a  tool to save time, then why am I having to double [21:58] check, triple check things? You know, that's just  having a a unreliable secretary that you're having [22:04] to constantly check the work for. So I think you  know breaking things down into into smaller pieces [22:09] um where that trust can be uh accumulated a  bit more readily and also you know establishing [22:17] uh you know the the agreements on you know where  does liability lie and um for which task right so [22:25] liability may look different for one task versus  another task but um yeah I think that's that's [22:29] just a conversation that needs to be happening  more I was thinking [clears throat] the tools [22:36] are getting better day by day. Uh but does is is  it really helping patients? I know from an ambient [22:43] AI perspective, it's definitely helping physicians  save their uh pajama time. Uh for folks who do not [22:50] know pajama time, so they have to finish all their  medical notes by the end of the day or end of two [22:56] days. So then they have to work uh after they go  home during their dinner time and dur also the [23:02] previsit summarization that some of the docs have  to do the previous day they spent by time. So all [23:07] of this the tools are definitely reducing it's  getting better day by day. Uh uh but but is it [23:14] empowering the patients? Uh is it filling the care  gap for patients? Is it making the health plan pay [23:23] better? All of these questions I still have. I  don't I don't know. Are the tools helping there? [23:29] No. Uh are there companies out there? I I hope so.  Are they focusing on patient care? That's what we [23:36] should. Uh so these are gaps that uh uh hopefully  AI will help us navigate and reduce it. Yeah. [23:45] Alex, what are your thoughts? Yeah. So um I would  perhaps approach you know this um this problem or [23:53] statement from you know the body of scientific  or clinical evidence and I think if we look at [23:58] the current state of affairs um I don't think that  any eye system right now um can replace clinician [24:04] judgment and so as a result as a human society  and I'm emphasizing the human part you know [24:11] um uh we certainly should be looking forward to  keeping clinicians in the loop and I would take it [24:17] even further keeping them at the helm of all the  decision-m processes at least based on how things [24:24] looking right now. And um to you know to kind of  analyze this a little bit from the human computer [24:31] interface interaction you know we know that um the  way we design these tools right now can actually [24:36] impact um the role the degree of contribution of  human to decision-m process and so we know there [24:43] are studies that show that for example you know  showing that the timing when the output of the [24:48] algorithm is shown to clinician actually matters  and if you show it at the same time when the [24:53] clinician is trying to make a you know decision  it can introduce automation bias and uh you know [24:58] will kind of force reliability on this tool for  subsequent cases. And so if we delay on the other [25:05] hand the AI output or um make it actually give the  opportunity for clinician to toggle back and forth [25:12] between their decision and you know the AI output  you can actually um enforce actually um clinical [25:19] decision making the way we view it as ideal  state uh this moment. So I think we still need uh [25:26] there's still tons of work that needs to be done  before we can you know make the statements and so [25:30] the we're looking forward to you know continuing  uh improving the existing body of evidence. Yeah, [25:36] and I agree. U and and for you know for us I  mean I manage the service line so there's a ton [25:41] of vendors coming all the time and say hey we had  this and this uh um and how many more ambient AI [25:47] we need I think we have enough like you know  everyone's going in that field please unless [25:51] you're creating quantum computing on the side you  know you know no more we we have enough I think [25:56] innovation that has to happen is in things that  actually helping our workflow so panyama time I [26:00] track it I actually tell them if you have pajama  time I force them to do ambient AI they like [26:06] there's no like a choice for them because we know  they can improve outcomes in terms of at least [26:10] the time that's the thing I can measure we need to  have some metrics and rubrics that we don't really [26:14] have yet to really establish how we can make the  return on investment on any of those tools because [26:19] the money is limited you know get reimbured by  Medicare by all of them and the pie is not bigger [26:25] someone something's got to give so you either  decrease you know one portion and use the AI tool [26:30] for that or you have to find the niche so that's  the important part of the decision support and and [26:35] why sometimes even you have the best solution in  the world no one will implement it because they [26:40] don't see the ROI for from the people who are  buying that unless you do a premium or do like [26:44] open evidence something that so pharma pays for  that and you just get ads based on the questions [26:49] you make right that's the only good model that  I've seen uh be able to be sustainable so u moving [26:55] forward on on on now we move to like regulatory  aspects which I think are so important and so [27:00] interesting because there's not a lot of talk  about it uh we see that things like doctrronic and [27:05] others are actually giving prescriptions basically  because you are allowed it and and I'm I'm okay [27:11] with some extent because I'm an enthusiast but  then what happens is there's an error like who [27:16] is the one who's going to carry liability and  up until now all these disclaimers the same as [27:20] when you read anything when you you know you get  an upgrade from your iPhone you accept yes or no [27:25] you don't accept then you don't get the upgrade  so everyone says yes uh so but then you get the [27:30] lawsuit or or the problem and and I had to deal  with a lot of those phone calls from patients. Um, [27:35] so how do we uh look into the regulatory framework  around clinical decision support uh and and make [27:42] sure that all these life dead decisions that  sometimes they entail can be trusted. Uh so I just [27:47] love your thoughts about it because to me that's  so interesting. Um I can start with Alex because [27:51] Yeah, sure. All right. So I guess we're talking  about uh explanability as a concept, right? And [27:58] um traceability and so on. So um I think obviously  for some fields you know pathology and radiology [28:06] um explanabilities as a concept you know is  probably a little bit more challenging to you [28:11] know implement. So um but I'll move on to you know  fields where um HR based uh clinical reasoning is [28:19] actually you know being trained to achieve through  AI models and so there we can actually design [28:26] algorithms uh design their architecture the way  that traceability is at least somewhat um you know [28:32] possible. And so as I mentioned before um with  the current approach to um I guess dealing with [28:39] unstructured and structured data and um combining  both probabilistic and deterministic um AI models, [28:47] we essentially can achieve um situations where  physicians can track the sources of the AI output [28:55] to either clinical notes or specific calculation  or a clinical guideline and as a result you know [29:01] this would improve the reliability of the output  and uh the explanability as well. And so I think [29:08] in this particular field it is possible to achieve  this to the degree when regulatory bodies will be [29:14] uh accepting this evidence as indication that  this tool is reliable and can be utilized in the [29:20] clinic. Yes. So s from the imaging perspective  what are your thoughts about it? Because what [29:24] if it tells you you know there's always this  interesting of sustaining attentional blindness [29:28] when there was a study where they have all these  radiologists looking there was a gorilla and the [29:33] cat scan and 83% of them missed them. So they say  well an AI tool can now help find this gorilla [29:39] in the room right so uh but um what if he's  missing it like how how do we make sure that [29:43] you know we have the best of both worlds. Yeah,  I think it's yeah, like you said, I think it's [29:48] it's managing those expectations and and making  sure that your your governance council is aware [29:53] of those expectations and the the capacities and  incapacities of AI and being very tasteful about, [29:59] you know, your application of them and, you know,  um the the ability to, you know, plan in advance, [30:06] okay, I'm going to build my infrastructure with  AI in mind. I think that's also a big deal. And [30:11] you especially see that with you know the advent  of digital pathology um becoming more prevalent. [30:16] I think we'll be able to build out uh kind  of those expectations as digital pathology [30:21] becomes more prevalent um in a way that we're  kind of retrofitting some of the thing uh the [30:25] standards that have kind of already existed for  radiology for so many decades. Um I think it'll [30:31] be interesting to see how that plays out. Yeah. So  and central I mean you you probably get a ton of [30:36] people reaching out to you about different models  and things they're doing. So just love to learn [30:40] from your perspective like how is that governance  being assessed? I know there's different degrees [30:44] of of you know sensitivity like and ambient AI  straightforward but other models that are more [30:50] specific like death and death sepsis and what  not. How are you approaching that from both [30:53] perspectives? Yeah. So the simplest approach is  like all we're looking at is data democratization [31:00] isn't it like as long as we have explainability  uh and bring this from a summary perspective [31:08] like we we took like I I don't know how many  of you use perplexity so for instance you you [31:12] can go there and it it tells you where I got this  information from isn't it it's not hallucinating [31:19] to some extent compared to other models so I want  to replicate that inside the healthcare obviously [31:24] open evidence is already doing it. Uh but to bring  it inside some specialty practices where we can [31:32] keep this as second pair of eyes uh is how I look  at it. And once we have this data democratization [31:40] so then look at the digital path lab reports  like we have 100page plus PDFs and you can't like [31:45] digest as a GI physician. So you don't have like  100 hours to digest all these 100page PDFs in it. [31:53] you have that 10 minutes or less than that to do  all of the uh findings within all these 100 200 [31:59] page. So that's where these tools will summarize  it, help them like navigate, make it searchable, [32:06] make it indexable. That's that's what I'm thinking  is is where all all practices should move towards [32:13] to make decision making a little bit easier for  the physicians cuz right now they're overburdened. [32:20] uh and and you can go look at all pretty much  all the rural area wait time for any of these [32:26] physicians or more than 3 4 months uh and and and  whenever I look at IBD patients or even um uh like [32:36] in any kind of subsp specialty if your weight time  goes above 2 3 months it could be a big decision [32:41] for the I mean life and death decisions in certain  cases. So I think these tools have to play a role [32:47] and we can put governance around it which which  is what the um the without replacement judgment. [32:53] That's that's how I look at it. I I agree with  that. I I think um so my philosophy is basically [32:59] that uh it would be nice to get to to get back  to a point where the main time allocation of [33:07] physicians was judgment and not clerical overhead  and some judgment. And I think that you know with [33:12] meaningful use we kind of got into this thing  where you you get to do some judgment after [33:16] you do all the factf finding the synthesis the  aggregation the documentation that healthcare much [33:22] better if we could just focus on judgment and I  think you know AI can obviously help that because [33:27] it will give provenence and rationale that's  explainable. I think a very real vector that our [33:32] field faces though that the amount of things to  judge is like exponentially blowing up. So you've [33:38] got more assays, more tests, more imaging. You  got a butterfly IQ at home with AI. You can say, [33:43] I don't know, do I have a liver cancer? Whatever,  right? And so and so now there there has to be [33:48] some point where we say you don't have to judge  that or I think what we'll end up with and this [33:54] would be sad but ironically very healthcare in  a way is a world where it's like we got a little [33:59] bit out of meaningful use a little bit less  documentation and now like 10x the burden of [34:03] double triple verification factf finding to where  it's at some point your decision space has to [34:09] have a boundary to it um and I think beyond that  boundary the systems need to aggregate liability [34:15] Um so and and there will be inevitably jobs that  I think must be done by the system just by sheer [34:23] volume and demand. But then that needs to be  the liable entity and you are you are able to [34:28] air traffic control and opine on it. But you  shouldn't have to then say but what if eight [34:32] layers deep there's a problem and you've got to  figure that out because I think that's just going [34:35] to be impossible. Today it's the way yes but I  don't think it can scale that way. Yeah. And and [34:41] and multimodality is becoming essential now. So  most of the models that are coming up is not just [34:46] looking at biology or pathology or genomics is  they're all combined and sometimes there's this [34:52] black box that there's impossible to discern  like and and something that is super important [34:58] is how can we judge something that we don't even  understand ourselves. So, so I think that uh when [35:03] I look at there's some radiomics things that I'm  seeing now that are completely crazy out of the [35:07] like we follow the rad biomarkers for lung cancer  for example and we say that will never work for [35:12] that patient because we have a randomized clinical  trial that show that that didn't work and then [35:17] it's like oh by the way there's this radomics and  if you look like this little kirk here and because [35:22] the vessels are aligned then that patient is going  to respond that only that one. So, uh, how can we [35:27] make sure that we you say, "Oh, sure. I'm going  to use it or I'm just going to say I just don't [35:31] want to go to the tr into that trouble. Uh, I'm  just going to not not do it." I'm completely the [35:35] opposite. I'm going to go for the end of one if  I can. But that's sometimes not practical. So, [35:41] you would love to hear sad what you think about  those things. Oh, no. I I completely agree. Like, [35:45] it kind of alludes to like the the broader topic  of, you know, decision support. I mean it's such [35:51] a huge you know uh ultimate you know uh uh task  that you're giving the model and there's so many [35:58] smaller subcomponents that even the smaller  subcomponents we don't have enough uh you know [36:03] model explanability model interpretability um so  I think there's you know new metrics that need to [36:07] be developed for things like uncertainty you know  different confidence levels for different subtasks [36:12] within a decision and you know I think building  it from the bottom up I think is is useful in in [36:19] not just getting overwhelmed overwhelmed like  kind of like Peter said where you can just you [36:22] know there's an infinite amount of data to  give a model right so it's like how do we [36:26] disentangle what information we're getting  you know what insights are we taking from [36:30] uh the different uh subprocesses that we're  trying to do here yeah uh I agree 100% so um [36:37] I wanted to like switch a little bit on  the practical angle here so for example [36:42] um do you prefer buy versus build um and do  you prefer EMR first versus a standalone. Uh, [36:52] and I would just I mean any any solution is  fine. I just want to understand that. I know [36:56] if if if if I don't say that EMR first um then  people are going to laugh at me. If if if it's [37:04] extra clicks for our physicians, if it's another  tool that they have to jump in to go look at it, [37:11] then the uh adoption rate is going to go down  because there is already click fatigue uh that [37:18] most physicians have. So you have to have the EMR  integration. And in our case um our clinicians I [37:27] mean for our physicians go to multiple different  hospitals and it everybody uses different EMRs. [37:34] We have to have our own EMR integrate with their  their EMRs to pull and push and that that itself [37:40] takes a lot of time for our uh team to focus on  or work on. And so that's why when we have to [37:48] build something new it takes a longer time because  they have a lot of other fish to fry every day. [37:53] uh because of all these integration issues. Um  so from an AI tool perspective so we are always [38:00] looking into newer vendors on who are doing who  has already done something and so bring that in [38:07] and then integrate with our uh primary EMR that's  that's what we are doing right now. So it it's [38:13] it's a classic build versus buy question. So if  if it's already available then customize it bring [38:19] it in is what my answer would be. Yeah. at least  resistive path uh and initially for a pilot maybe [38:25] an EMR integration is not needed because you will  have few physician champions who will run with it [38:31] but if you want to bring it to the whole practice  uh then you have to have the EMR integration yeah [38:37] know and and that's important around the accuracy  right because uh having the data based on our own [38:43] patients is what's going to drive the adoption as  well uh and so I'm going to ask you Peter about it [38:48] because pathology is such a unique space, right?  Because you can have the Oh, yeah. I build all [38:53] these gazillion WSI. Yeah, but they're all sourced  by Memoretin or MD Anderson. They're amazing, [39:00] but they don't use the same camera. So, the H&  gets a little bit messy and you cannot really step [39:05] it to my patient population. So, uh how is that  being approached now and and and particularly when [39:11] we're trying to establish things across multiple  networks? Yeah, I mean as with any type of assay, [39:17] you have to deliberately and rigorously validate  things like the H& staining technique and how it [39:22] impacts recovery. And that in turn relies on you  having a reliable ground truth set of cases for [39:28] which the actual diagnostic output is clear and a  system where there's a thing to measure. You know, [39:33] for example, I think people outside of AI in  general think that, you know, the model just [39:38] like spits out a diagnosis and you say agree,  disagree, but like do you have a bounding box? [39:42] Are there tiles? Where are they? What's my dice  coefficient? Can I get to a number? Can I profile [39:47] that number? Um, but it's not totally different  for what you would do in other things, but it is [39:52] a unique form of variance that's different maybe  from radiology except there's scanner differences, [39:57] but now you have scanner differences and  technique differences and glass differences. Uh, [40:01] and to some extent depending on your annotation,  pathologist differences if you don't even know [40:05] exactly what your ground truth is. So, you have to  na really nail that down first. Yeah. Uh so um and [40:12] and Alex um as you seen also different angles of  this um what is your your take on it? Yeah. So, [40:21] um I think for us I mean we see that the field  has been through some trials and tribulations [40:26] essentially and so we kind of we do have  a frameworks do have frameworks that allow [40:32] us to you know evaluate the solution and so for  us we looking at um usually how well the social [40:39] lines with clinical guidelines you know because we  know that clinicians need to have some skeleton to [40:44] you know uh build all the decision- making around  and So if you have that skeleton and the solution [40:51] aligns with it, it helps tremendously. Um the  other aspect I would say or this other pillar is [40:59] uh certainly the ownership. So we want to know  who owns the ultimate um decision you know for [41:05] the solution that is being proposed in the clinic  because no owner um no um you know integration [41:13] into the workplace. So and um I think the final  pillar that I would highlight is uh reimbursement [41:20] as funny as it is you know and it's perhaps  counterintuitive but it seems that assistive [41:26] AI tools in the current situation are actually  getting reimbursed much better than autonomous [41:34] in a way because they're tied assistive tools  are tied to actual work done by the physician. [41:39] And so this is another the third component that  um should be taken into consideration when making [41:44] all these decisions. Yeah, you brought up really  the right point because the thing is reimbursement [41:49] doesn't happen without a clinician in the room.  Um because we're as we take the liability and [41:54] the patient in front of us that we're trying  to convince to take the test or not that if it [41:58] doesn't have a a what's called work component, it  will never be passed by anything. I can tell you [42:03] firsthand. Uh so and and sag you know for the last  couple minutes uh how you're looking into you know [42:09] algorithm bias and things particularly in imaging  because uh you know we have all these amazing [42:12] algorithms for like you know mamography aid right  we we know based on a randomized 80,000 clinical [42:18] trial patient called the masai trial where they  found that actually you can optimize radiologist [42:24] time by like 40 49% I believe um and really just  use just one radiology so you can create that more [42:31] time so they can do more biopsies jeans paradox  get more radiologist in is so how are you looking [42:35] into this algorithm bias and in in imaging? Yeah.  So I mean like you said there's so many different [42:40] types of bias you know demographic types of bias  and you know uh whatever uh data the model was [42:47] trained with is obviously going to introduce bias  from the get- go. So I think it's a matter of [42:53] uh knowing as much as possible about the data  that was used to train the model. Um obviously [42:57] garbage in garbage out but you know even even if  it's uh well well uh curated data you still need [43:05] to have proper monitoring systems in check and  do that prior validation and you know do like a [43:11] uh a comprehensive study across uh different  demographics across different scanners and [43:16] see you know is this applicable in different uh  contexts or you know is there something that we [43:21] need to take a different approach on. Yeah improve  diversity on the on the data sets. Uh so I think [43:26] we are just u you know three seconds left on our  time. It just flew by. But what we can tell from [43:32] all the stories is that you we cannot this by  ourselves. There's not like a single monolith. [43:37] The same way we don't see a big model doing  everything. We need harnesses and agents to [43:42] do things from multiple data sources. The way  to make clinical decision support has to come [43:46] from a combination of all of us coming together to  bring the better frameworks. And hopefully this is [43:51] one of many conversations we're going to have in  this and future meetings. Thank you for the time.