Data Chiefs Mandate Upfront ROI as AI Agents Expose Broken Pipelines

AI Agent暴露数据管道缺陷，数据主管要求预先明确ROI 部署对话式AI工具正迫使企业重构底层数据治理与本体，因为 业内数据领袖探讨了从传统BI仪表板到对话式Agent及因果决策系统的运营演进。他们强调，尽管探索性查询受益于快速迭代与人机协同验证，但面向会员和受监管的工作流仍需要严格的确定性及经过验证的语义本体。专家小组还探讨了通过精准引用处理非结构化文档、新出现的“设计架构师”角色，以及衡量企业AI投资回报率（ROI）的不同框架——从前置的成本规避要求到风险投资式的组合实验不一而足。 商业智能（BI）经历了三个时代的发展：从人工原始SQL请求，到拖拽式平民数据科学工具，再到如今AI驱动的对话式决策智能。 (0:04) 在早期探索性数据分析中，保持战略连贯性以探究到“第10个问题”，比强求初始提示词达到100%的精准度更有价值。 (6:08) 不同受众对准确性的要求存在分歧：结合人机协同验证，内部AI工具可容忍较低的准确率（约80%），而面向客户的部署则要求近乎完美的精准度。 (10:16) 在缺乏扎实的数据完整性、规范治理及直接源引用的情况下部署前端对话式AI，会导致应用落地失败。 (13:41) 尽管“思维之速”对于欺诈防范和福利查询等实时操作至关重要，但战略决策需要深思熟虑的框架构建，而非快速、无引导的查询。 (19:31)

Enterprise analytics leaders indicate that moving beyond static dashboards toward AI-assisted decision intelligence requires tailoring model accuracy to operational risk, establishing rigorous semantic data foundations, and demonstrating tangible return on investment. Calibrating Accuracy to Risk and Workflow Context (6:08) Analytics leaders argue that universal standards for artificial intelligence accuracy are impractical, requiring organizations to vary thresholds based on task risk and target audience. Goutham Budati of The Farmer's Dog contends that early exploratory querying prioritizes conversational flow over immediate perfection, as strategic clarity often develops only by an executive's tenth query. To contain risk, early conversational agents were restricted to power users possessing deep operational context. Mona Hassanzadeh of Meridian Credit Union establishes a operational divide: internal-facing tools can tolerate lower accuracy thresholds (such as 80%) because internal staff act as a human-in-the-loop to catch anomalies, whereas client-facing applications require near-perfect precision to protect brand trust. Lovedeep Saini of Conner Strong & Buckelew observes an identical split in insurance workflows, where matching binder terms against final issued policies demands deterministic, zero-tolerance verification, while broad policy exploration tolerates directional parsing. Data Foundations and Semantic Ontologies (13:41) Deploying conversational user interfaces frequently exposes unaddressed weaknesses in enterprise data hygiene. Saini notes that Conner Strong & Buckelew initially built a front-end platform to meet executive demand following ChatGPT's release, only to realize the underlying data across customer relationship management systems, claims records, and unstructured policy documents was fragmented and unvetted, forcing an immediate pivot to data quality audits and governance. Budati describes AI as an organizational mirror that highlights deficiencies in internal documentation when public tools locate company information faster than internal systems. To prevent conflicting answers across departments, Ayman Elhalwagy of RootCause.ai emphasizes grounding models in purpose-built ontologies rather than relying on unconstrained large language model reasoning. Defining explicit business logic, min-max operating ranges, and cross-column dependency rules allows systems to deliver deterministic metric retrieval for core key performance indicators while reserving probabilistic calculations for speculative causal scenarios. Evolving Decision Frameworks and Organizational Roles (32:39) The panel highlights an operational shift from static reporting toward proactive decision design. Hassanzadeh identifies the "Design Architect" as an emerging organizational role bridging the gap between raw data infrastructure, semantic modeling, and business decision support, replacing traditional business intelligence roles that focused strictly on report visual execution. Concurrently, Budati projects that specialized titles such as "AI Product Manager" will disappear within three to five years as artificial intelligence becomes an assumed operational utility. To ensure analytical insights translate into business action, Budati instituted weekly cross-functional one-pagers produced by small teams. These structured insight sessions intentionally avoid routine operational weekly business reviews, focusing instead on testing business hypotheses and investigating operational anomalies to build shared strategic context. Measuring Portfolio Returns and Operational Productivity Evaluating returns on enterprise AI investments requires moving beyond adoption metrics toward quantified financial impact. Budati advocates for a venture capital portfolio model, noting that out of roughly 50 experimental projects over a 6-to-12-month period, only two or three delivered outsized, measurable financial value. Hassanzadeh enforces a contrasting upfront mandate, requiring projects to demonstrate clear cost avoidance—such as full-time equivalent headcount savings or vendor reductions—prior to development, while treating regulatory compliance and fraud detection as non-negotiable operational requirements. From a service perspective, Saini measures returns by operational capacity, explaining that automated document comparisons and review drafts enabled internal teams to scale high-touch advisory services previously reserved for elite accounts down to mid-sized clients. Elhalwagy divides causal AI value into first-order returns, measured by direct time saved from manual data pulls and interviews, and second-order returns, captured through the opportunity cost of identifying business issues early enough to intervene. While conversational platforms and causal models can streamline enterprise workflows, panellists emphasize that AI cannot drive operational action independently without explicit financial metrics, strong data ontologies, and executive buy-in to enforce behavioral change.

在企业分析从静态看板向智能决策演进的过程中，行业专家指出企业不应片面追逐全自动化或即时响应，而应将重点转向业务因果解释、底层数据治理以及与具体工作流严密咬合的确定性回报。 决策流重于即时精度与思维速度神话 (6:08) The Farmer's Dog 数据战略与洞察副总裁 Goutham Budati 提出，数据团队在过去二十年过度沉迷于构建数据仓库和仪表盘，却忽略了参与实际决策这一“最后一公里”。他认为在探索初期，连续的思考流比追求 100% 精度更为关键，因为业务人员往往需要迭代提问至第 10 个问题才能触及核心；为此，该团队初期仅向具备深厚业务背景的高级用户开放对话式数据智能体，以在控制风险的同时促成战略理解与预算支持。同时，专家团队对商业智能领域常宣传的“思维速度”（speed of thought）概念进行了反思。虽然在客户服务福利查询或毫秒级欺诈检测等场景下即时响应不可或缺，但在高层战略决策中，未经推敲的快速查询往往价值有限，快速迭代并不等同于仓促得出结论。 精度分层与底层本体防范单体思维 (10:16) 企业应用 AI 必须依据受众与场景实施精度分层。Meridian Credit Union 的 AI、数据科学与创新负责人 Mona Hassanzadeh 明确区分了对内与对外系统：内部智能体可容忍较低精度（如 80%），依靠员工这一“人在回路”在最终拍板前纠偏；但面向外部客户的工具必须达到极高精度，否则将直接损害品牌声誉与信任。Conner Strong & Buckelew 的首席分析官 Lovedeep Saini 以保险理赔与承保为例指出，核对客户暂保单（binder）与最终保单条款是否一致（例如 100 美元必须精准对应 100 美元）属于刚性零容忍场景，而保额宽泛探索则可容忍概率性偏差。Budati 进一步警示，切忌陷入认为静态看板或自主智能体能通包一切的“单体思维”（monolithic thinking）。针对语义混乱问题，RootCause.ai 首席执行官 Ayman Elhalwagy 强调必须建立定制化业务本体，明确业务逻辑、数值上下限约束与依赖规则，使标准指标检索保持 100% 确定性，同时对其余推测性场景输出明确的置信度评分，避免因各部门定义不一而产出相悖答案。 数据治理反思与设计架构师新兴角色 (13:41) 在非结构化文档处理与模型落地过程中，多位专家指出企业常因盲目跟风前沿技术而忽视底层基础。Saini 坦承其团队曾因 ChatGPT 热潮而在高管推动下优先构建了面向保险经纪人的前端 AI 工具，随后却发现底层数据存在缺失、碎片化且缺乏质量审核，最终不得不折返重新加固数据治理与完整性审计。他们后续构建的统一语义平台为所有分析结论附带了精确到文档、页码与段落的原文引注，以此建立专业风控团队的信任。针对内部流程缺陷，Budati 总结道：“AI 是一面镜子，能照出企业自身有多好或有多丑”，内部智能体若无法答题往往暴露的是文档规范和项目管理的缺失。Hassanzadeh 则倡导设立“设计架构师”（Design Architect）这一新兴组织角色，打破以往仅关注 Power BI 等可视化工具的窄化视角，全面统领从底层数据管道、语义层设计到决策支撑的全流程。 从风险合规到因果分析的投资回报测算 针对企业级 AI 的投资回报率（ROI），专家们提出了多元核算路径。Budati 提倡采用风险投资（VC）组合思维，接受高频试验中仅 2% 至 3% 的核心项目能产生决定性财务效益；其团队在过去半年至一年的约 50 项尝试中，正是依靠两到三个高影响力的头部项目拉动了实际美元收益。Hassanzadeh 则坚持在开发任何非强制性 AI 之前进行严格的前期 ROI 核算（如节省全职人力成本或削减第三方供应商开支），同时指明反洗钱与风险合规属于不可妥协的必做项目。Saini 将指标从使用率转向实际生产力与服务扩容，例如通过自动化保单对比与报告草拟，使运营团队能够以原本仅限顶尖客户的高端咨询标准去覆盖中小型客户。Elhalwagy 将因果 AI 的价值划分为两层：一阶 ROI 体现在替代人工跨源数据提取和访谈所节省的时间，二阶 ROI 则体现在提前识别运营瓶颈并介入调整所避免的决策机会成本。 专家一致认为，企业数据智能的成败最终取决于从单向看板转变为支持因果推断与实际行动的业务流闭环。但随着技术逐步演化为底层基础设施，如何在没有高管绝对支持或外部管理咨询介入的情况下，真正推动各业务部门打破惯性、采纳 AI 决策建议，依然是企业留待攻克的深层组织挑战。

## Background 本视频由科技频道“Ai4”于 2026 年 8 月 21 日发布在 YouTube 平台。根据页面简介，本场圆桌讨论聚焦商业智能（BI）如何从传统静态看板演进为由生成式与预测性 AI 驱动的实时洞察体系，多位来自金融、保险、D2C 及初创企业的领袖共同分享了跨部门落地 AI 决策智能与重塑数据交互的实战经验。 ## TL;DR 圆桌深入探讨了企业从传统静态报表迈向对话式 BI、因果 AI 与决策智能（Decision Intelligence）的演进路径。与会嘉宾指出，AI 探索早期应优先保障业务思考的思维流动（Flow State），而在核心财务与合规场景中则必须依托统一的语义层、业务本体与因果模型来锁定确定性并建立业务信任。讨论还揭示了 AI 作为组织成熟度“照妖镜”倒逼底层治理的现象，展望了“设计架构师”等新角色的崛起，并就风投下注模式、前置硬性财务指标及生产力扩容等不同维度的 AI 投资回报率（ROI）衡量标准展开了激烈思辨。 ## Atmosphere & style 整场圆桌讨论极度务实且充满思辨张力，主持人 Mark Ritzmann 以幽默犀利的破冰定调全场，开门见山摒弃空洞理论而直面实操教训。嘉宾们毫不避讳地自嘲“先做 AI 却发现没有数据”等踩坑经历，在面对“随思而动是否是神话”与“ROI 评估哲学”等尖锐议题时各抒己见、火花四溅。现场叙事节奏紧凑、层次分明，既有硬核的技术架构剖析，又有深刻的组织变革反思，带给观众极具启发性与沉浸感的业界前沿对话体验。 ## Key takeaways - [00:04](https://www.youtube.com/watch?v=KDOZqQ-tgqk&t=4s) 商业智能正在从早期的手工取数与静态看板报表，向以数据智能体（Data Agent）驱动的自然语言交互与全生命周期决策智能跨越。 - [06:08](https://www.youtube.com/watch?v=KDOZqQ-tgqk&t=368s) 在早期战略探索中，提问的“思维流动状态”（flow state）远比绝对准确率重要，追问推敲出的深度问题往往比初始问题更有价值。 - [10:56](https://www.youtube.com/watch?v=KDOZqQ-tgqk&t=656s) 对话式商业智能（Conversational BI）改变了取数模式，一线人员可通过 Web 智能体即问即答，但成败核心依然系于底层的统一数据基座（Data as a Service）。 - [13:37](https://www.youtube.com/watch?v=KDOZqQ-tgqk&t=817s) 企业级 AI 系统对准确率的要求因场景而异：外部面向客户以及财务、法务等场景容错率极低，而内部运营及早期推演更关注效率与人在回路（HITL）复核。 - [15:37](https://www.youtube.com/watch?v=KDOZqQ-tgqk&t=937s) 处理非结构化文档需要构建可信语义层与精准引文溯源（Citation）机制，以消除幻觉并打通跨系统数据孤岛建立 360 度全景视图。 - [17:34](https://www.youtube.com/watch?v=KDOZqQ-tgqk&t=1054s) 狂热风潮下容易出现“先做 AI 前台、后补底层数据”的本末倒置，导致必须回过头来做数据完整性校验与质量审计等基础补课。 - [20:23](https://www.youtube.com/watch?v=KDOZqQ-tgqk&t=1223s) 所谓“随思而动”（Speed of Thought）在应急客服或实时风控拦截中是绝对刚需，但在宏观商业战略分析中若缺乏清晰思考框架则只是营销神话。 - [23:11](https://www.youtube.com/watch?v=KDOZqQ-tgqk&t=1391s) 建立系统信任的核心在于因果推断（Causal AI）的可解释性，AI 系统应量化传达置信度与边界，而非仅关注表面的准确率指标。 - [25:26](https://www.youtube.com/watch?v=KDOZqQ-tgqk&t=1526s) 构建语义层与业务本体时必须以具体痛点和问题为导向，定义字段边界与约束规则，严防盲目“煮沸大海”。 - [27:53](https://www.youtube.com/watch?v=KDOZqQ-tgqk&t=1673s) 通用大模型在前瞻假设分析中具有随机性，结合语义层与因果推断模型才能保障输出的确定性或具有明确误差边界的推测。 - [32:39](https://www.youtube.com/watch?v=KDOZqQ-tgqk&t=1959s) 决策智能时代催生了“设计架构师”（Design Architect）新角色，从业者需跳出单一报表工具，掌握语义建模、数据架构与决策闭环设计。 - [34:57](https://www.youtube.com/watch?v=KDOZqQ-tgqk&t=2097s) AI 是一面检验组织成熟度的“照妖镜”，会无情暴露底层系统、语义层、知识沉淀及项目管理文档上的严重欠缺。 - [34:57](https://www.youtube.com/watch?v=KDOZqQ-tgqk&t=2097s) 未来不会存在专职的“AI 产品经理”头衔，AI 将如当下的互联网一样融入所有常规岗位中，组织需聚焦核心业务流程并降低摩擦。 - [39:11](https://www.youtube.com/watch?v=KDOZqQ-tgqk&t=2351s) AI 项目的 ROI 衡量模式存在多元哲学：包括类似风险投资的容错试错模型、前置严苛财务节约指标、生产力扩容指标，以及因果分析带来的决策时效性二阶价值。 ## Detailed walkthrough ### [00:04](https://www.youtube.com/watch?v=KDOZqQ-tgqk&t=4s) 开场破冰与圆桌议题定位 主持人 Mark Ritzmann（哥伦比亚大学人工智能与机器学习教师、SPS AI 实验室联合主任）首先为接下来的 45 分钟讨论做了整体框架设定。他以幽默的开场白化解气氛，笑称自己“从未为台上的任何一位嘉宾打过工，他们也没人为我打过工，彼此之间也无从属关系”，引来全场笑声。他明确指出，本次圆桌的核心原则是“极度务实”（extremely practical），拒绝空洞的理论堆砌，而是聚焦四位业界领袖的真实工作实践、踩过的坑（fumbles）、沉淀的最佳实践以及从教训中学到的经验。 ```mermaid flowchart TD accTitle: 现代商业智能与数据叙事演进路径 subgraph Legacy["传统阶段"] A["手动取数时代<br/>(手工写 SQL / 提需求等排期)"] --> B["自助 BI 时代<br/>(Cognos / Business Objects / 拖拽看板)"] end subgraph Modern["智能时代"] B --> C["公民数据科学家时代<br/>(业务人员探索建模)"] C --> D["AI Agent / 决策智能<br/>(自然语言交互与流动式思考)"] end ``` 谈及 BI 与数据叙事（Data Storytelling）的演变，主持人引用了 Pat Gelsinger 前一天的观点——“必须将技术置于问题或机会的近旁”。回顾数据交互的演进历史，从早期业务人员必须提工单、等技术团队手写 SQL 脚本返回结果，到 Cognos、Business Objects 等拖拽式自助 BI 的兴起，再到所谓“公民数据科学家”（citizen data scientist）的热潮，数据团队始终试图让业务更便捷地触达数据。如今生成式 AI 与 Agent 技术的爆发，正是让这一过程变得更快、更好的契机。接下来的讨论将纵贯数据准备、工作流优化、Human-in-the-loop（人在回路）、组织变革管理与心智模型构建，并前瞻因果推断（Causality）、长期 ROI 以及未来可能涌现的新角色与商业机遇。 --- ### [03:24](https://www.youtube.com/watch?v=KDOZqQ-tgqk&t=204s) 嘉宾阵容介绍：从高能物理科研到跨行业落地 圆桌嘉宾来自不同行业背景，分别介绍了各自的业务场景与技术专注方向： - **Ayman Elhalwagy**：RootCause.ai 联合创始人兼 CEO。公司深耕因果人工智能（Causal AI），核心使命是帮助企业不仅看清数据表象，更能理解“业务指标为何会发生变动”以及“下一步究竟该采取什么行动”。 - **Lovedeep Saini**：保险经纪公司 Conner Strong & Buckelew 的首席分析官（CAO）。他的职责是为客户提供风险覆盖建议，并通过 AI 和现代 BI 工具为业务用户赋能决策智能（Decision Intelligence）。令人瞩目的人才是他的跨界背景——他拥有瑞士欧洲核子研究组织（CERN）大型强子对撞机（LHC）的高能物理学博士学位，此前在学术界做质子碰撞的数据分析与挖掘，如今将高深的大规模数据处理经验降维迁移到商业保险决策中。 - **Mona Hassanzadeh**：Meridian（金融机构/信用社）的数据科学、AI 与创新负责人。她拥有工程学博士学位，带领的团队横跨四大支柱：Agent 开发、客户 AI 产品研发、前沿研究以及被其定义为“决策智能”的现代 BI 体系。 - **Goutham Budati**：D2C 宠物鲜粮独角兽 The Farmer's Dog 的数据与 AI 负责人。他在广告营销与代理商体系深耕多年，深谙“如果数据不能帮人建立直觉并驱动实际行动，就毫无价值，甚至会导致客户流失”的残酷现实；后来转向品牌方担任数据领导者，全力推动用数据讲好故事、倒逼关键决策对话。 --- ### [06:08](https://www.youtube.com/watch?v=KDOZqQ-tgqk&t=368s) The Farmer's Dog 实践：战略思考需要的是“流状态”，而非过早追求准确率 Goutham Budati 针对数据行业的现状发表了发人深省的见解。他指出，过去二十年来，数据和 BI 从业者在某种程度上“迷失了方向”——大家沉迷于搭建数据仓库、数据集市（Data Marts）以及深度绑定各种 BI 看板工具，却离真正的业务决策越来越远。因为决策需要深度的直觉和对利益机制的理解，而技术人员往往只交付了资产，却丢失了连接“收集数据”、“让数据可用”到最终“让数据产生商业价值”的最后一公里。 面对 AI 这个“如同魔法玩具箱般无所不能”的新工具，Goutham 认为最重要的不是“AI 能做什么”，而是“你问了什么样的问题”。他提出了一个反共识的鲜明观点：**在早期探索与战略思考中，持续的思维流动（flow state）远比绝对的准确率（accuracy）更重要。** > "I strongly believe the first question is never the best question; your 10th question or so on is the best question." ```mermaid flowchart LR accTitle: 洞察生成与业务沉淀飞轮 A["对话式 Data Agent<br/>(无缝多轮提问 / 激发直觉)"] --> B["业务骨干结对<br/>(探索边缘盲区 / 深度思辨)"] B --> C["每周单页洞察会<br/>(One-pagers of Insights)"] C --> D["组织级上下文沉淀<br/>(应对突发决策与长期变革)"] D --> A ``` 为了让提问的流速彻底爆发，The Farmer's Dog 早期在尚未实现 100% 准确率时，就将对话式 Data Agent 开放给极少数深谙业务逻辑的高阶超级用户（power users）。这些用户不仅未被小误差误导，反而被无阻碍的提问体验大幅激发了商业洞察，进而反向论证了企业加大 Agent 投入的价值。随后，团队才在此基础上层层补充数据治理与准确率保障。 - **拒绝用 AI 替代战略思考**：企业必须对自身最终目标（end goals）保持极度清醒，“千万不要去问 AI 我该做什么”，而是想清楚要做什么后，利用 AI 的力量去加速与优化。 - **制度化沉淀深度洞察**：坚决摒弃形式主义且枯燥的传统周度业务汇报（WBR），代之以每周一或周二举行的“单页洞察会”（One-pagers of insights）。由 2~3 人的跨职能小组结对，利用 AI 与数据深挖业务死角，撰写洞察并展开激烈攻防辩论，为企业积累长期的“决策认知疤痕”（scars and learnings）。 --- ### [10:16](https://www.youtube.com/watch?v=KDOZqQ-tgqk&t=616s) 从静态看板到全生命周期决策智能 接续 Goutham 的分享，Mona Hassanzadeh 强调 AI 在如今的企业环境中已不再是可有可无的“锦上添花”（nice to have），而是维持竞争力的“必选项”（must have）。传统的静态报表模式——散落在各处的 Excel 文件、固定排程的 SQL 脚本、Tableau 与 PowerBI 静态看板——正在迅速退场。企业当前的核心转变，是将割裂的数据源端到端地打通整合（涵盖云端与本地各种异构源），全面捕捉客户/会员的全生命周期旅程，从被动的指标呈现全面迈向全局实时的决策智能体系。 ### [10:56](https://www.youtube.com/watch?v=KDOZqQ-tgqk&t=656s) 对话式 BI 与数据智能体：从被动取数到主动交互 Mona Hassanzadeh 探讨了深入理解业务问题与客户行为的关键意义，强调只有明确业务目标，才能提出正确的问题并构建真正贴合实际的指标报告路径。当前企业的常态化实践正经历重大变革：团队虽然依然在使用 Power BI 等工具，但底层逻辑已经进化为以全景式、实时接入的数据为支撑，借助“数据智能体”（Data Agent）实现即问即答。这种全新的工作流被称为“对话式商业智能”（Conversational BI）。 以往的业务部门或一线人员需要跑来找数据团队提需求，比如询问“能不能给我拉一份特定类别的会员/客户名单用来做营销活动”，而现在他们可以直接登录后端的 Web 应用程序，通过智能体自主发起多轮提问、深入交互并获取精准解答。 谈及准确率（Accuracy）这一核心挑战时，Mona 指出必须严格区分应用场景与目标受众： - **面向外部客户/会员的智能体（Client-Facing / Member-Facing）：** 容错率极低，直接关乎企业的信誉与声誉。一旦回答失误或数据造假，将直接对品牌造成无法挽回的破坏。 - **内部员工使用的智能体（Internal-Facing）：** 容错空间相对更宽，数字若出现轻微浮动，内部团队能够及时发现，并且系统通常配备“人在回路”（Human-in-the-loop）机制来进行复核。 所有 AI 模型与智能体的成败最终都系于底层数据基座。数据资产不仅需要高度准确，更要全量覆盖业务的全生命周期——从会员的资产构成到客户的总价值，所有概念与实体之间的关联与交互都必须作为“数据即服务”（Data as a Service）坚实落地，才能为上层的大模型与智能体提供赖以信任的燃料。 ```mermaid flowchart TD accTitle: 对话式 BI 与数据智能体架构流转 subgraph ClientFacing["外部客户场景 (极高准度)"] CF["客户 / 会员"] -->|"多轮对话提问"| WebApp1["Web 智能体应用"] WebApp1 -.->|"严苛风控 / 声誉保护"| Rep["品牌信誉"] end subgraph InternalFacing["内部业务场景 (人在回路)"] Internal["业务 / 运营 / 营销团队"] -->|"自然语言交互"| WebApp2["对话式 BI 门户"] WebApp2 --> HITL["人在回路 (Human-in-the-Loop) 复核"] end subgraph Foundation["底层统一数据基座 (Data as a Service)"] DataLake["全域实时数据 / 语义层"] DataLake --> WebApp1 DataLake --> WebApp2 end ``` --- ### [13:37](https://www.youtube.com/watch?v=KDOZqQ-tgqk&t=817s) 保险经纪业务的痛点：海量 PDF 文档与 Nova / Iris 统一平台 Lovedeep Saini 顺着 Mona 关于用户画像与准确率的讨论展开。他直言，内部系统对准确率的耐受度完全取决于业务部门： - **运营部门（Operations）：** 应对趋势与概览时，也许能接受 80% 的准确率。 - **法务（Legal）与财务会计（Accounting）：** 对准确率有着刚性要求，绝无妥协余地。若财务向客户出具账单或收据时将应收 1000 美元写成 100 美元，将引发毁灭性灾难。 随后，Lovedeep 结合自己在保险经纪行业的实战经历，拆解了日常面临的最大痛点——非结构化文档。他们日常处理的核心数据往往“深埋在长达 200 页的 PDF 文件中”。运营人员必须花费数小时翻阅并比对当下的 200 页保单、上一年度已过期的旧保单，以及来自 10 家不同保险承保商（Insurance Carriers）的竞标报价方案，从而为客户寻找能够覆盖特定风险（XYZ Risk）的最优承保方案。 为了从繁重的人力审阅中解脱出来，团队构建了企业级 AI 与 BI 融合平台（内部命名为 Nova、Iris 等品牌）。一线人员无需在乎底层是 Python 脚本、SQL 查询还是连接了 Salesforce、代理机构管理系统（Agency Management System）或理赔系统，他们只需在前端以自然语言询问：“针对该客户或该行业，今年该风险的保费是多少？与去年相比涨了多少？为什么会上涨？” - **打破 20+ 数据孤岛：** 打通 Salesforce、理赔与保单管理等超过 20 种不同数据源，生成全方位的“客户 360 度全景视图”（Client 360 View）并实现同类客户的横向基准对比（Benchmarking）。 - **可信语义层与精细引文（Citation）：** 系统内部构建了统一的语义层（Semantic Layer），不仅输出答案，还会明确告知引证来源：“该数据点出自知识库 B 中的文档 A，具体位于第 200 页的第 10 页第 2 段”，以彻底消除幻觉、建立业务信任。 --- ### [17:34](https://www.youtube.com/watch?v=KDOZqQ-tgqk&t=1054s) 踩坑与反思：兴奋狂热之下的“先做 AI，后补数据” 回顾平台的搭建历程，Lovedeep 分享了全团队付出沉重代价才换来的一条惨痛教训。ChatGPT 问世引发狂热风潮后，公司上下都急于与 AI 互动，于是团队迅速搭建起了前台 AI 平台。但系统跑起来后才尴尬地发现：**底层根本没有数据**。 各业务系统的集成接口虽然搭建就绪，但所需的业务数据根本没有上传或保存在对应的文档目录和知识库中。现实逼迫技术团队不得不掉头补课，去反向要求业务人员补全数据并规范录入：“只有把数据录入完整，AI 才能给出你想要的答案。”原本理应“基础先行”的工程，在全员狂热追求 AI 的浪潮下变成了“先建空中楼阁，再回过头来做数据完整性校验（Data Completeness）、质量审计（Data Auditing）并补齐地基”。Lovedeep 自嘲道，这确实是本末倒置，但也是很多企业在 GenAI 早期必须经历的阵痛。 --- ### [18:35](https://www.youtube.com/watch?v=KDOZqQ-tgqk&t=1115s) 落地成效与业务场景：营销、财务反洗钱与续保运营 面对主持人关于“谁是这套系统最大、最主要的受益用户”的追问，几位嘉宾从各自的职能维度给出了鲜明对比的答案： - **Goutham Budati（营销与运营视角）：** 营销部门（Marketing）是 AI/ML 最核心的使用者，原因在于营销投资与回报之间的反馈闭环（Feedback Loop）极快，能快速通过数据检验模型有效性。尽管客服团队看起来是部署智能体最直接的场景，但由于直面终端消费者（Consumer-Facing），声誉与合规风险巨大，因此他们在此方向推进得极其审慎克制。 - **Mona Hassanzadeh（金融与风控视角）：** 虽身处金融机构，但作为企业级 AI 与数据科学团队，核心服务的是反洗钱（AML, Anti-Money Laundering）与反欺诈（Fraud Detection）这类高风险领域，其次才是驱动会员增长的营销分析与日常运营。 - **Lovedeep Saini（保险经纪视角）：** - **运营与续保（Operations & Renewals）：** 运营团队消耗了绝大部分查询吞吐量（Volume）。得益于企业极高的客户留存率，保单续保工作每天都在大量依赖这套 AI 工具来核实繁琐条款。 - **前台销售（Producers / Sales）：** 真正驱动企业新增营收（New Business）的是这群前台业务撮合人员。为了带来实打实的 ROI，技术团队正在重点为他们打造极具吸引力的赋能工具，帮助销售去外部市场争抢客户、拓展新业务。 --- ### [20:23](https://www.youtube.com/watch?v=KDOZqQ-tgqk&t=1223s) “随思而动”（Speed of Thought）是刚需还是神话？ 主持人抛出了商业智能和仪表盘领域多年来奉为圭臬的一个经典营销口号——“随思而动”（Speed of Thought，即人想到什么，数据系统就能毫无延迟地交互并跟进展现），并询问这究竟是名副其实的刚需，还只是供应商生造的营销神话。嘉宾们针对不同业务语境展现了深度思辨： | 嘉宾 / 业务场景 | 对“随思而动”的判断 | 核心理由与现实权衡 | |:--- |:--- |:--- | | **Lovedeep Saini**<br>（员工福利业务） | **绝对的刚需（高危即时场景）** | 雇员致电咨询救急问题（如“我该项急症治疗是否在保？”、“最近的药房能否买到该药并在保单内报销？”），业务人员必须分秒必争即时给出准确定论，属于现实中必须被满足的紧急响应。 | | **Mona Hassanzadeh**<br>（实时金融风控） | **实时是刚需，其余重在可用** | 在欺诈拦截（Fraud Detection）场景下，AI 的实时毫秒级判断属于核心红线；但在决策智能（Decision Intelligence）领域，“指尖上的对话式探索”只要能顺畅回答业务问题即可。 | | **Goutham Budati**<br>（战略与商业分析） | **在战略层面依然是“神话”（Myth）** | 如果分析师或决策者自身思路混乱，仅凭脑子里蹦出的零碎念头就随意索取数据，毫无意义。“随思而动”只有在建立起严密的思考框架时才有价值；追问过程中的第 10 个深入问题（经过层层筛选和推敲后的问题），永远比未经思考的前 9 个零碎闪念要重要得多。 | --- ### [22:15](https://www.youtube.com/watch?v=KDOZqQ-tgqk&t=1335s) 初创企业的解法：从通用抽象看 RootCause 平台与系统信任 主持人转向初创公司 RootCause.ai 的联合创始人兼 CEO Ayman Elhalwagy，指出企业内部构建系统往往是“按需定制、目标专一”（Purpose-Built），而作为独立软件开发商，初创团队必须在面对多元外部客户时做出截然不同的架构抉择。 Ayman 赞同这一观点，并以创业者的视角阐释了产品哲学：构建商业软件产品时，绝不能只盯着单个定制化需求，必须站在一定的“抽象层级”（Level of Abstraction）去洞察问题本质，打造具有高度泛化能力（Generalizable Solution）的工作流与核心算法，这样才能在市场上开辟并支撑起真正的商业赛道。在谈到打造 RootCause 平台时的底层思考时，Ayman 强调这与前面嘉宾们提到的核心痛点高度契合——归根结底，大模型时代智能系统最底层的生死基石就是“信任”（Trust）。 ### [23:11](https://www.youtube.com/watch?v=KDOZqQ-tgqk&t=1391s) 落地采纳、因果可解释性与业务本体层的构建 Ayman Elhalwagy 首先谈到了技术落地的核心挑战——“采纳度”（adoption）。企业可以构建出能够运行的系统，但模型给出的答案时对时错，如何提供抓手让业务人员持续使用并推动真正的组织变革，关键在于“因果关系”（causality）所带来的可解释性。面对探索性或推测性（speculative）问题时，系统不一定非要给出绝对精准的答案，但必须能够量化并向用户明确传达置信度（confidence），以便业务人员结合自身判断做出决策。目前许多 AI 产品往往只关注表面的准确率指标（headline accuracy metrics），却忽略了对置信度及其后续使用方式的量化评估。 ```mermaid flowchart TD accTitle: 基于业务本体与语义层的因果建模流程 subgraph Semantic_Layer["语义与本体层 (Single Source of Truth)"] direction TB Ontology["领域业务本体<br/>(Business Ontology)"] Logic["业务逻辑与规则定义<br/>(如: 客户流失 Churn / 活跃用户定义)"] Bounds["边界与约束<br/>(Min/Max 阈值、列间关联限制)"] Ontology --- Logic Ontology --- Bounds end subgraph Applications["下游推测与因果推理"] direction TB CausalModel["因果模型 (Causal Models)<br/>解释 Why & 推测 What-if"] Confidence["量化置信度输出<br/>辅助人类决策判断"] CausalModel --> Confidence end Semantic_Layer --> Applications ``` 紧接着，Ayman 强调了“语义层”（semantic layer）的重要性。如果跨系统或跨团队对同一个术语（如产品“品类” category）的定义存在偏差，最终得出的结果必然自相矛盾。RootCause.ai 采用的解决方案是构建一个单一事实来源（single source of truth）的业务本体，其中固化了特定业务场景下的核心逻辑与规则——例如公司如何精确计算流失率（churn）或定义活跃用户。通过这种方式，系统在面对具体指标查询时能给出可复现的确定性答案，在面对下游复杂的因果推断与情景推演时也能提供坚实可靠的上下文基准。 --- ### [25:26](https://www.youtube.com/watch?v=KDOZqQ-tgqk&t=1526s) 拒绝单一体思维：以问题为导向，避免“煮沸大海” Goutham Budati 针对单一体（monolithic）思维提出了警示。无论是过去认为仪表板可以解决所有问题，还是现在以为 AI 是包治百病的灵丹妙药，都是不切实际的。现实中许多企业甚至为同一项业务建了上百个重复或低效的仪表板，或者跟风部署了数百个目标模糊的 AI Agent。他指出，工具本身不是最终目的，唯有具体的使用场景（use case）和实际业务痛点才至关重要，准确率的要求也应完全依照业务场景来决定——有些场景允许误差，而有些场景则对精确度有严苛要求。 Ayman 对此深表赞同，并提醒在构建上下文层或本体层时绝不能试图“煮沸大海”（boil the ocean），否则团队会陷入为整个业务绘制巨细靡遗却毫无实用价值的模型泥潭中，带来巨大的维护负担甚至导致结论偏差。在构建语义层时必须坚持问题驱动： - 明确为了精准模拟某个业务场景和探索“假设分析”（what-ifs）到底需要哪些核心要素。 - 定义业务在现实运作中可能落入的边界范围，包括字段的最大最小值、类别枚举值。 - 确立列与列之间的业务约束规则（例如某一列的取值如何制约另一列）。 --- ### [27:53](https://www.youtube.com/watch?v=KDOZqQ-tgqk&t=1673s) 大模型的随机性推演与系统确定性保障 讨论转入系统的“确定性”（determinism）挑战。Ayman 指出，确定性失效通常发生在两个关键节点：一是语义层未对齐，导致大家各说各话；二是检索确定性答案时所用的底层系统不完善。例如，人类大脑清楚某项指标的具体计算公式，但直接与之交互的语言模型却往往一无所知，这正是语义层必须承载的内容。 ```mermaid flowchart LR accTitle: 语言模型自由推测与因果模型确定性对比 subgraph LLM_Approach["通用大模型直接推测"] Prompt["自然语言提问"] --> Hallucination["随机/无约束推测<br/>(缺乏固定算法、引入外部不可控数据)"] Hallucination --> Inconsistent["输出不可控 / 难以复现"] end subgraph Causal_Approach["因果模型 + 语义层"] Query["情景假设 / 指标查询"] --> Semantic["语义层约束 & 确定性算法"] Semantic --> Deterministic["确定性答案 / 有误差边界的概率结果"] end ``` 在面对前瞻性假设问题时，通用大模型往往表现出极大的随机性与不可控性，因为它们缺乏规范的推演算法，可能会随意编写代码，或随意引入外部数据源。相比之下，因果模型能够在可控的机制下输出具有确定性或带有明确误差边界（bound of error）的推测结果。Ayman 总结道，无论是从数据源中检索基础指标、生成图表，还是进行复杂的推测性情景规划，都必须建立起严格的确定性工作流。 --- ### [29:44](https://www.youtube.com/watch?v=KDOZqQ-tgqk&t=1784s) 保险业务场景下的工作流重塑与 ROI 衡准 Mark Ritzmann 将话题引向工作流变革与变革管理（change management），询问如何避免只是“更快、更轻松地制造出毫无用处的垃圾产物”。Lovedeep Saini 结合 Conner Strong & Buckelew 在保险领域的实践做出了回应。他指出，探讨 AI 的投资回报率（ROI）和生产力，归根结底取决于是否建立了客户信任与解决关键痛点。 在保险业务流程中，必须严格核对客户签署的“要保确认书/承保承诺”（binder）与保险公司出具的“最终保单”（final policy）： - **绝对准确的核心场景**：如果承保确认书承诺的保费是 100 美元，而最终保单上却显示为 105 美元，此时准确率就是生命线，必须确保前后数据 100% 严丝合缝。 - **容忍一定弹性的场景**：对于某些特定险种的承保范围或额度细则等其他组件，可以在一定容错范围内运作。 - **业务价值与效率提升**：AI 的深度检索与比对能力帮助业务团队从逐行审查保单的繁重劳动中解脱出来，大幅加快出具客户总结报告的速度，使传统静态的业务流程升级为 AI 原生（AI-native）的高效工作流，而人类专家则专注于结合 AI 建议做出最终判断。 --- ### [32:39](https://www.youtube.com/watch?v=KDOZqQ-tgqk&t=1959s) 决策智能时代下的用例优先级与“设计架构师”新角色 Mona Hassanzadeh 探讨了资源受限背景下的用例筛选机制以及企业角色的演进。尽管 AI 无疑能显著提升效率，但企业内部用于构建 AI 模型的资源始终是有限的。她在加入 Meridian 之初便与各个业务线负责人深入沟通，依据痛点程度与潜在的 ROI 进行排序——ROI 的衡量维度可以涵盖工作效率提升、风险控制、直接营收增长或减少损失。 Mona 提出了一个端到端的建设全景：将数据治理与文化（governance and culture）、数据解决方案（data solutions）、AI 与分析（AI and analytics）紧密融合为一个不可分割的支柱体系。 ```mermaid flowchart LR accTitle: 端到端决策智能支撑体系与能力演进 subgraph Pillars["端到端决策智能支柱体系"] Gov["治理与文化<br/>Governance & Culture"] --- Data["数据解决方案<br/>Data Solutions"] Data --- AI["AI 与高级分析<br/>AI & Analytics"] end subgraph Roles["分析人员能力模型演进"] OldRole["传统报表专家<br/>(仅精通 Power BI 等单一工具)"] --> NewRole["设计架构师<br/>(Design Architect)<br/>掌握语义建模、数据架构与决策设计"] end Pillars -.-> Roles ``` 在决策智能（decision intelligence）的推进过程中，组织内正在涌现出关键的新角色——**设计架构师（Design Architect）**： - 过去团队中可能拥有顶尖的 Power BI 报表专家，但往往缺乏对语义数据建模（semantic data modeling）与底层数据架构的深层理解。 - 现代决策智能要求面向报表与分析的从业者跳出传统工具的局限，主动学习语义层构建、数据优化及系统架构设计。 - 该角色既不同于纯粹的 AI 工程师或数据工程师，而是专注于打通从“底层数据”到“业务答案”再到“最终决策方案”的端到端路径。 ### [34:57](https://www.youtube.com/watch?v=KDOZqQ-tgqk&t=2097s) AI 作为组织成熟度的“照妖镜”与未来角色的演变 Goutham Budati 站起身来直接面向全场观众交流，分享了一个看似违反直觉（counterintuitive）的深刻洞察：构建对话式分析（Conversational Analytics）带来的最大改变，往往不是工具本身的直接效用，而是它暴露出企业在底层系统、语义层（Semantic Layer）以及项目管理与文档规范上的严重欠缺。他将 AI 比作一面镜子——“AI 是一面绝佳的镜子，能照出你到底有多优秀，或者有多丑陋”（"AI is a great interface that is a mirror that shows how good or how ugly you are"）。他举了一个具体例子：团队曾向内部的数据智能体（Data Agent）提问，智能体因缺乏内部数据而无法回答；随后向 Claude 提问，Claude 通过联网搜索给出了答案，而这部分信息原本理应在企业内部有清晰的沉淀。这一尴尬事件倒逼组织重构基础的项目管理流程与工具建设。 谈及 AI 职位的未来，Goutham 类比了互联网的发展史：20 年前人们会把“Internet”或“Digital”写进职位头衔，而今天人人都在使用互联网，早已没人这样标注；同样地，两到三年或五年后，如果有人自称“AI 产品经理（AI PM）”，大家都会感到尴尬（cringe）。企业必须回归基本面，绘制业务流程图，找准瓶颈来降低摩擦（例如在营销声明处理与营销运营流程中去摩擦化）。他提出了两层“同心圆”理念：内圈是实际要完成的业务工作，外圈是 AI 的赋能；团队目前在云端通过 AI 生成仪表板草稿，再沉淀回确定性的报表系统中，推动组织全面迈向决策科学（Decision Science）。 ```mermaid flowchart TD accTitle: 从 AI 驱动到确定性业务落地的双环模型 subgraph Core ["核心业务层 (Inner Circle)"] A["明确业务流程与瓶颈<br/>(业务决策 / 营销声明 / 流程去摩擦)"] B["沉淀确定性系统与报表<br/>(Deterministic Dashboards & Process)"] end subgraph AILayer ["AI 赋能层 (Outer Circle)"] C["AI 交互 / 镜子效应<br/>(暴露语义层与文档缺陷)"] D["AI 智能体探索与生成<br/>(Cloud Dashboards / Claude 辅助)"] end C -->|反推底层规范完善| A A -->|定义分析需求| D D -->|沉淀经验与逻辑| B ``` - **组织诊断**：AI 会无情暴露企业在语义层、知识沉淀和项目管理文档等基础层面的严重缺失。 - **职业趋势**：未来不会再有专设的“AI PM”头衔，AI 将像当下的“互联网”一样融入所有常规岗位中。 - **双环方法论**：内圈聚焦业务本质与确定性交付物，外圈利用 AI 探索与生成以加速决策闭环。 --- ### [37:59](https://www.youtube.com/watch?v=KDOZqQ-tgqk&t=2279s) 变革管理与客户信任的建立 面对主持人关于如何应对组织变革管理（Change Management）与工作流集成的提问，Ayman Elhalwagy 指出，这对于决策支持类软件至关重要。他引用了经典谚语并进行了拓展：“将马牵到水边是一回事，但你能让马喝水吗？答案是不能——除非借助管理咨询公司的力量，或者在最高管理层（C-Suite）建立起坚实的信任度。” 他回顾了自己过去为蓝筹企业（Blue-chip Enterprises）提供 AI 决策咨询的经历，强调无论技术多么先进，用户必须从心底信任系统输出的结果并愿意据此采取行动。对于初创公司而言，在当下的市场环境下让客户真正相信并采纳外部决策软件，是一项前所未有的巨大挑战。 --- ### [39:11](https://www.youtube.com/watch?v=KDOZqQ-tgqk&t=2351s) AI 的投资回报率（ROI）之辩：VC 风险模型、强制性财务标准与生产力指标 针对圆桌最后的焦点议题——如何评估 AI 项目的 ROI，各位嘉宾从不同业务场景给出了差异鲜明的衡量哲学： 1. **风险投资模型（Goutham Budati）**：Goutham 将 AI 项目的 ROI 衡量比作风险投资（Venture Capital）。企业会下很多注，其中可能仅有 2% 到 3% 会真正胜出并带来巨大收益，其他项目都是为下一次突破积累经验（"reps"）。以他过去半年的实践为例，团队有两三个项目带来了可衡量的真金白银收益，而长尾处充斥着大量效果尚不明确的应用，对此类应用无需苛求即时 ROI，只需设定好必要的安全护栏（Guardrails）。 2. **前置财务 ROI 标准（Mona Hassanzadeh）**：Mona 采取了截然相反的严格前置筛选策略。在 Meridian 开发 AI 智能体或模型之前，她强制要求业务方必须先拿出清晰的 ROI 预测，如具体节省的全职员工（FTE）编制成本或软件采购成本。除了合规与强监管下的“必做项目（Must-have）”，所有提升收入或减少损失的项目必须以财务 ROI 为立项前提。她坚信在当前阶段，“没有财务 ROI，其余一切都只是主观意见和无休止的争论”。 3. **分层生产力指标（Lovedeep Saini）**：Lovedeep 介绍了 Conner Strong & Buckelew 的指标演进路径：最初关注采纳率（Adoption Rate），如今聚焦深层生产力（Productivity）。这种生产力体现在服务扩容上——例如业务人员单日服务客户数从 10 家提升，保单审查（Policy Review）与对比速度加快；过去仅有头部精英客户（Elite Clients）能享有的季度业务回顾（QBR）和定制信函，现在得益于 AI 工具，运营团队能够将其覆盖至中小型客户群体。 4. **因果分析的一阶与二阶 ROI（Ayman Elhalwagy）**：Ayman 从 RootCause.ai 的角度将 ROI 拆解为两个层级，并强调软件销售必须最终换算为美元价值： | 评估维度 | 核心机制 | 业务体现与价值 | |:--- |:--- |:--- | | **一阶 ROI (First-Order)** | 自动化因果推断 | 省去大量跨部门访谈与手工拉取数据的繁琐时间，直接通过因果发现定位核心驱动因素（Direct & Indirect Drivers）。 | | **二阶 ROI (Second-Order)** | 机会成本与时效性 | 缩短认知与决策周期。在业务受损前提前洞察原因并快速调整，赚取或挽回因“决策滞后”而损失的潜在利润。 | 主持人最后对各位嘉宾的精彩见解表示感谢，并正式宣布圆桌讨论圆满结束。 ## Notable quotes > [00:04](https://www.youtube.com/watch?v=KDOZqQ-tgqk&t=4s) 必须将技术置于问题或机会的近旁 — Mark Ritzmann > [06:08](https://www.youtube.com/watch?v=KDOZqQ-tgqk&t=368s) I strongly believe the first question is never the best question; your 10th question or so on is the best question. — Goutham Budati > [06:08](https://www.youtube.com/watch?v=KDOZqQ-tgqk&t=368s) 千万不要去问 AI 我该做什么 — Goutham Budati > [17:34](https://www.youtube.com/watch?v=KDOZqQ-tgqk&t=1054s) 只有把数据录入完整，AI 才能给出你想要的答案 — Lovedeep Saini > [34:57](https://www.youtube.com/watch?v=KDOZqQ-tgqk&t=2097s) AI 是一面绝佳的镜子，能照出你到底有多优秀，或者有多丑陋 — Goutham Budati > [34:57](https://www.youtube.com/watch?v=KDOZqQ-tgqk&t=2097s) AI is a great interface that is a mirror that shows how good or how ugly you are — Goutham Budati > [37:59](https://www.youtube.com/watch?v=KDOZqQ-tgqk&t=2279s) 将马牵到水边是一回事，但你能让马喝水吗？答案是不能——除非借助管理咨询公司的力量，或者在最高管理层（C-Suite）建立起坚实的信任度 — Ayman Elhalwagy > [39:11](https://www.youtube.com/watch?v=KDOZqQ-tgqk&t=2351s) 没有财务 ROI，其余一切都只是主观意见和无休止的争论 — Mona Hassanzadeh ## Mentioned Mark Ritzmann, Ayman Elhalwagy, Lovedeep Saini, Mona Hassanzadeh, Goutham Budati, RootCause.ai, Conner Strong & Buckelew, Meridian, The Farmer's Dog, Columbia University, Pat Gelsinger, CERN, LHC, Cognos, Business Objects, Power BI, Tableau, Salesforce, Nova, Iris, Claude, ChatGPT ## Video description Goutham Budati - VP, Data Strategy and Insights, The Farmer's Dog Mark Ritzmann - Faculty in AI & ML, Co-Director SPS Artificial Intelligence Lab, Associate Dean - AI, Analytics, and Strategic Communications, Columbia University Mona Hassanzadeh - Head/Director of Data Science & AI, Meridian Lovedeep Saini - Chief Analytics Officer, Conner Strong & Buckelew Ayman Elhalwagy - Co-Founder and CEO, RootCause.ai BI is moving from static dashboards to real-time insights powered by generative and predictive AI. In this session, leaders will share how they’re enhancing business intelligence with AI—from building tools that understand natural language to enabling faster decisions through automated insight generation. Hear real use cases across finance, operations, sales and marketing, and learn how AI is reshaping what it means to ask a question of your data—and actually get an answer. [Source](https://www.youtube.com/watch?v=KDOZqQ-tgqk)

[0:04] Mark Ritzmann: 必须将技术置于问题或机会的近旁 [6:08] Goutham Budati: I strongly believe the first question is never the best question; your 10th question or so on is the best question. [6:08] Goutham Budati: 千万不要去问 AI 我该做什么 [17:34] Lovedeep Saini: 只有把数据录入完整，AI 才能给出你想要的答案 [34:57] Goutham Budati: AI 是一面绝佳的镜子，能照出你到底有多优秀，或者有多丑陋 [34:57] Goutham Budati: AI is a great interface that is a mirror that shows how good or how ugly you are [37:59] Ayman Elhalwagy: 将马牵到水边是一回事，但你能让马喝水吗？答案是不能——除非借助管理咨询公司的力量，或者在最高管理层（C-Suite）建立起坚实的信任度 [39:11] Mona Hassanzadeh: 没有财务 ROI，其余一切都只是主观意见和无休止的争论

[6:08] Goutham Budati: the first question is never the best question—your 10th question is [19:31] Goutham Budati: The 10th question is still the best question. [35:04] Goutham Budati: AI is a great interface that is a mirror that shows how good or how ugly you are. [35:04] Ayman Elhalwagy: It's one thing bringing the horse to water, but can you make the horse drink water? [40:15] Mark Ritzmann: without financial ROI, everything else is an opinion or a debate. [42:59] Ayman Elhalwagy: everything has to be converted to a dollar value

[0:04] Um, I want to spend a few minutes framing how  we're going to spend the next 45 minutes to uh [0:09] give you an idea of what we want to talk through  and what we want to get to. Uh, and then some [0:14] opening comments uh about dashboards and business  intelligence and storytelling in general. Uh, [0:19] and then I'll turn it over and have the panel  members introduce themselves and then we're [0:22] going to whip through a couple of questions. Um,  I feel it's important to start with saying I've [0:28] never worked for any of these people. [laughter]  Not true. They they've never worked for me and [0:36] they've never worked for each other. So, we debate  that later. Now that we got that out of the way, [0:40] let's go on. Um, so what we really want to focus  on and and what we've prepared for is to be [0:49] um extremely practical in our approach to the work  that we're doing. and uh into the information that [0:55] we share. So when we talk about being practical,  we want to talk about our current work and how [1:00] we've approached it, uh lessons learned, uh good  practices, maybe some fumbles along the way, [1:08] uh and things that we've learned. And when we  talk about these things, we want to be specific [1:13] uh in some of the elements that are around  storytelling and business intelligence. So [1:17] we want to make sure we touch on uh data prep uh  and workflows uh human in the loop some of the [1:24] change management aspects associated with this  and mindset and we have to talk about agents [1:30] because everyone here is talking about agents  um and then we're going to move to sort of a a [1:34] crystal ball what we think is going to happen  next and we want to talk about our experience [1:38] and our point of view around things like causality  uh ROI over the long term new jobs and new roles [1:46] that may emerge emerge in this new environment  uh and new opportunities for stakeholders and [1:51] business owners. So that's what we plan to cover.  Um so let me just whip through a couple of of [1:56] more setup points and then I'll have the panel  introduce themselves and then we'll launch into [1:59] our our series of questions. Uh so dashboards and  storytelling and business intelligence has been [2:06] around for a long time. Um and I think was Pat  Gisinger said yesterday that one of the keys is [2:11] you need to put the technology next to the problem  or next to the opportunity. And when we were doing [2:16] our prep sessions and talking through this, it  seems like business intelligence and dashboards [2:20] and storytelling has been moving for this for  a long time. Um, and and it sort of moved from, [2:26] you know, in the way back days, it used to be, you  know, SQL code and you had to drop off requests [2:30] and a personal return request. Uh, and then we  moved towards sort of selfservice BI and you had [2:36] all these drag and drop tools like business  objects and Cognos and and all those things. [2:41] If anybody remembers the era of the citizen data  scientist when when that was occurring. Um so more [2:48] and more of this is moving towards um you know  this merger and this ability to interact with data [2:54] and to to leverage data for business purposes. Um  and we believe that with artificial intelligence [3:03] it has the opportunity to become one of those  things that becomes better and becomes faster. [3:10] So back to the top when we talk about our work and  practical applications and practicality and advice [3:16] that's sort of the framework that we're coming  from and that's sort of the goal and the things [3:19] that we're moving towards. So let me start with um  having the panel introduce themselves. I'll start [3:24] with my life by my left. I mean if you want to  start and introduce yourself and tell us where you [3:27] work and where you're coming from. Um hi everyone.  I'm Aean. Uh it's great to be here. I'm one of the [3:33] co-founders of Root Cause AI. We are a cause of AI  company helping businesses understand why things [3:39] happen in their business and what to do about it.  That's it. Just a quick introduction right first. [3:49] Hello everyone. I'm Love Deep Snee and I work  for Connor Strong and Buffalo as chief analytics [3:55] officer and Connor Strong and Bucklow is insurance  brokerage. So we serve our clients we recommend [4:01] them that what they should do in order to cover  their risk etc. So I do data analytics for this uh [4:07] business and uh the purpose here is we could give  data and the tools to business users so they could [4:14] be closer to these uh decision making or decision  intelligence platforms. So that's what I'm trying [4:22] to do here with AI BI and all. And prior to that  I was uh working in uh academia as a research [4:30] scholar and I have a PhD I have earned PhD from  large hedron collider CERN at Switzerland. We [4:36] were doing the similar stuff data science data  analytics business intelligence there but with [4:42] proton collisions. So I'm bringing in the same  experience but to the industry world. Uh my name [4:48] is Mona Hassanade. Hello everyone. I'm from Radian  Credit Union. This is the financial industry. I'm [4:55] the head of AI, data science and innovation.  Under my team, we have different pillar from [5:01] agentic from uh customer AI development to the  research as well as the BI which we call it [5:08] decision intelligence. I have a PhD in engineering  a different word I know but um as AI is moving we [5:16] are moving towards uh financial industry as  well. Pass it to you everyone. Uh my name is [5:23] Gautam Bati. I lead data and AI at a pet food D2C  company called the farmer's dog. I uh have grown [5:31] up in my career around advertising primarily and  in those days when I worked at agency side uh what [5:38] good was data if you didn't allow it to build  intuition and ensure that you're taking actions [5:43] and we used to lose clients uh if we didn't have  the right kind of decision- making and that's how [5:48] uh that's my experience and it shaped me really  well into uh when I moved to the brand side or [5:53] working in companies as an internal data leader  essentially ensuring that we are telling the [5:58] right kind of stories so the moment and forcing  conversations to make the right kind of decisions. [6:03] That's my experience and I'm very excited to share  my thoughts on this panel. Why don't we stay with [6:08] you? So, um why don't you kick us off? Uh maybe  describe a little bit of your current work, [6:14] uh what the inspiration was or what the  opportunity was and then what you've [6:18] learned along the way that you can share with the  audience. my um I would say like more just zooming [6:24] out. Um I think uh over the last maybe two decades  or so as data and BI folks we kind of lost the way [6:32] uh because we have been you know working towards  building these assets building data marts building [6:36] data warehouses really attaching ourselves to the  tools the BI tools and do what to essentially help [6:44] with the decision making and I think we have been  participating less and less in that uh because [6:48] it's decision making comes with a lot of uh you  know uh intuition build the the work you know [6:54] incentives around things. what attracted me to the  opportunity here and uh what I'm doing is the uh [7:01] how do we connect the last mile from you know what  data we are collecting uh from making data usable [7:07] to making data valuable uh into the decision  making space that's what I've been uh focusing on [7:12] uh essentially um and you know partly building the  assets to do that uh but I'm also excited about [7:18] the last two years where uh this new tool this  magical toy box called AI you can do anything with [7:25] it But what do you ask? What kind of questions do  you ask to solve your problems? That's what I'm [7:31] uh most excited about. And um the way we are  solving some of the problems essentially are [7:37] um making sure we are enabling uh the flow of  questions uh from our stakeholders to be faster. [7:44] How do we do that? Uh we have a data agent or a  conversational uh you know uh BI that helps people [7:49] unlock. Uh it's actually not about the accuracy  in my opinion and and anybody can argue here. [7:55] I know we are striving towards accuracy but I  would say the flow state the strategic thinking [8:00] doesn't need accuracy it requires a continuous  flow of thoughts continuous flow of questions [8:04] because I strongly believe the first question is  never the best question your 10th question or so [8:10] on is the best question so how do you unlock that  flow and when we unleashed our data agent without [8:15] accuracy uh in place we only gave it to a select  few power users who understand the business flow [8:20] and it unlocked their uh you know u understanding  of the business uh that led us to investments [8:26] uh in the in the data agent uh that now we are  focusing on the governance and accuracy and what [8:31] good looks like but that's the long tale of use  cases u that is what we are solving here in terms [8:36] of uh you know the uh data problems as you just  said bringing you know the technology and the [8:42] business problem closer and working backwards to  a goal another one uh that we are focusing on is [8:48] uh what are the end goals we are going after  really being crystal clear about it and please [8:52] don't ask AI about it what should I do just  know what you need to be doing about it and [8:56] then you know optimizing it uh with uh with with  the power you have with AI. Uh the other thing is [9:03] these things won't happen magically if you don't  have a system around it. Um an example of a system [9:07] uh that I'm very passionate about is having these  insights sessions or one pages of insights that [9:13] you can conduct every Monday or Tuesday. These  are not weekly business reviews by the way which [9:17] are somewhat boring at sometimes. These are really  powerful like what can I do to look at the corners [9:23] of the business. I have this magical tool at my  disposal. I also know how to dive into the data. [9:28] I know the business. I can partner with folks. So  a duo or a trio come together write the insights [9:33] of the business and argue against it. And when the  time comes when it's very critical of solving the [9:39] problems, you have this rich set of insights that  you can really act on and hey this is all context [9:44] in the you know uh realm of insights that you're  building for the organization. I think that slow [9:49] build that scars you build by going through this  process that learnings you have will really help [9:54] you prepare for you know what the higher level  work is 10 20 years from now because this is not [10:00] going to stop here the new technology is going to  come you need to change manage and the next thing [10:04] is going to come um and you know I'd say like I  think we are now going towards uh the way that we [10:10] were lost and finding a path towards closing it  um that's what I'm excited about right now. All [10:16] right. Can you pick it up? Yeah, for sure. I'll  get back to accuracy one at the end. But what I [10:21] wanted to start with is that these days people are  not just doing AI for the sake of AI is not just [10:29] having oh nice to have it it's must have it. What  I see is that what we had before like PowerBI, [10:37] Tableau, all of us work with that bunch of  Excel file, all of the SQL, schedule, dashboard, [10:43] the static reporting. What we are moving towards  is having all of these data holistically together [10:50] either moving it to the cloud, different sources,  but making sure to see end to end of the member [10:56] or client. So to see what's going on, what  behavior we are following to be able to ask [11:01] the right question and provide the answer. So this  is important to know what the business problem is, [11:08] have the business shape the journey that  you are going through for your reporting. [11:14] What we are doing today as a current statement  or the current status of our job is yes we do [11:22] have a powerbi what in different word with the  connection to the data that holistically captured [11:29] in real time holistically answer all of the  question having a data agent that can answer your [11:37] question on the fly so you will ask the question  as a conversational BI that's what we call it to [11:44] answer the question that you have the business  problem that you have. So these days we are not [11:49] waiting for a lobby or the business to come to us  and say oh can you send me the list of the member [11:55] or the client with this specific category I want  that for the marketing. So they can go and ask [12:02] our web application which is behind all of those  the data agents that we have. So they can ask the [12:10] right question and get the answer. Even they can  go back and forth with the chatbot or the agents [12:16] that we have to get the proper answer they are  looking for. In terms of the accuracy that you [12:22] mentioned, I know we had that conversation that's  why I'm going to pick it up. Um it is important [12:27] to know who your audience is. We always says if  you have a aentic which is the member phase be [12:34] careful because client phase agentic is dealing  with your reputation when you have something [12:42] internal as a agent you have working with that. So  even if one number goes up and down the theme can [12:50] pick it up always you have a human in the loop  for the accuracy checking but when this is this [12:56] agent is member phase or the client face this is  out there so you are ruining your reputation if [13:02] the accuracy is down but having said that all of  this is based on the foundation of the data. Your [13:09] data should be accurate. Your data should cover  everything. You cannot afford missing one concept [13:16] of your data like for example the asset for your  members or the total value of the client all of [13:25] those are important to have end to end how they  are interacting with each other. So holistically [13:30] you should have the data good as a foundation  as a service to build up the good AI models [13:37] which I think you can uh pick that up. So Mona  has a really good point that who we are trying [13:42] to serve with these health platforms. So if this  is for external clients then accuracy matters. If [13:49] it's for internal clients then we can survive  with 80% maybe accurate answer and all that. [13:56] Again it depends internally. Are we trying to  serve the legal teams, accounting teams or the [14:03] operations which may be okay with 80% accuracy on  the data but if account teams they have to send [14:11] some sort of uh receipts or some sort of bills to  the clients then oh pay me $100 instead of $1,000 [14:18] that won't work right. So we want to build  accuracy there. So what we where I am coming [14:24] from insurance brokerage business. So where what  we deal with is most of the time PDF files. So [14:32] documents the data that lives under the 200page  PDF file. So our internal operation teams they [14:40] have to read that 200page document. Sometimes they  have to compare today's 200page document with the [14:47] expired policy with the last year's policy with  the quotes that we have received from 10 different [14:54] insurance carriers so that we could suggest our  client that this carrier is best for you to serve [15:01] your XYZ risk. So our operation teams they are  spending almost hours and hours of their time to [15:09] read through those documents to dig information  from those documents. So what we are trying to [15:15] build for them a conversational platform whether  it's AI BEI or a Python program or a SQL query [15:22] in the back end they don't need to know or are we  connecting to CRM or uh to the agency management [15:30] system they definitely want to know that is this  data coming from a reliable trustworthy source [15:38] or not but at the front end they just want to ask  a question that how much was the premium for this [15:46] risk for this client or for that industry this  year versus last year how much it has increased [15:53] and why it has increased. So all these questions  they could get these answers from the front end. [15:59] So what we built for them is we call it uh your  AI enterprise AI and BI platform. So we have [16:07] brand names for those Nova Iris and all that. So  the purpose there is within that single platform [16:13] we are trying to connect Salesforce data. We are  trying to connect where the claims information is, [16:19] where the policy information is. So we um  Gotha mentioned there were so many different [16:26] applications. So we know that Salesforce agency  management systems, claims management systems, [16:33] they all were different platforms applications  where we all had maybe 20 different data sources. [16:39] Now we are trying to unify them so that our users  they could talk to their data see the holistic [16:46] picture they could have the client 360 view they  could uh compare their um client against the [16:54] other similar clients benchmarking and all that  part. So what we are trying is connect your data [17:02] from all different sources, build the internal  semantic layer so that you could feed into these [17:10] AIs and also AIS they could tell these users that  this data point is coming from document A sitting [17:19] in repository B and uh here's the citation of  these 200 pages page number 10 paragraph who is [17:28] telling you this. So trust we want to build trust  in those systems. So that's what we are doing and [17:34] there are definitely many many many things lessons  learned. There are so many lessons learned that [17:39] uh we could spend the whole one hour there. So  but the main one that I would say is we built [17:47] the AI platform first. Everyone got excited after  the start GPT was there right? Everyone wanted [17:54] to interact with AI. We built the platform but  then we realized that data wasn't there. So we [18:00] were trying to build all the integration points  but data wasn't living in those uh doc document [18:06] folders in those repositories. So now we are going  back and what now we are suggesting our users that [18:13] you need to complete your data only then you would  get the answer that you're looking for. So that [18:18] data completeness data quality checks that data  auditing part we are doing that now. That's more [18:24] like you might say that you should have done this  first but everyone wanted AI first. So we did the [18:30] AI first and now we are going back and fixing  the basic foundation too. That's that's the [18:35] lesson learned. Yeah. Before we play just just  a quick followup question. Uh which business [18:40] units or business function is the biggest or most  predominant user of the the thing that you built? [18:48] So I'd say like uh uh marketing for me uh  mainly because it's the feedback loop uh [18:54] between you know your investment and what you get  is faster and I found that to be a place where [19:00] uh we you know depending on how you define AI  for me it's more ML and you know the data agent [19:05] uh you know the conversational that that  is where u you know I apply we also have [19:09] a customer service unit where you might think u  agent is the most easiest answer there but there's [19:14] a lot of risk in terms of uh consumerf facing  So uh we haven't skewed that much uh on that [19:20] front yet but marketing is where I'm I'm coming  from. How about you? I'm from finance. However, [19:25] because I'm the enterprise AI and data science,  we are covering AML fraud. That's one of the big [19:31] one within finance world that we are helping with  the AI as well as um like marketing analytics is [19:38] one of them because we wanted to grow our member.  So that's another area along with the operation of [19:44] course and it sounds like operations so operations  definitely they have the volume uh we have a [19:52] really good uh retention rate of our clients  we do really good business so renewals are the [19:58] ones that where they use most of the time these  tools they need data but where the revenue is [20:04] coming from new business where we want to make the  attractive tools that's the sales side producers [20:11] We do we're going to talk about ROI etc. but we  want to target that team. We want to build tools [20:17] for them so that they could go out and uh attract  more customers, bring new bring in new business. [20:23] And then just real quickly and then we're getting  punks. Um speed of thought has always been [20:30] like a grail for for dashboarding of business  intelligence systems or maybe just a marketing [20:34] tagline. Um how important is speed of thought?  Like is that just a myth? like or or is like [20:42] the interaction and and the ability for speed of  thought and followup question is really important. [20:46] Why don't we start with you this time and go  that way and then we'll come to you. [laughter] [20:52] Okay. Um see where we have employee benefits  business too. That's where our clients the [20:59] external clients and their employees they're  going to call our teams that am I covered for [21:04] XYZ treatment. That's where our employee benefits  team, they need to respond quickly to the answer [21:11] that is this uh pharmacy nearest to this person.  Is this drug covered by the policy that they have? [21:19] I have questions that need to go through. Yeah,  they have to get the answer as soon as they could [21:23] get from the actual emergencies that need to be  addressed. For sure. So, we have a fraud detection [21:28] at the real time. That's one of the key that you  have to be careful about it. So definitely that's [21:34] uh important for the AI detection. Uh but other  than that if we wanted to focus on the business [21:41] intelligence or decision intelligence I would  say conversational AI you know talking to your [21:46] data and get the the answer that you are looking  for at your fingertip. It would be important. [21:52] I think speed of thought is uh I'd say it's still  a myth that you know yes we need the answers but [21:58] to do what uh if you are unclear and you just want  answers to any thought that comes to your mind [22:05] it's of no use uh unless you have a really good  thought framed and now you're going on a quest [22:11] that helps question the 10th question I would  still stick with the 10th question being the best [22:15] question so um you have a different perspective  and a different set of challenges because I think [22:20] you can describe those you purpose-built systems,  specific objectives, whereas you've got to make, [22:25] you know, different choices and and different  addresses. So, you talk about your perspective and [22:29] your approach to this and how you're going after  it. Yeah, absolutely. So, as as a startup founder, [22:35] when you think about building a software product,  you have to look at the problem at a certain level [22:39] of abstraction to create a market for a solution,  right? And you have to look at the problem in a [22:43] way where you can build a generalizable solution,  whether it's a workflow or an algorithm, [22:49] whatever it is. And if I could kind of go into  some of the considerations that we were making [22:58] whilst building the root cause platform, uh it  it's very relevant to what we're talking about [23:04] here. So I think talking about trust and I I think  the the highle word for everything we're talking [23:11] about is really adoption. How do you get people  to use these things? You can build them. They [23:17] can work. They can sometimes come out with right  answers, sometimes come out with wrong answers. [23:21] Uh what leverage points do you have to make sure  that people carry on using this stuff and so that [23:27] it can drive real change in an organization? And  I think one of the cool things about causality [23:35] is explanability. So you don't need to always  be accurate with your answer, especially asking [23:42] a speculative question. But what you need to be  able to quantify is how confident you are in the [23:47] answer that you're giving the user so they can  use their own judgment and decide whether to use [23:52] it or not. And I think it's one thing often where  with a lot of the AI that gets released today, we [23:58] look at headline accuracy metrics. We do not look  at quantifying confidence on how the things that [24:05] come out get used. And then the second part which  I think uh all of my fellow panel mates touched on [24:11] is um really like it boils down to really I think  we've heard this word a lot but it's a semantic [24:17] layer. How do we know what we're talking about  is the same thing as what someone else is talking [24:21] about. Uh and that is also one way where we can  start asking the right questions because if you're [24:27] talking about one thing when you're talking about  category and then your colleagues talking about [24:32] category in a completely different perspective  coming from a completely different system then [24:36] you'll end up with mismatching answers. So, one of  the ways that we developed our software solution [24:44] is by building all our causal models that we  build that tell you why something is happening [24:50] from a single source of truth, an ontology  that's purpose-built to contain business [24:56] logic and information about that specific the the  the specific business context that your business [25:03] uses. So, how does your business calculate churn?  How does your business calculate what an active [25:07] user is? And through doing that, you can get  repeatable deterministic answers to questions when [25:14] you mention specific things, but also generally it  helps with when you're trying to model speculative [25:20] questions downstream using causal inference. I  could go on but I feel like I I don't want to [25:26] take up too long. I just want to yeah maybe I just  want to you know there respond that maybe what we [25:31] are talking essentially is we always tend to look  at things in a very monolithic way like dashboards [25:38] is it a pretty monolithic one that it's dashboards  and solve every problem AI is it monolithic that [25:44] solves every problem I think what we've been  making a point is essentially the use case and the [25:49] problem really matters and you can have dashboards  and how many of you folks had more than you know [25:55] 100 dashboards for the same thing. I'm sure you  know a few of you can attest or maybe more of you. [26:00] We build the wrong things like you know you come  to BI tool and you have dashboards and you build [26:05] it. That's not the end goal. You have AI agents  and now you have hundreds of agents to do what? [26:09] I think that's what I want to ensure that you  talk about use cases and problems and have the [26:14] right set of uh pains whether it's dashboards,  whether it's AI. I think that's where I think [26:18] uh you know we need to you know focus on and  accuracy should follow the use case. Some use [26:22] cases don't need accuracy, some use cases do and  that's where I think uh I want to focus on ju just [26:28] to add to what Gam is saying completely agree  and when you try and build let's say a context [26:34] layer or an ontology you know you see these words  getting thrown around you cannot boil the ocean [26:38] with it otherwise you're going to spend your  entire life trying to map out every little part [26:42] of your business and then you start mapping out  completely irrelevant things that no one's going [26:46] to use and you have to maintain or they start to  skew the answers that you're trying to actually [26:52] So again, when you know, let's say we're in in  the root cause platform, just to give you an [26:57] example of a problemdriven ontology or semantic  layer, it's what things do we need to be able to [27:04] accurately model a scenario and start to speculate  whatifs about that scenario. What are the realm of [27:11] possibilities that the business can realistically  operate in? And these are the things we build the [27:15] ontology around min minimums and maximums  and different categories that can exist in [27:20] a specific column and everything that goes around  that and business rules about how one column can [27:25] only can a value in one column can can restrict  the values in another column and all of these [27:31] things are extremely important. So that's what we  decide to model inside our ontology that goes into [27:36] uh the that essentially we derive causal models  from. So it it's completely along the same lines [27:41] as you have to be very problem driven with the way  you're building these things otherwise you end up [27:46] boiling the ocean and nothing gets achieved  at all. I I think autoi actually making a big [27:53] you know point and for that making decision. So  we're going to start from that one. Deterministic [28:00] was another another big challenge with with  systems that you mentioned. Yeah. Oh yeah, [28:05] absolutely. And I I think I think deterministic  can again it can there's there's a few points of [28:12] failure where determinism can fail. And the first  one is the semantic side of things talking about [28:17] the same thing first of all. But the second  part is the systems that are used to retrieve [28:23] deterministic answers. Right? If you're I mean we  can take a metric, you know how to calculate that [28:29] metric in your head. Does the language model that  you're talking to know? And that's what that's the [28:34] kind of stuff you should be considering putting in  a semantic layer. When we're talking about causal [28:39] models, for example, causal models are a way to  get deterministic speculative answers because you [28:45] ask each LLM a question and they will speculate  in a completely random way. They they they cuz [28:51] they don't have a specific mechanism or algorithm  by which to speculate. They'll either write some [28:56] code, they'll bring in some some of them will  bring in external information uh from one from [29:01] an exogenous source, some what others won't. So  you need to be really careful with a deterministic [29:08] workflow for achieving certain answers whether  it's a simple BI query of metrics or a chart or [29:14] a dashboard or asking completely speculative  whatif questions and scenario planning. And [29:19] there are ways to achieve determinism in all of  these things. Even if it's probabilistic within [29:25] a bound of error, it's still okay, especially  for speculative questions. But there's 100% a [29:30] way to achieve determinism when we're talking  about retrieving metrics, simple metrics from [29:35] a source. Um, let me just um love you, let's go  back to you. So from a from a bigger perspective, [29:44] um what did you come up against in terms of  workflow and change management and impacts? It was [29:52] like how did you make sure you just didn't keep  producing something useless faster and easier? [29:59] This is related to the um part that are we  using AI just for the sake of AI or are we [30:08] adding productivity and what's the ROI all these  are related now what's the ROI on building certain [30:16] platform or certain application for what use case  so we are all talking about accuracy building [30:26] trust and all that so for us let's Say we in our  world we always want to make sure that our clients [30:33] they get what they were promised. So what they  were promised we call it binding and what they [30:39] get we call it final policy. We want to make sure  that binder matches exactly with their policy. [30:46] So if binder says that hey client you have to  pay me $100 for this coverage and policy says [30:53] $15. So accuracy matters here for us because we  want to make sure that $100 matches exactly with [31:01] $100 there in the final prom is final issued  policy. So accuracy matters here but the rest [31:09] of the components that uh which um coverage is  um provided for how much limit and all that. [31:19] So there we can manage without accuracy. But as  long as this is about telling our clients that [31:26] your premium has increased, we do want to make  sure that we give them proper correct and well [31:33] verified answer. So ROI here is that uh we build  the client trust by not giving them wrong answers [31:42] but uh trustworthy answers, accurate answers. On  top of that uh we are helping our teams to be work [31:50] fast. So rather than they digging deep into each  and every coverage and each and every premium line [31:56] now they can work fast and then they can just  finally prepare for the client summary client [32:02] report and all that. So we make them fast. So we  are trying to change the workflows those static [32:09] the regular business procedures with help of AI  make them more AI native so that they could they [32:17] could be fast because AIS we all know that AI  can go dig deep into all the information that [32:24] we ground these AIS too and they can provide best  answers not best answers sorry fast answers and [32:31] human then use those recommendations and make the  final decision and yeah how about you Mar has has [32:39] anything changed and then with with with like I  said workflow or change management or new roles [32:45] or new functions or anything yeah of course  there is no doubt that AI will provide higher [32:50] efficiency better workforce however there is a  lack of resources to build the AI models as well [32:58] so you have to be careful in terms of selecting  the your use cases so I remember when I joined I [33:05] sat with different business leaders to hear their  pain point to see what is the best use cases with [33:12] the higher impact and the higher pain that they  are having uh within the organization. So you can [33:18] prioritize your AI model based on the ROI. The  ROI could be the efficiency, could be the risk, [33:25] could be revenue generation or could be the loss.  So all of those matters in terms of selecting the [33:31] AI use cases. when you select that one you know  that you need the data you need the governance [33:37] and all of this will go under one pillar so  you will have governance and culture data [33:42] solution AI and analytics that's how I envision  it end to end so one without each other without [33:49] the other doesn't work so you will see it end  to end together from the words of you know the [33:55] decision intelligence one thing that I see as  a new role in the future is design architect [34:02] So what I had before in my team, it was a very  good PowerBI um you know person who was really [34:11] expert in PowerBI. However, he did not know the  semantic data modeling sorry and so on so forth. [34:19] So what benefit him was knowing all of those  data and learning all of those semantic layering [34:26] optimizing the data and architecture and the  design for a better reporting or better decision [34:33] out of the report. So what I see these days for  the decision intelligence I'm not talking about [34:38] the AI engineer or data engineer I'm talking about  the powerbi or the you know the person to provide [34:45] the analytics or reporting or the solutioning of  the you know decision intelligence. So I do see [34:51] the design architecture to provide the end to end  from the data to the answer and the solutioning [34:57] would be a new role for the future. Um, let me put  a little twist on it and then we'll we'll conclude [35:04] with Cool. Um, you mind if I stand up? There's  audience there I couldn't see and I thought it [35:09] would be better to just address. I would say like  it's is it okay? Yeah, go ahead. Well, I could ask [35:14] you a more specific question, right? Is there  an example of a decision that has changed or is [35:22] different or is better informed because of what  you built? Uh, it's a bit of a counterintuitive [35:29] one. Mhm. The decision was not in the utility that  the conversational analytics is providing. The [35:35] decisions were in the lack of uh you know systems  we have or organization we have. AI is a great [35:42] interface that is a mirror that shows how good  or how ugly you are and you know the organization [35:48] context is really bad or projects such as uh you  know semantic layer obviously but also in terms [35:53] of like the uh you know uh project management  good documentation. There was an instance where [35:59] um you know we asked a data agent for something  and it didn't have the information but when we [36:03] asked claude it had web search so it searched on  the web and gave us the answer that we should have [36:08] had internally itself and that changed something  which is basically having some good processes [36:12] in the organization and ensuring that we are  building these u you know age-old you know project [36:18] management and good tooling uh that we wanted to  build in the organization that was one one thing [36:23] u that we found and just just one one address uh  one thing that I want to address in the previous [36:28] question is um I think I believe in the thesis  that uh how many of you are not working on AI [36:36] right now? Everybody is. How many of you are  working on internet? All of you are. I think [36:46] 20 years ago maybe uh people would have been  like fine with having internet in their title [36:50] or digital in their title. Now nobody has because  everybody is working on that. I think two three [36:56] years or maybe five years from now we will  all cringe if we say like I'm an AI PM or an [37:00] AI something and I do think that it goes back to  the fundamentals of you know the problem and what [37:06] we pick map out your business processes map out  where the bottlenecks are and what problems to [37:11] solve and for us we had challenges in terms of  how we you know process claims on some of our [37:16] you know marketing uh claims work or uh even in  the marketing processes that we do and we enabled [37:23] uh you know uh we removed or reduced of friction  on some of these uh areas. U that's that's the [37:28] one and the last one I want to say is uh there  are two concentric circles in my opinion. One is [37:33] the actual thing you do and then the second one is  enabled by AI and a lot of our work is enabled by [37:39] AI but doesn't mean that that is the AI that we're  going after like our dashboards are now built by [37:45] AI. uh whatever we build in cloud dashboards we  take it back into deterministic dashboards that [37:49] are enabled by AI and that is I think the journey  that we are going towards uh in general yeah like [37:55] almost moving more towards decision science  and like what is decision you make correct [37:59] um when you when you take your approach and work  through this um how much do you consider you know [38:06] elements like change management and and elements  like workflow like how do you account for that [38:10] working with different organizations yeah that's  actually a really interesting question and I think [38:15] It's a huge part of being a software solution  that does analytics and decision support because [38:22] there's it's one thing bringing the horse to  water, but can you make the water can you make the [38:27] horse drink water? Um, and the answer is no, not  without a management consulting firm or without [38:32] credibility in in some way with the seauite. So  ultimately yes it's a it's a major major part of [38:40] how we operate as a business how I used to operate  when I used to do boutique consulting for AI uh [38:47] driven sol AIdriven decision making in different  blue chip enterprises so of course of course you [38:54] have to make sure that people trust the outputs  that come out and are willing to act on them [38:59] um and it's a huge huge challenge as a startup  that that that didn't exist a few years ago to [39:05] make people trust technology that you're selling  to them. Um, I want to be a conscious time. So, [39:11] everybody gets one minute for this one. So, this  isn't a detailed question. I'll phrase it this [39:15] way. Um, in terms of ROI, do you feel like you  have a handle on how you're going to measure ROI? [39:24] Like not necessarily how, but like how confident  do you feel you can measure ROI? My answer to that [39:30] is u the venture capital world has a very good  model. you'll have a lot of bets and maybe you [39:37] know 2% 3% will pan out and they'll be the most  successful ones and a lot of these other things [39:43] will be reps uh for to do the next big thing and  that's how I'm going to measure ROI and even if I [39:48] reflect in my last 6 months one year of journey  we have two or three successful projects that [39:53] show real dollar value and we have about maybe 50  to you know I don't know maybe in in our land of [39:59] you know a lot of vibing apps that people produce  lots going on um and I don't know if I'll be able [40:05] to get true ROI there. So it would be essential  to understand true ROI the middle tier and the [40:10] longtail probably don't worry about it but have  necessary guard rails. How about you Mark? The [40:15] good thing about our company or actual actually  when we are selecting the use case is that I'm [40:20] forcing everybody to start from ROI. So I ask  people to come up with the ROI before us building [40:28] the AI agent or the AI model that we have. either  it could be saving the FD that they want to hire [40:35] or the vendor they want to you know um go for it  so we are saving that money so that consider as a [40:43] ROI for us when we are selecting the use case and  is that consistent application across all of them [40:49] so is it is it mostly a financial ROI or do you  look like reputational or employee we do have a [40:55] yes we do have um um employee we do have employee  experience or the member experience as well as the [41:02] reputation. So sometimes this is risk related. So  there is no choice. You have to go for it. This [41:07] is regulation. We are highly regulated company.  So we have to go for it. Those are must have. [41:13] But in addition to those that you are actually  bringing more accuracy, more efficiency. There [41:20] are some use cases that you are selecting because  you are gaining money either not losing money or [41:26] adding the revenue to that. So for those cases  I always always start with the ROI rather than [41:33] you know calculating afterward I would just say  like without financial ROI everything else is an [41:38] opinion or a debate and it's hard at least at  this stages. So I would really encourage to go [41:43] for financial ROI as much as possible right now  in this maturity we'll take that up [laughter] we [41:50] go for uh soft ones these days. So we started  with adoption. We all know that it was easy to [41:57] calculate adoption rate. But then productivity is  the final one these days. But within productivity [42:04] it depends on uh how we measure that. So did we  help our business users to serve 10 clients within [42:13] uh a day or did we help them do their policy  review and policy comparison fast? Are they [42:20] serving? are they satisfying their clients with  their uh quarterly reports or annual reports and [42:26] all that. So we have also added work for our  internal operation teams by giving them tools [42:33] that here you go you have a policy check that  you can do now here you go you have a quarterly [42:40] business review report annual client letters you  can produce all of those earlier they were doing [42:46] only for top elite clients now they can do for  midsize they can do for the small clients too [42:52] so it depends what we want to measure but we are  saying that are we adding productivity to their [42:59] bucket. So productivity matters. Are you serving  more? Are you doing more but more again matters [43:04] not the number of tasks but are those are you  spending less time and serving making your clients [43:11] satisfied. Then you you have again a bigger  challenge. So how do you how do you communicate [43:16] across the spectrum of organization? Oh everything  has to be converted to a dollar value. Uh [43:22] unfortunately that's just the way it's got to  be. uh otherwise no one's going to buy software [43:26] especially from outside their organization they  all think they can build it. Um so with causality [43:33] can break it down into first order and second  order ROI. The first thing is when you're trying [43:37] to actually understand why something's happening  it takes a long time to do that because you have [43:42] to go and search the organization you have to  interview people. You have to go pull data from [43:45] different places. So being able to do it through  causal discovery and getting causal drivers, [43:52] indirect and direct drivers of something, that's a  time saving. So you can see the first order effect [43:57] of having that instead of having to do it and and  then there's also the second order impact of well [44:02] what because we did this quicker, there's a bit  of an opportunity cost element where what if we [44:08] knew this later, how would that have impacted our  business? And often it's it's too late to make a [44:13] difference if you knew it later. So there's  there's also a very it's a very gray area, [44:18] but there's a calculation you can do on how  how much more money you can make by knowing [44:23] things earlier that rather than later, and it's  being able to be more adaptive. Uh but yeah, [44:29] cool. Thank you. We're out of time. Thank  you for joining us. We appreciate your time.