Sanofi and CVS Deploy Autonomous AI Agents to Cut Enterprise Data Costs

赛诺菲与CVS部署自主AI智能体以削减企业数据成本 企业级AI智能体的成功部署依赖于扎实的数据基础、多智能体编排与严格的Token成本管理，而非直接使用现成的大语言模型。 行业领袖详细介绍了企业级AI智能体的实际落地案例，包括赛诺菲用于商业数据分析的10至12个智能体网络，以及CVS Health用于模拟40万合成患者行为的“Agentic Twins”模拟器。专家小组成员指出了主要的部署障碍，强调企业的成功取决于坚实的数据基础、可解释性以及主动的变革管理，而非单纯依赖通用大模型。为控制成本并确保可靠性，发言人提倡采用模型中立的动态编排、按任务复杂度分层，并使Token消耗与核心业务指标严格对齐。最后，他们强调必须建立单一事实来源的数据架构和多LLM评估框架，以满足医疗等行业严苛的监管与可审计性要求。 赛诺菲利用由具备访问控制的编排层所协调的10至12个专业领域智能体网络，为商业团队提供自助式数据分析。 (3:28) CVS Health部署了包含40万个合成患者智能体的“Agentic Twins”，在实际推广前通过虚拟“飞行模拟器”模拟患者行为并测试医疗干预机制。 (6:58) 如果指望通用大语言模型在没有与统一的企业知识库和SOP集成的情况下开箱即用地解决复杂工作流，企业AI部署往往会以失败告终。 (9:34) 成功规模化扩展企业智能体取决于三大关键支柱：面向AI就绪的数据基础、透明且可解释的洞察，以及持续的变革管理。 (14:13) 可观测性、可解释性、透明度与信任度必须在AI架构设计初期就前置考量，在医疗等受严格监管的行业中尤为如此。 (19:03)

Deploying enterprise AI agents successfully requires shifting from generic large language models to multi-agent architectures grounded in curated data, strict cost orchestration, and transparent governance. Panelists from Sanofi, CVS Health, NICE, and ThinkingAI argue that business value depends on domain-specific execution and operational trust rather than unconstrained model capabilities. Specialized Multi-Agent Implementations in Production (3:28) Rather than relying on single generalist models, enterprise leaders are deploying specialized agent networks tailored to specific operational functions. Nitesh Soni describes Sanofi's conversational platform, which uses an orchestration layer and 10 to 12 domain-specific agents to translate commercial user queries into synthesized recommendations and next-best actions. This setup replaces manual multi-dashboard queries previously handled by data analysts. Synthetic Simulation and Pre-Deployment Testing (6:58) In heavily regulated sectors like healthcare, virtual agents allow organizations to model behavior before physical deployment. Anuj Maheshwari explains that CVS Health created 400,000 synthetic 'Agentic Twins' trained on simulator data to serve as a healthcare flight simulator. This platform enables the company to evaluate how patient populations might respond to intervention messages and therapy programs, addressing prescription abandonment while safeguarding patient privacy. Architecting Structured Data Foundations (32:36) Panelists caution that high pilot failure rates—cited as reaching up to 95% across the enterprise sector—stem directly from unvetted and fragmented data environments. Kevin Lee points out that retrieval-augmented generation falters when querying conflicting content systems. To solve this, Soni outlines Sanofi's three-tier data model: a source-aligned layer for raw feeds, a process-aligned layer standardizing enterprise schemas, and a consumption-aligned layer feeding domain-specific datasets to models. Balancing Compute Economics and Model Neutrality (21:25) Enterprise agent scale exposes firms to significant compute spend, notably 'token maxing' caused by inefficient prompts. Kevin Lee and Chris Han stress mapping task complexity to model capabilities: routine deterministic inquiries can run on conversational AI or lightweight models, reserving costly frontier models for complex reasoning. Maheshwari advocates a 'buy, rent, build' framework, purchasing base platforms, renting niche capabilities, and building proprietary business logic. Governance, Observability, and Speed as Core Metrics (22:42) Operational success is increasingly defined by reduced cycle times alongside transparent auditability. Maheshwari reports that agentic pipelines reduce analytical reporting tasks from a one-week, six-person manual workflow down to 20 minutes. To maintain regulatory compliance and user trust, Sanofi implements telemetry frameworks like TruLens and a multi-LLM setup where separate models inspect input data quality, generate insights, and audit output accuracy against source facts. While agentic systems deliver dramatic reductions in time to insight, long-term organizational viability remains bound to clear governance, explainability, and disciplined token economics to prevent unsustainable computational costs.

企业在推进AI智能体落地时，正从单纯的技术概念验证转向注重实际投资回报与运营效能的规模化部署。来自赛诺菲、CVS Health、NICE与ThinkingAI的技术管理者指出，构建三层数据基座、推行轻重模型动态编排、并建立全程可追溯的审计体系，是克服高达95%试点失败率的关键路径。 协同交互与孪生模拟赋能业务 (3:28) 赛诺菲（Sanofi）与CVS Health展示了智能体在具体业务中的创新落地形态。赛诺菲数据科学主管Nitesh Soni介绍，其商业化团队部署了基于对话的“与数据对话”（Talk to Data）平台。该平台通过主编排代理评估用户意图并执行权限控制，协同调度10至12个专业垂直智能体提取并综合数据，不仅免除业务人员依赖分析师手动跨看板处理数据的繁琐流程，还能主动提供下一步行动建议。 在医疗干预方面，CVS Health数据与分析策略副总裁Anuj Maheshwari分享了“智能体孪生”（Agentic Twins）的应用。为解决患者未能按方抓药或中断治疗的问题，该团队利用真实世界模拟数据构建了40万个合成患者智能体。这一机制充当了医疗项目的“飞行模拟器”，允许运营团队在门诊实际推行前，在兼顾隐私的环境中虚拟测试不同的沟通策略与干预效果，从而降低一线落地时的操作摩擦与潜在风险。 三层数据架构突破落地瓶颈 (32:36) 针对大量企业AI项目停滞在试点阶段的现象，与会者指出低质量和碎片化的底层数据是核心症结。NICE的Kevin Lee强调，企业普遍缺乏经过严格维护的单一权威知识库，直接通过检索增强生成（RAG）或模型上下文协议（MCP）让智能体查询分散在Drupal或AEM等平台中的文档，极易引发内容冲突和陈旧信息误导。 赛诺菲为此搭建了三层结构的数据基座：底层为接入IQVIA、CRM与销售记录等内外部原始信息的“源对齐层”（Source-Aligned Layer）；中层为统一企业业务规范并建立数据目录的“流程对齐层”（Process-Aligned Layer）；顶层则是面向AI模型特征库和数据分析准备的“消费对齐层”（Consumption-Aligned Layer）。Nitesh Soni表示，该体系形成了“利用AI治理数据，再以优质数据驱动AI”的协同循环，配合透明的可解释性机制，才能真正赢得终端业务人员的采纳与信任。 动态编排与分级管控算力成本 (21:25) 随着智能体进入常态化运行，频繁交互引发的“Token消耗过量”（Token Maxing）成为吞噬项目预算的现实隐患。Kevin Lee指出，NICE平台采取模型中立策略，通过实验室持续对比各类模型在具体企业任务中的认知表现，并利用动态编排引擎分流：将简单的确定性查询交由规则或小语言模型（SLM）处理，仅将高复杂度和意图模糊的任务路由至前沿大模型。ThinkingAI联合创始人Chris Han同样强调任务分级的重要性，建议企业将Claude Opus或GPT-4等重型推理模型严格限制在核心战略分析中，避免将算力浪费在华而不实的视觉交互上。 在预算与架构管理上，Anuj Maheshwari提出了“采购、租赁、自研”（Buy, Rent, Build）的组合策略：标准化的基础平台直接外购，专业垂直能力与小众大语言模型按需租赁，而体现企业核心差异化价值的专有工作流则坚持自研，以此在控制开发成本的同时保持长期的技术敏捷性。 缩短洞察周期与多模型审计 (22:42) 企业对智能体价值的度量正在从传统客户体验指标（如满意度CSAT和问题拦截率），延展至业务周期的绝对压缩。Anuj Maheshwari提出“获得洞察所需时间”（Time to Insight）是核心衡量标准，例如原本需要6名数据分析师耗时整整一周清洗和呈现的数据分析任务，智能体系统可在20分钟内以同等质量标准执行完毕。Nitesh Soni补充指出，在制药领域，缩短从药物研发到患者临床使用的时间是最高杠杆的KPI，研发与审批流程中节约的每一小时都直接关联到拯救患者生命的规模效应。 在强监管与医疗合规约束下，消除黑盒决策是智能体系统上线的前提条件。赛诺菲在架构设计中引入了多LLM制衡机制：分别设立专职的数据质量LLM把关输入、洞察生成LLM负责综合推演、以及准确性与审计LLM对照原始事实进行输出校验。配合TruLens遥测框架与用户反馈闭环，确保系统在面临监管审查时拥有端到端的推理可追溯性。 本场研讨展现了企业级AI智能体在缩短业务运转周期和辅助决策层面的实质价值，但其全面普及仍受制于模型计算成本控制、权威知识沉淀以及强监管下的合规审计能力。如何在释放智能体自主执行效率的同时，建立可持续的成本监控并保留必要的人机协同（Human-in-the-loop）监督，是企业迈向规模化落地时需持续权衡的关键课题。

## Background 本内容由 Ai4 于 2026 年 8 月 22 日发布在 YouTube 平台上。视频记录了一场由 VKTR 编辑总监 Michelle Hawley 主持的研讨会，邀请了来自赛诺菲（Sanofi）、CVS Health、ThinkingAI 及 NICE Systems 等领先企业的行业高管与技术专家，共同深入探讨 AI 智能体（AI Agents）在各行业落地并创造实际业务价值的成功案例与经验。 ## TL;DR 研讨会围绕 AI 智能体在跨国医药、医疗健康、游戏及客户体验领域的落地实践展开。赛诺菲与 CVS Health 等企业分享了通过多智能体协作平台及 40 万规模的“智能体分身”数字模拟器提升运营效率与患者体验的真实案例。专家们深入剖析了企业部署智能体时面临的数据基座、信任构建及变更管理等核心痛点，并针对 Token 成本控制、多模型分层架构、数据治理及可解释性等关键议题给出了具体的破解策略。 ## Atmosphere & style 整场研讨会呈现出极为专业、务实且高密度的行业对话氛围。几位跨国企业高管与技术领袖以坦诚且兼具前瞻性的视角，打破了对 AI 的盲目吹捧，直击项目落地中的痛点与挑战。现场探讨节奏紧凑、逻辑严密，在理论框架与具体业务数据之间切换自如，给听众带来沉浸式的行业前沿洞察体验。 ## Key takeaways - [00:00](https://www.youtube.com/watch?v=NnYCjJmovDk&t=0s) 研讨会由 VKTR 主持，汇聚来自赛诺菲（Sanofi）、CVS Health、ThinkingAI 及 NICE Systems 的多位高管，共同探讨跨行业 AI Agent 的落地与实践经验。 - [03:00](https://www.youtube.com/watch?v=NnYCjJmovDk&t=180s) 赛诺菲（Sanofi）打造了基于 10-12 个多智能体协同的“Talk to Data”数据对话平台，实现了企业级数据洞察的自助化与民主化。 - [06:00](https://www.youtube.com/watch?v=NnYCjJmovDk&t=360s) CVS Health 构建了包含 40 万个智能体分身（Agentic Twins）的数字飞行模拟器，用于低成本模拟并优化复杂的患者用药与履约干预策略。 - [09:00](https://www.youtube.com/watch?v=NnYCjJmovDk&t=540s) 从单体 Agent 走向企业级应用需要克服上下文统一、知识库对齐以及将人类技能转化为智能体能力的巨大鸿沟。 - [14:00](https://www.youtube.com/watch?v=NnYCjJmovDk&t=840s) 面对 95% 的 AI 试点项目难以迈向生产环境的挑战，核心痛点集中在“AI-Ready”数据基座、洞察的可解释性/信任以及业务端的变更管理。 - [19:03](https://www.youtube.com/watch?v=NnYCjJmovDk&t=1143s) 高监管行业中需将可观测性与可解释性融入底层架构；同时，评估 Agent 成功落地的 KPI 已从 CSAT 扩展至 Token 消耗控制、缩短洞察时间（Time to Insight）及上市/疗法送达时间。 - [26:15](https://www.youtube.com/watch?v=NnYCjJmovDk&t=1575s) 通过个人 AI Agent 预先提取员工动态与潜在挑战，可以重塑内部会议流程，省去冗长汇报并聚焦于核心瓶颈解决。 - [27:42](https://www.youtube.com/watch?v=NnYCjJmovDk&t=1662s) 为应对 Token 成本暴涨，企业应采取“买、租、自建”结合的策略，并建立任务复杂度分级，避免一律调用顶级大模型。 - [32:31](https://www.youtube.com/watch?v=NnYCjJmovDk&t=1951s) 准备数据基础设施需要确立“单一事实之源”，构建从源头对齐、流程对齐到消费对齐的三层数据架构，并形成“Data for AI / AI for Data”的双向闭环。 - [36:13](https://www.youtube.com/watch?v=NnYCjJmovDk&t=2173s) 确保 Agent 可追溯与可解释需要将其作为设计原则，并可通过多模型架构（如质量评估、洞察生成与裁判校验模型）进行交叉验证。 ## Detailed walkthrough ### [00:00](https://www.youtube.com/watch?v=NnYCjJmovDk&t=0s) 开场致词与嘉宾介绍 研讨会由 VKTR 编辑总监 Michelle Hawley 主持。在简短的开场致谢后，几位来自跨国医药、AI 分析、医疗保健及客户体验领域的重磅行业高管依次进行了自我介绍。 * **Nitesh Soni**：赛诺菲（Sanofi）全球数据科学负责人（数字商业）。赛诺菲总部位于法国，是全球领先的跨国制药巨头。他领导着一支 AI 专家团队，致力于构建规模化 AI 系统，重塑赛诺菲的商业运营模式，从而为全球医疗从业者及患者提供更优的医疗成效。 * **Chris Han**：ThinkingAI 联合创始人兼总裁。ThinkingAI 是一个智能体分析（Agentic Analytics）与实时运营（Live Ops）平台，主要服务于游戏等高速迭代的行业，旨在帮助企业打通既有数据资产并转化为业务增长。 * **Anuj Maheshwari**：CVS Health 医药数据与分析策略副总裁。CVS Health 是一家具备海量一手数据的多元化医疗健康企业。拥有 20 年医疗分析经验的 Anuj，致力于寻找推进医疗保健分析现代化的可行路径。 * **Kevin Lee**：NICE Systems 现场 CTO 兼关键项目负责人（Key Pursuits Leader）。NICE 是全球客户体验（CX）与联络中心领域的领军企业，Kevin 专注于站在产品、技术与客户合作的交汇点上，协助企业从新技术中提取真正的“业务成果”。 ``` [研讨会嘉宾阵容] ├── Michelle Hawley (主持人) - VKTR 编辑总监 ├── Nitesh Soni - 赛诺菲 (Sanofi) 数据科学全球负责人 ├── Chris Han - ThinkingAI 联合创始人兼总裁 ├── Anuj Maheshwari - CVS Health 数据与分析策略副总裁 └── Kevin Lee - NICE Systems 现场 CTO ``` --- ### [00:03](https://www.youtube.com/watch?v=NnYCjJmovDk&t=3s) 赛诺菲：基于 10-12 个智能体的“Talk to Data”数据对话平台 赛诺菲全球数据科学负责人 Nitesh Soni 分享了他们成功落地的智能体项目——为商业团队打造的“Talk to Data”（与数据对话）平台。传统的分析流程非常繁琐：品牌经理、销售代表或市场准入团队如果想了解产品在特定区域的业绩、潜在增长点或资金人力调度方案，必须向数据分析师提出需求。分析师需要耗费大量时间跨多个仪表板切片数据、提炼 AI 方案并反复沟通，才能给出解答。 为改变这一现状，赛诺菲构建了基于多智能体协作的对话式平台。用户只需以自然语言提问，底层便会有“智能体军团”协同运作：首先由通用编排智能体识别用户意图并分发请求，同时进行数据权限与安全控制；随后，任务专属智能体协同提炼信息并返回答案。该平台不仅能回答提问，还能智能引导后续追问并提供最佳下一步行动（Next Best Actions）建议，成功实现了企业级数据洞察的自助化与民主化。 * **系统架构**：部署了 10 至 12 个智能体，分为负责意图识别与权限管控的通用智能体（Generic Agents），以及针对特定业务的数据/计算任务智能体（Task-Specific Agents）。 * **应用群体**：覆盖品牌经理（Brand Managers）、现场销售代表（Field Reps）及市场准入团队（Market Access Team）。 * **核心价值**：实现企业级 Self-Serving Analytics（自助式分析），大幅缩短从提问到决策的周期。 > "Behind the scene we have an army of agents where one agent is basically trying to understand the intent and route request... bring back to the end users." — Nitesh Soni --- ### [00:06](https://www.youtube.com/watch?v=NnYCjJmovDk&t=6s) CVS Health：“智能体分身”构建 40 万规模的虚拟飞行模拟器 CVS Health 医药数据与分析策略副总裁 Anuj Maheshwari 详细阐述了他们推出的“智能体分身”（Agentic Twins）项目。虽然 CVS Health 并非原生 AI 公司，但作为以消费者为中心的企业，他们致力于利用海量一手数据建立竞争优势。针对医疗行业中“数以百万计患者未按时取药或未坚持医嘱疗程”的痛点，CVS Health 没有直接在实体零售店盲目上线新项目，而是构建了一个高保真、保护隐私的“虚拟飞行模拟器”。 基于真实世界模拟数据，CVS Health 训练了高达 **40 万个**不同类型的智能体分身。这些智能体能够模拟真实患者在面对医疗选择时的心理与行为反应。通过在数字空间中提前测试各种干预方案与沟通策略，团队可以不断迭代优化，避免将未经验证或体验不佳的方案直接推向终端消费者，从而实现了业务成效的指数级提升。 ``` [真实世界患者数据] ──> [隐私保护/模型训练] ──> [40万个 Agentic Twins] │ ▼ [业务决策/干预方案] ──> [数字飞行模拟器测试] ──> [优化后推向实体零售店] ``` * **训练规模**：构建并运行了 400,000 种不同类型的 AI 智能体分身。 * **解决痛点**：应对患者中断治疗、未按处方履约（Prescription Non-Adherence）等复杂行为决策。 * **核心逻辑**：以低成本的数字模拟替代高风险的线下试点，提前验证并优化用户体验。 > "We actually created 400,000 types of agents trained on real world simulator data that creates a bit of a flight simulator for us." — Anuj Maheshwari --- ### [00:09](https://www.youtube.com/watch?v=NnYCjJmovDk&t=9s) 游戏与客户体验领域的智能体应用与对齐挑战 ThinkingAI 联合创始人 Chris Han 与 NICE Systems 现场 CTO Kevin Lee 带来了技术落地与客户体验视角补充。Chris 提到，在智能体时代，虽然每个人都能轻松创建单体 Agent，但大语言模型（LLM）与企业级应用之间仍存在巨大鸿沟。企业必须将人类积累的技能集（Skill Sets）转化为智能体的能力，并确保多智能体在统一的上下文环境（Context）和知识库下协同运作。尤其在游戏等超高速迭代行业，每日可能产生上千个 A/B 测试与营销活动，如何让智能体自动从无数仪表板中提取真正能驱动业务增长的洞察，是当前最大的技术瓶颈。 Kevin Lee 则从客户体验（CX）视角警示：技术人员或企业管理者想解决的问题，往往与终端客户真正需要解决的问题存在错位。许多企业盲目过度套用生成式 AI（Generative AI），试图用通用 LLM 解决一切，却忽视了企业内部标准作业程序（SOP）的特殊性。Kevin 强调，企业沉淀的历史交互数据中隐藏着真正的答案，唯有深度挖掘数据并将其与实际的 ROI、KPI 挂钩，才能避免 AI 部署沦为花架子。 * **企业级智能体鸿沟**：缺乏统一上下文、知识库对齐不足、自主工作流（Autonomous Flow）与人类团队协作机制不完善。 * **数据利用误区**：企业往往坐拥大量 Dashboards 却缺乏实质洞察；过度依赖通用 LLM 忽略了历史交互数据对 SOP 的指导意义。 > "There is still such a big gap between the LLM and enterprise level agents... to align your agents can work together." — Chris Han --- ### [00:14](https://www.youtube.com/watch?v=NnYCjJmovDk&t=14s) 智能体落地的三大核心痛点：数据基座、信任与变更管理 针对 MIT 研究指出的“95% 的 AI 试点项目难以走向生产环境（Production）”这道坎，嘉宾们深度剖析了智能化落地过程中的实际阻碍与破局之道。赛诺菲的 Nitesh Soni 归纳了三大核心挑战： 1. **数据基座（Data Foundation）**：如果数据没有做到“AI Ready”（缺乏高质量与良好治理），任何 Agent 都只是“建在沙滩上的城堡”，无法实现规模化（Scale）。尽管数据清洗与治理工作既枯燥也不光鲜，但它是智能体可持续运行的生命线。 2. **洞察的可解释性与信任（Trust & Transparency）**：AI 模型的准确率再高，如果无法向业务端用户清晰解释推导逻辑，用户就不会采纳。没有业务行动，AI 的组织价值即归零。 3. **变更管理（Change Management）**：这是最艰难的一环。破局的关键在于在项目极早期就邀请业务团队深度参与，构建持续的反馈闭环（Continuous Feedback Loop），并在业务端培养“AI 智能体倡导者”（AI Champions），让其从使用者变为推广者。 Anuj Maheshwari 补充道，在医疗等高监管行业，变更管理还必须兼顾“人类智慧（Human Intelligence）与 AI 增强”的平衡，在设计系统架构时就把合规、审查与信任建立考量进去。 ``` [企业 AI 智能体落地的三大支柱] ┌─────────────────────────────────────────┐ │ 1. AI-Ready 数据基座 (治理与质量) │ ├─────────────────────────────────────────┤ │ 2. 透明可解释性 (建立业务信任) │ ├─────────────────────────────────────────┤ │ 3. 早期参与+AI Champion (深度变更管理) │ └─────────────────────────────────────────┘ ``` * **关键数据**：行业研究表明高达 95% 的 AI 项目停滞在 Pilot（试点）阶段。 * **成功路径**：从第一天起就启动业务端沟通与教育，建立从概念验证（PoC）到深度信任的跨越 bridge。 > "If your data is not AI ready, you won't be able to go from small concept to scale... without that, you would be building your agents on sand." — Nitesh Soni ### [19:03](https://www.youtube.com/watch?v=NnYCjJmovDk&t=1143s) 可观测性、可解释性与企业级 KPI 衡量指标 在监管要求严格的行业中，AI Agent 的安全性、透明度与合规性是不可妥协的底线。CVS Health 的 Anuj Maheshwari 强调，在医疗健康等高监管领域，可观测性（Observability）、可解释性（Explainability）和建立患者信任必须从一开始就融入架构设计中。虽然这种前期的大量测试和安全信号监控可能会让人感觉推进速度稍慢，但它实际上是在为系统奠定坚实的基础。 在评估 Agent 是否成功落地时，各家企业采用的关键绩效指标（KPI）侧重点各有不同： - **客户体验与运营效益**：NiCE 的 Kevin Lee 指出，传统的客户满意度（CSAT）和拦截率（Containment rate）仍是核心指标。但与此同时，企业对交互经济性（Interactions Economics）的审查正在大幅增加，必须严密监控底层的 Token 消耗（Token Burn）与“Token Maxing”现象，防止成本失控。 - **缩短洞察时间（Time to Insight）**：Anuj 提到，一个关键的证明点在于大幅压缩数据处理和生成分析结果的时间。例如，过去需要半尽职（6人）的团队耗费整整一周时间才能整理完成的复杂数据分析与图表生成，现在在保证同等精度、安全性和归因标注的前提下，仅需 20 分钟即可完成。从终极目标来看，其核心 KPI 是能否真正改变市场和改善患者体验，降低患者在就医及用药过程中的摩擦。 - **临床与业务转化效率**：Sanofi 的 Nitesh Soni 补充道，在医药行业中，最核心的 KPI 始终是“缩短从分子发现到终端疗法送达患者手中”的时间（Time to Market/Therapy）。无论系统节省的是几小时还是几天，在庞大的患者基数下，这种时间的节省最终都能转化为挽救生命的成果。 - **用户留存与平台粘性**：ThinkingAI 的 Chris Han 强调，无论 Agent 架构如何演进，数据始终是底层基石。在衡量商业增长时，用户的留存率（Retention）以及产品/平台的粘性是不可替代的顶层指标。 --- ### [26:15](https://www.youtube.com/watch?v=NnYCjJmovDk&t=1575s) 利用 AI Agent 重塑企业内部管理与开会流程 Chris Han 分享了一个关于利用 Agent 改善企业内部沟通透明度与运营效率的具体案例。传统的公司周会或例会往往需要每个人花费 15 分钟轮流汇报最新进展，效率极其低下。为了改变这一模式，他们构建了高度透明的个人动态更新 Portal： ``` [员工个人更新网页 (HTTP)] │ (动态更新工作进展) ▼ [个人 AI Agent 预先读取] ──► [自动提取潜在问题与挑战] │ ▼ [高效率研讨会议] - 免去冗长口头汇报 - 聚焦核心瓶颈与争议 - 直接产出待办清单 (To-Dos) ``` 通过这一流程变革，参会人员无需在会议中口头陈述基础信息，而是将精力 100% 集中在解决关键挑战和拟定下一步行动计划（To-Dos）上，实现了组织管理与运营模式的真正升级。 --- ### [27:42](https://www.youtube.com/watch?v=NnYCjJmovDk&t=1662s) Token 成本控制与模型分层调用策略 随着新模型不断涌现、旧模型被淘汰以及使用量暴增，Token 成本管理成为了企业面临的巨大挑战。与会嘉宾从多个维度探讨了成本控制的落地策略： - **买、租、自建（Buy, Rent, Build）的动态组合**：Anuj Maheshwari 建议企业采用混合策略。对于通用的基础能力，可以直接购买商品化的平台（Buy）；对于具备特定专业技能或需要保持灵活性的模块，采用租赁形式（Rent）；而对于企业自身最具核心差异化竞争力的逻辑，则选择自建（Build）。 - **架构效率与多模型降级/分流**：Nitesh Soni 与 Chris Han 指出，绝不能将所有任务一股脑抛给最顶级的旗舰大模型（如 Claude Opus 等）。企业需要梳理日常任务清单与特殊项目，建立任务复杂度分级（Complexity Level）： - **简单/确定性任务**：如查询订单状态、提取会议纪要信息，可直接使用规则引擎、传统对话式 AI 或低成本的小型语言模型（SLM）。 - **复杂/多意图任务**：仅在面对复杂的多轮推理、跨线程对话或深层逻辑分析时，才调用高级的大语言模型（LLM）。 - **AI 读写素养与成本意识（AI Literacy）**：企业内部需要建立关于 AI 研发与运行成本的认知。不能为了追求表面花哨的功能（如消耗大量 Web Token 构建炫酷的地图）而偏离核心业务目标。每一笔 AI 投资都必须直接绑定到具体的业务目标（Business Goal）和 ROI 计算上。 --- ### [32:31](https://www.youtube.com/watch?v=NnYCjJmovDk&t=1951s) 数据基础设施准备与“Data for AI / AI for Data”双向闭环 在问答环节，针对现场观众提出的“如何准备数据基础设施（Data Foundations）”问题，专家们给出了系统性的解答： Kevin Lee 透露，绝大多数企事业单位在部署 Bot 或 Agent 时，最核心的痛点在于缺乏一个“单一事实之源”（Single Source of Truth）。企业往往将知识分散沉淀在 Drupal、AEM 等各种历史遗留系统中，甚至同一种问题存在 5 个不同的历史版本，企图单纯依靠 RAG 或 MCP 服务去连接这些杂乱的系统并让 LLM 自动判定谁对谁错是行不通的。正确做法是给业务团队（如产品经理、项目经理）直接开通权限，让他们在离源头最近的地方进行知识库（KMS）的维护与精细化治理。 Nitesh Soni 进一步阐述了大型企业常见的数据三层架构： 1. **源头对齐资产（Source-aligned Assets）**：整合内部销售数据与外部供应商数据（如 IQVIA 数据），统一汇集至底层。 2. **流程对齐资产（Process-aligned Assets）**：将数据与公司整体业务流程相匹配，进行严格的数据编目（Cataloging）与字段元数据定义，这是构建高可靠 AI 的关键关键层。 3. **消费对齐资产（Consumption-aligned Assets）**：面向特定业务领域打造可以直接被 AI 或 Dashboard 调用的数据衍生层。 同时，Nitesh 提出了一个重要的概念闭环：**"Data for AI, and AI for Data"** —— 我们不仅需要准备好可供 AI 调用的高质量数据（Data for AI），也在广泛利用 AI 自动化技术来清洗、标注和构建这些数据资产（AI for Data）。 --- ### [36:13](https://www.youtube.com/watch?v=NnYCjJmovDk&t=2173s) AI Agent 的可追溯性、可解释性与架构多模型校验 针对观众关于“如何确保 Agent 的行为可解释（Explainable）与可追溯（Traceable）”的提问，Panel 嘉宾总结道： - **将可追溯性作为底层设计原则**：Anuj 强调，可追溯性必须从第 0 天起就作为系统的核心设计原则（Design Principle），确保每一个决策步骤都能追溯到具体的底层依据，从源头上杜绝黑盒系统（Black boxes）的产生。 - **技术打标与多模型互查机制**：在实际落地中，除了收集用户的正负反馈（Thumbs up/down）外，企业还会引入 Trace 框架等技术追踪链条。Nitesh 补充道，在“与数据对话”（Talk to Data）等高级场景中，他们采用了**多模型架构（Multi-LLM Architecture）**进行分工与交叉校验： - 一个 LLM 专门负责评估数据质量（Data Quality LLM）； - 一个 LLM 负责生成业务洞察（Insight LLM）； - 另一个独立的 LLM 则专门充当裁判，评估并校验这些洞察的准确性（Accuracy Verification LLM）。 通过技术链路追踪与多模型协同纠错，确保了 Agent 输出结果的透明、精准与可信。 ## Notable quotes > [03:00](https://www.youtube.com/watch?v=NnYCjJmovDk&t=180s) Behind the scene we have an army of agents where one agent is basically trying to understand the intent and route request... bring back to the end users. — Nitesh Soni > [06:00](https://www.youtube.com/watch?v=NnYCjJmovDk&t=360s) We actually created 400,000 types of agents trained on real world simulator data that creates a bit of a flight simulator for us. — Anuj Maheshwari > [09:00](https://www.youtube.com/watch?v=NnYCjJmovDk&t=540s) There is still such a big gap between the LLM and enterprise level agents... to align your agents can work together. — Chris Han > [14:00](https://www.youtube.com/watch?v=NnYCjJmovDk&t=840s) If your data is not AI ready, you won't be able to go from small concept to scale... without that, you would be building your agents on sand. — Nitesh Soni ## Mentioned Michelle Hawley, VKTR, Nitesh Soni, 赛诺菲 (Sanofi), Chris Han, ThinkingAI, Anuj Maheshwari, CVS Health, Kevin Lee, NICE Systems, Drupal, AEM, IQVIA, Claude Opus ## Video description Michelle Hawley - Editorial Director, VKTR Nitesh Soni - Global Head of Data Science (Digital Commercial), Sanofi Anuj Maheshwari - Vice President, CVS Health Chris Han - Co-Founder & President, ThinkingAI Kevin Lee - Field CTO & Key Pursuits Leader, NiCE Explore how pioneering organizations across sectors such as finance, healthcare, retail, and manufacturing are achieving significant business outcomes with advanced AI agents. Industry executives and technology experts will share detailed accounts of their experiences, highlighting measurable benefits like increased revenue, enhanced operational efficiency, customer loyalty, and reduced costs. Gain actionable insights from their journeys, learn about emerging best practices, and understand how to navigate implementation challenges effectively. [Source](https://www.youtube.com/watch?v=NnYCjJmovDk)

[3:00] Nitesh Soni: Behind the scene we have an army of agents where one agent is basically trying to understand the intent and route request... bring back to the end users. [6:00] Anuj Maheshwari: We actually created 400,000 types of agents trained on real world simulator data that creates a bit of a flight simulator for us. [9:00] Chris Han: There is still such a big gap between the LLM and enterprise level agents... to align your agents can work together. [14:00] Nitesh Soni: If your data is not AI ready, you won't be able to go from small concept to scale... without that, you would be building your agents on sand.

[14:13] Nitesh Soni: AI agents built on poorly governed or low-quality data are built "on sand" and cannot scale sustainably across an enterprise. [14:13] Nitesh Soni: High algorithmic accuracy is useless if the outputs cannot be clearly explained to end-users. Without explainability, user adoption drops to zero. [22:42] Anuj Maheshwari: Processes that previously required half a dozen data analysts working for a full week to clean data and generate visual summaries can now be executed by AI systems in under 20 minutes to the exact same analytical standard. [22:42] Nitesh Soni: Accelerating the timeline from molecular discovery to direct patient therapy directly impacts human lives. Whether optimizing code generation, insight synthesis, or clinical trials, saving even a few hours per patient scales into thousands of lives saved globally. [30:58] Chris Han: Every token spent must directly tie back to measurable business targets. [32:36] Kevin Lee: Asking an AI agent to query fragmented content systems—like scattered Drupal or Adobe Experience Manager (AEM) sites—via Retrieval-Augmented Generation (RAG) or Model Context Protocol (MCP) servers fails when the underlying documents contain conflicting or outdated information. [32:36] Nitesh Soni: Utilizing AI tools to clean, catalog, and construct high-quality data pipelines ("AI for data") to power downstream agent models ("data for AI"). [36:14] Anuj Maheshwari: Full traceability down to the specific rationale for every agent decision must be an absolute design requirement.

[0:00] Morning [0:04] everybody. Thank you so much for being here.  Uh I'm Michelle Holly. I'm going to be hosting [0:09] this panel today. I'm the editor at Vector  where we cover the enterprise AI. Uh first [0:16] to get us started, we're going to do some quick  introductions so everybody knows who you guys [0:19] are. If you guys want to start down the line  here. Okay. Okay. Hi everyone. N Sony here. [0:25] Uh I'm head of data science at Sophi. If you  don't know Sonophi, Soni is a leading global [0:31] pharmaceutical company headquartered in France and  uh has operation all around the world. Uh I lead [0:40] a team of uh AI experts uh who are basically  building AI systems at scale to transform the [0:47] commercial operations at Soni uh to deliver the uh  better outcomes for our healthcare practitioners [0:54] and the patients uh worldwide. Hello everyone.  I'm Chris Han, co-founder of Thinking AI. We are [1:02] a Agentic Analytics and live ops platform to help  our clients to drive the business growth. You know [1:11] everybody is saying they're agentic plus+ what but  before this way we were a data [snorts] analytics [1:18] and live ops platform and uh top challenge from  me and my company is you know try to figure out [1:25] what can we leverage we have already in the past  and to amplify everything to the customers. So [1:32] most of our customers are in the game industry.  It's a very fast industry. So I'm really excited [1:38] to be here. Hi everybody. My name is Enoj Mher  for work CVS health. I'm the vice president of [1:46] pharma data and analytic strategy which is a  mouthful uh but essentially my role is how do [1:51] we actually leverage uh the the breadth and the  depth of the the data and the analytics that we [1:57] have at CVS as an enterprise. It's a diversified  healthcare company with a lot of firstparty data [2:02] uh and information. How do we actually try to  maximize the the outcome and the impact off of [2:07] that? Uh background, I've been within healthcare  analytics for 20 years. Uh and part of what I'm [2:13] hoping to do and accomplish here is that find  a way forward for us to modernize the way that [2:20] healthcare actually deals with analytics. Uh and  there's ton of ton of opportunities. I'm looking [2:26] forward to this discussion. Hi everybody. My name  is Kevin Lee. I'm hoping this microphone is on. [2:32] There it is. I'm the voice behind the podium for  those of you that are literally over there. Uh my [2:37] name is Kevin Lee. I am the field CTO and key  pursuits leader for NICE systems. Uh for those [2:42] of you who don't know who NICE is, uh we are  one of the world's leading organizations inside [2:47] of customer experience and the contact center  realm. I'm at the intersection of product and [2:53] technology and our client or partners that we  work with to extract these new fangled things [3:00] everyone calls outcomes. Uh apparently that is  the new buzzword desour and uh I'm excited for [3:06] the panel today. So thank you very much. And for  those of you who also stood in that 9-hour line [3:11] with me that didn't get a cocktail at the end  of all that uh we can all commiserate after. [3:18] So today we're going to get into the nitty-gritty  of AI agents, specifically what's working with AI [3:23] agents, uh challenges that you guys have come up  against and how you've solved them. So let's start [3:28] with an AI agent project that you've successfully  launched. Natash, do you want to go first? Sure. [3:34] Um so I I think uh obviously there there are  a lot of exciting initiatives in the company. [3:41] Uh what the one which is generally interesting to  me is building the talk to data platform for our [3:49] commercial teams. Uh where you can think about and  that talk data platform is basically powered by [3:54] uh the agents. Uh you can think about as a brand  managers or uh field reps or salespersons or [4:03] market access team. uh if you want to know uh on  daily basis okay how uh my products is performing [4:10] in certain territory or where are the biggest  opportunity where I should be focusing on or [4:15] actually uh where exactly I should be shifting my  workforce or or the funds etc etc basically you [4:22] take all these requests to analyst and analysts  basically uh try to find those answer by slicing [4:28] and dicing the data through multiple dashboards  or picking up information some of the AI solutions [4:34] spend and number of hours back and forth talking  to different teams before constructing the final [4:39] answers. But we want to change that experience and  that's why we are building this talk to data kind [4:43] of a platform. Uh and here now what is happening  the users can come with uh and interact with this [4:50] platform in a more conversational way. they can  ask questions and behind the scene we have a army [4:55] of agents where one agent is basically trying to  understand the intent and route request to the [5:00] right um right underlying data elements or or  the AI intelligence and so on and then we also [5:07] have like a system where agent agent collaboration  is happening which basically help to collaborate [5:12] across different agents and try to synthesize  information and bring back to the end users. [5:17] Uh so this is this is very interesting because  it's not only uh providing these insights uh or [5:23] or responses to the questions but also helping to  uh basically give them an opportunity to ask next [5:30] follow-up questions in a more smart way actually.  uh and not only that but also uh this sort of [5:37] platform basically are also helping to provide the  recommendations you know okay maybe you would like [5:42] to take this recommendations to perform next  best actions uh if uh you are really looking [5:47] for you know kind of optimum answer so this is  very interesting and the other uh exciting thing [5:52] which I saw here is uh it's basically helping to  democratize uh the insights so for examples the [6:00] uh brand managers or sale managers uh they can uh  do lot of self-s serving analytics at enterpriseg [6:06] grade level using this solution. So I'm I'm super  excited. Of course, if we have a lot of other [6:11] challenges how we can bring the uh acquisitiones  of those insights and how do we do other controls [6:16] but that is maybe uh this second part of it. I'll  park there for the later. When you say an army of [6:23] agents, how many are we talking about? Actually  u probably it really depends you know it's [6:29] uh right now I'm talking about 10 to 12  agents. uh some are like generic agents, [6:34] some are task specific agents and when I say  generic agents means okay how do we build more [6:39] like intellectual orchestration layer which can  understand the intent of questions and also uh [6:46] put a little bit of a control around you know the  who is asking this question is that person should [6:51] be exposed to all the data elements or only in  the restricted areas so basically 10 to 12 let's [6:58] kick it down to you thanks so much uh first Let me  just set the stage, right? CVS Health um doesn't [7:05] quite scream out as a native AI company. What we  are is a consumer centric company. Our ambition [7:12] is actually to use that as a unique selling point  and start to like embed places in which that that [7:18] advantage actually moves further with agents. So  the project that we have scaled at the company is [7:25] what we call agentic twins. Now what does that  what does that actually mean? Uh the the idea [7:31] here is that this is a project that brings to life  uh highfidelity privacy privacy preserved flight [7:39] simulators to understand how patients actually  work within healthcare choices. The idea and the [7:46] impact of of what this work is is that instead of  uh instead of just fielding programs around at our [7:53] at our at our retail stores, uh we actually want  to understand what the challenges of of patients [7:58] are on a day-to-day basis, right? And so that we  can test them before we actually roll them out. [8:03] The impact that we're we're hoping for around  this is that when we actually like put programs [8:08] out there, it actually helps move the needle  from an exponential p perspective. Um, I don't [8:13] know how many of you know this, but that like the  the genesis of this was like last fall. Uh, we we [8:20] know that millions of patients just don't end up  filling a prescription at the pharmacy. Millions [8:26] more actually don't continue on the therapy as  much as they as that their doctors prescribe. We [8:32] wanted to understand how, right? Uh, and Michelle,  I think you answer your question about like what [8:36] do we do? We actually like created uh 400,000  types of agents, right, off of like that would [8:43] actually are trained on real world simulator data  that creates a bit of a flight simulator for us, [8:49] right? Um so that the actually the idea is how can  you actually like test these concepts in a virtual [8:54] digital fashion, right? So to understand like  how how you might actually receive this message, [8:59] how can you actually improve it without having  to take the day like put a grand scale off of [9:04] a failed program right out there or like tweak  the programs and change the experience on the [9:08] consumers on on a day-to-day basis. So that's  the that's what we have actually like been able [9:13] to do and accomplish. Um there's been a tons of  tons of learnings but this is actually like part [9:18] of the bigger push off of where CVS wants to go  in the future off of going just for purely from [9:24] a healthcare perspective to also then after that  taking into the technology and products as well. [9:28] So like that's part of the the broader strategy  and how we want to take it. Uh Chris would you [9:34] like to chime in? Yes I think you know what  we've been working on is not changed. I think [9:41] we just you know change the methodology and the  philosophy especially in this era. I think there [9:46] are two main you know perspectives because the  top priority in a company or in a pri enterprise [9:52] is always the growth how to fix the issue of  business growth. I can see there are two main [9:59] challenges from technical issue. I think you know  in this agent era everyone can build your agent [10:05] very easily. Everyone can talk to you know bunch  of agents daily your personal agents or your team [10:11] agents but I still can see there is such a big  gap between the LM and enterprise level agents [10:18] you know to align your agents can work together we  need to input enough you know the knowledge base [10:25] and we all the enterprise level agents should work  on the same context and uh also like how to align [10:32] the you know team of agents on the same you know  same target in the workflow. So you need to figure [10:38] out the appropriate skill set which is leveraged  you know among humans before now we need to [10:45] leverage those skills to the agents and also like  you know the autonomous flow because the you know [10:54] the previous workflow or the you know find a find  the analyst insights drive those insights to your [11:01] business and find you know the right campaigns or  product updates it's all through the humans. But [11:08] right now we need to figure out when agents joined  in the flow how that flow can be. So we still have [11:15] a lot of gaps in terms of the technology you  know challenges and there is another you know [11:22] you know functional of a function you know level  perspective. I from what I heard you know in a lot [11:29] of studio here a lot of companies here we have a  lot of dashboards and we sit sit on you know you [11:36] know tons of dashboards but literally there's no  or nothing or few of insights we really can drive [11:44] to the growth business for example you know what  insights drive here for the next version product [11:51] you know what kind of you know live campaigns we  should launch today or next week because in game [11:56] industry we worked out in you know maybe probably  in one day there are thousands of AB tests and in [12:04] terms of UA and you know marketing you know there  campaigns going on so I see those two challenges [12:11] we start you know we have something going on we  finished part of it we already see some success [12:18] stories from our pilot programs but I I see  you you know there's still a long way to Well, [12:24] yeah. Kevin, did you want to add anything?  Um, yeah, I guess I would just uh kind of go [12:31] back to the original point of what are the the  business problems that we're trying to solve and [12:38] uh my organization has the unique perspective  of looking at uh the problem statement through [12:44] the the customer lens, right? So, the CX side  of things and that doesn't always align with [12:51] the business desires, right? and uh you know what  the business wants to solve and what technologist [12:56] wants to solve uh versus what do the customers  actually need solved that's something that needs [13:02] to get reconciled and just to dubtail onto the  last point what we see being missed often in [13:09] the journey mapping and BRD and whatever other  acronym and term we want to use is utilizing the [13:15] actual data right and what we're finding from our  side of the equation is that the data holds the [13:24] Uh and so that data is is underutilized in many  cases and more often than not what we're seeing is [13:32] this over application of the uh the generative AI  space and just saying hey you know what an LLM can [13:39] solve this problem well not always right because  the SOPs the the ways that we handle things inside [13:46] of our organizations whether it's seni or CVS are  going to be very very different and so analyzing [13:53] those interactions, understanding what's in those  uh historical conversations actually helps us [13:59] determine what the problems to solve are. And that  needs to then reconcile to ROI, right? Because we [14:05] all have KPIs and MBOs's and uh so I would say  that's actually ironically overlooked all too [14:13] often. Well, I'm sure you're all familiar with the  research. I think it came out of MIT that 95% of [14:20] AI deployments fail in pilot phases. It's a bumpy  road to get from the pilot to into production. Uh [14:27] Chris, you had started to talk a little bit about  some of the challenges that come along with this [14:30] process, but let's dive a little bit deeper.  When you guys started working on your agent [14:34] deployments, what challenges came up that you  had to solve? How did you solve it? Uh or what [14:38] learning moments came up? Uh Natasha, let's start  with you. Okay. uh some challenges obviously you [14:46] know one thing which I think just highlight that  okay always start with the business problem it's [14:51] it's quite obvious one not start the technology  itself but I I I'll highlight three on top of this [14:57] uh first one is uh data actually if your data is  not AI ready you won't be able to go from uh small [15:06] concept to scale and in the bigger organizations  uh I think scale matters uh So uh investing in [15:13] uh building the data foundation which means  having a better quality data which is well [15:18] governed I think this is the biggest ask and the  biggest thing and it's also not a very fancy or [15:23] sexy work right so don't not many people want to  do that but without that you would be building [15:28] your agents on sand uh they will not s they won't  be more like a sustainable so that is one thing [15:35] uh second is uh trust in your insights so uh no  matters how accurate ely you build your insights [15:43] but if you are not able to explain in a more  transparent way to your end users they're not [15:48] going to act on it if they're not going to act  on it adoption will be low and you're building [15:52] something which is [clears throat] might look very  fancy in terms of accuracy etc etc but if you're [15:57] not able to explain it yeah uh the impact for the  organization is is is very minimal or sometime [16:02] it's zero uh the third probably uh is the hardest  one uh is the change management uh and that people [16:10] always talk about but I think uh where we see  level of success that partnering very closely with [16:16] the business teams when we are early enough in  fact when we are talking about all these ideas how [16:22] we're trying to solve this problem with agents and  so on and getting their continuous feedback uh not [16:27] one time it's more like a continuous feedback loop  right and then improving those solutions and then [16:33] uh asking them to adopt basically basically so  and that adoption happen naturally if you start [16:38] explaining you know how these solutions build how  the feedback has been incorporated and then within [16:44] the business side creating more like a champions  uh Asian champions AI Asian champions right so [16:49] they are not only the user but they are also more  like people who are advocating for your products [16:54] so these are the three things actually uh which  I would like to highlight and uh the I think the [17:00] biggest one probably still the data foundation in  in the in the various organizations when you're [17:06] talking about change management uh trying to get  the end user to use the platform to trust it. How [17:12] soon in the process does that start? I think  right at the beginning you know when you start [17:17] talking about these things uh you know you know  there always hype you know there was generative [17:22] AI agents agentic and so on. So the moment you  know start talking about any new projects or any [17:29] new use case when you're doing prioritizations  all those conversation happening up front before [17:34] you can go into the education mode. Any other  challenges you guys would like to share? Yeah. Um, [17:40] just building on what Nate says, uh, I could  couldn't agree more about like building trust. [17:46] I mean, through this process, you're likely going  to need to bridge from what exists today to what [17:52] you want it to be in the future for a number of  reasons. Like kind of going back to the example [17:56] that I described, you like market research exists  today, right? Understanding customer insights is a [18:01] process that exists, but it is slow and painful,  right? Like you take 10 weeks in order to like [18:06] gather information and put it into a PowerPoint  slide. But if you want to then after that plus [18:10] it onto onto a new solution, you have to ground  it on you have to have the rigor right on saying [18:16] that hey this is what was existing or this is what  exists today. I'm going to get you all comfortable [18:22] with what it's going to look like just to make  sure that you have all see what's under the hood. [18:26] Validate that that takes time right that that  building off of trust on people who are going to [18:31] do that I think is going to be really important.  Another just interesting part of of like how you [18:36] to architect these solutions especially working in  the industries that that we do and like healthcare [18:40] there's always going to be increased scrutiny  right so as you're thinking about managing [18:44] change you also need to think about pulling that  into your design and finding the right balance [18:48] between human intelligence right that always just  is the right thing to do and putting that and then [18:52] augmenting that intelligence appropriately it  has a profound effect on how you even think [18:57] about these systems right what do you want an AI  to do what versus what it might be able to do, [19:03] right? So like sometimes one overrides the other.  Um you also just be like conservative and make [19:08] sure that like we are ready for the scrutiny  that's going to that's going to come onto it, [19:12] right? So those are the two things that I'd add  on. Working in a more regulated industry, how do [19:16] you think about observability and explanability  as you're building? Yeah, you can't. Absolutely. [19:24] Observability, explanability, transparency,  trust, like all of those things are are just [19:30] things that you have to frontload and think about  where what is the right thing to do. That kind [19:34] of falls right in the middle of it, right? And  working with your working within the constructs [19:38] of of what has been defined and what is evolving,  right? It's going to be like the most important. [19:43] Making maintaining like patient trust uh is is of  paramount important. ensure that safety signals [19:48] are also pulled in the right places so that you're  actually like doing what's right. And that's why [19:52] like testing out all of this all these tools  and technology profusely is often the right way [19:57] to go. It might be it might feel like you're  moving a little bit slower up front, but no, [20:04] you're actually building a strong foundation,  right? Kevin, you had mentioned KPIs briefly, [20:10] and I think that's one important thing. You have a  successful agent deployment. How are you actually [20:15] measuring that it's successful? What KPIs are you  guys specifically looking at? Yeah, so I think the [20:22] observability point that was just brought up  is is important, right? That there's this um [20:26] desire to design fast, iterate and deploy. Uh but  then especially in the at the enterprise scale, [20:32] uh there's a certain level of rigor, right,  inspection that we have to to have. Um and so [20:37] KPRs are one of those things. uh from the CX  side obviously we have a couple of industry [20:43] specific stats I would say seesat or customer  satisfaction in particular uh uh containment is [20:49] another that we're looking at um but the economics  of the actual interactions now are becoming much [20:57] more highly scrutinized uh apparently token maxing  from the other side of the equation isn't so good [21:03] uh so yeah understanding you know how much that  burn is happening happening on the other side. [21:10] So all of these different KPIs and stats are  things that our organizations or our partners [21:14] that we work with are looking at and so that  observability and those dashboards into that just [21:18] to add to your dashboard pains um are something  that uh we're looking at all the time. Well, [21:25] digging deeper into the cost side of things,  what do you have in place to make sure that [21:29] you're managing token costs? Yeah, so from uh  the nice platform, we're uh we're we're neutral, [21:36] right? So we work with all the major uh models  out there and so uh every model from the front [21:43] uh frontier models and the back round models  I'll call them. Um so uh we do also have a labs [21:50] organization that is uh constantly evaluating the  efficacy and abilities of these different models [21:56] uh so that we can help partner with our brands  that we work with uh to drive them to say hey um [22:02] when you're looking for something that is just  hey what is the status of my order you may not [22:08] even need a generative model right we can look  at deterministic and conversational uh AI for [22:13] those use cases uh but then if it's much more  complex, multi-threaded, uh multi-intent needs [22:19] conversationality, then we're going to look at  some of the much more capable models. Um, so that [22:25] uh reconciliation of abilities is something that  we we do with uh the brands that we work with and [22:31] then we have an orchestration uh engine as well  that helps determine which model to utilize as [22:36] well. Going back to the KPIs then uh Anz would  you like to jump in? Yeah, a couple of things [22:42] that we we track um is first one is what I like to  call like time to insight, right? So it's actually [22:51] like saying how do you how do you actually try to  try to say that this process used to take us x x [22:57] time now with the same sort of fidelity, accuracy,  safety, you're not shrinking it down. Like part [23:04] of what one of the early proof points are that it  used to take like a half dozen people like a week [23:09] to crunch through like a bunch of data to create  like a visual, right? Uh of course footnoted like [23:14] saying that this is all that went into it.  Now you can try to do it within 20 minutes, [23:18] right? With the same sort of degrees and standard.  So like that would be like that shrinkage and that [23:22] truncation off of here's the business value that  you can create and here's how it goes. Like so [23:27] that's that's one of the angles again like as  and building to to your point Kevin like the [23:33] angle for us is that on where this is all going I  think like connecting it back to the use case that [23:37] I described hopefully like down the road right  the the real value the eye on the the the key [23:44] thing is that are we able to affect change in the  marketplace right so like if our original problem [23:51] point was that we think that it's a bad thing that  like patients do not or is a missed opportunity [23:56] it's a burden on healthcare or it's a it's not  the right thing for the patients to actually [24:01] um not take particular medications or not not  take these actions. Can it actually change? Is it [24:08] actually easier for for for patients to experience  healthcare which is really challenging at the best [24:13] of times? Uh so that's that's like the overall  overall aspiration. Uh Chris Nesh would you guys [24:20] like to jump in? Maybe I can add because uh we  come from farmers. So I think uh I know mentioned [24:26] few times trust I mentioned few times trust but I  think the other important KPI actually uh is the [24:34] time time from uh obviously molecular discovery  to taking it to to the to the therapy actually to [24:40] the direct to the patient how we can shrink that  time. time doesn't matter whatever you do whether [24:45] you writing a code producing insights or or or  you know taking that molecules to to the market [24:52] whatever you're doing so wherever you can save  time uh is important for us whether it's a Genti [24:58] system your traditional AI system or whatever it  is so that time parameters is very very important [25:02] so that is this the most important KPI for me  whatever I do actually if I can save even uh you [25:09] know couple of hours for each patient I can think  But at a scale you can save a lot of lives. Yeah, [25:16] you know I really agree and that's what he just  said. No matter what kind of the format is the [25:22] data is still the foundation of everything you  know no matter what the agents is when you know [25:28] agent team scale data is always the foundation.  If you cannot get the right data you cannot get [25:35] you know the process data collection and uh you  will get you won't get anything. So in terms of [25:40] the metric is we work with a lot of clients about  business growth. I think uh the in according to [25:48] my experience the retention you know no matter  the agents or whatever other things I think the [25:55] retention of the users is still the key of the  business growth how sticky is your platform or [26:02] product is so building that flow to contribute to  the like the the the the retention I think that's [26:09] that's the you know always the the top priority  so somebody just said about the management and [26:15] operation things. I I think I want to share a  little example here. What I've been trying to [26:21] uh change in our company is the meeting flow.  you know the previous meeting flow is everybody [26:28] has your maybe 15 minutes updates blah blah blah  and um in terms of trans transparency you know [26:35] and just you know mentioned about that I build  up like the portal of everything my personal [26:41] work in there and uh everybody should build your  portal and it's HTTPL page and dynamic update so [26:49] before the meeting and we just you know let our  personal agent to read everything you know about [26:55] others updates so you don't have to spend time to  talk about the updates. So during the during the [27:01] meeting and uh because the agents already read the  updates and got the personal you know potential [27:07] questions and challenges for others and during the  meetings we can focus on the challenges and focus [27:14] on the questions and from there probably you know  we drive a lot of to-do list and actions I think [27:22] yeah you know to to to to reach the final goal  pros you know you don't have you you don't you [27:29] the the tools upgrade the pro update they're not  enough and we can think more about the management [27:36] you know the the operation the transparency  a lot of things yeah I want to circle back to [27:42] the cost discussion I think as a lot of people  are building out processes with AI new models [27:47] are coming out old models are being phased  out and costs are rising dramatically so I [27:52] I wanted to get some more perspective from  you guys on how you're managing those costs [27:59] anyone who wants to go. Uh yeah, I can go first.  Um again, like we're early in this journey, [28:05] right? Uh off of off of just understanding the  different options. I think there's so many so [28:11] many different directions that you can go. Uh,  so there's a little bit of conjecture from my [28:14] side of where this is going to like end up and  it'll probably be different for every for for [28:18] all of you, but the mix is probably still going  to be the same off of places where you're going [28:25] to just buy a foundational platform that is a  it's not going to be like a one size fits all, [28:29] but like the composition would be here's  a base off of capabilities that you have, [28:34] right? Like perhaps those are commoditized, right?  Then after that there's going to be like things [28:39] that where there are specialists, some skills,  right? as you're seeing now the the world divide [28:43] up between like broader LLMs and specific skills,  right? There'll be places where there will be that [28:48] that uh that place where you'd want to rent those  and that's going to be the areas in which you're [28:54] um from a strategy perspective thing like if  things keep on changing maybe you can you want [28:59] to maintain the ability to change those and then  there's going to be things that are intrinsically [29:02] like unique to your your your offering in this  whole thing that might be better for you to build [29:08] right so the idea is I think like when it where  when all said and done there'll be a combination [29:12] of buy rent and build yourself and as things kind  of evolve of like the ability for you to be able [29:18] to like move where those lines are probably helps  you maximize between your cost and your outcome. [29:26] Oh, I think very well said. I I think another  point is of course double click on this arc [29:30] textual efficiencies basically making choices you  know probably you don't need large language model [29:36] to solve everything in in the workflows and  also in some cases you don't need AI to solve [29:41] automation everything in the workflow itself right  so so the choices and also you know when people [29:46] are interacting with tools uh what exactly you  want to log what exactly you don't want to log [29:50] so there are a lot of things which you need to uh  understand there's no perfect answer right now you [29:56] know where exactly we can say but still there are  signals okay the probably we can create a you know [30:02] choices between large language model and small  language model to solve certain things or maybe in [30:05] some cases we just need like rulebased algorithm  right now which agent can handle automatically [30:10] and so on so uh th those are the things um which  matters uh but I think uh the other thing which I [30:19] just want to add is AI literacy element uh for for  the organizations uh you know creating little bit [30:26] of awareness you know what does it takes to build  these solutions within the organizations so that [30:32] every user understand it's not like something is  not free there's a cost behind it so every time [30:37] we talk about you know when we when we say okay we  want to prioritize something which can create this [30:42] much of impact there's a reason behind it because  we don't want to work on small small things and [30:47] the cost is huge and the impact is small so we  don't want to do that in the organization so we [30:50] need we need to make a trade-off as well yeah I  just want to add a like more best uh case study [30:58] from my personal work. I think everybody have your  multi model right now and uh obviously you don't [31:04] want to throw everything to fo that's going to be  very very expensive and uh so you can take a look [31:12] at your routine task list and your special project  you're working on. I think both based on those [31:18] task list, I think you can build a complexity  level, you know, based on like different tasks, [31:26] you know, probably you want to throw more complex  jobs tasks to to Fable to Opus maybe [snorts] [31:34] throw some like if you want to just steal just  information from the meeting nodes and probably [31:39] going to s them to the cheaper and uh models. So  take a look at your you know routine task list and [31:47] try to think about the the best strategy to you  know consume your tokens and secondly I think you [31:54] know every the token or economics should related  to your business for sure. I think everything we [32:01] done like uh I made a lot of mistakes you know  sometimes oh this is so easy this is so like [32:08] you know just take steps to build a fancy maps you  know probably everyone oh let's consume web token [32:14] to build a very fancy things but it doesn't really  tie back to your business so I think everything [32:20] you should think about your target should be  clear and your business goal should be clear [32:26] and tie everything back to that and uh once you  figured out That's probably going to save a lot [32:31] of money. We have a little less than 10 minutes  left, so let's open it up to the audience and [32:36] see if anyone has any questions they'd like to ask  just so everyone could hear. So the question was, [32:41] "How are you guys preparing your data  foundations?" I guess I could take a first [32:46] pass at this um from the purveyor side. So partner  organizations that we work with often go, "Okay, [32:52] we want a bot or an agent that does things." And  then I asked them, well, do you actually have one [32:57] place that has the repo that has the quantifiable,  definitive, curated, contemporary answer? And the [33:04] answer is always no. Uh, and so what we find  is that getting that uh that source of truth [33:12] to be curated closest to the source, which is  typically program managers, product owners, [33:18] right? and giving them that direct access so they  can do this is something that in the past just [33:23] didn't happen right so we're all familiar with  ragging and we've talked about knowledge bases but [33:30] did we actually give them creds and the ability to  go edit and MACD those those pieces of information [33:35] into that repository uh KMS if you will and uh  the answer across every organization I work with [33:43] is always no right it's just yeah we've got stuff  that uh is sitting in everything from Drupal to AM [33:52] uh and you know the person that actually  managed that is no longer here. So we have [33:57] five different versions of the same answer and  then we just say no problem. Uh we're going to [34:01] have an MCP server hook into all of them and  then now a model is supposed to uh adjudicate [34:08] which one's right. It it's uh the answer is no.  It doesn't work. So um starting with that uh [34:15] data estate making sure it's create uh curated  appropriately we find is the right first step. [34:24] Yeah. All agree apparently. [laughter] they agree  that nice is the way to do it. So just put that [34:29] out there. Go ahead if you haven't thought. No,  I I I think you already nicely describe what [34:35] summarized uh the uh normally in in the bigger  organizations we we talk about three layers of [34:42] these datas. One is source align assets being for  example if you want to get into sales informations [34:48] you know how your sales are happening. So you  have like external information captured in some [34:52] system you have also external provider for example  IQBI in our case. uh but then those become more [34:59] like source aligned assets. you know bringing  everything at one common place and from there [35:03] you start with something called as process  align assets means how this is aligned with [35:07] overall company process right so that is another  layer which is basically been build and that [35:14] layer is very important because if that is not  built nicely you cannot create any AI after that [35:19] right so uh here you should have like the data  is properly been managed properly been cataloged [35:25] every element should have description it's  it's not that easy but of course that is intent [35:30] and and from there you can create another layer  which is something called as consumption line [35:33] assets or maybe the data which is ready for  certain domains to conduct AI right and there [35:39] you can create lot of derivation of extra  features this and that and you can even put [35:45] dashboard on top of it or you talk to data kind of  a capability or you can build AI on top of it the [35:50] one thing which I want to mention here is that  we we we said like you know we need to have AI [35:56] ready data to build AI on top of it at a scale  But we are also using AI to build AI data. So [36:02] that so data for AI AI for data actually. So this  these are two which I want to highlight. So it's [36:08] this both way in today's world. We have about  one minute left. So we have time for one more [36:14] quick question over here. So how are you making  sure that the AI agents that you're building are [36:19] explainable, traceable? Whoever would like  to go. I mean I'll give a really easy one, [36:26] right? Like I mean that's a requirement simply  right like essentially like whatever you do like [36:33] like you should have all the way down to like  why did it do it and like it it's not a it's [36:38] not a it has to be part of your foundational  right like if that's your design principle [36:43] then you're most likely to not create things that  are black boxes at least that's like from from my [36:50] yeah and then and from practical standpoint you  have different ways for example if asking you [36:54] know user is asking question you you always say  okay give you know you like this answer not like [36:59] this answer not like this answer why you don't  like the answer we also do that as using certain [37:04] technology behind it like trace app and so on so  we uh we really look into different things and uh [37:11] for example this the solution which I describe  now uh the talk capability we also uh use multi [37:18] lm architect you know llm to address the data  quality lm to generate inside lm as a just to [37:24] uh you know uh understand the accuracy of those  insights and so on. So it's a combination of [37:29] many things. All right, that's all the time  we have. Thank you everybody for joining us [37:34] for this panel and thank you panelists for  being here. Thank you for your conference.