Audible and Tokio Marine Target Prop Data to Beat Base AI Models

Audible与东京海上聚焦专有数据，力求超越通用AI模型 Audible与东京海上的企业高管敦促各公司将专有数据嵌入灵活架构中，而非依赖标准基础模型，并警告称静态基础模型无法带来竞争护城河。 行业专家探讨了企业如何摆脱开箱即用的生成式AI模型，构建能够推动实际业务价值的定制化解决方案。他们概述了评估定制化程度的分析框架，例如平衡投入成本、准确度与客户价值，同时强调企业的真正竞争护城河在于其专有数据与工作流，而非可互相替代的底层模型。专家小组探讨了数据治理、保障安全性的混合RAG架构以及合规Copilot等运营痛点，并对小型专用模型、动态实时适配、智能体治理以及对照ISO 42001标准进行的企业基准评估做出了预测。 企业在决定定制生成式 AI 时应基于阶梯式路径评估：Audible 采用衡量价值（Value）、投资（Investment）与精确度（Accuracy）的 BIA 框架，主张从提示词工程逐步进阶到 RAG 与微调，而非盲目直接微调模型。 (1:55) 底层基础模型正逐渐沦为可替换的通用商品，企业的核心护城河在于私有数据、业务工作流与专有 IP，并应建立每 3 到 6 个月灵活轮换底层模型的系统架构，避免在非核心开发工具上重复造轮子。 (7:42) 数据治理、单一事实来源以及跨部门对齐通常占生成式 AI 定制项目总工作量的一半；针对法律或保险等受监管领域，基础模型 95% 的基准准确率远远不够，往往需要达到约 99% 的精确度。 (12:08) AI 战略应从单纯的降本增效转向拓展营收增量，例如定制核保模型若能将保单接收率从 4% 提升至 5%，即可直接带来 25% 的收入增长。 (17:30) 未来企业模型定制将向小型专业化模型、动态实时自适应及多智能体网络（Agent Swarms）演进，同时需通过本地混合 RAG 架构、离境数据过滤以及 ISO 42001 等标准化治理体系来防范安全合规风险。 (33:43)

Enterprise AI leaders argue that off-the-shelf foundation models are commoditized utilities, meaning sustainable competitive advantage depends on coupling proprietary data and core workflows with adaptable, loosely coupled model architectures. Frameworks for Model Customization (1:55) Customizing foundation models exists on a progressive spectrum rather than an immediate jump to fine-tuning. Turing Enterprises' Brent Blum notes that enterprises typically start with prompt engineering or customized workspace instructions before advancing to Retrieval-Augmented Generation (RAG) and hybrid fine-tuning. Audible uses a 'BIA' rubric (Value, Investment, Accuracy) to decide when to customize; for its recommendation tool Maven, combining proprietary content tags and behavioral listening data with RAG proved sufficient to deliver domain-specific relevance without expensive full-model retraining. Customization becomes mandatory when off-the-shelf systems fail linguistic, cultural, or precision requirements. Argos Multilingual applies client style guides and reference datasets to overcome generic direct translations in low-resource languages. Meanwhile, regulated industries often require up to 99% accuracy—compared to typical 95% off-the-shelf baselines—and need specialized tuning to parse complex document layouts such as legal strikethroughs. Architectural Decoupling and Avoiding Non-Core Traps (7:42) Patrick Gallic of Tokio Marine Group cautions that base models function like cell phones—widely available tools that offer no proprietary moat. Instead, an organization's lasting value resides in its intellectual property, operational workflows, and actuarial or business logic. Because frontier models evolve rapidly, enterprises must design an interchangeable model flywheel that allows underlying commercial or open-source engines to be benchmarked and swapped every 3 to 6 months without rebuilding workflows. Organizations must also avoid building tools where external commercial software is advancing faster than internal development. Tokio Marine previously attempted to construct proprietary developer tooling, only to abandon it when vendor tools like Cursor, Windsurf, and Amazon Q outpaced their internal build. Gallic advises limiting customization strictly to core functional drivers that directly affect business growth. Governance, Privacy, and Hybrid Sandboxes Addressing data integrity and clean single-source-of-truth consensus frequently consumes up to half of an AI project's effort. In highly regulated sectors such as life sciences and insurance, deploying custom assistants requires mitigating privacy and compliance vulnerabilities. Comcast leverages isolated sandboxes to validate custom models against business outcomes before committing capital, while financial teams deploy PRD-trained co-pilots that advance proposals 80% to 90% toward compliance prior to human review. To safeguard data when querying external foundation models, organizations deploy hybrid architectures. Blum points to enterprise patterns where vector databases and RAG pipelines are hosted entirely on-premises, backed by custom outbound security filters that dynamically redact personal identifiers or block unauthorized requests before payloads reach public provider APIs. To formalize oversight, enterprises are turning to ISO 42001 certification to establish verifiable risk management and auditability. The Frontier of Multi-Agent Systems (35:08) While Robotic Process Automation (RPA) proved fragile due to brittle rule-based scripts, panelists view autonomous multi-agent networks as the next frontier for executing complex workflows on internal data. By leveraging chain-of-thought reasoning, agent swarms can evaluate diverse solution paths simultaneously. However, enterprise adoption faces friction: linking agents to enterprise applications via APIs or the Model Context Protocol (MCP) mirrors the slow, heavy integration overhead historically seen in legacy enterprise IT platforms like MuleSoft. Panelists anticipate that future customization will trend toward smaller, industry-tailored models and real-time contextual adaptation. As multi-agent architectures expand, organizations will increasingly deploy specialized agentic governance layers—in effect, autonomous oversight agents designated to monitor permissions, collocated data access, and security policies across other operational AI agents. While custom generative AI can unlock significant top-line gains, enterprises face recurring tradeoffs between custom builds and vendor capability leaps, alongside persistent IT integration friction when attempting to operationalize multi-agent networks at scale.

在由Ai4主办的研讨会上，来自Comcast、Tokio Marine、Audible、Argos Multilingual与Turing Enterprises的行业高管指出，通用基础大模型本身无法构成长期竞争壁垒，企业的核心护城河在于私有数据、业务流程与专有知识产权。企业推进生成式AI定制化时应采取渐进演进策略，将底层模型设计为可定期轮换的解耦引擎，并结合混合检索架构与ISO 42001等标准化治理框架控制合规与精度风险。 定制化决策框架与渐进演进路径 (1:55) Audible产品负责人Sue Shlapakovsky提出BIA评估框架，即从客户价值（Value）、长期投入与维护成本（Investment）以及领域精度要求（Accuracy）三个维度衡量模型定制化的必要性。她认为基础模型在摘要生成与通用自然语言理解上表现充分，但在个性化推荐等场景中，将专有数据与检索增强生成（RAG）结合即可满足业务需求，无需承担全量微调的高昂维护成本。Argos Multilingual的Raffaele Pascale补充指出，当通用工具在小语种、特定品牌语调或文化语境中的表现不达标时，企业摄取专有风格指南和参考资料进行定制是保障本地化质量的必要手段。 Turing Enterprises的Brent Blum强调定制化存在清晰的渐进尺度。企业通常无需起步即进行大规模模型微调，而应从提示词工程或项目级工作区配置（如带有自定义指令和文档上传的ChatGPT Projects）开始探索，随后根据业务投资回报率的基准评估，再逐步升级至基础RAG、混合RAG或微调模型。 专有数据壁垒与可插拔系统架构 (7:42) Tokio Marine的AI主管Patrick Gallic提出，基础模型如同普及的智能手机，无法为企业提供独特的竞争优势；真正的护城河沉淀在企业的专有数据、核心运营流程与知识产权中。鉴于基础大模型迭代极其迅速，Gallic建议企业构建敏捷的系统飞轮，将底层模型视作每隔3至6个月便可进行基准测试并随时替换的通用引擎，避免架构与单一供应商发生技术锁定。 Gallic同时分享了反面经验，警示企业切勿在非核心功能工具上过度定制自研。Tokio Marine早期曾耗费巨资自研内部开发者工具，但很快被市场上快速迭代的Cursor、Windsurf和Amazon Q等商用产品超越。在有效利用专有数据方面，Audible的Shlapakovsky介绍了其在2024年底上线的对话式搜索引擎Maven，该系统将内容深度理解（细粒度主题与叙事弧）、收听行为模式及用户偏好三层专有数据输入模型，实现了高度定制化的推荐能力。 业务增量价值、精度痛点与组织文化 多位专家指出，AI定制化投资的战略重心应从单一的成本缩减转向驱动核心业务增长。以非准入类保险业务为例，承保人通常仅接受4%的投保申请，Gallic指出若通过定制化AI将捕获率提升至5%，即可直接撬动25%的保费收入增长。在全球化布局上，亚洲机构偏重利用自动化应对严峻的劳动力短缺，而美国业务则侧重利用定制模型分析复杂的风险多样性。然而在落地受监管行业时，标准大模型约95%的基线准确率往往不足以支撑要求达到约99%精度的法律与保险审核，且模型在解析带有删除线等特殊格式的复杂文档时仍存在明显短板。 为消除复杂流程中的审批阻塞，部分企业在产品需求文档（PRD）上训练定制化合规助手，在人工介入前即可完成80%至90%的合规自检。在组织文化保障层面，Audible建立了三层推进机制：底层由管理层通过报销GenAI课程与季度成果展提供支持；中层制定容错规则，并采用采纳率和反馈等过程指标而非严苛的产出KPI；顶层则由基层AI爱好者通过工作坊与分享会自发驱动创新。 混合RAG安全架构与ISO治理标准 (33:43) 在涉及生命科学与法律等强监管领域时，数据隔离与安全防护是推进定制化的关键考量。Comcast等企业通过部署独立的沙箱环境，在投入巨资前验证定制模型的业务成效与合规性。针对敏感数据调用公共模型API的安全隐患，Blum分享了一种混合RAG架构：企业将向量数据库及检索管道完整保留在本地内部网络中，仅在向外部模型发起请求前部署定制的安全过滤层，动态脱敏敏感信息或阻断违规查询。 面对行业内将“负责任AI”停留在宣传口号的现状，研讨会探讨了合规标准化的必要性。与会专家证实，多家大型企业已开始积极对照人工智能管理体系标准ISO 42001进行内部基准评估，部分业务部门正在申请正式认证，第三方咨询机构亦逐步将ISO 42001合规成熟度评估作为专业服务推向市场。 智能体蜂群演进与动态治理趋势 (35:08) 探讨AI技术演进时，Blum指出相比规则僵化的传统机器人流程自动化（RPA），基于思维链推理的自主智能体和“智能体蜂群”能够动态分析庞大的潜在解空间并执行实际业务操作。但多位专家一致认为，将智能体连接至企业现有工具体系的底层架构目前面临较大阻力，其接口开发与上下文协议封装的繁琐程度堪比传统的企业级IT集成工程。 在未来12个月的趋势预测中，专家认为行业将加速向针对智能体任务优化的专业小模型迁移，用户交互界面也将进一步向非技术业务人员下放定制能力。针对多智能体网络中的权限漏洞和数据穿透风险，企业将必须开发由专用监督智能体审查执行智能体的自动化治理机制，以确保复杂系统运行的可视性与合规可控性。 生成式AI定制化并非即插即用的单点采购，而是涉及数据资产化、架构解耦与流程重塑的系统工程。尽管混合检索和智能体网络能够释放显著的业务增量，但模型接口集成的工程负担、受监管业务的超高精度门槛以及多智能体协同的安全可见性，仍是企业在规模化应用中亟需持续应对的结构性挑战。

## Background 本内容由 YouTube 频道 Ai4 于 2025 年 8 月 24 日发布。该研讨会汇聚了来自 Comcast、东京海上集团（Tokio Marine Group）、Audible、Argos Multilingual 及 Turing Enterprises 的行业领袖，重点探讨了企业在定制生成式 AI 大模型与多模态模型时的技术与战略考量，包括模型微调、私有数据融合、平衡算力与准确率，以及如何在各垂直行业中交付真正驱动业务结果的定制化解决方案。 ## TL;DR 研讨会深入剖析了企业评估与落地生成式 AI 定制化的全套方法论。专家们指出，基础大模型应被视为可随时插拔替换的“通用引擎”，企业的核心护城河在于自身的私有数据、业务工作流与专有知识产权（IP）。针对不同场景，企业可采用包含轻量 Prompt 工程、RAG 混合架构到深度微调的渐进式路径，并借助 Audible 的 VIA 框架等模型进行 ROI 权衡。此外，嘉宾们还重点探讨了在强监管行业中的数据治理与沙箱机制、多智能体协同（Agent-to-Agent）的治理难题，以及对标 ISO/IEC 42001 等国际 AI 合规标准的必要性。 ## Atmosphere & style 整场研讨会专业务实、节奏明快，充溢着资深企业实践者冷静而清醒的工程思维。主持人与嘉宾之间的交流默契流畅，对话完全跳脱了泛泛的“AI 狂热”宣传，直面数据治理繁琐、API 集成痛点以及合规风险等一线真实挑战。现场氛围在严谨的架构拆解与幽默的行业自嘲（如将数据比作“自家特别的孩子”）之间切换自如，观众 Q&A 环节更是针对强监管下的 Agent 失控隐患展开了极具深度的攻防式讨论，给听众带来极高信息密度的前沿企业 AI 落地图景。 ## Key takeaways - [01:55](https://www.youtube.com/watch?v=EVKg3Z0TCuU&t=115s) 企业在决定是否定制模型时应依据价值、投资与精度（VIA 框架）进行评估；多数场景下结合私有数据与 RAG 即可满足需求，无需盲目进行深度微调。 - [07:42](https://www.youtube.com/watch?v=EVKg3Z0TCuU&t=462s) 企业的核心护城河在于专有数据、工作流与业务 IP，而大模型只是 3 至 6 个月即可插拔迭代的引擎，企业应将核心业务逻辑与具体模型实现解耦。 - [10:04](https://www.youtube.com/watch?v=EVKg3Z0TCuU&t=604s) 企业应停止在成熟通用工具（如代码生成助手）上重复造轮子，而将研发资源集中在能带来竞争差异化的核心业务数据与职能流程中。 - [11:48](https://www.youtube.com/watch?v=EVKg3Z0TCuU&t=708s) 专有数据的核心价值在于与基础模型锚定（Grounding），通过结合深度内容理解与用户行为数据，构建基础模型单独无法生成的个性化体验。 - [13:44](https://www.youtube.com/watch?v=EVKg3Z0TCuU&t=824s) 前期数据清洗与建立跨部门“单一事实来源”占据了 AI 项目大半精力，面对未成熟技术场景，企业需在重金攻坚自研与等待成熟方案之间理性权衡 ROI。 - [17:30](https://www.youtube.com/watch?v=EVKg3Z0TCuU&t=1050s) 破除管理层对 AI“廉价魔法”的误解，将模型定制深度融入核心运营（如将保险审核转化率提升 1% 带来 25% 业务净增），其价值远超单纯的表层降本。 - [19:48](https://www.youtube.com/watch?v=EVKg3Z0TCuU&t=1188s) AI 定制化必须以业务战略为先导，跨国企业需根据不同区域的实际痛点（如亚洲的人力短缺 vs 欧美的复杂核保风控）推行差异化的定制策略。 - [20:36](https://www.youtube.com/watch?v=EVKg3Z0TCuU&t=1236s) 定制化 Copilot 承担前期 80% 至 90% 的基础合规审查工作，能够大幅降低业务部门的创新阻力，让资深专家专注于最终关键裁决。 - [21:55](https://www.youtube.com/watch?v=EVKg3Z0TCuU&t=1315s) 推动 GenAI 落地需建立“三层金字塔”体系：以高管支持与文化为基石、以安全沙箱和前置指标为流程保障、自下而上激发全员实验热情。 - [24:26](https://www.youtube.com/watch?v=EVKg3Z0TCuU&t=1466s) 为防范微调过拟合与模型偏见风险，必须建立严格的输入数据基准评测、业务流程中的自动化检查点以及关键环节的人机协同复核机制。 - [29:59](https://www.youtube.com/watch?v=EVKg3Z0TCuU&t=1799s) 多模型部署与 Agent 爆发式增长会带来运维人力过载的风险，未来需探索“智能体监督智能体”（Agent-to-Agent Governance）的自动化治理模式。 - [30:49](https://www.youtube.com/watch?v=EVKg3Z0TCuU&t=1849s) 在强监管和敏感数据行业中，可采用混合 RAG 架构（核心知识库驻留本地 + 算力调云端），并在出站请求前配置安全过滤层进行实时脱敏与拦截。 - [35:08](https://www.youtube.com/watch?v=EVKg3Z0TCuU&t=2108s) 多智能体协同（Agent Swarms）展现出远超传统 RPA 的灵活性与抗脆弱性，但目前大规模落地的主要瓶颈仍在底层工具调用协议与复杂的 API 集成。 ## Detailed walkthrough 以下是根据视频转录整理的详细内容记录与分析： ### [00:04](https://www.youtube.com/watch?v=EVKg3Z0TCuU&t=4s) 研讨会开场与与会嘉宾自我介绍 主持人、Comcast 人工智能战略与转型高级总监 Sean Singer 宣布研讨会正式开始。本次研讨会聚焦于“通过定制生成式 AI 模型提升企业业务价值”这一核心议题。随后，四位来自不同行业的重磅嘉宾依次介绍了各自的背景与业务领域： * **Patrick Gallic**（东京海上集团 Tokio Marine Group 国际 AI 中心负责人）：东京海上是日本最大的保险公司，市值约 700 亿美元，在全球 45 个国家/地区拥有 100 多家分支机构，采用联邦化（federated）运营模式。 * **Sue Shlapakovsky**（Audible 副总裁兼产品负责人）：负责 Audible 的用户增长、内容发现产品管理团队以及应用科学（Applied Science）团队，业务覆盖从用户订阅到内容推荐的全流程。 * **Raffaele Pascale**（Argos Multilingual 产品创新与解决方案总监）：专注定制化生成式 AI 解决方案，帮助客户扩展多语言业务、验证并准备数据集，以及提升内容质量，解决现有通用方案无法满足的全球化需求。 * **Brent Blum**（Turing Enterprises 解决方案负责人）：Turing 是一家 AI 咨询公司，主要负责两类业务：一是为 OpenAI、Anthropic、Google 等顶尖实验室训练 AI 模型；二是将这些前沿洞察转化为定制化 AI 解决方案，交付给《财富》1000 强企业。 ### [01:55](https://www.youtube.com/watch?v=EVKg3Z0TCuU&t=115s) 评估 AI 模型定制化需求的核心标准 针对主持人提出的“企业在决定是否定制生成式 AI 模型时应考量哪些标准”这一问题，各领域专家分享了不同的评估维度与实战经验： **语言与品牌契合度（Argos Multilingual）** Raffaele Pascale 指出，当现成（off-the-shelf）模型无法满足客户在语言准确性、文化语境或品牌一致性方面的要求时，定制化就显得至关重要。特别是在小语种（less commonly spoken languages）领域，无论某种语言的使用人口多少，客户都要求在所有语言翻译中保持统一的语气和专业术语。Argos 通过向系统注入各语言的样式指南（style guides）和参考资料，生成真正由本地化视角产出的“本土化内容”，而非单纯的字面翻译。 **VIA 决策框架（Audible）** Sue Shlapakovsky 强调，通用基础模型在文本摘要和自然语言理解等任务上表现优异，很多场景下配合检索增强生成（RAG）或提示词工程即可满足需求。但在权衡是否进行深层定制（如微调）时，Audible 采用 **VIA 框架** 进行评估： ```mermaid flowchart TD accTitle: VIA 决策框架图 subgraph VIA["VIA 决策框架 (Value, Investment, Accuracy)"] A["业务需求分析"] --> B["Value 客户价值"] A --> C["Investment 投资成本"] A --> D["Accuracy 领域精确度"] B --> E{"评估与匹配"} C --> E D --> E E -->|"基础需求/低成本"| F["通用基础模型 + Prompt工程"] E -->|"中度精准/自有数据"| G["RAG + 私有数据组合"] E -->|"高度特定/深度绑定"| H["模型深度微调 (Fine-tuning)"] end end ``` 在 Audible 的内容发现（discovery）业务中，尽管客户价值（Value）与投资成本（Investment）均极高，但实践表明，**将私有数据（proprietary data）与 RAG 技术相结合**，无需进行复杂的模型微调，即可达到理想的客户价值水平，成功实现了运营效率与用户价值的平衡。 **渐进式定制路线（Turing Enterprises）** Brent Blum 提倡一种渐进式（progressive）探索策略。他认为许多企业倾向于盲目追求深度定制，但实际上，编写高质量的提示词（Prompt Engineering）、使用 ChatGPT Projects 上传特定指令和文档，本身就是一种轻量级的定制化手段。企业可以先尝试这些低成本方案，观察是否能获得优于基线 LLM 的效果，再根据明确的业务问题决定是否跟进全套 RAG、混合 RAG（hybrid RAG）或模型微调。 **交互评估与产品路线图（Comcast）** Sean Singer 补充指出，Comcast 作为全美最大的宽带互联网服务提供商，客户交互是其核心业务。通过将现成模型与定制化奖励模型（customized reward model）进行对比，不仅能评估客户交互质量，还能挖掘出传统模型无法发现的新需求，从而反哺产品路线图规划。 ### [07:42](https://www.youtube.com/watch?v=EVKg3Z0TCuU&t=462s) 企业护城河打造与“可替换模型”飞轮策略 Tokyo Marine 的 Patrick Gallic 深入探讨了企业选择**不定制**或**精细化定制**的策略选择。他直言，虽然自己个人在各种 AI 许可证上花费巨大，但从企业价值角度来看，单纯接入基础模型（foundation models）的门槛极低，如同“人人都有手机”，无法构成竞争壁垒。 企业真正需要的是利用 AI 重塑核心运营流程（core operations），而非仅仅覆盖辅助性的功能区域（functional areas）。 ```mermaid flowchart LR accTitle: 企业AI核心资产与模型飞轮架构 subgraph EnterpriseMoat["企业核心护城河 (不可替代资产)"] D1["私有数据 (Data)"] D2["工作流 (Workflows)"] D3["精算模型/业务流程 (IP & Processes)"] end subgraph InterchangeableEngine["可替换模型引擎 (3-6个月迭代)"] M1["模型 A (如 GPT-4)"] -.->|"定期替换"| M2["模型 B (新一代模型)"] end EnterpriseMoat <-->|"数据与指令交互"| InterchangeableEngine ``` Patrick 提出了企业 AI 部署的核心逻辑： * **企业护城河在于私有 IP**：企业的投资重点应放在自身的私有数据（Data）、工作流（Workflows）和独特业务流程（IP/Processes）上。 * **模型是“可替换的引擎”**：鉴于大模型技术迭代极快（每 3 到 6 个月可能就需要更换一次），企业不应过度绑定或死磕某单一模型的定制，而是要构建一套可持续运转的“飞轮（flywheel）”，使模型能够像引擎一样随时插拔替换。 * **应对模型更新带来的挑战**：例如 GPT-5/OpenAI 的新模型选择器（model picker）机制改变了模型调用的底层逻辑，对处于高合规监管行业（regulated industry）的企业工作流造成了很大冲击，这进一步印证了将业务工作流与具体模型解耦的重要性。同时，对于非核心的功能区域，企业应避免盲目投入研发资源。 ### [10:04](https://www.youtube.com/watch?v=EVKg3Z0TCuU&t=604s) 寻找核心差异化：从自研工具转向专注核心业务数据 小组讨论首先探讨了企业在拥抱 AI 时的弯路与认知迭代。发言者分享道，AI 刚兴起时，团队对代码生成能力感到极其兴奋，投入了大量资金和时间试图为内部程序员自研编程工具。但随着时间推移，他们清醒地意识到市场上成熟的专用工具（如 Cursor、Windsurf、Amazon Q 等）在底层能力和工程打磨上投入巨大，体验远超内部自制；与其把时间浪费在通用工具的定制上，不如将精力和资源聚焦在能带来“竞争差异化”（Competitive Differentiation）的核心业务职能领域。 针对如何识别哪些数据真正具备定制价值，嘉宾们打了个形象的比方：“数据就像小孩子，每个人都觉得自己家的那个最特别（Data can be a little bit like children. Everyone thinks theirs is special）。”东京海上集团（Tokio Marine Group）的 Patrick Gallic 指出，企业通常很清楚自身独特的资产所在——要么是历经数年沉淀的行业专有数据，要么是从关键客户交互中采集的一手信息。在推进数据定制时，基准测试（Benchmarking）至关重要：企业应先评估基础大模型（Baseline LLM）的表现，再比对加入少量定制数据后的效果。采用“小步快跑、循序渐进”（Crawl, Walk, Run）的方法，才能在早期阶段准确判断某项数据是否值得进一步深耕投入。 ```mermaid flowchart TD accTitle: 数据价值评估与定制决策流程 subgraph S1["1. 资产梳理"] A["历史长期沉淀数据"] B["核心客户交互数据"] end subgraph S2["2. 基准评测 (Benchmarking)"] C["Baseline 基础模型测试"] D["轻量定制模型测试"] C --> E{"效果对比 (Crawl-Walk-Run)"} D --> E end subgraph S3["3. 决策分流"] E -- "提升明显" --> F["深度定制并整合业务"] E -- "提升有限" --> G["暂缓投入 / 等待行业演进"] end A --> C B --> D ``` --- ### [11:48](https://www.youtube.com/watch?v=EVKg3Z0TCuU&t=708s) 深度内容理解与行为数据：Audible 的个性化实践 Audible 产品副总裁兼主管 Sue Shlapakovsky 详细拆解了如何利用专有用户数据与内容数据构建真正的竞争壁垒。在 Audible，数据的核心使命在于改善用户体验与驱动个性化推荐。他们构建了多层级的数据体系： 1. **深度内容理解（Deep Content Understanding）：** 超越了传统的书籍流派（Genre）或基础元数据，深入解析故事弧线（Story arcs）、主题标签和语境细节——例如不再仅仅标记为“心理惊悚小说”，而是细化为“10小时以内、烧脑反转的心理惊悚有声书”； 2. **用户行为数据（Customer Behavioral Data）：** 细致追踪用户的真实收听习惯，包括收听模式、在何处播放、在何处暂停、哪些内容会引发连续刷听（Binge listening）； 3. **用户偏好与上下文（Customer Context）：** 综合用户的喜好、厌恶偏好与具体场景。 Sue 以 Audible 去年底推出的 AI 对话式搜索引擎 **Maven** 为例，说明了数据落地的威力。Maven 允许用户输入自然语言查询并返回高度定制化的结果。其核心壁垒在于并非简单调用基础大模型直接输出结果，而是将基础模型锚定（Grounding）在上述丰富、专有的内容与行为数据集上，从而产生基础模型自身绝不可能单独生成的独特、相关且高度个性化的体验。 --- ### [13:44](https://www.youtube.com/watch?v=EVKg3Z0TCuU&t=824s) 模型定制落地中的痛点：数据治理、技术瓶颈与 ROI 抉择 在探讨模型定制的具体痛点时，嘉宾们指出了从准备工作到技术实现的多重阻碍： - **前期数据治理（Data Governance）：** 数据清洗、数据完整性确认以及各部门之间就“单一事实来源”（Source of Truth）展开的跨部门拉扯，往往占到了整个 AI 项目“一半以上的战役”。 - **特定场景的技术瓶颈：** 例如在处理带有删除线（Strikethrough）的法律文档时，现有大模型在未微调前无法正确解析，导致无法直接进行文档比对；而在引入 RAG（检索增强生成）处理专有数据时，全行业的召回率与准确率表现也参差不齐。 - **自研与跟随的权衡（Build vs. Wait）：** 面对未成熟的技术点，企业必须权衡是自己花重金攻坚，还是等待行业基础设施完善后直接采购成熟产品。例如在保险业务的提交材料摄入解析（Submission Ingestion）场景中，业务需要 99% 的准确率，而当前技术只有 95%，差这 4% 就无法商用；团队需要持续进行敏捷基准评测，决定何时跟进而非盲目消耗研发周期。 - **客户专有数据的价值释放：** 来自 Argos Multilingual 的 Raffaele Pascale 补充道，尽管训练和定制方案需要耗费大量时间与算力资源，但只要结合客户提供的差异化专有数据，生成的结果就能精准契合客户预期，从通用泛化的输出跃升为高质、可扩展且具备商业独特性的解决方案。 --- ### [17:30](https://www.youtube.com/watch?v=EVKg3Z0TCuU&t=1050s) 培育实验文化与管理层认知：AI 不是廉价的“魔法” 讨论转向如何在组织内部构建测试、学习与实验的文化。发言者从第一性原理出发，将 AI 技能普及与职业发展历史做了生动类比： - **全员 AI 素养基线：** 就像早年求职者在简历上写“精通 MS Office”或“精通 Lotus Notes”一样，如今让全员具备“精通 AI”（Proficient in AI）的能力已成为企业运转的底线能力。员工应经历从合规使用、工具操作培训到激发业务想象力的标准路径。 - **破除管理层的“魔法幻觉”：** 真正的挑战在于高管层的认知教育。许多管理者仍误以为 AI 极其廉价且如魔法般神奇，“随便丢进去就能自动跑通”。必须让管理层认识到，若要利用 AI 重塑核心业务，必须进行全流程的深度投资与模型定制。 - **商业价值实例：** 以非承保（Non-admitted）保险业务为例，传统人工审核下承保人可能只能处理 4% 的投保申请（Submissions）。通过定制 AI 提升审核吞吐与精准度，哪怕将承保比例从 4% 提升至 5%，在业务体量上就意味着实现了 **25% 的业务净增长**。这种从底层改造业务创造的增量价值，远非单纯的“降本增效（Cost Optimization）”所能比拟。 ### [19:48](https://www.youtube.com/watch?v=EVKg3Z0TCuU&t=1188s) 业务导向的定制化策略与跨国差异 专家们探讨了生成式 AI 在不同业务场景下的落地逻辑：不应盲目兜售技术再寻找问题，而是要反转思维，先厘清业务战略，再基于战略需求思考如何定制和应用 AI。跨国企业在不同地区的诉求差异尤为明显。例如在东京海上集团（Tokio Marine Group），亚洲分支机构面临的核心痛点是人力资源紧缺（Headcount 不足），因此 AI 的首要任务是流程自动化与效率优化；而在美国市场，业务面临更多样化的复杂风险，AI 定制的核心目标则转向如何提升核保能力（Underwriting）。这种业务教育让团队理解“为何要定制”，进而推动针对性开发。 ```mermaid flowchart TD accTitle: AI 定制化业务驱动架构 subgraph Strategy["业务战略分析"] A["明确业务战略与痛点"] end subgraph Scenarios["区域与职能差异化诉求"] B["亚洲市场：人力短缺/流程优化"] C["欧美市场：复杂风险/核保能力"] D["金融中台：合规审核瓶颈"] end subgraph Implementation["AI 定制落地"] E["流程自动化 Agent"] F["风险核保评估模型"] G["PRD 合规助理 Copilot"] end A --> B & C & D B --> E C --> F D --> G ``` ### [20:36](https://www.youtube.com/watch?v=EVKg3Z0TCuU&t=1236s) 案例分享：金融合规 Copilot 破解产品创新瓶颈 Brent Blum 分享了为一家金融服务公司构建 Copilot 聊天机器人的实际案例。该公司产品经理不断提出出色的新产品方案，但合规团队（Compliance Team）人力严重不足，而产品经理本身并非合规专家，导致产品需求文档（PRD）在合规审核阶段形成严重堵塞，进而让产品团队在提出创新想法时畏手畏脚。 项目团队引入定制化合规助手，陪伴产品经理撰写文档并进行实时合规性校验。虽然它不能完全替代最终的人工签批，但能完成 80% 到 90% 的前期准备工作，确保提交给合规专家审阅时已基本达标。整个团队文化随之转变，大家将 AI 视为“新入职的初级合规专员”，虽向合规负责人汇报，但极大解放了生产力并消除了创新阻力。 - 产品经理在前期缺乏合规确定性导致创新意愿受阻，AI 助手充当了“随行副驾（Right-along Assistant）”。 - AI 承担了 80%–90% 的基础合规排查，使稀缺的资深合规专家只需专注于最终裁决。 - 将 AI 拟人化为“团队新员工”，有效降低了组织对新技术的抵触情绪。 ### [21:55](https://www.youtube.com/watch?v=EVKg3Z0TCuU&t=1315s) 受监管行业的沙箱治理与 Audible 的三层金字塔模型 在受高度监管的行业中，客户隐私和数据安全是首要红线。Comcast 的 Sean Singer 指出，必须建立完善的治理框架与沙箱（Sandbox）测试环境，让业务人员能够安全、快速地试错与学习，从而确保大规模投资定制化模型时能够稳定实现预期的商业价值。 Audible 的 Sue Shlapakovsky 随后分享了 Audible 推动生成式 AI 落地的“三层金字塔”方法论： ```mermaid flowchart TD accTitle: Audible AI 落地三层金字塔模型 subgraph Top["顶层：个体赋能 (Individual Empowerment)"] T1["自下而上的 AI 极客生态"] T2["午餐学习会 (Lunch & Learns) / 工作坊"] end subgraph Mid["中层：流程与指标演进 (Processes & Input KPIs)"] M1["安全使用规范 (隐私/安全/容错沙箱)"] M2["关注前置指标 (Leading Indicators/活跃度/质性反馈)"] end subgraph Base["底层：文化与领导力 (Culture & Leadership)"] B1["管理层早期全面支持与资金投入"] B2["全员 GenAI 课程报销计划"] B3["季度 POC 项目成果展示会 (Showcases)"] end Top --- Mid Mid --- Base ``` - **底层（文化与领导力）**：管理层早期全力支持，推行全员生成式 AI 课程费用报销政策，并举办季度展示会（GenAI Showcases），让开展 POC（概念验证）的员工直接向高管展示成果。 - **中层（流程与指标演进）**：制定清晰的内部安全使用规范，划定隐私与数据安全红线，提供“允许失败（Safe to fail）”的试验环境；考核机制从单一死盯业务终点指标，转为关注“前置指标/输入指标（Input Metrics / Leading Indicators）”，如参与度、采纳率以及客户质性反馈，给团队足够的迭代周期。 - **顶层（个体赋能）**：激活内部的“AI 极客（AI Enthusiasts）”，通过自下而上的工作坊与午餐分享会（Lunch and Learns），营造自发的探索氛围，弥补自上而下推动的不足。 ### [24:26](https://www.youtube.com/watch?v=EVKg3Z0TCuU&t=1466s) 应对模型定制风险：基准测试与人机协同检查点 针对现场观众关于定制化过程中“模型未对齐（Model Misalignment）”及“偏见引入（Bias）”等风险的担忧，专家们给出了实操建议： Sean Singer 强调必须回归严格的基准测试（Benchmarks）。定制化过程中容易出现过度微调（Overtuning）或错误微调（Mistuning），必须保证输入数据的纯净度，并通过明确的业务目标持续进行指标比对，从而判断模型是否达到预期效果且未发生偏移。 Argos Multilingual 的 Raffaele Pascale 补充道，许多客户误以为生成式 AI 是“即插即用（Plug and Play）”的现成工具，但现实远非如此。交付过程中不仅要训练模型，更要培训使用模型的客户；必须在业务流程中嵌入自动化检查点（Checkpoints），并辅以人工复核（Human-in-the-loop）对最终交付结果进行精细化微调。同时，设置明确的成功阈值（Success Thresholds）与试点项目（Pilots）是验证模型质量是否达标的关键。 ### [26:49](https://www.youtube.com/watch?v=EVKg3Z0TCuU&t=1609s) 未来一年生成式 AI 模型定制的核心趋势预测 在进入观众问答环节前，主持人与各位嘉宾展望了一年内模型定制领域的发展方向： - **定制化小语言模型（SLMs）驱动 Agent 任务**：主持人认为行业将更聚焦于轻量级、针对特定 Agentic 任务深度定制的小型语言模型。 - **行业专用模型与开箱即用知识库接入**：Brent Blum 指出，行业特定模型以及具备自我定制能力的模型正在兴起，例如类似“Claude for Financial Services”等方案，通过 RAG 架构让企业即时连通彭博（Bloomberg）等垂类数据源。 - **动态实时定制（Real-time Customization）**：Raffaele Pascale 预测未来的解决方案将告别固定的预训练模型形态，转变为根据用户所在地区、使用语言、个人偏好及当前任务进行实时调整的动态环境。 - **定制化能力平民化（Democratization）**：Sue Shlapakovsky 提出，当前依赖技术团队进行 Prompt 工程或微调的模式终将遇到产能瓶颈，未来通过高度抽象简化的交互界面，一线业务人员将能自主定制自动化工作流。 - **多 Agent 治理与架构安全突破**：Patrick Gallic 强调，现有 RAG 架构在大规模应用时面临严峻的安全挑战，且受监管行业中的 Agent 权限管理体系难以扩展，未来亟需发展“用于监管 Agent 的 Agent（Agents to monitor other agents）”来实现合规治理。 ### [29:59](https://www.youtube.com/watch?v=EVKg3Z0TCuU&t=1799s) 智能体安全隐患与“以 Agent 治理 Agent”的人力困境 在微调与 RAG 系统的实际落地中，安全隔离机制常常无法顺畅传递到具体的智能体（Agent）层级。当大量企业数据被集中汇聚在一个共享位置时，原本各系统之间设定的安全防线若未能同步跟进，就会在安全合规层面形成巨大的隐患。理想的解决路径是建立更加坚固的安全体系，并借助“智能体监督智能体”（Agent-to-Agent Governance）的方式来进行自动化治理，从而避免抽调大量人力去做繁重的人工巡检与维护。 运维与管理规模的迅速膨胀是当前企业面临的核心风险之一。企业可以非常迅速地拉起一个模型并完成微调上线，但紧接着就必须分配专门的人力去持续监控它的运行状态。如果团队快速上线了 30 个不同的定制化模型，大部分技术人员就不得不被绑在这些日常运维工作上，导致待办任务（Backlog）不断积压。 ```mermaid flowchart TD accTitle: Agent Governance and Hybrid RAG Security subgraph MultiAgentGov["多智能体协作与治理"] E["执行智能体 (Executor)"] -->|"输出初稿"| R["审查智能体 (Reviewer)"] R -->|"评估与纠偏"| G["治理智能体 (Overseeing Agent)"] end subgraph HybridRAG["混合式 RAG 数据流与安全防护"] LocalData["本地私有数据 (On-Prem RAG)"] --> Filter{"出站过滤层 (Filter)"} Filter -->|"脱敏处理 / 阻断违规"| LLM["外部云端模型 (如 OpenAI)"] Filter -->|"安全查询"| LLM end ``` ### [30:49](https://www.youtube.com/watch?v=EVKg3Z0TCuU&t=1849s) 提问环节：强监管行业在端到端微调与多模型聚合时的安全担忧 现场进入 Q&A 互动环节，来自 Think GX 的 Alex Estrada 提出了第一个问题。他指出自己身处强监管的生物医药（Biotech）行业，对前端用户随意进行模型定制化感到担忧，尤其担心模型与底层“唯一事实来源”（Source of Truth）脱节，从而引发输出不可靠的问题。 Alex 分享了一个切身体验：作为非 IT 专业人员，他仅用一个下午就搭建了一个多语言翻译工作流；随后又花 5 分钟写了第二个流程，能同时调用 4 个大语言模型（LLMs）、汇聚它们的响应并自动分析交叉痛点与差异点。这种极低的开发门槛令他既感到兴奋，又对其潜在的验证与可靠性漏洞感到担忧。针对这一痛点，Argos Multilingual 的 Raffaele Pascale 指出，在语言服务领域通常会将不同特性的模型进行组合：让一个擅长生成的模型担任“执行者”，另一个模型担任“审查者”，两者相互协作以保障交付质量。同时，企业必须严格界定敏感数据的存储与暴露边界，避免在云端直接裸露生命科学或法务等高度敏感的数据。 Brent Blum 随后补充了一个来自 Turing 客户的混合式 RAG 落地案例： - 该客户采用混合架构：计算端调用 OpenAI，而核心 RAG 知识库完全保留在本地私有环境（On-Premise）。 - 系统在所有出站请求（Outbound Query）之前架设了专用的安全过滤层（Filter）。 - 该过滤器会自动审查所有发往外部模型的查询内容，一旦识别出敏感信息，会立即将其进行匿名化脱敏（Anonymize），或者直接拦截该查询以防外泄。 ### [35:08](https://www.youtube.com/watch?v=EVKg3Z0TCuU&t=2108s) 智能体的规模化落地、失控风险与集成瓶颈 来自 Aerospike 的 Sini 提出了关于智能体未来演进的疑问：传统机器学习（ML）推理早已在广告技术（AdTech）、反欺诈和推荐引擎中实现了大规模应用，那么基于生成式 AI 的智能体究竟何时能够实现同等规模的普及？人类又是否会在某个时间点“失去对这一切的控制”？ 现场嘉宾用幽默化解了关于失控的焦虑，随后深入探讨了智能体相比传统 RPA（机器人流程自动化）的技术跃迁。过去的 RPA 在企业内大规模铺开时暴露出极高的脆弱性，流程稍有变动就会报错失效；而多智能体协同（Agent Swarms）、思维链（Chain of Thought）以及 Agent-to-Agent 交互则展现出远超 RPA 的灵活性。与其只派一个员工处理单一任务，未来企业可以瞬间调动“上千个虚拟员工”并发探索所有解法，并提炼出最优方案。 然而，智能体大规模落地目前受限于底层工具调用与系统集成瓶颈： - 智能体必须依赖模型上下文协议（MCP）等机制来调用外部工具，目前工具与 MCP 的打通本质上仍是定制化的 API 集成。 - 这种集成工作与传统的 IT 实施项目（例如早年通过 MuleSoft 集成 Salesforce 系统）一样繁琐而痛苦。 - 只有突破传统 API 接口的笨重模式、探索更高效的连接方式，智能体的大规模落地才能真正提速。 Audible 的 Sue Shlapakovsky 强调，通过专业化智能体（Specialized Agents）、分层编排智能体（Orchestrated Agents）来承担原本无法顾及的任务，将为企业效能带来阶梯式的跃升（Step Function Improvement）；但这一切的前提是必须建立配套的监督机制，在扩展规模与保障质量、安全之间取得平衡。 ### [39:03](https://www.youtube.com/watch?v=EVKg3Z0TCuU&t=2343s) AI 治理国际标准 ISO/IEC 42001 的企业对标与结语 现场观众 Vinay 抛出了最后一个关于合规认证的问题：正如食品行业区分普通宣称与“USDA 有机认证”一样，业界都在宣称自己的 AI 负责、安全，但国际标准化组织已经推出了针对人工智能管理体系的新国际标准 ISO/IEC 42001（涵盖治理、风险、合规与安全）。他询问各位嘉宾所在的企业是否已经主动对标或通过了该标准。 - 嘉宾确认，其所在的部分业务团队与机构已经明确将自身 AI 体系对标 ISO/IEC 42001 进行基准测试与合规审查。 - 涉及标准认证业务的团队也已经跟进相关的 ISO 认证实施。 - Turing 等技术服务商目前已将 ISO 42001 的合规评估与对标作为一项专业服务向企业客户提供。 会议在对全体嘉宾和参会观众的致谢中正式圆满结束。 ## Notable quotes > [10:04](https://www.youtube.com/watch?v=EVKg3Z0TCuU&t=604s) Data can be a little bit like children. Everyone thinks theirs is special — Patrick Gallic > [07:42](https://www.youtube.com/watch?v=EVKg3Z0TCuU&t=462s) 人人都有手机，无法构成竞争壁垒 — Patrick Gallic > [17:30](https://www.youtube.com/watch?v=EVKg3Z0TCuU&t=1050s) 精通 AI — Patrick Gallic > [17:30](https://www.youtube.com/watch?v=EVKg3Z0TCuU&t=1050s) 哪怕将承保比例从 4% 提升至 5%，在业务体量上就意味着实现了 25% 的业务净增长 — Patrick Gallic > [20:36](https://www.youtube.com/watch?v=EVKg3Z0TCuU&t=1236s) 新入职的初级合规专员 — Brent Blum > [26:49](https://www.youtube.com/watch?v=EVKg3Z0TCuU&t=1609s) 用于监管 Agent 的 Agent — Patrick Gallic > [35:08](https://www.youtube.com/watch?v=EVKg3Z0TCuU&t=2108s) 专业化智能体、分层编排智能体来承担原本无法顾及的任务，将为企业效能带来阶梯式的跃升 — Sue Shlapakovsky > [01:55](https://www.youtube.com/watch?v=EVKg3Z0TCuU&t=115s) 将私有数据与 RAG 技术相结合，无需进行复杂的模型微调，即可达到理想的客户价值水平 — Sue Shlapakovsky ## Mentioned Sean Singer, Patrick Gallic, Sue Shlapakovsky, Raffaele Pascale, Brent Blum, Alex Estrada, Vinay, Sini, Comcast, Tokio Marine Group, Audible, Argos Multilingual, Turing Enterprises, Inc., OpenAI, Anthropic, Google, ChatGPT Projects, GPT-4, GPT-5, Cursor, Windsurf, Amazon Q, Maven, Bloomberg, Salesforce, MuleSoft, Aerospike, Think GX, RAG, MCP, ISO/IEC 42001, MS Office, Lotus Notes ## Video description Raffaele Pascale, Director of Product Innovation & Solutions, Argos Multilingual Sean Singer, Senior Director for AI Strategy and Transformation, Comcast Patrick Gallic, Head of International AI Hub, Tokio Marine Group Brent Blum, AI Solutioning Lead, Turing Enterprises, Inc. Sue Shlapakovsky, VP and Head of Product, Audible Abstracts This session will dive into the technical and strategic considerations for customizing generative AI models such as LLMs and multimodal models.to meet your organization’s unique requirements. Experts will explore how to fine-tune pre-trained models, incorporate proprietary data, and adapt AI solutions to industry-specific challenges. Panelists will also discuss common customization challenges, such as balancing accuracy with computational efficiency, and share insights on how to deliver impactful, tailored AI solutions that drive real business outcomes. [Source](https://www.youtube.com/watch?v=EVKg3Z0TCuU)

[10:04] Patrick Gallic: Data can be a little bit like children. Everyone thinks theirs is special [7:42] Patrick Gallic: 人人都有手机，无法构成竞争壁垒 [17:30] Patrick Gallic: 精通 AI [17:30] Patrick Gallic: 哪怕将承保比例从 4% 提升至 5%，在业务体量上就意味着实现了 25% 的业务净增长 [20:36] Brent Blum: 新入职的初级合规专员 [26:49] Patrick Gallic: 用于监管 Agent 的 Agent [35:08] Sue Shlapakovsky: 专业化智能体、分层编排智能体来承担原本无法顾及的任务，将为企业效能带来阶梯式的跃升 [1:55] Sue Shlapakovsky: 将私有数据与 RAG 技术相结合，无需进行复杂的模型微调，即可达到理想的客户价值水平

[0:04] Sean Singer: geek out for the next 45 minutes [1:55] Sue Shlapakovsky: BIA [1:55] Brent Blum: progressive scale [7:42] Brent Blum: crawl, walk, run [12:08] Sue Shlapakovsky: mind-bending psychological thriller under 10 hours [12:08] Sue Shlapakovsky: Maven [22:32] Sue Shlapakovsky: safe-to-fail [26:49] Small & Industry-Specific Models [26:49] Agentic Governance (Agents Monitoring Agents) [35:08] Brent Blum: We lose control in 2030 on March 3rd—no, I'm just joking, I'm from the future. [35:08] Brent Blum: agent swarms [39:03] Vinay: USDA Organic

[0:04] Good afternoon everyone. My name is Sean [0:06] Singer. I'm a senior director for AI [0:08] strategy and transformation at Comcast. [0:11] We're here to geek out for the next 45 [0:13] minutes on customizing Gen AI models for [0:16] business impact. We have an awesome [0:19] panel and I'm going to ask the panelists [0:21] to introduce themselves starting down at [0:23] the far end. [0:24] >> Sure. Good afternoon everyone. My name [0:26] is Patrick Alec. I lead AI uh for Tokyo [0:29] Marine, which is a the largest Japanese [0:31] insurance company that now operates over [0:33] a 100 companies throughout the world in [0:35] 45 countries. Um it's about a 70 billion [0:38] market cap, but um you know, we're [0:40] federated throughout the world. [0:47] >> Hi, my name is Slabakovski from Audible. [0:50] I lead the growth and content discovery [0:52] product management teams as well as the [0:53] applied science team at Audible. Um, so [0:56] my teams do everything from signing up. [0:58] So, excuse me. My teams do everything [1:00] from helping customers sign up for a [1:02] subscription all the way through helping [1:04] you find your next title. [1:06] >> Hi everybody, my name is Rafael Pascal. [1:09] I'm a director of product innovation and [1:11] solution at Argos Multilingual. We do uh [1:14] customized genai solutions in order to [1:17] uh help our clients to scale [1:19] multilingual operations or validate and [1:22] prepare data sets or even improve the [1:24] quality of their content. All these to [1:26] meet uh global requirements that [1:28] actually is not really possible with [1:30] existing solutions. [1:32] Hey everyone, I'm Brent Bloom with [1:34] Touring. Uh I'm the solutioning lead and [1:37] Touring is a small AI consultancy that [1:39] does two main things. Um, we train all [1:42] the AI models at the leading labs, [1:44] OpenAI, Anthropic, Google, etc. And then [1:47] we take those insights and we deploy [1:49] custom AI solutions for Fortune 1000 or [1:53] so. [1:55] Great. So, first question, the first [1:58] step that you have to take when [2:00] customizing a Genai model is deciding [2:03] that's the path you want to go down. And [2:06] so I'm wondering what are the different [2:08] criteria you think about in making that [2:11] decision and maybe we could start with [2:13] you Raphael. [2:15] >> Sure. So uh at Argus for us [2:17] customization becomes essential when [2:20] off-the-shelf solutions cannot really [2:22] meet uh client expectations in terms for [2:26] example linguistic accuracy or [2:29] contextual context like cultural context [2:32] within the content or also like uh [2:35] giving the brand alignment that they [2:37] that they need. So in especially when we [2:40] talk about less commonly spoken [2:42] languages for us customization is [2:44] essential because our clients regardless [2:48] if the language is spoken or not like [2:51] commonly or not uh the tone the the [2:54] terminology should be consistent for all [2:56] the translations for all the languages [2:59] and uh also like uh in terms of output [3:03] we need to be able to generate something [3:06] that is in line with the brand alignment [3:08] that actually they have all around the [3:10] world. So this is uh like um achieved [3:14] thanks to the fact that we ingest the [3:16] machine, we ingest the solutions with [3:18] all the style guides, reference [3:19] materials that we can collect from all [3:22] the languages and giving basically a not [3:24] really a translated content but a truly [3:27] local content generated from locals. [3:34] Sue, any thoughts there and how you're [3:37] approaching this topic at Audible and [3:39] thinking about the situations where you [3:40] want to customize? [3:42] >> Yeah, so for those of you who aren't [3:43] familiar with Audible, we are a leading [3:45] provider and producer of premium audio [3:47] storytelling content. So, um, we have [3:51] millions of customers around the world [3:52] and we deliver audiobooks, podcasts, and [3:54] original content. [3:56] So for us, we are actually thinking just [3:58] like many of you about how we use AI to [4:00] personalize the experience, which means [4:03] we are also thinking about what are the [4:05] trade-offs between customization and [4:07] non-customization. [4:09] So for a lot of the big foundational [4:11] models out there, the good news is they [4:12] are really good at a lot of things like [4:14] summarization and um sort of um natural [4:19] language understanding, which means for [4:20] a lot of use cases, they actually do a [4:22] good job out of the box. [4:24] you might need to add rag or you might [4:26] do some type of prompt engineering, but [4:28] for the most part they're pretty good. [4:29] Of course, there's a lot of use cases [4:30] like we were talking about Rafael where [4:32] you do need to do fine-tuning. Um, but [4:34] the good news is there's a lot of [4:36] spectrum out there in terms of the level [4:38] of customization that you need to do. So [4:40] for Audible, one thing that we like to [4:42] do, I use this framework called BIA, [4:45] value, investment and accuracy. So [4:47] basically um value being what is the [4:51] value that this thing delivers to the [4:53] customer, [4:55] investment being what is the incremental [4:56] cost to doing this customization and [4:59] then accuracy, what kind of domain [5:02] specificity do you need [5:05] and can you get away with just a custom [5:07] model out of the bench. And so for my [5:10] role in discovery, the goal is to help [5:13] customers find the perfect title. And so [5:16] when you think about applying that [5:17] framework, I think about value very high [5:21] because that is part of the core [5:23] promise. I think about investment also [5:26] very high because customizing is not [5:28] cheap. You have to maintain it. You have [5:30] to scale it. You have to customize it. [5:33] And then accuracy. What we actually [5:35] found was that when we combine our [5:38] proprietary data with rag, we actually [5:42] get a really good level of customer [5:45] value that we don't need to fine-tune. [5:47] So, at the end of the day, for us, it [5:48] was actually about finding that balance [5:51] and sort of figuring out how to balance [5:54] the operational efficiency with customer [5:56] value. [6:00] Brad, how do these types of [6:02] conversations look for you when you're [6:03] having them with your clients? [6:06] >> Yeah, I mean, I think there's almost a [6:08] default mode that says, well, our our [6:10] enterprise is unique. Like, we need to [6:12] customize. And to an extent, I want to [6:15] say yes, you do. But, but I have a [6:17] slightly different twist on that, which [6:19] is to say like I think even proper [6:21] prompt engineering is a form of [6:23] customization. I think using the [6:25] projects feature within chat GPT, you [6:28] know, where you can upload custom [6:29] instructions and documents and so forth, [6:32] that's a form of customization. And so [6:34] you can dip your toe in the water and [6:36] see if you're getting better results [6:37] than just the baseline LLM. And then [6:40] from there decide, okay, well, here's my [6:42] clearly defined business problem and [6:44] let's start to measure that against, you [6:46] know, doing a full-blown solution like [6:48] rag or a hybrid rag um or even doing [6:52] some fine-tuning. And so it's almost [6:53] like a progressive scale where you can [6:56] start to experiment, see more value, and [6:58] then get encouragement that like this is [7:00] working and I'm getting better results. [7:03] That's really insightful. I think about [7:05] this a little bit internally a lot. [7:07] Comcast, you know, nation's largest [7:09] provider of broadband internet, and our [7:12] customer interactions are such a core [7:14] aspect of everything we do. Uh, and [7:17] looking at an off-the-shelf model, being [7:19] able to assess those interactions for [7:21] quality versus the incremental [7:23] performance you might get from a [7:24] customized reward model is a really [7:28] illustrative exercise to go through. And [7:29] you can find ways to not only identify [7:33] opportunities to improve that customer [7:34] interaction, but inform product road [7:36] maps in new ways that you might not have [7:38] been able to with an off-the-shelf [7:41] model. [7:42] Patrick, I know part of this is also [7:44] knowing when not to customize and uh any [7:48] you had mentioned an experience you had [7:50] had at Tokyo Marine. [7:51] >> Yeah. Uh so I think just first off the [7:53] bat um the base model sort of out of the [7:56] gate are great, right? I probably spend [7:57] way too much money. I could probably [7:59] like finance a very nice car with how [8:00] much I'm spending across all my licenses [8:02] personally. Um but in terms of like [8:04] enterprise value, I think they're pretty [8:06] limiting. And the reason why is because [8:08] when we go and we talk to sort of like [8:10] the leaders of the business and we say, [8:11] "Hey, we want to use AI." What they're [8:13] asking for is where can we have like a [8:15] competitive differentiation um using AI [8:18] that other people don't have? And and so [8:21] sort of like those foundation models are [8:22] like a cell phone. Yeah, everyone has a [8:23] cell phone. That's great. You can give [8:24] it out, but that really doesn't matter. [8:26] What we need is we need to change our [8:28] core operations. And so not sort of like [8:29] our functional areas, but our core [8:31] operations. We need to bring our own [8:32] data so we can chat with it. we need to [8:34] connect our own tools around actuarial [8:36] models or other tools that we have. Um [8:38] and then we needed to behave in our own [8:40] ways, right? And so what we've learned [8:42] um and of course we're new on this [8:43] journey and I'm sure that you guys are [8:45] all probably have just as many lessons [8:47] learned as I do, but I'll just share [8:48] what what I've learned so far uh [8:50] painfully is that our investments around [8:53] customization are absolutely necessary. [8:55] Um but the way that we're thinking about [8:57] this is that we will invest where we the [9:00] discernable moat is our IP. So this is [9:03] on our data, our workflow or our [9:04] processes. And then what we need to do [9:06] is think about an interchangeable like [9:08] engine which is the model. And so the [9:10] model needs to be able to go in and out, [9:11] right? So there the two focuses number [9:13] one is that our investments are in those [9:15] areas. And because it's on that area is [9:17] what is it? It's not really on just [9:18] customizing one model. It's actually [9:20] creating the flywheel because we know [9:22] that every 3 to 6 months we need to [9:24] replace that model. Um and I think like [9:26] uh GPT5 I think illustrated to probably [9:29] many of you I know showed to me like we [9:31] have a problem. We have a problem. They [9:33] fundamentally changed how we access [9:35] models and the fact that there's a model [9:37] picker now means that from an enterprise [9:39] perspective that my workflows don't work [9:42] anymore especially in a regulated [9:43] industry. So like that's a huge issue [9:45] which which makes me more invested in [9:47] saying I need to focus on sort of my [9:50] workflow. Um and so I think the one [9:52] example we had talked about before was [9:54] um besides that is is us deciding we are [9:57] not going to invest in um non-core sort [10:01] of functional areas. And I say this [10:02] broadly um there's 100 group companies. [10:04] I think each one kind of has a little [10:06] bit of a spin but but generally what do [10:08] we mean by this? When AI first came out [10:10] um I was excited about the programming [10:12] capabilities and we said hey let's [10:13] invest a whole bunch of money and time [10:15] in sort of creating like programming um [10:18] for our programmers. And over time we [10:20] realized, you know what, this is much [10:21] better served by like a cursor and a [10:23] windale and Amazon Q, right? They're [10:25] investing sort of that the the ability [10:28] to make those functions work. Um, and so [10:31] that's not a great way for me to spend [10:33] sort of my time customizing the models [10:34] to work according to, you know, my [10:36] needs. So what we're doing is we're [10:38] focusing on really like what are the [10:39] core functional areas where we can have [10:41] a competitive differentiation. Um, and [10:44] that's that's where we're we're spending [10:45] our time. And Brad, you alluded to this [10:48] a little bit. You know, data can be a [10:50] little bit like children. Everyone [10:52] thinks theirs is special. Um was [10:55] wondering, you know, what what is it? [10:57] How do you spot that differentiation? [10:59] Patrick, you mentioned, you know, RIP [11:02] core to our business operations. How do [11:04] you identify [11:07] data that you're going to use to [11:08] customize? [11:09] >> Yeah, I mean, I think most enterprises [11:11] know when they've got something really [11:13] special. It's either, you know, data [11:14] that's been retained over the years, um, [11:17] or data that's harvested from, you know, [11:19] important client interactions. And, and [11:22] really with every time that we, uh, [11:24] consider customizing that data, it's [11:26] about benchmarking. We take a look, uh, [11:29] at a baseline LLM and then we take a [11:31] look with a little bit of customization. [11:33] And, um, you know, this is early. We're [11:36] still we're all kind of still learning. [11:38] Um, but with that sort of like crawl, [11:41] walk, run approach, you can determine if [11:44] the data in question is worth taking [11:45] another step with. [11:48] And Sue, I would I would think your user [11:51] data is probably a really rich source, [11:53] particularly when you're thinking about [11:55] recommendation algorithms. You know, a [11:57] lot about my reading habits as an [12:00] Audible customer. um is that one of the [12:03] places you look to as a real [12:05] differentiator for you in building new [12:07] experiences? [12:08] >> Yeah, so when we think about data, it's [12:10] a lot about how we can use this data to [12:13] improve the lives of our customers. And [12:15] so um we think about the different types [12:19] of data that we can use to inform [12:21] personalization. So we have deep content [12:24] understanding. So we go deeper than just [12:26] genre or metadata. We understand story [12:29] arcs. we understand um topics, tags. So [12:32] for example, it's not just a [12:34] psychological thriller. [12:36] It's a mindbending psychological [12:38] thriller under 10 hours for example. So [12:40] we can go level deeper. Then we have [12:42] customer behavioral data. So we [12:44] understand how customers listen, their [12:46] listening patterns, where they play, [12:48] where they pause, where they binge. Then [12:50] we have um sort of customer context, so [12:53] their likes, their dislikes, their [12:54] preferences. And we combine all of that [12:56] information to create a more [12:58] personalized experience for our [12:59] customers. Um, a good example of us [13:03] combining that data and making it real [13:04] is Maven, which is our AI powered [13:07] conversational search engine that we [13:08] launched late last year. And what Maven [13:11] does is it allows a customer to enter a [13:13] natural language query and then it [13:15] returns a set of personalized results. [13:18] And what is differentiated about Maven [13:20] is that we're not just using a base [13:22] model and spitting out a bunch of [13:23] results. We're taking the base model, [13:25] we're grounding it in those data sets [13:27] that we have. And then what pops out at [13:30] the other end is this unique, [13:32] personalized, relevant output that you [13:35] couldn't have done with just base model [13:37] alone. [13:44] Where are some of the pain points along [13:47] the process of customization that you [13:48] would call out to people in the [13:50] audience? things to look out for, you [13:53] know, processes that take more time or [13:55] resources than you might anticipate. [14:01] I I can start. I mean, some of that's [14:03] just around data governance, data [14:06] integrity, cleansing. Um there may even [14:08] be some internal I won't call them [14:10] battles, but conversations to be had uh [14:13] amongst different departments as to like [14:15] where the source of truth is. So many [14:18] times at least we're finding like [14:20] there's a series of steps that precede [14:22] what I'll what I'll call like the AI [14:24] work um that really is like almost half [14:28] the battle. Yeah. And I'll add to this. [14:31] I I think practically uh there's a lot [14:34] of challenges right to customizing the [14:36] data. I think part of this is that the [14:37] road map's not really clear. So where do [14:39] I need to fine-tune? Like if I'm looking [14:41] at a legal document and it has [14:42] strikethroughs right now the LMS cannot [14:45] read that, right? So you can't do like a [14:47] legal compare for your documents without [14:49] fine-tuning the model like that's just [14:51] like okay you need to know that then if [14:52] you say okay well we want to bring our [14:54] own data in some sort of rag the recall [14:56] and accuracy is not great like across [14:58] the board I mean there was a you know on [15:00] the keynote presentation it looks like [15:01] they're making advancements there and [15:03] the question becomes do we invest in [15:05] that advancement or do we sort of wait [15:06] for the industry to make that investment [15:09] and then we'll sort of just use them as [15:10] a product right are we too early in this [15:12] game should we wait should we not wait [15:14] um and so I It's really a matter of like [15:16] exploring each one of these use cases [15:18] and and trying to determine can we get [15:21] business value today, right? And if not, [15:24] do we invest so we solve this problem or [15:25] do we wait for sort of the industry to [15:27] solve that problem and then we'll be [15:28] like a follower once it's solved, right? [15:30] With sort of like a note saying we know [15:32] that this is an issue, right? For [15:34] submission ingestion, we need 99% [15:36] accuracy. It's 95. Great. Doesn't work [15:38] for me yet, right? Every new model that [15:40] comes out or every new solution, I can [15:41] quickly benchmark that. [15:43] should I spend those cycles or should [15:45] someone else spend those cycles? Um, I [15:47] think there's there's lots of challenges [15:48] and I think anyone that has gone through [15:49] the model customization knows it's not [15:52] like a clear blueprint of where to start [15:54] and where to end. [15:56] >> I think that's a great point around your [15:59] your risk level, your tolerance for [16:01] accuracy that might be lower than you [16:03] want, what your loss function is, right? [16:04] Because in your case, and I can think of [16:06] some on our end as well, you know, you [16:09] really need to get to that absolutely [16:11] high level of accuracy, like 90% is just [16:14] simply not going to cut it. And that's [16:16] not where a lot of the off-the-shelf [16:18] models are on some of these more [16:19] specialized use cases. [16:22] Any other experiences with some of the [16:23] the pain points in the customization [16:25] process? [16:27] Well, what we experience is that [16:29] customization is true that uh requires [16:32] time, resources of course uh and efforts [16:36] to train the new models or to train the [16:38] solutions with the new models. But once [16:41] you customize those solutions with [16:43] exactly uh you know the data coming from [16:46] the client that are differentiator for [16:48] us like those are truly differentiated [16:51] data like what comes directly from the [16:53] client. Then you see that the outputs [16:56] that you get. It's exactly what the [16:58] client expects and not just a generic [17:01] output that actually is can be whatever [17:04] like helpful just because it's an output [17:07] generated with those requests or [17:09] whatever. So it truly matters when [17:12] actually in the process quality together [17:15] with scalability are key points of the [17:18] entire program. So once you have that [17:21] your solution starts like having a value [17:24] and then being unique. [17:30] So in terms of you know building a [17:32] culture of experimentation, testing and [17:35] learning, [17:37] how are your organizations fostering [17:39] that? Right. Um what's been your [17:41] success? What's been successful for you [17:43] in driving a appetite for customization [17:49] experimentation and education about what [17:51] this approach can do and cannot do? [17:55] >> Do you want to start with that one? All [17:56] right. Um, so I think from like a a [17:59] first principal standpoint, I think of [18:01] when I when I started my career, [18:03] probably like many of you, uh, I had on [18:05] my resume like I'm proficient in MS [18:07] Office, right? And like maybe I think I [18:10] even had like Lotus notes in there as [18:11] well. Right. So I I think like one of [18:13] the the foundations is from a a training [18:15] standpoint and like how do we we gather [18:17] that is that we need everyone to be able [18:19] to put on the resumes like I'm [18:20] proficient in AI like that just that's [18:23] like a baseline functionality right and [18:25] then after that the question becomes [18:26] like well what does the adoption look [18:28] like? and the adoption looks like [18:30] probably starting off with you know just [18:31] like acceptable use and then training on [18:33] tools and then maybe some imagination [18:35] right um but it follows a pretty clear [18:38] path I think I think what's more [18:39] challenging what you're talking about [18:41] here is that from an executive [18:43] standpoint you know how do we get the [18:45] executives educated so that way they [18:47] know we need to make these investments [18:48] in AI it's not just magic right there is [18:51] a sense that like AI is cheap and it's [18:53] magical and like you can just plop it in [18:55] and it's going to work um and I think [18:57] really like if If you want to transform [18:59] the core parts of your business, what [19:01] you need to do is you need to think [19:02] about that whole process. So in [19:04] insurance, what we need to look at is [19:06] from an underwriting standpoint, are [19:08] there risks that we can now underwrite [19:10] that we couldn't underwrite before? Um, [19:12] we have lines of business in like the [19:13] non-mitted space. I'm not to tech out on [19:15] on insurance here, but like you might [19:17] only write 4% of the submissions that [19:19] come in, right? Um, by using AI, if you [19:22] can write 5% instead of 4%, that's a 25% [19:25] increase. Like that's huge. But in order [19:27] to do that, there's going to be [19:28] significant investments in customizing [19:30] your models and that becomes sort of [19:32] like a competitive differentiation [19:35] because there's a 25% increase as [19:37] opposed to just cost optimization. Um, [19:40] and so, you know, I I think when it [19:42] comes to training, there's those two [19:43] tracks. And on the executive track, it's [19:45] having them understand what is AI, what [19:48] is the art of the possible. I then [19:50] flipping it over and instead of saying [19:52] here's the technology, give me the [19:53] problem. I think it's saying what is [19:55] your business strategy and then based on [19:57] your business strategy how can I apply [19:59] AI um and I'll just give you like one [20:01] last example on that sort of like the [20:03] education where you customize um as an [20:06] international company it's very [20:07] different the challenges so over in our [20:09] Asia side really the challenges around [20:11] headcount and not having enough people [20:14] um and so they they have a need to be [20:15] able to optimize these processes because [20:18] there's just not enough people right and [20:20] so we need to optimize those processes [20:21] whereas in the US there's a lot more [20:23] diversity iversity and risk and what we [20:24] want to do is looking at like how do you [20:26] underwrite more right um so both of [20:28] those require like the education and [20:30] then the education will lead to hey why [20:32] we're customizing and then you can [20:34] customize [20:36] I can I can tell a quick story um we [20:39] built a chatbot kind of a co-pilot for a [20:43] financial services company recently and [20:45] the problem they were experiencing is [20:47] that their product managers would write [20:49] you come up with these great products [20:50] and everyone wants to use them super [20:52] interesting stuff, but then like the [20:54] compliance teams that they partner with [20:56] were in short supply and the product [20:59] managers aren't really expected to be [21:00] compliance experts. And so there was [21:02] this bottleneck and just generally this [21:04] kind of culture whereby like the product [21:07] managers were a little hesitant to put [21:09] forth new ideas and they never quite [21:10] knew what was compliant and what wasn't. [21:13] And so by implementing this chatbot, it [21:16] gave them uh, you know, a right-along [21:19] assistant to make sure that all the [21:22] PRDs, the product requirements documents [21:24] were going to be compliant. And of [21:26] course, it wasn't the last word. It [21:27] didn't go out the door that way, but it [21:29] took them 80 or 90% of the way there. [21:31] And so that they could continue the [21:34] process of evolving and shaping these [21:36] products. And when they got the time of [21:38] the compliance team, it was good to go. [21:40] And so that that culture was more about [21:42] like AI is the new employee. AI is the [21:45] new junior compliance officer. It still [21:48] reports to the head of compliance, [21:49] right? But um it was almost like there [21:52] was a new teammate on the team and that [21:53] was super helpful to them. [21:55] >> That's a great example. And I I think [21:58] for those of us who work in highly [22:00] regulated industries, there's certainly [22:01] a huge concern around customer privacy. [22:04] And you know, we take a lot of pride in [22:06] our commitment to that at Comcast. And [22:08] so thinking about one the governance [22:10] piece of it, but also the sandbox [22:13] environments you can create to allow [22:15] people to experiment, test, learn very [22:18] safely and quickly so that when you do [22:20] make these larger investments to go [22:22] forward with the customization approach, [22:24] you're able to do that with a degree of [22:25] confidence that you're going to see the [22:27] business impact that you promised when [22:29] you started. [22:31] >> Sue, I think you had something you want [22:32] to add. [22:32] >> Yeah. So at Audible, we actually have [22:34] kind of a three-pronged approach, kind [22:35] of like a pyramid. So the foundation is [22:37] all about culture and leadership. So our [22:41] leadership team was actually very bought [22:42] in very early on. So we have a genai [22:45] reimbursement program for anybody in the [22:47] organization who wants to take a course [22:48] in genai. Then we have these genai [22:51] showcases quarterly where employees [22:53] across the company who are doing PC's [22:55] can come and talk about it get that [22:56] visibility with leadership. That's kind [22:59] of your foundation. Then in the middle [23:01] are the processes. So we created a set [23:04] of very clear guidelines for internal [23:06] employees on how to use Genai safely [23:09] accounting for things like privacy and [23:11] security or the constraints of your [23:12] organization so that they could [23:14] experiment and test in a way that they [23:16] were safe to fail. Um we also evolved [23:18] the way we look at our KPIs. So rather [23:20] than being laser focused on business [23:22] metrics and outcomes, we actually pay [23:25] closer attention to input metrics. [23:27] That's an internal term for leading [23:28] indicators. So engagement, adoption, and [23:32] we're paying closer attention to [23:33] qualitative feedback from our customers [23:35] so we can give this team the space to [23:38] iterate and get it right cuz it's it's a [23:41] longer runway. Um, and it's important to [23:45] appreciate that because otherwise you're [23:46] putting the wrong incentives in front of [23:48] the team. So that's the second tier. And [23:50] then the last sort of like tip of the [23:52] pyramid is all about individual [23:53] empowerment. So I'm sure you guys all [23:55] have folks within your team who are [23:56] these AI enthusiasts in the [23:58] organization. Um, I have a few of those [24:00] on my team and they were the ones who [24:01] were sort of bottoms up doing the [24:03] workshops, doing the lunch and learns, [24:05] just creating the environment and [24:07] creating that culture of curiosity [24:08] around AI that I think is hard to push [24:11] top down. And so the combination of [24:13] those three things is really what drove [24:15] adoption and engagement of AI for us. So [24:18] it wasn't just about creating the [24:19] technology for the organization, it was [24:21] about creating the incentives and the [24:23] culture as well. [24:26] So if we might have some folks in the [24:28] audience who are exploring use cases [24:30] that would leverage customized models [24:33] but are concerned about some of the [24:34] risks could be model misalignment, [24:36] introduction of bias. Uh how would you [24:39] advise our attendees to think through [24:41] some of the risks involved with uh model [24:44] customization? [24:51] >> I I can start. Um, [24:54] you know, I would say that we go back to [24:56] our benchmarks again. Um, you can [24:59] certainly overtune or misune a model and [25:02] and you know, you have need to have [25:04] clean data going in. Make sure you have [25:05] the right data. Um, but if you've got it [25:08] benchmarked and you know what your you [25:10] know, clearly defined business goals [25:12] are, it can help mitigate some of that [25:14] risk. Um, just by ensuring that you you [25:17] know what the outcome you're looking for [25:19] is. And when you've, you know, used the [25:22] right data, gone through the right [25:24] tuning, you finally are seeing the right [25:26] results, like then you know you've [25:28] you've hit it right and you haven't gone [25:29] too far. [25:33] >> Well, yeah. Um actually uh this comes to [25:36] the same adoption basically where we [25:38] were mentioning before because uh many [25:42] most of the clients think that jai is [25:44] just a plugandplay tool that um you put [25:46] and you go but actually it's not the [25:49] case we wish but it's not yet the case. [25:51] So you always need to train uh basically [25:55] not only the model but also the clients [25:57] that are using those solutions that you [26:00] are developing together with also in [26:02] order to make sure that the solution is [26:04] working with embedded checkpoints that [26:07] actually check the quality for you uh [26:10] where actually humans maybe will be then [26:14] fine-tune even more the final outputs [26:17] before delivery or whatever. And uh um [26:21] lastly there are pilots that definitely [26:24] help you to understand if you are [26:26] getting what actually is the [26:28] expectations from the client together [26:30] with you know establishing um success [26:32] thresholds or uh whatever to uh [26:36] basically show that uh your solution is [26:39] giving the quality expected from uh uh [26:42] the requirements. ments. [26:49] Great. So, this will be the last [26:51] question before we open it up to [26:53] audience Q&A. So, when we're all back [26:55] here a year from now, um, and obviously [26:59] Genai, as we all know, is moving at [27:01] breakneck speed, what is the trend as [27:06] relates to model customization that [27:08] we'll be talking about? And I can [27:10] volunteer myself first. Um, I I I'll say [27:13] we'll be talking a lot more about [27:15] smaller language models being customized [27:18] to drive very specific agent tasks, [27:20] agentic tasks. [27:24] >> Go ahead. [27:25] >> I was I would add to that um probably [27:28] also industry specific language models [27:31] um or even models that start to [27:34] customize themselves or aid in the [27:36] customization process. Um, one of the [27:39] tools that was released interest uh [27:41] recently that I thought was interesting [27:42] was Claude for financial services where [27:45] you can instantly import all your like [27:47] Bloomberg data and all that very [27:49] financial services uh relevant [27:51] information. I shouldn't say import as [27:53] much as like it's almost a raglike [27:56] approach. And I think we'll start to see [27:58] more industries come up with those kinds [28:00] of solutions where existing knowledge [28:03] bases can be easily imported and can be [28:06] a part of um an existing AI strategy. [28:12] Well, I think that in a year from now we [28:14] will be talking about real time [28:17] customization of the solutions because [28:20] uh I mean imagine a user where actually [28:23] will be presented no more with a [28:26] pre-trained model but uh a dynamic [28:29] environment where actually adjusts uh to [28:33] don't know where the this user lives or [28:36] the region or the language that he [28:38] speaks or the task that needs to [28:41] perform. So it will be more something [28:44] real time I mean real time customized [28:47] like on the preferences of the user and [28:49] definitely it will open topics like [28:52] privacy control and quality of course [28:57] >> I can see a version of the world like [28:58] you Brent where we've democratized the [29:00] customization of AI. So like today you [29:03] need a technical team to customize [29:04] whether that's prompt engineering or [29:07] fine-tuning or anywhere on the spectrum [29:09] but that team will become a bottleneck [29:12] at some point. So imagine a world where [29:14] you have all this demand but you've [29:16] simplified the technology with [29:17] abstracted interfaces like you talked [29:19] about and now your front line is [29:20] empowered to customize the automated [29:23] workflows and things like that. I think [29:25] that will be powerful. [29:27] I'm going to say what I hope that we're [29:29] talking about which is um agents to [29:32] monitor other agents for governance um [29:35] because I don't think it's sustainable. [29:36] I think that um like you're saying in [29:38] the last question as we're developing [29:40] more and more of these solutions I love [29:41] this real time um there's security risks [29:44] there's there's legitimate big security [29:46] challenges around like the existing rag [29:48] architecture which make it not scalable. [29:50] Um I think also like agent permissions [29:53] right now are are not scalable in a lot [29:55] of areas if you want to customize it in [29:56] a regulated industry. Um the the access [29:59] for the security do not flow through to [30:00] the agents and on rag where there's like [30:02] a bunch of data is all colllocated now [30:05] in in one sort of spot without sort of [30:07] all the securities that you have in [30:08] between um is a huge problem that we're [30:10] seeing on the security side. And so I [30:12] hope that what we're talking about is we [30:14] have much more robust security. Um and [30:16] then we can use agents in order to [30:19] monitor the other agents to have [30:21] governance so that way we don't have to [30:23] deploy a certain number of our people um [30:25] to sort of maintain uh that that is a [30:28] huge risk that I see is that we can [30:30] easily spin up a model and we can [30:32] fine-tune it. We can get it operational. [30:34] I then I need to allocate staff to be [30:36] able to go and monitor that and make [30:37] sure it's working on an ongoing basis. [30:39] And if I can quickly run up like 30 [30:41] different, you know, customized models, [30:43] now I need to allocate a majority of my [30:45] staff towards that. And the backlog [30:46] keeps growing and growing. [30:49] >> Great. So, why don't we open it up to [30:51] the audience? If you have a question, [30:53] please raise your hand um and wait for [30:56] our MC to come around with the [30:58] microphone. [31:00] >> Please come out. [31:05] >> Be gentle. [31:06] >> Hi, thank you for your presentation. And [31:08] my name is Alex Estrada with Think GX. [31:10] Uh you guys bring up some bunch of great [31:13] points. I work in a very regulated [31:15] industry uh biotech and the front end [31:19] user customization scares me and and the [31:22] points that you make about the [31:24] reliability to the source of truth. I [31:26] think at the granular level that's where [31:29] the problem is going to be because [31:30] everybody wants to use AI and uh as I [31:33] said in a different uh uh workshop [31:36] people who know in intuitively know the [31:39] business are going to know where those [31:41] pin potential pain points are. My [31:44] question is what how do you see the [31:48] security the validatable information [31:50] coming from these agents and definitely [31:52] like the idea of agents overseeing other [31:55] agents. I I may I created a workflow to [31:58] do language um translation in an [32:02] afternoon and I am not an IT person by [32:05] any stretch. And I wrote a second one [32:08] that could go out to four LLMs, converge [32:10] the responses, analyze them and give me [32:12] the pain or the the pivot points on all [32:15] four responses. It took five minutes. [32:18] And so this scares me because I think [32:20] it's both exciting but creating a lot of [32:22] dangerous points for for validation and [32:24] reliability. Thank you. [32:28] Anyone want to take a stab at that? [32:34] >> Well, uh I can speak about uh like our [32:38] experiences because uh we actually face [32:41] uh many issues in terms of quality [32:44] output especially the languages. So the [32:47] combination of the models actually helps [32:50] a lot to give back what actually is the [32:53] expected output because you leverage [32:56] basically skills of each LLM. So one LLM [32:59] is a very good executor while another [33:02] LLM is a very good reviewer. So you want [33:05] basically this mix and this combo and [33:08] the security of course stands on the way [33:10] that you then store those data. if you [33:13] use something opened in the server or if [33:16] you use your own version of the model. [33:19] So of course it's always depend then on [33:22] which kind of level security we are [33:25] talking about like if there are [33:27] information that can be shared outside [33:29] or maybe use only part of them in order [33:32] to still leverage the information but [33:35] not give too much outside and still have [33:38] like the results that you want from the [33:41] requirements again from what is [33:43] expected. But yeah, security is uh very [33:47] like uh close point to us and uh al also [33:52] talking about languages uh you you touch [33:55] basically any kind of topics from life [33:57] science, legal and many other topics [34:00] that are very sensitive and in this case [34:03] you always try to not leverage something [34:06] that is totally exposed uh let's say on [34:10] the provider that is giving you this LLM [34:12] but definitely [34:14] uh in half enough I want to say so in [34:17] this case you can still be sure that um [34:20] the data that you are exposing are not [34:23] too much uh exposed again [34:27] >> I'll add to that briefly um we have a [34:30] client that implemented a hybrid rag [34:33] solution so they they use open AAI uh [34:36] but they keep the rag on prem and the [34:38] wrinkle that I thought was interesting [34:40] was they've added a filter that checks [34:43] every outbound query to make sure that [34:45] you know the rag's not sending something [34:47] it shouldn't and that filter either then [34:50] anonymizes the information or prevents [34:52] the query from going out at all. Um that [34:56] that approach may help additionally. [35:00] >> Any other questions? [35:08] >> Yeah. Hi uh my name is Sini from [35:10] Aerospike. Um I have some familiarity [35:12] with some of the I'm not an expert in AI [35:14] by any means but um ML inferencing [35:16] usually is is done at scale you know [35:18] it's used at adtech fraud detection [35:20] recommendation engines and so on and [35:22] then gen of course has taken the world [35:24] by storm and now there is aentic um so [35:27] my main question to you is like we have [35:30] certain kinds of algorithms that work at [35:32] scale uh when do you think um something [35:36] like agents will just take over and then [35:39] uh they will also be at scale. And the [35:42] second part of the question is when do [35:44] we lose control of all of this? [35:47] >> Well, question is up for vocation. [35:49] >> Go to tomorrow morning's keynote. [35:52] >> We lose control in 2030 on March 3rd. [35:55] No, I'm just joking. I'm from the [35:57] future. Um, [36:02] I think agents are the most promising [36:03] thing for me. I think uh that's that's [36:05] the most exciting part of sort of my my [36:08] work. Um, and I think because [36:12] because the idea is that we want to be [36:14] able to to work within our own data and [36:16] actually take action in order to move [36:18] things forward. Agents allow us to do [36:20] this. I I was a big proponent of like [36:22] RPA um when it first started off and [36:24] there was a lot of excitement around RPA [36:26] and it quickly was too fragile and [36:28] brittle and it just didn't work right. [36:30] And I think anyone that has sort of done [36:32] this realizes that like it worked to a [36:34] certain extent and then it stopped. Um, [36:36] agents are are super exciting. Um and I [36:39] think that when you actually go from the [36:40] agent to agents actually becomes more [36:42] exciting because then you can put out [36:43] like agent swarms, you can do chain of [36:44] thought, you can do there's so many um [36:47] possibilities. I think once you go to [36:49] the agent to agent model which which [36:51] really accelerates it the the biggest [36:52] dependency I see um and and not to get [36:55] like too technical here is that uh the [36:57] agents are going to require like MCPS in [36:59] order to call your tools and right now [37:01] that tool to MCP is an integration. And [37:04] if you don't have an API or if you're [37:06] trying to do that integration, it looks [37:08] like every other IT project. Um, and so [37:11] it reminds me very much if anyone has [37:13] done like a like a Salesforce [37:15] implementation and you went through like [37:16] Mulesoft or something like that. It it's [37:18] like a painful process. Um, the hope is [37:21] that maybe there's a better way for us [37:23] to do that instead of just like an API [37:25] integration. Um, maybe there's other [37:26] ways that we can integrate. I think from [37:28] a consumer side there's a there's broad [37:29] applicability. Um, but I think the real [37:32] value is going to be in agents and more [37:35] specifically like the agentto agent [37:37] where instead of you just launching like [37:39] one employee at a task, you can launch [37:40] like a thousand employees and they can [37:42] look at every single possibility and [37:44] then come back with the best solution. [37:47] It's it's a great point and I your [37:49] reference to RPA triggered something for [37:51] me which is thinking back to that period [37:54] and the proliferation across an [37:57] enterprise of different RPA solutions [38:00] and the struggle to get visibility [38:02] across all of those and what that can [38:04] mean. So it like I share your excitement [38:07] certainly about the potential for agents [38:10] running throughout an organization's [38:12] operations, but getting that level of [38:14] visibility necessary to feel confident [38:17] in what they are doing, the actions [38:19] they're taking, and the outcomes of [38:21] those actions is going to be an [38:23] important part of that process. Sue, I [38:26] think you wrote it. [38:26] >> I was just going to say like plus one to [38:28] what you said. To me, I think in order [38:30] to scale the organization and bring [38:33] above the line all the other things that [38:35] you want to get done that you can't get [38:36] done today, the world of specialized [38:38] agents, agents of agents, orchestrated [38:41] agents, I think that is what is going to [38:43] step function improve the impact of the [38:44] organization. But then balancing that [38:47] with the right mechanisms to oversee and [38:50] maintain the quality and the security is [38:52] where the balance is going to come in. [38:56] >> I think we have time for one more [38:57] question. [39:03] Thank you. Uh my name is Vinay have have [39:08] u you know ISO 4201 is the new global [39:12] standard for um AI and it talks about [39:16] governance risk compliance security and [39:19] so forth. Have any of you benchmarked [39:22] your AI against ISO 4201? Everybody [39:25] talks about oh our AI is responsible [39:28] it's safe we look at security but it's [39:30] like saying my product is organic versus [39:33] my product is USDA organic right so have [39:37] you maybe you don't get yourself [39:39] certified but have you at least [39:40] benchmarked yourself against ISO 4201 or [39:44] thought about it thank you [39:47] >> yeah in in short in short yeah some of [39:50] our groups are are absolutely [39:51] benchmarking against [39:55] short. We do ISO certification for the [39:58] 4001 as well. Yes. [40:00] >> Yes. [40:02] >> Touring offers that as a service. Um I [40:04] couldn't I couldn't personally comment [40:06] on it. [40:10] >> Well, great. Thank you everyone for [40:12] attending. Uh it's been a great session. [40:14] Thank you to our panelists for doing [40:16] such a wonderful job. Enjoy the rest of [40:18] the conference.