Vanguard, Domino's Shift AI to Agent Support Over Raw Call Deflection

Vanguard 与 Domino's 将 AI 策略从单纯通话分流转向赋能人工客服 企业发现激进的 AI 分流会损害客户信任，因而将切实的投资回报转向自动化一线行政工作流，以赋能人工团队。 Vanguard 与 Domino's 的企业领袖联合 Assembled 和 LevelAI 的 CX 平台创始人，共同探讨了生成式 AI 如何超越激进的客户分流，转向赋能一线人工客服。专家小组详细阐述了自动化“围绕核心工作的辅助事务”（如通话后摘要、结构化 CRM 记录和升级转派）如何在为复杂咨询保障人工支持质量的同时，带来直接且高价值的投资回报。演讲者强调，企业的成功取决于构建 API 与数据就绪能力、摆脱平均处理时长（Average Handle Time）等短视的效率指标，并建立大型评估框架。最后，针对团队如何适应基础模型的持续更迭，他们权衡了专有 API 与开源模型在实际应用中的取舍。 企业对客服 AI 的认知已从完全取代人工转向人机协同模式，前线落地最快产生 ROI 的场景往往是通话自动总结、工单生成等管理型任务，而非直接替换人工对话。 (0:04) 底层基础大模型的快速跃升使传统机器学习流水线相形见绌，但企业落地的核心瓶颈仍集中在基础数据治理与 API 集成等系统就绪度上。 (3:45) 过度追求平均处理时间（AHT）或单纯指标拦截率易削弱客户信任，企业正转向涵盖 CSAT、长期客户情绪及内部采纳率的综合评估体系。 (19:28) 所谓“Level 5”客户体验自动化并非追求 100% 自动分流，而是在人机互学、严密评测与合规护栏下，守住品牌信誉与深度客户连接。 (24:49)

Enterprise leaders and technology providers report that the primary return on investment in AI customer operations has shifted from full human replacement toward frontline enablement and internal workflow automation. However, capturing durable value requires overcoming foundational API bottlenecks, redesigning operational metrics away from blunt deflection quotas, and maintaining rigorous evaluation infrastructure to manage brand risk. Frontline Enablement and Automating Administrative Overhead (7:03) While public attention often centers on consumer-facing generative bots, enterprise panelists emphasize that internal frontline augmentation delivers the most immediate, predictable business value. Jing Wang, leader of Vanguard's Enterprise AI Center of Excellence, explains that assisting financial advisors with automated call summarization reduces post-call administrative logging by 20 to 30 minutes following a 45-to-60-minute consultation, while simultaneously synthesizing six months of client relationship history to prep agents before interactions. Ryan Wang, CEO of Assembled, reinforces this dynamic, characterizing the core opportunity as automating "the work around the work." For example, support platform Canva achieved a 15% increase in cases solved per hour largely by automating routine administrative wrap-up tasks, such as translating unstructured customer issues into rigidly formatted Jira bug reports for engineering teams. Ashish Nagar, CEO of LevelAI, adds that enterprises see the highest return with the least friction by implementing assistive intelligence and workflow automation layers before pursuing fully autonomous deflection. Re-evaluating Metrics: Beyond Handle Time and Deflection (19:28) Pursuing aggressive deflection rates often creates user friction and damages customer sentiment. Bryce Macher, VP of Decision Science at Domino's, observes that widespread exposure to low-quality, automated touchpoints risks turning human customer service into a scarce "luxury experience." To protect long-term customer lifetime value, Domino's tracks sentiment metrics such as trust, happiness, and delight alongside standard operational indicators. Internally, optimizing solely for speed can distort employee behavior. Jing Wang notes that Vanguard initially targeted Average Handle Time (AHT) reductions when rolling out agent copilots, but the metric backfired: representatives rushed complex financial consultations to hit time targets. Recognizing that over 50% of calls contain multi-intent inquiries requiring deep discovery, Vanguard shifted toward holistic measures like CSAT. Nagar similarly defines advanced customer service automation not as a 100% automated contact center, but as a symbiotic ecosystem where human agents and AI copilots continuously collaborate. In operational practice, Ryan Wang points out that companies like Ramp evaluate AI success primarily by internal agent adoption rather than theoretical ROI models. Structural Bottlenecks: Data Readiness and Model Infrastructure (13:15) AI deployment is frequently constrained by legacy backend readiness rather than model capability. Nagar highlights an "API before AI" bottleneck across large enterprises, where ambitious automation plans are stalled by multi-year engineering backlogs required to expose backend systems to AI agents. Ryan Wang notes that at the most basic level, companies still struggle because operational knowledge lives informally in workers' heads rather than in structured documentation. To address this, Vanguard actively rewrites public and internal content specifically to optimize ingestion for large language models and reduce hallucinations. Architectural choices also influence enterprise economics. Nagar argues that relying on proprietary closed APIs becomes unsustainable at scale due to latency, token pricing, domain personalization, and data privacy limits when organizations process hundreds of millions of interactions. Consequently, LevelAI focuses on custom, open-source model pipelines to maintain throughput and reduce per-token tolls. Concurrently, rapid foundation model innovation—yielding benchmark gains of 20% and 10x cost drops out of the box—is breaking traditional multi-year procurement cycles, compelling organizations to re-evaluate their tech stacks every few months. Enterprise Governance, Evaluation Pipelines, and Scaling Traps (34:36) Deploying generative AI in sensitive consumer contexts requires resolving output accuracy before investing in massive scale. Macher cautions that engineering teams frequently make the mistake of debating infrastructure scaling before solving the "last mile" of reliable output. Ryan Wang cautions against the "vibe coding" trap: while modern generative tools make it easy to build an initial 80% prototype, the remaining 20%—consisting of reliable evaluations, data pipelines, and production scaling—demands the deepest engineering rigor. In heavily regulated settings, high general accuracy is insufficient. Jing Wang points out that Vanguard historically rejected customer-facing models operating at 97% accuracy because a 3% failure rate poses unacceptable financial risk. To deploy safely, Vanguard enforces eight internal Trusted AI standards, runs massive simulated evaluations across query distributions, conducts phased pilots, and partners with third-party monitoring vendors for real-time output interception. She highlights an ongoing industry gap: software vendors often aggressively promote generative features while leaving the burden of designing evaluation tooling entirely to enterprise buyers. As consumer search behavior expands into conversational engines and underlying foundation models turn over rapidly, panelists argue that enterprises must abandon static multi-year software contracts in favor of flexible, vertical team alignments capable of continuous model re-evaluation and adaptation.

企业在客户体验（CX）领域的AI应用正在经历深刻转向：从早期全面替代人工客服的激进构想，演进为以一线赋能与人机协同为核心的务实落地。来自 Vanguard、Domino's、Assembled 与 LevelAI 的业内高管一致指出，当前最显著的价值创造并非直接面向客户生成对话，而是消除“工作周边的冗余流程”，并通过重构底层数据、评估体系与服务指标，确保品牌信任不被粗劣的自动化侵蚀。 赋能一线与消除工作周边冗余 (7:03) 企业应用生成式AI的最直接价值并非让AI直接生成客服回复并交由人工纠错，而是赋能一线团队，自动化繁重的辅助性行政任务。Vanguard 负责管理超11万亿美元资产及5000万全球投资者，其企业AI卓越中心负责人 Jing Wang 指出，内部一线赋能是最大的价值驱动力。理财顾问通常需要进行45至60分钟的深度咨询，过去每次通话后需耗费20至30分钟手动整理记录；通过引入AI自动化摘要提取结构化事实，并整合过去6个月的客户互动历史，系统能在拨入前提供情绪与背景简报，大幅释放顾问精力。 Assembled 联合创始人兼 CEO Ryan Wang 同样强调，应聚焦于消除“工作周边的冗余流程”。例如其客户 Canva 在使用 Assembled 自动生成符合工程严格格式规范的 Jira 故障单并消除客服收尾工作后，每小时解决工单量提升了15%。这一趋势反映出行业认知的演变：从早期的“全盘自动化替代”转向混合式的人机协作。 衡量指标重构与人际连接溢价 (19:28) 单纯追求效率指标（如平均处理时间 AHT 或单向分流率）往往会导致客户体验恶化。Jing Wang 介绍，Vanguard 最初将缩短 AHT 作为评估坐席辅助工具的核心基准，但这导致员工因担心超时而匆忙挂断电话；由于管理个人资产涉及深层复杂的财务需求，且超50%的呼入包含多重意图，Vanguard 迅速调整策略，摒弃单一的 AHT 目标，转而将客户满意度（CSAT）等全流程体验作为核心指标。 Domino's 决策科学副总裁 Bryce Macher 援引 Mark Cuban 的观点指出，随着低劣自动化交互泛滥，高质量的人工支持正在变为“奢侈体验”，企业必须依靠快乐度、信任感等领先情绪指标维护客户终身价值。面对管理层对70%高分流率的狂热预期，Ryan Wang 提及金融科技公司 Ramp 将运营与AI领导权合二为一，直接以坐席采纳率而非复杂的 ROI 模型作为衡量标准；LevelAI 创始人 Ashish Nagar 则指出，最高阶的“Level 5 自动化”并非零人工介入，而是人机共生协作，这正是 Apple、Disney 等卓越品牌维系长期忠诚度的根基。 架构断层、API瓶颈与自建陷阱 (13:15) 企业部署生成式AI的首要技术瓶颈并非模型能力本身，而是后端数据的可用性。Ashish Nagar 提出由浅入深的四层落地框架（智能洞察、工作流自动化、人机协作者、全面自动化），并强调“在AI之前先有 API”（API before AI），许多大型企业受阻于长达两年的工程积压与陈旧系统。Jing Wang 补充道，数据就绪是一项持续工作，例如 Vanguard 重新改写了公共网站与内部文档的文案，以便大模型无幻觉地检索与理解。 针对近期盛行的“氛围编程”（vibe coding）与全栈自建风潮，Ryan Wang 警告原型开发仅占前80%，而评估流水线（evals）、高可用数据流以及应对极端情况等非核心工程构成了最困难的20%。在底层模型选择上，Ashish Nagar 指出在处理数亿至数十亿数据点时，直接依赖商业闭源 API 难以维持两秒内的企业级低延迟与经济性；企业应依据性能、价格、个性化与隐私（4P原则）评估开源定制架构，并警惕传统软件厂商在陈旧架构上“粉饰AI”所签订的多年期锁定合同。 严格风控基建与全生命周期交互 (34:36) 在高合规与高风险场景中，AI的精准度直接关系到企业存亡。Jing Wang 透露，在金融管理领域，即便模型准确率达到97%，剩余3%的错误率仍是不可接受的财务风险。为此 Vanguard 建立了由“八项可信AI标准”、海量模拟评估、受控试点实验以及实时防护监控拦截构成的治理闭环。她特别指出，外部供应商往往只销售模型能力而将评估责任完全甩给采购方，迫使企业必须自主建立完备的内部评测基建。 与此同时，消费交互模式正在发生外部迁移。Bryce Macher 强调在扩大底层基建规模前必须先攻克“最后一公里”的准确性瓶颈，并关注从“出生到坟墓”的完整消费者生命周期。随着用户日常使用 Claude、Grok 或 Perplexity 等工具，企业不仅需应对传统搜索，还必须布局生成式引擎优化（GEO）、代理式下单工作流，并监控用户在外部对话平台中的抱怨信号，将其融入全局客户心声策略之中。 该研讨表明，企业AI在客户服务领域的成熟度不再取决于模型参数规模或激进的分流承诺，而取决于能否将技术沉稳嵌入端到端业务流。然而，基础模型供应商格局的高频动荡、遗留系统 API 与数据治理的沉重负债，以及第三方厂商评测基建缺位所引发的自建成本，仍是企业在全面规模化生成式AI之前必须化解的现实瓶颈。

## Background 本内容由 Ai4 于 2025 年 8 月 28 日发布在 YouTube 平台上。视频记录了一场关于“利用 AI 重塑客户体验”的圆桌研讨，先锋领航（Vanguard）、达美乐（Domino's）、Assembled 和 LevelAI 的行业专家共同探讨了如何通过智能体、自动化与大模型提升服务效率并优化客户参与度。 ## TL;DR 圆桌讨论围绕生成式 AI 在客户体验（CX）与联络中心的实际落地展开，嘉宾们一致认为 AI 当前最大的价值在于赋能一线员工消除辅助工作，而非简单粗暴地取代人类。企业在推进 AI 时普遍面临“API 先于 AI”、数据治理及指标体系失真等现实瓶颈，过度追求分流率和压缩平均处理时间（AHT）往往会伤害品牌信任。与会高管强调，企业应采取分层推进策略，构建严格的评测防线与敏捷的架构体系，在人机共生中实现长期的商业与品牌价值。 ## Atmosphere & style 整场研讨会专业且务实，主持人的风趣穿插与四位来自甲乙方高管的坦诚分享形成了极佳的互动张力。现场既有对传统大厂工程排期与遗留系统痛点的辛辣调侃，也有对大模型时代盲目“自建”与虚假数字触点的清醒反思。讨论节奏紧凑、干货密集，在探讨技术前沿的同时始终牢牢锚定真实的商业回报与组织现实，呈现出一种理性克制却又充满洞见的实战派氛围。 ## Key takeaways - [07:03](https://www.youtube.com/watch?v=RwSeeVqA7-0&t=423s) 客户体验（CX）中应用 AI 的最大变革并非直接替代人工，而是解决“围绕工作的辅助工作”（work around the work），通过通话即时摘要、跨周期关系回顾与跨部门工单自动填充来赋能一线员工。 - [10:10](https://www.youtube.com/watch?v=RwSeeVqA7-0&t=610s) 面对底层大模型快速迭代与成本骤降，技术团队的核心竞争力在于工程架构的高效“代谢与吸收（Metabolize）”能力，即通过标准化评测链路在午餐时间内快速验证新模型带来的提效与降本。 - [11:06](https://www.youtube.com/watch?v=RwSeeVqA7-0&t=666s) 生成式 AI 重构了传统组织架构，推动企业走向“垂直对齐（Vertical Alignment）”，将底层算法基建与业务洞察角色整合至同一业务汇报线下以缩短反馈闭环。 - [13:15](https://www.youtube.com/watch?v=RwSeeVqA7-0&t=795s) 客户服务落地 AI 遵循四层推进框架（智能层、工作流自动化、协同辅助、完全自动化），企业真正的实施瓶颈往往不是算法，而是遗留系统中缺乏标准接口的“API 先于 AI”困境。 - [16:11](https://www.youtube.com/watch?v=RwSeeVqA7-0&t=971s) 基础准备工作往往处于更原始的“文档化”阶段，企业不仅要先把口口相传的业务规则写下来，还需要主动清洗并重构语义友好的“AI 就绪数据（AI-Ready Data）”。 - [17:30](https://www.youtube.com/watch?v=RwSeeVqA7-0&t=1050s) 单纯以降本和分流率作为考核指标极易制造虚假的数字触点，当消费者对低质 AI 交互产生疲劳时，真实的人际互动反而成为高价值体验，指标设计必须向“信任”与“幸福感”等全流程体验倾斜。 - [19:28](https://www.youtube.com/watch?v=RwSeeVqA7-0&t=1168s) Vanguard 等企业对考核机制进行反思，指出盲目追求 0% 转接率与极低的平均处理时间（AHT）会引发坐席恐惧、忽视深层财务需求，应以端到端体验与 CSAT 代替单一效率指标。 - [24:34](https://www.youtube.com/watch?v=RwSeeVqA7-0&t=1474s) 客服领域的最高境界并非自动驾驶式的“L5 级全自动化”，而是 AI 与人工坐席各司其职的人机共生（Symbiotic）状态，过度的全自动化反而会瓦解品牌的长期忠诚度。 - [29:04](https://www.youtube.com/watch?v=RwSeeVqA7-0&t=1744s) 大模型系统必须首先突破“最后一公里”的精准响应验证；很多企业出于过度的品牌风险担忧而将转人工敏感度调得过高，过早陷入庞大架构自建的隐形维护泥潭。 - [32:28](https://www.youtube.com/watch?v=RwSeeVqA7-0&t=1948s) 2020-2021 年是系统架构的分水岭，企业若继续在老旧传统架构上“外挂生成式 AI”将失去竞争力，应打破 5 年长约限制，转向数月一次的敏捷评估机制。 - [35:10](https://www.youtube.com/watch?v=RwSeeVqA7-0&t=2110s) 金融等高监管行业面对生成式 AI 幻觉设立了严苛的准入门槛，即使 97% 的准确率也因可能带来直接损失而不可接受，倒逼企业建立包含八项标准、海量模拟测试与实时阻断的三层防御体系。 - [40:37](https://www.youtube.com/watch?v=RwSeeVqA7-0&t=2437s) AI 时代品牌触达逻辑正在从传统搜索引擎优化（SEO）延伸至生成式引擎优化（GEO），企业需要前瞻布局智能体订购交互以及公域 AI 平台中的客户心声捕捉。 - [42:35](https://www.youtube.com/watch?v=RwSeeVqA7-0&t=2555s) 企业级 AI 落地需权衡“性能、价格、个性化、数据隐私”（4P 模型），面对数亿级并发与严苛延迟，开源模型与专属自研架构展现出比单纯调用商业闭源 API 显著更高的 ROI 优势。 - [44:28](https://www.youtube.com/watch?v=RwSeeVqA7-0&t=2668s) 成熟企业的 AI 投资组合策略应当将确定性极高、ROI 显著的后端流程自动化作为压舱石，同时保持对前端多模态交互的实验创新，并持续夯实底层数据治理基建。 ## Detailed walkthrough ### [00:04](https://www.youtube.com/watch?v=RwSeeVqA7-0&t=4s) 圆桌开场与主持人介绍 圆桌讨论聚焦于人工智能（AI）、客户体验（CX）与客户服务的交汇点。主持人 Andrew Seaman（LinkedIn News 特约编辑）在开场时指出，本次研讨汇聚了来自企业内部实践者与外部技术供应商的双重视角。他风趣地强调，虽然技术被称为“人工智能”（Artificial Intelligence），但面对客户体验时绝不能显得虚假或敷衍（"you can't be artificial when it comes to customer experience"），否则整个客户体验都会被毁掉。 --- ### [00:51](https://www.youtube.com/watch?v=RwSeeVqA7-0&t=51s) 嘉宾阵容介绍：从传统机器学习演进到大模型时代 四位嘉宾依次分享了自己的职业背景、公司定位以及对客户体验演进的洞察： - **Ryan Wang（Assembled 联合创始人兼 CEO）**：Assembled 专注于打造支撑客户支持运营的 AI 平台，涵盖自动化率超过 70% 的语音/聊天 AI 智能体，以及覆盖 20 到 20,000 人规模呼叫中心的劳动力管理（WFM）系统（排班、预测等）。Ryan 回顾道，自己早在“大家还称之为 ML（机器学习）而非 AI 的年代”就进入该领域，曾在 Stripe 仅有 80 人规模时负责基于 ML 的反欺诈检测，随后转向客户支持 ML。他指出，行业预期已迅速从早期的“用自动化取代所有人”，演变为通过人机协作（Humans and AI）实现更深思熟虑的方案。 - **Bryce Macher（Domino's 决策科学副总裁）**：Bryce 拥有语言学背景，经历从学术界到流媒体、酒店业再到快餐连锁（QSR）达美乐的跨界转型。达美乐将消费者研究、数据分析、面向消费者的处方性 ML 功能以及客户声音（VoC）呼叫中心分析统一归入决策科学（Decision Science）体系。Bryce 幽默地调侃，自己见证了该领域从“语言计算”、“计算语言学”、“自然语言处理（NLP）”再到 GPT-2.5 发布的演变，感叹自己“踩准了市场风口，真希望自己的投资组合也能这么准”。此外，他目前高度关注下一代“感官计算与感官 AI”（sensory computing / sensory AI）的多模态潜力。 - **Ashish Nagar（Level AI 创始人兼 CEO）**：Level AI 是一家端到端 AI 与 CX 平台，服务包括十余家财富 500 强在内的全球品牌。产品线从面向人工客服的自动化质检（QA）、全渠道非结构化数据客户声音洞察、客服 Co-pilot，扩展到全自动化的语音和聊天 Agent。Ashish 曾是亚马逊 Alexa 团队的产品负责人。他分享了一段生动的故事：在 2019 年，为了让 Alexa 能够与人类就社交话题持续对话 20 分钟，亚马逊投入了 7 个不同团队与多个 PhD 项目去分头攻坚；而如今在 LLM 面前，当年的传统架构已被彻底取代。 - **Jing Wang（Vanguard 企业 AI 卓越中心负责人）**：先锋领航（Vanguard）作为全球最大的共同基金与资产管理机构之一，管理资产规模（AUM）刚刚突破 11 万亿美元，服务全球约 5,000 万投资者。Jing 在 Vanguard 工作 15 年，其中近 10 年致力于构建数据科学、ML 与 AI 的应用体系。其团队支持所有分销业务与企业服务职能，运用 AI 提升客户体验、运营效率并驱动增长。 ```mermaid flowchart TD accTitle: AI in CX Evolution Architecture accDescription: Shows the transition from traditional ML methods to unified Generative AI applications in contact centers. subgraph Evolution["技术演进历史"] T1["传统机器学习 / 计算语言学<br>(多团队离散攻坚)"] --> T2["大语言模型与多模态<br>(端到端一体化生成)"] end subgraph CX_Applications["核心落地场景"] T2 --> A1["全自动智能体 (Agent)<br>处理 70%+ 常见交互"] T2 --> A2["人机协作 Co-pilot<br>实时辅助与知识召回"] T2 --> A3["运营后台质检与 WFM<br>跨周期全景洞察 / 自动归档"] end ``` --- ### [07:03](https://www.youtube.com/watch?v=RwSeeVqA7-0&t=423s) AI 带来的最大变革：赋能一线员工与“围绕工作的辅助工作” 当主持人问及过去数年到十年间 AI 带来的最深刻变革时，嘉宾们一致指出：**最大的价值爆发点并非直接取代人类，而是深度赋能一线人员，解决繁琐的“衍生辅助工作”（work around the work）。** Jing Wang 指出，尽管外界对直接面向客户的生成式 AI 充满热情，但 Vanguard 目前看到最大业务回报的领域在于赋能一线顾问（Advisors）与客户经理（Relationship Managers）： 1. **通话即时摘要与自动归档**：投资顾问与客户的一通深度沟通往往长达 45 至 60 分钟。以往顾问在通话结束后需要花费 20 到 30 分钟进行整理和归档，而 AI 通话摘要工具直接省去了这部分时间，实现了标准、高质量的关键信息沉淀。 2. **跨越 6 个月的全周期关系回顾**：系统不仅摘要单次通话，更会回溯过去半年的所有客户互动历史，为顾问生成整体客户关系摘要，使其在下一次沟通前迅速掌握客户核心诉求与最佳沟通切入点。 Ryan Wang 同样认同这一观点。他强调，AI 带来的效能提升往往不是“让 AI 替我写回复、我稍作修改就发出”，而是消除工作流程中的各种结构性阻碍： - **Canva 效率提升案例**：Canva 借助此类工作流优化，将客服团队每小时解决工单数（cases solved per hour）提升了 15%。 - **跨部门协同模板自动填充**：在技术支持场景中，当客服需要向工程团队提交 Bug 时，必须严格遵循 Jira 模版格式，稍有不合就会被工程团队驳回。通过让 AI 提取对话信息并严格按模板生成 Jira 工单，极大减少了沟通摩擦与重复劳动。 - **核心数据与事实汇总**： - **Vanguard 规模**：管理资产总规模突破 11 万亿美元，服务全球 5,000 万投资者。 - **顾问单次通话提效**：45–60 分钟的客户交流后，AI 摘要为一线顾问节省 20–30 分钟的整理归档时间。 - **Assembled 自动化指标**：AI Agent 能够自主处理超过 70% 的支持问题；助力 Canva 客服每小时解决工单率提升 15%。 - **Alexa 2019 vs 现状**：2019 年需要 7 个独立团队攻关的 20 分钟多轮对话技术栈，如今已全部被现代 LLM 架构所取代。 ### [10:10](https://www.youtube.com/watch?v=RwSeeVqA7-0&t=610s) 大模型快速迭代与工程架构的高效吸收 Ryan Wang 分享了 Assembled 在大语言模型迭代中的切身体验。AI 不仅能够快速生成标准问候语，还能严格遵循品牌的风格指南（Style Guide）与品牌基调（Brand Voice），大幅节省人工撰写与校对的时间。更关键的是底层技术的演进速度——从 GPT-4 到最新模型（如讨论中提及的 GPT-5 级别模型），模型的能力不断增强，而使用成本却呈现断崖式下跌。 为了抓住这种技术红利，Assembled 在工程组织架构上投入了巨大精力，确保团队能够快速“消化吸收（Metabolize）”外部模型的升级变更。Ryan 举例提到，当新模型刚发布时，团队在午饭还没吃完的时间里，就已经将其跑通了内部的评测集（Evals）。在未经任何微调（No Tuning）的原始状态下，基准评测效果直接提升了 20%，成本降低了近 10 倍，并且支持查询缓存（Query Caching）。这种底层算力与模型的跨越式突破，让搭建在平台之上的 AI 协同副驾（Co-pilot）和自主智能体（Agent）瞬间变得更加强大。 ```mermaid flowchart LR accTitle: 模型迭代与工程吸收链路 subgraph Model["前沿模型能力跃升"] A["新一代基座模型发布"] --> B["成本下降 10 倍 / 支持缓存"] A --> C["基准评测 (Evals) 提升 20%"] end subgraph Eng["工程吸收架构"] B & C --> D["标准化评测与集成链路"] end subgraph Product["上层产品赋能"] D --> E["AI 协同副驾 (Co-pilot)"] D --> F["自主服务智能体 (Agent)"] end ``` --- ### [11:06](https://www.youtube.com/watch?v=RwSeeVqA7-0&t=666s) 生成式 AI 带来的组织架构重组与垂直整合 随着技术的非线性突破，传统企业的组织架构正在受到猛烈冲击。主持人提出疑问：当企业引入 AI 时，往往会发现内部有两三个团队在做重复的事情，或者向不同的业务条线汇报，管理者该如何理顺这种组织阵痛？ Domino's（达美乐）决策科学副总裁 Bryce Macher 指出，技术能力的飞跃必然会带来组织形态的波动期。未来最显著的趋势将是“垂直对齐（Vertical Alignment）”——将整个客户旅程（包括底层技术基础设施与偏向洞察输出的业务角色）整合在同一个组织架构下，以便紧贴所支持的业务领域快速行动。例如，在达美乐，Bryce 直接向首席营销官（CMO）汇报。在疫情前这或许显得不可思议，但在生成式 AI 时代，那些“既懂深度技术又通晓业务语言、能直接驱动商业结果”的人才，正处于最核心的加速位上。组织架构究竟该以技术为主导还是营销为主导，取决于企业自身的基因，而且随着生成式 AI 重构工作流，这一组织架构探索仍将是一个长期的动态过程。 - **组织变动的核心驱动**：模型能力的阶跃式发展导致业务可能性不断被重新定义。 - **垂直整合的核心优势**：缩短技术基建与商业落地之间的反馈周期，让业务与算法在一个部门内闭环。 - **复合型人才的价值**：兼具深厚技术实力与商业对话能力的跨界角色成为生成式 AI 落地的主力军。 --- ### [13:15](https://www.youtube.com/watch?v=RwSeeVqA7-0&t=795s) 生成式 AI 落地的四层框架与“API 先于 AI” 在平衡“客户对极速响应的诉求”与“人性化关怀（Personal Touch）”的矛盾时，LevelAI 首席执行官 Ashish Nagar 提出了企业在客户服务中落地生成式 AI 的“四层推进框架”，并指出行业普遍陷入了过早追求“全自动化”的误区。 ```mermaid flowchart TD accTitle: 生成式 AI 落地四层演进框架 subgraph Layer1["第一层：智能层 (Intelligence)"] L1["分析洞察 / 意图识别 / 极低落地阻力"] end subgraph Layer2["第二层：工作流自动化 (Workflow Automation)"] L2["通话摘要 / QA 质检自动化"] end subgraph Layer3["第三层：协同增强 (Co-pilots)"] L3["坐席辅助 / 知识推荐"] end subgraph Layer4["第四层：全自主自动化 (Full Automation)"] L4["端到端自主处理 / 极高实施复杂度"] end Layer1 --> Layer2 --> Layer3 --> Layer4 ``` Ashish 强调，企业推行 AI 时面临的最大壁垒并非算法本身，而是系统集成与数据接口。他幽默地表示：“如果每次走进大型企业，听到他们说‘我们喜欢你们的 AI 路线图，但你们能不能先帮我们把 API 路线图搞定，因为我们工程排期已经堆积了两年’，我都能拿到一毛钱的话，我现在早就发财了。”底层的核心数据往往深嵌在古老的传统系统里，若缺少能与 AI 对话的 API，企业就只能做表面化的摘要工作。最健康的路径是先从阻力小、ROI 高的智能层切入，再用 3 到 12 个月的时间逐步向协同副驾与全自动化推进。 - **四层落地路径**： 1. **智能层（Intelligence）**：投入产出比最高、落地阻力最小，侧重于数据洞察与分析。 2. **工作流自动化（Workflow Automation）**：如会话摘要、服务质检（QA）自动化。 3. **协同辅助（Co-pilots）**：为人赋能，辅助坐席实时检索与应对。 4. **完全自动化（Full Automation）**：由 AI 自主端到端闭环解决问题，对底层 API 依赖极深。 - **核心痛点“API before AI”**：绝大多数大型企业的数据资产无法通过标准 API 暴露给大模型，导致两年的工程负债成为 AI 落地的真正瓶颈。 --- ### [16:11](https://www.youtube.com/watch?v=RwSeeVqA7-0&t=971s) 基础准备的认知进阶：文档化与“AI-Ready 数据” 针对 Ashish 提出的接口问题，Ryan 进一步补充道：“API before AI”其实已经是“第二阶段（Level 2）”的问题了；在许多企业里，“第一阶段（Level 1）”往往更加原始——那就是“先把业务规则写下来（Just write stuff down）”。很多客户急着要搞自动化，但一问他们想自动执行什么业务政策，才发现公司的政策甚至都没有文字文档，一切流程都停留在不同员工的脑子和口口相传里。 先锋领航（Vanguard）的 Jing Wang 随后分享了甲方的实战认知：企业的演进是一个迭代过程，不可能等到所有 API 和文档都完美无瑕了才开始。当下企业面临的新课题是重新定义什么是“AI-Ready 数据（适合 AI 消费的数据）”——它与传统意义上的“优质数据”完全不同。为了让大模型更好地理解并提升输出准确率，先锋领航甚至开展了重写网站内容等专项举措，以结构化、语义友好的方式重新组织企业知识库。整个支持 AI 的生态系统一直在飞速演化，企业必须在建设中摸索迭代。 --- ### [17:30](https://www.youtube.com/watch?v=RwSeeVqA7-0&t=1050s) 商业目标重新校准：虚假数字触点下的指标设计 当企业管理层高喊全面拥抱 AI、追求降本时，指标（KPI）体系的设立变得至关重要。Bryce 指出，AI 时代的指标设计包含两个维度：一个是企业传统的度量设计（区分前导指标与滞后指标），另一个是更加关键的“语境变化（Contextual Element）”。 Bryce 引用了马克·库班（Mark Cuban）的一条引发热烈反响的推文：生成式 AI 的滥用正在制造海量“虚假的数字触点（Inauthentic Digital Touch Points）”。当消费者在日常生活中被各种冷冰冰的 AI 交互包围轰炸、感到筋疲力尽时，未来最高端、最具奢华感的客户体验反而是“能够与一个真实的人交流”。 更严峻的是，当员工在生活中受够了虚假 AI，上班打卡后依然不得不整天对着 AI 工作时，内外部的情绪反馈会直接改变原本的业务平衡。因此，企业绝不能仅仅将“降低成本”视作核心 KPI，而必须重塑衡量“信任（Trust）”、“客户愉悦度（Delight）”以及“幸福感（Happiness）”等体验维度的指标，因为这些软性指标才是决定客户终身价值（LTV）与业务健康度的核心前导指标。 ### [19:28](https://www.youtube.com/watch?v=RwSeeVqA7-0&t=1168s) 内部员工指标与先锋领航（Vanguard）的全流程客户体验重构 讨论首先聚焦于内部指标的量化问题。随着 AI 工具的深入应用，企业不仅要衡量客户，还必须设法量化员工的敬业度（Employee Engagement），比如员工在工作中是否感到快乐，以及这种情绪如何与生产力和效率加速挂钩。如果一味追求冷冰冰的数字和虚假的互动体验，工作场所很容易演变成令人窒息的反乌托邦——员工每天打卡上班只是为了机械地完成类似“TPS 报告”一样的琐碎任务。 主持人随后转向先锋领航（Vanguard）的 Jing Wang，询问她在引入 AI 时如何重新评估指标，并重新培训人工客服不必急于挂断电话。Jing 解释称，先锋领航始终坚持以客户为中心，衡量客户体验必须考量客户与企业互动的整个端到端旅程。在客服中心的实际场景中，团队利用 AI 构建了 IVR（交互式语音应答）路由模型和能够自动应答的机器人（Bot）；当问题无法由机器解决时，再无缝转接给人工客服。为了评估全流程满意度，他们最初将“转接率”（Transfer Rate）作为衡量 IVR 表现的核心指标，认为分流做得越准转接越少。但他们很快意识到，超过 50% 的客户来电抱有多重意图（Multi-intent），绝不可能把转接率降到 0%，盲目追求 0% 转接率反而背离了以客户为中心的初衷。 更典型的偏差发生在人工服务环节。团队最初为人工坐席配备 AI 辅助工具时，把考核目标定为削减平均处理时间（AHT, Average Handle Time），以此判断工具的好坏。然而，过度聚焦 AHT 让客服人员产生了恐惧心理，导致大家只想着赶紧应付完当下的通话、跳到下一个客户。先锋领航处理的是客户的财务人生（Financial Lives），很多时候更长、更深入的交谈才能真正发掘客户的深层需求并提供有力帮助。认识到这一点后，他们重新评估了考核机制，明确 AHT 绝不能作为唯一指标，转而引入了 CSAT（客户满意度）等更全面的全流程体验衡量标准。 ```mermaid flowchart TD accTitle: Vanguard 端到端客户旅程与指标评估流 subgraph IVR_Bot["机器自助阶段"] A["客户多重意图来电"] --> B["AI IVR 智能路由与 Bot"] B -- 解决常规问题 --> C["自助完成服务"] end subgraph Human_Agent["人工服务阶段"] B -- 复杂/未解决问题转接 --> D["人工客服接听"] D --> E["AI 辅助工具赋能"] E --> F["深入挖掘客户财务需求"] end subgraph Metric_Shift["指标体系转变"] G["单一指标<br/>(追求 0% 转接率 & 压缩 AHT)"] -.->|反思纠偏| H["综合指标<br/>(端到端全流程体验 & CSAT)"] end ``` --- ### [22:22](https://www.youtube.com/watch?v=RwSeeVqA7-0&t=1342s) 客户分流陷阱与 Ramp 公司的纯“采纳度”驱动模式 Assembled 首席执行官 Ryan Wang 谈到了向企业级客户推销 AI 解决方案时的现实博弈。他打趣道，现场的嘉宾态度都很理性温和，但现实中去见高管和投资人时，大家聊的都是硬核的 ROI。客户服务领域一直存在一个公开的秘密——“大家都在想方设法进行呼叫分流（Deflection）”。供应商经常吹嘘能用 AI 自动化分流 70% 的服务单，但站在用户的角度，没人愿意被当成皮球踢走，大家遇到问题时的第一反应往往是：“别废话，马上给我转人工！” 因此，管理层、客服负责人与投资者之间始终在拉扯：既想最大化分流、降本增效，又不想牺牲客户体验。Ryan 观察到，最成熟的 AI 买家往往是那些自己深度使用这些工具、对 AI 的能力边界有清晰认知的人。他以客户 Ramp 为例：该公司的用户运营负责人同时兼任 AI 负责人，他们在考核 AI 落地成效时，既不看生产力提升幅度，也不看投资回报率（ROI），而是单纯聚焦于“工具采纳度”（Adoption）。 这一做法打破了 Ryan 的固有认知。因为 Ramp 的负责人亲自大量使用了 Assembled 的产品，对 AI 适合嵌入哪些具体业务工作流了如指掌。他们从定性角度完全明白这能带来巨大价值，因此根本不想耗费数据科学团队的大量时间去搞前置和后置的 A/B 测试对比，他们的唯一目标就是：“让更多人更快地用上它”。这种沟通让 AI 供应商从费力说服客户转变为直接协助客户加速落地。 - 客服软件领域常以自动化分流 70% 服务单作为核心 ROI 卖点，但极易引发消费者反感。 - Ramp 将用户运营主管与 AI 主管职责合并，以内部使用经验直接指导采购。 - 对成熟团队而言，避免繁琐的 A/B 测试度量，将核心 KPI 设定为“采纳度”（Adoption），反而能大幅加快组织赋能进程。 --- ### [24:34](https://www.youtube.com/watch?v=RwSeeVqA7-0&t=1474s) 客服自动化的“L5 级”迷思：人机共生与品牌持久价值 LevelAI 首席执行官 Ashish Nagar 对此深表赞同。他指出，许多项目虽然带着自动化、控本或销售转化的初衷立项，但向上汇报到执行管理层时，核心指标终究会回到 CSAT、客户体验或 NPS（净推荐值），而盲目的自动化与优质体验往往是互相冲突的。 针对业内热议的自动化层级，Ashish 提出了一个发人深省的类比：在自动驾驶领域，最高级别的自动化（L5）是车辆完全不需要人类干预、实现百分之百自主驾驶；但在客户服务领域，最高境界绝非 100% 的全自动化。客服领域的“L5 级自动化”应该是一种人机共生（Symbiotic）状态——AI 与人工坐席相互学习、实时协同，AI 明确自身最擅长自动化的场景，客户体验团队也清楚人类最能发挥温度的环节。尽管硅谷和科技媒体喜欢宣传 90% 甚至 99% 的全自动处理，这在 TechCrunch 上看起来很吸睛，但这绝不是现实中企业应有的服务形态，也不是消费者期望被对待的方式。 世界上最顶级的品牌之所以基业长青，核心就在于其无可替代的客户体验。消费者对苹果（Apple）、迪士尼（Disney）或 Hulu 的忠诚与深刻记忆，正是来自于有温度的连接，而这种长久的品牌价值绝不可能单纯依靠全自动化来实现。 --- ### [26:46](https://www.youtube.com/watch?v=RwSeeVqA7-0&t=1606s) 达美乐的风险权衡：从智能披萨追踪器到 GenAI 落地边界 主持人随后切入品牌形象与风险控制话题，调侃起自己在大学时期对达美乐披萨的痴迷，不仅怀念经典的“555 特惠交易”（555 deal），更对达美乐 App 上那个实时步进的披萨追踪器（Pizza Tracker）记忆犹新。面对生成式 AI 可能带来的幻觉（Hallucination）、品牌调性不一致等问题，企业该如何设立安全防护栏？ 达美乐的 Bryce Macher 回应称，早期的披萨追踪器其实就是一个包含机器学习能力的智能项目，它绝非凭空瞎猜；但从安全角度来看，这种体验几乎不存在任何风险，因此决策非常容易推进。达美乐作为一个受全球消费者信赖的成熟品牌，首要任务是挖掘消费者的真实痛点，并选用合适的工具去解决，随后再评估技术手段带来的潜在风险、危害以及对品牌形象的冲击。 Bryce 直言，达美乐目前的实践表明，消费者真正需要的解决方案往往并不需要过度复杂的尖端 AI 架构。现阶段行之有效的创新方案大多边界清晰、受控度高，既能扎实驱动业务增长，又不会背负生成式 AI 脱轨失控的高风险。对于面向消费者的财富 500 强企业而言，随着生成式 AI 走向成熟，最大的挑战其实是“从哪里起步”。在整个数据科学领域，大家往往有一种思维定势，觉得构建模型才是最硬核、最困难的环节，但如何平衡商业创新与品牌安全防护，才是更为复杂的系统性考验。 ### [29:04](https://www.youtube.com/watch?v=RwSeeVqA7-0&t=1744s) 最后一公里与自建还是购买的权衡 在构建基于大语言模型的系统时，无论采用 RAG（检索增强生成）、模型微调（Fine-tuning）还是自主研发的模型，让 API 给出精准无误的响应始终是最困难的一关。很多数据科学家在遇到困难时容易本末倒置，急于去探讨系统的工程扩展性与架构规模，却忽略了一个根本前提：如果“最后一公里”（Final Mile）的业务效果和准确性没有打通，后端搭建再宏大的工程架构也毫无意义。因此，企业必须认清这一现实：在“先买后建”或“先买后定制”的路线选择中，核心在于明确如何测试风险边界与消费者收益，甚至要做好某些早期探索最终会被推翻重写的心理准备，而不是过早沉迷于底层基础设施的扩展性。 针对企业如何权衡自建与购买以及管理数据风险的提问，Assembled 联合创始人兼 CEO Ryan Wang 分享了两个核心观察。首先，许多企业在保护品牌形象上显得过于保守。例如某家支付公司的语音 AI 表现非常出色，能够解答绝大多数常规问题，但他们将转人工客服的敏感度调到了极致——只要用户一提到类似“我想找人”，系统就立刻无条件转接，哪怕人工客服给出的回答与 AI 别无二致。Ryan 调侃道，大众根本记不住此前新闻里哪家航空公司因为聊天机器人乱给退款承诺而上了头条，品牌风险并没有想象中那么致命，企业其实可以更大胆一些，让 AI 去真正解决问题。 另一方面，目前业内兴起了诸如“氛围编码”（Vibe Coding）等一股盲目追求自建的热潮。甚至有客户向 Ryan 宣称要用 Vibe Coding 徒手搓一套劳动力管理系统（WFM），对此 Ryan 幽默地回应：“祝你们好运，大概一年后再见吧，到时候我们再来帮你们收拾烂摊子。”AI 与软件系统的难点从不在于做 Demo 或完成初始原型的 80%，而在于如何将其推向数百万乃至数千万用户规模，以及背后的数据治理、全面评测（Evals）等看不见的重活。企业必须理性决定把宝贵的“创新筹码”（Innovation Chips）押注在何处。 - **核心矛盾**：没有验证“最后一公里”的精准响应，任何底层架构的规模化扩展都是无用功。 - **品牌风险的误区**：企业往往因过度担忧品牌声誉而将转人工敏感度拉满，抑制了 AI 真正发挥效能的潜力。 - **自建的隐形成本**：演示（Demo）极其容易，但支撑海量并发、数据清洗与严格评测才是真正的工程深水区。 --- ### [32:28](https://www.youtube.com/watch?v=RwSeeVqA7-0&t=1948s) 淘汰旧系统与在 LLM 时代敏捷采购 LevelAI CEO Ashish Nagar 深入探讨了企业遗留系统与长周期合同的问题。他指出，2020至2021年是技术架构的天然分水岭：在此之前或2010年代诞生的老旧传统软件，绝大多数只是在原先的陈旧底座上“撒上一层生成式 AI 的胡椒粉”（Sprinkling GenAI on top），根本不具备以 AI 为核心的原生架构。在客户体验（CX）领域，企业若继续守着这些旧系统，每一天都在流失市场竞争力，因为现代消费者能够清晰分辨出企业是否真正合理地运用了 AI，这已经演变为品牌的核心差异化优势。 谈到软件采购模式，Ashish 甚至开玩笑说自己的销售团队听到这番话可能会抓狂：企业应该每隔几个月就对供应商进行一次重新评估，而不是被长达五年的长期合同所套牢。底层大语言模型每周都在迭代突破，技术基准（Benchmarks）日新月异，而行业分析机构却依然沿用每季度评选一次象限、每年发布一次报告的陈旧套路。面对飞速变化的技术，企业应当将长期死板合同先放在一边，从低风险的生成式 AI 辅助场景切入，逐步从人工增强过渡到全面自动化。 ```mermaid flowchart TD accTitle: AI 实施与演进路径 accDescription: 展示企业如何从低风险辅助场景演进到全自动化的实施路径 subgraph LegacyVsModern["系统架构选型"] A["传统遗留系统<br/>（表面叠加 GenAI）"] -.->|淘汰/搁置| B["原生 AI 架构<br/>（以大模型为核心）"] end subgraph AdoptionPath["落地与演进路径"] B --> C["阶段一：低风险场景切入<br/>（辅助与增强现有工作流）"] C --> D["阶段二：人机协同与流程增强<br/>（Augmentation）"] D --> E["阶段三：成熟场景全自动化<br/>（Full Automation）"] end subgraph EvaluationCycle["评估机制"] F["摒弃 5 年死板长约"] --> G["建立数月一次的敏捷评估体系"] G --> B end ``` - **代际差异**：2020年前后是分水岭，传统系统仅是外挂 GenAI，缺乏原生技术基因。 - **采购模式重塑**：大模型每周更新，企业对技术供应商的考评周期应缩短至数月，以适应技术迭代。 - **推进策略**：以低风险用例切入，经历“辅助增强 $\rightarrow$ 流程重塑 $\rightarrow$ 全面自动化”的三步走演进。 --- ### [35:10](https://www.youtube.com/watch?v=RwSeeVqA7-0&t=2110s) Vanguard 的高压监管应对、八项标准与评测防线 先锋领航集团（Vanguard）的 Jing Wang 阐述了资产管理巨头在合规与品牌信任双重高压下的 AI 治理实践。Vanguard 处于极高监管力度的金融服务业，客户往往将全部身家托付于此，品牌声誉就是企业最大的生命线。面对生成式 AI 天生的“幻觉”风险，Vanguard 设立了极为严苛的准入门槛。早先他们曾坚决否决过准确率已达 97% 的方案，因为在金融投资场景下，那被误导的 3% 客户可能会因此错失入市交易的时机并蒙受财产损失。 为在利用技术红利的同时有效防御风险，Vanguard 全面建立了企业级 AI 治理体系。他们发布了明确定义“可信 AI 系统”的八项标准（Eight Standards），所有拟上线的 AI 方案必须达标。同时，他们构建了支持海量数据模拟的自动化评测工具，在上线前通过小范围试点（A/B Testing / Pilot）收集真实洞察，并在运行阶段引入强效的实时监控（Real-time Monitoring），一旦拦截到有害或错误交互便立刻阻断。Jing 还指出了当前市场的一个显著脱节：许多外部 AI 供应商只热衷于堆砌功能特性，却懒得提供严密客观的评测工具，甚至在被问及准确率时将评测包袱甩回给企业客户，这也倒逼像 Vanguard 这样的企业必须自建强大的防守生态系统。 - **97% 准确率困境**：普通场景下极高的准确率在金融投资领域依然意味着 3% 客户的直接利益受损，不容妥协。 - **八项可信 AI 标准**：作为全公司 AI 方案必须强制满足的治理基线。 - **三层防御生态**： 1. **海量模拟评测**：针对大模型概率系统开展大规模合成与仿真测试。 2. **小步灰度试点**：通过严格的实验机制追踪业务成效。 3. **实时拦截监控**：线上逐笔交互全量监控，实时阻断异常输出。 - **供应商脱节现状**：多数商业供应商缺乏完备的评估体系，企业必须具备自主评测与监控鉴别能力。 ### [39:15](https://www.youtube.com/watch?v=RwSeeVqA7-0&t=2355s) 技术栈快速迭代与供应商敏捷应对 在被问及未来六个月最关注的行业变化时，Assembled 联合创始人兼 CEO Ryan Wang 首先从技术提供商（Vendor）的视角分享了他的切身体会。他指出，当前 AI 演进的迅猛节奏给供应商带来了巨大的商业与技术交付压力。许多客户出于对技术更新换代的担忧，开始倾向于签订按月履约的短期合同，这对于需要时间沉淀和交付实际成果的供应商而言极为痛苦，Ryan 半开玩笑地呼吁企业“请对供应商好一点，至少签一年期合同，因为实现成效需要时间”。 ```mermaid flowchart TD accTitle: 供应商面对技术迭代的演进策略 subgraph S1["传统架构模式"] A["固定云基础设施 (如 AWS)"] --> B["长期技术选型锁定"] end subgraph S2["生成式 AI 时代挑战"] C["技术生态剧烈变动 (如语音 AI 供应商洗牌)"] --> D["预测未来失灵"] end subgraph S3["敏捷应对机制"] E["高频代谢变化能力"] --> F["新模型/提供商快速接入"] F --> G["超高敏捷响应客户需求"] end ``` 在底层技术架构的选择上，Ryan 强调了过去与现在的巨大差异。以 AI 语音代理（AI Voice Agent）为例，六个月前整个行业格局看似十分明朗，几家融资规模巨大、声势浩大的明星企业成为了首选，但押注之后的结果却并不理想，整个技术版图迅速被颠覆重构。过去企业可以简单地决定“全盘基于 AWS 构建”，而如今没有任何人能够精准预测未来的技术走向。企业唯一可行的路径是提升组织“代谢变化”的能力：当底层大模型迭代、新供应商涌现或客户需求发生转变时，团队必须能够以极高的敏捷度做出响应与调整。 --- ### [40:37](https://www.youtube.com/watch?v=RwSeeVqA7-0&t=2437s) 全生命周期交互重塑与生成式引擎优化（GEO） Domino's 决策科学副总裁 Bryce Macher 将视角投向了消费者全生命周期（"Womb to tomb"，从始至终）。他深入思考的是，当普通消费者日益频繁地与 ChatGPT、Perplexity、Claude 以及 Grok 等消费级生成式 AI 工具互动时，品牌的客户体验版图将发生怎样的根本性变化。这不仅局限于 Domino's 自身的官方应用，而是延伸到整个开放的 AI 生态中。 ```mermaid flowchart LR accTitle: 消费级 AI 生态中的客户体验链路 subgraph Search["前链路：发现与订购"] A["传统 SEO"] --> B["GEO 生成式引擎优化"] B --> C["智能体订购交互 (Agentic Ordering)"] end subgraph Feedback["后链路：反馈与声誉"] D["公域 AI 平台对话 (ChatGPT/Claude 等)"] --> E["品牌诉求与情绪表达"] E --> F["全局客户体验优化 (Voice of Customer)"] end Search --> Feedback ``` Bryce 提到，随着搜索引擎对各类 AI 交互内容的索引，消费者的提问与反馈方式正在发生巨变。作为品牌方，核心目标依然是为消费者打造以“美味、热气腾腾、行业领先的披萨”为中心的神奇体验，但触达路径正在演变： - **生成式引擎优化（GEO，Generative Engine Optimization）**：企业需要思考 GEO 如何与传统搜索引擎优化（SEO）并行，以确保在生成式对话中精准呈现品牌价值。 - **智能体订购变革（Agentic Ordering）**：AI 代理直接接管点单流程后，用户表达订餐意图的方式将变得更加自然和多样化。 - **AI 咨询与售后反馈**：消费者遇到服务体验问题时，可能会直接向 ChatGPT 咨询如何维权或处理，企业必须从宏观全局的角度捕捉这些信号，将其纳入客户心声（Voice of the Customer）的分析体系中。 --- ### [42:35](https://www.youtube.com/watch?v=RwSeeVqA7-0&t=2555s) 企业级 AI 落地的四大支柱与开源模型优势 LevelAI 首席执行官 Ashish Nagar 重点剖析了驱动企业级 AI 转型的关键要素，并提出了企业在评估底层 AI 方案时必须权衡的四个核心维度（4P 模型）： | 评估支柱 (4P) | 企业核心考量点 | |:--- |:--- | | **性能 (Performance)** | 在海量并发下的响应延迟与吞吐能力，例如处理数亿级数据流时的低延迟要求。 | | **价格 (Pricing)** | 长期运行的成本效益与投入产出比（ROI），避免高昂的 API 调用账单。 | | **个性化 (Personalization)** | 针对企业私有业务场景和领域知识进行深度微调与适配的能力。 | | **数据隐私 (Privacy)** | 确保企业核心资产与用户敏感数据在受控环境中处理，符合合规标准。 | Ashish 明确指出，当前 AI 基础模型形成了两条鲜明路径：一条是以 OpenAI、Google 等为代表的闭源商业模型路线；另一条则是 Meta 以及海外活跃社区推动的开源模型路线。他强调，在企业级生产环境中，基于开源模型自研专属 AI 架构展现出了极其显著的 ROI 优势。 对于拥有数亿乃至数十亿数据点的成熟大型企业而言，单纯调用 OpenAI API 不仅难以承受长期的调用成本，更无法在极端负载下保证如两秒以内的响应延迟。Ashish 认为，仅仅在产品中封装一个闭源 API 与从底层构建可扩展的企业级 AI 架构之间存在着巨大的专业壁垒，而拥抱底层开源模型生态并构建端到端能力，正是企业 AI 走向真正成熟的核心标志。 --- ### [44:28](https://www.youtube.com/watch?v=RwSeeVqA7-0&t=2668s) 投资组合策略、流程自动化与底层基建 Vanguard 数据分析与洞察中心负责人 Jing Wang 分享了金融巨头在面对 AI 浪潮时的投资组合规划逻辑。她指出，未来六个月 Vanguard 将继续保持对 AI 的大规模投入，但核心挑战在于如何将有限的资源精准配置在投资组合的最有效节点上，以平衡客户体验、业务增长与运营效率。 ```mermaid flowchart TD accTitle: 企业 AI 投资组合配置战略 subgraph Foundation["底层数据与架构基础"] F1["AI-ready 数据资产治理"] F2["企业级 API 环境与服务集成"] end subgraph Investment["AI 投资组合落地方向"] P1["后端流程与工作流自动化\n(确定性极高的短期 ROI)"] P2["前端客户体验创新实验\n(语音 Agent / 多模态交互探索)"] end Foundation --> Investment ``` Jing Wang 从企业实践角度总结了投资配置的关键优先级： - **后端工作流自动化（Workflow Automation）**：在她看来，后端业务流程的自动化改造是当前确定性最高、投资回报率（ROI）最显著的投资方向。 - **前端客户体验实验（Frontend CX）**：尽管技术与外部供应商（如语音 AI 服务商）迭代剧烈，多模态模型不断推陈出新，但企业绝不能放弃在前端交互层面的探索与竞争。 - **坚实的基础设施建设（Foundations）**：持续深耕“AI 就绪数据”（AI-ready data）的治理体系以及高可用 API 环境的构建，为上层算法与业务敏捷对接提供支撑。 整场讨论在主持人 Andrew Seaman 的致谢与全场掌声中圆满结束，与会嘉宾一致认为，将 AI 技术演进与企业战略紧密锚定，是在快速变化的技术周期中实现长期商业价值的根本之道。 ## Notable quotes > [00:04](https://www.youtube.com/watch?v=RwSeeVqA7-0&t=4s) you can't be artificial when it comes to customer experience — Andrew Seaman > [00:51](https://www.youtube.com/watch?v=RwSeeVqA7-0&t=51s) 踩准了市场风口，真希望自己的投资组合也能这么准 — Bryce Macher > [13:15](https://www.youtube.com/watch?v=RwSeeVqA7-0&t=795s) 如果每次走进大型企业，听到他们说‘我们喜欢你们的 AI 路线图，但你们能不能先帮我们把 API 路线图搞定，因为我们工程排期已经堆积了两年’，我都能拿到一毛钱的话，我现在早就发财了。 — Ashish Nagar > [16:11](https://www.youtube.com/watch?v=RwSeeVqA7-0&t=971s) 先把业务规则写下来（Just write stuff down） — Ryan Wang > [17:30](https://www.youtube.com/watch?v=RwSeeVqA7-0&t=1050s) 能够与一个真实的人交流 — Bryce Macher > [22:22](https://www.youtube.com/watch?v=RwSeeVqA7-0&t=1342s) 别废话，马上给我转人工！ — Ryan Wang > [22:22](https://www.youtube.com/watch?v=RwSeeVqA7-0&t=1342s) 让更多人更快地用上它 — Ryan Wang > [29:04](https://www.youtube.com/watch?v=RwSeeVqA7-0&t=1744s) 祝你们好运，大概一年后再见吧，到时候我们再来帮你们收拾烂摊子。 — Ryan Wang > [32:28](https://www.youtube.com/watch?v=RwSeeVqA7-0&t=1948s) 在原先的陈旧底座上撒上一层生成式 AI 的胡椒粉（Sprinkling GenAI on top） — Ashish Nagar > [39:15](https://www.youtube.com/watch?v=RwSeeVqA7-0&t=2355s) 请对供应商好一点，至少签一年期合同，因为实现成效需要时间 — Ryan Wang ## Mentioned Andrew Seaman, Ryan Wang, Bryce Macher, Ashish Nagar, Jing Wang, Mark Cuban, Ai4, Vanguard, Domino's, Assembled, LinkedIn News, Level AI, Stripe, Amazon, Canva, Ramp, Apple, Disney, Hulu, OpenAI, Google, Meta, AWS, Alexa, GPT-2.5, GPT-4, GPT-5, ChatGPT, Perplexity, Claude, Grok, Jira, Pizza Tracker ## Video description Jing Wang, Principal, Head of Center for Analytics & Insights, Vanguard Bryce Macher, Vice President, Decision Science, Domino's Ryan Wang, Co-founder and CEO, Assembled, Inc. Andrew Seaman, Editor at Large for Jobs & Career Development, Linkedin News Ashish Nagar, CEO, LevelAI Abstracts This session will explore how AI is revolutionizing customer experience (CX) and customer service by automating interactions, personalizing support, and improving response times. Panelists will discuss how AI-powered tools, such as chatbots and virtual assistants, are transforming customer service operations in contact centers, enabling faster resolutions and reducing operational costs. Learn how organizations are integrating AI into their contact centers to improve customer satisfaction, streamline workflows, provide 24/7 support, and retain customers. The discussion will also examine the challenges of balancing AI automation with human interaction and strategies for creating a seamless, customer-centric experience. [Source](https://www.youtube.com/watch?v=RwSeeVqA7-0)

[0:04] Andrew Seaman: you can't be artificial when it comes to customer experience [0:51] Bryce Macher: 踩准了市场风口，真希望自己的投资组合也能这么准 [13:15] Ashish Nagar: 如果每次走进大型企业，听到他们说‘我们喜欢你们的 AI 路线图，但你们能不能先帮我们把 API 路线图搞定，因为我们工程排期已经堆积了两年’，我都能拿到一毛钱的话，我现在早就发财了。 [16:11] Ryan Wang: 先把业务规则写下来（Just write stuff down） [17:30] Bryce Macher: 能够与一个真实的人交流 [22:22] Ryan Wang: 别废话，马上给我转人工！ [22:22] Ryan Wang: 让更多人更快地用上它 [29:04] Ryan Wang: 祝你们好运，大概一年后再见吧，到时候我们再来帮你们收拾烂摊子。 [32:28] Ashish Nagar: 在原先的陈旧底座上撒上一层生成式 AI 的胡椒粉（Sprinkling GenAI on top） [39:15] Ryan Wang: 请对供应商好一点，至少签一年期合同，因为实现成效需要时间

[0:04] Ryan Wang: we're going to automate away everybody [2:02] Bryce Macher: Hey, I timed the market, that's great—wish my portfolio could do that. [7:03] Ryan Wang: the work around the work [10:10] Ryan Wang: metabolize this change really quickly [17:30] Bryce Macher: luxury experience [17:30] Bryce Macher: inauthentic digital touchpoints [24:49] Ashish Nagar: Level 5 CX automation [26:46] Bryce Macher: bringing the 5-5-5 deal back [29:47] Ryan Wang: vibe coding [31:54] Ashish Nagar: sprinkling GenAI on top [39:15] Ryan Wang: be kind to your vendors [40:37] Bryce Macher: magical pizza experiences centered around delicious hot industry-leading pizza [43:25] Ashish Nagar: good luck trying to push that into OpenAI's API and getting a two-second latency back from it

[0:04] All right, thank you everyone. Um, and [0:06] thank you for joining us. So, as you [0:08] heard, we're going to be talking about [0:10] sort of the intersection of AI, customer [0:12] experience, customer service, which I [0:14] think is such a great conversation to [0:17] have and we have such a great panel [0:19] because we have people who are internal, [0:20] we have people who are vendors to [0:23] companies. So we're getting a lot of [0:25] perspectives here and at such a crucial [0:29] and interesting area of business where [0:32] you you know despite the name you can't [0:33] be artificial when it comes to customer [0:35] experience because that that kind of [0:37] spoils the whole thing. So I'm Andrew [0:39] Seaman. I am an editor at large at [0:41] LinkedIn. Um LinkedIn has its own news [0:43] team. Some people don't know that but [0:45] I'm on it. Uh and I'll be leading the [0:48] panel and I'm going to let them [0:49] introduce themselves. So Ryan, do you [0:51] want to go first and tell us a little [0:53] bit about you, your company, but also [0:56] sort of where you're focused on the [0:58] customer service, customer experience [1:00] area? [1:01] >> Yeah, I'm happy to. I'm Ryan Ryan Wang. [1:04] I'm co-founder CEO at Assembled. So we [1:05] build the AI platform for customer [1:07] support operations that includes AI [1:10] automation through the agents, chat, [1:12] voice that handle over 70% of issues [1:15] automatically, but also through deep [1:16] into the contact center if you're [1:18] familiar with this workforce management. [1:20] So planning, forecasting, and scheduling [1:21] call centers of 20 all the way up to [1:23] 20,000 people and increasingly that [1:26] people footprint is including human and [1:28] AI. Um u my own background and things [1:31] that I'm focused on. So I I worked on [1:33] machine learning, you know, back when it [1:35] was ML, not AI. And I was working on ML [1:37] for fraud detection at a small payment [1:39] startup called Stripe when it was 80 [1:40] people. Ended up on working on ML for [1:43] support. And so we've seen this kind of [1:45] transition from ML to AI. And and I [1:47] think what I'm most focused on and [1:49] interested to talk about here as well is [1:51] is just the evolution of people's [1:53] expectations for this technology. You [1:55] know, it's really quickly gone from [1:56] we're going to automate away everybody [1:57] to uh maybe there's a more thoughtful [1:59] approach that involves humans and AI. [2:02] Uh good morning. I'm Bryce Mocker. I've [2:05] been working in the the kind of space of [2:08] like general consumer experience from a [2:09] data science perspective for for a long [2:12] while now. You know, starting with a [2:13] postto in like 2014 and and kind of [2:15] bouncing between academia and uh the [2:18] professional world. So, you know, I've [2:20] I've seen AI features in front of AI [2:24] features as you just said uh in front of [2:26] tens of millions of of consumers across [2:29] streaming and hospitality. uh and now [2:31] I'm in in QSR at Domino's. From a a [2:35] perspective of customer experience, the [2:37] exciting thing about what we've done at [2:38] Domino's is is we are realigned a lot of [2:41] teams under the the rubric of decision [2:43] science. So now I you know I have uh [2:45] purview across uh consumer research, [2:48] analytics, data science and the [2:50] consumerf facing kind of prescriptive ML [2:53] features all the way then back into [2:56] voice of customer and you know analytics [2:58] around call centers uh and things like [3:00] that. So it's a really exciting space at [3:02] Domino's to be in. Uh my background I [3:04] was trained as a linguist. So to your [3:06] point about like I remember when this [3:07] was called ML I remember when this was [3:09] called linguistic computing and [3:11] computational linguistics and then [3:12] natural language processing and now you [3:14] know at that time I was like wow I feel [3:16] like the odd duck out and then you know [3:18] as 2.5 came out GBT 2.5 came out in 2019 [3:21] I was like hey I time the market that's [3:24] great wish my portfolio could do that. [3:26] Um, so I love this space. I think it's a [3:29] really exciting and uh interesting [3:32] space. One of the futures that like I'm [3:34] keeping an eye on is as we talk about [3:36] linguistics in that space is what are [3:37] the other modalities? Can we get to a [3:39] place it's like sensorial computing or [3:41] sensory computing and sensory AI? [3:45] >> Hi, my name is Ashish. I'm the founder [3:46] and CEO of Level AI. So Level AI is an [3:50] end toend AI and CX platform. We have [3:52] products. We had our roots in human [3:55] agent technologies for human agents. So [3:58] automated quality assurance making sure [4:00] agents quality is uh evaluated with an [4:03] AI voice of the customer looking at [4:06] large pools of unstructured data within [4:09] the contact center within customer [4:10] service to find insights for product [4:12] operations sales compliance and so on [4:15] and then agent co-pilot and then over [4:17] the last year year and a half we re we [4:19] sort of realized this future where AI [4:21] agents and human agents we see will live [4:23] together so we've launched a series of [4:26] voice and chat agents which can do full [4:27] automation. We talked really and we'll [4:30] talk more about in the panel but this [4:32] idea of balance between human and AI [4:35] agents as we as we sort of service [4:38] global brands we s we service uh a half [4:41] a dozen or a dozen Fortune 500. So we've [4:44] really scaled these technologies over [4:45] the last few years. when I start when we [4:48] started the business we sit set like [4:50] luckily were right at the edge when LLMs [4:54] were starting to get uh you know get [4:56] popular and the reason was before [4:57] starting the business I was a product [4:59] leader in Amazon's Alexa team and at [5:02] Alexa we were using as as Ryan and and [5:05] the panelist mentioned we were using [5:07] classical ML techniques my last project [5:09] in Amazon Alexa was making Alexa talk to [5:11] a human for 20 minutes or any social [5:13] topic and the machine learning involved [5:15] in that were like seven different teams [5:17] trying to pick up every part of this [5:19] problem and that was 2019. It seems like [5:21] a lifetime ago and now all those teams [5:24] all those PhD programs all of those [5:26] approaches have been shelved in favor of [5:28] these new LLM systems. So really [5:30] exciting time and so I've been in AI for [5:33] the last 15 years but and really we have [5:36] a new set of uh capabilities now to work [5:39] on and so really looking forward to the [5:41] discussion today. [5:45] Hi everyone. Uh my name is Chin Wong. I [5:48] lead the enterprise uh AI center of [5:50] excellence at a Vanguard. I I hope [5:53] everybody knows Vanguard. But for those [5:55] who don't know, Vanguard is one of [5:57] world's largest mutual fund u asset [6:00] manager. We provide mutual funds and [6:01] ETF. In fact, we just crossed 11 [6:04] trillion uh asset under management um [6:08] last week and we serve about 50 million [6:11] investors across the globe. Um so I've [6:14] been with Vanguard for 15 years and 10 [6:17] nine 10 of that years has been uh [6:20] helping Vanguard to build up this muscle [6:23] in applying data science, machine [6:26] learning, AI in our parts of our [6:28] operations. Um my team the enterprise [6:31] center of excellence support all our [6:33] distribution business and corporate [6:35] services functions to really apply uh [6:38] machine learning AI to improve our [6:41] client experiences and operational [6:43] efficiencies drive growth etc. And it [6:46] was uh I think it's you know over the [6:48] last nine 10 years we just have seen [6:50] such amazing um evolution in this [6:55] technology and it's just uh phenomenal [6:58] to see how AI is transforming client [7:02] experiences [7:03] >> and Jing while you have the remote I or [7:06] the the microphone uh I I'd be [7:09] interested to hear where have you seen [7:11] the biggest difference in customer [7:14] experience? [7:15] brought on by AI, you know, over all [7:17] that time where what do you look at now [7:19] and say, "Wow, that is very different [7:21] than anything I've seen in the past, you [7:24] know, decade or more." [7:26] Yeah, I think over the last couple [7:28] years, uh, while everybody is super [7:30] excited about infusing generative AI [7:34] experience in direct clienting [7:36] interactions, where we see the biggest [7:39] impact so far is about creating tools to [7:42] enable our customer service uh, [7:45] representatives, those frontline crew [7:48] like advisors, so relationship managers [7:51] to really enable them to serve to have [7:54] better, higher, more capacity, better [7:56] insights, better tools to serve their [7:59] clients, right? Um, so we have built for [8:02] example call summarization for our [8:05] frontline crew. And that may be sounds [8:07] simple, but it's a massive uh it's [8:10] actually a very s big source of uh um um [8:14] value. So for advisors for example, they [8:17] have calls with their clients that last [8:20] 45 minutes for an hour and we save them [8:23] 20 30 minutes right after the call to [8:25] file away that conversation. And so [8:28] that's efficiency gain. Um we can [8:30] control in building those solutions we [8:32] control the type of insights that we [8:34] want to capture from those [8:35] conversations. So we have very high [8:37] quality standardized uh capture of those [8:40] insights and and so that make their prep [8:43] for the next call much easier and then [8:46] we are expanding that solution to look [8:48] not only for one call but we spend we [8:51] look backwards six months to look at the [8:53] entire relationship. We provide that [8:55] summary to the advisors or relationship [8:57] managers so they can immediately figure [9:00] out what's the best thing, what's the [9:02] client needs, what's the best topic to [9:04] bring to that uh clients. So we found [9:07] those type of tools are really now [9:09] massively in infusing into the frontline [9:12] workflow and are making them smarter, [9:14] more intelligent and then they have more [9:17] capacity, more time to spend with our [9:19] clients. [9:20] >> That's really fascinating. And um Ryan, [9:22] could you share your views on this? Like [9:24] where have you seen the biggest [9:25] difference like across, you know, your [9:27] experience in the space, you know, where [9:29] have you seen things evolve so quickly? [9:31] >> Yeah. Um I mean, definitely very similar [9:33] to Jing in terms of um I think the quick [9:36] change from let's automate everything [9:38] away to let's actually make people more [9:39] productive. And surprisingly, some of [9:41] the things that make people more [9:42] productive aren't a cure to write the [9:44] response for me and I'll just, you know, [9:46] edit it and send it off, but actually [9:48] the work around the work. Um, one of our [9:50] customers is Canva and they were able to [9:52] increase uh cases solved per hour in [9:54] customer support by 15%. Similar solving [9:57] the wrapups. If you need to, if anybody [10:00] works in technical support, sometimes [10:01] you need to send a bug over to [10:02] engineering. You have to have the follow [10:04] the Jira template. If you don't get the [10:06] template exactly right, they're going to [10:07] kick it back. So, just, you know, send [10:09] the information in. Here's the template. [10:10] Boom. That that saves a bunch of time. [10:12] And as well creating the um the greeting [10:15] and following your standard uh uh style [10:17] guide and your your brand voice. um [10:19] these things are very very easy for AI [10:20] to do. I think one other thing that uh [10:23] happened very recently with GPT5 was um [10:27] the models just keep getting better and [10:29] you know like getting cheaper and um [10:31] we've have invested a lot since GPT4 [10:34] roughly in being able to metabolize this [10:37] change really quickly as an engineering [10:38] organization inside of assembled. So [10:40] when GPT5 came out, before lunch was [10:42] even done, we had been able to run it [10:44] through our evals and see that it really [10:46] was 20% better in terms of our evals, [10:48] our standard evals with no tuning. It [10:50] was going to be 10x cheaper. You could [10:51] cach the queries. And so it's amazing [10:53] when you see all this kind of investment [10:55] going into the models. You can [10:56] immediately everything just gets way [10:58] better. And that's through our AI [10:59] co-pilot and also our AI agent. And and [11:02] so you're just sitting on top of a lot [11:03] of innovation and that makes life a lot [11:04] easier for us. [11:06] >> Yeah, that sounds great. Um, and Bryce, [11:08] I have a sort of [11:11] alteration to this question that I I I [11:13] thought of when you were um giving your [11:15] introduction. It seems like there AI [11:18] brings some reorganization to the side [11:21] of like, okay, we can't just keep our [11:23] current management structure or [11:24] reporting structure. [11:26] How how do you get your your head around [11:30] that when you sort of say like, "Okay, [11:31] we're bringing this in and we have like [11:34] three teams doing the same thing or [11:36] reporting in different people." Can you [11:37] tell us a little bit about that [11:38] experience because I imagine you you had [11:40] to deal with this when you went to [11:41] Domino's? [11:42] >> Yeah, it's an interesting question and I [11:44] think the it's going to be a work in [11:46] progress, right? as you were saying like [11:47] the the jump in performance from GPT4 to [11:50] 5 like eval everything like they're [11:52] incredible but how that reorganizes the [11:54] work as that mo as those models make [11:56] steps up is really important if we just [11:58] see that accelerate which I think we're [12:00] going to then that means there's going [12:01] to be a little bit of volatility in how [12:03] organizations are built around like how [12:05] you plan what's possible now so like I [12:08] think you know my bet would be you're [12:10] going to see a lot of vertical alignment [12:11] of teams so that you can have the whole [12:14] customer journey including its technical [12:16] infrastructure and those more insights [12:18] driven roles in one org so you could [12:20] move quickly by the business disciplines [12:23] you support. So for example you know at [12:24] Domino's I reported to the CMO and so [12:27] you know while that might seem really [12:29] strange from the organizational patterns [12:31] we saw like pre- pandemic as we're [12:34] seeing the acceleration from Genai it's [12:37] those people who are technical experts [12:38] but also speak the business and can also [12:40] drive the business that'll see the most [12:42] acceleration. Now, is that going to be [12:45] the permanent home or is that the right [12:46] structure? Again, like we're in a [12:49] volatile moment, so we've just got to [12:50] like work through it and figure it out [12:52] as it goes, but I think there's a lot of [12:54] promise there in driving acceleration, [12:56] thinking about what the problem you're [12:57] trying to solve is and organizing that [12:59] way. [13:00] >> Yeah. [13:00] >> And I assume that it's going to be [13:01] different by organization, too. [13:03] >> Absolutely. like some some places are [13:05] marketing organizations, some places are [13:07] techled organizations and that'll have a [13:09] huge impact on how you think about [13:11] organizational structure as genai [13:13] reconfigures workflows. And then Ashish, [13:15] I have a question for you when it comes [13:18] to sort of implementing AI because there [13:22] there seems to be almost two competing [13:24] goals here. You want things to be done [13:26] fast and this is also coming from the [13:27] customer. They want they want their [13:30] issue solved very quickly, but they also [13:33] want that personal touch. They want to [13:34] be heard. How do you balance that? Um, [13:39] you know, especially for your clients, [13:40] like how do you say like, listen, we're [13:41] we're going to do this. We're going to [13:42] do it fast, but we're also going to make [13:44] your customer feel valuable. [13:47] >> That's a great one. And let me sort of [13:49] try to uh also tie in the previous [13:51] question which you mentioned. When we [13:53] talk to customers, we look at geni [13:56] applications within customer service in [13:58] a framework of four categories. And [14:00] often it's kind of when we think of [14:02] geni, we often think of automation. But [14:04] what I would summit is there are four [14:07] layers. The the easiest and the highest [14:10] ROI in some cases is intelligence [14:12] followed by workforce automation. There [14:15] is a lot which as as was mentioned [14:17] there's a lot which you can do with [14:19] workflow automation with things like [14:20] summarization QA automation and so on. [14:23] Then there are co-pilots and I think [14:25] this framework holds in most geni [14:27] applications legal coding so on so [14:29] intelligence workflow automation [14:32] co-pilots and then full automation and [14:35] then linking to your question of how [14:37] does implementation [14:39] uh challenge change the biggest thing [14:42] which we realize before AI it's about [14:45] API [14:47] a lot of enterprise organizations don't [14:49] have the APIs to talk to the AI and So [14:53] and I could if I had a dime for every [14:55] time I walk into an enterprise situation [14:57] where people are like we love the AI [14:59] road map but can you help with the API [15:01] road map and we have you know two years [15:03] of engineering backlog on that API road [15:05] map right and the data underlying for [15:08] the API especially the larger the [15:09] organization is so deeply embedded which [15:12] AI could use you can do simp like you [15:14] can do things like summarization and so [15:16] on and so forth that's fine but that's [15:17] tough on top but if you're going deep in [15:20] implementation so API I versus AI number [15:23] one. Number two, quicker use cases like [15:26] intelligence which do not lead to [15:29] massive changes in workflows have lower [15:31] implementation barrier because it's an [15:33] intelligence layer on top. the higher [15:35] you go towards automation then [15:38] augmentation and full automation the [15:39] deeper the implementation challenge [15:41] right uh without taking more time but I [15:43] that's the framework we typically use [15:45] and we typically advise we we are seeing [15:48] customers go on that journey with us [15:50] they start with intelligence and then [15:51] over a 3 6 9 12 month period go towards [15:54] automation we wouldn't we wouldn't [15:57] advise anybody to take on all four in [15:59] one go or you know go in a go in a [16:01] halfert manner people have done that but [16:03] it just doesn't lead to the right [16:05] strategy because if you don't have the [16:06] underlying intelligence layer, you can't [16:09] do much with automation. For example, [16:11] >> can I jump in here because I like this [16:13] uh API before AI point, but I feel like [16:15] that's already level two. With some of [16:18] the companies we see, level one is just [16:20] write stuff down. It's just it's not, [16:22] oh, you want to automate what do you [16:24] want to automate? What what policy? Oh, [16:26] your policy is not even written down. [16:27] You have to go ask, you know, all these [16:29] different people. Um anyway, [16:32] >> yeah. Well, I I will also chime in. Uh I [16:35] think the progress is iterated, right? [16:38] Like any progress, you cannot wait for [16:40] us like let's get API cleaned up and [16:44] documented or today we look at AI ready [16:47] data like we need data to be able to do [16:50] really beautiful AI on top of it. Uh and [16:52] then we're going back to redefine what [16:55] is AI ready data that is different from [16:57] a traditionally good data. Right? Then [16:59] we're doing another initiative to say [17:01] let's now go that one step further to [17:03] get our AR data ready. Right? So now [17:07] you're looking at things maybe rewrite [17:09] your uh website content so that they can [17:11] be much easier uh to to be consumed by [17:14] AI models and gives you much better [17:16] accuracy in the results. So I think our [17:19] understanding of what is this system [17:22] ecosystem that needs to be built to [17:24] support better and better AI solutions [17:26] is evolving and then so you just have to [17:29] continue to build [17:30] >> and well and I have a question actually [17:32] and Bryce maybe you could um chime in on [17:34] this too is when you think of all those [17:36] things so you talk about okay I come in [17:38] and they have they sort of hey we want [17:40] AI uh but they don't actually have like [17:42] the roadmap is it and I wanted to ask [17:45] this a little later but KPIs [17:48] are so important because you might have [17:51] leaders who are like, "Hey, we're all in [17:52] on AI." Um, but they think like reduce [17:55] cost, things like that, but there's so [17:57] many and this also goes back to the idea [17:59] of like the human versus AI [18:01] interactions. [18:04] What are the KPIs? Like is that the [18:06] first point where you say like we need [18:07] to find our goal? [18:09] >> Yeah, it's a great question because it's [18:11] a there's two two levels of it. One is [18:13] just something we've always done in [18:14] enterprise which is just measurement [18:16] design, right? like it's like what is [18:17] the right metric we're trying to move? [18:19] Which one is a leading indicator of the [18:20] business? Are there lagging indicators [18:22] we need to pay attention to? Um but I [18:25] there's also a contextual element to it. [18:26] So like I'm thinking about that mic drop [18:28] uh tweet that Mark Cuban had about like [18:31] hey look Genai is just proliferating all [18:33] of these inauthentic digital touch [18:35] points. So as we're building these [18:37] experiences for consumers like consumerf [18:39] facing AI um you know the premium the [18:43] luxury experience is going to be talking [18:44] to a person right like how exciting to [18:46] talk to a person but I think one of the [18:48] things we don't pay attention to as much [18:50] and I I think about this a lot because I [18:52] have a consumer research team is that as [18:54] that external world changes so too does [18:56] the work world so you have all of these [18:58] inauthentic digital touch points that [19:00] you have to have on your day-to-day life [19:02] you're like man if I have to talk to one [19:03] more AI I swear and Then you get to work [19:06] and what are you going to do? Well, I [19:07] got to sit here and talk to these AIs. [19:10] Those KPIs that you designed are going [19:12] to move and you have to make sure you've [19:14] set them up. So like customerf facing, [19:16] right? We've always talked about the [19:17] ultimate goal for a business like ours [19:19] is like happiness because it's the [19:21] leading indicator of lifetime value or [19:24] like trust or you know delight or you [19:27] know any of those kind of more soft [19:28] metrics we then find a way to quantify. [19:30] And for internal initiatives you're [19:32] going to have to also figure that part [19:33] out. you're going to have to figure out [19:34] what are the ways we quantify employee [19:37] engagement. Like how do we think about [19:39] people being happy at work? How do we [19:41] connect that to productivity and and [19:43] acceleration? Because I do worry that as [19:45] we go deeper to your question of like [19:47] having all these inauthentic [19:48] experiences, we're going to run into a [19:50] workplace where it's pretty dystopic and [19:52] you you're just going in to like punch [19:54] the thing and create your TPS report. [19:57] Jin, can you because I think when we [20:00] talked first, you mentioned that you did [20:02] have to re-evaluate and retrain [20:05] basically the the humans to say no, you [20:07] can spend more time on a call, [20:09] >> right? So when we look at a a client [20:11] experience, you do want to, you know, [20:13] put a client at the center, right? And [20:15] then our measurement of a client [20:17] experience needs to consider this end to [20:20] end journey that they they spend with us [20:23] um when they interact with us and then [20:25] use that complete sort of journey to [20:28] measure their holistic experience. So [20:30] when we were working on a call center [20:33] use case for example um we used AI to [20:36] build IVR rotting models and then we [20:39] used AI to build um bots that will be [20:43] able to uh take questions automatically [20:46] and then we then pass the calls to the [20:48] human agent one um that those questions [20:52] cannot be solved by the bots. So we [20:54] measure that complete journey um [20:57] satisfaction. So we would use transfer [20:59] rate to say how good are we at avr [21:02] because if we get them right the call [21:04] won't be transferred but then we also [21:07] realize that people have multi- intent [21:09] when they call it just not most of 50% [21:13] of time it's not just about one thing so [21:15] we have to consider don't say hey [21:18] transfer rate the goal is to get down to [21:20] 0% because you can never get down to 0% [21:23] right because consider your customer at [21:25] the center and then for example then [21:27] when the the the calls are transferred [21:29] to the human agent. Um we built in those [21:32] tools. Initially we said let's cut down [21:34] average handle time and that's the goal [21:36] we're going to use to evaluate to say [21:38] how good the tool is. But then we [21:40] realized that if laser focus on average [21:43] handle time actually put some fears into [21:46] our call center agents. So they just [21:48] want to quickly finish that call and [21:50] move on to the next versus a lot of in [21:53] we deal with people's financial lives. A [21:55] lot of times it's a conversation that [21:57] allow us to deeperly more deeply [21:59] discover their needs and we can help [22:01] them much more right. We want those very [22:05] meaningful very valuable interactions to [22:07] still to happen. So we went back and [22:10] reevaluated average handle time alone [22:13] should not be the only goal and then we [22:15] added others like C said those kind of [22:18] more holistic measurement to make sure [22:19] we capture all the client experience. [22:22] >> Wonderful. And Ryan, how do you when you [22:25] walk into an enterprise situation or any [22:28] of your clients for that first meeting, [22:31] how do you walk them through like, you [22:33] know, okay, this is where we have to [22:35] start and this is where we're we're [22:37] going to go with the next meeting or [22:39] something like that. [22:40] >> Yeah. And I think connecting this to the [22:42] metrics point, I feel like this is a [22:44] very reasonable panel, you know, in [22:46] terms of like the role of human and AI. [22:48] I mean, when we're walking into [22:49] meetings, we're talking about ROI, and [22:51] the dirty secret of customer service is [22:53] we're trying to deflect. Like, one of [22:55] the metrics we tell people for AI [22:57] automation is we're deflecting 70% of [23:00] tickets. Um, I think you described this [23:03] Bryce. Nobody wants to be deflected. [23:05] You're just like, no, get me to a person [23:07] right now. Um and I think so so we're [23:10] trying to draw this line between [23:12] seuitees and support leaders and [23:15] investors who are increasingly pushing [23:17] the the envelope on AI you know [23:19] automation as well. Okay you want to [23:21] deflect more people you want to get more [23:23] efficiency out of the business you want [23:25] less stuff to go to people but you also [23:27] want to deliver a good customer [23:28] experience. One of the interesting [23:30] things that I've seen people start doing [23:32] is um would just show you what it looks [23:36] like. And the people who are most [23:38] sophisticated about buying AI tools, I [23:40] think, are the people who [23:42] use those AI tools themselves and have a [23:44] deep understanding of what is possible [23:46] and what is not possible. And RAMP is [23:48] one of our customers. I'll give you the [23:49] example of RAM. The head of user [23:50] operations is also the head of AI. [23:52] Interesting. Um [23:55] their KPIs are around adoption. [23:58] was not around impact to productivity. [24:01] It's not around ROI that broke my brain. [24:04] It's just adoption. And it's because [24:05] they've used the tools enough. They've [24:07] used our AI comply enough to say, "Okay, [24:08] I see exactly which parts of the [24:10] workflow this is good for. I [24:11] qualitatively understand this is going [24:13] to be really good. And I don't want to [24:15] spend the time going through data [24:16] science measuring the AB test of what [24:18] happened before and after. I just want [24:20] more people to use it and more people to [24:21] use it faster." Um, so when we can get [24:23] to that point of the conversation, I [24:25] think is really powerful because it's [24:26] not we're not trying to convince you [24:28] what to do with AI. We're helping you [24:30] accelerate your adoption. [24:32] >> Wow, that's really interesting. And [24:34] Ashish, do you have similar experiences [24:37] or, you know, how because I I think [24:39] there is that push and pull of like, you [24:41] know, AI is all about ROI, but then, you [24:44] know, you have to realize like, okay, [24:46] that we can't just focus on that purely [24:47] with customer experience. [24:49] >> Absolutely. And kind of like what R [24:51] mentioned, a lot of times we walk into a [24:53] conversation with an automation goal or [24:56] a cost control or a sales conversion [24:58] goal. But if you step back, it goes back [25:00] to what Jen mentioned, customer is at [25:01] the center. So if you as you go up the [25:04] executive ladder, it's mostly about [25:07] seesat or customer experience or NPS [25:09] score. And guess what? Those two things [25:11] are kind of conflicting. Another way we [25:14] think about this a lot like there's an [25:15] analogy to autonomous cars where as you [25:18] think about higher and higher levels of [25:20] automation the highest level in [25:22] automation is when the car can drive [25:24] itself. I would actually argue in [25:26] customer service the highest level is [25:28] not full automation. The highest level [25:30] is when humans and AI agents are in a [25:32] symbiotic situation where we have found [25:35] that perfect balance which they are [25:36] talking to each other learning from each [25:38] other and the AI has figured out what's [25:40] the right place to automate and then you [25:42] and the CX organizations have figured [25:44] out what's the right place for humans to [25:46] operate right that balance is great [25:49] level five automation in CX and there's [25:51] a lot of push currently uh and kind of [25:54] what Ryan was mentioning from Silicon [25:56] Valley from AI and so on and we are all [25:57] practitioners of AI But I just don't see [26:00] it it ending being a 99% or 90 plus% [26:04] auto. It looks good in tech crunch but [26:07] it's not the reality of how [26:09] organizations want their customer [26:11] service to be and how we want to be [26:13] treated as customers, right? And so u [26:16] that's that's how we approach it. What's [26:18] the right balance and what's the right [26:20] synergy eventually to sort of uh [26:23] maximize seesat or ma maximize brand [26:25] perception, customer experience and so [26:27] on. [26:28] The best brands in the world are known [26:31] for their customer experience. Like [26:32] that's sort of like the last, not the [26:35] last, but one of the most enduring [26:37] values in a brand, right? What you [26:39] remember Apple for or Disney for or Hulu [26:42] for, right? And that's not enabled by [26:45] full automation. [26:46] >> And Bryce, I jumping off the idea of [26:50] like brand identity. You know, in [26:51] college, as you could tell, I loved [26:53] ordering from Domino's. Um, and I love [26:56] seeing that little like, you know, [26:57] loading thing tick up. So, [27:00] >> bring 555 back. [27:04] >> When it comes to brand identity, [27:06] obviously there are risks with AI. So, [27:09] how do you manage that? Like, how how do [27:12] you sort of say like, okay, we're going [27:14] to have to allow for some incidents to [27:16] to occur, hallucinations, brand [27:18] inconsistency. How do you what [27:21] safeguards do you put in place? what um [27:23] you know what safety mechanisms have you [27:26] have you tried to employ? [27:28] >> It's a great question. I you know the [27:29] thing I'll say is you know like we were [27:31] talking about like ML was back in the [27:32] day you know our our pizza tracker is an [27:36] intelligent project. It's not just a [27:37] wild guess right like so like that [27:39] experience has a level of AI to it but [27:42] it's all it's like one where you're like [27:43] well what's the safety risk here? Oh [27:45] there's not really any safety risk [27:46] right? So, you know, on that side, it's [27:48] it's an easy decision to make when [27:50] you're looking at those trade-offs and [27:51] trying to understand what's good, what's [27:52] bad, what's bad experience versus unsafe [27:55] experience. All of those questions [27:56] filter through. And where we're at from [27:58] a a consumer side is, you know, we have [28:00] a great brand, we have a global brand [28:02] that people trust, and we're trying to [28:04] think through, you know, what do [28:06] consumers what are the actual real [28:08] consumer problems? How do we solve them [28:10] with the tools at hand? And then we jump [28:13] into the risk evaluation of okay if we [28:15] solve it this way what's the risk what's [28:16] the danger what's the impact to the [28:18] brand what's the impact to the consumer [28:20] and and we're just seeing that what [28:22] consumers are telling us is are you know [28:23] the solutions they want don't [28:25] necessarily revolve around the most [28:28] complex AI products where you have to [28:30] start worrying about well what if this [28:31] goes off the rails from a generative [28:33] perspective it's still pretty well- [28:35] contained it's still innovative it's [28:37] driving you know great business results [28:40] but it doesn't come with that high kind [28:42] of risk tag. I think as we get deeper [28:45] into maturity in the genai space, you [28:47] know, the the challenge for, you know, [28:50] folks in the Fortune 500 that are are [28:52] consumer focused brands is is having the [28:54] discussion about like, well, where do we [28:56] start, right? And I think the risk that [28:58] I've seen across my career is, you know, [29:01] data scientists like the hardest part is [29:03] the model, right? like getting the model [29:04] the API to respond correctly whether [29:06] it's a rag it's fine-tuning it's a [29:08] homebuilt model it's it's the single [29:11] hardest part and then as it gets more [29:14] and more difficult data scientists are [29:15] like well let's talk about scale let's [29:17] build the engineering around this thing [29:18] without realizing that until you have [29:20] that final mile working there's nothing [29:22] to scale right and so it's that's the [29:24] hard conversation that I think [29:25] businesses are going to have to have and [29:27] whether it's a you know buy first build [29:29] later or buy firsts customize later uh [29:32] mentality it all comes down to there are [29:34] going to be ways that you have to test [29:36] what that risk space might be, what the [29:39] benefit to consumers are. Some of it's [29:41] going to be throwaway and not get [29:43] obsessed with the scalability of the [29:45] infrastructure underneath those things. [29:47] >> And Ryan, can you because I assume this [29:49] is a conversation you have a lot with [29:51] clients. So like you know they they'll [29:53] say like should we buy you know what is [29:55] the risk here if I'm going to give you [29:57] access to data all those things. So so [29:59] how do you manage that in those [30:00] conversations? [30:02] >> Yeah. Um, two thoughts here. Um, one, a [30:06] lot of the companies we work with, I [30:08] think are a little bit too conservative [30:09] with their brands. I mean, Domino's and [30:12] Vanguard, I don't know if I would come [30:13] in and say you're too conservative, but [30:15] like um um some of our customers, [30:18] there's a there's one that does um um uh [30:20] it's a payments company and uh their [30:23] voice AI is quite good. It can answer a [30:26] lot of questions and yet the sensitivity [30:28] on escalating to a human is like all the [30:31] way dialed up. search like I want to [30:32] talk to no problem we'll get you to a [30:33] person right away right it's like it's [30:35] super super sensitive and it's like yeah [30:37] the person's kind of going to tell you [30:38] the same thing on this question [30:40] sometimes so maybe repeat it or be a [30:43] little bit um um more brave in terms of [30:45] like letting people just we're not going [30:47] to get you to a person I can really [30:49] answer your question seriously um and [30:51] and you know the news headlines of that [30:53] airline doing the refund I don't even [30:55] remember what airline that was anybody [30:56] remember which airline that was right so [30:58] it's like the brand risk is not that [30:59] high um but I think the other part of [31:02] this too is there is a lot more it's [31:05] interesting there's a lot more people [31:06] who are wanting to build um and and you [31:09] know vibe coding is a thing and um we've [31:11] had people tell us they're going to vibe [31:13] code a workforce management system and [31:15] um I was like good luck to you we'll [31:18] talk to you in a year or something [31:19] probably and we'll try to untangle it [31:21] there are some things that people I [31:23] think right now see as possible to build [31:25] and kind of like skipping past the other [31:27] 20%. The hard part of a lot of these [31:29] systems, AI systems, ML systems, just [31:32] software systems, is it's not doing the [31:34] demo. It's not doing the first thing. [31:36] It's scaling it to millions, tens of [31:39] millions of users and and and getting [31:42] the data right and doing the evals, [31:44] right? And everything that you don't [31:45] see. So, um um we're often telling [31:47] people, yeah, you could maybe build [31:50] this, but you know, decide where you [31:52] really do want to spend your innovation [31:53] chips because this is harder than it [31:54] looks. And Ashish, what is you know I [31:58] assume that also is an issue when you [32:00] come into an organization and I think I [32:03] heard you you might have talking about [32:05] this with Bryce earlier is like you have [32:07] established systems already. So you have [32:09] companies well we already bought this we [32:11] already have a contract here. So what is [32:14] you know how do you approach that to say [32:15] like all right I'm glad you still have [32:17] like five years on this contract it it [32:19] it's going to go to waste now. So, how [32:21] what what do you do when you walk in and [32:23] they say, "Oh, we already bought some [32:25] things and now you have to reconfigure." [32:28] >> That's that's a great one. I think a [32:30] couple a couple of things. One is and [32:33] this is uh for for organizations [32:36] thinking about adopting AI technology in [32:38] general. There there is around 2020 2021 [32:43] products and technologies which were [32:45] started there or were legacy systems [32:48] before that. There is a huge dividing [32:50] line and I would I would sum [32:54] is working which was you know started in [32:58] 2010s or before that uh they are just [33:01] most likely sprinkling genai on top and [33:04] they are not built with this technology [33:07] at its core. So every week every day [33:11] which you spend with that old tool is [33:13] value which you are losing from the [33:15] marketplace. I really mean it and this [33:17] is the first time in tech especially in [33:20] CX when customers will start realizing [33:23] if you are using AI properly or not and [33:27] it'll become a brand differentiation. So [33:29] it's no longer like so those [33:31] conversations are becoming easier for us [33:33] because people are willing to put those [33:36] long contracts on the side. The other [33:38] thing I and I I shouldn't say this as a [33:40] technology my sales team will hate me [33:42] but we we ask our customers to measure [33:46] us every few months because the [33:48] technology is changing so fast. We talk [33:50] about this in analyst briefings and and [33:52] you know expert conferences as well. If [33:55] the underlying models are changing every [33:56] week, why shouldn't why should analyst [33:59] quadrants be every quarter and and waves [34:02] be every year? You know that doesn't [34:03] make any sense. It's just the old way of [34:05] buying software. Right? where I'm going [34:07] with this the five-year dam contract [34:08] doesn't matter because of the underlying [34:10] model changed the benchmarks are [34:12] changing on a weekly basis on LLM [34:13] technology right so the five-year [34:15] contract is is is just that right uh now [34:19] the question very your precise question [34:21] was how do you start is start with [34:24] lowrisk use cases with jai which can be [34:26] a systems on top right and then you can [34:28] get on the journey of automation [34:30] augmentation and full automation but [34:32] start with lowrisk use cases that's how [34:35] we preach [34:36] and Jing. Um, I think we have a mic [34:38] coming for you. Oh, there we go. [34:42] >> Oh, swapping out. I like Vanguard [34:45] obviously, you know, I know people who [34:47] have pretty much their entire life kind [34:50] of up in Vanguard. So, like I can't [34:53] imagine the pressure that your company [34:55] is under to make sure people feel cared [34:57] for. How did you approach this? Like you [35:00] you've been there 15 years. How how did [35:03] you build this and say like okay if [35:04] we're going to take the steps here this [35:06] is how we do it how did that evolve [35:10] >> yeah so uh you know we are in financial [35:12] services so it's very heavily regulated [35:16] industry so we have regulators to uh to [35:20] um answer for right but on top of that [35:23] uh Vanguard's reputation is one of its [35:26] biggest asset frankly so people trust [35:29] Vanguard and we want to make sure [35:31] everything we do enforces that trust [35:34] rather than erode it. So I think we have [35:37] um pretty high standards for everything [35:39] we push out especially when it comes to [35:41] genai. You know the power of genai those [35:44] large language models comes with the [35:47] intrinsic um you know challenges like [35:50] hallucination. So we want to harvest the [35:53] power but we want to manage those [35:56] intrinsic risks. um in early days as as [36:00] um Orion and Bryce mentioned we rejected [36:04] solutions that are like 97% accurate [36:07] because the business just says what [36:09] about you are the 3% who got the right [36:11] answer and you miss opportunity to trade [36:14] to put your money in the market and and [36:17] when you put yourself in that truth and [36:19] you you you do feel very bad for our [36:21] clients. So um so what we are doing is [36:25] that while we are very actively [36:27] experimenting with those growth [36:30] experience improvement use cases we are [36:32] also investing to build the AI [36:34] governance side of muscle. So we have [36:37] published enterprise AI policies that [36:40] are defines eight standards for our [36:42] trusted AI systems and every AI [36:46] solutions we build needs to meet those [36:48] standards and we're connecting our [36:50] evaluation capabilities with those [36:53] standards so that as the teams to [36:56] building the solutions they're building [36:58] evolving towards those meeting those [37:00] eight standards and we're building tools [37:03] that are allow those massive of evals [37:07] happening right because you can't test [37:09] it's a statistical system probabilistic [37:12] system you have to test it with very [37:14] large volume of usage right you have to [37:16] simulate that and then when we are [37:19] rolling out we always use [37:20] experimentation to make sure we slowly [37:22] pilot so and then we can see the real [37:24] benefit and we get the insights through [37:26] those experimentations and to improve [37:28] the solution and finally very keen on [37:32] monitoring like we're looking for very [37:34] strong monitoring vendors So that when [37:36] our GNI solutions are in the in the [37:38] field, we know we have basically monitor [37:41] every single interactions and if real [37:44] time we intercept something harmful, we [37:46] can stop we can stop that. So those are [37:49] sort of different parts of our the other [37:52] defense side of the ecosystem if you [37:54] will. We I mean if you don't have a [37:56] strong system like that, you really [37:58] cannot scale up your direct genai client [38:02] experience use cases. And I assume that [38:04] especially like you mentioned it's such [38:06] a heavily regulated space that you know [38:09] regulations may not keep up with it. So [38:11] you sort of have to do the homework to [38:12] say okay if if this was written with AI [38:15] in mind like how would and that's where [38:17] those reports come in. Right. [38:18] >> Right. [38:19] >> Got it. [38:19] >> Right. That's exactly right. Um and I [38:21] think there is a for me um having [38:24] observed the industry for a while there [38:25] is a disconnect between the enterprise [38:28] uh companies trying to implement AI [38:30] versus the AI vendor providers uh [38:32] capabilities because vendor is very [38:34] focused on pushing out capabilities. [38:36] They're not so keen to build those [38:38] evaluation capabilities. So it's left [38:41] the companies are left on their own [38:44] >> to you know I often ask the vendors so [38:46] how accurate is your solution? often my [38:49] the answer I get is you have to do your [38:51] own evaluation. [38:53] >> The good ones are spending a lot of time [38:55] on evaluation. [38:58] >> Well, and we we have about five minutes [39:00] left and I would love to to end with [39:02] each of you going over because this this [39:06] space is changing so quickly. You know, [39:08] it seems like every every week there's [39:10] like five different news stories that [39:11] changes everything. What what is top of [39:15] mind for you over the next six months or [39:17] so that you're like, "Okay, I I'm [39:18] watching this. This is going to affect [39:20] my life in the space." Um Ryan, do you [39:23] want to go first? [39:25] >> Um sure. Yeah. Over the next six months, [39:27] I I I think it's a version of what [39:30] Shisha described like the incredible [39:32] pace means that we as a vendor are [39:34] getting a lot of pressure from people to [39:36] move faster, to do stuff on [39:38] month-to-month contracts, which is very [39:39] painful. do not like that's be kind to [39:41] your vendors you know at least one year [39:43] please because like you do need like [39:45] time to deliver results but we also see [39:47] that pressure as a as as builders who [39:50] are evaluating our own tech stack um I I [39:54] for example we have an AI voice agent [39:56] and the AI voice landscape seemed very [39:58] settled six months ago there were a [40:00] couple companies that have raised a lot [40:02] of money a lot of hype like they are [40:03] clearly very smart and we bet on them [40:06] and it hasn't gone well and the [40:08] landscape's completely changed And um [40:10] you know it used to be that you could [40:11] say AWS we're just going to build on AWS [40:14] and everything's good. And now you [40:16] really do have to be a lot more um [40:19] either able to read the tea leaves which [40:20] I think you know nobody can predict the [40:22] future or able to metabolize change [40:24] really quickly. And so what I'm really [40:25] pushing with our organization is when [40:27] things change when models drop new [40:29] providers come out when um our customers [40:32] have different needs how can we be super [40:34] super super reactive to that? [40:37] >> Nice. It's a great question. You know, [40:40] one of the things that in my career has [40:41] has paid dividends is thinking about [40:43] consumers from what we used to say womb [40:45] to tomb, right? Like the whole customer [40:47] life cycle. And so one of the things I'm [40:49] thinking about over the next six months [40:50] is like how do we like what does that [40:53] life cycle look like in the consumer [40:55] genai space not just you know on on [40:57] Domino site but just more generally like [41:00] as people engage with GPT5 as you have [41:02] you know perplexity out there doing [41:04] exciting things as you've got claude [41:06] getting a little more consumer friendly [41:08] and and facing um Grock you know getting [41:11] some usage as well and growing you know [41:14] what is what does the womb to tomb of a [41:16] customer look like? How do you how do [41:18] you mobilize that at once for kind of [41:19] strategic direction at the beginning of [41:21] the customer life? How do you find those [41:22] moments where people are going to those [41:24] chat clients and expressing frustration [41:26] about your brand? You know, there there [41:28] was the LinkedIn post that went viral. I [41:29] think we all remember it where it's [41:30] like, "Hey, by the way, Google's [41:32] indexing everybody saved chat GPT [41:34] searches and we were all like, "Oh yeah, [41:35] old school search engine language, you [41:37] know, querying." Uh, you know, those [41:40] kind of moments are really important [41:41] because as a brand, we want to [41:44] understand how we can make better [41:46] magical pizza experiences centered [41:48] around delicious hot industryleading [41:51] pizza. And so, how does chat GPT and and [41:54] all of these agents change what the [41:55] landscape looks like? Is it generative [41:57] engine optimization from a a strategy in [42:00] parallel with SEO? Is it like a way of [42:02] like figuring out how people ask for [42:04] pizzas? Is Aentek solutions go live? [42:06] like how does the ordering experience [42:08] change and then how does the feedback [42:10] experience change? Do people go to, you [42:11] know, chat GPT, you like have a problem [42:13] with, you know, a friend and you're [42:14] like, "Hey, give me some advice on how [42:15] to handle this." Are they going to chat [42:17] chat GPT and being like, "Hey, I had a [42:19] problem with my pizza experience. How do [42:20] I handle this?" And being able to use [42:22] those systems holistically to figure out [42:24] what customer experience looks like. [42:26] That's kind of a thing I'm thinking [42:27] about over the next six months as [42:29] someone who, you know, owns in many ways [42:30] the voice of the customer for Dominoes. [42:33] >> Interesting. Ashish [42:35] >> I'm curious about how how do our [42:37] customers think about performance [42:41] pricing [42:42] uh personalization [42:45] uh and the reason I mention that is that [42:47] there are two ways of building sort of [42:49] underlying AI models one is the closed [42:51] open AAI Google way cloud way and then [42:54] there's this open-source very active [42:55] open source universe we are personally [42:58] more you know we we believe in the open [43:00] source world we believe in building our [43:02] own models and so on and There is a huge [43:04] ROI difference between building your own [43:07] models and especially in enterprise [43:09] organizations and using sort of closed [43:13] models. And I think as organizations and [43:16] enterprises, we talk to very large [43:18] customers who are already recognizing [43:19] this. As organizations recognize the [43:22] impact of performance, pricing, [43:26] personalization and privacy or like [43:28] these are four underlying things which [43:30] drive enterprise adaption. And as [43:32] enterprises will recognize that the [43:34] world will enterprise AI will move more [43:37] and more towards underlying open source [43:39] models. China is doing incredible work [43:41] there. Meta is trying to figure out its [43:44] way back again. I'm really curious how [43:46] that plays out and that word is changing [43:48] every week, right? So the ability of AI [43:51] companies and organizations to adopt [43:53] those open source models in their stack [43:57] will hugely transform it because there's [43:59] a very different level of expertise [44:01] required to call OpenAI's API and put it [44:04] in your product versus build an AI stack [44:07] from ground up which we have done to be [44:09] able to really scale an AI AI product [44:12] for the enterprise when you are a lot of [44:14] our organizations have a 100 million [44:16] data points a billion data points coming [44:18] through. Good luck trying to push that [44:20] into OpenAI's API and getting a two-cond [44:22] latency back from it. Right. That's what [44:25] maturity would look like in AI. That's [44:26] what I'm looking to solve. [44:28] >> Jing, [44:30] well, for us for the next six month, I [44:33] think I think we're um we can as a [44:36] company, we're continue we're continuing [44:38] to invest in uh AI uh in a pretty big [44:42] way. So there is a big question around [44:44] when you think about AI investment as a [44:46] portfolio where do you want to invest [44:49] right? So I I think that has to be [44:50] rooted in your company's in my company's [44:53] uh strategy when it comes to customer [44:56] experience versus growth versus [44:58] operational efficiencies. uh where are [45:01] um where do you want to make a huge [45:03] difference today in the market space [45:05] relative to our competitors and then I [45:08] think there is also recognition as Ryan [45:10] said the technology text stack is [45:12] evolving so fast it's actually a [45:14] challenge for us to keep to keep up we [45:17] are like always evaluating new things so [45:21] how that kind of knowledge can inform us [45:23] where you want to double down investment [45:25] I think a lot of workflow automation [45:29] those are Backend processes in my in my [45:32] mind are the best place to invest right [45:34] now from ROI perspective but at the same [45:37] time you cannot just let it go lose lose [45:39] the race on front end customer [45:42] experience experimentation but your [45:45] vendor space is changing very quickly [45:46] like voice vendors for example and [45:49] multimodel models are coming out better [45:50] and better so and then you have to [45:53] invest in the foundation too like AI [45:55] ready data your API environment so where [45:58] is your AI investment strategy that is [46:01] very much aligned with your company [46:03] business strategy and allow you to you [46:06] know generate the highest return is is [46:09] something we're thinking about very care [46:11] >> thank you so much and thank you all for [46:13] sharing your knowledge um we covered a [46:15] lot of ground I really appreciate it [46:17] thank you [46:18] [Applause]