Google, Amazon AI Chiefs Say Infrastructure Gaps Stall Enterprise Scale

谷歌与亚马逊AI主管：基础设施短板阻碍企业级规模化 企业对生成式AI的应用受制于数据流水线集成以及每六个月一次的模型快速淘汰周期，而非基础模型的推理能力。 该专家组探讨了生成式AI在企业语音、银行业风险管理、电子商务和云模型托管等领域的生产级部署，并对比了专有托管API与自托管开放权重架构。嘉宾们详述了从提示与上下文工程转向参数高效微调（PEFT）和LoRA的过程，以化解延迟、准确性及领域适配瓶颈。他们还探讨了实际运营中的挑战，如多模态幻觉、六个月的模型淘汰周期以及对话语音代理的经济权衡。展望未来，与会发言者为通用人工智能（AGI）的过度炒作降温，认为真正的变革性价值存在于基础数据流水线、统一多模态和生命科学领域。 亚马逊采用基于大语言模型（LLM）的生成对抗网络（GAN）架构——由LLM同时充当生成器与判别器——以精选的人工示例小子集为条件，生成多样且逼真的合成数据。 (3:19) 尽管预计到2030年企业客户体验支出将实现90%的自动化，但由于语音疲劳和传统电话系统的限制，目前仅有不足1%的语音基础设施采用了生成式模型。 (3:19) 企业模型选型呈现三层分布格局，涵盖从追求开发速度的开箱即用专有API，到满足严苛数据驻留要求、HIPAA合规性及拥有完全基础设施控制权的自托管开放权重模型。 (7:44) 生产架构的重心正从简单简短的提示工程，转向全面的上下文与内容工程流水线，以此为紧凑高效的模型构建领域知识结构。 (11:05) 提示工程已从跨模型的试错重写演进为自动化提示优化流水线，由仲裁LLM以编程方式跨模型家族重写并评估提示。 (14:58)

Enterprise generative AI adoption is pivoting away from monolithic, prompt-only architectures toward modular, parameter-efficient systems tailored to strict latency, governance, and economic constraints. Panelists from Google Cloud, Amazon, Fifth Third Bank, and Rime argue that practical business value relies on mastering data integration pipelines, targeted fine-tuning, and multi-model orchestration rather than pursuing generalized intelligence. Deployment Tiers Across Latency and Governance (7:44) Enterprises navigate a three-tier model consumption spectrum on platforms like Google Cloud's Vertex AI, choosing among proprietary managed APIs (such as Gemini or Claude), managed open models, and self-hosted open-weight architectures like Llama. Keelin McDonell notes that selection depends heavily on regulatory and technical needs: high-throughput summarization benefits from managed APIs for developer velocity, whereas applications handling sensitive data—such as therapy chatbots constrained by HIPAA and strict data residency rules—rely on fine-tuned open models deployed on private regional infrastructure. Operational latency also dictates system boundaries. Hailong Wang observes that in financial services, latency-tolerant workflows can leverage large proprietary APIs with long-context prompting, but latency-critical banking functions require open-weight small language models (SLMs) fine-tuned on curated domain datasets. Furthermore, to mitigate multi-tenant cloud issues like noisy neighbors and variable peak latency, enterprise deployments increasingly depend on provisioned throughput to guarantee isolated compute capacity. Moving Beyond Prompts to Parameter-Efficient Tuning (21:23) While zero-shot, few-shot, and chain-of-thought prompting serve as low-friction entry points, panelists emphasize that enterprise production requires robust context and content engineering before model invocation. Because prompts leave model weights static, their performance caps when confronting domain-specific constraints or latency bottlenecks. However, full fine-tuning across hundreds of billions of parameters carries prohibitive compute costs and risks catastrophic forgetting, where base pre-trained knowledge degrades. Practitioners instead adopt Parameter-Efficient Fine-Tuning (PEFT) frameworks like LoRA and QLoRA. Lily Clifford highlights a speech synthesis scenario: when an existing voice model mispronounces new pharmaceutical terms like "Wegovy," full retraining risks degrading other vocal representations and forces enterprise clients to provision new hardware checkpoints. Training lightweight additive low-rank matrices resolves terminology gaps with minimal compute overhead. While massive context windows spanning millions of tokens allow prepending custom context directly into prompts, PEFT adapters and fine-tuned open weights remain essential where strict data sovereignty or local execution is demanded. Multi-Model Orchestration and Voice Economics (33:37) Interactive voice systems highlight the necessity of multi-model pipelines over monolithic LLMs. Complex enterprise tasks, such as hospital insurance verification, require navigating automated interactive voice response phone trees and conducting 45-minute conversations with human agents across structured questionnaires. In production, these voice agents orchestrate approximately 15 specialized, lightweight models—such as smaller Gemini variants—each dedicated to routing, state tracking, and backend database calls. This conversational flexibility transforms call center economics. Clifford notes that the enterprise customer experience sector accounts for $600 billion to $700 billion in annual spend, with only 3% to 4% currently allocated to software, while Gartner projects automated inbound calls will grow from 30% in five years to 90% by 2030. Fine-tuned open models have raised order completion rates from ~75% to 85–90% in fast-food settings. However, generative models are inherently chattier, expanding average handling times from two minutes to four or five minutes. Even as token prices fall, prolonged call lengths significantly increase text-to-speech audio synthesis and operational compute costs. Risk Tolerance, Multimodal Hallucinations, and Obsolescence (31:35) Addressing model reliability, Wang argues that hallucinations must be reframed as an inherent consequence of model autonomy and probabilistic behavior rather than treated as conventional software bugs. Organizations must define explicit risk tolerances and design guardrails proportional to the autonomy granted to an agent. In multimodal systems, Abhishek Bafna identifies a modality collapse issue where models over-rely on text tokens and ignore paired visual inputs; his team injects deliberate noise perturbations into text prompts to force the network to distribute attention across visual features, curbing hallucinations. Concurrently, corporate IT faces unprecedented lifecycle friction. McDonell observes that while corporate software historically relied on multi-year upgrade cycles, modern generative models face functional obsolescence within six months. Furthermore, broad industry benchmarks like MMLU often mask silent regressions in niche enterprise tasks when upgrading base models, forcing organizations to build continuous, proprietary evaluation pipelines. The panel concludes that immediate commercial impact lies in practical multimodal integration, computational biosciences, and refined operational pipelines rather than near-term artificial general intelligence (AGI), which panelists view as overhyped. Keelin McDonell emphasizes that the primary industry bottleneck is not foundational model reasoning capacity, but data pipelines, noting that the vast majority of existing model capabilities remain unexploited due to enterprise integration constraints.

来自 Google Cloud、Amazon、Fifth Third Bank 与 Rime 的行业专家指出，生成式 AI 的企业落地已从盲目追求基础模型，转向基于合规、延迟与成本的系统工程；企业竞争力的关键不在于追逐通用人工智能（AGI）假说，而在于理顺数据管线、选择适配的架构分层，并结合上下文工程与参数高效微调解决垂直业务问题。 架构选型：托管API、专有吞吐与开源微调 (7:44) Google Cloud 的 Keelin McDonell 提出了三层模型消费光谱：一是 Gemini、Claude 等专有托管 API，具高开箱即用质量与低 Token 计价，适合快速原型与常规文本摘要；二是开源模型的托管服务；三是自托管开源权重模型（如 Llama）。在医疗心理咨询等强合规与数据主权场景中，受 HIPAA 规范制约，企业必须采用私有部署的开源模型以掌握端到端控制权。针对公有云多租户环境中的“吵闹邻居”和峰值延迟波动，云厂商则通过“预配吞吐量”（Provisioned Throughput）提供专属计算资源。 金融行业对架构的权衡则高度受制于延迟。Fifth Third Bank 的 Hailong Wang 指出，高延迟容忍业务可采用基于专有 API 的大模型及长上下文提示词方案；但在严苛延迟限制下，银行必须采用经 LoRA 或 QLoRA 微调的开源小语言模型（SLM）。Google Cloud 对此提炼了包含 15 个问题的评估框架，系统评估企业合规、数据驻留与性能要求。 适配演进：从上下文工程走向参数高效微调 (11:05) 在企业生产环境中，单句提示词的局限日益明显，推动技术团队将重心转向“内容与上下文工程”，通过结构化领域知识训练小而强悍的专用模型。提示工程自身也从基础的零样本、少样本与思维链，演进至软提示（Soft Tuning）以及通过仲裁模型自动重写优化的流程，化解了早期跨模型迁移的不兼容痛点。在复杂的多轮对话中，单一提示词被多模型编排取代；例如在医疗收益周期管理中，系统需协调约 15 个轻量级模型分工处理意图路由、保单查询及处方核对。 当业务需要超越外部提示词所能达到的精度或更低推理延迟时，必须转向权重调整。全量参数微调因算力成本高昂且伴随“灾难性遗忘”风险，已逐渐被 LoRA 和 QLoRA 等参数高效微调（PEFT）取代。Rime 的 Lily Clifford 指出，语音合成模型在医疗场景中常遇到如 Wegovy 等生僻药物发音错误，若采用全量微调不仅破坏其他声学表征且会迫使客户更换硬件；通过低秩自适应仅调整极小比例参数，企业仅凭约百条样本即可实现专业术语校正，同时完整保留基座模型能力。 落地代价：幻觉容忍、业务经济学与静默退化 (33:37) 生成式 AI 在垂直业务中带来了显著的经济与行为转变。在年支出达 6000 亿至 7000 亿美元但软件渗透率仅 3% 至 4% 的企业客户体验（CX）领域，语音大模型应用占比尚不足 1%。Gartner 预测 5 年内企业呼入电话自动化率将达 30%，到 2030 年将达 90%。达美乐披萨在北美约 40% 门店内上线语音大模型后，点餐成功率由传统固定脚本系统的 75% 提升至 85%-90%；但由于大模型回复更为冗长，平均处理时长（AHT）从 2 分钟翻倍至 4-5 分钟，显著增加了语音合成（TTS）成本。然而，由于订单转化与客户满意度明显提升，企业普遍选择接受这一算力开销。 在质量与架构挑战上，Hailong Wang 强调大模型属于概率型行为模型，幻觉是赋予其自主推理能力时的必然副产物，团队需根据机构风险容忍度设立护栏。针对多模态大模型在图文理解中过度依赖文本先验的失衡问题，Amazon 的 Abhishek Bafna 提出了向文本注入扰动噪声的技术，强制模型注意力聚焦于视觉特征。与此同时，Keelin McDonell 指出模型迭代严重颠覆了 IT 周期：通用基准（如 MMLU）走高的同时，垂直特定任务频频发生隐性退化；企业软件长达数年的生命周期被急剧压缩为 6 个月的模型淘汰节奏，加剧了持续验证负担。 未来路径：原生多模态、数据管线与AGI反思 (42:05) 关于未来演进，专家一致看好计算生物学（如蛋白质折叠与药物设计）等前沿场景，并指出底层架构将从模块拼接转向原生端到端融合文本、音频与图像的多模态系统；同时，文本扩散等超越标准 Transformer 的新范式也已崭露头角。Keelin McDonell 强调，多模态输入能成倍释放实际办公价值，但行业目前对现有模型能力的利用尚不足 1%，核心瓶颈并非模型智力上限，而是底层数据管线的整合效率。 对于备受瞩目的通用人工智能（AGI），与会者普遍认为当前叙事存在过度炒作。人类婴儿的学习机制与基于统计词元的深度学习有本质区别。如同早期互联网仅凭 TCP 与 HTTP 等通信协议便重塑全球一样，现阶段 AI 技术在过去 12 到 24 个月内对工作生活范式的重塑，已足以构成深刻的工业级变革，无需依赖科幻式的 AGI 设想。 本场研讨表明，生成式 AI 的商业落地已步入注重投入产出比与系统工程的深化阶段。未来的核心挑战不再是盲目追求模型规模，而是在严格的合规、延迟与算力预算下，构建高韧性的数据管线与持续评估机制。如何在模型快速过时的迭代周期中维系业务稳定性，是企业必须应对的核心课题。

## Background 本内容由 YouTube 频道 Ai4 于 2025 年 9 月 29 日发布。页面描述显示，这是一场聚焦生成式 AI 模型落地应用与底层技术策略的圆桌研讨会，汇集了来自亚马逊、五三银行、诺德斯特龙、Rime 及谷歌的专家，共同探讨模型训练、微调、提示词工程以及跨行业攻克复杂 AI 挑战的最佳实践。 ## TL;DR 圆桌讨论围绕生成式 AI 在企业级落地中的真实工程挑战展开，涵盖从对抗合成数据生成、金融风控的 Agentic AI 与 Graph RAG 演化，到实时高保真语音客服的实际架构。专家们系统拆解了从单纯提示词工程向“上下文工程”、PEFT（如 LoRA/QLoRA）及开源模型自建微调的选型权衡。针对生产环境中严峻的幻觉、推理延迟、成本失控与大模型“六个月寿命”导致的 IT 迭代阵痛，嘉宾们提出了预置吞吐量、多模态文本加噪扰动等应对手段。最终，全场对 AGI 概念进行了理性去魅，一致强调现阶段应当专注于打通工程管线以释放现有大模型的未开发潜能。 ## Atmosphere & style 整场研讨会散发着高度务实、直面工程现实且充满行业洞见的专业极客氛围。嘉宾们既没有空洞贩卖未来概念，也不盲目推崇 AGI 叙事，而是以一线技术负责人的冷静视角，幽默且犀利地吐槽了语音点单把用户聊到“皮质醇飙升”、达美乐电话闲聊半小时导致账单爆表等真实踩坑经历。交流节奏紧凑、多模态与金融风控等垂直视角的碰撞极具信息密度，让听众在脚踏实地的工程技术探讨中体会到大模型落地从“玩具 Demo”迈入严苛工业级生产的真实阵痛与变革力量。 ## Key takeaways - [03:14](https://www.youtube.com/watch?v=vu5GebTRLeA&t=194s) 亚马逊通过双 LLM 驱动的 GAN（生成对抗网络）架构，结合少量精选人类真实数据，成功解决了传统合成数据机械感重、容易被识别的痛点，生成兼具高逼真度与多样性的数据。 - [03:14](https://www.youtube.com/watch?v=vu5GebTRLeA&t=194s) 五三银行（Fifth Third Bank）历时多年推动金融模型全生命周期数字化，通过融合 Agentic AI、RAG 与 Graph RAG 技术构建协同网络，实现了严格合规环境下的半自主决策。 - [05:57](https://www.youtube.com/watch?v=vu5GebTRLeA&t=357s) 企业客户服务市场年支出高达数千亿美元，但 AI 语音调用渗透率尚不足 1%，语音交互体验的关键不仅在于后端逻辑，更在于高拟真、低焦虑感的自然语音生成。 - [05:57](https://www.youtube.com/watch?v=vu5GebTRLeA&t=357s) 大模型消费模式主要分为专有托管 API、开源模型 API 托管与开源权重私有部署三类；强监管、高敏感的行业（如医疗、金融）更倾向于自托管开源模型以确保数据合规与主权。 - [11:05](https://www.youtube.com/watch?v=vu5GebTRLeA&t=665s) 简单提示词不足以支撑复杂企业系统，行业正转向涵盖领域知识与结构化内容管道的“上下文工程”（Context Engineering），并演进出软微调（Soft Tuning）与自动优化等高级范式。 - [17:29](https://www.youtube.com/watch?v=vu5GebTRLeA&t=1049s) 企业级复杂长流程应用（如长达 45 分钟的医疗电话核销）无法依靠单提示词完成，通常需要编排十几个专注特定意图分类的小模型协同并结合函数调用来实现。 - [21:23](https://www.youtube.com/watch?v=vu5GebTRLeA&t=1283s) 全量微调面临高昂成本、灾难性遗忘以及破坏预训练通识的风险，LoRA、QLoRA 等 PEFT（参数高效微调）技术通过外挂低秩适配器成为企业轻量化定制的首选方案。 - [25:11](https://www.youtube.com/watch?v=vu5GebTRLeA&t=1511s) 随着大模型上下文窗口扩大至百万级 Token，许多原本需要微调的场景可转为 Prompt 上下文前置注入，形成了“无极端低延迟诉求则优先做 Prompt 上下文设计”的工程实践原则。 - [29:24](https://www.youtube.com/watch?v=vu5GebTRLeA&t=1764s) 企业级落地需权衡幻觉、延迟与成本；公有 API 存在“吵闹邻居”与延迟波动问题，云厂商推出的预置吞吐量（Provisioned Throughput）可提供类似于独占算力的性能确定性。 - [31:35](https://www.youtube.com/watch?v=vu5GebTRLeA&t=1895s) 基座模型领先寿命缩短至约 6 个月，频繁的模型更迭导致版本退化（通用基准提升但细分场景劣化），给企业 IT 的基准评测、合规审查与系统重构带来沉重负担。 - [33:37](https://www.youtube.com/watch?v=vu5GebTRLeA&t=2017s) 语音智能客服引入 LLM 后成功率虽大幅提升，但也带来了回复冗长、TTS 字符消耗暴增、单次通话时长增加等成本失控与按量计费挑战。 - [36:53](https://www.youtube.com/watch?v=vu5GebTRLeA&t=2213s) 大模型本质是概率与行为感知系统，幻觉源自赋予模型的“自主性”（类似初级分析师可能犯错），企业应依据具体的风险承受度划定容忍边界，而非幻想绝对零误差。 - [38:36](https://www.youtube.com/watch?v=vu5GebTRLeA&t=2316s) 针对多模态大模型过度依赖文本先验而忽略图像物理表征的幻觉问题，可通过向文本通道注入噪声的扰动技术强制模型平衡注意力权重，锚定真实图文线索。 - [39:58](https://www.youtube.com/watch?v=vu5GebTRLeA&t=2398s) 嘉宾普遍对当下 AGI 的过度炒作持审慎态度，认为行业的核心突破应聚焦在分子生物计算、原生统一多模态模型以及打通工程数据管线以释放现有模型 99% 的未开发潜力。 ## Detailed walkthrough ### [00:04](https://www.youtube.com/watch?v=vu5GebTRLeA&t=4s) 圆桌嘉宾背景介绍与开场 本次圆桌讨论聚焦于生成式 AI（Generative AI）模型的落地应用场景与核心技术。主持人 Shubham Shukla 首先进行了自我介绍，他目前在零售百货巨头诺德斯特龙（Nordstrom）担任数据科学家，主导公司内部的生成式 AI 研发工作，重点利用生成式技术提升企业员工的内部生产力；在此之前，他曾深耕商品属性提取（Product Attribution）、推荐系统以及个性化搜索领域，并曾在亚马逊（Amazon）担任过数年的机器学习工程师。 随后，四位来自不同领域的行业专家依次介绍了各自的业务背景与技术方向： - **Lily Clifford**：语音合成初创公司 Rime 的首席执行官兼联合创始人。Rime 总部位于旧金山，专注于为企业级语音应用开发高拟真度语音合成模型。Lily 坦言大家日常拨打银行或电信运营商的免费客服电话体验往往非常糟糕，因此她与两位联合创始人 Brooke 和 Aries 创立了 Rime，旨在彻底重塑企业客户支持电话的听觉交互体验。目前北美达美乐披萨（Domino's Pizza）已有约 40% 具备生成式 AI 电话点单功能的门店接入了 Rime 的语音模型。 - **王海龙（Harry Wang）**：五三银行（Fifth Third Bank）模型风险管理分析总监兼环境创新应用负责人。他正带领团队推进金融服务领域模型全生命周期管理的数字化转型，探索大模型在严监管金融场景下的合规与落地。 - **Abhishek Bafna**：亚马逊资深应用科学家（Senior Applied Scientist），加入亚马逊已有六年以上，目前在 Alexa 团队负责视频理解（Video Understanding）与多模态模型研发。他拥有深厚的计算机视觉（CV）与机器学习背景，在加入亚马逊前曾从事医疗健康 AI 领域的研究。 - **Keelin McDonell**：谷歌云（Google Cloud）产品经理，负责 Vertex AI 平台上的核心资产库 Model Garden（模型中心）。Model Garden 汇聚了包括谷歌自研的 Gemini、开源领域的 Llama、DeepSeek 以及第三方合作伙伴在内的超过两万个模型，Keelin 的职责是确保开发者和企业客户能够高效检索、评估并使用各类第一方与第三方前沿模型。 --- ### [03:14](https://www.youtube.com/watch?v=vu5GebTRLeA&t=194s) 生成式 AI 实践案例：对抗网络合成数据与金融模型全生命周期 在进入正式提问环节后，主持人 Shubham 邀请各位嘉宾分享各自经历过最具代表性的生成式 AI 落地场景、面临的业务痛点以及底层的技术架构设计。 ```mermaid flowchart LR accTitle: 基于 LLM 的 GAN 合成数据生成架构 subgraph HumanInput["小样本标注数据"] H["精选人类真实数据\n(Human Curated)"] end subgraph GANFramework["GAN 对抗框架 (LLM 驱动)"] G["生成器 LLM\n(Generator)"] D["判别器 LLM\n(Discriminator)"] end subgraph Output["最终产物"] S["高多样性拟真人\n高质量合成数据"] end H --> D G -->|"生成候选数据"| D D -->|"对抗损失反馈"| G G --> S ``` 亚马逊的 Abhishek Bafna 首先分享了合成数据生成（Synthetic Data Generation）的技术突破。在许多前沿场景中，纯模型合成的数据往往具有明显的机械特征、过于容易被算法或人类识别（"too detectable"），缺乏真实人类表达的细微神韵。为了解决这一局限，他们搭建了一套基于生成对抗网络（GAN）的创新架构。在这个框架中，生成器（Generator）和判别器（Discriminator）两个核心组件均由大语言模型（LLM）担任。通过引入小规模经人工精选的高质量真实数据集（Human Curated Data）进行对抗训练，模型能够精准捕捉人类语言中的微妙细节（Nuances），最终生成的合成数据不仅高度逼真、拟人化，而且具备极高的数据多样性，能够广泛迁移并适配至各个不同的垂直业务领域。 紧接着，五三银行的王海龙分享了传统银行在模型生命周期管理（Model Lifecycle Management）数字化转型中的探索。这一转型是一场历时四到五年的渐进式工程： 1. **第一阶段：领域知识沉淀**——全面梳理并数字化银行内部的风控与模型规范等垂直领域知识体系； 2. **第二阶段：AI Agent 落地**——引入基础 AI 智能体辅助日常模型评估与风险审查； 3. **第三阶段：Agentic AI 协同网络**——构建多智能体协同系统，全面融合检索增强生成（RAG）与图检索增强生成（Graph RAG）技术，将海量领域专业知识与底层大模型紧密串联，实现模型全生命周期决策的半自动化与自主化，大幅提升金融风控系统的整体运行效率与决策准确度。 --- ### [05:57](https://www.youtube.com/watch?v=vu5GebTRLeA&t=357s) 企业级语音交互的万亿蓝海与大模型选型架构谱系 Rime 首席执行官 Lily Clifford 从商业与用户体验两个维度深入剖析了企业级语音客服的现状。她指出，全球客户体验（Customer Experience, CX）是一个年支出高达 6000 亿至 7000 亿美元的庞大市场，但目前仅有 3% 到 4% 的预算投入在软件上，其余绝大部分仍然依赖高昂的人力成本。据 Gartner 预测，未来 5 年内企业将有 30% 的电话接听实现全自动化，到 2030 年这一比例甚至将飙升至 90%。 然而现实情况极为骨感：在当下一线高频呼叫场景中，基于大模型的语音调用渗透率甚至不足 1%，甚至只有 0.25% 不到。大量跨国巨头（如大通银行 Chase Bank）的客服热线至今仍在使用 20 世纪末由配音演员预先录制的静态音频切片。Lily 强调，无论后台的语音 Agent 逻辑有多聪明，如果生成的合成声音生硬冰冷、"让用户的皮质醇飙升、充满焦虑与压力（raising my cortisol levels）"，用户就会立刻挂断电话。Rime 的核心使命正是从极致的用户体验切入，打造兼具自然情感与亲和力的实时语音生成系统，让 AI 语音真正从概念走向用户愿意日常交流的产品。 ```mermaid flowchart TD accTitle: Vertex AI 客户模型消费模式与选型决策路径 subgraph ConsumptionPatterns["三类模型消费模式 (Consumption Patterns)"] M1["专有托管 API\n(Proprietary Managed APIs)\n例: Gemini, Claude"] M2["开源模型托管 API\n(Open Models as API Services)"] M3["开源权重与私有部署\n(Open-weight Self-hosted)\n例: Llama, DeepSeek, Qwen"] end subgraph DecisionFactors["15项评估清单关键维度"] D1{"数据合规与主权\n(如 HIPAA / 跨区域数据控制)"} D2{"开箱即用与极致性价比\n(低延迟 / 低 Token 成本)"} end D1 -->|"需要自微调 & 严格数据隔离"| M3 D2 -->|"标准化通用客服 / 总结提取"| M1 ``` 谷歌云的 Keelin McDonell 则从底层平台与架构选型的视角切入。在管理 Vertex AI Model Garden 的两万多个模型时，团队将客户对模型的消费模式划分为三大阵营，并通过一份包含约 15 个关键问题的评估清单，帮助客户在不同需求维度下进行精准选型： - **开源权重与自建基础设施模式（Open-weight Models）**：针对拥有私有标注数据集、希望进行深度微调（Fine-tuning）、且对数据主权和部署区域有极高要求的客户。Keelin 列举了一个医疗心理咨询客户的案例：该客户希望利用真实心理咨询师的历史问诊实录训练一个心理治疗辅助对话机器人，受限于美国《健康保险可携性与责任法案》（HIPAA）及各国严格的医疗隐私法规，客户无法将敏感数据上传至第三方专有云端模型。最终，该团队选用了开源的 Llama 模型，在私有受控环境中进行微调并跨多区域合规部署，取得了极佳的业务效果。 - **专有托管 API 模式（Proprietary Managed APIs）**：针对通用客服、长文本总结等标准化场景，客户的核心诉求是开箱即用、极致的推理延迟以及高性价比的 Token 成本。对于此类需求，谷歌会优先推荐调用 Gemini 或 Claude 等成熟的托管 API。 - **开源模型 API 托管模式（Open Models as API Services）**：介于前两者之间，兼顾开源生态的灵活性与无服务器运维的便捷性。 随着各方对实际业务场景与架构选型的探讨告一段落，主持人 Shubham 将话题引向了生成式 AI 落地中的关键实战环节——提示词工程（Prompt Engineering），并向全场嘉宾抛出问题：在提示词工程逐渐变成一门"玄学艺术"的当下，究竟有哪些设计技巧能真正保障大模型输出的稳定性与可靠性，又有哪些常见做法在实践中被证明是低效或无效的。 ### [11:05](https://www.youtube.com/watch?v=vu5GebTRLeA&t=665s) 从提示词到“上下文工程”与系统级设计 王海龙（Harry Wang）首先接过了话题，阐述了在构建生成式 AI 应用时仅依靠简单提示词（Prompt Engineering）的局限性。很多用户以为写一两句提示词就能让大模型搞定一切，但在实际的企业级系统设计中，底层数据和内容架构才是核心。无论调用大模型 API 还是私有部署的小型语言模型（SLM），模型能够理解的本质上全赖输入的内容结构。 ```mermaid flowchart LR accTitle: Prompt and Context Engineering Flow subgraph UI ["用户交互层"] A["1-2 句用户意图提示词"] end subgraph ContextEngine ["上下文工程与数据底座"] B["领域知识整合"] C["结构化内容管道"] end subgraph ModelLayer ["模型调用层"] D["大模型 API"] E["精调小型语言模型 (SLM)"] end A --> ContextEngine B --> C C --> D C --> E ``` 在系统前端，用户或许只需要输入一两句话来表明“我想建什么样的模型”或“我想生成什么样的报告”；但在幕后，团队投入了大量精力进行“上下文工程”（Context Engineering）与领域知识体系的构建。王海龙强调，团队希望追求“小而强大”（small and mighty）的小语言模型，但要实现精准微调与稳定调用，必须把提示词、内容管道和底层数据整合为一套完整的上下文架构。 ### [12:37](https://www.youtube.com/watch?v=vu5GebTRLeA&t=757s) 提示词工程的进阶范式：思维链、软提示与自动微调 来自亚马逊的高级机器学习科学家阿比舍克·巴夫纳（Abhishek Bafna）深入拆解了提示词工程的技术演进。他指出，提示词工程早已脱离了最初的基础指令输入，发展成了一套非常精密的工程范式。提示词之所以是大模型应用落地的首选，是因为它完全不需要修改模型的底层权重，就能在各种定制化场景中快速见效。 巴夫纳系统梳理了主流与进阶的提示词技术： - **经典三分法**： - **零样本提示（Zero-shot Prompting）**：不提供任何示范样本直接下达指令。 - **少样本提示（Few-shot Prompting）**：在上下文中嵌入少量示例，通常能为绝大多数任务提供稳健的基线性能。 - **思维链（Chain-of-Thought, CoT）**：将复杂问题拆解为多个推理子步骤，引导大模型按逻辑推理，输出更合理缜密的回答。 - **角色/人设提示（Role / Persona-based Prompting）**：为大模型赋予明确的角色设定，从而精准规范回答的语气、风格以及专业准确度。 - **软微调（Soft Tuning / Soft Prompting）**：在输入文本的前端或后端拼接可训练的嵌入向量（Trainable Embedding Vectors）。这些非纯文本向量能从示例数据中直接学习并高度浓缩信息，仅占用极少量的 Token 预算，将更多宝贵的上下文窗口留给实际问题描述。 - **自动微调（Autotuning）**：让大语言模型自主迭代优化输入的提示词或输出结果，利用模型本身的能力自动重写并寻找最优表达。 ### [14:58](https://www.youtube.com/watch?v=vu5GebTRLeA&t=898s) 提示词的跨模型迁移与自动化优化演进 Google 产品经理基林·麦克唐奈（Keelin McDonell）分享了提示词工程近几年的行业演变轨迹。三年前，在第一代大语言模型刚起步时（如从 GPT-3 迁移到 GPT-3.5，或从 Google PaLM 迁移到 Gemini，亦或在 Gemini 与 Claude 之间切换），整个行业都深陷在“跨模型提示词迁移”的剧烈阵痛中。 当时不同模型之间的提示词几乎完全不兼容，换个模型就会导致输出结果天差地别。业务团队既没有科学的迁移方案，只能在客户面前盲目“试错与猜测不同大模型的性格”，体验极度割裂。 如今情况发生了根本性改善，提示词的跨模型可迁移性（Transferability）大幅提高。当前典型的评估流程往往是客户将同一组提示词分发给五六个模型，再通过电子表格打分对比。此外，自动化提示词工具也逐渐成熟。例如 Google Vertex AI 推出的“提示词优化”（Prompt Optimization）工具，用户只需按习惯写出原始提示词并调节偏好参数（如“更加正式”或“更加精炼”），系统便会调用一个仲裁大模型（Arbiter LLM）自动完成重写，再将其路由给目标模型执行。 ### [17:29](https://www.youtube.com/watch?v=vu5GebTRLeA&t=1049s) 真实复杂语音 Agent 的工程瓶颈：单提示词的幻觉与多模型编排 Rime 创始人兼 CEO 莉莉·克利福德（Lily Clifford）直言，构建企业级复杂语音应用（Voice Applications）依然极其困难，这与单纯基于知识库回答 FAQ 的一级客服场景有着云泥之别。 以医疗领域的收入周期管理（Revenue Cycle Management, RCM）为例，AI Agent 需要代表医院网络给保险公司打电话核实患者报销权益： - **AI 与 AI 的对抗与穿透**：AI 系统首先要自主拨号并穿透保险公司极其复杂的自动语音菜单（Phone Tree）。 - **长时间多轮人机交互**：接入人工坐席后，通话往往长达 45 分钟，AI 必须严格采集至少 7 项特定的核心信息（例如处方药是否报销、特定报销比例等）。 ```mermaid flowchart TD accTitle: Complex Voice Application Architecture subgraph Inbound ["电话呼叫链路"] A["呼叫保险公司"] --> B["自主穿透 IVR 语音菜单"] B --> C["接通人工坐席 (通话长达45分钟)"] end subgraph MultiModelLayer ["多模型协作中枢"] C --> D["调度层 / 状态机"] D --> E1["意图分类小模型 1 (Gemini)"] D --> E2["意图分类小模型 2 (Gemini)"] D --> E3["意图分类小模型..."] D --> E4["信息提取小模型 15 (Gemini)"] end subgraph Actions ["下游动作"] E1 & E2 & E3 & E4 --> F["函数调用 (Function Calling)"] E1 & E2 & E3 & E4 --> G["数据库读取与回写"] end ``` 克利福德指出，这种企业级场景绝不可能靠“单个提示词 + 单个大模型”搞定。实际生产中，团队往往需要编排 15 个独立的小型 Gemini 模型，分别负责在对话流的不同节点进行细粒度的意图分类与状态追踪，且每个小模型都必须配置专属的提示词。 克利福德幽默地吐槽道，任何人在 Rime 官网点击“实时试听体验”，听到的都只是单模型单提示词的玩具 Demo；而真正的企业级系统若要实现连续对话超过三分钟、支持函数调用（Function Calling）、集成数据库和外部 API，“背后需要海量的提示词工程与模型编排，这其中的复杂度远超外界的想象”。 ### [19:50](https://www.youtube.com/watch?v=vu5GebTRLeA&t=1190s) 迈向参数高效微调（PEFT） 主持人顺着提示词工程的话题引出了全量训练（Full Training）与微调（Fine-tuning）的讨论。传统机器学习时代主要聚焦于全量训练或针对特定任务的微调，而在大语言模型普及后，监督微调（SFT）、LoRA 以及参数高效微调（PEFT）等技术迅速占据主流。 阿比舍克·巴夫纳再次接棒解释道：提示词工程的上限在于“完全没有改变模型权重”，无论输入怎么变，底层仍然是同一个模型。虽然提示词能在一定程度上引导输出，但若要在特定专业领域追求更高的精度、可靠性或改变模型的内在行为，就必须更新模型权重进入微调阶段。面对动辄数百亿、上千亿参数的大模型，全量微调成本高昂，因此 PEFT 技术成为了企业落地中的关键路径。 ### [21:23](https://www.youtube.com/watch?v=vu5GebTRLeA&t=1283s) 全量微调的隐患与 PEFT（参数高效微调）的兴起 全量微调（Full Fine-Tuning）需要更新模型中的全部权重。随着模型规模与参数量的不断膨胀，全量更新不仅会消耗极其高昂的计算资源与财务成本，对训练数据的质量与体量也有着极苛刻的要求——数据必须极其庞大且多样化。如果只用少量特定数据进行全量训练，模型极易陷入灾难性遗忘（Catastrophic Forgetting）或过拟合（Overfitting）。基座模型是在数万亿 Token 上预训练出来的，本身具备非常强大的通用智能；直接覆盖原始权重往往意味着破坏这些宝贵的先验通识。 为了解决这一痛点，参数高效微调（Parameter-Efficient Fine-Tuning, PEFT）应运而生。PEFT 不直接改动原始权重，而是引入低秩矩阵（Low-Rank Matrices）等适配器（Adapters），将其嵌入到 Transformer 层中作为附加权重进行训练。这种方法只微调极小比例的参数，不仅大幅加快了训练速度、降低了算力开销，还完好保留了模型的原始能力。其中，LoRA 和 QLoRA 已成为当前业界最为流行、使用最广泛的微调方案。 ```mermaid flowchart TD accTitle: 全量微调与 PEFT 适配器微调的架构对比 subgraph FullFineTuning["全量微调 (Full Fine-Tuning)"] A1["预训练原始权重 (数十亿/百亿参数)"] --> B1["全部权重覆盖重写"] B1 --> C1["风险：灾难性遗忘 / 高成本 / 过拟合"] end subgraph PEFTApproach["PEFT 微调 (LoRA / QLoRA)"] A2["预训练原始权重 (参数完全冻结)"] B2["外挂低秩适配器 (仅占极小比例参数)"] A2 --> D2["融合输出 (保留通识 + 融入新知识)"] B2 --> D2 end ``` ### [23:22](https://www.youtube.com/watch?v=vu5GebTRLeA&t=1402s) 语音合成与医药名词：为什么微调必须追求“最少参数” Lily Clifford（Rime 创始人兼 CEO）举了一个生动的语音合成落地案例来阐明微调的实际挑战：假设 OpenAI 在 2020 年爬取了当时所有的有声书和 YouTube 视频音频，用语音识别模型转录后训练出一个通用的文本转语音（Text-to-Speech, TTS）模型，并将其部署到医疗健康场景中。然而，像礼来（Eli Lilly）或诺华（Novartis）这样的制药巨头每年都会推出全新的处方药（例如近几年火爆的减肥降糖药 Wegovy、Semaglutide 司美格鲁肽）。老模型根本无法正确发音这些新词。 要解决发音问题，团队只能采集大约 10 个人读这些新药名的样本数据。如果对整个超大模型进行全量微调，模型虽然能学会“Wegovy”和“Semaglutide”的正确读音，却会把之前学到的其他大量语音表征忘得一干二净。更糟糕的是，企业级交付不能动辄跟客户说“请接收一个全新的庞大模型 Checkpoint，并且你们得购买全新的硬件来运行它”。因此，工程上的核心考量变成了：**如何微调最少数量的参数，就能精准达成预期的业务改进？** - **典型发音误区案例**：新型处方药 Wegovy 与 Semaglutide（司美格鲁肽）。 - **全量微调缺陷**：仅靠 10 个样本做全量更新会导致严重的表征丢失与遗忘。 - **硬件与部署成本**：全量微调产生的新 Checkpoint 往往会倒逼企业客户升级底层硬件设施，交付阻力极大。 ### [25:11](https://www.youtube.com/watch?v=vu5GebTRLeA&t=1511s) 托管微调、超长上下文窗口与模型生命周期的博弈 针对 PEFT，Google 产品经理 Keelin McDonell 补充指出，目前像 Gemini 和 Claude 等专有模型平台已经提供了完善的托管微调服务。企业甚至不再需要深厚的 AI/ML 专业背景，只需上传一个包含大约 100 条示例的 JSON 文件，系统就能自动训练并挂载适配器。这对于让通用模型掌握企业特有术语、专有产品名称等客服或业务场景非常方便。 不过，随着长上下文窗口（Context Window）技术的飞跃，开发者面临着新的架构选择： - **Prompt 提示词前置注入**：以前模型只能处理几个段落，现在 Gemini 支持多达 100 万 Token，甚至某些新模型支持上千万 Token。很多原本需要微调的专有知识，现在直接作为上下文拼接到 API 请求中就能解决。 - **全量微调开源模型的特殊价值**：虽然长上下文和 PEFT 很流行，但某些特定场景（如 Keelin 之前提到的心理咨询/治疗模型）仍需要全量微调开源模型。一方面深度修改权重能带来更细腻的拟合效果；另一方面，金融科技和医疗健康等行业对数据驻留、私有化部署和生命周期自主权有极高要求。 - **企业升级周期的现实冲击**：当前大语言模型的平均“货架寿命”（Shelf Life）仅有 6 个月左右。相比 5 到 10 年前企业软件（如 Salesforce）漫长的升级周期，如此频繁的模型更迭让传统企业难以招架。很多企业宁可选择全量微调一个开源模型长期稳定运行，也不愿被动跟随前沿模型的半年淘汰周期。 ### [27:46](https://www.youtube.com/watch?v=vu5GebTRLeA&t=1666s) 金融行业的低延迟诉求与“Prompt 优先”原则 Hailong (Harry) Wang（Fifth Third Bank 分析总监）结合银行业实践指出，金融服务等传统行业对系统延迟（Latency）极其敏感。面对日益增大的上下文窗口，团队通常采取分流策略： - **通用组件**：直接精心设计 Prompt，调用 OpenAI 或 Google 等专有云端模型 API。 - **专业/高频组件**：对于要求严苛延迟、高度依赖金融垂直领域知识的组件，则倾向于采用开源的小语言模型（SLM），并通过 LoRA 或 QLoRA 等 PEFT 技术进行轻量微调。 Harry 还分享了来自创业公司朋友的实战建议：“如果你不需要为极端的推理效率和延迟发愁，那就根本不要做微调，直接去花心思设计好你的 Prompt 上下文（Context Design）即可。” ### [29:24](https://www.youtube.com/watch?v=vu5GebTRLeA&t=1764s) 落地挑战：幻觉、延迟与企业级吞吐保障（Provisioned Throughput） 主持人 Abhishek Bafna 将讨论引向生成式 AI 在实际业务落地中最核心的三大挑战：**幻觉（Hallucinations）、延迟（Latency）与成本（Costs）**。 ```mermaid flowchart LR accTitle: 大模型企业接入方案与权衡 Demand{"企业接入需求"} --> Cheap["追求极致性价比 / 通用问答"] Demand --> Stability["追求稳定延迟 / 隔离性保障"] Demand --> Control["追求生命周期掌控 / 本地合规"] Cheap --> ManagedAPI["标准托管 API (按 Token 计费)"] Stability --> PT["预置吞吐量 (Provisioned Throughput)"] Control --> SelfHost["自建私有化算力 / 开源模型"] ``` Keelin McDonell 从模型选型与消费模式的角度剖析了成本与性能的平衡： - **Token 成本大幅下行**：好消息是基座模型的 Token 单价在过去几年中呈断崖式下跌，且每隔几周或几个月就会涌现出更高性价比的新模型，极大地降低了入局门槛。 - **标准 API 的“吵闹邻居”困境**：初创团队做通用 Chatbot 直接按量购买托管 API 最划算；但大型企业在业务高峰期使用公共 API 时，常会遭遇“吵闹邻居”（Noisy Neighbors，即其他租户抢占算力）以及特定区域的并发延迟波动。 - **预置吞吐量（Provisioned Throughput）**：为了解决这一问题，Google 等主流云服务商（CSP）推出了预置吞吐量服务。它能为企业提供类似于独占 GPU 硬件的算力隔离与性能确定性，从而在保障高并发和低延迟的同时兼顾云端托管的便利。 ### [31:35](https://www.youtube.com/watch?v=vu5GebTRLeA&t=1895s) 模型迭代的“倒退困境”与企业六个月淘汰周期的生命周期重负 在算力基础设施与服务部署层面，企业可以依据自身的 GPU 预算进行灵活配置，并在优秀工程团队的协助下对推理服务栈进行极致优化以达成优异性能。然而，技术落地中最让企业客户抓狂的痛点之一在于大模型的“版本退化”现象。尽管诸如 MMLU 等通用行业评测基准的分数随着模型迭代一路飙升，但在客户具体落地的某个细分专属场景中，新一代模型反而可能出现性能衰退（regression）。面对海量的长尾应用，研发团队即便竭尽全力也无法捕捉到所有的微小缺陷，这种“基准提升但垂直场景恶化”的矛盾让下游团队十分受挫。 ```mermaid flowchart TD accTitle: 软件生命周期变迁对比 subgraph Traditional["传统企业级软件时代"] direction TB T1["就地部署/早期 SaaS"] --> T2["3 至 5 年漫长升级周期"] T2 --> T3["稳固的合规、安全与验证机制"] end subgraph GenAI["生成式大模型时代"] direction TB M1["模型频繁迭代演进"] --> M2["约 6 个月生命周期即面临淘汰"] M2 --> M3["IT 团队持续承受基准评测与安全合规重负"] end ``` 这种高速演化对企业 IT 与系统运维管理带来了前所未有的生存压力。在企业软件领域摸爬滚打超过十五年的从业者无不感叹，过去传统大型企业软件系统的升级周期通常以数年为单位，即便 SaaS 模式的兴起也只是轻微加快了迭代节奏。但在当下的生成式 AI 时代，一个顶尖基座模型的领先寿命往往只有区区六个月便面临淘汰。企业如果想立于技术潮头，就不得不持续追随最新模型，但这也意味着必须无休止地进行内部基准测试、安全审查与系统重构，这种剧烈的生命周期压缩正在重塑企业 IT 部署的底层现实。 --- ### [33:37](https://www.youtube.com/watch?v=vu5GebTRLeA&t=2017s) 语音智能客服的现实权衡：从刚性脚本到大模型的成本与体验博弈 在电话语音及呼叫中心场景中，LLM 的引入经历了一段从怀疑到颠覆的微妙轨迹。早期数据显示，大模型在刚上线时并不能立刻超越基于意图分类器与硬编码分支的传统刚性脚本流程，但随后其表现以每个月 3% 到 4% 的速率稳步爬升。业内顶尖团队通过对开源权重模型进行针对性微调，已经将快餐外卖电话点单等复杂场景的成功率从过去的 75% 推升到了 85% 甚至 90%。然而，这一质量飞跃伴随着意想不到的经济与架构代偿：由于大模型天生善于生成更加丰富、冗长且口语化的自然回复，导致语音合成（TTS）的字符消耗量暴增，直接推高了单次呼叫的处理成本。 与此同时，企业传统的关键绩效指标正在被彻底重构。以往呼叫中心高度看重“平均处理时间”（Average Handling Time, AHT），力求在最短时间内解决问题——过去的硬编码系统虽然点单成功率仅有 60% 左右，但成功的通话基本能在 2 分钟内利索结束；如今基于 LLM 的方案将成功率提高到了 85%，但平均单次耗时却延长到了 4 到 5 分钟。通话时间的拉长不仅成倍推高了按量计费的 API 账单，还因为长上下文窗口的记忆漂移加剧了模型幻觉。然而，由于消费者满意度出现了实质性增长，企业管理者被迫接受了这种平衡：大家宁愿让电话多打 3 分钟，只要用户能成功下单且挂断电话时心情愉悦。 - 行业内出现令人哭笑不得的极端案例：有顾客拨打达美乐（Domino's）披萨点单热线，由于大模型在对话中意外偏离主线，用户竟与其兴致勃勃地闲聊了长达 30 分钟。 - 在按调用量计费（consumption-based pricing）模式下，这种未产生实际订单却聊了半小时 ChatGPT 的通话，对企业而言无疑是成本失控的“最昂贵账单”。 - 业内将当前的语音 LLM 应用生态戏称为“狂野西部”（wild west）：尽管底层模型正在以人类历史上前所未有的速度商品化，但构建与运行实际应用的整体成本依然居高不下。 --- ### [36:53](https://www.youtube.com/watch?v=vu5GebTRLeA&t=2213s) 重新定义“幻觉”：从量化逻辑到初级分析师的自主性隐喻 围绕大模型饱受诟病的“幻觉”问题，来自银行业与量化分析背景的专家提出了发人深省的视角转换：我们或许应当重新审视幻觉的定义与归因。大模型在本质上是一种偏向行为感知与模式拟合的概率系统，而非专用于确定性数理逻辑的工具。如果将大模型直接当作严谨的量化分析工具来使用并期望绝对的确定性，随之而来的事实偏差究竟应当归咎于模型本身的缺陷，还是用户系统设计与选型的不当？ 即便在系统架构上进行了严密设计，幻觉依然不可避免，其根本原因在于应用场景赋予了模型“自主性”（autonomy）。这种机制恰如团队中新入职的初级分析师：管理层即便为其提供了全套标准化培训和详尽指南，当这位初级分析师独立输出成果时，最终结果中往往还是会出现某些偏差或错漏。企业在推进大模型落地时，必须脱离对绝对零误差的虚妄幻想，正视模型自主性与确定性之间的固有张力，根据自身具体的业务风险承受度（risk tolerance）来划定能够容忍的边界。 --- ### [38:36](https://www.youtube.com/watch?v=vu5GebTRLeA&t=2316s) 多模态大模型的幻觉破解：文本加噪扰动技术 传统文本大模型通常依靠检索增强生成（RAG）架构或外部防护栏（external guardrails）来在一定程度上抑制幻觉生成。然而，在以图文为代表的多模态大模型（MLLM）迅速崛起的背景下，一种独特的跨模态幻觉正在成为日益严峻的挑战：模型在接收图像与文本混合输入时，往往会表现出过度依赖文本先验信息的倾向，将注意力严重偏向文本提示，而忽略了图像内容的真实物理表征，导致图文之间出现事实性的逻辑断裂。 ```mermaid flowchart LR accTitle: 多模态抗幻觉文本加噪扰动技术 subgraph Inputs["多模态联合输入"] IMG["真实图像输入"] TXT["原始文本提示"] end subgraph Mitigation["抗幻觉扰动机制"] NOISE["人为注入噪声 / 文本扰动"] TXT --> NOISE end subgraph Fusion["平衡注意力机制"] NOISE & IMG --> MLLM["多模态模型 (MLLM) 推理"] end MLLM --> OUT["建立真实图文锚定，消除跨模态幻觉"] ``` 为了迫使模型在文本与图像输入之间建立平衡的注意力关联，研发团队探索出了一套极具对抗性思维的扰动技术（perturbation techniques）。该方案的核心在于主动干预输入端： - 研发人员会在多模态训练或推理过程中，蓄意向文本输入通道注入噪声，通过扰动文本的平滑性来降低模型对文本先验的过度自信。 - 这种文本降质手段强制模型将注意力权重重新转移到图像模态上，从而促使模型学会从多模态联合线索中提取依据。 - 该技术已成为解决涉及多模态融合输入场景下幻觉与事实失真问题的核心算法策略之一。 --- ### [39:58](https://www.youtube.com/watch?v=vu5GebTRLeA&t=2398s) 前瞻与去魅：看好生物科学与原生多模态，对 AGI 炒作保持审慎 在展望未来技术趋势与破除行业泡沫的圆桌总结环节，嘉宾们将目光投向了生物计算与原生多模态统一模型。生命科学领域被普遍视为极具潜力的蓝海：从基于生成式模型的药物发现到蛋白质空间结构的预测与设计，AI 在分子生物学中的未开垦空间依然极为广阔。与此同时，当前多模态模型普遍依赖于将不同模态的独立模型在前端或后端进行粗暴拼合，未来的核心突破点将在于构建统一架构模型（unified models），使其能够像人类感知世界一样，在统一的认知体系中无缝且连贯地同时消化、理解与生成多种模态的数据流。 与对具体技术路径的兴奋相反，嘉宾对当下甚嚣尘上的通用人工智能（AGI）概念提出了审慎的去魅。人类婴儿认知世界、建立逻辑常识的过程与目前大语言模型的参数学习机制存在着根本性的范式鸿沟，在许多专家看来，目前关于 AGI 的宏大叙事与舆论狂欢在很大程度上被过度炒作（overhyped）。相较于模糊且定义不清的 AGI 愿景，更加务实、可落地的技术演进将聚焦于多模态感知的深度演进，涵盖从精准高保真的语音合成到更复杂的视觉协同生成等具体能力的持续跃迁。 ### [42:05](https://www.youtube.com/watch?v=vu5GebTRLeA&t=2525s) 多模态、超越 Transformer 的架构与数据管线落地 Keelin McDonell 从产品经理的视角分享了对生成式模型未来演进的看法。他指出，当前多模态生成仅仅处于“冰山一角”（tip of the spear），而在日常工作中，模型的实用性与其多模态能力高度成正比——能够直接输入图表、数据图形以及语音备忘录等多样化模态，极大地提升了工作效率。此外，尽管“超越 Transformer”已经成为过去几年业内的老生常谈，但行业确实正在看到基于扩散（Diffusion）架构的文本模型以及科学计算模型的落地。例如在 Google，团队已经开始逐步推出基于扩散架构的科学模型。 谈及对通用人工智能（AGI）的狂热，Keelin McDonell 表达了理性的怀疑态度。他强调，当前行业真正面临的硬骨头并不是追求 AGI 的虚名，而是踏踏实实地“把工程管线（pipelines）做好”。目前全行业甚至连现有模型 99% 的潜能都还没有充分发挥出来；只要把底层管线和集成落地打通，即便仅凭现有的模型能力，就足以引发一场颠覆性的实用价值变革（sea change）。 - 多模态能力的深化让模型在处理图表、音频等非文本信息时展现出极高的实际效用。 - 扩散模型（Diffusion Models）开始跨界应用于文本生成与科学计算等垂直领域。 - 相比遥不可及的 AGI，打通与优化数据工程管线是释放现有 99% 模型潜力的核心所在。 ```mermaid flowchart TD accTitle: 释放生成式模型实用价值的路径 subgraph CoreCapabilities["模型演进能力"] M1["多模态拓展<br>(图表/语音/图像)"] M2["新兴架构探索<br>(文本与科学扩散模型)"] end subgraph EngineeringPipeline["落地关键瓶颈"] P1["优化数据与工程管线<br>(Pipelines)"] P2["释放现有模型未开发潜力<br>(>99% 价值空间)"] end subgraph BusinessImpact["实际业务影响"] I1["颠覆性生产力飞跃<br>(Sea Change)"] end CoreCapabilities --> EngineeringPipeline EngineeringPipeline --> BusinessImpact ``` --- ### [43:09](https://www.youtube.com/watch?v=vu5GebTRLeA&t=2589s) 技术的真实影响与总结致谢 讨论进一步延伸到对技术发展本质的思考。有嘉宾指出，人们完全不需要迷信 AGI，即使 AI 的底层技术进步从今天起彻底停滞，现存技术所带来的影响也已经具有颠覆性。这就像互联网相对于电报的演进——本质上同样是通过线缆传输信息，但依托 TCP 和 HTTP 协议，在过去 45 年中彻底重塑了人类世界；AI 并不需要达到《机械公敌》（I, Robot）那种科幻级别，就足以永远改变当代人及后代的生活。 随后，Fifth Third Bank 的 Hailong (Harry) Wang 补充道，只要回看过去一两年前人们的工作和生活方式，就会发现改变已经切实发生且极其深刻。无论终极的 AGI 是否到来，AI 技术的巨变已是既成事实，大家需要主动拥抱这一现实，并抓住把各项技术和应用串联整合的时代机遇。最后，主持人宣布圆桌研讨时间截止，全体嘉宾致谢并结束了本场分享。 - 技术变革的深远程度不依赖科幻式的 AGI，如同互联网协议在过去数十年对电报时代的重构一样。 - 过去一到两年内，个人与企业的工作生活流程已经发生了不可逆的剧烈转变。 - 拥抱技术现实、将各环节应用有效连接是当前各行各业面临的最大机遇。 ## Notable quotes > [05:57](https://www.youtube.com/watch?v=vu5GebTRLeA&t=357s) raising my cortisol levels — Lily Clifford > [11:05](https://www.youtube.com/watch?v=vu5GebTRLeA&t=665s) 小而强大 — Hailong (Harry) Wang > [17:29](https://www.youtube.com/watch?v=vu5GebTRLeA&t=1049s) 背后需要海量的提示词工程与模型编排，这其中的复杂度远超外界的想象 — Lily Clifford > [23:22](https://www.youtube.com/watch?v=vu5GebTRLeA&t=1402s) 如何微调最少数量的参数，就能精准达成预期的业务改进？ — Lily Clifford > [27:46](https://www.youtube.com/watch?v=vu5GebTRLeA&t=1666s) 如果你不需要为极端的推理效率和延迟发愁，那就根本不要做微调，直接去花心思设计好你的 Prompt 上下文即可。 — Hailong (Harry) Wang > [33:37](https://www.youtube.com/watch?v=vu5GebTRLeA&t=2017s) 狂野西部 > [42:05](https://www.youtube.com/watch?v=vu5GebTRLeA&t=2525s) 当前行业真正面临的硬骨头并不是追求 AGI 的虚名，而是踏踏实实地把工程管线做好 — Keelin McDonell > [43:09](https://www.youtube.com/watch?v=vu5GebTRLeA&t=2589s) 即使 AI 的底层技术进步从今天起彻底停滞，现存技术所带来的影响也已经具有颠覆性。这就像互联网相对于电报的演进 ## Mentioned Abhishek Bafna, Hailong (Harry) Wang, Shubham Shukla, Lily Clifford, Keelin McDonell, Amazon, Nordstrom, Fifth Third Bank, Rime, Google Cloud, Vertex AI, Model Garden, Gemini, Claude, Llama, DeepSeek, Qwen, GPT-3, GPT-3.5, Google PaLM, Domino's Pizza, Chase Bank, Eli Lilly, Novartis, Wegovy, Semaglutide, Salesforce, MMLU, HIPAA ## Video description Abhishek Bafna, Senior Machine Learning Scientist, Amazon Hailong (Harry) Wang, Director Analytics, Fifth Third Bank Shubham Shukla, Data Scientist II, Nordstrom Lily Clifford, CEO & Founder, Rime Keelin McDonell, Product Manager, Google Abstracts This session will explore the broad applications and technical strategies behind generative models, focusing on how different models and techniques can solve complex AI challenges. Panelists will discuss a variety of generative models, including the latest training techniques, fine-tuning approaches, and prompt engineering strategies. Learn how these models are being applied across industries to tackle hard problems, from creative content generation to advanced data synthesis. The discussion will provide insights into the evolving landscape of generative models and the techniques driving their success. [Source](https://www.youtube.com/watch?v=vu5GebTRLeA)

[5:57] Lily Clifford: raising my cortisol levels [11:05] Hailong (Harry) Wang: 小而强大 [17:29] Lily Clifford: 背后需要海量的提示词工程与模型编排，这其中的复杂度远超外界的想象 [23:22] Lily Clifford: 如何微调最少数量的参数，就能精准达成预期的业务改进？ [27:46] Hailong (Harry) Wang: 如果你不需要为极端的推理效率和延迟发愁，那就根本不要做微调，直接去花心思设计好你的 Prompt 上下文即可。 [33:37] 狂野西部 [42:05] Keelin McDonell: 当前行业真正面临的硬骨头并不是追求 AGI 的虚名，而是踏踏实实地把工程管线做好 [43:09] 即使 AI 的底层技术进步从今天起彻底停滞，现存技术所带来的影响也已经具有颠覆性。这就像互联网相对于电报的演进

[3:19] Lily Clifford: If you call and you're like, 'This voice is raising my cortisol levels. It's making me stressed out,' you're going to hang up. [14:58] Keelin McDonell: uncover what the LLM's personality is. [23:22] Lily Clifford: what is the fewest number of parameters that you can fine-tune in order to affect the change [31:35] Keelin McDonell: optimize the heck out of the serving stack [33:37] Lily Clifford: longer and more chattier [42:05] Keelin McDonell: tip of the spear [42:05] Keelin McDonell: beyond transformers [42:05] Keelin McDonell: I think in my experience, the hard thing is the pipelines. Like let's get the pipelines right. We haven't even gotten use out of 99% of these models at this point. And so once that happens, there's going to be like a massive sea change in the usefulness of these models even as they stand today.

[0:04] Welcome to generative AI model use case [0:06] and techniques session. We have a very [0:08] exciting session ahead of us. Um to get [0:12] things started, let's do a quick round [0:14] of introductions. Uh I'll start with [0:17] myself. I am Shubhham. Uh I'm a panel [0:19] moderator for this session. Uh I work at [0:23] Nordstrom as a data scientist. Uh my [0:26] focus is into generative AI efforts [0:29] leading generative AI efforts at [0:31] Nordstrom. Um focusing on projects [0:34] dealing with u internal productivity of [0:38] the employers [0:40] and prior to that I have worked in [0:42] product attribution space [0:44] recommendations [0:46] and pro and personalizations. [0:50] Um, prior to that, I have worked at [0:52] Amazon as a ML engineer for a couple of [0:55] years. And that's it. So, go ahead. [0:58] >> Awesome. Thanks for coming out to this [1:00] panel. My name is Lily Clifford. I'm the [1:03] CEO and one of the co-founders of a [1:05] startup based in San Francisco called [1:07] Rhyme. Rhyme develops speech synthesis [1:10] models for enterprise voice applications [1:13] prototypically in the enterprise. is [1:15] like if we all have the experience of [1:16] calling a toll-free number for a bank or [1:19] a telco, we generally don't like it. And [1:23] so, uh, I started a company a couple [1:25] years ago with my two co-founders, [1:26] Brooke and Aries, to help improve the [1:28] experience of calling enterprises to get [1:31] customer support. Um, one referenceable [1:34] customer I can mention is that if you [1:36] call Domino's Pizza North America, if [1:38] that Domino's Pizza has generative AI [1:41] based phone ordering, you're hearing our [1:44] voices. something like 40% of dominoes. [1:48] >> Okay. Uh my name is Harry Wong. I'm the [1:51] director analytics for model risk [1:55] management and also the head of [1:57] environment innovation app and fifth. So [2:00] I'm leading the effort to do the digital [2:03] transformation for the model life cycle [2:05] management for the financial services. [2:09] >> Uh hi everyone. Uh my name is Abishek. [2:12] Um and um I've been uh with Amazon for [2:16] over 6 years now. Uh I'm a senior [2:19] applied scientist. Um currently with [2:22] Alexa uh working on uh working with [2:26] models for video understanding [2:29] multimodal essentially. Uh my background [2:31] has been in uh CV uh mostly CVM ML and [2:36] uh I've worked in healthcare space [2:38] before as well before I joined Amazon. [2:40] Um, sorry that's okay. [2:43] >> Uh, hi everyone. My name is Keelin [2:44] McDonald. I'm a product manager at [2:46] Google Cloud. Um, I oversee a product, [2:49] excuse me, on our Vertex platform called [2:51] Model Garden. So, um, if you've ever [2:53] come into Vertex and looked for Gemini [2:55] or Llama or DeepSeek or any of 20,000 [2:59] other models, um, that is uh that all [3:01] sits in Vert.Ex Model Gardens. So really [3:03] my role is making sure that uh users can [3:06] find all of the great models that Google [3:08] offers but also uh third party and [3:11] partner models as well. [3:14] >> Cool. Um let's start with the first [3:17] question. [3:19] Um so generative AI has exploded uh with [3:23] diverse applications um be it text, [3:27] video, audio or images. Why don't you [3:30] share a specific use case you have [3:33] worked on involving generative AI and [3:37] what kind of problem it was? What [3:39] architecture did you use to solve that [3:41] problem? [3:45] Sure, I can go first. Uh one problem [3:48] specifically comes to mind uh and this [3:51] is in the realm of synthetic data [3:53] generation. Um we're trying to uh so as [3:59] some of people have pointed out one of [4:02] the limitations that we have in [4:04] synthetic data generation is the data [4:06] can be tooable [4:08] right um and so in this case we tried to [4:13] make the data more humanlike. [4:15] So the framework was quite [4:17] straightforward. We use a ganbased [4:20] framework uh generative adversarial and [4:22] network based framework uh which uh and [4:25] each of the discriminator and generative [4:28] modules over here where LLMs and by [4:31] using a small subset of human curated [4:34] data we are able to train these models [4:36] in such a way that the resultant data [4:40] captures the nuances of this human [4:42] curated data and [4:45] it's so diverse that it can be [4:47] applicable to any uh domain in per [4:49] particular and by using this we were [4:51] able to generate more uh humanlike [4:53] synthetic data. [4:55] >> Okay. uh just as I do my uh self [4:58] introduction uh as uh I currently need [5:01] the effort to do the digital [5:03] transformation for the model cycle [5:05] management actually we are on the [5:07] journey so this we we can call this is [5:11] the geni user case as well we are on a [5:13] journey uh to do this we've been for [5:16] like four and five years so the the case [5:20] we are doing is first is the first steps [5:23] we have to prepare our domain knowledge [5:25] Then we move on the AI agent. Now [5:28] actually we are in the agent AI stage [5:31] for to to connect everything together [5:35] and to implement all the dominoled to [5:37] the rag or or graph rag or otherwise to [5:41] implement all the the large model in [5:44] order to do the some autonomy for the [5:47] decision and to help us to more [5:50] efficient and more accurate to for the [5:54] system. Yeah. [5:57] So the industry that Rhyme is building [6:02] for is a very large industry. I think [6:04] conservatively there's probably 600 or [6:06] 700 billion of spend in customer [6:08] experience. [6:10] And [6:12] let's just say 3 or 4% of that is in [6:16] software today, right? And the rest is [6:18] in human labor. At the same time, right? [6:21] This comes from Gartner. [6:22] In the next 5 years, 30% of all phone [6:26] calls to enterprises could be fully [6:28] automated. And I think they're saying by [6:30] 2030 it's going to be 90%. Like even if [6:32] we don't believe that, that's a lot of [6:35] generative AI usage that today, right? I [6:38] can tell you from selling boots boots on [6:40] the ground into enterprises who have [6:41] these high volume calling applications. [6:44] How much of that today is LLM based? [6:47] I don't know, maybe like 1% uh less than [6:50] 1% quarter of a percent maybe less, [6:52] right? A lot of it's still like [6:54] pre-recorded assets. They got a voice [6:56] actor to record at the end of the 20th [6:58] century, right? That's what you're [6:59] hearing when you're calling Chase Bank. [7:01] And at the and and so like what is the [7:03] opportunity? The opportunity is huge at [7:07] the same time, right? Like very narrowly [7:09] focused. Ryme is very narrowly focused [7:12] on making it something that you want to [7:14] talk to, right? right? Like it doesn't [7:15] matter how smart a voice agent is. If [7:18] you call and you're like, "This voice is [7:19] raising my cortisol levels. It's making [7:21] me stressed out, you're going to hang [7:23] up." And so that's the really the [7:25] opportunity too is just from a user [7:27] experience perspective, how can we make [7:28] these applications [7:30] something that people want to use [7:32] instead of just um [7:35] I don't know, something that's [7:37] vaporware. I think that's like the point [7:39] that we're at. Um and I'm sure it's [7:41] going to happen at the same time. Yeah. [7:44] Yeah, I I might take this question from [7:46] like a slightly different angle just [7:48] given the the product that I oversee. So [7:50] I mentioned that Vert.Ex Model Garden [7:52] has you know 20,000 something models and [7:55] really those models are split across [7:58] different consumption patterns. So on [7:59] the one hand you have your proprietary [8:02] managed APIs. Those are going to be [8:04] things like Gemini but also quad other [8:06] kind of proprietary models. In the [8:08] middle you have um open models that are [8:11] also offered as as API services. And [8:13] then on the other extreme you have [8:15] openweight models where you're actually [8:16] provisioning the infrastructure to run [8:18] the models. And so often when we're [8:20] working with customers and talking about [8:22] you know what's the right [8:23] model/conumption [8:25] pattern for their use case we're we're [8:28] basically thinking about what do you [8:30] what are you trying to accomplish with [8:31] your use case. So for example um if a [8:34] customer has um a data set and they want [8:37] to really you know fine-tune and and uh [8:40] manage the model themselves and have you [8:42] know the ability to move it across [8:44] regions and lots of supportability we're [8:46] generally going to direct them to an [8:47] open model. So they might end up with [8:49] something like a llama or a deepse or a [8:50] quen. Um if they are looking for [8:53] basically just a really fast way to [8:55] consume tokens that are going to be [8:56] really state-of-the-art we're going to [8:58] be looking more at a managed service. [9:00] So, um, just to maybe kind of start on [9:02] the more open side, um, we worked with a [9:04] customer recently that does, um, they [9:06] have an application that delivers [9:08] therapy. And so, what they wanted to [9:10] accomplish was to, um, you know, have [9:13] some sort of, uh, basically kind of [9:17] therapy um, chatbot assistant that was [9:20] going to be trained on previous [9:21] transcripts that they had from actual [9:23] providers. And so, this was something [9:25] where they couldn't really do that [9:26] training on a proprietary model. they [9:28] really needed an open model and because [9:30] of HIPPA and various kind of um laws [9:33] that exist internationally, they needed [9:34] to really have very specific control [9:36] over where that um that model was going [9:38] to sit along with the data. Um so in [9:40] that case they um I believe they they [9:42] ended up using Llama. They fine-tuned [9:44] it. They were able to deploy it actually [9:45] in a few different places and got really [9:47] good results. Um on the other hand, I've [9:50] also worked with customers that want [9:51] something a little bit more out of the [9:53] box. So they might say, you know what, I [9:55] have a pretty generic, you know, [9:56] customer support use case or [9:58] summarization use case, and what I [10:00] really want is just something that [10:01] that's going to be really high quality [10:03] and is going to be really optimized for, [10:05] you know, great latency and an [10:08] affordable tokens. So in those cases, [10:10] we're generally going to push them [10:11] towards a managed API like a Gemini or a [10:14] Quad. Um, so again, you know, it's a [10:16] spectrum of of options. There's not [10:18] necessarily one right answer depending [10:20] on the use case, but um when we're [10:22] working with customers, we sort of have [10:24] a checklist of about 15 questions that [10:26] are going to kind of range across those [10:28] different priorities and based off of [10:29] the output of that checklist, we're [10:31] going to kind of direct them towards one [10:32] model or one architecture or another. [10:36] >> Cool. Uh pretty diverse use cases, I [10:39] must say. Uh let's talk about prompt [10:43] engineering. [10:45] uh it has almost become an art form [10:47] these days and for those working with [10:50] LLMs what techniques have you used [10:53] within prompt engineering and how have [10:56] you used them to get get reliable [10:58] outputs what techniques have worked for [11:01] you what techniques have not worked for [11:03] you [11:05] >> okay uh I can take this question first [11:08] uh uh because uh uh for our case I think [11:11] uh um I I think everybody that maybe now [11:15] uh when you use your the the gen [11:18] application or even for the genetic uh [11:21] system right so the primary engineering [11:23] maybe is a face the the the [11:26] people to use but however uh that's not [11:29] enough because when we design when we [11:32] design system or otherwise uh we talk [11:35] about a lot of things about the data [11:36] right so for the large model no matter [11:39] is small or the the the API All so [11:44] actually the data is all your content. [11:47] So your content if just like a one or [11:50] three sentence of a problem is not good [11:52] enough because you need to understand uh [11:54] what's the large model what they need to [11:57] take into their input. So that means you [11:59] have to have a contact engineering uh [12:02] the design for your content. So actually [12:04] for our system we are design right now [12:07] we have prompt which just one or two [12:08] sentence to ask okay what kind of model [12:11] I want to build what kind of report I [12:13] want to generate but hard behind the [12:15] scene we spend a lot of time to to [12:18] design the dominoled to their content [12:20] for our content content and our content [12:23] engineering in order to make our API [12:26] call or otherwise just go back to the [12:28] small language model we want to small [12:30] and mighty but how we can find the [12:32] tuning we have to put everything [12:34] together. Yeah. [12:37] >> Yeah. I can add something to this. Um [12:40] prompting prompt engineering has evolved [12:42] from the basic commands and now it has [12:45] become pretty sophisticated paradigm [12:47] that we use and uh it is the first thing [12:50] that you go to for an LLM because it [12:52] doesn't involve too much uh change [12:55] actually no change to to the web of the [12:57] model and uh you are still able to uh [13:00] achieve your personal use case from [13:02] LLMs. So generally you have the three uh [13:07] the basic types of prompting which is is [13:09] a zero short prompting few short [13:11] prompting and chain of thought zero [13:13] short you you already have done you [13:15] don't provide any examples uh few short [13:18] prompting you provide some examples in [13:20] in a lot of cases this gives you quite [13:23] good results as the baseline and such [13:25] and chain of thought breaks down the [13:28] problem into smaller problems so you [13:30] yourself break down and give it to the [13:31] LLM so that it the answers are more [13:34] reasonable and more thoughtful. Uh you [13:36] can say apart from these uh a couple [13:39] more techniques that I have found very [13:41] useful. Uh one is uh role or persona [13:45] based prompting wherein you assign a [13:47] role or a persona to the LLM and this [13:51] guides the tone and the style and also [13:53] the accuracy sometimes of the response. [13:56] Uh the other one is soft tuning. uh by [13:59] soft tuning what it means is you append [14:02] uh or prepend the input with trainable [14:06] embedding vectors right uh so this is [14:09] not text so these are just embedding [14:11] vectors and uh these can be extracted or [14:13] learned from the examples itself now [14:16] these will encode the information from [14:18] the examples in fewer tokens so now you [14:21] have more tokens left for your [14:23] description or the uh description of the [14:26] uh problem you're trying to solve [14:28] So this in a lot of cases it condenses [14:30] the prompt and it gives a much better [14:32] better result. And the third one is [14:36] autotuning. So you can get the LLM to [14:39] refine the prompt or the results by [14:42] itself uh iteratively. And in a lot of [14:45] cases you probably wouldn't figure out [14:47] the best way to get the answer or frame [14:50] the prompt but then the LLM can itself [14:52] help uh to frame it in a good way. So [14:55] these are some of the techniques I use. [14:58] Yeah, and and I can give a little bit of [15:00] background on kind of the the journey of [15:02] prompt engineering I've seen in the [15:03] industry over the last few years. So, [15:05] um, three years ago when, you know, the [15:07] kind of first generation of LMS were [15:08] coming out, uh, I think everyone was in [15:11] the same boat and experience [15:13] experiencing extreme pain in migrating [15:15] prompts from one model to another. So um [15:18] this was you know from GPT3 to 3.5 or in [15:21] our case we had a model called palm that [15:24] we had to migrate prompts from palm to [15:26] gemini and or even between different [15:29] types of models. So maybe you were going [15:30] from Gemini to quad or GPT to claude and [15:33] just quite honestly like the the prompts [15:36] were were were basically incompatible [15:38] between models. It was like shocking how [15:39] different of a response you would get. [15:41] And it was incredibly frustrating for [15:43] our customers and our field teams to try [15:45] to, you know, tell customers, hey, we [15:47] have this great new model, but by the [15:49] way, you have to rewrite all your [15:50] prompts and we don't really have a [15:52] technique to do it. We're just going to [15:54] sort of guess and kind of uncover with [15:56] you what the the LLM's personality is. [15:58] Um, so that's where we were three years [16:00] ago. Uh, I I'm very happy to say we are [16:02] not there today. Um what I typically see [16:04] now is very very little of those kinds [16:07] of like you know kind of you know major [16:11] frictions or gaps between models like um [16:14] you know a very common exercise we'll do [16:15] with customers evaluating different [16:17] models is they'll take the same prompt [16:18] or or roughly the same prompt and [16:21] they'll send it to five or six different [16:22] models and it'll all end up in like a [16:24] spreadsheet somewhere where they'll [16:25] either do an eval or some some human [16:27] being will look at it and you know for [16:28] the most part with some slight [16:30] exceptions you know prompts are I would [16:33] say more transferable than they were a [16:35] few years ago. Um so the things that I'm [16:37] seeing now um versus a few years ago are [16:41] um automatic prompt tuning um as Abishek [16:45] mentioned. Um the other thing that we've [16:46] rolled out on the vertex side and [16:48] there's other open source tools that do [16:49] similar things is a tool called prompt [16:51] optimization. And so basically what that [16:53] does is you put in the prompt in any way [16:55] you feel comfortable writing it. You [16:57] kind of turn some knobs based off of [16:59] what you want. um maybe you want it more [17:01] formal, maybe you want the response more [17:02] concise, etc., etc. And essentially the [17:05] LLM, uh there's kind of an arbiter LLM [17:08] that is going to rewrite the prompt for [17:09] you and then route that to the the [17:10] model. So I think like just kind of [17:12] looking across the journey, the the main [17:14] themes that I'm seeing are uh prompting [17:16] techniques are a lot more automated and [17:18] a lot more um sort of uh transferable [17:21] between models. Um but I think that [17:23] we'll continue to see that improve and [17:25] get a lot easier as we we go on. [17:29] I think, okay, here's my perspective on [17:31] voice applications. They're still very [17:34] hard to build, right? Like unlike uh an [17:37] LLM that accesses a knowledge base to [17:39] answer sort of FAQ type questions in the [17:42] context of like a firstline customer [17:44] support use case. We have customers that [17:47] are building applications in the [17:48] healthcare space to call for example an [17:50] insurer right on behalf of uh a hospital [17:54] network to verify benefits information. [17:56] And this is a revenue cycle management [17:58] use case. And in many cases, the people [18:00] on the other end of the call, okay, for [18:02] one, the AI system they're building is [18:04] navigating a whole other phone tree. So [18:06] you have essentially like AI talking to [18:08] AI. And then once they get through the [18:09] phone tree, they're talking to a human [18:11] agent and for some cases for like 45 [18:14] minutes and they have se they know they [18:16] have to get through seven questions, [18:17] right? Like is this prescription [18:19] covered? um what is um the the okay I'm [18:23] not totally up to date on like insurance [18:25] industry but like very specific [18:26] information they need to gather and you [18:29] you simply cannot do this with one LLM [18:33] today right like one prompt one LLM no [18:36] it's not possible right like they're [18:37] having to do a pretty sophisticated I [18:39] don't know I guess you could say like if [18:41] any of you in the audience have [18:42] experience building these kinds of [18:44] applications pre-LM right like they're [18:46] essentially having to reinvent the wheel [18:48] I'm not saying it's a worse application. [18:49] It's much better application and they're [18:51] able to do things that they were never [18:52] able to do before. But they're in some [18:54] cases right using 15 different very [18:57] small Gemini models for example just to [19:00] manage where in a conversation is [19:02] essentially doing like intentbased [19:04] classification using smaller LMS. Now, [19:07] each of those smaller LLMs need to [19:09] themselves have prompts, right? And so, [19:12] the problem space has really exploded in [19:14] a way that I don't think many people [19:15] appreciate. For what it's worth, right? [19:17] Like, you can go to the Rhyme website, [19:18] click try live chat, and you're talking [19:20] to our voices. It's just a single LM [19:22] prompt. Can it do anything? No, it can't [19:25] actually do anything. It can't make [19:26] function calls, right? It can't [19:27] integration integrate with a database. [19:29] It can't make external API calls to some [19:31] other service. It can't order pizza [19:34] today. Um, and so yeah, that I think [19:37] just want to convey right like how much [19:39] work it actually takes to build into to [19:41] build an enterpriseg grade application [19:43] that can have a conversation for more [19:44] than three minutes and actually do [19:45] stuff. It requires a ton of prompt [19:47] engineering. [19:50] >> Cool. Um, after prompting, let's talk [19:54] about [19:55] uh full training or find or fine-tuning. [20:00] Um [20:01] well you know before LLMs came into the [20:04] picture they were just either full [20:06] training of the models or fine-tuning of [20:08] the models for a particular task but [20:10] after LLMs there are few more techniques [20:13] available like SFT and and uh LORA and [20:18] parameter efficient finetuning uh have [20:21] what are your thoughts on these [20:22] techniques uh across the use cases you [20:25] have come across in your workplace use [20:30] >> I can take this uh first um [20:34] I'll probably shed some points about the [20:37] finetuning and specifically PF [20:40] techniques. So uh what is mentioned [20:43] mentioned is PFT is you must have heard [20:46] about the parameter efficient finetuning [20:48] techniques. So coming from prompt [20:52] engineering right uh so prompt [20:54] engineering can only do so much uh [20:56] because the model weights are not being [20:58] updated. So you have essentially the [20:59] same model. So by changing the external [21:02] inputs uh through prompts uh you can [21:04] achieve a degree of accuracy and a [21:07] degree of what you want. uh but in order [21:09] to uh do more or change things or [21:12] achieve more uh further accuracy you [21:15] need to update the model weights and [21:17] that's what we call fine-tuning. [21:20] Now these models have billions of [21:23] parameters which you all know and they [21:24] keep increasing uh with every model and [21:27] such. So in order to do a full [21:29] fine-tuning you have to update all of [21:31] these weights which incurs huge cost and [21:33] huge resources which typically [21:36] not you're not uh like you don't want to [21:40] do all of that for this. Uh the other [21:42] thing is um the data that you use to [21:46] fully fine-tune a model it needs to be [21:48] diverse and huge if you want to get [21:50] anything good out of it. uh otherwise [21:52] you tend to occur like go run into [21:56] issues like um [21:58] catastrophic forgetting or um you know [22:02] uh overfitting. What this means is that [22:05] by using the data that you're trying to [22:07] train the model you are overwriting the [22:10] weights that it has before and so you [22:13] are possibly incurring the risk of uh [22:16] the model trying to forget all the [22:18] generic information which it had [22:20] already. So all of these models are [22:22] trained on tremendous amounts of data [22:24] like trillions of tokens. So uh they [22:27] have a lot of innate uh intelligence or [22:29] smartness that we don't want to [22:30] eliminate. So here theft comes in which [22:33] is the parameter efficient finetuning [22:35] techniques. Uh by these uh you train [22:38] adapters. So these are just a fraction [22:41] of weights which the model has and uh [22:45] these can be trained in a way that it [22:47] doesn't disturb the original weights of [22:49] the model. So they are additive weights. [22:51] They are low rank matrices which are [22:53] embedded in the training in the [22:55] transformer layers. Uh and with this [22:58] fraction of weights you are able to [23:00] train uh much uh faster and uh get your [23:04] job done quickly. So these techniques [23:07] are like Lora and QQQ Laura. You should [23:10] go and read if uh you've not these are [23:12] very commonly used techniques and these [23:15] are pretty much used to uh fine-tune the [23:17] model. So those are uh the basic cases [23:20] of fine. [23:22] >> I can give a concrete example of why [23:25] this matters. Okay, let's say [23:29] someone like open AI say they scrape [23:32] every audio book and every YouTube video [23:35] just for the audio, right? Okay. And [23:36] then they use some speech recognition [23:38] network to transcribe every single audio [23:40] book in existence and every YouTube [23:42] video in existence and then train on [23:44] that tokenized data a speech synthesis [23:47] model. Let's say they do this in 2020. [23:50] Okay. Now they have this in production [23:53] this t this Texas speech model in [23:55] production powering healthcare use [23:56] cases. Guess what, right? Eli Liy [24:00] Novartis, they're making new [24:01] prescription medications every single [24:03] year. Right? and the text speech model [24:06] pronounces we go v incorrectly right how [24:10] do you fix that actually either you go [24:12] and collect data you're going to need to [24:14] collect data under this this paradigm of [24:16] right like 10 different people saying we [24:19] go vi or semutide or whatever it might [24:21] be and then you train again these are as [24:25] has been mentioned very large models if [24:27] you full do full parameter fine-tuning [24:29] on just those 10 examples the model [24:31] might pronounce semiglutide and we go [24:33] correctly but it's not going pronounce [24:35] it's going to forget a lot of the other [24:36] things that were contained in those [24:38] representations in the model and so and [24:40] by the way right like you don't want to [24:41] go to Eli Liy and be like here's an [24:44] entirely new checkpoint by the way [24:47] you're going to have to buy new hardware [24:48] to run it so then the consideration [24:51] becomes like in any of these product [24:54] paradigms what is the fewest number of [24:55] parameters that you can fine-tune in [24:57] order to affect the change the [24:59] improvement right that you're looking to [25:01] to find out again I think we're in wild [25:04] s territory here. Um but that that's [25:06] just to motivate concretely why someone [25:08] is interested in doing this. [25:11] >> Yeah, I I think that's a great example [25:13] and that's um generally with PEFT. So [25:16] for our uh proprietary models like I [25:18] mentioned before like Gemini and Claude [25:20] um we actually have a managed service to [25:22] do that type of tuning. So basically you [25:25] you don't need a deep a IML background [25:27] to do this now. You just need you know a [25:30] JSON blob of like a 100 examples. You [25:32] upload it, you get your adapters in [25:34] place, and it comes up a lot in exactly [25:35] the types of examples that Lily was [25:37] mentioning. So, for example, I run a [25:39] business and maybe Gemini has enough, [25:42] you know, general knowledge now to to do [25:44] a pretty good customer service bot, but [25:45] it doesn't know the names of my products [25:47] or it doesn't know, you know, specific [25:49] terms that we use. It can be really [25:51] really useful for for for those types of [25:53] use cases. Um, that being said, I will [25:55] say I've I've seen a little bit less [25:57] usage now that context windows are so [25:59] big and you can do um, you know, context [26:01] prepending for the um, API runs. So, for [26:04] example, you know, before when we [26:06] started off with LMS a few years ago, [26:08] you know, you could maybe throw a couple [26:10] paragraphs in at a time and that was [26:12] really the entire kind of memory window [26:14] of the LLM uh, to answer any query. Now, [26:18] um, you know, Gemini has uh, up to a [26:20] million tokens. I think uh, the new [26:22] llama had 10 million. And so some of [26:23] these models have really really massive [26:25] context windows. So there is a little [26:27] bit more of a debate sometimes with [26:28] customers deciding between those options [26:30] on full fine-tuning. Again that that is [26:32] something we still do see a lot with [26:34] open models that we offer. Um [26:36] particularly you know to return to the [26:37] example I gave earlier of this kind of [26:39] therapy model. um you know that might [26:41] have been something that they could [26:43] accomplish with PEFT with like a lot of [26:45] effort but it was a lot better for them [26:47] to train a model where they were [26:49] actually modifying the weights and [26:51] ultimately uh not only was that better [26:53] for them but the ownership over the life [26:55] cycle of the model and the ability to [26:57] deploy it in any specific you know data [26:59] center or region was super super [27:01] important for that customer. So, I know, [27:03] you know, one one one very real issue we [27:05] run into with a lot of enterprises these [27:07] days, especially if you're in like [27:08] fintech or healthcare, is the the the [27:11] average shelf life of an LLM is like 6 [27:13] months. And so, if your background is in [27:15] enterprise software, you know that these [27:17] are like um very very fast upgrade [27:20] cycles compared to like, you know, how [27:21] often you would like upgrade your [27:23] Salesforce instance five or 10 years [27:24] ago. And so, a lot of, you know, [27:26] companies are just not ready for that. [27:28] and they want to be able to control how [27:30] long they run a model even at the cost [27:32] of maybe not running the most [27:34] state-of-the-art model. So those are [27:35] reasons why, you know, it might be [27:37] better um in some cases to to full fully [27:39] fine-tune an open model. Um but we we do [27:41] still kind of see a range of different [27:43] uh different uh uses there. [27:46] >> Okay. Uh I just add a little bit um [27:48] about my my panelist we already [27:52] mentioned and uh for our user case uh is [27:56] specifically because we are more focused [27:59] on because I'm financial services is [28:01] more traditional one right so sometime [28:04] we are not uh uh we we can't we we can't [28:08] bear about the latency so for that is [28:10] the case and also with the content [28:14] window become larger and larger So for [28:16] this we might just give a really good [28:19] design of the content to the like a open [28:22] eye call right. So for API call for the [28:24] those type of uh the prioritary model [28:28] but however for some content some [28:31] component we require more efficient for [28:35] those type saying we require a lot of [28:38] domain knowledge for those type and also [28:41] for those uh component we might require [28:44] uh kind just I mentioned is a small [28:46] language model for those open weight so [28:49] for those things we need to do the [28:51] finetuning so we move more because I'm [28:53] maybe parameter uh efficiency type is [28:57] the Q laura or Lara type sense but [28:59] however the case is just one of my [29:02] friends also um discuss with me he he [29:06] he's from the uh startup company so he [29:10] told me if you are not worried about the [29:14] efficiency don't do any final tuning [29:17] just go to to design your content Yeah, [29:24] >> cool. Um, we have discussed about use [29:27] cases and fine-tuning techniques. Let's [29:30] talk about some of the challenges that [29:32] come into the picture with the usage of [29:34] generative AI models. Um, so let's talk [29:38] about words like hallucinations [29:41] and latency issues, costs issues. How [29:45] have you dealt with uh such issues in [29:48] your use cases? Have you come across [29:50] these problems? How did you deal with [29:52] it? [29:53] >> Okay, [29:54] >> I can I can kick off there because that [29:56] again is like a decision point that [29:58] often kind of flows from like the the [30:00] model you choose and the consumption [30:01] that that you want. Um so the good news [30:04] for you know generally for um for [30:07] foundation model users is that obviously [30:09] the cost of tokens has you know [30:11] dramatically decreased over the last few [30:13] years and every you know few weeks or [30:15] months we get a new cycle of some [30:17] unbelievably efficient new model. Um so [30:19] that's that's a really you know exciting [30:21] opportunity particularly if um you know [30:24] the cost seemed prohibitive a few years [30:25] ago when these models first started [30:27] rolling out. Um I think you know some of [30:30] the challenges that that I see uh and [30:32] these probably won't surprise anyone. Um [30:34] on the cost front it it's really going [30:37] to come down to uh what what is your [30:39] like what are your concurrency and [30:40] latency needs and what exactly is is [30:42] your use case. So, for example, um you [30:45] know, like I mentioned earlier, if [30:47] you're if you're you know, if you have a [30:48] chatbot startup or or you're you're [30:50] mostly just kind of trying to produce a [30:52] lot of tokens for a very generalized use [30:55] case, um generally speaking, uh you [30:58] know, a managed API is going to give you [31:00] the best option and there are many [31:01] wonderful managed API providers that [31:04] will um sell you very cheap tokens and [31:06] and you'll get a pretty good experience [31:07] there. for large enterprises sometimes [31:10] um you know one of the problems that [31:11] comes up with those APIs is like noisy [31:13] neighbors and just peak demand for [31:15] certain regions. Um at Google and and I [31:17] think all the CS the big CSPs do this um [31:20] you can uh buy into a product called [31:22] provision throughput which essentially [31:24] provides you with some isolation that is [31:26] not quite but more similar to if you [31:28] were you know owned your own GPU and [31:30] were were running the model yourself. [31:32] Um, you know, the other option of [31:33] obviously is to run an open model and [31:35] you can at that point then just decide [31:37] what your budget is for GPUs and spend [31:39] as much as as you're comfortable and and [31:41] if you have um a great engineering team, [31:43] they can really optimize the heck out of [31:45] the serving stack and and get you really [31:47] good results. And there's lots of very [31:49] cool benchmarks that um you can look [31:52] into there. So, from a cost perspective, [31:54] you know, again, it's it's really about [31:55] what your goals are. The good news is [31:58] there's lots of products that are going [31:59] to kind of meet whatever goals you have. [32:01] I think the other kind of challenges [32:03] that I see um one you know as mentioned [32:06] this morning you know that models aren't [32:08] perfect. We still see issues with [32:10] quality and hallucinations. I think [32:12] that's a lot better than it used to be. [32:14] Sometimes one thing that I find really [32:16] frustrates customers though is that with [32:18] each generation of model sometimes you [32:20] have improvements and sometimes you have [32:21] regressions in certain areas. So you [32:23] might see, you know, generic benchmarks [32:27] like MMLU keep going up, but then you'll [32:30] hear like, oh, it's actually worse at [32:32] this like one little tiny use case that [32:34] I have for my model. And it's it's just [32:36] it's very difficult. You know, the teams [32:38] do as much as they can to catch those [32:40] issues, but they they can't always do [32:41] it. Um, and then and then I think the [32:43] other one is like I mentioned earlier [32:45] for the enterprise, it's life cycle [32:46] management. Um, if you're an admin or an [32:49] IT person, I know this has probably been [32:51] stressing you out for the last year, but [32:52] but like we uh I've been working in [32:54] enterprise software for over 15 years [32:56] now. And the upgrade rate that that [33:00] companies were used to for these big, [33:02] you know, um, software programs was like [33:05] every few years at most. Even SAS like [33:07] only changed that a little bit. Now [33:09] we're talking about models, you know, [33:11] really being basically obsolete after [33:14] six months between generations. And so [33:16] this is a lot this this can create a ton [33:19] of a burden for for companies that are [33:21] looking to be at the cutting edge but [33:23] also want to have you know good [33:25] practices around safety and security and [33:27] and doing their own internal benchmarks. [33:29] So it it's just uh it's something that I [33:32] think is is just kind of a new reality [33:34] for how we we deploy these models. [33:37] >> Here's the thing, right? We have some [33:38] visibility into like calling [33:41] applications before and after LLM [33:43] adoption. And what do people find? They [33:46] find that the LLM application doesn't [33:47] immediately outperform the nonLLM [33:50] application, right? Some sort of [33:51] intentbased classifier that people are [33:53] using with really rigid call flows. [33:56] However, we've seen that creep up like [33:58] just month over month 3 or 4%. to the [34:01] point right where at the cutting edge [34:03] people are fine-tuning open weights [34:05] models to get from right like in the [34:08] fast food ordering context something [34:09] like 75% successful order rate to 85 [34:13] 90%. [34:15] At the same time, what do people know [34:17] notice, right? Like when they were by [34:20] hand writing call flows, right? Like [34:22] they're writing every single thing that [34:23] the bot could say, some human wrote. [34:26] When they switch to LLMs, what do they [34:27] find? The cost for text to speech [34:29] increases. The bots are generating [34:31] longer and more chattier responses. [34:33] They're really good at this, right? And [34:36] so cost has gone up for these [34:38] applications. I think at the same time [34:40] in many ways LLMs are probably the [34:44] fastest commodifying product in human [34:46] history. Um [34:49] does the fact that they're a commodity [34:50] mean it's cheap to run an application? [34:51] No, it doesn't mean it's cheap to run an [34:53] application. It's still expensive to run [34:55] an application. Um we also notice right [34:59] like when it comes to cost there's sort [35:01] of KP there's KPIs that are built into [35:03] enterprise calling that people are [35:05] rethinking right like one of those [35:06] things is average handling time. Average [35:08] handling time being the time uh from [35:11] which a customer is connected with the [35:13] agent to that pro that customer's [35:15] problem being solved. And a phone order [35:18] again success used to be like 60% but [35:21] for those 60% of orders that were [35:23] successfully automated they were h those [35:25] orders were taking place within like 2 [35:27] minutes. What they're noticing now right [35:29] is like order success is 85% but an [35:32] average order is taking like four or [35:33] five minutes. [35:35] That has implications for cost. It has [35:37] implications for hallucinations. Of [35:38] course, the longer the I'm sure you've [35:40] had this experience too, like one of [35:41] these consumerf facing LMS like on [35:44] Google AI studio or chat GPT, right? [35:46] Like the longer you talk to it, the more [35:47] you're like, does it remember the first [35:49] thing I said? Um, no. The answer is [35:52] often it doesn't. Um, [35:55] at the same time, right, like customer [35:56] satisfaction when you call these [35:58] businesses is going up, right? And so I [36:01] think there's some reconfiguration of [36:03] priorities that's happening where people [36:04] are saying, "Okay, maybe we're okay with [36:06] an order taking 5 minutes if people a [36:09] successfully order more often." And this [36:11] applies to customer support. It applies [36:13] to many different enterprise calling [36:15] applications. And b people are happier [36:17] when they get off the phone. Um, so I [36:21] don't know. It's it's it's I can't speak [36:22] necessarily directly to hallucinations. [36:24] I know it's a problem, by the way. Like [36:26] people are calling dominoes and talking [36:27] for 30 minutes because it went off the [36:29] rails at some point and they just liked [36:32] having the conversation at that point. [36:34] Um, and by the way, those are the most [36:37] expenses without saying on a consumption [36:38] based pricing model. If that doesn't end [36:40] in an order and you just spent right, [36:42] you ran up a bill for someone talking to [36:44] Chat GBT for 30 minutes, you're not [36:47] super happy with that. Um, so yeah, [36:50] again, wild west. [36:53] Yeah, I I I just rely on just add a [36:57] little bit um point on that especially u [37:01] we we talk about the human nation right [37:03] so the case is uh so sometime we need to [37:07] think about how we can you uh redefine [37:10] the definition of the humanization [37:13] because the case is this okay the the [37:17] model the no matter small model or large [37:20] model naturally they are more likely is [37:22] like a behavior type of sense model. [37:25] Okay. So sometime when we use a model we [37:27] use this model use as to do the more [37:30] likely because I'm a quant background we [37:32] use that as a quant. So this is of [37:36] course you use like quantum model [37:37] definitely you have hation actually for [37:39] another way around is is a model problem [37:42] or the users problem right. So this is [37:45] why you have to think about what you [37:47] design your system and how you can avoid [37:50] those types. But for another way around [37:52] is sometimes even you design system very [37:55] carefully you understand what's going on [37:57] you still get a hallucination why is [38:00] because you want give the model autonomy [38:04] right so if you want to give autonomy [38:06] they have some freedom so even for [38:09] example even you have a junior analyst [38:12] right he join the team you give him [38:14] training you give everything exactly [38:16] what he'll do in the end he give your [38:20] result you still say something is off. [38:23] So it then we we have to go back to see [38:26] what's your tolerance [38:28] if it's all depend on your tolerance [38:31] what's risk you want to take. So this [38:34] just my point. Yeah. [38:36] >> Yeah points actually. Um I'll offer like [38:40] a couple of points on hallucinations. I [38:43] think that was so um typical LMS uh they [38:48] you can use like rag based [38:49] architectures. So people use rag based [38:51] architectures or external guard rails to [38:54] deal with hallucinations up to a certain [38:56] point. [38:58] I also want to talk about certain use [39:00] cases for hallucinations in multimodal [39:03] large language models. So this happens [39:05] quite frequently because you have both [39:07] text and image inputs and often you see [39:11] the models relying on the text inputs [39:13] more rather than the image ones and [39:15] there's no linkage at all. So you you [39:18] you look at that more often and um these [39:21] days the MLM's use is uh increasing a [39:24] lot. So this has become a very important [39:26] issue to deal with and what we have [39:28] generally dealt with with this is trying [39:31] to make the model balance between both [39:34] the inputs uh the text and the image and [39:37] you do this by so by you you want to [39:40] make the model rely on the images more. [39:42] So you deliberately inject noise into [39:44] the text and this kind of perturbation [39:46] techniques are one of the techniques [39:48] that we've been using to deal with uh [39:50] hallucinations where more than one modes [39:52] are involved. [39:56] >> Great responses. [39:58] Um let's wrap up looking forward [40:02] which uh emerging technology or research [40:07] area are you looking forward to and [40:12] uh you're looking forward to personally [40:15] or which which area do you think is [40:17] overhyped? [40:19] I can go. Um I won't take a lot of time [40:23] but uh as I think a lot of people heard [40:28] earlier uh at the keynote uh we see a [40:31] lot of scope in uh uh biosciences like [40:36] data track discovery and uh structure of [40:38] proteins and things like that. I think [40:40] that is an area that has been untapped [40:41] yet and there's a lot of scope for [40:43] development. So I'm really excited about [40:45] that area. Um the other thing is the [40:49] different modes of data. So I'm very [40:51] excited about the modes like unified [40:53] models which can take into account the [40:56] different modes at the same time and [40:58] have a more co cohesive learning uh as [41:01] we do in in our everyday lives instead [41:04] of currently we have different models [41:06] taking different modes and then they [41:07] sort of merge them together either [41:09] before or after. So I think these two [41:11] are the areas that I feel were very [41:13] promising and I'm excited about those. [41:15] uh one overhead area I feel is AGI. Uh [41:22] I know people have different kinds of [41:24] opinions about it but personally I feel [41:26] like uh the basic framework of AGI comes [41:30] from like how we as human beings learn [41:32] like from babies and such uh how how we [41:35] understand things and I think like I [41:37] don't have a lot of information about [41:39] what exactly is going to be the [41:40] structure of AGI but sure looking [41:42] forward to that. Yeah, we have couple of [41:46] minutes. Just letting everyone know. [41:48] >> Uh I'll I'll I'll be quick. Um so what [41:51] I'm looking forward to, I think is uh [41:54] just improvements in multimodality. I [41:57] think like multimodal models have been [41:59] around for a while. Obviously a lot of [42:01] them now do you know everything from you [42:03] know speech synthesis to image [42:05] generation. But I think like we're just [42:06] sort of at the the tip of the spear [42:08] there. And it's I I like I personally [42:11] have noticed as a product manager I find [42:13] models to be way way way more useful the [42:15] more multimodal they are the more I can [42:17] you know feed in charts and graphs and [42:19] you know voice memos and stuff like [42:20] that. And so I think um as those those [42:23] things improve the the better off we'll [42:24] be. Um I'm also excited for the kind of [42:28] generation of models you know beyond or [42:31] building on the transformer. I feel like [42:32] this has kind of been a cliche for the [42:33] last few years like there's going to be [42:35] a thing beyond transformers. We're [42:36] starting to see like text diffusion [42:38] models and I work with science models at [42:40] at Google as well and we're starting to [42:42] roll some of those out that are [42:43] diffusion based. So, um, a lot of [42:45] exciting stuff. Um, you know, I won't I [42:48] won't be a hater on anything, but I will [42:50] say like I share your skepticism of AGI. [42:52] I think like in my experience, the hard [42:53] thing is the pipelines. Like let's get [42:55] the pipelines right. We haven't even [42:56] like, you know, gotten use out of 99% of [42:59] these models at this point. And so once [43:01] that happens, there's going to be like a [43:03] massive cheat sea change in the the [43:05] usefulness of these models even as they [43:06] stand today. [43:09] I I think we don't have to believe in [43:10] AGI to believe that even if progress [43:12] stopped today that the effects would be [43:14] hugely transformational, right? Like the [43:17] the internet wasn't that much of a [43:19] change over the telegraph, right? Like [43:22] we're talking about information over [43:23] wire behind TCP and HTTP and look what [43:26] that's done to our world in the last 45 [43:28] years. So, right? Like I think we don't [43:30] have to have iRoot to have something [43:32] that changes our lives and the lives of [43:35] all of our descendants forever. [43:37] >> Yeah. I just want to add one more point [43:40] here is I think uh right now if you're [43:43] looking back like one or two years ago [43:45] how you do your work, how you live a [43:49] completely change. So this changing [43:52] transforming effort is already here. So [43:54] we have to embrace reality about AI come [43:58] or not we'll see but however the the the [44:02] the technology uh is dramatically uh [44:07] change and we will have all all the [44:09] opportunity to connect all the together. [44:11] Yeah sorry [44:14] >> that's it we are out of time now. Thank [44:16] you. Thank you everyone. [44:17] >> Thank you everyone.