Snorkel AI Says Lack of Expert Data Stalls 80% of Enterprise GenAI

Snorkel AI称缺乏专家数据致80%企业生成式AI受阻 大多数企业生成式AI部署未能创造价值，是因为模型缺乏专业领域数据，这使得专家主导的程序化后训练成为迈向生产成功的真正瓶颈。 尽管企业AI采用率激增，但由于缺乏专业、高质量的领域数据，近80%的企业生成式AI项目未能产生切实的商业价值。迈向复杂的智能体工作流，需要超越基于网络抓取的常规预训练，转向由专家驱动的后训练和程序化评估框架，以克服领域专家资源有限的问题。Ray Mi阐述了Snorkel AI的“专家数据即服务”（Expert Data-as-a-Service）如何利用数万名专业领域专家并结合自动化验证，以两倍于传统方法的速度达到目标生产精度。归根结底，弥合企业性能差距依赖于一个迭代的“发现与修复”闭环，通过持续诊断系统瓶颈，从而交付稳健、可投入生产的AI。 尽管企业对生成式 AI 的采用率从 55% 攀升至 78%，但 80% 的项目未能创造实质业务价值，48% 的企业高管指出缺乏专业领域数据是实现业务落地的主要瓶颈。 (0:04) 生产级智能体（Agentic）系统在工具与 API 调用中容易发生级联错误，通用现成大模型与简单提示词已无法满足严苛的业务准确度要求。 (2:08) 实现企业级稳健性依赖三大支柱：后训练黄金数据集（SFT/RLHF）、专家评估及端到端开发流，需通过程序化评估框架放大稀缺领域专家的知识价值。 (5:47) 前沿 AI 训练正转向专家驱动的后训练模式；某大型电信公司借助覆盖核心分类的高质量数据与程序化评估，将账单客服模型的准确率从 55% 提升至 95%。 (8:21) Snorkel AI 依托跨越 1000 多个细分领域的专家网络提供专家数据即服务（Expert Data-as-a-Service），帮助客户以比传统方案快两倍的速度达成目标准确率。 (13:03)

Ray Mi of Snorkel AI argues that enterprise generative AI scaling is no longer constrained by compute or model parameter size, but by a lack of specialized, expert-curated domain data needed to ground agentic systems. The Enterprise AI Adoption and Value Disconnect (0:04) According to Stanford HAI research cited by Mi, enterprise AI adoption grew from 55% to 78% year-over-year. However, this deployment surge has exposed an operational gap: 80% of generative AI initiatives fail to produce tangible business value, and only 13% are deeply embedded into core business processes. Mi attributes this breakdown to data misalignment rather than architectural limits, noting that 48% of C-suite executives identify the absence of specialized domain data as their foremost operational bottleneck. Generic pre-training and basic prompting prove sufficient for prototypes like summarization, but falter when exposed to domain complexity and rigorous accuracy standards. Sequential Error Cascades in Agentic Systems (2:08) Production environments increasingly require transitioning from standalone large language models to agentic architectures. In these configurations, an orchestrator coordinates across user inputs, external tools, APIs, and LLM endpoints. Mi notes that agentic workflows introduce compounded vulnerability: errors cascade sequentially across tool usage, API integrations, and multi-turn user clarifications. Consequently, even advanced foundation models cannot compensate for upstream operational failures or inaccurate inputs, necessitating system-level robustness across post-training data, expert evaluation, and iterative development. The Post-Training Shift Toward Expert Data (8:21) Mi outlines an industry shift across three eras: pre-2023 models relied on massive, unlabeled web crawls for basic next-token completion; 2023 to 2024 emphasized general reinforcement learning from human feedback (RLHF) via non-specialists for conversational tasks; and 2025 onwards demands expert-driven post-training for domain-specific agents, deep research, and code generation. Indiscriminate web-scraped data contaminates advanced models, requiring data that is expert-generated, noise-free, diverse, and structured for multi-step reasoning. In one telecommunications case study, distilling an LLM into a smaller billing chatbot using seven distinct taxonomy categories lifted production accuracy from a 55% baseline to 95%. Scaling Domain Evaluation via Programmatic Feedback Loops (5:47) A critical barrier in deploying enterprise AI is the evaluation gap. While machine learning engineers can build agentic pipelines, they generally lack the specialized judgment required to verify domain-specific outputs. Conversely, qualified professionals, such as bank credit underwriters, typically have minimal availability—often as little as two hours per project. To resolve this scarcity, Snorkel advocates encoding expert judgment into programmatic evaluators paired with expert-versus-expert review loops. Mi stresses that production readiness depends on continuous 'find and fix' loops, where teams systematically isolate pipeline errors and apply targeted domain data to fine-tune retrieval-augmented generation and tool-calling components. Mi positions programmatic evaluation and specialized expert networks as essential infrastructure for achieving production-grade GenAI. However, the operational model assumes ongoing access to specialized domain experts and rigorous rubric design, leaving open the challenge of how smaller organizations without dedicated expert pipelines will maintain and audit rapidly drifting enterprise workflows.

Snorkel AI售前工程总监Ray Mi指出，企业生成式AI落地的核心瓶颈已从模型参数与算力规模转向高质量专业领域数据，单纯扩大模型无法消除落地阻碍，必须依靠专家数据与程序化评估构建专精AI系统。 企业AI落地率与实际业务价值脱节 (0:04) 根据斯坦福大学HAI的研究数据，企业AI的采用率同比已从55%攀升至78%，但目前高达80%的生成式AI方案未能创造实质性的商业价值，仅有13%的AI引擎被深度集成到核心业务流程中。Ray Mi指出，这种采用与价值创造之间的巨大落差，根源在于数据与实际业务的不匹配。尽管企业高管积极推进AI，但约有48%的最高管理层明确承认，缺乏高质量、垂直领域的专业数据是将生成式AI转化为生产力的最主要阻碍。基础大模型、参数量或底层算力不再是根本瓶颈，专业领域的高质量数据才是驱动企业业务价值的决定性要素。 通用模型局限与智能体系统的级联错误 (2:08) 开箱即用的基础模型和简单提示工程在文本摘要等原型验证中表现良好，但在面对复杂的垂直业务与严苛的准确性标准时会迅速失效。以医疗辅助问答为例，医生需要极其精准可靠的判断，一旦模型出现失误，临床人员便会重新依赖个人经验。在向高阶专用系统过渡时，行业正在进入由多工具调度构成的智能体架构体系。智能体负责接收请求、编排外部工具与API，并调用大语言模型生成最终答案。然而，在真实的多步骤工作流中，工具调用、用户追问及接口调用的细微失误会逐级累积，引发连锁级联错误，即便采用如GPT-5级别的高智能模型，也会因上游执行故障而导致整个业务流程瘫痪。 后训练范式转向专家数据与程序化评估 (5:47) 前沿AI研发正在经历范式转移：2023年之前依赖海量非结构化网络爬虫数据进行预训练，2023至2024年转向普通标注员的通用人类反馈强化学习（RLHF），而自2025年起，行业核心聚焦于领域专家主导的高阶后训练（SFT与RLHF）。低质甚至被污染的网络数据无法推动模型有效拓展，生产级系统依赖具备专家经验、高纯净度、分类平衡且包含复杂多轮交互的“黄金数据集”。例如，在某大型电信运营商的账单客服模型蒸馏项目中，团队提炼出7大核心分类法，将人工生成数据与程序化评估结合，使小模型在生产环境中的准确率从55%提升至95%。与此同时，数据科学家通常缺乏严苛的行业判断力，而信贷核保等领域专家的可用时间极度有限，这要求工程团队必须将专家智慧编码为程序化评估框架，以自动化方式扩大评测规模。 专家数据服务与端到端闭环优化实践 (13:03) 为解决高质量专业数据短缺的痛点，Snorkel AI推出了专家数据即服务模式，目前运营着由数万名跨越1000多个专业领域的行业专家组成的网络。在质量保障机制上，平台结合专家间相互校验与自动化程序评估，使客户达到89%目标准确率的速度比传统市场方式快达两倍。此外，团队在Hugging Face上开源了金融推理、保险核保以及Wordle游戏逻辑等6个复杂基准数据集，并跟踪展示单位美元性能指标，辅助工程团队在工具调用与图推理场景中平衡成本与性能。其核心实施方法在于构建“发现并修复”的持续迭代闭环：通过连续评估定位系统缺陷，针对性补充专用数据来微调嵌入模型、升级检索增强生成（RAG）管道或优化工具调用，从而持续推高生产就绪度。 该演讲强调，企业实现AI价值的核心路径在于通过专家数据与程序化评估持续修复系统漏洞；然而，在面对极其庞杂的多步骤智能体任务时，如何长期维持万级专家网络的评审一致性，并在严苛成本限制下实现微调收益的最大化，仍是工程落地中需要持续权衡的挑战。

## Background 该视频由行业会议频道 Ai4 于 2025 年 8 月 27 日发布在 YouTube 平台，由 Snorkel AI 售前工程总监 Ray Mi 博士主讲。视频简介指出，随着生成式 AI 从实验室走向生产环境，企业面临着模型扩展容易但性能扩展困难的困境，演讲重点探讨了数据为何成为规模化落地的核心瓶颈，并介绍了 Snorkel 基于专家在环（Expert-in-the-loop）的程序化数据开发平台。 ## TL;DR 演讲指出虽然企业 AI 采纳率已达 78%，但仍有 80% 的生成式 AI 项目未能创造实质业务价值，其根本原因在于模型缺乏高质量的垂直领域数据支撑。面对复杂的智能体（Agentic）系统与级联误差挑战，AI 数据范式已从早期的通用众包标注演进为以多样性、高复杂度和领域专家驱动为核心的专家数据后训练时代。通过北美头部电信运营商客服模型蒸馏（准确率从 55% 跃升至 95%）等实战案例，演讲展示了 Snorkel 的“专家数据即服务”与程序化评估能力。最终，团队提出了一套闭环的“评估-开发”与“发现-修复”迭代飞轮，强调唯有依靠高质量专家数据才能让模型真正达到生产就绪标准并转化出商业价值。 ## Atmosphere & style 整场演讲充满务实严谨且极具说服力的行业洞察氛围，主讲人 Ray Mi 以深厚的统计学学术底色与丰富的工业界实战经历娓娓道来。通过即兴的现场互动与亲历项目趣事（如专家库建立初期亲自上阵编写高难度数学题难倒模型），主讲人将复杂的 Agentic 架构与数据瓶颈拆解得生动透彻。整体叙事节奏紧凑、逻辑层层递进，既有对行业痛点的犀利剖析，又有清晰可执行的方法论指导，给观众带来专业且富有启发的临场体验。 ## Key takeaways - [00:04](https://www.youtube.com/watch?v=l16i2sigGgw&t=4s) 企业 AI 采纳率虽升至 78%，但高达 80% 的生成式 AI 项目未能创造实际商业价值，核心瓶颈在于缺乏高质量的专业领域数据。 - [02:08](https://www.youtube.com/watch?v=l16i2sigGgw&t=128s) 企业级生产环境要求兼顾高任务复杂度与高输出准确度，简单的开箱即用基础模型已无法满足业务需求，系统正向复杂的智能体（Agentic）范式演进。 - [04:47](https://www.youtube.com/watch?v=l16i2sigGgw&t=287s) 智能体系统的多步骤工具调用和长执行链路极易引发误差级联，必须依靠能够覆盖边界案例的黄金数据集、程序化评估框架与端到端开发闭环来保障稳健性。 - [08:21](https://www.youtube.com/watch?v=l16i2sigGgw&t=501s) AI 数据范式已从通用预训练、众包后训练演进至专家数据后训练时代，仅凭海量通用互联网语料不仅无法扩展性能，还会引发数据污染。 - [08:21](https://www.youtube.com/watch?v=l16i2sigGgw&t=501s) 构建高质量专业数据需遵循四大核心标准：专家驱动（具备真实从业经验）、高品质、多样性以及适配多步推理的前沿复杂性。 - [10:28](https://www.youtube.com/watch?v=l16i2sigGgw&t=628s) 针对北美头部电信运营商的实战案例中，通过提炼 7 大账单分类并结合专家合成与程序化评估，专属 SLM 客服模型基线准确率从 55% 暴增至 95%。 - [13:03](https://www.youtube.com/watch?v=l16i2sigGgw&t=783s) Snorkel AI 推出专家数据即服务，依托覆盖 1000+ 细分领域的专家网络，在达成 89% 严格准确率的同时比同类竞品交付速度快 2 倍。 - [13:03](https://www.youtube.com/watch?v=l16i2sigGgw&t=783s) 开源评测榜单首创性地引入“每美元性能比”（Performance per Dollar）指标，为企业在模型性能与 Token 调用成本之间权衡选型提供依据。 - [15:06](https://www.youtube.com/watch?v=l16i2sigGgw&t=906s) Snorkel 采用“专家对抗”同行审核与程序化评估器双重质检机制，通过“评估诊断-定向开发”的数据飞轮闭环靶向优化工具调用和 RAG 检索。 - [17:08](https://www.youtube.com/watch?v=l16i2sigGgw&t=1028s) AI 落地需依托持续滚动的“发现-修复”迭代闭环，只有在生产环境中达到极高的业务采纳率，才能真正将生成式 AI 转化为商业价值。 ## Detailed walkthrough ### [00:04](https://www.youtube.com/watch?v=l16i2sigGgw&t=4s) 模型无法扩展，而数据可以：企业 AI 落地的现实与瓶颈 Snorkel AI 售前工程总监 Ray Mi 博士在演讲开篇分享了自己的统计学与机器学习学术背景，并以一组震撼的行业数据切入主题。在向现场观众进行即兴调研时，Ray 发现台下已有大量企业和组织将 AI 解决方案部署到了生产环境中。根据斯坦福大学以人为本人工智能研究院（Stanford HAI）的研究报告，企业对 AI 的采纳率已从一年前的 55% 飙升至 78%，这表明在组织层面，技术采纳（Adoption）本身已不再是核心阻碍。然而，光鲜的采纳率背后隐藏着严峻的落地鸿沟：高达 80% 的生成式 AI 项目未能为企业创造实质性的商业价值，且仅有 13% 的 AI 引擎真正深度融入了关键业务流程中。 造成这种商业价值与采纳脱节的核心根源在于“数据不匹配”（Data Misalignment）。高达 48% 的企业 C-suite 高管已深刻意识到，缺乏高质量、专业化的领域数据是阻碍生成式 AI 方案实现规模化运营的最大痛点。尽管业界都在热议并投入 AI，算力与基础模型也在持续扩张，但真正的扩展瓶颈从来都不是模型架构本身，而是喂给模型的数据——即“模型无法自行扩展，只有数据才能驱动扩展”（Models Don't Scale, Data Does）。 - **企业 AI 采纳率**：从一年前的 55% 攀升至 78%（数据来源：Stanford HAI）。 - **商业价值转化鸿沟**：80% 的生成式 AI 项目未能为企业带来切实可衡量的业务价值。 - **关键业务集成度**：仅有 13% 的 AI 系统实现了与核心业务流程的深度融合。 - **高管核心关切**：48% 的管理层指出，缺乏高质量专业领域数据是 AI 方案落地的首要制约。 --- ### [02:08](https://www.youtube.com/watch?v=l16i2sigGgw&t=128s) 从开箱即用到专业智能体：复杂性与准确性的双重挑战 回顾当前 AI 的发展格局，随着各类开源与商业大语言模型（LLM）、多模态模型的涌现以及算力资源的充沛，企业面临的真正瓶颈不再是模型参数量或 GPU 显卡资源的匮乏。在 ChatGPT 刚问世时，大众倾向于直接使用“开箱即用”（Off-the-shelf）的通用基础模型，通过简单的提示工程（Prompting）来完成通用问答、文本摘要或简单的内容生成。这些通用方法对于构建概念验证（PoC）原型行之有效，但在面对严苛的企业级生产环境时却往往折戟沉沙。 要将通用 AI 推进为能够真正创造业务价值的专用 AI，必须在两个关键维度上取得突破：**任务复杂度**与**输出准确度**。以医疗场景为例，如果开发一个协助医生诊断和制定治疗方案的医疗问答机器人，其输出的准确性和可靠性必须达到极高标准；否则，医生宁可凭借自身数十年的临床经验行事，也不会信任或采纳 AI。随着任务从单步问答向深层领域工作流推进，系统必然向智能体（Agentic）范式演进。在 Ray 对现场听众的二次互动中，全场无一人在过去两天的会议中未曾听过“Agentic”一词，足见其火爆程度。从逻辑架构上看，一个典型的智能体系统包含三大核心要素：用户交互层、工具集调用接口（Tools），以及作为推理和指令中枢的 LLM 接口。 ```mermaid flowchart LR accTitle: 智能体系统核心逻辑交互架构 subgraph ClientLayer ["用户交互层"] U["最终用户 (User Request)"] end subgraph AgentCore ["智能体调度中枢 (Agent Engine)"] A["智能体中枢 (Orchestrator)"] end subgraph ExecutionLayer ["模型与工具执行层"] LLM["大语言模型 (LLM Endpoint)"] T["业务工具集 (APIs / DB Tools)"] end U -->|"输入请求与追问"| A A -->|"发送 Prompt / 上下文"| LLM LLM -->|"推理决策 / 工具调用指令"| A A -->|"执行 API / 数据查询"| T T -->|"返回结构化数据"| A A -->|"输出高质量结果"| U ``` --- ### [04:47](https://www.youtube.com/watch?v=l16i2sigGgw&t=287s) 智能体系统的稳健性基石：训练、程序化评估与端到端开发 在现实业务落地中，智能体系统的端到端链路极其错综复杂，涉及用户意图理解、澄清追问、多阶段 API 调用以及模型端点路由等多个环节。这种长链路架构带来了一个致命问题：上游任何一个微小的误差都会在执行链中逐级放大，产生连锁级联故障。即便底层采用像假设中的 "GPT-5" 那样极其聪明、极少犯错的顶尖模型，一旦工具调用或澄清交互环节出错，整套系统的表现就会彻底崩塌。为了让脆弱复杂的智能体系统在金融、医疗等强监管行业中达到高稳健性（Robustness）并推动实际采纳，必须牢牢把握数据层面的三大关键抓手： 1. **模型训练与基准构建（Training & Benchmarking）**：在后训练（Post-training）、监督微调（SFT）和强化学习（RLHF）中，必须依赖能够精确覆盖罕见事件和边界案例（Edge Cases）的“黄金数据集”（Golden Dataset），以此教会模型遵循特定的专业业务准则（Rubrics）。 2. **程序化评估（Evaluation）**：数据科学家和机器学习工程师擅长搭建系统架构，却普遍缺乏深度的垂直业务知识，无法准确评估模型在特定业务下的好坏；而真正的业务专家（如银行信贷审批员）日常精力极其有限，往往在一周内只能抽出两小时配合。解决这一矛盾的关键在于通过“程序化评估框架”，将专家的专业判断与标注规则提炼并放大为自动化评估器，以极低的专家时间成本实现与人类专家高度对齐的持续质检。 3. **端到端开发闭环（Development）**：除了优化单个组件模块，还必须有一套敏捷、可扩展的开发框架，将数据、评估与各子模块严密拼接，确保整套生成式系统的端到端输出质量达到生产上线标准。 --- ### [08:21](https://www.youtube.com/watch?v=l16i2sigGgw&t=501s) AI 数据演进三阶段：专家数据驱动前沿落地 从 2023 年至今短短三年内，AI 模型的数据范式经历了三个极其清晰的发展阶段。在 2023 年 ChatGPT 爆发前，行业处于以海量无标注互联网语料进行自监督**预训练（Pre-training）**的时代，模型的核心目标是在通用语料库中预测下一个 Token；随后进入以通用人类反馈强化学习（RLHF）为代表的**早期后训练阶段（Post-training）**，这一时期的数据主要由未经垂直专业训练的通用众包标注员提供，只能满足通用对话和泛化场景的需求；而当前行业已全面迈入**专家数据后训练时代（Post-training with Expert Data）**，专注于攻克复杂的领域特定任务，并支撑起代码智能体（Code Agents）与深度研究（Deep Research）等前沿应用。 海量且低质的通用互联网数据不仅无法帮助模型在专业垂直场景中实现性能扩展，反而会产生严重的数据污染，损害系统的可靠性。推动模型能力前沿向前演进的唯一动力就是“专家数据”。构建用于后训练或基准评测的高质量专业数据，必须严格遵循四项核心标准： - **专家驱动（Expert-driven）**：数据必须来自长期深耕该领域、具备真实从业经验并与业务对象频繁互动的专业人士，唯有他们才洞悉真实世界中的业务边角案例与隐性逻辑。 - **高品质（High Quality）**：标注过程需具备严密的逻辑推导与严格的事实验证。 - **多样性（Diversity）**：不仅关注类别分布，更强调各类业务属性组合下的丰富情境覆盖。 - **前沿复杂性（Frontier Complexity）**：摆脱传统的简单单轮问答或选择题，全面适配智能体所需的多步骤推理、多轮工具交互与多模块协调复杂度。 --- ### [10:28](https://www.youtube.com/watch?v=l16i2sigGgw&t=628s) 数据多样性与实战案例：从河流监测到电信客服模型蒸馏 在阐述数据多样性时，Ray 指出企业客户在向 Snorkel AI 提出数据规范要求时，最为看重的就是类别分布以及多条件组合下的场景覆盖广度。他列举了一个计算机视觉中的典型案例：在进行特定河流的防汛监控时，网络上能采集到的公开图像几乎全部是白天的晴朗画面；但从实际防灾角度来看，夜间往往是依赖机器自动化监控的核心时段，因为夜间缺乏人工巡检。如果在构建数据集时忽略了夜间低照度图像的多样性补全，模型在真正需要发挥作用的关键时刻就会彻底失效。 另一个极具代表性的工业级案例，是 Snorkel AI 协助北美排名前三的一家头部电信运营商（Telco）构建账单客服系统的项目。该客户希望将复杂庞大的基础大语言模型蒸馏（Distill）为精简高效的小语言模型（SLM），以便低成本、低延迟地处理用户关于月度账单明细的咨询。 ```mermaid flowchart TD accTitle: 电信运营商账单客服模型专业数据开发流程 subgraph Step1 ["分类体系提炼"] T["识别 7 大核心账单业务类别 (Taxonomy)"] end subgraph Step2 ["专家数据合成与质检"] E["领域专家生成高难度样本"] P["程序化数据扩展与多样性合成"] V["自动化对齐评估与过滤"] end subgraph Step3 ["模型蒸馏与业务上线"] S["精调专属小语言模型 (SLM)"] D["基线准确率 55% 提升至 95%"] P2["正式投入生产环境部署"] end T --> E E --> P P --> V V --> S S --> D D --> P2 ``` 在该项目中，团队首先提炼梳理出用户最常咨询的 7 大核心账单分类体系（Taxonomy），随后结合人类领域专家的标注生成与程序化数据评估流程，构建了覆盖各种真实边缘场景的高度多样化数据集。这一专业数据流水线成功将该电信客户的 SLM 客服模型基线准确率从原先的 **55% 暴增至 95%**，顺利实现了生产环境上线。Ray 幽默地向观众提到：如果你恰好是这三家头部电信运营商之一的用户，并在与其账单客服机器人对话，你很可能完全意识不到对方是 AI，因为该模型在应对各种复杂账单问题时已经表现得极其出色。 --- ### [13:03](https://www.youtube.com/watch?v=l16i2sigGgw&t=783s) Snorkel 专家数据即服务与开源基准评测体系 为了系统化解决各行业对专业数据的渴求，Snorkel AI 正式推出了“专家数据即服务”（Expert Data-as-a-Service）解决方案。在演讲前一周，团队刚刚庆祝了向一家全球顶尖前沿 AI 实验室交付首个专家数据项目的一周年纪念。在过去一年的实践中，Snorkel AI 建立起了一个庞大的专家网络： - 涵盖 **1,000 多个细分专业领域**； - 汇聚了数以万计具备实战经验的垂直行业专家； - 在达成客户设定的 **89% 严格准确率目标**时，交付速度比市场同类竞品快出 **2 倍**； - 获得了前沿顶尖大模型实验室与财富 500 强（Fortune 500）企业的高度信赖。 与此同时，Snorkel AI 积极回馈开源社区，在 Hugging Face 等平台开源了 6 套针对高阶推理任务的基准数据集，涵盖金融推理、保险核保乃至 Wordle 字谜等复杂任务，专门评测各主流大模型在工具调用与多步推理上的边界表现。此外，Snorkel 还推出了公开排行榜，不仅展示各模型在各项专业任务中的性能评分，还首创性地引入了“每美元性能比”（Performance per Dollar）指标。Ray 强调，尽管每 Token 的 API 调用成本在持续下降，但对于动辄数百万次调用的规模化生产系统而言，成本仍是至关重要的商业决策维度，该榜单为企业在性能与 Token 成本之间权衡选型提供了透明的参考基准。 --- ### [15:06](https://www.youtube.com/watch?v=l16i2sigGgw&t=906s) 专家对抗与程序化评估：专业 AI 的数据飞轮 谈及 Snorkel AI 的核心技术壁垒与差异化优势，Ray 回顾了自己早期亲身参与的一个高难度项目：为前沿数学与统计学大模型定制包含复杂多步推理的问题集。由于项目要求极高，目标是设计出能够“难倒”顶尖大模型、使其通过率压低至 0% 到 20% 之间的极高难度题目，在成立初期专家库尚在搭建的一个月内，Ray 亲自作为统计学专家上阵充当标注员与“出题人”。 在保障数据质量方面，Snorkel 采用了独特的“专家对抗”（Experts challenging experts）审核机制，由一位专家对另一位专家的输出进行严苛挑战与同行评审；同时将长期积累的实战经验固化为程序化评估器，构建起“人类专家 + 自动化机器”双重质检防火墙。Snorkel AI 的核心使命十分明确——成为“专业化 AI 的核心数据层”（Data Layer for Specialized AI）。 ```mermaid flowchart TD accTitle: Snorkel AI 评估与开发迭代数据飞轮 subgraph EvalPhase ["评估诊断阶段 (Evaluate)"] A1["系统端到端性能评测"] --> A2["定位具体错误点与失效根因"] end subgraph DevPhase ["针对性开发阶段 (Develop)"] B1["定向生成高质量专家数据"] B2["优化智能体工具调用 (Tool Calling)"] B3["微调 Embedding 模型优化 RAG 检索"] end A2 -->|"针对性数据策略"| B1 B1 --> B2 B1 --> B3 B2 -->|"性能提升与验证"| A1 B3 -->|"系统闭环迭代"| A1 ``` 这一使命的达成依赖于一套高度闭环的“评估-开发”迭代飞轮（Iterative Approach）： 1. **评估阶段（Evaluate）**：在复杂的智能体工作流中，首先利用评测体系精准诊断系统究竟在哪个环节出错、为何出错； 2. **开发阶段（Develop）**：定位具体瓶颈后，利用针对性的专业数据对特定子模块进行靶向修复与优化——例如注入特定的专家轨迹以提升模型的工具调用能力，或是生成高质量的数据切片用于微调嵌入（Embedding）模型以强化 RAG 系统的检索精度。通过持续的闭环迭代，驱动整套生成式 AI 系统稳步跨越生产落地的门槛。 ### [17:08](https://www.youtube.com/watch?v=l16i2sigGgw&t=1028s) 持续评估与“发现-修复”迭代循环 在演讲的最后，讲者总结了模型落地的核心逻辑：AI 系统的构建绝非一次性的工程，而是一个必须持续滚动的动态闭环。团队需要不断对模型进行评估（Evaluate），在实践中反复执行“发现与修复”（find and fix loop）的迭代流程。通过在循环中持续定位数据与表现层面的缺陷并快速修正，确保模型能够逐步收敛并适应复杂的真实业务场景。 这一整套以数据为中心的方法论，终极目标是让模型真正达到可投入生产使用的成熟状态（ready to use）。只有当模型在实际业务流程中获得极高的采纳率与认可度（acceptance rate）时，企业才能真正跨越 GenAI 规模化落地的瓶颈，切实将技术转化为可见的商业与业务价值。 ```mermaid flowchart LR accTitle: 模型评估与修复闭环 A["模型评估 (Evaluate)"] --> B["定位缺陷 (Find)"] B --> C["数据修正 (Fix)"] C --> D["模型生产就绪 (Ready to Use)"] D --> E["实现业务价值 (Generate Value)"] D -.->|持续滚动迭代| A ``` - **“发现-修复”迭代闭环（Find and Fix Loop）**：通过持续评估机制定位模型与数据的薄弱点，依托敏捷的数据开发流进行针对性修复。 - **业务价值与采纳率（Acceptance Rate）**：衡量模型落地的最终标准不仅是基准跑分，更在于生产环境中的高采纳率与实际创造的业务价值。 ## Notable quotes > [00:04](https://www.youtube.com/watch?v=l16i2sigGgw&t=4s) 模型无法自行扩展，只有数据才能驱动扩展 — Ray Mi > [00:04](https://www.youtube.com/watch?v=l16i2sigGgw&t=4s) 80% 的生成式 AI 项目未能为企业创造实质性的商业价值，且仅有 13% 的 AI 引擎真正深度融入了关键业务流程中。 — Ray Mi > [02:08](https://www.youtube.com/watch?v=l16i2sigGgw&t=128s) 要将通用 AI 推进为能够真正创造业务价值的专用 AI，必须在两个关键维度上取得突破：任务复杂度与输出准确度。 — Ray Mi > [04:47](https://www.youtube.com/watch?v=l16i2sigGgw&t=287s) 上游任何一个微小的误差都会在执行链中逐级放大，产生连锁级联故障。 — Ray Mi > [08:21](https://www.youtube.com/watch?v=l16i2sigGgw&t=501s) 海量且低质的通用互联网数据不仅无法帮助模型在专业垂直场景中实现性能扩展，反而会产生严重的数据污染，损害系统的可靠性。 — Ray Mi > [08:21](https://www.youtube.com/watch?v=l16i2sigGgw&t=501s) 推动模型能力前沿向前演进的唯一动力就是‘专家数据’。 — Ray Mi > [10:28](https://www.youtube.com/watch?v=l16i2sigGgw&t=628s) 如果你恰好是这三家头部电信运营商之一的用户，并在与其账单客服机器人对话，你很可能完全意识不到对方是 AI，因为该模型在应对各种复杂账单问题时已经表现得极其出色。 — Ray Mi > [15:06](https://www.youtube.com/watch?v=l16i2sigGgw&t=906s) 成为‘专业化 AI 的核心数据层’ — Ray Mi > [17:08](https://www.youtube.com/watch?v=l16i2sigGgw&t=1028s) AI 系统的构建绝非一次性的工程，而是一个必须持续滚动的动态闭环。团队需要不断对模型进行评估，在实践中反复执行‘发现与修复’的迭代流程。 — Ray Mi > [17:08](https://www.youtube.com/watch?v=l16i2sigGgw&t=1028s) 衡量模型落地的最终标准不仅是基准跑分，更在于生产环境中的高采纳率与实际创造的业务价值。 — Ray Mi ## Mentioned Ray Mi, Snorkel AI, Ai4, 斯坦福大学以人为本人工智能研究院 (Stanford HAI), ChatGPT, GPT-5, Expert Data-as-a-Service, Hugging Face, Wordle, Fortune 500 (财富 500 强) ## Video description Ray Mi, Dr, Pre-Sales Engineering, Snorkel AI Abstracts As GenAI systems move from research labs to production environments, organizations are discovering a painful truth: scaling models is easy, scaling performance is not. Despite access to powerful foundation models, many deployments stall. The common thread? Insufficient or misaligned data. In this session, we’ll explore why data is the primary bottleneck in GenAI deployment at scale. We’ll introduce a new approach to data development through programmatic, expert-in-the-loop workflows, powered by Snorkel's Expert Data-as-a-Service platform. Whether you're deploying GenAI agents or tuning domain-specific LLMs, the path to scalable performance starts with your data. [Source](https://www.youtube.com/watch?v=l16i2sigGgw)

[0:04] Ray Mi: 模型无法自行扩展，只有数据才能驱动扩展 [0:04] Ray Mi: 80% 的生成式 AI 项目未能为企业创造实质性的商业价值，且仅有 13% 的 AI 引擎真正深度融入了关键业务流程中。 [2:08] Ray Mi: 要将通用 AI 推进为能够真正创造业务价值的专用 AI，必须在两个关键维度上取得突破：任务复杂度与输出准确度。 [4:47] Ray Mi: 上游任何一个微小的误差都会在执行链中逐级放大，产生连锁级联故障。 [8:21] Ray Mi: 海量且低质的通用互联网数据不仅无法帮助模型在专业垂直场景中实现性能扩展，反而会产生严重的数据污染，损害系统的可靠性。 [8:21] Ray Mi: 推动模型能力前沿向前演进的唯一动力就是‘专家数据’。 [10:28] Ray Mi: 如果你恰好是这三家头部电信运营商之一的用户，并在与其账单客服机器人对话，你很可能完全意识不到对方是 AI，因为该模型在应对各种复杂账单问题时已经表现得极其出色。 [15:06] Ray Mi: 成为‘专业化 AI 的核心数据层’ [17:08] Ray Mi: AI 系统的构建绝非一次性的工程，而是一个必须持续滚动的动态闭环。团队需要不断对模型进行评估，在实践中反复执行‘发现与修复’的迭代流程。 [17:08] Ray Mi: 衡量模型落地的最终标准不仅是基准跑分，更在于生产环境中的高采纳率与实际创造的业务价值。

[0:04] Ray Mi: corporate AI adoption has surged from 55% to 78% year-over-year. [0:04] Ray Mi: 80% of generative AI initiatives fail to create tangible business value, and only 13% are deeply integrated into business-critical operations. [0:04] Ray Mi: Scaling models, parameter counts, and raw compute are no longer the bottleneck; rather, high-quality expert data serves as the critical fuel required to drive enterprise value. [2:08] Ray Mi: Transitioning from generic LLMs to specialized AI requires tackling two key dimensions: domain complexity and strict accuracy requirements. [2:08] Ray Mi: In complex, real-world agentic workflows, errors cascade sequentially through tool usage, user clarification steps, and API calls. [8:21] Ray Mi: Massive volumes of low-quality or web-scraped internet data contaminate models rather than helping them scale. [13:03] Ray Mi: Achieves 89% client-defined accuracy targets twice as fast as industry competitors. [15:06] Ray Mi: data layer for specialized AI [17:08] Ray Mi: Rather than treating model development as a one-time handoff, teams must repeatedly evaluate performance, diagnose alignment and data issues, and apply programmatic corrections.

[0:04] So um my name is Ray. I'm the director [0:06] of pre-sales from Snorkel AI. Uh Snorkel [0:09] is a um company that is helping client [0:12] get expert data for their AI machine [0:14] learning initiatives. The client [0:16] including like frontier AI labs as well [0:18] as Fortune 500 companies. That's why I'm [0:20] here to talk about why models don't [0:23] scale but data does. [0:27] My background is from like statistics [0:29] and machine learning. So I like to start [0:31] my uh talk with numbers. Um you might [0:34] already um hear this from all the talks [0:37] you've been through today and yesterday. [0:39] And before we jump into this details, I [0:41] want to do a like quick survey here. [0:44] Raise your hand if you or your [0:46] organization already deploy AI solution [0:48] in production. [0:50] Great. We have a lot of uh early [0:52] adopters. So um according to Stanford [0:55] HAI research um there about 78% of [0:58] organization already adopt AI in their u [1:01] process. So adoption is not really a [1:03] problem now. Um compared to a year ago [1:05] the number is up from 55 to 78%. Um and [1:10] then but the um problem there is 80% of [1:14] those generative AI uh initiative and [1:16] solutions is not really creating [1:17] tangible value for business and that's [1:20] the main problem and also adoption is [1:22] low. So only 13% of those um AI engine [1:25] really deeply integrated with the [1:27] business critical process reason um data [1:31] misalignment that's why we're here to [1:33] talk about why data matters when you are [1:34] scaling up your AI AI solution 48% of [1:38] the uh seuite realize recognize that [1:41] lack of high quality specialized data is [1:44] the problem for them to operationalize [1:46] their generative AI solution. So to wrap [1:49] it up um expert everyone is talking [1:52] about AI is doing AI but the real [1:54] bottleneck is not model instead it's [1:57] data and then throughout the session I [1:59] will tell you more about why data is the [2:01] critical fuel for um generative [2:03] initiatives and how snorkel is helping [2:05] our client to get expert data. [2:08] All right. So before we jump into the [2:11] data, let's see take a look at like what [2:13] um where we are at the moment in the AI [2:15] regime, right? Um at the be well I think [2:18] recently with all the different LM um [2:21] providers are providing different [2:23] language model or multi multimodel LM um [2:27] like genai solution and also with the [2:29] increased amount of compute power the [2:31] bottleneck is not really the choice of [2:33] model the number of parameters or the [2:35] GPU resources. Um at the beginning when [2:39] everyone get exposure to chat GBT we use [2:42] out oftheshelf AI to do prompting and [2:44] get some quick question answer you do [2:46] like simple task like summarization and [2:49] um um simple generation tasks those [2:52] works for proto prototype but it's not [2:54] really working for production grade um [2:56] implementation. So in order to move from [2:59] that off-the-shelf AI to specialized AI [3:02] there are two dimension we need to [3:03] tackle one is the complexity because [3:05] there will be more use case specific [3:07] domain specific um use cases that is [3:10] more much more complex than simply [3:12] prompt ALM and on the other hand a [3:15] accuracy requirement is also important [3:17] for this model to actually generate [3:19] value and also drive adoption. So [3:21] thinking about if you have a chatbot [3:23] that is helping doctor to get [3:24] information for them to better treat the [3:26] patients the accuracy should be very [3:28] reliable and trustworthy for them to use [3:30] it otherwise they will trust their own [3:32] knowledge and experience in the past [3:34] instead of using a chatbot generate um [3:36] powered by AI. So if we push that [3:38] frontier from like the lower left corner [3:41] to the upper right corner we're [3:42] gradually get getting into the agentic [3:45] um agentic regime. So I've been joining [3:49] a few sessions today um in this [3:51] conference but uh raise your hand if you [3:54] didn't heard agentic in any of the [3:56] sessions you had in the past two days. [3:58] All right zero. So so that's kind of a [4:02] pretty hot topic right um from my point [4:04] of view this is this is one look of a [4:07] gentic system right from my personal [4:08] point of view agentic system has three [4:11] major elements which could take [4:13] different shape and form like this [4:14] diagram. Um but from my point of view [4:16] you agent will have access to a user uh [4:19] get requests from the user and agent [4:22] also have access to tools right so what [4:24] kind of how you can solve the problem [4:26] use um based on the user's request and [4:28] also the agent have access to LM and [4:31] that LM will be the interface for the [4:33] agent to send some prompt over to so LM [4:36] can say hey user asked for this you have [4:38] that amount of tools use those tools to [4:40] assemble the data for the user for them [4:42] to get the answer they want so that's [4:44] kind of a more logical view of agentic [4:47] system and the realistically it can look [4:50] as complex like this right so the end [4:52] this end to end process basically [4:54] touching the user the agent is basically [4:56] doing all different orchestration [4:58] between tools user interaction API calls [5:00] LM endpoints in order to have the answer [5:03] get back to the client to to the user [5:06] it's very complex and think about like [5:08] all the errors in the upstream in this [5:10] process will cast all the way to the end [5:12] right even though the lm the like let's [5:15] say GPD5 is very smart make very little [5:19] errors any of the other steps tool [5:21] calling user interaction clarification [5:23] questions will also drive the [5:26] performance of the entire system so that [5:29] could be very complex and how do we kind [5:31] of make sure we have a good benchmark [5:33] about the performance of the agendic [5:35] system as well as improve the perform [5:37] performance so it's getting a production [5:39] grade performance and get ready to um [5:42] get ready to um to implement will be the [5:44] key topic for us to discuss today. All [5:47] right, we have a very complex system, [5:49] right? And now we want to make that [5:51] complex system from like very complex to [5:55] robust. And how do we make that [5:57] robustness happen will also drive [5:59] adoption especially in the highly [6:01] regulated industry. First we hit the [6:04] keyword data. So um data is important [6:07] for um two different foes. blindness [6:10] training including post training um like [6:13] reinforcement learning with human [6:14] feedback um supervised fine-tuning and [6:16] benchmarking. So a golden data set that [6:19] is fitting into the agenda workflow that [6:22] can tell you uh what's the kind of [6:24] rubric to follow and what's the high [6:26] quality answer you you can um get is [6:29] important to kind of capture the edge [6:31] cases and capture the rare events. So [6:33] the data is smart enough to in to um [6:36] instruct the model to think in a very [6:38] unique way that is uh that is uh fitting [6:40] to per purpose for your use case and [6:43] second is evaluation. So I've been [6:45] talking with a lot of data scientists in [6:47] my current job in my previous jobs [6:48] because my me myself was a um data [6:51] scientist when I was working in the [6:52] banking and financial services industry [6:55] they've been building a lot of agentic [6:57] uh system and genai use cases but the [6:59] problem is they couldn't really validate [7:01] whether the model is actually performing [7:03] well because the data scientist team and [7:05] machine learning engineer they have a [7:06] skill sets to assemble the system but [7:08] they don't really have the domain [7:10] knowledge to assess whether the model is [7:11] performing well or not. So um that's [7:14] create a bottleneck and also those [7:16] experts they're not available all the [7:18] time because data scientists might have [7:20] like thousands of initiatives but they [7:22] only have two hours access to one person [7:25] for that bank underwriting like credit [7:27] underwriting use case. So how do you [7:29] kind of amplify the human judgment via a [7:32] more programmatic evaluation framework [7:34] is important and that's how snorkel is [7:36] helping our customer addressing this [7:38] ground your um system by human [7:40] annotation but you don't really need to [7:43] consume huge amount of time from them [7:45] and then amplify that domain knowledge [7:47] through the pro programmatic development [7:49] to make your evaluator much more aligned [7:52] with human so that evaluator can be used [7:54] more programmatically and automatically. [7:58] Last but not least, development, the [8:00] data and evaluation. Those will tackle [8:02] all the smaller modules we see in that [8:04] agentic system. But how do you stitch [8:06] all the end-to-end process together and [8:08] make the final quality um good enough [8:10] for the production deployment uh require [8:13] a fast and scalable development [8:15] framework so that your geni system can [8:17] deploy deployed. [8:21] So let's take a look at why data is so [8:23] important for frontier AI development, [8:25] right? And let's put everything into a [8:28] timeline. And I was surprised that we [8:30] only need three years in this plot. 20 [8:32] before 2023 and then now we we're in [8:35] 2025. Um before CHGBD came to the [8:38] market, what we're doing pre-training [8:40] using unlabelled data basically scan [8:42] through the entire uh corpus in the [8:44] internet and then trying to understand [8:46] how to autocomplete next token. And then [8:49] a post-training era was where like Chad [8:51] GBD start to get into the public view [8:54] and that's basically uh majorly fired by [8:58] uh reinforcement learning with human [8:59] feedback but also kind of serving for [9:01] generic purposes because those [9:03] reinforcement learning is done by a [9:06] labeler that is not trained for spe [9:09] specialized task. Then now we're in the [9:11] era of post- training with expert data [9:14] and that's where we can solve more [9:15] complex andme driven use cases and also [9:19] um enable more um advanced task like [9:22] code agents like deep research which is [9:25] a pretty hot topic in the market right [9:26] now. [9:29] So just to conclude this part large [9:31] volume of internet data and lowquality [9:33] data is actually not helpful for the [9:36] model to scale scale out for different [9:38] specialized use cases especially low [9:40] quality data because that will [9:42] contaminate the the your your u your AI [9:45] system. So expert only expert data can [9:48] move the frontier forward. So your [9:49] system can be specialized and those are [9:52] a few requirements when if you think [9:54] about building um specialized and expert [9:57] data to do post- training or uh [10:00] benchmark evaluation. Those are a few [10:02] requirements. First one pretty [10:04] straightforward. It should be driven by [10:05] experts and that experts should be the [10:08] one that is actually working on this [10:09] domain has training experience and also [10:12] empirical interactions with their [10:15] ultimate counterpart so that the expert [10:18] know all the real world scenarios and [10:20] edge cases and different um and [10:22] different scenarios and second high [10:25] quality as I mentioned that's pretty [10:26] straightforward no more explanation [10:28] diversity actually is one thing that we [10:30] work a lot with clients when they're [10:32] reaching out to us for um data with [10:34] providing their data specs. They already [10:36] care about diversity distribution on [10:39] each categories and how those [10:41] combination of uh categories end up [10:43] being a large variety of cases [10:46] scenarios. For example, when customers [10:48] could be asking for images for a [10:50] specific river and all the images you [10:54] can collect from internet is the daytime [10:56] image. So in order to make that model [10:58] working, you might also need some [10:59] nighttime images in order to make the [11:01] model active um as good as the daytime [11:05] um especially nighttime monitoring for [11:07] like flooding could be more critical for [11:09] daytime monitoring because you you rely [11:11] on machine instead of human to do it. Um [11:13] that's just one example, right? There's [11:15] all other examples that we're are trying [11:17] to help a telco customer distill a large [11:19] language model into small language model [11:22] for their um customer service um [11:24] chatbot. that chatbot is um basically [11:27] answering question for your um billing [11:29] statement. Um and that small language [11:32] model needs a specialization in regards [11:35] to how to answer specific questions from [11:37] the client. When we are doing that, we [11:39] actually identify seven different [11:41] taxonomy. Those are common topics the [11:43] client will ask. And from there we um [11:46] generate um we use human generated data [11:49] plus uh from that data generation and [11:51] evaluation process to make the data [11:54] really diversified and then fit into the [11:56] real world scenario and that help the [11:58] client get the model performance from [12:00] 55% baseline to 95% accuracy and they [12:04] deploy in production. So if you're using [12:06] one of the top three telco services and [12:08] chat with the chatbot about your monthly [12:11] statement information um highly possible [12:14] you'll be chatting a chatbot without [12:16] noticing it noticing it because the [12:17] model is very good at handling all [12:19] different cases. Last but not least [12:21] frontier complexity. [12:24] Um at the beginning if you originally if [12:27] you want to think about like getting [12:28] data for uh model fine-tuning you might [12:30] want to only simply consider question [12:33] answer pairs or multi- question uh [12:34] choices questions but with the [12:36] evolvement of agentic system there more [12:39] requirement for like multi-steps [12:40] multi-interaction multi- um module um [12:44] data that is fitting into the complexity [12:47] of real world um scenario. So those are [12:49] different requirements when you are [12:51] considering develop your data um with [12:54] help like from company like snorkel [12:57] those will be different consideration [12:58] for how you get your model specialized. [13:03] So um snorkel announced the expert data [13:05] as a service offering recently and we've [13:08] been doing this for a while. Uh actually [13:10] last week we celebrate our first year uh [13:12] first anniversary of the first data [13:14] deliverable to one of the top AI um [13:16] frontier lab. Throughout this journey we [13:19] built a big network of experts from a [13:22] thousand different domains as well as um [13:25] and like we have tens of thousands [13:27] experts in those domains that is [13:29] handling different type of questions. Uh [13:31] we also achieved the 89% of accuracy [13:34] goal required by the client two times [13:36] faster than other players in the market. [13:38] um very trusted by the top LM labs as [13:41] well as the Fortune 500 companies. [13:45] Um we will wrap up soon but this is one [13:48] thing I want to share with you um [13:49] because there's this um QR code over [13:51] there. Snorkel is also contributing to [13:53] the open source community. We built so [13:55] far six different benchmark data set. [13:57] Each of them is tackling different [13:59] tasks. Some of those are financial [14:01] reasoning, insurance center writing or [14:03] even like games like Wordle. how LM is [14:06] is performing on those different tasks [14:08] with uh agent like two calling and [14:10] reasoning capability. We've run all the [14:13] LMS against those benchmark data set and [14:15] trying to put them on a leaderboard so [14:16] we know how their model is performing. [14:19] one helping our like broader audience to [14:22] understand like when you are picking up [14:24] LN for reasoning task you might choose [14:26] model A versus when you are trying to um [14:29] kind of make a tool calling or graph [14:32] related um agentic system you might use [14:34] model B but also we tell you the [14:36] performance per dollar so there's a [14:38] trade-off even though the to um like [14:41] dollar cost per token is reducing [14:43] significantly when you implement your AI [14:45] solution at scale cost is still a [14:47] consideration to also tell you in order [14:49] to achieve those performance what's the [14:51] token cost uh for each of those LM so [14:54] that's openly available in our website [14:55] and the data is also available on [14:57] hugging face if you want to use data to [14:59] benchmark your own your own um like [15:01] in-house fine-tune LM for example [15:06] and um just a quick differentiator about [15:09] why snorkel is unique in this place [15:11] first it's very complex so the first use [15:13] case I got involved it actually curate [15:16] very advanced questions for like [15:17] mathematics and statistics six question [15:19] answer pairs with a like multi-steps [15:21] reasoning process and the goal is to [15:24] trick LM 20% of like I have the pass [15:27] rate between zero to 20%. So I actually [15:30] serving as an expert in that use case. [15:33] Back then we are we don't have that big [15:35] network like in the first month. So I [15:36] was working on creating very difficult [15:38] mathematics reasoning uh steps as a as a [15:41] human annotator and human um expert to [15:44] trig the LMS and second quality like we [15:47] have a QA process. We use experts to [15:49] challenge the other expert to make sure [15:51] um the quality is good. Also we build [15:54] evaluators programmatically based on the [15:57] like empirical experience we have. So [15:59] we're baking that in place. So we use [16:01] human as well as machine to challenge [16:02] the quality before we deliver the data [16:05] to the customer. Speed is also a key [16:07] because kind of the faster you can [16:09] deploy a model in production the better [16:11] uh the the more value you can the [16:13] earlier you can generate value for the [16:15] production environment. Yeah. So this is [16:17] our mission. Basically it's very simple [16:18] very straightforward. I don't want to [16:20] put a lot of words there. Um we are [16:22] trying to become a data layer for a [16:24] specialized AI and why all the wise is [16:27] already mentioned just now and how we do [16:30] it is through this iterative approach. [16:32] So there are two major step in all the [16:34] LM use case and agentic workflow. First [16:36] you need to evaluate your system to [16:38] understand where the problem is and why [16:42] the problem occurred and then you once [16:45] you find the error you go fix error and [16:47] then the development part is where you [16:49] can actually leveraging those [16:51] specialized data to fix different parts [16:53] in that agentic system could be instruct [16:56] LM to be better at tool calling could be [16:58] a uh slice of curated data for your [17:00] embedding model fine-tuning to make the [17:02] rag system better so all those will [17:04] eventually get you into to a different [17:06] stage and now the loop the the firewall [17:08] is keep rolling that you can evaluate [17:11] again do that find and fix loop uh [17:14] iteratively eventually the goal is to [17:16] have your model ready to use get a good [17:19] acceptance rate and then generate value [17:21] for the This