Harvey Uses Synthetic Data and Open Weights to Challenge Frontier Labs

Harvey利用合成数据与开放权重挑战前沿实验室 应用层初创企业可以通过将开放权重模型与领域指导的合成数据集及多供应商服务基础设施相结合，以有限预算媲美顶尖人工智能研究。 Gabe Pereyra 阐述了应用层企业如何利用开放权重后训练、领域专家指导的合成数据以及多供应商基础设施，与资金雄厚的前沿实验室展开竞争。Harvey 并没有在单纯的机器学习团队规模上正面硬拼，而是构建了贴合实际的法律评测基准（如 LegalAgentBench），并与专业的新兴实验室合作训练诸如“Associate 1”等专用模型。Pereyra 认为，将行业专业知识与智能路由及开源模型相结合能够拉平竞争格局，从而改变人工智能开发的经济模式。 Harvey 联合创始人兼总裁 Gabe Pereyra 指出，应用层初创公司无须与前沿实验室进行算力或人才军备竞赛，而应通过利用评测基准、开源模型后训练及多云基础设施构建垂直领域的“前沿智能”。 (0:00) 针对法律客户数据高度机密、不可直接用于训练的限制，Harvey 开创了由领域专家引导的合成数据生成方案，从预设缺陷的评测标准反向生成高达 8000 万 token 的虚拟尽调数据室，并通过超过 1000 项单元测试检验模型能力。 (3:16) 得益于开源权重模型能力的提升，Harvey 携手 Fireworks.ai、Baseten 等多家细分领域的“新实验室”进行并行后训练实验，成功打造出结合开源模型与商业 API 的垂直领域模型“Associate 1”。 (5:57) 为了向 60 个国家和地区的企业级客户交付服务，Harvey 部署了具备自动容灾回退的多端模型矩阵，并建立覆盖通用指标、特定产品路径及领域专家盲测的多层级上线评估流水线以确保服务可用性。 (8:49)

Harvey co-founder Gabe Pereyra argues that application-layer AI startups can compete with multi-billion-dollar frontier labs by combining domain-expert synthetic data, open-weight post-training partnerships, and hyper-vertical workflow infrastructure. Rubric-First Synthetic Data and Public Benchmarks (3:16) Enterprise legal data is strictly privileged and cannot be used to train general models. To bypass this constraint, Harvey designed a synthetic data pipeline guided by in-house legal experts. Rather than drafting thousands of contracts manually, legal specialists design the evaluation rubric first, planting deliberate errors, missing clauses, and conflicting agreements across simulated transactions. Generative models then draft the documentation around those flaws, scaled with partners such as Mercor and Snorkel and tested against more than 1,000 unit tests via LLM-as-a-judge patterns. Post-Training Open Weights via Neo-Lab Partnerships (5:57) While early fine-tuning quickly became obsolete against fast-improving base models, Pereyra states that modern open-weight architectures like Kimi k1.5, GLM-4, and NeMo-Megatron now make post-training highly viable. Instead of building extensive in-house research teams or relying on a single vendor, Harvey distributes specialized research tasks across external 'neo-labs,' collaborating with Fireworks.ai, Baseten, Ngram, Trajectory, and Applied Compute on KV-cache compaction, search, and domain adaptation. Production Serving and Multi-Tiered Evaluation (8:49) Serving enterprise clients across 60 countries requires a multi-provider infrastructure that balances uptime, latency, cost, and strict regional privacy mandates. Harvey routes between closed frontier APIs (OpenAI and Anthropic, backed by Bedrock and Azure) and open post-trained models hosted across Fireworks.ai and Baseten. Lower-cost open models handle routine sub-tasks, such as citation generation with GLM, reducing dependence on expensive frontier endpoints. Talent Strategy and Unresolved Technical Hurdles (14:53) Pereyra noted that trying to match the nine-figure compensation packages offered by frontier labs for pure ML researchers was an early misstep. Harvey shifted to pairing strong software engineers with domain specialists who understand legal logic. However, major technical bottlenecks remain unsolved, including managing context windows that reach up to 80 million tokens in corporate data rooms, bridging the distribution gap between synthetic benchmarks and daily user behavior, and enabling continual learning without compromising multi-tenant data privacy. Defensibility Through Enterprise Workflow Orchestration (26:56) In distinguishing Harvey from horizontal productivity tools, Pereyra explains that horizontal platforms focus primarily on individual efficiency. In contrast, legal enterprises and corporate law departments require coordination across multi-month projects, dozens of attorneys, outside counsel, and complex billing constraints. Harvey focuses on vertical orchestration that manages human-and-agent workflows and resource allocation, which Pereyra contends provides a moat against generic tools. Pereyra analogizes Harvey's approach to the Moneyball thesis: application startups do not need the largest compute budgets or research staffs to build competitive domain intelligence. Success hinges on rigorous domain-grounded evaluation and orchestration, though maintaining privacy while learning continually from live enterprise usage remains an ongoing challenge.

Harvey 联合创始人兼总裁 Gabe Pereyra 阐述了应用层企业在缺乏前沿实验室庞大算力与资金的条件下，如何通过领域专家引导的合成数据、开源核心评测基准、联合新型基础设施实验室进行开源权重后训练，以及构建多层生产验证体系，在特定垂类中以精简预算打造前沿智能系统。 规则前置与专家驱动的合成数据管道 (3:16) 企业客户与律所的法律数据属于受特权保护的高敏感隐私，无法直接用于训练基础模型。针对这一困境，Harvey 探索出由领域专家主导的合成数据生成模式。据 Pereyra 介绍，公司律师 Julio 提出了一种“评测规则前置”的逆向构建技术：专家不直接手写上万份合同，而是先设定审计规则，在模拟交易数据中植入故意遗漏的条款、冲突协议与逻辑漏洞，随后调用 AI 模型围绕这些问题自动生成完整的虚拟数据室。 为了保证数据体量与真实度，Harvey 与 Mercor 及 Snorkel 等外部伙伴合作扩充并润色数据集，并与 LangChain 合作优化 LLM-as-a-judge 评估机制，部署了 1,000 多个单元测试来检验模型识别隐藏法律缺陷的能力。基于这套流程，Harvey 构建了包含高达 8,000 万 token 虚拟数据室的 Diligence 数据集，用于检验复杂长上下文推理能力，以此在严格合规前提下解决了客户隐私与高质量训练数据匮乏的双重矛盾。 开源评测基准与行业生态飞轮 (5:08) 高质量评测是模型训练与上线的前提。过去一年中，Harvey 先后创建并开源了 LegalAgentBench（涵盖律所初级律师复杂任务分类体系）、Contracting 数据集（用于多轮合同谈判训练）以及 Diligence 数据集。Pereyra 坦言开源基准在当时曾引发争议，因为主流前沿实验室会利用公开基准训练模型，但他基于在 Google Brain 和 DeepMind 的研究经验强调，如同 ImageNet、CIFAR 与 MNIST，数据集只有被社区广泛采用才能持续优化。 开源策略不仅为 Harvey 吸引了社区的漏洞修复与代码拉取请求（PR），还推动各大前沿实验室主动接入这些标准来测试其最新模型。Pereyra 认为，行业应用企业的核心壁垒不在于静态的公开基准，而在于基于客户私有数据微调的工作流与编排架构。公开评测基准既为客户评估不同模型提供了客观透明的标准，也使外部研究团队能在统一基准上测试新型优化技术，进而将 Harvey 转变为行业创新的生态枢纽。 联合新型实验室的开源权重后训练 (5:57) 在早期阶段，基础模型的快速更迭容易让应用公司的微调成果迅速过时，后训练性价比极低。然而，随着 Kimi k1.5、GLM-4、NeMo-Megatron 及 Inkling 等高性能开源权重模型的出现，针对特定细分任务进行专业化后训练已具备极高可行性，甚至能在窄领域匹配前沿大模型的表现。 受 Cursor 的 Composer 工具启发，Harvey 开发了名为 Associate 1 的垂直模型。Harvey 并没有单独依赖单一服务商或盲目自建基础设施团队，而是选择与多家专业新型基础设施实验室（Neo-Labs）并行展开研究：与 Fireworks.ai 探索训练 GLM 与 Claude 3.5 Sonnet 等导师模型协同交互，与 Baseten 合作优化 KV-cache 压缩与推理服务，与 Ngram 联合研发企业知识库检索，以及与 Trajectory 和 Applied Compute 分别针对 NeMo-Megatron 后训练与 Vault 产品进行特定优化。借助第三方平台与 Tinker 类工具，企业能够以低成本同时推进多项平行研究，避免了过早膨胀内部研发团队。 多层生产服务矩阵与复合人才策略 (8:49) 在跨越 60 个国家交付企业级服务时，Harvey 建立了具备自动故障降级功能的多供应商矩阵，以确保服务等级协议（SLA）与区域合规要求。其闭源前沿模型以 OpenAI 和 Anthropic 为主终结点，以 AWS Bedrock 和 Azure 作为降级备选；开源后训练模型则通过 Fireworks.ai 与 Baseten 在指定区域独立部署。在模型上线前，必须依次通过四级严密验证：基础通用自动化评估、与现有生产模型的通用盲测对比、针对具体产品界面要求的用户关键旅程（CUJ）自动化测试，以及由法律领域专家执行的真实工作流人工检验。 针对行业人才竞争，Pereyra 承认早期曾试图单纯与开出上亿美元薪酬包的前沿实验室争夺顶级机器学习研究员，实践证明这是策略失误。随后 Harvey 调整方向，将资深软件工程人才与经过系统 AI 直觉训练的法律领域专家进行深度绑定。随着第三方训练和推理工具的标准化普及，应用层公司不再需要庞大的底层基建工程队伍，这使得团队能够将核心资源聚焦于业务理解与系统工程融合之上。 Pereyra 将这一策略总结为应用层初创企业的“魔球理论”（Moneyball）：依靠非传统团队结构与有限预算打破重资本垄断。但他同时指出了垂类 AI 尚未解决的三大技术瓶颈：合成数据与真实客户日常操作之间的数据分布偏差（数据漂移）、超过 8,000 万 token 超长上下文的稳定推理极限，以及在保护多租户数据隐私前提下实现客户交互驱动的模型持续学习。与侧重个体效率的横向平台不同，Harvey 试图通过深度绑定复杂企业组织级项目协同来构筑长期防御壁垒。

## Background 该视频由红杉资本（Sequoia Capital）于 2026 年 8 月 18 日在 YouTube 平台发布，记录了在红杉资本举办的“Own Your Intelligence”活动中，法律 AI 独角兽 Harvey 联合创始人兼总裁 Gabe Pereyra 的主题分享。视频简介指出，演讲聚焦于应用层初创公司如何在缺乏巨额资金、顶尖算力与海量数据的劣势下，通过善用前沿 AI 生态，在有限预算内搭建自主研究实验室并跑通后训练全流程。 ## TL;DR Gabe Pereyra 深入拆解了 Harvey 打造垂直研究实验室的实操手册，核心支柱包括构建垂直基准评测集（如 LegalAgentBench 与 Diligence Dataset）、由领域专家引导生成高质量合成数据以绕开敏感数据红线，以及与多家新型后训练实验室（Neo Labs）协同打磨法律基石模型“Associate 1”。在工程落地上，Harvey 建立了覆盖 60 多个国家的高可用模型服务矩阵、严格的上线前评估门禁与多区域容灾机制，并通过简单开源替换与模型路由推动渐进式降本增效。面对未来竞争，Harvey 坚信应用层的终极壁垒不在于单点文本生成的个人生产力，而在于深入法律长周期、多协作复杂业务场景的“组织级生产力”编排系统。 ## Atmosphere & style 整场演讲务实笃定且极具感染力，Gabe Pereyra 凭借深厚的研究员背景与独角兽创业历程，将严肃晦涩的后训练工程讲得清晰透彻、层层递进。他用电影《点球成金》中以弱胜强的名台词打破行业对基础大模型实验室的盲目崇拜，展现出应用层创业者不甘为简单“套壳”的破局野心与坚韧底气。问答环节交流紧凑坦诚，面对数据隐私、人才争夺及系统瓶颈等刁钻提问毫无回避，带给听众一种如临一线研发作战室般的实战沉浸感与战略启发。 ## Bulletin - [02:04](https://www.youtube.com/watch?v=MGouk8W51v0&t=124s) Harvey 公开了包含 8000 万 Token 上下文的数据室（Data Room）尽职调查数据集，并配套 1000 多个基于 LLM-as-a-judge 的单元测试，成为当前规模最大的强化学习法律评测环境之一。 - [14:53](https://www.youtube.com/watch?v=MGouk8W51v0&t=893s) 面对顶尖律所客户数据绝对保密的红线，Harvey 采用“评分标准逆向倒推法”，由内部懂大模型的律师指导生成逼真的虚拟数据室，成功规避了直接使用客户敏感数据训练模型的合规风险。 - [05:54](https://www.youtube.com/watch?v=MGouk8W51v0&t=354s) 开源模型与开放权重生态（如 GLM、Kimi 等）的成熟使得后训练性价比大幅提升；Harvey 联合 Fireworks、Baseten 等多家 Neo Labs 分工后训练，打造专属于法律场景的基石模型“Associate 1”。 - [26:57](https://www.youtube.com/watch?v=MGouk8W51v0&t=1617s) Harvey 将长期防御壁垒从“个人生产力”（如辅助写邮件）转向“组织级生产力”，聚焦于跨越数月、涉及数十人与多 Agent 协同的重度企业级工作流，以此抵御横向通用 AI 产品的竞争。 ## Key takeaways - [00:00](https://www.youtube.com/watch?v=MGouk8W51v0&t=0s) 应用层公司在资源劣势下突围的核心在于“杠杆化前沿生态”：聚焦三大支柱——专业基准与数据、联合 Neo Labs 后训练、生产级推理与评测基础设施。 - [02:04](https://www.youtube.com/watch?v=MGouk8W51v0&t=124s) 基准评测是模型训练与上线的前置条件：若没有高质量 Benchmark，后续的模型训练与生产 Serving 就无从谈起。 - [02:04](https://www.youtube.com/watch?v=MGouk8W51v0&t=124s) Harvey 发布了三大法律基准：涵盖律所初级律师任务分类的 LegalAgentBench、合同谈判博弈数据集，以及长达 80M Token 的尽调强化学习环境。 - [03:36](https://www.youtube.com/watch?v=MGouk8W51v0&t=216s) 在律师保密特权限制下，由领域专家引导生成高质量合成数据，并结合 Mercor 等平台规模化放大，成为解决数据枯竭的有效路径。 - [03:36](https://www.youtube.com/watch?v=MGouk8W51v0&t=216s) 开源基准数据集能借助社区力量疯狂找 Bug、倒逼前沿大模型厂商在垂直榜单上优化，同时充当筛选外部合作方案的天然过滤器。 - [05:54](https://www.youtube.com/watch?v=MGouk8W51v0&t=354s) 随着开源基座质的跃迁，垂直后训练迎来窗口期；Harvey 同时与多家 Neo Labs 在不同算法路线上并行合作，分工探索蒸馏、检索增强与架构调优。 - [08:49](https://www.youtube.com/watch?v=MGouk8W51v0&t=529s) 针对全球 60 多个国家的严苛合规与高可用要求，必须建立包含多 Provider 冗余热备与毫秒级 Fallback 的多区域模型服务矩阵。 - [10:47](https://www.youtube.com/watch?v=MGouk8W51v0&t=647s) 模型推向生产前需通过自动化基准（定界）与人工双盲对比（Side-by-Side）双轨评估，并结合延迟、成本与各业务界面的关键用户旅程综合决策。 - [11:56](https://www.youtube.com/watch?v=MGouk8W51v0&t=716s) 务实的后训练飞轮应从“简单开源替换”起步：先在引文生成等轻量环节替换开源模型，建立混合部署工程肌肉记忆，再逐步演进至智能路由与全量微调。 - [14:53](https://www.youtube.com/watch?v=MGouk8W51v0&t=893s) 利用“从评分标准逆向倒推”的数据构造法，在虚拟数据室中预埋合同冲突与漏洞，可彻底打破“无数据无法证明能力，无能力换不来数据”的死结。 - [17:16](https://www.youtube.com/watch?v=MGouk8W51v0&t=1036s) 应用层公司无需在薪酬上与头部实验室硬拼全栈天才；随着训练与推理工程 API 的解耦成熟，高性价比吸纳垂直研究人才成为可能。 - [18:46](https://www.youtube.com/watch?v=MGouk8W51v0&t=1126s) 评测无需刻意对大模型出偏门难题，真实法律复杂场景（如涉及万份合同的并购尽调）的原貌本身就足以让前沿大模型触壁碰颈。 - [20:25](https://www.youtube.com/watch?v=MGouk8W51v0&t=1225s) 生产链路设立分层门禁（Gates）：通过自动化测试阻断劣质模型，通过灰度与 OOD 测试暴露通用能力崩溃与基准过拟合。 - [24:51](https://www.youtube.com/watch?v=MGouk8W51v0&t=1491s) 当前法律 AI 面临三大前沿瓶颈：未来向合成数据与多变生产请求的分布差距、80M Token 超长上下文的注意力失效，以及多租户机密隔离下的持续学习。 - [26:57](https://www.youtube.com/watch?v=MGouk8W51v0&t=1617s) 企业级垂直软件对抗通用平台侵蚀的终极法则：走向“极致深度垂直”，从辅助单兵作业升级为统揽多方协同、计费核算与项目编排的组织级平台。 ## Detailed walkthrough ### [00:00](https://www.youtube.com/watch?v=MGouk8W51v0&t=0s) 开场：应用层公司如何低成本打造自己的研究实验室 红杉资本（Sequoia Capital）在 “Own Your Intelligence” 活动中邀请到了 Harvey 的联合创始人兼总裁 Gabe Pereyra。主持人回顾道，Gabe 曾是 Google DeepMind 与 Meta 的研究科学家，后来他向大学室友展示了 GPT-3 的惊人能力，两人由此联合创立了如今法律科技领域的独角兽 Harvey。当前许多应用层公司都在思考同一个命题：如何开始进行自主研究、后训练（post-training）属于自己的模型并成立专属的 AI 实验室。Harvey 在这方面树立了标杆，此次分享的核心主题便是如何“在有限预算内打造一家研究实验室（building a research lab on a budget）”。 面对资金、顶尖人才、算力基础设施和海量数据都远超普通团队的前沿大模型实验室（Frontier Labs），应用层公司常常处于极其不利的竞技场中。Gabe 坦言这就像一场不公平的游戏：“世界上有富有的队伍，有贫穷的队伍，然后就是处在应用层的我们。”四年前 Harvey 刚成立时，周边的配套 AI 生态几乎不存在或刚刚起步，创业团队只能要么全靠自研，要么将精力集中在产品开发和 GTM（Go-To-Market）销售体系构建上。然而在当今环境下，只要善于借助外部的“前沿生态系统（Frontier Ecosystem）”，应用层公司完全有能力与前沿大厂同台竞技，打造出属于自己的前沿级智能。Gabe 总结的战略主要分为三大核心支柱： 1. 构建基准评测集（Benchmarks）与高质量训练数据； 2. 联合新兴的新型实验室（Neo Labs）开展后训练； 3. 构建可在生产环境中高可用、低延迟运行的模型推理服务（Serving）与评测体系。 --- ### [02:04](https://www.youtube.com/watch?v=MGouk8W51v0&t=124s) 奠定根基：构建法律领域基准评测集与环境 Gabe 指出，一切模型研发的第一步必须是基准评测集：“如果你没有一个好的 Benchmark，你就无法训练模型；而如果你无法训练模型，你根本无需操心后续的线上服务。”今年 Harvey 集中发布了三套针对法律业务垂直场景的数据集： - **LegalAgentBench**：系统梳理了大型律师事务所（BigLaw）初级律师（Associates）日常所承担的任务分类体系，涵盖多个专业业务领域（Practice Areas），包括极其复杂的基金设立法律文件起草（fund formation documents）、判例法检索研究（case law research）等高难度任务。 - **Contracting Dataset（合同数据集）**：用于训练智能体学习企业法务部门（In-house Legal Department）在商业交易中实际进行的合同谈判与修改博弈。 - **Diligence Dataset（尽职调查数据集）**：这是 Harvey 最新且最引以为傲的工作。Gabe 认为它是目前公开发布的规模最大的强化学习（RL）环境之一，其中最大的数据室（Data Room）上下文长达 8000 万 token（80M tokens），专门用于研究长文本（long context）与极度复杂的端到端法律推理任务。 为了将海量尽调数据转化为高效可运行的 RL 环境，Harvey 与 LangChain 合作进行了深度优化。在尽职调查数据集中，Harvey 部署了超过 1,000 个单元测试（Unit Tests），利用“LLM 作为裁判（LLM as a judge）”对模型输出结果进行打分。如果在大模型上频繁进行 RL rollout 探索，推理与判分成本将极速飙升，因此构建轻量、高效的评测执行环境是落地 RL 的关键前提。 ```mermaid flowchart TD accTitle: Harvey法律基准评测体系 accDescr: 展示Harvey发布的三大法律基准数据集及其对应的能力验证维度 subgraph Datasets["Harvey 核心评测与训练基准"] A["LegalAgentBench<br/>律所协作者任务分类体系"] B["Contracting Dataset<br/>企业法务合同谈判博弈"] C["Diligence Dataset<br/>80M Tokens 尽职调查长文本"] end subgraph Evaluation["评测与训练转化"] D["复杂文本起草与判例检索验证"] E["多轮谈判策略评估"] F["1000+ 单元测试<br/>LLM-as-a-judge 自动化打分"] end A --> D B --> E C --> F ``` --- ### [03:36](https://www.youtube.com/watch?v=MGouk8W51v0&t=216s) 领域专家指引的合成数据生成与开源策略 在数据收集上，Harvey 面临着比一般应用层公司更加严苛的合规壁垒。Harvey 的客户群均为全球顶尖律师事务所和大型企业，客户的法律文档拥有绝对的保密特权（Attorney-Client Privilege），具有极度敏感性。Harvey 既不能将这些数据输入通用基础模型，更不能将其直接拿来训练自己的模型。为了破局，Harvey 找到了今年成效显著的解法——**由领域专家指导的合成数据生成（Domain experts guiding synthetic data generation）**。 Gabe 引用了 Reka 联合创始人 Brendan 的比喻：正如现代工程师不再逐行手写代码，而是通过“结对/氛围编程（vibe coding）”去引导和调优代码模型一样，法律专家也在采取同样的模式。Gabe 透露，他的亲弟弟就是 Harvey 的一名在职律师，他精通如何使用前沿代码模型，并指导培训了公司内的一整批执业律师。这些懂法律业务的律师亲自指导模型，生成极其逼真、具备实战价值的合成数据，供模型训练和产品评测使用。在基础合成数据建立后，Harvey 进一步联合 Mercor 和 Snorkel 等平台将这一流程工业化放大，制造出规模庞大、标注精准的后训练数据。 针对把这些耗费心血的数据集予以开源（Open Sourcing）的决策，Gabe 承认这在当时内部有很大争议，但他坚定认为：“除非有大量的人在上面训练模型，否则你很难确信自己的数据集究竟好不好。”这一理念承袭自他在 Google Brain 和 DeepMind 从事研究的经历——历史上最好的数据集（如 ImageNet、CIFAR、MNIST）全都是开放的。开源后，社区会疯狂找 Bug、提交大量 Pull Request 和修改建议；不仅各家前沿实验室在发布新模型时开始主动在 Harvey 的 Benchmark 上跑分，甚至还引来了 Elon Musk 在社交平台上的转发背书。 --- ### [05:54](https://www.youtube.com/watch?v=MGouk8W51v0&t=354s) 联合 Neo Labs 攻克后训练：走向 “Associate 1” 在基准评测就位后，下一步是模型的后训练。过去几年，后训练对于垂直应用层公司往往“不划算”——基础预训练模型（Pre-trained base models）迭代极快，开发者费尽心思后训练挤出的性能优势，很快就会被下一代预训练基座完全抹平。但当前开源与开放权重（Open-weight）生态发生了质的跃迁，诸如 Kimi k1.5（Kimmy 3）、GLM-4 / 5.2、NeMo-Megatron 等模型的出现，使得应用团队可以在强力开源基座之上，通过定向后训练达成前沿水准（Frontier Intelligence）的领域智能。虽然这不代表通用 AGI，但在特定专业法律任务上已足以比肩顶尖闭源模型。 Harvey 启动后训练的策略不是单打独斗，而是广泛与**新型后训练实验室（Neo Labs）**展开生态合作。这些 Neo Labs 拥有成熟的基础设施和专属训练配方（recipes），能快速帮应用团队诊断训练集的好坏；如果与他们合作无法带来指标提升，往往意味着底层数据集本身存在缺陷。Harvey 没有押注单一供应商，而是并行为不同课题寻找最佳搭档： - **Fireworks.ai**：探索将 GLM 模型与更强推理模型（如 Claude 3.5 Opus / 4.8）配合，将后者作为“顾问模型（Advisor Model）”进行蒸馏与微调。 - **Baseten**：合作进行知识库压缩（KB compaction）相关的后训练。 - **Ngram**：攻关企业级搜索（Enterprise Search）与律所私域知识检索融合。 - **Trajectory**：合作进行 NeMo-Megatron 系列架构的定制训练。 - **Applied Compute**：支持 Harvey 核心产品 Vault 的专属计算与微调。 面对“为什么同时与多家 Neo Labs 合作”的疑问，Gabe 解释称，Harvey 的研究项目数量远超单一团队甚至单家 Neo Lab 的承载能力；每家实验室在算法路线上各有取舍，探索的开源模型也不尽相同。与多家合作让 Harvey 的学习曲线极其陡峭。与此同时，随着 Fireworks、Baseten 等基础设施以及 Tinker 等微调 API 的普及，Harvey 内部团队也正在独立进行越来越多的自主后训练。 受到编程工具 Cursor 打造其自研底层模型 Composer 的启发，Harvey 的最终目标是将内部专家合成数据、Mercor 规模化标注以及与各家 Neo Labs 沉淀的技术全部打包，打造出专属于法律从业者的基石模型——**“Associate 1”**，并将其与顶尖闭源模型并排部署在生产中。 --- ### [08:49](https://www.youtube.com/watch?v=MGouk8W51v0&t=529s) 生产级模型推理矩阵与上线前评估决策 后训练出模型仅仅是半程胜利，将其平稳接入生产环境（Serving in Production）才是真正的硬仗。很多行业人士仍误以为应用层公司仅仅是写几个 Prompt、调用单一的云端模型 API 做一个聊天机器人界面，但事实远比这复杂。 Harvey 目前已在全球超过 60 个国家和地区开展业务，覆盖极为多元的产品界面与工作流，各家跨国律所和企业客户对底层模型的使用策略（甚至数据驻留地域）有着严苛且互不相同的偏好限制。为了在保证合规与满足 SLA（服务级别协议）的前提下稳定运行，Harvey 搭建了一套立体的“模型服务矩阵（Model Serving Matrix）”： - **多模型族与多实例冗余**：针对主流闭源模型家族（如 OpenAI、Anthropic 等），在不同可用区均需配置热备和备用提供商（Fallbacks），一旦某一基础模型提供商发生宕机或限流，系统必须能毫秒级故障转移。 - **开源模型的原生整合**：通过与 Fireworks.ai 等推理服务平台结合，将自主后训练的开源模型无缝插拔进这套生产调用矩阵中。 ```mermaid flowchart LR accTitle: Harvey生产级模型准入与评估流程 accDescr: 展示新模型在进入生产环境前的双轨评测体系，包含自动化基准与人工盲测 subgraph Input["待选模型流"] M1["开源基座 / 后训练模型"] M2["前沿闭源商业模型"] end subgraph PreProd["上线前评估 (Pre-production Evals)"] direction TB E1["自动化评测: LegalAgentBench<br/>确定能力边界与法律分项得分"] E2["人工通用盲测 (Side-by-Side)<br/>双模型对比测试"] end subgraph Decision["准入决策"] Pass{"综合质量与SLA达标?"} Prod["接入多区域高可用 Serving 矩阵<br/>支持跨 Provider Fallback"] end Input --> PreProd E1 --> Pass E2 --> Pass Pass -- 是 --> Prod ``` 面对层出不穷的新模型（无论是闭源更新、开源上新还是 Harvey 自研微调版本），Harvey 建立了一套严谨的**准入评估决策管线（Pre-production Evals）**。在上线前，新模型必须经过两道关卡： 1. **自动化评测（Automated Benchmark）**：将模型放入 LegalAgentBench 等专属基准中快速扫描，瞬间得出该模型是否达到前沿水平、综合推理能力多高、在法律具体各垂直细分（如并购、诉讼、税务）中有何长短板； 2. **通用人工盲测（Human Testing / Side-by-Side）**：组织专业人员进行严格的双模型匿名对照输出对比（Side-by-Side），用定性分析验证自动化评测未暴露的语感、排版与推导细微差异，确保模型有足够把握才获批推向生产集群。 ### [10:47](https://www.youtube.com/watch?v=MGouk8W51v0&t=647s) 严苛的模型上线准则与生产环境监控 在决定是否将一个模型推向生产环境时，Harvey 建立了一套多维度的评估闭环。Gabe 强调，仅仅模型通用能力出色是远远不够的，它很可能无法契合某个具体的产品界面。为此，Harvey 针对每一个产品表面（Product Surface）都梳理了关键用户旅程（Critical User Journeys），并配套了严格的自动化产品测试，随后再进行人工产品测试。工程团队将这些表现与成本、延迟（Latency）以及区域可用性（Region Availability）等启发式规则综合权衡，以此作为是否上线的终极判断依据。 这一整套评估体系对后训练模型（Post-trained Models）与未后训练的基座模型完全一视同仁。Gabe 特别提醒所有开发者，这套评测与服务基础设施是可以复用的，而且必须在团队着手考虑做模型后训练之前，就已经彻底搭建就绪。 ```mermaid flowchart TD accTitle: 模型上线与生产监控工作流 accDescr: 展示 Harvey 如何从离线测试、多维决策到上线后的线上监控来评估模型表现。 subgraph Eval ["上线前评估"] A["关键用户旅程测试"] --> C{"上线决策标准"} B["自动化与人工测试"] --> C D["成本 / 延迟 / 区域限制"] --> C end subgraph Prod ["生产环境监控"] C -->|"达标放行"| E["A/B 分流测试"] E --> F["用户参与度监控"] E --> G["Token 效率与可用性"] E --> H["客户反馈与客诉邮件"] end ``` 模型一旦成功部署到生产环境，同样会进入严密的监测周期： - **灰度与 A/B 测试**：在推出重大版本或变更时，通过严格的对照试验评估实际表现。 - **业务参与度指标**：密切追踪用户参与度，验证模型是否达到了预期的业务效果。 - **系统运行表现**：全天候盯紧系统正常运行时间（Uptime）与 Token 生成效率。 - **最直接的现实反馈**：Gabe 风趣地将其总结为“产品反馈，其实就是愤怒客户发来的邮件”（"angry customer emails"）。 只有当这一整套监控信号确认一切运行平稳时，模型才算真正站稳脚跟。这也再次印证了核心逻辑：在考虑模型推理部署之前，基础设施必须先行。 --- ### [11:56](https://www.youtube.com/watch?v=MGouk8W51v0&t=716s) 渐进式开源替换与后训练飞轮 在正式开启复杂的后训练之前，Harvey 推荐采用一种被称为“简单开源替换”（Simple Open-Source Switches）的务实策略。团队应首先审视产品全流程中所有调用模型的环节，排查哪些细分功能可以被开源模型直接替换。例如在 Harvey 的产品中，部分负责生成引文（Citations）的模块并不需要最庞大、最昂贵的前沿大模型；将诸如 GLM 等轻量模型无缝接入，既能大幅缩减成本，又能保障甚至提升性能。这种“即插即用”的替换能够迅速帮助团队建立起混合部署、同时运维开源模型与闭源模型的工程肌肉记忆。 对于无法直接进行全局替换的复杂场景，进阶方案则是“模型路由”（Model Routing）。系统能够根据用户的特定查询请求与任务类型，智能分流并将特定流量路由至开源模型，从而在成本与效果之间取得最佳平衡。 当开源部署与路由机制运转成熟后，启动后训练飞轮（Post-Training Flywheel）所需的前提条件便已全部齐备。将模型投入生产、收集真实交互反馈，飞轮即可加速转动。Gabe 在此做出了极其严格的合规界定：所谓的“收集反馈”**绝对不代表拿客户的敏感数据进行模型训练**；Harvey 收集的是用户交互的质量信号与人工测试的反馈模式，借此指导团队构建下一代高质量合成数据集，从而持续反哺并迭代这些模型。 --- ### [13:55](https://www.youtube.com/watch?v=MGouk8W51v0&t=835s) 《点球成金》时刻：颠覆行业规则的预算之战 在 Gabe 看来，未来每一家公司都将不得不转型成为 AI 公司，并探索出属于自己的大模型落地方法论。然而眼下，绝大多数人依然选择押注大模型实验室，看衰应用层创业公司与垂直生态的破局可能。 为了表达应用层突围的决心，Gabe 引用了经典电影《点球成金》（Moneyball）中的震撼片段。在电影中，奥克兰运动家队即便拿下了令人瞠目结舌的 20 连胜，总经理 Billy Beane 却依然保持清醒与克制：如果无法赢下最后的总冠军，世人根本不会铭记这支队伍所创造的奇迹。 > “如果我们用这个预算和这支队伍赢了，我们就彻底改变了游戏规则。”（"But if we win on our budget with this team, we'll have changed the game."） Gabe 向在场的所有创业者与开发者呼吁：前沿生态的成熟已经赋予了应用层前所未有的机会，大家完全能够以极具性价比的资源撬动颠覆性的成果，“去改变这个游戏规则吧”。 --- ### [14:53](https://www.youtube.com/watch?v=MGouk8W51v0&t=893s) Q&A：攻克法律敏感数据，“逆向倒推”合成数据室 在问答环节中，现场观众针对法律行业的特殊性提出了关键疑问：法律领域充斥着极其敏感的机密数据，Harvey 的内部专家和律师到底是如何在不接触违规数据的前提下，合成出最顶尖的数据集来训练模型的？ Gabe 深入拆解了法律并购（M&A）业务面临的残酷现实： - **无法获得端到端输入数据**：当一家大律所代理企业并购时，律师面对的是包含多达 10,000 份合同的数据室（Data Room），外加海量的谈判邮件和会议纪要。这些核心过程资产从未对外公开，根本不存在像 GitHub 开源仓库那样现成的数据源。 - **公开数据的断层**：外界最多只能在公开渠道找到最终的《资产购买协议》（Purchase Agreement）等终审产物，却完全缺失推导出这一结果的全部前置材料。 - **人工捏造难以成立**：过去若直接要求一群律师凭空伪造一个拥有上万份彼此呼应、环环相扣合同的虚拟数据室，工作量与复杂度几乎难以实现。 面对这一死结，Gabe 的联合创始人兼兄弟 Julio 想出了一个极具创造性的“逆向破局法”： ```mermaid flowchart LR accTitle: 从评分标准倒推的数据室生成方案 accDescr: 展示从设定评分细则、埋设法律缺陷，到批量生成合同与验证模型能力的完整闭环流程。 A["确定评估标准<br/>(Rubric)"] --> B["预埋漏洞与缺陷<br/>(条款冲突/合同缺失)"] B --> C["合成虚拟数据室<br/>(海量关联合同)"] C --> D["专家润色调优<br/>(模拟真实业务质感)"] D --> E["模型生成尽调备忘录<br/>(Diligence Memo)"] E --> F{"对照预埋缺陷验证<br/>是否全部命中？"} ``` 1. **从评分标准（Rubric）倒推**：不再盲目构造合同，而是首先制定审核细则，并在底层预先埋设各类法律陷阱（例如关键合同缺失、不同条款之间互相冲突、权属瑕疵等）。 2. **生成完整关联数据室**：基于预先埋设的问题场景，批量生成互相印证的整套数据室，并通过专业平台与工具（如 Mercor 等）确保所有合同从格式到法律用语都极致逼真。 3. **闭环自动验证**：让模型阅读这套数据室并撰写尽职调查备忘录（Diligence Memo），由于所有陷阱都是提前埋入的，评估系统能精准判断模型是否完整揪出了所有预设问题。 这一突破彻底打破了以往的“先有鸡还是先有蛋”困境——过去律所总是要求：“先证明你们能训好，我们才给真实数据”，而技术方只能无奈回答：“不给我们数据，我们怎么证明给你看”。如今凭借逼真的合成基准，Harvey 成功赢得了顶尖律所的信任与主动合作意向。 --- ### [17:16](https://www.youtube.com/watch?v=MGouk8W51v0&t=1036s) Q&A：打破前沿实验室的人才垄断与研发解耦 对于现场提问的“应用层公司如何与 Anthropic、OpenAI 等前沿实验室争夺顶级研究员”，Gabe 坦承这是 Harvey 创立初期踩过的大坑之一。 - **薪酬壁垒与早期弯路**：Gabe 早年曾在主流大模型实验室工作，创业之初曾试图游说旧部加入。然而当时头部实验室的研究人员动辄能拿到高达上亿美元级别的薪酬包（$100M+ pay packages），当时体量的 Harvey 根本无法在正面战场竞争。 - **人才格局的结构性迁移**：随着技术逐步普及，高校在读博士以及厌倦了超大规模实验室官僚氛围的优秀研究者越来越多，顶尖人才的供给正在向外溢出。 - **全栈壁垒的瓦解与研发解耦**：早期之所以极度依赖稀缺的全能天才（如 Gabe 的前室友曾主导 OpenAI 的后训练业务），是因为当时的研究员不仅要设计算法，还得从零手搓庞大复杂的底层训练与推理部署基础设施。如今随着诸如 Fireworks 等类 Tinker API 服务生态的成熟，团队完全无需自己造轮子维护底层工程，这极大降低了算法落地的门槛，让应用层以更务实的团队规模吸纳高水平研究人才成为可能。 --- ### [18:46](https://www.youtube.com/watch?v=MGouk8W51v0&t=1126s) Q&A：无需“刻意做难题”，真实法律场景足以让前沿大模型触壁 有观众问及，在内部专家缺乏针对大模型的出题经验时，该如何设计出能够真正拉开前沿模型性能差距、具备足够区分度的评分基准（Rubrics）。 Gabe 的回答直截了当：团队完全不需要处心积虑去“专门为大模型设计难题”，只要**原汁原味地还原真实业务**即可。法律顶尖大案的复杂程度极其恐怖，只要将顶尖律所真实面对的客户业务原貌呈现出来，现有的前沿大模型在这些场景下依然会感到力不从心并频频碰壁。 - **深耕沉淀的直觉**：Julio 作为公司最早的员工之一，与大模型高强度磨合了整整 4 年，早已沉淀出极为敏锐的模型行为直觉，并把这套数据构建方法论推广到了内部整个律师团队。 - **核心焦点**：团队唯一的精力始终聚焦于打造最逼真、最严谨的数据室场景与尽调评分准则。一旦在真实业务模拟中运行测试，模型能力的断层与瓶颈自然会暴露无遗。随后，来自现场 Box 公司的观众及 Trolysis 的 Ross 紧接着开启了关于端到端工程管线（数据生成、算法研究与基础设施）中系统瓶颈的深入探讨。 ### [20:25](https://www.youtube.com/watch?v=MGouk8W51v0&t=1225s) 全链路排障：端到端系统的分层门禁机制 当被问及端到端工作流出现故障时，问题究竟更多出在数据层、研究层还是基础设施层，以及团队如何定位并编排排障流程时，Gabe Pereyra 指出这本身就是一个值得单开一场演讲的庞大课题。在他看来，系统故障并非由单一层面引起，而 Harvey 能够高效排查的关键在于他们早已找到了产品市场契合点（Product-Market Fit, PMF），并且拥有一套在生产环境中持续运行多年的成熟产品。早期 Harvey 深度依赖闭源模型，这反而帮助团队沉淀出了对整套端到端系统的运行信心——他们不仅建立了能够实时监控线上模型健康状态的推理服务基础设施，还磨炼出了无论是闭源还是开源新模型上线，都能迅速接入并进行评估与调度的“肌肉记忆”。 在这一套包含测试框架工程（Harness Engineering）、上下文管理等模块的复杂系统内，后训练（Post-Training）实际上只是整体链路中的一个小输入环节。研发团队完成一个模型的后训练后，会将其视作一个全新的外部模型接入主系统，从而触发全套评估指标。为了防止缺陷进入下游，Harvey 设立了多道清晰的“门禁”（Gates）：后训练模型若无法在基准测试中达标，流程就会被立刻阻断；而即便模型刷榜表现优异，一旦进入产品灰度测试后用户反馈“感觉不对劲”，往往意味着模型在基准集上发生了过拟合。例如，在面对通用助手式的宽泛提示词或分布外（Out-of-Distribution, OOD）场景时，这类模型就极易崩溃。这种分层的门禁设计，让研发团队能够迅速判断问题究竟卡在数据、模型还是工程层。 ```mermaid flowchart TD accTitle: 模型评估与发布门禁机制 accDescr: 展示模型从后训练经过基准测试、产品集成到通用鲁棒性检测的多阶段门禁流转过程。 subgraph EvaluationStage ["基准测试评估阶段"] A["后训练模型交付"] --> B{"基准测试 (如 LegalAgentBench)"} B -- "未达到阈值" --> C["阻断并回退：检查数据/训练参数"] end subgraph StagingStage ["灰度集成与产品验证"] B -- "达标通过" --> D["接入生产评估环境 (Serving Matrix)"] D --> E{"通用/分布外场景验证 (OOD Testing)"} E -- "过拟合/通用能力崩溃" --> F["阻断并排查：泛化能力与上下文工程"] end subgraph ProductionStage ["正式发布"] E -- "表现稳定" --> G["正式推向生产环境"] end ``` - **生产闭环优势**：凭借早期在闭源模型上验证成功的 PMF，团队积累了成熟的生产监控指标，使模型能力的波动能够被实时捕获。 - **定位过拟合风险**：专用基准测试存在盲区，模型可能在法律专业题库上表现抢眼，但在日常交互或起草通用邮件等分布外任务中失去稳定性。 --- ### [22:50](https://www.youtube.com/watch?v=MGouk8W51v0&t=1370s) 开源基准测试的博弈与战略考量 现场观众提到了 Harvey 从早期的 Big Law Bench 迭代到 LegalAgentBench，并询问在基准测试开源与闭源之间的权衡：开源基准容易被前沿大模型厂商用来针对性刷榜，而闭源又难以确立行业权威性。Gabe 坦言这确实是一种长期的平衡取舍，但在开源之前，Harvey 就已经与各大前沿实验室保持着密切合作，向其提供数据以帮助其提升底座模型的法律推理能力，进而反哺 Harvey 自身的产品表现。 随着业务发展，Gabe 认识到行业内部需要清晰透明的评估标准，帮助律师事务所与企业客户直观理解不同模型在具体法律场景下的优势与短板。创立之初，他曾以为“只要做出一款最强模型，客户就会买单”，但现实是每家客户的诉求与偏好千差万别。与此同时，前沿 AI 生态的爆发超出了预期，海量初创公司带着新技术主动寻求合作，开源数据集正好成为了天然的筛选过滤器——Harvey 无需消耗有限的研发人力逐一对接，只需让对方在公开数据集上验证效果，见效后再深入投入。从战略上看，Harvey 的核心防御壁垒并非这些用于通识提升的合成基准，而是帮助客户在各自私域数据上构建深度定制化系统的工程与数据能力。 - **赋能前沿底座**：通过与前沿实验室共享定向数据，促使基础模型底座更适应法律任务，Harvey 直接享受底层能力红利。 - **生态准入机制**：公开评测集成为外部创新技术（Neo-labs）的“试金石”，大幅降低了 Harvey 评估外部合作伙伴方案的沟通成本。 - **私域与公域区隔**：开源合成数据与评测集以推动行业基准演进，同时将企业私有数据的沉淀与定制视为不可动摇的商业壁垒。 --- ### [24:51](https://www.youtube.com/watch?v=MGouk8W51v0&t=1491s) 研发攻坚：分布对齐、超长上下文与持续学习 在谈及目前团队最希望前沿实验室协同攻坚的三大未解难题时，Gabe 首先列举了“合成数据与生产真实分布的脱节”。虽然 Harvey 能够利用领域专家生成极其逼真且由人工增强的合成数据，但这些数据往往是“面向未来场景设计”的（Future-Facing）。例如，系统可以模拟出极其真实的尽职调查虚拟数据室（Data Room），但在实际生产中，客户的使用行为并非局限于严格的尽职调查，还混杂了大量日常的邮件起草与琐碎沟通。受限于无法直接窥探客户隐私数据，如何消除合成任务与多变生产请求之间的分布差距（Distribution Gap），依然是最棘手的数据难题。 其次是超长复杂上下文的管理瓶颈。在尽职调查任务中，一个大型数据室的上下文体量可高达 8000 万 Token（80 million tokens）。当前哪怕是最先进的模型，在处理如此庞大且杂乱的环境时依然会频繁失效，模型架构与工程层均缺乏有效应对超复杂上下文的治理方案。最后则是企业级场景下的“持续学习”（Continual Learning）机制。Gabe 强调，Harvey 的终极目标并非单向输出一个通用法务大模型，而是让每个大型律所或企业客户在处理案件的过程中，其 AI 系统能够实现无缝进阶与自我迭代，同时在组织架构与安全层严格隔离客户机密，避免数据污染与泄露。这一课题交织着算法、架构与合规的多重挑战。 - **超大上下文规模**：真实尽调场景的数据量峰值达到 8000 万 Token，模型在超长序列下的注意力分配与环境交互存在巨大性能鸿沟。 - **分布不对齐困境**：因严守客户隐私红线，团队无法回流真实生产交互，导致面向复杂尽调构造的合成数据无法完全覆盖简单的日常助理工作流。 - **安全持续学习**：探索如何在严格保护客户机密（Client Matter Isolation）的前提下，实现模型在企业业务闭环中的增量进化。 --- ### [26:57](https://www.youtube.com/watch?v=MGouk8W51v0&t=1617s) 垂直化壁垒：从个人效率走向组织级工程编排 面对来自通用横向生产力工具（如各种通用办公与工作流平台）的竞争挑战，Gabe 阐明了 Harvey 战略重心的转变：软件定位正在从“个人生产力”（Individual Productivity）全面转向“组织级生产力”（Organizational Productivity）。传统的 AI 辅助工具通常着眼于个人律师如何更快地写完一段案情陈述或回复一封邮件，但在顶级律所的真实商业运作中，核心挑战在于如何安全、高效且盈利地统筹支撑上万家客户的多线并行交付。 以一个常规的法律项目为例，其难点往往不在于单点文本的起草，而在于它通常跨越长达 6 个月周期，牵扯全所 20 至 30 人的跨职能团队，还要与外部多方机构频繁协作。在这一层面上，软件的核心演变为集成了人类律师与 AI Agent 的复杂项目编排系统。进一步上升至千人规模的律所或财富 500 强法务部门，系统还需要接管资源配比、计费工时管理（Billing）、项目竞标（Pitching）以及上千家外部律所的跨组织协作。Gabe 总结道，历史上企业级软件克制通用型产品侵蚀的法则从未改变，那就是“极度深度垂直”（Hyper Vertical）——构建通用横向产品不愿涉足、也无法轻易理解的重度业务工作流。 - **竞争范式转移**：从单兵作战的代码/文本生成，转向多人、多 Agent 协作的长周期复杂项目管理与编排平台。 - **深耕组织痛点**：直击大型组织的经营本质，解决资源调配、成本核算、多团队权限与外部律所协同等综合治理诉求。 - **垂直防御逻辑**：通用产品倾向于打磨通用工作流，而面对法律行业高度非标、责任极重、协同极繁的特有生态，极致垂直的领域深度构成了天然的护城河。 ## Notable quotes > [00:00](https://www.youtube.com/watch?v=MGouk8W51v0&t=0s) 世界上有富有的队伍，有贫穷的队伍，然后就是处在应用层的我们。 — Gabe Pereyra > [02:04](https://www.youtube.com/watch?v=MGouk8W51v0&t=124s) 如果你没有一个好的 Benchmark，你就无法训练模型；而如果你无法训练模型，你根本无需操心后续的线上服务。 — Gabe Pereyra > [03:36](https://www.youtube.com/watch?v=MGouk8W51v0&t=216s) 除非有大量的人在上面训练模型，否则你很难确信自己的数据集究竟好不好。 — Gabe Pereyra > [10:47](https://www.youtube.com/watch?v=MGouk8W51v0&t=647s) 产品反馈，其实就是愤怒客户发来的邮件 — Gabe Pereyra > [13:55](https://www.youtube.com/watch?v=MGouk8W51v0&t=835s) 如果我们用这个预算和这支队伍赢了，我们就彻底改变了游戏规则。 — Gabe Pereyra > [14:53](https://www.youtube.com/watch?v=MGouk8W51v0&t=893s) 先证明你们能训好，我们才给真实数据 — Gabe Pereyra > [14:53](https://www.youtube.com/watch?v=MGouk8W51v0&t=893s) 不给我们数据，我们怎么证明给你看 — Gabe Pereyra > [22:50](https://www.youtube.com/watch?v=MGouk8W51v0&t=1370s) 只要做出一款最强模型，客户就会买单 — Gabe Pereyra > [26:57](https://www.youtube.com/watch?v=MGouk8W51v0&t=1617s) 极度深度垂直 — Gabe Pereyra ## Mentioned Harvey, Gabe Pereyra, Sequoia Capital (红杉资本), Google DeepMind, Meta, Google Brain, OpenAI, Anthropic, LangChain, Mercor, Snorkel, Fireworks.ai, Baseten, Ngram, Trajectory, Applied Compute, Box, Trolysis, LegalAgentBench, Contracting Dataset, Diligence Dataset, Associate 1, Big Law Bench, Vault, Cursor, Composer, GPT-3, Kimi k1.5, GLM-4 / 5.2, NeMo-Megatron, Claude 3.5 Opus, Tinker, ImageNet, CIFAR, MNIST, Billy Beane, Elon Musk, Julio, Brendan, 《点球成金》(Moneyball) ## Video description How does an application company compete with frontier labs that have more money, talent, compute, and data? At Sequoia Capital’s Own Your Intelligence event, Harvey co-founder and president Gabe Pereyra shares the playbook: leverage the frontier ecosystem instead of building everything yourself. Gabe walks through how Harvey built its research lab, starting with benchmarks like LegalAgentBench and its open-sourced diligence dataset, using domain experts to guide synthetic data generation, partnering with multiple neolabs for post-training, and building the serving and evaluation infrastructure to put models into production with confidence. He explains why open-weight models have made post-training worth it for the first time, and why every company will eventually need some version of this playbook. 00:00 Introduction 00:37 Building a research lab on a budget 02:28 Legal Agent Bench, contracting, and the diligence dataset 03:57 Domain experts guiding synthetic data generation 05:23 Why Harvey open sourced its datasets 06:55 Working with the neo labs — and why more than one 08:20 Post-training in-house: building "Associate 1" 09:44 The model serving matrix: 60 countries, fallbacks, SLAs 11:05 Deciding what stays in production 12:29 Simple open source switches and model routing 13:55 Moneyball: "If we win on this budget, we change the game" 14:53 Q&A: Training with sensitive data 17:16 Q&A: Competing for research talent 18:46 Q&A: Designing rubrics that actually challenge frontier models 20:19 Q&A: Where the pipeline breaks — data, research, or infra 22:59 Q&A: The tension in open sourcing a benchmark 25:02 Q&A: Biggest remaining open problems 27:10 Q&A: Competing with horizontal products [Source](https://www.youtube.com/watch?v=MGouk8W51v0)

[0:00] Gabe Pereyra: 世界上有富有的队伍，有贫穷的队伍，然后就是处在应用层的我们。 [2:04] Gabe Pereyra: 如果你没有一个好的 Benchmark，你就无法训练模型；而如果你无法训练模型，你根本无需操心后续的线上服务。 [3:36] Gabe Pereyra: 除非有大量的人在上面训练模型，否则你很难确信自己的数据集究竟好不好。 [10:47] Gabe Pereyra: 产品反馈，其实就是愤怒客户发来的邮件 [13:55] Gabe Pereyra: 如果我们用这个预算和这支队伍赢了，我们就彻底改变了游戏规则。 [14:53] Gabe Pereyra: 先证明你们能训好，我们才给真实数据 [14:53] Gabe Pereyra: 不给我们数据，我们怎么证明给你看 [22:50] Gabe Pereyra: 只要做出一款最强模型，客户就会买单 [26:57] Gabe Pereyra: 极度深度垂直

[13:55] Gabe Pereyra: If we win on this budget with this team, we'll have changed the game.

[0:00] Next up, we are honored to have Gabe [0:02] with us. Gabe is co-founder and [0:03] president of Harvey. Um you were a [0:06] research scientist at DeepMind ages ago, [0:09] um and then at Meta I think before you [0:11] showed your college roommates uh [0:13] what GPT-3 could do. And this duo uh [0:17] then became Harvey. Um we're really [0:19] excited to have you here. I think [0:21] everybody here in the audience is an [0:22] application company um thinking about [0:24] how to start doing their own research, [0:26] post-training their own models, um and [0:28] uh create their own labs. And so I think [0:31] Harvey has really set the example here, [0:32] and we're really delighted to have you [0:33] give a talk on how to how you guys built [0:35] Harvey Labs. [0:37] >> Awesome. [0:38] >> [applause] [0:44] >> So the alternative title to this talk is [0:48] building a research lab on a budget. [0:51] And [0:52] it's an unfair game competing with the [0:54] frontier labs if you're an application [0:57] layer company. [0:59] There are rich teams, there [snorts] are [1:01] poor teams, [1:02] and then there's us in the application [1:03] layer. [1:06] The frontier labs have more money, more [1:08] talent, more compute infrastructure, and [1:11] data. [1:12] So how do you compete? [1:16] By using the frontier ecosystem. [1:19] When we started Harvey 4 years ago, [1:22] most of these companies either didn't [1:24] exist or were just getting started. [1:27] And so we either had to build everything [1:29] ourselves [1:30] or in most cases focus on something [1:32] different. Like building our GTM org and [1:35] a great product. [1:37] But today [1:39] using the frontier ecosystem [1:41] I think you can compete with the [1:43] frontier labs and build frontier [1:45] intelligence. [1:47] This talk is going to be our high-level [1:49] playbook for doing this. [1:51] And I'm going to talk about how we build [1:53] benchmarks and training data, [1:55] how we work with the Neo Labs to do post [1:58] training [1:59] and how we serve these models in [2:01] production. [2:04] To start, [2:06] you want to build a benchmark. If you [2:07] don't have a good benchmark, you can't [2:09] train models, and if you can't train [2:11] models, [2:12] you don't need to serve them in [2:13] production. [2:14] So, this year we released three of these [2:17] data sets that we built. [2:19] We started by building Legal Agent [2:21] Bench, which is a taxonomy of tasks that [2:25] associates would do at a large law firm. [2:28] And [2:29] these cover multiple practice areas, and [2:31] they're complex tasks like doing [2:34] drafting complex fund formation [2:36] documents, doing case law research, [2:38] things like that. [2:40] We followed this up by building a [2:41] contracting data set. This teaches This [2:44] allows us to teach agents to do [2:46] negotiation like you would in an [2:47] in-house department. [2:49] And then the one I'm most excited about [2:51] that we recently released is a large [2:53] diligence data set. [2:55] This is, I think, one of the largest RL [2:57] environments that's been released. The [2:59] largest data rooms here are 80 million [3:01] tokens, and it lets us do research on [3:04] long context, very complex tasks. [3:08] And I think the most interesting thing [3:10] about these data sets is how we built [3:12] them. [3:14] One challenge we always had at Harvey [3:16] for training models is we can't train on [3:19] our customers' data. We work with the [3:21] largest law firms and enterprises, [3:24] and their legal data is incredibly [3:25] sensitive. It's privileged, and so you [3:28] can't put it in generic models. We can't [3:30] even put it in our models. And so, how [3:33] do you train models given that? [3:36] And the thing that started working [3:38] really well this year [3:39] is using domain experts to guide [3:43] synthetic data generation. [3:45] And [3:46] uh Brendan from Reka had a good analogy [3:48] that the same way that engineers now [3:51] don't write code, they vibe code and [3:53] guide these coding models. We're [3:55] starting to do the same thing. And so, [3:58] my younger brother is actually a lawyer [4:00] at Harvey, and he's gotten very good at [4:02] using the coding models. He's trained [4:04] our other lawyers to do that, and they [4:05] can generate incredibly realistic data [4:08] sets that we can use for training and [4:10] also evaluating our product. [4:14] And once you've done that, [4:16] synthetic data isn't good enough, but [4:19] it's a way to get started. [4:21] And so, we work with companies like [4:23] Mercor and Snorkel, who let you scale up [4:26] this process and build larger sets [4:29] particularly for training. [4:33] Once you've done that, you need to turn [4:35] these data sets into efficient RL [4:37] environments. It gets really expensive [4:40] as these data sets get larger and [4:42] evaluation gets very expensive. For [4:44] example, in our diligence data set, we [4:46] have over 1,000 unit tests that are [4:49] grading model outputs using LLM as a [4:52] judge. If you use the largest models and [4:55] you want to do RL rollouts and things [4:57] like this, it gets very expensive. And [4:59] so, there's a lot of work, and here's [5:01] some we did with LangChain, of making [5:03] these very efficient. [5:06] And then the last thing we did that I [5:09] think was a little controversial at the [5:11] time, was open-sourcing some of these [5:13] data sets. And the motivation for this [5:16] was [5:18] it's very hard to know your data set is [5:20] good unless a lot of people train on it. [5:23] When I used to do research at Google [5:25] Brain and DeepMind, the best data sets [5:28] were open, like ImageNet, CIFAR, MNIST, [5:31] and everyone used them, and you were [5:33] able to find all of the issues, and we [5:35] get a ton of pull requests, we get [5:37] suggestions. Um [5:39] and then I think increasingly we're [5:41] having the labs when they report new [5:43] models benchmark on our data set. And [5:46] then most importantly, we had Elon [5:48] retweet it. [5:52] So, once you've built the benchmark, now [5:54] you have something to train models [5:56] against. [5:57] And [5:59] the thing that is exciting now is [6:01] open-source models are getting [6:02] competitive. In the past, it wasn't [6:05] worth doing post training because the [6:07] models were improving so quickly from [6:09] pre-training that any post training you [6:12] did quickly got absorbed by the next [6:14] pre-trained model. [6:15] But now with models like Kimmy 3, GLM [6:18] 5.2, NeMo-Megatron, Inkling, and others, [6:21] it's possible to take these very strong [6:23] open-source base models and post train [6:26] them to levels of frontier intelligence. [6:29] Maybe not general frontier intelligence, [6:31] but if you have a specific task like us, [6:34] they are competitive. [6:36] And so, the way we recommend getting [6:37] started is working with the NeMo labs. [6:41] They have a bunch of expertise and [6:43] infrastructure already in place to help [6:46] you make sure that your training data [6:48] sets are good. They have recipes. And [6:50] usually, if you work with them and [6:53] you're not able to get better results, [6:55] there's probably something you're doing [6:56] wrong with your data set, and this is a [6:58] very good way to bootstrap it. [7:01] And so, some of the interesting work we [7:03] did with these different providers, um [7:06] Fireworks, we got some very interesting [7:08] results training GLM 5.1 to use Fable or [7:12] maybe Opus 4.8 as an advisor model. [7:16] Um Base 10, we did some interesting work [7:18] on KB compaction. N gram, who I think is [7:21] here, we're doing interesting work on [7:23] enterprise search and firm knowledge. [7:26] Trajectory, um we worked with them to [7:28] train NeMo-Megatron models. And Applied [7:30] Compute, we're doing some interesting [7:32] work on our Vault product. [7:34] And I think one question we got is why [7:37] work with multiple NeMo labs. Why not [7:39] just pick one? [7:41] And for us, as we're scaling the [7:43] research lab, we have more research [7:45] projects than we have bandwidth to do [7:47] internally or just with a single Neo [7:49] lab. [7:50] And every Neo lab is taking a different [7:53] bet. They have different ways they think [7:55] about research. We have different open [7:57] source models we want to train. And the [8:00] more we work with, the more we learn. [8:04] And it's getting easier than ever to do [8:06] post training. So one, working with the [8:08] Neo labs, we're learning a lot in [8:10] partnership with them. [8:12] And then we're doing more and more post [8:13] training ourselves internally with APIs [8:16] like Tinker and the infrastructure [8:18] Fireworks and Base 10 have built. It's [8:20] never been easier to post train these [8:22] models and then serve them. And there's [8:24] increasingly more post training talent [8:26] that we're hiring and is available. [8:29] And inspired by Cursor, the goal of [8:32] these efforts is for us to build our [8:34] version of Composer one. How do we [8:37] package all of the work we've done with [8:39] synthetic data, scaling it with Mercor, [8:41] the work with the Neo labs, into a model [8:44] we can serve alongside the closed source [8:45] models. [8:49] Now, once you've post trained a model, [8:51] you need to be able to serve it in [8:53] production. [8:54] And this is non-trivial. [8:57] So I want to start first by talking [8:59] about our model serving infrastructure [9:02] because I think sometimes people still [9:03] think about application layer companies [9:05] as you're calling a single model [9:07] endpoint and it's maybe a chat product. [9:10] And so a big problem we've solved over [9:12] the past four years is [9:15] we operate in 60 countries. We have [9:17] multiple product surface areas. [9:19] Customers have different model [9:20] preferences. [9:21] And even with the closed just the closed [9:23] source models, how do you serve these at [9:26] scale? [9:27] And so [9:28] this matrix gives you a sense of all of [9:31] the things we need to handle when we're [9:32] thinking about serving these models at [9:34] scale. [9:35] And as a simple example for each of [9:37] these model families, we need to serve [9:39] multiple of these models. We need to [9:41] have fallbacks across providers to hit [9:43] our SLAs. [9:45] And now with companies like [9:47] Fireworks.ai, we can add open-source [9:49] models into this mix. [9:54] In order to think about when you serve [9:56] models in production and how you modern [9:59] them modern them in production, you need [10:01] to have this infrastructure in place [10:04] thinking of even before you think about [10:05] post-training. And so this is how we [10:07] think about when there's a new model [10:09] released, whether it's open-source, [10:11] closed-source, or a model we've [10:13] post-trained, how we make decisions [10:15] whether to put it in production. And [10:17] then once it's in production, how we [10:18] make decisions whether to keep it in [10:20] production. [10:21] >> [snorts] [10:21] >> And so pre-production, we have a set of [10:24] generic evals. So in terms of automated, [10:28] we have the lab benchmark that I talked [10:30] about where whenever a new model comes [10:32] out, this gives us a very quick sense of [10:34] is it a frontier model, how strong is [10:36] it, what areas of legal is it good. [10:39] We have human testing in the generic [10:41] case where we run side-by-sides of this [10:44] model with other models to compare them. [10:47] And then for every product surface, we [10:49] have critical user journeys and [10:51] automated product tests because a model [10:53] could be very good generically, but it [10:55] could not be a good fit for a specific [10:58] product surface. [10:59] And then we also have human product [11:01] testing. And together these signals [11:04] along with heuristics around cost, [11:06] latency, region availability is how we [11:08] decide whether we put a model into [11:11] production. And I think the important [11:13] point here is [11:14] this is the case for post-trained models [11:17] or non-post-trained models. And so you [11:19] can reuse this infrastructure and you [11:21] should have it in place before you think [11:22] about post-training. [11:24] Once a model's in production, same [11:26] thing, doesn't matter if the model's [11:27] post-trained or not. We do AB testing if [11:30] we're rolling out a large change, we [11:32] look at engagement to track if this [11:34] model's performing how we expect. Um [11:37] and then we look at things like uptime, [11:39] token efficiency, and then we call it [11:41] product feedback, I call it angry [11:43] customer emails. And so there's There's [11:45] all of these signals that tell you this [11:47] is working as expected. [11:50] And so you need that infrastructure in [11:52] place before you think about serving [11:54] models. [11:55] And then [11:56] the thing you need to do even still [11:58] before serving models is what I call the [12:01] simple open-source switches. [12:04] And so the first one is look at all the [12:07] places you're serving models and find [12:10] are there places in my product where I [12:12] can just naively swap open-source [12:14] models? And so for example, we have [12:16] parts of our product that generate [12:18] citations that don't need the largest [12:19] models and there's opportunities to swap [12:22] in GLM 5.2 and get cost or performance [12:25] benefits. And so that's the first thing [12:27] and doing this builds the muscle of [12:30] serving open-source models alongside [12:32] closed-source models. [12:34] And then the second which has gotten uh [12:37] very popular now is model routing. So [12:40] there could be places where you can't [12:41] naively swap an open-source model, but [12:44] what you can do is on certain queries [12:46] route to open-source models. [12:50] And then once you've done this, you have [12:52] everything in place to start building [12:54] the post-training flywheel. [12:56] And [12:58] once you start serving them in [12:59] production and collecting feedback, and [13:02] caveat here, you need to be very careful [13:04] about what collecting feedback means. In [13:06] our case, it does not mean training on [13:08] customer data, [13:10] but we do get feedback signals from our [13:12] user testing and other things that can [13:14] inform how we build future data sets and [13:17] improve these models. [13:20] And that is our playbook for building a [13:23] research lab. [13:24] We think in the future every AI comp- [13:27] every company is going to need to become [13:30] an AI company and figure out some [13:33] version of this playbook. [13:35] And I think despite that, most people [13:38] are still betting against this playbook [13:41] and application layer companies and the [13:43] frontier ecosystem. [13:46] But [13:47] >> But if we win [13:49] on our budget with this team [13:56] we'll change the game. [13:59] >> This is a scene from Moneyball, which [14:01] hopefully you've seen this movie, [14:03] where they're talking about we just won [14:06] 20 games in a row. [14:08] And Billy Beane says, "It doesn't [14:10] matter. If we don't win the [14:12] championship, no one's going to [14:14] appreciate what we've done here." [14:16] And the quote is, "But if we win [14:20] on this budget with this team [14:23] we'll have changed the game." [14:25] And [14:26] I think now with the frontier ecosystem [14:29] all of you have the opportunity to do [14:31] the same. [14:33] Go change the game. [14:34] Thank you. [14:36] >> [applause] [14:42] >> Do you want to do Q&A? [14:44] >> You okay with that? [14:45] >> Yeah, of course. [14:46] >> [snorts] [14:48] >> Yeah. [14:49] >> I'm sorry, who's speaking? Um you talked [14:51] about the challenges with having uh [14:53] legal customers with very sensitive data [14:56] sets and talked a little bit about your [14:58] brother's lawyer and other in-house [14:59] experts doing their version of live [15:01] coding to produce the best work. What [15:03] does that look Can you give us a little [15:04] more detail about what that actually [15:06] looks like? [15:06] >> Yeah, great question. So [15:09] the biggest challenge with legal work is [15:12] when you're at a large law firm, the [15:13] type of work you're doing is, for [15:15] example, you're representing a company [15:17] doing an M&A. [15:19] And you get a data room, which is all [15:21] the contracts of the company you're [15:22] trying to acquire, plus there's all [15:24] these emails and meetings about the [15:26] negotiation. [15:27] None of that data is public, and so you [15:29] don't have this analogy of like open [15:31] source GitHub repos. [15:33] And the biggest challenge you run into [15:34] when training is you can maybe find some [15:37] of the final work product publicly, so [15:39] like a public purchase agreement, but [15:41] you don't have any of the input data. [15:43] And historically it was very awkward to [15:47] get a bunch of lawyers and say, "Hey, [15:48] make a fake data room." Because these [15:51] data rooms can be 10,000 contracts. They [15:53] all need to fit together. [15:55] And so what Julio figured out is [15:59] a very clever way to generate these data [16:01] sets. And so he started from the rubric, [16:04] and so he planted all these issues and [16:06] said, "Here's all the problems in the [16:07] data room and what I'm going to check [16:09] for [16:10] in a scenario, and then use that to [16:12] generate the data room, so you can plant [16:14] all of these issues like these contracts [16:16] don't tie together, this contract's [16:18] missing, and generate all of the data, [16:21] and then we'll use Mercor Shenorcal to [16:22] make all the contracts look realistic, [16:25] but now you have this input data set, [16:27] and you can have the model generate like [16:28] a diligence memo, and this way of [16:30] checking it to say, "Hey, did you catch [16:32] all of these issues that we know that [16:34] are in here because we planted them?" [16:36] And obviously, depending on the work [16:37] you're doing, you need to be clever of [16:39] how you create it, but that to me feels [16:42] like the really big unlock because now [16:44] we can start [16:46] doing this training that before you just [16:49] you had this chicken-and-egg problem [16:50] where we'd go to law firms and say, [16:51] "Hey, we can train you this model on [16:53] this client data." And they'd be like, [16:54] "Okay, prove it." And we were like, [16:55] "Show us the data." And they're like, [16:56] "No." And so now you can prove it here. [16:59] And now we have a lot of interest from [17:00] law firms saying, "Oh, this is really [17:02] interesting. Can we do it with our [17:03] data?" [17:05] Good question. Yeah. [17:06] >> Um [17:06] so, we're also [17:08] hiring for this. [17:09] >> We're also building an implied AI there. [17:11] Um from a hiring point of view, [17:14] it's difficult for you to compete with [17:16] researchers from Anthropic and and the [17:18] labs. [17:19] Do you try and play that game or do you [17:21] try and hire domain logic and lawyers [17:22] and experience? And where has that [17:24] worked and where has that not worked? [17:26] >> Yeah, I think this was one of the big [17:28] mistakes I made like when we first [17:29] started the company because I had worked [17:32] in these labs and so I knew a lot of the [17:34] folks and I would say, you know, come [17:35] work with us and then they were getting [17:37] these, you know, 100 million plus pay [17:39] packages and we weren't at the scale [17:41] where this made sense. And so I'd say [17:43] we're now just getting to the company [17:44] size where I would say we're not [17:47] competing with kind of the frontier [17:48] talent, but there is increasingly [17:51] folks that are doing PhDs, folks that [17:54] don't want to work at large labs and so [17:55] I think that is changing. And then I [17:57] think the second thing is [17:59] a lot of why you needed that talent [18:02] early on was it wasn't just doing the [18:05] post training, it was you need to build [18:08] the training infrastructure, the serving [18:09] infrastructure, and all of those things [18:11] combined, I think the talent to do that [18:14] was like very unique. Like my old [18:15] roommate was one of the folks who like [18:17] ran post training at OpenAI and he was [18:19] one of the best researchers I've ever [18:21] worked with, but now you can use things [18:23] like Fireworks, uh, tinker-like APIs, [18:26] and so you don't need to build the [18:27] training and serving infrastructure, and [18:29] so I think it opens up the pool of [18:31] talent and it feels very feasible to get [18:33] some of this talent now. [18:35] Yeah. [18:37] >> Uh, yeah, just another question on the [18:38] benchmark creation. So, when you're like [18:40] creating these rubrics, you still have [18:42] to create them in such a way that [18:43] they're separating them the frontier [18:45] models and you have to do that with the [18:46] experts themselves. Like how do you find [18:48] like navigating that challenge? So, the [18:50] experts kind of have to create this [18:51] rubric knowing or like trying to figure [18:53] out how the models are going to perform [18:54] on them. [18:54] >> So, the What do you mean separating the [18:57] >> Uh, I guess so, you have to create the [18:58] rubrics that can adequately separate [19:01] separate the front They they have to [19:02] challenge the the frontier, right? [19:03] >> Yep. [19:04] >> Um, I guess how do you navigate that [19:05] especially when you're like having these [19:06] in-house legal experts who might not [19:08] exactly know how to create that rubric [19:10] designed for a model. [19:12] >> Yeah, that's a good So, we think of it [19:14] more as how do we make them realistic [19:17] representations of the work our [19:18] customers are doing? And I think part of [19:21] why we picked the legal domain is if you [19:23] just build a realistic client matter [19:25] that these top firms are working on, [19:27] like the frontier model still can't do [19:30] this. [19:31] And [19:32] I think the thing over the past 4 years [19:34] we've built is So, for example, my [19:36] brother was one of our first hires. And [19:38] so, he's been working with the models [19:40] for 4 years. And so, his intuition of [19:43] how the models work, how to generate [19:44] these data sets is incredibly good. And [19:47] he's trained a bunch of lawyers to do [19:48] this. But I would say the focus is still [19:50] more of how do we build a really [19:52] realistic data room, and then rubrics [19:55] for the diligence. And then when we run [19:56] these models, we find, okay, there's [19:58] gaps and there's there's room for [19:59] improvement. But I think depends on the [20:01] domain. [20:03] Uh God. [20:04] >> Hey, uh Shadeen from Box. [20:06] >> Uh sorry, let me do that one and I'll [20:07] >> Great. [20:08] >> I mean, if you think [20:10] Great talk, by the way. Um Ross from [20:12] Trolysis. If you think of the uh your [20:15] you know, the the end-to-end motion is [20:17] like data generation, the environment [20:18] piece, then there is the algorithm piece [20:20] of like what how do we train, etc., the [20:23] research piece, and then there's the [20:24] infra piece. [20:25] If something is not working, like [20:28] anecdotally, what have you seen? Is it [20:31] like [20:31] operationally or dollar-wise or human [20:34] hours-wise, is it usually in the data [20:35] layer? Is it in the research layer? Or [20:37] is it in the infra layer? And how do you [20:39] go from there? [20:40] >> Who's orchestrating or debugging this [20:42] whole end-to-end pipeline in some ways? [20:44] >> Yeah, this is This is a great question. [20:46] I mean, this is a whole 'nother talk. [20:48] Um [20:50] I don't think there's a single [20:52] thing. So, I would say [20:55] what makes it easier for us is [20:59] we found product-market fit. We have a [21:01] product that's being used in production. [21:04] And we did this largely with the [21:05] closed-source models to start. [21:08] And so, that let us build a lot of this [21:11] kind of confidence in the thing is [21:13] working end-to-end. And so, we have a [21:16] product, people are using it, people [21:17] have been using it for many years. [21:19] The models work in this way, we can [21:21] monitor whether they're working in [21:22] production. So, that was kind of the [21:23] point of serving of you need a lot of [21:25] this in place. [21:27] And then, we have built the muscle of [21:30] new models come out, how do we put those [21:32] into the product, whether they're [21:34] closed-source or open-source models. [21:36] And so, that's you can kind of think [21:38] about the full cycle. And then, [21:39] obviously we've done a bunch of work of [21:41] like Harness engineering and context [21:43] management and all of these things. And [21:46] so, now you can think of post-training [21:47] as one small input into this broader [21:50] system. And so, we have our team [21:52] post-train a model, and then you feed it [21:54] into this broader system of, you know, [21:56] you treat it just as another new model. [21:58] And so, then you have all these signals [22:00] of, you know, what's going wrong. And [22:03] so, [22:04] I would say, but across these steps, [22:07] there's kind of easy ways to gate it. [22:08] So, if you start with the benchmarks and [22:10] you post-train a model and it doesn't [22:11] work well on those, then it's unlikely [22:13] to go farther. If it works well on that, [22:16] and then you put in a product, but [22:18] someone uses the product and they say [22:19] things feel weird because, for example, [22:22] on the benchmarks we've built, there is [22:24] still things that they definitely don't [22:26] catch. Like, the best example is you can [22:29] have a model that does very well on our [22:31] benchmarks, but they've overfit to some [22:33] degree, and then you put it in a more [22:34] generic assistant-like product, and it [22:36] kind of falls apart for when you go out [22:38] of distribution. But, yeah, you need to [22:41] have kind of all of the like stages and [22:43] gates in place, and then it kind of [22:45] becomes obvious where things are [22:46] breaking. But, good question. [22:48] >> Yeah. [22:50] >> Oh, I had a question. Hey, great to meet [22:52] you. I'm from Rocks. [22:53] Uh on the on the benchmark side, legal [22:56] agent, I think you guys had Big Law [22:58] before, Big Law bench, and then now [23:00] legal agent of [23:01] What's your philosophy on open sourcing [23:03] the benchmark? Because if you open [23:04] source, of course you're getting [23:06] traction with other people trying out [23:07] the benchmark, but the labs can help [23:08] climb as well. And then if you close [23:11] source, then there's a question of is [23:13] this a real benchmark? So how do you how [23:15] do you how do you how do you what's your [23:16] philosophy? [23:17] >> I I think that's like a good tension [23:19] that we think about. I would say even [23:21] before we open source this benchmark, we [23:23] already work closely with the labs. And [23:25] so we share data with them to help them [23:27] improve their models, which improved our [23:29] product. [23:30] And so [23:31] I would say the way we think about it is [23:34] there is a huge advantage in your [23:37] industry, particularly in legal, in [23:40] terms of helping our customers [23:42] understand how good different models are [23:44] at different things. And so I think when [23:46] we when I started Harvey [23:49] I kind of had the intuition of oh, we'll [23:50] just build the best model and then [23:53] customers will be happy because it's the [23:55] best. [23:56] And it's very clear now that every [23:57] customer has different preferences, [23:59] different models are good at different [24:01] things. And then I think the second that [24:04] I didn't anticipate is kind of how big [24:06] the frontier ecosystem would become. And [24:08] so now we have so many companies [24:10] reaching out saying, "Hey, we have this [24:12] new technique or we did this thing. Can [24:13] we try it?" And before we just didn't [24:15] have the bandwidth where we're just like [24:17] we're working on these other things. But [24:19] now with this open source data set we [24:20] could be we can say, "Hey, go try this [24:22] and if it works, then great. This is [24:24] something that's interesting to invest [24:26] in." Um and then I think the way we [24:28] think about it strategically is [24:31] the valuable data for us is going to be [24:33] helping law firms train their own [24:35] systems and how they work on private [24:38] data. And then we want to help everyone [24:41] improve these systems with synthetic [24:43] data and kind of some of the data we [24:45] open source, but obviously a balance for [24:47] the reasons that you mentioned. [24:49] Yeah. [24:51] >> You mentioned [24:52] finding a lab. [24:53] What are the remaining open questions [24:54] for for that you want help on? [24:57] >> Yeah, that's a great question. [24:58] Um [25:00] I would say one big challenge, one of [25:02] the biggest challenges [25:03] so we can generate these very realistic [25:06] synthetic data sets, we can augment them [25:08] with humans, but the distribution of [25:10] that data still doesn't match our [25:12] production distribution. And so I would [25:14] say these data sets we're generating are [25:16] somewhat future facing. And so what I [25:18] mean by that is we can generate a really [25:21] realistic [25:22] data room, [25:23] but right now our our product isn't used [25:26] purely to do a due diligence. And so if [25:28] a model does well on that, but then [25:29] someone uses it to draft an email, maybe [25:31] it doesn't work as well. And so I think [25:33] there's still that gap because we can't [25:35] look at customer data, and so how do you [25:36] like bridge that gap? So I think that's [25:39] to me that that's the biggest question [25:40] on the data set side, and that one that [25:42] one is challenging. [25:44] There is [25:46] I think there is a bunch of questions [25:48] around [25:49] for example, again with the diligence [25:51] data, like the largest data room is 80 [25:53] million tokens of context. Like the [25:55] models don't do a good job of managing [25:57] this context, and how do you train these [25:59] models to operate in kind of these very [26:01] complex environments? So I think that's [26:03] another one of just there's still a big [26:05] performance gap. [26:07] Um and I would say the third is [26:11] like we're making good progress on how [26:13] do we post train our [26:14] post train models ourselves, but I think [26:16] figuring out some form of continual [26:17] learning is I think the end game here is [26:20] not for us to build the best legal [26:21] model. [26:22] It's for us to help every law firm or [26:24] enterprise customer customize it to the [26:27] type of work they're doing. [26:29] And so just thinking about how do you [26:31] operationalize that, where you have a [26:33] large law firm and every time they work [26:36] on a client matter, [26:37] their AI system gets better, but you're [26:39] also protecting the client data. I think [26:41] that's kind of one huge like both [26:44] technical, operational, and like AI [26:46] challenge that that is super [26:47] interesting. [26:49] Yeah. [26:50] Uh [26:51] We're done. [26:52] Or more? I can do more. I can [26:56] Uh yeah, maybe last one. [26:57] >> Yeah. So, you know, you talked a lot [26:59] about like, you know, the the [27:01] the foundation readiness, right, to [27:02] compete with Front Row Live, right, [27:04] including your open source model, [27:05] including your infrastructure kind of [27:07] like Ruby. [27:08] From product perspective, right, Coda as [27:10] cloud product, they're general product, [27:12] right? So, from product perspective, how [27:14] do you guys compete with the, you know, [27:17] Coda as cloud workflow, methodology, [27:20] efficiency? How do you think about that? [27:23] >> Yeah. I think the big shift that we're [27:26] thinking about is [27:28] like our original product and then [27:30] things like Coda's Coda Work are [27:33] very individual-focused products, and so [27:36] they're about individual productivity. [27:38] And increasingly, the product we're [27:39] building is about organizational [27:41] productivity. And so, if you think of a [27:43] large law firm, [27:45] the problem they're trying to solve is [27:47] not how do I make my individual lawyers [27:49] more productive. The problem they're [27:51] trying to solve is I have 10,000 [27:53] clients. I'm working on client projects [27:55] for all of them. I need to make sure [27:57] that all those client projects go very [28:00] well, and then I can also do them in a [28:02] way that's profitable. And so, a lot of [28:04] the infrastructure we're building for [28:05] the law firms is how do you operate that [28:08] machine? And when you start thinking [28:10] about [28:12] an individual client project, [28:14] most of the challenge is not how do I [28:16] draft this one section. It's I'm working [28:20] on this project for 6 months. I have a [28:21] team of 20, 30 people across the firm. I [28:24] need to coordinate all of them. I need [28:25] to make sure that I'm coordinating all [28:27] the outside parties. And so, a lot of it [28:29] is starting to look like project [28:31] management that is orchestrating these [28:33] humans and these agents. And that's at [28:36] the, I would say, at the team level. And [28:38] then, if you think about it at the [28:39] organization level, now you have a [28:41] thousand of these projects, and you need [28:43] to start thinking about resource [28:45] allocation, what am I billing, what am I [28:47] pitching? And then with enterprises, [28:49] it's even more complicated where kind of [28:51] a Fortune 500, they're working with a [28:53] thousand law firms. They have a thousand [28:55] people internally. What are all the [28:57] systems to kind of start work is trading [28:59] that work? And so I would say the simple [29:01] answer is just, and this is I think [29:02] historically what enterprise has has [29:04] done is how do you just go hyper [29:06] vertical into your domain in a way that [29:08] the horizontal products won't. Yeah, [29:10] good question. [29:13] >> Thank you, Gabe. That was an iconic [29:14] talk. Thank you for joining us. [29:17] >> [applause]