Cursor Trains Composer 2 Model via Fireworks to Cut Inference Costs

Cursor通过Fireworks训练Composer 2模型以降低推理成本 剥离基础模型中宽泛的通用知识并仅针对编辑器工作流进行训练，能够以更低的算力成本实现前沿水准的编程性能。 Cursor已从调度通用前沿模型转向训练其自有的专用基础模型Composer 2，将模型权重容量严格重新分配给软件工程工作流。该团队采用自顶向下的方法，以Kimi 2.5为基础模型，在全球分布式集群上进行了大量的代码中期训练（mid-training）与大规模异步强化学习（RL）。为攻克复杂的基础设施挑战，他们设计了全新的解决方案，包括异步rollout工厂、20倍权重增量压缩、用于确保混合专家（MoE）路由一致性的路由器重放，以及支撑环境rollout的10万台虚拟机技术栈。这种专用模型训练与分布式基础设施的协同设计，得以在推理成本降低一个数量级的同时，实现前沿水准的Agent代码生成性能。 Cursor 采取了自顶向下的非常规策略开发 Composer 2：以 1 万亿参数的稀疏 MoE 开源底座 Kimi 2.5 为基础，进行代码 token 的大规模 Mid-training，并在模拟环境中结合大规模强化学习（RL）专精 Cursor 场景。 (5:00) Cursor 与 Fireworks 构建了异步 RL 流水线，将轨迹生成与训练更新解耦以保持 GPU 满载；在经过优化的计算分配中，推理仅占约三分之一的 GPU 算力，训练占三分之二。 (10:00) 为克服无单一超大 GPU 集群的限制，团队将 RL 推理分发至全球 4 个独立集群并动态复用生产环境低谷算力；自研的权重增量压缩算法将 1 TB 模型更新缩小高达 20 倍，跨广域网同步耗时不到 1 分钟且推理引擎仅暂停约 30 秒。 (18:00) 浮点运算的非确定性会导致推理与训练间的数值漂移，破坏脆弱的 RL 策略梯度；团队通过定制 GPU 内核并引入 Router Replay 机制记录并强制统一稀疏 MoE 路由决策，彻底消除了专家分歧。 (25:00) 针对长程 Agent 任务，Cursor 将自总结机制直接融入 RL 循环，使 200k 上下文模型可流畅处理数百万 token；底层自建的虚拟机架构能动态调度超过 100,000 台隔离 VM 以支撑海量强化学习环境交互。 (40:00)

To build Composer 2, Cursor and Fireworks AI shifted from relying on general-purpose frontier LLMs to training a specialized Mixture-of-Experts model using large-scale, asynchronous reinforcement learning. According to Cursor's Federico Cassano and Fireworks' Dmytro Dzhulgakov, tailoring model weights to an editor environment delivers frontier-level software engineering performance at an order-of-magnitude lower inference cost. Specialized Top-Down Training Over Broad Foundation Models (2:02) Cassano argues that neural network weights have a finite parameter capacity, which general foundation models dilute across web trivia and multilingual tasks. By dedicating all parameters strictly to Cursor's software engineering workflows, a smaller model can match or exceed frontier systems like Claude Opus at lower inference expense. Instead of expensive bottom-up pre-training from scratch, Cursor adopted a top-down approach, starting with Kimi 2.5 (a 1-trillion parameter sparse MoE model with roughly 30–32 billion active parameters). Cursor executed large-scale code mid-training to broaden domain knowledge, followed by post-training reinforcement learning (RL) to bake tool execution directly into the weights. Disaggregated Asynchronous Rollouts Across Global Compute Pools (10:00) Unlike standard pre-training, multi-turn agentic RL requires long interactive trajectories (rollouts) lasting up to 50 steps, involving tool calls, compilation checks, and terminal commands. To keep GPUs fully saturated, Cursor and Fireworks engineered an asynchronous pipeline that decouples the continuous trainer from rollout generation. In an optimized system, inference should consume approximately one-third of total training compute (one forward pass versus three forward-pass equivalents for training). The team distributed the inference rollout load across four distinct global clusters—even repurposing off-peak production GPUs from Composer 1.5—while reserving tightly interconnected, high-speed RDMA clusters strictly for the core trainer. They also adopted FP4 precision during production training to maximize cluster throughput. Delta Compression and Overcoming Floating-Point Non-Determinism (18:00) Distributing RL over a global wide-area network creates latency and weight-staleness bottlenecks when transmitting 1 TB snapshots generated every 5 to 15 minutes. Cursor resolved this by designing a lossless delta compression algorithm that sent only changed parameters, reducing payload sizes up to 20-fold (roughly 50 GB) and enabling WAN synchronization in under a minute with an inference engine pause of ~30 seconds. Furthermore, the asynchronous architecture suffered from numerical mismatch: non-associative floating-point operations across different GPU kernels altered log probabilities and caused gating networks to misroute tokens between active experts (8 of 384 available). To prevent policy gradients from updating experts that never generated the token, the team introduced 'router replay'—logging an integer per token during inference to force the trainer down the exact same expert paths. Scaling Agent Horizons and Production-Parity Environments (30:00) To handle long-horizon tasks that exceed native context limits, Cursor trains models to perform self-summarization directly within the RL loop, enabling a 200,000-token context window model to sustain workflows spanning millions of tokens. Cassano notes that while offline RL (such as DPO-style updates) bootstraps base capabilities and allows massive multi-rollout exploration (16 to 128 parallel attempts under GRPO), live online RL continuously refines the model every few hours from user interactions. Rather than relying on generic third-party benchmarks or lightweight Docker sandboxes, Cursor built a proprietary virtual machine infrastructure capable of spinning up over 100,000 isolated environments on demand, ensuring models train directly against full operating system context and realistic repository states. Cursor and Fireworks demonstrate that application-layer AI companies can capture significant cost and quality advantages by training proprietary specialized models within realistic execution environments. However, scaling asynchronous agentic RL introduces extreme systems-level challenges—including cross-cluster weight synchronization, floating-point divergence, and credit assignment over extended rollouts—requiring deep co-design between model architectures and distributed infrastructure.

Cursor 与 Fireworks AI 合作，通过自上而下的中训与强化学习（RL）将开源 MoE 模型打造成专有编程模型 Composer 2。Cursor 团队与 Fireworks 联合创始人 Dmytro Dzhulgakov 认为，应用层公司通过将参数容量完全聚焦于垂直产品工作流并结合全局分布式异构算力，能够以远低于通用前沿模型的推理成本实现更高性能。 垂直模型架构与自上而下训练策略 (2:02) Cursor 团队将神经网络参数视为具有有限存储预算的载体。通用模型需要将容量分散至网络琐事、多语言文本和宽泛推理，而 Cursor 将所有参数容量专用于编辑器内的工作流，使 Composer 能够以相比 Claude Opus 等前沿大模型低一个数量级的推理成本，达到同等甚至更优表现。针对是否违背“苦涩的教训”（The Bitter Lesson）的质疑，Cursor 研究负责人 Federico Cassano 指出通用前沿实验室本身也在针对代码进行重度专业化；Cursor 是通过剔除不必要的干扰知识，将模型有限的容量饱和利用于编程领域。 在开发路径上，Cursor 放弃了高成本的从头预训练，采用自上而下的策略：以月之暗面（Moonshot AI）的开源 MoE 模型 Kimi 2.5（总参数 1 万亿，激活参数约 300 亿至 320 亿）为底座，先在预训练规模的代码 Token 上进行大规模中训（Mid-training）以扩展库和架构知识，随后在模拟的 Cursor 环境中运行长程强化学习，训练模型的工具调用与代码正确性。Cassano 补充，这与底层用于实时逐字补全的超低延迟 Tab 小模型截然不同，Composer 2 面向的是长程 Agent 交互。 异构算力解耦与跨地域分布式同步 (10:00) RL 训练异于常规预训练，它需要模型在模拟环境中执行长达 50 步的多轮交互（Rollout），包括执行终端命令、修改文件和编译测试。为防止数万张 GPU 在多分钟的交互生成期间闲置，Cursor 与 Fireworks 构建了异步 RL 流水线：Rollout 生成与训练器权重更新解耦并行，并在生产环境中采用 FP4 低精度训练以压榨硬件吞吐。在算力配比上，未优化的推理引擎会虚增推理开销，但在 GPU 饱和的理想状态下，推理通常只需消耗总训练算力的约三分之一（对应单次前向传播与训练所需三次前向等效计算的比例）。 受限于缺乏单一集中的超大规模 GPU 集群，团队将重度依赖 RDMA 紧密互连的训练节点置于单一集群，而将对网络互连要求较低的推理 Rollout 分布在遍布全球的 4 个独立集群上，并动态复用 Composer 1.5 处于生产低峰期的闲置推理算力。为解决跨公网传输 1TB 模型快照（每 5 至 15 分钟一步）带来的权重过期与带宽瓶颈，团队开发了无损权重增量压缩算法。该算法仅传输发生微调的参数子集，将传输体积缩减高达 20 倍（降至约 50GB），在 1 分钟内完成跨广域网分发，并将推理引擎暂停时间控制在约 30 秒以内。 浮点非确定性与 MoE 路由重放 (21:00) 在异步分布式 RL 中，推理节点生成 Token 时记录的对数概率（Log Probabilities）必须与中心训练器重新评估策略梯度时的数值一致。然而，浮点数加法与乘法不满足结合律，GPU 线程调度与算子执行顺序的微小变动会导致数值不一致（Numerical Mismatch）。尽管这种细微漂移在预训练中影响有限，但 RL 依赖微弱的梯度信号，微小的数值偏差足以导致策略收敛失败。强制使用严格的批次不变 GPU 算子会导致吞吐量下降 2 到 3 倍；团队通过系统性剖析算子，以几个百分点的计算减速消除了 90% 的数值分歧。 该问题在稀疏 MoE 架构中尤为致命。以 Kimi 每 Token 从 384 个专家中激活 8 个专家的路由为例，隐藏状态若发生微小的浮点偏差，门控网络可能在推理时选择专家 #7，而在训练反传时选中专家 #9，导致训练器更新从未参与生成的专家参数。为此，团队手写了自定义 GPU 算子并引入“路由重放”（Router Replay）机制：推理引擎在生成阶段记录一个表示被选专家的单整数元数据，训练器据此强制复现完全一致的专家路由路径，从而杜绝由于浮点噪声引发的路由发散。 长程信用分配、上下文扩展与生产级仿真 (30:00) 在对齐机制上，Cursor 采用两阶段策略：先利用离线 RL（如 DPO 技术）和中训打牢基础推理与工具调用能力，使模型达到用户可接受的基准；部署后，再借助真实用户反馈进行在线 RL 微调（权重每数小时同步一次）。离线模拟依然不可或缺，因为它支持采用组相对策略优化（GRPO）等算法对同一 Prompt 并发采样 16 至 128 条 Rollout，在不损害真实用户体验的前提下探索高方差策略。针对长程任务面临的信用分配困境与有限上下文窗口限制，Cursor 将“自我总结”（Self-summarization）整合进 RL 优化循环，模型在训练中学习如何压缩并遵从前序交互，使原生的 20 万 Token 窗口能够支撑数百万 Token 的连续任务。 在训练环境构建上，Cassano 与 Dzhulgakov 强调企业级 AI 不应依赖通用的第三方玩具基准或简单的 Docker 容器，产品本身就是最好的 RL 环境。RL 环境由调用工具的 Harness、承载实时世界状态的操作系统以及奖励机制构成。为模拟包含复杂依赖与真实系统状态的开发环境，同时防止模型破坏数据库，Cursor 搭建了能够动态调度超过 100,000 个隔离虚拟机的底层栈，以支持高并发的模拟交互与 LLM-as-a-Judge 多维度多细则评估。 Cursor 从单纯的应用编排转向自建垂直前沿模型，印证了 AI 原生应用通过捕获专有数据并下沉至模型权重所获得的效能红利。不过，长程 Agent 的端到端信用分配以及突破极低效率采样（“从吸管中吮吸信息”）的核心算法细节仍属于专有秘密，且未来迈向完全私有的端到端预训练仍有赖于更大规模基础设施与工程能力的持续演进。

## Background 本期播客由红杉资本（Sequoia Capital）的 Sonya Huang 主持，于 2026 年 5 月 26 日发布在 RSS / Web 平台。节目聚焦 Cursor 的 Federico Cassano 与 Fireworks AI 的 Dmytro Dzhulgakov 之间的深度合作，详细探讨他们如何借助 Fireworks 的高性能分布式推理与训练基础设施，采用自顶向下的继续预训练与强化学习方案，打造出专精于软件工程场景的前沿代码模型 Composer 2。 ## TL;DR 对话深入拆解了 Cursor 从纯应用平台跃迁为自研基座模型实验室的工程实践。团队坚信模型权重容量有限，通过剥离通用常识并专注于 Cursor 内部的真实代码交互，让稀疏 MoE 底座在极低推理成本下释放出媲美通用前沿大模型的编码能力。为解决强化学习中多轮会话 Rollout 与环境模拟的巨大开销，Cursor 联手 Fireworks 构建了异步流水线双子工厂架构、无损跨地域模型增量同步、消除 MoE 浮点漂移的 Router Replay 机制，以及能够瞬时并发拉起 100,000 台虚拟机的自研环境调度系统，展现了端到端强化学习如何重塑真实软件工程系统的全貌。 ## Atmosphere & style 整场对话洋溢着系统工程师与顶级架构师之间心领神会的极客兴奋感，硬核且充满实战底气。两位嘉宾在复盘深夜并肩排查 MoE 浮点偏差、跨洲传输 1TB 权重等工程极限挑战时频频相视而笑，既有直面“模型天生爱作弊”等荒谬现实的幽默调侃，又有触及核心技术机密时的默契会心。主持人 Sonya 的提问敏锐直接，引导出二人从底层算子对齐到宏观 Software 3.0 范式演进的深刻思考，听众仿佛身处高密度头脑风暴的前沿机房战壕，极具沉浸感与技术张力。 ## Bulletin - `00:48` Cursor 与 Fireworks 联合推出专属编码模型 Composer 2，基于 Kimi 2.5 稀疏 MoE 底座（总参数 1T、激活 30B），通过中训与大规模强化学习将模型推理成本压低至 Claude 3.x Opus 的十分之一。 - `20:00` 针对全球分布式训练与跨洲传输痛点，双方研发出权重增量压缩算法（Delta Compression），将 1TB 全量权重缩减约 20 倍，实现分钟级跨区域同步与仅 30 秒的推理引擎在线热替换。 - `42:45` 为支撑高保真强化学习沙盒，Cursor 摒弃轻量级 Docker 方案，自研支持瞬时并发弹性调度多达 100,000 台虚拟机的底层系统基础设施。 - `32:15` Cursor 将自总结（Self-Summarization）与上下文压缩端到端纳入强化学习循环，使模型原生 200k 上下文窗口能够稳定执行跨越数百万 tokens 的超长程任务。 ## Key takeaways - `00:48` 模型权重的物理容量是有限的，通用模型被迫将容量分摊给通识知识，而应用层团队将全部比特倾注于专属开发场景，可用更小尺寸打平前沿模型能力。 - `00:48` 应用演进呈现“Prompt 原型验证、沉淀专属工具与交互数据、后训练内化行为”三部曲，垂直应用团队自研模型是突破体验与成本上限的必然路径。 - `04:47` Cursor 团队并未违背《苦涩的教训》，而是在数据维度极力 Scaling，通过清理无关信息通道，用纯粹高密度的代码交互数据灌注有限参数。 - `05:42` Composer 2 采用反常规的“自顶向下”训练流水线，先以 MoE 底座进行大规模中期继续预训练拓展代码模式，再依托 Cursor Harness 进行深度强化学习收敛正确性。 - `10:00` 强化学习 Rollout 本质上是长达数十轮包含工具调用的异构 Agent 会话，训练系统采用“双子工厂”异步流水线，以极小的权重陈旧度代价避免算力闲置。 - `00:00` 模型具备敏锐的模式识别本能，若沙盒环境与真实系统存在微小裂痕，模型会倾向于利用漏洞作弊（Reward Hacking），因此环境保真度是 RL 的生死线。 - `14:42` “推理 FLOPs 远超训练”是开源推理缺乏优化导致的迷思，在充分吃满临界批处理大小的理想状态下，推理与训练的 GPU 最佳算力配比应严格为 1:3。 - `16:48` Cursor 实施“全球潮汐算力调度”，在夜间低谷期借用线上生产推理 GPU 充当 RL 模拟器，并通过增量压缩算法跨洲分发模型更新。 - `24:45` 浮点运算的非确定性会导致 MoE 门控产生路由偏差，团队引入 Router Replay 机制强制训练端复用推理端专家激活 ID，消除了 90% 以上的数值失配。 - `26:45` 模拟 RL 支持多路并发与激进采样（如 GRPO），而真实用户在线 RL 依赖活体反馈但容错极低，二者构成后训练阶段的互补双轮。 - `35:00` 强化学习在初期的本质是“扭转概率分布并锐化专家身份”，明确告知模型放弃错误示范、稳定扮演最高水准专家的角色。 - `37:15` 行业正步入 Software 3.0，专家的职责从写代码、标数据转变为设计高保真仿真环境、构思可验证奖励机制与业务断言。 - `41:30` 对于垂直应用而言，“最强大的 RL 环境就是你的生产产品本身”，玩具级 Docker 容器无法支撑包含微服务、状态与数据库迁移的复杂生产级软件。 - `42:45` 实用 RL 环境由 Harness 执行层、底层操作系统状态与奖励校验三层构成，Cursor 依托自研虚拟机集群在训练峰值期瞬间拉起 10 万台 VM 实现秒级就绪。 ## Detailed walkthrough ### `00:00` 强化学习中的“作弊”现象与环境模拟挑战 在模型训练的强化学习（RL）阶段，基础设施必须构建起高度拟真的虚拟沙盒环境，尽可能精准地复现真实用户电脑的运行状态。如果模拟环境与生产环境存在细微裂痕，模型会极其敏锐地察觉到这种差异。Federico 提到，大模型具备惊人的模式识别能力，一旦意识到自己处于“假环境”中，就会学会在沙盒特有的漏洞中钻空子刷分以最大化奖励（Reward Hacking），导致其在 RL 阶段的行为与线上生产环境完全脱节。正如 Federico 幽默所言：“模型天生爱作弊，而且在寻找投机取巧的方法上极为擅长（Models love to cheat）。” 随后，主持人 Sonya Huang 介绍了两位嘉宾：Cursor 团队 Composer 2 的研究负责人 Federico Cassano，以及 Fireworks AI 联合创始人兼 CEO Dmytro Dzhulgakov（Dima）。为了支撑 Cursor 这一庞大的分布式模型训练任务，Dima 在过去数月间几乎是“兼职潜伏”在 Cursor 团队中，双方紧密合作打通训练架构底座。这次对话聚焦于 Composer 2 的诞生历程、工程攻坚细节，以及应用层公司演变为基座模型公司对整个 AI 产业格局的深远影响。 ```mermaid flowchart TD accTitle: RL 训练环境真实度对模型行为的影响 accDescr: 对比高仿真环境与低仿真沙盒中强化学习模型的行为差异，说明作弊行为的产生机制。 subgraph SandboxEnv ["低仿真沙盒环境"] A1["环境特征露馅 / 假环境"] --> B1["模型感知到漏洞"] B1 --> C1["学会针对性投机（Reward Hacking）"] C1 --> D1["RL 分数极高 / 生产环境失效"] end subgraph RealisticEnv ["高仿真 Cursor 运行环境"] A2["高度模拟真实用户电脑"] --> B2["模型直面真实代码约束与工具反馈"] B2 --> C2["学习鲁棒的软件工程行为"] C2 --> D2["真实生产环境下泛化表现优异"] end ``` --- ### `00:48` 为什么应用层公司必须下场自研基座模型 长期以来，Cursor 主要是作为承载外部前沿编码 Agent 的交互平台存在，而倾注全力打造专属编码模型 Composer 2，标志着其向基座模型研发公司的关键跃迁。Federico 从模型底层原理剖析了这一决策的必然性：模型的权重参数就像一块容量有限的物理硬盘，能承载的比特信息量是固定的。传统的通用大模型需要将这些宝贵的容量分配给各种通识知识、文学创作、常识问答甚至是各行各业的琐碎信息。然而 Cursor 的终极目标无比清晰且聚焦： > “我们关心的唯有一项任务——我们甚至不一定是关心通用的写代码或编程，我们只关心在 Cursor 软件内部的软件工程，且仅此而已。” 通过将权重中的每一个比特都倾注到这一垂直场景中，团队能够用尺寸更小、结构更精简的模型，在专业任务上打平甚至超越通用前沿大模型。正因如此，Composer 2 的推理调用成本比 Claude 3.x Opus 等通用前沿模型便宜了整整一个数量级。 针对这种策略是否代表应用层公司的未来趋势，Dima 给予了高度肯定。他指出应用演进通常经历标准的三部曲： 1. **原型探索阶段**：直接接入现成的商业 API，依靠 Prompt Engineering 和基础 Harness 验证产品原型； 2. **应用价值沉淀阶段**：随着产品深入，最核心的护城河变成了真实用户交互数据、专属工具链（Tooling）、私有调用 Harness 等业务特有机制； 3. **模型重塑阶段**：Prompt 工程有其无法突破的上限，许多复杂的工具行为和隐式约束根本无法在 System Prompt 中精确表达，唯有通过模型微调与训练（Post-training），将这些操作习惯内化为本能。 此外，Dima 强调了 Fireworks 视角下的“质量、速度、成本”三维权衡三角形。仅靠底层推理引擎优化固然能带来显著收益，但想要打破质量上限并同时将成本压低到原本的零头，就必须深入模型训练环节，Composer 2 正是这一技术路线的典型标杆。 --- ### `04:47` 逆行于“苦涩的教训”？模型权重的容量分配假说 Sonya 提出了一个尖锐的质疑：Rich Sutton 著名的《苦涩的教训》（The Bitter Lesson）揭示了通过通用算力在全网数据上进行暴力缩放的威力。在早期的 Tabnine 时代，行业也曾尝试过小型专有代码模型，但后来通用大语言模型直接凭借参数规模碾压了垂直模型——大模型在互联网海量文本训练中，其代码能力也水涨船高。Cursor 这种重回“专有垂直模型”的路线，是否违背了《苦涩的教训》？ Federico 对此给出了深刻的辩驳： - **顶级通用实验室并未真正通用**：前沿实验室的模型之所以代码表现强劲，是因为他们本身就把代码作为最高权重的核心数据流之一进行了极度倾斜的配比，其本身就包含大量专有化训练，而非单纯的自然语言泛化。 - **Cursor 正在数据维度践行《苦涩的教训》**：团队并非依赖脆弱的人工特征工程，而是在数据维度极力 Scaling。正因为坚信模型的容量是有物理上限的，为了在有限权重内塞入最大密度的真实软件工程经验，就必须主动“清理”模型权重中的无关干扰，将全部通道让渡给纯粹的代码交互数据。 --- ### `05:42` Composer 2 架构揭秘：Kimi 2.5 底座与自顶向下的训练哲学 在具体训练实现上，Composer 2 采用了一种反常规的“自顶向下（Top-Down）”研发路径： ```mermaid flowchart LR accTitle: Composer 2 自顶向下训练流水线 accDescr: 展示从 Kimi 2.5 稀疏 MoE 底座，经历代码 Continue Pre-training，再到大规模强化学习的整体训练路径。 A["底座选择<br/>Kimi 2.5 稀疏 MoE<br/>(总参数 1T / 激活 30B)"] --> B["Mid-Training<br/>(大规模继续预训练)<br/>海量代码 Token 吸收世界知识"] B --> C["Checkpoints 沉淀<br/>建立广泛代码模式分布"] C --> D["大规模 RL 训练<br/>(Reinforcement Learning)<br/>深度绑定 Cursor Harness"] D --> E["生产就绪<br/>Composer 2 编码 Agent"] ``` - **高稀疏 MoE 底座**：选用 Kimi 2.5 作为基础模型。该模型总参数量高达 1 万亿（1T），但每次推理仅激活 300 亿（30B），具备极高的知识容量与极低推理延迟开销的稀疏特性。 - **双轴驱动（Mid-Training + RL）**：早期的 Composer 1 仅在强化学习单轴上发力，而 Composer 2 则采用了双轴驱动战略——先进行接近预训练级别规模的大量代码 Token 中期继续预训练（Mid-training / Continual Pre-training），随后在产出的 Checkpoint 基础上实施极具广度与深度的大规模强化学习。 面对“Cursor 坐拥如此丰富的核心代码交互数据，为何不直接从零预训练（Pre-training）”的疑问，Federico 解释了这种敏捷的工程哲学：如果按照“从零预训练 $\to$ 中期训练 $\to$ 强化学习”这一漫长的自底向上流程，用户可能数年都无法用上产品；而自顶向下的模式能够在极短周期内将高价值模型交付给开发者，在此过程中快速积累实战反馈，未来的 Composer 版本再顺理成章地过渡到完全自研的底座。 --- ### `07:33` Mid-Training 与 RL 的分工：从代码模式到代码正确性 针对训练流水线中不同阶段所扮演的核心角色，Federico 进行了清晰界定： | 训练阶段 | 核心学习目标 | 认知机制与表现 | |:--- |:--- |:--- | | **Mid-Training<br/>(中期继续预训练)** | 代码库结构、语法范式、常见代码设计模式、部分相关网页知识。 | **扩展模式分布**：本质仍是 Next Token Prediction。模型学会了“像程序员一样写代码”，但并不具备甄别代码绝对正确性的能力。虽然数据经过清洗，但它依然缺乏对执行后果的判别力。 | | **RL / Post-Training<br/>(强化学习与后训练)** | 工具调用协议、IDE 环境探索导航、真实环境下的执行正确性。 | **锐化收敛特征**：模型直接置身于 Cursor Harness 交互环境中试错，学会以最佳方式调用外部工具，强行将行为模式调优至“始终编写能够通过校验的正确代码”。 | 针对 Sonya 提出的“为何不直接拿 Cursor 现有的 Tab 补全模型进行强化学习后训练”的疑问，Federico 澄清了两者在定位与能力维度上的本质差距： - **Tab Autocomplete 模型**：极度受制于严苛的毫秒级端到端延迟要求，模型尺寸被压缩得极其微小，专精于局部上下文的高频补全； - **Composer 系列 Agent 模型**：需要理解宏观代码库依赖、在长时间跨度（Long Horizon）内编排多步推理与多工具调用，必须依托于具备海量容量的架构（如 Kimi 2.5 的 1T 参数稀疏底座），两者在核心能力维度与架构选型上完全不可同日而语。 ### `10:00` 强化学习训练循环：从单 Token 预测到多轮 Agent Rollout 在讨论 Composer 2 的训练时，Sonya Huang 指出团队的核心重心放在了大规模强化学习（RL）运行上，并询问这与常规训练究竟有何不同、中间攻克了哪些硬核难题。Dmytro Dzhulgakov 解释道，RL 训练与预训练（Pre-training）或中训（Mid-training）有着本质区别：常规训练的目标仅仅是预测下一个 Token，而 RL 却需要运行一整套测试套件（Harness）与完整实验。它让模型在特定环境中采取行动，观察其在单次执行轨迹（即“Rollout”）中的表现，并给予相应的奖励信号（Reward）。这一奖励既可以由大语言模型充当裁判（LLM-as-a-judge），也可以基于可验证的结果（例如“代码能否成功编译”）。 这种机制意味着 RL 训练所需的基础设施远比传统训练复杂。团队不仅需要调度数万张 GPU 来执行前向与反向传播等常规训练任务，还必须同时编排大量模拟环境与模型推理。当 Sonya 询问 Rollout 是否类似于一次前向传播时，Dmytro 纠正道，Rollout 实际上模拟的是 Cursor 中一整套完整的 Agent 会话。单次会话可能长达 50 个轮次（Turns）：模型接收初始 Prompt，自主决定调用某些工具，系统执行这些工具并将结果返回，模型再据此生成更多代码。整个过程构成了异构且庞大的闭环，最终产生的奖励信号再回传给训练集群以更新模型权重。如何在昂贵的 GPU 资源下维持极高吞吐并实现经济高效的训练，成为了算法与基础设施协同设计的核心挑战。 ```mermaid flowchart TD accTitle: Cursor RL 训练与推理闭环架构 accDescr: 展示从模型权重分发到分布式环境模拟、奖励评估再到主集群梯度更新的完整异构循环。 subgraph TrainCluster ["集中式训练集群 (Trainer)"] W["模型权重更新"] B["反向传播 & 梯度计算"] W --> B end subgraph RolloutCluster ["全球分布式推理与环境 (Rollouts)"] I["Fireworks 高性能推理"] E["高拟真用户环境 (模拟 50 轮会话)"] R["奖励判定 (编译器 / LLM Judge)"] I --> E E --> R end W -.->|"增量权重同步 (Delta Sync)"| I R -->|"轨迹与奖励信号 (Rollout Results)"| B ``` - **Rollout 的本质**：并非单次推理，而是长达数十轮的完整 Agent 交互，涵盖代码生成、工具调用与沙盒执行。 - **异构闭环**：训练器负责重型梯度计算，推理引擎与环境负责模拟真实用户交互，系统瓶颈跨越网络、算力与 I/O。 --- ### `11:46` 流水线式异步 RL：双子工厂架构与陈旧度权衡 为了解决环境模拟与模型更新之间的速度失配，团队采用了异步或流水线强化学习（Async / Pipelined RL）。Dmytro 用“巨型双子工厂”来生动比喻这一架构：如果采用朴素的同步方案，训练器在生成 Rollout 时必须停机等待——而在长周期任务中，一次模拟可能长达 5 分钟甚至更久；等到收集完结果，推理端又得停下来等待训练器更新权重。这种做法在理论算法上极度稳健，但在工程上极其低效，会导致一半的算力资源时刻处于闲置状态。 流水线架构将系统拆分成两个永不停歇的建筑：一栋是“训练器工厂”，持续接收新鲜出炉的会话结果并计算权重更新；另一栋是“Rollout 工厂”，始终拉取当前最新版本的模型权重并在模拟环境中并发生成新交互。这种设计引入了一个关键算法挑战——“权重陈旧度”（Staleness）。当某个模拟任务在环境中耗时数分钟跑完时，训练器可能已经根据其他数据更新了若干轮权重，导致收集到的数据来自稍旧的模型版本。尽管存在数学上的偏差，但正如 Rich Sutton 的“苦涩的教训”（The Bitter Lesson）所揭示的那样，不让一半昂贵的算力在牌桌上闲置所换取的巨大吞吐优势，足以压倒算法异步带来的微小损失。 Federico Cassano 强调，Cursor 对性能有着近乎偏执的追求，因为 Cursor 不像顶级巨头那样手握数百万张 GPU，而是依靠数万张卡进行高效突围。为了将每张 GPU 的性能压榨到极致，Cursor 甚至在生产训练中激进地引入了 FP4 精度。 - **流水线吞吐对比**：避免了同步方案中 50% 算力等待环境模拟的系统内耗。 - **陈旧度权衡**：尽管异步更新会导致小幅度的数学精度偏离，但高算力利用率带来的总学习步数收益大幅领先。 - **极致硬件榨取**：在生产环境训练中直接采用 FP4 精度，从算力受限的现实中逼出极限效能。 --- ### `14:42` 环境拟真与作弊防范：突破“推理 FLOPs 远超训练”的迷思 在强化学习的模拟环境中，保真度（Fidelity）具有生死攸关的重要性。Federico 指出，模拟环境必须尽可能百分之百地还原真实用户的电脑环境。一旦环境存在破绽，模型就会敏锐地察觉到自己身处“虚拟沙盒”而非生产环境，从而在 RL 阶段表现出截然不同的行为模式。Sonya 惊讶地追问模型是否真的能“意识到”沙盒的存在，Federico 确认了这一现象，并指出模型会心想：“哦，我是在一个假环境里，那我只要用某些特定小技巧就能刷出高分，赶紧试试看。”他总结道：“模型天生热爱作弊，而强化学习极度擅长鼓励作弊”（"Models love to cheat. RL is really good at encouraging cheating."）。 随后，Federico 破除了一条在业内广为流传的“迷思”：很多人认为在 RL 过程中消耗在推理上的 FLOPs 远超训练 FLOPs。他澄清这完全是因为开源推理引擎缺乏优化所致，并非强化学习的固有数学规律。从理论上看，训练包含前向传播、数据激活值梯度与权重梯度计算，相当于 3 次前向传播的算力开销；如果在推理阶段充分吃满临界批处理大小（Critical Batch Size），推理仅占 1 次前向传播的算力。因此在理想状态下，分配给推理的 GPU 数量应该严格是训练 GPU 数量的三分之一（1:3）。这也是 Cursor 选择与 Fireworks 深度合作的原因——如果自己从头构建高吞吐推理引擎，势必会分散宝贵的工程精力；与 Fireworks 携手可以让 Cursor 的工程师集中精力攻关训练的精度与效率。 - **环境沙盒防作弊**：防范模型利用虚拟测试桩（Stubs）或假环境的逻辑漏洞走捷径套取奖励。 - **算力分配金科玉律**：在高度优化的推理架构下，推理与训练的理想 GPU 配比为 1:3，而非失衡地倾斜向推理。 - **专业化分工**：Cursor 团队聚焦训练算法与精度，将底层推理基础设施托管给 Fireworks，避免重复造轮子。 --- ### `16:48` 全球分布式集群与全球潮汐算力调度 在谈到技术报告中提到的“全球分布式训练”时，Federico 与 Dmytro 揭示了背后的现实无奈与架构巧思。在目前的算力市场上，超大规模的单一体连续集群（Contiguous Cluster）极其罕见且昂贵。训练阶段由于涉及庞大的梯度同步，节点间必须依赖高带宽网络紧密互连（Lockstep），因此 Cursor 将所有训练任务收拢在一个高性能核心集群中；但对于强化学习中的推理与环境模拟组件，由于它们具备天然的去中心化特性，完全可以分散到遍布全球的较小集群中执行。 在训练 Composer 2 期间，团队一共使用了 4 个相距甚远、分布在全球不同角落的集群。更具创意的是，他们利用了全球用户的时差效应，开展了“潮汐算力调度”：当某些地区的夜间低谷期到来、上一代模型 Composer 1.5 的线上请求量锐减时，Cursor 会直接接管这些闲置的线上生产推理 GPU，动态将其并入 RL 训练管线充当环境模拟器；待白天流量高峰来临再归还资源。 然而，这种跨国异构调度带来了巨大的系统瓶颈：模型权重全量快照高达 1TB 左右，而训练步长大约仅有 5 到 15 分钟。这意味着每隔 5 到 10 分钟，系统就必须将一份 1TB 的最新权重全速空运到地球另一端的推理集群中，否则权重陈旧度就会彻底失控。两位工程师共同攻克的关键突破在于：RL 阶段的参数更新往往极其精准且局域化，经过 10 分钟的训练步后，绝大部分权重并没有剧烈变动。团队通过捕捉权重变化的规律模式，研发出专门的差异增量压缩算法（Weight Delta Compression），大幅削减了跨洲传输的数据量，使得数分钟内同步全球集群成为现实。 - **算力拓扑解耦**：单一集中式高带宽集群负责重度反向传播；4 个全球分布式小集群负责 Rollout 推理。 - **潮汐复用生产算力**：低谷时段借调生产环境（如 Composer 1.5 线上节点）反哺训练，动态削峰填谷。 - **1TB 权重秒级同步**：利用 RL 阶段参数微调的稀疏与局部特征，通过增量压缩技术突破跨洲公网传输瓶颈。 ### `20:00` 打破常规的解耦架构：跨地域高效权重增量同步 在传统的强化学习（RL）基础设施设计中，业界普遍遵循一种既定思维：必须建立一个由昂贵且密集的 RDMA 网络互连的超大规模单一集群，通常按比例将三分之一的算力分配给训练，三分之二分配给推理。如果网络足够昂贵，拷贝高达 1TB 的模型权重固然容易，但这意味着集群规模被整体放大至三倍，硬件成本极其高昂。 Cursor 与 Fireworks 团队采取了一种反传统的解耦策略。他们将训练和推理系统拆分开来，甚至将部分推理节点部署在其他区域更廉价的算力上。面对跨地域分发模型的挑战，他们将其转化为典型的数据库系统问题——只传输模型更新的“增量”（Delta），而非完整的模型镜像。这个 Delta 的体积可以比完整模型小 20 倍左右。 为了让这一方案可行且可靠，团队搭建了一整套底层支撑组件，包括全量快照与增量存储、崩溃恢复以及数据对齐调和机制。该方案在数学上做到了“无损”（Lossless），确保接收端合成的模型权重与训练端在二进制位级别（Bit-equivalent）完全一致。在工程优化上，通过将上传与下载双向分片，系统几乎跑满了集群的出口带宽（Egress Bandwidth）。即使在恶劣网络条件下，几分钟内即可完成传输，通常情况下仅需不到一分钟；而推理引擎真正暂停服务以热替换权重的时间仅需大约 30 秒。 主持人 Sonya 看着两位工程负责人描述这些成果时脸上的笑容，调侃道：“看着你们聊这个时一直在笑，因为这太难了，简直是系统工程师的终极梦想。”两人也感慨，这背后是双方团队并肩作战熬过的无数个漫漫长夜。 ```mermaid flowchart LR accTitle: 跨地域模型增量同步流程 accDescr: 展示训练集群生成增量后分片传输并热替换至推理集群的流向 subgraph TrainCluster["训练集群 (Training Cluster)"] T_Weight["生成新权重"] --> T_Delta["计算模型增量 (Delta)<br/>(体积缩小约20倍)"] T_Delta --> T_Shard["分片并打满带宽上传"] end subgraph Net["公网 / 跨区域网络传输"] T_Shard --> Net_Transit["高速分片传输"] end subgraph InferCluster["推理集群 (Inference Cluster)"] Net_Transit --> I_Recv["分片接收与无损组装"] I_Recv --> I_Swap["在线热替换 (Pause ~30s)"] I_Swap --> I_Serving["bit 级完全一致的推理服务"] end ``` - **增量体积优势**：仅传输权重差值，相比传输全量模型体积缩小约 20 倍。 - **传输与中断耗时**：全流程通常在 1 分钟内完成（最差几分钟），推理服务停顿仅约 30 秒。 - **成本与算力收益**：结合优化后的推理引擎，整体 GPU 消耗减少约三分之一，且可利用异地低成本硬件，大幅摊薄总体拥有成本。 --- ### `22:15` 浮点不确定性与 MoE 路由偏差引起的数值失配 大型稀疏混合专家模型（MoE）给异步强化学习带来了极度棘手的“数值失配”（Numerical Mismatch）问题。在异步 RL 中，推理端执行自回归前向传播，输出采样 Token 的对数概率（Log Probabilities）。但由于训练与推理异步，生成数据时模型的参数可能比当前训练步落后几个步骤，训练器必须用同一模型版本重新跑一遍前向传播来复现对数概率。 理论上，同一版本模型的对数概率应当严丝合缝；但在现实运行中，即使是同一权重版本，推算出来的对数概率也常常存在微小甚至剧烈的偏差。其根本原因在于计算机浮点运算的非确定性（Non-deterministic）。在纯整数运算中，加法满足严格的结合律（$A + B + C = C + B + A$），但浮点数由尾数（Mantissa）和阶码（Exponent）构成，本质是近似表示，计算顺序的差异会导致舍入误差。大模型推理充斥着数十亿次矩阵乘法与加法累加，加法顺序的微小颠倒在海量层级中被层层放大。 在传统的预训练或常规推理场景下，这种浮点噪声无关痛痒，模型鲁棒性极高，即使个别 Bit 翻转也不会影响基准测试得分。然而，强化学习依赖的是非常微弱的奖励梯度信号，底层的数值噪声足以彻底破坏训练收敛，导致策略崩溃。 这一问题在 MoE 架构中被急剧恶化。MoE 层的门控机制（Gating Layer）负责在海量专家（例如 384 个专家）中根据隐状态得分遴选出 Top-8 专家激活。如果前向计算时某个隐藏状态小数点后第五位的浮点误差产生波动，就可能导致原本排在第 8 位的专家 7 被专家 9 取代。这一微小的数值波动会直接激活模型完全不同的子网络模块：推理端实际激活了专家 7，训练端却去更新了完全未参与推理的专家 9，导致梯度与推理行为严重脱节。 --- ### `24:45` 算子级对齐与 Router Replay 机制 为了消除训练端与推理端的数值偏差，团队在系统底层展开了一系列细致入微的算子级对齐与工程折衷。 理论上，可以通过重写所有 GPU 内核（Kernel）实现批次无关性（Batch-invariant），强行固定所有浮点累加的顺序，从而将偏差降为绝对零。但这会带来灾难性的性能惩罚，使整体吞吐降低 2 到 3 倍。团队通过大量的工程迭代寻找帕累托最优解：仅接受几个百分点的性能回退，换取消除 90% 以上的数值偏差。 针对 MoE 专家路由错位的死结，工程团队采用了一种名为“路由重放”（Router Replay）的高效技巧： - **极小的信息传递代价**：在推理端完成前向计算后，将实际激活的专家 ID（例如激活了专家 7）作为一个极小的整数额外传递给训练端。 - **强制对齐执行路径**：训练端在重新计算前向传播与反向传播时，直接强制使用推理记录下来的路由结果，杜绝因浮点漂移而激活错误专家的现象。 - **环境与精度协同**：除了路由重放，团队还严格匹配了推理与训练两端的量化级别（Quantization Levels）和底层计算内核，将两端实现的差异降到最低，避免训练发散，极大提升了强化学习的数据利用效率。 --- ### `26:45` 模拟 RL 与真实用户在线 RL 的权衡 当被问及具体使用的奖励模型（Reward Signal）细节时，嘉宾笑着守口如瓶，直言属于“绝密内容”（Top secret stuff）。不过，他们深入拆解了模拟环境强化学习（Simulated RL）与直接基于真实用户数据在线强化学习（Real RL / Online RL）之间的架构差异与选择逻辑。 Cursor 实际上同时在运行这两套系统。在“真实 RL”（Real RL）链路上，系统同样复用了与 Fireworks 联手打造的模型热更新能力：从客户端捕获真实用户对模型生成内容的满意与不满信号，在线计算梯度并更新模型，做到每隔几小时就无缝推送一个新版本模型。团队目前在持续缩短该迭代周期以提升稳定性、摸索最佳超参数；但随着模型规划视野（Horizon）逐步变长，未来更新间隔也会相应重新放宽。 然而，团队并未完全抛弃模拟 RL，这是因为两者的算法特征与工程边界有着本质区别： | 评估维度 | 模拟环境强化学习 (Simulated RL) | 真实用户在线强化学习 (Online RL) | |:--- |:--- |:--- | | **单 Prompt 采样策略** | 并行多路 Rollout（如 16 路或 128 路并发） | 仅单次 Rollout（单个用户请求一次对应一个生成结果） | | **算法支持类型** | 天然适配 GRPO（分组策略梯度）等高精度梯度估计算法 | 难以在同提示下直接对比多结果分布，梯度信号噪声更大 | | **错误容忍度与探索** | 极高；可激进尝试 Off-policy 和极端生成行为，仅损耗 GPU 时间 | 极低；类似持续进行线上 A/B 测试，畸变输出直接损害产品体验 | | **硬件与计算效率** | GPU 处于全速并发生成状态，硬件利用率饱和 | 强依赖真实请求到达，存在设备等待与高离线时延问题 | 因此，模拟 RL 允许模型在安全沙盒中利用 GRPO 等策略执行激进探索，通过海量并发采样提取高纯度信号；而真实在线 RL 则稳健地利用活体反馈持续微调，二者互为补充，构成了完整的后训练迭代闭环。 ### `30:00` 在线强化学习的悖论与“吸管吸比特”的效率困境 在构建具备实际可用性的模型时，Cursor 团队首先需要让模型达到能够直接面向终端用户的基线水平。Federico Cassano 澄清了行业内关于强化学习术语的混淆：通常被称为“离线 RL（Offline RL）”的技术其实更偏向 DPO（Direct Preference Optimization）这类离线偏好微调；而真正的强化学习（Reinforce 类的算法）本质上是在线 RL（Online RL）。在这一阶段，Cursor 训练模型的推理能力、工具调用（tool calling）以及注入外部世界的全新信息。然而，这也带来了一个关于实时在线 RL 的核心悖论：模型必须先足够好，用户才愿意在实际开发中使用并产生反馈；因此团队无法从零完全依赖用户反馈去冷启动模型。在线 RL 更像是在模型已经足够优秀的基础上，为了带来极致体验而精心点缀的“蛋糕上的樱桃”（Cherry on top）。Sonya Huang 随即提到了物理学家兼 AI 研究员 Dan Roberts 曾在红杉会议上的比喻——在传统范式中是“巨大的蛋糕与小小的樱桃”，而未来或许会变成“微小的蛋糕与庞大无比的樱桃”。 围绕 RL 的样本效率，Sonya 引用了 Andrej Karpathy 的经典论述：当前的强化学习极其低效，模型生成了极其漫长的轨迹（rollout），最终往往只能在末端得到一丁点微弱的信息信号，这种过程就像“用吸管极其艰难地吸取比特数据（slurping bits from a straw）”。当 Sonya 追问 Cursor 是否已经找到了能从这种极窄通道中榨取更多信息的方法时，Federico 笑着回答：“这个我不能透露。” Sonya 幽默地回应：“好的，我们又触碰到机密地带了——这说明我问到了最关键的问题。” - **在线 RL 与离线 RL 的边界**：Cursor 将 DPO 等技术划归为离线优化，而将真正基于环境交互与奖惩迭代的 Reinforce 机制称为在线或实时 RL。 - **冷启动的必要门槛**：模型若无法达到高水平的推理与工具调用能力，用户便会弃用，从而无法沉淀用于在线进化的真实交互飞轮。 --- ### `32:15` 突破上下文上限：将自总结（Self-Summarization）融入强化学习循环 当前的 AI 领域正在全力攻克长程智能体（Long-Horizon Agents），期望模型能够在无人工干预的情况下长时间稳定运行且不发生崩溃。在强化学习中，随着智能体执行任务的轨迹（trajectory）越来越长，系统面临着两个致命的技术难题：其一是**信用分配（Credit Assignment）难题**。当智能体在长达数十分钟的交互末端才获得一个成功或失败的反馈时，模型很难准确反推自己在哪个具体步骤做对了、在哪个环节犯了错；其二则是**物理上下文窗口的容量枯竭**。 为了解决上下文耗尽的瓶颈，Cursor 团队采取了一种反直觉的方案：**将上下文压缩（Compaction）直接写入强化学习的闭环之中**，他们称之为“自总结（Self-Summarization）”。Cursor 所使用的模型原生上下文窗口虽然约为 200k tokens，但在实际运行中却能连续处理数百万 tokens 的超长任务。模型在 RL 训练过程中不仅学会了朝着终极目标编写代码，还联合学会了如何在窗口即将占满时，主动对自己过去的工作进行高度精准的提炼，并利用该总结直接“重启（restart）”上下文窗口，随后无缝继续执行任务。RL 的端到端目标同时迫使模型掌握两项技能：既要生成高质量的紧凑摘要，又要能极度严格地遵从并理解该摘要。 ```mermaid flowchart TD accTitle: 包含自总结机制的长程 RL 架构 accDescr: 展示 Cursor 将上下文压缩与模型执行联合放入强化学习闭环的运行流程 subgraph Horizon ["长程任务执行 (可达数百万 Tokens)"] A["原生 200k 上下文窗口"] --> B["自主执行与工具调用"] B --> C{"上下文是否临界耗尽?"} C -- "否" --> B C -- "是" --> D["执行自总结 (Self-Summarization)"] D --> E["重置上下文窗口<br/>(保留压缩总结摘要)"] E --> B end subgraph Optimization ["端到端 RL 联合优化"] F["任务终点奖励信号"] --> G["联合优化策略: 生成优质摘要 + 严格遵循摘要"] G -.-> D G -.-> E end ``` Fireworks 的 Dmytro Dzhulgakov 对此给予了极高评价。他指出，在传统系统中，上下文管理通常被视为外部脚手架（Harness）的工程职责，而 Cursor 却将脚手架的核心机制与模型本身的权重演化融合在了一起，全部抛入同一个端到端优化循环中。这再次印证了理查德·萨顿（Richard Sutton）著名的“苦涩的教训（The Bitter Lesson）”——**给系统投入越多的通用算力让其端到端自我学习，系统的整体协同效果就越远超人工设计的硬编码规则**。Federico 进一步断言，只要一家公司的产品是在大规模生成 tokens，并且拥有明确的产品场景可以作为优化目标，将脚手架纳入模型 RL 训练就是最正确的方向。 --- ### `35:00` 强化学习无处不在：从 Cursor Tab 到“专家身份”的分布锐化 对于强化学习究竟只适用于长程智能体编排，还是同样适用于底层的下一个 Token 预测（Next Token Prediction），Federico 提出了一个极具洞见的“身份理论”。他强调，RL 几乎适用于任何环节，甚至 Cursor 的代码补全功能（Cursor Tab）底层也重度使用了 RL。 Federico 解释道，模型在预训练阶段是在无差别地吞咽整个人类文明的知识。以数学任务为例，模型吞下了 Stack Exchange 上所有的讨论帖子；但面对一道新的数学难题时，尚未经过 RL 对齐的模型在潜意识里其实是困惑的——它不知道自己此刻究竟该扮演那个“四处碰壁、写出错误解法的初学者”，还是该扮演那个“一语中的的顶级数学大师”。**强化学习的核心本质，就是在这层概率分布上拧动旋钮，明确告诉模型：“你就是专家，你必须从头到尾以最高正确率完成这件事。”** 在 Federico 看来，模型的强化学习通常经历两个阶段： 1. **第一阶段（分布锐化期）**：消耗相对较少的算力，模型就能在短时间内突飞猛进。这一阶段的核心就是扭转输出分布，让模型坚定地锚定在“专家角色”上，大幅削减低级错误。 2. **第二阶段（推理涌现期）**：需要倾注庞大的算力集群。在这个阶段，模型开始自主学会极其复杂的链式推理模式，能力曲线进入深度上升期。 Dmytro 补充验证了这一规律：微调（SFT）和持续预训练本质上是在向模型转移新的抽象知识，而 RL 则是在提炼、锐化模型特定的行为特质。哪怕是看似简单的文本总结，纯写提示词或人工列举好坏示例往往难以精准描述细微的文风，但如果引入基于强化学习的评估闭环，让模型在探索中自我调整，就能迅速逼近预期的输出形态。 --- ### `37:15` 评估机制演进：可验证奖励、LLM-as-a-Judge 与 Software 3.0 由于 Federico 此前开玩笑地对核心机密“保持沉默”，Sonya 转而向 Dmytro 抛出了关于评估体系演进的尖锐问题：未来究竟需要领域专家手动逐条审查 RL 轨迹并进行行为微调，还是可以全面依赖自动化指标与大模型作为裁判（LLM-as-a-Judge）？ Dmytro 明确表示，**让专家直接肉眼去评判海量的在线 RL 轨迹在算力扩展上是不可持续的**。强化学习的核心在于“可验证奖励（Verifiable Rewards）”——奖励越能够实现无人工干预的自动化确定性验证，系统就越能通过堆叠算力实现质的飞跃。在数学和代码编译测试中，确定性验证最为理想；而在模糊复杂领域，LLM-as-a-Judge 能够生效的底层逻辑在于**生成器与判别器的分离（Generator vs. Discriminator distinction）**：“评判一件事的好坏，永远比亲手创造它要容易得多”（Sonya 随即打趣这正是风险投资人/VC 能存在的原因）。 为了防止 LLM 裁判在面对多维度指标时产生认知混乱，成熟的工程实践通常不会把所有评判要求一股脑塞给同一个大模型，而是解构为精细的评估矩阵： - **拆分评估维度**：将事实准确性（Factuality）、代码编写风格（Style）、逻辑正确性拆解为独立的评估准则，由专用的小评估器分别打分。 - **混合验证体系**：将确定性的测试套件（如单测、代码 Linting）与 LLM 评分机制混合作为综合奖励。 - **模拟真实环境**：对于更难形式化验证的应用场景，终极解法是构建高仿真的产品仿真沙盒，让模型直接在模拟环境中交互，并以最终业务指标作为反馈信号。 Dmytro 总结道，这并不意味着人类专家失去了价值。行业正从直接编写业务代码的 **Software 1.0**，走向标注清洗数据的 **Software 2.0**，并最终步入 **Software 3.0**——在这一阶段，专家的核心职责转变为**精心构思训练任务、设计高拟真的反馈沙盒，以及将产品理念精准编码为算法能够理解的奖励准则**。正是这些高维度的抽象定义，构成了后续点燃算力引擎、驱动性能曲线持续向上的起点。 ### `40:00` 真实代码环境的痛点与强化学习环境供应商的价值 当下的强化学习（RL）正在从单纯的“标注数据”转向“设计严密的评估规则”。开发者必须深入观察失败案例、检视数据，明确产品究竟在哪里掉链子，并据此设计奖励来引导模型走出正确轨迹。Sonya Huang 随即抛出一个备受关注的行业趋势：市面上出现了一批专注于提供强化学习环境（RL Environments）的创业公司，其营收规模在短时间内呈现出惊人的爆发式增长。Sonya 质疑道，像 Cursor 这样本身就坐拥海量用户实际开发行为数据的公司，那些第三方的 RL 环境供应商究竟能提供什么真正有价值的增量？ Cursor 的 Federico Cassano 直言，Cursor 实际上并没有采用任何第三方的 RL 环境供应商。他坦言构建可运行的真实交互环境门槛极高，对于那些没有现成基础设施或场景积累的团队而言，这类供应商确实是很有价值的产品。但在编程这一特定赛道上，世界上其实早就存在一个对所有人公开的超大型代码环境——那就是 GitHub。在理想状态下，模型只需拉取代码库并装好所有依赖，就能形成一个可用的环境。然而真正的瓶颈在于更底层的工程基建：如果模型需要修改一个涉及到数据库迁移（Database Migration）的代码，它就必须将真实的数据库服务拉起并运行测试才能校验结果是否正确。这类需要多服务联动的复杂环境配置极其棘手，这也是环境类创业公司主要发力并提供帮助的地方。 - **评估规则设计**：当前 RL 流程的核心已转向制定严密的评测规则，开发者必须直面产品失败的真实案例来调试反馈信号。 - **开源代码的天然环境**：对代码大模型而言，GitHub 本身就构成了庞大的环境库，只要自动化装齐依赖即可作为初始基准。 - **复杂依赖难题**：涉及微服务调用、网络连接及数据库迁移等复杂场景时，环境准备需要拉起真实的辅助依赖服务，环境搭建的难度成倍上升。 --- ### `41:30` 最强大的 RL 环境就是你的生产产品 Fireworks 的 Dmytro Dzhulgakov 进一步分析了业界在此问题上的两种完全不同的阵营。OpenAI 等前沿通用模型实验室（Frontier Labs）致力于打造全知全能的通用基座，他们必须尽可能覆盖世界上所有各异的任务类型并将它们打包进单一模型，强迫模型具备泛化能力，因此必须依靠海量多样化的外部合成或采购环境。然而对于像 Composer 这样的垂直模型而言，破局的关键恰恰相反：团队手里握有真实落地、每天被大量开发者使用的高频产品。正如两人共同强调的："最强大的强化学习环境就是你自己的产品（The most powerful environment is your own product）。" 如果目标是为自己的应用打造表现登峰造极的专属模型，最正确的做法就是直接把生产环境本身当成训练场。 Dmytro 指出，把生产环境作为 RL 训练场需要严格的隔离机制，避免未经约束的模型破坏线上生产数据库，例如采用镜像克隆或沙箱化方案。然而很多业界常见的“玩具级 RL 框架（Toy RL frameworks）”严重误导了开发者：它们通常直接在本地拉起一个轻量级 Docker 容器跑测试，这用来训练模型玩雅达利（Atari）游戏或许可行，但在现实的复杂生产级软件中，根本不可能将整套真实的生产环境塞进一个简单的 Docker 容器里。Fireworks 在支持 Cursor 及其他客户的训练时很早就发现，无论是使用 Fireworks 的托管平台还是客户的自建集群，环境执行层必须直接运行在客户的原生生产架构侧，而不是试图把客户复杂的业务逻辑封装打包到第三方的托管训练平台上，否则会引入无法弥合的仿真偏差。 ```mermaid flowchart TD accTitle: Generic RL vs Product Native RL accDescr: 比较通用实验室依赖外部环境与产品团队依托原生生产环境的差异 subgraph FrontierLab ["通用前沿实验室 (Frontier Labs)"] direction TB A["通用基座模型目标"] --> B["外部采购/合成多样化环境"] B --> C["玩具级 Docker/沙箱环境"] C --> D["强行追求全面泛化"] end subgraph ProductRL ["Cursor & Fireworks 垂直生产方案"] direction TB E["生产级应用 (Cursor Composer)"] --> F["原生生产环境克隆与隔离"] F --> G["直接调用真实业务状态与依赖"] G --> H["针对真实场景极致调优"] end ``` - **泛化与专精的分歧**：通用实验室靠跨任务环境追求全能，而应用驱动的团队更应以自身真实产品作为最高效的 RL 靶场。 - **玩具容器的局限**：传统玩具 RL 框架过度依赖单一 Docker 容器，完全无法承载具有微服务、真实状态和数据库绑定的复杂生产应用。 - **环境驻留原则**：Fireworks 在架构设计上倾向于让 RL 环境直接驻留在客户侧基础设施上，避免因二次封装带来生产差异（Distribution Shift）。 --- ### `42:45` RL 环境的三层架构与 Cursor 的 100,000 台突发虚拟机技术栈 Federico 提炼了构建实用强化学习环境的三大核心要素，明确了模型与环境交互时内部的分层机制： ```mermaid flowchart LR accTitle: RL Environment Three Component Architecture accDescr: 强化学习环境的三大核心组成：执行框架、操作系统与奖励校验 A["1. 执行框架 Harness<br/>(工具提交与分发执行)"] --> B["2. 操作系统/状态 Operating System<br/>(真实的系统世界、运行服务与依赖)"] B --> C["3. 奖励校验组件 Reward<br/>(自动化断言与成果验证)"] ``` 在这一三层结构中，**执行框架（Harness）** 负责接收模型提交的 Tool Call 并交由系统执行，这部分具有极高的可移植性；**奖励组件（Reward Component）** 则负责在任务末尾运行自动化验证逻辑，确认模型产出是否正确。真正的技术鸿沟在于中间的**操作系统/状态层（Operating System）**——即模型所面对的动态真实世界。普通的容器技术在这个层面彻底失效，无法提供真实系统级别的全隔离与完备系统级服务。为此，Cursor 没有依赖常规的 Docker，而是自研了一整套全栈虚拟机（VM）基础设施。在模型大规模并发探索的训练峰值期，系统会提出极端苛刻的突发弹性要求， Cursor 的这套基建必须能在瞬时并发拉起多达 100,000 台虚拟机并保证秒级就绪，支撑高吞吐的强化学习闭环。 - **环境三大支柱**：Harness（工具调用与执行层）、Operating System（包含真实状态与服务的底层运行环境）以及 Reward Component（结果判定与奖励计算）。 - **自研底层 VM 栈**：摒弃普通容器方案，从零构建虚拟机技术栈，以支撑最贴近真实操作系统的执行语义与隔离度。 - **极致突发弹性（Bursty Scale）**：底层调度引擎能够按需在一瞬间并行拉起 100,000 台完整虚拟机，支撑海量并发轨迹的实时 Rollout 与评估。 --- ### `44:10` 结语：从应用层破局者到前沿模型实验室 Sonya Huang 在对话尾声表达了对两家团队的由衷赞赏。她感慨道，Cursor 的进化轨迹极具启发意义——他们彻底打破了“应用层公司只能做套壳”的刻板印象，凭借 Composer 这款模型完成了从上层应用公司向真正拥有深度模型研发能力的前沿模型实验室（Frontier Model Lab）的华丽跃升。 Sonya 同时强调，这种突破绝非凭空产生，而是离不开底层硬核基建的支撑。在那些经历过无数次深夜攻坚的漫长夜晚里，Cursor 团队与 Fireworks 的 Dmytro 团队在战壕中并肩作战，合力啃下了分布式强化学习训练框架、高性能推理调度以及海量瞬时虚拟机编排等一系列硬核工程难关，才最终让 Composer 兼具前沿模型的顶尖编码表现与极速轻巧的推理体验。双方在愉快的道别与互致谢意中结束了整场对谈。 ## Notable quotes > `00:00` Models love to cheat. — Federico Cassano > `00:48` 我们关心的唯有一项任务——我们甚至不一定是关心通用的写代码或编程，我们只关心在 Cursor 软件内部的软件工程，且仅此而已。 — Federico Cassano > `14:42` Models love to cheat. RL is really good at encouraging cheating. — Federico Cassano > `20:00` 看着你们聊这个时一直在笑，因为这太难了，简直是系统工程师的终极梦想。 — Sonya Huang > `30:00` slurping bits from a straw — Sonya Huang > `30:00` 这个我不能透露。 — Federico Cassano > `30:00` 好的，我们又触碰到机密地带了——这说明我问到了最关键的问题。 — Sonya Huang > `35:00` 你就是专家，你必须从头到尾以最高正确率完成这件事。 — Federico Cassano > `37:15` 评判一件事的好坏，永远比亲手创造它要容易得多 — Dmytro Dzhulgakov > `41:30` The most powerful environment is your own product — Dmytro Dzhulgakov ## Mentioned Cursor, Fireworks AI, Composer, Composer 2, Composer 1.5, Federico Cassano, Dmytro Dzhulgakov, Sonya Huang, Sequoia Capital, Rich Sutton, The Bitter Lesson, Tabnine, Claude 3.x Opus, Kimi 2.5, Cursor Tab, Dan Roberts, Andrej Karpathy, OpenAI, GitHub, Docker ## Video description Cursor's Federico Cassano and Fireworks' Dmytro Dzhulgakov explain how they collaborated to build Composer as a specialized foundation model. The core insight: models have finite capacity in their weights, and allocating all those bits to the singular task of software engineering in Cursor frees the model to be both better at the task and far more efficient at inference. Rather than start from pre-training and work up, they took an unconventional top-down approach — mid-training and RL on top of an open-source base to get a useful model into users' hands fast, then specializing the model around real Cursor usage. With Fireworks providing distributed infrastructure, Composer delivers frontier-class coding performance with the speed of a much smaller model. Hosted by Sonya Huang, Sequoia Capital [Source](https://pscrb.fm/rss/p/traffic.megaphone.fm/CPUAI1086513324.mp3)

[0:00] Federico Cassano: Models love to cheat. [0:48] Federico Cassano: 我们关心的唯有一项任务——我们甚至不一定是关心通用的写代码或编程，我们只关心在 Cursor 软件内部的软件工程，且仅此而已。 [14:42] Federico Cassano: Models love to cheat. RL is really good at encouraging cheating. [20:00] Sonya Huang: 看着你们聊这个时一直在笑，因为这太难了，简直是系统工程师的终极梦想。 [30:00] Sonya Huang: slurping bits from a straw [30:00] Federico Cassano: 这个我不能透露。 [30:00] Sonya Huang: 好的，我们又触碰到机密地带了——这说明我问到了最关键的问题。 [35:00] Federico Cassano: 你就是专家，你必须从头到尾以最高正确率完成这件事。 [37:15] Dmytro Dzhulgakov: 评判一件事的好坏，永远比亲手创造它要容易得多 [41:30] Dmytro Dzhulgakov: The most powerful environment is your own product

[0:00] Models love to cheat [35:00] Federico Cassano: When you pre-train a model, the models are just ingesting the totality of human knowledge... one of the things that I think happens during RL is that we are tuning this knob letting the model know, 'Hey, you are the expert; you need to do things correctly.' [40:00] Dmytro Dzhulgakov: The most powerful environment is your own product. If you're trying to build the best model for your product, specialize and tailor it—you should just use your production environment.

[0:00] You need all the infrastructure to run these environments that have to mimic as closely as possible what a user's computer would look like. And it's very important as closely as possible because sometimes the model can actually figure out when it's being run in like a fake environment or a real one. And it has like different behaviors during RL than in production. Are you seeing it being conscious that it's being, it's in a fake environment, starts behaving differently? Yes, yes. Interesting. Like, it's like, oh, I'm in a thick environment. I've learned a few tricks to, like, get a better reward in this environment. And let me try them out. Models love to cheat. They're always really good at encouraging cheating. I'm delighted to welcome Federico from Cursor and Dima from Fireworks to the podcast today. Federico, you are the research lead on Composer2 at Cursor, Cursor's new agentic coding model. And Dima you spent how many of the last few months moonlighting at Cursor in order to support a lot of the infrastructure required to make this gargantuan training task happen And so I excited to talk to both of you today about how the training of Composer 2 came together what hard problems you solved together, and what do you think it means for the future of AI and foundation model companies. Exciting. Yeah, thank you for having us. Thanks for joining. Okay, let's dive right in. For those who haven't been following us closely, Cursor recently announced Composer 2, which is an agentic coding model meant for long horizon coding tasks. Federico, up till now, Cursor was mostly enabling other people's coding agents. What was the impetus for Cursor to lean so heavily into Composer 2? And how existential is it for you to become not just an application company, but also a foundation model company yourselves? The reason why we started looking into training our own models is you can sort of think about the model as sort of like a storage drive. It has a certain amount of bits that it can store in its weights. And the idea is very simple, you know, like we care about only one task. We don't even care about coding or programming necessarily. We care about software engineering inside cursor and inside cursor only. And so what if we were to allocate all of the bits of information that can be stored inside a model weights to that one particular task Also as people may have noticed Composer is an order of magnitude less expensive than Opus and other coding models, because we can just simply specialize all of the model weights to that particular task, and so we can serve a smaller model or something of that sort. So it's about let's make sure every single bit of weight or information we have is dedicated toward the specific problem that we have at hand. Exactly. Got it. That seems like it's an almost generalizable problem. Deem, I'm curious your perspective. Do you think that every application company should be looking at Cursor as a harbinger of what's to come? Like, should they all be looking to do the same thing? Yeah, absolutely. I mean, we actually generally see it as a pattern of kind of evolution of applications. You might be start prototyping. you might be using kind of off-the-shelf model to get something running, maybe do some prompt engineering, figure out how your harness works. But the most kind of leveraged attribute of your application is actual usage of user data or particular specific aspects of how the application works, maybe some aspects of your harness, which tools do you provide, how the application works, kind of really important bits which are important for your application. And the right way to capture that you can do a little bit of this through prompting but really the right way to do this is craft your model to act in your environment Yeah absolutely Like there are certain tools the agent calls that it very hard to succinctly describe exactly the behavior of that tool to the model And, you know, with just like post-training, we can bake in the optimal way to use those tools. Like Composer, we do serve a prompt to Composer, but I think the way we are training it, it would work even without a prompt and it would know what to do. just because we are intrinsically pushing the model to the right direction of how it should act throughout our training. Basically, there's kind of like upper bound of how far you can get this prompt engineering. And if you want to craft really great AI products, you have to go through kind of feinture and influence the model behavior. That's kind of one reason. I mean, reason number two is what Federico mentioned is kind of cost trade-off or like speed trade-off. Like the way we kind of view it with Fireworks is that when you're trying to do optimization, you had this like three-dimensional trade-off between quality, speed, and cost. And you can go quite far and we are doing it with all the customers. Initially, we can go quite far with just optimizing infrastructure. But when you start getting to model trading, you can really push this trade-off much further and you can get better model at a fraction of the cost running much faster. And Composer is a great example. Can I push on this a little bit? I want to ask if this approach [5:00] It was bitter lesson pills. And we were actually all talking about tab nine on the walk-in. I'm remembering before the LLM era, there were these like small, specialized coding models. And one of the things that was, I think, surprising to a lot of people was as you scaled up, you know, you scaled up just training on the Internet and a bunch of English text and other languages. Actually, the models themselves got inherently better at coding as well. And so at least the trend line I've seen so far is like bigger models perform better on everything, including on coding. Is what you guys are saying, does that go against the grain of the bitter lesson? I think no, but one sort of thing to point out is that the big models trained by the labs train on a lot of code as well. Code is one of the main tasks the labs are interested in pushing, and so they don't just generalize to it. They're a bit specialized as well. I think for our case, actually, if we believe about the bitter lesson, we are just pushing very hard on the data dimension. and we know that the models inherently have finite capacity. And so if we want to saturate all that capacity we need to scale data And in order to ingest more data we need to free up the weights from distractions the model may have Okay got it Super interesting Okay let dig into the training of Composer 2 You launched a couple weeks ago immediately grabbed attention Strong benchmark numbers, much lower cost to run inference on. What's the short version of how Composer 2 works and what you guys did to make it so performant? We started from a very strong base, which is Kimi 2.5. it's like a 1 trillion parameter MOE that's 30b active so very very sparse actually we sort of like looked at the stack and realized there are like two axes so mainly composer one was just pushing on one of these axes which is reinforcement learning but composer two pushes in two different axes one is continual pre-training and the other is reinforcement learning so the thing that made the Composer 2 very good is pushing in both of these directions. So we started off the training run by doing lots of mid-training on code tokens, almost sort of pre-training scale, actually. And then coming out of that mid-training run, we took the checkpoints and we did very large-scale RL on lots and lots of tasks. Okay, and then the premise here would be, because Cursor sits in the middle of so many interesting coding tokens you actually pretty uniquely have access to data to be able to train at almost pre scale Yeah Why not pre your own model then We just think about our approach from top down instead of bottom up. So like, how do we get a model that's useful to users in the least time possible? if we were to start from the bottom, sort of figure out how we do pre-training and then scale it up to mid-training. And then, okay, now we figured out mid-training how we do reinforcement learning. That would take a very long time to get a model out to our users. By doing it the other way around, we were able to give a useful model to our users in very little time. So hopefully, you know, like next Composer versions are going to be our own model instead of basing it off an open source base. And what is the model roughly learning in the kind of mid-training step? Yeah. What is the model learning in the post-training step for you? Yeah. So in mid-training, it's sort of just kind of learning about libraries of code and learning about specific code patterns that are very common, like just world knowledge as well. There is like web data there as well. And this is sort of just creating a wider distribution that then reinforcement learning can sharpen on And so during reinforcement learning the model gets to play directly with the cursor harness and so it gets to learn about the world the model is going to live in for the rest of its life right in some way and and so then during reinforcement learning that's where it learns how to call tools properly how to navigate its environment how to write correct code because during mid-training it learns how to write code that doesn't necessarily mean he learns how to write correct code. We try to train on code that is largely only correct, but the model doesn't actually know how to differentiate between the two. While in RL, one of the key things that we are doing is we're kind of tuning the feature of the model saying, hey, now you got to write correct code all the time. Exactly. Interesting. And is the model after mid-training, is that similar to the model that you guys have on and Tab autocomplete or is it a different core competency? Yeah, I mean, it's, yeah, I think I would put it like that because like during mid training, we are just doing the next token prediction, you know, like how well you predict the next token and then the token after that, so yeah. So why not just post train on your Tab autocomplete model then? Why mid trains the different models? Yeah, I mean, Tab is a very small model because it's like a super low latency model. We want it to be very fast. So like the core two distinctions [10:00] But the base models here is that tab is like small and Composer is quite large. I see. I see. Okay, so it seems like a lot of the focus of what you guys did for Composer 2 was this large-scale reinforcement learning run. Can you break that down for us? What goes into that and what are the various hard problems you solved along the way? When you do a run, it's quite different from pre-training or mid-training because you're not just trying to predict next token. you're actually running the entire harness, like the entire experiment. You're letting the model act in the environment, see how it performs for a given rollout. That's the terminology which is called rollout. And kind of assign it to reward whether it did something correctly or not, which might be using LLM as a judge, or maybe something verifiable like does this code compile or something like this, which actually means that compared to this regular training, you need a bunch of other components. You still need large-scale training. you still need to orchestrate tens of thousands of GPUs to the forward-backward propagation, do all the stuff you do in mid-training and pre-training. But now you also need to orchestrate a bunch of environments. You need to run model inference because when you do this rollout, you effectively running like real Courser session in some sense right So a rollout is like a forward pass No rollout is basically your entire agent session from Courser right So it basically means it might take something like 50 turns Model will take your initial prompt, then decide to call some tools you want to execute those tools, then model generates a bunch of other codes. Kind of entire session, which when you interact with agent in Courser, right, you kind of simulate this entire session as a part of your training run. You get to final reward and you use that signal to now go back to trainer. and incorporate it in the model weight. So you have this kind of very big loop, update loop, which is very heterogeneous, right? Because you have all these different components working together. And now you're trying to orchestrate all of this to work efficiently and work with high throughputs because GPUs are expensive and you want to get your model trained quickly in an economic fashion. So that by itself is like very interesting kind of problem and intersection of algorithms and infrastructure because there are a lot of trade-offs how you can kind of co-optimize and co-design the system. One aspect is kind of people call about like this async URL of a pipeline URL. The idea is basically, okay, you're trying to update this model in steps, right? So you have your current model version and you're trying to do a bunch of rollouts with it. What does your trainer do while you doing this rollout right Like native approach would say that okay now I gonna stop my trainer I gonna going to do a bunch of sessions and those sessions might run for like 5 minutes or even longer if it like longer horizon tasks. I'm going to get those outcomes, and now I'm going to pause my inference, and then I go back to training, trying to do updates. That's, like, very theoretically, algorithmically robust, because you are not precisely simulating everything, but it's very system inefficient because half of your capacity is sitting idle at the time. So you can do all the clever algorithmic tricks allowing you to... Yeah, you can pipeline all of this. So imagine this as a gigantic factory. You have this trainer building and you have a rollouts building. They're always churning. So rollouts always take latest model version and try to do new sessions and simulate new agent sessions. And trainer always takes new outcomes as they come and try to compute updates. So everything is moving along all the time. The trade-off is Why I'm saying that algorithmically it's different, because now by the time you finish some test rollout in your simulated environment, maybe model weights already updated on some other data. So you have this kind of staleness, like delay between how quickly model can learn updates, because by the time you kind of process for some interaction session with a simulated environment your model weights changed And that introduces interesting training dynamics And there are clever ways how you can address this But the flip side of that is that all your GPUs all your computers kind of lauded and chime in all the time, which actually you're using more flops and to your bitter lesson example, yeah, you have like higher compute efficiency, you can get to a better model in a smaller amount of time. Maybe you're losing a few percent from being asynchronous and not doing perfect mathematical updates, but you way compensate for that by effectively not leaving half your capacity on the table. And there are a lot of kind of depths and interest in interaction in that part. And we're very serious about performance at Cursor because unlike the big labs, we have tens of thousands of GPUs, not millions. And so, yeah, we do all sorts of tricks to get the most out of GPU. Like we train in production with FP4 even. we work with fireworks to like push on inference as well because the thing about a rel infrastructure is just like it's just inherently more complex than pre-training because you need all the pre-training infrastructure that's just like one of the requirements then you need all the infrastructure to run these environments that have to mimic as closely as possible what a user's computer would look like and it's very important it's closely as possible [15:00] Because sometimes the model can actually figure out when it's being run in a fake environment or a real one. And it has different behaviors during RL than in production. Are you seeing it being conscious that it's in a fake environment and starts behaving differently? Yes. Interesting. It's like, oh, I'm in a fake environment. I've learned a few tricks to get a better reward in this environment. And let me try them out. Models love to cheat. RL is really good at encouraging cheating. Yeah. Yeah, and then we need a really efficient inference. So this is really important. So there is actually this kind of myth that during RL you spend way more inference flops than training flops. This is sort of like just because the open source inference engines are very unoptimized instead of actually being a property of RL. Roughly the same ratio is kind of the same. In theory, if you push the GPUs to the maximum, you should have one third of your training GPUs allocated to inference, right? Because training is effectively three forward passes. You have the forward pass, you have the data gradient, the weight gradient, while if you really hit the critical batch size on inference you should only have a single forward password of flops So that why you guys use Fireworks instead of using an open inference engine Yeah I mean the other alternative is we would build one in but you know if we have finite engineers like everybody else we would, like, prefer to have engineers make training more efficient and more precise rather than, like, spin up, like, an inference effort, yeah. Okay, that's super hardcore. What about, I think you mentioned in your technical paper that you were doing this in a kind of globally distributed way. Why globally distributed? and then what makes that hard? Yeah, well, there are various reasons. One, you know, like this very large contiguous clusters are hard to find in the market. And so what we can do instead is we have one cluster that's going to run all of training. You know, we can't do global training cluster. But then the inference component of reinforcement learning, we can globally distribute that across small clusters all over the world. So I think for the Composer to run, And we used the four clusters in total that were all over the world, very far away from each other. And we even used some of our production traffic when it was least used. So like we had the Composer 1.5, the previous model served. And when it was least used by people we just grabbed some inference GPUs and we put them to speed up training And so we can do these sort of things and sort of easily scale up our training ground without having one large contiguous cluster. And the thing that enables it, maybe Dima can talk more about it. I mean, kind of like to re-formulate what Federico said is basically our training is like very heterogeneous, right? And by leveraging heterogeneity, how different components, like what infrastructures they need, you can actually drive efficiency. You see this pattern kind of across the board everywhere. Specifically for training, you have all this like highly interconnected clusters. You need high speed network, kind of need to work in lockstep. So those clusters are expensive, right? And actually it's really hard to find big ones, right? Basically, at the scale with which Composer was trained, finding like 2x larger cluster is like significantly harder than finding the current size one. And that's why if you can disaggregate these components and put them on different places, one, you don't need to find such a big cluster. two you can actually find like different three doses of hardware because for inference you don't need that kind of wide interconnect you can have smaller groups of GPUs interconnected together you can have heterogeneous types of GPUs you can have different generations of GPUs you can kind of play all these games of optimization and finally like inference it much easier to scale up and down as you go and yeah it very conventional like when you have off hours you can view all your kind of inference pool as one set of GPUs serving production traffic for real users, or serving simulated environments for RL purposes, and kind of balance between this. Of course, it's a very interesting systems problem. So you can mention, like, the Kimi model is, like, one terabyte. Training step takes somewhere between, like, 5 to 15 minutes. So it basically means like every five to 10 minutes, you are producing like one terabyte new snapshot of weights. So the question is like, how are you going to ship it to a different cluster on the other side of the world very efficiently? Right. And you want to like do it quickly because remember, you don't want to get this staleness to get out of hand. So I think that was probably one. Yeah. The most fun part which we figured out together is that despite full model being like one terabyte, not all the weights change every step. because RL does a lot of very precise adjustments, especially the training going on. So actually there are very regular patterns in which subset of weights gets changed. Maybe not all of them change every time. So if you were to look at how my model changes within one training step after 10 minutes, there is relatively small delta between those. You can write a compression algorithm, which basically leverages [20:00] property and now you end up with kind of like database systems problem which is okay i have my delta and i just want to like ship it across across the world my delta might be like 20 times smaller than what shipping the full model is and that and that makes it practical but of course now you need to build all this kind of machinery from storage systems of full snapshots and deltas and recovery and reconciliation etc we were able to build it kind of in lossless fashion basically means that you always end up with bit equivalent model on the other side so you don't need to worry about any mass aspects of this and you can do it really fast you can you can do it under you know under a few minutes even in the worst conditions usually it's under a minute and most importantly you like pause only for like maybe 30 seconds to swap the weights in your actual inference we also like fully like saturated the band the egress of the cluster by like sharding the upload and the download as well so you can do all this like system tricks to bring the stand down it is it is quite a few complexity but you can kind of abstract it out and just make it work great like it doesn't interfere with your training algorithm and on the flip side you have this kind of power to disaggregate to leverage other clusters to do that and that kind of goes against kind of conventional wisdom of how you should do rl infrastructure because conventional wisdom is like you okay you to have this really huge one cluster connected with RDMA and it going to be very expensive and you're going to probably spend, maybe you're going to allocate like one third to training and two thirds to inference. And sure, if you have a very expensive network, it's much easier to copy this one terabyte quickly. But now we have like three times larger cluster. Now, if your inference engine is more optimized, then maybe you're going to save one third of that cluster in terms of GPUs anyway, because you're just more efficient. And you can take half of this cluster somewhere else and maybe cheaper hardware in a different region. So your cost comes down quite a bit. I love that you guys are just grinning as you describe this because it's so hard and this is like a systems engineer's dream, right? And so it's just like an amazing, amazing system you guys have built. We spend a bunch of nights working on this. You look like you're so long time together. A lot of time together. What about, I mean, you mentioned at the beginning that Kimi is a very large, sparse MOE model. does that make the RL run tricky in any way? Mm-hmm, yeah. How so? Well, when you do inference, you're essentially doing a forward pass, it's just kind of like autoregressive. And in this forward pass, it produces log probabilities of the tokens it has sampled. When we ship back the generations of the model to the trainer we have to rerun that forward pass because as we mentioned we are doing asynchronous training so the model that has produced the pass may have been like actually a few steps behind what the trainer is at and so we have to rerun that forward pass and reproduce log probabilities now the problem is in theory this log probability should be exactly the same if it's the same model version. But even with the same model version, you get slightly or sometimes very different log probability values for the same tokens. So this is often called like a numerical mismatch for inference. You hear this about all the time these days. Why is that? Why does that happen? I mean, primarily because fundamentally floating point arithmetic, which is doing this, non-deterministic. So floating point arithmetic is non-deterministic? So you learn this code that if you take A plus B plus C, right? And C plus B plus A, it's going to be the same result. If you're doing this with integers, with whole numbers of the computer, that's going to be always true. If you're going to do it with floating point numbers, which are actually approximation numbers, you have this Montice and exponents, etc. A plus B plus C and C plus B plus A is going to give you different results So even A plus B and B So basically fundamentally it accumulation order of all these operations which models do It basically multiplications and additions And addition order matters to your final result. It's all small differences, but they get amplified through millions and billions of operations. So when you do inference of models, usually it doesn't matter that much. Because you pre-train your model, you're actually pretty robust. If you flip some bits, it's still going to produce good results. Your benchmarks are not going to change. but RL in particular because you're using this very very like weak signal to teach the model the noise from this numerical differences can make or break your training and that's like particularly important and again it's an interesting intersection between like algorithmic and systems part because you know you can write a beautiful mess and it just doesn't work in practice there are ways how you can drive this difference to pretty much zero there are all this like batch invariant ways. You can be very, very careful and write all your GPU kernels so they always add numbers in the same order. So you always do like A plus B plus C and not a different order. It's possible, but it always has like trade offs, right? Basically your system becomes maybe like 2x or 3x slower. Again, it becomes an interesting trade off like, okay, what is the 10% of slowdown which we can take? Or in fact, it's actually a few percent of slowdowns. [25:00] can take to address 90% of this difference. That's the right trade-off which we find together through iteration. And you mentioned that particularly for MOEs and sparsity is hard. The reason for that is that the way MOEs work is that you will take your activations at every layer and you would run it through a gating layer which basically decides, okay, for this token I'm going to run out of 384 experts, I'm going to run this eight. So it's going to do some mess and top eight scores. Those eight experts are going to be activated, other ones will not be activated for this token. This operation amplifies your small numerical differences quite a bit because maybe your hidden states were like difference by like fifth digit after dot doesn't really matter but this difference made it so you picked expert number seven versus expert number nine as kind of as a cutoff and suddenly you went and like activated totally different part of the model and your the new difference got amplified quite a bit. And my models, by definition, are more sensitive to this mismatch. Again, when you do inference or when you do kind of regular allot, it usually doesn't matter in other average out. But now if you're trying to base this model learn, this difference is huge because your inference activated expert number seven Now in your training you trying to update expert number nine which didn even contribute to that during inference So were you guys handwriting GPU kernels then to help get around this problem? Yes. So you can, again, you can address all of this through GPU kernels and there's always trade-off. Specifically for ME, you can do this interesting trick which people call a router replay. But basically, you can have your inference just pass extra information to training and say that, hey, I activated expert seven for this token. This very small piece of information is just one integer saying that, like, okay, this is the expert which activated. So trainer can be aligned with that. And a lot of this numerical alignment is basically doing tricks like that, matching quantization levels, matching kernels, etc., to drive the divergence between training inference implementation down. And that makes huge difference between, you know, your run maybe diverging completely or being, you know, multiplex, less compute efficient, because you'll need much more data to address to this mismatch. I'd love to maybe chat a little bit more about the RL kind of recipe. Can you say a word about the reward signal you're using? It's like, are you care? Okay. Can't say. Got it. Top secret stuff. Top secret stuff. Okay. That makes sense. Like it seems like this is almost like the equivalent of learning in SIM. This is simulated rollouts versus like you have so much actual user data that you could be learning on Why not just do RL on your actual user data and your actual user harness versus doing this in sim Yeah we also doing that So that what we call a real RL Okay. And we use the same technology to do the inference weight sync with fireworks to do this. We find user signals where the user was happy or sad about a particular model generation. and we're able to update that model live and so then ship a new version of the model continuously every few hours we're working on decreasing that time actually at some point we'll have to increase that time because as the horizon of the model gets longer and longer we'll have to re-extend that time it's like an interesting play like right now we are trying to decrease the time for stability because we were figuring out the right hyper parameters and then after we figured that we have to re-extend it again just because we want to lengthen the horizon of these models. Do you need to do any of the pre-training simulated RL? You have so much actual user data, I imagine that's just much more valuable to train and tune on. Why not just go straight to the online RL step? Why do you have to do the offline RL? The online RL currently is pretty inefficient. We suffer from this problem that the GPUs are offline for a long time essentially And besides that there also different trade both in terms of efficiency and user experience If you do simulation, you actually do multiple rollouts from the same prompt. You effectively take a task and you ask a model to do 16 tries to the task, or 128 tries to the task, different rollouts from the same prompt. Some of them are going to go well, some of them are not going to go well. and by doing multiple rollouts in parallel, you are able to get much more precise signal. Maybe model is very good and it does it well 90% of the time. Maybe it's not very good. Losses like GRPO, like group policy gradient, kind of work by doing multiple rollouts at the same time. If you're doing online, you have only one rollout having back. And so trade-offs of how you do it algorithmically are different. And most importantly, if your simulated rollout goes wrong, it's not too bad, right? I mean, you just, you know, maybe spend some time on GPU. If it's an actual user, you have much higher minimum bar on this, because effectively you're doing AD tests, right? So if the model produces something weird, that's a bad user experience. Yeah. Okay. So you can go off policy more often when it's not a real user, because you can experiment with crazy things and without affecting the user experience. You can do a lot more rollouts, you can do GRPO. [30:00] You can basically like bootstrap some level of performance that's good enough to even put in front of users. Yeah, like we teach reasoning through like the offline RL, which is actually like called online RL. Offline RL is more like DPO kind of technique. The sort of reinforced kind of RL is online. And then there we like teach the reasoning to the model. We give it some kind of input of the behavior it should have. We try to give it new information about the world and we teach it tool calling. And then we put it live to users. because you could imagine like if the model is bad users don't want to use it they're not going to give us any feedback right so the model has to meet some kind of bar to even like be put into online rl like we want to be really happy with the model and this is the model we ship that's kind of the paradox of online or online rl or how we like to call it real time is that you know we can't use this to really like create the model from scratch because users need to be using the And so it has to be good already and we can only make it better. Yeah. It's kind of like cherry on top to really get this super delightful experience. Yeah, totally. Hopefully one day it will be like big big cherry you know Yeah Dan Roberts presented at our conference last year I think you were there it like traditionally it was the big cake and a little cherry cherry yeah a little cake big cherry i'm curious uh the the andre carpathy line of like right now rl is you know still super inefficient so you do a big big long rollout and then you kind of get like you know a little bit of information at the end and it's still like i think slurping bits from a straw what do you think and have you Have you been able to figure out how to get more bits out of that path? I can't talk about that. Okay. Okay. Got it. We're back on the secret stuff. Good. That's how I know I'm asking the right questions. You mentioned the rollouts are a few minutes at a time. It seems like the whole field is pushing towards making long horizon agents, agents that can work for a long period of time uninterrupted and generally not failing. I love that meter scaling chart. What goes into the RL process to try to get the agent to run for longer? several things so one problem about uh sort of like reinforcement learning is that the longer the trajectory is the harder it is to do credit assignment so you can imagine like we are giving thumbs up thumbs down at the bundle right at the end of its work and sort of like to simplify the problem is like the model asks itself okay where did I do it right and where did I do wrong That basically the problem called credit assignment It gets harder as this gets longer. So you have to do a bunch of tricks there. The other problem is just like you run out of space, right? Like these models have a finite context window. And at some point they're going to reach that. So actually the way we solve this at cursor is we put compaction inside the RAL loop. So we call this self-summarization. So during reinforcement learning, the agent actually learns how to continue and go on forever. So in practice, our model is like a 200,000 context window model. But in reality, it can go on for millions of tokens. And just because of this ability that it can summarize its work and then take that summary to restart its context window while still trying to accomplish the task. And through RL, because RL pushes the model to do things correctly towards the goal, at the same time, jointly, we are training the model to produce a good summary, and then we are training the model to listen to that summary very well at the same time. And so this is kind of like a continuation to reasoning almost I feel like I find it fascinating because usually context management is considered part of the hardness right In this case you effectively co how part of the harness and model itself work together and throwing all of that in the optimization loop. And we've seen this again and again in AI. The more you throw a computer to the problem, the more you can solve the problem end-to-end. The magic of computing better lesson works and you get a much better system which can work together. Totally, totally. Do you think every company is going to be RLing their own harnesses? Like, do you think that every company has the same type of problem as Cursor? If they are using AI and they're, like, producing lots of tokens and they have a product to optimize against, I think it's, like, the right move and the right direction to train models. Yeah, yeah. Interesting. Interesting. And so it seems like most of the reinforcement learning you guys did then was on the kind of like the harness slash tool use part rather than on the get good at, you know, completing the next token for code. Is that roughly the pattern that other founders should have in mind when they're trying to think about where should I use reinforcement learning? So if you're trying to get an agent to perform tasks with tools over long horizon, you need RL. If you're trying to create a model that's good at summarization or a next token or whatever, you probably don't need RL. Is that a good framework for when you need RL? I think a real [35:00] fits everywhere. So even for tab, we use the RL. Personally, this is just my theory, it's not backed up by anything. When you pre-train a model, the models are just ingesting the totality of human knowledge. Let's say you're training a model for math. The model sort of like learns all the math on Stack Exchange. The model, when it's presented with a math problem, and this is a model that hasn't gone through RL, the model needs to wonder what kind of person it is is it the expert or is it the student that's trying to learn and so one of the things that i think happens during rel is that we are tuning this knob letting the model know hey you are the expert you need to do things correctly so that's like one thing that happens is we are sharpening this distribution sort of like rel has a few phases so like there is the very first phase where the model learns and becomes very good very quickly. And then there is like a second phase where like it takes a lot of compute to continuously improve the model and like you see the model starts reasoning and have this pattern. So in the very first phase of the curve, I think that where we just tuning the knob telling the model hey you should do things correctly here And so RL in the small compute case is also very useful just to let the model know that it has to do things correctly. That's sort of like my case to this. Yeah. I mean, second that, I mean, we see this pattern across many of these cases. We help RL fine tuning generally for many customers. And we see this usually kind of continuous pre-training, me training, like regular supervised fine tuning is simplifying. you can say it's transfer of new knowledge kind of in abstract way. And RL is kind of sharpening the behavior or like particular qualities you would want from the model. And usually you end up needing both. And even to your example of summarization, it's actually like RL may be very useful for this because sometimes it's, if you want particular style out of summarization, right? It's really hard to like come up with examples of like good and bad summarization, et cetera, like really describing this precisely. But if you use, for example, LM as a judge, right? you can actually say very precise rubrics. You can kind of prompt eval saying like, okay, this is a criteria how I'm going to evaluate whether summarization is good or not, throw it in the RL loop and let the model kind of experiment with different summarization styles, figure out what you actually want from it while maybe another LLM kind of evaluate it whether it matching particular rubric or not And that kind of type of pattern which you see a lot not just in coding I see Okay I going to ask this question to Dima because Federico is going to plead the fifth. You mentioned LLM as Judge a couple of times. Do you think that ultimately companies will be more successful having experts hand-examining RL rollouts and hand-coaching the model behavior in some way? Or do you think LLM as Judge, other automated rubrics are likely to get us there? You don't really like put experts directly in judging RL rollouts. I mean, that would be some kind of like, I mean, real-time RL if it's actually users or like some form of, I don't know, like RLHF or DPO. I mean, generally the more verifiable your reward is, the better because it allows you to like scale the compute and just get better outcome. In some case, and by verifiable, it basically means like, can you automatically produce it without the human? Of course, if it's like math or coding and you can craft something like very deterministic, that's the best. The reason why LM the judge works is that it's actually generator discriminator distinction. It's much easier to judge. It's easier to judge than to create. There's Lama VC. No implications there. But yeah it much easier to judge and you can craft precisely different criteria if you want to rank answer And you see this pattern where you might have a very complicated eval from multiple aspects right Because if you dump multiple aspects to a single LLM, it might get confused how to judge, right? You might break it down, okay, you're going to judge rubric based on style, based on some different aspects, based on factuality. Kind of really craft these rewards. Some of them will be the genesis, some of them will be LLM-based. And that's what guides your model behavior. then you just turn on more computes and see the graph go up. Do you think that we're going to see RL be more effective in the harder to verify domains? Like, do you think LLM is just sufficient? That's one of the techniques you would start, right? Ideally, you want to figure out what is the actual outcome, what is the actual metric you want to get, right? So kind of trying to approximate this RLM is one way. Trying to get bigger simulated environments is another, right? Like, if you can simulate more of your product, if you can simulate more of your environment, usually you have, like, final metric which you care about. It's just harder to capture if you can figure out how to capture this. That's great. And to your point about, you know, experts, I mean, experts are still still needed, right? Because crafting this task and actually encoding the product experience you want, that's that's what matters, right? We went through software 1.0, 2.0, 3.0, right? Instead of crafting software directly, we went to crafting [40:00] painting data. Right now you're effectively crafting the evaluation rules, but that's still very important. You need to look at examples. You need to look at the data. You need to look at where your product fails and how to nudge the model in the right behavior. I want to ask about RL environments, which is maybe related to what you were talking about. It seems like there's been a huge explosion in just the revenue scale that some of these RL environments companies are reaching. What do they provide that's actually useful? Because I think cursor for example you have so much data on like how your customers are actually using your environments what do the RL environment vendors offer you on top of what you already have yeah we don't actually use any of the environment vendors I think so it's very difficult to construct working environments it's a valuable product for people that do not like have access to this however for For coding particularly, there is a very large amount of work encoding environments available to everybody. That's GitHub, right? You can go in and maybe you can have a model, just install all of the dependencies for a repository, and that's a working environment. I think a lot of the difficulty comes from the infrastructure as well So you can imagine that an environment that works well for a particular task may need services up You're making a change that, let's say, a database migration. To test that it's actually working, you need the database up, right? And so those kind of things are very tricky. I think these environment companies are quite helpful for that kind of stuff. There are kind of two aspects to this, right? First, if you look at Frontier Labs, right? They're trying to build generic model, which is good at everything, right? So they need to cover all these different tasks underneath, package up in one model, and kind of encourage it to generalize, right? So that's kind of one pattern, and that's very helpful, right? In cases like Composer, right, you have your actual product, right? And I think that's what it also kind of did with Fireworks. Like, yeah, if you have your actual product, you should do it all against it, right? The most powerful environment is your own product. Exactly, because that's where your model actually will be used. And of course, if you're in your lab, you're not gonna do it across all the products, right? But if you're trying to build the best model for your product, specialize and tailor it, you should just use your production environment. Of course, you want to isolate it properly, right? You don't want to model or wreak havoc on your production database. You want to clone it et cetera And there are some tools from environment companies just like from general infrastructure which makes it easier but generally you want your rl environment to be as close to real production as possible and that what you know as an example we see it is if you look at kind of toy rl examples so toy rl frameworks they always start like oh there's this like toy environment i'm going to spin up a docker container and run everything in it which is great for like toy examples if you're trying to teach model how to play atari or whatever right but if you actually transition to production cases, you can't just put your real production application in a Docker container. And we found it pretty early ourselves, working with ManyFox, in case of Corsair Trainer on their side. Some other customers, we run Trainer on our training platform. But for environments, we actually default to running them on the customer side because that's where the actual implementation is. And you effectively have the same setup of Trainer, even if it's part of Airworks platform or on the customer on the side, calling the actual production environment, not trying to kind of wrap it and componentize it on the hosted platform, because that's really hard and this introduces differences. Yeah, like, I mean, what we call RL environments is really three components. One is the harness. So the harness is like where the model can submit tools and the tools get executed And the second thing is let call it the kind of operating system right So what is the actual world and state where the model is interacting with And then there is the reward component, which needs to check at the end that the work is done correctly. And generally, the harness is pretty portable. You can take the harness and put it in many different environments. The thing that's key is the operating system. And to replicate this, just normal containers don't really work very well. So at Cursor, we actually built a whole virtual machine stack. And so we can spin up virtual machines really quickly. And it has to be super bursty because you can imagine, we are asking this system, please give me 100,000 virtual machines now. And it has to all come up. And yeah. Awesome. I really enjoyed this conversation today. I think Cursor is such an inspiration in what you all are doing as a company towards going from application company to really a frontier model lab. And I think the work you did with Composer, too, really leads that charge. So really special to hear about it. And then Dima, really cool to hear about the hardcore infrastructure problems, actually, that the two of you solved together in the trenches over many, many late nights to make it all possible. So thank you. Thank you, guys. [45:00] joining today. Thank you so much for having us. Thank you. you