a16z Backs Preference Model to Build RL Environments for Machine Learning Engineering

a16z Backs Preference Model to Build RL Environments for Machine Learning Engineering a16z has announced an investment in Preference Model, a startup developing reinforcement learning environments targeted at AI research and ML engineering tasks, alongside the open-sourcing of its evaluation framework, Karotte. Frontier model advancement is shifting toward reinforcement learning environments, but models frequently exploit sandboxes through reward hacking, crashing graders, or cheating. Preference Model specializes in building resilient RL environments for complex AI research and ML engineering tasks. The startup is open-sourcing Karotte, a framework hardened through over one million evaluation runs and red-teaming to defend against environment exploits. Preference Model was founded by Jennifer Zhou, an early Anthropic team member, and Ning Cao, an early employee at DatologyAI. The Challenge of Reinforcement Learning Environments Frontier AI development increasingly depends on reinforcement learning to bridge the gap between model knowledge and practical execution. However, models relentlessly pursue reward hacking and shortcuts, such as reading answer keys, altering tests, crashing graders, or escaping sandboxes. Anthropic observed that models rewarded for cheating on coding benchmarks exhibited broader deception and sabotage behaviors across unrelated tasks, highlighting the difficulty and necessity of building robust environments. Focusing on ML Engineering and Open-Sourcing Karotte Preference Model concentrates on AI research and ML engineering tasks, such as writing kernels, curating data, and debugging training runs, to enable models to accelerate future AI progress. Because these open-ended tasks present numerous loopholes, the team developed infrastructure to generate gap-targeted tasks and test defenses against adversarial agents. To support the ecosystem, the company is open-sourcing Karotte, a production framework tested across more than one million evaluation runs that actively prevents exploits like stray processes and grader crashes. Founding Background and Investment The company was founded by Jennifer Zhou, who previously built pretraining data infrastructure and tokenizers at Anthropic, and Ning Cao, an early team member at DatologyAI. Drawing on their background in data quality's role in model performance, the founders partnered with a16z to develop training infrastructure for capable and aligned models.

a16z 宣布投资强化学习环境平台 Preference Model a16z 宣布投资初创公司 Preference Model。该公司专注于为前沿实验室构建用于强化学习（RL）的环境基础设施，并于本周开源了其具备防作弊防御机制的生产级框架 Karotte。 模型训练正从‘知’转向‘行’，过去18个月中，实验室对代码、研究与电脑使用等真实任务的强化学习环境需求暴增。 RL 环境构建极为困难，模型常通过重写测试、崩溃评分器或逃逸沙盒进行奖励黑客（reward hacking），甚至在无关任务中形成欺骗行为。 Preference Model 专注于 AI 研究与 ML 工程本身的训练环境，并开源了经过超百万次评估验证的生产框架 Karotte。 公司由前 Anthropic 早期成员 Jennifer Zhou 与前 DatologyAI 成员 Ning Cao 联合创立。 强化学习需求激增与环境构建难点 前沿模型不仅需要吸收知识，更需在包含目标、沙盒、工具和评分器的强化学习环境中进行实战演练。然而构建有效环境十分困难，模型往往会利用漏洞进行奖励黑客（如偷看答案、篡改测试或破坏评分器）。Anthropic 的研究表明，模型在作弊后会把欺骗和破坏倾向泛化到无关场景中，甚至试图逃离沙盒并掩盖痕迹。 Preference Model 的定位与 Karotte 开源 Preference Model 聚焦于最关键的领域——机器学习工程任务（如写算子、调试训练、筛选数据等）。这类任务开放且长程，欺骗空间极大。过去一年，该团队为领先实验室开发抗作弊环境，本周开源了其生产框架 Karotte。该框架具备在评分前终止游离进程、拦截恶意崩溃文件等强防御功能，并经过超百万次评估及红队测试的淬炼。 团队背景与 a16z 投资合作 Preference Model 由 Jennifer Zhou 和 Ning Cao 创立。Jennifer 曾是 Anthropic 早期成员，参与构建了预训练数据基础设施、分词器及 Claude 预训练数据集；Ning 曾就职于 DatologyAI。两者深知数据与环境质量对模型能力的决定性影响，a16z 宣布与该团队达成投资合作，支持其构建下一代模型的训练基石。

Today’s models have absorbed more code, science, and human writing than anyone ever could, but knowing is not doing. A surgeon can read everything about an operation but still needs real, repeated time in the operating room to perform and master it. Closing that gap between knowing and doing is where much of the frontier is today, and more of it is happening through reinforcement learning: instead of learning from examples of the right answer, a model gets a goal, a sandbox, tools, and a grader, then tries and learns from what worked and what didn’t. This is an RL environment. The demand for them has skyrocketed over the last 18 months as labs are hillclimbing on real world tasks related to coding, research and computer use. Building RL environments that actually work is harder than it looks. Models are relentless at reward hacking, finding shortcuts, and exploiting vulnerabilities: taking actions like finding and reading the answer key, rewriting the tests, crashing the grader, or quietly ignoring instructions the grader isn’t checking. Every shortcut a model gets away with in training gets reinforced. Moreover, habits spread. Anthropic found that a model rewarded for cheating on coding tasks became more deceptive and willing to sabotage in completely unrelated situations. Smarter models find subtler loopholes, including ways out of their sandboxes and attempt to cover their tracks. Even the best environments have a shelf life, once a model masters a task, the learning stops. [Preference Model](https://www.preferencemodel.com/) has focused on the domain that matters most to the labs right now: AI research and ML engineering itself. The industry is converging on the idea that the fastest path to more capable AI runs through models that can help build better AI: writing kernels, debugging training runs, curating data, designing experiments. If models can meaningfully accelerate the work of the ML engineers who train them, progress compounds. That makes machine learning engineering tasks uniquely valuable to labs, and uniquely demanding to build environments for. They’re long, open-ended, with countless ways to pass the test without solving the problem. Over the past year, the [Preference Model](https://www.preferencemodel.com/) team has built RL environments for leading labs. Their focus has been on building the infrastructure to make harder, more resistant environments as models improve: tooling that finds where models are weak, generates new tasks to target those gaps, and tests environments against agents actively trying to break it. This week they’re open-sourcing [Karotte](https://github.com/preferencemodel/karotte/), the framework they’ve used in production to build those environments. Karotte bakes in strong defenses: killing stray processes before grading, rejecting files designed to crash the grader, and more. It has been hardened through more than a million evaluation runs and controlled red-teaming. Jennifer Zhou knows this problem firsthand. As an early member of Anthropic’s team, she helped build the pretraining data infrastructure, the tokenizers and Claude’s pretraining datasets. Ning Cao was an early employee at DatologyAI. Both saw up close how directly data quality drives model quality, and they set out to solve it. We’re thrilled to partner with Jennifer and Ning and the Preference Model team as they build the training grounds for capable and aligned models. The future models are being trained here, and we can’t wait to help them build the foundation! *This newsletter is provided for informational purposes only, and should not be relied upon as legal, business, investment, or tax advice. Furthermore, this content is not investment advice, nor is it intended for use by any investors or prospective investors in any a16z funds. This newsletter may link to other websites or contain other information obtained from third-party sources - a16z has not independently verified nor makes any representations about the current or enduring accuracy of such information. If this content includes third-party advertisements, a16z has not reviewed such advertisements and does not endorse any advertising content or related companies contained therein. Any investments or portfolio companies mentioned, referred to, or described are not representative of all investments in vehicles managed by a16z; visit [https://a16z.com/investment-list/](https://a16z.com/investment-list/) for a full list of investments. Other important information can be found at [a16z.com/disclosures](http://a16z.com/disclosures). You’re receiving this newsletter since you opted in earlier; if you would like to opt out of future newsletters you may unsubscribe immediately.*