Y Combinator Deploys Agent Harnesses to Quadruple AI Benchmark Scores

Prime Agent 助力 Claude Opus 在 ARC-AGI 基准测试中达到 95.5% 裸语言模型表现如同在流体智力测试中停滞于 30% 的纸带图灵机，而高表达力智能体框架提供了内存层级、REPL 状态和子智能体编排，完全基于相同权重即可释放出 95.5% 的性能。 该研讨表明，对于自主任务完成而言，高表达力的脚手架和测试期执行环境比单纯模型权重的微小提升更为重要。Seth Karten 概述了 Prime Agent 的递归语言模型架构，该架构将上下文作为 L1 到 L3 的缓存层级进行管理，并为模型配备持久化 REPL，从而在无需特定领域提示工程的情况下在 ARC-AGI 上达到 95.5%。Jon Saad-Falcon 介绍了 OpenJarvis，展示了如何通过前沿云端模型的离线元优化，使本地开源权重模型在个人硬件上以低 800 倍的成本运行。最后，Josh France 和 Regan Bell 详细介绍了 YC 的生产级智能体框架 QM，解释了他们为何将智能体内存解耦至 PostgreSQL、将计算沙盒转变为一次性工具，并引入了强制性的“grind 工具”以防止模型过早放弃。 高表达力的框架脚手架表现可大幅超越单纯的模型规模扩展：Claude Opus 开箱在 ARC-AGI 上的得分为 30%，但在没有权重更新的情况下，借助 Prime Agent 的非特化 REPL、子智能体衍生和工具调用框架达到了 95.5%。 (30:04) 斯坦福大学的 OpenJarvis 项目表明，利用前沿云端 LLM 作为离线元优化器来调优本地提示词、工具和执行策略，相比持续调用云端智能体 API，成本最高可降低 800 倍。 (42:47) Y Combinator 彻底重构了其内部智能体架构 QM，将智能体的内存、状态和交互日志从脆弱的本地虚拟机转移至中心化 PostgreSQL 数据库，将执行沙盒视为一次性计算资源。 (51:04) 为了应对基础模型过早放弃复杂任务的倾向，YC 引入了“grind 工具”，强制设定最低 token 预算或实际运行时间（例如两小时），以要求模型在测试期进行自我纠错。 (57:16)

Presenters at Y Combinator's Paper Club argued that external execution harnesses and scaffolding determine real-world artificial intelligence capabilities far more than raw model weights alone. By supplying persistent memory, programmatic execution environments, and recursive task delegation, modern harnesses transform stateless sequence predictors into durable, long-horizon problem solvers across benchmarks, personal hardware, and enterprise workflows. Harnessing Fluid Intelligence Beyond Static Weights (1:40) While academic critics often dismiss harnesses as superficial wrappers or low-effort prompt engineering, speakers highlighted evidence showing that execution scaffolding dictates model capability. On the ARC-AGI benchmark—designed to evaluate test-time adaptation on novel task distributions—frozen weights of Claude 3 Opus scored only 30% under standard scaffolding. Wrapped in Prime Intellect's Prime Agent harness, the exact same model weights reached 95.5%, while NVIDIA's AVO harness achieved 100% accuracy. Similarly, minor iterations in harness design can yield immediate double-digit performance gains, bridging the gap where in-context learning notoriously plateaus after 40 to 50 demonstrations. Prime Intellect researcher Seth Karten noted that an isolated large language model is fundamentally a stateless autoregressive processor—functioning like a theoretical Turing machine that merely maps input tokens to output probabilities. Equipping that core with an external harness that reads and writes external memory transforms it into a von Neumann architecture. By granting models runtime agency to execute code, inspect errors, and manage context dynamically, the harness unlocks problem-solving behaviors that models cannot execute on their own. Hierarchical Memory Caches and Sub-Agent Swarms (21:30) To prevent runaway context bloat and operational failure during extended execution, Karten conceptualized context management through a CPU cache hierarchy. Model weights represent the compute core; the active input token window serves as an L1 cache sustained through compaction tools; an interactive IPython REPL and live sub-agents act as an L2 cache in RAM; and local file storage operates as an L3 cache containing persisted skills and memories. Karten argued that L2 and L3 layers require active lifecycle maintenance through agentic garbage collection and refinement to prevent memory crashes. Prime Agent operationalizes this design using Recursive Language Model (RLM) principles, spinning up persistent sub-agent sessions backed by local background daemons. Rather than terminating upon task completion, sub-agents enter an idle state in RAM or offload to disk, preserving their accumulated context for future queries. During a continuous 7-day Factorio experiment that generated 23 million output tokens across 633 autonomous agents, this structured context compression and sub-agent division of labor prevented the stagnation and infinite loops typical of long-horizon autonomous runs. On-Device Personal AI via Offline Cloud Optimization Addressing the high recurring costs, privacy risks, and energy consumption of cloud-dependent agents like OpenClaw and Hermes, Stanford researcher Jon Saad-Falcon presented OpenJarvis. The project models an agent stack across five core primitives: user interfaces, agentic logic, underlying models, inference engines, and tools or memory. Noting that open-source weights lag proprietary frontier models by only 6 to 12 months, the team demonstrated that consumer systems, including Apple Silicon Mac minis, can run daily operational workloads locally at near-zero inference cost. To close the remaining capability gap, OpenJarvis utilizes a hybrid optimization workflow. Frontier cloud models (such as Claude 3.5 Sonnet or GPT models) are invoked offline to analyze execution failure traces and automatically refine the harness specifications, system prompts, and tool configurations. Once optimized, the tuned agent specification executes entirely on local runtimes like Ollama and vLLM, which Saad-Falcon reported achieves up to an 800x cost reduction compared to routing ongoing interactive queries through commercial APIs. Decoupling Execution Sandboxes in Enterprise Deployments (51:04) Josh France and Regan Bell detailed QM, Y Combinator's internal agent harness designed for company operations across Slack and web interfaces. QM evolved away from YC's previous deployment of 50 isolated virtual machines running Hermes agents, an architecture that created heavy administrative maintenance and locked institutional memory inside individual machine instances. QM inverts this model by pulling the agent's brain out of the execution environment: all global conversational histories and traces are stored in a central PostgreSQL database, while sandboxes are provisioned dynamically as disposable, on-demand compute resources. To counter the tendency of models to abandon complex tasks prematurely, the QM harness introduces a "grind tool" that enforces non-negotiable budgets measured in wall-clock run time or minimum token spend. However, the speakers warned of serious constraints when deploying agents into multiplayer corporate channels. Because models completely lack intuitive social discretion, sensitive company information can easily leak into public conversations unless access is governed by strict, fine-grained organizational permission engines. The presenters converged on the view that future AI breakthroughs will be driven by runtime execution harnesses and test-time compute allocation rather than static parameter scaling alone. Significant operational questions remain, including whether complex memory scaffolding will eventually be subsumed by base models, how to prevent human operators from blindly rubber-stamping agent-proposed modifications, and how to enforce social context boundaries in multi-user collaboration.

[0:00] I'm not sure this kind of prompt engineering belongs at a top tier machine learning conference [0:00] Context engineering is not a research problem. [7:13] smear the compute and logic over many more tokens [18:35] Seth Karten: just tokens in, tokens out [21:30] Seth Karten: to make sure that our RAM doesn't crash my computer laptop every day. [24:30] Seth Karten: the most expressible thing you can imagine. [51:04] Regan Bell: unhobbling [51:04] Regan Bell: main character syndrome [56:32] AGI anticipating harness [56:32] paper over rough edges in the system [57:16] often give up way too early [57:16] grind tool [58:22] agents really don't understand social contexts

[0:00] [music] [0:08] Welcome to YC Harness Club. I worked [0:11] really hard. It took me like 10 its with [0:12] Gemini to have Harsha riding a lobster. [0:16] Gemini didn't want to do it, but we [0:18] figured it out. Okay, welcome to harness [0:20] night. Um, first order of business. Uh, [0:24] how does everyone like the new YC paper [0:26] club look? [0:29] Well, this is like an idea that I kind [0:31] of blurted out at EV, our head of design [0:34] at YC, and two days later, she came back [0:37] with this, and I'm like, "This is [0:38] amazing." So, Ev in the back there, [0:40] please take a bow. [0:43] [applause] [0:45] Very exciting. All right, why harnesses? [0:49] Um, I mean, it's just a rapper. This is [0:51] just scaffolding. This is just like [0:53] prompt engineering. Um, why would this [0:56] be at all worthy of a night? Uh, here's [0:59] a great little Reddit that was only a [1:00] month ago, which is actually uh the most [1:03] aggressive. I'm not sure this kind of [1:04] prompt engineering belongs at a top tier [1:06] machine learning conference. Um, here's [1:09] another great one. Uh, context [1:11] engineering is not a research problem. [1:14] Um, so I think harnesses have long been [1:17] belittled uh as um subpar research. yet [1:22] it is uh literally gives us an 18% bump [1:26] in the difference between harness one [1:28] and harness two. Um, and as Seth will [1:31] tell us is the difference between [1:32] getting ArcGI to work and not. And so [1:35] this is a uh obviously worthy of some [1:38] some amount of research. And I think if [1:40] you look at this the classic meter plots [1:43] release date and how long uh an agent [1:46] can be running. Um, literally we go a [1:49] lot of this progress has been because of [1:50] harnesses. And so I call this the static [1:52] harness era where there isn't [1:54] self-improvement on the harness. And [1:56] then this later latest latest um maybe [1:59] the last 6 months has all been on the [2:00] self improving harnesses and think and [2:02] we'll get through all these and it's I [2:05] actually saw this plot in a presentation [2:07] by the CEO of trajectory. I actually [2:09] really liked it. was talking about how [2:12] the we we keep measuring perplexity and [2:16] you know this somewhat correlates to IQ [2:17] and and how intelligent the model is and [2:19] so we keep pushing more and more up this [2:21] intelligence uh IQ dimension but we're [2:25] not leveraging test time experience very [2:28] much so we're generating all this test [2:30] time experience we have a new domain uh [2:32] and it does and we don't really quickly [2:34] adapt and if you remember one of the [2:36] first YC paper clubs we did um I've been [2:39] doing this experiment where um the as I [2:42] increase the number of samples online [2:44] how do you actually learn from just [2:45] batch size one we don't really don't [2:47] have that structure we have ICL and then [2:50] once ICL gets saturated after just [2:52] loaded like 40 or 50 it doesn't actually [2:54] improve um on the valet at all then you [2:57] have to go to Laura small rank then you [2:59] go to Laura big rank then you go to SFT [3:02] and it's kind of weird that we have this [3:03] like different training procedures and [3:05] so I think that's really where harnesses [3:08] are shining right [3:10] And what Arc AGI exposes, that's kind of [3:12] the main uh um uh point of ARGI is how [3:16] quickly it adapts to uh a new problem, a [3:20] new a new distribution and does well in [3:23] it. And so, um, ArcGI actually went [3:26] through the batch with me, winter 26, [3:28] and we helped Greg, you know, look at [3:30] this and like the amount of thought and [3:32] attention, as I mentioned last time, [3:33] that goes into these these these games [3:35] to make sure they're all orthogonal [3:37] skills from game one to game two, so [3:39] that it's isolating this fluid [3:40] intelligence measure. Um really uh [3:44] Claude uh Opus was one of the the first [3:47] that was actually verified um on the the [3:49] hold out on the private that no one else [3:51] has access to other than Greg and [3:53] Chalet. Um and the best that they got [3:55] was 30%. And just with um some harness [3:59] uh this thing that doesn't deserve any [4:01] research just some wrapper and some [4:03] scaffolding we can get to 95 and AVO [4:06] from Nvidia got to 100%. So, Prime Agent [4:08] and Nvidia, which both recently just [4:11] came out. [4:12] And the other thing I want to add, so [4:14] when uh Carpathy launched his auto [4:17] researcher thing in March, I want to say [4:20] it was um I forked it and I was playing [4:23] around with it. And all I wanted to do [4:24] was make like a little user interface to [4:25] kind of see what's happening and track [4:27] it and and I ended up building a harness [4:30] by accident. I didn't mean to, but it [4:32] was just like I wanted to see it. And [4:34] and basically what it is is you specify [4:37] a purpose. And in this example, which is [4:39] actually a true one I gave, is like [4:41] diffusion LM don't beat ARLM. But maybe [4:45] if I ensemble, if I shard the diffusion [4:48] LM into a bunch of different ones [4:49] because there's such high arithmetic [4:51] intensity per GPU on a diffusion model [4:53] versus AR that I can actually um in [4:57] aggregate by sharding them, I can get [4:58] better better results. And so I just [5:00] give this to a per as the purpose to the [5:02] to my uh uh to my suite of agents um my [5:07] swarm of agents. I give it some seed [5:09] ideas. I want to vary the ensemble size [5:12] shard at different amounts [5:14] 100 times, 10 times, five times, [5:16] whatever. Um and then I specify a [5:19] valometric. Um and maybe I want to do [5:21] GSMAK and like GPT2 setup or something [5:24] like that. And then I have a scoping [5:26] agent that will kind of look up papers [5:29] that have that are are similar, look up [5:31] GitHub repos. Um, we'll give it to a PI [5:34] agent named Chris Ray. Um, who will then [5:37] uh give it to a research agent um named [5:40] John South Khan. Uh, there he is. Um, [5:44] and then he'll work on it and then he'll [5:45] start doing some stuff, give it to a [5:47] council for some help. That's where I [5:49] come in and me and Yaso will give you [5:51] some feedback and then you work on it a [5:52] bit more and then whenever you're ready [5:54] and Chris Ray will kind of keep tabs on [5:56] you, keep nagging you every every hour [5:58] or so then it goes to this author agent [6:00] to say okay now start stop freeze the [6:03] idea start writing ablations and um and [6:06] then start writing the paper and so then [6:08] what this has turned into so this is the [6:10] scoping agent uh and then you have this [6:12] cockpit that kind of you can view from [6:14] anywhere we do the uh tail scale up so [6:17] you can actually view this URL from [6:19] anywhere and see how it's progressing [6:21] and and talk to it right there. Um, it [6:24] just sends you email updates if you [6:26] build the whole 1B thing in here. And [6:28] then it actually starts publishing some [6:29] papers and like I've read the papers. [6:31] They're actually like good. They started [6:33] out in March, April like not so good [6:35] like you know May or whatever. But and [6:37] now I just basically give eight ideas to [6:40] eight H100 nodes and each one has eight [6:42] eight H100s. and just keep going. And [6:44] then I check in and they give give me [6:46] back papers. And it's kind of wild what [6:48] we're what we're dealing with there. And [6:50] what's changed really is just the [6:51] scaffolding that you put on top of it to [6:53] allow it. And I I didn't spend an [6:54] enormous amount of time on this, but [6:56] this is largely how I I do a lot of my [6:58] research, at least the the initial idea. [7:00] Um, and so yeah, so I'll just give this [7:03] six different ideas of things I want to [7:04] try and just leave it alone and let it [7:06] rip. And so the things that we can now [7:08] do just because of harnesses on the same [7:10] exact weight file is just wild. Um, so I [7:14] looked on I spent the the weekend just [7:16] in prep for this reading um a whole [7:19] bunch of the classic literature from [7:22] self-refined to reflection to Voyager to [7:24] all the tool former and I just wanted to [7:27] just do like a my my shot at a five [7:29] minute like history how did we get here? [7:32] Um I think it was it's kind of important [7:34] and I don't think a lot of people that [7:36] are entering AI now kind of like have [7:38] the context. So this is not going to be [7:40] in chronological order. It actually [7:41] doesn't make sense to to teach it that [7:43] way or show it that way. I'm not I'm [7:45] going to skip a lot of papers and I may [7:47] course grain the papers absurdly. So, [7:49] apologies. So, the initial harness the [7:51] GPT2 February 2019 is a while not end of [7:55] end of sequence loop. That's and then we [7:58] basically have top P sampling and then [8:00] we have the environment and that's the [8:02] harness. And there's really not that [8:03] much. There's no tool calling, there's [8:05] no skills, there's nothing like that. [8:07] And so that's the V0ero harness. [8:10] And so basically if you have this GSMAK [8:12] example, you're a math, remember the [8:14] system prompt. You are a math teacher. [8:15] This is like the persona stuff we used [8:17] to have. Um there's a context. Susie has [8:19] five bucks, she spends three. How much [8:21] does she have now? And then there's just [8:23] no chain of thought. It was just like [8:24] four hashes and two end of state and end [8:27] of sequence that is uh the GS MK format. [8:30] We measure uh what happens after the [8:33] four hashes. We get accuracy and you get [8:35] a plus one or minus one if it's wrong. [8:37] And then the entire last six years has [8:40] been giving more functionality into the [8:42] harness into a static harness. And so we [8:45] said okay well what if we give that back [8:47] into the context a bunch of examples [8:49] like that like well here's an example [8:51] and let's say now we'll switch it to [8:52] four and one and hopefully we'll have um [8:56] from that previous example we'll say oh [8:58] okay this makes sense I can will help to [9:01] learn and actually this is with a fshot [9:02] learners paper um back in July in 2020 [9:05] and then we had chain of thought and [9:07] says well directly predicting hash hash [9:10] 2 is maybe difficult what we'll do is [9:12] we'll smear the the compute the logic [9:15] over many more tokens and we'll train it [9:17] to get to that two now rather than just [9:19] output the two and that was the chain of [9:22] thought idea uh very cool and that was [9:25] all context innovations uh and output [9:27] space innovations action space [9:29] innovations um and then then we came up [9:31] with these tool former and webgbt webgbt [9:33] actually came out first then tool former [9:35] which is the idea of giving it tools and [9:37] it can call a tool that's just really [9:39] just a JSON object with a bunch of [9:41] things specified but in this example [9:43] just let's say subtraction instead of me [9:45] calculating in the weight file what is 5 [9:48] - 2 I can just call Python and call sub [9:51] 53 and it'll tell me two and so that's [9:54] pretty cool and you can expose all the [9:56] tools in the system prompt and that's [9:59] where tools come from and then we [10:01] figured out megpt which is big the one [10:03] of the coolest tools is being able to [10:04] read and write to your own context and [10:07] so before then all we could do is just [10:08] append append append append to the [10:10] context now we said what if I actually [10:12] give you create read, update, delete on [10:14] the context itself and we'll se separate [10:17] just this one little chunk called a [10:18] memory that you'll be able to to update. [10:22] And then Voyager said, okay, well, we [10:24] have these tools, but like what if I [10:26] want to chain together these tools to [10:27] achieve a task and then I learn it and [10:30] how do I distill it back into the the [10:32] system prompt to make it learn forever? [10:35] And this is the skill this where skills [10:37] kind of came about. And uh they did this [10:40] on Minecraft. Um and this the Voyager [10:42] paper and it's very cool paper. This is [10:44] like largely now what a skill is. And so [10:46] I have this skills.md and I have the [10:49] name of it that I can go and search and [10:51] here's the procedure. And then uh [10:53] intercode this idea of like again in [10:55] action space innovation if I can [10:57] actually output code uh then I can [10:59] basically now I have on the-ly uh tools [11:02] or on the fly skills. I actually don't [11:04] know if it's a considered a tool because [11:06] a tool is technically an API. Am I [11:08] outputting a function as a tool or a [11:10] skill? I actually am still unsure. Um [11:14] and then the react came first then [11:17] self-refine then reflection. But this [11:19] idea if I have multiple um agents that [11:23] that have different roles and they can [11:24] help self-improve self-improve on the [11:27] context uh then I can um get smarter and [11:30] smarter. So, I'll have take some action. [11:32] In this example, I I I flipped the three [11:35] and the five. Whoops. Um, I sent it to [11:38] an internal evaluator to say, is this [11:40] right? No, it doesn't look right. I can [11:42] either, you know, keep looping here or I [11:44] can go to the actual environment to get [11:46] back a reward signal. Um, I go here uh [11:49] uh to the reflection. They'll say, "Hey, [11:51] you actually flip these two, go back, [11:53] and I can improve the result." Um and [11:56] this is the one of the first like uh uh [12:00] ideas of like being multi- aent uh being [12:02] let the letting the agent reflect on its [12:05] own um output and improve it. And then [12:08] this idea of multi- aent goes even [12:11] further where I can actually spawn one [12:13] of the tools could be I can spawn a a [12:16] set of sub aents um and they they [12:19] persist in these persistent ripples and [12:21] they'll be able to be running and I can [12:22] interact with them and I'll have the sub [12:25] aent list. I can invoke those ones and I [12:26] can keep adding to that the the launch [12:28] sub aents and then RLM went even crazier [12:32] to allow this in a recursive fashion so [12:34] that um and I exposes this RLM query so [12:36] I keep uh recursively calling the RLM [12:39] query to uh solve a larger class of [12:41] problems and that the leafs whenever I [12:43] want I can call the LM query to spawn [12:45] that LM agent as well and then I have [12:49] this main orchestrator agent that's [12:51] running all that and that's what I call [12:54] like harness v1 one, this whole thing is [12:56] like static harnesses. I'm not improving [12:57] on the system prompt. I'm not um [13:00] updating the harness itself. And this is [13:03] this you, of course, we're going to have [13:04] to have GStack on the top. He's the one [13:06] that lets us do all this. So, we thank [13:07] you, Gary. Love GStack. Um but there's [13:10] some other ones that deserve a call out [13:11] as well. And basically what that means [13:14] to summarize all this, you have some [13:15] agent spec, you have some system prompt [13:18] here. You say, "How many turns am I [13:19] allowed? How many tool calls am I [13:21] allowed?" You don't want to allow [13:22] infinite. You specify a tool list. you [13:24] specify a skills list, the sub agent [13:26] list, and that's largely what the V1 um [13:29] is, and then you put it in a loop. And [13:32] this can be spawned on a prompt if I'm [13:34] asking it to do something in my Slack [13:37] channel, which we'll hear about QM, [13:39] which is like I use it every day. It's [13:40] the team that made it is here. It's [13:42] super exciting. Um and or it's on a [13:45] crunch and just wakes up every hour and [13:46] it decides to do work just like you [13:48] know, anyone else. Um, and so there's [13:50] some session management, there's a loop, [13:52] there's this context compilation. We're [13:54] actually creating the context. I'm [13:56] putting uh uh all that into an LM call. [13:59] I get back uh the action. And then I I [14:02] may or may not have some tools that I [14:04] need to invoke and append back into the [14:07] context. And that's basically harness [14:09] v1. And then the cool part, this is the [14:12] most exciting part where we're seeing a [14:14] lot of advancements, um, where you're [14:16] letting the harness itself learn. either [14:18] we're learning the system prompt or [14:20] we're learning the harness itself, which [14:22] is very trippy. And so, one of the [14:24] famous ones that I actually wanted them [14:26] to talk, but they're actually running a [14:29] 150 person DSPY meetup tonight and in in [14:32] this in San Francisco. They couldn't [14:33] make it. Um, but I've done some podcast [14:35] with them before and a super great [14:37] community. Is this DSPY? So, [14:39] demonstrate, search, uh, predict. uh [14:42] they have this idea of of basically uh [14:45] taking a train set a small set of [14:48] examples and then learning the optimal [14:50] system prompt. So I keep iterating, [14:52] iterating, iterating. I can't back prop [14:54] through that process, but I can do uh [14:56] something called genetic programming [14:57] where I'm finding candidates, I'm [14:58] merging candidates, and with some merge [15:00] rule, I'm evaluating, seeing what [15:02] happens, and I keep working and working. [15:04] And I basically gives me CRUD over the [15:06] system prompt itself, and allows me to [15:08] choose any system prompt. And then [15:10] Darwin machines actually go a step [15:12] further. Not only are you allowed to [15:14] change the system prompt, but you're [15:15] allowed to change the harness itself, [15:17] the harness code that is actually [15:19] running. And so you can imagine [15:21] basically what happens is you have this [15:23] archive of um many different uh agents [15:27] which is harness and system prompt. Uh [15:30] and you sample from them. You push them [15:33] uh through you'll actually evaluate how [15:35] how they did on some fit fitness [15:38] function. Uh you'll uh add it back into [15:41] into this archive state. And I skipped [15:43] over the the self modify. You can [15:45] actually you have a a meta harness that [15:48] actually allows the the agent to modify [15:49] its own uh harness so that it can become [15:52] a different harness and then that [15:53] basically loops around loops around and [15:56] eventually you get better and better [15:57] agents over time. And then the meta [15:59] harness which is the main harness is to [16:01] produce harnesses, right? Which is a [16:03] really meta concept. Um and so this is [16:05] the output space where you're [16:06] configuring multi- aent context [16:08] compilation. Um, and you're you're doing [16:11] this and you have allow CRUD on all of [16:13] it. And so it just keeps adding more and [16:15] more uh CRUD into the harness code, all [16:18] the green that you see here, the uh meta [16:21] prompt, the system prompts of all the [16:23] agents, how many agents there are. Um, [16:25] and you kind of grow this uh this meta [16:28] harness over time. And then one of the [16:30] authors of this paper is the lead author [16:32] is actually here tonight. Super [16:34] exciting. You got to drive down and talk [16:35] to him a lot of it about um continual [16:37] harness, which is I love it. And they go [16:40] even a step further where one they add [16:43] they they add some extra color on the [16:46] classes of memory and so they add this [16:49] history thing. You know he'll he'll go [16:50] into a bunch bunch more detail on that [16:52] memory breakdown. But the coolest part I [16:54] think is for the classic RL people. I [16:56] see Robert back there. He definitely [16:58] would would enjoy this Dagger style [17:00] online learning where you can actually [17:03] update the weight file itself. So you're [17:05] actually doing a test time training on [17:08] the LLM based on a small amount of [17:10] examples that you just learned which I [17:12] think is actually a huge huge important [17:14] research direction that we should get [17:16] working. Anyway, that's it. How you do [17:20] all [17:23] right so um tonight we have uh three [17:26] authors. Uh Ben got food poisoning this [17:28] morning so he couldn't make it. It was [17:30] very sad but we have three tremendous [17:32] authors. uh Seth uh student under shein [17:37] is I say it right uh at Princeton a [17:39] researcher at Prime Intellect and the [17:41] author of Prime Agent um John Sadvalone [17:44] who's a the first time we've had a a [17:47] call back um pres presenter very excited [17:49] about that PhD under Chris an Aelia um [17:53] and he's going to be he's the author of [17:56] open Jarvis with Ivonica uh from my lab [17:59] uh hazy which is a personal uh open [18:01] Jarvis [18:02] and which I think is really cool. And [18:03] then for Josh and Rean who Josh just got [18:07] promoted to be head of YC labs which is [18:09] really exciting which I think deserves a [18:11] little round of applause as well. [18:13] [applause] [18:14] Um be talking about QM and so QM there [18:18] at YC we had this general agent that we [18:20] use and they came out with QM a month [18:21] ago and it is like meaningfully uh [18:24] better and is so functional and I use it [18:26] every day. All right, that's all I got. [18:28] Thank you so much. [applause] [18:35] >> Hi everyone. Uh, great to be here [18:37] tonight. Uh, my name is Seth and I'm [18:38] going to be presenting Prime Agent, [18:40] which is a self-improving ROM harness. [18:42] Um, my job here tonight, I think, is to [18:45] try to convince all of you to take a [18:48] very first principles style approach to [18:52] thinking about how you build your [18:53] harness. So, we're going to be very, [18:54] very basic here to begin with. If you [18:56] think about and I love the uh [18:58] introduction right from of all the [19:00] background literature um which I thought [19:02] was so cool I think is very [19:03] complimentary the how I think about this [19:04] as well um but if you think about just [19:06] the raw LLM itself it's actually just [19:09] this sequential processor that you know [19:11] has some fixed weights and has some [19:14] visible context and it's it's taking [19:16] some tokens in and it's putting some [19:17] tokens out uh to make the next decision. [19:20] Um we don't really think of an LLM that [19:24] way these days. We have a set of files [19:26] that it has access to. We give endless [19:29] programs and tools for it to use. Um, [19:31] you can even message other sessions of [19:33] LLMs that are going on and create sub [19:35] agents in order to do all of these very [19:37] cool things. But, but in its basics, [19:39] it's just tokens in, tokens out. It's a [19:41] neural network making a prediction. The [19:43] harness itself is the layer between the [19:46] LLM and the world that adds things like [19:48] this persistent state tools and compute. [19:51] Uh, here we have a diagram of how we [19:53] think about prime agent. um from the [19:55] human's perspective. So you open up your [19:58] prime agent uh on your computer just [19:59] like you would cloud code, codec, pi, [20:01] etc. Um and it gets you into this agents [20:03] view and this agents view is an overview [20:05] of all the agents that you have going on [20:06] for all your parallel sessions with like [20:08] a very tight like tight summary of what [20:10] you're using and then you can hop into [20:12] one of those and check it out. there [20:14] you're going to be at this root session [20:16] and this [clears throat] root session is [20:17] your basically the project orchestrator [20:20] over all of these different sub aents [20:22] that it's controlling and you don't have [20:23] to ask it to start sub agents it will [20:26] leverage sub agents when they're useful [20:28] um and these are all programmatically [20:30] called because this is all based on uh [20:32] the recursive language model principle [20:34] where everything is in this IPython [20:37] shell um all your tools all your [20:39] memories all your sub aents uh and and [20:42] we expose uh for further coordination [20:44] these uh these messaging paradigms. So [20:46] you can manage all of these um and then [20:49] these agents can directly interact with [20:50] your environment which could just be [20:52] like the programs and files on your [20:53] computer or running an H200 node cluster [20:56] maybe for your auto research. Each of [20:57] these agents are then backed by this [20:59] persistent dam on your computer. Uh this [21:01] is so that you know when you close uh [21:03] your laptop or you you control C out of [21:05] the session, it's still running in the [21:07] background. You have to actually stop [21:08] the session so that make sure that [21:09] you're continuing to running. And then [21:11] we also have these other features we [21:13] exposed from continual harness where [21:15] it's able to provide like live CRUD [21:18] operations on all of the components that [21:20] we mentioned um in order to manage its [21:23] uh memory skills sub agents um [21:26] persistent and and prompt its own system [21:28] prompt persistently. The way I like to [21:30] think about all of this context that [21:32] we're building up is that we have this [21:34] sort of almost like like here I have [21:37] like L1, L2, L3. is like a cache, right? [21:40] It's like what is the most accessible [21:42] information that we're working with and [21:44] at the very like fastest like readily [21:47] available information, you know, the [21:48] models to be able to retrieve that [21:49] really quickly. It's the model weights. [21:51] So, everyone always wants to get all the [21:52] information in the model weights. Um, [21:54] but then we said, okay, well, maybe we [21:55] don't have all the information because [21:56] we don't want to have to fine-tune every [21:57] single time to update because that's [21:59] very expensive. So, we have this uh [22:01] active input context. So, we're using [22:03] lots and lots of tokens on the input. we [22:06] might have some in context examples like [22:08] we've seen previously in order to add to [22:10] these different capabilities but at a [22:12] certain point we run out of context. Um [22:15] and so the very like earliest form of [22:18] harnesses that we've seen that are still [22:20] used to this day even by those who say [22:21] we want the most minimal harness [22:23] possible is compaction because [22:25] compaction is a very generalized tool [22:28] for the agent to be able to uh summarize [22:31] its own context history in order to work [22:33] past its context length working window. [22:36] You can think about um once we go beyond [22:39] like what are directly like inputs and [22:40] outputs from the the model here into [22:42] this L2L3. You might be familiar with [22:44] the L3 which is more of the dispatch [22:46] state. So if you're working with a file [22:47] system, you can read and write from uh [22:50] main memory. Uh if we're at the L2, [22:52] which is I could think uh at a means in [22:55] between uh what the active context is [22:57] and working with your file system, you [22:59] might have a live uh ripple, which could [23:02] just be running things directly in [23:04] Python uh an IPython shell like you're [23:06] in a Jupyter notebook. And all of those [23:08] variables are saved directly in your [23:10] RAM. your agent can then [23:11] programmatically manipulate them and run [23:13] all sorts of programs directly on the [23:14] information there saving tons of tokens [23:16] rather than putting it directly into [23:18] context. Uh you can also create sub [23:20] agents and it's the same thing you're [23:21] basically saving context here because [23:23] you can task the agent with a specific [23:25] set of information in order to perform [23:27] some operations and the report back at [23:29] the end. What is interesting here is [23:31] what we talked about compaction for the [23:34] active context, right? You have your [23:36] context history. This is helping to [23:38] update it over time. So you can continue [23:39] to leverage this. But once we go beyond [23:42] this, we need to be thinking about how [23:43] are we doing these update. We talked [23:45] about CRUD. How are we do beyond just [23:47] creating reading? How are we updating [23:49] and deleting our context over time [23:52] beyond our uh or the state over time [23:55] beyond the context line. Uh I like to [23:57] think of this at the ripple is this [23:59] aentic garbage collection where we're [24:01] just cleaning up the variables in our [24:02] state as well as like what sub agents [24:04] could be used. And then so to make sure [24:06] that our RAM doesn't crash my computer [24:08] laptop every day. And then on top of [24:10] that, we have this uh notion of [24:11] refinement where we're updating and [24:14] deleting the skills and memories and [24:16] prompts that are stored on your system. [24:17] You can think of that so that way you [24:19] don't crash your actual uh out of space [24:21] on your hard drive as well. And so this [24:24] very much is a here's how we express [24:26] this thing and here's how we revise it [24:29] over time. The other perspective that I [24:30] really like to think about and I really [24:32] trying to push because harnesses are [24:34] almost going towards this like agentic [24:35] operating system that we're creating uh [24:37] is when I think of it um more [24:39] metaphorically here is that when you [24:42] look at the raw LLM it kind of looks [24:44] more like a touring machine where you [24:45] have this ticker tape uh and you have [24:47] all these instructions that are going in [24:48] and then it's performing some set of [24:50] operations and going out. But when you [24:52] look at a harness it's looking a lot [24:55] more vono like a vonoyman computer. [24:57] you're able to do these read and write [24:58] operations on external memory and that [25:01] makes it much more powerful and another [25:03] class of problems than just what a [25:05] touring machine is able to express on [25:06] its own. And so yeah, the idea of like [25:08] how do you build a good hardness? You [25:10] want it to be the most expressable thing [25:12] you can imagine. So some some like early [25:16] harnesses before it gets into the data [25:18] flywheel where the models can do [25:19] themselves are very specific. Plan, act, [25:22] critique, do these exact specific um [25:25] steps. Well, now har uh the models are [25:28] able to do that themselves. You can [25:30] imagine like we we don't have like a [25:31] react loop that we necessarily need to [25:33] explicitly impose. The models kind of [25:35] have natively uh figured this out. But [25:37] what they haven't figured out is how to [25:39] um you know they have to be able to have [25:41] the expressibility to call compact. They [25:43] have to be able to have a Python ripple [25:45] so they can run programs. Um they have [25:47] to have the ability to programmatically [25:49] create sub agents and access state and [25:51] have different feedback mechanisms. Th [25:53] those are model controlled [25:54] expressibility features and if you [25:56] removed one of those you're actually [25:57] removing a capability that it won't be [25:59] able to do otherwise. The way we manage [26:01] um and I'm sure you're all familiar with [26:02] the RLM paper uh from my co-author Alex [26:05] um fantastic bit work. What we do beyond [26:08] what was in the RLM paper is we think [26:10] about the age sub aents as these [26:12] persistent subsessions. [26:14] So each the parent station can create uh [26:17] spin up a new RLM sub aent and each of [26:20] these are then emitted. They run some [26:22] task and then they finish and report [26:24] back to some end state to the parent [26:27] session. These are then idle. They're [26:28] still working in your RAM. At any point, [26:30] the parent session can then send a [26:32] message to one of the sub aents to [26:35] continue working and it has all that [26:36] good context that you built up over time [26:38] so that you're not missing information [26:40] or have to reuse information that was [26:42] already developed in a prior context. [26:44] And then of course you know we don't [26:46] want to use a lot of RAM. So we can move [26:47] them offloaded uh in an inactive state [26:50] which then can be called back at any [26:51] time by messaging them in this [26:53] persistent sub agent setup. I talked a [26:55] little bit about uh continual harness uh [26:57] which we have in a a prior paper of mine [26:59] which talks about cuding the entire uh [27:02] harness state. Um this is another [27:04] feature that we want in our coding [27:05] agents leverage all of our prior [27:08] history. So you can imagine like some [27:09] set of trajectories where they have some [27:11] actions and outcomes or something [27:12] happened um at a at each turn. And so we [27:16] just kind of want to expose the ability [27:17] for the agent to leverage all that [27:19] information in order to update what the [27:22] future harness is going to look like. [27:23] Are we do we need to change our system [27:24] prompt? Do we need to create some [27:26] skills? And skills I think of as a set [27:29] of instructions or a program in order to [27:31] to achieve some some specific goal. uh [27:34] memory which could just be long-term [27:36] storage about things that are important [27:38] as well as the sub aent specifications [27:40] that we talked about in this very [27:41] persistent manner. Were there uh certain [27:43] sub aents that we want to reuse at a [27:45] later time because the context is useful [27:48] and just having the ability to do this [27:49] kind of reflection or refinement um over [27:52] time. It's very powerful for the models [27:55] to have. They're not perfect at this [27:56] right now, but this is one of the the [27:58] capabilities that you want to you want [28:00] to build your harness such that it is a [28:03] bit better than what the current models [28:05] are able to do. So then you can get [28:06] those reasoning traces and use that to [28:08] leverage your next iteration of model [28:09] and they'll be able to handle the [28:12] harness and be able to bootstrap [28:13] themselves into a higher and higher [28:14] performance. One of the coolest features [28:16] that we have in uh Prime Agent um that [28:18] we we've had since the beginning of when [28:21] I was working on this, this is one of [28:22] the first things I added um is the [28:24] ability to message between any any two [28:27] agents um within like some nuclear [28:29] family setup, parents, children, uh [28:31] siblings. Um and the reason why I did [28:34] this is because I was I was constantly [28:35] trying to figure out what's the best way [28:36] to like myself to manage all of the [28:38] agents I have doing everything for me in [28:40] five different directions, five billion [28:42] different directions every day. Um, and [28:44] it would be so much better if they could [28:45] just like share their contacts directly [28:46] with each other and coordinate. And [28:48] turns out that's fantastic for like [28:50] typical software engineering and long [28:52] horizon jobs as well. Uh, the last thing [28:54] that we look at when it comes to how did [28:57] we want to design our harness is we were [28:59] really thinking about long horizon [29:01] performance. I want to go run some jobs [29:03] and I don't want to have to babysit my [29:05] agents the entire time and when when I'm [29:06] ready to come back and check in, I can [29:08] check in with them and see what's going [29:10] on. And this is a perspective that I [29:12] also really lack seeing in a lot of the [29:14] evaluations that we're looking at. Uh a [29:17] lot of times if you run a model for not [29:19] enough time or say, oh well the model [29:21] stopped working after this amount of [29:22] budgets, but then this other model kept [29:25] working with using more budgets. Well, [29:26] first of all, you're not even using the [29:28] same fixed expenditure to compare the [29:30] models. But second of all, that could [29:31] also be hiding performance that you're [29:33] missing. Uh the way that I look at long [29:35] horizon performance eval is that I want [29:37] to see what's the practical plateau. At [29:39] what point will we only get incremental [29:42] gains in performance as I throw more [29:44] test time tokens at it? I have a couple [29:45] experiments that I'm going to show after [29:47] we've shared design philosophy here [29:49] about how we created uh prime agent. Um [29:52] we're going to talk a little bit about [29:53] test time scaling and uh our our TI [29:55] results as well as looking at um you [29:58] know does is it actually helpful and why [30:00] is it actually helpful for our [30:01] information management for the ripple [30:03] that we're working on these long [30:04] contexts and then um when we have these [30:06] really really long like almost ultra [30:08] horizon long horizon uh tasks uh how do [30:11] we sustain these like multi-day work and [30:13] like what actually goes on when we have [30:15] these refinements um over like these [30:18] settings that can last like a week at a [30:19] time or more. So, this is a result that [30:21] you probably all seen. We actually have [30:23] a one additional data point that we [30:24] added here that we didn't include in our [30:26] original result uh just to compare [30:28] across harnesses. We solved this uh we [30:30] went out, we're trying to figure out [30:31] what is the the best uh eval that people [30:34] care about these days when we're running [30:35] our harnesses and we're like, "Oh, we [30:37] should do RKGI." I like, "Oh, yeah. [30:38] Yeah, I remember. I I ran some results [30:40] with continual harness and we got 20% [30:42] with um Gemini Flash uh or sorry, Gemini [30:45] Pearl." Um so, I I think we can get at [30:47] least 20%. people who think that's [30:48] really cool that our like general [30:49] harness that didn't even like wasn't [30:51] even structured for ARHI did really [30:53] well. So I went online I was like okay I [30:55] need to find a good system prompt [30:56] because I don't want to make sure that [30:57] we're losing information. So I found a [30:59] another community leaderboard called [31:01] prolong and I just grabbed their system [31:02] prompt and I was like okay I'm going to [31:03] grab their system prompt forget the rest [31:05] and I'm just going to throw this [31:06] directly into prime agent. Uh and then I [31:08] ran this and I was like oh my god the [31:11] first run that I got it hit 99.9% [31:14] and then I looked at the logs and I was [31:15] cheating. Okay. So, I was like, "Okay, I [31:17] got to do proper sandboxing here. Like, [31:18] let's set this up properly." Uh, and [31:20] then so I spent another day on this. And [31:21] then, and then I went back and I was [31:23] like, "Oh my god, [31:26] I got 78% with GPT soul. Like, this is [31:28] going to be a great result." Um, and [31:30] then we're back. It's like, oh, let's [31:31] let's compare a couple other ones. And [31:33] so, it's again, we just took the prompt, [31:35] uh, general prompt that basically says, [31:37] uh, use a world model to solve ARC AGI [31:40] 3. Uh, here are the actions that you can [31:42] take. um you have uh and then the [31:44] general system prompt for prime agent [31:45] which is like you have a ripple you can [31:46] call sub agents uh you can use the it [31:49] programmatically um and and so we went [31:51] through I went through the traces and [31:52] it's basically doing a bunch of [31:53] different um like calls of the coding in [31:56] order to like check out these different [31:58] scenarios and analyzing the images and [32:00] doing like image processing and it's a [32:01] lot of really cool um stuff that uh it [32:04] seems like it was doing reasonable [32:06] reasoning while leveraging the the [32:08] ripple that we had um as like one of the [32:10] main things that was able to enable [32:12] build this. Uh so I went through and I [32:13] ran a couple other ones. We did GPT tero [32:15] 25.7% which is really cool. You can see [32:17] that compared to like what were the um [32:20] like the week before we did this uh open [32:22] AAI was like the guys the harness [32:24] matters a lot when you're doing [32:24] evaluations. We use the responses API. [32:27] This is the result that we got. Um and [32:30] we we ran Terra and and got almost like [32:34] we we didn't run to completion this one [32:36] but we got really good results in [32:38] comparison. And then we go um that that [32:41] we're already achieving higher than some [32:43] of like the GBT soul extra high which [32:45] was crazy. And then uh we went and we [32:48] did Opus and hit 95.5%. We're like [32:50] that's insane. We also compared to a lot [32:53] of the other harnesses. So some people [32:54] ask me like did you run this with cloud [32:55] code? Uh I did. Um unfortunately the [32:58] results weren't very good. Um, and so [33:00] rather than having bad results, I just [33:02] deferred to the the original cloud code [33:04] results and some other people have run [33:05] it uh with similar configurations to [33:07] prime agent and gotten much better [33:08] results since then. Um, but what's [33:10] interesting is that a lot of the really [33:12] popular harnesses don't necessarily do [33:14] well when prime agent does well. So like [33:16] for air agent, um, we spent a lot of [33:18] money very quickly and uh, we had to cut [33:20] it off because I spent like $5,000 [33:22] without making much performance. Um, not [33:24] saying this is the best they could do, [33:25] but it cost a lot of money to do so. Uh [33:28] so I think that the cost to performance [33:31] uh ratio is very important and one of [33:32] the things that does save money is being [33:33] able to programmatically work with your [33:36] context. Uh we ran a bunch of long [33:38] horizon um evals as well like oolong and [33:42] some coding uh emulator bench which is [33:44] going to come out soon which is a [33:45] program bench alternative and we found [33:47] that it was mainly parody or slightly [33:49] better than these other harnesses like [33:52] you across different models versus doing [33:54] like pimono cloud codecs with glm 5.2 to [33:56] Opus 5 and 5.6 as our setting. Another [34:00] one I thought was really cool is we have [34:01] this like program bench alternative [34:04] called emulator bench where we're trying [34:05] to reproduce entire emulators of [34:07] computer systems or in this case [34:09] creating like a Game Boy Color and check [34:11] that out. And we found that what's [34:13] really interesting is because it has [34:14] this um ripple access in the RLM, it's [34:18] able to use these programs in order to [34:20] kind of do these like out of experiment [34:22] uh loop designs in order to um try [34:26] things out in a lot more expressable and [34:29] free way before submitting the final [34:30] solution to the greater. Uh we also [34:32] tried this with uh GPU kernels um and we [34:35] got about par results uh across [34:37] different um both soul and kimico. One [34:39] is better, one is worse. about par um [34:42] which so we we're not overfit to like [34:43] any one particular um evaluation here. [34:47] Um what's interesting for the long [34:49] horizon stuff is we had some auto [34:51] research uh experiments that we did with [34:54] the nano GPT speedrun but we scaled it [34:56] up. We said let's give it uh 8 by H200 [34:59] for um a week and see what happens. And [35:02] you might be like, okay, prime age is [35:04] going to do so much better, right? [35:04] Because it's able to do all this [35:05] programming. Uh, it's a little high [35:07] variance. We can't attribute um any of [35:09] the benefits to with the harness versus [35:11] the model there because it's a very hard [35:12] task. But what we can do is inspect a [35:14] lot of the behavior that we've seen. And [35:16] what's really interesting is that we're [35:18] seeing models like deep 6v4, GLM 5.3, [35:20] and Kim K3. Um, you can tell these were [35:22] done a little more recently than our [35:24] first results. Uh, and we took these and [35:27] they were doing like what we call out of [35:28] loop experiments. So we were trying to [35:30] say how can I run experiments on like [35:32] the CPU and like look at the [35:33] parameterization and do hyperparameter [35:35] search and analyze the data so that I [35:38] don't have to spend like all my time [35:40] running expensive H200 experiments uh [35:43] because that takes the majority of the [35:44] time. So it's it's running experiments [35:46] that are not the main experiment in [35:48] order to optimize them. I think that's [35:49] really cool behavior that we're seeing [35:51] uh as we we shape what would be what [35:54] kind of things we need to for the [35:56] expressability for prime agent. So you [35:57] can use like really good auto research [35:59] because you can imagine if it's good at [36:00] auto research, it'll be good with you. [36:02] It be even better with a human in the [36:04] loop to bootstrap your experiments. Uh [36:05] and finally, we also streamed a 7-day [36:07] factorial run which used a total of 633 [36:11] agents um across uh 23 million tok [36:16] output tokens in order to make like [36:18] steady uh tech technological advancement [36:20] across the tech tree to continue to [36:22] progress over time. And here it uh one [36:25] of the main benefits is I can use like [36:27] these sub aents that can divvy up into [36:29] different tasks in the factory in order [36:31] to research and build and gather [36:33] resources and build the next items to [36:35] design the factory. Um as well as it can [36:38] use the refinement to leverage what [36:40] happened in the past in order to help in [36:42] the future um over these very long [36:44] context so it doesn't get stuck. And one [36:46] of the most interesting things here is [36:47] that it does not get stuck and it [36:48] continues to make technology progression [36:50] even at the end of our uh stage. Um this [36:54] is more like a Gemini plays Pokemon kind [36:55] of uh conclusion here. If there's one [36:57] thing that uh I find interesting today u [36:59] but like what takeaways you should [37:00] actually add to your own harness. Um I [37:02] think that you should think about [37:03] agentic context management. Uh you [37:06] should think about swarms and looking [37:07] into further depth RLMs and trying to [37:10] run standardized eval. All of the [37:11] results that we can they showed today [37:13] can be run with our uh verifiers uh [37:15] package that we have at Prime Inslect. [37:18] Um and shout out to my collaborators who [37:20] are fantastic and I love working with. [37:22] Thanks. [37:24] [applause] [37:27] >> All right, next up we have John. [37:31] >> Hey everybody. I'm super excited to talk [37:33] about a project um that we've been [37:34] working on at Stanford. Um, I've been [37:37] working on this with Ivanka Orion, my my [37:39] co-lead author, as well as our adviserss [37:41] Hazeni and Christopher Ray. So, personal [37:46] AI is everywhere, but it's mostly [37:47] cloudbound today. Uh, we see lots of [37:49] different harnesses and projects focused [37:52] on making daily writing, research, [37:54] coding, and scheduling. But projects [37:57] like OpenClaw and Hermes agent typically [37:59] rely on cloud LMS um for most of the [38:02] intelligence and for most of the most of [38:04] the queries. Um, what does this mean? It [38:07] means that it's pretty costly. You're [38:09] getting thousands and thousands of [38:11] dollars in API costs if you aggregate it [38:13] over a year. Um, it's not private. [38:14] You're often sending your most personal [38:17] um, data to LMS up in the cloud and you [38:20] don't necessarily know where all that [38:21] data is going. Um, it also requires you [38:23] to rent your intelligence as opposed to [38:25] just simply owning it out of the box. [38:26] And finally, it tends to consume orders [38:28] of magnitude more energy than just [38:30] running these LMS on your laptop. And so [38:33] the local LMS are finally good enough to [38:35] actually run a lot of these queries that [38:37] people care about. And so we see that um [38:40] the the current LMS of today are only 6 [38:43] to 12 months uh behind whatever is the [38:45] state-of-the-art frontier models um of [38:48] before. So you see um LMS today such as [38:50] Quen 3.8 27B um that achieve roughly the [38:54] same performance as like Claude 4.6 Opus [38:57] um back in the day. So that was kind of [38:58] the state-of-the-art model back in [38:59] August 2025. Um, and that gap seems to [39:02] be closing uh more and more as the [39:05] hardware accelerators that we have um [39:07] for our laptops and for our workstations [39:09] get better and better. Uh, just this [39:10] week we saw a new release from Apple um [39:12] with the new Mac Mini. And so we're [39:14] seeing this renewed focus from Apple as [39:16] well as Nvidia to build accelerators [39:19] specifically for personal use cases. And [39:21] so with this project, we wanted to [39:23] explore the the main question of can we [39:25] build the core of a personal AI stack, [39:27] namely the model inference, the agent [39:30] execution, the memory, the learning, [39:32] basically the parts that are mostly [39:34] reliant on the cloud today entirely on [39:36] device while staying competitive with [39:38] these cloudonly stacks. And so we [39:40] decided to propose open Jarvis. Um name [39:43] needs no needs no explanation. Um but we [39:45] wanted to explore just how much of this [39:46] we could run on device completely for [39:48] for free uh while preserving uh [39:51] security, privacy and quality. And so to [39:54] construct open Jarvis we wanted to [39:55] create the simplest set of primitives [39:57] for which you define any sort of harness [39:59] or or personal AI stack. Um the first [40:01] one is whatever user interfaces you need [40:03] to use. Um the second one is the actual [40:05] agentic logic around composable [40:07] reasoning and using different kinds of [40:09] intelligence and tools. Uh for the [40:11] intelligence, it's whatever LM you're [40:12] using as your engine for keeping [40:14] everything going. Um so this could be [40:16] Quen, GBDO, OSS, Gemma 3N. Um and then [40:19] whatever actual inference engine you [40:21] need to run it. So this could be O Lama, [40:23] um Llama CBP, VLM, SG Lang, um including [40:27] whatever hardware you're running it on. [40:29] So this could be Apple Silicon, Nvidia, [40:31] whatever you need. um for actually [40:33] making all of these agents and [40:34] intelligence useful you need some set of [40:36] tools in memory that can be run through [40:38] a standard MCP protocol um and you need [40:40] some sort of uh set of primitives for [40:42] actually doing learning whether it's [40:44] prompt based techniques like Japa or [40:46] DSPI um whether it's weight based [40:48] techniques like gpo and sftt and Laura [40:51] um you need some way to actually get [40:52] this agent to improve over time and [40:54] actually be able to make it more [40:55] personal and more effective and so to [40:57] kind of walk through like what opens [40:59] looks like um we tried to go with all of [41:01] the standard um interfaces that people [41:04] are already accustomed to. Um so we [41:06] wanted to give people the ability to [41:07] interact with it through a desktop and [41:09] actually just run it as they would [41:10] normally expect, but then see all of the [41:12] savings that they're getting in terms of [41:13] dollars and energy. Um we also wanted to [41:16] give people the ability um to run [41:18] different kinds of continuous agents. So [41:21] different kinds of agents that are [41:22] persistent in terms of cron jobs and [41:24] being able to run standard protocols um [41:26] day after day. Um, basically we just [41:28] wanted to to make this like [41:30] plug-and-play with all of the workflows [41:32] that people are already accustomed to [41:33] running. Um, and we wanted to make this [41:35] something that can get people to have [41:37] their first experience with LMS on [41:39] device the same way people had their [41:41] first experience with ChatGpt or Claude [41:43] back in the day. Um, so yeah, and so [41:46] yeah, to step through a little quicker, [41:47] but yeah, here's like a nice way to like [41:49] set up new persistent jobs. Um, yeah, we [41:52] have all of these different components. [41:54] We need some way to actually optimize [41:55] it. And so we wanted to get out of the [41:57] way of the LM as much as possible by [41:59] just creating a simple spec of these [42:01] five primitives by which they could go [42:03] through the optimization. And what we [42:04] found is that by going through this [42:06] whole optimization loop, we were not [42:08] only able to get significant dollar [42:09] costs uh dollar cost reductions, but [42:11] also significant latency reduction and [42:14] significant um improvements to overall [42:16] quality on these tests. And so this this [42:18] configuration is meant to simplify down [42:20] to just the five main things that people [42:22] care about when they're building these [42:23] these LM uh harnesses. So the [42:25] intelligence, the engine, the actual [42:27] agentic logic around it, the tools or [42:29] learning systems required for running it [42:31] um and the whole optimization um for the [42:34] whole spec as a whole. And so something [42:35] that we thought could be interesting to [42:37] help bridge this gap between local and [42:39] cloud LMS is to actually have the cloud [42:41] LM go through and manual and uh and [42:43] automatically optimize the whole LM the [42:46] whole local stack. And so this is a nice [42:48] way of taking advantages of the [42:49] capabilities of cloud LM to diagnose [42:52] proposed changes and gate um to create [42:55] improved solutions for these local LMS [42:57] while not incurring the cost of those [42:59] cloud LMS when you actually deploy these [43:01] um local stacks at inference. And so [43:04] what we found is that these uh these [43:06] open Jarvis jobs that were these open [43:08] Jarvis um configurations that were [43:10] actually optimized by cloud LMS like [43:12] cloud or chat GPT um were much more [43:15] effective than uh local stacks that were [43:17] just deployed out of the box because you [43:19] could actually cater to the specific LMS [43:21] the specific harness uh that was needed [43:24] uh for for different kinds of workloads. [43:26] And what we found is that even with the [43:27] ondevice LMS of today, we can rival [43:30] cloud LMS on different workflows around [43:32] personal AI um personal use cases, [43:35] coding, agentic tasks. While there [43:37] remains like many tasks for which um [43:40] like local local size LMS are not enough [43:42] um the gap is surprisingly closing um [43:45] month after month um as these LMS become [43:48] better distilled, more effective and [43:50] also we get better accelerators um for [43:52] running them. And so even with LM of [43:54] today, we can get 800x lower lower [43:57] costum in terms of actually running them [44:00] as well as a significant reduction in [44:01] latency. Um what we also found is that [44:03] no matter which cloud LM that we cloud [44:05] LM we picked um it was useful um in [44:08] terms of optimizing the whole aentic [44:10] loop for these uh local local open [44:13] Jarvis configurations. Um we found that [44:15] um the Opus series, Opus 5 as well as [44:18] GBD 5.6 Soul were were naturally the [44:20] best. Um but was interesting to see is [44:22] that you could pick Gemini, you could [44:23] pick um other other uh larger um LM [44:27] families like Kimmy and GLM and use them [44:30] to optimize these local configurations [44:32] so that you could capture those [44:33] efficiency gains, capture those [44:34] performance gains um for local inference [44:37] later. What we also found is that the [44:38] whole open Jarvis harness was cheaper to [44:40] optimize than alternatives which might [44:43] might require more data or more LM [44:45] calls. Um we found that like this set of [44:47] specs um and this set of primitives um [44:49] was most effective for local LM settings [44:51] because it got um the whole optimization [44:54] loop and the whole um set of LM [44:57] abstractions out of the way of the cloud [44:59] LM to just optimize the whole system and [45:01] just make it make it really fast and [45:02] really effective. Uh looking forward, [45:04] we're excited to keep building out this [45:05] project. Uh we think in the very near [45:07] future you're going to see um a huge maj [45:10] a huge proportion maybe even a majority [45:12] of uh people's daily inference calls [45:15] going to local devices and on-prem uh [45:17] laptops or on-prem workstations as [45:19] opposed to the kind of standard of today [45:21] where everything's being pushed out to [45:22] the cloud. Um we think these trends are [45:24] only going to continue because the [45:25] accelerators keep getting better and the [45:26] LMS keep getting better. Um and so if [45:28] you're excited about um anything in the [45:30] stack, whether it's uh better local LMS, [45:33] better accelerators, better inference [45:34] engines uh for deploying uh beyond data [45:37] centers, um please reach out. Uh we'd be [45:39] excited to chat. Include a QR code of [45:41] the project. Um if if folks are around [45:43] here afterwards, we'd love to chat. [45:45] Thanks. [applause] [45:49] All right. Now, we have our own YC. We [45:52] have Josh and Rean. [45:58] Hi, I'm Josh and uh this is Rean and [46:01] we're working on QM, which is YC's [46:03] open-source [46:05] uh agent harness for work. QM is one [46:08] system that uh gives every employee at [46:10] YC an open claw-like assistant uh that's [46:14] like fully customizable and available in [46:17] Slack or via web UI, which is uh what [46:20] we're looking at here. Um each person [46:23] works within QM in their own personal [46:26] context that has its own sandbox files [46:30] uh and crons and it can they can also [46:34] work with QM in a multiplayer setting [46:36] like a slack channel. People use QM for [46:39] a pretty broad range of things uh like a [46:43] lot of automations like email triage [46:45] like legal and finance workflows. It's [46:48] really good at editing documents and [46:49] pulling data out of our internal [46:51] database. Uh it can also spin up live [46:54] internal web apps and help with stuff [46:57] like planning events. Uh but it's [46:59] designed to be broadly helpful for the [47:02] range of tasks that someone might [47:04] encounter at YC uh on a day-to-day [47:07] basis. [47:09] So you might be wondering uh why we [47:13] built this and it's really the result of [47:16] a string of internal agent projects that [47:19] have kind of unwound throughout the [47:21] years and all of which were really [47:24] riding this tailwind of increasingly [47:26] capable models. Um [47:29] the first one we built was in like [47:32] January of 2025. We internally refer to [47:34] it as the quote unquote like general [47:37] agent. Uh, but it was pretty [47:39] straightforward. Just kind of a system [47:40] prompt with tools uh in a loop. It was [47:44] one sizefits-all. [47:46] Um, [47:47] sort of like everyone was talking to the [47:49] same thing. Uh, and it was pretty [47:51] straightforward architecturally, but [47:53] like still uh very or surprisingly good [47:57] at answering data questions. Uh, [48:00] interestingly like the scope of what the [48:02] general agent was good at just increased [48:05] I guess unsurprisingly as the under [48:06] underlying models got better. Uh, and we [48:09] eventually hooked it up to Slack. We [48:10] added crons uh and gave it a a few more [48:14] tools so that it could be more be [48:16] capable across more domains. Um, in June [48:20] of 2025, uh, [48:23] by then like a lot of our engineers [48:25] started using cloud code and codecs. Uh, [48:28] and we realized that you could pretty [48:29] easily run these in a VM. And then, uh, [48:32] we hooked that up to a Slack tag, which [48:34] was a pretty powerful medium for people [48:36] who just wanted to like run a one-off [48:38] code change. Uh, we also configure [48:41] configured it to run our CI pipelines [48:43] and then spin up dev environments for [48:46] testing. Uh, and so people could come in [48:49] like describe a bug or something they [48:51] wanted to see happen and the bot would [48:54] go off and actually solve it, which is [48:55] like a pretty powerful um thing for [48:58] someone who like maybe hadn't uh made a [49:01] code change before in their life even. [49:03] Uh but we also on top of that had a [49:07] small loop going where we would observe [49:09] sort of how the bot failed uh where it [49:12] went wrong and then update the [49:14] agents.mmd which was uh present in the [49:17] codebase at the time um to make sure [49:20] that the thing got better as as uh we [49:23] like observe the usage and so in January [49:26] of this year a lot of the partners [49:28] started using openclaw and one thing to [49:30] know about YC partners is that they're [49:32] incredibly busy [49:33] uh between like office hours uh they get [49:36] tons of inbound email they're always [49:38] reading applications so like any tools [49:40] that can give them additional leverage [49:42] are incredibly valuable to YC so open [49:45] cloud particular was useful because it [49:47] was the first agent that a lot of them [49:49] had used uh that had their own computer [49:51] that had its own computer and so this [49:53] made it like very customizable in a way [49:55] that the previous paradigm of agents was [49:57] not uh and it functioned almost like a [50:00] personal assistant [50:02] And so in April uh the question became [50:04] like could we provide this to every [50:06] employee at YC uh without uh like buying [50:10] everyone a Mac Mini effectively. And so [50:12] we ended up provisioning a fleet of like [50:14] 50 plus uh Hermes agents that were [50:17] running in VMs. And these were [50:20] definitely pretty helpful but they [50:22] required a lot of configuring for people [50:24] to get value out of them. And it was [50:27] just like inherently kind of difficult [50:28] to manage this fleet. Um, it was sort of [50:32] like a whack-a-ole situation where I [50:34] would have to sort of like SSH into [50:36] these individual instances and fix them. [50:38] And so the follow-up question became [50:40] like we've got we've gotten a lot of [50:43] value out of these agentic systems. Uh, [50:46] like let's build something that tries to [50:48] address some of the downsides of running [50:50] this big fleet of Hermes agents. Uh, [50:52] while still maintaining the [50:53] personalizability and some of the like [50:56] the stuff that people were really [50:57] getting value out of. [51:00] And so, [51:04] >> um, yeah. Okay. So, basically, uh, [51:06] there's a pretty clear trend from [51:08] whether you could see there from what [51:09] Josh was showing you. Basically, um, the [51:12] models are getting better exponentially. [51:14] Um, and we were starting to see just [51:16] increasingly impressive returns from [51:18] giving them more and more capabilities. [51:19] So, OpenCloud gives the agent its own [51:21] computer and we start to see really [51:23] impressive returns from that. So around [51:25] May this year, we started thinking just [51:27] like how far can we push this if we just [51:28] keep pulling on this threat. Um we [51:31] really like the the lens of sort of [51:33] unhobling. I don't know if you guys uh [51:35] read situational awareness when it came [51:36] out in like 2024. Um but that was kind [51:38] of an era when like test time compute [51:40] was just starting to become a thing and [51:42] like you know we're starting to give [51:43] agents tools for the first time and [51:45] there's sort of this intuition that what [51:47] agents can do like there's a little bit [51:49] there's kind of more intelligence in the [51:50] models than we're than we're using in a [51:52] lot of cases. Um, and it's like if we [51:54] really push uh the frontier in terms of [51:57] just like what we're offering up the [51:58] agents as as capabilities that they can [52:00] make use of uh like magic can start [52:02] happening. Um so the first way that we [52:06] do that with QM um is by essentially [52:09] pulling the brain of the system up out [52:12] of the sandbox. So with uh you know with [52:15] Hermes and with OpenClaw you effectively [52:17] have um the agent has its own computer [52:20] which is super powerful but it's also [52:22] trapped inside that computer. So that [52:24] causes a few issues just from you know [52:26] uh us trying to administer that system [52:28] when there were you know uh even a few [52:30] dozen of these things it starts to [52:31] become unwieldy almost immediately. Um [52:33] but the other issue with that is that [52:35] you um all all of the like all the [52:38] sessions that you would have uh they're [52:40] trapped inside that computer. And so [52:42] what we did instead is we uh we just [52:45] offload everything into Postgress. So [52:47] everything is centralized um from all [52:48] the agent conversations that people are [52:50] having. Um and then we expose those to [52:52] the agent itself. So it can look at all [52:54] the context that's sort of aggregating [52:55] from across the system. And then the [52:57] other thing we do is we start thinking [52:59] about sandboxes less as this home where [53:02] the uh where the agent lives and where [53:04] it's kind of stuck in a lot of ways. And [53:06] sandboxes become more of this thing uh [53:08] more of a resource that the agent can [53:09] dip into and use as needed. Um but it's [53:12] it's um it's a lot less limiting. Uh the [53:15] other thing that this starts to open up [53:17] um is this idea of you're accumulating [53:19] this large eval set of all the traces [53:20] that you have from the conversations [53:22] that people are having with the agent. [53:23] And in principle, you can think about [53:25] going and and hill climbing on that um [53:27] and sort of having this automated [53:28] improvement loop. Um we've had sort of [53:30] mixed results with that. I would say I [53:32] think typically if you're just [53:34] dispatching this like torrent of agents [53:36] that are supposed to um fix all of the [53:38] bugs that they're encountering when you [53:40] have the LLM as a judge, you start to [53:42] get this kind of uh like main character [53:44] syndrome where the agents are making [53:46] fixes that are, you know, only seeing [53:48] their their um their piece of the [53:51] elephant effectively. like they're [53:52] they're really um they can be sort of [53:55] yeah not not seeing the whole um whole [53:57] system and so having the human in the [53:58] loop there has continued to be really [53:59] important although we're really looking [54:01] forward to this uh working uh all the [54:04] way around. Um so the other major thing [54:06] we do that's pretty obvious is just wire [54:08] the agents to all of the resources [54:09] across the company that we can. Uh we [54:11] already happened to have a CLI at YC [54:14] that worked really well um that wired a [54:16] lot of systems together. Um uh but [54:20] anything that wasn't in there uh we [54:21] basically allow um adding just arbitrary [54:25] API keys um that sort of thing. And then [54:28] we also want to ensure that we have par [54:30] with uh just an employee working on [54:32] their laptop. So you know device code [54:34] OOTH [54:35] um we go ahead and ingest that into a [54:37] keychain and then refresh it for you. So [54:39] it's it's uh ideally supposed to imitate [54:41] the experience of a person on their [54:43] computer. We mostly keep this to to be [54:46] read only uh in the database but we do [54:48] allow for rights um via uh human [54:52] reviewed bulk upserts. So the way that [54:54] works is the agent will put forward a [54:55] plan to um edit the database um that a [54:59] person can kind of give a once over and [55:00] ensure it's not doing anything crazy [55:01] before the write actually happens. Um [55:04] one thing we've observed with this is [55:05] that we've started just kind of rubber [55:07] stamping these. It's a little bit like [55:09] um I think if you guys use cloud code in [55:11] the early days like you might have been [55:12] reviewing the tool uses very closely and [55:15] eventually um you sort of build up more [55:17] trust in the agent. So this is something [55:18] that we're uh looking at very closely [55:20] over the over the next few months. Yeah. [55:22] So sort of like I was saying um the [55:24] sandboxes in this system we like to [55:26] think of as a resource for the agent. So [55:29] and and less where the agent actually [55:30] lives. Um so in QM the agent can um [55:35] basically uh by default it's going to be [55:37] using a particular sandbox. So that's [55:39] going to be one that's been allocated to [55:41] the user that it's talking to. But in [55:43] general uh there are there are [55:45] environments that the agent can kind of [55:47] converge on and uh and can collaborate [55:49] with. Um, and then the other uh key [55:52] thing is that um if the agent is working [55:56] on like a um like a heavier dev [55:58] workload, it can go and reach for a [55:59] machine that has more resources. Um if [56:01] it's working on something that's [56:02] simpler, it'll just go for a sandbox [56:04] that's that's less powerful. Um and so [56:06] pushing that decision into the agent [56:08] itself rather than the harness has been [56:09] a really um really powerful thing. [56:12] Similarly, um allowing the agent to uh [56:15] tap into its own runtime. So basically, [56:17] if it can choose the provider um that [56:20] it's working with, you start to get out [56:22] of situations like um I'm sure you guys [56:24] have run into this with Fable. If you [56:25] try to do AI research, if you try to do [56:27] cyber security, anything, you'll get a [56:29] bunch of refusals. Um so what we can do [56:31] when the agent can control it on [56:32] runtime, it can just pop out into [56:34] another model uh when it needs to avoid [56:35] a situation like that. Um similarly, uh [56:39] like in the earlier situation, um it's [56:41] often useful to pop between different [56:43] sandbox providers. Um and so that's [56:46] something we can do quite easily. And as [56:47] a general rule, what we've tried to do [56:49] is keep the harness extremely thin. Uh [56:52] we think of the kind of core of the [56:54] system as being these three tools um [56:56] where you have execution in a remote [56:58] sandbox, reading and writing from object [57:00] storage and then uh publishing internal [57:02] apps, a pretty simple sort of getbacked [57:05] uh system. And then we have other tools [57:07] for interacting with memory and crons [57:09] and that sort of thing. But we really [57:11] think of these as sort of temporary um [57:14] papering over rough edges in the system. [57:16] And really the core of it is is these [57:18] three up here. Um it's we we really try [57:20] to keep it as small as we can. Yeah, [57:22] we're we're sort of trying to be this [57:23] like uh AGI anticipating harness. [57:26] Although uh since we aren't there yet, [57:29] um there are a few things that we've run [57:31] into. Um, one of these is that the [57:34] agents have been we've tried to put them [57:36] in this really capable environment where [57:37] they have all these uh tools available [57:39] to them, but um they often give up way [57:42] too early. Uh so one thing we've [57:44] experimented with especially over the [57:45] past month or so has been setting uh we [57:48] call it like a grind tool or um [57:50] basically we set budgets on goals. So [57:53] the agent is not allowed to give up on [57:55] its task before a certain amount of like [57:57] walk clock time. just like a couple [57:59] hours uh or a certain amount of token [58:01] spend. And so um what that can [58:03] accomplish is just like uh really a lot [58:06] better, you know, uh research outputs, [58:08] better reports, that sort of thing. Um [58:10] and it's been really fun actually to see [58:12] uh like OpenAI and anthropic um you [58:15] know, crack some open problems in math [58:16] with like a very similar technique. Uh [58:18] but it also works for just you know, [58:20] normal office work stuff too. Um the [58:22] other thing that we've seen a lot of is [58:24] so this harness is supposed to work uh [58:26] it works in multiplayer. It works in [58:28] Slack. Um, but because of the the [58:30] artifacts of its of its training, [58:32] effectively uh what we see is that uh [58:35] the agent gets can get very confused [58:37] about the situation that it's in even if [58:39] we specify this pretty clearly in the [58:40] system prompt. So having like local [58:42] affordances for this has been something [58:44] that's been uh that's been really [58:45] important. [58:50] So yeah, um the other thing that has [58:53] been a problem is that agents really [58:55] don't uh understand social contexts. Um [58:59] to make this a little more concrete, [59:01] like if I tell Regan a piece of [59:03] information, uh he intuitively sort of [59:05] knows uh or at least has like a good [59:08] mental framework of where it is okay to [59:10] share that information. Uh but it takes [59:13] some actual work to recreate this with [59:15] an agent. uh like privileged information [59:18] can very easily just leak into these [59:20] contexts where it should not be and so [59:24] uh the information that you can put in [59:26] the brain is effectively like bounded by [59:28] how good your permission system is uh [59:30] and so YC luckily has an existing [59:33] software system with like fine grained [59:35] permissioning uh that has been built [59:37] over the years but a lot of people just [59:39] don't have that uh and so it takes uh [59:42] work to allow for knowledge sharing in [59:45] an in nuanced Okay, so thanks everybody [59:48] for listening. Um, you can try out QM. [59:51] It's open source. Uh, and coding agents [59:54] are pretty good at standing it up. If [59:56] you run into any problems, feel free to [59:57] put up an issue and we'll look at it. [59:59] Uh, we're also hiring. So if any of this [60:01] resonated with you or would be exciting, [60:03] then feel free to send us an email. [60:07] [applause]