Andrej Karpathy Warns Software 3.0 Requires Shift From Vibe Coding to Rigorous Agentic Engineering

Andrej Karpathy：软件3.0需从“氛围编码”转向严谨的“智能体工程” 这位OpenAI联合创始人解释了开发者为何必须从随意的提示词编写转向管理随机性AI系统，同时保留人类的品味与系统级理解能力。 Karpathy回顾了2025年底发生的巨大范式转变：随着智能体工作流开始生成无瑕疵的代码，促使开发方式从随意的“氛围编码”（vibe coding）转向严谨的“智能体工程”（agentic engineering）。他提出了“软件3.0”概念，即大语言模型（LLM）作为直接处理提示词的主操作系统，令传统的中间层应用架构走向过时。尽管自动化和生产力实现了巨大飞跃，但AI的智能依然高度“参差不齐”，虽在代码等可验证领域表现卓越，却欠缺基本常识。Karpathy最终指出，尽管底层实现和语法可以外包给AI，但人类的基本功、架构设计、品味和核心理解力依然不可替代。 Karpathy 指出编程范式在 12 月迎来重大转折，智能体工具能够一次性可靠生成完美代码块而无需手动修正，开发流程已从主动调试转向无缝协同的引导模式。 (0:02) 计算领域正迈入 Software 3.0 时代，大语言模型成为可编程计算机，传统中间件应用架构被大幅淘汰，未来神经计算机甚至可能让神经网络成为宿主操作系统、传统 CPU 仅作为协处理器。 (2:28) LLM 呈现出明显的“参差智能”特征，在数学和代码等高可验证性领域表现惊艳，但在基础常识推理上频频失灵；若应用场景超出前沿实验室预训练范围，开发者必须自建可验证的强化学习环境。 (9:41) “Vibe Coding”降低了普通人的开发门槛，而“Agentic Engineering”（智能体工程）则是一门管理随机且易错智能体的严谨专业工程规范，顶级工程师由此获得的生产力提升远超传统的“10倍”概念。 (15:45) 尽管底层实现与低级 API 可以外包给智能体，但架构设计、审美与计算机基础原理依然依赖人类，正如 Karpathy 强调的“思考可以外包，但理解绝无法外包”。 (27:42)

Andrej Karpathy argues that computing is entering a "Software 3.0" era where large language models act as central interpreters and autonomous agents replace manual coding, fundamentally altering engineering workflows and hiring. Software 3.0 and the Shift to Neural Computing (2:28) Karpathy outlines a transition from Software 1.0 (explicit programming) and Software 2.0 (weights learned from data) to Software 3.0, where prompts and context windows direct an LLM acting as a programmable computer. This model can bypass traditional software layers entirely. For example, rather than building an app like Karpathy's "MenuGen"—which ran OCR, image generation, and Vercel rendering via glue code—a modern multimodal model can directly manipulate pixels to overlay images onto a restaurant menu in response to a simple prompt. Looking toward 2026, Karpathy anticipates that the computational architecture may invert: neural networks will act as the host operating system, while traditional deterministic CPUs become co-processors for specialized tasks. In this regime, dynamic user interfaces could be synthesized on the fly via diffusion from raw video and audio inputs. The Irregularities of Jagged Intelligence (9:41) Because frontier AI models are trained using massive reinforcement learning (RL) loops reliant on automated verification, their capabilities develop in an uneven, "jagged" pattern. Models excel at verifiable tasks like mathematics, vulnerability discovery, and refactoring large codebases. However, they struggle with basic, non-verifiable common-sense logic; Karpathy cites an instance where a model instructed a user to walk rather than drive to wash a car located 50 meters away. This dynamic also affects code quality. Because standard RL setups lack aesthetic reward functions, agent-generated code is often bloated, heavily duplicated, and poorly abstracted. Karpathy observed this when attempting to prompt an LLM to simplify training code for his "micro GPT" project, finding that radical simplification failed because it sat outside the model's RL training circuits. From Vibe Coding to Agentic Engineering (15:45) Karpathy distinguishes "vibe coding"—which lowers the barrier to entry by letting casual users build software through conversational steering—from "agentic engineering," which maintains rigorous quality and security standards. Rather than relying on simple code completions, agentic engineers manage stochastic, fallible agents as force multipliers, achieving productivity gains that Karpathy states far exceed the traditional "10x engineer" label. This shift alters how developers work and should be hired. Developers no longer need to memorize specific library APIs, such as tensor dimension syntax, but they still require deep computer science fundamentals to catch architectural flaws. Karpathy points out that during MenuGen's development, an agent attempted to link Stripe payments to Google accounts by matching email strings instead of using persistent user IDs. To evaluate candidates in this environment, Karpathy proposes moving away from isolated coding puzzles toward adversarial tests, such as tasking candidates to build complex applications and defending them against automated LLM penetration attacks. Agent-Native Infrastructure and Human Comprehension (25:16) Current computing infrastructure remains heavily geared toward human interfaces, creating friction when configuring DNS, navigating deployment dashboards, or reading manual documentation. Karpathy envisions an agent-native ecosystem where systems are decoupled into LLM-legible sensors and actuators, enabling agents to deploy services directly or negotiate logistics with other agents autonomously. Despite this operational delegation, Karpathy highlights that human comprehension remains an essential bottleneck. While automated tools can process synthetic information and compile custom personal knowledge bases, users cannot outsource their fundamental understanding of system specifications and objectives without losing the ability to steer the underlying models. Karpathy concludes that while almost any domain can theoretically be automated given verifiable feedback or synthetic evaluation councils, current LLMs still lack genuine understanding, leaving humans as the necessary directors of complex systems for the near future.

Andrej Karpathy 认为，大语言模型正在推动计算体系从手写代码转向以提示词和上下文窗口为控制中枢的“软件 3.0”，开发模式也随之从普通人凭直觉探索的 Vibe Coding 升级为需要专业工程把控的 Agentic Engineering。尽管 AI 凭借强化学习可验证性在代码生成等领域展现出超越传统“十倍工程师”的效能，但在常识推理、审美抽象及端到端业务逻辑上仍存在明显的“锯齿状智能”缺陷，这使得计算机科学底层基础与人类审美判断力依然是不可替代的核心瓶颈。 软件 3.0 与神经网络计算机 (2:28) Karpathy 将大语言模型定义为一种颠覆性的计算新范式。相较于人类手写规则的“软件 1.0”以及依赖数据集训练权重的“软件 2.0”，“软件 3.0”直接把模型视作可编程计算机，提示词和上下文窗口构成了驱动解释器的核心指令。在这种架构下，传统繁琐的中间层应用胶水代码将大量失效。以其自研的“MenuGen”菜单工具为例，过去需要结合 OCR、图像生成 API 及云端重绘代码的多层架构；而在软件 3.0 模式下，用户只需将图片上传给 Gemini 等多模态模型并下达指令，模型即可直接在像素层面完成图像叠加与重绘，使传统应用架构沦为多余的中间媒介。 展望 2026 年，Karpathy 预测计算体系可能发生根本性反转：神经网络将跃升为主机操作系统，传统 CPU 则退化为负责确定性计算的协处理器。届时计算设备可以直接接收视频或音频输入，依靠扩散模型实时动态渲染出完全契合当下上下文的即时交互界面。 智能体工程与开发评估变革 (15:45) Karpathy 指出，2024 年 12 月的模型能力演进是一个关键转折点，智能体生成的代码片段首次达到无需人工频繁纠错的连贯度。这一突破分化出了两个技术层级：“氛围编程”（Vibe Coding）显著降低了非专业人员开发软件的技术门槛，使任何人都能通过宏观引导模型进行创造；而“智能体工程”（Agentic Engineering）则要求在应对具有高度随机性（stochastic）与能力尖刺的智能体时，依然保持专业软件的质量与安全基线。Karpathy 认为，善用这套体系的顶尖工程师所获得的效能提升已远超传统的“十倍工程师”范畴。 这一范式转换进一步迫使传统工程招聘模式进行重构。Karpathy 批评基于孤立算法谜题的传统面试已然过时，未来的评估应当转向实战项目验证：例如要求候选人构建一个供智能体交互的复杂社交平台克隆体，并利用高级模型发起自动化安全渗透与红蓝对抗，以此检验候选人设计架构抵御机器攻击的真实工程韧性。 锯齿状智能与可验证性边界 (9:41) 当前模型能力呈现出极不均衡的“锯齿状”特征。Karpathy 分析，这是由于前沿实验室高度依赖基于强化学习（RL）的环境与自动化验证奖励回路；模型因而在代码重构、数学证明及挖掘零日漏洞等具备明确验证标准的领域表现卓越，但在缺乏自动化反馈的常识推理中频频失手。典型案例是模型在面对“前往 50 米外洗车店是该步行还是驱车”的问题时，会荒谬且笃定地建议用户步行。此外，若特定应用脱离了主流实验室预训练数据的覆盖范围，开发者便无法依赖模型开箱即用的推理能力，必须自建可验证的微调流水线。 这种智能的非连续性同样反映在代码美学上。Karpathy 坦言审查智能体生成的代码时常常感到心悸，因为代码普遍存在冗余堆砌和脆弱抽象；在尝试让模型极致简化“micro GPT”训练脚本时，模型也因欠缺审美奖励机制而屡屡受挫。Karpathy 将大模型比喻为由统计回路构成的“幽灵”，而非具备好奇心等演化生物本能的实体，开发者对其输出必须保持清醒的技术审视。 架构判断力与智能体原生生态 (19:29) 随着智能体承揽了调用语法、API 参数（如张量维度命名规则）等细节琐事，人类核心价值正在向系统架构、逻辑边界与底层原理迁移。Karpathy 在开发实践中发现，智能体曾试图通过直接匹配 Google 和 Stripe 的邮箱字符串来核验用户账户，导致不同邮箱用户无法绑定付费数据；这类缺乏全局常识的荒谬逻辑表明，唯一用户标识等核心约束仍必须由人类设计规范来严格把控，开发者同样必须深入理解张量排布与内存机制以避免潜在的性能损耗。 在更宏观的层面上，数字基础设施正逐步迈向“智能体原生”（Agent-Native）阶段，未来的软件架构与数据格式将直接为模型的感知和执行服务，彻底剥离人类手动配置域名与云服务的繁琐交互。Karpathy 强调，“你可以外包思考，但无法外包理解”；在智能体高度自动化的时代，人类对于系统本质的认知与导向判断，依然是整个工程体系不可替代的终极瓶颈。 Karpathy 总结认为，虽然从长远来看几乎所有任务都有望通过设立可验证反馈（如 LLM 评审委员会）实现自动化，但就当下而言，大语言模型在真正的深度“理解”上依然欠缺质的突破。开发者与研究者的终极挑战在于构建更优的知识整合工具与人机协同中枢，在拥抱智能体爆发式生产力的同时，确保人类对复杂系统的掌控力与认知深度不被技术执行速度所抛离。

[0:02] Andrej Karpathy: vibe coding [0:02] Andrej Karpathy: never felt more behind as a programmer [2:28] Andrej Karpathy: use Nano Banana to overlay the things onto the menu [2:28] Andrej Karpathy: spurious [7:37] Andrej Karpathy: very weird [7:37] Andrej Karpathy: historical appendages [9:41] Andrej Karpathy: This is insane. [14:22] Andrej Karpathy: vague posting [14:22] Andrej Karpathy: Everything is automatable. [15:45] Andrej Karpathy: raising the floor for everyone in terms of what they can do in software [15:45] Andrej Karpathy: agentic engineering [19:29] Andrej Karpathy: This is such a weird thing to do [22:13] Andrej Karpathy: pulling teeth [22:13] Andrej Karpathy: summoning ghosts [25:16] Andrej Karpathy: Why are people still telling me what to do? Like, I don't want to do anything. What is the thing I should copy-paste to my agent? [27:42] Andrej Karpathy: You can outsource your thinking, but you can't outsource your understanding.

[0:02] We're so excited for our very first [0:03] special guest. He has helped build [0:06] modern AI, [0:07] then explain modern AI, and then [0:10] occasionally rename modern AI. [0:14] He actually helped co-found OpenAI right [0:16] inside of this office, was the one who [0:18] actually got autopilot working at Tesla [0:20] back in the day. [0:22] And he has a rare gift of making the [0:24] most complex technical shifts feel both [0:27] accessible and inevitable. [0:30] You all know him for having coined the [0:31] term vibe coding last year, but just in [0:34] the last few months he said something [0:36] even more startling, [0:38] that he's never felt more behind as a [0:40] programmer. [0:41] That's where we're starting today. Thank [0:42] you, Andre, for joining us. Yeah, hello. [0:44] I'm excited to be here and to kick us [0:46] off. [0:47] Okay, so just a couple months ago you [0:49] said that you've never felt more behind [0:51] as a programmer. That's startling to [0:52] hear from you of all people. Um can you [0:55] help us unpack that? Was that feeling [0:57] exhilarating or unsettling? [1:00] Uh yeah, mixture of both for sure. [1:02] Uh well, first of all, um [1:05] I guess like as many of you I've been [1:06] using agentic tools like Alpha code [1:08] adjacent things uh for a while, maybe [1:09] over the last year as it came out. And [1:11] it was very good at, you know, chunks of [1:13] code. And sometimes it would mess up and [1:15] you have to edit them, and it was kind [1:16] of helpful. And then I would say [1:17] December was this uh clear point where [1:20] for me uh I was on a break, so I had a [1:22] bit more time. I think many other people [1:23] were similar. And uh I just start to [1:26] notice that with the latest models uh [1:28] the chunks just came out fine. And then [1:29] I kept asking for more, and just came [1:31] out fine. And then I can't remember the [1:32] last time I corrected it. And then I was [1:34] I just uh you know, trusted the system [1:36] more and more. And then I was vibe [1:37] coding. [1:38] >> [laughter] [1:39] >> And uh so it was kind of a [1:41] I do think that it was a very stark [1:42] transition. I think that a lot of people [1:44] actually I tried to I tried to stress [1:46] this on uh Twitter and or X because I [1:49] think a lot of people experienced AI uh [1:51] last year as ChatGPT adjacent thing, uh [1:53] but you really had to look again, and [1:55] you had to look as of December uh [1:57] because things have changed [1:58] fundamentally and uh especially on this [2:00] like agentic coherent workflow that [2:03] really started to actually work. Um and [2:05] so I would say that um [2:08] yeah, it was just that realization that [2:10] really uh [2:11] had me um go down the whole rabbit hole [2:13] of just, you know, infinity side [2:15] project. Uh my side projects folder is [2:17] like extremely full with lots of random [2:19] things and uh just I've been coding all [2:21] the time. Uh so uh yeah, that kind of [2:23] happened in December, I would say. [2:25] And I was looking at the repercussions [2:26] of that since. [2:28] Um you've talked a lot about this idea [2:29] of LLMs as a new computer. Um that it [2:32] isn't just better software, it's a whole [2:35] new computing paradigm. And um software [2:37] 1.0 was explicit rules, software 2.0 was [2:40] learned weights, software 3.0 is this. [2:43] Um if that's actually true, what does a [2:46] team build differently the day they [2:48] actually believe this? [2:50] Right. So uh yeah, exactly. So software [2:53] 1.0 I'm writing code, software 2.0 I'm [2:55] actually programming by creating data [2:57] sets and training uh training neural [2:59] networks. So the programming is kind of [3:00] like arranging data sets and maybe some [3:02] objectives and neural network [3:03] architectures. And then what happened is [3:05] that basically if you train one of these [3:07] GPT models or LLMs on a sufficiently [3:09] large set of tasks implicit basically [3:12] implicitly because by training on the [3:14] internet you have to multitask all the [3:15] things that are in the data set. Uh [3:17] these actually become kind of like a [3:18] programmable computer in a certain [3:19] sense. So software 3.0 is kind of about [3:22] uh you know, your programming now turns [3:24] to prompting and what's in the context [3:26] window is your lever over the [3:28] interpreter that is the LLM that is kind [3:30] of like interpreting your context and uh [3:32] performing computation in the digital [3:34] digital information space. So I guess um [3:37] yeah, that's kind of the transition and [3:39] I think there's a few examples of that [3:41] really drove it home for me and maybe [3:42] that might be instructive. Uh so for [3:44] example, when you when Open Claw came [3:46] out [3:48] when you want to install Open Claw, you [3:49] would expect that normally this is a [3:50] bash bash script like a shell script. [3:52] So, run the shell script to run uh to [3:54] install OpenClaw. [3:55] Um but the thing is that in order to [3:57] target lots of different platforms and [3:59] lots of different types of computers you [4:00] might run an OpenClaw, uh this these [4:02] shell scripts usually ballooned up and [4:03] become extremely complex. But the thing [4:05] is you're still stuck in a software 1.0 [4:06] universe of wanting to write the code. [4:08] And actually the OpenClaw installation [4:10] is a is a copy-paste of a bunch of text [4:13] that you're supposed to give to your [4:14] agent. Uh so, basically it's it's a [4:16] little skill of uh you know, copy-paste [4:18] this and give it to your agent and it [4:19] will install OpenClaw. And the reason [4:21] this is a lot more powerful is you're [4:22] working now in the software 3.0 paradigm [4:24] where you don't have to precisely uh [4:26] spell out, you know, all the individual [4:28] details of that setup. The agent has its [4:30] own intelligence that it packages up and [4:31] then it kind of like follows the [4:32] instructions and it looks at your [4:35] environment, your computer, and it kind [4:36] of like performs intelligent actions to [4:37] make things work and debugs things in [4:39] the loop. And it's just like so much [4:40] more powerful, right? So, I think that's [4:43] a very different kind of like way of [4:44] thinking about it. It's just like, what [4:46] is the piece of text to copy-paste to [4:47] your agent? That's the programming [4:48] paradigm now. I think one more maybe uh [4:51] example that comes to mind that is even [4:52] more extreme than that is when I was [4:54] building um MenuGen. So, MenuGen is this [4:57] idea where [4:59] you um you come to a restaurant, they [5:01] give you a menu, there's no pictures [5:02] usually, so I don't know what any of [5:04] these things are. Uh usually I like 30% [5:06] of the things I don't have no idea what [5:07] they are, 50%. So, I wanted to take a [5:09] photo of the restaurant menu and to get [5:12] pictures of what those things might look [5:14] like in a generic sense. [5:16] And so, I built I built coded this app [5:17] that basically lets you upload a photo [5:19] and it does all this stuff and it runs [5:21] on Vercel and uh it basically re-renders [5:24] the menu and it gives you like all the [5:26] items and it gives you a picture that it [5:28] uses an image um you know, generator uh [5:30] for to basically OCR all the different [5:33] titles, uh use the image generator to [5:35] get pictures of them and then shows it [5:36] to you. [5:37] And then I saw the software 3.0 version [5:39] of this, which is which blew my mind, [5:41] which is literally just take your photo, [5:43] give it to Gemini, and say use Nano [5:46] Banana to overlay the the things onto [5:49] the menu." Uh Uh [5:50] and Nana Banana basically returned an [5:52] image that is exactly the picture of the [5:53] menu that I took, but it actually put [5:55] into the pixels, it rendered the [5:57] different things in the menu. [5:59] And this blew my mind because [6:02] actually all of my menu gen is spurious. [6:04] It's working in the old paradigm that [6:06] app shouldn't exist. Uh and uh yeah, the [6:09] software 3.0 paradigm is a lot more kind [6:11] of raw. It just um [6:12] your neural network is doing more and [6:14] more of the work, and your prompt or [6:16] context is just the image, and the [6:18] output is an image, and there's no need [6:20] to have any of the app in between. [6:22] Um so, I think that people have to kind [6:25] of like reframe, [6:26] you know, not to work in the existing [6:28] paradigm of what things existed and just [6:30] think about it as a speed up of what [6:32] exists. [6:33] It's actually like new things are [6:34] available now. [6:35] And going back to your programming [6:36] question, it's not even I think that's [6:38] also an example of working in the in the [6:40] old mindset because it's not just about [6:41] programming and programming becoming [6:42] faster. This is more general information [6:44] processing that is automatable now. So, [6:47] um it's not just even about code. So, [6:49] previous code worked over a kind of like [6:51] structured data, right? And uh you write [6:53] code over structured data. But like for [6:55] example with my LLM knowledge bases [6:56] project, um [6:58] uh basically you get LLMs to create [6:59] wikis for your organization or for you [7:01] in person, etc. This is not even a [7:03] program. This is not something that [7:04] could exist before because there was no [7:06] there was no code that would create a [7:07] knowledge base based on a bunch of [7:09] facts. But now you can just take these [7:10] documents and [7:12] uh [7:12] basically uh recompile them in a [7:14] different way, and uh reorder them, and [7:16] create something that is uh new and [7:17] interesting uh as a reframing of the [7:19] data. And so, these are new things that [7:22] weren't possible. Uh and so, I think [7:24] this is uh something that I keep trying [7:26] to get back to as to not only what can [7:28] we do that existed that is faster now, [7:30] but I think there's new opportunities of [7:33] just things that couldn't be possible [7:34] before. And I almost think that that's [7:35] more exciting. [7:37] I love the menu gen progression and [7:39] dichotomy that you laid out, and I think [7:41] even I'm sure many folks here followed [7:43] your own progression of programming from [7:45] last October to early January, February [7:48] this year. If you extrapolate that [7:50] further, what is the 2026 equivalent [7:54] for building websites in the '90s, [7:56] building mobile apps in the 2010s, [7:59] building SaaS in the last cloud era? [8:02] What will look completely obvious in [8:04] hindsight that is still mostly unbuilt [8:06] today? [8:07] Um [8:09] >> [clears throat] [8:09] >> Well, going with the example of MenuGen, [8:11] I guess. So, a lot of this code [8:12] shouldn't exist and it's just neural [8:13] networks doing most of the work. [8:16] Um I do think that the extrapolation [8:17] looks very weird because you could [8:19] basically imagine [8:21] I don't think I Yeah, so you could [8:23] imagine completely neural computers in a [8:25] certain sense. Uh you feed a raw videos [8:28] like imagine a device that takes raw [8:29] videos or audio into basically what's a [8:31] neural net and uses diffusion to render [8:34] a UI that is kind of like, you know, [8:36] unique for that moment in a certain [8:38] sense. And um [8:40] I kind of feel like in the early days of [8:41] computing actually, people were a little [8:43] bit confused as to whether computers [8:44] would look like calculators or computers [8:46] would look like neural nets. And in '50s [8:48] and '60s, it was not really obvious [8:49] which way would go. And of course, we [8:51] went down the calculator path and ended [8:53] up building classical computing and then [8:55] neural nets are currently running [8:56] virtualized on existing computers. But [8:58] you could imagine I think that a lot of [9:00] this will flip and that the neural net [9:01] becomes kind of like the host process. [9:03] And the CPUs become kind of like the [9:05] co-processor. So, we saw the diagram of, [9:08] you know, intelligence compute is going [9:09] to neural networks is going to take over [9:11] and become the dominant spend of flops. [9:13] So, you could imagine something really [9:15] weird and foreign when where neural nets [9:18] are doing most of the heavy lifting, [9:19] they're using tool use as just like, you [9:21] know, [9:22] historical appendage for some kinds of [9:23] like deterministic tasks. But what's [9:25] really running the show is these [9:27] neural nets that are networked in a [9:28] certain way. Um so, you can imagine [9:30] something extremely foreign as the [9:32] extrapolation, but I think we're going [9:33] to probably get there sort of piece by [9:35] piece. [9:36] And I don't Yeah, I don't that that [9:38] progression is TBD, I would say. [9:40] >> [snorts] [9:41] >> I'd love to talk a little bit about um, [9:43] uh, this concept of verifiability. The [9:45] fact that AI will automate faster and [9:47] more easily domains where the output can [9:49] be verified. [9:51] Um, if that framework is right, what [9:53] work is about to move much faster than [9:54] people realize? And what professions do [9:57] we have that people actually think are [9:59] safe, but they're actually highly [10:01] verifiable? [10:02] Uh, yes, so I I spent uh, some time [10:05] writing about verifiability and um, [10:07] basically like traditional computers can [10:09] easily automate what you can specify [10:12] in code. And uh, [10:13] kind of this latest round of LLMs can [10:15] easily automate what you can uh, verify [10:17] in a certain in a certain sense. Uh, [10:19] because the way this works is that when [10:20] frontier labs are training these LLMs, [10:22] these are giant reinforcement learning [10:23] environments. So, they are given a [10:25] verification rewards. And then because [10:27] of the way that these models are [10:29] trained, they end up basically uh, [10:30] progressing and creating these like [10:33] jagged entities that really peak in [10:35] capability in kind of like verifiable [10:36] domains like math and code and adjacent. [10:38] And kind of like stagnate and are a [10:40] little bit um, [10:41] you know, rougher on the edges when uh, [10:43] things are not kind of like in that in [10:44] that space. So, I think the reason I [10:46] wrote about verifiability is I'm trying [10:47] to understand why these things are so [10:49] jagged. [10:50] Um, and some of it has to do with how [10:52] the labs train the models, but I think [10:53] some of it also has to do with um, the [10:55] focus of the labs and what they happen [10:57] to put into the data distribution. Uh, [11:00] because some things basically are [11:01] significantly more valuable in economy [11:02] and end up creating more environments [11:04] because the labs wanted to work in those [11:05] settings. So, I think code is a good [11:07] example of that. There's probably lots [11:08] of verifiable environments they could [11:10] think about that happen not to make it [11:11] into the mix because they're just not [11:12] that useful to have the capability [11:13] around. [11:14] Um, [11:15] but I think to me the big um, I guess [11:18] like the big mystery is uh, [11:20] the favorite example for a while was [11:21] that how many letters are are in a [11:23] strawberry? And the models would [11:24] famously get this wrong and it's an [11:25] example of jaggedness. Uh, the models [11:27] now patch this, I think, but the new one [11:29] is I want to go to a car wash to wash my [11:31] car, and it's 50 m away, should I drive [11:34] or should I walk? And state-of-the-art [11:37] models today will tell you to walk [11:39] because it's so close. How is it [11:41] possible that state-of-the-art Opus 4.7 [11:44] will simultaneously refactor a 100,000 [11:46] like [11:47] >> [laughter] [11:47] >> code base a line code base or find [11:49] zero-day vulnerabilities and yet tells [11:51] me to walk to this car wash? This is [11:54] insane. And to whatever extent these [11:57] models are remain jagged, it's an [12:00] indication that number one, maybe [12:01] something slightly off. [12:03] Or [12:04] number two, you need to actually [12:06] be in the loop a little bit and you need [12:07] to treat them as tools and you do have [12:09] to kind of stay in touch with what [12:11] they're doing. And so I think all of my [12:13] writing, long story short, about [12:14] verifiability is just trying to [12:15] understand um [12:17] why these things are jagged, is there [12:19] any pattern to it? And I think it's a [12:20] some kind of combination of verifiable [12:23] plus labs care. Maybe one more anecdote [12:25] that is instructive [12:27] is [12:28] from GPT-3.5 to GPT-4, people noticed [12:30] that chess improved a lot and I think a [12:32] lot of people thought, oh well, it's [12:34] just a progression of the capabilities. [12:35] But actually it's it's more that I think [12:37] this is public information, I think I [12:39] saw it on the internet. Um a huge amount [12:40] of like [12:42] data of chess made it into the [12:43] pre-training set. [12:45] And just because it's in the data [12:46] distribution, basically the model [12:48] improved a lot more than it would just [12:50] by default. [12:51] So someone at OpenAI decided to add this [12:53] data and now you have a capability that [12:55] just peaked a lot more. And so that's [12:57] why I think I'm stressing this [12:59] dimension of it as we are slightly at [13:02] the mercy of whatever the labs are [13:03] doing, whatever they happen to put into [13:04] the mix and you have to actually explore [13:06] this thing that they give you that has [13:08] no manual and it works in certain [13:10] settings but maybe not in some settings [13:12] and you have to kind of explore it a [13:14] little bit and if you're in the circuits [13:17] that were part of the RL, you fly and if [13:19] you're in the circuits that are out of [13:21] the data distribution, you're going to [13:22] struggle and you have to kind of figure [13:24] out which which circuits you're in in [13:26] your application. And if you and if [13:28] you're not in the circuits, then you [13:29] have to really look at fine-tuning and [13:30] doing some of your own work because it's [13:32] not going to necessarily come out of the [13:34] LLM out of the box. [13:36] I'd love to come back to the concept of [13:37] jagged intelligence in a little bit. Um [13:40] if you were a founder today and thinking [13:42] about building a company, you are trying [13:44] to solve a problem that you think is [13:46] tractable, something that uh is a domain [13:49] that is verifiable, but you look around [13:51] and you think, "Oh my gosh, well the [13:52] labs have really really started uh [13:56] got getting to escape velocity and the [13:58] ones that seem most obvious, math, [14:00] coding, and others." What would your [14:01] advice be to to the founders in the [14:03] audience? [14:05] Um [14:08] So, I think maybe that comes to the [14:10] previous question of I do think that [14:11] verifiability because it um [14:14] Let me think. So, verifiability makes [14:16] something tractable in the current [14:17] paradigm because you can throw huge [14:19] amount of RL at it. Um [14:22] So, maybe one way to see it is that uh [14:25] that remains true even if the labs are [14:26] not focusing on it directly. So, if you [14:28] are in a a verifiable setting where you [14:30] could create these RL environments or [14:31] examples, then that actually sets you up [14:34] to potentially do your own fine-tuning [14:35] and you might benefit from that. But, [14:36] that is fundamentally technology that [14:38] just works. You can pull a lever. If you [14:39] have huge amount of diverse data sets of [14:41] RL environments, etc., uh you can use [14:43] your favorite fine-tuning framework and [14:45] um and uh [14:47] pull the lever and get something that [14:48] actually uh works pretty well. So, um [14:51] I don't know what the examples of this [14:52] might be. Um [14:54] but I do think there are some very [14:55] valuable uh reinforcement learning [14:57] environments that people could think of [14:58] that I think are not part of the [15:00] Yeah, I don't want to give away the [15:01] answer, but there is one domain that I [15:03] think is very uh Oh, okay. Sorry. I [15:05] don't mean to vague post on on the [15:06] stage, but [15:07] uh there are some examples of this. On [15:09] the flip side, what do you think still [15:11] feels automatable only from a distance? [15:14] I do think that ultimately almost [15:17] everything can be made uh verifiable to [15:19] some extent, some things easier than [15:20] others. Um [15:22] because even for like things that are [15:23] like writing or so on, you can imagine [15:25] having a council of LLM judges and [15:27] probably get get to some get something [15:30] reasonable out of the [15:31] from from this kind of an approach. So, [15:33] it's more about what's easy or hard. [15:35] Um [15:36] So, I I do think that ultimately um [15:39] Uh yeah, I think uh [15:41] Everything. [15:42] >> [laughter] [15:43] >> Everything is automatable. [15:45] Amazing. Okay. Um so, last year you [15:47] coined the term vibe coding and today [15:49] we're in a world that feels a little bit [15:50] more serious, more agentic engineering. [15:52] What do you think is the difference [15:54] between the two and what would you [15:55] actually call what we're in today? [15:57] Uh yeah, so I would say vibe coding is [15:58] about raising the floor for everyone in [16:00] terms of what they can do in software. [16:02] So, the floor rises, everyone can vibe [16:04] code anything, and that's amazing, [16:06] incredible. But then I would say agentic [16:08] engineering is about preserving the [16:09] quality bar of what existed before in [16:11] professional software. So, you're not [16:13] allowed to introduce uh vulnerabilities [16:15] due to vibe coding. Um you are um you're [16:18] still responsible for your software just [16:20] as before, but can you go faster? And [16:22] spoiler is you can, but how do you how [16:24] do you do that properly? And so, to me [16:26] agentic engineering when I I call it [16:27] that because I do think it's kind of [16:29] like an engineering discipline. You have [16:30] these agents which are these like spiky [16:32] entities, they're a bit fallible, a [16:33] little bit stochastic, but they are [16:35] extremely powerful. And it's how do you [16:37] how do you coordinate them to go faster [16:39] without sacrificing your quality bar? [16:41] And doing that well and correctly um is [16:45] the the realm of agentic engineering. [16:47] Um so, I kind of see them as as [16:49] different. Like one is about maybe [16:50] raising the raising the floor, and the [16:52] other is about um you know, [16:53] extrapolating. And what I'm seeing I [16:55] think is [16:56] there is a very high ceiling on agent [16:58] engineer uh capability. And you know, [17:01] people used to talk about the 10x [17:02] engineer previously. I think that this [17:04] is uh magnified a lot more. Uh 10x is uh [17:07] is not uh the speed up you gain. Um [17:10] and I think uh it does seem to me like [17:12] people who are very good at this um [17:15] peak a lot more than 10x uh from from my [17:17] perspective right now. I really like [17:18] that framing. Um [17:20] one thing that when Sam Altman came to [17:22] AI sent last year, one memorable thing [17:24] he said was that people of different [17:26] generations use ChatGPT differently. So, [17:28] if you're in your 30s, you use it as a [17:30] Google search replacement, but if you're [17:32] in your teens, ChatGPT is your gateway [17:34] to the internet. What is the parallel [17:36] here in coding today? If we were to [17:38] watch two people code using open claw, [17:41] cloud code, codex, one you'd consider [17:44] mediocre at it and one you would [17:46] consider fully AI native, how would you [17:48] describe the difference? [17:51] I [clears throat] mean, I think it's [17:52] just trying to get the most out of the [17:54] tools that are available, utilizing all [17:56] of their features, investing into your [17:58] own [17:59] kind of setup. So, just like previously, [18:01] all the engineers are used to basically [18:03] getting the most out of the tools you [18:04] use, either it's Vim or VS Code or now [18:06] it's you know, cloud code or codex or so [18:09] on. So, um [18:11] just investing into your setup [18:13] and [18:14] utilizing a lot of the, you know, tools [18:17] that are available to you. [18:18] Um and I think it just kind of looks [18:20] like that. I do think that [18:22] maybe related thought is [18:24] um [18:26] a lot of people are maybe hiring [18:28] for this, right? Because they want to [18:30] hire strong agentic engineers. I do [18:32] think that [18:34] what I'm seeing is that [18:36] the, you know, most people are still not [18:37] refactored their [18:39] their hiring process for agentic [18:41] engineer capability, right? Like if [18:44] you're giving out puzzles to solve, then [18:46] this is still the old paradigm. I would [18:47] say that hiring have to has to look like [18:49] give me a really big project and see [18:51] someone implement that big project. Like [18:53] let's write, say a Twitter clone [18:55] for agents and then make it really good, [18:58] make it really secure, and then have [19:00] some agents simulate some activity on [19:03] this Twitter. And then I'm going to use [19:05] 10 [19:06] codex 5.4 x high to try to break your [19:09] break your [19:12] this website that you deployed and [19:14] they're going to try to basically break [19:15] it and they should not be able to break [19:16] it. And so maybe it looks like that, [19:18] right? And so yeah, watching people in [19:20] that that setting and building some [19:22] bigger projects and [19:24] utilize utilizing the tooling is maybe [19:26] what I would look at for the most part. [19:29] And as agents do more, what human skill [19:31] do you think becomes more valuable, not [19:33] less? [19:34] Also, yeah, it's a good question. I [19:36] think um [19:38] Well, right now the answer is that the [19:39] agents are catalog these internal [19:41] entities, right? So [19:42] it's remarkable [19:44] um [19:45] you basically still have to be in charge [19:47] of the aesthetics, the the judgment, the [19:49] taste, and a little bit of oversight. [19:51] And maybe one one of my favorite [19:52] examples of like the the weirdness of [19:54] agents is um [19:56] for menu gen, you sign up with a Google [19:58] Google account, but you [20:00] purchase credits using a Stripe account [20:03] and both of them have email addresses. [20:04] And my agent actually tries to basically [20:08] um [20:09] like when you purchase credits, it [20:10] assigned it using the email address from [20:12] Stripe to the Google email address. Like [20:15] there wasn't a persistent user ID that [20:18] that for people. It was trying to match [20:20] up the email addresses, but you could [20:21] use different email address for your [20:22] Stripe and your Google and basically [20:24] would not associate the funds. And so [20:26] this is the kind of thing that these [20:28] agents still will make mistakes about. [20:30] It's like why would you use email [20:31] addresses to try to cross-correlate the [20:32] funds? They can be arbitrary. You can [20:34] use different emails, etc. Like this is [20:36] such a weird thing to do. So I think [20:38] people have to be in charge of this [20:40] spec, this plan, and [20:43] actually don't even like the plan mode. [20:44] I would [20:45] I mean, obviously it's very useful, but [20:47] I think there's something more general [20:48] here where you have to work with your [20:49] agent to design a spec that is very [20:51] detailed and maybe it's a maybe [20:53] basically the docs and then get the [20:55] agents to write them. And you're in [20:56] charge of the oversight and the [20:58] top-level categories, but the agents are [21:00] doing a lot of the under the hood. And [21:02] so I think you're not caring about some [21:04] of the details. So as an example also [21:05] with um, a race or tensors in neural [21:09] networks, um, there's a ton of details [21:11] between PyTorch and NumPy and all the [21:13] different like pandas and so on for all [21:14] the different little API details. And [21:16] I'll I already forgot about the keep [21:18] dims versus keep dim or whether it's dim [21:20] or axis or reshape or permute or [21:22] transpose. I don't remember this stuff [21:23] anymore, right? Because you don't have [21:24] to. This is the kind of details that are [21:26] handled by the intern because they have [21:27] very good recall. And but you still have [21:29] to know for example that um, you know, [21:31] there's underlying tensor, there's an [21:33] underlying view and then you can view of [21:36] the same storage or you can have [21:37] different storage which will be less [21:38] efficient. As we still have to have an [21:40] understanding of what this stuff is [21:42] doing and some of the fundamentals [21:44] um, [21:45] so that you're not copying memory around [21:46] unnecessarily and so on. But uh, the [21:48] details of the APIs are now handed off. [21:51] So it um, you're in charge of the taste, [21:53] the engineering, the design uh, and that [21:55] it makes sense and that you're asking [21:57] for the right things and that you're [21:58] saying that okay, that these have to be [22:00] unique user IDs that we're going to tie [22:01] everything to. Um, [22:03] and so you're doing some of the [22:05] design and development and the engineers [22:07] are doing the fill in the blanks. And [22:08] that's currently kind of like where we [22:09] are and I think that's what everyone of [22:11] course is seeing I think right now. Do [22:13] you think there's a chance that this um, [22:15] taste and judgment matters less over [22:18] time or will the ceiling just keep [22:19] rising? [22:21] Um, yeah, it's a good question. I would [22:22] say [22:25] um, [22:26] I mean I'm hoping that the that it [22:28] improves. I think probably the reason it [22:30] doesn't improve right now is again it's [22:31] not part of the RL. There's probably no [22:33] aesthetics cost or reward or it's not [22:36] good enough or something like that. Um, [22:39] I do think that when you actually look [22:41] at the code, sometimes I get a little [22:42] bit of a heart attack because it's not [22:44] like super amazing code necessarily all [22:45] the time and it's very bloated and [22:46] there's a lot of copy-paste and there's [22:48] awkward abstractions that are brittle [22:50] and like it works but it's just really [22:52] gross. Um, [22:54] and I do I do hope that this can improve [22:56] in future models. Um, a good example [22:58] also is this uh, you know, the micro GPT [23:00] project uh, which where I was trying to [23:01] simplify uh LLM training to be as simple [23:04] as possible. The models hate this. They [23:06] can't do it. I tried to I keep I kept [23:08] trying to prompt an LLM to simplify [23:10] more, simplify more, and it just can't [23:13] You feel like you're outside of the RL [23:14] circuits. It feels like it you're [23:16] obviously, you know, you're pulling [23:18] teeth. It's not like light speed. [23:21] So, I think um [23:23] I do think that people are still remain [23:24] in charge of this, but I do think that [23:26] there's nothing fundamental again that's [23:27] preventing it. It's just the labs [23:28] haven't done it yet almost. [23:30] Yeah. [23:31] So, I'd love to come back to this idea [23:32] of uh [23:34] jagged forms of intelligence. You wrote [23:36] a little bit about this with uh very [23:38] thought-provoking piece around animals [23:39] versus ghosts. [23:41] Um and the idea is that we're not [23:43] building animals. We are summoning [23:44] ghosts. Um and these are jagged forms of [23:47] intelligence that are shaped by data and [23:50] reward functions, but not by intrinsic [23:52] motivation or fun or curiosity or [23:54] empowerment, uh [23:56] things that kind of came about via [23:58] evolution. [23:59] Um why does that framing matter? And [24:02] what does it actually change about how [24:04] you build and deploy and evaluate or [24:06] even trust them? [24:08] Uh [24:09] yes, so [24:11] Yeah, I think the reason I wrote about [24:12] this is because I'm trying to wrap my [24:14] head around what these things are, [24:15] right? Because if you have a good model [24:16] of what they are or are not, then you're [24:18] going to be more competent at uh using [24:20] them. Um and I do think that um [24:23] I don't know if it has I'm not sure if [24:25] it actually has like real power. [24:27] >> [laughter] [24:28] >> I think it's a little bit of [24:29] philosophizing. [24:31] But I do think that um [24:34] I think it's just um [24:36] coming to terms with the fact that these [24:37] things are not, you know, animal [24:39] intelligences. Like if you yell at them, [24:40] they're not going to work better or or [24:42] worse or it doesn't have any impact. Um [24:44] and uh [24:46] it's all just kind of like these [24:47] statistical simulation circuits where [24:49] the the substrate is pre-training, so [24:52] like statistics. And then but then [24:54] there's RL bolting on top, so [24:57] it kind of like increases the [24:58] disadvantages and um maybe it's just [25:00] kind of like a mindset of what I'm [25:02] coming into or what's likely to work or [25:04] not likely to work or how to modify it, [25:05] but I don't actually I don't know that I [25:07] have like here are the five obvious [25:09] outcomes of how to make your system [25:11] better. It's more just being suspicious [25:12] of it and um [25:14] figuring out over time. [25:16] That's where it starts. Okay, so you are [25:18] so deep in working with agents that [25:19] don't just chat. They have [25:22] real permissions. They have local [25:23] contacts. They actually take action on [25:25] your your behalf. What does the world [25:27] look like when we all start to live in [25:29] that world? [25:31] Yeah, I think [25:33] I think a lot of people probably here [25:34] are excited about what this agentic [25:37] you know, native agentic environment [25:39] looks like and everything has to be [25:40] rewritten. Everything is still [25:41] fundamentally written for humans and has [25:43] to be moved around. I still use most of [25:46] the time when I use different frameworks [25:48] or libraries or things like that. They [25:49] still have docs that are fundamentally [25:51] written for humans. This is my favorite [25:53] pet peeve. Like I don't Why are people [25:55] still telling me what to do? Like I [25:57] don't want to do anything. What is the [25:58] thing I should copy paste to my agent? [26:00] >> [laughter] [26:01] >> So it's just every time I'm told, you [26:03] know, go to this URL or something like [26:05] that. It's just like ah. [26:06] >> [laughter] [26:07] >> You know. [26:08] >> [snorts] [26:08] >> So um [26:09] everyone is I think excited about how do [26:11] we decompose the workloads that need to [26:13] happen into fundamentally sensors over [26:15] the world, actuators over the world. How [26:17] do we make it agent native? Basically [26:19] describe it to agents first. Um and then [26:22] I have a lot of automation around [26:26] you know, the [26:28] Yeah, around data structures that are [26:29] very legible to the LLMs. [26:31] So I think [26:33] yeah, I'm hoping that there's a lot of [26:34] agent first infrastructure out there and [26:37] that you know, for MenuGen famously when [26:39] I wrote the not I'm not sure how [26:40] famously, but when I wrote the blog post [26:42] about MenuGen [laughter] [26:43] a lot of the work a lot of the trouble [26:46] was not even writing the code for [26:47] MenuGen. It was deploying it on Vercel [26:49] because I had to work with all these [26:50] different services and I just string [26:51] them up and I just go to their settings [26:52] and the menus, and [26:54] you know, configure my DNS, and it was [26:56] just so annoying. And so, that's a good [26:59] example of I would hope that MenuGen [27:01] that I could give a prompt to an LLM, [27:03] build MenuGen, and then I didn't have to [27:05] touch anything, and it's deployed in [27:06] that same way on the internet. I think [27:09] that would be a good kind of a test for [27:10] whether or not a lot of our [27:12] infrastructure is becoming more and more [27:13] agent native. And then ultimately, I [27:15] would say [27:16] yeah, I I do think we're going towards a [27:17] world where there's agent representation [27:20] for people and for organizations, and um [27:25] you know, I'll have my agent talk to [27:26] your agent to figure out some of the [27:28] details of our meetings or things like [27:30] that. So, [27:31] >> [laughter] [27:32] >> um I do think that that's roughly where [27:34] things are going, but um yeah, I think [27:36] everyone here is excited about that. I [27:38] really like the visual analogy of [27:39] sensors and actuators. I actually hadn't [27:41] thought of that. That's super [27:42] interesting. [27:42] >> Right. Um okay, I think we have to end [27:44] on a question about education, um [27:46] because you are probably one of the very [27:48] best in the world at making complex [27:50] technical concepts simple and deeply [27:52] thoughtful about how we design education [27:55] around it. Um [27:57] what still remains worth learning deeply [28:00] when intelligence gets cheap as we move [28:02] into the next era of AI? [28:05] Yeah. [28:06] Uh there was a tweet that blew my mind [28:08] recently, and I keep thinking about it [28:09] like every other day. It was something [28:10] along the lines of um [28:12] you can outsource your thinking, but you [28:13] can't outsource your understanding. [28:16] And um I think that's really nicely put. [28:19] I so [28:21] yeah, because I still I'm still part of [28:23] the system, and I still I still have to [28:25] somehow information still has to make it [28:26] into my brain, and I feel like I'm [28:27] becoming a bottleneck of just even [28:29] knowing what we're trying to build, why [28:30] is it worth doing, uh how do I direct [28:32] you know, how do I direct my my agents, [28:34] and so on. So, [28:35] I do still think that ultimately [28:37] something has to direct the thinking and [28:39] the processing, and so on. And um [28:43] that's still kind of fundamentally [28:44] constrained somehow by understanding. [28:45] And this is one reason I also was very [28:46] excited about all the all the knowledge [28:48] bases because I feel like that's that's [28:50] a way for me to process information. And [28:52] anytime I see a different projection [28:53] onto information, I always like feel [28:55] like I gain insight. So, it's really [28:57] just [28:57] a lot of prompts for me to do synthetic [28:59] data generation kind of over over some [29:01] fixed data. Uh so, I I really enjoy uh [29:04] whenever I read an article, I have my uh [29:05] you know, my wiki that's being built up [29:07] from these articles. And I love asking [29:09] questions about things or um and I I [29:11] think that ultimately these are tools to [29:14] enhance understanding in a certain way. [29:16] And this is still kind of like a bit of [29:18] a bottleneck because then you can't [29:19] direct the [29:21] uh you you can't be a good director if [29:22] you still [29:23] uh cuz the LLMs certainly don't excel at [29:25] understanding. You still are uniquely in [29:27] charge of that. So, [29:28] uh yeah, I think uh tools to that effect [29:30] I think are incredibly interesting and [29:32] exciting. I'm excited to be back here in [29:34] a couple years and to see if we've been [29:35] fully automated out of the loop and they [29:38] actually take care of understanding as [29:39] well. Uh thank you so much for joining [29:41] us, Andre. We really appreciate it. [29:42] [applause]