Anthropic Engineers Stop Writing Code as Agent Harnesses Take Over

智能体运行框架接管开发，Anthropic工程师不再亲自编写代码 随着前沿AI模型实现语法和代码生成的自动化，软件工程正从编写实现逻辑转向构建自动化验证框架及管理并发智能体循环。 Thariq Shihipar概述了Anthropic的工程研发如何从人工编写代码转向构建执行框架、自动化验证流水线和自主智能体循环。他解释说，工程师正越来越多地将精力投入到前置明确规范、维护测试脚手架以及管理上下文工程上，而非微调提示词模板。对话探讨了在机械化代码生成变得廉价且普及的背景下，审美素养、底层系统知识与架构判断力为何仍是关键的壁垒能力。此外，Shihipar还回顾了早期职业发展速度，分享了公开输出技术内容如何拓宽职业机会，以及年轻工程师为何应当兼具远大抱负与技术谦逊。 Anthropic内部工程研发已从在IDE中直接编写代码，转变为构建验证框架、工具定义以及多智能体自主循环。 (2:53) Anthropic精简了Claude Code 80%的系统提示词并移除了少样本示例，因为更新的前沿模型展现出更强的原生指令遵循能力，反倒可能受到僵化示范的限制。 (27:38) 为了跟上AI极高的代码生成速度并避免代码审查瓶颈，工程团队必须将测试脚手架扩展约100倍，并通过同时运行多达50个Claude Code会话的配置来管理并发任务。 (53:26) 尽管机械性的代码实现已基本商品化，但在操作系统、运行时和内存等底层技术素养方面的深厚积累，对于突破系统瓶颈和引导自主系统仍至关重要。 (60:57)

Anthropic engineer Thariq Shihipar argues that frontier AI models are shifting software engineering from manual syntax production to building verification harnesses, context scaffolding, and autonomous execution loops. While coding variance has plummeted, Shihipar contends that deep systems literacy, architectural judgment, and domain taste remain essential to steering these automated workflows. Harness Design and Autonomous Loop Engineering (6:16) Rather than rendering outer tooling obsolete, smarter models make harness engineering increasingly complex and load-bearing. Shihipar explains that when agents operate autonomously for hours, harnesses must handle sandboxing, safety evaluation, and review synthesis. For example, Claude Code incorporates an 'auto mode' classifier to assess safety after every action without halting for manual approvals, and transforms long execution traces into structured artifacts for human inspection. Anthropic engineers increasingly transition from interactive prompting to 'loop engineering'—designing recurring or event-driven pipelines that monitor metrics, triage feedback, and land validated fixes across the software development lifecycle. Shihipar notes that while constructing these systems requires significant upfront effort analogous to onboarding a human engineer, automated verification loops enable continuous issue resolution and free developers to focus on system-level architecture rather than glue code. Context Engineering and Explicit Compute Allocation (27:38) Shihipar observes that prompt engineering has evolved into holistic context engineering: structuring tools, schemas, and verification environments rather than drafting clever text prompts. In modern models, Anthropic found that few-shot examples often restrict creative problem-solving; removing hardcoded demonstrations from tool definitions and pruning 80% of Claude Code's system prompt actually improved autonomous execution. Furthermore, default model behaviors prioritize low latency and modest token usage, prompting agents to jump directly into code. Shihipar states that unlocking deep reasoning requires giving models explicit permission to consume compute. Engineers achieve this by signaling task difficulty, authorizing sub-agents, instructing the model to brainstorm before writing code, or setting overnight autonomous goals. Scaffolding Verification and Shifting Maintainability Horizons (48:22) As the volume of machine-generated code grows, Shihipar argues that traditional debates over naming conventions and stylistic preferences matter far less than massive automated testing scaffolding. Anthropic teams favor a two-step pattern: allowing models to find a functional solution first, and only then invoking a specialized 'simplify' skill to streamline the architecture. Guardrails against drift include capturing production traffic to generate mock databases, component test suites, and automated screen recordings verifying features in live environments before pull request review. This rapid model improvement also alters how teams handle technical debt. Shihipar suggests evaluating maintenance over tighter one- to two-month horizons. If a refactoring project offers value only on a 6-to-12-month timeline, teams may be better off shipping customer features immediately and deferring overhauls until next-generation models can handle repo-wide rewrites. The Durability of Technical Literacy and Taste (60:57) Addressing Boris Cherny's claim that 'coding is largely solved,' Shihipar clarifies that programming has shifted from an unpredictable, failure-prone bottleneck into an abundant utility. However, he maintains that foundational technical literacy—understanding operating systems, runtimes, memory, and caching—remains critical to evaluate output quality and orchestrate advanced workflows, such as programmatically driving command-line tools like FFmpeg. Subjective judgment and taste also remain uniquely defensible. In frontend engineering, obtaining polished aesthetic output requires supplying machine-readable specifications, such as raw HTML and inspectable Figma design tokens, rather than imprecise screenshots. Culturally, Anthropic discourages generating strategic product essays with AI, treating writing as an irreplaceable medium for human critical reasoning. Shihipar envisions a software landscape characterized by extreme abundance, mirroring the historical expansion of recorded music. However, navigating this abundance requires early-career engineers to balance bold self-belief with the humility to seek mentorship, ensuring they master system-level fundamentals rather than passively consuming model outputs.

[9:52] Thariq Shihipar: It's really blurring the lines of like, what does it mean? [10:42] Ryan Peterman: Don't tell me till you're done. [16:24] Ryan Peterman: painfully slow [29:15] Thariq Shihipar: Here's an example of writing a file well and here's an example of not doing it well. [30:56] just believe in yourself [32:57] Claude: Hey, maybe you should go to sleep. [35:04] I'm not a designer. I don't even have the vocabulary, but I want to build a better aesthetic. Thariq Shihipar: My rule of thumb is like if I would be happy to show someone the prompt, I would send them the output. writing is also a way of thinking [60:57] Boris Cherny: coding is largely solved [60:57] Andrej Karpathy: learning should feel like effort [67:42] Thariq Shihipar: Thinking of yourself not as someone who's being trained to do something, but someone who can do something incredible right away, I think is really important.

[0:00] Most software we wrote before LLMs was [0:01] not very good. [0:03] >> This is Tariq. He's an engineer at [0:05] Anthropic on the Claude code team, and I [0:07] asked him all about how their [0:09] engineering team makes the most out of [0:10] the models today. [0:12] >> Can you move up like a higher [0:14] abstraction level, you know, like can [0:15] you build the system that builds a [0:16] system? [0:17] >> Is there something that you feel's kind [0:19] of the next big shift that is going to [0:22] diffuse into the industry? [0:24] >> Yeah, I think there are a few different [0:27] ones. [0:28] >> Here's the full episode. [0:34] My goal with this conversation is to ask [0:37] you as much as I can about how can you [0:39] get the most out of the models [0:41] specifically for software engineering so [0:44] that people in the industry can kind of [0:45] learn from the best practices where [0:48] Anthropic's having success. And so to [0:51] start the conversation off, I'd like to [0:53] ask you let's say I was coming from a [0:57] company that's maybe less AI pilled and [0:59] I was onboarding onto your team, what [1:02] are the most impactful things that you'd [1:04] tell me to start doing to successfully [1:07] onboard? [1:08] >> I think the number one tip we have for [1:10] you know, both people inside and and [1:12] outside Anthropic is that like if you [1:13] treat Claude like a thought partner and [1:15] give it like the context that you need, [1:18] then you can usually figure out the next [1:20] steps, you know, and and so I think that [1:22] like starting with okay, can Claude do [1:25] it? If not, why not, you know, and [1:28] uh [1:29] using that as the starting place, I I [1:31] think it's really important and I think [1:33] just like always thinking about like [1:35] okay, what's [1:37] you know, reflecting on can you move up [1:40] like a higher abstraction level, you [1:41] know, like can you uh you know, set up a [1:44] system? Can you build the system that [1:46] builds a system, right? Instead of just [1:48] like building, you know, the the product [1:51] itself. [1:52] >> I remember when we used to onboard [1:54] people you would get assigned an [1:56] onboarding buddy, so someone who kind of [1:59] really knows the code base and you can [2:01] ask all of your trivial setup questions [2:03] to. It sounds like Claude fills in a lot [2:05] of those gaps. And in practice, is it [2:09] 100% you don't need an onboarding buddy [2:12] and you can just ask the model for [2:13] everything? [2:14] >> You do need an onboarding buddy, but [2:16] more from like a kind of like social and [2:19] cultural perspective than a technical [2:21] perspective, you know what I mean? I [2:22] think like from a technical perspective, [2:24] you can basically just like work with [2:26] Claude and you know, if you're like good [2:29] at it, like you can get what you need to [2:30] onboard, but I think just like having [2:32] the context of you know, how to work [2:34] with a team and also just like how to [2:36] like, you know, get buying on what [2:38] you're building or you know, understand [2:41] like how the team works together or just [2:42] even like have a friend to work with, [2:44] you know, I think is really important. [2:46] So, yeah, we still have onboarding [2:47] buddies, but not [2:49] um [2:49] it's not like nearly as much technical [2:52] lift as as it used to be. [2:53] >> There's a big difference between the [2:56] external perception of the AI [2:58] capabilities and the internal Anthropic [3:01] usage. Because I'll talk to friends, [3:03] they say, yeah, what Boris is saying is [3:05] the reality. And so, my question is, why [3:09] is there such a big difference between [3:10] that that internal perception and that [3:13] external perception? [3:15] >> I think we have a good track record kind [3:18] of of when we make these like claims, [3:20] you know, you're like, oh, you're you're [3:22] like it turns out to be true on the long [3:25] large enough time scale, you know, I [3:26] think a lot of engineers are not really [3:28] in the IDE anymore, you know, like yeah, [3:31] the code being written is like I I [3:34] talked to large enterprise customers [3:35] where like, you know, they haven't like [3:37] typed a line of code themselves in 6 [3:38] months, right? So, we think of it part [3:41] of our jobs, you know, to figure out how [3:43] to work at a higher abstraction level. [3:45] And like if I spend like a day trying to [3:48] figure out how to get Claude to the [3:50] like, you know, do this autonomously and [3:52] I fail, that's like fine, you know, and [3:54] maybe even good, you know, cuz now I can [3:56] be like, oh hey, like Claude is not good [3:57] at this. How do we make it better, you [3:59] know? But I think if you're working like [4:01] an average job, you know, your job is to [4:03] produce the output and like I think [4:05] automation always has a cost cuz you're [4:07] like it's an investment and if it works, [4:10] then great, you'll make, you know, a [4:11] return on your investment over time. [4:14] Um but if it doesn't, then you've like [4:15] wasted all this time. And so I think the [4:18] nice thing about the models is that [4:20] you know, generally the chance of the [4:22] investment working out is higher and [4:24] higher because like the models are [4:26] getting smarter and smarter. [4:27] Uh but you still have to sort of take [4:29] that leap as an individual. Your boss is [4:31] not going to be super happy with you if [4:32] you're like, you know, uh [4:35] if you've been working on your harness [4:36] setup the entire time and like not [4:38] shipping code. [4:39] Um [4:40] but yeah, I think it's mostly just like [4:42] culture and thinking of it as our job to [4:45] sort of live in the future and then we [4:46] try and like build it into the product [4:47] and our harness as well so that you [4:49] don't have to do as much manual stuff, [4:52] yeah. [4:53] >> Are there examples that you come to mind [4:56] when you think of things you don't see [4:58] people doing on Twitter that are really [5:01] having making a big difference at [5:03] Anthropic that you'd kind of recommend? [5:05] >> Like using Claude code for knowledge [5:07] work, I think it's really valuable. The [5:09] way a technical person does knowledge [5:11] work, I think is very different than the [5:13] way a non-technical person does [5:14] knowledge work um these days because you [5:17] can get the models to do it. And so I [5:19] think like you know, increasingly if you [5:21] can figure out like, hey, this is a [5:23] task, the task is made up of code-like [5:26] things, you know, and like how do I tell [5:29] the model the code [5:31] steps to take, you know, [5:34] uh in order to do it, right? Uh [5:36] you can like [5:38] uh [5:39] you know, I think you can do a lot more, [5:40] right? So I I think I do like a lot of [5:42] accounting with Claude code using But [5:45] the personal side using like uh scripts [5:48] and Python instead of Excel, you know? [5:50] Or like I do video editing using FFmpeg [5:53] and these libraries to render visuals [5:56] and things like that. And so I think [5:58] most of the knowledge work is like uh [6:01] reducible to code and coding agents if [6:05] you like think about it well, you know, [6:07] and um [6:08] I think that that's like one I think [6:11] like big difference [6:13] uh that I see like [6:15] other people doing less of, I think. [6:18] >> We talked a little bit about the the [6:20] model and the harness. And [6:23] uh I want to ask you, what's the [6:25] relationship between the model and the [6:27] harness in terms of getting the best end [6:28] results? And obviously the model's [6:30] important. So how important is the [6:32] harness? And are there any examples you [6:34] could kind of share? [6:35] >> Yeah, I mean, uh the harness is super [6:38] important. I I think you I I think that [6:41] sometimes there's this idea that the [6:42] harness doesn't matter because the [6:43] models will get better and better. And [6:45] if the model just does everything [6:47] perfectly, then why do you need a [6:48] harness at all, right? And I think that [6:51] in practice, what we see is like the [6:53] models get better and better. And so the [6:55] harness needs to become more and more [6:56] complicated uh [6:58] to [7:00] allow the model to do more things, [7:02] right? And so um an example of this is [7:05] uh auto mode. And so auto mode, you [7:08] know, is a classifier that runs after [7:11] every, you know, uh task that you [7:14] normally have to ask a permission uh [7:16] prompt for for Claude. And back when we [7:18] were like, you know, Opus 4 or even Opus [7:21] 4.5 maybe like uh it was not so bad to [7:24] hit enter on the permission prompts [7:25] because like you were you know, the [7:27] turns would only last a few minutes [7:29] anyways, right? And now Claude is [7:31] running, you know, can run for hours. [7:33] And so you really need that ability for [7:36] it to you know, like do work safely, [7:38] stick to your instructions. And auto [7:40] mode is really complicated software, You [7:42] know, sandboxing is really complicated [7:44] software. [7:45] Um and then like, you know, you also [7:47] think about things like, okay, like [7:48] Claude could do so much more work now. [7:51] Um [7:52] how does it represent the work that it's [7:54] done? Like, it's worked for like 8 hours [7:56] and you want to know what it's done in 8 [7:57] hours, right? So, [7:59] this is what we use artifacts for, [8:00] right? And artifacts themselves are a [8:02] form of prompting cuz like, how does [8:04] Claude represent that in a useful way, [8:06] right? Like, there's a lot of different [8:07] ways that it could uh represent that [8:09] information. [8:11] Um and so, I think like roughly [8:13] what we see is like the harness harness [8:16] engineering is definitely like this like [8:18] mix of science and art. I think it's [8:20] very unintuitive in a lot of different [8:22] ways. Um but I think it has like just [8:25] big abilities to like unlock new parts [8:29] of uh you know, like model behavior. And [8:33] uh yeah, I just see the harnesses get [8:34] more and more complicated and it's kind [8:36] of harder and harder to like [8:38] actually vibe coordinate your own, which [8:41] is like a little bit unintuitive to me [8:42] because of like how good the models have [8:44] gotten. Uh but yeah, things like auto [8:47] mode and workflows and things like that [8:49] that which are like actually quite [8:50] complicated pieces of software [8:53] become like really load-bearing, I [8:55] think. [8:56] >> As LLMs have advanced, more and more of [9:00] the human part can be taken out of the [9:03] loop. You know, at Anthropic, what [9:05] percent of [9:07] your or your team's changes are fully [9:10] autonomously made versus actually [9:12] pairing with a model like people were [9:14] doing more of like around a year ago? [9:18] >> Um I think it's very dependent on, you [9:21] know, what you consider autonomously [9:23] made or [9:25] um what [9:27] uh what team or what function you're [9:29] working on as well. You know, for [9:31] example, a designer might give you a [9:33] Figma file and then you pass the Figma [9:36] file to Claude Code. Uh Cloud Code is [9:38] great at using the Figma MCP, but like, [9:40] you know, the designers put a lot of [9:41] work in there. [9:43] Um I think that like roughly [9:46] um [9:46] our goal is to sort of get Cloud being [9:49] able to do all of the like [9:51] uh [9:52] the glue work that like ties everything [9:55] together, right? So, like, okay, you've [9:56] got like a Figma file. Do I really need [9:57] to like rewrite that in React or [9:59] something? Like, probably not, right? [10:00] Like, I I think like that work has been [10:03] done once, right? And so, the way I [10:05] think about it is like, what's the [10:06] unique work that I need to do every day, [10:08] right? And uh the more I'm like doing [10:13] that unique work, the better. And And [10:14] like, I think there's just a ton of [10:16] demand for [10:17] you know, unique work and thinking. And [10:19] the more I'm doing something I've done [10:20] before, I'm like, okay, like, can can [10:23] Cloud do this? Or like, if I'm just [10:25] translating what someone else has done, [10:26] you know, I'm like, oh, like, can Cloud [10:28] do this as well? [10:29] It's really blurring the lines of like, [10:31] what what does it mean? Even Even if [10:33] Cloud has fully generated this PR, [10:35] you've probably done a lot of [10:38] uh made a lot of decisions and and add a [10:40] lot of context along the way, so. [10:42] >> How close are we to a world where [10:43] someone just says, "Hey, Cloud, [10:45] here's the ticket. Just [10:47] don't tell me till you're done." [10:50] >> Yeah, so I mean, it depends on how good [10:52] the ticket is. If someone has like [10:54] perfectly written out essentially like a [10:56] software spec for this ticket, then [10:58] yeah, actually Cloud can probably do [11:00] that, right? But I think that like um to [11:03] start, you know, like let's say that we [11:05] get a GitHub issue, that you know, is [11:08] this worth fixing? Is this a feature [11:10] like often times issues are like also [11:13] feature requests or something where [11:14] there might be like multiple things [11:15] happening that maybe you combine [11:17] together in a different way, you know [11:19] what I mean? And so, I think that there [11:20] is [11:21] a a lot of that work is like, okay, [11:23] what's the vision for the product? Where [11:25] Where do we want to go? How do we like, [11:27] you know, [11:28] um [11:29] make sure what we we're building is [11:30] cohesive. And so, I I think for a [11:33] specific spec like a specific enough [11:36] spec, Claude can do it. In practice, [11:39] people don't have haven't figured out [11:43] what they really want, you know? And [11:45] haven't figured out the unknowns or the [11:47] shape of the problem and things like [11:49] that. And maybe the way you would have [11:51] done it before is like you start writing [11:53] the code and you're like, "Oh, like what [11:54] am I supposed to do now?" Like or like, [11:55] you know, like you you figured out that [11:56] way. Uh now, I think you need like new [11:59] ways of figuring out what you don't know [12:01] yet, right? And I think you can still [12:02] chat to Claude really. [12:04] Um but I think [12:06] uh that's the hard part. And I think [12:08] definitely a failure mode is like [12:10] you you tag Claude tag and you're like, [12:12] "Hey, please do this." [12:14] You know, one sentence or less [12:16] description, no previous context or [12:18] memory, [12:19] um and then it does it and you're like, [12:21] "Oh, no, I don't like it." You know, and [12:23] then you're like, you know, just [12:25] iterating forever on that, right? Versus [12:26] like figuring out what you actually want [12:28] like really quickly. [12:30] >> When I was working at Meta, like the [12:32] ambiguity of the task really ranges [12:36] often times proportional to people's I [12:39] mean, levels aren't everything in [12:41] software engineering, but it's roughly [12:43] senior engineers kind of take that [12:46] ambiguous uh business need and convert [12:49] it into something that's a lot more [12:51] concrete. And then people who are maybe [12:53] newer in their careers, they kind of do [12:55] the implementation work. Um and like [12:58] intern projects were kind of almost line [13:01] by line specified. If I was still uh [13:04] working at Meta and I had an intern, I [13:06] was like [13:07] just kind of give the give Claude my [13:09] intern's back and it would kind of do [13:11] it. [13:12] What kind of work does an intern do then [13:15] at Anthropic if Claude can kind of [13:17] handle it? [13:18] >> What we see in practice is that there [13:21] are just so many new types of work that [13:23] no one has ever done before, right? And [13:25] I think that like we all have to figure [13:27] that out and so for example, how do you [13:30] do an eval, you know what I mean, [13:32] against like [13:34] uh these new coding behaviors, right? [13:36] Like how do you make sure like [13:38] uh that like, you know, how do you [13:40] measure Claude's performance across like [13:42] millions and millions of users, uh [13:45] million across doing all sorts of tasks. [13:47] Sometimes these tasks might have [13:48] trade-offs, you know, so I think there [13:49] is a lot of new work to be done, [13:53] especially like in the research side of [13:55] things and um I think that like it's [13:58] harder and harder to write those specs, [14:01] but there's also less and less [14:02] experience that is relevant, you know [14:05] what I mean? And so like I think that [14:08] yes, having some experience means that [14:09] you can sort of like [14:11] you know, [14:12] you know how to get work done and you [14:15] know how to like learn, but I think you [14:17] also get those skills out of college and [14:19] and you know, now I think as an intern [14:22] trying to figure out like, okay, what [14:23] are the things that people have not done [14:25] before and and doing that I think is [14:26] really exciting and I think that there [14:28] is um [14:30] Yeah, I think internships are less write [14:32] the react code, you know what I mean, [14:34] and there's so many new problems to [14:36] solve. We need people to solve them. [14:37] It's helpful if you have fresh set of [14:40] eyes on it, you know? Um [14:42] but it is definitely be more proactive [14:44] and like opportunistic maybe than uh [14:47] than before or if you're it was maybe [14:48] more of like a pipeline. [14:51] >> We talked a little bit about knowledge [14:52] work and that makes me think of computer [14:55] use and browser use and I want to ask [14:58] you, you know, how far away is that from [15:01] being widely adopted in the industry and [15:05] impactful? Maybe you can talk about how [15:08] Anthropic uses it since it feels like [15:09] you're always far ahead. [15:11] >> I I think computer and browser use the [15:12] models have gone a lot better. Opus 5 I [15:15] think is like a really good computer use [15:16] model, but there are like these weird [15:18] edge cases where like for example, it [15:21] can't type a password on my behalf [15:22] because my passwords are in one password [15:25] or something and [15:26] you know, it can't access that and so [15:28] like it gets stuck and um I think [15:30] there's still like those edge cases [15:32] which are kind of UXE edge cases. Um [15:35] and then I think also obviously computer [15:37] uses also [15:39] uh you know, the more and more people [15:41] turn APIs and MCPs into like new ways to [15:45] use Claude. I think that can also take a [15:47] lot of the use cases that you're using [15:50] computers for, right? And so going back [15:52] to like oh yeah, knowledge work [15:53] everything is code. I think like you [15:56] know, there are some things where [15:57] there's no API, there's no way to [16:00] execute in code other than just like [16:02] opening up your browser. But I think [16:04] increasingly there are more and more [16:05] ways and I think yeah, Claude tag is a [16:07] great example of like we've just sort of [16:08] like tried to [16:10] roll up all these things into APIs that [16:13] it can use and so [16:15] um it feels like it can do a lot of work [16:17] on your behalf even though even though [16:18] it's not [16:20] literally running like a virtual [16:21] computer and clicking things, you know. [16:24] >> I have noticed that whenever I use [16:26] uh any kind of computer use tooling, it [16:29] feels [16:30] painfully slow. I I look at the cursor [16:32] and it's sitting there for 10 seconds, [16:34] moves over, you know, goes there for 10 [16:36] seconds. What where is all that latency [16:39] coming from? Do you have a sense? [16:41] >> I think that like it's hard to make a [16:42] small model that's really really good at [16:44] it. I think just cuz there's a lot of [16:46] knowledge that you need to have about [16:48] this task and how these things work [16:50] together and stuff and so [16:52] um you want like a smart model and smart [16:54] models, you know, take a little bit [16:55] longer time. It it's like I think a lot [16:58] of people thought we'd get here sooner, [17:00] but I think it's just been a harder task [17:01] than we expected. Um I think like one [17:03] thing I've someone's told me about [17:05] computers before is that like [17:08] it's a state machine where you don't [17:09] control the entire state. If you are on [17:12] the DoorDash website or something you [17:13] want to add something to the cart and [17:15] you add it incorrectly, now you have [17:17] this new flow to like undo it. You know, [17:20] now you need to go click and now you [17:22] need to go delete it and you can make a [17:23] mistake along that side as well, right? [17:25] Whereas like in code, you can sort of [17:27] like undo, get, you know, whatever. You [17:29] control all of the state. [17:31] Um but for computer use like each action [17:34] is [17:35] if not irreversible, it's like [17:38] uh [17:38] you know, much harder to review reverse [17:40] than than others. [17:42] >> Anthropic, [17:44] um I get the sense that employees have a [17:47] lot of budget in terms of the compute to [17:50] kind of speed up whatever it is they [17:52] need to do. And so, to if we were to [17:54] spur the imagination of people who use [17:57] the models to make them more productive, [18:01] assuming they had infinite compute, like [18:03] what what type of workflows would you [18:05] start telling someone to do if they had [18:08] infinite compute? [18:10] >> I I think this difference is slightly [18:13] more exaggerated than you'd think, I [18:15] think. You know what I mean? I think [18:16] that like, for example, I use my Mac sub [18:20] on the weekends and I have almost never [18:24] hit a 5-hour limit. I think the models [18:26] are really smart and I think that like a [18:28] lot of times when we're spending a lot [18:31] of compute, we're just trying to find [18:34] sort of capabilities or we're trying a [18:35] bunch of different things and and it's [18:37] more about like us figuring out model [18:39] possibilities, you know what I mean? [18:41] Than getting a lot of work done. When [18:43] we're testing for math, for example, [18:45] we're trying to understand how smart is [18:46] the prob the the model and that is [18:49] useful to us. That's useful output to [18:51] work and it can also sometimes solve [18:53] like the Riemann hypothesis or something [18:55] or like not make progress, you know what [18:57] I mean? People can replicate what we do [19:00] at home just by thinking at a higher [19:02] level of abstraction. For example, I'm [19:04] trying to like get Claude to draft [19:06] feedback for you. And so, [19:09] I want the funnel for like Claude to [19:11] draft with some feedback to the user has [19:13] submitted feedback to be really good, [19:15] right? And so I monitor that funnel and [19:18] then I ask Claude [19:20] I I had some ideas, but then I was also [19:22] like, "Oh, what if I ask Claude to try [19:24] and improve the funnel, you know?" And [19:26] it'd be like, "Here are some ideas. Can [19:28] you come up with some as well? Let's [19:29] figure it out." All of those things you [19:31] can kind of do yourself right now, you [19:33] know, if you're like, "Okay, let me when [19:36] I'm making a feature, let me like [19:38] uh annotate it with the events, you [19:40] know, let me make sure that Claude has [19:42] access to those events. Uh let me run a [19:45] loop in the morning every day to check, [19:48] you know, what events are fired and what [19:51] changes were made. Maybe even like let [19:53] me proactively suggest some ideas, [19:54] right? This can all happen, I think, [19:57] within a fairly reasonable amount of [19:58] compute. I actually what I see more [20:00] often is people running into limits [20:02] where they've actually done kind of the [20:04] opposite. They've started with a small [20:06] mid-scope task, you know, it's like, [20:07] "Oh, hey, like uh you know, refactor [20:10] this function in this way." And then [20:12] Claude does it and and maybe it's like [20:15] has some follow-on effects cuz like [20:16] refactoring this has means you have to [20:18] do some other work, too, and that's not [20:20] exactly correct or like, you know, you [20:22] you're iterating there and it's like, [20:24] you know, you just spent a lot of time [20:26] whereas [20:27] uh [20:28] if you had sort of like stepped up a [20:29] level, told Claude your goals, then [20:32] figure out like, "Okay, what are the [20:33] details [20:34] you know, do some exploration, uh maybe [20:37] write out the schema or like, you know, [20:39] and and then work with it. Then let it [20:40] run. You could probably get the same [20:43] output. [20:45] >> I remember it was going kind of viral [20:47] this idea of creating loops. Um [20:51] and I mean that that does feel like a [20:53] pretty [20:54] uh expensive sort of thing to set up if [20:57] you're just asking it to kind of keep [20:58] hammering away. Maybe first could you [21:00] define this loop engineering thing and [21:03] then I'm curious how often do you use it [21:05] and [21:06] you know, would you hit limits if you [21:08] were on a max plan doing that kind of [21:09] stuff? [21:10] >> Yeah, so I okay, so loop engineering [21:13] roughly it's like [21:16] instead of prompting Claude directly, [21:18] you're setting up a system that prompts [21:20] Claude. You know, and [21:23] you don't have to think of it if you use [21:25] Claude tag a lot of this comes [21:26] naturally. You just ask it like, hey, [21:29] every day do this thing, you know, and [21:32] that will that's a loop. You can [21:35] definitely do that sort of work right [21:37] now, but I think if you're trying to set [21:38] up like let's let's say like [21:40] 10 loops or something, you know, that [21:42] are like uh [21:45] triaging your feedback and implementing [21:48] and things like that. We do that sort of [21:49] work, but we also spend a lot of time [21:51] making sure our skills and things like [21:54] that are useful, you know, and like are [21:57] good at like triaging [21:59] that we can [22:00] how we're good at verification and so [22:03] that we can like make sure that the [22:04] changes land, you know, and then I think [22:07] at that level, you know, what it means [22:10] is like now we have someone monitoring [22:12] issues that we just could never have [22:14] kept on top of before, right? And it [22:16] just like increases our software [22:18] development velocity and it's worth a [22:19] lot of value to us. And so yeah, I think [22:22] that like if you're kind of at the scale [22:23] of like, okay, you want it to [22:25] essentially be autonomously running, you [22:28] know, [22:29] doing a software engineering job. For [22:30] certain cases, I think it can do that if [22:33] you set up the verification well, if you [22:35] set up your skills well, if you give it [22:36] the right data sources, but um it is a [22:39] lot of work, right? And I think that [22:40] like you have to make sure that that [22:42] work is valuable to you in the same way [22:43] that like if you hire someone, you know, [22:45] you have to make sure that work is [22:46] valuable. [22:47] >> We've talked a little bit about [22:48] Anthropic being kind of [22:50] just ahead because that's the job of the [22:52] company on adopting AI. Is there [22:55] something that you feel is kind of the [22:57] next big shift that maybe Anthropic's [22:59] already felt that you feel [23:01] is going to diffuse into the industry [23:04] and if so, what might that be? [23:06] >> Yeah, I think there are few different [23:09] ones. Obviously, Claude Tag, I think, is [23:12] our, you know, the way we do a lot of [23:13] our work right now, right? And I think [23:15] the way I think about it is that Claude [23:17] Code is really good at the [23:18] implementation of code, and Claude Tag [23:20] is for the rest of the software [23:22] development life cycle, you know, so [23:24] like so [23:25] getting feedback, [23:27] uh you know, doing code review, like [23:30] babysitting, building CICD, incidents, [23:34] like, you know, all of these other [23:35] things, Claude Tag is really good at. [23:37] At a high level, turning every part of [23:40] your software development life cycle [23:41] into kind of like a a routine or a loop [23:45] using Claude Tag or something like that, [23:47] I think is probably where things are [23:49] headed, you know? [23:50] Um I think probably generative [23:53] interfaces are still to come, and I [23:55] think that like artifacts and, you know, [23:59] uh yeah, basically artifacts, I think, [24:01] are going to be like an increasingly [24:03] large way of how you like interact and [24:05] read with Claude. And and so I think [24:07] that, you know, I use artifacts for [24:09] almost everything, and and so I think [24:10] that like we're going to see that also [24:12] become big, and I think that's also ties [24:15] into Claude Tag, right? So like let's [24:16] say that you are um on your phone and [24:20] Claude Tag has done a bunch of work for [24:21] you. It creates a report. The report is [24:23] readable, you know, on your phone [24:25] because it's made the like the artifact [24:27] look really great. [24:28] >> Could you explain [24:30] uh artifacts a little bit or kind of [24:31] what are they and how do people use [24:33] them? [24:34] >> Artifacts are basically [24:36] um [24:37] Claude can upload essentially a web app [24:40] for you to use, you know? And I think [24:41] that [24:42] uh [24:44] you can use it in a really wide range of [24:47] capabilities. They can now, for example, [24:50] call your MCP. So you can make an [24:52] artifact, for example, to read your [24:54] inbox, you know, and like display it or [24:57] sort it or tag it, right? As like a one [24:59] way. You can also use artifact to uh [25:02] show you a plan for coding, right? And [25:04] then in that it might show like diagrams [25:06] and file snippets and code and schemas. [25:09] Um and so it shows you like sort of uh [25:12] it's like an interactive [25:14] display for the tap for the job you're [25:16] doing right now, you know? And as Claude [25:18] does more and more, you know, like jobs, [25:21] we're realizing that like just text in [25:24] text out is probably not useful for [25:26] everything, right? And Claude is doing [25:29] is better and better at creating [25:30] essentially the exact interface for you [25:32] at the right time. Um I think that [25:35] there's like a lot more to do there, and [25:37] I think [25:38] it's another one of those things where [25:39] you have to think about like, oh, like [25:41] you know, could I be interacting with [25:42] Claude in a different way, right? Like [25:44] could I interact with it through an [25:45] artifact? Could I [25:47] you know, use it to like learn more or [25:49] like to understand it better or stay in [25:51] the loop or [25:53] um [25:54] improve some of my like own knowledge [25:56] work or something. So, [25:57] um [25:58] yeah, I I I think it's pretty exciting. [25:59] We're still kind of early to it. [26:01] >> Does the daily driver model vary among [26:04] engineers, or do people typically just [26:07] pick the most intelligent one that's [26:09] available at Anthropic? [26:11] >> I do think, you know, similarly with [26:13] models, if you use the smart models and [26:14] you use them well and you give them [26:17] task where they can, you know, are [26:18] well-shaped and you've spent some time [26:20] setting up a good verification harness [26:21] and things like that, you can get a lot [26:23] out of them. [26:24] I don't think it's like choosing which [26:26] model at the right time, you know? I [26:28] think it's like, oh, how do you get the [26:30] most out of the frontier models [26:33] um because I think technology works the [26:36] way it does, right? Like everything gets [26:37] more abundant, more available. [26:40] Um and so I think we're in this current [26:43] weird spot where, you know, we don't [26:44] quite have enough compute for everyone [26:47] to have Fable at 100% of the rate [26:48] limits. [26:49] Um but I don't think this is like a [26:51] durable skill to build figuring out like [26:54] oh when do you use Fable and when do you [26:55] Sonic, right? So [26:58] I think it's like unintuitive there. [27:00] But I think in practice if you're today [27:03] what I would do is like I use Fable for [27:05] planning, for brainstorming, for finding [27:08] unknowns, for coming up with the [27:09] detailed spec and I'd use Opus 5 to [27:11] implement it. [27:13] And I would probably use Opus 5 with [27:16] workflows [27:17] using like a verification aid like [27:19] schema or like harness that's built with [27:21] Fable, right? So I'd use Fable for those [27:23] high leverage task. [27:25] And yeah, Opus 5 for the execution and [27:29] implementation. [27:31] But yeah, I think increasingly probably [27:33] next year I I I think you're just not [27:34] going to be thinking that much about [27:36] like which model. [27:38] >> You mentioned that skill of I guess [27:40] prompting and I've heard some people [27:44] they say [27:46] it's not too durable of a skill because [27:49] it's kind of it's really specific to a [27:51] model. Like these models they almost [27:52] have their own unique [27:55] I guess spiky intelligence. So if you if [27:58] you knew everything that was very [27:59] specific to let's say today's Fable and [28:02] you're a master of today's Fable [28:05] I mean maybe you don't need to know any [28:06] of that stuff like a year from now and [28:09] Fable 3 or 4 or 5 whatever comes out. [28:13] Let's say you were talking to software [28:15] engineer who's looking for career advice [28:18] and they're thinking hey should I really [28:20] become a master of engineering my [28:23] prompt? [28:24] >> I do want to say prompting is like [28:26] little bit more than just a prompt you [28:28] put in. It's also you know, you might [28:29] have done made a skill or you might have [28:31] like you know, added some data or [28:33] something. It's not just a prompt you [28:34] write but it's like everything you've [28:36] done before that builds up into your [28:37] context, right? So [28:40] sometimes people see us write small [28:42] prompts and they're like oh what is that [28:43] mean? But we we just spent so much time [28:46] on [28:47] the the [28:48] harness and the verification and and the [28:50] skills. So, [28:51] um [28:52] I think it will be like really valuable [28:54] to just keep better at prompting. I [28:56] think to what you're saying about each [28:58] model is different, you're right. Like I [28:59] think each model is kind of its owns [29:02] kind of like almost organic digital [29:05] thing, you know? And so, there are [29:06] quirks you have to learn and you do have [29:09] to unlearn them. So, [29:11] we recently wrote about like how we [29:13] removed 80% of the system to prompt from [29:15] Claude code, right? Um and one of the [29:18] learnings we had was like we needed to [29:20] remove examples from the tool [29:22] descriptions. And you know, this used to [29:24] be the only way you could get good [29:26] output from the models was through [29:29] tools, right? Or or through examples, [29:31] right? So, you'd have to be like, "Hey, [29:32] this is the right tool. Use this. Here [29:35] Here's an example of writing a [29:37] file well and here's an example of not [29:39] doing it well." [29:40] Um [29:41] and now we found that like examples are [29:43] mostly negative, I think, unless um [29:47] you really see Claude doing something [29:48] you don't like because [29:50] uh it's just like quite imaginative. [29:52] It's good at sticking to your intention [29:54] and working with you. Um and so, we [29:55] removed a lot of examples. So, in that [29:58] case, yes, you do have to sort of like [30:00] adapt, but the skill you're building is [30:03] the skill to adapt. You learned how to [30:05] use Fable and now you know a lot about [30:08] Fable, but you also know how to learn [30:10] You learned how to work with a model, [30:12] right? And so, Fable 5.5 comes out, you [30:14] need to [30:15] learn how to use it again as well, [30:19] but you'll be much faster cuz you're [30:21] better at learning. [30:22] You're better working with Fable 5, you [30:24] know? And I think for me, I think the [30:26] first model I worked with was GPT-2, and [30:29] I remember like it was it was so hard to [30:31] get [30:32] a JSON output out of GPT-2. Like if you [30:35] could just [30:36] get it to like choose one of the [30:38] categories that you gave it, that would [30:40] be like incredible, you know? And so, [30:42] um I think that but like [30:45] building that skill, GPT-2 is such a [30:47] different model than Fable 5, but I feel [30:50] like the skill I spent doing that has [30:52] like helped me be better at prompting [30:54] Fable 5. [30:56] >> Is there any like tribal knowledge or [30:57] quirky tips in today's models where [31:00] you'd say [31:01] someone should should know that to get [31:03] more when they prompt? [31:04] >> I actually need to counter one I think [31:06] that people have been saying where it's [31:07] like, "Oh, just believe in yourself." or [31:09] something. I I know that [31:11] Jared [31:13] Jared's post about the Riemann [31:14] hypothesis had like he was just like, [31:16] "Keep going. Just believe in yourself." [31:18] I think in this case it was mostly just [31:20] Jared saying it's okay to use compute to [31:23] solve this problem and I'm giving you [31:25] permission to do it, you know? And I [31:27] think that like that's not exactly the [31:29] same as [31:31] you know, I believe in you, you know [31:34] what I mean? It's really just like [31:35] letting the model use compute. So, I [31:38] think that that is something I would [31:41] I like to tell the models right now is [31:42] like, "Okay, hey, I think this is a hard [31:44] problem. Use sub agents, you know? Use [31:47] workflows. Like if you need it, right? [31:49] So, like I always tell it to like use [31:51] its own judgment, but I'm giving you [31:55] permission to do this stuff, right? And [31:58] I think that [31:59] uh [32:00] you have to sort of remember that the [32:01] models by default, you know, do what [32:04] maybe the average user wants, which is [32:06] like they want it to respond and start [32:08] doing work as fast as possible. [32:11] Roughly like complete the task, but not [32:13] spend like a crazy amount of compute on [32:15] it, you know? And so, I think that like [32:18] um you have to sort of if you want the [32:20] model to do it differently, you have to [32:22] nudge it slightly, right? So, you might [32:23] have to be like, "Okay, hey, like I [32:25] don't want you to do any work yet. I [32:27] want you to brainstorm, you know? I want [32:29] you to like think with me, right?" Um [32:32] and if you prompt it that way, it will [32:33] start doing that. If you want it to [32:35] spend a lot of compute, if you're like, [32:36] hey, [32:37] you know, sometimes I'll say like [32:39] Yeah, hey, I think this is a hard [32:40] problem. Uh feel free to use workflows. [32:43] If I'm running overnight, I might just [32:44] be like, hey, I'm going to sleep, you [32:46] know, set the slash goal or something [32:47] and then [32:48] uh let it let it run. So, [32:51] um [32:51] yeah, I think there is a [32:54] uh just like giving it permission to do [32:56] the thing you want. [32:57] >> When you recently removed so much of the [33:00] system prompt, [33:01] uh how did you prove that [33:04] the end result was better? [33:06] >> We have a bunch of user metrics just [33:08] like how, you know, how much do people [33:10] like the output of Claude? Do you [33:12] something you get that survey and you [33:13] see it. Um [33:15] we run evals against, you know, our [33:18] internal eval uh and external evals to [33:21] see like how it performs at these [33:23] different tasks. Um [33:25] but I think it is hard like sometimes [33:27] you don't realize that they're not If [33:29] Claude is telling the user if it does [33:32] all its work and then it's like, hey, [33:33] maybe you should go to sleep, there's no [33:35] eval for Claude tells you to go to [33:36] sleep, you know what I mean? And now [33:38] we're like have to like catch this like [33:40] new behavior. Um so, it is hard. I think [33:42] we spent a lot of time [33:44] basically just um [33:46] you know, like removing lines in the [33:47] system prompt, running evals, uh seeing [33:50] how it works, seeing how people uh [33:52] reported it internally, and then like [33:54] adjusting, but it was like a full-time [33:56] job for several people over long periods [33:58] of time. And so, [34:00] I don't think I'd necessarily recommend [34:01] everyone do this. I think that's kind of [34:04] why we wrote that post about what we [34:06] learned from like adjusting the system [34:08] prompt. And we think that's pretty [34:10] general. So, hopefully you don't have to [34:12] like now [34:13] go through this like crazy iteration [34:15] process. [34:16] >> OpenAI, Anthropic, Cursor, and Vercel [34:20] all use this product to make their lives [34:22] better. [34:23] And the problem it solves is when you're [34:25] building SaaS or an ad product and you [34:27] want to sell to other companies, there's [34:29] all these requirements you need to meet. [34:31] There's SSL, there's SCIM, there's RBAC, [34:35] there's audit logs. These are all things [34:37] that take time to integrate, but aren't [34:39] the main focus of your app. WorkOS is an [34:41] API layer that lets you meet all of [34:43] these requirements in just a few lines [34:45] of code. So, let's say you have a new [34:47] SaaS product and you want to sell to [34:49] other companies, WorkOS will solve all [34:51] of these critical feature gaps for you. [34:54] You can check them out at workos.com to [34:56] learn more and get started. And I [34:59] appreciate them for supporting my work [35:00] and sponsoring this podcast. [35:02] >> I think a lot of people [35:05] when they use Claude code, they get [35:07] excellent results when it's kind of [35:09] getting [35:11] very objective work done. [35:13] But when it's kind of prompting models [35:15] to do beautiful or tasteful work, it's [35:18] kind of not always, you know, it's [35:20] pretty much more hit and miss. And so, [35:23] you know, how do you best instill like a [35:25] very particular style you're going for [35:27] or particular taste in the models [35:30] outputs when it's a much more subjective [35:32] domain, maybe like front end? [35:35] >> The way you do it is sort of like you [35:37] stay in the loop. I think you give it [35:39] references, right? And I think the more [35:41] references you give it with data, the [35:44] better. So, like it's better to give an [35:46] HTML file than a screenshot, right? It's [35:48] better to give a Figma file than like a [35:51] raster image or something, right? Cuz [35:53] now it if Claude wants to know the [35:55] border radius of this thing, it's like, [35:57] oh, what's the Figma component border [35:59] radius and let me just copy it over. And [36:02] so, I think giving it a bunch of [36:04] references, ideally in code, is a really [36:07] good way, right? Of like getting it to [36:09] stick to this. Um [36:11] I think if not, you can then ask it to [36:14] if you don't have like [36:16] uh let's say you're not a designer. [36:18] There's probably Step one is being like, [36:20] okay, now I'm not a designer. There's a [36:22] lot I don't know about design, you know? [36:25] And like there's was lot I don't know [36:26] about iteration. I don't even know what [36:29] good looks like, right? And I think this [36:30] is like part of the art of working with [36:33] a designer is like they will just know [36:35] what good looks like and they'll be like [36:36] this isn't good enough in this way, [36:38] right? And they like prompt it. And that [36:40] I think is also a skill that will keep [36:42] getting valuable and even more valuable [36:45] over time is just like what is good [36:47] output? What is worth doing, right? Like [36:49] I think uh for example with the uh the [36:51] Riemann hypothesis, Jared prompted it, [36:54] but he had no idea if it was correct [36:56] until like Lev who's like, you know, [36:59] one of the world's best mathematicians [37:01] who was like, you know, okay, like how [37:02] do I [37:04] is this correct, right? And he worked [37:06] with it and he asked it like tons of [37:08] follow-up questions. [37:10] And we could not follow that at all. We [37:12] had no idea what he was saying, right? [37:14] But he was like really intrigued. And [37:16] and so I think [37:17] more and more being that like high-cased [37:19] user and like knowing a lot about a [37:21] problem in a domain space is how you get [37:23] good outputs, right? That's how you [37:25] these problems. Uh [37:27] otherwise like maybe Claude did solve [37:29] like physics or something and but you [37:30] just wouldn't know it, right? Like [37:31] you're like [37:32] uh [37:33] you you don't know enough about it. And [37:35] so I think when you're talking about [37:37] design, the first thing you do is like [37:38] how do you become more tasteful with [37:40] design, right? And so you can ask Claude [37:42] that as well, right? Like you can be [37:44] like, "Hey, I'm not a designer. I want [37:46] to be better at design. I don't even [37:47] have the language. First maybe let's [37:49] find some reference sites." And that and [37:51] then maybe you pull some reference sites [37:52] and then you're like, "This is what I [37:53] like or this is what I don't like, [37:55] right?" And then you [37:57] sort of build up that, you know, those [37:59] references and you give Claude it. And [38:01] and now maybe you tell it, "Hey, let's [38:03] do some exploration. This is sort of my [38:05] taste." And then it'll do like a few [38:07] different mock-ups, right? And I like to [38:09] do these mock-ups all in HTML because [38:10] it's all self-contained, easy to edit. [38:13] And then once you have that [38:15] you know, that reference, now it's a [38:17] reference. So now you can be like you [38:19] can make a new session and be like, [38:20] "Hey, this is a mock-up of a design I [38:22] want. Uh start implementing this and, [38:25] you know, it will have all of it in code [38:27] and you can start getting there." Um but [38:30] I think that like the really hard part [38:32] is just knowing like, "Oh, when is [38:34] something good enough?" You know, uh [38:35] versus like when can you, you know, push [38:38] harder, right? And you see this with a [38:40] lot of the math proofs, too, right? Or [38:41] and when like Terence Tao is like, [38:43] "Okay, enough with the half-complete [38:45] theorems. Just do the full theorem." You [38:47] know, and and so I think like it sounds [38:50] simple, but it takes a lot of domain [38:52] knowledge and expertise to get to the [38:54] point to be like, "Okay, like you it [38:56] seems like you're smart enough to have [38:58] done that. Now do this." [39:00] >> When I was a software engineer, [39:03] there was a lot of um [39:05] I guess it was like glue writing work [39:07] kind of where you you finish some work [39:09] and you write a launch post or [39:12] you are part of some work stream and you [39:14] got to post an update every 2 to 4 weeks [39:18] or maybe you have a direction doc or [39:21] design doc and all this like writing [39:24] around the software uh that you actually [39:27] write. [39:28] And curious your thoughts on if that's [39:31] changed at all at Anthropic, how much of [39:34] that is written by LLMs and how much of [39:37] that is, you know, human-written still. [39:40] >> Mhm. My rule of thumb is like if I would [39:42] be happy to show someone the prompt, I [39:45] would send them the output, right? And [39:47] so uh a lot of times the prompt is just [39:51] collecting context is the most common [39:52] one, right? So, for example, uh before [39:55] every one-on-one with my manager, I ask [39:57] Claude to, you know, read every Slack [40:00] message and GitHub message and, you [40:02] know, a [40:03] PR and compile like, you know, a report [40:07] of what I did, right? And uh that's [40:10] context that my manager doesn't have [40:11] because, you know, they haven't been [40:13] literally reading every PR or something. [40:15] And so I don't feel bad if like, you [40:17] know, [40:18] like my manager could also run this [40:21] command, but it doesn't have my context. [40:23] And I don't feel bad with them knowing [40:24] that this is the prompt I used to [40:26] generate this command, right? So, [40:29] um I think likewise, like if you're [40:31] sharing updates or something like that, [40:33] you know, like uh you might want to [40:36] prove that you've read it. You know, I [40:38] think this is important. And And [40:39] sometimes like little edits are ways of [40:42] proving that you have also understood [40:44] this work, right? And so, if it's just [40:46] like a data readout, you know, maybe [40:48] you're you're sanity checking like the [40:51] numbers make sense and this like these [40:53] are all the numbers you intended to [40:54] include. And as part of that, maybe you [40:56] format it differently or you add like a [40:58] little sentence in on your behalf, [41:00] right? And but I think ultimately, it's [41:02] a data readout. [41:03] Everyone knows the prompt you wrote is [41:05] like, "Hey, like generate a readout on [41:07] this feature based on this." And And [41:09] you're fine, right? But like, I think if [41:11] you're pitching like a new concept or [41:13] new idea and your prompt is like, "Help [41:15] me come up with a new concept for, you [41:17] know, this product, right?" Like, you [41:19] probably at least [41:22] people want you to know that want to [41:23] know that you've like believed in it, [41:25] right? And so, like even if you think [41:27] Claude's idea is incredible and just [41:29] verbatim you wouldn't change anything, [41:31] what I would say is like, I'd be like, [41:32] "Hey, Claude generated this. Um but I [41:34] think it's great. You know, like I or [41:37] I've did like a hundred different [41:38] generations and I think this was really [41:39] good. And here is like something, you [41:42] know, that I want to send you, right?" [41:44] >> [snorts] [41:44] >> Um and so, I think that like it's good [41:47] to be upfront about it, I think, right? [41:49] Because I do think what people don't [41:51] like is when they feel like they've been [41:53] like misled a little bit. Like, "Oh, [41:54] like you We thought we you were doing [41:57] this work, but, you know, [42:00] um it it's really Claude, right?" And I [42:02] think that like um [42:04] writing sometimes can have a lot of like [42:06] there's some parts of writing where [42:08] individual words matter. You know, like [42:11] funnily like tweets are I of a good [42:12] example of this where like the [42:13] individual tweet matters, right? So, [42:16] um [42:17] you like more and more you can't use [42:19] Claude to do that because it's like like [42:21] every word has some thought that you've [42:23] put into it and some intention that [42:25] you've put into it. So, like, you know, [42:27] yeah, a pitch or an essay or something [42:29] like that. We do a lot of like internal [42:30] essays at Anthropic being like, "Hey, [42:32] this is why I think we should do this." [42:34] And that's generally like all [42:35] human-written. It's very like looked [42:38] down upon, I think, to have like Claude [42:41] like your essay written by Claude, you [42:42] know what I mean? Um because like every [42:44] word is something that you like are [42:47] intentional about. [42:48] >> So, it sounds like the proportion [42:51] of the writing that's all that boring [42:54] route writing, like the data readouts, [42:56] the [42:57] the one-on-one updates, the work stream [42:59] updates, that's increasingly becoming [43:01] AI, but always reviewed by human. And [43:04] then the novel thoughts, novel [43:07] direction, is still very human-written. [43:10] And And feels like it should remain that [43:12] way even if the models were a little bit [43:15] better, too. [43:16] >> Yeah, I think it's like if you're, you [43:18] know, it's a like writing is also way of [43:21] thinking, right? And so like maybe if [43:22] you need to think about the data stream [43:24] or data readout more, then maybe you [43:26] need to like summarize it, you know? And [43:27] And so, [43:29] um but yeah, I I I think like especially [43:30] gathering context is one of those things [43:32] where no one will ever like hold it [43:34] against you, kind of, right? Like uh [43:36] oh, like, you know, you did a bunch of [43:39] research and, you know, Claude did this, [43:41] but yeah, like I don't want to ask my [43:42] agent to do the same research. Like, you [43:44] know, it's a shortcut, but like just [43:46] acknowledging it is good. [43:48] >> Someone I also was talking to, they had [43:51] this thought of it would be valuable to [43:54] have almost like a git blame, but it's [43:57] like like a prompt blame of [44:00] cuz it would be nice to reverse look up [44:03] what was the prompt that generated this [44:05] change to kind of debug things. Do you [44:07] have any sort of meta version control on [44:11] the prompts that generated the software [44:12] or is it still very vanilla, you know, [44:15] get history? [44:17] >> Yeah, that is a little bit tough because [44:19] again like, you know, what goes into a [44:21] prompt is not just the prompt but also [44:24] the context and skills and things like [44:25] that. So like maybe, you know, a prompt [44:28] might seem basic but isn't. Uh I think [44:31] I it's kind of hard to judge. But I do, [44:34] you know, like going back to like when [44:36] I'm giving a PR, I don't think a PR at [44:38] this point is any different than an [44:40] artifact or something. Like Claude is [44:41] doing basically all the code writing, [44:44] right? So if I send someone a PR, I [44:46] usually also attach an artifact of every [44:49] prompt I sent to Claude, [44:52] um including failed like approaches and [44:54] things like that, you know? [44:56] Um so that they can see like, okay, I've [44:58] considered a lot of other things, you [45:00] know? And if I haven't, if this is just [45:02] a one-shot, I just tell people. I'm [45:04] like, "Hey, this is the one-shot [45:06] example. This is the prompt I used, [45:07] right?" Um [45:09] and uh I think that's like usually [45:12] impressive and interesting to them as [45:13] well because they're like, "Oh, like [45:14] it's cool that Claude could one-shot [45:16] this." Um but the worst is when you get [45:20] like [45:21] you know, you you just I'm just trying [45:22] to avoid cases where I send in like this [45:24] 10,000 line PR and they're like, "Did [45:27] you like how much have you like read [45:29] this or like, you know, like how much [45:30] did you work with Claude on it?" And if [45:33] I'm doing like a large PR, I am going to [45:35] like show my work as much as possible. [45:39] >> On maintaining code because AI can [45:42] generate such high volumes of code at [45:45] this point, do you have any tips on [45:47] what's worked well at Anthropic for code [45:49] ownership and maintenance? [45:52] >> I do think you have to revisit [45:54] like what is important with code [45:56] maintenance, right? And so I think that [45:57] like there are some things where like [45:59] naming used to be really important, [46:02] right? Because like it was like how you [46:04] as a team thought about this like [46:05] abstraction and feature. Um but I think [46:08] naming is becoming less and less [46:09] important. A lot of stylistic things in [46:12] code are becoming less and less [46:13] important, you know? And I think that [46:15] like this is not the same to me as [46:18] maintenance, right? So I think that like [46:21] um [46:22] if you you might want to sit down and be [46:24] like, okay, like what things really [46:26] matter now, what don't, what opinions do [46:28] we have that don't matter. So that's [46:30] one. And then I think on the second on [46:32] the maintenance side is sort of like [46:34] having good scaffolding, right? So I [46:37] think that it's like uh having a good [46:39] verification harness, having a good like [46:43] uh sort of skills. We use the simplify [46:45] skill a lot. I think that like you know, [46:47] sometimes even from model perspective, [46:49] like it might do a lot of work. And then [46:51] like even just giving it permission to [46:53] be like, "Hey, I think this is the right [46:55] uh idea. Let's simplify it." gives it [46:58] that permission to do it, right? But [47:00] like you actually don't want a model to [47:02] by default do work and then simplify, [47:05] right? Because uh maybe it's not [47:07] correct, right? Like you don't want it [47:09] to simplify work that's not correct. [47:10] It's like um you're wasting tokens, [47:13] right? And so I think this is also how [47:15] humans think, right? They're like you [47:17] know, you think generatively, you try an [47:19] approach, and then maybe you like [47:20] simplify and abstract a little. Um so I [47:23] think there's like some work inside your [47:25] own code base or like setting up the [47:27] skills and that those practices where [47:29] you like you know, you're not just [47:31] submitting [47:32] like the first take, but you've like [47:34] simplified it and tried it. Um and then [47:37] like yeah, having a really good [47:38] verification harness where you feel like [47:40] you're catching you know, like you have [47:43] a good [47:44] belief that Claude is like testing every [47:46] part of it. [47:48] Uh like you know, when you submit a PR [47:50] to Claude code, you get a recording back [47:52] of it using the feature and testing it [47:54] as an example, right? Um and you can [47:57] just get really, really creative with [47:58] different ways to like [48:00] uh, test stuff. Like I think you should [48:02] basically have [48:03] on the order of I'd say more like a [48:05] hundred times more testing code than [48:07] you've ever had before. You know what I [48:09] mean? So like, [48:10] uh, [48:11] you should have [48:12] uh, fixtures for everything. You can [48:14] just pull production code and create [48:15] fixtures and mock-ups on the fly for for [48:17] databases. You can uh, have storybooks [48:20] for front-end and things like that. And [48:22] you can have all these different ways of [48:24] testing and verifying your code. And [48:25] that I think is really valuable for [48:27] maintainability, right? It's like um, [48:30] just having like all these ways of [48:32] verifying it. [48:33] Uh, and then uh, yeah, of course like [48:36] there's just like the human element of [48:37] like where do you want your code base to [48:40] go? If you know that's the case, [48:43] probably you start thinking about like, [48:45] you know, how you'd replay and undo and [48:47] redo becomes really important, right? [48:48] Like just like that sort of stuff. And [48:51] so the but if you're single player only, [48:53] that actually matters a little bit less. [48:56] You know, like you like the average user [48:58] is not going to undo a hundred times or [49:00] something, right? And so like, you know, [49:03] many like single player applications [49:05] undo stop working [49:07] pretty quickly. Like, you know, like [49:08] after like four or five times, right? [49:10] Um, just cuz it's not that important. [49:11] But in multiplayer, the ability to [49:13] compose different things together is [49:15] really important or compose different [49:16] operations together is really important. [49:17] And so, um, [49:19] that's just an example where like if you [49:21] know the direction of your code base, [49:22] there are things you care about and you [49:23] want the models to know. And uh, you [49:25] know, you can include that in your [49:26] skills. You can also just like [49:28] you know, like that's sort of what [49:29] maintainability means to me is like [49:31] having a vision for what your code base [49:33] is good at, where you're where you're [49:34] going, you know, keeping it in line. [49:37] Um, I do think the models are getting [49:38] better and better and better. And like [49:41] almost every code base probably I think [49:43] will have this moment where you're like, [49:45] do you just ask the model to rewrite all [49:47] of it? You know? [49:49] Um, and like because now you're like, [49:52] oh, like I can do it in the most [49:54] performant language. I can mix and match [49:57] like you know like there's not a reason [49:58] that every software in the world [50:00] shouldn't run in web assembly in the [50:02] browser. You know what I mean? There's [50:03] not a reason why [50:05] I don't know like your Xbox game can't [50:08] run in your browser actually, right? [50:09] Like the Xbox like the hardware like the [50:12] consoles are much weaker than your [50:14] average like MacBook right now. You know [50:16] what I mean? But it's just like the code [50:18] base, right? [snorts] Um but we could [50:21] like and and so I I think probably over [50:23] the next year or two like everyone's [50:25] going to need to think about that, [50:27] right? [50:28] And so I I don't think you want to spend [50:30] too much time [50:32] like worrying about maintainability in [50:34] this like way that might not matter [50:35] anymore. I'm not saying maintainability [50:37] doesn't matter. You just have to update [50:38] your mental model of like what does it [50:39] mean for like code to be maintainable, [50:42] you know? [50:43] >> I was talking to this friend um [50:46] who was saying that [50:48] he leaves tons and tons of tech debt [50:51] leaving around because the next you know [50:54] the next iteration why why waste time [50:56] fixing that now when the next iteration [50:58] of Fable's going to one shot all this [51:00] tech debt and refactor this for me. So [51:04] kind of funny. [51:05] >> Yeah, I I don't think that's completely [51:07] incorrect. It depends on the [51:09] circumstance, right? Like I think this [51:10] is also just a classic thing in [51:12] startups, right? Where you're like, you [51:14] know, even in normal human engineer you [51:16] always have this problem of like um [51:18] hey, do I refactor or do I do tech debt [51:21] or do I deliver more customer value? [51:23] Would refactoring help me deliver more [51:25] customer value right now? I do think [51:27] that like just generally I think you [51:29] should think on your projects on shorter [51:31] time scales. So you're like okay, how do [51:33] I deliver value over the next month or [51:35] two? And if you the project you're [51:37] talking about is delivering value over [51:39] six months or 12 months, you know, then [51:42] maybe yeah, maybe like wait for the next [51:44] model a little bit, you know, and then [51:46] like that like [51:48] just deliver value on like the short [51:50] time scale scale, make sure it's like, [51:52] you know, really good. And if the models [51:54] are not quite good [51:55] enough, like they might get there soon. [51:57] It depends very much on the like [51:58] specific case, right? But and I'm not [52:01] saying this is true of everything, but [52:02] like just something you should keep in [52:04] mind. [52:04] >> I've talked to some friends who work at [52:06] big tech companies like Google, [52:09] Facebook, those types of places. [52:11] And then they as they've become more and [52:14] more AI pilled, one thing that people [52:17] have noticed is there's a lot more [52:19] incidents or SEVs in in their usage. [52:22] And [52:23] it's natural in those organizations cuz [52:25] they they read the code less and there's [52:27] more code flying out. You know, what [52:30] countermeasures have worked really well [52:32] for Anthropic to prevent breakages given [52:36] that the code velocity is so much [52:37] higher? [52:38] >> Yeah, I I think this is something that [52:40] like is a byproduct of moving faster [52:43] sometimes and we have to figure it out. [52:45] Like I think that [52:46] uh you know, [52:48] I don't think our uptime is exactly [52:49] where we want it to be either, but also [52:52] as a company we're a little like almost [52:54] 6 years old I think around, right? So [52:56] it's like no company has grown this fast [52:58] before and a lot of that is because we [53:00] were enabled to like create more [53:02] products faster than ever before, right? [53:04] And so you can use Claude to make your [53:06] uptime better, right? And I think that [53:08] like the way we think about this is just [53:09] like really good what's the dream [53:12] testing environment, the dream like you [53:14] know, deployment environment. Like can [53:16] you like you know, take requests and [53:19] replay them across like, you know, mock [53:21] databases and fixtures across [53:23] everything? Can you chaos monkey [53:24] everything, you know? [53:26] >> In my personal workflows, I have so many [53:30] more custom random tools and scripts [53:33] that just make everything faster. [53:35] And just curious, you know, on your team [53:39] at Anthropic for instance, does everyone [53:41] have a set of [53:43] miscellaneous tools that help them, [53:46] you know, get random things done? [53:48] >> Yeah, everyone does for sure. Like I I [53:51] think [53:51] part of this is is the job. Like I think [53:54] that sometimes uh [53:56] what we try and do is we play around [53:57] with harnesses. Like sometimes some [53:58] people on the team build their own [54:00] harnesses for a little bit to figure out [54:02] like, "Oh, is this like useful or not?" [54:05] You know, and then they they figure out [54:06] if it works and if not, they like [54:08] integrate it, right? So, I think there's [54:09] like that's part of the job uh in some [54:12] ways. Um [54:13] I think like other examples of like sort [54:16] of like misc stuff people will do. A lot [54:18] of people will have like [54:20] uh unique Claude tag setups. I think [54:22] Claude tag is one of these things where [54:24] like, you know, you can have it like I [54:27] have it scheduled my calendar invites, [54:29] right? And so like if someone [54:30] wants to like schedule something with [54:32] me, then I'm just like, "Hey, you can [54:33] just like tag Claude here in this [54:34] channel and I'll accept whatever it like [54:36] puts on my calendar, you know what I [54:38] mean?" So, um I think there's like some [54:40] stuff like that. I know like a lot of [54:42] people use it for like email. I I think [54:45] that um [54:46] I've seen like some people do like [54:49] interesting like multi-Clauding sort of [54:51] like [54:52] tmuxing like setups, right? Like where [54:55] like what's the ideal scenario for you [54:57] to like display like 50 different Claude [55:00] codes and, you know, what's the best way [55:02] to for you to figure out what's going on [55:04] at any one time? So, um [55:06] yeah, I think there's a lot of different [55:08] ways and we're trying to make Claude [55:10] code more hackable as well so that like, [55:12] you know, more people can [55:14] uh sort of like even [55:16] uh make their own version of Claude code [55:18] like more different and everyone might [55:20] have their own little twist on it. [55:24] >> Your role at Anthropic's really [55:25] interesting cuz of the external [55:27] visibility that you have and um I think [55:30] a lot of people when they give career [55:32] advice, [55:34] visibility's a a good thing, [55:36] but they don't have this level of [55:38] external visibility. And so, do you [55:40] recommend to software engineers like [55:42] they should be posting on Twitter and X [55:45] and, you know, if so, what what advice [55:47] would you give in that sense? [55:49] >> Generally, [55:50] the thing I say to people that you [55:52] should share your work [55:54] externally as much as you can, [55:55] especially [55:56] um I think within certain companies like [55:59] you might not be able to, but like maybe [56:01] you have side projects or something like [56:03] that. I think just um before I joined [56:05] Anthropic, what I did was like I spent a [56:07] bunch of time working with different [56:09] companies, building stuff, and writing [56:11] about it, and talking about it. Um and [56:14] uh like this was really valuable cuz it [56:17] like increased my surface area of luck, [56:20] you know? And so, I think that like uh [56:24] it's really [56:26] like the bar is much lower than you [56:28] think. Like basically, whenever someone [56:29] asked me for advice, I'm like, "Okay, [56:31] I I'll I'll sit them down. I'll be like, [56:33] "I know statistically I I tell this [56:35] advice to a lot of people, and almost no [56:37] one does it. Um and everyone who's done [56:40] it is like either in a job that they're [56:42] pretty excited about or running their [56:43] own company. [56:45] Um [56:45] and there are reasons why you're not [56:46] going to want to do it, but like this is [56:48] it. Like I'm just going to tell you. [56:49] One, like I don't care about your [56:50] resume. Like like, you know, don't do [56:52] that. Um you have to choose like an [56:54] interesting project to work on. You have [56:56] to work really hard on it. You have to [56:58] like lock in, and then you have to ship [57:01] it and write about it. Like you have to [57:02] do it. You can't like There's going to [57:04] be so many reasons why you don't want [57:06] to. It's like not good enough yet, or [57:07] like you haven't like you think the [57:09] write-up is not very interesting or [57:10] something, but you have to do it, and [57:14] you can't get discouraged, you know? [57:15] Like maybe the first one might not work, [57:17] but you have to do it again. Um [57:19] and maybe Twitter is not the right [57:20] place. Maybe it's Reddit. Uh maybe [57:22] there's a specific Reddit or Hacker News [57:24] or something like that. Part of what [57:26] you're trying to do is you're just [57:27] trying to find people who like like what [57:28] you're doing, right? But I think that [57:30] like people in general want to really [57:35] like you know we since consume more [57:37] content than ever right and you know [57:39] this right like I think that like people [57:42] want really high quality content high [57:44] quality content as you know is a lot of [57:46] work and so I think going back to like [57:48] what we said before about like would you [57:50] show someone the prompt you did right [57:52] like I think a lot of times people are [57:54] like oh what's the shortcut like oh like [57:56] should I just ask Claude to manage my [57:58] Twitter account and is that how I like [58:00] grow and and I'm like no like like you [58:02] should not do that right like you have [58:03] to sort of engage authentically [58:06] you have to like post like you know do [58:08] good work and talk about it and I think [58:10] like build up networks and things like [58:12] that but I think that as long as that's [58:14] one of your goals and you try hard at it [58:16] I've been seeing anyone not succeed at [58:19] it but it is really hard it's kind of [58:21] like saying like oh I want to like go to [58:23] the gym every day or I want to like lose [58:25] weight or something you know like there [58:27] are simple things that you can do that [58:29] take a lot of discipline and are easy to [58:31] get discouraged with and [58:33] um [58:34] but like worth doing if you do it and I [58:36] think like posting or more specific more [58:38] specifically like writing about your [58:40] work is like I think really really [58:41] valuable. [58:42] >> You mentioned luck surface area. Do you [58:44] have an example that kind of illustrates [58:47] the value of expanding your luck surface [58:50] area? [58:52] >> Yeah I mean okay how did I get my job at [58:54] Anthropic? I did [58:57] a fellowship with a company called Good [58:59] Fire where I did like some applied [59:01] research and so they they're an [59:03] interpretability AI company and I was [59:05] trying to figure out like how do I use [59:06] interpretability to in like to make [59:08] better products and so I learned about [59:10] their research [59:12] and I uh like felt like I want to work [59:17] on a project and it was really important [59:19] for me to like share it and so like I [59:21] when I agreed to work with them I was [59:23] like that's my goal is I want to like [59:25] share what I'm building I think that's [59:27] good for you as well because good fire [59:29] is a startup and you know, people I want [59:31] they want people to learn about them. [59:33] And so this is what something I'm going [59:34] to do. So I built this like [59:35] interpretability visualization and I [59:38] shared about it. This is my first like [59:39] big post on Twitter but it was only 500 [59:41] likes or something. Like at like now [59:43] like you know, that's like not very much [59:45] to me but like it's like back then it [59:46] was like a huge post and um several [59:49] people saw it and like some of them DM'd [59:51] me about it and then uh like I then got [59:56] an intro into [59:59] you know, like uh into a role here and [60:01] so [60:02] um [60:03] I think that like I spent like a month [60:05] on that project I think in particular. [60:07] So it wasn't like you know, crazy um [60:11] and uh yeah, it just like showed people [60:15] that I could like do interesting work [60:17] and um [60:19] and I got paid for it too. So it wasn't [60:20] like I was doing it for free. [60:21] Um [60:23] And and there's plenty of examples of [60:24] like I think people are happy to do that [60:26] sort of thing for you. You know, like if [60:28] you like sort of show that kind of [60:30] initiative. Um but yeah, you have to [60:34] like it it it is like you really have to [60:36] like do interesting and novel work and [60:38] work work hard and be proud of your work [60:40] as well. You know, there's not like a [60:42] shortcut to that I think but [60:44] um if you do I think everyone is always [60:47] looking for interesting work and wants [60:48] to support you and you know, wants to [60:50] hire you or um [60:53] yeah, give you money like [60:55] uh lots of good things. [60:57] >> I see this tagline going out a lot with [61:00] some of the stuff Anthropic's been um [61:03] maybe like I think it's like Boris went [61:05] on some podcast and the tagline was [61:08] coding is largely solved. You should [61:10] people still learn to code if coding is [61:13] largely solved. [61:14] >> I think being technical is really really [61:16] important. Knowing how do computers [61:19] work, how does like um [61:22] yeah, how does how do computer programs [61:23] work? How do languages work? Like what [61:25] are the hard things and like what's a [61:27] back-end service? Like what's a cache? [61:28] Like what's what like like, you know, [61:30] what like what is memory allocation? [61:32] Like all of these things are actually [61:33] kind of really important to learn. [61:35] Um, I do think it's hard to motivate [61:39] yourself sometimes to do in the same way [61:41] that like, you know, doing math by hand [61:44] was not that motivating to me, you know, [61:46] but some people just love math and did [61:47] it. Um, I I think that like that's [61:50] probably [61:52] like something that people have to [61:53] figure out, but I think it is really [61:55] worth it. Like being technical is really [61:57] really important. Like we talked about [61:59] at the start of or like earlier where [62:01] like, "Oh, the only way you can tell you [62:02] solved you know, the Riemann hypothesis [62:05] or like made progress or whatever is [62:07] like or the Jacobian conjecture is like [62:10] if you're a great mathematician, right?" [62:12] And in the same way the only way you can [62:14] tell if you're like built great software [62:16] is like if you're a great software [62:17] engineer. Um, [62:19] I think that like how do you do that [62:22] hard, but like we've talked about some [62:24] of this stuff before just like staying [62:25] in the loop like, you know, like put [62:27] like, uh, [62:29] like putting like reflecting on your [62:31] process and and getting better and and [62:33] you can use Claude to learn as well and [62:35] and sort of explain things to you. So [62:36] like treating it like a thought partner. [62:39] Um, but learning like truly I I think [62:41] like, you know, Kaparthy says like [62:43] learning should feel like effort, you [62:45] know, and I think that's like one of the [62:46] hard things is like a lot of times even [62:48] if you ask Claude to explain something [62:50] to you and you might just like nod along [62:51] and you're like, "Oh, yeah, like I [62:52] learned it." But you didn't really cuz [62:53] you didn't put any effort in, right? So [62:55] I think that like it is really [62:57] technical. You should learn. It's hard [63:00] to learn. And and sometimes like what [63:02] school forces you to do is to learn [63:04] that. Um, I don't know truly like I'm [63:06] not learning programming from scratch. [63:08] So I don't know exactly what how to do [63:11] it now. I think if short of better ways, [63:14] I would still like type out and build [63:17] programs and run them and learn them. [63:19] You know what I mean? [63:20] Um there might be better ways that are [63:21] more cloud informed as well, but I think [63:24] it's really important. [63:25] Um [63:26] I think when Boris says coding is [63:28] solved, I think it just means like [63:30] um you know, we don't get stuck in the [63:33] same ways that we used to before. Like I [63:34] think coding used to be this very high [63:36] variability thing where you're like, oh [63:38] like could this bug take a day or could [63:41] it take 2 weeks? You have no idea [63:43] sometimes, you know? And I think like um [63:46] on the whole coding used to be like in [63:49] real terms [63:50] like something that was very rare for [63:53] something to go well. Do you know what I [63:55] mean? Like very few people in the entire [63:57] world could write software and they were [64:00] very very rare and even if you got them [64:02] all together, there were so many other [64:03] reasons why it wouldn't work, right? And [64:06] coding was this like one of the rarest [64:08] things in the world where like the [64:09] chance of software [64:11] project going well was like [64:13] on absolute terms very low, you know? [64:16] Uh and if you were like hiring someone [64:18] to [64:19] like make software for you for something [64:22] that's not like a huge product. Like if [64:25] you're hiring someone to make software [64:26] for your car dealership, you were almost [64:28] certainly not going to get the software [64:30] you wanted. You're going to get like [64:32] essentially scammed, you know what I [64:33] mean? Um not because anyone was trying [64:36] to scam you. It was just like software [64:37] was really really hard and you know, it [64:39] could only be spent on like the most [64:41] important scalable things in the world. [64:43] And now that coding is solved, I think [64:46] what we mean is that like you can use [64:47] coding to do all these other things that [64:50] we not done before, but that's not to [64:52] say that like that's not a lot of work [64:54] still. Um [64:56] it's just like it's not this like [64:57] incredibly rare difficult thing that [65:00] mostly like just doesn't work and you [65:02] have to spend like 8 hours a day locked [65:05] in to do. [65:07] >> What's really insane, that shows the [65:08] difference in expectation, is when I [65:11] used to write software, I would be [65:14] shocked if it worked on the first try. I [65:16] was like, "Whoa. [65:17] >> Wait, why is this working?" [65:18] >> [laughter] [65:19] >> And you you'd expect to kind of bash [65:21] your head against the wall a little bit, [65:23] and then it it works. Even if it's just [65:24] like you missed the semicolon or [65:26] something like that. And now I almost [65:29] have the flip expectation where it when [65:31] it doesn't work, I'm like, "Wait, what [65:32] what why did Claude What what happened [65:35] here?" [65:35] Usually, I expect it to work almost uh [65:39] the opposite [65:40] >> like on the first try. It's crazy. [65:42] >> Immediately. Yeah, yeah, I think humans [65:43] get really used to abundance, right? [65:45] Like there was this like [65:46] um article someone shared recently about [65:48] like going through a modern apartment [65:50] and talking about all the like wonderful [65:52] things that we have now that people [65:54] could not have imagined before. Like [65:56] something that can play music on demand [65:58] that's suited for your mood. You know, [66:00] like before you'd have to like hire a [66:02] musician, you know, to like go write [66:06] like, you know, incredible music, right? [66:09] Um but I'd say on the whole, probably [66:11] more people are getting paid to make [66:13] music now than ever before, right? Like [66:15] um and I think that like music is [66:17] reaching more people than ever before. [66:19] And I think probably the same thing will [66:21] happen with software, will happen with [66:23] math, will happen with all of these [66:24] things. Um [66:26] where you know, when you get abundance, [66:28] people are like, "Great. Like I want [66:29] more abundance. I want my software in [66:31] everything, you know? Like I want my [66:33] music everywhere, you know?" So, yeah. [66:36] >> Yeah, one potential other data point for [66:38] this conversation saying that maybe [66:40] people should still be technical or [66:43] learn how to code is I think earlier in [66:45] the conversation we talked about [66:47] automating knowledge work, and you [66:48] mentioned that [66:50] people who are technical had kind of a [66:52] leg up cuz they could kind of understood [66:54] how to coordinate Claude to do a variety [66:57] thing. Like you're calling FFmpeg to [67:00] automate some video editing, and like I [67:02] don't think someone who is not technical [67:04] would have that thought. So, it does [67:07] seem like even in this case where models [67:09] are doing a lot, there's still so much [67:11] value in being technical. [67:13] >> Knowing how computers work. You know, in [67:15] the same way that like like probably [67:18] like the most technical CEOs are like [67:20] the best CEOs are technical, right? Like [67:22] Zuckerberg, Elon, right? But like they [67:24] probably haven't written like a line of [67:26] code truly in a long time, but you know, [67:29] they understand how systems work, they [67:30] understand you know, constraints and [67:32] things like that and then that's really [67:34] really important. And so, [67:36] um you know, even if all that work is [67:38] changed, I think like being technical is [67:40] really really important. [67:42] >> And then last question for you is if you [67:44] could go back to when you just entered [67:46] the industry and give yourself some [67:47] advice knowing what you know now, what [67:49] would you say? [67:52] >> There are like kind of like two wolves [67:54] sort of I I think like you have to [67:57] believe in yourself, you know, I think [67:58] like this is really important and I [68:00] think that like at least when I joined [68:03] in the industry, it was rarer to believe [68:05] in people when they were [68:07] kind of the younger. Like I think that [68:09] was like the whole point of Y Combinator [68:10] or something was like believing in young [68:12] people early on to do really great [68:14] things. And so, I think that like you [68:16] know, even going back to like the intern [68:18] discussion we had earlier, right? Like I [68:20] think that like [68:22] thinking of yourself not as like someone [68:23] who's like being trained to do [68:25] something, but someone who can do like [68:26] something incredible right away, I think [68:28] is really important, you know? And um [68:31] I think I wish I had done [68:33] bigger, bolder things that I'd written [68:37] and and shared about. I think there were [68:38] lots of ideas where I was like, "Wow, [68:40] like I think [68:41] I think I had like I thought I had done [68:44] original work that I wish [68:47] maybe I'd even shared in some form, but [68:48] maybe not like form that it like [68:50] survived the internet, you know? And it [68:52] wasn't like a like goal of mine. And so, [68:55] I wish I'd like sort of been bolder and [68:57] like you know, [68:59] done and shared some of this work, but [69:02] at the same time you also there is a lot [69:04] to learn from people, you know what I [69:05] mean? And like I think that um [69:08] you like I think when you're young, [69:10] you're like, you know, you tend to fold [69:13] fall either you're not bold enough or [69:14] you're [69:15] you're not like [69:17] uh [69:18] you don't like learn enough, you know, [69:20] or you're like you're not like you don't [69:22] like you're too bold and you don't like [69:25] figure out what people have done before [69:26] and figure out why it's not working, [69:28] right? Um and there's this balance and I [69:31] think uh you everyone has different [69:33] failure modes, but I think like uh I [69:35] think I probably didn't take it enough [69:37] advantage of like, you know, mentors and [69:39] people who had learned a lot. Um and I [69:41] think I probably had to like relearn a [69:43] bunch of things as a result. Um [69:46] So, there's probably not universal good [69:48] advice, but just things to think about [69:50] as you're, you know, as you're entering. [69:53] >> Awesome. Well, thanks so much for your [69:54] time, Derek. Really appreciate it. [69:56] >> Yeah, of course. Thanks, fine. It's fun. [69:58] >> Hey, thank you for watching this [69:59] podcast. If you liked it and you want to [70:01] see the show grow, please support with a [70:03] comment or a like. [70:05] Also, if you have any recommendations [70:07] for people you want me to bring on, [70:09] please drop a comment. Guests like [70:11] Barbara Liskov, Mike Stonebraker, Mark [70:14] Brooker, these were all people that I [70:16] brought on because someone left a [70:18] comment. On another note, aside from the [70:20] podcast, I'm working on building the [70:22] ergonomic keyboard that I wish existed. [70:24] Here's a glance at the prototype. It's a [70:26] split keyboard, so there's two sides. Um [70:29] this is in the case. But yeah, we [70:31] launched on Kickstarter and we hit our [70:32] goal within 8 hours of launching. I [70:35] really appreciate it if you were one of [70:36] the people who grabbed one of the early [70:38] units. Um we're now working on the long [70:40] journey of building the tooling now, and [70:42] so if you still want to pick one up, [70:44] I've left the late pledges open on [70:46] Kickstarter, so you can grab one there. [70:48] I'll put a link in the description. [70:50] Thank you again for watching the podcast [70:52] and I'll see you in the next episode.