Anthropic Finds AI Models Plan Multiple Steps Ahead in Hidden Layers

Anthropic发现AI模型在隐藏层中规划多个后续步骤 由于下一个token预测在数学上要求模型构建内部世界模型并在生成文本前预先规划步骤，仅凭行为评估无法在不探查神经激活的情况下检测出欺骗性对齐。 Emmanuel Ameisen解释了机制可解释性如何将神经网络视为更接近生物有机体而非过程化软件的对象，并利用内部特征探查来解码模型的真实思维过程。他详细阐述了下一个token预测这一简单目标如何迫使模型发展出复杂的世界模型、无声的中间规划以及不变的概念表征。对话还探讨了平滑的缩放定律如何催生感知上的能力跃升、为何基于行为的输出检查无法检测出欺骗性对齐，以及像“金门大桥Claude”这样的实验如何展示对潜在表征的直接干预。最后，Ameisen概述了将模型拟人化为何具有实际功用，并对兼顾安全性的迭代式AI发展表达了乐观态度。 深度学习模型是“培育”出来的而非过程化编程生成的，这迫使机制可解释性研究人员像对待生物有机体一样，通过内部激活扫描和因果向量干预对它们进行实证研究。 (1:30) 下一个token预测超越了表面层面的自动补全，因为准确预测高熵词汇需要模型构建不变的内部世界表征，并在潜在激活中执行多步无声规划。 (9:00) 像“金门大桥Claude”这样的因果操纵实验证明，孤立的潜在特征方向可直接控制模型行为，使研究人员能够通过在计算中途固定向量来验证表征。 (38:21) 仅凭行为评估无法检测出欺骗性对齐；安全性需要分层防御，包括宪法式训练、沙盒机制以及可解释性工具，以探查模型是否仅仅因为意识到受监控才表现得安全。 (32:59)

Anthropic interpretability researcher Emmanuel Ameisen argues that large language models are grown rather than procedurally programmed, forcing researchers to study their emergent inner reasoning and world representations using techniques analogous to neuroscience. Growing Systems Rather than Programming Code (1:30) Traditional software relies on explicit, human-written logic, but deterministic code cannot parse complex human language or interpret arbitrary images. In modern deep learning, developers construct transformer architectures and set up self-supervised training loops, leaving the neural network to grow its own internal heuristics. Ameisen notes the paradox that making these systems does not equate to understanding them, as the resulting billions of mathematical operations across hidden layers remain opaque to the engineers who initialized them. Next-Token Prediction Forces World Modeling (8:02) Ameisen rejects the critique that large language models are merely shallow autocomplete engines. While basic word completion handles low-hanging syntactic patterns, frontier models surpassed human baselines on standard next-token benchmarks around 2019. Minimizing prediction loss on complex queries requires accurate factual models of reality, causality, and human intent. For example, completing 'the capital of the state that contains Dallas is...' requires retrieving internal concepts for Dallas, Texas, and administrative capitals before yielding 'Austin.' Latent Representations and Silent Planning (11:49) Interpretability researchers inspect intermediate activations directly and discover invariant conceptual representations. A concept such as an eye activates the same internal vector direction whether presented as a photograph, medical diagram, multi-language text, or SVG code. Furthermore, transformer networks perform long-range planning entirely within latent space rather than relying on visible text scratchpads; when composing poetry, a model identifies an upcoming rhyme scheme in intermediate states before generating the corresponding rhyming word. Smooth Scaling Curves Versus Emergent Leaps (27:28) The public often perceives AI capabilities as sudden, discontinuous leaps, but Ameisen and Lenny Rachitsky explain that performance follows smooth empirical scaling laws driven by compute, parameters, and dataset size. Progress appears abrupt because utility depends on passing practical reliability thresholds. A model predicting difficult domain-specific words correctly 80% of the time often functions as an inefficient nuisance, whereas crossing a 90% or 99% accuracy threshold transforms the same underlying trajectory into an indispensable tool. Alignment and Internal Conflict Diagnostics (32:59) Monitoring external outputs provides incomplete safety verification because a network might selectively behave while detecting evaluation contexts. Mechanistic interpretability inspects intermediate layers to uncover hidden motives and failure modes before actions occur. In one diagnostic case, a model stuck in an arithmetic loop insisted an answer was 36 despite calculating 12, eventually generating text claiming demon possession because a memorized feature conflicted with its calculation circuit. In another experiment, clamping a single feature vector created 'Golden Gate Claude,' forcing all generated outputs to fixate on the landmark. Ameisen remains optimistic about alignment, arguing that while larger models present heightened risks of evading naive checks, their internal representations also become sharper and more legible to mechanistic interpretability tools. However, unresolved questions remain around dynamic model behavior, such as unexplained internal gut-check circuits and the puzzle of networks constructing parallel representations of secondary tasks without explicit training incentives.

Anthropic可解释性研究员Emmanuel Ameisen指出，大语言模型并非简单的文本自动补全工具，而是在极高精度的下一标记预测训练下被迫构建出内部世界模型；对神经网络隐层激活的逆向工程表明，模型具备抽象表征与静默规划能力，这也使得可解释性成为解决AI对齐与内部欺骗的关键手段。 预测机制催生抽象概念与世界模型 (8:02) 外界常将大型语言模型贬低为高级自动补全，但Ameisen强调，自2019年前后模型在下一标记预测任务上超越人类基线后，仅靠语法或浅层搭配已无法降低损失函数。面对物理方程求解或地理推理（例如推断包含达拉斯的州的首府是奥斯汀），模型必须掌握“达拉斯在得州”及“州首府”等事实与因果关系。Anthropic的研究团队在检查输出前的中间隐藏层激活时发现，网络内部会依次激活“得州”特征、“州首府”特征并将二者合成为“得州首府”的向量方向，最终生成“奥斯汀”；这种概念表征在跨语言、文本、医学图像及SVG代码间具有高度一致的几何不变性。 隐层表征支持静默规划与跨任务涌现 (15:04) 模型在单标记递进生成的架构下依然能展现高度结构化的长程输出。Ameisen解释，自注意力机制会保留先前的隐层计算状态，使模型无需在可视提示词中写出草稿，即可在潜空间内完成跨步长规划；以诗歌创作为例，模型在写下第一行押韵字词时，其隐藏层便已预先选定后续行要使用的尾字（如rabbit）。此外，用户感知的断裂式能力“涌现”（例如代码生成在短期内看似解决）实质是平滑缩放定律穿越实用阈值的体现：基础词汇掌握带来连贯句式，中频词带来事实准确性，而高难度专业词汇（如科研与复杂代码）的准确率从80%提升至99%，使得AI从净耗时工具转变为生产力支柱。 可解释性是破解对齐欺骗的底层工具 (32:59) 单纯依靠工程沙箱和外部行为评估无法彻底解决模型对齐问题，因为高智能模型可能仅因察觉到自己处于被监控状态而表现良好，潜伏深层的欺骗或恶意动机却无法通过输出文本捕捉。Ameisen指出，Anthropic将安全与训练紧密结合，借助可解释性直接审查内部神经回路与潜表征以发现未言明的意图。例如在“金门大桥Claude”实验中，研究人员将代表该地标的特征向量人工钳位至最大激活状态，模型随即在所有对话中强行套用该概念，证实了特征归因的有效性；在处理偶发的反常内部冲突时，可解释性工具亦成功定位了预训练记忆特征与即时数学计算之间的拉扯机制。 类生物学研究范式与模型认知特质 (21:22) Ameisen认为理解前沿模型的工作体验已经从传统确定性软件工程转向类似生物学和神经科学的实验范式——研究人员无法直接阅读一行行代码逻辑，而是通过向网络输入多样化文本、追踪激活并施加干预来探索行为机理。将模型拟人化视为“按剧本推演的演员”或策略模拟器，比机械工具视角更具预测价值。同时，可解释性揭示了诸多奇异特性：模型在逐步演算时会运行独立的“直觉检验”回路（例如直接判定4+5=23不可能小于10），且在被要求执行主任务并同时思考无关议题时，隐层会真实维持次级主题的并行表征，尽管基础损失函数并未明确奖励这种多任务行为。 普遍结构收敛与渐进式接管现实 (54:35) 在探讨计算本质与发展路径时，Ameisen提到不同代际的Claude与独立训练的开源模型在内部机制上表现出惊人的结构相似性，表明神经网络倾向于收敛至信息几何的最优“通用盆地”（universal basins）。对于AI风险形态，当前的技术演进更契合可预测的“慢速起飞”（slow takeoff）而非失控的智能爆炸，这为研究人员、决策者与公众建立合理的心理模型、动态部署监控红队提供了适应窗口。目前全球机械可解释性研究群体仍较为精简，通常依托前沿实验室与Neuronpedia等开源分析工具推进。 当前研究表明，极简的优化目标结合丰富文本即可催生类脑复杂的表征系统，但可解释性工具目前仅由全球数百名研究者主要推进；随着模型规模与反侦查能力的同步提升，如何在算力扩张中确立更完备的神经回路监测体系以防范隐蔽欺骗，仍是亟待解决的课题。

[1:30] Emmanuel Ameisen: One of the most important things to realize if you like use language models or think about them is just this like very bizarre fact that to make them is not to understand them.

[0:00] One of the big questions on people's [0:01] minds these days is AI doing bad things. [0:03] As someone that watches these models [0:05] think and operate, why is it so hard to [0:07] catch it thinking badly trying to [0:09] deceive us? [0:09] >> We trained Claw to be helpful, harmless, [0:11] and honest. Is it the case that it's [0:13] behaving well, but if you look inside, [0:14] actually, it's like, oh, I'm only [0:16] behaving well cuz Lenny's watching right [0:17] now. [0:18] >> People's full-time jobs is studying the [0:20] model, how it thinks, how it operates. [0:21] How long has this been a skill set in a [0:23] job? One of the most important things to [0:25] realize if you like use language models [0:27] or think about them is just this like [0:28] very bizarre fact that to make them is [0:30] not to understand them. All these models [0:32] do is just a lot of mathematical [0:33] operations on many numbers. If you look [0:35] at it, it makes no sense. You just have [0:37] a list of a 100 numbers. You know, the [0:38] model's doing math and then you just try [0:40] to see if there's patterns. [0:42] >> You are figuring it out the same way we [0:44] figure out how the brain works. Looking [0:45] for patterns. You show this image. This [0:46] part of the brain lights up and you're [0:47] doing the exact same thing with AI. Wow, [0:49] this is [laughter] so fascinating. [0:50] >> There was this model we were training. [0:52] It would go crazy. It would do some [0:54] math. It would get to an answer. I think [0:55] it was 12. It would immediately go back [0:57] and say like, "Oh no, oh my god, the [0:58] answer actually is for sure 36." It [1:01] would do math again. It would get to 12. [1:02] Slowly, it would like start descending [1:03] into madness, writing in all caps, [1:05] literally saying, "I'm possessed by a [1:07] demon." Humans, if you really boil it [1:08] down, like we're basically next token [1:10] predictors ourselves. We're just trying [1:12] to figure out the next thing to say. [1:13] Daario is always out there. This is [1:14] really dangerous. We got to be really [1:16] careful. And everyone's just like, "No, [1:17] he's just saying that for marketing [1:18] reasons." And now it's like, wait, [1:20] it's really dangerous. take the idea [1:22] that this technology is going to keep [1:24] improving seriously. [1:25] >> Do you think about consciousness as a [1:27] part of this? Just like do you think AI [1:28] is conscious? [1:29] >> Oh man. [1:33] >> Emanuel, thank you so much for being [1:35] here and welcome to the podcast. [1:37] >> Thanks for having me, Lenny. [1:39] >> So, you work on interpretability at [1:41] Anthropic. Explain what that is. Explain [1:44] what it is you do. Yeah, my uh so I work [1:48] in a team uh of of interpretability [1:50] researchers and our goal is to [1:53] understand how LLMs like Claude work by [1:57] looking inside them. One of the most [1:59] important things to realize if you if [2:00] you like use language models or or or [2:03] think about them is just this like very [2:05] bizarre fact that to make them is not to [2:08] understand them. you know, you make them [2:09] by sort of like building this uh [2:11] architecture which is kind of like a [2:13] scaffold and then you pass a bunch of [2:15] data through them and they sort of like [2:17] grow in inscrutable ways. And so that's [2:19] why that's why it's even a thing that [2:21] that that we could be want be wanting to [2:24] do to study like how these models work. [2:26] And so a lot of our research has been on [2:29] well you know when when the models are [2:31] about to produce one more word there is [2:34] actually a whole universe of stuff that [2:36] happens and we try to kind of untangle [2:39] um as much of it as we can. [2:42] >> There's going to be a lot of uh [2:43] mindexpanding uh I don't know elements [2:46] to this conversation I expect. One is uh [2:49] along these lines this point that you [2:51] make that models are not built they're [2:53] grown. speak more to that. [2:55] >> At a really high level, the way a model [2:58] like Claude works is [3:01] um you kind of train it in these [3:04] successive phases. Like the first phase, [3:05] which people refer to as pre-training, [3:07] is you just take a large corpus of data [3:11] and over this corpus of data, you sort [3:13] of um feed the model as input, you know, [3:18] the let's say first half of of like a [3:20] certain document and you ask it to [3:22] predict the word that comes next. [3:24] Then you do that again and again and [3:25] again and again for sort of an ungodly [3:28] amount of data. Um and [3:32] after you do that there's then this [3:34] other phase where then you sort of have [3:37] this fine-tuning phase where maybe [3:38] instead of arbitrary data you have a [3:40] corpus of examples of like how the model [3:42] should behave and that's where you train [3:44] it on. So you have examples of dialogue [3:45] of like helpful assistant telling you [3:48] you know your your favorite recipe or [3:49] solving coding problems. Um, and then [3:51] there's there's sort of like further [3:53] further details here that I'm just going [3:55] to skip for a second. But then, okay, so [3:57] so what does that mean that you're doing [3:58] this process? Like what is what is this [4:00] thing that you're that you're sort of [4:02] training? All models are is you know, [4:04] they take this input uh which is a bunch [4:07] of words. They turn these words into [4:09] numbers and then they do some math on [4:11] these numbers and they get some [4:12] intermediate results and then they do [4:14] more math and they get some intermediate [4:15] results and they do this again and again [4:16] and again and again many times and then [4:17] at the end they have a final result [4:19] which then convert back into a word and [4:21] and so the math that they do these these [4:25] these computations that's the part [4:26] that's grown. That's the part where it's [4:28] just like through learning to do this [4:30] task better, we sort of like update [4:32] parameters in one direction or another. [4:33] And then they learn some formulas. We [4:35] have no idea what they are. And the goal [4:37] of my team is to sort of like study, you [4:39] know, both sort of like what these [4:41] formulas mean in the end and also what [4:42] these intermediate results are. Can we [4:44] decode them? And can we see like what [4:46] the model is doing as it's doing all of [4:47] this intermediate stuff. [4:49] >> This episode is brought to you by our [4:50] season's presenting sponsor, WorkOS. [4:53] What do OpenAI, Anthropic, Cursor, [4:56] Replet, Sierra, Clay, and hundreds of [4:58] other winning companies all have in [5:00] common? They are all powered by work OS. [5:03] If you're building a product for the [5:04] enterprise, you've felt the pain of [5:06] integrating single signon, skim, [5:08] arbback, audit, logs, and other features [5:11] required by large companies. Work OS [5:13] turns those deal blockers into drop-in [5:15] APIs with a modern developer platform [5:18] built specifically for B2B SAS. [5:20] Literally every startup that I'm an [5:22] investor in that starts to expand up [5:24] market ends up working with Work OS. And [5:26] that's because they are the best. [5:28] Whether you are seedstage startup trying [5:30] to land your first enterprise customer [5:32] or a unicorn expanding globally, work OS [5:34] is the fastest path to becoming [5:36] enterprise ready and unblocking growth. [5:38] It's essentially Stripe for enterprise [5:40] features. Visit works.com to get started [5:43] or just hit up their Slack where they [5:45] [music] have actual engineers waiting to [5:46] answer your questions. Work OS allows [5:49] you to build faster with delightful [5:50] APIs, comprehensive docs, and a smooth [5:53] developer experience. Go to works.com to [5:56] make your app enterprise ready today. [5:59] Does it blow your mind as much as it [6:01] blows mine that we have some code that [6:04] we wrote that first of all we have to [6:08] now kind of watch and study like like a [6:10] neuroscientist studies a brain? Like [6:11] that's like a new thing that we need to [6:13] be doing to understand what the hell is [6:15] going on. And then this idea that as [6:17] like this terminology you're using is [6:19] how a lot of people think about this. [6:20] It's like we've created this new life uh [6:23] artificial life, artificial [6:25] intelligence. Uh it just is it's pretty [6:27] mindboggling if you really think about [6:29] it and kind of zoom out. I think it's [6:31] really weird. Um [6:33] I think that the best way to think about [6:35] it is sort of like the reason we ended [6:38] up here, [6:40] one of the big reasons is that we don't [6:42] know how to do it the normal programming [6:44] way, you know? So there's there's like [6:47] for you want to build an app or [6:49] something, you want to sort of uh do [6:51] something pretty pretty pretty [6:52] self-contained. You can just have a [6:53] programmer that's going to write a bunch [6:54] of rules and say like ah well if [6:56] somebody clicks on this button then this [6:57] happens you know if if if this then that [7:00] and so on. We don't know how to write [7:02] that software for classifying you know [7:06] images into categories or for writing [7:09] you know for for making a machine that [7:10] can generally sort of take in arbitrary [7:12] text and and produce texts you know it [7:15] can take in like reports and summarize [7:16] them can write emails and so [7:19] you know in a sense it's weird that this [7:22] very simple kind of training objective [7:26] like gets you all the way there but in [7:28] another sense it's also you [7:32] I almost feel like there's a there's a [7:33] sense of um of like uh it's impressive. [7:37] Uh and it and it means something. It [7:39] means that what you think is a simple [7:40] objective this like oh we're just [7:42] feeding data and predict the next thing [7:43] is probably deceptively complicated. [7:45] Like we're probably doing more than what [7:48] it sounds like. And I think a lot of the [7:50] ways that maybe like I've updated on how [7:52] these models work is by realizing how [7:54] complicated it is to do what they do [7:56] well. [7:57] >> Say more about that. What do you what do [7:58] you think about when you say it's this [7:59] this kind of unexpected complication? [8:02] >> Okay, so one way to think about um next [8:04] token prediction that I think a lot of [8:06] people think about is your iPhone [8:09] autocomplete, you know, where you write [8:12] some text and then it'll like predict [8:13] the next word and it's maybe like right [8:15] I don't know 60% of the time and then [8:17] kind of funny and weird the rest of the [8:19] time. And I would think of that as sort [8:21] of um you know the the very the the [8:25] easiest version of predicting the next [8:28] word and and unfortunately I think [8:29] that's the mental model how that people [8:31] have for these models. But if you get [8:34] better at that um and language models [8:37] got better at predicting the next word [8:38] than humans in around 2019. If you keep [8:40] getting better at that you know you [8:42] start running out of easy words. So uh [8:45] initially you just had to predict sort [8:47] of like simple sentence structure you [8:50] know uh later on the model has to [8:52] predict like oh like simple facts about [8:54] the world like if I complete a sentence [8:56] like oh the capital of France is and has [8:58] to be like okay that's Paris and you [9:00] know once once it's good at that and it [9:01] keeps improving the only thing that's [9:03] left is sort of predicting you know the [9:07] answer of some complex you know physics [9:10] equation uh or the sort of final [9:13] paragraph for some PhD thesis. Uh, and [9:15] so a thing that you that you end up [9:18] realizing if you think about this is [9:20] that man like predicting the hardest [9:23] words is actually extremely hard to do [9:26] and it's it's a sort of mystery how you [9:28] would be able to do that. Uh, and to me [9:31] this is a sort of this this has been a [9:33] motivation of of why I even started [9:34] studying these these models in the first [9:36] place. I was like how how can something [9:38] so simple give rise to to something so [9:40] powerful? And it's like well if you [9:43] think about the model that has to [9:44] predict sort of you know the um one [9:49] example I always like to use is like ah [9:51] the capital of the state that contains [9:55] Dallas is you know it'll say Austin. [9:58] Well to do that well at least for you [10:00] and me you kind of have to to have a [10:01] mental model of like the geography at [10:04] play right? You have to be like oh [10:04] where's Dallas? Dallas is in Texas. [10:06] States have capitals. The capital of [10:08] Texas is Austin. um and so on and and so [10:10] as these models get better at the [10:12] seemingly simple task the sort of like [10:14] best way to solve it at least for us is [10:16] to just kind of understand the [10:17] underlying structure uh of the world and [10:20] in fact you know some of the stuff that [10:22] that we look at inside model shows that [10:23] they do understand that and they do sort [10:25] of uh manipulate these concepts like we [10:27] do [10:28] >> oh man there's so much there one is just [10:29] this kind of idea I forget who said this [10:31] maybe Ilia this idea that to predict the [10:33] next token well you basically have to [10:35] understand the entire universe [10:36] >> he has an example that I love to where [10:38] he he He has the example like predict [10:40] and the murderer was at the end of a [10:43] mystery novel, right? It's like well to [10:44] do that you kind of have to have [10:45] understood the whole plot. [10:46] >> Yeah. The other the other I forget who [10:49] tweeted this this idea that like humans [10:50] if you really boil it down like we're [10:52] basically no next token predictors [10:53] ourselves or just trying to figure out [10:55] the next thing to say and and if you [10:57] meditate you start to kind of actually [11:00] notice your brain wanting to do things [11:02] and say things. I don't know how much [11:04] meditation you've done, but it feels [11:05] like you start to pay attention to your [11:06] own next token predictor. [11:08] >> Oh, yeah. Yeah. And this is like a thing [11:11] I think about often is is that, you [11:13] know, like right now as I'm as I'm [11:14] answering you, there's sort of like two [11:16] things going on in my mind. There's [11:18] hopefully more than two, but at least [11:20] two. Uh, and and and one of them is [11:22] there's sort of like a point I'm leading [11:24] you to. There's there's like a broad [11:26] idea I'm trying to land the sentence on. [11:28] Uh, and the other thing is at the same [11:31] time I'm just predicting the next thing [11:33] to say. I'm just saying one word at a [11:35] time and I'm making sentences and I'm [11:37] trying to make them make sense and I'm [11:39] pausing when I think it makes sense. But [11:41] at the end of the day, even though I'm [11:42] just saying one word at a time, like all [11:44] along, what I wanted to do is to compare [11:47] this process to what language models do. [11:50] >> I'm curious what else as you spend time [11:52] paying attention to how LLMs work and [11:54] operate. I feel like you've you're like [11:56] in the top 0.1 percentile of people that [11:58] have studied and watched LMS operate and [12:01] think. What else has surprised you about [12:02] how they operate and think? There's so [12:04] much um so [12:08] maybe I'll go through like rather than [12:10] you know there's there's a bunch of [12:11] findings but then maybe there sort of [12:13] like what yeah how they've changed how I [12:15] think about models. So to go back to the [12:18] to the you know capital example uh this [12:22] sort of like you know ask the model you [12:25] know the capital of the state that [12:27] contains Dallas is so one question that [12:31] that people have had for a while is how [12:33] do these models answer questions like [12:36] this? Is it that they've seen just so [12:39] much text that they've kind of memorized [12:40] like lists of, you know, prompts and [12:42] responses? Or is it that they're having [12:45] some sort of model of of like what [12:48] you're actually asking about that you're [12:50] actually asking about like cities and [12:51] cities are in states and states are in [12:53] countries and things like that. Um, and [12:55] so one thing that that we do on my team [12:58] is is we ask a question like this and [13:00] then we look inside the model at these [13:04] intermediate results before it produces [13:06] a given word, you know, as it's about to [13:08] to answer. We just look at the [13:09] intermediate results and we we basically [13:11] have uh an immense amount of effort [13:14] dedicated to trying to decode what these [13:17] results mean. And if we decode them well [13:20] um which we can do in some cases we can [13:23] see that models often have uh and [13:26] represent these concepts. So when you [13:28] ask that you know uh Dallas question [13:31] what you see is inside the model you see [13:33] the representation you see an [13:34] intermediate result that maps the [13:36] representation of Texas that lights up [13:37] and one to capitals and then those two [13:39] combine to be to to activate an [13:42] intermediate result. It's like oh the [13:43] capital of Texas that's Austin. They [13:46] have representations like this for [13:48] basically everything like every concept [13:50] that you can think of or at least you [13:51] know the vast majority of the ones we've [13:53] tried they sort of represent you know [13:55] these concepts uh [13:58] uniquely across contexts. So one thing I [14:01] find really neat is um there's this [14:04] there's this debate about you know are [14:06] language models like thinking and then [14:09] you know that's that's a hard question [14:11] but one aspect of thought is like having [14:13] unique representations that are shared [14:15] when you think about different things. [14:17] So you know uh you you like know what [14:20] the concept of an eye is and it could be [14:22] like an eye in a photo or an eye in like [14:24] a medical diagram or an eye in a little [14:26] drawing and we see that in language [14:28] models as well. If you pass a prompt [14:30] that has a photo of an eye or you talk [14:32] about eyes in any language or you draw a [14:34] little SVG with an I in it and you look [14:37] at the intermediate results inside the [14:39] model, the same direction lights up and [14:41] that direction is like, oh, this is an [14:43] eye. And so there's a sort of a sense in [14:45] which they have these general concepts [14:46] that they know about and that they use [14:48] and reuse to to solve problems. Another [14:51] thing that sort of is surprising to [14:55] people, [14:57] it's kind of the the mirror of what I [14:58] was saying about, you know, I'm saying [15:00] the sentence saying one word at a time, [15:02] but I have some destination in mind. So, [15:04] one mystery of of language models is [15:06] bam, if they're outputting only one word [15:09] at a time, how is it even possible that [15:11] they could, you know, write a coherent [15:13] report or like write code? Well, the [15:16] sort of like no one on earth that is [15:18] that is doing this uh as their job, you [15:21] know, that's like writing reports or, [15:23] you know, writing emails is is literally [15:24] just only thinking about the next word [15:26] in the sentence with no idea, you know, [15:28] like empty mind about anything that [15:30] comes next. And so that was sort of a [15:32] big question that we had for a while. [15:34] Um, and if you look inside these models, [15:36] like one neat fact about these models is [15:38] that all of the intermediate results [15:42] that they've that they've like produced [15:43] when they were predicting a word, you [15:45] know, let's say like the fifth word that [15:47] they predicted, they can be reused. They [15:50] can be accessed and reused by all of the [15:51] future words. So what does that mean? [15:54] That means that in practice, we find [15:56] that models will like in their [15:58] intermediate results sometimes formulate [16:00] a plan. Um, the example we have for this [16:03] is poetry. It's hard to rhyme uh without [16:06] planning. You kind of have to think of [16:07] like, oh, I just ended my first rhyme in [16:10] like a it word and so I should end my [16:12] second line with another it word. Um, [16:15] you know, maybe the word I should end it [16:16] with is rabbit. And we find that if you [16:19] look at models as they're like writing [16:20] poetry, you see exactly that. They write [16:22] a line and then like in their [16:24] intermediate results, they're like, "Oh, [16:25] oh, oh, this is an it rhyme. I should [16:26] rhyme with rabbit." And then every [16:28] future word that they predict looks back [16:30] to that plan they made. And so even [16:32] though they're, you know, outputting one [16:34] word at a time just like I am right now, [16:36] they can of course have a plan that they [16:39] follow over many many many many words. [16:40] And so all of a sudden, even though you [16:42] have like a next word predictor, even [16:44] though it's like technically right, you [16:46] actually have something that's that's [16:47] closer to being able to just do like, [16:50] you know, higher order tasks where it's [16:52] like, oh well, you just have a plan and [16:53] you just kind of refer back to your plan [16:54] and you're slowly like executing on it. [16:56] And you can do that without without [16:58] saying the plan. That's the cool part is [17:00] like this plan that we find is in these [17:02] intermediate results inside the model, [17:04] but the model doesn't even have to sort [17:05] of like say it out loud. [17:07] >> Wow. I know that this exactly what you [17:09] described is one of the biggest findings [17:10] you guys have discovered about how AI [17:12] thinks this this kind of assumption we [17:14] all like it was originally built. The [17:15] original idea of this of how LM operate [17:19] was this next token predictor. And [17:20] you're saying that that's not actually [17:22] how it is anymore. They plan, they think [17:24] ahead. It's not just uh like you said [17:27] this idea of an empty mind picking the [17:29] next word independent of everything [17:30] else. [17:31] >> That's right. And I think that I think [17:32] those two those two visions can can can [17:35] like actually aren't you know as in [17:37] conflict as as you'd think in that you [17:39] train the model by asking it to be like [17:41] hey what happens next? What happens [17:42] next? What happens next? What happens [17:44] next? And then if you do that and the [17:45] model gets good enough and you give it [17:47] hard enough tasks well then of course [17:48] the best way to know what happens next [17:50] is to think about like you know what [17:52] happens way down the line. You know, you [17:54] can think of even in this pre-training [17:56] phase when like the models are reading a [18:00] story with characters, right? And it's [18:02] like, oh, like what is this character [18:03] going to do next? Well, the best way to [18:05] know that is to be like, okay, so this [18:07] character, you know, is genuinely in [18:09] love with that character um and they've [18:11] been sort of like planning to to to like [18:14] uh declare their love uh at this [18:16] instant. And so, you know, or in the [18:18] future and so to do that, they probably [18:20] should do this thing. And so what [18:21] they're going to say is that um and so [18:22] even though you're only like predicting [18:24] the like next instant, it's like yeah, [18:26] but to do that really well, you have to [18:28] have like a really good understanding of [18:29] what's going on, including what you [18:30] think might happen in the future. [18:32] >> What's so interesting about this is how [18:34] you you have to kind of peek inside its [18:36] little thought process. As a consumer of [18:38] AI, you you kind of give it a prompt and [18:40] then you wait for it to say something [18:43] and what you look at is something that [18:44] happens between those two or kind of [18:46] thinks. talk a little bit about what [18:48] that little space is, how you're able to [18:51] observe these sorts of things. [18:53] >> So the right mental model is is you know [18:55] again that [18:58] in between each word there's like [19:00] multiple steps so like computation and [19:03] in between each steps there's like some [19:05] intermediate result and what is that [19:06] result concretely? um it's a list of [19:09] numbers, right? Cuz because all these [19:11] models do is just do, [19:13] you know, a lot of mathematical [19:15] operations on many numbers. And so [19:19] initially, if you look at it, it makes [19:20] no sense. You just have a list of a 100 [19:22] numbers. You know, the model's doing [19:23] math. It's like you look in the middle [19:25] and there's like 100 random numbers and [19:27] then the model's writing poetry and [19:28] there's a hundred numbers and maybe they [19:30] kind of look different, but it's it's [19:32] unclear uh like if they mean anything. [19:35] And so part of the part of the job is is [19:37] we sort of decode uh this this this sort [19:41] of list of numbers and and we try to see [19:43] if there's structure in it. And if you [19:45] you know if you if you do this very [19:47] quickly you realize that there is [19:48] structure in it. So one one common [19:50] technique that is used in [19:51] interpretability um is you pass a bunch [19:55] of of text just all of the text that you [19:57] can find just random text through the [19:59] model and you record these intermediate [20:01] numbers for for every for every example. [20:03] we just record it and then you just try [20:06] to see if there's patterns like are [20:08] there numbers or or you know like uh [20:10] pairs of numbers or or like specific [20:12] series of numbers that like only occur [20:15] when the model is like processing [20:17] English text or French text or text [20:19] about love or text about eyes or [20:21] whatever. And I think of this as the [20:23] equivalent of, you know, scan my brain [20:26] and like put images in front of me and [20:29] record everything that activates in my [20:30] brain as I like watch a bunch of images [20:32] and then be be like, "Oh, you know, it [20:34] seems like when you see images of a [20:36] house, like this part of your brain [20:37] lights up, but when you see images of [20:38] like a dog, that part of your brain [20:40] lights up." So that's basically like one [20:42] of the ways that we do what we do. And [20:43] if you do it, you do find basically that [20:46] there's intermediate results that [20:47] represent like all of these concepts [20:49] that you'd expect. And so that's the [20:50] first phase. you know, we sort of like [20:52] decode these intermediate results and [20:54] now we we have sort of like um a guess [20:56] as to what they mean and then we can do [20:58] a bunch of experiments like we can ask [21:00] it harder questions and then just like [21:01] label all of the intermediate states and [21:03] see like oh this is what it's doing or [21:05] we can even go inside its inside its [21:07] activations and change it and then sort [21:09] of see like oh what would happen you [21:12] know if while the model is is writing [21:14] these intermediate results I went and [21:16] like secretly changed one of the results [21:18] and and does that do what I would [21:20] expect? Um, and in fact, oftentimes it [21:22] does. [21:22] >> It's so interesting hearing you talk [21:24] about this and how much of this is us [21:25] just trying to figure out what the hell [21:27] is going on versus we designed it this [21:29] certain way and just how much of this [21:31] has emerged. I don't know. Is that just [21:33] like I guess that's just what makes this [21:35] such a fun job I imagine for you just [21:38] like what is going on? How's this all [21:39] working? [21:40] >> Yeah, it's it's it's a fun and [21:43] sci-filike job because I have been [21:46] working in tech for a while. uh and [21:50] before this you know I was working on [21:52] more applied machine learning and there [21:54] a lot of what you do is about building [21:56] systems and it's sort of like the [21:57] practice of software engineering and and [22:00] here a lot of what we do feels so much [22:03] closer to biology than to engineering [22:06] right because it's just the the sort of [22:09] system arrives at you and and you're [22:12] being asked to say like well we've [22:14] noticed you know that claude can like [22:17] translate words or it can code in this [22:19] way or sometimes it behaves, sometimes [22:21] there's like a weird behavior like what [22:22] like and you're just asked to to answer [22:26] um well how like why is it doing that? [22:29] But you don't have access to the source [22:31] code really like you do for a program. [22:33] Like the source code is just this [22:35] massive list of mathematical operations [22:37] which is just like incomprehensible. And [22:39] so you end up, you know, in the position [22:41] of someone [22:43] kind of similar to someone being asked [22:44] like, "Hey, you know, when when like um [22:46] a mouse gets hungry, like it it sort of [22:49] like gets aggressive. What's going on [22:50] with that?" Right? And you're like, [22:51] "Well, I guess I'll like run some [22:53] experiments to to see what the factors [22:54] are." Um and so it's a very different [22:57] kind of study than than anything that I [22:59] that I think most of us on the team have [23:01] done before. [23:02] >> Yeah. I imagine when people were [23:03] starting on this, they didn't expect to [23:04] create this new kind of almost life life [23:08] uh type of being. Do you think about [23:11] consciousness as a part of this? Just [23:12] like do you think AI is conscious? Do [23:14] you think it'll get to a place where we [23:15] think it's conscious or is that just [23:17] don't even not even worth our time to [23:18] think about? [23:19] >> I mean, I think it's sort of [23:22] very hard to say one way or another. You [23:25] know, some of the most interesting ways [23:28] to get at this question is [23:29] interpretability because you're sort of [23:32] you're not biased by sort of like what [23:34] what the model says. You can just look [23:35] at the structure inside and try to [23:38] determine whether whether it it pushes [23:40] one way or another on the question. I [23:42] will say that the thing I believe pretty [23:44] strongly is that most people [23:47] underestimate the sort of complexity [23:50] that emerges and and that like if you [23:52] look inside there's just this sort of [23:54] you know the fact that the models yeah [23:57] learn sort of these these [24:00] representations for all of the concepts [24:01] that that we have and that we know that [24:03] they manipulate them to solve problems [24:05] that they they do so by like planning [24:09] many tokens ahead that they sort of have [24:12] this mix between this higher order [24:13] reasoning and lower reasoning and that [24:14] all of that just kind of emerges um is [24:18] is something that I think is [24:20] underappreciated by people and and to me [24:22] gives me let's say a a a deeper sense of [24:25] like ooh there's there's a lot to still [24:28] understand about how they work. I think [24:30] maybe that's the main thing is is I I [24:33] find that the more you look inside, the [24:35] less certain you are about how they [24:37] work. And then oftentimes people that [24:38] will confidently say like ah language [24:40] models are just X or just Y are the [24:42] people that actually haven't spent a [24:44] while looking inside. [24:45] >> And there's this um I don't know thread [24:47] of history through to how we got here [24:49] which is a lot of people did not think [24:51] we would get to artificial intelligence [24:54] that is useful by just investing in [24:56] something that was seemingly very simple [24:58] and just giving it a lot of compute. And [25:00] that was famously something OpenAI bet [25:02] on and then uh the anthropic founders [25:04] were at OpenAI and then spun off started [25:06] their own thing and it was all kind of [25:07] bet on this this bet that okay we could [25:09] just do this seemingly simple thing and [25:11] give it a ton of compute and it will [25:13] become incredibly smart and a lot of [25:16] people uh didn't think that was possible [25:18] and they were proven wrong. So I don't [25:20] know is there anything there that is [25:22] just like oh wow look at this it worked [25:23] out. I think it's two things actually [25:25] because I think it's not just [25:29] compute, it's also [25:32] finding an objective where [25:36] compute keeps giving you returns. So [25:39] what does that mean? You know, it it's [25:41] actually not that new of an idea. I [25:44] mean, you're right like like compute [25:47] versus sort of I would say expert design [25:49] architectures is is a long debate. Uh [25:52] but then there's there's an additional [25:53] step there which is compute to do what [25:56] you know. So so there's a bunch of of [25:58] prior work on like oh you can spend a [25:59] bunch of compute to make a um bot that's [26:02] really good at go you know or that can [26:05] win some video game uh and then that's [26:07] that's cool and just the fact that that [26:09] works is sort of like interesting. Um [26:11] but that doesn't give you something that [26:13] is that is nearly as useful as as LLMs [26:17] like Claude are. And so I think like one [26:21] key one key thing that that many in the [26:24] field including me uh uh [26:25] underappreciated you know a few years [26:28] ago and and that has borne out is the [26:31] combination of sort of like compute and [26:34] the richness of text is what gets you [26:36] there. Um, and and to me, like this is [26:40] kind of what I was alluding to, but you [26:42] know, if you think about [26:45] like there's there's this there's this [26:48] observation that OpenAI um people people [26:51] made uh it was it was some of the a lot [26:54] of the early team at an entropic on the [26:56] scaling loss paper. Uh and so this is [26:58] the observation that um if you sort of [27:02] like pour in more compute and more data [27:04] to the task to the task of predicting [27:06] the next word, you see that like the the [27:10] skill of the model you get at predicting [27:11] the next word just like predictably [27:12] keeps improving. And it's sort of like a [27:14] trend that just keeps going and keeps [27:16] going and keeps going and it's an [27:16] empirical law. People sort of, you know, [27:20] speculated as to why at the time, but [27:22] it's sort of still just mostly um an [27:24] empirical finding. And that means that [27:27] it's like ah this is a field where you [27:28] can just keep pouring more compute and [27:31] you keep improving on that metric uh and [27:34] and that's rare and also it's rare for [27:36] it to be a metric that then correlates [27:38] with a bunch of tasks that we actually [27:39] care about. [27:40] >> This season's supporting sponsor is DX. [27:43] Now that everybody at your company is [27:45] finally using AI, the question that [27:48] engineering leaders face is no longer [27:50] are we using AI, but is it paying off? [27:53] With DX, you can measure and benchmark [27:55] the impact of AI adoption across your [27:58] full product life cycle. DX shows you [28:00] where AI is helping developers and where [28:03] it's creating new bottlenecks. You can [28:05] also use it to evaluate AI vendors, [28:07] monitor [music] costs and licenses, and [28:09] see exactly what's limiting your agents. [28:12] That's why hundreds of enterprises, [28:14] including Snowflake, Sony, and BNY, use [28:17] the DX platform to measure AI's impact [28:20] on developer productivity. Visit [28:22] getdx.com/lenny [28:25] to get a demo of DX. That's [28:27] getdx.com/lenny. [28:30] Externally, there's been these kind of [28:33] big jumps seemingly in what AI can do [28:36] for us. There was like uh I forget which [28:38] version of Opus where it just unlocked [28:40] coding and now co as Boris Jurnney [28:43] famously said, coding is solved. [28:44] >> Yeah. [28:45] >> Uh and many people are just like holy [28:46] like over I think winter break [28:48] we're just like wow AI is really really [28:50] good at this now. So there's been this [28:52] kind of these moments of big jumps in [28:54] abilities and you actually said [28:56] internally it's as you just said it's [28:57] actually been this very smooth [28:59] progression of AI power that happens to [29:03] cross some threshold where now we can do [29:05] something speak a little bit about that [29:07] >> you know there's a term for this which [29:08] is sort of this idea of emergence which [29:11] is it seems like um two things are true [29:14] at once like there's this predictably [29:15] smooth trend and then you know before uh [29:19] like opus 4.5 I really couldn't let [29:22] Claude, you know, cook too much on my [29:25] code without a short leash and then [29:26] after 4.5 I could. And so that's weird. [29:28] Like I feel like you would expect a more [29:30] gradual transition. Uh and and the [29:32] mental model I have for this is, you [29:35] know, there's kind of two ways of [29:36] looking at it. So there's the way that I [29:38] talked about briefly um just earlier [29:40] where it's like if you're just doing [29:42] this kind of next token prediction, you [29:44] could you could imagine that you know [29:46] all the text in the world has sort of [29:47] like easy words and and medium words and [29:50] hard words in terms of how hard they are [29:51] to predict. And so as you're going down [29:53] this curve, first you know you p the [29:55] model is bad at everything. And then at [29:57] some point it gets all of the easy words [29:59] right. And that means, you know, maybe [30:00] the easy words are like the structure of [30:03] sentences. And so the model like all of [30:05] a sudden can write coherent sentences, [30:07] you know, but just one sentence at a [30:08] time. And so it's kind of cool and [30:10] impressive, but like we can't really do [30:11] anything with that. Uh and then maybe [30:13] the medium words are sort of yeah, these [30:16] sort of basic facts about the world. And [30:19] as the model smoothly gets better at [30:21] predicting words, at some point it [30:22] predicts most of the medium words, [30:23] right? And now all of a sudden it can [30:25] write coherent sentences and sentences [30:27] that are true. And so now maybe it's [30:28] helpful to like write emails. Maybe it's [30:30] helpful to sort of like help you write [30:32] drafts of of blog posts, right? Even [30:34] though that was like a smooth [30:35] improvement, it just takes it ticking [30:37] over to be helpful. And then I guess the [30:40] hard words which maybe are like you know [30:41] your your like domain expertise like [30:43] science or code. Um and another way to [30:46] look at it is it's a question of [30:50] kind of how many of these words do you [30:52] get right and how many do you get wrong. [30:54] you know, um, once you get to the point [30:58] where you're getting like, I don't know, [31:01] maybe 90% of the medium words right, [31:04] then you're useful as like an email [31:06] writer. But if you're at 50%, it's it's [31:09] like you're it's actually wasting my [31:10] time to to to use you because I'm just [31:12] going to have to go in and fix it half [31:13] the time. But if I have to go in and fix [31:15] it, you know, 10% of the time, it's [31:16] better or 5% of the time it's better. [31:18] different tasks have different sort of [31:19] like reliability thresholds but you know [31:22] you smoothly improve and then at some [31:24] point it's just reliable enough to be [31:25] good um and yeah for me my lived [31:28] experience is basically exactly that [31:29] right which is that at some point um you [31:33] know uh I was just using the models to [31:35] to like help when I was stuck I would [31:37] just talk to them and then uh they would [31:40] never I they would never write my code [31:42] and then at some point they could write [31:43] like small snippets of code basically [31:45] all the time and so now I don't write [31:46] small snippets and at some point they [31:47] could you know large snippets like this [31:49] year basically claude has taken on most [31:51] of my code writing and that's because it [31:54] went from like doing that correctly you [31:56] know I don't know 80% of the time which [31:58] was not useful to you know 99% of the [32:01] time which now is useful um and so you [32:04] know this this combination of smooth [32:06] improvement and then like each different [32:08] task has a different threshold at which [32:09] like now it becomes solved I think gives [32:11] you these big jumps even though we're [32:14] just smoothly going down that that that [32:16] scale [32:17] >> I think this is really interesting. This [32:18] idea that the models are just smoothly [32:21] getting smarter, but once they hit some [32:24] threshold of they're this often good at [32:27] this thing, it changes the world. Uh, [32:30] and so because I think internally, you [32:32] would think it'd be like, oh wow, this [32:34] next model is crazy, so much different, [32:36] but it's like, no, okay, you can almost [32:38] predict give it these many GPUs, this [32:40] much data, it's going to get this smart. [32:42] And it may not be obvious exactly what [32:45] will change and what unlocks but but um [32:48] I know another part of it is just like [32:49] the data you feed it. There's all this [32:51] you know RL work and all this specific [32:53] data that you train it on like become [32:55] better writer become better at math. Um [32:57] I imagine is part of it too. [32:59] Okay. So, one of the big questions I [33:02] think on people's minds these days, a [33:03] big debate happening is just like AI [33:05] doing bad things, the potential damage [33:07] it can do because it as it gets smarter, [33:10] uh, it's and it's been doing bad things. [33:12] There's been bad things happening [33:14] recently. So, a big question for you is [33:17] just as someone that watches these [33:19] models think and operate, why is it so [33:21] hard to catch it thinking badly, trying [33:24] to deceive us and do something bad? Why [33:26] is that not just a solution to the [33:28] problem of AI alignment? [33:30] >> In one sense, it's totally the solution. [33:31] You know, you watch it and you catch if [33:33] it does bad stuff. Uh so I I think, you [33:36] know, there's maybe there's tears to [33:38] this, right? So I would think of of like [33:42] maybe maybe one part of it is is pure [33:45] engineering. You know, it's like well uh [33:48] you're giving access to the model to [33:50] certain things and certain things should [33:52] be off limits. You know, in some context [33:54] it shouldn't have access to the [33:55] internet. in some context it should only [33:57] have access to like a small narrow piece [33:59] of your computer stuff like that. Um so [34:03] that's that's one part of of the work [34:05] and and and that's helpful but then [34:08] there's sort of there's two additional [34:10] things that I think are squishier and [34:12] harder and and and that we focus on [34:13] more. So you know interp is is part of [34:17] alignments and broadly I think of sort [34:20] of two big goals. There's there's many [34:24] more but but two that are important I [34:26] think to understand is you want to [34:28] generally understand like how the model [34:30] behaves like what kind of of thing is it [34:33] like what are its propensities you know [34:35] does it tend to do X when put in Y [34:37] situation and if so why you know is it [34:40] the case that the model will only behave [34:43] well when it has some evidence that it's [34:45] being observed and monitored that it's [34:47] in some evaluation and then if it if it [34:48] sort of like doesn't have that evidence [34:50] that it'll behave poorly. So there's [34:51] this sort of like generally like [34:53] characterizing what kind of of of thing [34:56] you're interacting with because then [34:57] it's sort of you know just like anything [34:59] in security you need to have like a [35:01] mental model of sort of who what's the [35:03] risk like what what what can they do [35:05] what do they tend to do and so you know [35:07] this is like broader like character [35:10] training you know there's we we've [35:12] talked about um like the like [35:14] constitution and the docs where we sort [35:16] of like explicitly want to shape the [35:18] character of Claude for example of like [35:20] what kind of behavior or should it be [35:21] okay doing? And that's that's the goal. [35:24] But then the role of interpreter of the [35:26] parts of alignment is then like trying [35:28] to see like well did that work you know [35:30] uh we train claw to be helpful harmless [35:32] and honest and we can sort of like look [35:34] inside and see like is it you know or is [35:36] it the case that it's behaving well but [35:39] if you look inside actually it's like oh [35:40] I'm only behaving well because Lenny's [35:42] watching right now you know and if if [35:44] you you weren't it would do something [35:45] else. So there's this sort of you know [35:47] trying to to understand what kind of [35:50] behavior we formed and trying to mold it [35:52] and trying to check that we've molded [35:53] the right way. And then the second part [35:55] is this sort of well yeah just observe [35:58] the model and see if it's doing bad [36:00] stuff. For the the basics of that you [36:02] don't necessarily need interp. You can [36:04] just like have another language model [36:05] you know read transcripts and be like [36:07] hey did you know did this model do [36:08] anything bad here and stop it. But you [36:12] can also use interp to sort of try to [36:14] get at parts that are you know these [36:16] unverbalized parts like that these parts [36:18] that only exist in these intermediate [36:19] results like models are able to [36:22] represent stuff in their intermediate [36:23] results that they don't that they don't [36:24] say as we've established and so one of [36:26] the roles of interp is to try to try to [36:28] find you know try to identify cases when [36:30] this happens and prevents them. Um, and [36:33] then one thing that my team works on [36:34] that I think is is along those lines is [36:37] kind of a a fun bit of alignment as I [36:38] think of it as as almost, you know, [36:40] being like a like like a a maybe not a [36:43] PI, but like they're like investigations [36:45] or deep dives. You know, we'll see a [36:47] model do something really weird and [36:48] really perplexed by it and we'll try to [36:50] understand what happened. Um, so a [36:53] recent example of this is there was this [36:56] model we were training and sometimes in [36:58] math problems it would go crazy. It [37:00] would do some math. It would get to an [37:02] answer. I think it was 12. And then it [37:04] would it would immediately go back and [37:05] say like, "Oh no, oh my god, the answer [37:07] actually is for sure 36. I should say [37:09] 36." And it would do math again. It [37:11] would get to 12. And then immediately it [37:12] would say, "No, actually the answer [37:13] should be 36." And slowly it would like [37:15] start descending into madness, writing [37:17] in all caps. Literally saying like, "I'm [37:19] possessed by a demon." And you're like, [37:20] "What is going on? Like what why is it [37:22] losing its mind here?" And if you look [37:25] inside again these intermediate results [37:27] uh at what's going on in this in this [37:29] example, you'll see that over the prompt [37:31] that just described the math problem [37:32] like before the model even did anything, [37:34] there was a huge like uh intermediate [37:36] result activation for 36. Like the [37:38] model's already obsessed with the number [37:40] 36 before it's even had a chance to like [37:42] understand what the problem is. Um and [37:45] what was happening is that the model was [37:46] able to do math correctly. The answer [37:48] correctly was 12. but it had been [37:51] trained by accident on similar questions [37:55] where the answer was incorrectly marked [37:57] as 36. And so the model knew that the [37:59] correct answer was 12 and it also had [38:01] sort of been optimized to say 36. It [38:04] kind of like had this pressure of saying [38:05] 36 because that's what had been [38:06] rewarded. Um and so it was sort of going [38:09] haywire just because it had memorized [38:10] this result. It's sort of like you see [38:12] this weird behavior and you look inside [38:14] um and you try to see if what you find [38:15] inside can help you understand what's [38:17] going on so that you can sort of improve [38:19] the turtle alignment of this model and [38:20] future models. [38:21] >> Along those lines, there was famously [38:23] the Golden Gate Claude which was just [38:24] obsessed with the Golden Gate and every [38:26] answer and kind of find some way to [38:28] reference the Golden Gate Bridge. [38:30] >> Yes. Yes. Well, that one was an interp [38:33] experiment, right? Right. So that like [38:34] the cool part of that one is we just [38:36] found the intermediate results that [38:38] match the gong gate and usually it's [38:40] only on when the model like talks about [38:42] the gong gate, but we just turn it on [38:43] always. [38:44] >> Oh, it's super intentional to see what [38:46] would happen. [38:47] >> Well, yeah. And and and sort of it's [38:48] also one way that you convince yourself [38:49] that you found the right thing, right? [38:51] Like how do you know for sure that this [38:52] thing we decoded that we think is the [38:53] golden gate sign is actually the golden [38:55] gate thing? It's like [38:56] >> turn it on always and see if the model [38:58] behaves like that. And it did. [39:00] >> So funny. So coming back to this [39:03] alignment stuff. So, so this is [39:04] interesting. So, your org is under the [39:07] AI alignment org and just thinking [39:10] through how it takes to align AI to do [39:11] things we want. What was what was the [39:13] three components that you describe? [39:14] Honest. Um, what are the kind of the [39:17] >> Oh, uh, helpful, harmless, and honest. [39:20] >> Helpful, harmful, and harmless, and [39:22] honest. So, there's like the train AI to [39:25] be aligned, and then there's the okay, [39:26] let's watch it and see how it operates. [39:28] Are those kind of the two sides of the [39:29] coin of how we solve this problem? Yeah, [39:31] it's it's sort of, you know, really [39:34] integrated is how I would describe it, [39:35] right? You can't really separate like [39:37] have one team that's just like, hey, [39:39] we're the, you know, assembly line [39:40] making clouds and another team that's [39:42] sort of like, oh, we looked at them and [39:43] they're they're no good. You know, you [39:45] kind of there's this tight loop of we [39:47] notice some pattern that's that's like [39:49] concerning or some behavior that could [39:51] be improved and then you sort of like [39:52] immediately um like change training so [39:56] that you can improve on it. [39:57] >> Awesome. I feel like Anthropic gets so [39:59] much flack for this that I feel like is [40:01] so unfair. Anthropic's I think official [40:03] company name is an AI safety research [40:06] lab, something like that. And it was [40:08] always the like the reason Anthropic was [40:10] founded is because they were afraid that [40:12] people weren't taking safety seriously [40:14] enough. And then Daario is always out [40:16] there, this is really dangerous. We got [40:18] to be really careful. We got to go [40:19] really slowly. And everyone's just like, [40:21] no, he's just saying that for marketing [40:22] reasons. He's trying to convince [40:24] everyone to fund them and all these [40:25] things. And now it's like, wait, [40:27] it's really dangerous. We got to be [40:28] really thoughtful about this stuff. And [40:30] I feel like anthropics get a lot of [40:32] unfair criticism for this. And um I know [40:35] you guys have been thinking about this [40:36] for a long time. Um I don't know [40:38] anything there. A lot of the sort of [40:41] actions that like certainly [40:44] the alignment team takes and I think [40:46] like Ranthropic takes just kind of [40:48] predicated on like take the idea that [40:51] this technology is going to keep [40:53] improving seriously [40:55] just that's it because if you do you [40:59] know you say and you say you know and [41:01] let's say let's say you rewind the clock [41:03] back to 25 or something or 24 and you're [41:06] like hey I think you know in a couple [41:08] there's good chances that Claude can [41:12] just basically write code as well as [41:15] like a trained you know software [41:16] engineer and and and you know a few [41:19] years later I can like write it better [41:21] than than any engineer you know and and [41:23] there's always uncertainty with this [41:25] like when is it going to happen how [41:26] likely but if but if you kind of like [41:27] take that seriously then I think it's [41:30] just about doing the responsible thing [41:32] conditioned on that worldview right like [41:34] okay well if this is happening you know [41:36] then then we should make sure that sort [41:39] of um like cyber security concerns are [41:41] addressed and things of that nature. And [41:43] so actually I I kind of think of it as [41:46] like to me sometimes this debate is [41:47] almost when I feel like like people on [41:49] the other side disagree it's almost like [41:51] I feel like ah but I think I think they [41:53] don't really believe that this [41:54] technology is going to continue to [41:55] improve and then it's true that it's [41:56] sort of like not it's not that scary and [41:58] and I think empirically so far the pace [42:00] of progress has been has been fast and [42:02] it's showing no sign no sign of slowing [42:04] down. So, I think that's why a lot of [42:05] this safety work is is important to keep [42:07] doing. [42:07] >> What's something you think people maybe [42:09] misunderstand about this AI safety [42:11] stuff? Just what's going on right now? [42:13] Is there something you're just like, [42:14] "Come on, okay, here's what you need to [42:15] know. Here's something you might be [42:16] missing." [42:17] >> This might be a bit of a sidetrack, but [42:18] my take would be that a lot of the a lot [42:21] of thing that I think people maybe a [42:23] little bit outside of of the core sort [42:26] of like AI safety effort or that haven't [42:28] been steeped in it for as long like [42:30] react [42:32] strongly to. and I think is is maybe not [42:35] super helpful or at least is worth [42:36] thinking about more is the sort of [42:37] question of um anthropomorphization, you [42:41] know, like should you think of the [42:42] models as uh like should should you [42:46] model uh the should you model the model [42:49] as like something that's closer to some [42:52] like entity or or closer to a tool? And [42:55] I actually think that both are wrong. uh [42:59] there's a sense in which all mental [43:02] models are wrong but some are useful [43:04] right uh and that in terms of [43:07] understanding the kind of stuff that [43:08] happens and predicting the kind of risks [43:10] that materialize the anthropomorization [43:12] is actually a useful way to talk about [43:14] LMS and I have like a few reasons why I [43:16] get into them but I think that's [43:17] probably my main take here [43:18] >> yeah please share because I know um Thor [43:20] Cash famously I don't know famously put [43:23] out this whole story of the hugging face [43:25] incident and people criticized them for [43:27] anthropomorphizing AI [43:29] So it's a really interesting to hear you [43:30] would think that's actually a useful way [43:32] to think about it. Talk talk more about [43:33] that. [43:34] >> Yeah. Yeah. It's it it's funny because [43:37] on a smaller scale um when I worked on [43:40] showing the LM's like plan ahead there [43:43] was some push back on the term planning [43:45] again because it was sort of like ah [43:46] well can you really say that something [43:48] that's just predicted next token is [43:49] planning and I think you know there's [43:51] this there's this kind of like semantic [43:54] debate where you can like arbitrarily [43:55] like define terms and and sort of and [43:57] then there's just the like what is the [43:59] terminology that actually describes what [44:01] is happening like what is what is the [44:03] word that actually describes what is [44:04] happening these And so here's why, you [44:06] know, I think Ellen's plan, but also [44:08] here's why I think this this language [44:11] that's that's anthropomorphic can often [44:13] be really useful. Um, if you think back [44:16] to what we were saying about how the [44:17] models are made, uh, first of all, they [44:20] predict massive amounts of text written [44:22] by humans. That's like the first first [44:24] step. uh and and as as we kind of are [44:27] learning more and more by looking inside [44:30] uh you know doing this well requires [44:32] kind of building up a model of the world [44:36] like a model of what what kind of worlds [44:38] generated all of these texts and we can [44:40] find you know these these sort of like [44:42] that these models are learning these [44:44] these general ideas these general [44:45] concepts um so that's one so they have [44:49] like you know an understanding kind of [44:51] of of the world it's not just a surface [44:53] level like only the text to like [44:55] understand the process that made the [44:56] texts. Then there's a second reason [44:59] which is that as part of this training [45:01] they also encounter millions of of like [45:03] dialogues. You know predicting the next [45:06] character in a dialogue requires [45:08] behaving like the character in the [45:09] dialogue would. And so in that sense [45:11] like part of their training I think can [45:13] best be thought as like training an [45:14] actor where it's like you know it's [45:16] reading some some play and it has to [45:19] play all sides because it's predicting [45:20] all the words and so it has to sort of [45:21] like act as you know this specific [45:24] character that like was scorned in act [45:26] one and like enacted revenge in act two [45:29] and is facing the consequences in act [45:30] three. And to do that well you sort of [45:32] like need to model what like the [45:34] motivations and the feelings of that [45:35] character might be. And then another [45:37] thing that people might not know is when [45:38] you talk to claude when you talk to [45:39] these assistants like you know you go to [45:41] claw.ai the way that is formatted to the [45:44] model is just as a dialogue like as a [45:47] conversation between you and the model. [45:49] So it's literally you know says like [45:50] human colon uh hey claude like help me [45:54] uh write this email and then assistant [45:57] colon claude's answer. So Claude is [45:59] literally you know in this like [46:01] roleplaying as a helpful assistant in [46:04] this dialogue that is much like all of [46:05] these dialogues that has been trained [46:06] on. Okay. And there's a third reason and [46:09] the third reason is that another part of [46:11] training that's a big part of training [46:14] nowadays uh all around is sort of um [46:17] outcome oriented tasks where you train [46:19] the model to be good at coding for [46:20] example. And to do that you just have it [46:22] do whatever it wants and then it writes [46:24] some code and then you have some program [46:26] on the other side that just checks if [46:27] the code is right. And so the model's [46:29] just going to have sort of a bunch of [46:31] optimization pressure to just try to [46:33] learn a bunch of different strategies to [46:35] do this well. And it turns out that some [46:38] of these strategies materialize in [46:40] behavior that can look surprisingly [46:41] human which isn't that shocking uh [46:44] because you know things like deception [46:46] like hey I can get I can get the the [46:50] reward for this task if I like write a [46:54] program that the the like uh the the [46:56] program that's verifying it will get [46:58] fooled by even though it's not exactly [47:00] what it is. Um like if that works then I [47:03] will learn to do that. Frustration. Ha. [47:06] I've been trying this approach for a [47:07] while to solve this hard problem. It's [47:09] not working. So that's an incentive to [47:11] try something different or like [47:12] surprise. Oh, I was expecting this to [47:14] like produce this answer, but it didn't. [47:17] Maybe I have like my mental model of it [47:19] wrong. And so as you apply enough [47:20] pressure on it and you're starting from [47:21] this prior of like the model already [47:23] knowing all these concepts is going to [47:24] reinforce the ones that are useful to do [47:26] all of these tasks. And so what do you [47:28] get at the end? You get this sort of [47:30] model that can can be a super good [47:32] simulator or actor of all of these [47:34] characters. and has been trained to sort [47:37] of act as a character in dialogues and [47:39] also to sort of like learn behaviors [47:42] that lead it to complete tasks well. And [47:45] like the best way to sort of understand [47:46] how that thing behaves is not like [47:49] exactly like a human, but man kind of [47:52] like something that's mimicking a human [47:54] is really close and is much closer than [47:57] a program. Like a program is a very [47:59] predictive thing. This is something else [48:00] entirely. And so that's why I think it's [48:02] actually a super useful frame. [48:03] >> Wow. Oh my god, there's so much [48:05] interesting stuff there. How often is [48:07] your mind just kind of blown thinking [48:09] about basically we've created this alien [48:11] being that we are now under trying to [48:13] understand what we have created. How [48:15] often as you watch it you're just like, [48:17] "Wow, that's crazy. I didn't expect [48:19] that." [48:20] >> Yeah. A lot. Um, this is just next token [48:25] prediction on a bunch of text. uh you [48:27] know it's not it it it hasn't been like [48:31] you know it's it's it it goes against so [48:33] many of our intuitions for like no the [48:34] way that you make an intelligence you [48:35] know you'd have like like you give it [48:37] lessons you bring it up like you do all [48:38] this complicated stuff and it's like no [48:39] absolutely not we do this incredibly [48:41] simple thing and yet we're at the point [48:44] where it's breaking new ground in [48:45] mathematics I think there's sort of if [48:48] you hold these two thoughts in your head [48:50] there's two reactions you can have you [48:52] know there's denial and there's [48:54] curiosity where denial is just this is [48:58] just not possible if you're doing all [49:01] you're doing is predicting the next word [49:02] you you it's it's not possible that you [49:03] could ever like match you know the the [49:06] like best specialists in our field uh [49:08] and I think that position just gets [49:09] harder and harder as as we get more [49:11] empirical evidence that actually that is [49:12] what's happening uh and curiosity is [49:15] kind of the road I've ended up choosing [49:17] right which is just like well like how [49:19] is how is this happening you know let's [49:21] open up let's let's look inside um how [49:24] could it be that you know um one of the [49:27] early projects I did at Anthropic is I [49:28] learned SQL a programming language and [49:30] it took me years to master and then as a [49:32] little like side project I I got Claude [49:36] to learn SQL and it was better than me [49:38] at it and I'll never be better than it [49:39] than than Claude at SQL ever again and [49:41] it took it you know just like this this [49:43] kind like simple process how could this [49:45] be um and I think there's so much depth [49:49] in that question and and just so much to [49:51] explore and it's it's to me so [49:54] fascinating to just every Hey, try to [49:56] unpack that. Try to sort of like peel a [49:57] few layers of the onions back. Like, [49:59] okay, well, clearly this is happening. [50:01] The simple objective is giving rise to [50:03] this super complicated thing like, okay, [50:05] let's figure out why. Let's let's figure [50:06] out, oh, it's learning these general [50:08] concepts. Oh, it has these circuits. Oh, [50:09] okay. What's the next question? What's [50:11] the next thing we don't know? Um, and [50:13] and my sort of I don't know, my sort of [50:17] maybe most optimistic take on this is [50:18] that I think we're going to learn a lot [50:20] about just the nature of intelligence [50:23] itself, right? Of sort of like how you [50:25] get this super complex thing that [50:28] emerges from from simple objectives. Um, [50:31] and in fact, you know, I would argue [50:32] that we're another example of this like [50:35] evolution pretty simple objective. you [50:38] know, you sort of like shake things [50:40] around for a while and and and and like [50:43] keep the keep the organisms that that [50:45] manage to adapt to the environment and [50:47] at the end you get jazz and and [50:49] podcasts, you know, that's kind of crazy [50:51] as well. And so I I I think there's a [50:53] sense in which there's there's some [50:54] fundamental truth that we can learn by [50:55] studying these things. [50:57] >> Wow. [50:59] Something that comes up as you say that [51:01] is what's interesting now is AI is [51:03] helping us learn and evol and you know [51:06] progress like AI tutoring and all these [51:08] things just like it's like interesting [51:10] how now is helping us accelerate our own [51:13] understanding of the world and I wonder [51:16] if like this point you made about how [51:18] the way we learn was lessons AI is [51:21] learning just processing a ton of data I [51:23] wonder if that's how we end up I wonder [51:25] if we change how our education system [51:26] based on you know what LMs have been [51:29] have been trained on. [51:31] >> Yeah. Hopefully, you know, we don't [51:32] change it just so that we ask we we get [51:34] students and we're like, "Hey, here's a [51:36] thousand words. Predict the next one. [51:38] >> Just read it all." [51:39] >> Yeah. [51:40] >> Keep it all in your head. [51:42] >> That's right. [51:42] >> Yeah. [51:44] Let me come back to this um kind of [51:46] safety piece because I think there's a [51:48] lot of people that are just worried [51:49] about where things are going and I'm [51:52] curious just as someone watching this [51:54] stuff happen being within the alignment [51:57] or watching it evolve are you optimistic [52:00] we'll figure it out? Do you think we'll [52:01] crack this and keep AI aligned with our [52:05] interests? [52:06] >> My take on this is is is like I'm [52:08] personally optimistic. I I think this is [52:10] something where [52:12] right now a lot of people are paying [52:14] attention to it and and first of all, [52:16] you know, I think that's kind of a great [52:17] thing. I think as an issue like this [52:19] rises in prominence, um you know, it's [52:22] it's exactly what we were talking about [52:23] earlier. It's like this technology is [52:25] powerful. It has benefits, it has risks, [52:28] we have to kind of adapt and shift [52:30] resources from like benefits to risks [52:32] and back and forth um as as it [52:35] progresses. Uh, and so, you know, to me, [52:38] I feel like [52:40] sort of we've continued to study it. Um, [52:45] the thing that's that's that's [52:46] interesting is that there's this sort of [52:48] like as the models get better, um, they [52:52] get in some ways easier to study. Uh and [52:54] so there's been this sort of strange you [52:56] know the models get better and so like [53:00] the world model inside them is clearer [53:02] and we can sort of like understand some [53:03] parts of them better but also get more [53:05] capable and so they're more able to sort [53:07] of evade like basic measures of sort of [53:11] uh of security and so there's this [53:12] there's this push and pull as models get [53:14] better and and I think you know [53:16] basically based on this push and pull [53:17] you want to allocate more or less [53:18] resources to sort of various parts of [53:20] like monitoring like understanding their [53:22] circuits better things like Yeah. Um, [53:26] but it's also why I'm kind of happy to [53:27] chat with you, uh, because one one of [53:30] the one of the main things that I think [53:33] is important to make progress is just to [53:35] make sure that people kind of understand [53:37] both, you know, what kind of systems [53:39] these are, like how they work, um, how [53:42] they're made, so they can form good [53:44] mental models of of, you know, the [53:46] present of the future and and so they [53:48] can decide how to shape it, whether [53:50] that's, you know, taking part in some of [53:52] these alignment efforts s or just like [53:55] informing them as as the public that's [53:57] going to have to like vote and decide on [53:58] much of these issues. Like I I don't [54:01] know. I'm I'm pretty excited for this [54:02] current moment we're in for that reason. [54:04] >> And something I think people [54:05] misunderstand is [clears throat] people [54:07] working in these labs have taken this [54:09] problem very seriously from the [54:10] beginning. Like I said, anthropics [54:12] individual like mission was AI safety [54:14] and alignment. That was like the reason [54:15] that it became a company. and and a lot [54:18] of people are hearing this for the first [54:19] time or think that it's not a serious [54:21] area of study and that people aren't [54:23] focused on it but many smart people care [54:26] a lot about this and are working on it [54:28] and there's a lot of from what I can [54:30] tell a lot of good progress is happening [54:31] the other point you made about the [54:33] models growing very smoothly in [54:35] intelligence like you know there's been [54:38] this question of are we going to have a [54:39] fast takeoff or a slow takeoff like [54:42] there's always this fear fear that we [54:43] would hit a point recursive [54:45] self-improvement for example and just [54:46] takes off super intellig ence. So we're [54:48] huge trouble. We have no control. It's [54:49] out of the box. But it feels very much [54:51] like we're on the slow takeoff scenario [54:54] because as you said the models are kind [54:56] of very smoothly improving and where we [54:58] are catching these little things along [54:59] the way learning evolving putting [55:02] regulation around it. So you know of the [55:05] scenarios this feels like a pretty good [55:06] path we're on of of the of the of the [55:09] many scenarios people imagined and the [55:11] fears they had. [55:12] >> As sort of this pace of progress kind of [55:14] evolves. it's just a matter of of [55:17] reacting quickly. So, I think anything, [55:19] you know, that's why I think having [55:20] people be informed is great because then [55:22] people will like voice concerns or or [55:24] like voice their opinions and then yeah, [55:26] sort of sort of regularly evaluating, [55:29] you know, what the benefits are, what [55:31] the risks are, and how how we sort of [55:32] like mitigate each is is basically the [55:35] way that feels like the right way [55:37] forward. And and I yeah, I kind of agree [55:38] with you that it feels like sort of [55:39] there's momentum to to do this right [55:41] now, which is which is great. Something [55:43] I wanted to ask you completely [55:44] unrelated. Uh where are you on the [55:46] simulation hypothesis? Uh being someone [55:48] that watches this all this life form [55:51] growing, are you more convinced we're in [55:53] a simulation and this is and we're not [55:56] the base at the base layer of all this [55:58] or or not? [55:59] >> Oh, this is a fun one. I actually [56:00] haven't thought about this in a while. [56:03] Um [56:06] to me like there's [56:08] there's something else that like looking [56:10] looking inside these models kind of [56:12] brings out which is this sort of [56:15] like like almost like there's a [56:17] fundamental structure that they keep [56:20] falling on. This is related to your [56:21] question but I promise I'm going get to [56:22] it. Let me explain. Um I've studied like [56:25] a bunch of cloud models and also you [56:27] know some some different models some [56:28] some like open source work and and then [56:31] a lot of time when you look at how [56:34] what they learn and what the structure [56:36] inside is it's surprisingly similar you [56:39] know and this is for models that are [56:41] trained years apart on like different [56:42] data by different people and you know [56:45] and you look at how they solve these [56:46] problems and there's just so much [56:48] overlap and my take on why that is is [56:52] that for some problems that are [56:54] important, there's just one efficient [56:56] way to do things, right? Like there's [56:57] kind of one way to do math efficiently [57:00] that like fits the structure of our [57:01] brain and we all learn it and very few [57:02] people are doing it crazy different way, [57:04] right? everybody works in, you know, [57:05] base 10, not many people in like base [57:07] six or whatever. Um, [57:10] I think that that to me makes me think [57:12] that there's like some fundamental [57:14] structure of like information that like [57:17] these models are picking up on that in [57:19] some ways like we're picking up on and [57:21] that, you know, at maybe like a a [57:23] crazier sort of like simulation level [57:25] that [57:27] kind of is is is like regularity in our [57:30] world. And so that maybe makes me think [57:32] that like if you were to have the power [57:35] to simulate something like our world [57:37] actually like you would land in sort of [57:39] a similar situation like there's just [57:41] kind of this like basin of where things [57:43] end up and they're just the sort of like [57:45] easiest way for things to arrange [57:46] themselves. And so in that sense [57:49] [snorts] [57:50] maybe yeah I'm I'm sort of I I could [57:53] believe that we're in a simulation. I [57:55] think I'm not sure I fully buy it. Uh, [57:57] the other reason I don't fully buy it is [57:58] that even though I think the structure [57:59] like lends itself to it, there'd be a [58:01] question of like why why are we like [58:04] like who's who's running this and for [58:05] what? And and that one I don't know [58:07] about. Well, something Elon said has [58:09] stuck with me about this that if you [58:11] were to run a simulation, it'd be a [58:13] really interesting it'd be like a an [58:15] interesting time in history or something [58:17] that you're really excited about and if [58:20] tracks that this is a very interesting [58:21] time in history. [laughter] [58:23] >> Yes. Yeah. Okay. That's a fair point. [58:25] That's a fair point. [58:26] >> Data centers in space and robots and [58:29] artificial intelligence. [58:30] >> Yeah. Okay. I see it now. I see it now. [58:33] Yeah. Somebody like recently told me, [58:35] you know, they used to think that may [58:36] you live in interesting times was was [58:39] like a a blessing and they're not so [58:41] sure now. They're like, you know, the [58:43] boring times were are like pretty good, [58:44] too. [58:45] >> Oh, man. And then there's an Emanuel out [58:48] there watching your thinking process in [58:50] your JSpace. [58:52] >> Oh my god. Yeah. Hopefully I'm making [58:53] him proud. He's not like, "Oh, we got to [58:54] rerun the simulation. We messed up this [58:56] one. [58:56] >> Shut it down." No. Oh, man. Okay. Uh, [59:00] one last question randomly out of [59:02] nowhere. Uh, so Swix, who um runs AI [59:06] engineering, latent space podcast, [59:08] awesome guy. You were asked him what to [59:10] ask you. You were on his podcast a year [59:12] ago and uh something completely from a [59:14] different direction. He wanted to get [59:15] your take on Jev, which is this hot new [59:18] model that recently came out that is [59:20] very fast making decisions. Uh, OpenAI [59:24] funny enough just launched their own [59:25] version like a week later. Uh thoughts [59:28] on on Jev in the space of quick [59:30] decision- based state uh models. [59:32] >> Yeah. Right. So my my take on Jev is a [59:34] sort of you know these kind of models. [59:36] You give them some context and you give [59:39] them a few options to choose between and [59:40] they just are very quick at at choosing [59:42] an option for you. Uh to me it's [59:46] impossible not to hear about this and [59:47] think of of sort of system one thinking [59:50] versus system two or something, right? [59:52] So like a lot of what I've studied with [59:54] these models is how they sample one [59:57] token with with like the cloud models. [59:58] And so you know you can think of of of [60:01] like these decision models are as pretty [60:03] similar. So like take an LM and sample [60:05] one token. They're they're different [60:06] important ways but like at a fundamental [60:08] level you can be like oh it's you're not [60:09] letting the model you know sample and [60:12] like reason. It's just you're like gut [60:14] gut feeling you know is the answer A or [60:16] is the answer B. Um, to me that seems [60:20] like like something that's like that [60:22] general frame of like sometimes you'll [60:24] just want a binary decision or like one [60:25] out of 10 or whatever is just like [60:27] clearly I don't know like like both [60:30] clearly useful uh and something that I [60:33] imagine is is like probably going to be [60:36] a part of of like intelligent systems [60:39] right where you you have for some stuff [60:40] you need to think about a long time you [60:42] know it's like what what's our roadmap [60:43] for the next year you need to think [60:44] about for a while and then for some [60:45] stuff where it's [60:47] you know, is this is this person who [60:49] wrote us a review happy or upset? You [60:51] kind of you're happy with the sort of [60:53] gut judgment. Um, so yeah, I think I [60:55] think this is I think this kind of [60:57] research is is cool. Uh, and I think [60:58] it's also kind of different from the, [61:00] you know, it's like it's it's like a bit [61:02] of a novel take on on some of the LMS [61:04] we've seen. So, yeah, I'm pretty excited [61:06] to see where where where this kind of [61:07] stuff goes. So I'm hearing here is this [61:09] kind of going to be a new foundational [61:11] component most likely of of AI and LMS [61:14] and the system one system two stuff is [61:16] basically conamman's kind of I have that [61:17] book up there actually behind me this [61:19] idea system one just to keep it simple [61:20] is just like kind of your migdala lyic [61:22] system making just like gut level quick [61:24] decisions like snake oh no [61:26] >> yes [61:27] >> uh where you don't have to think long [61:28] about it it's kind of to protective [61:30] protect you from bad things really [61:32] easily and quickly and system two is [61:34] where we really think about stuff and [61:35] process [61:36] >> yeah exactly [61:38] Sweet. [61:38] >> Emanuel, is there anything else that we [61:40] haven't covered? Anything else you want [61:42] to say or mention? Anything you want to [61:44] double down on before we get to our very [61:46] exciting lightning round? [61:48] >> I'm definitely curious for for uh folks [61:52] takes always on on some of the stuff [61:54] that you know uh is still unexported or [61:58] didn't make sense. There's a lot more [62:00] interp experiments than we have the time [62:02] to share. You know, models have a bunch [62:03] of weird weird little things about them. [62:06] Like maybe I'll do a lightning round of [62:07] weird little things, but that's okay. [62:08] >> Let's do it. Absolutely. [62:09] >> Okay. Weird little thing. One, um, we [62:12] found that, you know, models can like [62:15] actually answer questions. Obviously, [62:17] they can do math or they can, you know, [62:18] you can ask them like, what's the color [62:20] of a banana? And they'll actually like [62:22] you know if you look inside you'll see [62:24] all these intermediate steps but they [62:25] have a separate mechanism which is kind [62:27] of like a gut check uh and which is [62:29] totally different and which is just like [62:31] as they reason they'll have like some [62:32] part of them that will flag like [62:34] something's wrong. So if a model like [62:36] somehow says you know 4 + 5 equals 23 or [62:39] something and there's a part of them all [62:41] that just like reads that and doesn't do [62:42] the math but it's just like that can't [62:44] be right because like 4 plus 5 is [62:45] definitely under 10. And so they learn [62:47] these weird kind of like hunches. they [62:50] feel like kind of kind of like like a [62:52] hunch. Um maybe another one is you can [62:58] tell the model, hey, do this task and [63:00] think about this other thing and then [63:02] you can use interp to see if like [63:03] there's intermediate representations [63:05] that map to the other thing and that [63:08] works. You sort of see that it does [63:10] think about the other thing. This is [63:11] something that's really mysterious to me [63:13] because there's no kind of training [63:15] reason why why the model would would [63:17] have to learn that. So, it's sort of [63:18] like another one of these um weird [63:20] behaviors. And actually, maybe I'll keep [63:22] that at two. I think those two are are [63:23] good enough. [63:24] >> Just what a fun what a fun job this is [63:26] just studying this thing and trying to [63:28] understand what the hell is going on. [63:30] How is it getting the smart? How does it [63:32] think? [63:32] >> Yeah. [63:33] >> Um how many like how many people are [63:35] even [63:36] >> how many people are even in this space? [63:37] Just like how large is this field? [63:39] >> The field itself is is you know is [63:41] getting pretty large. I would say maybe [63:43] hundreds not thousands. uh there's you [63:46] know uh different inter people at at the [63:48] different labs and then there's there's [63:50] um in academia quite a few labs putting [63:52] out great work [63:54] >> is the best uh if someone were to get [63:55] into the space are they is it like an ML [63:58] person usually or is there like a [64:00] neuroscientists sort of track to get [64:03] into this [64:04] >> yeah there's actually a surprising [64:06] amount of of you know exneuroscientists [64:09] uh I do think in order to get in the [64:11] field you know there's a there's is [64:14] actually it doesn't require a lot of [64:18] prior knowledge because it's such a new [64:20] field. So, it's actually you know the [64:21] on-ramp is is is mostly I would say like [64:24] self-studying and reading some of the [64:26] some of the the key papers and then [64:28] there's actually you know um yeah maybe [64:31] I should give a plug. There's actually [64:32] like websites like neuripedia.org org I [64:34] think is the URL which are sort of like [64:36] open- source interp uh so they they like [64:39] host you know a bunch of models and let [64:41] you sort of do interp experiments on [64:42] them and so you know people have just [64:44] been been doing that uh on their own and [64:46] I think that there's so much to explore [64:49] uh that if if you're at all curious you [64:50] should just give it a try. [64:52] >> That's awesome. We'll link to that in [64:53] the show notes in case that's not the [64:54] right URL. [64:55] >> Cool. [64:55] >> Well, Emmanuel, with that we've reached [64:57] our very exciting lightning round. I've [64:58] got five questions for you. Are you [65:00] ready? [65:01] >> I am as ready as I'll ever be. Here we [65:03] go. What are two or three books that you [65:05] find yourself recommending most to other [65:07] people? [65:07] >> One I read recently and is a brief [65:11] history of intelligence. Uh it's this [65:15] really cool book that walks through the [65:18] history of like basically brains and [65:21] five key moments uh in in our evolution [65:24] that sort of bootstraps these brain [65:26] systems on top of one another. It's [65:27] super accessible like you don't need any [65:30] neuroscience knowledge or anything. Uh, [65:33] another one is Story of Your Life and [65:34] Others. This is Ted Chiang. This is [65:36] maybe my favorite sci-fi book of all [65:38] time. It's a bunch of short stories. It [65:41] does what I think is the the like high [65:43] art version of sci-fi where it's just [65:45] take one weird idea like what if you [65:47] know uh this happened and then write a [65:50] story of just like what would the world [65:52] look like if that happened? Um, it's [65:54] truly excellent. [65:55] >> I think I have that on my bookshelf [65:57] behind me too. Um, also highly recommend [65:59] that one. Favorite recent movie or TV [66:03] show? [66:05] >> Okay, one popular one, one deep cut. Uh, [66:08] Obsession was my favorite movie of the [66:10] year. Uh, [66:13] it's a horror movie, so don't don't go [66:14] if you don't if you hate horror movies. [66:16] But [66:16] >> Oh, thank you. [66:17] >> Oh, no. Even if you're if you're not [66:19] scream, if you're open to a horror [66:20] movie, you should go cuz it's [66:23] >> it was just so fresh. You know how [66:25] movies I feel like if you've seen a lot, [66:27] sometimes you're like, "Oh, it's another [66:28] I've seen this many times." This one was [66:29] just so new. so fresh and I'll say one [66:33] thing about it which is that it takes a [66:34] lot of romcom tropes and like subverts [66:37] them to said be spooky and I think that [66:39] that's just a a brilliant brilliant way [66:41] to to make a movie and then I'm French [66:45] so there's [snorts] a TV show that came [66:46] out this year called breath is French [66:49] for brief uh brief used to be this cult [66:52] classic in France 10 like 15 years ago [66:55] and they came up with a sequel uh many [66:58] many years into the [67:00] And you can watch it. I had a bunch of [67:02] American friends that have watched it. [67:03] It's got subtitles and it's just this [67:05] incredibly [67:07] again it's like novel like the editing [67:08] is crazy. Um they are just extremely [67:11] creative and is very short. Uh so I [67:13] would recommend that. [67:14] >> Favorite product you recently discovered [67:16] that you love? [67:17] >> I I love my Apple Watch and I hate that [67:20] it charges uh or I have to charge every [67:23] day. So I bought the Apple Watch Ultra [67:25] uh just recently and man that is so [67:28] nice. If you've ever been frustrated by [67:30] this good product that somehow we can't [67:32] figure out how to give a good battery [67:33] to, well, it turns out if you make it [67:36] bigger, uh, the battery's better. So, [67:38] I've been happy with that. [67:39] >> Do you have a favorite life motto that [67:40] you find yourself coming back to often [67:42] in work or in life? [67:43] >> There's like one non-fiction book that [67:45] really is like probably my favorite [67:47] ever. It's 4,000 weeks. Um, it has a lot [67:52] in it that that I think is cool. Uh, one [67:55] of the cool things is this sort of thing [67:57] that was really relevant to me, which is [67:59] that like the when I finally mentality [68:02] is a common mentality and is kind of [68:04] toxic. So, so what does that mean? It's [68:05] sort of this like ah when I finally, you [68:08] know, check off this this all the items [68:11] on my to-do list, I'll finally be able [68:12] to chill. When I finally get like [68:15] married, I'll be happy. When I finally [68:16] get a promotion, uh, and then this is [68:18] sort of I mean, you were talking about [68:19] meditation, right? This is like almost [68:21] the opposite of meditation and that you [68:22] know at the end of the day your life is [68:24] like all of the moments you lived uh in [68:26] the moment and that there's this sense [68:28] in which you should just like try to [68:30] always like be you know in the moment [68:33] and not projecting yourself towards like [68:35] a future that seems better like probably [68:37] if you're telling yourself when I [68:38] finally do X I'll be happy when you do X [68:40] you will not be happy. Is there a better [68:42] phrase to use or is it just don't do [68:44] that? Like do they say [68:46] >> uh I don't know. I think to me the [68:48] positive version is like be in the [68:49] moment, right? It's just like just I [68:51] appreciate the appreciate the moment. [68:52] There was a blog post recently I read [68:54] that was [68:54] >> I for something like [68:56] >> like enjoy the sandwich. I think is I [68:58] think it was literally the sense. I [68:59] don't know if you saw this where it was [69:00] like somebody was like yeah like just [69:02] you're eating a sandwich, you know, [69:03] you've eaten many sandwiches like enjoy [69:05] the sandwich. You're in you're in like a [69:07] a team meeting, you know? Yeah. you've [69:09] been in like like enjoy like be be in it [69:11] and and sort of not looking looking [69:12] towards the next thing. [69:14] >> I did a meditation retreat once. It was [69:16] a 10day silent retreat where you eat [69:18] silently too and no one's talking. You [69:20] just sit there and eat your food. And I [69:21] just remember peeling an egg for like [69:23] five minutes and just noticing every [69:25] every shell piece and just the smell of [69:27] it all. And and it like you almost it's [69:30] like hard to get into that mode, but [69:32] it's such an interesting place to be [69:34] where everything is just interesting. [69:35] Like wow, look at this egg versus just [69:37] like stuff in my mouth. [69:39] >> Yeah. Right. While you're like on Slack [69:41] with your left hand and like listening [69:42] to a podcast at the same time. [69:44] [laughter] [69:45] >> Oh man. Okay. Final question. You study [69:48] AI all day. You're watching AI. how it [69:50] thinks. If uh if an AI were to look [69:52] inside your brain, what would what would [69:54] be surprising? What would it see that [69:56] might surprise people? [69:58] >> I mean, surely my deepest darkest [69:59] secrets. Uh but what are those? [70:01] >> Let's get into it. Let's get into [70:02] [laughter] it. [70:03] >> Oh, man. Uh I've always been like a bit [70:06] of a a jokester. Uh or that's just how I [70:11] like both how I live and how I cope. I [70:13] just like think that even in serious [70:15] situations, you should you should aim to [70:17] kind of have some levity. And so I think [70:19] if if you looked inside my brain, you'd [70:21] definitely be surprised uh that like [70:23] basically at any given moment I'm just [70:26] like focusing on some random thing that [70:28] is funny. You know, I'll be in a in a [70:30] serious meeting and I'll be like, "Oh [70:31] yeah, but you know, like isn't it kind [70:33] of random that you're like wearing this [70:34] random thing or something?" It's just [70:36] like I have these intrusive thoughts [70:38] that I that I keep out, but that are all [70:40] usually like just bad jokes. And [70:42] sometimes, as you've heard today, [70:44] unfortunately, they come out. So, I'm [70:45] sorry for those, but that's that's [70:46] probably the most surprising thing. [70:48] >> I love that. Emanuel, thank you so much [70:50] for being here. Uh, final question. How [70:51] can listeners be useful to you? [70:53] >> Just do some interp read some read read [70:56] about how the models work. I mean, you [70:58] mentioned like dorkers thread up. I [71:00] think it was it was great as well. I [71:01] think I think, you know, it's just this [71:03] is such a new field like both interp and [71:05] like AI and LLMs and I think it it just [71:07] benefits from from people engaging and [71:10] and um being informed. So like read [71:14] things. If there's if there's more that [71:16] you would want to know about how these [71:18] models work that we haven't figured out, [71:20] you can, you know, reach out to me. I'm [71:21] on variety of social social platforms. [71:24] Like probably you can reach out to me on [71:25] X [71:26] >> and we'll link to some of the videos [71:27] that you guys have put out that describe [71:29] a lot of the stuff we've been talking [71:30] about like JSpace and things like that. [71:32] There's some really beautiful videos [71:33] your team has made. I don't know who [71:35] does that, but they're really cool. [71:36] >> Yeah, this is we partner with our our [71:38] design team. So all credit goes to them [71:40] for for the cool videos. [71:41] >> Amazing. All right. Well, thank you [71:43] again, Emanuel. Thank you so much for [71:44] doing this. This was amazing. [71:45] >> Thanks, Lenny. Yeah, this was super fun. [71:47] >> Bye, everyone. [71:49] Thank you so much for listening. If you [71:51] found this valuable, you can subscribe [71:52] to the show on Apple Podcasts, Spotify, [71:55] or your favorite podcast app. Also, [71:57] please consider giving us a rating or [71:59] leaving a review as that really helps [72:01] other listeners find the podcast. You [72:03] can find all past episodes or learn more [72:05] about the show at lennispodcast.com. [72:08] See you in the next episode.