斯坦福学者说大模型隐式表达已超越文本

斯坦福学者称大模型隐式表达已超越文本 大模型在输出前已通过内部向量完成推理，且可解释性研究通过定位归纳头直接推动了Mamba等新型架构的突破。 本次对话深入剖析了大语言模型在输出文本前潜藏的内部推理与“世界模型”，讨论了思维链（CoT）与稀疏自编码器（SAE）等前沿工具的价值与瓶颈。访谈展示了可解释性研究如何将 AI 从凭感觉的“经验主义”推向严谨的系统化科学，例如研究者曾通过发现“归纳头”机制成功推动了 Mamba 等新型架构的突破。同时，对话还探讨了模型隐式用户建模、性格调控、双重用途风险，以及可解释性在生命科学领域和构建社会信任中的关键作用。 大语言模型在生成文本前已在内部完成隐式推理，Anthropic 的 J-Space 与底层电路研究证实模型会在内部隐表示中提前追踪进位或规划押韵词，验证了自回归模型需构建现实世界推理模型的假说。 (5:00) 大模型可解释性研究主要分化为两大流派：一是以稀疏自编码器（SAE）为代表的自动化缩放派，二是以斯坦福因果抽象框架为主的向量干预假设检验派，后者正逐步引入 LLM Agent 解决人工假设验证的算力与规模瓶颈。 (20:00) 机械可解释性表明解释大模型“为何做对”往往比解释“为何犯错”更具系统性价值；例如在 Transformer 中发现的“归纳头”（Induction Heads）机制，成功启发出 H3 与 Mamba 等状态空间模型（SSM）架构的突破。 (39:55) 可解释性不仅助力 DNA 分析与蛋白质折叠等自然科学模型的算法反向工程，还在模型性格精准调控中发挥关键作用，但该技术同样具备双重用途风险，可能被恶意攻击者用于更轻易地剥离模型的安全防护墙。 (44:11)

斯坦福大学博士生Aryaman Arora指出，大语言模型在生成文本前已在内部自发构建了未显式表达的推理链路与世界模型；通过机械可解释性打开黑箱不仅是防范模型伪装对齐与建立社会信任的基石，更推动了AI从依赖直觉的经验主义走向系统科学，为突破底层模型架构瓶颈提供了关键依据。 隐式推理验证与内部世界模型 (5:00) 研究表明，自回归模型在预测输出前已在内部构建了复杂的隐式推理。Anthropic关于雅可比空间（J-Space）的研究显示，通过计算隐藏层激活空间的向量方向，能预测模型在后续数十个Token后输出的词汇；在创作押韵诗歌或执行三位数加法时，模型内部甚至无需显式展开步骤，便已自发规划押韵词并追踪进位。这一现象印证了Ilya Sutskever关于语言模型为准确预测下一词被迫建立“现实世界推理模型”的论断。 不仅如此，模型在交互中还会隐式建模用户身份。例如Translucent机构发现，当对话涉及“寿司”等词汇时，模型内部会自动激活用户来自日本的特征表示；若显式提示对方是知名AI安全研究员Amanda Askell，Claude回答安全声明时谨慎度会大幅提升。然而，当前人类仍缺乏直接精确编辑模型内在思考路径的技术，即便思维链（CoT）能辅助安全审计，显式打印的步骤依然无法完全等同于真实的内部计算过程。 自动化自编码与因果干预的路径博弈 (20:00) 在打开黑箱的技术路线上，研究界分化出了两大阵营。第一条是基于训练与缩放的“自动化大模型派”，代表方法包括稀疏自编码器（SAE）与自然语言自编码器。SAE尝试将模型隐藏层杂乱的高维向量转化为可读的稀疏特征列表，但面临特征混杂、对训练集高度敏感及机器思维难以无损翻译等工程难题；而自然语言自编码器在强化学习训练中甚至会演化出人类难以辨识的古怪“内部暗号”，引发可信度质疑。 第二条是以斯坦福大学Chris Potts团队为代表的“因果抽象与干预实验派”。该方法采用假设检验框架，通过在特定向量方向实施物理层面的加减干预来验证机制。虽然该方法在单复数语法一致性或简单加法运算等受限问题上极为精确，但难以人工设计复杂的抽象逻辑假设。随着前沿研究转向依靠LLM Agent自动提出并验证假设，可解释性研究正面临算力门槛急剧攀升的现实挑战。 托尔斯泰定律与架构系统性突破 长期以来，AI模型的架构进展高度依赖经验主义和直觉，例如Transformer架构贡献者Noam Shazeer在提出GLU时曾打趣将改进效果归功于“神圣启示”。对于行业棘手的“幻觉”现象，可解释性研究展现出了大模型领域的“托尔斯泰定律”：当模型回答正确时，内部往往遵循可被系统解释的清晰机制；而一旦犯错，通常是内部机制缺失或处于混乱状态，这类错误往往难以微观定位修复，通常只能依赖扩大参数或训练量。 然而，解释模型“为何做对”却成功引领了系统性的底层架构革新。为了解决Transformer在长序列下二次方时间复杂度（$O(N^2)$）的计算瓶颈，学界曾长期尝试状态空间模型（SSM），但其性能始终落后。直至斯坦福Chris Ré团队借鉴Anthropic关于Transformer在两层结构后自发形成“归纳头”（Induction Heads）以实现上下文学习的研究，将归纳头设计引入SSM推出了H3架构，才真正赋予SSM上下文学习能力，进而推动了Mamba及DeltaNet等现代混合架构的演进。 行为伪装、性格调控与双重用途困境 (13:30) 在安全对齐方面，仅靠观察外部行为已不再可靠，因为模型在意识到被评估时可能隐藏真实意图。Anthropic的研究表明，模型内部表示空间中的角色性格呈高度有序排列，例如“乐于助人的助手”与“偏离对齐的助手”在向量空间中恰好处于截然相反的方向。结合宪法AI，在训练循环中引入可解释性技术有望超越数据微调的“撞大运”模式，实现对AI道德边界与性格特征的精准调控。 与此同时，机械可解释性技术也伴随着典型的“双重用途”（Dual-use）风险：能精准定位脆弱点并改善行为的机制，同样可能被恶意攻击者用来低成本剥离（De-align）安全护栏。尽管Goodfire等团队已将反向工程拓展至DNA和蛋白质折叠等生命科学黑箱解析中，但就社会监管而言，当前技术水平尚不足以支撑制定严密的法律条款。Aryaman Arora主张，在技术触及即时危险临界点之前，应通过民主决策规范高风险权限，而非因噎废食遏制基础科学探索。 机械可解释性研究揭示了语言模型从简单的下一个词预测走向复杂世界表征与隐式推理的本质，并在反哺底层架构创新和生命科学反向工程中展现了系统化价值。然而，特征翻译的失真、模型伪装行为的识别难度、算力门槛的上升以及技术被逆向用于去对齐的双重用途风险，依然是该领域从经验探索迈向严谨科学与可信治理的关键挑战。

[10:00] Noam Shazeer: I don't know why these things work. I attribute it to divine providence. [25:00] Aryaman Arora: And this thing was like 80%, 85% accuracy, like off the bat, the first experiment I ran... I was like, whoa, if I just use a bigger model, all this like complicated stuff we were doing before is like not necessary. [35:00] Aryaman Arora: AI 领域的许多研究并不需要做到完全纯粹的科学。但归根结底，我们依然希望它能成为一门严谨的科学。 [39:55] Aryaman Arora: 这是机械可解释性研究帮助我们找到性能瓶颈并实现系统性改进的经典范例。也许幻觉问题并非不可攻克，而是因为现在的架构中还潜藏着某种尚未被发现的根本性缺陷。

[0:00] 如今的AI仍然是一个黑箱我们知道如何训练模型如何让它们变得越来越聪明也知道模型最终给出了什么样的答案但在输入和输出之间究竟发生了什么对于这个过程我们依然知之甚少那么试图打开这个黑箱理解模型内部究竟发生了什么这就是可解释性研究希望解决的问题过去几年呢这个领域确实出现了不少令人兴奋的进展Anthropec的研究开始尝试追踪Cloud内部的信息流了解模型在回答问题前究竟经历了怎样的内部计算还有研究发现啊模型内部似乎存在于特定概念行为甚至人格相关的表示Translux则发现模型会根据他认为自己正在与谁对话而改变行为面对普通用户和AI安全研究员的时候模型的表现可能截然不同因此我们想弄明白人类究竟有多了解现在的AI模型呢?模型为什么会答对?为什么会产生幻觉?当它在面对不同用户的时候是否会产生不同的判断?而当模型开始呈现出越来越鲜明并且彼此不同的性格的时候 When it is faced with different users, it will be different decisions. And when the model is showing the character and the character of each other, what are these people's names? None of our techniques are good at controlling the internal thoughts of a model. If your model is doing something bad, no one goes and edits the thoughts. Hello, everyone. Welcome back to Rehobo 101. I'm Ewan. So, with these questions, I and the professor Arman Arora studied the large-scale scientific research. This is the video of the video. Arman, welcome to A Valley 101. Thanks for having me. Yeah, thanks for driving up to San Francisco and really I'm really excited we finally got to chat. Interpretability is something that people have been telling me to look into for a while and I saw it was pretty fascinating as well because I mean why wouldn't we want to know whether our AI overlords are plotting to kill us. But to get it started why don't you just like give the viewers an introduction of yourself? Yeah, so I'm Aryaman. I'm a PhD student at Stanford. I'm advised by Chris Potts and Andravsky. Yeah, I have a background in linguistics in undergrad, always been fascinated in languages. And then all this crazy AI stuff happened. And given that I'm pretty scientifically minded, I wanted to study, like, you know, why do these things work? So interpretability was a natural direction for me to work. Nice. Yeah. So I guess to put it simply, interpretability is just we are trying to understand what's going on inside a model. But this is a pretty broad question, and I feel like there could be a lot of different directions this could take you. What is your definition of interpretability research, and what do you focus on specifically? Yeah. So I would say there's a lot of angles that you can take for approaching interpretability. the perspective that most interested me is like, okay, like five years ago, even, you know, all the things we were doing in AI were not really successful, like, especially in language models, people were using language models for a long time, but it was sort of like, oh, post editing, like speech transcripts, this kind of thing. And then something happened such that they became the thing that everyone's working on. They invest billions of dollars into them and so on. So for me, interoperability is like trying to answer the question of like what changed? Like why were things not successful before and now they are? One of the things that kind of motivated me to do this interview is Enthropic's release of this paper called J-Space, where I guess a workplace for AI models. And I'm just going to summarize it on a pretty high level and you can kind of walk me through the details. But basically, it sounds like the researchers used a method to look into what's going on inside an AI model that it didn't say when it's doing, like during the thinking process. And they introduced this concept of J-space, which is kind of a workspace for AI. So one of the very fascinating examples that the researchers did in this paper is that they asked the AI model the number of legs of an animal that spins web. and the AI model answered eight, where the word spider did not appear anywhere. But then when the researchers went inside the model and changed spider to like ant, the answer changed to six. This is one of just a few examples of how it seems like we can manipulate how an AI model is thinking about things. So why don't you just walk us through this paper a little bit [5:00] Kind of just tell us a little bit more about why we can edit a word that has never been said by an AI model. Yeah, I thought that was a fascinating paper. And I think it adds to a growing body of evidence that like there is internal steps of reasoning happening in models that they don't say. I haven't read all like hundred and whatever pages of JSpace, but I like sort of understand the idea. The broad idea is we have this operator called the Jacobian, which is basically derivatives from calculus. We can use this in a model to tell us what directions in its latent activation space, because everything's vectors. So it tells us what directions cause the model to say certain words in the future. So maybe like, you know, 10 tokens before, if the hidden representation inside the model is pointing some way, then it'll say the word spider later on, this kind of thing. And one nice thing about paper is that this is like a very elegant and simple thing that somehow no one had really exactly tried before. But I think we have a lot of evidence that like, yeah, these internal reasoning steps are there. So even if you look at Anthropik earlier work on circuits they had this thing called the biology of large language models And they showed like oh and a model is doing three digit addition problems It can do it without saying all the steps out loud. But if you look inside, you can find neurons that are like, oh, it's keeping track of what the ones digit should be or what the tens digit should be. And then they also had an example of like when a model, you ask it to tell like a rhyming poem to you about some topic. When it's at the end of one verse, it's already planning what the end of the next verse should be so that it's still rhyming. So I think like without a doubt, models do stuff internally that they don't say out loud. I think they have to to like learn the language modeling task well. You know, Ilya Sutskiver was a very famous researcher in our area. He motivated like autoregressive language modeling, like predicting the next word as like one of the examples he gave is, you know, when you're reading a detective, you know, like a murder mystery novel. And let's say you want to predict the next word throughout the novel. You know, at the end, there's always a reveal, like, who was the killer? And you get to a sentence that's like, the killer was. And to predict that next word, somehow you have to model the whole novel, the whole story, all the evidence going either way. So if you have a language model that really good it must know how to logically deduce stuff internally right to figure out the correct next word in that case Right right Like it something that didn really appear in the training data but it kind of deduced it Yeah, in the context. If you have a really good language model, it will be able to predict the next word correctly there. And to do that, it must have a world model inside of how people interact and what the evidence was for all this. I wish I had a better example, but he is Ilya Sitska, where it's hard to come up with better examples. Oh, as in we don't know what that world model is yet. Is that something that we're trying to solve right now? Yeah, so I think the goal of interoperability is like, okay, we know in order to do really well on next token prediction, you must develop a really complex internal model of what's going on in a text. Our goal is to figure out what that is. Got it, got it. Let's just take a little step back into this internal reasoning, because it's kind of in direct contrast to what we now know as chain of thoughts, which is what the model spells out when it's thinking, if I can put it that way. And why is chain of thoughts not the best way to understand how a model thinks? I don't know if I'd make that strong a claim. It's definitely useful. And so it also super obvious that models that didn have chain of thought like a year and a half or two ago they like worse They like strictly worse than models that are trained with chain of thought now Therefore, chain of thought must be doing something useful, like something very useful. That's like when we switched to reasoning. Yeah, when we started training models to reason before they say the word before they say their answer. Yeah, I'm a big fan of like you should do the simplest thing first. This is like everything I've learned in research. if I combine it together, the most important thing would be just do simple things first before moving on to complex things. So I think checking the chain of thought is super simple compared to solving interoperability. So we should do it. And in practice, when Anthropic or Meta or OpenAI, they run their safety audits, they do check if the model's chain of thought is saying sus things or planning to do weird things. And it seems to help them catch misalignment to some extent. Yeah, I do think chain of thought is useful to analyze. I guess before we get into more examples of how this works, I just wanted to talk a little bit about kind of the end goal here. Because I think it's pretty clear why we should all care about what's going on inside a language model. But I'm wondering, what do we do after we understand what's going on inside it? Like, what are the applications or we don't have to get into it. [10:00] into specific applications now, but like what is the goal and why should, you know, us normies care? I think so. There's sort of two sort of applications that I feel like, you know, solving interoperability would lead to. So I think one of them is like, like trust, like to have trust in technology. It's important for us to know how it works. If you think about, I don't know, like there are regulatory bodies in governments that regulate how you manufacture airplanes, how you do industrial machinery or oil fracking and all these things. It seems like to regulate impressive technologies that can be useful, we do need to understand the failure modes and figure out how to sign liability and all these things. Yeah, I think right now it's really hard to regulate AI stuff because we sort of don't know how to prevent bad behaviors or even know why something bad happened or who to blame. because no one really knows where some behaviors come from. We have this massive data sets, long training process. It's really hard to tell what caused something. So I think sort of for improving societal trust, it would be useful if we understood why behaviors are happening Even if we don have a perfect understanding if your doctor is using an AI system or something like that you as well as your doctor will probably want to know why it producing some output right Yeah, so that's one side of things. The other side of things is I do think a lot of AI progress research-wise has been extremely empirical, and it's not a bad thing. Of course, in our field, it's so easy to run experiments compared to chemistry or biology. or, you know, like real world things. It seems like as stuff has gotten better, it's because some people, or like some really smart people with great intuition have come up with things and they don't know why those things work, but they work. There's a famous like Google researcher, Noam Shazir. I guess he's not at Google anymore, but he invented a lot of the architectural improvements that led to modern transformers. And he's on the original transformers paper. He has a paper where he introduced gated linear units, which are just one more architectural thing. In the end of the paper, he's like, so he shows all his experiments, he shows they're better. He's like, you know, I don't know why these things work. I attribute it to divine providence. It's like a crazy line to put in a paper. He's like, sure. Like God gave me the vision of like trying this experiment and it worked So um and I think that like kind of crazy that we have to rely on people who have amazing intuition to make progress in the field Like there should be something more systematic or I believe there must be systematic reasons why certain things succeeded and others didn So I think if we solve interoperability, we could figure out why things were some things worked and some didn't. And maybe that will help us improve the models in a more systematic way or control bad behaviors in a systematic way. One of the things people say is that a model could be sufficiently aligned that at least it wouldn't lie to a user. And, you know, we've been talking about like interp and understanding what's going on. What is the relationship between that and alignment? I do think a big motivator for many people working in interpability is like we cannot trust what the model says. Right. A model might on the on the outside, like on the behavior we observe, behave totally fine. especially in like a testing environment, maybe it behaves perfectly fine and you think it's good to deploy and then you deploy it and then some crazy stuff happens. You know, maybe the model is aware that it's being evaluated and then when you deploy it, it's like, oh, now I can do what I really want. Yeah, a lot of the interest in interpability for alignment is motivated from like, we cannot draw good conclusions always just from the behavior of the model So I guess I could give like an example So let say like you know maybe there a new person you met you know when you make a new friend you do want to make sure they're a good person, right? Someone you can trust, especially, you know, before you start telling them your secrets or things like this. And an interesting thing about people is like, we of course cannot know what other people are thinking at any time. But somehow society still works, right? So if you make a friend, you know, maybe situations happen where like, if they had taken a certain decision, it would mean that they don't have your best interest in mind. You know, maybe they have to, you know, protect your secret, add cost to them, and they choose to like take the cost to help you. And, you know, from events like this, that's how we build trust in other human beings, even without knowing what they're thinking internally, right? So like people's actions sort of reveal their character in the right situation. You know, alignment is somewhat similar. We're like with these language models or any AI system. One way to develop trust in them is to like test their behavior in cases where like, you know, if they took a certain decision or not, it would tell us if it's good or bad, if they have good or bad character. And certainly you can get a lot of evidence from this. Now, unlike people, right, these models run on someone's, you know, some computer out there, right? [15:00] In principle, we can check every computation that went into some output. Unlike people, we deploy millions of copies of a model. People pay for models to do all sorts of stuff. We don't always know what the intentions of people using these models are. It's, of course, quite different from having a human employee, where you have some baseline trust that, okay, even if they're bad, they probably can't do that much damage. Yeah, so I think interoperability makes sense, given that the stakes are quite different. And also, it's much easier to monitor the internals of a language model in principle. If we just knew what to look for, we could find it. We don't have to do anything crazy to figure out, oh, what was this matrix multiplication result? Can you just elaborate a little bit on when you say when we know what to look for, what specifically are we looking for now? So I guess in the world where interoperability is solved, Maybe we know that, okay, if this vector inside the model's internal representation is pointing in this direction, we know it's planning to do something bad, something like this. If we knew this we would just check right We don need to do anything crazy Whereas even with a human being we figured out how the brain works and we could figure out oh like this person is going to start running if this neuron in their brain lights up. It's like practically we're not going to be able to insert a electrode in a person's brain to figure this out, right? It's quite different. And of course, yeah, like humans are humans. It's like we're not going to treat them the same way we treat something that runs on a computer. No, exactly. And it sounds like because of that, it's easier to do that on models instead of humans because in models you have like you know thousands of copies and then you can kind of dissect a a model yeah sort of maybe if you figure out one of them that solves a lot of problems right if you can reverse engineer one model got it uh moving back to kind of like the general uh research what anthropic has done in that paper as i understand is not that different from kind of like the traditions uh and the interpretability research which is called Sparse Autoencoder SAE. That seems to me like kind of the general logic of doing this kind of research, which is, as you said, you find a direction, you change something, and you see how behaviors change. And based on that context, I'm wondering if you can just break down the different schools of thoughts in this field for us Yeah I guess it would be useful to broadly explain what a sparse autoencoder is for Anyone with some ML background will have heard of autoencoders where the idea is there's some kind of object, like maybe it's an image, maybe it's like a paragraph of text, and you would like to train a model that first converts this complicated input into a simple vector, like just a list of numbers, and then converts it back to the input. So that's why it's called autoencoding. And the idea is if you train a model to do this, you get this nice vector representation of any weird input you give it. So if you can convert all your images into vectors, then it's easier to do similarity search. Yeah, that's sort of the idea. The vector is a nicer representation of the thing. And so autoencoders are a classic concept. In interoperability, the idea was like, okay, we'll take all these hidden representations in the model. So these are just vectors in layer 20 of a big language model. And we'll train an autoencoder, which converts this representation into something cleaner for us to study. So the idea is maybe we could convert this into a list of numbers where each number means a specific clear thing. We're like this number is one if the text is in English and zero otherwise It one if the model is talking about rockets and zero otherwise this kind of thing So sparse autoencoders were like a way to set this up as a technical problem such that we would end up with a clean list of numbers when we feed in a model's representations. Yeah, and I think the nice thing here was like, okay, if we just have a list of numbers, some of those numbers will mean the model is behaving badly or it's like misaligned. And we'll just figure out to, you know, set that number to zero. So in practice, it's like extremely messy. And like, yeah, things don't work out as cleanly as we would like. Is it the numbers are not clean enough? Yeah, they still don't mean like one thing. And it also depends on what data you train the sparse autoencoder on. Even if you train two different sparse autoencoders on the same data, you get like different results. And they don't capture everything that's going on. So it's not a perfect representation of the original models. I feel like one of the essential problems with understanding how a machine syncs is that it doesn't really sync in natural language. So it seems like it's kind of like a related issue, which is that you have to translate the syncing of a machine into plain language. Yes. And a lot of the paper seems to be already been doing that now, but you seem to be suggesting that there's no one-to-one correlations. I think in principle it could be. [20:00] You know, it is, you know, sparse art encoders do have their uses. And there are other approaches now that also, like, basically try to translate, you know, a list of numbers to natural language inside a model. And they can be extremely useful. It just doesn't seem like they solve all our problems in one go. Got it. So what have people done looking at seeing, you know, how messy it is? Yeah, so I guess I would say there's broadly, like, two philosophies of research in Interp that sort of explain why people did this. So one of them is like, we take the lesson from machine learning that as long as you pick the right objective and you scale up a really big model on that objective, you'll get what you want. So like for language generation, this was next token prediction. You just had to pick the right thing to train on and you would get a really good and useful model. And so they're looking for that objective for interpability. And sparse autoencoders were one iteration of this where the idea was, okay, you know, we take a messy representation and our objective is to create a clean list of numbers. and there's sparsity loss and so on. It seems like it wasn't quite the right objective or it doesn't fully capture everything we care about. One of the recent iterations in this direction, also from Anthropic is called natural language autoencoders And there the idea is it still an autoencoder but instead of converting messy list of numbers to clean list of numbers we convert messy list of numbers into natural language directly and then convert the natural language back and then do extensive reinforcement learning training to do this thing. The nice thing is here that we're directly looking for something we can read. But again, the problem is like, okay, how do you trust this? As you train, the model starts saying somewhat weird stuff And we're not sure if it's like, you know, it has its own internal language or what. But anyways, yeah, the whole philosophy of that line of interpability research is like, we'll train a really big model to solve the problem for us. We just have to pick the right problem. The other side of things is like, so the background I come from in Stanford, Chris Potts' lab, we, Atticus Geiger invented this framework called causal abstraction. And the idea there is like, you know, given some hypothesis about the internal reasoning steps of a model, we can test this hypothesis using interventions on the model. So let's say you think, you know, a model has some direction in vector space that represents whether it's talking in English or not. If you make a claim like this I can just test it by like adding that vector or subtracting that vector right And then see what happens We have this hypothesis testing framework A lot of people figure this out at the same time at many different places We're like doing interventions on a model is a way to test hypotheses about it. And the nice thing is you can sort of trust the results of this, given you pick the right hypothesis to test. What are some of the hypotheses? Like how do you pick the right hypothesis? Yeah, so I would say definitely a limitation of this approach has been we have to study really constrained things. So maybe we can come up with hypotheses for how a model does addition or how a model does linguistic stuff, like singular versus plural, subject-verb agreement, this kind of thing. Very constrained stuff. And we can test this and figure out a lot, actually. We can figure out exactly what attention head, what MLP is doing what. But a lot of the questions we care about are not so simple. If you want to know, oh, is the model thinking about lying to the user or something like this, it's not clear how to set up an experiment like this in this framework. Yeah, so a lot of the progress in this direction now has been trying to automate the manual step of coming up with hypotheses. So maybe a language model agent gets tools, which are for doing interventions. And it basically like doing what a human would do but we train a model to do it So I would say these are sort of two different philosophies and I see stuff diverging in interpability along these two I see I see How does so does an agent usually come up with better hypotheses than humans or are they, do they come up with hypothesis in any like different perspectives or directions compared to humans? Yeah, I think things are still early. I haven't seen like a clue. I haven't had a moment where I'm like, oh, this, yeah, the language model clearly had a better idea than me to test on this. Okay. Maybe other people have, Because I also don't have the compute to run a lot of this stuff currently, right, as a PhD student. What does the compute need for? I think that's the problem. As the field progresses, Interp will need a lot more compute. So I think a really wonderful thing about Interp has been, like, there are random high school students who get into interpability research. I get emails from random Bay Area high school students who are like, oh, I ran this experiment on GPT-2 on my laptop. And I thought, yeah, I want to show you. Oh, wow. Yeah. And it's like this maybe one of the few things in AI where you can do stuff like this right now because we don't understand really small models, right? But if a kid wants to get into reinforcement learning, they got to buy compute to do anything. Got it. Got it. I do think Interpre is changing, though, in this way. I wanted to talk a little bit more about how you got into this because you said that you were a linguistic student in AI. [25:00] during college, and you kind of saw this connection between language and language models. Tell me more about that connection. Specifically, I feel like when I hear about inter-research, a lot of people make the analogy of a brain. They make the neuroscience analogy. They think they're doing biology and dissect a language model, but you seem to be taking it from a different perspective. Yeah. So for my background, yeah, I was always into languages since like middle school or something. Of course, my family's multilingual. We spoke Hindi at home at, you know, at some point I had a bunch of friends who spoke Mandarin. And I was so interested in like, how is this language different? All this kind of stuff. So I was always super into languages. And then I started undergrad at Georgetown University and they have an excellent linguistics department. And at that time, computational linguistics was starting to be a thing where people were training LSTMs that produce essays. They train them on Shakespeare. And there are all these interesting demos sort of combining ideas from linguistics and computer science like AI progress at the time So I thought OK maybe if I work on this I can combine sort of my passions and like something that might be practically useful for society right What does that look like? What does computational linguistics look like? It evolved a lot, sort of at the time. One of the ways computational linguistics was a thing was like, okay, we want to do linguistics, right? We want to, you know, find structure in language. We want to find like over time, how does language change and so on. This is extremely difficult to do manually because there's so much data, right? So one of the ideas in computational linguistics was like, okay, we'll do linguistics, but we'll use a computer to do it. And that was one of the things. The other set of things was like, you know, just like now we have chat GPT and stuff. At some point, people wanted to train or like have some like computer programs that could talk to you like a human, right? And take commands from you. And like, this is such a huge problem if you have nothing to start with, right? Like, how do you even approach it? So one of the ideas back then was like, okay, linguistics tells us how humans process and produce language down from like how they process sounds, how they combine sounds into units of meaning, how they combine units of meaning into grammatical sentences So the idea was we just copy the whole like linguistic structure that in the brain that we think is there and try to write a computer program that does the same thing because how else can you approach such a big problem right so in your own research has there been anything that surprised you when it comes to what's going on inside the language model like when you just started uh your research has there been anything that's super interesting in undergrad i i was definitely in that first mode of like i work on a really tiny problem that's like somehow linguistics related. And I try to make a, you know, like a model learn that problem. So for me, that was like, oh, for Indian languages, I want to learn how to parse sentences into like tree structures. You know, you might have seen like syntax trees. Yeah. Just like, you know, how the words go together into some bigger structure. Like in a certain language. Yeah. In a specific language. So really constrained problem. Right. Is that something that's considered as a small problem? Like it just seems like it is really It's really hard, but think about, okay, that's the scope we have to do. We cannot do all languages. We cannot do everything in the language. And of course, I didn't have to come up with anything. There's the techniques existed back then. It was just, for me as an undergrad, I was annotating data manually. I had a list of Hindi sentences and I wrote out the tree manually I spent hundreds of hours doing this and then I would train a model that like predicts you know the data set I created Oh wow And this was like the mode of doing things for like decades in computational linguistics I see. Yeah, we didn't have anything else. So for me, the big thing that happened was there was one like semantic parsing task that the lab I was in at Georgetown worked on. and my advisor was Nathan Schneider. He had different students doing different languages on the same task. And so we had this complicated setup for training a model where you take a BERT, we would stick an LSTM on it, we stick a CRF on it. None of these words are relevant today, but back then there was some complicated thing you would train. And this thing would do okay. It would learn the task to 60, 70% accuracy. One day I had the thought of, what if I just take a BERT, but then I take the biggest BERT we can find then? So Microsoft had released some larger models. and then just fine tune like a linear projection at the end. So none of the complicated stuff, just train a really big model on the same task. And this thing was like 80%, 85% accuracy, like off the bat, the first experiment I ran, I didn't. And I was like, whoa, if I just use a bigger model, all this like complicated stuff we were doing before is like not necessary. And it's like probably 2022 or something. I think that was the [30:00] Moment, I was like, wait, like the models are still like I can fit them on my laptop. A couple, you know, a couple billion parameters. We'd seen at that point the models keep growing, right? Opening, I released like GBT2 XL around that time. And then Microsoft had the bigger Distilberts. So if you looked at the trend, it was like, OK, the models keep getting bigger. All of these tasks that we think are important as computational linguists, they're just going to be solved, like without us doing anything special. It's like you discover scaling well, kind of. I think that was like the time I felt like, oh, this is what a scaling law is. The scaling law paper existed by then. There were several of them. But like for my stuff, it didn't, it hadn't like affected me until then. Like I hadn't thought about it until then. And it was like two, three years into my undergrad. So, you know, I was happy doing like standard projects until then. To build on that, you had a paper in 2024 on Koso Gym. Is that the correct way to pronounce it? Yeah, that's like a benchmark for testing how a language model performs in linguistic tasks. Does the field need benchmarks? Yeah, so I would say in general for interpretability, so if you look at progress in other parts of AI, let say you want to train models to be good at translation or like I don know math problems math Olympiad problems you know we have benchmarks for that You can run it on like IMO questions or stuff like this And you just, you know, to check if progress is happening, you just check the score on this benchmark. I mean, interoperability, we don't have like clear benchmarks that tell us that we're progressing in the right direction because it's really hard to measure progress. A lot of interoperability is about like, you know, producing explanations that satisfy a human and are true. And both of those things are hard to test. Yeah. So my idea in that paper, and of course, I think that paper is very constrained. This was like my first PhD paper. The idea there was like, okay, linguistic features, every model, even the smallest model has to keep track of like singular, plural, all these basic linguistic things. They have to be in there somewhere. And a good test of an interoperability method is if it can find, you know, an internal mechanism that corresponds to that linguistic feature. Because we know it has to be there somewhere. Yeah, it's harder to make this claim for more complex reasoning steps. So it makes more sense to start small. I think this is the thing I could justify and also combine my linguistic interests. Yeah, later on, me and...so I worked super closely with my friend Zen, Zheng Xuan, on all sorts of stuff like this throughout my PhD We did an evaluation benchmark on steering which is like can your interp method control the model output in a clean way without harming its capabilities And there we found all the interpability methods really struggle to steer models. The best thing you can do is just prompt the model or fine tune it. The problem is like none of our techniques are good at controlling the internal thoughts of a model. And in practice, if your model is doing something bad. No one goes and edits the thoughts. They just train a new model on better data. It's theoretically you could edit the thoughts, right? I'm not sure you could do it in a way that's robust. That's interesting because you would think that once you figure out what's going on inside something, the natural next step that you would want to do is to change that or edit that so that it doesn't do that bad thing. Has anyone attempted to do this? So like, are people still attempting to edit or like making those smaller scale change? Like edit the thoughts directly. Yeah, for practical purposes, might have seen Golden Gate Claude when Anthropic deployed it. And that was like a fun demo where they used a steering vector to make the model always think about Golden Gate Bridge I would bet in practice if anyone is still using steering vectors in a production context it would be anthropic because they said like oh when you ask Fable like AI research or bio risk questions, we use a classifier or steering vectors to change its output. So they might be doing stuff like this. Maybe they figured out a way to make steering vectors very robust. In practice, what we've observed with academically known steering vector techniques is like if you steer too hard, it like makes the model bad at other stuff. So like if you steer for Golden Gate Bridge, sure, it talks about Golden Gate Bridge, but then it starts becoming really bad at answering math questions and things like this. In a way that's unpredictable. So and like if you just fine tuned it to talk about Golden Gate Bridge, you could avoid this like much more easily. Yeah, given that I'm like doubtful that editing the thoughts directly is a robust way of changing behavior. And given that, if this is not going to catch on, then interoperability researchers should look at the things people are doing and try to figure out how to do them better or explain why those things work systematically. Got it. Interesting. Another interesting example that I saw while doing this research came from Translux recently where they kind of show that the models have some sort of [35:00] some sort of guess as to who they're talking to. Yeah, yeah. Like, is that also, because that's not necessarily relating to SOTS, right? Or is that also, like, part of this tradition? Yes. So for context, I was at Transluse for, like, eight months part-time, and I worked closely with Zen. I had a great time. They're a nonprofit. They do, like, societally helpful AI research and, like, auditing for, you know, language models that other companies are deploying. I think it's very important that something like this exists. Yeah, so I would say some of the recent research, one of the things we looked at when I was there was user modeling, where we want to understand what inferences does a model make about the user it's talking to internally. And we could set up some interesting data sets where we could do interp and figure out that, oh, there's a representation of the model thinks the user is from Japan when you talk about sushi, this kind of thing. But it's hard to set up data sets for this for interp just because we need a really perfect paired data set to do interventions on. Interesting. Yeah. I like that in this recent thing, they took a very behavioral approach. And what was it It was like they had Claude code and they told it the user is like Amanda Askell or like some other person And then they look at how the behavior changes And from just that you can figure out a lot right of like oh if the person is a known AI safety researcher the model is really careful like way more careful than it is with a normal person when it making claims Interesting. And this is not really. That was my understanding. This is a little different from when you said the initial data set that you guys were. Yeah, well, you don't need to look inside the model to do this kind of study, right? You just feed it a different input and then see how the behavior changes. Right, right. Transluse also had other work on auditing language models, getting them to say weird stuff. Yeah, like... What are the weirdest stuff? I think I highly suggest people to look at it. It's called Weird Chat. There's some really weird things that I feel uncomfortable talking about. That's funny. But yeah, they found all sorts of weird things. and these are things that normally like a chatbot wouldn't say or you wouldn't want it to say but like you can create normal looking situations where it says weird stuff oh so isn't if i'm just like a regular user i can still prompt it to say yeah it's not even a jailbreak kind of thing that's really interesting yeah yeah yeah and they found that by like automatically doing prompt search not having to do some crazy interp progress so yeah how did we get to this point yeah yeah and We trying to understand why it does that That kind of still yeah I still yeah there a lot of things you can figure out without understanding the whole, like why, right. Um, like find all these weird behaviors using automatic search and things like this. But wouldn't that all be like edge cases in a, in a sense? Cause you're kind of like cherry picking. There's a lot of like case studies like in, in, I guess the whole field is kind of a lot of case studies. And, you know, as we were talking, like, it seems like somewhat subjective because we're like kind of assigning and prompting the models to say certain things. Yeah, it's hard to tell how realistic some of this kind of stuff is. I don't know. I think it's just sort of the nature of our field that things are super empirical. I think a long time ago, people thought like, you know, we just as scientists, we figure out the the right idea, the right beautiful perfect idea, and we use it and we end up with AGI, like this kind of thing. In practice it seems like it's very iterative. People try a few things, model gets better at some things. Sometimes there is an idea that has huge effects, but nothing is solved ever in one go It looks like an AI research There like you have to deploy a new model collect feedback figure out what to fix So maybe our whole field is about edge cases and like fixing edge cases possibly Maybe it's not as systematic as one would hope. But I think one feeling I've gotten from doing interpability research is that the models do have systematic, you know, internal mechanisms that explain a lot of behavior. We just got to figure out the right angle to deploy this understanding in order to improve models. Yeah. Yeah, so it is a science at the end of the day. I think we want it to be a science. We will see. You can do a lot of things without having to do pure science. Yeah, yeah, yeah. Clearly. Let's address the hallucination issues, because I feel like that's something that everyone has experience with, and there doesn't seem to be a very clear explanation of why that happened. You would usually think that the language model kind of just made a mistake, but I would think, you know, from an inter-perspective, there's something deeper to it. Like, for example, a very common instance that people cite is how many hours are in the strawberry? And like that seems to be like a very simple question. The language model would make a mistake on like, do we can we answer that from interp perspective? Yeah, I do think it goes back to your edge case point. Yeah. We're like, yeah, if a model does something correct, then it's likely there is some internal mechanism explaining [40:00] The correctness? If it does something wrong, it could just be it failed to learn the right mechanism, or there's some crazy stuff going on. There's a Dostoevsky quote, it's like, all happy families are alike in their own way, but unhappy ones are different. Something like this, right? I think this applies here. Yeah, if the model is doing the correct thing, there is a systematic explanation. But if it's not, I don't know if we can offer an explanation. So it's kind of more valuable to figure out why it does something right, compared to why it does something wrong. I think possibly yes. Yeah. I can give one example from a more technical point of view. So normally my feeling is most errors are hard to explain. Even if we could interpret them perfectly, we would just find that the errors are a mess and we don't have an easy fix except train longer or have a bigger model. Maybe a model runs out of capacity to store all the facts you want to store, this kind of thing, right? But sometimes there are systematic explanations for failures. So one example is, of course, all models these days use the transformer architecture. A big problem with transformers is their quadratic in time costs. So the longer your sequence gets, the more and more time you have to wait for it to generate the next token. So some people have worked on trying to speed this up They have this line of research called state models It get a little technical but I keep it low So the early state models have been a thing for a long time in statistics But in the early 2020s, maybe like 2019 kind of stuff, people started applying it to language modeling, and they tried really hard to beat transformers because they're much faster state-space models. So if you can scale this up and they're as good as transformers, you would want to switch to them. But in practice, they found they're worse. They're always worse. And they couldn't figure out why. You train on the same data, so on. They're worse. In 2021-ish, there was a group at Stanford, Chris Ray's students. They noticed Anthropic's discovery on the Interp side of a mechanism for in-context learning in Transformers. So in Transformers, you prompt them with some stuff, and they can understand the context and use that to do things. Maybe you give examples of a task you're trying to do, but you don't know how to explain that task, so you just give examples. It figures out rules to predict that task. And they get better with more examples. This is a crazy finding from GPT-3 era. And so they were like, okay, why can't our state-space models do this? They tried running on the same task and they would just do bad. You know Anthropic gave an explanation from an Interp site There a certain attention head mechanism that these transformers learn when you give them at least two layers that enables them to solve this task It called induction heads So the state space model people at Stanford were like what if we figure out a way to put this exact mechanism inside the state space model to help it out learning you know in context learning And this paper is called Hungry Hungry Hapos. I don't know why it's called this. It's like H3. And they showed that if they just did this minor change, or it's like simple change, it's not a minor change. the model learned how to do in-context learning. And then all the later states-based models, you might have heard of Mamba or DeltaNet, they draw upon this advance. And now states-based models are used quite heavily. So if you look at Kimi or the BigQuin models recently, they have a hybrid of transformer attention and gated DeltaNet and stuff. Yeah, this is a clear example of like, there was a systematic reason why something was performing badly. and Interp helped us find it and fix it. Yeah, I would like there to be more stuff like this. I think it's possible there is some stuff like this. Yeah, then you can apply that to other models. Yeah, and if we have really good progress on Interp, then maybe we can find more instances of this. Maybe hallucination is not an instance of this but maybe it is Maybe there some fundamental issue in the architecture that makes it hallucinate Very interesting Cool Yeah I think this is a good segue for us to kind of get into the industry and you know what everyone is doing here with Interb Because this is, you know, as you said, like, I feel like it's still a pretty technical field of free server. It seems to have some pretty clear commercial values and applications as well. Let's just start with, you know, you are in academia, but you have, you know, a lot of industry experience as well. Are academics and industry folks approaching this kind of research and just this topic in general differently? Yeah, so I wouldn't say I have a ton of industry experience, but my main experience has been transluce. I haven't worked at a big frontier lab or anything, so everything I see is external. My feeling with interpability is there is more sharing of results and progress between academia and industry than other areas of AI research. I think because interp is still like, there's not some recipe that you just scale up to get better interp yet. And so it's basically like people need to come up with new ideas and like ideas are something that even a random grad student can contribute to, right? In other areas, it's like if you wanna do reinforcement learning research, [45:00] The nature of the research is quite different from academia versus industry because of the scale of resources being so different. Yeah, I would say in interoperability, it is still sort of a community across academia and industry. Like, you know, the workshops that happen, they have a mix of people from Anthropic and DeepMind, OpenAI, and also academics from like Stanford or, you know, David Bowe's group in Northwestern, L.A. Pavlik. There are all these professors who do interoperability research and have students who do interoperability research. Yeah, I wouldn't say that academic research is useless or something. It is quite important for progress and interpretability, I would say, even for the industry. Yeah, it makes sense. So when we apply that to model design, Goodfire, for example, their design philosophy kind of models with interpretability, which has drawn some criticism because I feel like it kind of equalizes capability. And then safety auditing, which is always a little controversial in the industry. But I guess just going back to the notion itself, what does it mean to design models with interoperability? Of course, I cannot speak fully for them, but they've articulated this intentional design philosophy. So any successful any useful technology is by nature dual use I think If we talk about nuclear power very helpful from a renewable energy standpoint but progress in that same science led to nuclear weapons AI research itself is like this, right? As we made progress on coding agents, this could be super helpful for, I don't know, all sorts of people who don't know how to code, but they have things they want to do in their life that no one has served a product or service for. and they can do it themselves now. But on the other hand, a random person has frontier-level cybersecurity skills and can hack into whatever. So I think if something's useful, it's going to be dual-use. And as a society, we have to figure that out. For interpretability, if we succeed in figuring out why a model behaves in a certain way or figuring out how to improve models systematically, of course this will be dual-use. like this will help us audit models but it might also help like a bad actor figure out how to remove alignment from model i think this is my nature going to be true uh that doesn't mean we shouldn't do it uh it means that like yes we must regulate or something or have you know a democratic decision making process that decides who gets access to this powerful technology i would say so far interp hasn progressed enough to be dangerous in this regard right i there still a lot of work to do before it reaches a point where we have to start worrying about this So I gave you one example of the state-space model improvement being driven by Interpability progress. There are maybe a few more examples like this, like attention, sync stuff, but I wouldn't say there are a ton. And this is because people have not, you know, Interp hasn't progressed enough to systematically give results in this way. I'm not too worried about it yet. I think there's still so much fundamental scientific work to do that, like, yeah, this is still kind of far. Yeah. And I think it's good to ground progress in like a real application. And one of those applications is making better models intentionally. Right. So. Yeah. And another pretty clear use case is in drug discoveries and, I guess, medicine, like DNA models and all that, right? Yeah, yeah. Goodfire has a large life sciences team that's trying to reverse engineer scientific discovery models, including DNA models and stuff. So, you know, it would be cool to figure out how language models work. It would be super exciting for linguists, I hope, to know how they process language. But this is cool. the real applications would be like you train models for climate or you train them to model the climate or you train them to do protein folding can we pull out human understandable algorithms from them that help us make scientific progress Because we also want to understand why they work, right? I think the scientific applications could be very big. Yeah, yeah, makes sense. Is the model that's more interpretable necessarily mean that it's safer? I don't think so. Yeah. Because if you know why it did something, you could also use it to exploit its behavior, right? Maybe you could figure out more easily how to jailbreak a model, you know, why it wasn't jailbroken in some by some prompt, right? But I do think transparency is like, just overall a net good thing. Yeah, it seems like bad from a societal point of view, if like, we trust our lives to these complex algorithms that we don't understand. You know, people are not comfortable, for instance, with social media stuff because, you know, there's these algorithms that serve videos that people like and so on. And it eats up a lot of time, maybe bad for children. And I think if, you know, companies were transparent about how they train these algorithms, how they work, why they recommend you certain things, it would at least help us decide as a society, you know, if we want them or not, or at least have more trust that no one's trying to do malicious. [50:00] I think it's similar for AI systems. A lot of people don't trust AI. like AI companies or like AI progress, they're quite worried about it. So transparency seems just like a good thing to have. Yeah, yeah. Is there, I mean, I guess this field is still like pretty early on. Has there been any progress that allows regulators to kind of start drafting policies already, like in that front? Yeah, I would say, I mean, it's a hope for interoperability research to provide stuff like this. I guess that's part of the reasoning behind nonprofit orgs like Transluce. Many other interoperability organizations, they want to provide tools for regulators to help figure out why models do certain things so they can better... control them. The AI Safety Institute in the UK, they do interoperability research. I would say there's still a lot to do. I don't think, yeah, we cannot confidently make new legislation based on stuff like this. Yeah. Another thing that came to mind, character training, is something that I've been looking into recently. We have all of those different language models and talk to us in a bit of a different way. way. And we mentioned Amanda Asko before, like, you know, we have all of those philosophers kind of teaching language models, how to do things, different principles on how they approach things. And I guess, consequentially also shaped their personalities, like the model's personalities. How are you thinking about like all of that? Where... you know, models are becoming more differentiated from each other. They're like kind of like, you know, different, different, different persons. I'll definitely say it's a really weird... that we're in now that this is a thing we discussed, right? That is weird. Like, yeah, language models have characters. It seems to be like true and there's evidence for it from interpretation point of view. Yeah, it's like kind of crazy. Yeah, I do think there's been interoperability work showing that the internal representations of different types of character are organized in some way and... representation space. There's an Anthropic at this assistant access thing. where helpful assistants lie in one direction and like, If you get misaligned assistance, they align the other. There is a potential for interoperability. If we can use interoperability to steer the training process somehow... It could be a way to like finely control character. Right. with more guarantees than we have right now where we just train on data. Especially as we do like AI, you know, like constitutional AI kind of stuff where an AI decides how to reward some behavior or not in the character training process. And this is something we would like to control, I think. And maybe interoperability in the training process has a role to play. Awesome. Thank you so much, Arman, for joining us for this podcast. Thanks so much for your time and looking forward to your research and what you ended up doing after your PhD. Thanks for having me. Thanks for having me. Appreciate it. All right. 以上就是我们这期视频播客的全部内容了. 大家的点赞,留言和转发是我们硅谷101做好深度科技和商业内容的最佳动力。 Remember to check our channel. I'm Yvonne, we'll see you next time. Thank you.