Sparse Autoencoders Fail to Explain Neural Networks, Researchers Find

研究发现稀疏自编码器无法有效解释神经网络 当前的可解释性工具会导致模型性能下降高达90%，且无法捕捉神经网络内部的非线性和几何计算结构。 本次研讨全面梳理了机制可解释性，重点探讨了五大核心实际应用：监控、修改、预测、改进 AI 模型，以及通过“Microscope AI”解构模型。研讨对比了逆向工程与基于概念的可解释性，分析了研究人员如何利用稀疏自编码器（SAE）分解网络以解决多义性与叠加问题。讨论重点指出了一系列关键技术瓶颈——例如高昂的性能对齐税、极端的算力需求、特征吸收以及线性假设失效，同时对论文的整体分类体系进行了评述。与会者还将逆向工程与基于概念的方法比照为无监督与有监督学习范式，并结合了 SAE Bench 等评估工具展开探讨。 机械可解释性（Mech Interp）主要涵盖五大实际应用领域：监控 AI 系统（白盒评估与异常检测）、修改模型（激活引导与知识编辑）、预测行为（木马与分布外检测）、优化模型架构与蒸馏，以及提取超人类认知洞见的 Microscope AI。 (0:00) 该领域划分为两大核心范式：逆向工程（自下而上、类似无监督地分解网络组件与回路交互）与基于概念的可解释性（自上而下、类似有监督地搜索人类已知概念），两者亦可通过在特征空间训练线性探针（如 SAE Bench）进行结合。 (13:41) 为解决单个神经元的多语义性（Polysemanticity）和特征叠加（Superposition），稀疏自编码器（SAE）成为主流解构工具，但面临极高的对齐税与信号损失；例如 OpenAI 在 GPT-4 上的 16M 潜在特征 SAE 使性能大幅倒退至仅相当于 10% 预训练数据的水平，且标准线性 SAE 难以捕捉时间或加法等任务中的圆周或螺旋非线性几何表征。 (21:42) 逆向工程需同时评估节点为何激活（通过梯度归因与特征合成）及其下游因果效应（通过激活打补丁追踪）；由于目前基于可解释性的工程干预效果仍不及传统微调，研究者提议引入类似生物学果蝇（Drosophila）的“模式生物”以安全、低成本地迭代假设验证。 (161:14)

Reviewing the survey by Lee Sharkey et al., the reading group argues that mechanistic interpretability offers essential white-box methods for monitoring, modifying, and predicting neural network behavior, but currently faces severe performance penalties, geometric blind spots, and verification bottlenecks. Practical Motivations Across AI Safety and Optimization (0:00) The discussion organizes the practical value of mechanistic interpretability into five operational domains. For monitoring, white-box evaluation seeks to uncover deceptive alignment and evaluation sandbagging—where models deliberately underperform—by tracking internal activations and refusal vectors rather than surface outputs. For modification and behavioral prediction, researchers target activation steering to fix shallow safety fine-tuning, diagnose out-of-distribution failures, and detect hidden Trojan triggers, with the long-term goal of extracting synthetic programs that define catastrophic boundary conditions. Beyond safety, interpretability provides structural levers for model optimization and discovery. Identifying specific functional sub-circuits enables targeted distillation and layer skipping, which prunes unneeded computation when tasks resolve early. Finally, under the 'Microscope AI' paradigm, analyzing internal representations could yield original domain knowledge, mirroring how systems like AlphaZero uncovered novel chess strategies or AlphaFold modeled complex biological rules. Sparse Autoencoder Bottlenecks and Geometric Inadequacies (21:42) To address polysemanticity and superposition—where networks compress more concepts than available activation dimensions—researchers widely rely on Sparse Autoencoders (SAEs) to map activations into high-dimensional, sparse latent spaces. However, the speaker notes that SAEs impose a steep alignment tax. In OpenAI's GPT-4 experiments, an SAE with 16 million latents degraded model performance to the level of a base model trained on only 10% of its data, while single-layer SAEs on GPT-2 caused performance drops between 10% and 40%. Full-scale SAE training can also match the compute cost of pretraining an entirely new model. Furthermore, the foundational assumption that concepts exist as linear directions is flawed. Standard SAEs fail to capture multi-dimensional geometries, such as circular representations of time or the spiral geometries identified by Max Tegmark and colleagues for numerical addition in language models. Optimization also introduces structural pathologies, including feature splitting into hyper-specific sub-concepts and feature absorption, where functional context (such as 'begins with') inadvertently attaches to specific concept latents (such as 'elephant'). Reverse Engineering Workflows and Causal Tracing (154:22) Reverse engineering decomposes a network into components, formulates mechanistic hypotheses, and validates them by assessing both upstream triggers and downstream effects. Upstream analysis combines natural dataset activations, gradient attribution, and synthetic feature generation to determine what activates an internal node. Downstream analysis then maps how that activation influences the network through direct logit lens projections, indirect effects via causal activation patching, and sequential propagation across multiple future token steps. Because conducting separate forward passes for activation patching is computationally expensive across large architectures, researchers frequently rely on gradient approximations. Combining upstream triggers with downstream effects enables end-to-end causal tracing, detailing how features interact to produce final outputs. If an explanatory model fails validation, researchers must adjust their hypotheses or alter their decomposition granularity. Empirical Verification and Interpretability Taxonomy Critiques (161:14) Validating mechanistic explanations requires rigorous empirical benchmarking, including predicting activations on held-out text, testing against adversarial failure modes, and hand-crafting minimal circuits from scratch. Because interpretability-based interventions currently remain uncompetitive with standard fine-tuning or model editing for practical bug-fixing and safety enforcement, the authors advocate studying simplified 'model organisms' analogous to Drosophila in biology to iteratively test tools on ground-truth mechanics before scaling to massive foundation models. During open discussion, participants questioned the paper's rigid conceptual division between reverse engineering and concept-based interpretability. Discussant Victor argued that both share the same fundamental goal, proposing instead that reverse engineering functions as unsupervised feature discovery while concept-based probing acts as supervised, hypothesis-driven feature search. Participants highlighted hybrid methods like SAE Bench, which trains linear probes over SAE feature representations, as evidence that the boundary between these two paradigms is fluid. The presentation highlights that while mechanistic interpretability provides indispensable tools for inspecting internal model mechanics, the field still lacks theoretical certainty over whether superposition and linear representations reflect ground truth or merely convenient empirical approximations. Scaling interpretability to practical engineering standards will require resolving the severe compute costs, architectural placement challenges, and information loss inherent in current dictionary learning techniques.

本期读书会围绕李·夏基（Lee Sharkey）等人的论文《机械可解释性的未决问题》展开，探讨了如何通过逆向工程构建白盒评估，以实现对人工智能模型的内部监控、行为预测与精准编辑；与会者同时强调，作为当前主流解构工具的稀疏自编码器（SAE）仍面临高昂的对齐税、特征吸收及无法捕捉高维非线性几何表征等理论与工程瓶颈。 实践应用：从黑盒监测迈向内部干预 (0:00) 传统评估方法将人工智能视为黑盒，仅依靠最终输出来衡量能力，这在检测模型欺骗或故意隐瞒实力的“沙袋现象”（sandbagging）时存在严重盲区。导读指出，机械可解释性致力于通过检查模型内部认知模式建立白盒监控，包括实时标记异常激活模式，以及通过定位表征层面的“拒绝向量”（refusal vectors）来指导更具针对性的红队测试。在模型修改层面，研究人员无需全量微调，即可利用激活导向（activation steering）和知识编辑直接修正模型能力，并尝试解释安全微调为何往往流于表面且容易被绕过。 此外，该领域还被寄予预测未见行为与结构优化的期望。研究者长远希望提炼出反映神经网络运作机理的“合成程序”，以明确界定灾难性故障或木马后门的触发边界；在工程改进上，定位特定任务的关键回路可实现跳层计算与针对性蒸馏。讨论中还将可解释性比作“显微镜”，类比 AlphaZero 发现全新国际象棋策略或 AlphaFold 揭示蛋白质折叠法则，探讨直接从高维表征中提取超越人类既有认知领域知识的潜力。 研究范式：无监督发现与概念驱动之辨 (13:41) 论文将机械可解释性划分为“逆向工程”（Reverse Engineering）与“基于概念的可解释性”（Concept-Based Interpretability）两大范式。逆向工程侧重于自底向上拆解组件并绘制其交互回路，以解释复杂涌现行为；基于概念的方法则从人类预定义的语义单元（如性别或语法成分）出发，在网络中定向搜寻对应的神经元或向量方向。 与会者 Victor 对此分类提出质疑，认为两者均服务于理解内部机制的终极目标，强行二分显得有些武断。讨论随后将这种分歧重塑为更为直观的“无监督与有监督”框架：逆向工程类似于无监督探索，在无预设前提下盲目提取权重中固化的全部特征；而概念法则属于有监督搜索，定向检索特定语义。读书会指出，两者已在实践中出现交叉融合，例如基准测试 SAE Bench 正是通过在稀疏特征空间上训练线性探针，评估提取特征的质量与语义对齐度。 稀疏自编码器的对齐税与表征缺陷 (21:42) 早期可解释性研究以单个人工神经元为分析单位，但神经元普遍存在响应多个无关概念的多语义性（polysemanticity），且传统的降维方法无法化解网络在有限维度内压缩海量概念的“叠加”（superposition）问题。当前主流学界依赖稀疏自编码器（SAE）将残差流或多层感知机（MLP）激活映射到极高维的稀疏潜在空间，以分离出单语义特征。然而，读书会着重列举了该方法的一系列严重缺陷，首先是重构带来的巨大“对齐税”与性能退化：OpenAI 在 GPT-4 上使用 1600 万潜在单元的实验表明，重建后模型的性能骤降至仅相当于用 10% 数据预训练的基础模型水准，意味着损失了 90% 的已学能力；即便在小型模型 GPT-2 上，单层 SAE 也会诱发 10% 至 40% 的性能损失。 此外，在庞大基座模型上逐层解构的计算成本极其高昂，甚至不亚于重新预训练一个新模型。在理论层面，SAE 依赖概念呈线性可读的假设，难以刻画网络内部非线性变换；在极端稀疏性约束下，模型还会产生“特征分裂”（单一概念分裂为过于狭隘的细分子特征）与“特征吸收”（例如广义前缀特征不当吸收特定概念特征，导致潜在表征携带多余语境冗余）。针对如何将字典学习从标准 Transformer 推广到诸如 Mamba 等状态空间架构，学界目前亦缺乏明确的接入准则。 因果验证：几何表征与模型有机体探索 (146:26) 在组件解构完成后，逆向工程需建立端到端的因果追踪（causal tracing）。这不仅需要分析输入端的激活原因（结合高激活数据集范例、梯度归因或人工合成最大激活样本），还必须测定下游影响。下游分析涵盖了通过 Logit Lens 评估直接输出效应、利用激活修补（activation patching）进行因果干预以分离间接效应，以及评估跨越多个后续 Token 的序列传播效应；为规避海量因果介入带来的计算开销，研究人员正探索利用梯度近似手段。 讨论进一步指出，真实模型往往依靠多维非线性几何空间来组织概念，例如将时间表示为圆周旋转，或在加法计算中呈现螺旋几何分布（如 Max Tegmark 团队的研究发现），这些连续流形根本无法被 SAE 的一维离散向量所还原。鉴于当前机械可解释性手段在代码纠错或安全干预等工程基准上仍远落后于传统微调，论文与读书会建议效仿生物学引入“模式生物”（如果蝇 *Drosophila*），优先在已知真值算法的小型合成网络中迭代假说验证工具，再逐步向大型模型扩展。 当前机械可解释性研究仍面临理论根基不明的困境，学界尚无法证实线性表征假说与叠加现象究竟代表了神经网络的真实运作机制，还是仅为便利研究的经验近似。面对事后分解方法在算力与性能保留上的瓶颈，将稀疏专家混合（MoE）或双线性激活等内生可解释性先验直接融入模型架构设计被视作破局方向；然而在形成成熟范式之前，开发更严谨的因果验证与多维几何表征工具仍是该领域的核心挑战。

[166:08] Victor: The division between reverse engineering and concept-based interpretability seems like a not super natural division to make... seems kind of arbitrary. [57:01] I was to some amount just talking right now to see if it will create some discussion, but I'm fine with leaving it here.

[0:00] if you're writing on the chat uh so if [0:02] you don't want to speak um but you want [0:05] to like [0:08] communicate you can write on the chat um [0:12] like I have a second [0:14] window that allows me to see the chat [0:17] and so I I would be able to see your [0:20] comments and I [0:23] think if Victor if you if if it's ready [0:27] I would [0:28] start uh should working now I think [0:32] okay uh I check the YouTube channel but [0:36] yeah okay yeah so I I did the [0:41] introduction yeah yeah I did the [0:43] introduction already [0:45] um um we are skipping so we are [0:49] not uh going through the paper by itself [0:54] I like I prepared the [0:56] presentation the paper starts like [0:59] within [1:00] introduction uh on on mechanistic [1:04] interpretability um which I think most [1:07] of you are aware and then it it talks [1:10] about the types of open problems and in [1:14] particular it it makes these five let's [1:18] say five broad [1:21] categories of practical [1:23] applications of me [1:25] typ [1:27] um it's [1:30] sorry I need [1:32] to there's something on my screen that I [1:34] cannot [1:35] see uh that you you can you can see but [1:39] I got uh sorry let me up the [1:46] [Music] [2:02] yeah let's do it [2:10] again [2:13] okay yeah and and so the the five broad [2:18] categories that that they Define as as [2:21] possible practical applications of [2:23] making turp are applications where you [2:25] try to to monit monitor AI [2:28] systems uh [2:30] for instance um like a a good broad [2:34] question that that would characterize [2:36] this type of applications is can you [2:38] monitor um the cognition patterns and [2:41] and detect dangerous Behavior Uh while [2:44] it is occurring um they give a couple of [2:49] examples uh in practice um the fact the [2:53] the current way that we do uh [2:56] evaluations for instance um is very [2:58] blackbox you don't look at the model [3:01] internals you just look at the for [3:05] instance the prediction of the model and [3:07] so it's it's hard to to evaluate if the [3:10] model is is is doing a deception or if [3:14] it's or if it's send bagging on the [3:17] evaluation and so one type of practical [3:20] applications would be can we make [3:22] evaluations a bit more white box uh and [3:25] look at at model internals to try to to [3:30] understand the the cognition patterns [3:32] that they're [3:33] doing uh they also [3:35] mention um trying to do anomaly [3:40] detection uh anomaly detection uh the [3:43] way they frame it and I think that's [3:45] normally the framing uh would would work [3:48] similarly to to like content moderation [3:51] so you'd be monitoring the internal of [3:55] the model and uh if some a kind of [3:59] pattern would emerge that's that's not [4:04] um a normal pattern that you have seen [4:07] uh that would flag the system and you [4:10] you you look better at at the model to [4:13] see what what triggered this [4:15] behavior and uh the third type of uh [4:20] monitoring application that they [4:22] mention is is the ability to to improve [4:27] red teing um [4:31] effects uh if if you're able to [4:35] understand better what leads to to [4:38] deception and and bagging and and have [4:41] anomality detection you can [4:44] probably develop [4:46] better um raming effects uh for instance [4:50] they they [4:52] mention uh the work on on refusal [4:55] vectors and using optimizing prompts [5:00] that minimize the refusal Direction [5:03] instead of uh optimizing prompts to to U [5:08] minimize refusal also works and so this [5:11] is a kind of um mechanistic [5:14] interpretability application that that [5:17] then improved uh red [5:19] teaming the [5:21] second uh big um pratical application is [5:26] is in modify AI where they talk about AC [5:30] steering uh machine and learning and any [5:33] in general um knowledge editing but also [5:38] uh fight [5:39] tuning uh it it it can help all [5:44] this uh mechanistic intility can can [5:47] help uh understand how to improve Uh [5:52] current methods of control and Inspire [5:55] others like the case of of activation [5:58] steering which was a a mechanistic [6:01] interpretability [6:02] work um and it can also try to to [6:07] understand more fundamental questions [6:09] and in particular the example that they [6:11] give about fine tuning is that we we've [6:15] seen that fine tuning especially for [6:17] instance refusal file fine tuning is [6:19] very [6:21] shallow uh and and metic [6:23] interpretability can can help us [6:26] understand why it is shallow and and if [6:29] there's [6:30] um ways [6:32] to to improve it and to to make um fine [6:36] tuning more more [6:38] robust the the third [6:41] point is about uh predicting AI Behavior [6:45] so can we can we predict uh for instance [6:48] out of distribution Behavior or future [6:51] capabilities that that a model can get [6:55] from um from training or F tuning [6:59] they [7:00] mention uh using the mech inter to to [7:04] identify troen models um or models that [7:08] have been in in such a in some way like [7:13] modified uh in a even way and try to to [7:15] use Mech inter to to find those but also [7:20] um understanding better adversarial [7:23] examples and why jail brakes [7:26] work um and [7:30] uh [7:31] in in in a a broad way uh one of the [7:36] like the interpret me inter dreams would [7:38] be to to be able to to make a a [7:41] synthetic [7:42] program that that reproduces Model [7:45] Behavior and this would be like the the [7:49] the the the best possible case of being [7:52] able to predict it the Model Behavior [7:54] because now you have a synthetic program [7:57] that's much easier to analyze than [8:00] than a large language model for instance [8:03] uh and You' be able [8:05] to to study all [8:09] the all the let's say Pro program [8:13] boundaries and and try to to understand [8:17] what inputs would lead to to [8:19] catastrophic [8:21] outputs uh they they also [8:25] mention that and and maybe this is a bit [8:28] of an area [8:30] at least on on on my end that I I don't [8:33] know so [8:34] much but uh trying to use making turp to [8:40] to look at emergence emergent Behavior [8:44] both on scale or on fine tuning so [8:46] understanding taking a [8:49] network that was unable to do a task and [8:53] then a network that is able to do a task [8:56] and maybe the only [8:58] difference there were end on the same [9:00] data but one of the networks is bigger [9:02] than the [9:03] other uh or or both of them are the same [9:07] size but one of one was fine-tuned and [9:10] the other wasn't and trying to [9:13] understand if if it's possible through [9:16] mechanistic interpretability to to [9:18] measure the difference between these [9:20] these networks and pinpoint better um [9:24] techniques that would allow you to to [9:27] predict um if if if a given Network can [9:30] have a certain um ability without [9:34] measuring that ability per se then the [9:38] fourth point that they mention is is [9:41] improve [9:42] AI sorry these names are are names that [9:45] I gave to try to to make them them all [9:49] all the same but [9:52] um the [9:53] [Music] [9:54] the the central question in the in the [9:57] box is closer to what they they they [10:01] talk about in the in the [10:04] text uh in improve AI they [10:07] mention um using MEC turp to to better [10:14] understand um how to do distillation of [10:18] models uh for instance by by correctly [10:22] locating uh some circuits you could [10:25] potentially make a smaller model that [10:27] has the same uh Behavior as the bigger [10:29] model that just on that specific task [10:33] and so you could make smaller models [10:37] um from bigger models um which would be [10:41] cheaper to to make inference [10:44] on and they they also talk about for [10:47] instance uh skipping layers because you [10:50] know the computation is almost [10:52] finished um and also designing better um [10:58] architect Lees if if you're able to [11:01] understand which kinds of components Mak [11:05] certain tasks um you could start [11:09] designing uh components that have those [11:11] inductive [11:12] biases uh items and so that that could [11:15] help you design better [11:17] models and then the the fifth type of [11:21] practical [11:23] application is what is is microscope AI [11:26] so uh using uh what language language [11:30] models or or any kind of model uh [11:33] learned about the [11:35] world uh and and by doing mechin you you [11:39] extract the knowledge that the model has [11:41] about the world and and hopefully you [11:45] learn something new uh one of the [11:48] examples that that they give and they [11:50] give several examples but I think this [11:52] one is the the one [11:55] that that maybe is easier to understand [12:00] is uh the fact that when when they first [12:06] trained um Alpha [12:10] zero they were able to teach some some [12:14] Grand [12:15] Masters uh new new things about Chess [12:19] that they then were able [12:21] to uh to to to to put into practice um [12:27] and and they have an example which I now [12:29] forget but I think in the example that [12:33] they give [12:35] um it it wasn't exactly [12:38] MEP um they just looked at the model [12:42] doing moves and and and analyze that but [12:46] in [12:47] principle uh for instance if you can do [12:50] uh inp on Alpha fold you could [12:54] potentially learn more about chemistry [12:57] and biology on on what the the proteins [13:00] fold uh the way they [13:03] fold um I don't know [13:06] if anyone has uh I I'm I I think I'm [13:10] going to stop on every slide on almost [13:13] all slides because they're they're very [13:14] dense and I I talk on them for for long [13:17] time so if anyone has any [13:19] question uh like specifically about [13:22] these practical [13:23] applications [13:25] um you you can do it now if if not I'll [13:29] go to the next [13:36] slide okay so [13:41] um the as I said the the the paper [13:46] broadly [13:48] splits meing tur into two kinds of of of [13:52] of ways to do Mak turp uh one of them [13:57] they the other call reverse Eng [14:00] ing uh where you take a [14:05] network and you try to be to [14:08] to to take it apart into smaller [14:13] components that don't have to be exactly [14:15] the components of the network they can [14:18] be some kind of abstract [14:20] component and then after taking it [14:23] apart uh you you try to describe the the [14:27] component interactions [14:29] that lead to to the behavior and so from [14:33] the network you kind of like [14:35] extract a [14:36] possible meanings that the network has [14:40] uh the other broad approach um is what [14:45] they call Concept based [14:47] interpretability and in that case you [14:51] you think about some some some concept [14:55] and you you look for them on the network [14:58] uh like like you're trying to [15:01] understand uh if networks understand um [15:05] the concept of gender for instance on [15:07] language or or PL plurals or adjectives [15:13] so you you set up uh in mind with with a [15:16] concept and then you try to to [15:19] identify uh which components in the [15:22] network uh produce a have the the the [15:26] roles um that produce the the kind of [15:29] behavior that you're looking [15:31] for uh and and we're going to be this [15:35] week we're going to be focusing on on [15:37] reverse [15:38] engineering and then next week uh we'll [15:42] talk about uh concept [15:44] based uh [15:47] interpretability the idea of of RSE [15:49] engineering as as I mentioned is that [15:52] you you want to be able to first um [15:57] decompose uh neural network [16:01] um the idea and and the picture is that [16:05] you you want to be able to take apart uh [16:08] the network at the joints uh such that [16:12] um if you know every piece of the [16:15] network and then you know the [16:17] interaction that happens at the joints [16:19] you should be able to to decompose the [16:22] network completely and explain [16:26] it um the [16:32] the dec composition part doesn't have to [16:35] be [16:37] um [16:39] across architecturally defined um points [16:45] so initially me and this is something [16:50] that we'll see also next slide uh [16:52] focused on on using neurons for [16:55] instance as the basis of the composition [16:59] um but you could you could think of of [17:03] doing the composition in a more course [17:05] grained way uh looking at full [17:08] components like the the MLP or ATT [17:10] tension at a given layer you can even be [17:12] more cross grained and consider full [17:16] layers uh or you can do something that's [17:20] not based on any given layer or any [17:23] given neon but a combination of [17:27] them and and the the way that they [17:32] Define reverse engineering mechanistic [17:36] interpretability is that first you do [17:38] this decomposition you describe each [17:41] part of the [17:44] decomposition um and then you you make [17:47] an [17:48] hypothesis on how this DEC [17:52] composition uh makes a [17:54] certain um makes the process that you're [17:58] trying to to [17:59] analyze and you have some tests you try [18:02] to validate these [18:04] hypothesis and and if you're able to to [18:07] validate the [18:09] hypothesis uh you're good you explain [18:11] the network if you weren't able to [18:14] validate the hypotheses you go back and [18:16] you try to to find either better [18:20] descriptions or different uh [18:25] decompositions um as as I was saying [18:30] uh [18:31] initially um one of the first tests was [18:35] to to use neurons [18:38] um as as the natural [18:41] decomposition uh but as maybe most of [18:45] you know U neurons uh are polysemantic [18:49] meaning that if you look at their [18:53] activations uh they activate in examples [18:57] um [18:59] that may not be not have a s a single [19:03] meaning for a human um and [19:08] so after that other other kind of of [19:12] techniques um started to develop the [19:17] other [19:18] mention using dimensionality [19:21] reduction um on on different parts of [19:24] the network uh but the the problem with [19:26] using dimensionality reduction [19:29] is that it it is not [19:32] uh useful to deal with superposition so [19:36] the idea with of superposition is that [19:38] um [19:39] networks uh want to represent Concepts [19:43] um with their param and in their [19:46] parameters but they want to represent [19:49] more uh Concepts than [19:52] Dimensions that they [19:54] have which means that uh some dimensions [19:59] uh can can can be more than one concept [20:03] and so when you do dimensionality [20:06] reduction uh you you will have always be [20:09] in super position and you you cannot [20:12] take these Concepts apart [20:15] and and because of [20:17] this recently people started working [20:22] with with sparse learning so uh with [20:25] sparse Auto encoders if you're familiar [20:27] with that the [20:30] idea is that you you go [20:33] from um a lower dimensional space let's [20:37] say the residual stream the output of [20:39] the MLP or or the output of ATT [20:42] tension and you train a a special simple [20:47] Network that um goes from that Dimension [20:50] to a much large larger [20:52] Dimension uh in a sparse way such that [20:55] you can identify um individ idual [20:58] Dimensions that are active and hopefully [21:02] these these intermediate [21:05] dimensions are [21:08] um are [21:10] interpretable [21:13] and because because of of like this [21:17] technique has has [21:20] seen a significant let's say [21:24] success um people everyone is doing it [21:29] uh there's there's multiple ways of [21:31] doing it you can do sp AO encoders or uh [21:35] transcoders or cross coders the authors [21:38] mentioned the different techniques uh [21:41] the [21:42] problem is that uh these [21:45] techniques [21:48] have there's multiple things that that [21:51] are not completely [21:53] explained one of them is uh that their [21:56] reconstruction is too high um they they [21:59] talk about how for [22:02] instance uh when when people from open [22:05] AI trained SP AO encoders on GPT [22:09] 4 uh the [22:11] Reconstruction uh of those passo [22:13] encoders even though they had 16 million [22:16] latent was the same as having trained [22:19] the model for only [22:21] 10% of of of their data so it's like [22:24] they're throwing away [22:26] 90% of their model [22:29] by by by using an sa a single sa on a [22:32] single [22:33] layer which is [22:35] obviously uh too much alignment tax uh [22:39] uh and even when you do this on smaller [22:41] models on on models like [22:44] gpt2 uh you you can get performance [22:47] drops of 10 to [22:50] 40% uh and [22:53] so developing new techniques that that [22:57] improves [22:59] um the the Reconstruction errors seem [23:03] seems to be something uh important if we [23:06] are uh if we want to continue using uh [23:10] sparse [23:12] mods uh another thing that they they [23:14] mention as a problem is the fact that uh [23:17] it's very expensive to train um sparse [23:21] dictionary um to do sp diary learning on [23:26] large models uh especially [23:29] if you want to have [23:31] a auto encoder or or coder that has a [23:36] large number of latent you may end up [23:42] um having a more computer intensive than [23:45] the original layer that you're training [23:47] training on and so if you want to do [23:49] something if you want to De compose [23:52] every component of of of the network you [23:55] end up having to train a new models [23:58] basically in terms of comput uh to to [24:01] explain uh the base model and and this [24:04] is obviously um not feasible for for [24:07] large models where companies may be [24:10] compute [24:12] constrainted uh they also mention the [24:16] fact that we are assuming um the the [24:20] linear representation hypothesis so we [24:22] are assuming that it's possible to [24:24] linearly read the concepts that the [24:27] model uses [24:29] and that this this have any sense uh [24:33] which is [24:34] obviously [24:37] um partially uh it it has to be [24:40] partially um just an hypothesis because [24:45] the model models are nonlinear and so um [24:49] and we keep finding ways that that [24:51] models represent Concepts that are a bit [24:54] more complicated than just the the the [24:56] linear representation hypothesis and so [24:59] it's possible that uh we are losing [25:01] something by by using Spar ditionary [25:04] learning the the other thing that they [25:09] discuss about is about sparity not being [25:13] the best proxy for [25:16] interpretability [25:18] um the the reason why spice dictionary [25:21] learning uh works or or the idea behind [25:25] it is that can go to to a higher [25:29] Dimension and then by making activation [25:32] sparse you can look individually at at [25:35] each activation and try to understand [25:37] what's [25:38] happening uh the problem is that as as [25:41] as you make models sparer and [25:44] sparer uh you start getting uh an effect [25:47] that's called feature splitting that [25:49] they represent here by originally having [25:53] just a direction for cuts uh but then [25:56] you start having a direction for person [25:58] cut and SS and Main Coons I think that's [26:02] what's written there um and also um you [26:06] start having feature absorption where [26:11] before you had [26:14] um a feature that was begins with e and [26:18] then a feature for elephant but because [26:20] you want to your constraining for [26:24] spess the model [26:26] learns uh uh feature for elephant and [26:30] the feature for begins with the except [26:32] elephant so it no longer activates on [26:35] elephant and and the Elephant feature [26:39] absorbs a bit of the begins with feature [26:43] and this just makes um latent less [26:46] interpretable because now uh what looks [26:49] like a an elephant feature can can bring [26:53] with it some some other [26:56] baggage they they mentioned that it's [26:57] not [26:58] straightforward to appli to apply spary [27:02] learning for all all models um for new [27:06] architectures in the future if people uh [27:09] start using mamba or start using [27:12] other [27:14] um other models we may need to to think [27:17] about where where [27:19] to where to train our SP out encoders it [27:23] even on Transformers it's not obvious uh [27:27] how you deal uh with attention ads uh [27:30] where do you train it and this is [27:32] something that they [27:34] mention um another problem uh is that [27:38] even though spary [27:40] learning uh describes [27:44] activations it doesn't [27:46] completely describe the mechanism that [27:48] the model is using so it's not looking [27:52] at at the weights of the model and it's [27:55] not understanding why why that input [27:59] gives that activation why that [28:02] activation leads to to a certain output [28:05] um and and [28:07] they they in this section they talk [28:09] about maybe looking at um weight [28:14] interpretability instead of just [28:16] activation [28:18] interpretability then [28:21] uh they I I skipped one well I skipped [28:26] two so uh the [28:29] related to to the linear representation [28:32] hypothesis uh is the fact [28:35] that Spar diary learning completely [28:38] ignores um that there could be some kind [28:42] of [28:43] geometry um in the in in the place where [28:47] where where the the model is being [28:49] trained and for [28:51] instance uh it has been [28:55] observed that uh models represent time [28:58] uh like in a circular way where you you [29:03] you [29:04] have a features that are more than [29:06] onedimensional that are two dimensional [29:09] and that that rotate and the model [29:11] represents January next to February next [29:14] to March uh with just some some angle [29:17] separating them and and current I want [29:21] to mention like this a recent paper by [29:24] uh maxman and you know his students [29:27] where [29:28] they found that also seems like numers [29:33] for addition are also representing some [29:35] kind of like it's also a circular thing [29:38] but it's like some spiral thing uh I [29:41] haven't read the paper yet but it's [29:43] something like that it's like related to [29:44] the to the modular addition example but [29:48] the modular addition thing was some like [29:50] very small models and this I think it [29:53] was actually L models uh and yeah that's [29:58] the kind of thing that you might not [30:01] miss since you are just doing SP col [30:03] although at the same time you will still [30:07] get like a a feature for each number you [30:09] just wouldn't get the thing you wouldn't [30:11] get is that the the features are [30:14] represented in some way that's related [30:15] to each other in a way that like [30:18] probably matters for the actual [30:20] computations but yeah yeah so so you'd [30:24] still you'd still get the direction for [30:26] January and still get the the direction [30:28] for February but uh without you having [30:32] to look at it specifically you would not [30:35] see that the direction for January and [30:37] the direction for February are related [30:40] by by a certain maybe there is some way [30:42] of like if you have a bunch of features [30:44] that you got from person col you can [30:47] like so there is maybe there is some [30:50] automated way of finding like if some of [30:53] them are related in some kind of [30:55] interesting geometry but I don't it's [30:58] not obvious that you can but maybe it is [31:00] possible to do [31:03] something um yeah and then then the the [31:07] next thing that let's say it's not the [31:11] last problem but another problem that [31:13] they mention is that it's possible that [31:17] in your training data or um just because [31:22] of inductive of of of the biases of [31:24] training the concept that you are [31:26] actually trying to look for is not [31:29] captured by the SA or or the other uh [31:33] dictionary learning model that you you [31:36] trained uh and you can think about [31:39] this in in a very simple way for [31:42] instance if you train on data that [31:45] doesn't talk about um [31:48] biology uh you will never learn any [31:51] concept related to [31:53] biology and so then if you if you're [31:56] trying to [31:59] to evaluate the model in some kind of [32:01] biology question your saay will not give [32:05] any any interesting results or or you [32:07] you'll not be able to use your [32:11] essay um they they finally [32:15] mention uh that we don't have like a [32:19] strong a the theoretical [32:23] bases uh for any of this so we still we [32:27] still aren't sure um if the the the [32:31] linear representation hypothesis is even [32:33] a good [32:35] approximation um we don't have we don't [32:39] know if if superposition is a concept [32:41] that that [32:43] actually uh makes sense or if it's just [32:46] um like an apparent thing that we that [32:49] we've been able to use and they describe [32:53] a couple of [32:54] different more U theoretical framework [32:59] um that that that should be pursued but [33:02] um that are still let's say in in in in [33:06] in infancy and uh it's not obvious how [33:09] how they relate with [33:11] interpretability and they end the [33:13] section [33:16] uh [33:18] mentioning that uh it probably makes [33:20] sense that you try to embed some of [33:23] these advantages of of dictionary [33:26] learning right into your models and make [33:29] them [33:30] interpretable uh for instance using uh [33:33] mixture of experts or using uh sparer [33:36] activation functions they talk about um [33:40] using bilinear um activations in in in [33:44] MLPs to make interpretation of them [33:48] easier uh uh and and they give a bunch [33:51] of examples um on on techniques that [33:54] would make models interpretable and and [33:57] this they present this as kind of an [34:00] alternative uh to the the problems of of [34:03] Spar sectionary [34:04] learning because hopefully if your if [34:07] your models are more interpretable then [34:09] you you don't need to to deal with uh [34:13] polic semanticity or or with um [34:18] superposition um after you decompose the [34:22] network you're obviously not done you [34:25] still need to describe what each part of [34:28] network does and you need to describe uh [34:32] the [34:33] interactions uh between the different [34:36] parts um they they divide this problem [34:39] into into two parts uh the first part is [34:45] understanding why a component activates [34:48] and then the second part is [34:51] understanding uh what the component does [34:54] uh in in terms of Downstream effects and [34:57] so [34:58] if you're able to to to do this for all [35:00] the components that you care about you [35:02] can do this kind [35:04] of causal [35:06] tracing um where if you want to explain [35:09] what this um model does you'll look at [35:13] all the inputs you you look at all [35:16] the all the outputs that are activated [35:20] by these inputs and then you look at all [35:22] the outputs of the node that you want [35:25] and finally you'll observe the the the [35:28] output effect uh and and if you're able [35:31] to explain all of these you you you can [35:35] describe uh what [35:37] um what this component is [35:40] doing uh they they start by talking [35:43] about [35:44] using ACT uh looking at activating [35:47] examples to to try to to explain the the [35:50] input of models uh but but and describe [35:54] some problems with using activating [35:57] examples [35:59] and then they [36:00] describe um attribution methods as well [36:05] uh that and and they give a list of of [36:08] several different attribution methods [36:10] most of them uh that are gradient [36:14] based and and they describe some some [36:16] problems with using attribution methods [36:19] and and now there are to interpret but [36:22] but mentioned [36:25] that efficient and accurate attribution [36:28] methods is is an open problem that's [36:31] that's very important if we want to be [36:32] able to to describe uh [36:35] components and then they they mention a [36:38] strategy that kind of combines these two [36:41] so looking at activating examples in [36:44] nature let's say um and and and and and [36:48] using attribution uh to make what they [36:52] call feature synthesis where you try [36:55] to produce artificially synthet [36:59] synthetize um highly activating data and [37:03] and through this process it's possible [37:05] that you that you learn [37:08] um about what what the component is [37:12] doing um but but some criticisms about [37:16] about doing this instead of just using a [37:19] natural data [37:21] sets [37:26] the having explained the input of your [37:30] component you you may want to explain [37:33] the downstream [37:34] effects they talk [37:36] about um three different ways that you [37:41] can learn yeah [37:44] really maybe H maybe B and I should head [37:48] first people want to head [37:50] over you you have your [37:55] microphone no sorry I'm going to mute [37:58] this person because they have the MH [38:02] sorry [38:06] um okay I was talking [38:11] about yeah the output uh behavior that [38:15] they talk about studying three different [38:17] effects one of them is the direct effect [38:20] if you if you know about the logic lens [38:23] the idea is that you can look at what [38:25] the output produces what the the [38:27] component produces and and try to expand [38:31] it as if it was already the end of the [38:34] network and look at at the effect and [38:36] they talk about of about different [38:38] variations of this and this is a direct [38:42] effect [38:44] because [38:48] uh you you [38:50] see what what the component does without [38:54] passing through the [38:56] network and that then they talk about [38:58] indirect effects and and especially [39:01] about uh doing causal interventions [39:04] where you you patch parts of the [39:08] network uh with different values of the [39:11] component and you try to [39:13] observe uh the kinds of paths that [39:18] that through which the information [39:20] passes to try to understand uh what the [39:24] component [39:25] does and they again they they talk about [39:29] different [39:30] methods um that are all of this type of [39:34] casual [39:36] interventions um [39:41] and they talk about some of the [39:45] drawbacks of these that for each [39:47] intervention you kind of need a forward [39:49] pass and if you have a lot of components [39:51] and if you have of of different things [39:54] that you want to explore um this may be [39:57] become very [39:59] expensive uh and instead they they [40:02] mention uh some kind of interventions [40:06] that that use gradients to to [40:09] approximate [40:12] um the the normal causal intervention uh [40:16] to try to better understand the [40:19] downstream effects and [40:22] finally they talk about looking at the [40:26] effect [40:28] um on on a sequential kind of way so [40:32] instead of just looking at the direct [40:34] effect so in in the case of of of [40:36] language models instead of looking just [40:38] at the next token you you see how that [40:41] effect propagates for for for [40:44] longer uh uh across multiple token [40:48] positions uh and they they talk about [40:52] how you could basically patch the [40:54] forward pass of a model into a different [40:57] part of the model so in like into a [40:59] future forward PA and try to to to [41:03] see what kind of influence that has in [41:07] in Model [41:09] Behavior finally the the the third step [41:14] of of reverse engineering is let's say [41:17] the fundamental step to to close the [41:21] process where you try [41:23] to um [41:27] see if if your hypothesis about the your [41:31] description of the different parts and [41:33] their interaction makes [41:35] sense they they give different ways [41:39] where where that you'd be able [41:42] to to to do this validation uh one of [41:46] them [41:47] is if you're able to predict the [41:51] activations um of of certain [41:55] parts uh you can then uh make [41:59] predictions and check if your [42:01] description [42:03] matches uh with with with what we [42:06] observe for instance uh in Sp diary [42:09] learning um you may have a [42:12] component so a certain feature on your [42:15] on your Spar Auto encoder that you think [42:18] activates on text about cats and so you [42:23] can if if you think if that's your [42:25] hypothesis you can through the model a [42:28] bunch of text about [42:29] cats uh and and see if indeed um [42:34] predicting that it activates on cat [42:37] actually predicts the behavior that you [42:39] observe on that data [42:41] set uh in a similar way they they talk [42:46] about how you could look at adversarial [42:50] examples and and failure [42:51] mode and understand [42:55] um if by by doing adversarial examples [42:59] uh that your [43:01] description still is still valid [43:06] um they talk about uh and [43:10] crafting a a network so if if you're [43:13] able to [43:14] describe uh the [43:16] circuits and all the components of the [43:18] circuits and how they interact you could [43:22] just make a Network that has that part [43:23] of C of the circuit and check if if if [43:27] the circuit is able to to perform a [43:30] certain task or if you if you have [43:32] access to the ground rot so if you're [43:34] able to to make a [43:36] um a network that uh by hand that you [43:40] know the algorithm that the network is [43:42] implementing uh you you could test out [43:47] uh some kind of [43:49] interpretability technique on that nwork [43:51] and if you find the real algorithm you [43:55] know that the technique uh is probably [43:58] um good and and and they [44:02] end they they end the the The Benchmark [44:05] on on the the hardest kinds of things [44:08] that you can [44:10] do which is you have a certain uh [44:14] engineering goal um like you want to to [44:18] make sure that uh that models uh refuse [44:23] or um you you want to understand why [44:27] models make a certain bug in code and [44:30] you you use your your description to to [44:36] fix that problem in a way that's [44:39] competitive uh they they talk about how [44:43] most interpretability models are still [44:45] far away from from being competitive to [44:49] other model other methods that that that [44:52] you may use um to fix a certain [44:55] engineering goal [44:58] uh finally they they talk about making [45:01] Ben benchmarks and uh [45:04] designing [45:06] um a parallel to to the dropil in in [45:10] biology if you don't know what the [45:11] dropil is is it it is a kind of it's an [45:15] organism uh that's very [45:17] simple and that people can if if they [45:22] want to understand some kind of [45:24] mechanism in biology in living things [45:27] they normally look at [45:29] dropa to kind of understand the the pro [45:33] the what kind of processes is it makes [45:36] and so they talk about the [45:38] possibility of of creating more um model [45:44] organisms uh that are made to have [45:47] certain behaviors and then you could [45:49] study them and and check if your [45:53] hypothesis generation Tool uh works [45:57] or [45:58] not and this [46:01] is this has been the the part on on [46:06] rever engineering [46:08] networks um the next three slides that I [46:12] have just about like these open problem [46:16] it's sorry it's more than three slides [46:18] it's six slides and [46:20] so I [46:23] think uh it would make sense if people [46:26] have questions either about what I [46:27] discussed or if they read the open [46:30] problems um that we can now talk about [46:35] them um I think in [46:38] general I mean the way that I [46:40] presented most of these problems that [46:43] are open uh I I I mentioned them um like [46:49] briefly [46:51] um and so if like if you if you have any [46:53] questions or if you think if you read [46:55] the problems and you think they were [46:57] interesting just say something otherwise [47:00] I don't think it's so interesting to [47:02] just to go through there's not that [47:05] much uh to [47:08] discuss I mean just if we have something [47:11] to discuss about them they make sense to [47:13] discuss but [47:15] otherwise um reading them doesn't bring [47:18] much [47:25] information e [48:20] um [48:28] I don't know if if [48:40] people if people don't have any [48:42] questions I think we can wrap it [48:50] up I guess I can comment something just [48:53] so it's not justos to say something uh [48:57] so the the division [49:00] between uh like the reverse engineering [49:03] and concept based interpretability seems [49:06] like a not [49:08] supernal like division to make seems [49:11] kind of arbitrary I guess it doesn't [49:13] completely not make sense but still [49:18] uh feels like the the things that they [49:23] put there is like there are some [49:25] techniques there is still more motivated [49:27] by by trying to understand the model [49:32] and yeah I don't [49:43] know I I also agree with you Victor it [49:46] seems like quite arbitrary in the sense [49:48] that um but rather if you then read the [49:52] description uh and the the text uh for [49:55] the figure you probably like have a [49:58] better understanding but yes the the the [50:00] drawings are not that clear to me at [50:03] least um and also because we like we are [50:06] hypothesizing that the um like the the [50:10] models learn something always based on [50:13] some kind of Concepts right otherwise it [50:15] wouldn't make any sense like try to to [50:18] interpret anything inside the models [50:20] right so it's like we are just starting [50:23] from that [50:25] hypothesis so like I guess you you can [50:28] think of it I guess it's not think about [50:30] it is not that natural because you can [50:32] think of it as uh the first kind of [50:36] thing is the like the reverse the thing [50:38] they call reverse engineering is like [50:40] something like UNS [50:42] supervis uh finding like trying to find [50:46] features in the model and the thing they [50:50] call [50:51] Concept is like half a human think of [50:54] specific things they want to look for in [50:57] the model and so I guess in that sense [51:00] like there is some distinction there [51:04] uh those seems like two different [51:07] methods to yeah I guess yeah are [51:10] different ways of doing things not [51:12] like way you can divide things up [51:18] but another thing I I thought when [51:22] reading this paper not this part of the [51:25] paper was that the I [51:31] they they have a section of the the the [51:34] thing that Gonzalo called like uh [51:37] practical problems or something like [51:40] that they call it yeah they applications [51:44] they don't call it that on the paper [51:45] they call it like types of problems of [51:48] problems seems like a weird like it's I [51:53] think it makes more sense to talk about [51:55] in terms of application [51:57] because these are basically things you [51:59] can use making their while like open [52:03] problems to me are like are more like [52:06] the kind of thing you will divide in uh [52:08] I feel like there are some more [52:10] divisions that you could do uh than just [52:12] whether they are reverse engineering or [52:14] not could talk about there are open [52:17] problems that are about how we divide [52:21] the how do we divide the modeling [52:24] features better something like maybe you [52:26] do better Sport and colors maybe you do [52:28] whatever the new Apollo research thing [52:31] is that I haven't read that paper yet [52:33] that's supposed to be like a replacement [52:35] for colors or you do whatever diality [52:39] reduction and a separate another kind of [52:42] problem is once you have some features [52:44] how do you do attribution of them or [52:47] like how do you find circuits of them [52:50] how they are connected and what the [52:51] features are used for uh also yeah what [52:55] when you have one feature this kind of [52:58] relat but not exactly the same what some [53:01] feature means in some sense which that's [53:05] where you get H some kind of problem [53:09] kind of techniques that you get with [53:13] h with you know Auto kind of when they [53:17] use chpt or whatever to generate the [53:19] planation [53:20] Foria and the related open ey paper so [53:25] and the paper to about all these things [53:27] it just feels like it's not organizing [53:31] it the way I will organize them which I [53:33] not that b problem is just kind of it's [53:37] not kind of it's not ideal but in the [53:41] end I I for what I say still like talks [53:44] about everything it should talk about [53:46] just in kind of weird [53:49] of yeah so about about like this [53:52] division between reverse engineering and [53:54] concept based I think like one of the [53:57] things that you [53:58] said is probably the best way to look at [54:01] it in like in terms of um supervised [54:05] versus not [54:06] supervised like yeah when you're doing [54:08] reverse engineering you're just trying [54:10] to find everything that the model has [54:13] while while you're doing concept based [54:15] which we didn't explore and and we'll [54:17] talk about next time uh their idea is [54:20] more of of supervised methods where you [54:22] do probing and stuff like that uh in [54:25] terms of [54:27] this thing that you were [54:29] saying um yeah I think I think the the [54:32] way that they thought about the the [54:34] paper was [54:35] more uh what are like open big open [54:38] questions like theoretical open [54:40] questions uh and in this section the [54:45] that section of of the theoretical open [54:47] questions they called [54:50] it uh open problems in methods and [54:55] Foundations and then they have an A next [54:58] section that's the the open problems the [55:00] Practical [55:01] applications I agree that it's it's a [55:05] bit like their types of op problem [55:08] section is a bit weird the way that it's [55:11] framed it's a bit [55:12] weird yeah no specifically it's frame [55:16] it's frame out [55:19] applications rather that about problems [55:22] in Mak in a sense I feel but yeah anyway [55:27] another thing like about that about the [55:33] the supervision and supervis thing I [55:35] feel like you could also like I you [55:37] there you could also mix the two like I [55:40] can I think it might make sense for [55:43] example I me thinking of trying this [55:46] maybe make sense for but like doing [55:49] proving on like the spone col like [55:53] features basis to see which feature [55:57] are encoding some human concept that you [56:00] want to find on the model and you just [56:04] Pros mix up with the spars Anders to [56:07] like trying to find which spars and [56:10] colod featur by you know correspond to [56:13] the thing you are looking for yeah the [56:18] like in in SE Benchmark they do [56:19] something like that where they use they [56:22] train probes on sa activations [56:26] on [56:28] the yeah because I think been thinking [56:32] of doing something like that for [56:34] the I mean you weren't the one that [56:37] introduced that someone else introduced [56:39] that but in in sa Benchmark they they [56:41] link to the correct [56:42] Source yeah I probably look at that yeah [56:46] I just thought of it and I I knew this [56:50] probably either it doesn't make sense [56:52] for some thought or someone else had [56:55] already done it so so I should just go [56:57] look at what other people have done yeah [57:01] so if if there's [57:02] nothing if there's nothing else because [57:06] we already arrived to our our Mark yeah [57:10] I would I don't have anything else to [57:12] say I mean I was in to some amount just [57:15] talking right now to see if it will [57:18] create some discussion [57:19] but uh I'm fine with leaving it [57:24] here okay so next next [57:26] next season next week we'll do the rest [57:31] of the [57:32] paper uh and then from then on [57:36] we'll we'll talk about on review papers