Interpretability Probes Risk False Positives, ETH Researchers Find

ETH研究人员发现：可解释性探针存在假阳性风险 分布式对齐搜索（Distributed Alignment Search）中不受约束的非线性对齐映射能够迫使任意神经网络匹配任意因果算法，这意味着除非严格限制映射容量，否则基于干预的可解释性测试将不再忠实可靠。 本内容探讨了因果抽象与分布式对齐搜索（DAS）是否足以识别神经网络中的内部表征。作者提出了一个核心理论定理，证明不受约束的非线性对齐映射可以轻易迫使任意因果图与任意网络相对齐。通过在合成任务和Pythia模型家族（IOI任务）上的实验，他们证明了高表达能力的非线性映射即使在未经训练、随机初始化的模型中也能提取出虚假的因果图。该讨论强调了一个根本性困境：线性映射可能因遗漏真实的非线性特征而产生假阴性，而非线性映射则面临严重的假阳性风险以及人为强行拼凑表征的问题。 论文提出核心理论定理指出，在极弱的假设条件下，若允许使用无约束的非线性对齐映射，任何目标因果图都可以被证明是任意神经网络的因果抽象，使得算法假设验证失去可证伪性。 (8:05) 在分层相等性任务与 Pythia 语言模型的实验中，线性映射的交换干预准确率（IIA）会如实反映模型能力，但高表达能力的非线性映射（如 RefNet）不仅在各层维持近 100% 的 IIA，在完全未经训练的随机模型上也达到了 80% 以上。 (10:36) 研究者阐明了“非线性表征困境”：严格限定在线性映射会因忽略周期性环状等真实非线性结构而产生假阴性，而采用非线性映射又极易通过过度拟合内部噪声“操纵”出虚假因果对齐。 (25:43) 针对分布式对齐搜索（DAS）的实际应用，作者建议默认使用受限的正交旋转映射以最大程度规避假阳性；若探索非线性表征，则必须配合严格的分布外（OOD）泛化测试与人工可视化审计。 (30:01)

Denis Sutter and Julian Minder demonstrate that causal abstraction frameworks like Distributed Alignment Search face a fundamental dilemma: unconstrained nonlinear alignment maps can trivially verify any arbitrary algorithm in any neural network, while restrictive linear maps risk missing genuine nonlinear representations. Theoretical Triviality of Unconstrained Causal Abstraction (8:05) The authors present an impossibility theorem establishing that, under mild assumptions, any causal graph can be proven to be an input-restricted distributed abstraction of any neural network. This construction requires only that the target causal graph map to at least one neuron per node, that inputs remain countable, and that the neural network layers remain injective and surjective across their domain. Because input-restricted interventions isolate latent states that uniquely identify specific inputs, an unconstrained nonlinear alignment map can construct intermediate target values that artificially produce the causal algorithm's expected output. As Julian Minder notes, without structural encoding assumptions, causal abstraction loses its falsifiability because researchers can mathematically force any network to match any target algorithm. Empirical False Positives in Untrained Models (10:36) Empirical evaluations on both synthetic and real-world architectures confirm that expressive alignment maps fabricate causal mechanisms out of random noise. In experiments using a three-layer Multi-Layer Perceptron trained on a Hierarchical Equality Task, linear alignment maps appropriately lost Interchange Intervention Accuracy (IIA) in deeper layers and closely tracked actual task learning. Conversely, nonlinear RefNet maps sustained near-100% IIA across all layers and achieved over 80% IIA on randomly initialized, untrained models. This failure mode replicated on large language models: when tested on the Indirect Object Identification task using the Pythia model family, nonlinear alignment maps applied to the residual stream achieved near-perfect IIA on untrained models incapable of solving the task. The authors argue this proves that high intervention accuracy under expressive maps reflects the probe's independent computational capacity rather than internal model mechanics. Geometric Nonlinearities and the Representation Dilemma (25:43) The central tension—termed the Non-Linear Representation Dilemma—arises because neural networks sometimes rely on non-planar representations, such as circular or helical manifolds identified in modular arithmetic and representations of days of the week. Restricting searches to orthogonal rotation maps reliably prevents false positives by disallowing the probe from computing features itself, but simple linear projections crush circular geometries by overlapping opposing points. Participants observed that standard interpretability methods like Sparse Autoencoders (SAEs) risk taking linear cross-sections of continuous geometric structures. This slicing can create misleading feature artifacts, generating visually compelling attribution graphs that are merely partial components or red herrings of larger nonlinear systems. Bounding Probe Capacity and Generalization Audits (33:17) To counter alignment gerrymandering without overlooking nonlinear encodings, Sutter outlines several methodological controls. For routine exploratory analysis, researchers are advised to default to restricted orthogonal rotation maps to minimize false positives. When nonlinear probes are unavoidable, practitioners must combine them with capacity bounds scaled to downstream layer depth, manual inspection, and out-of-distribution (OOD) generalization testing. Sutter notes that even general unconstrained linear transformations (matrix multiplications without orthogonality constraints) can exploit unintended features to fake alignment. In an empirical study tracking carry-over features in arithmetic, untangling whether an expressive transformation reflected genuine mechanisms versus spurious correlations required two months of dedicated manual auditing. The source concludes that interventional consistency alone does not guarantee feature faithfulness, as current mechanistic interpretability lacks an agreed-upon, ground-truth definition of what constitutes a faithful circuit. Resolving the dilemma will require bridging pragmatic, safety-oriented heuristic searches with formal theoretical boundaries, particularly through out-of-distribution validation and structurally bounded alignment algorithms.

Denis Sutter 与 Julian Minder 等研究者指出，若不对特征对齐映射施加严格的结构约束，因果抽象在理论上可将任意计算图平凡地映射至任意神经网络。实证测试表明，高表达能力的非线性映射甚至能在完全未经训练的随机模型中构建出虚假的因果对齐，暴露了机械可解释性研究在避免假阳性与捕捉真实非线性表征之间的根本两难。 无约束对齐导致因果抽象在理论上平凡化 (8:05) 因果抽象（Causal Abstraction）与分布式对齐搜索（DAS）旨在通过干预实验，检验神经网络内部是否忠实实现了某种高层因果算法。在形式化定义中，研究者寻找一个对齐映射 $\Phi$，将网络隐藏层激活投影到算法节点的潜空间。然而论文提出的核心定理表明：在输入空间可数且网络层具备单射与满射特性的极弱假设下，任意因果图都可以被证明是任意神经网络的输入受限分布式抽象。 该定理的构造逻辑在于，如果允许使用无约束的非线性对齐映射，连续的内部激活状态将保留原始输入的完整标识信息。映射函数利用这种单射性质，完全能够凭空重构出目标算法在各中间节点所需的数值，并借助满射性导向预期的输出分类。Julian Minder 强调，因果抽象原本承诺提供严格的算法假设检验，但一旦脱离了特征编码形式的具体约束，其数学验证机制便会失去可证伪性。 非线性映射在未训练模型上伪造算法结构 (10:36) 为了检验理论预测，研究人员在分层相等任务 $((x_1 == x_2) == (x_3 == x_4))$ 上评估了一个包含 3 个 60 维隐藏层的 MLP 模型。当使用受限的线性对齐映射时，交换干预准确率（IIA）随着网络层深入而逐渐衰减，且 IIA 的提升与模型实际训练表现紧密同步，在未训练模型上准确率接近 0%。相反，基于可逆非线性神经网络（RefNet）的对齐映射打破了层级衰减规律，不仅在所有隐藏层均取得接近 100% 的 IIA，甚至在完全随机初始化的未训练网络上达到了 80% 以上的干预准确率。 这一假阳性现象在真实大语言模型中同样得到了复现。研究者在 Pythia 模型家族上针对间接宾语识别（IOI）任务进行测试，对最终提示词标记的残差流执行干预。实验显示，正交旋转映射的表现与模型实际能力相符，在小模型或未训练模型中表现低迷；但高表达能力的非线性映射却能在完全不具备 IOI 任务能力的未训练 Pythia 模型中达成近乎完美的 IIA，证实强大的探针会直接从激活噪声中“杰利蝾螈式”（gerrymander）地拼接出虚假因果机制。 非线性表征两难与稀疏自编码器的几何切片 (25:43) 这一发现确立了机械可解释性中的“非线性表征两难”：严格限定在线性或正交旋转映射能够有效压制虚假对齐，但会因表达能力不足而产生假阴性；放开非线性映射虽能探索复杂几何，却极易诱发严重的假阳性。实证研究已证实神经网络并非纯粹线性，例如星期概念在表征空间中呈现闭合圆环，模块化算术任务则形成高维螺旋流形。简单的单维线性投影会将圆环两端重叠，从而彻底破坏其固有的关系拓扑。 研讨成员进一步指出，当前广泛依赖的稀疏自编码器（SAE）由于采用线性编码器与解码器，在面对连续或周期性几何时会强行进行“线性切片”。这种切片会制造出极具误导性的离散伪影，例如将所有以 5 结尾的数字归为同一方向特征，而这实际上只是高维连续螺旋流形上的孤立截面。研究人员在特征归因图中看到的清晰离散回路，很可能只是更庞大非线性几何投射出的“红鲱鱼”。 保真度定义缺陷与实操中的约束准则 (30:01) 讨论会反思了当前机械可解释性对“特征保真度”操作定义的循环论证缺陷：学界往往仅因干预能够改变模型输出就认定特征真实存在，但未训练模型上的假阳性反例推翻了这种因果充分性。Denis Sutter 指出，即便是不加正交约束的普通线性矩阵乘法，也可能通过利用旁路关联特征来作弊；在多位数加法进位特征的研究中，团队曾耗费整整两个月的人工审查才厘清复杂映射究竟学到了什么。 针对 DAS 方法的应用实践，Sutter 建议：在初步探索阶段，研究者应默认采用正交旋转映射，以极低的假阳性风险作为安全基线；若必须探查非线性表征，则必须将对齐映射的计算容量限制在干预层之后剩余网络深度之内，并配合人工可视化审查。此外，团队发现尽管理论证明未提供泛化保证，但经训练的有效对齐映射在未见数据上表现出意外良好的干预效果，因此极端分布外（OOD）泛化测试是筛除非物理拟合伪影的关键手段。 本场讨论揭示了因果抽象从理论到实证层面的局限性。当前技术仍无法断言大语言模型在执行多位数加法时是在运行严格的分步算法，还是依赖分布式启发式记忆与局部子网络投票。可解释性学界在实用主义干预与基础理论严密性之间存在分歧，若无法对探针计算容量建立原则性限制并在全分布空间进行验证，因果干预指标便无法作为证明模型内部机制的充分证据。

[22:06] Julian Minder: If you don't make assumptions about how things are encoded, then this breaks, because you can show that any algorithm is implemented in any network. [30:01] Denis Sutter: If you just want to get a short view on it, use rotation maps. You might get some false negatives, but at least it's very unlikely that you get false positives. [50:34] Denis Sutter: Part of the problem is actually figuring out what it means for a feature to be faithful... I don't think there's like a ground truth definition of a feature being faithful. [0:00] Denis Sutter: The Non-Linear Representation Dilemma: Is Causal Abstraction Enough for Mechanistic Interpretability? [62:29] Participant: it's not like I've been that productive on the other thing to be honest.

[0:00] Wait. Yeah, you're going to start when [0:02] you want. [0:04] >> Sure. Um, you can see my screen, right? [0:08] >> Yes. [0:10] Give me one moment. Um, [0:14] >> it's also on the on YouTube. Uh, [0:18] >> okay. [0:19] >> Um, I think I'm ready. [0:24] >> Perfect. [0:26] >> So, um, can I start, right? [0:29] >> Yes. DC. [0:31] >> Great. Okay. Welcome to my presentation [0:34] about our paper. In this work, I will [0:36] present the nonlinear representation [0:38] dilemma. Is coal abstruction enough for [0:40] mechanistic interpretability? Please [0:42] feel free to ask questions anytime [0:44] during the presentation and do not [0:45] hesitate to interrupt me. [0:49] I will begin with a brief introduction [0:51] on on causal obstruction. [0:58] Great. Let us start by having some task [1:01] for which we present the zoomed [1:02] underlying algorithm as a directed [1:04] asyclic causal graph. To make this more [1:07] intuitive, consider the hierarchical [1:09] equality task as example. In this task, [1:12] four input numbers are provided. The [1:14] first two and the second two numbers are [1:16] each tested for equality and the results [1:18] of these two tests are then compared for [1:20] equality themselves. [1:22] For example, consider the input 1 9 6 [1:25] and six. The first pair one and nine are [1:28] not equal yielding false while the [1:30] second pair six and six are equal [1:32] yielding true. Comparing these two [1:34] equality results false and true gives [1:36] the final output false since they are [1:39] not equal. [1:40] An intervention on this causal graph [1:42] could for instance involve severing the [1:44] incoming connections to the first [1:45] equality node and manually setting its [1:48] value to true. [1:51] Now let us discuss what a tow [1:53] abstraction is. Suppose we have assumed [1:55] algorithm represented as a causal graph. [1:58] In addition, consider a neural network [2:00] for which we want to determine whether [2:02] the algorithm is a t abstraction of that [2:04] network. [2:05] Let fn denote the function implemented [2:07] by the neural network and fa the [2:09] function implemented by the algorithm. [2:12] Now assume an input x. For a tow [2:15] obstruction, there must exist a [2:17] subjective function to x that translates [2:19] the input of the neural network into the [2:22] corresponding input for the algorithm. [2:25] Causal obstruction defines the equality [2:26] between a neural network and algorithm [2:28] in terms of interventions. Let us [2:31] therefore assume perform an intervention [2:33] on the neural network as illustrated [2:35] here. We now require the existence of a [2:38] function that transforms interventions [2:40] on the neural network into corresponding [2:42] interventions on the coal graph. [2:45] Let us denote the inner node as s in [2:50] and the inner nodes of the cos graph as [2:52] ea in. [2:54] Finally, we require a subjective mapping [2:56] towel that aligns the values of s in [2:58] with those of eta int. [3:01] Now that all key terms have been [3:03] introduced, we can express the condition [3:05] required for coal graph to constitute a [3:08] tow abstraction of the network. This [3:10] means that for all inputs X and for [3:12] interventions given the given [3:13] intervention set, performing an [3:15] intervention on the neural network and [3:17] mapping the resulting inner nodes values [3:19] to the coal graph must yield the same [3:21] values for the inner nodes as first [3:24] translating the inputs to the coal [3:26] graphs input and then applying the [3:28] corresponding mapped intervention on the [3:30] coal graph. [3:32] You can now define [3:36] which holds when this equivalence [3:38] applies for all possible interventions [3:39] on the algorithm. So now note that in [3:42] principle any type of intervention on [3:44] the neural network is permissible. [3:48] To refine this we introduce the concept [3:50] of constructive abstraction. This is a [3:53] tow this is a strong tow abstraction in [3:55] which each node in the cosal graph is [3:58] assigned a distinct non-over overlapping [4:00] set of neurons as illustrated here. [4:03] This constraint modifies the required [4:05] mapping function such that each neuron [4:07] is assoc associated exclusively with a [4:10] designated neuron set. [4:13] The way in which we map nodes to neurons [4:16] in the DNN implies different assumptions [4:18] about how the network encodes [4:20] information. In this example where a set [4:22] of neurons corresponds to a single node, [4:25] we assume the privilege basis hypothesis [4:29] under which information in the DNN is [4:31] encoded within subspaces aligned with [4:33] individual neurons. Alternatively, there [4:36] is a linear representation assumption [4:40] posits that hidden neuron dimensions [4:42] need not directly represent concepts. [4:45] Instead, concepts corresponds to [4:46] dimensions within a linear [4:47] transformation. [4:49] To implement this within our causal [4:52] abstraction framework, we obtain we [4:54] first apply a reversible linear map [4:56] referred to as the lineup map denoted [4:58] five. This allows us to obtain a latent [5:01] space. [5:03] Those dimensions can then be assigned to [5:05] specific variables. Once these [5:07] dimensions are defined, we can intervene [5:09] on them directly by manipulating the [5:11] corresponding latent dimensions. [5:13] Finally, we map the modified latent [5:15] representations back into the model's [5:17] hidden space using the inverse of the [5:19] alignment map and continue the forward [5:22] path of the network to obtain the final [5:24] output. [5:26] Under the privileged basis hypothesis, [5:28] the alignment map V is a permutation of [5:31] the identity matrix which implies that [5:33] individual neurons themselves can [5:35] represent distinct concepts. In [5:37] contrast, the linear representation [5:38] hypothesis assumes that concepts are [5:40] encoded within linear subspaces. [5:43] For a nonlinear representation [5:45] hypothesis, the alignment map FE is a [5:47] reversible nonlinear function such a [5:49] such as a reversible neuronet network, [5:52] for example, a refet reflecting that [5:55] reflecting that there is no inherent [5:57] reason to expect features to be linearly [5:59] encoded. [6:01] Building on this we introduce [6:03] distributed abstraction which is a [6:05] strong tower obstruction in which the [6:07] nodes of the coal graph are mapped to [6:08] non-over overlapping partitions of the [6:10] latent space dimensions produced by the [6:12] alignment map fee. So how do we find [6:15] this distributed alignment maps in [6:17] practice? [6:19] The most prominent method is called [6:20] distributed alignment search or short [6:22] does. Let's start with a data set sample [6:25] that includes a base input along with [6:28] two source inputs that define the [6:30] interventions. Each source input [6:32] corresponds to a separate run both in [6:34] the algorithm and the neural network. [6:36] For the base input, we run the algorithm [6:39] and network up to an intervention layer. [6:41] At that layer, we intervene then [6:43] continue running both systems to [6:45] completion where we calculate a [6:47] similarity loss. for example, a ne [6:49] negative lock probability and apply back [6:51] propagation to train the alignment map [6:53] fee by keeping everything else frozen. [6:57] In some samples, we have only one order [7:00] of the source input, which means that we [7:02] intervene on only a subset of the [7:04] variables. [7:06] To measure how well the algorithm and [7:07] the neuronet network align under the map [7:09] fee, we use interchange intervention [7:11] accuracy with which measures the [7:13] percentage of matching behavior between [7:15] them under intervention. [7:18] We can now introduce our final concept. [7:20] The input restricted distributed [7:22] abstraction. This refers to a dist [7:24] distributed abstraction composed [7:26] entirely of input restricted [7:28] interventions [7:29] where each intervention can be [7:31] constructed from values of other input [7:33] restricted interventions. An example is [7:36] shown here where where we replace [7:38] certain values with those from another [7:40] input. On a high level, if an algorithm [7:42] is an input restricted distributed [7:44] abstraction of a network, there exists a [7:46] reachable latent space with some [7:48] bjective alignment map where the [7:50] different dimensions are assigned to [7:51] only one or the or non-variable behaving [7:54] under input reachable intervention [7:56] values as expected under the causal [7:58] graph. It can be seen as a formalization [8:01] of distributed alignment search. [8:05] That concludes the background on coal [8:07] abstraction. Let's now move to the [8:08] nonlinear representation dilemma. The [8:11] main theorem shown in my in our paper [8:14] states that under specific assumptions [8:16] any cosal graph can be shown to be a [8:18] input restricted distributed abstraction [8:20] of any neural network. [8:22] Now let me give the assumption required [8:24] for this theorem. Aligning any cosal [8:26] graph with any neural neural network [8:29] using some feed to get a latent space [8:31] from input restricted distributed [8:33] abstraction. [8:35] We assume that each node in the causal [8:37] graph can be associated with at least [8:39] one neuron in the in the model. [8:41] Furthermore, both the algorithm and the [8:43] neural network are required to solve the [8:45] given task and to operate over countable [8:48] inputs. [8:50] Finally, the neural network must be both [8:52] injective and subjective. [8:56] To provide some intuition for how we [8:58] prove the theorem, let us first fix the [9:00] mapping from the input nodes to the [9:03] input neurons as the identity function. [9:07] For the mapping from the output neurons [9:09] to nodes, we use arcmax. [9:12] We then only need to demonstrate the [9:14] existence of an analignment map fee such [9:17] that the existence such that there [9:20] exists a t to prime between the [9:22] remaining neurons and nodes while [9:24] preserving the correct output for the [9:25] argmax mapping. [9:28] To achieve this we first leverage the [9:29] countable inputs to define fee as inject [9:33] as injective in each dimension. [9:36] We now consider an input restricted [9:38] intervention and account for the input [9:40] injectivity. [9:42] We then observe both nodes in the latent [9:44] space can identify which input caused [9:46] their activation. [9:48] This enables us to define a suitable to [9:51] prime using this information to [9:53] determine the corresponding values in [9:55] the causal graph. [9:57] If we now also leverage the output [9:59] subjectivity, we now know that there [10:01] exists points in the hidden space that [10:03] produce the correct argmax when passed [10:05] through the remainder of the neural [10:06] network. By combining this with our [10:09] knowledge of which inputs generated [10:11] these latent space points, we can define [10:13] the inverse of y accordingly. [10:17] This results in the required model [10:18] outputs for argmax. We note that we only [10:21] showed it here for one layer, but this [10:23] can be extended to multiple layers by [10:25] induction. We can assume for the [10:27] previous layers these properties and [10:29] then proved that interventions work the [10:32] same in the DNN and algorithm layer by [10:34] layer. [10:36] We now evaluated this in this [10:38] theoretical finding in practice. For the [10:41] algorithm we use the previously [10:42] presented hierarchical equality causal [10:44] graph and test for two equality [10:47] variables it inner one and two. [10:50] For the neural network, we employ a [10:52] multi-layer perceptron with a 16 [10:55] dimensional input followed by three [10:57] hidden layers with 60dimensional hidden [10:59] spaces and a two-dimensional output. [11:01] Train it on this task. [11:04] Each node corresponds to a group of four [11:06] neurons in the neural networks input [11:08] layer. [11:10] You then train fi with dust to extract a [11:13] suitable latent space of each of those [11:14] hidden layers and used inter and used [11:17] intervention sizes [11:20] to finally compute the resulting IIA [11:23] by assigning the first eight dimensions [11:25] to the first inner node and the [11:27] remaining eight to the second inner node [11:28] for interventions. We observe similar [11:31] IAS across the three different used [11:34] alignment maps. [11:36] Intervening on only one only two nodes [11:38] for each variable already reveals bigger [11:41] differences where the weaker alignment [11:43] map based on privileged basis assumption [11:46] no longer performs that well. [11:48] Intervening on only a single dimension [11:50] per variable shows performance losses in [11:52] all alignment maps except for the [11:54] strongest, the nonlinear alignment map. [11:57] We now perform the same evaluation [11:59] across the remaining two hint layers to [12:01] gain our first K insight using linear [12:04] alignment maps. The IIA decreases with [12:06] each layer suggesting that the two inner [12:08] variables are most likely encoded in the [12:10] earlier layers. However, this pattern [12:13] does not hold for the nonlinear [12:14] alignment map which shows nearly perfect [12:16] alignment across all layers [12:18] contradicting the earlier intuition. [12:21] In a second experiment, we examine how [12:24] the IIA evolves during training. As [12:27] usual, the DNN's performance increases [12:29] over time. When applying the linear [12:32] alignment map trained with DAS, we [12:34] observe that IIA closely tracks the [12:36] DNN's accuracy. [12:38] However, using Refnets as alignment maps [12:40] trained with DA with increasing [12:43] complexity, this correspondence no [12:45] longer holds. Most strikingly with the [12:48] most complex refetment map we observe [12:51] over 80% IIIA even when the models is [12:55] randomly initialized and untrained. This [12:57] demonstrate that a high IIA which would [13:00] normally suggest that the algorithm is [13:01] an approximate input restricted [13:03] distributed abstraction of the model can [13:05] be misleading as a randomly initialized [13:08] model has not actually learned any [13:10] algorithm. [13:12] We then present in our paper experiments [13:14] on large language models. For this we [13:16] use the task of indirect object [13:18] identification. [13:21] For this task we apply the following [13:23] algorithm. We begin with prompt such as [13:26] friends Joanna and Christy found a mango [13:28] at the bar. Joanna Gy 2. We then can [13:31] assign labels. For example, Joanna is a [13:33] and Christian B. This allows for two [13:36] possible completed sentence structures [13:38] abab or abba. [13:40] In our example we have an abab sentence. [13:44] This information then can be used to [13:45] determine the correct output which in [13:48] this case is Christie. An example for an [13:51] ABA sentence would be obtained by [13:53] swapping the last Janna with Christie in [13:54] the prompt. [13:57] We now evaluate whether the transformer [13:59] model family pia aligns with this [14:01] algorithm. Transformers take a string as [14:04] input, tokenizes it, passes it through [14:07] an embedding layer, then through several [14:09] transformer blocks and finally applies [14:12] an embedding to produce the output. [14:16] For the experiments, we apply an [14:17] alignment map to the residual stream of [14:20] the last token. [14:22] Examining the IOI task performing across [14:25] uh examining the IOI task performance [14:28] across different pay models sizes. We [14:30] observed that even in trained the two [14:32] smallest models fail to solve the task. [14:36] When randomly initialized, none of the [14:37] models can solve the task. Using a [14:40] rotation map as the alignment map and [14:43] intervening on the first half of the [14:44] latent space and trained by DA, we find [14:47] that the resulting IIA again tracks the [14:49] performance of the model. This pattern [14:52] again does not hold for the nonlinear [14:53] alignment maps. Even though both the [14:56] trained and untrained versions of the [14:57] two smallest PIA models fail to perform [14:59] the task, we still observe nearperfect [15:01] IIA when using nonlinear alignment maps. [15:07] Evaluating this on the train on the [15:09] training process of a PIA model, we see [15:11] behavior similar to the previous [15:13] hierarchical equality task where near [15:16] perfect alignment is found even at [15:17] random initialization. [15:20] Therefore, we see that causal [15:21] obstruction alone is not sufficient. [15:25] It is crucial to balance the complexity [15:26] of the alignment map with the observed [15:28] IIA. Highly capable alignment maps can [15:31] produce near perfect accuracy even if [15:34] the evaluated not actually implemented [15:36] in the model. An overly weak alignment [15:39] maps may fail to detect the encoded [15:40] features. [15:43] He knows that our theoretical proof [15:45] effectively effectively corresponds to a [15:48] strong overfitting to testing the [15:51] generalization of discovered alignment [15:53] maps is essential to detect [15:54] gerrymandered alignment maps which we [15:56] exploited in orarium. [15:59] Looking forward it will be important to [16:01] continue investigating how feature [16:03] representations are encoded in order to [16:06] avoid using alignment maps that are too [16:07] weak or too strong. [16:10] Now to conclude this presentation, [16:12] we are able to answer the central [16:14] question of our paper. Is coal [16:16] abstraction enough for mechan for [16:17] mechanistic interpretability? [16:20] While promising current causal [16:22] abstraction is not enough, multiple [16:23] weaknesses need to be addressed in [16:25] future work. [16:27] That concludes my presentation. I hope [16:29] you were able to follow along with at [16:31] least parts of it. And I'm now happy to [16:33] take any questions. [16:56] Yeah, I guess if no one is asking [16:58] questions, I can say something. Yeah, I [17:01] I don't think I completely follow all of [17:04] it during for only the presentation, but [17:07] I do think I got some idea what this is [17:10] about. [17:11] uh [17:15] and [17:17] like the the first kind of the the [17:19] theoretical [17:21] result I I I also didn't completely [17:24] understand it but I it seems like the [17:27] kind of thing where uh [17:32] you can make some theoretical result but [17:35] about that in practic and practice there [17:38] is some more structure to the [17:40] abstractions where the I guess that's [17:43] partly what you were saying [17:46] and like [17:48] I I I like I feel like I have [17:52] I have some skepticisms of that kind of [17:54] theoretical result uh in general and the [17:59] the empirical thing where you show that [18:03] the [18:05] in tactics you can find with some [18:08] noncompletely arbit arbitrary uh [18:11] nonlinear thing you can find fake [18:15] representations on actual networks is [18:18] more like compelling [18:22] >> I think yeah Dennis go ahead [18:26] >> um what you if I understand you [18:29] correctly um the theoretical part [18:33] actually is something which is somehow [18:36] relying on some cherry mandered um [18:38] alignment maps which in practice for [18:41] example distributed alignment search um [18:44] does train on one set and then test on [18:46] other set. So um our our actually does [18:52] not show that it will work on some [18:54] untrained data for example it is just [18:57] showing that we can make and therefore [19:00] it's more an existence proof. So for any [19:02] kind of data we have um we can show that [19:06] there exists some alignment map which [19:09] >> which says that this is this uh that [19:12] this causal graph is actually [19:13] implemented by the model even though it [19:16] it it does not need to be implemented [19:19] but um the empirical experiments [19:21] actually show us which is trained on [19:23] some um the train data and tested on [19:26] some test data. So our pharm actually [19:29] does not really get any doesn't give [19:31] great um guarantees in that setting that [19:34] we actually find something which will [19:36] generates to test data and we actually [19:38] also were quite surprised that that we [19:41] can train on one set and test it on [19:43] another set and there's actually then um [19:45] it works. So the alignment map still [19:49] results in um interventions which are [19:52] behaving as we expect to even though [19:55] these data points are never seen and [19:57] that is actually something which we [19:58] followed up with some future research. [20:02] [clears throat] [20:03] >> Also maybe Victor to kind of go into [20:06] this a bit more. I think so why the [20:09] theoretical part matters a lot is [20:11] because causal abstraction or [20:14] distributed alignment search is very [20:16] much built on a set of theory and um we [20:21] basically show that this theory breaks [20:24] down if you don't make assumptions about [20:27] your um abstraction map which in a very [20:31] high level intuitive thing is just a [20:33] thing that maps from the network to your [20:36] causal graph. Right? Distribute [20:38] alignment search is basically you have [20:40] you you think oh my model implements [20:43] this algorithm in this causal graph and [20:46] then you want to see whether you can [20:47] find a map from the network to your [20:49] causal graph that actually correlates w [20:52] with this [20:53] >> and we show that [20:55] >> sorry uh [20:56] >> yeah go ahead [20:58] >> finish I was the thing I'm wondering is [21:01] more is this how much this is just the [21:05] kind of result Like you can have a there [21:08] is there are infinite models that you [21:10] can show that there are infinite models [21:12] that for some input like for some set of [21:17] uh inputs and outputs have the same [21:21] infinite models that have the same [21:22] result. But in practice there are some [21:26] reason like things like neural networks [21:28] works because the the actual [21:32] things you can have in real life have [21:35] some structure where you can actually [21:37] have some assumptions that they are some [21:41] they are simple or something and that [21:44] works. I I I wondering how much this is [21:46] the analog results where you can show [21:50] that there is always some graph that is [21:52] equivalent but maybe the the graphs turn [21:56] out to be like the when you are having [22:00] pattern solutions the graphs that are [22:01] that are equivalent are super [22:03] complicated or something. [22:06] So, so I think the point here is not [22:08] that you can always find a graph that is [22:10] equivalent to the network but the other [22:12] way around actually you the point is [22:15] that any graph that you choose can be [22:18] shown to be equivalent to the network [22:21] which is right this is what causal [22:23] abstraction or distributed alignment [22:25] search uh kind of promises right it [22:28] promises that if you claim I think my [22:31] network is doing this and then I apply [22:34] causal abstraction theory and distribute [22:36] alignment search to evaluate whether my [22:38] network actually does this. That's what [22:41] it does does. But we act we show that if [22:45] you don't make assumptions about how [22:47] things are encoded [22:50] um then this breaks because you can you [22:53] can show that any algorithm is [22:55] implemented in any network. So it's kind [22:58] of the other way around and and and [23:02] this matters for the causal abstraction [23:04] theory because this is what it promised [23:06] initially. It promised to give you a [23:08] theory that if you have a graph it can [23:12] tell you whether it's implemented in [23:14] network or not and so far people have [23:18] just always assumed that everything is [23:20] encoded linearly but now more recently [23:22] this has been a bit more in discussion. [23:24] I I personally I think most things are [23:27] actually encoded linearly but it's just [23:30] we need to we need to discuss how things [23:33] is how things are encoded in network and [23:36] if you if you just state that we don't [23:39] care how it's encoded it can be highly [23:41] nonlinear we just care about whether [23:43] this algorithm is implemented then our [23:46] paper our theory and also our empirical [23:48] results show that distributed alignment [23:52] search and causal abstraction breaks on [23:54] um without any additional analysis. I I [23:58] hope that helps a bit. [23:59] >> Yeah, I think that that makes sense. I I [24:01] also yeah I also already had the [24:03] impression that this kind of uh [24:07] automatic search uh was much dependent [24:11] on people on papers were first doing [24:13] things like uh just use neurons and uh [24:19] later yeah they started doing using [24:21] sparse encoders when that was popular [24:24] and it does feel like a lot of it [24:26] depends on how exactly you divide the [24:30] model to do interventions. [24:33] Uh [24:36] and yeah, people are going to have to [24:38] figure out some better. So like what [24:41] exactly features are and how to divide [24:44] the model if they want to get and yeah I [24:47] think your result is is a cool is [24:50] interesting that [24:52] >> yeah so so if you want to abstract it [24:53] even more I think it's just uh you can [24:57] maybe even make some claims about interp [25:00] in general which is just if you're [25:03] optimizing something in your interp [25:06] method you need to be really careful [25:09] about what assumptions you make and what [25:11] assumptions you don't make and whether [25:14] making or not making these assumptions [25:17] breaks down anything um in particular. [25:20] For example, here if you don't make the [25:22] linear the linear assumption the theory [25:25] breaks down like you you don't have to [25:27] do like it doesn't make sense to use [25:28] does if you don't make the linear [25:30] assumption at least in its current uh [25:33] state without [25:36] analyzing generalization or something [25:38] like this. Um [25:41] >> yes. [25:43] >> Yeah, go ahead. [25:44] >> Yes. And as Julian said, there was there [25:47] is research out there which um suggests [25:50] or or already shows that some features [25:53] are um nonlinear encoded. So that is [25:56] actually what we call the why we call it [25:58] the nonlinear representation dilemma. So [26:00] we have on one side if you are um just [26:04] um using linear maps you on one side [26:07] mitigates this problem which we have [26:08] shown here. So you cannot really diary [26:11] the alignment map but on the other side [26:14] there are features which are not are [26:16] encoded nonlinearly. So if you're just [26:19] using linear alignment maps you will not [26:20] be able to find them and that is this [26:23] dilemma. So the question which one [26:25] question one thing is actually the [26:27] generalization how well does it [26:29] generalize and the other thing is other [26:31] discussion how what kind of encoding do [26:34] we assume in the model is that as [26:37] example there is some work which shows [26:39] that um weekdays are encoded in a circle [26:41] in a neural network and so the question [26:44] is how [26:48] >> yeah and as the question then of course [26:51] becomes how How do we select the right [26:54] alignment map to on one side prevent [26:59] such germanated features on the other [27:01] side [27:03] um yeah to prevent them and still get [27:06] the correct uh encodings. So there is [27:09] and um we I think the authors are um [27:12] agreeing that coal abstraction is [27:14] actually quite promising but as we [27:16] showed in this work there's still a lot [27:18] of work to do in case of generalize [27:22] generalization testing and also on um [27:25] figuring out how complex um an alignment [27:28] map has to be which is uh surprisingly [27:30] actually um the previous the predecessor [27:33] of cosal obstruction can be said to be [27:35] probing and this actually probing [27:37] actually at the in problems. How strong [27:38] do we probe for something? And so what [27:41] is also shown in this work is actually [27:42] that the problems which were shown in [27:44] probing followed causal abstraction. So [27:46] there's still work to do. We cannot just [27:48] assume it will when it finds something [27:50] that it's automatically [27:52] saying that it's included. So that is [27:54] actually what uh one of the takeaways of [27:57] this work is there's still work to do to [27:59] >> yeah probing you don't like probing is [28:02] usually a bad idea to use a something [28:06] like a neural network for the probing [28:08] instead of a linear probe because then [28:10] you always get good results because your [28:14] neural network figures out the the thing [28:16] you were trying to figure out it was [28:17] represented there. [28:19] >> Yeah. Especially if your network is [28:20] injective, right? Which there was just [28:22] this paper. [28:23] >> Yeah, a new paper thing. I saw it. [28:26] >> Yeah. So, okay, there's quantization [28:29] which is a problem. They assume [28:30] continuous states which we also but uh [28:34] if you assume it's mostly injective then [28:36] as you said if you have a nonlinear [28:37] highly nonlinear probe it can basically [28:40] deduce the whole input. So you can make [28:42] any conclusions about the input. So you [28:44] cannot really make any conclusions about [28:46] whether [28:47] >> things are truly encoded in network or [28:50] not with a probe here. You you have to [28:52] trade off. [28:52] >> Yeah. [28:53] >> Um [28:55] >> exactly. And coal abstraction actually [28:57] tried to alleviate this problem in [28:59] probing by also requiring that the [29:02] outputs behave on the interventions as [29:05] expected which probing just tries to [29:07] find out if it's it is it is it encoded [29:10] and cause abstraction tries to figure [29:12] out also when we intervene on this [29:15] feature does it behave as it should be [29:17] given our assumptions what it uh what is [29:19] implemented as algorithm and what [29:22] actually is then shown here that the [29:24] problems still come up still too strong [29:26] alignment maps still allow for um [29:29] figuring out things which are not there [29:31] by simply gerrymander this whole thing. [29:34] So figuring out what is the feature and [29:37] then figuring out what it where it [29:38] should put it. So that is actually the [29:40] main proof that that is possible even in [29:42] coal abstraction. [29:51] Anyone else want to uh ask a question or [29:55] something because yeah I don't want to [29:56] be uh I can ask more things but I don't [29:59] want to be as if [30:01] >> hello um what thanks for the [30:04] presentation I a question maybe it's a [30:07] followup um on what you just said it's [30:10] um do you have any hint on what would be [30:14] a good methodology to [30:16] >> maybe maybe I could maybe I could ask [30:19] something that is So [30:22] >> if if if like I I want to use D to [30:27] analyze my network, [30:30] >> how would you go about this now? Like [30:32] knowing this result? [30:40] >> Yeah. Um hear me? [30:44] >> Yes, sure. Um there is um I think two [30:47] people asked at the same time a question [30:49] but yeah um I think the question is the [30:51] same one [30:53] >> is the question the same one how to use [30:55] distributed lent this methodologies to [30:58] still get you [31:00] >> I don't maybe there is some connection [31:03] issue on my on my [31:04] >> side another person so I don't know [31:08] >> there [31:10] okay I think I will get go first with [31:12] Julen's question and then with your [31:14] questions Is that fine? Because I [31:15] already hear his and I'm not sure if I [31:17] hear you completely. [31:19] >> Okay. [31:20] >> But um [31:21] >> yeah, I I did hear you by the way. So I [31:23] don't know. [31:26] >> Yeah. Okay. So the right now with the [31:30] insights of this paper the best [31:33] suggestion I can give to still use the [31:36] framework of course obstruction which um [31:38] as I said I still think it's highly um [31:42] useful is for now to [31:45] um there's if you just want to get a [31:48] short view on it daily rotation maps you [31:51] might not find um the feature even if [31:54] it's there but at least you will not [31:56] find some you you might get some false [32:00] negatives but at least you don't get uh [32:03] at least it's very unlikely that you get [32:04] some false positives. If you wants to [32:07] dive deeper into how uh dive deeper and [32:10] also find features which may not be um [32:14] may not be linear. The best suggestion I [32:18] can give you is probably the same as I [32:20] did after this work is to apply this [32:25] with alignment search with some with the [32:28] with some alignment map you think could [32:30] be useful and then look and then analyze [32:33] what actually is found. So make some [32:35] visualizations [32:37] to figure out is it uh to figure out [32:39] what actually happens in the what [32:41] actually is found as encoding of the [32:44] feature. But if you are just wanting to [32:46] apply it, um the suggestion is stay with [32:50] rotation maps. You they are not that uh [32:53] vulnerable vulnerable for um [32:57] false positives. So you might get some [33:00] false negatives, but if it's positive, [33:02] it's it is possible that it's [33:04] implemented. It's much more likely than [33:06] when use some strong alignment maps. [33:08] Does that answer the question? [33:10] >> Yeah, sure. [33:13] >> Great. So okay then um your question. [33:17] >> Yes. Yeah. So sorry maybe there was some [33:20] misconnection. My question is maybe a [33:22] bit of the same but it's do you have any [33:25] idea on how could be a what would be a [33:28] good way to distinguish between what [33:30] would be um uh something that is too [33:33] strong where uh assumption too strong [33:35] where the the the probe or the alignment [33:39] map does heavy lifting and something [33:41] that is uh really inside the network. Do [33:44] you have any maybe [33:47] idea about analy analyzing the the [33:50] volume that it's it's [33:54] you can get out of this or some [33:55] assumptions that would be good that is [33:58] stronger than linear uh linear features [34:02] but maybe could be inside the the [34:05] network. [34:07] That's a good question. I have um that [34:11] is actually um concerns probably [34:13] followup work and so there are two [34:16] directions which can be go which can be [34:19] which we can follow. One is for example [34:21] there is some work in probing which um [34:24] just allows um probing maps which are [34:28] equally strong than the remainder of the [34:30] model um to find the input so to say. So [34:34] it's uh dynamically adapted to the model [34:36] and where you look at the model. That [34:38] would be one direction which could be [34:41] interesting. So using alignment maps [34:43] which reflect um how much the model [34:46] actually can um [34:49] the com the processing capabilities of [34:52] the model after the layer you look into. [34:54] On the other side actually we made some [34:57] follow-up work on this and analyzed [35:00] actually what is found when using a [35:02] refet on um the arithmetic task of the [35:06] mechanistic interpretability benchmark. [35:08] That's just some task which we have as a [35:10] prompt as some summation and then we [35:14] look at if there's the carry on feature. [35:15] The carry on feature in a summation is [35:17] the when you the first digits of both [35:19] numbers together and they higher than [35:21] nine probably as you learned to add in [35:23] school on paper. So um then we looked at [35:27] what is found by RefNet and then we [35:30] actually scaled down the complexity to [35:33] figure out what actually is found and we [35:35] actually found that even linear [35:37] alignment maps. So there is two [35:39] different grades of linear maps in this [35:42] work are linear alignment maps are [35:43] rotation maps. So there are a bit more [35:46] constraints than just linear alignment [35:47] maps but we called them linear maps [35:49] because also in the previous works they [35:51] are called linear maps. But if you use [35:53] the linear maps which are [35:54] interventionable so they don't need to [35:56] have um just the properties of rotation [35:58] but can be any way can be any other form [36:01] which is possible by a matrix multiplic [36:03] multiplication and we actually are in [36:06] that work we can show that even then [36:08] there is a possibility that the model is [36:11] exploiting some other feature and [36:13] doesn't really return something [36:15] faithful. So um if you want to get out [36:20] of just using rotation maps which um [36:24] allow to prevent uh mitigate such things [36:27] which we saw there. So rotation maps is [36:31] relatively [36:33] um faithful just because it is so [36:36] constrained that the that the alignment [36:38] map cannot really hack too much so to [36:41] say. But if you want to go even out or [36:43] even just to linear maps which are [36:46] probably the simplest um mapping which [36:49] you can have after rotations even then [36:51] you already run into problems that you [36:54] can find something which actually is not [36:56] existing there. So my suggestion is [36:59] there if you want to apply anything more [37:03] complex than linear alignment maps, you [37:06] probably have to get to um to get your [37:09] hands dirty so to say and figure out [37:11] what actually is found and analyze it. [37:14] And that actually can right now be a bit [37:16] a process. I spent two months just [37:20] hunting down one feature, this carryon [37:23] feature to figure out what what is [37:24] actually doing. And so my suggestion [37:29] right now, if you don't want to spend [37:32] one one two months just to figure out [37:33] what you found, then probably just stay [37:37] at rotation until there are improvements [37:41] made to coal abstraction which allow to [37:44] prevent such behavior. there I'm quite [37:47] confident that there is such [37:49] improvements. I also talked with Adekus [37:51] Geger which is the original author of [37:54] this alignment search and there he also [37:57] is aware of this and there needs to be [37:59] some improvements. So that and on the [38:02] other side one other direction which can [38:05] be used besides looking what is found is [38:08] on the other side also uh improving [38:11] generalization methods so to say. So try [38:13] to figure out if the feature actually [38:16] the fe the the feature you found [38:17] actually transfer from your setting to [38:19] more out of distribution settings. [38:22] That is something else which could be [38:24] useful. So um either stable rotation [38:27] maps if you want to spend a bit more [38:29] work try to experiment with as much test [38:33] data which is as out of distribution as [38:35] possible. That is probably so the middle [38:38] ground of work versus work quality um [38:41] tradeoff. And if you want to be quite [38:44] sure what you found, you probably have [38:46] to go in there and spend some time [38:48] figuring out what you found. Does that [38:49] answer the question? [39:41] Yes. Um, yeah, that's actually also a [39:44] good point. We, yeah, we actually showed [39:46] that it follows it. That's also that's [39:50] actually the intuition which we had [39:51] first to why rotation is might good be. [39:54] There's still the problem with rotation [39:56] that um it will just find linear maps [39:59] and they need to be orthogonal to some [40:02] other uh to the other features because [40:04] it's just rotation. So you have to find [40:08] something like you can just find [40:09] something like that and that actually [40:11] the dilemma part comes in there is [40:13] enough work which shows that some [40:15] features are not linear and you will not [40:17] find them when you just use linear maps [40:19] it will not work. So as said if you just [40:23] want to get some fast results with does [40:26] stable rotations [40:28] it will give you some false negatives [40:31] probably but at least it mitigates um [40:35] many false positives and that's probably [40:37] also the reason why in current research [40:40] um rotation maps actually are seemingly [40:43] faithful because they simply um they are [40:46] so restricted that exploitations of [40:49] different structures is just very hard. [40:55] Thank you for the question by the way. [41:15] Are there any other questions? I'm happy [41:17] to answer any questions which are there. [41:47] Okay. [41:50] >> I have minor thing to ask if nobody else [41:53] ask anything else. Uh [41:56] >> but yeah. So you this is uh not pretty [42:00] important but you earlier mentioned like [42:02] the days of the week being on a circle [42:06] as an example of a nonlinear feature bas [42:09] isn't that still a linear representation [42:12] like there is some structure there that [42:14] you wouldn't find on something like [42:16] color but it's we still a direction on [42:20] the circle right [42:23] >> yes but it's in the uh hidden space It's [42:27] still there is no you cannot draw any [42:29] line through this um any line which [42:32] actually represents the days so to say. [42:35] So if you are drawing a line through a [42:37] circle and then projected down the [42:39] circle down to the line there will [42:41] probably be overlap between different [42:43] days which are on opposite sides of the [42:45] circle. [42:47] So it is it is it is a direction so to [42:51] say but it's a direction in rotation and [42:54] to find a direction in a circle and to [42:56] find such directions you need some [42:57] complex more complex alignment maps [43:00] which are given in that paper actually. [43:03] So but uh to find to the alignments maps [43:06] which are mentioned in that paper you um [43:09] actually all no not linear they are all [43:13] having this um circle geometry in it. By [43:16] the way, that paper actually got an [43:18] update and it's no longer a circle. It's [43:20] um it's still a circle, but a circle in [43:24] a circle like structure in more [43:27] dimensions. So, it's not just a circle. [43:30] >> Okay. [43:32] >> Although maybe the the the intuition [43:34] here is like the day the day is still [43:37] linearly encoded, but the relationship [43:39] between the day is nonlinear. [43:41] >> So like Monday is still clearly linear [43:44] direction. Yeah, I think related [43:47] >> in that on the paper anthropic paper [43:50] where they do [43:52] >> uh like like there is there is a paper [43:55] where the they found how GPT a bunch of [44:00] models it was PC and the thing at GPT2 [44:03] and whatever like this edition and they [44:05] found it with uh some something similar [44:08] to the modular edition thing that Neil [44:10] found but with and some kind of elixru [44:14] structure. [44:15] And there is a there is the anthropic [44:18] [clears throat] paper I don't remember [44:19] right now the one where they found a [44:22] bunch of this uh found a bunch of graphs [44:26] of like features uh how they try to find [44:30] some algorithms by looking at features [44:33] and it feels like the one where they do [44:37] they explain addition and they get a [44:40] bunch of complicated things with modulus [44:42] and whatever [44:44] Seems like that might be what happens if [44:47] you take like it looks by to be maybe I [44:50] just wrong but it looks like the what [44:52] you take if you have something like the [44:54] elixir structure thing and you are [44:56] taking linear slices [44:59] uh from it and it's trying to explain a [45:02] single example of addition and they [45:05] might just be missing like the the [45:07] complete thing with the whole structure. [45:14] I didn't fully get it but uh yeah yeah [45:18] >> I think that's some weird idea I had [45:21] like when I was uh reading that paper [45:23] for a previous group necessary. [45:26] >> Okay. Okay. [laughter and gasps] [45:32] But um yeah, I I think we also read this [45:36] uh helix paper where they show this [45:38] helix. [45:40] But yeah, we actually also at some point [45:44] in the uh remaining time of my master's [45:48] thesis actually we looked also we had [45:51] some hypothes at least I had some [45:53] hypothesis that maybe what is this found [45:55] is some some way of cutting through the [45:59] true paper a true feature when we using [46:01] just linear ones but yeah um it didn't [46:06] turn out to be that but it was about the [46:09] carry on feature in the add arithmetic [46:13] task which is summation so to say [46:29] yeah I guess my my my idea is just that [46:32] it seems likely that in addition [46:35] specifically [46:37] bigger models [46:39] the [46:41] do do something like the elix thing. Uh [46:44] but methods like the ones the the [46:48] methods that are used they just use [46:51] sparse encoders to find linear encodings [46:54] can't find that kind of structure and [46:57] and like the things like the anthropic [47:01] paper where they yeah that [47:06] were just looking at graph of graphs of [47:09] linear fe you know how linear features [47:12] do cost. song output might just be [47:16] missing that kind of nonlinear [47:19] representation [clears throat] of you [47:20] know addition or of numbers or whatever [47:24] it seems related [47:26] >> but not super true either [47:31] >> yeah I think I see your point that's [47:34] true as far as outers that's also [47:36] something I saw actually are not able to [47:38] find anything which is nonlinear at [47:40] least in the papers I read they always [47:42] used some kind of linear mapping a [47:45] linear encoder and decoder so to say [47:48] with a refet uh activation function with [47:51] hidden space so they probably also not [47:54] find yeah they will not find any [47:56] rotation that's actually also yeah I [48:01] agree with that completely and there is [48:03] also the other question of course um [48:06] does does a large language model [48:08] actually do any computations at all as [48:11] we use them for example on writing huge [48:14] summations down on paper or are they [48:17] just um some kind of memorization [48:21] machines which just picks out the right [48:23] place the right answer and there's also [48:25] work which suggests that um large [48:28] language models um are some kind of [48:30] something in between some kind of [48:31] horistic machine where they have [48:33] different features inside of them where [48:36] some features in some cases uh put push [48:39] the output in the right direction so the [48:42] tense [48:42] and the the first and the second digits [48:44] and so on. So it's still an open [48:47] question actually how um is encoded uh [48:51] how are so how addition actually is [48:53] encoded in models and as far as I see [48:56] there is still no consensus on how it's [49:00] done and also in my work here and in my [49:03] work which followed after this there was [49:06] not really any um there is no clear [49:09] answer to this so cause obstruction [49:13] right now cannot give any answer to the [49:16] question of what actually is done in [49:17] summation [49:19] um is still as as I want to repeat it's [49:23] um still a very promising method but [49:25] it's it just needs more work on it it to [49:30] get to iron out these problems because [49:34] it right now it is as far as I can see [49:36] it's more applied with the with the [49:38] thing with the thinking it will just [49:41] work because it's causally um correlated [49:45] So it does actually what we think it [49:47] does when we intervene and so then is [49:49] the thought um then we actually found [49:52] something and my work so far shows that [49:56] while that's there is still work that [49:58] needs to be done to actually that we can [50:01] and say we actually found something the [50:02] model actually is doing and not just [50:04] something um the alignment map is [50:08] exploiting some other things in the [50:11] model. [50:23] I hope I deviated with that answer not [50:25] too much from your question. [50:32] Yeah, [50:34] I guess I I I guess I just was thinking [50:36] about like how [50:40] like in this I I in this I should my [50:44] screen this this attribution graph in [50:46] the entropic paper uh seems to me like [50:50] the kind of thing where I wonder if this [50:54] is common where uh so they are doing the [50:59] attribution [clears throat] graph for [51:00] like some specific [51:02] addition, right? And they get Oh, sorry. [51:04] I I removed the And they get all of [51:08] this. I I it feels to me that this may [51:12] just be [51:14] this explanation here might just be [51:17] taking a linear slice of a more [51:20] complicated or some nonlinear thing in [51:23] terms of you know there you find things [51:25] like [51:27] uh some feature that represent [51:31] the number mode something and things [51:35] like that. And that seems like the kind [51:38] of thing you might find if you are [51:42] if the actual [clears throat] [51:43] representation is something like a [51:45] circle uh or something like the elix [51:48] from that paper and and you are just [51:51] taking uh linear slices from it and you [51:56] find things like all of the numbers that [52:00] ended five have are in some direct you [52:03] know are some activate some feature in [52:06] some direct ction because the actual [52:08] thing is some kind of elix and and it [52:10] has a module of 10. So it happens to be [52:12] pointing in the same direction every [52:14] five every you know all the numbers that [52:16] end on five and something like that. Uh [52:22] and that's just that kind that kind of [52:25] thing seems related to to all of this. [52:29] Yeah, I agree. It just seems [52:31] seems seems reasonable that something [52:35] could be happening. Although it's also [52:37] not sure what Yeah. [52:39] >> Yeah. [52:39] >> We just [clears throat] don't know. [52:42] >> Yeah. I also agree. It's an interesting [52:44] um hypothesis [52:47] and as I said um [52:51] as I saw in my work, it's it's not [52:54] always [52:55] I what we see is not always what [52:58] actually happens. So or so it actually [53:03] could be something like that. It's [53:05] actually also something which we does in [53:08] discussed internally that um such um [53:13] interpretability methods and especially [53:14] visualizations [53:17] can actually include some kinds of um [53:20] red herrings which seem reason uh seem [53:24] um seem important or seem structurally [53:28] but in the ends turn out to actually not [53:31] be that important or actually be part of [53:33] something bigger. like for example this [53:35] cuts into this sphere. So that's a good [53:38] point. So and actually I also have yet [53:42] to mention probably that um while [53:44] they're looking through literature I [53:46] never really saw [53:48] I'm not really sure if there is any good [53:50] definition of what it means for a found [53:52] feature to be actually faithful. Most of [53:54] what I found in these regards is [53:56] actually a feature is faithful if our [53:58] method finds it or it's if it's costly [54:01] related to the output or something like [54:03] that and as we showed it doesn't really [54:06] need to be faithful because if we have [54:08] these extreme cases for example we can [54:10] find for a random model actually that it [54:12] implements it but the random model of [54:14] course doesn't implement it or we can [54:17] find for any neuronet network any [54:19] algorithm. So, um, the discussion of [54:22] what is a faithful feature is probably [54:24] still open. As far as I know, it's [54:28] >> part of the problem is actually figuring [54:30] out what it means for a feature to be [54:33] faithful. Uh [54:36] because I mean if we have some clear [54:40] idea of that we could already be doing [54:43] things like testing like figure out it [54:46] would be easier to get more to make [54:50] better method is have some clear idea [54:52] what it means for future to be safer [54:54] that we could just test on new methods. [54:57] Uh [54:57] >> I'm not sure whether there's a solution [54:59] to this problem. I think it's about you [55:01] need to make assumptions what you think [55:04] what what you define as being faithful [55:06] and then you can [clears throat] do [55:08] stuff with it. I don't think there's [55:09] like a ground truth definition of a [55:12] feature being faithful. [55:14] How does it look? [55:16] >> But I don't know. [55:18] >> I mean do you know the mechanistic [55:20] interpretability is mechanistic [55:22] interpretability identifiable paper? [55:25] Which kind of reminds me a bit of this [55:27] discussion. [55:29] Um, Dennis, I think you know it for [55:32] sure. Gets [55:34] >> I think so, but um, yeah, [55:37] >> by Maxim, [55:43] >> I'll send it. [55:45] >> Right. Then [55:48] yeah and in the end I think the best at [55:50] least I can do it do is uh at least the [55:54] best we were able to do is not to show [55:56] what is faithful but taking uh the [55:59] definition of what causal abstraction [56:00] gives us and then show that there are [56:03] some situation which it intuitively [56:07] seems wrong that that is the correct [56:09] definition for a faithful feature. So in [56:12] our case it is that we can align any [56:14] algorithm with any neural network and [56:17] that we can find for random initialized [56:19] networks good I a good yeah IIA so to [56:22] say thank you for sending them uh paper [56:29] ah this everything everything yeah [56:31] that's actually one I know [56:38] this discusses [56:40] if interpret for if [56:45] if a feature which is found needs to be [56:48] um [56:50] the only one so to say which encodes it [56:53] and only if it's the the one feature [56:56] which encodes it. Yeah, it basically [56:59] asks it discusses this question that we [57:02] also have been discuss discussing of [57:04] like [57:06] is there a ground proof feature or is [57:08] this just Yeah. Can we identify? Yeah. [57:13] Yeah. I guess it seems to me like it's [57:16] that there's probably [57:19] uh some at least some set of reasonable [57:24] natural assumptions you can find. Uh [57:29] in [57:30] in the same sense that in general in [57:35] science this kind of sense questions [57:38] people just find some reasonal [57:41] assumptions that work a lot of the time. [57:45] Uh but I don't know. [57:48] >> Yeah. Um in my personal opinion is [57:51] probably that's um a question of [57:54] iteratively refining the definition. For [57:57] example, there was this recent paper [57:59] from Attekus [58:01] which he took a more uh philosophical [58:05] um stance on it and one of the important [58:09] points there shown was that it needs to [58:11] generalize so to say. So it needs to [58:14] generalize to all possible data. [58:17] But of course that is um not testable in [58:21] reality. [58:22] >> But still that is [58:25] >> sorry. [58:26] >> Yes. Some people I feel like some people [58:28] like Neil uh lately are just uh and I [58:32] thought it's good if it works on [58:35] practical examples or something like [58:37] that and have I kind of gave up on [58:42] having a super clear idea of what is [58:46] truly represented on the model or [58:47] something like that. Uh [58:49] >> yeah I agree. So Neo has Neo has much [58:52] more a much more practical view more [58:55] recently more he his his uh approach to [58:58] making more recently is more of this [59:01] thing of [59:02] uh let's do something that has a direct [59:06] practical impact and while I think this [59:08] is like very interesting and we need to [59:10] do this because this is kind of in the [59:13] end what what interpretability should uh [59:16] should give us but I still think there's [59:18] like a lot of interesting more [59:20] theoretical problems to show solve that [59:23] will downstream [59:25] also help us. But yeah, this is the the [59:29] normal um problem of pra more practical [59:33] applied research and more foundational [59:36] um theoretical research. Yeah. And you [59:38] do like find a lot of like you do find [59:42] solutions to more practical research by [59:44] thinking a lot by doing more base [59:47] research and thinking about more the [59:50] philosophy of it and things like that. [59:52] That also happens. I don't know what's [59:55] best in this case. Uh probably better if [59:59] >> both [60:01] people doing both. Yeah. [60:03] >> Yeah. I also agree. Here is by the way I [60:06] send in a chat the paper from Atticus [60:08] which [60:10] takes some more philosophical question [60:12] on how causal abstraction is showing [60:15] faithful features. I found it quite [60:17] interesting. Great. [60:25] And yes, al I agree that it probably is [60:28] necessary to do both because on one side [60:30] for it can help um practical applying of [60:35] it to find so to say more [60:40] better features or more faithful feature [60:42] whatever that means. And on the other [60:44] side practical appliance will also then [60:47] give some insights for example insights [60:51] into how theoretic in with what [60:54] theoretical parts by the by now ignore. [60:57] So yeah, as example I made some [61:00] experiments uh which are quite practical [61:02] which are just trying to find a feature [61:04] and then figure out what this was found [61:06] and that also informed some of yeah that [61:09] is also quite informative of how to [61:11] improve cost abstraction theoretically. [61:15] So [61:18] don't give up on either of them. Neither [61:20] theoretical nor practical. [61:25] Um neither of them are not used. Both [61:28] are very important. [61:36] Agreed. [61:39] Okay guys, I slowly have to leave but [61:42] thanks a lot for having us. [61:45] >> Um yeah, bye guys. [61:49] Goodbye. Still around for questions if [61:51] there are any. [61:54] >> Anyone else want to ask some question? I [61:56] think I for me I can end it here. But [61:59] yeah, that's [62:00] >> Yeah, if for me it's fine as well, but [62:03] if someone has questions also feel free [62:05] to ask later on some questions. I think [62:09] you found [62:09] >> Oh, by the way, uh I probably should [62:11] have said earlier when there were more [62:12] people, but uh people who are here, I I [62:17] made a when to meet for like the when [62:20] the reading group when people are [62:22] available for the reading group because [62:23] I might change it after this week. Uh so [62:27] people should just look at that and put [62:29] them when they are available if yeah if [62:32] you want to be on the reading group in [62:34] the future because I I probably change [62:38] the time invest [62:47] so I have finally revived because I'm [62:49] now in the EU. Uh [62:52] so do you still want to work on like the [62:54] chess um project? [62:57] >> Oh right. Uh [62:59] uh we we can talk later about that. [63:03] Yeah, I was working on something else. [63:05] Uh because I got a grant. So but I I [63:09] might still want to do a bit on the [63:13] Yeah, I just been busy. But I I'm fine [63:16] with we can talk one of these days [63:18] project and maybe do something. It's not [63:20] like I've been that productive on the [63:23] other thing to be honest. [laughter] [63:25] But uh yeah, I was more of the I was [63:28] supposed to be working on the other [63:29] things. So I was [laughter] thinking [63:31] about other projects, but I also was not [63:33] that working on the thing I was supposed [63:35] to be doing. So [laughter] [63:37] I was I've been doing things but not [63:40] all. It was not like I don't have free [63:42] time anyway. So [63:44] >> yeah. Yeah. I mean yeah that makes [63:45] sense. It sounds like you know the usual [63:47] kind of struggle. Uh yeah. Okay. I text [63:49] you uh on that chat channel later then. [63:55] >> All right. See you. [63:58] >> See you. [64:00] >> Byebye. [64:01] >> Okay. p.m.