Méloux Shows AI Interpretability Methods Fail Without Random Baselines

Méloux指出缺乏随机基线会导致AI可解释性方法失效 由于研究人员未针对随机基线模型进行统计原假设检验，标准AI可解释性技术往往会从纯随机噪声中提取出看似合理的结构化解释。 本报告探讨了AI可解释性方法中的统计脆弱性与不可识别性，并借用著名的fMRI“死三文鱼”研究作类比，展示了现有技术常常如何从随机噪声中提取出虚假的结构化解释。论文诊断了线性探针、稀疏自编码器（SAE）和回路发现中存在的欠明晰、过度确定与不可识别性等核心理论问题。为解决这些缺陷，作者提出了一套严谨的框架，将结构因果模型、针对随机基线模型的原假设检验以及明确定义的替代类别相结合。报告最后深入讨论了实用主义与实在主义可解释性、统计评估的计算成本，以及使用合成模型进行实际压力测试等内容。 论文《AI 可解释性的死鲑鱼》借鉴神经科学著名的死鲑鱼 fMRI 实验，指出当前可解释性工具频频在纯随机噪声中寻找虚假规律，例如参数完全随机化的 BERT 仅靠初始 Token 嵌入的信息残留就能使线性探针达到约 0.8 的准确率。 (1:09) 特征归因（如 SHAP、Integrated Gradients）、稀疏自编码器（SAE）及电路发现等方法普遍存在统计脆弱性，其深层理论根源在于欠规范（Under-specification）、过度决定（九头蛇效应）以及不可识别性（Non-identifiability）。 (11:53) 作者建议引入结构因果模型（SCM）与严格的零假设检验，通过在相同结构的随机基线网络上运行相同的解释流程，并借助蒙特卡洛抽样计算经验 p 值，以排除噪声造成的虚假发现。 (17:04) 该框架将解释任务形式化为由查询空间、代理模型类与误差度量构成的三元组推断问题，并证明可识别性是解释方法本身的属性而非目标模型固有属性，研究者可通过约束代理类或缩小查询空间将不可识别问题转化为可识别问题。 (31:37)

In the position paper "The Dead Salmons of AI Interpretability", Maxime Méloux and co-authors argue that explainability methods suffer from severe statistical fragility, frequently extracting structured, plausible explanations from pure noise due to a lack of null baselines and formal identifiability. The Dead Salmon Problem in Neural Networks (1:09) Drawing an analogy to Bennett et al.'s 2009 fMRI experiment—where uncorrected multiple comparisons detected statistically significant brain activity in a dead Atlantic salmon—the authors warn that AI interpretability produces equivalent false positives. In an illustrative experiment using BERT on 300 IMDb sentiment sentences, randomizing all transformer weights while preserving only initial token embeddings still yielded linear probing classification accuracies of approximately 0.8 ± 0.15 with statistically significant correlations. Because rich semantic signals persist through random transformations, interpretability tools falsely infer that the network has actively learned the task. The presenter surveys similar vulnerabilities across the literature: vision saliency maps remain visually plausible under layer randomization, sparse autoencoders (SAEs) extract coherent features from randomly initialized networks, and linear probes conflate concept representation with downstream utilization without proving causal necessity. Theoretical Causes: Under-Specification and Overdetermination (11:53) Méloux diagnoses these empirical failures through two theoretical mechanisms: under-specification and overdetermination. Under-specification occurs when analysis relies solely on input-output behaviors; because an infinite number of internal computational graphs can produce identical mappings, predictive techniques overfit to spurious patterns. Conversely, overdetermination complicates causal and interventional methods. Neural networks contain redundant, independently sufficient computational pathways and superposition—termed the "hydra effect"—meaning ablating an individually predictive component does not cause task failure because the network reroutes computation. Both failures undermine statistical identifiability, the mathematical guarantee that an underlying explanation can be uniquely recovered from observation. When tasks are non-identifiable, expanding the hypothesis space drastically increases false discoveries across Concept Activation Vectors (CAVs), SAEs, and circuit discovery methods. Grounding Interpretability in Statistical Causal Models (17:04) To counter dead salmon artifacts, the authors propose formalizing interpretability through Structural Causal Models (SCMs), mapping network inputs as exogenous variables, internal activations as endogenous variables, and matrix operations as structural equations. Under this framework, an explanation is defined as a surrogate model evaluated across a specific triplet: a query space of causal questions, an admissible surrogate class (such as subgraphs or linear probes), and an error metric quantifying divergence from the true model. Crucially, the presenter argues that identifiability is a property of the explanation method rather than the target model; researchers can enforce identifiability by narrowing the query space or restricting the surrogate class. Operationally, methods must incorporate null hypothesis testing against identically structured, randomized baseline networks, using Monte Carlo sampling to compute empirical p-values and establish confidence intervals. Pragmatic Abstractions Versus Reductionist Realism (67:50) During the discussion, participants debated whether interpretability should seek a single ground-truth algorithm or settle for functional approximations. Méloux defended a "pragmatic" stance over an absolute "realist" view. Just as medical science models organs and physiological systems without tracking every atom, pragmatic interpretability relies on task-specific, compressed explanations evaluated by their predictive utility, avoiding reductionist descriptions of raw weights. However, attendees highlighted practical obstacles to this rigor. Rigorous statistical sampling and confidence bounds add heavy computational overhead to procedures like training SAEs on frontier models. Additionally, while synthetic benchmarks like Google DeepMind's Tracer compile algorithmic code into known ground-truth circuits, their near-binary weights differ drastically from the smooth, continuous representations formed by gradient descent. The authors urge AI interpretability to undergo a statistical revolution akin to neuroscience and econometrics by treating explanations as estimators bounded by quantifiable uncertainty. While formalizing explanation triplets and testing against randomized baselines provides a path toward cumulative science, the field faces unresolved tensions between the high compute cost of rigorous controls, commercial incentives to scale frontier models, and the non-identifiability inherent in complex, redundant architectures.

Maxime Méloux 等人在立场预印本《The Dead Salmons of AI Interpretability》中指出，当前机械可解释性研究普遍存在“死大马哈鱼效应”，即解释工具常从随机噪声中提炼出虚假但看似合理的计算模式；作者主张引入统计因果推断框架与严格的零假设检验，以解决解释方法的统计脆弱性与非可识别性问题。 死大马哈鱼隐喻与解释工具的统计脆弱性 (1:09) 该研究借鉴了 2009 年 Bennett 等人的经典功能性磁共振成像（fMRI）神经科学实验：研究人员在对一条死大西洋鲑鱼展示人类社交照片时，由于未进行多重比较校正，在纯随机噪声中检测出具有极低 p 值的虚假神经激活。演讲者指出，当前人工智能可解释性领域正面临类似的“死大马哈鱼问题”，即解释方法频繁将结构噪声误判为模型的真实计算逻辑。在作者的基线实验中，当对 BERT 模型的全部 Transformer 权重进行随机初始化、仅保留初始词嵌入时，针对 300 条 IMDb 情感分类样本的线性探针仍能取得约 0.8 ± 0.15 的准确率，主成分分析（PCA）也检测出极低 p 值的虚假相关性。这表明词嵌入中预存的语义信号可穿透无意义的随机变换，诱使探针误判模型正在执行主动计算。 广泛的文献调研进一步证实了此类脆弱性。在计算机视觉领域，逐层级联随机化测试显示，即使模型权重被完全打乱，显著性图依然能生成在视觉上看似合理的解释，且易受对抗扰动篡改；理论上的不可行性定理亦证明，积分梯度（Integrated Gradients）与 SHAP 等线性特征归因方法在特定条件下表现并不优于随机猜测。在线性探针方面，研究人员长期混淆了概念在激活中的“表征存在”与下游决策中的“实际利用”；而稀疏自编码器（SAE）同样能在随机网络中提取出结构化特征，概念激活向量（CAV）等基于概念的方法也常因数据分布微调而产生非因果的“解释幻觉”。 欠拟合、过度决定与非可识别性的理论根源 (11:53) 演讲者将上述统计缺陷归结为两大理论机制：欠拟合（under-specification）与过度决定（overdetermination）。欠拟合多见于预测性解释方法中，即仅依赖输入输出行为来推导解释，然而理论上存在无限种内部计算图均能实现完全相同的输入输出映射，导致解释工具严重过拟合。过度决定则主导了因果干预分析：由于神经网络包含多条独立的冗余计算通路，当通过干预消融某条关键通路时，模型可通过状态叠加与剩余通路动态重构计算流（即“九头蛇效应”），使研究人员难以分离出必需的因果依赖项。 欠拟合与过度决定共同导致了统计学中的“非可识别性”（non-identifiability），即无法仅通过有限观测唯一确定系统真实的潜在参数或解释模型。在非可识别的情境下，假设空间的盲目扩大会急剧增加伪发现率。演讲者分析指出，CAV 与线性探针受制于欠拟合而本质上不可识别；激活修补（activation patching）虽在局部重要性得分计算上具备数学可识别性，但常被过度解读为全局行为的排他性定位；而现行的回路发现（circuit discovery）方法则高度非可识别，微小的数据集或参数变动即可重构出截然不同却同样自洽的回路子图。 基于结构因果模型与零假设检验的解决方案 (17:04) 为消除解释工具中的伪相关性，论文提出了一套基于结构因果模型（SCM）的形式化因果推断框架。在此框架中，神经网络的计算图被直接形式化为因果图，输入数据被界定为外生变量，层激活、注意力头与权重属于内生变量，确定性计算操作则充当结构方程。研究人员必须显式定义解释的三元组结构：一是查询空间（即解释试图回答的干预性因果问题集合，例如特定激活钳制后的行为变化），二是替代模型类（即允许采纳的简化解释形式，如子网络回路或因果抽象），三是误差度量（用于衡量替代模型与真实系统输出差异的指标，如 KL 散度）。 该框架将可解释性任务转化为客观的参数估计问题，并指出可识别性本质上属于解释方法三元组的属性，而非目标模型本身的物理属性；研究者可通过收窄替代模型结构或精简查询空间来使任务具备可识别性。此外，作者主张推行零假设检验：在分析训练后的目标模型时，必须在相同架构的随机初始化基线网络上运行完全相同的提取流程，并借助蒙特卡洛抽样计算经验 p 值。在情感分类探针对比中，训练后 BERT 随网络层深呈现出探针精度的阶梯式上升，而随机基线网络的精度轨迹保持平坦或遵循纯噪声分布；在词性标注与世界坐标预测（$R^2$ 相关性评估）实验中，引入随机权重对照后，观察到的真实效应量均大幅缩减，从而剔除了底层嵌入残留带来的假象。 实用主义对立、计算开销与基准验证困境 在研讨环节中，与会者就解释哲学的路线选择展开了讨论。论文提倡“实用主义”视角，将神经网络视同气象或生物组织般的复杂系统，承认因局部最优解与 SGD 动态噪声影响，网络极少实现教科书式的纯粹算法，解释的价值在于特定任务上的功能性逼近与预测精度；反之，若走向极端的“现实主义”甚至还原主义（将模型视作单一真实算法或底层物理原子堆叠），将彻底抹杀人类理解系统所需的高阶功能抽象。与会者亦将该框架与 Neel Nanda 提出的以对齐为核心的实用主义进行了对比，指出本篇论文更接近一种元可解释性（meta-interpretability）审视。 然而，将形式化统计检验落地于现代前沿模型面临严峻现实挑战。首先，在超大规模语言模型上训练稀疏自编码器已耗费高昂算力，若叠加多重比较校正、蒙特卡洛采样与置信区间估计，计算成本将成倍攀升。其次，在非可识别性验证上，与会者探讨了借助 Google DeepMind 的 Tracer 库将明确算法代码编译进 Transformer 权重的合成基准，但此类手工编译产生的近二值化参数与梯度下降训练得到的连续网络存在根本形态差异，使得针对真实网络平滑表征开发的工具在此类基准上往往失真。此外，对于模块化加法等经典小模型工作，虽然利用数学傅里叶公式替换神经组件可验证回路因果性，但在向大规模模型迁移时，多回路冗余与特征分离等非可识别性问题仍未得到根治。 本讨论确立了将 AI 可解释性视作统计估计推断的范式转型方向，强调必须汇报置信集合并以随机权重为零假设基准。然而，在前沿超大参数模型上的庞大计算开销、联合训练可解释架构的能力权衡，以及缺乏高拟真度真实机理基准等现实难题，依然是该统计框架由理论走向普遍实践的核心阻碍。

[24:01] The inputs are not really a property of the network. It's just whatever [is put] in there and you need to specify their distribution. [27:39] It essentially turns [the] task of explaining into an objective inference problem, and when it's an objective inference problem, we know that we have mathematical tools to treat that. [41:48] You can find as many potential working sparse autoencoders as you can imagine, all produce slightly different results... that's one thing that tends to prove that there's probably not identifiability. [51:36] As long as things work, you can just put a lot of stops, and when it stops working, you need scientists to understand what's happening. [60:51] Are you open to collaborations, Maxime? [60:51] Maxime: Yeah, I am absolutely. [79:36] He used to take fully reverse engineering the neural network as the main objective, and now his main objective is trying to do whatever is useful for helping with alignment with interpretability. Ablations on humans are terrible, [98:56] I was just mostly asking Claude to do things to see how well it did... to see if Claude would get to the same conclusions I got for just letting it run experiments. [102:16] I kind of want to try taking a random model and asking Claude to try to figure out what it does to see if it comes up with some random fake explanation.

[0:00] Okay, just let me know. [0:01] >> Yeah, it should be streaming now. You [0:03] can start whenever you want. [0:05] >> Okay. Um, so hi, I'm going to be [0:08] presenting this uh preprint called the [0:10] dead elements of AI interpretability and [0:13] uh it's a preprint we submitted uh just [0:16] last week I think or two weeks ago. Uh [0:19] >> maybe put your mouth a bit away from the [0:22] mic since you're [0:24] >> is it into it [0:25] >> saturating a bit. [0:27] >> Okay. Uh, hang on. Okay, let me know now [0:31] if this is a bit better. [0:34] >> That's a little bit better. Yeah. [0:36] >> Yeah, I don't have an excellent mic, but [0:40] um Okay, just let me know if needed. And [0:44] uh yeah, this is a this is a very much a [0:46] position paper, so not a classic [0:48] experimental paper even though there are [0:50] a few experiments. [0:53] Uh and so basically uh there are four [0:56] parts in this presentation. The first [0:58] one is kind of a related work showing [1:00] what we call the statistical fragility [1:02] of AI interpretability [1:04] uh and then figuring out why that [1:06] happens and uh what we can do about it. [1:09] So uh the the name of the paper uh comes [1:12] from a pretty famous study in [1:14] neuroscience that came out in 2009 and [1:18] uh basically neuroscience researchers [1:20] put a dead salmon so a dead fish in an [1:23] MRI machine and uh so of course it was a [1:26] dead fish and they knew that it was dead [1:27] and they showed the salmon series of [1:30] pictures showing humans in situations [1:32] like in a picnic or dancing things like [1:35] that and they asked again the dead fish [1:38] to determine what emotion the person in [1:40] the photo must have been experiencing. [1:43] And what they did is that they applied [1:45] the standard analysis science to measure [1:48] brain activity on the on the salmon. [1:50] [snorts] And they found that there were [1:52] several regions of brain that were [1:54] statistically significant with the [1:57] actual task the salmon was asked to do [2:00] uh even with a very high uh very low p [2:02] value. And uh obviously it's not like a [2:06] study about magic salmon or anything, [2:09] but what was going on is that obviously [2:10] even though the measurement correct the [2:13] inference that it was flawed uh and [2:16] essentially they were just random noise [2:19] because again it was a dead fish with no [2:20] actual brain activity. And uh and here [2:25] goal of the study was to show that if [2:26] you don't correct for multiple [2:28] comparisons then you can end up with a [2:30] nonsensical result. [2:33] And the reason why this uh study was [2:36] important is that in neuroscience it was [2:38] in neurosciences it was part of what we [2:39] called a statistical revolution where [2:42] people started identifying other [2:44] statistical issues such as that one. So [2:47] typically issues where methods would [2:49] find false positives issues of [2:51] overfitting uh spirious correlations so [2:54] correlation between different phenomenon [2:56] that were not actually based on [2:58] causation. [3:00] And uh what happens is that similar [3:02] issue [3:04] for example in psychology, econometrics, [3:06] social sciences during the replication [3:09] crisis there were issues where also [3:11] people were finding that there were [3:12] things that were not done properly in [3:14] terms of statistics and that wasing to [3:16] wrong results. And so essentially what [3:19] we call dead salmon issues, it's uh [3:22] things like reading patterns where [3:24] there's actually only random even though [3:26] we seem to produce what looks like a [3:28] plausible interpretation. [3:31] Uh and in all of those fields uh the way [3:33] it is over time by building stronger [3:37] statical frameworks correcting [3:41] the entire field of causal inference is [3:43] more or less born from revolution. Uh so [3:46] that's how it was done. [3:48] And so uh what we do with this work is [3:50] that first of all we try to find similar [3:53] dead and more generally statistical [3:55] fragility the field of XAI [3:58] uh then we try to uh find what is the [4:01] where those issues happening and then [4:03] one uh possible solution which is [4:06] tentative for now. [4:08] Uh so first let's do the the related [4:10] work part and uh here's a minimal [4:13] example actually [4:16] experiment. [4:17] Uh simply put we took BERT and [4:19] randomized all of its weights pretty [4:21] much just keeping the embeddings and [4:23] then randomizing everything else. And [4:25] what we did is that we fed BERT with 300 [4:29] IMDb sentences and measured the [4:32] activation uh the average tokens and we [4:36] tried to see if we could get this was a [4:38] sentiment analyzed data set if we could [4:40] get an actual positive correlation [4:42] between the activations this now [4:44] randomized network and um the actual [4:47] label of the data. And what you can see [4:49] in the in the top half of the picture is [4:52] that we were finding a lot of similar [4:54] correlations even with very very low p [4:57] values. [4:58] Uh in the second step we did something [5:01] similar as well where this time still [5:03] kept randomized bird and this time we [5:05] trained to probe uh to we trained a [5:07] probe on different layers uh of the [5:09] model uh to see again if it could [5:11] predict the label from the uh from the [5:15] uh activations. And what we can see is [5:18] that the probe uh significantly all of [5:20] the time outperform random guessing. [5:22] We're getting u accuracy of about 0.8 [5:26] plus or minus 0.15. [5:28] Uh so we have a double double type of [5:31] analysis. On one hand we have PCA on the [5:33] top. This is PCA and the second one it's [5:35] probing. And so we have this random [5:37] network but we're still able [5:40] infer things that looks like the model [5:42] is actually doing something even though [5:44] it's again a completely random network. [5:46] And uh this mostly comes from the fact [5:48] that there is already data in token [5:50] embeddings and even though the model [5:52] does nothing after that is pretty much [5:54] preserved. Uh and that's why we can find [5:56] such correlations. [5:59] Um so this kind of dead salmon artifacts [6:02] they were found even feature attribution [6:05] methods uh where people showed that even [6:08] if you randomize pretty much all of uh [6:11] vision models layers you can still find [6:13] that salency map produce what look like [6:17] plausible explanations. So you can see [6:18] in this bird image for different methods [6:21] uh from left to right it's um [6:23] randomizing more and more layer and the [6:25] end is completely randomized pretty much [6:27] and you can see that scency map still [6:28] produce things that uh pretty much look [6:31] like the original image. So this was um [6:34] kind of the first work showing this kind [6:36] of issue. Um other similar more [6:40] generally statistical issues were that [6:44] for example explanations can be [6:45] manipulated. [6:47] Uh so here on the the top uh you can see [6:50] that we can um add adversarial [6:53] perturbations to an image and then the [6:56] either salency map or depending on your [6:58] explanation method you can produce [6:59] pretty much whenever you want. [7:02] U interpretation of neural networks is [7:04] fragile was [7:06] and then at the bottom impossibility [7:09] theorems for feature attribution. This [7:10] was theoretic work which showed that if [7:14] you take any feature attribution method [7:17] and linear so for example integrated [7:20] gradients sharp and so on uh these you [7:23] can mathematically prove that they fail [7:25] in some [7:27] can simply not be better than random [7:30] guessing and this was tested on a large [7:32] class of models. [7:35] Uh then we have probing which is still [7:37] widely uh we fairly old studies again as [7:41] well that probing can recover [7:43] information even if the [7:46] contextualized embeddings are [7:47] randomized. We know that there are a lot [7:50] of probes don't really generalize very [7:53] well and also the issue is more [7:56] fundamental probing will pretty much [7:58] check if some concept is present in the [8:00] activation of a model but it doesn't [8:02] necessarily mean that actually use [8:05] activations to do anything. [8:09] uh now more recently we have and uh we [8:12] also know a number of issues with SAEs [8:16] that they can be used to interpret even [8:18] for randomly [8:20] uh they also have generalization [8:23] issues that have been shown two papers [8:25] in the middle pretty much show that [8:28] >> [snorts] [8:28] >> uh and the last paper at the bottom show [8:31] parts of encoders are sensitive to [8:33] adversarial input perturbations. So all [8:36] of us again it's not necessarily exactly [8:37] dead points to a general lack of [8:40] robustness in those methods. [8:44] Uh we also have a concept based [8:47] explanations. [8:48] Um so one paper from 2021 is the [8:53] presents what we call interpretibility [8:55] illusions and what those are is that [8:57] essentially find spirious correlations [9:00] between activations of some neurons and [9:03] concepts. So they are they seem to be [9:04] correlation but they are not actually [9:07] causal [9:08] produce correlation. [9:10] Um then we also know that concept [9:13] activation scores can produc [9:15] inconsistentation [9:17] and in general concept based methods [9:18] they tend to have poor generalization [9:21] and they are highly sensitive to the [9:23] data set produce the explanation. [9:27] uh then a bit more recently we've had a [9:30] coal approaches [9:32] uh typically for example causal [9:33] mediation analysis which [9:36] patching so again when we perform [9:38] interventions on intermediate work to [9:42] try and measure the effect of component [9:44] and here again studies have shown that [9:46] these are experiment design and there's [9:49] also what has been termed the hydra [9:51] effect which shows that even if you uh [9:54] find a component that has an important [9:56] score. So score meaning that the [9:58] component seems to be important the [10:00] behavior that you're explaining. If you [10:02] remove that neuron or that component uh [10:05] the model still works because there's of [10:08] superpositions of states and different [10:11] redundant pathways in the computational [10:13] graph that makes it so that neural [10:15] network is still able to perform the [10:16] task. [10:18] uh then in actual mechanistic interpret [10:21] also of [snorts] [10:24] um similar issues that have been showed [10:25] more recently. So interpretability [10:28] again they seem to app appear even in [10:31] methods such as subspace activation [10:34] patching. Uh there are some studies that [10:36] have shown multiple times that circuits [10:39] don't generalize very well. uh I also uh [10:43] submitted a preprint recently that [10:45] showed that circuits that we observe [10:47] with secret discovery methods they are [10:49] sensitive to a very small minor choice [10:51] if you perturbate your data set a little [10:53] bit you can very different. [10:56] Uh another study that I presented in [10:58] this group a while ago shows that in a [11:01] single network you can find multiple [11:03] circuits that are completely different [11:05] even if the network is randomized. And [11:08] then uh another paper which I think was [11:10] also in the reading a few weeks ago [11:13] shows that even methods like coal [11:14] abstraction can produce explanations for [11:17] random networks. [11:19] Uh so all of these again uh point to [11:23] some issues. Oh and there's one other [11:24] thing that I didn't mention here which [11:26] is explanation natural language [11:29] learning [11:31] of thought things like that through the [11:32] model. uh there are also been studies [11:34] that shows that the natural language [11:37] explanation are not necessarily faithful [11:39] to the actual computation going on [11:42] inside the model. So if you had ask the [11:44] model how did it come up with the result [11:46] it will produce an explanation that [11:49] uh it actually computed the answer. [11:53] So what are those uh coming from? Well, [11:57] essentially there are two big uh issues [12:00] that we call under specification and [12:02] overation. [12:04] Under specification is the fact that [12:07] sometimes if you just look at input [12:09] output behaviors and you want to explain [12:11] that uh well simply put you can just [12:14] overfit and there are uh an infinite [12:16] number of different explanations that [12:18] can fit the data equally well. So that [12:20] is under specification. So input uh to [12:23] output data is not enough to produce a [12:25] single explanation [12:28] and overdetermination uh a bit of the [12:31] opposite and overdetermination is the [12:33] fact that [12:34] you have multiple paths in your neural [12:36] networks that are redundant and uh each [12:39] of these path is independently [12:41] sufficient meaning that work with the [12:43] other and that removing part of the [12:44] network doesn't necessar [12:47] that it will fail on the task that [12:49] you're uh evaluating. really not. And uh [12:53] if you look at generally explanation [12:54] method, we have predictive methods and [12:57] methods uh predictive methods they tend [13:00] to be of the first kind. So input to [13:03] output behaviors and therefore they are [13:05] likely to have issues of under [13:06] specification [13:08] and if you look at causal methods well [13:10] there there's the second issue. So the [13:12] problem of uh overdetermination [13:15] and uh more generally more generally [13:18] there is a a stronger concept which is [13:20] the concept of identifiability or [13:22] nonidentifiability [13:24] and uh in statistical inference [13:26] identifiability is uh the idea that if [13:30] you have a model and you're trying to [13:32] recover its parameters can you do that [13:34] uniquely from observations. So by just [13:37] observing your model can you recover the [13:39] parameters in a way that you will the [13:42] actual parameters and that there's no [13:43] multiple uh valid values [13:47] and uh why is identify not always [13:51] happening? Well, as I said earlier, if [13:54] you have poor generalization of or [13:56] overfitting, uh that means that your [13:58] models are likely to produce different [14:00] explanations depending on how it's [14:02] overfitting. Or you can have an [14:04] explanation that generalizes badly in [14:07] many ways and uh that means that you [14:09] don't know which explanation you should [14:11] select because again they don't [14:13] generalize. So it's easy to generate as [14:15] many explanation as you want. [14:17] Um if methods are sensitive to a choice [14:20] it also you're going nonidentity because [14:24] again if you change slightly the [14:26] parameters of your method you might end [14:28] up with a different observations cover [14:30] the the unique true value. [14:33] Uh and there's also cases of uh false [14:36] discovery in general. Um, and the [14:40] problem is that if you have a problem [14:42] that is not identifiable, then that [14:44] means that the bigger your hypothesis [14:46] space is, then the larger the chance is [14:48] that you're going to recover a false [14:51] explanation, an explanation that is just [14:54] peruse in general. [14:57] And uh if we look a bit at some of the [15:00] problem [15:01] that we can find in the literature. So [15:03] for example for conceptbased methods [15:06] concept activation vectors is quite a [15:08] popular thing but we can actually show [15:10] that it's a non problem meaning that [15:13] without additional [15:15] not produce a unique explanation. Uh [15:17] more generally this is an instance of [15:20] causal representation learning and that [15:23] shows that it's underspecified. Uh you [15:25] can over fit in many different ways and [15:27] produce explanation [15:30] for probing and essays. Again you are [15:33] mostly input output or input and hidden [15:36] state behavior. Uh so this is a non [15:40] nonidentifiable again because this issue [15:42] of unpecification. If you are only parts [15:45] of the network you can have an [15:46] explanation that generalizes badly in [15:49] plenty of different way and so that [15:51] doesn't let you select a unique [15:53] explanation. [15:55] Uh in the case of activation patching uh [15:57] we can actually show mathemat is [16:00] identifiable. If you do patching and you [16:02] fix your model and behavior only find [16:05] one exact value but the problem is that [16:08] people tend to over interpret the [16:10] results of causal media analysis. Uh [16:13] what this really tells you is that you [16:15] tell that one neuron or layer [16:19] certain score of importance a certain [16:21] effect score for a given behavior but [16:24] that doesn't mean that the behavior is [16:26] located in place which was the issue [16:28] that we had with uh probing earlier and [16:31] that is again a case of [16:33] overdetermination. you might remove and [16:35] still have a behavior happening [16:37] somewhere else and uh circuit discovery [16:40] again suffersation [16:42] as well. So for current methods it's [16:44] very much nonidentifiable [16:49] um and also identify [16:56] ways of explaining things where we don't [16:58] know are uh identifiable or not. This is [17:01] a very much an open problem. [17:04] So now here uh I will present one of the [17:08] proposed to all of these issues which is [17:09] called statistical causal inference [17:13] and uh here's a simple solution to the [17:16] largest dead salmon issues like the ear [17:20] we try to run it on [17:24] what seemed like random explanations. So [17:26] a simple solution is just to do [17:28] hypothesis testing against a null [17:31] hypothesis. So for example when we [17:34] explain a neural network we [17:37] produce one single explan circuit or one [17:40] single probe etc. And one simple [17:44] solution is to compare that explanation [17:46] that we're producing against the same [17:48] explanation but produced on a randomized [17:51] neural network with exactly the same [17:52] architecture. So for example, if you are [17:55] applying circuit discovery, [17:57] I don't know lama or whatever, just [18:00] apply the same circuit discovery to a [18:02] randomized model and then compare those [18:04] two circuits together and [snorts] then [18:06] mathematically we have estimate the p [18:10] value using parlo and that will score on [18:13] how much your probe is is extracting an [18:16] actual explanation where and not just [18:18] reading random data essentially. [18:22] And so if we do that uh we made three [18:25] again very small experiment it's not [18:27] really the main goal of the paper but [18:28] for the sentiment analysis [18:32] uh you can see on the left this was the [18:34] probe accuracies that we trained on the [18:36] on the neural network and um you can see [18:40] that the the actual model is in blue. So [18:43] when we train a bunch of different [18:45] probes on the neural network we get [18:47] accuracy that tends to increase. And [18:49] then if we train the same probes on this [18:51] time the randomized neural network we [18:53] those trajectories that are in gray. And [18:56] so uh essentially if you have a random [19:00] the beginning of the model because it's [19:02] simply encoded in the embeddings but uh [19:04] as you go through the random lay [19:08] whereas the actual train model obviously [19:11] is getting a better and better [19:12] representation probe increases. And what [19:15] you can see is that on the right we [19:17] measure the effect size of difference [19:19] between those two groups. So again the [19:21] the random neural networks and the [19:24] actual one and uh you can see that so [19:28] the the orange bars here are when probe [19:32] versus random guessing and you can see [19:34] that here on the left random guessing [19:35] 0.5 for a two label classification task [19:39] and that even the random network are [19:41] outperforming [19:43] and of course the actual train models [19:45] probe is get extreme high [19:49] extremely [19:51] Uh however when we do the difference not [19:53] with random guess but with the randomly [19:56] reinitialized model we get much uh [19:59] smaller effect score and actually none [20:01] of the are so the probe is actually not [20:04] performing significantly better uh than [20:07] on a random network. So that is a way to [20:10] correcting for this kind of issue. [20:12] Um we have two similar experiments here. [20:15] Uh this one is uh pretty much the same [20:18] thing but instead of being uh used on [20:21] sentiment analysis, it's simply on uh [20:24] part of speech tagging [20:26] uh prediction. And again here you can [20:28] see that if you just look at the probe [20:30] on the train model, it performs very [20:32] highly uh way better than random chance [20:35] because there is again part of speech [20:37] information encoded in the embeddings. [20:39] Uh so you get extremely high effect [20:41] score again. But if you compare it to [20:43] the same probe but trained uh on a [20:46] random randomized network then you can [20:48] see that there is significant effect [20:52] between layer one and nine but the other [20:54] layers [20:56] between your trained probe on the actual [20:58] model and on a randomized model. So this [21:00] is again the kind of uh comparison we're [21:03] looking for. [21:04] Uh and the last experiment here uh we [21:07] tried to repro reproduce the experiments [21:09] of a paper that is called uh language [21:12] models represent space and time and what [21:15] the authors did in that paper is that uh [21:17] they prompted a large languual model [21:20] place names such as New York City, [21:22] Berlin city names and then they uh [21:25] looked at the activations of the last [21:27] layer and they trained a probe on those [21:30] activations to try to predict the real [21:32] world coordinate of the [21:35] So the longitude and latitude on the [21:37] actual world map and they looked at the [21:40] the correlation so the accuracy of the [21:41] probe. So in terms of R square [21:44] correlation between the actual location [21:45] of and the one predicted from the [21:48] activations and uh again as you can see [21:50] the probe was getting very high accuracy [21:54] and uh we did the same but again [21:56] randomiz [21:59] tend to get slightly lower accuracy but [22:01] still better than random chance again [22:04] due to the fact that most likely [22:07] is some spatial information about [22:09] places. [22:10] And [snorts] uh here we can see that the [22:12] result is that here the result much [22:15] always significant but by comparing to a [22:17] random network uh instead of comparing [22:20] to random guessing we get much smaller [22:22] size. So it allows us to uh to show [22:25] things with a little bit more [22:29] uh and so the more general idea is that [22:33] if you look at uh statistics in general [22:36] the process of statistical estimation is [22:39] that you have some kind of hidden [22:41] process and you know that this process [22:42] has true parameters theta that you [22:45] cannot read directly and what you do is [22:48] that from that process you visualize [22:50] your observations. So samples the [22:53] process things like that and then you [22:55] have an estimator that takes into input [22:58] the uh those finite observations and try [23:00] to uh feed with a hat. So [23:05] parameter [snorts] [23:06] and we know that when we do that in then [23:10] all kinds of properties that we might [23:11] want estimator. So bias which is simply [23:14] put does the estimator recover your [23:17] correct parameter on average variance [23:20] across different data sets and [23:22] consistency. So if you have a [23:25] observations tends to infinity does your [23:27] estimator converge to the true value of [23:30] theta. Uh and in this case [23:33] identifability not a property of the [23:35] estimator property of a model it. Uh so [23:39] what that means is that simply put uh [23:42] identicality means if you had a [23:43] different process with a different value [23:45] of theta would you produce a different [23:48] uh distribution of patients. [23:52] Uh and so the idea is to kind of use [23:54] this kind of framework in our case uh or [23:57] explanation AI explanation. [24:01] And so the the first thing that we do is [24:04] represent [24:06] structural causal models. And structural [24:09] causal models uh are essentially just [24:11] graphs uh in which the nodes are [24:14] variables and the uh edges between nodes [24:17] they say you obtain a variable based on [24:20] its parents. And this framework works [24:23] very well. You can very easily to uh [24:25] represent neural networks because you [24:27] need a causal graph and the coal graph [24:29] in the case of simply computation graph [24:32] of a network. [clears throat] [24:34] Uh we have in a causal model what we [24:37] call ex variable external variable that [24:40] act on the system. Uh so in the case of [24:42] a neural network uh it would simply [24:44] would be the inputs. The inputs are not [24:46] really a property of the network. It's [24:48] just whatever puts in there and you need [24:51] to specify their distribution. Then we [24:54] have what are called the endogenous [24:55] variables, internal variables of a [24:58] system, [25:00] things like hidden states, outputs, a [25:02] weight, things like that. And the last [25:04] thing you need is uh what are called [25:06] structural equations which explain how [25:08] do you go how is obtained from the [25:11] previous variable, the parent variable. [25:13] And in a neural network, how to obtain [25:14] the value of a neural network? world [25:16] it's simply the propagation uh of a [25:19] previous uh weights and values. So here [25:22] it would be things like attention [25:23] activation functions so on [25:27] and then what we do is that we formalize [25:30] causal queries [25:33] and uh a coal query is basically any [25:36] question that you can answer if you know [25:38] the true kosal model. So if you have [25:41] your neural network and have the graph, [25:44] what kind of queries would you like to [25:45] know? For example, what happens? What [25:48] does the network produce inution if we [25:51] force an activation to have a specific [25:53] value or how much uh a certain attention [25:58] contributes to a certain task like [26:00] factual recall and these are generally [26:02] the kind of explanation that we're [26:05] uh so we call these causal query again [26:10] and then the the last uh thing we do is [26:13] that we formal [26:15] surrogate models [26:18] And what we say is that an explanation [26:22] a simpler description of your model that [26:25] can answer some kinds of causal queries [26:27] about the underlying [26:29] uh causal model. And so specifically we [26:32] have three components in what we called [26:34] an explanation defined this way. There's [26:37] a query space meaning what are the [26:40] queries that your explanation should [26:41] answer. So what are the things you are [26:43] trying [26:45] behavior? What kind of questions do you [26:46] want to be able to answer? [26:49] Then what we have is the surrogate class [26:51] which means which type of explanations I [26:54] which which type of explanation are [26:56] admissible. Am I looking for circuits? [26:58] Am I looking for probe looking for coal [27:00] graphs? And then we need some kind of [27:03] error measure. Uh meaning that given an [27:05] explanation, how can we measure that u [27:09] the uh that our explanation correctly [27:12] answers the queries that we are trying [27:14] to do? And so these are pretty much [27:17] things that research when you define an [27:21] explanation when you're trying to answer [27:23] some questions about you need to say [27:25] which type of question you are willing [27:27] to answer which type of explanations are [27:29] you accepting and which error measure [27:32] are you using. So these need to be uh [27:35] precise by the researcher who is doing [27:36] the explanatory work on the model. [27:39] And then to evaluate uh explanation [27:43] uh basically what we want to measure is [27:46] expected value of the the distance [27:49] between what your surrogate model [27:51] produces and what the true uh answer to [27:54] the query is. And what we say is that a [27:57] task is identifiable if there [28:01] uh a unique way to minimize uh this. uh [28:05] so essentially a unique surrogate model [28:07] that distance between the actual query [28:10] answer and the one that your surrogate [28:12] model is predicting. [28:14] Of course in practice you cannot compute [28:16] the actual because if you're working can [28:21] only use a finite amount of data uh and [28:25] even for well you can first of all [28:27] answer only a finite questions and even [28:29] for each question you can only do a [28:31] finite number of sampling inputs and [28:34] measuring for example activations. [28:36] uh but that's fine because we can [28:38] basically see this whole method and just [28:42] theoretical [28:44] estimator. [28:46] Uh since we have a finite data set we [28:48] are [28:50] an explanation that is an estimate. [28:54] So uh here's a little example because I [28:56] know [28:58] so um as you can see again the coausal [29:01] model in term of work we're going to [29:03] have some kind of input distribution [29:05] that we are assuming then the neural [29:07] network which is your causal model so [29:09] your computation graphs and then for [29:12] circuit finding we need to specify what [29:14] are the three things that we need for [29:16] explanation. The first one is the type [29:18] of queries that we want to answer. So [29:20] this depends why we're doing circuit [29:22] finding. [29:24] uh I don't know uh indirect object [29:27] identification then our queries are [29:29] going to these type of questions [29:32] then what are uh type of space what type [29:36] of are we allowing finding accepting [29:40] only circuits which are subgraphs of [29:43] your neural network and then what is the [29:46] error measure well here for uh a given [29:48] query and explanation your error measure [29:51] would basically be uh the divergent [29:53] between the distribution that your uh [29:56] model produces and the distribution that [29:59] your model produces if [30:03] so this is just one uh such examples uh [30:06] but what's really interesting is that [30:07] you [30:09] kind of theoretical framework to reframe [30:11] all kinds of different [30:14] uh results. So for example if you if you [30:17] just want to do performance benchmarking [30:19] on your model that is technically a type [30:21] of explanation It's just that your [30:24] surrogate model [30:26] of a model task then uh basically what [30:30] you're looking for a model color that [30:33] summarizes your prediction renewal [30:35] accuracy. [30:37] And here uh what would be your causal [30:39] query? While your causal query is [30:41] assuming a certain distribution of [30:43] inputs, what should be the uh the [30:46] distribution of the output actual [30:49] theoretical accuracy and then you are [30:51] going to measure that accuracy finite [30:54] number of samples and so you can measure [30:56] the error which was simply going to be [30:58] the difference between the expected [31:00] value your uh estimated value versus the [31:03] true one. You can also reframe all kinds [31:06] of things like this [31:08] probing, concept based methods and so [31:11] on, circuit discovery, causal tracing, [31:14] coal abstraction. Uh and eventually the [31:18] idea is really just for each thing you [31:20] want to do is you need to specify what [31:22] type of questions do I want to answer. [31:24] How do I formalize the causal models? [31:27] Then what types of uh answers am I [31:31] do I use to judge if a uh an explanation [31:35] is good? [31:37] [snorts] Uh so what is the what are the [31:39] benefits of doing this whole uh [31:44] theoretical I guess grounding. So the [31:47] the nice thing is that essentially turns [31:49] task of explaining into an objective [31:53] and when it's an objective inference [31:55] problem we know that we kind of [31:57] mathematical tools to treat that and uh [32:00] check uh for the check for the fact that [32:03] it's actually robust and so on. [32:06] Uh however there's one thing which is uh [32:09] it's both an additional constraint but [32:11] it's also nice. If we do the statistical [32:13] framing, it's that we need to report [32:15] uncertainty. So for example, for a [32:17] circuit, you cannot just have a method [32:20] produces one circuit here. Since you're [32:22] working with a probabilistic input, you [32:24] need to give some kind of confidence set [32:26] or posterior distribution. So you need [32:28] to qualify the uncertainty that your [32:30] method will produce. [32:33] Uh in general we know that if tasks are [32:35] notifiable tend to produce a result with [32:38] high uncertainty task is identifiable we [32:41] should get uh low uncertainty. [32:44] And the other thing that is nice is that [32:46] if weability [32:49] is a uh a root probability a property of [32:53] the method and not the model meaning [32:55] that for a given model you can change [32:58] your explanation queries that you're [33:01] trying to answer and tweak these to try [33:03] to improve of what you are doing. [33:09] uh and then in terms of future [33:12] done with it uh in coal inference what [33:15] they try to do [33:17] tasks are identifiable or not and here [33:19] is the same thing. So given a collection [33:21] of queries uh given allowed explan [33:26] divergence measurement uh can we tell if [33:29] that triple is identifiable and that [33:32] would help us uh figure out again which [33:34] task are easier or harder grand scheme [33:37] of explainability. [33:39] Um what's also nice is when [33:43] everything [33:46] approaches so for example saying in [33:48] advance you can specify a prior over [33:50] type of explanations that looking for [33:53] and then applying classical posterior [33:55] update rules to try to refine your uh [33:59] your explanation over time [34:02] and finally this is also nice [34:05] essentially opens the door to meta [34:07] analysis and cumulative science. So for [34:09] example, you can simply aggregate [34:11] evidence multiple studies and apply [34:14] those findings together to get a [34:16] stronger like a smaller confidence [34:19] interval or a stronger bound of whatever [34:21] effect you are trying to measure. [34:25] Uh so thank you very much. Uh if you [34:28] have any questions or if you want me to [34:30] go over things again, please feel free. [34:35] Okay. [34:36] >> Uh thanks for the talk. I was only able [34:38] to make some of it, but I'm glad I was [34:40] able to make some of it. Uh I have [34:42] questions about the last two slides. Um [34:47] uh let's the one that comes to mind most [34:50] right now would probably be this future [34:52] work here. So [34:54] >> um [34:56] uh can you explain a bit? I I think I'm [34:59] a bit cautious. I I definitely think [35:02] mechanistic interpretability is a field [35:04] that is like in strong need of becoming [35:07] a cumulative science [35:09] >> and it's far from it right now. Um I [35:11] think like it's not clear exactly how [35:17] um [35:19] these this beian approach that you kind [35:21] of loosely sketched out will actually [35:24] lead to that because I think there's a [35:25] lot like there's too many distinct [35:28] things that the cumulative effects of [35:31] different studies work on like too many [35:34] like there's like the comatorial [35:36] explosion is a bit too big such that I [35:38] don't think things will align so that [35:40] you relate them in this way. [35:42] Uh yeah no I would say those are two [35:44] different the theative science part is [35:48] just gathering evidence in general for a [35:50] specific behavior over different uh [35:52] different studies but the biogen [35:54] approach would really just be for a [35:56] single task that you're trying to [35:57] explain and just accumulating evidence [35:59] for that task on a single model probably [36:01] within the context of a of a single [36:03] study and yeah I agree that for the part [36:08] uh I mean it is a bit of it is a bit [36:10] speculative [36:12] because again you have so many so many [36:14] data sets and so on that it's like I [36:16] could see the the amount of data [36:18] actually being available being very [36:20] sparse and not necessarily lending [36:22] itself to this kind of analysis. Uh but [36:25] at least it opens the door to it is what [36:27] I would say. [36:28] >> Yeah. Cool. Um [36:32] let's go one slide back. Sorry I I bike [36:36] ride. I had to go bike between two [36:37] places. So I missed like the 10 minutes [36:39] before this. But um I just wanted to [36:42] spend a bit more time on this uh [36:46] sentence at the bottom which is uh [36:49] identifiability is a pro probability of [36:51] the method not the model. [36:53] >> Yes. [36:54] >> Um yeah can you just like say a bit more [36:56] words because I I missed the time before [36:58] that. [36:58] >> Yeah. So essentially uh again in if you [37:02] just look at statistical estimation [37:04] identicality is a property of your [37:05] model. It's just simply put given an [37:08] infinite number of observation of your [37:09] model uh can you always well no it's not [37:12] even that it's just if you have [37:13] different values of your parameters does [37:14] that produce a different distribution [37:17] over the observations that you're [37:18] measuring and so essentially if your [37:20] process is not identity [37:29] is not really uh I mean we we have a [37:31] slightly different definition of [37:33] identility and it's just saying that if [37:35] you have a queries you're trying to [37:37] answer. You have class of uh [37:39] explanations that you accept and uh you [37:42] have a uh a discrepancy measure. Then [37:44] the identity is a property of this [37:46] triple this triplet and say using this [37:50] triple can is there a unique explanation [37:52] that you're divergent the set of queries [37:56] uh [37:58] based on the model that you have. And so [38:00] if you have something that is non [38:02] identifiable, you can simply make it [38:03] more identifiable by decreasing the size [38:05] of the allowed explanations or just [38:07] trying to answer less queries. So this [38:10] is something you can influence on [38:11] because property of the model but of the [38:14] explanation method that you are [38:15] proposing. [38:18] Does that make sense? [38:20] >> Yeah, I think so. Um this is a big [38:23] question but obviously my question is [38:25] motivated by my own research. Um [38:28] uh what do you think about the [38:33] plausibility or tractability of kind of [38:36] jointly tuning the model and the method [38:41] conjunctly [38:42] um to uh allow greater identifiability. [38:47] >> Uh so training the model at the same [38:49] time as the middle. Yeah. I mean uh I [38:53] mean yeah I don't see why that wouldn't [38:55] work because I mean there are a number [38:57] of approaches where you I mean this is [38:59] what a lot of people do is just try to [39:01] train the model to be more explainable [39:03] that's one way to do it [39:04] >> but if you can also train uh so refine [39:08] your the type of explanation that you're [39:10] producing yeah that should be another [39:11] way of doing it see any immediate issues [39:14] with that [39:17] >> uh yeah I I have more questions but if [39:20] anyone else wants to ask and I don't [39:22] want to I want to let them ask ask well [39:25] first um [39:28] >> I have just one question so you listed [39:32] almost every method of that's used in me [39:35] interp except I guess for tearing [39:38] vectors and I'm wondering what what do [39:40] you think how this could be you know [39:43] basically applied to also make steering [39:45] vectors more empirical like in a sense [39:47] we sort of already do it by having [39:50] random vector and seeing if that does [39:52] something as well. But other than that, [39:54] maybe we could improve on it as well. [39:55] Like what's your take on that? [39:57] >> Uh so I will admit that I'm very unear [40:00] vectors in general. It's kind of black [40:02] spot I have in in my literature work. Uh [40:07] but well again I guess the the simple [40:11] thing to do would be like to begin with [40:13] just the comparison like the hypo [40:15] testing again like you compare with I [40:17] don't know a random vector and you see [40:18] if you get something that really [40:20] different. Uh I think that would be a a [40:23] pretty good start to that. Uh and can [40:26] you can you reframe the whole idea of [40:28] steering vectors into this? Yeah, I mean [40:31] the the framework should be able to [40:32] accommodate that and you can uh try to [40:35] quantify exactly uh what kind of again [40:38] what kind of questions are you [40:39] answering? Uh and then here your [40:42] explanation class would be like a set of [40:44] vectors essentially. [snorts] uh and the [40:46] hard part may be the measuring part but [40:49] the the yeah the basic thing would be at [40:51] least doing the hypothesis test against [40:53] the baseline and seeing if you get I [40:57] don't know if there are already studies [40:58] that do this kind of but that would be [41:01] the first thing that would come to mind [41:05] >> yeah yeah that was my question so [41:07] basically something like null hypothesis [41:09] exists for steering vectors which is [41:11] that you try to see if adding a random [41:13] direction vector would also [41:16] make a similar change which is like a [41:18] one sanity check that exists but yeah [41:20] based on uh your presentation here more [41:23] could be done in that direction I guess [41:25] yeah [41:25] >> yeah most likely I mean always can I [41:27] mean more can always be done but the [41:28] problem is that again it makes it harder [41:30] for the you don't just blindly apply the [41:33] method but now you need to say exactly [41:34] what do I want to answer and how to [41:37] answer it so it it shifts the burden uh [41:39] in that way but I think that's really [41:41] the way to go uh rigorous [music] [41:43] mechanistic interpolity in general. [41:48] >> Okay, I'm back on my laptop now. Uh, [41:50] does anyone else have any questions? [41:53] >> I I had some as well. Um [41:57] I guess this is more specific like uh [42:00] just wondering for if if for instance if [42:03] I get it correctly sometimes you can [42:04] train you can train sparse autoenccoders [42:07] on on transformers and or on language [42:10] models and if you increase the [42:11] dimensionality of the sparse [42:13] autoenccoder you can possibly get like [42:16] more granular features basically. So [42:19] that in a way is a form of nonidentified [42:21] ability right? [42:23] >> Yeah. Yeah. I mean again the thing is [42:26] that you're just trying to predict one I [42:28] mean a series of output from just one [42:32] activation layer or or what if you just [42:36] >> you can have a whole way to just make [42:38] something that doesn't generalize on [42:39] other types of [42:42] you can find as many potential working [42:44] sparse autoenccoders as you can imagine [42:46] all produce slightly different result. [42:48] So yeah, what you're saying is [42:51] changing the dimension of a hidden state [42:54] uh you end up with different concept and [42:56] things and yeah that's one thing that [42:58] tends to prove that there's probably not [43:00] identifiability. [43:01] >> Yeah. And also all sorts of like uh [43:04] configurations like that that can [43:07] produce different feature splitting for [43:09] instance different reconstruction loss [43:12] problems and so on. So that seems to be [43:14] very very brittle in terms of uh this [43:17] nonidentifiability. [43:19] That's good point. Yeah. So we can [43:21] mistake ourselves. Oh, this is the model [43:23] what it's actually doing in the sparse [43:24] to encoder in the attribution graph. But [43:26] for but in reality this is just one [43:29] possible train spar encoder. [43:31] >> Yeah, pretty much. But that I mean [43:32] that's also normal like you need to [43:35] recent mechanistic like five six years [43:38] old. So if we compare to other like [43:41] natural sciences that 50 100 200 years I [43:45] think it's normal that we have we're [43:46] doing this kind of sort of baby step but [43:49] uh I think part of the growing up part [43:51] of mechanising is going to understand [43:53] this kind of issues and trying to solve [43:55] them. [43:57] >> Yeah [43:58] alo okay maybe more general question [44:01] there but thank for the answer there. [44:04] That's true. and is is uh like I think [44:08] at least I as far as I understand it [44:11] like in science like to truly identify a [44:13] causal relationship you have to uh do [44:16] some kind of intervention uh and it has [44:18] to be done correctly. So I wonder if you [44:21] have like some uh some good uh some good [44:23] rule book uh that like uh to do it [44:27] correctly basically to not fall to not [44:29] delude oursel as Nanda would say. Well [44:32] uh that is one of the things that I I [44:34] did mention [44:38] I don't remember where that was uh but [44:40] that is a uh yeah that is a uh a causal [44:44] method right your actual patching so [44:46] you're intervening and that if you your [44:48] component or change this activation with [44:50] a zero or bean or noisy baseline uh [44:54] whether you have a difference so uh I [44:57] mean there there's a lot of perspective [45:02] But I think we're currently already [45:04] doing some of the nice things. The hard [45:06] part again is not [45:09] just because you have a causal effect [45:11] neuron does that mean like yeah you can [45:13] prove mathematically again effect this [45:15] neuron has an effect but again that [45:16] doesn't mean oh that's where my behavior [45:18] is because you can remove that network [45:20] that neuron and still have the behavior [45:22] happening in other so uh it can be used [45:25] to quantify things well but I think the [45:28] hard part is jumping to the [45:29] interpretation [45:31] uh and that I don't know if coausal [45:34] causality in general has any great [45:36] answers to that. Uh I work a lot with [45:38] causal abstractions. So I think uh a [45:41] nice idea in general is to to give [45:43] explanations as abstractions and then [45:46] you can kind of try to uh verify [45:49] a study by uh s and um [45:53] Julian Mer has shown like it's it's also [45:56] has other type of so yeah I don't really [45:59] know but in general yeah if you want to [46:02] if you go be [46:06] You're you're jump cutting off a bit. I [46:09] think my sensitivity is too too low [46:12] basically. [46:13] >> Oh, sorry. [46:14] >> Okay. [46:16] >> Yeah. But essentially, yeah, you if you [46:17] want to go beyond correlations, you need [46:20] to perform some kind of interventions [46:22] that otherwise you're not even at level [46:24] you're at level zero because it's not [46:28] >> Yes. [46:33] >> Okay. Uh [46:34] >> but but the hydra effect is a thing for [46:36] example. [46:37] >> Yes. Yeah. It is very much [46:39] >> like what if I can give an example there [46:42] like what comes to mind is that people [46:44] thought that like uh [46:48] okay I'm not sure okay well well the [46:50] example that came is that people thought [46:52] that uh smoking causes no no that [46:54] alcohol causes lung cancer but actually [46:57] uh the correlation the correlation was [46:59] there because people who smoke also [47:01] tended I mean people who took alcohol [47:04] also tended to smoke so that's why the [47:06] correlation basically [47:08] But like for instance if that's if in [47:10] that person you would you would ablate [47:12] the lung uh then or okay that's not a [47:17] good idea that's yeah ablations on [47:20] humans are terrible basically you can [47:22] [laughter] [47:24] >> trying to think of other examples where [47:26] you can like see see like ablation [47:29] actually doesn't give you close answer [47:31] >> well here the intervention would like [47:35] [laughter] Yeah or that but I guess in [47:38] that case yeah if you want to show if [47:39] you have a person who's uh smoking and [47:42] drinking and you see that sore cancer [47:44] well well you cannot really do it on one [47:46] person your entire population but here [47:48] the intervention would be make a part of [47:50] a group stop smoking or stop drinking [47:53] and see if [47:55] >> uh okay well you need a control group [47:57] who doesn't have cancer because stopping [47:59] smoking is not going to make your cancer [48:01] disappear but you get the idea. [48:03] >> Yeah [snorts] yeah yeah okay that works. [48:07] ablate the other causal variable. [48:09] >> Yeah. [48:10] >> But the issue is with the alcohol thing [48:11] you you [48:13] they didn't think about the other causal [48:15] variable. Sometimes in make we wouldn't [48:17] like we wouldn't think oh actually the [48:20] causal variable is somewhere else and we [48:22] wouldn't look for it and just [48:24] >> but that's why I mean that's why like [48:26] the [48:29] especially these kind of questions in [48:31] econometry and science if you want to I [48:34] mean the field had to mature to be able [48:36] to see is something really a causal [48:37] factor for a disease or a phenomenon. [48:43] >> Yeah. like that's that's how the whole [48:46] the entire field of coal inference was [48:48] born really. [48:51] >> Thanks. Okay, makes sense. [48:54] >> Uh I have a question. Um just curious [48:57] like uh when did you work on this [49:00] project over what time period and who um [49:04] >> yeah like who helped you the most I [49:07] guess? [49:08] >> Uh so this was mostly done with my [49:11] adviser Maxim Per. So the other maxim um [49:13] here at the end and uh we've been I mean [49:17] we've been working on this for [49:19] essentially my previous were kind of uh [49:22] showing those dead summons issues. So [49:24] the the paper that I one of them was [49:26] showing nonidentifiability of circuit [49:29] discovery and then recently I had [49:31] another try to show the the variant [49:35] circuit methods in general. I was like, [49:37] "Oh, so interesting. We have we have [49:39] identity issues. [49:41] Be interesting to try and reframe all of [49:43] that as a statistical problem because we [49:45] have the same type of issues they [49:47] encounter in statistics. So really [49:49] working on this paper uh it wasn't very [49:53] probably like a month or two, but there [49:55] was a lot of uh um earlier work or like [50:00] preparing work that was done over since [50:02] the the beginning of my PhD pretty much. [50:06] Well, um, is this the is this a final [50:11] installment or is there like more like [50:14] the series? [50:15] >> Well, there I have another big work [50:18] coming up which is uh but it's it's more [50:21] on causal abstraction [50:23] statistical part. Um, however, I think [50:25] it would be interesting to try and do [50:27] some Beijian coal stuff. So this might [50:29] be a problem that I something I would be [50:31] interested in working on. Just see if we [50:33] can do like I don't know Beijian circuit [50:35] discovery, Beijian probing, whatever you [50:37] want. Uh so this might be [50:41] very serious the studies part. [50:45] >> Cool. Um [50:50] rectifies Victor, do you have any [50:51] questions? [50:56] >> Yeah, not really. Uh I do want to [50:58] mention that there is some uh mic [51:00] artifacts on the that but I'll probably [51:03] be able to fix them on the recording. [51:05] >> Uh yeah, I'm very sorry. I'm with my [51:06] work headset which is extremely poor. [51:10] >> Yeah, don't worry. I think some of it I [51:12] might fix it for YouTube later. Uh but [51:16] yeah. [51:16] >> Okay. Thank you. [51:17] >> Yeah, I think just the uh mic [51:20] sensitivity like in the Discord setting [51:22] you could uh lower it so that it catches [51:25] up your voice more often. [51:27] uh even if you're speaking slightly less [51:29] loud, slightly quieter. [51:32] >> Yeah. [51:33] >> Yeah. At the moment, [clears throat] [51:36] >> there's also like the small noises for [51:38] but but I think the small noises like [51:39] for the you know when you talk that's uh [51:42] probably fixable. Uh but yeah, [51:45] >> by the way, do you think uh I think [51:47] something that could be useful for [51:48] Mkinter people uh if if you want to jump [51:51] from this topic, Victor, uh is is uh is [51:54] uh if you could like take this uh and [51:57] make like some kind of cheat sheet or [51:59] something like for Mckinel researchers [52:02] or something uh like uh here's do this [52:05] new hypothesis or uh think about think [52:09] about like the non fair issue etc. like [52:12] I think that would be like useful for [52:14] [laughter] [52:15] yeah the whole M inter field I think [52:18] needs more rigorous statistics and well [52:20] generally the whole machine learning [52:22] field needs more statistics uh so [52:25] >> uh I think uh yeah [52:28] >> yeah that's yeah that's true that's [52:31] something we can look into propagate the [52:34] good word [52:36] >> yeah but but but it's funny in a way [52:39] like how you know how like uh like how [52:44] in the in the beginning like when deep [52:46] learning people started scaling [52:48] everything up and not caring about [52:49] proper statistics and just benchmarking [52:52] it's and and it somehow works but now [52:55] we're trying to understand it and those [52:57] methods are kind of problematic a bit uh [53:00] if we don't have regress statistics to [53:02] understand it. Uh [53:04] >> yeah it's always like where like as long [53:06] as things work you can just put a lot of [53:07] them [53:10] stops and when it stops working you need [53:11] to scientist to understand what's [53:14] [laughter] [53:15] >> like do do you like last two weeks ago [53:19] or something like that in mech [53:20] interpreting group we were looking at a [53:22] paper that started using language models [53:24] to interpret language models the I think [53:27] it's called activation oracles is it [53:29] called like that Victor or do you know [53:31] it [53:32] >> yeah it's activation oracles [53:34] So that makes me kind of think oh okay [53:36] well now we are using blackbox methods [53:38] to interpret a blackbox method that [53:40] doesn't seem to be like what I'm not [53:43] sure what to think about that like from [53:45] a from this statistical perspective or [53:47] like this more rigorous perspective it [53:49] doesn't seem to be really rigorous as in [53:51] like nonident nonidentifiability [53:54] for instance seems to be like very big [53:56] there like you know you can train [53:59] neuronet network in so many different [54:00] ways if uh if you're using lang language [54:03] model if you're training language model [54:05] and train language model for instance or [54:07] in that case it was fine dealing so [54:12] >> I mean there was a I remember that there [54:14] were a short period of time thankfully [54:16] it was pretty short where people were [54:17] even uh trying to do cycle [54:21] so they were trying to understand the by [54:23] giving it like personality tests and and [54:25] things [54:26] um so yeah as you can imagine that's not [54:29] really [54:31] >> yeah or the IQ tests are very viral. [54:34] >> Yeah. Yeah. Yeah. Stuff like that. [54:36] >> Yeah. I can imagine that that's that's a [54:38] big issue [54:41] statistically. [54:41] >> I mean for the activation oracle kind of [54:43] thing maybe just kind of works. I think [54:47] like you could have identifiability [54:49] there if like the model actually gives [54:52] different explanations for you know the [54:55] for different models and actually works. [54:58] But I think the hard part there will be [55:00] being sure that that's actually the [55:03] case. I I I guess it will be interesting [55:06] to like uh take some of the activation [55:10] oracle models and uh you know ask ask it [55:13] the same questions for like a random [55:15] model uh or something [55:17] >> uh or even not a random model. Maybe [55:21] something [55:22] erh it will be a better erh you know [55:26] test of this kind of thing uh to try [55:31] things on some real model but that has [55:35] been train on a completely different [55:37] data set that is about something else. [55:40] Uh right like you ask like you take [55:44] alpha follow or something and you use it [55:47] on the [55:49] uh [55:50] uh on the [55:53] you know on on the uh some language [55:56] question uh because maybe like maybe you [56:01] have some method that seems like it's [56:03] not giving for responsing a completely [56:05] random model because the completely [56:06] random model is very different from real [56:09] models in some Okay. Uh but maybe you [56:11] could have some real model that you are [56:13] pretty sure doesn't have whatever the [56:15] thing you're looking for is uh you know [56:18] because maybe it's a narrow model that [56:20] is string on some other set that doesn't [56:22] have it. Uh and maybe that will be a [56:24] cool test of some things. [56:27] >> Yeah, for sure. I mean [56:30] yeah you don't I mean you don't need to [56:32] always necessarily adopt the full [56:34] statistical framework. even like this [56:37] type of comparative studies would be a [56:39] great thing to have. [56:42] >> I'm gonna I'm gonna put my conspiracy uh [56:44] theorist hat on right now and I'm going [56:46] to throw myself into the wild. I uh [56:51] I reject I cautiously reject Verie's [56:54] statement that to truly identify a [56:57] causal relationship has to do an [56:58] intervention. [56:59] >> Cautiously reject. [57:02] Can you elaborate? [laughter] [57:05] >> Throwing myself into the into the pool [57:07] of sharks here. Um, [57:10] uh, let's let's I'm going to put a very [57:13] extreme example. [57:15] Um, okay. And of course, we're very far [57:17] from this extreme example. [57:18] >> I think I think I agree with you, but [57:21] >> hear me out. Hear me out. Hear me out. [57:23] If the model is linear, you do not have [57:26] to perform an intervention. [57:29] uh because we can read from the weights [57:32] directly uh through through SVD for [57:35] example or or through other means we can [57:37] calculate the causal relationship [57:40] directly. [57:41] >> Okay. [57:42] >> Um [57:43] >> that makes sense. [57:43] >> This is an example of a very simple [57:46] neural network where uh you do not need [57:51] an intervention to truly identify a [57:53] causal relationship. [57:54] >> Okay. I I understand we are we are in a [57:56] very far world from this. [57:58] >> [laughter] [57:59] >> Like yeah like that makes sense. I I I [58:03] think like I guess it depends also like [58:05] on the field. I guess for instance in [58:07] medicine like you cannot really [58:08] calculate it from humans from human [58:11] states that you can measure. So yeah it [58:13] makes but it makes sense in in if you [58:15] have a such a simple model that you can [58:18] in a way solve it analytically or and so [58:21] or like numerically or something like [58:22] that. [58:23] >> Yeah. Yeah. I think having well I also [58:28] have a hot take about this [laughter] [58:30] which is like uh doing interventions [58:33] helps but like in principle [58:37] you [58:39] uh I guess like I I it's not like uh [58:45] interventions are like magically [58:48] separate from other interventions or [58:50] something. Uh [58:53] and if you imagine some like idea leg [58:56] agent looking at data like Solomon of [58:59] indivision or something uh it can figure [59:02] out causality just from observations. [59:06] Uh so [59:09] uh I think this relates more to like [59:11] people saying that neural networks won't [59:13] be able to to learn like uh causality or [59:16] whatever only from text data uh more [59:19] than in practice people needing [59:22] interventions to figure out stuff. Uh [59:24] but yeah. [59:28] >> H [59:30] yeah. Another thing I came up with just [59:32] now is just like okay I don't know how [59:34] well this ties into things but like [59:36] let's say you had a model and you in in [59:39] advance you did not know that it was [59:40] actually like a linear transformation. [59:42] Okay you didn't know you just you [59:45] someone gives you like an API to a [59:47] neural network and you're like okay [59:48] figure out the causal relationship. Then [59:50] you're going to like do some causal [59:51] thingies. [59:53] How many like interventions [59:57] do does it take for you to be like with [59:59] very high confidence that it has some [60:01] structure such as linear linearity? I [60:03] don't actually know how to answer this [60:04] question. [60:05] >> It ties very well into things because [60:06] that's basically paper more or less. [60:09] >> Right. [laughter] [60:12] Right. You have to [60:14] yeah you can start with with a prior [60:16] with Beijian prior uniform Beijian prior [60:19] and then just slowly update with each [60:21] new evidence [60:24] >> but yeah I mean that's what I'm working [60:26] on is essenti [60:29] oh yeah it's cutting that's weird uh you [60:31] have your neural network then you have [60:33] your explanation which is going to be a [60:35] causal model and the goal is to have a [60:37] metric that tells you how well aligned [60:39] they actually are and the goal is to [60:41] also have a confidence interval also [60:42] like how many interventions it takes you [60:44] to detect if your explanation is not [60:45] actually correct. [60:48] >> Yeah, [60:51] this sounds very useful. Definitely like [60:54] what I'm what I'm also thinking of that [60:55] this if you would apply this to making [60:57] interp like you know how expensive is it [61:00] to like for example train a sparel [61:02] encoder on frontier models. So like if [61:04] you want to do all sorts of rigorous [61:06] statistics like this the computational [61:08] budget will go very far as well. Yeah, [61:12] for sure. [61:12] >> Is an issue. [61:16] It's also interesting how how liner [61:19] probes on on the random in it networks [61:22] are are still so good compared to random [61:24] chance. I I wouldn't [61:26] Yeah. [61:26] >> Yeah. I think it's just crazy the amount [61:28] of stuff that is [61:31] we just don't realize [61:33] >> that just I wouldn't expect that and [61:36] like so like and then you can measure [61:39] the statistical significance I guess. uh [61:41] between the liner probes on random [61:43] minute network and on the actual network [61:45] some other one. [61:46] >> Yeah. [61:46] >> Yeah. [61:50] >> Just [61:51] >> are you open are you open to [61:52] collaborations Maxim? [61:53] >> Yeah, I am absolutely [61:55] >> cool. Um how do I reach out? Did I DM [61:58] you or what? [61:59] >> Yeah, just DM me on. It's probably the [62:01] easiest. [62:01] >> Excellent. Excellent. Cool. Cool. [62:07] Nice. [62:09] Uh, rectifies. You haven't said [62:12] anything. Are you Are you alive? [62:14] >> Not to call you out or anything, but I'm [62:16] calling you out. [62:20] >> Might be. [62:20] >> You mean Nika? [62:23] >> No, [62:25] >> the one right below them on my screen. [62:27] Anyway, okay. They don't have they don't [62:29] have to speak up. It's fine. [62:31] >> They can they can do observation without [62:33] intervention. [62:35] [laughter] [62:39] That's nice. [62:41] Oh yeah. [62:42] >> Uh any other questions or [62:48] >> Yeah. You also said that some people are [62:52] trying to build build more interpretable [62:54] models from scratch like for instance [62:56] the sparse transformers from OpenAI [62:58] which incorporates the spar. [63:01] >> I am on this train by the way of [63:03] architectures. So, so yes, same in a [63:08] way. [63:09] >> Yeah, I mean that's the other thing. If [63:10] you can't interpret things, just train [63:12] them to be interpretable. [63:14] >> But seems like at least at least in the [63:17] open architecture like uh that we went [63:20] through like two months ago or something [63:21] here. Uh seems like there's a trade-off [63:24] between interpretability and capability [63:26] at least in in terms of deep learning [63:27] basically. So um that that seems to be [63:32] an issue in terms of like you know [63:34] commercial interests and so on being [63:37] more interesting capability being more [63:39] interesting to most people than like [63:41] actual scientific understanding and [63:44] explanations basically what the is [63:46] trying to do. So [63:48] >> yeah, that's [63:49] >> I think I think that's also it really [63:51] really makes sense that this trade-off [63:53] exists because I mean interpretability [63:55] is essent [63:58] like how humans can understand very [64:00] subjective and if you want something [64:03] with better capabilities than you then [64:05] probably it's not something that you can [64:06] explain easily to a human if a human [64:08] could just do that immediately right. [64:12] >> Yeah. Also, so so a problem is when you [64:15] are doing things like training the [64:17] models to be more interpretable is that [64:19] like uh I mean maybe you'd have a method [64:22] where you have the model and train it to [64:25] be more interpretable without changing [64:26] how the model works but otherwise [64:30] uh like what we actually want is to [64:33] interpret the ones that people use in [64:36] practice and then uh we need to be able [64:41] to to interpret those even if they are [64:43] not trained to be interpreter or [64:45] whatever but at the same time I think we [64:48] could probably learn uh very like we [64:52] could learn probably learn thing if we [64:54] had some mod right now we don't have [64:56] know really know how to really [64:58] understand any big model and if we had [65:01] some big language model even if it was [65:04] worse or it was something like the [65:06] sparse thing that the open did that we [65:09] actually understood we probably could uh [65:13] learn a lot of things that generalize to [65:15] to you know whatever models people [65:17] actually use. [65:21] >> I think I think the the really big uh [65:23] constraint like I think I think I think [65:27] the OpenAI approach is wrong. Okay, I'm [65:29] okay with this dis disagreeing with with [65:31] Open AI and Big Labs. I I think interpol [65:34] architectures is a is definitely a play. [65:36] I think their play is wrong. um uh in [65:41] the sense that they just trade off [65:43] massive amounts of of performance. I [65:46] think like I think we should I think [65:50] there are ways we should we should [65:51] specifically search for things that [65:54] enhance interpretability [65:55] that uh uh are not extremely harmful to [66:02] performance. [66:04] Maybe we could find a kind of trade-off [66:06] where training gets harder but inference [66:08] doesn't get worse or something like [66:10] that. [66:10] >> Yeah, I think these are all all valid [66:12] approaches. I think like I think we're [66:14] we've not invested enough like thinking [66:18] power into exploring uh I I definitely [66:22] think that improving interpretability [66:25] there's definitely a trade-off. There's [66:26] no such thing as like a free lunch here. [66:28] It'll cost something. Um, and the [66:31] easiest resource, the most e the easiest [66:34] to articulate resources is performance, [66:37] but I think there are nice favorable [66:40] trade-offs and we should be searching [66:42] along this PTO frontier. That's that's [66:44] my take. [66:47] >> I think [66:49] it it will be nice if we have a model [66:52] like Yeah, it would be nice if we had a [66:54] method like that. But that also if [66:57] people find some right now I would also [67:00] be okay if someone finds some method [67:02] that has huge tradeoffs but actually [67:04] works actually lets you interpret the [67:06] model h very well because we right now [67:10] we don't even have that and I feel like [67:13] that would already be produced [67:14] >> linear thinker bilinear [67:19] >> think about it it's the I've already [67:21] calculated how much extra comput it [67:23] costs it's 10% [67:26] It's not that much. I mean, okay, maybe [67:28] maybe for a billion dollar training runs [67:30] is $100 million. Sure. But like, you [67:32] know, like as a multiplier of the [67:35] original compute spent, it's not that [67:37] high. [67:42] Anyway, sorry, I'm just like I'm just [67:44] whispering. I'm just [laughter] like [67:46] basically [67:50] >> anyway [67:50] >> there there's a question in the chat [67:52] from the guy who didn't talk. [67:54] >> Oh yeah. [67:55] >> He said yeah I will I will verbalize it. [67:58] Uh quick question. I see the paper [67:59] advocates largely for a pragmatic [68:01] approach rather than a realist approach [68:03] to interpretability. Wanted to ask why [68:05] like just skipping through it. If [68:07] there's no true explanation to how model [68:09] works and just approximations, what does [68:11] it mean for alignment and how can we [68:12] trust interpretability based cases if [68:14] the explanations are pragmatic? [68:18] >> Uh, wow, that's a that's not the [68:20] simplest question. Um, [68:24] so we do advocate for pragmatic approach [68:28] because [68:30] actual No, it's a good question. It's a [68:32] it's an excellent um [68:38] yeah uh I'm not even sure how to [68:40] honestly [68:42] um I mean the thing is I think there are [68:45] cases we're going to have to decide with [68:47] pragmatic explanations because I don't [68:51] >> [snorts] [68:51] >> um realistic ones for sure if I just [68:54] don't think the current methods are [68:56] anywhere near enough to do this kind of [68:58] thing. I mean think about it. I I think [69:01] one thing is that neural networks are [69:03] essentially complex systems right just [69:05] like uh I don't know like they are [69:07] dedicated to having entire fields [69:09] complex system [69:11] uh just like you have neuroscience that [69:13] studies the brain meteorology that [69:15] studies the web and the thing is that [69:19] neural networks are kind of yes [69:21] complexes that we [69:24] because we can measure what's going [69:26] inside them infinite precision can them [69:30] as much as we want. Uh and so it's it's [69:34] kind of like the model like can use for [69:38] explanation of scientific systems in [69:40] general. So uh yeah, if we find that [69:43] there's no [69:45] like how to explain brain explain [69:49] something like a neural network where we [69:50] can actually measure accuracy and so on. [69:56] Uh I I keep saying that my icon flashes [69:58] so I don't [70:00] >> Yeah. Per cutting. [70:01] >> No. Yeah, it's okay. I mean, at this [70:02] point it's fine. Like [laughter] [70:04] >> Okay. Okay. [70:05] >> We're not We're not running it back an [70:06] hour and 15 minutes. So [70:08] >> Okay. Yeah. [70:09] >> But I think [70:10] >> we will make do with what we have. [70:14] I feel like during the while when you [70:17] were explaining the paper, you were [70:19] mostly understandable and like there is [70:21] a few annoying extra noises, but I I can [70:25] probably fix that later. [70:28] >> Okay. Well, that part works. That's the [70:30] best. [70:32] I have a question of like um [70:36] what [70:39] okay maybe this is just me like not [70:40] understanding properly like pragmatic [70:42] versus realist seem similar to me in my [70:45] head [70:46] um [70:48] what maybe pragmatic I don't know if [70:51] pragmatic is the right word here um like [70:55] I don't see [70:58] I'm a bit confused [71:01] hm M [71:01] >> unless Maxim got it. [71:04] >> I mean I I I think by realist it was [71:07] more like theoretical or like actual [71:09] guarantees of understanding things. [71:11] That's the part that I meant versus [71:13] pragmat. Yeah, I think pragmatic and [71:14] realist are the same thing. It's just [71:16] probably the person meant the yeah more [71:18] like theoretical by realist is what I [71:21] assumed at least. [71:23] >> Okay. I I more I more understood it as [71:26] like in terms of like predictive power [71:29] as in like really would be like ultimate [71:32] predictive power in a way and and [71:34] pragmatic is fine with error bars etc. [71:38] But yeah, I'm not sure realist is [71:41] realistic. [laughter] [71:43] Um, so or maybe you want to define it. [71:47] Rectifies in the chat. [71:50] >> Yeah, he's trying to define it now. [71:52] >> Yeah, [71:53] >> this would be like network has an [71:55] algorithm and and our job is to find it. [71:58] >> So basically upsold certainty. That's [72:00] what I mention. [72:03] >> Okay. M [72:06] >> yeah I don't necessarily know that [72:07] realist is the right word for that [72:11] >> that is certainly a take that is [72:12] certainly a position held by some people [72:16] I don't think it is held by everyone [72:18] >> maybe held by [72:21] >> like I feel like uh you can say that [72:23] this would be the the ground truth like [72:25] if you exactly want to one matched [72:28] exactly the dynamics in a way so like [72:32] And but but uh but in practice [72:36] everything we have so far is like as far [72:38] as I understand different approximations [72:40] for different parts of the network like [72:43] different behaviors from different [72:45] angles with all those issues that you [72:47] are showing in this paper basically. Um [72:52] >> yeah, [72:53] >> I mean I guess also we have to consider [72:55] that even if in theory these problems [72:58] would have perfect solutions, networks [73:00] don't actually implement perfect [73:02] solutions cuz you know they go by local [73:04] minima. So they might actually [73:06] legitimately not be implementing a [73:08] perfect algorithmic solution even if one [73:10] exists in theory. [73:13] >> That's true. Um [73:14] >> like [73:15] >> okay yeah yeah I think pragmatic is [73:17] there's no single true algorithm. [73:19] There's only explanations that work [73:21] better or worse purposes. I mean, I [73:23] think there's space in the field to be [73:25] taking both approaches. [73:26] >> Yeah. [73:27] >> Absolutely. [73:28] >> Like so [73:31] I I'm just happy that people that are [73:34] people are actually doing both [73:36] approaches, right? Uh the realist like I [73:38] think open AI's work is quite similar to [73:41] to network has an algorithm and our job [73:43] is define it except with an asterisk [73:45] that they are conditioning the network [73:46] to have this property. [73:48] Um and then uh like pragmatic I think [73:58] in a sense your description of pragmatic [74:00] is the the more realistic view of uh in [74:04] terms of like [74:06] how the field will will evolve into. Um [74:11] but anyway yeah like I'm team pragmatic. [74:15] I like that, you know, but but I think [74:17] it's good that we have both. [74:19] >> I think it also really depends on why [74:22] you are trying to explain your system in [74:23] the first place. If you're just trying [74:25] to do like predictive stuff, then like [74:26] yeah, you probably find pragmatic [74:28] approach. But I would also argue that [74:30] science in general is kind of trying to [74:32] understand systems as much as we can [74:34] like really get to the heart of it. And [74:35] I think for that you would need probably [74:37] more of what you call the realist [74:39] approach. Uh but I don't know if it's [74:40] actually doable in practice. That's the [74:42] main thing. [74:45] Medicine seems to be very pragmatic [74:49] [laughter] [74:50] [snorts] [74:53] >> yeah especially medicine targeting the [74:56] brain [laughter] [75:01] but that's maybe because I've explored [75:03] that one the most and I don't see the [75:04] the issues in all the other sub fields [75:07] yeah [75:13] Like [75:15] it it feels to me kind of like that even [75:18] like the only like quoteunquote realist [75:21] thing would be exactly the network on [75:23] itself like as it strain with all its [75:26] weights in all its blackbox complexity [75:30] >> and like maybe not but [75:32] >> I [snorts] feel like at that point uh [75:35] >> there is some isomorphism maybe [75:37] >> or or you might as well say that the the [75:41] true realist thing is a complete [75:43] description of all of the atoms in their [75:45] computer or whatever. And like [75:47] [laughter] [75:48] you you kind of [75:51] like remove the concept of explaining [75:54] something which in the end it is about [75:57] having some high level abstraction that [76:00] is kind of works like that most explains [76:04] most of the things. uh even if you have [76:09] some noise or whatever that you don't [76:12] care about. Uh right like you can [76:15] explain a human body by saying by either [76:19] saying it has organs at those that do [76:22] specific things at their steps or [76:24] whatever or you can explain the the the [76:27] human body as a giant pile of atoms that [76:30] follow the laws of physics. uh and those [76:33] are correct [76:35] >> the in some sense the the BM is more [76:38] correct that is uh that there are other [76:43] things that you are not going to capture [76:45] on the other thing uh but that's [76:50] uh not like when we talk about [76:53] understanding something we're already [76:55] thinking about higher levels of [76:58] subtraction and ignoring some detail and [77:01] things like that [77:04] >> the the one detail is also here when [77:07] we're doing mech inter we're not trying [77:09] to explain usually the entire network [77:12] but more like the behavior of the [77:13] network on one specific task so I don't [77:16] think it's uh it's not necessarily an [77:17] insane thing to think that surely we can [77:20] do better than having the full [77:21] computation surely there's all kinds of [77:23] parts that are inactive even with poly [77:25] semanticity and redundancy and so on [77:28] like probably there's some kind of [77:29] compressed explanation that we can [77:32] Yeah, like you can you could you can [77:35] find a compressed explanation for the [77:37] entire network like uh in the same way [77:41] you you can I think you can like [77:44] medicine can understand the human body [77:46] without having uh you know without [77:50] having a complete list of all of the [77:52] atoms or whatever you can find [77:54] compresses like compressor explanations [77:56] for all the different parts of the thing [77:59] and uh get have predictions in all kinds [78:04] of situations. [78:06] Uh and and that feels like a separate [78:09] thing from [78:11] another meaning of like thinking more [78:14] about pramatic of more the thing that [78:16] people do in medicine versus biology [78:18] where in medicine people usually [78:22] uh [78:24] yeah I think I think that you're [78:26] thinking those edges into philosophy but [78:29] yeah I think there seems like a separate [78:30] thing here where you can have different [78:34] you know explan not be trying to explain [78:37] something where you just care about some [78:40] local property like will this medicine [78:42] work and then you might just be blackbox [78:47] looking at it and trying different [78:50] substances and seeing what happens. [78:53] Uh and yeah, it feels like there is like [79:00] the this question of uh [79:05] what does it mean to explain some things [79:09] in separate for like the more there are [79:12] more pragmatic methods and less [79:15] pragmatic methods that are more about [79:17] understanding. [79:19] Uh [79:21] but so what did it look like? So, what [79:23] I'm trying to say is uh [79:28] yeah, I'm not sure what I mean trying to [79:29] say. I feel like I kind of getting lost [79:31] in the philosophy right now. Sorry. [79:33] [laughter] [79:36] >> Makes me think about like have you seen [79:39] uh Maxim? Have you seen the Neil's uh [79:42] pragmatic interp [79:45] paper or blog post? [79:47] >> I have. [79:48] >> I see some some related patterns. which [79:52] are talking about [79:53] >> um make uh Neil posted that he's quote [79:56] unquote moving from from trying to find [79:59] the algorithm to to figure out methods [80:02] that work for alignment. So this way [80:04] like being pragmatic basically. So like [80:06] methods that work the best not that [80:08] explained the most or something like [80:10] that like to summarize it very high [80:12] level and probably strolling or [80:14] something. [80:15] >> Yeah. I I feel like Neil's [80:18] is more about Neil's thing is more about [80:22] uh he's trying like more about a [80:26] changing approach of how he period you [80:29] know he and my team periodize research [80:33] more than anything and it's not that and [80:36] he's also giving up on trying to fully [80:39] reverse engineer networks but I think if [80:44] if if He thought it would be likely to [80:47] be you know doable to to fully reverse [80:50] engineer your networks. I think it's a [80:52] pragmatic approach which will you know [80:55] recommend to that. So it's not like I I [80:58] think these are separate things kind of [81:02] it's kind of complicated. I I asked him [81:04] about it on the channel and he said some [81:06] interesting things like on the it's on [81:08] the reading group channel some a while [81:10] ago when he we read his post on the [81:13] reading group. Uh [81:16] but yeah where he it's more that he used [81:22] to take fully reverse engineering the [81:25] neal network as the main objective and [81:29] now he's uh trying to has his main [81:34] objective is trying to do whatever is [81:38] useful for helping with alignment with [81:41] interpretability. [81:43] Uh [81:46] >> yeah okay that makes sense actually. [81:48] >> Yeah but like if it if he thought that [81:52] fully reverse engineer the neural [81:54] network was going to be easy and useful [81:57] for interpretability he will do that. uh [82:00] it's just that he's not focusing on that [82:03] as his current objective where he thinks [82:06] of all the methods if they are good in [82:08] terms of do they help for that or not [82:10] and [snorts] he's more fine with kind of [82:13] hacking like the kind of methods that [82:15] just work for now and just solve some [82:18] problem like tell if the model is lying [82:20] or whatever even if they don't help with [82:23] reverse engineering the model [82:24] understanding it better [82:27] >> I gotta I got to hop out But before I [82:30] leave, um I would like to uh if anyone [82:34] has ideas for the server, um DM me those [82:38] ideas. I'm going to try instead of [82:40] talking about the server, DM me [82:41] directly. Um and also if they want to [82:43] talk to me about any research at all, DM [82:45] me as well. This extends to the people [82:48] strictly in this voice call to keep it [82:50] tractable. [82:52] Um, [82:54] >> there will also be people on the [82:56] watching this on the YouTube channel [82:58] maybe though. [83:00] >> Oh yeah. Uh, extends to them as well. [83:02] >> But but no one else. I mean, who else is [83:05] going to hear it? But anyway, uh, I'm [83:06] going to [laughter] [83:07] >> now the video somehow become super [83:09] violent and you start. [83:10] >> Yeah. Yeah. Anyway, [83:11] >> I don't think so. But yeah, [laughter] [83:13] >> this is a good talk. Uh, thanks for [83:15] giving it. I'll try. [83:16] >> Thanks. [83:17] >> Okay. See you. [83:18] >> Byebye. [83:19] >> Yeah, it's true. This paper is more like [83:22] I I I kind of see it as kind of like me [83:25] meta interpretability like trying to [83:28] interpret how the different interpretive [83:30] methods work and don't work and how to [83:31] make them better basically as a result [83:34] which is good uh which is which is [83:37] different from Neil's rep [83:38] prioritization. [83:40] So this is more like trying to actually [83:43] make the methods work better and [83:45] actually reverse engineer better and so [83:47] on. Yeah. And like and we we should do [83:52] meta meta interpretability [laughter] [83:56] and yeah that would be meta meta I guess [83:59] this this is also a form of meta [84:01] analysis since you kind of review the [84:02] different papers there. [84:06] >> Yeah. Yeah. Yeah it is in a way. [84:10] >> So we need more of these I think. But [84:14] this is good. [84:15] Like I I I [84:18] it's interesting how in science like the [84:21] more you go into the methods, the more [84:23] holes you see in them and so on. Oh [84:26] well, makes [snorts] you makes you feel [84:28] less and less confident about the [84:30] results as a result. [84:33] >> Yeah. But I mean on the other hand, if [84:34] you think about it, we are in a much [84:36] better position than neuroscientists [84:38] because neuroscientists can't get the [84:40] entire activations of every neuron. they [84:42] get only population averages and things [84:44] like that and they have to go from [84:46] there. [84:46] >> So we should be better off in making [84:49] interp. [84:50] >> Yeah. and you can't remove half of a [84:52] patient's brain and see if a behavior [84:54] [laughter] [84:56] that reminds me of like I I saw uh in a [85:00] conference I was with the bit ba and he [85:02] was trying to convince one of his uh [85:05] students for him was when he was doing a [85:08] neuroscience to come work in making turf [85:11] with saying basically that that you [85:13] don't have to do like experiments that's [85:15] much cheaper and you don't have to spend [85:17] months to like remove parts of a mouse [85:19] brain or whatever [85:20] >> [laughter] [85:22] >> Uh-huh. [85:24] Yeah. But if you want to do it on [85:26] frontier models, then then you need very [85:29] big pockets of money. [85:31] >> Yeah, that's true. [85:33] >> That's I guess [85:35] >> but yeah. [85:37] >> Yeah. [85:38] >> For for frontier models, even if the [85:42] open source best models even running [85:45] them is expensive. So [85:48] >> yeah, [85:49] >> unless you want to run a very very [85:51] slowly uh using swap or something, but [85:55] yeah. Uh [85:59] huh. I wonder if there is something like [86:03] if you could like train part of the [86:06] convers different different sizes of [86:09] models and see some scaling claws in [86:11] interpretability or something like that. [86:13] people have that kind of thing like I [86:17] guess it kind of hard like how you [86:19] measure I think I mean there's a bunch [86:21] of papers I think we saying on the radio [86:23] group that have different I don't know [86:27] skull scaly loves but yeah that show for [86:30] different models of different sizes [86:31] things or like show some relationship [86:36] with the sparity of the spar encoder or [86:39] whatever with some other measure of how [86:43] Well, Woody is interpreting, [86:45] but I don't know. [86:51] >> Makes me Yeah. makes me wonder if the [86:54] bigger models also like have more paths [86:56] to solve some some some concrete task. [87:00] So like more you could identify more [87:02] circuits in a way for the same. That [87:04] feels more like this feels like the kind [87:06] of thing where you can probably figure [87:09] something out from existing papers that [87:12] have already like I feel it's a bunch of [87:14] papers that were not that were like [87:17] looking into things related to that that [87:20] like were not mainly trying to figure [87:23] the thing you are asking out but where [87:25] they were trying different models and [87:29] some of them are bigger and some of them [87:30] are smaller and you can probably figure [87:32] it out by like doing literature search [87:34] or whatever and asking CL or something [87:37] for look for papers or that and yeah [87:41] looking at what they found or even [87:43] there's probably also papers that look [87:46] at that kind of thing only small models [87:49] and other papers that look at the same [87:51] kind of thing on bigger models and you [87:52] can figure maybe that out I don't know [87:55] [snorts] [87:59] >> yeah [88:01] So [88:03] with yeah you could have multiple [88:06] circuits in a model and you could have [88:09] nonidentifiability [88:10] in terms of in using different methods [88:13] to identify them and finding [88:16] uh [88:18] yeah [88:20] >> having multiple circuits for the same [88:21] circuit basically and there being [88:23] multiple circuits in the model as well [88:25] and in the interpreted uh method in the [88:28] method that interprets it. about this uh [88:31] god [laughter] [88:34] >> something I I feel it would be cool to [88:37] test of all kind of methods it will be [88:40] really nice uh if we actually like [88:44] that's why I like the open thing and [88:46] think like if I don't know uh w manag to [88:50] find some pilinear thing or something [88:52] that just works that will nice but even [88:55] if we have a method that makes models [88:58] that are like that are you know trained [89:02] to be more interpreter or whatever and [89:03] they are much worse but we actually [89:05] understand them that will be really good [89:07] because then we could test things like [89:09] identifiability by having a model where [89:12] we know what the answer is and testing [89:15] methods against it. [89:18] >> Yeah. But we also don't have a model [89:21] where we are we uh like not only like we [89:26] don't even have a model that where we [89:28] have some vague guess of how it works. [89:30] Uh we don't we will need a model where [89:32] we are pretty sure we know what it works [89:34] and seems even harder. Uh right now we [89:37] don't even have like a some model where [89:39] we have a full circuit thing that [89:42] explains the model that uh we are not [89:46] sure it's true but it's at least some [89:49] full explanation that like explains most [89:51] of the the assets of the model whatever [89:54] >> there's the mentter benchmark actually [89:57] that's kind of how I started my PhD. I [89:59] wanted to make a benchmark to try to [90:01] have like provably interpretable [90:03] networks and then we could try the [90:05] methods on them and that's what they did [90:06] pretty much in the make inter benchmark. [90:09] Uh they used interchange intervention [90:11] training to do this like causality based [90:13] training and then they submitted it as a [90:15] benchmark and people can just try and [90:17] recover the original circuit but it's [90:18] just for finding so it's only one [90:20] method. [90:24] Yeah, I think there's there are some [90:26] things you can do with synthetic [90:28] circuits that you made and then you know [90:30] what the current truth is, but uh I [90:33] worry that kind of thing is worse in [90:36] that they don't like the the real models [90:39] are going to be pretty different in some [90:41] ways like I don't know you could for [90:43] example you could train with uh you [90:46] could code there is a thing called [90:49] tracer from Google I think I think it's [90:51] from demand yeah where you can compile [90:53] write code and compile it to a [90:55] transformer. Uh but I don't think I [90:58] expect that looks like very different [91:00] from actually train neural networks ways [91:03] that's probably you know it testing your [91:05] method does probably [91:07] >> yeah I mean the the weights are [91:08] basically binary like of course it's [91:10] very different but at the same time I I [91:11] don't know if our methods don't even [91:13] work on super simple uh things even if [91:16] they look very different like I think [91:18] it's unlikely complex [91:20] >> that probably is yeah it probably should [91:24] be tested like kind of similar to the [91:27] random model thing. Uh uh yeah, that it [91:31] will be nice to see what happens if you [91:34] want to try what happens if you have [91:36] those models tested. I I was thinking [91:39] more even if it does work uh that still [91:42] doesn't mean it's going to work on a [91:44] real network. Uh but yeah, maybe the [91:48] other way around. uh your method pro [91:52] should probably also work on the on the [91:55] binary unless I don't know your method [91:58] does exploit some feature of uh real [92:03] networks that that the binary thing like [92:06] the tracer thing doesn't have uh like [92:08] maybe the turns out the [92:13] something like the the silly samp I [92:17] don't think it's actually true but maybe [92:19] everything is the linear representation [92:21] hypothesis is just true and so uh and [92:25] you can just use sp some kind of spoder [92:29] uh to interpret the networks and some [92:33] tracer kind of thing wouldn't work. Uh [92:38] I I don't think that's actually the case [92:40] for that example, but something like [92:42] that. [92:42] >> If [laughter] you had a method that [92:44] really needs your model to be noisy and [92:46] messy and having polynticity and [92:48] multiple function and stuff, I mean [92:51] [clears throat] if you manage to find [92:52] something like that. Yeah, sure. [92:57] >> Like I can [92:59] mod [93:01] transfer paper from the the paper. This [93:04] makes me wonder [93:06] [clears throat] [93:10] I not hearing you very well right now [93:12] like it's not very maybe now. [93:14] >> Oh now it works. Yeah. [93:16] >> Yeah. Yeah. So so so uh you know the [93:18] original neon growing paper with mod [93:21] edition algorithm reverse engineered [93:25] being the trigonometric algorithm. That [93:27] makes me wonder if how how many holes [93:30] are in this paper like how is it is it [93:33] also non identifiable for example or [93:35] like could if you did this method on a [93:39] on a random init network what results [93:41] would you get etc etc basically do this [93:44] paper on on on this grouping paper like [93:47] which seems to be like [93:48] >> as far as I understand the the most it [93:51] feels the closest to the ground truth of [93:53] what the transformer is doing but maybe [93:55] we're actually building ourselves. [93:58] >> I know I I I did. Okay. So, I have been [94:00] looking uh recently again to my to my [94:04] thing with auto context learning stuff [94:06] on the on a modular multiplication model [94:09] and I don't know the rest but okay some [94:12] things on the original model does seem [94:15] like pretty clear those work that way [94:17] and also seeing the same things on a [94:20] different model as model multiplication [94:22] instead. Uh [94:24] so so I don't know I I would want I I [94:28] know I want to try this the the [94:30] experiment I were running on the other [94:31] day like on random network. Uh [94:36] but I think at least the fiery de [94:39] composition stuff it's unlikely you you [94:42] find like so so clear results as the as [94:46] in the paper with a random network. Uh [94:49] uh [94:51] but I yeah let's feel about the other [94:53] stuff. [94:56] >> Yeah. [94:58] >> Also I don't know the [95:01] so so in one hand they they do this [95:05] thing where they replace the the model [95:09] the parts of the model with their math [95:14] for how it works and that doesn't hurt [95:16] the performance pretty much. Uh, [95:20] but I don't know. I I will have to check [95:24] like the details of that and how much it [95:26] works to be sure that it actually makes [95:30] sense. Uh [95:33] but yeah, [95:36] but going to going to that I will see [95:39] like people working more on that kind of [95:42] rocking model like with the modular [95:45] edition model and like actually [95:49] getting pretty confident you know how it [95:52] works and maybe that could be a a good [95:56] test of some methods. see if they [96:00] recover the [96:03] uh the same you know groing the same I [96:09] mean the same addition structure for [96:12] that it will be nice to have more [96:15] examples at the modular edition thing at [96:18] the same time I do worry that you know [96:20] this kind of thing this kind of toy [96:22] model it's also pretty different from [96:25] bigger models so uh it might not uh [96:30] generalize that much but yeah could be [96:32] it should at least work like your [96:34] methods should also work even for those [96:37] small small models [96:42] >> also on the related to that I I do [96:45] expect [96:47] the I wonder if the Groki model also has [96:50] a nice I was thinking about that that [96:54] has also has a nice uh geometric [96:57] explanation [96:58] like with the other like in the other uh [97:03] >> yeah I actually was looking for that the [97:06] other day the model multiplication model [97:08] uh but it's uh it instead it doesn't [97:13] have a nice spiral like it instead has [97:16] like a bunch of clustering where it I [97:19] think this also the same on the modular [97:21] edition model though h where it's it's [97:24] more that uh has a bunch of uh a bunch [97:30] of uh directions on the on the embedding [97:34] space where each direction like uh the [97:39] model the each number is clustering like [97:42] are cluster in some direction and it [97:45] some direction means that the number is [97:49] uh some module or something and it just [97:54] finds the correct number with that like [97:56] has a bunch of uh this number mod five [98:00] is five or whatever. Uh this number mod [98:04] 18 is whatever and it just combines [98:07] those uh to to differentiate all the [98:11] numbers and it doesn't have a nice [98:13] continuous corpor [98:15] >> but [98:16] >> that's this composition maybe. [98:18] >> Yeah. So it does look like a cool [98:22] structure thing, but it's not a a cool I [98:25] think it's I mean it does [98:29] end up being like a nice for your like [98:33] the composition figure. I feel like the [98:35] has to be some way of plotting it that [98:38] it where it looks like a circle. Uh but [98:41] yeah, I need to to see to think more [98:44] about that. I was I was just mostly [98:46] asking the clothes uh for prov to to do [98:51] things to see how well it did at the [98:52] kind of [98:55] uh yeah [98:56] >> but I hadn't done much like I was [98:59] working on something else in that [99:01] project now I was not looking at the [99:03] modular multiplication anymore I was [99:05] looking at real language model now uh [99:07] but I went back for the modular [99:09] multiplication thing for a bit to test [99:11] uh to to see if clo will get to the same [99:15] conclusions I got for like just mis [99:18] letting it run experiments. Hm. I wonder [99:21] if if you would strain it in some [99:23] different way if you would get a more [99:25] clear geometrical structure as a result. [99:27] But yeah. [99:29] >> Yeah. I mean do you do actually Okay. So [99:33] big language models like GPT2 do have a [99:36] nicer geometric structure because they [99:39] have a there's one of there was the one [99:42] of the papers the language models use [99:44] trigonometry to do addition. Uh, and so [99:47] they do have a they have an elix for [99:49] that. [99:50] >> Uh, [99:54] I [99:55] >> but I think the thing on the ging paper [99:57] is not the same, but maybe I I'm not [100:00] completely sure. I I feel like the the I [100:02] thought they were the same, but then I [100:04] was confused yesterday and I might still [100:07] just be confused. I'm not seeing how it [100:10] is a thing. But it's not the case that [100:14] like if you just plot the PCA like you [100:16] when you plot the PCA on the on the on [100:19] the on the other spas paper from [100:22] anthropic the more recent one is it [100:25] doesn't look like that. If you plot the [100:26] PC it looks looks like a bunch of uh [100:29] like it looks like some of the numbers [100:31] are clustered and and if you look at [100:34] that it turns out the clustering depends [100:35] on the mod of the you know being mod [100:38] five or something. Uh but uh maybe you [100:41] take the for your transform or something [100:43] and then it looks uh more like a sitkill [100:46] or something. Uh I I not sure I will [100:50] look into that and tell you tell you [100:51] later if I feel wor about [100:56] >> h okay [100:59] >> I wonder [101:01] from the original topic and the paper [101:02] and maybe we could or or like [101:05] >> I don't know if we could also end it uh [101:08] here if people don't uh have a lot more [101:11] to talk about this paper. [101:12] >> Uh yeah I will have to go shortly [101:15] actually. Yeah. Does anyone else has any [101:18] questions then before uh Maxim has to go [101:32] doesn't seem like it [101:36] you know [101:38] okay yeah says he doesn't have [101:42] questions. [101:43] Uh [101:46] all right. Uh [101:48] >> thank you everyone for the questions. Uh [101:50] the discussion was super interesting. Uh [101:52] so thank you a lot for that. [101:56] >> Yeah. [101:56] >> Yes. Thanks for the presentation Maxim. [101:59] >> Yeah. Thanks for coming. If you want to [102:02] uh you make another paper and you want [102:04] to come talk here just say it on the [102:07] reading group. I'll probably uh go [102:10] >> the next point is probably going to be [102:12] in a while but sure. [102:14] >> Yeah. [laughter] Okay. [102:16] Uh, also a last thing I want to say is [102:19] yeah and now I was thinking of this and [102:22] uh maybe uh I I kind of want to try make [102:27] taking a random model and asking Claus [102:31] to try to figure out what it does to see [102:34] if it comes up with some random fake [102:37] explanation. [102:41] >> Yeah, if you do that I'm curious to see [102:43] the results actually. [102:45] Yeah, [102:47] I I I I don't have that that much usage [102:51] for clo because I only pray in the $20 [102:54] thing. So, it's going to be a while. But [102:56] yeah. [laughter] [103:04] Yeah. I I also want to see that for the [103:06] activation oracles. [103:09] uh like if you take a random model does [103:14] does it actually answer the [103:17] uh the question or if you ask a question [103:20] of the activation oracle that it can't [103:23] like it's impossible for it to answer [103:27] >> like what makes me think they [103:28] technically tested it on different [103:30] models I think and they found like a [103:32] loss in performance so I would assume [103:34] completely random model would be big [103:36] loss in performance but I wouldn't be [103:38] surprised anymore after this. [103:40] >> Yeah, [laughter] [103:41] like it is probably a bit close in [103:42] performance. But my my question is does [103:46] it like does it still does the model say [103:50] it also know how to explain it or does [103:52] it elucidate explanations like how does [103:54] it fail? It probably [103:57] you can already see on the other model's [103:59] case like how does it fail like it does [104:02] it like make up things? Uh maybe on the [104:05] completely maybe it fa it originates on [104:07] the on the other models but on the [104:10] completely random one it it says it [104:13] either goes crazy or just says it [104:16] doesn't know I would have I kind of [104:19] expect it doesn't it kind of just mix [104:21] things up [104:26] that's a problem because like on right [104:28] when you are testing you do know h for [104:30] the course they do know the ground truth [104:33] but they The interesting thing is [104:35] finding out like [104:39] uh yeah like I I guess from the back [104:43] door I wonder if they tried [104:46] uh asking it for a I don't remember if [104:48] they tried that they tried asking [104:51] testing it on a model that to look for a [104:54] back door in a model that doesn't [104:56] actually have a back door. Uh [105:00] but yeah, [105:06] anyway, yeah, I guess we can leave it [105:09] here [105:13] if nobody else has anything to say. [105:18] >> Okay. [105:19] >> Okay. Yeah, I'll I'll stop the recording [105:23] then. Uh [105:27] uh where was it here? I'm still