OpenAI Researchers Force Network Sparsity to Uncover Transformer Circuits

OpenAI 研究人员通过强制网络稀疏揭示 Transformer 电路 施加严格的权重与激活稀疏性可将 AI 内部机制简化为可读电路，尽管这种极端约束会直接降低模型性能。 读书会分析了一种权重稀疏 Transformer 架构，该架构在所有层中强制实施物理权重稀疏性与 Top-K 激活稀疏性，从而发掘更清晰、更具可解释性的子图。他们详细探讨了利用直通估计器（straight-through estimators）进行自动电路发现的机制，以及旨在将稀疏激活与稠密模型对齐的桥接损失（bridge loss）构建方法。讨论还深入审视了论文中的特殊细节，包括为了提升性能而特意保留的嵌入层 bug，并将文中的技术与 Anthropic 的 transcoder 工作进行了对比。最后，他们权衡了可解释性与模型能力之间的得失，指出 L0 稀疏性虽能减少多义性串扰，但这是以牺牲原始计算效率和损失表现为代价的。 权重稀疏 Transformer 通过直接限制权重矩阵中的非零连接，提供了一种不同于事后稀疏自编码器（SAE）的端到端可解释性替代方案，能够直接从数据中学习到必要且充分的独立电路。 (7:48) 该模型架构在 Embedding、注意力投影和前馈矩阵中全面实施权重稀疏，并结合过滤掉 75% 维度的 AbsTopK 激活稀疏化，以优化模型能力与可解释性之间的权衡。 (34:34) 研究者设计了跨模型的“桥接函数”（Bridges），在训练阶段通过混合前向传播与转换损失函数将稀疏模型与稠密模型的隐藏状态表征紧密对齐。 (50:20) 分析显示更大规模的模型能将特定任务逻辑压缩进更紧凑、更直观的剪枝电路中，尽管严苛的 L0 权重稀疏正则化目前仍会削弱模型的整体泛化容量。 (14:47)

In a reading group discussion of the OpenAI paper "Weight-sparse transformers have interpretable circuits," participants analyzed how enforcing structural sparsity during training produces clean, isolated, and verifiable sub-circuits for specific tasks. While this approach bypasses the reconstruction errors and arbitrary hyperparameter choices of post-hoc Sparse Autoencoders, it imposes capability penalties and faces steep computational and scaling trade-offs. End-to-End Sparsity Versus Post-Hoc Analysis (7:48) Traditional mechanistic interpretability primarily analyzes dense, entangled models or applies post-hoc techniques such as Sparse Autoencoders (SAEs). Reading group co-host Aman noted that SAEs and transcoders suffer from reconstruction errors, layer-isolated application (typically targeting MLPs while missing attention heads), and arbitrary hyperparameter choices such as dictionary size. In contrast, training an end-to-end weight-sparse transformer forces the entire computation—across all attention matrices and MLP layers—to learn sparse, organic circuits directly from data. The group noted that standard dense networks rely on diffuse residual representations where deleting primary pathways triggers messy fallback duplicate circuits. By contrast, sub-circuits in the paper's weight-sparse models proved both necessary and sufficient. In an evaluated Python quote-matching task ('hello' vs. "hello"), the network condensed computation into a minimal program: one channel detected quote marks, a second classified single versus double quotes, and an attention skip connection routed the correct closing quote to the end token. Training Architecture, L0 Regularization, and Top-K Activation (34:34) The participants dissected the paper's specific training implementation details documented in Appendix A.1. The architecture omits standard positional embeddings entirely, learns a separate bi-gram matrix to offload trivial token adjacency statistics, and pretrains purely on Python code. Sparsity is imposed universally across token embeddings and attention projection weights (W_Q, W_K, W_V, W_O) by combining an absolute Top-K (AbsTopK) activation threshold—retaining the top 25% of activations and zeroing the rest—with structural L0 weight regularization. Dynamic L0 weight sparsity is enforced post-optimizer step by retaining only the highest-magnitude weights per matrix and zeroing the remainder, with the target L0 parameter linearly annealed over the first 50% of training steps. Participants observed that zeroed weights are not permanently dead; unmasked parameters receive gradient updates before thresholding, allowing an entry to regain active status if its magnitude surpasses the matrix cut-off. Automated Circuit Discovery and Bridge Networks (38:49) Rather than relying on manual inspection to identify pathways, the authors introduced an automated task-pruning algorithm. Nodes (defined as attention heads or residual channels) and edges (non-zero weight entries) are pruned using mask parameters trained with a straight-through estimator and a surrogate gradient derivative, guaranteeing the model maintains a specified target loss threshold (such as 0.5) on a given benchmark. To align sparse architectures with dense representations, the authors explored "bridges" in Appendix C. Bridge translation functions (f_i and g_i) use auxiliary loss terms, including normalized mean squared error and hybrid forward-pass KL divergence, to map activations bidirectionally between a frozen dense model and the sparse model. Group members noted that these bridge components are omitted entirely during inference, serving strictly as alignment training tools. Capability Penalties, Scaling frontiers, and MoE Comparisons (126:06) The group analyzed the paper's Pareto frontier comparing total model size, non-zero parameters, loss, and interpretability (measured by pruned circuit size K). While holding parameter count fixed and decreasing non-zero weights degrades task capability, scaling total model size shifts the frontier upward. Larger models compressed simple tasks into significantly smaller, tighter algorithmic circuits than smaller models, which the group compared to grokking dynamics where algorithmic solutions collapse the high bit-complexity required by memorization networks. However, participants concluded that weight sparsity serves primarily as an interpretability research vehicle rather than a production replacement for dense or Mixture of Experts (MoE) architectures. Unlike MoE models (such as DeepSeek's 500B total/32B active setup), arbitrary sparse weight matrices offer poor FLOP efficiency and fail to accelerate standard GPU matrix multiplication without specialized kernels. A key unresolved limitation is whether weight-sparse architectures eliminate polysemanticity or merely compress it into sparse nodes. Furthermore, the participants questioned whether the manual effort required to audit thousands of sub-circuits can be overcome, noting that while scaling weight-sparse architectures to frontier models faces severe hardware and capability constraints, achieving verifiable interpretability at even a GPT-3 scale would represent a major milestone.

OpenAI 研究人员在论文《Weight-sparse transformers have interpretable circuits》中提出，通过在预训练阶段对 Transformer 施加端到端权重稀疏性与激活约束，可以促使模型自发形成清晰、独立的计算回路。参与讨论的机械可解释性研究者指出，该方案旨在从底层架构解决密集模型及事后稀疏自编码器所面临的重构误差、超参数任意性与多义性纠缠难题。 架构路线：端到端权重稀疏替代事后自编码器 (7:48) 传统的机械可解释性主要针对全连接密集网络，但其神经元之间存在高度复杂的纠缠。后验解释工具如稀疏自编码器（SAE）或转码器（Transcoders）虽能展开潜在特征，但存在固有缺陷：模型重构存在不可忽略的误差、特征字典大小与损失惩罚依赖人工主观设定、分析通常受限于 MLP 层而忽略注意力机制，且不完备的特征基底容易在消融测试中破坏不相关的语义。 针对上述瓶颈，论文提出在预训练阶段直接构建端到端权重稀疏 Transformer，在词元嵌入、注意力投影（$W_Q, W_K, W_V, W_O$）及 MLP 权重矩阵中将绝大多数连接物理置零。讨论成员强调，这种全局约束迫使网络直接从数据分布中学习精简的连接通路，避免了后验解释方法带来的重构失真与人为结构偏置。 训练动力学：L0 动态退火与 AbsTopK 约束 (103:37) 为避免静态随机连接带来的性能瓶颈，模型采用动态 $L_0$ 调度算法。在每次优化器更新未掩码参数后，训练流程执行基于绝对量级的 Top-$k$ 阈值化操作，仅保留各权重矩阵中幅值最大的非零项并将剩余权重重置为零。这一 $L_0$ 目标在前 50% 的训练步数中进行线性退火，允许网络在锁定连接拓扑前动态探索最优路径，被置零的连接在累积足够梯度后仍可重新恢复激活。 在结构权重稀疏之外，模型进一步叠加了激活稀疏性约束，引入绝对值 Top-$K$（AbsTopK）激活函数（通常保留节点隐层维度的前 25% 幅值并清零其余 75%）。为简化任务机制，该模型完全移除了标准位置编码，引入独立的双词元（bi-gram）矩阵分流低阶词邻接统计，并在纯 Python 代码语料上完成了训练。 回路提取：自动化剪枝与必要充分性验证 (17:40) 在任务回路分析阶段，研究团队采用基于直通估计器（Straight-Through Estimator）与 Sigmoid 替代梯度的自动化掩码剪枝算法，替代繁琐的人工逐层溯源。该算法在指定任务分布下约束目标损失阈值（例如保持损失在 0.5 以下），自动剪除与特定行为无关的注意力头与残差通道节点，从而提取出最小功能子网络。 以 Python 字符串单双引号匹配任务为例，提取出的稀疏子网络展现出高度清晰的因果链条：专用通道分别负责检测引号出现与分类单双引号属性，并通过注意力机制绕过中间文本直接输出对应的闭合符号。与密集网络中冗余杂乱的替代通路不同，该稀疏回路表现出“必要且充分”的数学特性，移除子网络之外的所有模型权重后，该极简程序仍能独立稳定运行。 规模效应：大模型对回路复杂度的算法压缩 (126:06) 讨论组分析了模型规模、损失与回路复杂度之间的帕累托前沿。在固定参数规模下，减少非零参数数量（强化 $L_0$ 约束）会导致任务损失上升，但能显著缩小剪枝后的任务回路尺寸（Pruned Circuit Size）。相反，扩大模型总参数规模可向上平移能力前沿，使大型稀疏模型在保持较低损失的同时维持极小的回路规模。 研讨人员援引 Neel Nanda 关于模加法与顿悟（Grokking）的研究指出，这一反直觉现象揭示了表征机制的本质差异：容量较小的小模型往往依赖散乱、高熵的记忆网络硬编码训练样本，导致剪枝回路庞大；而参数充足的大模型倾向于学习紧凑、通用的算法解。因此在简单基准任务上，规模更大的模型反而能沉淀出节点更少、更易被人类理解的算法回路。 桥接对齐机制与算力落地局限 (77:53) 为建立稀疏模型与密集模型的表征关联，论文附录提出了“桥接”（Bridges）架构。该方案先训练并冻结密集模型，再通过编码器 $f_i$ 与解码器 $g_i$ 映射中间层激活，引入包括归一化均方误差在内的三项损失构建混合前向传播，强制稀疏模型复现密集模型的潜在表征，且桥接参数在推理阶段完全剔除。 然而研讨成员指出，该技术目前具有明显的实用边界：非结构化的细粒度权重清零在缺乏专用稀疏内核的现代 GPU 硬件上无法带来推理加速，FLOP 计算效率远落后于混合专家（MoE）等块状路由架构；此外，针对特定任务剪枝会直接损害未剪枝任务的表现，模型内部仍残留一定程度的特征多义性，使其目前更适合作为可解释性基础研究工具，而非前沿生产模型的架构替代方案。 总体而言，研讨组认为端到端权重稀疏 Transformer 为自动化回路发现与严格的因果验证提供了突破性的研究范式，证明了神经网络能够自发形成模块化的人类可读程序。但由于其对模型泛化性能存在实质性损耗、现有硬件执行效率受限且难以完全根除多义性，该技术在短期内仍属于专用分析工具，能否成功拓展至前沿通用大模型仍有待检验。

[46:08] If you ablate a sparse encoder feature, you might be ablating whatever the true features... are... you don't have the right basis to do ablations on to disentangle the model nicely. [63:00] So first, they apply the translation from the dense to the sparse. Then they do all the layers in the sparse model until the end and then they take the KL... Okay, I just grokked it now! [77:53] It's just the error of the bridge basically... how well the bridge reconstructs hidden activations in the second model... in both directions. [84:43] At some point, we do find a bug... fixing this hurts quality in terms of capabilities and interpretability... They're just like, 'works better.' OpenAI in a nutshell. [87:36] We find that some amount of activation sparsity helps on top of weight sparsity, but that too much activation sparsity hurts the capabilities-interpretability frontier. [98:49] This reminds me of the big Anthropic paper... that we spent two days on for transcoders and activation circuits. This one is similarly complicated, but they have compressed it in a relatively short paper. [114:29] I need to review a lot of linear algebra stuff [118:39] It's better to let it pick which connections it wants, but forcing it to only have a few of them... it's almost like a super easy indicator of like important neurons if one neuron feeds into 20 different non-zero others. [143:39] Improving interpretability this way in this paper is harming the capability basically [170:09] I feel like the paper is too big to do one day... very dense paper.

[0:02] Okay, [0:05] let's let's [0:09] and [0:10] let me check that the recording is [0:13] working and stream to YouTube. [0:20] Yeah. [0:26] It is streaming. [0:31] Yeah. And the volume sounds okay. Okay. [0:34] Anyone anyway let's let's get started. [0:36] Uh understanding neural networks to the [0:39] spark sequence. [0:42] Uh okay. [0:44] Train. We train model models to think in [0:48] simple more traceable steps so we can [0:50] better understand how they work. Uh [0:53] okay. So it's about training models [0:56] apparently [0:58] uh [1:03] neuron networks uh neuron networks [1:06] powers today power to most capable [1:09] systems but they remain difficult to [1:10] understand we don't write these models [1:13] with explicit stepbystep instructions [1:16] instead they learn by adjusting billions [1:19] of internal connections or weights under [1:21] the master attach some hats. [1:26] Oh yeah, that that's the obvious thing. [1:28] I mean they really need to write this [1:30] because I I I feel like uh a lot of [1:34] people who are not in AI don't [1:37] it they don't understand and people even [1:40] people who are in AI [1:44] uh [1:46] they [1:48] don't really think as I think there's a [1:51] lot of people who just think uh oh sorry [1:54] someone is writing on chat [1:59] Yeah, I I I I will go to the paper. I I [2:03] just was going to go through the blog [2:05] first. I mean, if you want, we can just [2:06] skip to the paper, but I I I I I was [2:09] going to just read the this the blog [2:14] first because it's a summary of their [2:16] thing. Uh [2:21] yeah, that I I Yeah, makes sense. By the [2:24] way, uh for the Yeah, this might not I [2:27] don't know if the chat shows on the [2:29] Yeah, it's not showing the recording. [2:30] For the people uh who maybe watching the [2:33] recording, I just responding to the car [2:35] saying that this the pause is for lay [2:38] people and the paper might be better. [2:40] But yeah, okay, let me [2:43] continue. How you [2:48] this is [2:51] essential. Yeah, this how typical stuff [2:54] how interpretability is nice. [2:57] Um [3:02] I I am curious about this about what how [3:04] they they what they say interpret [3:09] about [3:11] nice [3:13] interesting they say it does completely [3:18] maybe let's talk more about that on the [3:20] paper but they say it does completely [3:22] reverse engineer a model computation [3:25] because I feel like like these days Neil [3:28] is not very optimistic about that more [3:32] ambitious view of making ter but people [3:34] are think of more understand some things [3:37] about how the model works [3:41] uh [3:51] low [3:54] difficult [3:56] uh uh [4:03] permission to perform networks. [4:07] Okay. So yeah, this is about training [4:08] models that are more interpretable [4:12] and yeah I have I have thoughts about [4:15] that but I I think it's pro probably [4:17] should actually read the get to what [4:20] they are doing first. Uh okay previous [4:23] mechanistic has start from [4:28] in this network six individual network [4:30] is connected to thousands of uh other [4:34] neurons [4:37] uh [4:39] by the way if people do want me to go to [4:41] the paper I can just directly and look [4:44] at some specific thing we can do that I [4:46] just uh [4:49] doing this because I thought it was the [4:51] best idea. And uh nobody apart from Tekk [4:55] nobody else said anything [5:01] most perform [5:04] many functions making it seem impossible [5:06] to understand. Uh but what if we train [5:09] and tangle neural networks [5:12] uh which may with many more networks in [5:15] which neuron only has a few dozen [5:18] connections. [5:19] Huh? [5:22] I guess this is interesting, but and [5:25] maybe they explain why like they have [5:30] they have some counter point to this on [5:31] the paper, but the the thing I would [5:35] wonder here is how much [5:39] this tells you about actual networks [5:41] that are not training on this way. But [5:44] at the same time, I mean, we don't [5:48] we not don't don't know much about [5:50] models in general. Maybe understanding [5:52] this [5:54] if if it did turn out that it was very [5:56] easy to understand this weird sparse [6:00] kind of network, even if the [6:01] [clears throat] [6:02] real networks are very different in some [6:04] ways. Maybe you can learn some [6:08] interesting things about it or even like [6:10] how to make with interpretability [6:12] methods. [6:14] Uh [6:16] and I will wonder how [6:19] how good you can get it to be like if [6:21] there is some performance but you can [6:23] get an interpreter network there is [6:25] actually like even is worse than the [6:26] other it's at least you know somewhat [6:28] good and you can you use it to I can do [6:33] the kind of things we don't have any [6:34] idea how to write a program to do like a [6:37] lot of things language models do even if [6:40] it turns out that real networks are [6:43] words do things differently having [6:44] strong example of how strong network [6:46] will do it could be very interesting [6:49] but yeah [6:52] anyway let's let's keep going uh we [6:55] train langu uh language models with a [6:57] very similar theaters to deputy tube but [7:00] yeah I mean I guess they been a [7:02] transformer they probably say in the [7:03] paper uh [7:07] all the ways to be seen No. [7:12] Okay. [7:24] It's a circuit sp [7:38] Yeah. So, uh [7:48] ah yeah, it's okay. Yeah. Uh by the way, [7:50] so I'm I going to put the this so people [7:52] can see the chat for a moment. Yeah. So [7:54] N say or whatever your username is [7:56] pronounced is saying uh that uh they are [8:00] wondering why not use a normal model [8:03] then dimensional illusion [8:08] I mean that's kind of already [8:11] that's kind of already [8:14] like spar colors. I guess this person [8:16] calls are not as sparse in these ways in [8:19] that you're not doing sparse in the [8:23] connections or something. [8:26] Uh [8:29] I I think it's [8:34] there that this might have some [8:37] advantages in that [8:39] uh in the if you do something like the [8:42] some dimensional [8:44] introduc on a real network you are going [8:49] to miss things like you when people do [8:51] things like spar encoders right well [8:53] that's not really delusion Sometimes [8:56] they they have multimeders but yeah when [8:58] you do you you're trying to summarize in [9:01] some way with some some method that are [9:05] network that already exists you are [9:08] going to miss things and there we don't [9:10] have like super good there's people are [9:13] making better methods but they are not [9:16] great methods that that don't have a lot [9:18] of error [9:20] of reconstruction error and things like [9:22] that [9:24] and in this case you don't know like [9:27] your the the sparse representation is [9:30] actually how the network works like is [9:33] the full full picture in this case so I [9:35] could see why it's the case but also [9:38] let's see if they say in the paper later [9:43] like why are they doing this uh and we [9:46] have some better explanation that uh my [9:49] idea from the for that [9:52] or why this this is a useful thing to do [9:54] as opposed to just try to look at actual [9:58] existing networks. [10:01] Uh [10:06] starting with the first model [10:07] computation tangle we consider very [10:09] simple model behaviors. [10:12] Oh, hi. [10:14] Uh, [10:16] wait. People feel free to just interrupt [10:20] me or talk any point like on voice chat [10:22] to you have to just write on text. [10:34] Someone is writing. [10:36] Uh but I I I'll keep uh looking at this [10:42] while you while you write. Anyway, uh we [10:46] we to me [10:53] responsible for this behavior with them. [10:56] Yeah, this is just [11:11] So it's not I they are not um [11:22] they are not training as part to encoder [11:26] in the [11:29] Yeah, [11:31] that's also True. [11:35] Yeah. So, so first the they are not uh [11:40] doing something like sports and codes to [11:42] the model. They are training a model [11:43] that has some sparity in it already like [11:46] in the architecture but they are just [11:48] putting weights to zero but uh like they [11:52] are training a sparse model not taking a [11:54] train model and doing a sparity on it. [11:56] I'm also like as Aman said on the chat [12:03] and yeah person are also not end to end. [12:07] Uh [12:09] so you are only doing it on a specific [12:12] layer. [12:16] Uh [12:19] and yeah here is like the whole network [12:22] is sparse now. [12:27] because it was trained to be sparse. And [12:30] this might mean that this probably means [12:32] that the network is worse than uh I mean [12:37] we we will see. Well, [12:42] I I sllor mods are more capable [12:45] interpretable, but I I think people were [12:49] complaining on Twitter about this title. [12:52] And let's let's see it doesn't actually [12:56] mean that yeah if you just make models [12:58] pass this actually probably increase. [13:02] Uh [13:06] >> yeah, just one thing to add is like why [13:08] you might want to do this end toend [13:09] sparity thing is because otherwise like [13:12] there are so many design choices as to [13:13] like how you enforce sparity and like [13:15] where you do it like you know and like I [13:19] guess the concern could be that those [13:21] choices [13:22] like you know might affect like what [13:25] your sparity [13:27] uh lets you learn. While on the other [13:30] hand like if you have a technique which [13:32] is like more organically sparse then you [13:35] might ask the question okay like yeah [13:37] you can argue maybe that the sparity is [13:40] now learning things based on the data [13:41] rather than like the kind of arbitrary [13:43] design choices you make as to like I'll [13:46] you know I I'll I'll just train an [13:48] autoenccoder in this layer or I'll [13:50] combine these layers to make a [13:52] transcoder or something. [13:53] >> Yeah. And also like you are the model is [13:56] trained with the spacity. know uh like [13:59] the you know that the the sparse the the [14:05] sparse version of the model is the the [14:07] spark model here is the model so you you [14:10] know it has all of the thing you are not [14:13] missing anything while on the spar [14:15] encoder if you use the spar encoder and [14:18] replace the start encoder in the the [14:20] layer uh you might be in the middle you [14:24] do have reconstruction error and you [14:26] might missing some mechanisms that [14:28] are not captured by this person. Well, [14:38] yeah. Uh let's let's [14:42] Yeah. So, uh let me [14:47] params. So, the bigger armor params, [14:51] right? And [14:54] uh this is [14:58] PR loss [15:02] uh with an interpretability [15:07] simply size. [15:20] I guess we need to see how they're [15:22] measuring this interpretability [15:24] because I mean the capability [15:25] [clears throat] is obvious. [15:27] >> Yeah. Yeah. So this is just saying [15:30] bigger models uh [15:33] are have lower loss uh for the same [15:38] sparity. [15:41] Uh I will be interesting if like if it's [15:44] actually [15:47] I I'm confused by here they say [15:54] so it says interpretability pruned [15:57] circuit size but how is that not the [16:01] same as non zero params [16:05] uh like we'll go to the paper and look [16:06] at that but yeah I don't know like yeah [16:11] the uh yeah for a first spark model size [16:14] in case sparity setting more weights to [16:17] zero [16:19] I but if if this is a sparity [16:27] if they are here measuring this supposed [16:30] to represent the prune cirqu size I [16:33] confuse why it's not the same as the non [16:37] zero zero params Right. [16:45] Unless it's [16:48] this is actually the size of the [16:50] original circuit and this is actually [16:52] but then it's not in the never mind. [16:54] Let's okay let's [16:59] leave that for later and we look at it [17:01] on the paper. Uh [17:04] to make this uh concrete consider the [17:06] trade where a model type of python code [17:08] has to complete a string with the [17:10] correct type of code. In Python hello [17:13] with single quote and hello M code the [17:16] model our code the model can sort this [17:19] by remembering which quote type open [17:21] string and reproducing at the end. Make [17:23] sense? [17:25] Uh [17:40] this is so make this [17:43] graph [17:44] wait. [17:53] Oh, this is cool. [18:29] Yeah, it's really cool if they if this [18:31] gets you like tiny. Yeah, if you you [18:34] only have a few numbers and they do [18:38] something interpretable. [18:42] Uh [18:48] I I need to read the paper first to to [18:51] understand some of the like what they [18:53] mean uh neurons [18:58] by residual channels. I mean I guess [19:01] they mean like the the strip here is [19:04] only a few numbers now. [19:11] or well I'm not sure I I probably [19:14] there's some explanation on the paper [19:15] that's [clears throat] better than me [19:16] trying to figure it out getting it from [19:19] the figure uh uses layer to convert this [19:24] into one channel that detects any quote [19:26] another classifies between single and [19:28] double quotes [19:30] using a compression to ignore [19:33] intervening tokens find the previous [19:35] code and copies type to the final token. [19:38] Uh, please close. [19:43] Wait, wait. Just [19:47] just random thought that can hand to the [19:49] current thing. Uh, I could try we we are [19:54] doing this already, but I I could try [19:56] for next week. I I might ask Leo out. He [19:59] was also even after we discuss it uh [20:02] along we could ask Leo to if he wants to [20:06] come to the reading group to talk about [20:08] this. I don't know if it how much they [20:12] is open publishing this okay for them to [20:15] talk about it. Uh [20:18] and [20:20] yeah, he made me busy, but uh [20:25] uh I'm stealing his small server that he [20:28] made. Uh with only the people who pass [20:32] his test on the was funny. Uh I might [20:36] ask him there or something or just DM [20:39] him. Uh [20:44] anyway, [20:48] yeah, but I I wouldn't be surprised if [20:52] he's not Yeah, he's too busy to But I I [20:56] might as well try. Anyway, let's let's [20:58] just let's first try to at least look at [21:03] it ourselves. So So if he does come, we [21:06] have uh interesting questions. [21:10] Uh [21:12] in our definition uh the set connections [21:15] shown above are sufficient to perform [21:17] the task. Uh they are also necessary. Uh [21:23] so so yeah if you they remove the rest [21:25] of the model the small sequence still [21:27] works which that's actually [21:31] pretty nice because this often is not [21:35] the case with I feel this is more [21:39] unnecessary [21:40] causes multip yeah this is a kind of a [21:44] more stringent uh like definition of [21:48] what a circuit doing that what people do [21:52] with actual with normal networks where [21:54] is kind of a lot most more of a mess and [21:58] sometimes you have some that you think [21:59] does the things but there's also other [22:02] that also do the thing if you delete the [22:04] the current one it starts maybe it does [22:07] it less often but it still kind of works [22:11] and and if you remove the entire other [22:14] parts of the network or I don't think [22:17] are good enough that most of the time it [22:20] works works [22:22] also tiny things like I don't know uh [22:25] Neil's modular addition paper or [22:30] uh things like that uh yeah you can just [22:33] replace the [22:35] the in that case it that's true but also [22:38] it's a tiny model that's just doing a [22:40] single thing uh [22:46] more complicated behaviors [22:48] a hard way completely [22:50] And we can still simple part [22:53] explanations which are relative model [22:55] behavior [23:00] current. Oh, someone left. Anyway, yeah. [23:03] Uh [23:06] the type of variable current [snorts] [23:09] one attention operation copies the name [23:12] of variable into the set token when it's [23:14] defined. Another letteration [23:17] time from the set into a subsequence [23:22] use the variable allowing the body to [23:24] infer the corres. Oh, that's a neat [23:28] way of [23:30] uh doing things. [23:50] Yeah. Yeah. That's yeah that's a nice [23:54] way of doing Python stuff like the [23:58] copying [24:01] current into here and then [24:05] yeah [24:07] and you have the type here [24:12] okay yeah maybe people are not [24:15] understanding but I think I I got it at [24:18] this kind of uh yeah the [24:22] to as an early goal making a model [24:25] computations easier to understand but [24:28] still a way to go more models and [24:35] models [24:37] with motives and more composition models [24:41] we could develop [24:43] ination for the models. Yeah, that makes [24:46] sense. [24:48] Strat [24:53] particles for systems mod [24:56] I mean yes but this is just going back [25:02] to spar colors right I guess more than [25:06] sparse color it's again back to the [25:09] thing that [25:11] was say among was saying where is this [25:14] will be like some you are doing a [25:17] sparity on the whole thing. I mean, I [25:20] guess you multiple layers also like [25:22] transcoders and also that uh [25:32] yeah, but I think uh like even even [25:35] transcoders don't really like look into [25:37] the like the I mean there's nothing they [25:39] don't do anything with the attention [25:41] weights and those things, right? So, [25:42] it's all MLP focused. So maybe you could [25:45] generalize to like yeah some other [25:47] technique. [25:48] >> Yeah. I mean you could reasonable [25:50] transcoder with all there's you couldn't [25:53] do with the entire thing right like do a [25:58] sparse on all the parts like you can [26:00] people do do train parts on the on the [26:03] attention sometimes like not usually but [26:07] there are a few of them on European [26:09] things like that. uh and like I guess [26:15] you could do something with all of them [26:17] but I don't know. Yeah. So that that [26:19] it's true that no I don't think nobody [26:21] has done that or maybe someone has but [26:24] in some obscure people that we haven't [26:26] seen. Uh [26:28] so so yeah it will be it's a bit [26:31] different from the things people are [26:32] doing. [26:35] Uh [26:36] >> yeah the other thing that I'm curious is [26:37] like you know the the interpretations [26:39] that they have shown right like I mean [26:42] you know like of the specific circuits [26:44] like whatever like the matching I mean [26:46] I'm curious if [26:47] >> that has been like manually done or is [26:50] that still automated because I mean if [26:53] you otherwise if you scale up like scale [26:56] this up like let's say even if you could [26:58] do this then [26:59] >> you know how do you identify the [27:01] circuits like even if they are sparse [27:02] right like I mean uh doing that manually [27:05] I mean the hope with sparse auto [27:07] encoders was in a way that they would [27:09] they could like help automate some of [27:11] that but like it's been tricky uh is [27:14] what it looks like and so [27:18] >> u maybe it's easier if it is manual but [27:20] it's probably still not sorry if it is [27:22] parser the the t the circuits [27:25] >> maybe it's easier but it's probably [27:26] still like non-trivial to like you know [27:28] if you have like a huge network then to [27:31] do it like well [27:33] >> yeah I I think that makes sense At the [27:35] same time [27:38] like I feel like a lot of method al like [27:42] a lot of there are a lot of me things [27:43] you can do with uh try to do causal [27:48] inference kind of things where if you [27:51] actually have some unit of analysis [27:54] where well not really I was going to say [27:56] if you had some set of features that are [28:00] independently meaningful you can do uh [28:03] things with that but right you you might [28:06] still have even if it's sparse you kind [28:08] of still have poly semanticity on the on [28:10] the actual [28:13] this part ones uh so and you probably do [28:16] have so yeah but yeah I do yeah that's [28:20] the main questions I I have uh from this [28:26] remaining for the paper of how did they [28:30] figure out the that I I guess we can go [28:34] to the paper and try to to see that. And [28:39] there was something else. Yeah, I want [28:41] someone want to figure out uh [28:46] what's going on with this graph because [28:49] slightly confusing why they mean how do [28:51] we measure interpretability here or it's [28:52] just the number of of of you know how [28:55] sparse it is and then why it's different [28:57] from the number of zero parameters. [29:00] Uh but yeah uh I don't remember if I [29:04] have any other [29:05] >> I think paper will will define it [29:07] hopefully. [29:09] >> Yeah I I don't remember if I there was [29:11] anything else that I said that I wanted [29:13] to see on the paper but let's okay we [29:15] have seen this. Let's actually go to the [29:17] paper. [29:20] Okay the [29:23] the abstra is probably the same thing [29:24] that we already read. [29:28] Uh, I'm going to click. There is [29:30] something new that I didn't mention [29:32] there. [29:41] Oh uh [29:46] Amant says on the chat I'm putting here [29:48] so people on the video later can see [29:51] that he thinks the paper also me [29:54] something called britches spar sports [29:56] machine dlayer [29:59] but yeah it's hard to know how it's [30:01] different for maybe it's like a like a [30:03] sparse color but it's they mean on the [30:09] is not the same as sparse city as in [30:12] sparse to encoder, right? Uh in sparse [30:16] encoders, you are sparse in the sense [30:18] that you are trying to reduce the [30:22] network to a set of a large actually [30:26] even bigger than the than the model than [30:29] the other the original thing. sometimes [30:32] set of uh activations [30:35] that uh summarize the that are sparsely [30:41] activated where only a few activate on [30:44] each text. And here you have you this is [30:49] parsing the meaning of you have few [30:54] connections between weights right uh I I [30:58] will see to the bridges part later if [31:00] that's what I they mean but but it seems [31:03] like a different way so like there are [31:06] different kinds of sparity [31:17] Yeah, it does seem different in [31:19] definition though. I'm not sure like [31:22] like [snorts and clears throat] um at [31:24] the end of the day like there is [31:26] sparsity of some kind. So I'm not sure [31:28] like how different it is in practice. [31:32] Uh [31:33] >> yeah, not not I'm not sure how to think. [31:36] I feel I would need to think about that [31:39] lot more and like how do the things [31:42] relate and how they might be different [31:45] right and yeah no but uh let's see what [31:49] they say about it anyway first uh [31:54] we can always ask if Leo does come here [31:56] we can always ask him how they are [31:58] different [32:00] but let's see if this already is playing [32:02] on the paper anyway Okay. [32:17] Okay. Yeah. First learning about basing [32:19] the situations of pairs and then [32:21] attempted to understand computations of [32:22] the model with however this approach [32:25] attain human sequence by extracting away [32:27] complex computations that are only [32:28] person understood. Yeah, that's just [32:30] that's what I think may reflect the [32:33] chosen instructions in addition to the [32:34] most true mechanisms. [32:37] Here we introduce a new paradigm with [32:38] less substantially simpler and more [32:40] general circuits that we can fully [32:42] understand even at the lowest levels of [32:44] structure. To do this, we take [32:46] transformers where the mass majority of [32:48] weights are zero yet [32:51] and then [33:04] channels the model from distributed [33:06] concept across multiple channels. So [33:08] using more neurons that distinly need to [33:12] represent a single code. Yeah, I want to [33:14] understand this [33:17] like this concept of residual channels [33:19] in this in this [33:22] connection, right? [33:26] So [33:33] okay so in a normal [33:36] let me quickly check that they don't [33:39] actually explain it better somewhere. [33:43] Yeah let before I try to figure it out [33:47] myself. [33:50] I mean someone else knows like how [33:54] the [33:56] what a residual channel means in this [33:59] context in that you have normally I have [34:02] the residual stream and that's what I [34:05] have uh what I what they mean but uh I [34:09] I'm less sure of how to think about the [34:12] residual stream in terms of [34:15] uh [34:17] you know when you have a lot of zero [34:19] those [34:22] uh [34:32] wait [34:34] uh yeah wait is the sparity only on the [34:39] MLP or do they do something with [34:41] attention because we were talking about [34:43] that earlier but maybe okay we spacity [34:47] on all weight services token bearings [34:49] That's Oh, there's also detention [34:51] weights then, right? [34:53] >> Yeah. I mean, even token embeddings. So, [34:56] >> yeah. So, that's [35:02] Yeah, I I kind of this feels like I need [35:06] to spend a lot more thinking the type [35:09] thinking about how that even works in [35:12] terms of what does it means in terms of [35:13] the attention layer. [35:17] uh [snorts] [35:18] and not something I'm going to [35:20] understand like today directly. [35:23] >> Yeah. I mean I'm not sure like if how do [35:26] they choose I mean is it just like [35:27] purely random? It could just be like [35:29] purely random but I mean at least at [35:31] least I could be wrong but like right [35:33] now my understanding is maybe they just [35:34] set certain weights to zero and so like [35:38] any linear combination with those sets [35:40] of weights essentially ends up being [35:42] zero. [35:47] Oh yeah yeah yeah they have syn I do [35:50] wonder if random and the things like do [35:53] we ensure that each neuron has some [35:56] connections [35:58] uh how does that work in terms of [36:02] attention and like the kind of attention [36:05] patterns that can be done and I guess [36:06] attention patterns are not affect they [36:09] are still calculated they just [36:10] calculated with less neurons kind I it [36:14] feels like like the concept is simple [36:17] but like the the the actual implications [36:20] of how the thing works seems like I I I [36:23] I too uh I'm not awake enough right now [36:27] to to figure it. I did this like spend [36:31] like a while like going outside and [36:33] thinking about it for a while instead of [36:34] trying to read a paper on this. [36:36] [laughter] [36:38] >> Yeah. No, I I took a quick look at that [36:40] appendix 8.1 and there's some like weird [36:42] things there, I think. Uh yeah, I mean, [36:45] so there's Yeah, I think Decker just [36:47] posted. So there's apart from uh [36:49] weights, they also enforce sparsity on [36:52] like activations and like basically at [36:56] each location like you know uh they they [36:59] only retain like the top K activations [37:02] uh and zero zeroing out all all other [37:04] activations. [37:06] So [37:08] But yeah, I think this thing right the [37:10] the thing the purple on the [37:11] >> Yes. Yes. Yes. Yes. And this is on top [37:13] of like the weights, right? So the they [37:15] they zero out the weights and then they [37:16] also zero out the activations. Uh and [37:20] and there are some other weird things [37:22] like they don't use positional [37:23] embeddings. I don't know why. Uh I don't [37:26] know. Yeah, there's there seem to be [37:28] some things which I don't Yeah, it maybe [37:31] maybe it's well motivated but it seems [37:34] bit arbitrary. [37:36] they also like learn a separate or I [37:38] don't know they learn or they something [37:39] there there's like a separate like a [37:41] biogram matrix that they I don't know [37:44] learn or they maintain they say that you [37:46] know they don't want like this bars [37:47] model to learn like things like bgrams [37:50] and so they have a separate matrix for [37:52] that again I have no idea how that works [37:55] but yeah yeah it seems like there are a [37:58] lot of small things yeah [38:05] Uh yeah, I I guess this also seems like [38:10] there's probably a lot of things on the [38:11] paper and we might not be able to [38:16] like figure out what is this today. Uh [38:19] but let's let's try uh look at [38:25] it. [38:31] Yeah. So [38:34] um [38:39] anyway [38:41] uh okay let let's let's continue uh so [38:44] par mod spar circuits okay let's first [38:49] okay first I think yeah interpretability [38:53] metric let's see that [38:55] hometric mean of number edges in the [38:58] circuit a cursor has created task to [39:07] Uh okay let's let's mess from the start [39:11] with sparse models for models with the [39:13] parameter set to zero. Uhhuh. They are [39:16] pretrained on Python code. Okay. So they [39:19] are just training on Python. Uh yeah [39:22] probably that also makes things simpler. [39:25] uh simple [39:28] where they are forced to choose between [39:30] one or two compressions. Your your idea [39:34] seems to be the latest paper is train [39:37] some simple models that are easier to [39:39] understand and we can do science on them [39:42] instead of trying to understand [39:44] something that's more like real models. [39:48] That's that seems like a good idea [39:54] of our models. We isolate the small [39:56] spirits the model performance time using [39:59] a novel pruning method. Okay. Yeah. [40:01] Yeah. So they they do have some novel [40:02] pruning method to uh automated to [40:06] actually figure it out. They they didn't [40:08] just manually [40:10] uh look at them. I maybe they did have [40:12] to manually figure out what they the the [40:15] parts mean but yeah they have some way [40:18] of finding it interpretable model should [40:21] be easy to entangle in behavior should [40:23] be implemented by compact standalone [40:26] circuits. [40:29] H [40:30] this is not [40:34] obvious to me [40:38] but okay [40:43] like you know maybe there is some [40:45] behavior that happens because a [40:48] combination of a lot of different [40:49] circuits that uh do a lot of complex [40:52] things and this only true of very simple [40:55] behaviors [40:56] uh uh And maybe this is also true like [41:00] on this small very small models that [41:03] don't have like a very complicated [41:04] things going on. Uh [41:09] so [41:12] well I guess that that mean that will [41:16] also mean they will be harder to [41:17] untangle. I don't know. [41:20] A span circuits are defined as a set of [41:22] nose created by edges. Uhhuh. Definition [41:26] not is maximally covers the root and [41:28] covers weight to matrices. Uhhuh. [41:32] So so they actually yeah the the [41:37] we define a node to mean individual [41:39] attention p head or channel right. [41:43] Uhhuh. So yeah the makes sense. [41:47] uh [41:49] an edge then is a nonzero entry in a [41:52] weight matrix that unconnect the two [41:54] nodes. [41:58] So yeah basically anything that where uh [42:01] you do some operation and you get some [42:04] vector at the end uh notes are the the [42:07] things you get and the and h are the the [42:11] weights that those vectors get uh [42:15] operate get. Yeah, that they produce [42:18] that part of the matrix of the resulting [42:21] vector [42:26] also. Okay, someone found the part about [42:29] learning when I was reading that pro the [42:32] model the smile with achieves a target [42:35] loss on the T distribution the target [42:38] loss is 0.5 everywhere [42:41] we prove by deleting some susceptible [42:44] nodes alone across all token positions [42:50] and [43:01] Oh, so they're meablating the [43:05] uh notes [43:10] uh until they [43:13] Yeah. trying to keep the same loss. [43:25] Well, not less than some target loss [43:28] anyway. Uh [43:35] but then that does not I mean [43:40] [snorts] [43:41] okay but then those nodes which are like [43:42] quote unquote deleted then also have an [43:44] impact right? [43:47] I think trying to I think they [43:49] automatically try to find notes that uh [43:52] which notes don't uh reduce the loss too [43:56] much. [44:02] Oh, let let me so so let me let me go to [44:05] the parting uh [44:20] Oh, okay. So, there a set of masks which [44:24] they used to get the respective node [44:27] locations. [44:29] Let's have a step for no idea what that [44:32] means. [44:34] uh we train the mass parameters by using [44:36] sigmoid derivative super gradient [44:40] to back properate to the whatever that [44:43] function is [44:46] oh I don't know what the straight [44:48] estimation is I think it's a a thing [44:51] where [44:52] well not not that I think there was a [44:56] thing where there are some operations [44:58] that are not uh not differentiable uh [45:02] But you can you can put them on middle [45:04] one network and you can still get a [45:07] gradient uh by there are some that can [45:10] be estimate that's you can you just skip [45:13] them in the gradient calculation and [45:15] that works uh [45:20] >> yeah no I think for non-ifferiable [45:23] uh functions that is an estimator that's [45:25] used for the gradient [45:27] uh uh no but okay so but this looks like [45:30] that they have like this is an automated [45:33] algorithm for like circuit discovery. I [45:35] mean not circuit interpretation but [45:38] basically you take a task and then you I [45:41] mean you know you have like data points [45:43] for the task and then you run this [45:45] optimization to find like the circuit [45:48] which is you know which for which like [45:51] you know for a given level of loss which [45:53] minimizes uh the circuit size in some [45:56] sense. [45:57] >> Yeah. while keeping the loss above [45:58] certain level and then you I guess yeah [46:01] I mean then the next thing would be you [46:03] try to interpret what different nodes in [46:05] the circuit actually [46:06] >> I mean in the end the the the the second [46:09] part is harder to to automate you can [46:13] probably automate it with something like [46:15] the way you try to automate labeling [46:17] sports color which is [46:20] >> as tragic to do it [laughter] [46:22] >> the [46:24] yeah the the labeling is is automated [46:27] for example in Wipedia by by LMS [46:30] labeling it but at at the same time now [46:32] there are some people attempting to [46:34] automate the the attribution graph [46:37] construction uh like like in the biology [46:40] of LM's paper like constructing those [46:42] attribution graphs so like automating [46:46] super node clustering and stuff like [46:47] this um there's attempts at that uh [46:51] which I haven't read yet and that would [46:53] be interesting to look into I can send [46:55] some or the one I know about. [laughter] [46:58] >> You you have song that you want to look [47:01] at. You can put it on the reading group [47:03] channel and uh we could I could pick it [47:06] for next week or whatever. Uh [47:09] >> yeah, [47:10] that there's outer of that in the AI [47:13] discord. So he could also maybe come and [47:15] present it. [47:17] >> Yeah. Uh yeah, I don't know. Yeah, you [47:21] can ask him or or I can try ask him uh [47:24] put the channel on the channel and yeah, [47:27] we can try to find out. [47:30] Uh [47:35] anyway, yeah, let let's let's wrap to it [47:39] and we I think later how which papers we [47:42] want to read and how which ones we might [47:44] get the out for. uh [47:48] minglated. [47:52] Okay. So yeah, they they are doing some [47:54] some pruning automated pruning and they [47:57] get some circuit and I guess part of the [48:01] idea is also uh so far they they might [48:05] have done maybe they say something but [48:08] this seems simple enough that they might [48:10] have just figured it out manually what [48:12] the things mean. Uh [48:17] I think this kind of method like the [48:20] reason part of the reason this doesn't [48:23] work if you do something like this but [48:25] with the sparok encoder is that uh the [48:29] spar decoder is not a good [48:34] uh two things first the spar code is not [48:36] that good at reconstruction so you are [48:38] going to break things if you try to [48:40] apply patient pro and Second, uh it's [48:46] also not completely disangling [48:50] the features such that if you ablate a [48:56] sparse encoder feature, you might be [48:59] ablating [49:01] whatever the true features in some more [49:03] vague philosophical sense, whatever that [49:05] means, uh are uh you might be uplating [49:10] more than one thing and your so your [49:13] casual you don't have the right basis to [49:15] to do abilations on to disentangle the [49:19] model nicely and I guess the idea here [49:21] is this might be easier here because the [49:23] model might be more disentangled [49:26] even here it might not be the case [49:31] I will have to think of how much this we [49:35] actually can be sure that or how much it [49:40] will be true for this version of these [49:41] models that they are more disentangle [49:44] and each neuron means a single thing or [49:47] less things [49:49] uh [49:52] such that you can [49:55] yeah you can really say I don't know [49:57] this is the neuron that does that [49:59] represents whatever uh and not have that [50:03] and not have poly semanticity and things [50:05] like that uh uh [50:17] bridge. Uh anyway [50:20] also yeah we here let's also look at the [50:23] bridges thing [50:27] away sparse mode [50:29] of bridges [50:31] mapping [clears throat] with intense [50:33] sparse mode activation [50:38] was a person layer. [50:43] Wait, what? [50:45] The sponge model size a sense of breeze [50:50] the densation [50:53] layer. [51:05] >> I probably need to go. Yeah, know they [51:07] say that it is basically you know each [51:10] it does seem very much like a sparse [51:12] autoenccoder like if you look at that [51:13] note they have at the bottom. [51:16] >> Oh uh [51:23] each splits can be view as a sparse [51:26] encoder where the lat space is the spar. [51:35] Okay. [51:37] But [51:39] so they are turning away the sparse [51:42] model of bridges. But but the the the [51:46] weight sparse. So they have a way spar [51:49] models and they have some sparse colors [51:52] that somehow map from the rear model to [51:55] that one [51:57] or is the [52:00] match compress will be separate with the [52:02] sp and the civations. Okay. Yeah. [52:06] So compared to this we use multiple [52:08] bridge going bridge loss terms in [52:10] addition to loss. [52:14] Okay. So they are doing a weird thing [52:15] that also this also seems like it could [52:17] be some paper [52:20] like but so they are they so are they [52:25] training [52:26] uh a new model that has some weird loss [52:31] that has to have the [52:34] yeah that that that has is trying to [52:36] keep it aligned to the spar encoder of [52:40] the original Then [52:52] want the weight sparse model to match [52:53] the smallest computations with the [52:55] bridges accative translating between the [52:57] spar and the activations. To compare [52:59] this we use three bridge laws terms [53:03] addition to predate laws. [53:08] Okay. [53:10] I feel like this could be its own paper. [53:13] We could spend an entire day on the [53:15] reading group like trying to understand [53:17] these bridge things and talking about [53:19] what proper what things they might have. [53:23] Uh the is just an appendix for [53:31] a cool idea though. [53:33] >> Yeah. [53:35] Yeah, [53:35] >> they they they are training the dense [53:38] and sparse model at the same time [53:41] basically. [53:42] >> Yeah, [53:43] >> with disconnected loss. [53:45] >> It seems like they actually train the [53:47] dense model first and then it seems like [53:49] they freeze it and then trains the [53:51] sparse model based on that. Although I'm [53:53] not totally sure. [53:54] >> Yeah, I think I think they they train [53:57] spar colors. [54:00] >> Uh no. Yeah, I think they they freeze [54:02] the they freeze the original model. Uh [54:06] >> oh. Then how does it differ from normal [54:08] way? [54:09] >> No, because you have a they have a [54:11] second model not just a sparse models [54:13] that is sparse and has a loss that like [54:17] like like they are trying to map the the [54:22] original model into this new sparse [54:25] model. So it's not just reconstructing [54:28] the different uh the different MLP uh [54:32] layers for example. It's it's also like [54:35] taking account the the loss of the whole [54:37] model on some data sets. [54:38] >> Yes. [54:39] >> Okay. That's the difference. Okay. [54:40] >> Yeah. [54:44] >> Go ahead. [54:45] >> Oh yeah. I was just saying that they are [54:47] trying to make a mod a new model sparse [54:51] model that matches the sparse of the [54:55] encoder [54:56] latent on the on the real model. [54:59] >> Just the idea of like yeah keeping the [55:01] latent spaces aligned with a loss term [55:03] like do you guys know any like research [55:05] on that? That seems like a very [55:06] interesting idea. [55:11] I don't know [55:14] maybe [55:14] >> I mean there there was work I mean this [55:16] is like much older work but like you [55:18] know like domain adaptation [55:21] uh there this I think cycle GAN and [55:23] those kind of things they used to have [55:24] this thing where like you would align [55:26] latent space by like adding like a loss [55:28] term I can I can find some specific [55:30] papers I'm not even sure if cycle GAN is [55:32] the right example I have in mind but [55:34] there were like some [55:36] yeah there there was some work like in [55:37] aligning latence spaces for like domain [55:39] adaptation and those kind of things. [55:42] I I feel like this idea is too cool to [55:45] be an appendent and should be a throne [55:47] paper and hopefully they publish more on [55:49] that [laughter] [55:50] but but yeah uh [55:58] >> so this bridge I mean this bridge has [56:01] its own set of weights [56:05] and like does it participate in like the [56:08] feed forward of the sparse model? I mean [56:10] I'm I'm a little confused like [56:12] >> so the the bridge it's uh they say [56:15] elsewhere that the bridge is kind of [56:16] like a sparse [56:19] like it's it's not sadly but yeah [56:21] something like that and the the new [56:24] model is just train with a with a loss [56:27] the sparse model just train with a loss [56:29] with includes matching the the the [56:32] bridge [56:37] Right. [56:39] Yeah. So, so here we want this weight [56:42] sparse model to match a dense model's [56:44] computations which the bridge is [56:46] accurately translating between the [56:48] sparse and densivations. [56:51] To accomplish this, we use three bridge [56:53] loss terms in addition to normal [56:55] pre-training loss. So in addition to so [56:57] they train a model. [57:00] So, so they are training a a sparse [57:03] model that is trained to have low lower [57:06] loss uh model data but also at the same [57:09] time is train with this bridge loss [57:14] [snorts] uh which [57:22] uh [57:24] near [57:26] normalize mean square error. [57:33] Wait. So is this of the H1? [57:39] This is the residual activations of the [57:42] BS of the T and the spars. [57:46] Uh [57:52] what's the this function [57:55] encoders? Ah okay. So this they are [57:58] isn't this just if I understand this [58:01] correctly isn't this just they are [58:02] adding the first one is just they are [58:05] adding the the the [58:11] error of the bridge to the to the [58:15] >> to the loss. So it's they they're trying [58:17] to minimize the error of the bridge. [58:25] Uh, I'm curious how they train the [58:27] bridge if they train. So, so yeah, what [58:30] I'm not sure if they So, [58:33] do they train the bridge first and the [58:36] bridge is also frozen and then they do [58:38] this or they are truly training the [58:40] bridge at the same time? [58:43] Wait, please [58:52] location [59:02] multiply the bridge. [59:11] Yeah, but [59:14] well, [59:16] okay, makes it. But what's the loss for [59:19] the bridge though? [59:27] I mean, it's I guess [59:29] end to end, right? So [59:32] like all the loss terms [59:38] uh or is it like a separate head [59:44] >> that's [59:48] >> because if you look at the feed forward [59:50] like I mean I still I mean like if I I [59:53] mean I can understand during training [59:54] sure you have this loss but but like [59:57] during inference like you basically [59:58] don't need bridge right if you're doing [60:00] any only inference through the sparse [60:02] model. You don't need the bridge [60:03] weights. [60:31] Yeah, this is definitely like [60:34] complicated. [60:35] >> Yeah, I feel like this is the [60:40] separate paper that has been put in the [60:42] appendics on this one. [60:45] >> Yeah. And if it works like I mean I mean [60:47] I would suspect maybe like you know it [60:49] doesn't work very well because if it [60:50] does work it does feel like you know you [60:52] might even [60:54] >> maybe put it here because it didn't work [60:56] very well and if they had people have [60:58] made it's on paper. I I have a kind of [61:01] hypothesis is that this is like first [61:03] time after a long time open I released [61:05] some open paper with some actual details [61:08] like this. So maybe the the people in [61:10] the interpretability team are like oh [61:12] let's put as much as possible they [61:14] finally let us release something so [61:16] >> oh yeah maybe [61:16] >> fill in everything [laughter] that we're [61:19] doing. Uh [61:20] >> yeah like maybe wouldn't have let them [61:22] put it to paper or something. I I don't [61:24] know what how the inside politics of [61:28] work. [61:28] >> They just shout another paper in the [61:30] appendix. Basically [61:32] >> this seems like if we don't get Leo to [61:35] come, we could ask him why they did this [61:38] [laughter] [61:40] >> and he would be like I cannot say [61:43] anything. Sorry. [61:45] >> That's okay. [61:45] >> Yeah. [laughter] [61:47] >> Maybe maybe not. Yeah. I still remember [61:51] asking something asking him saying [61:54] something on on [61:57] a Luther like some speculation about GP [62:00] where about yeah the years ago and he he [62:04] just responds with the uh with a [62:11] like a sus emoi emoji that he couldn't [62:14] say anything but I was actually right [62:17] [laughter] [62:18] I think or at least vaguely night [62:21] was later funny. [62:23] >> Nice. [62:28] >> Uh yeah, never mind. Um [62:35] I also saying other things that I don't [62:37] know through. Anyway, [snorts] [62:42] >> I I I still kind of struggle to see when [62:45] when they mentioned mean like the the [62:48] pruning algorithm [62:51] uh when they say mean [62:54] ablated. [62:57] >> Yeah, let's let's switch a bit to the [62:59] programming and if people still want we [63:01] can go back to the other thing. I try to [63:03] understand the bridge thing again. Uh [63:05] after [63:06] >> if you want we can stay at the bridge up [63:08] to [63:10] yeah I don't know uh okay yeah better [63:16] then okay let's finish figuring out the [63:18] bridge so the main thing [63:21] I I'm [63:28] okay [63:28] >> I guess the uh the last paragraph on the [63:30] page is is really interesting to me at [63:32] least it seems like the way they [63:34] construct a sparse model is by just [63:36] subtracting two different activations [63:38] basically. So like double versus single [63:41] quote task like to to just get the [63:44] circuits for that specific task. [63:46] Although it's kind of not clear to me [63:48] how they actually sparify it though like [63:50] that you know [63:53] is there some cut off like below which [63:55] like the weights just get set to zero. [63:57] It's not very clear. [64:00] >> That's part of the pruning algorithm. [64:01] No. [64:04] Oh yeah. Yeah. Probably. So they they do [64:06] this and then they apply that to the the [64:08] result. Yeah. [64:29] >> Yeah. I don't know. This feels like if [64:30] people have like taken the first few [64:33] important papers uh about something like [64:36] sports and coders and put them all on [64:38] the same thing and say here it is [64:42] [laughter] [64:52] or at the beginning I remember they had [64:54] something about [64:56] some threshold where it's set to zero. [64:59] I forgot how it exactly was. [65:04] >> Uh [65:08] oh. Yeah. Goodbye. [65:11] Yeah. I'm only saying if he's uh [65:14] going how to I don't know how long [65:18] people [65:19] want to keep talking about this. I'm [65:22] fine with keep going. People uh want I [65:25] don't have much else to do today. Uh [65:27] well maybe later not that well I don't [65:30] want a lot [65:33] longer but yeah let let's go for a bit. [65:36] Okay so let's going back to the B so [65:39] quickly. I [65:42] Okay. Yeah. So the car is doing a [65:44] diagram of the SD term. I Yeah. I I was [65:49] not paying that much attention to that. [65:51] I just don't have it figured that out. [65:55] But uh that's is still right that those [66:00] terms are still to train the only the [66:05] train the spar well you train the the [66:08] sparse model right [66:17] or wait is the so so is this SD term so [66:22] so with the SD term are you training the [66:24] the original model with that or or or [66:26] are you taking the gradient of that but [66:29] only updating the the first [66:40] what what I'm not sure is what they [66:42] train the bridge on or do they take this [66:47] okay [66:48] the bridges have encoders and decoders [66:56] Ah, okay. [66:58] Wait. So the do do you mean that? So so [67:01] these losses are for the bridges and [67:05] this is for the the Okay. Okay. I was [67:10] understanding it wrong. I thought all [67:12] the all these three losses were on the [67:17] images of them. [67:26] Here it says we have a kayan to train [67:29] the sparse model to asset activation [67:31] from them models. Are they counting the [67:34] bridge as part of the spar model there [67:36] or what? [67:42] Uh so so they they train both the spars [67:45] model and the bridges on those loes. [67:50] So they do train both in parole. I [67:53] >> I I know not not sure. Uh that's what [67:56] Decar is saying but yeah I know not not [68:00] I'm not that confident on the [68:01] explanation like they say bridges to [68:05] convert one kind of activations to the [68:07] other at various single locations. So it [68:10] seems like the the bridge is not part of [68:13] the spark [68:14] >> here. [68:15] So, so the the understanding is that the [68:18] yeah the the fence model is frozen but [68:21] the par models and bridges are not [68:33] supported yet [68:36] possible combinations of spades and [68:39] devil slayers. Wait, what? [68:44] What does that mean? [68:49] layers. Yeah, I I'm not sure I [68:51] understand that. Uh yeah, [68:56] I I think I'm not sure, but I think it [68:59] means [69:02] >> they want to like swap the sparse. [69:05] Actually, maybe I'm not sure, [69:08] >> but it seems to me like it means that [69:10] they want to be able to swap any layer [69:11] of the spars for the dense and like vice [69:13] versa. [69:16] >> Mhm. [69:19] >> Does it make sense? I guess [69:24] >> and vice versa. So, they want to [69:27] translate from dense to sparse and from [69:29] sparse to dance activations. They will [69:32] not translate [69:34] >> like there's there's a loss term for [69:36] each of those for the bridges. Those are [69:37] two of the three loss terms bridges [69:41] >> and the dimensions of the sparse [69:43] activations are higher [69:46] right [69:48] this [69:53] does the wait does the paper write [69:56] anywhere I was going back to the this [69:58] section to see if they say something [70:00] that [70:02] Okay. [70:10] Uhhuh. [70:13] >> Yeah, it makes sense to me. Does it make [70:16] sense to you? [70:17] >> Yes. We use a normaliz terms that trains [70:21] the bridge encoder to accurately predict [70:23] the sparse activation from dense ones. [70:31] Ah [70:33] okay. to this [70:36] is this also the [70:41] okay so so so yeah I I think they didn't [70:45] explain it again here but yeah I think [70:47] that does mean this is yeah that this is [70:51] the also the loss for the for the bridge [70:54] right [70:56] >> it also says uh we run hybrid forward [71:00] passes of the sparse and dense models [71:02] using the bridges to convert one kind of [71:04] activations to the other at various [71:06] single locations. So I guess they [71:08] actually do even part of the training is [71:10] swapping players. It seems like [71:15] >> I know I feel like this is like [71:19] >> kind of not very well explained because [71:22] it's all like they have this whole [71:24] complicated this kind of a whole paper [71:27] in like one page. [laughter] [71:30] And we'll have to see if if they put [71:35] more things about previous or if people [71:39] use them or something. I don't know. [71:45] I mean maybe the reason this is only an [71:47] appendix like we don't know if it's the [71:51] open AI [71:53] only only they only publishing this so [71:55] they have to publish all their stuff or [71:58] they try bridges and it didn't work that [72:01] well I guess we could is this from [72:05] is do they have any [72:09] are this for bridges or is this [72:16] like I'm wondering if they have any [72:19] interesting result from the bridge [72:23] thing. [72:25] There is code. Yeah, if there is code [72:27] then that's nice because uh people can [72:31] actually look at it and figure it out [72:33] how it works. And uh yeah, hopefully [72:36] someone will do that. I don't think I [72:38] will, but if some of you want to [72:40] actually look at the code and put some [72:42] explanation somewhere, that will be nice [72:45] and I link it on the discord or whatever [72:48] if it's done. [72:50] Uh [72:58] anyway, let's let's see if there is [73:00] anything we will focus on this thing. Uh [73:04] I'm curious if there's other things we [73:06] were missing and all of this. Let's look [73:09] a bit part [73:13] city or well I mean if people want me to [73:15] keep looking at the bridges or look at [73:17] some specific part just uh tell me and I [73:21] can go to there. [73:25] >> I'm still trying to par pass the mouth [73:28] [laughter] [73:29] behind the bridges the terms. The C says [73:32] the the code does not include bre which [73:35] is okay. They have to be about like [73:39] there's they have to probably they're [73:42] probably going to publish a new paper on [73:43] that right or like what like uh it seems [73:49] kind of I maybe it did actually [73:53] didn't work. Uh does a paper anywhere [73:57] have any results from bridges? Maybe it [73:59] didn't work well and they just mention [74:00] it because they spent a lot of time of [74:03] their lives actually getting the bridges [74:06] to work and they didn't actually work [74:08] that well and then they just wanted to [74:11] put that but mention it but not uh it [74:14] didn't work well enough to actually you [74:16] know make it a big thing on the paper. I [74:18] don't know. [74:21] I if we if we can I maybe they they also [74:26] I wouldn't be surprised if Leo already [74:29] or some of the other say something [74:31] something Luther [74:37] >> like when I'm still looking at the [74:39] bridge bridges math like when I look at [74:42] the L K it seems like they compose like [74:48] they so when they go from D to S, [74:53] they first [74:54] do the F function. [74:57] So, so they first first basically [74:59] translate from from from dense to sparse [75:02] and then they do all the sparse layers [75:05] until the end. Uh, no, wait. [75:11] Okay. Well, [75:13] yeah. Like one thing I'm confused about [75:15] is couldn't the FI be at arbitrary like [75:20] the the the the translate the bridge [75:22] basically couldn't that happen in [75:24] arbitrary place because when I look at [75:26] the visualization they they [75:30] Oh no it's only [75:34] after the first layer. [75:39] No I'm so [ __ ] confused. [laughter] [75:42] >> At least this is much more fun than on [75:45] the reading groups when I I'm just [75:47] reading the paper. I [laughter] [75:49] >> I'm not confused anymore because I read [75:51] unbedding as embedding. So So okay, now [75:54] I see now. Okay. So So okay, so now the [75:56] composition works normally like in math. [75:58] That that made me confused. So they [75:59] first [76:01] so they first uh apply apply the [76:04] translation from from from the dense to [76:07] the to the sparse. Then they do all the [76:10] all the all the layers in the sparse [76:12] model until the end and then they take [76:14] the KL and they do the analogous thing [76:18] for for the from sparse students and [76:20] that's part of the loss function. I [76:22] guess one thing what I'm confused about [76:24] is why they first [76:28] hit ah for each eye. Oh okay I see now. [76:32] So because they do it for each eye and [76:36] at each layer eye. So so the loss Okay. [76:38] Okay. Guys, you know the loss goes [76:42] uh for each like [76:45] so like first term is one layer uh in [76:48] sparse model uh translates to bridge uh [76:52] uh translates through bridge to dense [76:54] model and then go go to the end of the [76:58] dense model and and take the take take [77:00] the final uh final output and then do [77:05] then do the the KL with the ground [77:07] truth. [77:08] And the second one in the sparse model [77:11] goes two times. So sparse model once one [77:14] layer sparse model second layer then [77:16] translation to bridge and then it goes [77:18] the the uh into the dense model and then [77:21] it goes to through the rest of the dense [77:24] model one layer another layer another [77:25] layer until the end then they so is this [77:28] thing that they can draw here right [77:32] >> yeah yeah yeah oh yeah he did draw it [77:35] right yeah okay I just grooked it now so [77:37] [laughter] [77:39] okay [77:40] >> I'm curious if if you understand the [77:42] first loss term because it seems like [77:44] the second two loss terms don't actually [77:46] depend on the bridge at all like uh fi [77:50] and gni are the the bridge encoder and [77:53] decoder and those are not anywhere in [77:54] the second two loss terms it's just [77:56] about the [77:57] >> they are in the second two loss term [77:59] there is fi [78:01] at at that's the first first thing [78:04] that's applied [78:07] first fi then msi up to ms building when [78:12] they [78:12] >> I see it now. Yeah, you're right. [78:15] >> So the the bridge is needed. Yeah. [78:18] >> Yeah. No. [78:22] >> Do you understand? [78:22] >> Oh my god. I I Go ahead. [78:25] >> What is the first lost term for I think [78:27] I understand the second two, but I don't [78:29] understand. [78:31] >> Uh it's [78:34] it's just [78:36] pure translation. [78:40] It's just the act so that the [78:42] activations are aligned. [78:44] >> I don't wait. [78:46] >> No, the first [78:48] you apply the [78:51] you [78:53] the activations of the [78:56] of the Yeah, it's just the difference [78:58] between using the B applying the bridge [79:00] to the to the dense model and the [79:03] activations on the on the sparse model. [79:06] So, it's just the error of the bridge [79:07] basically. [79:09] >> Yeah. how well the bridge reconstructs [79:11] hidden activations in the second model. [79:15] Uh [79:16] >> both like both both directions like [79:19] >> in both directions. [79:20] >> Yes. [79:21] >> Okay. I think I get it now. [79:23] >> Uh but I still I'm still not completely [79:26] sure is okay. So they have these three [79:29] love loss terms. So what are they [79:34] training with the those loss terms? the [79:37] sparse model or the sparse model and the [79:40] bridges and [79:43] do the bridges. [79:45] Yeah, basically that [79:48] mod [79:49] >> I'm not sure. [79:50] >> Okay, [79:50] >> but why do they say that? I guess it [79:54] makes sense because they don't explain [79:56] otherwise where the uh what how they [80:01] train the bridges, right? Or maybe they [80:04] say it earlier or later, but but [80:07] like in this this par is kind of weirdly [80:13] uh weirdly explaining because they are [80:20] really no [80:24] wasn't there at the top some training [80:26] section for the bridges. Maybe not. [80:30] Okay. [80:37] Yeah, but I guess you it makes sense. I [80:41] guess you you could train the bridges [80:43] more like SP encoder, but I guess we are [80:46] not doing that. Then it's not a there's [80:48] not that similar to encoder because you [80:50] are not training the construction model [80:52] on the loss of the original one. Uh [80:58] yeah [81:00] makes sense. I guess what you do what we [81:04] are what the way they are not mentioning [81:06] that they are not really mentioning this [81:08] because it yeah in their minds is the [81:12] same because if you you train in the [81:14] model right if you you you have the [81:16] bridges and you freeze the first model [81:18] by default what happens if you are using [81:21] the bridges as some Python thing that it [81:23] gets multiplied by the other thing is [81:25] that when you do back prop it just gets [81:28] everything gets updated the field is not [81:30] for everything so by default you will [81:32] train on the bridges if they are not [81:34] frozen. [81:37] So I guess unless they specifically mean [81:39] they they they fro they [81:44] Yeah, I think it could be more. Okay, [81:45] they we train a weight sparse model. [81:48] Okay, yeah, that makes sense. Maybe I'm [81:50] just being dumb and not reading it well [81:52] because I have in my mind the previous [81:55] idea that is like a sparse encoder which [81:58] is staying on the thing and then it's I [82:00] was being confused by that but yeah it [82:04] could be precisely the way [82:07] by being is open [laughter] [82:10] >> like alongside could mean first the that [82:14] first dense model then the second like [82:16] do they do it in parallel or serly Like [82:20] but alongside seems to imply more of the [82:22] parallel instead of [82:24] >> I think the obvious like thinking about [82:25] it like the obvious thing to do like the [82:28] the foul thing to do is training at the [82:31] same time because if you imagine having [82:33] like the actual code uh and having some [82:36] bridges right uh that you are [82:39] multiplying by the other thing right [82:42] you're taking the activations and you [82:43] are you are doing back pro through [82:46] everything uh by default All when you do [82:49] back prop the model uh you are also [82:52] getting the the gradient for the bridges [82:56] right? [82:58] Yeah. Yeah. Okay. That would make sense. [82:59] Yeah. To to do it in serial. That would [83:02] make it so complicated. Why would they [83:03] do that? [83:04] >> If you specifically freeze them, you are [83:06] going to also train the bridges if I [83:09] think about this correctly. [83:11] >> But yeah, the gradient would be more [83:13] warped in some sense. uh [83:17] like [83:19] since the since you go since that one lo [83:23] goes like from one to the other and so [83:25] on. It's like uh that would I think that [83:27] would make it confusing. [83:28] >> No, but I I the thing that said that you [83:32] Yeah. SPAR model and the bridges it [83:33] might not work. I was thinking if you [83:36] like the the other option that made some [83:39] sense was if they train first the [83:41] bridges and then the spar [83:44] the sorry and then the sparse model to [83:46] match the bridge. Uh but I guess [83:52] it's not always me that that couldn't [83:54] also work but [83:57] uh [84:01] right but you could try I could imagine [84:04] training the bridges uh making the [84:07] bridges as sparse at all color where you [84:10] train it on reconstruction laws on the [84:13] on the actual model and then you connect [84:15] to the thing but then you wouldn't call [84:17] bit will just be a sports encoder. Uh so [84:20] in that sense that's the way that it's [84:23] different from sports encoders. [84:29] Yeah. Anyway, [84:31] you are right. [84:33] [laughter] Not I'm not going to try to [84:35] talk about why my weird confusions make [84:39] some sense. You're just right. Never [84:41] mind. [laughter] [84:42] [gasps] [84:43] Hey, [84:45] >> there's a pretty funny paragraph on page [84:47] 16 where it says they identified a bug, [84:50] uh, but fixing it reduced performance, [84:52] so they just left it in and didn't [84:53] really explain anything about it. [84:56] >> Oh, wait, what? [84:58] >> On page 16. [84:59] >> Uh, okay, let me go to that. [85:04] >> At it's the paragraph that starts with [85:06] at at some point. [85:10] Anyway, before this, I I do think they [85:13] really should put the the code for the [85:16] Brig somewhere because if they did, [85:17] that's very annoying. [85:23] At some point, we do find a booking or [85:27] Ling that the metending [85:33] embedding [85:47] Okay. So they kept them. [laughter] [85:50] Okay. [85:52] Does that mean while [85:54] serious? [86:05] >> They're just like works better. [86:07] Open A in a nutshell. However, [86:10] uh however, oh okay, fixing this hurts [86:16] >> quality in terms of capabilities and [86:18] interpret. Okay. So, it's not just [86:20] capabilities but also interpretability. [86:21] [clears throat] Okay. Okay. [86:23] >> That made it better. Okay. [86:25] >> But so if I [86:28] >> this is the divine intervention. [86:30] If I understand my intervention [86:33] correctly, [laughter] [86:35] they [86:37] the the book is the book that [86:42] the [86:46] mean maybe I'm failing the context on [86:49] this on this thing. Uh [86:55] great also oh also oh yeah there's some [86:58] details here we start we got to the page [87:00] part I kind of skip and started not [87:03] reading everything else and maybe we [87:04] should read a bit more of the other [87:05] things but okay so they they they have [87:08] everything has at least four nonzero [87:12] uh non zero elements or whatever uh yeah [87:16] weights I guess values yeah [87:21] the LC linearly [87:26] so that the model becomes parser. [87:36] I'm not I'm not sure what does that [87:38] mean. So So they said the the sparity [87:41] thing is that they [87:44] wait. So this separate for the setting [87:46] waste to zero thing, right? This is just [87:49] the normal [87:51] uh you know norm the normal penalty or [87:56] what [88:07] is it doesn't it's not that important I [88:09] know it conf I don't sure I completely [88:12] understand but the the change in the [88:16] thing the book this LC [88:19] schedule in the book is changing is if [88:21] it's just the normal thing where you [88:26] have in the loss you have a term for the [88:30] uh [88:32] you know for the [88:36] uh [88:37] what was the name of the of the measure [88:40] where you have to how big the weights [88:41] are which just the yeah if it's I don't [88:45] know if it's the normal thing that there [88:48] is more losses or or it's the the top K [88:52] thing. [88:57] Oh, because it's the way Spar wait no uh [89:02] no [89:06] wrong. [89:16] Okay. [89:18] But [89:23] >> I I I I gave GPT the question like if [89:26] they are trained in parallel or not and [89:28] it's it's it's it's been thinking for [89:30] like two minutes now. [89:32] >> Did you just copy paste the paper into [89:34] it? [laughter] [89:35] >> Yeah. Well, Apple did a PDF. Yeah. [89:39] >> Uhhuh. Let's favorite [89:50] Also they use some weird norm that is [89:53] not layer norm. [laughter] [89:55] Okay. [90:01] Use so much things on this paper like [90:03] already getting the normal sparse models [90:06] to work. She's like insane amount of [90:08] work. [laughter] [90:10] Like this is this is reminds me of the [90:13] of the anthropic uh of the big anthropic [90:17] paper with the with the [90:21] that we spent two days on uh for the [90:24] transcoders and all that and the [90:26] activation circuits. Uh but that one [90:30] seems a lot longer. This one is [90:32] similarly complicated but they have [90:34] compressed it in a relatively short [90:36] paper. [90:39] I wonder if they like trying tried [90:42] simpler versions that just didn't work. [90:45] So they had to do all these changes. [90:48] >> I don't think they right about that. [90:50] Yeah. [90:51] >> Another thing we we could ask if we [90:55] >> Yeah. Yeah, [91:01] I mean I I am liking it and I feel like [91:04] this is the the day in the one of the [91:07] days on the reading group since the [91:08] entropic paper that we have spent more [91:11] time actually discussing the paper [91:13] instead of just [laughter] [91:18] reading it and being okay. Part of it is [91:20] because we don't understand it and we [91:22] kind of need to figure it out together. [91:24] But [laughter] [91:33] we train a weight sparse model. Alex [91:35] said starting with an existing dense [91:38] model, we train the weight sparse model [91:42] jointly with the bridges. Wait, is that [91:44] house nation or is that real quote? [91:49] Oh, that's Oh, if you if you go at the [91:51] very top, they say that they start with [91:53] the existing dense model and train the [91:56] weight sparse model jointly with the [91:58] bridges. Therefore, they do not train it [92:00] in parallel. They first they start with [92:03] train existing train. [92:04] >> The has found a phrase where they say we [92:08] train a weight sparse model alongside a [92:10] set of bridges. So yeah, I guess it's [92:13] Yeah, I guess that [92:14] >> they Yeah, they say both. [92:17] >> This is confirmation that yeah that yeah [92:21] this also teleps also reads it this way. [92:23] It's just me that I was with them. [92:25] [laughter] [92:27] Uh [92:28] uh [92:29] >> okay. [92:32] >> Anyway, uh [92:35] where was I looking at? Oh yeah, here. [92:40] Yeah, in I understand this abs stop K [92:44] thing then, right? Because so so my [92:46] initial idea is you have the model and [92:48] you you put some waste at zero, right? [92:51] But this sounds like it gradually lowers [92:55] the waste to zero. Oh no, wait. Is there [93:02] the [93:08] wait? Where does this this app stop [93:11] thing is playing? Uh, [93:22] okay. Yeah. What? [93:29] Okay. [93:44] Right. But I know you understand. [93:49] So, so when that game methods you have [93:53] the sparse to encoder, right? And uh [93:58] you take the activations that are most [94:01] activated and keep those. But I Yeah. In [94:07] the context of this paper, how where are [94:10] they using that? [94:12] Uh [94:16] you mean where where they are using the [94:18] app stop? Okay. [94:19] >> Yes. There there's a there's a how um [94:24] there's a diagram of the architecture [94:27] at page [94:31] uh page 17 the thing in order [94:38] >> if you go to page 17 does this this [94:40] diagram [94:41] >> the okay so this is on the bridges right [94:43] but they don't use on the normal span [94:50] in some way in the normal sparse model. [94:53] Uh that is just not that is not this [94:56] bridge thing. [94:59] >> Mhm. [95:00] >> Or or wait, is this film for the bridges [95:02] or just happens to be on the same page [95:03] as the expiration the brides? Oh, maybe [95:06] it's not. Yeah, it might not be because [95:08] it's on the methods thing. Wait, what? [95:11] Why is there a map stop K then? [95:15] I I guess it makes sense to put it but [95:17] uh well some sense but it seems non [95:22] like seems like an extra thing that is [95:26] uh okay yeah figure for that I I guess I [95:31] I conclusive they say why they did the [95:33] abs stop case sc [95:38] right you could just do the thing where [95:41] you set weights to zero [95:46] or or is that [95:48] >> they actually they actually do mention [95:50] like the weights versus activations [95:52] trade-off on page 15. They say um here [95:56] let me [96:00] we find that some amount of activation [96:02] sparity helps on top of weight sparity [96:04] but that too much activation sparity [96:06] hurts the capabilities interpretability [96:08] frontier. Again [96:19] we also have some evidence that [96:20] activation sparity emerges naturally [96:22] from weight sparity. [96:25] It's interesting. [96:29] I confuse why they decide to do both [96:33] instead of [96:35] writing a paper where they do weight [96:38] sparity showing that and then doing some [96:42] other thing where okay we also this we [96:45] figure out that if you also add [96:48] activation sparity on top of it you get [96:51] better results instead of this feels [96:53] pretty much like they they just tried [96:56] added a lot of things until you that [96:58] working like it seems like very like a [97:01] bit over engineer or something for the [97:03] like it probably works better but I feel [97:07] like there are multiple separated [97:09] details that that they are just putting [97:12] together here at the same time right [97:19] because unless I misunderstanding [97:22] something you could remove the app stop [97:26] K thing and still [97:29] have a sparse model by doing the weight [97:33] sparity thing, right? [97:40] Yeah. And I think they even say that the [97:41] weight sparity creates some activation [97:44] sparity. It does kind of feel like like [97:46] they're both pretty I don't know things [97:48] and they don't really explain either of [97:50] them like in full detail. [98:00] Okay. Okay. Okay. So this [98:07] right first. [98:22] Yeah. I feel this paper has a lot of [98:24] results and we we spend a lot of time on [98:26] the bridges and we haven't even read [98:28] most of the other things. Uh [98:32] so [98:36] oh uh in I saw [98:40] get in and then go out. He's been doing [98:42] that on groups. I don't know why but [98:44] yeah never mind. [laughter] [98:46] Uh [98:49] another aspect of interpretability is [98:52] feature character which empirically [98:53] seems closely related to sparity. [98:58] We find that increasing the weight [99:00] spacity naturally increasing [99:02] stream activations. [99:09] Uh [99:19] yeah. So [99:33] wait, but [99:41] Okay, but here they're talking zero [99:45] uh [99:46] as [99:48] as referring to the hidden dimension of [99:51] the model, right? For the number of [99:54] layers concerned, [99:56] not the top K thing, unless I [100:00] misunderstanded something and they are [100:03] the same thing. [100:19] They said for absolute top K uh they [100:21] normally set K to 1/4 of the dimension [100:24] of whatever the note is. So I think that [100:27] would be um you know the the the number [100:30] changes based on like where it is. But [100:34] it's like 75% of basically all [100:36] activations are just thrown away. Like [100:37] the bottom 75%. [100:40] >> Yeah. [100:42] Okay. So, where do they first [100:49] where do they first start talking about [100:51] the apptop K and why do they do that? [100:54] Because uh okay so that's activation [100:57] spar. So, so Rtop K is the activation [101:00] sparity. But in the results, so so here [101:03] they are only talking about weight [101:06] sparity, right? They are not mentioning [101:08] this apptop K [101:12] uh thing, but I'm wondering if they are [101:15] all of the results have also have the [101:18] abs thing for whatever reason or if they [101:23] have a train models without it and with [101:25] it and check both. Right? [101:29] Does the thing I'm saying make sense or [101:32] I just misunderstanding something? [101:48] Okay. A1 [101:54] kind of weird if if it's only on the [101:56] appendix, but okay. [101:59] >> Yeah, there's there's one occurrence of [102:01] ABS top K outside of appendix and it's [102:03] just in a footnote. It is pretty weird [102:04] that they [laughter] I feel like they [102:06] should talk about it more. [102:07] >> It's kind of weird that they add like a [102:10] different they have two ways to two [102:12] kinds of sparity. They add like a second [102:15] kind of sparity to the thing that just [102:19] didn't appendix. [102:34] Yeah. Yeah. I I think that's that that [102:36] that that's true. [102:38] But [102:40] but but then that means I'm confused. [102:43] I'm talking about the thing the C was [102:45] thinking that the LCD for the weight [102:47] sparing and abstract but then I [102:50] confused. But do why this uh let me go [102:55] to the [102:59] we were earlier on the the part with the [103:02] book, right? Uh let me find the the text [103:05] on the channel and go search for it. [103:11] Uh [103:15] but um [103:18] I guess I just set on the thing. [103:24] Ah [103:25] okay. [103:27] Why is there L0 scheduleuling if it's [103:32] Yeah. What do you mean scheduling? I [103:34] will I I would have imagine [103:37] uh that people [103:41] uh yeah that you just at the start [103:44] before training set some weights at zero [103:48] and that's it and that is how you [103:50] architecture work. Uh, but this sounds [103:53] like they [103:55] they have a schedule where they they [103:58] change them to zero slowly or something [104:00] or I just misunderstanded what L0 means. [104:04] Or maybe L0 means something else in this [104:07] context and it means like the the thing [104:12] where uh like normally there is an L1 [104:19] penalty on the law. So maybe that means [104:22] that uh yeah [104:31] >> I wish they were more clear [104:33] uh it seems like the L0 thing is only [104:36] for when they train sparse models from [104:39] scratch but it seems like the bridge [104:41] thing is mostly they're not training [104:43] sparse models from scratch there and it [104:45] seems like to me I might be wrong but it [104:47] seems like the pruning is used for the [104:50] uh like creating sparse model from a [104:53] dense model but it seems like they also [104:55] >> I think the the pruning is for getting [104:58] fils from a sparse model. The br the the [105:02] bridge thing is is how they they they [105:05] using the bridge and they are training a [105:06] sparse model that matches [105:09] uh that matches a a dense model and they [105:13] they do pruning on the sparse model to [105:15] find circuits like the pruning is [105:17] something like AC/DC or something like [105:19] where they are doing like automated [105:21] circuit search by doing pruning uh it [105:24] doesn't have to do with [105:26] the translation part which is the what [105:28] the training with the bridges we were [105:30] talking about earlier I think uh but I [105:33] think I can understand this. I think [105:35] they are just misexplaining things again [105:37] in that. Okay. So there is the L zero [105:41] norm. I I'm not sure this but I think [105:43] it's okay. So they talk about the L [105:45] norms of the weight which is just [105:49] uh [105:53] the the the norm is just I think isn't [105:56] this the square of the waist or [105:58] something? this the square of adding the [106:00] weights or something like that. [106:03] Uh [106:05] and so they they are just saying talking [106:09] about L0 as a way of saying the ways [106:12] that there are more ways that are zero [106:14] not here. But when they talk about L0 [106:18] penalty uh like scheduling elsewhere, [106:21] they mean they just mean the normal [106:24] machine learning thing where you add the [106:26] L0 to the loss uh so like as a way to [106:33] get for models to be uh like you know to [106:38] not over fit and whatever and and this [106:41] is just a conf they confusingly use the [106:43] term for both L for things because it [106:46] makes sense. [106:48] Uh but it's kind of a weird way of [106:51] talking that makes it [106:54] that or I'm not I'm misunderstanding [106:56] something and they are somehow gradually [106:59] putting the weights to zero but that [107:02] will confuse that will seems less [107:05] sensical way of doing things. Uh does [107:09] that make sense or are saying nonsense? [107:21] they they they [107:23] define it as [107:26] total number of non-zero elements in the [107:29] vectors [107:31] >> in this normal way normal machine [107:32] learning there is like such thing as [107:34] L0ization and there is some official uh [107:38] like there is some definition which is [107:39] like I think the square of the sum of [107:41] the uh yeah What's zero norm? And and [107:46] the just that the zero norm is uh yeah [107:50] it's like a there's a vector norms like [107:53] the yeah uh [107:55] >> you mean some of the magnitudes of the [107:57] vector [107:57] >> the monitors like so I think [107:59] >> isn't that L1 norm and this is L0 norm. [108:03] >> Uh [108:06] >> yeah I don't I don't remember what the L [108:09] norm is. [108:12] L0 norm seems to be to total number of [108:15] nonzero elements in a vector while the [108:18] L1 norm is the square of the magnitude. [108:21] So I guess that would make sense. It's [108:24] like [108:25] if you want as many weights as possible [108:28] to be zero then if you include the L0 [108:30] norm like the total number of non-zero [108:33] elements. [108:34] >> Okay [108:36] [clears throat] with the L1 norm then. [108:38] Okay. [108:38] >> Yeah. Yeah. Okay. But then what's what's [108:41] the LC scheduling then? [108:46] >> Not sure. [108:48] Yeah. [108:49] >> Like let me search scheduling. [108:53] >> Maybe maybe [108:54] >> because L1 scheduling is just [108:56] [clears throat] when the the the thing [108:58] where the No, what is actually L1 [109:00] scheduling? L rate scheduling makes [109:02] sense. L1 scheduling doesn't make sense [109:04] now that I think about it. We just don't [109:06] change anything on the Yeah. L0 and [109:09] Ling. What? What does February 5? No, [109:12] this is just [109:16] >> No, they they talk. Is that a typo? [109:19] Maybe because figure 25 is about [109:23] learning rate scheduling and and [109:30] I think they might be just mistype and [109:32] they make a learning rate scheduling [109:34] which makes more sense. [109:38] Let let me let me check. Okay. So, [109:41] uh [109:45] >> they do say the same term two times, but [109:48] yeah, it's possible. [109:50] >> Okay. [109:52] So, here they say learning rate [109:55] is but yeah. [110:07] Oh, no. Learning rate is connected to uh [110:09] L0. They say it's learning rate is one [110:13] over Yeah, I mean I think you're looking [110:15] at two. [110:17] >> Oh yeah, I saw that equation there. They [110:20] they they combine the the the [110:24] classic glaring light with this L0. [110:27] >> They change L0 [110:31] during training, right? [110:39] Uh [110:40] well [110:43] it I guess they always like compute the [110:46] total amount of non-zero vectors and [110:48] that just makes the [110:51] >> yeah but don't schedule dynamic in a way [110:54] >> the some vectors to zero [110:57] >> I understand the start right [111:01] >> yeah that would well that that would [111:03] decrease [111:05] another vectors are not going to become [111:08] zero normally they just get increasingly [111:12] smaller and start having if they get [111:14] really really small they will I guess [111:18] yeah normally the weights don't get to [111:20] zero right they just get like very small [111:22] float values right now [111:24] >> I I really think it might be related to [111:26] the pruning because in the appendex [111:29] section about pruning they mentioned [111:30] that like they train the pruning like [111:33] along with task loss. It's not just like [111:36] a static thing, you know, and like they [111:39] have binary masks for zeroing out [111:42] connections, which seems pretty related [111:44] to the L like I think that they might [111:45] have just explained it really badly, but [111:47] like L0 scheduling is how the pruning [111:49] works maybe. [111:54] >> Like if you look at uh page 21, it's [111:57] like [111:58] >> okay, [112:00] >> I don't know. But they really like if if [112:02] that is true, I feel like they really [112:03] should like connect them more. [112:08] >> Like my previous understanding of [112:09] pruning is that it's just something they [112:12] do after they train the model to just [112:14] find [112:15] >> No, they they say pretty clearly it's [112:17] trained like with the model. Like if you [112:19] look at um [112:24] >> it seems [112:26] >> Yeah, look at the loss function. [112:29] uh although maybe I'm not so sure now [112:31] actually we optimize our masks on a [112:34] linear combination of task cross entropy [112:36] and K the number of non-zero elements in [112:39] the mask [112:48] >> yeah know that sounds like they just [112:50] train the mask afterwards right [112:52] >> maybe yeah [112:54] >> that feel maybe we will need to look at [112:57] the code for some of these things but uh [113:00] >> oh they define the L0 norm as as I said [113:03] it while holding the number of non-zero [113:07] parameters in parenthesis L0 exactly [113:10] constant in [113:14] uh they hold it exactly constant wait [113:17] what [laughter] [113:21] yes but that's [113:25] I mean like this scheduling [113:30] I'm kind of I later going to look at [113:34] Luther and Twitter and see if people's [113:36] what people are saying about this paper. [113:38] I haven't seen anything because it was [113:42] it didn't came out came out like last [113:44] week, right? And I haven't been checking [113:48] Twitter that much. [113:50] I don't think people Well, I don't see [113:52] Twitter. I don't think people were [113:53] talking that much of this paper. Uh that [114:08] maybe Neil Neil is posting videos lately [114:12] uh a lot I think because some maths [114:16] maybe he'll do a video on this on this [114:18] paper and he'll understand it better or [114:20] also also complain about not understand [114:23] it and will feel vindicated. [laughter] [114:29] Oh, after each atom we update they zero [114:33] all zero out all but the largest [114:36] magnitude entries in each weight weight [114:38] matrix such that every matrix has the [114:40] same fraction of non-zero elements. So [114:43] they enforce [114:45] enforce [114:47] that there's always the same fraction of [114:49] non-zero elements. [114:51] >> Wait wait wait wait. So but [114:54] do which elements are non zero change [114:57] then [115:00] >> that's I think the case because they at [115:03] every ADB update they do the zeroing out [115:09] >> if an element [115:11] >> after applying every time [115:14] >> if an element is zero can it stop being [115:18] zero to grade that side when have we [115:23] don't gradient zero things or maybe not [115:26] we misunderstand it like [115:28] >> maybe there's plus bias [115:32] >> that makes it not zero [115:38] >> I I feel like I wouldn't think about [115:42] this in terms of like feel like there is [115:44] some math things I right now need to [115:48] remember to out your gray understand [115:50] that what happens if you have a zero and [115:52] you do that that matrix multiplication [115:56] and if things like that uh but yeah [116:01] >> you need addition yeah [116:08] which makes sense from biases as far as [116:10] I understand [116:11] >> like if you not have biases and multiply [116:14] by like have some weight that is zero [116:17] can the weight stop being zero or does [116:20] the gradient and not uh [116:22] >> weight m [116:25] weight matrix times input plus bias. So [116:28] if the bias so the times become zero but [116:32] the bi plus bias if that's not not n not [116:34] n not n not n not n not n not n not n [116:34] not n not n not non zero then that [116:35] becomes the value [116:38] that's my understanding now [116:40] >> but if the if the if we would have the [116:43] bias if it's still zero can it have non [116:47] like can the gradient not be okay zero [116:54] >> I guess it like okay see so let me think [116:57] you got a zero I increase a bit you it [116:59] could it could increase the thing so [117:02] maybe yeah I don't know why I thought it [117:04] it doesn't make sense uh yeah if you [117:08] don't keep [117:10] your weights are going to increase [117:12] probably if they are all zero [117:15] >> uh I I guess normally what happens if [117:18] you have things like relu and then your [117:22] zero [117:23] gains don't matter because there is a [117:27] nonlinearity [117:28] And it always Yeah, maybe that was I [117:32] that was why I was a little confused. I [117:34] was thinking like normally if you had [117:36] the weight to zero, it's going to stay [117:37] at zero because [117:40] uh [117:42] it has to increase a lot to matter for [117:44] the rail, right? Well, no, it could add [117:45] to something. Never mind. I don't know. [117:47] I'm confused. I don't think I know what [117:48] I'm talking about right now. Sorry. Uh [117:53] yeah, I need to review a lot of linear [117:57] algebra stuff. Well, [117:59] >> I think it can be changed. Yeah. [118:02] >> Uh [118:05] Rubic prop. [118:07] >> Yeah. Anyway, uh [118:14] yeah, I'm still confused about this, but [118:16] I guess I could try to gradient can [118:19] still update it. Yeah. [118:22] like I still confuse about what's the [118:26] what the zero scheduling is and why [118:30] these [118:33] I maybe I will have to read the whole [118:35] thing instead of the jumping around like [118:37] we are doing but uh [118:39] >> it seems to me like L0 scheduling like [118:41] the way they're like they basically just [118:43] mean progressively zeroing out more [118:46] weights because they say we enal the L0 [118:49] linearly over the first 50% of training [118:52] so that the model becomes sparer [118:53] throughout training. [118:55] >> Okay. [118:55] >> So, [118:56] >> but they enforce the the amount of zero. [119:03] >> Yeah, [119:03] >> every matrix has the same fraction of [119:04] non-zero elements. [119:07] >> So, they enforce it to be constant. [119:10] >> I think so. Yeah. [119:16] >> Okay. Yeah, I'm not sure that but okay. [119:19] That [119:20] That makes some sense. I was imagining [119:22] the model as having just having some [119:27] weights that are permanently zero from [119:28] the start and you just train the model [119:30] there. But I guess it's not that you [119:32] like increase the amount of zeros for [119:34] some reason. Uh [119:37] that seems like it's also kind of weird [119:42] and like I I wonder is it wouldn't it be [119:46] simpler and why why not just try [119:51] just do a model with less connections? [119:55] Uh like sure is if you want to do it [119:58] efficiently, you you still do matrix [120:01] multiplication uh but why not put zeros [120:06] and freeze the zeros, right? Like put [120:07] some amount of free zero from the start [120:09] and freeze those weights. Maybe you [120:11] can't freeze individual weights and not [120:13] entire matrix and try to make it [120:16] annoying. So it is actually easier to [120:18] code this thing. Uh but concept simpler [120:22] to just not have the connection right [120:25] >> like [120:27] it's seems like the L0 is like the [120:29] hyperparameter for the like how many [120:31] edges can exist in the circuit. So like [120:35] that's like it's like one axis on the on [120:37] where we were like confused at the very [120:39] beginning on the graph. [120:43] Uh [120:46] the training loss and prune circuit size [120:53] I think that the L0 is 1 X is the prune [120:56] circuit size right [121:01] okay so the [121:03] the thing the is saying is that yeah if [121:06] you right it didn't make sense so if you [121:08] initialize with some random empty [121:10] connections [121:12] uh the connections might not make sense. [121:18] Uh [121:21] like that they might not be like it's [121:24] nicer if you do the other thing because [121:26] the model can decide which things to [121:29] connect to each things. [121:32] And so it might want to [121:36] uh like for example [121:40] uh like you can imagine instead doing [121:43] four like taking each neuron and [121:46] connected it randomly to for the other [121:48] neurons. Uh but that makes it hard. The [121:53] model could learn to make the field with [121:55] uh with the connection with us. But it's [121:58] better to let it pick which connections [122:01] it wants, but forcing it to only have a [122:03] few of them. Uh, [122:07] and maybe it wants to some specific [122:10] neuron have have more connections or [122:12] something. Uh, [122:17] >> yeah. Okay. Make makes sense. [122:21] >> Mhm. [122:21] >> That's almost like a a super easy [122:23] indicator of like important neurons. if [122:26] if one neuron feeds into like, you know, [122:29] 20 different like nonzero other ones. [122:32] >> Yeah, that makes sense. [122:33] >> Pretty cool. I I really like that like [122:36] they I feel like people don't think [122:38] about complexity enough in general. Like [122:40] this is just kind of like my personal [122:41] thing. Um but I really like that like [122:43] they they kind of give like a effective [122:45] seemingly effective way to like [122:49] continuously vary the complexity almost [122:51] with the L0 scheduling. [122:54] >> Pretty cool. [122:58] I mean I guess normally you can also [123:00] vary the schedule if by making a video [123:03] model which is going to be more complex [123:05] but yeah [laughter] [123:11] and a model with lots of stand yeah [123:16] has this not been done before uh I don't [123:20] know I don't think anyone has train [123:23] sparse models in this Okay. Or like I [123:26] don't I don't know any paper [123:30] uh newspaper [123:32] >> maybe there is but do they mention [123:34] anything on I mean if there is some [123:37] important paper about this well let me [123:40] let's check the introduction in case the [123:41] or maybe there is a related work section [123:43] and they mention yeah related work [123:47] circuits I mean if not [123:51] uh okay yeah let's check the related [123:54] work and if there is and it's not under [123:57] related work will later I will probably [124:01] see people uh complaining about that on [124:05] the internet video I made a about this [124:09] on the 80s or whatever [laughter] [124:11] and uh we I'll mention that on the [124:15] channel if I see it but okay let's see [124:18] sports and color yeah this is just the [124:20] work and colors [124:24] But do I agree correctly that the L like [124:27] in the graph in the in the uh capability [124:32] interpretability graph that we were [124:34] confused at the very beginning that the [124:37] the one axis is the L0 [124:41] the interpretability axis is the L0 [124:43] hyperparameter. [124:44] >> But then what's the non-zero params [124:47] thing? [124:47] >> Oh yeah non-zero params. [124:50] So [124:51] >> why is why is this different from this [124:53] then? I that's why I'm confused. [124:58] >> Okay, [125:00] maybe [laughter] [125:01] maybe prune circuit size correlates with [125:04] the non-zero params. Then [125:08] this wait I'm [125:12] prune synchron size. This is the L0. [125:15] This is the size of the thwits that they [125:17] find with the pruning. [125:22] So, so higher means that they find more [125:26] bigger more complicated circuits. I [125:29] don't think that doesn't seem a bit [125:31] measure of interpretability because it [125:33] does mean that the model has more [125:34] complicated mechanism size or seems like [125:38] it if you try a [125:41] hard I guess if it's the same task. No, [125:44] but you could have the same task and you [125:47] are comparing for the same task, you [125:51] might still find like the bigger models [125:57] or whatever have more complicated. [125:58] >> Yeah, that will make sense because if [126:00] you look if you look Yeah. If you look [126:03] at at like you can see that that the [126:06] bigger the smaller models [126:09] because the the the the other axis the [126:11] color is another axis which is total [126:14] model size. Uh so so and you can see [126:18] that the bigger bigger ones at the very [126:21] bottom [126:22] >> you have you have you have [snorts] uh [126:25] the non-zero params you have it higher [126:31] and [126:34] the inter the circuit size also is lower [126:36] so like [126:38] uh [126:41] >> okay yeah I confuse because [126:44] >> uh trying to formulate what I'm what I'm [126:47] trying to what I'm what I'm meaning uh [126:51] the relationship like because because [126:53] the the amount of non-zero params like [126:56] if you have a bigger model then you need [126:59] bigger non-zero params to achieve the [127:02] same parity [127:05] >> to achieve the same room circuit size. [127:08] >> Yeah. But shouldn't this be a lot more [127:12] sparse then [127:14] >> right like for the same size [127:17] >> be also why [127:20] >> that's a bigger model so [127:26] what we saying [127:28] >> yeah that's a bigger model [127:32] shouldn't be it be the other way around [127:33] though [127:35] >> yeah like it's a I'm curious why so Why [127:38] is this one not why didn't trying [127:42] this one with more non-zero parameters? [127:44] >> Oh, it's nonzero params, not zero [127:46] params. It's like the inverse of L0 or [127:50] wait. [laughter] [127:51] >> Yeah, [127:55] uh [127:56] let's say you have why [127:58] >> no it's not okay. [128:00] So that you need both like why is not [128:03] the why don't we have one with more like [128:07] right [128:08] this will make sense if the colors were [128:10] the other way uh if this was the bigger [128:13] one because uh then the smaller one [128:17] couldn't have the biggest circle because [128:19] the biggest circle wouldn't fit but this [128:21] is the biggest model right is there any [128:24] reason why they can't have a model like, [128:30] okay, so if if this was backwards, [128:34] uh it would make sense to me because 10 [128:40] 10 to the 7 [128:45] uh [128:47] is so 10 to the 7 is 10 million, right? [128:51] So the smallest model can't have the [128:56] this level number parameters because [128:58] bigger than but this is not the smaller [129:01] model. This is the biggest right? Maybe [129:04] it's [129:05] >> maybe it's compressing the the features [129:08] much more. So the resulting resulting uh [129:12] uh circuit is much smaller because [129:14] there's much more compression. [129:18] But isn't this on the Isn't this the L? [129:23] If this L0, why it's not like why are is [129:27] there no uh you know model with 10 to [129:30] the 9 parameters and uh 14.8 [129:35] million non-zero parameters. [129:42] Setting more weights to zero reduces [129:44] capability but increases [129:46] interpretability. Scaling up model size [129:48] shifts the frontier upward suggesting we [129:50] can build larger models that are both [129:52] capable interpret [130:01] decreasing. [130:07] Wait, wait, wait. Save this frontier [130:10] one. So is the colors. Wait, no better [130:14] it's [130:17] making model sparser aka decreasing the [130:20] L zero norm of the weights. So [130:23] decreasing the total amount of non-zero [130:26] elements while holding the total [130:29] parameter count fixed trades off the two [130:32] harming capability but improving [130:34] interpretability. [130:36] I wish they had a sentence about the [130:38] colors as well. [130:40] >> Maybe. Okay, maybe is this graph not not [130:44] here? Maybe [130:46] at the top. [130:47] >> Okay, let's go to the [130:48] >> basically the same [130:50] >> but maybe it has some more explanation [130:53] here. Yeah, [130:54] >> it's a column. So, yeah, let's let's [130:56] see. So, more parameters then. [131:01] So more parameters [131:03] equals [131:05] um [131:07] better [131:10] capability and lower [131:13] cell. So yeah [131:16] >> and [131:19] lower interpretability. [131:21] >> Okay. So makes sense. So, so the more [131:26] >> and increasing the total amount of [131:28] non-zero parameters [131:30] uh [131:34] so decreasing the amount of zeros [131:38] makes the circuit size smaller. [131:43] No, it makes the circuit size bigger [131:46] >> and that's more interpretable. [131:49] >> And circuit side being bigger is more [131:51] interpretable. Yeah, I'm not sure this [131:53] is a good measure of being [131:55] interpretable. Wait, no. I think [131:59] >> smaller smaller is more interpretable, [132:00] right? [132:02] >> Yes. They say smaller smaller is more [132:06] interpretable. Yeah. [132:08] >> Because like this is the station. They [132:10] say better on this. [132:12] >> Here you want to go down. You don't want [132:14] to go up. [132:15] >> Down. Down. Yeah, it says better here. [132:18] Like this corner is the the best [132:20] according to them. I'm not convinced [132:22] that this is the Okay, so I think if you [132:25] keep the the model [132:29] uh Okay, so I feel like there's two [132:32] variables here like you could have the [132:35] the circuit be bigger because the model [132:37] is smarter and has a more complicated [132:40] algorithm to the thing, right? [132:43] uh like thinking about the damp sample [132:45] like with Neil's ging paper you have the [132:49] memorizing [132:51] uh the memorizing algorithm and the and [132:55] the actual modular edition algorithm and [132:58] the actual modular edition algorithm [133:01] I'm not it might have more nodes than [133:04] the memorized circuit [133:06] or well no actually no it will have less [133:08] because the memorized secret is just [133:10] going to have a lot of notes that are [133:11] kind of just doing [133:15] for each hard coding against sample. [133:17] Never mind. But anyway, it could be that [133:22] uh there is some that you have better a [133:25] better model and you have a more [133:27] complicated filid but is better in some [133:29] way like it maybe does more things. uh [133:34] and so so it increases the size without [133:37] and there is not a problem with [133:38] interpretability just the actual thing [133:40] with the actual thing is just [133:42] complicated [133:44] >> the [133:46] actually like incre [133:48] increasing the model size [133:51] >> decreases the prune circuit size [133:54] >> that's really counterintuitive to me [133:56] actually like it's almost like bigger [133:58] models like they learn like I'm not sure [134:00] how this factors into like memory but [134:02] it's like cuz you know I think this also [134:05] this is given a certain task like they [134:07] print it for that task you know [134:09] >> so it's a given [134:10] >> we can go to the here how they [134:13] >> there's more compression in the bigger [134:15] models so the circuits are smaller [134:17] >> yeah huh [134:19] >> yeah yeah it makes sense okay so then it [134:22] makes sense so then it's more the the [134:25] opposite thing where the bigger models [134:26] are finding simpler ways to do things [134:30] like they are going for memorizing to [134:32] having actual algorithms for the simple [134:35] tasks. Uh [134:39] at the same time, yeah, you you need to [134:41] call the law check. [134:45] You also take into account to the loss [134:47] because you could have a simpler model. [134:51] I wouldn't be surprised. The thing is [134:53] that on this task, all of the models can [134:55] do the task because it's very simple. [134:57] But maybe there's some more complicated [134:59] task where you have a better model that [135:02] has lower loss but is slightly more [135:04] complicated because uh it's doing more [135:09] things and the original model just [135:11] didn't do the task correctly. So it [135:12] didn't even have that quit complete or [135:15] whatever. [135:17] I don't know. [135:18] >> Yeah, I I can see that there might be [135:20] edge cases. But right now on this task [135:23] it seems like they figured out this in a [135:25] way general law the scaling law of in [135:29] all those axes and I wonder yeah that [135:32] they ask I think in the conclusion or [135:35] somewhere in the blog post maybe like [135:37] they don't know how it generalizes to [135:39] other tasks and bigger models. So that's [135:42] I guess open question. [135:48] Yeah, this reminds me of the whole like [135:51] how people think deep learning will go [135:53] before deep learning to go off of like [135:56] uh that it will just overfit [136:00] by default but actually in reality it [136:02] seems like a counter example. Another [136:05] one. Yeah, I mean that like again the [136:09] the [136:10] I was starting to argue earlier with the [136:14] Neil thing, but I did realize in the [136:18] middle that actually the the the prune [136:22] size on the on the groin task probably [136:25] goes down as the model learns the actual [136:27] algorithm. [136:29] Even if some more abstract sense is yeah [136:33] less comp like is more complicated [136:35] because the other thing is just [136:36] memorizing a bunch of things but yeah [136:38] you need a lot of you probably need need [136:40] a lot of notes to memorize a lot of [136:42] things [136:43] >> you need more bit yeah you need more [136:46] bits for memorization [136:48] since everything else is in a way a [136:50] compression but yeah [136:54] >> I guess not sure how general that is [136:57] yeah [136:57] >> seems like it will cases, but like where [137:02] like you will expect bigger better [137:04] models to have like for the same thing a [137:09] simpler circuit or whatever but [137:14] if you are testing a task it maybe use [137:17] and you don't check that you have the [137:19] same performance that I was saying you [137:22] could have two one that's more [137:26] complicated but great still like is more [137:28] because it gets a lot lower laws [137:31] uh [137:34] >> shortcut learning as well. [137:36] >> Oh yeah. So the the thing the car says [137:39] she mentions about sparse attention uh [137:42] but I think there are a lot of papers [137:44] that do sparse attention uh for the [137:46] musician this a whole pile of people [137:50] trying to find this ways to do attention [137:53] sparsely but that's [137:56] not the same thing they are not doing [137:59] zeros on way on weights right I think [138:03] they just put the normal the test [138:07] presentation thing they are [138:10] not calculating some attention values [138:13] instead of you know some parts of the [138:15] weight matrix or whatever and that's [138:16] just if you have very long context to [138:19] you to not spend as much compute on that [138:26] >> uh [138:29] like I don't know there is like flash [138:31] attention all the stuff I think it's [138:33] that right I haven't actually sent the [138:36] paper let Let me check the papers the [138:37] sent. [138:41] Yeah, I I don't not sure of this, but I [138:45] have this is like sparity on the [138:47] attention [138:49] uh scores, not on the weights or [138:53] whatever. [138:54] >> And here they just care about the [138:56] weights. [138:57] >> Mhm. [139:00] >> The pruning thing. There's also this [139:02] pruning. Let me put them here. So, [139:04] there's also this pruning thing. No [139:07] idea. [139:10] >> I haven't really looked at those papers [139:12] yet. So, it just seemed like they were [139:14] like I'm just interested in the idea of [139:16] like sparifying as you train, you know? [139:19] It seems like that what they were doing. [139:22] >> Yeah. After Hadam, we they they do the [139:25] spification. [139:28] I I guess but I guess people weren't [139:31] maybe people did these things already [139:33] but they were trying to [139:36] make inference more efficient. They [139:38] weren't trying to uh do something like [139:42] make it more interpretable. [139:45] Uh it's seems like it's probably like [139:48] the idea of let's train a a [139:52] network that's that's sparse [139:56] uh for interpretability [139:58] might be new in some seems like an [140:02] obvious idea but doing a sporting code [140:04] was also an obvious idea and took a [140:05] while for people to think of it. [140:08] >> [laughter] [140:09] >> Uh [140:12] in this case people might not have [140:14] thought of it until right now. So [140:17] [laughter] [140:18] uh [140:21] uh yeah they don't they talk about this [140:23] oh there is pass weight training section [140:26] wait [140:30] okay so [140:33] there okay so this saying that there is [140:35] there are paper for sparse train [140:38] they might not have done forability [140:41] Okay, [140:46] I guess we have to [140:52] let me go back to to the [141:09] Let's go to this face [141:26] Uh [141:28] no. [141:30] Yeah. [141:32] Uh [141:33] old papers. [141:39] It's funny that papers from 2018 in [141:43] extremely old now. [141:51] >> Uh yeah, this seems like people were [141:54] trying to do sparity for other reasons. [142:01] They were doing things like yeah brains [142:04] are sparse. They were trying to do for [142:07] optimization reasons. Uh [142:08] >> yeah like like there's this I remember [142:11] some papers saying like that you can [142:13] remove most of the weights and still [142:16] don't lose performance really. Yeah. And [142:18] there is like people these days what are [142:21] using mixture of experts some way. So [142:24] like [142:24] >> yeah that's another thing [142:26] >> papers training like there are there's a [142:29] lot of stuff in the research training [142:32] sparse models. [142:34] I don't know if there's a lot of things [142:36] about this specific way of doing sparse [142:38] models. How much maybe all of this over [142:41] complication is from there is some prep [142:43] literature that was training sparse [142:45] models for optimization resource and [142:46] they are copying some ideas and that's [142:48] maybe how explaining that but they [142:51] didn't make that very clear if so uh but [142:57] uh anyway [143:00] um [143:02] a I think it might the novel part might [143:05] just might be apart from the specific [143:07] details that they did for [143:09] interpretability and then did the [143:10] pruning thing and found uh [143:15] things the same way that it's not like [143:17] dictionary learning was a new problem or [143:19] people haven't done the spers [143:23] for other things that aren't [143:25] interpretability before. [143:27] uh is asked that uh people thought of [143:31] using it for interpity. [143:37] >> Uh yeah. [143:39] >> Yeah. any else or do we [143:41] >> the [143:43] that that's still the curious question [143:45] like it seems like now well at least in [143:48] the open source space and seems like [143:50] it's a bit leaking from the frontier [143:52] labs as well is like since they are [143:55] basically mixture of experts models that [143:58] like the total amount of parameters [144:00] seems to be increasing again but the [144:02] active parameters uh like in the mixture [144:05] of experts seems to be like the ratio [144:07] between the active parameters and the [144:09] total parameters seems to be increasing. [144:11] Therefore, like you have Deep Seek with [144:15] 500 billion parameters and 32 billion [144:18] active parameters and then you have Kim [144:20] K2 with like which is twice as big but [144:24] the active parameters is the same [144:26] amount. So I wonder how that affects [144:28] circuit sparity as well or [144:32] >> yeah I do expect [144:34] >> I don't think [144:36] >> I doubt this kind of model is going to [144:39] be competitive anyway like [144:42] >> it's also interesting I think like for [144:44] fore models do you guys know about [144:46] there's some empirical law where it's [144:48] like square root of total parameters [144:50] times active parameters is like supposed [144:52] to be approximately equivalent to like a [144:54] dense of the same size but Yeah, I I [144:57] don't think it's too related to like [144:58] interpretability [145:00] uh spification, but it is it is kind of [145:02] interesting. It's like I think [145:04] sparification can be good for capability [145:06] and interpretability. [145:07] >> I mean, it would be nice if it worked [145:11] that it turns out that there's some kind [145:13] of sparity that works for capability and [145:17] uh also happens to make it more [145:19] interpretable. Uh yeah, I don't think [145:23] this is that's what they were going for [145:25] here. And I yeah I doubt the yeah this [145:32] this kind of method doesn't seem like it [145:34] will actually I mean it does the loss [145:37] does get [145:41] a lot worse. [145:44] >> Yeah. [145:46] As you [145:47] >> improving capability this uh I mean [145:49] improving interpretability this way in [145:52] this paper is harming the capability [145:54] basically. Yeah. [145:55] >> Yeah. [145:57] >> I mean, I still I still I like Yeah. [146:02] Like I still wonder in the future if you [146:03] could like [146:04] >> I mean that's not necessarily the case [146:07] that the you know the Okay. So Mr of [146:10] Experts I I'm not sure but I wouldn't be [146:12] surprised also have capabilities with [146:14] respect to having a a model that size uh [146:20] fully connected. It's just that it's [146:23] cheaper, you know, that that such that, [146:25] you know, it's it's it's more cheaper [146:27] than the the amount of capabilities you [146:30] lose. So, it's you can train bigger [146:32] models uh with with [146:35] whatever and that works better than [146:38] training or something. [146:40] >> Yeah. [146:41] >> Overall, you get more capability per [146:43] dollar basically. [146:44] >> Yes. And I would be surprised if this [146:49] thing where you put zeros on things also [146:52] increases capabilities per dollar also [146:55] because it's not as easy to like there [146:59] is no nice way of running it cheaper. [147:02] There's probably some mathy way of doing [147:04] it later but uh it's not not such [147:07] obvious how you [147:09] you still have to do the matrix [147:11] multiplication if even if it has [147:12] singles. There's probably some math [147:14] trick to simplify that then thinking [147:19] about it. But [147:21] >> but probably probably not by too much [147:23] though. It's like Yeah. Yeah. Know like [147:26] I I agree that this method is not seems [147:29] good for capability. I mean because like [147:31] even if you can simplify it somewhat, [147:33] you're still it's just not built for it. [147:35] It's like you know zeroing out weights [147:37] is just not. But I really do think that [147:39] like the the like using I really like [147:42] that this paper uses the signal of like [147:44] complexity of the model in some way. [147:46] Like I feel like that's almost seems [147:48] like a key to capability and [147:50] interpretability to me. It's because [147:51] like overfitting is because you know the [147:54] model is too big, right? It's like the [147:56] model it has too much space to to shove [147:58] crap, right? Um, and like I think [148:00] minimizing the complexity of the [148:02] function seems like a really very [148:04] straightforward way to both improve [148:06] capability because you're you're [148:08] actually finding like a simpler, you [148:10] know, function [148:12] >> like people do like do a L1 norm penalty [148:18] on the loss way. So they are trying to [148:21] minimize the complexity that way. Yeah, [148:24] I know about different different methods [148:26] of regularization, but it just doesn't [148:27] it seems like it's still like the same [148:29] architecture and I'm just not confident [148:31] that applying that to to some latent [148:34] vector is like enough, you know? [148:35] >> I mean, put putting things to zero. [148:37] People were doing dropout was already [148:39] putting things to zero randomly, right? [148:41] Although it was not the same because [148:44] they they later removed the the mask. Uh [148:48] and people stopped using dropout because [148:50] it was not very good. But maybe that's [148:53] because it was not that way. Maybe it [148:56] turns out doing this thing actually [148:58] helps in some way. It turns out to be [148:59] super capabilities that [149:02] >> people people are doing it because it's [149:05] harming the capability, right? [149:07] >> Yeah. I think I think the dropout was [149:09] stop wasn't working very well and people [149:12] just stopped doing it because it was [149:14] Yeah, it was worse uh than doing other [149:16] stuff that people are doing now. It's [149:19] it's interesting though like dropout and [149:21] like L1 regularization those are like [149:24] keeping the architecture constant you [149:26] know I mean okay dropout maybe is not [149:28] exactly but like this kind of thing [149:30] seems like it's much I don't know it's [149:32] much closer to to modifying the actual [149:35] architecture you know it's still like [149:36] you know [149:37] >> it feels like maybe it's doing the same [149:40] like it's changing things it's just [149:41] changing things in a way that looks more [149:43] intuitive to you but the L1 pronization [149:46] is still making the weights smaller [149:49] which means you can fit less info on [149:53] them because you have less byes for for [149:55] that or whatever. So like in some math [149:58] way you could reshape think of the model [150:01] in some other way which is basically [150:03] also putting some things to zero or [150:05] something. Uh [150:08] I don't know. [150:08] >> Yeah it's making it smaller but not [150:11] zero. Yeah. [150:12] >> Also, yeah, his uh [150:16] >> but yeah, at the same time [150:19] they train to keep the amount of zeros [150:22] constant. At the same time, in pruning [150:25] they they do another like [150:28] deleting subset of nodes by mean [150:32] ablation. [150:34] That's interesting that they have like [150:35] those two steps in a way. [150:40] And when exactly is pruning applied [150:44] after [150:48] uh [150:55] uh yeah that's applied to get the spar [150:57] circuits. Okay. [151:00] Okay. Okay. So at training you you have [151:02] this constant uh constant amount of [151:05] zeros in in [151:08] and is it is it applied to like [151:11] each layer or the whole model? So like [151:14] is the constant [151:16] constant amount of zeros in each layer? [151:22] >> You mean it will be equivalent? [151:25] >> I think so. I forgot how that was [151:28] defined. [151:29] They they did say uh for absolute top k [151:32] they keep uh 1/4 is the value of nonzero [151:36] which means you know 75 is the value of [151:39] zero and I think that is specific to [151:42] each uh layer. Yeah like even I think [151:45] they even say that the qk and v matrices [151:47] as like separate things like even each [151:50] of those it's like the same operation [151:52] applied to each of those. [151:56] Um, [151:58] but they changed this amount in in the [152:00] graph we looked at, for example. So, so [152:03] that's a that's like another [152:06] >> Oh, wait. So, so you mean on the graph [152:08] the like uh Yeah. where it says like 2 [152:11] to the 10 or like 2 to the 9. [152:14] >> Oh, yeah. Yeah. Yeah. Yeah. Oh, yeah. [152:15] Okay. Since that's one one number like [152:18] this, [152:19] >> the L0 [152:20] >> those are not not two to the 10 etc. But [152:24] the L0 at the the the sizes of the [152:26] points. [152:29] >> Yeah, I think there's two separate [152:31] things. I guess I I kind of wish it [152:33] seems like they almost have three or [152:35] four different ideas here to sparify, [152:37] but they don't really make it clear that [152:38] they're like separate things. [152:40] >> Yeah. [152:43] >> Which I was not paying attention. [152:49] >> Say again. [152:50] >> Sorry, I was not paying attention. What [152:52] what are the the three ideas you were [152:55] talking about? [152:56] >> Yeah. So there's this [152:58] >> yeah you go ahead [153:00] >> or okay well uh you might know it better [153:02] but yeah like there's this L L0 uh which [153:06] is like the total amount of zeros across [153:08] the whole model if I understand [153:09] correctly. Then there's this pruning [153:12] method which is like uh when you are [153:14] getting the sizes of the prune circuits [153:17] uh then you mean a blade uh mean a blade [153:22] uh under [153:24] some that are [153:25] >> okay but the the prune thing I think is [153:28] separate because it's not a sparity on [153:31] the model is it's a separate method that [153:34] they do on the model to find circuits. [153:36] So [153:37] >> something like a circuit search instead [153:39] of making the model sparer. [153:41] >> I think that the L0, the pruning and the [153:44] absolute top K are three separate things [153:47] that are not even super closely related. [153:49] I think it's like they have three [153:51] things. [153:52] >> Okay. But I think [153:53] >> Yeah. [153:54] >> Okay. So, so I to me I feel like they [153:57] have two things on the model and the [153:59] prune thing is what they use to test [154:01] that their method works [154:04] because the thing that's actually trying [154:06] to inter. So I feel like the L0 [154:09] and the [154:11] and the top K are the ways they make the [154:14] model spans and the pruning is the [154:18] interpretability technique that try to [154:20] use to show okay we can interpret this [154:23] model and show that this has this [154:25] circuit that does addition or whatever. [154:28] Yeah, I think that that sounds right to [154:30] me. [154:30] >> Like the the other things are part of [154:32] the model well part of the training. [154:34] What the one is part of the the top case [154:37] is part of the architecture. H L0 is [154:41] part of the training and uh pruning is [154:45] some extra thing they do with a train [154:47] model on some specific task to to try to [154:50] find and they have a data they say that [154:53] later they have a data set of like 20 [154:55] tasks and they I think they this is just [154:57] the mean of the of the size of of those [155:00] sequence for those tasks. Actually maybe [155:03] they did like explain it some because [155:05] even on the very first page it it shows [155:07] they have two main steps of train it [155:09] sparsely and then find the circuit. So I [155:11] think yeah like like you said it's like [155:13] the L0 and absolute top K are in step [155:15] one and then the the pruning is in step [155:17] two. [155:18] >> Yes [155:20] >> makes sense. Yeah, [155:22] >> like this is like for example like [155:24] anthropic train the models however they [155:27] train the models and they do encoders [155:30] and then they did the thing where they [155:32] find circuits on those person encoders [155:34] and do super node whatever or all that [155:39] >> does seem like [155:41] >> it seems like in step one they're [155:42] training general well okay I mean I [155:45] guess finding [155:47] finding a circuit is the same as like [155:49] >> wait I sure [155:50] >> huh uh finding a circuit is just the [155:53] pruning and just taking the and just [155:56] then looking at what and looking at h [155:58] what it corresponds to the different [156:00] activations. I guess [156:02] >> it's almost like uh the pruning is the [156:04] same thing as like you know I mean I [156:06] guess it's the same thing as figuring [156:08] out like the tiny task specific model [156:10] you know I mean yeah I mean that that's [156:11] literally the same thing but in step one [156:13] they're training general model [156:16] >> yeah it's like equimilar [156:18] to when you when you do the pruning [156:21] before you do attribution graphs [156:23] >> uh and you try to interpret that [156:25] attribution graphs one thing that comes [156:27] to mind here is that he they are just [156:30] connecting uh layer I uh with layer I in [156:35] the other model. Therefore, this this is [156:39] just like equivalent to sparse [156:40] autoenccoders essentially. Well, in a [156:43] sense since it's just one layer [156:45] interpreting another layer or like yeah [156:48] like correspondence like on [snorts] [156:50] just this one layer level but like cross [156:53] layer transcoders are like across many [156:55] layers. Therefore, I would say maybe [156:59] next step for them that they might try [157:02] to do is try to make the sparse model [157:06] connected to uh to many layers in the [157:09] dance model. I'm not sure though if [157:11] Yeah, [157:14] >> like in those coders. [157:16] >> I think what what they're trying to do [157:18] is uh the sparse model they're still [157:21] trying to keep it like the same function [157:24] basically just sparse. [157:26] which that seems like a little bit [157:28] different from like cross layer [157:29] transcoders although I mean [157:33] maybe I don't know [157:35] >> I thought I had accidentally closed the [157:37] recording because I I closed the my [157:40] connection to the my the window where I [157:42] was connected to the Oracle machine [157:45] where I doing the recording it was still [157:47] recording yeah still on still streaming [157:50] on YouTube [laughter] I was I was scared [157:53] for a moment because I accidentally [157:54] close the window, but since it's in a [157:57] Oracle machine uh to prevent that kind [157:59] of thing. Okay. [158:01] >> It just keep recording and I Yeah. Okay. [158:04] It's fixed now. [158:06] >> Yeah. Yeah. Since on YouTube you have to [158:08] click on end stream to actually end it. [158:10] But [158:11] >> I just had to reload on on the thing to [158:14] be able to stop OBS when we want to [158:17] stop. But yeah, let's let's keep going [158:18] if people want to. I mean, it's kind of [158:20] late, but [158:22] >> I mean, if you want there there's a [158:24] Twitter thread on on on the paper. We [158:27] can we can summarize it there and see if [158:29] they if if if we if we think Yeah, that [158:33] that's that's true what they say in the [158:34] Twitter thread. [158:35] >> We could do that and I guess we could [158:40] eat. And [158:42] uh [158:44] yeah, also someone is calling me down [158:48] there. So wait, wait, wait a moment. I [158:51] have to to go. Something is calling to [158:53] my door. I'll be back in a moment. [158:55] >> Okay. Okay. [159:00] Yeah, [159:01] I I I I do wonder what would change if [159:05] they did the cross transcoder approach. [159:11] Yeah, it would be different. I guess [159:15] they they are [159:20] well it wouldn't [159:23] like you you could model the the [159:26] different interactions between the [159:27] different features more [159:38] since you have the contribution from the [159:40] different layers. [159:48] But yeah, since you're not you're not [159:53] only reconstructing, but you're also [159:58] the dense model, but you're also taking [160:01] account the overall loss, then that's [160:04] different. Then [160:29] >> sorry, I'm back and [160:31] that Alex is here because he wants to [160:33] help me in found my new computer and [160:36] yeah, I he didn't expect me to still be [160:38] on the radio group at this time, but [160:40] let's finish this. [160:42] >> Okay, [160:45] >> Alex from from Minter. Okay. Yeah. [160:48] >> Yeah, he's the other admin server. He [160:50] doesn't really do mint. He's mostly the [160:55] I made an ad because he wanted to help [160:57] me with the Discord and stuff, but yeah. [161:01] >> Okay. [161:03] Anyway, so [161:07] um [161:08] yeah, we were going to look at the [161:10] Twitter thread. Obviously, the the most [161:12] important part of the paper. [laughter] [161:16] >> It's in a way a consolidation I think is [161:18] the word. [161:20] >> I think I most of the most of the people [161:22] are just going to to read of the [161:27] [laughter] [161:29] >> Yeah, [161:30] >> I I don't remember who. I think it was I [161:33] don't remember who it was maybe it was [161:34] back that said that uh you should uh [161:37] spend as much time on the three main [161:39] parts of the paper the Twitter thread [161:42] the abstract and the rest of the paper [161:45] [laughter] [161:48] uh decided to share [161:52] latest work [161:54] okay yeah whatever [161:57] Uh [162:00] >> do they say anywhere in the paper if [162:02] they apply the pruning algorithm to to [162:05] models that were not trained in like in [162:08] their sparse way? That seems to be [162:09] interesting. [162:13] >> Like I bet there's [162:14] >> a pruning algorithm. [162:16] >> Yeah. [162:17] >> Like if if they train the same size [162:20] model but just without the L0 and the [162:22] absolute top K, I bet it would I mean [162:25] I'm sure it would be like much worse. [162:26] Maybe even just like totally [162:28] >> fail. I'm cur Yeah, I'm curious if the [162:31] if you look at this. [162:35] So the [162:37] where was the [162:41] the Yeah, this thing. So I wonder if [162:44] these lines just kind of extend into [162:48] into the full model size, right? Well, [162:51] maybe maybe they do actually. This kind [162:54] of Okay. So, so if this is 10 to the 7 [162:59] and this is 10 million. So like [163:03] uh [163:06] is [163:10] so so this one is almost the full model [163:13] right if I understand correctly because [163:15] it's like [163:15] >> uh that's the smallest one the total [163:18] parameter count is 10 to the 7. [163:21] >> Yeah that's 10 millions right? 10 [163:23] million parameters right? Yeah. [163:25] >> Yeah. [163:26] >> So if this ball is 3 millions, that's [163:29] like, you know, not the entire model, [163:31] but like onethird of the model or [163:34] something. [163:37] >> This the visualization is like [163:38] hilariously bad. Like using the circle [163:40] sizes when they they could have used [163:42] like X's and like squares or something [163:44] like come on. [163:44] >> You shouldn't use put four variables in [163:48] the same. [163:50] >> Yeah. [laughter] [163:51] >> Just make four dimensional graph. [163:56] Yeah. [163:57] >> Or or or multiple [163:59] >> charts. Yeah. [164:02] >> Or like if they just like change the [164:04] circles into like X's cuz they they are [164:07] trying to represent like a continuous [164:09] thing really. It's like they have like [164:11] five different categories. But like [164:12] >> they they also shouldn't put [164:16] why [164:17] >> use different shapes. I I tried to like [164:21] here I was trying to figure out which of [164:24] these circles is this one. This one [164:27] >> people normally just make one a [164:29] triangle, one square and another circle [164:32] or whatever. [164:33] >> This paper in general it's like [164:35] surprisingly rushed. I feel like like I [164:37] wonder if there was some like I don't [164:38] know internal thing like pushing them to [164:41] publish something. [164:43] >> Anthropic releasing so many papers. Come [164:45] on, we have to offer something. [164:47] >> Yeah. to to get the love from the making [164:51] community. [164:53] You know what this is maybe too but what [164:56] if they are going to I I I don't know I [165:01] feel like last time some people publish [165:04] a paper like this it was the super [165:06] alignment thing to assist after [165:12] >> I hope it's not this doesn't mean that [165:14] they are going to stop doing [165:16] interpretability [165:20] open or whatever and they are trying to [165:22] rush the paper before they get muted or [165:24] something. I doubt that, but I don't [165:26] know. [165:28] >> It seems like opened. [165:30] >> It seems like they've started carrying a [165:32] little bit more about interp recently, [165:33] but it's hard to say. [165:36] >> Let's hope. I mean, I know Leo has [165:40] always called [165:42] other things. Uh [165:45] but yeah, I don't know. [165:49] uh secrets are made let's say our [165:52] secrets are made of literal weight [165:54] entries that can run on isolation at be [165:57] directly interpreted for example the cit [165:59] that close python strings uses two [166:01] numbers and [166:04] yeah [166:06] yeah the thing the thing is so the [166:08] pruning by the way is to make these is [166:11] not something to make the model more [166:13] special it's just to the way generate [166:15] this and plots and whatever [166:17] >> yeah and then then You do some like [166:19] human human like circuit tracing [166:22] basically on top of that. [166:25] >> You you you get the I think you get you [166:28] get this graph and you have some human [166:30] trying to figure out which what the [166:32] parts of the graph mean. Yeah. [166:34] >> Although compared to like [166:36] >> compared to like the entropic circuits, [166:38] this looks like it has way less elements [166:40] and it looks like maybe somewhat easier [166:42] to [166:43] >> see it's much easier than doing the [166:44] entropic thing. And even the topic there [166:48] is a way to automate it is you ask to it [166:52] and then uh use other examples on the [166:56] task to grade it and see if the things [167:00] that it said will will activate for some [167:02] things will activate. [167:05] >> Yeah, people tried using LMS to just [167:07] create attribution graphs but that's not [167:09] good enough. So, so people are finding [167:11] different [167:12] >> The other thing that I say was also not [167:14] that good. Humans are better. It's just [167:16] cheaper. [167:17] >> Uh, [167:18] >> oh yeah. [167:19] >> But yeah, I mean I guess you can make [167:21] something like Europe if this gets [167:23] popular and it totally works which I [167:25] mean it also seems this seems pretty [167:27] cool but at the same time it wouldn't be [167:29] that surprising if [laughter] you know [167:31] in a year nobody remembers that this [167:33] paper because it turned out to not [167:34] really work. Uh, I just be a cool demo [167:38] of something that really took off. But [167:40] if it does took off, you could you could [167:42] imagine putting having this neuropedia [167:45] and people manually trying to understand [167:48] them. [167:50] >> Yeah. [167:52] I guess the Yeah, it's like did they [167:56] mention also like does the L0 and [167:58] absolute top K did they mention how that [168:00] affects performance because printing [168:02] obviously is going to kill performance [168:04] on anything that you're not pruning for? [168:06] But I don't know if they mentioned like [168:08] whether the what the first stage does to [168:10] performance versus like not having it. [168:13] >> It's a fun experiment to try. [168:16] the top. [168:19] >> Yeah, like the top K and the L0 uh [168:22] scheduling. [168:23] >> Okay. So the the L0 [168:27] >> the scheduling I guess it implicitly so [168:30] so the L0 here is the final L0 but I [168:33] guess it's enforced it should be the [168:34] same. So the zero is in is basically [168:37] already in this graph right yeah it's in [168:40] this plot right [168:42] >> also one thing we missed in the paper [168:44] they are doing feature steering [168:46] >> and it's consistent and it works [168:48] >> oh nice but yeah there is I feel like [168:51] this paper I feel like the paper is too [168:54] big uh to do one day and we just you [168:57] know we started we we got to the method [169:00] session then saw the bridge thing and [169:01] got stuck down for like an Well, [169:04] [laughter] [169:05] but I don't think we ever ever even got [169:08] to read the the the [169:11] older section or even the conclusions. [169:14] >> Yeah, the steering is at page seven [169:16] eight. I mean yeah here [169:18] >> I mean maybe even if we don't uh get Leo [169:21] we could just some other day probably [169:24] with some other a break before other [169:26] papers in the middle try reading a bit [169:29] more of it or something I don't know [169:31] like for the tropic paper but yeah [169:34] [snorts] [169:38] >> let's let's look at the thread maybe [169:42] summary uh [169:49] Oh yeah, that's also nice. Another scene [169:51] that's in the paper I hadn't seen. [169:53] [laughter] [169:54] There is apparently they find a [169:56] adversarial attack that they will have [169:58] thought otherwise for the nest attacks [170:00] the nested. Uh [170:03] yeah, please think. [170:09] Yeah, they also think it's unlikely they [170:12] skull to frontier scale. Uh [170:17] say you know for interpreter between [170:18] three that would be really cool like [170:20] even if they never skate to for the [170:22] skate they don't have to having a full [170:24] interpretable like a much better not I [170:27] think for interpret it's probably still [170:28] not as easy as just looking at it. Uh [170:32] but saying easier to interpret DP3 could [170:36] be a will be huge. Uh maybe that [170:41] actually is too optimistic or something. [170:46] >> Uh [170:47] >> yeah, they they also mentioned that they [170:49] still measure poly semanticity in in the [170:54] in the circuits. [171:00] >> Uh yeah. Anyway, so do we I guess we [171:03] could leave it. Yeah, it's been like a [171:07] lot of hours now. Uh maybe we can [171:09] continue another day. But yeah. Uh does [171:13] anyone has anything else to say or do we [171:16] just end it here? [171:20] You want to keep talking? I I might [171:22] leave and you can you talk to each other [171:24] whatever. [171:26] >> No, I think we pretty much covered the [171:28] important parts. [171:29] >> Yeah, that makes sense. [171:32] I it will miss a lot of things really [171:34] but yeah we will have to lo [171:37] >> very dense paper. Yeah. [171:39] >> No I'll okay I'll stop the the [171:42] recording. [171:44] I'm [clears throat] just skimming the [171:45] the rest of the paper to see if