Sparse Autoencoders Fail to Capture Concept Manifolds in Neural Models
Mech Interp Discord Server Recordings
Standard sparse autoencoders fragment smooth, non-linear concept manifolds into dense clusters of discrete linear features, causing severe feature aliasing and dictionary inflation across neural model activation spaces.
## Background
This video was published on YouTube on June 9, 2026, by the Mech Interp Discord Server Recordings channel. It features a community reading group session exploring the paper "Do Sparse Autoencoders Capture Concept Manifolds?" by Usha Bhalla, Thomas Fel, Can Rager, and collaborators. The description provides links to the arXiv preprint and invitations to the group's Discord community.
## TL;DR
The session investigates whether Sparse Autoencoders (SAEs) capture non-linear concept manifolds in neural network activation spaces or merely approximate them through linear tiling. Moving past the traditional linear representation hypothesis, the group analyzes how SAEs interact with curved geometries like circular concepts or modular arithmetic. Using theoretical tools such as the Ising model framework, the discussion covers failure modes like feature shattering and dilution, post-hoc topology recovery, and empirical validations across real-world language models. Ultimately, the participants examine how features co-activate and debate the parallels between artificial dictionary learning and population coding in biological neuroscience.
## Atmosphere & style
The tone is relaxed, highly collaborative, and intellectually candid, feeling like an informal late-night research seminar among passionate peers. Speakers freely interject with 'hot takes,' self-deprecating humor about getting side-tracked by math rabbit holes, and casual banter while navigating technical figures and complex proofs. The energy seamlessly shifts between high-level conceptual debates on causality and deep technical dives into equation definitions, offering an engaging, transparent look into how AI researchers digest dense papers in real time.
## Key takeaways
- [00:00](https://www.youtube.com/watch?v=1AX0tm3Gf50&t=0s) The linear representation hypothesis—which assumes concepts are stored as 1D linear directions in activation space—is increasingly being challenged by evidence that models represent concepts as low-dimensional non-linear manifolds.
- [02:24](https://www.youtube.com/watch?v=1AX0tm3Gf50&t=144s) When standard linear SAEs encounter curved manifolds, they approximate them by creating a swarm of discrete linear features that tile the surface, resulting in feature aliasing and inflated dictionary sizes.
- [05:51](https://www.youtube.com/watch?v=1AX0tm3Gf50&t=351s) Concepts with cyclic or continuous properties (such as days of the week, color spaces, or modular arithmetic) naturally form geometric shapes like circles, tori, or generalized helices in internal activation space.
- [14:05](https://www.youtube.com/watch?v=1AX0tm3Gf50&t=845s) A key distinction exists between the input data manifold hypothesis (raw input space) and the representation manifold hypothesis (how internal hidden activations organize learned concepts).
- [31:00](https://www.youtube.com/watch?v=1AX0tm3Gf50&t=1860s) Empirical experiments show that causally steering model activations smoothly along curved manifold paths produces predictable changes in model outputs, proving manifolds actively drive computation.
- [34:00](https://www.youtube.com/watch?v=1AX0tm3Gf50&t=2040s) Depending on feature capacity and sparsity, SAEs capture manifolds through three primary regimes: Compact Capture (matching intrinsic dimension), Shattered Tiling (over-tiling via localized linear segments), or Delution (overlapping, poorly tuned features).
- [01:03:41](https://www.youtube.com/watch?v=1AX0tm3Gf50&t=3821s) Even when an SAE shatters a continuous manifold into discrete linear atoms, the underlying global topology can be fully reconstructed post-hoc by analyzing pairwise feature co-activation correlations.
- [01:09:55](https://www.youtube.com/watch?v=1AX0tm3Gf50&t=4195s) The way SAE linear atoms tile continuous concept spaces closely mirrors population coding and tuning curves in biological neuroscience.
- [01:14:23](https://www.youtube.com/watch?v=1AX0tm3Gf50&t=4463s) High-frequency, pervasive features create severe distortion under strict L0/L1 sparsity constraints, causing autoencoders to smear global background features across domain-specific sub-features.
- [01:18:00](https://www.youtube.com/watch?v=1AX0tm3Gf50&t=4680s) The paper leverages an Ising model framework from statistical physics to factor out indirect correlations and superposition artifacts, cleanly isolating structured feature communities.
- [01:33:56](https://www.youtube.com/watch?v=1AX0tm3Gf50&t=5636s) Empirical evaluations on real language models reveal that state-of-the-art SAEs fall between compact capture and shattering, using fragmented local patches rather than optimal sub-space allocations.
- [01:51:16](https://www.youtube.com/watch?v=1AX0tm3Gf50&t=6676s) Setting the top-K sparsity parameter too low (K=1 or 2) forces individual feature atoms to be polysemantically shared across completely unrelated manifolds, exacerbating shattered representations.
- [01:59:45](https://www.youtube.com/watch?v=1AX0tm3Gf50&t=7185s) Many structural concept manifolds (such as circular representations for periodic sequences) appear invariant across different model architectures, supporting the Platonic representation hypothesis.
## Detailed walkthrough
### [00:00](https://www.youtube.com/watch?v=1AX0tm3Gf50&t=0s) Stream Setup and the Shift Beyond the Linear Representation Hypothesis
The session opens with informal housekeeping as the host gets the YouTube live stream up and running for participants who want to watch later or catch up asynchronously. After a brief pause to double-check audio and visual feeds, the host confirms that the stream is working properly and transitions straight into the core theme of the reading group. For years, the dominant paradigm in mechanistic interpretability has been the *linear representation hypothesis*—the belief that concepts and features inside neural network activation spaces are represented as one-dimensional linear directions. Standard techniques such as probing, activation steering, and standard Sparse Autoencoders (SAEs) all rely heavily on this linear assumption.
However, a growing body of recent research over the past few years—and particularly over the last year—has begun to challenge this foundational premise. Papers emerging from groups like Goodfire demonstrate that models often represent concepts not as single straight vectors, but as low-dimensional non-linear manifolds embedded within activation space.
- Traditional interpretability methods (standard probing, linear steering, baseline SAEs) rely explicitly on the linear representation hypothesis.
- Emerging research shows activation spaces frequently organize concepts into low-dimensional non-linear manifolds rather than pure linear directions.
[02:24](https://www.youtube.com/watch?v=1AX0tm3Gf50&t=144s) Goodfire's Manifold Steering and the SAE Tiling Problem
The host references prior work from Goodfire discussed in an earlier reading group session, where researchers demonstrated "manifold steering." In toy examples with known manifold structure, steering along the curved geometry of the manifold yielded superior control compared to traditional linear vector addition. The paper under discussion today ("Do Sparse Autoencoders Capture Concept Manifolds?") investigates how standard SAEs interact with these non-linear geometric structures—specifically asking whether SAEs capture manifolds globally (via compressed sub-spaces or spans) or locally by creating a swarm of discrete linear features that "tile" the continuous manifold surface.
A participant chimes in to emphasize the broader debate: while many researchers stubbornly believe everything is represented as a linear direction, this new line of work proves that activation structures can form far more complex shapes. The host notes that when a standard linear SAE encounters a continuous curve or manifold—such as a continuous sequence of numbers or a circular concept—it tends to approximate the structure by creating numerous linear features that tile different points along the curve. This results in severe feature aliasing and requires a massive dictionary of redundant features, whereas a true geometric representation could capture the entire manifold far more efficiently.
- Standard SAEs forced to approximate a non-linear manifold will tile the continuous surface using multiple discrete linear features.
- Tiling leads to feature aliasing and inflates dictionary size, replacing a single continuous curve with dozens of point-wise linear approximations.
- Goodfire researchers have teased upcoming work on "geometric SAEs" that incorporate manifold-aware feature primitives instead of pure linear vectors.
[05:51](https://www.youtube.com/watch?v=1AX0tm3Gf50&t=351s) Geometric Primitives and Formal Definitions of Manifolds
Discussion turns to the theoretical promise of building SAEs with geometric primitives rather than standard direction vectors. The host mentions asking Goodfire researchers on social media about this, confirming that manifold-native autoencoders are actively in development. A participant asks whether the paper provides a strict, formal mathematical distinction between a concept manifold and standard linear representations.
The host explains that while the machine learning community often uses the term "manifold" loosely, a manifold in this context refers to a topological space that looks locally Euclidean when zoomed in, even if its global topology is non-Euclidean (such as a torus, sphere, or closed curve). Linear directions are simply a one-dimensional special case of a manifold. If a model represents days of the week or modular addition, those representations naturally form closed circles or higher-dimensional geometric manifolds rather than infinite straight lines.
- A linear representation is technically just a 1D linear manifold; non-linear manifolds generalize this to curves, circles, tori, and higher-dimensional surfaces.
- In mechanistic interpretability, concepts with cyclic or continuous properties (e.g., days of the week, modular arithmetic) naturally form non-linear geometric shapes in activation space.
[08:45](https://www.youtube.com/watch?v=1AX0tm3Gf50&t=525s) Stream Adjustments and the Scope of the Paper
Interrupting the conceptual discussion briefly, the host realizes that the screen share on YouTube had not been capturing the paper document properly. After adjusting the window setup and zooming in on the PDF for stream viewers, the discussion resumes around the paper's core contributions. The host notes that while the paper sets up a framework for evaluating how SAEs handle manifolds, it serves primarily as a formal diagnostic paper rather than introducing a complete, universal unsupervised manifold autoencoder.
The group scrutinizes why the paper focuses heavily on formalizing how SAEs fail or tile manifolds rather than jumping straight to a replacement architecture. The host suspects there are subtle caveats and edge cases when trying to extract manifolds completely unsupervised across arbitrary real-world models.
- The paper acts as a diagnostic framework to formalize how linear SAEs tile and misrepresent concept manifolds.
- The authors show that SAEs can recover geometric structure through post-hoc analysis (such as clustering feature activations), even if the underlying SAE loss function encourages discrete linear tiling.
[14:05](https://www.youtube.com/watch?v=1AX0tm3Gf50&t=845s) The Manifold Hypothesis: Data Geometry vs. Learned Features
The conversation broadens into a foundational debate regarding the traditional "manifold hypothesis" in machine learning versus manifold representations inside model activations. A participant points out that the machine learning community has long assumed data lies on low-dimensional manifolds, and that topological deep learning specifically studies connectivity and metric spaces without assuming linear distance.
The host clarifies an important distinction that often causes confusion: classic machine learning papers discuss the manifold of the *input data distribution* (e.g., natural images occupying a tiny low-dimensional surface within raw pixel space), whereas mechanistic interpretability is specifically concerned with the geometry of *internal learned feature representations* inside the model's hidden layers. While related, a model operating on input data does not automatically guarantee that its internal hidden representations cleanly mirror that external geometry.
- Data manifold hypothesis: raw input data (e.g., image pixels) lies on a low-dimensional manifold in input space.
- Representation manifold hypothesis: internal hidden activations organize conceptual knowledge into low-dimensional geometric manifolds.
- Topological deep learning focuses on connectivity and topological properties rather than strict Euclidean distances or linear directions.
[18:25](https://www.youtube.com/watch?v=1AX0tm3Gf50&t=1105s) Does Gradient Descent Actually Learn Objective Algebraic Structures?
The participants debate whether neural networks truly internalize objective mathematical or geometric structures, or if they merely learn superficial approximations that happen to satisfy the objective function. One speaker notes that when training toy models on algebraic structures like mathematical groups, models often struggle with properties like associativity, showing that gradient descent offers no guarantee of discovering clean, exact algebraic manifolds.
The host highlights that while models frequently develop geometric representations (like circular representations for periodic sequence tasks) because they are computationally efficient for predicting target tokens, it remains an open question how cleanly these internal spaces align with real-world domain geometry. The discussion concludes that researchers must be extremely precise about which space they are analyzing—whether pixel space, token embedding space, or abstract latent activation space—before claiming a model has discovered a "true" concept manifold.
- Models trained on algebraic group operations often fail on specific properties like associativity, proving gradient descent does not automatically converge on perfect algebraic structures.
- Internal representations reflect whatever geometric layout minimizes loss, which may be a distorted or under-specified approximation of the true data domain structure.
<ElicitationsGroup message="To explore the concepts from this transcript further:">
<Elicitation label="Summarize the core differences between linear representation and concept manifolds" query="Summarize the key differences between the linear representation hypothesis and the concept manifold hypothesis as discussed in the reading group."/>
<Elicitation label="Explain how Sparse Autoencoders tile non-linear manifolds" query="Explain how standard Sparse Autoencoders tile non-linear concept manifolds and why this leads to feature aliasing."/>
</ElicitationsGroup>
[24:55](https://www.youtube.com/watch?v=1AX0tm3Gf50&t=1495s) Continuous Spaces, Discrete Tokens, and the Nature of Representations
The discussion opens with an ongoing debate regarding input spaces and whether concepts in LLMs exist in a continuous or discrete format. The speakers point out that in text-based transformers, the raw input space isn't continuous because text consists of discrete tokens; instead, models map these onto learned continuous embeddings. Some participants bring up the idea of a "platonic space of ideas," though it is quickly dismissed as an unhelpful rabbit hole so the group can pivot back to the core paper: *"Do Sparse Autoencoders Capture Concept Manifolds?"*
```
Continuous vs. Discrete Conceptual Space
┌───────────────────┐ Learned ┌───────────────────────────┐
│ Discrete Text │ ────────────> │ Continuous Embedding │
│ Tokens │ Embeddings │ Space / Representation │
└───────────────────┘ └───────────────────────────┘
```
The conversational dynamic is casual and candid, with speakers acknowledging their own tendency to digress before steering the session back on track. They transition into examining how Sparse Autoencoders (SAEs) reconstruct activations and where standard assumptions about feature representation fall short.
- **Learned Embeddings:** Continuous vector spaces constructed from discrete token inputs.
- **Platonic Representation Hypothesis:** Mentioned in passing as an abstract framing for universal concept alignment across models.
[26:00](https://www.youtube.com/watch?v=1AX0tm3Gf50&t=1560s) Manifolds and the Limits of Linear Feature Hypotheses
The group transitions to discussing the Ising model—a mathematical model of ferromagnetism from statistical mechanics—which the paper references in relation to concept detectors and feature representations. The conversation turns to how prevalent concept manifolds actually are in neural models, contrasting this with the popular "linear representation hypothesis" where features are assumed to be simple 1D directions in activation space.
One participant offers a "hot take": perhaps *everything* in internal representation space is actually a nonlinear manifold, and perfectly linear 1D direction features don't actually exist in real-world models. They argue that every linear direction is just an approximation with inherent "wiggle room."
> "Hot take is that everything is actually a manifold and there's no such a thing as a feature that's a perfect line... every line is broken."
The group debates whether discrete categorical concepts force models to make discontinuous jumps or whether smoothly steerable manifolds dominate representations. They suspect researchers bias their findings toward manifolds simply by testing on continuously varying domains (like color spaces), rather than evaluating truly random or discrete categories.
- **Ising Model:** Used in statistical mechanics to model ferromagnetism via discrete spin states, serving as an analogy for structured discrete/continuous concept interactions.
- **Linear Representation Hypothesis:** The assumption that concepts map directly to 1D linear directions in activation space.
- **Manifold Hypothesis:** The alternative view that concepts lie on multi-dimensional, curved geometries that cannot be captured by a single linear vector.
[31:00](https://www.youtube.com/watch?v=1AX0tm3Gf50&t=1860s) Causal Steering Along Concept Manifolds
The participants review the paper's experimental setup, where the authors prove that manifold representations aren't just decorative geometric artifacts—they actively drive model behavior. By identifying prototypical "centroids" on a manifold and interpolating smoothly between them, the authors demonstrate that steering activation vectors along these curved paths produces smooth, predictable changes in model output.
This triggers a meta-commentary on causality in mechanistic interpretability. The group references heated debates in machine learning forums over what constitutes "real causation," with some critics arguing that activation steering doesn't represent true causal mechanism. The speakers note how nicely these empirical manifold interventions mirror recent findings from other research groups working on non-linear concept representation.
- **Manifold Steering:** Adding vectors that move along non-linear concept surfaces between prototypical centroids.
- **Causal Evidence:** Smooth output changes under continuous intervention prove the model's internal computation relies on the manifold structure.
[34:00](https://www.youtube.com/watch?v=1AX0tm3Gf50&t=2040s) How SAEs Fail: Shattered Tiling, Delusion, and Compact Capture
The group explores how Sparse Autoencoders (SAEs) attempt—and frequently fail—to represent multidimensional, curved geometries. Because standard SAEs are trained to extract discrete linear directions, they struggle when forced to reconstruct a continuous $D$-dimensional manifold, leading to distinct failure modes:
```
SAE Representation Modes
┌───────────────────────┬────────────────────────────────────────────────────────┐
│ Mode │ Description │
├───────────────────────┼────────────────────────────────────────────────────────┤
│ Shattered Tiling │ The SAE uses a dense cluster of discrete, highly │
│ │ anti-correlated linear features to approximate a curved │
│ │ surface (over-tiling). │
├───────────────────────┼────────────────────────────────────────────────────────┤
│ Delusion │ Features activate across the manifold but have mixed, │
│ │ inconsistent selectivities (positively and negatively │
│ │ correlated). │
├───────────────────────┼────────────────────────────────────────────────────────┤
│ Compact Capture │ A small set of features aligns directly with the intrinsic│
│ │ dimensions/axes of the underlying manifold. │
└───────────────────────┴────────────────────────────────────────────────────────┘
```
The speakers tie this back to Anthropic's work on circuit attribution graphs and arithmetic representations. They point out how number representations often form a "generalized helix" in activation space (as shown by Max Tegmark's group). When an SAE tries to represent a helix using linear directions, it creates periodic features (like "number mod 10" or "number mod 8") to tile the manifold, creating complex trigonometric sub-circuits.
- **Generalized Helix:** The non-linear structure LLMs use to represent ordered numerical sequences and modular arithmetic.
- **Anthropic Attribution Graphs:** Computational graphs extracted via SAEs that often unintentionally capture periodic slices of higher-dimensional manifolds.
- **Base-10 Internal Reuse:** Mentioned via a paper (Goodfire) showing LLMs like Llama convert modular arithmetic (e.g., month math modulo 12) into base-10 representations internally before converting back, sacrificing precision to reuse base-10 circuitry.
[43:48](https://www.youtube.com/watch?v=1AX0tm3Gf50&t=2628s) Structural Analogies and Reconstructing the Manifold
The conversation pivots to the theoretical proof presented in the paper regarding SAE reconstruction error. The speakers attempt to parse the mapping between the Ising model from statistical mechanics and dictionary learning. In the Ising model, discrete magnetic dipole moments interact on a lattice—a framework historically foundational to Hopfield networks and Boltzmann machines.
The group works through the paper's central theorem: if an SAE achieves a low enough reconstruction error on activations that lie on a manifold, its dictionary must either over-tile the space or achieve "compact capture" where the sparsity regime directly aligns with the ambient dimension of the manifold.
The session concludes with the speakers acknowledging the mathematical density of the paper's formal definitions, agreeing to summarize the core intuition rather than getting bogged down in line-by-line proofs.
- **Ambient Dimension:** The dimensionality of the surrounding activation space in which a lower-dimensional concept manifold is embedded.
- **Reconstruction Error Bounds:** The mathematical threshold below which an SAE is forced to either faithfully map a manifold's dimensions or break it into an inefficient web of linear tiles.
<FollowUp label="Would you like to proceed with Part 3 of the transcript walkthrough?" query="Please provide the detailed walkthrough for Part 3 of the transcript."/>
Here is a detailed, faithful walkthrough covering Part 3 of the reading group session on *"Do Sparse Autoencoders Capture Concept Manifolds?"*.
---
[49:34](https://www.youtube.com/watch?v=1AX0tm3Gf50&t=2974s) Feature Alignment, Ambient Dimensions, and Paper Structure
The group opens this segment discussing how sparse autoencoder (SAE) latent features align with underlying data manifolds. The speakers consider how a manifold's structural dimensionality dictates the number of features or "atoms" needed to capture it. They note that when the number of SAE features allocated to a manifold exceeds its intrinsic or ambient dimension, the representations shift.
A meta-discussion unfolds regarding how to approach reading the paper during the live call. One participant highlights how refreshing it is that the authors explicitly detail their math notation right at the start, remarking that *"more papers should do that."* The group decides to prioritize getting the core conceptual gist and big picture first before diving into dense equations, allowing less math-focused attendees to follow along before getting into the technical weeds later.
- **Notation:** The paper explicitly defines its mathematical conventions early on to improve readability.
- **Reading Strategy:** The group agrees to cover conceptual findings first and reserve full step-by-step mathematical derivations for the end of the meeting.
---
[51:30](https://www.youtube.com/watch?v=1AX0tm3Gf50&t=3090s) Manifold Capture Regimes: Compact Capture vs. Shattering
The speakers dissect the paper's theoretical framework describing how SAEs represent continuous concept manifolds. When an SAE dictionary allocates features to a manifold, the relationship depends heavily on dictionary width, embedding dimension, and model linearity:
* **Compact Capture:** The ideal regime where the number of learned features matches the intrinsic dimension of the manifold ($D$). Each feature aligns directly with a primary axis or directional vector of the manifold structure.
* **Shattering:** Occurs when the features are linear and the manifold is curved, or when the dictionary provides more features than the manifold's intrinsic dimension. The model "shatters" the continuous manifold into discrete, nearly disjoint localized regions, where different atoms cover different sub-patches.
The participants debate whether linear features can inherently represent curved manifolds. While linear features can theoretically represent curves via continuous linear combinations, in practice—under sparsity constraints—models often resort to breaking the curve into localized piece-wise linear segments (shattering).
---
[53:53](https://www.youtube.com/watch?v=1AX0tm3Gf50&t=3233s) Deciphering Figure 3 and Feature Correlation Matrices
The discussion shifts to interpreting Figure 3 in the paper, which displays pairwise feature interactions and correlation matrices (such as Ising model interactions). The speakers work through what the matrix off-diagonals and zero-diagonal values represent.
They observe that negative correlations naturally emerge in cyclic or geometric concepts. For example, on a 2D circular manifold, increasing one coordinate direction ($D_1$) forces a decrease in another ($D_2$). The group notes that off-diagonal structure in these correlation matrices reflects how different SAE atoms interact when tiling a single underlying concept manifold.
- **Ising Pairings & Correlation:** Zero entries on the diagonal with high positive or negative off-diagonal values indicate structured inter-feature relationships.
- **Geometric Trade-offs:** Negative off-diagonal correlation arises naturally when features represent opposing directions along a constrained continuous shape.
---
[56:23](https://www.youtube.com/watch?v=1AX0tm3Gf50&t=3383s) Synthetic Validations and the Sparsity/Width Sweep
The group reviews the paper's synthetic validation experiments. To rigorously test when and how SAEs capture manifold structures, the authors evaluate SAE performance across systematically controlled synthetic benchmarks featuring synthetic geometries (e.g., circles, spheres, curves).
The speakers discuss how changing SAE hyper-parameters—specifically dictionary width and top-$k$ sparsity ($k$)—drives transitions between dictionary regimes:
Participants compare these findings to known SAE failure modes reported by Anthropic and other research labs, such as feature duplication and feature absorption.
---
[01:00:03](https://www.youtube.com/watch?v=1AX0tm3Gf50&t=3603s) Distinguishing Shattering from Feature Dilution
The call engages in a detailed comparison between **Shattered** and **Dilute** representation regimes:
* **Shattering:** The dictionary contains more atoms than the manifold dimension. The manifold is cleanly partitioned into a set of nearly disjoint, localized atoms. Each specific point on the manifold triggers a minimal, unique subset of localized features.
* **Dilution:** Occurs when many redundant, broadly tuned features activate simultaneously for any given point on the manifold. Individual atoms lose their specific, sharp tuning, leading to high overlap in feature support.
A participant asks for clarification on parameter notation ($k$), confirming that $k$ denotes the sparsity setting (e.g., in Top-$K$ SAEs), while dictionary width refers to the total number of SAE latent dimensions.
---
[01:03:41](https://www.youtube.com/watch?v=1AX0tm3Gf50&t=3821s) Post-Hoc Topology Recovery via Feature Correlations
A key takeaway highlighted by the speakers is that even when an SAE shatters or dilutes a manifold into discrete atoms, the global topology and geometry of the concept can still be fully recovered post-hoc.
By analyzing the coupling matrix or pairwise correlation matrix of the learned SAE features, researchers can apply clustering or graph-based recovery algorithms to reconstruct the continuous underlying shape (e.g., recovering a smooth 1D circle or 2D manifold from scattered point activations). The group connects this finding to previous mathematical work showing that smooth underlying spaces can be reconstructed from high-dimensional discrete correlations.
- **Block-Diagonal Structure:** Co-activating feature groups form block-diagonal structures matching ground-truth manifold partitions.
- **Activation vs. Feature Space:** Reconstructing manifold topology directly from raw model activations is computationally prohibitive, whereas computing pairwise feature correlations across discrete SAE latents is computationally feasible and clean.
---
[01:09:55](https://www.youtube.com/watch?v=1AX0tm3Gf50&t=4195s) Parallels to Neuroscience and Population Coding
The session transitions into a philosophical and technical comparison between mechanistic interpretability in artificial neural networks and biological neuroscience.
The paper points out that the "tiling" and "shattering" observed in SAE latents closely mirrors **population coding** in the brain, where individual biological neurons act as localized tuning curves with overlapping support to jointly encode continuous perceptual spaces (like spatial orientation or color).
The participants discuss how AI interpretability researchers frequently rediscover established principles from neuroscience and signal processing:
- **Neuroscience Overlap:** Concepts like sparse coding, population codes, and receptive field tiling were originally developed in biological brain modeling.
- **Cross-Disciplinary Connections:** Similar mathematical tools appear in EEG signal processing and sparse dictionary learning.
- **Shared Constraints:** Both biological brains and artificial networks face identical geometric constraints: representing complex high-dimensional data using combinations of bounded linear/non-linear processing units under efficiency constraints.
<ElicitationsGroup message="Explore related concepts from this transcript section:">
<Elicitation label="Mathematical breakdown of SAE manifold capture" query="Break down the mathematical definitions of Compact Capture, Shattering, and Dilution in Sparse Autoencoders as described in the paper 'Do Sparse Autoencoders Capture Concept Manifolds?'"/>
<Elicitation label="Compare SAE feature shattering vs. neuroscience population codes" query="How does SAE feature shattering relate to population coding and tuning curves in biological neuroscience?"/>
</ElicitationsGroup>
Here is a detailed, faithful walkthrough of this part of the transcript:
[01:14:23](https://www.youtube.com/watch?v=1AX0tm3Gf50&t=4463s) Co-activation Mechanics and the High-Frequency Feature Problem
The discussion opens with an examination of how sparse autoencoder (SAE) "atoms" or features co-activate when capturing concept manifolds. When two atoms sample similar positions along a continuous manifold, moving through input space causes them to fire together or in a smooth sequence. One speaker notes that when moving along a line on a manifold, activation in one atom naturally makes activation in adjacent atoms more likely. However, the group distinguishes between structural co-activation (where atoms span the same underlying geometric manifold) and raw correlational co-occurrence. In correlational co-occurrence, features point in entirely different directions in activation space, but the base model happens to activate one feature whenever another is present.
This leads directly into a critique of how standard SAEs handle high-frequency, "universally firing" features. A speaker explains that enforceability of $L_0$ or $L_1$ sparsity creates severe issues when dealing with concepts that appear across almost all inputs:
* **Sparsity penalties vs. pervasive concepts:** If a feature (such as "English text") is active 90%+ of the time, forcing top-$k$ sparsity (e.g., $k=4$) consumes a valuable active slot on every single token.
* **Feature absorption/smearing:** Because sparse autoencoders penalize non-zero activations, the optimization process may refuse to dedicate a single persistent feature to high-frequency concepts. Instead, it smears those activations into all other correlated sub-features.
* **Prior literature connection:** The participants recall previous mechanistic interpretability reading sessions that specifically addressed high-frequency feature distortion in sparse autoencoders.
> "If you are trying to do sparsity, you'll have problems with features that are always there... if it has top-4, that's already one of four features that it has to remove."
---
[01:18:00](https://www.youtube.com/watch?v=1AX0tm3Gf50&t=4680s) Causality Critiques and the Ising Model Framework
The group pivots to discuss broader critiques of machine learning and interpretability, particularly arguments raised by causal abstraction researchers. While some causal purists dismiss standard ML as merely learning spurious correlations, the participants argue that deep models often develop internal representations that reflect true causal structure via simple inductive biases (drawing parallels to Solomonoff induction).
To address spurious correlations and structural dependencies rigorously, the paper under discussion introduces a formulation based on the **Ising model**—a mathematical model from statistical physics used to study ferromagnetism and spin interactions:
* **Spin alignment analogy:** In the Ising model, individual spins (+1 or -1) interact with neighboring spins, seeking local or global energy minima through positive or negative couplings.
* **Factoring out spurious correlations:** Unlike a basic correlation matrix, the Ising formulation factors out indirect dependencies and superposition artifacts by construction.
* **Decoupling high-frequency features:** Universally firing features end up with large bias/field parameters but weak pairwise couplings, allowing the method to separate global background noise from true manifold geometry.
A participant shares a helpful concrete scenario posted in the chat by a user named Usagi to illustrate why simple correlation fails where conditional independence models succeed:
* **Weather Example:** "Cold weather," "winter," and "drinking hot chocolate" all co-occur frequently in data. A naive correlation matrix shows strong pairwise links between all three.
* **Conditional Independence:** Once you condition on "winter" or "cold weather," the direct interaction between arbitrary sub-features drops, isolating the core underlying driver.
> "If you cast it in an Ising framework, then by construction you get some mathematical results from that field... indirect correlations and superposition are factored out."
---
[01:26:13](https://www.youtube.com/watch?v=1AX0tm3Gf50&t=5173s) Tiling Regimes and Unsupervised Manifold Identification
Moving deeper into the mechanics of the paper, the group analyzes how the interaction parameters (positive and negative couplings) distinguish between different feature organization regimes:
* **Cooperative Tiling Regime:** Positive couplings indicate atoms that smoothly tile a shared continuous manifold, firing cooperatively as inputs transition across the space.
* **Exclusion/Shattering Regime:** Negative couplings highlight mutual exclusion, where features actively suppress one another or partition space into disjoint fragments.
* **Community Detection:** By analyzing the interaction matrix, concept manifolds can be recovered in an unsupervised manner as "communities of atoms" exhibiting strong pairwise interactions, regardless of whether those interactions are cooperative or competitive.
The speakers reflect on how this physics-inspired framework allows researchers to extract structured concept manifolds directly from activation co-occurrences without having to train specialized autoencoders for every candidate manifold geometry.
---
[01:33:56](https://www.youtube.com/watch?v=1AX0tm3Gf50&t=5636s) Empirically Testing Real Model Architectures: Capture vs. Shattering
The session transitions from theoretical toy models to empirical tests run on full-scale language models. The paper authors tested five primary sparse autoencoder architectures (including standard ReLU SAEs, Top-$K$ SAEs, and Matryoshka SAEs) across varying feature dictionary sizes and sparsity parameters.
They evaluated these architectures against known continuous concept manifolds (such as synthetic color spaces, HDR temperature sweeps, and style vector transitions in generative models) to test whether SAEs cleanly capture continuous manifolds or shatter them into fragmented pieces.
* **The Main Finding:** Across almost all tested architectures and dictionary sizes, current SAEs fall squarely between clean capture and complete shattering.
* **Patchwork Reconstruction:** Instead of learning smooth, low-dimensional coordinate representations of a manifold, SAE features form local "patches." Reconstructing a point on the manifold requires combining $K$ distinct local features, which only partially cover the global geometry.
* **Failure of Compact Allocation:** Standard SAEs fail to allocate compact, dedicated feature sub-spaces to continuous manifolds. Increasing the dictionary expansion factor often increases feature fragmentation rather than improving manifold fidelity.
> "The results place current autoencoders in between capture and shattering... local patches of the manifold assemble to cover the geometry, but they do not allocate compact representations."
<ElicitationsGroup message="Explore related concepts in sparse autoencoders and feature geometry:">
<Elicitation label="Explain how Top-K and Matryoshka SAEs differ in feature allocation" query="Explain the structural differences between standard ReLU, Top-K, and Matryoshka sparse autoencoders and how they impact feature allocation."/>
<Elicitation label="Deep dive into the Ising model formulation for feature co-activation" query="Explain the mathematical formulation of the Ising model used to disentangle feature co-activations in sparse autoencoders."/>
</ElicitationsGroup>
[01:38:05](https://www.youtube.com/watch?v=1AX0tm3Gf50&t=5885s) Evaluating Manifold Reconstruction and Geometry Recovery
The discussion begins with participants closely analyzing how Sparse Autoencoders (SAEs) capture specific concept manifolds rather than averaging across all representations. They note that the paper under discussion—*"Do Sparse Autoencoders Capture Concept Manifolds?"*—picks specific 24-dimensional manifolds to check rather than computing an average across the entire activation space.
The speakers transition into discussing the geometry of these representations and how methods like the Ising model interaction coupling can recover pre-known manifolds. They note that the pipeline introduces a framework for analyzing novel manifold encodings, including epistemic uncertainty in scientific contexts. However, a participant clarifies that the paper's approach relies on using identified meaningful features to reconstruct manifolds rather than uncovering manifolds directly from raw model activations in a fully unsupervised manner.
- The authors specifically analyze targeted 24-dimensional manifolds rather than taking global averages across all activation spaces.
- Applying Ising model coupling techniques to SAE features recovers ground-truth manifold topologies previously known to exist.
- A distinction is made between discovering manifolds directly from raw activations versus reconstructing them post-hoc from extracted SAE feature representations.
[01:42:11](https://www.youtube.com/watch?v=1AX0tm3Gf50&t=6131s) Feature Co-activation Metrics and Spatial Attraction Models
The conversation turns to how features jointly represent a manifold space. The participants debate whether cosine similarity or co-activation patterns offer a better metric for determining if two features belong to the same concept manifold. Under the linear model of SAEs, feature atoms spanning manifold spaces may have near-orthogonal decoder directions, meaning raw decoder geometry carries no privileged information about the underlying non-linear topology.
```
Attraction (Co-activation)
+-------------------+
| |
v v
[ Feature Atom A ] <-----> [ Feature Atom B ]
(Repulsion / Separation across distinct manifolds)
```
The speakers draw parallels to physical models of manifold geometry where feature interactions involve principles of attraction and repulsion:
- Features that jointly represent a localized manifold region tend to fire together (co-activate).
- Features covering distant or distinct conceptual spaces remain separated or anti-correlated.
One speaker highlights potential edge cases where concepts might be anti-correlated depending on dataset sampling. For example, sampling exclusively within group theory texts creates strong correlations among abstract algebra concepts, whereas broader sampling across set theory might show neutral or anti-correlated patterns with specific algebraic structures.
- Raw decoder vector directions often fail to reveal manifold structure because decoder atoms are constrained to be nearly orthogonal.
- Five distinct similarity metrics were evaluated in the paper, with Ising interactions/couplings and conditional co-activation yielding the highest correlation with true manifold geometry.
- Data sampling distributions heavily influence feature co-activation statistics and observed manifold correlations.
[01:47:45](https://www.youtube.com/watch?v=1AX0tm3Gf50&t=6465s) Unsupervised Manifold Discovery and System Bottlenecks
The group addresses the final conclusions of the paper, questioning why the authors demonstrated their Ising-based discovery pipeline on selected examples rather than automatically extracting hundreds of manifolds across entire models. They debate the current operational bottlenecks of unsupervised manifold discovery.
While the feature clustering aspect can be run automatically over SAE dictionaries, labeling and verifying that the resulting clusters correspond to coherent human-interpretable manifolds still requires manual oversight. The participants express interest in taking open-source SAE implementations and running the Ising coupling pipeline across full model layers to check how many novel, uncurated manifolds emerge automatically.
- The primary bottleneck in scaling manifold discovery isn't the clustering calculation itself, but the manual verification and labeling of extracted feature clusters.
- Automatically clustering co-activating SAE features could potentially uncover hundreds of latent geometric structures without prior hypotheses.
[01:51:16](https://www.youtube.com/watch?v=1AX0tm3Gf50&t=6676s) Dissecting Sparsity Regimes and Atom Sharing ($K$ Parameter)
Prompted by a request to review figures in the paper, the group analyzes how varying the sparsity parameter ($K$ in top-$K$ SAEs or $\text{L}_0$ norm) affects manifold representation. The discussion focuses on why manifolds become "shattered" or shared across unrelated concept groups when $K$ is set too low.
```
Low Sparsity (Low K: 1-2) Optimal Sparsity (Intermediate K: 4+)
+------------------------------+ +-------------------------------------+
| Few features available. | | Compact set of dedicated feature |
| Single atom forced to span | | atoms jointly span each individual |
| multiple unrelated manifolds | | continuous manifold. |
| ("Shattering" / Polysemantic)| | |
+------------------------------+ +-------------------------------------+
```
A speaker clarifies the distinction between an **SAE feature** (the general latent concept) and an **atom** (the specific directional vector in the decoder weight matrix). At extremely low values of $K$ (such as $K = 1$ or $K = 2$), the SAE lacks sufficient capacity to allocate distinct feature combinations to every distinct manifold. Consequently:
1. Individual atoms become broadly shared across completely unrelated conceptual domains (an effect akin to superposition).
2. The continuous geometry of a single manifold gets shattered across disparate feature groups.
3. At intermediate $K$ values (e.g., $K \ge 4$), a compact, dedicated set of feature atoms clean-spans each continuous manifold.
- At very low $K$ (high sparsity), the number of active latent slots is smaller than the ambient dimension required to represent complex concept manifolds.
- An "atom" specifically denotes the normalized direction vector in the SAE decoder matrix corresponding to a given feature.
- Intermediate sparsity levels allow SAEs to form dedicated basis sets that smoothly tile non-linear manifolds.
[01:56:55](https://www.youtube.com/watch?v=1AX0tm3Gf50&t=7015s) Non-Continuous Manifolds and Geometric Abstractions
The conversation shifts to broader theoretical perspectives on representational geometry in neural networks. Participants explore whether future interpretability work should move beyond smooth, continuous manifolds toward non-continuous, disconnected, or non-linear geometric structures.
They discuss how neural networks compress knowledge into "a set of potentially disconnected, non-linearly curved geometric structures." One speaker points out that while global concepts may be highly complex and non-linear, networks often represent them as locally linear sub-spaces or "rays" emanating through activation space.
- Conceptual spaces in LLMs are frequently formalized as non-linearly curved, potentially disconnected geometric manifolds.
- Local linearity remains a primary inductive bias, causing neural representations to approximate curves and complex topologies using linear rays or low-dimensional slices.
[01:59:45](https://www.youtube.com/watch?v=1AX0tm3Gf50&t=7185s) Cross-Model Platonic Representations and Neuroscience Connections
In the final segment, the speakers examine the paper's findings regarding cross-model consistency ("Platonic representation hypothesis") and look for connections to systems neuroscience. They discuss whether concept manifolds—such as the circular geometry representing the months of the year—are invariant across different model architectures and training runs.
```
[ Model A (e.g., Llama) ] [ Model B (e.g., Gemma) ]
\ /
v v
+-----------------------------------+
| Universal Platonic Manifold |
| (e.g., Circular Month Topology) |
+-----------------------------------+
```
The discussion touches on whether techniques from neuroscience (such as neural population manifold analysis in biological brains) could be directly imported into mechanistic interpretability. The session wraps up by examining how variance within a manifold is explained by identifying key directional sub-components within the SAE feature space.
- Standard concept topologies (such as circular representations of cyclic concepts like months or days) tend to recur consistently across diverse model architectures.
- The group highlights an untapped opportunity for greater cross-pollination between biological neuroscience manifold discovery techniques and artificial neural network interpretability.
- Variance within an extracted concept manifold can be decomposed into specific linear sub-directions captured by the autoencoder.
[02:05:50](https://www.youtube.com/watch?v=1AX0tm3Gf50&t=7550s) Evaluating SAE Metrics and Feature Recovery
The discussion begins with a brief technical observation regarding how metrics evaluate Sparse Autoencoders (SAEs) and feature discovery. The speakers note that even if multiple features exist that perform different functions, the metric will still rate an SAE highly as long as it recovers the core correlated features responsible for the main task behavior.
- The metric evaluates performance based on whether key correlated features are captured, even if additional redundant or auxiliary features exist in the representation.
[02:07:09](https://www.youtube.com/watch?v=1AX0tm3Gf50&t=7629s) Wrapping Up the Main Paper Presentation
With the primary paper presentation concluded, the hosts decide it is a natural stopping point for the official reading group session. They reflect on whether to keep recording while diving deeper into the technical appendices and math derivations, or to bring the public portion of the event to a close.
One speaker mentions that they might record an additional deep-dive video later if they thoroughly work through the paper's mathematical details, though upcoming workload and immediate project deadlines might prevent a fully comprehensive review. The host encourages them to record any extra analysis whenever time permits so it can be uploaded to the official YouTube channel.
- The primary presentation of "Do Sparse Autoencoders Capture Concept Manifolds?" is officially concluded.
- Additional deep-dive discussions on the appendix math may be recorded separately and uploaded to the YouTube channel.
- Immediate priorities shift toward practical application work due to impending deadlines.
[02:08:31](https://www.youtube.com/watch?v=1AX0tm3Gf50&t=7711s) Reading Group Duration and Final Remarks
As the session wraps up, the speakers joke about the event's duration. Hitting around the two-hour mark, they remark that this duration is actually "lower than the average" for their reading groups, noting that long-format discussions are common across other AI research communities such as the EleutherAI Discord server.
One participant lightheartedly admits to getting distracted while trying to teach someone on Machine Talking how to read mathematical equations in papers instead of focusing entirely on the paper at hand, laughing about being "so sniped constantly" and joking that someone needs to "do mech interp a bit on my psychology" to figure out why they get drawn into technical rabbit holes. Finding no further questions or comments from the audience, the host thanks the attendees and stops the recording.
- The session clocked in right around two hours, which the hosts note is shorter than their typical reading group average.
- "I'm basically right now trying to teach someone... how to read math in papers. I need to stop doing that and actually read the math in this paper instead. Why am I so sniped constantly?"
- "I have to do mech interp a bit on my psychology..."
## Notable quotes
> [26:00](https://www.youtube.com/watch?v=1AX0tm3Gf50&t=1560s) Hot take is that everything is actually a manifold and there's no such a thing as a feature that's a perfect line... every line is broken.
> [49:34](https://www.youtube.com/watch?v=1AX0tm3Gf50&t=2974s) more papers should do that
> [02:24:23](https://www.youtube.com/watch?v=1AX0tm3Gf50&t=8663s) If you are trying to do sparsity, you'll have problems with features that are always there... if it has top-4, that's already one of four features that it has to remove.
> [02:38:00](https://www.youtube.com/watch?v=1AX0tm3Gf50&t=9480s) If you cast it in an Ising framework, then by construction you get some mathematical results from that field... indirect correlations and superposition are factored out.
> [01:33:56](https://www.youtube.com/watch?v=1AX0tm3Gf50&t=5636s) The results place current autoencoders in between capture and shattering... local patches of the manifold assemble to cover the geometry, but they do not allocate compact representations.
> [02:08:31](https://www.youtube.com/watch?v=1AX0tm3Gf50&t=7711s) I'm basically right now trying to teach someone... how to read math in papers. I need to stop doing that and actually read the math in this paper instead. Why am I so sniped constantly?
> [02:08:31](https://www.youtube.com/watch?v=1AX0tm3Gf50&t=7711s) I have to do mech interp a bit on my psychology...
## Mentioned
Do Sparse Autoencoders Capture Concept Manifolds?, Usha Bhalla, Thomas Fel, Can Rager, Sheridan Feucht, Tal Haklay, Daniel Wurgaft, Siddharth Boppana, Matthew Kowal, Vasudev Shyam, Jack Merullo, Atticus Geiger, Ekdeep Singh Lubana, Goodfire, Anthropic, Max Tegmark, Llama, Gemma, Sparse Autoencoders (SAEs), Ising model, Hopfield networks, Boltzmann machines, Top-K SAEs, Matryoshka SAEs, ReLU SAEs, Linear Representation Hypothesis, Platonic Representation Hypothesis, Solomonoff induction, EleutherAI, YouTube, Discord, Mech Interp Discord Server Recordings
Participants in the Mech Interp Reading Group examine the paper 'Do Sparse Autoencoders Capture Concept Manifolds?', which challenges the linear representation hypothesis by demonstrating that neural networks often represent concepts as low-dimensional non-linear manifolds.
Challenging the Linear Representation Paradigm (0:00)
Traditional mechanistic interpretability relies heavily on the linear representation hypothesis, which assumes concepts and features occupy one-dimensional linear directions in activation space. However, recent findings cited by the group—including studies from Goodfire—indicate that models organize conceptual information, such as periodic sequences or cyclic concepts like days of the week, into curved, non-linear manifolds. In tests with known manifold geometries, steering activations along curved paths rather than straight linear vectors yielded smoother, causal control over model outputs.
SAE Tiling, Shattering, and Failure Modes (2:24)
Standard Sparse Autoencoders (SAEs) struggle to reconstruct non-linear geometric structures because their loss functions encourage discrete, linear feature representations. When forced to represent a continuous manifold, an SAE typically breaks the geometry into a dense patchwork of discrete linear features—a failure mode termed 'shattered tiling' or over-tiling. This process inflates dictionary sizes and causes feature aliasing, replacing a single continuous manifold with dozens of disjoint linear approximations.
The group outlined three primary representation regimes observed across varying dictionary widths and sparsity levels: compact capture, shattered tiling, and dilution. Compact capture occurs when feature counts match the intrinsic dimension of the manifold, while dilution features broadly tuned, overlapping activations. In empirical evaluations across five SAE architectures—including standard ReLU, Top-K, and Matryoshka models—representations consistently fell between compact capture and complete shattering, forming local patches rather than clean coordinate systems.
Extracting Manifold Topology via Ising Couplings (78:00)
Although linear SAEs fail to natively capture manifolds as single primitives, the underlying global topology remains recoverable post-hoc. The paper adapts the Ising model from statistical physics to analyze pairwise interactions across learned SAE features. Unlike standard Pearson correlation matrices, which conflate structural relationships with high-frequency background features or co-occurrence noise, the Ising coupling framework conditions on shared activations to isolate direct geometric couplings.
Evaluating the resulting coupling parameters separates cooperative tiling (atoms smoothly spanning a continuous space) from mutual exclusion (features partitioning disjoint fragments). Applying community detection algorithms to these interaction matrices allows researchers to reconstruct pre-known ground-truth manifolds—such as 24-dimensional test manifolds—without needing to discover them directly from high-dimensional, raw activation spaces.
Sparsity Dynamics and Representational Analogies (69:55)
The behavior of SAE features depends critically on the sparsity setting, such as the top-k parameter. The group noted that at very low sparsity levels (such as k=1 or k=2), the SAE lacks sufficient active slots to cover multi-dimensional geometries, forcing individual decoder atoms to be shared across unrelated concepts. At intermediate levels (k >= 4), the SAE can form dedicated basis sets that adequately tile the manifold.
Participants observed striking parallels between SAE feature tiling and population coding in biological neuroscience, where neural populations use overlapping tuning curves to jointly encode continuous perceptual variables. Additionally, representations of recurring concepts, such as circular month geometries, appear consistently across disparate architectures, supporting the idea of cross-model Platonic representations.
While the diagnostic framework demonstrates that concept manifolds can be recovered post-hoc through feature co-activation analysis, fully unsupervised, end-to-end manifold discovery remains limited. The primary practical bottleneck is not computational clustering, but the human verification required to interpret discovered structures, leaving the development of native 'geometric SAEs' as an active open challenge.
[0:00] 主持人: 稍等一下……按理说应该在录制了……好吧，现在没问题了。
[3:45] 主持人: 如果不使用连续的几何流形表征，SAE 就会用许许多多不同的方向去“平铺”这条曲线，将连续的几何结构任意别名化（Aliasing）为大量离散的特征。
[29:48] 也许模型中的一切表征本质上都是流形，根本不存在绝对完美的“直线特征”（Line Feature），每一条表示线在某种程度上都是“弯曲或破碎”的。
[49:34] 如果一个流形只有 3 个维度，理想情况是模型刚好拥有 3 个特征来完整表达它；但如果你给予它过多的特征，就会出现粉碎（Shattering）或稀释（Delusion）的情况。
[63:41] 这就像我们在之前的数学论文里看到的那样，即使特征被打碎成了很多份，只要拿到特征之间的相关性矩阵，你依然能够像拼图一样把原来的几何形状还原出来！
[69:24] 这其实证明了机理可解释性（Mech Interp）很多时候是在重新发现神经科学里早就成熟的概念！
[74:23] 像“英语文本（English）”这种在绝大多数数据中都存在的特征，SAE 的学习机制极难处理。为了维持 Top-K 稀疏性，模型可能会将“英语”这一高频激活强行挤压分布到所有相关特征中，而非将其单独抽离为独立特征。
[93:00] 种高相关性仅仅是外部环境（如季节）混杂所致，Ising 模型能够将无直接耦合的特征剥离开来。
[100:08] 论文并非直接从原始激活值（Activations）中以纯无监督的方式“凭空”生成流形，而是先提取出有意义的 SAE 特征，再基于这些特征去重建与发现底层的几何流形。
[115:07] 当稀疏度极高、允许激活的特征极少时，SAE 缺乏足够的自由度去单独表征每一个独立的流形。为了降低重构误差，模型不得不强行使用极少数的特征方向去拼接和近似多个完全无关的流形。
[125:50] 主讲人: I'm teaching someone how to read math instead of reading the math in this paper up
[127:09] 与会成员: because deadlines
[128:31] 与会成员: lower than the average actually
[128:31] 主讲人: that is a rabbit hole
Source quotes
[26:00] Hot take is that everything is actually a manifold and there's no such a thing as a feature that's a perfect line... every line is broken.
[49:34] more papers should do that
[144:23] If you are trying to do sparsity, you'll have problems with features that are always there... if it has top-4, that's already one of four features that it has to remove.
[158:00] If you cast it in an Ising framework, then by construction you get some mathematical results from that field... indirect correlations and superposition are factored out.
[93:56] The results place current autoencoders in between capture and shattering... local patches of the manifold assemble to cover the geometry, but they do not allocate compact representations.
[128:31] I'm basically right now trying to teach someone... how to read math in papers. I need to stop doing that and actually read the math in this paper instead. Why am I so sniped constantly?
[128:31] I have to do mech interp a bit on my psychology...
Original transcript
[0:00] I'm going to be
[0:02] streaming to YouTube, by the way.
[0:05] Uh,
[0:10] in case people want to watch later, uh,
[0:14] let's
[0:26] Okay. So uh yeah everything is working.
[0:31] I think the streaming should be working.
[0:39] Uh
[0:47] no wait it's not uh a moment.
[0:55] Uh, of course
[1:05] it should be I mean still in case it's
[1:09] recording but
[1:13] okay now it's working. Never mind.
[1:21] A lot of previous work was assuming like
[1:26] the linear representation hypothesis
[1:29] where like some uh like things are
[1:33] represented as directions in some space
[1:37] in the activation space. Uh and you know
[1:39] things like sparse encoders make this
[1:41] assumption and uh also for things like
[1:45] probing and the usual methods where
[1:47] people do steering work. But uh good
[1:51] fire has the and there is some other
[1:54] papers that kind of point on this
[1:55] direction. uh in recent three years uh
[2:00] well more last year uh that
[2:05] actually trans like models represent
[2:09] uh things from at least sometimes uh as
[2:15] vectors as as manifolds low dimensional
[2:19] manifolds in activation space.
[2:23] um
[2:24] like there are some paper finding C. So
[2:28] like there previous uh the previous
[2:31] paper we read on the reading group about
[2:33] this from them uh was like showing
[2:36] manifold steering where they have some
[2:38] toy examples where they knew there were
[2:41] things that were presented as uh as uh
[2:46] manifolds and they found the manifolds
[2:49] and they did the steering along the
[2:51] manifold instead of linearly and that
[2:53] work better in some ways
[2:56] and yeah this This is a new one. I
[2:59] haven't actually read it yet, but we
[3:01] will go through it. Uh
[3:05] uh yeah, I go through it uh also for the
[3:08] people who haven't read it. And uh if
[3:11] you have and have some takes, you can
[3:13] also talk whenever you want. Uh
[3:18] >> some people that believe everything is
[3:20] represented as a linear direction, but
[3:23] these people are trying to show actually
[3:25] it's more complex. that are complex
[3:27] shapes.
[3:28] >> Yeah. Also, you don't need to talk on
[3:30] chat. You can also write on the chat if
[3:32] you don't want to talk. Uh
[3:35] >> I'll try to look at the the chat. Uh
[3:41] okay. So
[3:45] are the user to start interpretable
[3:47] feature for network presentations.
[3:50] Yeah. So
[3:53] uh
[3:56] direction. Yeah, this the thing they I
[3:57] was talking about. They they just say
[3:59] here that
[4:02] papers and express some things are
[4:04] linear and they are there are new papers
[4:08] that uh many concepts are organized at
[4:11] low dimension many fonts
[4:14] go
[4:16] and we don't know how much
[4:19] this is like are all concepts manifolds
[4:22] are most of them are other things
[4:26] linearly linearly presented
[4:28] Uh I don't know. I think you can
[4:31] probably
[4:33] get get some information for all the
[4:35] existing papers.
[4:37] uh a problem is that
[4:41] uh
[4:44] uh I guess I'll talk about it here that
[4:47] uh the thing is they say we here like
[4:50] there's a three question like first what
[4:51] does it mean for me to cut to a manifold
[4:56] and
[4:58] do somehow I think there's a yeah
[5:00] there's a thing here that I think that
[5:02] maybe get into where if you are taking
[5:06] linear near uh directions like you're
[5:09] taking directions as features. You can
[5:12] subdivide the
[5:15] uh some more complicated manifolding a
[5:18] bunch of points and directions
[5:21] and that can make it seem like you line
[5:26] methods that assume linear
[5:29] uh things and work.
[5:32] Uh
[5:34] but yeah
[5:38] show that person and coders can catch to
[5:40] many different ways globally by locating
[5:43] a compat group of fans linear spans
[5:45] entire manifer or locally by
[5:47] distributing features that tiles.
[5:51] Yeah, I think this is the thing that was
[5:53] to go that uh
[5:56] if you are doing
[5:58] normal sports colors or uh linear pros
[6:04] or whatever and your actual the actual
[6:08] representation is some manifold you are
[6:10] going to be tiling the
[6:14] the manifold with uh some linear
[6:19] directions and arbitrary aliasing them
[6:22] to different things even though the the
[6:25] actual representation might be some
[6:26] continuous uh manifold where uh maybe
[6:31] you represent the numbers up some curve
[6:34] and you pick each number as having some
[6:36] direction that happens to match with the
[6:39] correct point in the curve. Uh but you
[6:42] need a lot of features when you could
[6:44] just have some representation of the
[6:45] entire curve and that will work.
[6:49] Um
[6:50] empire empirically we find the SP and
[6:53] colors automatically recover structure
[6:55] missing the global spaces along
[6:57] solution.
[7:00] Okay.
[7:02] Uh we'll see what about explains why
[7:05] manif structure visible
[7:10] discovers methods that ser
[7:14] also suggest that future should
[7:17] geometric as the basic interity. Yeah, I
[7:21] think that's what's uh
[7:25] that's what I'm getting from this uh
[7:27] good fire paper. Although I'm kind of
[7:30] waiting for them to actually release a
[7:32] paper which uh it sounds like this is
[7:35] not that yet where they
[7:38] they actually
[7:41] make something like sparse to encoders
[7:43] but with geometric uh some kind of
[7:47] manifold thing as their features instead
[7:50] of uh directions but yeah they haven't
[7:53] done that yet but they they had said on
[7:55] Twitter that they are working on this Uh
[8:08] >> specifically ask them and they replied
[8:10] soon or something like that.
[8:12] >> Yeah.
[8:18] So so yeah they
[8:21] I mean it was the obvious thing to do
[8:23] after the previous paper. It wasn't
[8:25] surprising they were not doing but yeah
[8:27] they they did confirm to me on Twitter
[8:28] that they are actually working on that
[8:31] they they did at least bake post about
[8:33] it. Uh
[8:35] >> yeah basically
[8:39] >> so okay so what does this actual paper
[8:43] do? Okay. So,
[8:45] uh so this is introduction stuff
[8:52] here presentation
[8:54] map.
[9:01] So yeah. So, so assumes you have linear
[9:06] things.
[9:09] Uh,
[9:16] yes, like someone the stream started.
[9:19] Yes, I'm streaming by the way.
[9:32] Uh anyway,
[9:35] is that Oh, I I I I I was not that that
[9:39] was that remind me I was not actually
[9:41] streaming the
[9:42] >> on on the stream.
[9:44] >> I just noticed
[9:46] >> uh I fixed it. Yeah, I accidentally
[9:48] changed it earlier because uh I was not
[9:50] sure the screen the stream was working.
[9:54] But yeah, so I should also
[10:00] remove this so it's better
[10:04] and maybe zoom a bit because otherwise
[10:06] it sometimes it's hard to see on the
[10:08] stream. Uh
[10:13] yes,
[10:16] where was I? Yeah. So
[10:19] yeah this is more introduction person
[10:21] encoders good and but they have problems
[10:24] whatever
[10:27] uh with applying sperm
[10:31] things a lot that's very good
[10:37] representation is more indicated than
[10:40] linear representation Yes.
[10:47] Uh,
[11:00] >> we should go through some of these
[11:01] papers that are showing different things
[11:04] that are
[11:05] >> Yeah, maybe maybe at some point we
[11:06] should uh read some of those.
[11:11] But yeah, I mean there's maybe some of
[11:13] them as well that I read. Uh it's hard
[11:17] to remember just from the this kind of
[11:19] sitation where it's just the
[11:22] name name of the authors.
[11:23] >> Uh
[11:26] but yeah.
[11:38] Yeah. asations
[11:45] from service but is organized in the way
[11:49] making for contributions. Yeah. Okay.
[11:51] So, so I guess that's the question of
[11:54] maybe the so so the maybe the the sparse
[11:59] of encoders are taking the wrong
[12:00] assumption that is linear but by making
[12:03] a lot of features that are some line
[12:06] they could actually
[12:10] uh have the representation
[12:13] uh the correct representation right but
[12:18] uh
[12:20] that those uh yeah but uh is that
[12:23] actually like you could maybe find the
[12:26] geometric extructors by clustering and
[12:28] that kind of thing on the on the spark
[12:32] encoder features. Uh but I guess a
[12:35] question is is that actually the the
[12:38] case or does the the spar to encoder
[12:42] like does the bad assumptions of sparers
[12:45] makes for example that so that the
[12:48] sparity
[12:49] uh
[12:52] moves the model in the wrong direction
[12:54] because it's trying to be sparse as in
[12:56] the wrong basis or something like that
[13:01] like it's it's trying to be as sparse
[13:04] on tiling some manifolds and it with it
[13:08] we wanted some representation where it's
[13:10] actually uh where the assumption is the
[13:14] part is correct you will need to talk
[13:17] about manifolds or something
[13:20] uh and yeah and yeah what problems does
[13:22] this cause if theation actually at least
[13:27] sometimes some meaningful thing and
[13:30] we're assuming they are linear
[13:33] uh is that what I think this paper is
[13:35] trying to do formalize the program doing
[13:39] by the way at any point other people can
[13:42] say something or comment or something. I
[13:46] mean I I guess I'll try.
[13:47] >> Is there like a formal definition for
[13:50] what u uh for like a manifold versus
[13:55] something that would be captured under
[13:57] the traditional obsumptions of the
[13:58] linear representation hypothesis? Is
[14:01] that anywhere in the paper?
[14:04] >> Uh we'll see. I haven't you know got in
[14:06] there yet but manifold is like the thing
[14:11] is people usually use this uh word a bit
[14:14] vaguely in machine learning but but
[14:16] usually manifold is is a is a space that
[14:20] looks locally ukidian if you zoom in but
[14:22] if you zoom out it's not ukidian. So
[14:24] like taurus for example is a or like a
[14:27] case of a body like
[14:29] >> you can think of it as like
[14:31] a like generalization to more dimensions
[14:35] of like a corp or a surface or something
[14:37] like that.
[14:39] >> Yeah. So so yeah. So like the
[14:41] >> Yeah, they they here they using it kind
[14:43] of vagely where it doesn't actually have
[14:46] to be a surface or a curve like it can
[14:48] have like some gaps or something but
[14:51] approximately a manifold is what they're
[14:53] talking about I think.
[14:55] >> Yeah. Basically uh line representation
[14:58] hypothesis assumes that everything is a
[15:01] line which is a special case of a line
[15:03] manifold essentially where it's like one
[15:06] dimension uh and uh yeah you can have uh
[15:11] you can have instead of lines you can
[15:13] have curves it's and you can have
[15:15] circles instead which are like curves
[15:19] that are connected. Uh then you can go
[15:22] into more dimensions where we have
[15:24] planes or like uh uh uh spheres or
[15:28] something uh
[15:30] >> cubes for example. So uh and the
[15:34] features
[15:35] >> like the circuits in the groin paper has
[15:38] uh a circular feature right something
[15:42] like that.
[15:44] >> Yeah something like that. Uh a lot of
[15:47] other things are uh maybe better
[15:49] examples are like uh days of a week are
[15:53] in a circle. So you could say it's on a
[15:56] circle manifold. Um like here's some
[15:59] examples of some manifolds.
[16:03] Yeah.
[16:05] which is what they basically work on in
[16:08] this paper on uh trying to find these
[16:11] manifold in unsupervised ways from SA
[16:14] features since most methods right now
[16:17] are just hey let's plot it and see it
[16:19] ourselves find the money ourselves
[16:22] >> do they actually do this on the I'm not
[16:25] getting impression actually see maybe
[16:28] maybe I'm misunderstanding
[16:30] >> okay so so we we'll see but my
[16:32] impression from the stuff I read so is
[16:34] that this one is not yet their promise
[16:39] unsupervised
[16:40] sparse to encoder manifold thing but
[16:42] this is still yet another paper of
[16:46] formalizing stuff about why that's
[16:49] necessary what in what ways the linear
[16:53] manif you know linear procs assuming
[16:57] that it's actually represented as a
[16:59] manifold
[17:01] >> we define it
[17:02] >> okay yeah never mind there is I There is
[17:05] an unsupervised
[17:07] uh discovery of many forest structure
[17:09] section. So maybe there is
[17:12] >> yeah okay
[17:14] >> okay but towards a pretty problem. Okay.
[17:17] So,
[17:20] so they do have an
[17:24] Oh, but
[17:28] we'll see what they do exactly. But uh
[17:30] it the physic is a secondary thing and
[17:33] the paper is mainly about the other
[17:35] stuff and uh so that makes me think it's
[17:39] probably not like the uh method that
[17:42] works for any that's you know the
[17:44] perfect method that work for any model
[17:46] yet or like it's maybe has some problem.
[17:51] We'll see what what it says though. Um,
[17:55] >> what do you mean the supervised data is
[17:57] a problem or
[17:58] >> No, I don't know what it I I it seems
[18:01] weird that it was the the thing we
[18:03] actually were most interested in that
[18:07] that wouldn't be the main thing on the
[18:10] paper. Uh,
[18:13] >> okay. I see what you mean. Yeah.
[18:14] >> So, I I'm guessing it's uh there is some
[18:17] caveat here. Uh
[18:21] mhm
[18:22] >> but we see so
[18:25] personal style manifolds.
[18:28] That's interesting.
[18:32] >> Yeah. Hasn't it always been the
[18:33] prevailing theory though that um uh you
[18:36] know um these lower dimen dimensional
[18:39] manifolds exist just like from the
[18:41] machine learning um community in
[18:42] general, right? And then even like
[18:44] possibly
[18:46] >> Yeah. Yeah. Yeah. It it is the uh the
[18:49] manical hypothesis,
[18:51] >> right? And then that local linearity
[18:53] just like being the thing that um is
[18:55] emphasized on, right, which is the fact
[18:57] that um lo things are locally linear. So
[19:00] um presumably that's why uh even making
[19:03] the linear um assumption um typically
[19:06] kind of works. And then more on the
[19:08] extreme end right you then have uh folks
[19:11] in topological
[19:13] um deep learning right which don't make
[19:15] the assumption around um distance or
[19:18] direction and just on connectivity right
[19:20] and then like that you know obsession in
[19:23] that space being um like finding uh
[19:26] where things disconnect. So possibly the
[19:29] fact that even like the manifold
[19:31] hypothesis um has holes in it, pun
[19:34] intended. Um but uh yeah,
[19:38] >> I think about that as well.
[19:41] >> I think the thing on typical machine
[19:43] learning is not talking about feature. I
[19:46] think on making the a lot of papers had
[19:48] been assuming that features are
[19:51] have assuming the linear representation
[19:53] hypothesis partly just because it was
[19:55] easier more than because they thought it
[19:57] was true and it seemed to worked. Uh and
[20:00] I think like often the machine typical
[20:04] machine learning thing was talking about
[20:08] uh I was thinking about this the other
[20:10] day because I think I was a bit confused
[20:11] about this but it's more talking about
[20:14] manifolds in like the data in that like
[20:18] the data lies in some low dimensional
[20:20] manifold of very true input space and
[20:23] this is about learned features and I
[20:27] guess they are related. Yeah.
[20:29] >> Uh that if you know think data license
[20:32] and manifold and and also that I don't
[20:35] know the outputs are also smooth on that
[20:39] uh
[20:40] >> basic yeah it's basically impossible to
[20:43] measure empirically the body holds the
[20:44] data as far as I know
[20:47] >> I mean on on input is space you should
[20:50] be able to like it's just
[20:51] >> uniquely
[20:53] >> uh say or not okay not on text though
[20:57] because like text the continuous things
[21:01] learn embeddings. But if you are imagine
[21:04] a bunch of like even image model you can
[21:08] talk about you know manifolding uh the
[21:11] data. The dream as far as I know is that
[21:13] these models are actually learning the
[21:16] low dimen the low dimensional uh
[21:19] manifold latent manifold
[21:21] >> uh inside the the the high dimensional
[21:24] data sets like that they're actually
[21:26] like properly learning them which we
[21:30] don't know if that's the case uh uh but
[21:32] I think it's true in a weak degree uh
[21:36] >> yeah I think this also like becomes what
[21:40] does it even me in that okay like
[21:43] >> yeah I don't know how to test that
[21:46] >> okay okay so if you are I think I think
[21:48] this is because of what low dimensional
[21:50] manifold are you even talking about
[21:52] because uh like okay if you are talking
[21:56] as like that training data falls into
[22:00] long low dimensional manifold in input
[22:03] space then okay that's uh
[22:07] >> yeah this is assumed to object some
[22:10] objective structure in the data that the
[22:13] bottle is approximating.
[22:15] No, but I mean if you Okay, so there can
[22:19] be that structure like I I think this
[22:22] very like on on space is kind of okay.
[22:26] It's kind of meh
[22:29] but on on pixel space is just clear that
[22:33] there is if there is some structure it's
[22:35] just some structure on the space of
[22:38] >> possible combination of pixels. So that
[22:40] is uh right
[22:42] >> it's like yeah like it's like similar
[22:44] that like uh a group like if you make
[22:47] the model learn groups those groups have
[22:50] explicit uh objective structure uh which
[22:53] you represent as just uh like which you
[22:56] can represent as just tokens. Uh uh like
[23:00] so this is like similar assuming that
[23:03] that the any data uh like this uh that
[23:07] we feed into the models have this
[23:08] objective.
[23:10] Yeah, the the thing you want the model
[23:14] to learn like the model needs to output
[23:16] something like so the thing you want the
[23:18] model to learn is I mean maybe you want
[23:21] it to learn the structure of the data
[23:22] maybe it learns it like for example
[23:24] clearly there is some structure on on
[23:27] the relationship between years or
[23:29] whatever you know big between days or
[23:32] whatever and the models learns it
[23:33] because it has this geometric structures
[23:36] uh but I mean I I think at some uh what
[23:41] we want is the model to output the right
[23:43] thing uh and then uh like maybe learning
[23:48] some structure is useful for that but
[23:50] it's not
[23:53] you know
[23:55] necessarily the case that I assume also
[23:58] see
[23:58] >> yes it is not it is not always the case
[24:00] they learn the structure so like I don't
[24:03] know I think the the the example there
[24:06] is like when you put when you make it
[24:08] learn groups or some other algebraic
[24:10] structure like it has problems with
[24:12] associivity for example um um
[24:17] in in one paper we looked at the where
[24:20] you had the toy model where it land
[24:22] groups so it doesn't all like there's no
[24:24] guarantee that gradient descent will
[24:26] learn this structure
[24:29] >> it open doors though
[24:31] >> but yes I'm just saying that it that
[24:35] like the whole that is kind of becomes
[24:38] under spec both at first it becomes
[24:40] under specified and you need to be very
[24:42] clear about what exactly like in which
[24:46] space the structure you are talking
[24:48] about is like is in some vague compet
[24:52] space of the real war or something or is
[24:55] it just directly on input space and the
[24:59] input space is undefined on transformers
[25:02] that are using text input because it's
[25:04] not continuous uh I mean I guess you do
[25:08] some discrete thing but it's kind of a
[25:11] is just using token is just using
[25:13] embeddings and embeddings are learned uh
[25:17] >> yes some people are talking about
[25:19] structure in a platonic space of ideas
[25:21] but that's unfort
[25:23] >> yeah
[25:25] I was thinking we were arguing about I
[25:27] guess I talk about data because I kind
[25:29] of let's go back to reading the paper uh
[25:31] but I I think I have takes I was
[25:35] thinking on the talking with thinking
[25:37] about more about the toilelet stuff
[25:40] about the you know kitting stuff and I
[25:42] think I was thinking about it to bang I
[25:47] understand
[25:48] >> we may be diverging
[25:49] >> bore what he's talking there but I'll
[25:52] leave that for after the paper
[25:55] let's go back to the actual paper uh
[26:00] >> reconstruction error okay so going back
[26:03] to Uh
[26:10] >> I like that they use model. I model
[26:14] is a very good model.
[26:21] >> Wait,
[26:26] it's a model of feromagnetism in
[26:28] statistical mechanics.
[26:31] represent many
[26:35] detectors.
[26:38] Yes, that's uh also a secret
[26:46] model presentations are likely to be
[26:48] okay. So this is they are saying they
[26:52] they have some evidence that uh this
[26:56] manifold stuff is actually pretty
[26:58] prevalent on the models probably for
[27:01] like the sparse encoder stuff and how
[27:04] they fail. Uh
[27:08] yeah, I think they wrote somewhere or
[27:10] I've seen somewhere or maybe it's also
[27:13] in this paper that
[27:15] uh essays
[27:17] make the make it more
[27:22] uh since they since they they are
[27:24] learned to extract linear direction uh
[27:28] that uh and also discretize it and so on
[27:32] that they can uh uh forget quote unquote
[27:37] or not not properly reconstruct some of
[27:39] the geometry.
[27:41] >> Yeah.
[27:43] >> Uh I think you you were talking about
[27:45] something like that actually.
[27:46] >> Yeah. Yeah. It's also more about uh
[27:51] uh here I think they are saying that so
[27:54] so far we have evidence from previous
[27:56] paper that there are some
[27:59] representations of some concept as are
[28:01] manifolds but we don't that doesn't tell
[28:04] us how what which what percent of
[28:07] features are manifolds I think they're
[28:09] saying it's actually a lot and we'll see
[28:12] why they claim by but it's something
[28:14] about like spar encoders H they did some
[28:18] testing on sports encoders and find that
[28:20] they fail in some way that they will
[28:22] spend if there is a lot of manifolds but
[28:24] we'll see what the address says. Okay,
[28:27] >> maybe hot take is that everything is
[28:28] actually a manifold and there's no such
[28:30] a thing as a feature that's a perfect
[28:33] line like
[28:37] >> every line is broken.
[28:40] >> It could be approximate to the uh
[28:44] direction.
[28:46] Yeah,
[28:46] >> but I mean obviously it's never exactly
[28:49] a direction. There is some wiggle room
[28:54] that line representation hypothesis
[29:00] I'm making.
[29:00] >> But yeah,
[29:02] >> I I don't know. We'll see.
[29:10] This time we will showing mod
[29:18] the structure that give us multip.
[29:22] Yes, I mean not all of them. I I wonder
[29:25] are there cases where the thing is not
[29:32] uh
[29:36] yeah I I wonder if there is if the there
[29:38] is some doesn't that seems like it
[29:40] should be this kid but there is still
[29:42] some manifold probably don't have those
[29:43] but I guess we wouldn't know to look for
[29:45] them. Uh
[29:48] >> yeah, maybe you have to sometimes jump
[29:52] uh if something is very discreet
[29:53] category instead of uh line line
[29:58] interpolation
[30:00] uh line or like not uh instead of
[30:04] continuous steering basically you have
[30:06] to like add like 10.
[30:10] >> Okay. So so they take us with the
[30:13] concept was continuous. Yeah, I I this
[30:16] is the the thing where they take domains
[30:18] with continuous but uh you should also
[30:22] look at random domains too like maybe
[30:24] they are just doing like finding
[30:27] manifolds because they are looking for
[30:29] the places where is more likely that
[30:31] there are many folds and maybe there's a
[30:32] lot of other features I don't know maybe
[30:34] because they don't have some content
[30:36] that can continuously vary uh
[30:44] Yeah,
[30:49] the PC PCA and they see this kind of
[30:52] cool nonlinear geometries
[30:56] conor
[30:58] work like D the week they say new ones
[31:02] but wasn't the colors already a thing
[31:07] though like it's on the or it's only or
[31:11] the colors new from this one I'm not
[31:14] Maybe here they also have lightness
[31:16] dimension or something.
[31:18] >> Oh yeah.
[31:18] >> And special one. Yeah, maybe it's more
[31:21] complex colors than the before.
[31:24] >> We see distance and neighborhize
[31:33] this resource part and highlight more
[31:36] scope of linearies.
[31:38] We show guys that these manifolds are
[31:40] not merely geometrical artifacts but are
[31:42] functionally relevant by measuring the
[31:44] effect of model behavior. Okay. Then we
[31:48] can steer along the manifold by steing
[31:50] between protypical centroidids
[31:53] and the smoothly interpolated between
[31:55] these points.
[31:58] does that depend on the variable
[32:02] that the border output change smoothly
[32:04] and predolations.
[32:08] This indicates the medical structure is
[32:10] not only present in the representation
[32:12] but also casually influences
[32:15] behavior.
[32:27] Yeah.
[32:31] So it's kind of similar to the other
[32:33] paper where they also so they they take
[32:37] the model and they do
[32:40] uh and you know they they do
[32:42] interventions uh by doing doing uh they
[32:47] do manifest steering like the other the
[32:48] manifest steering paper where they steer
[32:51] by moving adding vectors moving through
[32:55] the uh different points on the of the
[33:01] manifold and that changes uh the output
[33:05] smoothly
[33:13] and as the sources cast causal is
[33:17] another example of uh uh making the
[33:20] people trying testing causality and like
[33:24] what the guy in the machine learning
[33:25] thread talk you were arguing with says
[33:28] >> wait wait wait Wait, wait, wait. Uh,
[33:30] well, these people have some other
[33:32] arguments, but let's not go in that
[33:34] rabbit hole. He thinks this is not real
[33:36] cause. Okay,
[33:38] >> okay, I see.
[33:39] >> Because because
[33:42] I mean I think I partially understand,
[33:44] but yeah. Uh there's
[33:45] >> concrete is what it means to
[33:47] successfully capture many for the shelf
[33:50] retainers protocols
[33:54] fraction
[33:56] that stands representations consist with
[33:59] multi-dimensional nonlinear curve
[34:01] geometries.
[34:02] >> Uhhuh.
[34:03] >> Yeah.
[34:05] Uh
[34:09] >> curve geometries. That's the very good
[34:13] uh words. Uh I mean I'm saving that into
[34:16] my
[34:17] >> set of words.
[34:23] >> We are not making a normative class that
[34:26] never let us satisfy this model or
[34:28] representations
[34:29] instead of to make concrete scenario
[34:31] where we can regify super and the self
[34:35] models. Okay. So they are yeah trying to
[34:39] so they have this uh linear
[34:40] representation hypothesis
[34:43] and this cont it's a rain activation
[34:45] space a singleation created by a
[34:47] coicient aha
[34:50] the geometric structure identified
[34:54] uh say this is special case of a richer
[34:58] phenomenon with this con continues over
[35:00] large dimension structures
[35:04] can be the previous civil wars. Okay.
[35:09] Uh
[35:14] probably not that worth for like the to
[35:17] get to we're going to spend too much
[35:20] time look getting too far into the
[35:22] specific math. If someone wants to, they
[35:25] can go through it and maybe explain it
[35:27] or or if someone has already looked
[35:29] through it, we can explain it.
[35:31] Otherwise, I think we
[35:33] >> kind of want to, but it will take more
[35:36] time.
[35:37] >> Yeah. So, for now, let's just uh kind of
[35:40] keep it continue.
[35:43] So, yeah, they
[35:55] a small fix.
[36:00] uh
[36:02] the color consistency groups for every
[36:04] on every input on the main port. Okay.
[36:08] Uh
[36:13] that makes sense. So they are uh
[36:17] >> mixture of manifolds.
[36:20] Yeah, this is about like if you have a
[36:22] spark of encoder with lines, it's like
[36:25] trying to say does this recover the
[36:29] manifold by you know does the manifold
[36:31] have
[36:33] uh enough lines. Okay, so there's
[36:35] actually a uh there's here a bunch of
[36:37] regiments. First like we do this you do
[36:39] this thing where you take the manifold
[36:42] and have a bunch of points that are each
[36:43] one is a feature. Like for example uh if
[36:47] this is the number
[36:50] uh
[36:51] you you can get a uh sports encoders
[36:55] that like has a feature for each number
[36:57] and then this this mess uh
[37:02] and there is this also this compa
[37:04] capture where you have some feature for
[37:06] each of the dimensions of manifold and
[37:09] you are kind of capturing it. Do you see
[37:13] this or I think on anthropic stuff like
[37:17] on the anthropic paper of uh
[37:21] >> the no not the line breaking I mean the
[37:24] line breaking show also was talking
[37:26] about this but I mean there was the
[37:28] example about addition
[37:31] on the
[37:34] anthropic paper about um
[37:38] graph what is the the name uh I'm
[37:41] missing the right now.
[37:44] >> Yes, that one where we're doing
[37:45] attribution graphs. That's the name. So
[37:47] they were doing attribution graphs and
[37:49] they I think they they were implicitly
[37:52] doing accidentally doing something like
[37:53] this where one of the features like I
[37:56] think numbers are there are also
[37:57] probably represented as some like this
[38:00] and one of the features was something
[38:03] like number is
[38:06] uh mod 8 is mod 10 is eight or whatever
[38:11] and that
[38:13] is a feature that is basically
[38:16] allowing here like it's one if you
[38:19] imagine D2 here and the numbers are on
[38:22] the circle and it comes around every
[38:24] tiny numbers like this direction is
[38:27] pointing towards the numbers that is
[38:29] more most activated by numbers who are
[38:31] most whatever fits here and here and
[38:33] here right uh so
[38:36] >> like
[38:38] you said that you think that it's it's
[38:40] on a shape like like uh there's maxar
[38:43] paper that's just explicitly that
[38:44] numbers in dollars are presented as a
[38:46] generalized helix.
[38:47] >> Yes.
[38:47] >> So this is a helix.
[38:49] >> Yeah.
[38:49] >> Yeah. So So yeah, it's a helix. Uh but
[38:53] the activation graphs are from sports
[38:56] colors. So I think it is doing this like
[39:01] it has a theation graph has like a few
[39:04] features and one of them is the like you
[39:08] know the number most thing I think
[39:10] that's what will be here the one and
[39:13] another one is like the number most the
[39:16] first digit which I think is the two
[39:18] right so so like you
[39:21] uh have a bunch of features that if you
[39:24] use person to the song like they some
[39:27] weird trigonometry stuff. Uh, but they
[39:30] are actually just trying to
[39:34] hit this spiral thing. I think it's
[39:37] what's going on, but I don't know.
[39:40] I just heavily suspect it.
[39:43] >> Can you repeat your point?
[39:45] I I think that you have to get some
[39:49] features when you use activation graphs
[39:51] on uh arithmetic that uh are actually
[39:56] just pointing towards the model actually
[39:58] represents the number with uh with uh
[40:00] the numbers as a c like this spiral like
[40:03] in the 10 paper.
[40:06] >> Yeah. Like as far as I know this is how
[40:07] mostly the shapes were discovered. you
[40:09] just sample a lot then you do this
[40:12] dimension reduction or PCA or etc.
[40:15] >> Yeah. But I'm saying that the the spar
[40:18] >> the sparse to encoder from the biology
[40:21] of large models is actually capturing
[40:24] something is like doing something like
[40:25] compact the compact capture thing here
[40:28] where the one of the features is like
[40:31] the number mod whatever. Another feature
[40:34] is the number mod 10. Uh no not not the
[40:38] remote the like the
[40:41] >> uh yeah yeah uh like because you see the
[40:44] periodic patterns of Yeah. Yeah. Yeah. I
[40:47] think that's the case like uh it's
[40:49] represented as this manifold and the
[40:51] different features that just curve
[40:52] different subsets of the body fold in
[40:55] different dimensional ways.
[40:58] >> Yeah. Anyway, going back to the paper,
[41:01] uh
[41:03] speaking uh the features that feel the
[41:05] quality relationship was low dimension
[41:08] for them
[41:09] >> like at the same time what might be the
[41:12] case that uh like there may be multiple
[41:15] of those manifolds for numbers because
[41:17] the model fractures it or something like
[41:19] that. Yeah, I mean I I I and and I think
[41:22] that case was a manifold for addition
[41:26] specifically. I think uh it doesn't
[41:29] necessarily use the same representation
[41:30] for other other things
[41:32] >> and sometimes the models reuse like some
[41:36] of these circuits when they
[41:38] >> you should expect them to reuse them but
[41:41] uh yeah
[41:44] >> even like per modulo 12 valve it's
[41:46] reusing uh base 10 calculator in llama
[41:50] because uh there's just so much more
[41:53] data in base 10 and that's how it reuse
[41:55] the parameters apparently.
[41:57] >> Oh, where is where is that from?
[42:00] >> This is one good fire paper. Did we I
[42:02] think we did read it. I'm not sure. No,
[42:04] we didn't actually. I I just read it in
[42:06] my Okay. Yeah, this is one good fire
[42:07] paper. it that's also about finding a uh
[42:12] uh more general like calculator where
[42:15] like when the model wanted to do some
[42:18] modal arithmetic with months uh it
[42:21] literally like first uh converted it
[42:25] internally into base 10 uh arithmetic
[42:29] did the arithmetic then converted it
[42:30] back to the mod 12 space uh because you
[42:34] shall put it somewhere because that's
[42:36] interesting Okay. And as a result uh you
[42:39] have accuracy loss. But I my hypothesis
[42:42] or I think they also say that in the
[42:44] paper uh is that like it is just useful
[42:47] to reuse it uh because finite parameters
[42:51] finite data finite time etc etc. I
[42:55] >> it's just not about number of
[42:56] parameters. It's more about the general
[43:00] pull towards compression for a bunch of
[43:02] reasons about like the the shape of the
[43:05] loss landscape and all that and also the
[43:07] regularization and yeah
[43:09] >> yeah yeah yeah
[43:13] this one
[43:15] >> uh
[43:17] yeah we have modular subm module to to
[43:20] calculate uh uh stuff that's living in
[43:23] the in the in in module
[43:27] Like this is so stupid.
[43:41] >> Okay. Yeah. So, so here they are
[43:43] separating.
[43:44] >> What is the figure with the icing
[43:47] pairings mean?
[43:49] Uh I'm not sure what their map is
[43:51] exactly but I I think model is a is a is
[43:55] a is a model of feromagnetism
[43:58] statistical like so uh the discrete
[44:02] there's like discrete variables that
[44:04] represent the the magnetic dipole
[44:06] moments of atomic spins in in the graph
[44:08] in in the latice and they interact. Uh I
[44:12] think this was this like you know the
[44:16] guy who got uh uh who got Noble for
[44:20] machine learn AI the he he he got it uh
[44:26] because he b like the or the very
[44:31] I hope field networks I think are based
[44:33] on the icing model uh which are like
[44:38] boltsman as well.
[44:40] >> Yeah yeah yeah.
[44:42] So I'm interesting interested to see the
[44:45] mapping they have here which I don't
[44:47] know yet how exactly they map it.
[44:49] >> Yeah let's let's head to that first.
[44:52] Okay. So first they here they are
[44:54] talking about three things which they
[44:56] have like tiling versus capture. They
[44:58] have uh two types of ting where it has
[45:03] patterning where you just have a
[45:04] direction for each different for for the
[45:07] different
[45:09] uh
[45:11] yeah features manifesto
[45:19] features to manif fromivity resulting in
[45:22] positive corpus for the subport sport.
[45:24] Uhhuh.
[45:26] Delions occur when many atoms activated
[45:29] the manifold but with feature sets of
[45:32] mix seleivity.
[45:35] Uh okay. So you have
[45:40] yeah I'm not sure I completely
[45:41] understand the difference between
[45:42] delusion and shat by now but okay. Uh
[45:52] By the way, does anyone else want to say
[45:55] something? Uh
[45:59] because I've been talking a lot. Mostly
[46:02] >> I think uh I think what they mean by
[46:04] mixed selectivity is that the features
[46:07] are sometimes positively correlated and
[46:10] sometimes negatively correlated.
[46:15] So it it must mean that
[46:18] cuz like in in the case of shattered
[46:20] tiling they say that the features are
[46:22] anti-correlated
[46:24] right. So this is this is probably kind
[46:26] of like what they mention in uh toy
[46:28] models of superp position from anthropic
[46:32] >> maybe
[46:42] I'm going to check uh
[46:46] okay so they many attemps activate the
[46:49] manifold but which feature sets of
[46:51] mixivity okay I
[46:56] Yeah, it's mixed. Yeah, that makes sense
[46:59] because uh the
[47:03] pairings are also mixed.
[47:11] They have this theorem that I will have
[47:13] to read all of the definitions to
[47:14] understand but it says something about
[47:16] how the if the port achieves uh
[47:19] reconstruction error of certain amount
[47:22] then either part the color over uh
[47:29] captures the manifold at some precision.
[47:34] Okay.
[47:36] Uh I guess the the explained here with
[47:39] more with less math essentially where is
[47:42] low enough the sparity regime is aligned
[47:44] with the ambient dimension of M and the
[47:48] dictionary
[47:48] >> the ambient dimension of the body fault
[47:51] is the space where the body fault is
[47:53] embedded.
[47:53] >> Yeah. So
[47:56] what does it mean for the amp for the
[48:01] sparity regimen to be aligned with the
[48:05] ambient dimension of the manifold is my
[48:07] question here
[48:11] because they also talk about
[48:12] >> if I understand it properly but my uh
[48:16] >> about pictures aligning with it. Yeah,
[48:19] since since uh since since I would have
[48:21] to go through the details, but my
[48:23] understanding is that there is like some
[48:25] some uh like ground proof uh manifold
[48:30] dimension and if if the reconstruction
[48:32] loss is low enough then this the the
[48:36] spar just capture the the the the
[48:40] dimension of the actual manifold. Uh but
[48:43] maybe they're saying slightly something
[48:45] else here.
[48:48] Is it say ambient mod uh damage?
[48:52] Uh how do they define sparity regime?
[48:56] I get here they say some something clear
[48:58] they talk about
[49:01] where the features align with the amp
[49:03] space of manifold. So you can also let's
[49:05] talk about the features. I mean the
[49:07] sparity regime I assume like it's
[49:11] talking about the where the model is a
[49:15] spar which is on features
[49:18] uh
[49:20] like it's trying to
[49:22] uh like like okay I think I this is how
[49:26] I mean if you have your your features
[49:30] are aligned with the if the manifold is
[49:34] aligned with the features
[49:36] uh you could you can
[49:40] uh capture it with few features so the
[49:44] sparities align with the features is
[49:46] maybe what they're talking about but I
[49:48] will have to you know read the whole
[49:50] thing to to that and let's
[49:53] have them to to that
[49:56] um
[50:04] if anyone has read that has you know
[50:07] better explanation of this they feel
[50:09] free to just talk and say it or write it
[50:12] on the chat.
[50:13] >> I just want to say it's nice that when
[50:16] they start the math they they start
[50:17] talking about uh the notation so that
[50:22] people Yeah more papers should do that
[50:24] basically.
[50:26] >> Yeah I mean uh we can read that later
[50:29] but like for I feel like uh people here
[50:33] maybe more I'm up for like trying to
[50:36] figure out the M later, but I want to
[50:38] get through I think more people who are
[50:40] more in like reading like getting the
[50:44] the gift of the paper and all that. So,
[50:45] we'll first do that and later we can
[50:49] start going through the maths and if
[50:50] people are bored by that, they can leave
[50:52] by then.
[50:54] >> Yeah,
[50:56] >> I'm going to go through maths right now
[50:57] anyway. But yeah.
[50:59] >> Yeah, feel free to do that while I keep
[51:01] looking at the more
[51:05] Yeah. Okay. Number of atoms are located
[51:07] to a manifold.
[51:09] Uh
[51:14] when the number of atoms are located to
[51:15] a manifold extends in same building
[51:18] dimension
[51:21] this protocol contain
[51:25] different atoms to different regions of
[51:26] M.
[51:28] Okay.
[51:30] >> So yeah. So basically where there are
[51:34] uh where you have more features for a
[51:39] manifold that there are uh dimen that
[51:43] the ambient dimension. Why why the
[51:46] ambient dimension? Because
[51:49] well yeah
[51:50] >> because the features I yeah I think they
[51:53] just mean the dimension of manifold. Uh
[51:59] >> right because the ambient dimension of
[52:00] the manifold is the is the number of uh
[52:03] if you you are trying to do compact cut
[52:06] like this you need like one feature for
[52:09] each uh ambient dimension right
[52:16] >> uh
[52:17] >> yes I think that's exactly what it means
[52:19] so what they're I guess what they're
[52:21] trying to say is that all the features
[52:23] are not automatically able to um to uh
[52:27] to like capture the manifold and they
[52:30] and that's a that it's a precondition
[52:33] for the manifold capture.
[52:36] >> Yeah. Like the the ideal case is this
[52:38] when the model like this manifold has
[52:41] like three dimensions and you just have
[52:43] three features for it and if you have
[52:46] more uh you are going to have uh Saturn
[52:49] or delusion,
[52:51] right?
[52:52] >> No. And also when it's like curved or
[52:54] something then then uh it yeah since I
[52:58] say features are linear then you just
[53:00] have to shutter it.
[53:03] No but even if the the features are
[53:06] linear you can still represent the curve
[53:11] as a combination of features.
[53:15] >> Okay I see what you mean. Yeah,
[53:17] >> like on the capture thing here, like in
[53:19] the compass capt
[53:21] you can perfectly represent the curve by
[53:23] just having one feature that represents
[53:24] one direction of the curve, another
[53:27] direction of the curve
[53:29] curve. But the problem is if you have
[53:34] >> shouldn't it be free? Okay, that's the
[53:35] ambient thing. Okay, I think I see it
[53:37] now. It should be free. Yo, it is free.
[53:39] I see. Yeah, there's D3. Uh, I didn't
[53:41] see it. Yeah. Okay. Yeah, that makes
[53:42] sense. Yeah.
[53:45] Yeah.
[53:46] >> Uh
[53:53] >> yeah,
[53:54] >> diagonal zero in figure three should be
[53:56] right.
[53:58] >> Uh on
[54:00] >> uh okay. So
[54:04] this is
[54:06] uh listening fightings. So
[54:12] no because
[54:16] uh wait no
[54:18] so so what does this I I wasn't what
[54:21] does this graph mean exactly is the
[54:23] question uh
[54:26] right this says uh ising partner pirings
[54:32] which is they probably spread it uh
[54:35] somewhere here but like h Let's have two
[54:42] vectors and three is
[54:46] uh
[54:47] 3 one is uh one and do 2 two two is like
[54:53] the di is zero but the others are are
[54:56] high and here and there is plus one.
[55:00] Yeah, I just have no idea what uh uh
[55:03] what this means. If you understand it,
[55:06] Saki, maybe you can explain it.
[55:11] You meant this.
[55:12] >> I thought that was the correlation
[55:14] matrix.
[55:15] >> Oh, I made it this. Yeah.
[55:16] >> No. Well,
[55:21] >> maybe
[55:23] uh like uh in the sig model
[55:27] they interact
[55:30] >> like if you increase d1 then d2
[55:32] decreases like in a circle. If you
[55:35] increase x then y decreases.
[55:38] So that's negative correlation.
[55:44] >> Yeah. So D1 should be correlated with
[55:47] theelves, right? Uh
[55:49] >> yeah, yeah, I understand why. Yeah,
[55:55] >> of the matrix. Uh like they they usually
[55:57] I feel like that's a thing that people
[55:58] do when uh
[56:02] >> Yeah, maybe these correlation matrices.
[56:05] >> Yeah, it could be.
[56:08] That's what
[56:13] Okay.
[56:23] Uh synthetic validation for manif.
[56:28] Uh so the course they have a benchmark
[56:34] uh
[56:37] I mean yeah so this is a
[56:51] so this is a bunch this is a bunch of
[56:53] things. Uh so they they have this so so
[56:58] they have this synthetic data uh where
[57:01] they have some manifolds on the data I'm
[57:04] assume uh yeah as part of many embedding
[57:10] some of the divisionations and
[57:14] I'm
[57:15] guessing they are trying to get to
[57:20] something personal course
[57:24] on this.
[57:27] Yeah,
[57:34] switch those
[57:46] manifold break down sweep and the
[57:52] Yeah, I guess how I wonder how much I
[57:55] think there's other recent papers on
[57:57] what fire blue is like how much you can
[58:00] go through other papers about you know
[58:03] ways that sporting colors fail and
[58:08] things like you know uh features being
[58:12] abs feature absorption all that and uh
[58:16] related to this the many manifold stuff.
[58:22] I like how things change with sparity
[58:25] and all that.
[58:29] This is person
[58:34] atoms are broadly shared and the manifed
[58:37] groups.
[58:39] >> Oh, this is good.
[58:41] H
[58:43] >> yeah I just want to say this is good
[58:45] that like uh in how they model
[58:47] increasing parity like this was my I I
[58:50] well it's maybe obvious I don't know
[58:52] like if uh
[58:55] it was seen from from from some graphs
[58:57] that from anthropic and others like as
[59:00] you increase sparity you get much more
[59:01] granular features so when it comes to
[59:03] modeling this manifold you get much more
[59:06] granular
[59:07] >> yeah thinking like there was all those
[59:10] preview all that kind of thing. I wonder
[59:12] how they translate into the manifold
[59:15] stuff. Uh
[59:16] >> yeah. So if if you have a lot uh like a
[59:20] very big uh SAPE dimensions and it just
[59:24] it just uh then the the the money
[59:26] manifold is just shattered on all the
[59:28] different parts uh uh like uh templed uh
[59:33] if if if you for to use much less
[59:37] dimensions then it will it will have
[59:41] more uh I don't know feature for yeah
[59:44] like the compact structural capture
[59:46] basically instead of shuttering. Like I
[59:48] I'm not sure if that's general rule
[59:51] maybe
[59:52] >> like as in I'm not sure if like you
[59:55] still also get features for the compact
[59:57] fure capture at the same time shattering
[60:01] uh as well.
[60:03] It seems like you do more shattering you
[60:06] you see more shattering from the SA
[60:08] features I've seen when they are like
[60:10] plotted uh uh across the different part
[60:14] regimes basically. Oh, sparity regime is
[60:17] how many dimensions? I guess that makes
[60:20] sense.
[60:22] Yeah, because you have infinite
[60:23] parameter regime for Okay, that would
[60:24] make sense terminologically.
[60:29] >> Uh, also does anyone else want to say
[60:32] something? Because I think I saw someone
[60:34] uh
[60:37] >> Oh, nice. Uh yeah, I'm trying to think
[60:39] of the compact capture as some vector
[60:41] space with some no minimum number of
[60:44] features needed to expand the vector
[60:45] space.
[60:46] >> Ah
[60:48] yeah, I think that's true.
[60:49] >> Yeah, that that makes sense. Setting is
[60:51] when you have more basic features than
[60:53] the minimal while the you have a lot of
[60:57] features but can turn it entirely.
[61:01] Yeah, I I I think the comp thing is
[61:04] true. I'm not sure for the I'm not sure.
[61:06] I don't completely yet like the
[61:07] difference between spattering and
[61:11] >> like the
[61:13] okay uh if I understand cori it's like
[61:16] you you basically you don't have enough
[61:20] uh you just don't have enough power to
[61:23] to to get the whole manifold.
[61:26] So you deletute and just get subset of
[61:29] it. Many redundant attempts activate.
[61:32] No, like both in both shattering and
[61:34] delusion you have more feature than you
[61:37] did.
[61:39] >> Okay. Okay.
[61:42] Oh, maybe maybe it's like because it's
[61:44] mixel activity then I I I just have to
[61:48] go through the map step by step.
[61:51] >> Yeah, you have to move the mask. Okay.
[61:54] It says
[61:54] >> but like another like that customize
[61:57] that. Okay. So you first go first go in
[62:00] one direction then then then in
[62:02] >> okay so here they're explaining
[62:05] many the delusion as many redundant
[62:08] atoms fire per point and uh individual
[62:12] atoms loss specifically. Yeah. So there
[62:16] are more features than point
[62:20] >> this is uh this is feature uh feature uh
[62:24] duplication.
[62:26] Yeah, I think that's true. True. Yeah.
[62:31] >> Uh, can someone clarify what K means?
[62:34] >> What does K mean?
[62:36] >> Oh, on this graph they are talking about
[62:38] K is sparity.
[62:41] >> Yeah, like yeah, here it says dictionary
[62:43] width and sparity K4
[62:47] >> which I think it like is
[62:52] I'm not sure if this is how they
[62:53] calculated here. uh like the dimensions
[62:55] of the essays is the dictionary. might
[62:58] be a top KSA and they just you know
[63:02] uh put the K K at uh
[63:08] because like the dimensions of the SA I
[63:10] think it's the maybe the dictionary with
[63:13] here because I mean spending
[63:20] yeah I don't
[63:28] Uh
[63:41] even outside this is interesting I
[63:43] think. Okay. So uh the capture regime
[63:46] man for structure can be recovered post
[63:47] hawk firing firec model can which I'm
[63:53] not sure is it something we'll see maybe
[63:55] I think we found the thing we said maybe
[63:57] it's like related to correlation
[64:00] uh
[64:02] just a couple matrix yeah that sounds
[64:04] like it's actually just fancy
[64:07] correlations
[64:08] uh block diag structure align with the
[64:13] ground through manifold partition
[64:17] they call it through the recover atoms.
[64:22] Oh I think yeah this is related to the
[64:25] thing when you have you can get
[64:30] uh curves from
[64:33] cor from correlation matrix related to
[64:37] that paper. Uh do you remember Bernie
[64:40] the the math paper that was talking
[64:42] about how Yeah. So, so I I think this is
[64:45] related to that uh
[64:49] >> recording through recovered atom groups
[64:52] faithfully reconstructs the topology
[64:54] geometry for multiple types of
[64:55] supervision.
[64:58] Like if you have a bunch of features
[65:01] even if they there is shattering or
[65:03] delusion you have a bunch of features
[65:05] and you you can get for the numbers for
[65:08] example you can get the you can get back
[65:12] the shape from the correlation matrix
[65:15] right right
[65:17] yeah something like that was done in the
[65:19] paper that you mentioned
[65:22] maybe that's what they're doing here
[65:24] yeah
[65:25] >> also
[65:25] >> why can we do this for activations why
[65:27] do we have to do just for say features
[65:32] >> uh method general to activations
[65:34] >> do you because you will need
[65:37] correlations between all possible
[65:39] activations and that seems like it's
[65:43] >> understandable
[65:45] >> right like if you have all of the I mean
[65:47] you could what does like you don't have
[65:49] correlation between activ like each
[65:52] activation only happens once so you
[65:53] can't doation between them you
[65:56] I
[65:59] actually what if you did this a thing
[66:06] okay yeah that's stupid
[66:14] weird okay so there's a bunch of this
[66:17] these are very small graphs but okay uh
[66:22] yeah so this is the graph for the
[66:26] thing is capture and other thing is the
[66:29] sparity. Yeah, it's stop K from the the
[66:32] K. Uh and they yeah they did show for
[66:38] I'm not sure what here is but or maybe
[66:41] it's the
[66:43] the dimension of the circle but
[66:48] uh or
[66:52] no. Yeah, I'm not sure what 10 is. Go
[66:58] let me see the graph
[67:01] and two de which is the number of
[67:03] dimensions on the circle which makes
[67:05] sense but yeah I know maybe it's the
[67:09] plate to the capture
[67:12] uh
[67:16] a
[67:17] uh
[67:18] >> this is a sparity seat spot.
[67:29] I think it's sparity.
[67:33] >> No, sparity is the x-axis.
[67:38] >> Yeah, I guess it doesn't matter. I think
[67:39] the main thing is the the bright line
[67:41] and I don't know what other things are.
[67:46] And yeah, aggregate also. Yeah, if you
[67:48] aggregate, you get this one.
[67:54] capture R3. I guess it means how much
[67:57] you capture R3
[67:59] as in
[68:00] >> Yeah, I think the super stealthy ship is
[68:04] is correct. Like you
[68:07] like you can kind of extrude it in all
[68:10] cases but in you have in the capture
[68:13] case you just have one like the minimal
[68:16] amount of dimensions you have one
[68:18] dimension for each uh dimension on the
[68:20] manifold and everything is nice. Yes. Uh
[68:24] at each point is some linear combination
[68:27] of each uh feature. Uh on the shattering
[68:31] case, you have a bunch of latent for
[68:34] each uh for each point. And the delusion
[68:39] cases you when you have even more uh
[68:42] features and you have the you just have
[68:45] more than one feature for each position
[68:47] on the on the graph which I think you
[68:51] need normally that will just you have
[68:56] very
[68:57] you don't have a lot of sparity and you
[68:59] are uh you know uh
[69:06] normally sports are trying to not do
[69:08] that. I trying to have less features
[69:10] that not have more than one feature for
[69:12] the same thing. Uh
[69:17] okay. So yeah, this thing here is Yeah,
[69:19] you
[69:21] have this capture score and capture
[69:23] happen more with this sparity and then
[69:26] it doesn't go down.
[69:31] Someone in the chat by the way try to
[69:37] Yeah, I was uh talking about that.
[69:40] >> Okay, I see.
[69:42] >> Yeah, I think it's it's right more or
[69:44] less. Uh okay, so
[69:49] yeah, it makes sense. It's like feature
[69:51] duplication.
[69:55] >> So there is a bunch of Okay, so here
[69:56] what do they say about this? So it's
[69:58] atom with Yeah, this because
[70:02] neuroscience. Oh, there's a Oh, this is
[70:04] interesting. Oh, there's some
[70:05] neuroscience thing where new are to be
[70:08] sensitive to different values of a
[70:10] concept
[70:12] covering overall geometry by the
[70:14] population code. I mean that sounds like
[70:16] actually the neurons are uh the
[70:21] representation with neurons are also
[70:23] some manifold to the the neuroscience
[70:26] people are just uh being confused by
[70:29] looking at each nebu individually. The
[70:32] way you would also be confused if you
[70:33] were looking at each neuron
[70:35] individually.
[70:35] >> They they also Yeah. They also look at
[70:37] the aggregates and so on. There's also
[70:40] sparse going. Yeah. Like
[70:41] >> neurom manifolds are different. Yeah.
[70:44] >> Uh
[70:45] >> in line with the we tiling localis with
[70:49] overlap and support. Uh yes continues to
[70:52] be uh we need to look into
[70:56] neuroscience for making
[70:58] >> I think that's I'm agreeing that
[71:00] actually. I think a lot of making
[71:02] therapist actually rediscovering
[71:04] concepts that exist in neuroscience and
[71:06] not everything but all of it.
[71:08] >> Yeah,
[71:09] >> maybe it may be the case also because
[71:10] neuroscience people are coming into and
[71:13] then showing the method. Yeah, as we saw
[71:17] in I think a lot of is both in
[71:20] neuroscience and here you have some uh
[71:25] amount of things that are doing some
[71:30] combination maybe even maybe even like
[71:34] linear combinations to with some
[71:36] nonlinearities to do stuff and you're
[71:40] trying to represent data with that. Uh
[71:45] and yes, so you are going to see a lot
[71:49] of the
[71:51] a lot of the same, you know, phenomenon
[71:54] like you're going to have things with
[71:56] superposition all that I mean like
[72:00] >> a lot of things are geometry and you
[72:02] should expect the same thing to happen
[72:04] on the other things because also
[72:05] geometry right uh
[72:08] >> I I think yeah I think in the original
[72:10] essay paper from entropic they say that
[72:13] there there's a corresponding
[72:16] uh method in neuroscience to to the
[72:18] essays uh to sparse coding uh I don't
[72:23] know how it's called it it has a
[72:24] different name also also also it's used
[72:27] in signal processing
[72:30] uh oh learning is used for EEG
[72:34] I I just
[72:35] >> yes you should expect that to happen
[72:38] because it is just a lot of this stuff
[72:40] are general facts about like you know
[72:45] the geometry of having representing
[72:48] thing as combination for a lot of uh
[72:52] things like neuron activations.
[72:56] Uh
[72:58] and if you want to do that have
[73:00] something that does that efficiently you
[73:02] are going to see the same kind of shapes
[73:04] and so because they are just efficient
[73:07] way of representing things on that
[73:10] space.
[73:12] Uh I mean if the actual potential but a
[73:15] different learning algorithm or whatever
[73:18] uh
[73:24] yeah
[73:27] okay so
[73:33] yeah so so they have talk about ting and
[73:35] inside taling they have shattering and
[73:37] uh delusion
[73:40] uh
[73:42] where yeah the difference as we said a
[73:44] bunch of times here is that in
[73:46] shattering they have a set of nearly in
[73:52] this yeah they have a set of nearly
[73:55] disjoint pictures or whatever atoms
[73:57] partition the manifold and illusion when
[74:00] there is overlap
[74:03] uh
[74:05] and yeah regardless of whether according
[74:08] to dynamism the group of the code
[74:11] manifold is unknown among discovered
[74:14] from the codes alone.
[74:18] Yeah, to this end one must use quivation
[74:23] statistics atoms that usually represent
[74:25] a manifold fire together or in a smooth
[74:27] session of input zone m
[74:30] yes that's the thing also thing we were
[74:32] talking about uh raw cultivation of
[74:35] course conforms to sources
[74:38] structural coativation yeah basically
[74:40] because they are uh which is yeah atoms
[74:45] span at the same manifold yeah so that
[74:48] you
[74:49] two things, two atoms like sampling
[74:51] their similar position tiling so many
[74:53] fall and so when you move through it you
[74:57] start you know when you one is active is
[75:00] more likely the other is active because
[75:02] they are on the same uh like they're on
[75:06] the same line or something and you're
[75:07] moving through the line that they are uh
[75:10] on the same manifolding
[75:13] um and correlational cos
[75:16] that data So yeah, where there is also
[75:19] the case where this
[75:21] they are not on the same manifold at all
[75:24] or like they are completely different
[75:26] directions in activation space but it
[75:28] just happens what the when the model uh
[75:32] uh activates a fe one feature that tends
[75:35] to activate the other for this or the
[75:37] air later.
[75:43] is also dominated by atoms that fire
[75:46] universally with activation everything
[75:48] with car manifestation tool. Okay. Yeah,
[75:51] that makes sense. This is also atomos
[75:53] that fire universally
[75:55] are like features that f universally are
[75:58] a problem with for sport encoders too.
[76:01] uh because if you are trying to do a
[76:03] sparity you you'll
[76:07] have problems with features that are
[76:09] always there because either you want to
[76:11] always represent them or like uh I mean
[76:15] you can not have a feature for them and
[76:17] just make the decoder always write them
[76:20] or something but they they could be that
[76:22] they doesn't always fire just fires most
[76:25] of the time and yeah there was some
[76:28] papers about this about like a high
[76:31] frequency features on spark encoders. Uh
[76:39] I pair and high frequency
[76:45] features in express encoders. There was
[76:47] a paper yeah that we read on that.
[76:51] >> I mean I would expect like I don't know
[76:53] English to be one high frequency
[76:55] feature. Well it's
[76:57] >> yes that an example if there is some
[76:59] English feature it's activated most of
[77:01] the time. uh uh you you like the the the
[77:06] that generates problems with like the
[77:08] learning I I don't remember sly what was
[77:10] the the details where that yeah like the
[77:12] learning process where it wants to do a
[77:15] sparity and only represent certain
[77:18] features but it could if I always
[77:21] activate the English feature uh but then
[77:24] if it let's say it has top K there's
[77:27] already one less fe and has tech four
[77:29] that's already one of four features that
[77:32] it has to remove. Maybe instead it wants
[77:35] to if only English text it will want to
[77:39] do something like put English
[77:43] activations in all things that are
[77:45] correlated with being English text
[77:47] instead of making it a feature or
[77:49] something. Uh
[78:00] uh
[78:02] spiral relations. Wow, this is cool.
[78:04] They try to look at spir Yeah. By the
[78:07] way, this is one of the issu one of the
[78:11] one of the critiques of the causality
[78:13] people is that spirious correlations and
[78:16] compounds everywhere. And this paper
[78:18] specifically talks about compounds and
[78:20] spirations
[78:22] chefskis. I like it. Anyway, uh yeah,
[78:25] they also have other critiques. I think
[78:26] we can learn a lot from the coal by the
[78:28] way. Uh
[78:32] >> okay. I mean I think
[78:34] >> I mean the ma mathematical consultative
[78:36] people not the philosophical
[78:39] people as well but honestly I have very
[78:41] much low good experience with that. I
[78:44] don't know maybe I'm just not
[78:46] openminded. I mean I think I mean yeah
[78:47] because this it's true the yeah I don't
[78:52] know people are already know about
[78:54] causality and like the I feel like a lot
[78:57] of papers already are trying you know
[79:00] trying different interventions
[79:03] >> but
[79:04] >> if you want we can we can read this
[79:05] paper that I linked which is the most
[79:08] quote unquote causal paper that I could
[79:10] find um so which criticizes a lot of
[79:14] McKinter methods at the same It's it's
[79:16] basically saying causal abstraction
[79:18] theoretical foundations for making D. So
[79:20] it's like saying this is how you do
[79:22] properly. Uh
[79:23] >> I mean I'm missing but did we read this
[79:26] paper or a similar one? Maybe
[79:27] >> I don't think so maybe long time ago. I
[79:31] don't know why it wasn't here. Anyway,
[79:33] going back to this icing page and leg
[79:38] what I I what I'm curious what's icing I
[79:41] I'm get I get that is uh some way of
[79:45] measuring correlations by now by context
[79:48] but I'm not sure exactly what it is.
[79:51] Uh
[79:54] it it uh it uh it wait
[79:58] uh
[79:59] I sometimes go through the icing
[80:01] >> like the
[80:05] ferroagnetic thing
[80:06] >> and uh it it shows how like the
[80:09] different spins interact with each
[80:11] other. So it it like makes them line up
[80:14] uh in positive or negative direction uh
[80:18] depending on the definitions.
[80:20] So, so yeah, but how
[80:23] >> it can get to some local minima and that
[80:27] and from that uh the the first like
[80:30] habian I mean not hibian the
[80:33] >> like what's like in how does what size
[80:37] like how does it work here because it
[80:41] seems like it's working as some kind of
[80:43] way of talking about correlation between
[80:46] uh between
[80:48] color features Right. But like how does
[80:51] that work on?
[80:52] >> Yeah, I I don't know. I I I have to go
[80:55] step by step through I have to go step
[80:57] by step through Ruba. I was start going
[80:59] through the map and and then I starting
[81:01] to
[81:02] >> like so far. I'm I'm like the main thing
[81:05] I'm confused is why what does this thing
[81:08] that would you wouldn't get by doing all
[81:10] the same stuff but like just correlation
[81:12] a correlation matrix between features.
[81:15] Uh
[81:16] >> maybe
[81:19] maybe they okay I I I don't know yet
[81:21] exactly but maybe they do it because if
[81:23] you cast it and rising uh this frame
[81:26] then you by poor pre get some
[81:29] mathematical results from from the other
[81:31] field.
[81:32] >> That is my hypothesis why they is
[81:34] usually the reason why they cast it to a
[81:37] different domain through someorphism or
[81:39] something at least if they do that. Yeah
[81:42] >> after conditional rest population. Okay.
[81:44] So,
[81:47] uh it's a dependency that arise from
[81:49] atoms represent their manifold
[81:54] uh out of frequently simpl.
[82:02] It seems like they do get some nice
[82:04] properties from this but okay. So
[82:07] indirect correlations in this position
[82:09] are factory out by construction
[82:14] as a result.
[82:19] Okay, they are basically
[82:23] uh Bernie they basically answering my
[82:26] question uh here that for a bunch of
[82:30] that it it has some nice properties uh
[82:33] that I don't understand why because I
[82:35] don't understand how icing works but
[82:36] whatever uh like where it's h somehow it
[82:41] avoids counting superposition
[82:45] and uh
[82:48] >> arcted by construction
[82:51] >> for some reason that I don't understand
[82:53] by construction and uh
[82:59] and also atoms that are you know
[83:02] features that are high frequency
[83:04] features uh have some large whatever
[83:10] number and then we couplings
[83:13] so you can separate them.
[83:16] Uh so so yeah they're saying that for
[83:18] these reasons is better than just doing
[83:20] uh correlation.
[83:23] So so basically saying that the all of
[83:25] the problems that uh they mentioned
[83:28] before is that maybe the correlational
[83:30] co occurrence uh are solved by just
[83:33] using this uh free thing. Uh I yeah no
[83:38] idea why.
[83:41] By the way, if someone if someone thinks
[83:43] I'm getting something wrong, just tell
[83:45] me. I'm just kind of trying to figure
[83:47] out as as I go and tell you what my
[83:50] current idea is. But like often when I
[83:52] reading these papers, I I'm wrong for a
[83:55] bit and then realize I was think more
[83:57] about it and realize I was wrong and
[83:59] then I was I'm wrong again and then I
[84:01] change my mind and then I end that, you
[84:03] know, only understanding some things
[84:04] until like the end of the paper if ever.
[84:10] Hey,
[84:21] if you're interested in some specific
[84:22] part of the paper, we can briefly skip
[84:24] to it and read it. Uh but yeah,
[84:34] the science
[84:40] in the ting
[84:42] specialization
[84:48] like the spin model. Yeah, I'm not I
[84:51] don't see the the the correspondence
[84:53] yet, but in in the spin model like each
[84:56] spin tries to agree with the spins next
[84:58] to it. So like if you have one and minus
[85:01] one next to each other, then they will
[85:03] both like one will go down, one will go
[85:06] up and eventually they might like agree.
[85:09] Uh
[85:12] also by the by the way yeah so this
[85:15] thing is not correlations is this icing
[85:17] thing which is kind of like correlations
[85:19] but it's not correlations and maybe
[85:21] that's why the di the actual reason why
[85:23] diagonal is white but no idea. Yeah. And
[85:27] like the the idea there is like that the
[85:29] the aligned neighbor lower energy uh
[85:35] and like there is no rule that uh
[85:37] depending on how you start the whole of
[85:40] spins that try to align with each other.
[85:42] There is no rule that it will eventually
[85:45] all like go to some local minima where
[85:49] everyone lines up globally basically.
[85:51] So, so it can just do some oscillation
[85:53] or something infinitely.
[85:58] >> I don't know the mapping this paper yet.
[86:00] I have to go through it in more detail.
[86:07] >> Okay. Yeah. So,
[86:12] uh
[86:13] the tailing regimen atoms regimen. Yeah.
[86:18] So,
[86:20] input positive couplings within the
[86:22] group uh the diarrhe
[86:27] negative couplings that exclusions.
[86:30] Okay. Yeah, this is again the the thing
[86:32] with the icing on the on the previous
[86:34] figure.
[86:40] Okay. So the basic thing is they they
[86:43] say they they claim they can you you can
[86:46] separate the capture and the two kinds
[86:48] of tiling depending on what the how
[86:52] negative or positive the couplings are.
[86:55] Uh I mean on the real there was going to
[86:56] be a mess with some parts and some
[86:59] things are going to be negative some
[87:01] positive and yeah uh
[87:06] is it centration matrix identifying
[87:09] manif through method symmetric relations
[87:12] we can recover
[87:15] them as communities of atoms with a
[87:17] strong pairway interactions
[87:20] irrespectable those interactions are
[87:22] cooperative.
[87:23] Okay, that makes sense.
[87:27] Depends in activations with Okay, so I
[87:31] guess this may be related to the things
[87:33] where they talk about unsupervised
[87:35] fighting of medievals. Maybe it's
[87:39] looking at
[87:41] reference from spar colors but it's not
[87:44] yet from you know training something
[87:46] that serves color but with var
[87:50] uh
[87:52] maybe that's a separate thing I don't
[87:53] know
[87:56] uh
[88:01] okay yeah that's nice so so you have
[88:02] this person colors and from correlations
[88:06] But in this physics framing uh you can
[88:11] find many faults on those uh on on on
[88:15] the uh from the uh correlations between
[88:19] activations of uh different features.
[88:22] Uh I guess this yeah still has the
[88:25] confound of the of the
[88:30] know of correlations of of correlations
[88:33] that are not about structure. But you
[88:36] can probably separate that out though.
[88:39] Uh
[88:45] >> like one what like
[88:47] >> yeah like
[88:49] there there's
[88:52] an architecture called coal ja by lood
[88:55] and one critique of the coality people
[88:57] is that anything this architecture can
[88:59] learn is a spreation
[89:01] well yeah uh might be more general for
[89:04] ml
[89:06] >> I feel like the general I think the cons
[89:09] people are for that say that the ML can
[89:12] only learn correlations are wrong and I
[89:15] have taken but I
[89:18] >> it is not that they only well okay some
[89:20] people say that but like so some some
[89:24] people that are more nuanced say that
[89:26] like the models struggle to like uh uh
[89:30] separate previous correlations and
[89:32] actual correl and like actual
[89:34] correlations and and causal relations
[89:36] and so at the same time I I think the
[89:39] same for humans but uh that's uh I think
[89:42] we can build architectures that actually
[89:44] uh can separate it better than humans in
[89:46] superhuman way.
[89:47] >> Okay. Uh I mean I think it's just a
[89:49] question of well I won't get into this
[89:51] slide now because I kind of want to
[89:53] finish the paper but uh I think it's
[89:54] just a question of finding of inductive
[89:57] biases and finding simply simple
[89:59] explanations of uh the training data and
[90:02] how you can see how like if you think
[90:06] about solommon induction it finds the
[90:09] correct causality even though it it's
[90:14] only getting the the data And I feel
[90:18] there is some kind of people have
[90:20] philosophical views that are broken by
[90:22] that because they will think that you
[90:24] can only if you're you're only seeing I
[90:26] don't know combination of of simples or
[90:28] whatever and then you can't get the true
[90:30] causality because you're not doing
[90:31] causal interventions or whatever but
[90:34] that's not true and
[90:39] the answer is about how you can actually
[90:42] get causality from correlations with
[90:44] some inductive bias. But let's ignore
[90:47] that and go back to the paper. Also,
[90:50] uh Usagi posted something that with a C
[90:54] explanation of the
[90:57] of something. Let me see
[91:00] on the chat. Uh a concrete sample. Three
[91:03] features uh for weather, winter, hot
[91:07] chocolate
[91:09] will be correlated on the data.
[91:13] don't directly interact.
[91:17] Okay. Yeah. So,
[91:20] uh cold weather and hot chocolate
[91:22] together a lot. Yeah.
[91:25] Uh
[91:31] has high value. This model will be near
[91:34] still copy because of the condition the
[91:37] three on D1, D2, everything.
[91:40] Okay. I still feel like I will need to
[91:43] understand
[91:46] how the icing model works to understand
[91:50] why is that the case. But yeah, I
[91:52] understand that this also seems related
[91:54] to the correlations versus conversation
[91:56] stuff, right? Uh
[92:03] uh
[92:05] I I mean the thing that if you read the
[92:07] thing that posted on the chat.
[92:11] >> Okay. I see. Close explains.
[92:14] >> Yeah. I'm going to check
[92:22] >> related to how like you can see the
[92:24] causality as being a way
[92:28] uh efficient way to talk about condition
[92:31] the structure of conditional
[92:33] correlations.
[92:35] Uh
[92:39] example imagining pre features.
[92:57] Anyway, so
[93:00] so they have these toy models. Anyway,
[93:02] going back to going back to this. So,
[93:06] they do the toy model. There was on a
[93:08] previous figure uh and they
[93:14] this is the figure for stuff. Uh
[93:18] >> condition.
[93:39] Okay. Yes, that's nice. But that's on
[93:44] some s data where they just have some
[93:46] actual manifold and they just try to
[93:48] reconstruct it. Uh the question is does
[93:52] what happens on real on real models.
[93:56] Okay. So now that that's what they are
[93:59] doing now. Uh
[94:06] so we train fire architectures standard
[94:10] L yelu top k matrioska
[94:16] uh with a bunch of different parameters
[94:20] uh
[94:23] do not even compat capture they apply
[94:27] the same to for each manif we
[94:33] I I never actually look at the how how
[94:35] they actually test think this how they
[94:38] test that but okay mission
[94:43] quality as a support size fix shows the
[94:48] average across many false architectures
[94:50] versus print with the number of features
[94:54] produces
[94:56] manifestations
[94:58] huh Yeah.
[95:01] Uh this indicates that do not allocate
[95:04] the components contest to the manifold.
[95:07] I mean is that but are there not so many
[95:11] manifolds where they do that though? uh
[95:16] >> they do not allocate compact to spend
[95:18] the money
[95:20] >> like they are amazing about uh manifolds
[95:23] and architectures but uh it seems
[95:25] possible to me that uh like the actual
[95:27] thing is more like it's capturing some
[95:30] manifolds and not capturing at all
[95:32] others and so on. Uh
[95:38] >> yeah,
[95:40] >> also they're talking about manifolds,
[95:42] but uh do they
[95:46] how they know which manifolds to pick?
[95:50] Are they okay? We'll see.
[96:11] Yeah, thanks super.
[96:13] >> Okay, they're looking at the HDR
[96:15] temperatures and you know those fighters
[96:17] that
[96:19] >> Yeah, come on. Come on manifolds.
[96:23] >> Yeah.
[96:27] Uh so so uh this is always resting in
[96:31] the fact that uh them knowing that those
[96:34] are actually those manifolds. I mean you
[96:37] could be wrong but I I mean I think it's
[96:39] probably right
[96:41] but
[96:42] >> they say they find it in unsupervised
[96:44] way. So
[96:46] >> how do they do that?
[96:48] >> Yeah.
[96:50] >> I think you go
[96:52] that's another part finding a
[96:55] supervisor. That's I think that's
[96:58] after this
[97:00] uh here they have some manif that they
[97:02] found earlier and they are uh testing
[97:05] how the models uh detect them
[97:09] like how the spark encoders like
[97:11] features uh
[97:32] style force detectors
[97:35] to comply. How do they represent them?
[97:38] Uh fix five contra activations with
[97:42] partical construction using case number
[97:44] features middle features reconstruct
[97:46] local patches of the many falling
[97:48] awesomely
[97:52] covers the full geometry. The results
[97:54] place current colors in between capture
[97:58] and shattering. Huh.
[98:01] Okay.
[98:03] So yeah, they're saying that they're
[98:05] kind of captur a bit and shatter a bit
[98:08] or like probably around middle. I think
[98:10] it's kind of they're taking average not
[98:14] necessarily that.
[98:17] Oh, they're they're looking at specific
[98:19] 24. So
[98:22] yeah, so for days
[98:30] Yeah.
[98:34] So it was not right because they're pick
[98:36] they're not doing the average man. So
[98:38] they're picking some specific manif and
[98:40] checking that one. It seems like it's
[98:42] kind of doing a mess. Uh,
[99:51] Okay,
[99:58] put it on the
[100:08] geometry.
[100:13] Okay. So
[100:20] yeah. So so if you do the whole icing
[100:22] thing, you they're saying that you
[100:25] recover
[100:26] uh
[100:28] you do the icing thing on the sports
[100:30] encoders and you recover these manifolds
[100:33] that they they already knew were there.
[100:39] Same pipeline profess a novel manifold
[100:42] encoding epistemical center in
[100:44] scientific context. This seems also
[100:47] related to
[100:50] uh
[100:52] a novel manifold encoding epistemic
[100:56] uncertaintity.
[100:57] Oo
[101:00] >> okay
[101:02] BCA.
[101:04] Yes.
[101:10] >> Okay. Then supervised manifold
[101:12] discovery. Uh
[101:15] >> okay. Let's see.
[101:26] >> Yeah.
[101:29] Okay. So I think this section is so so
[101:32] the the reason why this is not the the
[101:35] yeah the the paper that that they talk
[101:39] about right that they that they were
[101:42] talking about Twitter is that this I
[101:45] understand it is about taking the
[101:48] meaningful features and recovering
[101:50] manifold from them
[101:53] not unsupervised
[101:56] way but it's not unsupervised from The
[101:59] activations directly get in manifolds.
[102:11] Okay. So a natural starting point this
[102:14] is the thing actually work uh better
[102:17] than just the simple thing of uh
[102:20] similarity or whatever by the similarity
[102:23] work. However uh under space capture the
[102:28] atoms spanning money for spaces maybe
[102:30] near oral under ting atoms convert
[102:34] converting adjustable
[102:36] have similar decoded directions. Okay
[102:39] that makes sense. uh decoded geometry
[102:42] that carries no privilege information
[102:44] about the meaningful topology that
[102:46] favors quile.
[102:48] Uh so that's
[102:51] yeah okay co vision st offer a more
[102:54] principal alternative features that
[102:56] jointly represent
[102:58] a manifold fire together
[103:02] uh in the center in case the features
[103:04] join
[103:07] as I mean do they
[103:10] >> oh
[103:10] >> I I guess they only fire together if
[103:14] they are close
[103:16] right And you call
[103:23] how? Yeah,
[103:24] >> like this reminds me of the uh uh
[103:28] physics model in uh I think it was in
[103:31] the uh in the geometry of manifold where
[103:33] you also have like features which are
[103:36] yeah which are like uh repelling and and
[103:40] and
[103:42] uh uh attracting.
[103:45] Uh
[103:46] >> yeah, although that was about the model
[103:49] putting things in the in a manifold
[103:52] because it wants them to be separated in
[103:56] when uh they are be close and when they
[103:59] are close and separate when they are
[104:00] separate. Uh and this is kind of
[104:05] imperson that in some sense because this
[104:07] out once you have the features then the
[104:09] one that are close will fire close and
[104:11] we you know fight together and the ones
[104:13] that are separate will separate
[104:14] together. Uh
[104:16] >> yeah it makes sense.
[104:16] >> Uh but
[104:20] this is yeah
[104:23] yeah this is related to coactivation. I
[104:27] I kind of think there is some reasons
[104:30] why not me always the case. I need to
[104:33] think what about this but if it's like
[104:35] uh
[104:37] you compare five simulator measures. Oh
[104:39] my god.
[104:43] Like
[104:47] do you like if you have do you you don't
[104:50] really have
[104:53] I guess I guess depends on what what you
[104:58] are
[105:00] uh only if it's styling only if it's
[105:05] uh you know uh compact because like how
[105:10] much increasing one increases the
[105:14] others.
[105:17] I need to think about it. It's not
[105:18] obvious to me. It's always the case that
[105:20] you you have uh that symbol
[105:25] >> and you sample globally enough then that
[105:28] makes sense
[105:30] like you can locally
[105:32] >> are things on the same manifold
[105:33] necessarily correlated like you could
[105:36] imagine different points being
[105:39] >> if you s if you sample globally enough I
[105:42] think so if you sample you can sample
[105:46] like
[105:47] like the the idea that I'm thinking of.
[105:49] >> I guess they Yeah, they like one is
[105:53] they're corrected with the near one and
[105:55] then the near Y is correct.
[105:57] >> Yeah, like like for math concepts, I
[105:59] don't know like you can have a set is
[106:03] very it can be very correlated if you
[106:05] sample only with like group or something
[106:08] if you sample only group theory text.
[106:11] But if you sample uh uh I don't know
[106:15] pure set theory that's not in algebraic
[106:17] structures or something set will be
[106:19] there but it will be completely
[106:21] anticorrelated with group uh so I don't
[106:25] know maybe that was the
[106:29] yeah the point you said you was also
[106:30] making I don't know
[106:32] >> uh I don't know
[106:35] uh
[106:37] >> so how they correlate with each other is
[106:39] dependent on the exact thing that the
[106:42] exact data points that you're sampling
[106:43] and getting activations from.
[106:53] So they try different
[106:56] okay so they try different methods of
[106:58] checking similarity and they find that
[107:00] the
[107:02] icing coplins and conditional
[107:04] cultivation re the curation. Okay. So
[107:08] also conditional quativation. I feel
[107:10] like I think open is related to
[107:12] conditional quativation by explanation
[107:14] on the on the thing that CL did. Uh but
[107:19] yeah, I feel like I need to learn more
[107:21] about that. Maybe ask CL more.
[107:24] Uh
[107:24] >> like uh yeah, what I'm trying to do now
[107:27] is just going where the buff math
[107:30] starts. It's just every single
[107:32] definition, every single variable I
[107:34] don't know. Just googling it basically.
[107:37] or clothing it.
[107:45] Okay. So, yeah, this is this is here. I
[107:51] I'm kind of
[107:54] Okay. So, so,
[107:58] so this this is the end of the paper.
[108:01] The the this is nice as a way of
[108:03] testing. So this is not yet a method for
[108:06] the money force in general. I don't I
[108:08] don't know feels like they should be
[108:11] able to what is the bottleneck here on
[108:14] just not finding you know finding
[108:16] hundreds of manifolds in the models from
[108:19] the from the person goers like yeah it
[108:22] would be better if you had some method
[108:23] than supervised just from the street
[108:25] that directly finds manifolds. Uh but
[108:31] >> can you say again your point? Why I
[108:34] curious why what's the bottleneck for
[108:37] why this paper has like an example of of
[108:41] manifold
[108:43] uh
[108:44] uh of of manifold that they found using
[108:47] the icing base discovery pipeline and
[108:51] it's not like we use this pipeline and
[108:54] we found like 300 manifolds in the model
[108:58] you know what I mean
[109:01] >> can you try to rephrase your it or say
[109:03] again
[109:04] >> why they didn't like what's the
[109:07] bottleneck for just finding hundreds of
[109:09] manifolds in the model using this method
[109:12] >> more confused I don't know
[109:15] >> yeah I mean I guess there is the
[109:18] labeling part of this manual but uh I
[109:21] don't know seems like it should be
[109:24] easier to fight more maybe but yeah I
[109:28] don't
[109:32] Like if we maybe I don't know if they
[109:34] have code or not but if they have if we
[109:36] just go apply their stuff and run it on
[109:40] the models can we find another manifold
[109:42] and if no why not is the question. H
[109:48] yeah, that could be done
[109:54] because it's not like the other stuff
[109:58] was supervised. So you require having
[110:00] coming up with some idea of something
[110:02] that maybe represent as manifold but
[110:04] this will be just basically doing
[110:06] clustering on automatically on on person
[110:09] feature. So yeah,
[110:13] new science. So that's basically it. I
[110:15] wish we can look a bit to the appendix
[110:18] but I think we mostly
[110:21] but we we mostly finished now. Uh I
[110:24] guess people feel free to talk about
[110:29] your takes on the paper and all that uh
[110:32] general uh if you want.
[110:36] Uh
[110:39] >> yeah, I want to go through through
[110:41] through the map in more detail though.
[110:43] >> Otherwise, I think the thing to do now
[110:44] is like uh yeah, maybe
[110:48] I we need to maybe I'll end the
[110:51] recording or maybe not. I'm not sure.
[110:53] But uh the thing uh people don't have
[110:56] takes now. I think maybe we just can
[110:59] spend some time trying to understand the
[111:02] the the
[111:03] >> question like can you scroll up?
[111:07] >> Yes.
[111:10] >> Uh which page?
[111:12] >> So um
[111:16] it was from uh Yeah, you have to scroll
[111:18] up a lot actually.
[111:19] >> Yeah. Uh
[111:26] >> uh one of the figures I think.
[111:29] >> Wait, uh is this one?
[111:30] >> Yeah, I I think this one. Yes. Yes. The
[111:32] one with sparosity. Yes.
[111:35] >> So um I think they mention that uh
[111:41] Okay. Can you scroll down a little?
[111:44] Yes. So um they mentioned that uh at low
[111:49] K atoms are broadly shared and the
[111:52] manifold is shattered across unrelated
[111:54] groups. So uh is this like an example of
[111:59] superposition cuz like the groups are
[112:02] unrelated but the shattering is done
[112:07] across groups. So do they mean are are
[112:11] they saying that
[112:13] >> uh multiple groups are involved?
[112:16] >> Is that what they mean to say?
[112:18] >> Okay. So the you're talking about B then
[112:20] right?
[112:23] >> Yes. Uh at low the low regime.
[112:28] >> Yeah.
[112:28] >> What does it mean exactly?
[112:30] >> Atoms are broadly shared at the monifold
[112:34] is shatter related groups. Okay. So
[112:39] uh
[112:40] >> specifically I also like it's also
[112:43] unclear to me what an atom means like
[112:45] could you explain that?
[112:46] >> Okay so I think atoms here
[112:51] it just means the in the call just means
[112:54] the features right uh
[112:58] they're just talking about
[112:59] >> I understand what a feature means in an
[113:01] essay but like how is an atom different
[113:04] from a feature? Uh it's just a feature
[113:07] but in this fancy is that the atoms have
[113:11] something to do with the decoder in the
[113:13] SA
[113:15] I
[113:15] >> I think it's just the the feature is
[113:18] just it's a more general term where it
[113:20] doesn't
[113:20] >> Oh yeah no I think of the decoder I
[113:22] think it's the the atom here is talking
[113:25] about
[113:27] like the direction of the decoder for
[113:29] that feature. Yeah.
[113:32] >> Okay.
[113:35] Uh
[113:37] so so they saying that so they're saying
[113:42] atoms are broadly shared.
[113:46] Okay. Okay. So money for shattered and
[113:48] cross unrelated groups.
[113:50] Uh
[113:52] yeah I'm not pretty sure I that what
[113:54] that means exactly either but okay. So
[113:57] you have lo K. So we have a lot of
[114:00] sparity. Uh let's say only three fe
[114:04] three features or whatever. Uh
[114:09] I guess if you have
[114:12] more
[114:22] money for sub. Okay. I think you have
[114:26] low. Okay. I think I understand. If you
[114:28] have okay, you have a bunch of
[114:29] manifolds. It's just saying that there
[114:32] is like few enough features that uh the
[114:36] multiple
[114:37] uh manifolds share a single feature
[114:43] like this is the the regime where it's
[114:47] here like where you uh I'm guessing it's
[114:50] like where K is one or two or whatever.
[114:57] Okay, got it. I think
[114:59] >> so like you have very few features so
[115:03] you can't represent different manifolds
[115:04] with them. So you just have some feature
[115:06] that points different manifolds,
[115:08] >> right?
[115:10] >> Got it.
[115:11] because the okay so they has to be very
[115:14] low because the they say at intermediate
[115:17] K and it says about a compact set of
[115:20] atom spans each manifold thing and here
[115:23] that K is like uh four right so that
[115:28] means they they mean this this part
[115:30] where the support is low so it has to be
[115:33] basically one and two and maybe three so
[115:37] probably like one and two so of course
[115:40] It it makes sense to me that if you have
[115:43] uh K2
[115:45] in some top K SAE, you are not going to
[115:50] have representations for all of the for
[115:53] each manifold because you basically
[115:55] barely have any feature any features
[115:56] like a very high sparity. So you have to
[116:01] represent a lot of the activations in
[116:04] basically combinations of two features.
[116:08] Uh so you have like more than too many
[116:10] falls you are yeah you can't like it's
[116:14] not possible for you to represent them.
[116:19] >> Got it.
[116:20] >> Like I guess it's even less than the
[116:25] like it's even less than the than the
[116:28] ambient dimension of a lot of these
[116:30] manifold. So
[116:37] anyway, anyone else has anything to talk
[116:40] about or ask or whatever.
[116:55] Uh maybe if anyone wants to play some
[116:58] more high level as in do you see future
[117:00] in this unsupervised manifold discovery
[117:03] or making their more broadly because I
[117:06] personally do but I think it should be
[117:08] generalized to also uh non-ontinuous
[117:12] manifolds.
[117:14] I'm not sure if there's a ter for that
[117:16] >> is that I mean is that
[117:18] >> manifolds
[117:20] >> I guess
[117:22] that like maybe turns out you just can't
[117:25] think of the that as just a bunch of
[117:27] separate manifold right uh
[117:29] >> yeah yeah non how do they say it
[117:31] nonlineer the thing that I really liked
[117:35] >> yeah but
[117:36] >> so a set of potentially disconnected
[117:38] nonlinearly curved geometric structures
[117:41] with co you should say Oh yeah, that is
[117:46] general enough for me. It is so general
[117:49] that it might be.
[117:52] >> Yeah, I guess but if it's I I think
[117:55] >> we're getting into the
[117:56] >> There are reasons why you since you
[117:59] might expect things to be locally l for
[118:01] there's a bunch of machine learning
[118:03] reasons why people say that maybe like
[118:06] locally linear. So like you uh maybe can
[118:11] expect things to even if it's actually
[118:13] like that to de compon
[118:18] uh line are you know uh
[118:23] many falls the same way that the many
[118:25] falls decomposing a bunch of uh lines of
[118:29] like you know linear linear lines.
[118:34] It's not slice. It's like they they call
[118:36] them rays which I think is makes more
[118:39] sense because like it's not like the
[118:42] thing that the linear probes
[118:45] do do are like the pro are like
[118:49] onedimensional metaphors because that's
[118:51] a curve. Uh it's actually more specific
[118:55] than that. It's like uh some direction
[118:58] starting from the origin and you know
[119:01] going to the well it doesn't necessarily
[119:02] for the origin because it have a bi but
[119:04] yeah some some direction on the on on
[119:08] the space
[119:10] uh and it can't be a line that's like
[119:14] not uh on that like you can imagine some
[119:18] other cor line or whatever on that other
[119:21] direction interface that's not a
[119:24] direction
[119:26] from there. I don't know.
[119:32] >> Can can you can you try to uh
[119:35] >> I not sure I need to think more about
[119:36] that because I'm not sure I was saying
[119:38] it's completely censical.
[119:39] >> Wait, platonic something they said
[119:42] uh if you grew up a bit.
[119:45] >> Oh,
[119:49] I was going to scroll a bit for the
[119:51] through the appendices. There's
[119:52] something interesting but
[119:56] this is an interesting section
[120:00] compare land.
[120:12] But do they mean here defense person go
[120:15] on different models which is or they or
[120:18] the same model.
[120:24] >> I guess they mean the same model because
[120:26] otherwise it makes sense to say the same
[120:28] manifold.
[120:30] I mean you could when I think when
[120:33] people talk about platonic something
[120:34] they I they usually they I I they I
[120:38] think they are not using it the way
[120:39] other
[120:40] >> different models but they're studying
[120:42] the same model in this section.
[120:44] >> Yeah that's sad. I I think it will there
[120:46] will be something interesting of looking
[120:48] if you I think you probably find the
[120:50] same very they tried on different models
[120:53] and I'm assuming they they don't mention
[120:55] something like you know we we didn't
[120:57] find the colors were shaped on this way
[121:02] on I I'm assuming that a that I don't
[121:04] know months is always a circle in all of
[121:07] the models so there's probably something
[121:09] to be found there but I guess it's not
[121:11] the thing they're testing Yeah.
[121:28] I try to certain color
[121:37] this this
[121:39] similarity.
[121:41] This by the way does have once on the on
[121:43] the diraphy
[121:56] cosmetry
[122:05] as direction points.
[122:33] Oh no m
[122:38] this for me.
[122:47] Nice picture. So,
[122:49] but I no idea what
[122:52] compressor motor equivalent is, but
[122:55] okay. Nice
[122:58] points make the shape
[123:18] I mean this so we were talking about
[123:20] early you know unsupervised finding of
[123:22] manifolds today
[123:25] Don't the like what about the
[123:27] neuroscience people? Do they do that on
[123:29] the brains?
[123:32] >> Uh I'm not sure. I don't know.
[123:35] >> Maybe close.
[123:36] >> Maybe we can we can extract their
[123:38] knowledge.
[123:40] >> Uh
[123:42] are there no
[123:45] if that's the case? Why are there no
[123:47] more like there should be at least a few
[123:50] neuroscience people getting into making
[123:52] term now be like oh we could just do
[123:54] this that why is there not
[123:57] enough that maybe they just already a
[124:00] lot of their of the 16 insights well
[124:04] already here but I don't
[124:36] All
[124:47] right.
[125:01] I have many falls.
[125:19] for instance.
[125:27] Okay.
[125:32] Okay. So this is basically just testing
[125:35] if the other thing is just testing if
[125:37] you can find some set of directions that
[125:40] captures the that that splines explains
[125:44] the the variance inside the manifold.
[125:50] not a measure of
[125:54] like there could be more that do other
[125:57] things like you could have more than one
[125:59] feature but as long as it's actually
[126:01] already doing the thing the the hard
[126:03] corate ones it's still as high on this
[126:06] metric I think
[126:09] uh
[126:38] Okay.
[126:40] I think
[126:49] I'm teaching someone how to read math
[126:51] instead of reading the buff with this
[126:53] paper up. So I will stop doing that and
[126:55] now I will read the math in this paper.
[126:57] Okay. Um let's go.
[127:09] Okay, I guess that's okay. So, I think
[127:11] yeah, we can the main paper thing is
[127:14] done.
[127:15] >> Uh, I guess we now can get to the part
[127:17] where we spend a lot of time read the B
[127:20] and try and understand it. But I don't
[127:21] know if we should record that or you
[127:24] know take the reading group presented by
[127:26] now and
[127:29] uh
[127:31] yeah I I I
[127:36] might also now now uh check some
[127:38] application before going deeper. So
[127:41] maybe
[127:42] >> Okay. Yeah, maybe we can just end end
[127:44] the reading group here then. Yeah, but
[127:47] uh
[127:50] >> if I go through the paper in detail, I
[127:52] can additionally record something and
[127:54] potentially post it on the YouTube
[127:56] channel.
[127:57] >> Yeah, you can do that if you want.
[128:00] >> But that's assuming I will go through it
[128:02] in a lot of detail, which because of
[128:05] other things I have to do, I'm not sure
[128:06] I will do that in the in full
[128:08] comprehensive way, but I I really want
[128:10] to.
[128:12] >> Okay. Yeah, I think we can get it. If
[128:14] you want to do that at some point, we
[128:17] can just uh you can feel free to record
[128:19] it and send me the video or something.
[128:21] I'll post it to the YouTube.
[128:22] >> Yeah. Yeah. I think I'll first now do
[128:25] some application though. Uh because
[128:28] deadlines
[128:29] >> uh anyone else wants to comment
[128:32] something or something or do I just end
[128:34] the recording now?
[128:38] We going for two hours which is the
[128:40] typical reading group time these days.
[128:44] >> That's actually uh I think lower than
[128:47] the average actually.
[128:49] >> Yeah, probably.
[128:51] >> Yeah. I mean we are not the only one the
[128:54] data that long you see the for exampic
[129:01] server they also
[129:02] >> yeah I'm basically right now trying to
[129:04] teach someone the machine talk how to
[129:06] read math in papers I need to stop doing
[129:09] that and actually read the math in his
[129:10] paper instead why am I so sniped
[129:13] constantly I don't know uh whatever uh I
[129:15] have is the answer but
[129:17] >> I have to do me interpret a bit on my
[129:19] psychology because uh whatever Uh that
[129:22] is a rabbit hole. Yes. So
[129:25] >> yeah. And I guess nobody say anything so
[129:28] I just stop the recording. I guess.
[129:35] All right. Thank you guys.
[129:39] Goodbye.
