Softmax Replaces AI Values Alignment With Multi-Agent Emergence Math
Ai4
Top-down values alignment inevitably fails as models scale, requiring AI safety to evaluate information-theoretic emergence and constituent agent welfare modeled on biological symbiosis.
## Background
Published by Ai4 on YouTube on August 26, 2025, this presentation features Adam Goldstein, Co-Founder and Head of Research at Softmax. According to the description, Goldstein outlines the mathematics behind organic alignment—a biologically inspired approach that moves away from control-and-enslavement paradigms toward symbiotic multi-agent intelligence where parts form thriving wholes.
## TL;DR
Goldstein argues that mainstream values alignment systematically fails because it relies on top-down coercion rather than peer-level symbiosis. He introduces organic alignment as a two-dimensional framework optimizing for both macro-level emergence and the measurable well-being of constituent parts. To quantify these dynamics mathematically, he presents Partial Information Decomposition (PID) to detect synergistic information and Causal Emergence 2.0 to test macro-scale predictive power in discrete dynamical systems. Finally, he outlines Metagrid, an open-source multi-agent simulation testbed that models open-ended cooperation, network sparsity, and generational lifespans.
## Atmosphere & style
The presentation carries an intellectually rigorous, academic, and deeply thoughtful tone, fulfilling Goldstein's opening promise of delivering an unapologetically nerdy talk. Goldstein speaks with articulate clarity and quiet enthusiasm, seamlessly connecting mathematical concepts like Markov chains and mutual information with intuitive biological models of cancer, slime molds, and cellular sparsity. The concluding Q&A transitions the session into a collegial, open-ended exchange where complex philosophical inquiries about agent happiness and mortality are addressed with candor and lighthearted humor.
## Key takeaways
- [00:04](https://www.youtube.com/watch?v=SmJ7KOAhyxQ&t=4s) Mainstream values alignment relies on a top-down reward-and-punishment control loop that systematically fails and compounds errors as AI models scale.
- [02:36](https://www.youtube.com/watch?v=SmJ7KOAhyxQ&t=156s) Organic alignment draws from multi-scale biology and symbiosis, proposing that humans and AIs should interact as non-hierarchical peers within mutually beneficial collective wholes.
- [05:50](https://www.youtube.com/watch?v=SmJ7KOAhyxQ&t=350s) Emergence alone is not inherently beneficial; higher-order structures can become parasitic or authoritarian, requiring alignment to optimize across both emergence and constituent well-being.
- [08:15](https://www.youtube.com/watch?v=SmJ7KOAhyxQ&t=495s) Partial Information Decomposition (PID) mathematically measures emergence by isolating synergistic mutual information that cannot be reduced to the sum of individual components.
- [14:00](https://www.youtube.com/watch?v=SmJ7KOAhyxQ&t=840s) Causal Emergence 2.0 demonstrates that coarse-graining complex micro-states into macroscopic partitions can provide stronger predictive power over future system states than tracking noisy individual parts.
- [20:48](https://www.youtube.com/watch?v=SmJ7KOAhyxQ&t=1248s) The welfare of constituent AI agents in multi-agent systems can be quantitatively measured and aggregated using established ML metrics such as loss, reward, surprisal, and free energy.
- [23:07](https://www.youtube.com/watch?v=SmJ7KOAhyxQ&t=1387s) Metagrid provides an open-source, open-ended simulation platform to test heterogeneous multi-agent cooperation and competition without the constraints of fixed-endpoint benchmarks.
- [29:58](https://www.youtube.com/watch?v=SmJ7KOAhyxQ&t=1798s) When individual components fail to benefit from a larger system, they adopt localized survival strategies that can fatally fracture the host organism, directly analogous to biological cancer.
- [30:58](https://www.youtube.com/watch?v=SmJ7KOAhyxQ&t=1858s) Sparsity in network connections—mirroring biological cells connecting to fewer than ten neighbors—is essential to prevent information overload in agent world-modeling.
- [32:00](https://www.youtube.com/watch?v=SmJ7KOAhyxQ&t=1920s) Metagrid incorporates bounded agent lifespans and mortality as an evolutionary mechanism, enabling generational adaptation to changing environments while inheriting learned traits.
- [33:45](https://www.youtube.com/watch?v=SmJ7KOAhyxQ&t=2025s) Organic alignment requires active mutual reciprocity between parts and the emergent whole, distinguishing it from scenarios where agents merely share independent, coincident preferences.
## Detailed walkthrough
### [00:04](https://www.youtube.com/watch?v=SmJ7KOAhyxQ&t=4s) Introduction and the Failure of Values Alignment
Opening what he promises to be "one of the nerdier talks you'll hear at this event," Adam Goldstein introduces the mathematics behind organic alignment. Following up on the morning keynote delivered by his co-founder and Softmax CEO Emmett Shear on the philosophy of the concept, Goldstein shifts the focus to actionable quantitative methods: defining the math, measuring it, and applying it directly to AI systems. He frames the central problem around the inevitable failure of mainstream AI safety, widely known as "values alignment."
Goldstein breaks down the dominant paradigm into three straightforward steps: humans decide what an AI should say or do, reward desired behavior while punishing deviations, and hope this feedback loop magically yields a docile machine that does everything humans want "without complaint." Despite consuming nearly a decade of mainstream AI capabilities and safety research, this control-and-enslavement mindset has compiled an "atrocious" track record. As models scale, they repeatedly demonstrate behaviors entirely outside their training guardrails and contrary to human intent, with failure modes only compounding over time.
- **The Values Alignment Loop:** Specify desired behavior $\rightarrow$ Reward compliance / punish deviation $\rightarrow$ Hope for an obedient system that never deviates.
- **The Core Problem:** Mainstream values alignment treats AI through a paradigm of top-down control and enforcement, which systematically breaks down in practice.
```mermaid
flowchart LR
accTitle: The Traditional Values Alignment Loop
subgraph Traditional["Mainstream Values Alignment"]
A["1. Define Desired Behavior"] --> B["2. Reward Compliance / Punish Deviation"]
B --> C["3. Hope for Flawless, Obedient Execution"]
C -.->|"Repeated Failures in Practice"| A
end
```
---
[02:36](https://www.youtube.com/watch?v=SmJ7KOAhyxQ&t=156s) The Organic Alignment Alternative: Symbiosis Across Scales
Instead of enforcing top-down control, organic alignment frames intelligence around group problem-solving. Rather than commanding individual models, researchers measure whether groups of AIs form coherent, functional wholes—analogous to a human organization or cells coalescing into a living tissue—and evaluate whether that higher-level entity benefits or harms the individual constituents. The ultimate ambition is to construct symbiotic groups of humans and AIs operating as peers without masters or slaves, cooperating to solve collective challenges where all participants benefit.
Goldstein draws this conceptual foundation from evolutionary biology and multicellular life. Slime molds, for instance, consist of single-celled organisms that assemble into collective structures to navigate complex physical and chemical environments beyond the reach of any single amoeba. Symbiosis provides an even stronger parallel: human cells perish in environments with a pH of 1 or 2, but gut microbiota thrive in that acidic environment, breaking down food and unlocking nutrients human biology cannot process alone. Drawing on the work of developmental biologist Michael Levin—with whom Goldstein studied and co-founded an incubator prior to launching Softmax—he stresses that intelligence is scale-free: it exists in single cells, tissues, organs, bodies, societies, and entire ecosystems.
- **Organic Alignment Premise:** AI agents solve problems collectively as peers, forming emergent functional organizations that mutually benefit both human and machine participants.
- **Biological Precedents:** Slime mold aggregation and human–microbiome symbiosis demonstrate how vastly different entities cooperate without centralized coercion.
- **Levin's Multi-Scale Intelligence:** Intelligence has neither an absolute floor nor an absolute ceiling; minimal cognition in cells scales up into tissues, organisms, and social groups.
---
[05:50](https://www.youtube.com/watch?v=SmJ7KOAhyxQ&t=350s) Emergence and the Two-Dimensional Alignment Space
In complex systems, emergence describes the phenomenon where small components (such as atoms or molecules) form higher-order structures (such as cells or bodies) without the underlying parts disappearing. Goldstein highlights that an emergent system can be analyzed from either level: one can inspect the collective organism as a whole or reduce it to its underlying parts. Crucially, however, emergence in itself does not imply a positive outcome.
Emergence can be deeply destructive. In nature, parasitic fungi hijack an ant's brain to force the insect to act exclusively in the fungus's reproductive interest—a novel collective dynamic that neither organism could manifest alone, but one fatal to the host. Similarly, human social systems can produce authoritarian regimes or toxic ideologies that actively suppress or liquidate their individual members. Consequently, organic alignment cannot settle for mere emergence; it requires a two-dimensional optimization space where a system exhibits both strong emergence and net-positive well-being for its constituent parts.
```mermaid
flowchart TD
accTitle: The Two Axes of Organic Alignment
subgraph Space["Organic Alignment Framework"]
axisY["Axis 1: Emergence (Whole > Sum of Parts)"]
axisX["Axis 2: Well-being (Benefit to Constituent Parts)"]
Ideal["Target: Upper-Right Quadrant\n(Strong Emergence + Mutual Thriving)"]
Parasitic["Failure Mode: Emergent Parasitism / Exploitation"]
Isolated["Failure Mode: Disconnected Parts / No Synergy"]
- **Definition of Emergence:** The whole exhibits macroscopic behaviors and properties while its underlying micro-components remain intact and observable.
- **Pathological Emergence:** Parasitic fungus-ant relationships and oppressive social systems prove that higher-order emergence can actively harm the parts.
- **The Two Crucial Axes:** True organic alignment requires navigating to the upper-right quadrant: maximizing *emergence* while maximizing *part-level benefit*.
---
[08:15](https://www.youtube.com/watch?v=SmJ7KOAhyxQ&t=495s) Measuring Emergence: Partial Information Decomposition (PID)
To mathematically quantify emergence, Goldstein points to information theory, specifically Partial Information Decomposition (PID). The core inquiry of PID is determining whether a collective whole contains information beyond what can be extracted from its individual components. To illustrate, Goldstein uses the behavior of bird flocks across three distinct regimes:
1. **Redundant Dynamics:** If every bird flies in the identical direction at the exact same velocity, inspecting a single bird provides complete knowledge of the entire flock. The information is entirely redundant.
2. **Independent/Unique Dynamics:** If every bird flies in random directions independently, the group possesses no macro-level structure, and information remains strictly unique to individual parts.
3. **Synergistic Emergence:** When birds interact dynamically, making decentralized group-level steering decisions, the flock's trajectory cannot be deduced by examining any individual bird in isolation.
In PID mathematics, the mutual information between source variables $X, Y$ and a target variable $T$ is exhaustively decomposed into three mutually exclusive components: redundant information, unique information, and synergistic information. A system exhibits genuine emergence only when synergistic information is high, meaning the state of the macroscopic whole is irreducible to the sum of its individual parts. Because PID is grounded entirely in standard information theory, its equations can be applied universally to biological cells, human teams, or networks of large language models.
- **Information Types in PID:**
- **Redundant Information:** Information present across multiple individual parts simultaneously (e.g., synchronized flocking).
- **Unique Information:** Information contained exclusively within a single component (e.g., uncoordinated, purely autonomous agents).
- **Synergistic Information:** Information that emerges strictly from the interaction of multiple parts and cannot be recovered from any component in isolation.
- **Criterion for Emergence:** Higher synergistic mutual information directly reflects a more coherent, irreducible macroscopic whole.
---
[14:00](https://www.youtube.com/watch?v=SmJ7KOAhyxQ&t=840s) Causal Emergence 2.0 and Macro-Level Predictive Power
The second mathematical pillar discussed is Causal Emergence 2.0, developed in research led by Erik Hoel (a long-time collaborator of Integrated Information Theory founder Giulio Tononi) and supported by Softmax. While PID focuses on static information content, Causal Emergence centers on causality: evaluating whether coarse-graining a system into a macro-level description yields superior predictive power about its future states compared to tracking all micro-level details.
To explain the mechanics, Goldstein introduces discrete dynamical systems modeled as Markov chains with transition likelihoods across state spaces. In a sample eight-state Markov chain, some micro-states transition deterministically, others transition probabilistically to multiple states, and some act as terminal absorbing states (like remaining stuck in state two forever). Because Markov chains are computationally universal—capable of expressing any dynamical or computational system much like a Turing machine—grouping fine-grained micro-states into macroscopic partitions allows researchers to measure whether causal models at the macro scale outperform noisy, low-level micro descriptions.
- **Core Question of Causal Emergence:** Does tracking the aggregate whole provide stronger causal predictive power than tracking the micro-level parts?
- **Computational Universality:** Markov chains with transition matrices can formally represent any dynamical computational process.
- **State Space Partitioning:** By grouping micro-states into macro-states, one can mathematically identify whether causal mechanisms emerge at a higher level of abstraction.
[16:25](https://www.youtube.com/watch?v=SmJ7KOAhyxQ&t=985s) Causal Emergence and Coarse-Graining
Adam Goldstein details how coarse-graining works by systematically grouping fine-grained micro-states into broader macro-states. Using an illustrative system, he demonstrates how coloring individual states step-by-step—first linking state 7 with state 4, then incorporating state 6 and state 5—aggregates the graph into fewer, merged state sets. The primary objective is to evaluate whether grouping these states together mathematically yields higher predictive power regarding the system's future state compared to tracking each micro-state independently.
```mermaid
flowchart LR
accTitle: Coarse-Graining Process in Causal Emergence
subgraph Micro["Micro-State Level"]
direction TB
S1["State 4"]
S2["State 7"]
S3["State 6"]
S4["State 5"]
end
subgraph Macro["Coarse-Grained Macro-State"]
M1["Grouped Macro-State<br/>(Blue Cluster)"]
end
subgraph Predict["Predictive Outcome"]
P1["100% Deterministic<br/>Future Prediction"]
end
Initial grouping steps significantly raise predictive certainty; for instance, grouping three states allows predicting the next state with 75% probability, rapidly ascending to 100% certainty once all interconnected states are grouped together under a unified macro-state. Beyond this threshold, further coarse-graining yields no incremental gain in predictive ability. This optimal partition marks the core thesis of causal emergence: an elegant, path-finding-like algorithm rooted in information-theoretic causality that identifies the precise level of abstraction best suited to predict a system's future.
- **Conceptual Basis:** If treating a complex system as a unified whole allows a observer to predict its future better than observing individual components, it serves as a robust scientific foundation for asserting that the system is acting as an emergent whole.
- **Human Analogy:** Humans interact with other people as singular entities rather than clusters of billions of cells because the macro-level abstraction dramatically simplifies prediction and interaction.
- **Decision Rule:** When holistic grouping improves future state predictions, macro-level emergence exists; if predictive accuracy declines, the analytical focus must remain strictly on the component parts.
[20:48](https://www.youtube.com/watch?v=SmJ7KOAhyxQ&t=1248s) Measuring the Well-Being of Parts in Multi-Agent AI
Shifting focus from measuring emergence to evaluating component welfare, Goldstein presents a straightforward approach: existing machine learning metrics can directly evaluate whether AI agents "benefit" from system integration. Machine learning already utilizes quantitative standard metrics—loss, reward, free energy in active inference models, or surprisal—to assess whether an individual model is functioning effectively. Assessing collective well-being requires calculating these exact metrics across an entire population of models rather than evaluating them in isolation.
Goldstein warns against assuming that emergent system behavior inherently benefits its individual components. Emergent architectures can just as easily give rise to parasitic or dictatorial structures where high-level system coordination occurs at the explicit expense of the individual agents. Evaluating part-level metrics ensures that emergent organization aligns with the well-being of the constituent models.
- **Aggregation Metrics:** Evaluating group welfare requires adopting specific social philosophy frameworks, such as measuring average population loss, tracking the performance of the worst-off agent, or measuring distributional variance.
- **Agent Types:** Multi-agent evaluation techniques apply universally across Large Language Models (LLMs), Reinforcement Learning (RL) agents, and active inference systems.
- **Organic Alignment Definition:** Organic alignment is defined by the dual criteria of high macro-level emergence coupled with positive welfare outcomes across the constituent parts.
[23:07](https://www.youtube.com/watch?v=SmJ7KOAhyxQ&t=1387s) Multi-Agent Simulation with Metagrid
To evaluate multi-agent dynamics and test whether AI models successfully form mutually beneficial emergent wholes, researchers require rich simulation environments. Goldstein notes that standard multi-agent frameworks—like PettingZoo or DeepMind's Melting Pot—frequently rely on fixed endpoints and predefined objectives that do not reflect open-ended real-world conditions. Softmax sponsors and actively contributes to Metagrid, an open-source multi-agent simulation platform designed to emulate real-world environmental flexibility.
```mermaid
flowchart LR
accTitle: Metagrid Simulation Environment Capabilities
subgraph Agents["Heterogeneous Agents"]
direction TB
AG1["LLM Agents"]
AG2["RL Agents"]
AG3["Custom Rule Models"]
end
subgraph Environment["Metagrid Platform"]
E1["Arbitrary Map Topologies"]
E2["Open-Ended Interaction Dynamics"]
end
subgraph Behaviors["Observed Behaviors"]
B1["Cooperation"]
B2["Competition"]
B3["Neutral Coexistence"]
end
Metagrid enables researchers to construct custom maps of arbitrary complexity populated by flexible numbers of heterogeneous agents. The environment allows agents to compete, cooperate, or ignore one another, providing a clean testbed for measuring foundational skills (such as navigation) alongside complex multi-agent behavior metrics.
- **Infrastructure & Execution:** Metagrid runs efficiently on local hardware—such as lightweight laptop models—as well as scaled-up cloud clusters for large-scale training runs.
- **Architectural Agnostic:** Supports arbitrary agent backends, including custom-trained RL models, LLM-driven agents, or specialized language synthesis architectures.
- **Open Access:** All related research papers, frameworks, and Metagrid toolsets are fully open-source and released into the public domain without licensing restrictions.
[26:00](https://www.youtube.com/watch?v=SmJ7KOAhyxQ&t=1560s) Q&A: Decomposing Information and Defining Agent Happiness
During the Q&A session, an audience member asks whether the mathematical breakdown of information refers to semantic decomposition. Goldstein clarifies that it relies on mutual information—a fundamental statistical measure of correlation evaluating how well one vector or string of bits predicts another. The participant then questions whether evaluating "happiness" via loss functions is a validated metric, pointing out that happiness is an inherently subjective quality.
Goldstein acknowledges the subjective nature of happiness but highlights that human policy-making relies on proxy metrics like GDP per capita to make practical quantitative assessments. Because modern machine learning models are explicitly optimized around loss reduction, loss serves as a natural proxy for agent preference within the paradigm we have constructed.
> "The emergence math is the really interesting stuff... But as far as the multi-agent well-being math, that stuff you can just think about yourself: 'What does it mean to me for an AI to be happy?' If you know the answer for one AI, then you know the same answer for a group of AIs."
- **Information Measure:** Mutual information provides a purely statistical evaluation of mutual predictability between bit strings or embedding vectors.
- **Welfare Proxies:** Alternative metrics for agent preference include surprisal, free energy, or direct human feedback ratings.
[29:58](https://www.youtube.com/watch?v=SmJ7KOAhyxQ&t=1798s) Q&A: System Dynamics, Cancer Analogies, and Sparsity
An audience member asks what happens in modeled systems when small sub-groups of components fail to benefit from the broader structure. Goldstein references his background in cancer research, explaining that when components experience adverse conditions, they adapt by exploring new survival strategies. In biological systems, lung cells exposed to excessive smoking or liver cells exposed to alcohol experience conditions outside their normal parameters, forcing them into alternative survival behaviors that ultimately profit the localized cells at the severe detriment of the broader organism.
```mermaid
flowchart TD
accTitle: Pathological Fracture vs Healthy System Adaptation
subgraph Trigger["System Stress / Adverse Environment"]
T1["Localized Stress Factors"]
end
subgraph Adaptation["Cellular / Agent Strategy Shift"]
A1["Exploration of New Survival Strategies"]
end
subgraph Outcome["Systemic Impact"]
O1["Local Survival Gain"]
O2["Pathological Breakdown of Whole Body (Cancer)"]
end
Addressing a follow-up question regarding optimal architectural structures, Goldstein emphasizes the systemic necessity of sparsity. Hyper-connected networks make it difficult for individual components to parse system dynamics due to information overload.
- **Biological Sparsity:** Most cells in the human body maintain direct connections to fewer than 10 neighboring cells, which simplifies local world-modeling.
- **Multi-AI Safety Risk:** Without explicit monitoring of agent-level metrics, unaligned sub-groups in multi-AI ecosystems may default to localized survival strategies that fracture the stability of the entire system.
[32:00](https://www.youtube.com/watch?v=SmJ7KOAhyxQ&t=1920s) Lifespans, Mortality, and Generational Adaptation in Metagrid
Addressing the challenge of multi-agent inference as environments scale up, Adam notes that inference is straightforward when an agent's world is small, but becomes progressively harder as the population of other agents expands. An audience member connects an earlier mention of cancer to Robert Axelrod’s seminal work on cooperation—which eventually led Axelrod into cancer research—and asks how the dimension of time influences cooperation in open-ended simulations, noting that most multicellular organisms have finite lifespans (with rare exceptions like jellyfish or planaria).
```mermaid
flowchart LR
accTitle: Generational Adaptation Flow in Metagrid
subgraph S1["Agent Lifecycle"]
A["Active Agent Policy<br>(Fixed Weights)"] --> B["Operate for Lifespan<br>(~1,000 Timesteps)"]
end
subgraph S2["Generational Turnover"]
B --> C["Agent Termination & Succession"]
C --> D["Inherit Learned Traits &<br>Adapt to New Environment"]
end
```
Adam explains that time is handled deliberately in Metagrid by enforcing limited lifespans. Agents—defined by policies with a specific set of weights—persist for a set duration, such as a thousand timesteps, before dying and being replaced by successor agents. This architectural choice directly mirrors natural living systems, from cellular regeneration in the human body to generational turnover across children and grandchildren. Adam argues that mortality is an evolutionary feature rather than a flaw, because "sometimes the best way to adapt to a new environment is to start from scratch but to inherit some of the things that your parents learned."
- In Metagrid, agent policies run for bounded lifespans (roughly 1,000 steps) rather than existing indefinitely.
- The turnover mechanism balances fresh environmental adaptation with the inheritance of successful learned traits across generations.
---
[33:45](https://www.youtube.com/watch?v=SmJ7KOAhyxQ&t=2025s) Defining "Organic Alignment" vs. Mere Agreement
An audience member raises a conceptual question regarding the flocking analogy: if every bird in a flock possesses a preference to be alone and they all disperse outward from a central point, every bird achieves maximum happiness, but it seems counterintuitive to describe them as aligned. Adam clarifies the core distinction between conventional alignment and "organic alignment" by echoing a foundational question: "aligned to what?"
In organic alignment, parts must be aligned to the larger whole of which they are a constituent part. Healthy cells in a biological body are aligned only when working for the collective well-being of the organism, while the organism reciprocally sustains the cells; cancer represents cellular misalignment, where cells prioritize their individual proliferation at the expense of the whole body. While dispersing birds might be aligned in the trivial sense that their subjective preferences coincide, their collective behavior fails the test of emergence because they do not form or sustain a coherent higher-order entity.
| Concept | Agent State / Preference | Higher-Order Emergence | Organic Alignment? |
|:--- |:--- |:--- |:--- |
| **Dispersing Flock** | Agents share identical desires (wanting to be alone) | Fails to form or maintain a cohesive larger whole | **No** (Individual satisfaction without emergence) |
| **Cancerous Cells** | Cells optimize locally for their own growth | Actively damages and consumes the larger organism | **No** (Cellular misalignment / parasitism) |
| **Organic Alignment** | Agents coordinate mutually supportive actions | Whole thrives while simultaneously supporting parts | **Yes** (Mutual emergence and reciprocal benefit) |
---
[35:42](https://www.youtube.com/watch?v=SmJ7KOAhyxQ&t=2142s) Research Community and Open Collaboration
Adam concludes by emphasizing that organic alignment is an evolving paradigm and invites researchers to help refine its definitions, mathematical foundations, and frameworks. Softmax is building an active research community dedicated to developing novel AI architectures, defining emergence metrics, and running simulations within the Metagrid framework.
He encourages researchers with diverse perspectives across AI and adjacent scientific fields to reach out directly via email (`adam@softmax.com`) to explore research collaborations, while humorously asking vendors to refrain from sending software sales pitches.
## Notable quotes
> [00:04](https://www.youtube.com/watch?v=SmJ7KOAhyxQ&t=4s) one of the nerdier talks you'll hear at this event — Adam Goldstein
> [00:04](https://www.youtube.com/watch?v=SmJ7KOAhyxQ&t=4s) without complaint — Adam Goldstein
> [00:04](https://www.youtube.com/watch?v=SmJ7KOAhyxQ&t=4s) atrocious — Adam Goldstein
> [26:00](https://www.youtube.com/watch?v=SmJ7KOAhyxQ&t=1560s) The emergence math is the really interesting stuff... But as far as the multi-agent well-being math, that stuff you can just think about yourself: 'What does it mean to me for an AI to be happy?' If you know the answer for one AI, then you know the same answer for a group of AIs. — Adam Goldstein
> [26:00](https://www.youtube.com/watch?v=SmJ7KOAhyxQ&t=1560s) What does it mean to me for an AI to be happy? — Adam Goldstein
> [26:00](https://www.youtube.com/watch?v=SmJ7KOAhyxQ&t=1560s) If you know the answer for one AI, then you know the same answer for a group of AIs. — Adam Goldstein
> [32:00](https://www.youtube.com/watch?v=SmJ7KOAhyxQ&t=1920s) sometimes the best way to adapt to a new environment is to start from scratch but to inherit some of the things that your parents learned. — Adam Goldstein
> [33:45](https://www.youtube.com/watch?v=SmJ7KOAhyxQ&t=2025s) aligned to what? — Adam Goldstein
## Mentioned
Adam Goldstein, Softmax, Emmett Shear, Michael Levin, Erik Hoel, Giulio Tononi, Robert Axelrod, Partial Information Decomposition (PID), Causal Emergence 2.0, Integrated Information Theory, Metagrid, PettingZoo, DeepMind, Melting Pot
## Video description
Adam Goldstein, Co-Founder, Chairman, Head of Research, Softmax
Abstracts
The AI field is focused on a future where either we control the AIs or they control us. This is extremely dangerous and also completely unnecessary. In the living world, we see cells work together to form multicellular life, and different organisms work together in symbiosis, without any piece being "in control" of the rest. Inspired by such examples from biology, the field of organic alignment is charting an alternative approach to AI. In this talk I will focus on the math of organic alignment: understanding what it means (and how to measure) when parts form a larger whole, and how the whole affects the parts. I will provide an overview of recent research in emergence, learning, and compassion, and highlight open areas of research in the field. If you want to live in a world where AIs and humanity thrive together, this talk will help you understand what's already possible and how your research can fit in.
Softmax co-founder Adam Goldstein argues that conventional top-down AI values alignment is fundamentally unworkable, proposing instead an organic alignment model where AI agents and humans form symbiotic, self-organizing wholes. This approach evaluates safety along two quantitative dimensions: whether the collective forms an irreducible emergent whole, and whether that emergent organization directly benefits its constituent parts.
Failure of Coercive Alignment and the Symbiotic Alternative (0:04)
Goldstein characterizes mainstream AI values alignment—specifying rules, punishing deviations, and expecting complete obedience—as a top-down control paradigm that consistently breaks down as models scale. Drawing on developmental biologist Michael Levin's theory of scale-free intelligence, Goldstein asserts that cognition operates across multiple tiers, from single cells and slime molds to organisms and societies. Rather than enforcing master-servant relationships, organic alignment seeks to develop cooperative, peer-to-peer networks of humans and AI models that solve shared problems without centralized coercion, modeled on biological mutualism like the human gut microbiome.
The Dual Axes: Emergence and Constituent Welfare (5:50)
Emergence occurs when macroscopic properties appear in a system while its underlying micro-components remain intact. However, Goldstein warns that emergence alone does not guarantee a benign outcome. Natural systems frequently exhibit destructive emergence, such as parasitic fungi hijacking ant nervous systems for reproduction, just as human social systems can produce authoritarian regimes that suppress their members.
Consequently, organic alignment requires navigating a two-dimensional optimization space. A system must demonstrate both high emergence—where the collective whole is distinct from its isolated parts—and positive welfare for its individual participants. Systems that achieve high emergence at the expense of their components are classified as parasitic rather than aligned.
Quantifying Emergence via Synergy and Causal Power (8:15)
To evaluate emergence quantitatively, Goldstein points to Partial Information Decomposition (PID) and Causal Emergence 2.0. In PID, mutual information is split into redundant, unique, and synergistic components. A flock of birds or a network of language models exhibits genuine emergence only when synergistic information is high, indicating that the macroscopic state cannot be deduced by observing individual parts in isolation.
Complementing PID, Causal Emergence 2.0—developed in research led by Erik Hoel and supported by Softmax—models systems as Markov chains to test coarse-graining. By grouping micro-states into macroscopic clusters, researchers measure whether macro-level descriptions yield superior predictive and causal power over system trajectories compared to tracking noisy, fine-grained micro-states.
Welfare Metrics, Network Sparsity, and Generational Simulation (20:48)
Assessing component well-being in multi-agent environments relies on existing machine learning metrics—such as loss, reward, free energy, or surprisal—aggregated across populations using social welfare functions like average loss or the status of the worst-off agent. To test these interactions under open-ended conditions, Softmax contributes to Metagrid, an open-source simulation platform that models heterogeneous agents without fixed endpoints.
Metagrid incorporates biological design principles, including generational adaptation through bounded agent lifespans (typically around 1,000 timesteps) to refresh environmental learning while inheriting prior traits. Goldstein also highlights the systemic necessity of network sparsity, noting that biological cells typically maintain fewer than 10 direct connections; excessive connectivity can overwhelm agent modeling and trigger localized, cancerous survival strategies that destabilize the whole system.
Goldstein presents organic alignment as an evolving research agenda rather than a settled solution, calling on the broader scientific community to refine its mathematical tools, test multi-agent dynamics in open-source environments like Metagrid, and establish robust proxies for agent well-being.
精炼阅读摘要
Softmax 联合创始人 Adam Goldstein 提出，传统自上而下的 AI“价值观对齐”范式在实践中系统性失效；他主张转向基于演化生物学与信息论的“有机对齐”（Organic Alignment），通过数学工具量化系统的高阶涌现与微观个体福祉，旨在构建人类与 AI 之间无主从关系的共生协作生态。
批判控制范式与确立双维对齐空间 (0:04)
Goldstein 认为，主流价值观对齐将研究重点置于“人类制定规范—奖惩反馈—寄望机器顺从”的控制闭环，实质上是一种单向奴役思维。随着模型参数规模扩大，系统频繁突破训练边界并违背人类意图，表明该机制具有系统性缺陷。受发育生物学家 Michael Levin 的无标度智能理论及黏菌、肠道菌群共生现象的启发，有机对齐主张将智能视为多层次的群体自组织，让人类与机器作为对等主体共同应对复杂挑战。
然而，群体层面的“涌现”（Emergence）并不必然带来良性结果。Goldstein 举例指出，寄生真菌操纵蚂蚁神经系统或人类社会出现极权统治，均属于高阶涌现结构严重摧毁微观个体的病态案例；而群鸟若仅各自散开，虽满足个体偏好却无法维持宏观结构。因此，真正的有机对齐必须在一个二维坐标空间中优化：系统既要实现宏观整体大于部分之和的强涌现，又要确保所有构成单元获得正向福祉，避免陷入系统对个体的寄生与剥削。
涌现的数学量化：协同信息与因果粗粒化 (8:15)
为了使涌现能够被定量计算，Goldstein 引入了部分信息分解（Partial Information Decomposition, PID）。在信息论框架下，系统的互信息被拆解为冗余信息、唯一信息与协同信息（Synergistic Information）。如果群鸟以相同速度沿同一方向飞行，信息完全冗余；如果各自随机运动，信息则完全独立；只有当智能体通过去中心化互动产生无法从任何单个个体中还原的宏观转向时，系统才表现出高协同信息。高协同信息是判定宏观整体不可还原为微观个体的核心数学依据。
在动力学层面，Softmax 支持了由 Erik Hoel 主导的“因果涌现 2.0”（Causal Emergence 2.0）研究。该理论利用具有计算通用性的离散马尔可夫链，通过对微观状态进行系统的“粗粒化”（Coarse-Graining）分组，评估宏观抽象是否比细粒度微观追踪具有更强的因果预测能力。在测试案例中，将多个微观状态聚类合并后，系统未来状态的预测确定性可从 75% 跃升至 100%。当针对系统的宏观描述能提供优于微观细节的预测表现时，便在数学上确立了因果涌现的存在。
智能体福祉代理与系统稀疏性机制 (20:48)
在衡量构成单元的利益时，Goldstein 指出无须重新发明标准，机器学习现有的定量指标——如损失函数、奖励信号、主动推理中的自由能（Free Energy）或惊奇度（Surprisal）——均可作为智能体偏好的有效代理，这类似于人类经济决策中将人均 GDP 视作福祉的统计代理。对群体福祉的评估则借用社会哲学框架，综合考察整个智能体群体的平均损失、处境最差者的表现以及分布方差，以防系统演化出以牺牲部分个体为代价的高效协调。
针对个体利益受损后的动态演化，Goldstein 结合其癌症研究背景指出，当细胞面临局部压力（如吸烟刺激肺部或酒精损伤肝脏）时，会探索替代生存策略，最终以整体崩溃为代价换取局部增殖；多智能体系统若缺乏微观监测，同样可能因局部子群体的自主生存策略而导致系统瓦解。此外，他强调拓扑“稀疏性”（Sparsity）是维持系统健康的关键：人体绝大多数细胞仅与不足 10 个邻近细胞保持直接连接，低连接度降低了信息过载，有助于智能体构建稳定的局部环境模型。
Metagrid 开放仿真与有限生命周期 (23:07)
为验证多智能体能否在没有集中强制的情况下形成良性共生，Softmax 资助并开发了开源仿真平台 Metagrid。与 PettingZoo 或 DeepMind 的 Melting Pot 等预设固定目标的基准不同，Metagrid 支持任意拓扑地图与异构智能体（涵盖大语言模型、强化学习模型与自定义规则智能体），能够模拟竞争、合作或共存等开放式动态，可在轻量级笔记本电脑及大规模云端集群上高效运行。
在应对大规模种群的推理复杂度与演化动力学方面，Metagrid 引入了有限生命周期机制。平台中智能体策略并非永久运行，而是在经历约 1,000 个时间步后终结，并由继任者接替。Goldstein 指出，这一设计借鉴了生物世代交替与细胞更新规律：死亡并非缺陷，而是一种关键的演化适应机制，使系统能够在继承亲代习得特征的同时，通过从零启动有效适应新的环境压力。
Goldstein 强调，有机对齐目前仍处于演化形成的初期阶段，其核心数学工具与平台代码均已开源。该范式依赖将算法损失作为智能体福祉的代理指标，未来仍需在更广泛的跨学科合作中检验其在异构人类-AI 复合系统中的有效性与边界条件。
Detailed notes — zh
## Background
本内容由 YouTube 频道 Ai4 于 2025 年 8 月 26 日发布，主讲人为 Softmax 联合创始人、董事长兼研究负责人 Adam Goldstein。视频简介指出，当前 AI 领域陷入了“非控制即被控制”的危险迷局，而有机对齐受生物学多细胞与共生现象启发，旨在通过量化涌现、学习与福祉的数学原理，探索人机协同繁荣的替代路径。
## TL;DR
Adam Goldstein 深入阐述了“有机对齐（Organic Alignment）”的量化数学框架，指出传统价值观对齐已走入死胡同。演讲围绕“涌现度量”（主信息分解 PID 与因果涌现 2.0 粗粒化）和“福祉评估”（多智能体收益分布与社会选择理论）两大数学支柱展开，剖析了健康共生与 AI 独裁、系统异化（如癌症机制）的本质区别。同时，他介绍了开源的 Metagrid 多智能体仿真平台与代际演化机制，呼吁跨学科研究者共同构建超越主奴控制范式的人机共生体系。
## Atmosphere & style
整场演讲呈现出高密度、严谨且硬核的技术学术基调，同时不乏生动风趣的演说魅力。演讲者在深入推导抽象信息论公式与马尔可夫链状态图之余，巧妙穿插黏菌、肠道菌群与癌症病理等生物学直观比喻，将复杂的数学概念落地为通俗易懂的直觉认知。现场互动思辨氛围浓厚，问答环节探讨了哲学定义与演化机制，整体传递出一种打破旧范式束缚、开创人机协作新图景的探索激情与开放姿态。
## Key takeaways
- [00:04](https://www.youtube.com/watch?v=SmJ7KOAhyxQ&t=4s) 传统的“价值观对齐”（Values Alignment）通过奖励与惩罚强制 AI 顺从人类，不仅难以消除意外行为，在逻辑上也已陷入死胡同；AI 对齐应转向从生物学与复杂系统汲取灵感的“有机对齐”（Organic Alignment）。
- [00:04](https://www.youtube.com/watch?v=SmJ7KOAhyxQ&t=4s) 有机对齐建立在两大数学支柱之上：第一是“涌现的数学”（评估分散个体何时构成功能整体），第二是“福祉的数学”（评估整体对局部组件是有益还是有害）。
- [01:23](https://www.youtube.com/watch?v=SmJ7KOAhyxQ&t=83s) 自然界中的多尺度智能（如黏菌协同、肠道菌群与人体的共生）表明，人与 AI 不应是主奴关系，而应以平等协作的伙伴身份在多智能体生态中实现共生与双赢。
- [05:50](https://www.youtube.com/watch?v=SmJ7KOAhyxQ&t=350s) 涌现本身并不等同于善好（例如僵尸真菌寄生或极权组织），真正的有机对齐必须同时满足“高涌现度（功能整体）”与“高局部福祉（个体受益）”两大维度。
- [08:15](https://www.youtube.com/watch?v=SmJ7KOAhyxQ&t=495s) 通过主信息分解（PID）框架，系统互信息可被划分为冗余信息、独有信息与协同信息（Synergistic Information），其中协同信息的占比是衡量多智能体系统产生高阶涌现的核心指标。
- [14:00](https://www.youtube.com/watch?v=SmJ7KOAhyxQ&t=840s) 因果涌现 2.0（Causal Emergence 2.0）借助状态粗粒化（Coarse-graining）将微观状态聚合为宏观状态；当宏观视角相较微观视角能显著提升因果预测能力时，系统在数学上即具备真实的因果涌现。
- [20:48](https://www.youtube.com/watch?v=SmJ7KOAhyxQ&t=1248s) 在评估多智能体系统的福祉时，必须防范牺牲个体利益的“AI 独裁”或“AI 寄生关系”，并可引入平均福祉、罗尔斯主义底线关怀（最差个体保障）及分布方差等多维度指标。
- [23:07](https://www.youtube.com/watch?v=SmJ7KOAhyxQ&t=1387s) 开源多智能体模拟平台 Metagrid 摆脱了传统基准的固定任务终点，支持自定义开放地图与多种智能体架构接入，用于验证和量化多智能体的因果涌现与有机对齐。
- [29:57](https://www.youtube.com/watch?v=SmJ7KOAhyxQ&t=1797s) 受损局部为了自身利益破坏整体健康的现象类似于生物学中的“癌症”，而网络连接的稀疏性（Sparsity）能够降低个体的局部建模负担，有利于维系宏观整体的健康稳态。
- [32:00](https://www.youtube.com/watch?v=SmJ7KOAhyxQ&t=1920s) Metagrid 模拟真实生命系统的有限寿命与代际更替，通过“局部重置重启 + 继承前代经验”的机制，帮助复杂系统以更优的适应力应对动态未知的环境挑战。
- [33:45](https://www.youtube.com/watch?v=SmJ7KOAhyxQ&t=2025s) 有机对齐特指局部与宏观整体之间的双向赋能；单纯满足个体孤立偏好但缺乏宏观功能结构涌现的行为（如四散飞离的鸟群），并不属于有机对齐。
## Detailed walkthrough
### [00:04](https://www.youtube.com/watch?v=SmJ7KOAhyxQ&t=4s) 破除“控制”迷局：有机对齐与量化数学的引入
在本次技术演讲中，Softmax 联合创始人兼研究负责人 Adam Goldstein 开宗明义地指出，这或许是全场技术密度最高、最硬核的一场分享。延续上午其联合创始人兼 CEO Emmett 关于“有机对齐（Organic Alignment）”哲学层面的探讨，本场演讲将完全转向数学与量化：如何度量局部何时能构成一个具有功能的整体、如何评估整体对局部带来的影响，以及如何将这些数学工具实际应用于 AI 的研究与工程中。Goldstein 指出，目前主流的 AI 对齐路径本质上是“价值观对齐（Values Alignment）”，但这套逻辑注定会失败；取而代之的，应该是从复杂系统与生物学中汲取灵感的“有机对齐”。
整个数学分析框架被清晰地拆解为两大核心轴：第一是**涌现（Emergence）的数学**，探究分散的个体在什么条件下会凝聚为一个更高阶的整体；第二是**福祉（Well-being）的数学**，评估当高阶整体形成后，它究竟是在滋养个体还是在压榨个体。为了降低技术门槛，团队还发布了一个开源项目，方便研究者轻松搭建多智能体（Multi-agent）模拟环境，以实际演练有机对齐的度量算法。
```mermaid
flowchart LR
accTitle: 有机对齐的双轴评估框架
subgraph S1["两大数学支柱"]
direction TB
E["涌现度量<br/>（部分是否构成功能整体）"]
W["福祉度量<br/>（整体对局部是有益还是有害）"]
end
subgraph S2["最终目标"]
O["有机对齐<br/>（人机共生与繁荣）"]
end
E --> O
W --> O
```
---
### [01:23](https://www.youtube.com/watch?v=SmJ7KOAhyxQ&t=83s) 价值观对齐的死胡同与人机共生的新图景
主流的价值观对齐方案设计得非常简单粗暴：首先由人类单方面决定我们希望 AI 展现何种行为（即定义何为“好 AI”），随后通过奖励合规行为、惩罚违规行为（如 RLHF 等机制）实施约束，最终期望 AI 能够“完美执行人类的一切吩咐且毫无怨言”。主流学界与工业界在这个方向上已经投入了将近十年，但其成效惨不忍睹。现实不断证明，AI 总会做出完全超出设计意图、甚至完全脱离训练轨道的意外行为，并且随着模型能力增强，失控风险只增不减。
与传统的“人类支配、奴役 AI”不同，有机对齐主张让 AI 智能体以群体协作的形式去解决问题，将研究重心放在系统整体的组织性与局部收益上。在这种构想下，人类与 AI 不存在主奴关系，而是以平等的协作伙伴身份，在多智能体生态中共同追求目标，实现双赢。Goldstein 用生物界生动的自然现象说明了这种关系：
* **黏菌（Slime Molds）：** 单个黏菌细胞能力极其原始，但当成千上万个细胞聚集在一起时，它们能够在物理与化学空间中协同寻优，解决任何单个个体绝对无法独立解决的复杂问题。
* **肠道菌群与人体的共生（Symbiosis）：** 人体组织非常脆弱，若置身于 pH 值为 1 或 2 的强酸环境中便会迅速溶解破坏；然而胃肠道内的共生菌群却最适应这种极端酸性环境。它们借此安家，同时协助人体分解难以消化的物质、萃取养分。人类并不需要与细菌以同一种方式思考或行动，但彼此之间形成的共生契约，让双方都活得更好。
* **多尺度智能（Multiscale Intelligence）：** 生物学家迈克尔·莱文（Michael Levin）的研究表明，智能在自然界是跨尺度连续分布的。从单个细胞微弱的自主性，到组织、器官、个体、人类群体，直至宏观生态系统，既不存在所谓“绝对最高等级”的智能，也不存在“零智能”的绝对底层。面对未来的超智能系统（Superintelligent AI），唯有学会跨尺度的协同融合，才能实现人机共存。
---
### [05:50](https://www.youtube.com/watch?v=SmJ7KOAhyxQ&t=350s) 涌现的双刃剑与有机对齐二维象限
在复杂系统科学中，“部分形成整体”的核心现象被称为**涌现（Emergence）**。之所以使用“涌现”而非“融合”，是因为在形成整体的同时，底层微观的局部并没有消失——由分子聚集而成了细胞，但分子依然存在；由细胞构成了人体，但底层细胞依旧各自运作。观察者既可以宏观审视系统整体，也可以微观剖析底层零件，这种“既见树木、又见森林”的双重视角正是涌现系统的本质。
然而，Goldstein 极其严肃地警告：“涌现并不等于善好。”自然界中充斥着病态和破坏性的涌现：
* **寄生真菌（僵尸蚂蚁）：** 寄生真菌入侵蚂蚁神经中枢，夺取其大脑控制权以达成真菌自身的繁殖目的。这种全新的行为模式是蚂蚁和真菌单方面都不具备的（属于典型的涌现现象），但对宿主蚂蚁而言却是毁灭性的灭顶之灾。
* **极权组织与极端意识形态：** 在人类社会中，由某个强权人物或极端教条主导的群体也会涌现出强大的组织力，但这往往伴随着对内部个体的压迫、剥削乃至肉体消灭。
```mermaid
flowchart TD
accTitle: 有机对齐二维评估象限
subgraph Q["整体性质象限图"]
direction TB
UR["【右上象限：有机对齐】<br/>高涌现度 + 局部受益（共生繁荣）"]
UL["【左上象限：寄生/压迫】<br/>高涌现度 + 局部受损（僵尸真菌/极权系统）"]
LL["【下半部分：无序状态】<br/>低涌现度（机械混杂或各自为政）"]
end
UL --- UR
LL --- UL
```
单纯追求系统能力的涌现是非常危险的。一个合格的“有机对齐”系统，必须落在**高涌现度（形成强有力的功能整体）**与**高局部福祉（各组成部分切实受益）**并存的象限内。
---
### [08:15](https://www.youtube.com/watch?v=SmJ7KOAhyxQ&t=495s) 度量涌现其一：主信息分解（PID）框架
为了用数学语言精确衡量“整体何时大于部分之和”，Goldstein 深入探讨了**主信息分解（Partial/Principal Information Decomposition, PID）**。这是一套根植于经典信息论、建立在互信息（Mutual Information）之上的数学框架。
假设我们试图预测一个目标变量 $T$（例如鸟群整体的运动趋势），而系统由多个输入源变量（如鸟群中的个体 $X$ 和 $Y$）组成。PID 能够将源变量与目标变量之间的互信息，严格且穷尽地分解为三种互斥的信息形态：
1. **冗余信息（Redundant Information）：** 仅观察 $X$ 或仅观察 $Y$，都能独立获取关于 $T$ 的全部信息。例如在排成固定队形、以完全相同的速度朝同一方向飞行的机械鸟群中，每一只鸟的行为完全一样，任何一只鸟都包含整个群体的信息。这属于高度冗余，不存在真正意义上的整体智能。
2. **独有信息（Unique Information）：** 预测 $T$ 所需的关键信息仅存在于某一个特定局部节点中（例如整个群体完全受唯一一只领头鸟的动作驱动，其余成员无关紧要），或者所有个体完全随机乱飞、各自为政。这种情况下整体同样可以直接还原为微观局部，并未发生高阶涌现。
3. **协同信息（Synergistic Information）：** 想要预测目标 $T$，单看 $X$ 或单看 $Y$ 都无法获知，唯有同时综合考量多个部分之间的互动关系时，这部分信息才会显现。以真实自然的鸟群为例，整体飞行的转弯或规避往往是群体动态协商的结果，整个系统知道的信息，任何单只鸟在微观上都无法独立全知。
协同信息的占比，正是衡量多智能体系统是否真正产生涌现的核心数学指标。无论系统是由碳基细胞、多大语言模型（LLM）智能体还是人类群体构成，都可以套用这套通用的信息论公式进行无差别计算。
---
### [14:00](https://www.youtube.com/watch?v=SmJ7KOAhyxQ&t=840s) 度量涌现其二：因果涌现 2.0 与马尔可夫链状态聚合
第二套度量理论源于前沿学者埃里克·赫尔（Erik Hoel）团队提出的**因果涌现 2.0（Causal Emergence 2.0）**。不同于 PID 纯粹基于概率分布的信息熵分析，因果涌现的核心聚焦点在于“因果预测力”——即**从宏观（整体）视角观察系统，相比微观（局部）视角，是否能够赋予我们更强的预测和因果干预能力**。
为了展示因果涌现的运作逻辑，赫尔利用在计算上图灵完备的**马尔可夫链（Markov Chain）**作为动力学系统模型：
* 系统在多个离散状态（State）之间随时间步推进演化。例如当系统处于某个状态（如状态 5）时，在下一时刻会依据转移概率跳转到其他相连状态；而某些吸收状态（如状态 2）则没有向外发散的箭头，系统一旦落入其中就会永远驻留。
* 传统的微观视角将每一个微观状态独立追踪，往往会因为转移概率的发散、局部噪声过大而导致预测熵增（因果模糊）。
* 因果涌现 2.0 提出的机制是：通过某种数学上定义的粗粒化（Coarse-graining）映射，将具有相似因果流向的微观状态集合归并为更高级别的“宏观状态”（在拓扑图上通过颜色聚合来区分）。如果经过这种宏观聚合之后，系统的因果决定论性与确定性显著提升，使得宏观模型比追踪每一个微观底层状态更加精确有效，那么在严格的数学意义上，这个宏观整体便具备了真实的“因果涌现”。这也为分析复杂多智能体协作网络提供了强大的图论与动力学工具。
### [16:25](https://www.youtube.com/watch?v=SmJ7KOAhyxQ&t=985s) 因果涌现与状态粗粒化：用宏观视角提升预测力
亚当·戈德斯坦（Adam Goldstein）通过演示一组状态图来解释因果涌现（Causal Emergence）的核心机制。他展示了当系统中的状态 7 与状态 4 一同被标记为蓝色时，其背后的概念是：“如果我们直接将状态 4 和状态 7 视为同一个状态，会发生什么？”随后，状态 6 和状态 5 也相继被归入这一集合。这一逐步将系统微观状态（Micro-states）合并为更少宏观状态（Macro-states）的过程，在信息论和物理学中被称为“粗粒化”（Coarse-graining）。当把大部分连通状态归为同一种颜色时，一个核心的数学问题随之浮现：将这些状态合并究竟能带来什么实际收益？
```mermaid
flowchart LR
accTitle: 状态粗粒化与因果涌现示意图
subgraph Micro ["微观状态 (Micro-states)"]
S4["状态 4"]
S7["状态 7"]
S6["状态 6"]
S5["状态 5"]
end
subgraph Coarse ["粗粒化 (Coarse-graining)"]
MacroBlue["单一宏观状态 (蓝色集合)"]
end
subgraph Outcome ["因果预测输出"]
Pred["预测确定性提升 (最高达 100%)"]
end
S4 --> MacroBlue
S7 --> MacroBlue
S6 --> MacroBlue
S5 --> MacroBlue
MacroBlue --> Pred
```
答案在于预测未来能力的变化。亚当强调了一个核心命题：“如果将这些状态聚合在一起，预测未来的准确率会比分开处理时更高还是更低？”在初始阶段，分散的微观状态使得系统的下一步跳转充满不确定性；但随着相关联的状态被合并为蓝色宏观状态，系统处于蓝色集合的概率迅速提升至四分之三，并最终达到 100% 的完全可预测性。然而，当所有相互连接的状态合并完毕后，继续将无关状态涂成蓝色便不再带来任何额外的预测增益。
- **因果涌现的判定标准**：通过类似寻路算法的分区/粗粒化操作，寻找能够使未来状态预测能力达到最大化的宏观划分。
- **算法优雅性**：该方法逻辑清晰、计算直观，让初级计算机科学家也能轻松理解，且原生内嵌了严格的信息因果度量。
---
### [19:36](https://www.youtube.com/watch?v=SmJ7KOAhyxQ&t=1176s) 整体与部分：为什么将系统视为整体能简化交互
因果涌现为科学界定义“系统是否作为一个整体运作”提供了坚实的数理依据。亚当打了个生动的比方：“我把你当成一个‘人’而不是‘一堆细胞’来对待，是因为这样能让我与你的互动变得简单得多。”虽然从物理层面上看两者皆属事实，但宏观视角的认知成本更低、解释力更强。
因果涌现的哲学与科学逻辑正是如此：若将系统视为一个整体能够更准确地预测其未来行为，那么将其确立为一个独立涌现的整体就是极具价值的科学假设。反之，如果将系统打包为整体并没有提升预测精度，就应当退回到微观视角，继续关注各个分散的零部件，因为此时系统并未产生超越部分的因果涌现。这种基于因果预测力的方法，提供了一种独立于互信息（Mutual Information）之外的精妙涌现度量框架。
---
### [20:48](https://www.youtube.com/watch?v=SmJ7KOAhyxQ&t=1248s) 多智能体福祉评估：警惕“AI 独裁”与寄生关系
在确立了涌现的度量后，亚当转向有机对齐的第二个核心支柱——如何衡量底层各个部分（智能体）的福祉（Well-being）。他风趣地提醒台下听众：“如果前几张幻灯片让你昏昏欲睡，现在可以醒醒了。”在现有的机器学习体系中，早已存在衡量单个 AI 表现或“幸福感”的成熟数学工具，例如损失函数（Loss）、奖励值（Reward）、主动推理中的自由能（Free Energy），以及数学上的惊奇度（Surprisal）。评估多智能体系统的福祉，本质上就是将这些度量扩展至群体中的所有智能体。
```mermaid
flowchart TD
accTitle: 多智能体系统涌现与福祉分布评估
subgraph SystemMetrics ["宏观系统状态"]
Emergence["因果涌现 / 整体形成"]
end
subgraph Risks ["潜在失衡风险"]
Dictator["AI 独裁 (AI Dictatorship)"]
Parasite["寄生关系 (Parasitism)"]
end
subgraph Aggregation ["社会选择聚合视角"]
Avg["群体平均值 (Average)"]
Rawls["最差个体保障 (Worst-off)"]
Var["分布方差 / 中位数 (Median/Variance)"]
end
Emergence -.可能导致.-> Risks
Emergence --> Aggregation
```
亚当特别发出警告：绝不能想当然地认为“只要系统产生了涌现，AI 就一定会获得更高奖励或更低损失”。现实中完全可能出现牺牲个体利益的“AI 独裁”（AI Dictatorship）或“AI 寄生关系”。因此，必须显式监测每一个组件的福祉变化。面对不同智能体福祉变化幅度与方向各异的情况，研究者需要借鉴社会哲学长久以来的思考，选择特定的聚合视角：
- **平均福祉视角**：监测整个智能体种群的平均损失或平均奖励。
- **底线关怀视角（罗尔斯主义）**：以群体中处境最差（Worst-off）的智能体状态作为衡量整体健康度的基准。
- **分布与方差视角**：评估中位数水平及个体间收益的方差波动。
---
### [23:07](https://www.youtube.com/watch?v=SmJ7KOAhyxQ&t=1387s) Metagrid 仿真平台：超越传统端点基准的开放世界测试
为了让研究人员验证算法能否促进有益的因果涌现，亚当介绍了多智能体模拟环境。他澄清道，这里的“智能体”指的是能够自主行动的 AI，远早于如今专指聊天机器人的狭隘定义。传统的强化学习框架（如 Melting Pot、PettingZoo）往往预设了固定的任务终点，而真实世界的问题极少具有明确边界。为此，Softmax 赞助并深度参与了开源项目 **Metagrid**。
Metagrid 旨在模拟生命系统的复杂互动，其核心特性包括：
- **无固定标准解**：支持构建任意复杂度的自定义地图，模拟真实世界的开放环境。
- **自由的互动模式**：支持任意数量的智能体自由选择合作、竞争或彼此忽略。
- **架构完全解耦**：无论是大语言模型（LLM）、自研或内置的强化学习（RL）智能体，还是主动推理与语言合成模型，均可无缝接入评测。
- **高效与轻量化**：既支持在个人笔记本电脑上高效运行超轻量模型，也能在云端集群进行大规模分布式训练。
- **多维度评估指标**：同时覆盖智能体的新环境导航基础技能，以及群体涌现与有机对齐行为指标。
---
### [26:01](https://www.youtube.com/watch?v=SmJ7KOAhyxQ&t=1561s) 总结与呼吁：价值观对齐的死胡同与有机对齐的生机
亚当断言，现行的“价值观对齐”（Values Alignment）从现有证据来看已经走入死胡同，没有任何迹象表明它会在未来突然奏效。相比之下，“有机对齐”（Organic Alignment）植根于生命系统乃至人类肉身在自然界演化共存的底层规律，具备更高的可行性。
他强调，有机对齐绝非“空中楼阁式的嬉皮士空想”（pie in the sky hippie thinking），而是拥有扎实的数学基础与全套免费的开源工具。演讲中提到的所有学术论文、数学框架与 Metagrid 项目代码均已完全开源并进入公有领域，没有任何使用限制，他热情邀请参会者共同参与探索这一方向。
---
### [27:12](https://www.youtube.com/watch?v=SmJ7KOAhyxQ&t=1632s) 现场问答：互信息本质与“AI 幸福度”的度量争议
在问答环节，现场观众就信息分解与“AI 幸福感”的量化方式提出了疑问：
- **关于信息分解的数学实质**：观众提问这是否指语义层面的分解。亚当澄清，这里使用的是互信息（Mutual Information），本质上是统计相关性的一种高级形式。它衡量的是两组向量（无论是嵌入向量还是底层比特串）之间的相互预测能力。
- **关于“AI 幸福度”的度量有效性**：观众质疑主观的“幸福”能否用损失函数来代表。亚当类比指出，人类制定国家政策时也会使用人均 GDP 等定量指标，尽管其代表性常有争议。在 AI 领域，当前的范式就是将“降低损失”定义为对 AI 有益。研究者完全可以根据需要更换代理指标——无论是惊奇度、自由能，还是人类对输出的评分。群体福祉计算的难点不在于数学本身，而在于确立价值立场；真正具备深刻学术挑战性的是因果涌现本身的数学度量。
---
### [29:57](https://www.youtube.com/watch?v=SmJ7KOAhyxQ&t=1797s) 现场问答：癌症隐喻、边缘化群体与网络稀疏性优势
针对“当系统内部分组件未能在协作中获益时会发生什么”的提问，亚当分享了自己从事癌症研究的背景经验，并给出了深刻的生物学类比：
```mermaid
flowchart TD
accTitle: 边缘化组件的异化与癌症机制类比
Normal["健康稳态：细胞/智能体遵循群体规则协作"]
Stress["局部受损：长期吸烟(肺部) / 酗酒(肝脏) / AI任务恶化"]
Rogue["策略突变：受损单元探索自主生存策略"]
Harm["整体崩溃：局部获益但以牺牲宿主整体健康为代价 (癌症)"]
Normal --> Stress
Stress --> Rogue
Rogue --> Harm
```
在健康身体中，细胞各司其职；然而一旦面临极端环境压力（如长期吸烟之于肺部细胞、过度饮酒之于肝脏细胞），受损细胞为了生存与繁衍，会被迫探索超出常规的新策略。这些策略虽然短期内让局部细胞自身得以维系，却以牺牲宿主整体健康为代价演变为癌症。多智能体系统若忽视局部个体的福祉，也会发生类似的系统异化与撕裂。
在谈及最优网络结构时，亚当指出了**稀疏连接（Sparsity）**的关键价值：在一个全互联的过度稠密系统中，个体面临的信息过载会导致其极难建模整体环境。人体细胞在物理上通常仅与少于 10 个相邻细胞保持直接连接。这种稀疏性使得单元能够以极低的认知负担对局部世界进行建模，从而更稳定地维系宏观整体的繁荣。
### [32:00](https://www.youtube.com/watch?v=SmJ7KOAhyxQ&t=1920s) 多智能体推理与时间尺度：有限寿命与代际演化
在多智能体系统中，当环境规模较小、仅有少量智能体存在时，推断（inference）相对容易进行；但随着智能体数量的剧化增加，实现优质推断的计算与协调难度也会呈指数级上升。现场观众就此延伸提问，联想到政治学家罗伯特·阿克塞尔罗德（Robert Axelrod）关于合作演化的经典研究及其对癌症现象的探讨，并提出“时间跨度”（time horizon）在合作中扮演的核心角色——自然界中绝大多数多细胞生物都有其寿命限制（水母和涡虫等少数物种除外），而在 Metagrid 这类开放式模拟环境中，时间机制是如何设计并影响系统演化结果的。
```mermaid
flowchart LR
accTitle: 代际演化机制与环境适应
subgraph S1["当前代智能体"]
A["策略与权重 (Policy Weights)"] --> B["环境互动 (~1000 步)"]
B --> C["个体消亡 (Mortality)"]
end
subgraph S2["继承与重置"]
C --> D["经验与特征继承"]
D --> E["生成继任智能体"]
end
subgraph S3["宏观适应性"]
E --> F["更高效适应动态新环境"]
end
```
Adam Goldstein 详细解释了 Metagrid 中对时间与寿命的具象建模方式。在模拟系统内部，团队为人为赋予了智能体有限的生命周期：每个具有特定权重参数的策略智能体只会持续存活一段固定的时间步（例如约 1000 个时间步长），随后便会“死亡”并被其继任者所取代。这种设计直接借鉴了真实生命系统的底层规律——人体通过不断更新细胞来维持运转，人类社会则通过一代代的子孙后代实现跨越漫长时间尺度的代际更替。Adam 强调，生命系统之所以普遍演化出“死亡与寿命限制”，是因为有时候适应全新未知环境的最佳途径就是“重置并重新开始”，同时在这一过程中选择性地继承父辈所习得的宝贵经验。
- **推断瓶颈**：随着群体内智能体数量的增加，有效推断彼此意图与全局状态的复杂度急剧上升。
- **寿命设定**：在 Metagrid 架构中，策略网络（Policy Weights）通常被限制在约 1000 个时间步的生命周期内。
- **演化智慧**：有限寿命（Mortality）通过“局部清零重启 + 继承前代经验”的机制，为复杂系统提供了应对动态环境的最优适应力。
---
### [33:45](https://www.youtube.com/watch?v=SmJ7KOAhyxQ&t=2025s) 重新审视“对齐”的定义：从个体偏好满足到有机涌现
针对鸟群模型的直觉设想，有观众提出深度哲学思辨：假设鸟群中的每一只鸟从根本上都渴望“独处”，当它们从中心点同时向四周散开飞离时，每只鸟都如愿以偿获得了快乐，彼此的愿望也完全一致，但这能否被称为“对齐”（Alignment）？该问题直指“部分与整体之间如何对齐”的深层语义。
```mermaid
flowchart TD
accTitle: 有机对齐与单纯偏好一致的区别
subgraph Mode1["偏好一致（非有机对齐）"]
A1["个体目标: 渴望独处"] --> B1["行为: 各自四散飞离"]
B1 --> C1["结果: 满足个体快乐，但系统零涌现"]
end
subgraph Mode2["有机对齐 (Organic Alignment)"]
A2["部分与整体双向赋能"] --> B2["行为: 协同构成多细胞/宏观系统"]
B2 --> C2["结果: 涌现出具备生命力的有机整体"]
end
```
Adam 借用当天上午 Emmett 提出的核心设问——“对齐究竟是向什么对齐（Aligned to what）？”来拆解这一概念。他明确指出，Softmax 所倡导的“有机对齐”（Organic Alignment）特指**局部与它们所构成的更大整体之间的双向对齐**。以生物学为例：
1. **健康共生**：人体的细胞只有在为整个身体的福祉工作时才算对齐，而身体也必须反过来滋养并保护这些细胞。
2. **癌症失齐**：癌症本质上就是一种细胞层面的“失齐”（Cellular Misalignment）——癌细胞只顾自身无限增殖的利益，却以牺牲整个身体为代价。
对于“四散飞走的鸟群”，Adam 认为虽然它们在“都想独处”的个体偏好上达成了一致，但这种行为模式无法在宏观尺度上涌现（emerge）为一个更大的功能性整体。它们或许实现了“个体幸福感的满足”，但在系统涌现维度上是彻底失败的，因此绝不属于“有机对齐”。
- **对齐客体**：有机对齐要求局部组件对齐于包含它们的宏观整体，而非仅仅满足孤立的个体偏好。
- **癌症隐喻**：将癌细胞定义为典型的系统内失齐——以整体系统的崩溃为代价换取局部的局部利益最大化。
- **涌现准则**：缺乏更高层级结构涌现（Emergence）的偏好一致性，不能被归类为有机对齐。
---
### [35:42](https://www.youtube.com/watch?v=SmJ7KOAhyxQ&t=2142s) 总结与呼吁：共建有机对齐的研究社群
在演讲尾声，Adam Goldstein 强调“有机对齐”仍是一个处于奠基阶段的前沿新兴领域，目前的理论框架与术语定义均属于抛砖引玉，旨在为人工智能的安全与发展提供全新的范式参考，而非最终定论。
Adam 向与会者发出开放合作邀请，呼吁对有机对齐、新型 AI 架构设计、涌现评估指标以及 Metagrid 模拟平台感兴趣的研究人员通过邮件（`adam@softmax.com`）取得联系。Softmax 正在积极构建一个多元化的研究者社群，希望吸纳来自人工智能、生物学、复杂系统等不同学科视角的跨界探索者共同推进该领域的发展。最后，Adam 幽默地提醒大家不要发送推销软件的垃圾商务邮件，真诚欢迎所有志在探索“人机协同繁荣”未来的学者与同行深入交流。
## Notable quotes
> [05:50](https://www.youtube.com/watch?v=SmJ7KOAhyxQ&t=350s) 涌现并不等于善好。 — Adam Goldstein
> [16:25](https://www.youtube.com/watch?v=SmJ7KOAhyxQ&t=985s) 如果我们直接将状态 4 和状态 7 视为同一个状态，会发生什么？ — Adam Goldstein
> [16:25](https://www.youtube.com/watch?v=SmJ7KOAhyxQ&t=985s) 如果将这些状态聚合在一起，预测未来的准确率会比分开处理时更高还是更低？ — Adam Goldstein
> [19:36](https://www.youtube.com/watch?v=SmJ7KOAhyxQ&t=1176s) 我把你当成一个‘人’而不是‘一堆细胞’来对待，是因为这样能让我与你的互动变得简单得多。 — Adam Goldstein
> [20:48](https://www.youtube.com/watch?v=SmJ7KOAhyxQ&t=1248s) 如果前几张幻灯片让你昏昏欲睡，现在可以醒醒了。 — Adam Goldstein
> [20:48](https://www.youtube.com/watch?v=SmJ7KOAhyxQ&t=1248s) 只要系统产生了涌现，AI 就一定会获得更高奖励或更低损失 — Adam Goldstein
> [26:01](https://www.youtube.com/watch?v=SmJ7KOAhyxQ&t=1561s) 空中楼阁式的嬉皮士空想 — Adam Goldstein
> [33:45](https://www.youtube.com/watch?v=SmJ7KOAhyxQ&t=2025s) 对齐究竟是向什么对齐 — Emmett
## Mentioned
Adam Goldstein, Emmett, Softmax, Michael Levin, Erik Hoel, Robert Axelrod, Metagrid, Melting Pot, PettingZoo
## Video description
Adam Goldstein, Co-Founder, Chairman, Head of Research, Softmax
Abstracts
The AI field is focused on a future where either we control the AIs or they control us. This is extremely dangerous and also completely unnecessary. In the living world, we see cells work together to form multicellular life, and different organisms work together in symbiosis, without any piece being "in control" of the rest. Inspired by such examples from biology, the field of organic alignment is charting an alternative approach to AI. In this talk I will focus on the math of organic alignment: understanding what it means (and how to measure) when parts form a larger whole, and how the whole affects the parts. I will provide an overview of recent research in emergence, learning, and compassion, and highlight open areas of research in the field. If you want to live in a world where AIs and humanity thrive together, this talk will help you understand what's already possible and how your research can fit in.
[Source](https://www.youtube.com/watch?v=SmJ7KOAhyxQ)
Source quotes
[5:50] Adam Goldstein: 涌现并不等于善好。
[16:25] Adam Goldstein: 如果我们直接将状态 4 和状态 7 视为同一个状态，会发生什么？
[16:25] Adam Goldstein: 如果将这些状态聚合在一起，预测未来的准确率会比分开处理时更高还是更低？
[19:36] Adam Goldstein: 我把你当成一个‘人’而不是‘一堆细胞’来对待，是因为这样能让我与你的互动变得简单得多。
[20:48] Adam Goldstein: 如果前几张幻灯片让你昏昏欲睡，现在可以醒醒了。
[20:48] Adam Goldstein: 只要系统产生了涌现，AI 就一定会获得更高奖励或更低损失
[26:01] Adam Goldstein: 空中楼阁式的嬉皮士空想
[33:45] Emmett: 对齐究竟是向什么对齐
Source quotes
[0:04] Adam Goldstein: one of the nerdier talks you'll hear at this event
[0:04] Adam Goldstein: without complaint
[0:04] Adam Goldstein: atrocious
[26:00] Adam Goldstein: The emergence math is the really interesting stuff... But as far as the multi-agent well-being math, that stuff you can just think about yourself: 'What does it mean to me for an AI to be happy?' If you know the answer for one AI, then you know the same answer for a group of AIs.
[26:00] Adam Goldstein: What does it mean to me for an AI to be happy?
[26:00] Adam Goldstein: If you know the answer for one AI, then you know the same answer for a group of AIs.
[32:00] Adam Goldstein: sometimes the best way to adapt to a new environment is to start from scratch but to inherit some of the things that your parents learned.
[33:45] Adam Goldstein: aligned to what?
Original transcript
[0:04] Thank you and welcome everyone. This is
[0:06] probably going to be one of the nerdier
[0:08] talks you'll hear at this event. Today
[0:11] I'm going to be talking about the math
[0:12] of organic alignment. For those of you
[0:15] who were at this morning's keynotes, you
[0:16] heard my co-founder and our CEO EMTT
[0:19] talk about the philosophy of organic
[0:21] alignment. Today I'm going to dive in a
[0:23] little bit more on the specifics, how
[0:25] you could even think about measuring
[0:26] this, quantifying it, and using it in
[0:28] your own AI work. So to start with, I'm
[0:32] going to talk about the failures, I
[0:34] think the inevitable failures of the
[0:36] standard approach to AI alignment, which
[0:38] is typically called values alignment.
[0:41] I'm going to talk instead about organic
[0:42] alignment. And I'm going to break it
[0:44] down into two categories. One is the
[0:47] question of uh emergence. When do a
[0:51] group of parts form a larger hole?
[0:55] That's going to be the first kind of
[0:56] math that I talk about. And the second
[0:58] is the math of well-being. When there's
[1:00] a larger hole that forms, is it good for
[1:03] the parts or bad for the parts? That's
[1:05] going to be the rough outline for the
[1:07] math. Then I'm going to talk a little
[1:08] bit about how you can actually measure
[1:10] this in simulation. Anyone here? We've
[1:12] got an open- source project to make it
[1:14] easy to run multi- aent simulations and
[1:17] to start to get more acquainted with
[1:20] this kind of organic alignment.
[1:23] So the approach of values alignment
[1:25] which is what basically everyone in the
[1:27] mainstream AI field is doing is a pretty
[1:31] straightforward approach which entails
[1:33] first deciding what we want the AIs to
[1:36] say or do what we consider a good AI
[1:39] then rewarding the AI when it does more
[1:42] of that slash punishing it when it does
[1:45] less of it. And then eventually we hope
[1:49] they hope this leads to something that
[1:51] makes AI perfect at doing everything we
[1:54] could ever hope for without complaint.
[1:57] That's the that's the dream. That's what
[1:59] you hear most people in the AI world, AI
[2:01] safety and AI capabilities talking
[2:02] about. This is the kind of alignment we
[2:04] want. We're going to make AIs that do
[2:06] our bidding, never complain, and never
[2:08] deviate from our wishes. But as EMTT
[2:11] pointed out this morning, the track
[2:12] record of this is atrocious. It is not
[2:15] working at all. Even though this has
[2:18] been the mainstream approach for AI
[2:19] alignment for going on 10 years, we just
[2:22] keep being
[2:24] shown the ways that the AIs do stuff
[2:26] that is totally outside of what we want,
[2:28] totally outside what we've trained them
[2:30] to do, and it just keeps getting worse.
[2:33] So maybe we should try something else.
[2:36] The idea of organic alignment is that we
[2:39] have AI solve problems in groups. The
[2:42] whole basis is around group problem
[2:44] solving, working together. And we
[2:46] measure whether these collections of AIs
[2:49] are in fact forming a functional whole
[2:52] like a functional organization of people
[2:55] or a functional group of cells in the
[2:56] body. And we see whether that larger
[2:59] hole is good for the parts or bad for
[3:02] the parts. And the hope is if we do this
[3:05] well enough that eventually we create
[3:07] groups of AIs and humans in which we're
[3:10] all working together to solve problems
[3:12] and satisfy all of our goals and we all
[3:14] benefit. So there's no control here.
[3:18] There's no us telling the AI what to do
[3:20] and enslaving them and rewarding them if
[3:21] they if they do what we want and
[3:23] punishing them otherwise. This is about
[3:25] working together as peers. Totally
[3:27] different approach than the mainstream.
[3:30] And the analogy that we draw from in
[3:32] softmax is the analogy of life,
[3:34] particularly multisellular life. If you
[3:37] think about, for example, slime molds,
[3:39] there are a bunch of cells, a bunch of
[3:41] individual beings working together to
[3:45] solve problems that none of them would
[3:46] be able to solve on their own. And they
[3:48] do these amazing things in physical
[3:50] space and chemical space. They do all
[3:52] sorts of things collectively that they
[3:54] wouldn't be able to do on their own. An
[3:56] even more interesting example, I think,
[3:59] is that of symbiosis. So, for example,
[4:02] the bacteria in our gut and our body,
[4:05] our human cells work together to solve
[4:08] problems better than either could do on
[4:10] their own. I mean, our bodies really
[4:12] hate like low pH environments. You know,
[4:17] if you put a a human inside something
[4:19] that had a pH of one or two, we would
[4:21] just dissolve. But the bacteria in our
[4:24] gut love it. that's like home for them.
[4:27] So they do really well in our stomach
[4:29] and they help us process food that would
[4:32] be otherwise very difficult for us to
[4:33] extract nutrients from. And that kind of
[4:36] symbiotic relationship is what you
[4:38] should have in mind when you think about
[4:40] organic alignment. There's no
[4:41] requirement that we and the AIs do the
[4:43] same thing or think in the same way. But
[4:47] we do have to work together in such a
[4:49] way that we both benefit from the
[4:51] interactions like us and our gut
[4:53] bacteria.
[4:54] And as inspiration for this approach, I
[4:57] encourage those of you who haven't read
[4:58] to take a look at the work of a biology
[5:01] professor named Michael Levan. I studied
[5:03] with him several years ago and I started
[5:05] an incubator with him. That was where
[5:07] softmax came from. And what he writes
[5:09] about is the the way in which
[5:12] intelligence exists across scales that
[5:15] there's even a minimal amount of
[5:16] intelligence
[5:18] cells. individual cells are a little
[5:20] intelligent and they lead to intelligent
[5:22] organs and intelligent bodies and then
[5:25] intelligent groups of people or groups
[5:26] of animals and intelligent ecosystem. So
[5:29] there's no highest level of intelligence
[5:32] and there's also no lowest level of
[5:33] intelligence. There's just a little bit
[5:35] as you get lower and lower down. And
[5:37] when you see things that way, it becomes
[5:39] perfectly obvious that we have to think
[5:41] about this problem of coming together
[5:42] and working together if we're going to
[5:44] make any progress on working well with
[5:47] super intelligent
[5:49] AI.
[5:50] So the the the principal concept of
[5:53] forming a larger hole is one that's
[5:55] known in in the field of complex systems
[5:58] and in other domains as emergence. This
[6:00] is something where you have a lot of
[6:01] small bits and say molecules or atoms
[6:04] and from those you get the emergence of
[6:07] cells and emergence is I think a very
[6:09] good word for this because the parts
[6:12] don't go away. When you have a cell,
[6:15] there's still molecules. If you wanted
[6:17] to, you could ignore the cell and just
[6:19] look at the molecules. Likewise, when
[6:21] there's a human body, they're still
[6:22] cells. Sometimes we forget that. We just
[6:24] treat other people as though they're
[6:26] these things, but they're made of cells
[6:28] just like we are. And those parts, you
[6:31] can look at them or you can ignore them.
[6:32] This ability to look at the parts or
[6:34] ignore the parts is what makes some
[6:36] something an emergent system. You can
[6:38] look at it as a whole or you can look at
[6:40] it as a group of parts. Being able to
[6:42] keep both perspectives in mind is key.
[6:45] But it's important to recognize that
[6:47] emergence doesn't guarantee benefit. So
[6:50] you can have emergent larger holes like
[6:52] parasitic relationships. The picture on
[6:55] this screen is on the left here. One of
[6:57] an ant being parasited by a fungus takes
[7:01] over the ant's brain and makes it do the
[7:03] fungus's bidding. That's something that
[7:05] the fungus couldn't do on its own and
[7:07] something the ant couldn't do on its
[7:08] own. So there is emergence, but it's not
[7:11] the good kind of emergence. It's the bad
[7:12] kind of emergence. Likewise, you can
[7:14] have the emergence of groups that are
[7:16] dominated by a particular person or
[7:19] ideology that works against the
[7:21] interests of the parts or some or all of
[7:25] the parts where it actually actively
[7:27] suppresses or tries to kill the parts.
[7:29] That's also a kind of emergence. There's
[7:31] something that didn't exist before.
[7:32] There's a larger level of organization,
[7:34] but it's not one that benefits the
[7:36] parts. So, these are the bad kinds of
[7:38] emergence. I think it's just important
[7:40] to name emergence is not enough. If you
[7:42] come away from this being like, "Oh, we
[7:44] just have to have AI's emerge." Uh-uh.
[7:47] No, that could be very, very bad. We
[7:48] need more than that. What we need is
[7:51] emergence and the parts to benefit. This
[7:53] is my very simple uh two-dimensional
[7:56] graph of organic alignment. We want
[7:59] there to be emergence and we want the
[8:01] parts to benefit. If you're in the upper
[8:02] right corner here, great. If you're
[8:05] somewhere else, not so good. Uh that's
[8:08] that's where we want to head for. So,
[8:10] I'm going to break these two axes down
[8:12] for the rest of this presentation and
[8:13] walk you through the math for both.
[8:15] We're going to start with the math of
[8:16] emergence. This is a really exciting
[8:19] field that was studied some in the 80s
[8:21] and 90s, but has really taken off in the
[8:23] last 10 years. Even this year, one of
[8:25] the papers I'm going to show you just
[8:27] was uh preprinted this year. So, there's
[8:30] there's several frameworks for it, but
[8:31] I'm going to focus today on two. One is
[8:34] called the principal information
[8:35] decomposition. This is an information
[8:38] theoretic way to get at the question of
[8:41] whether a whole is more than the sum of
[8:43] the parts. And the second is a paper
[8:45] that we sponsored this year that was
[8:47] preprinted a few months ago called
[8:48] causal emergence 2.0. So let's start
[8:51] with the principal information
[8:53] decomposition. There's many papers on
[8:55] this. This particular paper is the one
[8:57] that I'm going to be um drawing some uh
[8:59] language and some examples from. If you
[9:01] search P principal information
[9:03] decomposition, you can find a bunch of
[9:05] these on the internet. So the key
[9:08] concept in the P is that uh you want to
[9:12] know how much information is contained
[9:14] in the hole that's beyond the
[9:17] information in any of the parts. And it
[9:20] might seem a little weird to think that
[9:23] such a scenario is even possible. Isn't
[9:25] a hole always just the parts? Well, yes
[9:28] and no, right? An example of this would
[9:30] be a flock of birds, right? Imagine a
[9:33] flock of birds in which all of the birds
[9:35] are moving in exactly the same direction
[9:38] at exactly the same speed. Like each of
[9:39] my fingers is a bird, right? In that
[9:42] scenario,
[9:43] all of the information is in every
[9:45] single bird. Every bird has the same
[9:47] speed and the same same p uh direction.
[9:50] So there's nothing that the flock knows
[9:52] that the individual birds don't know.
[9:55] Right? That's one scenario. You could
[9:56] also imagine a scenario where birds are
[9:58] flying at random, right? Just imagine
[10:00] birds flying at random. The flock
[10:03] doesn't know anything the birds don't
[10:05] know there because there's no
[10:06] information to know. It's just a random
[10:08] mix, right? But there's this in between
[10:11] zone with flocks of birds. And if you
[10:13] ever go outside, you'll you'll you'll
[10:14] notice this where the birds are not
[10:16] moving at exactly the same speed and the
[10:18] flock is kind of making decisions as a
[10:21] group. Which means that in a sense the
[10:23] flock knows something that the birds
[10:25] don't individually know. You'd actually
[10:28] have to look at what the whole is doing
[10:30] to understand where the hole is going.
[10:32] You can't just look at any individual
[10:34] part. That's what the principal
[10:36] information decomposition is about. So
[10:38] the way that this is typically set up in
[10:40] the literature is that you have some
[10:42] target variable that you're trying to
[10:44] understand. call that t and you have
[10:46] source variables call those x and y
[10:49] which you're looking at to get
[10:51] information about t. So for example
[10:53] you're looking at the birds to
[10:55] understand the behavior of the flock.
[10:57] Simple simple example. So the question
[11:00] is where's the information in this? And
[11:02] there's three different kinds. If you
[11:05] wanted to know the information in T, one
[11:07] possibility is that you could look at
[11:09] just X or just Y
[11:12] and you would know everything you needed
[11:14] to know about T. That would be an
[11:16] example of a case of what's called
[11:18] redundant information. That's like when
[11:20] all the birds are flying in exactly the
[11:22] same direction. Totally redundant. Each
[11:24] bird is completely redundant. You only
[11:26] need one of them to know what's going
[11:27] on.
[11:28] The second possibility is that you might
[11:31] find the information for the flock in
[11:34] just one bird or just another bird but
[11:36] not both. This is a scenario where
[11:39] basically all of the dynamics of the
[11:42] system can be understood from a single
[11:44] part and all the other parts are are uh
[11:48] not necessary. That would be a scenario
[11:50] where the information is what's called
[11:52] unique. So instead of being redundant
[11:54] spread across all of the the birds, it's
[11:57] unique. it's in a particular bird.
[11:59] That's where you would look for the
[12:00] information.
[12:02] Neither of these, I should point out, is
[12:04] indicative of emergence. These are cases
[12:07] where the whole is reducible to the
[12:09] parts, these two scenarios. But there's
[12:11] a third scenario, which is that you can
[12:14] only find the information of the whole
[12:16] if you look at all the parts or at least
[12:19] more than one of the parts. That would
[12:21] be a scenario in which case there in
[12:23] which there's what's known as
[12:24] synergistic information where to
[12:26] understand the group you actually have
[12:28] to look at all the parts. You can't just
[12:30] look at some or one of them to
[12:32] understand the behavior of the whole.
[12:34] And there's you can you can look up
[12:36] there's math that decomposes the mutual
[12:38] information between the parts into these
[12:42] three terms. And these are exhaustive.
[12:44] So all of the mutual information can be
[12:47] bucketed into redundant, unique, and
[12:51] synergistic information. And generally
[12:53] speaking, the more synergistic
[12:55] information there is, the more that
[12:58] there's a sense in which there's a
[13:00] larger hole. So this is a really really
[13:03] nice framework for thinking about
[13:05] emergence because it's totally
[13:07] mathematically grounded in really simple
[13:10] uh information theoretic concepts like
[13:12] mutual information and uh there's these
[13:15] simple definitions for each of these
[13:17] kinds of information that you can apply
[13:18] to any system made of anything could be
[13:20] AIS could be cells could be people could
[13:23] be LLM doesn't matter you can you can
[13:25] run the exact same equations on it so
[13:27] that's the principal information
[13:29] decomposition and the reason that that's
[13:31] relevant for emergence is that if all
[13:33] the information between agents is
[13:34] redundant, there's nothing to be learned
[13:36] by looking at the whole. If all the
[13:38] information is unique, then there's
[13:41] nothing no sense in which they're acting
[13:43] coherently at all. They're just all
[13:45] doing their own thing. And if there's
[13:47] synergistic information, then there's a
[13:49] meaningful sense in which the whole is
[13:51] more than the sum of its parts. So, I
[13:53] recommend you read more about the
[13:54] principal information decomposition if
[13:56] you think that this might be relevant
[13:58] for your research.
[14:00] The second framework for understanding
[14:02] emergence is what's what is known in the
[14:04] literature as causal emergence 2.0. And
[14:07] this was preprinted just several months
[14:09] ago by a wonderful thinker named Eric Hy
[14:12] who's been working in this space of
[14:13] emergence for about a decade. For those
[14:16] of you who are familiar with integrated
[14:18] information theory, Julio Tenon's
[14:20] theory, Eric was a collaborator of his
[14:22] uh on that. So he's been spending a lot
[14:24] of time in this question of coherence
[14:26] and emergence since the field kind of
[14:28] began.
[14:30] So in causal emergence 2.0 the key
[14:32] question is how much predictive power do
[14:35] you get by looking at the whole instead
[14:37] of looking at the parts. So this is
[14:40] subtly different than the question of
[14:41] how much information is there. This is
[14:43] specifically information about what
[14:45] causes what. And I'll I'll dig in a
[14:48] little bit more to this just to to get
[14:50] the point across. But I think if you
[14:51] keep in mind causal emergence is about
[14:53] causality. It's about what causes what.
[14:56] That'll give you the best uh the best
[14:58] rubric. So the example that Eric uses in
[15:01] his paper is that of a Markov chain. For
[15:04] those of you who aren't familiar, just
[15:06] imagine this is a process where a system
[15:08] is moving through several different
[15:10] states over time. And in this diagram,
[15:13] what you can see is that there are eight
[15:15] states and there's a certain states
[15:18] transition to other states and other
[15:21] other states don't have any transitions
[15:23] at all. The way to interpret this is
[15:24] that if the system is in, for example,
[15:27] uh state five, that's the green one,
[15:30] right? Then it will move to another
[15:32] state, one of the ones that the arrows
[15:34] point to on the next step, the next
[15:36] period, the next time step. But if it's
[15:39] if you're in state two, for example, the
[15:41] yellow one on the bottom left, then
[15:43] you'll just stay in state two forever.
[15:45] You'll never transition to another
[15:47] state. And Markov chains are
[15:49] computationally universal in the same
[15:51] way that touring machines are, which
[15:52] means that any dynamical system, any
[15:54] computational process can in theory in
[15:57] principle be described by a Markoff
[15:59] chain like this with transition
[16:01] likelihoods. So what you'll notice if
[16:03] you look at this is that there's kind of
[16:05] a natural way you could partition the
[16:07] system into states that are connected to
[16:10] other states and states that are
[16:12] disconnected from other states. And what
[16:14] you can do, this is what Eric proposes,
[16:16] is that you can you can basically start
[16:18] to group states together. And the way he
[16:20] does this, just to make it easy to
[16:22] visualize, is by changing the colors of
[16:25] the states. So what you'll see in the
[16:27] next one is that uh now state number
[16:30] seven has turned blue just like state
[16:33] number four. That means that
[16:35] conceptually he's saying what if we just
[16:37] treat state four and state seven as the
[16:39] same. What if we just treat them as
[16:41] though they're the same? Not two
[16:42] different states but the same state. And
[16:44] then he adds another one. He says what
[16:46] if you treated state six as part of the
[16:48] same the same set of states. And then
[16:50] what if you treated state five as part
[16:52] of the same state?
[16:53] step by step, he's just grouping the
[16:56] system into fewer and fewer states by
[16:58] coloring one set of them the same. And
[17:01] you keep going all the way up until at
[17:03] the end you've got uh all but one of the
[17:05] states in the same the same set. And the
[17:09] question is, well, what does that
[17:11] accomplish for you? If you treat these
[17:13] states as all being the same, what does
[17:16] that actually get you? And you can
[17:17] answer that question numerically. So
[17:19] what he does is he uses these uh
[17:21] information theoretic concepts of
[17:23] causality and he says if you group
[17:26] states together can you predict the
[17:28] future better than if you treat the
[17:30] states as separate? I'm going to say
[17:33] that one more time. It's a really simple
[17:34] but important point. If you group the
[17:37] states together, can you predict the
[17:40] future better or worse than if you treat
[17:43] the states as separate?
[17:46] And the answer is at first as you group
[17:49] states together, the system becomes
[17:52] easier to predict. Right? Look here.
[17:55] When those three are colored blue, you
[17:58] know with three quarters probability
[18:01] what state the system's going to be in
[18:04] next. It's going to be in one of the
[18:05] blue ones. Right? At the beginning, you
[18:08] only knew with much lower likelihood
[18:10] what state it's going to be in next. But
[18:11] once you group them as the same, then
[18:14] you get more and more confident. Hey,
[18:15] it's going to be a blue one next. It's
[18:17] going to be a blue one next. It's going
[18:18] to be a blue one next. And eventually
[18:20] you get to perfect predictability. If
[18:22] you group all of the ones that are
[18:24] connected together the same color, then
[18:26] you know with 100% certainty that the
[18:29] next time step it's going to be a blue
[18:31] one because it was a blue one last time.
[18:34] But as you keep grouping them together,
[18:36] you get no increase in predictive power.
[18:39] you can keep coloring more of them blue.
[18:42] It doesn't improve your predictions at
[18:43] all. So the idea here is that in any
[18:46] system you do this this this
[18:49] partitioning or he calls it coarse
[18:50] graining. You take these micro states
[18:54] and you group them together into bigger
[18:56] and bigger and bigger macro states until
[19:00] you no longer get any improvements in
[19:01] your predictions by grouping them
[19:03] together. That's the concept of causal
[19:05] emergence. Really, really simple, really
[19:08] elegant. The algorithm is something that
[19:10] you know introductory computer
[19:12] scientists could understand. It's like a
[19:13] path finding algorithm. Um and it has
[19:17] these nice uh these nice uh concepts of
[19:19] causality baked in. So you can actually
[19:21] say hey if we get the correct grouping
[19:24] the correct coarse graining then we can
[19:27] demonstrate that that's the the uh
[19:29] chunking the coarse graining that best
[19:32] predicts the future of the system. So
[19:34] why is this relevant for emergence?
[19:36] Well, because if you treat a system as a
[19:39] whole, you're able to predict the future
[19:41] better, then it's a really good
[19:44] hypothesis to use as a scientist that
[19:46] the system is acting as a whole. I mean,
[19:49] when we say is a system acting as a
[19:51] larger hole, what else do we mean except
[19:54] for that it makes our life easier if we
[19:57] treat it as though it's a larger hole?
[19:59] Right? I treat you as a person, not a
[20:02] bunch of cells, because it makes it way
[20:04] easier for me to interact with you. If I
[20:06] just believe that you're a person
[20:07] instead of believing you're a bunch of
[20:09] cells. Of course, you're both, but it's
[20:11] just easier to think about if I treat
[20:13] you as a person. Likewise, in causal
[20:15] emergence, if you're better able to
[20:17] predict the future by treating the
[20:18] system as a whole, then basically it's
[20:20] saying it is a hole because it makes
[20:23] your life easier. You can predict the
[20:24] future better. And if uh if you can't
[20:27] predict the system better by by treating
[20:29] it as a whole, then it says no, actually
[20:32] just keep paying attention to the parts.
[20:34] There's no larger hole here. There's no
[20:35] emergence. It's just all about the
[20:37] parts. Again, a really simple, elegant
[20:40] framework for thinking about emergence
[20:42] here from the perspective of causality
[20:45] instead of mutual information.
[20:48] So, I've talked some about the notion of
[20:50] emergence and how you'd measure that.
[20:51] Now I want to talk about the notion of
[20:53] are the parts better off? This is going
[20:55] to be a much simpler concept. So if the
[20:57] last few slides kind of had you going to
[20:59] sleep, you can wake up. Now the idea of
[21:02] the parts benefiting is we already have
[21:04] all of these uh mathematical
[21:06] measurements in machine learning for
[21:10] whether an AI is happy or not. You can
[21:12] look at the loss or the reward or the
[21:15] free energy if you're into active
[21:16] inference. You can look at surprisal if
[21:18] you're a mathematician. All of these
[21:20] things point at, hey, is the AI doing
[21:22] something and is it happy about what
[21:23] it's doing? Okay, so here's the simple
[21:26] thing. If you want to know if all the
[21:27] parts are happy, you just do the same
[21:29] thing, but you look at all of the parts.
[21:31] That's it, right? You can take all of
[21:33] the metrics that you're already using
[21:34] for your ML models, but instead of
[21:37] looking at them for individual models,
[21:39] you just look at them across a group of
[21:40] models. Could be LMS, could be RL
[21:43] agents, could be active inference
[21:44] agents, doesn't matter. You just look at
[21:46] the math across all of them. Okay? It's
[21:48] kind of antilimactic, but I think it's
[21:50] still important to note because you can
[21:52] definitely have situations in which you
[21:55] have emergence and the AIs are unhappy
[21:59] about it, right? It's basically like an
[22:01] AI dictatorship or an AI parasitic
[22:03] relationship. You can 100% have that. So
[22:06] don't assume that just because you have
[22:08] emergence that the AIs are going to
[22:10] experience, you know, higher reward or
[22:12] lower loss. There's no guarantee of
[22:14] that. You actually do have to check.
[22:17] Of course, I've I've like done a little
[22:19] slight of hand. I've said, "Hey, just
[22:21] look at the well-being of all of the
[22:22] parts. Just look at the well-being of
[22:24] all of the AIs." Okay, but what if their
[22:26] well-being changes different amounts or
[22:29] different directions? Now, we get into
[22:30] the questions that social philosophers
[22:32] have been grappling with for a long
[22:33] time. Do you look at the average
[22:36] well-being in the population? Do you
[22:38] look at how the worst off are doing, and
[22:41] that's your measure of how well the
[22:42] group is doing? Do you look at the
[22:45] median? Do you look at the variance? I
[22:47] mean these I'm not I'm not going to make
[22:48] an answer to this. You do have to pick a
[22:50] perspective. Maybe you look at the
[22:52] average loss or the average reward of
[22:54] the agents or maybe you look at the how
[22:57] well the worst off of them is doing or
[22:59] all of these metrics. You do have to you
[23:00] do have to take a point of view on that.
[23:02] But once you've taken that it becomes
[23:03] very straightforward to compute this.
[23:07] Okay. So I've now laid out for you just
[23:09] high level the math of how you measure
[23:11] emergence and how you measure the
[23:12] well-being of a of agents in a
[23:14] multi-agent scenario. And together that
[23:16] gives you a way to quantify degrees of
[23:19] organic alignment among AIs or among any
[23:22] kinds of agents that we might think
[23:24] about. But then you might say, "Okay,
[23:26] great. How do I test this out? I've got
[23:28] some great AI that I think is really
[23:30] good at working with other other AIs and
[23:33] I want to demonstrate that it actually
[23:35] is good at forming emergent larger
[23:37] holes. And those larger holes are good
[23:39] for the AIs. I want to prove that. And I
[23:41] say to you, great. Come up and talk to
[23:43] me after this. Maybe we want to offer
[23:45] you a job. Maybe we want to sponsor your
[23:47] research. But at a minimum, I'm glad
[23:49] you're thinking about it. So, how do you
[23:50] test it out? Well, there are many
[23:52] multi-agent frameworks out there, and
[23:54] these predate the use of the term agent
[23:56] to just mean like chatbot. To be clear,
[23:58] these are multi-agent frameworks that go
[24:00] back many years when agent meant just an
[24:03] AI that did stuff. Um, melting pot,
[24:06] petting zoo. You probably come across
[24:07] others. Um, but real world problems
[24:10] rarely have defined endpoints. the way
[24:13] that most multi-agent uh reinforcement
[24:16] learning frameworks do. So if you
[24:18] imitate something like the real world,
[24:20] if you want to emulate the real world,
[24:22] you actually want a kind of multi-agent
[24:24] simulator where you can design the
[24:26] environment and you can design what the
[24:28] goals are. And uh that's why we we
[24:31] sponsor and contribute to the Metagrid
[24:33] open-source project that EMTT touched on
[24:35] this morning. Metagrid is uh a project
[24:39] where we have lots and lots of agents,
[24:42] as many as you want, that are
[24:43] interacting with each other. And as they
[24:46] interact, they can cooperate, they can
[24:48] compete, they can ignore each other,
[24:49] they can do all the kinds of things that
[24:51] basic living systems do um that help us
[24:55] to study whether they're forming a
[24:56] larger hole. And uh what makes it cool
[25:00] is that you can design your own maps of
[25:02] arbitrary complexity. So there's no
[25:04] fixed correct solution. You can have as
[25:07] many agents or as few agents as you
[25:09] want. And you can pick your own
[25:11] architecture too. So you can train an
[25:14] LLM and see if it'll cooperate in this
[25:16] environment. You can have an RL agent
[25:18] that you've trained yourself or use one
[25:20] of ours. You can build a a you know
[25:23] language synthesis model like whatever
[25:25] you want. Doesn't matter. You can you
[25:27] can test it out and see how it does. And
[25:29] you can train it locally or in the
[25:31] cloud. So, I've trained these on my own
[25:33] laptop, which is cool. It's nice to have
[25:35] really, really small agents that run
[25:36] really efficiently to test this stuff
[25:38] out. Or if you want to do like a big
[25:40] training run, you know, if you got a lot
[25:42] of money, great. You can do that, too.
[25:44] Um, and we've got these different
[25:46] evaluation metrics that allow you to
[25:48] test out how your agents doing in terms
[25:49] of basic skills like navigating a new
[25:52] environment, as well as metrics related
[25:55] to multi-agent behavior like the ones
[25:57] that I've talked about in this
[25:58] presentation.
[26:00] So to summarize, the approach of values
[26:02] alignment is a dead end as far as I can
[26:05] tell. There's no evidence whatsoever
[26:06] that it's going to start working all of
[26:08] a sudden. And we have lots of evidence
[26:09] that it's not. But the organic alignment
[26:12] framework is much more plausible because
[26:15] it's based on the principles that living
[26:16] systems, including our own bodies, use
[26:19] to navigate the world and to interact
[26:21] with others. There's a math for this.
[26:24] This is not just pie in the sky hippie
[26:26] thinking. There's an actual math for
[26:28] understanding emergence and well-being
[26:30] in multi-agent systems and there's tools
[26:33] that are free that anyone here can use
[26:36] for actually testing out your own AI
[26:38] models in those kinds of scenarios. Um,
[26:41] all of this is free. All of it's in the
[26:43] public domain. Every single paper that
[26:45] I've listed here, every single project
[26:46] is something that's free to use. There's
[26:48] no restrictions. You can just go have
[26:50] fun.
[26:52] uh if you are on the fence about what to
[26:55] do about alignment, I would encourage
[26:57] you to try this out and see how it fits.
[26:59] I think it will lead to a better future
[27:01] for all of us and also for the AIS and
[27:04] that's a future that we can all be
[27:05] excited about. So I wanted to leave some
[27:07] time for Q&A because I imagine some of
[27:10] you are curious about stuff. Yeah.
[27:12] >> Uh you spoke about like you know
[27:14] decomposing information. Um so you you
[27:16] said that there's a math to decomposing
[27:18] information and and you can also see
[27:20] like what is more than the parts. U so
[27:23] are you talking about like semantic
[27:24] decomposition or
[27:26] >> No, I'm talking about mutual information
[27:29] which is like a fancy form of
[27:31] statistical correlation. So it's just
[27:33] literally saying if I have if I have two
[27:35] different vectors right from to what
[27:37] extent can I predict the second one from
[27:39] the first?
[27:39] >> You're talking about the embeddings
[27:40] themselves.
[27:41] >> Yeah, you could look at them as
[27:42] embeddings. You can also just literally
[27:44] look at them as sequences of bits. I
[27:46] mean like they're it's just a super
[27:48] basic statistical computation. It's
[27:50] saying how much mutual predictability is
[27:52] there between two strings.
[27:54] >> Okay.
[27:54] >> And and one more question that was the
[27:55] was the one where you talk about the
[27:57] happiness of the LM or like the I guess
[27:59] the harmony of the system, right? You're
[28:01] talking about like measuring the loss
[28:02] functions and all this like I mean is
[28:04] that really like a validated approach to
[28:07] like because when you talk about like
[28:08] happiness or something these are
[28:09] subjective qualities
[28:10] >> of course. Yeah. Yeah. Happiness is
[28:11] subjective. I don't know what makes you
[28:12] happy. You might not know what makes me
[28:14] happy. Right?
[28:15] >> So,
[28:15] >> but when we when we do like policym as a
[28:19] country or as a state, right, we do look
[28:21] at things that are quantitative. We look
[28:23] at, for example, GDP per capita, right?
[28:26] Like how much how much overall money is
[28:29] there in the economy or how much is
[28:31] there per person?
[28:32] >> It's it's debatable as to whether that's
[28:34] useful or not. Actually, you're bringing
[28:35] up a good point, you know.
[28:36] >> Absolutely. So u the when you when you
[28:39] talk talk about the loss functions and
[28:40] all this like you're talking about the
[28:41] it it makes theoretical sense but has it
[28:44] like turned out like you know you're
[28:45] seeing that okay my measurements are
[28:47] actually leading to a more harmonious
[28:49] system where the results are more
[28:50] optimal well this is the question right
[28:52] optimal for whom I would say that the
[28:55] way we've designed AIs to date is that
[28:57] we've said that what's good for the AI
[28:59] is to improve its loss right we've
[29:02] decided that so because we've decided
[29:05] that that's what we're going to pick as
[29:06] our measurement We're going to look at
[29:08] the loss for every agent. If you say,
[29:09] "No, no, I actually don't want to look
[29:11] at the average loss. I want to look at
[29:13] surprise, right?" Great. So, you can use
[29:15] surprise. If you want to look at free
[29:17] energy, you can look at free energy. If
[29:18] you want to look at how does a human
[29:20] rate the output, then you can use that
[29:23] as your metric. My point is just, and
[29:24] this is why I kind of joked about it,
[29:26] the emergence math is the really
[29:28] interesting stuff. That's where you
[29:29] actually want to like read the papers
[29:31] and whatever. But as far as the
[29:32] multi-agent well-being math, that stuff
[29:34] you can just think about yourself. What
[29:36] what what does it mean to me for an AI
[29:38] to be happy? How would I know if the AI
[29:40] is happy? If you know the answer for one
[29:41] AI, then you know the same answer for a
[29:44] group of AIs. That's that's what I'm
[29:45] suggesting. But yeah, it's totally new
[29:47] territory. Like no one knows whether
[29:50] that's how it's going to work out. I'm
[29:52] just suggesting this is something that
[29:53] is measurable. Um yeah, we got another
[29:56] question.
[29:58] Um so what happens in systems when that
[30:02] you've modeled out when um like small
[30:05] groups of the components are are not
[30:08] benefiting from the system. Yeah. Well
[30:11] the the the thing that led me into this
[30:12] research was studying cancer and I think
[30:15] that's a really good analogy to what
[30:17] happens when some of the parts are not
[30:19] benefiting and others are. What you see
[30:22] is that the parts that aren't benefiting
[30:24] start trying to figure out new
[30:25] strategies to start doing better, which
[30:28] is by the way what we train the AIs to
[30:30] do, right? It's also what cells have
[30:32] evolved to do. If the situation is hard,
[30:34] try something new. So all the cells in a
[30:37] healthy body, you know, we're born, all
[30:38] the cells are doing the things that
[30:40] they're supposed to be in the group. But
[30:42] then if we, you know, smoke a lot, then
[30:44] the cells in our lungs are having a
[30:47] really, really tough time, right? If we
[30:49] drink a lot, the cells in our liver are
[30:51] having a really, really hard time.
[30:53] That's way beyond what they've expected
[30:56] is going to happen. And so they start to
[30:58] explore other strategies to survive and
[31:01] to thrive. And those strategies lead
[31:03] them actually to do better, but at the
[31:06] expense of the rest of the body. That's
[31:09] that's generally the the way that these
[31:11] these fractures lead to to poor health.
[31:15] And I think we should expect to see the
[31:16] same thing in multiAI systems unless
[31:20] we're paying attention to how well the
[31:21] AIs are doing.
[31:23] >> Um, have you found any like kind of
[31:25] optimal ratios that exist or
[31:28] stratifications?
[31:29] >> No, I mean what I'll say kind of
[31:31] qualitatively is that we do know that
[31:33] there's benefits to sparity, right? that
[31:36] if you have a super hyperconnected
[31:38] system, then it's actually very hard for
[31:41] the cells or the parts to make sense of
[31:44] the whole because there's so much going
[31:46] on. What we see in our bodies is that
[31:48] most of the cells in our body are only
[31:51] directly connected to a few other cells,
[31:54] like fewer than 10. And I think there's
[31:56] a reason for that. It's much easier to
[31:57] to model your world and to do good
[32:00] inference when your world is small. When
[32:02] there's only like a small number of
[32:04] other agents as the number of agents
[32:06] grows, it becomes harder and harder to
[32:08] do good inference.
[32:11] Question over here.
[32:12] >> Thanks. Uh that mention of cancer was
[32:14] timely because I was your work was
[32:16] making me think of Robert Axelrod and
[32:17] his work on comm cooperation led him to
[32:19] to cancer as well. Um so related to that
[32:21] I was thinking about this in terms of
[32:23] time because different you know agent
[32:26] you know a a studies of agents
[32:27] cooperating has shown like the the
[32:29] length of time you're looking at like
[32:31] really determines the outcome you know
[32:32] similarly like
[32:33] >> we don't have multisellular organisms
[32:36] that live forever as far well I guess
[32:37] maybe jellyfish maybe depending
[32:39] >> yeah and pleneria are also a fun example
[32:41] but most don't you're right
[32:42] >> right so I was and you mentioned that
[32:43] the meta was you know sort of an
[32:45] open-ended simulation so I was just
[32:47] wondering sort of how time affects what
[32:49] you're looking at Yeah. So, so just to
[32:50] be really specific on the time thing in
[32:52] Metagrid, we create these um basically
[32:55] limited time lifespans. So, agents,
[32:57] which is to say policies with a
[32:59] particular set of weights, persist for a
[33:02] period of time, you know, maybe a
[33:03] thousand time steps or something like
[33:05] that. And then effectively they die and
[33:08] get replaced by their successors. And um
[33:11] that in that way it sort of mirrors
[33:13] living systems where we're constantly
[33:15] regenerating cells in our bodies and we
[33:17] are being replaced by people our our
[33:20] children and our grandchildren over
[33:22] longer time scales. Yeah. So I I think
[33:24] there's a reason that that's the way it
[33:26] works for most living systems. I think
[33:28] there's a reason why we have mortality
[33:30] and the reason is because sometimes the
[33:32] best way to adapt to a new environment
[33:35] is to start from scratch but to inherit
[33:37] some of the things that your parents
[33:39] learned. So yeah, we're we're we're
[33:41] imitating that. We're actually borrowing
[33:43] from that in our system. Yeah.
[33:45] >> Um I'm curious about the word alignment.
[33:48] Um kind of your uh flock example. If all
[33:52] of the birds wanted to be alone and they
[33:54] were at a central point and they all
[33:56] just flew out,
[33:57] >> they'd all be very happy, but I wouldn't
[33:59] think they'd be aligned, right?
[34:01] >> And so
[34:02] >> is there a a subtle meaning behind what
[34:05] you think of as the parts are aligning
[34:07] to some
[34:08] >> Yeah. Yeah. Well, just to make sure I
[34:09] understand your question, you're saying
[34:10] literally if the center stayed at the
[34:12] center, but all the parts like exploded,
[34:15] would that be alignment? Is that what
[34:17] you're asking?
[34:18] >> Well, if they wanted to be alone, then
[34:20] they would all fly apart,
[34:21] >> right?
[34:22] >> And they'd all be happy.
[34:23] >> Yeah. Okay, great. I think I've got the
[34:25] question. You can tell me. So, so I
[34:27] think it's important when we talk about
[34:28] alignment, EMTT mentioned this this
[34:30] morning, aligned to what? Right. The way
[34:33] that we're using the term alignment is
[34:34] that they're aligned to a larger whole
[34:36] of which they're a part. Right? So the
[34:39] cells in our body are only aligned if
[34:41] they're working for the well-being of
[34:43] our body. And our body is only aligned
[34:46] with our cells if our body is helping
[34:47] our cells. Cancer would be cellular
[34:50] misalignment. Some of the cells are not
[34:52] working for the benefit of the body.
[34:54] They're working for their own benefit at
[34:55] the body's expense. Right? So that's how
[34:57] we mean organic alignment. You can also
[35:00] use a different kind of alignment, you
[35:03] know, and say, well, the cells or the
[35:04] the birds are all aligned in wanting to
[35:06] be alone. True, they're all aligned in
[35:09] the sense that they all agree on what
[35:10] they all want, but there's no sense, I
[35:13] would argue, in which following that
[35:15] preference leads them to behave as a
[35:17] larger whole. So, they fail on the
[35:19] question of emergence. They might be
[35:21] doing well in terms of happiness, but
[35:23] they're not emerging into a larger hole.
[35:25] So, that we would say is not organic
[35:27] alignment. Yeah.
[35:29] This is a new though. So like we're
[35:31] coming up with these terms, you know,
[35:33] this is like I'm proposing how how you
[35:35] might think about it. I'm not saying
[35:36] it's the definitive answer. We have
[35:38] another question or two.
[35:42] Okay. Well, here's what I'd say. If you
[35:44] found this interesting, shoot me an
[35:46] email at adamsoftmax.com. We've got a
[35:48] whole community of researchers who are
[35:50] thinking about organic alignment and
[35:52] building new AI architectures and new
[35:54] evaluation metrics and basically doing
[35:57] research using this Metagrid framework
[35:59] and the ideas that I pointed out today.
[36:01] And we're always looking for research
[36:03] collaborators, people who've got new
[36:05] perspectives from either AI or from
[36:07] other fields. And if you know, please
[36:09] please don't email me with like trying
[36:11] to sell me your software or something
[36:12] like I got so many of those emails
[36:14] before this conference. But if you're
[36:15] genuinely interested in organic
[36:17] alignment, let's chat. Sound good? All
[36:20] right. Thanks, guys.
