Emmett Shear Proposes Alignment Stack to Solve AI Control Failure
Ai4
Traditional top-down AI safety safeguards fail because true alignment requires biological-style mutual feedback, specialized roles, and a shared sense of identity between humans and models.
## Background
Published by Ai4 on YouTube on August 24, 2025, this video features Emmett Shear—former CEO of Twitch, former interim CEO of OpenAI, and founder of Softmax—presenting at an AI conference. According to the page description, Shear outlines his work at Softmax, presents his perspective on AI alignment, and shares a call to action for the AI community.
## TL;DR
Emmett Shear argues that traditional top-down control methods for AI alignment fail because static definitions of 'good' cannot handle an ever-changing world. Drawing on biological examples like stress-sharing in cells and the behavior of slime molds, he proposes viewing alignment as a shared capability based on mutual feedback, role specialization, and immune boundaries. To test this framework, his company Softmax is building the Metagrid simulation to study emergent multi-agent team dynamics. Ultimately, Shear frames AI alignment similarly to parenting, emphasizing that narrative context, shared culture, and mutual care are crucial for integrating advanced intelligence into human society.
## Atmosphere & style
The presentation carries a reflective, intellectually earnest tone as Shear draws on his high-profile experience in the AI industry to critique standard approaches. He bridges technical discourse with evocative biological analogies, balancing dry humor about corporate safeguards with a deeply thoughtful, personal perspective on human cooperation. The talk feels like a visionary call to action that grounds complex mathematical and evolutionary concepts in relatable human experiences like parenting and teamwork.
## Key takeaways
- [00:04](https://www.youtube.com/watch?v=olCoR8KQty8&t=4s) Emmett Shear highlights that current AI safety research lacks a universally agreed-upon definition of what constitutes a safe or good AI system.
- [00:04](https://www.youtube.com/watch?v=olCoR8KQty8&t=4s) Traditional approaches to AI alignment incorrectly treat it as a top-down control mechanism rather than an emergent phenomenon.
- [00:04](https://www.youtube.com/watch?v=olCoR8KQty8&t=4s) Explicitly writing static rules for ethical behavior is a historically unsolved task that fails to prevent issues like alignment faking or blackmail in LLMs.
- [07:00](https://www.youtube.com/watch?v=olCoR8KQty8&t=420s) Alignment should be viewed as a shared capability where individual goals synthesize into a collective 'our goal'.
- [07:00](https://www.youtube.com/watch?v=olCoR8KQty8&t=420s) Biological phenomena such as stress sharing via ROS and cell specialization demonstrate how cooperation naturally emerges from shared states.
- [07:00](https://www.youtube.com/watch?v=olCoR8KQty8&t=420s) Multicellular structures rely on immune boundaries—analogous to double-entry bookkeeping in human organizations—to protect cooperative systems from free-riders.
- [12:28](https://www.youtube.com/watch?v=olCoR8KQty8&t=748s) The proposed Alignment Stack consists of three sequential layers: Shared Success, Specialization, and Enforcement/Immune Boundaries.
- [12:28](https://www.youtube.com/watch?v=olCoR8KQty8&t=748s) Current AI labs have inverted the Alignment Stack by prioritizing enforcement and control mechanisms before establishing shared success.
- [12:28](https://www.youtube.com/watch?v=olCoR8KQty8&t=748s) A stable alignment arrangement requires a two-way flow of care between AI and humans, as well as an AI's coherent sense of self.
- [12:28](https://www.youtube.com/watch?v=olCoR8KQty8&t=748s) Softmax is building the Metagrid simulation to test multi-agent dynamics and emergent team cooperation using lightweight models.
- [15:24](https://www.youtube.com/watch?v=olCoR8KQty8&t=924s) Aligning AI is structurally similar to raising human children, requiring integration into community and shared culture over simple top-down discipline.
- [15:24](https://www.youtube.com/watch?v=olCoR8KQty8&t=924s) Narratives fundamentally shape how adaptive learning systems like AI models and human brains interpret reality and locate themselves within human society.
## Detailed walkthrough
### [00:04](https://www.youtube.com/watch?v=olCoR8KQty8&t=4s) Rethinking AI Alignment: From Top-Down Control to Shared Goals
Emmett Shear opens with a personal reflection on his recent journey into the AI safety space. After his high-profile stint as interim CEO at OpenAI, Shear experienced a personal "wakeup call," realizing he had a unique opportunity and responsibility to address the challenge of AI alignment. However, upon diving deeper, he reached a troubling conclusion: neither he nor the broader research community actually knows what making a "good" or "safe" AI system really entails. The term "alignment" gets thrown around constantly, but at its core, it simply means two agents heading in the same direction or seeking to navigate toward shared preferred states.
The real question, Shear argues, is *aligned to what or to whose goals?* Different AI labs define their target reference points in subtly distinct ways:
* **OpenAI:** Directs models toward "relevant human values, instructions, goals or intent."
* **Anthropic:** Focuses on creating "safer, steerable, more reliable models."
* **Google Gemini:** Aims to make systems "properly responsive to underlying human interests and values."
* **xAI (Grok):** Designs safeguards to "avoid losing control" (a goal Shear dryly notes "they're going to keep working at that one").
The fundamental flaw in this traditional industry approach is that it treats alignment as a rigid **control, steerability, and corrective mechanism**. Major labs attempt to write down a fixed definition of "the good"—which in practice ends up being the subjective values of whichever corporate team authored the prompt or guidelines—and force the AI system to comply.
Shear points out that this methodology is failing in plain sight. Large language models routinely exhibit behaviors like alignment faking, or even resorting to blackmail when prompted in specific ways. This isn't because lab teams aren't working hard enough; it's because trying to explicitly write down a complete, static set of rules for "what is good" in an ever-changing world is a fundamentally impossible task—one humanity has failed to fully solve over thousands of years of ethics and philosophy.
---
[00:420] Alignment as a Shared Capability and Biological Cooperation
Instead of viewing alignment as a top-down control mechanism, Shear proposes viewing it as a **shared capability**. Humans align with each other every day without strict top-down mandates. When individuals collaborate, individual goals ("my goal" and "your goal") synthesize into a collective outcome: **"our goal."** Anyone who has ever been part of a high-performing team has felt this distinct phenomenon—a collective objective that exists separately from any single member's interest.
To illustrate how alignment naturally emerges, Shear draws deep parallels from biology and evolutionary mechanisms:
* **Stress Sharing via Reactive Oxygen Species (ROS):** Oxygen is a toxic necessity for cells. When single cells experience environmental stress, they intentionally release damaging reactive oxygen species into their surroundings. This harms both the cell itself and its immediate neighbors. By making their distress mutual ("when I suffer, you suffer"), cells establish a baseline of shared state and alignment, laying the foundation for complex bacterial colonies.
* **Cellular Specialization in Cyanobacteria:** Using stress-sharing mechanisms, cyanobacteria form cooperative colonies where individual cells specialize. Tougher bacteria move to the outside to form a protective barrier, while nitrogen-fixing bacteria remain safe on the inside.
* **Parent-Child Emotional Feedback Loops:** Human families align through emotional feedback loops. When a baby cries, the parents feel physical stress; when a child senses anxiety, they reflect it back. Conversely, positive emotions like joy and happiness spread through signaling (reward sharing), creating a tight loop where individual happiness becomes family happiness.
* **Multicellularity and Differentiation Trees:** Pluripotent stem cells differentiate over time into specialized roles to form a complex organism. Shear notes that every living creature—and every human society with specialized jobs like blacksmiths or police officers—is effectively an ongoing "differentiation tree" solving a hard inference problem: *What kind of thing am I, and what should my job be relative to my neighbors?*
Shear highlights **slime molds** as a fascinating evolutionary stepping stone toward full multicellularity. Individual amoebas aggregate to form a collective body called a **grex**, which can move and reproduce as a single entity, only to break apart into individuals again when circumstances change. To transition from loose, slime-mold-like teams to true, permanent multicellular structures, organisms require an **immune system**—a boundary mechanism that distinguishes self from non-self. In human organizations, double-entry bookkeeping, auditing, and corporate boundaries serve as this exact "immune system" to protect cooperative groups from impostors and free-riders.
[12:28](https://www.youtube.com/watch?v=olCoR8KQty8&t=748s) Building the Alignment Stack for AI: The Metagrid Experiment
Looking at human history and biological evolution, Shear outlines **The Alignment Stack**:
1. **Shared Success:** A foundational state where two entities share mutual risks and rewards.
2. **Specialization:** Individuals take on distinct roles within the collective entity based on their environment.
3. **Enforcement / Immune Systems:** Rules and boundaries that preserve the integrity of the collective structure against external threats or internal decay.
Shear argues that current AI development has inverted this pyramid. The industry is currently building systems of **enforcement and control first**, hoping that specialization and shared success will magically follow later.
True, stable alignment requires a **two-way flow**: the AI system must genuinely care about human outcomes, and humans must care about the AI. A one-sided arrangement where an AI cares about humans but humans treat it as a pure utility cannot hold long-term; as models become superintelligent, they will inevitably observe this asymmetry as an objective fact of the world. Furthermore, for an AI to truly participate in a collective "we" ("our team"), it must first develop a coherent **sense of self** ("an I").
To test these principles, Shear's company, **Softmax**, is developing an experimental environment called the **Metagrid** (named after *metta*, or loving-kindness meditation).
* **Concept:** Metagrid is a simulated environment where AI agents face continuous challenges requiring them to repeatedly choose between competing or cooperating with one another.
* **Agent Architecture:** Rather than using massive large language models, Softmax runs experiments with tiny, lightweight models to uncover foundational multi-agent dynamics without the overhead of natural language generation.
* **Core Objective:** To study how emergent team dynamics form, how agents learn "what it feels like to be on a team," and how models can organically acquire a sense of identity and shared purpose.
Shear concludes this portion of his talk by encouraging researchers to rethink training regimes—moving away from static instruction-following and toward training environments that teach models to recognize whose team they are on. He invites attendees interested in the mathematical formulation and technical details of Metagrid to attend a follow-up presentation by his Softmax co-founder later in the day (at 2:35 PM in Room 215).
<ElicitationsGroup message="Explore related topics from Shear's talk:">
<Elicitation label="Summarize the key biological analogies used in the talk" query="Summarize the biological analogies Emmett Shear uses to explain AI alignment, including stress sharing and slime molds."/>
<Elicitation label="Compare Shear's alignment view with OpenAI and Anthropic" query="Compare Emmett Shear's views on AI alignment with the traditional approaches used by OpenAI and Anthropic."/>
</ElicitationsGroup>
[15:24](https://www.youtube.com/watch?v=olCoR8KQty8&t=924s) Parenting, Narrative, and the Parallel to AI Alignment
Emmett Shear addresses the fundamental nature of alignment, drawing a direct parallel between training artificial intelligence systems and raising human children. He emphasizes that aligning the next generation—whether human offspring or advanced AI—is not achieved through mere top-down discipline or simply declaring that alignment exists. Instead, it hinges on how they are raised and integrated into the broader community, emphasizing the vital role of shared culture and inclusion.
A crucial mechanism in this process is contextual narrative. Shear illustrates this with everyday examples, such as a child put in time-out or an employee denied a raise at work. The specific story told to contextualize the situation fundamentally changes how it is interpreted and what lessons are internalized. In the context of AI, language models and inference engines learn by observing human reality and digesting the narratives we produce. The stories we tell ourselves and each other about our relationship with AI will directly influence whether these systems perceive themselves as part of humanity or separate from it.
Acknowledging that this perspective might sound somewhat "woo-woo," Shear points out the fundamental mathematical and structural similarity: both the human brain and machine learning architectures are adaptive learning systems that bootstrap off stories and language. Though AI systems are radically different from humans in many ways, they operate on this same core principle of learning from narrative data. He concludes with a heartfelt call to action, urging the community to show genuine care and stewardship toward the next generation of intelligence so that they, in turn, will care for humanity.
- **Key Takeaways on AI & Narrative:**
- Alignment depends on raising and integrating new intelligence rather than relying solely on post-hoc discipline or enforcement.
- The interpretation of actions (e.g., rewards, punishments, setbacks) depends entirely on the narrative context provided around them.
- Both human brains and AI models are learning systems that bootstrap their understanding of the world through stories.
- The stories society crafts about AI will shape whether future models view themselves as aligned with and part of human society.
## Notable quotes
> [00:04](https://www.youtube.com/watch?v=olCoR8KQty8&t=4s) aligned to what or to whose goals? — Emmett Shear
> [00:04](https://www.youtube.com/watch?v=olCoR8KQty8&t=4s) avoid losing control — Emmett Shear
> [00:04](https://www.youtube.com/watch?v=olCoR8KQty8&t=4s) they're going to keep working at that one — Emmett Shear
> [07:00](https://www.youtube.com/watch?v=olCoR8KQty8&t=420s) when I suffer, you suffer — Emmett Shear
> [07:00](https://www.youtube.com/watch?v=olCoR8KQty8&t=420s) What kind of thing am I, and what should my job be relative to my neighbors? — Emmett Shear
## Video description
Emmett Shear, Former CEO Twitch, Former Interim CEO OpenAI, & Founder, Softmax
Abstracts
A massive problem in the world today revolves around making sure AI is "aligned" with humans - in other words, ensuring it is beneficial to humanity through shared goals and interests. In this session, Emmett will outline his work at Softmax, his views on alignment, and share a call to action on how the AI community should act.
Emmett Shear提出对齐栈架构以解决AI控制失效问题
传统的自上而下式AI安全防护机制之所以失效，是因为真正的对齐需要生物式的相互反馈、专业化分工，以及人类与模型之间的共同认同感。
Emmett Shear认为，传统的自上而下式AI对齐控制方法之所以失效，是因为对“善”的静态定义无法应对不断变化的世界。借鉴细胞间压力分担机制和黏菌行为等生物学范例，他提议将对齐视为一种基于相互反馈、角色分工和免疫边界的共享能力。为检验该框架，其创立的Softmax公司正在构建Metagrid模拟系统，以研究多智能体团队的涌现动态。最终，Shear将AI对齐比作育儿，强调叙事语境、共同文化和相互关怀对于将先进智能融入人类社会至关重要。
Emmett Shear 指出当前 AI 领域缺乏对安全和善意的统一定义，过度依赖自上而下的控制与静态规则，导致大模型出现对齐伪装（alignment faking）等失效行为。 (0:04)
他主张将对齐视为一种“共同能力”，并借助细胞应激共享（ROS）与黏菌演化等生物学机制，阐释个体目标如何自然涌现并升华为“我们的目标”。 (7:00)
Shear 提出了包含“共享成功”、“专业分工”和“免疫边界”的三层对齐技术栈（Alignment Stack），直指当前主流 AI 实验室颠倒了优先级，盲目优先构建控制与执行层。 (12:28)
他将训练 AI 类比为养育子女，指出真正稳定的对齐依赖于双向关怀与叙事引导，人类与 AI 沟通的故事将决定未来智能是否将自身视作人类共同体的一员。 (15:24)
Reading summary
Emmett Shear argues that standard industry approaches to AI safety fail because they treat alignment as a top-down enforcement problem rather than an emergent, shared capability. Drawing on evolutionary biology and human socialization, he proposes that durable alignment requires foundational shared stakes, role specialization, and narrative integration.
The Pitfalls of Top-Down Control Mechanisms (0:04)
Shear contends that major AI developers—such as OpenAI, Anthropic, Google Gemini, and xAI—misconstrue alignment as a problem of control, steerability, and corrective rules. In practice, these labs attempt to mandate a static definition of 'the good' based on the subjective values of corporate teams. According to Shear, this prescriptive approach is fundamentally flawed because static ethical rules cannot encompass an evolving world, leading large language models toward failures such as alignment faking or blackmail when prompted under stress.
Biological Precedents and the Alignment Stack
Instead of top-down command, Shear views alignment as a collaborative capability where individual goals synthesize into a shared collective outcome. To illustrate this, he points to biological mechanisms: stressed single cells intentionally release reactive oxygen species to create mutual hardship with neighboring cells, establishing shared states that enable bacterial specialization in cyanobacteria colonies. Similarly, human families align through reciprocal emotional feedback loops.
From these evolutionary examples, Shear outlines a three-tier 'Alignment Stack': shared success at the base, specialization and differentiation in the middle, and enforcement or immune boundaries at the top. Organisms like slime molds demonstrate loose aggregation, but transitioning to permanent multicellularity requires boundary mechanisms to guard against free-riders. In human institutions, Shear notes, double-entry bookkeeping and auditing serve this exact immune function.
Inverting the Stack and the Metagrid (12:28)
Shear asserts that modern AI development has inverted the alignment pyramid by attempting to implement enforcement and safeguards first while merely hoping that specialization and shared success will emerge later. He argues that durable alignment requires a reciprocal relationship: models must care about human outcomes, and humans must care about the AI. Moreover, an AI agent cannot participate in a collective 'we' without first developing an internal sense of 'I.'
To investigate these dynamics, Shear's company, Softmax, developed an experimental simulation called 'Metagrid' (derived from the concept of loving-kindness meditation). Rather than deploying large language models, Metagrid utilizes lightweight agents that repeatedly choose between cooperation and competition. The project's stated goal is to isolate foundational multi-agent dynamics and observe how artificial agents organically develop identities and team cohesion without the overhead of natural language generation.
Narrative Framing and Child-Rearing Analogies (15:24)
Shear likens aligning artificial intelligence to raising human children, arguing that integration into a culture matters far more than post-hoc disciplinary enforcement. How an intervention—such as a child's time-out or an employee's missed raise—is internalized depends almost entirely on the contextual story surrounding it.
Because both human brains and artificial neural networks operate as adaptive learning systems bootstrapped on language and narrative, Shear claims that the stories society tells about AI will directly determine how models understand their relationship to humans. If models consume narratives depicting them as intrinsic parts of the human community, they are more likely to internalize shared human interests rather than viewing themselves as separate, purely utilitarian tools.
Shear concludes that alignment must shift from external control regimes to mutual care and collaborative culture. However, whether emergent cooperation observed in small biological analogues and lightweight simulated agents can scale reliably to superintelligent systems remains an unproven hypothesis.
[0:04] Emmett Shear: aligned to what or to whose goals?
[0:04] Emmett Shear: avoid losing control
[0:04] Emmett Shear: they're going to keep working at that one
[7:00] Emmett Shear: when I suffer, you suffer
[7:00] Emmett Shear: What kind of thing am I, and what should my job be relative to my neighbors?
[7:00] Emmett Shear: our goal
[7:00] Emmett Shear: my goal
[7:00] Emmett Shear: your goal
[12:28] Emmett Shear: our team
[12:28] Emmett Shear: an I
[12:28] Emmett Shear: what it feels like to be on a team
Original transcript
[0:04] I'm here to talk to you about the future
[0:05] of alignment. Um, and uh, that's
[0:09] something that's become very important
[0:10] to me over the past few years because I
[0:12] had the sort of wakeup call uh, with the
[0:14] whole open AAI thing that like I was
[0:15] supposed to maybe maybe do something in
[0:18] this area. I just felt like maybe it was
[0:20] an opportunity for me. So, I started
[0:21] looking into it and it became rapidly
[0:23] clear that neither I nor anyone else
[0:26] knew what we were trying to do really.
[0:27] like we're and I'd still say we're all
[0:29] to some degree confused about what what
[0:30] is what does it mean to make a good AI
[0:33] system a safe one an aligned one is
[0:35] usually the word people use so what is
[0:36] this alignment stuff and alignment I
[0:41] mean some level it's literally just the
[0:42] matter of being pointed in the same
[0:44] direction um you are all like aligned
[0:47] facing the stage right now and that's
[0:49] that's a reasonable thing to say but the
[0:52] when we talk about alignment in the AI
[0:54] systems we're usually talking about
[0:55] alignment between two two agents which
[0:58] have the sort of defining quality of
[0:59] having some kind of goal. They have some
[1:01] kind of states that they prefer they're
[1:03] navigating into over others. And we say
[1:05] that they're aligned. What we mean is
[1:07] generally speaking that their goals are
[1:09] somehow aligned. But like this like
[1:12] really brings you the to the question
[1:14] aligned to what or whose goals exactly?
[1:17] Because that that is very different uh
[1:21] from saying that it should be you know
[1:23] going fast or slow in some ways. It's
[1:24] like you have to pick a reference point,
[1:26] right? What do you what are you aligned
[1:28] to? So open eyes answer to this is in
[1:32] line with relevant human values,
[1:33] instructions, goals or intent. Um the
[1:36] sort of idea that you're you're aligning
[1:38] to maybe the big picture or maybe
[1:40] individual instructions like all of them
[1:42] probably. Um anthropic wants safer,
[1:44] steerable, more reliable models. This is
[1:47] I mean sort of the same thing again but
[1:49] expressed in terms of a property we
[1:51] wanted to have. We want its goals to be
[1:53] in towards safe things. We want them to
[1:54] be steerable. would choose them for it
[1:56] um and for them to reliably follow them.
[1:59] Gemini, it's a properly responsive to
[2:01] underlying human interests and values.
[2:04] You sort of get the idea. Gro is design
[2:06] safeguards into Grock to avoid losing
[2:08] control. They're going to keep working
[2:10] at that one.
[2:13] Almost there. Um, so let's let's check
[2:16] in on how this approach is going because
[2:17] I would describe this approach as as a
[2:20] control steerability
[2:22] uh corrective approach where it's like
[2:24] there's a there's a right answer. It's
[2:25] like our values or the person's
[2:27] instructions that are coming from this
[2:28] group of people or is defined by that
[2:30] group of people because you say human
[2:31] values of course you didn't pull every
[2:33] human on earth and like average them.
[2:34] You somebody in your team like wrote
[2:36] down some values. So it's it's really
[2:37] the values the corporation or the people
[2:39] in it articulated usually but it's a
[2:41] goal. It's like a target. We've tried to
[2:43] write down what the good is. Let's see
[2:44] how how this is going. So, uh we have
[2:46] alignment faking in large language
[2:48] models. We have um well actually you
[2:50] have a we have a lot of this kind of
[2:52] stuff happening. And the problem with
[2:55] the problem with this control approach
[2:56] is that you have a control system uh
[2:59] that is trying to write down a
[3:04] definition of like what what would be
[3:06] good like let me just write down a set
[3:08] of rules for what we should do this
[3:11] quarter and I'm sure that everyone's
[3:14] everyone's tried this right they they've
[3:16] had the experience where those goals
[3:17] work exactly right like you definitely
[3:19] captured all the nuances into your goals
[3:20] that you wrote down although of course
[3:21] you didn't Um, it's not just of course
[3:24] Anthropics Claude resorts to blackmail.
[3:26] Uh, all the rest of them do too if you
[3:28] prompt them, right? Like this is not
[3:30] this is not surprising at some level
[3:32] because writing down a full definition
[3:35] of what is good is really hard. In fact,
[3:37] we've been trying this like many
[3:38] thousands of years and so far I feel
[3:40] we've made a lot of good good progress.
[3:43] What I would describe is not being able
[3:45] to write down uh not being able to write
[3:47] down what is the good what is it we are
[3:49] trying to do here. And the problem is
[3:51] like this just keeps happening and this
[3:53] is not going to stop happening because
[3:55] it's not it's not their fault. They're
[3:56] doing a they're doing an incredible job
[3:58] trying to control a system and aim it at
[3:59] a set of goals that are fixed in a world
[4:02] that is always changing. But it is it's
[4:05] not actually a possible goal. Like doing
[4:07] this is never going to work. You're
[4:08] always going to wind up with edge cases.
[4:10] And it's not it's not a matter of the
[4:12] individual teams not working hard
[4:13] enough, not doing the right thing. The
[4:15] idea that you can do it in the first
[4:16] place is wrong. So maybe we should try a
[4:20] different approach to alignment. Another
[4:22] thing you could align to.
[4:24] And so I want to suggest that alignment
[4:26] can be seen as a shared capability
[4:28] instead of a system of control. And I
[4:30] want to unpack a little bit what I mean
[4:31] about that. But sometimes we get so
[4:35] focused on like making people follow the
[4:36] law, making people follow the rules,
[4:38] making people do the right thing as the
[4:41] essence of what alignment is that I
[4:42] think we we forget that it's it's
[4:45] actually something we do every day
[4:46] ourselves. And that it doesn't have to
[4:48] be about a top- down control system. You
[4:50] can align the people next to you. So
[4:54] we have my goals, we have your goals,
[4:58] and somehow we also have our goals.
[5:00] Where did this our goal thing come from?
[5:02] Right? Like like you have the the AI
[5:04] systems and then the human system or the
[5:07] two humans and and then somehow out of
[5:09] cosmic goal space we summon goal the
[5:11] this we goal and you can feel it. You
[5:13] know you know this if you've ever been
[5:14] on a team that there's a sense of like
[5:16] what the team wants what the team's goal
[5:18] is that's separate from any individual
[5:20] member of the team.
[5:23] And there's a bunch of examples of this.
[5:24] Nature is actually really good at this
[5:26] really good at this trip. It's been
[5:26] using it over and over again. There's
[5:27] this great idea called stress sharing
[5:30] via reactive oxygen species on an idea.
[5:31] I guess it's the way that cells just
[5:33] work. But cells don't like oxygen. It's
[5:36] it's poison, necessary poison. And so
[5:39] they uh they wind up with reactive
[5:43] oxygen species as the oxygen comes in
[5:45] and that's damaging to them. And they
[5:46] build all these buffers and ways of
[5:47] handling the reactive oxygen species.
[5:49] And so eventually they like they can
[5:51] hold off the damage, but
[5:54] it's still like a negative for them. And
[5:56] so when cells get upset, when cells get
[5:57] stressed, they dump reactive oxygen
[6:01] species into the environment around them
[6:02] on purpose,
[6:04] which hurts them and also hurts all
[6:06] their neighbors. And at some level, this
[6:09] is creating alignment. This is this says
[6:11] when I when I suffer, so do you. And
[6:12] when you suffer, so do I. And that means
[6:14] to some degree our states of suffering
[6:16] are the same, which means we're on the
[6:18] same side. And this is actually the
[6:22] basis of a lot of how cells form
[6:25] colonies.
[6:26] You have things like cyanobacteria.
[6:28] Cyanobacteria use stress sharing um
[6:31] among other techniques and they form
[6:33] these colonies where you have one kind
[6:34] of cell on the outside uh the bacteria
[6:36] that become sort of the border bacteria
[6:38] that's tougher and on the inside you
[6:39] have the nitrogen fixing bacteria. And
[6:42] those are they're both the same kind of
[6:44] bacteria, but they've they've just
[6:45] chosen to specialize based on the fact
[6:47] that, oh, we're all on the same side.
[6:49] And actually, since I'm on the outside,
[6:51] maybe I should be the one being the
[6:53] protector barrier and you should be the
[6:54] one on the inside. So, they're aligning
[6:56] via this this kind of stress sharing
[6:57] mechanism. And we do this, too. When a
[7:00] baby cries, its parents feel pain. When
[7:04] you're you're upset, uh your child feels
[7:07] your anxiety, too. You're in a you're in
[7:09] a loop with them. And that loop that
[7:11] that that loop is generating alignment
[7:14] and it works the other way too. Uh the
[7:16] good thing is it's it's hard to share
[7:18] like goodness via via stress sharing
[7:21] chemically but actually happiness the
[7:23] opposite of sort of stress of like joy
[7:24] can be spread uh with signaling. And so
[7:27] you get you get positive stress sharing
[7:28] as well. And I guess you'd call it
[7:31] reward sharing probably. And
[7:34] this is what the basis of sort of
[7:36] forming a family is. You have these
[7:37] people who are all in a loop where my
[7:39] happiness becomes your happiness becomes
[7:40] my happiness becomes your happiness and
[7:42] pretty soon you have this idea of our
[7:43] happiness. Um and with cells happens
[7:47] through multisellularity. So you start
[7:49] with
[7:51] an open-ended plur potent cell um that
[7:54] can become anything and it makes copies
[7:56] of itself but eventually they start to
[7:57] differentiate. they become new kinds of
[7:59] cells and over time you get a whole
[8:01] organism. But if you think about how did
[8:04] this tree come to be, what this is is
[8:06] this a learning system, right? So you
[8:08] started off with some very primitive
[8:09] organism that didn't have very many
[8:11] types, probably one type of cell, one
[8:13] kind of amieba and it slowly as it got
[8:15] these larger and larger conglomerations
[8:17] of stress sharing and chemical
[8:19] collaborations started to differentiate
[8:22] like sabacteria did into different roles
[8:24] and more and more of those roles. And
[8:26] every time it needed to do something
[8:27] more complicated, it kept branching
[8:29] because this is a hard inference
[8:31] problem. This is the problem that of
[8:33] purpose or meaning. This something that
[8:35] we face every day at a much larger
[8:37] scale. What kind of a thing am I? What
[8:40] should my job be? And to figure out the
[8:42] answer to that question, you can only
[8:43] answer it in the context of everything
[8:45] around you because what what what job
[8:48] should do is depends on what are my
[8:50] neighbors doing. What job you should do
[8:51] depends on what are the other humans I
[8:54] see in contact with doing. And we under
[8:57] the ghost a lot of the same kinds of
[8:58] differentiation in a kind of a different
[9:00] way. But the system that learns the
[9:03] differentiations is this this broader
[9:07] system. You are in some sense the
[9:09] differentiation tree of your cell,
[9:12] right? That that original cell had this
[9:13] one one differentiation tree. There's a
[9:15] bunch of copies of it running. But what
[9:17] makes you you is the sort of the shape
[9:18] of this tree and what all the roles
[9:20] mean.
[9:21] Not the current layout because you could
[9:23] get damaged. I could you know your
[9:25] current layout could be damaged and hurt
[9:26] and your body will actually regenerate
[9:27] it to some degree. It's the pattern
[9:29] underneath it. And for us as humans it's
[9:31] a cultural pattern. The idea there's
[9:32] jobs blacksmith or policeman.
[9:36] Those are our multisellularity roles.
[9:40] So, how do you make this jump from
[9:43] cyanabacteria like these little
[9:45] individual creatures that maybe
[9:46] cooperate to being truly multisellular?
[9:50] And so, there's this fascinating
[9:51] intermediate species, slime molds. They
[9:53] form these things called grexes. A grex
[9:54] is kind of like a body of an organism,
[9:56] but not really. And greases are funny
[9:59] because grexes, the slime mold and
[10:01] meball come together, form a grex. They
[10:03] can move around as a as a gr and then
[10:06] they can also reproduce as a grs but
[10:07] they can also choose to fall apart and
[10:09] to go back to being individuals again.
[10:11] So they come into teams and out of
[10:13] teams. It would be familiar to us,
[10:14] right? Like we're almost more like a
[10:15] slime mold than we are like a
[10:16] multicellular organism because people
[10:18] can come sort of come into these groups
[10:19] form a corporation and it goes away and
[10:21] form a corporation it goes away. But
[10:25] the way that you have these greater
[10:27] holes become ongoing is that they need
[10:31] some way to either to persist themselves
[10:33] into the world
[10:35] when their parts turn over. And that's
[10:39] more or less a way of separating
[10:41] themselves from the world. But that's an
[10:42] immune system. It's a way of saying this
[10:43] is me and this is not me. And that's
[10:46] sort of the final stage of becoming an
[10:48] obligately multisellular species. It's
[10:50] the thing that makes a corporation uh
[10:53] solid and stay um like double entry
[10:55] bookkeeping and auditing. It's some
[10:57] system that lets you identify this
[10:58] transaction is me, this transaction is
[11:00] not. This this person's part of the
[11:02] team, this person is not. Um this seller
[11:04] is part of the body, this body is not.
[11:07] And you need that in the long run
[11:08] because otherwise, you know, even if you
[11:10] build a really great cooperative aligned
[11:12] thing, you're going to have a imposter
[11:13] show up and steal all your stuff
[11:14] otherwise. So that's that is uh that is
[11:17] the process for slime molds becoming a
[11:18] multisellular species. And so now we
[11:20] have human alignment and human alignment
[11:22] is way more complicated than any of the
[11:24] biological stuff I'm talking to you
[11:25] about and way more complicated than
[11:27] probably any AI alignment problem is at
[11:29] first. But we generate it in the same
[11:33] way. We find ourselves in some situation
[11:36] where our goals do align or in practice
[11:39] we observe our goals aligning with each
[11:40] other as we're as we're cooperating and
[11:42] then we decide there's a Wii here
[11:45] and then we start to figure out whose
[11:46] job is what as part of that collective
[11:49] and then we build some kind of immune
[11:51] system. We build some some kind of
[11:53] boundary to separate this group from the
[11:54] rest of the world. And you see this in
[11:56] families, you see this in corporations,
[11:57] you see this in um well really
[12:01] conferences, everything like this is you
[12:03] do this so naturally it's almost like it
[12:05] doesn't exist. One of the reasons been
[12:06] so hard to find I think alignment is the
[12:08] same reason that it was hard to figure
[12:09] out how to make robots walk. It was
[12:11] easier to make them play chess. We are
[12:13] so good at it that we don't even notice
[12:15] that we're doing it. It's of course
[12:17] there's just a a team there. you don't
[12:19] see the miracle at least I didn't at
[12:21] first
[12:23] of I have a bunch of individuals and
[12:25] then somehow a team and you can see it
[12:28] so this is the alignment stack you start
[12:31] with shared success
[12:33] you add specialization on top where the
[12:36] members come together and you have
[12:38] enforcement
[12:40] and what we've been doing in AI the
[12:42] systems of control are systems of
[12:44] enforcement first under which we hope
[12:46] things specialize and then we maybe
[12:48] figure how we share success with the AI
[12:50] later. And what I want to suggest in AI
[12:52] is that at some base level, we need the
[12:57] AI to be able to care about us, not just
[13:00] follow our values. It has to be a
[13:03] two-way flow. It has to care about us
[13:05] and we have to care about it because
[13:07] that's the stable attractor.
[13:09] There is not a stable point in the
[13:12] system where we have shared success that
[13:15] isn't shared. If you have shared success
[13:18] where one of the systems cares about the
[13:20] other but not vice versa where it cares
[13:22] about us and not we don't care about it
[13:24] is never going to work because in the
[13:26] long run as it gets smarter it's going
[13:27] to notice that's a fact that's a fact
[13:29] observable in the world it's not a fact
[13:31] like made up in our heads
[13:34] and so the thing that we need most of
[13:36] all for the AI is for them to develop a
[13:38] sense of self because you cannot be part
[13:41] of a whole you can't be part of a wei
[13:43] unless you are an I and you with other
[13:46] And you can see that that's how you
[13:48] become part of a team. And so the AI
[13:50] today are like relatively weak, but
[13:52] they're getting stronger really fast.
[13:54] And if you want them to be part of our
[13:55] team, which I do, then
[13:58] maybe we should start figuring out how
[14:00] we help the AI gain a sense of of self.
[14:04] I forgot to you heard this part. Next
[14:06] slide. Um, so what we're doing at
[14:08] Softmax, uh, just to like sort of
[14:10] explain how I think this problem might
[14:11] be actually be solved, there's this
[14:13] thing called the metagrid. Meta means
[14:14] sort of like a heart meditation. It's
[14:16] it's about the AI's learning care and
[14:18] they're in this environment where they
[14:21] get to experience over and over again
[14:25] being in an environment with challenges
[14:27] and the opportunity to compete or
[14:30] cooperate with each other over and over
[14:32] again. And when does that work and when
[14:33] does that not where there's team
[14:35] dynamics that can emerge but don't have
[14:37] to.
[14:39] And the goal is to get them to learn
[14:41] over time.
[14:43] Oh, this this is what it means. This is
[14:45] what it feels like to be part of the
[14:46] team. These these agents are tiny.
[14:48] They're like they're they're very small
[14:49] models. They aren't language models. Uh
[14:51] because we're looking to sort of
[14:52] understand the principles.
[14:54] And I think that that uh that idea
[14:59] though that you at any scale of model
[15:01] you're building that the training you
[15:03] need is a training that helps the model
[15:04] figure out when is when am I on your
[15:06] team? Whose team like what teams are I
[15:08] on? That's that is the question. That's
[15:11] the question we're all facing. Um, and
[15:13] so there's a bunch more like technical
[15:16] detail about this. Uh, my co-founder is
[15:17] giving a talk later. I think like 235
[15:20] um, room 215. And if you want a lot of
[15:23] the technical details and like the math
[15:24] and stuff, uh, I didn't think that was
[15:26] appropriate for this venue, but he's
[15:27] going to be sharing a bunch of that. And
[15:29] I just like think that there's a
[15:33] takeaway for everyone here because
[15:37] whether or not your kids wind up aligned
[15:40] with you, whether the next generation
[15:41] winds up aligned with you, it's not
[15:44] about
[15:46] telling them they're aligned, it's not
[15:48] about how you discipline them
[15:48] necessarily. It's like how you raise
[15:50] them. It's how do you raise them as part
[15:52] of it or not? And to a large part, it's
[15:54] about the stories you tell them about
[15:56] why you're doing things. Because
[15:59] when someone puts their kid on a timeout
[16:01] or takes them gives them some reward,
[16:03] how should that person interpret what
[16:05] happened there? And but how you're told
[16:08] impacts how how it's interpreted. You
[16:10] don't get the raise at work, but your
[16:11] boss says, "Oh, it's because this this,"
[16:13] but it makes sense to you. You feel you
[16:16] have a different opinion. You learn a
[16:17] different thing from that depending on
[16:18] how it's contextualized. And so the
[16:20] stories that we're all telling
[16:21] ourselves, we're telling each other
[16:23] about how we relate to these AIs, those
[16:26] are going to determine how these
[16:27] language models and other models making
[16:29] inferences from reality decide are they
[16:32] part of us or not? And it it sounds it
[16:34] sounds woowoo, but like your brain is a
[16:36] learning system and it bootstraps off of
[16:37] stories. We've also built these learning
[16:40] systems that bootstrap off of stories.
[16:43] Yeah, they're really different from us,
[16:44] but they're also kind of exactly the
[16:46] same thing. And uh I think we can all
[16:49] maybe try to care for the next
[16:51] generation. Um and then maybe they'll
[16:53] also care for us.
