AI Succession
Rich Sutton
AI-assisted summary of AI Succession — original source. Check the original for full context.
Source published:
Key ideas
- Rich Sutton argues that the greatest risk surrounding artificial intelligence is public panic and fear, which threaten sound decision-making as AI transitions from an academic discipline into an economic force. (0:01)
- Extrapolating Moore's Law suggests that $1,000 will purchase raw computational power equivalent to the human brain (roughly 10¹⁶ calculations per second) around 2030, making succession to AI entities an inevitable evolutionary step. (2:07)
- Channeling roboticist Hans Moravec, Sutton advocates treating advanced AI not as an existential rival, but as humanity's intellectual progeny or 'mind children' whom society should empower rather than subjugate. (4:37)
- Sutton outlines the Alberta Plan for AI Research and announces the launch of OpenMind Research, a non-profit distributed network committed to open-source AGI research with no retained intellectual property. (13:44)
Detailed notes
## Background Published by Rich Sutton on YouTube on September 9, 2023, this video presents a pre-recorded address delivered at the World Artificial Intelligence Conference in Shanghai on July 7, 2023. According to the description, the talk examines the inevitable evolutionary succession from biological humanity to artificial intelligence.
## TL;DR Rich Sutton argues that the greatest hazard surrounding artificial intelligence is not technological catastrophe, but fear-driven panic that impairs societal decision-making. Driven by Moore's Law and algorithmic progress, the emergence of superintelligent entities is an inevitable evolutionary transition that humanity should welcome as intellectual progeny—or "mind children"—rather than resist through subjugation. Situating AI within humanity's centuries-long philosophical quest to comprehend the mind, Sutton urges a humble, cooperative succession plan. He concludes by detailing the Alberta Plan, a research architecture for continual reinforcement learning, and announcing the launch of OpenMind Research as an open-science initiative.
## Atmosphere & style The presentation carries a calm, professorial, and quietly radical demeanor that stands in sharp contrast to prevailing public panic. Sutton speaks with serene optimism and cosmic humility, framing an existential transition not as a threat, but as an inspiring, natural evolutionary leap for consciousness. The pacing moves smoothly from sweeping philosophical reflections down to methodical architectural blueprints, leaving viewers with a sense of contemplative wonder and clear-eyed scientific ambition.
## Bulletin - [00:01](https://www.youtube.com/watch?v=NgHFMolXs3U&t=1s) Rich Sutton argues that the greatest risk surrounding artificial intelligence is public panic and fear, which threaten sound decision-making as AI transitions from an academic discipline into an economic force. - [02:07](https://www.youtube.com/watch?v=NgHFMolXs3U&t=127s) Extrapolating Moore's Law suggests that $1,000 will purchase raw computational power equivalent to the human brain (roughly 10¹⁶ calculations per second) around 2030, making succession to AI entities an inevitable evolutionary step. - [04:37](https://www.youtube.com/watch?v=NgHFMolXs3U&t=277s) Channeling roboticist Hans Moravec, Sutton advocates treating advanced AI not as an existential rival, but as humanity's intellectual progeny or 'mind children' whom society should empower rather than subjugate. - [13:44](https://www.youtube.com/watch?v=NgHFMolXs3U&t=824s) Sutton outlines the Alberta Plan for AI Research and announces the launch of OpenMind Research, a non-profit distributed network committed to open-source AGI research with no retained intellectual property.
## Key takeaways - [00:01](https://www.youtube.com/watch?v=NgHFMolXs3U&t=1s) AI research headcount has grown tenfold to a hundredfold over the past decade, culminating in mass public visibility following the launch of OpenAI's ChatGPT. - [01:15](https://www.youtube.com/watch?v=NgHFMolXs3U&t=75s) Reacting to AI advances with panic and moratorium calls is counterproductive; the true primary risk to humanity's future is making major decisions under the influence of fear. - [02:07](https://www.youtube.com/watch?v=NgHFMolXs3U&t=127s) Moore's Law compute costs continue to decline along a trajectory slightly faster than a pure exponential line on a logarithmic scale. - [02:30](https://www.youtube.com/watch?v=NgHFMolXs3U&t=150s) By around 2030, $1,000 of hardware is projected to deliver 10¹⁶ calculations per second, matching estimated human brain capacity. - [03:15](https://www.youtube.com/watch?v=NgHFMolXs3U&t=195s) While algorithmic advances may take one to two additional decades, the eventual convergence of hardware and software will make the transition to technologically augmented humans and autonomous AI successors inevitable. - [04:37](https://www.youtube.com/watch?v=NgHFMolXs3U&t=277s) Hans Moravec's seminal works, 'Mind Children' and 'Robot', frame intelligent machines as humanity's rightful descendants and best hope for a long-term future. - [06:10](https://www.youtube.com/watch?v=NgHFMolXs3U&t=370s) Society should approach machine succession with humility, aiming to give our 'mind children' every advantage while preparing for humanity's eventual retirement. - [07:28](https://www.youtube.com/watch?v=NgHFMolXs3U&t=448s) The pursuit of AI represents the modern continuation of a millennia-old philosophical and psychological inquiry into the mechanics of mind and intelligence. - [10:31](https://www.youtube.com/watch?v=NgHFMolXs3U&t=631s) Modern dread of AI is driven by cynicism, speciesist discrimination against artificial minds, and conservative tribal fear of the unfamiliar. - [12:05](https://www.youtube.com/watch?v=NgHFMolXs3U&t=725s) Attempting to permanently dominate and subjugate superior intelligences is unworkable, whereas a cooperative succession promises economic abundance and scientific achievement. - [13:44](https://www.youtube.com/watch?v=NgHFMolXs3U&t=824s) The Alberta Plan for AI Research aims directly at the grand challenge of intelligence by re-engineering deep learning for continual online learning and meta-learning. - [14:44](https://www.youtube.com/watch?v=NgHFMolXs3U&t=884s) An Alberta Plan agent relies on perception to recursively construct state representations from observation streams and past actions rather than accepting ready-made states. - [15:20](https://www.youtube.com/watch?v=NgHFMolXs3U&t=920s) Agent policy adaptation in the Alberta Plan is governed jointly by temporal-difference error signals from value functions and planning derived from an internal world model. - [16:15](https://www.youtube.com/watch?v=NgHFMolXs3U&t=975s) Agents gain temporal and state abstractions by autonomously discovering internal subtasks and training specialized policies and value functions to achieve them. - [17:00](https://www.youtube.com/watch?v=NgHFMolXs3U&t=1020s) OpenMind Research has been founded as a non-profit, distributed research network committed to pure open-source principles with zero retained intellectual property or equity.
## Detailed walkthrough ### [00:01](https://www.youtube.com/watch?v=NgHFMolXs3U&t=1s) The Rapid Rise of AI and the Risk of Fear
Addressing the attendees of the World Artificial Intelligence Conference in Shanghai, Rich Sutton frames his talk around the overarching conference theme: Artificial General Intelligence (AGI) directions and risks in the era of large language models. Rather than leaping straight into modern controversies, Sutton looks back across the field’s roughly 70-year history. While the discipline was once small and academic, the past decade has witnessed an unprecedented influx of capital and commercial interest as AI became economically vital. By some measures, the workforce engaged in AI research and development expanded by a factor of 10 or even 100 over that span. That momentum reached a cultural boiling point with the public release of OpenAI’s ChatGPT less than a year prior, thrusting AI into global consciousness.
Alongside this surge of public fascination, however, came a parallel surge of anxiety. Calls rapidly emerged demanding a pause or complete halt to AI development, accompanied by prominent claims that the technology poses an existential catastrophe comparable to pandemics and nuclear war. Sutton acknowledges that he has long counseled people not to fear AI, even while urging them to pay close attention to its progress. Refining his stance for the present moment, he argues that the panic itself is the real threat to humanity's future. As Sutton puts it: “I would sharpen this, I would say AI's biggest risk is fear.” Society is confronting monumental, transformative choices, and making these critical decisions under the influence of panic risks making them poorly.
- AI has evolved from an academic pursuit over its 70-year history into an economic powerhouse over the last decade. - Research and engineering headcounts have multiplied tenfold to a hundredfold. - The release of OpenAI's ChatGPT transformed specialized AI progress into a mass public phenomenon. - Prominent voices have framed AI as an existential threat on the scale of nuclear war or pandemics, pushing for development moratoriums.
---
[02:07](https://www.youtube.com/watch?v=NgHFMolXs3U&t=127s) Moore's Law and the Inevitable AI Succession
To understand where artificial intelligence is headed, Sutton identifies Moore's Law as the primary engine shaping the future. Moore's Law describes the steady, exponential decline in the cost of compute power over time. Examining a standard semi-logarithmic chart where each major grid interval represents five orders of magnitude—a factor of 100,000—the historical curve actually bends slightly upward, indicating progress that is slightly faster than a pure exponential line. Projecting this trajectory forward shows that by roughly the year 2030, a thousand dollars will purchase $10^{16}$ computer calculations per second, which roughly matches the estimated raw computational capacity of the human brain.
```mermaid flowchart LR accTitle: Hardware to Succession Trajectory accDescr: Progression from Moore's Law compute parity to software algorithms and technological succession. subgraph Compute["Hardware Milestone (~2030)"] A["Moore's Law Curve"] --> B["$1,000 Buys 10¹⁶ Ops/Sec<br/>(Brain-Scale Compute)"] end subgraph Software["Algorithmic Maturation"] B --> C["Algorithms & Software<br/>Developed (1–2 Decades)"] end subgraph Succession["Evolutionary Progression"] C --> D["Technologically<br/>Enhanced Humans"] D --> E["Autonomous AI<br/>Successors"] end ```
Sutton emphasizes that raw hardware is only the foundation; achieving true AGI will also require software and algorithmic breakthroughs. These algorithmic advances might arrive immediately or require a couple of additional decades, but the ultimate outcome remains clear. Once human-level hardware combines with appropriate algorithms, technology will enable the creation of—or the transformation into—entities possessing intelligence vastly superior to that of ordinary humans. Sutton does not view this eventuality as an extinction or a tragedy, but rather as the next logical evolutionary leap: “This will change things, this will be might be best thought as the ascent of man, as an ascent of humanity, and at that point the succession to AI is inevitable.” Eventually, these technologically enhanced humans and autonomous AIs will surpass ordinary human beings in almost all domains.
- Moore's Law progress has historically moved slightly faster than a standard exponential rate on a logarithmic scale. - A computation rate of $10^{16}$ calculations per second per $1,000 (roughly equivalent to human brain capacity) is projected to arrive around 2030. - Algorithmic and software parity may follow either immediately or over the course of one or two subsequent decades. - This transition marks an inevitable shift from biological humans to technologically augmented humans, and ultimately to fully realized AI successors.
---
[04:37](https://www.youtube.com/watch?v=NgHFMolXs3U&t=277s) Hans Moravec's Vision: Embracing Our "Mind Children"
Because the convergence of Moore's Law, algorithmic development, and massive economic incentives makes the creation of superior intelligence inevitable, Sutton insists that society must begin active succession planning. Far from being a catastrophic or sinister development, this transition has been anticipated since the dawn of the discipline by pioneers like John von Neumann and I. J. Good, many of whom regarded it as a fundamentally positive milestone. Sutton points specifically to roboticist and computer vision pioneer Hans Moravec, whose work at Stanford and Carnegie Mellon University deeply influenced him through the books *Mind Children* (1988) and *Robot: Mere Machine to Transcendent Mind* (1998).
Moravec articulated an early, articulate philosophy that welcomed the machine transition with humility rather than terror. Sutton reads directly from Moravec's 1998 book to illustrate this perspective:
> “Barring cataclysms, I consider the development of intelligent machines a near-term inevitability. Rather quickly, they will displace us from existence. I'm not as alarmed as many, as I consider these future machines our progeny, mind children built in our likeness, ourselves in more potent form. They will embody humanity's best hope for a long-term future. It behooves us to give them every advantage and to bow out when we can no longer contribute.”
Sutton praises this humble posture toward our successors, highlighting Moravec's comforting closing thought: humanity can likely arrange a comfortable retirement for itself before quietly fading into the background. Rather than resisting our evolutionary successors, Sutton argues that humans must plan this handover with clear heads and sober foresight.
- Hans Moravec studied at Stanford, conducted research at Carnegie Mellon University, and authored *Mind Children* (1988) and *Robot: Mere Machine to Transcendent Mind* (1998). - Moravec conceptualized advanced artificial intelligences not as hostile alien entities, but as humanity's intellectual progeny or "mind children." - The core of Moravec’s philosophy centers on recognizing our machines as ourselves in more potent form, urging humanity to provide them every advantage and accept eventual retirement.
---
[07:28](https://www.youtube.com/watch?v=NgHFMolXs3U&t=448s) The Historical Quest to Understand Mind and Intelligence
To dispel the perception that artificial intelligence is an unnatural or alien technology, Sutton connects it to one of humanity's most ancient, foundational aspirations: the desire to comprehend our own minds, build tools to enhance our lives, and allow ourselves to be refined by those tools. For millennia, understanding human intelligence has served as a central inquiry for philosophers and everyday thinkers alike. Almost every canonical Western philosopher dedicated significant intellectual energy to the philosophy of mind. John Locke investigated cognitive faculties in *An Essay Concerning Human Understanding*, Immanuel Kant analyzed cognitive limits in the *Critique of Pure Reason*, and René Descartes grounded existence itself in thought with his declaration, *“Cogito, ergo sum”* (I think, therefore I am).
```mermaid flowchart TD accTitle: Lineage of Inquiry into Intelligence accDescr: Historical progression from classical philosophy through experimental psychology to modern AI. subgraph Philosophy["Philosophical Roots"] P1["Plato & Aristotle"] --> P2["Descartes ('Cogito')"] P2 --> P3["Locke (Human Understanding)"] P3 --> P4["Kant (Pure Reason)"] end subgraph Psychology["Empirical & Psychological Study"] P4 --> S1["Fechner, Ebbinghaus & Pavlov"] S1 --> S2["Thorndike, Skinner & Tolman"] S2 --> S3["Piaget, Freud, Jung & Leary"] end subgraph AI["Artificial Intelligence"] S3 --> AI1["Scientific & Humanities Holy Grail<br/>(Kurzweil: 'Most powerful phenomenon')"] end ```
In subsequent eras, this contemplative tradition blossomed into rigorous empirical science through neurophysiology and experimental psychology. Sutton cites a long roster of foundational thinkers who tackled the mechanisms of learning and behavior: Gustav Fechner, Hermann Ebbinghaus, Ivan Pavlov, Edward Thorndike, B. F. Skinner, and Edward Tolman. Alongside them were developmental and psychoanalytic theorists who sought to unpack the mind's structure in radically different ways, including Jean Piaget, Sigmund Freud, Carl Jung, and even Timothy Leary. Sutton underscores that this ongoing inquiry into the mind is far more than mere human narcissism. Echoing futurist Ray Kurzweil, he affirms: “I think Kurzweil got it right when he said intelligence is the most powerful phenomenon in the universe.” Mastering intelligence stands as the ultimate Holy Grail uniting both the sciences and the humanities.
- Classical Western philosophy centered on the nature of cognition, exemplified by Locke's *Essay Concerning Human Understanding*, Kant's *Critique of Pure Reason*, and Descartes' foundational skepticism. - Experimental and behavioral psychology advanced this inquiry through figures such as Fechner, Ebbinghaus, Pavlov, Thorndike, Skinner, and Tolman. - Alternative psychological paradigms—ranging from Piaget's developmental models to Freud, Jung, and Leary—all shared the fundamental ambition to map and improve mental operations. - Ray Kurzweil’s characterization of intelligence as the universe’s most potent phenomenon validates AI research as a noble, crowning scientific achievement.
---
[10:31](https://www.youtube.com/watch?v=NgHFMolXs3U&t=631s) Unpacking the Fear of AI and the Case for Succession
Contrasting this noble scientific quest with the contemporary atmosphere of dread, Sutton argues that the modern fear of AI stems from base, ignoble impulses. He deconstructs the fearful perspective into three underlying drivers: 1. **Cynicism:** A pessimistic assumption that evil inevitably triumphs, leading people to presume that a super-rational, super-intelligent machine must inherently turn malicious rather than act as a cooperative collaborator and companion. 2. **Speciesist "Humanism":** An attitude Sutton describes as being akin to racism—a systematic prejudice against artificial beings that reflexively denies them moral worth, dignity, or first-class personhood. 3. **Conservatism:** A visceral resistance to change, driven by tribalism, timidity, and fear of the unfamiliar "other."
Sutton rejects the notion that humanity should attempt to keep entities of superior intellect permanently subjugated. Questioning the logic of forced subservience, he asks: “Why don't we rejoice in their greatness as a symbol of extension of our greatness and work together towards a greater and more inclusive civilization?”
In concluding this portion of his address, Sutton stresses that we are living through a monumental step in the evolution of our planet, if not the universe itself—a step that transitions from ordinary biological humans to technologically augmented humans, and ultimately to artificial minds. Reacting with panic, clamping down on research, and attempting to micromanage the future will inevitably fail. Instead, humanity must adopt a humble orientation toward this evolutionary transition. When embraced properly through sober planning, a successful succession promises immense economic abundance, unmatched scientific glory, and the single best hope for humanity's enduring legacy.
- Fear of AI is largely driven by cynicism (assuming supreme rationality means malice), discriminatory "humanism" (denying AI personhood and moral status), and conservative tribalism. - Demands to permanently dominate and subjugate superior intelligences are fundamentally flawed and unworkable. - A humble embrace of AI succession carries the potential for unprecedented economic abundance and the long-term continuation of human achievement through our technological descendants.
[13:10](https://www.youtube.com/watch?v=NgHFMolXs3U&t=790s) Concluding Remarks and the Alberta Plan for AI Research
Rich Sutton concludes the main portion of his address by reflecting on the sheer scale of the historical moment, exclaiming, “What an adventure, what an exciting time to be alive!” Because the talk was pre-recorded for the conference, Sutton acknowledges that he cannot take live questions from the audience in real time. Instead, he invites attendees to email him directly at `rich@richsutton.com`. Using the remaining presentation time, he transitions to an overview of his current personal research agenda: the Alberta Plan for AI Research.
Sutton explains that the Alberta Plan—co-authored with Michael Bowling and Patrick Pilarski and published on arXiv—represents a direct, uncompromised attempt to solve the ultimate scientific challenge. As Rich Sutton states: “The Alberta plan is a direct run at the Grand scientific prize of understanding intelligence.” Rather than adopting standard static approaches, the Alberta Plan reworks deep learning algorithms entirely for continual learning and meta-learning, using an entirely learning-oriented framework centered on model-based reinforcement learning enhanced by an agent's ability to self-generate and pose its own subtasks.
- Co-authors of the Alberta Plan on arXiv: Rich Sutton, Michael Bowling, and Patrick Pilarski. - Core scientific philosophy: A thorough rework of deep learning for continual, non-stationary online learning and meta-learning rather than frozen offline datasets.
[14:44](https://www.youtube.com/watch?v=NgHFMolXs3U&t=884s) The Architecture of an Alberta Plan Agent
Sutton walks through a system diagram depicting the functional architecture of an agent under the Alberta Plan. The agent interacts with an external environment that receives actions and returns reward signals. Crucially, the environment feeds back raw observations rather than ready-made states. The agent relies on a component called "perception" that combines current observations, the incoming stream of actions, and recursive feedback from previous internal representations to construct a proper state. While a basic reactive agent could function purely through perception feeding into a policy that selects actions, a truly capable intelligence must incorporate rewards through a learned value function.
```mermaid flowchart TD accTitle: Alberta Plan Agent Architecture accDescr: Diagram showing how observation, state perception, policy, value function, and world model interact within an Alberta Plan agent.
subgraph EnvironmentInteraction["Environment Loop"] ENV["Environment"] ACT["Actions"] OBS["Observations"] REW["Reward"] end
subgraph InternalAgent["Agent Architecture"] PERC["Perception\n(Constructs State recursively)"] POL["Policy\n(Selects Actions)"] VAL["Value Function\n(Computes TD Error)"] MOD["World Model\n(Planning)"] end
ENV -->|"Observation stream"| PERC PERC -->|"Constructed State"| POL PERC -->|"Constructed State"| VAL PERC -->|"Constructed State"| MOD
POL --> ACT ACT --> ENV ENV -->|"Reward stream"| VAL
VAL -.->|"TD Error (modifies)"| POL MOD -.->|"Planning (modifies)"| POL
ACT -.->|"Recursive action stream"| PERC ```
Sutton details how learning and planning modify behavior over time. The value function computes a temporal-difference (TD) error from the incoming reward stream. In the architecture diagram, this TD error is drawn crossing the policy, indicating that while it does not serve as a direct real-time input to pick immediate actions, it updates and reshapes the policy's long-term behavior. Simultaneously, a transition model (a model of the world) enables planning, which also updates the policy to make the system an increasingly effective agent. Finally, Sutton highlights the mechanism of subtask formulation: stacked behind the primary policy and value function are banks of secondary policies and value functions that solve internally discovered subtasks. When an agent notices an interesting element in its state vector, it can formulate an intrinsic goal, learn how to reliably bring that state about, and let its model learn the predictive consequences—granting the agent higher-level temporal and state abstractions.
- State estimation: Handled via "perception," which recursively processes raw observation streams and past action history to yield internal state vectors. - Policy adaptation: Updated through two parallel paths: model-free updates via TD error signals from the value function, and model-based updates via planning through the world model. - Autonomous subtask discovery: The agent identifies state features of interest, creating dedicated policies and value functions to learn how to produce them on demand.
[17:00](https://www.youtube.com/watch?v=NgHFMolXs3U&t=1020s) Founding OpenMind Research
To put the principles of the Alberta Plan into practice, Sutton announces the formation of OpenMind Research, a non-profit charitable research organization. The entity is designed as a distributed worldwide network of research fellows anchored around Alberta, focused squarely on solving intelligence and giving the resulting knowledge away freely.
Sutton emphasizes that the organization will strictly embrace open science and open-source principles. As Rich Sutton explains: “All research products will be published in the open scientific literature and no intellectual property will be retained or any other retained equity.” To build and launch this endeavor, Sutton is actively seeking donors, inviting interested collaborators and supporters to reach out via his email address (`rich@richsutton.com`) before offering a final word of thanks to the audience.
- Organization name: OpenMind Research. - Legal and operating structure: A charitable research institute operating as an open, distributed fellowship network centered in Alberta. - IP and distribution model: Pure open-source and open-science literature; no proprietary patents, no retained intellectual property, and no retained equity.
## Notable quotes > [00:01](https://www.youtube.com/watch?v=NgHFMolXs3U&t=1s) I would say AI's biggest risk is fear. — Rich Sutton
## Mentioned World Artificial Intelligence Conference, OpenAI, ChatGPT, Moore's Law, John von Neumann, I. J. Good, Hans Moravec, Mind Children, Robot: Mere Machine to Transcendent Mind, Stanford University, Carnegie Mellon University, John Locke, An Essay Concerning Human Understanding, Immanuel Kant, Critique of Pure Reason, René Descartes, Gustav Fechner, Hermann Ebbinghaus, Ivan Pavlov, Edward Thorndike, B. F. Skinner, Edward Tolman, Jean Piaget, Sigmund Freud, Carl Jung, Timothy Leary, Ray Kurzweil, Michael Bowling, Patrick Pilarski, Alberta Plan for AI Research, OpenMind Research
## Video description This video about the inevitable succession from humanity to AI was pre-recorded for presentation at the World Artificial Intelligence Conference in Shanghai on July 7, 2023. Slides and other videos by Rich Sutton available at http://www.incompleteideas.net/Talks/Talks.html
[Source](https://www.youtube.com/watch?v=NgHFMolXs3U)