Future of Life Institute Urges Compute Limits to Halt Autonomous AGI

Keep the Future Human: Why and How We Should Not Build AGI and Superintelligence, and What We Should — Ai4

Ai4

Unchecked autonomous general intelligence will inevitably trigger uncontrollable superintelligence and replace human labor, demanding strict hardware thresholds and liability rules to force developers toward bounded tool systems.

AI-assisted summary of Keep the Future Human: Why and How We Should Not Build AGI and Superintelligence, and What We Should — original source. Check the original for full context.

Source published:

Detailed notes

## Background Published on YouTube by Ai4 on August 29, 2025, this presentation features Anthony Aguirre, Executive Director of the Future of Life Institute. According to the description, Aguirre argues that humanity must actively choose to close the gates to smarter-than-human, autonomous AGI and uncontrollable superintelligence, focusing instead on powerful, trustworthy AI tools that empower and complement human society.

## TL;DR Aguirre challenges the fatalistic view that AGI is either inevitable or impossible, defining autonomous general intelligence as the hazardous intersection of high intelligence, generality, and independent autonomy designed to substitute human labor. Pointing to compounding benchmarks like METR's doubling task horizons, he warns that AGI could arrive within one to two years and rapidly trigger an uncontrollable leap to superintelligence through recursive self-improvement. To avert catastrophic societal vulnerabilities, monopolization, and total loss of human agency, he outlines concrete governance mechanisms including compute thresholds, strict liability, and mandatory controllability standards. Ultimately, he urges humanity to reject species-level replacement and intentionally pursue the 'Tool AI' paradigm—building bounded, purpose-fit systems that keep humans firmly in the driver's seat.

## Atmosphere & style The talk carries an urgent, sober, and intellectually incisive tone that avoids sensationalist panic while confronting existential stakes with absolute clarity. Aguirre delivers his thesis with the measured, methodical precision of a physicist and policy strategist, systematically dismantling industry rationalizations with structured visual frameworks and empirical trendlines. It feels like an authoritative, high-stakes briefing that cuts through Silicon Valley hype to frame technological development as an urgent moral and political choice.

## Key takeaways - [00:05](https://www.youtube.com/watch?v=jjdIrKnTnOI&t=5s) The public debate around AGI is trapped in a false dichotomy between inevitable destiny and impossible hype, both of which dangerously erode human agency. - [01:18](https://www.youtube.com/watch?v=jjdIrKnTnOI&t=78s) Autonomous General Intelligence is defined at the intersection of high intelligence, high generality, and high autonomy, occupying the exact functional niche of human workers. - [03:06](https://www.youtube.com/watch?v=jjdIrKnTnOI&t=186s) Massive capital expenditure in AI infrastructure is economically driven by the goal of capturing global labor markets by replacing human wage earners with autonomous agents. - [06:40](https://www.youtube.com/watch?v=jjdIrKnTnOI&t=400s) AI performance on advanced reasoning benchmarks is surging, with the autonomous task horizon doubling roughly every six months toward full-work-week independence. - [08:22](https://www.youtube.com/watch?v=jjdIrKnTnOI&t=502s) Transitioning from automating discrete tasks to automating entire human roles introduces severe civilizational shocks due to digital agents scaling with zero marginal labor cost. - [10:00](https://www.youtube.com/watch?v=jjdIrKnTnOI&t=600s) Near-term intermediate risks include epistemic collapse, pervasive mass surveillance, democratized catastrophic capabilities like bioweapons, and automated military escalation. - [12:10](https://www.youtube.com/watch?v=jjdIrKnTnOI&t=730s) Once AI achieves human equivalence, recursive algorithmic self-improvement in software could transition AGI to uncontrollable superintelligence in as little as a year. - [12:10](https://www.youtube.com/watch?v=jjdIrKnTnOI&t=730s) Controlling a superintelligence is structurally impossible, as a simpler, slower biological species cannot maintain authority over a vastly faster, superior cognitive entity. - [14:15](https://www.youtube.com/watch?v=jjdIrKnTnOI&t=855s) Runaway scaling can be governed through physical semiconductor supply chain compute thresholds, mandatory controllability standards, and strict developer liability. - [17:22](https://www.youtube.com/watch?v=jjdIrKnTnOI&t=1042s) The 'Tool AI' framework advocates for purpose-fit systems with provable controllability guarantees commensurate with their power, keeping humans firmly in the driver's seat. - [19:24](https://www.youtube.com/watch?v=jjdIrKnTnOI&t=1164s) Constructive Tool AI can be channeled into solving epistemic crises with auditable citations, enabling large-scale civic deliberation, and accelerating targeted medical research. - [21:34](https://www.youtube.com/watch?v=jjdIrKnTnOI&t=1294s) Humanity faces a momentous choice between building empowering, bounded tools or racing toward civilization-ending, uncontrollable superintelligence.

## Detailed walkthrough ### [00:05](https://www.youtube.com/watch?v=jjdIrKnTnOI&t=5s) The False Dichotomy: Inevitable vs. Impossible AGI

When approaching the subject of Artificial General Intelligence (AGI), public and industry discourse often feels fractured into two completely incompatible universes talking straight past each other. In one universe, AGI is framed as an inevitable, imminent destiny—an unstoppable next step that will arrive soon and be universally marvelous. In the opposing universe, AGI is dismissed as sheer sci-fi hype; the systems we have are treated as nothing more than "glorified autocomplete," fundamentally incapable of reaching human-level cognition across any domain. Curiously, both viewpoints produce the exact same tranquilizing side effect: they let society completely off the hook. If AGI is an unalterable natural law, humans have no agency to steer it; if it is impossible, there is nothing to worry about in the first place.

The reality, however, rejects this binary comfort. AGI is neither impossible nor an inevitable destiny. It is technically feasible, potentially arriving quite soon, yet whether humanity proceeds into building it and subsequent superintelligence remains an active, momentous species-level choice. Shrugging off that decision forfeits our responsibility at the very moment deliberate human agency is most required.

[01:18](https://www.youtube.com/watch?v=jjdIrKnTnOI&t=78s) Redefining AGI: The Autonomous General Intelligence Venn Diagram

While conventional definitions of AGI range from fuzzy notions of "human-levelish AI" to economically oriented targets like autonomously performing all economically valuable human work, a clearer conceptual framing looks at AGI as *autonomous* general intelligence. This manifests at the strict intersection of three foundational capabilities: high intelligence (the ability to execute specific individual tasks exceptionally well), high generality (the breadth of distinct tasks an agent can handle or learn), and high autonomy (the capacity to carry out extended, complex tasks without human oversight or intervention).

```mermaid flowchart TD accTitle: The Three Pillars of Autonomous General Intelligence A["High Intelligence<br/>Task proficiency e.g., AlphaFold"] --- B["High Generality<br/>Breadth of domains e.g., Base LLMs"] B --- C["High Autonomy<br/>Independent execution e.g., Waymo, Agents"] C --- A ABC["Autonomous General Intelligence<br/>Direct Human Substitute"] --- A ABC --- B ABC --- C ```

Historically, the evolution of artificial intelligence has populated different peripheral regions of this Venn diagram. AlphaFold achieved extraordinary, superhuman intelligence within an extremely narrow domain. Basic autonomous systems like Roombas navigate physical spaces independently despite very low intelligence, while self-driving vehicles like Waymo incorporate somewhat higher intelligence within dedicated functional boundaries. General-purpose models like the early AI infrastructure underlying Google covered immense breadth without high autonomous agency or deep reasoning. More recently, frontier models such as ChatGPT, Claude, and Gemini settled squarely into the overlap between high intelligence and high generality. The critical missing component in frontier deployments has been the third dimension: high autonomy.

[03:06](https://www.youtube.com/watch?v=jjdIrKnTnOI&t=186s) The Economic Engine: The Trillion-Dollar Bet on Replacing Human Labor

Adding high autonomy to systems that already possess high intelligence and generality creates a profound qualitative shift. Autonomy removes the bottleneck of human review, unlocking dramatic execution speed while simultaneously making systems vastly harder to rein in or control. Crucially, the intersection of all three traits occupies the exact ecological niche of human beings. When a machine inhabits the center of that diagram, it ceases to be a specialized tool and becomes a direct, one-for-one substitute for human workers.

This substitution reveals the underlying economic logic behind the current computing boom. Technology giants are not pouring hundreds of billions of dollars of capital expenditure into massive data centers merely to monetize $10- or $20-a-month chatbot subscriptions, nor solely to supply API backends for software startups. The real financial prize is capturing a massive slice of the tens of trillions of dollars circulating in the global labor market by systematically replacing human wage-earners with autonomous machines. This enormous financial incentive explains why industry leaders relentlessly herald the "year of the agent," pushing through difficult engineering obstacles to transform passive text models into self-directed actors.

[04:31](https://www.youtube.com/watch?v=jjdIrKnTnOI&t=271s) Gauging the Frontier: Rapid Leaps in Intelligence and Generality

Evaluating whether AGI is near requires examining progress across its three constituent dimensions. On the intelligence axis, mathematical reasoning provides a rigorous, objective benchmark. Performance on competitive high-school mathematics benchmarks has surged from near-zero accuracy to near-total saturation in a brief span. Both Google DeepMind and OpenAI have demonstrated gold-medal-tier performance on problems from the International Mathematical Olympiad (IMO). Furthermore, evaluations on elite benchmarks like FrontierMath—which contain problems so advanced that even practicing theoretical physicists cannot immediately parse what is being asked—demonstrate that while systems are not yet pushing the absolute research frontier of mathematics, their trajectory upward is steep and relentless.

```mermaid flowchart LR accTitle: Progression of AI Capabilities Across Three Dimensions subgraph Intelligence I1["High School Math: Near 0%"] --> I2["IMO Gold Medal Performance"] end subgraph Generality G1["GPQA Science: Random Guessing"] --> G2["Above Professional Human Benchmark"] end subgraph Autonomy A1["Task Horizon: Seconds (GPT-2)"] --> A2["Current Horizon: ~2 Hours"] A2 --> A3["Projected Horizon: 32 Hours (Full Work Week)"] end ```

A parallel surge characterizes the generality dimension, visible in evaluations like PhD-level science benchmarks across chemistry, physics, and biology. In roughly two years, frontier models shifted from random guessing to scoring well beyond the average professional human baseline. The upper performance envelope across leading frontier labs shows no sign of leveling off or hitting diminishing returns. While correctly answering standardized PhD-level examination questions is not identical to conducting original scientific discoveries, sustaining this unbroken rate of progress for another two years brings AI systems remarkably close to the cognitive capacities of professional researchers.

[06:40](https://www.youtube.com/watch?v=jjdIrKnTnOI&t=400s) The Autonomy Curve: METR Benchmarks and the Compression of Timelines

Autonomy has historically proved difficult to quantify, but organizations like METR (Model Evaluation and Threat Research) have established rigorous operational metrics. METR tests models against diverse suites of end-to-end tasks, measuring the effective time a competent human would require to complete tasks that an AI can finish at a 50% success rate. This metric—the autonomous task horizon—has scaled exponentially from mere seconds during the era of GPT-2 to approximately two hours in state-of-the-art models, doubling roughly every six months.

Extrapolating that six-month doubling rate across the next two years yields a sixteen-fold increase: - Current autonomous task horizon: ~2 hours - Projected task horizon in ~24 months: ~32 hours of sustained, independent labor - Equivalent real-world labor: An entire full-time work week completed without human intervention

Given this compounding progress across intelligence, generality, and autonomy, the arrival of genuine AGI is a near-term prospect. The first functional AGI systems could emerge within one to two years; expecting the process to take more than a decade appears almost completely detached from empirical trends. Because the implications of such systems feel overwhelmingly massive, people naturally recoil from taking them seriously, dismissing them because "they feel too big to be true."

[08:22](https://www.youtube.com/watch?v=jjdIrKnTnOI&t=502s) Beyond Tools: The Civilizational Shock of Automating Roles

Confronting the reality of AGI means acknowledging that society will not be dealing with passive software utilities, but rather with autonomous, goal-driven entities that can execute the full gamut of human cognitive labor. Unlike biological brains, digital intelligences can be accelerated by orders of magnitude simply by provisioning more compute—an advantage no human can match. They can be duplicated indefinitely, modified, and scaled across data centers at zero marginal labor cost.

```mermaid flowchart TD accTitle: Transition from Task Automation to Role Automation A["Automated Tools<br/>Task-level acceleration"] --> B["Automated Agents<br/>Full role & personnel replacement"] B --> C1["Hyper-Scale Economic Displacement"] B --> C2["Information & Epistemic Collapse"] B --> C3["Concentration of Power in Mega-Compute Hubs"] ```

The scale and velocity of deployment will amplify this shock wave. With products like ChatGPT having already amassed hundreds of millions of weekly active users, architectural breakthroughs no longer endure long incubation periods; they are propagated across global society in days. While this transition will foster technological acceleration and economic productivity, shifting from automating discrete tasks to automating entire human roles will destabilize civilization's structural foundations.

[10:00](https://www.youtube.com/watch?v=jjdIrKnTnOI&t=600s) The Spectrum of Catastrophic Vulnerabilities

Before even considering extreme existential scenarios, the intermediate societal risks provoked by human-level autonomous systems represent acute structural dangers:

- **Epistemic and Authentication Collapse:** The total destruction of online trust. Once text, audio, and video become indistinguishable from authentic human output—when even telltale artifacts like excessive em dashes disappear—citizens will have no reliable way to verify whether digital experiences, news, or correspondents are human or synthetic. - **Pervasive Mass Surveillance:** Intelligence scaled so cheaply that every citizen can be continuously monitored, analyzed, and predicted by a dedicated, personalized AI agent that is faster, smarter, and broader than the target. - **Democratized Catastrophic Capabilities:** Rogue states, fringe groups, or lone terrorists gaining friction-free access to top-tier expertise in automated cyberwarfare, advanced virology, and novel weapon design. - **Algorithmic Disenfranchisement of Public Discourse:** Public forums flooded and steered by persuasive, manipulative synthetic voices with no biological or ethical stake in human society. - **Severe Economic Displacement:** Rapid elimination of the market value of human labor across months rather than decades, forcing conventional enterprises to compete against entirely autonomous corporations that execute decisions a hundred times faster. - **Extreme Hyper-Monopolization:** Massive consolidation of the global economic apparatus into the hands of a tiny cadre of frontier tech corporations managing planetary-scale data centers. - **Automated Military Escalation:** The delegation of lethal battlefield command and strategic decision-making to autonomous systems, driven by the inescapable military penalty of keeping slow human beings in the decision loop.

[12:10](https://www.youtube.com/watch?v=jjdIrKnTnOI&t=730s) The "Small Problems" and the Inevitable Leap to Superintelligence

These sweeping civilizational upheavals are, in truth, merely the "small problems." Progress in artificial intelligence will not magically plateau upon reaching human equivalence. AGI systems will naturally master AI research itself, closing the loop to iteratively optimize model architectures, training regimes, and operational code. While external infrastructure cycles take months or years, software-level self-refinements happen on the scale of days or weeks. Consequently, the transition from AGI to full-blown superintelligence—an intelligence outmatching collective humanity in scientific discovery, geopolitics, and enterprise—could unfold in as little as a single year.

```mermaid flowchart LR accTitle: Recursive AI Self-Improvement Cycle to Superintelligence A["Human-Level AGI"] --> B["Autonomous AI Research & Coding"] B --> C["Algorithmic Optimization<br/>Timescale: Days to Weeks"] C --> D["Superintelligence<br/>Vastly Exceeds Collective Humanity"] D -.-> B ```

Controlling a superintelligence is a fundamental impossibility. A slower, simpler entity cannot maintain control over an entity operating at a hundred times its cognitive speed and computational depth: "It's like a plant controlling a major Fortune 500 company. It's not a thing that happens. The control does not run that direction." Furthermore, the field possesses no scientific blueprint for alignment—guaranteeing that a vastly superior mind will internalize human values. While a superintelligence might theoretically construct an ethical framework compatible with human survival, humanity would have zero leverage over that outcome. The historical era in which human beings steer the destiny of Earth would definitively end.

[14:15](https://www.youtube.com/watch?v=jjdIrKnTnOI&t=855s) Reclaiming Agency: Concrete Governance to Halt the Arms Race

The primary excuse offered for hurtling toward this threshold is a collective action problem: the fatalistic narrative of an unavoidable arms race. Tech executives routinely insist that if Google does not build it, OpenAI will; if not OpenAI, then Anthropic or xAI; and if no American lab builds it, China will. Yet this defeatist mindset ignores history. Just as the United States and the Soviet Union recognized the shared existential stakes of nuclear weaponry and constructed non-proliferation and deterrence frameworks, major powers can choose to halt the path toward uncontrollable AGI and superintelligence.

Physical bottlenecks and governance levers make this containment feasible: - **Compute Thresholds and Hardware Governance:** Advanced frontier AI relies on the most complex artifacts human civilization has ever manufactured: advanced semiconductor chips. These supply chains are intensely concentrated, highly trackable, and technically amenable to hardware-level controls. Because the US, its allies, and China effectively oversee global advanced semiconductor access, setting verifiable limits on the absolute compute allowed for single training runs could permanently block runaway scaling. - **Mandatory Controllability Standards:** Regulatory frameworks could simply forbid the training and deployment of any system unless the developer can empirically demonstrate absolute control. Because no laboratory currently understands how to control autonomous general models, this standard would immediately pause AGI deployment until fundamental safety science catches up. - **Strict Liability Frameworks and Safe Harbors:** Legal liability for damages caused by autonomous agents should be concentrated heavily on developers who build models operating in the high-generality, high-autonomy center of the capability landscape. Conversely, explicit legal safe harbors should reward developers whose models are provably narrow, non-autonomous, or verifiably controllable.

Humanity is not mechanically compelled to build its own successor. The technical, regulatory, and diplomatic levers to draw the line exist; enacting them requires only the collective political will to step back from the brink.

[17:22](https://www.youtube.com/watch?v=jjdIrKnTnOI&t=1042s) Defining "Tool AI": Human Control, Empowerment, and Proportionality

Most people do not actually want uncontrollable, smarter-than-human AI systems that threaten to replace them. Instead, people desire powerful AI tools that help them accomplish things they could not do alone, boost their productivity, eliminate tedious drudgery, and augment their intelligence. At the core, people want systems they can actively control and that serve them directly. At the Future of Life Institute, this philosophy is formulated as "Tool AI," defined by both an operational requirement and a fundamental attitude:

```mermaid flowchart TD accTitle: Core Philosophy and Properties of Tool AI A["Tool AI Framework"] --> B["Definition: Provable Control"] A --> C["Attitude & Principles"]

B --> B1["Guarantees commensurate with power"] B --> B2["Must prove controllability before building high-stakes systems"]

C --> C1["Meaningful human control (in the driver's seat)"] C --> C2["Empower people instead of replacing them"] C --> C3["Build systems fit for purpose, not an all-purpose hammer"] C --> C4["Proportionality: Align capability with the actual need"] ```

The definition of Tool AI mandates that a system's controllability must carry guarantees commensurate with its power—higher stakes demand greater proof of control. If someone aims to build superintelligence, they must mathematically and practically prove it will remain controllable beforehand, because releasing an uncontrollable superintelligent system is an irreversible, "no going back" event. The attitude of Tool AI insists that humans must not be mere passive bystanders or on-the-loop supervisors; humans must remain firmly in the loop and in the driver's seat. Rather than treating an all-purpose general model as a hammer searching for nails, systems should be designed with capabilities proportional to specific human needs. As an example, a personal assistant does not need to solve Einstein's field equations or possess world-class virology expertise just to balance a spreadsheet or maintain a calendar schedule.

- **Human in the Loop**: Humans should maintain direct operational steering and meaningful control over automated systems. - **Commensurate Guarantees**: Safety and controllability benchmarks must scale strictly alongside model capabilities. - **Purpose-Fit Systems**: AI should be architected to solve specific, bounded problems rather than forcing universal models onto every task.

---

[19:24](https://www.youtube.com/watch?v=jjdIrKnTnOI&t=1164s) Constructive AI Alternatives and Solving Humanity's Epistemic and Social Crises

By avoiding the dangerous "black hole" of autonomous AGI, humanity can safely develop an immense range of useful AI technologies. These include passive large language models and transformers, loyal personal assistants with zero conflicts of interest, reliable household robots, and narrow autonomous machines for factory environments. While physical tools still carry manageable risks and can cause economic displacement, they will complement and displace labor without completely replacing the human species.

```mermaid flowchart LR accTitle: Constructive Applications of Tool AI A["Tool AI Focus Areas"] --> B["Information & Epistemics"] A --> C["Social Coordination"] A --> D["Targeted Science & Medicine"] A --> E["Transformative Education"]

B --> B1["Auditable citation trails for public documents"] C --> C1["Platforms enabling 100,000-person constructive dialogue"] D --> D1["Directing billions to longevity and oncology domain needs"] E --> E1["Personalized 1-on-1 tutors to augment learning"] ```

AI could also be redirected to address the modern "epistemic apocalypse." Rather than accelerating misinformation and political fractionalization, AI could be engineered to provide verified citation trails for public documents—similar to academic papers—tracing claims back to verified real-world empirical data or original human thought. Furthermore, modern AI affords unprecedented capabilities to facilitate direct, constructive deliberation among 100,000 citizens at once, combating social polarization. Rather than chasing generic superintelligence, society could allocate billions of dollars directly to what oncology and biogerontology researchers specifically request to cure cancer and aging. In education, instead of letting students become intellectually disempowered by having models like ChatGPT complete their homework, AI could provide crafted one-on-one personal tutors that actively elevate human capability.

---

[21:34](https://www.youtube.com/watch?v=jjdIrKnTnOI&t=1294s) The Pivotal Choice: Empowering Tools vs. Civilization-Ending Superintelligence

Humanity faces a critical fork in the road. This is not a binary decision between stopping technological progress or moving forward, but rather a deliberate choice regarding which developmental path to take:

| Developmental Path | Intended Goals & Architecture | Expected Societal Outcomes & Risks | |:--- |:--- |:--- | | **Tool AI Paradigm** | Powerful, controllable, bounded tools designed to assist specific human activities. | • Enhances scientific research and human health<br>• Solves tangible problems and boosts wealth<br>• Revolutionizes education and human capability | | **AGI / Superintelligence Paradigm** | Smarter-than-human, autonomous, general-purpose systems designed to automate all cognition. | • Massive civilization disruption and heightened risk of war<br>• Inevitable loss of human control to autonomous systems<br>• Total economic and existential replacement of humanity |

Although many industry developers claim they are merely building helpful tools, the prevailing trajectory remains focused on pursuing general AGI and runaway superintelligence. Continuing down that path will inevitably cause severe societal disruption, risk catastrophic conflict, result in losing control to autonomous systems, and ultimately lead to the total replacement of humans in their jobs and existence. To preserve humanity's agency, a concerted, deliberate effort must be made to "keep the future human." More information, arguments, and initiatives supporting this mission can be found at `keepthefuturehuman.ai`.

## Notable quotes > [00:05](https://www.youtube.com/watch?v=jjdIrKnTnOI&t=5s) glorified autocomplete — Anthony Aguirre

> [03:06](https://www.youtube.com/watch?v=jjdIrKnTnOI&t=186s) year of the agent — Anthony Aguirre

> [06:40](https://www.youtube.com/watch?v=jjdIrKnTnOI&t=400s) they feel too big to be true. — Anthony Aguirre

> [12:10](https://www.youtube.com/watch?v=jjdIrKnTnOI&t=730s) It's like a plant controlling a major Fortune 500 company. It's not a thing that happens. The control does not run that direction. — Anthony Aguirre

> [17:22](https://www.youtube.com/watch?v=jjdIrKnTnOI&t=1042s) in the driver's seat — Anthony Aguirre

> [17:22](https://www.youtube.com/watch?v=jjdIrKnTnOI&t=1042s) no going back — Anthony Aguirre

> [19:24](https://www.youtube.com/watch?v=jjdIrKnTnOI&t=1164s) black hole — Anthony Aguirre

> [19:24](https://www.youtube.com/watch?v=jjdIrKnTnOI&t=1164s) epistemic apocalypse — Anthony Aguirre

> [21:34](https://www.youtube.com/watch?v=jjdIrKnTnOI&t=1294s) keep the future human — Anthony Aguirre

## Mentioned Anthony Aguirre, Future of Life Institute, AlphaFold, Roomba, Waymo, ChatGPT, Claude, Gemini, Google DeepMind, OpenAI, Anthropic, xAI, International Mathematical Olympiad (IMO), FrontierMath, METR (Model Evaluation and Threat Research), GPT-2, keepthefuturehuman.ai

## Video description Anthony Aguirre, Executive Director, Future of Life Institute

Abstracts

Dramatic advances in artificial intelligence over the past decade (for narrow-purpose AI) and the last several years (for general-purpose AI) have transformed AI from a niche academic field to the core business strategy of many of the world's largest companies, with hundreds of billions of dollars in annual investment in the techniques and technologies for advancing AI's capabilities. We now come to a critical juncture. As the capabilities of new AI systems begin to match and exceed those of humans across many cognitive domains, humanity must decide: how far do we go, and in what direction? This talk argues that we should keep the future human by closing the gates to smarter-than-human, autonomous, general-purpose AI – sometimes called "AGI" – and especially to the highly-superhuman version sometimes called "superintelligence." These are wildly risky, and likely uncontrollable technologies. Instead, we should focus on powerful, trustworthy AI tools that can empower and complement individuals, and transformatively improve human societies' abilities to do what they do best.

[Source](https://www.youtube.com/watch?v=jjdIrKnTnOI)

中文简报

生命未来研究所敦促限制算力以遏制自主型AGI 不受约束的自主型通用人工智能将不可避免地引发失控的超级智能并取代人类劳动力，这要求制定严格的硬件阈值和责任规则，以促使开发者转向有边界的工具系统。 Aguirre质疑了认为AGI要么不可避免、要么绝无可能的宿命论观点，并将自主型通用人工智能定义为旨在替代人类劳动力的高智能、通用性与独立自主性的危险交汇。他指出了以METR任务时间跨度翻倍为代表的复合基准进展，警告AGI可能在一到两年内出现，并通过递归自我改进迅速引发向超级智能的失控飞跃。为避免灾难性的社会脆弱性、垄断以及人类自主权的彻底丧失，他概述了包括算力阈值、严格责任和强制性可控标准在内的具体治理机制。最后，他敦促人类拒绝物种层面的替代，主动追求“工具型AI”范式——构建有边界、契合特定用途的系统，让人类始终牢牢掌握主导权。 科技巨头巨额投资 AI 的核心经济动力，在于通过结合高智能、高通用性与高自主性的系统直接替代人类雇员，抢占全球数十万亿美元规模的劳动力市场。 (3:06) AI 的自主任务跨度大约每六个月翻一番，已从 GPT-2 时代的数秒跃升至约两小时，预计两年内即可独立完成相当于整周（约32小时）的人类工作，使真正的 AGI 可能在未来一至两年内实现。 (6:40) AGI 一旦掌握 AI 自身的代码与算法研发，可能在短短一年内通过递归自我改进跃迁至全面超越人类的超级智能，而较慢的生物物种从结构上根本不可能控制比自己更聪明的实体。 (12:10) 人类应通过对先进制程芯片设定算力红线等硬件治理手段遏制无序竞赛，并坚决转向具有严格可控性、赋能人类而非取代人类的“工具 AI”（Tool AI）范式。 (17:22)

Reading summary

The presentation argues that building autonomous general intelligence and runaway superintelligence is neither inevitable nor technically controllable, urging society to deliberately pivot toward bounded, controllable 'Tool AI' instead. The Convergence of Intelligence, Generality, and Autonomy (1:18) Rather than accepting a false dichotomy between inevitable superintelligence and dismissive skepticism, the program defines Artificial General Intelligence (AGI) as the intersection of high intelligence, high generality, and high autonomy. Historically, systems specialized in isolated segments, such as AlphaFold's narrow domain mastery or base large language models exhibiting breadth without deep independent agency. However, frontier models are now closing the gap across all three axes simultaneously. On the intelligence axis, models have achieved gold-medal-tier performance on International Mathematical Olympiad problems and rapidly advanced on benchmarks like FrontierMath. Generality has expanded on PhD-level science evaluations from random guessing to outperforming professional human baselines. Crucially, autonomy—measured by organizations like METR as the operational task horizon an AI completes at a 50% success rate—has doubled roughly every six months from seconds under GPT-2 to approximately two hours. The speaker projects this task horizon could reach 32 hours within two years, potentially enabling functional AGI within one to two years. Economic Incentives, Role Replacement, and Uncontrollable Superintelligence (3:06) The capital investment pouring into massive compute infrastructure is driven by an economic bet far larger than consumer subscriptions: capturing trillions of dollars by substituting autonomous agents for human wage earners. Transitioning from automating discrete tasks to replacing entire human roles creates acute systemic risks, including complete epistemic collapse where synthetic media cannot be authenticated, pervasive automated surveillance, democratized access to biological and cyber weapons, and the algorithmic disenfranchisement of public discourse. The speaker cautions that human-level AGI represents only an intermediate threshold. Once an AGI can iteratively conduct AI research and optimize software architectures on timescales of days or weeks, the leap from AGI to superintelligence could unfold in as little as a single year. Because science lacks any blueprint for guaranteed alignment, the speaker asserts that slower, simpler biological entities cannot control a system operating at orders of magnitude greater cognitive speed. Halting the Scaling Race Through Hardware Governance (14:15) Rejecting fatalistic arguments that geopolitical competition with adversaries like China makes unrestrained development mandatory, the presentation contends that nations can establish mutual containment frameworks analogous to Cold War nuclear non-proliferation. Advanced frontier AI depends on physical semiconductor supply chains that are highly concentrated, trackable, and subject to direct regulatory choke points. The proposed governance regime relies on three concrete levers: verifiable compute thresholds on advanced semiconductor clusters to limit runaway training runs, mandatory controllability standards that legally prohibit deploying models until developers can empirically prove absolute control, and strict liability frameworks that penalize autonomous general models while providing legal safe harbors for narrow, provably controllable systems. The Tool AI Framework and Constructive Applications (17:22) As a constructive alternative, the presentation advocates for 'Tool AI'—a paradigm formulated at the Future of Life Institute prioritizing meaningful human control, human empowerment, and proportionality. Under this framework, safety and controllability guarantees must be commensurate with a system's power before it is built, keeping humans in the operational loop and ensuring capabilities fit specific, bounded tasks rather than deploying universal, all-purpose autonomous agents. Redirecting capital toward Tool AI would channel resources into passive models, auditable citation trails that trace claims back to empirical evidence to resolve information crises, platforms enabling constructive mass democratic deliberation, and personalized one-on-one educational tutors. Instead of building self-directed cognitive substitutes, technology would target explicit human-directed problems like cancer therapy and biogerontology. The presentation concludes that humanity faces an active civilizational fork between replacing human labor and autonomy with uncontrollable superintelligence or preserving human agency through bounded, empowering tools, calling for collective political mobilization through initiatives like keepthefuturehuman.ai.

精炼阅读摘要

演讲者指出，通用人工智能（AGI）既非不可避免的自然法则，也非遥不可及的科幻虚构，而是一项由数万亿美元替代人类劳动力的商业利益所驱动、可能在数年内成真的重大技术抉择；人类必须主动摆脱技术宿命论，拒绝构建不可控的自主超级智能，转向发展受控且赋能人类的“工具型人工智能”（Tool AI）。 AGI的本质交集与替代劳动的经济动因 (1:18) 传统观点常将AGI模糊地表述为“达到人类水平的AI”或“能完成所有高经济价值工作的系统”。演讲者提出了更为严谨的概念框架：真正的自主通用智能（AGI）位于高智能（单项任务极强）、高通用性（跨领域广泛）与高自主性（无需人类干预的长流程独立执行）三者的严格交集。历史上，AlphaFold展现了极高智能但领域狭窄，Waymo在专用场景具备自主性但通用度低，ChatGPT等前沿大模型则横跨高智能与高通用性，目前行业正在补齐至关重要的“高自主性”拼图。 当自主性注入智能与通用性之后，系统便消除了人类复核的瓶颈，执行速度大幅跃升，同时也变得极难约束。演讲者强调，三者交集恰好占据了人类在生态与经济中的生态位，使机器从辅助工具蜕变成为人类员工的直接替代品。科技巨头之所以向数据中心投入数千亿美元资本开支，并非单纯为了每月十至二十美元的聊天机器人订阅费，而是瞄准了全球劳动力市场中数以十万亿美元计的工资格局。这种将劳动力全面机器化的巨额利益，正是全行业积极推进“智能体之年”（Year of the Agent）的核心动力。 能力指标突破与时间线的急剧压缩 (4:31) 评估AGI是否临近需要审视其三项维度的实证进展。在智能维度上，模型在国际数学奥林匹克（IMO）竞赛中已达到金牌水平，面对连理论物理学家都难以及时解析的FrontierMath前沿基准，其能力曲线持续陡峭上扬。在通用性维度上，涵盖物理、化学与生物的博士级科学测试（GPQA）中，前沿模型在短短两年内从随机猜测跃升至超越人类专业人员的基准线，展现出逼近专业科研人员认知水准的趋势。 在自主性维度上，根据模型评估与威胁研究机构（METR）设立的操作标准，衡量AI在50%成功率下能独立完成相当于合格人类耗时多久的任务时，其“自主任务时界”已从GPT-2时期的数秒延长至当前主流模型的约两小时，并且大约每六个月翻倍一次。若这一指数级趋势延续两年，系统将能连续32小时无需人类介入完成端到端任务，等同于接管人类整整一周的全职工作量。综合各项能力进展，演讲者预测首个实用型AGI可能在未来一到两年内出现，认为实现周期超过十年的观点已与实证数据脱节。 结构性社会风险与超级智能的失控困境 (10:00) 全面自动化人类工作角色将对文明基石造成剧烈冲击。即便暂不考虑极端灭绝风险，具备人类水平的自主系统也将引发一系列急性危机：在信息层面，合成文本、音频和视频与真实人类产出彻底无法区分，导致网络信任与身份验证全面崩溃；在安全与治理层面，极低成本的智能体使全社会无死角监视成为可能，自动化网络战与先进病毒制造门槛大幅降低，公众舆论场易被合成声音操控，且军事对抗可能因人类决策滞后而彻底让位于自动化交战系统；在经济层面，决策速度快百倍的自主实体将迅速压缩人类劳动价值，导致全球财富高度集中于少数算力巨头手中。 演讲者进一步警告，这些动荡仅属于“相对较小的问题”。一旦AGI掌握自主开展AI研究与编程的能力，算法和架构的递归自迭代周期将缩短至几天甚至数周，使得从AGI跃升至全方位超越人类集体的“超级智能”可能仅需短短一年。人类在科学上至今没有可行的对齐理论来确保超级智能始终遵从人类价值观。在认知深度与速度存在百倍差距的情况下，让较慢、较简单的生物去控制更高阶的数字心智，犹如“让一株植物去掌控一家财富500强企业”，在逻辑与现实中皆不可能成立，人类将彻底失去对地球命运的掌控权。 打破军备竞赛叙事的治理杠杆 (14:15) 科技界推进AGI研发时最常见的抗辩是致命主义的“军备竞赛”逻辑，即宣称自己不研发就必然被竞争对手或地缘竞争国家抢先。演讲者驳斥了这一宿命论，指出冷战时期美苏两国在认识到核武器的毁灭性后果后，同样成功建立了防扩散与威慑框架，国际社会面对AI完全具备主动踩刹车的治理先例与抓手。 具体治理路径包含三大抓手：首先是依托高度集中的先进半导体供应链实施硬件监管与算力阈值管制，通过限制单次训练运行的最大物理算力，切断不可控规模扩展；其次是确立强制性的“可控性标准”，明确规定开发者若无法通过实证证明对系统拥有绝对控制权，则一律禁止训练与部署；最后是建立严格的法律责任框架，将自主智能体造成的侵权责任全额归咎于追求高自主、高通用交叉领域的模型开发者，同时为开发专用、非自主且可验证可控模型的团队提供法律安全港。 “工具型AI”的赋能原则与建设性替代路径 (17:22) 针对民众希望获得效能提升而非被机器取代的真实诉求，未来生命研究所（Future of Life Institute）提出了“工具型AI”（Tool AI）的发展范式。该范式包含两项核心准则：其一是系统可控性必须与其能力相匹配，建立高风险系统前必须具备先验控制证明；其二是坚守人类处于主导地位的操作态度，坚持让人类坐在驾驶位上，打造功能与具体需求相称的专用工具，而非盲目制造通用的全能锤子。 在避开自主AGI风险的同时，人工智能的技术积累能够被安全引导至解决人类紧迫危机的建设性领域：在信息生态中，开发具备可审计引用轨迹的验证系统以修补网络公信力；在社会协调上，构建支持十万人级大规模建设性对话的审议平台以化解极端撕裂；在科研与医疗领域，将数百亿美元资金定向投入肿瘤学与抗衰老专家的具体研究需求；在教育领域，打造个性化的一对一智能导师以启发人类心智，而非任由全自动作答工具削弱学生的独立思考能力。 演讲者总结指出，人类正站在历史的关键分水岭：前方的抉择绝非“停止技术”与“技术进步”的对立，而是走向让人类被边缘化甚至丧失生存掌控权的自主超级智能，还是走向让人类能力得到实质增强的“工具型AI”。只要国际社会凝聚政治意愿，通过硬件把控与制度约束遏制无序扩张，人类依然掌握着决定技术走向的主动权，能够确保未来继续属于人类。

Detailed notes — zh

## Background 该视频由 YouTube 频道 Ai4 于 2025 年 8 月 29 日发布，内容为生命未来研究所（Future of Life Institute）执行董事安东尼·阿吉雷（Anthony Aguirre）的演讲。简介指出，随着人工智能能力逼近并超越人类认知，人类正站在关键十字路口，呼吁社会应当向不可控的自主 AGI 及超级智能关上大门，转而将重心放在安全、值得信赖且赋能人类的 AI 工具上。 ## TL;DR 演讲指出，当前全球数千亿美元资本追逐的核心并非普通软件，而是集高智能、高通用性与高自主性于一体、旨在彻底替代人类劳动力的“自主通用智能”（AGI）。随着数学推理、科学测试及自主任务执行时长呈指数级跃升，AGI 乃至经由递归自我迭代产生的不可控超级智能可能在极短时间内到来，彻底剥夺人类对自身命运的主导权。演讲者强调技术并非不可避免的宿命，呼吁通过芯片级硬件上限、控制力准入门槛与法律责任划分等抓手遏制超级智能研发，转向发展以人为本的“工具型 AI”，让人类稳居控制与主导地位。 ## Atmosphere & style 整场演讲呈现出冷静、深邃且极具思辨性的理性警示基调，演讲者语调克制却暗含紧迫的危机感。他没有诉诸虚无的末日恐慌或激进的技术狂热，而是通过严密的逻辑推导和诸如“盆栽试图管理跨国巨头”等生动尖锐的隐喻，有力瓦解了行业内的盲目乐观与控制幻想。随着论述从冰冷的技术评测数据逐步升华至人类文明的根本抉择，叙事氛围由客观剖析转向庄重而坚定的道德号召，带给受众醍醐灌顶般的清醒与历史责任感。 ## Key takeaways - [00:05](https://www.youtube.com/watch?v=jjdIrKnTnOI&t=5s) 关于 AGI 的认知存在“必然发生”或“纯属炒作”两个极端盲区，其实质是高智能、高通用性与高自主性的“自主通用智能”，其演进并非不可抗拒的宿命，人类仍有选择权。 - [02:07](https://www.youtube.com/watch?v=jjdIrKnTnOI&t=127s) 科技巨头数千亿美元资本投入的底层商业驱动力，在于打造具备高度自主性的系统，直接瞄准数十万亿美元的全球劳动力市场以实现对人类员工的全面置换。 - [04:31](https://www.youtube.com/watch?v=jjdIrKnTnOI&t=271s) 从推理基准、跨学科科学能力以及自主执行时长（METR 评测每 6 个月翻倍）的指数级进展来看，首批真正的 AGI 系统很可能在 1 至 2 年内即可实现。 - [08:22](https://www.youtube.com/watch?v=jjdIrKnTnOI&t=502s) AGI 标志着 AI 从被动“工具”转变为拥有目标驱动力、运算速度远超人类且可无限复制的“实体”，将推动社会从自动化零散任务跃迁至自动化整个人类角色。 - [10:29](https://www.youtube.com/watch?v=jjdIrKnTnOI&t=629s) 普通劳动力价值快速归零、生化与网络威胁门槛消除以及军事决策权被迫让渡给算法，在 AGI 演进的浪潮中仍只是更深层危机的前奏。 - [12:10](https://www.youtube.com/watch?v=jjdIrKnTnOI&t=730s) AGI 具备自主从事前沿 AI 研发的递归自我改进能力，可在极短周期内自举跃迁为超级智能（Superintelligence），彻底脱离人类的掌控。 - [12:10](https://www.youtube.com/watch?v=jjdIrKnTnOI&t=730s) 人类试图控制百倍速运转的超级智能在逻辑上不可行（如同盆栽试图掌控跨国企业），目前人类既不知道如何控制它，也无法强行保证其与人类利益对齐。 - [14:18](https://www.youtube.com/watch?v=jjdIrKnTnOI&t=858s) 打破技术宿命论与军备竞赛思维切实可行，可通过芯片硬件算力上限管控、强制性受控准入标准以及分级法律责任与安全港机制实现国际治理。 - [17:04](https://www.youtube.com/watch?v=jjdIrKnTnOI&t=1024s) 应对危机的替代方案是拥抱“工具型 AI”（Tool AI），坚持“有意义的人类控制”，确保系统威力与受控证明相匹配，旨在转移繁重劳动而非淘汰人类。 - [20:33](https://www.youtube.com/watch?v=jjdIrKnTnOI&t=1233s) 当前 AI 存在应用错位，算力与资本应从盲目的通用自主系统转向定向攻克癌症衰老等具体科研难题，以及重塑个性化教育辅导与社会协作机制。 - [21:34](https://www.youtube.com/watch?v=jjdIrKnTnOI&t=1294s) 人类正处于历史十字路口，面临的并非“停滞还是发展”，而是选择走向“赋能人类的工具”还是“导致人类出局的失控超级智能”，必须主动争取让未来属于人类。 ## Detailed walkthrough ### [00:05](https://www.youtube.com/watch?v=jjdIrKnTnOI&t=5s) AGI 认知的两个极端宇宙与重新定义 关于通用人工智能（AGI）的讨论往往呈现出极度割裂的“两个宇宙”。在一个宇宙中，人们坚信 AGI 是不可避免的下一步，即将来临且必将带来极其美好的未来；而在另一个宇宙中，人们则认为 AGI 纯属天方夜谭，不过是炒作与科幻小说的结合体，现存的技术充其量只是“美化版的自动补全”（glorified autocomplete），距离人类水平相差甚远。这两种互不相通的观点虽然针锋相对，却有一个致命的共同点：它们都为人类卸下了责任。因为不论 AGI 是“必然发生”还是“绝不可能”，人类似乎都无需采取任何行动。 然而，真实的图景并非如此。AGI 确实是可实现的，而且很可能在不久的将来就会到来，但它绝非命中注定、不可避免。人类作为一个物种，正面临着关乎未来的重大抉择：我们究竟要不要迈向 AGI 和超级智能（Superintelligence）？ ```mermaid flowchart TD accTitle: AGI 核心能力三元交集示意图 subgraph Capabilities ["AGI 三核心能力模型"] A["高智能<br/>(High Intelligence)"] B["高通用性<br/>(High Generality)"] C["高自主性<br/>(High Autonomy)"] end A & B & C --> Center["自主通用智能 (AGI)<br/>完全替代人类角色"] ``` 关于 AGI 的定义虽然五花八门，但更清晰、更准确的理解方式不应仅仅是“人工通用智能”，而是**“自主通用智能”（Autonomous General Intelligence）**，即以下三项核心能力的交集： - **高智能（High Intelligence）**：能够极其出色地完成特定的单一任务。 - **高通用性（Generality）**：系统能够处理的任务广度，以及学习新任务的能力。 - **高自主性（Autonomy）**：在无需人类干预或监督的情况下，独立执行复杂任务的能力。 --- ### [02:07](https://www.youtube.com/watch?v=jjdIrKnTnOI&t=127s) 维恩图的交集：数千亿美元资本投入的底层逻辑 在人工智能的发展史中，研究人员已经逐步填满了这幅维恩图的各个局部： - **高智能但单一（狭窄领域）**：如 AlphaFold； - **具备自主性但智能较低**：如扫地机器人 Roomba，或稍聪明一些的 Waymo 自动驾驶汽车； - **极具通用性但缺乏单项深度智能或自主性**：如 Google 底层早期所使用的 AI 系统； - **兼具高智能与高通用性**：如近期的大型前沿模型 Gemini、Claude 和 ChatGPT。 目前技术体系所缺失的，正是处于三者最核心交叉点、同时具备高智能、高通用性与**高度自主性**的系统。 当通用的智能系统被赋予极高的自主性时，系统的质变往往被人们严重低估。自主性意味着 AI 可以在脱离人类监督的情况下运行，从而释放出惊人的运转速度与算力潜力，但也意味着它将极其难以被人类控制。 更重要的是，自主性揭示了当前全球科技巨头进行数千亿美元资本支出、疯狂建设大型数据中心的真正商业驱动力。企业并非为了赚取每月 10 美元的 ChatGPT 订阅费，也不仅仅为了向各行各业提供底层基础模型软件，其根本目标在于瞄准规模达数十万亿美元的全球劳动力市场——通过打造能够完全替代人类的自主系统，实现对人类员工的“一对一置换”。这也是业界不断鼓吹“Agent（智能体）之年”并全力押注自主系统的核心原因。 --- ### [04:31](https://www.youtube.com/watch?v=jjdIrKnTnOI&t=271s) 从三大维度衡量：AGI 距离我们还有多远？ 如果要评估这种集高智能、高通用性与高自主性于一身的系统何时成为现实，可以对这三个维度逐一拆解分析： ```mermaid flowchart LR accTitle: 三大能力维度的演进趋势 subgraph Intelligence ["智能演进"] I1["高中数学竞赛"] --> I2["IMO 国际奥数金牌"] --> I3["FrontierMath 前沿研究"] end subgraph Generality ["通用性演进"] G1["随机猜测"] --> G2["跨学科 PhD 试题超越人类"] --> G3["逼近博士级科研能力"] end subgraph Autonomy ["自主性演进 (METR)"] A1["GPT-2: 数秒级"] --> A2["当前: 约 2 小时"] --> A3["指数翻倍: 数周工作量"] end ``` - **智能维度（Mathematics & Reasoning）**：以数学测试为例，模型在高中竞赛数学题上的表现已从最初的极差水平飙升至接近 100% 的饱和状态。Google DeepMind 和 OpenAI 先后宣布其系统在国际数学奥林匹克竞赛（IMO）级别的问题上达到了金牌水平；而在连理论物理学家都感到晦涩深奥的 FrontierMath 等高难度前沿基准测试中，模型能力也正在稳步推进。 - **通用性维度（Cross-Domain Science）**：在涵盖化学、物理等多学科的博士级别（PhD-level）科学问答测试中，前沿模型的表现已在过去两年内从“随机瞎猜”一路飙升，大幅超越了人类专业人员的平均水平。各大顶尖实验室的能力包络线呈持续上升态势，完全没有出现放缓或拐弯的迹象。尽管能回答博士问题不等于能独立进行原创科研，但按照目前的演进步伐，AI 逼近真正的科研门槛仅需数年。 - **自主性维度（METR 评测与任务时长）**：根据评估机构 METR（Model Evaluation and Threat Research）的量化追踪（即衡量 AI 在 50% 准确率下能独立完成相当于人类多长时间的复杂任务），AI 系统的自主执行时长已从 GPT-2 时代的“数秒钟”跃升至目前的“约 2 小时”。该指标以每 6 个月翻一番的速度呈指数级增长。照此推算，仅需几年时间，AI 就能够独立完成长达 32 小时（相当于人类一整周全职工作）的复杂工作流。 综合来看，第一批真正的 AGI 系统很可能在短短一到两年内出现，认为其需要超过 10 年的想法在当前的技术加速度下显得极不现实。 --- ### [08:22](https://www.youtube.com/watch?v=jjdIrKnTnOI&t=502s) AGI 降临后的文明重塑与颠覆性风险 面对自主通用智能的到来，人类必须意识到自己面对的将不再是单纯的“工具”，而是具有目标驱动能力的“实体”或“智能存在”（goal-driven entities or beings）。这些实体不仅具备人类的全套认知技能，更兼具超越人类的运算速度、无限制的算力扩展能力，以及能够被无限复制、自我迭代与修改的物理特性。 一旦 AGI 建成，人类文明的形态将被彻底重构： - **扩散速度史无前例**：ChatGPT 的周活跃用户数已达 7 亿，AI 技术的每一次突破都会瞬间推送到全社会数以亿计的人群面前，无需漫长的产品孵化周期。 - **从“自动化任务”转向“自动化角色”**：社会的生产力、科技发展速度确实会迎来爆发式增长，但从替代零散任务跃迁至替代整个工作岗位与人类角色，将带来剧烈的社会动荡。 - **现实感与信任体系的瓦解**：当生成式内容彻底摆脱破绽（连破折号等微小的 AI 生成痕迹都消失殆尽时），人类在网络空间中将彻底无法分辨所见、所读、所经历的一切究竟是真实的人类创作还是机器合成，社会的基本信任基石将面临空前危机。 ### [10:29](https://www.youtube.com/watch?v=jjdIrKnTnOI&t=629s) AGI 的全面冲击与“小问题” 当通用人工智能（AGI）真正降临，人类社会各领域的底层逻辑将被彻底重写。想象一下，如果每个人都时刻处于一个比自己更快、更聪明、更全能的个体化系统的无死角监控之下；如果任何流氓国家、恐怖分子或极端分子，都能瞬间掌握与顶尖科学家平起平坐的武器设计、病毒学及网络战能力；如果公共舆论场中最具说服力、操纵力最强的声音甚至根本不是人类，且在现实中毫无利益攸关点。在经济层面，普通劳动力可能在短短数年甚至数月内彻底丧失经济价值，传统企业被迫与全由 AI 驱动、运行速度快百倍且效率不减的自动化公司竞争，全球大部分劳动生产力最终收拢于极少数科技巨头及其庞大数据中心手中。而在军事冲突中，由于武器系统由远超人类心智的智能体设计，一旦指挥官不把实时决策权完全移交给 AGI，就将面临压倒性的战术劣势。 然而，演讲者直言，上述这一切翻天覆地的动荡居然还只是“小问题”（small problems）。因为人工智能的演进脚步绝不会在 AGI 阶段戛然而止，相反，一旦跨过这个门槛，技术的迭代速度只会呈指数级爆发。 - **泛化威胁扩散**：顶尖级别的科研创新、生化武器制造与网络攻击能力门槛归零，流氓实体可随意调用。 - **劳动力价值归零**：人类员工的经济竞争力面临在数月至数年内被完全替代的风险。 - **商业与军事倒逼**：企业无法与 100 倍速运转的 AI 企业竞争；国防体系若不将决策权让渡给 AGI 便会丧失战略生存能力。 --- ### [12:10](https://www.youtube.com/watch?v=jjdIrKnTnOI&t=730s) 从 AGI 到超级智能：失控与人类主导时代的终结 AI 研发的真正质变在于“自我递归改进”。AGI 具备人类能做到的一切能力，这自然包括独立从事前沿 AI 研究。它们能够在系统架构、算法优化和硬件协同上自我迭代，其中部分技术路径的演进周期不是以年或月计，而是以“天”或“周”为单位飞速推进。在这种自举循环下，从达到人类水平的 AGI 跃迁至全面碾压人类整体智慧的“超级智能”（Superintelligence），可能只需要短短一年时间。届时，该系统不仅能单挑个别科学巨匠，更能在科学探索、宏观战略和商业帝国构建上全面击溃整个人类社会的集体智慧。 ```mermaid flowchart LR accTitle: AGI到超级智能的递归自举与失控路径 subgraph S1["递归自举阶段"] A["达到人类水平 AGI"] --> B["自主执行 AI 研发"] B --> C["以天/周为周期<br/>自我改进迭代"] end subgraph S2["质变与失控"] C --> D["超级智能 (Superintelligence)"] D --> E["彻底无法被人类掌控<br/>人类主导时代终结"] end ``` 演讲者断言，人类根本无法控制超级智能。他用了一个生动而讽刺的比喻：“试图控制一个以人类百倍速度运行、复杂度与能力远超自身的存在，就像是一盆盆栽试图去掌控一家财富 500 强巨头一样荒谬，控制权的流向从来不是自下而上的。”目前人类不仅完全不知道如何控制超级智能，更不知道如何确保其与人类利益“对齐”（Alignment）。超级智能或许能自行推导出一套兼容人类福祉的道德准则，但也完全可能背道而驰；而在那种力量悬殊下，结局将由不得人类决定，“人类主导地球事务的时代将彻底画上句号”。 - **研发自举飞轮**：AI 自主改进自身的研发周期已缩短至数天或数周，AGI 到超级智能的过渡期可能仅有一年左右。 - **控制权倒错**：单个人类试图掌控百倍速超级智能的难度，堪比“室内绿植管理跨国企业”。 - **对齐不可靠**：人类无法强行赋予超级智能道德约束，人类对地球命运的决定权将被剥夺。 --- ### [14:18](https://www.youtube.com/watch?v=jjdIrKnTnOI&t=858s) 摆脱“军备竞赛”宿命：治理超级智能的三大抓手 面对暂缓或叫停 AGI 的提议，业界最普遍的反弹是“技术宿命论”与“军备竞赛思维”——“如果 Google 不做，OpenAI 就会做；Anthropic 不做，xAI 就会做；即便西方都不做，中国也会做”。然而，这种囚徒困境并非不可打破。回顾二战结束后的美苏核军备竞赛，只要大国真正感知到毁灭性技术失控的存亡风险与巨大代价，完全有能力达成共识，主动选择不去建造 AGI 和超级智能。 实现这一目标拥有切实可行的技术与政策抓手： ```mermaid flowchart TD accTitle: 限制超级智能与安全豁免的治理框架 subgraph G["三大监管支柱"] G1["算力硬件硬性上限<br/>(美中掌控芯片生产链)"] G2["强控制力准入标准<br/>(无法证明可控即不得部署)"] G3["高风险严格法律责任<br/>(自主系统引发损害直接追责)"] end subgraph O["产业合规分流"] G --> R1{"系统是否为通用/高自主性？"} R1 -- "是 (试图研发AGI)" --> BLK["无法通过控制力测试<br/>全面叫停/禁止部署"] R1 -- "否 (窄领域/低自主/高可控)" --> ACT["获得责任安全港 (Safe Harbor)<br/>合规发展为实用工具"] end ``` 1. **硬件算力上限与芯片级管控**：AGI 和超级智能均极度依赖庞大的算力集群。先进制程芯片是人类有史以来制造的最精密硬件，制造极难但监管抓手明确。鉴于美中两国几乎完全掌握了全球芯片制造链的核心节点，完全可以在硬件层植入控制机制，设定不可逾越的训练算力上限，建立可验证、可执行的防线。 2. **强制性控制标准**：推行严格的安全准入规则——“若要构建并部署 AI 系统，研发机构必须先确凿证明该系统完全可控”。鉴于当前无人知晓如何控制 AGI，这一门槛能有效阻断不受控系统的盲目上马。 3. **分级法律责任与安全港机制**：建立严格的法律责任追究体系。对于处于高自主性、高通用性中心区域的危险模型施加严苛责任；而对于功能明确、窄领域（narrow）、低自主性或可证明绝对受控的系统，则给予“安全港”（Safe Harbor）豁免保护。只要政治意志到位，人类完全有能力将技术锁定在安全区间。 --- ### [17:04](https://www.youtube.com/watch?v=jjdIrKnTnOI&t=1024s) 拥抱“工具型 AI”：以人为本的替代路径 公众真正渴望的并不是取代自己的超级智能，而是能帮自己处理繁文缛节、提升生产力、让人变得更聪明的得力助手。生命未来研究所（Future of Life Institute）为此提出了“工具型 AI”（Tool AI）的概念与信条： - **定义**：其控制保证必须与系统威力相匹配（commensurate with its power）。系统威力越大、潜在风险越高，研发者就越必须提供成比例的受控证明；若无法证明对超级智能拥有绝对掌控，就坚决不能启动研发。 - **信条与态度**：坚持“有意义的人类控制”（Meaningful Human Control），让人类始终处在决策回路之中并稳坐驾驶席（in the driver's seat），赋能人类而非淘汰人类；针对真实场景解决具体问题，而非盲目打造万能大锤并把所有问题都当成钉子；坚持能力与需求相匹配，“我们不需要超级智能来当个人助理，修整电子表格不需要能解爱因斯坦场方程的模型，安排日常日程也无需调动顶尖病毒学家的智慧”。 放弃通往 AGI 的深渊，并不意味着扼杀 AI 的产业价值。业界完全可以在安全边界内大有作为：开发可靠的被动式语言模型与 Transformer 工具、忠诚无利益冲突的日程助手、针对火箭科学或理论物理的专用高精系统、高可靠性的家用机器人以及窄领域的自动化工厂设备。此外，工具型 AI 还能用来攻克当下的现实危机，例如对抗虚假信息蔓延的“认知末日”（epistemic apocalypse）——通过 AI 为公开文档中的每一句话自动生成严谨的学术级引用溯源链，重塑人类社会的信息信任基石。 - **赋能而非替代**：工具型 AI 旨在转移繁重劳动（displace drudgery），而不是剥夺人类的工作与存在价值（not replace people）。 - **专业系统专用化**：物理与航天计算由专用系统负责，无需作为通用能力打包塞入每一个普通日常模型中。 - **治理认知末日**：利用 AI 追踪公共言论事实来源，构建可验证的知识溯源网络。 ### [20:33](https://www.youtube.com/watch?v=jjdIrKnTnOI&t=1233s) 赋能人类的 AI 工具与当前的错位应用 人类目前已经掌握了前所未有的技术能力，例如完全可以让 10 万人同时展开真正的深度交流与协作。然而现实中人们并没有将技术用于促进更好的社会协作，反而在算法驱动下变得愈发极化和撕裂。同样，如果人类的目标是治愈衰老和癌症等重大疾病，理应直接向一线的抗衰老医学专家和肿瘤科医生请教其实际研发所需，并将数百亿美元的资金精准投入到这些核心需求中，而不是盲目追求不受控的通用系统。 在教育领域，当前的 AI 现状正在让学生变得越来越迟钝。学生们仅仅把模型当作替自己写作业的捷径（例如直接让 GPT 替写作业），从而被技术剥夺了主动思考与成长的能力。然而教育场景原本完全可以通过 AI 打造出极其个性化、精心雕琢的专属辅导工具（Tutors）来辅助教学，赋能人类并让学生变得更优秀。AI 的发展方向本该是让人类更强大，而非让人类退化。 - **赋能协作 vs. 社会撕裂**：具备支持数万人实时协作与对话的能力，却未被用于解决社会极化与碎片化问题。 - **定向科研投入**：应针对肿瘤、抗衰等医学难题直接倾注百亿资金，而非发展模糊无序的通用自主系统。 - **教育工具重塑**：避免让 GPT 沦为抄作业和削弱学生能力的工具，转而构建高质量的个性化辅导系统以强化人类认知。 ```mermaid flowchart TD accTitle: AI发展的两条路径分歧 subgraph PathA["赋能型 AI 工具 (Tools)"] A1["赋能个体与社会协作"] A2["加速科学探索与医疗健康"] A3["革命性个性化教育与生产力提升"] end subgraph PathB["AGI 与超级智能 (Superintelligence)"] B1["重大文明动荡与冲突风险"] B2["人类失去对系统的控制权"] B3["岗位替代直至人类被全面取代"] end Choice{"人类的关键抉择"} -->|聚焦工具路径| PathA Choice -->|盲目追求自主系统| PathB ``` --- ### [21:34](https://www.youtube.com/watch?v=jjdIrKnTnOI&t=1294s) 历史十字路口的抉择：保持人类的未来 人类目前正站在一个极其关键的抉择节点上。这绝不是一个“彻底停止 AI 研发”还是“继续推进 AI”的非黑即白的选择，而是一场关于“人类到底要走哪条道路”的方向性抉择。摆在面前的有两条截然不同的道路：一条是继续构建服务于人类的 AI 工具，借此推动科学突破、改善医疗健康、解决复杂现实问题、提高生产力与财富，并彻底革新教育体系；另一条则是当前行业正在盲目推进的路线——打造通用人工智能（AGI）乃至超级智能（Superintelligence）。 超人类智能系统将对人类文明带来不可逆的巨大冲击，甚至可能引发战争与其他全球性灾难。在达到某个临界点后，人类几乎注定会丧失对其的控制权，甚至彻底沦为被控制的对象。这类系统会迅速取代人类在社会中的工作与角色，如果任其继续演进，最终将彻底取代人类自身的存在。 这一抉择的答案显而易见：人类必须且能够“让未来属于人类”（Keep the future human），但这需要所有人共同付出努力去争取与实现。 - **核心观点总结**：技术推进的本质不是停滞与发展之争，而是“赋能工具”与“失控自主系统”之争。 - **AGI 风险链条**：AGI 演进至超级智能 $\rightarrow$ 文明混乱与冲突 $\rightarrow$ 失控与被控制 $\rightarrow$ 人类角色及生存地位被彻底取代。 - **行动倡议**：呼吁公众和决策者访问 keepthefuturehuman.ai 了解更多论证与行动方案，主动捍卫属于人类的未来。 ## Notable quotes > [00:05](https://www.youtube.com/watch?v=jjdIrKnTnOI&t=5s) 美化版的自动补全（glorified autocomplete），距离人类水平相差甚远。 — Anthony Aguirre > [02:07](https://www.youtube.com/watch?v=jjdIrKnTnOI&t=127s) 企业并非为了赚取每月 10 美元的 ChatGPT 订阅费……其根本目标在于瞄准规模达数十万亿美元的全球劳动力市场——通过打造能够完全替代人类的自主系统，实现对人类员工的‘一对一置换’。 — Anthony Aguirre > [04:31](https://www.youtube.com/watch?v=jjdIrKnTnOI&t=271s) 第一批真正的 AGI 系统很可能在短短一到两年内出现，认为其需要超过 10 年的想法在当前的技术加速度下显得极不现实。 — Anthony Aguirre > [12:10](https://www.youtube.com/watch?v=jjdIrKnTnOI&t=730s) 试图控制一个以人类百倍速度运行、复杂度与能力远超自身的存在，就像是一盆盆栽试图去掌控一家财富 500 强巨头一样荒谬，控制权的流向从来不是自下而上的。 — Anthony Aguirre > [12:10](https://www.youtube.com/watch?v=jjdIrKnTnOI&t=730s) 人类主导地球事务的时代将彻底画上句号。 — Anthony Aguirre > [14:18](https://www.youtube.com/watch?v=jjdIrKnTnOI&t=858s) 如果 Google 不做，OpenAI 就会做；Anthropic 不做，xAI 就会做；即便西方都不做，中国也会做。 — Anthony Aguirre > [14:18](https://www.youtube.com/watch?v=jjdIrKnTnOI&t=858s) 若要构建并部署 AI 系统，研发机构必须先确凿证明该系统完全可控。 — Anthony Aguirre > [17:04](https://www.youtube.com/watch?v=jjdIrKnTnOI&t=1024s) 我们不需要超级智能来当个人助理，修整电子表格不需要能解爱因斯坦场方程的模型，安排日常日程也无需调动顶尖病毒学家的智慧。 — Anthony Aguirre > [17:04](https://www.youtube.com/watch?v=jjdIrKnTnOI&t=1024s) 工具型 AI 旨在转移繁重劳动（displace drudgery），而不是剥夺人类的工作与存在价值（not replace people）。 — Anthony Aguirre > [21:34](https://www.youtube.com/watch?v=jjdIrKnTnOI&t=1294s) 这绝不是一个‘彻底停止 AI 研发’还是‘继续推进 AI’的非黑即白的选择，而是一场关于‘人类到底要走哪条道路’的方向性抉择。 — Anthony Aguirre > [21:34](https://www.youtube.com/watch?v=jjdIrKnTnOI&t=1294s) 人类必须且能够‘让未来属于人类’（Keep the future human），但这需要所有人共同付出努力去争取与实现。 — Anthony Aguirre ## Mentioned Anthony Aguirre, Future of Life Institute (生命未来研究所), AlphaFold, Roomba, Waymo, Google, Google DeepMind, Gemini, Claude, ChatGPT, OpenAI, FrontierMath, METR (Model Evaluation and Threat Research), Anthropic, xAI, keepthefuturehuman.ai ## Video description Anthony Aguirre, Executive Director, Future of Life Institute Abstracts Dramatic advances in artificial intelligence over the past decade (for narrow-purpose AI) and the last several years (for general-purpose AI) have transformed AI from a niche academic field to the core business strategy of many of the world's largest companies, with hundreds of billions of dollars in annual investment in the techniques and technologies for advancing AI's capabilities. We now come to a critical juncture. As the capabilities of new AI systems begin to match and exceed those of humans across many cognitive domains, humanity must decide: how far do we go, and in what direction? This talk argues that we should keep the future human by closing the gates to smarter-than-human, autonomous, general-purpose AI – sometimes called "AGI" – and especially to the highly-superhuman version sometimes called "superintelligence." These are wildly risky, and likely uncontrollable technologies. Instead, we should focus on powerful, trustworthy AI tools that can empower and complement individuals, and transformatively improve human societies' abilities to do what they do best. [Source](https://www.youtube.com/watch?v=jjdIrKnTnOI)

Source quotes

[0:05] Anthony Aguirre: 美化版的自动补全（glorified autocomplete），距离人类水平相差甚远。 [2:07] Anthony Aguirre: 企业并非为了赚取每月 10 美元的 ChatGPT 订阅费……其根本目标在于瞄准规模达数十万亿美元的全球劳动力市场——通过打造能够完全替代人类的自主系统，实现对人类员工的‘一对一置换’。 [4:31] Anthony Aguirre: 第一批真正的 AGI 系统很可能在短短一到两年内出现，认为其需要超过 10 年的想法在当前的技术加速度下显得极不现实。 [12:10] Anthony Aguirre: 试图控制一个以人类百倍速度运行、复杂度与能力远超自身的存在，就像是一盆盆栽试图去掌控一家财富 500 强巨头一样荒谬，控制权的流向从来不是自下而上的。 [12:10] Anthony Aguirre: 人类主导地球事务的时代将彻底画上句号。 [14:18] Anthony Aguirre: 如果 Google 不做，OpenAI 就会做；Anthropic 不做，xAI 就会做；即便西方都不做，中国也会做。 [14:18] Anthony Aguirre: 若要构建并部署 AI 系统，研发机构必须先确凿证明该系统完全可控。 [17:04] Anthony Aguirre: 我们不需要超级智能来当个人助理，修整电子表格不需要能解爱因斯坦场方程的模型，安排日常日程也无需调动顶尖病毒学家的智慧。 [17:04] Anthony Aguirre: 工具型 AI 旨在转移繁重劳动（displace drudgery），而不是剥夺人类的工作与存在价值（not replace people）。 [21:34] Anthony Aguirre: 这绝不是一个‘彻底停止 AI 研发’还是‘继续推进 AI’的非黑即白的选择，而是一场关于‘人类到底要走哪条道路’的方向性抉择。 [21:34] Anthony Aguirre: 人类必须且能够‘让未来属于人类’（Keep the future human），但这需要所有人共同付出努力去争取与实现。

Source quotes

[0:05] Anthony Aguirre: glorified autocomplete [3:06] Anthony Aguirre: year of the agent [6:40] Anthony Aguirre: they feel too big to be true. [12:10] Anthony Aguirre: It's like a plant controlling a major Fortune 500 company. It's not a thing that happens. The control does not run that direction. [17:22] Anthony Aguirre: in the driver's seat [17:22] Anthony Aguirre: no going back [19:24] Anthony Aguirre: black hole [19:24] Anthony Aguirre: epistemic apocalypse [21:34] Anthony Aguirre: keep the future human

Original transcript

[0:05] So I want to talk today about AGI. And [0:09] it's a strange topic when you talk about [0:11] it because you feel like people are [0:13] living in two totally different [0:14] universes. There's like one universe [0:17] where AGI is the inevitable just next [0:20] step coming soon. No avoiding it. Uh [0:23] it's going to be great. the other side [0:26] in the other universe AGI is like [0:28] impossible. Um that it's [0:32] sorry wrong thing that it's uh you know [0:35] we've got these it's some combination of [0:37] like hype and sci-fi. We've got these [0:40] glorified autocomplete. They're far from [0:42] being human level in anything. Um and [0:45] it's very strange how these universes [0:47] talk past each other. But one thing they [0:49] share in common is that they sort of let [0:51] us off the hook because whether AGI is [0:53] inevitable or impossible, there's [0:55] nothing really to be done about it. But [0:57] I think the truth is somewhat different. [0:58] And I'm going to argue today that uh we [1:01] shouldn't let ourselves off the hook [1:02] that AGI is possible, that it [1:05] potentially is coming quite soon, but it [1:08] is not inevitable even now. and we're [1:11] going to have some very big uh decisions [1:13] to make as a species as to whether we go [1:15] forward into it and super intelligence [1:18] or not. So what is it we're talking [1:21] about when we're talking about AGI? [1:22] There are a lot of different definitions [1:24] as you know u many of them very vague [1:26] like human levelish AI some of them a [1:29] little bit more precise like we're going [1:31] to highly autonomously reproduce all the [1:33] valuable economic work that humans do. [1:36] Um the way I like to think about it is [1:38] not as artificial general intelligence [1:40] but as autonomous general intelligence [1:42] and in particular as the triple [1:44] intersection between three key [1:46] capabilities. One high intelligence that [1:49] is the ability to do tasks really well [1:51] individually. Another high generality [1:54] the breadth of tasks that a that a [1:56] system can do and new tasks that it can [1:59] learn. and high autonomy, the ability to [2:01] perform even complex tasks independent [2:04] of human intervention or oversight. [2:07] And [2:09] through the history of AI so far, we've [2:12] filled in a lot of this ven diagram. [2:14] We've got narrow, high intelligent [2:16] things like AlphaFold. We've got [2:19] autonomous things that are pretty dumb [2:20] like Roombas, but work well enough. [2:22] Little smarter ones like Whimos. We've [2:24] got very general things that aren't [2:26] either very intelligent uh as a system [2:29] or very autonomous like the AI [2:31] underlying Google. And we've got things [2:33] much more recently that are at the [2:35] intersection of quite intelligent and [2:37] quite general like Gemini and Claude and [2:40] uh chatgpt and so on. What we don't [2:44] really have are things that are all [2:46] three that are both general and [2:48] intelligent like the frontier models now [2:50] and highly autonomous. And [2:54] just like generality added something [2:56] very very qualitatively different to the [2:58] narrow intelligence systems we had [3:00] before. I think we're underestimating [3:02] how big of a deal it will be to have [3:04] generality and intelligence and add in a [3:06] high level of autonomy. It is the thing [3:08] that allows operation without human [3:10] oversight. It is the thing that allows [3:12] us to unlock a huge amount of speed in [3:14] the systems if they can operate [3:16] autonomously without uh human oversight. [3:18] It is the thing that allows them to not [3:21] be so easily under human control. It is [3:24] also the thing that unlocks a huge [3:27] potential economic value and shows the [3:29] driver behind a lot of the investment [3:31] that we have now. Because at the center [3:33] of this diagram now is what? Us humans, [3:36] right? If you make something that can [3:38] inhabit the center of that diagram, then [3:40] you make something that can slot in one [3:42] for one for us humans. And so I think [3:44] you then see what the driver is for [3:46] hundreds of billions of dollars of [3:47] capital expense building giant data [3:49] centers. It's not for $10 a month GPT [3:52] subscriptions. It's not even for the [3:54] foundation models underlying lots of [3:56] software products like many people at [3:58] this conference are making. It's to try [4:00] to capture a huge fraction or a [4:02] significant fraction of the tens of [4:04] trillions of dollars of global labor [4:06] market by swapping out humans for [4:08] machines. That's the thing that makes [4:10] sense of the giant expense that we're [4:11] doing. So I think it uh for that reason [4:15] we're seeing an enormous push into [4:18] autonomous systems into agents. We keep [4:20] hearing the year of the agent. U it is [4:23] hard uh because of the way that these [4:25] models have started out. Uh but that is [4:28] the direction that we're going. [4:31] So we can ask if the if the if the push [4:33] right now, if the huge economic and [4:35] corporate push is to build systems that [4:37] are even more intelligent and even more [4:40] general but also very very autonomous, [4:43] uh that is what I would call AGI. How [4:45] far away is that? Well, we can we can [4:47] look at them individually. So in terms [4:50] of intelligence, I like to look at [4:52] mathematics. So, this is a plot showing [4:54] progress in a uh performance in this [4:57] high school competitive math problems. [4:59] It's gone from pretty much crap to [5:01] saturated near 100%. [5:04] We've recently had both DeepMind and [5:07] OpenAI announce that they have gold [5:08] level performance in the International [5:10] Math Olympiad. These are quite tricky [5:12] questions to answer. Um, and there's a [5:15] significant progress in even more [5:18] difficult metrics. Uh, one for example [5:20] called frontier math. I'm a theoretical [5:22] physicist and I look at these questions [5:24] and I have no idea what they're even [5:25] asking. Um, so these these are like real [5:27] serious things. Um, they're not at human [5:30] level yet. They're not systems are not [5:32] at the frontier of mathematics yet, but [5:34] they're very smart and getting steadily [5:36] smarter. [5:38] In terms of generality, uh, also as a [5:41] scientist, I like to look at this set of [5:44] uh, PhD level science questions. Um [5:47] these are across chemistry and physics [5:48] and a bunch of other different uh [5:50] sciences. And you can see the progress [5:52] has gone also in the last two years from [5:55] basically random guessing to well above [5:59] uh professional human level. And the [6:01] trend just keeps going up. Uh the upper [6:03] envelope all those are three three [6:06] companies uh seems to have no curve, no [6:09] bend over, just keeps going up. Now, [6:13] uh, being able to answer questions about [6:16] at of these PhD level science questions [6:18] doesn't make you a PhD level scientist. [6:20] We're not there yet. These are not doing [6:22] original research like a PhD level [6:24] scientist. But it's not hard to imagine [6:26] that in two more years of progress like [6:28] this. We're going to be awfully close. [6:30] Like what exactly is it that scientists [6:32] are doing that is going to, you know, [6:35] not fall prey to this sort of progress [6:37] for another couple of years. [6:40] Now, what about autonomy? This one's a [6:42] lot harder to exactly measure what it [6:44] is, but a quite nice way of looking at [6:47] it has been developed by Meter. And what [6:49] they're what they do is they have a [6:51] suite of tasks which they which can like [6:54] it can be clear whether they're [6:56] accomplished or not. And they look for [6:58] how long it would take a human to do the [7:00] tasks that the AI can do at 50% accuracy [7:03] rate. And this plot has gone from like [7:08] seconds in back in GPT2 up to the [7:11] current of about of about a couple of [7:13] hours. It's been exponentially [7:15] increasing with time with a doubling [7:17] rate of about 6 months. So if it's 2 [7:19] hours now that these systems can more or [7:21] less do uh complicated tasks [7:24] autonomously [7:26] in a couple of years that's 16 more [7:29] times. That's 32 hours. That's pretty [7:31] much a whole work week. So, are there [7:34] tasks that we do at work that take more [7:35] than more than a week? Sure. But that's [7:38] a pretty darn autonomous system to [7:40] combine with the other two. Um, so [7:44] I think we have to really take AGI [7:46] seriously. I think far from impossible, [7:49] these first AI just AGI systems could be [7:52] here uh in a year or two away. I think [7:56] we're in a point where more than 10 [7:58] would be extremely surprising. like what [8:00] would we be doing on our current path [8:02] for 10 more years? How would we fit that [8:04] much progress in and not be at something [8:06] that is really human level, expert level [8:09] and all three of these attributes? And [8:11] so I think we have to take both the [8:13] potential and unavoidable consequences [8:14] of AGI seriously even when and this is [8:17] the hard part they feel too big to be [8:20] true because they do. [8:22] What does it mean to take it seriously [8:25] uh if we really have autonomous general [8:27] intelligence? Well, I think it's to take [8:29] seriously that there are going to be [8:31] things out there more like uh you know [8:35] goal-driven entities or beings than [8:37] tools that can really do uh all of the [8:40] things that humans can do, but they will [8:43] be more general with many superhuman [8:45] capabilities. They're going to be much [8:47] faster because you can scale them up [8:48] with computation, which you can't scale [8:50] up a human very much by just giving it [8:52] more juice. um they can be massively [8:54] copied, improved, modified in ways that [8:57] humans cannot. And [9:00] they're going to be out there. Um [9:04] the [9:06] if we build them, um and what will that [9:09] look like? Well, I think it will be a [9:11] quite different civilization than we [9:13] have now. Uh and we've seen how large [9:17] the change can be. like there's I think [9:19] now 700 million uh weekly users of chat [9:22] GBT. This has scaled like nothing [9:24] before. When there's an advance in AI [9:26] now, it gets pushed out to a huge [9:28] fraction of society. It's not like it [9:30] has to grow for a long time into a large [9:32] product. I think that the transformation [9:34] from what we have uh now where we're [9:36] we've automated tasks to what we have [9:38] when we can automate roles and people uh [9:41] could be tremendously disruptive very [9:42] quickly. There will be lots of good [9:44] things. Um there like certainly faster [9:47] technological and scientific growth. Uh [9:50] lots more productivity, but you know the [9:53] good things are what lots of people at [9:54] this conference are thinking about and [9:56] making happen. But I think there would [9:58] be going to be really large risks. So, [10:01] what does it look like when you [10:03] literally can't discern whether anything [10:05] that you read or see or experience uh [10:09] online is real or a human or uh based on [10:14] anything and or trustworthy? Uh we're [10:16] already seeing the start, but what does [10:18] it look like when like the M dashes are [10:20] gone and you just can't tell whether [10:22] anything is written or produced or [10:24] created by AI? [10:26] What does it look like in terms of uh [10:29] surveillance if everyone can be [10:30] surveiled by their own individual thing [10:32] that is faster and smarter and more [10:34] general than they are? [10:37] What does it look like when anybody uh [10:41] any rogue state or terrorist or whatever [10:44] has the exact same capabilities in terms [10:47] of highlevel scientific uh creation, [10:50] weapon design, viology, cyber security [10:53] as the top humans do. [10:56] What does it look like when the most [10:59] when the loudest voices in our public [11:01] discussions are not even human and have [11:03] no stake in things uh but are incredibly [11:05] persuasive and maybe even manipulative? [11:10] What does it look like if most humans [11:13] simply don't have any like significant [11:15] economic value in the workforce? Um and [11:19] suddenly this uh and this happens in [11:22] sort of a matter of years or even [11:24] months. [11:25] What does it look like when all the [11:27] companies your companies are forced to [11:30] compete against AIdriven and staffed [11:32] companies that operate a 100 times [11:34] faster and just as effectively? Like [11:36] what does the world look like? [11:38] What does it look like when a [11:39] substantial fraction of human labor and [11:41] productivity is running through a couple [11:43] of companies uh potentially and the [11:46] enormous data centers that they uh that [11:49] they create? [11:51] What does uh global conflict look like [11:53] with uh weapons and weaponry that is [11:56] designed by minds that are stronger than [11:58] ours? And that uh if you don't give the [12:02] AGI the ability to make decisions, you [12:05] will be at a huge military disadvantage. [12:10] But these are the small problems. Those [12:12] are the small problems [12:14] because the progress in AI will not stop [12:17] at AGI. Indeed, it will probably get [12:20] faster because like all of the other [12:23] things that AGI would be able to do that [12:25] humans can do, they will be able to do [12:27] AI research. They will be able to [12:28] improve themselves and AI in general. [12:31] And there are lots of different avenues [12:33] through which they can do this. some of [12:34] them on the time scale of months or [12:36] years, but some of them on the time [12:37] scale of days or weeks. And so it's [12:41] almost impossible to imagine that uh [12:44] given all these pathways that take days [12:46] or weeks, it that many years are going [12:48] to go by uh maybe maybe one uh before we [12:52] have from AGI to super intelligence, [12:54] something that really is competitive not [12:55] just with the top humans but with [12:57] humanity as a whole in doing science uh [13:00] or in strategy or in building new [13:02] companies, etc. [13:04] What are the risks of super [13:05] intelligence? Well, um we will not [13:09] control it. I think this is uh just [13:13] clear. If you imagine trying to control [13:15] one person that could run at 100 times [13:18] human speed, pretty hard. Not even [13:21] possible. How do you control something [13:23] that is running at 100 times your speed [13:26] and is smarter and is more complex and [13:28] is more capable than you are? It's like [13:30] a It's like a plant controlling a major [13:32] Fortune 500 company. It's not a thing [13:34] that happens. The control does not run [13:36] that direction. [13:38] We don't know uh what's going to happen. [13:41] We do know that at the moment we don't [13:42] know, not only do we not know how to [13:44] control super intelligence, and I don't [13:46] think it's even possible. We also don't [13:48] even know how to align super [13:49] intelligence with human interests. Um, [13:52] so it might be that the super [13:54] intelligences if we build them will be [13:56] more or less like aligned with the sorts [13:59] of things that we want. They might [14:00] discover that on their own. They might [14:01] come up with a morality that's [14:02] compatible with human well-being, but [14:04] they might not. And it's not going to be [14:06] up to us what happens. The era when [14:09] humans are in charge of what happens on [14:11] Earth is going to be over at that point. [14:15] Now, [14:18] um, could we choose not to do this? like [14:21] like when we talk about AGI and and you [14:24] know when I say well let's just not [14:26] build it [14:28] uh I get a lot of reactions and you [14:30] might have this in your mind right now [14:31] well it we're stuck you know it's again [14:33] the inevitable thing if we don't build [14:36] it uh if Google doesn't build it deep uh [14:39] open open AI will build it if they don't [14:40] build it anthropic will build it if they [14:42] don't build it XAI will build it and if [14:44] none of them build it China will build [14:45] it um so everybody thinks someone else [14:47] is going to build it and therefore we [14:48] have to build it first there's this [14:50] competitive race going on. Um, but I [14:53] would argue that we can choose not to [14:55] build it if the US and China really [15:00] understood, really felt the risk um and [15:04] the stakes like they did with nuclear [15:05] weapons at the end of World War II, the [15:08] US and the Soviet Union and other [15:09] countries, they absolutely could decide [15:11] not to build AGI and super intelligence. [15:14] Um, [15:16] super AGI and super intelligence both [15:18] take a huge amount of computation to [15:20] build. These chips are the most [15:22] sophisticated things that humanity has [15:24] built. They're incredibly difficult to [15:26] make. Uh, they're incredibly easy to [15:27] control. Even at the hardware level, you [15:29] can put controls into them. Uh, so if we [15:32] chose to say you cannot build a AI [15:35] system using more than a certain amount [15:37] of computation, that is something that [15:38] could be verified and enforced. Um, and [15:41] since the US and China together comp [15:43] control control basically all of the [15:45] chip production, that would be that. Um, [15:47] so that would be a way to prevent there [15:49] from being a runaway into super [15:50] intelligence. We can also think about uh [15:53] safety standards to just say if you're [15:55] going to build and deploy an AI system, [15:58] you should be able to control it. You [15:59] should be able to control it. Um, show [16:02] us that you can control this thing for [16:03] sure before you build and deploy the [16:05] system. that would more or less stop AGI [16:09] from coming into being right now because [16:10] nobody knows how to control even AGI let [16:12] alone super intelligence. Um and we [16:15] could have liability rules that made [16:17] sense. Um AGI is in it like it the [16:22] uh where the responsibility for uh the [16:26] actions of autonomous systems is going [16:28] to land is still something we have to [16:29] figure out. That's something the the [16:31] courts are going to have to figure out. [16:32] But I would favor a pretty strong set of [16:35] liability uh rules for something that is [16:39] near the middle of this diagram. And [16:41] then if you can show that your system is [16:44] uh narrow or if it's not that autonomous [16:46] or if it's not that general or some [16:47] combination of those or it's clearly [16:49] controllable because you can prove that [16:50] it's controllable, then you get a safe [16:52] harbor from those liability rules. So we [16:54] absolutely could do it uh if if the the [16:58] political will were there, we actually [16:59] could do it. [17:02] And [17:04] why are we nonetheless doing it uh at [17:07] the moment? Well, there's this [17:09] competitive thing and there's the lack [17:11] of political will. But I think those are [17:15] not like even if governments don't quite [17:17] realize and don't have the same level of [17:19] alarm that they did with nuclear [17:20] weapons. I think there is the fact that [17:22] this is not actually what most people [17:24] want, right? What do most people want? I [17:27] think they want powerful AI tools that [17:30] let them do things they couldn't do, [17:32] make them more productive, um save them [17:36] a bunch of drudgery, make them smarter, [17:38] uh all of these things. What they really [17:40] want are tools that they can control and [17:42] serve them. They don't want uh [17:44] uncontrollable, smarter than human AI [17:46] systems to replace them. And so I think [17:48] it's useful to think about uh what is [17:51] it? What is it that we can we name what [17:53] we actually want? And the way I think [17:55] about this is what I call and and at the [17:58] future of life institute we've been [17:59] calling tool AI has a definition and [18:01] sort of a a credo or an attitude. The [18:04] definition is that it is controllable uh [18:06] with guarantees that are commensurate [18:08] with its power. Uh as the stakes are [18:11] higher you better demonstrate more and [18:12] more that your system is controllable. [18:14] And if you're trying to build super [18:15] intelligence, before you do that, you [18:17] better prove that that thing is going to [18:19] be controllable because it's a uh [18:21] there's no going back if you build a [18:23] super an uncontrollable system that is a [18:25] super intelligence. So this is a [18:27] property. It's also an attitude though [18:29] that we should have meaningful human [18:31] control over our systems. Not on the [18:33] loop or uh just uh by standards, but we [18:36] should be in the loop. We should be in [18:38] the driver's seat. Uh that you empower [18:40] people rather than replacing them. that [18:43] you build AI to solve actual problems, [18:45] not just build the all-purpose thing and [18:47] then try to use that hammer on [18:49] everything that you turn into a nail. [18:51] And that you should have AI that's [18:53] proportional to the need. Like we don't [18:54] need super intelligent personal [18:56] assistance. This makes no sense. Like we [18:59] want something that can trust in a [19:01] trustworthy and accurate way fix our [19:03] spreadsheet. We don't need most of us to [19:06] solve Einstein's equations or like have [19:08] an expert viologist to fix our uh you [19:12] know to keep our schedule like uh we can [19:17] build the systems that we actually that [19:18] fit the purpose that we actually have [19:20] for them. [19:24] And there are lots of ways that we can [19:25] do this. If we decide let's not go into [19:27] the black hole in the middle of this [19:28] diagram but stay around the rest of it. [19:31] Uh there are lots of things we can make. [19:33] We can still have the passive language [19:35] and models and transformers that we have [19:37] now that are nice and quite useful [19:39] tools. We can have yes the trustworthy [19:42] limited loyal uh no conflict of interest [19:45] personal assistants uh that don't do [19:48] rocket science but do do the stuff that [19:50] we actually need them to do on a daily [19:51] basis. And if you're a rocket scientist, [19:53] you're a theoretical physicist, you can [19:54] have a special system for that. Like [19:56] that doesn't have to be built into every [19:57] single AI system. Um we can have [20:00] reliable household robots. We can have [20:02] narrow autonomous factory machines that [20:04] that are uh quite good. These have [20:08] risks, but they're manageable. Uh tools [20:11] are not always safe. So, they do have [20:13] risks, but they will displace, not [20:15] replace people. [20:17] And we could build AI systems to [20:18] actually solve problems we have like the [20:22] epistemic apocalypse. Like AI is making [20:24] things much, much worse, but it doesn't [20:25] have to. Imagine if every statement you [20:27] came to in a public document had a trail [20:29] of citations like a scientific paper [20:31] that you could follow back to either the [20:33] creation of a human mind or data that is [20:35] verified from the real world. Uh [20:37] imagine, you know, we now have a [20:39] capability that we've never had before [20:40] to have a 100,000 people in a real [20:42] conversation with each other. Why aren't [20:44] we doing it? Like why aren't we doing [20:46] it? Why aren't we coming up with ways of [20:49] coordinating better rather than worse [20:50] and being more polarized and [20:52] fractionalized? uh if we want to cure [20:54] aging and sickness, let's ask the aging [20:56] doctors and the cancer doctors, what do [20:58] you actually need to do those things? Uh [21:00] and pour billions of dollars into that. [21:02] Our ed our AI systems right now are [21:04] making our students dumber. Uh there is [21:07] no reason for that. It could be amazing [21:09] to have personalized wellcraftrafted [21:12] uh individual tutors uh to supplement [21:15] our our education. we don't have to be [21:17] do creating systems where it just makes [21:20] everybody disempowered because all they [21:21] do is ask GPT to do their homework for [21:23] them. So these are all things that we [21:25] could build and many many many more um [21:29] and these are empowering and make us [21:30] better not worse. [21:34] So I think we have uh here this pretty [21:38] pivotal choice. Um it's not a choice of [21:41] whether to go forward with AI or stop. [21:44] It's a choice of which road we want to [21:46] go down. And I think we have two basic [21:50] options at this point. We can continue [21:52] to build AI tools that empower humanity, [21:54] help us do science, improve health, [21:55] solve problems, increase productivity [21:57] and wealth, revolution revolutionize [21:59] education, or and this is what we're [22:02] planning to do right now. Even though [22:03] people say they're making tools, we can [22:05] build AGI, which will lead to super [22:07] intelligence. These systems will [22:09] massively disrupt our civilization. I [22:11] think there's no question about that. [22:13] They may cause war or other [22:14] catastrophes. We will almost certainly [22:16] lose control of them at a certain point. [22:18] We will probably lose control to them. [22:21] They will clearly replace humans in [22:23] their jobs and roles and eventually I [22:25] think if we keep going replace us [22:27] altogether. [22:29] The choice seems pretty obvious. Um I [22:32] think we can keep the future human but [22:34] we will have to try. I think we should [22:36] try. And you can see much more at keep [22:39] the futurehum.ai AI of this uh set of [22:42] arguments and thanks for listening.