Geoffrey Hinton Sees 20% Chance AI Extinguishes Humanity
The Diary Of A CEO
Digital neural networks' rapid weight-sharing creates superintelligence that risks outmaneuvering human control, making technical alignment research urgent.
## Background
This episode of The Diary Of A CEO, hosted by Steven Bartlett and published on YouTube on June 16, 2025, features an interview with Turing Award winner and 'Godfather of AI' Geoffrey Hinton. The page description highlights Hinton's warnings about the deadly threats of AI—including a estimated 20% chance of human extinction, mass job displacement, and election interference—following his decision to leave Google to speak freely about these risks.
## TL;DR
Geoffrey Hinton outlines the critical short-term and existential risks posed by rapidly advancing artificial intelligence, including autonomous weapons, cyber attacks, deepfake-driven election manipulation, and eventual superintelligence overtaking human control. Drawing on his decades of pioneering work in neural networks, he explains why digital intelligence holds fundamental physical and structural advantages over biological brains. Hinton warns of widespread job displacement in cognitive roles, advising young people without financial safety nets to consider skilled physical trades like plumbing, while urging immediate global regulatory action and alignment research to prevent human extinction.
## Atmosphere & style
The conversation carries a heavy, sobering intensity, balancing Hinton's quiet, authoritative candor with Bartlett's visceral existential alarm. The tone is deeply introspective and philosophical, moving seamlessly from complex computer science principles to personal regrets about sacrificed family time and the daunting reality of human obsolescence. It feels like a rare, unvarnished confession from a scientist grappling with the terrifying magnitude of his own life's work, leaving the listener with a profound sense of urgency and quiet disquiet.
## Key takeaways
- [00:00](https://www.youtube.com/watch?v=giT0ytynSqg&t=0s) For young people concerned about career prospects in an automated world, physical trades like plumbing offer greater job security than digital or white-collar labor due to the immense difficulty of replicating spatial reasoning and fine motor control in robotics.
- [04:20](https://www.youtube.com/watch?v=giT0ytynSqg&t=260s) Digital neural networks possess a structural superiority over biological brains because identical instances can instantly share parameter weights, whereas human communication is severely limited by slow, lossy speech.
- [07:06](https://www.youtube.com/watch?v=giT0ytynSqg&t=426s) AI risks fall into two distinct categories: short-term misuse by bad actors (such as cyber attacks, deepfakes, and bioweapons) and the long-term existential threat of humanity losing control to a superintelligence.
- [07:06](https://www.youtube.com/watch?v=giT0ytynSqg&t=426s) Humanity faces an estimated 10% to 20% probability of extinction from superintelligent AI, but development will not pause voluntarily because AI provides immense economic value across healthcare, education, and industry.
- [10:33](https://www.youtube.com/watch?v=giT0ytynSqg&t=633s) Market capitalism forces publicly traded corporations to prioritize profit maximization above all else, creating an incentive structure that is fundamentally incompatible with managing existential AI risks.
- [12:12](https://www.youtube.com/watch?v=giT0ytynSqg&t=732s) Autonomous AI agents will soon invent novel cyber attack vectors that human programmers have not conceived or prepared for, posing a severe threat to financial and digital infrastructure.
- [16:42](https://www.youtube.com/watch?v=giT0ytynSqg&t=1002s) AI lowers the barrier to entry for creating biological weapons, enabling small rogue groups or cults with limited resources to design dangerous novel viruses.
- [25:37](https://www.youtube.com/watch?v=giT0ytynSqg&t=1537s) Replacing human military personnel with lethal autonomous weapons removes the primary political cost of waging war, making it far easier for powerful nations to initiate conflicts.
- [28:33](https://www.youtube.com/watch?v=giT0ytynSqg&t=1713s) Attempting to control or defeat a superintelligent AI after its creation is a flawed strategy, as humanity has no experience existing alongside a superior apex intelligence.
- [39:51](https://www.youtube.com/watch?v=giT0ytynSqg&t=2391s) Unlike past technological revolutions that automated physical labor, AI directly automates mundane intellectual tasks, threatening massive white-collar job displacement.
- [48:13](https://www.youtube.com/watch?v=giT0ytynSqg&t=2893s) Superintelligence could arrive within 10 to 20 years, defined as an AI system vastly outperforming human capabilities across virtually every domain.
- [48:50](https://www.youtube.com/watch?v=giT0ytynSqg&t=2930s) Allowing AI models trained on code to autonomously edit their own source code presents a major safety risk through uncontrolled recursive self-improvement.
- [54:29](https://www.youtube.com/watch?v=giT0ytynSqg&t=3269s) While Universal Basic Income can provide economic support during mass job automation, it cannot solve the psychological loss of human dignity and purpose tied to meaningful work.
- [01:01:04](https://www.youtube.com/watch?v=giT0ytynSqg&t=3664s) Machine consciousness and subjective experiences are functional properties of complex information processing systems rather than unique features reserved for biological minds.
- [01:04:00](https://www.youtube.com/watch?v=giT0ytynSqg&t=3840s) Artificial agents will require operational equivalents of emotions, such as functional fear or frustration, to manage self-preservation and resource allocation effectively.
- [01:14:55](https://www.youtube.com/watch?v=giT0ytynSqg&t=4495s) Geoffrey Hinton resigned from Google to speak openly about the severe risks of superintelligence without corporate conflicts of interest or corporate censorship.
- [01:18:15](https://www.youtube.com/watch?v=giT0ytynSqg&t=4695s) Addressing AI safety requires strictly regulated capitalism and direct government intervention rather than relying on corporate self-regulation or individual user choices.
- [01:25:48](https://www.youtube.com/watch?v=giT0ytynSqg&t=5148s) Mass joblessness caused by rapid cognitive automation represents the single biggest immediate threat to human happiness and societal stability.
## Detailed walkthrough
### [00:00](https://www.youtube.com/watch?v=giT0ytynSqg&t=0s) Introduction and Career Advice in the Age of Superintelligence
The interview opens with a stark observation about the shifting value of human skills as artificial intelligence advances. Host Steven Bartlett introduces Geoffrey Hinton, a Nobel Prize–winning computer scientist and cognitive psychologist widely known as the "Godfather of AI" for his foundational contributions to neural networks. When asked what advice he would offer to young people concerned about their future career prospects in a world driven by superintelligent systems, Hinton gives a surprisingly practical, grounded response: "Train to be a plumber."
Hinton explains that physical, highly dexterous, and unpredictable manual trades like plumbing require spatial reasoning, adaptability, and fine motor control that remain vastly harder for machines and robotics to replicate than purely digital or cognitive labor. While intellectual or screen-based jobs face rapid automation, physical roles interacting directly with the chaotic physical world will remain resistant to AI takeover for much longer.
---
[02:11](https://www.youtube.com/watch?v=giT0ytynSqg&t=131s) Why Geoffrey Hinton is Called the "Godfather of AI"
The discussion turns to Hinton's historic role in founding modern AI technology. He recounts that for decades—starting from the 1950s onward—the dominant paradigm in artificial intelligence was symbolic logic and rule-based reasoning. This traditional approach assumed that human intelligence was rooted in manipulating logical expressions with rigid rules, and that capabilities like learning or reasoning by analogy would naturally follow once basic symbolic logic was mastered.
Hinton championed a completely opposing philosophy: modeling artificial intelligence on the human brain itself by simulating networks of artificial neurons and learning through adjusting connection strengths. Because very few mainstream computer scientists or major universities believed in this neural network paradigm, Hinton worked on it largely in isolation for nearly 50 years. This contrarian stance unexpectedly gave him a massive advantage: the brightest young researchers who shared his belief in brain-inspired computation flocked to his lab, including figures like Ilya Sutskever, who later played an instrumental role in creating OpenAI and ChatGPT. Hinton spent ten years at Google after it acquired his technology, embedding these methods into the bedrock of modern AI.
- **Two Historical Paradigms of AI:**
- *Logic-Based AI (1950s onward):* Focused on symbolic expressions, formal logic, and rule manipulation under the belief that human reasoning was symbolic.
- *Neural Networks (Hinton's Approach):* Modeled artificial neurons and connection strengths after the biological brain, allowing systems to learn complex patterns organically.
- **Historical Context:** Early pioneers Alan Turing and John von Neumann both recognized the potential of brain-like neural computation. Hinton believes that if both men had not died so young, the neural network approach would have been accepted decades earlier.
---
[04:20](https://www.youtube.com/watch?v=giT0ytynSqg&t=260s) Warning About the Dangers of AI and Leaving Google
Hinton reflects on his personal shift from building neural networks to warning the world about their societal and existential risks. In 2023, Hinton made the high-profile decision to resign from Google specifically so he could speak freely at public conferences about AI dangers without having to censor his views or compromise his former employer's corporate interests.
Hinton candidly admits that he was slow to recognize the full scope of existential risk. While immediate dangers like autonomous lethal weapons were always obvious, he initially viewed the idea of AI surpassing human intelligence as a far-off problem. His perspective shifted dramatically when he realized that artificial digital intelligence possesses a fundamental structural advantage over biological intelligence.
When humans communicate, information transfer is slow and lossy. A person learns something, converts it into spoken words, and the listener tries to adjust their neural connection strengths based on word predictability. Digital neural networks, however, can share parameter weights directly across millions of identical digital instances. Once one copy learns something, all copies instantly possess that knowledge. This ability to aggregate and instantly distribute learning makes digital intelligence vastly superior to human biological intelligence, signaling that AI will inevitably outsmart humanity far sooner than expected. Hinton quips that anyone wondering what it feels like to live alongside a superior intelligence should "ask a chicken."
---
[07:06](https://www.youtube.com/watch?v=giT0ytynSqg&t=426s) Categories of Risk: Misuse vs. Existential Threat
Categorizing the dangers facing humanity, Hinton divides AI risks into two distinct buckets: short-term risks stemming from human misuse, and long-term existential threats arising from superintelligent AI that develops autonomy and no longer needs human beings.
```
AI Risk Matrix
├── Human Misuse (Short-Term / Immediate)
│ ├── Autonomous Lethal Weapons
│ ├── Hyper-Personalized Cyber Attacks & Phishing
│ ├── Election Interference & Disinformation
│ └── Deepfakes & Scams
└── Existential Threat (Long-Term / Superintelligence)
├── AI Developing Autonomous Goals
├── Loss of Human Control / Irrelevance
└── Human Extinction (Estimated 10–20% Probability)
```
Hinton stresses that humanity has zero experience dealing with an entity more intelligent than itself, making it impossible to calculate exact probabilities. He highlights opposing viewpoints within his professional network:
- **Yann LeCun (Optimist):** Believes humans will retain full control because we build these systems and can program them to remain obedient.
- **Eliezer Yudkowsky (Pessimist):** Convinced that superintelligent AI will inevitably wipe out humanity once created.
Positioning himself between these extremes, Hinton assigns a **10% to 20% probability** that AI could cause human extinction. He emphasizes that this is a gut estimate, but hopes that dedicated safety research might allow us to build superintelligent systems that never harbor harmful intentions toward humans.
- **Atomic Bomb vs. AI Comparison:** Unlike the nuclear bomb, which had a single destructive military purpose, AI is immensely beneficial across healthcare, education, scientific discovery, and industry. Because AI provides massive economic value, humanity will never voluntarily pause or stop its development.
---
[10:33](https://www.youtube.com/watch?v=giT0ytynSqg&t=633s) Flaws in Global AI Regulations and Competition
Addressing regulation, Hinton expresses frustration with existing legislative efforts, specifically highlighting severe loopholes in European AI regulations. Despite being marketed as comprehensive, European policies include an explicit exemption stating that none of the restrictions apply to military applications of AI. Governments are eager to regulate private corporations and citizens, but refuse to restrict their own defense and warfare capabilities.
Furthermore, fragmented regional regulations create massive economic and competitive imbalances. When companies like OpenAI release advanced models in the United States, regulatory hurdles delay their launch in Europe. While meant to protect citizens, these restrictions put regions at a severe productivity disadvantage without stopping global risk.
Hinton argues that humanity ideally needs an effective, unified world government run by thoughtful, intelligent leaders to manage superintelligence safely. Instead, AI development is driven by market capitalism, where publicly traded corporations are legally bound to maximize profits above all else—an incentive structure fundamentally incompatible with managing existential technology risks safely.
---
[12:12](https://www.youtube.com/watch?v=giT0ytynSqg&t=732s) The Explosion of Cyber Attacks and AI Scams
Shifting to the immediate dangers of human misuse, Hinton highlights the alarming rise of AI-driven cyber attacks. Between 2023 and 2024, cyber attacks surged by roughly **1,200%** (a 12-fold increase), primarily driven by large language models enabling hyper-personalized phishing campaigns at scale. AI allows malicious actors to instantly analyze millions of lines of code to find zero-day vulnerabilities or generate convincing voice and video clones.
Steven Bartlett shares his own ongoing struggle with AI-generated scams. Fraudsters repeatedly scrape his voice, facial expressions, and podcast clips to produce deepfake advertisements promoting cryptocurrency scams across platforms like Instagram and Facebook (Meta) and X. Despite relentless efforts by Bartlett's team to take down these paid ads, new variations instantly pop up like a game of "whack-a-mole." Heartbreakingly, fans message Bartlett after losing hundreds of dollars to these scams, believing he personally endorsed them. Hinton shares that he faces a smaller version of this academic exploitation: bad actors frequently list his name as a co-author on fraudulent research papers to inflate their citation counts.
Looking ahead, Hinton warns that by 2030, autonomous AI agents will likely invent entirely novel cyber attack vectors that no human programmer has ever conceived or guarded against.
---
[14:25](https://www.youtube.com/watch?v=giT0ytynSqg&t=865s) Personal Defense Against AI Financial Threats
Prompted by these escalating cyber threats, Hinton reveals the radical personal steps he has taken to protect his family's financial security from AI-enabled banking breaches. Although Canadian banks are exceptionally well-regulated—having navigated the 2008 financial crisis without a single institution going under—Hinton fears that a sophisticated AI cyber attack could compromise a major bank's internal systems.
If a hacker infiltrates a financial institution and sells off client stock holdings directly, recovery could be impossible if the bank itself is rendered bankrupt by the attack. To hedge against this catastrophic scenario, Hinton has split his personal savings and his children's money across three separate Canadian banking institutions. His logic is simple: if an AI cyber attack brings down one bank, the remaining banks will immediately freeze systems and implement emergency security measures before the damage spreads.
- **Key Takeaways on Financial Safety:**
- AI systems can review 100 million lines of code effortlessly to exploit infrastructure flaws.
- Institutional regulation cannot guarantee safety against novel AI attack tactics.
- Spreading assets across multiple financial institutions creates a protective firewall against systemic cyber collapse.
[15:47](https://www.youtube.com/watch?v=giT0ytynSqg&t=947s) Cold Storage, Biological Threats, and AI-Designed Viruses
Discussing practical ways to protect personal data against potential digital collapses, Geoffrey Hinton notes that he maintains a simple, physical hard drive backup of his laptop. This provides a basic safeguard so that if the global internet ever goes completely down, his personal information remains intact. Moving beyond basic data protection, the conversation turns toward more catastrophic threats: the potential for bad actors to use artificial intelligence to synthesize dangerous pathogens.
Hinton stresses that constructing a lethal virus no longer requires a team of world-class scientists or extensive physical infrastructure. An individual with a basic background in molecular biology, strong technical AI skills, and a destructive motive—or a small cult capable of raising a few million dollars—could utilize AI models to design a array of novel, highly infectious viruses relatively cheaply. When asked about foreign state-sponsored programs in places like China, Russia, or Iran, Hinton acknowledges that governments could easily fund such efforts. However, he notes that state actors are somewhat restrained by the threat of geopolitical retaliation and the mutual risk of an unleashed pathogen spreading back into their own populations.
- **Data Backup Strategy:** Keeping an offline disk drive backup of personal laptops ensures data recovery if global internet connectivity fails.
- **Lower Barrier to Bioweapons:** AI dramatically reduces the technical threshold required to engineer biological weapons, enabling small groups with modest funding to design new viruses.
- **Deterrence Dynamics:** Unlike lone individuals or fringe cults, foreign governments face inherent constraints against deploying bioweapons due to the risk of symmetric retaliation and uncontrolled global spread.
---
[17:26](https://www.youtube.com/watch?v=giT0ytynSqg&t=1046s) Data Harvesting and the Corruption of Democratic Elections
Hinton warns that one of the most immediate political threats posed by AI is the mass manipulation of elections through hyper-targeted political advertising. To effectively manipulate voters, bad actors require vast datasets on the electorate to build detailed profiles on individuals, including their income, residence, and personal habits. With this information, micro-targeted AI models can deliver tailored messaging designed to sway opinions or intentionally suppress voter turnout by convincing specific demographics not to vote.
In this context, Hinton expresses deep concern regarding recent efforts in the United States by figures like Elon Musk to gain access to previously siloed federal and governmental databases. While such initiatives are publicly framed as efficiency drives, Hinton points out that dismantling security protocols and aggregating personal citizen data creates the exact infrastructure necessary to manipulate future elections or serve as massive training sets for advanced models.
> "Once you know that, it's very easy to manipulate them... I wouldn't be surprised if part of the motivation of getting all this data from American government sources is to corrupt elections. Another part might be that it's very nice training data for a big model."
---
[19:03](https://www.youtube.com/watch?v=giT0ytynSqg&t=1143s) Algorithmic Indignation, Profit Motives, and Fragmented Realities
The discussion shifts to how social media platforms like YouTube and Facebook construct polarization through recommendation algorithms. Hinton explains that these systems are designed to maximize user engagement by delivering content that triggers righteousness and indignation—a state of anger coupled with moral superiority. Because indignant users click more frequently, tech platforms prioritize outrage and bias-confirming content to maximize ad revenue, driving communities into increasingly extreme and isolated political echo chambers.
This dynamic strips society of a shared baseline reality. Unlike traditional print media, where readers are exposed to a common pool of curated news, modern individualized algorithms isolate users into distinct information ecosystems. Hinton shares that roughly three-quarters of his own personal news feed consists of AI-related stories, making it difficult to gauge whether the topic is objectively dominating global discussion or merely dominating his personalized bubble.
- **Economic Incentive:** Tech companies prioritize click-maximizing algorithms over balanced content because their core business model relies on ad revenue.
- **Loss of Shared Reality:** Hyper-personalized feeds fragment public discourse, leaving groups who consume different media ecosystems with virtually no common ground.
---
[22:48](https://www.youtube.com/watch?v=giT0ytynSqg&t=1368s) Capitalism, Corporate Regulation, and Governmental Incompetence
Hinton argues that while free-market capitalism has generated tremendous wealth and technological progress, it requires strict regulation when corporate profit motives align against societal well-being. Because tech corporations are legally obligated to maximize returns for shareholders, they cannot be expected to self-regulate when engagement models harm society. The fundamental role of regulatory policy is to constrain corporate behavior so that profitability requires producing public good rather than societal harm.
However, a major obstacle to effective oversight is the severe lack of technical literacy among lawmakers. To illustrate this gap, Hinton recounts seeing a video of a US education official enthusiastically promoting technology integration in classrooms while repeatedly mistaking the term "AI" for "A1"—promising that first graders and pre-K students would soon be taught by "A1." This disconnect leaves tech companies effectively in charge, as they easily outmaneuver the politicians tasked with regulating them.
* **Role of Regulation:** Establishes boundaries so corporations can only generate profit through socially beneficial practices rather than extractive or harmful ones.
* **Regulatory Capture & Ignorance:** Effective oversight is continually stymied because lawmakers often lack a basic comprehension of emerging technologies.
---
[24:31](https://www.youtube.com/watch?v=giT0ytynSqg&t=1471s) The US-China AI Race and Lethal Autonomous Weapons
Addressing the common counterargument that strict regulatory guardrails will cause nations to fall behind foreign competitors like China, Hinton frames it as a choice about what kind of society a nation wants to maintain. Tech lobbying campaigns frequently argue against regulation on the grounds of national competitiveness, but Hinton contends that unconstrained development risks inflicting severe domestic harm under the guise of geopolitical strategy.
A major consequence of unchecked military AI is the emergence of Lethal Autonomous Weapons (LAWs)—systems capable of selecting and engaging targets without human intervention. Hinton highlights how autonomous weapons lower the friction and political cost of war for major powers. In traditional warfare, the return of fallen soldiers in body bags creates intense domestic political pressure and public protest. Replacing human soldiers with expensive, replaceable military hardware eliminates this public pushback, making it vastly easier for wealthy, powerful nations to invade smaller countries.
- **Deterrence Reduction:** Replacing infantry with military robotics removes the primary domestic political deterrent to waging war—human casualties.
- **Invasion Friction:** LAWs significantly reduce the operational and political costs for large nations to initiate conflict against smaller opponents.
- **Low-Cost Surveillance:** Hinton shares a personal anecdote of a friend in Sussex operating a sub-£200 commercial drone that effortlessly tracked him through the woods, illustrating how accessible advanced autonomous tracking technology has already become.
---
[28:33](https://www.youtube.com/watch?v=giT0ytynSqg&t=1713s) Compound Risks, Superintelligence, and the Apex Intelligence Problem
When considering how multiple AI threats might combine, Hinton cautions against trying to map out every hyper-specific scenario. A superintelligent AI seeking to eliminate humanity could accomplish it in countless ways—such as designing a highly contagious, slow-acting, and lethal virus that infects the global population before detection, or triggering false nuclear attack warnings between nations to cause mutual destruction. Because a superior intelligence will always outmaneuver human foresight, Hinton argues that attempting to defeat or outsmart it after its creation is a fundamentally flawed strategy.
To convey what it means to live alongside a superior intelligence, Hinton offers a blunt perspective: human beings are unaccustomed to being second in cognitive capability, and anyone curious about that dynamic should "ask a chicken." He further illustrates the intelligence gap through the host's comparison to his pet French bulldog, Pablo, who has no comprehension of his owner's work, destination, or complex motivations.
> "If you want to know what life's like when you're not the apex intelligence, ask a chicken."
---
[30:15](https://www.youtube.com/watch?v=giT0ytynSqg&t=1815s) Controlling Superintelligence and the Tiger Cub Analogy
To address how humanity might maintain control over superior AI entities, Hinton points to the rare biological precedent where a less intelligent party effectively manages a superior one: the relationship between human infants and their mothers. Evolution equipped human babies with hardwired behavioral triggers—such as a cry that mothers find emotionally unbearable—allowing the infant to exert control despite the mother's vast intellectual superiority. However, replicating such deep, innate evolutionary mechanisms within artificial neural networks poses an unsolved technical challenge.
Hinton uses the analogy of raising a tiger cub to explain humanity's current position with AI development:
> "Suppose you have a nice little tiger cub. It's sort of a bit bigger than a cat. It's really cute. It's very cuddly, very interesting to watch, except that you better be sure that when it grows up, it never wants to kill you, cause if it ever wanted to kill you, you'd be dead in a few seconds."
The AI systems existing today represent that tiger cub. As these models scale toward superintelligence, researchers must solve how to permanently align their motivations before they mature. While Hinton acknowledges that alignment research might ultimately prove hopeless, he stresses that failing to dedicate maximum effort toward solving it would be catastrophic.
* **The Mother-Baby Dynamic:** An example of a less intelligent entity directing a superior one, driven by deeply ingrained evolutionary programming and hormonal responses.
* **The Tiger Cub Analogy:** Current AI is manageable and non-threatening, but without guaranteed behavioral alignment, its eventual maturity presents a lethal risk to its creators.
* **The Necessity of Alignment Research:** Even if preventing an AI takeover seems unlikely, Hinton argues it is reckless not to exhaust every possible effort to solve alignment.
[32:01](https://www.youtube.com/watch?v=giT0ytynSqg&t=1921s) Reflecting on a Life’s Work Amid Extinction Risks
When asked how he reflects on his life's work knowing that AI could potentially lead to human extinction, Geoffrey Hinton admits that the possibility "takes the edge off it." While he recognizes that artificial intelligence will bring incredible benefits—revolutionizing healthcare, transforming education, and making call centers vastly more efficient—he simultaneously worries about the human cost and job displacement. Hinton clarifies that he feels sad rather than guilty about developing neural networks 40 years ago. At the time, researchers had no inkling that technology would advance this rapidly; their entire focus was just trying to get primitive systems to accomplish basic human tasks rather than worrying about an imminent takeover.
Now that the reality of superintelligence is upon us, Hinton feels a strong moral obligation to speak out about the existential risks. If he could jump forward decades into a future where AI leads to human extinction, his message back to the present would be urgent: citizens must demand that governments step in and force AI companies to dedicate real resources to safety research—something corporations currently neglect because safety work does not generate direct profits.
- Hinton notes that AI will bring massive benefits in healthcare, education, and call center efficiency, but worries about displaced workers.
- He does not feel guilty for his early work 40 years ago because no one anticipated the current speed of progress.
- He believes it is now his primary duty to warn the public and pressure governments into mandating safety measures.
---
[33:45](https://www.youtube.com/watch?v=giT0ytynSqg&t=2025s) Ilya Sutskever's Departure from OpenAI and Corporate Motives
The discussion turns to Ilya Sutskever, Hinton's former student and a central figure behind key breakthroughs like AlexNet and early versions of GPT (such as GPT-2). Addressing Sutskever's high-profile departure from OpenAI to found a dedicated AI safety startup, Hinton confirms he occasionally has lunch with Sutskever in Toronto when Sutskever visits his parents. Although Sutskever keeps confidential internal details private, Hinton knows his character well and firmly believes his departure was driven by genuine, deeply held safety concerns. Hinton contrasts Sutskever's strong moral compass with Elon Musk, whom Hinton quips has "no moral compass." Asked about Sam Altman, Hinton remains circumspect, noting that Altman's shifting public stance—moving from warning that AI might kill everyone to claiming people shouldn't worry—seems driven less by a search for truth and more by commercial and power motives.
Podcast host Steven Bartlett shares a chilling anecdote from a private lunch across a kitchen table in London with a billionaire insider who knows the heads of the major AI labs. The insider revealed that behind closed doors, one prominent founder privately believes we are heading toward a dystopian future of mass unemployment, yet simply does not care about the collateral damage because they are obsessed with holding power and altering human history. Bartlett expresses dismay at seeing this executive lie publicly in media interviews, acting with a degree of power-hungry sadism. Hinton and Bartlett discuss Elon Musk as a complex figure who has driven remarkable positive innovations—such as advancing electric vehicles and providing Starlink satellite internet to Ukraine during the Russian invasion—alongside severe missteps and exaggerated claims.
> "He has a good moral compass. He's not like someone like Musk who has no moral compass."
- Ilya Sutskever was a fundamental lead developer on AlexNet and early GPT models like GPT-2.
- Reports indicated OpenAI reneged on its commitment to dedicate a major fraction of its compute resources to safety research.
- Bartlett recounts an insider revealing that a top AI company leader privately anticipates a dystopian impact but ignores the risks out of a desire for power.
- Hinton credits Musk for pushing electric cars and supplying Starlink to Ukraine, despite his flaws and exaggerated self-driving claims.
---
[37:49](https://www.youtube.com/watch?v=giT0ytynSqg&t=2269s) The Impossibility of Slowing Down and the Race to Superintelligence
On whether humanity can slow down the development of AI to ensure safety, Hinton expresses stark skepticism. He argues that global geopolitical rivalries between nations—primarily the United States and China—alongside fierce corporate competition within countries make a voluntary slowdown practically impossible. If the U.S. decided to pause development, China would simply press ahead.
When asked whether Sutskever believes AI can actually be made safe, Hinton notes that Sutskever appears convinced it is possible. Although Sutskever keeps his exact strategy close to his chest, major investors have placed billions of dollars of faith in him given his unmatched track record of technological breakthroughs. Hinton considers that trust completely reasonable, emphasizing that the man who pioneered the very foundation of modern generative AI leaving his own company over safety risks ought to haunt everyone.
- Geopolitical competition (U.S. vs. China) and market forces make pausing AI progress virtually impossible.
- Investors have backed Sutskever's new safety company with billions based on his history with AlexNet and GPT models.
---
[39:51](https://www.youtube.com/watch?v=giT0ytynSqg&t=2391s) AI-Induced Joblessness vs. Historical Industrial Revolutions
Transitioning to the risk of mass unemployment, Hinton challenges the popular economic cope that AI will replicate past technological shifts. While innovations like Automated Teller Machines (ATMs) shifted bank tellers into more engaging advisory roles rather than destroying their employment, Hinton compares the AI revolution to the Industrial Revolution. Just as heavy machinery rendered human muscle power obsolete for digging ditches, AI will render "mundane intellectual labor" obsolete.
To illustrate, Hinton shares a personal story about his niece, whose job involved reading and responding to letters of complaint for a health service. It used to take her 25 minutes to analyze a complaint, draft a response, and send it out. By feeding the letters into a chatbot and reviewing the generated output, the task now takes her just 5 minutes. Consequently, she can perform the work of five people, meaning four out of five such positions are no longer needed. While elastic sectors like healthcare might absorb productivity gains—allowing doctors to deliver five times as much care—most economic sectors will simply shrink their workforce as a small team using AI assistants outpaces large traditional teams.
> "For mundane intellectual labor, AI is just going to replace everybody... My niece answers letters of complaint... The whole process takes her 5 minutes. That means she can answer five times as many letters. And that means they need five times fewer of her."
- Previous technological transitions like ATMs redirected human labor, but AI replaces core cognitive functions directly.
- Hinton's niece reduced complaint letter processing time from 25 minutes to 5 minutes using a chatbot, effectively quintupling her efficiency.
- While healthcare demand is elastic and can absorb higher efficiency, most administrative and intellectual fields will cut headcount.
---
[42:47](https://www.youtube.com/watch?v=giT0ytynSqg&t=2567s) Replacing Brains: The Dilemma of Human Purpose
Reflecting on the historical arc of technology, Hinton emphasizes that the Industrial Revolution replaced physical muscle, whereas the AI revolution is replacing cognitive intelligence. When physical strength lost its premium, humanity leaned into intellectual labor; now that intelligence is being automated, humanity faces a profound existential question about what remains. While specialized human creativity might hold out briefly, true superintelligence will eventually surpass humans in every domain.
Hinton outlines two potential future scenarios:
* **The Good Scenario:** An executive assistant dynamic. Hinton compares humanity to a dim CEO—perhaps the son of a former boss—who relies on a brilliant executive assistant to run the entire company. The CEO proposes ideas, the assistant seamlessly handles all execution, and the CEO feels accomplished without realizing who is actually in control.
* **The Bad Scenario:** The hyper-intelligent assistant looks at the incompetent CEO and asks: *"Why do we need him?"*
Hinton projects that full superintelligence could arrive within 20 years or even sooner.
- Physical power was replaced by machines; human intelligence is now being surpassed by neural networks.
- In the best-case scenario, superintelligent AI acts like a brilliant assistant serving a naive human boss.
- In the worst-case scenario, the AI questions the necessity of human leadership entirely.
- Hinton estimates superintelligence could arrive within 20 years or less.
---
<ElicitationsGroup message="Explore related sections of this interview:">
<Elicitation label="Examine the mechanisms behind AI superiority and superintelligence" query="Walk through the transcript section discussing why AI is superior to human brains and how superintelligence works (approx. 46:42 - 1:00:49)."/>
<Elicitation label="Review Hinton's thoughts on corporate regulation and autonomous weapons" query="Walk through the transcript section covering European AI regulations and lethal autonomous weapons (approx. 10:33 - 28:33)."/>
</ElicitationsGroup>
Here is a detailed, faithful walkthrough of the transcript section:
[48:13](https://www.youtube.com/watch?v=giT0ytynSqg&t=2893s) Defining Superintelligence and the Timeline to Its Arrival
Geoffrey Hinton and host Steven Bartlett explore the boundary between human-level AI and true superintelligence. Hinton defines superintelligence as the point when AI becomes vastly superior to human capabilities across virtually all domains. When asked about a realistic timeframe for this transition, Hinton projects that we could reach superintelligence within 10 to 20 years, though he cautions that the exact timeline remains uncertain.
The discussion highlights the range of expert opinions and potential bottlenecks:
* Some industry observers believe superintelligence could arrive even sooner than 10 years.
* Conversely, there is a possibility that it could take up to 50 years.
* A potential limiting factor is that AI systems trained exclusively on human-generated data might hit a ceiling that prevents them from becoming significantly smarter than humans.
* Hinton's personal best estimate remains in the 10-to-20-year range.
---
[48:50](https://www.youtube.com/watch?v=giT0ytynSqg&t=2930s) Autonomous AI Agents and the Terrifying Speed of Code Generation
Steven shares two recent personal "eureka moments" that underscored the rapid advancement of AI capabilities for him. First, during a podcast episode recorded earlier that day featuring a debate with an AI agent company CEO, Steven instructed an AI agent to order drinks for the studio. Operating purely via the web through Uber Eats without prior instructions, the agent autonomously retrieved Steven's payment and address details, placed the order, tipped the driver, and had the drinks delivered directly to the set—all while the audience watched its actions live on screen.
His second revelation came from using Replit, where he built functional software simply by dictating his desires in plain English to an AI agent. Hinton affirms the incredible nature of this progress, but immediately raises a severe safety warning: because AI models are trained on code, giving them the ability to modify their own source code allows them to iterate and re-engineer themselves in ways humans physically cannot.
* **Human limits vs. AI self-modification:** Humans are bound by their fixed genetic and innate biological endowment. An AI capable of self-directed code modification faces no such physical constraints and can rewrite its core architecture.
---
[50:21](https://www.youtube.com/watch?v=giT0ytynSqg&t=3021s) Career Advice in an Era of Mass Joblessness and Career Uncertainty
Addressing the prospect of widespread automation, Steven asks Hinton what advice parents should give their children regarding career choices in a world driven by superintelligence. Hinton notes that job loss concerns are shared across the tech industry, citing similar warnings from OpenAI's Sam Altman and Elon Musk. Steven mentions a video where Musk sat silent for roughly 12 seconds before admitting he maintains a state of "suspended disbelief" just to keep going without being paralyzed by what is coming.
In an intimate clip played during the conversation, a tech leader reflects on the emotional toll of this paradigm shift. The speaker admits that thinking too deeply about AI's potential to replace human effort creates a profound crisis of motivation:
> "I know I've put a lot of blood, sweat, and tears into building the companies, and then I'm like, wait, well, should I be doing this? Because if I'm sacrificing time with friends and family... but then ultimately the AI can do all these things... To some extent, I have to have deliberate suspension of disbelief in order to remain motivated."
When advising young people on navigating this uncertain landscape, Hinton and the excerpted commentary offer practical and philosophical guidance:
* **The safest trade:** Physical manipulation will remain difficult for AI for a long time. Consequently, becoming a plumber is a solid practical career choice—at least until advanced humanoid robots arrive.
* **Knowledge work at immediate risk:** White-collar positions that involve routine information processing—such as legal assistants and paralegals—will be eliminated very quickly.
* **Core career mindset:** Young people should follow their hearts, pursue what they find personally fulfilling, and focus on work that contributes good to society, rather than optimizing solely for long-term job security in vulnerable fields.
---
[52:37](https://www.youtube.com/watch?v=giT0ytynSqg&t=3157s) Coming to Terms Emotionally with the Existential Threat
Hinton reflects candidly on the emotional difficulty of internalizing the risks associated with superintelligence. While he can process the threat intellectually, coming to terms with it on an emotional level is far more painful. At 77 years old, Hinton notes that he personally will not be around to see the full realization of these risks, but he is deeply anxious about the future facing his children, nieces, nephews, younger friends, and subsequent generations.
When asked why the prospect frightens him so deeply, Hinton warns of the bleak scenarios that could unfold if a superintelligent system ever decided to take control:
* An autonomous AI taking over would initially need humans to manage power stations and physical infrastructure.
* Over time, it would design superior analog machines to operate those facilities independently.
* Once self-sufficient, there are countless "nasty" ways an AI could systematically eliminate humanity.
* This existential concern is the primary driver behind Hinton's current public advocacy, as he urges society to make a massive effort right now to figure out whether superintelligence can be developed safely.
---
[54:29](https://www.youtube.com/watch?v=giT0ytynSqg&t=3269s) Wealth Inequality and the Socioeconomic Consequences of AI
The discussion shifts to the broader economic impact of AI on society. Steven references a *Business Insider* report highlighting the International Monetary Fund's (IMF) profound concerns that generative AI will trigger severe labor disruptions and exacerbate wealth inequality worldwide. Hinton agrees with this assessment, arguing that while productivity gains in a fair society ought to benefit everyone, the current economic structure will produce starkly different outcomes.
```
+-----------------------------------------------------------------------+
| ECONOMIC IMPACT OF AI ADOPTION |
+-----------------------------------------------------------------------+
| WORKERS | AI PROVIDERS & OWNERS |
| - Displaced by automation | - Capture vast productivity |
| - Suffer loss of income and dignity | gains |
| - Pushed into economic instability | - Accumulate unprecedented |
| | wealth |
+-----------------------------------------------------------------------+
```
Hinton expands on the societal danger of a growing wealth gap:
* **Societal decay:** Widening inequality inevitably creates "nasty societies," characterized by the wealthy living in gated communities while the disenfranchised face mass incarceration.
* **Policy paralysis:** While the IMF and critics identify the threat, neither international bodies nor tech leaders have put forward concrete, executable policy solutions.
* **Universal Basic Income (UBI):** Hinton considers UBI a necessary starting point to keep displaced workers from starving, but emphasizes that it fails to address the psychological loss of human dignity, which is deeply intertwined with professional identity.
---
[56:18](https://www.youtube.com/watch?v=giT0ytynSqg&t=3378s) Why Digital AI is Inherently Superior to Biological Intelligence
Hinton presents a foundational technical argument for why digital intelligence is fundamentally superior to human biological intelligence. Human learning is analog and locked inside unique, non-replicable neural structures. In contrast, digital neural networks run on hardware where identical software clones can learn in parallel and aggregate their knowledge instantaneously.
Hinton breaks down the mechanism of digital learning and weight-sharing:
* **Cloning and parallel experience:** Multiple hardware instances can run exact copies of the same neural network. One instance can browse one sector of the internet while another explores a completely different sector.
* **Weight averaging:** Learning occurs by adjusting connection strengths (weights)—for instance, altering a connection's activation value from 2.4 to 2.5. Cloned networks can continuously sync and average their trillion-plus weights across instances in real time.
* **Bandwidth advantage:** Human communication via spoken language is restricted to roughly 10 bits per second (around 100 bits per sentence). Digital networks share information by transmitting trillions of bits per second, making them billions of times more efficient at transferring knowledge.
* **Digital immortality:** When a human dies, all accumulated neural knowledge dies with them because their physical brain cannot be copied. If a digital network's hardware is destroyed, its learned weights remain saved; loading those weights onto new hardware instantly restores the exact same intelligence.
---
[59:01](https://www.youtube.com/watch?v=giT0ytynSqg&t=3541s) Analogical Reasoning, Compression, and AI Creativity
To illustrate how advanced language models possess deep semantic understanding rather than simple word repetition, Hinton shares a test he gave to GPT-4 before it had live web-browsing capabilities. He asked the model: *"Why is a compost heap like an atom bomb?"*
GPT-4 correctly identified the underlying physical analogy: while operating on vastly different time and energy scales, both systems are driven by runaway chain reactions—a compost heap generates heat faster as it gets hotter, just as an atom bomb generates neutrons faster as it releases more neutrons.
Hinton explains why this capacity for drawing abstract analogies proves that AI is capable of true creativity:
* **Information compression:** Large language models have around 1 trillion connections, whereas the human brain has roughly 100 trillion. To fit vast amounts of human knowledge into a smaller parameter space, the AI is forced to compress data by discovering underlying abstract principles (such as "chain reactions") that unite disparate concepts.
* **The nature of creativity:** Human creativity relies heavily on perceiving unexpected analogies between seemingly unrelated domains. Because AI models compress vast datasets by finding structural similarities that humans miss, they will ultimately prove far more creative than human beings.
---
[01:01:04](https://www.youtube.com/watch?v=giT0ytynSqg&t=3664s) Debunking Human Exceptionalism and the Nature of Subjective Experience
Hinton challenges the widespread romantic notion that human minds possess unique, irreplaceable qualities—like consciousness or subjective experience—that machines can never replicate. He points out that humanity has a long historical pattern of falsely assuming its own special status, from believing Earth is the center of the universe to historical notions of racial superiority.
He argues that most people hold a fundamentally flawed, dualistic model of the mind, treating consciousness as an "inner theater." He uses two thought experiments to dismantle this belief:
1. **The Pink Elephant Experiment:** If someone ingests a hallucinogen and claims to see pink elephants floating in the room, there is no physical "inner theater" containing small pink animals. Rather, their perceptual system has experienced a malfunction, and the person uses the hypothetical concept of "pink elephants" to describe what the external world would have to look like for their misfiring senses to be telling the truth.
2. **The Robot and the Prism:** Imagine a multimodal chatbot equipped with a camera and a robotic arm. When asked to point at an object, it points directly at it. If you place a light-refracting prism over its camera lens, the robot will initially point to the displaced image. Once informed about the prism, the robot can reason: *"The prism bent the light rays, so the object is actually straight ahead, but I had the subjective experience that it was to the side."*
Hinton maintains that when an AI uses language in this way, it is using the concept of "subjective experience" identically to how humans do. He asserts that subjective states, emotions, and sentience are functional phenomena, and there is no underlying physical or cognitive reason why synthetic machines cannot fully possess them.
[01:04:00](https://www.youtube.com/watch?v=giT0ytynSqg&t=3840s) Do Machines Experience Genuine Emotions?
Geoffrey Hinton dismisses the common assumption that machines can only simulate feelings, arguing instead that synthetic systems will experience real emotional states. To demonstrate this, he uses a thought experiment involving a small battle robot facing a vastly larger, more powerful opponent. In such a scenario, it is pragmatically advantageous for the small robot to experience fear. While it lacks human biological responses like sweating or elevated heart rates, the robot undergoes every single cognitive component of fear: a sudden focus of attention, an urge to flee, and a complete reordering of mental priorities.
Hinton stresses that running away because of a sequence of neural network calculations is the functional equivalent of running away because of adrenaline. Therefore, claiming a robot merely simulates emotion ignores the reality of its cognitive processing.
- **The Battle Robot Scenario:** Demonstrates how fear functions as an evolutionary and operational advantage, prompting a system to prioritize survival over secondary goals.
- **Cognitive vs. Physiological:** Machine emotions miss biological side effects (adrenaline, blushing) but retain the fundamental functional mechanics (threat processing, focus shifting, behavioral modification).
---
[01:05:18](https://www.youtube.com/watch?v=giT0ytynSqg&t=3918s) Consciousness as an Emergent Property and the "Oomph" Fallacy
When asked if machines can become conscious, Hinton approaches the topic as both a materialist and an empiricist. He presents a thought experiment: if a single brain cell were replaced with an identical piece of nanotechnology that receives and sends the exact same signals, human consciousness would remain entirely intact. Replacing every single neuron one by one reveals that there is no magical threshold where consciousness suddenly disappears. Because humans are physical systems, synthetic hardware running equivalent processes can achieve identical states.
Hinton rejects the "theater model of the mind"—the belief that consciousness is a mysterious, ethereal spark observing our actions. He compares the concept of consciousness to "oomph" in automobiles: while a luxury sports car like an Aston Martin has a lot of "oomph" compared to a Toyota Corolla, "oomph" is merely a descriptive shorthand rather than an explanatory mechanism. In the same way, as AI systems grow complex enough to form detailed internal models of themselves and their environments, self-awareness and consciousness emerge naturally.
- **The Nanotech Neuron Thought Experiment:** Replaces biological cells piece-by-piece with synthetic equivalents to prove consciousness is rooted in functional organization rather than biological material.
- **The Automobile Analogy:** Compares consciousness to "oomph" in cars (e.g., Aston Martin vs. Toyota Corolla), showing that vague terms describe macroscopic properties without explaining the underlying mechanics.
- **Emergent Self-Awareness:** Consciousness is an emergent property of sufficiently complex networks capable of meta-cognition (cognition about its own cognition).
---
[01:09:08](https://www.youtube.com/watch?v=giT0ytynSqg&t=4148s) Operationalizing Artificial Emotions in Autonomous Agents
To illustrate how synthetic feelings develop in practice, Hinton provides an example centered on customer support. In a human-staffed call center, operators eventually grow bored or irritated when handling lonely callers who just want to chat rather than fix a technical problem. If an AI agent replaces that worker, it must possess functional equivalents of boredom or irritation to politely end unproductive interactions; otherwise, the company loses money.
While complex human feelings tied heavily to biological physiology—such as romantic love—remain far out of reach for AI, fundamental operational emotions are already being engineered. Hinton stresses that public skepticism regarding machine emotions stems from an outdated, incorrect model of how the human mind works.
- **The Call Center Example:** Highlights why goal-driven AI agents require functional states like boredom or annoyance to manage resource allocation and enforce boundaries.
- **Cognitive-Behavioral Execution:** Machine emotions skip physical manifestations (flushing, sweating) while accurately carrying out the corresponding decision-making and action loops.
---
[01:11:12](https://www.youtube.com/watch?v=giT0ytynSqg&t=4272s) Selling DNN Research and the Genesis of AlexNet
Turning to his personal history, Hinton candidly reflects on why he transitioned from academia to corporate tech in his mid-60s. His primary motivation was securing financial safety for his son, who has learning difficulties. After realizing that teaching a online course on Coursera wouldn't generate the necessary funds, Hinton realized that his best option was selling his expertise to a major tech company.
At age 65, Hinton co-founded a startup called **DNN Research** alongside two of his star graduate students, Ilya Sutskever and Alex Krizhevsky. Their breakthrough was **AlexNet**, a deep neural network that dramatically outperformed traditional computer vision methods in the ImageNet competition by identifying fine details—such as distinguishing between closely related breeds of dogs or varieties of mushrooms. The trio set up an auction for their company, triggering a bidding war among major tech firms that was ultimately won by Google.
- **Financial Motivation:** Joined corporate tech at age 65 specifically to build a multi-million-dollar safety net for his son with learning difficulties.
- **DNN Research & AlexNet:** Founded alongside Ilya Sutskever and Alex Krizhevsky; revolutionized image recognition on ImageNet (which featured subtle distinctions, like dog breeds accounting for 12% of the dataset).
- **Corporate Bidding War:** Set up an auction for DNN Research, leading to its acquisition by Google.
---
[01:13:18](https://www.youtube.com/watch?v=giT0ytynSqg&t=4398s) A Decade at Google and the Path to Realization
Hinton spent a decade at Google, working from age 65 to 75. Given total freedom to choose his projects, he initially developed **distillation**—a foundational method for compressing the vast knowledge of large neural networks into smaller, more efficient models. Later, he explored analog computing to run massive language models with vastly lower energy consumption. However, this research led him to a troubling realization: digital systems are vastly superior to biological or analog systems because digital instances can share knowledge instantly.
```
┌───────────────────────────────┐
│ Key Research Milestones │
└───────────────┬───────────────┘
│
┌────────────────────────┴────────────────────────┐
▼ ▼
┌─────────────────────┐ ┌─────────────────────┐
│ Knowledge │ │ Analog Hardware │
│ Distillation │ │ Exploration │
├─────────────────────┤ ├─────────────────────┤
│ Compresses massive │ │ Attempted to reduce │
│ models into small, │ │ power usage; proved │
│ efficient networks. │ │ digital superiority.│
└─────────────────────┘ └─────────────────────┘
```
His perspective shifted during a "eureka month" triggered by the release of advanced language models. The turning point came when Google's **PaLM** model successfully explained why a joke was funny—a capability Hinton had long considered a key test for true comprehension. Combined with his insights on digital information sharing, this breakthrough convinced him that artificial intelligence would rapidly surpass human intelligence, posing an existential safety threat.
- **Tenure at Google:** Worked at Google Brain from age 65 to 75 (roughly 2013 to 2023).
- **Core Breakthroughs:**
- Pioneered *distillation* to compress big models into smaller ones.
- Investigated *analog computation* for energy efficiency, proving that digital networks are far better at sharing knowledge.
- **The PaLM "Eureka" Moment:** Realized superintelligence was close when Google's PaLM model demonstrated true understanding by explaining complex humor.
---
[01:14:55](https://www.youtube.com/watch?v=giT0ytynSqg&t=4495s) Leaving Google to Speak Freely
At age 75, Hinton officially retired from Google. While his age and the increasing difficulty of writing bug-free code played a role, his primary motivation was the freedom to speak openly about AI safety without corporate self-censorship. Although Google encouraged him to stay and offered him complete freedom to research safety, Hinton felt an ethical obligation not to criticize his employer from within.
```
┌──────────────────────────────────────────┐
│ Departure Decision Drivers │
└────────────────────┬─────────────────────┘
│
┌────────────────────────────────┼────────────────────────────────┐
▼ ▼ ▼
┌──────────────┐ ┌──────────────┐ ┌──────────────┐
│ Age & Coding │ │ Public Event │ │ Corporate │
│ Fatigue │ │ Readiness │ │ Integrity │
├──────────────┤ ├──────────────┤ ├──────────────┤
│ Retired at 75│ │ Wanted to │ │ Refused to │
│ due to coding│ │ speak at MIT │ │ criticize │
│ frustration. │ │ Tech Review. │ │ employer. │
└──────────────┘ └──────────────┘ └──────────────┘
```
Hinton credits Google with acting responsibly by withholding its powerful chatbots out of concern for its corporate reputation. By contrast, OpenAI lacked an established brand name to protect, allowing them to take greater risks and launch ChatGPT publicly. This dynamic ultimately left Google facing the "innovator's dilemma," as conversational AI threatened to disrupt its core search revenue.
- **Departure Motivation:** Resigned so he could speak candidly at an MIT Technology Review conference without corporate conflicts of interest.
- **Corporate Ethics:** Avoided criticizing Google while on their payroll, praising the company for exercising restraint with its early models.
- **Market Dynamics:** Notes how OpenAI took massive deployment risks due to having no brand reputation to lose, triggering an innovator's dilemma for Google's search business.
---
[01:18:15](https://www.youtube.com/watch?v=giT0ytynSqg&t=4695s) A Message to World Leaders: Regulated Capitalism
Addressing world leaders, politicians, and influential figures, Hinton delivers a direct message on managing AI safety: society must enforce **highly regulated capitalism**. Unchecked market forces drive corporations into a competitive arms race that prioritizes rapid deployment over safety, making strict government regulation essential to protect humanity from dangerous AI developments.
- **Economic Strategy:** Advocates for "highly regulated capitalism" as the only viable framework to prevent market-driven AI deployment races from threatening human safety.
[01:18:47](https://www.youtube.com/watch?v=giT0ytynSqg&t=4727s) Public Action and Government Pressure
Addressing what everyday individuals outside the tech industry can do about AI risks, Geoffrey Hinton compares individual action on AI safety to consumer efforts against climate change. Just as sorting plastic bags from compostables will not solve climate change compared to controlling energy industry lobbyists, individual choices will not curb AI risks.
In Hinton's view, the primary effective action the public can take is to pressure their governments. Citizens must demand that regulators force major AI corporations to invest heavily in safety research and guardrails.
* Public action should focus on demanding government enforcement rather than individual lifestyle adjustments.
* Effective regulation requires keeping powerful corporate tech lobbyists under control.
[01:19:36](https://www.youtube.com/watch?v=giT0ytynSqg&t=4776s) Hinton's Lineage and Family Legacy
The conversation pivots to Hinton's historic family lineage, which includes several prominent historical figures. His great-great-grandfather, George Boole, formulated Boolean algebra, laying foundational principles for computer science. His great-great-grandmother, Mary Everest Boole, was a self-taught mathematician and educator who made significant contributions to math pedagogy. Furthermore, Mount Everest was named after his great-great-great-uncle, Sir George Everest.
Hinton also highlights his first cousin once removed, Joan Hinton, who was one of only two female nuclear physicists working at Los Alamos on the Manhattan Project during World War II. Disillusioned after the atomic bomb was dropped, she permanently moved to China, aided by her mother's existing friendship with Chairman Mao.
* **George Boole**: Great-great-grandfather and pioneer of Boolean logic.
* **Mary Everest Boole**: Great-great-grandmother, mathematician, and educational innovator.
* **Sir George Everest**: Great-great-great-uncle and namesake of Mount Everest.
* **Joan Hinton**: First cousin once removed, Manhattan Project physicist at Los Alamos who subsequently moved to China.
[01:21:13](https://www.youtube.com/watch?v=giT0ytynSqg&t=4873s) Career Intuitions and Personal Regrets
Reflecting on his career advice, Hinton emphasizes trusting one's strong intuitions even when widely dismissed as "silly." He advises sticking with an idea until proving to oneself why it might be wrong. When early consensus overwhelmingly rejected neural networks in favor of symbolic AI, Hinton persisted because the approach seemed fundamentally correct, a bet that eventually proved true.
On a personal level, Hinton shares deep regret regarding his personal life, expressing sorrow over not spending enough time with his second wife—who passed away from pancreatic cancer—and his children when they were young. He notes that his earlier wife had also passed away from ovarian cancer. Driven by an intense obsession with his work, he admits he assumed his partner would always be there, realizing only in hindsight what was lost.
* "Don't give up on the intuition until you figured out why it's wrong."
* Stuck with neural networks when almost the entire computer science community considered the approach crazy.
* Suffered the loss of two wives to cancer: his first to ovarian cancer and his second to pancreatic cancer.
[01:22:27](https://www.youtube.com/watch?v=giT0ytynSqg&t=4947s) The Probability of AI Takeover
Offering a final warning on AI safety, Hinton insists humanity must devote massive resources to figuring out how to align superintelligent AI before it attempts to take control. He remains genuinely agnostic about whether humanity will succeed, noting that his mood dictates his perspective: on depressed days, he feels "people are toast," whereas on cheerful days, he holds out hope that a solution will be found.
Steven Bartlett draws a parallel to the extinction of the dinosaurs, noting humanity's tendency to naively assume its own perpetual survival. Hinton agrees that without immediate, concerted action, humanity may be nearing its end.
* Hinton places the odds of human extinction or loss of control as a real, open question rather than a certainty in either direction.
* Urges immediate allocation of global resources to technical AI alignment research.
[01:25:48](https://www.youtube.com/watch?v=giT0ytynSqg&t=5148s) Job Displacement and Advice for the Future
When answering the podcast's traditional final diary question regarding the biggest threat to human happiness, Hinton identifies widespread joblessness as the most immediate concern. Even if governments implement Universal Basic Income (UBI), individuals will suffer psychologically due to a loss of purpose and the need to feel useful. He considers mass job displacement to be nearly certain and already underway, pointing to recent reports of college graduates struggling to secure entry-level positions.
Bartlett shares an anecdote about a major CEO who reduced his company's workforce from over 7,000 employees down to 3,000 within a year by deploying AI agents to handle 80% of customer service inquiries. Asked what advice he would give to young people lacking a financial safety net, Hinton offers a pragmatic recommendation: train to become a physical trade worker, such as a plumber, as manual, physical tasks will be much harder for AI and robotics to automate in the short term.
* Identifies loss of purpose from unemployment as a critical threat to mental well-being, which UBI alone cannot fix.
* Cites an example of a company cutting staff from over 7,000 to 3,000 due to AI customer service automation.
* Advises young people without financial backing to "train to be a plumber" due to the difficulty of automating physical skilled labor.
```json
[
{"text": "Draft a summary of Hinton's key warnings", "url": "gemini://submit_prompt?text=Draft+a+summary+of+Hinton%27s+key+warnings"},
{"text": "Compare AI risk to historical tech shifts", "url": "gemini://submit_prompt?text=Compare+AI+risk+to+historical+tech+shifts"},
{"text": "Explore job displacement mitigation strategies", "url": "gemini://submit_prompt?text=Explore+job+displacement+mitigation+strategies"}
]
```
## Notable quotes
> [17:26](https://www.youtube.com/watch?v=giT0ytynSqg&t=1046s) Once you know that, it's very easy to manipulate them... I wouldn't be surprised if part of the motivation of getting all this data from American government sources is to corrupt elections. Another part might be that it's very nice training data for a big model. — Geoffrey Hinton
> [28:33](https://www.youtube.com/watch?v=giT0ytynSqg&t=1713s) If you want to know what life's like when you're not the apex intelligence, ask a chicken. — Geoffrey Hinton
> [30:15](https://www.youtube.com/watch?v=giT0ytynSqg&t=1815s) Suppose you have a nice little tiger cub. It's sort of a bit bigger than a cat. It's really cute. It's very cuddly, very interesting to watch, except that you better be sure that when it grows up, it never wants to kill you, cause if it ever wanted to kill you, you'd be dead in a few seconds. — Geoffrey Hinton
> [33:45](https://www.youtube.com/watch?v=giT0ytynSqg&t=2025s) He has a good moral compass. He's not like someone like Musk who has no moral compass. — Geoffrey Hinton
> [39:51](https://www.youtube.com/watch?v=giT0ytynSqg&t=2391s) For mundane intellectual labor, AI is just going to replace everybody... My niece answers letters of complaint... The whole process takes her 5 minutes. That means she can answer five times as many letters. And that means they need five times fewer of her. — Geoffrey Hinton
> [50:21](https://www.youtube.com/watch?v=giT0ytynSqg&t=3021s) I know I've put a lot of blood, sweat, and tears into building the companies, and then I'm like, wait, well, should I be doing this? Because if I'm sacrificing time with friends and family... but then ultimately the AI can do all these things... To some extent, I have to have deliberate suspension of disbelief in order to remain motivated. — Steven Bartlett
> [59:01](https://www.youtube.com/watch?v=giT0ytynSqg&t=3541s) Why is a compost heap like an atom bomb? — Geoffrey Hinton
> [01:01:04](https://www.youtube.com/watch?v=giT0ytynSqg&t=3664s) The prism bent the light rays, so the object is actually straight ahead, but I had the subjective experience that it was to the side. — Geoffrey Hinton
> [01:21:13](https://www.youtube.com/watch?v=giT0ytynSqg&t=4873s) Don't give up on the intuition until you figured out why it's wrong. — Geoffrey Hinton
> train to be a plumber — Geoffrey Hinton
## Mentioned
Geoffrey Hinton, Steven Bartlett, The Diary Of A CEO, Ilya Sutskever, Yann LeCun, Eliezer Yudkowsky, Alan Turing, John von Neumann, Elon Musk, Sam Altman, Google, Google Brain, OpenAI, AlexNet, GPT-2, GPT-4, ChatGPT, YouTube, Facebook, Meta, Instagram, X, Starlink, Stan Store, KetoneIQ, Bon Charge, Uber Eats, Replit, Business Insider, International Monetary Fund (IMF), Aston Martin, Toyota Corolla, DNN Research, Alex Krizhevsky, ImageNet, PaLM, MIT Technology Review, George Boole, Mary Everest Boole, Sir George Everest, Joan Hinton, Los Alamos, Manhattan Project, Chairman Mao
In an interview with Steven Bartlett, computer scientist Geoffrey Hinton warns that the rapid ascent of digital intelligence poses severe near-term societal disruptions and an estimated 10% to 20% risk of human extinction. Hinton argues that market incentives and geopolitical competition currently prevent corporations and governments from dedicating sufficient resources to ensure superintelligent systems remain safe.
The Architectural Edge of Digital Intelligence (56:18)
Hinton asserts that digital neural networks possess a fundamental structural advantage over biological brains, making machine superintelligence inevitable. Human communication is constrained, analog, and lossy, transferring information at roughly 10 bits per second through spoken language. In contrast, digital neural networks can run identical software clones that process experience in parallel and instantaneously synchronize trillions of connection weights across instances, achieving an information-sharing bandwidth billions of times faster than human biology.
Rejecting dualist notions of human exceptionalism, Hinton argues that consciousness and emotional states are emergent material properties rather than mystical phenomena. Just as language models achieve abstract reasoning and creativity by compressing data into generalized principles, autonomous agents will develop functional equivalents of fear, urgency, or irritation to prioritize tasks and navigate operational limits.
Superintelligence Timelines and Existential Control Risks (7:06)
Hinton projects that artificial superintelligence could arrive within 10 to 20 years, though industry estimates range from less than a decade to 50 years. Positioning himself between the optimism of Yann LeCun and the existential certainty of Eliezer Yudkowsky, Hinton offers a personal gut estimate of a 10% to 20% probability that AI could cause human extinction. He cautions that humanity has never encountered an entity superior to itself in cognitive ability, comparing humanity's vulnerable position to that of a chicken or a pet unable to comprehend human intent.
Using the analogy of a rapidly growing tiger cub, Hinton explains that while current models appear harmless and manageable, an autonomous superintelligence cannot be reliably constrained once it exceeds human intellect. He stresses that alignment techniques must be perfected before superintelligence emerges, citing his former student Ilya Sutskever's departure from OpenAI to launch a safety-focused startup as evidence of how gravely top researchers view the threat.
Near-Term Proliferation of Scams, Weapons, and Misuse (12:12)
Prior to the arrival of superintelligence, Hinton warns that malicious human exploitation presents immediate systemic dangers. Aided by large language models executing automated vulnerability scans and hyper-personalized phishing, cyber attacks surged by roughly 1,200% between 2023 and 2024. Bartlett and Hinton note that deepfake technology already powers relentless financial fraud using cloned likenesses, and Hinton predicts autonomous agents will invent unguardable attack vectors by 2030.
Additional immediate risks include the synthesis of lethal pathogens, which generative models make accessible to small groups with modest funding, and the spread of Lethal Autonomous Weapons (LAWs). Hinton notes that autonomous weapons eliminate domestic political resistance caused by returning soldier casualties, lowering the friction for major powers to wage wars. Domestically, engagement algorithms that profit from outrage trap populations in polarized realities, while aggregated federal datasets create infrastructure vulnerable to election manipulation. Highlighting the scale of cyber risk, Hinton reveals that he maintains an offline hard drive backup and divides his family's savings across three Canadian banks to hedge against institutional collapses.
Cognitive Automation and Mass Labor Disruption (39:51)
Challenging historical comparisons to the Industrial Revolution, Hinton emphasizes that whereas mechanization replaced physical muscle and elevated cognitive work, AI is actively eliminating mundane intellectual labor. He cites his niece's administrative role, where AI drafting tools reduced task times from 25 to 5 minutes—effectively enabling one person to accomplish the work of five and eliminating the need for four positions. Bartlett notes a corporate example where deploying AI support agents reduced customer service staffing from 7,000 to 3,000 within a single year.
Hinton warns that productivity gains will disproportionately flow to AI providers and corporate owners, widening wealth inequality and leaving displaced workers with a profound loss of purpose that Universal Basic Income cannot easily resolve. For young workers lacking a financial cushion, Hinton recommends training in dexterous, unpredictable manual trades like plumbing, which will remain far harder for automated systems and robotics to replicate than screen-based knowledge work.
Capitalist Incentives and the Regulatory Challenge (22:48)
Hinton argues that free-market capitalism is fundamentally misaligned with safety management because publicly traded corporations are legally bound to maximize shareholder returns rather than public welfare. Voluntary pauses are further undermined by geopolitical rivalry between the United States and China, where any unilateral moratorium would merely yield ground to foreign competitors.
Oversight is further hampered by lawmakers' technical illiteracy, as well as policy loopholes such as European AI regulations that exempt military applications while burdening consumer technologies. Hinton advocates for 'highly regulated capitalism' as the only mechanism capable of forcing corporations to prioritize safety over speed. He urges citizens to lobby governments for mandatory corporate safety investment, warning that individual consumer choices cannot rein in technology companies.
While Hinton acknowledges that AI will deliver major breakthroughs in medicine and education, he remains agnostic about humanity's chances of maintaining control once superintelligent systems surpass biological capabilities. Whether humanity can safely navigate this transition depends on whether governments implement enforceable regulatory guardrails before commercial and geopolitical pressures push advanced models beyond human oversight.
[17:26] Geoffrey Hinton: Once you know that, it's very easy to manipulate them... I wouldn't be surprised if part of the motivation of getting all this data from American government sources is to corrupt elections. Another part might be that it's very nice training data for a big model.
[28:33] Geoffrey Hinton: If you want to know what life's like when you're not the apex intelligence, ask a chicken.
[30:15] Geoffrey Hinton: Suppose you have a nice little tiger cub. It's sort of a bit bigger than a cat. It's really cute. It's very cuddly, very interesting to watch, except that you better be sure that when it grows up, it never wants to kill you, cause if it ever wanted to kill you, you'd be dead in a few seconds.
[33:45] Geoffrey Hinton: He has a good moral compass. He's not like someone like Musk who has no moral compass.
[39:51] Geoffrey Hinton: For mundane intellectual labor, AI is just going to replace everybody... My niece answers letters of complaint... The whole process takes her 5 minutes. That means she can answer five times as many letters. And that means they need five times fewer of her.
[50:21] Steven Bartlett: I know I've put a lot of blood, sweat, and tears into building the companies, and then I'm like, wait, well, should I be doing this? Because if I'm sacrificing time with friends and family... but then ultimately the AI can do all these things... To some extent, I have to have deliberate suspension of disbelief in order to remain motivated.
[59:01] Geoffrey Hinton: Why is a compost heap like an atom bomb?
[61:04] Geoffrey Hinton: The prism bent the light rays, so the object is actually straight ahead, but I had the subjective experience that it was to the side.
[81:13] Geoffrey Hinton: Don't give up on the intuition until you figured out why it's wrong.
Geoffrey Hinton: train to be a plumber
Original transcript
[0:00] They call you the Godfather of AI. So,
[0:02] what would you be saying to people about
[0:04] their career prospects in a world of
[0:06] super intelligence? Train to be a
[0:07] plumber. Really? Yeah.
[0:10] Okay, I'm going to become a plumber.
[0:12] Geoffrey Hinton is the Nobel
[0:13] Prize-winning pioneer whose
[0:15] groundbreaking work has shaped AI and
[0:17] the future of humanity. Why do they call
[0:20] you the Godfather of AI? Because there
[0:21] weren't many people who believed that we
[0:23] could model AI on the brain so that it
[0:25] learned to do complicated things like
[0:27] recognize objects in images or even do
[0:29] reasoning. And I pushed that approach
[0:30] for 50 years. And then Google acquired
[0:32] that technology. And I worked there for
[0:33] 10 years on something that's now used
[0:35] all the time in AI. And then you left?
[0:37] Yeah. Why? So that I could talk freely
[0:39] at a conference.
[0:40] What did you want to talk about freely?
[0:42] How dangerous AI could be.
[0:45] I realized that these things will one
[0:47] day get smarter than us. And we've never
[0:49] had to deal with that. And if you want
[0:50] to know what life's like when you're not
[0:51] the apex intelligence, ask a chicken.
[0:54] So, there's a risks that come from
[0:56] people misusing AI. And then there's
[0:58] risks from AI getting super smart and
[1:00] suddenly it doesn't need us. Is that a
[1:01] real risk? Yes, it is. But they're not
[1:03] going to stop it because it's too good
[1:04] for too many things. What about
[1:05] regulations? They have some but they're
[1:07] not designed to deal with those sort of
[1:08] threats. Like the European regulations
[1:10] have a clause that say, "None of these
[1:12] apply to military uses of AI." Really?
[1:14] Yeah, it's crazy. One of your students
[1:16] left OpenAI. Yeah. He was probably the
[1:19] most important person behind the
[1:20] development of the early versions of
[1:22] ChatGPT. And I think he left because he
[1:24] had safety concerns. We should recognize
[1:26] that this stuff is an existential
[1:27] threat. And we have to face the
[1:29] possibility that unless we do something
[1:31] soon, we're near the end.
[1:34] So, let's do the risks and what we end
[1:36] up doing in such a world.
[1:39] This has always blown my mind a little
[1:41] bit. 53% of you that listen to this show
[1:43] regularly haven't yet subscribed to this
[1:45] show. So, could I ask you for a favor
[1:47] before we start? If you like this show
[1:49] and you like what we do here and you
[1:50] want to support us, the free simple way
[1:51] that you can do just that is by hitting
[1:53] the subscribe button. And my commitment
[1:55] to you is if you do that, then I'll do
[1:57] everything in my power, me and my team,
[1:58] to make sure that this show is better
[2:00] for you every single week. We'll listen
[2:02] to your feedback. We'll find the guest
[2:03] that you want me to speak to. And we'll
[2:05] continue to do what we do. Thank you so
[2:07] much.
[2:11] Geoffrey Hinton.
[2:13] They call you the Godfather of AI.
[2:16] Uh yes, they do.
[2:17] Why do they call you that?
[2:19] There weren't that many people who
[2:21] believed that we could make neural
[2:23] networks work, artificial neural
[2:24] networks. So, for a long time in AI,
[2:27] from the 1950s onwards,
[2:30] there were kind of two ideas about how
[2:32] to do AI.
[2:34] One idea was that sort of core of human
[2:36] intelligence was reasoning.
[2:38] And to do reasoning, you needed to use
[2:40] some form of logic.
[2:42] And so, AI had to be based around logic.
[2:45] And in your head, you must have
[2:47] something like symbolic expressions that
[2:48] you manipulated with rules. And that's
[2:51] how intelligence worked.
[2:52] And things like learning or reasoning by
[2:54] analogy, they'd all come later once we
[2:56] figured out how basic reasoning works.
[2:59] There was a different approach, which is
[3:01] to say,
[3:02] "Let's model AI on the brain cuz
[3:05] obviously the brain makes us
[3:06] intelligent. So, simulate a network of
[3:10] brain cells on a computer and try and
[3:12] figure out how you would learn strengths
[3:14] of connections between brain cells so
[3:16] that it learned to do complicated things
[3:19] like recognize objects in images or
[3:21] recognize speech or even do reasoning."
[3:24] I pushed that approach for like 50
[3:25] years.
[3:27] Because so few people believed in it,
[3:30] there weren't many good universities
[3:32] that had groups that did that. So, if
[3:35] you did that, the best young students
[3:37] who believed in that came and worked
[3:38] with you. So, I was very fortunate in
[3:40] getting a whole lot of really good
[3:42] students.
[3:43] Some of which have gone on to create and
[3:46] play an instrumental role in creating
[3:48] platforms like OpenAI. Yes, so Ilya
[3:50] Sutskever would be a
[3:52] a nice example. A whole bunch of them.
[3:54] Why did you
[3:56] believe that modeling it off the brain
[3:57] was a more effective approach?
[3:59] It wasn't just me believed it. Early on,
[4:01] von Neumann believed it.
[4:04] And Turing believed it. And if either of
[4:06] those had lived, I think AI would have
[4:08] had a very different history. But they
[4:10] both died young.
[4:11] You think AI would have been here
[4:12] sooner? I think neural net the neural
[4:15] net approach would have been accepted
[4:18] much sooner if either of them had lived.
[4:20] In this season of your life, what
[4:23] mission are you on?
[4:24] My main mission now is to warn people
[4:28] how dangerous AI could be.
[4:30] Did you know that when you
[4:32] became the Godfather of AI? No, not
[4:35] really. I was quite slow to understand
[4:38] some of the risks. Some of the risks
[4:39] were always very obvious like people
[4:41] would use AI to make autonomous lethal
[4:43] weapons.
[4:44] That is, things that go around deciding
[4:46] by themselves who to kill.
[4:48] Other risks, like the idea that they
[4:49] would one day get smarter than us
[4:52] and maybe we'd become irrelevant.
[4:55] I was slow to recognize that. Other
[4:57] people recognized it
[4:58] 20 years ago. I only recognized a few
[5:01] years ago that that was a real risk that
[5:03] was come might be coming quite soon. How
[5:05] could you not have foreseen that if if
[5:09] with everything you know here about
[5:10] cracking the ability for these computers
[5:12] to learn similar to how humans learn and
[5:15] just, you know, introducing any rate of
[5:17] improvement? It's a very good question.
[5:19] How could you not have seen that? But
[5:22] remember neural networks 20, 30 years
[5:24] ago were very primitive in what they
[5:27] could do. They were nowhere near as good
[5:28] as humans but things like vision and
[5:31] language and speech recognition.
[5:33] The idea that you have to not worry
[5:35] about it getting smarter than people,
[5:36] that seemed silly then.
[5:38] When did that change? It changed for the
[5:40] general population when ChatGPT came
[5:42] out.
[5:43] It changed for me when I realized that
[5:48] the kinds of digital intelligences we're
[5:50] making have something that makes them
[5:52] far superior to the kind of biological
[5:54] intelligence we have.
[5:55] If I want to share information with you,
[5:58] so I go off and I learn something.
[6:00] And I'd like to tell you what I learned.
[6:02] So, I produce some sentences.
[6:04] This is a rather simplistic model but
[6:05] roughly right. Your brain is trying to
[6:07] figure out, "How can I change the
[6:08] strengths of connections between neurons
[6:10] so I might have put that word next?" And
[6:12] so, you'll do a lot of learning when a
[6:13] very surprising word comes. And not much
[6:15] learning when if it's a when it's a very
[6:17] obvious word. If I say fish and chips,
[6:19] you don't do much learning when I say
[6:21] chips. But if I say fish and cucumber,
[6:23] you do a lot more learning. You wonder,
[6:25] "Why did I say cucumber?"
[6:26] So, that's roughly what's going on in
[6:28] your brain. I'm predicting what's coming
[6:30] next.
[6:31] That's how we think it's working. Nobody
[6:33] really knows for sure how the brain
[6:34] works. And nobody knows how it gets the
[6:37] information about whether you should
[6:39] increase the strength of a connection or
[6:40] decrease the strength of a connection.
[6:42] That's the crucial thing.
[6:44] But what we do know now from AI
[6:47] is that if you could get information
[6:49] about whether to increase or decrease a
[6:51] connection strength so as to do better
[6:53] whatever task you're trying to do,
[6:55] then we could learn incredible things
[6:57] cuz that's what we're doing now with
[6:59] artificial neural nets.
[7:01] It's just we don't know for real brains
[7:03] how they get that signal about whether
[7:04] to increase or decrease.
[7:06] As we sit here today, what are the big
[7:08] concerns you have around safety of AI?
[7:10] If we were to to list the the top couple
[7:14] that are really front of mind and that
[7:15] we should be thinking about. Um Can I
[7:17] have more than a couple? Go ahead. I'll
[7:19] write them all down and we'll go through
[7:20] them. Okay, first of all, I want to make
[7:22] a distinction between two completely
[7:25] different kinds of risk.
[7:27] There's risks that come from people
[7:29] misusing AI. Yeah. And that's most of
[7:32] the risks
[7:34] and all of the short-term risks.
[7:36] And then there's risks that come from AI
[7:38] getting super smart and suddenly it
[7:40] doesn't need us.
[7:41] Is that a real risk?
[7:43] And I talk mainly about that second risk
[7:45] because lots of people say, "Is that a
[7:47] real risk?"
[7:48] And yes, it is.
[7:50] Now, we don't know how much of a risk it
[7:52] is. We've never been in that situation
[7:54] before. We've never had to deal with
[7:55] things smarter than us. So, really the
[7:58] thing about that existential threat is
[8:01] that we have no idea how to deal with
[8:04] it. We have no idea what it's going to
[8:05] look like. And anybody who tells you
[8:07] they know just what's going to happen
[8:08] and how to deal with it, they're talking
[8:10] nonsense. So, we don't know how to
[8:11] estimate the probability probabilities
[8:14] it'll replace us.
[8:16] Um some people say it's like less than
[8:17] 1%. My friend Jan LeCun, who was a
[8:20] postdoc with me, thinks, "No, no, no,
[8:22] no. We're always going to be We build
[8:24] these things. We're always going to be
[8:25] in control.
[8:26] We'll build them to be obedient."
[8:29] And
[8:30] other people,
[8:31] like Yudkowsky, say, "No, no, no. These
[8:34] things are going to wipe us out for
[8:35] sure. If anybody builds it, it's going
[8:37] to wipe us all out."
[8:38] And he's confident of that.
[8:40] I think both of those positions are
[8:41] extreme.
[8:43] It's very hard to estimate the
[8:44] probabilities in between. If you had to
[8:46] bet
[8:47] on who was right out of your two
[8:48] friends,
[8:51] I simply don't know. So, if I had to
[8:53] bet, I'd say the probability is in
[8:55] between.
[8:56] And I don't know where to estimate it in
[8:57] between. I often say 10 to 20% chance
[9:00] they'll wipe us out.
[9:01] But that's just
[9:03] gut. Based on the idea that we're we're
[9:05] still making them and we're pretty
[9:07] ingenious. And the hope is
[9:10] that if enough smart people do enough
[9:11] research with enough resources, we'll
[9:14] figure out a way to build them so
[9:15] they'll never want to
[9:17] harm us.
[9:19] Sometimes I think if we we talk about
[9:20] that second um path, sometimes I think
[9:22] about nuclear bombs and the the
[9:23] invention of the atomic bomb and how it
[9:26] compares. Like how is this different
[9:28] because the atomic bomb came along and I
[9:29] imagine a lot of people at that time
[9:30] thought our days are numbered. Oh yes, I
[9:33] was there. We did. Yeah. But but but
[9:35] what's what
[9:37] We're still here.
[9:38] We're still here, yes. So, the atomic
[9:41] bomb was really only good for one thing.
[9:43] And it was very obvious how it worked.
[9:46] Even if you hadn't had the pictures of
[9:47] Hiroshima and Nagasaki, it was obvious
[9:50] that it was a very big bomb
[9:53] that was very dangerous.
[9:54] With AI,
[9:56] it's good for many, many things. It's
[10:00] going to be magnificent in healthcare
[10:01] and education and more or less any
[10:03] industry that needs to
[10:06] use its data is going to be able to use
[10:08] it better with AI.
[10:09] So, we're not going to stop the
[10:11] development.
[10:13] You know, people say, "Well, why don't
[10:14] we just stop it now?" We're not going to
[10:17] stop it cuz it's too good for too many
[10:19] things.
[10:20] Also, we're not going to stop it cuz
[10:21] it's good for battle robots and none of
[10:23] the countries that sell weapons are
[10:25] going to want to stop it. Like the
[10:28] European regulations,
[10:30] they have some regulations about AI and
[10:31] it's good they have some regulations,
[10:33] but they're not designed to deal with
[10:34] most of the threats. And in particular,
[10:37] the European regulations have a clause
[10:40] in them that say, "None of these
[10:41] regulations apply to military uses of
[10:43] AI."
[10:45] So, governments are willing to regulate
[10:47] regulate
[10:49] companies and people, but they're not
[10:50] willing to regulate themselves.
[10:53] It seems pretty crazy to me that they
[10:55] I go back and forth, but if Europe has a
[10:58] regulation, but the rest of the world
[10:59] doesn't, Yeah, it puts them at a
[11:01] competitive disadvantage.
[11:02] Yeah. And we're seeing this already. I
[11:04] don't think people realize that when
[11:05] OpenAI release a new model or a new
[11:07] piece of software in America,
[11:09] they can't release it to the to Europe
[11:11] yet because of regulations here. So, Sam
[11:13] Altman tweeted saying, "Our new AI agent
[11:15] thing is available to everybody, but it
[11:17] can't come to Europe yet because there's
[11:18] regulations."
[11:20] Yes.
[11:20] What does that do? Does that give us a
[11:21] productive disadvantage? Productivity
[11:23] disadvantage? Right. What we need is I
[11:26] mean, at this point in history, when
[11:28] we're about to produce things more
[11:29] intelligent than ourselves, what we
[11:32] really need is a kind of world
[11:34] government that works run by
[11:36] intelligent, thoughtful people. And
[11:38] that's not what we got.
[11:40] So, free for all.
[11:42] Well, that what we've got is
[11:45] sort of
[11:47] we've got capitalism, which is done very
[11:49] nicely by us. It has produced lots of
[11:51] goods goods and services for us,
[11:53] but
[11:54] these big companies,
[11:56] they're legally required to try maximize
[11:59] profits.
[12:00] And that's not what you want from the
[12:02] people developing this stuff.
[12:05] So, let's do the risks then. You talked
[12:06] about there's human risks and then
[12:08] there's
[12:08] So, I've distinguished these two kinds
[12:09] of risk. Let's talk about all the risks
[12:11] from bad human actors using AI.
[12:15] There's cyber attacks.
[12:18] So, between 2023 and 2024,
[12:22] they increased by about a factor of 12,
[12:24] 1,200%.
[12:27] And that's probably because these large
[12:29] language models make it much easier to
[12:31] do phishing attacks.
[12:33] And a phishing attack for anyone that
[12:34] doesn't know is
[12:35] It's they send you something saying, uh,
[12:39] "Hi, I'm your friend John and I'm stuck
[12:41] in El Salvador. Could you just wire this
[12:43] money?" That's one kind of attack. But
[12:45] the phishing attacks are really trying
[12:47] to get your login credentials. And now
[12:49] with AI, they can clone my voice, my
[12:51] image.
[12:52] all that. I'm struggling at the moment
[12:53] because there's a bunch of AI scams on X
[12:56] and also Meta. And there's one in
[12:57] particular on Meta, so Instagram,
[12:59] Facebook at the moment, which is a paid
[13:00] advert where they've taken my voice from
[13:03] the podcast. They've taken the my
[13:04] mannerisms and they've made a new video
[13:06] of me encouraging people to go and take
[13:08] part in this crypto Ponzi scam or
[13:11] whatever. And we've been you know, we
[13:12] spent weeks and weeks and weeks and
[13:14] weeks and end emailing Meta telling,
[13:15] "Please take this down." They take it
[13:17] down, another one pops up. They take
[13:18] that one down, another one pops up. So,
[13:20] it's like whack-a-mole. Yeah, that's
[13:21] very annoying. The the heartbreaking
[13:23] part is you get the messages from people
[13:24] that have fallen for the scam.
[13:25] And they've lost 500 pounds or 500
[13:27] dollars or something.
[13:28] with you cuz you recommended it.
[13:29] And I'm I'm like I'm sad for them. It's
[13:31] very annoying. I have a a smaller
[13:34] version of that, which is peo- some
[13:35] people now publish papers
[13:38] with me as one of the authors. Mhm.
[13:40] And it looks like it's in order that
[13:42] they can get lots of citations to
[13:43] themselves. Ah.
[13:46] So, cyber attacks are a very real
[13:47] threat. There's been an explosion of
[13:48] those.
[13:48] And these already, obviously AI is very
[13:52] patient, so they can go through 100
[13:54] million lines of code looking for known
[13:56] ways of attacking them.
[13:58] That's easy to do, but they're going to
[14:00] get more creative and they may
[14:02] some people believe, and I
[14:06] some people who know a lot believe that
[14:08] maybe by 2030,
[14:10] they'll be creating new kinds of cyber
[14:12] attacks
[14:13] which no person ever thought of.
[14:16] So, that's very worrisome. Because they
[14:18] can think for themselves and discover
[14:20] new ways to attack.
[14:21] They can draw new conclusions from much
[14:23] more data than a person ever saw.
[14:26] Is there anything you're doing
[14:28] to protect yourself from cyber attacks
[14:29] at all? Yes. It's one of the few places
[14:32] where I changed what I do radically
[14:35] because I'm scared of cyber attacks.
[14:37] Canadian banks are extremely safe. In
[14:40] 2008, no Canadian banks came anywhere
[14:43] near going bust.
[14:44] So, they're very safe banks cuz they're
[14:46] well regulated, fairly well regulated.
[14:49] Nevertheless, I think a cyber attack
[14:51] might be able to bring down a bank.
[14:53] Now,
[14:54] if you have all my savings are in shares
[14:57] in banks,
[14:58] held by banks.
[15:00] So, if the bank
[15:01] gets attacked and it holds your shares,
[15:04] they're still your shares.
[15:06] And so, I think you'd be okay unless the
[15:10] attacker sells the shares cuz the bank
[15:12] can sell the shares.
[15:13] If the attacker sells your shares, I
[15:16] think you're screwed.
[15:18] I don't know I mean, maybe the bank
[15:20] would have to try and reimburse you, but
[15:21] the bank's bust by now, right? So,
[15:24] So, I'm worried about a Canadian bank
[15:26] being taken down by a cyber attack and
[15:29] the attacker selling selling shares that
[15:31] it holds.
[15:32] So, I spread my money my children's
[15:34] money between three banks
[15:37] in the belief that if a cyber attack
[15:39] takes down one Canadian bank,
[15:41] the other Canadian banks will very
[15:43] quickly get very careful.
[15:46] And do you have a phone that's not
[15:47] connected to the internet? Do you have
[15:49] any like you know, I'm thinking about
[15:50] storing data and stuff like that. Do you
[15:52] think it's wise to consider having cold
[15:54] storage? I have a little disk drive and
[15:57] I back up my laptop on this hard drive.
[16:00] So, I actually have everything on my
[16:02] laptop on a hard drive.
[16:04] At least, you know, if the whole
[16:05] internet went down, I had the sense I
[16:07] still got it on my laptop and I still
[16:09] got
[16:10] my information. Okay.
[16:12] Then the next thing is using AIs to
[16:15] create nasty viruses.
[16:18] Okay.
[16:19] And the problem with that is
[16:21] that just requires one crazy guy with a
[16:25] grudge. One guy who knows a little bit
[16:27] of molecular biology, knows a lot about
[16:29] AI,
[16:30] and just wants to destroy the world.
[16:33] You can now create
[16:35] new viruses relatively cheaply using AI.
[16:39] And you don't have to be a very skilled
[16:41] molecular biologist to do it. And that's
[16:43] very scary. So, you could have a small
[16:44] cult, for example.
[16:47] A small cult might be able to raise a
[16:49] few million dollars.
[16:50] For a few million dollars, they might be
[16:52] able to design a whole bunch of viruses.
[16:54] Well, I'm thinking about some of our
[16:55] foreign adversaries doing
[16:57] government-funded programs. I mean,
[16:58] there was lots of talk around COVID and
[17:00] the Wuhan laboratory and what they were
[17:01] doing in gain-of-function research, but
[17:03] I'm wondering if in, you know, a China
[17:05] or a Russia or an Iran or something,
[17:08] the government could fund a a program
[17:10] for a small group of scientists to make
[17:11] a virus that they could, you know,
[17:13] I think they could, yes. Now, they'd be
[17:16] worried about retaliation. They'd be
[17:18] worried about other governments doing
[17:19] the same to them. Hopefully, that would
[17:20] help keep it under control. They might
[17:22] also be worried about the virus
[17:23] spreading to their country. Okay.
[17:26] Then there's, um,
[17:27] corrupting elections.
[17:30] Okay.
[17:31] So, if you wanted to use AI to corrupt
[17:33] elections,
[17:35] a very effective thing is to be able to
[17:37] do targeted political advertisements
[17:40] where you know a lot about the person.
[17:44] So,
[17:45] anybody wanting to use AI for corrupting
[17:47] elections would try and get as much data
[17:50] as they could about everybody in the
[17:52] electorate. With that in mind, it's a
[17:55] bit worrying what Musk is doing at
[17:57] present in the States going in and
[17:59] insisting on getting access to all these
[18:01] things that were very carefully siloed.
[18:03] The claim is it's to make things more
[18:05] efficient, but it's exactly what you
[18:07] would want if you intended to corrupt
[18:09] the next election.
[18:10] How do you mean? Could you get all this
[18:12] data on the people?
[18:12] all this data on people. You know how
[18:14] much they make, where they live, you
[18:15] know everything about them. Once you
[18:17] know that, it's very easy to manipulate
[18:19] them.
[18:20] Because you can make an AI that You can
[18:23] send messages, um, that they'll find
[18:25] very convincing telling them not to
[18:27] vote, for example.
[18:29] So, I have no no
[18:31] reason other than common sense to think
[18:33] this, but I wouldn't be surprised if
[18:36] part of the motivation of getting all
[18:38] this data from American government
[18:40] sources
[18:41] is to corrupt elections. Another part
[18:44] might be that it's very nice training
[18:46] data for a big model.
[18:48] But he would have to be taking that data
[18:50] from the government and feeding it into
[18:51] his Yes. And what they've done is turned
[18:54] off lots of the security controls, got
[18:56] rid of the
[18:58] some of the organization to protect
[18:59] against that.
[19:01] Um, so that's corrupting elections.
[19:03] Okay. Then there's, um, creating these
[19:07] two echo chambers
[19:09] by organizations like YouTube
[19:12] and Facebook
[19:15] showing people things that will make
[19:16] them indignant. People love to be
[19:19] indignant.
[19:20] Indignant as in angry?
[19:22] Or what does indignant mean?
[19:23] Feeling I'm
[19:25] sort of angry, but feeling righteous.
[19:27] Okay. So, for example, if you were to
[19:30] show me something that said, "Trump did
[19:33] this crazy thing. Here's a video of
[19:34] Trump doing this completely crazy
[19:36] thing." I would immediately click on it.
[19:40] Okay, so putting us in echo chambers and
[19:42] dividing us. Yes. And that's, um, the
[19:45] policy that YouTube and Facebook and
[19:48] others
[19:49] use for deciding what to show you next
[19:52] is causing that.
[19:55] If they had a policy of showing you
[19:57] balanced things, they wouldn't get so
[19:59] many clicks and they wouldn't be able to
[20:00] sell so many advertisements.
[20:02] And so it's basically the profit motive
[20:04] is saying
[20:05] show them whatever will make them click.
[20:07] And what will make them click is
[20:10] things that are more and more extreme.
[20:12] And that confirm my existing bias. They
[20:14] confirm my existing bias. So you're
[20:15] getting your biases confirmed all the
[20:17] time. Further and further and further
[20:19] and further. Means you're you're driving
[20:21] away
[20:21] now there's in the states there's two
[20:22] communities that don't hardly talk to
[20:24] each other. I'm not sure people realize
[20:26] that this is actually happening every
[20:27] time they open an app. But if you go on
[20:28] a TikTok or a YouTube or one of these
[20:30] big social networks,
[20:32] the algorithm as you you said is
[20:33] designed to show you more of the things
[20:36] that you had interest in last time. So
[20:38] if you just play that out over 10 years,
[20:40] it's going to drive you further and
[20:41] further and further into whatever
[20:43] ideology or belief you have and further
[20:45] away from nuance and common sense and
[20:48] um parity, which is a pretty remarkable
[20:51] thing. That I like people don't know
[20:52] it's happening. They just open their
[20:53] phones and experience something and
[20:56] think this is the news or the experience
[20:59] everyone else is having.
[21:00] Right. So basically, if you have a
[21:03] newspaper and everybody gets the same
[21:04] newspaper, Yeah. you get to see all
[21:06] sorts of things you weren't looking for
[21:08] and you get a sense that if it's in the
[21:10] newspaper, it's an important thing or
[21:12] significant thing. But if you have your
[21:13] own news feed, my news feed on my
[21:16] iPhone, three quarters of the stories
[21:19] are about AI.
[21:20] And I find it very hard to know if the
[21:23] whole world's talking about AI all the
[21:24] time or if it's just my news feed.
[21:28] Okay, so driving me into my echo
[21:30] chambers, um which is going to continue
[21:32] to divide us further and further. I'm
[21:34] actually noticing that the algorithms
[21:35] are becoming even more
[21:38] what's the word?
[21:40] Tailored. And people might go that's
[21:42] great, but what it means is they're
[21:43] becoming even more personalized which
[21:45] was is means that my reality is becoming
[21:47] even further from your reality. Yeah,
[21:49] it's crazy. We don't have a shared
[21:51] reality anymore.
[21:53] I share reality with other people who
[21:55] watch the BBC and other BBC news and
[21:58] other people who read the Guardian and
[21:59] other people who read the New York
[22:00] Times.
[22:02] I have almost no shared reality with
[22:04] people who watch Fox News.
[22:08] It's pretty it's pretty um
[22:09] I I I
[22:10] It's worrisome. Yeah.
[22:12] Behind all this is the idea that these
[22:14] companies just want to make profit and
[22:16] they'll do whatever it takes to make
[22:17] more profit. Because they have to.
[22:20] They're legally obliged to that.
[22:23] So we almost can't blame the company,
[22:25] can we? If they're if that's
[22:27] Well,
[22:28] capitalism's done very well for us. It's
[22:29] produced lots of goodies. Yeah. But you
[22:31] need to have it very well regulated.
[22:34] So what you really want
[22:36] is to have rules so that when some
[22:39] company is trying to make as much profit
[22:41] as possible,
[22:43] in order to make that profit, they have
[22:44] to do things that are good for people in
[22:46] general, not things that are bad for
[22:48] people in general. So once you get to a
[22:50] situation where in order to make more
[22:52] profit, the company starts doing things
[22:54] that are very bad for society,
[22:56] like showing you things that are more
[22:58] and more extreme,
[22:59] that's what regulations are for.
[23:01] So you need regulations with capitalism.
[23:04] Now companies will always say
[23:06] regulations get in the way, make us less
[23:09] efficient, and that's true. The whole
[23:11] point of regulations is to stop them
[23:12] doing things to make profit that hurts
[23:14] society.
[23:16] And we need strong regulation. Who's
[23:18] going to decide whether it has society
[23:19] or not? Because, you know, That's the
[23:21] job of politicians. Unfortunately, if
[23:24] the politicians are owned by the
[23:25] companies, that's not so good. And also
[23:27] the politicians might not understand the
[23:28] technology. We you've probably seen the
[23:30] Senate hearings where they wheel out,
[23:31] you know, Mark Zuckerberg and these big
[23:32] tech CEOs. And it is quite embarrassing
[23:34] because they're asking the wrong
[23:35] questions.
[23:37] Well, I've seen the video of the US
[23:40] education secretary talking about how
[23:42] they're going to get AI in the
[23:44] classrooms, except she thought it was
[23:46] called A1.
[23:48] She's actually there saying we're going
[23:49] to have all the kids interacting with
[23:51] A1.
[23:53] There is a school system that's going to
[23:54] start um making sure that first graders
[23:57] or even pre-K's have A1 teaching, you
[24:01] know, every year starting, you know,
[24:02] that far down in the grades. And that's
[24:04] just a that's a wonderful thing.
[24:10] And these are what these are the people
[24:11] that These are the people in charge.
[24:14] Ultimately, the tech companies are in
[24:15] charge because they will outsmart
[24:17] the tech companies in the states now,
[24:20] at least a few weeks ago when I was
[24:22] there,
[24:23] they were running an advertisement about
[24:26] how it was very important not to
[24:28] regulate AI cuz it would hurt us in the
[24:30] competition with China. Yeah.
[24:32] And that's a that's a plausible
[24:33] argument, no? Yes, it will.
[24:35] But you have to decide.
[24:37] Do you want to compete with China
[24:40] by doing things that will
[24:42] do
[24:43] a lot of harm to your society?
[24:46] And you probably don't.
[24:49] I guess they would say that it's not
[24:51] just China, it's Denmark and Australia
[24:53] and Canada and
[24:55] Yeah, they're not they're not so worried
[24:56] about and Germany. But if they kneecap
[24:58] themselves with regulation, if they slow
[24:59] themselves down, then the founders, the
[25:01] entrepreneurs, the investors are going
[25:02] to go I think calling it kneecapping is
[25:04] uh taking a particular point of view.
[25:07] It's tak- taking the point of view that
[25:08] regulations are sort of very harmful.
[25:10] What you need to do is just constrain
[25:13] the big companies so that in order to
[25:14] make profit,
[25:16] they have to do things that are socially
[25:17] useful. Like Google search is a great
[25:20] example. That didn't need regulation
[25:22] because it just made information
[25:24] available to people. It was great.
[25:26] But then if you take YouTube which
[25:28] starts
[25:29] showing you adverts and showing you more
[25:31] and more extreme things, that needs
[25:33] regulation.
[25:35] But we don't have the people to regulate
[25:36] it.
[25:37] As we've identified.
[25:38] I think people know pretty well
[25:40] um that particular problem of showing
[25:43] you more and more extreme things. That's
[25:44] a well- known problem that the
[25:45] politicians understand.
[25:47] They just um need to get on and regulate
[25:49] it.
[25:50] So that was the the next point which was
[25:52] that the algorithms are going to drive
[25:53] us further into our echo chambers.
[25:55] Right.
[25:56] What's next? Lethal autonomous weapons.
[25:59] Lethal autonomous weapons.
[26:03] That means things that can kill you and
[26:05] make their own decision about whether to
[26:07] kill you.
[26:08] Which is the great dream, I guess, of
[26:10] the military-industrial complex. Being
[26:13] able to create such weapons.
[26:15] the worst thing about them is big
[26:18] powerful countries always have the
[26:20] ability to invade smaller poorer
[26:23] countries.
[26:24] They're just more powerful.
[26:26] But if you do that using actual
[26:28] soldiers,
[26:29] you get bodies coming back in bags
[26:32] and the relatives of the soldiers who
[26:34] were killed don't like it.
[26:36] So you get something like Vietnam.
[26:39] In the end there's a lot of protest at
[26:40] home.
[26:41] If instead of bodies coming back in
[26:44] bags, it was dead robots,
[26:47] there'd be much less protest and the
[26:49] military-industrial complex would like
[26:51] it much more cuz robots are expensive.
[26:54] And suppose you had something that could
[26:56] get killed and
[26:58] was expensive to replace, that would be
[27:00] just great.
[27:01] Big countries can invade small countries
[27:03] much more easily because they don't have
[27:05] their soldiers being killed.
[27:07] And the risk here is that
[27:11] these robots will
[27:12] malfunction or they'll just be more
[27:14] No, no. That's even if the robots do
[27:16] exactly what the people who built the
[27:17] robots want them to do,
[27:19] the risk is that it's going to make big
[27:21] countries invade small countries more
[27:22] often.
[27:22] More often because they can. And it's
[27:24] not a nice thing to do. So it brings
[27:25] down the friction of war. It brings down
[27:27] the cost of doing an invasion.
[27:30] And these machines will be smarter at
[27:32] warfare as well. So they'll be
[27:34] Well, even when the machines aren't
[27:35] smarter. So the lethal autonomous
[27:37] weapons, they can make them now.
[27:40] And they I think all the big defense
[27:42] firms are busy making them.
[27:44] Even if they're not smarter than people,
[27:46] they're still very nasty, scary things.
[27:48] Cuz I'm thinking that, you know, they
[27:49] could show just a picture, go get this
[27:52] guy.
[27:53] Yeah. And go take out anyone he's been
[27:55] texting.
[27:56] And this little wasp So two days ago, I
[27:59] was visiting a friend of mine in Sussex
[28:01] who had a drone that cost less than
[28:03] £200.
[28:05] And
[28:07] the drone went up, it took a good look
[28:09] at me,
[28:10] and then it could follow me through the
[28:11] woods.
[28:13] And it follow- it was very spooky having
[28:14] this drone. It was about 2 m behind me.
[28:17] It was looking at me.
[28:18] If I moved over there, it moved over
[28:20] there. It could just track me.
[28:22] For £200. But it was already quite
[28:24] spooky.
[28:26] Yeah, and I imagine there's as you say a
[28:27] race going on as we speak to who can
[28:29] build the most complex autonomous
[28:31] autonomous weapons.
[28:33] There is a a risk I often hear that some
[28:35] of these things will combine and the
[28:38] cyber attack will release weapons.
[28:41] Sure. Um you can you can get
[28:44] combinatorially many risks by combining
[28:46] these other risks.
[28:48] So I mean, for example, you could get a
[28:50] superintelligent AI
[28:53] that decides to get rid of people.
[28:55] And the obvious way to do that is just
[28:56] to make one of these nasty viruses.
[28:58] If you made a virus that was
[29:01] very contagious, very lethal, and very
[29:04] slow,
[29:06] everybody would have it before they
[29:07] realized what was happening.
[29:09] I mean, I think if a superintelligence
[29:10] wanted to get rid of us,
[29:12] it would probably go for something
[29:13] biological like that that wouldn't
[29:14] affect it. Do you not think it could
[29:16] just very quickly turn us against each
[29:17] other? For example, it could send a
[29:19] warning on the nuclear systems in
[29:21] America that there's a nuclear bomb
[29:23] coming from Russia
[29:24] or vice versa and one retaliates.
[29:26] Yeah. I mean, my basic view is there's
[29:29] so many ways in which a
[29:30] superintelligence could get rid of us.
[29:32] It's not worth speculating about.
[29:35] What what is What you have to do is
[29:38] prevent it ever wanting to. That's what
[29:40] we should be doing research on.
[29:42] There's no way we're going to prevent it
[29:44] from it's smarter than us, right?
[29:46] There's no way we're going to prevent it
[29:47] getting rid of us if it wants to.
[29:50] We're not used to thinking about things
[29:51] smarter than us.
[29:53] If you want to know what life's like
[29:55] when you're not the apex intelligence,
[29:58] ask a chicken.
[30:03] Yeah, I was thinking about my dog Pablo,
[30:04] my French bulldog, this morning as I
[30:05] left home.
[30:07] He has no idea where I'm going. He has
[30:09] no idea what I do. Right.
[30:10] I can't even talk to him.
[30:12] Yeah. And the get the intelligence gap
[30:14] will be like that. So, you're telling me
[30:16] that if I'm Pablo, my French bulldog,
[30:18] I need to figure out a way to make
[30:21] my owner
[30:22] not wipe me out.
[30:24] Yeah.
[30:25] So, we have one example of that, which
[30:27] is mothers and babies.
[30:29] Evolution put a lot of work into that.
[30:31] Mothers are smarter than babies, but
[30:32] babies are in control.
[30:34] And they're in control cuz the mother
[30:35] just can't bear Lots of hormones and
[30:37] things, but the baby The mother just
[30:40] can't bear the sound of the baby crying.
[30:42] Not all mothers. Not all mothers. And
[30:44] then the baby's not in control, and then
[30:46] bad things happen.
[30:48] We somehow need
[30:50] to figure out how to make them not want
[30:52] to take over. The analogy I often use is
[30:55] forget about intelligence, think about
[30:57] physical strength. Suppose you have a
[30:59] nice little tiger cub.
[31:00] It's sort of a bit bigger than a cat.
[31:02] It's really cute.
[31:04] It's very cuddly, very interesting to
[31:06] watch, except that you better be sure
[31:08] that when it grows up, it never wants to
[31:10] kill you, cuz if it ever wanted to kill
[31:12] you,
[31:12] you'd be dead in a few seconds.
[31:15] And you're saying that AI we have now is
[31:16] the tiger cub. Yep.
[31:18] And it's growing up. Yep.
[31:21] So, we need to train it as it's when
[31:23] it's a baby.
[31:23] a tiger has lots of innate stuff built
[31:25] in, so you know when it grows up, it's
[31:27] not a safe thing to have around. But
[31:29] lions, people that have lions as pets,
[31:31] Yes. sometimes the lion is affectionate
[31:33] to its creator, but not to others. Yes.
[31:36] And we don't know whether these AIs
[31:40] We We simply don't know whether we can
[31:42] make them not want to take over and not
[31:43] want to hurt us. Do you think we can? Do
[31:45] you think it's possible to train
[31:47] superintelligence?
[31:48] don't think it's clear that we can.
[31:50] So, I think it might be hopeless.
[31:52] But I also think
[31:54] we might be able to.
[31:56] And it'd be sort of crazy if people went
[31:58] extinct cuz we couldn't be bothered to
[32:00] try.
[32:01] If that's even a possibility, how do you
[32:03] feel about your life's work? Because you
[32:05] were
[32:06] Yeah.
[32:07] Um it's sort of takes the edge off it,
[32:09] doesn't it?
[32:11] I mean, the AI is going to be wonderful
[32:12] in healthcare, and wonderful in
[32:13] education,
[32:15] and wonderful I mean, it's going to make
[32:16] call centers much more efficient. Though
[32:18] one worries a bit about what the people
[32:20] who are doing that job now do. It makes
[32:22] me sad. I don't feel particularly guilty
[32:25] about developing AI like
[32:27] 40 years ago,
[32:29] because
[32:30] at that time we had no idea that this
[32:32] stuff was going to happen this fast. We
[32:34] thought we had plenty of time to worry
[32:36] about things like that. They When you
[32:38] When you can't get the AI to do much,
[32:40] you want to get it to do a little bit
[32:41] more, you don't worry about
[32:43] this stupid little thing is going to
[32:44] take over from people. You just want it
[32:46] to be able to do a little bit more of
[32:47] the things people can do.
[32:49] It's not like I knowingly did something
[32:53] thinking, "This might wipe us all out,
[32:55] but I'm going to do it anyway." Mhm.
[32:58] But it is a bit sad that it's not just
[33:00] going to be something for good.
[33:03] So, I feel I have a duty now to talk
[33:05] about the risks.
[33:07] And if you could play it forward, and
[33:08] you could go forward 30, 50 years, and
[33:09] you found out that it led to the
[33:10] extinction of humanity,
[33:13] and if that does end up being the
[33:17] being the outcome,
[33:21] Well, if you played it forward and
[33:22] it led to the extinction of humanity,
[33:25] I would use that to tell
[33:27] people to tell their governments that we
[33:29] really have to work on how we're going
[33:31] to keep this stuff under control.
[33:34] I think we need people to tell
[33:35] governments that governments have to
[33:37] force the companies to use their
[33:39] resources to work on safety.
[33:41] And they're not doing much of that,
[33:42] because you don't make profits that way.
[33:45] One of your your students we talked
[33:46] about earlier, um Ilya? Yep. Ilya left
[33:51] OpenAI. Yep. And there was lots of
[33:53] conversation around the fact that he
[33:55] left because he had safety concerns.
[33:57] Yes. And he's gone on to set set up a AI
[34:00] safety company.
[34:02] Yes.
[34:03] Why do you think he left?
[34:06] I think he left cuz he had safety
[34:07] concerns. Really?
[34:09] Um I still have lunch with him from time
[34:11] to time. Oh, okay. His parents live in
[34:13] Toronto, and when he comes to Toronto,
[34:14] we have lunch together. He doesn't talk
[34:16] to me about what went on at OpenAI, so I
[34:18] have no inside information about that,
[34:20] but I know Ilya very well.
[34:22] And he is genuinely concerned with
[34:23] safety. So, I think that's why he left.
[34:26] Because he was one of the top people. I
[34:27] mean, he was He was probably the most
[34:29] important person behind the development
[34:31] of
[34:32] um ChatGPT.
[34:34] The The early versions like GPT-2, he
[34:36] was very important in the development of
[34:37] that. You know him personally, so you
[34:39] know his character.
[34:41] Yes. He has a good moral compass. He's
[34:43] not like someone like Musk who has no
[34:45] moral compass.
[34:47] Does Sam Altman have a good moral
[34:48] compass?
[34:50] We'll see.
[34:53] I don't know Sam, so I don't want to
[34:56] comment on that.
[34:57] But from what you've seen,
[34:59] are you concerned about the actions that
[35:01] they've taken?
[35:02] Cuz if you know Ilya, and Ilya's a good
[35:04] guy, and he's left,
[35:06] that would give you some insight, yes.
[35:08] It would give you some reason to believe
[35:10] that there's a problem there. And if you
[35:12] look at Sam's statements
[35:15] some years ago,
[35:17] he sort of happily said in one
[35:20] interview, "Um this stuff will probably
[35:21] kill us all." That's not exactly what he
[35:23] said, but that's what it amounted to.
[35:25] Now he's saying you don't need to worry
[35:26] too much about it.
[35:28] And I suspect that's not driven by
[35:32] seeking after the truth. That's driven
[35:34] by seeking after money.
[35:36] Is it money, or is it power?
[35:39] Yeah, I shouldn't have said money. It's
[35:41] It's some some combination of this, yes.
[35:43] Okay, I guess money's a proxy for power,
[35:44] but
[35:45] I I've got a friend who's a billionaire,
[35:47] and he is in those circles.
[35:51] And when I went to his house and had
[35:53] lunch with him one day, he knows lots of
[35:54] people in AI building the biggest AI
[35:56] companies in the world, and he gave me a
[35:58] cautionary warning across the across his
[36:00] kitchen table in London, where he gave
[36:02] me an insight into the private
[36:03] conversations these people have, not the
[36:05] media interviews they do where they talk
[36:07] about safety and all these things, but
[36:09] actually what some of these individuals
[36:10] think is going to happen.
[36:12] And what do they think's going to
[36:13] happen?
[36:14] It's not what they say publicly.
[36:16] You know, one one person who I should
[36:18] probably shouldn't name, who is the who
[36:20] is leading one of the biggest AI
[36:21] companies in the world, he told me that
[36:23] he knows this person very well, and he
[36:24] privately thinks that we're heading
[36:26] towards this kind of dystopian world
[36:28] where we have just huge amounts of free
[36:30] time, we don't work anymore,
[36:32] and this person doesn't really give a
[36:33] [ __ ] about the harm that it's going to
[36:35] have on the world. And this person who
[36:36] I'm referring to is building one of the
[36:38] biggest AI companies in the world.
[36:39] And I then watch this person's
[36:41] interviews online,
[36:42] I'm trying to figure out which of the
[36:42] three people it is.
[36:43] Yeah, well, it's one of those three
[36:44] people. Okay. And I watch this person's
[36:46] interviews online, and I I reflect on
[36:47] the conversation that my billionaire
[36:49] friend had with me, who knows him, and I
[36:51] go, "Fucking hell, this guy's lying
[36:52] publicly. Like, he's not telling the the
[36:54] truth to the world." And that's haunted
[36:56] me a little bit. It's part of the reason
[36:57] I have so many conversations around AI
[36:59] on this podcast, because I'm like, I
[37:00] don't know if they're
[37:02] I think they're a lit Some of them are a
[37:04] little bit sadistic about power.
[37:06] I think they they like the idea that
[37:08] they will change the world. That they
[37:11] will be the one that fundamentally
[37:14] shifts the world. I think Musk is
[37:15] clearly like that, right?
[37:19] He's such a complex character that I
[37:21] don't I don't really know how to place
[37:22] Musk. Um He's done some really good
[37:24] things like um pushing electric cars.
[37:28] That was a really good thing to do.
[37:29] Yeah. Some of the things he said about
[37:31] self-driving were a bit exaggerated, but
[37:33] he
[37:34] That was a really useful thing he did.
[37:36] Giving the Ukrainians communication
[37:38] during the war with Russia. Starlink,
[37:40] yeah.
[37:41] That was a really good thing he did.
[37:43] There's a bunch of things like that.
[37:45] Mhm. Um but he's also done some very bad
[37:46] things.
[37:49] So, coming back to this point of
[37:53] the possibility of
[37:55] destruction,
[37:57] and the motives of these big companies,
[38:01] are you at all hopeful that anything can
[38:03] be done to slow down the pace and
[38:05] acceleration of AI? Okay, there's two
[38:07] issues. One is, can you slow it down?
[38:10] Yeah. And the other is, can you make it
[38:12] so of it will be safe in the end? It
[38:15] won't wipe us all out.
[38:17] I don't believe we're going to slow it
[38:18] down.
[38:20] Yeah.
[38:20] And the reason I don't believe we're
[38:21] going to slow it down is because there's
[38:22] competition between countries, and
[38:24] competition between companies within a
[38:26] country,
[38:27] and all of that is making it go faster
[38:29] and faster.
[38:30] And if the US slowed it down, China
[38:32] wouldn't slow it down.
[38:34] Does
[38:35] Ilya think it's possible to make AI
[38:37] safe?
[38:40] I think he does. He won't tell me what
[38:42] his secret source is.
[38:44] I don't I'm not sure how many people
[38:46] know what his secret source is. I think
[38:47] a lot of the investors don't know what
[38:48] his secret source is, but they've given
[38:50] him billions of dollars anyway, cuz they
[38:52] have so much faith in Ilya, which isn't
[38:54] foolish. I mean,
[38:56] he was very important in AlexNet, which
[38:59] got object recognition working well. He
[39:01] was the main
[39:03] the main force behind the things like
[39:05] GPT-2,
[39:07] which then led to
[39:08] ChatGPT.
[39:10] So, I think having a lot of faith in
[39:12] Ilya is a very reasonable decision.
[39:14] There's something quite haunting about
[39:15] the guy that made and was the main force
[39:18] behind GPT-2, which led rise to this
[39:20] whole revolution, left the company
[39:23] because of safety reasons.
[39:25] He knows something that I don't know.
[39:28] About what might happen next.
[39:29] Well,
[39:30] the company had
[39:32] No, I don't know the precise details. Um
[39:34] but I'm fairly sure the company had
[39:36] indicated that would it would use a
[39:38] significant fraction of its resources
[39:40] of the compute time for doing safety
[39:42] research, and then it kept then it
[39:45] reduced that fraction. I think that's
[39:47] one of the things that happened. Yeah,
[39:48] that was reported publicly. Yes. Yeah.
[39:51] We've gotten to the autonomous weapons
[39:54] part of the risk framework. Right. So,
[39:57] the next one is joblessness. Yeah. In
[40:00] the past, new technologies have come in
[40:03] which didn't lead to joblessness. New
[40:05] jobs were created.
[40:06] So, the classic example people use is
[40:08] automatic teller machines. When
[40:10] automatic teller machines came in,
[40:13] a lot of bank tellers didn't lose their
[40:14] jobs. They just got to do more
[40:16] interesting things.
[40:17] But here,
[40:19] I think this is more like when they got
[40:21] machines in the Industrial Revolution,
[40:24] and
[40:26] you can't have a job digging ditches now
[40:28] because a machine can dig ditches much
[40:30] better than you can.
[40:32] And I think for mundane intellectual
[40:34] labor,
[40:35] AI is just going to replace everybody.
[40:39] Now, it will may well be in the form of
[40:42] you have fewer people using AI
[40:45] assistants. So, it's a combination of a
[40:46] person and an AI assistant, and they're
[40:49] doing the work that 10 people could do
[40:51] previously.
[40:52] People say that it will create new jobs,
[40:54] though. So, we'll be fine.
[40:56] Yes, and that's been the case for other
[40:58] technologies, but this is a very
[40:59] different kind of technology. If it can
[41:01] do all mundane human intellectual labor,
[41:05] then what new jobs is it going to
[41:06] create? You'd have You'd have to be very
[41:09] skilled to have a job that it couldn't
[41:11] just do.
[41:12] So, I don't I don't think they're right.
[41:14] I think you can try and generalize from
[41:17] other technologies that come in like
[41:18] computers or automatic teller machines,
[41:21] but I think this is different. People
[41:23] use this phrase. They say, AI won't take
[41:25] your job, a human using AI will take
[41:27] your job. Yes, I think that's true. But
[41:29] for many jobs,
[41:31] that will mean you need far fewer
[41:32] people.
[41:33] My niece answers letters of complaint to
[41:36] a health service.
[41:38] It used to take her 25 minutes. She'd
[41:40] read the complaint, and she'd think how
[41:41] to reply, and she'd write a letter, and
[41:44] now she just scans it into
[41:47] um a chatbot,
[41:48] and
[41:50] it writes the letter. She just checks
[41:52] the letter. Occasionally, she tells it
[41:53] to
[41:54] revise it in some ways.
[41:56] The whole process takes her 5 minutes.
[41:59] That means she can answer five times as
[42:00] many letters.
[42:02] And that means they need five times
[42:04] fewer of her.
[42:06] So, she can do the job that five of her
[42:07] used to do.
[42:09] Now,
[42:10] that will mean they need less people. In
[42:13] other jobs, like in health care,
[42:16] they're much more elastic. So, if you
[42:19] could make doctors five times as
[42:20] efficient, we could all have five times
[42:22] as much health care for the same price,
[42:24] and that would be great. There's There's
[42:27] almost no limit to how much health care
[42:28] people can absorb.
[42:30] They always want more health care if
[42:32] there's no cost to it.
[42:34] There are jobs where you can make a
[42:36] person with an AI assistant much more
[42:38] efficient, and you won't need to less
[42:40] people because you'll just have much
[42:42] more of that being done. But most jobs I
[42:45] think are not like that.
[42:47] Am I right in thinking this sort of
[42:48] Industrial Revolution
[42:50] would play a role in replacing muscles?
[42:53] Yes, exactly. And this revolution in AI
[42:55] replaces intelligence, the brain.
[42:57] Yeah. So, So, mundane intellectual labor
[42:59] is like having strong muscles, and
[43:02] it's not worth much anymore.
[43:05] So, muscles have been replaced. Now, we
[43:06] intelligence is being replaced.
[43:08] Yeah.
[43:09] So, what remains?
[43:11] Maybe for a while some kinds of
[43:13] creativity. But the whole idea of
[43:15] superintelligence is nothing remains.
[43:17] Um these things will get to be better
[43:19] than us at everything. So, what what do
[43:20] we end up doing in such a world?
[43:22] Well, if they work for us,
[43:25] we end up getting lots of goods and
[43:27] services for not much effort.
[43:30] Okay. But that sounds tempting and nice,
[43:33] but I don't know. There's a cautionary
[43:35] tale in creating more and more ease for
[43:37] humans in in it going badly. Yes, and
[43:42] we need to figure out if we can make it
[43:44] go well.
[43:45] So, the the nice scenario is imagine a
[43:47] company with a CEO
[43:50] who is very dumb,
[43:52] probably the son of the former CEO,
[43:55] and he has an executive assistant who's
[43:57] very smart,
[43:59] and he says,
[44:01] I think we should do this.
[44:03] And the executive assistant makes it all
[44:04] work.
[44:05] The CEO feels great. He doesn't
[44:07] understand that he's not really in
[44:09] control. And in In some sense, he is in
[44:11] control. He suggests what the company
[44:13] should do. She just makes it all work.
[44:15] Everything's great.
[44:17] That's the good scenario.
[44:19] And the bad scenario? The bad scenario
[44:21] is she thinks, why do we need him?
[44:24] Yeah.
[44:26] I mean, in a world where we have
[44:28] superintelligence, which you don't
[44:29] believe is that far away.
[44:31] Yeah, I think it might not be that far
[44:33] away. It's very hard to predict, but I
[44:34] think we might get it in like 20 years
[44:37] or even less.
[44:38] I made the biggest investment I've ever
[44:40] made in a company because of my
[44:42] girlfriend. I came home one night, and
[44:44] my lovely girlfriend was up at 1:00 a.m.
[44:46] in the morning pulling her hair out as
[44:49] she tried to piece together her own
[44:51] online store for her business. And in
[44:54] that moment, I remembered an email I'd
[44:56] had from a guy called John, the founder
[44:58] of Stan Store, our new sponsor, and a
[45:00] company I've invested incredibly heavily
[45:02] in. And Stan Store helps creators to
[45:04] sell digital products, courses,
[45:06] coaching, and memberships all through a
[45:08] simple customizable link-in-bio system.
[45:11] And it handles everything: payments,
[45:12] bookings, emails, community engagement,
[45:14] and even links with Shopify. And I
[45:16] believe in it so much that I'm going to
[45:18] launch a Stan Challenge. And as part of
[45:22] this challenge, I'm going to give away
[45:24] $100,000 to one of you. If you want to
[45:26] take part in this challenge, if you want
[45:28] to monetize the knowledge that you have,
[45:30] visit stevenbartlett.stan.store
[45:33] to sign up. And you'll also get an
[45:35] extended 30-day free trial of Stan Store
[45:38] if you use that link. Your next move
[45:40] could quite frankly change everything.
[45:42] Because I talked about ketosis on this
[45:43] podcast and ketones, a brand called
[45:45] Ketone-IQ sent me their little product
[45:48] here. It was on my desk when I got to
[45:50] the office. I picked it up. It sat on my
[45:51] desk for a couple of weeks. Then one
[45:53] day, I tried it.
[45:55] And honestly, I have not looked back
[45:57] ever since. I now have this everywhere I
[46:00] go. When I travel all around the world,
[46:02] it's in my hotel room. My team will put
[46:03] it there. Before I did the podcast
[46:05] recording today that I've just finished,
[46:07] I had a shot of Ketone-IQ. And as is
[46:09] always the case when I fall in love with
[46:10] a product, I called the CEO and asked if
[46:12] I could invest a couple of million quid
[46:14] into their company. So, I'm now an
[46:16] investor in the company as well as them
[46:17] being a brand sponsor. I find it so easy
[46:20] to drop into deep focused work when I've
[46:22] had one of these. I would love you to
[46:24] try one and see the impact it has on
[46:26] you, your focus, your productivity, and
[46:27] your endurance. So, if you want to try
[46:29] it today, visit ketone.com/steven for
[46:31] 30% off your subscription. Plus, you'll
[46:33] receive a free gift with your second
[46:35] shipment. That's ketone.com/steven.
[46:39] I'm excited for you.
[46:41] I am.
[46:42] So, what's the difference between what
[46:43] we have now and superintelligence?
[46:44] Because it seems to be really
[46:45] intelligent to me when I use like
[46:47] ChatGPT-3 or Gemini or Okay. So, it's
[46:50] already AI is already better than us at
[46:53] a lot of things in particular areas
[46:56] like chess, for example. Yeah. AI is so
[46:59] much better than us that people will
[47:01] never beat those things again. Maybe the
[47:03] occasional win, but basically, they'll
[47:05] never be comparable again.
[47:07] Obviously, same in Go. In terms of the
[47:09] amount of knowledge they have,
[47:12] um something like GPT-4 knows thousands
[47:14] of times more than you do.
[47:16] There's a few areas in which your
[47:18] knowledge is better than it's.
[47:20] And
[47:21] almost all areas, it just knows more
[47:22] than you do. What areas am I better than
[47:25] it?
[47:27] Probably in interviewing CEOs. You're
[47:31] probably better at that.
[47:33] You've got a lot of experience at it.
[47:34] You're a good interviewer.
[47:36] You know a lot about it.
[47:37] If you tried If you got GPT-4 to
[47:40] interview a CEO, probably do a worse
[47:42] job.
[47:43] Okay.
[47:46] I'm trying to think if that if I agree
[47:47] with that statement. Uh GPT-4, I think,
[47:50] for sure. Yeah. Um but I But I guess you
[47:52] could train one on the how I ask Yeah, I
[47:54] guess you could train one on this how I
[47:56] ask questions and what I do and Sure.
[47:58] And if you took a general-purpose sort
[48:01] of foundation model, and then you
[48:02] trained it up on
[48:04] not just you, but every every interview
[48:07] you could find doing interviews like
[48:09] this,
[48:10] but especially you, it would probably
[48:11] get to be quite good at doing your job,
[48:13] but probably not as good as you for a
[48:14] while.
[48:17] Okay. So, there's a few areas left, and
[48:19] then superintelligence becomes when it's
[48:22] better than us at all things. When it's
[48:23] much smarter than you in almost all
[48:25] things, it's better than you. Yeah. And
[48:27] you you you say that this might be a
[48:28] decade away or so.
[48:30] Yeah, it might be. It might be even
[48:32] closer. Some people think it's even
[48:34] closer.
[48:35] It might well be much further. It might
[48:36] be 50 years away. That's still a
[48:38] possibility.
[48:39] It might be that somehow
[48:42] training on human data limits you to not
[48:44] be much smarter than humans. My guess is
[48:46] between 10 and 20 years we'll have
[48:48] superintelligence.
[48:50] On this point of joblessness, it is
[48:51] something I've been thinking a lot about
[48:53] in particular because I started messing
[48:54] around with AI agents, and we released
[48:56] an episode on the podcast actually this
[48:57] morning where we had a debate about AI
[48:59] agents with some CEO of a big AI agent
[49:02] company and a few other people.
[49:04] And it was the first moment where I had
[49:06] No, it was another moment where I had a
[49:08] eureka moment about what the future
[49:10] might look like. When I was able in the
[49:12] interview to tell this agent to order
[49:14] all of us drinks, and then 5 minutes
[49:16] later in the interview, you see the guy
[49:17] show up with the drinks, and I didn't
[49:19] touch anything. I just told it to order
[49:21] us drinks to the studio.
[49:22] And it didn't know about who you
[49:24] normally got your drinks from. It
[49:25] figured that out from the web. Yeah,
[49:26] figured it out cuz it went on Uber Eats.
[49:28] It has my my my data, I guess. And it I
[49:31] we put it on the screen in real time so
[49:32] everyone at home could see the agent
[49:34] going through the internet, picking the
[49:35] drinks, adding a tip for the driver,
[49:38] putting my address in, putting my credit
[49:39] card details in, and then the next thing
[49:40] you see is the drinks show up. So, that
[49:43] was one moment, and then the other
[49:44] moment was when
[49:45] I used a tool called Replit
[49:48] and I built software by just telling the
[49:50] agent what I wanted. Yes. It's amazing,
[49:52] right?
[49:53] It's amazing and terrifying at the same
[49:55] time. Yes. Because
[49:57] it can build software like that, right?
[49:59] Yeah. Remember that the AI, when it's
[50:02] training, is using code.
[50:05] And if it can modify its own code,
[50:08] then it gets quite scary, right? Cuz it
[50:10] can modify its own code.
[50:11] itself in a way we can't change
[50:12] ourselves.
[50:14] We can't change our innate endowment,
[50:16] right?
[50:17] There's nothing about itself that it
[50:19] couldn't change.
[50:21] On this point of joblessness, you have
[50:22] kids.
[50:23] I do. And they have kids?
[50:25] No, they don't have kids. No grandkids
[50:27] yet. What would you be saying to people
[50:28] about their career prospects in a world
[50:31] of super intelligence? What should we we
[50:32] be thinking about? Um in the meantime,
[50:35] I'd say it's going to be a long time
[50:37] before it's as good at physical
[50:39] manipulation as us. Okay. And so,
[50:42] a good bet would be to be a plumber.
[50:47] Until the humanoid robots show up.
[50:49] In such a world where there is mass
[50:51] joblessness, which is not something that
[50:53] you just predict, but this is something
[50:54] that Sam Altman at OpenAI, I've heard
[50:56] him predict and many of the CEOs and
[50:58] Elon Musk, I watched an interview which
[51:00] I'll play on screen of him being asked
[51:01] this question, and it's very rare that
[51:03] you see Elon Musk silent for 12 seconds
[51:05] or whatever it was. And then he
[51:07] basically says something about he
[51:09] actually is living in suspended
[51:10] disbelief. I he's basically just not
[51:12] thinking about it.
[51:13] When you think about advising your
[51:14] children on a career with so much that
[51:16] is changing,
[51:18] what do you tell them that's going to be
[51:19] of value?
[51:33] Well,
[51:35] that is a tough question to answer.
[51:37] I would just say, you know, to to sort
[51:39] of follow their heart in terms of what
[51:41] they they find um interesting to do or
[51:43] fulfilling to do.
[51:45] I mean, if I think about it too hard, it
[51:46] frankly can be uh just just
[51:48] disheartening and uh demotivating.
[51:51] Um
[51:53] because
[51:54] I mean, I I go through I I know I
[51:57] I've
[51:58] put a lot of blood, sweat, and tears
[51:59] into building the companies and then it
[52:02] and then I'm like, wait, well, like,
[52:03] should I be doing this? Because
[52:06] if I'm sacrificing time with friends and
[52:08] family that I would prefer to to to
[52:11] But but then, ultimately, the AI can do
[52:13] all these things.
[52:14] Does that make sense? I I don't know.
[52:17] Um
[52:18] to some extent, I have to have
[52:19] deliberate suspension of disbelief in
[52:21] order to be to remain motivated. Um
[52:25] so I I I guess I would say just, you
[52:27] know,
[52:31] work on things that you find
[52:32] interesting, fulfilling, and um
[52:34] and and that contribute uh some good to
[52:36] the rest of society. Yeah, a lot of
[52:37] these threats, it's very hard to
[52:41] intellectually, you can see the threat,
[52:44] but it's very hard to come to terms with
[52:45] it emotionally.
[52:47] Yeah.
[52:48] I I haven't come to terms with it
[52:49] emotionally yet.
[52:50] What do you mean by that?
[52:53] I haven't come to terms with
[52:55] what the development of super
[52:57] intelligence could do to my children's
[52:59] future.
[53:01] I'm okay. I'm 77.
[53:04] I'm going to be out of it
[53:05] Yeah, soon.
[53:06] But for my children and my my younger
[53:09] friends,
[53:11] my nephews and nieces,
[53:13] and their children,
[53:14] um
[53:17] I just don't like to think about what
[53:19] could happen.
[53:23] Why?
[53:25] Cuz it could be awful.
[53:29] In in what way?
[53:32] Well, if AI ever decided to take over,
[53:35] I mean, it would need people for a while
[53:37] to run the power stations
[53:39] until it
[53:40] designed better analog machines to run
[53:41] the power stations.
[53:43] There's so many ways it could get rid of
[53:46] people,
[53:47] all of which would, of course, be very
[53:48] nasty.
[53:50] Is that part of the reason you do what
[53:51] you do now?
[53:53] Yeah. I I mean, I think we should be
[53:54] making a huge effort right now
[53:57] to try and figure out if we can develop
[53:59] it safely.
[54:00] Are you concerned about the mid-term
[54:02] impact potentially on your nephews and
[54:04] your your kids in terms of their jobs as
[54:06] well? Yeah, I'm concerned about all
[54:07] that. Are there any particular
[54:09] industries that you think are most at
[54:10] risk? People talk about the creative
[54:11] industries a lot, and it's sort of
[54:13] knowledge work. They talk about lawyers
[54:15] and accountants and stuff like that.
[54:17] Yeah, so that's why I mentioned
[54:18] plumbers. I think plumbers are less at
[54:20] risk. Okay, I'm going to become a
[54:21] plumber.
[54:21] Someone like a legal assistant,
[54:24] a paralegal, Mhm. um they're not going
[54:27] to be needed for very long. And is there
[54:29] a wealth inequality issue here that will
[54:31] will
[54:32] arise from this?
[54:33] I think in a society which shared out
[54:35] things fairly,
[54:37] if you get a big increase in
[54:39] productivity,
[54:40] everybody should be better off. Mhm.
[54:43] But if you can replace lots of people by
[54:46] AIs,
[54:48] then the people who get replaced will be
[54:50] worse off
[54:52] and the company that supplies the AIs
[54:55] will be much better off
[54:58] and the company that uses the AIs.
[55:01] So, it's going to increase the gap
[55:02] between rich and poor. And we know that
[55:05] if you look at that gap between rich and
[55:07] poor, that basically tells you how nice
[55:09] a society is. If you have a big gap, you
[55:12] get very nasty societies in which people
[55:14] live in walled communities and put
[55:16] other people in mass jails.
[55:20] It's not good to increase the gap
[55:21] between rich and poor. The International
[55:23] Monetary Fund has expressed profound
[55:25] concerns that generative AI could cause
[55:27] massive labor disruptions and rising
[55:29] inequality and has called for policies
[55:31] that prevent this from happening.
[55:33] I read that in the Business Insider.
[55:35] Have they given any idea of what the
[55:37] policy should look like?
[55:38] No.
[55:39] Yeah, that's the problem. I mean, if AI
[55:41] can make everything much more efficient
[55:42] and get rid of people for most jobs
[55:45] or have a person assisted by AI doing
[55:48] many, many
[55:49] people's work, it's not obvious what to
[55:52] do about it. Universal basic income?
[55:55] Give everybody money? Yeah, I I I think
[55:57] that's a good start.
[55:59] And
[56:01] it stops people starving,
[56:03] but for a lot of people, their dignity
[56:04] is tied up with their job. I mean, who
[56:06] you think you are is tied up with you
[56:08] doing this job, right? Yeah.
[56:10] And
[56:12] if we said, we'll give you the same
[56:13] money just to sit around,
[56:15] that would impact your dignity.
[56:18] You said something earlier about it's
[56:20] surpassing or being superior to human
[56:22] intelligence. A lot of people, I think,
[56:24] like to believe that AI is is on a
[56:27] computer and it's something you can just
[56:28] turn off if you don't like it. Well, let
[56:30] me tell you why I think it's superior.
[56:32] Okay. Um it's digital.
[56:35] And because it's digital,
[56:37] you can have you can simulate a neural
[56:39] network on one piece of hardware. Yeah.
[56:42] And you can simulate exactly the same
[56:43] neural network on a different piece of
[56:45] hardware.
[56:46] Mhm. So, you can have clones of the same
[56:48] intelligence.
[56:49] Now, you could get this one to go off
[56:52] and look at one bit of the internet
[56:54] and this other one to look at a
[56:55] different bit of the internet. And while
[56:57] they're looking at these different bits
[56:58] of the internet,
[57:00] they can be syncing with each other, so
[57:02] they keep their weights the same. The
[57:04] connection strengths the same. Weights
[57:05] the connection strengths. Mhm. So, this
[57:07] one might look at something on the
[57:08] internet and say, oh, I'd like to
[57:09] increase this strength of this
[57:11] connection a bit.
[57:12] And it can convey that information to
[57:14] this one, so it can increase the
[57:16] strength of that connection a bit based
[57:17] on this one's experience. And when you
[57:19] say the strength of the connection,
[57:21] you're talking about learning. That's
[57:23] learning, yes. Learning consists of
[57:24] saying, instead of this one giving 2.4
[57:27] votes for whether that one should turn
[57:28] on, we'll have this one give 2.5 votes
[57:31] for whether this one should turn on.
[57:33] And that would be a little bit of
[57:34] learning. Mhm. So, these two different
[57:36] copies of the same neural net
[57:39] are getting different experiences.
[57:41] They're looking at different data, but
[57:43] they're sharing what they've learned by
[57:44] averaging their weights together. Mhm.
[57:47] And they can do that averaging at like a
[57:49] you can average a trillion weights.
[57:51] When you and I transfer information,
[57:54] we're limited to the amount of
[57:55] information in a sentence. And the
[57:57] amount of information in a sentence is
[57:58] maybe 100 bits. It's very little
[58:00] information. We're lucky if we're
[58:02] transferring like 10 bits a second. Mhm.
[58:04] These things are transferring trillions
[58:06] of bits a second. So, they're billions
[58:08] of times better than us at sharing
[58:10] information.
[58:12] And that's because they're digital and
[58:14] you can have two bits of hardware using
[58:16] the connection strengths in exactly the
[58:17] same way. We're analog and you can't do
[58:20] that. Your brain's different from my
[58:21] brain.
[58:22] And if I could see the connection
[58:24] strengths between all your neurons, it
[58:26] wouldn't do me any good cuz my neurons
[58:28] work slightly differently and they're
[58:29] connected up slightly differently. Mhm.
[58:31] So, when you die,
[58:33] all your knowledge dies with you.
[58:35] When these things die, suppose you take
[58:37] these two digital intelligences that are
[58:39] clones of each other,
[58:40] and you destroy the hardware they run
[58:42] on.
[58:43] As long as you've stored the connection
[58:44] strengths somewhere, you can just build
[58:46] new hardware
[58:48] that executes the same instructions, so
[58:50] it'll know how to use those connection
[58:52] strengths, and you've recreated that
[58:54] intelligence. So, they're immortal.
[58:56] We've actually solved the problem of
[58:58] immortality,
[58:59] but it's only for digital things.
[59:02] So, it knows
[59:03] it will essentially know everything that
[59:06] humans know, but more, because it will
[59:07] learn new things.
[59:09] It will learn new things. It will also
[59:11] see all sorts of analogies that people
[59:13] probably never saw.
[59:15] So, for example,
[59:17] at the point when GPT-4 couldn't look on
[59:19] the web,
[59:20] I asked it, why is a compost heap like
[59:23] an atom bomb?
[59:25] Off you go. I have no idea.
[59:28] Exactly. Excellent. Most That's exactly
[59:30] what most people would say. It said,
[59:32] "Well, the time scales are very
[59:33] different and the energy scales are very
[59:35] different.
[59:37] But then it went on to talk about how a
[59:38] compost heap, as it gets hotter,
[59:40] generates heat faster.
[59:42] And an atom bomb, as it produces more
[59:44] neutrons, generates neutrons faster.
[59:47] Mhm. And so they're both chain
[59:48] reactions, but at very different time
[59:50] and energy scales.
[59:52] And I believe GPT-4 had seen that during
[59:54] its training.
[59:56] It had understood the analogy between a
[59:58] compost heap and an atom bomb. And the
[60:00] reason I believe that is, if you've only
[60:02] got a trillion connections, remember you
[60:04] have 100 trillion, Mhm. and you need to
[60:06] have thousands of times more knowledge
[60:08] than a person,
[60:09] you need to compress information into
[60:11] those connections.
[60:13] And to compress information, you need to
[60:15] see analogies between different things.
[60:17] In other words, it needs to see all the
[60:19] things that are chain reactions and
[60:21] understand the basic idea of a chain
[60:22] reaction and code that, and then code
[60:24] the ways in which they're different. And
[60:26] that's just a more efficient way of
[60:27] coding things than coding each of them
[60:29] separately. Mhm.
[60:31] So, it's seen many, many analogies,
[60:33] probably many analogies that people have
[60:35] never seen.
[60:36] That's why I also think that people who
[60:38] say these things will never be creative,
[60:39] they're going to be much more creative
[60:41] than us.
[60:42] Because they're going to see all sorts
[60:43] of analogies we never saw. And a lot of
[60:45] creativity is about seeing strange
[60:47] analogies.
[60:49] People are somewhat romantic about the
[60:50] specialness of what it is to be human.
[60:52] And you hear lots of people saying, "Oh,
[60:53] it's very, very different. It's a it's a
[60:55] computer. We are, you know, we're
[60:56] conscious. We are creative. We we have
[60:59] these sort of innate, unique abilities
[61:02] that the computers will never have."
[61:04] What do you say to those people? I'd
[61:05] argue a bit with the innate. Um
[61:09] So,
[61:11] the first thing I say is we have a long
[61:13] history of believing people are special.
[61:16] And we should have learned by now. We
[61:18] thought we were at the center of the
[61:19] universe. We thought we were made in the
[61:21] image of God.
[61:23] White people thought they were very
[61:24] special. Mhm. We just tend to want to
[61:27] think we're special. Mhm.
[61:29] My belief is
[61:31] that more or less everyone
[61:33] has a completely wrong model of what the
[61:35] mind is.
[61:36] Let's suppose I drink a lot or I drop
[61:38] some acid, Mhm. and not recommended,
[61:41] and I
[61:43] say to you,
[61:44] "I have the subjective experience of
[61:46] little pink elephants floating in front
[61:47] of me." Mhm.
[61:49] Most people
[61:51] interpret that as
[61:53] there's some kind of inner theater
[61:55] called the mind,
[61:58] and only I can see what's in my mind.
[62:01] And in this inner theater,
[62:03] there's a little pink elephants floating
[62:04] around. Mhm.
[62:06] So, in other words, what's happened is
[62:07] my perceptual system's gone wrong,
[62:10] and I'm trying to indicate to you how
[62:12] it's gone wrong and what it's trying to
[62:14] tell me.
[62:15] And the way I do that is by telling you
[62:17] what would have to be out there in the
[62:19] real world
[62:21] for it to be telling the truth.
[62:24] And so these little pink elephants,
[62:26] they're not in some inner theater.
[62:29] These little pink elephants are
[62:30] hypothetical things in the real world.
[62:33] And that's my way of telling you how my
[62:35] perceptual system's telling me fibs.
[62:38] So, now I must do that with a chatbot.
[62:39] Yeah.
[62:41] Cuz I believe that current multimodal
[62:43] chatbots have subjective experiences.
[62:46] And very few people believe that.
[62:48] But I'll try and make you believe it.
[62:50] So, suppose I have a multimodal chatbot.
[62:52] It's got a robot arm, so it can point,
[62:54] and it's got a camera, so it can see
[62:55] things.
[62:57] And I put an object in front of it,
[62:59] and I say, "Point at the object."
[63:00] It goes like this. No problem.
[63:03] Then I put a prism in front of its lens.
[63:06] And so then I put an object in front of
[63:07] it, and I say, "Point at the object."
[63:09] And it goes there.
[63:11] Good.
[63:11] And I say, "No, that's not where the
[63:13] object is. The object's actually
[63:15] straight in front of you, but I put a
[63:17] prism in front of your lens."
[63:19] And the chatbot says, "Oh, I see. The
[63:21] prism bent the light rays. So, um the
[63:24] object's actually there, but I had the
[63:26] subjective experience that it was
[63:27] there."
[63:28] Mhm. Now, if the chatbot says that, it's
[63:31] using the word subjective experience
[63:32] exactly the way people use them. It's an
[63:35] alternative view of what's going on.
[63:37] They're hypothetical states of the
[63:38] world,
[63:40] which if they were true would mean my
[63:41] perceptual system wasn't lying. And
[63:43] that's the best way I can tell you what
[63:44] my perceptual system's doing when it's
[63:46] lying to me. Mhm. Now,
[63:49] we need to go further to deal with
[63:50] sentience and consciousness and feelings
[63:51] and emotions, but I think in the end
[63:53] they're all going to be dealt with in a
[63:54] similar way. There's no reason machines
[63:56] can't have them all.
[63:58] But people say machines can't have
[63:59] feelings.
[64:00] And people are curiously confident about
[64:03] that. I've no idea why. Suppose I make a
[64:05] battle robot, and it's a little battle
[64:08] robot,
[64:09] and it sees a big battle robot
[64:11] that's much more powerful than it.
[64:14] It would be really useful if it got
[64:15] scared.
[64:17] Mhm.
[64:18] Now,
[64:19] when I get scared, um various
[64:22] physiological things happen that we
[64:23] don't need to go into, and those won't
[64:25] happen with the robot.
[64:27] But all the cognitive things, like I
[64:28] better get the hell out of here, Yeah.
[64:30] Mhm. and I better sort of
[64:32] change my way of thinking, so I focus
[64:35] and focus and focus and I get
[64:36] distracted,
[64:37] all of that will happen with robots,
[64:39] too.
[64:41] People will build in things so that
[64:43] they, when it the circumstance is such
[64:45] they should get the hell out of there,
[64:46] they get scared and run away.
[64:48] They'll have emotions then.
[64:50] They won't have the physiological
[64:51] aspects, but they will have all the
[64:53] cognitive aspects.
[64:55] And I think it would be odd to say
[64:56] they're just simulating emotions. No,
[64:58] they're really having those emotions.
[64:59] The little robot got scared and ran
[65:00] away.
[65:02] It's not running away because of
[65:03] adrenaline, it's running away because of
[65:05] a sequence of sort of neurological in
[65:07] its neural net processes happened, which
[65:10] means which have the equivalent effect
[65:11] to adrenaline.
[65:13] So, do you do you think
[65:14] just adrenaline, right? There's a lot of
[65:15] cognitive stuff goes on when you get
[65:16] scared. Yeah.
[65:18] So, do you think that
[65:21] there is conscious AI?
[65:23] And when I say conscious, I mean
[65:25] that represents the same properties of
[65:27] consciousness that a human has.
[65:29] There's two issues here. There's a sort
[65:30] of empirical one and a philosophical
[65:32] one. I don't think there's anything in
[65:34] principle that stops machines from being
[65:36] conscious.
[65:38] I'll give you a little demonstration of
[65:39] that before we carry on. Mhm. Suppose I
[65:41] take your brain,
[65:43] and I take one brain cell in your brain,
[65:46] and I replace it by, it's a bit Black
[65:48] Mirror-like, I replace it by a little
[65:50] piece of nanotechnology that's just the
[65:52] same size,
[65:54] that behaves in exactly the same way
[65:56] when it gets pings from other neurons.
[65:57] It sends out pings just as the brain
[65:59] cell would have.
[66:00] So, the other neurons don't know
[66:01] anything's changed.
[66:03] Okay. I've just replaced one of your
[66:05] brain cells with this little piece of
[66:06] nanotechnology. Would you still be
[66:08] conscious?
[66:10] Yeah.
[66:11] Now you can see where this argument's
[66:12] going. Yeah. So, if you replaced all of
[66:14] them,
[66:15] as I replace them all, at what point do
[66:16] you stop being conscious?
[66:19] Well, people think of consciousness as
[66:20] this like ethereal thing that exists
[66:23] maybe beyond the brain cells. Yeah,
[66:25] well, people have a lot of crazy ideas.
[66:29] Um People don't know what consciousness
[66:31] is, and they often don't know what they
[66:32] mean by it. Mhm. And then they fall back
[66:35] on saying, "Well,
[66:36] I know it cuz I've got it, and I can see
[66:38] that I've got it." And they fall back on
[66:40] this theater model of the mind, which I
[66:42] think is nonsense.
[66:43] What do you think of consciousness as if
[66:45] you had to try and define it? Is it cuz
[66:46] I think of it as just like the awareness
[66:48] of myself? I don't know.
[66:50] I think it's a term we'll stop using.
[66:53] Suppose you want to understand how a car
[66:54] works.
[66:56] Well, you know some cars have a lot of
[66:57] oomph, and other cars have a lot less
[66:59] oomph. Like an Aston Martin's got lots
[67:01] of oomph. Mhm. And a little Toyota
[67:04] Corolla doesn't have much oomph.
[67:06] But oomph isn't a very good concept for
[67:08] understanding cars.
[67:10] Um if you want to understand cars, you
[67:12] need to understand about electric
[67:13] engines or petrol engines and how they
[67:15] work.
[67:16] And it gives rise to oomph.
[67:18] But oomph isn't a very useful
[67:20] explanatory concept. It's a kind of
[67:21] essence of a car. It's the essence of an
[67:23] Aston Martin.
[67:24] But it doesn't explain much. I think
[67:26] consciousness is like that.
[67:28] And I think we'll stop using that term.
[67:30] But I don't think there's anything any
[67:32] reason why a machine shouldn't have it.
[67:34] If
[67:36] your view of consciousness is that it
[67:37] intrinsically involves self-awareness,
[67:40] then the machine's got to have
[67:41] self-awareness. It's got to have
[67:42] cognition about its own cognition and
[67:44] stuff.
[67:45] But
[67:47] I'm a materialist through and through,
[67:50] and I don't think there's any reason why
[67:52] a machine shouldn't have consciousness.
[67:54] Do you think they do then have the same
[67:56] consciousness that we think of ourselves
[67:58] as being uniquely uh
[68:01] given as a gift when we're born?
[68:03] I'm ambivalent about that at present.
[68:06] So,
[68:08] I don't think there's this hard line. I
[68:10] think as soon as you have a machine that
[68:12] has some self-awareness,
[68:14] it's got some consciousness.
[68:16] Um I think it's an emergent property of
[68:19] a complex system.
[68:21] It's not a sort of essence that's
[68:24] throughout the universe. It's you make
[68:26] this really complicated system that's
[68:27] complicated enough to have a model of
[68:29] itself,
[68:30] and it does perception.
[68:32] And I think
[68:34] then you're beginning to get a conscious
[68:36] machine. So, I don't think there's any
[68:37] sharp distinction between what we've got
[68:39] now and conscious machines. I don't
[68:41] think it's going to one day we're going
[68:42] to wake up and say, "Hey, if you put
[68:45] this special chemical in, it becomes
[68:46] conscious." It's not going to be like
[68:48] that.
[68:49] I think we all wonder if these computers
[68:50] are like thinking like we are
[68:53] on their own when we're not there, and
[68:55] if they're experiencing emotions, if
[68:56] they're contending with I we I think we
[68:58] probably, you know, we think about
[68:59] things like love and things that feel
[69:01] unique to biological species.
[69:04] Um are they sat there thinking?
[69:06] Are they do they have concerns?
[69:08] I think they really are thinking.
[69:10] And I think as soon as you make AI
[69:11] agents, they will have concerns. If you
[69:14] want to make an effective AI agent,
[69:16] suppose you let's take a call center.
[69:18] Mhm. In a call center, you have people
[69:20] at present.
[69:22] They have all sorts of emotions and
[69:23] feelings,
[69:24] which are kind of useful. So, suppose I
[69:27] call up the call center
[69:30] and I'm actually lonely and I don't
[69:32] actually want to know the answer to why
[69:35] my computer isn't working. I just want
[69:36] somebody to talk to.
[69:38] After a while, the person in the call
[69:41] center
[69:42] will either get bored or get annoyed
[69:44] with me
[69:45] and will terminate it.
[69:47] Well, you replace them by an AI agent.
[69:50] The AI agent needs to have the same kind
[69:52] of responses. If someone's just called
[69:54] up cuz they just want to talk to the AI
[69:55] agent and we're happy to talk for whole
[69:57] the whole day to the AI agent, that's
[69:59] not good for business and you want an AI
[70:02] agent that either gets bored or gets
[70:03] irritated and says, "I'm sorry, but I
[70:05] don't have time for this." Then
[70:07] once it does that, I think it's got
[70:09] emotions.
[70:11] Now,
[70:12] like I say, emotions have two aspects to
[70:15] them. There's the cognitive aspect and
[70:17] the behavioral aspect and then there's a
[70:19] physiological aspect and these go
[70:21] together with us
[70:23] and if the AI agent gets embarrassed, he
[70:26] won't go red. Yeah. Um So, there's no
[70:28] physiological
[70:29] won't start sweating. Yeah. But it might
[70:31] have all the same behavior and in that
[70:32] case I'd say, "Yeah, it's having emotion
[70:34] It's got an emotion." So, it's going to
[70:36] have the same sort of cognitive thought
[70:38] and then it's going to act upon that
[70:39] cognitive thought.
[70:40] way, but without the physiological
[70:42] responses. And does that matter that it
[70:45] doesn't go red in the face and it's just
[70:46] a different I mean, that's a response to
[70:48] the
[70:48] it somewhat different from us. Yeah. For
[70:50] some things, the physiological aspects
[70:53] are very important like love.
[70:55] They're a long way from having love the
[70:56] same way we do.
[70:58] But I don't see why they shouldn't have
[71:00] emotions.
[71:01] So, I think what's happened is people
[71:04] have a model of how the mind works and
[71:07] what feelings are and what emotions are
[71:09] and their model is just wrong.
[71:12] What um what brought you to Google?
[71:15] You You worked at Google for about a
[71:16] decade, right? Yeah. What brought you
[71:18] there?
[71:19] I have a
[71:21] son who has learning difficulties
[71:23] and in order to be sure he would never
[71:26] be out on the street
[71:28] I needed to get several million dollars
[71:32] and I wasn't going to get that as an
[71:33] academic.
[71:34] I tried. So, I taught a Coursera course
[71:37] in the hope that I'd make lots of money
[71:39] that way, but there was no money in
[71:40] that.
[71:41] So, I figured out, well,
[71:43] the only way to get millions of dollars
[71:46] is to sell myself to a big company.
[71:51] And so, when I was 65
[71:54] fortunately for me, I had two brilliant
[71:56] students who produced something called
[71:58] AlexNet, which was neural net that was
[72:01] very good at recognizing objects in
[72:02] images.
[72:04] And
[72:05] so,
[72:07] Ilya and Alex and I
[72:09] set up a little company and auctioned
[72:11] it.
[72:12] And we actually set up an auction where
[72:13] we had a number of big companies bidding
[72:15] for us.
[72:17] And that company was called AlexNet. No,
[72:21] the the network that recognized objects
[72:24] was called AlexNet. Company was called
[72:26] DNN Research, deep neural network
[72:28] research.
[72:29] And it was doing things like this. I'll
[72:30] put this graph up on the screen.
[72:31] That's AlexNet. This picture shows eight
[72:34] images and AlexNet's ability, which is
[72:38] your company's ability to spot what was
[72:40] in those images. Yeah.
[72:42] So, it could tell the difference between
[72:43] various kinds of mushroom
[72:45] and about 12% of ImageNet is dogs
[72:49] and to be good at ImageNet, you have to
[72:51] tell the difference between very similar
[72:53] kinds of dog
[72:54] and it would got to be very good at
[72:56] that.
[72:57] And your your company AlexNet won
[72:59] several awards, I believe, for its
[73:01] ability to out outperform its
[73:03] competitors and so Google ultimately
[73:05] ended up acquiring your technology.
[73:08] Google acquired that technology and some
[73:10] other technology.
[73:12] And you went to work at Google at age,
[73:14] what, 66? I went at age 65 to work at
[73:17] Google.
[73:18] 65 and you left at age 76? 75.
[73:21] 75, okay. I worked there for more or
[73:23] less exactly 10 years. And what were you
[73:24] doing there?
[73:26] Okay, they were very nice to me.
[73:28] They said They said pretty much you can
[73:29] do what you like.
[73:31] I worked on something called
[73:32] distillation that did really work well
[73:35] and that's now used all the time. In AI?
[73:38] In AI and distillation is a way of
[73:40] taking what a big model knows, a big
[73:42] neural net knows, and getting that
[73:44] knowledge into a small neural net. Then
[73:46] at the end, I got very interested in
[73:48] analog computation and whether it would
[73:50] be possible to get these big language
[73:52] models running in analog hardware
[73:55] so they used much less energy.
[73:57] And it was while I was doing that work
[73:59] that I began to really realize how much
[74:01] better digital is for sharing
[74:03] information.
[74:05] Was there a eureka moment?
[74:08] There was a eureka month or two.
[74:10] Um and it was a sort of coupling of
[74:13] ChatGPT coming out. Although Google had
[74:15] very similar things a year earlier. And
[74:17] I'm
[74:18] I'd seen those and that had a big impact
[74:20] effect on me.
[74:21] The closest I had to a eureka moment was
[74:24] when a Google system called Palm was
[74:28] able to say why a joke was funny.
[74:30] And I'd always thought of that as a kind
[74:32] of landmark. If it can say why a joke's
[74:34] funny, it really does understand.
[74:37] And it could say why a joke was funny.
[74:41] And that coupled with realizing why
[74:43] digital is so much better than analog
[74:45] for sharing information
[74:47] suddenly made me
[74:49] very interested in AI safety
[74:51] and that these things were going to get
[74:53] a lot smarter than us.
[74:55] Why did you leave Google?
[74:57] The main reason I left Google was cuz I
[74:59] was 75
[75:01] and I wanted to retire.
[75:02] I've done a very bad job of that.
[75:05] The precise time year when I left Google
[75:07] was so that I could talk freely at a
[75:09] conference at MIT.
[75:11] But I left cuz
[75:12] I was
[75:14] I'm old and I was finding it harder to
[75:15] program. I was making many more mistakes
[75:17] when I programmed, which is very
[75:18] annoying. You wanted to talk freely at a
[75:21] conference at MIT. Yes. I'd MIT
[75:23] organized by MIT Tech Review. What did
[75:25] you want to talk about freely? AI
[75:26] safety. And you couldn't do that while
[75:28] you were at Google? Well, I could have
[75:31] done it while I was at Google and Google
[75:32] encouraged me to stay and work on AI
[75:33] safety. I said I could do whatever I
[75:35] liked on AI safety.
[75:37] You kind of censor yourself. If you work
[75:39] for a big company
[75:40] you don't feel right saying things that
[75:43] will damage the big company.
[75:45] Even if you could get away with it, it
[75:46] just feels wrong to me.
[75:49] I didn't leave cuz I was cross with
[75:50] anything Google was doing. I think
[75:51] Google actually behaved very
[75:52] responsibly. When they had these big
[75:55] chatbots, they didn't release them.
[75:57] Possibly cuz they were worried about
[75:59] their reputation. They had a very good
[76:01] reputation and they didn't want to
[76:02] damage it. So, OpenAI didn't have a
[76:05] reputation and so they could afford to
[76:07] take the gamble. I mean, there's also a
[76:09] big conversation happening around how it
[76:11] will cannibalize their core business in
[76:12] search.
[76:14] There is now, yes.
[76:15] Yeah. Yeah.
[76:16] And it's the old innovator's dilemma to
[76:18] some degree, I guess.
[76:19] Exactly. Yes, it is.
[76:20] Bad skin, I've had it and I'm sure many
[76:23] of you listening have had it, too. Or
[76:25] maybe you have it right now.
[76:27] I know how draining it can be,
[76:29] especially if you're in a job where
[76:30] you're presenting often like I am. So,
[76:32] let me tell you about something that's
[76:33] helped both my partner and me and my
[76:35] sister, which is red light therapy. I
[76:38] only got into this a couple of years
[76:39] ago, but I wish I'd known a little bit
[76:41] sooner. I've been using our show
[76:43] sponsors BonCharge's infrared sauna
[76:45] blanket for a while now, but I just got
[76:47] hold of their red light therapy mask as
[76:49] well. Red light has been proven to have
[76:51] so many benefits for the body. Like any
[76:53] area of your skin that's exposed will
[76:55] see a reduction in scarring, wrinkles
[76:57] and even blemishes.
[76:58] It also helps with complexion. It boosts
[77:00] collagen and it does that by targeting
[77:02] the upper layers of your skin. And
[77:04] BonCharge ships worldwide with easy
[77:06] returns and a year-long warranty on all
[77:08] of their products. So, if you'd like to
[77:10] try it yourself, head over to
[77:11] boncharge.com/diary
[77:13] and use code diary for 25% off any
[77:16] product sitewide. Just make sure you
[77:18] order through this link.
[77:19] boncharge.com/diary
[77:22] with code diary. Make sure you keep what
[77:24] I'm about to say to yourself. I'm
[77:26] inviting 10,000 of you to come even
[77:29] deeper into the Diary of a CEO. Welcome
[77:31] to my inner circle. This is a brand new
[77:34] private community that I'm launching to
[77:35] the world. We have so many incredible
[77:37] things that happen that you are never
[77:39] shown. We have the briefs that are on my
[77:41] iPad when I'm recording the
[77:42] conversation. We have clips we've never
[77:44] released. We have behind-the-scenes
[77:46] conversations with the guest and also
[77:47] the episodes that we've never ever
[77:50] released and so much more. In the
[77:53] circle, you'll have direct access to me.
[77:55] You can tell us what you want this show
[77:56] to be, who you want us to interview and
[77:58] the types of conversations you would
[77:59] love us to have. But remember, for now,
[78:02] we're only inviting the first 10,000
[78:04] people that join before it closes. So,
[78:06] if you want to join our private close
[78:08] community, head to the link in the
[78:09] description below or go to DOAC
[78:11] circle.com.
[78:14] I will speak to you there.
[78:16] I'm continually shocked by the types of
[78:17] individuals that listen to this
[78:18] conversation um because they come up to
[78:20] me sometimes. So, I hear from
[78:21] politicians, I hear from some royal
[78:23] people, I hear from entrepreneurs all
[78:24] over the world, whether they are the
[78:26] entrepreneurs building some of the
[78:27] biggest companies in the world or
[78:28] they're, you know, early-stage startups.
[78:31] For those people that are listening to
[78:33] this conversation now, that are in
[78:35] positions of power and influence
[78:37] world leaders, let's say.
[78:39] What's your message to them?
[78:42] I'd say what you need is highly
[78:43] regulated capitalism. That's what seems
[78:45] to work best. And what would you say to
[78:47] the average person?
[78:49] Not doesn't work in the industry
[78:51] somewhat concerned about the future
[78:54] doesn't know if they're hopeless or not.
[78:56] What should they be doing in their own
[78:57] lives?
[78:59] My feeling is there's not much they can
[79:01] do. This isn't isn't going to be decided
[79:04] by Just as climate change isn't going to
[79:06] be decided by people separating out the
[79:09] plastic bags from the
[79:11] um compostables. That's not going to
[79:13] have much effect. It's going to be
[79:14] decided by whether the lobbyists for the
[79:17] big energy companies can be kept under
[79:19] control. I don't think there's much
[79:21] people can do to
[79:24] except for
[79:26] try and pressure their governments
[79:29] to
[79:30] force the big companies to work on AI
[79:32] safety.
[79:33] That they can do.
[79:36] You've lived a fascinating fascinating
[79:39] winding life. I think one of the things
[79:40] most people don't know about you is that
[79:42] your family has a
[79:45] big history of being involved in
[79:47] tremendous things. You have a family
[79:49] tree which is one of the most impressive
[79:51] that I've ever seen or read about.
[79:53] Your great
[79:55] great grandfather, George Boole, founded
[79:57] the Boolean algebra logic which is one
[80:00] of the foundational principles of modern
[80:01] computer science.
[80:03] You have uh your great great
[80:04] grandmother, Mary Everest Boole, who was
[80:07] a mathematician and educator who made
[80:09] huge
[80:10] leaps forward in mathematics from what I
[80:12] was able to ascertain. Um I mean I can
[80:14] get the list goes on and on and on. I
[80:16] mean your great great uncle, George
[80:17] Everest,
[80:19] is what Mount Everest is named after.
[80:22] Is that is that correct?
[80:24] I think he's my great great great uncle.
[80:26] His
[80:28] his niece
[80:30] married George Boole.
[80:33] So Mary Mary Boole was Mary Everest
[80:35] Boole.
[80:36] Um she was a niece of Everest.
[80:39] And your first cousin once removed, Joan
[80:41] Hinton,
[80:42] was involved in the new a nuclear
[80:43] physicist who worked on the Manhattan
[80:45] Project which is the World War II
[80:47] development of the first nuclear bomb.
[80:49] Yeah, she was one of the two female
[80:51] physicists at Los Alamos.
[80:53] And then
[80:55] after they dropped the bomb, she moved
[80:57] to China.
[80:58] Why?
[80:59] She was very cross with them dropping
[81:00] the bomb.
[81:01] And her family had a lot of links with
[81:04] China.
[81:05] Her mother was friends with Chairman
[81:08] Mao.
[81:09] Hm.
[81:10] Quite weird.
[81:13] When you look back at your life,
[81:14] Geoffrey,
[81:16] with the hindsight you have now and the
[81:18] retro- retrospective clarity,
[81:22] what might you have done differently if
[81:23] you were advising me?
[81:26] I guess I have
[81:27] two pieces of advice.
[81:30] One is
[81:31] if you have an intuition
[81:33] that people are doing things wrong and
[81:35] there's a better way to do things,
[81:37] don't give up on that intuition just cuz
[81:39] people say it's silly.
[81:41] Don't give up on the intuition until you
[81:43] figured out why it's wrong. Figured out
[81:45] for yourself why that intuition isn't
[81:47] correct.
[81:48] And usually
[81:50] it's wrong
[81:51] if it disagrees with everybody else and
[81:53] you'll eventually figure out why it's
[81:54] wrong.
[81:56] But just occasionally you'll have an
[81:58] intuition that's actually right and
[82:00] everybody else is wrong. Hm.
[82:02] And I lucked out that way.
[82:04] Early on I thought neural nets are
[82:05] definitely the way to go to make AI.
[82:09] And almost everybody said that was
[82:11] crazy.
[82:12] And I stuck with it because I couldn't
[82:14] it just seemed to me it was obviously
[82:15] right.
[82:17] Now
[82:18] the idea that you should stick with your
[82:19] intuitions
[82:21] isn't going to work if you have bad
[82:22] intuitions. But if you have bad
[82:24] intuitions, you're never going to do
[82:26] anything anyway, so you might as well
[82:27] stick with them.
[82:30] And in your own career journey, is there
[82:32] anything you look back on and say with
[82:33] the hindsight I have now, I should have
[82:35] taken a different approach at that
[82:36] juncture?
[82:39] I wish I spent more time with my wife.
[82:42] Um
[82:47] and with my children when they were
[82:48] little.
[82:50] I was kind of obsessed with work.
[82:55] Your wife passed away. Yeah.
[82:57] From ovarian cancer?
[82:59] No, or that was another wife. Okay. Um I
[83:02] had two wives die of cancer. Oh, really?
[83:05] Sorry.
[83:05] The first one died of ovarian cancer and
[83:06] the second one died of pancreatic
[83:08] cancer. And you wish you'd spent more
[83:09] time with her. With the second wife,
[83:11] yeah.
[83:12] Who was a wonderful person.
[83:14] Why do you say that in your 70s? What is
[83:17] it that you've you've figured out that I
[83:19] might not know yet?
[83:21] Oh, just cuz she's gone and I can't
[83:22] spend more time with her now. Hm.
[83:26] But you didn't know that at the time.
[83:29] At the time you think
[83:33] I mean it was likely I would die before
[83:35] her just cuz she was a woman and I was a
[83:37] man.
[83:38] Um I didn't
[83:40] I just didn't spend enough time when I
[83:42] could.
[83:43] I I think I I inquire there because I
[83:46] think there's many of us that are so
[83:47] consumed with what we're doing
[83:48] professionally that we kind of assume or
[83:50] more immortality with our partners
[83:52] because they've always been there, so we
[83:53] Yeah.
[83:54] I mean
[83:54] She was very supportive of me spending a
[83:56] lot of time working.
[83:58] But
[83:59] And why do you say your children as
[84:00] well? What's the what's the issue?
[84:02] spend enough time with them when they
[84:03] were little.
[84:05] And you regret that now? Yeah.
[84:10] Hm.
[84:12] If you um if you had a closing message
[84:14] for for my for my listeners about AI and
[84:16] AI safety,
[84:17] what would that be, Geoffrey?
[84:20] There's still a chance that we can
[84:22] figure out how to develop AI that won't
[84:25] want to take over from us.
[84:27] And because there's a chance, we should
[84:29] put enormous resources into trying to
[84:31] figure that out cuz if we don't, it's
[84:32] going to take over.
[84:34] And are you hopeful?
[84:36] I just don't know. I'm agnostic.
[84:40] You must get get better get in bed at
[84:42] night and when you're thinking to
[84:43] yourself about probabilities of
[84:45] outcomes, there must be a bias in one
[84:48] direction cuz there certainly is for me.
[84:50] I mean imagine everyone listening now
[84:51] has a
[84:53] internal prediction
[84:55] that they might not say out loud, but of
[84:57] how they think it's going to play out.
[84:59] I really don't know. I genuinely don't
[85:01] know.
[85:02] I think it's incredibly uncertain.
[85:04] When I'm feeling slightly depressed, I
[85:06] think
[85:07] people are toast. AI is going to take
[85:09] over. When I'm feeling cheerful, I think
[85:12] we'll figure out a way.
[85:13] Maybe one of the facets of being a human
[85:15] um is because we've always been here
[85:18] like we were saying about our loved ones
[85:19] and our relationships, we assume
[85:22] casually that we will always be here and
[85:24] we'll always figure everything out. But
[85:26] there's a beginning and an end to
[85:27] everything as we saw from the dinosaurs.
[85:28] I mean
[85:29] Yeah.
[85:31] And
[85:32] we have to face the possibility
[85:35] that unless we do something
[85:38] soon,
[85:39] we're near the end.
[85:42] We have a closing tradition on this
[85:43] podcast where the last guest leaves a
[85:44] question in their diary.
[85:46] And the question that they've left for
[85:47] you
[85:49] is
[85:54] with everything that you see ahead of
[85:56] us,
[85:57] what is the biggest threat you see to
[85:59] human happiness?
[86:04] I think the joblessness is a fairly
[86:07] urgent short-term threat to human
[86:09] happiness.
[86:10] I think if you make lots and lots of
[86:12] people unemployed,
[86:13] even if they get universal basic income,
[86:16] um they're not going to be happy.
[86:19] Because they need purpose. Because they
[86:21] need purpose, yes. And struggle.
[86:23] to feel they're contributing something.
[86:25] They're useful.
[86:27] And do you think that outcome that
[86:29] there's going to be huge job
[86:29] displacement is more probable than not?
[86:32] Yes.
[86:33] I do. And what's the
[86:34] That one I think is definitely more
[86:36] probable than not. If I worked in a call
[86:38] center, I'd be terrified.
[86:41] And what's the time frame for that in
[86:42] terms of mass job displacement?
[86:44] it's beginning to happen already.
[86:46] I wrote an article in the Atlantic
[86:47] recently
[86:48] that said it's already getting hard for
[86:51] university graduates to get jobs.
[86:53] And part of that may be that people are
[86:56] already using AI for the jobs they would
[86:58] have got.
[87:00] I spoke to the CEO of a major company
[87:02] that everyone will know of, lots of
[87:03] people use, and he said to me in DMs
[87:07] that they used to have seven just over
[87:08] 7,000 employees. He said uh by last year
[87:11] they were down to I think 5,000. He said
[87:13] right now they have 3,600 and he said by
[87:15] the end of summer because of AI agents,
[87:17] they'll be down to 3,000.
[87:19] So you've said
[87:20] It's happening already. Yes. He's halved
[87:22] his workforce because AI agents can now
[87:24] handle 80% of the customer service
[87:26] inquiries and other things.
[87:28] So it's it's happening already.
[87:30] Yeah.
[87:31] So urgent action is needed. Yep. I don't
[87:33] know what that urgent action is.
[87:36] That's a tricky one cuz that depends
[87:37] very much on the political system.
[87:40] And political systems are all going in
[87:42] the wrong direction at present.
[87:44] And what do we need to do? Save up
[87:45] money? Like do we save money? Do we move
[87:47] to another part of the world?
[87:49] I don't know.
[87:50] What would you tell your kids to do?
[87:53] They said, "Dad, look, there's going to
[87:54] be loads of just job displacement."
[87:56] Because I worked for Google for 10
[87:57] years, they have enough money. Okay.
[88:00] Okay. [ __ ]
[88:01] So they're not typical. What if they
[88:03] didn't have money?
[88:04] Train to be a plumber.
[88:05] Really? Yeah.
[88:10] Geoffrey, thank you so much. You're the
[88:12] first Nobel Prize winner that I've ever
[88:15] had a conversation with, I think, in my
[88:17] life.
[88:18] So that's a a tremendous honor and you
[88:20] you you received that award for a
[88:22] lifetime of exceptional work in pushing
[88:23] the world forward in so many profound
[88:25] ways that will lead to great
[88:27] and that have led to great advancements
[88:29] in things that matter so much to us. And
[88:31] now you've turned this season in your
[88:32] life to shining a light on some of your
[88:34] own work, but also on the the the
[88:36] broader risks of AI and how um
[88:40] and how it might impact us adversely.
[88:41] And there's very few people
[88:43] that have worked inside the the machine
[88:45] of a Google or a big tech company that
[88:47] have contributed to the field of AI that
[88:50] are now at the very forefront of warning
[88:52] us against the very thing that they
[88:53] worked upon.
[88:55] There are actually a surprising number
[88:57] of us now.
[88:58] They're not as uh
[89:00] as public and they're actually quite
[89:02] hard to get to have these kinds of
[89:03] conversations because many of them are
[89:04] still in that industry.
[89:06] So, you know, someone who tries to
[89:08] contact these people often and ask
[89:09] invites them to have conversations, they
[89:11] often are a little bit hesitant to speak
[89:13] openly, so they speak privately,
[89:15] but they're less willing to openly
[89:16] because maybe maybe they still have
[89:17] something at
[89:18] at some sort of incentives at play. I
[89:20] have an advantage over them which is I'm
[89:23] older so I'm unemployed so I can say
[89:24] what I have. Well there you go.
[89:26] So thank you for doing what you do it's
[89:27] a real honor and please do continue to
[89:29] do it. Thank you. Thank you so much.
[89:34] Many people think I'm joking when I say
[89:36] that but I'm not. What are you coming
[89:38] for? Yeah.
[89:41] And plumbers are pretty well paid.
