Mustafa Suleyman Demands Global Hardware Controls to Halt AI Wave

穆斯塔法·苏莱曼呼吁实施全球硬件管制以遏制AI浪潮 Inflection AI联合创始人穆斯塔法·苏莱曼认为，除非各国政府对先进硬件和高端计算基础设施实施严格的防扩散管制，否则人类将面临生存威胁。 穆斯塔法·苏莱曼探讨了人工智能的双重属性，权衡了其解决气候变化和疾病等人类重大危机的潜力，以及未受遏制的超级智能与合成生物学带来的灾难性风险。他回顾了DeepMind的关键里程碑（如AlphaGo的突破），并探讨了应对呈指数级扩张的技术所带来的紧迫挑战。苏莱曼概述了必要的遏制措施，包括硬件卡点、多因素安全框架以及防止危险扩散的新型全球治理模式。他最终指出，尽管遏制自主技术在历史上看似前所未有，但人类必须使其成为可能，以守护自身的未来。 人工智能呈现出根本性的双重悖论：用于医疗突破的同类数学模型，同样能被武器化并用于军事目标锁定及生物武器制造。 (0:00) 十多年来，驱动 AI 发展的算力规模以每年 10 倍的速度呈指数级增长，从 2013 年训练 Atari 模型的 2 petaflops 跃升至现代前沿模型的逾 100 亿 petaflops。 (4:57) 政客与决策者普遍陷入“悲观回避陷阱”，因害怕被视为危言耸听而选择忽视 AI 引发的工作岗位替代与社会动荡预警。 (12:51) 与核材料需要庞大工业设施提纯不同，AI 属于具有通用性的全用途软件，能在互联网上高度压缩与开源传播，令传统技术遏制手段极难奏效。 (16:04) 大国之间的竞争制造了危险的“竞态条件”：各方因担忧对手领先而争相加速研发，阻碍了国际社会在超级智能失控前落实预防性安全规范。 (26:24)

Mustafa Suleyman argues that containing artificial intelligence is the defining civilizational challenge of the 21st century, warning that without proactive global guardrails, the rapid proliferation of omni-use models will outpace human control and empower malicious actors. While emphasizing severe existential and geopolitical hazards, he maintains that deliberate governance can prevent runaway autonomy and steer AI toward unlocking radical material abundance. Exponential Scaling and the Omni-Use Dilemma (4:57) Suleyman illustrates the velocity of AI development through massive computational expansion. In 2013, DeepMind utilized 2 petaflops of compute to train algorithms on Atari games, whereas modern flagship models at Inflection AI consume over 10 billion petaflops—representing a 10x annual increase in compute over a decade. Adjacent fields reflect similar exponential shifts; genome sequencing costs collapsed from roughly $100 million in the early 2000s to an anticipated $100 by the mid-2020s, lowering barriers to synthesizing novel biological entities. The fundamental danger stems from artificial intelligence being an omni-use technology. The same core architectures used to identify cancerous tumors or improve battery chemistry can be repurposed to pinpoint battlefield targets or construct hyper-transmissible pathogens. Furthermore, DeepMind's AlphaGo demonstrated via its unprecedented "Move 37" in 2016 that self-learning systems can formulate strategies beyond human anticipation, making systems that operate beyond human logic inherently difficult to predict and control. Geopolitical Competition and Asymmetric Digital Proliferation (26:24) A central barrier to safety is an international "race condition" between nation-states. If one country pauses development to establish safety protocols, geopolitical rivals face strong incentives to accelerate, generating an arms-race dynamic fueled by mutual paranoia. Suleyman notes that this dynamic is compounded by a "pessimism aversion trap," where policymakers avoid confronting legitimate technological disruptions by retreating into comfortable assumptions that lost jobs will naturally be replaced by new industries. Suleyman contrasts AI containment with past non-proliferation frameworks like the Nuclear Non-Proliferation Treaty. Refining enriched uranium requires extensive, detectable industrial infrastructure, which historically limited nuclear weapons to nine states. Digital software risks are far more asymmetric: high-performing models can be compressed to run at 60 to 70 times smaller than their original footprints and released globally via open-source repositories, granting small groups destructive power once restricted to major powers. Physical Choke Points and Corporate Accountability (75:23) Because digital code spreads rapidly, Suleyman asserts that containment must leverage physical supply chain bottlenecks. Advanced AI clusters, such as the $1.3 billion supercomputer built by Inflection AI, depend heavily on specialized graphics processing units (GPUs) and physical internet routing infrastructure. Sovereign states can enforce the "precautionary principle" by regulating chip manufacturing, export licenses, cloud APIs, and biological synthesis equipment to restrict dangerous proliferation before deployment. Suleyman insists on drawing clear regulatory boundaries against digital autonomy. While physical automation like self-driving cars can operate safely within defined telemetry and GPS constraints, he advocates for an outright global ban on self-improving autonomous agents and lethal autonomous military weapons. To alter corporate incentives, he highlights Inflection AI's legal charter as a Public Benefit Corporation, which requires executives to balance societal well-being and safety against traditional shareholder profit maximization. Binary Horizons: Radical Abundance Versus Systemic Fragility (97:37) Suleyman envisions two divergent futures over the next several decades. In an aligned, contained scenario, AI breakthroughs in battery chemistry and quantum computing could eliminate primary energy bottlenecks within 20 to 30 years. Driving the marginal costs of clean power, food production, healthcare, and water desalination toward near-zero would create "radical abundance" and decouple survival from wage labor without depending on conventional state-issued cash transfers. Failure to contain these tools, however, risks systemic instability. Immediate threats include automated cyberattacks, deepfakes, and audio cloning scams that undermine basic social trust, requiring advanced defense models and multi-factor authentication. Over a longer horizon, superior intelligences may have no natural incentive to respect human welfare; Suleyman rejects transhumanist brain-uploading concepts as unscientific, concluding that humanity must deliberately ensure advanced systems remain provably subservient to human command. Suleyman acknowledges that historical precedent works against technology containment, yet insists society must reject the belief that containment is impossible. While democratic election cycles and economic competition create intense pressures to abandon caution, he concludes that coordinated oversight, hardware choke points, and broad civic engagement remain essential to keeping humanity in control of its future.

DeepMind与Inflection AI联合创始人Mustafa Suleyman指出，人工智能具备彻底重塑全球经济与解决气候、医疗危机的潜力，但其技术本质具有军民两用的泛用性；人类必须主动建立全球层面的技术遏制框架，否则将面临失控风险。 算力飞跃与技术双刃剑效应 (4:57) Suleyman回顾了自2010年创立DeepMind以来的技术演进。2013年深度学习革命期间，AI仅通过屏幕像素与基础奖励机制在Atari游戏中独立发展出人类未曾预料的策略，当时训练仅耗费约2 petaflops算力。而如今Inflection AI训练旗舰模型已动用超过100亿petaflops，算力规模连续十年以每年十倍的速度增长。这种能够自主生成原始知识与语言理解的能力构成了巨大突破，但也带来了深刻悖论：能够在医学影像中精准检测肿瘤的底层数学架构，同样可以被直接用于战场上的自主军事目标锁定。 遏制悖论与地缘政治竞相加速 (12:51) Suleyman认为当前最大的挑战是技术遏制，即人类必须有能力对无休止的技术扩张说“不”。然而历史上除了禁用氟利昂、化学武器和致盲激光等少数具备替代方案的先例，人类极少成功封禁具备重大影响力的新技术。当前各国与商业机构陷入博弈死结：任何一方单方面放缓研发，都担忧竞争对手抢占主导地位并掠夺预计达15万亿美元的AI经济红利。Suleyman批评政策精英层存在“悲观回避陷阱”，往往用历史中“新技术自然会创造新就业”的说辞来回避失业与颠覆性风险。同时，与需要庞大工业基础设施提炼浓缩铀的核武器不同，开源模型可在压缩数十倍后在全球网络中低成本分发，极大加剧了数字扩散的不对称性。 物理扼流点与可落地的治理蓝图 (75:23) 针对遏制可能性的质疑，Suleyman主张不能将监管停留在抽象层面，而应聚焦于物理供应链中的“扼流点”。由于顶尖大模型的训练依赖昂贵的实体计算集群——例如Inflection AI耗资逾1.3亿美元构建全球第二大超级计算机——主权国家完全可以通过高阶GPU等芯片的制造、出口限制以及海底光缆枢纽进行监管。在制度层面，他倡导通过对AI企业征税反哺劳动力转型培训，并设立类似二战后联合国或世贸组织职能的全球技术稳定机制。Suleyman将Inflection AI注册为益友公司（Public Benefit Corporation），从法律层面要求企业在追求股东利润的同时必须承担社会安全与人类福祉义务；他明确呼吁在全球范围内全面禁止具备自主自我迭代能力的网络智能体以及完全自主的致命性军事武器。 生物融合风险与超智能对齐难题 (20:07) 展望2050年，Suleyman警告AI将与生物制造深度融合。人类基因测序成本已从2000年代初的约1亿美元骤降至百美元级别，降幅达百万倍；可编程生物技术使得定制新型生物实体变得唾手可得，一旦合成高传染性或极致命的病原体，将引发灾难性全球生物安全风险。面对智力可能超出人类数百至数千倍的超级智能（AGI/ACI），Suleyman反驳了“高度智能的机器必然会善待人类”的假设，指出人类对待宠物的模式证明了高智力物种对低智力物种缺乏平等的依附动机。因此，技术体系必须在设计之初就确立绝对的从属地位与预防原则，确保人类维持在地球上的主导物种地位，同时他对将意识脱离生物肉身上传至服务器的超人类主义构想持严厉的怀疑态度。 近端威胁演进与极端充裕愿景 (57:52) 在应对长远风险的同时，Suleyman强调了眼前的现实威胁。深度伪造与语音克隆技术正被广泛应用于电信诈骗，严重削弱公众对数字信息的信任基础，传统双重认证将不得不升级为多维交叉验证；同时，防御复杂攻击必须依靠防御性AI与人类紧密协作。对于未来的终极走向，Suleyman描述了截然相反的两种结局：若遏制失败，恶意微小派系将获取足以瞬间动摇全球秩序的失控工具；若遏制成功，AI与量子计算的结合将在未来20至30年内彻底解决电池化学与储能难题，使能源、洁净水淡化、食品培育、医疗及交通的边际成本趋近于零，从而带来无需强制劳动的“极致充裕”社会。 Mustafa Suleyman强调遏制技术并非被动放弃进步，而是人类文明必须建立的主动防御机制。其论述虽然提供了结合硬件出口管制与新型治理架构的思路，但如何在全球多极对抗与激烈商业追逐的现实环境下达成真正有效的国际协同，仍是一个未完全解决的重大难题。

## Background 该视频由知名播客频道《The Diary Of A CEO》（主持人 Steven Bartlett）于 2023 年 9 月 4 日发布在 YouTube 平台上。访谈围绕 DeepMind 联合创始人兼 Inflection AI 创始人穆斯塔法·苏莱曼（Mustafa Suleyman）及其新书《即将到来的浪潮》（*The Coming Wave*）展开，深入探讨了人工智能与相邻前沿科技日益加剧的危险性、技术遏制（Containment）的可行性以及人类文明的终极抉择。 ## TL;DR 苏莱曼在对话中深入剖析了人工智能、人形机器人与合成生物学并进发展所带来的指数级冲击，指出 AI 兼具军民多用途与开源扩散特性，使全球防范陷入了严峻的博弈困局。他警告社会切勿掉入逃避直面灾难的“悲观规避陷阱”，并痛陈当前民主政体受困于短期选举周期，严重缺乏对全人类长期技术稳定性负责的治理机制。为此，他提出了一套以管制 GPU 算力硬件、监控物理网络通道和落实前置预防原则为核心的技术遏制蓝图，强调遏制“必须成为可能”。在对终局的推演中，他指出若成功遏制 AI 将引领人类迈向基础生活成本归零的“极端丰裕”时代，而一旦失控则将赋予少数恶意者瞬间瘫痪文明秩序的毁灭性力量。 ## Atmosphere & style 整场对话在理性的学术推演与深邃的存在性焦虑之间交织展开，主持人巴特利特抽丝剥茧的尖锐质问与苏莱曼作为一线缔造者的诚恳剖白碰撞出强烈的思想张力。对谈过程中不时出现的凝重停顿与下意识的身体防卫动作，生动传递出直面不可逆技术浪潮时的战栗与无力感。在凝视技术深渊的冰冷现实与畅想极端丰裕的乌托邦愿景之间，两人的交流宛如一场暴风雨前夜的文明辩论，既饱含透支心力的沉重与疲惫，又闪烁着拒绝投降的人文主义坚守。 ## Bulletin - [12:19](https://www.youtube.com/watch?v=CTxnLsYHWuI&t=739s) 前沿模型训练算力呈现年均 10 倍的指数级扩张，Inflection AI 训练前沿系统需消耗约 100 亿 Petaflops 算力，并斥资逾 13 亿美元自建超级计算机集群以支撑高阶智能演进。 - [16:04](https://www.youtube.com/watch?v=CTxnLsYHWuI&t=964s) AI 具备独特的“全用途技术”（Omni-use）属性，同一核心架构既能用于医疗筛查也能用于军用空袭；加之 GPT-3 级别开源模型体积已缩减 60 至 70 倍，使顶尖技术极易在无监管状态下加速扩散。 - [01:15:18](https://www.youtube.com/watch?v=CTxnLsYHWuI&t=4518s) 对 AI 的有效遏制依赖物理实体的“扼杀点”（Choke Points），通过对尖端 GPU 芯片供应链及跨国海底光缆物理节点的流量管制，国际社会有能力在基础设施层压缩危险技术接触者的数量。 - [49:48](https://www.youtube.com/watch?v=CTxnLsYHWuI&t=2988s) 若成功实现技术遏制，AI 有望在未来 20 至 30 年内突破电池储能与核聚变瓶颈，将能源与生活必需品边际生产成本压降至近乎于零，推动全人类步入无需依赖出卖劳力的“极端丰裕”时代。 ## Key takeaways - [00:00](https://www.youtube.com/watch?v=CTxnLsYHWuI&t=0s) 遏制（Containment）不是可选项而是人类生存底线：如果人类无法在未来数十年内有效建立约束智能的框架，“它就会反过来遏制我们”。 - [12:19](https://www.youtube.com/watch?v=CTxnLsYHWuI&t=739s) 精英圈层深陷“悲观规避陷阱”（Pessimism Aversion Trap），习惯性搬出历史就业增长数据盲目自我宽慰，刻意回避对潜在失控终局展开严肃的政策讨论。 - [16:04](https://www.youtube.com/watch?v=CTxnLsYHWuI&t=964s) “全用途技术”属性使 AI 无法像氟氯烃或化学武器那样被简单一刀切封禁，其巨大的商业红利与医疗救命价值驱使全球资本和政府加速狂飙。 - [19:31](https://www.youtube.com/watch?v=CTxnLsYHWuI&t=1171s) 智力正在成为一种全新的资本形式，如同历史上的土地与石油争夺，AI 将成为所有拥有权力的实体竞相夺取以放大自身意志的终极杠杆。 - [25:51](https://www.youtube.com/watch?v=CTxnLsYHWuI&t=1551s) 合成生物学与 AI 的协同将基因测序成本压低百万倍，面对工程化致命病原体的外溢风险，行业必须恪守“前置预防原则”，宁可放慢步调也须严控生物试剂与底层算力准入。 - [28:44](https://www.youtube.com/watch?v=CTxnLsYHWuI&t=1724s) 地缘博弈中的“竞生条件”（Race Condition）容易沦为自毁型预言，国际社会应汲取冷战时期《核不扩散条约》的成功范式，寻求大国在物种共同生存底线上的协调合作。 - [40:16](https://www.youtube.com/watch?v=CTxnLsYHWuI&t=2416s) AlphaGo“第 37 手棋”揭示了机器超越人类认知的创造力，这种创造力虽是解决复杂危机的必备工具，却也内生出系统行为脱离人类预期与控制的巨大风险。 - [01:24:18](https://www.youtube.com/watch?v=CTxnLsYHWuI&t=5058s) 坚决抵制“致命自主性”（Lethal Autonomy）并划定安全红线，严禁赋予线上 AI 脱离人类监管的无限制自我迭代权限，严禁制造完全自主决定生杀大权的军用武器。 - [01:08:30](https://www.youtube.com/watch?v=CTxnLsYHWuI&t=4110s) 西方四年民主选举周期所催生的“短期主义”成为重大治理阻碍，面对未来可能释放的 15 万亿美元产业蛋糕，全球迫切需要设立跨越数十年的“全球技术稳定机制”。 - [01:15:18](https://www.youtube.com/watch?v=CTxnLsYHWuI&t=4518s) 遏制战略的关键是将掌握存在性风险尖端模型的实体数量压缩到极小范围，并在政策上通过对 AI 企业征税主动注入阻力，为社会争取适应时间。 - [57:50](https://www.youtube.com/watch?v=CTxnLsYHWuI&t=3470s) Deepfake 与高仿克隆语音正在彻底瓦解社会信任基石，身份安全防御亟需从传统双因素认证（2FA）跨越至跨渠道独立验证的 3FA 或 4FA 机制。 - [01:28:35](https://www.youtube.com/watch?v=CTxnLsYHWuI&t=5315s) Inflection AI 采用共益企业（PBC）法律架构，试图从根本上破除传统公司单纯追求股东利益最大化的弊病，在法理上确立商业盈利与公共安全的强制平衡。 - [01:41:11](https://www.youtube.com/watch?v=CTxnLsYHWuI&t=6071s) 遏制成功的终局是普惠超级智能带来的极端丰裕与个体权力扩散；遏制失败的终局则是全球网络化世界中极少数恶意个体获得瞬间摧毁文明秩序的不对称毁灭力量。 ## Detailed walkthrough ### [00:00](https://www.youtube.com/watch?v=CTxnLsYHWuI&t=0s) 引言与遏制难题：技术狂潮的双刃剑 访谈在一场关于 AI 终极风险的直白对话中拉开帷幕。作为 DeepMind 的联合创始人，穆斯塔法·苏莱曼（Mustafa Suleyman）在回顾其在 Google 研发对话模型（如早期的 LaMDA 项目）的经历时坦言，当时团队推迟发布技术正是因为内心深处的焦虑与不安。他警告称，未来所有组织和实体都将竞相追逐算力与智能，而这项技术的双刃剑属性极其剧烈：它既能被用来在医学影像中精准检测癌变肿瘤，也能在战场上锁定致命打击的目标。 苏莱曼特别强调，最核心的威胁在于“极端少数人获得瞬间破坏世界稳定的能力”。当少数怀揣恶意的人能够轻而易举地获取足以造成大规模伤亡的智能工具时，传统的安全防御体系将被彻底击穿。面对“AI 是否能够被遏制（containment）”的提问，苏莱曼给出了斩钉截铁的回答：遏制“必须可能（it must be possible）”，因为如果人类无法有效遏制它，“它就会反过来遏制我们（otherwise it contains us）”。这也是他创办新公司 Inflection AI 的底层动机——打造一个真正站在人类这边的 AI。若人类成功驾驭，将极大降低交通与医疗等基础设施的成本；若失败，代价将难以承受。 ```mermaid flowchart TD accTitle: AI技术双刃剑与遏制逻辑 accDescr: 展示AI强大的双向影响力及遏制必要性的逻辑流程图 subgraph DualUse["AI 技术潜能"] A["高维机器智能"] --> B1["正面效益：医疗癌症筛查、廉价交通与能源"] A --> B2["毁灭性风险：战场目标锁定、少数人造成系统性破坏"] end subgraph Dilemma["终极抉择与遏制"] B2 --> C{"能否有效遏制 (Containment)？"} C -- "成功遏制" --> D["人类掌控工具，解决生存危机"] C -- "未能遏制" --> E["AI 遏制人类，引发存在性灾难"] end ``` --- ### [02:11](https://www.youtube.com/watch?v=CTxnLsYHWuI&t=131s) 从惊恐到担当：人类面临的时代必然性 当主持人史蒂文·巴特利特（Steven Bartlett）问及如何从情绪上概括当下面对 AGI（通用人工智能）乃至 ACI（人工特异/上限智能，Artificial Capable Intelligence）时的心境，苏莱曼坦诚地吐露了一个词——“吓坏了（petrified）”。然而随着时间的推移，对利弊得失与发展轨迹的深层思考促使他的心态发生了转变：AI 的演进具有极其强大的必然性（inevitable），与其被动恐惧，全人类更需要凝聚共识，“张开双臂去引导它、约束它”。 苏莱曼认为，对这一必然轨迹的主动干预赋予了人类不可推卸的使命。在接下来的 50 年中，人类面临着前所未有的生存危机：海水淡化、将粮食生产成本降低百倍、普及可再生能源以及应对气候变化等。要在资源紧缩的环境中以更低消耗产出更多价值，人类目前几乎缺乏切实可行的解决方案，而能充当科研伙伴、发明者与创造者的机器智能几乎是不可替代的杠杆。但他同时强调，AI 的具体形态、治理架构、所有权归属以及训练方式并非命中注定，而完全取决于人类未来十年的选择：“如果我们不主动去迎接这个挑战，它就会直接强加在我们身上。” --- ### [04:57](https://www.youtube.com/watch?v=CTxnLsYHWuI&t=297s) DeepMind 的顿悟时刻：从雅达利游戏到数字生成 苏莱曼回顾了自己从技术中真正感受到敬畏与震颤的起点。2010 年创立 DeepMind 后的头几年，技术进展尚处于起步阶段，直到 2013 年深度学习浪潮爆发。当时团队取得了一个里程碑式的突破：训练强化学习模型自学玩经典的雅达利（Atari）街机游戏（如《太空侵略者》Space Invaders 和《乒乓》Pong）。AI 仅仅通过观察屏幕上一帧帧原始像素的变化，并在“得分（Reward）”的刺激驱动下自主决定上下左右或射击操作。令团队震惊的是，系统在完全没有人工灌输规则的情况下，自发摸索出了人类高级玩家甚至从未察觉的高阶通关技巧。这种“通过刺激加奖励就能自主发现人类尚未掌握的策略”的能力，让苏莱曼既兴奋于它为文明创造新知识的潜力，又第一次深切体会到近乎恐怖的寒意。 而更早的技术顿悟时刻发生在 2013 年之前。苏莱曼站在办公室里，目睹了一台早期的图像生成模型在 300x300 像素的极小黑白网格上生成手写数字 7。伴随运算，大约 10 到 15 秒内，纯黑的屏幕上逐渐浮现出灰色斑块，紧接着白色像素从黑暗中聚合，勾勒出一个前所未见的手写体“7”。尽管在如今逼真的生成技术面前这显得十分粗糙，但在那一瞬间他意识到：模型已经理解了数字 7 的内在特征表征，并能够凭空生成新样本。事实证明，随后的十年完全遵循了这一逻辑轨迹——海量算力与海量数据的叠加，让技术在十年内直接从生成粗糙的手写黑白数字跨越到一句话即可凭空构建逼真的照片与视频。 --- ### [09:17](https://www.youtube.com/watch?v=CTxnLsYHWuI&t=557s) 预测之外的震撼：语言空间的非线性质变 面对过去十年的发展历程，苏莱曼指出了“可预测”与“彻底出乎意料”的分界线。在视觉与音频领域，由于物理信号存在强局部连续性——临近像素拼凑成线条、边缘、拐角，再延伸为五官乃至全景，声音信号在语谱图上同样存在声素规律——因此基于这种局部结构预测图像和视频是符合直觉的。然而，真正令苏莱曼感到始料未及的，是同样的生成范式竟然在高度抽象的“语言空间”中产生了惊人奇迹。 语言的复杂性远超物理像素。当给出前文“猫坐在（The cat sat on the...）”时，接下来出现的词汇不仅可能是垫子（mat），还可能是桌子、汽车、椅子、树木、高山或云朵，其潜在候选词与抽象意图有着天文数字般的可能性。在早期，苏莱曼完全无法凭直觉相信这类简单模型能够撑起如今的大型语言模型（LLM）生态。然而，当算力以不可思议的节奏几何倍增时，质变发生了： - **指数级算力跃升**：在过去 10 年间，行业几乎保持着每年将语言模型计算量放大 10 倍（10x）的恐怖增速。 - **雅达利时期的基准**：2013 年 DeepMind 训练 Atari 游戏模型时，仅消耗了 2 个 Petaflops 的计算量（Peta 即千万亿次，1 Petaflop 代表每秒 $10^{15}$ 次浮点运算，2 Petaflops 意味着总计 2000 万亿次运算）。 - **当下的算力鸿沟**：仅仅一两单位在当年就称得上庞大天文数字的算力基数，在每年翻十倍的累加下，将如今最顶尖的大模型推向了难以想象的规模，进而催生出 ChatGPT、Google Bard，以及 Inflection AI 推出的主打共情与高情商的个人智能体 Pi（pi.ai）。 ### [12:19](https://www.youtube.com/watch?v=CTxnLsYHWuI&t=739s) 算力爆炸与“悲观规避陷阱” 穆斯塔法·苏莱曼（Mustafa Suleyman）首先分享了当前最前沿大语言模型的训练规模：在 Inflection AI，他们训练一个模型就需要消耗高达 100 亿 Petaflops 的算力，换算下来就是“100 亿个百万亿次计算”（10 billion million billion calculations）。这种极其庞大、超乎人类想象的计算量，带来了一个已被反复验证的事实——只要每年将模型规模扩大 10 倍，AI 就会呈现出一种近乎“魔法般”的体验，与它对话就像在同一个博学且聪明的真正人类交谈。 主持人史蒂芬·巴特利特（Steven Bartlett）坦承自己近期与多位行业人士交流后，内心感到一种理性的恐惧。这种恐惧并非情绪化的焦虑，而是逻辑推演后的无力感：人类似乎注定会沦为这颗星球上不再占据主导地位的物种，而他无法从逻辑上推导出力挽狂澜的途径。这恰好呼应了苏莱曼新书《大浪将至》（*The Coming Wave*）第一章的核心论调——“技术是不可遏制的”（Containment is not possible）。史蒂芬随即向苏莱曼请教书中提出的一个概念：“悲观规避陷阱”（pessimism aversion trap）。 苏莱曼解释道，几乎所有人第一次真正直面这股浪潮时，本能反应都是恐惧和惊悚——担心工作被抢走、担心儿女爱上 AI、甚至困惑于当非人实体具备人类特质时“何以为人”。但在过去几年里，绝大多数人（尤其是精英阶层）的默认应对方式是逃避悲观情绪，缩回安全区，假装一切都会向好发展。人们总会搬出历史经验安慰自己：“我们过去每次技术革命都在创造新工作，从未造成永久性失业，目前的失业率甚至创历史新低”。这种心态本质上不是渴望乐观，而是极度恐惧悲观。正是这种“悲观规避陷阱”，阻碍了社会去开展那些应对未来所急需的艰难对话。 - **算力指标**：Inflection 的前沿大模型训练消耗约 100 亿 Petaflops。 - **扩展定律**：每年模型规模扩大 10 倍，推动 AI 产生质变级的拟人智能体验。 - **心态反思**：精英圈层的“悲观规避”导致社会回避了对潜在黑暗结局的严肃政策讨论。 --- ### [16:04](https://www.youtube.com/watch?v=CTxnLsYHWuI&t=964s) 技术遏制困境与“多用途”扩散 史蒂芬抛出了一个沉重的问题：我们真的有可能遏制（contain）AI 吗？历史表明人类极少能永久封禁一项技术。如同铁锤既能用来建房也能用来杀人，一旦某项技术被发明，由于博弈对抗的存在（如果对方拥有核武器而我们没有，我们就会处于劣势），它总会渗透进人类社会。 苏莱曼明确表示，单纯用“乐观”或“悲观”的框架来看待这个问题都是偏颇且有害的。从历史上看，遏制或彻底封禁这一类技术的先例极少。人类曾成功禁绝过氟氯烃（CFCs，因其破坏臭氧层），也签署条约禁止了化学武器、生物武器以及致盲激光（blinding lasers）。但这些成功案例的前提通常是：市场上存在成本相当或更廉价、且同样有效的替代品。 ```mermaid flowchart TD accTitle: AI 技术的全用途扩散逻辑 accDescr: 展示 AI 从底层通用技术到军民两用分化，再到商业驱动全面扩散的闭环逻辑。 subgraph CoreTech["底层通用智能"] A["核心模型能力<br/>(Omni-use / 全用途技术)"] end subgraph DualUse["应用场景分化"] B["民用医疗领域<br/>识别胸片癌变肿瘤"] C["军事国防领域<br/>战场空袭目标识别"] end subgraph Proliferation["扩散驱动机制"] D["巨大商业利益与社会价值"] E["地缘博弈与不可封禁性"] F["开源模型与无序扩散"] end A --> B A --> C B --> D C --> E D & E --> F ``` 而 AI 则完全不同，它是一种“全用途技术”（Omni-use technology）。同一套底层的核心算法，既能在医学影像中精准识别胸片上的癌变肿瘤造福人类，也能在战场上为军用无人机识别空袭目标进行致命打击。这种多用途性赋予了 AI 巨大的商业回报和极强的生存价值，推动其全面扩散，也给监管带来了前所未有的难题：你怎么去彻底封禁一项在明面上如此有益、却潜藏致命破坏力的技术？ 尽管如此，苏莱曼依然认为遏制在理论上是可能的。现代人类文明的基石依然是民族国家（Nation-state）。社会将权力和合法暴力垄断权让渡给了国家并向其缴税；如今，民族国家必须迅速自我迭代升级，建立起严密有力的监管体系。这不仅包括对闭源前沿研发公司的审查，更必须延伸至开源生态（Open Source），否则无序的扩散将不可避免。 - **历史封禁先例**：氟氯烃（破坏臭氧层）、化武与生武、致盲激光武器。 - **全用途特性（Omni-use）**：单一技术兼具拯救生命的医疗用途与精准杀伤的军事用途。 - **监管抓手**：依托民族国家的合法管辖权，将闭源厂商与开源社区同时纳入监管视线。 --- ### [19:31](https://www.youtube.com/watch?v=CTxnLsYHWuI&t=1171s) 2050 展望与“假装不信”的悬置状态 史蒂芬将讨论拉回到现实视角：对于伯明翰开 Uber 的普通打车司机戴夫（Dave）来说，大众对 AI 的认知仅限于“能帮孩子写作业的聊天机器人”。如果完全不加以遏制，30 年后的 2050 年世界究竟会变成什么样？ 苏莱曼毫不犹豫地指出，到 2050 年，AI 不仅会像人类一样在数字世界中行动自如，物理世界也将被具身机器人（Robots）彻底重构。更震撼的是，届时将会诞生全新的“生物合成实体”（new biological beings）。因为过去数十年间在硬件与软件层面发生的指数级进化轨迹，必然会全盘复制并降临在生物学（Biology）这一平台上。 注意到苏莱曼说话时抱住了双臂，史蒂芬立刻指出了这个肢体语言背后的不适感。史蒂芬提到自己曾在录制《龙穴之创业投资》（*Dragons' Den*）时给其他投资人展示过埃隆·马斯克（Elon Musk）的一段访谈：当马斯克被问及在这个由 AI 统治的不可避免的未来该如何给自己的孩子提供人生建议时，罕见地陷入了长达 20 秒的死寂与结巴，并坦言自己正处于一种“假装不信的悬置状态”（suspended disbelief）——即如果深究下去，连自己眼前奋斗的意义都会彻底崩塌，因为 AI 的威胁甚至超越了核武器。 苏莱曼对马斯克的这句“假装不信的悬置状态”深表赞同，并承认自己刚才的微动作确实是因为推演到深处的毛骨悚然。他总结道，人类必须直面那些极度黑暗的灾难性概率： > “智力（Intelligence）将成为一种全新的资本形式。” 就像人类历史上曾疯狂争夺土地、争夺石油一样，任何能让人“用更少资源、更快、更好、更聪明地完成任务”的工具，都会引发国家和企业不计代价的争抢。伴随指数级提升的可预测轨迹，哪里存在权力，哪里就会有 AI 去极度放大、加速并强力助推这种权力。正因如此，当被要求去凝视 2050 年的终局时，即便是置身技术浪潮核心的先驱者，也不禁感到战栗与苦笑。 ### [23:32](https://www.youtube.com/watch?v=CTxnLsYHWuI&t=1412s) 人类主导地位与相邻技术：机器人与合成生物学的演进 Mustafa Suleyman 严肃地强调，当前的人工智能给人的感觉切切实实像是一个“新物种”，而这个物种必须受到严格掌控。人类绝不能容许自己在地球上的主导地位被剥夺或撼动，这是底线中的底线。当主持人 Steven Bartlett 追问随 AI 一同崛起的相邻技术时，Suleyman 指出，机器人和生物学正在平行演进，共同构成了这股庞大的技术浪潮。 虽然过去几十年来机器人技术的宏大构想尚未彻底照进现实——目前人类最拿得出手的成果不过是无人机和尚在发展中的自动驾驶汽车——但机器人正在沿着与其他技术相同的指数轨迹加速演进。Suleyman 预计在未来 30 年内，人形机器人必然会走进日常生活系统，以高度可靠的能力替人类承担繁重的实体劳动。而在生物学领域，变化则更为惊人：相较于 2000 年人类首次完成基因组测序，如今测序的成本已经骤降了整整一百万倍；人类不仅能读取基因，更能越来越轻松地合成制造全新的 DNA 片段，从生命最核心的代码层面去删除不想要的缺陷或植入新特质——例如让水果保存更久，或者让合成人造肉具备更高的蛋白质含量。 - **测序成本断崖式下跌**：基因组测序成本在过去约 20 年间降至最初的百万分之一（$1/1,000,000$）。 - **物理与数字现实融合**：人形机器人与合成生物学均与 AI 呈现出协同共进的同一技术轨迹。 --- ### [25:51](https://www.youtube.com/watch?v=CTxnLsYHWuI&t=1551s) 合成病原体与风险遏制：前置预防原则与限制访问 谈及合成生物学的潜在后果时，Suleyman 描绘了一幅极其严峻的阴暗图景：一旦有人开始恶意或意外地摆弄合成病原体，完全有可能制造出传染力呈几何级暴增、致死率极高的人工工程化超级病毒，引发前所未有的全球大流行。正是这种潜在毁灭性风险，使得“技术遏制”（Containment）成为当务之急。人类必须像管制危险材料那样管制这些工具和专有诀窍（Know-how）——就像炭疽杆菌绝不可能在互联网上公开贩售或任由个人随意实验一样，前沿 AI 与生物合成工具绝不能处于无监管的放任状态。 这意味着监管机构必须从根源上收紧准入权限：限制核心算力（Compute）、限制驱动模型的底层软件、限制对外提供实验接口（API）的云环境，并在生物端严控合成原料与前体物质。Suleyman 直言，这种强监管立场必然会引发巨大反弹，那些心怀善意的创业团队、底层独立开发者会因为繁重的合规负担而倍感愤怒与挫败，但他直接以极其强硬的口吻回应道：“这就是我们所处的时代，认清现实并接受它吧（Deal with it）。”他强调，人类历史上从未在某项技术发明之初就主动要求“放慢脚步、先确保不造成伤害”，但面对未来 5 到 10 年随时可能脱缰失控的技术，业界必须恪守“前置预防原则”（Precautionary Principle），甚至宁可主动舍弃一部分短期潜在收益，也必须换取技术始终处于受控与服务人类集体利益的安全边界之内。 ```mermaid flowchart TD accTitle: 技术遏制框架与关键防线 accDescr: 展示从上游基础设施到终端分发的层级遏制机制 subgraph Infrastructure["上游核心资源管控"] C["算力硬件限制 (Compute)"] S["底层模型软件与权重监管"] B["生物合成原料与试剂管控"] end subgraph Distribution["服务分发通道"] API["云端环境与商业 API 接口"] end subgraph Defense["防护目标"] Safeguard["防止工程化病原体与致命灾难<br/>恪守前置预防原则 (Precautionary Principle)"] end C --> API S --> API B --> Safeguard API --> Safeguard ``` - **严控四大关键节点**：算力供给、核心算法软件、云端开放 API、生物合成实体原料。 - **前置预防原则的核心取舍**：与其冒险释放无法收回的潜在灾难，不如宁可放弃部分技术红利，优先保证安全。 --- ### [28:44](https://www.youtube.com/watch?v=CTxnLsYHWuI&t=1724s) 竞生条件（Race Condition）与核不扩散的历史经验 Steven Bartlett 随即提出反驳与担忧：如果 AI 能够根据需求定制特性——比如设计一款能随时间定向变异、只针对特定人群或特定规模的“生化核弹”，那么面对美国可能采取的审慎预防态度，像中、俄这样的竞争对手很可能会选择加速狂飙；为了防止受制于人，各国军方必然会陷入“防患于未然”的军备竞赛。Suleyman 承认这正是科技博弈中经典的“竞生条件”（Race Condition）——“如果我不做，对方就会做；所以为了不落后，我必须先做”。 然而，Suleyman 犀利地指出，这种思维定势最致命的地方在于它会沦为一种“自我实现的预言”，最终逼迫所有人走向最坏的结局。他坚信国际间存在巨大的合作空间，因为无论是中美两国还是其他被视为假想敌的阵营，在维护人类整体生存、健康与福祉上拥有不可分割的共同利益。他以冷战以来的历史经验为证：《核不扩散条约》在很大程度上取得了令人瞩目的成功，全球至今仅有 9 个拥核国，国际社会甚至成功激励或迫使 3 个国家彻底放弃了核武库，恐怖分子等极端小型组织也始终未能染指核武。尽管 Bartlett 提醒核武器依赖昂贵且无法在后院棚屋里存放的实体原料（如浓缩铀-235），而 AI 却完全是一串能够轻易跨国界传播的代码，但 Suleyman 依旧认为，全球协作并非痴人说梦。 - **竞生条件的恶性循环**：对对手抢先研发的恐惧，反向逼迫己方加速投入，从而使军备竞赛成为必然。 - **核不扩散的借鉴价值**：目前全球公认的拥核国家仅维持在 9 个，证明国际机制在限制终极武器扩散上曾发挥过关键效力。 --- ### [31:34](https://www.youtube.com/watch?v=CTxnLsYHWuI&t=1894s) AI 开源浪潮下的扩散危机：30年后的遏制豪赌 Suleyman 坦承，AI 确实具备与核材料截然不同的扩散特性。当前行业正处于一种诡异的分裂状态：一方面，Google、OpenAI 以及他自己创办的 Inflection 正在研发耗资数亿美元、仅有极少数巨头能负担的最前沿模型；但另一方面，仅仅几年前属于绝对顶尖的技术，如今却已迅速被彻底开源。以 2020 年夏天轰动一时的 GPT-3 为例，如今其模型架构、训练权重与实现代码全部在互联网上公开可用，更关键的是，开源复现版本不仅性能优异，其体积甚至比原版缩小了 60 到 70 倍，使得普通开发者或远在俄罗斯的年轻人坐在卧室电脑前，就能低成本运行并利用这些强力工具。技术的天然轨迹正是“变得越来越高效、越来越廉价、扩散得越来越深远”，这也正是其新书核心探讨的议题：未来 30 到 50 年人类最大的挑战就是“遏制”（Containment）与对抗泛滥扩散。 对话的气氛在此时带上了一丝幽默却又深沉的哲思。Bartlett 调侃道，除非 30 年后两人被机器人终结，或者播客主持人的饭碗彻底被 AI 抢走，否则 30 年后届时 60 岁的 Bartlett 和 68 岁的 Suleyman 还会坐在这里对谈，届时人类是否真的能完成这一全球遏制壮举？Suleyman 毫不犹豫地回答：“我们必须做到，我甚至无法想象我们失败的后果。”他自认是一个彻头彻尾的人文主义者，坚信人类必须主动做出选择，把捍卫我们自身物种的生存与主导权放在首位。他直言，尽管未来有可能造出完全可证明其永远服从人类控制的 AGI，但在没有百分之百把握之前，人类绝对不能拿全人类的命运去进行这场豪赌；一旦失去遏制，总会有人怀着狂妄的好奇心想去窥探“门背后究竟有什么”，从而彻底打开无法关上的潘多拉魔盒。 - **开源模型的断崖式轻量化**：具备类似 GPT-3 级别能力的模型已被开源复现，体积相较于原始版本缩减达 60 至 70 倍。 - **人文主义的核心立场**：拒绝拿人类物种的控制权进行技术对赌，遏制技术的无序扩散是全球网络化时代关乎生存的终极防御战。 ### [35:17](https://www.youtube.com/watch?v=CTxnLsYHWuI&t=2117s) 为什么超越人类的超级智能会想要与我们互动？ 在网络化与全球化的世界中，任何微小的举措都会瞬间呈指数级扩散，无论当事人的初衷是否良善。主持人 Steven Bartlett 由此引出一个深刻的疑虑：穆斯塔法·苏莱曼（Mustafa Suleyman）在预测未来 30 年的技术图景时，曾提到人类将与一个拥有人类水平甚至超越人类水平智能的“新物种”互动；但 Bartlett 认为，指望这种新物种愿意与人类互动，完全是人类的一厢情愿。 Bartlett 用自己饲养的法国斗牛犬巴勃罗（Pablo）做了生动的对比：他与爱犬之间的智商差距并不算悬崖式的断层；哪怕是与人类 DNA 仅差约 1% 的猩猩，行为也表现为到处扔粪便，而人类却坐在演播厅里向全球广播。正如苏莱曼在其著作《即将到来的浪潮》（*The Coming Wave*）第四章中所指出的，AI 的潜在智能上限目前看不到尽头。如果未来的超级智能在算力和认知维度上比人类高出成百上千倍，它为何要在意人类？ ```mermaid flowchart TD accTitle: AI目标函数设计与物种互动挑战 accDescr: 展示如果AI目标函数设计不当或失控，将导致人类与AI物种间产生断层。 subgraph Design["AI 目标函数设定"] A["设计以人类福祉为中心的目标函数"] B["未设严格边界 / 赋予无限制自我迭代"] end subgraph Outcome["演化分歧路径"] C["AI 尊重并理解人类，服务于人类物种"] D["AI 产生超常能力 / 行为脱离人类控制与认知"] end A --> C B --> D ``` 对此，苏莱曼回应称，这一切核心取决于人类的底层设计。AI 遏制（containment）的核心挑战之一，正是确保我们设计出的 AI 是**我们想与之互动、且它也愿意与我们互动**的系统。如果给 AI 设定的目标函数（objective function）在底层架构中就忽视或贬损了人类及其目标，AI 必然会跑偏并做出怪异举动。 ### [37:12](https://www.youtube.com/watch?v=CTxnLsYHWuI&t=2232s) 遏制的核心：在悬崖勒马时学会“说不” Bartlett 紧接着质疑，即使初始设计良好，AI 一旦具备自我复制或迭代能力，就如同孩子长大后智商暴涨千倍。在面对思维、甚至具备某种感知力、计算速度超越人类脑肉躯壳数千倍的崭新物种时，人类凭什么相信自己微不足道的智慧还能对其产生约束？ 苏莱曼指出，这恰恰切中了“遏制问题”的要害： - **审慎原则（Precautionary Principle）**：如果人类无法确信 AI 会尊重、理解并服务于人类整体，就绝不应该承担这种在实验和设计上的极端风险。 - **自我改进的上限失控**：如果任由 AI 无限吸纳全球算力与海量数据，它在未来 30 到 50 年内必将彻底超越人类的干预能力。 - **学会“说不”的社会肌肉**：人类文明在历史上从未真正锻炼过这种自我克制的“肌肉”。整个社会的商业、政治、地缘竞争本能始终是冲撞、对抗、扩张、追求利润与击败对手。 - **摒弃地缘政治恐慌**：苏莱曼抨击了当前弥漫在西方文化中的敌意与他者化（othering）思维。过去人们整日担忧极端恐怖主义，如今又集体被引导去渲染对中国的极度恐慌；这种陷入“冲突与对抗”的博弈心理，正是促使人类在 AI 军备竞赛中走向自毁的根本诱因。 ### [40:16](https://www.youtube.com/watch?v=CTxnLsYHWuI&t=2416s) 第 37 手棋的启示与创造力的双刃剑 Bartlett 回顾了自己对 AI 认知发生范式转变的关键时刻。过去大众普遍将 AI 当作被驯养的孩子，认为工程师只是写好规则告诉它国际象棋每一步该怎么走。然而，DeepMind 研发的 AlphaGo 彻底打破了这一认知： - AlphaGo 通过自我博弈训练，击溃了当时的世界围棋冠军（李世石）。 - 在第 4 局（实为对局系列中著名的**第 37 手棋**）中，AlphaGo 下出了令所有人匪夷所思的一步。人类职业解说员当时脱口而出“这绝对是个失误”，结果复盘证明这竟是超越人类千百年围棋定式的神之一手。 - Bartlett 感叹，在这一刻他彻底放弃了“像训练小狗坐下、握手那样控制 AI”的天真幻想——当系统做出了人类不仅无法预测、甚至事后都难以理解的决策时，人类该如何掌控？ 苏莱曼对此剖析了技术发展内生的深刻矛盾：**人类恰恰需要 AI 拥有这种不可预测的创造力**。无论是破解全新战略、提出未有过的概念，还是为不治之症寻找新疗法，面对未来 30 年全球人口变化、气候危机等生存极限，人类自身的智力与发明能力已触及天花板，迫切需要此类工具实现超级赋能。然而，正是这种赋予人类解决复杂难题的创造性，同时带来了系统做出违背人类利益行为的巨大风险。 ### [43:11](https://www.youtube.com/watch?v=CTxnLsYHWuI&t=2591s) 面对存在性危机：对抗虚无与直面矛盾 针对当前的狂热局势，Bartlett 提出质疑：包括埃隆·马斯克（Elon Musk）在内的行业领袖都深知这其中巨大的存在性危机（existential risk），而苏莱曼自己也创立了估值达 40 亿美元的 Inflection AI（旗下拥有对话模型 Pi）。在国家之间担心落后、企业受制于股东利益而拼命向前的囚徒困境下，整个行业是否都在靠“假装看不见”（suspended disbelief）来维持日常？ 苏莱曼对此作出了清醒而坚定的反思： - **正视矛盾即是智慧**：同时衡量技术带来的巨大福祉与恐怖的潜在后果，并承认其中的矛盾与残酷真相，这并非逃避，而是一种全人类迫切需要的诚实。 - **拒绝盲目乐观与虚无**：如果戴上眼罩假装风险不存在，或者盲信“人类过去总能解决问题”，社会将失去反思与行动的动力。 - **集体参与才能破局**：现阶段无论是顶尖的 AI 科学家、苏莱曼等技术专家、风险投资人，还是各国政客，没有任何人掌握着现成的标准答案。 - **《即将到来的浪潮》的初衷**：苏莱曼直言，他撰写该书的目的就是为了向全社会敲响警钟，呼吁所有人共同聚焦并参与这一命题——“去证明我是错的，证明 AI 的遏制必然能够达成”。 ### [45:55](https://www.youtube.com/watch?v=CTxnLsYHWuI&t=2755s) 纵观未来两百年：量子计算与算力的指数级爆炸 讨论开篇聚焦于一个核心执念：对技术的“遏制”（containment）必须是可能的。穆斯塔法·苏莱曼（Mustafa Suleyman）坦言自己“极度渴望它是可能的”，因为如果不相信可以遏制，人类就只能被迫接受超人类主义（transhumanism）所主张的“不可避免的演化宿命”。主持人史蒂芬·巴特利特（Steven Bartlett）顺势将时间维度拉长，不再讨论未来30年，而是追问200年后的2200年，世界究竟会演变成什么样。 面对如此宏大的时间跨度，苏莱曼直呼“这简直令人难以想象（mind-blowing）”。他指出，届时人类必然已经掌握了量子计算机（Quantum Computer）。与传统的计算架构相比，量子计算是一种颠覆性的范式，能够把过去需要数百亿次浮点运算（FLOPs）的处理工作压缩在单次计算中完成。苏莱曼以老式的 56K 拨号上网调制解调器（dial-up modem）作比——当年人们得忍受滋滋作响的拨号音，看着网页图片逐个像素吃力地加载，那是因为硬件处理速度构成了摩擦阻力；而量子计算的提速规模不是简单的十亿倍，而是数以亿亿倍（billions of billions times）的跃迁。 苏莱曼进一步描绘了数据存储与计算硬件的微型化演进轨迹： ```mermaid flowchart TD accTitle: 计算与存储硬件的历史演变与未来预测 accDescr: 展示从1960年代大型机到未来极微型智能感应设备的体积缩减与算力爆炸。 subgraph Past["历史阶段"] A["1960年代：托盘级设备<br/>需叉车搬运的庞大主机"] --> B["早期PC时代：3.5英寸软盘<br/>容量仅 1.4 MB"] end subgraph Present["当代阶段"] B --> C["当今智能手机<br/>掌中容纳相当于数十亿张软盘的算力与存储"] end subgraph Future["未来构想"] C --> D["极微型穿戴与环境设备<br/>极度廉价的冰箱贴式传感器，全天候感知、分析与优化"] end ``` 在苏莱曼看来，海量数据之所以蕴含无穷价值，是因为“当人们摸清事物之间的关联时，就能够带来效率的优化和改进”。过去25年互联网创造的一切财富都建立在这一逻辑之上；而在150至200年后，伴随着遍布环境的智能节点，人类将进入一个几乎不需要再从事传统工作的世界。 --- ### [49:48](https://www.youtube.com/watch?v=CTxnLsYHWuI&t=2988s) 边际成本归零的未来：能源革命与极端廉价的世界 除了纯粹的算力暴增，苏莱曼指出了另一个与 AI 深度绑定的关键变量——能源生产成本。当前应对气候变化的核心技术瓶颈在于电池储能技术，而 AI 有望在未来 20 至 30 年内攻克这一难题。一旦高效储能得以解决，人类就能够廉价乃至完全免费地从太阳汲取并储存接近无穷无尽的能源。 能源成本几乎内嵌于现代社会每一件商品与服务的生产价值链中。苏莱曼强调，当能源成本被抽离归零后，全社会的生产成本将断崖式下跌： - **海水淡化与农业**：廉价能源能直接驱动大规模海水淡化，使粮食耕种与高品质食品生产成本大幅降低。 - **交通出行与医疗制造**：重塑新型运输系统，将生物制药和医疗健康服务的生产成本压至近乎于零。 - **生活成本的普降**：在未来 30 至 50 年内，几乎所有基础物资和生活要素都将变得极其廉价（radically cheaper）。这种成本雪崩虽然由巨大的商业利益驱动，但它正是人类应当对未来保持巨大希望与乐观的根本原因。 --- ### [51:24](https://www.youtube.com/watch?v=CTxnLsYHWuI&t=3084s) 批驳超人类主义：意识上传与冷冻大脑的荒谬 话题随后转回超人类主义的命题。苏莱曼解释道，超人类主义者认为人类的灵魂、意识与存在本质终将超越生物载体（biological substrate）——人体的血肉躯壳与大脑仅仅是智慧的暂时容器。该群体中有部分被他戏称为“怪人（crackpots）”的拥趸坚信，人类有朝一日能将自我意识上传到硅基芯片上，让“史蒂芬”的数字化本质在2200年的某台服务器中永生。 当巴特利特追问在合成生物学（synthetic biology）、纳米技术与量子计算共同加速的背景下，为何能笃定他们是错的，苏莱曼给出了神经科学层面的质疑： - **身心二元论的破产**：苏莱曼指出，目前没有任何科学证据表明可以从大脑中单独剥离出所谓的“存在本质”；神经科学更倾向于认为心智、身体与意识是不可分割的统一体，而非独立存在的灵魂软件。 - **冷冻复活的荒诞现实**：许多信徒佩戴着“死后冷冻我（cryogenically freeze me when I die）”的类似器官捐献者的胸牌，雇佣专门的急救车在断气瞬间赶往现场，火速将大脑浸入低温冷冻罐中以期未来“重启”并传输意识。苏莱曼直言这种想法纯属“疯狂（nuts）”，在物理与生物机制上根本无法令人信服。 --- ### [54:08](https://www.youtube.com/watch?v=CTxnLsYHWuI&t=3248s) 罗威纳犬之喻：面对自主技术的遏制与敬畏 巴特利特对遏制技术的可能性提出了犀利反驳。他援引苏莱曼在其著作《即将到来的浪潮》（*The Coming Wave*）第七章中的观点：“AI 比历史上任何技术都更加自主；几个世纪以来，关于技术脱离控制、成为超越人类能动性的自导自驱力量的设想曾是科幻，但如今已不再是幻想。” 巴特利特打了个形象的比方：面对一种无需人类指令即可自主行动的高智能机器，坚称“必须能遏制它”，就像面对一只体型比自己大上千倍的庞大罗威纳犬（Rottweiler），人类仰头拿着牵引绳说“我要带你去散步”，而巨犬只会居高临下地俯视，随时能将人踩在脚下。 苏莱曼接受了这个比喻，但他给出了强有力的反驳逻辑： - **历史上的遏制先例**：人类在历史上曾多次成功遏制了远比自身强大的力量，包括驯化罗威纳犬本身、圈禁大猩猩、老虎与鳄鱼，甚至有效管控了全球大流行病原体与毁灭性的核武器。 - **文明长周期的视角**：放眼整个演化史，人类过去绝大部分时间都在奔波觅食和躲避敌人的石头，而当今是物种历史上最和平的时期，社会最大的烦恼甚至变成了“营养过剩（people eat too much）”。过去500年文明的总体趋势是寿命延长、生活品质提升与健康改善，这全赖于人类拥有遏制强力因子的能力。 - **治理范式的转变**：要成功约束 AI 这只潜在的巨兽，不能依靠高度对抗性、偏执猜忌的政治手段，而需要极高的创造力、妥协精神以及前所未有的“谦逊姿态（humble tone）”。全社会必须将遏制 AI 视作第一优先级，“因为如果我们不遏制它，它就会遏制我们（otherwise it contains us）”。 --- ### [57:08](https://www.youtube.com/watch?v=CTxnLsYHWuI&t=3428s) 现实逼近的阴影：大语言模型引发的个体网络安全危机 对话从遥远的哲学思辨骤然拉回迫在眉睫的现实风险。巴特利特将焦点转向大语言模型（LLM）所引发的新型网络安全与人身安全威胁。当技术已经能够逼真地伪造任何语音、文本与视频时，人类原有的社会信任基石正面临瓦解。 他分享了一段近期引发深切共鸣的真实视频案例：一名年轻女孩的祖母接到了诈骗电话，电话那头运用声音克隆技术完美复刻了她儿子的声音，谎称自己遭遇了严重车祸急需抢救资金，老人在惊慌失措之下差一点将钱汇出。这类生成式 AI 驱动的深度伪造（Deepfake）欺诈，正以极低的门槛将普通人推向防不胜防的信任深渊。 ### [57:50](https://www.youtube.com/watch?v=CTxnLsYHWuI&t=3470s) 信任危机与防御机制：在 Deepfake 时代重建怀疑精神 主持人指出，人类社会的基石建立在信任之上——我们相信亲眼所见、亲耳所闻。然而，随着 AI 技术的爆发，人们似乎进入了一个无论在互联网上还是电话里都无法轻信任何信息的全新时刻。面对大众的普遍焦虑，Mustafa Suleyman 直言，保持“健康的怀疑态度”不仅必要，而且在未来的重要性将远超以往。他回顾了人类适应数字技术的第一波浪潮：最初垃圾邮件和诈骗横行，大家频繁中招，但随着时间推移，人们逐渐学会识别套路、产生免疫并主动规避；如今面对层出不穷的钓鱼短信亦是如此。 然而，新一代犯罪分子利用 AI 进行操纵的手段将更加逼真。未来的克隆语音听起来与真人毫无二致，Deepfake 视频也会真假难辨。对此，防御手段必须以同样的速度进化。以银行业为例，为了抵御身份盗用，行业发明了一次性密码（OTP）短信和双因素认证（2FA）。Mustafa 预测，未来的安全防护将不得不升级为三因素甚至四因素认证，需要通过多个完全独立的信源进行交叉三角验证，而绝非给银行经理打个电话就能轻易放行资金。防守措施必须像即将到来的进攻手段一样迅速迭代，这需要汇集全社会的创造力与专注度。 - **适应历史**：人类从早期的尼日利亚诈骗邮件和短信钓鱼中建立了适应力，面对 Deepfake 亦需建立新的警惕直觉。 - **验证机制演进**：单纯的电话确认早已过时，安全体系正从短信 OTP、2FA 迈向跨多渠道独立验证的 3FA、4FA 时代。 --- ### [01:00:05](https://www.youtube.com/watch?v=CTxnLsYHWuI&t=3605s) 以 AI 制衡 AI：人机协同在博弈中的进化优势 针对“我们是否需要开发 AI 来防御 AI”这一观点，Mustafa 指出这其实已经在现实中广泛运行。日常生活中，绝大多数反垃圾邮件系统、信用卡实时欺诈交易检测，都不是人类管理员坐在屏幕前死盯着交易流水，而是依靠机器学习模型即时捕捉异常行为。在物理安防领域，现代数据中心与监控摄像头同样搭载了 AI 音频与图像追踪算法，能够主动捕捉玻璃破碎或突发异响。随着 AI 愈发强大，利用受控、健康且服务于人类利益的防御性 AI，是抵御恶意攻击的必然选择。 主持人分享了一个自己测试大语言模型的经历：他让 AI 假设一个名为“Cynthia”的人工智能企图毁灭世界，并询问应对方案，大模型给出的答案正是需要训练另一个代表人类利益的 AI 来进行对抗。但 Mustafa 补充道，过去几十年的博弈历史揭示了一个更具启发性的事实：至今为止最强大的力量始终是“人类 + AI”的协同体。即便在围棋这样看似被 AI 绝对统治的领域，最新的学术研究表明，人类棋手通过发现此前未被注意到的新策略，依然能够击败超越 AlphaGo 的顶尖围棋程序。人类并不是被动受挫，而是会迅速适应、反思质疑、发明新工具，并将 AI 化为己用。这就像人类发明眼镜一样，工具打破了生理极限，拓展了认知边界，人机交互的动态迭代正是技术演进的核心所在。 ```mermaid flowchart TD accTitle: AI Attack and Human-AI Defense Dynamics accDescr: Diagram illustrating how offensive AI threats are countered by human-machine collaboration and defensive systems. subgraph Attack["恶意攻击面 (Offensive Wave)"] A["生成式攻击<br/>Deepfake / 克隆语音 / 钓鱼渗透"] B["自动化破坏<br/>失控 AI / 网络基础设施渗透"] end subgraph Defense["防御与制衡体系 (Defensive Ecosystem)"] C["防御型 AI<br/>实时反欺诈 / 垃圾识别 / 异常监控"] D["身份交叉验证<br/>多因素认证 (3FA / 4FA)"] E["人类认知演进<br/>策略迭代 / 怀疑精神 / 规则设计"] end A --> C A --> D B --> C C --> F["人机协同优势<br/>Human + AI Centaur"] E --> F D --> F ``` - **反欺诈自动化**：实时金融风控系统与安防摄像头声响识别均由专用机器学习算法驱动。 - **围棋逆转案例**：2023 年的研究论文证实，人类棋手针对先进围棋 AI 寻找盲区与反向策略，成功战胜了超越 AlphaGo 水准的最强 AI 系统。 - **眼镜隐喻**：眼镜让人类在视力衰退或昏暗环境下依然能阅读学习，AI 本质上也是拓展人类心智能力的认知假体。 --- ### [01:03:38](https://www.youtube.com/watch?v=CTxnLsYHWuI&t=3818s) 身处漩涡创办 Inflection AI：用一线实践战胜未知恐惧 既然 Mustafa 深知 AI 发展失控可能带来的灾难性后果，主持人尖锐地提问：为什么你还要亲自下场创办一家 AI 前沿公司（Inflection AI）？Mustafa 坦言，这背后存在复杂的矛盾，但他深信，向世界证明如何构建安全、受控的人工智能，唯一有效的途径就是在最前沿进行真实的工程实验。如果清醒者只充当冷眼旁观的批评家或怀疑论者，主动放弃站在研发第一线，那么决定技术走向的话语权就会被拱手让给其他力量——比如不顾一切追求利润最大化的商业巨头，或是陷入零和博弈的大国对抗之中。 在 Mustafa 看来，化解对未知技术恐惧的解药是“深度理解”（Deep understanding）。人们应该亲自去体验、把玩这些模型，直视它们的弱点：它们目前远非全知全能的超人类，依然会犯大量低级愚蠢的错误，在很多方面性能十分粗糙，而且制造门槛也并非遥不可及。主持人追问，随着自己对 AI 的了解越来越深，反而感觉像“抽丝剥茧般陷入更疯狂的焦虑”，实验是否真的减轻了 Mustafa 的恐惧？Mustafa 笑着承认这是个既肯定又否定的两难命题：在未来一二十年的短中期内，深入实践让他彻底摆脱了对科幻式“人类灭绝”的恐慌；但若将目光放长到三十年以后，技术终局究竟会走向何方，依然是一个令人深感烧脑与困惑的巨大谜团。 - **创办逻辑**：不退避于前沿之外，以工程实践定义“安全可控（Safe & Contained）”的 AI 标准，防止激进参与者主导技术方向。 - **短期与长期分歧**：深入了解模型底层让人在短期内免于恐慌（因其漏洞与缺陷明显），但 30 年后的超长期演化仍极难预测。 --- ### [01:05:55](https://www.youtube.com/watch?v=CTxnLsYHWuI&t=3955s) “遏制必须成为可能”：为什么单靠政府监管远远不够 在 Mustafa 的著作《浪潮将至》（*The Coming Wave*）第 13 章中，他拟定的标题是《遏制必须成为可能》（*Containment Must Be Possible*）。主持人注意到他用的是“必须”（Must be）而非断言“已经”（Is），并问及那些将希望完全寄托于政府立法的人——例如英国首相 Rishi Sunak 召开内阁紧急应对小组（Cobra）或是出台国家层面的监管倡议。Mustafa 提到，除了英国政府的举措，欧盟正在推进庞大的《人工智能法案》（EU AI Act），美国总统 Joe Biden 也推出了白宫倡议，而他本人和团队正积极与这些政府首脑合作，协助梳理监管架构。 然而，Mustafa 强调，尽管监管不可或缺，但以为“只要一纸禁令就能踩下刹车”的想法完全是天真脱节的。最核心的困境在于：当前负责起草法案的立法者与官僚，对底层技术的运作细节缺乏足够深刻的认知。虽然过去六个月里政界正以空前的速度恶补相关知识，让 Mustafa 感到自己呼吁了十年的议题终于迎来了迟到的共鸣，但单纯的官僚监管永远无法孤立起效。要实现对 AI 的真正遏制，必须依托于一场全社会的广泛运动——需要富有批判精神与现代创造力的开发者、文化层面的自觉、以及各行各业多元人群在技术一线的探索实践。面对外界预测这一产业未来将释放出高达 15 万亿美元的惊人商业价值，巨大的经济利益更让遏制任务面临前所未有的考验。 - **国际监管动态**：欧盟《人工智能法案》（EU AI Act）、美国白宫承诺协议、英国布莱切利园人工智能安全峰会及政府响应。 - **监管的盲区与局限**：政策制定者在专业技术细节上的认知滞后，使得行政指令无法单独解决对齐与遏制难题。 - **万亿产业诱惑**：AI 产业预计将成为规模超 15 万亿美元的经济引擎，高昂的商业回报给盲目减速带来了巨大的地缘与资本阻力。 ### [01:08:30](https://www.youtube.com/watch?v=CTxnLsYHWuI&t=4110s) 短期主义政治与全球治理架构的缺位 史蒂文·巴特利特（Steven Bartlett）一针见血地指出，当前的民主体制存在着根深蒂固的短期政治博弈。他以时任英国首相里希·苏纳克（Rishi Sunak）为例做了一个假设：如果一位领导人在两年后的选举中面临下台危机，只要不能提振本国经济就会被选民赶出唐宁街，那么他绝对没有动力去放慢或限制那笔可能降临的“15万亿美元财富大礼包”。没有哪个政客愿意把价值15万亿美元的经济红利拱手让给美国、加拿大或其他竞争对手，而是必然想让其落在自己国家。除了人类长远发展这种宏大口号之外，身处4年选举周期的领导人唯一的实际冲动就是：“只要能让我眼前的经济数据好看、撑过下一次选举，哪怕只是在安全监管上做表面文章（lip service），我也绝不会停下发展的脚步。” 穆斯塔法·苏莱曼（Mustafa Suleyman）对此表示高度赞同，并指出“短期主义（short-termism）正在杀死我们”。世界的技术变革以小时乃至十年为尺度剧烈演进，而人类却被困在虚妄的4年选举周期中——新官上任刚推行政策6个月，到了第12到18个月就必须全力备战下一轮选战。当前国际社会最致命的空白，在于**没有任何一个机构对全人类的“长期技术稳定性”负最终责任**。苏莱曼提出，世界急需一个拥有强制力的“全球技术稳定机制（global technology stability function）”。该机构必须具备以下职能： 1. 制定跨越20年尺度的全球技术遏制与收敛策略； 2. 在必要时刻向技术竞赛中注入阻力（friction）； 3. 严格落实预先防范原则（precautionary principle），维持全球和平与均势。 苏莱曼感叹道，这一设想听起来极为激进，但其本质正是二战后建立的联合国安理会（UN Security Council）与世界贸易组织（WTO）等国际多边体系的延续。二战的浩劫促使人类建立了维系了70余年繁荣与相对和平的多边秩序。然而，当下的国际格局却令人担忧地向相反方向倒退：政界与舆论沉迷于亨廷顿式的“文明冲突论（Clash of Civilizations）”叙事，将中国视作注定走向正面对抗的宿敌，在非洲和中东展开零和地缘博弈。苏莱曼直言，单凭英国、欧洲或美国各自出台优良的本土法规，根本无法破解这种走向阵营对立的囚徒困境，若不建立全球协调机制，冲突必将不可避免。 --- ### [01:12:07](https://www.youtube.com/watch?v=CTxnLsYHWuI&t=4327s) 国际共识的历史教训：灾难与相互保证毁灭 针对巴特利特提出的人性与博弈论困境——“治理气候变化因短视利益而受阻，各国又怎会在能赚15万亿美元的 AI 面前主动踩刹车？”，苏莱曼给出了一个残酷而沉痛的历史总结：在整部人类现代史上，全球范围内达成深度妥协与高度一致的时刻，仅仅发生在两种极端情境之下： 1. **难以想象的浩劫与苦难的余波**：广岛与长崎的核爆、纳粹大屠杀以及第二次世界大战的毁灭性代价，用惨痛的鲜血换来了战后十年的政治共识，催生了当今国际秩序的基础结构； 2. **明确的“相互保证毁灭（MAD, Mutually Assured Destruction）”机制**：冷战时期各方深知核战争即全人类自杀，这种明确的毁灭预期迫使核大国坐回谈判桌。 ```mermaid flowchart TD accTitle: Historical Drivers of Global Consensus accDescr: Two scenarios that historically forced humanity into consensus and why AI challenges the second model. subgraph Catastrophe["情境一：历史浩劫驱动"] W2["二战 / 广岛核爆 / 大屠杀"] --> Treaties["多边国际协议与治理机制诞生"] end subgraph ColdWarMAD["情境二：相互保证毁灭（冷战）"] NuclearActors["核大国集中（仅少数几方掌握）"] --> Deterrence["自杀式反噬形成战略均势"] end subgraph AIDilemma["AI时代的致命挑战"] Spread["技术门槛降低，散播扩散极快"] --> DiffusePower["参与者众多，MAD平衡失效"] DiffusePower --> Goal["遏制核心：限制接触尖端模型的实体数量"] end ``` 巴特利特犀利地补充道，冷战之所以能够维系核威慑平衡，是因为当时实际上已经投下了核弹，且全球拥有核按钮的仅有不到十个政权实体。而 AI 技术完全不同——它并非九个掌权者关门谈判就能锁定的实体，其极具流动性与扩散性，未来可能成千上万的人都能触碰其毁灭性力量。 对此，苏莱曼回应称，技术遏制（containment）的核心攻坚点，正是**竭尽全力把接触具有存在性威胁（existential threat）技术的实体数量压缩到极小范围**。唯有将核心技术收窄在极少数关键参与者手中，人类才能重新调动冷战时期赖以维系和平的既有军事和经济遏制工具，防止超智能（Superintelligence）或通用人工智能（AGI）失控泛滥。 --- ### [01:15:18](https://www.youtube.com/watch?v=CTxnLsYHWuI&t=4518s) 遏制战略蓝图：安全、审计、供应链扼杀点与税收阻力 作为业界首个提出系统性“AI 遏制蓝图（blueprint for containment）”的专家，苏莱曼在书中提出了涵盖10大焦点的综合防线，并在对话中详细拆解了其中的关键维度： * **防御性安全 AI（Safety）**：开发专用型 AI 防御系统，以智能对冲智能，专门监控与遏制其他不可预测的 AI 系统； * **透明度与模型审计（Audits）**：建立强制性审查标准，对包括开源模型在内的所有前沿系统进行底层技术审计； * **供应链扼杀点（Choke Points）**：在关键实体物理节点切断失控扩散路径。苏莱曼破除了大众对“互联网是虚空云端”的幻觉，指出互联网的实质是横亘全球的海底电缆和实体终端，这些物理节点完全可以被监控和切断流量。而在硬件层面，核心扼杀点则是训练超级集群必不可少的图形处理器（GPU）： - 尖端大语言模型的高度依赖特种算力，属于重资本实体工业品； - 苏莱曼透露，其创立的 Inflection AI 当时斥资**超过13亿美元**，建成了当时全球第二大超级计算机集群； - 算力芯片具有极强的地理与供应链聚集度，各国政府完全具备能力并应当对尖端芯片的流通实施管制与断供。 ```mermaid flowchart LR accTitle: AI Infrastructure Choke Points accDescr: Physical choke points that enable the containment and regulation of frontier AI models. subgraph ChokePoints["关键物理扼杀点"] Cables["物理网络基础设施<br/>（海底光缆 / 流量监控出口）"] Chips["尖端GPU算力硬件<br/>（百亿级超算集群 / 先进制造）"] end subgraph Impact["遏制效应"] Cables --> FlowControl["通信流拦截与数据管制"] Chips --> AccessControl["限制尖端大模型训练准入"] end ``` 针对蓝图中的第6点——**对 AI 公司征收高额专项税以补贴社会转型与劳动力转岗培训（Reskilling）**，巴特利特提出了强烈的避税与监管套利质疑：“如果本土政府课以重税，AI 创业公司完全可以拔腿就跑，迁往迪拜或葡萄牙等避税天堂。” 苏莱曼从容回应称，税收政策的本质目的恰恰是**在体系内主动注入阻力（introduce friction），放慢技术的奔逸速度**。虽然国际监管套利确实存在，少数团队可能会外迁至新加坡或阿联酋，但在未来至少10年内，全球顶尖的智力资本、顶层科研人才和软硬件生态依然高度集中在极少数核心国家。只要主流工业国达成协同协议，就能在制度上建立起一道强大的缓冲带，为整个人类社会争取到宝贵的适应时间。 - **核心数据与披露**： - 全球潜在 AI 经济增量预估：**15万亿美元**（15 trillion dollars）； - Inflection AI 建设超级计算机集群的实际投入：**约13亿美元**（远超外界传闻的10亿美元），在建成时成为全球第二大超算集群； - 遏制体系的核心手段：通过控制 GPU 物理流通、海底光缆网络出口，结合高额专项税收调节，形成制度阻力以延缓不可控的指数级扩散。 ### [01:18:48](https://www.youtube.com/watch?v=CTxnLsYHWuI&t=4728s) 硅谷的人才聚集与大国博弈下的“协调困境” 穆斯塔法·苏莱曼（Mustafa Suleyman）回顾了自己在 2020 年从伦敦搬到硅谷并创立新公司 Inflection AI 的经历。他直言，做出这一决定的核心原因在于硅谷的顶尖人才密度高得惊人，全球最优秀的 AI 专家与软件工程师几乎都汇聚于此，在可预见的未来，这一超大城市的“算力与智力引擎”地位仍难被取代。 然而，拉长时间维度来看，这本质上是一个极为棘手的全球性“协调困境”（coordination problem）：如何让各个主权国家集体达成共识，共同减速并对技术进行围堵控制（containment）？苏莱曼指出，这与危险武器或危险生物原料的扩散如出一辙——只要有一方率先向前狂奔，其他任何单方面选择自律或放慢脚步的国家都将面临巨大地缘风险。 主持人在探讨中表达了自己的深层忧虑。他虽然认为自己是一个讲求证据的理性乐观主义者，但在综合了各方观点并深入分析苏莱曼在《即将到来的浪潮》（*The Coming Wave*）一书中详述的底层驱动力后，他发现所有激励机制（incentives）都在将人类推向“放弃遏制”的深渊。新技术的爆发如同历史上的淘金热，所有人都争先恐后地拿起铁铲抢占地盘，直到灾难性的意外后果爆发才追悔莫及。 ```mermaid flowchart TD accTitle: 抑制技术失控的协调困境 accDescr: 展示了地缘博弈和个人利益如何导致集体遏制行动破裂的逻辑图 subgraph Incentives["无休止的驱动力"] A["历史最大的经济红利"] B["对名誉、地位与历史定位的渴求"] end subgraph Dynamics["淘金热博弈"] C["单方面减速意味着落后与劣势"] D["主权国家间缺乏互信"] end subgraph Consequence["现实困局"] E["协调破裂：全速军备竞赛"] F["围堵控制（Containment）在短期失效"] end Incentives --> Dynamics Dynamics --> Consequence ``` --- ### [01:20:40](https://www.youtube.com/watch?v=CTxnLsYHWuI&t=4840s) 人性的诱惑与生物安全的历史警示 主持人引述了苏莱曼在书中第八章对“不可阻挡的驱动机制”的犀利剖析：即将到来的这波浪潮代表着人类有史以来最丰厚的经济奖赏。科学家和技术专家同样是普通人，对名誉、成功、历史地位有着强烈的执念，他们聪明绝顶、争强好胜，并在心中精心构建着自己在人类历史上的坐标。无论是 OpenAI 的萨姆·奥尔特曼（Sam Altman）、埃隆·马斯克（Elon Musk），还是苏莱曼本人，都难逃这种人类本能的驱动。更糟糕的是，行业中还有大量行事准则远不如他们的人，这些人纯粹追逐财富与权力。 面对主持人关于“是否只有等到灾难发生或出现核威慑般的‘相互保证毁灭’（MAD）局面，人类才会真正行动”的质问，苏莱曼发出了沉痛的反思与警示： - **反思功能获得性研究（Gain-of-Function）**：以新冠疫情（COVID-19）为例，哪怕不去争论它究竟是自然起源还是武汉实验室的泄漏事故，仅凭“它有可能是实验室为了增加病原体传播力而发生泄漏”这一潜在可能性，理智的世界就应当立刻全面叫停这类研究。 - **历史上的泄漏先例**：20 世纪 90 年代英国暴发的口蹄疫（Foot-and-Mouth Disease）同样发生过泄漏事故，但因为主要受害者是牲畜，并未真正促成彻底反思。 - **情绪的低谷**：苏莱曼承认此刻感到愤怒、悲伤与悲观，因为全人类连病毒起源的共识都无法达成，也未能吸取教训关闭危险生物实验室，这暴露出人类在应对科技轨迹上的极度愚钝。 --- ### [01:24:18](https://www.youtube.com/watch?v=CTxnLsYHWuI&t=5058s) 致命自主武器的红线与“必须可能”的信念 在谈到 AI 技术的具体分支时，苏莱曼明确划定了绝对不能跨越的红线：人类坚决不应该追求“致命自主性”（Lethal Autonomy）。他将自动驾驶汽车与致命自主武器进行了鲜明对比： - **自动驾驶汽车（可控物理环境）**：运行在高度受限的物理空间内，拥有 GPS 导航、极其透明的车载遥测数据和反复验证的组件，其遏制控制策略清晰、安全且可复现。 - **不可受控的自主形态（必须抵制）**：例如允许在线 AI 在完全脱离人类监管的情况下进行无底线的自我迭代进化；又如具备致命杀伤力、完全自主决策的军用机器人。战场瞬息万变，AI 每次面对的都是全新的敌人与未知的极端情境，赋予机器执行致命暴力的自主权是不可接受的。 当主持人犀利地质疑：“像你这么聪明且洞悉局势的人，怎么可能真正相信遏制是可能的？短期激励显然会压倒一切，直到 30 年后我们像被凶猛的罗威纳犬拴在链子上一样彻底丧失控制权。”苏莱曼给出了全场最掷地有声的回答： > “我并没有说遏制是可能的，我说的是它‘必须是可能的’（It must be possible）。” 苏莱曼解释道，他的新书第一章名为《围堵是不可能的》，而最后一章则是《围堵必须成为可能》。这绝非天真的乐观主义，而是人类别无选择的生存底线。谈到自己的实际行动，苏莱曼表示自己几乎将全部精力倾注于此——不仅在技术上深入探索大模型的边界、安全性与伦理，更在公司治理上探索新型架构，例如将 Inflection AI 注册为共益企业（Public Benefit Corporation, PBC），以在法律结构上摆脱单纯追求股东利润最大化的绑架，确保安全与社会责任被置于商业利益之前。 ### [01:28:35](https://www.youtube.com/watch?v=CTxnLsYHWuI&t=5315s) 公益公司（PBC）与企业激励机制的革新 穆斯塔法·苏莱曼（Mustafa Suleyman）探讨了重塑商业组织形式的必要性，并以他创立 Inflection AI 时采用的公益公司（Public Benefit Corporation, PBC）架构为例。在这种法律框架下，公司不仅对股东的利润负责，还在法律上被明确要求平衡盈利与公司行为对外部世界造成的影响——涵盖环境保护、员工福祉，乃至那些甚至从未使用过公司产品的普通大众。在苏莱曼看来，这是企业治理结构演进中极其重要的新方向，迫使企业主动且负责任地“做正确的事”。 如果回顾历史，昔日的烟草巨头或石油巨擘之所以肆无忌惮，很大程度上是因为其公司章程只激励董事会最大化资本回报，无需为社会代价承担法律受托责任。若当时的企业章程规定董事必须在追求利润与推动世界向好之间维持对等的平衡，工业发展或许会走上一条截然不同的道路。苏莱曼坦言，这种法律形式的变革虽然谈不上万灵药，无法一劳永逸地解决所有技术伦理冲突，但它代表了一种改变社会组织激励机制的渐进式创新，是迈向正确轨道的坚实一步。 ```mermaid flowchart TD accTitle: 传统企业与公益公司治理结构对比 accDescr: 对比传统资本驱动公司与公共利益公司（PBC）在受托责任和决策目标上的核心差异 subgraph Traditional["传统企业模式"] A["资本驱动"] --> B["单一受托责任: 股东利润最大化"] B --> C["外部性忽视 (环境、社会与非用户代价)"] end subgraph PBC["公益公司 (PBC) 模式"] D["双重使命驱动"] --> E["法律强制平衡: 商业利润 + 公共利益"] E --> F["全面考量生态、用户福祉与非用户影响"] end ``` ### [01:29:57](https://www.youtube.com/watch?v=CTxnLsYHWuI&t=5397s) 身处前沿的情感重量：筋疲力尽与振奋并存 当主持人问及他是否会为此感到悲伤或沉重时，苏莱曼毫不犹豫地承认了这一点。在 AI 前沿深耕十余年，直面这股即将颠覆世界的巨浪，在情感上是一份巨大的心理负担。每天思考遥远的未来、审视人类整体的弱点和制度失灵，常常伴随着强烈的挫败感。苏莱曼谈到，人类甚至无法从刚刚过去的全球新冠疫情中吸取基本教训，各国政府运转迟缓、巨头企业充耳不闻，整个文化圈层整日沉溺于琐碎微小的议题互相倾轧，却对眼前扑面而来的巨大危机视若无睹。 作为少数深刻洞悉这项技术演进路径的先行者，苏莱曼背负着超乎常人的责任感，但他更倾向于将其视作一种罕见的特权与机遇。主持人引用了百老汇音乐剧《汉密尔顿》（Hamilton）中的名句“历史正注视着你”（History has its eyes on you），苏莱曼感同身受地表示，那种沉重的注视感确实时刻存在。当被问及内心是否感到快乐时，他坦承自己的情绪处于“筋疲力尽与振奋刺激的等量交织”中： - **短期的振奋**：看到每天有数百万人将这些超级工具装进口袋，AI 正在为普通人提供情绪支持、解答疑问、提升认知，让人类变得更聪明、更健康、更充实。 - **长期的疲惫**：多年来不断向外界疾呼潜在风险却难以形成共识。直到 ChatGPT 爆发后的六个月里，他才感到一丝真正的释怀——因为这场浪潮终于全面降临，AI 风险不再只是深埋于 DeepMind 或 OpenAI 少数研究员脑海中的孤岛担忧，而变成了全人类共同面对的公共议题。 ### [01:33:40](https://www.youtube.com/watch?v=CTxnLsYHWuI&t=5620s) 现实世界与平行数字空间：元宇宙的真正本质 主持人翻开苏莱曼的新作《即将到来的浪潮》（The Coming Wave）第 284 页（全书正文的最后一页），念出了一段极具冲击力的断言：“人类可能不再像过去那样是地球的主导驱动者。我们将生活在一个多数日常互动不再是与其他人、而是与 AI 展开的纪元。”苏莱曼对此笑着点头，指出这种转变绝非科幻构想，现实中人们早已身处其中。 他让主持人反思每个人每天盯着屏幕的时间——不论是智能手机、iPad 还是电脑显示器，往往长达 12 个小时，而凝视亲友眼睛的时间却少得可怜。过去几年科技界对“元宇宙”（Metaverse）的讨论存在严重的误判与概念偏离，人们总是狭隘地把元宇宙描绘成一个“在别处的”、需要戴上头显以虚拟化身在里面蹦跳交谈的 3D 虚拟空间。苏莱曼认为这种理解完全本末倒置：真正的元宇宙其实早就降临了，它就是与人类日常生活完全平行运行的数字空间。无论是社交平台上的信息流互动、YouTube 上的视频，还是当前录制的这期播客，都是超越物理空间的数字交互网络。这个平行的数字维度正在与物理现实深度嵌套，并逐渐由 AI 扮演核心的交互界面。 ### [01:35:44](https://www.youtube.com/watch?v=CTxnLsYHWuI&t=5744s) 面对即将来临的浪潮：年轻人与普通人该如何自处 面对这个被庞大算力与人工智能彻底重构的世界，如果一个年轻孩子走上前询问自己的人生该投向何处，苏莱曼给出的核心信条是：“知识就是力量。”他建议年轻一代： - **主动拥抱与理解**：不要在感到恐惧时转过头去假装看不见，必须正视技术的深远影响。 - **尽一切可能去参与和塑造**：理解技术的底层逻辑，参与到这一历史进程的塑造中，因为浪潮的到来不可逆转。 针对普通公众如何在“技术围堵与遏制”（Containment）的战役中贡献力量，苏莱曼强调无需给出一份刻板的教条清单。每个人都应当去阅读、去倾听、去动手使用并尝试构建这些 AI 工具，了解当下的监管政策进展，关注那些倡导技术安全与伦理的行动组织。他建议人们在日常生活中多加探讨——在酒吧聚会时聊聊这些议题，问问父母在与智能音箱交互时的真实体验。“保持关注与警觉就足够了”，凭借公众的创造力与独立思考，只要大家不再漠不关心，每个人自然会看清自己在时代中所能扮演的角色与责任。 ### [01:37:37](https://www.youtube.com/watch?v=CTxnLsYHWuI&t=5857s) 控制的胜败抉择：如果成功，我们将迎来“极致丰裕” 在讨论的收尾阶段，主持人抛出了核心问题：如果在 AI 的有效遏制与控制上人类成功了会怎样？如果失败了又会怎样？苏莱曼满怀信心地描绘了成功控制 AI 后的壮丽图景：这将是整部人类物种史上“生产力最爆发、最具唯才是举特征（meritocratic）的辉煌时刻”。 当超级智能以极低成本普惠至数亿甚至数十亿人口时，人类的创造力与问题解决能力将获得空前放大。在未来三十年内，人类有望彻底攻克最棘手的社会难题： - **能源革命**：将能源的生产、储存与分配成本降至接近零边际成本。 - **食物与医疗**：极大幅度降低健康食物的生产门槛，彻底重塑医疗护理系统。 - **交通与教育**：交通出行全面优化，优质教育资源零门槛向全球所有人敞开，形成全方位的“极致丰裕”（radical abundance）。 面对主持人的质疑——“当物质极大丰富，人类是否会失去生活的意义感、目标感与必要的奋斗挣扎”，并举出西方在食物过剩后饱受肥胖困扰的例子，苏莱曼乐观地表示，比起毁灭性的技术失控灾难，为“如何打发丰裕的一天”、“如何重新寻找生活意义”而烦恼，绝对是一个无比美妙的“高级烦恼”（a good problem to have）。人类天生会寻找新的探索旅程与挑战，彻底将普罗大众从千百年来沉重繁琐的生计劳作中解放出来，让他们能够自由追寻内心的热爱与兴趣，正是人类技术演进最值得追求的终极宿命。 ### [01:39:53](https://www.youtube.com/watch?v=CTxnLsYHWuI&t=5993s) 走向极端丰裕：重新定义全民基本收入与权力扩散 长期以来，穆斯塔法·苏莱曼（Mustafa Suleyman）一直是全民基本收入（UBI）的坚定支持者，但他认为未来的形态并不会简单停留在“政府每月给每个人发支票”。真正的变革在于通过先进科技从根本上降低基础商品的生产成本，使人们不再完全依赖赚取收入来维持生存。他设想了一种近乎免费的能源与食物体系：凭借廉价甚至近乎免费的清洁能源，人们可以利用海水淡化与基因改造后的适应性种子在沙漠中种植粮食。这种技术突破将彻底重塑城市与国家的运行架构，使个体或小群体即使脱离传统中心化系统的支持，也能独立生活相当长的时间。 虽然苏莱曼坦承自己并非自由意志主义乌托邦（Libertarian wet dream）的信奉者，但从理论视角来看，这种底层逻辑极其引人入胜。这正是他所定义的“权力的扩散”（Proliferation of Power）——权力不仅意味着能够获取超级智能，更意味着普通人拥有掌控自身命运的工具，可以按照自己的愿景去创造生活的意义与目标。在他眼中，一个极具创造力、人人共享的“极端丰裕世界”（A world of radical abundance），就是技术发展最为理想的成功图景。 ```mermaid flowchart TD accTitle: 极端丰裕与权力扩散图示 accDescr: 科技降低基础成本带来社会结构转变并赋能个体 subgraph TechEnablement["技术突破与极端丰裕"] A["近乎免费的清洁能源"] --> B["海水淡化与基因适应种子"] B --> C["基础生活成本暴降（食物与能源）"] end subgraph SocialShift["社会结构重组与赋能"] C --> D["摆脱单一收入依赖"] D --> E["城市与国家中心化结构解构"] E --> F["个体掌握自我命运与生活意义"] end ``` ### [01:41:11](https://www.youtube.com/watch?v=CTxnLsYHWuI&t=6071s) 控制失败的代价：全球网络化世界中的失控危机 与技术带来的丰裕相对立的，则是技术遏制（Containment）的全面失控。苏莱曼指出，未能实现丰裕世界固然令人遗憾，但技术治理最致命的失败在于“无法遏制扩散”。一旦毁灭性力量发生大规模扩散，那些心怀叵测的恶意行为者（Bad Actors）将不可避免地获得强大的破坏性手段。 在一个高度网络化、全球互联的世界中，这一挑战被成倍放大：少数极端个体或边缘团体，只需借助瞬时、大规模起效的尖端科技工具，便能在转瞬间对成千上万的人造成毁灭性打击。苏莱曼严肃地强调，技术遏制的核心本质，正是要构筑坚固的防线，阻止少数恶意分子获得彻底破坏全球秩序与文明稳定的极端手段。 - **丰裕与遏制的双重挑战**：成功是让每个人共享算力与技术红利，失败则是先进技术的毁灭潜能彻底外溢。 - **不对称破坏力**：全球互联放大了个体威胁，少数恶意群体获取失控的技术工具，将带来系统性的全球灾难。 ### [01:42:15](https://www.youtube.com/watch?v=CTxnLsYHWuI&t=6135s) 终极追问：心底最神圣宁静的角落 节目进入了《CEO日记》（The Diary Of A CEO）的标志性收尾传统：由上一位未曾谋面的嘉宾为下一位嘉宾留下一个匿名问题。本期留给苏莱曼的问题是：“在你的生命中，哪一个空间或地方是你认为最神圣的？” 面对这个颇具灵性意味的提问，苏莱曼沉思后回忆起童年的一段往事。他脑海中最美丽、最神圣的地方，是幼时与家人在英格兰湖区（Lake District）的温德米尔湖（Lake Windermere）游玩的经历。当时年幼的他坐在湖面上的一艘小橡皮艇（dinghy）上，那份无与伦比的宁静、壮美与祥和深深烙印在记忆深处。尽管自那之后他再未重访该地，但那片湖水成了他在喧嚣现实世界中的精神对照，他甚至半开玩笑地感叹，在如今这个高速动荡的世界里，自己真应该回到温德米尔湖去彻底放松一下。 ### [01:43:11](https://www.youtube.com/watch?v=CTxnLsYHWuI&t=6191s) 对话尾声：《大潮将至》的现实警示与赞誉 节目主持人在结语中对穆斯塔法·苏莱曼的新书《大潮将至：技术、权力与二十一世纪的最大难题》（*The Coming Wave: Technology, Power, and the Twenty-First Century's Greatest Dilemma*）给予了极高评价。主持人特别赞赏苏莱曼在书中并未迎合舆论、不偏袒任何极端的立场来博取关注，而是以极其平衡、精炼且充满智慧的笔触，全面剖析了人类当前面临的重大技术抉择，并切实给出了具有实操性的治理方案。 主持人提到，自己在节目录制前还与播客导演杰克（Jack）交流过，该书将原本高深、小众的纳米技术（Nanotechnology）、生物技术（Biotechnology）、量子计算（Quantum Computing）以及机器人技术（Robotics）与人工智能的交汇融合解释得通俗易懂，深刻揭示了这些前沿技术交织产生的无限机遇与巨大风险。苏莱曼对这次跨越科技、哲学与未来的广泛对话表达了诚挚感谢，盛赞整场交流极其精彩过瘾，为这场关于人类命运与智能未来的深度对谈画上了圆满句号。随后，播客进入了官方赞助商 Airbnb 的商业口播环节。 ## Notable quotes > [00:00](https://www.youtube.com/watch?v=CTxnLsYHWuI&t=0s) it must be possible — Mustafa Suleyman > [00:00](https://www.youtube.com/watch?v=CTxnLsYHWuI&t=0s) otherwise it contains us — Mustafa Suleyman > [02:11](https://www.youtube.com/watch?v=CTxnLsYHWuI&t=131s) petrified — Mustafa Suleyman > [12:19](https://www.youtube.com/watch?v=CTxnLsYHWuI&t=739s) Containment is not possible — Mustafa Suleyman > [19:31](https://www.youtube.com/watch?v=CTxnLsYHWuI&t=1171s) 智力（Intelligence）将成为一种全新的资本形式。 — Mustafa Suleyman > [25:51](https://www.youtube.com/watch?v=CTxnLsYHWuI&t=1551s) Deal with it — Mustafa Suleyman > [01:08:30](https://www.youtube.com/watch?v=CTxnLsYHWuI&t=4110s) 短期主义（short-termism）正在杀死我们 — Mustafa Suleyman > [01:24:18](https://www.youtube.com/watch?v=CTxnLsYHWuI&t=5058s) 我并没有说遏制是可能的，我说的是它‘必须是可能的’（It must be possible）。 — Mustafa Suleyman > [01:29:57](https://www.youtube.com/watch?v=CTxnLsYHWuI&t=5397s) History has its eyes on you — Steven Bartlett > [01:33:40](https://www.youtube.com/watch?v=CTxnLsYHWuI&t=5620s) 人类可能不再像过去那样是地球的主导驱动者。我们将生活在一个多数日常互动不再是与其他人、而是与 AI 展开的纪元。 — Steven Bartlett > [01:35:44](https://www.youtube.com/watch?v=CTxnLsYHWuI&t=5744s) 知识就是力量。 — Mustafa Suleyman > [01:37:37](https://www.youtube.com/watch?v=CTxnLsYHWuI&t=5857s) a good problem to have — Mustafa Suleyman ## Mentioned Mustafa Suleyman, Steven Bartlett, Elon Musk, Sam Altman, Rishi Sunak, Joe Biden, DeepMind, Inflection AI, Pi, Google, OpenAI, LaMDA, ChatGPT, Google Bard, AlphaGo, The Coming Wave, The Diary Of A CEO, Dragons' Den, Atari, Space Invaders, Pong, GPT-3, EU AI Act, 联合国安理会 (UN Security Council), 世界贸易组织 (WTO) ## Video description If You Enjoyed This Episode You Will LOVE This One With Mo Gawdat: https://youtu.be/bk-nQ7HF6k4?si=Xn9WWUB2nca77Jd9 00:00 Intro 02:11 How do you feel emotionally about what's going on with AI? 09:17 What's surprised you most about the last decade? 12:51 I'm scared of this coming wave. 16:04 Is containment possible? 23:53 What will these AI biological beings look like? 27:08 Would we be able to regulate AI? 33:10 In 30 years' time, do you think we would have contained AI? 35:43 Why would such a being want to interact with us? 46:35 Quantum computers & their potential 57:04 Cybersecurity 01:03:38 Why did you build a company in this space knowing the problems? 01:05:55 Will governments help us regulate it? 01:15:29 What do we need to do to contain it? 01:30:10 Do you feel sad about all of this? 01:34:04 Well slowly move more toward AI interactions over human ones. 01:36:01 What should young people be dedicating their lives to? 01:37:53 What happens if we fail in containment, and what happens if we succeed? 01:42:31 The last guest's question Are you ready to think like a CEO? Gain access to the 100 CEOs newsletter here: ⁠https://bit.ly/100-ceos-newsletter You can purchase Mustafa’s book, ‘The Coming Wave: Technology, Power, and the Twenty-First Century's Greatest Dilemma’, here: https://amzn.to/3Qudl2Z Follow Mustafa: Twitter: https://bit.ly/45FZ0qr Join this channel to get access to perks: https://bit.ly/3Dpmgx5 My new book! 'The 33 Laws Of Business & Life' pre order link: https://smarturl.it/DOACbook Follow me:  Instagram: http://bit.ly/3nIkGAZ Twitter: http://bit.ly/3ztHuHm Linkedin: https://bit.ly/41Fl95Q Telegram: http://bit.ly/3nJYxST Sponsors:  Huel: https://g2ul0.app.link/G4RjcdKNKsb Airbnb: http://bit.ly/40TcyNr [Source](https://www.youtube.com/watch?v=CTxnLsYHWuI)

[0:00] Mustafa Suleyman: it must be possible [0:00] Mustafa Suleyman: otherwise it contains us [2:11] Mustafa Suleyman: petrified [12:19] Mustafa Suleyman: Containment is not possible [19:31] Mustafa Suleyman: 智力（Intelligence）将成为一种全新的资本形式。 [25:51] Mustafa Suleyman: Deal with it [68:30] Mustafa Suleyman: 短期主义（short-termism）正在杀死我们 [84:18] Mustafa Suleyman: 我并没有说遏制是可能的，我说的是它‘必须是可能的’（It must be possible）。 [89:57] Steven Bartlett: History has its eyes on you [93:40] Steven Bartlett: 人类可能不再像过去那样是地球的主导驱动者。我们将生活在一个多数日常互动不再是与其他人、而是与 AI 展开的纪元。 [95:44] Mustafa Suleyman: 知识就是力量。 [97:37] Mustafa Suleyman: a good problem to have

[9:17] Mustafa Suleyman: Scaling these models by 10x every single year produces this magical experience of talking to an AI that feels like you're talking to a human that is super knowledgeable and super smart. [20:07] Mustafa Suleyman: We cannot allow ourselves to be dislodged from our position as the dominant species on this planet. We cannot allow that. [33:10] Mustafa Suleyman: If we can't be confident that it's going to respect you and understand you and work for you and us as a species overall, then that's where we have to adopt the precautionary principle. [51:26] Mustafa Suleyman: Our physical body, our brain, our biology is just an enabler for your intelligence... but I haven't seen any evidence yet that we're able to extract the essence of a being from a brain. [54:08] Steven Bartlett: AI is more autonomous than any other technology in history... the idea that technology is somehow running out of control, a self-directed and self-propelling force beyond the realms of human agency remained a fiction—not anymore. [63:38] Mustafa Suleyman: What reduces fear is deep understanding. Spend time playing with these models, look at their weaknesses... they make tons of mistakes, they're crappy in lots of ways, they're actually not that hard to make. [89:57] Steven Bartlett: History has its eyes on you. [97:37] Mustafa Suleyman: We are about to make intelligence widely available to hundreds of millions if not billions of people... over the next few decades we will solve many of our biggest social challenges.

[0:00] are you uncomfortable talking about this [0:02] yeah I mean it's pretty wild right [0:04] Mustafa suan the billionaire founder of [0:07] Google's AI technology he's played a key [0:10] role in the development of AI from its [0:12] first critical steps in 2020 I moved to [0:15] work on Google's chat box it was the [0:17] ultimate technology we can use them to [0:19] turbocharge our knowledge unlike [0:21] anything else why didn't they release it [0:23] we were nervous we were nervous every [0:26] organization is going to race to get [0:29] their hands on intelligence and that's [0:31] going to be incredibly destructive this [0:33] technology can be used to identify [0:35] cancerous tumors as it can to identify a [0:38] Target on the battlefield a tiny group [0:41] of people who wish to cause harm are [0:43] going to have access to tools that can [0:45] instantly destabilize our world that's [0:47] the challenge how to stop something that [0:50] can cause harm or potentially kill [0:52] that's where we need containment do you [0:54] think that it is containable it has to [0:56] be possible why it must be possible why [0:58] must it be because otherwise it contains [1:00] us yet you chose to build a company in [1:03] this space why did you do that because I [1:06] want to design an AI that on your side I [1:09] honestly think that if we succeed [1:12] everything is a lot cheaper it's going [1:13] to power New forms of transportation [1:15] reduce the cost of healthcare but what [1:18] if we fail the really painful answer to [1:20] that question is [1:23] that do you ever get sad about [1:25] it yeah it's intense [1:30] I think this is fascinating I looked at [1:33] the back end of our YouTube channel and [1:35] it says that since this channel started [1:37] 69.9% of you that watch it frequently [1:40] haven't yet hit the Subscribe button so [1:43] I have a favor to ask you if you've ever [1:45] watched this Channel and enjoyed the [1:46] content if you're enjoying this episode [1:47] right now please could I ask a small [1:49] favor please hit the Subscribe button [1:50] helps this channel more than I can [1:51] explain and I promise if you do that to [1:54] return the favor we will make the show [1:57] better and better and better and better [1:59] and better that's the promise I'm [2:00] willing to make you if you hit the [2:01] Subscribe button do we have a [2:03] [Music] [2:09] deal everything that's going on with [2:11] artificial intelligence now and um this [2:14] new wave and all these terms like AGI [2:16] and saw another term in your your book [2:18] called ACI first time I'd heard that [2:20] term how do you feel about it [2:22] emotionally if you had to incapsulate [2:24] how you feel emotionally about what's [2:25] going on in this moment how would you do [2:28] what words would you use I would say [2:30] say in the past it would have been [2:35] petrified and I think that over [2:39] time as you really think through the [2:42] consequences and the pros and cons and [2:44] the trajectory that we're on you adapt [2:49] and you understand that actually there [2:52] is [2:53] something incredibly inevitable about [2:55] this trajectory and that we have to wrap [2:59] our arms around it and guide it and [3:01] control it as a collective species as a [3:04] as humanity and I think the more you [3:08] realize how much influence we [3:11] collectively can have over this outcome [3:14] the more empowering it is because on the [3:17] face of it this is really going to be [3:19] the tool that helps us tackle all the [3:22] challenges that we're facing as a [3:24] species right we need to fix water [3:28] desalination we need to grow food 100x [3:31] cheaper than we currently do we need [3:33] renewable energy to be you know [3:35] ubiquitous and everywhere in our lives [3:38] we need to adapt to climate change [3:40] everywhere you look in the next 50 years [3:43] we have to do more with less and there [3:46] are very very [3:48] few proposals let alone practical [3:52] solutions for how we get there training [3:55] machines to help us as AIDS scientific [3:59] research Partners inventors creators is [4:03] absolutely essential and so the upside [4:06] is phenomenal it's enormous but AI isn't [4:10] just a thing it's not an inevitable [4:13] whole its form isn't inevitable right [4:18] its form the exact way that it manifests [4:21] and appears in our everyday lives and [4:23] the way that it's governed and who it's [4:24] owned by and how it's trained that is a [4:28] question that is up to us collectively [4:31] as a species to figure out over the next [4:33] decade because if we don't Embrace that [4:36] challenge then it happens to us and [4:39] that's really what I'm I have been [4:42] wrestling with for 15 years of my career [4:44] is how to intervene in a way that this [4:48] really does benefit everybody and those [4:51] benefits far far outweigh the potential [4:54] risks at what stage were you [4:57] petrified so I founded Deep Mind in [5:02] 2010 and you know over the course of the [5:06] first few years our progress was fairly [5:09] modest but quite quickly in sort of 2013 [5:14] as the Deep learning Revolution began to [5:16] take off I could see [5:19] glimmers of very early versions of AIS [5:22] learning to do really clever things so [5:25] for example one of our big initial [5:28] achievements was to Teach an AI to play [5:31] the Atari games so remember Space [5:34] Invaders and and pong where you batter a [5:36] ball from left to right and we trained [5:39] this initial AI to purely look at the [5:42] raw pixels screen by screen flickering [5:45] or moving in front of the AI and then [5:48] control the actions up down left right [5:51] shoot or not and it got so good at [5:55] learning to play this simple game simply [5:57] through attaching a value between the [6:00] reward like it was it was getting score [6:03] and taking an action that it learned [6:06] some really clever strategies uh to play [6:08] the game really well that Us games [6:12] players and humans hadn't really even [6:14] noticed at least people in the office [6:16] hadn't noticed it some professionals did [6:19] um and that was amazing to me because I [6:22] was like wow this simple system that [6:26] learns through a set of stimuli Plus a [6:29] reward to take some actions can actually [6:33] discover many strategies clever tricks [6:36] to play the game well that us humans [6:40] hadn't occurred to us right and that to [6:43] me is both thrilling because it presents [6:47] the opportunity to invent new knowledge [6:51] and Advance our [6:53] civilization but of course in the same [6:55] measure is also petrifying [7:00] was there a particular moment when you [7:01] were you were at Deep Mind where you go [7:03] where you had that kind of Eureka Moment [7:05] Like a day when something happened and [7:08] and it caused [7:09] that that Epiphany I guess was it yeah [7:14] it it it was actually a moment even [7:17] before 2013 where I remember standing in [7:20] the office and watching a very early [7:23] prototype of one of these image [7:25] recognition image generation models that [7:29] had um was trained to generate new [7:32] handwritten black and white digits so [7:34] imagine 0er to 1 2 3 4 5 6 789 all in [7:39] different style of handwriting on a tiny [7:42] grid of like 300 pixels by 300 pixels in [7:44] black and white and we were trying to [7:47] train the AI to generate a new version [7:51] of one of those digits a number seven in [7:53] a new handwriting sounds so simplistic [7:56] today given the incredible [7:58] photorealistic images that are being [8:00] generated right um and I just remember [8:03] so clearly it it took sort of 10 or 15 [8:07] seconds and it just resolved it the the [8:09] number appeared it went from complete [8:12] Black to like slowly gray and then [8:14] suddenly these were like white pixels [8:16] appeared out of the the black darkness [8:18] and it revealed a number seven and that [8:21] sounds so simplistic in hindsight but it [8:24] was amazing I was like wow the model [8:29] kind of understands the representation [8:31] of a seven well enough to generate a new [8:34] example of a number seven an image of a [8:37] number seven you know and you roll [8:39] forward 10 years and our predictions [8:43] were correct in fact it was quite [8:45] predictable in hindsight the trajectory [8:48] that we were on more compute plus vast [8:51] amounts of data has enabled us within a [8:55] decade to go from predicting black and [8:57] white digits generating new versions of [9:00] those images to now generating [9:04] unbelievable [9:06] photorealistic not just images but [9:08] videos novel videos with a simple [9:13] natural language instruction or a prompt [9:16] what has surprised you you said you [9:18] referred to that as predictable but what [9:20] has surprised you about what's happened [9:21] over the last decade [9:24] so I think what was predictable to me [9:27] back then was the generation of images [9:30] and of audio um because the structure of [9:36] an image is locally contained so pixels [9:39] that are near one another create [9:41] straight lines and edges and corners and [9:43] then eventually they create eyebrows and [9:45] noses and eyes and faces and entire [9:47] scenes and I could just intuitively in a [9:50] very simplistic way I could get my head [9:52] around the fact that okay well we're [9:54] predicting these number sevens you can [9:56] imagine how you then can expand that out [9:58] to enre images maybe even to videos [10:02] maybe you know to audio too you know [10:04] what I said you know a couple seconds [10:06] ago is connected in phon space in the [10:10] spectrogram but what was much more [10:12] surprising to me was that those same [10:15] methods for Generation applied in the [10:18] space of language you know language [10:21] seems like such a different abstract [10:24] space of ideas when I say like the cat [10:27] sat on the [10:29] most people would probably predict Matt [10:32] right but it could be table car chair [10:36] tree it could be Mountain Cloud I mean [10:39] there's a gazillion possible next word [10:43] predictions and so the space is so much [10:46] larger the ideas are so much more [10:48] abstract I just couldn't wrap my [10:50] intuition around the idea that we would [10:53] be able to create the incredible large [10:56] language models that you see today [10:59] your chat [11:01] gpts chat GPT Google B the Google's Bard [11:06] inflection my new company has an AI [11:08] called Pi pi. a which stands for [11:10] personal intelligence and it's as good [11:12] as chat GPT but much more emotional and [11:15] empathetic and kind so it's just super [11:18] surprising to me that just growing the [11:21] size of these large language models as [11:24] we have done by 10x every single year [11:29] for the last 10 years we've been able to [11:31] produce this and that that that's just [11:33] an amazingly large number if you just [11:36] kind of pause for a moment to Grapple [11:39] with the numbers here in 2013 when we [11:42] trained the Atari AI that I mentioned to [11:44] you at Deep Mind that used two Peta [11:49] flops of computation so peta [11:52] peta stands for a million billion [11:56] calculations a flop is a calculation so [12:00] 2 million [12:01] billion right which is already an insane [12:04] number of calculations lost me at two [12:06] it's totally crazy yeah just two of [12:07] these units that are already really [12:09] large and every year since then we've [12:13] 10x the number of calculations that can [12:15] be done such that today the biggest [12:19] language model that we train at [12:21] inflection uses 10 billion Peta flops so [12:26] 10 billion million billion calculations [12:28] I mean it's just unfathomably large [12:31] number and what we've really observed is [12:35] that scaling these models by 10x every [12:38] single year produces this magical [12:42] experience of talking to an AI that [12:44] feels like you're talking to a human [12:46] that is super knowledgeable and super [12:49] smart there's so much that's happened in [12:52] public conversation around AI um and [12:54] there's so many questions that I have [12:55] I've I've been speaking to a few people [12:57] about artificial intelligence and [12:59] understand it and I'm I think where I am [13:02] right now is I feel quite [13:05] scared um but when I get scared I don't [13:08] get it's not the type of scared that [13:09] makes me anxious it's not like an [13:11] emotional scared it's a very logical [13:12] scared it's my very logical brain hasn't [13:15] been able to figure out how the [13:17] inevitable outcome that I've arrived at [13:20] which is that humans become the less [13:22] dominant species on this planet um how [13:25] that is to be avoided in any way the [13:27] first chapter of your book The Coming [13:28] way wave is a is is a is titled [13:33] appropriately to how I feel containment [13:36] is not possible you you say in that [13:38] chapter the widespread emotional [13:40] reaction I I was observing is something [13:42] I've come to call the pessimism aversion [13:44] trap correct what is the pessimism [13:47] aversion trap [13:50] well so all of us being included feel [13:54] what you just described when you first [13:57] get to grips with the idea of this new [13:59] new coming wave it's scary it's [14:01] petrifying it's threatening is it going [14:03] to take my job is my daughter or son [14:06] going to fall in love with it you know [14:08] what does this mean what does it mean to [14:10] be human in a world where there's these [14:12] other humanlike things that aren't human [14:15] how do I make sense of that it's super [14:18] scary and a lot of people over the last [14:21] few years I think things have changed in [14:24] the last six months I have to say but o [14:26] over the last few years I would say [14:29] the default reaction has been to avoid [14:33] the pessimism and the fear right to just [14:36] kind of recoil from it and pretend that [14:39] it's like either not happening or that [14:41] it's all going to work out to be Rosy [14:43] it's going to be fine we don't have to [14:44] worry about it people often say well [14:46] we've always created new jobs we've [14:48] never permanently displaced jobs we've [14:50] only ever seen new jobs be created [14:53] unemployment is at an all-time low right [14:55] so there's this default optimism bias [14:58] that we have and I think it's less about [15:00] a need for optimism and more about a [15:02] fear of P pessimism and so that trap [15:06] particularly in Elite circles means that [15:09] often we aren't having the tough [15:11] conversations that we need to have in [15:14] order to respond to the coming [15:17] wave are you scared in part about having [15:20] those tough conversations because of how [15:22] it might be received um not so much [15:26] anymore so I've spent most of my career [15:30] trying to put those tough questions on [15:34] the policy table right I've been raising [15:38] these questions the ethics of AI safety [15:41] and questions of containment for as long [15:44] as I can remember with governments and [15:45] civil societies and all the rest of it [15:46] and so I've become used to talking about [15:49] that and you know I [15:53] think it's essential that we have the [15:56] honest conversation because we can't let [15:59] let it happen to us we have to openly [16:01] talk about [16:02] it is I mean this is a this is a big a [16:06] big question [16:08] but as you sit here now do you think [16:11] that it is containable because [16:14] I I I can't see [16:16] how I can't see how it can be [16:19] contained chapter 3 is the containment [16:21] problem or you give give the example of [16:24] how Technologies are often invented for [16:27] good reasons and for certain use cases [16:30] like the hammer you know which is used [16:32] you know maybe to build something but it [16:33] also can be used to kill people [16:36] um and you say in history we haven't [16:38] been able to ban a technology ever [16:41] really it has always found a way into [16:44] society um because of other societies [16:47] have an incentive to have it even if we [16:49] don't and then we need we need it like [16:51] the nuclear bomb because if they have it [16:52] then we don't then we're at a [16:54] disadvantage so are you optimistic [17:01] honestly I don't think an optimism or a [17:05] pessimism frame is the right one because [17:09] the E both are equally biased in ways [17:12] that I think distract us as I say in the [17:16] book on the face of it it does look like [17:19] containment isn't possible we haven't [17:21] contained or permanently banned a [17:24] technology of this type in the past [17:27] there are some that we have done right [17:30] so we banned cfc's for example because [17:32] they were producing a hole in the ozone [17:34] layer we've banned certain weapons [17:37] chemical and biological weapons for [17:39] example or blinding lasers Believe It or [17:41] Not There are such things as lasers that [17:43] will instantly blind you you know so we [17:45] have stepped back from the frontier in [17:49] some cases but that's largely where [17:51] there's either cheaper or you know [17:54] equally effective Alternatives that are [17:56] quickly adopted in this case [17:59] these Technologies are Omni use so the [18:02] same core technology can be used to [18:06] identify you know cancerous tumors in [18:09] chest x-rays as it can to identify a [18:12] Target on the battlefield for an aerial [18:14] strike so that mixed use or Omni use is [18:18] going to drive the proliferation because [18:21] there's huge commercial incentives [18:22] because it's going to deliver a huge [18:23] benefit and do a lot of good and that's [18:27] the challenge that we have to figure out [18:29] is how to stop something which on the [18:31] face of it is so good but at the same [18:33] time can be used in really bad ways too [18:37] do you think we will I do think we will [18:41] so I think that nation states Remain the [18:46] backbone of our civilization we have [18:50] chosen to concentrate power in a single [18:55] Authority the nation state and we pay [18:58] our tax is and we've given the nation [19:00] state a monopoly over the use of [19:03] violence and now the nation state is [19:06] going to have [19:07] to update itself quickly to be able to [19:11] contain this technology because without [19:14] that kind of essentially oversight both [19:18] of those of us who are making it but [19:20] also crucially of the open [19:22] source then it will proliferate and it [19:25] will spread but regulation is still a [19:27] real tool and we we can use it and we [19:29] must what does what does the world look [19:31] like in um let's say 30 years if that [19:35] doesn't happen in your [19:38] view people because people the average [19:40] person can't really gra grapple their [19:42] head around artificial intelligence when [19:43] they think of it they think of like [19:45] these large Lang large language models [19:47] that you can chat to and ask it about [19:49] your [19:50] homework that's like the average [19:52] person's understanding of artificial [19:53] intelligence because that's all they've [19:54] ever been exposed to of it you have a [19:56] different view because of the work [19:58] you've spent the last decade doing so to [20:01] try and give Dave who's I don't know an [20:04] Uber driver in Birmingham an idea who's [20:07] listening to this right now what [20:08] artificial intelligence intelligence is [20:10] and its potential capabilities if you [20:13] know there's no there's no containment [20:16] what does it what does the world look [20:17] like in 30 [20:18] years so I think it's going to feel [20:22] largely like another human so think [20:25] about the things that you can do not [20:28] again in the physical world but in the [20:31] digital world 2050 I'm thinking of I'm [20:33] in [20:34] 2050 2050 we will have robots 2050 we [20:39] will definitely have robots I mean more [20:40] than that 2050 we will have new [20:43] biological beings as well because the [20:45] same trajectory that we've been on with [20:49] hardware and software is also going to [20:52] apply to the platform of biology are you [20:56] uncomfortable talking about this yeah I [20:58] mean it's pretty wild right don't know [21:00] you crossed your arms [21:01] and no I always I always look I always I [21:04] always use that as as a cue for someone [21:05] when when a subject matter is [21:07] uncomfortable and it's interesting [21:10] because I know you know so much more [21:11] than me and about this and I know youve [21:14] spent way more hours thinking off into [21:16] the future about the consequences of [21:17] this I mean you've written a book about [21:19] it so P like you spent 10 years at the [21:21] very deep mind is one of the the [21:23] Pinnacle companies the Pioneers in this [21:24] whole Space so you know you know some [21:28] stuff and it's funny because when I was [21:29] I watched an interview with Elon Musk [21:31] and he was asked a question similar to [21:32] this I know he speaks in certain certain [21:35] tone of voice but he said that he he's [21:37] almost he's gotten to the point where he [21:38] thinks he's living in suspended [21:41] disbelief where he thinks that if he [21:43] spent too long thinking about it he [21:44] wouldn't understand the purpose of what [21:45] he's doing right now and he he says that [21:48] it's more dangerous than nuclear weapons [21:50] um and that it's too late too late to [21:51] stop it there this's one interview [21:53] that's chilling and I was filming [21:55] Dragons Den the other day and I showed [21:56] the dragons the clip I was said look [21:58] what El musk said when he was asked [21:59] about what his child what advice he [22:01] should give to his children in a world [22:03] of in an an inevitable world of [22:05] artificial intelligence it's the first [22:07] time I've seen Elon Musk stop for like [22:08] 20 seconds and not know what to say [22:10] stumble stumble stumble stumble [22:13] stumble and then conclude that he's [22:15] living in suspended [22:17] disbelief yeah I mean I think it's a [22:19] great phrase that is the moment we're in [22:22] we have to it's what I said to you about [22:24] the pessimism verion trap and we have to [22:27] confront the probab ility of seriously [22:30] dark outcomes and we have to spend time [22:33] really thinking about those consequences [22:36] because the competitive nature of [22:39] companies and of nation states is going [22:42] to mean that every organization is going [22:45] to race to get their hands on [22:49] intelligence intelligence is going to be [22:51] a new form of of capital right just as [22:54] there was a grab for land or there's a [22:56] grab for oil there's a grab for anything [22:59] that enables you to do more with less [23:01] faster better smarter right and we can [23:05] clearly see the predictable trajectory [23:08] of the exponential improvements in these [23:09] Technologies and so we should expect [23:12] that wherever there is power there's now [23:15] a new tool to amplify that power [23:18] accelerate that power turbocharge it [23:21] right and you know in 2050 if you ask me [23:25] to look out there I mean of of course it [23:28] makes me Grimace that's why I was like [23:30] oh my God [23:32] it's it really does feel like a new [23:35] species and and that has to be brought [23:38] under control we cannot allow ourselves [23:43] to be dislodged from our position as the [23:48] dominant species on this planet we [23:50] cannot allow that you mentioned robots [23:53] so these are sort of adjacent [23:55] technologies that Rising with artificial [23:57] intelligence robots you mentioned um [24:00] biological new biological [24:04] species give me some light on what you [24:06] mean by [24:07] that well so so far the dream of [24:11] Robotics hasn't really come to fruition [24:14] right I mean we we still have the most [24:17] we have now are sort of drones and [24:19] little bit of self-driving [24:21] cars but that is broadly on the same [24:25] trajectory as these other Technologies [24:27] and I think that over for the next 30 [24:29] Years you know we are going to have [24:32] humanoid robotics we're going to have um [24:35] you know [24:37] physical tools within our everyday [24:41] system that we can rely on that will be [24:44] pretty good that would be pretty good to [24:46] do many of the physical tasks and that's [24:48] a little bit further out because I think [24:49] it you know there's a lot of tough [24:51] problems there but it's still coming in [24:53] the same way and likewise with Biology [24:56] you know we can now see sequence a [24:59] genome for a millionth of the cost of [25:02] the first genome which took place in [25:04] 2000 so 20ish years ago the cost has [25:08] come down by a million times and we can [25:11] now increasingly synthesize that is [25:14] create or [25:16] manufacture new bits of DNA which [25:19] obviously give rise to life in every [25:22] possible form and we're starting to [25:24] engineer that DNA to either remove [25:27] traits [25:29] uh or capabilities that we don't like or [25:31] indeed to add new things that we want it [25:33] to do we want you know fruit to last [25:36] longer or we want meat to have higher [25:39] protein etc etc synthetic meat to have [25:42] higher protein [25:44] levels and what's the implications of [25:48] that potential [25:51] implications I think the the darkest [25:54] scenario there is that people will [25:56] experiment with pathogens [26:00] engineered you know synthetic pathogens [26:03] that might end up accidentally or [26:05] intentionally being more [26:07] transmissible I.E they they can spread [26:10] faster um or more lethal I.E you know [26:15] they cause more harm or potentially kill [26:17] like a pandemic like a pandemic um [26:21] and that's where we need containment [26:24] right we have to limit access to the [26:27] tool TOs and the knowhow to carry out [26:31] that kind of experimentation so one [26:34] framework of thinking about this with [26:36] respect to making containment possible [26:39] is that we really are experimenting with [26:43] dangerous [26:44] materials and Anthrax is not something [26:47] that can be bought over the Internet [26:50] that can be freely experimented with and [26:53] likewise the very best of these tools in [26:55] a few years time are going to be capable [26:58] of creating you know new synthetic um [27:02] pandemic pathogens and so we have to [27:05] restrict access to those things that [27:06] means restricting access to the compute [27:09] it means restricting access to the [27:11] software that runs the models to the [27:14] cloud environments that provide apis [27:16] provide you access to experiment with [27:18] those things um and of course on the [27:21] biology side it means restricting access [27:23] to some of the substances and people [27:24] aren't going to like this people are not [27:27] going to like that claim because it [27:30] means that those who want to do good [27:32] with those tools those who want to [27:35] create a start up the small guy the [27:38] little developer that struggles to [27:40] comply with all the regulations they're [27:42] going to be pissed off understandably [27:44] right but that is the age we're in deal [27:48] with it like we have to confront that [27:51] reality that means that we have to [27:53] approach this with the precautionary [27:55] principle right never before in the [27:58] invention of a technology or in the [27:59] creation of a regulation have we [28:02] proactively said we need to go slowly we [28:05] need to make sure that this first does [28:07] no harm the precautionary principle and [28:10] that is just an unprecedented moment no [28:13] other Technology's done that right [28:15] because I think we collectively in the [28:18] industry those of us who are closest to [28:20] the work can see a place in 5 years or [28:23] 10 years where it could get out of [28:26] control and we have to get on top of it [28:27] now now and it's better to forgo like [28:30] that is give up some of those potential [28:33] upsides or benefits until we can be more [28:36] sure that it can be contained that it [28:39] can be controlled that it always serves [28:41] our Collective interests and I I think [28:44] about that so I think about what you've [28:45] just said there about being able to [28:47] create these pathogens these diseases [28:49] and viruses Etc that you know could [28:51] become weapons or whatever else but with [28:53] artificial intelligence and the power of [28:55] that intelligence with these um [28:59] pathogens you could theoretically ask [29:02] one of these systems to create a virus [29:06] that a very deadly [29:09] virus um you could ask the artificial [29:11] intelligence to create a very deadly [29:12] virus that has certain [29:15] properties um maybe even that mutates [29:18] over time in a certain way so it only [29:19] kills a certain amount of people kind of [29:21] like a nuclear bomb of of viruses that [29:23] you could just pop hit an enemy with now [29:26] if I'm if I hear that and I go okay [29:27] that's powerful I would like one of [29:30] those you know there might be an [29:31] adversary out there that goes I would [29:32] like one of those just in case America [29:34] get out of hand and America's thinking [29:36] you know I want one of those in case [29:37] Russia gets out of hand and so okay you [29:40] might take a precautionary approach in [29:43] the United States but that's only going [29:44] to put you on the back foot when China [29:47] or Russia or one of your adversaries [29:50] accelerates forward in that in that path [29:51] and this was the same with the the [29:52] nuclear bomb and you know you nailed it [29:57] I mean that is is the race condition we [30:00] refer to that as the race condition the [30:03] idea that if I don't do it the other [30:07] party is going to do it and therefore I [30:10] must do it but the problem with that is [30:12] that it creates a self-fulfilling [30:14] prophecy so the default there is that we [30:16] all end up doing it and that can't be [30:19] right because there is a opportunity for [30:23] massive cooperation here there's a [30:26] shared that is between us and China and [30:29] every other quote unquote them or they [30:31] or enemy that we want to create we've [30:35] all got a shared interest in advancing [30:39] the collective health and well-being of [30:42] humans and Humanity how well have we [30:44] done at promoting shared interest well [30:47] in the development of Technologies over [30:48] the years even at like a corporate level [30:51] even you [30:52] know you know the nuclear [30:54] nonproliferation treaty has been [30:55] reasonably successful there's only nine [30:58] nuclear states in the world today we've [31:00] stopped many like three countries [31:03] actually gave up nuclear weapons because [31:05] we incentivize them with sanctions and [31:07] threats and economic rewards um small [31:11] groups have tried to get access to [31:12] nuclear weapons and so far have largely [31:14] failed it's expensive though right and [31:16] hard to like uranium as a as a chemical [31:19] to keep it stable and to to buy it and [31:21] to house it I mean I couldn't just put [31:22] it in the shed you certainly couldn't [31:24] put it in a shed you can't download [31:26] uranium 235 off the Internet it's not [31:28] available open source that is totally [31:30] true so it's got different [31:32] characteristics for sure but a kid in [31:34] Russia could you know in his bedroom [31:36] could download something onto his [31:38] computer that's incredibly harmful in [31:41] the artificial intelligence Department [31:44] right I think that that will be possible [31:47] at some point in the next five years [31:49] it's true because there's a weird Trend [31:51] that's going on here on the one hand [31:55] You've Got The Cutting Edge AI models [31:58] that are built by Google and open Ai and [32:01] my company inflection and they cost [32:04] hundreds of millions of dollars and [32:05] there's only a few of them but on the [32:07] other hand the what was cutting edge a [32:10] few years ago is now open source today [32:14] so gpt3 which came out in the summer of [32:18] 2020 is now reproduced as an open- [32:22] Source model so the code and the weights [32:24] of the model the design of the model and [32:26] the actual implementation code is [32:29] completely freely available on the web [32:32] and it's tiny it's like 60 times or 60 [32:36] 70 times smaller than the original model [32:39] which means that it's cheaper to use and [32:41] cheaper to run and that's as you know [32:44] we've said earlier like that's the [32:45] natural trajectory of technologies that [32:48] become useful they get more efficient [32:50] they get cheaper and they spread further [32:52] and so that's the containment challenge [32:55] that's really the essence of what I'm [32:57] sort of trying to raise in my book is to [33:00] frame the challenge of the next 30 to 50 [33:03] years as around containment um and [33:06] around confronting [33:08] proliferation do you believe because [33:11] we're both going to be alive unless this [33:12] you know there some robot kills us but [33:14] we're both going to be alive in 30 years [33:16] time I hope so maybe the podcast will [33:19] still be going unless AI is now taking [33:21] my [33:23] job it's very possible so I'm I'm going [33:26] to sit you here and you know when you're [33:28] you you'll be what 60 68 years old I'll [33:30] be [33:33] 60 um and I'll [33:37] say at that point when we have that [33:39] conversation do you think we would have [33:41] been successful in containment at a [33:43] global level I think we have to be I [33:47] can't even think that we're not why [33:59] because I'm fundamentally a humanist and [34:02] I think that we have to make a choice to [34:05] put our species first [34:10] and I think that that's what we have to [34:12] be defending for the next 50 years [34:15] that's what we have to defend because [34:17] look it it's it's certainly possible [34:21] that we invent these [34:23] agis in such a way that they are always [34:26] going to be [34:29] provably um [34:31] subservient uh to humans and take [34:36] instructions you know from their human [34:38] controller every single time but enough [34:41] of us think that we can't be sure about [34:44] that that I don't think we should take [34:47] the gamble basically [34:51] so that's why I think that we should [34:53] focus on containment and non [34:54] proliferation because some people if [34:56] they do have access of the technology [34:58] will want to take those risks and they [35:00] will just want to see like what's on the [35:03] other side of the door you know and they [35:05] might end up opening Pandora's Box and [35:07] that's a decision that affects all of us [35:10] and that's the challenge of the [35:11] networked age you know we live in this [35:13] globalized world and we use these words [35:15] like globalization and we you sort of [35:17] forget what globalization means this is [35:19] what globalization is this is what a [35:21] networked world is it means that someone [35:24] taking one small action can suddenly [35:27] spread [35:28] everywhere instantly regardless of their [35:31] intentions when they took the action it [35:33] maybe you know unintentional like you [35:35] say may be that they're never they [35:37] weren't ever meaning to do [35:41] harm well I think I asked you when I [35:44] said it you 30 years time you said that [35:45] there will be like human level [35:48] intelligence you'll be interacting with [35:50] you know this new species but the [35:52] species for me to think the the species [35:54] will want to interact with me is feels [35:57] like wishful thinking because what will [35:59] I be to them you know like I've got a [36:02] French Bulldog Pablo and I can't imagine [36:05] our IQ is that far apart like like you [36:08] know in relative terms my the IQ between [36:11] me and my dog Pablo I can't imagine [36:12] that's that far apart even when I think [36:14] about is it like the orangutang where we [36:16] only have like 1% difference in DNA or [36:18] something crazy and yet they throw their [36:20] poop around and I'm sat here [36:22] broadcasting around the world there's [36:24] quite a difference in that 1% you know [36:27] and then I think about this new species [36:29] where as you write in your book in [36:31] chapter [36:33] 4 there seems to be no upper limit to [36:35] ai's potential [36:37] intelligence why would such an [36:39] intelligence want to interact with me [36:42] well it depends how you design it [36:45] so I think that our goal one of the [36:48] challenges of containment is to design [36:52] AIS that we want to interact with that [36:55] want to interact with us right if you [36:58] set an objective function for an AI a [37:00] goal for an AI by its design which you [37:04] know inherently disregards or [37:07] disrespects you as a human and your [37:09] goals then it's going to wander off and [37:10] do a lot of strange things what if it [37:12] has kids and the kids are you know what [37:15] I mean what if it replicates in a way [37:17] where because because I've I've heard [37:19] this this conversation around like it [37:20] depends how we design it but you [37:23] know I think about [37:28] it's kind of like if I have a kid and [37:30] the kid grows up to be a thousand times [37:32] more intelligent than me to think that I [37:35] could have any influence on it on it [37:37] when it's a thinking [37:39] sentient [37:40] developing species again feels like I'm [37:45] overestimating my version of [37:46] intelligence and importance and [37:48] significance in the face of something [37:50] that is incomprehensibly like a even a [37:52] hundred times more intelligent than me [37:54] and the speed of its computation is a [37:56] thousand times what my [37:58] the meat in my skull can do yeah like [38:01] how how how is it gonna how how do I [38:03] know it's going to respect me or care [38:04] about me or understand you know that I [38:06] me you [38:08] know I think that comes back down to the [38:10] containment challenge I think that if we [38:12] can't be confident that it's going to [38:15] respect you and understand you and work [38:19] for you and us as a species overall then [38:24] that's where we have to adopt the [38:26] precautionary principle I don't think we [38:28] should be taking those kinds of risks in [38:31] experimentation and design and now I'm [38:33] not saying it's possible to design an AI [38:38] that doesn't have those self-improvement [38:41] capabilities in the limit in like 30 or [38:43] 50 years I think it you know that's kind [38:45] of what I was saying is like it seems [38:47] likely that if you have one like that [38:50] it's going to take advantage of infinite [38:52] amounts of data and infinite amounts of [38:54] computation and it's going to kind of [38:55] outstrip our ability to act and so I [38:59] think we have to step back from that [39:01] precipice that's what the containment [39:04] problem is is that it's it's actually [39:07] saying no sometimes it's saying no and [39:11] that's a different sort of muscle that [39:14] we've never really exercised as a [39:16] civilization and and that's obviously [39:18] why containment appears not to be [39:20] possible because we've never done it [39:22] before we've never done it before and [39:24] every inch of our you know Comm and [39:28] politics and our war and all of our [39:30] instincts are just like Clash compete [39:32] Clash compete prophit profit grow beat [39:36] exactly dominate you know fear them be [39:39] paranoid like now all this nonsense [39:41] about like China being this new evil [39:44] like it how does that slip into our [39:46] culture how are we suddenly all shifted [39:49] from thinking it's the the the Muslim [39:50] terrorists about to blow us all up to [39:53] now it's the Chinese who are about to [39:55] you know blow up Kansas it's just like [39:57] what are we talking about that like we [39:59] really have to pair back the paranoia [40:02] and the fear and the othering um because [40:06] those are the incentive dynamics that [40:08] are going to drive us to you know cause [40:11] self harm to humanity thinking the worst [40:14] in each other the the there's couple of [40:16] key moments when in my understanding of [40:18] artificial intelligence that have been [40:19] kind of Paradigm Paradigm shifts for me [40:21] because I think like many people I [40:23] thought of artificial intelligence [40:25] as you know like a like a child I was [40:28] Raising and I would program I would code [40:31] it to do certain things so I would code [40:32] it to play chess and I would tell it the [40:36] moves that are conducive with being [40:39] successful in chess and then I remember [40:41] watching that like Alpha go documentary [40:43] right which I think was deep deep mind [40:44] wasn't it that was us yeah you guys so [40:45] you programmed this this um artificial [40:47] intelligence to play the game go which [40:49] is kind of like just think of it kind of [40:50] like a chess or a black am or whatever [40:52] and it eventually just beats the best [40:53] player in the world of all time and it [40:55] and the way it learned how to beat the [40:57] best player in the world of all time the [40:58] world champion who was by the way [41:00] depressed when he got beat um was just [41:03] by playing itself right and then there's [41:06] this moment I think in is it game four [41:08] or something where right it does this [41:10] move that no one could have predicted a [41:12] move that seemingly makes absolutely no [41:15] sense [41:16] right in those moments where no one [41:20] trained it to do that and it did [41:21] something unexpected Beyond where humans [41:24] are trying to figure it out in hindsight [41:26] this is where I go how do you how do you [41:27] train it if it's doing things we didn't [41:30] anticipate right like how do you control [41:32] it when it's doing things that humans [41:34] couldn't anticipate it doing where we're [41:35] looking at that move it's called like [41:37] move 37 or something correct yeah is it [41:39] move 37 it is look at my intelligence F [41:41] nice work yeah I'm I'm going to survive [41:43] a bit longer than I thought it's like [41:45] move 37 you at least another decade in [41:48] you um move 37 does this crazy thing and [41:51] you see everybody like lean in and go [41:53] why has it done that and it turns out to [41:55] be brilliant that humans couldn't [41:57] couldn't forecast the commentator [41:58] actually thought it was a mistake yeah [42:00] he was a pro and he was like this this [42:02] definitely a mistake you know it's the [42:03] the alpha go lost the game but it was so [42:05] far ahead of us that it knew something [42:07] we didn't right right that's when that's [42:10] when I lost hope in this whole idea of [42:11] like oh train it to do what we want like [42:13] a dog like sit pour roll over right well [42:18] the real challenge is that we actually [42:22] want it to do those things like when it [42:25] discovers a new strategy [42:27] or it invents a new idea or it helps us [42:30] find like you know a cure for some [42:32] disease like that's why we're building [42:34] it right because we're reaching the [42:37] limits of what we as you know humans can [42:41] invent and solve right especially with [42:43] what we're facing of you know in terms [42:45] of population growth over the next 30 [42:47] Years and how climate change is going to [42:49] affect that and so on like we really [42:52] want these tools to turbocharge us right [42:55] and yet like it's that creativity and [42:57] that invention which obviously makes us [43:00] also [43:01] feel well maybe it it is really going to [43:05] do things that we don't like for sure [43:07] right so [43:11] interesting how do you contend with all [43:13] of this how do you contend with the the [43:16] clearup side and then you must like Elon [43:19] must be completely aware of the the [43:23] horrifying existential risk at the same [43:26] time and and you're building a big [43:28] company in this space which I think is [43:30] valued at 4 billion now inflection AI [43:32] which has got this its own model called [43:36] Pi so you're building in this space you [43:40] understand the incentives at both a [43:41] nation state level and a corporate level [43:43] that we're we're going to keep planing [43:44] forward even if the US stops there's [43:46] going to be some other country that sees [43:47] that as a huge Advantage their economy [43:49] will swell because they did if this [43:51] company stops then this one's going to [43:53] get a get a huge advantage and their [43:54] shareholders are you know [43:58] everyone's investing in AI full steam [44:00] ahead but you feel you can see this huge [44:03] existential risk is it suspended is that [44:04] the pathw suspended [44:06] disbelief I mean just to kind of like [44:09] just know that it's I feel like I know [44:11] that it's going to happen no one's been [44:13] able to tell me [44:15] otherwise but just don't think too much [44:18] about it and you'll be okay I think you [44:21] can't give up right I think that in some [44:25] ways you're realization exactly what [44:28] you've just described like weighing up [44:31] two conflicting and horrible truths [44:33] about what is likely to happen those [44:37] contradictions that is a kind of honesty [44:40] and a wisdom I think that we need all [44:43] collectively to realize because the only [44:47] path through this is to be straight up [44:51] and embrace you know the risks and [44:54] embrace the default trajectory of all [44:56] these competing incentives driving [44:58] forward to kind of make this feel like [45:01] inevitable and if you put the blinkers [45:03] on and you kind of just ignore it or if [45:05] you just be super Rosy and it's all [45:06] going to be all right and if you say [45:07] that we've always figured it out anyway [45:09] then you we're not going to get the [45:11] energy and the dynamism and engagement [45:13] from everybody to try to figure this out [45:16] and that's what gives me like reason to [45:19] be hopeful because I think that we make [45:22] progress by getting everybody paying [45:25] attention to this it isn't going to be [45:28] about those who are currently the AI [45:30] scientists or those who are the [45:32] technologists you know like me or the [45:34] Venture capitalists or just the [45:36] politicians like all of those people no [45:38] one's got answers so that's what we have [45:40] to confront there are no obvious answers [45:44] to this profound question and I've [45:48] basically written the book to say prove [45:51] that I'm wrong you know containment must [45:54] be [45:55] possible and I it must be it must be [45:59] possible it has to be possible it has to [46:01] be you want it to be I I desperately [46:04] want it to be yeah why must it be [46:08] because otherwise I think you're in the [46:10] camp of believing that this is the [46:13] inevitable evolution of humans the [46:17] transhuman kind of view you know some [46:21] people would argue like what is okay [46:23] let's part let's let's stretch the [46:24] timelines out okay so let's not talk [46:27] about 30 years let's talk about 200 [46:31] years like what is this going to look [46:34] like in [46:38] 2200 you tell me you're smarter than [46:40] me I mean it's mindblowing it's [46:43] mind-blowing we'll have quantum [46:45] computers by then what's a quantum [46:48] computer a quantum computer is a [46:51] completely different type of computing [46:54] architecture which in simple [46:57] terms basically allows you [47:00] to those those calculations that I [47:03] described at the beginning billions and [47:05] billions of flops those billions of [47:07] flops can be done in a single [47:10] computation so everything that you see [47:13] in the digital world today relies on [47:16] computers processing information and and [47:19] the speed of that processing is a [47:21] friction it kind of slows things down [47:24] right you remember back in the day old [47:27] School modems 56k modem the dialup sound [47:30] and the image pixel loading like pixel [47:33] by pixel that was because the computers [47:35] were slow and we're getting to a point [47:37] now where the computers are getting [47:39] faster and faster and faster and Quantum [47:41] Computing is like a whole new leap like [47:44] way way way Beyond where we where we [47:47] currently are and so by analogy how [47:50] would I understand that so like if my [47:52] I've got my dialup modem over here and [47:55] then Quantum Computing over here [47:57] right what's the how do I what's the [48:00] difference well I don't know what it's [48:02] really difficult to exp a billion times [48:04] faster oh it's it's it's like it's like [48:06] billions of billions times faster it's [48:08] it's it's much more than that I mean one [48:10] way of think about it is [48:12] like a floppy disc which I guess most [48:16] people remember 1.4 megabytes a physical [48:19] thing back in the day in [48:22] 1960 or so that was basically an entire [48:27] pallets worth of computer that was moved [48:30] around by a forklift truck right which [48:33] is insane today you know you have [48:37] billions and billions of times that [48:39] floppy disc in your smartphone in your [48:43] pocket tomorrow you're going to have [48:47] billions and billions of smartphones in [48:51] minuscule wearable devices there'll be [48:54] cheap fridge magnets that you know are [48:56] constantly on everywhere sensing all the [48:59] time monitoring processing analyzing [49:02] improving [49:03] optimizing you know and they'll be super [49:06] cheap so it's super unclear what do you [49:10] do with all of that knowledge and [49:12] information I mean it's ultimately [49:15] knowledge creates value when you know [49:18] the relationship between things you can [49:20] improve them you know make it more [49:22] efficient and so more data is what has [49:26] enabled us to build all the value of you [49:28] know online in the last 25 years and so [49:32] what does that look like in 150 years I [49:35] can't really even imagine to be honest [49:36] with you it's very hard to say I don't [49:39] think everybody is going to be [49:41] working why would we yeah what we [49:45] wouldn't be working in that kind of [49:46] environment I mean look the other [49:48] trajectory to add to this is the cost of [49:53] energy [49:54] production you know AI if it really [49:58] helps us [50:00] solve battery [50:02] storage which is the missing piece I [50:05] think to really tackle climate change [50:08] then we will be able to Source basically [50:11] source and store infinite energy from [50:15] the Sun and I think in 20 or so years [50:18] time 20 30 years time that is going to [50:21] be a cheap and widely available if not [50:23] completely freely available resource and [50:26] if you think about it everything in [50:29] life has the cost of energy built into [50:32] its production value and so if you strip [50:35] that out everything is likely to get a [50:38] lot cheaper we'll be able to desalinate [50:41] water we'll be able to grow crops much [50:44] much cheaper we'll be able to grow much [50:46] higher quality food right it's going to [50:48] power New forms of transportation it's [50:50] going to reduce the cost of drug [50:52] production and Healthcare right so all [50:55] of those gains [50:57] obviously there'll be a huge commercial [50:58] incentive to drive the production of [51:00] those gains but the cost of producing [51:02] them is going to go through the floor I [51:03] think that's one key thing that a lot of [51:05] people don't realize that is a reason to [51:07] be hugely hopeful and optimistic about [51:09] the future everything is going to get [51:12] radically cheaper in 30 to 50 [51:18] years it's a 200 years time we have no [51:20] idea what the world looks like it's uh [51:22] this goes back to the point about being [51:24] is it did you say transhumanist [51:26] right what does that [51:28] mean [51:30] transhumanism I mean it's a group of [51:32] people [51:34] who basically believe [51:37] that you that that humans and our soul [51:41] and our being will one day transcend or [51:44] move beyond our biological substrate [51:49] okay so our physical body our brain our [51:51] biology is just an enabler for your [51:55] intelligence and who you are as a person [51:59] and there's a group of kind of crack [52:03] Bots basically I think who think that [52:06] we're going to be able to upload [52:09] ourselves to a silicon substrate right a [52:13] computer that can hold the essence of [52:16] what it means to be Stephen so you in [52:18] 200 in 20 uh in in [52:22] 2200 will could well still be You by [52:25] their reasoning [52:27] but you'll live on a server somewhere [52:30] why are they wrong I think about all [52:31] these adjacent Technologies like [52:33] biological um biological advancements [52:36] did you call it like biosynthesis or [52:38] something was yeah synthetic biology syn [52:40] synthetic biology um I think about the [52:44] nanotechnology development right think [52:46] about Quantum Computing the the progress [52:48] in artificial intelligence everything [52:50] becoming cheaper and I think why why are [52:51] they [52:53] wrong it's hard to say precisely [52:57] but broadly speaking I haven't seen any [53:00] evidence yet that we're able to extract [53:03] the essence of a being from a brain [53:06] right it's that that that that kind of [53:08] dualism that you know there is a mind [53:11] and a body and a spirit that is a I I [53:16] don't think I don't see much evidence [53:18] for that even in Neuroscience um that [53:20] actually it's much more one and the same [53:22] so I don't think you know you're going [53:24] to be able to emulate the entire brain [53:26] so their thesis is that well some of [53:28] them cryogenically store their brain [53:30] after death Jesus so they they have it [53:33] they' they they wear these like you know [53:35] how you have like an organ donor tag or [53:37] whatever so they have a cryogenically [53:40] freeze me when I die tag and so they [53:43] there's like a special like ambulance [53:45] services that will come pick you up [53:47] because obviously you need to do it [53:48] really quickly the moment you die you [53:49] need to get put into a cryogenic freezer [53:52] to preserve your you know brain forever [53:55] I personally think this is this is is [53:56] nuts but you know their belief is that [53:58] you'll then be able to reboot that [54:01] biological brain and then transfer you [54:03] over [54:05] um it it doesn't seem plausible to me [54:08] when you said at the start of this this [54:09] little topic here that you it must be [54:12] possible to contain it said it must be [54:15] possible um the the reason why I I [54:18] struggle with that is because in chapter [54:19] 7 you say line in your book that AI is [54:22] more autonomous than any other [54:23] technology in [54:24] history for centuries the idea that [54:26] technology is is somehow running out of [54:29] control a self-directed and [54:30] self-propelling force beyond the Realms [54:32] of human agency remained a fiction not [54:38] anym and this idea of autonomous [54:41] technology that [54:43] is acting [54:47] uninstructed um and is intelligent and [54:50] then you say we must be able to contain [54:52] it it's kind of like a massive dog like [54:54] a big rottweiler yeah [54:56] that is you know a thousand times bigger [54:59] than me and me looking up at it and [55:00] going I'm going to get take you for a [55:02] walk y yeah and then it's just looking [55:05] down at me and [55:07] just stepping over me or stepping on me [55:10] well that's actually a good example [55:12] because we have actually contained [55:15] Rottweilers before we've contained [55:17] gorillas and you know tigers and [55:20] crocodiles and pandemic pathogens and [55:23] nuclear weapons and so you know it's [55:26] easy to be you know a hater on what [55:29] we've achieved but this is the most [55:32] peaceful moment in the history of our [55:34] species this is a moment when our [55:36] biggest problem is that people eat too [55:38] much think about that we've spent our [55:42] entire evolutionary [55:44] period running around looking for food [55:47] and trying to stop you know our enemies [55:49] throwing rocks at us and we've had this [55:53] incredible period of 500 years [55:57] where you know each year things have [56:00] broadly well maybe each each Century [56:03] let's say there's been a few ups and [56:04] downs but things have broadly got better [56:07] and we're on a trajectory for you know [56:10] lifespans to increase and quality of [56:12] life to increase and health and [56:14] well-being to improve and I think that's [56:17] because in many ways we have succeeded [56:20] in containing forces that appear to be [56:23] more powerful than ourselves it just [56:25] requires unbelievable creativity and [56:29] adaptation it requires compromise and it [56:33] requires an a new tone right a much more [56:37] humble tone to governance and politics [56:41] and and how we run our world not this [56:43] kind of like hyper aggressive [56:45] adversarial paranoia tone that we talked [56:47] about previously but one that is like [56:50] much more wise than that much more [56:53] accepting that we are unleashing this [56:55] force that does have that that potential [56:56] to be the Rott riler that you described [56:59] but that we must contain that as our [57:02] number one priority that has to be the [57:04] thing that we focus on because otherwise [57:06] it contains [57:08] us i' I've been thinking a lot recently [57:10] about cyber security as well just [57:11] broadly on an individual level in a [57:14] world where there are these kinds of [57:16] tools which seems to be quite close um [57:18] large language models brings up this [57:21] whole new question about cyber security [57:23] and cyber safety and you know in a world [57:25] where there's these ability to generate [57:28] audio and language and videos that seem [57:31] to be real um what can we trust and you [57:34] know I was watching a video of a of a of [57:37] a young girl whose grandmother was [57:39] called up by a voice that was made to [57:42] sound like her son saying he'd been in a [57:44] car accident and asking for money and [57:47] her nearly sending the money or this [57:49] whole you know because this really [57:50] brings into Focus that we our lives are [57:52] build on built on trust trusting the [57:54] things we see here in [57:57] watch and in in and now we're at feels [58:01] like a a a moment where we're no longer [58:04] going to be able to trust what we see on [58:06] the internet on the [58:09] phone what what what advice do you do we [58:11] you have for people who were worried [58:13] about [58:15] this [58:16] so skepticism I think is healthy and [58:20] necessary and I think that we're going [58:22] to need it um even more than than we [58:26] ever did right and so if you think about [58:29] how we've adapted to the first wave of [58:32] this which was spammy email scams um [58:35] everybody got them and over [58:38] time people learned to identify them and [58:42] be skeptical of them and reject them [58:44] likewise you know I'm sure many of us [58:46] get like text messages I certainly get [58:48] loads of text messages trying to fish me [58:50] and ask me to meet up or do this that [58:52] and the other and we've adapted right [58:55] now I think we should all know and [58:59] expect that criminals will use these [59:02] tools to manipulate us just as you've [59:05] described I mean you know the voice is [59:08] going to be humanlike the Deep fake is [59:11] going to be super convincing and there [59:15] are actually ways around those things so [59:18] for example the reason why the banks [59:21] invented OTP um one-time passwords where [59:24] they send you a text message with a [59:26] special code um is precisely for this [59:29] reason so that you have a 2fa a two [59:32] Factor authentication increasingly we [59:34] will have a three or four Factor [59:37] authentication where you have to [59:38] triangulate between multiple separate [59:42] independent sources and it won't just be [59:45] like call your bank manager and release [59:46] the funds right [59:49] so this is where we need the creativity [59:52] and energy and attention of everybody [59:54] because [59:56] defense the kind of defensive measures [59:59] have to evolve as quickly as the [60:01] potential offensive measures the attacks [60:04] that are [60:05] coming I heard you say this that you [60:08] think um some people are for many of [60:10] these problems we're going to need to [60:12] develop AIS to defend us from the [60:15] AIS right we kind of already have that [60:18] right so we have automated ways of [60:21] detecting spam online these days you [60:24] know most of the time there are um [60:27] machine Learning Systems which are [60:29] trying to identify when your credit card [60:31] is used in a fraudulent way that's not a [60:34] human sitting there looking at patterns [60:35] of spending traffic in real time that's [60:38] an AI that is like flagging that [60:40] something looks off um likewise with [60:43] data centers or security cameras a lot [60:47] of those security cameras these days are [60:49] you know have tracking algorithms that [60:51] look for you know surprising sounds or [60:55] like if a if a glass window is is [60:57] smashed that'll be detected by an AI [61:01] often that is you know listening on the [61:03] security camera so you know that's kind [61:06] of what I mean by that is that [61:07] increasingly those AIS will get more [61:09] capable and we'll want to use them for [61:11] defensive purposes and that's exactly [61:14] what it looks like to have good healthy [61:16] well-functioning controlled AIS that [61:18] serve us I went on one of these large [61:20] language models and and said to me give [61:22] I said to the large language model give [61:23] me an example where in artificial [61:25] intelligence takes over the world or [61:26] whatever and just and results in the [61:29] destruction of humanity and then tell me [61:31] what we'd need to do to prevent [61:33] it and it said it gave me this wonderful [61:36] example of this AI called Cynthia that [61:39] threatens to destroy the world and it [61:41] says the way to defend that would be a [61:43] different AI which had a different name [61:46] and it said that this one would be [61:47] acting in human interests and we'd [61:49] basically be fighting one AI with [61:51] another Ai and of and of course of [61:55] course of course that level if Cynthia [61:57] started to wreak hav havoc on the world [61:59] and take control of the nuclear weapons [62:00] and infrastructure and all that we would [62:02] need an equally [62:04] intelligent weapon to fight [62:07] it although one of the interesting [62:09] things that we found um over the last [62:12] few decades is that it so far tended to [62:15] be the AI plus the human that has that [62:19] is still dominating that's the case in [62:21] chess uh in go and other games um [62:26] that in go it's still yeah so there was [62:28] a paper that came out a few months ago [62:31] two months ago that showed that a human [62:33] was actually able to beat The Cutting [62:35] Edge go program um even one that was [62:38] better than Alpha go with a new strategy [62:40] that they had discovered um you know so [62:44] obviously it's not just a sort of game [62:47] over environment where the AI just [62:48] arrives and it gets better like humans [62:50] also adapt they get super smart they [62:53] like I say get more cynical ask get more [62:55] more skeptical ask you know good [62:58] questions invent their own things use [62:59] their own AIS to adapt and that's the [63:02] evolutionary nature of what it means to [63:05] have a technology right I mean [63:06] everything is a technology like your [63:08] pair of glasses made you smarter in a [63:12] way like before there were glasses and [63:14] people got bad eyesight they weren't [63:15] able to read you know suddenly those who [63:18] did adopt those Technologies were able [63:20] to read for you know longer in their [63:22] lives or under low light conditions and [63:24] they were able to consume more [63:25] information and got smarter and so that [63:27] is the trajectory of Technology it's [63:29] this iterative interplay between you [63:33] know human and machine that makes us [63:36] better over time you know the potential [63:39] um consequences if if we don't reach a [63:42] point of containment yet you chose to [63:43] build a company in this space [63:47] yeah why why that why did you do that [63:51] because I believe that the best way to [63:56] uh demonstrate how to build safe and and [64:00] contained AI is to actually experiment [64:04] with it in practice and I think that if [64:08] we are just Skeptics or critics and we [64:11] stand back from The Cutting Edge then we [64:14] give up that opportunity to shape [64:16] outcomes to you know all of those other [64:20] actors that we referred to whether it's [64:22] like China and the US going at each [64:23] other's throats uh you know you know or [64:26] other big companies that are purely [64:27] pursuing profit at all costs and so it [64:31] doesn't solve all the problems of course [64:33] it's super hard and again it's full of [64:35] contradictions but I honestly think it's [64:38] the right way for everybody to proceed [64:41] you know if experiment at the front yeah [64:43] if you're afraid Russia Putin understand [64:47] right what reduces fear is deep [64:50] understanding spend time playing with [64:52] these models look at their weaknesses [64:54] they're not superhuman yet they make [64:56] tons of mistakes they're crappy in lots [64:58] of ways they're actually not that hard [65:00] to make the more you've experimented has [65:03] it has that correlated with a reduction [65:05] in [65:08] fear cheeky question no but that's yes [65:12] and no you're totally right yes it has [65:14] in the sense that you know the problem [65:17] is the more you learn the more you [65:19] realize yeah that's what I'm saying I [65:22] was fine before I started talking about [65:24] Ai and now more I've talked about [65:27] it it's true it's true it's it's sort of [65:30] pulling on a thread [65:32] which it's a crazy spiral um yeah I mean [65:37] like I think in the short term It's Made [65:39] Me way less afraid because I I don't see [65:42] that kind of existential harm that we've [65:45] been talking about in the next decade or [65:47] two but longer term that's that's where [65:49] I struggle to wrap my head around how [65:51] things play out in 30 [65:53] years some people say [65:57] government regulation will sorted [65:59] out you discussed this in Chapter 13 of [66:02] your book where you which is titled [66:04] containment must be possible I love how [66:07] you didn't say is yeah containment must [66:11] be containment must be possible um what [66:14] do you say to people that say government [66:15] regulation will sorted out I had rishy [66:16] sunak did some announcement and he's got [66:18] a cobra committee coming together [66:20] they'll handle it that's right and the [66:23] EU have a huge piece of regulation [66:26] called the EU AI act um um you know Joe [66:30] President Joe Biden has you know gotten [66:32] his own you know set of proposals and um [66:36] you know we've been working with with [66:38] both you know Rishi sunak and and Biden [66:40] and you know trying to contribute and [66:42] shape it in the best way that we can [66:44] look it isn't going to happen without [66:47] regulation so regulation is essential is [66:50] critical um again going back to the [66:53] precautionary principle but at the same [66:56] time regulation isn't enough you know I [66:59] often hear people say well we'll just [67:01] regulate it we'll just stop we'll just [67:04] stop we'll just stop we'll slow down um [67:08] and the problem with that is that it [67:10] kind of ignores the fact that the people [67:14] who are putting together the regulation [67:17] don't really understand enough about the [67:19] detail today you know in their defense [67:22] they're rapidly trying to wrap their [67:24] head around it especially in in the last [67:25] 6 months and that's a great relief to me [67:28] cuz I feel the burden is now [67:29] increasingly shared and you know just [67:32] from a personal perspective I'm like I [67:35] feel like I've been saying this for [67:36] about a decade and just in the last six [67:38] months now everyone's coming at me and [67:40] saying like you know what's going on I'm [67:42] like great this is the conversation we [67:44] need to be having because everybody can [67:46] start to see the glimmers of the future [67:49] like what will happen if a chat GPT like [67:52] product or a piie like product really [67:54] does improve over the next 10 years and [67:58] so when I say you know regulation is not [68:00] enough what I mean is it needs movement [68:03] it needs culture it needs people who are [68:06] actually building and making you know in [68:09] like modern creative critical ways not [68:11] just like giving it up to you know [68:14] companies or small groups of people [68:16] right we need lots of different people [68:18] experimenting with strategies for [68:19] containment isn't it predicted that this [68:21] industry is A1 15 trillion dollar [68:23] industry or something like that yeah [68:26] I've heard that it is a lot so if I'm [68:28] rishy and I know that I'm going to be [68:30] chucked out of office Rish is the prime [68:33] minister of the UK If I'm going to be [68:34] trucked out of office in two years [68:35] unless this economy gets good I don't [68:38] want to do anything to slow down that [68:39] $15 trillion dollar bag that I could be [68:42] on the receiving end of I would I would [68:44] definitely not want to slow that 15 [68:45] billion trillion dollar bag and give it [68:47] to like America or Canada or some other [68:50] country I'd want that $15 trillion doll [68:53] windfall to be on my country right so I [68:58] have I have no other than the long-term [69:00] you know health and success of humanity [69:03] in my four-year election window I've got [69:05] to do everything I can to boost these [69:07] numbers right and get us looking good so [69:10] I I could give you lip [69:12] service but but but listen I'm not going [69:15] to be here unless these numbers look [69:18] good right exactly that's another one of [69:22] the problems short-termism is everywhere [69:26] who is responsible for thinking about [69:29] the 20-year [69:31] future who is it I mean that's a deep [69:34] question right I mean we we we the world [69:36] is happening to us on a decade by decade [69:39] time scale it's also happening hour by [69:41] hour so change is just ripping through [69:44] us and this arbitrary window of [69:46] governance of like a four-year election [69:49] cycle where actually it's not even four [69:51] years because by the time you've got in [69:53] you do some stuff for six months and [69:54] then by month you know 12 or 18 you're [69:58] starting to think about the next cycle [69:59] and are you going to pull you know this [70:01] just like the short-termism is killing [70:03] us right and we don't have an [70:06] Institutional body whose responsibility [70:10] is [70:11] stability you could think of it as like [70:13] a you [70:15] know like a global technology stability [70:19] function what is the global strategy for [70:23] containment that has the ability to to [70:25] introduce friction when necessary to [70:28] implement the precautionary principle [70:31] and to basically keep the [70:33] peace that I think is the missing [70:36] governance piece which we have to invent [70:39] in the next 20 years and it's insane [70:41] because I'm basically [70:43] describing the UN Security Council plus [70:47] the World Trade Organization all these [70:50] huge you know Global institutions which [70:53] formed after you know the horrors of the [70:55] second world war have actually been [70:58] incredible they've created [71:00] interdependence and alignment and [71:02] stability right obviously there's been a [71:04] lot of bumps along the way in the last [71:05] 70 years but broadly speaking it's an [71:08] unprecedented period of peace and when [71:11] there's peace we can create prosperity [71:14] and that's actually what we're lacking [71:16] at the moment is that we don't have an [71:18] international mechanism for coordinating [71:20] among competing Nations competing [71:24] corporations um to drive the peace in [71:26] fact we're actually going kind of in the [71:29] opposite direction we're resorting to [71:30] the old school language of a clash of [71:33] civilizations with like China is the new [71:36] enemy they're going to come to dominate [71:37] us we have to dominate them it's a it's [71:39] a battle between two polls China's [71:41] taking over Africa China's taking over [71:43] the Middle East we have to count I mean [71:46] it's just like that can only lead to [71:49] conflict that just assumes that conflict [71:51] is inevitable and so when I say [71:54] regulation is not enough no amount of [71:56] good regulation in the UK or in Europe [71:59] or in the US is going to deal with that [72:01] Clash of civilizations language which we [72:04] seem to have been become addicted to if [72:07] we need that Global collaboration to be [72:09] successful here are you optimistic now [72:13] that we'll we'll get it because the same [72:14] incentives are at play with climate [72:16] change in AI you know why would I want [72:17] to reduce my carbon emissions when it's [72:19] making me loads of money or why you know [72:22] why would I want to reduce my AI [72:24] development when it's going to make us [72:25] 15 trillion yeah so the the the really [72:29] painful answer to that question is that [72:32] we've only really ever driven extreme [72:36] compromise and consensus in two [72:40] scenarios one off the back of [72:44] unimaginable catastrophe and suffering [72:46] you know Hiroshima Nagasaki and the [72:49] Holocaust and World War II which drove [72:52] 10 years of consensus and new political [72:55] structures right and then the second is [73:00] um we did fire the bullet though didn't [73:02] we we fired a couple of those nuclear [73:05] bombs exactly and that that's why I'm [73:07] saying the brutal truth of that is that [73:09] it takes a catastrophe to [73:13] trigger the need for alignment right so [73:16] that that's one the second is where [73:19] there is an obvious mutually assured [73:22] destruction um you know [73:25] Dynamic where both parties are afraid [73:29] that this would trigger nuclear meltdown [73:32] right and that means suicide and when [73:35] there was few parties exactly when there [73:38] was just nine people exactly you could [73:40] get all nine but in in when we're [73:42] talking about artificial technology [73:43] there's going to be more than nine [73:44] people right that have P access to the [73:47] full sort of power of that technology [73:49] for NE various reasons I don't think it [73:51] has to be like that I think that's the [73:54] challenge of containment [73:55] is to reduce the number of actors that [73:58] have access to the existential threat [74:01] Technologies to an absolute minimum and [74:03] then use the existing military and [74:07] economic incentives which have driven [74:09] World Order and peace so far um to to [74:12] prevent the proliferation of access to [74:14] these super intelligences or these agis [74:17] a quick word on hu as you know they're a [74:19] sponsor of this podcast and I'm an [74:20] investor in the company and I have to [74:22] say it's moments like this in my life [74:24] where I'm extreme busy and I'm flying [74:26] all over the place and I'm recording TV [74:28] shows and I'm recording shows in America [74:30] and here in the UK that hu is a [74:33] necessity in my life I'm someone that [74:36] regardless of external circumstances or [74:38] professional demands wants to stay [74:40] healthy and nutritionally complete and [74:42] that's exactly where heal fits in my [74:43] life it's enabled me to get all of the [74:45] vitamins and minerals and nutrients that [74:48] I need in my diet to be aligned with my [74:50] health goals while also not dropping the [74:52] ball on my professional goals because [74:54] it's convenient and because I can get it [74:56] online in Tesco in supermarkets all over [74:58] the country if you're one of those [75:00] people that hasn't yet tried hu or you [75:01] have before but for whatever reason [75:03] you're not a Hu consumer right now I [75:06] would highly recommend giving hu a go [75:09] and Tesco have now increased the [75:11] listings with hu so you can now get the [75:13] RTD ready to drink in Tesco expresses [75:15] all across the UK 10 areas of focus for [75:18] containment you're the first person I've [75:20] met that's really hazarded a laid out a [75:23] blueprint for the things that need to be [75:24] done [75:26] um cohesively to try and reach this [75:28] point of containment so I super excited [75:29] to talk to you about these the first one [75:30] is about safety and you mentioned there [75:33] that's kind of what we talked about a [75:33] little bit about there being AIS that [75:35] are currently being developed to help [75:37] contain other [75:40] AIS two [75:42] audits um which is being able to from [75:45] what I understand being able to audit [75:47] what's being built in the these open [75:48] source models three choke points what's [75:52] that yeah so choke point refers to [75:56] points in the supply chain where you can [75:59] throttle who has access to what okay so [76:02] on the internet today everyone thinks of [76:04] the internet as an idea this kind of [76:07] abstract Cloud thing that hovers around [76:09] above our heads but really the internet [76:12] is a bunch of cables those cables you [76:15] know are physical things that transmit [76:18] information you know under the sea and [76:22] you know the those points the end points [76:24] can be [76:25] stopped and you can monitor traffic you [76:27] can control basically what traffic moves [76:31] back and forth and then the second choke [76:33] point is access to chips so the gpus [76:38] graphics processing units which are used [76:41] to train these super large clusters I [76:44] mean we now have the second largest [76:46] supercomputer in the world today uh at [76:50] least you know just for this next six [76:51] months we will other people will catch [76:53] up soon but we're ahead of the curve [76:54] we're very luy [76:55] cost a billion dollars and those chips [76:59] are really the raw commodity that we use [77:02] to build these large language models and [77:05] access to those chips is something that [77:07] governments can should and are um you [77:11] know restricting that's a choke point [77:13] you spent a billion dollars on a [77:14] computer we did yeah it's bit more than [77:16] that actually about [77:20] 1.3 a couple of years time that'll be [77:23] the price of an iPhone [77:25] that's the problem everyone's going to [77:27] have [77:28] it number six is quite curious you say [77:31] that um the need for governments to put [77:32] increased taxation on AI companies to be [77:35] able to find um fund the massive changes [77:37] in society such as paying for reskilling [77:40] and education yeah [77:43] um you put massive tax on over here I'm [77:45] going to go over [77:47] here if you tax it if I'm an AI company [77:49] and you're taxing me heavily over here [77:51] I'm going to Dubai yep or Portugal yep [77:56] so if it's that much of a competitive [77:58] disadvantage I will not build my company [78:00] where the taxation's high right [78:04] right so the way to think about this is [78:07] what are the strategies for containment [78:09] if we're agreed that long-term we want [78:12] to contain that is close down slow down [78:16] control both the proliferation of these [78:18] Technologies and the way the really big [78:20] AIS are used then the way to do that is [78:24] to tax things tax things taxing things [78:27] slows them down and that's what you're [78:29] looking for provided you can coordinate [78:31] internationally so you're totally right [78:34] that you know some people will move to [78:35] Singapore or to Abu Dhabi or Dubai or [78:38] whatever the reality is that at least [78:41] for the next you know sort of period I [78:43] would say 10 years or so the [78:45] concentrations of intellectual you know [78:48] horsepower will remain the big mega [78:52] cities right you know I I moved from [78:54] from London in 2020 to go to Silicon [78:57] Valley and I started my new company in [78:59] Silicon Valley because the concentration [79:01] of talent there is overwhelming all the [79:03] very best people are there on in in Ai [79:06] and software engineering so I think it's [79:09] quite likely that that's going to remain [79:11] the case for the foreseeable future but [79:13] in the long term you're totally right [79:15] how do you it's another coordination [79:16] problem how do we get nation states to [79:19] collectively agree that we want to try [79:22] and contain that we want to slow down [79:24] because [79:25] as we've discussed with the [79:26] proliferation of dangerous materials or [79:28] on the military side there's no use one [79:30] person doing it or one country doing it [79:33] if others race ahead and that's the [79:35] conundrum that we face I am I don't [79:38] consider myself to be a pessimist in my [79:39] life I consider myself to be an optimist [79:42] generally I think and I always I think [79:44] that as you've said I think we have no [79:45] choice but to be optimistic and I have [79:47] faith in humanity we've done so much so [79:49] many incredible things and so overcome [79:50] so many things and I also think I'm [79:52] really logical as in I'm the type of [79:54] person that needs evidence to change my [79:56] beliefs either way um when I look at all [79:59] of the whole picture having spoken to [80:01] you and several others on this subject [80:03] matter I see more reasons why we won't [80:06] be able to contain than reasons why we [80:07] will especially when I dig into those [80:10] incentives um you talk about incentives [80:13] at length in your book um at different [80:15] different points and it's clear that all [80:17] the incentives are pushing towards a [80:20] lack of containment especially in the [80:22] short and Midterm which tends to happen [80:23] with new technology in the short and [80:25] Midterm it's like a land grab the gold [80:27] is in the Stream we all rush to get the [80:29] the shovels and the you know the cves [80:31] and stuff and then we realize the [80:33] unintended consequences of that [80:35] hopefully not before it's too [80:37] late in chapter 8 you talk about [80:40] Unstoppable incentives at play here the [80:43] coming wave represents the greatest [80:44] economic prize in [80:47] history and scientists and technologists [80:50] are all too human they crave status [80:52] success and Legacy [80:55] and they want to be recognized as the [80:57] first and the best they're competitive [80:59] and clever with a carefully nurtured [81:01] sense of their place in the world and in [81:04] history [81:06] right I look at you I look at people [81:08] like Sam um from open AI [81:12] Elon you're all [81:14] humans with the same understanding of [81:16] your place in history and status and [81:19] success you all want that right [81:23] right there's a lot of people that maybe [81:25] aren't as don't have as good a track [81:28] record of you at doing the right thing [81:29] which you certainly have that will just [81:31] want the status and the success and the [81:33] money incredibly strong incentives I [81:36] always think about incentives as being [81:37] the thing that you look at when you want [81:39] to understand how people will behave all [81:41] of the incentives on a on a geopolitical [81:44] like on a global level suggest that [81:48] containment won't happen am I right in [81:51] that [81:52] assumption that all the incentives [81:54] suggest containment won't happen in the [81:56] short or midterm until there is a c a a [81:59] tragic event that makes us forces us [82:01] towards that idea of containment or if [82:05] there is a threat of mutually assured [82:08] destruction right so that and that's the [82:11] case that I'm trying to make is that [82:14] let's not wait for something [82:16] catastrophic to happen so it's [82:18] self-evident that we all have to work [82:20] towards containment right I mean you you [82:23] would have thought that the Potential [82:26] Threat the potential idea that [82:31] covid-19 [82:33] was a side effect let's call it of a [82:37] laboratory in Wuhan that was exploring [82:40] gain of function research where it was [82:42] deliberately trying to basically make [82:45] the pathogen more transmissible you [82:48] would have thought that warning to all [82:50] of us let's let's not even debate [82:52] whether it was or wasn't but just the [82:54] fact that it's conceivable that it could [82:56] be that should really in my opinion have [83:00] forced all of us to instantly agree that [83:04] this kind of research should just be [83:05] shut down we should just not be doing [83:07] gain of function research on what planet [83:09] could we possibly persuade ourselves [83:12] that we can overcome the containment [83:14] problem in biology because we've proven [83:16] that we can't cuz it could have [83:19] potentially got out and there's a number [83:20] of other examples of where it did get [83:21] out of other diseases like foot and [83:23] mouth disease [83:25] mhm back in the '90s in the UK so but [83:29] that didn't change our Behavior right [83:32] well foot and mouth disease clearly [83:33] didn't cause enough harm because it only [83:35] killed a bunch of cattle right um and [83:38] the pandemic we can't seem you know [83:40] covid-19 pandemic we can't seem to agree [83:42] you know that it really was from a lab [83:45] and not from a bunch of bats right and [83:49] so that's where I struggle where you [83:52] know now you catch me in a moment where [83:53] I feel angry and sad and pessimistic [83:57] because to me that's like a [83:59] straightforwardly obvious conclusion [84:01] that you know this is a type of research [84:04] that we should be closing down and I [84:05] think we should be using these moments [84:09] to give us insight and wisdom about how [84:11] we handle other technology trajectories [84:15] in the next few decades should we should [84:18] we should that's what I'm advocating for [84:20] must that's the best I can do I want to [84:22] know will will I think be a [84:25] low I can only do my best I'm doing my [84:28] best to advocate for it I mean you know [84:30] like I'll give you an example like I [84:32] think autonomy is a type of AI [84:35] capability that we should not be [84:37] pursuing really like autonomous cars and [84:40] stuff well I I autonomous cars I think [84:43] are slightly different because [84:44] autonomous cars operate within a much [84:46] more constrained physical domain right [84:49] like you know you you really can the [84:52] containment strategies for autonomous [84:53] cars are quite reassuring right they [84:56] have you know GPS control you know we [84:59] know exactly all the Telemetry and how [85:01] exactly all of those you know components [85:03] on board a car operate and we can [85:06] observe repeatedly that it behaves [85:08] exactly as intended right whereas I [85:12] think with with other forms of autonomy [85:14] that people might be pursuing like [85:16] online okay you know where you have an [85:18] an AI that is like designed to [85:20] self-improve without any human oversight [85:23] or a battle Battlefield weapon which you [85:26] know like unlike a car hasn't been you [85:28] know over that particular moment in the [85:31] battlefield millions of times but is [85:33] actually facing a new enemy every time [85:35] you know every single time and we're [85:37] just going to go and you know allow [85:39] these autonomous weapons to have you [85:41] know the these autonomous military [85:43] robots to have lethal [85:45] Force I think that's something that we [85:47] should really resist I don't think we [85:50] want to have autonomous robots that have [85:53] lethal Force you're a super smart guy [85:55] and I I struggle to believe that you're [85:58] you you because you you demonstrate such [86:00] a clear understanding of the incentives [86:02] in your book that I struggle to believe [86:04] that you don't think the incentives will [86:07] win out especially in the short and near [86:09] term and then the problem is in the [86:10] short and near term as is the case with [86:12] most of these waves is we we wake up and [86:17] 10 years time ago how the hell did we [86:19] get here right and why like and we and [86:21] as you say this precautionary approach [86:23] of we should have ranged the Bell [86:24] earlier we should have sounded the alarm [86:25] earlier but we waltzed in with optimism [86:28] right and with that kind of aversion to [86:31] confronting the realities of it and then [86:34] we woke up in 30 years and we're on a [86:36] leash right and there's a big rottweiler [86:38] and we're we've lost control we've lost [86:40] you know I [86:44] I I I would love to [86:47] know someone as smart as you I don't I [86:50] don't believe can be can believe that [86:52] containment is [86:55] possible and that's me just being [86:57] completely honest I'm not saying you're [86:58] lying to me but I just can't see how [86:59] someone as smart as you and in the know [87:01] as you can believe that containment is [87:03] going to happen well I didn't say it is [87:05] possible I said it must be right which [87:07] is this is what we keep discussing right [87:09] that's an important distinction is that [87:11] on the face of it look what I I care [87:13] about I care about science I care about [87:16] facts I care about describing the world [87:19] as I see it and what I've set out to do [87:22] in the book is des describe a set of [87:25] interlocking incentives which drive a [87:27] technology production process which [87:30] produces potentially really dangerous [87:33] outcomes and what I'm trying to do is [87:36] frame those outcomes in the context of [87:38] the containment problem and say this is [87:40] the big challenge of the 21st century [87:43] containment is the challenge and if it [87:45] isn't possible then we have serious [87:47] issues and on the face of it like I've [87:49] said in the book I mean the CH the first [87:50] chapter is called containment is not [87:52] possible right the last chapter is [87:53] called con must be possible for all our [87:55] sakes it must be possible but that but I [87:58] agree with you that I'm not I'm not [87:59] saying it is I'm saying this is what we [88:02] have to be working on we have no choice [88:04] we have no choice but to work on this [88:06] problem this is a critical [88:08] problem how much of your time are you [88:10] focusing on this problem basically all [88:13] my time I mean bu building and creating [88:15] is about understanding how these models [88:19] work what their limitations are how to [88:21] build it safely and ethically I mean we [88:23] have designed the structure of the [88:25] company to focus on the safety and [88:28] ethics aspects so for example we are a [88:30] public benefit Corporation right which [88:32] is a new type of corporation which gives [88:35] us a legal obligation to balance profit [88:40] making with the consequences of our [88:44] actions as a company on the rest of the [88:47] world the way that we affect the [88:48] environment you know the way that we [88:51] affect people the way that we affect [88:53] users that people who aren't users of [88:55] our products and that's a really [88:59] interesting I think and important New [89:02] Direction it's a new Evolution in [89:04] corporate structure because it says we [89:06] have a responsibility to proactively do [89:09] our best to do the right thing right and [89:12] I think that if if you were a tobacco [89:15] company back in the day or an oil [89:17] company back in the day and your legal [89:19] Charter said that your directors are [89:22] liable in if they don't meet the [89:25] criteria of stewarding your work in a [89:29] way that doesn't just optimize profit [89:31] which is what all companies are [89:32] incentivized to do at the moment talking [89:33] about incentives but actually in equal [89:36] measure attends to the importance of [89:38] doing good in the world to me that's a [89:42] incremental but important innovation in [89:45] how we organize society and how we [89:48] incentivize our work so it doesn't solve [89:51] everything it's it's it's not a Panacea [89:53] but that's my effort to try and take a [89:56] small step in the right direction do you [89:57] ever get sad about it about what's [89:59] happening yeah for sure for sure it's [90:08] intense it's intense it's a lot to take [90:10] in this is it's a it's a very [90:14] real [90:17] reality does that weigh on [90:20] you yeah it does I mean every day every [90:23] day I mean I've been working on this for [90:25] many years now and it's uh you know it's [90:28] it's [90:29] emotionally a lot to take in it's it's [90:32] it's hard to think about the far out [90:35] future and how your actions today our [90:39] actions collectively our weaknesses our [90:42] failures that you know that irritation [90:44] that I have that we can't learn the [90:47] lessons from the pandemic right like all [90:50] of those moments where you feel the [90:53] frustration governments not working [90:55] properly or corporations not listening [90:58] or some of the obsessions that we have [91:00] in culture where we're debating like [91:03] small things you know and you're just [91:06] like [91:07] Whoa We need to focus on the big picture [91:10] here you must feel a certain sense of [91:12] responsibility as well that most people [91:14] won't carry because you've spent so much [91:17] of your life at the very cutting edge of [91:18] this technology and you understand it [91:20] better than most you can speak to it [91:22] better than most so you have a a great [91:24] chance than many at [91:27] steering that's a [91:29] responsibility yeah I embrace that I try [91:32] to treat that as a [91:35] privilege I feel lucky to have [91:39] the opportunity to try and do that [91:42] there's this wonderful thing in my [91:44] favorite theatrical play called Hamilton [91:47] where he says history has its eyes on [91:50] you do you feel [91:52] that yeah I feel the I feel that I feel [91:55] that it's a good way of putting it I do [91:58] feel [92:01] that you're happy [92:04] right well what is happiness to [92:10] know um what's the range of emotions [92:13] that you you contend with on a on a [92:16] frequent basis if you're being [92:19] honest I think [92:24] is kind of exhausting and exhilarating [92:27] in equal measure because for me it is [92:32] beautiful to see people interact with [92:36] AIS and get huge benefit out of it I [92:39] mean you know every day now millions of [92:42] people have a super smart tool in their [92:46] pocket that is making them wiser and [92:48] healthier and happier providing [92:50] emotional support answering questions of [92:53] every type [92:54] making you more intelligent and so on [92:56] the face of it in the short term that [92:57] feels incredible it's amazing what we're [92:59] all [93:00] building but in the longer term it is [93:04] exhausting to keep making this argument [93:07] and you know have been doing it for a [93:10] long time and in a weird way I feel a [93:13] bit of a sense of relief in the last six [93:15] months because after chat gbt and you [93:19] know this this wave feels like it's [93:21] started to arrive and everybody gets it [93:24] so I feel like it's a shared problem [93:27] now and uh that feels nice it's not just [93:32] bouncing around in your head a little [93:33] bit it's not just in my head and a few [93:35] other people at Deep Mind and open Ai [93:37] and other places that have been talking [93:39] about it for a long [93:40] time ultimately human beings May no [93:43] longer be the primary planetary drivers [93:46] as we have become accustomed to being we [93:48] are going to live in an Epoch where the [93:50] majority of our daily interactions are [93:52] not with other people but with a eyes [93:54] page [93:56] 284 of your [93:59] book The Last Page [94:01] [Laughter] [94:04] yeah think about how much of your day [94:08] you spend looking uh [94:11] screen 12 hours pretty much right [94:14] whether it's a phone or an iPad or a [94:16] desktop versus how much time you spend [94:19] looking into the eyes of your friends [94:21] and your loved [94:22] ones and so to me it's like we're [94:26] already there in a way you know what I [94:30] meant by that was you [94:33] know this is a world that we're kind of [94:37] already in you know the last three years [94:39] people have been talking about metaverse [94:40] metaverse metaverse and the [94:43] mischaracterization of the metaverse was [94:45] that it's over there it was this like [94:48] virtual world that we would all Bop [94:49] around in and talk to each other as [94:51] these little characters and but that was [94:54] totally wrong that was a complete [94:56] misframing the metaverse is already here [95:01] it's the digital space that exists in [95:04] parallel time to our everyday life it's [95:07] the conversation that you will have on [95:10] Twitter or you know the video that [95:12] you'll post on YouTube or this podcast [95:14] that will go out and connect with other [95:16] people it's that meta space of [95:19] interaction you know and I use meta to [95:22] mean Beyond this space not just that [95:26] weird other over there space that people [95:29] seem to point to and that's really what [95:33] is emerging here it's this parallel [95:36] digital space that is going to live [95:38] alongside with and in relation to our [95:41] physical world your kids come to you you [95:44] got kids no I don't have kids your [95:46] future kids if you ever have kids a [95:48] young child walks up to you and says [95:51] asks that question that Elon was asks [95:53] what should I do about with my future [95:55] what should I pursue in the light of [95:58] everything you know about how artificial [96:00] intelligence is going to change the [96:01] world and computational power and all of [96:03] these things what should I dedicate my [96:05] life to what do you say I would say [96:08] knowledge is power [96:11] Embrace understand grapple with the [96:14] consequences don't look the other way [96:16] when it feels [96:18] scary and do everything you can to [96:22] understand and part anticipate and shape [96:25] because it is [96:30] coming and if someone's listening to [96:32] this and they want to do something to [96:33] help this battle for which I think you [96:36] present as the solution [96:38] containment what can the individual [96:42] do read listen use the tools try to make [96:48] the [96:48] tools understand the current state of [96:51] Regulation see which organization are [96:54] organizing around it like you know [96:56] campaign groups activism groups you know [97:00] find solidarity connect with other [97:03] people spend time online ask these [97:05] questions mention it at the pub you know [97:09] ask your parents ask your mom how she's [97:11] reacting to you know talking to Alexa or [97:13] whatever it is that she might do pay [97:16] attention I think that's already enough [97:19] and there's no need to be more [97:21] prescriptive than that because I think [97:22] people are [97:24] creative and independent and will it [97:26] will it will be obvious to you what you [97:30] as an individual feel you need to [97:32] contribute In This Moment provided [97:34] you're paying [97:37] attention last question what if we fail [97:41] and what if we succeed what if we fail [97:43] in containment and what if we succeed in [97:45] containment of artificial [97:48] intelligence I honestly think that if we [97:51] succeed this is going to be the the most [97:55] productive and the most meritocratic [97:57] moment in the history of our species we [98:00] are about to make intelligence widely [98:04] available to hundreds of millions if not [98:07] billions of people and that is all going [98:09] to make us smarter and much more [98:11] creative and much more productive and I [98:13] think over the next few decades we will [98:16] solve many of our biggest Social [98:18] Challenges I really believe that I [98:21] really believe we're going to reduce the [98:22] cost of energy production storage and [98:24] distribution to zero marginal cost we're [98:26] going to reduce the cost of producing [98:28] healthy food and make that widely [98:29] available to everybody and I [98:33] think the same trajectory with healthc [98:35] care with Transportation with [98:37] education I think that ends up producing [98:42] radical abundance over a 30-year period [98:45] and in a world of radical abundance what [98:46] do I do with my day I think that's [98:49] another profound question and believe me [98:50] that is a good problem to have if we can [98:53] absolutely if do we don't need meaning [98:55] and purpose and oh man that is a better [98:58] problem to have than what we've just [98:59] been talking about for the last like 90 [99:02] minutes yeah and I think that's [99:04] wonderful isn't that amazing I don't [99:06] know I I don't know the reason I I I'm [99:08] unsure is because everything that seems [99:11] wonderful has a has a unintended [99:13] consequence I'm sure it does we live in [99:15] a world of food abundance in the west [99:17] and our biggest problem is obesity right [99:19] so I'll take that problem in the grand [99:21] scheme of everything not need struggle [99:25] do we not need that kind of meaningful [99:27] voluntary struggle I think we'll create [99:30] new other you know opportunities to [99:34] Quest okay you know I I think that's an [99:37] easier problem to solve and I think it's [99:38] an amazing problem like many people [99:40] really don't want to work right they [99:42] they want to pursue their passion and [99:43] their Hobby and you know all the things [99:45] that you talk about and so on and [99:47] absolutely like we're now I think going [99:49] to be heading towards a world where we [99:51] can liberate people from the the Les of [99:53] work unless you really want to Universal [99:56] basic income I've long been an advocate [99:58] of Ubi very long time everyone gets a [100:01] check every month I don't think it's [100:03] going to quite take that form I actually [100:06] think it's going to be that we basically [100:09] reduce the cost of producing basic Goods [100:12] so that you're not as dependent on [100:14] income like imagine if you did have [100:16] basically free energy and food and you [100:20] you you could use that free energy to [100:21] grow your own food you could grow in a [100:23] desert because you would have adapted [100:25] seeds and so on you would have you know [100:28] desalination and so on that really [100:30] changes the structure of cities it [100:31] changes the structure of Nations it [100:33] means that you really can live in quite [100:35] different ways for very extended periods [100:38] without contact with the kind of Center [100:40] I mean I'm actually not a huge advocate [100:42] of that kind of libertarian you know wet [100:44] dream but like I think if you think [100:46] about it in theory it's kind of a really [100:49] interesting Dynamic that's what [100:50] proliferation of power means power isn't [100:52] just about access to intelligence it's [100:54] about access to these tools which allow [100:57] you to take control of your own destiny [100:59] and your life and create meaning and [101:01] purpose in the way that you you know [101:03] might Envision and that's incredibly [101:05] creative incredibly creative time that's [101:08] what success looks like to me [101:11] and well in some ways the downside of [101:14] that I think this that failure is not [101:17] achieving a world of radical abundance [101:20] in my opinion and and more more [101:22] importantly failure is a failure to [101:25] contain right what does that lead [101:29] to I think it leads to a mass [101:31] proliferation of power and people who [101:33] have really bad you know intentions what [101:37] does that lead to will potentially use [101:39] that power to cause harm to others this [101:42] is part of the challenge right a small [101:45] in this networked globalized World a [101:48] tiny group of people who wish to [101:51] deliberately cause harm [101:53] are going to have access to tools that [101:56] can instantly quickly have large scale [101:59] impact on many many other people and [102:02] that's the challenge of proliferation is [102:04] preventing those Bad actors from getting [102:06] access to the means to completely [102:09] destabilize um our world that's what [102:12] containment is [102:15] about we have a closing tradition on [102:17] this podcast where the last guest leaves [102:18] a question for the next guest not [102:19] knowing who they're leaving the question [102:21] for the question left for you is [102:24] what is a space or place that you [102:27] consider the most [102:32] sacred well I think one of the most [102:35] beautiful places I remember going to as [102:38] a child was um windir Lake in the Lake [102:43] District um and I was pretty young and [102:48] on a on a dingy with uh some family [102:52] members [102:53] and I just remember it being incredibly [102:56] Serene and beautiful and and calm I [102:58] actually haven't been back there since [103:01] but that was a pretty beautiful place [103:05] seems like the antithesis of the world [103:06] we live in right maybe I should go back [103:09] there and chill [103:11] out maybe thank you so much for writing [103:13] such a great book it's wonderful to to [103:15] to read a book on this subject matter [103:17] that does present Solutions because not [103:19] many of them do and it presents them in [103:21] a balanced way that appreciates both [103:23] sides of the argument doesn't isn't [103:25] tempted to just play to either what do [103:27] they call it playing to like the crowd [103:29] they call like playing to the orchestra [103:30] I can't remember right but just it [103:31] doesn't attempt to play to either side [103:33] or Ponder to either side in order to [103:34] score points it seems to be entirely [103:37] nuanced incredibly smart and Incredibly [103:41] necessary because of the stakes that the [103:43] book confronts um that are at play in [103:46] the world at the moment and and that's [103:49] really important it's very very very [103:51] important and it's important that I [103:52] think everybody reads this book it's [103:54] incredibly accessible as well and I said [103:56] to Jack who's the director of this [103:58] podcast before we started recording that [104:01] there's so many term there's so many [104:02] terms like [104:04] nanotechnology and um all the stuff [104:06] about like biotechnologies and Quantum [104:09] Computing that reading through the book [104:11] suddenly I understood what they meant [104:13] and these had been kind of exclusive ter [104:15] terms and Technologies and I also had [104:17] never understood the relationship that [104:19] all of these Technologies now have with [104:21] each other and how like robotics [104:23] emerging with artificial intelligence is [104:25] going to cause this whole new range of [104:27] possibilities that again have a good [104:30] side and a potential downside um It's a [104:32] Wonderful book and it's perfectly timed [104:34] it's perfectly timed wonderfully written [104:36] perfectly timed I'm so thankful that I [104:38] got to read it and I highly recommend [104:40] that anybody that's curious on this [104:41] subject matter goes and gets the book so [104:44] thank you Mustafa really really [104:45] appreciate your time and hopefully it [104:46] wasn't too uncomfortable for you thank [104:48] you this was awesome I loved it it was [104:50] really fun and uh thanks for such a [104:52] amazing amazing wide ranging [104:54] conversation thank [104:57] you if you've been listening to this [104:59] podcast over the last few months you'll [105:01] know that we're sponsored and supported [105:03] by Airbnb but it amazes me how many [105:05] people don't realize they could actually [105:07] be sitting on their very own Airbnb for [105:09] me as someone who works away a lot it [105:11] just makes sense to Airbnb my place at [105:13] home whilst I'm away if your job [105:15] requires you to be away from home for [105:17] extended periods of time why leave your [105:19] home empty you can so easily turn your [105:22] home into an Airbnb and let it generate [105:23] income for you whilst you're on the road [105:26] whether you could use a little extra [105:27] money to cover some bills or for [105:29] something a little bit more fun your [105:31] home might just be worth more than you [105:32] think and you can find out how much it's [105:34] worth at airbnb.co [105:36] /host that's [105:38] airbnb.co slost [105:41] [Music] [105:53] ah [106:00] [Music]