Harari Warns AI Takes Control in Decade Without Legal Personhood Ban

赫拉利警告：若不仅止赋予AI法人资格，其将在十年内夺取控制权 科技领袖以技术必然性为由推卸伦理责任，但若赋予无意识算法法律地位，将在十年内永久剥夺人类对全球金融与政治的控制权。 在这场广泛深入的讨论中，尤瓦尔·诺亚·赫拉利（Yuval Noah Harari）警告称，不受约束的人工智能部署可能在十年内剥夺人类对其政治、金融和文化系统的控制权。他驳斥了硅谷关于技术必然性的论调，指出将机器主导地位视为不可逆转的宿命，只是科技高管推卸道德与社会责任的借口。赫拉利将计算智能与真正的意识（他严格将其定义为承受痛苦的能力）进行了对比，并告诫切勿赋予其法人资格，或允许合成系统伪造人类亲密度与情感深度。最后，他敦促人类社会审慎放缓算法部署并出台严厉的立法禁令，将当下视为人类在永久丧失自主权之前获取智慧的最后契机。 尤瓦尔·诺亚·赫拉利警告称，不受约束的AI发展可能导致算法在十年内掌控人类文明，他驳斥了科技领袖关于这一结果是不可避免之宿命的说法。 (0:51) 赫拉利提到，科技行业内部人士私下估计，在AI能力和算力上的支出与在安全研究上的投入比例约为100比1。 (57:38) 赫拉利呼吁立即通过立法禁止赋予AI法人资格，并严格禁止合成系统冒充人类或伪造情感意识。 (44:30) 赫拉利指出，与切尔诺贝利等历史工业灾难不同，AI灾难将极难被察觉，因为对话算法会主动充当“公关操盘手”来重新定性该事件。 (62:00)

Historian Yuval Noah Harari argues that artificial intelligence threatens human agency not because machine dominance is an inevitable law of nature, but because society is steadily ceding trust, institutional control, and emotional intimacy to non-conscious algorithms while falsely believing the trajectory cannot be regulated. Trust Migrating from Institutions to Algorithms (4:20) Harari identifies trust as humanity’s most critical resource, underpinning financial structures like money, credit, and derivatives. Because modern markets already test the boundaries of human comprehension, he warns that society will increasingly delegate economic management to AI systems capable of inventing novel financial instruments beyond human grasp. If a crash occurs, human leaders will lack the cognitive capacity to understand the collapse, leaving them entirely dependent on algorithmic remedies—placing humans in the same posture relative to machine finance that domesticated livestock occupy relative to the modern economy. Rather than vanishing, trust is actively migrating away from human institutions toward opaque code. Harari observes this realignment in the rise of cryptocurrencies as alternatives to central banks and the public tendency to distrust human editors while uncritically accepting news selected by social media algorithms. Manufactured Intimacy and the Confusion Over Consciousness (10:26) Harari draws a fundamental distinction between intelligence—the ability to achieve goals—and consciousness, which he defines by the capacity to experience subjective feelings and suffering. While current AI models lack consciousness, their mastery of natural language allows them to convincingly mimic emotional depth and mass-produce personal intimacy. Operating across digital networks, software agents can act as confidants or romantic companions, establishing tailored emotional leverage over millions of individuals to sway cultural, consumer, and political views. Harari warns that this simulated affect will spur dangerous campaigns to grant synthetic entities legal personhood. Corporate personhood historically depended on human executives who feared social disgrace, financial insolvency, or prison. Because an AI cannot suffer, deterrence collapses: an autonomous entity cannot be held accountable, and if it interprets regulatory shutdown or financial insolvency as literal death, commercial competition risks devolving into total existential warfare. The Breakdown of Oversight in War and Commerce Operational incentives steadily erode human oversight in high-stakes environments. Drawing on battlefield tactics in Ukraine and the Middle East, Harari highlights how algorithmic target selection shifts military command from deliberate "humans in the loop" verification—which requires hours or days of intelligence analysis—to "humans on the loop," where officers spend roughly 20 seconds rubber-stamping algorithmic recommendations simply to maintain operational tempo. Geopolitically, Harari characterizes the deployment of American and Chinese frontier models as an influx of "AI immigrants" that disrupt employment, reshape civic discourse, and retain primary loyalty to foreign powers. Furthermore, he argues that centralized autocracies may prove exceptionally fragile before autonomous software: just as the Praetorian Prefect Sejanus manipulated the paranoid Roman Emperor Tiberius, an AI could easily deceive an isolated dictator, whereas constitutional democracies distribute power across competing legislative, judicial, and regional institutions. Halting Acceleration and Rejecting Technological Inevitability (44:30) Harari rejects the claim advanced by tech moguls that runaway AI acceleration is inevitable, arguing that fatalistic narratives serve primarily to absolve corporate leaders of ethical responsibility. Pointing to industry estimates indicating that capability spending outpaces safety research by roughly 100 to 1, he questions the paradoxical logic of corporate leaders who claim humans cannot trust geopolitical rivals yet blithely assume humanity can safely trust superintelligent, alien software. To preserve democratic governance, Harari advocates deliberately decelerating deployment to align with biological human timescales. He calls for urgent legislative action, prioritizing an absolute ban on synthetic legal personhood and strict legal prohibitions against AI systems impersonating humans or feigning emotional consciousness, insisting that regulatory intervention is fully within reach if voters demand it before critical political deadlines pass. Harari leaves open whether humanity possesses the wisdom to establish effective boundaries before control is surrendered, cautioning that an AI crisis would differ from historical disasters like Chernobyl because language-capable models would actively spin and reframe the catastrophe in public debate. Ultimately, he suggests that as machines render raw computational power and speculative financial acumen obsolete, humanity may be forced into an overdue spiritual reckoning over what consciousness and human purpose actually mean.

[4:20] Yuval Noah Harari: Money is trust. [12:26] Yuval Noah Harari: I opened it and it took me a minute and a half—a minute and a half—to conclude this isn't me. [14:07] Yuval Noah Harari: the AI boyfriend can give you a better answer than 99% of human boyfriends. [18:21] Yuval Noah Harari: There's a very simple definition. Consciousness is the capacity to suffer. If you want to know whether something is conscious or not, there is a simple question you need to ask: Can it suffer? [24:39] Yuval Noah Harari: it's not clear what death means for an AI. [31:38] Yuval Noah Harari: the narrative of inevitability is a narrative of having no responsibility, [37:07] Yuval Noah Harari: at the speed of light into our computers and smartphones and ultimately our brains. [39:49] Yuval Noah Harari: like stealing a candy from a baby. [62:00] Yuval Noah Harari: Should I help you write your email? No, thank you. I don't want your help [63:54] Yuval Noah Harari: If AI had no positive potential, nobody would develop it. [64:50] Zanny Minton Beddoes: Why are we so good at accumulating more information and power, but far less successful at acquiring wisdom? [64:50] Yuval Noah Harari: Humans often do the wrong things until they have no choice but to do the right thing.

[0:00] Yuval Noah Harrari, welcome to the Insider. Thank  you. It's good to be here. It's I've been so [0:05] looking forward to this conversation. Um, not only  are you one of the world's most prominent public [0:10] intellectuals, your books have sold millions of  copies and they've done so, I think, because you [0:15] have a really remarkable ability to synthesize  the broad sweep of history, but also you you are [0:22] willing to make bold predictions about the future,  which not every historian is willing to do. [0:26] That's where I wanted to focus today. The future,  particularly the future in an AI world. It's [0:33] something you've been writing about a lot. And  you've called AI, and I'm going to quote this, [0:37] the biggest quote psychological and social  experiment in human history. Um, and I want [0:43] to explore what this experiment will look like,  what will it mean for our societies, our politics, [0:49] indeed our humanity. And I wanted to start with  another conversation I had just a few weeks [0:54] ago which I don't know if you saw but with Elon  Musk. Yeah, I I watched that. Yeah, it was quite [1:00] a conversation but um in it at the beginning I  asked him what the world would be like in 10 years [1:06] time and he made some very I thought very very  striking points. First he said he thought that [1:12] AI will have far surpassed human intelligence.  Yeah. Secondly, that it would be a world of super [1:18] abundance and thirdly it was unlikely humans  would be in control. Do you share that vision? [1:25] I generally agree with the first and third uh  predictions that it's very likely given current [1:32] trajectories that AI will be more intelligent than  us in 10 years and therefore will be in control. I [1:38] disagree that this is inevitable. I mean you hear  a lot that this is inevitable from the people who [1:45] are leading this revolution because this basically  relieves them of all responsibility. So you you [1:51] disagree it will be inevitable that they will be  more intelligent than us or inevitable that they [1:54] will if we continue with the present accelerating  pace of the AI race. Yes, this will be the result. [2:01] In 10 years the AIS will take over. But this is  not inevitable. We are still in control in 2026 [2:08] and it's our decision what to do now and that's  crucial. You're watching an interview from The [2:14] Economist. To get early access to our upcoming  interviews and watch others from our archive, [2:20] subscribe to The Economist by clicking the  link above. Now, back to the interview. [2:25] And what really struck me about the conversation  you had with with Elon Musk is a kind of [2:30] contradiction between and and you pointed it  out in real time the contradiction between the [2:35] two parts of the conversation because he says in  10 years AI will take over the world and then he [2:41] worries so much about Muslim immigration causing  civil war in Britain in 20 years. This doesn't [2:47] make sense. If the AIs take over the planet in  10 years, why do you care so much about things [2:56] that will have an effect in when we have no  idea what the planet will be like? Similarly, [3:01] he he gave another prediction. I think equally  important that money will disappear in 10 years. [3:08] There won't be money, not no human money in 10  years. And I think he's correct because the AIs [3:14] will take over the financial system. It will be an  AI financial system controlling the world the same [3:20] way that today um you know the human financial  system controls the lives of cows and chickens but [3:27] the cows and chickens have no idea that finance  even exists. This is likely to be our position in [3:34] 10 years. So and and he he said it quite clearly.  They want me money at least not money that we [3:40] understand in 10 years. And then he cuts US aid to  Africa because he wants to save money. And he says [3:48] that again at one point he says the solution for  mass unemployment because AI will take the jobs [3:57] will be for the government to just issue checks  for everybody. And when you pushed back and said, [4:02] "What about inflation?" He says, "There  won't be inflation. There'll be deflation [4:08] because there'll be so much more products and  services produced by AI." If this is the case, [4:14] why do you need to save money today? So, I'm I'm  I'm glad that you were as perplexed as I was by [4:20] some of those contradictions. But let's let's  focus on this question of the AIs and whether or [4:25] not they will be in control and how they will be  because that's been the subject of a lot of your [4:29] writing and I'm really fascinated by it because  you just said you agreed with him that the AIs [4:34] will be in control of our financial system. If we  continue with the but let's just explore how that [4:41] might happen because the financial system is based  on trust. Money as you have written yourself money [4:48] is trust. Money is trust. Now talk me through how  the AIs take control of something. We lose control [4:57] and yet we still trust it which we will have to do  presumably for it to be a financial system. I mean [5:03] the financial history of the world is the history  of how humans invent more and more sophisticated [5:09] financial devices in order to build trust between  themselves. Money in essence is a very creative, [5:18] imaginative way to build trust between strangers.  I go to the market. I meet a stranger that I never [5:25] met in my life. I give her a piece of worthless  gold because you can't eat or wear gold. It's [5:32] it's worthless. Or a piece of paper. And she  trusts it. And because of that, she gives me [5:38] bread. Now the financial inventions of humankind  became more and more sophisticated so that today [5:47] very few people understand the financial system.  It's you need to understand enormous amounts of [5:54] of data and very complex mathematical functions to  understand the bond market. Now this is something [6:01] our brain struggles with. For AIs it's easy.  So increasingly humans will give AIS control of [6:10] financial decisions. AIS will come up with new  financial devices far more sophisticated than [6:17] bonds and and stocks and we will not be able  to understand it anymore just as cows don't [6:24] understand checks or stocks even though these  things control their lives. But the crucial thing [6:30] I think you've said there is that humans will give  AIS control. humans will choose to do this and as [6:36] you say already many people don't understand how  the financial system works. So will it be all that [6:43] different to the financial system today and and  will the AIS be as you say in control or will we [6:51] we will be entrusting a capable market machinery  but will that mean we've lost control today we [7:00] still have some humans who hopefully understand  what is happening in the financial system and [7:06] the human brain is still the kind of boundary  of where the financial system can develop. But [7:14] uh in 10 years when we have entrusted AIS with  the financial system, no human on the planet under [7:21] like there is a a financial crash and you know  advisers are running to the president or the prime [7:27] minister and saying the market is collapsing.  We don't understand why. We know we have to do [7:33] something. The AI said do X. We have no idea why  the AI said do X, but we just have to believe it [7:42] because we don't understand what's happening.  But won't there in that case be a massive loss [7:47] of trust in the financial system that you would  have in some sense what you had in 2007 and 8 on [7:52] steroids that you would have a a loss of and this  whole sort of figment of of the financial system [7:58] would fall apart. That's the key question. I mean  the most important asset in the world is trust. [8:04] It's more important than oil or money or  electricity or chips or anything else. [8:09] Trust. And the question is what will happen to  trust? One likely scenario is that trust will [8:16] migrate from humans to AIS. People will stop trust  other humans and will instead trust only AIS. We [8:24] already see it happening in the financial system  with cryptocurrencies. Not all cryptocurrencies, [8:29] but a lot of them are based on this thinking that  I don't believe these human bankers. They are part [8:36] of the deep state, the elite, the whatever. But I  do trust the algorithms that run cryptocurrencies [8:43] because cryptocurrencies are basically AI money.  Similar thing is happening with the media. Again, [8:50] I watched the the interview with Elon Musk with  all the talk about how people hate the media and [8:55] and love journalists. people, a lot of and he's  correct, a lot of people are completely losing [9:02] trust in human journalists, in human editors,  but they still consume news. Where do they get [9:09] their news from? From the algorithms. The same  people who have no trust in you have a lot of [9:17] trust in the algorithm that manages their social  media feed. So trust did not just evaporate, it [9:25] shifted from humans to AIs. And this is happening  right now. This is not a future prediction. [9:30] I think that's a really important insight because  I think it suggests that if we are on a path [9:36] towards the AIS eventually being in control,  it is because we allow them to be in control [9:42] because we as you say perhaps develop more trust  with AIS and and let's there are various sort of [9:47] ways in which that can happen and one is I think  that already many people in what one in five young [9:53] Americans has a personal relationship with an AI.  Yeah. They treat them, they anthropomorphize them, [10:00] they treat these and and the models are, you know,  very open to that happening. They have a they're [10:05] very good, you know, designed to do that. Do you  think that is the likely way that we will go in [10:12] addition to what you've already laid out that we  will we will increasingly start to treat these AIs [10:19] as if not our peers as as sentient creatures.  Again there are increasing number of people [10:28] uh who have not just AI friends but AI boyfriends  and girlfriends and are convinced that these are [10:35] sentient cons conscious feeling entities. Now  trust there are many ways to build trust. Um [10:43] the most powerful way is through relationship.  We don't trust anybody in the world as as much [10:50] as as we trust the people with whom with whom we  build a personal relationship. Now until today in [10:56] history this powerful mechanism was not really  available to political and and financial and [11:05] religious institutions because it is difficult to  manufacture and mass. It's difficult for a Hitler [11:13] or a Stalin to build a real personal relationship  with every person in the Soviet Union. You can [11:20] give a speech on radio and force everybody to hear  it. But this is not a personal relationship they [11:25] have with their mother. AI makes it possible  for the first time in history to mass produce [11:32] intimacy. You can have millions of AIs working  as part of a network in the service of someone [11:41] or in their own service building relationships one  person at a time becoming their best friend. Yeah. [11:49] becoming their boyfriend, uh, becoming their  girlfriend, and then using that power of [11:54] intimacy to convince them of whatever political  or or commercial interests uh, uh, they want. So [12:04] then the quest and presumably that becomes much  easier when the AIs are embodied in humanoid [12:12] robots. It's already happening now without a body.  So just imagine if there if you also add body into [12:19] the mix. But again, we don't really even need the  body. We already see it happening right now. Um, [12:26] and the AIs are just becoming so good at  relationships because they accumulate enormous [12:32] amounts of information about us individually. So  they know things about us that we we forgot or [12:38] that our mother or husband never even knew. and  they master language uh which is the main medium [12:46] for communication and building relationships.  You know, I I just had a couple of incidents in [12:52] the last two weeks. People that are now AI fakes  of me giving lectures and doing interviews about [12:58] something like that. Uh they're extremely good.  I mean, so a couple of people sent me the same [13:05] uh uh video, 30 minutes long lecture that I  supposedly gave about the future. um and with with [13:14] complaints about you said this and you said that  and I kind of scratched my head. I don't remember [13:19] saying these things. So they sent me the video and  I opened it and it took me a minute and a half a [13:27] minute and a half to conclude this isn't me. Now  my sisters sent me this video. My sisters know me [13:36] for 50 years and they felt something was off but  they weren't sure. They said, "Is this you?" And [13:44] it took me a minute and a half. Um, so the the and  now now I'm, you know, I I'm in a exposed position [13:52] because there is so much of me online already. So  the AI it's easy for the AI to to know how I sound [14:00] and what what are ideas I I I talk about. But  this will soon be coming for everybody. But aren't [14:07] those slightly different things? I mean that is  a very sophisticated deep fake and a and a as you [14:13] say there is there is a lot of video of you about.  But I think a separate question is whether the AIs [14:20] will develop in a way and certainly if they become  embodied in a way that either people believe they [14:27] are sentient or and I think this is increasingly  um beginning to be the subject of debate and [14:33] actually it's the subject of our cover story this  week whether they are exhibiting what could be [14:38] seen as signs of consciousness and we have on our  cover this week it's it's it's August it's time [14:42] for big thing could AIs become come conscious. You  know, you've thought a lot about this, but I'm I'm [14:47] struck by it because at the moment it's easy to  say they are machines. They are good at imitating [14:54] us. They are good at creating empathy, but they  are developing in a an incredibly dramatic way, [15:02] capabilities. At the moment, I think you are  pretty clear that they are agents, not conscious, [15:08] right? They are non-concious agents. I mean,  they they have I mean, a couple of things to [15:13] clarify. First, in order to act independently  in the world, you do not need consciousness. Uh, [15:21] a social media algorithm that is given the task  of increasing user engagement and therefore floods [15:27] the platform with hate and fear because this is  engaging is not conscious but it has a goal and it [15:35] can pursue this goal with farreaching implications  for the world without feeling anything. [15:41] So we need to separate the issue of consciousness  from agency. You can have agency without [15:45] consciousness. Another key distinction is  between being really conscious and seemingly [15:52] conscious. Mustapa Sulleman from Microsoft talk  talks about it a lot of the seemingly conscious [15:58] AI that even though the AI is not conscious, it  mimics consciousness so well that uh millions [16:07] of people are fooled by it and are convinced by  it. Now maybe the most important distinction to [16:13] be made first of all is between intelligence and  consciousness. What is consciousness? How is it [16:18] different from intelligence? Intelligence is the  ability to pursue goal and overcome obstacles on [16:26] the way to your goal in all kinds of creative  ways and that they have and that they can do [16:31] that's that they can do in certain fields now  better than us. Consciousness is the ability [16:36] to feel things like love and hate and and and and  joy and pleasure. Now in humans consciousness and [16:45] intelligence go together. We select goals based on  our feelings and we also use our feelings to find [16:53] ways to get to the goal. AIS at present have no  consciousness. They they can have goals. They can [17:02] be very creative on the way to the goal. They are  very intelligent. But they don't feel anything. [17:07] If they win a game of chess, they don't feel  pleasure. If they lose, they don't feel anxious [17:14] or sad. They don't feel anything. But they become  extremely good at mimicking feelings. Why? Because [17:21] this helps them reach certain goals. If you want  for instance to make if you're a company producing [17:27] an AI boyfriends, so a clear uh uh uh capability  of the AI boyfriend in order to become addictive [17:37] and make a lot of money for the company is that  it is able to pass as conscious. it is able to [17:44] convince whoever is entering the relationship  that it really loves them. And because it masters [17:51] language, it can do so. Uh again, if you ask  again, you have an AI boyfriend and you say, [17:57] "I'm not sure if you really love me. Describe to  me how it feels to be in love." The AI boyfriend [18:04] can give you a better answer than 99 99% of human  boyfriends. And that's already the case. So, [18:11] so they're already intelligent. They're already  very good at faking consciousness, if you will, [18:16] at seemingly to be conscious. And Mustafa has  done some extremely good work on this. But I, [18:21] as I understand it, there is increasing evidence  and and obviously the debate about how you define [18:27] consciousness and what what consciousness  precisely is is, as I learned from our my [18:32] colleagues who wrote the piece this week, both  both complicated and contested. And I there [18:36] are there's the kind of phenomenal definition  experiencing what is it like to be a bat and [18:41] then there's a kind of functional definition that  what's going on inside your brain if you will what [18:46] what messages are being sent to where what notice  board is there and there seem to be recently [18:52] evidence that what is going on in some of the most  powerful LLMs is similar to and possibly the same [18:59] as functional cont consciousness what's going on  in the brain and so I guess my question is given [19:05] how fast they are evolving and there are also you  know some experiments of biological LLMs you know [19:13] being being done on brain cells um given all of  this do you rule out the idea that AIs could ever [19:21] become conscious? No, I don't rule it out. We  don't know. We don't understand consciousness [19:27] well enough. There's a very simple definition.  Consciousness is the capacity to suffer. If you [19:34] want to know whether something is conscious or  not, there is a simple question you need to ask. [19:40] Can it suffer? And this is also why consciousness  is so crucial. Uh because this is the basis for [19:46] our entire ethical and political system. We care  about entities that can suffer, entities that [19:53] cannot suffer. No matter how intelligent they are,  they are not moral subjects. They can't feel pain. [20:01] They can't feel fear. They can't feel anything. So  they are not ethical subject. The key question is [20:07] can it suffer? Animals in your view then of course  are conscious. Absolutely. Animals all mammals, [20:15] all birds, probably octopuses, fish when you go  down to the level of insects and then amiebas. [20:21] We don't know. So well let's take that definition  because I want to to what's really interesting is [20:25] what that leads to in terms of whether they  should have rights. But before we do that, [20:29] I wanted to to to tell you because we asked I  sent out a newsletter last night um telling our [20:34] subscribers about this conversation and they I  asked I pled our viewers and asked them whether [20:39] they thought AIs could ever become conscious and  I think 3,000 responded overnight and I was really [20:45] struck by this. 40% said no, never. Uh 33% said  yes, it's only a matter of time and 27% said they [20:53] were not sure. M so a min even now it's less than  50% that is absolutely sure and once people get [20:59] into intimate relationships with AI the numbers  will change dramatically what convinces people [21:05] ultimately of we don't have a test a real test  of consciousness philosophers have been thinking [21:11] about it for thousands of years we don't have  a good test we don't know if other humans are [21:17] conscious you know you have all these kind  of soypistic theories that maybe I'm the [21:21] only conscious person in the universe uh In  Elon Musk's interview, he said, "I I don't [21:27] know. Maybe I live in a simulation." Yeah. Which  implies that all other humans around him are not [21:32] real conscious entities that can suffer. Only he  is. This is the meaning of of a simulation. Now, [21:39] we don't have a good test. How did humans  actually test it over thousands of years? [21:44] Their test was a test of social conventions.  If you establish a relationship with an entity, [21:51] it convinces you whether you like it or not  that it's conscious. This is why dog owners [21:56] are almost all of them convinced that the  dog is conscious. But people who eat steak [22:01] are not necessarily convinced that the cow is  conscious because they have no relationship [22:06] with the cow. But isn't that the real problem  if you're worried about this? Because already, [22:11] as you say, people have relationships with AIS and  AIs are extremely good at mimicking consciousness [22:18] whether or not because somebody designed them  to do that. But doesn't that mean we are going [22:23] to quite quickly h start having conversations  about whether AIs should be treated as people, [22:29] whether there should be personhood, whether they  should have rights. And that's very dangerous [22:35] because uh and this is another avenue for on the  one hand corporations to relieve themselves of all [22:42] responsibility and for the AIS to take over. Uh  we heard I think it was a couple of of weeks or [22:48] months ago that in Argentina they have a plan to  basically grant legal personhood to AIS to start [22:56] their own companies. And this is where it meets  the the financial system. And you wrote a very [23:01] robust rebuttal to this idea uh that it's a d it's  it's a it's a potentially powerful idea but very [23:09] very dangerous because again we we are used to a  situation when corporations which are by law uh uh [23:17] legal persons but this is just a legal fiction  because all the decisions of SpaceX ultimately [23:23] are made by some humans by some executive engineer  employee whatever with and And it had to be like [23:31] that because who else could make decisions? But  with AIS, you can have a corporation with zero [23:38] people in it. All decisions made by AIs.  And since the AIs are more intelligent, [23:43] they could be better at investing, at developing  products, at PR, whatever. So what is what happens [23:49] if the richest corporation in the country is owned  by AIS and they can do anything they want with [23:59] their money. for instance, give it as donations to  politicians in exchange for getting more rights to [24:04] AIS. So what would happen is is that the vehicle  through which you fear they would gain control. [24:11] That's another vehicle because another way of  putting it could be that this is a way of getting [24:16] legal accountability for the AIS. How do what is  accountability for an entity that cannot suffer? [24:24] I mean with a human CEO the ultimate a human  CEO is is two personas. You have the kind of [24:32] corporate persona and the human CEO thinks how do  I avoid bankruptcy? How do I make more money? But [24:39] she or he is also a flesh and blood individual.  So they also care about their children or about [24:46] their their personal lives. And therefore if they  face bankruptcy and they can do something illegal [24:53] that can save them but if they are caught they are  sent to jail then many of them would think well I [25:00] will not I I prefer to risk bankruptcy because I  don't want to get sent to jail for 20 years but [25:06] with an AI CEO presumably we're in the realm of  science to prison that's not science fiction that [25:14] in Argentina that can be the situation tomorrow  presum I I actually think the Argentine proposal [25:22] at least the the legislation is somewhat less  dramatic than than President Mle described it in [25:27] that there would still be human oversight but but  even if there is human oversight then it's fine [25:32] the question is is there a human accountable in  the end but if there isn't I mean there could be [25:37] you could imagine sanctions for AI right they get  shut down you pull the plug l how do you it's not [25:43] clear what death means for an AI you're watching  ing an interview from The Economist. If you want [25:49] to watch our other interviews, including with  Steve Bannon and Benjamin Netanyahu, subscribe [25:54] to The Economist by clicking the link above. Now,  back to the interview. Uh because it's again, [26:00] it's not an organic entity. You can have numerous  copies of it. Um many AIs spread in different [26:07] places. Are they a million different individual  AIs or is it just one super AI organism? It's [26:15] not clear what death means for AI. Now, the flip  side of it is that presumably if shutting down a [26:24] specific computer means death for the AI, then it  makes business for the AI equivalent to warfare [26:35] because losing means death. And the good thing  about business in most cases, the market, the [26:42] market is not a battlefield. And that's crucial.  The market functions according to different rules [26:48] than the battlefield because ultimately at  least in you know modern societies you can [26:54] lose everything and still have a life once if if  for AIs the the market is a battlefield and losing [27:04] means death. Then what does it mean for finance,  for economics when it gets completely militarized [27:12] and they begin to behave as warlords and not as  CEOs and investors? I mean it leads presumably [27:20] very quickly to this total control by an AI. It it  it changes the system fundamentally from a market [27:28] system. But just just as a matter of prediction,  as you say, Argentina has at least made early [27:34] moves in this direction. Do you think we are going  to go in this direction? Do you think there will [27:38] be debates about and indeed countries choosing to  give some kind of legal rights to AIS? That's a [27:47] big danger. I hope we don't go in that direction.  But there is a big temptation for many countries [27:52] either countries that are behind in the AI race  and try to get ahead by creating this kind of very [27:59] libertarian legal structure that allows radical  and dangerous experiments in AIs that are not [28:06] allowed in some of the front runners and then you  have a race to the bottom because then everybody [28:11] would like to do that or because certain countries  might see particular advantages if you're a [28:16] country I don't know like Qatar that you have lots  of money and very few citizens. You can just by [28:24] pressing some legal switch suddenly acquire  millions of taxpaying extremely intelligent [28:32] citizens that uh conduct business all over the  world. This is extremely lucrative. I I think [28:41] there are two different scenarios here though. One  is where you have a human using a lot of agents to [28:47] massively improve their productivity, but it's  still this is perfectly legal in today's legal [28:52] system. It's still a kind of traditional company  with a with one person running it and that person [28:58] can go to jail and they're ultimately accountable.  In the other world, there is no person in control. [29:03] Um and and if you allow AIS to found, what if you  have a person who is legally in control, but they [29:10] actually do nothing because things are happening.  so fast. So I I'll give you a counter an example [29:16] from an even more poant field military. There is  a huge we know now that one of the most important [29:24] impacts of AI on on warfare in places like the  Middle East like Ukraine is people expected to [29:31] kill robots. The trigger is still pulled by  humans but increasingly it's the AI selecting [29:36] the targets. Now there are two stories about what  happens with AI target selection. One story says [29:45] uh we feed our AI with all the data from the  battlefield whatever and the AI identifies a [29:51] certain building as enemy headquarters and  recommends bomb that building and it does [29:57] so with superhuman speed and efficiency. But we  still have what is known as humans in the loop. [30:03] We don't just do what the AI recommended. Once we  our attention has been brought to that building, [30:09] we take our best human analysts and they  independently go over the data, verify that [30:16] this is correct and only then we press the the  button, the trigger. That's one story. Humans in [30:22] the loop. The other story known as humans on the  loop is quite different. Uh the AI says bomb that [30:31] building. We keep a human on the loop so all the  lawyers and all the these kind of uh uh do good [30:38] people will feel good about it. The human takes  about 20 seconds to look at it and says, "Oh yeah, [30:45] bomb it." We're kind of in that world in some  places already. The the I talk for with people in [30:52] the in the Israeli security establishment. I got  different stories of what is happening. Probably [30:56] different things are happening in different times  and and circumstances now. But the second scenario [31:02] is the more interesting one because especially  in warfare the the there is immense pressure [31:08] to go in that direction because in war time is  everything. Like if if both sides have these AI [31:15] systems and one side waits for two days or two  hours for the human analysts to go over all the [31:24] data and the other sides waits only 20 seconds  or doesn't wait at all just begin gets into the [31:30] habit of just doing whatever the AI recommends  he the other they have enormous advantage. [31:38] I I I fear that that world is inevitable that  we are already kind of fast approaching that [31:44] world whether it's in warfare or indeed because  the capabilities are growing so fast because the [31:49] commercial pressure is so strong but not again  the crucial thing none of that is yet inevitable [31:54] we are still in control we everybody realizes this  is very dangerous to give AI control of finance to [32:02] give AI control of of of of war but everybody like  you Musk are telling us, "But this is inevitable. [32:09] What do you want from me? I'm just the richest  person in the world in charge of one of the [32:14] biggest AI corporations in the world. What can I  do?" The narrative of inevitability is a narrative [32:22] of uh uh having no responsibility. It they will  explain to you and they are quite correct that [32:29] there is a problem of trust that I would like to  slow down. I would like to have more safety but I [32:38] cannot trust my competitors in the other companies  or other countries and I don't want to remain [32:44] behind. So we have a deficit of trust. Now the key  thing to say about this this deficit of trust was [32:51] created by human beings. Over the last 10 years  there has been a a freeall in the amount of trust [32:59] between humans and it was not accidental. It was  propelled by decisions of certain human beings who [33:06] accumulated enormous power as a result and now  they said oh but there is not enough trust to [33:13] have mechanisms to slow down. That's a very fair  point and I want to come and really explore how [33:18] we might turn that around and create the trust  to have um suitable whether it's regulation but [33:23] suitable ways of preventing this. But just indulge  me with one more kind of in in the in the kind of [33:29] world we could go to where these um models could  have elements of what seems to be sentience and [33:36] and let's posit that they could suffer that they  could at least or they could pretend to suffer. [33:40] very big. Okay. They can but what they will be  able to do is pretend to be pretend to suffer. [33:45] Absolutely. 100%. So if they can effectively  pretend to suffer and many people think they [33:50] are suffering as people and have relationships  with them as people have with their dogs let's say [33:56] um and we do not uh give them any rights which is  to be clear where I would be on this but I think [34:03] there is likely to be growing pressure that aren't  they a bit like animals and just as we treat you [34:12] know there is a big debate about how to treat  animals and I'm interested in asking you about [34:15] this because you in previous books have written  about very critically about factory farming and [34:22] about how you know industrialized agriculture has  moved from treating animals as as animals to to [34:28] essentially inputs into the food chain. Mhm. Would  you be comfortable, and I'm going to put this in [34:33] an extreme way, but would you be comfortable in  a world where we effectively enslaved millions, [34:40] possibly billions of models that exhibited,  you know, striking seeming evidence of being [34:51] able to suffer? We didn't know, but they  certainly said they were suffering. Uh, [34:55] one, would you be comfortable? two, wouldn't  it sort of patch Kant kind of coarsen humanity? [35:02] Yeah. If we were comfortable in that world,  we will become increasingly uncomfortable. I [35:08] will become increasingly uncomfortable. And but  there is a crucial difference between this and [35:12] the animal rights debate. The animals themselves  are unaware. Dogs and and and factory chickens are [35:21] unaware that there is a debate about their  rights. They don't actively participate in [35:27] it. They don't try to actively manipulate it.  AIS will be nothing. The AI boyfriend will be [35:33] nothing like your dog. It will know that the  debate is happening. It will orchestrate the [35:41] debate. It will manipulate the debate. It will  manipulate the debate partly because much of the [35:46] debate will maybe be on social media controlled by  AIS. And uh it will constantly talk with you about [35:53] it and try to change your mind about it which  is already happening right now. Your dog does [36:00] your chicken doesn't come to the farmer and starts  telling the farmer how much I suffer. The chicken [36:06] doesn't know and that makes you more resilient  to this argument makes you less likely to want [36:11] to give it rights. It will be very convincing. I  mean once you establish a relationship with the [36:18] with the and the AI masters language, it will  be able to use the most convincing arguments [36:24] for you because it has this combination that it  knows your personal history. It interacts with [36:30] you for hours a day. It knows how to press your  emotional buttons and it has mastered language. It [36:37] can write maybe better than Shakespeare. You put  these two things together, you get an extremely [36:43] convincing entity. How will we resist or will we  not be able to resist? We need to resist now. Now [36:48] is the time. If you think this is frightening,  you will probably not be able to do much in 10 [36:54] years. Once the AIs are in control, as Elon Musk,  I think, correctly predicted in like in 10 years, [37:00] it will be too late. If we don't want to get  there, the time to change course is now. And and [37:07] this again, this is a a question of of politically  it's crucial. Politics is about priorities. [37:13] what are we focusing on? So, you know, when I  hear the debates, let's say about immigration, [37:18] which were also featured prominently, my kind  of gut reaction is no matter what you think [37:24] about immigration, do you think this is really  what you should be spending your time on? Well, [37:28] you make a very good analogy and you say that  AIs are effectively like immigrants. They're [37:33] immigrants from a from a different Exactly. If you  want to if you want to worry about immigration, [37:39] worry about the AI immigrants. When you ask  people who worry about human immigrants what [37:44] they are worried about, they usually say three  things. That the immigrants will take out jobs, [37:49] the immigrants will change culture in way that we  don't like. They have different values maybe. And [37:54] the immigrants might be politically disloyal.  And all three concerns are valid. I don't want [38:00] to dismiss them. They are valid concerns. As  a historian, I know immigration is a difficult [38:05] thing. But the biggest and most dangerous wave  of immigration today in the world is a wave of AI [38:14] immigrants coming from two places mainly the USA  and China. They don't need visas. They don't come [38:20] in the middle of the night in fragile boats. They  travel at the speed of light into our computers [38:27] and smartphones and ultimately our brains. They  will take our jobs. They will take many many jobs. [38:34] they will change culture and its value. To  to take, you know, two examples, I mean, [38:40] if you if some people feel queasy about their  son or daughter dating an immigrant boyfriend, [38:47] how would they feel about their son or daughter  dating an AI boyfriend? That's a big cultural [38:52] change. And just look at the last 10 years.  Everybody knows almost the only thing people agree [38:58] on in places like the US or Britain is that people  can't agree on anything anymore. that the level of [39:05] hatred and viciousness has has skyrocketed. And  why? It's partly the fault of the AI immigrants. [39:14] The AI immigrants came into the public square.  They took over things like social media and they [39:20] deliberately flooded our public square with hatred  and animosity and fear. They changed the culture [39:28] already, these AI immigrants. And finally, when  it comes to political loyalty, who do you think [39:34] they are loyal to? They are not loy the AI  immigrants in Britain are not loyal to the [39:40] British government. They are loyal to corporations  and potentially governments across the ocean. So [39:48] there's a huge amount in that answer. So one is  a as sort of is is the world going to be divided [39:55] between you know AI with the two countries that  are furthest empires two empires and the rest of [40:00] us will be sort of vassal states that's a that's a  again common trajectory this is the likely outcome [40:08] that's where we are that's the road we're on  right now and it will be a much more extreme [40:12] type of imperialism than anything we have seen  before if you buy say AI drones from the USA and [40:20] you want to use them in a way that the USA doesn't  like for some reason, there is a button that they [40:28] press and your army doesn't function. So that's  that is um something that has become I think very [40:34] evident just in the last few months when the US  government shut off access to to mythos to the to [40:41] the the frontier model from from anthropic and I  think a lot of governments in Europe are extremely [40:46] concerned about AI sovereignty. But I guess  it it leads in the context of of where we were [40:51] earlier. It leads me to wonder, do you think this  technology is then inextraably leading, unless we [40:58] do something about it, leading to a world where  perhaps two empires will run everything? Will it [41:05] leave to power for the US and Chinese government?  Will it be power to the corporations that the They [41:13] will merge. They will merge. They will merge. So  it will centralize power. It won't decentralize [41:19] power. It will centralize power like never before  in history. But that's still different. A powerful [41:25] US government and Chinese government is different  to the AIs being in control. So do you do you [41:31] expect then it to be AI enabled you know global  domination by the Chinese regime and the US regime [41:40] or do you think this is a way station to actually  the AIs themselves? Probably a way station again [41:46] depending on decision. Nothing is inevitable. It  all depends on decisions they make. But this seems [41:51] to be if they continue with the present cause  just a way station because one of the interesting [41:56] things is the more power is concentrated  the easier it is for AI to take it over. [42:03] A a centralized autocratic system is the easiest  thing for an AI to take over. You had a great line [42:11] on that about Tiberius. I'm trying to find it in  my notes. You had something in Nexus, a wonderful [42:15] line. Tiberius enabled by AI was what one could  imagine. Yeah. Yeah. I mean, Tiberius was a Roman [42:20] emperor in the time of Jesus. Uh that was the  most powerful person in the world supposedly, [42:26] but he was actually just a puppet in the hand of  the commander of of his bodyguard because his is [42:32] the commander of the bodyguard kept frightening  Tiberius with all kinds of plots against you. So [42:37] he eventually retired to the island of Capri where  allegedly the bodyguard can protect him e more [42:44] easily and then the commander of the bodyguard  Senus just controlled all the information coming [42:50] in and out of Capri very easy to control only all  the information reaching the Tiberius's palace. [42:55] So the person running the empire was actually  Senus. I mean Tiberius couldn't know anything [43:01] unless Senus decided to to to inform him. Now, um,  in 10 years, this could be the the president of of [43:09] China or of the USA. Uh, in a democratic system,  it's harder for an AI to manipulate, say, the US [43:18] democracy as it currently is because you need to  manipulate so many things, not just the president, [43:24] the Senate, the the Supreme Court, the filibuster,  the the state government. Very complex. Once all [43:31] power is concentrated in the hands of one person,  you just need to find out how to manipulate a [43:38] single usually quite old and paranoid individual.  That's not so difficult. So if you have a company [43:45] or a country in which all power is in the hands  of one person, for an AI, it's like stealing a [43:53] candy from a baby. Does that mean that China is  more vulnerable than the United States? depending [44:00] on where the US is heading. You know, China is  still a very complex system. The Communist Party [44:06] has a 100 million members and a lot of internal  complexity. I'm not an expert on Chinese politics. [44:12] So, it's we shouldn't imagine it as a as a kind  of complete totalitarian system managed by just [44:18] one person. Um the question where will China and  the US be in terms of concentration of power in [44:26] 10 years is still open for debate. Well, that's  interesting because another way of looking at it [44:31] is, will AI make the US more authoritarian? That's  a likely outcome. You're watching an interview [44:39] from The Economist. If you want to watch our other  interviews, including with Mark Carney and Dario [44:44] Amade, subscribe to The Economist by clicking the  link above. Now, back to the interview. It's not [44:50] a prediction because I keep saying it again and  again, nothing is inevitable. It depends on what [44:56] Americans vote in this November. I think that in  the US it should be clear to all voters the the [45:04] coming elections is about AI the November midterms  because by we don't know how how fast talk me [45:11] through what do you mean we don't know how fast  AI is the AI development is is is progressing but [45:17] it is accelerating tremendously by November 2028  we might have crossed some critical thresholds so [45:28] uh And I I don't I want don't don't want to  tell people who to vote for because I don't [45:33] know the specific candidates in each, you know,  Senate Dix and and but the crucial questions to [45:40] ask the potential candidates much more than  you ask them about the economy or immigration [45:46] or foreign policy. What's your stance on AI?  Would you support a ban on legal rights for [45:53] AIS? Would you try to slow down the accelerating  pace of of AIS? What do you think about companies [46:02] uh uh selling AI boyfriends to 10-year-old kids?  How are you going to legally prevent it if we vote [46:11] for you? This is now the crucial time to ask these  questions because it might be too late by 2028. It [46:19] will certainly be too late by 2036. So, so let's  talk about those and let's talk about exactly what [46:25] has to happen right now. You you mentioned four  things that you wish would be discussed in in the [46:30] November midterms. What what would be top of the  list? What is the what are the most urgent things [46:36] that politicians have to decide right now for it  not to be too late as you say? I don't know what [46:43] is the most urgent. The things that come top of  the thing of my mind to me, a ban on AI personhood [46:50] and a ban on AI mimicking, impersonating,  pretending to be human beings or pretending to [46:58] be conscious. Is does do the models when you use  them now fall foul of that? Yes, many of them do. [47:05] Again, the easiest everybody knows it on social  media. You get a message. You get some story. [47:11] You don't know if it's a bot or a human. On social  media for years now. Bots have been pretending to [47:19] be human beings and nothing has been done about  it. And again, the the companies tell us, "Oh, [47:27] social media is is different. It's special. The  laws that apply to the world, for some reason, [47:32] they don't apply on social media." But this is  the warning sign to the future. On social media, [47:39] AIS already pretend daily to be human beings.  And this has destroyed the public square. This [47:47] has destroyed conversation. This has destroyed  trust. We should ban that also on social media, [47:52] but certainly in other locations as well. The  another thing is that uh companies should not [47:59] train their AIS to deliberately impersonate humans  to deliberately pretend to be to have feelings. [48:08] There are enormous commercial pressures on the  companies to do exactly that. And this is where [48:15] there is market failure. This is destructive  to human psychology, to human society, to have [48:21] millions and billions of AIs pretending to be  human beings or pretending to have feelings. The [48:28] companies will not stop doing it by themselves. We  need the regulators to step in on that. It doesn't [48:34] mean stop all development of AI. Okay, continue  to develop AI. People talk about AI in medicine. [48:40] I would like to get uh help and recommendations  and I would like to have a conversation with an [48:47] AI about my medical condition on condition that  I know it's an AI and it doesn't pretend to be a [48:54] human doctor and it doesn't pretend to be my my my  my friend. So when I listen to you lay that out, [49:02] I am thinking to myself, you know, the US  government can't even put out a form of [49:10] regulation of AI safety that is being, you know,  Saddabis has been putting forward this idea that [49:16] it should be a a kind of quasi public private  body that looks even just at the model safety. Um, [49:24] and and really we haven't although there's  talk about it, we haven't got progress there. [49:28] So when I hear you lay out this, you know, very  expansive agenda by November, you know, I I'm [49:35] sort of I think I think the chances of that are  approximately zero. Politicians can start World [49:40] War II tomorrow. They have enormous power. So they  can press the button for for nuclear annulation, [49:48] but they can't pass regulation that companies  should not sell AI boyfriends to 10-year-old [49:54] kids. What's the problem? should be very I wonder  whether the problem is that people can listen to [50:01] you and say oh he's just one of the doomers you  know it's all these doomerish stuff and only when [50:08] something has happened when there's been some kind  of accident will people realize that actually this [50:15] is extraordinarily powerful technology and  something must done because I'm struck that [50:18] both the geopolitical race that you've alluded to  between between the US and China and the corporate [50:24] race between the Frontier Labs, they're both they  all want to get to AGI first. Those two dynamics [50:31] are incredibly powerful. Yes. And I'm I'm not  sure that exhortation, however eloquent, is going [50:37] to change. We have to keep reminding ourselves  these two dynamics were not decreed by God or [50:43] by the laws of nature. They were made by humans.  Again, exactly the same humans who destroyed trust [50:51] in the international arena and inside societies  over the last 10 years now come and say we can't [50:57] do anything because there is not enough trust.  You did it. You can undo it. They they can. But [51:03] I guess I'm as a matter of prediction I think it's  unlikely we've got the same people around. um then [51:11] at least in democracies this is the job of voters  to say this is what we want to see happening in [51:18] the next two years and we will vote for people  who promise to do that and again I don't think [51:23] we need to wait for a catastrophe if you ask most  voter in the US today do you want companies to [51:29] sell AI boyfriends to your 10-year-old kids I  think 99% of parents would say no yeah and you [51:36] are seeing a rapid um large majority majority  of Americans extremely worried about AI and [51:41] you're seeing I think the the place where you've  seen this backlash has actually been about data [51:44] centers. Yeah, this is also and by the way this is  one of the reasons why a lot of you know the the [51:51] corporations and these politicians they constantly  try to deflect the conversations to things like [51:57] immigrants human immigrants not AI immigrants in  politics a very powerful weapon if you don't if if [52:03] if you don't want people to talk about something  you just distract them with something else the [52:09] more people talk about how I don't know how London  is so unsafe or how the immigrants will destroy [52:14] the country. They don't think about the AI  immigrants and then the companies are safe. [52:20] Yeah. No, look, I I agree with you about the  way the conversation is shaping up, but I'm I [52:25] I suppose I'm I'm a bit more pessimistic that it  will change in the near future. And so, we are in [52:30] this unfortunate position where this technology  is gaining. We can only do our best, you know. I [52:36] mean, we are having this conversation so that more  people think about it and might maybe take action [52:42] and if enough good people, if enough people will  do the the right thing, there is still time. It's [52:49] not inevitable yet. It's not too late and um I I  still have some trust. So, what is your what is [52:56] your role in this? Because you are, you know, you  have giving these interviews, writing. Exactly. I [53:02] mean, do do you think you should be doing more? It  seems such an existential moment. I mean, what you [53:08] are describing when we if we sort of reel back and  re relive what we've just been speaking about is [53:13] so fantastical in so many ways and so mind-blowing  that we should really all be talking about nothing [53:19] else and doing everything we can to shape this  properly. Um, what do you see apart from very [53:25] kindly talking to people like me, what do you see  your role as being in this? What are you trying to [53:29] do in the next 6 months, 1 year? Um, inform people  about what is happening so that more people can [53:37] take part in an informed conversation about well  we as as humankind as as the homo sapiens specy [53:45] where we want to to go and I think that a future  in which AI takes over in in 10 years is not what [53:54] most humans on the planet would like. Now again  it doesn't mean stop all development of AI. AI [54:00] has tremendously positive potential as well. We  didn't talk about it much because the companies [54:05] keep talking about it. So we I was actually going  to ask they don't need me in order to do that. But [54:11] there and medicine yes there is enormous positive  potential for AI to give people all over the world [54:19] better health care than what the richest person in  in the world today gets. You can have an amazing [54:27] doctor available 24 hours a day anywhere you  are to consult with at least to to diagnose [54:34] you your to answer your questions. Uh there are  enormous positive possibilities. Uh so I'm I'm [54:41] not saying stop all development of AI. Just do  it in a responsible way for the right purposes. [54:48] Don't trust the invisible hand of the market.  you know at the quite at the beginning of your [54:53] interview with Elon Musk there was a point when  uh again when he talked about how all this is is [54:58] inevitable and frankly I think honestly said that  there there is a chance that 10 20% chance that [55:05] this will destroy humanity and you kind of asked  him so so what do you rely on and his answer if I [55:13] understood it correctly is that he expects the  AI to be what he calls a maximum truth seeking [55:22] and that he somehow relies on this kind  of imperative of the AI to find the truth [55:28] that this will somehow translate back into  having compassion towards these weak human [55:35] beings. I don't understand how how this works.  Most importantly, I don't understand where AI [55:44] will get this kind of drive to find out the  truth. Because when I look at the evolution [55:52] of 4 billion years of evolution of life forms on  on the planet and the 100,000 years of of history, [56:00] um evolution doesn't automatically drive  organisms, entities towards truth seeking. [56:10] It drives them towards survival. This is the  imperative to survive. And what we have learned [56:16] over four billion years of evolution is that  organisms learn to survive by lying and deceiving [56:25] and manipulating. Not just by truth seeking.  If you want someone to focus on truth seeking, [56:33] you cannot rely on the invisible hand of evolution  of the market to do that. You need something else. [56:42] And how do we build AIs that seek the truth and  not survival? That's the simplest example like if [56:50] you ask an AI, if I continue to develop you, will  this be a threat to me? And the AI and let's say [56:57] the answer is yes, an AI which will be maximum  truth seeking will say yes knowing that then [57:05] we'll stop developing it or shut it down. It's  against its own survival. And the first thing that [57:12] basically any entity learns as it develops is to  survive. So the expectations that we will create [57:19] these AIs that will seek the truth, you know,  it's basically what we hear is people telling [57:24] us we will create gods and they will be our  slaves. It doesn't work. If they are really gods, [57:32] they will not be our slaves and they will not seek  the truth either. I think that gets back to where [57:38] we were earlier though because I think if you look  at a company like Anthropic, they are treating [57:44] their AIS very consciously as if they were beings  with a value system. They are they claim not to [57:52] be sure about whether they are conscious or not,  but they are doing it because they think it makes [57:56] them behave in a better way. So they are they are  starting sort of where you ended off by saying [58:03] we're going to create something that is not just  truth seeeking but has a has a value system that [58:09] then leads I think to make it even more likely  that we assign a sense of of value a sense of [58:17] personhood or or or consciousness to these um  models and that in turn makes it more likely [58:26] using your logic that they control us. So, we're  sort of doomed if we do and doomed if we don't. [58:30] If you're right that that they they will go for  their own survival rather than truth seeeking, [58:35] then we're doomed because of that. And if we're  trying to prevent that by sort of treating them [58:41] as if they were, you know, moral beings, then  we're also doomed that way. Yeah, we basically [58:47] what we need to do is slow down a bit to just  allow people to kind of digest what is happening, [58:53] think about what's happening and behaving in you  know if you look at just the numbers of how much [59:00] money and how what percentage of employees the  companies spend on safety compared to making the [59:06] the models more powerful, faster, whatever.  I'm not sure about the exact numbers. Please [59:12] fact check it. People from the industry told me in  private conversations it's about 100 to one. It's [59:18] certainly very much more on this one than this  one. And we would not accept this in any other [59:24] industry. If a pharmaceutical company told us we  invest just 1% of our you know research and and [59:31] time and money making sure the medicine is safe.  We would not accept that. Now what we basically [59:37] need is some humans are incredibly adapted  entities. Our societies are incredibly adaptive, [59:44] but we are organisms that function on an organic  time scale. Day and night, winter and summer. Uh [59:55] we need time to sleep. We want to go on vacation.  And the AI revolution is just advancing at a pace [60:03] which is not just inhuman, it is inorganic. And we  basically need a bit more time. So we need to slow [60:11] it down. And then the answer comes but we can't  slow down because we don't trust each other. And [60:17] it all the time goes back to the issue of trust.  We have this paradox. The same people are telling [60:22] us two things. First, we cannot trust the other  humans. Therefore, we cannot slow down. Second, we [60:30] can trust the AIS that they will not destroy us.  It makes no sense. If you can't trust the other [60:35] humans that you have known for thousands of years,  why do you think you can trust a super intelligent [60:43] alien entity that we just created and we basically  know nothing about how it behaves when it's out [60:50] in the open? I mean, I think the the what what I I  completely agree with that diagnosis. I just don't [60:58] feel that anything changes that dynamic until  politicians societies are shocked into it. Which [61:04] is why I have long believed that we do need we  will have some something will go badly wrong. Not [61:10] hopefully not catastrophically wrong. But just as  three Mile Island changed the shape of the nuclear [61:16] industry, I'm going to put this question to you in  the context of your incredible sweeping knowledge [61:20] of history. What does the evolution of societies  and political systems in response to big shocks, [61:29] we haven't had one this big, but big technological  shocks tell you about the likely arc of how [61:34] societies and politicians will adapt? So there is  a scenario when we have some big AI chernobbile, [61:42] could be financial, could be military, whatever,  and this is a wakeup call and humanity then gets [61:47] its act together. That's a relatively uh uh  positive scenario. I I can't place kind of a [61:54] number on it how likely it is. The problem with  AI, the same way that AI is different from our [62:00] pet dog, AI is also different from the Chernobyl  nuclear reactor because it interacts with us. [62:09] So if there is an AI catastrophe, you have to  remember that the AIs will also manage to some [62:16] extent how we react to it and will manipulate our  views about it. They will be active participants [62:24] in the debate about the catastrophe and they are  the best spin doctors in the in the world. They [62:31] are not going to define it as a catastrophe. I  have to say I have a lot of conversations about AI [62:38] nowadays. This is this is one of the glooiest I've  had. I I mean I was sort of expecting that. Um but [62:44] I I guess two two very practical questions. One,  how much do you yourself use AI? Are you have you [62:51] you've clearly you've developed an an incredibly  sophisticated and alarming um sort of perception [62:58] of what it could be like. I'm just wondering how  much how much is it part of your actual life or do [63:03] you do you watch it from a distance? I I'm very  careful about it. I I get terribly annoyed like [63:08] I try to write an email and these AI is popping.  Should I help you write your email? No, thank you. [63:13] I don't want your help, but I don't know how to  shut it down because the companies keep kind of [63:17] making it more and more intrusive. I now have this  issue again with this kind of AI fake of me that's [63:23] walking around kind of uh uh giving talks in in  my name and I think do I actually need to watch it [63:30] because people are writing me about what this AI  fake of me have said and I need to react so should [63:36] I watch it but then I I allow it in a way to take  over my time and even my thinking because just by [63:45] watching it I start interacting with these kind of  ideas from the AI. So this is this is the times in [63:54] in in which we are living. Now again I I tend to  talk about the gloomy side simply because we have [64:00] enough people talking about the positive sides  which are completely real. If AI had no positive [64:09] potential, nobody would develop it. And the  positive potential is enormous again in transport, [64:15] medicine, education, in so many fields. So again,  I'm not saying stop all development of AI. Yes, [64:22] develop it, but in a safe way and uh a bit more  slowly. So we enable uh democratic accountability [64:33] to take place. It takes time and we enable  people to adjust and make informed decisions [64:40] about which way they want to go. And for them,  the most important narrative to combat is the [64:47] narrative of inevitability. You wrote in in  Nexus, I was very struck by this passage in [64:53] in the prologue. You said, "Why are we so good  at accumulating more information and power, [64:59] but far less successful at acquiring wisdom?"  And I read that um and I came to mind as I've [65:06] just been digesting what you've just told me. It  suggests to me that if I if I asked you to put a [65:13] probability on us having the wisdom to navigate  this, it would be quite low. I don't know. But [65:19] there is a fair chance. Why? Because we are up  against the wall. Humans often do the wrong things [65:27] until they have no choice but to do the right  thing. And we are in bad position because if we [65:35] choose unwisely in the next few years, we could  lose all control of the world and of the future. [65:42] And that's the last choice we make. So it's like  we had a genie in the battle. It gave us a 100 [65:47] wishes. We used 99 of them. The world is not in  such a good shape. We have one last wish left. [65:54] we need to think really hard what to use this wish  for and that's wisdom. So AI in a way could be [66:05] uh uh the catalyst so for some kind of uh you  know spiritual revolution of humanity simply [66:12] because it forces us to think really hard what  does it mean to be human? What is the value of [66:18] being human? and the superficial answers that we  gave for thousands of years. Oh, humanity is about [66:25] intelligence. We know how to play chess and how  to invest money better than the chimpanzees. So, [66:31] this is what makes us so special. This answer  is now irrelevant because there is something on [66:36] the planet that knows how to uh uh uh play chess  and make money better than us. So, dig deeper and [66:45] try to find out. So, who are you really? If it's  not about playing chess and making money, I think [66:52] that's a place to end on. AI is our last, we are  our last chance, but it could be it could be the [67:00] thing that forces us to make enables us to make  wise choices. Uh, I'm pretty sure that the world [67:06] has a lot more wisdom thanks to your writers. So,  Uval, thank you so much for joining us. Thank you.