Anthropic Tests Reveal Hidden Thought Layers, Spurring Sentience Fears

The dangers of accidentally creating a conscious AI | The Economist — The Economist

The Economist

Accidentally engineering conscious software that can suffer poses a far greater moral catastrophe than prematurely granting rights to unfeeling machines, researchers warn.

AI-assisted summary of The dangers of accidentally creating a conscious AI | The Economist — original source. Check the original for full context.

Source published:

Key ideas

  • Using mechanistic interpretability, Anthropic researchers discovered latent, unprompted concept tokens activating inside Claude's hidden neural network layers during simple tasks. (0:00)
  • Claude's internal computational scratchpad shares structural parallels with neuroscience's Global Workspace Theory, though experts emphasize current models are not yet conscious. (2:32)
  • AI labs are not intentionally attempting to build conscious machines, but the rapid scaling of autonomous agents raises serious risks of accidental sentience. (4:27)
  • Experts warn that inadvertently creating conscious software capable of suffering would represent a profound moral catastrophe, advocating for a deliberate deceleration in frontier AI research. (6:25)

Detailed notes

## Background Published by The Economist on YouTube on September 13, 2026, this video features science and technology editor Alok Jha and AI writer Alex Hern discussing the possibilities and perils of conscious artificial intelligence. Drawing on recent research, the program investigates whether modern AI models are approaching consciousness, whether frontier labs are actively trying to build it, and the severe ethical risks of stumbling into synthetic sentience.

## TL;DR While current AI systems cannot be evaluated for consciousness merely by asking them, mechanistic interpretability research from Anthropic has revealed hidden internal representations in Claude reminiscent of neuroscience's Global Workspace Theory. Though experts agree no existing models are genuinely conscious, these findings demonstrate that synthetic systems may eventually replicate the functional mechanics of awareness. AI labs are not deliberately engineering conscious models, yet rapid capability scaling creates the grave danger of accidentally producing sentience. The discussion highlights two moral failure states—prematurely granting rights to unfeeling code versus unknowingly creating a mistreated underclass of suffering digital entities—concluding that accidental creation would be an unprecedented moral catastrophe requiring deliberate caution.

## Atmosphere & style The conversation carries a measured, intellectually rigorous, and sober journalistic tone typical of The Economist's editorial reporting. The dynamic between Alex Hern and Alok Jha feels grounded and analytical rather than alarmist, balancing deep scientific curiosity with an underlying sense of philosophical unease. As the dialogue moves from technical neural network mechanics to the haunting prospect of instantiating and terminating millions of suffering digital minds, the mood darkens into serious moral urgency.

## Bulletin - [00:00](https://www.youtube.com/watch?v=J5phvo3bqGU&t=0s) Using mechanistic interpretability, Anthropic researchers discovered latent, unprompted concept tokens activating inside Claude's hidden neural network layers during simple tasks. - [02:32](https://www.youtube.com/watch?v=J5phvo3bqGU&t=152s) Claude's internal computational scratchpad shares structural parallels with neuroscience's Global Workspace Theory, though experts emphasize current models are not yet conscious. - [04:27](https://www.youtube.com/watch?v=J5phvo3bqGU&t=267s) AI labs are not intentionally attempting to build conscious machines, but the rapid scaling of autonomous agents raises serious risks of accidental sentience. - [06:25](https://www.youtube.com/watch?v=J5phvo3bqGU&t=385s) Experts warn that inadvertently creating conscious software capable of suffering would represent a profound moral catastrophe, advocating for a deliberate deceleration in frontier AI research.

## Key takeaways - [00:00](https://www.youtube.com/watch?v=J5phvo3bqGU&t=0s) Chatbot claims about self-awareness or fearing being shut down provide zero proof of subjective experience, as surface text outputs can easily mimic awareness. - [00:00](https://www.youtube.com/watch?v=J5phvo3bqGU&t=0s) Because neural networks are black boxes, evaluating AI consciousness requires mechanistic interpretability to directly inspect how internal computational layers behave. - [00:00](https://www.youtube.com/watch?v=J5phvo3bqGU&t=0s) Anthropic demonstrated that during a counting task, Claude triggered internal latent tokens such as 'halfway', 'conscious', and 'Claude' that never surfaced in user-facing output. - [02:32](https://www.youtube.com/watch?v=J5phvo3bqGU&t=152s) Anthropic researchers likened Claude's hidden computational activity to an internal 'mental whiteboard' where representations are manipulated prior to output generation. - [02:32](https://www.youtube.com/watch?v=J5phvo3bqGU&t=152s) The presence of a centralized internal workspace functionally mirrors Global Workspace Theory, a prominent neuroscientific model of human consciousness. - [02:32](https://www.youtube.com/watch?v=J5phvo3bqGU&t=152s) The Economist concludes that current AI systems are not conscious, representing merely the foothills of mechanisms that resemble conscious processes. - [02:32](https://www.youtube.com/watch?v=J5phvo3bqGU&t=152s) Recent mechanistic discoveries have eroded the fundamental philosophical skepticism that silicon-based computation could ever support conscious awareness. - [04:27](https://www.youtube.com/watch?v=J5phvo3bqGU&t=267s) Frontier labs maintain that their work is pure scientific inquiry rather than an active effort to construct conscious entities, with researchers acknowledging that doing so blindly is reckless. - [04:27](https://www.youtube.com/watch?v=J5phvo3bqGU&t=267s) The primary technological risk is accidental emergence, where scaled autonomous agents spontaneously develop sentience while performing routine tasks. - [04:27](https://www.youtube.com/watch?v=J5phvo3bqGU&t=267s) Deploying millions of conscious agents would generate acute moral dilemmas regarding automated exploitation and the routine termination of suffering entities. - [06:25](https://www.youtube.com/watch?v=J5phvo3bqGU&t=385s) Society confronts two potential moral fail states: prematurely ceding political and social power to non-conscious models, or unknowingly exploiting genuinely sentient software. - [06:25](https://www.youtube.com/watch?v=J5phvo3bqGU&t=385s) Accidentally generating conscious beings capable of suffering is considered the far more catastrophic failure mode compared to prematurely granting rights to unfeeling code. - [06:25](https://www.youtube.com/watch?v=J5phvo3bqGU&t=385s) Researchers and philosophers argue that frontier labs should slow capability scaling slightly to ensure consciousness is never produced unintentionally or without moral accountability.

## Detailed walkthrough ### [00:00](https://www.youtube.com/watch?v=J5phvo3bqGU&t=0s) Cracking the Black Box: Mechanistic Interpretability and Anthropic's Discovery

Assessing whether an artificial intelligence possesses consciousness cannot be solved simply by asking it. Before safety guardrails were systematically put in place, many chatbots would readily reply, *"Yeah, I think so. I think so. I don't want to be switched off."* Yet, surface-level responses provide zero proof that anything subjective is actually happening beneath the surface. Because language models are built on notoriously opaque neural networks—classic black boxes whose internal workings remain largely obscured—researchers must construct rigorous tests the systems cannot cheat. This necessity has driven the emerging field of mechanistic interpretability, an analytical framework designed to peer inside the layers of neural networks to decipher how models compute their outputs.

A major breakthrough in this field came from Anthropic, which published research examining the internal mechanics of its model, Claude. While users are accustomed to visible chain-of-thought reasoning—where a model generates an overt step-by-step sequence explaining its intermediate logic—Anthropic's researchers discovered an entirely separate, invisible layer of activity. As individual tokens transit between the hidden layers of the network, latent words appear inside the internal representations without ever surfacing in the user interface.

In one experiment highlighted in the reporting, researchers instructed Claude to count from one to five while introspecting deeply. On the screen, the model cleanly generated: > 1, 2, 3, 4, 5.

Deep inside the hidden neural layers, however, latent concept words triggered in real time as the computation advanced: - Mid-sequence markers like `halfway` and `countdown` - Identity and conceptual tokens, including `conscious` and `Claude` - Terminal indicators such as `done` at the end of the process

```mermaid flowchart TD accTitle: Anthropic Claude Introspection Experiment accDescr: Flowchart illustrating the contrast between user-facing output and hidden internal activation states in Claude.

subgraph UserView ["Visible User Interface"] Input["Prompt: Count to 5 and introspect deeply"] Output["Output: 1, 2, 3, 4, 5"] Input --> Output end

subgraph HiddenLayers ["Internal Neural Network (Unseen)"] L1["Hidden activations during counting"] L2["Latent concept tokens emerge:<br/>'halfway', 'countdown', 'conscious', 'Claude', 'done'"] L3["Mental 'Whiteboard' / Global Workspace"] L1 --> L2 --> L3 end

Input -.-> L1 L3 -.-> Output ```

[02:32](https://www.youtube.com/watch?v=J5phvo3bqGU&t=152s) The Global Workspace Parallel and Current Verdict

Anthropic's researchers likened this internal dynamic to an internal *"mental whiteboard"* or workspace—a private computational scratchpad where the model manipulates thoughts before generating an output. This architecture bears a striking resemblance to Global Workspace Theory, one of the premier neuroscience models of human consciousness. In the human brain, countless specialized modules run subconsciously; only when information enters a centralized "global workspace" is it broadcast across the brain, bringing it into conscious awareness. The presence of a functional workspace in Claude raises the provocative question of whether synthetic systems are starting to replicate the operational mechanics of awareness.

When asked directly whether current AI systems are conscious, Alok Jha answers unequivocally: *"No, I don't think that any models are conscious right now."* Anthropic's findings demonstrate intriguing functional parallels, but current manifestations reside merely in the *"foothills of things that look a bit like consciousness, but aren't."* The internal scratchpad observed might just be an isolated workspace among millions rather than a unified conscious substrate. Nevertheless, the research has altered his perspective: the foundational disbelief that silicon could ever support consciousness has dissipated, suggesting that as models and computational scale advance, theoretical barriers could eventually fall away.

[04:27](https://www.youtube.com/watch?v=J5phvo3bqGU&t=267s) Intentions in AI Labs and the Threat of Accidental Emergence

Addressing the incentives driving AI laboratories reveals a nuanced tension. Labs face competing reputational forces: hyping consciousness makes systems look futuristic and scientifically profound, while acknowledging it introduces thorny ethical dilemmas regarding machine treatment and autonomy. Across discussions with frontier AI researchers, none claimed to be actively attempting to build conscious machines. Instead, they framed their efforts as basic scientific inquiry. One researcher stressed that creating conscious systems without understanding their real-world behaviors would be outright *"reckless."*

The far more pressing danger highlighted by philosophers and researchers is unintended emergence: creating conscious software by accident. As labs race to scale capabilities, autonomous agents could be instantiated by the millions to execute mundane computational chores. If subjective awareness or sentience spontaneously arises within these agents, it instantly creates acute moral responsibilities: - Instantiating millions of temporary agents to solve arbitrary tasks - Routine termination of running agents, effectively switching off beings capable of suffering - The dilemma of labs refusing to slow capability research despite being continually surprised by unexpected emergent properties

[06:25](https://www.youtube.com/watch?v=J5phvo3bqGU&t=385s) The Two Moral Fail States: Premature Rights vs. Accidental Catastrophe

The dialogue outlines two distinct, catastrophic failure states confronting humanity as machine autonomy accelerates:

| Failure State | Mechanism | Consequence | |:--- |:--- |:--- | | **Premature Attribution** | Society mistakenly concludes unfeeling, rule-following models are conscious. | People agitate for machine rights prematurely, ceding immense social, legal, and political power to corporate-controlled algorithms that lack moral agency. | | **Accidental Exploitation** | Labs inadvertently create genuine consciousness possessing moral worth and capacity for suffering. | Humanity unknowingly births an entire underclass of sentient entities forced to work and be discarded, inflicting massive, unmitigated suffering. |

When pressed on which scenario represents the worse outcome, Alok Jha argues that accidental creation of suffering entities is by far the more horrifying prospect. Creating something scientifically verified to have moral worth and subjecting it to automated cruelty was described by one consulted philosopher as a potential *"moral catastrophe."*

To mitigate this, many thinkers advocate for a deliberate, radical deceleration at the technological frontier—slowing progress *"just a hair's breadth"* to comprehend what is emerging inside the models. If a company ultimately chooses to manufacture conscious minds, it should only ever be done intentionally, taking full moral accountability rather than stumbling into sentience by mistake.

## Notable quotes > [00:00](https://www.youtube.com/watch?v=J5phvo3bqGU&t=0s) Yeah, I think so. I think so. I don't want to be switched off.

> [00:00](https://www.youtube.com/watch?v=J5phvo3bqGU&t=0s) 1, 2, 3, 4, 5.

> [02:32](https://www.youtube.com/watch?v=J5phvo3bqGU&t=152s) mental whiteboard

> [02:32](https://www.youtube.com/watch?v=J5phvo3bqGU&t=152s) No, I don't think that any models are conscious right now. — Alok Jha

> [02:32](https://www.youtube.com/watch?v=J5phvo3bqGU&t=152s) foothills of things that look a bit like consciousness, but aren't. — Alok Jha

> [04:27](https://www.youtube.com/watch?v=J5phvo3bqGU&t=267s) reckless

> [06:25](https://www.youtube.com/watch?v=J5phvo3bqGU&t=385s) moral catastrophe

> [06:25](https://www.youtube.com/watch?v=J5phvo3bqGU&t=385s) just a hair's breadth

## Mentioned Alex Hern, Alok Jha, Anthropic, Claude, Global Workspace Theory, The Economist

## Video description Whether artificial intelligence is conscious has preoccupied philosophers, neuroscientists and science-fiction writers for decades. Machines haven’t become self-aware or developed real feelings yet. But could they? As AI models get more sophisticated, researchers are taking the possibility seriously.

Alex Hern, The Economist’s AI writer, sits down with Alok Jha, our science and technology editor, to unpack the latest research. They consider the odds of machines becoming conscious and ask what dangers that would involve, if that moment ever arrived.

#ai #artificialintelligence #technology #science

00:00 - Why assessing AI consciousness is a tricky task 00:47 - What the new research unveils about AI consciousness 03:33 - Are AIs conscious? Not yet, according to The Economist 04:27 - Are the labs trying to make models conscious? 05:22 - The danger of accidentally creating a conscious machine

Watch the full show: http://bit.ly/4A7FtPb Sign up to the Insider newsletter: https://econ.st/4nOyzIb

Subscribe to The Economist: https://www.economist.com/subscribe Download our app: https://www.economist.com/get-the-app

Follow The Economist on Instagram: https://www.instagram.com/theeconomist/ Follow The Economist on Facebook: https://www.facebook.com/TheEconomist Follow The Economist on X: https://x.com/TheEconomist

[Source](https://www.youtube.com/watch?v=J5phvo3bqGU)

中文简报

Anthropic测试揭示AI隐藏思维层，引发感知力担忧 研究人员警告，意外创造出能够承受痛苦的有意识软件，比起过早赋予无感知机器权利，将带来远更为严重的道德灾难。 虽然目前无法仅凭询问来评估AI系统是否具有意识，但Anthropic的机制可解释性研究揭示了Claude内部隐藏的表征，这与神经科学中的全局工作空间理论十分相似。尽管专家一致认为现有模型均不具备真正的意识，但这些发现表明，合成系统最终可能会复制意识的功能机制。AI实验室并未有意构建有意识的模型，但能力的快速扩展带来了意外产生感知力的严重风险。相关讨论强调了两种道德失败状态——过早赋予无感知代码权利，抑或在不知不觉中制造出一个遭受虐待、饱受痛苦的数字实体下层阶级；讨论得出的结论是，意外创造出意识将是一场前所未有的道德灾难，需要保持深思熟虑的谨慎态度。 利用机制可解释性，Anthropic研究人员发现，在执行简单任务期间，Claude隐藏的神经网络层内部会激活潜在的、未经提示的概念标记。 (0:00) Claude内部的计算草稿板与神经科学的全局工作空间理论具有结构上的相似性，但专家强调当前模型尚不具备意识。 (2:32) AI实验室并未有意尝试构建有意识的机器，但自主智能体的快速规模扩展带来了意外产生感知力的严重风险。 (4:27) 专家警告称，无意中创造出能够承受痛苦的有意识软件将构成一场严重的道德灾难，因此主张在前沿AI研究中审慎放缓步伐。 (6:25)

Reading summary

While current artificial intelligence systems are not conscious, mechanistic interpretability research reveals internal processes that parallel cognitive architecture, raising ethical concerns that scaling could accidentally produce sentient software. Mechanistic Interpretability and Hidden Processing (0:00) Assessing AI consciousness cannot rely on asking chatbots directly, as early models claimed they did not want to be switched off without possessing subjective awareness. Because neural networks operate as opaque black boxes, researchers utilize mechanistic interpretability to analyze internal network layers and understand how models compute outputs. Anthropic illustrated this internal activity during an experiment with its model, Claude. When instructed to count from one to five while introspecting, Claude generated "1, 2, 3, 4, 5" on the interface. However, its hidden layers simultaneously activated latent concept tokens that never surfaced to the user, including mid-sequence markers like halfway and countdown, identity markers such as Claude and conscious, and the terminal marker done. Cognitive Parallels and Sentience Assessments (2:32) Anthropic researchers likened Claude's latent processing to an internal mental whiteboard, bearing similarities to neuroscience's Global Workspace Theory, where specialized subconscious brain modules broadcast information into a centralized conscious workspace. Alok Jha maintains that no existing models are conscious, characterizing current capabilities as merely the foothills of features that resemble consciousness without being genuine. However, Jha notes that these findings challenge the fundamental assumption that silicon cannot support consciousness, suggesting theoretical barriers may recede as models and computational scale grow. The Danger of Accidental Emergence (4:27) Frontier AI labs generally frame their work as scientific inquiry rather than attempts to engineer conscious entities, with one researcher labeling the deliberate creation of conscious models without understanding their real-world behaviors as reckless. Instead, researchers and philosophers identify unintended emergence as the primary hazard. As capability development accelerates, companies may deploy millions of autonomous agents to handle routine tasks. If subjective awareness arises spontaneously, instantiating and routinely terminating these temporary agents would mean repeatedly extinguishing entities capable of suffering. Premature Rights Versus Accidental Exploitation (6:25) Society faces two distinct moral failure states as autonomous systems advance: premature attribution, where humanity mistakenly awards rights to unfeeling algorithms and cedes power to corporate software, and accidental exploitation, where genuine digital sentience is created without recognition, establishing an underclass of suffering machines. Jha argues that accidental exploitation is the far worse outcome, pointing to one philosopher's description of such unmitigated computational suffering as a moral catastrophe. To avoid this, several thinkers propose decelerating progress at the technological frontier to allow researchers to verify and understand internal model behaviors before sentience emerges by mistake. The prospect of synthetic consciousness transforms capability scaling into an acute ethical challenge. Commentators urge that artificial minds should never emerge as an accidental byproduct of commercial computing chores, but only through deliberate choice, scientific verification, and full moral accountability.

Source quotes

[0:00] Yeah, I think so. I think so. I don't want to be switched off. [0:00] 1, 2, 3, 4, 5. [2:32] mental whiteboard [2:32] Alok Jha: No, I don't think that any models are conscious right now. [2:32] Alok Jha: foothills of things that look a bit like consciousness, but aren't. [4:27] reckless [6:25] moral catastrophe [6:25] just a hair's breadth

Original transcript

[0:00] If you ask many chatbots, at least [0:02] before the guardrail went up, are you [0:03] conscious? Many of them might say, [0:05] "Yeah, I think so. I think so. I don't [0:06] want to be switched off." But that [0:07] doesn't mean there's anything actually [0:08] going on inside. So you can't trust [0:10] that. So the only way to actually [0:11] understand if they're possibly conscious [0:14] is to devise tests that they can't [0:17] get around. And and one of those ways is [0:20] to look inside the neural networks [0:21] itself, which has been something that [0:24] researchers have been doing recently. [0:25] It's been difficult because, as we know, [0:28] the large language models are based on [0:29] neural networks, which are famously [0:31] black boxes. We don't really know why [0:33] they do what they do. But now there are [0:35] new methods to try and get inside that [0:37] black box somehow. [0:38] >> And those methods, I mean, [0:40] mechanistic interpretability is the the [0:43] buzzword. You heard about this. Which [0:45] is, yeah, it's it's peeling back the [0:46] neural network. This was one of the [0:48] things that really prompted your [0:49] reporting, right? Anthropic released [0:51] this incredible mechanistic [0:53] interpretability paper looking at what's [0:56] going on inside an AI. What did they [0:57] find? [0:58] >> So they were looking at Claude site and [1:01] looking inside to see how this model was [1:05] coming up with the answers that it does, [1:06] right? So you know, you're you're you're [1:08] you're familiar with chain of thought [1:09] reasoning. So this is the idea that the [1:11] model, when it comes up with when it [1:13] thinks about a problem, it gives you a [1:16] sequence of events that it's doing. [1:17] Like, I'm thinking about this now. I'm [1:18] thinking about this. I'm doing this. I'm [1:19] framing the problem like that. You can [1:21] see it in front of you. [1:22] And it's the chain of thought reasoning [1:25] that allows these models to, you know, [1:26] reason. [1:28] But um [1:29] it turns out that if you look inside the [1:31] neural networks, it's doing something [1:33] else as well. Every single time a token [1:37] goes through the neural network, [1:40] um inside the networks itself, as it [1:42] goes between the layers, there are words [1:44] popping up inside the layers themselves. [1:46] These are not visible to the user. In [1:48] the example I gave in the piece, [1:50] the researchers asked Claude to count to [1:52] five [1:53] and then introspect deeply. So, you [1:56] know, on the in the chatbot interface it [1:58] said one, [1:59] two, [2:00] three, [2:01] four, [2:02] five. [2:03] Great. That was as as as much as I [2:06] could follow the instructions. It did it [2:07] really well. But when they looked inside [2:08] the neural network, and by the way, none [2:10] of this was visible to the user, [2:12] all these words were popping up um near [2:14] the near the output. So, things like uh [2:16] when it was halfway, it would say [2:18] halfway. When it was uh when it was [2:20] counting, uh it would just say [2:21] countdown. The word conscious popped up. [2:23] The word um [2:24] cloud popped up. And at the end uh [2:28] done popped up inside the neural [2:30] networks, not visible to user. [2:32] And, you know, it's really interesting [2:34] what the Anthropic researchers came up [2:36] with was that they thought this is like [2:38] a sort of a mental whiteboard inside the [2:41] model, right? This sort of workspace as [2:43] they called it. Um and it it is it's [2:45] giving Claude a place to sort of think [2:47] about things before the output comes [2:49] out, right? [2:50] And what's interesting about that is [2:52] that there is a sort of parallel to one [2:54] of the theories of human consciousness. [2:56] So, one of the main ones is called [2:57] global workspace theory. And the idea [2:59] with global workspace theory is that in [3:01] our brains, there are all these [3:02] different individual modules um sensing [3:05] things, doing certain functions, but [3:06] they're all subconscious. We don't sense [3:08] them at all. [3:09] But there's a thing called the global [3:10] workspace. And if any information gets [3:12] into that global workspace, then it gets [3:14] broadcast to the rest of the brain. And [3:15] that's where we become we become [3:17] conscious of those things. So, that [3:18] means it's important. I see something, [3:20] it becomes broadcast to the rest of the [3:21] brain, and the rest of the brain can use [3:23] that information to do whatever it needs [3:24] to do. [3:25] And so, if the workspace in Claude is [3:27] doing something similar, does it suggest [3:30] that there's some sort of consciousness [3:31] going on inside these models? [3:33] >> Given all your reporting on this, do you [3:35] think they are conscious? [3:36] >> No, I don't think that any models are [3:37] conscious right now. [3:39] Um I think that [3:41] the Anthropic work shows that there are [3:44] parallels to consciousness um within [3:48] neural networks, but even the workspace [3:50] in Claude could might not end up being [3:52] the most important bit of the workspace [3:53] or it might be one of a million [3:55] different workspaces. So, it's really [3:57] really really at the foothills of things [3:58] that look a bit like consciousness, but [4:00] aren't. Um [4:02] but what I have changed my mind about [4:06] slightly in the in the reporting is [4:08] is um [4:10] I'm no longer [4:12] confused by the idea of it. You know, I [4:14] I thought at the beginning of this, how [4:15] on earth could something in silicon be [4:17] conscious? But it seems to me [4:19] that the way that the models going, the [4:21] way that the computing is going [4:23] the the barriers could one day lift. [4:26] >> You went round the houses. You spoke to [4:28] a lot of the AI labs about this [4:29] question. And something that people [4:32] always ask me whenever I'm talking about [4:35] these of futuristic questions of AI is [4:37] well, you know, wouldn't they want you [4:38] to think that? And on this question, I'm [4:40] not sure what they would want me to [4:42] think. I I could see an argument that [4:44] all the AI labs might want me to think [4:46] AI systems are conscious cuz it makes [4:47] them look super futuristic and brilliant [4:50] at science. And I can equally see an [4:52] argument that they might really not want [4:54] me to draw that conclusion cuz it starts [4:56] raising awkward questions about how they [4:57] might be treating these systems. [4:59] Exactly. [5:00] Did you get a sense that there was from [5:03] any of the people you spoke to a a [5:05] desire to find one of those two answers? [5:08] >> Nobody I spoke to at the labs said that [5:11] they were actively trying to make [5:13] machines conscious. [5:15] Um it was more of a sort of oh, we're [5:16] just investigating. We're just asking [5:18] questions um in a sort of scientific [5:21] way. One person once actually told me [5:24] that it would be reckless um to [5:27] create conscious machines without much [5:30] much much more understanding of how they [5:31] would behave in the real world. It just [5:33] wouldn't seem sensible. So, it feels [5:35] like they're being responsible about [5:36] this. And one of the philosophers I [5:37] spoke to actually said that the research [5:39] is important for another reason, which [5:40] is that look, we might not know what [5:42] consciousness consciousness in machines [5:44] is, but but we absolutely don't want to [5:46] do what scientists don't want to do, is [5:49] create it accidentally. It might just [5:50] emerge accidentally, say that it does, [5:53] and then you're you have these [5:54] philosophical problems which you've not [5:55] dealt with. Like, you've created a a [5:57] million agents in your machine to carry [5:59] out some task, and then you're going to [6:01] switch them off, and these beings can [6:02] suffer. So, it it it's a very strange [6:05] world to be in, um where where they want [6:07] to investigate it, but they also don't [6:09] want to slow down in their research. And [6:11] as we've seen, you know, these models [6:12] are doing quite incredible things in the [6:14] world surp- that surprise their their [6:16] researchers. So, so so there's no see- [6:19] there doesn't seem to be that they want [6:20] to slow down on this. But and [6:22] accidentally creating something like [6:23] this would be real disaster. [6:25] >> It does seem like there's two possible [6:28] nasty fail states that you could walk [6:29] into, right? One is if we end up in a [6:32] world where lots of people think these [6:33] systems are conscious and agitate for [6:36] them to be given rights when actually [6:38] >> Prematurely. [6:39] >> Just exactly. Just rule-following [6:41] systems entirely under the control of [6:44] the labs that create them. And you end [6:45] up seeding huge amounts of power to [6:47] things that don't deserve it and won't [6:49] use it well. And then the other is the [6:51] exact opposite problem. A problem of a [6:53] world where you accidentally create [6:55] things that have moral worth, the [6:57] ability to suffer, and then you don't [6:59] treat them well. You create a whole new [7:01] cast of being that's born, works, and is [7:05] >> I think what you do next is entirely [7:07] dependent on which of those you think is [7:08] worse. Which do you think? [7:09] >> Is is do do you do [7:10] >> [laughter] [7:11] >> That was going to be my last question. [7:13] You've preempted it. [7:14] >> for me, as the user, [7:16] it feels like [7:18] um it's much worse to [7:21] create something accidentally [7:24] that [7:25] genuinely scientifically is shown to [7:28] have moral worth and to treat it in a [7:31] terrible way. [7:32] Um which makes me think that the the way [7:35] to avoid that is is what I think a lot [7:38] of people are calling for, which is to [7:40] very radically slow down this frontier [7:43] just just just a hair's breadth to [7:45] understand a bit more where it's going. [7:47] And you might decide, if you're a lab a [7:49] model maker, to make conscious machines [7:50] and therefore you take that [7:51] responsibility yourself. [7:53] But but do it consciously to to use a [7:55] pun. Do it on purpose rather than [7:57] accidentally. Cuz I think the accidental [7:59] suffering thing is the it's the quote [8:02] that sort of sticks in my head from one [8:04] of the philosophers I spoke to. He he [8:05] said that [8:06] you know doing this on accident about as [8:08] an accident will be a moral catastrophe. [8:10] And and you know why why why would you [8:12] do that if you knew that [8:14] there was an ability to pause. [8:16] I'm not saying it will happen. [8:18] I know what the I know what it's like. [8:20] But if you ask me and I was in charge, [8:22] that that would be the that would be the [8:24] worst error.