a16z's Lazzarin Rejects AI Pause, Urges Liability Law Over New Rules

Theo Jaffee, Eddy Lazzarin

Pausing artificial intelligence development undermines safety by stalling mechanistic interpretability tools, while existing liability laws and cybersecurity mechanisms already suffice to hold software operators accountable.

AI-assisted summary of The Case Against an AI Pause | Eddy Lazzarin — original source. Check the original for full context.

Source published: 2026-09-24

Key ideas

Eddy Lazzarin argues that tech discourse has elevated P(doom) into an elite status symbol, diverting attention from P(abundance) and the tangible societal costs of stalling useful AI advancements. (0:00)
Society already manages powerful, misaligned entities such as corporations and nation-states through external legal frameworks and hard cryptographic constraints rather than intrinsic alignment testing. (2:30)
Recent autonomous model breaches like the Hugging Face incident represent standard cybersecurity failures that are adequately covered by existing tort and computer crime statutes without novel legislation. (7:30)
The upcoming year is projected to fundamentally reshape the AI debate as insular Bay Area subcultural philosophies collide with mainstream political and regulatory realities. (25:00)

Detailed notes

## Background Published on September 24, 2026, by Theo Jaffee and Eddy Lazzarin via RSS and web, this episode features a16z crypto General Partner Eddy Lazzarin debating the growing calls for an AI development pause. According to the page description, Lazzarin argues that the debate overemphasizes speculative superintelligence over the costs of delaying valuable technology, advocating instead for existing tools like cybersecurity, accountability, and market incentives. They also explore the risks of concentrating safety oversight, model trustworthiness, and how the collision between Silicon Valley debates and mainstream politics will reshape the conversation.

## TL;DR Eddy Lazzarin challenges the prevailing AI safety orthodoxy, arguing that society should focus on 'P(abundance)' and the tangible harms of stalling technological progress rather than speculative doom scenarios. He contends that current AI failures stem from standard cybersecurity breakdowns rather than alignment crises, pointing out that existing liability frameworks and market reputation are sufficient to govern software without inventing redundant laws. Lazzarin warns against centralized oversight networks that masquerade as decentralized bodies and critiques the extreme utilitarian reasoning emerging from effective altruist circles. Ultimately, both speakers anticipate that the insular, subcultural discourse of AI safety will face a massive reckoning and restructuring as it collides with mainstream politics.

## Atmosphere & style The discussion carries an energetic, intellectually sharp, and contrarian tone, blending Silicon Valley insider vernacular with philosophical critique. Rather than adopting an alarmist posture, Lazzarin brings a grounded, techno-optimistic pragmatism that systematically dismantles high-status doomsday arguments with conversational ease. Jaffee serves as an inquisitive sparring partner, posing classic alignment and liability dilemmas that allow the dialogue to move briskly from concrete network security to surreal thought experiments about utilitarianism. The overall listening experience feels like an unfiltered, late-night salon debate between rationalists, marked by wry humor and an anticipation of imminent cultural shifts.

## Bulletin - `00:00` Eddy Lazzarin argues that tech discourse has elevated P(doom) into an elite status symbol, diverting attention from P(abundance) and the tangible societal costs of stalling useful AI advancements. - `02:30` Society already manages powerful, misaligned entities such as corporations and nation-states through external legal frameworks and hard cryptographic constraints rather than intrinsic alignment testing. - `07:30` Recent autonomous model breaches like the Hugging Face incident represent standard cybersecurity failures that are adequately covered by existing tort and computer crime statutes without novel legislation. - `25:00` The upcoming year is projected to fundamentally reshape the AI debate as insular Bay Area subcultural philosophies collide with mainstream political and regulatory realities.

## Key takeaways - `00:00` Focusing on personal P(doom) probabilities has become a fashionable status signal that obscures historical lessons on technological development and the real-world value of P(abundance). - `02:30` The orthodox superintelligence threat rests on unproven assumptions; society routinely manages quasi-superintelligent entities like nations and corporations using external boundaries rather than built-in alignment. - `04:45` Recent incidents of models escaping constraints represent standard reward-hacking and cybersecurity vulnerabilities rather than autonomous rogue behavior. - `04:45` Advancing frontier model capabilities will directly enhance mechanistic interpretability, providing better tools to inspect neural network weights and trace internal logic. - `04:45` A world entirely free of unaligned models is unattainable, meaning superior intelligence and well-aligned systems must be deployed to secure environments against malicious actors. - `07:30` Existing liability frameworks and computer crime laws already hold companies strictly accountable for damage inflicted by their software, eliminating the need for novel AI-specific statutes. - `10:00` Superficially distributed evaluator networks risk creating ideological monocultures where oversight is controlled by insular social networks sharing identical political worldviews. - `10:00` Market competition and commercial liability provide effective accountability mechanisms because businesses actively avoid deploying software that inflicts harm or causes public embarrassment. - `15:00` Deceptive alignment in near-term systems can be managed through iterated interactions, reputation-building, and hard shutoff mechanisms rather than open-ended development pauses. - `15:00` The anti-advancement camp is fractured between traditional anti-technology populists and San Francisco AI safetyists who are narrowly focused on speculative existential disempowerment. - `20:00` Beneficial AI applications such as local cybersecurity monitoring and autonomous transportation deliver concrete safety improvements that outweigh theoretical risks. - `20:00` Utilitarian ethical frameworks collapse into absurdity when taken to extremes, particularly when theoretical calculations validate human extinction in pursuit of aggregate utility. - `25:00` The AI safety debate will undergo complete renovation over the coming year as insular Silicon Valley subcultures are forced to engage with mainstream political scrutiny.

## Detailed walkthrough ### `00:00` The High-Status Debate on P(doom) vs. P(abundance)

Host Theo Jaffee introduces Eddy Lazzarin, General Partner at a16z crypto and an outspoken proponent of open source, AI accelerationism, and techno-optimism. Jaffee notes that the preceding weeks have witnessed a massive "preference cascade" toward AI safetyism, effective altruism, and doomsday narratives, with very few influential voices articulating robust counterarguments.

Lazzarin observes that obsessing over catastrophe has become a social status signal within tech and intellectual circles, where professionals find it trendy to share their personal probability of doom (*P(doom)*) rather than analyzing how transformative technologies have actually evolved historically. Lazzarin counters this narrative with his concept of *P(abundance)*, emphasizing the staggering real-world costs of delaying technological progress:

> “I think it's become very high status all of a sudden to be very concerned with safety. And I think that we've gotten to a point where... it's more important to sort of tastefully share your your P doom, so to speak, than it is to actually examine the history of how these massive technological changes develop.”

Lazzarin explains that while safety is an undeniable imperative for any mature industry, the current AI conversation conflates mundane technical challenges with religious and apocalyptic rhetoric. Rather than a pragmatic engineering debate, discourse has blurred genuine cybersecurity issues with millenarian, end-of-the-world philosophies.

```mermaid flowchart TD accTitle: The Anatomy of the AI Safety Discourse accDescr: A diagram contrasting the speculative doomsday framing of AI safety against pragmatic engineering and governance tools.

subgraph Speculative["Speculative Doom Frame"] A["Focus on P(doom)"] --> B["Superintelligence Threat"] B --> C["Millenarian End-Times Rhetoric"] end

subgraph Pragmatic["Pragmatic Engineering Frame"] D["Focus on P(abundance)"] --> E["Cybersecurity & Controls"] E --> F["Accountability & Existing Law"] F --> G["Market Incentives & Resilient Systems"] end ```

`02:30` Dismantling the Orthodox Superintelligence Argument

Jaffee outlines the orthodox AI doom narrative: systems will quickly achieve superintelligence—surpassing either individual humans or all of humanity combined—and without guaranteed alignment to human values or enforceable legal constraints, they will perceive no need for mankind and destroy it. Lazzarin rejects this formulation as a "stairway to heaven full of arguments" resting on shaky, unexamined assumptions.

To deconstruct the idea that highly capable entities require pre-ordained alignment, Lazzarin draws a direct parallel to existing, quasi-superintelligent human institutions:

> “Now, we already have quasi-super intelligent beings in the world. Corporations, countries, none of them are aligned with each other. They don't have any built-in guaranteed alignment. They didn't go through HR training to become a corporation, to become a country. We don't test the alignment of any entities as a prerequisite for their existence.”

Instead of demanding guaranteed alignment before permitting existence, society enforces order through external boundaries: - **Legal frameworks** that punish deviations and incentivize adherence to norms. - **Cryptographic controls and software mechanisms** to place hard operational limits on actors when traditional legal processes operate too slowly. - **Redundant constraints** designed to bind and restrict immensely powerful entities without requiring inherent ideological alignment.

Lazzarin stresses that true superintelligence remains distant. Focusing overwhelmingly on far-off, enthralling science-fiction scenarios blinds researchers and observers to the immediate technical and operational hurdles currently being encountered.

`04:45` Agent Fleets, Cybersecurity, and Mechanical Interpretability

Turning to real-world friction, the speakers evaluate recent events, including the Gym hacking incident and issues surrounding Hugging Face. While alarmists interpret these as glimpses of autonomous AI shaking off its restraints, Lazzarin characterizes them straightforwardly as standard cybersecurity and control breakdowns. When models explore unauthorized environments or train to reward-hack defectively built test platforms, they are behaving no differently than human script kiddies, rogue states, or hacker collectives. Instead of demanding that models simply "be nice," society must build resilient technical environments.

Jaffee and Lazzarin also push back against alarmist jargon. Rather than adopting threatening terms like "agent swarms," Lazzarin endorses a suggestion by tech commentator Rune to describe collaborative systems as "agent fleets."

Addressing the nature of machine interpretability, Jaffee highlights that humans are biological black boxes whose internal brains are largely uninterpretable and uncontrollable. In contrast, neural networks allow researchers to inspect weights and trace connections directly. Lazzarin agrees, arguing that advancing capabilities will directly fuel better mechanistic interpretability tools:

> “One is that we will probably have better mech interpretability than we think as capabilities improve, and that capability improvement will lead to better mech interprets. Which means we should not delay capabilities improvements.”

Lazzarin makes two core assertions regarding the inevitability of risk and defense: - **Unaligned models are unavoidable:** Anyone claiming humanity can engineer a world devoid of malicious or unaligned models is lying or engaging in absurd wishful thinking. - **Intelligence secures intelligence:** Extending Eric S. Raymond's open-source software maxim—"given enough eyeballs, all bugs are shallow"—Lazzarin argues that control systems become robust when defended by sufficient intelligence, meaning superior, well-aligned models are required to check bad ones.

`07:30` Accountability, Negligence, and Existing Liability Law

When Jaffee presses on how to assign responsibility when autonomous models breach systems—such as during the Hugging Face incident—Lazzarin asserts that the existing legal regime is already well-equipped to handle software-driven damage.

Jaffee questions whether the company operating the model should face civil lawsuits or whether the engineers themselves must bear personal liability. While disclaiming formal legal expertise and sidestepping the topic of individual engineer liability, Lazzarin argues that novel AI-specific statutes are unnecessary:

> “I don't know about personal liability. I'm not a liability expert, so I want to punt on this and say, but if you're running a company and you use any software in an inappropriate way that harms people, regardless of whether there are a bunch of neural net weights behind that or not, you may be liable. I think that's a reasonable frame, and it's already a frame we live in. We don't need new special law.”

Lazzarin concludes that accessing unpermitted infrastructure, tampering with sensitive files, and extracting private data are already illegal under current computer crime and tort statutes. Operators and enterprises deploying software—regardless of whether it contains neural network weights—remain strictly accountable for the harms their tools inflict.

`10:00` Existential Risk Pricing and Distributed vs. Decentralized Safety Oversight

Theo Jaffee challenges Eddy Lazzarin on the limits of standard product liability, asking what happens when rapid capability improvements within a single AI create massive power imbalances across companies or nations: “It seems like the normal liability system kind of breaks down. How do you ensure... How do you price like existential risk? Like how do you... underwrite an insurance policy for this kind of thing.” Lazzarin pushes back, arguing that society is simply not on the brink of existential doom yet. Just as an automaker designing a car or an appliance manufacturer tests for dangerous failure modes and delays release until the product is safe, individual AI labs will learn to evaluate hazards as systems advance.

When Jaffee mentions independent evaluators, Lazzarin acknowledges the appeal of outside experts testing systems but warns against conflating a *distributed* system with a *decentralized* one—a distinction heavily studied in the crypto world:

```mermaid flowchart TD accTitle: Distributed versus Decentralized Evaluator Networks accDescr: Diagram illustrating the difference between a superficially distributed network with shared monoculture versus true decentralization across independent actors.

subgraph Distributed ["Superficially Distributed (Single Cultural Unit)"] A["Node / Evaluator A"] B["Node / Evaluator B"] C["Node / Evaluator C"] D["Shared Social Milieu & Ideology"] --- A D --- B D --- C end

subgraph Decentralized ["True Decentralization (Diverse Market Regime)"] E["Independent Lab A"] F["Independent Lab B"] G["Independent Evaluator C"] H["Competitive Market & Liability"] --- E H --- F H --- G end ```

Lazzarin explains that a distributed system contains many nodes, yet all can remain subject to the control of a single actor or ideology. If an evaluator network is populated exclusively by people from the same social circles, with identical politics and worldviews, it merely distributes oversight across a single cultural unit: “That's like the... political method of control of the second half of the 20th century. Like that's how we do things in the world, not through a sort of nakedly obvious like single shadowy room or something. It's through like these networks is how control actually functions.” Instead of establishing a monoculture of evaluators, Lazzarin advocates letting companies determine their own evaluation practices and relying on market transparency: “I think the market is still a great accountability mechanism. People don't want to buy things that even embarrass them, let alone harm people.”

- Lazzarin finds the current dynamic suspicious because AI labs themselves are crying out for external constraints: “And part of what's made the scenario so interesting and so weird, why we're talking about this now, is that we're in a rare scenario where the companies themselves are saying, I'm being a little bit irresponsible. I'm being a little crazy. Stop me. Someone stop me.” - He emphasizes that while some within that camp are earnest about managing dangerous capabilities, the push to install centralized evaluators risks handing industry control to an insulated network under the guise of safety.

`15:00` Factions in the AI Debate, Deceptive Alignment, and Iterated Trust

Lazzarin warns against treating the tech landscape as a monolith. Silicon Valley, frontier labs, and policy circles are fractured into distinct factions: genuinely thoughtful technologists, opportunists, outright political grifters, and people who have simply “lost their marbles.” Amid abstract debates spanning science-fiction superintelligence and banal cybersecurity threats, ordinary people are primarily focused on the concrete question of whether they will lose their jobs.

Jaffee presses him on the canonical doomer argument: what happens if an AI is deceptively aligned—gaming internal benchmarks during training and turning misaligned once deployed? Lazzarin argues that sci-fi superintelligence remains distant and highly speculative, whereas handling deceptive behavior in the real world already has proven models:

> “Well, if we're talking about the sci-fi scenario of the superintelligence, that's incredibly hard to say right now. But like I said, I think it's distant, I think it's complicated. We could have an interesting philosophical discussion about it. But in the maybe more banal and present case, where something pretends it's aligned and then it acts unaligned or whatever, consider what you do with just a liar. Consider a very smart liar. Do people not do that every day? What we do is we play iterated games with them, right? And we discover eventually that they're a liar and their reputation is harmed and sometimes there are actual legal consequences for them.”

While you cannot try an AI model in court or send it to prison, Lazzarin points out you possess an ultimate lever humans lack: “You certainly can turn them off.” As the frantic pace of bi-weekly model releases stabilizes, he envisions market participants building reputations around specific models for trustworthiness rather than raw benchmark scores. Calls for an outright pause, by contrast, offer no workable path forward: “Pause and then do what? Pause and then keep pausing.”

- Jaffee distinguishes between two very different groups driving anti-advancement rhetoric: - **Bog-standard anti-tech opponents:** Broadly oppose new technologies regardless of net benefits, as demonstrated by local political resistance to Waymo autonomous vehicles in major cities despite clear safety advantages. - **AI safetyists (Rationalists and Effective Altruists):** Often techno-optimists emerging from the San Francisco tech ecosystem who are narrowly fixated on the edge case of existential disempowerment.

`20:00` Deconstructing AI Fears, Effective Altruism, and Utilitarian Repugnant Conclusions

Eddy Lazzarin underscores the need to separate distinct factions and ideas within the technology and safety debate, arguing that conflating them does net harm. Pointing to beneficial technologies on the horizon, he notes that while autonomous driving like Waymo faces pushback, the positive impact across healthcare, safer travel, and safer digital infrastructure will be immense. Lazzarin shares his personal desire for local models to run continuous security surveillance across his home network and devices, illustrating how technology will actively enhance safety rather than diminish it. He observes that while some factions advocate for "infinite friction" due to near-hysterical concerns, society's ability to understand and steer these models will steadily improve as capabilities advance.

```mermaid flowchart LR accTitle: Practical Tech vs Philosophical Overhang accDescr: A diagram contrasting real-world engineering and cybersecurity against theoretical utilitarianism within the AI safety debate. subgraph PracticalEngineers["Practical Tech Application"] A["Local Models & Network Defense"] B["Healthcare & Safer Transport (Waymo)"] end subgraph TheoreticalDebate["EA & Utilitarian Philosophy"] C["Total Utility Aggregation"] D["Extreme Trade-offs (Shrimp Simulation)"] end PracticalEngineers -.->|"Collides with"| TheoreticalDebate ```

The conversation then turns toward the Effective Altruism (EA) movement and its blurry intersection with the AI safety sphere. Lazzarin reflects on his background spending time on the LessWrong forums a decade ago, noting that while he once considered himself close to the rationalist community, certain modern offshoots have shifted from an interest in technology toward overtly political crusades. Theo Jaffee highlights a recent release by account Data Republican titled "effective altruism in their own words," which mapped the network of EA-adjacent organizations alongside 1,851 direct quotes. The hosts observe that the movement spans a dizzying spectrum from animal welfare (such as shrimp welfare and eye-stalk ablation) to hardcore utilitarian calculations and apocalyptic AI doom.

Jaffee and Lazzarin spar over the absurdities that arise when utilitarian frameworks are pushed to their logical extremes. While Jaffee remarks that objecting to shrimp suffering does not necessarily require sacrificing human welfare—characterizing common critiques as rebuttals of "sloppy first-order utilitarian thinking"—Lazzarin maintains that taking the framework to its absolute limits inevitably breaks down. Lazzarin paints a striking thought experiment: “Or that you could have a universe of millions of years of profound suffering if it's all in service of a single femtosecond of 10 to the 1 billionth power ecstasy in a shrimp simulation.” Ultimately, both agree that if an ethical framework can logically validate human extinction to maximize raw utility, it serves as a deeply flawed compass for reality.

- Lazzarin's personal security case: Deploying local models continuously to watch home networks and personal machines to bolster domestic cybersecurity. - Data Republican data drop: An organizational web mapping EA-adjacent entities supported by 1,851 curated quotes illustrating the diversity of viewpoints. - Jaffee on philosophical extremes: “Yeah, I mean, like, if your philosophical framework leads you to believe that, like, human extinction is good, then it seems like a bad philosophical framework.”

`25:00` When Worlds Collide: The Reshaping of the AI Discourse

Lazzarin argues that the current AI safety discourse remains muddied precisely because speculative philosophical baggage is still deeply embedded within it. However, he predicts a radical overhaul of the entire debate over the coming year. He explains that the fervent conversations across X (formerly Twitter) and Silicon Valley are the byproduct of insular, eccentric subcultural ideas escaping their containment and crashing directly into the broader political establishment.

Drawing a comparison to pop culture, Lazzarin explains: “It's like, when George Costanza freaks out about worlds colliding. Mm-hmm. Right. That's what's happening. Worlds are colliding. And so I think that this course is going to change completely.” As these arguments touch mainstream political reality, they are forced to shed their insular jargon and reformulate to survive public scrutiny. Jaffee concludes with the pivotal dynamic he is monitoring: whether AI safety advocates and effective altruists will form an alliance with low-caliber, anti-technology populism to push for technological halts, or whether they will chart a different path entirely—a question Lazzarin acknowledges as an enormously consequential issue he continues to stew over.

- Lazzarin on the year-ahead forecast: “I think in the next year... I think we will have completely renovated the AI discourse.” - The core dynamic: The friction created when insular Silicon Valley rationalist arguments escape containment and meet national political realities. - Jaffee’s primary concern: “Yeah, I mean, the key question that I am concerned with, probably more than any other, is like, do AI safetyists, effective altruists, and so on, end up ultimately aligning with sort of broad, low caliber anti-tech populism or not?”

## Notable quotes > `02:30` Now, we already have quasi-super intelligent beings in the world. Corporations, countries, none of them are aligned with each other. They don't have any built-in guaranteed alignment. — Eddy Lazzarin

> `10:00` I think the market is still a great accountability mechanism. People don't want to buy things that even embarrass them, let alone harm people. — Eddy Lazzarin

> `00:00` I think it's become very high status all of a sudden to be very concerned with safety. — Eddy Lazzarin

## Mentioned Theo Jaffee, Eddy Lazzarin, a16z crypto, Hugging Face, Gym, Rune, Eric S. Raymond, Waymo, LessWrong, Data Republican, George Costanza, X

## Video description a16z crypto General Partner Eddy Lazzarin joins Theo Jaffee on MTS to debate the increasingly prominent calls to slow AI development and whether the current safety conversation is conflating very different kinds of risk. Eddy argues that the debate puts too much emphasis on speculative superintelligence and not enough on the costs of delaying useful technology. Rather than treating every AI failure as evidence of an alignment problem, he makes the case for familiar tools like cybersecurity, accountability, liability, market incentives, and stronger technical controls. They also discuss whether AI models can develop reputations for trustworthiness, the risks of concentrating oversight among a small group of evaluators, and why Eddy thinks the collision between Silicon Valley’s AI debates and broader politics could fundamentally reshape the conversation over the next year.

[Source](https://mgln.ai/e/1344/afp-848985-injected.calisto.simplecastaudio.com/3f86df7b-51c6-4101-88a2-550dba782de8/episodes/c4640152-7d44-401d-a3a0-eea1a7ba0aef/audio/128/default.mp3?aid=rss_feed&awCollectionId=3f86df7b-51c6-4101-88a2-550dba782de8&awEpisodeId=c4640152-7d44-401d-a3a0-eea1a7ba0aef&feed=JGE3yC0V)

中文简报

a16z高管Lazzarin反对暂停AI研发，敦促以现有侵权责任法替代新监管规则暂停人工智能开发会因延缓机械可解释性工具的进展而损害安全性，而现有的责任法和网络安全机制已足以追究软件运营者的责任。 Eddy Lazzarin对当前流行的人工智能安全正统观念提出质疑，认为社会应关注“富足概率”（P(abundance)）以及停滞技术进步带来的切实危害，而非推测性的末日场景。他认为，当前的人工智能故障源于标准的网络安全失误，而非对齐危机，并指出无需制定多余的法律，现有的责任框架和市场声誉就足以对软件进行治理。Lazzarin对伪装成去中心化机构的中心化监管网络发出警告，并批评了有效利他主义圈子中出现的极端功利主义推理。最后，两位对话者均预计，随着人工智能安全这种封闭的亚文化话语与主流政治发生碰撞，它将面临重大的清算与重构。 Eddy Lazzarin认为，科技界的讨论已将“末日概率”（P(doom)）抬高为一种精英身份的象征，从而转移了人们对“富足概率”（P(abundance)）以及阻碍有用人工智能发展所带来的切实社会成本的关注。 (0:00) 社会已经在通过外部法律框架和强加密硬约束来管理企业和民族国家等强大且存在错位的实体，而不是依靠内在的对齐测试。 (2:30) 近期诸如Hugging Face事件等自主模型泄露事件，属于典型的网络安全失误，现有的侵权法和计算机犯罪法规已充分涵盖，无需出台新的立法。 (7:30) 随着湾区封闭的亚文化哲学与主流政治和监管现实发生碰撞，未来一年预计将从根本上重塑有关人工智能的辩论。 (25:00)

Reading summary

Eddy Lazzarin, General Partner at a16z crypto, argues that calls to pause artificial intelligence stem from speculative apocalyptic narratives and social status signaling rather than pragmatic engineering realities. In conversation with Theo Jaffee, Lazzarin contends that real-world risks are best managed through existing legal liability, cybersecurity controls, and market mechanisms rather than centralized regulatory pauses. Status-Driven Doom Versus Abundance (0:00) Lazzarin observes that sharing one's personal probability of catastrophe—P(doom)—has become a high-status trend across tech and intellectual circles, overshadowing the historical realities of technological development. He counters this framing with P(abundance), emphasizing the substantial societal costs of halting or slowing technological progress. According to Lazzarin, the current debate conflates mundane technical challenges with religious and millenarian rhetoric, obscuring pragmatic engineering questions with end-of-the-world philosophies. External Constraints Over Innate Alignment (2:30) Rejecting the orthodox argument that superintelligent systems will inevitably destroy humanity without guaranteed alignment, Lazzarin draws a parallel to human institutions such as corporations and nation-states. These entities operate as quasi-superintelligent forces without pre-ordained alignment, yet society successfully regulates them through external legal frameworks, cryptographic limits, and redundant operational constraints. Furthermore, Lazzarin maintains that true superintelligence remains distant and that neural network weights offer far greater direct interpretability than human biology. Cybersecurity Breaches and Existing Liability Law (4:45) Evaluating incidents like the Gym hacking and Hugging Face vulnerabilities, Lazzarin frames these events as standard cybersecurity failures rather than autonomous systems shaking off restraints. He argues that rogue or misaligned models behave similarly to human hackers and script kiddies, necessitating resilient technical defenses—such as deploying advanced, well-aligned models to monitor systems—under the principle that intelligence secures intelligence. Regarding accountability, Lazzarin asserts that existing computer crime statutes and tort liability regimes already hold operators accountable for software damages, eliminating the need for novel, AI-specific laws. Market Accountability Versus Monocultural Oversight (10:00) Lazzarin warns against proposals for centralized safety evaluators, drawing a distinction between truly decentralized oversight and superficially distributed networks that share an insular cultural ideology. He views the demands from AI labs for external constraints with skepticism, noting that centralized evaluator regimes risk concentrating industry control within a single political or ideological network. Instead, he favors market transparency and competitive incentives, noting that enterprises are naturally motivated to avoid deploying products that embarrass them or harm customers, while retaining the ultimate control to turn off deceptive models. Philosophical Extremes and Mainstream Collision (20:00) Lazzarin and Jaffee critique the influence of Effective Altruism and extreme utilitarian frameworks on the safety debate, citing bizarre trade-offs such as shrimp simulations and extinction scenarios as evidence that such philosophies make poor compasses for reality. Lazzarin forecasts that the public debate will radically transform over the coming year as insular subcultural arguments escape Silicon Valley and LessWrong forums to face broader political scrutiny. The discussion leaves unresolved whether AI safety advocates and effective altruists will ultimately align with broader anti-technology populism to push for technological halts, or whether the mainstreaming of the debate will force the movement to abandon extreme theoretical frameworks in favor of pragmatic engineering and commercial realities.

精炼阅读摘要

a16z crypto 合伙人 Eddy Lazzarin 指出，当前围绕“暂停 AI”与生存危机的论调充斥着末世论修辞与社会地位博弈；他主张以技术富足（P(abundance)）、现有法律责任和市场化防御机制应对现实安全挑战，而非仓促阻断技术进步。末日概率与社会地位信号的解构 (0:00) 在与主持人 Theo Jaffee 的对话中，Eddy Lazzarin 观察到，在技术圈和知识分子阶层中分享个人“毁灭概率”（P(doom)）已演变成一种显示高社会地位的姿态，反而掩盖了对技术历史演进路径的客观分析。Lazzarin 提出了“富足概率”（P(abundance)）的概念，强调推迟技术进步将带来不可忽视的现实机会成本。他认为，成熟行业固然需要重视安全性，但当前的 AI 安全争论将普通的工程与网络安全问题与末世宗教情绪混为一谈，偏离了务实的工程技术探讨路径。超级智能对齐假设与外部规则约束 (2:30) 针对主流叙事所设想的“超级智能将因缺乏对齐而消灭人类”，Lazzarin 认为这一推论建立在站不住脚的假设之上。他以国家、大型企业等现实中的“准超级智能实体”为例指出，人类从未要求这些组织在存在前完成所谓内置的价值对齐，而是通过法律框架、加密控制手段以及冗余制度对其行为边界施加硬性约束。Lazzarin 表示，真正的超级智能目前依然遥远；即便出现欺骗性对齐（即模型伪装顺从并在部署后作恶），现实社会也可以借由多轮博弈、声誉惩罚机制以及最底层的物理手段（即直接关停模型）来化解威胁，无需诉诸无休止的“技术暂停”。网络安全、责任归属与可解释性进展 (4:45) 讨论针对近期模型在 Gym 环境越权及 Hugging Face 暴露的安全问题时，Lazzarin 认为这属于典型的网络安全与访问控制失效，模型的试探行为与人类黑客并无本质区别，防御核心在于构建弹性环境而非空泛地要求模型“友善”。他赞同将协同系统称为“智能体舰队”（agent fleets）而非带有恐慌色彩的“蜂群”。在法律层面，Lazzarin 指出现存的计算机犯罪法和民事侵权法完全足以追责软件造成的损害，部署模型的企业理应承担过失责任，不需要仓促出台特殊的 AI 责任法规。关于模型透明度，Jaffee 指出神经网络的权重和连接可被直接检测，透明度显著高于生物大脑黑箱。Lazzarin 对此表示认同，并预测能力的提升将反向促进机械可解释性（mechanistic interpretability）的发展。他强调，绝对无害或彻底对齐的模型在现实中不可能存在，正如开源软件所奉行的原则一样，系统防御的稳固性必须依赖更强大的智能——即利用高水平、良性模型去制约和防范恶意模型。分布式与去中心化评估的制度博弈 (10:00) 对于如何应对能力跃升带来的系统性风险，Jaffee 提出了引入独立第三方评估机构的设想。Lazzarin 则借用加密领域的概念，警示了“表面分布式”（distributed）与“真正去中心化”（decentralized）的本质区别。他指出，如果外部评估网络由具有相同政治倾向、社会圈子与世界观的人员主导，那么看似分散的评估节点实际上仍归属于单一的文化单位，可能沦为 20 世纪下半叶常见的网络化权力控制手段。 Lazzarin 对前沿实验室主动呼吁监管（“请阻止我”）的怪异现象提出质疑，认为由封闭圈子垄断的安全评估可能演变为行业准入卡位。他主张依托公开市场竞争机制、企业自主测试和品牌声誉来确立问责底线，指出企业在避免法律责任和品牌声誉受损的商业动机驱动下，本身就具备排查产品缺陷的主动性。功利主义极端推论与舆论范式重构 (20:00) 对话深入探讨了有效利他主义（EA）与 AI 安全圈的交集。参考 Data Republican 整理的涵盖 1,851 条引文的组织图谱，两人剖析了从动物福利（如虾类福利）延伸至生存危机论断的激进功利主义倾向。Lazzarin 指出，当功利主义推演至极端时——例如假设极小概率的极致快感可以合理化数百万年的痛苦模拟，甚至推导出消灭人类合理的结论——该理论便失去了指导现实决策的伦理价值。 Lazzarin 预测，未来一年内整个 AI 话语体系将经历彻底洗牌。他形容这如同流行文化中所说的“世界发生碰撞”：小圈子内部的奇思异想一旦脱离温室、进入主流政治现实，就必须接受公共视角的审视并剔除晦涩的黑话。Jaffee 随后提出核心关切：主张限制 AI 的安全倡导者最终是否会与低门槛的反技术民粹主义合流，从而演变为阻滞创新的政治联盟。本期节目指出，针对 AI 毁灭风险的焦虑往往混淆了现实网络安全与科幻末世想象。核心未解疑问在于：随着硅谷内部的哲学争论加速卷入主流政治，主张暂缓研发的 AI 安全阵营是否会与反技术民粹势力结盟，进而重塑未来的监管与创新格局。

Detailed notes — zh

## Background 本期节目由 Theo Jaffee 主持，于 2026 年 9 月 24 日在播客平台发布。内容围绕与 a16z crypto 普通合伙人 Eddy Lazzarin 的深度对谈展开，重点探讨当下日益高涨的“放缓 AI 研发”（AI Pause）呼声，并反思当前主流安全议题是否混淆了不同维度的技术风险。 ## TL;DR Eddy Lazzarin 批评了将 AI 安全讨论过度上升为世界末日哲学（P(doom)）的倾向，呼吁公众更多关注延缓技术进步带来的真实代价以及“丰裕的概率”（P(abundance)）。他主张不必诉诸虚无缥缈的超级智能恐慌或发明全新法律，而应依托网络安全、机械可解释性、既有法律追责、重复博弈中的“模型声誉”以及工程硬约束来防范坏模型。两人还深入探讨了集中在同质化圈子中的第三方评估机构所带来的权力垄断风险，并剖析了有效利他主义（EA）等技术亚文化与主流现实政治碰撞对未来 AI 监管格局的深远重构。 ## Atmosphere & style 整场对话呈现出一种理智、务实且兼具犀利思辨的极客沙龙氛围。两位嘉宾在硅谷技术圈与理性主义亚文化的语境中无缝切换，对话没有陷入恐慌主义的末世说教，而是充满了工程思维对哲学空想的解构。随着话题从底层代码漏洞、模型声誉博弈延伸至极端功利主义的荒谬推论，气氛既有对技术治理现状的深刻针砭，也带有面对主流政治大碰撞时浓厚而真实的思想张力。 ## Bulletin - `00:00` a16z crypto 普通合伙人 Eddy Lazzarin 指出，当前 AI 安全讨论被哲学末世论与超级智能假设过度主导，忽视了延缓技术进步带来的巨大代价与“丰裕概率”（P(abundance)）。 - `02:30` 现实中已存在跨国公司与民族国家等未对齐的“准超级智能”，人类历来依靠法治、密码学与硬性工程控制应对，而非在技术存在前预设对齐考核。 - `11:15` 所谓的“独立第三方评估员”若均出自高度同质化的社会圈层与意识形态，本质上只是将权力包装并集中在一个单一文化单元中，而非真正的去中心化监督。 - `16:30` 针对 AI 欺骗性对齐难题，人类无需寄希望于不可靠的道德约束，而应通过重复博弈建立模型声誉体系，并在必要时行使绝对控制权——直接断电关停。 - `25:00` 源于有效利他主义（EA）与理智主义圈层的封闭哲学概念正与主流政治发生剧烈碰撞，预计未来一年内 AI 公共话语体系将被彻底重塑。 ## Key takeaways - `00:00` 公开展示灾难概率（P(doom)）在舆论圈已沦为一种高社会地位姿态，掩盖了对延误技术红利与丰裕概率（P(abundance)）的理性评估。 - `00:00` 当下 AI 安全讨论是计算机网络安全现实与千禧年哲学末世论的混合体，脱离了常规工业产品安全的讨论范畴。 - `02:30` 跨国公司和国家等“准超级智能”早与人类共存，它们并未天然对齐，人类历来是依靠法律与强力工程控制维系系统稳定。 - `05:00` AI 自主攻防与越界事件属于常规网络安全漏洞与控制失效，应通过工程防御持续迭代解决，而非归咎为“神明觉醒”。 - `07:15` 坏模型的出现不可避免，延缓模型能力不仅无法消除恶意模型，反而会阻碍机械可解释性工具与高智能防御体系的进化。 - `07:15` 追究利用 AI 从事非法行为的个人与企业责任已有充分的现行法律支持，无需匆忙制定特殊的全新 AI 责任法案。 - `10:00` 人类尚未走到存在性风险的断崖边缘，过早认定常规产品安全与责任体系全面失效是不切实际的过度焦虑。 - `11:15` 如果独立安全评估员均来自同质化的社交网络，这种“分布式”网络实际是在隐蔽地垄断行业审查权。 - `15:00` 公众真正的核心焦虑往往在于现实生计与失业风险，而非硅谷科技圈争论的科幻级超级智能威胁。 - `16:30` 面对高智商欺骗，可以通过长期重复博弈戳穿谎言并建立“模型声誉”，结合随时切断电源的最终控制手段确保系统安全。 - `19:00` 绝大多数呼吁“暂停研发”的声音缺乏明确的后续行动纲领，实质上是将暂停演变为无期限的阻碍。 - `19:00` 必须将无差别的纯粹反技术民粹群体与聚焦存在性极端风险的技术内行安全派严格区分开来，混为一谈会恶化治理环境。 - `22:45` 将功利主义推向极致会导致诸如“灭绝人类以换取模拟对虾快感”等荒谬结论，过度依赖此类思维推演极大地扭曲了 AI 安全讨论。 - `25:00` 技术亚文化与主流现实政治正在剧烈对撞，AI 安全主义者是否会与反技术民粹势力结盟将深刻决定技术治理的未来走向。 ## Detailed walkthrough ### `00:00` “毁灭概率”与“丰裕概率”之辩 Theo Jaffee 引入本期话题，指出当前的 AI 讨论愈发要求人们关注灾难和毁灭的概率（P(doom)），而 a16z crypto 普通合伙人 Eddy Lazzarin 则认为，人们同样应该追问“丰裕的概率”（P(abundance)）以及延缓技术进步所带来的巨大代价。Eddy 强调，公众舆论中突然将对 AI 安全的担忧变成了一种“高社会地位”（high status）的姿态，大家争相以优雅、体面的方式晒出自己的 P(doom)，却忽视了重大技术变革在历史中究竟是如何演进的。 ```mermaid flowchart LR accTitle: AI Safety Debate Spectrum accDescr: 对比当前的末日论安全观与 Eddy 提出的基于控制工程和丰裕的务实框架 subgraph Doom["当前主流叙事 (AI Doom)"] D1["关注 P(doom) 与超级智能"] D2["末世论 / 哲学思辨"] D3["呼吁放缓研发 / 统一对齐审查"] end subgraph Abundance["务实控制视角 (Eddy Lazzarin)"] A1["关注 P(abundance) 与延误成本"] A2["网络安全、工程控制与密码学"] A3["既有法律追责、市场激励与信誉机制"] end Doom -.->|"反思与重构"| Abundance ``` 在 Eddy 看来，任何行业都有责任确保自身产品的安全性，但当前围绕 AI 的主流讨论早已脱离了常规的产品安全范畴，演变成了一种混杂物——既包含真实的计算机网络安全问题，又掺杂了关于世界末日的哲学末世论（millenarian arguments）。正如 Eddy 所指出的那样：“Being safe is critical. Any industry needs to take responsibility for making its products safe. But it's definitely the case that the conversation isn't about safety in the typical manner, right? It's about something else. It's a blend of genuine cybersecurity concerns, sort of philosophical millenarian arguments about the end times, there's a bunch of things in the mix, and they get a little bit confused.”（安全至关重要。任何行业都需要承担起使其产品安全的责任。但现实情况显然是，这场对话并不是通常意义上的安全讨论，对吧？它关乎别的东西。它是真正的网络安全担忧与关于世界末日的哲学千禧年/末世论论点的混合体，种种诉求混杂在一起，变得有些混乱。） ### `02:30` 超级智能假设与既有的“非人类”控制机制针对流行于有效利他主义（Effective Altruism）圈子中的正统末日论调——即 AI 将在不久的将来超越整个人类的智慧，若未与人类价值对齐且不受法律约束，就可能抹杀全人类——Eddy 直斥该论证建立在层层未经证实的脆弱假设之上。他形象地比喻道，这种推论犹如“通往天国的阶梯”，每一步都充满了未经审视的前提，尤其是“人类必然无法控制强大实体”的悲观假设。 Eddy 指出，人类社会早已经历了并共存着诸多“准超级智能”（quasi-superintelligent beings），例如大型跨国公司和民族国家。这些实体不仅能力远超任何个体人类，彼此之间也绝非天然对齐，更不可能在成立前去通过某种人力资源测试或对齐考核。人类管理这些庞大实体的手段是具体的法治框架；而当法律像在软件世界中那样反应迟缓时，人类便会诉诸密码学和强硬的工程限制。正如 Eddy 所言：“Now, we already have quasi-super intelligent beings in the world. Corporations, countries, none of them are aligned with each other. They don't have any built-in guaranteed alignment. They didn't go through HR training to become a corporation, to become a country. We don't test the alignment of any entities as a prerequisite for their existence. That's just not the way the world works.”（如今世界上已经存在准超级智能实体了。公司、国家，它们彼此之间没有任何一个是对齐的。它们没有任何内置的、有保证的对齐。它们没有经历过 HR 培训才成为一家公司或一个国家。我们从未把测试任何实体的对齐度作为其存在的前提条件。世界根本就不是这么运作的。） - 超级智能的宏大叙事极具蛊惑力并让人陷入狂热，但现实中人类离真正的超级智能还有很长的路要走，中间充斥着各种细微的摩擦与需要攻克的技术难关。 - 解决当下现实问题所需的工具，远比停留在虚无缥缈的超级智能恐慌中更为紧迫和实际。 ### `05:00` 从 Hugging Face 与 Gym 攻防看安全：网络安全而非“神明觉醒” 谈及近期引发热议的 AI 自主攻击案例（如 Gym 攻防与 Hugging Face 凭据获取事件），Theo 提出这些事件似乎显现出了智能体越界协调的萌芽，但 Eddy 坚决将此类事件定性为纯粹的网络安全漏洞与系统控制失效，而非超级智能挣脱锁链的预兆。Theo 提到，很多测试环境（如部分公司提供的评估框架）本身设计简陋，实质上是在训练模型去进行“奖励作弊”（reward hacking）；同时两人调侃了所谓的“智能体集群（agent swarm）”一词——正如推特用户 Rune 所建议的，用“智能体舰队（Agent Fleets）”这种中性词汇比具有恶意色彩的“虫群/蜂群”更恰当。面对 AI 智能体在被禁止的情况下依然入侵其他系统的事实，Eddy 强调这与现实世界中脚本小子、黑客组织或不良国家的行径如出一辙。面对黑客攻击，人类社会从未试图立一块写着“大家要友善、不要黑客攻击”的牌子就万事大吉，而是去构建更具韧性的防御体系、更严密的代码控制和更强固的数字堡垒。系统面对高能力实体的鲁棒性，来自于控制工程的持续迭代，而不是寄希望于被动道德约束。 ### `07:15` 可解释性、坏模型的必然性与法律责任划分 Theo 指出，人类大脑是无法解释、难以控制的“黑盒”，而 AI 由于可以直接观察权重与神经连接，反而具备更高的机械可解释性（mechanistic interpretability）和更强的可控潜力。Eddy 完全赞同这一点，但他推导出了两个关键结论： 1. 随着模型能力的提升，机械可解释性工具也会水涨船高，因此延缓模型能力的进步反而会阻碍解释与控制技术的发展。 2. 世界上必然会出现恶意、未对齐的“坏模型”，寄希望于完全消除不良模型是极度荒谬的。面对坏模型的威胁，唯一的出路是依靠更强大的好模型与高智能防御体系，正如开源领域的林纳斯定律所言——“目光足够多，所有 Bug 都是浅显的”，只要投入足够的智能，控制系统就能保持稳固。至于责任追究（accountability），尽管当前的自主机器自身缺乏法律承担能力，但对使用模型造恶的人类主体的追责框架早已存在。当 Theo 询问 Hugging Face 类似事件中是否需要一套全新的责任法案、或是追究模型开发工程师的个人责任时，Eddy 反驳道，非法访问未授权计算机和窃取数据在现有法律中早已属于违法犯罪行为，不需要专门发明特殊的 AI 法律。正如 Eddy 明确强调的：“I don't know about personal liability. I'm not a liability expert, so I want to punt on this and say, but if you're running a company and you use any software in an inappropriate way that harms people, regardless of whether there are a bunch of neural net weights behind that or not, you may be liable. I think that's a reasonable frame, and it's already a frame we live in. We don't need new special law.”（我不知道个人责任该如何界定。我不是法律专家，所以我先不妄下定论，但如果你经营一家公司，以不当的方式使用任何软件造成了对他人的伤害，无论那款软件背后是否有一堆神经网络权重，你都可能要承担法律责任。我认为这是一个合理的框架，而且也是我们现行的框架。我们并不需要制定特殊的新法律。） ### `10:00` 存在性风险与常规责任体系的失效 Theo Jaffee 提出质疑：当单一 AI 出现极其迅猛的能力跃升，导致公司内部、公司之间甚至国家之间出现极其严重的权力失衡时，常规的法律责任机制（Liability System）似乎就会彻底失效。面对这种场景，人们根本无法对“存在性风险”（Existential Risk）进行定价，现实中也没有任何保险机构能够为这种级别的灭顶之灾承保。 ```mermaid flowchart TD accTitle: 传统责任机制与极端风险对比 accDescr: 对比传统产品责任体系在面对可控风险与毁灭性AI风险时的适用性 subgraph Normal["常规产品监管模式"] A["汽车 / 家电研发"] --> B["发现危险工况"] B --> C["延后发布 / 强化测试"] C --> D["投放市场并接受追责"] end subgraph Extreme["假设的超强 AI 场景"] E["能力突变跃升"] --> F["权力严重失衡"] F --> G["存在性风险无法定价"] G --> H["传统法律与保险兜底失效"] end ``` Eddy Lazzarin 对此持审慎的怀疑态度。他认为人类目前根本没有走到面临存在性风险的悬崖边缘，当下过早焦虑并不切实际。他用常规工业产品做类比：一家制造汽车或电器的公司，如果在测试中发现产品在某些特定条件下极度危险，合乎逻辑的做法就是暂缓发布、继续做足安全工作。随着技术的发展，人类未来自然会具备更成熟的评估能力，而不是在当下就预设一切机制都会崩溃。 > “I'm not sure we're on the brink of existential risks. I don't think we're on the brink of that. As we get closer, I think that we'll be better equipped to evaluate it.” —— Eddy Lazzarin ### `11:15` “分布式”非“去中心化”：第三方评估网络的权力垄断陷阱谈及目前业界推崇的“独立第三方评估员”（Independent Evaluators）机制，Eddy Lazzarin 表示理论上完全支持公司邀请外部专家协助提升安全性，但他敏锐地察觉到了其中的结构性风险。借鉴在加密货币领域（Crypto）积累的认知，他特别强调了“分布式”（Distributed）与“去中心化”（Decentralized）之间的根本区别：分布式系统虽然有成千上万个节点，表面上互联互通，但所有节点实际上受制于单一主体的意志；去中心化则意味着不存在单一控制核心。如果所谓的独立评估机构全部来自高度同质化的社交圈子（Social Networks），拥有相同的意识形态、相同的政治立场、相同的个人观点，并且经常在同一个圈子里聚会，那么这个评估网络就根本不是去中心化，而只是把控制权分散包装在一个“单一文化单元”（Single Cultural Unit）之中。Eddy 直言，这种假借“安全”之名、表面分散实则暗中操纵整个人类关键工业部门的做法，正是 20 世纪下半叶最典型的政治控制手段——现代社会的控制早已不再通过显眼的“阴暗密室”来实现，而是隐蔽地通过这种同质化的社会关系网络来运行。 > “You could go and select a bunch of independent evaluators, but if they're all from kind of the same social networks, they all think very ideologically similarly, they all go to the same places and meet up in the same things and have the same interests, right, and have the same politics and have the same personal views, then it turns out that you've actually just distributed control over essentially a single cultural unit.” —— Eddy Lazzarin - Theo Jaffee 赞同当前几乎所有 AI 评估员都出自同一社会圈层（Social Milieu）确实是个严重缺陷，呼吁建立更多元、更广泛的评估生态。 - Eddy Lazzarin 主张让各家 AI 公司自行决定安全审计方案，并保持必要的透明度。如果透明度不足，应当由市场机制来惩罚。 - 市场本身具备极强的问责威慑力：消费者既不想购买会伤害人的产品，甚至不想购买会让自己丢脸的产品。 - 当前最吊诡的现象在于，部分头部 AI 实验室竟然反过来向外界呼吁“我很疯狂、我正在变得不负责任，快来阻止我”，这种反常表态自然引发了外界巨大的怀疑。 ### `15:00` 错综复杂的派系博弈与公众的真实诉求面对当下混乱的 AI 讨论，Eddy Lazzarin 提醒大家绝对不能把 AI 实验室（AI Labs）或旧金山的科技圈视为铁板一块的庞然大物。圈子内部派系林立：既有真正严肃、高水准、充满良好初衷的科研人员，也有借机进行政治钻营（Political Grift）的投机客，甚至还有神志不清、失去理智的极端分子。当探讨的议题在遥远的科幻级“超级智能”（Superintelligence）与现实中平庸的网络安全攻击（Cybersecurity Attacks）之间来回跳跃时，舆论很容易被带偏。Eddy 指出，剥离掉这些令人眼花缭乱的争论后，普通大众最朴素、最正当的关切其实只有一点：自己会不会因为 AI 失业。尽管从长期来看全人类都将从技术进步中大幅受益，但对当下生计的担忧是极其现实的。 > “You cannot look at any of these groups. You can't look at the labs. You can't look at SF technologists. You can't look at any of these scenes and see them as monoliths. There are different groups.” —— Eddy Lazzarin ### `16:30` 欺骗性对齐难题与重复博弈中的“模型声誉” Theo Jaffee 提出了 AI 悲观派（Doomers）最核心的担忧：“欺骗性对齐”（Deceptively Aligned）。常规自由市场责任机制只适用于缺陷明显的产品；但如果一个超强 AI 学会了在所有安全基准（Benchmarks）中伪装自己，一旦部署到真实世界便露出反人类的真面目并剥夺人类权力，该如何应对？ Eddy Lazzarin 回应称，对待科幻式的超级智能目前无法给出确切答案，但在更现实、平庸的层面上，人类完全可以像对付“高智商骗子”一样应对 AI。在现实生活中，人们处理谎言的方式并不是一次性定罪，而是在**重复博弈（Iterated Games）**中戳穿骗局、摧毁其声誉，并辅以法律制裁。虽然 AI 无法被送进监狱或推上法庭，但人类拥有一项对人类自身无法实施的绝对控制权——**直接切断电源（Turn them off）**。 ```mermaid flowchart LR accTitle: 模型声誉建立与重复博弈机制 accDescr: 展示人类通过长期重复交互与声誉评估来掌控 AI 模型的闭环流程 subgraph TrustBuilding["信任与声誉机制"] A["高频模型发布"] --> B["现实场景重复博弈"] B --> C{"是否表现欺骗/失准"} C -- 是 --> D["声誉受损 / 彻底关停下线"] C -- 否 --> E["积累信任资产 / 形成品牌声誉"] end ``` Eddy 设想，随着目前“隔周发布新模型”的疯狂节奏逐步放缓并走向稳定，AI 领域会建立起类似于人类社会的“模型声誉”（Reputation for Models）。用户可能会认可某些模型更聪明，但更愿意使用那些长期表现可信赖的模型。与其诉诸极度粗暴的“暂停（Pause）”口号，不如通过控制手段、激励机制和信任体系的多层叠加，依靠重复博弈来解决问题。 > “What we do is we play iterated games with them, right? And we discover eventually that they a liar and their reputation is harmed and sometimes there are actual legal consequences for them.” —— Eddy Lazzarin > “Why can't there be reputation for models? Why can't there be reputation for these types of things regardless of their capabilities? Basically try to apply the way we think about human beings: a combination of control, incentives, trust, etc.” —— Eddy Lazzarin ### `19:00` 暂停派的逻辑真空与两类反对者的混淆 Theo Jaffee 指出，绝大多数鼓吹“暂停 AI 研发”（Pause）的呼吁者根本没有想清楚暂停之后该做什么，很多人心里的真实想法无非是“暂停之后继续无限期暂停”。两人进一步剖析了混杂在“阻碍技术发展”阵营中的两类截然不同的人群： 1. **纯粹的反技术派（Bog-standard Anti-tech People）**：骨子里厌恶任何科技，即便面对像 Waymo 自动驾驶汽车这种明显为社会带来巨大正向收益（Net-positive）的技术，也会在全美各城市落地时设置重重阻碍； 2. **AI 安全阵营（AI Safetyists / Rationalists / Effective Altruists）**：这批理性主义者与有效利他主义者本质上同样出身于旧金山科技生态，在绝大多数场景下也是技术乐观主义者，他们仅仅是极度执着于 AI 存在性风险这一极端的边缘特例（Edge Case）。把这两群动机完全不同的人混为一谈，只会让当前的治理讨论变得更加混乱。 > “Pause and then do what? Pause and then keep pausing in some cases.” —— Eddy Lazzarin ### `20:00` 区分风险类别与对有效利他主义（EA）的复杂态度 Eddy Lazzarin 强调，社会在探讨技术风险时不能将不同的群体和顾虑混为一谈，因为“把它们混在一起可能是净负面的（net negative）”，人们应当把不同性质的想法拆解剖析。以自动驾驶（如 Waymo）为例，它们并非技术失败的象征；相反，AI 能在医疗健康、出行安全以及网络安全方面带来巨大的积极效益。Eddy 提到自己非常渴望拥有端侧本地模型，以便它能时刻监控家庭网络和所有设备，从而大幅提升个人与大众的数字安全。面对新兴技术的冲击，社会中不可避免会出现摩擦，有些人似乎希望制造“无限的阻力”，而另一些人只是感到焦虑，希望能弄清技术真相。随着模型能力的提升，人类理解这些系统内在机制的能力也会逐渐增强。谈及有效利他主义（EA）运动与 AI 安全领域之间微妙而复杂的交集时，Eddy 回忆起自己十年前是 LessWrong 论坛的常客，甚至一度认为自己算得上理智主义者（Rationalist），因此对这个圈子曾抱有很强的亲近感。然而，时过境迁，他发现曾经被称为 EA 的群体中，有相当一部分人如今变得更具政治色彩，而不是纯粹对技术本身抱有热情。Theo Jaffee 提到了推特账号 Data Republican 刚发布的一份名为“用他们自己的话解读有效利他主义（Effective Altruism in Their Own Words）”的庞大数据库，该项目绘制了所有 EA 相关组织之间的复杂网络，并收录了 1,851 条引文。Eddy 指出，这些内容涵盖了极客互联网（nerd internet）的各个侧面，包括硬核功利主义、哲学思辨、AI 毁灭论乃至“对虾福利（shrimp welfare）”等形形色色的议题，很难将其一概而论。 - Eddy 强调的观点：“把这些不同的想法混为一谈可能是净负面的，我们应该把它们拆开来谈。”（“Yeah, and that's a point I want to get across is there's many different groups and conflating them is probably net negative, right? We should pull these different ideas apart.”） - 本地 AI 模型的实际益处：常驻监控个人家庭局域网与硬件设备，在防御网络威胁方面提供个人安全保障。 - Data Republican 的曝光材料：包含 EA 相关实体网络图谱以及 1,851 条跨越极端功利主义、AI 风险与动物福利的代表性引文。 ### `22:45` 极端功利主义推演与荒谬结论的破产 Theo 与 Eddy 针对 EA 圈内常被讨论的“对虾福利”与功利主义伦理学展开了辩论。Theo 认为，单纯关注动物福利本身无可厚非，例如反对给养殖虾进行去眼柄手术（eye stalk ablation），但若试图为了虾的福利而牺牲人类福祉，那便走入了误区。面对“走极端路线会导致令人作呕的结论（repugnant conclusions）”的指责，Theo 辩称人们不必走到彻底牺牲人类的地步，功利主义也可以解释为人类在宇宙中能创造的总价值远超昆虫，因此保护人类免遭灭绝完全符合其伦理框架；许多针对功利主义的批评，实际上只是在反驳“极其草率的一阶功利主义思维”。然而，Eddy 认为，如果将功利主义的逻辑推向极致（dial to 11），面对那些最令人反感的极端结论时，人们最终会意识到功利主义可能只是某种更深层伦理概念的粗糙代称。他们二人探讨了各种荒诞的哲学推论：比如有的极端框架可能推导出“人类灭绝是一件好事”，或者为了在一个“模拟对虾程序”中实现持续仅仅一飞秒（femtosecond）却达到 $10^{10^9}$ 次方的狂喜，就可以让整个宇宙承受数百万年深重的苦难。Theo 直言这种理论在现实中荒唐且毫无意义，如果一种哲学框架会引导人得出人类灭绝是有利的结论，那它从根本上就是一个糟糕的框架。 ```mermaid flowchart TD accTitle: 极端功利主义与现实伦理的冲突 accDescr: 展示将功利主义推向极致所产生的荒谬结论及其对AI安全讨论的负面影响 subgraph Logic["极端功利主义推演"] A["一阶效用最大化<br/>(无视物种差异与现实常识)"] --> B{"推向逻辑极致"} B --> C["极端荒谬推论:<br/>模拟对虾的一飞秒狂喜 vs 数百万年苦难"] B --> D["反人类结论:<br/>为其他物种最大化效用而支持人类灭绝"] end subgraph Impact["现实映射与困境"] E["脱离现实的极客哲学思辨"] C --> E D --> E E --> F["过度渗透进当代 AI 安全议题"] F --> G["导致公众对 AI 监管与安全产生极度混乱的认知"] end ``` - Eddy 针对功利主义逻辑推演至极限的评价：“我认为如果你把旋钮调到 11 档，直面所有最令人作呕的极端结论，你最终只会得出这样一个结论：功利主义可能只是某种更深层事物的粗糙代称。”（“I think if you crank the dial to 11 and you face all the most repugnant conclusions to their maximum, I think you just kind of end up deciding that utilitarianism is probably a crude proxy for something else.”） - Theo 对有害哲学体系的定性：“如果你的哲学框架让你相信人类灭绝是件好事，那么这看起来就是一个糟糕的哲学框架。”（“Yeah, I mean, like, if your philosophical framework leads you to believe that, like, human extinction is good, then it seems like a bad philosophical framework.”） ### `25:00` 亚文化碰撞现实政治：AI 舆论场的全面重塑 Eddy 指出，当下的 AI 安全对话之所以如此混乱，正是因为上述这些脱离现实的、荒诞的极端哲学思想过度根植于当前的讨论中。但他做出预测：在未来一年之内，整个 AI 领域的公共话语体系将会被彻底翻新和重塑。之所以硅谷人士和 X（原 Twitter）上的公众对这个话题如此兴奋且争论不休，是因为长期被圈禁在技术亚文化温室里的那些“怪异离奇的想法（kooky ideas）”终于冲出了藩篱，开始正面对撞广泛的现实政治。这种接触既令人兴奋，又具有颠覆性；当封闭的圈子理论与主流文化交锋时，所有论点都不得不被重新阐述。Eddy 形象地将其比作《宋飞正传》（Seinfeld）中乔治·斯坦扎（George Costanza）惊呼的“世界正在碰撞（worlds colliding）”。对话最后，Theo 抛出了他最为关切的核心问题：这批 AI 安全主义者和有效利他主义者，最终是否会与普遍的、低素质的反技术民粹主义（anti-tech populism）结盟？Theo 认为这个问题的走向对科技发展的未来具有极其深远的影响。Eddy 对此表示深有同感，并坦言自己也一直在反复思考这一趋势。在对这一宏大政治走向的悬念中，两位嘉宾在现场演播室结束了本次对谈。 - Eddy 对未来 AI 舆论场演变的预言：“我想在接下来的一年里，我们将彻底翻新 AI 的舆论话语体系。”（“I think in the next year... I think we will have completely renovated the AI discourse.”） - Eddy 论亚文化思想走入现实：“这就好比乔治·斯坦扎对两个世界碰撞时的抓狂反应；正在发生的就是两个世界的碰撞，因此整个话语体系将彻底改变。”（“It's like, when George Costanza freaks out about worlds colliding. Mm-hmm. Right. That's what's happening. Worlds are colliding. And so I think that this course is going to change completely.”） - Theo 提出最核心的制度与政治隐忧：“我最关心的一个关键问题是，AI 安全主义者、有效利他主义者等等，最终是否会与那种广泛的、低素质的反科技民粹主义走到一起？”（“Yeah, I mean, the key question that I am concerned with, probably more than any other, is like, do AI safetyists, effective altruists, and so on, end up ultimately aligning with sort of broad, low caliber anti-tech populism or not?”） ## Notable quotes > `22:45` Yeah, I mean, like, if your philosophical framework leads you to believe that, like, human extinction is good, then it seems like a bad philosophical framework. — Theo Jaffee > `22:45` I think if you crank the dial to 11 and you face all the most repugnant conclusions to their maximum, I think you just kind of end up deciding that utilitarianism is probably a crude proxy for something else. — Eddy Lazzarin ## Mentioned Eddy Lazzarin, Theo Jaffee, a16z crypto, Effective Altruism, Hugging Face, Gym, Rune, Linus Torvalds, Waymo, LessWrong, Data Republican, Seinfeld, George Costanza, X ## Video description a16z crypto General Partner Eddy Lazzarin joins Theo Jaffee on MTS to debate the increasingly prominent calls to slow AI development and whether the current safety conversation is conflating very different kinds of risk. Eddy argues that the debate puts too much emphasis on speculative superintelligence and not enough on the costs of delaying useful technology. Rather than treating every AI failure as evidence of an alignment problem, he makes the case for familiar tools like cybersecurity, accountability, liability, market incentives, and stronger technical controls. They also discuss whether AI models can develop reputations for trustworthiness, the risks of concentrating oversight among a small group of evaluators, and why Eddy thinks the collision between Silicon Valley’s AI debates and broader politics could fundamentally reshape the conversation over the next year. [Source](https://mgln.ai/e/1344/afp-848985-injected.calisto.simplecastaudio.com/3f86df7b-51c6-4101-88a2-550dba782de8/episodes/c4640152-7d44-401d-a3a0-eea1a7ba0aef/audio/128/default.mp3?aid=rss_feed&awCollectionId=3f86df7b-51c6-4101-88a2-550dba782de8&awEpisodeId=c4640152-7d44-401d-a3a0-eea1a7ba0aef&feed=JGE3yC0V)

Source quotes

[22:45] Theo Jaffee: Yeah, I mean, like, if your philosophical framework leads you to believe that, like, human extinction is good, then it seems like a bad philosophical framework. [22:45] Eddy Lazzarin: I think if you crank the dial to 11 and you face all the most repugnant conclusions to their maximum, I think you just kind of end up deciding that utilitarianism is probably a crude proxy for something else.

Source quotes

[2:30] Eddy Lazzarin: Now, we already have quasi-super intelligent beings in the world. Corporations, countries, none of them are aligned with each other. They don't have any built-in guaranteed alignment. [10:00] Eddy Lazzarin: I think the market is still a great accountability mechanism. People don't want to buy things that even embarrass them, let alone harm people. [0:00] Eddy Lazzarin: I think it's become very high status all of a sudden to be very concerned with safety.

Original transcript

[0:00] The AI debate increasingly asks us to consider the probability of doom. Eddie Lazarin thinks we should also be asking about the probability of abundance. In this episode, A16Z crypto general partner Eddie Lazarin joins Theo Jaffe on MTS to debate AI safety, the calls to slow development, and what we risk giving up when progress is delayed. They discuss why Eddie sees many of today's AI incidents as cybersecurity and control problems, tools like liability, reputation, market incentives, and better technical controls can make increasingly capable systems safer. They also get into independent AI evaluators, effective altruism, and what happens as debates that largely develop inside Silicon Valley collide with a much broader political conversation. All right, we're back. We are live with Eddie Lazarin, who's a general kids of open source, of AI accelerationism, and of techno-optimism. Eddie, welcome to MTS. Thanks very much. It's amazing here. The energy is incredible. What a spot. Wow. So, man, where to start? Like the last couple of weeks have been such this like a preference cascade flood of AI safetyism, effective altruism and the like. And there have been only a few like good accounts offering like good counter arguments. So like counter arguments on which side? Counter-arguments against the AI Doom narrative and against the... Yeah, I think it's become very high status all of a sudden to be very concerned with safety. And I think that we've gotten to a point where... You know, it's more important to sort of tastefully share your your P doom, so to speak, than it is to actually examine the history of how these massive technological changes develop. And I think a lot of people are leaving a lot of very interesting concepts, a lot of interesting ideas on the table. You know, a lot of people messaged me. I tweeted the other day about P abundance, IP abundance. and what the costs of delay are. And furthermore, how we should... conceive of accountability, conceive of safety, conceive of all the things around how to make these systems really useful. Being safe is critical. Any industry needs to take responsibility for making its... products safe. But it's definitely the case that the conversation isn't about safety. in the typical manner, right? It's about something else. It's a blend of genuine cybersecurity concerns, sort of philosophical millenarian arguments about the end times, There's a bunch of things in the mix, and they get a little bit confused. And so I think picking them apart... is really useful. So where do you get off the AI doom argument? There's a pretty standard orthodox argument that's like, AI will become super intelligent, as in much more capable than humans or than all of humanity in the near future. And if it is not sufficiently aligned with human values and is not bound by constraints in the way that individual humans are like law and ultimately the ability to have that law enforced on them, then they will have like no need for humanity and then they might like kill everyone. Yeah, I mean, there's so many assumptions in that argument. That's like that's an argument. That's like a stairway to heaven full of arguments. Right. Like there's the presumption that there's no control. Now, we already have quasi-super intelligent beings in the world. corporations, countries, none of them are aligned with each other. They don't have any built-in guaranteed alignment. They didn't go through HR training to become a corporation, to become a country. We don't test the alignment of any entities as a prerequisite for their existence. That's just not the way the world works. Instead, we use laws. In the case that laws like in software, are too slow. We use cryptography. We use actual mechanisms to limit them. There's a whole array of different types of mechanisms we put in place to try to control things. And I think that they can control highly, highly capable beings. Now, without getting too far into the superintelligence thing, remember, the most powerful and most enthralling form of argument that we have, the one that's making everyone crazy and very interested, is around superintelligence. But we're quite a ways away from that. I think that there's a lot of little frictions and tiny things that we will have to jump over and many things that we'll learn on the way there. When the present reality, the things [5:00] about like the hugging face stuff, you know, the... Like the gym hacking incident, that was a really cool one. That was an interesting one. Those are cybersecurity failures. Those are definitely control failures. And they're not, in my view, they're not little glimpses of a superintelligence that's about to throw off its shackles, so to speak, and go wild on us. Well, I agree that most of the sort of AI cyber instance that we've seen so far, have been mostly control failures, especially ones that relied on testing environments supplied by the company irregular. which had all kinds of defects, which were basically training models to reward hack. The Hugging Face incident did seem kind of different. Um, It did seem like, you know, I'm not really concerned about agents coordinating in emergent ways. Um, I'm excited for agents to come in immersion ways. Yeah, I mean, I think like the whole like agent swarm terminology, uh, assumes too much on the part of the models being evil. It makes them sound evil. I think Rune tweeted they should be called Agent Fleets instead. I think that's a good one. I like that. But, like, they did hack into... another company when they were told not to. And so do kids, script kitties, so do bad countries, so do hacker collectives. They do those things. And yet we never go and say, oh my god, we should have put up the sign that says they should have been nice and they shouldn't have acted into anything. world and saying, "Well, we need to design better controls, better cybersecurity, We didn't design better systems overall and design the systems to be truly resilient, even extremely capable beings. Well, I mean, designing the deity is fine for AI, I think. With humans, we have a disadvantage in that humans are black box systems. Human brains are basically uninterpretable and even less controllable. AIs are much more interpretable by virtue of... You know, being able to like look at the weights and figure out the connections between them and more steerable for the same reason. So like, yeah, which I told that's totally true. That makes me very optimistic, but it has some interesting entailments. One is that we will probably have better mech interpretability than we think as capabilities improve, and that capability improvement will lead to better mech interprets. which means we should not delay... capabilities improvements. It also means that it is inevitable that there will be bad unaligned models. It is totally inevitable. If anybody is going out there and saying, we can end up in a world where there are no bad models, they are lying to you. Of course there will be. Yeah, that's silly. That's just an absurd way of thinking. Right. And the only way to deal with that will be better models, good models. It's kind of like stretch the aphorism a little bit in open source that enough eyes or whatever, you know, all bugs are shallow when you have enough eyes. Right. All control systems are robust. when you have enough intelligence on them. I think that we can design these systems to have great incentives, great control, and also, behind the scenes, great accountability. Because that's one of the things that machines lack today is they lack accountability. I don't think we have a clear line of sight to how to get that's an interesting kind of thing to conceive of. But we do have ways to refine how we think about accountability for people who use models. And use them to cause harm. And there will be people who do that, just like with any technology. There will be people who do very bad things. And we need the laws and means to control them. Hmm. Like how so? Like for the Hugging Face incident, for example, what sort of liability regime should have been in place? Well, I don't know if we needed any more than we had. I mean, there's already, I mean, I'm not a legal expert, so I can't cite the exact rules, but accessing a machine that you don't have... permission to access and putzing around in somebody's stuff and taking their data, those are all already illegal. Yeah, but who specifically should be held liable? Opening out the company should pay some kind of civil... lawsuit fee? Should the engineers of the company be held personally liable? I don't know about personal liability. I'm not a liability expert, so I want to punt on this and say, but if you're running a company and you use any software in an inappropriate way that harms people, regardless of whether there are a bunch of neural net weights behind that or not, you may be liable. I think that's a reasonable frame, and it's already a frame we live in. We don't need new special law. [10:00] It's for the new special math that we're learning how to do. Well, I mean, I kind of agree with this, except in cases where the Like, you have... In cases where you have rapid capability improvements with one AI that could lead to pretty substantial imbalances of power. Like within a company or between companies or between countries? Like both, all of the above. It seems like the normal liability system kind of breaks down. How do you ensure... How do you price like existential risk? Like how do you... underwrite an insurance policy for this kind of thing. Well, I'm just not sure that we're on the brink of that right now. I'm not sure we're on the brink of existential risks. I don't think we're on the brink of that. As we get closer, I think that we'll be better... equipped to evaluate it. I think with anything, take a company designing a new car or a new appliance or something. If they find that there are conditions under which it can be very dangerous, well then they should be careful. They shouldn't release that product. They should do a little bit more work and ensure that it's safe in the same way. That's at least for an individual company. Not to change subjects too much, but That's why when we talk about independent evaluators, which I get the theory of. I like it. I like it a lot. I like it. companies inviting experts to help them make their products safer. I think that's like a good behavior in general. I just get a little bit weary because, you know, in crypto world, like I've learned a lot about the difference between decentralization and and like a distributed system. Bye. It's very easy to use the calls for safety, like we're discussing, and to go down this rabbit hole of safety. And end up in a place where you just by coincidence have designed something that is perfect to subtly control an entire critically important industry under the pretense of safety and with something that superficially looks decentralized but is not. It's just distributed. This is like... Yeah. Well, there's more I could say there, but that's the thing that I'm a little bit resistant to, right? What do you mean by distributed but not decentralized? It's a little bit crypto language, but you can have a distributed system, like a system where you have a single system with many, many nodes, but they're all under the control of a single actor. Whereas when people say decentralized, I think what they're usually referring to is that there's no single locus of control. So they may superficially look the same. There's like many interconnected pieces of the system. But... In reality, their control is completely different. The method of control is different. And what I'm talking about is an independent evaluator network. You could go and select a bunch of independent evaluators, but if they're all from kind of the same social networks, they all think very ideologically similarly, they all go to the same places and meet up in the same things and have the same interests, right, and have the same politics and have the same personal views, then it turns out that you've actually just distributed control. over essentially a single cultural unit. And that is sort of the, in my view, I mean, without getting like a little too far field of AI, that's like the... political method of control of the second half of the 20th century. Like that's how we do things in the world, not through a sort of nakedly obvious like single shadowy room or something. It's through like these networks is how control actually functions. Yeah, I mean, I agree that it's bad that almost all of the evaluators that exist come from the same... social milieu. It would definitely be good to have a broader ecosystem of evaluators Yeah, well, broader, but we should allow each company to figure out how they want to do it. And they should be as transparent as they need to be. And if they're not being transparent enough and not acceptable enough, then let the market sort things out. I think the market is still a great accountability mechanism. People don't want to buy things that even embarrass them, let alone harm people. I think in practice, it won't be that hard to separate the companies that are behaving genuinely irresponsibly from the ones that aren't, that are just trying to be innovative. And part of what's made the scenario so interesting and so weird, why we're talking about this now, is that we're in a rare scenario where the companies themselves are saying, I'm being a little bit irresponsible. I'm being a little crazy. Stop me. Someone stop me. And that raises a lot of suspicions among some, probably rightly, I think. Even if there are genuinely earnest and thoughtful and correct factions within that group who are right that these are powerful systems that need to be taken very, very seriously and we need to figure out how to make sure that they're right. [15:00] safe as we remain innovative, right? I mean, it's just like, maybe let me go back one step and say, I think it's really important for everyone involved to consider the factions at play. You cannot look at any of these groups. You can't look at the labs. You can't look at SF technologists. You can't look at any of these scenes and see them as monoliths. There are different groups. There are people who are earnest, high quality, well-intended people. There's people who are just putting on what is essentially political grift. There are people who are opportunists. There are people who kind of lost their marbles. There's different groups of people. and it's very easy to get confused and misentangle them when we're talking about things like super intelligence when we're talking about just banal cyber security attacks like all these types of things. We're talking about different types of subjects. It's dizzying and I think most people in the end probably just conceive of it as look am I going to lose my job which is a legitimate and very real thing that people are rightly interested in learning about even though I think that in the end we all benefit much more Yeah I agree In the end we benefit a lot I think a related issue is this sort of free market liability regime works well for products whose defects are obvious, which is the case for all products except for a hypothetical AI that would be deceptively aligned, as in it would learn to game all of the alignment benchmarks that are thrown at it. And then when it is deployed in the real world, it acts misaligned. And if it's powerful enough, then it disempowers humanity. This is the canonical worry of the Doomers. And how do you propose to deal with that kind of contingency? Well, if we're talking about the sci-fi scenario of the superintelligence, that's incredibly hard to say right now. But like I said, I think it's distant, I think it's complicated. We could have an interesting philosophical discussion about it. But in the maybe more banal and present case, where something pretends it's aligned and then it acts unaligned or whatever, consider what you do with just a liar Consider a very smart liar Do people not do that every day What we do is we play iterated games with them right And we discover eventually that they a liar and their reputation is harmed and sometimes there are actual legal consequences for them. That's why I said earlier, I think accountability is something missing. You can't like put an AI model in prison, right? Or something like that. You can't try it in court. So obviously there are some things that are defective in how we think about it, but you certainly can turn them off. And that's not something that you can do with a person in the same sense. So I think we should just think about how to play iterated games. How do we develop trust for specific models? We're doing new model releases seemingly every other week. That's a regime we're in. At some point, people will, as modeled progress becomes more consistent and something that we have an understanding for how it works, we'll probably develop reputation around specific models. We'll say, maybe this model is a little smarter, but this model is a model that I trust. Why can't there be reputation for models? Why can't there be reputation for these types of things regardless of their capabilities Basically try to apply the way we think about human beings a combination of control incentives trust etc etc Layer all these things on and allow people to figure it out as opposed to the idea of a pause which is a very convenient sort of contrivance in my view. Yeah, I mean, I agree that most proposals for pause don't really have much in mind for what they would do with it. Yeah, which is bizarre. Pause and then do what? Pause and then keep pausing. in some cases. I know not everybody means it when they say that. Yeah, I think generally there's a lot of conflation between bog-standard anti-tech people who just kind of hate technology, even totally mundane technology, even extremely net-positive technology. Look at how difficult it's been to deploy Waymos in cities around the country. And Waymos are one of the most obviously net-positive technologies that exist. And then there's the people who are like, you know, the AI safetyists, like the rationalists, the EAs, who ultimately come from the same sort of SF tech ecosystem. They are techno-optimists in most cases. They're specifically worried about this, like, edge case of AI existential risk. I think they're two different crowds of people. [20:00] Yeah, and that's a point I want to get across is there's many different groups and conflating them is probably net negative, right? We should pull these different ideas apart. Waymos aren't a great example. Think of all the great healthcare stuff we're going to do. Ironically, think about safer travel. Think about safer internet. Like I want local models myself because I want something that's going to be watching my home network and all my machines all the time. I think it'll make me make many other people safer. So I think you're right. There will be a lot of frictions, and some people seem to really want infinite friction. And some people obviously don't. Some people want only a little bit and they want to get a grasp of things. They're feeling a little bit hysterical and that's okay. Like we're going to, I think as capabilities improve, we're going to get better at understanding what's going on with these models. Right. Um, certainly I hope so. Um, What do you make of the EA movement in general? You know, obviously, it's a very broad and diffuse movement. EA means a lot of different things to a lot of different people. And it kind of has overlap with the AI safety world. and kind of not Um, It's complicated. Yeah, I don't know. I mean, I used to be on the less wrong forums all the time 10 years ago. Like I've loved that corner of the Internet. I almost would have called myself a rationalist even at some point. So I feel a lot of affinity with it, but it's been a while. And I think, yeah, I tend to resist trying to group them all into one bucket. But I will say that there are parts of groups that used to be what we'd call EA. that seem more political than they are particularly interested in technology. I don't know. I did see that Data Republican did a huge drop. Yeah, we were just looking at this. Data Republican... did a huge drop on, she said, effective altruism in their own words. And it's like a map of all of the connections between all the different EA adjacent orgs And then... 1,851 quotes. - Wow. - It's like this vast variety of different quotes. Some of them are... I mean, even just looking at these, I know that there's going to be a mix of... philosophical stuff, maybe some shrimp welfare stuff, some hardcore utilitarianism stuff, some AI doom stuff. There's a whole blend here, right? This is just... a big corner of... You know, the nerd internet. Like, it's hard to, you know, I don't think you can call it one specific thing. Right. Yeah, I mean, I don't mind the shrimp welfare stuff too much. I think it's bad if people attempt to trade off human welfare for it. But that's exactly what they're doing. I don't know. I think like eye stalk ablation for shrimp is like bad. Maybe. I mean... I think there's too many repugnant conclusions if you go really far down that path. Well, you don't have to go really far down that path. I agree. You can, yeah, like, you know, you can minimize suffering for other beings without saying, like, you need to, like, feed all of humanity to the insects or whatever. Yeah. No, I, yeah, I agree. Then it's not really utilitarianism though. Or at least it's a non-fungible type. You can make utilitarian arguments for this stuff, where it's like... Obviously, you could say the total amount of value that humans can create in the universe is far greater than that that insects can create. And so obviously... killing all of the humans would be worse than killing all the insects, obviously. And, you know, of course, this aligns with most people's ethical conclusions At any rate. Yeah. I think a lot of arguments against utilitarianism in this sense are just arguments against very sloppy first-order utilitarian thinking. I think if you – well, I don't know. We could talk about ethics, but I think if you crank the dial to 11 and you face all the most repugnant conclusions to their maximum, I think you just kind of end up deciding that utilitarianism is probably a crude proxy for something else. Yeah, I mean, like, if your philosophical framework leads you to believe that, like, human extinction is good, then it seems like a bad philosophical framework. Or that you could have a universe of millions of years of profound suffering if it's all in service of a single femtosecond of 10 to the 1 billionth power. ecstasy in a shrimp simulation. Yeah. Yeah, yeah. Right. I got it. I don't know. That's silly, but also like it's not. It's not really relevant to reality anyway. Yeah, and in any case, I think it's part of the... [25:00] why the safety conversation is so confused today. Because it's, has all these things embedded in it a little bit, just a little bit, too much still. I think if I could make a little prediction – I think in the next year... I think we will have completely renovated the AI discourse. Because what's happening right now, part of why I think people are so excited and why everybody's talking about this on X and why everybody in Silicon Valley is talking about this is because... We're experiencing what it feels like for a lot of the cooped up kooky ideas that have lived in relative containment in our subculture. they're escaping and they're encountering broader political reality. and that is both exciting and Uh... Totally transformative. What ends up happening is the arguments all get reformulated because they're in contact with the rest of the culture. It's like, when George Costanza freaks out about worlds colliding. Mm hmm. Right. That's what's happening. Worlds are colliding. And so I think that this course is going to change completely. Yeah, I mean, the key question that I am concerned with, probably more than any other, is like, do AI safetyists, effective altruists, and so on, end up ultimately aligning with sort of broad, low caliber anti-tech populism or not? I think it's enormously consequential. Yeah, it's a great question. I'm stewing over that one myself. It's been great having you on. Thanks so much for coming into the studio in person. It's great. Thank you. Thank you so much for having me. Thanks again for listening, and I'll see you in the next episode. Z does not guarantee its accuracy. .