Anthropic Co-Founders Tie Model Safety Directly to Computational Scale

Anthropic联合创始人将模型安全与计算规模直接挂钩 领导团队认为，Constitutional AI和负责任扩展政策（Responsible Scaling Policies）等前沿安全框架只有应用于高度扩展的模型时才能发挥作用，这促使安全规范转化为产品的基线要求，并对行业竞争对手形成由市场驱动的牵引力。 在这场兼具回顾与前瞻的讨论中，Anthropic联合创始人梳理了他们共同经历的历程：从OpenAI早期的模型扩展突破，到创立以安全为导向的AI实验室。他们探讨了Constitutional AI与负责任扩展政策（RSP）等开创性框架的发展，重点阐述了安全措施如何自然而然地从先进模型的规模扩展中获得红利。创始人们详细说明了他们在权衡公司架构、落实实证评估以及推动全行业“向上竞争”时的务实态度。展望未来，他们对AI彻底变革生物学、医疗和软件工程的潜力充满期待，并表达了对机械可解释性研究的坚定承诺。 Anthropic 联合创始人回顾了早期扩展定律（Scaling Laws）的突破，指出安全与规模在结构上深度绑定——模型必须先扩展到具备足够的基础智能，才能有效应用人类反馈强化学习（RLHF）。 (2:30) 团队推出了“宪法式 AI”（Constitutional AI）与分级的“负责任扩展政策”（RSP），通过严密的基准评测（Evals）将前沿安全转化为常规的产品工程标准，并带动业内多家公司跟进。 (18:26) 团队经历反映了 AI 舆论的剧烈变迁：早期投身 AI 和关注安全曾被主流视作离经叛道，如今则已转变为与白宫高层沟通的核心议题，验证了逆共识押注的战略价值。 (11:13) 展望未来，创始人们对神经网络可解释性（Interpretability）以及 AI 推动生物医疗等科研突破充满期待，并指出近期在 Y Combinator 现场已有 95% 的人使用 Claude 进行代码编写。 (45:38)

Anthropic's co-founders argue that AI safety cannot remain an abstract academic exercise or a self-defeating purity test, but must instead be structurally integrated with empirical model scaling and commercial development. They contend that demonstrating competitive safety techniques creates a commercial and institutional gravitational pull that compels the wider AI industry toward a race to the top. Structural Interdependence of Safety and Scaling (2:30) Anthropic's core approach emerged from an early observation that AI safety and scale are fundamentally intertwined. The co-founders explain that during the development of GPT-2 and GPT-3, they realized language models could serve as mechanisms to absorb implicit human values, yet techniques such as Reinforcement Learning from Human Feedback (RLHF) required large-scale models to function effectively because smaller architectures lacked sufficient capability. Rather than viewing safety as an impediment to progress, the founders treated capability advances as prerequisites for sophisticated steering. This insight drove the development of Constitutional AI. Jared Kaplan and Dario Amodei recount that instead of designing convoluted alignment schemes, they leveraged the empirical finding that simple methods succeed when paired with scale. By providing a model with a written set of principles and evaluating its outputs through basic prompt-driven comparisons, the system learned to supervise its own behavior, validating the scaling hypothesis by turning alignment targets into direct training signals. Institutionalizing Safety via the Responsible Scaling Policy (18:26) To manage risks systematically, Dario Amodei and Paul Christiano drafted the Responsible Scaling Policy (RSP) in late 2022. Rather than imposing an arbitrary cap on model compute, the RSP introduces a tiered hierarchy of capability thresholds, known as AI Safety Levels (ASLs). Passing into higher tiers mandates progressively stricter cybersecurity and alignment protocols. The founders emphasize that safety must become as routine, predictable, and 'boring' as automotive safety standards or standard corporate financial audits, giving external partners and lawmakers confidence that extreme measures activate only when genuine capability thresholds are breached. Executing this framework relies on continuous model evaluation, spearheaded by groups such as the Frontier Red Team. The co-founders highlight that determining an upper bound on model capabilities is an asymmetric challenge; models must be aggressively probed for dangerous capabilities in Chemical, Biological, Radiological, and Nuclear (CBRN) domains, accounting for augmentations like tool use and chain-of-thought reasoning. To prevent operational bottlenecks, Anthropic simplified the RSP's language to ensure safety protocols function like actionable internal metrics rather than bureaucratic hurdles. Commercial Pragmatism and Industry Race to the Top (35:57) The decision to launch Anthropic as a commercial enterprise rather than a research nonprofit was dictated by the severe capital constraints of training frontier models. The founders established a corporate culture prioritizing low ego and cross-functional alignment, exemplified by their unanimous adoption of an 80% pledge for public benefit. Rather than centralizing hard trade-offs among executives, leadership expects product, engineering, and trust teams to collectively balance deployment velocity against safety requirements. Anthropic's leadership rejects ideological purity tests, arguing that demanding absolute adherence simply pushes conscientious researchers out of positions of influence. Instead, the company relies on commercial success to exert a 'gravitational pull' across the ecosystem. Daniela Amodei and colleagues observe that within months of Anthropic publishing its RSP, three competing AI labs instituted similar frameworks, while industry peers widely replicated Anthropic's Frontier Red Team model and interpretability methods. Interpretability, State Capacity, and Scientific Horizons (43:43) Looking forward, the co-founders identify interpretability and empirical science as the next frontiers where established consensus will shift. Chris Olah describes trained neural networks as an 'artificial biology' exhibiting intricate, discoverable internal order rather than opaque black boxes. The team argues that mapping these structures will not only prevent model misbehavior but may eventually yield foundational insights into human neuroscience and complex system failures. The founders also emphasize the critical emergence of state capacity, noting that newly established government evaluation bodies provide essential public oversight so private labs do not manage the transition alone. Beyond technical governance, the founders cite immediate productivity shifts—such as a recent room survey at Y Combinator where 95% of attendees reported using Claude for programming—and forecast that frontier models will accelerate transformative scientific breakthroughs akin to AlphaFold across medicine and cancer research. While the co-founders express optimism about steering frontier models and normalizing rigorous safety evaluations, they acknowledge that defining precise capability thresholds remains fraught with gray areas, demanding continuous recalibration as models encounter novel operational environments.

Anthropic 联合创始人回顾了团队从物理学与早期 AI 实验室到创建商业机构的历程，指出 AI 安全与模型规模扩展在结构上密不可分，必须通过务实的工程化框架融入前沿开发。他们认为，通过建立透明的负责任扩展政策与可解释性研究，安全措施不仅能够转化为商业壁垒，还能推动全行业建立“向上竞争”的安全标准。 规模扩展与安全研究的结构性交织 (2:30) Dario Amodei 与团队成员回顾了早期的探索阶段，当时主流观点担忧超级 AI 无法理解人类价值观。团队发现，语言模型是让人工智能理解隐性人类知识的关键，但更核心的洞察在于：要让基于人类反馈的强化学习（RLHF）生效，必须先扩大模型规模，因为参数量较小的模型不够聪明，无法有效执行对齐训练。这一认知确立了团队的核心哲学，即规模化（Scaling）与安全性并非对立，而是互为前提。 随后，Jared Kaplan 等人提出的“宪政 AI”（Constitutional AI）将这一理念转化为实际工程方案。与外界最初将其视为科幻概念不同，团队发现通过提供一套明确的书面原则，模型能够根据原则反思自身输出并进行自我对比训练。Dario 认为这印证了“苦涩的教训”（bitter lesson）：只要确定清晰的训练目标并供给适当的数据，大模型强大的模式匹配能力就能高效执行对齐，使安全解决方案成为模型能力增长的“自然红利”。 负责任扩展政策与安全评估落地 (18:26) 2022 年底，Dario Amodei 与 Paul Christiano 开始构思负责任扩展政策（RSP）。团队意识到对模型能力设定单一的硬性上限并不现实，转而设计了一套基于能力阈值的分级体系（如 ASL-2、ASL-3 等）。每当模型突破特定的能力门槛时，系统必须触发对应的安全与物理安防保障措施。Anthropic 期望将这一机制制度化，使其像汽车工业的安全测试或常规财务审计一样“枯燥且标准”，从而在内部消除部门目标冲突，并避免因频繁误报而引发警报疲劳。 在实际执行中，RSP 严重依赖密集的评估体系。团队指出，AI 能力评估具有不对称性：确定模型能力的下限相对简单，但探测其上限极其困难，因此需要 Frontier Red Team 等专门团队持续探索思维链提示、工具调用等可能绕过限制的危险能力（包括化学、生物、放射性及核风险等 CBRN 威胁）。为了避免文书过于官僚化，领导层近期重写了 RSP，降低理解门槛，确保基层工程师与董事会能够顺畅对接，像工厂工人检查安全带一样随时识别潜在安全漏洞。 反向共识与低政治内耗的组织构建 (9:37) 针对 AI 领域的早期研究文化，Jared Kaplan 与 Chris Olah 指出，传统学界长期受“AI 寒冬”的历史创伤困扰，普遍存在过度保守的心态，甚至认为设定雄心勃勃的愿景是不合规矩的。Dario Amodei 总结道，主流的“成熟共识”往往只是伪装成理性的从众行为。面对频繁变动的行业共识，团队选择坚定押注算力扩张与逆向非主流赛道。即便只有一半的判断正确，探索无人涉足的路径依然具备极高的系统价值。 离开 OpenAI 创办 Anthropic 是出于责任感与务实考量。Tom Brown 曾倾向于建立纯粹的安全研究非营利组织，但前沿 AI 对工程与资本的极高需求迫使团队转向商业架构，并确立了“少许诺、多兑现”的原则。Daniela Amodei 强调，团队在招募中严苛筛选，全员统一签署 80% 承诺，形成低内耗、低自负的文化。公司避免让产品、研究、合规与商业部门在孤岛中相互制约，而是将权衡取舍下放至跨职能协作中，由全员共同承担不完美决策。 商业驱动的向上竞争与未来技术愿景 (35:57) Anthropic 坚信安全不应是自我孤立的道德纯度测试，而应成为市场上的差异化竞争力。客户选择 Claude 的核心原因之一正是其“有益且无害”的可靠性。随着商业模式被验证，Anthropic 对行业产生了“重力拉引”，其 RSP 框架在发布后数月内即被三家主流 AI 企业借鉴，红队模型与可解释性研究亦被广泛采纳，形成了推动整个行业跟进的“向上竞争”。在应用端，代码生成已出现显著的相变突破，联合创始人提到在近期 Y Combinator 的演讲中，现场已有 95% 的人举手表示使用 Claude 辅助编程。 在基础科学层面，Chris Olah 将神经网络视作充满复杂结构的“人工生物学”，预测未来十年可解释性将演进为如同生物学般的独立学科，甚至有望为理解人类大脑运作及解决精神疾病提供全新工具。此外，团队观察到全球政府正在快速建立针对前沿 AI 的评测机构，这一公共能力的增长将平衡私人企业的影响。从辅助疫苗研发、癌症攻关，到受 AlphaFold 启发孵化更多生物学突破，团队寄望 AI 最终成为增强自由与民主制度的赋能工具。 Anthropic 联合创始人强调，当前科学界正处于能够直接观察并实证表征前沿模型潜在风险的关键窗口期。然而，如何在极高资本消耗与白热化竞争中持续严守安全底线，以及能否将内部的可解释性与评测体系有效推广为全行业的强制规范，依然是未来几年决定人工智能技术长期轨迹的核心考验。

## Background 该内容由 Anthropic 官方频道于 2024 年 12 月 20 日发布在 YouTube 平台上。视频简介指出，这是一场 Anthropic 联合创始人之间的深度对话，参与者包括 Chris Olah、Jack Clark、Daniela Amodei 等七位核心成员，共同探讨了公司的过去、现在以及人工智能的未来发展。 ## TL;DR 在这场对话中，联合创始人团队回顾了从早期在 OpenAI 确立规模法则（Scaling Laws）到创立 Anthropic 的历史使命感，深入阐述了将模型扩展与 AI 安全视为一体的共识。他们详细讨论了负责任扩展政策（RSP）与宪法级 AI（Constitutional AI）的诞生、演进及其在消除政治化争论与政策对齐中的务实价值。最后，团队展望了 AI 在神经网络可解释性、“人工生物学”、前沿科学医学研究以及增强民主等领域的未来突破前景。 ## Atmosphere & style 整个对话充满了一种极具韧性且真诚的“低自尊、低内耗”氛围，联合创始人之间展现出跨越十余年、极其罕见的深厚信任与默契。言谈中既有对早期被当成疯子等往事的轻松幽默与自嘲，又在谈及 AGI 历史使命、拒绝“高尚的失败”时流露出坚定、纯粹且极端的务实主义。观众能清晰地感受到这群顶尖科学家和跨界领袖在面对强大技术变革时，那份将科学好奇心与严谨工程实证融为一体的踏实感与责任感。 ## Key takeaways - [00:00](https://www.youtube.com/watch?v=om2lIWXLLN4&t=0s) 联合创始人团队大多有超过十年的深厚渊源，通过展示计算量和参数规模的增长趋势，团队早期便确信 AI 的发展远比预想的更为紧迫。 - [02:30](https://www.youtube.com/watch?v=om2lIWXLLN4&t=150s) 团队将语言模型（LLM）视为解决 AI 安全危机的天然载体，认为扩展模型规模（Scaling）与确保模型安全（Safety）在本质上是同一枚硬币的两面。 - [04:10](https://www.youtube.com/watch?v=om2lIWXLLN4&t=250s) 经典的《AI 安全中的具体问题》论文将抽象的哲学讨论锚定在具体的机器学习技术中，为学术界和工业界构建了 AI 安全的科学共识。 - [04:10](https://www.youtube.com/watch?v=om2lIWXLLN4&t=250s) 宪法级 AI（Constitutional AI）的核心逻辑是利用模型自身的选择与阅读能力，将人类撰写的原则转化为直接可优化的训练目标，证明了简单机制在 AI 中的有效性。 - [07:23](https://www.youtube.com/watch?v=om2lIWXLLN4&t=443s) 物理学背景赋予了团队敢于思考宏大框架的视角，从而打破了早期 AI 领域受“AI 冬天”阴影影响而缺乏野心的学术界保守主义共识。 - [10:56](https://www.youtube.com/watch?v=om2lIWXLLN4&t=656s) 近年来全球对 AI 的态度发生剧变，公众对宏观社会影响的担忧日益加剧，AI 从科技界边缘话题演变为受到白宫等国家高层密切关注的重大事件。 - [14:19](https://www.youtube.com/watch?v=om2lIWXLLN4&t=859s) 各类型工程师在 AI 团队中的关键作用已成为行业共识，OpenAI 早期明确表示可以通过工程工作为 AI 安全做贡献，这促成了团队的多元化构建。 - [15:42](https://www.youtube.com/watch?v=om2lIWXLLN4&t=942s) 在 GPT-3 研发尾声时，团队首次将信任与安全（Trust and Safety）概念引入大模型领域，为 AI 安全研究从理论迈向实际应用搭建了桥梁。 - [18:30](https://www.youtube.com/watch?v=om2lIWXLLN4&t=1110s) 负责任扩展政策（RSP）是 Anthropic 最核心的制度创新，它通过多级“安全阈值”和具体的评估测试，将抽象的 AI 安全转化为务实、可解的行业规范。 - [18:30](https://www.youtube.com/watch?v=om2lIWXLLN4&t=1110s) 明确且制度化的 RSP 有效消除了团队内部的“政治化”争论，防止滥用安全借口盲目推进或粗暴叫停项目，并避免了高频误报导致的“狼来了”效应。 - [28:08](https://www.youtube.com/watch?v=om2lIWXLLN4&t=1688s) 准确测定模型的“能力上限”极其困难，因为模型会利用思维链或工具调用等意想不到的方式协助高危操作，这使得具体的安全评估（Evals）成为驱动一切的核心。 - [28:58](https://www.youtube.com/watch?v=om2lIWXLLN4&t=1738s) 团队对 RSP 进行了文本重写以去除技术官僚化和对抗性语气，使其成为全员烂熟于心、人人皆可直观理解并执行的普适性企业文化文件。 - [30:42](https://www.youtube.com/watch?v=om2lIWXLLN4&t=1842s) 创办 Anthropic 并非为了公司本身，而是出于强烈的历史使命感，旨在通过掌握企业架构的主导权，避免 AI 发展陷入旧有科技巨头的套路，全员甚至签署了资产捐赠承诺。 - [32:45](https://www.youtube.com/watch?v=om2lIWXLLN4&t=1965s) 低自尊、低内耗和高度团结的“低政治”文化是 Anthropic 的核心战略优势，产品、研究、安全及政策等部门在同一个“变革理论”下协同作战。 - [39:08](https://www.youtube.com/watch?v=om2lIWXLLN4&t=2348s) Anthropic 的核心商业逻辑是“向顶端赛跑”（Race to the top），通过商业成功迫使竞争对手像素级复制其在安全领域的扎实投入，从而输出安全红利。 - [42:35](https://www.youtube.com/watch?v=om2lIWXLLN4&t=2555s) 团队强烈反对通过不计后果的“高尚的失败”来论证 AI 的危险性，坚持在不牺牲市场竞争力的前提下通过激励对齐来吸引并带动全行业走向安全彼岸。 - [43:43](https://www.youtube.com/watch?v=om2lIWXLLN4&t=2623s) 神经网络的训练展现出一个奇妙的“人工生物学”世界，对其内部优美结构的可解释性（Interpretability）研究，未来甚至可能为人类理解和治疗精神疾病提供全新模型。 - [45:38](https://www.youtube.com/watch?v=om2lIWXLLN4&t=2738s) 各国政府建立起测试和评估 AI 系统的全新机构，这种国家能力的构建分担了企业的安全责任；同时，Claude 在科学研究和编程领域的爆发式应用标志着相变时代的到来。 - [48:07](https://www.youtube.com/watch?v=om2lIWXLLN4&t=2887s) AI 在生物学中的应用正在打破旧有质疑，未来的目标是构建能创造出“100 个 AlphaFold”的工具，并进一步探索如何让 AI 成为促进自由与增强民主的工具。 ## Detailed walkthrough ### [00:00](https://www.youtube.com/watch?v=om2lIWXLLN4&t=0s) 为何投身 AI：早期的交集与信念的形成 对话在一场轻松的探讨中拉开序幕，联合创始人兼 CEO Dario Amodei 随机点名 Jared Kaplan，询问他最初为何决定从事 AI 领域的研究。Jared 坦言自己此前研究了很长时间的物理学，后来觉得有些枯燥，同时也希望能和更多朋友在一起。Dario 补充道，自己当时并没有对 Jared 进行什么“硬推销”，只是不断向他展示各种 AI 模型的运行结果，试图证明这些模型具有极强的通用性而非仅限于单一领域；在看了足够多的成果后，担任了大约六年教授的 Jared 终于认同了这一趋势。与此同时，Chris Olah 回忆起自己 19 岁首次访问旧金山湾区时便结识了当时还是博士后研究员的 Dario 和 Jared，后来在 Google Brain 期间与 Dario 成为相邻案头的同事，并与 Tom Brown 合作，最终又在 OpenAI 与大家重聚。联合创始人之间的渊源大多已跨越十余年。 Jack Clark 则分享了他在 2015 年作为一名科技记者尝试采访 Dario 的往事。当时 Dario 正在 Google 撰写开创性的论文《AI 安全中的具体问题》（*Concrete Problems in AI Safety*）。Daniela Amodei 提到，Dario 总是能给周围的人带来某种“醍醐灌顶”的震撼——通过展示计算量增长、参数规模以及大脑神经元数量的对比，让人瞬间意识到 AI 的发展远比预想的更为严肃和紧迫，Dario 搞笑地自嘲自己是“好消息的带来者”。随后，团队在 OpenAI 体验到了 Scaling Laws（规模法则）的威力：从 GPT-2 到 GPT-3，模型在不断扩大规模的过程中展现出惊人的效果，而正是这种“不断奏效”的奇迹将这群人紧密凝聚在一起。 - Chris Olah 在 19 岁首次前往湾区时便结识了 Dario Amodei 和 Jared Kaplan。 - Dario 与 Chris 在 Google Brain 工作期间曾紧密相邻而坐。 - Jack Clark 于 2015 年作为记者首次采访 Dario，当时 Dario 正专注于 AI 安全研究。 --- ### [02:30](https://www.youtube.com/watch?v=om2lIWXLLN4&t=150s) 语言模型、人类反馈强化学习（RLHF）与 Scaling 的安全纽带 在 GPT-2 和 GPT-3 的探索时期，团队逐渐意识到语言模型与安全研究之间不可分割的联系。当时业界普遍担忧未来的 AI 系统可能会极其强大，却无法理解人类价值，甚至无法与人类进行有效沟通。联合创始人团队将大语言模型（LLM）视为解决这一危机的天然载体，因为语言模型本身就要求系统必须理解人类社会中隐含的知识与价值体系。 为了使模型能够接受人类反馈强化学习（RLHF），就必须先将其规模（Compute & Parameters）扩大到足够聪明的程度。因此，扩展模型规模（Scaling）与确保模型安全（Safety）在本质上是同一枚硬币的两面。Dario 当时在 OpenAI 组建的 AI 安全团队领衔了许多 early scaling（早期模型扩展）的研究，其初衷正是为了通过精准预测 AI 发展趋势，从而让整个学术界和工业界真正重视 AI 安全问题。 - Jack Clark 回忆曾在英国某个机场提取 GPT-2 的生成样本撰写虚构新闻，并给 Dario 发 Slack 消息感叹其巨大的政策影响，Dario 当时以典型的极简风格回复了“Yes”。 - 早期 GPT-2 的发布与安全讨论，为 Anthropic 后来采取的大规模、安全导向的运作模式奠定了经验基础。 --- ### [04:10](https://www.youtube.com/watch?v=om2lIWXLLN4&t=250s) 从《AI 安全中的具体问题》到宪法级 AI（Constitutional AI）的突破 话题转向了 2016 年发表的经典论文《AI 安全中的具体问题》。Dario 笑着承认，这篇论文在某种程度上是他为了拖延当时手头其他工作而与 Chris Olah 共同头脑风暴的产物。当时关于 AI 安全的讨论大多停留在极其抽象、深奥的哲学层面上，而这篇论文的真正价值在于将其 anchor（锚定）在具体的机器学习技术之中。Chris 指出，虽然从现在的技术眼光来看论文中的某些具体问题可能显得有些过时，但它在当时起到了极大的共识构建（consensus-building）作用，让主流机器学习界开始承认 AI 安全是一个值得严肃对待的真实科学课题。 这种将创新点落地的思路延续到了 Anthropic 成立之初。当 Jared Kaplan 和 Dario 提出“宪法级 AI”（Constitutional AI，简称 CAI）的构想——即直接给语言模型写一套“宪法”来指导和改变其全部行为时，许多人觉得这个想法简直疯狂。然而 Jared 和团队坚信“简单的机制在 AI 中往往极其有效”。他们利用模型擅长解答多选题的特性，通过 Prompt 引导模型对照既定原则自我审查与修正行为，从而将极其复杂的伦理对齐问题转化为清晰可训练的目标。正如 Scaling Hypothesis（规模假设）或 Bitter Lesson（苦涩的教训）所揭示的，只要设定好清晰的数据和训练目标，模型就能在持续计算中学会执行。 - Dario Amodei 坦言撰写《AI 安全中的具体问题》最初包含对其他工作的“合法拖延”。 - Chris Olah 为了发布该论文，曾与 Google Brain 的 20 多位研究员逐一沟通以争取支持。 - Constitutional AI 的核心逻辑：利用模型自身的选择与阅读能力，将人类撰写的原则（Principles）转化为直接可优化的训练目标。 --- ### [07:23](https://www.youtube.com/watch?v=om2lIWXLLN4&t=443s) 物理学家视角、学术界保守主义与历史镜像 Jared Kaplan 结合自己从理论物理学转型到 AI 的经历，指出当时的 AI 研究者群体普遍带有“AI 冬天”（AI Winter）留下的心理创伤，导致整个领域不敢拥有过于宏大的野心或 vision，甚至认为讨论宏伟的技术前景是一种禁忌。然而，物理学家往往“相当自傲”，习惯于从宏观框架和宏大设想去思考问题。要真正重视 AI 安全，首先必须敢于相信 AI 能够变得无比强大。 Dario 和 Chris 分析指出，这种保守倾向实际上是学术界普遍存在的避险机制（risk-averse mentality），甚至连工业界在很长一段时间内也照搬了这种心态，直到 2022 年左右才真正打破。他们将这种现象与 1939 年核物理学早期关于原子弹的争论相类比：当时以 Fermi（费米）为代表的保守派出于谨慎对概念保持怀疑，而 Szilard（西拉德）和 Teller（泰勒）等人则因担忧潜在风险而选择极其严肃地对待这一技术。 Dario 强调，过去十年给他留下最深感悟的是：许多看似明智、成熟的“共识”，本质上只是伪装成成熟的从众行为（herding behavior）。当一个人多次目睹某种社会共识在短时间内彻底颠覆后，就会明白即使只有 50% 的把握，也应该坚定地对那些尚未被他人关注的核心非共识（counter-consensus）方向下注。 - Jared Kaplan 认为 AI 领域早期缺乏野心是受历史“AI 冬大”心理阴影的影响，而物理学背景带来了敢于思考宏大框架的视角。 - Dario 将 AI 发展的保守心态比作 1939 年 Fermi 与 Szilard、Teller 在核武器可行性上的分歧。 - Dario 的核心心得：打破所谓“成熟共识”的从众心理，在关键问题上敢于做出非共识的坚定下注。 --- ### [10:56](https://www.youtube.com/watch?v=om2lIWXLLN4&t=656s) 全球情绪的剧变与拜访白宫 近年来，公众和业界对 AI 的态度发生了戏剧性的转变。Daniela Amodei 提到，在 Anthropic 进行的用户调研中，普通大众和客户越来越频繁地表达对 AI 带来宏观社会影响的担忧——不仅关乎具体的就业岗位、偏见和毒性，更关乎 AI 是否会从根本上改变人类社会合作与运行的方式。有趣的是，机器学习领域的专业研究者过去在预测 AI 威力方面往往比普通大众更为悲观和保守。 Jack Clark 与 Dario 提及他们在 2023 年受邀前往白宫参加会议的经历。当时美国副总统 Harris（哈里斯）和商务部长 Raimondo（雷蒙多）明确告诉他们：“我们正密切关注你们，AI 将成为一件极其重大的大事。”这一场景在几年前是完全不可想象的。Jack 回忆起自己 2014 年作为科技记者绘制 ImageNet 图表、2015 年劝说彭博社（Bloomberg）报道 NVIDIA 的 GPUs 作用时被当成疯子，甚至 2016 年离职加入 AI 行业时还收到邮件劝阻称“你在犯下人生中最严重的错误”。短短几年间，整个世界对大模型与扩展范式的态度已不可同日而语。 - 2023 年，Dario Amodei 与 Jack Clark 受邀前往白宫参加 AI 专题会议，与 Vice President Harris 及 Commerce Secretary Raimondo 等高官面对面交流。 - Jack Clark 在 2015 年试图推动关于 NVIDIA GPU 在 AI 中关键作用的报道时，曾被媒体编辑认为“彻底疯了”。 - 2016 年 Jack 离开新闻界投身 AI 领域时，曾被同行警告“正在犯下人生最大的错误”。 ### [14:19](https://www.youtube.com/watch?v=om2lIWXLLN4&t=859s) 从 OpenAI 的早期日子谈起：转型期的决断力与工程师的崛起 几位联合创始人回忆起他们最初决定投身人工智能领域时的情景。Sam McCandlish 笑称，当时他为了给自己一个了断，甚至对前东家提出了“如果拒绝给我双倍薪水我就离职”的要求，虽然明知对方不会答应，但第二天醒来他还是轻松地递交了辞呈。大家调侃他是一个极其果断的人，而 Sam 将这种决断力归功于当时大环境带来的震撼——他那时每天沉浸在阅读 arXiv 的前沿论文中，甚至包括 Dario Amodei 早期在百度（Baidu）所做的研究。他清晰地意识到：“这里正在发生一些完全疯狂的事情。”在座的每一个人在职业生涯的某个节点，都曾凭借这种坚定的信念在 AI 领域压下了重注。 与 Sam 的果断不同，Jared Kaplan 承认自己当时纠结了大概六个月，在“是否要加入这家初创公司”之间反复权衡。Chris Olah 补充道，在那个时期，行业内普遍认为只有“研究员（Researcher）”才能从事 AI 工作，根本没有人谈论工程师在 AI 领域能发挥多大作用。而如今，各类型工程师在 AI 团队中的关键作用已成为行业共识。正是因为 OpenAI 当时明确表示“可以通过工程工作来为 AI 安全（AI safety）做出贡献”，才最终促成了 Jared 的加入，这种机会在当时是绝无仅有的。 --- ### [15:42](https://www.youtube.com/watch?v=om2lIWXLLN4&t=942s) 组织管理的演变：从 Stripe 到 OpenAI 的实用主义转型 随后，话题转向了 Daniela Amodei 的加入经历。Chris 提到 Daniela 曾是他在 OpenAI 时的直属主管。Daniela 分享道，在加入 OpenAI 之前，她在 Stripe 工作了大约五年半，当时的顶头上司正是 Greg Brockman。在 OpenAI 创立之初，正是 Daniela 将 Greg 与 Dario 牵线搭桥，她当时对 Greg 说：“我认识的最聪明的人就是 Dario，如果你能招到他，那将是非常幸运的。” 在离开 Stripe 后，曾有非营利组织和国际发展背景的 Daniela 原本打算回归公共卫生或政治领域，去帮助那些弱势群体。然而，当时的 OpenAI 作为一个拥有宏大远景的非营利组织，展现出了极具吸引力的交集。她坦言，当时的 OpenAI 虽然聚集了一批意图良好且极具天赋的人才，但组织管理上“有点混乱（a little bit of a mess）”，这恰好为她提供了一个完美的“多面手（utility player）”舞台。她开始接管人员管理，重组技术团队，并深度参与了扩展组织（scaling org）、语言团队以及政策制定等工作，极大地提升了公司的运转效率。 - **关键人物关系**：Daniela Amodei 曾引荐 Greg Brockman 与 Dario Amodei 结识，促成了 OpenAI 早期的核心团队构建。 - **信任与安全（Trust and Safety）的引入**：在 GPT-3 研发接近尾声时，Daniela 凭借在 Stripe 管理信任与安全团队的经验，首次向团队引入了这一概念，为 AI 安全研究从理论迈向实际应用搭建了桥梁。 --- ### [18:30](https://www.youtube.com/watch?v=om2lIWXLLN4&t=1110s) 核心文件的诞生：负责任扩展政策（RSP）的迭代与价值 话题自然地过渡到了 Anthropic 最核心的制度创新——负责任扩展政策（Responsible Scaling Policy, 简称 RSP）。Dario 回忆道，这个概念最早由他和 Paul Christiano 在 2022 年底提出。最初的设想非常朴素：是否应该在某个特定节点对模型扩展（Scaling）进行封顶，直到某些安全问题得到解决？但他们很快意识到单一的硬性封顶并不合理，于是将其演变为一系列多级的“安全阈值（Thresholds）”。在不同的阈值下，必须对模型执行特定的能力测试，并采取相应级别的安全防护措施。 为了让这项政策更具行业普适性，他们最初希望由第三方来主导设计，因此 Paul 离职去开展相关设计工作。当 Paul 公布该概念后，Anthropic 在一两个月内便推出了自己的 RSP 方案。这个文件经历了无数稿的修改，被团队调侃为“修改次数最多的一份文档”。Jack Clark 将其生动地比喻为 Anthropic 的“宪法（Constitution）”，是凝聚公司共识的“圣经”。整个 RSP 的落地不仅需要 Dario、Paul、Sam 和 Jared 等人提供底层的哲学原则，更需要强大的工程与组织运营支撑，包括近期为了明确权责而进行的 RSP 组织架构调整。 - **安全是可解的问题**：团队一致认为，AI 安全并非不可逾越的鸿沟，而是一个可以通过大量投入和机制建设来解决的硬课题。 - **消除“政治化”争论**：明确且制度化的 RSP 提供了清晰的衡量标准，有效防止了团队内部滥用“安全”借口来盲目推进或粗暴叫停某个项目。 - **防范“狼来了”效应**：Dario 强调，安全工作必须精准校准。如果像某些建筑的火警警报一样每周误报，当真正的危险（火灾）来临时，反而没有人会在意。 --- ### [25:11](https://www.youtube.com/watch?v=om2lIWXLLN4&t=1511s) 实践中的挑战与外部沟通：灰产地带与“无聊的”行业标准 虽然 RSP 的框架非常完美，但联合创始人也坦言其实施过程远比预想的要艰难和复杂。Jared 指出，当面对如此前沿的全新技术时，很多时候模型的能力并非非黑即白，而是存在大量的“灰色地带（gray areas）”，这些是无法提前预测的，必须通过三到四轮的实际迭代，才能逐渐摸索出正确的评估边界和政策肌肉。 负责 Anthropic 算力（Compute）资源的 Sam 则从外部沟通的角度肯定了 RSP 的巨大价值。在与不同预期速度的外部利益相关者（如政府官员或参议员）沟通时，RSP 变成了一种极其直观的语言。当面对那些认为 AI 威胁还很遥远的人时，Sam 可以明确告诉他们：“既然威胁还未达到阈值，我们目前就不需要采取极端安全措施。”这种务实的做法将原本听起来有些玄学的 AI 安全，转化成了一种类似于“财务审计”一样极其正常且枯燥的商业规范。 - **评估（Evals）驱动一切**：如今 RSP 已经深入到 Anthropic 的日常产品规划中。从前沿红队（Frontier Red Team）到训练团队，几乎所有团队都在疯狂地进行模型评估，以测量模型是否达到了可能引发危险的能力阈值。 ### [28:08](https://www.youtube.com/watch?v=om2lIWXLLN4&t=1688s) 评估模型的“危险上限”与政策对齐 在对人工智能模型的能力进行评估时，确定其能力的下限相对容易，但要准确测定其“上限”却极其困难。团队投入了大量的研究精力，只为了解答一个核心问题：“这个模型到底能不能做出某些危险的事情？”因为模型可能会利用一些人类尚未想到的技巧——比如思维链（Chain of Thought）、最佳结果筛选（best of N）或某种特定的工具调用方式，从而以意想不到的方式协助用户执行高危操作。 这种极其具体的评估方法在政策制定领域发挥了巨大的建设性作用。在此之前，“AI安全”对政策制定者而言一直是个抽象的政治概念。但当团队能够明确指出“我们拥有一套具体的评估测试（Evals），它的结果将直接决定我们是否部署该模型”时，情况就完全不同了。这使得团队能够与政策制定者、国家安全专家以及防范化解大规模杀伤性武器（CBRN）等领域的专业人员进行精准的对齐与校准。这种具体的评估机制极大地激发了外部专家的积极性，共同帮助推动评估准确性的提升，从而创造了原本不可能实现的合作反事实。 --- ### [28:58](https://www.youtube.com/watch?v=om2lIWXLLN4&t=1738s) 负责任扩展政策（RSP）的去技术官僚化与企业文化 负责任扩展政策（Responsible Scaling Policy, 简称 RSP）在公司的日常运营中无处不在。然而，团队近期对 RSP 进行了大规模的文本重写，核心在于调整其“语气”。原版的 RSP 显得过于技术官僚化（technocratic），甚至带有某种微妙的对抗性。团队花费了大量时间思考如何构建一个让全员都产生归属感的系统。 正如公司通过 OKR（目标与关键结果）进行管理一样，RSP 应当成为全公司所有员工都烂熟于心的共识。无论是技术人员还是非技术人员，大家都应该清晰地知道：RSP 的核心目标是什么？我们如何衡量是否达标？我们目前处于哪个 AI 安全等级（AI Safety Level），是 ASL-2 还是 ASL-3？如果 RSP 流于技术官僚化，变成只有少数特定人群才能理解的行业黑话，其生产力就会大打折扣。重写后的 RSP 变成了一份人人皆可轻松阅读的普适性文件，员工能直观地理解这些安全规则的合理性、警惕潜在风险的原因，并在实际工作中敏锐地察觉异常。这就好比汽车制造厂的工人在流水线上能一眼看出安全带的接头是否报错位一样。通过这种方式，在公司管理层、董事会以及一线研发人员之间建立起健康的反馈流，彻底避免因沟通脱节或“线路交叉”而导致灾难性错误的发生。 --- ### [30:42](https://www.youtube.com/watch?v=om2lIWXLLN4&t=1842s) 创立 Anthropic 的初衷：并非为公司而公司 事实上，创始团队中没有任何人最初的梦想是“创办一家公司”。大家之所以聚在一起，完全是出于一种强烈的责任感和历史使命感。在 2020 年，随着全员都曾参与或接触过的 GPT-3 的问世以及扩展定律（Scaling Laws）的确证，AGI 的未来轨迹已经清晰地展现在眼前。如果大家不迅速联合起来采取行动，整个行业很快就会跨越“无法挽回的点”。创办 Anthropic，是大家为了在这个剧烈变化的生态中保留改变环境能力而选择的必然路径。 在求学阶段或担任物理学教授期间，团队成员就曾致力于探索如何以促进公共利益的方式推进科学研究。随着 AI 展现出极为陡峭的发展轨迹，且对算力和资本的渴求远超学术界所能承受的极限，创办一个新的机构成为了唯一的解。创始人们坦言，如果只是为了追求个人财富或权力，他们绝不会推荐任何人去创办公司，这纯粹是达成技术向善这一最终目的的必要手段。如果不用这种方式自己掌握环境和企业架构的设立主导权，AI 的发展就会陷入科技界旧有的、令人窒息的套路中——同样的人群、同样的态度、同样的模式复刻。 * 创始团队成员曾发起过名为 "Project Vannevar" 的项目尝试，旨在探索让科学推进公共利益的制度化路径。 * 团队为了向共同的使命表达决心，全员共同签署了 80% 资产捐赠的 pledge 承诺。 --- ### [32:45](https://www.youtube.com/watch?v=om2lIWXLLN4&t=1965s) 信任、低政治与企业文化的高效扩展 坐在会议桌前的团队成员之间拥有一种极其罕见且深厚的“信任”，这是 Anthropic 最大的核心战略优势。在科技行业中，想要让一个庞大的团队长期保持高度一致且纯粹的使命感是极难做到的。团队将这种文化的成功扩展归功于团队内部的“首席小丑驯兽师”（Chief Clown Wrangler），正是因为她始终坚持极高的招聘标准，才成功把投机者阻挡在门外，让整个公司的文化显得既友善又纯粹。 Anthropic 内部呈现出一种极具韧性的“低政治、低内耗”特征。这得益于全员的“低自尊/低利己（low ego）”以及对政治手段天然的免疫与排斥反应。更为关键的是公司内部的“高度团结（Unity）”。在很多功能不健全的企业中，不同部门往往为了不同的利益目标而相互掣肘，甚至认为其他部门在暗中削弱自己的成果——例如安全部门在前线修补漏洞，而业务部门在后方制造破坏。但在 Anthropic，产品团队、研究团队、信任与安全团队、市场拓展团队以及政策团队，全部在同一个“变革理论（Theory of Change）”下协同作战。 * 联合创始人透露，当初离开 OpenAI 时其实内心充满挣扎与抗拒，因为他当时并不确定世界上多出一家 AI 实验室对人类是不是一件好事。 * 在创立初期，团队曾花很长时间争论是否应该成立一家纯粹的安全研究非营利组织，但最终面对现实的资金与算力约束，极端的务实主义促成了 Anthropic 这家公司的诞生。 --- ### [36:55](https://www.youtube.com/watch?v=om2lIWXLLN4&t=2215s) 极端务实主义与跨部门的权衡共担 在公司创立初期，团队就确立了一条重要准则：“少做承诺，多践行承诺”。保持清醒的认知、正视现实的残酷交替（trade-offs），因为信任与公信力远比任何单一政策更为重要。在 Anthropic，务实与安全的观念已经渗透到了非同寻常的深度：你可以看到首席技术官 Mike Krieger 在一线为了安全红线坚定地捍卫“暂缓发布产品”的决定，同时也能看到业务负责人 Vinay 积极地从商业视角出谋划策，探讨如何安全地将产品推过终点线；技术安全团队会主动考虑产品的实用性，而负责推理（Inference）的工程师同样在时刻关注安全风险。 健康的组织应当将复杂的权衡从最高管理层下放到每一个具体岗位的员工身上。在功能失调的职场中，安全、产品和研究部门各执一词、互不相让，最后将矛盾踢给高层，而高层掌握的信息其实远不如一线充分。而在 Anthropic，大家都清晰地意识到世界是不完美的，任何决策都是在特定岗位上共同面对次优解（suboptimal）的博弈，所有人都在为同一个综合权衡的整体目标而努力。 --- ### [39:08](https://www.youtube.com/watch?v=om2lIWXLLN4&t=2348s) “向顶端赛跑”与安全红利的市场化输出 Anthropic 的核心商业逻辑是“向顶端赛跑（Race to the top）”的赌注。这并非一个毫无风险的纯上行赌注，但大家有着高度一致的共识。市场本质上是极其务实的：Anthropic 在商业上越成功，市场就越有动力去像素级复制那些让我们获得成功的要素。当我们的商业成功与我们在安全领域的扎实投入深度绑定时，就会在整个 AI 行业中产生一种强大的引力场，逼迫其他竞争对手不得不加入这场关于安全的良性竞争。用形象的话来说就是：“我们负责把安全带造出来，其他人随便抄，这是个更好的世界。” 如果一家公司只因为害怕风险而选择不去研发前沿技术，或者做不出比别人更好的技术，这种消极防御是行不通的，因为你无法向世界证明“从这里走到安全彼岸”的现实可行性。无论是行业还是整个人类社会，都必须成功度过这个技术从无到有、再到极其强大的危险转型期。如果能在确保安全的同时完全不牺牲技术的市场竞争力，找到这种双赢的解法，那么外界的监管环境、顶尖人才的流向以及客户的自主选择，都会形成合力，共同推着整个行业往前走。 * 这种“安全红利输出”在实践中屡试不爽：在 Anthropic 推出 RSP 几个月后，行业内最知名的三家 AI 公司也迅速跟进推出了自己的 RSP 政策；前沿红队（Frontier Red Team）的评估机制也几乎立即被行业其他实验室像素级复刻。 * 这种机制同样体现在可解释性研究（Interpretability research）以及与各大国家级 AI 安全研究所（AI Safety Institutes）的深度合作中。 * 市场压力同样来自客户的真实心声：B端客户极度抗拒模型产生幻觉或被轻易提示词劫持（jailbreak），他们强烈渴求“既有用又无害（helpful and harmless）”的模型。在大量的客户电话中，客户明确表示选择 Claude 的原因就是“因为我们知道它更安全”。 --- ### [42:35](https://www.youtube.com/watch?v=om2lIWXLLN4&t=2555s) 拒绝“高尚的失败” 针对某种在科技界流传的叙事——即认为真正崇高且有美德的做法应该是“高尚的失败”（nobly fail），即通过一种完全不切实际、不计商业后果的方式去死守安全，从而以此来向世界论证 AI 的危险性并呼吁创新在此止步，团队对此表达了强烈的反对。Anthropic 的 co-founders 明确表示：“我们不是毁灭论者（doomers）。”团队希望能打造出积极、创新且让客户感到惊艳的正面事物，同时清晰且坚定地划出系统安全的底线。 ### [42:54](https://www.youtube.com/watch?v=om2lIWXLLN4&t=2574s) 激励对齐与良性竞争 在讨论决策机制时，发言者指出，如果以牺牲影响力和话语权为代价来追求所谓对“事业的纯洁性”，实际上是非常适得其反的（self-defeating）。这种做法会导致一个糟糕的后果：最终掌握决策权的人，将全都是通过逆向自我选择留下来的、根本不在乎安全性、也不以此为优先级的人。 相反，如果能竭尽全力去寻找一种方法来对齐激励机制（align incentives），确保在面临艰难决断时，最关键的节点能拥有最强大的推动力和最充足的证据来支持正确决定，就能触发一种“向顶端竞赛”（race to the top）的良性循环。这样一来，重视安全的人就不会被边缘化或挤出权力中心，反而能吸引并带动其他人一同跟随。 --- ### [43:43](https://www.youtube.com/watch?v=om2lIWXLLN4&t=2623s) 神经网络的可解释性与“人工生物学” 当谈到未来最让人兴奋的工作时，Chris Olah 分享了他对可解释性（interpretability）的独特视角。除了众所周知的安全性贡献外，他在情感层面上感到同样兴奋的是神经网络本身所蕴含的美感。目前人们往往把神经网络当成不透明的“黑盒子”，对其内部构造不感兴趣；但只要开始深入探究，就会发现其中充满了令人惊叹的优美结构。 这就像人们过去看待生物学一样，误以为进化是一个漫长而无聊的机械过程，最终只是凑巧产出了动物；但事实上，进化作为一种优化过程，其孕育出的每一个生命都包含着难以置信的复杂性。Chris 认为神经网络的训练也是一种优化过程，其内部正展现出一个完整的“人工生物学”（artificial biology）世界。他甚至畅想十年后自己走进书店，能够买到一本关于“神经网络生物学”的教科书，并为里面即将揭示的奇妙世界感到兴奋不已。团队成员随即幽默地打趣道，到时候他就能买一本印有自己头像的教科书了。 --- ### [45:38](https://www.youtube.com/watch?v=om2lIWXLLN4&t=2738s) 政府评测能力的提升与 AI 带来的社会变革 Jack Clark 随后分享了让他感到振奋的另一个领域：政府能力的建设。如果几年前预测各国政府会建立起全新且具备实际能力的机构来测试和评估 AI 系统，大多数人都会觉得不可思议。但如今这已成为现实，政府就像是为应对这种新型技术或 Chris 所研究的领域而建立了一座座“新大使馆”。这种国家能力（state capacity）的构建让他感到非常欣慰，这意味着应对这场社会转型将不再仅仅是企业单方面的责任。 Daniela Amodei 则从 AI 改变人类命运的宏大愿景出发，表示看到 Claude 目前在疫苗研发、癌症研究和生物学领域展现出的蛛丝马迹就已经是件“疯狂”的事。放眼未来三到五年，她难以想象 Claude 会在多大程度上帮助人类解决医疗健康等方面的底层基础问题。这让她回想起自己 25 岁从事国际发展工作时的经历，并感叹如果当年能有 Claude 来承担她当时做得不够高效的工作，那将是非常美妙的体验。 - **Claude 在工作与编程中的爆发式应用**：团队成员提到，现在不仅致力于将 Claude 引入全球的企业中，个人在生活中也越来越多地使用 Claude 交流。尤其在编程（coding）方面，六个月前团队自身甚至都不怎么用 Claude 编代码，而如今发生了“相变”（phase difference）级别的巨大改变。在两周前 YC（Y Combinator）的一次演讲中，当询问现场有多少人使用 Claude 编写代码时，全场竟然有 95% 的人举手，这与四个月前的景象完全不可同日而语。 --- ### [48:07](https://www.youtube.com/watch?v=om2lIWXLLN4&t=2887s) 打破共识与 AI 在科学、医学和民主中的未来 Dario Amodei 补充道，最让他兴奋的是那些看似属于“智者共识”、但即将被打破的领域。首先依然是可解释性，它不仅是引导和构建安全 AI 的关键，更包含着关于智能优化问题以及人类大脑工作原理的深刻洞察。Dario 甚至半开玩笑且十分严肃地预测：“Chris Olah 将会是未来的诺贝尔医学奖获得者。”作为前神经科学家，Dario 怀疑精神分裂症或心境障碍等尚未攻克的精神疾病，其背后存在某种更高层面的系统性问题，而由于人类大脑过于黏糊（mushy）且难以解剖交互，研究一直困难重重。虽然神经网络并非完美类比，但随着时间推移，它将成为越来越好的研究模型。 其次，AI 在生物学中的应用正在打破旧有质疑。AlphaFold 斩获诺贝尔化学奖是一个里程碑式的成就，而 Anthropic 应该致力于构建能够帮助人类创造出“100 个 AlphaFold”的工具。最后，Dario 提出了利用 AI 增强民主的设想。人们常常担心 AI 误入歧途会成为威权主义的工具，因此探索如何让 AI 成为促进自由与自决的工具虽然仍处于早期阶段，但其重要性不言而喻。 随后发言者补充道，许多人加入 Anthropic 最初是出于对 AI 的科学好奇，随后被技术进步所说服，从而凝聚成共同的愿景：不仅要推进技术，更要深刻理解技术并确保其安全。随着近期更先进系统的出现，团队开始真正窥见到超强系统可能带来的具体风险，这使得他们能够以可解释性及其他安全机制为武器，展开直观、深入且实证的科学研究。在接下来的六个月里，如何利用对高级系统潜在问题的理解来界定风险并避开陷阱，将是工作的重中之重。随着讨论的圆满结束，创始人们在欢呼和欢笑声中为这场精彩的对话画上了句号。 ## Notable quotes > [02:30](https://www.youtube.com/watch?v=om2lIWXLLN4&t=150s) 因为语言模型本身就要求系统必须理解人类社会中隐含的知识与价值体系。 > [04:10](https://www.youtube.com/watch?v=om2lIWXLLN4&t=250s) 简单的机制在 AI 中往往极其有效 — Jared Kaplan > [07:23](https://www.youtube.com/watch?v=om2lIWXLLN4&t=443s) 许多看似明智、成熟的“共识”，本质上只是伪装成成熟的从众行为 — Dario Amodei > [10:56](https://www.youtube.com/watch?v=om2lIWXLLN4&t=656s) 我们正密切关注你们，AI 将成为一件极其重大的大事。 — Kamala Harris / Gina Raimondo > [14:19](https://www.youtube.com/watch?v=om2lIWXLLN4&t=859s) 如果拒绝给我双倍薪水我就离职 — Sam McCandlish > [15:42](https://www.youtube.com/watch?v=om2lIWXLLN4&t=942s) 我认识的最聪明的人就是 Dario，如果你能招到他，那将是非常幸运的。 — Daniela Amodei > [18:30](https://www.youtube.com/watch?v=om2lIWXLLN4&t=1110s) 修改次数最多的一份文档 > [39:08](https://www.youtube.com/watch?v=om2lIWXLLN4&t=2348s) 我们负责把安全带造出来，其他人随便抄，这是个更好的世界。 > [39:08](https://www.youtube.com/watch?v=om2lIWXLLN4&t=2348s) 因为我们知道它更安全 — B端客户 > [42:35](https://www.youtube.com/watch?v=om2lIWXLLN4&t=2555s) 我们不是毁灭论者（doomers）。 — Anthropic 联合创始人 > [48:07](https://www.youtube.com/watch?v=om2lIWXLLN4&t=2887s) Chris Olah 将会是未来的诺贝尔医学奖获得者。 — Dario Amodei ## Mentioned Dario Amodei, Jared Kaplan, Chris Olah, Jack Clark, Daniela Amodei, Sam McCandlish, Tom Brown, Mike Krieger, Paul Christiano, Greg Brockman, Kamala Harris, Gina Raimondo, Enrico Fermi, Leo Szilard, Edward Teller, Anthropic, OpenAI, Google Brain, Stripe, NVIDIA, Bloomberg, White House, Y Combinator, Claude, GPT-2, GPT-3, AlphaFold, Responsible Scaling Policy (RSP), Constitutional AI (CAI), Scaling Laws, RLHF, Concrete Problems in AI Safety, Machines of Loving Grace, Project Vannevar [Source](https://www.youtube.com/watch?v=om2lIWXLLN4)

[2:30] 因为语言模型本身就要求系统必须理解人类社会中隐含的知识与价值体系。 [4:10] Jared Kaplan: 简单的机制在 AI 中往往极其有效 [7:23] Dario Amodei: 许多看似明智、成熟的“共识”，本质上只是伪装成成熟的从众行为 [10:56] Kamala Harris / Gina Raimondo: 我们正密切关注你们，AI 将成为一件极其重大的大事。 [14:19] Sam McCandlish: 如果拒绝给我双倍薪水我就离职 [15:42] Daniela Amodei: 我认识的最聪明的人就是 Dario，如果你能招到他，那将是非常幸运的。 [18:30] 修改次数最多的一份文档 [39:08] 我们负责把安全带造出来，其他人随便抄，这是个更好的世界。 [39:08] B端客户: 因为我们知道它更安全 [42:35] Anthropic 联合创始人: 我们不是毁灭论者（doomers）。 [48:07] Dario Amodei: Chris Olah 将会是未来的诺贝尔医学奖获得者。

[1:17] Dario Amodei: the bringer of happy tidings. [6:00] Jared Kaplan: Oh, we're just gonna write a constitution for a language model and that'll change all of its behavior. [6:00] Jared Kaplan: simple things just work really, really well in AI. [7:23] Jared Kaplan: psychologically damaged by the AI winter. [9:37] Dario Amodei: herding behavior masquerading as maturity and sophistication. [11:13] Kamala Harris and Gina Raimondo: We've got our eye on you guys... AI's gonna be a really big deal. [15:42] A speaker: the smartest person that I know [15:42] A speaker: a little bit of a mess [21:31] US Senators: Yes, that's a completely normal thing to do. Are you telling me not everyone does this? [25:11] A speaker: evals, evals, evals [35:57] Tom Brown: make less promises and keep more of them. [30:42] Daniela Amodei: Chief clown wrangler. [43:43] Chris Olah: artificial biology [48:07] A speaker: really not joking [48:07] A speaker: a hundred AlphaFolds.

[0:00] - Why are we working on AI in the first place? [0:03] I'm just gonna arbitrarily pick Jared. [0:05] Why are you doing AI at all? [0:07] - I mean, I was working on physics for a long time [0:09] and I got bored, [0:10] and I wanted to hang out with more of my friends, [0:12] so yeah. [0:14] - Yeah, I thought Dario pitched you on it. [0:15] - I don't think I explicitly pitched you at any point. [0:17] I just kind of, like, showed you results of, like, AI models [0:21] and then was trying to make the point that, like, [0:24] they're very general and they don't only apply to one thing [0:27] and then just at some point, [0:28] after I showed you enough of them, you were like, [0:30] "Oh yeah, that seems like it's right." [0:32] - How long had you been a professor before? [0:34] Like, when you started? [0:34] - I think like six years or so. [0:36] I think I helped recruit Sam. [0:37] - I talked to you and you were like, [0:38] "I think I've created a good bubble here-" [0:40] - Yeah. [0:41] - "And like my goal is to get Tom to come back." [0:43] That worked. [0:43] - And did you meet everyone through Google [0:45] when you were doing the interpretability stuff, Chris? [0:48] - No. [0:49] So I guess [0:50] I actually met a bunch of you when I was 19 [0:52] and I was visiting- - Oh yeah? [0:53] - In the Bay Area for the first time. [0:54] So I guess I met Dario and Jared then, [0:57] and I guess they were postdocs, [0:58] which I thought was very cool at the time. [1:00] And then I was working at Google Brain and Dario joined [1:03] and we sat side by side, actually, for a while. [1:05] We had desks beside each other, [1:07] and I worked with Tom there as well. [1:08] And then, of course, [1:09] I got to work with all of you at OpenAI when I went there. [1:13] - Yeah. [1:14] - So I guess I've known a lot of you for, like, [1:15] more than a decade, which is kind of wild. [1:17] - If I remember correctly, [1:18] I met Dario in 2015 when I went to a conference you were at [1:22] and I tried to interview you and Google PR said [1:24] I would've read all of your research papers [1:27] that you needed to- [1:27] - Uh, yeah, I think I was writing [1:29] "Concrete problems in AI safety" [1:31] when I was at Google. [1:32] - I think you wrote a story about that paper. [1:34] - I did. [1:35] - I remember right before I started working with you, [1:38] I think you invited me to the office [1:40] and I got to like come chat [1:41] and just like tell me everything about AI. [1:44] And you like explained, [1:46] I remember afterwards being like, [1:48] "Oh, I guess this stuff is [1:50] much more serious than I realized. [1:51] And you were, like, [1:53] probably explaining the big blob of compute [1:54] and like parameter counting [1:56] and how many neurons are in the brain and everything. [1:58] - And I feel like Dario often has that effect on people; [2:00] "This is much more serious than I realized." [2:02] - Yeah, I'm the bringer of happy tidings. [2:06] - But I remember when we were at OpenAI [2:09] when there was the scaling law stuff [2:11] and just making things bigger [2:12] and it started to feel like it was working, [2:14] and then it kind of kept on eerily working [2:16] on a bunch of different projects, [2:18] which I think is how we all ended up [2:20] working closely together, [2:22] 'cause it was first GPT-2 [2:24] and then scaling laws and GPT-3 and we ended up- [2:26] - Yeah, we were the blob of people [2:28] that were making things work. - Yeah. [2:30] - That's right. [2:31] - I think we're also excited about safety, [2:32] 'cause in that era there was sort of this idea that [2:34] AI would become very powerful [2:37] but, like, potentially not understand human values [2:41] or not even be able to communicate with us. [2:42] And so I think we were all pretty excited about [2:44] language models as a way to kind of guarantee [2:48] that AI systems would have to understand [2:50] kind of implicit knowledge that- [2:52] - And RL from human feedback on top of language models, [2:56] which was the whole reason for scaling these models up [2:59] was that, you know, the models weren't smart enough [3:03] to do RLHF on top of, [3:06] so that's the kind of intertwinement of [3:08] safety and scaling of the models that we, you know, [3:10] still believe in today. [3:11] - Yeah, I think there was also an element of like [3:14] the scaling work was done as part of the safety team [3:17] that, you know- - Mm-hmm. [3:18] - Dario started at OpenAI, [3:19] because we thought that forecasting AI trends [3:22] was important to be able to have us taken seriously [3:25] and take safety seriously as a problem. [3:27] - Correct. [3:28] - Yeah, and we took, [3:29] I remember being in some airport in England [3:31] sampling from GPT-2 and using it to write fake news articles [3:35] and Slacking Dario and being like, [3:37] "Oh, this stuff actually works. [3:38] It might have, like, [3:39] huge policy implications." [3:41] I think Dario said something like, "Yes." [3:46] His typical way. [3:48] But then we worked on that a bunch [3:49] as well as the release stuff, which was kind of wild. [3:51] - Yeah, I remember the release stuff. [3:53] I think that was when we first started working together. [3:54] - Yeah. - That was a fun time. [3:56] - Yes. - The GPT-2 launch. [3:57] - Yeah. [3:58] But I think it was good for us 'cause we did a kind of [4:00] slightly strange, safety-oriented thing altogether, [4:03] and then we ended up doing Anthropic, [4:05] which is a much larger, slightly strange, [4:08] safety-oriented thing- [4:09] - That's right. - Together. [4:10] - So I guess just like going back to the concrete problems, [4:13] 'cause I remember, [4:14] so I joined OpenAI in like 2016, [4:17] like one of the first 20 employees or whatever [4:19] with you, Dario, [4:20] and I remember at that time [4:21] the concrete problems in AI safety [4:23] seemed like it was like the first mainstream AI safety- [4:26] - Yes. - Paper, I guess? [4:27] I don't really know if I ever asked you what the story was [4:29] for how that came about. [4:31] - Chris knows the story, 'cause he was involved in it. [4:34] I think, you know, we were both at Google. [4:36] I forget what other project I was working on, [4:38] but like with many things, [4:40] it was my attempt to kind of procrastinate from [4:43] whatever other project I was working on [4:44] that I've now [4:45] completely forgotten what it was. [4:47] But I think it was like Chris and I decided to write down [4:52] what are some open problems in terms of AI safety, [4:55] and also AI safety you usually talked about [4:57] in this very kind of abstruse, abstract way. [5:01] Can we kind of ground it in the ML [5:03] that was going on at the time? [5:04] I mean, now there's been, like, you know, six, seven years [5:08] of work in that vein, [5:10] but it was almost a strange idea at the time. [5:12] - Yeah, I think there's a way in which it was almost a [5:14] kind of political project, where at the time, [5:18] a lot of people didn't take safety seriously. [5:20] So I think that there was sort of this goal [5:21] to collate a list of problems [5:23] that sort of people agreed were reasonable, [5:26] often already existed in literature, [5:28] and then get a bunch of people across different institutions [5:33] who are credible to be authors. [5:34] And like, I remember I had this whole long period [5:37] where I just talked to 20 different researchers at Brain [5:39] to build support for publishing the paper. [5:42] Like in some ways, [5:42] if you look at it in terms of the problems [5:44] and a lot of things that emphasized, [5:45] I think it, you know, hasn't held up that well [5:47] in that it's, you know, [5:48] I think it's not really the right problems, [5:49] but I think if you sort of see it instead as [5:52] a consensus-building exercise, [5:53] that there's something here that is real [5:56] and that is worth taking seriously, [5:58] then it was a pretty important moment. [6:00] - I mean, you end up in this really weird sci-fi world [6:02] where I remember at the start of Anthropic, [6:04] we were talking about constitutional AI [6:06] and I think Jared said, [6:07] "Oh, we're just gonna write a constitution [6:09] for a language model [6:10] and that'll change all of its behavior." [6:12] And I remember that sounded incredibly crazy [6:15] at the time, [6:16] but why did you guys think that was gonna work? [6:18] Because I remember that was one of the first early, like, [6:21] big research ideas we had at the company. [6:23] - Yeah, I mean, [6:24] I think Dario and I had talked about it for a while. [6:25] I guess I think simple things [6:27] just work really, really well in AI. [6:29] And so like, I think the first versions of that [6:32] were quite complicated, [6:33] but then we kind of like whittled away into, like, [6:36] just use the fact that AI systems are good at [6:38] solving multiple choice exams [6:40] and give them a prompt that tells them [6:42] what they're looking for and that was kind of a lot of [6:46] what we needed. [6:47] - And then we were able to just write down these principles. [6:50] - I mean, it goes back to, like, the big blob of compute [6:52] or the bitter lesson or the scaling hypothesis. [6:55] If you can identify, you know, [6:57] something that you can give the AI data for [7:00] and that's kind of a clear target, [7:01] you'll get it to do it. [7:02] So like here's this set of instructions, [7:05] here's this set of principles. [7:07] AI language models can like read that set of principles [7:10] and they can compare it to the behavior [7:12] that they themselves are engaging in. [7:14] And so like you've got your training target there, [7:16] so once you know that, [7:17] I think my view and Jared's view is [7:19] there's a way to get it to work; [7:20] you just have to fiddle with enough of the details. [7:23] - Yeah. [7:23] I think it was always weird for me, [7:25] especially in these early eras, [7:26] 'cause like I was in physics and then coming from physics, [7:30] and I think now we forget about this [7:32] 'cause everyone's excited about AI, [7:34] but like, I remember talking to Dario about [7:37] concrete problems and other things, [7:38] and I just got the sense that AI researchers [7:41] were very, very kind of psychologically damaged [7:43] by the AI winter [7:45] where they just kind of felt like [7:47] having really ambitious ideas or ambitious visions [7:50] was, like, very disallowed. [7:52] And that's kind of how I imagine it was [7:53] in terms of talking about safety. [7:55] In order to care about safety, [7:56] you have to believe that AI systems could actually be [7:58] really powerful and really useful, [8:01] and I think that there was kind of a prohibition [8:02] against being ambitious. [8:04] - And I think [8:05] one of the benefits is that physicists are very arrogant [8:07] and so they're constantly [8:09] doing really ambitious things [8:10] and talking about things in terms of grand schemes, [8:12] and so- [8:14] - Yeah. [8:15] - I mean, I think that's definitely true. [8:17] Like I remember in 2014, [8:18] it was like there were just like, I don't know, [8:21] there were just some things you couldn't say, right? [8:23] But I actually think it was kind of an extension of problems [8:26] that exist across academia, [8:28] other than maybe theoretical physics, [8:30] where they've kind of evolved into [8:32] very risk-averse institutions for a number of reasons. [8:35] And even the industrial parts of AI [8:37] had kind of transplanted or fork lifted that mentality. [8:40] And it took a long time. [8:41] I think it took it until like 2022 [8:43] to get out of that mentality. [8:45] - There's a weird thing about like, [8:46] what does it mean to be conservative and respectful, [8:49] where you might think like, [8:51] one version you could have is that [8:53] what it means to be conservative is to take the risks [8:55] or the potential harms of what you're doing [8:57] really seriously and worry about that. [8:59] But another kind of conservatism is to be like, [9:01] "Ah, you know, taking an idea too seriously [9:04] and believing that it might succeed [9:05] is sort of like scientific arrogance." [9:07] And so I think there's like kind of two different kinds of [9:11] conservatism or caution, [9:12] and I think we were sort of in a regime [9:14] that was very controlled by that one. [9:15] I mean, you see it historically, right? [9:17] Like if you look at the early discussions in 1939 [9:20] between, you know, people involved in nuclear physics [9:22] about what the nuclear bombs were, [9:24] sort of a serious concern. [9:25] You see exactly the same thing [9:27] with Fermi resisting these ideas, [9:30] because it just seemed kind of like a crazy thing. [9:32] And other people, like Szilard or Teller, [9:34] taking the ideas seriously [9:36] because they were worried about the risks. [9:37] - Yeah. [9:38] Perhaps the deepest lesson [9:40] that I've learned in the last 10 years, [9:42] and probably, you know, [9:43] all of you have learned some form of it as well, [9:45] is there can be this kind of seeming consensus, [9:48] these things that kind of everyone knows, [9:51] that, I don't know, [9:52] seem sort of wise, seem like they're common sense, [9:55] but really, they're just kind of herding behavior [9:59] masquerading as maturity and sophistication. [10:02] - Mm-hmm. [10:02] - And when you've seen the consensus can change overnight- [10:06] And when you've seen it happen a number of times, [10:08] you suspected but you didn't really bet on it [10:11] and you're like, "Oh man, I kind of thought this, [10:13] but what do I know?" [10:14] You know, "How how can I be right [10:16] and all these people are wrong?" [10:17] You see that a few times, then you just start saying, [10:19] "Nope, this is the bet we're gonna make. [10:21] I don't know for sure if we're right, [10:22] but like, just ignore all this other stuff, see it happen, [10:26] and, I don't know, even if you're right 50% of the time [10:29] being right 50% of the time contributes so much," right? [10:32] You're adding so much [10:33] that is not being added by anyone else. [10:35] - Yeah. [10:36] - And it feels like that's where we are today [10:37] with some safety stuff, [10:39] where there's like a consensus view that [10:40] a lot of this safety stuff is unusual [10:42] or doesn't naturally fall out of the technology. [10:45] And then at Anthropic, we do all of this research [10:47] where weird safety misalignment problems [10:49] fall out as a natural dividend of the tech we're building, [10:53] so it feels like we're in that [10:54] counter-consensus view right now. [10:56] - But I feel like that has been shifting over the past, [10:58] even just like 18- [11:00] - We've been helping to shift- [11:00] - We've definitely been helping. [11:01] - No, I mean- - Yeah. [11:02] - By publishing and doing research. [11:03] - Constant publishing. - This constant force. [11:04] Yeah. - But I also think, just, [11:06] world sentiment around AI has shifted really dramatically, [11:09] and, you know, [11:10] it's more common in the user research that we do [11:12] to hear just customers, regular people say, [11:15] "I'm really worried about what the impact of AI on the world [11:18] more broadly is going to be." [11:19] And sometimes that means, you know, [11:21] jobs or bias or toxicity, but it also sometimes means like, [11:24] is this just gonna mess up the world, right? [11:26] How is this gonna contribute to fundamentally changing [11:29] how humans work together, operate? [11:31] Which is, I wouldn't have predicted that actually, [11:33] you know? - Oh yeah. [11:34] But yeah, for whatever reason, [11:35] it seems like people in the ML research sphere [11:37] have always been more pessimistic about [11:40] AI becoming very powerful- [11:41] - Yeah. - Than the general public. [11:43] - Maybe it's a weird- [11:44] - General public's just like- [11:44] - Humility or something, yeah. [11:45] - When Dario and I went to the White House in 2023, [11:48] in that meeting, like Harris and Raimondo and stuff [11:50] basically said, I'll paraphrase, [11:53] but basically said like, "We've got our eye on you guys. [11:55] Like, AI's gonna be a really big deal [11:57] and we're now actually paying attention," [11:58] which is- - And they're right. [11:59] - They're right. [12:01] - They're absolutely right. [12:02] Absolutely right. - But I think in 2018 [12:03] you wouldn't have been like, [12:04] "The President will call you to the White House [12:06] to tell you [12:07] they're paying close attention to [12:08] the development of language models." [12:10] - Yeah. [12:11] - Which is like a crazy place- [12:11] - That was not on the bingo card. [12:13] - That was like 2018. [12:15] - One thing that I think is interesting, too, [12:17] just is, I guess, like all of us kind of got into this [12:22] when it didn't seem like there was- [12:24] - Mm-hmm. [12:25] - Like we thought that it could happen, but yeah, [12:27] it was like Fermi being skeptical of the atomic bomb. [12:30] It was like he was just a good scientist [12:33] and there was some evidence that it could happen, [12:35] but there also was a lot of evidence against it happening. [12:38] - Mm-hmm. [12:39] - And he, I guess, decided that it would be worthwhile [12:41] because if it was true, then it would be a big deal. [12:44] And I think for all of us, it was like, yeah, [12:45] like 2015, 2016, 2017, there was some evidence [12:49] and increasing evidence that this might be a big deal, [12:51] but I remember in 2016, like talking to all my advisors- [12:55] - Yeah, yeah. [12:55] - And I was like, "I've done startup stuff. [12:57] I wanna help out with AI safety, [12:59] but I'm not great at math. [13:01] I don't exactly know [13:03] how I can do it." [13:04] And I think at the time people either were like, [13:07] "Well, you need to be super good at decision theory [13:09] in order to help out." [13:10] And I was like, "Eh, that's probably not gonna work." [13:12] Or they were like, [13:14] "It doesn't really seem like [13:14] we're gonna get some crazy AI thing," [13:17] and so I had only a few people, basically, [13:20] that were like, [13:21] "Yeah, okay, that seems like a good thing to do." [13:24] - I remember in 2014 [13:25] making graphs of ImageNet results over time [13:27] when I was a journalist [13:28] and trying to get stories published about [13:31] and people thought I was completely mad. [13:32] And then I remember in 2015 [13:34] trying to persuade Bloomberg [13:35] to let me write a story about NVIDIA, [13:37] because every AI research paper had started mentioning [13:40] the use of GPUs, [13:41] and they said that was completely mad. [13:43] And then in 2016 when I left journalism to go into AI, [13:46] I have these emails saying, [13:48] "You're making the worst mistake of your life," [13:51] which I now occasionally look back on, [13:55] but it was like it all seemed crazy at the time, [13:58] from many perspectives, [13:59] to go and take this seriously, that scaling was gonna work, [14:02] and something was maybe different about [14:04] the technology paradigm. [14:06] - You're like Michael Jordan and that coach [14:07] that didn't believe in him in high school. [14:09] - How did you actually make the decision, though? [14:11] Was it did you feel torn or was it obvious to you? [14:14] - I did a crazy counter-bet where I said, [14:16] "Let me become your full-time AI reporter [14:19] and double my salary," [14:20] which I knew that they wouldn't say yes to. [14:22] And then I went to sleep and then I woke up and resigned. [14:25] It was all fairly relaxed. [14:27] - You're just a decisive guy. [14:29] - In that instance, I was. [14:31] I think it's because I was like going to work, [14:33] reading archive papers, [14:35] and then printing archive papers off [14:36] and coming home and reading archive papers, [14:39] including Dario's papers from the Baidu stuff [14:42] and being like, [14:43] "Something completely crazy is happening here." [14:46] And at some point I thought you should bet with conviction, [14:48] which I think everyone here has done in their careers, [14:50] is just betting with conviction that this is gonna work. [14:53] - Yeah. [14:54] - I definitely was not as decisive of you. [14:56] I spent, like, six months [14:58] flip flopping, like, "Okay, should I, actually? [15:01] Should I do it? [15:02] Should I try to do a startup? [15:02] Should I try to do this thing?" [15:04] - But I also feel like back then, [15:05] there wasn't as much talk of engineers [15:08] and the impact that an engineer can have [15:10] on AI, right? - Yeah, yeah, no way. [15:10] - That feels so natural to us now. [15:12] And we're at the same sort of talent raise [15:14] for engineers of all different types, [15:17] but at the time, it was like, you're a researcher, [15:19] and that's the only people [15:20] that can work on AI. - Yeah. [15:21] - So I don't think it was crazy [15:22] that you were spending time thinking about that. [15:24] - Yeah. [15:25] Yeah, and I think that that was basically the thing [15:27] that got me to join OpenAI, [15:28] was like I messaged the people there and they were like, [15:31] "Yeah, we actually think that you can help out-" [15:33] - Yeah. - "By doing engineering work." [15:34] - Yeah. [15:35] - "And that you can help out with AI safety in that way." [15:37] - Mm-hmm. - Which I think [15:38] there hadn't really been an opportunity for that, [15:40] so that was what- [15:41] - That's right. - Brought me there. [15:42] - You were my manager at OpenAI. [15:44] - I was, that's right. [15:44] - I think I joined after you'd been there for a while. [15:47] - A little bit. [15:48] - 'Cause I was at Brain for a bit. [15:49] - Yeah. [15:50] - I don't know if I ever asked you [15:50] what it was that got you to join? [15:53] - Yeah, so I had been at Stripe [15:55] for about five and a half years [15:57] and I knew Greg; he had been my boss. [15:59] He was my boss at Stripe for a while, [16:01] and I actually introduced him and Dario, [16:04] because I said, when he was starting OpenAI, [16:06] I was like, "The smartest person that I know is Dario. [16:10] You would be really lucky to get him." [16:12] So Dario was at OpenAI, [16:13] I had a few friends from Stripe that had gone there, too. [16:16] And I think sort of like you, [16:18] I'd been thinking about what I wanted to do after Stripe. [16:22] I had gone there just 'cause I wanted to get more skills [16:24] after working in, you know, [16:26] nonprofit and international development, [16:28] and I actually thought I was gonna go back to doing that. [16:31] Like, essentially, I had always been working. [16:33] I was like, "I really wanna help people that have, you know, [16:36] less than I do," [16:37] but I didn't have the skills [16:38] when I was doing it before Stripe. [16:39] - Yep. [16:40] - And so I looked at going back to public health. [16:42] I thought about going back into politics very briefly, [16:44] but I was also looking around at other tech companies [16:47] and other sort of ways of having impact, [16:50] and OpenAI, at the time, [16:51] felt like it was a really nice intersection. [16:52] It was a nonprofit, [16:54] they were working on this really big, lofty mission. [16:56] I really believed in sort of the AI, you know, potential, [16:59] because, I mean, I know Dario a little bit, [17:01] and so he was- [17:02] - And they needed management help. [17:04] - The definitely needed help. [17:06] That is a fact. [17:07] And so I think that it felt very me-shaped, right? [17:10] I was like, "Oh, there's this nonprofit [17:12] and there's all these really great people [17:14] with these really good intentions, [17:16] but it seems like they're a little bit of a mess." [17:18] - Yeah. [17:19] And that felt really exciting to me, [17:21] to get to come in and even, you know, [17:23] just I was such a utility player, right? [17:25] I was running people, [17:27] but I was also running some of the technical teams- [17:29] - Scaling orgs, yep. [17:29] - Yeah, the scaling org, [17:30] I worked on the language team, [17:31] I took over- [17:32] - You worked on policy- [17:33] - I worked on some policy stuff, [17:34] I worked with Chris, [17:35] and I felt like there was just so much goodness [17:38] in so many of the employees there, [17:40] and I felt a very strong desire to come [17:42] and sort of just try to help make the company [17:44] a little more functional. [17:46] - I remember towards the end, after we'd done GPT-3, [17:49] you were like, [17:50] "Have you guys heard of something called trust and safety?" [17:51] - Yes, I remember that! [17:55] That did happen. - Yeah. [17:57] - Yeah. - I said, you know, [17:58] "I used to run some trust and safety teams at Stripe. [18:00] There's a thing called trust and safety [18:02] that you might wanna consider for a technology like this." [18:05] And it's funny because it's sort of is the intermediary step [18:08] between, you know, AI safety research, right, [18:11] which is how do you actually make the model safe [18:13] to something just much more practical. [18:15] I do think there was a value in saying, you know, [18:18] this is gonna be a big thing; [18:19] we also have to be doing this sort of practical work [18:21] day to day to build the muscles [18:23] for when things are gonna be a lot higher stakes. [18:26] - That might be a good transition point [18:28] to talk about things like the responsible scaling policy [18:31] and how we came up with that, [18:33] or why we came up with it and how we're using it now, [18:35] especially given how much trust and safety work [18:37] we do on today's models. [18:39] So whose idea was the RSP? [18:41] Was you and Paul? - So, yeah, [18:43] it was me and Paul first talked about it in late, [18:47] Paul Christiano, in late 2022. [18:49] - Mm-hmm. [18:50] - First it was like, oh, should we cap scaling at a, [18:52] you know, particular point [18:54] until we've discovered how to solve certain safety problems? [18:57] And then it was like, well, you know, [18:59] it's kind of strange to have this one place [19:01] where you cap it and then you uncap it, [19:02] so let's have a bunch of thresholds, [19:05] and then at each threshold you have to do certain tests [19:07] to see if the model is capable [19:09] and you have to take [19:10] increasing safety and security measures. [19:12] But originally we had this idea, [19:14] and then the thought was just look, you know, [19:17] this'll go better if it's done by some third party. [19:20] Like, we shouldn't be the ones to do it, right? [19:23] It shouldn't come from one company, [19:24] 'cause then other companies are less likely to adopt it. [19:26] So Paul actually went off and designed it, [19:28] and, you know, many, many features of it changed [19:31] and we were kind of, on our side, [19:33] working on how it should work. [19:36] And, you know, once Paul had something together, [19:39] then pretty much immediately after he announced the concept, [19:42] we announced ours within a month or two. [19:44] I mean, many of us were heavily involved in it. [19:46] I remember writing at least one draft of it myself, [19:49] but there were several drafts of it. [19:50] - There were so many drafts. [19:54] - I think it's gone through the most drafts of any doc. [19:56] - Yeah. [19:56] - Which makes sense, right? [19:57] It's like, I feel like it is [19:59] in the same way that the US treats like the Constitution, [20:03] as like the holy document. [20:04] - Yeah. - Which like, [20:05] I think is just a big thing that like strengthens the US. [20:07] - Yes. [20:08] - And like we don't expect the US to go off the rails, [20:10] in part, because just like every single person in the US [20:12] is like, "The Constitution is a big deal, [20:14] and if you tread on that-" - Yeah. [20:15] - "Like, I'm mad." [20:16] - Yeah, yeah. [20:17] - Like I think that the RSP is our, [20:19] like, it holds that thing. [20:21] It's like the holy document for Anthropic. [20:22] So it's worth doing a lot of iterations getting it right. [20:26] - Some of what I think has been so cool to watch [20:27] about the RSP development at Anthropic, too, [20:31] is it feels like it has gone through [20:32] so many different phases [20:33] and there's so many different skills [20:34] that are needed to make it work, right? [20:36] There's like the big ideas, [20:37] which I feel like Dario and Paul and Sam and Jared [20:40] and so many others are like, "What are the principles? [20:43] Like what are we trying to say? [20:45] How do we know if we're right?" [20:46] But there's also this very operational approach [20:49] to just iterating [20:50] where we're like, "Well, we thought that [20:51] we were gonna see this at this, you know, safety level, [20:54] and we didn't, [20:55] so should we change it so that we're making sure that [20:57] we're holding ourselves accountable?" [20:58] And then there's all kinds of organizational things, right? [21:01] We just were like, [21:01] "Let's change the structure of the RSP organization [21:03] for clearer accountability." [21:05] And I think my sense is that for a document [21:08] that's as important as this, right, [21:09] I love the Constitution analogy, [21:11] it's like there's all of these bodies and systems [21:13] that exist in the US [21:15] to make sure that we follow the Constitution, right? [21:17] There's the courts, there's the Supreme Court, [21:18] there's the presidency, [21:20] there's, you know, both houses of Congress [21:23] and they do all kinds of other things, of course, [21:26] but there's like all of this infrastructure that you need [21:28] around this one document, [21:29] and I feel like we're also learning that lesson here. [21:31] - I think it sort of reflects a view [21:33] a lot of us have about safety, [21:34] which is that it's a solvable problem. [21:37] - Mm-hmm. [21:37] - It's just a very, very hard problem [21:39] that's gonna take tons and tons of work. [21:40] - Mm-hmm. - Yeah. [21:41] - All of these institutions that we need to build up, [21:43] like there's all sorts of institutions built up around [21:46] like automotive safety, built up over many, many years. [21:49] But we're like, "Do we have the time to do that? [21:51] We've gotta go as fast as we can to figure out [21:54] what the institutions we need for AI safety are, [21:56] and build those and try to build them here first, [21:58] but make it exportable." [22:00] - That's right. [22:00] - It it forces unity also, [22:01] because if any part of the org [22:04] is not kind of in line with our safety values, [22:08] it shows up through kind of the RSP, right? [22:10] The RSP is gonna block them from doing what they want to do, [22:13] and so it's a way to remind everyone over and over again, [22:16] basically, to make safety a product requirement, [22:20] part of the product planning process. [22:22] And so, like, it's not just a bunch of kind of like [22:24] bromides that we repeat; [22:26] it's something that you actually, [22:27] if you show up here and you're not aligned, [22:30] you actually run into it. [22:31] - Yeah. [22:32] - And you either have to learn to get with the program [22:34] or it doesn't work out. - Yeah. [22:37] - The RSPs become kind of funny over time [22:39] because we spend thousands of hours of work on it, [22:42] and then I go and talk to senators and I explain the RSP, [22:44] and I'm like, "We have some stuff [22:46] that means it's hard to steal what we make, [22:49] and also that it's safe." [22:50] And they're like, [22:51] "Yes, that's a completely normal thing to do. [22:55] Are you telling me not everyone does this?" [22:57] You're like, "Oh, okay, yeah." [22:58] Yeah. [23:01] - It's half true that that not everyone does this. [23:03] - Yeah. [23:03] - But that kind of, [23:04] it's amazing because we've spent so much effort on it here [23:07] and when you boil it down, they're like, [23:08] "Yes, that sounds like a normal way [23:10] to do that." - Yeah, that sounds good. [23:12] - That's been the goal. [23:13] Like Daniela was saying, [23:14] "Let's make this as boring and normal. [23:16] Like let's make this a finance thing." [23:17] - Yeah, imagine it's like an audit. [23:19] - Yeah, yeah. - Right? [23:20] Yeah. - No, boring and normal [23:22] is what we want, [23:23] certainly in retrospect. - Yeah. [23:25] Well also, Dario, [23:26] I think in addition to driving alignment, [23:27] it also drives clarity- - Mm-hmm. [23:29] - Because it's written down what we're trying to do [23:32] and it's legible to everyone in the company, [23:34] and it's legible externally [23:36] what we think we're supposed to be aiming towards [23:38] from a safety perspective. [23:39] It's not perfect. [23:40] We're iterating on it, we're making it better, [23:42] but I think there's some value in saying like, [23:44] "This is what we're worried about, this thing over here." [23:46] Like you can't just use this word to sort of [23:49] derail something in either direction, right? [23:51] To say, "Oh, because of safety, we can't do X, [23:53] or because of safety, we have to do X." [23:55] We're really trying to make it clearer what we mean. [23:57] - Yeah, it prevents you from worrying about [24:00] every last little thing under the sun. [24:02] - That's right. [24:03] - Because it's actually the fire drills [24:05] that damage the cause of safety in the long run. [24:07] - Right. [24:08] - I've said like, "If there's a building, [24:08] and, you know, the fire alarm goes off every week, [24:12] like, that's a really unsafe building." [24:13] - Mm-hmm. [24:14] - 'Cause when there's like actually a fire, [24:15] you're just gonna be like- - No ones gonna care. [24:16] - "Oh, it just goes off all the time." [24:17] So- - Yeah. [24:18] - It's very important to be calibrated. [24:20] - Yeah. - That's right. [24:21] - Yeah. [24:22] A slightly different frame that I find kind of clarifying [24:24] is that I think that RSP creates healthy incentives [24:27] at a lot of levels. - Mm. [24:28] - So I think internally it aligns [24:30] the incentives of every team with safety [24:32] because it means that if we don't make progress on safety, [24:34] we're gonna block. [24:36] I also think that externally [24:38] it creates a lot of healthier incentives [24:40] than other possibilities, at least that I see, [24:42] because it means that, you know, if we at some point [24:44] have to take some kind of dramatic action, [24:45] like, if at some point we have to say, [24:46] "You know, our model, we've reached some point [24:48] and we can't yet make a model safe," [24:50] it aligns that with sort of the point where [24:52] there's evidence that supports that decision [24:54] and there's sort of a preexisting framework [24:57] for thinking about it, and it's legible. [24:58] And so I think there's a lot of levels at which the RSP, [25:01] I think in ways that maybe I didn't initially understand [25:04] when we were talking about the early versions of it, [25:07] it creates a better framework [25:08] than any of the other ones that I've thought about. [25:11] - I think this is all true, [25:12] but I feel like it undersells sort of like [25:15] how challenging it's been to sort of figure out [25:18] what the right policies and evaluations [25:21] and what the lines should be. [25:24] I think that we have and continue to [25:26] sort of iterate a lot on that, [25:27] and I think there is a question also that's difficult [25:30] of sort of you could be at a point where [25:32] it's very clear something's dangerous [25:34] or very clear that something's safe, [25:35] but with some technology that's so new, [25:37] there's actually like a big gray area. [25:39] And so I think that has been, like, [25:41] all the things that we're saying [25:43] were things that made me [25:45] really, really excited about the RSP at the beginning, [25:47] and still do, [25:48] but also I think enacting this [25:51] in a clear way and making it work [25:54] has been much harder and more complicated [25:56] than I anticipated. [25:58] - Yeah, I think this is exactly the point. [25:59] Like- - Yeah. [26:00] - Like the gray areas are impossible to predict. [26:02] There's so many of them. [26:03] Like, until you actually try to implement everything, [26:05] you don't know what's going to go wrong. [26:07] So what we're trying to do is go and implement everything [26:09] so we can see as early as possible what's going to go wrong, [26:12] so- - Yeah, you have to- [26:13] - The gray areas are- [26:14] - You have to do three or four passes before- [26:15] - Yeah. - Yeah. [26:16] - Before you really, really get it right. [26:17] Like, iteration is just very powerful and, you know, [26:20] you're not gonna get it right on the first time. [26:23] And so, you know, if the stakes are increasing, [26:25] you want to get your iterations in early; [26:27] you don't want to get them in late. [26:28] - You're also building [26:29] the internal institutions and processes, [26:32] so the specifics might change a lot, [26:34] but building the muscle of just doing it [26:36] is the really valuable thing. [26:37] - I'm responsible for, like, compute at Anthropic, and so- [26:41] - That's important. [26:42] - So, thank you. [26:43] I think so. [26:44] So like for me, [26:48] like I guess we have to deal with external folks- [26:51] - Yeah. [26:51] - And different external folks [26:53] kind of are on different spectrums of the, like, [26:55] how fast do they think stuff is gonna get? [26:56] - Mm-hmm. - And like, [26:57] I think that's also been a thing, [26:58] where I started out like not thinking stuff [27:00] would be that fast and have changed over time. [27:02] And so, I have sympathy for that. [27:04] And so I think the RSP has been extremely useful for me [27:07] in communicating with people [27:08] who think that things might take longer [27:10] because then we have a thing where it's like, [27:12] we don't need to do extreme safety measures [27:15] until stuff gets really intense, [27:18] and then they might be like, [27:21] "I don't think stuff will get intense for a long time." [27:22] And then I'll be like, [27:23] "Okay, yeah, we don't have to do extreme safety measures." [27:25] And so that makes it a lot easier [27:27] to communicate with other folks externally. [27:29] - Yeah, yeah, it makes it like [27:30] a normal thing you can talk about, [27:33] rather than something really strange. [27:34] - Yeah. - Yeah. [27:35] How else is it showing up for people? [27:37] You are- [27:38] - Evals, evals, evals. [27:40] - Good. [27:41] - It's all about evals. [27:42] Everyone's doing evals. [27:43] Like, your training team is doing evals all the time. [27:45] We're trying to figure out, like, [27:47] has this model gotten enough better [27:48] that it has the potential to be dangerous? [27:51] So how many teams do we have that are evals teams? [27:54] You have Frontier Red Team. [27:56] There must be, I mean there's a lot of people- [27:58] - Every team produces evals, basically. [28:00] - And that means you're just measuring against the RSP, [28:03] like measuring for certain signs of [28:05] things that would concern you or not concern you. [28:07] - Exactly. [28:08] Like it's easy to lower bound the abilities of a model, [28:14] but it's hard to upper bound, [28:15] so we just put tons and tons of research effort [28:17] into saying, like, [28:19] "Can this model do this dangerous thing or not? [28:21] Maybe there's some trick that we haven't thought of, [28:23] like chain of thought or best event [28:25] or some kind of tool use that's gonna make it [28:27] so it can help you do something very dangerous." [28:30] - It's been really useful in policy, [28:31] because it's been a really abstract concept, [28:34] what safety is, [28:35] and when I'm like, "We have an eval [28:37] which changes whether we deploy the model or not," [28:39] and then you can go and calibrate with [28:40] policymakers or experts in national security [28:43] or some of these CBRN areas that we do, [28:45] to actually help us build evals that are well-calibrated [28:48] and that, counter-factually, [28:50] just wouldn't have happened otherwise, [28:52] but once you've got the specific thing, [28:54] people are a lot more motivated [28:55] to help you make it accurate, [28:56] so it's been useful for that. [28:58] How has it shown up for- - The RSP shows up for me, [28:59] for sure. [29:01] Often. [29:03] I actually think, weirdly, [29:04] the way that I think about the RSP the most is like [29:08] what it sounds like- [29:09] - Mm-hmm? [29:10] - Just like the tone. [29:10] I think we just did a big rewrite of the tone of the RSP [29:13] because it felt overly technocratic, [29:15] and even a little bit adversarial. [29:17] I spent a lot of time thinking about like, [29:18] how do you build a system [29:19] that people just wanna be a part of? [29:21] - Mm-hmm. - Right? [29:21] It's so much better if the RSP is something [29:23] that everyone in the company can walk around and tell you, [29:26] you know, just like with OKRs [29:28] like we do right now- - Yeah. [29:29] - Like, what are the top goals of the RSP? [29:32] How do we know if we're meeting them? [29:32] What AI safety level are we at right now? [29:35] Are we at ASL-2? [29:35] Are we at ASL-3? [29:37] That people know what to look for [29:38] because that is how you're going to have [29:40] good, common knowledge of if something's going wrong, [29:42] right? [29:43] If it's overly technocratic, [29:44] and it's something that [29:45] only particular people in the company [29:47] feel is accessible to them, [29:48] it's just like not as productive, right? [29:50] And I think it's been really cool [29:52] to watch it sort of transition into this document [29:54] where I actually think most, [29:56] if not everybody at the company, regardless of their role, [29:58] could read it and say, "This feels really reasonable. [30:00] I wanna make sure that we're building AI [30:03] in the following ways, [30:04] and I see why I would be worried about these things, [30:07] and I also kind of know what to look for [30:08] if I bump into something," right? [30:10] It's almost like make it simple enough [30:11] that if you are working at a manufacturing plant [30:15] and you're like, [30:15] "Huh, it looks like the safety seatbelt on this [30:17] should connect this way, but it doesn't connect," [30:19] that you can spot it. [30:20] - Mm-hmm. [30:21] - And that there's just like healthy feedback flow [30:23] between leadership and the board [30:25] and the rest of the company [30:26] and the people that are actually building it, [30:28] because I actually think the way this stuff goes wrong [30:30] in most cases is just, like, the wires don't connect [30:34] or like they get crossed, [30:35] and that would just be like [30:37] a really sad way for things to go wrong, right? [30:39] It's just all about operationalizing it, [30:41] making it easy for people to understand. [30:42] - Yeah, the thing I would say is [30:43] none of us wanted to found a company. [30:45] We just felt like it was our duty, right? [30:49] - It felt like we had to. [30:51] - Like, we have to do this thing. [30:52] This is the way we're gonna make things go better with AI. [30:55] Like that's also why we did the pledge, right? [30:57] - Yeah. - Because we're like [30:59] the reason we're doing this is it feels like our duty. [31:01] - I wanted to invent and discover things [31:05] in some kind of beneficial way. [31:06] That was how I came to it, and that led to working on AI, [31:10] and AI required a lot of engineering [31:12] and eventually AI required a lot of capital, [31:15] but what I found was that [31:19] if you don't do this in a way where [31:22] you're setting the environment, [31:23] where you set up the company, [31:26] then a lot of it gets done, [31:27] a lot of it repeats the same mistakes [31:30] that I found so alienating about the tech community. [31:35] It's the same people, [31:36] it's the same attitude, [31:38] it's the same pattern-matching, [31:39] And so at some point it just seemed inevitable [31:42] that we need to do it in a different way. [31:44] - When we were hanging out in graduate school, [31:45] I remember you had kind of this whole program [31:48] of trying to figure out how to do science [31:50] in a way that would sort of advance the public good. [31:52] And I think that's pretty similar to [31:54] how we we think about this. [31:57] I think you had this like Project Vannevar or something, [31:59] to do that. [32:01] I was a professor. [32:02] I think basically I just looked at the situation [32:04] and I was convinced that AI [32:06] was on a very, very, very steep trajectory [32:09] in terms of impact, [32:10] didn't seem like because of the necessity for capital, [32:14] and as a physics professor, I could continue doing that [32:16] and I kind of wanted to work with people that I trusted [32:20] in building an institution to try to make AI go well. [32:24] But yeah, I would never recommend founding a company. [32:29] Or really want to do it. [32:30] I mean, yeah, I think it's just a means to an end. [32:32] I mean, [32:33] I think that's like usually how things go well, though. [32:35] If you're doing something just to sort of like [32:37] enrich yourself or gain power or, like, [32:39] you have to sort of actually care about [32:41] accomplishing a real goal in the world [32:43] and then you find whatever means you have to. [32:46] - Well, something I think about a lot [32:47] as just a strategic advantage for us is, [32:50] I mean, it sounds really funny to say, [32:52] but just like how much trust there is at this table, [32:55] right? - Mm-hmm. [32:56] - Like I think that's not, I mean, [32:57] Tom, you were at other startups. [32:58] I was never a founder before, [33:00] but it's actually really hard to get a group of, [33:02] like, a big group of people, to have like the same mission. [33:07] Right? [33:08] And I think the thing that I feel like the happiest about [33:10] when I come into work, [33:11] and probably the most proud of at Anthropic, [33:13] is how well that has scaled to a lot of people. [33:16] It feels to me like in this group [33:18] and with the rest of leadership, [33:19] everyone is here for the mission, [33:21] and our mission is really clear- [33:22] - Yep. - And it's very pure, right? [33:25] And I think that is something that I don't see as often, [33:27] to Dario's point, in sort of the tech industry. [33:29] It feels like there's just a wholesomeness [33:32] to what we're trying to do. [33:33] Like, no, I agree, none of us were like, [33:35] "Let's just go found a company!" [33:37] I felt like we had to do it, right? [33:39] It just felt like [33:40] we couldn't keep doing what we were doing [33:41] at the place we were doing it. [33:42] We had to do it by ourselves. [33:44] - And it felt like with GPT-3, [33:46] you know, which all of us had like touched or worked on, [33:48] and scaling laws and everything else, [33:50] we could see it in front of us in 2020. [33:53] And it felt like, well, [33:54] if we don't do something soon, all together, [33:56] you're gonna hit the point of no return. [33:58] And you have to do something [34:00] to have any ability to change the environment. [34:02] - Mm-hmm. [34:03] - I think, building off Daniela, [34:05] I do think that there's just like a lot of trust- [34:07] - Mm-hmm. - In this group. [34:08] I think each of us knows that we got into this [34:11] because we wanna help out with the world. [34:13] - Yeah. [34:14] - We did the the 80% pledge thing, [34:17] and that was like a thing that everybody was just like, [34:20] "Yes, obviously we're gonna do this." [34:21] - Mm-hmm. - Yeah, yeah. [34:22] - And yeah, I do think that the trust thing [34:26] is a special thing that's extremely rare. [34:29] - Yeah. [34:29] - I credit Daniela with keeping the bar high. [34:33] I credit you with the fact- [34:34] - Keeping out the clowns. [34:36] Keeping out the clowns. [34:37] - Chief clown wrangler! [34:39] That's my job. [34:40] - No, but you're the reason the culture scaled, I think. [34:41] - Yeah. [34:42] People say how nice people are here. [34:43] - Yeah. [34:44] - Which is actually a wildly important thing. [34:47] - I think Anthropic is really low politics, [34:49] and of course, [34:50] we all have a different vantage point than average, [34:52] and I try to remember that. - It's because of low ego. [34:54] - But it's low ego, [34:55] and I think I do think our interview process [34:57] and just the type of people who work here, [34:59] like, there's almost a like allergic reaction to politics. [35:03] - And unity. [35:04] - Mm-hmm. - Unity is so important. [35:06] The idea that the product team, the research team- [35:09] - Yes. [35:10] - The trust and safety team, [35:12] you know, the go-to market team, the policy team- [35:14] - Yeah. [35:15] - Like, the safety folks, [35:18] they're all trying to contribute to kind of the same goal, [35:22] the same mission of the company, [35:23] right? - Yes. [35:24] - I think it's dysfunctional when [35:26] different parts of the company [35:28] think they're trying to accomplish different things- [35:30] - Yeah. [35:31] - Think the company's about different things [35:32] or think that other parts of the company [35:34] are trying to undermine what they're doing. [35:36] - Yeah. [35:37] - And I think the most important thing [35:38] we've managed to preserve is, [35:40] and again, things like the RSP drive it, [35:42] this idea that it's not, you know, [35:45] there's some parts of the company causing damage [35:47] and other parts of the company trying to repair it, [35:50] but that there are different parts of the company [35:52] doing different functions [35:54] and that they all function under a single theory of change. [35:57] - Extreme pragmatism, right? [35:58] - Yeah. [35:59] - You know, the reason I went to OpenAI in the first place, [36:02] you know, it was a nonprofit, [36:03] it was a place where I could go and focus on safety, [36:06] and I think over time, you know, [36:08] that maybe wasn't as good a fit [36:10] and there were some difficult decisions. [36:12] And I think, in a lot of ways, [36:13] I really trusted Dario and Daniela on that, [36:16] but I didn't want to leave. [36:18] That was something that I think I was actually [36:21] pretty reluctant to go along with, [36:24] because I think, for one thing, [36:25] I didn't know that it was good for the world [36:27] to have more AI labs. [36:29] And I think it was something that [36:30] I was pretty, pretty reluctant for. [36:32] And I think, as well, when we did leave, [36:34] I think I was, you know, I was reluctant to start a company. [36:38] I think I was arguing for a long time [36:40] that we should do a nonprofit instead, [36:41] and just focus on safety research. [36:42] - Yes. [36:43] - And I think it really took pragmatism [36:46] and confronting the constraints [36:49] and just being honest about what the constraints implied [36:52] for accomplishing that mission- [36:53] - Mm-hmm. [36:54] - That led to Anthropic. [36:55] - I think just a really important lesson [36:57] that we were good about early on [36:59] is make less promises and keep more of them. [37:02] - Yeah. - Right? [37:03] - Yeah. [37:03] - Like, try to be calibrated, be realistic, [37:08] confront the trade-offs, because, you know, [37:11] trust and credibility [37:12] are more important than any particular policy. [37:14] - Yeah. [37:15] - It is so unusual to have what we have, [37:19] and watching Mike Krieger defend safety things, [37:23] of like reasons why we shouldn't ship a product yet, [37:25] but also then to watch Vinay sort of say like, [37:28] "Okay, we have to do the right thing for the business. [37:31] Like how do we get this across the finish line?" [37:32] - Mm-hmm, yep. [37:33] - And to hear, you know, [37:34] people like deep in the technical safety org [37:36] talking about how it's also important that we build things [37:39] that are practical for people, [37:41] and hearing, you know, [37:42] engineers on inference talk about safety. [37:43] That's amazing. [37:44] Like, I think that is, again, [37:47] one of the most special things about working here, [37:48] is everybody with that unity is prioritizing the pragmatism, [37:52] the safety, the business. [37:54] That's wild. [37:55] - The safest move- - I think about it as- [37:56] - Yeah. [37:57] - Spreading the trade-offs- - Yeah. [37:59] - From just the leadership of the company to everyone, [38:04] right? - Yeah. [38:04] - I think the dysfunctional world is like, [38:06] you have a bunch of people who only see a big, you know, [38:10] safety is like, "We always have to do this," [38:11] and product is like, "We always have to do this," [38:13] and research is like, you know, [38:15] "This is the only thing we care about." [38:17] And then you're stuck at the top, [38:19] right? - Yeah. [38:20] - You're stuck at the top. [38:20] You have to decide between [38:21] you don't have as much information as either of them. [38:24] That's the dysfunctional world. [38:25] The functional world is when [38:26] you're able to communicate to everyone, [38:29] "There are these trade-offs we're all facing together." [38:31] - Yeah. [38:32] - The world is a far from perfect place. [38:34] There's just trade-offs. [38:36] Everything you do is gonna be suboptimal. [38:38] Everything you do is gonna be [38:40] some attempt to get the best of both worlds that, you know, [38:43] doesn't work out as well as you thought it was, [38:46] and everyone is on the same page about [38:48] confronting those trade-offs together, [38:50] and they just feel like they're confronting them [38:53] from a particular post. [38:54] - Mm-hmm. [38:55] - From a particular job, as part of the overall job of [38:57] confronting all the trade-offs. [38:59] - It's a bet on race to the top, right? [39:00] - It's a bet on race to the top, yeah. [39:02] - Like it's not a pure upside bet. [39:03] Things could go wrong, but- [39:04] - Yeah. - Like, [39:05] we're all aligned on like, [39:06] "This is the bet that we're making." [39:08] - And markets are pragmatic, [39:10] so if the more successful Anthropic becomes as a company, [39:14] the more incentive there is for people to copy [39:16] the things that make us successful. [39:18] And the more that success is tied to [39:20] actual safety stuff we do, [39:22] the more it just creates [39:23] a gravitational force in the industry [39:25] that will actually get the rest of industry to compete. [39:28] And it's like, "Sure, we'll build seat belts [39:30] and everyone else can copy them." [39:32] That's good. [39:33] Yeah. [39:34] - That's like good world. [39:34] - That's really good. - Yeah. [39:36] This is the race to the top, right? [39:37] But if you're saying, [39:38] "Well, we're not gonna build the technology, [39:40] you're not gonna build it better than someone else," [39:42] that in the end, [39:44] that just doesn't work because you're not proving [39:46] that it's possible to get from here to there. [39:48] - Mm-hmm. [39:49] - Where the world needs to get, [39:50] nevermind the industry, nevermind one company, [39:53] is it needs to get us successfully through from [39:56] this technology does-doesn't exist [39:58] to the technology exists in a very powerful way [40:01] and society has actually managed it. [40:04] And I think the only way that's gonna happen [40:05] is that if you have, at the level of a single company, [40:09] and eventually at the level of the industry, [40:11] you're actually confronting those trade-offs. [40:13] You have to find a way to actually be competitive, [40:18] to actually lead the industry, in some cases, [40:20] and yet manage to do things safely. [40:22] And if you can do that, [40:24] the gravitational pull you exert is so great. [40:28] There's so many factors, [40:29] from the regulatory environment, [40:31] to the kinds of people who wanna work at different places, [40:34] to, even sometimes, the views of customers- [40:36] - Yeah. [40:37] - That kind of drive in the direction of [40:39] if you can show that you can do well on safety [40:43] without sacrificing competitiveness, right, [40:45] if you can find these kind of win-wins, [40:48] then others are incentivized to do the same thing. [40:50] - Yeah, I mean I think that's why [40:52] getting things like the RSP right is so important, [40:55] because I think that we ourselves, [40:59] seeing where the technology is headed, have often thought, [41:02] "Oh wow, we need to be really careful of this thing," [41:05] but at the same time we have to be even more careful [41:07] not to be crying wolf, [41:09] saying that like, "Innovation needs to stop here." [41:12] We need to sort of find a way to make AI useful, innovative, [41:19] delightful for customers, [41:20] but also figure out what the constraints really have to be [41:24] that we can stand behind, that makes systems safe, [41:27] so that it's possible for others to think that [41:30] they can do that too, [41:31] and they can succeed, they can compete with us. [41:34] - We're not doomers, right? [41:35] Like, we wanna build the positive thing. [41:37] - Yeah. [41:38] - We wanna like build the good thing. [41:38] - And we've seen it happen in practice. [41:40] A few months after we came out with our RSP, [41:43] the three most prominent AI companies had one, right? [41:46] Interpretability research, [41:48] that's another area we've done it. [41:49] Just the focus on safety overall, [41:52] like collaboration with the AI safety institutes, [41:54] other areas. [41:56] - Yeah, the Frontier Red Team got cloned almost immediately, [41:59] which is good. [42:00] You want all the labs to be testing for like, [42:02] very, very security-scary risks. [42:04] - Export the seat belts. [42:05] - Yeah, exactly. - Mm-hmm, [42:06] export the seat belts. [42:07] Well, Jack also mentioned it earlier, [42:08] but customers also really care about safety, right? [42:12] Customers don't want models that are hallucinating. [42:14] They don't want models that are easy to jailbreak. [42:16] They want models that are helpful and harmless, [42:19] right? - Yeah. [42:20] - And so a lot of the time, [42:21] what we hear in customer calls is just, [42:22] "We're going with Claude because we know it's safer." [42:24] I think that is also a huge market impact, right, [42:26] because our ability to have models [42:29] that are trustworthy and reliable, [42:31] that matters for the market pressure [42:33] that it puts on competitors, too. [42:35] - Maybe to unpack something that Dario said [42:37] a little bit more, [42:38] I think there's this narrative or this idea [42:40] that maybe the virtuous thing [42:42] is to almost like nobly fail, right? [42:45] It's like you should go [42:46] and like put safety, [42:48] you should go and put things, [42:50] you should sort of demonstrate like in an impragmatic way [42:53] so that you can sort of demonstrate [42:54] your purity to the cause or something like this. [42:58] And I think if you do that, [43:00] it's actually very self-defeating. [43:02] For one thing, it means that [43:03] you're gonna have the people who are are deciding, [43:06] making decisions, [43:06] be self-selected for being people who don't care, [43:09] and for people who aren't prioritizing safety [43:11] and who don't care about it. [43:13] And I think, on the other hand, [43:14] if you try really hard to find the way [43:17] to align the incentives [43:19] and make it so that if there are hard decisions, [43:22] they happen at the points where [43:23] there is the most force to go and support [43:26] making the correct hard decisions, [43:28] and where there's the most evidence, [43:29] then you can sort of start to trigger this race to the top [43:32] that Dario is describing, [43:33] where instead of going and having, you know, [43:37] the people who care get pushed out of influence, [43:39] you instead pull other people to have to go and follow. [43:43] - So what are you all excited about [43:45] when it comes to the next things we'll be working on? [43:48] - Well, I think there's a bunch of reasons [43:50] you can be excited about interpretability. [43:52] One is obviously safety, [43:54] but there's another one that I think I find, [43:56] at an emotional level, you know, [43:58] equally exciting or equally meaningful to me, [44:02] which is just that I think neural networks are beautiful [44:05] and I think that there's a lot of beauty in them [44:07] that we don't see. [44:08] We treat them like these black boxes that [44:10] we're not particularly interested in the internal stuff, [44:14] but when you start to go and look inside them, [44:15] they're just full of amazing, beautiful structure. [44:19] You know, it's sort of like if people looked at biology [44:22] and they were like, "You know, evolution is really boring. [44:27] It's just a simple thing that goes and runs for a long time, [44:31] and then it makes animals," [44:32] and like, instead, [44:33] it's like, actually, you know, [44:34] each one of those animals that evolution produces, [44:36] and I think that, you know, it's an optimization process, [44:39] like training a neural network. [44:40] You know, [44:41] they're full of incredible complexity and structure, [44:43] and we have an entire sort of artificial biology [44:47] inside of neural networks. [44:48] If you're just willing to look inside them, [44:51] there's all of this amazing stuff. [44:52] And I think that we're just starting to slowly unpack it, [44:55] and it's incredible, and there's so much there, [44:57] but there's just so much we discovered there. [44:59] We're just starting to crack it open [45:01] and I think it's gonna be amazing and beautiful. [45:04] And sometimes I imagine, you know, [45:07] like a decade in the future, [45:08] walking into a bookstore and buying, you know, [45:10] the textbook on neural network interpretability [45:12] or really, like, on the biology of neural networks, [45:15] and just the kind of wild things [45:17] that are gonna be inside of it. [45:18] And I think that in the next decade, [45:20] in the next couple of years even, [45:22] we're gonna go and start to go in [45:24] and really discover all of those things. [45:26] And it's gonna be wild and incredible. [45:29] - It's also gonna be great [45:31] that you get to buy your own textbook. [45:34] - Just have your face on it. [45:35] - I mean, yeah. [45:38] - I'm excited that a few years ago if you had said like, [45:41] "Governments will set up new bodies [45:44] to test and evaluate AI systems [45:46] and they will actually be competent and good," [45:49] you would've not thought that was going to be the case. [45:51] But it's happened, [45:52] it's kind of like governments have built [45:55] these new embassies, almost, [45:57] to deal with this new kind of class of technology [46:00] or, like, thing that Chris studies. [46:03] And I'm just very excited to see where that goes. [46:05] I think it actually means that we have state capacity [46:08] to deal with this kind of societal transition, [46:10] so it's not just companies. [46:12] And I'm excited to help with that. [46:14] - I'm already excited about this to, you know, [46:17] to a certain extent today, [46:18] but I think just imagining the future world [46:20] of what AI is going to be able to do for people [46:24] is it's impossible to not feel excited about that. [46:28] Dario talks about this a lot, [46:30] but I think even just the sort of glimmers of [46:32] Claude being able to help with, you know, [46:35] vaccine development and cancer research [46:37] and biological research [46:39] is crazy, like, just to be able to watch what it can do now, [46:42] but when I fast forward, you know, [46:44] three years in the future or five years in the future, [46:46] imagining that Claude could actually solve [46:48] so many of the fundamental problems [46:50] that we just face as humans. [46:53] Even just from a health perspective alone, [46:56] even if you sort of take everything else out, [46:58] feels really exciting to me, [47:00] just like thinking back to [47:01] my international development times. [47:03] It would be amazing if Claude was responsible for [47:05] helping to do a lot of the work that I was trying to do [47:07] a lot less effectively when I was, like, 25. [47:10] - I mean, I guess similarly, [47:11] I'm excited to build Claude for work. [47:13] Like, I'm excited to build Claude into the company [47:17] and into companies all over the world. [47:19] - I guess I'm excited just for, I guess like, personally, [47:23] like I like using Claude a lot [47:26] at work, [47:27] and so like, definitely, [47:29] there's been increasing amounts of like home times [47:31] with like me just like chatting with Claude about stuff. [47:34] I think the biggest recent thing has been code- [47:39] - Mm-hmm. [47:39] - Where like six months ago, [47:41] I didn't use Claude to do any coding work. [47:43] Like, [47:44] our teams didn't really use Claude that much for coding, [47:46] and now it's like just phase difference. [47:48] Like, I gave a talk at YCU like week before last, [47:52] and at the beginning I just asked like, [47:54] "Okay, so how many folks here use Claude for coding now?" [47:59] And literally 95% of hands- [48:01] - Wow. - Wow. [48:02] - Like, all the hands in the room, [48:03] which just is totally different than how it was [48:05] four months ago or whatever. [48:06] - Mm-hmm. - Yeah. [48:07] - So when I think about what I'm excited about, [48:09] I think about places where, you know, like I said before, [48:12] where there's this kind of consensus that, again, [48:15] seems like consensus, [48:17] seems like what everyone wise thinks, [48:18] and then it just kind of breaks, [48:21] and so places where I think that's about to happen [48:23] and it hasn't happened yet. [48:25] One of them is interpretability. [48:27] I think interpretability is both the key to [48:30] steering and making safe AI systems, [48:33] and we're about to understand, [48:35] and interpretability contains insights about [48:39] intelligent optimization problems [48:40] and about how the human brain works. [48:43] I've said, and I'm really not joking, [48:45] Chris Olah is gonna be a future Nobel Medicine Laureate. [48:48] - Aw, yeah. [48:49] - I'm serious. [48:50] I'm serious, [48:51] because a lot of these, [48:53] I used to be a neuroscientist, [48:54] a lot of these mental illnesses, [48:56] the ones we haven't figured out, right, [48:58] schizophrenia or the mood disorders, [49:00] I suspect there's some higher level system thing going on [49:04] and that it's hard to make sense of those with brains [49:06] because brains are so mushy [49:08] and hard to open up and interact with. [49:11] Neural nets are not like this; [49:12] they're not a perfect analogy, [49:13] but as time goes on, they will be a better analogy. [49:16] That's one area. [49:17] Second is, related to that, [49:19] I think just the use of AI for biology. [49:22] Biology is an incredibly difficult problem. [49:26] People continue to be skeptical, for a number of reasons. [49:31] I think that consensus is starting to break. [49:33] We saw a Nobel Prize in chemistry [49:34] awarded for AlphaFold; remarkable accomplishment. [49:37] We should be trying to build things [49:39] that can help us create a hundred AlphaFolds. [49:42] And then finally, using AI to enhance democracy. [49:46] We worry about if AI is built in the wrong way, [49:49] it can be a tool for authoritarianism. [49:52] How can AI be a tool for freedom and self-determination? [49:56] I think that one is earlier than the other two, [49:59] but it's gonna be just as important. [50:01] - Yeah, I mean, I guess two things that [50:04] at least connect to what you were saying earlier, [50:05] I mean, one is I feel like people frequently join Anthropic [50:12] because they're sort of [50:13] scientifically really curious about AI [50:17] and then kind of get convinced by AI progress [50:21] to sort of share the vision of the need, [50:24] not just to advance the technology, [50:26] but to understand it more deeply [50:28] and to make sure that it's safe. [50:31] And I feel like it's actually just sort of exciting [50:33] to have people that you're working with [50:36] kind of more and more united in their vision [50:39] for both what AI development looks like [50:41] and the sort of sense of responsibility associated with it. [50:44] And I feel like that's been happening a lot [50:46] due to a lot of advances [50:47] that have happened in the last year, [50:48] like what Tom talked about. [50:50] Another is that, I mean, [50:51] going back really to concrete problems, [50:53] I feel like we've done a lot of work on AI safety [50:56] up until this point. [50:57] A lot of it's really important, [50:59] but I think we're now, with some recent developments, [51:01] really getting a glimmer of what kinds of risks [51:04] might literally come about from systems [51:06] that are very, very advanced- [51:07] - Mm-hmm. [51:08] - So that we can investigate and study them [51:11] directly with interpretability, [51:13] with other kinds of safety mechanisms, [51:15] and really understand what the risks from [51:17] very advanced AI might look like. [51:21] And I think that that's something that is [51:23] really gonna allow us to sort of further the mission [51:26] in a really deeply scientific, empirical way. [51:29] And so I'm excited about sort of the next six months of [51:31] how we use our understanding of [51:33] what can go wrong with advanced systems to characterize that [51:37] and figure out how to avoid those pitfalls. [51:40] - Perfect. [51:41] Fin! - Yay! [51:42] - We did it! [51:43] - Woo! [51:44] - Good time. - I know. [51:45] We gotta do this more often.