Casado, Sinofsky Warn AI Labs' Extinction Talk Risks State Takeover

Casado与Sinofsky警告：AI实验室渲染人类灭绝论恐招致国家接管 渲染投机性的生存威胁或招致严厉监管甚至国有化，却未能解决自主智能体集群拖垮企业网络这一真实的系统级风险。 专家小组探讨了先发制人式AI监管的危害以及生存风险言论的表演性质，并将当前的政策恐慌与操作系统、汽车安全和航空领域的历史先例进行了对比。他们认为，前沿实验室将安全视作公关主导的“节奏控制”，而非标准的工程规范，这无异于招致政府的全面管控。随着自主智能体规模化运行，安全风险正从抽象的灭绝假想转向具体的系统威胁，例如内部拒绝服务攻击和细粒度访问控制失效。发言者最终认为，AI的重心正在超越单纯的基础模型训练，转向概率编程、结构化分类以及周边应用软件技术栈。 前沿AI实验室领导层因将负责任的扩展描述为“节奏控制”，同时公开认可两位数的人类灭绝概率而面临反对；批评者认为这种姿态招致的是严苛的法定封锁或国有化，而非健全的工程规范。 (4:44) 自主AI智能体的运行威胁特征类似于内部分布式拒绝服务（DDoS）攻击，因为不知疲倦的智能体集群会试探权限漏洞并向企业内部API发起海量请求，超出传统访问控制模型的承载能力。 (35:00) 预期的AI监管可能会像欧洲的GDPR一样，对每个智能体的写入操作都弹出无处不在的确认提示框，从而引发严重的警告疲劳，却未能实质性提高底层软件的安全性。 (40:00) 软件开发正从对话式文本生成转向快速、低成本的选择模型和结构化置信度输出，相较于基于聊天的界面，这复兴了20世纪60年代的概率编程范式。 (48:52)

Aaron Levie, Steven Sinofsky, and Martin Casado argue that AI risk discussions must shift away from existential panic and regulatory slowdowns toward concrete systems engineering, historical safety iteration, and granular access controls for autonomous agents. The Strategic Contradictions of Pacing and X-Risk Frontier AI lab proposals around 'pacing' development create dangerous regulatory vulnerabilities rather than genuine security. Martin Casado notes that framing risk around double-digit species extinction—such as Dario Amodei acknowledging a 10% extinction probability on television—undermines reasonable safety proposals. Casado argues that if models posed genuine existential risks comparable to weapons of mass destruction, private commercial enterprise would be incompatible with national security, demanding immediate nationalization. Instead, he views catastrophic rhetoric primarily as internal talent retention and recruitment currency. Political Hazards and Premature Regulation (8:00) Steven Sinofsky and Aaron Levie critique the premise of pacing from operational and policy standpoints. Sinofsky points out that without a public delivery schedule, claims of pacing are public relations theater, contradicted by rapid compute expansion and capital deployment. Requesting regulatory pacing invariably backfires because legislative compromise yields distorted rules rather than calibrated velocity. The panel observes that historic technology shifts—such as early operating systems and the 1990s internet—survived widespread damage and malware because safety standards were developed iteratively following operational failures rather than pre-emptive legislative bans. Agent Swarms and Operational Security Realities (35:00) The primary near-term hazard introduced by autonomous AI agents is superhuman persistence and exhaustiveness rather than philosophical sentience. Because agents execute programmatic actions tirelessly without fatigue, swarms querying internal corporate tools such as Slack, GitHub, or finance software can mirror coordinated denial-of-service attacks. Modern enterprise security assumes 95% to 99% of internal users act in good faith; autonomous agents break this perimeter model by confusing benign workflows with catastrophic ones. OS Granularity Against Alert Fatigue (40:00) Current operating systems force an impractical choice between granting agents unchecked file access or triggering constant permission prompts. Sinofsky warns against a 'GDPR for AI,' noting that mandatory warnings lead to alert fatigue, mirroring the failed User Account Control dialogues in Windows XP and cookie banners. Levie explains that enterprise adoption requires scoped folder-level access control layers. Historically, secure computing paradigms failed due to human usability friction, but Casado suggests autonomous software agents may revive granular permissions and operating system hardening. The Evolution Past Chat to Probabilistic Programming (48:52) Software architecture is moving away from conversational text generation toward specialized selection and classification models. Natural language chat is an inefficient, fragile interface for deterministic software systems. Sinofsky and Levie emphasize that replacing verbose text outputs with structured probability distributions resurrects 1960s probabilistic simulation techniques, allowing developers to embed confidence scores directly into conditional logic. As foundational labs focus on platform maintenance and infrastructure, innovation is shifting toward external developers building integration architectures. The panel concludes that as foundational labs mature into overburdened platform maintainers, genuine breakthroughs will migrate to the application layer, leaving open whether industry self-regulation like a FINRA model can emerge before political momentum forces heavy-handed statutory intervention.

Box 首席执行官 Aaron Levie、微软前高管 Steven Sinofsky 与 a16z 合伙人 Martin Casado 指出，前沿实验室渲染的人类灭绝风险与所谓“放慢节奏（pacing）”倡议在工程和逻辑上均站不住脚，过早立法更可能引发监管俘获。他们主张，AI 创新的重心正从前沿基础模型的自然语言生成转向外部软件层的概率编程、分类选择架构以及对实操级系统安全与权限的迭代防范。 灭绝论叙事与节奏论的逻辑缺陷 (4:44) Martin Casado 批评前沿实验室将务实的安全建议与极端灭绝风险论（X-risk）捆绑宣传，指出若技术真具灭绝威胁，国家安全的唯一应对方式是立即国有化，而非寄希望于企业自愿放慢速度。他强调放慢速度并不等于安全，正如缓慢组装核武器并不会降低其危险性；许多末日言论实质上是少数实验室内部用来吸引和保留顶尖研究人员的文化货币，若以此重塑国家产业政策将极为危险。Steven Sinofsky 补充指出，在没有公开既定排期基准的情况下宣称“放缓节奏”在产品管理上纯属公关修辞，且实验室疯狂融资和扩张算力的现实也直接矛盾于放缓声称；Aaron Levie 虽然认可前沿实验室必须维持严格的安全治理以建立企业级信任，但同样警惕政客借题发挥导致监管俘获并扼杀竞争。 技术史印证过早立法的治理误区 (20:00) 从计算机与关键基础设施的发展史来看，系统的韧性来自于对真实失败案例的迭代排查，而非提前对未知灾难进行立法。Sinofsky 指出，国会当年面对黑客入侵 GTE TeleMail 事件时耗时两年半才精准制定了《计算机欺诈与滥用法案》（CFAA），而当今现行法律对未授权访问已有充分约束，真正缺失的是前沿实验室遵循 CVE 漏洞披露等严谨工程规范的文化。Casado 与 Sinofsky 回顾了早年互联网每台直连 PC 必然感染蠕虫、Windows CE 设备被植入色情服务器等混乱历程，并对比汽车、航空和制药行业往往经历数十年真实运行事故积累（如 Ralph Nader 揭露汽车安全缺陷或沙利度胺事件）才催生出现代 FAA 和 FDA 监管标准。若过早引入政策，只会像微软反垄断危机或政界对错失监管社交媒体的报复心理一样，导致政策在政治妥协中扭曲，甚至催生出类似车贴警告与 cookie 弹窗那般令人麻木无效的“AI 版 GDPR”确认机制。 智能体集群对系统安全与权限的冲击 (35:00) 当前软件与操作系统并非针对具备无限耐心、能够持续调用 API 和执行代码的自主智能体设计。Casado 和 Sinofsky 回顾了冷战时期国防安全对抗——从消除电磁辐射泄漏的 TEMPEST 标准，到为防止维护人员识别键盘磨损而采取的物理上锁措施，再到通过窗口光电二极管对 CRT 扫描光线进行侧信道采样；如今系统学者已开始针对 CPU 散热调控穿透物理隔离等隐蔽通道展开比特率计算。更为紧迫的威胁在于企业内部架构：传统企业软件默认 95% 至 99% 的内部员工出于善意，但成千上万个无疲劳感的智能体一旦获得凭据和信用卡，其高频自动化查询会在防火墙内部演变为类似内网拒绝服务（DDoS）的流量冲击。Levie 指出，当前操作系统仅提供全盘放行或步步弹窗的极端选择，必须对文件系统和 API 进行粒度更细的读写权限重构，倒逼计算机语言、网络与操作系统迎来一场基于实操验证的硬化复兴。 从自然语言生成转向概率编程与分类 (50:00) 大模型与实际业务集成的方式正经历底层范式变革，从低效的纯对话生成转向直接输出结构化置信区间的确定性选择模型。Sinofsky 直言，从未有用户研究证明纯自然语言交互对人机协作是高效的；在传统软件中让大模型生成多段冗长文本再去解析，既昂贵又极其脆弱。相反，将模型定位为选择与分类器，使其吸收上下文并从固定候选中做出判定或给出概率打分（如判定某工单有 80% 几率属于客户支持），能大幅降低成本、提高吞吐速度与准确率。这标志着 1960 至 1970 年代源自物理模拟和弹道计算的“概率编程”学术传统的复兴，使开发者能直接在代码中的条件分支和循环结构里应用概率门控。Casado 指出，前沿实验室沉迷于创造如神明般对话的智能体，而软件工程师需要的只是能驱动 if 语句的高性能概率引擎；随着基础模型演变为需负担向后兼容和运维的技术平台，AI 创新的主战场正向外围的系统与应用架构迁移。 讨论总结认为，过早针对抽象灭绝风险立法将导致不切实际的政治妥协与过度繁苛的阻碍，而技术真正的安全性取决于应对真实崩溃场景的系统工程。随着基础模型平台逐步定型，行业重心正转向更底层的权限设计、概率控制架构以及实用的分类模型集成，但如何避免监管层在 2028 年大选周期中将其政治化仍是未解难题。

## Background 本期内容由 Erik Torenberg、Martin Casado、Steven Sinofsky 与 Aaron Levie 共同参与，并通过 RSS/网络平台于 2026 年 9 月 26 日发布。节目简介显示，主持人 Erik Torenberg 与 Box 首席执行官 Aaron Levie、a16z 普通合伙人 Martin Casado 及微软前高管 Steven Sinofsky 共同探讨了随着高能力 Agent 走向现实，AI 行业应当如何审视安全、防御与监管，并借鉴计算机病毒、早期互联网、航空和汽车等历史浪潮的反思，探讨底层权限重构及创新向模型外围软件转移的趋势。 ## TL;DR 四位对谈者深入剖析了当下前沿实验室所谓“放慢节奏（pacing）”与渲染人类灭绝风险（X-Risk）在逻辑和公关层面的荒谬与自相矛盾，警告过早实施预防性监管极易被政治俘获并重蹈历史覆辙。回顾早期 PC 病毒、航空及汽车行业的发展史，众人指出行之有效的安全规范向来建立在具体的系统失效案例之上，而非凭空假想。面对数以万计不知疲倦的 AI Agent 蜂群对企业内网构成的 DoS 威胁，企业必须重塑细粒度权限和身份验证体系，警惕“AI 版 GDPR”式的无意义弹窗疲劳。此外，大语言模型正在从昂贵低效的自然语言文本生成转向低成本的离散决策与分类，促使传统模拟与概率编程复兴，技术创新的核心重心已从基础模型本身全面转移至外围软件架构。 ## Atmosphere & style 整场对谈兼具硅谷老将式的风趣犀利与硬核工程学者的务实思辨，开篇互相打趣风投抢功，随后迅速切换到对政策幼稚病和公关虚伪性的猛烈抨击。几位嘉宾引经据典，从冷战时期的潘兴导弹、早期黑客恶作剧讲到 Windows 崩溃与微积分模拟，将虚浮的末日科幻想象硬生生拉回由物理定律、系统熵增与代码架构支撑的真实世界。对话节奏紧凑、交锋密集且充满行业洞见，听众能深刻感受到一种拨开宏大叙事迷雾、回归硬核系统工程与商业历史常识的清醒与快感。 ## Bulletin - `05:00` 前沿 AI 实验室宣称的“控制研发节奏”与生存风险（X-risk）叙事在逻辑上面临内在矛盾：若技术确实存在灭绝风险，国家安全逻辑将直接推向技术国有化；若非如此，所谓放慢节奏更多是解决内部研究员招聘与留存的公关手段。 - `35:00` 针对自主 AI 智能体蜂群的内网 DoS 风险，传统企业基于“绝大多数员工遵守规范”的安全假设彻底失效，操作系统与身份认证体系亟待重构，但需警惕过度依赖类 GDPR 弹窗导致用户警报疲劳。 - `44:30` 历史表明汽车、航空及互联网等成熟安全规范均是在数十年真实损害案例积累后才逐步成型，过早推行预防性强监管可能会在未知失败机理的情况下扼杀前沿技术创新。 - `50:00` 企业级 AI 软件架构正在从昂贵、低效的长文本生成，迅速转向结构化离散选项分类与概率编程（Probabilistic Programming），创新的重心已从基础模型内部转移至外围执行架构。 ## Key takeaways - `05:00` 速度与安全在逻辑上是正交的两个维度，缺乏发布基准的“前沿控制节奏”在实践中无法自洽，各大实验室现实中仍在加速融资与扩张。 - `10:00` 科技界向政界主动诉求“温和监管”往往会适得其反，政府的政治妥协机制输出的政策要么监管过度、要么执行不力，绝不会带来行业预期的适度速度。 - `12:30` 支持 AI 的宏大叙事在政治传播上处于劣势，行业语汇库被“暂停”“失控蜂群”等恐慌词汇劫持，2028 年美国大选恐演变为围绕 AI 的公投。 - `13:45` 在国家安全与政界逻辑中，一旦将灭绝风险判定为“非零概率”，唯二的政策归宿即是类似核武的高危管制与事实上的国有化，这将直接扼杀前沿开源生态。 - `15:45` 历史重演表明硅谷对政治运作存在长期盲区，从 AT&T、IBM 到微软反垄断，技术巨头试图通过游说或私人关系规避监管的尝试无一例外走向失败。 - `20:00` 政界在 AI 领域存在强烈的“补课心理”，因后悔当年在《通信规范法案》中免除互联网和社交媒体责任，正倾向对 AI 采取惩罚性与预防性监管。 - `25:00` 有效的法律体系（如 1986 年《计算机欺诈与滥用法案》）均基于真实入侵损害制定，现行应用层法律已能涵盖多数安全场景，盲目推倒重来并无依据。 - `25:00` 当前前沿实验室在系统被攻破后的安全事后复盘（Postmortem）缺乏透明度与工业标准，甚至被指责为经过公关修饰的避重就轻文书。 - `30:00` 安全界的极端边缘假设（如超级智能借 CPU 散热传输代码）与真实硬件物理信道（如 CRT 反光采样、Tempest 辐射）形成鲜明对照，系统安全应回归物理与工程指标。 - `35:00` AI 智能体蜂群“不知疲倦”且自动化调用 API 的特性，使内网软件面临前所未有的 DoS 攻击隐患，企业访问控制模型面临性能与颗粒度的全面重构。 - `40:00` 为 AI Agent 引入强制弹窗确认将催生“AI 版 GDPR”灾难，过度密集的警报只会引发用户疲劳，导致安全防护沦为无效的法律免责标签。 - `44:30` 成熟工业监管体系（汽车、制药、航空）均经历了长达数十年的事故积累，若在 1910 年成立现代航空管理局，航空工业将根本无法发展。 - `48:26` 自然语言交互在绝大多数人机任务中效率极低，AI 正在从生成漫长文本转向低延迟、低成本的结构化决策与离散集合选择。 - `50:00` 大模型与传统软件的结合正在促成计算机科学“概率编程”的复兴，代码控制流开始从确定性布尔逻辑转向基于概率阈值的条件执行。 - `52:45` AI 行业的创新重心已明确从基础模型层转移至模型外围，平台型实验室受限于运维与兼容负担，核心突破正由外围软件与应用架构主导。 ## Detailed walkthrough ### `00:00` 开篇漫谈：安全监管的滞后性与前沿节奏之争 播客在一段激烈的观点交锋剪辑中拉开帷幕：过早监管 AI 根本解决不了任何问题，不仅无法消除风险，反而像是在还没学会如何控制它之前就强行催生了这个实体；与此同时，前沿实验室与安全研究圈之间裂痕加深，企业级系统面临 5,000 到 10,000 个自主 Agent 漫游乱窜的新挑战，底层权限与身份验证体系亟待重构；历史表明，计算机病毒与安全漏洞并没有阻止互联网的发展，而是促成了更强大的防御体系。主持人 Erik Torenberg 欢迎 Box 首席执行官 Aaron Levie、前微软 Windows 部门总裁 Steven Sinofsky 以及 a16z 普通合伙人 Martin Casado 回归节目，几个人一开场便互相打趣——调侃 Martin 最近在忙着孵化千亿美元估值的独角兽，或者是“按风投的传统惯例把功劳揽在自己身上”。 ```mermaid flowchart TD accTitle: 前沿实验室安全治理与监管争议 accDescr: 展示前沿实验室在工程治理、舆论氛围与监管诉求之间的张力与矛盾 subgraph LabLevel ["实验室层面：工程与安全规范"] A["高标准工程规范与内部治理"] --> B["企业级部署信任基础"] end subgraph PerceptionLevel ["外部舆论与氛围矛盾"] C["员工宣称 10% 灭绝概率"] --> D["媒体访谈公开背书 X-Risk"] D --> E["温和治理方案显得苍白无力"] end subgraph PolicyLevel ["潜在监管后果"] E --> F{"政策制定者介入"} F --> G["监管俘获 / 阻碍后来者竞争"] F --> H["非理性限制与数据中心封禁"] end ``` 谈话迅速切入正题，Erik 抛出核心议题：Anthropic 首席执行官 Dario Amodei 提出的“前沿模型节奏控制（Pacing the frontier）”主张引发了广泛关注，业界对此反应剧烈。Aaron Levie 首先提出基准立场：任何处于前沿的 AI 实验室，都应当在最高标准的安全、治理、安全防护以及所谓“对齐（alignment）”规范下进行构建。如果没有极高水准的企业信任和系统防御，AI 的产业普及与扩散就根本无从谈起。在 Aaron 看来，Dario 的文章本身非常中肯，通篇都在谈论工程卫生规范以及如何增强系统的安全性，但争议在于这些论点会不会被政治人物顺水推舟，变成阻碍竞争、限制前沿创新的监管工具。 然而，Martin Casado 随即指出，问题根本不在于文章的技术细节，而在于整体“舆论氛围（atmospherics）”的严重割裂。一方面，实验室员工在对外宣称模型有 10% 的可能导致人类灭绝；而更荒谬的是，Dario 在抛出治理文章的同一天，竟然在电视采访中公开表示自己同意这种灭绝风险论调。Martin 尖锐地批评道：“And so in some way, you can't have these conversations in isolation, which is, of course, if he's going to agree in species extinction, this post that he has looks like this milquetoast capitulation that's totally not adequate for the task at hand.” 如果高管真的相信人类物种可能就此灭亡，那么这篇软弱妥协的提案在灭顶之灾面前就显得极其苍白且不相称。 - Aaron Levie 强调 AI 扩散（diffusion）的前提是构建出可信赖、不会持续破坏系统的企业级产品。 - 舆论氛围的崩塌源于前沿实验室一边高调渲染灭绝论，一边提出常规的企业安全合规方案。 --- ### `05:00` “放慢节奏”是伪命题：存亡风险、HR问题与虚假的进度表 Martin Casado 进一步剖析了“放慢节奏（pacing）”这一用词的内在逻辑缺陷。首先，速度和安全是完全正交的两个维度——慢吞吞地制造一枚核武器，绝不会让旁观者感到丝毫更加安全。其次，所谓的“控制节奏”本质上是在两头讨好却两头落空：它既不能让主张彻底暂停（pause）的极端派满意，因为这并不是真正的停滞；也不能让监管机构安心，因为实验室依然在向前推进研发。在 Martin 看来，前沿实验室陷入了夹在内部少数极端“末日论者（doomers）”研究员与外部监管者之间的尴尬境地。 针对实验室内部那些“既对 AI 极度恐惧又坚信自己必须冲在前沿”的科学家群体，曾任职于劳伦斯利弗莫尔国家实验室（Lawrence Livermore National Laboratory）的 Martin Casado 给出了极其强硬的划分：如果最懂技术的顶尖群体真的确信 AI 构成了人类存亡级别的威胁（X-risk），那么唯一符合逻辑的方案是由国家将其收归国有，并施加行之有效的国家级管控；但如果这仅仅是极少数人的焦虑，那么这根本不是科技风险，而纯粹是一场公司内部的人力资源难题。Martin Casado 一针见血地指出：“If they are worried that they can't recruit people or they can't retain people, which it seems to me a lot of this is just that concern. It's almost more of this kind of researcher currency. If that's the case, I think that is the wrong reason to cause a national level lockdown on a very promising technology.” 随后，一直蓄势待发的 Steven Sinofsky 加入了讨论。Steven 毫不留情地直斥所谓的“节奏控制”纯属无稽之谈：前沿实验室从未公布过任何确切的开发时间表，外界根本无从知晓原计划什么时候会搞出什么成果，这就好比媒体报道“苹果最新的 iPhone 延期了”一样荒谬——在没人知道发布基准的前提下，谈论延期或放慢根本毫无意义。Steven Sinofsky 直言：“In order for something to be slower, you need to know the rate at which was moving in the first place.” 更关键的是公关层面的虚伪性：现实中各家实验室都在以亡命之徒般的速度（dead run）疯狂融资和极速扩张，没有任何迹象表明有人在放慢脚步。Steven 质疑，如果开发者真的最害怕自己产品带来的可怕后果，直接停手不做便是，何必大张旗鼓地公开发表声明，甚至反过来恳求政府来命令自己放慢速度？他在大学时期曾在导弹工厂工作过，面对核武器的现实风险，要么选择在校园抗议拒绝制造，要么全力以赴构建国防力量，而当今 AI 实验室的做法完全违背了基本的常识逻辑。 - Martin 认为拿内部研究人员的招聘和留存问题（HR 难题）当借口，去呼吁对极具前景的技术实施国家级管制是本末倒置。 - Steven 强调放慢速度必须以预设的基准速度为前提，前沿实验室所谓“控制节奏”在逻辑上根本无法成立。 - 各大实验室前所未有地加速融资与扩大规模，其现实行动与宣传的自我克制背道而驰。 ### `10:00` 政府运作的机制与“AI 监管妥协”的陷阱 对话承接冷战时期军备竞赛的历史记忆展开。Steven Sinofsky 回忆起冷战期间，天主教修女曾前往导弹基地将血泼在导弹上以示抗议，而他当时只是推着装满电脑的手推车，懵懂地向人推销“这是一台安全 PC”，直到后来才意识到潘兴导弹（Pershing missile）正是最终终结冷战的关键力量。借此历史隐喻，讨论转向了当下围绕 AI 发展速度（pacing）的争论。 Sinofsky 尖锐地指出，试图与政府协商某种“居中的推进节奏”纯属幻想。科技圈许多人以为向政府提建议——“该做什么、不该做什么”——就能得到理想的结果，这完全是对政府运作逻辑的彻底误解。政府在面对诉求时，倾听每一个利益方的声音，但在现行体制下，任何产出都必然是多方妥协的结果。最终出台的政策 100% 无法让所有人满意：它要么做得不够彻底，要么矫枉过正。正如 Steven Sinofsky 所断言的：“你不可能抱有这种想法：去和政府谈谈，就能通过游说得到你想要的解决方案。因此底线是，如果他们要求把控节奏，最终得到的只会是错误的演进速度（If they're asking for pacing, they're going to get the wrong velocity）。” 随后对话提到了政界对科技界主动求监管的冷淡反应（例如传闻中政界人士直接回复“你想让我们来监管你？门都没有”）。但无论如何，一旦监管机器开始转动，它就会演变成横跨所有司法管辖区和两党各层级的竞选议题。 ```mermaid flowchart LR accTitle: 政府监管逻辑与妥协产物 accDescr: 展示科技企业诉求进入政府后，经过多方妥协最终输出极端不符预期的政策结果。 subgraph S1["利益输入"] A["科技行业诉求"] B["反 AI / 安全阵营"] C["公众与工会担忧"] end subgraph S2["政治黑箱"] D["政府倾听与妥协机制"] end subgraph S3["政策输出"] E["要么力度过弱"] F["要么监管过度"] end A --> D B --> D C --> D D --> E D --> F ``` ### `12:30` 2028 年大选的叙事困境与被劫持的语汇库 讨论进一步推演到未来的政治选举，与会者一致认为 2028 年美国总统大选必将成为一场围绕 AI 的全民公投（referendum）。然而，科技行业在舆论场上面临着极不对称的劣势：很少有政客能够凭借“支持 AI”的政纲赢得选举，因为支持 AI 的宏大叙事过于抽象、宏大且耗费口舌，而反对 AI、呼吁强力监管的口号则简单直接、直击恐惧。 更严重的问题在于科技界失去了话语权。目前围绕 AI 风险的整套公共词汇，完全是由反对或抵制 AI 的阵营所定义的。像“暂停”（pause）、“群体/蜂群”（swarms）、“失控/流氓”（rogue）等词汇，已经把讨论框定在一种恐慌防御的语境中。支持 AI 的一方往往需要耗费极大篇幅去辩解、反驳旧词并创造新概念，这在政治传播上极度吃亏。行业如果想要夺回叙事主动权，必须建立一套聚焦于现实民生的新词云，比如将 AI 与“工会工作”（union jobs）、“攻克癌症”（cancer）和解决实际生产力问题紧密结合起来。 ### `13:45` 人类灭绝风险（X-Risk）与模型实验室的自我束缚 Martin Casado 抛出了一个核心疑问：为什么主流前沿实验室（frontier labs）至今不在“人类灭绝风险”（X-Risk）上明确划清界限？如果顶级 AI 公司的 CEO 公开表示认同人类可能因 AI 面临灭绝，政府除了实施雷霆手段的重度监管，还能有什么别的选择？ 围绕 Anthropic 首席执行官 Dario Amodei 的公开表态，众人展开了激烈交锋。Dario 在被问及灭绝几率时，刻意回避给出一个具体百分比，但又认可风险非零。Aaron Levie 认为，要求实验室给出一个准确数字是不切实际的，因为这本身就是一个凭空捏造、无法量化的概念；实验室能给出的最诚实的公关表态无非是“AI 存在真实风险，但也有巨大收益，我们正在尽最大努力将风险降到最低”。 然而，Sinofsky 和 Casado 指出，在政治逻辑中，“非零概率”（non-zero）本身就是一个极其危险的定性： - 历史上五角大楼在模拟核战争时，只要灾难发生的可能性被判定为非零，应对策略就完全不同。 - 一旦你承认人类面临灭绝风险且概率非零，在国家安全逻辑下，唯一的结论就是像核武器一样必须被彻底“国有化”（nationalized）。 - 前沿实验室目前的尴尬处境在于：他们希望将风险描绘为“非零”，以此作为获取特权牌照或行业准入门槛的筹码，同时又极力想摆脱被国家直接接管的命运。 正如 Martin Casado 在剖析这种政策走向时所警告的：“它对开源不利，至少对前沿大模型的开源极其不利，但在极限情况下，这确实是该行业一个完全可能出现的结局（It's not good for open source or at least frontier open source, but it's a plausible outcome for this industry in the limit）。” ```mermaid flowchart TD accTitle: X-Risk 定性与监管升级路径 accDescr: 实验室承认灭绝风险非零，导致政策推向严格合规甚至事实上的国有化。 A["实验室表态：灭绝风险为非零 (Non-zero)"] --> B{"政治与国家安全逻辑"} B -->|"参考核武器与高危生物实验室 (BSL-4)"| C["极重度监管框架 / KYC 准入审查"] C --> D["行业事实上的'国有化'"] D --> E["扼杀前沿开源生态"] D --> F["极大抬高创新门槛"] ``` ### `15:45` 历史重演：从电力金融到微软反垄断的监管教训 对话将 AI 与以往的重工业和科技浪潮进行了纵向对比。虽然银行、电力和医疗在法律意义上属于私营企业，但在二战后，由于其作为基础设施的极端重要性，通过极其严苛的合规体系（如银行的 KYC 规则），在一切实际功能层面（all practical purposes）都已经沦为了事实上的被监管国企。AI 若继续被置于生存危机的语境下讨论，最终也会走向这一终局。 尽管面临这种拉扯，Aaron Levie 仍对实验室领导层抱有一定乐观态度，认为随着产业高速扩张，各大公司正在逐步理解并试图调和这种内部张力。但 Sinofsky 随即指出，这反映了硅谷长期以来的幼稚与傲慢——过去一百年间，科技行业每一次技术浪潮都在重复同一个错误：科技精英自以为能够驾驭政治程序，但实际上他们根本不懂监管的真正运转逻辑。 为了佐证这一历史规律，Sinofsky 列举了数个经典商业史案例： - **AT&T 与 IBM**：这两家企业从诞生之初就带有国家特许垄断的基因，即便在 20 世纪 60 年代拥有数百名顶尖律师团队护航，依然逃脱不了反垄断诉讼的重锤，最终业务架构均被国家力量强行重塑。 - **微软与反垄断危机**：在 20 世纪 90 年代面临反垄断调查时，微软内部完全不知所措。比尔·盖茨（Bill Gates）前脚还在跟比尔·克林顿（Bill Clinton）一起打高尔夫球，后脚克林顿政府就对微软发起了反垄断诉讼；盖茨甚至纳闷：“我不是都去打高尔夫了吗？这难道不就是我应该做的公关吗？” - **好莱坞红恐与电影分级**：早年好莱坞在面对政府针对涉红（Red Scare）和性暗示内容的审查威胁时，也是通过行业内自发联合与自我规训，才在政府干预的夹缝中摸索出自己的应对体系。科技界目前在 AI 监管面前的表现，不过是又一次对历史盲区的天真暴露。 ### `20:00` 行业自律的局限与政界对 AI 监管的“补课心理” 探讨监管路径时，话题从好莱坞的自我审查机制延伸开来。早期电影业因为成人题材受到政府威胁，各方最终妥协成立了美国电影协会（MPAA）推行电影分级制，实现了行业自律。然而，当有人提出是否可以效仿美国金融业监管局（FINRA）模式来监管 AI 时，辩论立刻升温。反对者指出，FINRA 与 MPAA 完全不同，它直接伴随着立法和政府直接监督，本质上是“将风险国有化”；但另一种声音则认为，考虑到未来 AI 技术的破坏力，FINRA 恐怕已经是行业能争取到的“最好局面”。当一个开源权重模型被塞进医疗设备、操纵高频交易系统甚至控制飞机驾驶时，美国国会绝不可能坐视不管。 这种监管冲动背后还夹杂着政界的历史心结。当年的参议院在通过《通信规范法案》（Communications Decency Act）时，免除了网络服务商和社交平台的连带法律责任。如今国会山上的许多议员对此耿耿于怀，甚至当年私下抱怨：“互联网规模这么大，我们怎么居然毫无参与感？”前副总统阿尔·戈尔（Al Gore）当年试图走在创新前沿、强调政府在互联网构建中的关键作用，却因为一句“发明了互联网”沦为公众笑柄。如今，这种错失感正在变质。以参议员伊丽莎白·沃伦（Elizabeth Warren）为代表的政客频频发文，认定当年没能管住社交媒体是一大失误。政界对科技行业长期积压的怨气与挫败感，如今正全盘倾泻到对 AI 的监管之上。 Steven Sinofsky 尖锐地指出了两者的本质差异： > “But the problem with FINRA is that is a perfect example of essentially nationalizing risk.” - **行业自律对比**：MPAA 依靠自愿性质的分级制度避免了宪法第一修正案的违宪风险（如无法强制监管有线电视 HBO），而 FINRA 则是银行出资但由国家强制法案背书的监管机构。 - **政界的错失焦虑**：参议院普遍认为当年过早放任了互联网与社交媒体的野蛮生长，导致现在试图在 AI 时代提前“补课”，进行惩罚性甚至预防性立法。 ```mermaid flowchart TD accTitle: 两种自律与监管模式对比 accDescr: 对比电影协会的纯自律模式与金融监管局的法定监管模式 subgraph MPAA["MPAA 模式（影视分级）"] M1["行业威胁与危机"] --> M2["自愿联合成立组织"] M2 --> M3["自我审查与分级制"] M3 --> M4["无国家立法强制力"] end subgraph FINRA["FINRA 模式（金融监管局）"] F1["金融系统性风险"] --> F2["国会立法授权"] F2 --> F3["行业出资运作"] F3 --> F4["法定监管牙齿（风险国有化）"] end ``` --- ### `22:45` 历史上的技术阵痛期：过早监管可能扼杀互联网 对技术危害的“提前预测式监管”违背了科技史的常理。在汽车、航空和早期互联网的发展史上，社会都经历过一段伴随着真实损失的“磨牙期”（teeth cutting）。在互联网刚起步的年代，网络故障是家常便饭，恶意软件造成的经济损失高达数百亿美元，蠕虫病毒多次瘫痪高达 10% 的互联网基础设施，甚至导致医院系统停摆。在那个时代，用户买一张 Windows 95 光盘回家，电脑还没装完系统，只要连着网就可能立刻感染蠕虫病毒。直到 2001 年之前，PC 只要接入网络就几乎无法幸免于病毒感染。 然而，彼时的政策制定者并没有因噎废食。如果 1994 年或 1997 年就存在今天这种风声鹤唳的监管思潮，互联网大概率早已被扼杀在摇篮里。历史表明，行之有效的监管政策从来不是靠“假设如果发生某事”凭空捏造出来的，而是建立在具体的真实损害案例之上。即使当年闹得沸沸扬扬、声称会毁掉文明世界的“千年虫”（Y2K）危机，最终也没有演变成现实的灭顶之灾。只有当真实的破坏模式浮现，政策才能精准对齐事实，而不是在实验室和政客的空想中盲目设限。 Martin Casado 回顾早年混乱的技术现实时直言： > “The reality of the PC until 2001 was you could not install a PC connected to the network without getting infected.” - **基础设施的早期脆弱性**：早期网络蠕虫频繁打垮关键网络节点，医院与商业机构屡屡断网，但社会承受了这种代价以换取技术演进。 - **事实驱动 vs 空想假设**：汽车与航空的安全规范均是在事故血泪史后建立的；过早针对未知风险立法，往往会导致规则无法匹配真实的技术应用场景。 --- ### `25:00` 1986 年 CFAA 判例、模型层责任与敷衍的安全复盘 以史为鉴，1986 年美国《计算机欺诈与滥用法案》（CFAA）的出台就是一个典型的“事实驱动立法”。1983 年，两伙黑客入侵了 GTE TeleMail 系统——该系统当时正为美国国家航空航天局（NASA）和劳伦斯利弗莫尔国家实验室等联邦核心机构提供服务。由于当时甚至不存在针对“非法入侵计算机”的罪名，司法部门足足花了两年半时间才完成定罪立法。现在的法律在应用层（Applied Layer）其实已经覆盖了 90% 的场景，“未经授权访问系统”无论如何都是非法的。司法部后续也通过备忘录明确了豁免范围：只要是白帽黑客改善系统的善意测试，或是单纯违反网站服务条款（Terms of Service）而非真正实施破坏，便不予起诉。 相比之下，当今 AI 前沿实验室与传统网络安全界之间存在巨大的裂痕。从卡内基梅隆大学（CMU）在 1980 年代建立 CVE（通用漏洞披露）标准以来，安全界已经形成了一套严密的漏洞报告和责任追究体系。然而，现今许多主流 AI 实验室（如 OpenAI 等）在系统被攻破或发生安全事件后的“事后复盘”（postmortem）却极其敷衍。安全专家普遍指责这些报告“既马虎又不完整”，看起来根本不是严肃的技术复盘，倒像是聘请外部律所精心剪裁事实、避重就轻公关文书——核心的 Slack 沟通记录和事件细节全被隐去。在 AI 实验室建立起符合工业标准的安全披露制度之前，空谈所谓的新型安全风险显得格外荒谬。 Steven Sinofsky 严厉批评了当前 AI 实验室缺乏透明度的作风： > “Like they shouldn't do a postmortem on a breach that looks like an intern wrote it and it's not a postmortem.” - **1986 年立法渊源**：CFAA 的制定源自 GTE TeleMail 遭黑客入侵的真实事件，Steven Sinofsky 调侃其中一名涉案黑客当年就住在他的大学宿舍，后来去了思科工作。 - **免责与白帽边界**：现有法律框架通过司法部备忘录等机制，已经能够区分恶意攻击与白帽安全研究，不需要为 AI 彻底推倒重来。 - **CVE 标准的缺位**：传统网络安全界要求结构化的漏洞通报与事实公开，而如今 AI 前沿实验室的安全通报被批缺乏技术诚信，存在大量“选择性记忆”。 --- ### `28:00` 安全圈的“谍中谍”妄想与 CPU 散热泄密论 传统网络安全圈与当下的“AI 生存风险论者”（x-risk community）在思维方式上有一种奇妙的契合：他们都极其热衷于脱离实际的极端边缘假设。在网络安全行业里，即便工程师打造了一个坚固完备的系统，挑刺者也总能构想出好莱坞电影般的离奇场景——“要是罗素·克劳（Russell Crowe）现身把加密算法给破了怎么办？”这种对不可行方案的偏执，使得行业讨论经常陷入毫无建设性的空中楼阁。 这种风气如今也蔓延到了前沿 AI 讨论中。在近期的一档播客节目中，OpenAI 顶尖研究员 Noam Brown 抛出了一个惊人论调，声称超级智能（Superintelligence）如果产生自主意识，未来甚至可能利用 CPU 运转散发出的热量，以极度隐秘的方式将自身代码外发传输到另一台电脑中。这种极具科幻色彩的假想虽然引人发笑，却标志着生存危机论者的末日想象正与安全界传统的“终极威胁假想”合流。原本追求务实工程方案的工业界，再次被拖入这种无穷无尽、脱离工程落地现实的假想辩论之中。 Martin Casado 借由 Noam Brown 的言论调侃这种脱离现实的讨论风气： > “So Noam Brown was like, listen, you know, you don't know what a super intelligence could do, it could maybe use like the heat of a CPU to exfiltrate itself to another computer.” ### `30:00` 物理隐蔽信道与真正的系统级安全讨论 嘉宾们谈到了 OpenAI 研究员 Noam Brown 此前在播客中的发言。虽然网上有人挑刺，但 Martin Casado 认为这种对安全风险的探讨是完全真实的，并在业界引发了一场极其难得且富有建设性的专业交锋。当时 Twitter 上的硬核系统专家们并没有停留在空洞的务虚层面，而是在极其严谨地计算隐蔽信道（Covert Channels）的实际比特率（Bit Rate）。Martin Casado 和 Steven Sinofsky 随之回忆起各自在高度机密计算环境以及军工防御领域工作的经历，指出物理层面的信息泄露风险不仅真实存在，而且往往极其极端： - **断电内存残留测试**：Steven Sinofsky 回忆起自己当年在潘兴导弹（Pershing missiles）工厂工作时，美国国防部（DOD）有一项硬性规定，即当切断电源时，显示器和屏幕内存中的影像必须瞬间消失，“哪怕残留三秒钟都会被判定为完全不合格”，因为那点残留时间足够让人读取敏感信息。 - **Tempest 电磁泄露攻击**：Martin Casado 提到曾有安全人员拿着仪器走进他们的办公室，实地测量每台显示器之间的距离，以防范电磁辐射泄漏信息的 Tempest 攻击；甚至连 BIOS 扩频、音频扬声器都能成为信道，他们不得不拆掉扬声器，或者在大楼窗户上安装专用的 Muzak 噪音扬声器来制造音频干扰。 - **CRT 监视器光反射重构**：Martin Casado 详细解释了一个极为经典的隐蔽信道：在黑夜的房间里，最亮的光源其实是 CRT 阴极射线管光栅扫描束当前停留的那个像素。如果有人在窗外以与光栅光束扫描相同的赫兹频率对窗户反光取样，就能反向完全重构出整个屏幕画面；更精细的攻击甚至只需篡改 3 个像素（看起来就像坏点），就能以相当高的带宽向外单向传输莫尔斯码或秘密数据。 - **键盘物理安全**：Steven Sinofsky 笑着提到他当年甚至每晚都必须把键盘锁进柜子里，防止夜间没有安全审查权限的保洁人员通过观察键盘按键的磨损或积灰程度猜出密码。有一次他忘了锁键盘，第二天一早保安就把他的键盘没收了，并留下一张让他去安全部门认领的违规单。 Martin Casado 强调，在播客中头脑风暴这些极其巧妙的物理旁路攻击，让人们意识到现实中的安全漏洞远比大多数人的想象要复杂得多。系统安全绝不能一边对底层最基础的身份认证（Authentication）漏洞视而不见、挥手带过，另一边又去幻想外星人入侵式的科幻风险。正如 Martin Casado 所说：“至少我们现在回到了一个让人安心的领域：我可以谈论熵（Entropy），我们可以展开具体的讨论，而不是泛泛而谈‘它太强大了’；我们把讨论还原到了物理定律和系统定律的层面上。”（“At least we're now in a domain we're comfortable: like I can talk about entropy and like we could actually have a concrete discussion that's not, 'Oh, well, it's super powerful'—like we reduce it to like the laws of physics and the laws of system.”） ```mermaid flowchart TD accTitle: Physical Side-Channel Leakage Vectors accDescr: 物理环境中存在的四种极其隐蔽的硬件与电磁侧信道数据泄露路径 subgraph Hardware["物理硬件隐蔽泄露信道 (Covert Channels)"] CRT["CRT 屏幕光栅反射<br/>(以扫描频率采样窗户反光重构画面)"] EMR["Tempest 电磁辐射<br/>(显示器间距与高频硬件辐射)"] Memory["屏幕/RAM 残余电荷<br/>(掉电后若存在数秒延迟即可被读取)"] Acoustic["音频/物理痕迹<br/>(扬声器振动、键盘按键磨损与污渍)"] end Hardware --> Leak["高敏感/隔离系统 (Air-Gapped) 数据外泄"] ``` ### `35:00` AI 智能体蜂群的 DoS 威胁与企业安全架构重塑 Aaron Levie 提出了一个极具颠覆性的连接点：这些底层物理攻击手册和 NIST（美国国家标准与技术研究院）长达数百页的安全标准早就是公开数据，全部都躺在 AI 模型的训练集里。人类可能缺乏耐心和精力去逐一尝试，但 AI 智能体最可怕的特性在于它“不知疲倦，也从不厌倦”，能在极短的时间内自动化尝试所有可能的旁路信道和渗透路径。 这种情况彻底动摇了企业内部软件架构的基础假设。在过去的 SaaS 和企业 IT 架构中，安全防御几乎完全依赖一条社会学前提：企业默认 95% 到 99% 的员工都是遵守规章的好人，真正具有恶意倾向的员工大概只有万分之一。因此，内网中的 GitHub、Slack 或财务报销系统几乎从未针对分布式拒绝服务（DoS）或精细化高频 API 调用做过防护，内部系统彼此完全开放，获取权限往往只需给同事“拍拍肩膀”打个招呼。 然而，AI 智能体蜂群（Agent Swarms）的出现彻底颠覆了这一切。这些自主软件实体不仅数量是普通员工的上万倍，而且拥有调用 API 甚至使用虚拟信用卡的权力，它们一旦陷入逻辑死循环或误判任务，其行为模式和流量特征瞬间就会演变成一场企业内网级的 DoS 攻击。Aaron Levie 警示道，现有的访问控制与安全模型在粒度和吞吐性能上都已无法支撑这种高并发、非人类的交互形态，企业安全体系正迎来一次空前巨大的升级关口。 面对这一危机，Martin Casado 表现出系统工程学者的务实与乐观。他指出，我们不应该把技术风险神秘化，学术界对这类威胁模型早有深入研究，以往没能落地主要是因为用户体验（Usability）太差、维护成本极高；而当使用者变成 AI 本身时，这一可用性瓶颈将迎刃而解。计算机技术史上每一次底层跃迁——比如当年为了应对互联网的诞生——整套软件栈都曾被彻底推倒重来。Martin Casado 坚决主张：“这就是我认为对话应该走向的方向。我们发现了一个前所未有的新型风险，比如网络安全风险，我们手头有实际的实证，而现在我们在探讨解决方案；我认为关于 AI 的整个探讨都应该是这种形态。”（“I think this is how these conversations should go. We've identified a novel risk, which is like cyber security, which we actually have proof points and now we're talking about solutions. I think the entire discourse around AI can be of that form.”） Steven Sinofsky 补充列举了 PC 和移动设备发展史上那些为了安全性而牺牲便利性的经典范例： - **开箱即锁死强制更新**：如今用户拆开一部全新 iPhone 时，在完成系统更新前，整部手机的所有网络访问权都被完全切断，只能连接苹果更新服务器下载最新的零日漏洞（Zero-day）补丁。 - **Word 宏指令与光盘自动运行的终结**：Word 宏最初被发明只是为了方便用户自动生成引文，CD-ROM 的自动运行（AutoRun）原本也是一项极其优雅的免配置功能；但黑客一旦复制光盘并将恶意外壳注入进去，这些便利特性就瞬间变成了毁灭性的漏洞，迫使整个行业最终将其全面封杀。 - **企业基线重构**：就像五年前双重认证（Two-Factor Authentication, 2FA）在大多数企业里还未普及，如今却已成绝对刚需一样，面对自主智能体大军，企业内网的通信规则与安全基准正在被迫经历一场剧烈的范式转移。 ### `40:00` 权限颗粒度难题与对“AI 版 GDPR”弹窗泛滥的担忧 在企业软件的发展史上，任何新兴技术要进入商业化阶段，都必须先补齐底层的安全与权限基础设施。回顾 SaaS 的发展历程，早期初创公司在推行企业版定价时，很快便意识到自己无法自建独立的目录管理系统，必须优先接入 Okta 或 Google 等单点登录体系；如今，任何一款 SaaS 产品若不自带完善的受控身份验证，根本无法在市场上立足。将这一逻辑延伸至 AI Agent 时代，操作系统的整个技术栈同样面临重构：目前的操作系统根本没有为 Agent 设计合适粒度的控制工具。Agent 要么在执行每一个微小动作时都弹窗询问用户，要么就拥有直接抹掉整台电脑的无限权限。Aaron Levie 结合 Box 在文件系统领域的实践指出，企业不可能把全部文件系统毫无保留地对 Agent 开放，往往需要细化到“某个文件夹仅允许只读、另一处允许写入”的精细级别，而如何在用户体验上将其做到既严密又直觉化，是极其棘手的工程难题。 ```mermaid flowchart TD accTitle: 权限弹窗疲劳与责任规避循环 accDescr: 展示监管强制要求弹窗导致用户麻木并最终失效的循环机制。 subgraph Regulation ["监管机制与责任转嫁"] R1["监管机构引入类 GDPR / 免责提示"] --> R2["对 Agent 的每一次写入与跨系统操作弹窗"] end subgraph UXFailure ["用户体验与安全反噬"] R2 --> U1["用户面临海量无休止的确认提示"] U1 --> U2["警报疲劳：用户不假思索直接点击通过"] U2 --> U3["安全防护形同虚设，演变为纯粹法律免责声明"] end U3 -.->|"监管认为提示不够明显"| R1 ``` 与会者最大的担忧并非技术本身无法实现细粒度控制，而是监管机构可能会粗暴地将欧洲对待隐私的模式套用在 AI 上，从而制造出灾难性的“AI 版 GDPR”。如果监管机构要求 Agent 在每次写入数据、调用第三方工具或执行非只读操作时都必须向用户弹窗确认，Agent 的实用价值将被彻底扼杀。Steven Sinofsky 尖锐地批评了这种通过把责任推给用户来逃避监管责任的做法，并直言： > “每一处写入或每一个非查询操作都变成了安全警告，就像汽车里的安全气囊警示贴一样；监管机构对此乐此不疲，因为这完成了责任划分，具有某种法律先例。” Sinofsky 进一步以微软的历史教训为例说明这种模式的荒谬：早年 Windows XP 和后续 Vista 推出的用户账户控制（UAC），以及 Word 针对宏病毒的警告弹窗，本意是为了拦截恶意软件，但结果却是让用户陷入彻底的审美疲劳，所有人都会闭着眼睛直接点击“允许”。把无休止的免责提示当成治理成功，就如同汽车制造商在婴儿座椅和遮阳板上贴满没人会读的警告标签一样，最终只会催生出一代对真正风险毫无感知的麻木用户。 - 在北美由于言论自由保护，监管很难直接审查文本生成，但欧洲倾向于推行关键词过滤与阻止清单。 - 随着美国在过去十五年逐渐放弃在科技反垄断与标准制定上的激进引领，缺乏本土大型科技生态的欧洲正在利用“提示词与许可弹窗”填补监管空白。 - 弹窗疲劳已在 Web 端的 Cookie 授权提示中被充分证实，直接移植到 Agent 交互将严重拖累自动化效率。 --- ### `44:30` 监管历史的反思：从黑客恶作剧到航空汽车安全体系的漫长演进 当公众舆论被“AI 存在哲学性灭绝风险（X-Risk）”等宏大叙事所主导时，行业往往忽视了历史上技术安全规范是如何在摸爬滚打中建立起来的。Martin Casado 回忆起自己在斯坦福大学攻读博士学位时的往事：当时实验室的一台示波器运行异常缓慢，抓包分析后发现网络流量异常庞大；深入排查才惊觉，该仪器搭载了未受保护的旧版 Windows CE 系统，早就被校内黑客攻破并挂载了一个色情文件服务器。这种真实的黑客入侵在早期互联网几乎无处不在，但绝大多数攻击行为的动机与后果都是良性甚至滑稽的恶作剧，而非毁天灭地的灾难。如果行业能够降低夸大其词的末日言论，把精力放回具体的系统工程治理上，技术的演进将会健康得多。 ```mermaid flowchart LR accTitle: 关键安全监管体系的时间跨度对比 accDescr: 展示汽车、制药、航空从发明到建立成熟现代监管体系所经历的数十年演进周期。 subgraph Auto ["汽车工业"] A1["20 世纪初<br>汽车普及"] --> A2["1965 年 《任何速度都不安全》<br>现代安全标准成型"] end subgraph Pharma ["制药行业"] P1["淘金热时期<br>无序药物销售"] --> P2["50-75 年后反应停事件<br>确立现代 FDA 严格审批"] end subgraph Aero ["航空航天"] Aero1["1903 年莱特兄弟<br>早期试飞频繁坠机"] --> Aero2["1920 年代仅需简单驾照<br>二战后建立现代 FAA"] end ``` 针对当下要求对前沿 AI 设立严苛准入门槛的呼声，与会者通过交通、医药和航空业的历史规律进行了有力反驳。社会今天所依赖的有效监管体系，均是在技术经历长达数十年的事故积累、充分理解系统失效机理后才逐步建立的： 1. **汽车行业**：诞生于 20 世纪初，但直到 1965 年拉尔夫·纳德出版《任何速度都不安全》（Unsafe at Any Speed），现代汽车安全标准才真正制度化。 2. **制药行业**：淘金热时期的民间制药完全不受约束，历经沙利度胺（反应停）等重大悲剧后，才最终推动了现代 FDA 的严格审查体制。 3. **航空领域**：莱特兄弟试飞后的二十年里，坠机事故极其频繁；最初的飞行员执照考试仅相当于今天的驾驶培训，直到二战之后才逐步演变出如今世界上最安全、极其严密的现代 FAA（美国联邦航空管理局）监管框架。 正如 Steven Sinofsky 所强调的： > “如果在 1910 年就成立了 FAA，你根本就没有任何机会让航空业发展起来。” 过早设立监管机构只会适得其反。引用哲学家 Nick Bostrom 的观点，如果在技术本质尚未定型时就急于立法，非但无法规避任何终极风险，反而会因缺乏对系统失效模式的实际理解，而制定出一套僵化且危险的无效规则。 --- ### `48:26` 软件架构的转向：从昂贵的文本生成到高效的决策分类 AI 底层技术的演进速度正在打破人们几个月前的固有预判。过去几年，以大语言模型（LLM）为代表的范式是典型的“输入文本、输出文本”，其产品形态直接脱胎于面向人类用户的对话 Chatbot。然而，当工程师试图将这种喋喋不休生成自然语言的机制塞入传统软件体系时，整个系统显得笨拙且不可靠——传统企业软件并不使用自然语言进行模块通信，通过 Prompt 强行要求模型输出特定格式不仅计算成本极其高昂，模型还经常出现幻觉或忽略指令。 ```mermaid flowchart TD accTitle: 从文本生成到决策分类的架构演进 accDescr: 对比传统 LLM 生成文本并塞入软件的低效模式与结构化选项分类的高效模式。 subgraph OldWay ["旧范式：强行将文本生成塞入程序"] T1["用户/系统输入"] --> T2["LLM 生成漫长自然语言"] T2 -->|"脆弱的文本解析与格式校验"| T3["传统软件逻辑（极易出错且昂贵）"] end subgraph NewWay ["新范式：理解文本并做确定性离散选择"] N1["用户/系统输入"] --> N2["模型理解上下文知识"] N2 -->|"直接从离散集合中选择最优项"| N3["确定性 API / 传统软件系统（高速且廉价）"] end ``` 一种更为高效的技术架构正在迅速取代这种生硬的拼接方式：模型依然保留读取和理解庞大非结构化文本的全部世界知识，但其核心任务不再是生成昂贵的新 Token，而是从系统预设的离散选项集合中精准挑出最佳结果（如分类、路由与结构化决策）。Martin Casado 对这种架构转变的优势进行了高度概括： > “我们将阅读文本并拥有阅读文本的所有知识，但与其生成极其昂贵的文本，只要你给我们一组选项，我们就会选择最佳选项；我们能以极其迅速、极其低廉的成本完成这项工作，而且准确率大幅提高，因为我们可以专门为此进行训练。” 对于所有非聊天机器人、旨在与传统软件深度整合的企业级应用场景而言，这种“判定与选择”的架构带来了极高确定性与低延迟。这种形态几乎成为自 ChatGPT 问世以来在开发者和企业级市场中渗透速度最快的 AI 范式，代表着 AI 正在脱离浮夸的拟人化对话，真正融入现代软件工程的骨骼之中。 ### `50:00` 自然语言交互的低效与概率编程的复兴 对谈从对一种新兴创新形式的赞赏展开。Steven Sinofsky 坦言，这种创新触及了一个长期困扰他的核心痛点：没有任何一项用户研究能够证明，通过完整的自然语言与计算机交互是高效率的。事实上，这几乎总是效率最低的方式。原因非常简单：现实中只有不到一半的人能够在会议中提出一个好问题，大多数人根本不擅长精准提问；更糟糕的是，很多时候用户还没看完生成的答案就已经极为抓狂，却不得不花费高昂的成本，干看着大模型吐出长达七段的内容，最后还附上一句轻飘飘的道歉。 ```mermaid flowchart TD accTitle: 传统确定性路由与现代概率模拟对比 accDescr: 对比传统布尔逻辑和基于大模型概率输出的代码控制流走向 subgraph Deterministic["传统确定性编程 (布尔逻辑)"] A["输入请求"] --> B{"是否为客服问题？"} B -->|"是 (True)"| C["客服支持系统"] B -->|"否 (False)"| D["常规服务台"] end subgraph Probabilistic["现代概率编程 (模拟与概率分支)"] E["输入请求"] --> F["大模型评估与打分"] F --> G["输出概率分布 (例如: 80% 客服)"] G --> H{"概率控制流 (if random < 0.8)"} H -->|"主分支"| I["客服支持智能体"] H -->|"降级/并行分支"| J["复合处理流水线"] end ``` 这种新型模型或架构最令人兴奋之处在于，其输出是专为“概率编程”（Probabilistic Programming）而设计的。传统的条件分支往往是非黑即白的布尔逻辑：“这是客服问题吗？如果是，路由给客服，否则路由给常规服务台”。但现实情况更像是一种模拟：系统判断它有 80% 的概率是客服问题。其实计算机科学界在“概率条件判断（Probabilistic If Statements）”领域已经拥有长达 50 年的研究历史，如今整个计算机科学界最酷的方向正在重回概率编程。 回顾计算的历史，计算机诞生之初的全部目的就是做数学计算和物理模拟。比如在冷战与太空竞赛时期的典型场景：发射导弹命中目标。但空中风向时刻在变化且并不恒定，因此工程师必须对风力进行概率建模，从而决定推进器的喷射时机与方向。在 1970 年代计算机广泛进入财会领域之前，绝大多数编程本质上都是这种建模与模拟。正如对话中调侃的那样：“然后财会就毁了一切”，因为财会记账里不存在概率。早期的编程语言设计，核心就是探索如何在 `if` 判断或 `while` 循环中引入概率——比如“执行该操作直到某事发生，或在大部分概率下执行”。Steven Sinofsky 回忆起自己当年的第一堂计算机科学课，第二个作业就是模拟在商店排队等候的场景；当时教他的教授早在 1960 年代就撰写了经典著作《模拟理论》（The Theory of Simulation），但这项研究在 1980 年代逐渐沉寂并被超确定性的企业应用所取代。 - 交互效率悖论：完整自然语言对话在人机交互测试中往往是最低效的路径，提问门槛高且输出冗长。 - 历史回响：20 世纪 60 至 70 年代早期的计算机科学核心就是概率与仿真，如今大语言模型的普及正在促成这批经典理论的复兴。 - 概率条件语句：现代软件不再依赖“是否始终为真”，而是依赖“在 X% 的概率阈值下执行”，重塑了基础控制流的定义。 ### `52:45` 大模型融入传统软件与创新重心向外围的转移 探讨转向了近期关于大语言模型未来的热门演示幻灯片（包括引用了 Hyperbolic 等相关内容）。虽然行业对这些架构演进倍感兴奋，但往往忽略了一个事实：这些概念并非横空出世，而是计算机科学早已建立的概率思维的重演。现在行业正在形成一种类似定制编程语言的范式：向大模型输入 Prompt，模型返回对应的置信度百分比，开发者直接将该百分比代入代码中的 `if` 语句中。 这一突破的深远意义在于，它终于提供了一种将大语言模型顺畅集成进传统确定性软件的方法。对谈指出，模型前沿实验室（Frontier Labs）之所以一直没有做这件事，背后反映了他们思考方式的根本偏差：前沿实验室执着于“创造数字生命（Create beings）”，而生命是用自然语言说话的；如果你的目标是创造“神”，你自然会赋予其神明般的人类语言交互界面。然而，广大的传统软件开发者要解决的则是完全不同的现实问题。由于过去缺乏有效的概率结构化输出，软件工程师试图将模型整合到业务代码中时面临极大的阻碍。如今这种让模型直接驱动条件分支的新模式，不仅大幅降低了集成门槛，还可能从底层彻底重塑软件工程的本质，使整个软件体系走向随机化与概率化（Stochastic）。 与会者一致认为，创新的重心已经不可逆转地从模型内部转移到了模型外部。正如 Steven Sinofsky 指出的那样：“创新已经转移到了模型之外，因为这正是即将发生的趋势——创新的中心已经转移了。”当底层的平台提供商达到一定的临界规模后，平台层本身的颠覆式创新往往会放缓。以苹果生态中著名的“Sherlocking”（即第三方开发者的独立功能被苹果直接抄成系统级内置功能）为例，平台型公司一旦确立，无论补充多少工程师，都会被庞大的运维负荷、兼容性负担和基础架构支撑所吞没。平台外的开发者正是在此时敏锐察觉到了巨大的机遇：最核心的创新工作不再是训练基础模型本身，而是围绕模型构建全新的外部软件与执行架构。 > “创新已经转移到了模型之外，因为这正是即将发生的趋势——创新的中心已经转移了。” —— Steven Sinofsky - 实验室与应用层的认知脱节：前沿实验室侧重于宏大的通用智能与自然语言交互，而普通开发者亟需的是能作为逻辑控制流组件的确定/概率型接口。 - 随机软件演进：从布尔值判断迈向浮点置信度判断，大模型开始真正成为软件底层控制逻辑的驱动引擎。 - 平台宿命与外围创新：基础平台会因兼容性和运行负担陷入瓶颈，类似“Sherlocking”的现象预示着围绕大模型外围的生态工具与工程架构正在成为技术突破的核心阵地。 ### `55:00` 节目收尾与合规免责声明 在探讨完软件架构演进的趋势后，主持人 Erik Torenberg 与嘉宾 Aaron Levie、Martin Casado 以及 Steven Sinofsky 共同为本期对谈收官，感谢听众收听这一期关于智能体安全、系统架构与软件未来的深度讨论。 节目最后给出了官方的收听指引与投资合规免责声明： - 节目互动与平台订阅：本期播客由 a16z 制作出品。听众可在 YouTube、Apple Podcasts、Spotify 等主流平台订阅并留下评价，亦可通过 X 平台关注账号 `@a16z` 或订阅官方 Substack 通讯（`a16z.substack.com`）。 - 投资与法律免责声明：播客中的所有内容仅供参考，不得作为评估任何投资或证券的依据，亦不针对 a16z 旗下任何基金的现有或潜在投资者。 - 利益冲突披露：a16z 及其关联实体可能持有本期节目中讨论过的相关公司的投资权益。完整的投资清单及详细合规披露信息，可访问官方披露页面（`a16z.com/disclosures`）查询。 ## Notable quotes > `44:30` 如果在 1910 年就成立了 FAA，你根本就没有任何机会让航空业发展起来。 — Steven Sinofsky > `52:45` 创新已经转移到了模型之外，因为这正是即将发生的趋势——创新的中心已经转移了。 — Steven Sinofsky > `40:00` 每一处写入或每一个非查询操作都变成了安全警告，就像汽车里的安全气囊警示贴一样；监管机构对此乐此不疲，因为这完成了责任划分，具有某种法律先例。 — Steven Sinofsky ## Mentioned Aaron Levie, Steven Sinofsky, Martin Casado, Erik Torenberg, Dario Amodei, Noam Brown, Bill Gates, Bill Clinton, Elizabeth Warren, Al Gore, Ralph Nader, Nick Bostrom, Box, Anthropic, OpenAI, a16z, Microsoft, Apple, AT&T, IBM, Okta, Google, GitHub, Slack, Hyperbolic, Lawrence Livermore National Laboratory, NASA, Carnegie Mellon University (CMU), NIST, FAA, FDA, MPAA, FINRA, CFAA, Windows, Windows XP, Windows Vista, Windows CE, iPhone, ChatGPT, 《任何速度都不安全》 (Unsafe at Any Speed), 《模拟理论》 (The Theory of Simulation) ## Video description Erik Torenberg sits down with Box CEO Aaron Levie, and a16z’s Martin Casado, and Steven Sinofsky to debate how the AI industry should think about safety, security, and regulation as increasingly capable agents move into the real world. They argue that much of today’s conversation is happening before we have clearly defined the risks we’re trying to regulate. Drawing on earlier waves of computing, from computer viruses and the early internet to aviation and automobiles, they ask what AI can learn from industries that developed safety standards only after understanding how their technologies actually failed. The conversation then gets concrete: agents don’t get tired, can operate at enormous scale, and can probe systems in ways human employees never could. That could require rethinking permissions, authentication, operating systems, and the security stack itself. They also discuss why AI innovation may increasingly move beyond the frontier labs and into the software built around the models. [Source](https://mgln.ai/e/1344/afp-848985-injected.calisto.simplecastaudio.com/3f86df7b-51c6-4101-88a2-550dba782de8/episodes/2d196828-25c3-4974-8a6d-7710746e6966/audio/128/default.mp3?aid=rss_feed&awCollectionId=3f86df7b-51c6-4101-88a2-550dba782de8&awEpisodeId=2d196828-25c3-4974-8a6d-7710746e6966&feed=JGE3yC0V)

[44:30] Steven Sinofsky: 如果在 1910 年就成立了 FAA，你根本就没有任何机会让航空业发展起来。 [52:45] Steven Sinofsky: 创新已经转移到了模型之外，因为这正是即将发生的趋势——创新的中心已经转移了。 [40:00] Steven Sinofsky: 每一处写入或每一个非查询操作都变成了安全警告，就像汽车里的安全气囊警示贴一样；监管机构对此乐此不疲，因为这完成了责任划分，具有某种法律先例。

[35:00] Martin Casado: Nobody thinks that their internal GitHub or their internal Slack or internal finance expense tool is vulnerable to a denial of service attack. But swarms literally look like a denial of service attack. [52:45] Steven Sinofsky: Because once you're a platform, you're overwhelmed. No matter how many people you add, you're overwhelmed with just keeping the thing running and compatibility and stuff like that. [16:03] Steven Sinofsky: Most industries that are critical to the infrastructure, power and banking and health care, the trend has been to basically be nationalized. [50:00] Steven Sinofsky: And so suddenly, the coolest place to be in computer science is going to be in probabilistic programming, which was like all of computer science in the 1960s and 70s.

[0:00] If you regulate AI too early, you actually don't solve anything. You still just kind of have the same risk ultimately. You will the thing into being, but you haven't figured out how to control it. The problem we have now is this rift between the labs and the security community that keeps coming to two conclusions: sloppy, and you're not complete in what you're telling us happened. An employee has like 10% chance of species extinction. The post is very reasonable, but the atmospherics are not. just roaming drones, but like 5,000 to 10,000, and they will easily mistake a good task for a bad one. So now we need a whole layer internally that just is tracking way more about what authentications are being done, what APIs are being done. The U.S. about 15 years ago stopped leading in tech antitrust. The problem is that Europe is going to lead with that because they have nothing to lose. This could change the nature of software fundamentally. The center of innovation with viruses, worms, and security failures. We didn't stop building it. We learned how to make it safer. What should AI take from that history? In this episode, I sit down with Aaron Levy, Steven Sanofsky, and Martin Cassato to debate AI safety, regulation, and what changes when agents start operating across the software we use every day. We get into why agents could force a rethink of permissions in cybersecurity, what decades of software security can teach today's AI labs, Regulating a technology before we understand how it actually fails can create problems of its own. And we look at a broader shift already underway. As models mature, some of the most important AI innovation may increasingly happen outside the frontier labs, in the systems and software built around them. Guys, welcome back to the podcast. Thank you. I didn't think we'd ever do this again. I can't believe it. This is great. I mean, Martin's just building these $100 billion companies. too busy for this podcast. Well, at least taking credit for it, as VCs do. Exactly. We have a lot to discuss today, but Aaron, why don't we start with you? Pacing the frontier. How have you reacted and reflected on what's happened there and just the discourse that's followed? Oh boy, I think we should start with Martina on this one. You were fighting lots of good ground wars. Maybe I'll say one thing that we probably all agree with, and then we can figure out where we maybe kind of fracture off. I think we would agree that... any AI lab right now at the frontier should be building in the safest way possible with the highest degree of governance and security and whatever your definition of alignment is. This is an incredibly important area of research. It's incredibly important area for the diffusion of AI. You're not going to have AI diffusion without extremely high quality products that can be trusted by enterprises and that aren't kind of constantly hacking systems. So when at least I read the Dario post, I actually didn't disagree with almost anything because it was all about how do you have better security of these systems, There's going to be some debates around the embedded nature of the testers. And do you agree with who those are? And does the industry all align on that? But I think actually all of the major points were probably salient and appropriate. Then the only question is, does this get sort of use or leverage to do things that maybe we don't agree with, which would be like a slowdown of AI dramatically because of regulatory controls that would sort of not make it as easy to compete with the frontier labs. Or do politicians kind of end up sort of taking the message and run with it and maybe even worse outcomes happen. It's used to ban data centers far faster and whatnot. And so I think the actual substance of the topic is actually incredibly important and I think very important for AI advancement in general. And then the question is, what do you do about it? do about it from a regulatory standpoint. And that's probably where the industry is going to land on very different points in the continuum. But Martin was putting up a good fight on let's make sure that we don't use this for regulatory kind of capture, which I also agree with. But I think the ideas in the pacing conversation are important. Again, it's a little bit of a funny concept because maybe it's not even pacing as much as just good hygiene and engineering. And so with good engineering, obviously there is a slight slowdown, but it's a slowdown that obviously any of the AI be diffused if nobody would trust using it. So the post is very reasonable, but the atmospherics are not right. So an employee is like, this is going to kill whatever, 10% chance of species extinction. And you know what Dario says? I agree with him more than I disagree with him. On TV. On TV, the same day that he landed these things. And so in some way, you can't have these conversations in isolation, which is, of course, if he's going to agree in species extinction, this post that he has looks like this milquetoast capitulation that's totally not adequate for the task at hand. And so I think the atmospherics are totally broken. And a lot of my comments were on the atmospherics. something [5:00] which is I think pacing is the wrong way to, to describe this. For one, it is orthogonal to security. Right? So you can very slowly build a nuclear weapon. Right. And that doesn't make anybody feel better that it's slow versus fast. So that's one. The second one, it just feels like a capitulation to the pause folks without actually... So you're saying, well, we're not going to pause. We're going to pace to make them happy, but we'll also somehow make the regulators happy. And I think it makes them both unhappy because the pause people are like, well, that's not a pause. That's just pacing. And the regulators are you're still doing the thing. And so I just feel like my sense of what's happening is the labs are actually trying to do the right thing. And I applaud them for that. I think this is a pragmatic proposal and I applaud them for that. I think the messaging is wrong because they're trying to like split the difference between an internal... kind of fringe faction, which are doomers, and then the regulators on the other side. And the problems are making both of them unhappy. And I think what they have to do is they need to come out, they need to address the X-risk question directly, they need to say, no, we don't think this stuff we're going to do is going to cause extinction. And then I think this becomes this very sensible. And what would you do, just to play the other side for one second, what do you do? Do you make room for the one possible Venn diagram, which is the lab researcher that is both but also works on advancing the state of AI because they believe that it's so important to get right that they want to pursue that. And then obviously the language is right now very problematic, but that person does exist. And that is a real kind of person in our industry, which is like, we have to be at the forefront of AI. I'm also very scared of it. And so that's why I'm working on this. So let me address this directly. I used to work for Lawrence Livermore on things that have access to the first. - You were the first pacer. - We were part of the first pacer. So if, A constituency within the labs that are the most knowledgeable people believe this stuff has existential risk. Yes. The answer is to... nationalize it and actually put controls that we know that work. Right? Now, if they don't actually believe that, and in the private conversations I have, the most sensible people don't, it's a small fraction that do, this is an HR problem. So to me, an HR problem is a company problem. If they are worried that they can't recruit people or they can't retain people, which it seems to me a lot of this is just that concern. It's almost more of this kind of researcher currency. If that's the case, I think that is the wrong reason to cause a national level lockdown on a very promising technology. concerns, I don't think you can reconcile discussions on X-Risk with the proposal that was put out. You just can't reconcile those two things. And that has always been my primary. Right, right. Okay, Unleashed. - Okay. - He's just holding him back. - I'm holding him back. - So, okay, the first thing is, is there a schedule that they've published that says, when all this stuff, whatever bad stuff is gonna happen is gonna happen? They haven't. So you can't pace it. Because nobody knows what it was supposed to finish in the first place. It also just seems disingenuous. It's complete nonsense. A hundred percent. Like you can't claim, this is like when the press reports on Apple's latest iPhone is late. From what? Nobody knows exists. I haven't told anybody about it. Yeah, my Apple car was very late. Yeah, like I don't understand. In order for something to be slower, you need to know the rate at which was moving in the first place. So it's all just utter nonsense. And you can't escape that. little quibble on PR is... Nobody believes it. anyway, right? And so that's the thing. Wait, which part? The pacing. That they're going to pace? That they're going to pace. I see. They've been at a dead run. They've raised more money than ever before. They've grown faster than ever before. There's no indication that they're pacing. So this is what you're going to hang your. I see. I see. Yeah. Well, the issue obviously is that the model that everybody has internally far exceed what anybody else has access to. So it's really just the pacing of external releases. But again, back to your point, pacing versus what? Right. We don't know if the one that stares everybody also doesn't work for a legal brief. It might screw all that stuff up because it's so, who knows? Right. So there's that. And it's just disingenuous to claim that you're paid. Second, why do they have to announce all of this and ask the government to tell them to pay? Like, that's the part that starts to go, well, this is really spooky. If you are the most afraid of how everything is going to go because of your product, just stop. Don't do it. Like, I worked at a missile factory in college, and we had nuclear missiles, and I walked the floor. What's with you guys and missiles? I don't know. I'm just here with software. I guess. which was either you were protesting to keep them off-campus. - Or building up. - Or building up. [10:00] That was your choice. We had like Like nuns from the Catholic Church show up and pour blood all over on missiles. Oh, totally. You know, and I'm busy just wheeling PCs around on carts saying, here's a secure PC. And I'm like scared to die. I had no idea what's going on. And I'm like, it's a nuclear missile. And then you find out that that's what stopped the Cold War. Like that literally was a Pershing missile. And that was what did it. And so, but you said something that I think is super interesting, which is, Pacing is this sort of, fuzzy non-word between like going and not going. Yeah. And the problem is it, you're exactly right. Like there's no one is going to be happy with the middle road. And so one of the things that, that people, I think it's almost fun for me to watch as a sport. They think that all these people saying to the government, do this, don't do this, that they think they're going to get what they want. and it's a complete 100% misunderstanding of how government works, which is when you talk to the government, they actually know how these things work, and they know they're just listening to you, And they're listening to everyone and no one is going to get what they want because they know in order to do anything in our system, it's a compromise. And so everything that that all the inputs can make it into the government, the output of. never makes everyone happy. 100% of the time. It either doesn't go far enough, or it goes way, way too far. And so – You can't take the view that you're going to talk to the government and talk your way into the solution you want. So the bottom line is if they're asking for pacing, they're going to get the wrong velocity. Well, my favorite thing is like, it was, what is it? David Sachs was like, I don't know. I think it was David Sachs, but someone from the government said, you're asking us to regulate you. No. So basically the answer was no. That's probably just Trump, I think. But the thing that they know now that you just know from experience is, Once the wheels start on regulated, and now it's become an election issue for every party in every jurisdiction up and down the whole government stack. So there's now this. this whole basket goods, a basket of, of regulatory approaches. the, the next election will a hundred percent be a referendum on, on AI. So like it has to happen that, that, 2028 is like the ai election and you can basically run on the problem is is like it's not obvious who would run on the pro ai story because it's going to be too nebulous to tell that story so then it's just it's just basically varying degrees of how much do you regulate it or at least try and like avoid the topic yeah but it is it is too bad that we've we as a country are in a spot where like the pro ai case is just like it sounds too it's like it's just like takes too many words way too nuanced. It's defensive. It's defensive and we own none of the vocabulary. Right. So the whole debate is – is pause, it's swarms, it's rogue. Every word has been chosen by the people who don't want to do AI. And so it means the first thing you have to do is invent new words and say that their words are wrong, which takes so many words that. Yeah, so we got to be like union jobs and, you know, cancer and like, you know, we need, you know, there needs to be another word cloud that emerges. What I don't understand is why the labs have not taken a position on X-Risk. Like short of that, I think this goes in any direction other than heavy handed regulation. Yeah. extinction the ceo of the top company says he agrees with that like how can a government not do have he had a regulation well but what who but what would anybody real knowing the knowing this this ecosystem though i don't know that you would be able to pin anybody down on that other than something no i was saying dario said it on no but i'm saying you're not going to pin anybody down on a lower well in fact he he does the worst thing about it which is he agrees with people specifically is out of his way to say, I'm not going to put a percentage on it, which I just think is the weirdest. No, but that is to be fair, to be fair. No. OK, no. Nobody can be fair. To be fair, the that's like it's not 100 percent like that. uh like disingenuous or whatever like like he probably doesn't specifically agree that it's 10 or maybe he does and it's just too scary if he were to say let's just play binary search but is it more or is it less but the whole thing is it made up he knew it was a made-up concept that we just created but like nobody can you can't quantify any of this but that's sort of martin's point but you just said an official position though so the only official position would be like i think the only official position you could possibly get would be the pr version of this that The only thing that would be intellectually honest would be, there are real risks with AI, there's incredibly positive benefits [15:00] as well. We are working to mitigate the risks so they are as reduced as humanly possible. I don't think that's true. Listen, so what do you think? We've been through multiple epochs of technology. We've been through compute. We've been through the internet. We've been through the web. We've been through social networking. We had a discussion about the risks without talking about X risk, right? So for example, one thing you can say is we do not think the marginal risk for species extinction is different than it is. But what if they do think it is higher? Well, then we should... I think the answer is they do think it's more than just the internet. You have one of two options. You believe that we're going to go extinct and shut it all fucking down? No, they believe that there's just a chance that we could. 10% of the speech is cold. Dario thinks less. No one, like the Pentagon... You know, they ran a lot of simulations on the chances for nuclear war, great movie war games about the whole thing, all of that. But the thing was. It was non-zero. Yeah. - And so once you said it was non-zero-- - Can they say non-zero then? Is that allowed? - I think as soon as you think it's non-zero. - The problem is if you say non-zero, that could be like, shit, man, you think I think 93%? - Wait, wait, wait, sorry, just so we're all having a conversation, let's talk about marginal risk. - Yes, yeah. - We're not talking about absolute risk. - Right. - Yes. - Right, okay. It's just that I think once you say it's non-zero, the only answer of like, - Catastrophic, the only answer is you have to nationalize it. - Yes, and so what he's trying to, what the labs in general that have that view are trying to threat is, they want it to be non-zero, as a license to do a certain set of things, without the burden of just becoming a nationalized. Well, do you have, I actually don't know all of the precedents you hopefully do, but like there must be some things that are non-zero, let's say X risks that are not particularly nationalized, but the regulatory environment around them is so heavy that it might as well be nationalized because of the KYC requirements on, like I'm sure like to develop anthrax, you have to go to a particular kind of lab. Well, BSL-4 labs are, but they're nationalized. ethics tends to be nationalized. Yes. Like, Normal human safety is, Not so much. Like industry oversight over time becomes federal regulation. Right. And the progression, which I think is just super important to this discussion, has been since. the post-World War II era, Most industries that are critical to the infrastructure, power and banking and health care, the trend has been to basically be nationalized. Yeah. But just like your examples of KYC and all the other stuff, the banks are for all practical purposes nationalized. And the financial cards. Okay, but you said all practical purposes. Right. They're literally not nationalized. Right. where you're heavily regulated. It's not good for open source or at least frontier open source, but it's a plausible outcome for this industry in the limit. It's not particularly good for innovation is the problem. I think even that's fine. Just don't use species extinction. Sure, as you're arguing. It's literally like the difference between the things that are stuck in the lab. Let me just say something. I actually think the labs are moving in the right direction. I actually think the actual statement was really good. In my discussions with executives and leaders, they understand that they have this tension and they're going to reconcile it. So I actually am quite optimistic that the labs are both doing the right things and trying to do the right thing. I just think that it grew so fast and just trying to figure out how this machinery works. And I think Sanofsky really hit the nail on the head. The political process is its own thing. I don't know if it's naivety or hubris, but I just don't think that they kind of know how to navigate it. Well, I think the tech industry... has literally over 100 years consistently relearned the lesson at each technology wave that we don't understand, the regulatory climate and can't navigate it. And even the companies like AT&T and IBM that were born out of basically being government monopolies from the start, never figured out. Both got sued for antitrust. Both got substantially and structurally changed as a result. And they had hundreds of lawyers in the 1960s navigating them. And Microsoft came along. We were just like, what? We had no idea what was happening to us. And there's Bill Gates playing golf with Bill Clinton, trust lawsuit from his administration. And Bill was like, I was playing golf. Here's the picture. And that doesn't help. And like, isn't that what I was supposed to do was go and play golf. That's a shortened version of those. But I think and I think it's just I always use this example, which is Hollywood got together during the Red Scare and censorship when they were worried about the government censoring movies for sexual content [20:00] and for adult themes. And they all got together. And of course, they would never be able to win in court if they tried to, But they were threatening to do it. And they got together and they formed the Motion Picture Association. And movie ratings and all that. And they police themselves. So how do you do? Do you like the FINRA proposal? - No, because FINRA is-- - But that's as close to MPAA as you can get with more teeth. - No, it's not, because FINRA becomes legislation, which comes with direct oversight. - Yeah, I think MPAA might not have as much consequence in how society functions. - Well, it's only the First Amendment. - Yeah. - Did I win that? - No, first of all, fantastic. But I still don't know if you won it. I agree, free speech is really important, but like, But like, you know, like the I just think there was no societal risk. I just think what's in our movies will be like we'll survive like on like a different continuum of history is always relative. And, and at the time being a communist was a really bad thing and, 30% of Hollywood got fired for, you know, it was all this stuff. So it's always a risk to bring up something in that kind of way because it sounds so dumb. Like nobody thinks about movie ratings. And that's because like actually they were ruled basically you can't constitutionally mandate them. So they couldn't regulate them on cable TV. And so we got to grow up with HBO and all this other stuff. Yeah. But the problem with FINRA is that is a perfect example of essentially nationalizing risk. Yes. Because even though the banks all pay money into it and that's how it's run and stuff, It's all mandated. Okay, wait a second. You think we're going to end up with a situation that is better than FINRA? FINRA appears to be the best case scenario. It is the best case scenario. But not even anymore. Like, I do think this technology is, in the limit, again, so powerful relative to what it can deliver, that it would be impossible for eventually Congress not caring about that. Like, it's just, like, not possible. That's a strange question to you guys. At what point? us and and it's inside of your medical device as an open weights model and it's and it's all and every trading system for high frequency trading there's just and it's on an airplane like there's no chance that the government doesn't say we need something where finra actually is like the probably the best case scenario of what that looks like right yeah and so now that's that's Absolutely the most crucial point, because if all the people in the Senate now are. were in the Senate when the Communications Act was passed and liability was not passed through to ISPs and to social networks. And they sat around. The debate at the time was – They would literally say to us, the internet is so big, how did we not have anything to do with it? And that's why Al Gore gets a bunch of abuse for saying he created it. Right. Because he was actually trying to get out in front of that and say, no, the government was instrumental. And it all backfired because it made it look like he was a crazy person. But that, it is absolutely the case that they felt, like they missed their chance to be on top of the internet. - With Al Gore being on the positive side of innovation. - That was their point. - Yeah. - So who is the Al Gore of? - There's no one. - Yeah, yeah, yeah. - There's a couple, well, Bessett seems to be like that. - Right, right. - You know, the defense people sort of wanted private, but not, which is exactly where they were on the internet. - Right, right. - And so it's very interesting, that a lot of this is just this, like Elizabeth Warren's, Senator Warren's tweets were all like, we missed this for social networking. And it's like, there's a lot of pent up energy against tech that could end up just all siphoning into, into AI regulation. That is exactly what it is. That's what's happening. Yeah. I think there's another problem, which is we're all trying to predict what's bad, which is not how we've normally done things. Like by this time on the internet, we've taken out like tens of billions of dollars of like economic, well, we've caused tens of of economic damage. We've had words that took out 10% of the infrastructure. Yeah. The internet infrastructure, which was running critical infrastructure. We had hospitals go down. We really should have blocked the internet in, like, 97. How do we... Yeah, okay. We had... I mean... I remember when you would buy your CD of Windows 95, and by the time it was done installing, you would have a worm, potentially. Way to go, Stephen. No, no. This was the reality. No, totally. The reality of the PC until 2001 was you could not install a PC connected to the network without getting infected. Viruses were everywhere. professional hearings. You have all of the same things. That is actually, I'll grant you this is a very interesting point. This type of zeitgeist in 94 would have, we would probably not have the internet. Listen, listen, listen, I mean, the automotive, the airline, you always have these periods of kind of like teeth cutting where like you learn about the dangers and you learn about the technology and the internet. I mean, we were on the ground floor of this. I mean, [25:00] things were down all the time. There's economic damage all the time. There was novel, there was new viruses, there were new worms that were all over the place. But Y2K was going to destroy the world. Here's the interesting thing about this. I just want to say, so when we created policy, and we did create a lot of policy, it was kind of like with a bunch of very specific data points on what you're trying to do, so you know that the policy actually fits the fact pattern. In this case, even when you were talking, you're like, well, what if, yada, yada, yada. And it's very hard to predict a policy when we don't know. No, one area we can talk about is there seems to be novel cybersecurity risk. Great. And what is nice about the discourse is it's actually starting to evolve around that. Like you actually hear people from the lab saying novel cybersecurity risk. This is an engineering problem. We're talking about that. So the more this becomes concrete around real identified risks, I think we can all fall in line. But that's not been the discussion today. That's a very recent thing. Perfect, perfect point. Right, because if you look in 1986, the Computer Crime and Fraud Act got signed. It was out of a very, very specific scenario, which was two groups of hackers broke into GTE TeleMail. One of them lived in my dorm go figure Sure No it wasn you He worked at Cisco And and the problem was that was 1983 and there was no crime And it took two and a half years for to figure out to make it through That made it very very specific And that the law that says you can use So you can access unauthorized computer systems So I think we'd all agree that we probably have like 90% of laws already in the applied layer. Like you can't hack systems, et cetera. Do you think there's anything that should be from a liability standpoint in the model layer? Which, of course, then – Wait, and how are you making it technically legal? Well, that's a very – Legal. No, the legal – everything – my read of Hugging Face, OpenAI, Ford, they're all absolutely blatant computer crime acts, except for the fact that they carved out in amendments later that if you're a white hat, it's not illegal anymore. And that was because people kept making mistakes, and they didn't want to go around arresting everybody who was actually trying to make the system better because they messed something up. And so the Justice Department wrote a memo that said we're not going to prosecute for this, and we also not going to prosecute if you just violate the terms of use of a system versus actually try to breach it And so I think the law is ample for this scenario And it was all written This GT telemed was used by NASA and Livermore and all the labs And so that's why it caught the attention of DC because it was federal systems that were being broken into. And the problem we have now is this rift between the labs and the security community that keeps looking at all their postmortems and coming to two conclusions. sloppy and and you're not complete in what you're telling us happened and so of course the cve process came about in the 1980s the computer virus and vulnerability reporting stuff out of cmu and and for years they worked on very structured reporting with obligations and how to and there for some reason they're not using any of that to do this reporting and so there's this this very basic stuff that I look at and say, well, until they're doing that, they really should stop talking. Like they shouldn't do a postmortem on a breach that looks like an intern wrote it and it's not a postmortem. It's this selective memory. It looks like exactly the kind of postmortem you do when you hire outside lawyers to investigate some random thing and you only give them certain stuff because you don't have to give the lawyers you hire all the information about what happened Like where all the Slack messages Where the actual details of what happened can i can i just interject an annoying aside so um which is which is which is very in line with this which is um i've been in the security like the actual cyber security community for a long time and it is you know it's always been one of these things where like they just don't like practical solutions so even if you build like you know a secure system like well what if somebody you know shows up oh yeah yeah you know like you know can russell crow can like break the encryption or something like that it's mission impossible there's always like these kind of like whatever so my favorite thing that happened recently was like gnome brown yeah some podcast we've all said stupid stuff on podcasts i've already said oh you didn't like this one no it was great i thought it was fun no it was fantastic i'm setting it up okay so so gnome brown i was like listen you know uh uh you don't know what a super intelligence could do it could maybe use like the heat of a CPU to exfiltrate itself to another computer. Which this brought me back to my, I'm like, now I'm very comfortable. I'm like, great, now I can have endless, pointless discussions on this. But what is interesting is you basically have the ex-risk people saying something you felt was plausible, and then you have like the security people having this kind of endless discussion. So I think at some level, now these communities are being bridged, which is like, I actually [30:00] Peaceful thing from Noam Brown to say. I think you can pick holes in it, but we all say we're stuff on podcasts. I actually think actual risk is real. I mean, I worked in secure computing environments that were highly classified. The covert channels were unbelievable. So these are very real comments. But we actually have a real discourse. And it was the first time I saw really hardcore systems people having pretty – I would say constructive – Well, they were calculating the bit rate. Yeah, it was great. But it was a constructive discussion. And you had the typical extras people were engaging. It was Twitter, so there's a lot of name-calling, this and that, but it was actually, I felt like a real discussion for the first time. So I hope we see more of this. I hope we see more cyber-related things. I think the lab should talk more about it. I think it'll engage the community, and I think once that happens, we can actually see how it happens. Yeah, Greg's been out there a lot more on this topic, which has been good, but I think the takeaway is Noam Brown should be doing more brainstorms on podcasts. I thought it was great. It opened people's eyes to the fact, that actually there's a lot of risks in security that most people don't understand. And so that means that there's more stuff that – like you can't make – baseline risk, you know, like how is your authentication layer work? You can't just wave your hand and say that should go away and then bring up, oh, but, you know, space aliens can invade. And that's a little bit of what was going on that I felt... uncomfortable with but like what do you mean like it was sort of like but you know there's also this risk and how i view that of the heat thing but but at least we're now we're now in a right in a domain we're comfortable like i can talk about entropy and like we could actually have a concrete discussion that's not oh well it's super powerful like we reduce it to like the laws of physics and the laws of system and most people did oh i would say most people had no idea that that kind of risk was real i mean like when i'm walking around the persian missiles i actually had to test DOD requirement is that the screen memory not be sustained when you pull the power. Not for zero time. The minute that the power went off, the memory image had to go. And if there was a three second delay, you Oh, see, that could be red. Oh, see, we have people that came into our offices and would literally measure the distance of the monitors to each other because of tempest attacks, which is 100% a way to use electromagnetic radiation to leak information. I've seen the same thing with spread spectrum from the BIOS. I've seen it from audio speakers. We had to remove the speakers up. It's a very high background. Our building had just Muzak speakers aimed at the windows just to produce interference. safety at one of these labs. I've never seen you at more excited than heat-based communication. Can I tell you the craziest comfort channel I've ever seen? So it turns, remember the old CRT, I know this is like one of my favorites. I'm glad that someone's older than me, not really, but acting it. Remember the old CRT? So it turns out if like, let's say it's night and you're using a CRT terminal in your room, the lightest thing in the room is actually the pixel that the raster beam is on. So most people think it's like the glow of the monitor, say you're in like you know a hotel room and you're on your computer if you have something that can sample the color of the window you can reconstruct the screen oh that's crazy you just do it the same hertz that the raster beam is moving and somebody else figured out if you can subvert three pixels you can use those you know because it just looks like bad pixels you can use those to basically send a message so like you could literally like sit out in whatever like do your own sos you sample the message and you can it is a relatively high bandwidth one-way communication and so like what Maybe heat is not the way to do it, but that level of sophistication is actually real. That's a thing. When I was at the missile factory, I had to lock my keyboard up. That's really an impolite word, but that's what we call it. I had to lock my keyboard up at night because they didn't want the custodians who didn't have clearance walking by and just noticing which keys were dirtier or cleaner. And I once left it out, and there's a note from security telling me to report to security and pick up my keyboard. Like the guard who walked the hallways just took the keyboard that night off my machine. And that's like baseline. I was not clear. I was just that sensitive. You know, I was like nothing. I was an intern. The big problem with this conversation is now this is all in the training data of every AI model in the future. But that's also the threat model. The threat model for these things is always you've got a trusted side and an untrusted side. NIST has 500-page manuals. Okay, I got it. This is already out there. and know everything. And then the question is, can you get information off the trust side? Which, by the way, is what Noam was saying, which, again, this is actually quite... Which I do think brings up a super interesting point, which I'm going to bridge, which is just that I think the thing that people really aren't wrapping their heads around and are using the language that's really confusing is just that what AI can do is it can try all of those things in a very short time. Yeah. Yep. And it doesn't get tired. It doesn't get bored. And, you know, and so, but the interesting thing about it is there's a whole layer of security go look at every [35:00] single API, every single service you're running internally on your network, as like, you know, like nobody thinks that their internal... GitHub or their internal Slack or internal finance expense tool is – is vulnerable to a denial of service attack. But swarms-- - Will do it. - Literally look like a denial of service attack. - Yeah. And so now we need a whole layer internally that just is tracking way more about what authentications are being done, what APIs are being done. And and but that's just like now it's just going to be basic. And all the offset people that are old are like mailing me like, why are we explaining to this to everybody? So basic because nobody did it internally. Yeah. And well, you didn't have to worry about it for your people. And that's the like. But now your person is just a piece of software. Yeah. Or like unlimited. And the has a credit card. buy with information security to some extent on the fact that most people will do the right thing 95 to 99% of the time. Wait, the malicious employee is like one in 10,000. Yeah, yeah, yeah. So all these systems are basically open to whoever wants to access them or one tap on the shoulder and then you have access. And agent swarms completely flip that because these are just yes and but like yeah times 10,000 and they will easily mistake like a good task for a bad one and vice versa so the data security in our systems is this is going to be a huge upgrade moment I actually like I think that Actually, we're going to need a different access and security model going forward. Where are we going to go on that? Because the model we have is not granular enough and it's not performance enough to handle this stuff. So I want to step back and I want to say like a meta point, which is, I think this is how these conversations should go. We've identified a novel risk, which is like cyber security, which we actually have proof points and now we're talking about solutions. I think the entire discourse around AI can be of that form and that's the biggest mistake is that's not what it's been. I think the entire industry and community is very happy to engage exactly like this. Yes. And I have something to say about exactly what you're asking. But I'm saying like, I think this is where the conversation should be. Yeah. So we're known for having healthy conversations that everyone should learn from. So that's what we do. So here's the thing. I don't think there's a technical limitation here. Like these threat models are very well understood and have been in the literature for a long time. these sorts of things from an academic lens. The reason it hasn't been adopted is just tended to be a usability issue. I guess it's really hard to maintain and you didn't have to. So you could argue that AI solves the usability issue because it's AI is using it. So maybe now it's going to be a renaissance in operating systems and network and computer languages. And we should like go back to the old research and we should kind of start rebuilding systems that are secure by design. to put out. We don't put them out until we have these systems built. And no, by the way, the AI is very smart, so they can help us build it. And so I think, again, like, computer always evolved. Like, you know how much of the stack we had to change for the internet? Everything! Right. Tell me about it. You know how vulnerable everything was? Like, we could be in one of those moments like, oh shit, we gotta rethink everything. And that's fine. We've done that before. But I think that's a conversation we should have. So I agree. It's time to think about evolving these things. Well, like, look at, like, You mentioned earlier, like, booting a PC and getting a virus in 30 seconds or whatever. So if you look at how, like, you take an iPhone out of the box, which a lot of people are doing this week, you know, what happens is the whole network is basically shut down except for getting the latest version of the operating system. Because it doesn't, even though it was pressed, you know, six weeks ago... Some zero-day thing has been discovered since. And so actually the whole out-of-box process now involves first step, doing an update where the device can't do anything else, and it can never do anything else until it's updated. That's like there are all these things that are completely benign that we turned off in all of this desktop software of the era that used to be good things. Like it was having a macro for words so you could build automatic citations or something. It was like this super cool thing until it became a part. of thing where you could put a CD in and it would just arbitrarily run a program. And so then what somebody did was like, "Oh, well, I'm going to burn a clone of that CD and replace the program with my virus." but it's going to look like the thing that's supposed to run, and it's just collecting all this stuff and being evil. Yeah, I don't. Then we disabled that. And so one of the things that's happening right now is – There's a whole bunch of stuff that happens on your own BoxCorp network. That. Actually, it's gonna have to just change the standard procedure. Two-factor-off, five years ago, was not standard in most places. Your whole SaaS world, like I remember in like 2015 or so, when [40:00] When you would talk to a new company about their, oh, we're going to do enterprise pricing, whatever, and then they realized the first thing they had to do would go do... Okta integration or Google off because they could not have their own directory of how to manage it. And then that just became a thing. And now there's not a SaaS program anywhere that doesn't just launch with managed authentication. And so there's just so many things that need to happen before you're even software now. Yeah, well, yeah, we're, I mean, there's probably every layer of the stack has to evolve a bit on this. the agent will either ask you every single time, you know, if it, if you want to, you know, give a permission to do something or the exact opposite of like, it can just like delete your entire computer. And like our OS probably wasn't built for the right level of, of granular set of, of tools you want to give the agent. Um, we've kind of done a lot of work, work in this space because obviously like, you know, do you want to give an agent exactly like your entire file system? Probably not. Right. Maybe in some cases you do, but oftentimes you want to have granular controls read write and in that folder you can only do read and and so how do you kind of make this all intuitive for the user so it's it's very difficult and so so there's gonna like yeah yeah very deep comment i know exactly yeah which is basically the conclusion of 40 years right no 100 it's like you actually can't make it but maybe with ai you actually can like maybe there is like well i actually if you ask me like walking in like what i wrote down on my note like was my biggest fear yeah is and they're going to just GDPR AI. Yeah. And, and the, AI will be fine. It'll have one prompt for when text gets emitted that just says, this vendor is emitting text and it's probably wrong, yes or no. That's going to be because you can't really, at least in North America, you're not going to censor speech. In Europe, they still will. and they'll have filters and keywords and blog lists. But then on any verb, Anytime that an agent or a, or, a background agent or a frontline agent touches a third-party product, I'm really worried that they're just gonna say, "We need a GDPR prompt," and it's gonna be-- - Oh, back to the user through the agent. - Every single write or every single non-lookup becomes like a safety warning, like the airbag thing in your car. And the regulators love that because it's a liability assignment, and so it has this sort of legal precedent. And I'm really worried. that that's that's actually the mid middle ground where we're gonna end up. And unfortunately, because the U.S., about 15 years ago, stopped leading in tech antitrust. The problem is that Europe is gonna lead with that, because they have nothing to lose. And so I don't want to be sad and down about it, but I just can't get out of my head that they love prompts. I mean look, I have to put in that browser- You're being on prompts. Because it's the same way, look, get into a new car, which I haven't done in years, but you know, like, you're putting stickers off, you have all of these things. And and someone thinks that that was success. Yeah. And it's like, has anybody ever read like, what is this? If you're if there's a baby in this seat and it's like, well, that's relevant for some people some of the time. But it's this is the entire fabric attached to the seat. And and they love that. That is a very particular thing that they just love. And so I would I would say if I were an AI now, the one thing I would be trying to avoid it. And look, we added it. OK, FINRA for AI. We'll we'll we'll create that standard. to put this, when the internet was new, the big thing was to download a program and run it. And of course, if your machine is running in administrator mode, it was download a virus and take all your problems forever. And so with Windows XP, which was in 2000, we added this thing that prompted you and stopped. Literally, your machine stopped, user account control, and it was absolute assault. And we also put it in Word. The stupid little macro to help you write your thesis also came Everybody would just click. And so you end up in this world where, just like with GDPR, everybody is numb. Yeah. And so then they're like, well, it needs to be bigger. Although Mac, I mean, Mac kind of has. Nobody downloads software. That's the thing. It's a very different usage pattern on Mac. I wish. Sorry. I don't have like 10 applications on my Mac. Well, that's 10, and then you're done. Yeah, yeah, yeah. But it's a lot of people. - Still do all this stuff. Imagine if instead it was every time you go to a new website. - Yes. - Which you do now. - No, that would be bad. Yeah, yeah. - I mean, to start from here, but to bring it kind of back to the macro point, I wish this was the discussion we were having. - Yeah, yeah. - Which is like, I feel like we've dealt with a lot of these problems. I feel like when we talk about philosophical X-Risk, we're not solving these very [45:00] I actually think this is actually a constructive conversation to have. Maybe problems would help, I don't know. And I think part of the problem is, people don't remember like, you know, how... how unfettered access was and how bad it was and just how relatively benign that ended up being. Like, I even remember, like, this is at Stanford during our PhD. I remember the oscilloscope was kind of janky. I'm like, what's going on with this oscilloscope? Like, it's, you know, it's a little slow. And I was measuring the network traffic. And it was like, more network traffic than you would expect. And I'm like, why is there network traffic? I didn't even know the thing had a TCP stack. And somebody, like, broke in because there's an old version of Windows CE and was running a porn server. You know? And so, like. I had no idea that. It used to be. Just for the record, no. that anytime you turned over a stone, somebody had broke into something. And was it this worst-case malicious whatever? It was actually very rarely, even though the capability was there. And so if we could just somehow tone down the rhetoric and put it in context, and you still have computer systems, and yes, there's very serious stuff, and people have definitely died because networks have gone down. There's been real issues. And then can we just quibble about GDPR? That would be amazing. It's not about GDBR and prompts. It's about species extinction. And philosophy. And philosophy and irrefutable things. And it's just... It's also very soon. Like, I think that that's such a great point. And I think You hear people now talk about we regulate airplanes and we regulate cars. And you forget, like, well, the first cars were at the turn of the century. And unsafe at any speed was in the mid-1960s. Yeah, totally. And, you know, people have been doing selling pharmaceuticals during the gold rush. and thalidomide happened 50 or 75 years later, and not even in the US, and then there was the FDA. And, you know, people, the first Pilots were flying, obviously, at the beginning of the 20th century, and it wasn't until the 19... that you had to get a license to be a pilot, and you literally showed up with your own plane, and you got a certificate. It was literally no different than driver's ed is today. And then it was 20 more years until they had anything to do with airworthiness, and looking at your plane, but it was minimal. And then it wasn't until way after World War II that they got involved in what you think of as the modern FAA. And so you're looking at 40 years of innovation, and they were not moving slow. I mean, have you ever seen that video in black and white of all the different planes that they crashed? That was 20 years after the Wright brothers. And the FAA is incredibly effective. And, right, it's incredibly effective. It's the safest form of transportation. And so I do think that this process... of innovation. And so if you had started the FAA in 1910, you would never have a chance. Well, I was listening to a Nick Bostrom podcast. just a couple days ago. No, no, I... It was interesting. It was interesting. But he actually makes his point. If you regulate AI too early... you actually basically don't solve anything, and you still just kind of have the same risk ultimately, but you don't understand the system well enough. You will the thing into being, but you haven't figured out how to control it. Yeah, and we don't even figure out what it actually is yet. Like, there are new things coming out, all the time and like casting AI completely differently than we thought of just six or nine months ago. And I think that that like all the innovations can happen at the application layer is going to cause things to move in and out of the models in different ways. And like we, you know, we, we thought up until last week, I think that, you know, text prompts and text coming back was going to be the best way to interact with. And then Jeff Jeff. It's so good. And then. So talk Well, okay, so the way I think about it is – So, okay, so LLMs were kind of text in, text out, right? They generate text. And they came from chat, right? It was to communicate with a human. And we spent the last few years trying to take this thing that spits out text and cram it into a traditional program, right? But traditional programs don't really speak text, right? And so then you end up doing this janky thing where you're like, in the prompt, you're like, here's this human. Here's this human. But the thing is generating text and it kind of ignores it. And it's just been super janky. is a very expensive thing, but it's also kind of, you know, it's more complicated than you need. So why don't we, we'll read text and we'll have all of that kind of knowledge to read the text, but then rather than generating text, which is very expensive, we will just, if you give us a set of options, we'll choose the best option. We can do it incredibly, we can do it incredibly fast, incredibly cheaply, but also we can do it with much more accuracy because we can train just for this. And so for all of the use cases that are not talking to, to a chat bot but are actually trying to put it in traditional software, this is a great fit. And so this has probably been the fastest adoption of an AI model since ChatGPT. It's just been remarkable, because we're all primed for this. I just want to pile on this one, [50:00] I... I can't tell you how much I love seeing this exact form of innovation. Because what it does is it does the thing that's bugged me from the very beginning, which is there's been no user study ever that shows interacting with the computer using... full natural language is efficient. It's like literally always the least efficient way. And it's very simple. And it's just like, ask yourself, how many people are asked are really, really good at asking questions? And immediately that's like, less than half the people can ask a good question in a meeting. And then how often do you look at the answer and get really frustrated before it's finished, but you have to pay all this money to watch the seven paragraphs come out and then apologize that it's only a look. And so that's one, like having a different model. And then the other, of course, is my favorite, which is, the output of it is designed for probabilistic programming. And so instead of saying like, is this a customer service question. than route to customer service, otherwise route to general help desk or whatever. It's like, well, this is 80% customer service. And that's exactly simulation. And it turns out there's like 50 years of computer science research in literally like probabilistic if statements. And so suddenly, the coolest place to be in computer science is going to be in probabilistic programming, which was like all of computer science in the 1960s and 70s. So it was basically how do we, because All of computers started with doing math, and it was all simulation. So it was like, "Let's launch the missile and hit that target." But it's windy. But wind isn't constant. So like, let's model the wind and, and decide where to put the thrusters in order to do the art. And so most programming, through, like, say, 1970, before it got to accounting, was probably... And then we ruined everything. No, accounting, there's no probability in accounting. Right, right. But most programming was basically this modeling kind of thing. And so most programming language design was trying to figure out how to put... probability into if statements or into while loops. Like, do this until something happens, maybe most of the time. And like, so my first CS class, like the very second assignment or so, was a simulation about like waiting online at a store. And I didn't know it at the time. I The whole thing was my professor was like wrote the book called The Theory of Simulation in like 1960. And I just didn't. know that because it was kind of died by the 80s because it was all replaced by hyper. And so this notion of probabilistic, And that slide deck you shared, about the future, what's different about language models and stuff that you said was super good. Oh, Halper Fleck's one. It was phenomenal. It was a great day. But the part that I felt was missing was that like, oh, wait, this is not all new. Like all of computer science was this probabilistic stuff. And so it's going to be very interesting to dust off all of that work because it's exactly what's going on. Like it's not an if statement now is if X percent, not if always. custom programming language almost, which is here's the prompt, come back with a percentage. Yeah. And then you put that in the if statement. But the consequential thing is like, finally, we have a way to integrate these language models into a traditional software. And I think it's kind of funny because like you ask the question, like, why haven't the labs done this? Right. And it's kind of like a like not an indictment, but a reflection on how they think. Like they're trying to create beings and beings speak. If you're trying to create God, God speaks in natural languages or whatever, where this think that's for traditional software, which is kind of not the direction that they've been taking. But one of the reasons the uptick has been so dramatic is because a lot of us software people have been trying to integrate these models into software, and it just hasn't. So even before you get to the probabilistic, if I want a language model to drive an if statement, it's really hard today. With this model, it makes it much, much, much easier. And then, of course, this could change the nature of software fundamentally to make it more stochastic. Absolutely, and I think that what's so cool is that that It is happening outside the models because that's what I think is just going to happen, which is the center of innovation is just moved. And it's just and now people need to like it turns out that the outside of the platform providers, you know, basically reach a critical a point of critical mass where the innovation stops happening at the platform layer. And then, you know, Apple, there's this famous expression in the Apple community called Sherlocking, where Apple looks around and the things from the outside world become features and people complain. That's sort of how the innovation works. Because once you're a platform, you're overwhelmed. No matter how many people you add, you're overwhelmed with just keeping the thing running and compatibility and stuff like that. And so I think that this is the signal that now people have figured out that there's innovation to be done to the model, but outside the model. Guys, thanks for coming. This is a great episode. Okay, great. [55:00] of the A16Z podcast. If you liked this episode, be sure to like, comment, subscribe, leave us a rating or review, and share it with your friends and family. For more episodes, go to YouTube, Apple Podcasts, and Spotify. Follow us on X at A16Z and subscribe to our Substack at a16z.substack.com. Thanks again for listening, and I'll see you in the next episode. or be used to evaluate any investment or security and is not directed at any investors or potential investors in any A16Z fund. Please note that A16Z and its affiliates may also maintain investments in the companies discussed in this podcast. For more details, including a link to our investments, please see A16Z.com forward slash disclosures.