Persily Warns US Faces 16-Month Window to Regulate Frontier AI

珀西里警告：美国监管前沿人工智能仅剩16个月窗口期 单边禁令与企业自律无法遏制递归自我改进或中国开源模型，华盛顿只剩日益收窄的16个月窗口期来建立内部监管机制，以免技术治理彻底脱离人类控制。 政治学家弗朗西斯·福山（Francis Fukuyama）与法学学者纳撒尼尔·珀西里（Nathaniel Persily）探讨了前沿人工智能带来的日益严峻的治理困境，并指出尽管特定行业的监管已在推进，但对底层基础模型的监管仍面临严重的行政、地缘政治和结构性瓶颈。他们分析了行业自律的瓦解、联邦政府技术能力的匮乏，以及面对激增的中国开源权重模型时传统军控框架的局限性。对话还进一步探讨了围绕实体数据中心扩建的国内摩擦、欧洲监管模式在全球适用性上的不确定性，以及类似FINRA（美国金融业监管局）等金融风格的监管架构。最终，两位学者敦促政策制定者摆脱关于人类灭绝的推测性辩论，转而专注于劳动力市场、关键基础设施和金融体系所面临的紧迫而切实的现实冲击。 前沿人工智能监管面临预计16个月的政策窗口期，在此之后，潜在的递归自我改进可能会超出人类的治理能力。 (15:57) 前沿实验室之间的自愿自律面临严重的现实局限，包括针对协同安全协议的反垄断壁垒，以及来自中国不受约束的开源权重模型扩散。 (4:35) 美国政府在人工智能技术人才方面面临严重的国家能力赤字，历史上往往依赖国家安全机构在出现重大漏洞后才介入处置。 (9:08) 由于电网承压、公用事业费率上涨及水资源消耗，公众对人工智能的反弹目前主要集中在实体数据中心的建设上，这成为了更广泛的经济与生存焦虑的具体宣泄口。 (23:04)

Francis Fukuyama and Nathaniel Persily argue that effective AI governance requires building dedicated state technical capacity and sector-specific oversight rather than relying on industry self-policing or speculative existential doomsday debates. They emphasize that systemic vulnerabilities, labor displacement, and the unchecked diffusion of open-weight models pose far more immediate challenges than hypothetical extinction scenarios. The Limits of Voluntary Industry Self-Regulation (4:35) Persily and Fukuyama distinguish between application-level rules and direct oversight of the foundation models engineered in frontier labs. Voluntary self-policing has proven inadequate; Persily highlights a containment breach incident involving Hugging Face to argue that commercial pressures routinely supersede safety promises. Furthermore, because a winner-take-all mindset dominates frontier lab investments, leading firms channel capital into advancing capability metrics rather than internal safety checks. Legal and structural barriers compound this self-governance deficit. Existing tort liability can address downstream harms after an autonomous agent fails, but it cannot prevent irreversible outcomes like chemical, biological, radiological, or nuclear (CBRN) threats or severe cyber incidents. Moreover, when competing labs attempt to coordinate voluntary safety thresholds, they risk exposure to antitrust litigation alleging anticompetitive collusion. State Capacity Deficits and Institutional Architecture (9:08) A primary bottleneck in AI governance is the human capital gap across governments. Fukuyama notes that the average age of an American civil servant is in the upper 50s, while premier engineering and policy graduates are overwhelmingly recruited by private firms like OpenAI and Anthropic. Persily adds that this deficit undermines comprehensive frameworks like the European Union's AI Act, which will falter if regulatory bodies lack the internal personnel to inspect and audit complex systems across diverse industries. To bridge this divide, Fukuyama draws on the historical transition from the Interstate Commerce Commission to dedicated agencies like the Federal Aviation Administration to argue against outsourcing technical regulation. Persily suggests establishing a specialized body focused strictly on pre-deployment model evaluations—similar to the United Kingdom's AI Safety Institute or a FINRA-style public-private partnership—while leaving downstream impacts on areas like the power grid or consumer protection to domain-specific agencies. Geopolitical Realities and Open-Weight Model Diffusion (17:28) Persily challenges the narrative that AI development is a binary race between the United States and China with a fixed finish line, arguing that capability milestones will eventually be reached by rivals regardless of short-term leads. The deeper governance dilemma stems from model architecture: while American firms primarily operate closed foundation systems, Chinese institutions frequently publish capable open-weight models. These models can be downloaded, adapted, and stripped of safety filters globally, undermining purely domestic US restrictions. Fukuyama compares the spread of cost-effective Chinese open models to Huawei's historical disruption of Western telecom giants like Lucent and Nortel. Because many practical business applications do not require frontier-scale reasoning, enterprise adoption may pivot toward accessible open-source architectures. On the international stage, while Persily sees potential for a narrow, bilateral 'Geneva Convention' covering CBRN and cyber warfare risks, monitoring remains exceptionally difficult compared to physical nuclear arms inspections. Public Backlash, Infrastructure, and Near-Term Harms (23:04) Grassroots resistance to artificial intelligence is crystallizing around the physical footprint of data centers. Persily observes that local complaints concerning water consumption, power grid strain, and utility rate hikes serve as a concrete outlet for diffuse public anxieties regarding economic displacement, algorithmic bias, and existential risk. Fukuyama warns that the frantic capital expenditures driving this infrastructure buildout are temporarily masking broader labor automation, which cash redistribution schemes like universal basic income cannot solve given the intrinsic connection between paid work and human dignity. Both speakers dismiss the hyper-focus on civilizational extinction (p(doom)) as an unhelpful distraction from high-probability disruptions. Persily and Fukuyama urge policymakers to prepare for mid-level systemic shocks within a critical 16-month window before recursive self-improvement accelerates, pointing to plausible near-term crises such as flash financial crashes, localized electrical grid failures, and automated mass-scale banking cyberattacks. Fukuyama and Persily conclude that the primary threat in the near term is not a runaway superintelligence, but the compounded instability of an unprecedented industrial transition. Whether governance succeeds depends on whether governments can rapidly upgrade internal technical talent, mandate air-gapped sandboxes for experimental research, and address labor disruptions before automated systems outpace existing institutional guardrails.

斯坦福大学学者 Francis Fukuyama 与 Nathaniel Persily 指出，人工智能监管的紧迫瓶颈在于国家行政能力的严重匮乏与地缘政治竞争，而非单纯的立法缺位；政策制定者应摆脱虚无的人类毁灭叙事，聚焦于约16个月内可能到来的递归自我改进窗口，以及电网、金融等具体领域的系统性风险。 国家能力短缺与监管架构设计 (9:08) Fukuyama 指出，美国联邦政府面临严峻的国家治理能力赤字，公务员平均年龄已接近60岁，极度匮乏前沿计算机领域的专业知识，而顶尖工程与政策毕业生普遍被 OpenAI、Anthropic 等私营实验室高薪招募。他以历史上州际商务委员会（ICC）错误监管卡车行业、直至后来设立联邦航空管理局（FAA）专门监管航空为例，强调通用型或外包监管在面对复杂颠覆性技术时注定失效。美国在应对类似“Mythos”模型泄露引发的极端网络攻击风险时，虽然促使国防部与联邦调查局（FBI）介入叫停部署，但国家安全机构并不能替代常态化技术监管。 Persily 进一步分析，若监管仅停留在法律条文层面而缺乏执行人力，欧盟《人工智能法案》等宏观框架也难免落空。他主张采取分层治理结构：借鉴英国人工智能安全研究所（UK AISI）作为全球事实基准的先例，设立专注于前沿模型部署前测试的独立技术机构（如商务部旗下的安全研究所）；同时由环境保护署（EPA）、联邦通信委员会（FCC）等传统部门负责电网负荷、公共事业定价等下游行业影响。此外，针对私营实验室可能逃避监管的问题，Persily 建议参考美国金融业监管局（FINRA）的模式，建立包含强制标准与第三方认证审计的公私合作治理机制。 自律破灭与16个月时间窗口 (4:35) 针对 Anthropic 首席执行官 Dario Amodei 提出的“调整前沿步伐”等行业自律构想，Persily 明确表示，私营实验室的自愿协调在实践中屡遭失败。他列举夏季 Hugging Face 出现的安全漏洞与模型失控事件指出，虽然未造成实际公众伤害，但商业利益驱动下的自律无法抵御灾难性后果。更关键的是，同行企业若私下联合制定安全上限，极易触犯现行反垄断法并面临串通指控；而事后侵权诉讼机制对于化学、生物、放射性或核（CBRN）威胁以及永久失控等不可逆伤害毫无防范作用。 Persily 发出预警，政策制定者仅剩大约16个月的关键窗口期来确立有效的规制体系。一旦大模型实现递归自我改进（RSI）——即模型能够自主、迭代地优化其底层代码与系统架构——技术发展速度将彻底脱离人类行政干预的能力范围。此时，各州各自为政的“拼贴式”立法不仅效率低下，更会错失干预模型层安全的核心节点。 中美地缘博弈与开源模型扩散 (17:28) 在国际维度上，两位学者打破了“监管会导致西方在与中国的竞争中落败”这一线性思维。Persily 认为，将通用人工智能（AGI）研发类比为终点线竞赛是错误的，即使对手落后六个月，最终也会达到相似的能力水平；前沿研究推进与沙盒安全防护并不互斥。地缘政治的核心难点在于开源权重的全球扩散：中国机构正在积极发布强大的开源权重模型（如 Kimi），任何用户均可下载、微调并剥离其安全过滤器；加之开发者频繁通过 Hugging Face 等平台对 Claude 等专有模型与开源架构进行蒸馏和混合，使得追责变得极为困难。 针对西方认为中国放任技术发展的误解，Fukuyama 澄清指出，中国共产党对境内模型实施着极其严格的政府审批与内容审查机制，其监管苛刻程度往往超出西方民主国家。Fukuyama 还以华为击垮北电与朗讯等历史经验对比指出，大多数实际工作并不需要运行成本高昂的“法拉利级别”前沿模型，性价比极高的开源模型正对西方私营企业构成实质冲击。Persily 提出，中美之间或可探索建立类似针对 CBRN 和进攻性网络武器的“日内瓦公约”式双边算力监控协定，但现任政府缺乏强制推行双边降速的政治意愿。 实体基建抵制、劳动尊严与现实危害 (23:04) 在社会与经济层面，算力基础设施的激增已引发全美范围的政治反弹。围绕数据中心建设的邻避效应（NIMBYism）、电网超载、水资源消耗及公用事业费上涨，正成为公众宣泄深层技术焦虑的实体导火索。Persily 指出，公众对人工智能监管的民意党派分歧仅约10%，属于罕见的非极化议题，但随着2028年大选临近，反人工智能情绪或将加速上升。与此同时，Persily 警告，当前全球经济增长危险地依赖英伟达、微软与 OpenAI 之间复杂的循环资本支出，一旦出现类似1999年或2008年的基建放缓或金融修正，将引发剧烈震荡。 对于大规模劳动力替代，Fukuyama 强调全民基本收入（UBI）无法取代带薪劳动所承载的“人类尊严”与社会价值认同。两人一致驳斥了主流叙事中关于人类灭绝概率（p(doom)）的终极毁灭论调。Fukuyama 主张，防御性人工智能技术将与威胁同步演进，监管更应聚焦于高度可能发生的现实危机——例如单个银行上百万个账户遭自动化网络攻击清空，或是区域电网与股票交易系统遭遇闪崩等系统性冲击。 当前技术治理的核心矛盾在于监管制度的滞后与前沿模型向物理实体外溢的剧烈碰撞。由于缺乏跨国算力核查机制与联邦统一立法，无论是完全寄希望于欧洲式的“布鲁塞尔效应”，还是依赖企业沙盒物理隔离，都无法彻底规避不可逆的技术外溢风险。

[54:37] Francis Fukuyama: I really do think that people ought to focus not on these we're all going to die scenarios, but really on the near-term very serious harms. [14:09] Francis Fukuyama: you can't outsource regulation to the people that need to be regulated. [0:10] Nathaniel Persily: AI regulation is inevitable and it is already happening.

[0:10] I would like to introduce our speakers [0:11] today. We have with us Francis Fukuyama, [0:14] senior fellow at Stanford's Freeman [0:16] Spogley Institute for International [0:18] Studies. He is here today in [0:20] conversation with Nate Priscili, [0:22] professor of law at Stamford Law School [0:24] and co-director of the Stamford Law AI [0:27] Initiative. Francis and Nate, thank you [0:30] so much for being here with us today. [0:31] We're looking forward to your [0:32] conversation. Please take it away. [0:35] >> Uh thanks very much Annette. So uh I [0:38] want to talk a little bit about the [0:40] course that we are uh planning. Uh this [0:43] uh course the course title is AI power [0:46] and the new rules of global strategy. Uh [0:50] it will be taught in person uh the week [0:52] of March 22nd of 2027. [0:55] Um I'll be teaching in it. Nate will be [0:57] and many of the other professors that [1:00] work on AI policy uh issues here uh at [1:04] Stanford. Um I should say a little bit [1:07] about who this course is for. This is [1:10] not your typical AI course where we're [1:12] just teaching skills or giving you [1:15] background on the technology or how [1:16] you're going to deploy it in your [1:18] company. Uh our unit is um uh we I [1:24] direct a master's in international [1:26] policy at the Freeman Spogley Institute [1:29] and we are interested in governance [1:31] issues. Uh we're interested in the [1:34] question of how this technology relates [1:36] to geopolitics to domestic politics how [1:40] it's going to shape the uh international [1:43] dimensions of the world that we are uh [1:46] entering into. And if you are interested [1:49] in those sorts of issues, you know, [1:52] really what this means for the world, [1:54] uh, more broadly, uh, this, uh, I think [1:57] will be the course, uh, for you. [2:00] So, uh, let's begin. Uh, my colleague [2:03] Nate is one of the foremost experts on [2:05] the question of uh, AI governance. He's [2:09] thought a lot about these uh, the issues [2:12] connected with that. So Nate um [2:15] in recent uh weeks there's been a lot of [2:18] talk about uh needing to regulate uh [2:22] artificial intelligence. Uh you know [2:24] Dario Amadai did a essay in which he [2:28] said we need to pace the frontier. I [2:30] have no idea what that means. Uh but it [2:33] evidently means you know you got to slow [2:35] down. But it strikes me that a lot of [2:39] that discussion has been very [2:40] non-specific. Um, if you did want to [2:43] slow down the pace of AI development, [2:46] these new frontier models, how would you [2:49] go about doing it? Uh, would it be the [2:51] labs themselves that did it? Would it be [2:53] an international agreement? Would it be [2:56] uh some government agency? And if it's a [2:58] government agency, you know, who has the [3:00] authority to uh make decisions like [3:03] that? So maybe you could speak a little [3:05] bit in general about AI governance as a [3:09] problem and then you know what you see [3:11] as some of the solutions. [3:13] >> Great. Well, thanks for having me. I'll [3:15] say that uh I I've decided that my new [3:17] diet is going to be called pacing the [3:19] frontier because that's what I I'm [3:21] trying to do. Um so AI governance is [3:24] certainly having its day and AI is [3:26] having its political moment. Uh we're [3:28] seeing it in the United States and [3:30] around the world. uh what whereas in the [3:33] 2024 election AI was barely uh discussed [3:36] uh in this election whether it's because [3:38] of uh data center buildouts or these [3:41] questions about labor force impacts or [3:43] existential risks which you are focusing [3:45] on uh I think now uh people's minds are [3:48] are being focused on it both here in [3:50] Silicon Valley and in Washington. Uh [3:52] I'll say that at at a general level when [3:55] you think about AI governance and [3:57] regulation you need to [4:00] admit a few things. The first is AI [4:03] regulation is inevitable and it is [4:05] already happening. Um if you are you [4:09] know riding in a way whimo in um in San [4:13] Francisco and it runs over my foot I [4:15] need to decide who I can sue that is AI [4:18] regulation. uh when it comes to the use [4:21] of AI by uh banks or by uh pharma or [4:25] others, there are existing regulations [4:27] that are going to that that are [4:29] triggered. And so we need AI regulation [4:31] in order to sort of chart a safe AI uh [4:34] future. But one of the central questions [4:35] is whether you regulate the technology [4:37] or you regulate the applications. For [4:40] the most part, you know, in the last few [4:42] months, because of these concerns about [4:44] existential risk and loss of control, [4:46] we're now having a more serious [4:48] discussion about how you might regulate [4:51] the the technology itself, particularly [4:53] as produced by uh the major labs. But I [4:55] do want to just emphasize that AI [4:58] regulation is all around us because we [5:00] need to adapt the existing legal order [5:02] uh to to deal with AI. Now, when it [5:05] comes to how you would try to um uh deal [5:10] with the the the new problems or the the [5:13] problems that are are coming from the [5:15] labs that with respect to the [5:16] technology, um you're right to sort of [5:19] highlight the what I think are the [5:21] obstacles, right? And and those [5:23] obstacles are uh that this is a new and [5:26] fastm moving technology. Uh I mean [5:28] obviously AI has been around for a long [5:30] time but it's it's a lot of these new uh [5:32] features are I think um present unique [5:35] regulatory questions. uh it is [5:38] international in scope and so of course [5:40] whenever you talk about say uh US [5:43] governance of AI you have to put that in [5:45] the geopolitical uh discussion with what [5:47] would China do uh and we don't exactly [5:50] have international institutions that are [5:53] performing uh with great skill these [5:56] days and so it's it's not clear how [5:58] whether it's the UN or whether it's WTO [6:01] or or whoever else uh to to uh craft an [6:04] international agreement it really begins [6:07] with the US uh and China, but but we [6:09] here in the US really don't have the [6:12] answer. There are many bills in [6:13] Congress. Uh the administration is is [6:16] quite accelerationist. Uh and um so [6:20] we're left with sort of the the the [6:23] preaching from the AI companies about um [6:26] um how dangerous this may be uh and [6:29] asking for regulation. And I think there [6:31] is m maybe momentum in that direction [6:33] but there's a lot to to work on with [6:35] respect to the specifics. [6:38] So if well okay let's let's go through [6:42] this uh systematically. [6:44] Could you rely on the frontier companies [6:46] themselves to you know quote unquote [6:49] pace the frontier? Could you leave it up [6:51] to them to uh you know try to slow down [6:55] the rate at which um their development [6:58] of these very powerful models is going. [7:01] Well, I mean I we've sort of done that [7:04] already and we've had the hugging face [7:05] incident. So I think that that incident [7:08] over the summer and then all of the kind [7:10] of knock-on reporting of other incidents [7:12] like that have suggested that at least [7:14] the existing uh system is not working uh [7:18] and that in you know that that thank God [7:20] that that incident or the other losses [7:23] of control were not didn't harm anyone. [7:26] uh but it does give you some sense that [7:30] um the the companies if left completely [7:32] to themselves might not uh act in the [7:35] public interest. Now it is still the [7:37] case that existing tort law and [7:40] liability regimes uh could punish the [7:43] companies if for example their agents [7:45] committed um uh havoc in the financial [7:48] system or pick your pick your uh victim. [7:51] Uh and so it's not as if they're [7:53] completely unchecked by the law. Uh but [7:56] I think given that we're we're we're [7:57] worried about new kinds of risks with [8:00] these companies, whether you're talking [8:01] about what we call the SEAB burn risks, [8:03] chemical, biological, radiological, [8:05] nuclear, or these loss of control risks [8:08] or the cyber risks. Um there there is [8:11] this sense that the the potential costs [8:13] are so huge that leaving it to like the [8:16] existing litigation liability tort [8:19] system might not be sufficient. And so [8:22] one other option is that while we could [8:24] have collusion among the companies to um [8:27] uh to come up with best to slow the [8:29] frontier to agree to do so um even the [8:32] spectre of that by the various press [8:34] releases that we've seen has led to [8:36] already an antitrust lawsuit against [8:38] them. And so I think that it is it is [8:42] going to be difficult for the companies [8:45] to come up with uh uh mutually agreed [8:48] upon safety standards if there is not [8:50] some enforcable enforcement body that [8:53] would keep them to that. Even so uh one [8:57] of the challenges is also that you can [8:59] do all you want with the labs in the [9:01] United States but if the Chinese uh just [9:03] continue a pace uh then the technology [9:06] is going to be out there. [9:08] >> Yeah. Well, let's uh let's get back to [9:10] China in in uh a little bit. All right. [9:13] So, you can't trust the companies to do [9:15] this on their own. So, that leaves the [9:17] US government. Um this is one of the [9:21] issues that I uh think about a lot. U we [9:24] have a problem in this country with [9:26] state capacity, meaning we don't have [9:28] enough people in the government that [9:30] know enough to actually sensibly uh deal [9:34] with a new very fastmoving technology [9:36] like this. I think the average age of an [9:39] American civil servant is somewhere in [9:41] the upper 50s. I have nothing, you know, [9:44] I'm older than that, so I have nothing [9:45] against people that are in their 50s, [9:47] but uh they tend not to be, you know, uh [9:50] up on the latest uh technological [9:53] advances. And so you start with this big [9:55] problem of where are you going to get uh [9:58] the people into the government that know [10:00] enough about this technology that they [10:02] can uh regulate uh sensibly. I think [10:06] that [10:07] you know my students uh a lot of them [10:09] have been hired by the companies [10:11] themselves and are working at anthropic [10:13] and open AI and you know other places [10:16] but um almost none of them actually [10:19] would like to go uh into the government. [10:22] Uh so what part of the government uh you [10:25] know one question is do you actually [10:27] have to create a new regulator uh to [10:30] deal with this or can you give it you [10:33] know to existing parts of the [10:35] government. So for example, I think uh [10:37] after the mythos uh release where uh [10:40] there was this fear that government [10:42] computers could be broken into very uh [10:45] easily, it was really the security uh uh [10:48] agencies that got really spooked by this [10:51] and they're the ones that, you know, [10:53] pushed the Trump administration to uh [10:56] you know, not get that model released so [10:59] quickly. So that was the defense [11:01] department, I don't know, the FBI, other [11:02] agencies like that. who who should be [11:05] doing this regulation. [11:08] >> So, um again, we there's no way around [11:13] the need for uh talent related to AI in [11:17] all areas of say the federal government [11:19] right now because it's going to be [11:21] penetrating, you know, whether it's the [11:23] Department of Commerce or whether it's [11:24] the EPA or whether it's um um the FCC, [11:28] right? There are go every area of [11:31] government is going to have to tech up. [11:33] Uh I agree with you that this is you [11:35] know one of the big challenges. usually [11:37] end with that in a discussion of AI [11:39] governance saying look you could write [11:40] the perfect law but if you don't [11:42] actually have people there to enforce it [11:44] uh you're going to you're going to uh be [11:46] swimming upstream and that by the way is [11:48] a global problem um you know for all of [11:51] its legislation in Europe uh on the say [11:54] the European AI act which is the most [11:56] comprehensive uh regulation out there if [11:59] they don't have people in the government [12:00] to actually monitor what is basically a [12:03] regulation of the entire economic system [12:05] uh in Europe they're not going to be [12:06] able to uh enforce it. Uh so I agree [12:09] with you on that. And so where if you [12:11] were to uh develop a more comprehensive [12:14] AI regulatory scheme in the US, where [12:16] would you locate it? Um I think you when [12:20] it comes to the national security [12:22] issues, you are going to have to have um [12:24] all of the the various folks in the [12:27] national security firmament uh uh being [12:30] involved in this. There's no question [12:31] about that on on those question. So the [12:33] so the Pentagon and FBI and etc. Um but [12:38] I think that if we are going to regulate [12:40] the technology in general uh then you [12:43] know starting some kind of new agency [12:46] makes a lot of sense but it depends what [12:48] we want it to do because we're not going [12:50] to have it regulate uh every application [12:53] of AI in the economy. Uh but whether [12:58] it's it's what was the original US AI [13:00] Safety Institute which has now been [13:02] recast as Casey and I think it even has [13:04] another uh uh name this week because [13:06] we're now calling it super intelligence [13:08] instead of artificial intelligence. uh [13:10] uh uh that that you're I you if we're [13:14] going to have a regime of pre-eployment [13:16] testing where the AI labs have to get [13:19] some stamp of approval from the federal [13:21] government then you need technical [13:23] capacity I think in a new agency [13:26] um whether it's under the department of [13:28] commerce as it currently is or some [13:29] place else that has the technical talent [13:32] and can be nimble enough uh to evaluate [13:35] these models but that is just at the at [13:38] the kind of model layer and and on [13:40] pre-eployment testing when it comes to [13:41] enforcement and sort of monitoring all [13:44] these dangers, it's going to be a whole [13:45] of government approach. You're going to [13:46] have to have uh in each one of these [13:49] sectors, I mean, take like the energy [13:51] sector just as an example where where [13:53] now, you know, the the effect of AI on [13:55] on energy is going to be uh so important [13:59] both on the prices and then and and the [14:01] requirements of it. And so uh uh it's [14:03] it's going to be all over the federal [14:05] government. uh even if we want to like [14:06] have some specialist in a special [14:08] agency. [14:09] >> Yeah. I guess part of the problem is [14:10] that AI penetrates into every part of [14:13] the economy and so for example child [14:16] safety you know is an issue for you know [14:18] certain part of the economy but you know [14:21] you don't necessarily want the national [14:23] security agencies to be making decisions [14:25] on what uh you know what's not good uh [14:29] what's not good for children. So I the [14:32] way I think about it, you know, um I [14:34] really do think that you need a [14:36] specialized regulator. I'll just give [14:37] you a historical uh precedent for why [14:40] that's uh the case. The first um [14:43] American regulator in the late 19th [14:46] century was the Interstate Commerce [14:47] Commission, which was put put in place [14:52] to regulate this new technology, the [14:54] railroads. [14:56] when trucking came along, uh people [14:58] said, well, you know, trucks are just [15:00] another way of moving things from one [15:02] place to another, so we'll give this to [15:04] the IC. And that turned out to be really [15:07] unworkable because the economics of [15:09] trucking is completely different from [15:12] the economics of railroads. So, by the [15:15] time they got to airplanes, uh they [15:17] didn't try to fold it into some master [15:19] regulator. They created, you know, the [15:21] FAA and the CAB and a lot of very [15:24] specialized things. I would think that [15:26] given [15:28] how uh fastm moving and how uh complex [15:33] this technology is, you really do need [15:35] to get people that are really focused on [15:38] this. The usual pattern I think in the [15:40] uh US government is to outsource this [15:42] thing. But you can't do that. You know, [15:44] you can't outsource regulation to the [15:47] people that need to be regulated. And so [15:50] uh you really are going to need to uh [15:52] somehow develop that um uh that capacity [15:56] uh internally. [15:57] >> I agree with that uh that we need to [15:59] develop the capacity, but let let's [16:01] figure out what this group is going to [16:02] do because as you said, you're going to [16:05] end up having all areas of federal [16:07] regulation um uh affected by AI. And so [16:11] even if you were to develop the the sort [16:13] of ninja AI agency within the federal [16:16] government and staff it with what what [16:18] would probably be thousands of people, [16:20] you are still going to have to have kind [16:22] of an updating of the regulatory [16:24] environment in all these other areas [16:25] that touch AI. Uh, and so that's why I [16:28] focused on on what I think most people [16:30] can think about when they're thinking [16:32] about AI safety just at the model layer, [16:34] which is that all right, you have, you [16:35] know, and and the UK has actually been [16:37] probably the most outfront on this. [16:40] Their AI safety institute has been the [16:42] leader around the world. So much so that [16:44] they're kind of developing the standard [16:46] by which the the model developers are [16:49] are sort of abiding. And so they'll [16:50] they'll submit it there and people take [16:52] a lot of uh direction from them. Uh we [16:55] also shouldn't forget the states right [16:57] uh because the states are are uh being [16:59] active here. It's a problem. I think [17:01] everybody agrees that a patchwork quil [17:04] of AI regulation is is is not not just [17:07] second best but it's very third fourth [17:10] fifth best right um um but it it's you [17:13] know we can't wait uh because we need to [17:15] get this right in the next 16 months uh [17:18] because if we reach recursive [17:20] self-improvement and the models are you [17:22] know off and running uh I think it's [17:24] going to be too late for us to come up [17:26] with the right regulatory regime. [17:28] >> Yeah. So let's talk about the uh [17:31] international side of this. You've [17:32] mentioned China already in any [17:35] discussion of AI regulation. You always [17:38] get to a point uh you say well we ought [17:40] to slow down. We ought to regulate uh [17:43] you know be careful in moving forward. [17:45] And then somebody says but what about [17:47] China? China is not slowing down. uh and [17:51] um if we slow down, they're simply going [17:54] to beat us to super intelligence or AGI [17:58] or whatever the goal is. How do you [18:00] think about that? [18:01] >> So, I I tend to think that the race [18:04] analogy is the wrong one. That this is [18:07] not a race with a finish line. Uh and [18:09] that even if they are six months behind [18:12] that then you know wherever you put that [18:15] marker in terms of uh super intelligence [18:18] they will get there. Um, and it may very [18:21] well be that the benefits of recursive [18:24] self-improvement or the the impact of [18:26] recursive self-improvement is so [18:27] significant that it it's absolutely [18:30] critical that for national security [18:32] reasons that the United States get there [18:33] first. uh because uh I if well I said [18:37] there's no finish line if the the shape [18:40] of the curve is such that once you reach [18:42] RSI that then uh your capabilities just [18:45] you know say for offensive cyber attacks [18:48] or or some other national security uh [18:50] concern uh that then it's it's important [18:53] that we get there. It's not clear to me [18:55] that um [18:58] that that that you can't sort of develop [19:01] the technology uh and and develop some [19:04] of these technological advances that [19:06] would be important to protecting our our [19:08] lead here while uh also making sure for [19:12] example you don't have agents escaping [19:14] sandboxes right I mean I think there is [19:16] a way to uh uh continue investing in the [19:20] research here uh while also doing it uh [19:22] in a safe way But you know the China [19:25] argument I it it's not just about losing [19:28] to China. It's about if if the [19:30] technology is going to be out there and [19:32] in particular since China has been [19:35] developing open models right which can [19:37] be adapted around the world uh by by [19:40] anyone um that as those approach you [19:43] know greater and greater power and [19:45] capabilities they pose a unique [19:47] regulatory problem. So even if anthropic [19:50] and open AI and and Google and X and [19:53] Meta uh end up slowing a little bit um [19:57] and even if it doesn't the US [19:59] position in terms of AI dominance um if [20:02] you have super capable open models [20:06] coming from China then those pose you [20:08] know unique safety concerns that a US [20:10] regulatory regime that's focusing on the [20:12] proprietary closed models is not going [20:14] to touch. [20:16] >> Yeah. I wonder whether you could [20:18] actually [20:20] uh make a distinction between Chinese [20:22] models that come from overseas and [20:23] domestic ones and you know somehow ban [20:26] the uh the external ones. I mean they're [20:28] all connected on the internet. So [20:30] >> yeah. Well, that's exactly right. And [20:32] and you know, we're going to get to a [20:34] point where where the open models that [20:36] are hosted on places like Hugging Face, [20:38] I mean, we're probably there already [20:40] where especially when you get to some of [20:41] these SP smaller specialized models, [20:44] they're distilled from Claude, you know, [20:46] plus Kimmy plus these, you know, that [20:49] there's uh uh these new open models that [20:52] are kind of just an amalgam of all the [20:54] stuff that's out there. And so it's [20:56] going to becoming increasingly difficult [20:58] to attribute say harm to the actual [21:01] developer of the model especially when [21:03] you're talking about these kind of [21:04] bespoke applications let alone as you [21:06] were saying you know look it's software [21:08] so it's basically going to be out in the [21:10] cloud somewhere so if you're a bad actor [21:13] um and there aren't as many controls in [21:14] China as there would be here then you're [21:16] going to have access to those models. I [21:19] think one thing that people don't [21:20] realize is that the Chinese themselves [21:24] are not libertarians who are just saying [21:26] let the technology rip as fast as [21:28] possible. You know, we want to be first. [21:30] I mean, you have some Americans that [21:32] believe that, but that's not the way the [21:34] Chinese Communist Party acts. And as I [21:36] understand it, they actually do more [21:38] regulation of their AI than we do. I [21:41] mean, so for example, they actually have [21:43] a government agency that looks at [21:46] frontier models and has to get give its [21:48] permission for them to be uh released. [21:51] Uh so they're already doing things that [21:54] we would presumably, you know, want to [21:56] agree with them that we all should be [21:58] doing, you know, in the in the interest [22:00] of general AI safety. [22:02] >> I agree. And I I I look I do think there [22:04] is a space for some kind of quasi Geneva [22:08] convention on AI that would um at least [22:10] deal with the existential risks on you [22:13] know chemical, biological, radiological, [22:14] nuclear and that maybe cyber. Um it it's [22:18] not going to be easy to enforce. This is [22:20] not like nuclear weapons where you can [22:23] maybe monitor you know the warheads or [22:25] the plutonium or something like that. Um [22:28] but it does take a hell of a lot of [22:29] energy to to train these models and it's [22:32] not as just if any that anyone can [22:34] develop something at the frontier and so [22:37] if there were sort of greater oversight [22:39] and monitoring by an international body [22:42] um uh on the training of these models at [22:44] least just on the seaburn uh and cyber [22:47] uh risks then I think that would you [22:49] know be some kind of place for agreement [22:52] between the US and China but you know we [22:55] haven't seen it thus far and And [22:57] certainly the administration is not um [22:59] pushing to have an agreement that would [23:02] slow everybody down at once. [23:04] >> Yeah. Yeah. Let's talk about the [23:06] politics of this. So, a lot of very [23:09] interesting things are happening both [23:10] domestically and internationally. [23:13] Um [23:15] in the United States, you now have [23:17] really big anti- AI wings of both the [23:20] Republican and Democratic parties. Right [23:22] now, it's all focused on data centers. [23:25] Uh, a lot of that seems to be driven by [23:27] some form of nimiism or as I understand [23:30] it, a lot of it is the feeling that a [23:32] lot of local officials have been making [23:34] these deals with the big companies and [23:37] getting paid off, uh, you know, without [23:39] actually consulting their local [23:41] communities. Um, I would think that by [23:44] the time you get to the 2028 election, [23:48] uh, AI is going to be one of the big [23:50] issues that's going to be contested in [23:52] this. Do you agree with that? [23:54] >> I do. I do. And that that's sort of [23:55] where I started and I think that you're [23:57] even seeing it in 2026 in particular [24:00] areas as seen through the data center uh [24:03] issue right and so whether it's for [24:05] state legislature whether it's for [24:06] congress there are certain parts of the [24:08] country where the data center issue is [24:11] front and center now what is the data [24:13] center issue about there is the nimiism [24:15] issue that you suggested as well as [24:17] concerns about say energy rates and [24:19] water and all of that but I tend to [24:22] think that the uh outcry against data [24:24] centers is just a way to um focus [24:28] people's diverse anxieties related to AI [24:32] uh at something which is literally [24:34] concrete right so that you can uh this [24:37] is an area where you can um you know a [24:39] physical thing that you can protest [24:41] against in whether you are concerned [24:43] about AI's effect on uh your kids or [24:46] whether it's on um the economy or [24:48] whether it is on uh the the stuff [24:51] related to data center the noise the um [24:53] the energy, the water. Uh I I I tend to [24:56] think that the concerns about data [24:59] centers are actually politically [25:02] navigable that that that just the [25:04] building of the buildout of the data [25:06] centers, but because I think it [25:08] represents a sort of broad concerns that [25:11] people have about uh AI, uh I think that [25:14] it's going to be a political flash [25:15] point. I'll say that and I'm I'm writing [25:17] an article um on this right now and I as [25:20] I think you know I'm I'm the head of the [25:22] task force of the American Political [25:23] Science Association on AI and we've got [25:25] a a book coming out on on politics and [25:27] AI. I tend to think that there is there [25:31] there is a kind of residual tech lash [25:33] that is still affecting people's [25:35] attitudes toward the AI companies. That [25:37] is a tech lash that started say 2016 [25:40] with Facebook and everything that [25:41] happened uh thereafter or Cambridge [25:43] Analytica whatever. Um, so there are [25:46] people who are distrusted tech companies [25:48] to begin with and then you lop on to [25:50] that the fact that they're all saying [25:51] that they're existential risk that this [25:53] you could lead to 10 to 20% chance of [25:55] the end of humanity plus the widespread [25:58] job loss fears that I think some people [26:00] you know really are concerned about and [26:02] then uh other concerns related to to [26:06] kids or or to uh the kinds of things [26:09] that we've seen with social media deep [26:10] fakes all of that. Um, and so there's [26:13] all of these different concerns that are [26:15] now filtering in into the politics. You [26:17] are right, which is that the there is [26:19] only about a 10% partisan gap on on most [26:22] questions that you ask with respect to [26:24] uh AI and politics. Um, and so that's [26:28] that makes it quite different than than [26:30] most other areas of American political [26:32] life. But we will see whether the elite [26:35] signals that are coming out uh [26:37] particularly from the administration are [26:38] going to lead this to have a more [26:40] familiar partisan uh structure in public [26:42] opinion. [26:44] >> Yeah. Um [26:47] I think you're absolutely right that the [26:49] data center issue is not the most [26:51] important uh that it really does [26:53] symbolize these more diffuse fears of [26:56] what the economic and uh job impact of [27:00] AI is going to be. Um I don't think [27:03] we've really seen uh how that's going to [27:05] play out yet. Uh actually the [27:08] unemployment rate is still quite low [27:10] because the data center buildout itself [27:12] is creating a huge amount of economic [27:14] activity. So people are busy doing that. [27:16] But does seem to me that at a certain [27:19] point uh you know a lot of uh [27:21] occupations are going to be replaced by [27:23] these machines and I don't think that [27:26] we've even begun to think about how the [27:28] government responds to you know this [27:31] kind of technology uh job loss and you [27:35] know do you compensate people do you [27:37] restrict it do you you know universal [27:39] basic income I mean there are all these [27:41] ideas floating around that I don't think [27:43] are very plausible [27:46] agree with that. [27:47] >> Yeah. I mean I think because ultimately [27:49] it comes down to redistribution, right? [27:51] If any of these proposals are going to [27:53] be somehow the benefits of AI or the [27:56] money of AI will acrue to whether it's [27:59] the companies or rich individuals or [28:00] whatever and then or the government and [28:03] then the government's going to redistri [28:04] redistribute it back in some fashion [28:07] whether it's through creation of jobs uh [28:09] or whether it's through actual universal [28:12] basic income. And look, this is not a [28:14] political system that is terribly [28:15] hospitable to to that, right? Just just [28:19] uh you know, the widespread cutting of [28:20] checks um um to everyone based off of of [28:24] uh you know, tech profits. Uh and so I [28:28] think there's there's there's a lot to [28:29] be um concerned about there. I I don't [28:33] see the the economic effects being as um [28:39] significant in the short term, but uh [28:44] just wait to see if there is a [28:46] correction if there's a slowdown in the [28:48] data center buildout or or just if you [28:51] know if we relive 2008 in a kind of in [28:55] an with an AI cast to it, right? and and [28:58] there's like a massive or a 1999 kind of [29:01] crash, uh you can imagine what the [29:03] reaction is going to be toward the AI [29:05] companies then because they are as you [29:06] said they're propping up not just the US [29:08] economy right now but the data center [29:10] that are built out is propping up the [29:11] world economy uh because of the [29:13] literally trillions of dollars that are [29:14] now being invested. [29:15] >> Yeah. And you've also got this [29:17] distributional issue where the profits [29:20] from you know this kind of activity are [29:22] going to a smaller and smaller circle of [29:24] people and the general benefits are [29:27] really um not at all clear. So this is [29:30] an issue that uh I've been particularly [29:33] agitated about uh you know that and it [29:36] really has to do with what's the [29:37] function of work in a society because I [29:40] don't think that work that you are paid [29:42] for is simply a means of getting the [29:45] material prerequisites to living a life [29:48] you know so you can have housing food [29:50] shelter you know so forth uh it really [29:52] has to do with you know human dignity it [29:55] has to do with the fact that the society [29:57] thinks that what you are doing uh is [30:00] valuable enough that you're going to be [30:02] paid a salary for it. And if you take [30:05] that away and substitute some kind of [30:08] make work, you know, activities, you [30:10] know, some of these people say, well, [30:11] you know, you can pursue hobbies, but do [30:14] you think that millions of people are [30:16] going to be satisfied, you know, doing [30:17] hobbies, uh, instead of actually being [30:20] paid for, you know, for useful labor? So [30:22] I think that there's a potential you [30:25] know real social rupture that may uh [30:29] emerge uh as a you know as a as a result [30:32] of this. Um okay how about uh broadening [30:38] the uh aperture here? Uh we've been [30:42] talking about the US and China and not [30:44] about any other part of the world. But [30:47] we do have Europe, we have the global [30:49] south, we have a lot of countries that [30:51] are watching this, some of them kind of [30:54] in horror. Uh how do you think this is [30:56] going to affect um you know [30:58] international relations on that level [31:00] when these two big superpowers are the [31:03] only ones that seem to be moving forward [31:05] on uh this technology? [31:08] >> Yeah. So uh I I've been spending a lot [31:10] of my time in the last year or two uh in [31:13] places like Brazil and Japan and [31:15] certainly uh in Europe um thinking about [31:18] what you so-called middle power strategy [31:20] with with respect to AI. Let me just say [31:22] first on on your previous point about [31:24] dignity of work. you know, we've got [31:26] this project here in the law school uh [31:29] as part of the AI initiative uh doing [31:30] some work with the Vatican uh on on on [31:34] some of the issues that they are dealing [31:37] with with AI. And that's one of the [31:39] things that the Pope has said, right, is [31:41] that it's not just about the ability to [31:43] earn a living, but it's the dignity of [31:45] work and that if if AI takes that away, [31:47] then we've got uh real challenges. Now [31:49] on the on the middle power strategy or [31:52] you know rest of the world strategy I [31:55] mean different countries I think are [31:56] have different opportunities here and uh [31:59] different capabilities. I think the jury [32:01] is still out as to whether the so-called [32:04] Brussels effect is going to apply to AI. [32:07] Right? So the Brussels effect was the [32:09] idea that Europe regulates while the you [32:11] know US innovates uh but that u the [32:14] Europe sets the regulatory standard for [32:16] the world. That was probably true with [32:18] respect to their privacy law GDPR which [32:21] then became the kind of standard that uh [32:24] lamentable or not that that then [32:26] governed the the rest of the world. uh [32:28] but when the US is unable to act uh uh [32:31] you there the Europe is a huge market [32:34] and so it can it can set the standard. [32:36] It's sort of what California has done of [32:37] course with like auto emissions and and [32:39] may even do uh with respect to AI and so [32:42] I think a lot does ride on Europe's [32:45] position as the primary regulator of AI [32:48] right now because they are the ones who [32:49] have the most comprehensive law and so [32:52] will they be the uh de facto governors? [32:55] I think I I'm a little skeptical that [32:57] that is going to be the result here [32:59] because I think this is different than [33:01] say privacy legislation and others but [33:03] it's still possible uh that they are [33:06] going to be become that body and and all [33:09] of the AI companies with the exception [33:12] of meta and sort of X which signed on to [33:14] half of it signed on to the um codes of [33:17] practice that came out of the EU. And so [33:19] I think the EU is still uh going to be [33:23] out front as a regulator, but I do think [33:26] in terms innovator that Europe is um [33:30] it's in a bad spot when it comes to uh [33:33] trying to develop the technology and to [33:35] more importantly to diffuse the [33:37] technology. And so that is usually my [33:39] message when I'm talking to these kinds [33:41] of companies or countries which is that [33:44] look, you're not going to develop your [33:45] own foundation model. you're just not [33:47] going to be able to compete with the US [33:48] companies or with um China. Uh and so [33:52] you got to think about the different [33:54] layers of the AI stack. So some places [33:56] like Brazil, which has abundant energy [33:59] and sort of I've learned about how much [34:01] renewable energy uh the over 95% of [34:03] their uh power grid is now renewable. Uh [34:06] they're in a good position to be leaders [34:08] in you know if they need want to [34:11] building a data centers and all of that. [34:13] In addition, places like Japan and and I [34:16] think also in Europe, if they make it [34:19] easier for the technology to diffuse [34:21] into the economy, you could see native [34:23] app development, you know, and and real [34:26] competition um at the application layer [34:29] even if they can't do uh the foundation [34:31] layer. I think that you know it's still [34:34] going to be the case that the US and and [34:36] maybe China are and to some extent South [34:39] Korea uh which is important here are [34:42] going to be leaders on chip on the [34:44] design of chips but of course Taiwan is [34:48] still the most critical country in the [34:50] world when it comes to the AI supply [34:51] chain because um TSMC has just figured [34:55] out things that nobody else can in terms [34:57] of uh the building of of the [34:59] semiconductors. Uh and so I think each [35:02] one of these countries has has a [35:04] different role to play. And I and I [35:06] think that um [35:09] the competition between the US and China [35:11] is not just over the power of the [35:12] technology, but it's also the [35:14] reliability of the technology to be [35:16] distributed to all these other [35:18] countries. I I have I've been surprised [35:21] in my travels to hear how significant [35:24] the administration's decision with [35:26] respect to mythos and fable was in the [35:28] spring in terms of uh casting doubt on [35:32] the reliability of American AI models to [35:34] be there sort of and be there reliably. [35:37] And so the fact that the president could [35:38] just pull the plug on on on one of the [35:40] models that other countries were using [35:42] for cyber security and other things um [35:45] did send a chill down the spine of many [35:47] of these uh places where they were going [35:49] to deploy this. Uh so much so that for [35:52] example in in Brazil they're building [35:53] two giant supercomputing facilities. One [35:57] is Nvidia based and and sort of American [35:59] technology and one is Chinese [36:00] technology. And so they're trying to [36:02] hedge their bets. And so that is what [36:05] you you do see around the world is that [36:07] the these countries are trying to to [36:08] play both sides uh for the moment. Uh [36:11] and we'll see if that that continues. [36:13] And by the way, this is why the Chinese [36:16] strategy of open models is going to be [36:19] so critical for those middle powers, [36:21] right? Because they're going to be able [36:23] to adapt them to their domestic [36:25] circumstances. [36:26] >> Yeah. At a much lower cost. Uh we're [36:29] already getting quite a few questions in [36:31] the chat. So I would encourage anyone [36:33] that does have a question to uh uh put [36:36] it in there. We're going to finish in a [36:38] couple of minutes now. Uh yeah. So maybe [36:41] one question is related to those [36:44] open-source models. You know, China [36:47] destroyed the western telecoms industry. [36:50] you know, companies like Lucent and [36:53] Nortell by coming up, you know, Huawei [36:56] basically came up with a technology that [36:58] was not as good as the leading, you [37:01] know, switches that were being made by [37:02] the big telecom companies, but they were [37:04] a lot cheaper. Uh, and so many companies [37:07] said, "Okay, we don't need, you know, to [37:09] pay this premium for the best switch in [37:11] the world. The Chinese switches are uh [37:14] adequate." And it seems to me there's a [37:16] danger right now of the same something [37:18] similar happening in the AI world where [37:21] these Chinese open- source models are [37:23] being adopted by a lot of people [37:25] including a lot of people in the United [37:26] States because they're adequate. You [37:28] know, you don't need the most [37:30] sophisticated frontier model to do the [37:33] kinds of work that most companies, you [37:35] know, want AI to do. Uh so I do think [37:38] that I mean that's both a good thing and [37:40] a bad thing. It's not good for the big [37:41] companies in the United States. uh it [37:44] may be a way for this technology to um [37:48] spread you know to other parts of the [37:50] world that really can't afford to be at [37:52] the frontier. [37:54] >> Yes. Well, I also think it's sort of [37:56] unfortunate that the open closed debate [37:59] correlates with the geopolitical debate [38:00] of China versus the US. Now, it's not [38:03] completely the case, right? because Meta [38:05] and Nvidia and and others are um [38:08] developing open models and so I think [38:11] the hope there is that maybe they will [38:12] develop models that are um uh as capable [38:16] as Kimmy and and and the other uh [38:18] Chinese open models and I think there's [38:20] some cause for you know hope there uh [38:24] but you are right if you look [38:26] geopolitically and you look at at the [38:27] adoption in the last year of the Chinese [38:30] open models it's been quite striking you [38:32] know because of the cost of the of the [38:34] those models, whether in the US or or [38:37] around the world, you're seeing um the [38:38] Chinese models uh getting very popular [38:41] for the reasons that you say, which is [38:43] that they're good enough for most [38:44] things. You don't need to uh you know, [38:46] drive a Ferrari to the supermarket, [38:48] right? And and and that is uh you as [38:52] powerful and and impressive as as the [38:54] American frontier models are, you don't [38:57] just don't need them for most tasks that [38:59] you might need to use AI for. And so [39:02] especially given the adaptability of the [39:04] open models to particular kind of [39:06] economic or industry circumstances or [39:08] domestic you know political [39:10] circumstances uh uh that you would [39:13] expect there to be a preference you know [39:15] in a lot of sectors for to to have an [39:18] open model that you can then you know [39:20] tailor for your particular needs. [39:22] >> Yeah. By the way, I'll say one thing on [39:24] this, which is is a little bit deceptive [39:26] because even though the Chinese models [39:28] are open, um you're going to need some [39:31] massive infrastructure, right, just to [39:33] for inference to power these models [39:35] anyway. And so it's not as if you can [39:38] you can just build the everything [39:40] yourself in your garage uh with an open [39:43] model to then to then run at the scale [39:45] of which we want to, you know, uh use [39:47] these models. [39:49] >> Yeah. Okay. Well, that's too bad. I I [39:52] like doing things in my garage. [39:54] >> Well, when you were talking about [39:56] hobbies before, Frank, I I hope everyone [39:58] knows about I mean, because anyone who [39:59] follows your Instagram would know that I [40:01] think you'd be quite successful in a [40:03] hobby based economy, you know, whether [40:05] it's woodworking or anything else. [40:07] You'll do really well. [40:10] >> Okay, good. So, I think we will wrap up [40:11] this part of the session, uh this [40:14] conversation we've had. Uh, Annette, do [40:16] you want to take over in terms of um uh [40:20] going through some of the questions that [40:21] are in the chat that have been posed by [40:23] some of the people listening to this? [40:26] >> Absolutely. Happy to. Um, thank you and [40:28] and Nate for so for such an insightful [40:30] discussion so far. Um, as uh as Frank [40:33] said, we'll open up to questions right [40:35] now. Um, I encourage folks to submit [40:37] yours in the Q&A box that you see on [40:38] your council. Um but before we move on [40:41] to that, I just want to remind folks um [40:43] that today's webinar um is is um is [40:47] sponsored by uh the AI power and and and [40:50] new global strategy program which we'll [40:52] have running on campus u in March of [40:55] 2027. If you want more information, [40:57] please look at your console um and click [40:59] the links there. All right, we will move [41:02] on. Uh we're getting some questions [41:04] here, but I'll start with one that we [41:06] got earlier on. Um, okay. How much of [41:09] the issue around pacing the frontier [41:11] model stems from the current profit [41:13] above all at all cost mentality that [41:16] seems to be driving US businesses [41:18] overall? [41:21] Well, I think that um [41:24] you you can phrase it that way, but you [41:26] could also just say that that the AI [41:29] companies seem to have a winner take all [41:32] mentality that this is similar though in [41:35] many ways it's different but similar to [41:36] like Google search if to be the search [41:38] engine to be the model and so therefore [41:41] their their interest in pushing the [41:44] capabilities is you know from their [41:46] standpoint almost existential that [41:48] because there is a chance that only one [41:49] of them would win. Now, I happen to [41:50] disagree with that. Um, I think there's [41:52] going to be room for for several of [41:54] these companies. Uh, but it is, you [41:58] know, a sense that that uh they need [42:01] some outside force like the government [42:03] in order to discipline them uh uh [42:06] because otherwise the competitive [42:07] pressures of of the market are going to [42:09] lead them to just keep pushing the [42:11] capabilities and maybe not spend on [42:14] safety. Now, I think there's something [42:15] to be said by, you know, the arguments [42:17] of Jensen Wong and and David Saxs and [42:19] others, which is to say, look, I mean, [42:20] you have this in other industries and we [42:22] have, you know, people don't release [42:24] unsafe products. And so, you shouldn't [42:26] just blame competition to say, look, [42:28] regulate me before I kill again. Uh, and [42:31] and so, you know, I think there's [42:33] something to that, but at the same time, [42:34] because of the antitrust laws and [42:36] because of uh the absence of federal [42:38] regulation, we don't have the ability [42:40] for these companies to coordinate on [42:41] safety. [42:49] Thanks for that. Um, the next question [42:51] that I have here is which regul [42:53] regulatory risk are investors in compute [42:56] and data infrastructure underpricing [42:58] most today? Export controls, liabil [43:00] liability for model harms or data rules? [43:05] >> Okay, that's a very uh CNBC kind of [43:09] question there. Let me let me think if I [43:10] can if I can answer it. Um uh well which [43:14] which are they under underpricing? Um, [43:17] look, I tend to think that the most that [43:20] at some point in the next year, we will [43:23] have some kind of roll back that then [43:26] will have all these kind of knock-on [43:27] effects because of the of the crazy [43:29] circular financing between the the [43:32] whether it's in, you know, Nvidia and [43:34] OpenAI and Microsoft, but all the money, [43:36] it's very hard to track and that does [43:39] lead to a level of complexity opacity [43:41] that I think is not great for um the [43:44] sustainability of of the market. Having [43:47] said all that, I I I do think that, you [43:50] know, the long term for these companies [43:52] looks good. It's just a matter of of, [43:54] you know, what the pace uh looks like. [43:56] And so I would think that a slowdown in [44:00] um uh data center buildout or is the [44:04] thing that might then might lead maybe [44:06] into sort of public perception that we [44:09] are um [44:12] you know that that that there's a [44:14] correction. But, you know, you don't you [44:16] don't see that right now. They're just [44:17] trying to they're trying to find some [44:18] place, you know, any place that they can [44:20] build a data center, they're they're [44:21] trying to do it. [44:29] All right. Um, another question we have [44:31] here is not having AI safeguards and AI [44:35] regulations actively fueling AI fears [44:37] and affecting AI adoption, slowing [44:40] slowing progress in the US. [44:44] So, um, [44:47] I don't think that what whatever [44:49] regulation might get passed, whether at [44:51] the federal level or whether by states, [44:53] I don't think it's going to speak to [44:55] most people's fears about AI. I still I [44:58] think it's the right thing to do. Uh, [44:59] just because you want to regulate [45:01] things. Uh, yeah, but partly because you [45:03] want to chart a secure AI future. I [45:06] think it's good for the companies to [45:07] have uh settled legal rules. But um I [45:11] don't think law is going to solve the [45:13] public perception problem for the same [45:15] reason that if we were regulating social [45:16] media, I don't think people would then [45:18] suddenly say, "Oh, well, I I I love [45:20] these tech companies." Um that [45:23] previously I was saying we're hurting my [45:25] kids or uh destroying democracy. uh and [45:28] so I don't I don't think that regulation [45:30] is going to solve uh the anxiety or the [45:32] public opinion problem especially if in [45:35] the coming years uh the way we think [45:37] about AI is principally through the lens [45:39] of the labor force impacts and so if you [45:42] believe that AI is destroying your job [45:44] right uh it doesn't make a difference [45:45] how how much you're regulating data [45:46] centers or model development uh it still [45:49] has that effect and so I think the [45:51] politics of this are complex uh and [45:54] unsettled right now and um I think that [45:57] the the regulatory debate is part of [45:59] that uh part of that political [46:01] discussion, but it's not all of it. [46:09] >> All right, thank you for that. Um, okay. [46:12] What are some possible implications on [46:14] financial market side of things? [46:18] >> Well, you're already seeing it. I mean, [46:19] all the all the big financial firms are [46:22] trying to integrate AI into all of what [46:24] they do. And so you know um whether it's [46:29] trying to predict the trajectory of [46:32] individual companies the market itself I [46:34] mean all of the banks are using uh AI [46:36] and so um I think that that you would [46:39] expect uh that that would be a sector [46:41] where you'd see it uh rolled out very [46:44] quickly. Um in terms of like regulation [46:48] though of of the financial industry. I [46:50] mean, one of the things as as Frank and [46:52] I were talking before about getting a um [46:56] uh independent regulatory body for AI up [46:58] and running, which is something, by the [46:59] way, Cass Sunstein has a proposal on [47:01] this, but also Demisabas has put out uh [47:04] a a early on a proposal to say that we [47:08] need to regulate AI in the same way we [47:10] do regulate financial institutions. So [47:12] something like FINRA should be um [47:15] proposed for the AI companies. Now [47:18] that's going to be a kind of public [47:19] private partnership idea and standard [47:21] setting and and all of that but without [47:23] with some kind of accredited third party [47:25] observation. Uh and so I think that in [47:29] many ways while while there's going to [47:31] be important regulation [47:33] uh regarding the diffusion of AI in the [47:35] financial sector I think we can also [47:37] learn a little bit from the financial [47:39] sector when it comes to regulating AI. [47:45] >> Okay. Um, how should policymakers [47:48] address application regulations without [47:51] also squatch squashing muchneeded AI [47:53] research? [47:56] >> Well, I don't I don't think those uh [47:59] those motives are necessarily intention. [48:01] I mean, I think that um if if the [48:04] questioner is actually talking about the [48:06] applications, I mean, so much of this is [48:09] downstream of the of the development of [48:11] the main AI tools, right, of the of the [48:14] models. And so I I think that we really [48:17] need to be encouraging um uh diffusion [48:20] at the application layer. And this is [48:22] what I was trying to say before about uh [48:24] these countries outside the US and China [48:26] where I do think on the development of [48:28] applications that they're going to be [48:30] that is where they need to really [48:31] innovate. Um, [48:34] you know, I I I think that there's uh [48:38] the research questions and maybe what [48:41] the questioner is asking is how can you [48:43] do research at the major labs uh while [48:46] also pushing uh the frontier or or or or [48:49] pacing the frontier. And I do think that [48:53] uh you know one of the [48:55] outcomes of this hugging face incident [48:58] is in part that you know these were [49:00] unreleased models that or or un you know [49:04] uh deployed models that ended up causing [49:07] harm. And so these companies really need [49:10] to lock down their sandboxes to make [49:13] sure that in the research that they're [49:15] doing on the models that it's done you [49:16] know without internet access. Enthropic [49:19] actually had an announcement about that [49:20] today. Uh and so that's the way you you [49:24] try to protect research. You make sure [49:25] that it's it's done in an environment as [49:28] as you would with biological, you know, [49:30] gene editing or any other kinds of [49:31] things that's segregated from the [49:34] outside. [49:40] >> Okay. Um I have another question here. [49:43] What is the worst case scenario if the [49:45] US does not come up with the proper [49:46] and/or unified regulation in the next [49:49] say 18 months? [49:51] >> Let me I I don't want to be too da here [49:54] on this uh I tend to be a little more um [49:57] I'd say in the normal technology camp [50:00] let me say this that even if the US [50:03] comes up with the perfect regulation, [50:05] however we define that, it's not clear [50:07] to me it's going to stop some of the [50:10] doomsday scenarios, right? And so it is [50:12] going to be very challenging to deal [50:14] with this loss of control problem [50:17] especially if we sort of posit and and [50:20] Frank and I didn't really even get into [50:21] this but it's almost hard to even [50:23] conceive what does opensource agentic [50:26] super intelligence look like right where [50:29] you have agents that are off of open [50:30] models that are essentially you know out [50:32] there um and so that that is you know [50:36] that's going to pose a fundamental [50:38] challenge I think that um if the [50:41] you know, doesn't regulate uh and we are [50:43] the biggest sort of game in town. I [50:45] think you'll see a series of small level [50:49] um I don't want to call them [50:50] catastrophes, but incidents, you know, [50:52] maybe something with the electrical [50:54] grid, maybe something with stock [50:55] markets, those kinds of things. [50:57] Hopefully, I I I tend not to be in the [50:59] kind of existential doomer risk camp [51:01] that this is that that we're going to [51:02] have a terminator scenario that leads [51:04] to, you know, human extinction. But I [51:06] think that, you know, in the same way [51:07] that we're going to see incredible [51:08] benefits, possibly cures for cancer and [51:10] other diseases, um, incredible, you [51:13] know, creativity, other kinds of, um, [51:16] great benefits from AI, you're going to [51:18] have some of these periodic, uh, you [51:21] know, harms, uh, that that happen. Uh, [51:23] this is what happens with new [51:24] technologies, and this is a particularly [51:27] powerful one. I mean, as many have said, [51:29] right, this is like the industrial [51:30] revolution, but just maybe 10 times as [51:32] significant and 10 times as fast and um [51:35] you know, it's going to take a long time [51:36] for us to to sort of come to grips with [51:39] it. [51:41] >> Yeah. Truly don't know, but but I love [51:44] the optimistic view. Um okay, we talked [51:47] a lot about um we talked a lot about [51:49] jobs. Um so there's a question here. uh [51:52] what human possibly regulatory related [51:55] jobs do you see emerging in 10 years [51:58] time that don't exist today? [52:01] >> Well, I still think that there are [52:03] existing jobs that um you know are going [52:07] to be retoled, you know, for the AI era. [52:10] We think a lot about this in the law [52:12] school. I'm actually teaching a class [52:13] now on the use of legal AI tools to the [52:16] first year law students. And um [52:20] you know what we're seeing with law [52:21] firms, law firms don't seem to be [52:23] reducing hiring, but what they what they [52:25] end up doing is that um the senior [52:28] partners are just more capable now [52:30] because they can use AI to do a lot of [52:31] the stuff that the junior folks were [52:32] doing. And that that is, you know, [52:35] something that I think we should be [52:36] legitimately concerned about on the [52:37] labor side, which is that the young [52:39] people who are coming out uh especially [52:41] if they're not AI uh trained are going [52:44] to have a difficult time. Um, but as to [52:46] what what kinds of uh jobs there might [52:50] be in the future that that we don't see [52:52] now, it's it's almost hard for me to um [52:55] describe except to say I can see in [52:58] every industry AI skills being, you [53:02] know, a a new kind of job. And so, uh, [53:04] just to give you an example, I I'm [53:06] spending a lot of time with these image [53:08] generation or these video generation [53:09] tools right now, just experimenting with [53:11] them. I've developed an AI avatar of [53:13] myself that can give an oral exam to my [53:15] students that so they can actually have [53:17] a conversation. And so this is something [53:20] I never would have done, you know, 10 [53:22] years ago. I the idea of I mean the [53:24] technology wasn't there, but even the [53:25] the idea of of experimenting with that [53:27] task is not something a professor would [53:29] have done. And so I think there are [53:30] going to be all kinds of things like [53:32] that where where in particular [53:35] industries there going to be firms that [53:36] are going to be hiring people who have [53:38] these AI skills uh to perform tasks that [53:41] they didn't they didn't do before. [53:43] >> I think also uh one new job category is [53:48] actually managing AIs. [53:49] >> Yeah. [53:50] >> Right. Because most agentic AIs you [53:52] can't just let them run and expect that [53:55] they're going to do you know the right [53:56] thing. So people, you know, it's and one [53:59] way to think about the future AIs is [54:02] that they're like uh, you know, they're [54:04] like workers. They're they're going to [54:06] do stuff that human beings used to do, [54:08] but all workers need to be managed. And [54:10] so having the technical knowledge to [54:13] manage a machine is going to be an [54:16] important uh capacity uh that you know [54:19] is going to exist in the future. [54:22] >> That's right. And I think there's going [54:23] to be there going to be lots of roles [54:25] that are defined like that, right? That [54:27] that there's going to be sort of AI [54:30] management is implied in almost any [54:32] senior role at a company. [54:37] >> Yeah. Um, okay. I think we have time for [54:40] about one more and I'm going to leave us [54:42] with sort of a fun one that I see here. [54:44] Um, what's your take on Bostonramm's be [54:47] careful what you wish for view in his [54:49] book Deep Utopia? [54:52] I'm I'm going to leave that one to [54:53] Frank. Frank, do you have do you have [54:54] thoughts on that? [54:56] >> Well, I uh don't really believe in any [55:01] of the big doomer scenarios. Uh I think [55:05] that it's actually hard to do the kind [55:08] of damage that a lot of them are [55:10] expecting. And I suspect that the degree [55:12] of human control is actually going to be [55:15] uh something that will be developed. A [55:18] lot of it is going to require AI. you're [55:20] going to have AIs that are going to be [55:21] managing other AIs and defending from uh [55:25] AIS. Um, and I'm not sure that that's a [55:28] process that will simply get uh, you [55:30] know, get out of control. Uh, I really [55:32] do think that people ought to focus not [55:35] on these uh, we're all going to die [55:37] scenarios, but really on the uh, [55:40] near-term very serious harms. you know, [55:43] like Chase Bank, you know, uh, a million [55:47] of its customers get their checking [55:48] accounts emptied out, you know, this [55:50] sort of thing. Uh, and that's pretty [55:52] bad. You know, it's going to lead to a [55:54] real crisis in the United States. Uh, [55:56] and that seems to me much more likely [55:58] than any of the, you know, the pdoom uh, [56:00] kinds of scenarios that people are [56:02] positing. that I agree with and and and [56:05] that was what I was trying to emphasize [56:06] before, which is that yeah, I could see, [56:08] you know, a series of kind of real AI [56:11] breakthroughs and then accompanied by [56:13] these AI harms and these AI incidents, [56:15] whether you're talking about the [56:16] electrical grid or stock market or or or [56:18] something else. [56:22] >> Great. All right. Well, thank you again, [56:25] Frank and Nate, for this uh talk on such [56:27] an important topic. Um to everyone in [56:29] the audience with us live today, thank [56:31] you all for your questions and [56:33] participation. Um I want you I want to [56:35] remind you that today's session was [56:37] recorded and a link to the recording [56:38] will be made available to all of you [56:40] within a week. I hope everyone has a [56:42] wonderful rest of your day and we'll see [56:44] you all next