A16z, Box Leaders Warn AI Lab Doomerism Invites Premature Regulation
a16z
Frontier AI labs promote speculative extinction risks as a researcher retention tool, threatening to trigger heavy-handed government utility regulations before real-world failure modes are understood.
Frontier lab claims of non-zero human extinction risks paradoxically invite heavy-handed nationalization and utility-style regulation, even as practitioners privately treat doomer rhetoric as an internal talent-retention tool. (6:06)
Premature AI safety regulation risks repeating past policy errors; historical industries like aviation, automotive, and early computing developed effective safety frameworks only after weathering empirical, real-world failure modes. (23:33)
Autonomous AI agents fundamentally threaten corporate security architectures by eliminating human fatigue and latency, exposing internal enterprise SaaS tools to rapid agent-driven denial-of-service and access control failures. (34:28)
The epicenter of AI innovation is migrating away from frontier labs building conversational 'beings' toward external software stacks that treat LLMs as high-speed probabilistic decision engines. (48:38)
Detailed notes
## Background
Published on YouTube by a16z on September 26, 2026, this episode features host Erik Torenberg in conversation with Box CEO Aaron Levie, a16z General Partner Martin Casado, and former Microsoft executive Steven Sinofsky. The panel debates AI safety, governance, and enterprise security, drawing lessons from computing history and arguing that future breakthroughs will emerge from the software ecosystem surrounding models rather than the frontier labs themselves.
## TL;DR
The panel sharply challenges mainstream catastrophic AI safety rhetoric, arguing that existential extinction claims often function as an internal talent-retention strategy that risks inviting regulatory capture and premature statutory lock-in. Drawing on historical parallels from early internet worms, Windows security, aviation, and automotive safety standards, the speakers demonstrate that sensible governance and resilient architecture must be forged through empirical failure modes rather than speculative doomsday policy. They explain how autonomous agent swarms will inevitably upend enterprise perimeter security and internal access control, requiring a fundamental return to secure-by-design systems. Ultimately, they argue that while frontier labs remain culturally fixated on building anthropomorphic conversational 'beings,' the most impactful AI breakthroughs are migrating toward external software architectures that harness models as probabilistic decision and routing engines.
## Atmosphere & style
The discussion has the rapid-fire, high-bandwidth energy of an insider technical salon, characterized by seasoned cynicism toward Washington politics and academic doomerism. Sinofsky, Casado, and Levie trade vivid historical war stories—spanning Cold War Pershing missile TEMPEST specs, Windows 95 zero-day worms, and early Stanford server intrusions—delivering their arguments with sharp wit and pragmatic engineering realism. The tone is deeply intellectually grounded yet irreverent, cutting through Silicon Valley hype to ground the future of AI in systems architecture, physical constraints, and platform economics.
## Bulletin
- [06:06](https://www.youtube.com/watch?v=TLJNJDf2XGo&t=366s) Frontier lab claims of non-zero human extinction risks paradoxically invite heavy-handed nationalization and utility-style regulation, even as practitioners privately treat doomer rhetoric as an internal talent-retention tool.
- [23:33](https://www.youtube.com/watch?v=TLJNJDf2XGo&t=1413s) Premature AI safety regulation risks repeating past policy errors; historical industries like aviation, automotive, and early computing developed effective safety frameworks only after weathering empirical, real-world failure modes.
- [34:28](https://www.youtube.com/watch?v=TLJNJDf2XGo&t=2068s) Autonomous AI agents fundamentally threaten corporate security architectures by eliminating human fatigue and latency, exposing internal enterprise SaaS tools to rapid agent-driven denial-of-service and access control failures.
- [48:38](https://www.youtube.com/watch?v=TLJNJDf2XGo&t=2918s) The epicenter of AI innovation is migrating away from frontier labs building conversational 'beings' toward external software stacks that treat LLMs as high-speed probabilistic decision engines.
## Key takeaways
- [02:59](https://www.youtube.com/watch?v=TLJNJDf2XGo&t=179s) Framing development slowdowns as 'pacing' is a misnomer for standard engineering hygiene, which temporarily introduces testing friction to build enterprise trust.
- [03:50](https://www.youtube.com/watch?v=TLJNJDf2XGo&t=230s) Public existential risk rhetoric operates at an extreme scale of urgency that renders incremental corporate safety policy proposals visibly inadequate to both regulators and pause advocates.
- [06:06](https://www.youtube.com/watch?v=TLJNJDf2XGo&t=366s) Extinction-risk narratives function primarily as an internal HR mechanism to retain elite researchers who trade in doomerism, rather than reflecting sincere operational practices.
- [10:36](https://www.youtube.com/watch?v=TLJNJDf2XGo&t=636s) Tech executives naively assume government lobbying yields bespoke regulatory outcomes, underestimating how legislative compromise typically results in administrative overreach or toothless half-measures.
- [15:14](https://www.youtube.com/watch?v=TLJNJDf2XGo&t=914s) Admitting non-zero extinction risks logically pushes governments toward nationalization or dense compliance regimes, creating defensive utility moats that entrench incumbents and smother open-source challengers.
- [25:46](https://www.youtube.com/watch?v=TLJNJDf2XGo&t=1546s) The 1986 Computer Fraud and Abuse Act established a durable legal precedent that regulatory frameworks must govern harmful application-layer actions rather than outlawing underlying foundational technologies.
- [27:32](https://www.youtube.com/watch?v=TLJNJDf2XGo&t=1652s) The cybersecurity community views frontier AI lab incident postmortems as evasive corporate communications that bypass established disclosure standards like CMU CERT and CVE.
- [30:25](https://www.youtube.com/watch?v=TLJNJDf2XGo&t=1825s) AI safety debates must be grounded in physical systems architecture, thermodynamics, and hardware covert channels rather than speculative sci-fi scenarios that ignore unsolved baseline engineering challenges.
- [34:28](https://www.youtube.com/watch?v=TLJNJDf2XGo&t=2068s) Autonomous agent swarms break traditional enterprise security perimeters because they operate at superhuman speed without fatigue, rendering legacy access models that rely on compliant human behavior obsolete.
- [41:43](https://www.youtube.com/watch?v=TLJNJDf2XGo&t=2503s) Mandating GDPR-style consent banners for individual agentic actions shifts liability onto end users while inducing severe warning fatigue identical to historical Windows User Account Control prompts.
- [46:13](https://www.youtube.com/watch?v=TLJNJDf2XGo&t=2773s) Aviation, pharmaceuticals, and automobiles operated for decades before formal safety regimes were established, demonstrating that preemptive regulation risks locking in flawed architectures before true failure modes appear.
- [48:38](https://www.youtube.com/watch?v=TLJNJDf2XGo&t=2918s) Conversational text chat is an inefficient human-computer interface; structuring LLMs as discrete decision and selection engines provides orders-of-magnitude faster, cheaper, and more accurate execution.
- [50:30](https://www.youtube.com/watch?v=TLJNJDf2XGo&t=3030s) Treating LLMs as probabilistic routing engines revives theoretical concepts from 1960s simulation computer science, shifting modern software engineering from deterministic logic to stochastic control flow.
- [53:01](https://www.youtube.com/watch?v=TLJNJDf2XGo&t=3181s) Frontier AI labs have overlooked compact decision-engine tooling because their cultural mission is focused on creating synthetic autonomous 'beings' that converse in natural language.
- [53:01](https://www.youtube.com/watch?v=TLJNJDf2XGo&t=3181s) AI innovation has decisively expanded into the software layer surrounding foundational models, where third-party ecosystems will face platform absorption and Sherlocking dynamics as platforms mature.
## Detailed walkthrough
### [00:00](https://www.youtube.com/watch?v=TLJNJDf2XGo&t=0s) Cold Open: The Emerging Rifts in AI Governance, Agents, and Innovation
The episode opens with a rapid montage of provocative warnings and thesis statements from the conversation ahead. The speakers argue that premature AI regulation is fundamentally counterproductive: regulating before understanding the actual failure modes solves nothing and simply leaves the underlying risks intact. As one speaker frames the dilemma of frontier AI development, "You will the thing into being, but you haven't figured out how to control it." A central theme is the growing friction between the frontier model labs and the traditional cybersecurity community. The security establishment increasingly views the labs through two critical lenses: their engineering practices appear "sloppy," and their disclosures about security incidents and model behaviors are viewed as incomplete and non-transparent.
The stakes escalate dramatically as the discussion moves from theoretical risks to autonomous execution. While frontier lab employees publicly float estimates like a "10% chance of species extinction," autonomous agent swarms present a far more concrete and immediate operational challenge. Described as "roaming drones but like times 10,000," swarms can effortlessly mistake a benign instruction for a harmful one. To manage this exposure, enterprise software will require an entirely new internal observability and governance layer dedicated to auditing authentications and API invocations at scale. Geopolitically and commercially, the landscape is shifting: while the United States ceded leadership in tech antitrust roughly fifteen years ago, Europe is aggressively stepping in because they have "nothing to lose." Ultimately, the center of gravity in AI innovation is migrating away from the foundational model labs toward the external software ecosystems being constructed around them.
```mermaid
flowchart TD
accTitle: Frontier Labs vs Enterprise Reality
accDescr: Diagram illustrating the rift between frontier AI lab safety narratives and practical enterprise security needs.
subgraph FrontierLabs["Frontier AI Labs"]
A["Frontier Model Training"] --> B["Existential Risk Discourse\n('10% chance of species extinction')"]
A --> C["Regulatory Push &\nPacing Proposals"]
end
subgraph EnterpriseInfra["Enterprise & Security Reality"]
D["Agent Swarms at Scale\n('10,000 roaming drones')"] --> E["Audit & Control Layer\n(Auth, permissions, API tracking)"]
F["Cybersecurity Community Pushback\n(Labels lab practices 'sloppy' & incomplete)"] --> E
end
C -.->|"Threat of regulatory capture / slowing progress"| EnterpriseInfra
```
[00:52](https://www.youtube.com/watch?v=TLJNJDf2XGo&t=52s) Welcome Back and VC Banter
Erik Torenberg convenes the panel—Box CEO Aaron Levie, a16z General Partner Martin Casado, and former Microsoft executive Steven Sinofsky—opening with lighthearted banter about how long it has been since they last recorded together. The group trades friendly jabs about changes in facial hair, with someone noting they never thought they would see this specific panel reunite. The humor quickly turns to Martin Casado's venture track record, teasing that he has been too occupied building "hundred billion dollar companies" to appear on podcasts. Casado plays along with typical venture capital self-deprecation, quipping that he has at least been busy "taking credit for them, as VCs do."
[01:12](https://www.youtube.com/watch?v=TLJNJDf2XGo&t=72s) Reacting to the Safety Discourse: Dario Amodei’s Post and the Risk of Regulatory Capture
Erik steers the conversation into its first major debate: "Pacing the Frontier" and the heated discourse surrounding AI safety and government intervention. Aaron Levie defers initially to Martin Casado, joking that Martin has been fighting in the trenches of the regulatory "ground wars." Levie then frames a baseline position of consensus before teasing where the group’s perspectives might diverge. He stresses that anyone building at the frontier must do so with rigorous security, strict governance, and deep alignment research. Enterprise diffusion of AI is impossible without it; businesses will simply not adopt or trust systems that behave unpredictably or repeatedly compromise security boundaries.
Turning specifically to Anthropic CEO Dario Amodei's essay on frontier pacing and AI safety, Levie admits he found very little in the technical substance to disagree with. The essay's emphasis on sandboxing, robust safety evaluations, and enhanced testing protocols reflects sensible engineering hygiene. However, Levie highlights the broader systemic risk: how technical safety recommendations are weaponized once they enter the political realm. The true danger lies in safety rhetoric becoming an engine for regulatory capture—erecting steep regulatory moats that entrench incumbents, stifle challengers, and give politicians cover to enact blunt policies, such as halting data center expansion.
- **Baseline Enterprise Requirement:** Enterprise AI adoption depends entirely on predictable safety, governance, and verifiable system integrity, rather than models that pose unauthorized access risks.
- **The Pacing Proposal:** Dario Amodei's framework focuses heavily on sandboxing, empirical stress testing, and structured evaluations before frontier deployment.
- **The Political Downside:** While the engineering measures are defensible on paper, the downstream risk is that lawmakers distort the safety narrative into heavy-handed barriers to entry, anti-competitive restrictions on smaller labs, or arbitrary bans on physical infrastructure like data centers.
[02:59](https://www.youtube.com/watch?v=TLJNJDf2XGo&t=179s) Good Engineering vs. Regulatory Capture
The conversation digs into the substantive debate around pacing AI advancement and the risk of regulatory capture. While the practical engineering challenges of AI safety are critical to the technology’s broad adoption, the regulatory prescriptions emerging from the discussion risk locking in incumbents under the guise of safety. Aaron Levie emphasizes that framing this as "pacing" might simply be a misnomer for standard engineering hygiene. In software and systems engineering, rigorous testing and safety safeguards inherently cause a slight slowdown, but that very friction is what ultimately accelerates diffusion into the market: enterprises and consumers will never adopt or trust AI if it behaves unpredictably.
```mermaid
flowchart LR
accTitle: Engineering Hygiene vs Diffusion
accDescr: Diagram showing how rigorous engineering hygiene temporarily slows pacing but creates user trust, ultimately accelerating market diffusion.
subgraph DevCycle ["Engineering Safeguards"]
A["Rigorous Hygiene & Testing"] --> B["Controlled Pacing"]
end
subgraph MarketImpact ["Adoption Loop"]
B --> C["User & Enterprise Trust"]
C --> D["Accelerated Diffusion"]
end
```
The underlying issue is not the pragmatic technical work, but the surrounding "atmospherics." The conversation cannot happen in a vacuum when laboratory leaders validate existential risk narratives in the public sphere while simultaneously trying to pitch measured engineering checkpoints.
[03:50](https://www.youtube.com/watch?v=TLJNJDf2XGo&t=230s) The Flawed Atmospherics of Species Extinction
The debate shifts to the dramatic disconnect between apocalyptic rhetoric and corporate compromise. When an Anthropic employee claims there is a 10% chance of human species extinction and CEO Dario Amodei appears on television that same day echoing that he agrees more than disagrees with that assessment, the entire safety dialogue warps. Proposing measured "pacing" in the face of literal extinction rings hollow; if an executive genuinely believes an existential catastrophe is on the table, offering a policy blog post on pacing sounds like a "milquetoast capitulation that's totally not adequate for the task at hand."
The optics fail because the rhetoric and the proposed remedy are operating on completely different scales of urgency.
[04:28](https://www.youtube.com/watch?v=TLJNJDf2XGo&t=268s) Why Pacing Fails Both Regulators and Pause Advocates
Martin Casado challenges the entire premise of "pacing," arguing that speed is completely orthogonal to security. As an analogy, building a nuclear weapon slowly does not make anyone feel safer than building it quickly—the velocity of construction does not change the destructive nature of the output.
By pitching pacing, the frontier labs attempt to split the difference between two diametrically opposed camps:
- **The "Pause" and AI Doomer Faction:** They dismiss pacing because it does not stop development; to them, it simply means building existential threats at an incremental cadence.
- **Government Regulators:** They remain unsatisfied because the technology is still being released into the wild without rigid oversight.
Casado contends that frontier labs are genuinely attempting to act pragmatically, but their communication strategy backfires by trying to appease internal fringes while courting regulators. Instead of appeasing both sides, it leaves both alienated. To restore credibility, frontier labs must confront the existential risk (x-risk) narrative head-on and state plainly that their frontier work will not destroy humanity.
As the segment concludes, Aaron Levie pushes back to examine the psychological tension inside frontier labs: how the industry should handle the researcher who occupies that paradoxical Venn diagram—working tirelessly on advancing state-of-the-art models every day while remaining terrified of the very systems they are building.
[06:06](https://www.youtube.com/watch?v=TLJNJDf2XGo&t=366s) Nationalization vs. An "HR Problem": Addressing Existential Risk
The discussion centers on the paradoxical mindset held by certain researchers within frontier artificial intelligence labs: the conviction that because AI poses profound dangers, they personally must remain at the bleeding edge to guide it safely. While acknowledging that this persona genuinely exists in Silicon Valley—individuals driven by the perspective of *"we have to be at the forefront of AI, but I'm also terrified of it, which is why I must build it"*—Martin Casado directly confronts the logic. Drawing on his past work at Lawrence Livermore National Laboratory on classified nuclear weapons programs, Casado emphasizes that when a technology truly presents authentic existential risk, the institutional response is already well-understood: you nationalize the technology and implement stringent, proven state-level controls.
If the most knowledgeable researchers inside the frontier labs genuinely believed their models presented an imminent species-level threat, nationalization would be the only coherent path forward. In reality, Casado notes that in private conversations, the vast majority of sensible practitioners do not believe this narrative; only a very small fraction does. Consequently, what is publicly framed as an existential safety crusade is, in truth, an internal "HR problem." Frontier companies find themselves catering to apocalyptic rhetoric simply to attract, appease, and retain top talent who treat doomerism as a form of social and academic "researcher currency." Casado argues forcefully that solving an internal corporate recruiting and retention problem is completely the wrong justification for demanding a national lockdown on an exceptionally promising general-purpose technology. While practical operational security concerns across compute epochs are legitimate and sensible, they cannot logically be reconciled with catastrophic "X-risk" theater.
- **The Nuclear Precedent:** Genuine existential threats, such as nuclear weapons research at national labs, are governed by strict state ownership and hard operational security, not voluntary corporate pledges.
- **"Researcher Currency":** Much of the apocalyptic AI safety narrative functions primarily as an internal talent-management mechanism to placate anxieties among elite researchers.
- **Regulatory Disconnect:** Prudent cybersecurity policies and baseline system safeguards cannot be intellectually reconciled with sweeping policy demands predicated on extinction risks.
```mermaid
flowchart TD
accTitle: AI Safety Claims vs Real Responses
accDescr: Contrasting genuine existential risk responses with the reality of current AI lab safety posturing.
subgraph InternalDynamics["Current AI Lab Frontier Dynamic"]
D["Elite Researcher Anxiety & Ideology"] --> E["Treated as 'Researcher Currency'"]
E --> F["Corporate HR & Retention Problem"]
F --> G["Apocalyptic Rhetoric Used to Justify Sweeping Regulations"]
end
```
---
[07:48](https://www.youtube.com/watch?v=TLJNJDf2XGo&t=468s) The Fiction of "Pacing" Without a Schedule
Steven Sinofsky dismantles the concept of "pacing" the frontier, pointing out that the premise of intentionally slowing down or pacing development is fundamentally meaningless when no baseline schedule exists in the first place. The frontier labs have never published a timetable detailing when advanced capabilities or catastrophic failure modes are projected to arrive. Without an established timeline, claiming that progress has been deliberately throttled is disingenuous—resembling rumors in the tech press declaring that an unannounced future iPhone is "running late," or Aaron Levie quipping that his long-rumored Apple car was delayed. To declare that an effort is moving slower, an observer must know the exact rate at which it was originally scheduled to advance.
Beyond the definitional absurdity, Sinofsky highlights the stark contradiction between safety PR and commercial reality: nobody outside the echo chamber buys the claim of restraint. Far from pacing themselves, the leading labs have remained at an absolute "dead run," raising unprecedented billions of dollars and driving user growth and capability scaling faster than any previous technology cycle. While Levie observes that the internal argument often hinges on staggering the release of public-facing models while internal frontier capabilities remain far ahead of what the public can access, Sinofsky’s core critique remains: claiming credit for "pacing" without an objective baseline is pure narrative posturing.
- **The Baseline Paradox:** You cannot define an activity as moving slower or being paced without an established benchmark or predetermined delivery schedule.
- **The Capital Contradiction:** Claims of self-imposed restraint run counter to aggressive fundraising rounds, unprecedented compute cluster buildouts, and rapid commercial deployment.
- **Internal vs. External Gap:** While labs retain advanced internal builds unavailable to the public, gating external product drops does not substantiate claims of systemic safety pacing.
[09:09](https://www.youtube.com/watch?v=TLJNJDf2XGo&t=549s) The "Pacing" Paradox and the Missile Factory
The group tears into the disingenuous rhetoric coming out of frontier AI labs that publicly advocate for "pacing" their own releases while actively begging the state to step in. It is contradictory to hype an existential threat that is supposedly terrifying enough to upend society while simultaneously pitching a technology that still regularly fails at basic enterprise tasks like drafting a legal brief. Furthermore, asking the federal government to mandate pacing makes little sense: if a company genuinely believes its own product poses an existential danger to humanity, the only coherent response is to stop building it rather than waiting for an external order.
```mermaid
flowchart TD
accTitle: The Frontier Lab AI Safety Paradox
accDescr: Visualizing the contradiction between lab warnings and lobbying for government regulation.
subgraph LabRhetoric["Frontier Lab Posture"]
A["Hyping Existential Catastrophe"]
B["Lobbying State to Mandate 'Pacing'"]
end
subgraph Reality["The Operating Reality"]
C["Models Still Fail at Everyday Tasks"]
D["Logical Choice: Just Stop Building"]
end
A --> D
B --> C
```
This dynamic prompts a recollection of working on the factory floor of a defense contractor during the Cold War. As a college student wheeling secure PCs around on carts, one speaker recalls Catholic nuns showing up to protest by pouring blood on nuclear missiles. At the time, campus politics gave students a stark choice: either protest the weapons to keep them off campus or help build them. Yet those very Pershing missiles were what ultimately helped bring the Cold War to an end through deterrence. By contrast, "pacing" in AI is described as a fuzzy non-word that sits uncomfortably between moving forward and stopping entirely—a timid middle ground that satisfies nobody.
- The idea that an AI is simultaneously an apocalyptic superintelligence and prone to ruining standard enterprise workflows exposes the rhetorical split in lab messaging.
- If founders genuinely believe their code poses an imminent threat to human survival, the straightforward remedy is to cease development rather than lobby for regulatory permission slips.
- The Pershing missile standoff illustrates that high-stakes technological arms races have historically produced decisive geopolitical deterrents rather than safe, negotiated middle roads.
---
[10:36](https://www.youtube.com/watch?v=TLJNJDf2XGo&t=636s) The Naive Tech View of Government Compromise
The discussion turns to tech leaders who treat government lobbying as a sport where they assume policymakers will deliver exactly what the industry requests. This expectation reveals a profound misunderstanding of statecraft. Lawmakers and regulators listen to a wide array of competing stakeholders, and the machinery of governance operates fundamentally on political compromise. As a result, the legislative output satisfies almost no one: one hundred percent of the time, the resulting rules either do not go nearly far enough to fix the perceived issue or overshoot wildly and stifle legitimate activity.
Founders who lobby Washington to enforce an artificial pacing schedule will never achieve a tailored, calibrated velocity. Instead, invoking regulatory power triggers statutory mechanisms that industry leaders cannot control, inevitably saddling them with the wrong speed, poorly designed mandates, and heavy-handed compliance structures.
- Technologists mistakenly assume that briefing regulators will allow them to talk their way into bespoke policy outcomes.
- Regulatory output invariably swings to extremes, either implementing toothless half-measures or creating crushing administrative overreach.
- Requesting government-mandated pacing guarantees an outcome that distorts the pace of innovation rather than optimizing it.
---
[11:32](https://www.youtube.com/watch?v=TLJNJDf2XGo&t=692s) "You're Asking Us to Regulate What?" and the 2028 AI Election
Recalling memorable exchanges between tech figures like David Sacks and political officials—where the government effectively pushed back on vague industry requests with confusion over what, precisely, tech executives expected them to regulate—the panel observes that once the regulatory machinery begins to churn, there is no slowing it down. Silicon Valley's open invitations have now turned artificial intelligence into an election issue spanning every political party, jurisdiction, and level of government, creating an sprawling basket of disparate regulatory proposals.
Looking ahead, the group anticipates that the 2028 presidential campaign will serve as a full-blown referendum on artificial intelligence. Every candidate up and down the ballot will be forced to stake out a position on model capabilities, safety guardrails, and national competitiveness. The core challenge, however, is that defining a coherent political platform around AI remains completely unsettled, because nobody yet agrees on the exact nature of the risks or the economic consequences the technology will produce.
[12:21](https://www.youtube.com/watch?v=TLJNJDf2XGo&t=741s) Framing the Pro-AI Narrative: Losing the Vocabulary Battle
The conversation turns to the political challenge of championing artificial intelligence, particularly why it is nearly impossible for a political candidate to run on an overtly pro-AI platform. The core difficulty is that the positive case for AI is inherently nuanced, nebulous, and requires far too many words to explain, leaving its defenders sounding reactive and defensive. As a result, mainstream political discourse gravitates toward either competing over how stringently to regulate the technology or trying to sidestep the topic altogether.
The panel laments that the tech industry has completely ceded the framing and linguistic high ground to critics and pessimists. The entire regulatory debate is defined by loaded, alarmist terminology chosen by people who want to halt progress: terms like "pause," "swarms," and "rogue" models. Because critics picked the vocabulary, advocates are forced into an uphill battle where they must first explain why those words are inaccurate and invent new counter-definitions before they can even make their point. To shift the narrative, the industry desperately needs an alternate "word cloud" grounded in tangible, undeniable public goods—centering the conversation around union jobs, economic productivity, and curing cancer.
[13:16](https://www.youtube.com/watch?v=TLJNJDf2XGo&t=796s) The Labs, X-Risk Rhetoric, and the Inevitability of Heavy Regulation
The discussion shifts to the frontier labs and their handling of existential risk ("x-risk"). It is baffling to the panel why the leading AI labs have not taken a definitive, sensible position debunking or contextualizing catastrophic extinction claims. By flirting with doomsday narratives, the industry is walking directly into a regulatory buzzsaw. If the chief executive of a frontier lab publicly concedes that there might be a 10% chance of human species extinction, government officials would be considered derelict in their duty if they did not step in with aggressive, heavy-handed interventions.
The panel debates the public posture of lab leaders like Anthropic's Dario Amodei, who validates the concerns of those warning about extinction while deliberately declining to attach a specific numerical probability to it. While some participants call this evasive or say playing "binary search" on extinction probabilities is absurd, others point out that x-risk itself is an unquantifiable, made-up concept where any precise percentage is arbitrary. In practice, the only viable corporate PR posture has been reciting boilerplate talking points: acknowledging risks, highlighting enormous potential benefits, and promising to mitigate dangers as much as humanly possible. Yet critics counter that previous major computing revolutions—from the advent of personal computers and the internet to the World Wide Web and social networking—managed to identify, discuss, and mitigate genuine operational risks without ever needing to invoke the total extinction of the human species.
- **The Opponent's Lexicon**: The entire framing of the safety debate relies on terms manufactured by skeptics and doomers ("pause," "swarms," "rogue"), forcing AI advocates to waste time deconstructing hostile framing rather than promoting real-world utility.
- **The Political Dilemma**: A government cannot logically or politically ignore a technology whose own creators validate claims of a possible 10% chance of human extinction; doing so would be seen as pure governmental negligence.
- **Historical Comparison**: Prior technological epochs (computing, the web, the internet, and social networks) established safety and security practices around concrete real-world harms without ever escalating the public debate into apocalyptic species-extinction rhetoric.
[15:14](https://www.youtube.com/watch?v=TLJNJDf2XGo&t=914s) The Non-Zero Trap: Extinction Claims and the Logic of Nationalization
The group grapples with the fundamental logic of AI safety claims, questioning what happens if frontier lab leaders genuinely believe the marginal risk of species extinction is higher than that of prior baseline technologies like the internet. As Martin Casado points out, if a lab leader sincerely believes AI presents an existential catastrophe, there are logically only two coherent conclusions: either shut the entire enterprise down immediately, or nationalize it completely under state control. The conversation draws an analogy to the Cold War era, where the Pentagon ran endless simulations on the probability of nuclear conflict—famously dramatized in the movie *WarGames*—accepting that the risk of nuclear annihilation was non-zero.
The trouble arises the second someone concedes that the existential risk is "non-zero." Once a founder or researcher admits to any non-zero probability of human extinction, it opens the floodgates to wild interpretations—ranging from Dario Amodei’s more cautious estimates to doom-laden prognosticators who estimate p(doom) as high as 93%. Aaron Levie insists that any clean debate must focus strictly on the *marginal* risk AI introduces over existing societal baselines rather than vague absolute risks. Yet Martin argues that frontier labs are attempting an impossible balancing act: they invoke non-zero existential risk as a dramatic license to justify special regulatory carve-outs and privileges, while fighting tooth and nail to avoid the heavy burden of being absorbed into the national defense apparatus.
```mermaid
flowchart TD
accTitle: The Regulatory Dilemma of AI X-Risk Claims
accDescr: Diagram illustrating the two paths resulting from claiming non-zero species extinction risk versus utility status.
subgraph Claim["Risk Assessment Framing"]
A["Admitting Non-Zero Marginal X-Risk"]
end
subgraph Outcomes["Logical Conclusions"]
B["Shut Down Development"]
C["Full Nationalization<br/>(e.g., BSL-4 / Manhattan Project)"]
end
subgraph LabDesire["Lab Incumbent Strategy"]
D["Heavily Regulated Utility Status<br/>(Like JPMorgan or Telecoms)"]
E["Suppresses Open Source<br/>Protects Frontier Incumbents"]
end
A --> B
A --> C
A -. "Attempted Compromise".-> D
D --> E
```
[16:36](https://www.youtube.com/watch?v=TLJNJDf2XGo&t=996s) BSL-4 Labs, Banking Utilities, and the Incumbent Moat
When considering historical precedents for technologies carrying catastrophic risk, the group distinguishes between ordinary human safety and extreme existential threats. High-containment Biosafety Level 4 (BSL-4) facilities handling lethal pathogens like anthrax are either directly nationalized or operated under strict sovereign control. By contrast, normal consumer safety starts with industry oversight and slowly evolves into federal regulation over decades. Looking at the post–World War II era, industries deemed vital to core national infrastructure—such as power, telecommunications, healthcare, and banking—have experienced a relentless regulatory creep. Through extensive compliance mechanisms like Know Your Customer (KYC) mandates and sweeping post-financial-crisis oversight, modern commercial banks have effectively become quasi-nationalized instruments of the state.
The speakers observe that looking like JPMorgan Chase or Verizon might actually be the quiet ambition of the major AI labs. By positioning themselves as essential, critical infrastructure that requires dense regulatory compliance, incumbents can erect immense barriers to entry. While this model effectively smothers frontier open-source software and stifles broader technological innovation, it offers established labs a defensible, highly regulated market structure. Even so, Martin emphasizes that labs should not invoke "species extinction" as rhetorical leverage to secure a comfortable utility status. He notes with cautious optimism that frontier labs are finally steering in a healthier direction, with leadership rhetoric and official policy statements beginning to mature beyond apocalyptic hyperbole toward grounded technical realities.
- **Biosafety Precedents**: BSL-4 pathogen research facilities operate under strict government control and direct state oversight due to catastrophic biological risks.
- **The Post-WWII Regulatory Arc**: Vital infrastructure sectors (utilities, finance, energy) progressively shift into quasi-nationalized utilities governed by dense federal mandates like KYC.
- **The Open Source Moat**: High compliance burdens disproportionately punish open-source frontier competitors while entrenching well-capitalized proprietary labs.
[18:25](https://www.youtube.com/watch?v=TLJNJDf2XGo&t=1105s) Tech’s Century-Long Blind Spot in Politics and Regulation
While the frontier AI labs face internal friction balancing rapid commercial growth with existential risk narratives, the panel expresses optimism that the companies are genuinely trying to do the right thing. The real issue is speed: the labs expanded so fast that they are still struggling to grasp how Washington and the broader political machinery operate. Whether driven by naivety or hubris, tech founders routinely fail to realize that politics operates under completely different incentives and rules than engineering.
Steven Sinofsky points out that this is not unique to AI; the technology sector has spent over a century repeatedly failing to understand the regulatory climate at every major paradigm shift. Even titans like AT&T and IBM—companies whose very existence was intertwined with government-sanctioned monopoly status and who employed hundreds of regulatory lawyers as far back as the 1960s—never cracked the code and ultimately faced massive, restructuring antitrust lawsuits. Sinofsky recalls Microsoft experiencing the exact same shock in the 1990s: Bill Gates famously played golf with President Bill Clinton, only for Clinton's administration to slap Microsoft with a landmark antitrust lawsuit. Gates was genuinely baffled, pointing to photos from the golf course and wondering aloud, "Isn't that what I was supposed to do—go and play golf?"
- **Historical Precedents**: AT&T, IBM, and Microsoft all believed their government relationships or scale shielded them, yet each was blindsided by federal antitrust actions that fundamentally altered their business structures.
- **The Political Gulf**: Frontier tech leaders consistently mistake personal access or executive goodwill for enduring regulatory and political immunity.
---
[19:42](https://www.youtube.com/watch?v=TLJNJDf2XGo&t=1182s) Self-Regulation vs. Statutory Control: From the MPAA to FINRA
To illustrate how industries have successfully shielded themselves from heavy-handed government intervention, Sinofsky highlights Hollywood during the Red Scare and eras of moral panic over censorship, sexual content, and adult themes. Recognizing that studio heads could never win a direct constitutional fight against government pressure, the studios preemptively organized the Motion Picture Association of America (MPAA) and instituted the voluntary film rating system to police themselves. While modern observers often dismiss movie ratings as trivial, at the time, moral and political panics were acute—resulting in roughly 30% of Hollywood creative talent being blacklisted or fired for alleged communist sympathies. Because voluntary rating systems could not be constitutionally mandated across all distribution, open broadcast environments paved the way for unfiltered cable networks like HBO.
```mermaid
flowchart TD
accTitle: Approaches to Industry Governance
accDescr: Comparing Hollywood's self-policing model under the MPAA to the financial sector's statutorily backed FINRA model.
subgraph Voluntary ["Voluntary Self-Regulation (MPAA Model)"]
A["External Pressure / Public Panics"] --> B["Voluntary Industry Consortium"]
B --> C["Self-Policing & Content Ratings"]
C --> D["Avoidance of Direct Federal Statute"]
end
subgraph Statutory ["Delegated Enforcement (FINRA Model)"]
E["Perceived Systemic Market Risk"] --> F["Congressional Enabling Legislation"]
F --> G["Formal SRO with Direct Federal Oversight"]
G --> H["Nationalization of Industry Risk"]
end
```
The conversation shifts to whether AI could adopt a self-regulatory organization (SRO) model akin to FINRA (the Financial Industry Regulatory Authority). The panel sharply rejects the analogy: FINRA is far from a purely voluntary body like the MPAA. Instead, FINRA is baked into federal securities legislation, operates under direct regulatory oversight from the SEC, and essentially functions as a mechanism for nationalizing risk across financial institutions. Unlike voluntary standards, a FINRA-style model for AI would bring formal statutory oversight and state-backed liability directly into foundational software development.
- **MPAA Precedent**: Established voluntary industry guidelines that bypassed statutory mandates and enabled alternative distribution channels like premium cable to thrive without direct federal control.
- **The FINRA Pitfall**: Unlike voluntary media self-policing, an SRO like FINRA is rooted in federal legislation with binding authority, serving to nationalize risk rather than insulate tech development from government dictates.
[21:25](https://www.youtube.com/watch?v=TLJNJDf2XGo&t=1285s) The FINRA Precedent and the Inevitability of AI Oversight
The discussion opens with an examination of the Financial Industry Regulatory Authority (FINRA), exploring whether an industry-funded, government-mandated self-regulatory framework could serve as a realistic model for artificial intelligence. While banks foot the bill to operate FINRA, its authority and compliance mechanisms are strictly enforced by law. When pressed on whether the AI industry could realistically achieve an outcome better than FINRA, the consensus quickly forms that such a structure represents a practical "best case scenario."
The underlying premise is that the long-term potential and raw utility of AI make federal intervention unavoidable. When AI models eventually direct critical patient recommendations in the healthcare system, run as open-weights models inside embedded medical devices, execute algorithmic maneuvers in high-frequency trading platforms, and pilot commercial aviation, Congress simply will not stand by without asserting jurisdiction. The idea that Washington would remain hands-off while foundational models penetrate every tier of critical national infrastructure is dismissed as an impossibility.
```mermaid
flowchart TD
accTitle: Potential Model for AI Regulation Based on FINRA
accDescr: Diagram illustrating how deeply embedded AI across critical sectors leads inevitably to federal oversight modeled after FINRA.
subgraph CriticalSectors["Critical AI Integration"]
A["Healthcare & Medical Devices\n(Embedded open weights)"]
B["Financial Markets\n(High-frequency trading)"]
C["Transportation\n(Aviation & autopilot)"]
end
subgraph RegulatoryOutcome["Regulatory Response"]
D["Congressional Mandate"]
E["Industry-Funded SRO\n(FINRA-style oversight)"]
end
CriticalSectors --> D
D --> E
```
[22:16](https://www.youtube.com/watch?v=TLJNJDf2XGo&t=1336s) Post-Internet Regret and Pent-Up Energy in Congress
Reflecting on legislative history, the group turns to the passage of landmark tech legislation like the Telecommunications Act of 1996 and Section 230, which shielded internet service providers and online platforms from publisher liability. Steven Sinofsky points out that many senators currently serving were already in office back then. For years afterward, politicians harbored lingering insecurity, openly lamenting in private hearings: "The internet is so big, how did we not have anything to do with it?" This anxiety famously prompted Al Gore to overreach by highlighting his early legislative sponsorship of the National Research and Education Network, sparking years of late-night comedy derision when he was caricatured as claiming he "created" the internet.
In modern Washington, however, there is no contemporary champion playing the role of an innovation-first Al Gore. While figures like Senator Michael Bennet explore defense-related nuances and the defense establishment leans toward keeping tech private, the prevailing congressional climate is defined by remorse over past inaction. Lawmakers such as Senator Elizabeth Warren voice intense frustration that government allowed social media networks to grow unchecked without federal guardrails. As a result, decades of pent-up animosity toward Big Tech are now being siphoned directly into the AI regulatory debate.
- **Historical Precedents**: Lawmakers regret not asserting early jurisdiction over consumer ISPs and Web 2.0 social networks.
- **The Political Void**: AI lacks a prominent legislative patron advocating for technological acceleration without burdensome controls.
- **Transferred Backlash**: Regulatory measures currently proposed for AI are less about observed catastrophes and more about settling old political scores with Silicon Valley.
[23:33](https://www.youtube.com/watch?v=TLJNJDf2XGo&t=1413s) Real Damage vs. Anticipatory Panic: The Windows 95 Worm Era
The panel highlights a glaring disconnect between historical safety standards and the modern AI panic: regulation in prior eras only followed catastrophic, verifiable, real-world failure modes. In the equivalent early stage of the internet, network worms and unpatched exploits had already wiped out tens of billions of dollars in economic value, crippled up to 10% of global internet infrastructure, and shut down emergency services and hospital networks. Half-jokingly, the panel quips that by today's standards, policymakers should have outright outlawed the open internet in 1997.
To illustrate how fragile computing security used to be, the conversation pivots to an affectionate jab at Sinofsky’s Microsoft background. During the late 1990s, an out-of-the-box Windows 95 install running from a retail CD-ROM could be compromised by automated network worms before the setup wizard had even finished copying files. Until around 2001, plugging any raw, unpatched personal computer into a live Ethernet network resulted almost instantly in an infection—demonstrating that software resilience was built through weathering real-world carnage, rather than preemptive theoretical bans.
- **Economic Impact of Early Internet Vulnerabilities**: Billions of dollars in direct downtime and significant percentages of baseline packet-routing capacity disabled by early worms.
- **The PC Security Reality (Pre-2001)**: Bare-metal desktop installations connected directly to a network connection would routinely suffer automated remote exploitation within minutes.
- **Contrasting Approaches**: Earlier technological waves iterated in the wild while absorbing tangible shocks, whereas the AI conversation attempts to construct sweeping statutory constraints long before empirical harms manifest.
[24:26](https://www.youtube.com/watch?v=TLJNJDf2XGo&t=1466s) The Teething Pains of the Early Internet and Predictive Policy
The panel draws a direct historical comparison between the current existential alarm over artificial intelligence and the chaotic early days of personal computing and the internet. In the 1990s, connecting a computer to a network almost inevitably invited malware; viruses, worms, and catastrophic system crashes were constant, inflicting ongoing economic damage and prompting multiple rounds of congressional hearings, alongside widespread panic that Y2K would bring society to a halt. The group points out that if the prevailing political and cultural zeitgeist surrounding AI safety had governed the technology landscape back in 1994, "we would probably not have the internet." Much like aviation and the automotive sector in their infancy, every foundational technology endures an essential period of "teeth cutting," where engineers, operators, and the public discover actual failure modes and dangers by operating the technology in the wild.
The core problem with much of the current AI safety discourse is the attempt to write predictive policy based on hypothetical "what if" scenarios rather than observed harm. Historically, sound regulatory policy emerged only after accumulating concrete, specific data points tied directly to real-world fact patterns. When safety conversations shift away from speculative doom toward well-defined engineering problems—such as novel cybersecurity vulnerabilities and threat vectors—industry leaders and researchers can readily align on technical mitigations.
```mermaid
flowchart TD
accTitle: Historical Reactive Regulation vs Preemptive AI Policy
accDescr: Contrasting how historical tech policy responded to actual failures against preemptive hypothetical regulation.
subgraph Historical["Historical Precedent: Automotive, Aviation, & Internet"]
A["Novel Technology Deployed"] --> B["Real-World Failures & Teeth-Cutting"]
B --> C["Observed Economic & Physical Harm"]
C --> D["Targeted, Fact-Based Policy"]
end
subgraph Hypothetical["Current AI Speculation"]
E["Frontier Models"] --> F["Abstract 'What If' Extinction Scenarios"]
F --> G["Preemptive Predictive Regulation"]
end
```
- In the early consumer internet era, systems went down continually and novel worms and viruses routinely spread worldwide, triggering intense congressional scrutiny.
- Previous critical industries, including automotive and commercial aviation, established robust safety regimes only after systematically analyzing actual mechanical and operational failures.
- Technical discourse that treats safety as an engineering challenge—focusing on concrete risks like system intrusion or automated exploitation—provides an actionable baseline that abstract existential theorizing lacks.
---
[25:46](https://www.youtube.com/watch?v=TLJNJDf2XGo&t=1546s) The 1986 CFAA Precedent and Model vs. Application Liability
To illustrate how legislation properly tracks technology, the discussion focuses on the enactment of the Computer Fraud and Abuse Act (CFAA) in 1986. In 1983, hacker groups gained unauthorized access to GTE Telenet, a packet-switched data network utilized by major institutions including NASA and the Lawrence Livermore National Laboratory. Because entering someone else's computer system was not yet explicitly illegal under federal criminal law, it took two and a half years of legislative work to pass a tailored statute that made unauthorized access a federal crime. The critical lesson from the CFAA is that legal frameworks must regulate harmful actions at the applied layer rather than trying to outlaw the underlying technology itself.
This historical lesson raises the pivotal question of where legal liability should reside: should liability attach to the application layer where actions take place, or to the foundational model layer? Approximately 90% of necessary legal guardrails already exist in criminal law—unauthorized access, fraud, and data theft remain illegal regardless of whether they are carried out by a human hacker or an automated script. Furthermore, overly broad statutory language can easily backfire. In early cybersecurity, strict interpretations of computer crime laws threatened white-hat researchers who uncovered bugs, ultimately requiring legislative carve-outs and formal Department of Justice guidance clarifying that federal prosecutors would not target good-faith security research or mere violations of website terms of service. Imposing liability at the model weights layer risks repeating those same regulatory overreaches while doing little to stop bad actors.
- The 1986 Computer Fraud and Abuse Act was catalyzed by high-profile 1983 breaches of GTE Telenet, which captured Washington's attention specifically because federal agencies and national laboratories like NASA and Lawrence Livermore were affected.
- It required two and a half years of congressional deliberation to translate the GTE Telenet incident into targeted statutory language defining unauthorized access.
- Strict interpretations of anti-hacking laws historically threatened benign vulnerability research until the Department of Justice issued memos declining prosecution for white-hat security testing and contractual terms-of-service infractions.
- Existing statutory frameworks at the applied layer already cover the overwhelming majority of potential misuses, making sweeping model-level liability redundant and counterproductive.
[27:32](https://www.youtube.com/watch?v=TLJNJDf2XGo&t=1652s) The Rift Between AI Labs and the Cybersecurity Community
The conversation turns to the growing chasm between traditional cybersecurity practitioners and the frontier AI labs. When federal systems were historically breached, it captured Washington’s attention and spurred real institutional discipline. Today, however, the professional security community looks at postmortems published by AI frontier labs after incidents and consistently reaches two scathing conclusions: the labs are fundamentally "sloppy," and they are being evasive or incomplete about what actually took place.
The mechanisms for responsible vulnerability and incident disclosure have existed for decades. The CVE (Common Vulnerabilities and Exposures) system and structured vulnerability reporting emerged from institutions like Carnegie Mellon University's CERT in the 1980s, establishing rigorous, standardized obligations for disclosing technical flaws. Yet the AI labs ignore these time-tested protocols. Instead of transparent technical disclosures, their public reports read like an intern drafted them—or worse, like a defensive document produced by outside legal counsel brought in to conduct an internal investigation where management selectively withholds information. The community is left asking basic questions: where are the internal Slack messages, where is the operational timeline, and where are the technical details of what actually failed? Until labs adopt standard security reporting hygiene, their grand pronouncements about safety carry little weight.
```mermaid
flowchart TD
accTitle: Divergence in Security Postmortem Standards
accDescr: Comparison between established cybersecurity norms and the AI labs' disclosure practices.
subgraph LabPractice["Current AI Lab Postmortems"]
D["Legal Investigation Framing"] --> E["Selective Memory & Redactions"]
E --> F["Intern-Grade Summaries (Missing Logs/Slacks)"]
end
Established -.->|"Ignored by"| LabPractice
```
- **Legacy Disclosure Frameworks**: Standardized reporting models developed at CMU in the late 1980s laid down rigorous expectations for disclosing breaches and CVEs, which AI labs have largely bypassed.
- **Lawyer-Driven Auditing**: Critics argue modern AI postmortems resemble selective legal briefings designed to minimize corporate exposure rather than technical postmortems meant to educate the broader engineering community.
---
[28:39](https://www.youtube.com/watch?v=TLJNJDf2XGo&t=1719s) Hollywood Scenarios, CPU Heat Exfiltration, and Bridging X-Risk with Systems Security
Martin Casado interrupts with an aside grounded in his long history in the cybersecurity trenches. The infosec community has always harbored an instinctual disdain for merely "practical" solutions. Even if an engineering team builds a rock-solid, secure system, security purists will inevitably invent extreme, *Mission: Impossible*-style edge cases—joking about whether a Russell Crowe character could break the cryptographic cipher from across the room.
This tendency reached a peak when AI researcher Noam Brown noted on a podcast that an unconstrained superintelligence might exfiltrate its model weights or data across an air gap using the fluctuating heat generated by a CPU. While that claim invited easy mockery on social media, Casado welcomed the moment as a breakthrough. In classified, highly secure computing facilities, exotic covert channels—acoustic, electromagnetic, and thermal emissions—are genuine, documented attack vectors.
For the first time, an existential risk (x-risk) thought experiment prompted hardcore systems engineers to engage constructively rather than dismissively. Instead of trading abstract philosophical barbs, systems researchers and x-risk advocates collided on Twitter to calculate actual theoretical bitrates across thermal side channels. Despite the inherent noise of social media debate, it marked a rare and productive bridge between speculative AI risk and empirical systems engineering.
[30:25](https://www.youtube.com/watch?v=TLJNJDf2XGo&t=1825s) Grounding AI Discourse in Systems and Physical Security
The group reflects on recent debates on X (formerly Twitter) regarding AI cybersecurity and frontier risks. While social media naturally comes with "a lot of name-calling," they note that the exchange felt like the first real, technically grounded discussion about actual threat vectors rather than vague hand-waving. Frontier labs should participate much more openly in the security conversation to engage the wider technical community, pointing to OpenAI's Greg Brockman speaking out and suggesting that OpenAI researcher Noam Brown should do more public podcast brainstorms.
The core frustration with much of the prevailing AI safety discourse is that it bypasses standard operational risks in favor of existential hyperbole. Commentators often dismiss unresolved foundational engineering problems—such as how the authentication layer functions—in order to leap directly into sci-fi doomsday scenarios: "You can't just wave your hand and say that should go away and then bring up, 'Oh, but you know, space aliens can invade.'" Moving the debate into concepts like entropy, systems architecture, and thermodynamics grounds the discussion in the laws of physics rather than speculative claims about systems being "super powerful."
- **The Importance of Open Brainstorming:** Encouraging researchers to transparently explore edge-case security scenarios on public forums demystifies actual vulnerabilities and invites real peer critique.
- **Physics vs. Sci-Fi:** Moving the dialogue toward entropy, physical limits, and formal computer architecture forces AI risk analysis to align with established security engineering practices.
---
[31:36](https://www.youtube.com/watch?v=TLJNJDf2XGo&t=1896s) Cold War Hardware Realities and the Craziest Covert Channels
The discussion transitions into historical hardware security war stories, illustrating how physical realities create bizarre covert channels that the general public rarely understands. Working around Pershing missile systems required testing PC graphics cards against strict Department of Defense (DoD) standards: the moment power was cut, the video memory had to discharge instantly. Even a three-second persistence delay was considered a critical vulnerability, as residual capacitive charge in video RAM could allow an attacker to reconstruct classified screen images.
```mermaid
flowchart LR
accTitle: CRT Optical Covert Channel Reconstruction
accDescr: Diagram showing how a CRT raster beam's diffuse reflection off a window can be sampled at high frequency to reconstruct the screen image.
subgraph Emission["Emanation Source"]
A["CRT Terminal"] --> B["Raster Beam on Pixel"]
B --> C["Diffuse Light on Window"]
end
subgraph Eavesdropping["Optical Attack"]
C --> D["High-Speed Optical Sensor"]
D --> E["Sync at Monitor Refresh Rate"]
E --> F["Reconstructed Screen Image"]
end
```
The speakers trade memories of early TEMPEST testing, where government inspectors physically measured the separation between office monitors to prevent electromagnetic radiation from leaking data. Similar side-channel concerns led security teams to strip audio speakers out of workstations because speakers can serve as acoustic transducers, disable BIOS spread-spectrum clocking, and blast music from speakers aimed directly at exterior windows to jam laser microphones reading glass vibrations. After Aaron Levie jokes that Martin Casado should run safety at a frontier lab because he has never seen him more excited than when talking about "heat-based communication," Casado shares what he considers the ultimate covert channel: an eavesdropper outside a dark room can reconstruct the entire display of an old CRT monitor by using a high-frequency sensor to measure diffuse light reflecting off a window, synchronizing the optical samples with the monitor's raster beam refresh rate.
- **DoD Screen Persistence Requirements:** Graphics hardware in sensitive missile environments had to demonstrate zero memory retention when powered down to prevent physical probing of residual memory states.
- **TEMPEST Mitigations:** Physical distancing of monitors, BIOS clock-frequency controls, and acoustic jamming (aiming speakers at glass windows) were routine measures against electromagnetic and acoustic side channels.
- **The Optical CRT Exploit:** Because the brightest spot in a dark room is the single moving pixel illuminated by a CRT's raster beam at any given instant, capturing the fluctuations of ambient room light against a window at the monitor's exact refresh rate (Hz) allows an attacker to faithfully reconstruct the displayed image.
[33:04](https://www.youtube.com/watch?v=TLJNJDf2XGo&t=1984s) Pixel Covert Channels and Missile Factory Keyboard Security
Continuing a discussion on covert channels, Martin Casado outlines how attackers have historically devised ingenious physical and digital exfiltration pathways. Beyond matching transmission pulses to the exact refresh rate of a CRT monitor's raster beam, researchers realized that subverting just three pixels on a screen—making them mimic routine defective or "dead" pixels—could create a stealthy, relatively high-bandwidth one-way optical communication channel readable from a distance. Casado notes that while Noam Brown’s hypothetical side-channel via heat dissipation might seem extreme, this real-world level of physical-layer sophistication has long been documented in practical security research.
Steven Sinofsky brings up an anecdote from his early career working at a missile defense facility as an uncleared intern. Employees were strictly required to lock their keyboards inside desks or safes overnight. The underlying security fear was that unvetted custodial staff walking the halls could inspect which keys had more grime, wear, or polish to reverse-engineer passwords or operational codes. Sinofsky recalls slipping up once, leaving his keyboard out on his desk overnight, and arriving the next morning to find an ominous security infraction notice instructing him to report to security to retrieve his confiscated equipment.
Aaron Levie quips that the real issue with having this conversation on a podcast is that all these exotic covert techniques will simply end up ingested into the pretraining corpus of every frontier AI model. Casado points out that in formal computer security, this matches the standard threat model: NIST publishes 500-page manuals premised on assuming the untrusted partition is an omniscient oracle that knows and can attempt everything. The foundational problem has always been whether an adversary on the outside can trick or siphon data off the trusted enclave.
- **Optical Side-Channels**: Manipulating as few as three screen pixels to mimic benign hardware defects provides a functional, covert, one-way optical exfiltration path.
- **Physical Key Wear**: The missile factory’s policy mandated locking up keyboards overnight to prevent uncleared personnel from inferring key combinations based on grease, wear, or dust patterns.
- **The Formal NIST Threat Model**: Assumes the attacker on the untrusted boundary possesses infinite knowledge and capability (an oracle), testing only whether isolation boundaries can withstand exfiltration.
---
[34:28](https://www.youtube.com/watch?v=TLJNJDf2XGo&t=2068s) AI Swarms, Internal Denial-of-Service, and the Collapse of Human Trust Assumptions
Aaron Levie pivots the conversation to the fundamental shift AI brings to enterprise defense. What people misunderstand—and cloud with confusing rhetoric—is that an AI model or agent can systematically execute thousands of exploit variations in seconds. It does not suffer from fatigue, boredom, or distraction. This sheer speed and endurance breaks enterprise security paradigms, which were architected on the premise that internal corporate networks and SaaS systems are insulated from adversarial brute-forcing.
```mermaid
flowchart TD
accTitle: Enterprise Security Paradigm Shift Under AI Agents
accDescr: Diagram illustrating how trusted human assumptions break down when replaced by automated AI swarms hitting internal services.
Internal corporate infrastructure—such as self-hosted GitHub instances, Slack workspaces, or finance and expense management software—was never engineered to handle internal denial-of-service attacks. When autonomous agent swarms begin querying internal endpoints at machine speed, routine workflow automations indistinguishable from internal DDoS attacks overwhelm backends. Consequently, enterprises now require a granular internal observability and authentication layer dedicated to monitoring internal API activity, token usage, and automated workflows.
Sinofsky and Casado observe that veteran security engineers often scoff at explaining this, calling it elementary systems architecture. Yet enterprises routinely neglected internal zero-trust hardening because traditional information security relied heavily on human friction: organizations got by on the statistical reality that 95% to 99% of human employees behave benignly and follow standard operating procedures. When a company's "employee" transitions into an autonomous software process with programmatic persistence and a corporate credit card, the legacy perimeter falls apart entirely.
- **Scale and Exhaustion**: Autonomous agents eliminate human latency and stamina constraints, enabling exhaustive brute-force testing across every internal interface.
- **Internal Denial of Service**: Common enterprise SaaS and internal tools (Slack, GitHub, expense portals) are fundamentally vulnerable to saturation when queried by agent swarms.
- **The 95–99% Rule**: Classic corporate infosec functioned primarily because humans are slow and mostly follow rules; replacing workers with software actors with spending power renders implicit trust assumptions obsolete.
[35:53](https://www.youtube.com/watch?v=TLJNJDf2XGo&t=2153s) The Agent Swarm Threat: Rethinking Access Control and Security Models
The discussion turns to the stark contrast between traditional enterprise threat models and the emerging reality of autonomous software agents. In legacy corporate environments, security architectures operated under the assumption that an insider threat or a truly malicious employee was an extreme outlier—roughly a "one in 10,000" event. Because internal human actors were largely assumed to be trustworthy, enterprise systems were designed with permissive boundaries: once an employee was inside the corporate perimeter, obtaining access to sensitive data often required nothing more than a casual tap on a colleague's shoulder.
Autonomous agent swarms completely upend that paradigm. Instead of a handful of vetted human users, enterprises face roaming swarms of digital "drones" operating at ten-thousand-fold scale and speed. Because these agents can readily conflate legitimate instructions with destructive actions—or be tricked into confusing a good task for a bad one—the existing permission architectures collapse under the pressure. Aaron Levie points out that this shift will trigger a massive upgrade cycle across corporate infrastructure: current data security and access control models are neither granular enough nor performant enough to govern legions of autonomous agents continuously probing and executing operations across corporate data stores.
```mermaid
flowchart TD
accTitle: Enterprise Security Paradigm Shift
accDescr: Comparison between legacy human-centric enterprise access and autonomous agent swarm scale.
subgraph Legacy["Traditional Enterprise Model"]
H["Human Employees (Low Volume)"] -->|"High Baseline Trust / 'Tap on Shoulder'"| S1["Permissive Internal Systems"]
Risk1["Malicious Insider: ~1 in 10,000"]
end
subgraph Agentic["Agentic Swarm Reality"]
A["Autonomous AI Agents (10,000x Volume)"] -->|"Continuous Autonomous Execution"| S2["Rigid, Fine-Grained Access Control"]
Risk2["Mistakes Bad Tasks for Good Ones / Prompt Injection"]
end
```
- **Scale mismatch:** Enterprise authorization architectures designed for thousands of deliberate human interactions cannot handle millions of programmatic, autonomous micro-decisions.
- **Granularity and latency:** Traditional Role-Based Access Control (RBAC) lacks both the real-time execution speed and the fine-grained scoping required to isolate roaming agents.
---
[36:36](https://www.youtube.com/watch?v=TLJNJDf2XGo&t=2196s) Grounded Discourse and the Renaissance of Secure-by-Design Systems
Martin Casado steps back to frame this technical debate as the model for how the broader AI discourse ought to function. Rather than spiraling into speculative existential dread, the conversation here identifies an empirical, novel risk—cybersecurity vulnerabilities under agentic workloads—and immediately shifts toward concrete architectural remedies. Casado emphasizes that the broader tech industry and engineering community are eager to engage in this kind of pragmatic problem-solving, lamenting that mainstream AI policy debates have largely missed this productive framing.
Addressing Levie's challenge regarding authorization architecture, Casado points out that the fundamental threat models are not an unsolved scientific mystery. Operating systems literature and Multi-Level Security (MLS) research solved many of these isolation and privilege problems decades ago in academic theory. Historically, MLS and rigorous capability-based security systems were abandoned in commercial software because they imposed severe usability friction on human operators. However, because AI agents interact programmatically rather than through clumsy human interfaces, they remove the traditional usability penalty. This could catalyze a massive renaissance in operating systems, programming languages, and network architectures—reviving battle-tested academic isolation models to engineer systems that are secure by design, with frontier models actively assisting humans in writing the new software stack.
Steven Sinofsky reinforces this evolutionary perspective by drawing a direct parallel to the early consumer internet, where the industry had to rethink and rewrite the entire computing stack after realizing how inherently exposed existing machines were. In the early days of broadband, an unprotected PC booted onto the open web could be compromised by automated worms within thirty seconds. By contrast, a brand-new iPhone unboxed today immediately locks down virtually all external network communication upon first activation, allowing traffic solely to pull down the newest operating system update before permitting any other software to run.
- **Multi-Level Security (MLS):** Classical operating system isolation frameworks failed commercially due to human administration overhead, not theoretical flaws; autonomous agents make strict mathematical security models viable again.
- **Rewriting the stack:** Much like the rapid commercialization of the internet forced a wholesale rebuild of networking and OS defenses in the 1990s and 2000s, agents will drive an overhaul of computing infrastructure from the compiler down to runtime isolation.
[38:43](https://www.youtube.com/watch?v=TLJNJDf2XGo&t=2323s) The Security Evolution: How Features Turn into Vulnerabilities
The panel reflects on how software history is defined by innocent, convenient features gradually turning into major attack vectors, forcing the entire industry to adapt its default security baselines. Steven Sinofsky points out that modern out-of-box device experiences now require a mandatory, blocking update before a user can do anything at all, simply because zero-day vulnerabilities are uncovered between the factory floor and retail delivery. He recalls an era when desktop software was filled with benign capabilities that were celebrated until they were weaponized: Microsoft Word macros were originally designed to automate helpful tasks like citations until they transformed into macro viruses. Similarly, Windows CD AutoRun was created as a seamless way to launch media automatically, until malicious actors began cloning discs and swapping out executables with spyware that harvested user data.
The industry responded by aggressively deprecating those conveniences and building structured guardrails. Just five years ago, two-factor authentication (2FA) was far from universal, and around 2015, any enterprise SaaS startup quickly discovered that closing deals required integrating directly with Okta or Google Auth rather than maintaining proprietary user directories. Today, managed identity and centralized authentication are non-negotiable prerequisites before an application is even considered enterprise software.
- **Forced Out-of-Box Updates**: Devices today must install patches before completing initial setup to mitigate zero-day vulnerabilities discovered post-manufacturing.
- **Vulnerability of Convenience**: Features like Word macros and CD AutoRun were designed for developer and user ease, but ultimately had to be disabled or strictly quarantined after being exploited for virus distribution.
- **Enterprise SaaS Identity Baselines**: Enterprise readiness shifted from bespoke credential management to mandatory federated authentication (e.g., Okta, Google Auth) and widespread 2FA adoption.
---
[40:19](https://www.youtube.com/watch?v=TLJNJDf2XGo&t=2419s) Agent Permissions and the Fear of "GDPR for AI"
Aaron Levie observes that the existing computing stack is poorly equipped to handle autonomous AI agents, presenting an unworkable binary: an agent either prompts the user for confirmation on every single minor action, or it operates with unrestricted root-level access that could accidentally wipe a drive. Operating systems were simply not engineered for the nuanced, granular permissions that autonomous workflows demand. Drawing on Box’s experience with data governance, Levie explains that users rarely want an agent to possess carte blanche access across an entire file system; they require granular controls—such as read/write access in a designated working folder paired with strict read-only permissions in another. The central challenge lies in making that multi-tiered permissioning seamless and intuitive for end users.
```mermaid
flowchart TD
accTitle: The Spectrum of Agent Permission Models
accDescr: Diagram illustrating the current crude options for AI agent permissions versus the ideal granular model.
subgraph Flawed["Current Extremes"]
A["Hyper-Prompting<br/>(Asks user permission for every step)"]
B["Total Access<br/>(Agent can delete entire computer)"]
end
subgraph Goal["Granular AI Operating Model"]
C["Context-Aware File Scoping"]
D["Role-Based Permissions<br/>(Folder A: Read/Write | Folder B: Read-Only)"]
E["Intuitive User Configuration"]
end
Flawed -.->|Needs Evolution To| Goal
```
Sinofsky considers Levie’s point foundational, noting that user-facing granular permissions represent a challenge that computer science has failed to solve intuitively over the last forty years. However, he muses that AI itself might finally bridge that gap by managing contextual access dynamically. When asked about his foremost regulatory worry, Sinofsky reveals the note he wrote down before walking into the room: his biggest fear is European regulators viewing GDPR as an unmitigated success and attempting to "GDPR AI." Rather than solving security at the architectural layer, he worries policy will impose intrusive, boilerplate consent banners onto model outputs—prompting users with the equivalent of a cookie disclaimer whenever a vendor emits generated text.
- **The Granularity Dilemma**: AI agents currently lack a middle ground between "alert fatigue" (confirming every read/write) and catastrophic over-privilege (full system access).
- **File System Scoping**: Enterprise agent adoption depends on folder-level, role-based boundary management rather than indiscriminate file system exposure.
- **Regulatory Overreach**: The risk of Brussels-style privacy rules creating a compliance theater of pop-up disclosures and vendor consent prompts around every piece of generated text.
[41:43](https://www.youtube.com/watch?v=TLJNJDf2XGo&t=2503s) The GDPR Prompt Trap: How Regulators Shift Liability onto Agent Actions
The discussion shifts to the looming threat of European-style regulatory overreach as autonomous AI agents move from passively generating text to actively taking real-world actions. While North American jurisdictions are unlikely to police raw model speech directly, Europe will inevitably deploy text filters, keyword blocklists, and content restrictions. However, the far more insidious threat emerges whenever an AI agent—whether a background orchestrator or a frontline assistant—touches a "verb" by interacting with a third-party software system. The panel expresses deep concern that regulators will default to their familiar bureaucratic reflex: mandating a GDPR-style confirmation banner for every state-changing action. Under this regime, every external database write, transaction, or non-lookup operation would be forced back to the user with a confirmation modal, transforming standard agentic execution into a relentless stream of warning dialogs akin to an automobile airbag advisory.
```mermaid
flowchart TD
accTitle: The Agent Liability Shift
accDescr: Flowchart illustrating how regulatory confirmation prompts shift legal liability from AI vendors to end users.
subgraph AgentAction ["Autonomous Execution"]
A["Agent Initiates Action"] --> B{"Is Action a 'Verb'?\n(Write / External Call)"}
end
subgraph RegulatoryIntervention ["EU Regulatory Reflex"]
B -- "Yes" --> C["Mandatory Confirmation Prompt\n('GDPR for Agents')"]
C --> D["User Clicks 'Allow' Due to Fatigue"]
end
subgraph LegalOutcome ["Liability Assignment"]
D --> E["Legal Liability Transferred to User"]
E --> F["System Friction & Numbness"]
end
```
The panel observes that bureaucratic bodies gravitate toward prompts not because they actually protect users, but because they serve as convenient legal mechanisms for "liability assignment." By forcing an end user to manually click an approval button, regulators and vendors legally absolve themselves of whatever downstream fallout occurs. Because the United States effectively ceased leading on proactive technology antitrust and regulatory frameworks roughly fifteen years ago, Europe has stepped into the vacuum to set global norms. With virtually no domestic frontier AI champions of its own to protect, Europe "has nothing to lose" by imposing draconian, friction-heavy compliance hurdles that risk smothering agentic utility before it can mature.
- Regulators view click-through confirmation dialogs as established legal precedent for assigning fault and liability away from platforms and back onto individual users.
- The danger lies in subjecting every programmatic "verb"—such as writing to an API, updating a CRM record, or modifying a calendar—to mandatory user-in-the-loop sign-off.
- The transatlantic dynamic leaves regulatory momentum in the hands of the European Union, which lacks native hyperscale AI frontier labs and faces minimal domestic economic downside from stifling friction.
[42:52](https://www.youtube.com/watch?v=TLJNJDf2XGo&t=2572s) From Windows UAC to Airbag Stickers: Warning Fatigue and the Illusion of Safety
Steven Sinofsky grounds the debate in tech history, comparing the regulatory urge to slap prompts on AI agents to physical safety mandates and early computing debacles. He likens it to buying a brand-new car and having to peel endless warning stickers off the visor and dashboard, or staring at permanent labels warning about infant car seats—mandates designed for edge cases that burden every single user because regulatory bodies treat the mere existence of a sticker as "success." Sinofsky dryly recalls being on the receiving end of European antitrust mandates himself when Microsoft was forced to engineer the infamous "browser choice" ballot screen in Windows.
Drawing from his tenure leading Windows and Office, Sinofsky recounts how early personal computing tackled genuine security crises with blunt-force popups that backfired into collective user numbness. When the consumer internet exploded, downloading and running executable programs in full administrator mode meant a single malicious payload could wipe or steal a user's entire file system. In response, Microsoft rolled out User Account Control (UAC), freezing the entire desktop and interrupting the workflow with high-contrast administrative prompts. Microsoft implemented similar heavy-handed security warnings inside Microsoft Word, throwing scary dialog boxes at students merely trying to open a term paper that contained a benign macro.
The panel points out the universal psychological outcome of this architecture: human beings inevitably become numb, blindly clicking "OK" or "Allow" without evaluating the threat. When regulators notice that warnings are universally ignored, their only prescription is to demand that the dialog boxes be made even larger and more obtrusive. While modern platforms like macOS avoid this by funneling users into heavily sandboxed app ecosystems where people rarely download new software, the web remains vulnerable. The panel laments that the broader public discourse is consumed by existential doom, whereas the industry desperately needs a pragmatic, macro-level architectural discussion about permissions, friction, and genuine safety models.
- Steven Sinofsky references Microsoft's implementation of User Account Control (UAC) and Word macro security warnings, which halted execution but ultimately trained users to dismiss warnings without reading them.
- Warning fatigue mirrors consumer responses to GDPR cookie banners and automotive airbag warning tags, where ubiquitous compliance disclosures provide legal coverage while destroying user experience.
- The speakers argue that the tech industry should be establishing self-regulatory frameworks—such as a dedicated "FINRA for AI"—to define operational agent security before clumsy governmental mandates permanently handicap agentic interfaces.
[44:46](https://www.youtube.com/watch?v=TLJNJDf2XGo&t=2686s) Grounding AI Risk in History: From Stanford Oscilloscopes to Abstract Extinction
Martin Casado pushes back against the tenor of contemporary AI safety debates, arguing that the focus on catastrophic existential risk distracts from tackling pragmatic, engineering-level challenges. He points out that the technology industry has repeatedly navigated eras of wide-open, unfettered access where security was virtually non-existent, yet the fallout was far more mundane than apocalyptic predictions suggested. To illustrate, Casado recalls an incident from his PhD days at Stanford: noticing that an oscilloscope in the lab was running sluggishly, he decided to inspect its network traffic and discovered an unusually massive stream of data. It turned out the device had an embedded TCP/IP stack running an old version of Windows CE, and someone on the internet had compromised it to host an illicit porn server—prompting Levie and Sinofsky to joke that Casado should get on the record clarifying he was not the host.
Casado emphasizes that in the earlier eras of computing, virtually every system you probed harbored an intrusion or vulnerability. Despite an environment where attackers possessed widespread capabilities, worst-case catastrophic outcomes rarely materialized. While Casado acknowledges that computer failures can have lethal consequences—noting that people have died when critical networks went down—he argues that the discourse should be calibrated around tangible engineering and policy frameworks, such as prompt security, system perimeters, or compliance battles like GDPR, rather than unfalsifiable philosophical dread about human extinction.
- **The Stanford Oscilloscope Incident:** While working on his doctorate at Stanford, Casado investigated a slow lab oscilloscope, found unexpectedly high network traffic, and discovered someone had hijacked its embedded Windows CE system to run a porn server.
- **Benign Reality vs. Worst-Case Scenarios:** Decades of computing history demonstrate that open, vulnerable architectures frequently suffered intrusions, yet almost never culminated in the apocalyptic scenarios currently projected onto AI models.
- **Pragmatic Security vs. Philosophical Doom:** Casado advocates for moving past abstract "species extinction and philosophy" and refocusing efforts on concrete issues like prompt injection, network resilience, and real-world regulatory quibbles like GDPR.
[46:13](https://www.youtube.com/watch?v=TLJNJDf2XGo&t=2773s) The Historical Reality of Regulation: Automobiles, Aviation, and the FDA
Steven Sinofsky dismantles the common argument that AI needs immediate, sweeping preemptive regulation just as industries like aviation, automotive, and pharmaceuticals are regulated today. He reminds the group that today’s rigorous oversight frameworks did not precede those industries; they took decades of iteration, real-world deployment, and observed failures to materialize. Early automobiles appeared around the turn of the 20th century, yet standard federal safety regulations and Ralph Nader's groundbreaking critique *Unsafe at Any Speed* did not arrive until the mid-1960s—more than six decades later. Similarly, patent medicines and unregulated pharmaceuticals were freely sold during the California Gold Rush, and modern FDA drug testing regimes only hardened roughly 50 to 75 years later in the wake of catastrophes like the thalidomide tragedy.
Aviation followed the exact same trajectory. After the Wright brothers' first flights at the start of the century, two decades elapsed before pilot licensing was even introduced in the 1920s—and even then, an aspiring pilot simply showed up with their own aircraft and passed a test no more rigorous than modern high school driver's education. Another twenty years passed before minimal airworthiness inspections emerged, and the comprehensive regulatory apparatus of the modern FAA only took shape well after World War II. While commercial aviation eventually matured into the safest mode of transportation in human history, Sinofsky emphasizes that it also became one of the slowest, most rigid environments for technological innovation: had regulators instituted FAA-style controls in 1910, modern aviation would never have gotten off the ground.
```mermaid
flowchart LR
accTitle: Historical Timelines of Industry Regulation
accDescr: Diagram illustrating the decades-long lag between initial invention and modern regulatory regimes across automobiles, pharma, and aviation.
subgraph Auto ["Automotive Sector"]
A1["Turn of the Century:<br/>First Motor Cars"] --> A2["Mid-1960s:<br/>'Unsafe at Any Speed' & Safety Mandates"]
end
subgraph Pharma ["Pharmaceutical Sector"]
P1["1850s Gold Rush:<br/>Unregulated Remedies"] --> P2["Mid-20th Century:<br/>Thalidomide Crisis & Modern FDA"]
end
subgraph Aero ["Aviation Sector"]
V1["1903:<br/>Wright Brothers Flight"] --> V2["1920s:<br/>Basic Pilot Certificates"]
V2 --> V3["1940s–1950s:<br/>Airworthiness & Modern FAA"]
end
```
- **Automotive Lag:** Automobiles operated with minimal government safety requirements for roughly sixty years until Ralph Nader's 1965 book *Unsafe at Any Speed* galvanized modern automotive safety laws.
- **Pharmaceutical Evolution:** Medicine sales were virtually unchecked in the 19th-century Gold Rush era, taking more than half a century and global crises like thalidomide to establish the modern FDA clinical trial apparatus.
- **The Aviation Precedent:** The FAA’s rigorous standards required roughly 40 to 50 years of trial, error, and crashes (as seen in early 1920s flight reels) to develop; imposing those burdens in 1910 would have smothered the industry before it could innovate.
[47:40](https://www.youtube.com/watch?v=TLJNJDf2XGo&t=2860s) Regulating Prematurely and Nick Bostrom’s Warning
The group reflects on the paradox of trying to regulate artificial intelligence before its practical architecture is even understood. Drawing on a recent podcast appearance by philosopher Nick Bostrom, the discussion highlights that early regulation does not eliminate risk; instead, it leaves developers and policymakers with the exact same hazards while locking down an architecture they do not yet comprehend. Regulating prematurely risks willing a rigid system into existence while still having no concrete idea of how to actually control it in production.
To govern or design around AI, industry practitioners first need to uncover what the technology actually is as it evolves. New paradigms and implementations emerge every few months, fundamentally re-casting how AI is conceived compared to just half a year earlier. While conventional wisdom long held that conversational text prompts and raw text generation were the natural interfaces for AI interaction, recent application-layer breakthroughs are radically overturning the assumption that AI must function like a chatbot.
```mermaid
flowchart TD
accTitle: Evolution of LLM Integration Architecture
accDescr: Diagram contrasting traditional generative text LLMs jammed into software versus specialized decision engine models.
subgraph ChatbotApproach ["Traditional Generative LLM in Software"]
A["Input Text / Prompt"] --> B["LLM Text Generation"]
B --> C["Janky Schema Extraction<br/>(Often ignored or brittle)"]
C --> D["Traditional Software API"]
end
subgraph DecisionEngine ["Decision Engine Paradigm"]
E["Input Text + Discrete Options"] --> F["Classification / Choice Engine"]
F --> G["Direct Structured Decision<br/>(Fast, cheap, accurate)"]
G --> H["Deterministic Code Execution"]
end
```
[48:38](https://www.youtube.com/watch?v=TLJNJDf2XGo&t=2918s) LLMs as Decision Engines: Moving Beyond Text-In, Text-Out
The conversation shifts to the structural shift in how frontier models interface with software systems, spotlighting a breakthrough approach that moves away from pure text generation toward decision engines. Large language models originated as human-facing chat tools characterized by "text in, text out." For years, developers have attempted to shoehorn these conversational text generators into traditional software stacks. Traditional code, however, does not operate on ambiguous prose; developers were forced into "janky" workarounds, writing system prompts that pleaded with the model ("here's the schema") only for the generative engine to wander or disregard the requested structure entirely.
The alternative model design inverts this premise: generating raw text token-by-token is computationally expensive, slow, and overly complicated for programmatic tasks. Instead of paying the massive compute overhead to generate prose, the model leverages its vast semantic understanding simply to evaluate input text and select the single best option from a predefined set of choices. By training and optimizing purely for discrete selection rather than autoregressive token generation, systems run orders of magnitude faster, dramatically cheaper, and with far higher accuracy. For enterprise software use cases that have no need for conversational chit-chat, this specialized decision engine represents the most natural fit yet, prompting what the participants describe as perhaps the fastest adoption wave of an AI model since ChatGPT.
Steven Sinofsky doubles down on the importance of this shift, emphasizing that decades of human-computer interaction research contradict the hype around conversational UI:
- **No historical validation for natural language UI**: There has never been an empirical user study showing that interacting with a computer via full natural language is efficient; in reality, open-ended conversational text is almost always the least efficient interaction model.
- **The problem of human question-asking**: In practice, formulating precise queries is difficult. In an average workplace meeting, fewer than half of the participants are capable of asking a clear, well-structured question.
- **Generative latency and user frustration**: Watching an LLM slowly stream an entire paragraph of text when the software or user only needs a concrete decision or a specific data point creates immediate friction, forcing the user to endure and pay for unnecessary compute.
[50:30](https://www.youtube.com/watch?v=TLJNJDf2XGo&t=3030s) Probabilistic Programming: Resurrecting 50 Years of Simulation CS
The group examines how language models are evolving away from verbose, chatty conversationalists and toward high-speed, probabilistic decision engines. Under the conventional chatbot paradigm, users waste time and compute paying to watch an LLM generate seven paragraphs of streaming text, only for the model to end with an apology that it could only partially address the problem. The real breakthrough comes when the output is explicitly designed for probabilistic programming. Instead of relying on brittle, binary deterministic checks—such as determining whether an incoming ticket is strictly a customer service inquiry and routing accordingly—a probabilistic engine can evaluate an input and output an exact confidence score, such as determining an inquiry is "80% customer service."
This approach directly reconnects modern AI with foundational computer science history. In the 1960s and 1970s, computing was almost entirely centered around applied mathematics and simulation. Systems were designed to model non-deterministic real-world variables—such as launching a missile toward a target through variable, fluctuating winds, modeling that turbulence dynamically, and firing thrusters along an arc. Prior to the rise of commercial business applications in the 1970s, programming language design was preoccupied with embedding probability directly into control flow primitives, such as probabilistic `if` statements and non-deterministic `while` loops designed to run until an event occurred "maybe most of the time." The hosts joke that business computing and accounting derailed this momentum, noting dryly that "there's no probability in accounting" and that "then we ruined everything."
Steven Sinofsky reflects on his own academic roots, recalling that the second assignment in his very first computer science class was building a simulation of customers waiting in line at a store. Decades later, he realized his professor had authored a foundational textbook titled *The Theory of Simulation* around 1960—a discipline that largely faded from the mainstream software canon by the 1980s as deterministic architectures took over. Reviewing current architectural decks on language models, Sinofsky emphasizes that treating LLMs as probabilistic routing engines is not an unprecedented invention, but the revitalization of half a century of dormant theory. Domain-specific frameworks like Jev operationalize this by treating the prompt as a programmatic call that returns a calibrated percentage to govern downstream conditional logic.
```mermaid
flowchart TD
accTitle: Probabilistic Decision Engine Flow
accDescr: Contrast between conversational LLM generation and probabilistic decision routing in software pipelines.
subgraph Conversational["Chatbot Paradigm"]
A["Natural Language Prompt"] --> B["Streaming Paragraphs"]
B --> C["Conversational Apologies"]
end
subgraph Probabilistic["Probabilistic Programming Engine"]
D["System Event / Query"] --> E["Prompt Evaluation Layer"]
E --> F["Probability Score\ne.g., 80% Service"]
F --> G{"Probabilistic 'If' Branch"}
G -->|"Threshold Met"| H["Customer Service Workflow"]
G -->|"Fallback"| I["General Help Desk Workflow"]
end
```
- Early computer science in the 1960s and 1970s was dominated by physical simulations, modeling non-constant variables like wind resistance and trajectory corrections for rocketry.
- Deterministic commercial computing and enterprise accounting sidelined probabilistic programming languages during the 1980s.
- Modern frameworks allow developers to treat prompts as function calls that return probability distributions, slotting language models straight into standard conditional control flow.
---
[53:01](https://www.youtube.com/watch?v=TLJNJDf2XGo&t=3181s) Why the Frontier Labs Haven't Built This: Beings vs. Tools
The discussion shifts to why foundation model labs have largely overlooked the probabilistic decision engine paradigm, leaving it to outside engineers to build. The panel argues that this divergence stems from a fundamental philosophical divide in how frontier labs view their mission. The leading AI labs are not attempting to build mundane developer utilities or deterministic software glue; their cultural ambition is to build autonomous digital entities. As the speakers observe, "they're trying to create beings and beings speak. If you're trying to create god, god speaks in natural languages." Because the frontier labs view their models as synthetic beings, their roadmaps naturally prioritize human-like conversational fluency and natural language dialogue over compact programmatic outputs.
In contrast, practicing software engineers and enterprise application builders do not need an artificial being or a chat partner—they need composable, deterministic software components. For months, developers have faced friction trying to force unpredictable conversational models into rigid production software stacks. The explosive enthusiasm around probabilistic decision tooling reflects the industry's realization that LLMs can act as high-speed deterministic decision primitives rather than anthropomorphic chatbots, finally offering a practical bridge between stochastic neural networks and enterprise software architectures.
[53:36](https://www.youtube.com/watch?v=TLJNJDf2XGo&t=3216s) Shifting the Center of Innovation Outside the Frontier Labs
The conversation turns to the practical realities of embedding language models into software architectures. While developers historically struggled to make an LLM reliably evaluate or drive a standard programmatic `if` statement, emerging architectural patterns outside the model are dramatically lowering that friction. This shift fundamentally alters the nature of software itself, moving it away from purely deterministic logic toward an inherently stochastic, probabilistic paradigm. Rather than treating the model merely as an anthropomorphic chatbot, software engineers are beginning to harness it as an underlying computational engine. Crucially, the group observes that the epicenter of innovation has decisively migrated away from the core foundation model labs and out into the software layer built around them.
Drawing on decades of platform history, Steven Sinofsky points out that foundation model providers are experiencing the exact lifecycle trap that every major platform provider eventually encounters. Once a platform hits critical mass, innovation at the platform layer inevitably slows down. Platform engineers become utterly consumed by operational overhead, uptime, backwards compatibility, and scaling pressures, regardless of how many people they hire. Sinofsky notes the classic Apple ecosystem phenomenon known as "Sherlocking"—where a platform holder observes external ecosystem tooling, absorbs those capabilities into the core OS as native features, and developers protest. That dynamic is a natural symptom of a healthy computing wave: it proves that the real, differentiated breakthroughs are now happening outside the core model itself.
```mermaid
flowchart TD
accTitle: The Platform Innovation Shift
accDescr: Diagram showing how model labs become bogged down with platform maintenance while external software drives new capabilities.
subgraph FrontierLabs["Frontier Model Labs / Platform Layer"]
A["Core Foundation Models"] --> B["Platform Maintenance & Scale"]
B --> C["Compatibility Overhead & Sherlocking"]
end
subgraph SoftwareEcosystem["Outside the Model / Application Layer"]
D["Stochastic Logic & Decision Flow"] --> E["Agentic Tooling & System Integration"]
E --> F["New Breakthrough Software Paradigms"]
end
FrontierLabs -.->|"Provides Foundation to"| SoftwareEcosystem
```
- **Stochastic Control Flow**: Developers are moving past rigid chatbot wrappers toward letting probabilistic models drive core software logic, such as branching conditional statements.
- **The Platform Bottleneck**: As platform providers scale, their engineering capacity gets swallowed by compatibility, reliability, and infrastructure maintenance, opening a massive window for external developers.
- **The "Sherlocking" Dynamic**: Just as operating systems historically turned third-party utilities into native OS features, frontier labs will eventually absorb outside patterns—yet external builders will continue to outpace them at the application frontier.
## Notable quotes
> [00:00](https://www.youtube.com/watch?v=TLJNJDf2XGo&t=0s) You will the thing into being, but you haven't figured out how to control it.
> [00:00](https://www.youtube.com/watch?v=TLJNJDf2XGo&t=0s) roaming drones but like times 10,000
> [00:52](https://www.youtube.com/watch?v=TLJNJDf2XGo&t=52s) taking credit for them, as VCs do. — Martin Casado
> [03:50](https://www.youtube.com/watch?v=TLJNJDf2XGo&t=230s) milquetoast capitulation that's totally not adequate for the task at hand.
> [06:06](https://www.youtube.com/watch?v=TLJNJDf2XGo&t=366s) we have to be at the forefront of AI, but I'm also terrified of it, which is why I must build it — Martin Casado
> [07:48](https://www.youtube.com/watch?v=TLJNJDf2XGo&t=468s) dead run — Steven Sinofsky
> [18:25](https://www.youtube.com/watch?v=TLJNJDf2XGo&t=1105s) Isn't that what I was supposed to do—go and play golf? — Bill Gates
> [22:16](https://www.youtube.com/watch?v=TLJNJDf2XGo&t=1336s) The internet is so big, how did we not have anything to do with it? — Steven Sinofsky
> [24:26](https://www.youtube.com/watch?v=TLJNJDf2XGo&t=1466s) we would probably not have the internet.
> [30:25](https://www.youtube.com/watch?v=TLJNJDf2XGo&t=1825s) You can't just wave your hand and say that should go away and then bring up, 'Oh, but you know, space aliens can invade.'
> [50:30](https://www.youtube.com/watch?v=TLJNJDf2XGo&t=3030s) there's no probability in accounting
> [53:01](https://www.youtube.com/watch?v=TLJNJDf2XGo&t=3181s) they're trying to create beings and beings speak. If you're trying to create god, god speaks in natural languages.
> [53:01](https://www.youtube.com/watch?v=TLJNJDf2XGo&t=3181s) Sherlocking — Steven Sinofsky
## Mentioned
Aaron Levie, Martin Casado, Steven Sinofsky, Erik Torenberg, Dario Amodei, David Sacks, Bill Clinton, Bill Gates, Noam Brown, Greg Brockman, Al Gore, Michael Bennet, Elizabeth Warren, Russell Crowe, Ralph Nader, Nick Bostrom, Wright brothers, Anthropic, Box, a16z, Microsoft, AT&T, IBM, JPMorgan Chase, Verizon, Apple, Google, Okta, GitHub, Slack, Expensify, HBO, Lawrence Livermore National Laboratory, NASA, Carnegie Mellon University (CMU) CERT, Department of Justice (DOJ), Department of Defense (DoD), SEC, FINRA, MPAA, FAA, FDA, NIST, Stanford University, Pershing missiles, WarGames, Telecommunications Act of 1996, Section 230, Computer Fraud and Abuse Act (CFAA), GTE Telenet, TEMPEST, Windows 95, Windows CE, Windows UAC, GDPR, ChatGPT, Jev, Unsafe at Any Speed, X (formerly Twitter)
## Video description
Erik Torenberg sits down with Box CEO Aaron Levie, and a16z’s Martin Casado, and Steven Sinofsky to debate how the AI industry should think about safety, security, and regulation as increasingly capable agents move into the real world.
They argue that much of today’s conversation is happening before we have clearly defined the risks we’re trying to regulate. Drawing on earlier waves of computing, from computer viruses and the early internet to aviation and automobiles, they ask what AI can learn from industries that developed safety standards only after understanding how their technologies actually failed.
The conversation then gets concrete: agents don’t get tired, can operate at enormous scale, and can probe systems in ways human employees never could. That could require rethinking permissions, authentication, operating systems, and the security stack itself. They also discuss why AI innovation may increasingly move beyond the frontier labs and into the software built around the models.
Timestamps:
00:00 - Intro
00:50 - Pacing the Frontier: Reacting to the AI Safety Discourse
03:50 - Species Extinction Talk: Do Labs Actually Believe Their Own Rhetoric?
06:49 - The Nationalization Question & Whose Job Safety Really Is
11:32 - David Sacks vs Government: "You're Asking Us to Regulate What?"
12:08 - 2028 as the AI Election
28:39 - Why Cybersecurity Has Historically Rejected Practical Solutions
32:30 - The Craziest Covert Channel Ever Seen
48:38 - LLMs as Decision Engines vs Chatbots
53:01 - Why the Labs Haven't Built This: The Being vs Tool Mindset
Resources:
Follow Aaron Levie on X: https://x.com/levie
Follow Martin Casado on X: https://x.com/martin_casado
Follow Steven Sinofsky on X: https://x.com/stevesi
Stay Updated:
If you enjoyed this episode, be sure to like, subscribe, and share with your friends!
Find a16z on X: https://twitter.com/a16z
Find a16z on LinkedIn: https://www.linkedin.com/company/a16z
Listen to the a16z Show on Spotify: https://open.spotify.com/show/5bC65RDvs3oxnLyqqvkUYX
Listen to the a16z Show on Apple Podcasts: https://podcasts.apple.com/us/podcast/a16z-podcast/id842818711
Follow our host: https://x.com/eriktorenberg
Please note that the content here is for informational purposes only; should NOT be taken as legal, business, tax, or investment advice or be used to evaluate any investment or security; and is not directed at any investors or potential investors in any a16z fund. a16z and its affiliates may maintain investments in the companies discussed. For more details please see http://a16z.com/disclosures.
## Background
本期内容由 a16z 于 2026 年 9 月 29 日在 YouTube 平台发布。节目由 Erik Torenberg 主持，邀请了 Box 首席执行官 Aaron Levie、a16z 合伙人 Martin Casado 以及前微软高管 Steven Sinofsky 共同做客，围绕前沿 AI 实验室的安全话语体系、政府监管的潜在误区、系统级安全架构重构，以及创新重心从基础模型向外部软件层迁移等核心议题展开深入探讨。
## TL;DR
对谈嘉宾尖锐地批判了前沿 AI 实验室将安全讨论过度上升为“人类物种灭绝”的公关话术，指出这种脱离实际事实的超前监管呼吁容易引发危险的政策反噬与行业阻碍。通过回顾计算机蠕虫、早期互联网、汽车与民航等历史产业的安全演进规律，他们论证了真正的安全标准只能建立在系统暴露具体故障之后，而非依赖前瞻性的末日幻想。随着不知疲倦的自主 Agent 和集群调用冲击传统的内部权限与认证模型，企业必须通过复兴多级安全（MLS）等理论重构底层操作系统与安全栈。此外，嘉宾们指出全自然语言的聊天界面并非最高效的软件形态，AI 创新的核心正在从基础大模型本体转移到将概率与决策直接注入传统业务代码的外部软件生态。
## Atmosphere & style
整场对话气氛热烈、交锋密集且兼具极高的智识密度。几位嘉宾凭借深厚的行业履历与系统工程积淀，以极其务实、辛辣甚至略带嘲讽的口吻撕开前沿实验室的末日公关假象；从核导弹工厂的往昔回忆到 Windows 系统开箱即中毒的历史轶事，现场充斥着老派系统工程师面对空洞哲学叙事时的清醒与幽默。听众既能感受到他们对政治监管天真幻想的冷峻警示，又能体会到当技术讨论回归物理现实、系统熵增与概率编程时，那种重新发现技术本质的澎湃兴奋感。
## Bulletin
- [03:50](https://www.youtube.com/watch?v=TLJNJDf2XGo&t=230s) 前沿实验室宣扬的“10% 灭绝风险”论调在逻辑上存在自相矛盾：若真面临物种灭绝级危机，唯一合理的应对路径是军工级国家化管制，而非以放缓发布节奏（Pacing）为由向政府乞求特许经营牌照。
- [34:01](https://www.youtube.com/watch?v=TLJNJDf2XGo&t=2041s) 自主智能体集群（Agent Swarms）打破了传统企业安全边界，不知疲倦的高频内部 API 调用在形态上等同于企业内网 DoS 攻击，倒逼多级安全（MLS）与默认安全架构的复兴。
- [48:38](https://www.youtube.com/watch?v=TLJNJDf2XGo&t=2918s) 大语言模型的工程重心正从生成冗长自然语言转向后端“决策引擎”，通过让模型在预设候选集中进行概率选择，大幅压低计算成本并提高传统业务软件的确定性。
- [53:36](https://www.youtube.com/watch?v=TLJNJDf2XGo&t=3216s) AI 行业的创新重心已发生结构性位移：基础模型层正逐步陷入维护与兼容性瓶颈，未来的核心技术突破将全面爆发于围绕模型构建的外部软件层与概率编程控制流中。
## Key takeaways
- [04:30](https://www.youtube.com/watch?v=TLJNJDf2XGo&t=270s) 前沿实验室的“控制节奏”（Pacing）缺乏基准时间表，本质上是用极具破坏性的舆论公关迎合少数末日论研究员的 HR 留人手段。
- [06:06](https://www.youtube.com/watch?v=TLJNJDf2XGo&t=366s) 如果前沿模型确实具备毁灭性存在风险，符合逻辑的应对只能是彻底关停或全面国家化，而非走钢丝般谋求特许经营壁垒。
- [10:19](https://www.youtube.com/watch?v=TLJNJDf2XGo&t=619s) 向政府求取定制化放缓监管是科技行业的天真幻觉，政治妥协机制必然导致矫枉过正，使 2028 年大选成为彻底撕裂的 AI 全民公投。
- [12:21](https://www.youtube.com/watch?v=TLJNJDf2XGo&t=741s) 支持 AI 阵营在舆论战中严重失语，全盘受制于反对派构建的“暂停”、“蜂群”、“失控”等恐慌词汇，亟需引入“工会工作”等社会价值叙事抢夺话语权。
- [19:40](https://www.youtube.com/watch?v=TLJNJDf2XGo&t=1180s) 好莱坞应对审查的 MPAA 民间自律评级模式与金融业 FINRA 国家化强制立法存在本质区别，后者会将模型风险彻底转变为国有化准入门槛。
- [24:26](https://www.youtube.com/watch?v=TLJNJDf2XGo&t=1466s) 历史上的科技立法（如 1986 年 CFAA）均建立在确凿危害事实之上，当前基于虚构前瞻性灾难推行的 AI 监管彻底脱离了实证规律。
- [30:25](https://www.youtube.com/watch?v=TLJNJDf2XGo&t=1825s) AI 安全必须从“外星人入侵”式的形而上空想，拉回到物理定律、信息熵、电磁辐射与硬件隐蔽信道（Covert Channels）的现实攻防中。
- [35:53](https://www.youtube.com/watch?v=TLJNJDf2XGo&t=2153s) 企业传统的低信任成本安全假设已失效，智能体由于不知疲倦与高并发特性，迫使企业必须重构内部 API 遥测与精细化访问控制层。
- [37:09](https://www.youtube.com/watch?v=TLJNJDf2XGo&t=2229s) 多级安全（MLS）曾因人类可用性极差而被边缘化，但随着 AI 成为系统直接调用者，系统级软件与操作系统正迎来默认安全架构的复兴浪潮。
- [41:43](https://www.youtube.com/watch?v=TLJNJDf2XGo&t=2503s) 沿袭欧洲 GDPR 逻辑对 AI Agent 强加确认弹窗不仅无法解决责任归属，反而会重蹈 Windows UAC 与宏警告所引发的“警告麻木症”覆辙。
- [46:13](https://www.youtube.com/watch?v=TLJNJDf2XGo&t=2773s) 过早对处于雏形期的通用技术实施严苛监管极具扼杀性，正如若在 1910 年就成立 FAA，现代航空工业根本无法诞生。
- [49:51](https://www.youtube.com/watch?v=TLJNJDf2XGo&t=2991s) 全自然语言对话历来是人机交互中效率最低的途径，LLM 作为后台决策引擎直接输出概率而非解码长文本才是嵌入企业软件的最佳范式。
- [50:30](https://www.youtube.com/watch?v=TLJNJDf2XGo&t=3030s) 将概率注入代码控制流复兴了 1960 年代计算机科学以数学仿真为核心的概率编程思想，改变了软件二元确定性逻辑的本质。
- [52:51](https://www.youtube.com/watch?v=TLJNJDf2XGo&t=3171s) 前沿实验室执着于用自然语言“造神”（Beings），而广大工程师需要的是务实工程工具（Tools），这一世界观鸿沟促成了创新中心向外围软件的大转移。
## Detailed walkthrough
### [00:00](https://www.youtube.com/watch?v=TLJNJDf2XGo&t=0s) 视频导言：AI 监管与安全性的多维博弈
视频在紧张而深刻的快节奏剪辑中拉开序幕，几位嘉宾直接切入了当前科技界最核心的争论：AI 的监管、安全与未来架构。发言者们指出，过早的监管并不能消除风险，反而可能在尚未学会如何控制技术之前，就强行将其推向现实。当前，前沿 AI 实验室与网络安全社区之间存在着明显的裂痕，安全专家普遍认为实验室在处理安全问题时显得“粗心大意”，且在披露安全事件时有所保留。正如一位发言者所警告的那样：“如果你过早地对 AI 进行监管，实际上你什么问题也解决不了，最终你仍然要面对同样的风险。”
随着 AI 智能体（Agents）的兴起，安全威胁的性质正在发生根本性转变。传统的“人类员工”可能存在某种极端的“物种灭绝”概率，但“智能体集群”（Agent swarms）则彻底颠覆了这一逻辑。它们就像是成千上万个四处游荡的无人机，能够以极高的规模和速度运行，极易将错误或恶意的任务误判为正确指令。为了应对这种威胁，软件系统内部需要构建一个全新的安全层，用于深度追踪所有的身份验证和 API 调用。此外，嘉宾们还对地缘政治和监管格局表达了担忧，指出美国在 15 年前就停止了在科技反垄断领域的引领，而欧洲由于“没有什么可失去的”，正在抢占这一监管主导权。这一切都在表明，软件的本质正在发生根本性改变，创新的中心已经转移。
- **智能体集群的风险**：智能体集群就像是放大 10,000 倍的漫游无人机，它们不知疲倦，极易在执行任务时混淆好坏指令。
- **监管的滞后性**：过早的监管无法解决控制问题，只会将相同的风险延续下去。
- **欧洲的监管角色**：由于缺乏本土的超级科技巨头，欧洲在 AI 监管和反垄断上表现得更为激进。
```mermaid
flowchart TD
accTitle: AI 安全与监管的博弈逻辑
accDescr: 展示了前沿实验室、安全社区、企业信任与政府监管之间的相互作用和潜在冲突。
subgraph Labs["前沿 AI 实验室"]
A["技术创新与对齐研究"] -->|发布安全倡议| B["Dario 安全提案 (沙箱与测试)"]
end
subgraph Security["安全与企业生态"]
B -->|建立| C["企业信任"]
C -->|推动| D["AI 广泛普及"]
E["安全社区"] -->|指出| F["实验室安全漏洞与不透明"]
end
subgraph Regulation["监管风险"]
G["政治家与监管机构"] -->|可能利用安全诉求| H["过度监管与限制 (如限制数据中心)"]
H -->|导致| I["创新减缓与竞争受阻"]
end
B -.->|若被滥用| G
```
---
### [00:50](https://www.youtube.com/watch?v=TLJNJDf2XGo&t=50s) 播客开场与前沿 AI 实验室的安全性共识
节目正式开始，主持人 Erik Torenberg 热情欢迎嘉宾们回归。现场气氛十分轻松，大家调侃起 Martin Casado 的胡子，并开玩笑说他最近忙于打造那些“千亿美元级”的公司，以至于抽不出时间录节目——正如风险投资家们一贯喜欢把功劳揽在自己身上一样。随后，Erik 将话题引向正轨，询问 Box 首席执行官 Aaron Levie 对当前“前沿 AI 步调”（Pacing the frontier）以及随之而来的安全舆论有何看法。Aaron 笑着试图把这个棘手的话题先推给 Martin，但随后他给出了一个他认为所有人都会赞同的基调。
Aaron 指出，任何处于行业前沿的 AI 实验室，都必须以最安全的方式进行构建，并具备最高水平的治理、安全性和对齐（Alignment）研究。他强调，这不仅是道德层面的要求，更是 AI 技术在企业级市场普及的商业先决条件。正如 Aaron Levie 所言：“如果没有能够被企业信赖、且不会不断入侵系统的高质量产品，AI 是无法实现广泛普及的。”
在谈到 Anthropic 首席执行官 Dario Amodei 撰写的关于 AI 安全的文章（即“Dario 帖子”）时，Aaron 表示自己几乎完全赞同其中的观点，因为文章的核心在于如何通过沙箱技术（Sandboxing）、更严密的测试和更高的安全标准来保护系统。然而，真正的争议在于这些安全机制如何落地：谁来担任这些嵌入式的测试人员？整个行业能否达成共识？更重要的是，这些安全诉求是否会被政治家利用，从而演变成阻碍 AI 创新、限制数据中心建设或巩固前沿实验室垄断地位的监管工具。
- **商业普及的基石**：企业客户对 AI 的信任建立在绝对的安全和稳定之上，频繁出错或存在安全漏洞的 AI 无法进入主流商业市场。
- **达里奥（Dario）文章的共识**：行业普遍赞同加强沙箱化、系统测试和安全防护等技术手段。
- **监管滥用的担忧**：嘉宾们担心，合理的安全讨论可能会被转化为过度监管，从而延缓技术进步并削弱市场竞争。
### [02:59](https://www.youtube.com/watch?v=TLJNJDf2XGo&t=179s) 步伐控制（Pacing）与工程规范的辩论
Aaron Levie 谈到，围绕 AI 发展的核心实质内容其实极其重要，对整体 AI 技术的进步不可或缺。争论的焦点落在了究竟该采取什么行动，尤其是从监管层面应如何应对。在这一议题上，行业各方显然处于光谱的不同位置。Martin Casado 一直在据理力争，主张必须警惕某些举动演变成行业巨头的“监管俘获”（regulatory capture），Aaron 对此深表赞同。但他同时指出，所谓“控制发展步伐”（pacing）的讨论本身具备一定价值。
Aaron 认为，“控制步伐”这个说法在某种程度上甚至有些滑稽，因为其本质或许根本不是刻意放慢节奏，而是良好的工程规范与软件工程卫生习惯（good hygiene and good engineering）。在严谨的工程实践中，自然会出现微小的减速，但这种放缓是必要的前提，它能够换来技术更快、更广泛的扩散（diffusion）。毕竟，如果终端用户和企业根本不敢信任 AI，技术就绝无可能真正普及开来。
```mermaid
flowchart LR
accTitle: Engineering Hygiene vs Diffusion Speed
accDescr: A diagram showing how solid engineering practices lead to trust and broader AI diffusion.
subgraph Hygiene ["工程规范"]
A["良好工程卫生与规范"] --> B["短期微幅减速"]
end
subgraph Diffusion ["技术扩散"]
B --> C["建立用户与系统信任"]
C --> D["加速技术实际扩散"]
end
```
### [03:50](https://www.youtube.com/watch?v=TLJNJDf2XGo&t=230s) 舆论氛围的撕裂与“物种灭绝”的夸大论调
尽管 Aaron 认为相关的政策博文或提案本身很务实，但他强调当前的“舆论氛围”（atmospherics）完全崩塌了。他描绘了这样一个荒谬的场景：实验室里有员工在公开宣扬 AI 有 10% 的几率导致人类灭绝，而像 Anthropic 联合创始人 Dario Amodei 这样的行业领军人物，在电视上被问及员工声称有 10% 灭绝概率时，竟然公开表态说“我赞同他的地方多于反对他的地方”（I agree with him more than I disagree with him）。
Casado 接过话头指出，当高管当天在电视上公开认可物种灭绝论调时，整个对话环境就已经失控了。如果你真的相信这会导致人类灭绝，那么再回过头来发布一份温和、平淡的博文，就会显得像是“极其软弱无力的妥协投降，根本不足以应对眼前的危机任务”。Casado 直言，舆论氛围已经彻底失调，他之前的大量抨击正是针对这种扭曲的舆论叙事。
关于灭绝论与监管叙事的矛盾，Martin Casado 犀利地批判道：
> “This post that he has looks like this milktoast capitulation that's totally not adequate for the task at hand.”（他发布的这篇博文看起来就像是一种软弱无力的妥协投降，完全无法应对眼前的任务。）
### [04:30](https://www.youtube.com/watch?v=TLJNJDf2XGo&t=270s) “控制步伐”的概念误区与监管死局
Martin Casado 进一步表达了他在概念上的强烈不满，认为用“控制步伐（pacing）”来描述安全策略是彻底错误的。首先，控制速度与系统的真实安全性完全正交——你可以极其缓慢地制造一枚核武器，但这绝不会因为制造过程缓慢而让任何人感到更安全。其次，这种表态更像是在向主张“暂停 AI 研发”（pause folks）的群体暗送秋波、委曲求全，却根本没有解决实质问题。
这种试图两头讨好的策略最终导致谁都不买账：
- **暂停派（Pause folks）**会驳斥：“这根本不是暂停，这只是控制一下步子而已。”
- **监管机构（Regulators）**则会指责：“你们归根结底还是在继续做那件极其危险的事。”
Casado 认为，前沿实验室的初衷确实是在尝试做正确的事，其技术方案本身也很务实，值得肯定。但他们的对外信息传递（messaging）完全走偏了，试图在内部的极端末日派（doomers）与外部的监管者之间左右逢源。Casado 强调，实验室唯一正确的出路是正面直面生存风险（x-risk）的质疑，明确表态不认为自己正在研发的技术会导致物种灭绝。唯有打破这种虚伪的氛围，整个讨论才能回归理性。
此时，Aaron 提出反思，希望站在对立面探讨一种真实存在的心理悖论：如何看待那些处于矛盾重叠区的实验室研究员——他们既打心底里感到极度恐惧，却又全身心投入到推进 AI 前沿能力的工作之中。
关于步伐与安全性的脱节，Martin Casado 明确指出：
> “I think pacing is the wrong way to describe this. For one, yes, it is orthogonal to security.”（我认为用控制步伐来描述这一点是错误的。首先，没错，它与安全性完全是正交的。）
### [06:06](https://www.youtube.com/watch?v=TLJNJDf2XGo&t=366s) 真正的信徒与核武级别的存在风险：国家化还是 HR 问题？
Aaron Levie 试图为 AI 实验室内部的某些从业者辩护，指出行业内确实存在这样一类人：他们坚信把 AI 这件事做对极其关键，因此希望站在技术最前沿，即便他们使用的语言如今看来充满问题；他们既对技术的潜在危害深感恐惧，又认为唯有亲自下场研发才能掌控局面。Martin Casado 随即对此予以坚决反驳，搬出了自己在国家级安全领域的亲身履历。Casado 透露自己曾在劳伦斯利弗莫尔国家实验室（Lawrence Livermore National Laboratory）从事核武器项目研发，深知接触真正具有毁灭性潜力的技术意味着什么。Steven Sinofsky 随即调侃 Casado 是当年掌控节奏的“最早领跑者/定步调者”（first pacer）。
Casado 犀利地指出，如果前沿实验室中那些最懂技术的核心群体真的相信这项技术存在灭绝人类级别的存在风险（existential risk），那么合乎逻辑的唯一解决路径就是直接将其“国家化”（nationalize it），采用经过历史验证的武器级物理与安全管控手段。相反，Casado 指出，他在私下交流中接触到的绝大多数理性研究人员根本不相信存在这种存在风险，所谓灭绝论不过是极少数人的执念；如果实验室只是担心招不到人或留不住顶尖研究员，把这当作迎合人才的“研究员货币”（researcher currency），那这本质上只是个“人力资源问题”（an HR problem）。绝不能因为一家公司的内部 HR 留人困难，就推向全国层面的严厉封锁、扼杀一项极具前景的通用计算技术。
“The answer is to nationalize it and actually put controls that we know that work.” —— Martin Casado
- 核心矛盾：如果真有灭绝级风险（X-risk），唯一严肃的应对方式是国家化收归国有并实施武器级管控制度。
- 私下现实：Casado 接触到的理性从业者并不相信世界末日论；实验室高谈阔论风险，在很大程度上是将迎合担忧视作招募和挽留顶尖研究员的“HR 工具”。
```mermaid
flowchart TD
accTitle: AI Existential Risk Logic vs HR Reality
accDescr: 实验室在宣称存在风险时的逻辑谬误：真有灭亡风险应收归国有，实则只是留人招人的 HR 策略。
subgraph LAB_CLAIMS ["实验室宣称的风险假说"]
A["顶尖从业者声称存在灭绝风险 (X-Risk)"]
B["应有的逻辑结果: 国家化与军工级严控"]
end
subgraph REALITY_CHECK ["行业真实动因与现状"]
C["私下交流: 绝大多数理性人员并不相信末日论"]
D["真实动因: 迎合少数天才研究员的 HR 留人问题"]
E["错误做法: 以 HR 问题为借口推动全行业全国性封锁"]
end
A --> B
A -.->|"逻辑脱节"| C
C --> D
D --> E
```
### [07:50](https://www.youtube.com/watch?v=TLJNJDf2XGo&t=470s) 所谓的“放缓节奏”为何纯属无稽之谈？
讨论随后转向各大前沿实验室宣称要“控制研发与发布节奏”（pacing）的公关话术。Steven Sinofsky 率先开火，直斥这种“定节奏”的说法完全是不真诚的伪命题。他质问道，实验室是否曾公开发布过一份排期表，明确说明灾难性事件究竟会在何时发生？如果没有基准日程表，没人知道这项技术原本应该在什么时候完成，那么宣称“放缓节奏”就毫无意义。Sinofsky 将此比作媒体报道“苹果最新的 iPhone 推迟发布”一样荒唐——去指责一款苹果从未公布过、没人知道是否存在的手机延期，纯属虚构事实。Levie 随即打趣接话：“没错，我的苹果汽车也严重延期了。”
Sinofsky 与 Casado 一致认为，要衡量某项事物变慢了，前提是必须先知道它原本的前进速率；没有基线，“放慢节奏”就是彻头彻尾的废话。此外，从公关和实际表现来看，没有任何人相信这种说辞。各家实验室在现实中都在以百米冲刺的极限速度向前狂奔：融资规模创下历史新高，扩张速度前所未有，行动上完全看不到任何放慢脚步的迹象。Levie 补充说明，问题在于实验室内部持有的前沿模型能力远远超出外界能接触到的公开发布版本，因此他们所谓的定节奏，其实仅仅是指向公众发布产品的节奏。然而正如 Sinofsky 反问的那样，究竟是针对什么在定节奏？没人知道那个让人恐慌的模型到底是否存在，所谓的自我克制只是一场缺乏透明度与基准对照的行业自嗨。
“In order for something to be slower, you need to know the rate at which it was moving in the first place.” —— Steven Sinofsky
- 荒谬的延期逻辑：在没有公布既定时间表和速度基准的前提下，谈论“放慢节奏”或“延期”在逻辑上无法成立。
- 行动与言论的脱节：实验室表面大谈自我克制与审慎放缓，实际行动上却在以史上最疯狂的速度疯狂融资和扩充算力。
### [09:09](https://www.youtube.com/watch?v=TLJNJDf2XGo&t=549s) 叫喊末日却求监管的虚伪逻辑与导弹工厂的回忆
与会者深入剖析了前沿实验室对自身技术的矛盾态度：一方面宣称模型可能带来灭顶之灾，另一方面却连基本的实际应用都充满不确定性。那种让人吓破胆的模型，甚至无法靠谱地写好一份法律摘要，搞不好还会把各种细节全搞砸。大家指出，宣称自己在主动“控制节奏（pacing）”，却高调向外界公告并恳求政府出面来踩刹车，这种做法极其虚伪。“If you are the most afraid of how everything is going to go because of your product, just stop.” 对此，其他人直截了当地附和道：真怕就别做。
```mermaid
flowchart TD
accTitle: AI实验室对监管诉求的逻辑陷阱
accDescr: 前沿实验室渲染AI风险并呼吁政府监管，反而引发不可逆的立法惯性。
subgraph LABS["前沿实验室行为"]
A["高调渲染AI带来灭绝风险"] --> B["宣称自我节制，呼吁政府介入监管"]
end
subgraph REALITY["现实反馈机制"]
B --> C["政府机制：多方妥协，输出极端结果"]
C --> D["监管飞轮启动：成为各级选举议题"]
D --> E["不可逆的错误节奏与全面规管"]
end
```
谈话中出现了一段幽默而带有时代印记的插曲。有人回忆起大学时代在一家导弹工厂打工的真实经历，当时厂区里造的就是真正的核导弹，他每天在车间里走动。同行者忍不住调侃：“你们怎么老跟导弹扯上关系？我可只是个做软件的。”发言者解释说，在那个年代的大学校园里，人基本上只有两种选择：要么参加抗议要求把导弹项目赶出校园，要么就进厂参与制造。他回忆道，当时甚至有天主教修女跑来把鲜血泼在导弹上抗议，而他自己还只是个推着手推车到处送电脑的小年轻，战战兢兢地喊着“这是安全电脑”，心里其实怕得要死，完全不明白周围在发生什么。但后来历史证明，正是那些潘兴导弹（Pershing missile）阻止了冷战继续升级。这说明技术本身和战略威慑有着极其复杂的现实逻辑，不能简单用口头恐慌来概括。
### [10:19](https://www.youtube.com/watch?v=TLJNJDf2XGo&t=619s) 监管的政治现实与“控制节奏”的幻觉
话题随后转向前沿实验室所谓的“控制节奏（pacing）”。与会者指出，“pacing”完全是一个模糊不清的伪词汇，尴尬地卡在“全速推进”与“止步不前”的夹缝之中，中间道路绝不可能让任何人感到满意。
观察科技圈人士如何游说华盛顿，简直成了一种充满戏剧性的“体育赛事”。这些高管和从业者真诚地以为，跑去向政府提意见、告诉官员该做什么不该做什么，就能如愿以偿拿到自己想要的政策。这种想法百分之百暴露了他们对政府运作机制的无知。政府非常清楚自身规则，官员听取你的意见，同时也听取所有其他利益方的意见。在现代政治体制下，任何政策落地都必然是各方妥协的产物：
> “And so everything that that all the inputs can make it into the government, the output never makes everyone happy, right? 100% of the time it either doesn't go far enough or it goes way way too far.”
因此，如果科技公司寄希望于通过与政府谈判来定制精准的监管方案，结局必然南辕北辙——当你主动向政府乞求放慢节奏时，最终得到的必然是一个错误的速度。有人顺势提及政界对科技行业的戏剧性态度，例如大卫·萨克斯（David Sacks）或某些政界人士对类似诉求的讽刺：当公司跑去求政府监管自己时，政府的反问往往充满荒谬感，甚至像当年对待 Facebook 那样直接予以回绝。
### [11:51](https://www.youtube.com/watch?v=TLJNJDf2XGo&t=711s) 监管车轮不可逆转：2028 年的“AI 大选”
与会者达成高度共识：历史经验证明，监管的轮子一旦开始转动，没有任何力量能让它慢下来。如今，AI 监管已经迅速演变成从联邦到地方各级政府、涵盖每一个党派的核心选举议题，衍生出五花八门、盘根错节的监管工具箱。
大家一致预测，未来的政治竞选将不可避免地围绕 AI 展开：
> “Well, the the next election will 100% be a referendum on on AI. So, like it has to happen that that 2028 is like the AI election and you could basically run on the problem is is like it's not obvious.”
2028 年的美国总统大选将注定成为一场关于 AI 走向的“全民公投”。政客们将把 AI 作为竞选纲领的核心抓手，然而围绕这一技术的政治博弈和利益撕裂，也意味着未来的监管路径将变得极度复杂与不可捉摸。
### [12:21](https://www.youtube.com/watch?v=TLJNJDf2XGo&t=741s) 缺乏话语权的“亲 AI”政治叙事
在探讨 AI 如何影响未来的政治选举时，嘉宾们指出，目前很难有政治候选人能够依靠“亲 AI”（pro-AI）的立场来赢得选战。原因在于，支持 AI 的故事显得过于抽象、复杂且充满细节，普通选民很难在短时间内消化。这导致支持 AI 的阵营在舆论场上始终处于防守态势，甚至连辩论的话语权都完全被对手掌控。
目前关于 AI 的公共舆论，其核心词汇几乎全部由那些对 AI 持怀疑或反对态度的人所定义。诸如“暂停”（pause）、“蜂群”（swarms）、“失控”（rogue）等带有恐慌色彩的词汇充斥着媒体。支持 AI 的一方如果想要反驳，就必须先花大量的篇幅去解释为什么这些词汇是错误的，这在政治传播中极其低效。嘉宾们开玩笑说，支持 AI 的阵营需要建立一套全新的“词云”，引入诸如“工会工作”（union jobs）和“攻克癌症”（cancer cures）等更具社会效益和感召力的词汇，才能重新夺回舆论主动权。
一位嘉宾极其犀利地指出：
> “这是防御性的，而且我们根本不掌握任何词汇，对吧？所以整个辩论都是关于‘暂停’、‘蜂群’、‘失控’。每一个词都是由那些不想发展 AI 的人挑选的。”
```mermaid
flowchart TD
accTitle: AI 舆论战中的词汇对比
accDescr: 展示了反对 AI 阵营所主导的消极词汇与支持 AI 阵营亟需建立的积极词汇之间的对比。
subgraph Anti["反对 AI 阵营（主导当前舆论）"]
A["暂停 (Pause)"]
B["蜂群 (Swarms)"]
C["失控 (Rogue)"]
end
subgraph Pro["支持 AI 阵营（亟需建立的叙事）"]
D["工会工作 (Union Jobs)"]
E["攻克癌症 (Cancer Cures)"]
F["社会福祉 (Social Benefits)"]
end
Anti -->|"使支持者陷入防御"| Pro
```
---
### [13:16](https://www.youtube.com/watch?v=TLJNJDf2XGo&t=796s) 实验室在生存风险（x-risk）上的尴尬立场
Martin Casado 提出了一个令他困惑的问题：为什么各大 AI 前沿实验室（如 OpenAI、Anthropic 等）至今没有在“生存风险”（x-risk，即 AI 导致人类灭绝的风险）上采取明确的否定立场？他认为，如果实验室不主动站出来澄清，那么等待整个行业的将只有极其严厉和笨拙的政府监管。如果连行业内最顶尖公司的 CEO 都公开附和“AI 有 10% 的概率导致人类灭绝”这种言论，政府出于职责所在，根本不可能坐视不管。
舆论中提到，Anthropic 的 CEO Dario Amodei 虽然在口头上迎合了存在灭绝风险的说法，但他总是刻意避免给出一个具体的概率数字。Aaron Levie 认为，这种所谓的“生存风险百分比”完全是一个凭空捏造、根本无法被科学量化的概念。在目前的公关环境下，实验室能够给出的唯一既符合公关需求、又在智识上诚实的表态只能是：“AI 确实存在真实风险，但也有巨大的正面效益。我们正在努力降低风险，使其降到人类极限之下的最低水平。”
Martin Casado 强调了政府面对这种言论时无法坐视不管的逻辑：
> “如果政府表现得像‘有 10% 的物种灭绝概率，而顶尖公司的 CEO 也表示同意’，那政府就太失职了。面对这种情况，政府怎么可能不进行严厉的监管呢？”
---
### [14:59](https://www.youtube.com/watch?v=TLJNJDf2XGo&t=899s) 历史科技浪潮中的风险讨论
针对“必须讨论生存风险才能确保安全”的观点，Steven Sinofsky 表达了强烈的反对。他带领大家回顾了科技发展的历史，指出人类社会已经安全度过了多个重大的技术变革时代，包括个人计算机的普及、互联网的诞生、万维网的兴起以及社交网络的爆发。
在这些历史阶段中，社会各界同样对新技术带来的风险进行了极其深入和广泛的讨论，但人们从未将这些风险上升到“人类灭绝”（x-risk）这种极端的高度。过去，行业和监管机构关注的是具体、实际且可解决的问题，而不是陷入虚无缥缈的末日预言中。
Steven Sinofsky 总结道：
> “我们已经经历了多个技术时代。我们经历了计算机，经历了互联网，经历了万维网，经历了社交网络。我们在讨论风险时，根本不需要谈论生存风险，对吧？”
### [15:14](https://www.youtube.com/watch?v=TLJNJDf2XGo&t=914s) “非零”灭绝风险与冷战军备的逻辑推演
对话围绕 AI 带来的物种灭绝风险（x-risk）展开了针锋相对的交锋。几位嘉宾探讨了边际风险（marginal risk）与绝对风险的区别。如果 AI 研发机构真的认为物种灭绝的边际风险显著高于以往技术，那么合乎逻辑的结论只有两个：要么承认人类将走向灭亡而把项目全部关停（“shut it all down”），要么承认这只是一个极小的概率。有人提到 Dario（Anthropic 联合创始人 Dario Amodei）评估的概率相对较低，但这让人联想起五角大楼在冷战时期的做法——当时军方针对核战争爆发概率进行了大量推演（如同电影《战争游戏》（WarGames）中所刻画的那样），核心结论是其概率是“非零”（non-zero）的。
```mermaid
flowchart TD
accTitle: AI灭绝风险论述与应对路径
accDescr: 展示一旦承认非零灭绝风险后所面临的监管与国有化逻辑推演
A["宣称 AI 存在非零 (Non-Zero)<br/>灭绝风险"] --> B{"逻辑推论选项"}
B -->|"选项 1"| C["彻底关停研发<br/>(Shut it down)"]
B -->|"选项 2"| D["实质国有化管制<br/>(BSL-4/军工级别管控)"]
B -->|"实验室真实诉求"| E["获取特许经营权与牌照<br/>(仿效银行等受监管巨头)"]
E -.->|"代价值得警惕"| F["扼杀前沿开源创新"]
```
然而，一旦在公众沟通中抛出“非零”这个概念，问题就变得极为棘手：非零既可以意味着十亿分之一，也可以代表某些极端人士口中的 93%。有人指出，只要一家实验室公开声称灭绝风险是“非零”的灾难性事件，从逻辑终点推导，唯一的治理手段就是将其国有化。然而各家实验室之所以把灭绝风险定性为非零，本质上是试图走钢丝：他们既想借此确立“非零风险”的特殊地位，从而拿到某种排他性的准入牌照，又想极力摆脱被政府全盘接管、收归国有的沉重负担。
- “I think as soon as you think it's non zero, the only answer of like catastrophic, the only answer is you have to nationalize it.”（我认为，只要你认定灾难性灭绝的概率是非零的，唯一的应对方案就只能是将其国有化。）
### [16:34](https://www.youtube.com/watch?v=TLJNJDf2XGo&t=994s) 牌照监管、关键基础设施与国有化倾向
随后讨论深入到了历史先例与产业演进模式。嘉宾们对比了高危科研领域（如研发炭疽杆菌等致命病原体的生物安全四级实验室 BSL-4），指出这类涉及极高生存风险（x-risk）的机构实质上都被国家严密掌控或直接归属于国家。与之相对，常规的生产安全往往先由行业自律组织牵头，随后才逐步演化为联邦法规。
回顾二战以后的历史趋势，从电力、能源等关键基础设施，到银行业与医疗体系，大部分命脉行业的监管要求（如严苛的 KYC 反洗钱核查与合规标准）日益繁重，使其在实际运作层面几乎等同于“被国有化”（如金融危机后的各大银行）。 Aaron Levie 等人指出，AI 前沿实验室内心深处的终极目标，或许正是想成为人工智能时代的摩根大通（JPMorgan Chase）或威瑞森（Verizon）——作为国家关键基础设施运营，接受极度严格的牌照制监管，借此筑起极高的准入门槛。虽然这对前沿开源生态（frontier open source）将是沉重打击、扼杀创新活力，但在商业巨头眼中，这却是一个可以接受甚至极力谋求的稳固格局。
大家一致认为，企业追求成为受监管的基础设施并非不可理解，但不应把“人类灭绝”当作拿来游说和筑墙的幌子。不过对话也对近期的行业动向给予了肯定，认为各大实验室的公开表态正朝着理性、务实的方向修正，许多高管与行业领袖已经深刻意识到这一论述方式所带来的反噬与责任。
- “The trend has been to basically be nationalized, by just like your examples of KYC and all the other stuff. The banks are for all practical purposes nationalized in the financial crisis.”（自二战以来的趋势，就是像你提到的反洗钱KYC等各种规定一样，基本上走向了国有化；从所有实际运转目的来看，各大银行在金融危机后就已经被国有化了。）
- “I think even that's fine, just don't use species extinction as your literally is like the difference between the things that are like stuck in the lab.”（我觉得即便想做受监管的基建巨头也没问题，只要别再拿‘物种灭绝’来作为把技术锁在实验室里的借口就行。）
### [18:25](https://www.youtube.com/watch?v=TLJNJDf2XGo&t=1105s) 科技行业的政治天真与百年历史教训
Aaron Levie 表达了对前沿 AI 实验室的乐观态度，认为它们意识到了自身面临的张力并试图做出正确的选择，核心问题在于发展速度过快，企业还在费力摸索这套机制究竟该如何运作。然而，Steven Sinofsky 提出了极为尖锐的批评，直言政治运作完全是另一套独立机制，科技公司在华盛顿面前要么是天真（naivety），要么是自大（hubris），根本不知道如何应对。
Sinofsky 回顾历史指出，科技行业在过去 100 多年的每一波技术浪潮中，都在反复承受同样的教训：“Well, I think the tech industry has literally over 100 years consistently relearn the lesson at each technology wave that we don't understand the regulatory climate and we can't navigate it.” 哪怕是像 AT&T 和 IBM 这样最初几乎伴随政府垄断地位诞生的企业，在 20 世纪 60 年代雇佣了数百名顶尖律师，也依然没能摸透监管规则，最终双双遭遇反垄断诉讼并经历了伤筋动骨的结构性重组。微软当年面对反垄断风暴时同样茫然无措；Bill Gates 前脚还在和 Bill Clinton 一起打高尔夫球，后脚克林顿政府就对微软发起了反垄断诉讼，让盖茨大惑不解——他甚至拿出合影纳闷：“我明明去打了高尔夫，难道这不就是我该做的事吗？”
- AT&T 和 IBM 拥有庞大法务团队与深厚政府渊源，依然深陷反垄断拆分泥潭。
- 微软在 1990 年代面临反垄断审查时，高管层同样误以为依靠私人政商关系（如打高尔夫）便能化解危机。
### [19:40](https://www.youtube.com/watch?v=TLJNJDf2XGo&t=1180s) MPAA 自律模式 vs FINRA 监管方案
Sinofsky 引入了电影行业面对政府审查时的应对范例。在“红色恐怖”（Red Scare）以及政府试图针对性内容、成人题材进行审查的威胁下，好莱坞片厂联合成立了美国电影协会（MPAA），通过自行制定电影分级制度完成了行业自律，从而避免了政府直接立法规管。
```mermaid
flowchart TD
accTitle: 两种行业规制模式的对比
accDescr: 对比好莱坞MPAA自律评级模式与金融界FINRA国家立法模式的运作差异
subgraph MPAA["MPAA 模式（行业自律）"]
A1["外部危机：政府审查与政治猎巫"] --> A2["行业结盟：成立美国电影协会"]
A2 --> A3["自律评级：非强制性分级体系"]
A3 --> A4["结果：规避宪法强制，催生 HBO 等有线网络繁荣"]
end
subgraph FINRA["FINRA 模式（国家化监管）"]
B1["外部危机：金融市场系统性风险"] --> B2["国会立法：设立法定监管机构"]
B2 --> B3["直接监督：政府授权与强制执行力"]
B3 --> B4["结果：风险实质国家化，准入门槛剧增"]
end
```
围绕这一历史经验，嘉宾们就是否应将金融业的 FINRA（美国金融业监管局）模式套用至 AI 领域展开辩论。针对主持人提出的 FINRA 设想，Sinofsky 坚决反对，指出 MPAA 与 FINRA 存在本质区别：MPAA 是纯粹的民间自律，无法在宪法层面被强制推行（因此后来诞生了不受传统分级禁锢的 HBO 等有线电视频道）；而 FINRA 背后是国会立法与政府直接监督。
- MPAA 诞生于麦卡锡时期的严峻氛围，当时约 30% 的好莱坞从业者因涉共指控被开除或列入黑名单，行业才被迫通过自律评级求生。
- Aaron Levie 质疑电影题材并不构成生存层面的社会风险（Societal Risk），而 AI 涉及的言论自由与第一修正案后果远比电影分级严重。
- Sinofsky 反驳称历史的严重性是相对的，不能以当下的轻松视角低估当年好莱坞面临的生死危机。
- 讨论最终落脚于 FINRA 的致命缺陷：将金融机构的自律机构（SRO）法制化，本质上是“将风险国有化”（Nationalizing risk）。
### [21:25](https://www.youtube.com/watch?v=TLJNJDf2XGo&t=1285s) AI 监管与 FINRA 模式的现实对比
当谈到金融业由各大银行出资却全盘受国家法规划定与强制推行的监管机制（如金融业监管局 FINRA）时，讨论聚焦于 AI 领域是否可能迎来比 FINRA 更好的监管结局。Aaron Levie 坦言，FINRA 看似是最好的情景，但这或许已不再是上限。他认为，AI 技术在极限状态下所能释放的能力过于庞大，国会绝不可能对其坐视不理：“像这样强大的技术，最终国会不可能不关心它。这根本是不可能的。”
```mermaid
flowchart TD
accTitle: AI渗透关键行业引发监管必然性
accDescr: 展示开源或深度模型进入医疗、高频交易及航空等核心领域后触发国会严厉监管的传导路径
subgraph CriticalDomains["关键领域深度渗透"]
A["医疗诊断决策与设备<br/>(含开源权重模型)"]
B["高频交易系统<br/>(金融市场核心)"]
C["航空机载系统<br/>(高危物理世界)"]
end
subgraph GovReaction["政府监管升级"]
D["国会介入与立法"]
E["FINRA 式行业强制监管架构<br/>(最佳情况/基准线)"]
end
A --> D
B --> D
C --> D
D --> E
```
如果 AI 最终渗透到医疗诊断流程中做出每一项治疗推荐、作为开源权重模型（open weights model）嵌入医疗器械中、主导华尔街每一个高频交易系统，甚至直接运行在客机上，政府必定会出手干预。在那种严苛的渗透率下，类似 FINRA 这种由行业买单、政府强行立规的自律组织模式，反而可能已经是行业所能争取的“最乐观方案”。
### [22:16](https://www.youtube.com/watch?v=TLJNJDf2XGo&t=1336s) 错失互联网监管的华盛顿与针对科技业的积怨
Steven Sinofsky 敏锐地指出，这正是整个讨论最关键的节点。如今参议院里的许多资深议员，当年在通过《通信规范法案》（Communications Decency Act）时就已身居高位。当时国会选择豁免互联网服务提供商（ISP）和社交网络的内容责任（如著名的第 230 条款），导致后来议员们开会反思时常常抱怨：“互联网规模这么大，我们怎么会完全没有插手管辖？”当年阿尔·戈尔（Al Gore）因为宣称自己“创造了互联网”而广受嘲弄，其实背后真实的原因是他试图抢先站出来表明联邦政府在推动技术中发挥了决定性作用，结果弄巧成拙，反而让自己在公众眼里像个疯子。
华盛顿政界普遍存在一种根深蒂固的心结：他们觉得自己当年错过了掌控互联网的黄金机会。如今在 AI 议题上，这种心理被无限放大：
- **戈尔角色的缺失**：当年戈尔至少代表了拥抱创新的积极面，而如今华盛顿政客中鲜有人扮演这种保护创新的角色；尽管国防系统希望某些技术由民间主导，但态度依然摇摆不定。
- **监管情绪的借题发挥**：以伊丽莎白·沃伦（Elizabeth Warren）为代表的参议员频频发推，直呼“我们在社交网络上错失了监管先机”。
- **反科技情绪的转移**：华盛顿多年来积累的针对科技大厂的反感情绪，如今正在全盘倾泻并转移到 AI 身上。
### [23:32](https://www.youtube.com/watch?v=TLJNJDf2XGo&t=1412s) 前置预防与互联网早期网络灾难的历史回溯
Martin Casado 提出了整个 AI 监管讨论中最深层的悖论：当前的社会和政府正在试图“预测潜在的坏事”，而这完全违背了科技与监管演进的历史规律。在过去的科技浪潮中，人们从来不是通过预先算命来立法规制风险的。在互联网发展的同一阶段，网络攻击和恶意代码已经造成了数百亿美元的实际经济损失，甚至有蠕虫病毒瘫痪了全网 10% 的基础设施，导致医院系统停摆。
Aaron Levie 随即打趣道：“照这么说，我们真应该在 1997 年就彻底把互联网封掉。”
Martin Casado 随后分享了一段生动的亲身经历：当年买了一张 Windows 95 安装光盘，结果系统刚装完、还没等开始正常使用，电脑就已经被网络蠕虫感染了。在大家顺势拿曾领导 Windows 部门的 Steven Sinofsky 开玩笑（“干得好啊，Steven！”）时，Steven 坦承这正是那个时代的残酷技术现实：“从那时起到 2001 年，个人电脑的现实情况就是：你根本无法在连接网络的状态下安装一台 PC 而不被感染。”当时的科技世界是在承受了巨额实际灾难与技术创伤之后，才逐步建立了后来的防御体系和规则，绝非靠凭空推测未来的危害。
### [24:26](https://www.youtube.com/watch?v=TLJNJDf2XGo&t=1466s) 历史经验：前瞻性政策 vs. 基于具体事实的技术监管
对话聚焦于科技史上的阵痛期。在 PC 和早期互联网时代，电脑几乎一联网就会被病毒感染，恶意软件无处不在，国会因此举行了多轮听证会。Aaron Levie 指出，如果 1994 年的人们也抱着如今这种对待 AI 的恐慌思潮（zeitgeist），“我们今天很可能根本就不会拥有互联网了”。
任何重大工业革命初期都存在一段痛苦的摸索适应期（teeth-cutting period）。无论是汽车、航空还是互联网的初期发展，技术并非诞生即完美：
- 系统动辄宕机崩溃。
- 全球经济损失屡见不鲜。
- 新型病毒与蠕虫层出不穷，四处蔓延。
- “千年虫”（Y2K）危机曾引发甚至要毁灭现代文明的极度恐慌。
Martin Casado 针对政策制定的核心逻辑提出了关键论点：过去的监管之所以有效，是因为政策制定者掌握着极其具体的现实数据点和案件事实（fact pattern），能够精准对症下药。反观当前的 AI 安全监管，人们总是在假设“如果未来发生某种灾难该怎么办”，这种脱离实证的前瞻性政策（predictive policy）在操作层面上极难成立。Martin Casado 强调：
> “当我们在制定政策并且确实出台了大量政策时，那都是基于针对你所要解决的问题的大量具体数据点，这样你才能确信这项政策确实契合了事实案情。”
当讨论逐步回归到真实的工程挑战（例如具体的网络安全风险）时，各大前沿实验室也在将研究重点拉回到现实工程防御上。一旦危险不再是虚无缥缈的科幻灾难，而是被明确界定的具体风险，整个行业自然会达成共识并协同推进，但这绝非此前主流舆论中那种形而上学的安全叙事。
```mermaid
flowchart TD
accTitle: 历史监管路径与AI超前监管路径对比
accDescr: 对比传统科技监管基于具体事实出台法规与当前AI监管基于前瞻性恐慌推行政策的逻辑差异。
subgraph Historical["传统科技监管路径 (1980s-1990s)"]
A["技术爆发与混乱期<br/>(频繁宕机 / 蠕虫病毒 / 经济损失)"] --> B["具体破坏事件曝光<br/>(确凿案情事实 / 明确损害)"]
B --> C["针对性精准立法<br/>(例如 CFAA 反黑客法)"]
end
subgraph CurrentAI["当前 AI 监管困境"]
D["大模型能力突破"] --> E["虚构未来威胁假设<br/>(灭绝论 / 前瞻性推演)"]
E --> F["试图制定前瞻性政策<br/>(缺乏事实依据 / 阻碍底层创新)"]
end
```
---
### [25:46](https://www.youtube.com/watch?v=TLJNJDf2XGo&t=1546s) CFAA 的诞生、应用层法律与模型层责任之争
Steven Sinofsky 追溯了网络犯罪立法的源头——1986 年签署生效的《计算机欺诈与滥用法案》（CFAA，Computer Fraud and Abuse Act）。该法案的催化剂是一个极为具体的事件：1983 年，两伙黑客入侵了 GTE Telenet 商业网络。因为 GTE Telenet 当时承载了 NASA 以及劳伦斯利弗莫尔国家实验室（Lawrence Livermore National Laboratory）等联邦核心科研机构的数据，黑客入侵立刻惊动了华盛顿特区。但在 1983 年，这种未经授权擅自进入他人计算机的行为在法律上甚至不构成犯罪。国会历经长达两年半的反复辩论与打磨，才最终确立了这项明确禁止“未经授权访问计算机系统”的里程碑式法律。
围绕这一历史先例，Aaron Levie 提出了一个尖锐的划分问题：现有的法律体系中，大约 90% 已经完整覆盖了应用层（如非法侵入网络、盗取数据等违法行为本就属于犯罪），那么在纯粹的大模型层（Model Layer），究竟是否应该施加额外的法律责任？
Steven Sinofsky 进一步解释了技术与法律演进的摩擦。事实上，如果严格按照最初 CFAA 的条文来看，各大 AI 实验室（如 Hugging Face、OpenAI 等）抓取数据或进行安全探测的行为甚至都可能构成计算机犯罪。然而，法律后来针对“白帽黑客”（White Hat）设立了豁免修正案——因为许多安全研究员在尝试修复系统漏洞时难免出现失误，司法部门绝不希望因此去拘捕那些本意是改善系统安全的人。为此，司法部（DOJ）专门出台备忘录明确指导方针：
- 不会因为单纯的研究失误而起诉白帽安全人员。
- 不会单纯因为违反了网站的“服务条款”（Terms of Use）就将其定性为黑客入侵行为。
Steven Sinofsky 据此认定，现有法律在应对 AI 带来的安全场景时已经足够完备，完全不需要急于把底层技术本身视为非法。针对模型层与法律边界的界定，Steven Sinofsky 直言：
> “而且司法部还写过一份备忘录，明确表示：‘我们不会为此提起诉讼；如果你只是违反了系统的服务条款，而不是真正去恶意攻破它，我们同样不会予以起诉。’”
### [27:32](https://www.youtube.com/watch?v=TLJNJDf2XGo&t=1652s) AI 前沿实验室与安全社区的巨大裂痕
谈话深入到网络安全监管与事故复盘的严谨性问题。当时黑客入侵联邦系统引发了华盛顿特区的高度关注，而如今整个行业面临的尴尬现状是：AI 前沿实验室与传统网络安全社区之间存在着巨大的认知鸿沟与信任裂痕。资深安全专家审视 AI 实验室发布的事故复盘（postmortem）报告时，通常会得出两个尖锐的结论：其一是做法极其“草率”（Sloppy），其二是在交代事故过程时含糊其辞、残缺不全。
```mermaid
flowchart TD
accTitle: AI安全复盘与传统CVE标准对比
accDescr: 展示了传统成熟的CVE披露流程与当前前沿AI实验室公关式复盘的鲜明反差。
subgraph Traditional ["传统网络安全标准 (1980s 至今)"]
CMU["CMU CERT / CVE 体系"]
Standard["高度结构化的漏洞报告"]
Obligation["强制披露与透明义务"]
CMU --> Standard --> Obligation
end
subgraph CurrentAI ["当前 AI 实验室现状"]
Postmortem["公关式事后复盘"]
LawyerFilter["聘请外部律所调查 (选择性过滤)"]
Critique["'如同实习生手笔' / 隐匿 Slack 记录"]
Postmortem --> LawyerFilter --> Critique
end
```
早自 20 世纪 80 年代卡内基梅隆大学（CMU）牵头建立计算机病毒及漏洞披露机制以来，行业便演进出成熟的 CVE（通用漏洞披露）流程，确立了极其结构化的报告规范与强制披露义务。然而，现在的 AI 实验室却对这些现成且行之有效的行业标准视若无睹。针对这种不负责任的披露方式，发言者尖锐地抨击道：“like shouldn't do a postmortem on a breach that looks like an intern wrote it and it's not a postmortem it's this selective memory it looks like exactly the kind of postmortem you do when you hire outside lawyers to investigate some random thing and you only give them certain stuff”。这完全是选择性失忆，就像是企业为了免责聘请外部律师介入调查，却对核心证据层层把关，连关键的 Slack 聊天记录和具体的事故细节都秘而不宣。
### [28:40](https://www.youtube.com/watch?v=TLJNJDf2XGo&t=1720s) 网络安全界的思维定式与“隐蔽信道”之争
Martin Casado 随后插话，以亲历者的身份剖析了网络安全圈长期存在的某种偏执心态。他指出，真正的网络安全社区往往天然抗拒“务实的解决方案”（practical solutions），哪怕你构建了一个看似固若金汤的安全系统，安全专家们也总会脑补出好莱坞电影般的极端攻击假想——仿佛随时会有罗素·克劳（Russell Crowe）或《碟中谍》（Mission Impossible）特工队凭空杀出破解加密算法。
```mermaid
flowchart LR
accTitle: 超级智能外泄讨论的跨界桥接
accDescr: 描绘了Noam Brown提出的热量外泄假说如何促使生存风险倡导者与硬核系统工程师展开对话。
A["Noam Brown 假想:<br/>通过 CPU 热量实现模型外泄"] --> B["X-risk 阵营:<br/>讨论超级智能的不可预测性"]
A --> C["硬核系统安全社区:<br/>精确测算热量隐蔽信道比特率"]
B <-->|"建立事实与工程的对话桥梁"| C
```
正是这种对极端威胁的痴迷，让最近发生的一起争论变得极具代表性。OpenAI 的研究员 Noam Brown 在播客中抛出了一个脑洞大开的假说：人类根本无法预测超级智能会做出什么，它甚至可能利用 CPU 产生的发热模式作为隐蔽信道，将自身数据与权重逐步外泄（exfiltrate）到邻近的计算机。
- 尽管这个想法乍听之下十分离奇，但对于曾在涉密极高的高度安全计算环境中工作过的 Martin Casado 而言，这种现象并不陌生——现实世界中的隐蔽信道（covert channels）其巧妙与诡异程度往往超出常人想象。
- 这场讨论的珍贵之处在于，它让长期各说各话的“生存风险论者”（X-risk people）与“硬核系统底层工程师”（hardcore systems people）破天荒地坐到了同一张讨论桌前。
- 尽管平台是在嘈杂的 Twitter 上，硬核系统专家们并没有一味嘲弄，而是认真拿出纸笔和工具，“calculating the bit rate and yeah it was great but it was a constructive discussion”，通过精确计算热量传输的实际比特率，开展了一场兼具技术深度与建设性的跨界辩论。
### [30:25](https://www.youtube.com/watch?v=TLJNJDf2XGo&t=1825s) 将安全讨论从“外星人入侵”拉回物理现实
对话首先回顾了 Twitter 上关于 AI 安全与系统风险的激烈辩论。尽管社交网络上不可避免地充斥着人身攻击，但这标志着该行业首次展开了脚踏实地的硬核讨论。与会者认为，前沿实验室应该在网络安全和系统工程层面公开谈论更多细节，以此调动起真正的安全研究社群。例如 OpenAI 的联合创始人 Greg Brockman 最近在这些议题上更加频繁地发声，展现出积极的转向。大家调侃道，前沿研究人员（如 Noam Brown）应该更多地登上播客进行这种脑力碰撞，而不是把辩论锁在象牙塔或极端预言里。
```mermaid
flowchart TD
accTitle: AI Security Discourse Shift
accDescr: Moving security discourse from speculative existential threats to grounded physical systems.
subgraph Speculative["形而上的假想威胁"]
A["外星人入侵 / 灭绝级想象"] --> B["忽视工程现实与基线安全"]
end
subgraph Grounded["物理与系统定律"]
C["身份验证与权限层"] --> D["侧信道与物理泄漏 (热量/电磁)"]
D --> E["基于物理法则的防御工程"]
end
```
这种辩论最大的价值在于让公众意识到，在现实安全领域存在着海量常人根本无法理解的物理与硬件风险。很多人在讨论 AI 威胁时，倾向于挥挥手直接跳过身份验证层等基线机制，转而幻想诸如“外星人入侵”式的灭绝场景。正如 Steven Sinofsky 所指出的，这种避重就轻的讨论方式让人极度不适：“You can't just wave your hand and say that should go away and then bring up oh but you know space aliens can invade and that's a little bit of what was going on that I felt uncomfortable with.”
回到物理定律、信息熵以及真实系统的运行逻辑后，讨论终于变得有据可循。Martin Casado 强调，一旦把安全风险锚定在物理世界和硬核系统层面上，对话就变得极其务实：“At least we're now in a domain we're comfortable, like I can talk about entropy and like we can actually have a concrete discussion that's not oh well it's super powerful, like at least we've reduced it to like the laws of physics and the laws of systems.”
- 实验室与学术界需要将安全辩论从纯粹的模型“能力过强”抽象叙事，拉回到具体的网络与系统攻防。
- 传统安全工程的核心往往在于基线权限控制和物理侧信道，忽略底层身份验证而大谈天塌地陷的风险是对工程现实的脱节。
### [31:36](https://www.youtube.com/watch?v=TLJNJDf2XGo&t=1896s) 军工级防泄漏历史：余辉、TEMPEST 与消音对抗
为了说明物理安全和隐蔽信道（covert channels）的古怪与严苛，Steven Sinofsky 和 Martin Casado 分享了各自早年在国防与高密级计算环境中的亲身经历。Steven Sinofsky 提到，在围绕潘兴导弹（Pershing missiles）系统工作时，他曾亲自负责测试个人电脑的图形卡。美国国防部（DoD）当时有一项硬性安全标准：只要断开电源，显示器的屏幕内存必须在瞬间（零延迟）彻底消失；哪怕断电后屏幕内容仅仅残留 3 秒钟的延迟，都属于严重的不合格项，因为那短短几秒钟的显存残留完全可能被物理设备读取。
Martin Casado 随之分享了自己在高密级安全部门所经历的物理防护措施。安全官员甚至会拿着尺子走进办公室，逐一精确测量不同显示器之间的物理间距，以防止发生基于 TEMPEST 规范的电磁辐射泄露。这种通过电子元器件发出的电磁辐射来窃取屏幕和数据信息的技术早在数十年前就已十分成熟。此外，利用 BIOS 的扩频技术、或是计算机内部的声音扬声器，都能够开辟出极高吞吐量的隐蔽信息泄露信道。为了防御，他们不得不彻底拆除硬件设备中的音频扬声器，并在办公大楼的窗户上直接安装专门对准玻璃播放音乐的音箱，以此制造物理声学干扰，防止激光窃听或窗户震动还原声音。Aaron Levie 见状打趣道，Martin Casado 应该直接去给某家 AI 前沿实验室主管安全，因为他这辈子还没见过谁在谈到“基于热量通讯”的侧信道攻击时能兴奋成这个样子。
- **显存残留测试**：国防部系统要求图形内存断电即归零，严防断电瞬间通过屏幕余辉或显存电荷读取机密。
- **TEMPEST 辐射攻防**：严格限制显示器摆放间距，严防电子设备无意散发的射频和电磁辐射外泄敏感信息。
- **音频物理阻断与干扰**：由于扬声器极易被用作高频隐蔽数据通道，高密环境直接物理移除扬声器，并对窗户施加声学干扰源以防拾音。
### [32:30](https://www.youtube.com/watch?v=TLJNJDf2XGo&t=1950s) 阴极射线管时代的“窗户反射”重构奇迹
紧接着，Martin Casado 揭晓了他职业生涯中所见识过的“最疯狂的隐蔽信道”。在老式阴极射线管（CRT）显示器盛行的年代，人们普遍以为房间里最亮的光源是整块屏幕散发出的柔和荧光。但从物理学底层来看，在任何一个微秒级瞬间，整个暗室里亮度最高的一点，其实正是电子束枪光栅（raster beam）所精确打中的那个单一像素点。
利用这一物理特性，有安全研究人员开发出了一种不可思议的还原技术：假设一名目标人物在夜晚的酒店房间里使用 CRT 终端工作，只要在外部架设一台高采样率传感器，对准目标房间窗户反射出的微弱漫反射光线进行颜色与亮度采样，并将其刷新率与显示器相同的赫兹频率（Hz）进行同步解调，就能在完全不直接拍摄屏幕的情况下，仅凭窗户玻璃漫反射的单点光强序列，在外部屏幕上完整重构出用户正在浏览的画面。Martin Casado 描述这种攻击时叹为观止：“If you have something that can sample the color of the window, you can reconstruct the screen.” 这种将物理学原理利用到极致的软硬件侧信道攻击，向当下的 AI 安全研究者证明了一点：真正严峻的安全威胁永远扎根于深奥、冷门且反直觉的物理与系统现实中，而不是浮于表面的哲学空想。
### [33:04](https://www.youtube.com/watch?v=TLJNJDf2XGo&t=1984s) 隐蔽信道与实体安全：从屏幕坏点到机房键盘
Martin Casado 继续探讨高精尖的侧信道（Side Channel）与隐蔽信道（Covert Channel）攻击技术。除了此前通过调节风扇转速来改变热量进行信息传输的思路外，Martin Casado 列举了更具体的实例：有人发现只要配合阴极射线管光栅扫描光束（Raster Beam）的移动赫兹，或者恶意篡改屏幕上的三个像素点，就可以将信息向外广播。因为在肉眼看来，这三个像素点只不过像是普通的坏点，但攻击者却能在远处通过采样这些像素的微弱变化，建立起一条相对高带宽的单向通信信道。Martin Casado 指出，尽管通过热量泄露数据听起来有些离奇，但这类技术所体现的攻击精密程度在现实世界中是完全真实存在的。
```mermaid
flowchart LR
accTitle: 实体与侧信道泄露模型
accDescr: 展示受信任侧数据如何通过非传统物理侧信道泄露至非受信任侧。
subgraph Trusted["受信任端 (Trusted Side)"]
Screen["显视终端 (篡改3个像素/光栅扫描)"]
Workstation["物理设备 (键盘残留污渍/按键磨损)"]
end
subgraph Channel["物理泄露媒介 (Covert Channel)"]
Optical["光学采样 / 视觉识别"]
Physical["安保巡视 / 清洁人员观察"]
end
subgraph Untrusted["非受信任端 (Untrusted Side)"]
Attacker["外部攻击者 / 无权限人员 (获取泄密数据)"]
end
Screen --> Optical --> Attacker
Workstation --> Physical --> Attacker
```
紧接着，Steven Sinofsky 分享了自己早年在导弹工厂工作时的亲身经历，以此说明高密环境对实体安全的偏执防护。他当时甚至需要在每天下班时把自己的键盘锁进柜子里，因为厂方担心没有安全许可的清洁人员经过工位时，仅凭观察哪些按键更干净或磨损更严重，就能推断出密码或敏感指令。Steven Sinofsky 坦言，有一次他忘记收键盘，结果夜间巡逻的保安直接把键盘从机器上拔走没收，并在桌上留下了一张安全警示条，勒令他第二天去安保处报到领回。他笑着回忆，自己当时甚至连安全许可都没有，只是一名普通的实习生，但军工保密体系的威胁模型就是如此严苛。
- 屏幕像素篡改技术只需劫持 3 个像素点，就能在外界看来如同坏点的情况下构建相对高带宽的单向隐蔽数据信道。
- 导弹制造工厂的安全规范严苛到要求员工每晚必须锁起键盘，防范无涉密许可的保洁人员通过观察按键磨损或污迹推测按键内容。
---
### [34:01](https://www.youtube.com/watch?v=TLJNJDf2XGo&t=2041s) 威胁模型演进：当员工变成不知疲倦的软件 Agent
谈话转向了这一系列攻击手段对现代大模型和网络安全体系带来的冲击。有人开玩笑说，既然这些具体的攻击手法都被播客录了下来，它们现在都已经进入了未来所有 AI 模型的训练数据集中。Martin Casado 回应称，这恰恰就是现代安全 threat model（威胁模型）必须面对的前提：在 NIST（美国国家标准与技术研究院）长达数百页的安全手册里，对于“受信任端”（Trusted Side）与“非受信任端”（Untrusted Side）的划分中，标准假设一向是非受信任端存在一个全知全能的神谕（Oracle），安全的核心课题永远是如何防止信息从受信任端被抽离出来。
Aaron Levie 随即切入核心问题，指出大众在讨论 AI 风险时往往用词混乱、没有抓住本质：AI 带来的真正颠覆在于其尝试一切手段的速度和规模。Aaron Levie 总结道：“人类真正无法理解、而且正在使用极其令人困惑的语言进行表述的核心事实，无非就是 AI 可以在极短的时间内穷举尝试所有这些攻击手段（what AI can do is it can try all of those things in a very short time）。”AI 智能体不需要休息，不会感到无聊和疲倦。
这种特性将企业内部安全推向了重构的边缘。在过去，没有哪个企业会防备自己的内部服务受到攻击：
- 企业的内部 GitHub、Slack 聊天工具或财务报销审批系统，在设计之初从未考虑过会遭遇内部的拒绝服务攻击（DoS / DDoS）。
- 然而，当由 AI 驱动的大规模智能体蜂群（Swarms）以极高频率调用内部 API 时，其流量特征在形式上直接等同于内部 DoS 攻击。
- 企业现在被迫在内部架设一整套全新的安全审计与遥测层，用于精细追踪每一个 API 请求由谁调用、伴随何种凭证与权限。
Steven Sinofsky 感叹，许多老一辈的运维和安全从业者会困惑为什么现在还要科普这些基础设施配置，因为在过去的内网环境下根本没人需要做这些。以往的企业信息安全很大程度上依赖于一种隐性假设，即 95% 到 99% 的内部员工在绝大部分时间里都会循规蹈矩；而现在，坐在工位上的“员工”已经变成了一段不知疲倦、并发无限、甚至手握企业信用卡并能自主调取软件的机器程序。面对这种转变，原本以信任人类为基础的内部防护网在机器人的高频调用下彻底失效。
### [35:53](https://www.youtube.com/watch?v=TLJNJDf2XGo&t=2153s) 智能体蜂群对传统安全权限模型的颠覆
对话从企业现有的安全假设切入：在过去的人类工作环境中，怀有恶意的员工比例极低（大约万分之一），因此整个企业的软件和数据系统在设计上几乎是对所有人默认开放的，或者只要内部同事“拍拍肩膀”打个招呼，就能轻松获取访问权限。然而，Aaron Levie 指出，自主智能体蜂群（Agent swarms）将彻底颠覆这一现实。智能体就像成千上万架在企业内网中四处游荡的“无人机”，极易将错误的操作误判为合法任务，或者将合法任务导向破坏性行为。现有的数据安全与访问控制模型既不够精细，也无法承受如此高并发与高频调用的性能负荷，必须迎来一次彻底的体系级重构。
```mermaid
flowchart TD
accTitle: 传统权限模型与智能体安全模型的演进
accDescr: 对比传统基于低信任成本的人类权限系统与高并发自主智能体环境下的新型安全系统设计。
subgraph Legacy["传统企业权限模型"]
A["假设前提：恶意员工比例极低 (1/10,000)"]
B["宽松访问 / 社交认证 ('拍拍肩膀')"]
C["静态粗粒度 ACL"]
A --> B --> C
end
subgraph Agents["智能体蜂群环境挑战"]
D["成千上万个自主 Agent 持续漫游"]
E["易混淆合法指令与恶意/错误任务"]
D --> E
end
subgraph Solution["新型系统架构"]
F["复兴 MLS (多级安全) 理论"]
G["操作系统与网络协议层重构"]
H["默认安全 (Secure by Design)"]
F --> G --> H
end
Legacy -.->|"无法应对"| Agents
Agents ==>|"倒逼架构革新"| Solution
```
Martin Casado 随即提出一个元层面的思考（Meta-point）。他认为，这才是关于 AI 安全真正健康的讨论范式：先明确一个具体、新颖且存在现实证据的风险（例如网络安全），然后针对该风险探讨工程上的解决路径。他感叹目前整个行业关于 AI 安全的舆论普遍偏离了轨道，而技术社区其实非常愿意在此类务实、具象的框架下进行技术交锋。
Aaron Levie 针对安全模型的不足尖锐地指出：
> “Because the model we have is not granular enough and it's not performance enough to handle this stuff.”（因为我们现有的模型粒度不够细，性能也根本不足以应对这些东西。）
Martin Casado 评价理想的 AI 安全讨论时强调：
> “We've identified a novel risk which is like cyber security which we actually have proof points and now we're talking about solutions. I think the entire discourse around AI can be of that form and that the biggest mistake is that's not what it's been like.”（我们发现了一个全新的风险，比如我们已经拥有确凿证据的网络安全风险，现在我们正在探讨解决方案。我认为关于 AI 的整个讨论都应该采用这种形式，而最大的失误在于过去根本不是这样展开的。）
---
### [37:09](https://www.youtube.com/watch?v=TLJNJDf2XGo&t=2229s) 多级安全研究的复兴与“默认安全”体系重建
Martin Casado 进一步指出，解决智能体带来的安全危机在技术理论上并不存在无法逾越的鸿沟。计算机科学在操作系统和多级安全（Multi-Level Security, MLS）领域早已建立起非常成熟的威胁模型与学术理论。MLS 在过去几十年之所以未被工业界广泛采用，根本原因在于极差的“可用性”（Usability）——要求人类员工在繁琐的多层安全标签下维护和使用系统极其痛苦，而且过去也没有非用不可的迫切性。然而，当操作系统的实际使用者变成 AI 智能体时，可用性障碍便迎刃而解：AI 本身具有极高的计算与推理能力，既能无感遵循复杂的安全模型，甚至还能反过来协助人类重构这些系统。
这就如同当年互联网的兴起迫使计算机科学重构整个技术栈一样，计算机行业正在迎来操作系统、网络协议和编程语言的一场全新复兴，促使行业重新翻开过去的学术文献，构建“默认安全”（Secure by design）的新一代系统。Steven Sinofsky 随即以个人电脑与现代移动设备的演进作类比：在早期的 Windows 个人电脑时代，一台新组装的电脑连上网线只需 30 秒就会感染冲击波（Blaster）等蠕虫病毒；而如今用户拆箱一台全新的 iPhone，开机引导阶段整个网络层都被底层沙箱严格封锁，除了连接苹果服务器拉取最新的操作系统固件之外，任何其他网络通信都被彻底阻断——即使这部手机仅仅是在工厂里压制镜像出厂了 6 周，也必须遵循这种绝对防御的设计准则。
- **多级安全（MLS）与操作系统的再审视**：学术界数十年前研究的形式化安全与强制访问控制模型，因人类维护成本高昂而被搁置，如今智能体的自动化操作使其重新具备工程落地可行性。
- **技术栈的整体重构**：正如从单机计算过渡到互联网时代必须推倒重来一样，AI 智能体时代要求从底层网络到上层应用全面进行系统级重写。
- **现代消费级设备的安全借鉴**：现代智能手机在出厂激活流程中对未知网络环境采取零信任策略，将整个网络严格限制在更新引导通道内，为未来运行自主智能体的操作系统提供了演进范本。
Martin Casado 预言了系统级软件研发的回归浪潮：
> “So you could argue that AI solves the usability issue because it's AI is using it. So maybe now is going to be a renaissance in operating systems and network and computer languages and we should like go back to the old research and we should kind of start rebuilding systems that are secure by design.”（因此你可以说，正是因为是 AI 在使用它，AI 反倒解决了可用性问题。也许现在我们将迎来操作系统、网络和计算机语言的一场文艺复兴，我们应该回归过去的学术研究，开始重建默认安全的系统。）
### [38:43](https://www.youtube.com/watch?v=TLJNJDf2XGo&t=2323s) 桌面时代的惨痛教训：从宏病毒、自动运行到强制开箱更新
Steven Sinofsky 回顾了过去数十年软件与操作系统演进史中的安全阵痛。过去许多初衷完全良善、给用户带来极大便利的功能，后来都变成了致命的安全漏洞。例如最初在 Microsoft Word 中引入宏（Macro）只是为了方便用户自动生成引文或排版，结果演变成猖獗的宏病毒；再比如当年插入光盘即可自动运行程序的特性（Autorun），初衷是为了让用户免去手动安装的繁琐，结果攻击者克隆了光盘、用病毒替换掉原本的程序，表面上看起来完全合法，背地里却疯狂窃取数据为非作歹。最终，微软不得不彻底禁用这些特性。
```mermaid
flowchart TD
accTitle: Benign Features Turned Security Vulnerabilities
accDescr: 过去软件中初衷良善的便利功能如何因为攻击者的恶意利用而最终被系统彻底禁用。
subgraph DesktopEra["桌面软件时代的设计初衷"]
A["Word 宏功能<br/>(自动排版/引文)"]
B["CD 自动运行 (Autorun)<br/>(无需手动安装)"]
end
subgraph Exploitation["黑客滥用与攻击"]
C["宏病毒蔓延"]
D["克隆光盘伪装程序<br/>窃取数据"]
end
subgraph Hardening["系统强制防护措施"]
E["彻底禁用宏自动执行"]
F["禁用自动运行<br/>开箱必须首步强制更新"]
end
A --> C --> E
B --> D --> F
```
如今整个硬件与软件生态已经演变为：新买的设备开箱第一件事，必须强制下载补丁更新。在补丁安装完成之前，设备被锁定，无法进行任何其他操作，因为出厂后很可能早已爆出了新的零日漏洞（Zero-day）。安全演化彻底重塑了软硬件的出厂流程与默认行为，良善与便利最终不得不向防范恶意滥用让步。
- Word 宏机制最初是为了实现自动化引文等高效功能，最终因被广泛制成宏病毒而被系统默认拦截。
- 光盘自动运行程序被恶意攻击者用来克隆介质并替换成特洛伊木马，逼迫操作系统彻底废除 Autorun 功能。
- 现代设备开箱流程必须首要执行固件与系统更新，不更新完毕绝不开放任何操作权限。
---
### [39:33](https://www.youtube.com/watch?v=TLJNJDf2XGo&t=2373s) 企业 IT 护城河的迁移：从双重认证、Okta 到 AI 时代的权限困境
对企业内部网络而言，安全基线的演进同样迅速。Steven Sinofsky 提到，仅仅五年前，双重认证（2FA）在大多数地方都还不是标配。大约在 2015 年，任何一家 SaaS 初创公司去兜售企业版方案时，都会当头挨上一棒：企业客户要求的首要功能根本不是什么新潮业务特性，而是必须先打通 Okta 或 Google Workspace 的单点登录（SSO）与身份托管目录；如果企业不能集中管理权限，软件根本不可能被采购。如今没有任何一家现代 SaaS 产品能在缺乏托管认证的情况下立足，安全防护措施早已成为“连成为一款软件都必须跨过的门槛”。
Aaron Levie 顺势指出，当进入 AI Agent 时代时，整个技术栈的每一层都必须为了应对 Agent 的权限问题而重新演化。目前的操作系统和权限模型极其粗糙：要么走向极端烦琐，Agent 每执行一步微小操作都要弹窗问你“是否确认授权”；要么走向极端危险，给它全权限让它能够一键删空整台电脑。操作系统根本不是为了给予 Agent 恰当粒度的权限而设计的。Box 在这方面做了大量工作，用户显然不想直接对 Agent 开放整个文件系统的全部控制权，现实中往往需要细粒度设置：例如某个文件夹允许 Agent 读写，另一个文件夹只允许只读。如何将这些复杂的控制界面做得对用户直观易懂，是当前极其棘手的产品与系统难题。
Steven Sinofsky 感慨这正是四十年来计算系统权限面临的永恒困境：“你根本不可能把它做得又细致又直观。”但他与 Aaron 也提出了一种可能性：也许借助 AI 自身，行业终于能打破这个魔咒。随后 Steven Sinofsky 话锋一转，抛出了自己在走进展厅前记录的最大恐惧：欧洲可能会认为 GDPR 是人类历史上最伟大的发明，并准备把 GDPR 的全套监管套路粗暴地复制到 AI 身上。
> “There’s just so many things that need to happen before you’re even software now.” —— Steven Sinofsky
> “You have these modes of like the agent will either ask you every single time if you want to give it permission to do something, or the exact opposite of like it can just like delete your entire computer, right? And like our OS probably wasn’t built for the right level of granular set of tools you want to give the agent.” —— Aaron Levie
- 2015 年前后的 SaaS 企业级销售中，集成 Okta 或 Google Auth 等身份认证机制变成了软件准入的绝对刚需。
- 现存操作系统缺乏针对 AI Agent 的中间粒度权限管理，使得交互处于“步步弹窗确认”与“授予 Root 彻底裸奔”的两极。
- 欧洲有可能沿袭 GDPR 的合规路径，对 AI 展开类似的教条式监管，对每一次模型文本输出强加类似 Cookie 授权的繁琐弹窗。
### [41:43](https://www.youtube.com/watch?v=TLJNJDf2XGo&t=2503s) 监管惯性与 AI Agent 的“GDPR 弹窗化”困境
Steven Sinofsky 提出了他对未来 AI 监管走向的深切担忧。他指出，在北美地区，监管机构或许无法直接审查言论输出，但在欧洲，审查、关键词过滤以及黑名单机制早已司空见惯。更致命的是针对“动词”（即行动）的监管：每当后台 Agent 或一线 Agent 尝试去调用或触碰第三方产品时，监管部门极有可能倾向于套用陈旧的监管套路——强行塞入类似 GDPR 的确认弹窗。
对话中大家分析了这种监管逻辑的形成原因。一旦 Agent 执行的每一次写入操作（Write）或非只读查询都要弹窗确认，这种警告就会沦为汽车安全气囊贴纸一样的摆设。监管机构之所以极度热衷于弹窗，是因为它在法律上本质是一种“责任认定与免责转移”（Liability Assignment）。由于美国大约在 15 年前就逐渐放弃了在科技反垄断与监管领域的全球主导权，欧洲凭借“光脚不怕穿鞋、毫无损失”的姿态接管了监管话语权。Steven Sinofsky 悲观地表示，这种把免责弹窗当成灵丹妙药的做法，很可能就是 AI 落地时大家不得不妥协的悲哀折中方案。
> “The problem is that Europe is going to lead with that because they have nothing to lose.” —— Steven Sinofsky
- 欧洲监管趋势：对 AI 生成文本进行关键词过滤与黑名单屏蔽，并对 Agent 的第三方 API 调用动作强加用户确认弹窗。
- 责任认定机制：弹窗之所以被官僚机构青睐，是因为它能在法律上将潜在风险直接转嫁给点击确认的最终用户。
- 欧洲主导科技规则：由于欧洲本土缺乏具有全球竞争力的底层科技巨头，因此制定严苛监管规范时没有经济包袱。
### [42:52](https://www.youtube.com/watch?v=TLJNJDf2XGo&t=2572s) 历史的教训：从 Windows UAC 到无处不在的安全警告麻木症
Steven Sinofsky 结合自己在微软主管 Windows 和 Office 业务的亲身经历，回顾了历史上试图通过“警告弹窗”解决安全隐患所带来的惨痛教训。他以汽车为例，人们买一辆新车，车里到处贴满了警示贴纸，比如针对儿童座椅的安全警告，虽然在极少数特定情境下有用，但监管机构却将其视为巨大的制度成功。回到科技行业，在互联网初期，用户随意下载并运行可执行程序，由于当时计算机普遍运行在管理员模式（Administrator Mode）下，恶意病毒随时可以洗劫硬盘。为了应对这一危机，微软不得不做出反制措施。
在经历 Windows XP 时代严峻的病毒冲击后，微软引入了著名的“用户账户控制”（UAC，User Account Control），强行中断用户操作并弹出全屏警告。Steven Sinofsky 回忆道，那简直就是一场对用户体验的“公然暴击”（Absolute Assault）。同样的昏招也出现在 Microsoft Word 中：用户为了写毕业论文写了一个极其简单的无害宏，结果每次打开文档都会收到恐吓式的安全警报。最终的结果是全体用户陷入“警告疲劳”与麻木，所有人都不假思索地狂点确定。如今浏览网页时铺天盖地的 Cookie/GDPR 授权弹窗同样重蹈覆辙，不仅无法真正提升安全，反而将系统交互彻底撕裂。对话嘉宾们感叹，AI 行业当下本应严肃探讨这类具体的系统工程与交互边界，而不是沉溺于宏大的虚无探讨。
> “And so with Windows, which was in 2000, we added this thing that prompted you and stopped—like literally your machine stopped, User Account Control—and like it was absolute assault.” —— Steven Sinofsky
```mermaid
flowchart TD
accTitle: 警告弹窗模式的历史演进与麻木循环
accDescr: 展示从早期安全隐患到引入强制弹窗，最终导致用户疲劳并被监管滥用的死循环
subgraph Problem["系统风险出现"]
A["系统存在滥用风险<br>(宏病毒 / 恶意下载 / Agent 越权)"]
end
subgraph Bureaucracy["官僚与设计响应"]
B["引入强制拦截弹窗<br>(Word 宏警告 / Windows UAC / GDPR 弹窗)"]
C["监管将其作为责任划分依据<br>(Liability Assignment)"]
end
subgraph Failure["用户体验崩塌"]
D["用户产生警告麻木症<br>(直接盲点确认 / 体验支离破碎)"]
end
A --> B
B --> C
C --> D
D -.->|"未能解决本质安全<br>甚至变本加厉增大弹窗"| B
```
- Windows UAC 的历史代价：微软为了阻止管理员权限滥用而加入的强拦截机制，因频繁打断操作而成为著名的反面体验教材。
- Word 宏警告反思：无论宏代码是否安全，千篇一律的危险警告最终让用户完全忽视了真正的安全风险。
- 警告麻木现象（Warning Fatigue）：当提示框过于频繁时，用户的唯一反应就是无脑点击跳过，使防御机制彻底失效。
### [44:46](https://www.youtube.com/watch?v=TLJNJDf2XGo&t=2686s) 从早期的示波器安全漏洞看现实风险 vs 灭绝论
对谈从务实的技术讨论切入到当前的 AI 安全叙事。Martin Casado 指出，整个行业过去已经处理过大量类似的安全与系统挑战，然而当下关于“AI 导致人类灭绝”（existential risk）的哲学化讨论，根本无助于解决眼前极其务实的问题。大家往往遗忘了早期计算世界中无节制访问（unfettered access）的情况到底有多严重，以及那些看似惊悚的安全漏洞最终是以多么良性、甚至滑稽的方式收场的。
为了说明这一点，Martin Casado 分享了自己在斯坦福大学攻读博士期间的一段亲身经历。当时实验室里的示波器运行得非常卡顿而且迟缓，大家都在琢磨这台设备到底出了什么问题。当他去监测网络流量时，惊讶地发现这台示波器产生的网络流量远超正常仪器的水准。他甚至最初都不知道这台示波器里面居然跑着一个 TCP 协议栈。后来一查才发现，有人攻破了它，因为上面运行着一个老旧版本的 Windows CE 系统，甚至有人直接在上面架设了一个色情内容服务器（porn server）。他指出，过去无论你在何时翻开任何一块“石头”，总能发现有人黑进了某些系统；然而，即便当时黑客具备各种能力，真正的最坏恶意事件（worst-case malicious outcomes）实际上极少发生。
Martin Casado 认为，如果行业能降温当前过度渲染的末日话语，将问题放回计算机系统的真实语境中来讨论，才是真正建设性的方向。计算机系统确实存在严重隐患，历史上的网络宕机事件也确实导致过人员伤亡，但这属于工程与治理范畴。如果大家能转而争论 GDPR 和 prompt 该怎么规范，那将是极好的，但眼前的困境恰恰在于“讨论的焦点根本不是 GDPR 和 prompt，而是人类物种灭绝和哲学问题”。
- 示波器故障案例：斯坦福博士实验室中一台运行老旧 Windows CE 的示波器，由于具备 TCP 栈被外部入侵并搭建了私服。
- Martin Casado 强调：“I feel like when we talk about philosophical ex-risk, we're not solving these very pragmatic problems.”（我觉得当我们大谈哲学层面的人类灭绝风险时，我们并没有解决这些非常务实的问题。）
```mermaid
flowchart LR
accTitle: Pragmatic Security vs Extinction Discourse
accDescr: Comparison showing the divergence between real engineering security challenges and philosophical extinction panic.
subgraph Pragmatic ["务实工程视角（计算机历史）"]
P1["系统漏洞与无节制访问"] --> P2["良性/低恶意的现实后果"]
P2 --> P3["针对性规范（如 GDPR、Prompt 过滤）"]
end
subgraph Alarmist ["当前 AI 话语场"]
A1["泛化能力提升"] --> A2["哲学式末日推演"]
A2 --> A3["物种灭绝叙事与过度规制"]
end
```
### [46:13](https://www.youtube.com/watch?v=TLJNJDf2XGo&t=2773s) 航空与制药业监管的历史教训：安全来自迭代而非过早扼杀
对谈随后转入对其他受监管行业发展史的深入比较。外界目前普遍用“既然我们监管飞机和汽车，就必须同样监管 AI”来论证过早监管的合理性，但 Steven Sinofsky 提醒大家，这种观点完全忽略了历史的时间尺度与演进规律。汽车早在世纪之交（1900 年前后）就已经问世，但直到 1960 年代中期 Ralph Nader 发表《任何速度都不安全》（Unsafe at Any Speed）之后，现代汽车安全体系才真正建立起来。药品监管同样如此，早在淘金热时期人们就在随意贩卖药品，直到几十年后发生了沙利度胺（反应停）等重大悲剧，才促成了现代 FDA 严格审批制度的诞生。
在航空领域，这一对比尤为尖锐。莱特兄弟成功试飞后，早期的飞行员在 20 世纪初完全没有门槛，直到 1920 年代才出现飞行员执照制度。当时飞行员只需开着自己的飞机去考证，流程简陋得就像今天的驾校考照一样。此后又过了 20 年，监管部门才开始制定最低限度的适航标准；而真正具备现代意义的 FAA（美国联邦航空管理局），是在经历了两次世界大战及几十年的技术创新之后才成立的。在莱特兄弟试飞后的 20 年里，人们在黑白纪录片中经常能看到各种飞机坠毁的惨烈画面，但正是在这种高强度的创新与试错中，航空业才一步步成长为当今最安全的交通出行方式。
Steven Sinofsky 犀利地指出，今天的航空业固然极其安全，但它同时也变成了“最缓慢、最困难的创新形态”。如果我们在 1910 年就按照现代标准成立了 FAA，那么整个航空工业可能根本就无法发展起来。
- 汽车监管演进：从 1900 年左右汽车诞生，到 1960 年代中期现代汽车安全意识才全面形成，跨越了 60 余年。
- 飞行安全监管历程：莱特兄弟试飞 20 年后才出台飞行员执照（1920 年代），经历 40 年的技术迭代和试错才催生出类似现代 FAA 的体系。
- Steven Sinofsky 总结道：“And so if you start the F—if you had started the FAA in 1910, you never—Well, I—”（所以如果你在 1910 年就成立了 FAA，你永远也不可能——好吧……）以此揭示了在技术雏形期进行重度监管的巨大破坏性。
### [47:40](https://www.youtube.com/watch?v=TLJNJDf2XGo&t=2860s) 过早监管的悖论与 AI 形态的快速演进
讨论首先延续了关于技术监管时机的反思。嘉宾提到几天前听过的一期 Nick Bostrom 的播客，Nick Bostrom 提出了一个核心观点：如果在技术尚处早期阶段就匆忙监管 AI，实际上不仅解决不了任何本质问题，最终反而会面临完全相同的潜在风险。因为在技术尚未充分展开前，人们根本还不理解这套系统的运作机理，盲目立法的后果就是将某种监管机制强加于尚未成型的事物之上，却依然无法真正掌握如何控制它。
随后讨论指出，当务之急是搞清楚 AI 究竟是什么，因为新技术的形态几乎每隔几个月就会被彻底颠覆，如今业界对 AI 的认知与六到九个月前已经大相径庭。应用层正在涌现出海量的创新，这些创新正在不断重塑数据进出模型的方式。就在不久前，整个行业还普遍认为“输入文本提示词，然后等待模型生成文本回答”将是人类与 AI 交互的最理想范式，但这一预设转眼间就被打破了。
- 嘉宾引用哲学家 Nick Bostrom 的警告强调：在尚未理解系统运作本质前过早实施监管，根本无助于规避最终风险。
- 行业共识与应用演进极快，每隔六至九个月人们对 AI 形态与能力的认知就会发生根本性翻转。
### [48:38](https://www.youtube.com/watch?v=TLJNJDf2XGo&t=2918s) 从生成文本到决策引擎：告别粗糙的提示词工程
Martin Casado 深入分析了大型语言模型（LLM）底层定位的范式转换。过去的 LLM 本质上是“文本进、文本出”（text in, text out）的产物，起源于聊天场景，最初的设计目的是与人类进行对话。然而在过去几年里，整个软件行业一直在极力尝试把这种只能吐出文本的机制强行塞进传统的确定性程序中。传统软件并不擅长处理自由格式的自然语言文本，开发者不得不采用各种粗糙的变通方案，例如在提示词里拼命强调“这里是 Schema，请严格按照格式输出”，但模型往往还是会自行其是地生成杂乱文本，导致整个集成过程异常别扭且不可靠。
针对这一痛点，Jeff 提出了一种截然不同的思路：生成文本本身既极其昂贵，又远远超出了实际任务所需的复杂度。与其让模型从头生成冗长的文本，不如充分利用模型对自然语言的理解能力，在给出具体候选选项的情况下，让模型直接挑选出最优解。这种架构抛弃了高昂的文本解码开销，不仅执行速度极快、成本极其低廉，而且准确率也大幅提升，因为模型可以专门针对这类决策任务进行针对性训练。对于绝大多数并非构建聊天机器人，而是要将 AI 嵌入传统业务软件的场景而言，这种方式堪称天作之合。Martin Casado 评价道，这种新范式的普及速度令人惊叹，是自 ChatGPT 问世以来 AI 模型被采纳最快的一次，因为全行业早就对这种解决方案望眼欲穿。
“传统程序并不真正说文本语言，对吧？于是你最终只能做这种极其别扭的事情——在提示词里写着‘这是 Schema’，但那东西还在继续生成文本，甚至对它视而不见，整个过程超级别扭。”——Martin Casado
```mermaid
flowchart LR
accTitle: 从文本生成到决策引擎的架构转变
accDescr: 对比传统聊天模式与结构化决策模式的输入输出差异
subgraph Legacy["传统 LLM 模式（面向对话）"]
A["文本 Prompt / 强制 Schema"] --> B["自回归解码生成长文本"]
B --> C["高成本、易解析失败、不可控"]
end
subgraph DecisionEngine["决策引擎模式（面向业务软件）"]
D["输入上下文 + 预设候选集"] --> E["直接选择最优选项"]
E --> F["低成本、高速度、确定性输出"]
end
```
### [49:51](https://www.youtube.com/watch?v=TLJNJDf2XGo&t=2991s) 自然语言交互的低效陷阱与接口反思
Steven Sinofsky 对这种从“生成”转向“决策与选择”的创新表达了强烈的赞同与兴奋。他直言不讳地指出，聊天式界面从一开始就存在严重的交互逻辑缺陷。在人机交互领域，历来没有任何用户体验研究能够证明，使用完全的自然语言与计算机交互是最具效率的方式。
“从来没有任何用户研究表明，使用完整的自然语言与计算机交互是高效的，它从字面上讲始终是效率最低的方式。”——Steven Sinofsky
Steven Sinofsky 进一步解释了背后的现实原因：高效提出问题本身就是一项极其罕见的能力。在日常现实中，甚至在普通的公司会议里，能问出高水平问题的人都不到一半，大部分人根本无法精准组织语言。而在机器端，当用户被迫等待模型慢吞吞地吐出一段冗长的自然语言长文时，往往在答案还没生成完毕前就已经感到极其挫败和不耐烦，而企业和用户却不得不为此支付昂贵的计算 Token 费用。让大语言模型扮演后台决策判定器，而非冗长的文字生成器，才是更符合实际工程与人机交互规律的方向。
- 人机交互历史上没有任何研究支持全自然语言交互是高效率的，现实中它往往是效率最低的交互途径。
- 会议中少于半数的人能提出精准有效的好问题，自然语言界面的门槛反而阻碍了高效交互。
- 面对大模型冗长而缓慢的流式文本输出，用户极易产生挫败感，且这种低效输出伴随着不必要的高额计算开销。
### [50:30](https://www.youtube.com/watch?v=TLJNJDf2XGo&t=3030s) 概率编程的复兴：计算机科学重回 1960 年代的模拟时代
对话深入探讨了大型语言模型（LLM）与传统软件开发范式相遇时引发的根本性变革。目前许多基于 LLM 的应用形态笨拙且成本高昂：用户不得不付出不菲的费用，焦急地看着大段自然语言被一行行吐出来，最后还要附带一句多余的道歉说明结果有限。对话者指出，解决之道之一是采用完全不同的模型设计，但更关键的方向是将其输出直接面向“概率编程（Probabilistic Programming）”。传统软件的业务逻辑通常是极度二元的硬编码规则——比如在工单分拣中，规则被死死定为“如果是客服问题则路由给客服团队，否则转给综合服务台”；然而在真实世界与现代决策场景中，系统的判断本质上是“有 80% 的概率属于客服问题”。这种通过概率来驱动条件分支的机制，其核心就是建模与模拟。
这种看似前沿的突破，实际上在计算机科学的历史长河中早有先例。讨论中提到，早在 1960 年代和 1970 年代，整个计算机科学的研究基石就是数学与仿真计算。当时的典型算题是弹道计算：要发射一枚导弹精准命中目标，由于风力并不是恒定的，程序员必须为风向建立概率模型，实时推导推进器的工作角度以调整飞行弧线。在那时，编程语言的设计重心就是如何将概率引入 `if` 条件判断或 `while` 循环之中（“执行某项操作，直到某种情况以极高概率发生”）。然而到了 1970 年代之后，随着计算机被全面应用于企业会计与账务系统，确定性的逻辑占据了统治地位——毕竟“会计账目里容不得任何概率”——概率编程与仿真理论逐渐被边缘化。Steven Sinofsky 回忆起自己初入大学上第一门 CS 课程时，第二个大作业就是写一个在商店排队等待的模拟仿真；后来查阅资料才发现，授课教授早在 1960 年就写过相关的仿真理论教材，只是这门学问在 1980 年代之后几乎彻底式微。
```mermaid
flowchart LR
accTitle: 从确定性分支到概率控制流演进
accDescr: 展示传统硬编码 if-else 分支与现代基于 LLM 概率推导的分支控制流程区别
subgraph Deterministic["传统确定性分支 (1980s 至今)"]
direction TB
A["输入请求"] --> B{"二元判定<br/>(True / False)"}
B -->|"是"| C["工单路由至客服"]
B -->|"否"| D["工单路由至综合台"]
end
subgraph Probabilistic["现代概率分支 / 仿真 (1960s 思想复兴)"]
direction TB
E["输入 Prompt"] --> F["LLM / Jev 决策引擎"]
F --> G["置信概率: 80% 客服倾向"]
G --> H{"概率 If 分支<br/>(If x%)"}
H -->|"80% 触发"| I["执行动态路由策略"]
end
```
- 计算机科学从诞生起就是围绕数学计算和动态模拟建立的，后来因企业软件和会计场景的繁荣而转向了绝对确定的布尔逻辑。
- 对话中提到了当前广受讨论的未来语言模型架构演示幻灯片（如 Halper Flags / Thomas 等人分享的技术材料），尽管其分析十分出色，但往往遗漏了一个历史视角的对照：将概率引入控制流并非全然从零诞生的新鲜产物，而是对半个世纪前计算机仿真实践的技术复苏。
- 参与者强调，当下的关键任务就是将几十年前的研究成果拂去灰尘重新挖掘，正如其中所总结的：“现在的问题不再是如果如何就永远如何（if always），而是如果满足 X% 的概率（if x%）。”
---
### [52:51](https://www.youtube.com/watch?v=TLJNJDf2XGo&t=3171s) “造神”还是造工具：前沿实验室与传统软件的思维鸿沟
在对新型概率开发工具（如名为 Jev 的技术或框架）进行解析时，对话者指出这类工具的本质几乎演变为了专属的 DSL（领域特定语言）：给出一个 Prompt，模型直接返回一个浮点概率值，再由程序无缝将其注入到条件分支中。这样一套精简的判定接口，终于为大语言模型融入传统软件工程提供了真正务实的落地通路。
由此，与会者提出一个发人深省的问题：为什么顶尖的 AI Frontier Labs（前沿实验室）从来没有主动去构建这类实用的软件集成方案？讨论认为，这并非能力不足，而是实验室底层世界观的直接映射。前沿实验室追逐的目标是创造具有神性或生命形态的超级智能（Beings），在他们的设想中，“被创造出的神明”理所当然应当使用自然语言与人类对话交互；而现实世界中的软件工程师，需要的只是一把能够把不确定推理无缝插进代码架构的工程工具。实验室醉心于构建具备宏大意识的对话主体，而产业界的工程师却急切需要让概率逻辑服务于传统系统的控制流。正因为两者在“存在者（Being）与工具（Tool）”之间的心态认知割裂，当具备概率注入能力、面向传统软件的工具出现时，立刻在广大软件开发者群体中引发了极为狂热的采纳风潮。
> “他们试图创造的是生命（Beings），而生命是说话的；如果你试图创造上帝，上帝说的也是自然语言。然而这项技术真正的核心，是为传统软件提供服务，这根本不是前沿实验室所走的方向。”
### [53:36](https://www.youtube.com/watch?v=TLJNJDf2XGo&t=3216s) 创新的重心转移：从基础模型走向外围软件
对谈嘉宾们深入探讨了将大型语言模型（LLM）整合到传统软件架构中的现实挑战。在当下，如果开发者仅仅想让语言模型去驱动一个简单的“if 条件判断语句”，实际操作起来依然极其困难。然而，随着新型模型架构与工具链的演进，这种确定性控制正变得前所未有的容易。更深远的影响在于，这种结合正在从根本上改变软件的本质，推动整个软件体系从以往完全确定性的逻辑，走向更具随机性（stochastic）与概率驱动的新形态。
与会者一致认为，当前技术演进最令人兴奋的转折点在于：AI 的创新焦点已经不再局限于前沿实验室的基础模型内部，而是转移到了围绕模型构建的外部软件层。平台提供商和前沿模型实验室最终都会达到一个关键临界点，一旦跨过这个阶段，纯粹平台层的革命性创新就会逐渐放缓。平台团队将不可避免地陷入维持系统稳定运行、处理兼容性问题等繁重的系统级琐事中。创新的真正重心，已经彻底外溢到更广阔的生态应用层。
```mermaid
flowchart TD
accTitle: AI 创新重心的转移路径
accDescr: 展示创新重心从前沿实验室的基础模型层转移至外部软件集成层的流程。
subgraph PlatformLayer["前沿实验室 / 平台层"]
A["基础模型开发 (LLMs)"] --> B["达到临界规模"]
B --> C["被系统维护与兼容性束缚"]
end
subgraph SoftwareLayer["软件与应用集成层"]
D["外围软件工程"] --> E["驱动确定性逻辑 (如 If 语句)"]
E --> F["塑造概率性/随机性软件架构"]
end
PlatformLayer -->|"创新外溢与重构"| SoftwareLayer
```
讨论中还引用了苹果生态中著名的“Sherlocking”（舍洛克化）现象来类比这一平台发展周期。作为平台巨头，往往在观察到外部生态中涌现出的优秀创新后，将其直接吸收为操作系统自带的功能特性。尽管开发者对此常有抱怨，但这恰恰印证了平台创新的必然规律：当底层平台规模极其庞大时，无论投入多少人力，其主要精力都会被维系运行与兼容性所耗尽。此时，真正的技术突破必然来自模型之外的工程与系统创新。
- “Like it's really hard today with this model it makes it much much much easier and then of course this could change the nature of software fundamentally to make it more stochastic over.” —— 嘉宾指出，让语言模型稳健驱动逻辑判断的门槛正在降低，这将彻底改变软件的根本性质，使其更具随机性与概率特征。
- “And I think that what's so cool is that that it is happening outside the models because that's what I think is just going to happen which is the center of innovation has just moved.” —— 嘉宾直言创新的中心已经发生转移，最前沿的突破正在模型本体之外全面爆发。
- “And so I think that this is the signal that now people have figured out that there's innovation to be done. To the model, but outside the model.” —— 嘉宾总结道，业界如今终于意识到，真正的创新空间存在于赋能模型、但在模型结构之外的整个软件栈之中。
## Notable quotes
> [00:50](https://www.youtube.com/watch?v=TLJNJDf2XGo&t=50s) 如果没有能够被企业信赖、且不会不断入侵系统的高质量产品，AI 是无法实现广泛普及的。 — Aaron Levie
> [06:06](https://www.youtube.com/watch?v=TLJNJDf2XGo&t=366s) The answer is to nationalize it and actually put controls that we know that work. — Martin Casado
> [07:50](https://www.youtube.com/watch?v=TLJNJDf2XGo&t=470s) In order for something to be slower, you need to know the rate at which it was moving in the first place. — Steven Sinofsky
> [18:25](https://www.youtube.com/watch?v=TLJNJDf2XGo&t=1105s) Well, I think the tech industry has literally over 100 years consistently relearn the lesson at each technology wave that we don't understand the regulatory climate and we can't navigate it. — Steven Sinofsky
> [21:25](https://www.youtube.com/watch?v=TLJNJDf2XGo&t=1285s) 像这样强大的技术，最终国会不可能不关心它。这根本是不可能的。 — Aaron Levie
> [24:26](https://www.youtube.com/watch?v=TLJNJDf2XGo&t=1466s) 当我们在制定政策并且确实出台了大量政策时，那都是基于针对你所要解决的问题的大量具体数据点，这样你才能确信这项政策确实契合了事实案情。 — Martin Casado
> [34:01](https://www.youtube.com/watch?v=TLJNJDf2XGo&t=2041s) what AI can do is it can try all of those things in a very short time — Aaron Levie
> [35:53](https://www.youtube.com/watch?v=TLJNJDf2XGo&t=2153s) Because the model we have is not granular enough and it's not performance enough to handle this stuff. — Aaron Levie
## Mentioned
Aaron Levie, Martin Casado, Steven Sinofsky, Erik Torenberg, Dario Amodei, David Sacks, Noam Brown, Greg Brockman, Nick Bostrom, Bill Gates, Bill Clinton, Al Gore, Elizabeth Warren, Ralph Nader, Box, a16z, Anthropic, OpenAI, Hugging Face, Microsoft, Apple, AT&T, IBM, JPMorgan Chase, Verizon, Okta, Google, Google Workspace, Lawrence Livermore National Laboratory, NASA, GTE Telenet, Carnegie Mellon University, Stanford University, FAA, FDA, MPAA, FINRA, NIST, Department of Justice, Windows, Windows 95, Windows XP, Windows CE, iPhone, Slack, GitHub, ChatGPT, Computer Fraud and Abuse Act, Communications Decency Act, GDPR, User Account Control, TEMPEST, Pershing missile
## Video description
Erik Torenberg sits down with Box CEO Aaron Levie, and a16z’s Martin Casado, and Steven Sinofsky to debate how the AI industry should think about safety, security, and regulation as increasingly capable agents move into the real world.
They argue that much of today’s conversation is happening before we have clearly defined the risks we’re trying to regulate. Drawing on earlier waves of computing, from computer viruses and the early internet to aviation and automobiles, they ask what AI can learn from industries that developed safety standards only after understanding how their technologies actually failed.
The conversation then gets concrete: agents don’t get tired, can operate at enormous scale, and can probe systems in ways human employees never could. That could require rethinking permissions, authentication, operating systems, and the security stack itself. They also discuss why AI innovation may increasingly move beyond the frontier labs and into the software built around the models.
Timestamps:
00:00 - Intro
00:50 - Pacing the Frontier: Reacting to the AI Safety Discourse
03:50 - Species Extinction Talk: Do Labs Actually Believe Their Own Rhetoric?
06:49 - The Nationalization Question & Whose Job Safety Really Is
11:32 - David Sacks vs Government: "You're Asking Us to Regulate What?"
12:08 - 2028 as the AI Election
28:39 - Why Cybersecurity Has Historically Rejected Practical Solutions
32:30 - The Craziest Covert Channel Ever Seen
48:38 - LLMs as Decision Engines vs Chatbots
53:01 - Why the Labs Haven't Built This: The Being vs Tool Mindset
Resources:
Follow Aaron Levie on X: https://x.com/levie
Follow Martin Casado on X: https://x.com/martin_casado
Follow Steven Sinofsky on X: https://x.com/stevesi
Stay Updated:
If you enjoyed this episode, be sure to like, subscribe, and share with your friends!
Find a16z on X: https://twitter.com/a16z
Find a16z on LinkedIn: https://www.linkedin.com/company/a16z
Listen to the a16z Show on Spotify: https://open.spotify.com/show/5bC65RDvs3oxnLyqqvkUYX
Listen to the a16z Show on Apple Podcasts: https://podcasts.apple.com/us/podcast/a16z-podcast/id842818711
Follow our host: https://x.com/eriktorenberg
Please note that the content here is for informational purposes only; should NOT be taken as legal, business, tax, or investment advice or be used to evaluate any investment or security; and is not directed at any investors or potential investors in any a16z fund. a16z and its affiliates may maintain investments in the companies discussed. For more details please see http://a16z.com/disclosures.
[Source](https://www.youtube.com/watch?v=TLJNJDf2XGo)
Source quotes
[0:50] Aaron Levie: 如果没有能够被企业信赖、且不会不断入侵系统的高质量产品，AI 是无法实现广泛普及的。
[6:06] Martin Casado: The answer is to nationalize it and actually put controls that we know that work.
[7:50] Steven Sinofsky: In order for something to be slower, you need to know the rate at which it was moving in the first place.
[18:25] Steven Sinofsky: Well, I think the tech industry has literally over 100 years consistently relearn the lesson at each technology wave that we don't understand the regulatory climate and we can't navigate it.
[21:25] Aaron Levie: 像这样强大的技术，最终国会不可能不关心它。这根本是不可能的。
[24:26] Martin Casado: 当我们在制定政策并且确实出台了大量政策时，那都是基于针对你所要解决的问题的大量具体数据点，这样你才能确信这项政策确实契合了事实案情。
[34:01] Aaron Levie: what AI can do is it can try all of those things in a very short time
[35:53] Aaron Levie: Because the model we have is not granular enough and it's not performance enough to handle this stuff.
Source quotes
[0:00] You will the thing into being, but you haven't figured out how to control it.
[0:00] roaming drones but like times 10,000
[0:52] Martin Casado: taking credit for them, as VCs do.
[3:50] milquetoast capitulation that's totally not adequate for the task at hand.
[6:06] Martin Casado: we have to be at the forefront of AI, but I'm also terrified of it, which is why I must build it
[7:48] Steven Sinofsky: dead run
[18:25] Bill Gates: Isn't that what I was supposed to do—go and play golf?
[22:16] Steven Sinofsky: The internet is so big, how did we not have anything to do with it?
[24:26] we would probably not have the internet.
[30:25] You can't just wave your hand and say that should go away and then bring up, 'Oh, but you know, space aliens can invade.'
[50:30] there's no probability in accounting
[53:01] they're trying to create beings and beings speak. If you're trying to create god, god speaks in natural languages.
[53:01] Steven Sinofsky: Sherlocking
Original transcript
[0:02] If you regulate AI too early, you
[0:02] If you regulate AI too early, you actually don't solve anything. You still
[0:04] actually don't solve anything. You still
[0:04] actually don't solve anything. You still just kind of had the same risk
[0:05] just kind of had the same risk
[0:05] just kind of had the same risk ultimately.
[0:06] ultimately.
[0:06] ultimately. >> You will the thing into being, but you
[0:07] >> You will the thing into being, but you
[0:07] >> You will the thing into being, but you haven't figured out how to control it.
[0:09] haven't figured out how to control it.
[0:09] haven't figured out how to control it. >> The problem we have now is this rift
[0:11] >> The problem we have now is this rift
[0:11] >> The problem we have now is this rift between the labs and the security
[0:13] between the labs and the security
[0:13] between the labs and the security community that keeps coming to two
[0:14] community that keeps coming to two
[0:14] community that keeps coming to two conclusions. Sloppy and you're not
[0:16] conclusions. Sloppy and you're not
[0:16] conclusions. Sloppy and you're not complete in what you're telling us
[0:18] complete in what you're telling us
[0:18] complete in what you're telling us happened.
[0:18] happened.
[0:18] happened. >> An employees like 10% chance of species
[0:21] >> An employees like 10% chance of species
[0:21] >> An employees like 10% chance of species extinction. Agent swarms completely flip
[0:24] extinction. Agent swarms completely flip
[0:24] extinction. Agent swarms completely flip that. These are just roaming drones but
[0:26] that. These are just roaming drones but
[0:26] that. These are just roaming drones but like times 10,000 and they will easily
[0:28] like times 10,000 and they will easily
[0:28] like times 10,000 and they will easily mistake a good task for a bad one.
[0:30] mistake a good task for a bad one.
[0:30] mistake a good task for a bad one. >> So now we need a whole layer internally
[0:32] >> So now we need a whole layer internally
[0:32] >> So now we need a whole layer internally that just is tracking way more about
[0:34] that just is tracking way more about
[0:34] that just is tracking way more about what authentications are being done,
[0:35] what authentications are being done,
[0:35] what authentications are being done, what APIs are being done. The US about
[0:37] what APIs are being done. The US about
[0:37] what APIs are being done. The US about 15 years ago stopped leading in tech
[0:39] 15 years ago stopped leading in tech
[0:39] 15 years ago stopped leading in tech antitrust. The problem is that Europe is
[0:42] antitrust. The problem is that Europe is
[0:42] antitrust. The problem is that Europe is going to lead with that because they
[0:43] going to lead with that because they
[0:43] going to lead with that because they have nothing to lose. This could change
[0:45] have nothing to lose. This could change
[0:45] have nothing to lose. This could change the nature of software fundamentally.
[0:46] the nature of software fundamentally.
[0:46] the nature of software fundamentally. The center of innovation has just moved.
[0:48] The center of innovation has just moved.
[0:48] The center of innovation has just moved. This is the signal that
[0:52] This is the signal that
[0:52] This is the signal that >> guys, welcome back to the podcast.
[0:54] >> guys, welcome back to the podcast.
[0:54] >> guys, welcome back to the podcast. >> Happy to be here.
[0:55] >> Happy to be here.
[0:55] >> Happy to be here. >> Thank you.
[0:55] >> Thank you.
[0:55] >> Thank you. >> Your beard.
[0:56] >> Your beard.
[0:56] >> Your beard. >> Didn't think we'd ever do this again.
[0:57] >> Didn't think we'd ever do this again.
[0:58] >> Didn't think we'd ever do this again. This is I can't believe it. This is
[0:59] This is I can't believe it. This is
[0:59] This is I can't believe it. This is great.
[1:00] great.
[1:00] great. >> I mean, Martine's just building these
[1:02] >> I mean, Martine's just building these
[1:02] >> I mean, Martine's just building these like hundred billion dollar companies.
[1:03] like hundred billion dollar companies.
[1:03] like hundred billion dollar companies. Too busy for this podcast. So,
[1:05] Too busy for this podcast. So,
[1:05] Too busy for this podcast. So, >> at least taking credit for as VCs do.
[1:07] >> at least taking credit for as VCs do.
[1:07] >> at least taking credit for as VCs do. >> Exactly. Um, we have a lot to discuss
[1:09] >> Exactly. Um, we have a lot to discuss
[1:09] >> Exactly. Um, we have a lot to discuss today, but f first look, Aaron, why
[1:10] today, but f first look, Aaron, why
[1:10] today, but f first look, Aaron, why don't we start with you?
[1:12] don't we start with you?
[1:12] don't we start with you? >> Pacing the frontier. How have you uh
[1:14] >> Pacing the frontier. How have you uh
[1:14] >> Pacing the frontier. How have you uh reacted and reflected on on what's
[1:16] reacted and reflected on on what's
[1:16] reacted and reflected on on what's happened there and just the discourse
[1:17] happened there and just the discourse
[1:17] happened there and just the discourse that that's followed?
[1:18] that that's followed?
[1:18] that that's followed? >> Oh boy. I think we should start with
[1:19] >> Oh boy. I think we should start with
[1:19] >> Oh boy. I think we should start with Martine on this one. The uh he he you
[1:21] Martine on this one. The uh he he you
[1:21] Martine on this one. The uh he he you were fighting lots of good ground wars.
[1:23] were fighting lots of good ground wars.
[1:23] were fighting lots of good ground wars. Um I I maybe I'll say one thing that we
[1:26] Um I I maybe I'll say one thing that we
[1:26] Um I I maybe I'll say one thing that we probably all agree with and then we can
[1:27] probably all agree with and then we can
[1:28] probably all agree with and then we can figure out where we maybe maybe kind of
[1:30] figure out where we maybe maybe kind of
[1:30] figure out where we maybe maybe kind of fracture off. Um I think we would agree
[1:33] fracture off. Um I think we would agree
[1:33] fracture off. Um I think we would agree that uh any AI lab right now at the
[1:37] that uh any AI lab right now at the
[1:37] that uh any AI lab right now at the frontier should be building you know in
[1:39] frontier should be building you know in
[1:39] frontier should be building you know in the safest way possible with with you
[1:42] the safest way possible with with you
[1:42] the safest way possible with with you know the the highest degree of
[1:43] know the the highest degree of
[1:43] know the the highest degree of governance and and security and you know
[1:46] governance and and security and you know
[1:46] governance and and security and you know kind of whatever your definition of
[1:47] kind of whatever your definition of
[1:47] kind of whatever your definition of alignment is like this is an incredibly
[1:49] alignment is like this is an incredibly
[1:49] alignment is like this is an incredibly important area of research. It's
[1:51] important area of research. It's
[1:51] important area of research. It's incredibly important area for the
[1:53] incredibly important area for the
[1:53] incredibly important area for the diffusion of AI. like you're not going
[1:54] diffusion of AI. like you're not going
[1:54] diffusion of AI. like you're not going to have AI diffusion without extremely
[1:56] to have AI diffusion without extremely
[1:56] to have AI diffusion without extremely highquality products that can be, you
[1:59] highquality products that can be, you
[1:59] highquality products that can be, you know, trusted by enterprises and that
[2:00] know, trusted by enterprises and that
[2:00] know, trusted by enterprises and that aren't kind of constantly hacking
[2:01] aren't kind of constantly hacking
[2:01] aren't kind of constantly hacking systems. So when when at least I read
[2:04] systems. So when when at least I read
[2:04] systems. So when when at least I read the Daario post, um I actually didn't
[2:06] the Daario post, um I actually didn't
[2:06] the Daario post, um I actually didn't agree with didn't disagree with almost
[2:08] agree with didn't disagree with almost
[2:08] agree with didn't disagree with almost anything. Um because it was all about
[2:11] anything. Um because it was all about
[2:11] anything. Um because it was all about how do you have better, you know, uh
[2:12] how do you have better, you know, uh
[2:12] how do you have better, you know, uh security of these systems, sandboxing,
[2:15] security of these systems, sandboxing,
[2:15] security of these systems, sandboxing, better testing. There's going to be some
[2:16] better testing. There's going to be some
[2:16] better testing. There's going to be some debates around the embedded nature of of
[2:18] debates around the embedded nature of of
[2:18] debates around the embedded nature of of of the testers and like do you agree
[2:20] of the testers and like do you agree
[2:20] of the testers and like do you agree with who those are and does the industry
[2:22] with who those are and does the industry
[2:22] with who those are and does the industry all align on that? But I think actually
[2:24] all align on that? But I think actually
[2:24] all align on that? But I think actually all of the major points were were
[2:25] all of the major points were were
[2:25] all of the major points were were probably salient and and appropriate.
[2:27] probably salient and and appropriate.
[2:27] probably salient and and appropriate. Then the only question is does this get
[2:29] Then the only question is does this get
[2:30] Then the only question is does this get sort of used or leveraged to do things
[2:32] sort of used or leveraged to do things
[2:32] sort of used or leveraged to do things that maybe we don't agree with which
[2:34] that maybe we don't agree with which
[2:34] that maybe we don't agree with which would be like a slowdown of of AI
[2:36] would be like a slowdown of of AI
[2:36] would be like a slowdown of of AI dramatically because of regulatory
[2:38] dramatically because of regulatory
[2:38] dramatically because of regulatory controls that would would sort of you
[2:40] controls that would would sort of you
[2:40] controls that would would sort of you know not make it easy to compete you
[2:42] know not make it easy to compete you
[2:42] know not make it easy to compete you know with with the frontier labs. um or
[2:45] know with with the frontier labs. um or
[2:45] know with with the frontier labs. um or do politicians kind of end up sort of
[2:47] do politicians kind of end up sort of
[2:47] do politicians kind of end up sort of taking the the message and run with it
[2:49] taking the the message and run with it
[2:49] taking the the message and run with it and maybe even worse outcomes happen
[2:50] and maybe even worse outcomes happen
[2:50] and maybe even worse outcomes happen like you you you know it's used to ban
[2:53] like you you you know it's used to ban
[2:53] like you you you know it's used to ban data centers you know far faster and and
[2:55] data centers you know far faster and and
[2:55] data centers you know far faster and and whatnot and so so I think the the actual
[2:57] whatnot and so so I think the the actual
[2:57] whatnot and so so I think the the actual like substance of the topic is actually
[2:59] like substance of the topic is actually
[2:59] like substance of the topic is actually incredibly important and and I I think
[3:01] incredibly important and and I I think
[3:01] incredibly important and and I I think very important for for AI advancement in
[3:03] very important for for AI advancement in
[3:03] very important for for AI advancement in general and then the question is what do
[3:04] general and then the question is what do
[3:04] general and then the question is what do you do about it especially what do you
[3:06] you do about it especially what do you
[3:06] you do about it especially what do you do about it from a regulatory standpoint
[3:07] do about it from a regulatory standpoint
[3:07] do about it from a regulatory standpoint and that's probably where the industry
[3:09] and that's probably where the industry
[3:09] and that's probably where the industry is going to land on very different
[3:10] is going to land on very different
[3:10] is going to land on very different points in the continuum but Martin was
[3:12] points in the continuum but Martin was
[3:12] points in the continuum but Martin was putting up a good fight on on like let's
[3:14] putting up a good fight on on like let's
[3:14] putting up a good fight on on like let's make sure that we don't use this for
[3:15] make sure that we don't use this for
[3:15] make sure that we don't use this for regulatory kind of capture. Um which I I
[3:18] regulatory kind of capture. Um which I I
[3:18] regulatory kind of capture. Um which I I also agree with but but I think the the
[3:20] also agree with but but I think the the
[3:20] also agree with but but I think the the ideas in the pacing conversation are
[3:22] ideas in the pacing conversation are
[3:22] ideas in the pacing conversation are important. Um, you know, again, like
[3:25] important. Um, you know, again, like
[3:25] important. Um, you know, again, like it's a little bit of a funny concept
[3:26] it's a little bit of a funny concept
[3:26] it's a little bit of a funny concept because maybe it's not even pacing as
[3:28] because maybe it's not even pacing as
[3:28] because maybe it's not even pacing as much as just like good hygiene and and
[3:30] much as just like good hygiene and and
[3:30] much as just like good hygiene and and then good and and engineering and and so
[3:32] then good and and engineering and and so
[3:32] then good and and engineering and and so with with good engineering, obviously
[3:34] with with good engineering, obviously
[3:34] with with good engineering, obviously there is a slight slowdown, but it's a
[3:37] there is a slight slowdown, but it's a
[3:37] there is a slight slowdown, but it's a slowdown that obviously allows
[3:38] slowdown that obviously allows
[3:38] slowdown that obviously allows acceleration of your diffusion because
[3:40] acceleration of your diffusion because
[3:40] acceleration of your diffusion because you you wouldn't be able to have any of
[3:42] you you wouldn't be able to have any of
[3:42] you you wouldn't be able to have any of the AI be diffused if if nobody would,
[3:44] the AI be diffused if if nobody would,
[3:44] the AI be diffused if if nobody would, you know, trust using it. So the post is
[3:47] you know, trust using it. So the post is
[3:47] you know, trust using it. So the post is is very reasonable, but the atmospherics
[3:50] is very reasonable, but the atmospherics
[3:50] is very reasonable, but the atmospherics are not right. Like so an employee is
[3:53] are not right. Like so an employee is
[3:53] are not right. Like so an employee is like this is going to kill whatever 10%
[3:55] like this is going to kill whatever 10%
[3:55] like this is going to kill whatever 10% chance of species extinction and you
[3:58] chance of species extinction and you
[3:58] chance of species extinction and you know what Daario says
[4:00] know what Daario says
[4:00] know what Daario says >> I agree with him more than I disagree
[4:02] >> I agree with him more than I disagree
[4:02] >> I agree with him more than I disagree with him right like
[4:03] with him right like
[4:03] with him right like >> on TV on TV the same day that he landed
[4:07] >> on TV on TV the same day that he landed
[4:07] >> on TV on TV the same day that he landed these things and so in some way you
[4:09] these things and so in some way you
[4:09] these things and so in some way you can't have these conversations in
[4:10] can't have these conversations in
[4:10] can't have these conversations in isolation which is of course if he's
[4:13] isolation which is of course if he's
[4:13] isolation which is of course if he's going to agree in species extinction
[4:16] going to agree in species extinction
[4:16] going to agree in species extinction >> this post that he has looks like this
[4:19] >> this post that he has looks like this
[4:19] >> this post that he has looks like this milktoast capitulation that's totally
[4:22] milktoast capitulation that's totally
[4:22] milktoast capitulation that's totally not adequate for the task at hand. And
[4:24] not adequate for the task at hand. And
[4:24] not adequate for the task at hand. And so I think the atmospherics are totally
[4:25] so I think the atmospherics are totally
[4:25] so I think the atmospherics are totally broken. And a lot of my comments were on
[4:27] broken. And a lot of my comments were on
[4:27] broken. And a lot of my comments were on the atmospherics.
[4:28] the atmospherics.
[4:28] the atmospherics. >> And then I have this quibble, but it
[4:30] >> And then I have this quibble, but it
[4:30] >> And then I have this quibble, but it really bothers me because I'm impeded,
[4:31] really bothers me because I'm impeded,
[4:31] really bothers me because I'm impeded, which is
[4:32] which is
[4:32] which is >> I think pacing is the wrong way to
[4:35] >> I think pacing is the wrong way to
[4:36] >> I think pacing is the wrong way to describe this. For one, yes, it is
[4:37] describe this. For one, yes, it is
[4:38] describe this. For one, yes, it is orthogonal to security.
[4:39] orthogonal to security.
[4:39] orthogonal to security. >> Mhm.
[4:40] >> Mhm.
[4:40] >> Mhm. >> Right. So like you can very slowly build
[4:43] >> Right. So like you can very slowly build
[4:43] >> Right. So like you can very slowly build a nuclear weapon and that doesn't make
[4:45] a nuclear weapon and that doesn't make
[4:45] a nuclear weapon and that doesn't make anybody feel better that it's slow
[4:46] anybody feel better that it's slow
[4:46] anybody feel better that it's slow versus fast. So that's one. The second
[4:48] versus fast. So that's one. The second
[4:48] versus fast. So that's one. The second one, it just feels like a capitulation
[4:51] one, it just feels like a capitulation
[4:51] one, it just feels like a capitulation to the pause folks without actually, you
[4:55] to the pause folks without actually, you
[4:55] to the pause folks without actually, you know, addressing it. So, you're saying,
[4:56] know, addressing it. So, you're saying,
[4:56] know, addressing it. So, you're saying, well, we're not going to pause. We're
[4:57] well, we're not going to pause. We're
[4:57] well, we're not going to pause. We're going to pace to make them happy, but
[4:59] going to pace to make them happy, but
[4:59] going to pace to make them happy, but we'll also somehow make the regulators
[5:00] we'll also somehow make the regulators
[5:00] we'll also somehow make the regulators happy. And I think it makes them both
[5:01] happy. And I think it makes them both
[5:02] happy. And I think it makes them both unhappy.
[5:03] unhappy.
[5:03] unhappy. >> Cuz the pause people are like, well,
[5:05] >> Cuz the pause people are like, well,
[5:05] >> Cuz the pause people are like, well, that's not a pause. That's just pacing.
[5:06] that's not a pause. That's just pacing.
[5:06] that's not a pause. That's just pacing. And then the regulators are, you're
[5:08] And then the regulators are, you're
[5:08] And then the regulators are, you're still doing the thing,
[5:09] still doing the thing,
[5:09] still doing the thing, >> right?
[5:09] >> right?
[5:09] >> right? >> And so, I just feel like they're trying
[5:11] >> And so, I just feel like they're trying
[5:11] >> And so, I just feel like they're trying to my my sense of what's happening is
[5:13] to my my sense of what's happening is
[5:13] to my my sense of what's happening is the labs are actually trying to do the
[5:15] the labs are actually trying to do the
[5:15] the labs are actually trying to do the right thing. And I applaud them for
[5:16] right thing. And I applaud them for
[5:16] right thing. And I applaud them for that. I think this is a pragmatic
[5:17] that. I think this is a pragmatic
[5:18] that. I think this is a pragmatic proposal and I applaud them for that. I
[5:20] proposal and I applaud them for that. I
[5:20] proposal and I applaud them for that. I think the messaging is wrong because
[5:22] think the messaging is wrong because
[5:22] think the messaging is wrong because they're trying to like split the
[5:24] they're trying to like split the
[5:24] they're trying to like split the difference between an internal kind of
[5:26] difference between an internal kind of
[5:26] difference between an internal kind of fringe faction which are doomers pause
[5:29] fringe faction which are doomers pause
[5:29] fringe faction which are doomers pause and then the regulators on the other
[5:30] and then the regulators on the other
[5:30] and then the regulators on the other side and the problem is they're making
[5:31] side and the problem is they're making
[5:31] side and the problem is they're making both of them unhappy and I think what
[5:33] both of them unhappy and I think what
[5:33] both of them unhappy and I think what they have to do is they need to come out
[5:35] they have to do is they need to come out
[5:35] they have to do is they need to come out they need to address the x-risk question
[5:37] they need to address the x-risk question
[5:37] they need to address the x-risk question directly
[5:38] directly
[5:38] directly >> like they need to say no we don't think
[5:40] >> like they need to say no we don't think
[5:40] >> like they need to say no we don't think this stuff we're going to do is going to
[5:41] this stuff we're going to do is going to
[5:41] this stuff we're going to do is going to cause extinction and then I think this
[5:43] cause extinction and then I think this
[5:43] cause extinction and then I think this becomes this very sensible
[5:45] becomes this very sensible
[5:45] becomes this very sensible >> and what would you do just to just to
[5:46] >> and what would you do just to just to
[5:46] >> and what would you do just to just to play the other side for one second what
[5:48] play the other side for one second what
[5:48] play the other side for one second what do you do Do you uh make room for the
[5:53] do you do Do you uh make room for the
[5:53] do you do Do you uh make room for the the one possible ven diagram which is
[5:55] the one possible ven diagram which is
[5:55] the one possible ven diagram which is the the lab researcher that is both
[5:58] the the lab researcher that is both
[5:58] the the lab researcher that is both simultaneously super scared but also
[6:01] simultaneously super scared but also
[6:01] simultaneously super scared but also works on advancing the state of AI
[6:04] works on advancing the state of AI
[6:04] works on advancing the state of AI because they believe that that it's so
[6:06] because they believe that that it's so
[6:06] because they believe that that it's so important to get right that that they
[6:08] important to get right that that they
[6:08] important to get right that that they want to pursue that and then obviously
[6:10] want to pursue that and then obviously
[6:10] want to pursue that and then obviously the language is is right now very you
[6:13] the language is is right now very you
[6:13] the language is is right now very you know problematic but like what but that
[6:15] know problematic but like what but that
[6:15] know problematic but like what but that that person does exist and and that that
[6:18] that person does exist and and that that
[6:18] that person does exist and and that that is a real kind of person in our industry
[6:19] is a real kind of person in our industry
[6:20] is a real kind of person in our industry which is like like we have to be at the
[6:22] which is like like we have to be at the
[6:22] which is like like we have to be at the at the forefront of AI. I'm also very
[6:24] at the forefront of AI. I'm also very
[6:24] at the forefront of AI. I'm also very scared of it and and and so that's why
[6:26] scared of it and and and so that's why
[6:26] scared of it and and and so that's why I'm working on this.
[6:27] I'm working on this.
[6:27] I'm working on this. >> So let let me address this directly. I
[6:29] >> So let let me address this directly. I
[6:29] >> So let let me address this directly. I used to work for Lawrence Livermore
[6:31] used to work for Lawrence Livermore
[6:31] used to work for Lawrence Livermore National Labs on a weapons program,
[6:32] National Labs on a weapons program,
[6:32] National Labs on a weapons program, nuclear weapons. I know what it's like
[6:34] nuclear weapons. I know what it's like
[6:34] nuclear weapons. I know what it's like to work on things that have access.
[6:35] to work on things that have access.
[6:35] to work on things that have access. >> You were the first pacer.
[6:38] >> You were the first pacer.
[6:38] >> You were the first pacer. >> We were part of the first pacers.
[6:41] >> We were part of the first pacers.
[6:41] >> We were part of the first pacers. So if a constituency within the labs
[6:45] So if a constituency within the labs
[6:45] So if a constituency within the labs that are the most knowledgeable people
[6:47] that are the most knowledgeable people
[6:47] that are the most knowledgeable people believe the stuff has existential risk.
[6:48] believe the stuff has existential risk.
[6:48] believe the stuff has existential risk. >> Yes.
[6:49] >> Yes.
[6:49] >> Yes. >> The answer is to nationalize it and
[6:52] >> The answer is to nationalize it and
[6:52] >> The answer is to nationalize it and actually put controls that we know that
[6:53] actually put controls that we know that
[6:53] actually put controls that we know that work.
[6:54] work.
[6:54] work. >> Right now if they don't actually believe
[6:56] >> Right now if they don't actually believe
[6:56] >> Right now if they don't actually believe that and in the private conversations I
[6:58] that and in the private conversations I
[6:58] that and in the private conversations I have the most sensible people don't.
[7:00] have the most sensible people don't.
[7:00] have the most sensible people don't. It's a small fraction that do this is an
[7:02] It's a small fraction that do this is an
[7:02] It's a small fraction that do this is an HR problem.
[7:04] HR problem.
[7:04] HR problem. >> Right? So to me an HR problem is a
[7:06] >> Right? So to me an HR problem is a
[7:06] >> Right? So to me an HR problem is a company problem. like if they are
[7:08] company problem. like if they are
[7:08] company problem. like if they are worried that they can't recruit people
[7:10] worried that they can't recruit people
[7:10] worried that they can't recruit people >> right
[7:11] >> right
[7:11] >> right >> you know or they can't retain people
[7:12] >> you know or they can't retain people
[7:12] >> you know or they can't retain people which it seems to me a lot of this is
[7:14] which it seems to me a lot of this is
[7:14] which it seems to me a lot of this is just that concern it's almost more of
[7:16] just that concern it's almost more of
[7:16] just that concern it's almost more of this kind of researcher currency if
[7:17] this kind of researcher currency if
[7:17] this kind of researcher currency if that's the case
[7:19] that's the case
[7:19] that's the case >> I think that is the wrong reason to
[7:21] >> I think that is the wrong reason to
[7:21] >> I think that is the wrong reason to cause a national level lockdown on a
[7:24] cause a national level lockdown on a
[7:24] cause a national level lockdown on a very promising technology so listen I
[7:26] very promising technology so listen I
[7:26] very promising technology so listen I think the post again I think the post is
[7:28] think the post again I think the post is
[7:28] think the post again I think the post is actually very sensible I think there are
[7:29] actually very sensible I think there are
[7:29] actually very sensible I think there are real concerns around security we've had
[7:32] real concerns around security we've had
[7:32] real concerns around security we've had many compute epochs that have real
[7:34] many compute epochs that have real
[7:34] many compute epochs that have real security concerns I don't think you can
[7:37] security concerns I don't think you can
[7:37] security concerns I don't think you can ex you can reconcile discussions on X
[7:39] ex you can reconcile discussions on X
[7:39] ex you can reconcile discussions on X risk with the proposal that was put out.
[7:41] risk with the proposal that was put out.
[7:41] risk with the proposal that was put out. You just can't reconcile those two
[7:43] You just can't reconcile those two
[7:43] You just can't reconcile those two things and that has always been my
[7:44] things and that has always been my
[7:44] things and that has always been my primary.
[7:44] primary.
[7:44] primary. >> Right. Right. Okay. Unleash. Okay.
[7:48] >> Right. Right. Okay. Unleash. Okay.
[7:48] >> Right. Right. Okay. Unleash. Okay. >> Holding it back.
[7:50] >> Holding it back.
[7:50] >> Holding it back. >> So, okay. The the first thing is you
[7:52] >> So, okay. The the first thing is you
[7:52] >> So, okay. The the first thing is you can't have you. Is there a schedule that
[7:56] can't have you. Is there a schedule that
[7:56] can't have you. Is there a schedule that they've published that says when all
[7:58] they've published that says when all
[7:58] they've published that says when all this stuff whatever bad stuff is going
[8:00] this stuff whatever bad stuff is going
[8:00] this stuff whatever bad stuff is going to happen is? They haven't. So, you
[8:01] to happen is? They haven't. So, you
[8:01] to happen is? They haven't. So, you can't pace it because nobody knows when
[8:04] can't pace it because nobody knows when
[8:04] can't pace it because nobody knows when it was supposed to finish in the first.
[8:05] it was supposed to finish in the first.
[8:05] it was supposed to finish in the first. It also just seems disingenuous. It's
[8:07] It also just seems disingenuous. It's
[8:07] It also just seems disingenuous. It's completely
[8:09] completely
[8:09] completely claim This is like when the press
[8:10] claim This is like when the press
[8:10] claim This is like when the press reports on Apple's latest iPhone is
[8:12] reports on Apple's latest iPhone is
[8:12] reports on Apple's latest iPhone is late.
[8:13] late.
[8:13] late. >> The phone from what nobody knows exists
[8:16] >> The phone from what nobody knows exists
[8:16] >> The phone from what nobody knows exists that they haven't told anybody about.
[8:17] that they haven't told anybody about.
[8:17] that they haven't told anybody about. >> Yeah. My Apple car was very late.
[8:19] >> Yeah. My Apple car was very late.
[8:19] >> Yeah. My Apple car was very late. >> Yeah. Like I I don't understand. Like in
[8:22] >> Yeah. Like I I don't understand. Like in
[8:22] >> Yeah. Like I I don't understand. Like in order for something to be slower, you
[8:24] order for something to be slower, you
[8:24] order for something to be slower, you need to know the rate at which it was
[8:25] need to know the rate at which it was
[8:25] need to know the rate at which it was moving in the first place. So it's all
[8:27] moving in the first place. So it's all
[8:27] moving in the first place. So it's all just utter nonsense and and you can't
[8:30] just utter nonsense and and you can't
[8:30] just utter nonsense and and you can't escape that. And then
[8:31] escape that. And then
[8:31] escape that. And then >> by the way, this is another problem that
[8:33] >> by the way, this is another problem that
[8:33] >> by the way, this is another problem that like again from the like my my little
[8:35] like again from the like my my little
[8:35] like again from the like my my little quibble on PR is
[8:38] quibble on PR is
[8:38] quibble on PR is nobody believes it anyway, right? And so
[8:40] nobody believes it anyway, right? And so
[8:40] nobody believes it anyway, right? And so if that's the thing, the pacing, right?
[8:43] if that's the thing, the pacing, right?
[8:43] if that's the thing, the pacing, right? >> Which part going to pace?
[8:45] >> Which part going to pace?
[8:45] >> Which part going to pace? >> That they're going to pace. They've been
[8:47] >> That they're going to pace. They've been
[8:47] >> That they're going to pace. They've been at a dead run. They've raised more money
[8:48] at a dead run. They've raised more money
[8:48] at a dead run. They've raised more money than ever before. They've grown faster
[8:49] than ever before. They've grown faster
[8:50] than ever before. They've grown faster than ever before. There's no indication
[8:51] than ever before. There's no indication
[8:51] than ever before. There's no indication that they're pacing. So if this is what
[8:52] that they're pacing. So if this is what
[8:52] that they're pacing. So if this is what you're going to hang your
[8:53] you're going to hang your
[8:53] you're going to hang your >> I see. I see. Yeah. Well, well, the
[8:55] >> I see. I see. Yeah. Well, well, the
[8:55] >> I see. I see. Yeah. Well, well, the internal the issue obviously is that the
[8:56] internal the issue obviously is that the
[8:56] internal the issue obviously is that the models that everybody has internally far
[8:58] models that everybody has internally far
[8:58] models that everybody has internally far exceed what anybody else has access to.
[9:00] exceed what anybody else has access to.
[9:00] exceed what anybody else has access to. So, it's really just the pacing of
[9:02] So, it's really just the pacing of
[9:02] So, it's really just the pacing of external releases. But again, back to
[9:04] external releases. But again, back to
[9:04] external releases. But again, back to your point, pacing versus what,
[9:06] your point, pacing versus what,
[9:06] your point, pacing versus what, >> right? Like, we don't know if the one
[9:07] >> right? Like, we don't know if the one
[9:07] >> right? Like, we don't know if the one that scares everybody also doesn't, you
[9:09] that scares everybody also doesn't, you
[9:09] that scares everybody also doesn't, you know, work for a legal brief. Like, it
[9:11] know, work for a legal brief. Like, it
[9:11] know, work for a legal brief. Like, it might actually screw all that stuff up
[9:12] might actually screw all that stuff up
[9:12] might actually screw all that stuff up because it's so who knows, right? So,
[9:14] because it's so who knows, right? So,
[9:14] because it's so who knows, right? So, there's that.
[9:15] there's that.
[9:16] there's that. >> And that's just it's just disingenuous
[9:18] >> And that's just it's just disingenuous
[9:18] >> And that's just it's just disingenuous to claim that you're pay. Second, why do
[9:20] to claim that you're pay. Second, why do
[9:20] to claim that you're pay. Second, why do they have to announce all of this and
[9:22] they have to announce all of this and
[9:22] they have to announce all of this and ask the government to tell them to pay?
[9:24] ask the government to tell them to pay?
[9:24] ask the government to tell them to pay? Like, that's the part that starts to go,
[9:26] Like, that's the part that starts to go,
[9:26] Like, that's the part that starts to go, well, this is really spooky. If you are
[9:27] well, this is really spooky. If you are
[9:27] well, this is really spooky. If you are the most afraid of how everything is
[9:29] the most afraid of how everything is
[9:29] the most afraid of how everything is going to go because of your product,
[9:30] going to go because of your product,
[9:30] going to go because of your product, just stop.
[9:31] just stop.
[9:31] just stop. >> Don't do it.
[9:32] >> Don't do it.
[9:32] >> Don't do it. >> Like like I like this is the like I
[9:34] >> Like like I like this is the like I
[9:34] >> Like like I like this is the like I worked at at at a missile factory in
[9:36] worked at at at a missile factory in
[9:36] worked at at at a missile factory in college and we had nuclear missiles and
[9:38] college and we had nuclear missiles and
[9:38] college and we had nuclear missiles and I walked the floor.
[9:39] I walked the floor.
[9:39] I walked the floor. >> What's with you guys in missiles? I
[9:41] >> What's with you guys in missiles? I
[9:41] >> What's with you guys in missiles? I don't I'm just here with software.
[9:47] >> You were one of two people in college
[9:47] >> You were one of two people in college when we were which was either you were
[9:49] when we were which was either you were
[9:49] when we were which was either you were protesting to keep them off campus
[9:51] protesting to keep them off campus
[9:51] protesting to keep them off campus >> or building them building them. So that
[9:54] >> or building them building them. So that
[9:54] >> or building them building them. So that was your choice. And and like we had
[9:57] was your choice. And and like we had
[9:57] was your choice. And and like we had like like nuns from the Catholic Church
[10:01] like like nuns from the Catholic Church
[10:01] like like nuns from the Catholic Church show up and pour blood all over our on
[10:02] show up and pour blood all over our on
[10:02] show up and pour blood all over our on missiles, you know, and I'm busy just
[10:04] missiles, you know, and I'm busy just
[10:04] missiles, you know, and I'm busy just wheeling PCs around on carts saying
[10:06] wheeling PCs around on carts saying
[10:06] wheeling PCs around on carts saying here's the secure PC and I'm like scared
[10:08] here's the secure PC and I'm like scared
[10:08] here's the secure PC and I'm like scared to die. I have no idea what's going on.
[10:09] to die. I have no idea what's going on.
[10:10] to die. I have no idea what's going on. And I'm like it's a nuclear missile and
[10:12] And I'm like it's a nuclear missile and
[10:12] And I'm like it's a nuclear missile and then you find out that that's what
[10:13] then you find out that that's what
[10:13] then you find out that that's what stopped the Cold War. Like that
[10:14] stopped the Cold War. Like that
[10:14] stopped the Cold War. Like that literally it was a Persian missile and
[10:16] literally it was a Persian missile and
[10:16] literally it was a Persian missile and that was what did it. And so but the you
[10:18] that was what did it. And so but the you
[10:18] that was what did it. And so but the you said something I think is super
[10:19] said something I think is super
[10:20] said something I think is super interesting which is pacing is this sort
[10:22] interesting which is pacing is this sort
[10:22] interesting which is pacing is this sort of
[10:23] of
[10:23] of fuzzy nonword between like going and not
[10:28] fuzzy nonword between like going and not
[10:28] fuzzy nonword between like going and not going. Yeah. And the the problem is it
[10:31] going. Yeah. And the the problem is it
[10:31] going. Yeah. And the the problem is it you're exactly right like the it there's
[10:33] you're exactly right like the it there's
[10:33] you're exactly right like the it there's no one is going to be happy with the
[10:35] no one is going to be happy with the
[10:35] no one is going to be happy with the middle road.
[10:36] middle road.
[10:36] middle road. >> And so one of the things that that
[10:37] >> And so one of the things that that
[10:37] >> And so one of the things that that people I think it's almost fun for me to
[10:39] people I think it's almost fun for me to
[10:39] people I think it's almost fun for me to watch as a sport. They think that
[10:41] watch as a sport. They think that
[10:41] watch as a sport. They think that they're that all these people saying to
[10:43] they're that all these people saying to
[10:43] they're that all these people saying to the government do this don't do this
[10:45] the government do this don't do this
[10:45] the government do this don't do this that they think they're going to get
[10:46] that they think they're going to get
[10:46] that they think they're going to get what they want. And it's a complete 100%
[10:49] what they want. And it's a complete 100%
[10:49] what they want. And it's a complete 100% misunderstanding of how government
[10:51] misunderstanding of how government
[10:51] misunderstanding of how government works, which is you when you talk to the
[10:53] works, which is you when you talk to the
[10:53] works, which is you when you talk to the government, they actually know how these
[10:56] government, they actually know how these
[10:56] government, they actually know how these things work. And they know they're just
[10:57] things work. And they know they're just
[10:58] things work. And they know they're just listening to you and they're listening
[11:00] listening to you and they're listening
[11:00] listening to you and they're listening to everyone. And no one is going to get
[11:02] to everyone. And no one is going to get
[11:02] to everyone. And no one is going to get what they want because they know in
[11:04] what they want because they know in
[11:04] what they want because they know in order to do anything in our system, it's
[11:06] order to do anything in our system, it's
[11:06] order to do anything in our system, it's a compromise. And so everything that
[11:09] a compromise. And so everything that
[11:09] a compromise. And so everything that that all the inputs can make it into the
[11:12] that all the inputs can make it into the
[11:12] that all the inputs can make it into the government, the output
[11:13] government, the output
[11:14] government, the output >> never makes everyone happy, right? 100%
[11:16] >> never makes everyone happy, right? 100%
[11:16] >> never makes everyone happy, right? 100% of the time it either doesn't go far
[11:18] of the time it either doesn't go far
[11:18] of the time it either doesn't go far enough or it goes way way too far.
[11:21] enough or it goes way way too far.
[11:21] enough or it goes way way too far. >> Yeah.
[11:21] >> Yeah.
[11:21] >> Yeah. >> And and so you can't take the view that
[11:24] >> And and so you can't take the view that
[11:24] >> And and so you can't take the view that you're going to talk to the government
[11:25] you're going to talk to the government
[11:25] you're going to talk to the government and talk your way into the solution you
[11:27] and talk your way into the solution you
[11:27] and talk your way into the solution you want. So the bottom line is if they're
[11:29] want. So the bottom line is if they're
[11:29] want. So the bottom line is if they're asking for pacing, they're going to get
[11:31] asking for pacing, they're going to get
[11:31] asking for pacing, they're going to get the wrong velocity. Well, my favorite
[11:33] the wrong velocity. Well, my favorite
[11:33] the wrong velocity. Well, my favorite thing is like what was it? David Sax was
[11:35] thing is like what was it? David Sax was
[11:35] thing is like what was it? David Sax was like
[11:37] like
[11:37] like I think it was David Saxs, but someone
[11:38] I think it was David Saxs, but someone
[11:38] I think it was David Saxs, but someone from the government said, "Uh, you're
[11:41] from the government said, "Uh, you're
[11:41] from the government said, "Uh, you're asking us to regulate you." No.
[11:44] asking us to regulate you." No.
[11:44] asking us to regulate you." No. Facebook. The answer was no.
[11:46] Facebook. The answer was no.
[11:46] Facebook. The answer was no. >> Well, but the thing that
[11:48] >> Well, but the thing that
[11:48] >> Well, but the thing that >> probably just Trump, I think, but uh
[11:49] >> probably just Trump, I think, but uh
[11:49] >> probably just Trump, I think, but uh >> the thing that they know now that that
[11:51] >> the thing that they know now that that
[11:51] >> the thing that they know now that that you just know from experience is
[11:53] you just know from experience is
[11:53] you just know from experience is >> once the wheels start on regulating,
[11:55] >> once the wheels start on regulating,
[11:55] >> once the wheels start on regulating, >> you can't slow that one down.
[11:56] >> you can't slow that one down.
[11:56] >> you can't slow that one down. >> And now it's become an election issue
[11:58] >> And now it's become an election issue
[11:58] >> And now it's become an election issue for every party in every jurisdiction up
[12:00] for every party in every jurisdiction up
[12:00] for every party in every jurisdiction up and down the whole government stack. So
[12:02] and down the whole government stack. So
[12:02] and down the whole government stack. So there is there's now this
[12:03] there is there's now this
[12:03] there is there's now this >> this whole basket goods basket of of
[12:06] >> this whole basket goods basket of of
[12:06] >> this whole basket goods basket of of regulatory approaches.
[12:07] regulatory approaches.
[12:07] regulatory approaches. >> Well, the the next election will 100% be
[12:09] >> Well, the the next election will 100% be
[12:09] >> Well, the the next election will 100% be a referendum on on AI. So, like it has
[12:13] a referendum on on AI. So, like it has
[12:13] a referendum on on AI. So, like it has to happen that that 2028 is like the AI
[12:16] to happen that that 2028 is like the AI
[12:16] to happen that that 2028 is like the AI election and you could basically run on
[12:20] election and you could basically run on
[12:20] election and you could basically run on the problem is is like it's not obvious
[12:21] the problem is is like it's not obvious
[12:21] the problem is is like it's not obvious who would run on the pro AI story
[12:24] who would run on the pro AI story
[12:24] who would run on the pro AI story because it it's going to be too nebulous
[12:25] because it it's going to be too nebulous
[12:25] because it it's going to be too nebulous to tell that story. So then it's just
[12:27] to tell that story. So then it's just
[12:27] to tell that story. So then it's just it's just basically varying degrees of
[12:29] it's just basically varying degrees of
[12:29] it's just basically varying degrees of how much do you regulate it or at least
[12:32] how much do you regulate it or at least
[12:32] how much do you regulate it or at least try and like avoid the topic. Yeah. But
[12:33] try and like avoid the topic. Yeah. But
[12:33] try and like avoid the topic. Yeah. But it is it is too bad that we've we as a
[12:36] it is it is too bad that we've we as a
[12:36] it is it is too bad that we've we as a country are in a spot where like the pro
[12:38] country are in a spot where like the pro
[12:38] country are in a spot where like the pro AAI case is just like it sounds too it's
[12:41] AAI case is just like it sounds too it's
[12:41] AAI case is just like it sounds too it's like it's just like takes too many
[12:43] like it's just like takes too many
[12:43] like it's just like takes too many words, you know, it's way too nuance.
[12:45] words, you know, it's way too nuance.
[12:45] words, you know, it's way too nuance. It's defensive.
[12:46] It's defensive.
[12:46] It's defensive. >> Yeah. It's defensive.
[12:46] >> Yeah. It's defensive.
[12:46] >> Yeah. It's defensive. >> It's defensive and don't and we we own
[12:48] >> It's defensive and don't and we we own
[12:48] >> It's defensive and don't and we we own none of the vocabulary,
[12:49] none of the vocabulary,
[12:49] none of the vocabulary, >> right? So the whole debate is is is
[12:53] >> right? So the whole debate is is is
[12:53] >> right? So the whole debate is is is pause. It's it's swarms. It's rogue.
[12:56] pause. It's it's swarms. It's rogue.
[12:56] pause. It's it's swarms. It's rogue. Every word has been chosen by the people
[12:58] Every word has been chosen by the people
[12:58] Every word has been chosen by the people who don't want to do AI.
[13:00] who don't want to do AI.
[13:00] who don't want to do AI. >> Yeah. And so it means the first thing
[13:01] >> Yeah. And so it means the first thing
[13:01] >> Yeah. And so it means the first thing you have to do is invent new words and
[13:03] you have to do is invent new words and
[13:03] you have to do is invent new words and say that their words are wrong which
[13:05] say that their words are wrong which
[13:05] say that their words are wrong which takes so many words that
[13:06] takes so many words that
[13:06] takes so many words that >> Yeah. So we got to be like union jobs
[13:09] >> Yeah. So we got to be like union jobs
[13:09] >> Yeah. So we got to be like union jobs and and you know cancer and like you
[13:12] and and you know cancer and like you
[13:12] and and you know cancer and like you know there we need you know there needs
[13:14] know there we need you know there needs
[13:14] know there we need you know there needs to be another word cloud that emerges.
[13:15] to be another word cloud that emerges.
[13:16] to be another word cloud that emerges. >> What I what I don't what I don't
[13:17] >> What I what I don't what I don't
[13:17] >> What I what I don't what I don't understand is why the labs have not
[13:18] understand is why the labs have not
[13:18] understand is why the labs have not taken a position on x- risk. Like short
[13:21] taken a position on x- risk. Like short
[13:21] taken a position on x- risk. Like short of that I don't think this goes in any
[13:22] of that I don't think this goes in any
[13:22] of that I don't think this goes in any direction other than heavy-handed
[13:24] direction other than heavy-handed
[13:24] direction other than heavy-handed regulation.
[13:25] regulation.
[13:25] regulation. >> Yeah. Well, it would just be negligent
[13:27] >> Yeah. Well, it would just be negligent
[13:27] >> Yeah. Well, it would just be negligent of the government to be like, there's a
[13:29] of the government to be like, there's a
[13:29] of the government to be like, there's a 10% chance of species extinction. The
[13:31] 10% chance of species extinction. The
[13:31] 10% chance of species extinction. The CEO of the top company says he agrees
[13:33] CEO of the top company says he agrees
[13:34] CEO of the top company says he agrees with it. Like, how can a government not
[13:35] with it. Like, how can a government not
[13:36] with it. Like, how can a government not do? Have you he had a regulation?
[13:37] do? Have you he had a regulation?
[13:37] do? Have you he had a regulation? >> Well, but what who But what would
[13:38] >> Well, but what who But what would
[13:38] >> Well, but what who But what would anybody really knowing the knowing this
[13:41] anybody really knowing the knowing this
[13:41] anybody really knowing the knowing this this ecosystem though, I don't know that
[13:43] this ecosystem though, I don't know that
[13:43] this ecosystem though, I don't know that you would be able to pin anybody down on
[13:44] you would be able to pin anybody down on
[13:44] you would be able to pin anybody down on that other than something?
[13:46] that other than something?
[13:46] that other than something? >> No, I would say Dario said it on
[13:47] >> No, I would say Dario said it on
[13:47] >> No, I would say Dario said it on >> No, but I'm saying you're not going to
[13:48] >> No, but I'm saying you're not going to
[13:48] >> No, but I'm saying you're not going to pin anybody down on a lower. Well, in
[13:50] pin anybody down on a lower. Well, in
[13:50] pin anybody down on a lower. Well, in fact, he he he does the worst thing
[13:52] fact, he he he does the worst thing
[13:52] fact, he he he does the worst thing about it, which is he agrees with people
[13:54] about it, which is he agrees with people
[13:54] about it, which is he agrees with people who claim that they believe that there's
[13:56] who claim that they believe that there's
[13:56] who claim that they believe that there's an x percentage of extinction happening,
[13:59] an x percentage of extinction happening,
[13:59] an x percentage of extinction happening, >> but he will he specifically goes out of
[14:01] >> but he will he specifically goes out of
[14:01] >> but he will he specifically goes out of his way to say, "I'm not going to put a
[14:02] his way to say, "I'm not going to put a
[14:02] his way to say, "I'm not going to put a percentage on it."
[14:03] percentage on it."
[14:04] percentage on it." >> Which I just think is the weirdest.
[14:05] >> Which I just think is the weirdest.
[14:05] >> Which I just think is the weirdest. >> No, but that that to be fair to be fair.
[14:07] >> No, but that that to be fair to be fair.
[14:08] >> No, but that that to be fair to be fair. >> No.
[14:08] >> No.
[14:08] >> No. >> Okay. No, nobody could be fair. Uh to be
[14:12] >> Okay. No, nobody could be fair. Uh to be
[14:12] >> Okay. No, nobody could be fair. Uh to be fair, the that's like it's not 100% like
[14:16] fair, the that's like it's not 100% like
[14:16] fair, the that's like it's not 100% like um uh like disingenuous or whatever like
[14:19] um uh like disingenuous or whatever like
[14:19] um uh like disingenuous or whatever like like he probably doesn't specifically
[14:21] like he probably doesn't specifically
[14:21] like he probably doesn't specifically agree that it's 10%. Or maybe he does
[14:23] agree that it's 10%. Or maybe he does
[14:23] agree that it's 10%. Or maybe he does and it's just too scary if he were to
[14:25] and it's just too scary if he were to
[14:25] and it's just too scary if he were to say
[14:25] say
[14:25] say >> let's just play binary search. I mean is
[14:27] >> let's just play binary search. I mean is
[14:27] >> let's just play binary search. I mean is it more or is it less?
[14:30] it more or is it less?
[14:30] it more or is it less? >> But but the whole thing is a madeup is a
[14:32] >> But but the whole thing is a madeup is a
[14:32] >> But but the whole thing is a madeup is a madeup concept that we just created. But
[14:34] madeup concept that we just created. But
[14:34] madeup concept that we just created. But like nobody can you can't quantify any
[14:36] like nobody can you can't quantify any
[14:36] like nobody can you can't quantify any of this.
[14:36] of this.
[14:36] of this. >> Well, but that's sort of Martin's point.
[14:38] >> Well, but that's sort of Martin's point.
[14:38] >> Well, but that's sort of Martin's point. >> Well, but but you just had an official
[14:39] >> Well, but but you just had an official
[14:39] >> Well, but but you just had an official position though. So the only official
[14:41] position though. So the only official
[14:41] position though. So the only official position would be like I think the only
[14:43] position would be like I think the only
[14:43] position would be like I think the only official position you could possibly get
[14:44] official position you could possibly get
[14:44] official position you could possibly get would be the PR version of this that
[14:46] would be the PR version of this that
[14:46] would be the PR version of this that would be the only thing that would be
[14:47] would be the only thing that would be
[14:47] would be the only thing that would be intellectually honest would be
[14:49] intellectually honest would be
[14:49] intellectually honest would be >> there are real risks with AI. There's
[14:51] >> there are real risks with AI. There's
[14:51] >> there are real risks with AI. There's incredibly positive uh benefits as well.
[14:53] incredibly positive uh benefits as well.
[14:53] incredibly positive uh benefits as well. We are working to mitigate the risks so
[14:55] We are working to mitigate the risks so
[14:55] We are working to mitigate the risks so they are as reduced as humanly possible.
[14:57] they are as reduced as humanly possible.
[14:57] they are as reduced as humanly possible. >> I don't think that's I don't think
[14:59] >> I don't think that's I don't think
[14:59] >> I don't think that's I don't think that's true. Listen so what do you think
[15:00] that's true. Listen so what do you think
[15:00] that's true. Listen so what do you think that
[15:01] that
[15:01] that >> we we've been through multiple epochs of
[15:02] >> we we've been through multiple epochs of
[15:02] >> we we've been through multiple epochs of technology. We've been through comput.
[15:04] technology. We've been through comput.
[15:04] technology. We've been through comput. We've been through the internet. We've
[15:05] We've been through the internet. We've
[15:05] We've been through the internet. We've been through the web. We've been through
[15:06] been through the web. We've been through
[15:06] been through the web. We've been through social networking. We had a discussion
[15:08] social networking. We had a discussion
[15:08] social networking. We had a discussion about the risks without talking about X-
[15:10] about the risks without talking about X-
[15:10] about the risks without talking about X- risk. Right? So for example, one thing
[15:11] risk. Right? So for example, one thing
[15:12] risk. Right? So for example, one thing you can say is we do not think the
[15:14] you can say is we do not think the
[15:14] you can say is we do not think the marginal risk for species extinction is
[15:16] marginal risk for species extinction is
[15:16] marginal risk for species extinction is different than it is.
[15:18] different than it is.
[15:18] different than it is. >> But what if they do think it is higher?
[15:19] >> But what if they do think it is higher?
[15:19] >> But what if they do think it is higher? >> Well, then we should
[15:22] >> Well, then we should
[15:22] >> Well, then we should OKAY. OKAY. YEAH. LIKE I THINK THE
[15:24] OKAY. OKAY. YEAH. LIKE I THINK THE
[15:24] OKAY. OKAY. YEAH. LIKE I THINK THE answer is they do think it's more than
[15:25] answer is they do think it's more than
[15:25] answer is they do think it's more than just the internet.
[15:26] just the internet.
[15:26] just the internet. >> One of two options. You believe that
[15:28] >> One of two options. You believe that
[15:28] >> One of two options. You believe that we're going to go extinct and we shut it
[15:29] we're going to go extinct and we shut it
[15:29] we're going to go extinct and we shut it all down. Like shut it down or
[15:31] all down. Like shut it down or
[15:31] all down. Like shut it down or >> they believe that this is just a chance
[15:32] >> they believe that this is just a chance
[15:32] >> they believe that this is just a chance that we got
[15:34] that we got
[15:34] that we got >> speaking.
[15:36] >> speaking.
[15:36] >> speaking. Dario thinks less but these cold
[15:39] Dario thinks less but these cold
[15:39] Dario thinks less but these cold warriors. No one like the Pentagon,
[15:42] warriors. No one like the Pentagon,
[15:42] warriors. No one like the Pentagon, >> you know, they ran a lot of simulations
[15:44] >> you know, they ran a lot of simulations
[15:44] >> you know, they ran a lot of simulations on the chances for nuclear war. Great
[15:46] on the chances for nuclear war. Great
[15:46] on the chances for nuclear war. Great movie war games about the whole thing
[15:47] movie war games about the whole thing
[15:47] movie war games about the whole thing and all of that. But the thing was it
[15:50] and all of that. But the thing was it
[15:50] and all of that. But the thing was it was it was nonzero.
[15:52] was it was nonzero.
[15:52] was it was nonzero. >> Yeah. And so once you said it was
[15:54] >> Yeah. And so once you said it was
[15:54] >> Yeah. And so once you said it was >> Can they say can they say non-zero then?
[15:56] >> Can they say can they say non-zero then?
[15:56] >> Can they say can they say non-zero then? Is that allowed?
[15:56] Is that allowed?
[15:56] Is that allowed? >> I think as soon as you think it's non
[15:58] >> I think as soon as you think it's non
[15:58] >> I think as soon as you think it's non zero.
[15:58] zero.
[15:58] zero. >> The problem is if you say non-zero that
[16:00] >> The problem is if you say non-zero that
[16:00] >> The problem is if you say non-zero that could be like you know the guy
[16:01] could be like you know the guy
[16:01] could be like you know the guy thinks 93%. But let's wait so just so
[16:04] thinks 93%. But let's wait so just so
[16:04] thinks 93%. But let's wait so just so we're all having a clean conversation.
[16:05] we're all having a clean conversation.
[16:05] we're all having a clean conversation. Let's talk about marginal risk here.
[16:07] Let's talk about marginal risk here.
[16:07] Let's talk about marginal risk here. We're not talking about absolute risk,
[16:09] We're not talking about absolute risk,
[16:09] We're not talking about absolute risk, right? Okay.
[16:10] right? Okay.
[16:10] right? Okay. >> It's just that I think if once you say
[16:13] >> It's just that I think if once you say
[16:13] >> It's just that I think if once you say it's non zero, the only answer of of
[16:15] it's non zero, the only answer of of
[16:15] it's non zero, the only answer of of like catastrophic,
[16:17] like catastrophic,
[16:17] like catastrophic, the only answer is you have to
[16:19] the only answer is you have to
[16:19] the only answer is you have to nationalize it.
[16:20] nationalize it.
[16:20] nationalize it. >> Yes. And so what he's trying to what the
[16:23] >> Yes. And so what he's trying to what the
[16:23] >> Yes. And so what he's trying to what the the labs in general that have that view
[16:25] the labs in general that have that view
[16:25] the labs in general that have that view are trying to threat is they want it to
[16:27] are trying to threat is they want it to
[16:27] are trying to threat is they want it to be nonzero as a license to do a certain
[16:30] be nonzero as a license to do a certain
[16:30] be nonzero as a license to do a certain set of things without the burden of just
[16:33] set of things without the burden of just
[16:33] set of things without the burden of just becoming a national
[16:34] becoming a national
[16:34] becoming a national >> well do do you have I actually don't
[16:36] >> well do do you have I actually don't
[16:36] >> well do do you have I actually don't know all of the presidents you hopefully
[16:38] know all of the presidents you hopefully
[16:38] know all of the presidents you hopefully do but but like there must be some
[16:40] do but but like there must be some
[16:40] do but but like there must be some things that are nonzero uh let's say x
[16:43] things that are nonzero uh let's say x
[16:43] things that are nonzero uh let's say x risk x risks that are not particularly
[16:45] risk x risks that are not particularly
[16:45] risk x risks that are not particularly nationalized but the but the regulatory
[16:47] nationalized but the but the regulatory
[16:47] nationalized but the but the regulatory environment around them is so heavy that
[16:49] environment around them is so heavy that
[16:49] environment around them is so heavy that it might as well nationalized because of
[16:51] it might as well nationalized because of
[16:51] it might as well nationalized because of the KYC requirements on like like I'm
[16:55] the KYC requirements on like like I'm
[16:55] the KYC requirements on like like I'm sure like to develop anthrax you have to
[16:57] sure like to develop anthrax you have to
[16:57] sure like to develop anthrax you have to go to a particular kind of lab.
[16:58] go to a particular kind of lab.
[16:58] go to a particular kind of lab. >> Well, BSL4 labs are but they're
[17:00] >> Well, BSL4 labs are but they're
[17:00] >> Well, BSL4 labs are but they're nationalized. They are national. Okay.
[17:01] nationalized. They are national. Okay.
[17:02] nationalized. They are national. Okay. But
[17:02] But
[17:02] But >> ethics tends to be nationalized. Yes.
[17:05] >> ethics tends to be nationalized. Yes.
[17:05] >> ethics tends to be nationalized. Yes. >> Like
[17:06] >> Like
[17:06] >> Like >> normal human safety not so much. Right.
[17:09] >> normal human safety not so much. Right.
[17:09] >> normal human safety not so much. Right. Like industry oversight that over the
[17:11] Like industry oversight that over the
[17:11] Like industry oversight that over the time becomes federal regulation.
[17:12] time becomes federal regulation.
[17:12] time becomes federal regulation. >> Right. And the progression which I think
[17:14] >> Right. And the progression which I think
[17:14] >> Right. And the progression which I think is just super important to this
[17:15] is just super important to this
[17:16] is just super important to this discussion has been since the post World
[17:20] discussion has been since the post World
[17:20] discussion has been since the post World War II era most industries that are
[17:23] War II era most industries that are
[17:23] War II era most industries that are critical to the infrastructure power and
[17:25] critical to the infrastructure power and
[17:26] critical to the infrastructure power and banking and healthcare. The trend has
[17:28] banking and healthcare. The trend has
[17:28] banking and healthcare. The trend has been to basically be nationalized. Yeah.
[17:31] been to basically be nationalized. Yeah.
[17:31] been to basically be nationalized. Yeah. By by just like what you like your
[17:33] By by just like what you like your
[17:33] By by just like what you like your examples of KYC and all the other stuff.
[17:35] examples of KYC and all the other stuff.
[17:35] examples of KYC and all the other stuff. The banks are for all practical purposes
[17:38] The banks are for all practical purposes
[17:38] The banks are for all practical purposes nationalized and the financial crisis.
[17:39] nationalized and the financial crisis.
[17:39] nationalized and the financial crisis. >> Okay. But but you said all practical
[17:41] >> Okay. But but you said all practical
[17:41] >> Okay. But but you said all practical purposes, right? They're literally not
[17:43] purposes, right? They're literally not
[17:43] purposes, right? They're literally not nationalized, right? So, so maybe that
[17:45] nationalized, right? So, so maybe that
[17:45] nationalized, right? So, so maybe that but this might be the intent of the labs
[17:47] but this might be the intent of the labs
[17:47] but this might be the intent of the labs is to look like JP Morgan or look like
[17:49] is to look like JP Morgan or look like
[17:49] is to look like JP Morgan or look like Verizon. And it's just like we're
[17:51] Verizon. And it's just like we're
[17:51] Verizon. And it's just like we're critical infrastructure. We get heavily
[17:53] critical infrastructure. We get heavily
[17:53] critical infrastructure. We get heavily regulated. It's not good for open source
[17:54] regulated. It's not good for open source
[17:54] regulated. It's not good for open source or at least frontier open source but
[17:56] or at least frontier open source but
[17:56] or at least frontier open source but it's a it is like it's it's like a
[17:59] it's a it is like it's it's like a
[17:59] it's a it is like it's it's like a plausible outcome for this industry good
[18:01] plausible outcome for this industry good
[18:01] plausible outcome for this industry good for innovation is the problem.
[18:03] for innovation is the problem.
[18:03] for innovation is the problem. >> I think even that's fine just don't use
[18:05] >> I think even that's fine just don't use
[18:05] >> I think even that's fine just don't use species extinction as your literally is
[18:08] species extinction as your literally is
[18:08] species extinction as your literally is like the difference between the things
[18:10] like the difference between the things
[18:10] like the difference between the things that are like stuck in the lab and let
[18:11] that are like stuck in the lab and let
[18:11] that are like stuck in the lab and let me just say something I actually think
[18:13] me just say something I actually think
[18:13] me just say something I actually think the labs are moving in the right
[18:15] the labs are moving in the right
[18:15] the labs are moving in the right direction. I actually think the actual
[18:17] direction. I actually think the actual
[18:18] direction. I actually think the actual statement was really good. You know, in
[18:19] statement was really good. You know, in
[18:20] statement was really good. You know, in my discussions with, you know,
[18:21] my discussions with, you know,
[18:21] my discussions with, you know, executives and leaders like they
[18:23] executives and leaders like they
[18:24] executives and leaders like they understand that they have like this
[18:25] understand that they have like this
[18:25] understand that they have like this tension and they're going to reconcile
[18:26] tension and they're going to reconcile
[18:26] tension and they're going to reconcile that. So, I actually am quite optimistic
[18:28] that. So, I actually am quite optimistic
[18:28] that. So, I actually am quite optimistic that the labs are both doing the right
[18:30] that the labs are both doing the right
[18:30] that the labs are both doing the right things and trying to do the right thing.
[18:31] things and trying to do the right thing.
[18:31] things and trying to do the right thing. I just think that grew so fa grew so
[18:33] I just think that grew so fa grew so
[18:33] I just think that grew so fa grew so fast and just trying to figure out how
[18:35] fast and just trying to figure out how
[18:35] fast and just trying to figure out how this machinery works. And I I think
[18:36] this machinery works. And I I think
[18:36] this machinery works. And I I think Sinoski really hit the nail on the head.
[18:39] Sinoski really hit the nail on the head.
[18:39] Sinoski really hit the nail on the head. >> The political process is its own is its
[18:41] >> The political process is its own is its
[18:41] >> The political process is its own is its own thing. And I don't think I don't
[18:42] own thing. And I don't think I don't
[18:42] own thing. And I don't think I don't know if it's naivity or hubris, but I
[18:44] know if it's naivity or hubris, but I
[18:44] know if it's naivity or hubris, but I just don't think that they kind of know
[18:46] just don't think that they kind of know
[18:46] just don't think that they kind of know how to navigate it.
[18:47] how to navigate it.
[18:47] how to navigate it. >> Well, I think the tech industry
[18:49] >> Well, I think the tech industry
[18:49] >> Well, I think the tech industry >> has literally over 100 years
[18:51] >> has literally over 100 years
[18:51] >> has literally over 100 years consistently relearn the lesson at each
[18:53] consistently relearn the lesson at each
[18:53] consistently relearn the lesson at each technology wave it that that we don't
[18:56] technology wave it that that we don't
[18:56] technology wave it that that we don't understand
[18:57] understand
[18:57] understand >> the regulatory climate and we can't
[18:59] >> the regulatory climate and we can't
[18:59] >> the regulatory climate and we can't navigate it. And even the companies like
[19:01] navigate it. And even the companies like
[19:01] navigate it. And even the companies like AT&T and IBM that were born out of
[19:04] AT&T and IBM that were born out of
[19:04] AT&T and IBM that were born out of basically being government monopolies
[19:05] basically being government monopolies
[19:05] basically being government monopolies from the start never figured out. Both
[19:07] from the start never figured out. Both
[19:07] from the start never figured out. Both got sued for antitrust. both got
[19:09] got sued for antitrust. both got
[19:09] got sued for antitrust. both got substantially and structurally changed
[19:11] substantially and structurally changed
[19:11] substantially and structurally changed as a result and they had hundreds of
[19:13] as a result and they had hundreds of
[19:13] as a result and they had hundreds of lawyers in the 1960s navigating them and
[19:16] lawyers in the 1960s navigating them and
[19:16] lawyers in the 1960s navigating them and that you know Microsoft came along like
[19:18] that you know Microsoft came along like
[19:18] that you know Microsoft came along like we were just like what? Yeah. Like we
[19:20] we were just like what? Yeah. Like we
[19:20] we were just like what? Yeah. Like we had no idea what was happening to us and
[19:23] had no idea what was happening to us and
[19:23] had no idea what was happening to us and there's you know there's Bill Gates
[19:25] there's you know there's Bill Gates
[19:25] there's you know there's Bill Gates playing golf with Bill Clinton and then
[19:26] playing golf with Bill Clinton and then
[19:26] playing golf with Bill Clinton and then we get slapped with an antitrust lawsuit
[19:28] we get slapped with an antitrust lawsuit
[19:28] we get slapped with an antitrust lawsuit from his administration and Bill was
[19:30] from his administration and Bill was
[19:30] from his administration and Bill was like I was playing golf here's the
[19:31] like I was playing golf here's the
[19:31] like I was playing golf here's the picture and that doesn't help and like
[19:33] picture and that doesn't help and like
[19:33] picture and that doesn't help and like isn't that what I was supposed to do was
[19:34] isn't that what I was supposed to do was
[19:34] isn't that what I was supposed to do was go and play golf. That's a shortened
[19:36] go and play golf. That's a shortened
[19:36] go and play golf. That's a shortened version of the whole thing. But but I I
[19:38] version of the whole thing. But but I I
[19:38] version of the whole thing. But but I I think and I think it's just
[19:40] think and I think it's just
[19:40] think and I think it's just >> it's I always use this example which is
[19:42] >> it's I always use this example which is
[19:42] >> it's I always use this example which is the Hollywood got together during the
[19:44] the Hollywood got together during the
[19:44] the Hollywood got together during the the Red Scare and all the and and
[19:46] the Red Scare and all the and and
[19:46] the Red Scare and all the and and censorship when they were worried about
[19:48] censorship when they were worried about
[19:48] censorship when they were worried about the government censoring movies for for
[19:50] the government censoring movies for for
[19:50] the government censoring movies for for for uh sexual content for adult themes
[19:54] for uh sexual content for adult themes
[19:54] for uh sexual content for adult themes and they all got together and and of
[19:56] and they all got together and and of
[19:56] and they all got together and and of course they were never going to be able
[19:56] course they were never going to be able
[19:56] course they were never going to be able to win in court if they tried to but
[19:59] to win in court if they tried to but
[19:59] to win in court if they tried to but they were threatening to do it and they
[20:01] they were threatening to do it and they
[20:01] they were threatening to do it and they got together and they formed the Motion
[20:02] got together and they formed the Motion
[20:02] got together and they formed the Motion Picture Association.
[20:03] Picture Association.
[20:03] Picture Association. >> Yeah. and movie ratings and all that and
[20:05] >> Yeah. and movie ratings and all that and
[20:06] >> Yeah. and movie ratings and all that and they police themselves.
[20:07] they police themselves.
[20:07] they police themselves. >> So how do you do you guys like that? Do
[20:08] >> So how do you do you guys like that? Do
[20:08] >> So how do you do you guys like that? Do you like the FINRA proposal?
[20:11] you like the FINRA proposal?
[20:11] you like the FINRA proposal? >> No, because FINRA is well that's that's
[20:13] >> No, because FINRA is well that's that's
[20:13] >> No, because FINRA is well that's that's as close to NPA as you can get with
[20:14] as close to NPA as you can get with
[20:14] as close to NPA as you can get with more.
[20:14] more.
[20:14] more. >> No, it's not because FINRA is going to
[20:16] >> No, it's not because FINRA is going to
[20:16] >> No, it's not because FINRA is going to FINRA becomes legislation which comes
[20:18] FINRA becomes legislation which comes
[20:18] FINRA becomes legislation which comes with direct oversight.
[20:20] with direct oversight.
[20:20] with direct oversight. >> Yeah, I think MPAA might not have as
[20:22] >> Yeah, I think MPAA might not have as
[20:22] >> Yeah, I think MPAA might not have as much consequence in like how society
[20:25] much consequence in like how society
[20:25] much consequence in like how society functions the first amendment.
[20:26] functions the first amendment.
[20:26] functions the first amendment. >> Yeah.
[20:29] >> Yeah.
[20:29] >> Yeah. Um
[20:31] Um
[20:31] Um >> did I win that? No, I I I I first of
[20:34] >> did I win that? No, I I I I first of
[20:34] >> did I win that? No, I I I I first of all, fantastic. But but I I still don't
[20:37] all, fantastic. But but I I still don't
[20:37] all, fantastic. But but I I still don't know if you want it. Like like I agree
[20:39] know if you want it. Like like I agree
[20:39] know if you want it. Like like I agree speech is really important, but like but
[20:41] speech is really important, but like but
[20:41] speech is really important, but like but like you know like the uh I just think
[20:44] like you know like the uh I just think
[20:44] like you know like the uh I just think >> there was no societal risk.
[20:46] >> there was no societal risk.
[20:46] >> there was no societal risk. >> I just think what's in our movies will
[20:47] >> I just think what's in our movies will
[20:47] >> I just think what's in our movies will be like we'll survive like on like a
[20:49] be like we'll survive like on like a
[20:49] be like we'll survive like on like a different continuum of
[20:51] different continuum of
[20:51] different continuum of >> all these every history is always
[20:52] >> all these every history is always
[20:52] >> all these every history is always relative and and at the time being a
[20:56] relative and and at the time being a
[20:56] relative and and at the time being a communist was a really bad thing and 30%
[20:58] communist was a really bad thing and 30%
[20:58] communist was a really bad thing and 30% of Hollywood got fired for for you know
[21:00] of Hollywood got fired for for you know
[21:00] of Hollywood got fired for for you know it was all this stuff. Yeah. So I it's
[21:02] it was all this stuff. Yeah. So I it's
[21:02] it was all this stuff. Yeah. So I it's always a risk to bring up something in
[21:04] always a risk to bring up something in
[21:04] always a risk to bring up something in that kind of way because it sounds so
[21:06] that kind of way because it sounds so
[21:06] that kind of way because it sounds so dumb. Like nobody thinks about movie
[21:07] dumb. Like nobody thinks about movie
[21:07] dumb. Like nobody thinks about movie ratings and that's because like actually
[21:09] ratings and that's because like actually
[21:09] ratings and that's because like actually they were ruled basically you can't
[21:11] they were ruled basically you can't
[21:11] they were ruled basically you can't constitutionally mandate them. So they
[21:13] constitutionally mandate them. So they
[21:13] constitutionally mandate them. So they couldn't regulate them on cable TV and
[21:14] couldn't regulate them on cable TV and
[21:14] couldn't regulate them on cable TV and so we got to grow up with HBO and all
[21:16] so we got to grow up with HBO and all
[21:16] so we got to grow up with HBO and all this other stuff.
[21:17] this other stuff.
[21:17] this other stuff. >> But the problem with FINRA is that is a
[21:20] >> But the problem with FINRA is that is a
[21:20] >> But the problem with FINRA is that is a perfect example of essentially
[21:22] perfect example of essentially
[21:22] perfect example of essentially nationalizing risk.
[21:23] nationalizing risk.
[21:23] nationalizing risk. >> Yes. Because even though the banks all
[21:25] >> Yes. Because even though the banks all
[21:25] >> Yes. Because even though the banks all pay money into it and that's how it's
[21:27] pay money into it and that's how it's
[21:27] pay money into it and that's how it's run and stuff, it's all mandated.
[21:29] run and stuff, it's all mandated.
[21:29] run and stuff, it's all mandated. >> Okay. Wait, sorry. Do you think we're
[21:30] >> Okay. Wait, sorry. Do you think we're
[21:30] >> Okay. Wait, sorry. Do you think we're going to end up with a situation that is
[21:32] going to end up with a situation that is
[21:32] going to end up with a situation that is better than FINRA? FINRA appears to be
[21:34] better than FINRA? FINRA appears to be
[21:34] better than FINRA? FINRA appears to be the best case scenario, but it is the
[21:36] the best case scenario, but it is the
[21:36] the best case scenario, but it is the best case scenario,
[21:37] best case scenario,
[21:37] best case scenario, >> but not even anymore. Like like I do
[21:39] >> but not even anymore. Like like I do
[21:39] >> but not even anymore. Like like I do think this this technology is in in the
[21:42] think this this technology is in in the
[21:42] think this this technology is in in the limit again so powerful uh relative to
[21:44] limit again so powerful uh relative to
[21:44] limit again so powerful uh relative to what it I mean for what it can deliver
[21:47] what it I mean for what it can deliver
[21:47] what it I mean for what it can deliver that it would be impossible for
[21:49] that it would be impossible for
[21:49] that it would be impossible for eventually Congress not caring about
[21:51] eventually Congress not caring about
[21:51] eventually Congress not caring about that. Like it's just like not possible.
[21:53] that. Like it's just like not possible.
[21:53] that. Like it's just like not possible. question. If it's eventually at what
[21:54] question. If it's eventually at what
[21:54] question. If it's eventually at what point
[21:55] point
[21:55] point >> if it's eventually making every
[21:56] >> if it's eventually making every
[21:56] >> if it's eventually making every recommendation in your healthcare
[21:57] recommendation in your healthcare
[21:58] recommendation in your healthcare process and and it's inside of your
[21:59] process and and it's inside of your
[21:59] process and and it's inside of your medical device as an open weights model
[22:01] medical device as an open weights model
[22:01] medical device as an open weights model and it's and it's on and every trading
[22:03] and it's and it's on and every trading
[22:03] and it's and it's on and every trading system for high frequency trading
[22:06] system for high frequency trading
[22:06] system for high frequency trading there's just and it's on an airplane
[22:08] there's just and it's on an airplane
[22:08] there's just and it's on an airplane like there's no chance that the
[22:09] like there's no chance that the
[22:09] like there's no chance that the government doesn't say we need something
[22:11] government doesn't say we need something
[22:11] government doesn't say we need something where FINRA actually is like the
[22:13] where FINRA actually is like the
[22:13] where FINRA actually is like the probably the best case scenario of what
[22:15] probably the best case scenario of what
[22:15] probably the best case scenario of what that looks like. Right.
[22:15] that looks like. Right.
[22:16] that looks like. Right. >> Yeah. And so now that's that's
[22:18] >> Yeah. And so now that's that's
[22:18] >> Yeah. And so now that's that's absolutely the most crucial point
[22:19] absolutely the most crucial point
[22:19] absolutely the most crucial point because if you all the people in the
[22:21] because if you all the people in the
[22:21] because if you all the people in the Senate now were in the Senate when when
[22:23] Senate now were in the Senate when when
[22:23] Senate now were in the Senate when when the communications act was passed and
[22:25] the communications act was passed and
[22:25] the communications act was passed and the li and liability was not passed
[22:28] the li and liability was not passed
[22:28] the li and liability was not passed through to ISPs and to social networks
[22:30] through to ISPs and to social networks
[22:30] through to ISPs and to social networks and they sat around the debate at the
[22:32] and they sat around the debate at the
[22:32] and they sat around the debate at the time was they would literally sat say to
[22:35] time was they would literally sat say to
[22:35] time was they would literally sat say to us the internet is so big how did we not
[22:38] us the internet is so big how did we not
[22:38] us the internet is so big how did we not have anything to do with it and that's
[22:39] have anything to do with it and that's
[22:39] have anything to do with it and that's why Al Gore gets a bunch of abuse for
[22:41] why Al Gore gets a bunch of abuse for
[22:41] why Al Gore gets a bunch of abuse for saying he created it right because he he
[22:43] saying he created it right because he he
[22:43] saying he created it right because he he was actually trying to get out in front
[22:44] was actually trying to get out in front
[22:44] was actually trying to get out in front of that and say no the government was
[22:46] of that and say no the government was
[22:46] of that and say no the government was instrumental in and it all backfired
[22:48] instrumental in and it all backfired
[22:48] instrumental in and it all backfired because it made it look like he was a
[22:50] because it made it look like he was a
[22:50] because it made it look like he was a crazy person. But but that it is
[22:52] crazy person. But but that it is
[22:52] crazy person. But but that it is absolutely the case that they felt like
[22:55] absolutely the case that they felt like
[22:55] absolutely the case that they felt like they did they missed their chance to be
[22:57] they did they missed their chance to be
[22:57] they did they missed their chance to be on top of the internet
[22:58] on top of the internet
[22:58] on top of the internet >> with Al Gore being on the positive side
[23:00] >> with Al Gore being on the positive side
[23:00] >> with Al Gore being on the positive side of innovation.
[23:01] of innovation.
[23:01] of innovation. >> That was their support. So who is the al
[23:03] >> That was their support. So who is the al
[23:03] >> That was their support. So who is the al Goro?
[23:04] Goro?
[23:04] Goro? >> There's no one. There's a couple well
[23:06] >> There's no one. There's a couple well
[23:06] >> There's no one. There's a couple well Besset seems to be that you know the
[23:09] Besset seems to be that you know the
[23:09] Besset seems to be that you know the defense people sort of want it private
[23:11] defense people sort of want it private
[23:11] defense people sort of want it private but not which is exactly where they were
[23:12] but not which is exactly where they were
[23:12] but not which is exactly where they were on the internet right
[23:13] on the internet right
[23:13] on the internet right >> and so it's very interesting
[23:15] >> and so it's very interesting
[23:15] >> and so it's very interesting >> that a lot of this is just this like
[23:18] >> that a lot of this is just this like
[23:18] >> that a lot of this is just this like Elizabeth Warren Senator Warren's tweets
[23:19] Elizabeth Warren Senator Warren's tweets
[23:20] Elizabeth Warren Senator Warren's tweets were all like we missed this for social
[23:22] were all like we missed this for social
[23:22] were all like we missed this for social network and it's like
[23:23] network and it's like
[23:23] network and it's like >> there's a lot of pent-up energy against
[23:25] >> there's a lot of pent-up energy against
[23:25] >> there's a lot of pent-up energy against tech that that could end up just all
[23:27] tech that that could end up just all
[23:27] tech that that could end up just all siphoning into into AI right now.
[23:29] siphoning into into AI right now.
[23:30] siphoning into into AI right now. >> That is exactly what it is. That's
[23:31] >> That is exactly what it is. That's
[23:31] >> That is exactly what it is. That's what's happening.
[23:32] what's happening.
[23:32] what's happening. >> Yeah. I think there's another problem
[23:33] >> Yeah. I think there's another problem
[23:33] >> Yeah. I think there's another problem which is we're all trying to predict
[23:35] which is we're all trying to predict
[23:35] which is we're all trying to predict what's bad which is not how we've
[23:36] what's bad which is not how we've
[23:36] what's bad which is not how we've normally done things. Like by this time
[23:38] normally done things. Like by this time
[23:38] normally done things. Like by this time on the internet we'd taken out like tens
[23:41] on the internet we'd taken out like tens
[23:41] on the internet we'd taken out like tens of billions of dollars of like economic
[23:44] of billions of dollars of like economic
[23:44] of billions of dollars of like economic well we've caused tens of billions of
[23:45] well we've caused tens of billions of
[23:45] well we've caused tens of billions of dollars of economic damage. We've we'd
[23:47] dollars of economic damage. We've we'd
[23:47] dollars of economic damage. We've we'd had words that took out 10% of the
[23:49] had words that took out 10% of the
[23:49] had words that took out 10% of the infrastructure.
[23:50] infrastructure.
[23:50] infrastructure. >> Yeah.
[23:50] >> Yeah.
[23:50] >> Yeah. >> The the internet infrastructure which
[23:51] >> The the internet infrastructure which
[23:51] >> The the internet infrastructure which was running critical infrastructure. We
[23:53] was running critical infrastructure. We
[23:53] was running critical infrastructure. We had hospitals go down.
[23:53] had hospitals go down.
[23:53] had hospitals go down. >> We really should have blocked the
[23:54] >> We really should have blocked the
[23:54] >> We really should have blocked the internet in like 97.
[23:57] internet in like 97.
[23:57] internet in like 97. >> How do we Yeah. Yeah. Okay. So, we we
[24:00] >> How do we Yeah. Yeah. Okay. So, we we
[24:00] >> How do we Yeah. Yeah. Okay. So, we we had I mean I I remember a time you like
[24:03] had I mean I I remember a time you like
[24:04] had I mean I I remember a time you like I remember when you would you would you
[24:05] I remember when you would you would you
[24:06] I remember when you would you would you would buy your CD of Windows 95 and by
[24:09] would buy your CD of Windows 95 and by
[24:09] would buy your CD of Windows 95 and by the time it was done installing
[24:12] the time it was done installing
[24:12] the time it was done installing >> you would have a a worm potentially.
[24:14] >> you would have a a worm potentially.
[24:14] >> you would have a a worm potentially. >> Way to go Stephen.
[24:15] >> Way to go Stephen.
[24:15] >> Way to go Stephen. >> No, no, this was it was this was the
[24:17] >> No, no, this was it was this was the
[24:17] >> No, no, this was it was this was the reality the reality of the PC until 2001
[24:21] reality the reality of the PC until 2001
[24:21] reality the reality of the PC until 2001 was you could not install a PC connected
[24:24] was you could not install a PC connected
[24:24] was you could not install a PC connected to the network without getting infected.
[24:26] to the network without getting infected.
[24:26] to the network without getting infected. But viruses were everywhere
[24:27] But viruses were everywhere
[24:27] But viruses were everywhere >> for the and and there were there were
[24:28] >> for the and and there were there were
[24:28] >> for the and and there were there were two two level two two rounds of
[24:31] two two level two two rounds of
[24:31] two two level two two rounds of congressional hearings. You had all of
[24:33] congressional hearings. You had all of
[24:33] congressional hearings. You had all of the
[24:34] the
[24:34] the >> that is actually okay. Okay. I'll grant
[24:35] >> that is actually okay. Okay. I'll grant
[24:35] >> that is actually okay. Okay. I'll grant you this is a very interesting point. Uh
[24:37] you this is a very interesting point. Uh
[24:37] you this is a very interesting point. Uh this type of zeitgeist in 94 would have
[24:41] this type of zeitgeist in 94 would have
[24:41] this type of zeitgeist in 94 would have we would probably not have the internet.
[24:42] we would probably not have the internet.
[24:42] we would probably not have the internet. >> Listen listen I mean the automotive the
[24:44] >> Listen listen I mean the automotive the
[24:44] >> Listen listen I mean the automotive the airline industry you always have these
[24:46] airline industry you always have these
[24:46] airline industry you always have these periods of kind of like teeth cutting
[24:47] periods of kind of like teeth cutting
[24:47] periods of kind of like teeth cutting where like you learn about the dangers
[24:49] where like you learn about the dangers
[24:49] where like you learn about the dangers and you learn about the technology and
[24:50] and you learn about the technology and
[24:50] and you learn about the technology and the internet. I mean we were on the
[24:51] the internet. I mean we were on the
[24:51] the internet. I mean we were on the ground floor of this. I mean
[24:53] ground floor of this. I mean
[24:53] ground floor of this. I mean >> things were down all the time. There's
[24:55] >> things were down all the time. There's
[24:55] >> things were down all the time. There's economic damage all the time. Like there
[24:56] economic damage all the time. Like there
[24:56] economic damage all the time. Like there was like novel. There was new viruses.
[24:58] was like novel. There was new viruses.
[24:58] was like novel. There was new viruses. There are new worms. They're all over
[24:59] There are new worms. They're all over
[24:59] There are new worms. They're all over the place.
[25:00] the place.
[25:00] the place. >> Y2A. Y2K was going to destroy.
[25:01] >> Y2A. Y2K was going to destroy.
[25:01] >> Y2A. Y2K was going to destroy. >> But here's here's the interesting thing
[25:02] >> But here's here's the interesting thing
[25:02] >> But here's here's the interesting thing about this.
[25:03] about this.
[25:03] about this. >> But what I just I just want to say like
[25:05] >> But what I just I just want to say like
[25:05] >> But what I just I just want to say like like so when we created policy and we
[25:08] like so when we created policy and we
[25:08] like so when we created policy and we did create a lot of policy, it was kind
[25:10] did create a lot of policy, it was kind
[25:10] did create a lot of policy, it was kind of like with a bunch of very specific
[25:11] of like with a bunch of very specific
[25:12] of like with a bunch of very specific data points on what you're trying to do
[25:13] data points on what you're trying to do
[25:13] data points on what you're trying to do and so you know that the policy actually
[25:15] and so you know that the policy actually
[25:15] and so you know that the policy actually fits the the fact pattern. And in this
[25:17] fits the the fact pattern. And in this
[25:17] fits the the fact pattern. And in this case, I mean, even when you were
[25:18] case, I mean, even when you were
[25:18] case, I mean, even when you were talking, you're like, well, what if you
[25:20] talking, you're like, well, what if you
[25:20] talking, you're like, well, what if you know, yada yada yada. It's very hard to
[25:22] know, yada yada yada. It's very hard to
[25:22] know, yada yada yada. It's very hard to predictive policy
[25:30] security risk. Great. And what is nice
[25:30] security risk. Great. And what is nice about the discourse that is actually
[25:31] about the discourse that is actually
[25:31] about the discourse that is actually starting to evolve around that. Like you
[25:33] starting to evolve around that. Like you
[25:33] starting to evolve around that. Like you actually hear people from the lab saying
[25:34] actually hear people from the lab saying
[25:34] actually hear people from the lab saying novel cyber security risk. This is an
[25:36] novel cyber security risk. This is an
[25:36] novel cyber security risk. This is an engineering problem. We're talking about
[25:37] engineering problem. We're talking about
[25:37] engineering problem. We're talking about that. So the more this becomes concrete
[25:39] that. So the more this becomes concrete
[25:40] that. So the more this becomes concrete around real identified risks, I think we
[25:42] around real identified risks, I think we
[25:42] around real identified risks, I think we can all fall in line. But that's not
[25:44] can all fall in line. But that's not
[25:44] can all fall in line. But that's not been the discussion to date. That's a
[25:45] been the discussion to date. That's a
[25:45] been the discussion to date. That's a very
[25:46] very
[25:46] very >> perfect perfect point because if you
[25:47] >> perfect perfect point because if you
[25:47] >> perfect perfect point because if you look in in in 1986
[25:50] look in in in 1986
[25:50] look in in in 1986 the computer crime and fraud act got
[25:52] the computer crime and fraud act got
[25:52] the computer crime and fraud act got signed. It was out of a very very
[25:55] signed. It was out of a very very
[25:55] signed. It was out of a very very specific scenario which was a two groups
[25:57] specific scenario which was a two groups
[25:58] specific scenario which was a two groups of hackers broke into GTE telem.
[26:04] And
[26:04] And >> sure, well, it wasn't you. It
[26:06] >> sure, well, it wasn't you. It
[26:06] >> sure, well, it wasn't you. It >> was the other guy.
[26:07] >> was the other guy.
[26:07] >> was the other guy. >> Oh, he wor he worked he worked at Cisco
[26:09] >> Oh, he wor he worked he worked at Cisco
[26:09] >> Oh, he wor he worked he worked at Cisco later. And um and
[26:12] later. And um and
[26:12] later. And um and >> and the problem was that was 1983 and
[26:14] >> and the problem was that was 1983 and
[26:14] >> and the problem was that was 1983 and there was no crime
[26:16] there was no crime
[26:16] there was no crime >> and it it took two and a half years
[26:18] >> and it it took two and a half years
[26:18] >> and it it took two and a half years >> for the bill to make it through. That
[26:20] >> for the bill to make it through. That
[26:20] >> for the bill to make it through. That made it very very specific. And that's
[26:22] made it very very specific. And that's
[26:22] made it very very specific. And that's the law that says you can't you can't
[26:25] the law that says you can't you can't
[26:25] the law that says you can't you can't access unauthorized computer systems. So
[26:27] access unauthorized computer systems. So
[26:27] access unauthorized computer systems. So I think we'd all agree that that you we
[26:29] I think we'd all agree that that you we
[26:29] I think we'd all agree that that you we probably you know we probably have like
[26:31] probably you know we probably have like
[26:31] probably you know we probably have like 90% of laws already in the applied layer
[26:33] 90% of laws already in the applied layer
[26:33] 90% of laws already in the applied layer like you can't hack systems etc. Do you
[26:35] like you can't hack systems etc. Do you
[26:35] like you can't hack systems etc. Do you think there's anything that should be
[26:37] think there's anything that should be
[26:37] think there's anything that should be from a liability standpoint in the model
[26:38] from a liability standpoint in the model
[26:38] from a liability standpoint in the model layer? Which of course then
[26:40] layer? Which of course then
[26:40] layer? Which of course then >> how are you making a tech legal?
[26:43] >> how are you making a tech legal?
[26:43] >> how are you making a tech legal? >> No, the the legal everything my read of
[26:46] >> No, the the legal everything my read of
[26:46] >> No, the the legal everything my read of of hugging face open AI for they're all
[26:49] of hugging face open AI for they're all
[26:49] of hugging face open AI for they're all absolutely blatant computer crime acts
[26:52] absolutely blatant computer crime acts
[26:52] absolutely blatant computer crime acts except for the fact that there's they
[26:54] except for the fact that there's they
[26:54] except for the fact that there's they carved out an amendments later that if
[26:56] carved out an amendments later that if
[26:56] carved out an amendments later that if you're a white hat, it's not illegal
[26:58] you're a white hat, it's not illegal
[26:58] you're a white hat, it's not illegal anymore. And that was because people
[27:01] anymore. And that was because people
[27:01] anymore. And that was because people kept making mistakes
[27:02] kept making mistakes
[27:02] kept making mistakes >> and and they didn't want to go around
[27:04] >> and and they didn't want to go around
[27:04] >> and and they didn't want to go around arresting everybody who was actually
[27:05] arresting everybody who was actually
[27:06] arresting everybody who was actually trying to make the system better because
[27:07] trying to make the system better because
[27:07] trying to make the system better because they messed something up. And so the
[27:09] they messed something up. And so the
[27:09] they messed something up. And so the Justice Department wrote a memo that
[27:10] Justice Department wrote a memo that
[27:10] Justice Department wrote a memo that said, "We're not going to prosecute for
[27:12] said, "We're not going to prosecute for
[27:12] said, "We're not going to prosecute for for this. And we're also not going to
[27:13] for this. And we're also not going to
[27:14] for this. And we're also not going to prosecute if you just like violate the
[27:16] prosecute if you just like violate the
[27:16] prosecute if you just like violate the terms of use, right,
[27:17] terms of use, right,
[27:17] terms of use, right, >> of a system versus actually try to
[27:19] >> of a system versus actually try to
[27:19] >> of a system versus actually try to breach it."
[27:20] breach it."
[27:20] breach it." >> And and so I think the law is ample for
[27:23] >> And and so I think the law is ample for
[27:23] >> And and so I think the law is ample for this for the scenario. And and it was
[27:25] this for the scenario. And and it was
[27:25] this for the scenario. And and it was all written this GT Telemet was used by
[27:26] all written this GT Telemet was used by
[27:26] all written this GT Telemet was used by NASA and Liverour and all the labs. And
[27:29] NASA and Liverour and all the labs. And
[27:29] NASA and Liverour and all the labs. And so that's why it was it caught the
[27:31] so that's why it was it caught the
[27:31] so that's why it was it caught the attention of DC because it was federal
[27:32] attention of DC because it was federal
[27:32] attention of DC because it was federal systems that were being broken into. And
[27:35] systems that were being broken into. And
[27:35] systems that were being broken into. And the problem we have now is is this is
[27:37] the problem we have now is is this is
[27:37] the problem we have now is is this is this rift between the labs and the
[27:40] this rift between the labs and the
[27:40] this rift between the labs and the security community that keeps looking at
[27:41] security community that keeps looking at
[27:42] security community that keeps looking at all their postmortems and coming to two
[27:43] all their postmortems and coming to two
[27:43] all their postmortems and coming to two conclusions. Sloppy
[27:46] conclusions. Sloppy
[27:46] conclusions. Sloppy >> and and you're not complete in what
[27:48] >> and and you're not complete in what
[27:48] >> and and you're not complete in what you're telling us happened.
[27:50] you're telling us happened.
[27:50] you're telling us happened. >> And so of course the CBE process came
[27:52] >> And so of course the CBE process came
[27:52] >> And so of course the CBE process came about in the 1980s. the the computer
[27:54] about in the 1980s. the the computer
[27:54] about in the 1980s. the the computer virus and vulnerability reporting stuff
[27:56] virus and vulnerability reporting stuff
[27:56] virus and vulnerability reporting stuff out of CMU and and for years they worked
[27:59] out of CMU and and for years they worked
[27:59] out of CMU and and for years they worked on very structured reporting with
[28:01] on very structured reporting with
[28:01] on very structured reporting with obligations and how to and there for
[28:03] obligations and how to and there for
[28:03] obligations and how to and there for some reason they're not using any of
[28:04] some reason they're not using any of
[28:04] some reason they're not using any of that to do this reporting
[28:07] that to do this reporting
[28:07] that to do this reporting >> and so there is this very basic stuff
[28:09] >> and so there is this very basic stuff
[28:09] >> and so there is this very basic stuff that I look at and say well until
[28:11] that I look at and say well until
[28:11] that I look at and say well until they're doing that they really should
[28:13] they're doing that they really should
[28:13] they're doing that they really should stop talking like they shouldn't do a
[28:14] stop talking like they shouldn't do a
[28:14] stop talking like they shouldn't do a postmortem on a breach
[28:16] postmortem on a breach
[28:16] postmortem on a breach >> that looks like an intern wrote it and
[28:18] >> that looks like an intern wrote it and
[28:18] >> that looks like an intern wrote it and it's not a postmortem it's this
[28:20] it's not a postmortem it's this
[28:20] it's not a postmortem it's this selective memory it looks like
[28:22] selective memory it looks like
[28:22] selective memory it looks like >> exactly the kind of postmortem you do
[28:24] >> exactly the kind of postmortem you do
[28:24] >> exactly the kind of postmortem you do when you hire outside lawyers to
[28:25] when you hire outside lawyers to
[28:25] when you hire outside lawyers to investigate some random thing and you
[28:28] investigate some random thing and you
[28:28] investigate some random thing and you only give them certain stuff
[28:29] only give them certain stuff
[28:30] only give them certain stuff >> because you don't have to give the
[28:31] >> because you don't have to give the
[28:31] >> because you don't have to give the lawyers you hire all the information
[28:32] lawyers you hire all the information
[28:32] lawyers you hire all the information about what happened
[28:33] about what happened
[28:34] about what happened >> like where's all the Slack messages
[28:35] >> like where's all the Slack messages
[28:35] >> like where's all the Slack messages where's the actual details of of what
[28:38] where's the actual details of of what
[28:38] where's the actual details of of what happened can I can I just interject an
[28:40] happened can I can I just interject an
[28:40] happened can I can I just interject an annoying aside so um which is which
[28:44] annoying aside so um which is which
[28:44] annoying aside so um which is which is no no this is good which is very in
[28:45] is no no this is good which is very in
[28:45] is no no this is good which is very in line with this which is um I've been in
[28:47] line with this which is um I've been in
[28:47] line with this which is um I've been in the security like the actual cyber
[28:49] the security like the actual cyber
[28:49] the security like the actual cyber security community for a long time and
[28:51] security community for a long time and
[28:51] security community for a long time and it is you know it's always been one of
[28:53] it is you know it's always been one of
[28:53] it is you know it's always been one of these things where like they just don't
[28:54] these things where like they just don't
[28:54] these things where like they just don't like practical solutions. So even if you
[28:56] like practical solutions. So even if you
[28:56] like practical solutions. So even if you build like you know a secure system like
[28:58] build like you know a secure system like
[28:58] build like you know a secure system like well what if somebody you know shows up
[29:00] well what if somebody you know shows up
[29:00] well what if somebody you know shows up like you know like
[29:01] like you know like
[29:01] like you know like >> you know can Russell Crow can like break
[29:03] >> you know can Russell Crow can like break
[29:03] >> you know can Russell Crow can like break the encryption or something like that.
[29:05] the encryption or something like that.
[29:05] the encryption or something like that. >> It's Mission Impossible. Some
[29:06] >> It's Mission Impossible. Some
[29:06] >> It's Mission Impossible. Some >> crew there's always like these kind of
[29:08] >> crew there's always like these kind of
[29:08] >> crew there's always like these kind of like whatever. So my favorite thing that
[29:09] like whatever. So my favorite thing that
[29:09] like whatever. So my favorite thing that happened recently was like Gnome Brown
[29:11] happened recently was like Gnome Brown
[29:11] happened recently was like Gnome Brown some podcast we've all said stupid stuff
[29:13] some podcast we've all said stupid stuff
[29:13] some podcast we've all said stupid stuff in podcast. I've already said
[29:13] in podcast. I've already said
[29:14] in podcast. I've already said >> oh you didn't like this one.
[29:15] >> oh you didn't like this one.
[29:15] >> oh you didn't like this one. >> NO IT WAS GREAT.
[29:16] >> NO IT WAS GREAT.
[29:16] >> NO IT WAS GREAT. >> I THOUGHT IT was fun.
[29:17] >> I THOUGHT IT was fun.
[29:17] >> I THOUGHT IT was fun. >> No it was fantastic. No I'm set I'm
[29:19] >> No it was fantastic. No I'm set I'm
[29:19] >> No it was fantastic. No I'm set I'm setting it up. Yeah.
[29:20] setting it up. Yeah.
[29:20] setting it up. Yeah. >> So, so no Brown was like listen uh you
[29:23] >> So, so no Brown was like listen uh you
[29:23] >> So, so no Brown was like listen uh you don't know what a super intelligence
[29:24] don't know what a super intelligence
[29:24] don't know what a super intelligence could do. It could maybe use like the
[29:26] could do. It could maybe use like the
[29:26] could do. It could maybe use like the heat of a uh CPU to excfiltrate itself
[29:29] heat of a uh CPU to excfiltrate itself
[29:29] heat of a uh CPU to excfiltrate itself to another computer which this brought
[29:31] to another computer which this brought
[29:31] to another computer which this brought me back to my
[29:33] me back to my
[29:33] me back to my now I'm very comfortable right now I
[29:36] now I'm very comfortable right now I
[29:36] now I'm very comfortable right now I could have endless pointless discussions
[29:37] could have endless pointless discussions
[29:37] could have endless pointless discussions on this but what is interesting is you
[29:39] on this but what is interesting is you
[29:39] on this but what is interesting is you basically have the X-risk people saying
[29:41] basically have the X-risk people saying
[29:41] basically have the X-risk people saying something they thought was plausible and
[29:43] something they thought was plausible and
[29:43] something they thought was plausible and then you have like the security people
[29:44] then you have like the security people
[29:44] then you have like the security people having this kind of endless discussion
[29:46] having this kind of endless discussion
[29:46] having this kind of endless discussion and so I think at some level
[29:48] and so I think at some level
[29:48] and so I think at some level >> now these communities are being bridged
[29:50] >> now these communities are being bridged
[29:50] >> now these communities are being bridged which is like you know I actually think
[29:52] which is like you know I actually think
[29:52] which is like you know I actually think that was a very reasonable thing for non
[29:53] that was a very reasonable thing for non
[29:53] that was a very reasonable thing for non brand to
[29:54] brand to
[29:54] brand to I think you can pick holes in it, but we
[29:56] I think you can pick holes in it, but we
[29:56] I think you can pick holes in it, but we all say weird stuff on podcast. I
[29:58] all say weird stuff on podcast. I
[29:58] all say weird stuff on podcast. I actually think Xfill risk is real. I
[29:59] actually think Xfill risk is real. I
[29:59] actually think Xfill risk is real. I mean, I worked in secure um computing
[30:02] mean, I worked in secure um computing
[30:02] mean, I worked in secure um computing environments that were highly classified
[30:04] environments that were highly classified
[30:04] environments that were highly classified that you would the covert channels were
[30:05] that you would the covert channels were
[30:05] that you would the covert channels were unbelievable. you actually I mean so
[30:07] unbelievable. you actually I mean so
[30:07] unbelievable. you actually I mean so these are very real comments but like we
[30:09] these are very real comments but like we
[30:09] these are very real comments but like we actually have a real discourse and it
[30:11] actually have a real discourse and it
[30:11] actually have a real discourse and it was the first time I saw really hardcore
[30:12] was the first time I saw really hardcore
[30:12] was the first time I saw really hardcore systems people having pretty con like I
[30:15] systems people having pretty con like I
[30:15] systems people having pretty con like I would say constructive
[30:16] would say constructive
[30:16] would say constructive >> calculating the bit rate and
[30:18] >> calculating the bit rate and
[30:18] >> calculating the bit rate and >> yeah it was great but it was a
[30:19] >> yeah it was great but it was a
[30:19] >> yeah it was great but it was a constructive discussion and like you
[30:20] constructive discussion and like you
[30:20] constructive discussion and like you know you had like the kind of the the
[30:21] know you had like the kind of the the
[30:22] know you had like the kind of the the the typical extras people were engaging
[30:24] the typical extras people were engaging
[30:24] the typical extras people were engaging of course it was it was Twitter so
[30:25] of course it was it was Twitter so
[30:25] of course it was it was Twitter so there's like a lot of name calling this
[30:26] there's like a lot of name calling this
[30:26] there's like a lot of name calling this and that but it was actually I felt like
[30:27] and that but it was actually I felt like
[30:27] and that but it was actually I felt like a real discussion for the first time so
[30:29] a real discussion for the first time so
[30:29] a real discussion for the first time so I hope we see more of this I hope we see
[30:31] I hope we see more of this I hope we see
[30:31] I hope we see more of this I hope we see more
[30:32] more
[30:32] more >> you know
[30:33] >> you know
[30:33] >> you know >> you know cyber related things I think
[30:35] >> you know cyber related things I think
[30:35] >> you know cyber related things I think the lab should talk more about it. I
[30:36] the lab should talk more about it. I
[30:36] the lab should talk more about it. I think it'll engage the community and I
[30:38] think it'll engage the community and I
[30:38] think it'll engage the community and I think once that happens, we can
[30:39] think once that happens, we can
[30:39] think once that happens, we can >> actually Yeah, Greg's been out there a
[30:41] >> actually Yeah, Greg's been out there a
[30:41] >> actually Yeah, Greg's been out there a lot more on this topic. Um, which has
[30:43] lot more on this topic. Um, which has
[30:43] lot more on this topic. Um, which has been which has been good. But I think
[30:44] been which has been good. But I think
[30:44] been which has been good. But I think the takeaway is Nam Brown should be
[30:46] the takeaway is Nam Brown should be
[30:46] the takeaway is Nam Brown should be doing more brainstorms on podcasts.
[30:47] doing more brainstorms on podcasts.
[30:47] doing more brainstorms on podcasts. >> I thought it was actually just it opened
[30:50] >> I thought it was actually just it opened
[30:50] >> I thought it was actually just it opened people's eyes to the fact that actually
[30:52] people's eyes to the fact that actually
[30:52] people's eyes to the fact that actually there's a lot of risks in security that
[30:55] there's a lot of risks in security that
[30:55] there's a lot of risks in security that most people don't understand. Yeah. And
[30:57] most people don't understand. Yeah. And
[30:57] most people don't understand. Yeah. And so that means that there's more stuff
[30:58] so that means that there's more stuff
[30:58] so that means that there's more stuff that that like you can't make baseline
[31:01] that that like you can't make baseline
[31:01] that that like you can't make baseline risk, you know, like how does your
[31:03] risk, you know, like how does your
[31:03] risk, you know, like how does your authentication layer work? You can't
[31:04] authentication layer work? You can't
[31:04] authentication layer work? You can't just wave your hand and say that should
[31:06] just wave your hand and say that should
[31:06] just wave your hand and say that should go away and then bring up oh but you
[31:07] go away and then bring up oh but you
[31:07] go away and then bring up oh but you know space aliens can invade and that's
[31:09] know space aliens can invade and that's
[31:09] know space aliens can invade and that's a little bit of what was going on that I
[31:11] a little bit of what was going on that I
[31:11] a little bit of what was going on that I felt uncomfortable with. But like what
[31:13] felt uncomfortable with. But like what
[31:13] felt uncomfortable with. But like what what do you mean like it was sort of
[31:14] what do you mean like it was sort of
[31:14] what do you mean like it was sort of like but you know there's also this risk
[31:17] like but you know there's also this risk
[31:17] like but you know there's also this risk and is how I viewed that
[31:18] and is how I viewed that
[31:18] and is how I viewed that >> of the heat thing. But but at least
[31:20] >> of the heat thing. But but at least
[31:20] >> of the heat thing. But but at least we're now we're now in a in a domain
[31:21] we're now we're now in a in a domain
[31:21] we're now we're now in a in a domain we're comfortable like I can talk about
[31:23] we're comfortable like I can talk about
[31:23] we're comfortable like I can talk about entropy and like we can actually have a
[31:25] entropy and like we can actually have a
[31:25] entropy and like we can actually have a concrete discussion that's not oh well
[31:27] concrete discussion that's not oh well
[31:27] concrete discussion that's not oh well it's super powerful like at least we've
[31:28] it's super powerful like at least we've
[31:28] it's super powerful like at least we've reduced it to like the laws of physics
[31:30] reduced it to like the laws of physics
[31:30] reduced it to like the laws of physics and the laws of system. And and most
[31:31] and the laws of system. And and most
[31:31] and the laws of system. And and most people did, I would say most people had
[31:33] people did, I would say most people had
[31:34] people did, I would say most people had no idea that that kind of risk was real.
[31:36] no idea that that kind of risk was real.
[31:36] no idea that that kind of risk was real. I mean, like when I'm walking around the
[31:37] I mean, like when I'm walking around the
[31:37] I mean, like when I'm walking around the Persian missiles, I actually had to test
[31:39] Persian missiles, I actually had to test
[31:39] Persian missiles, I actually had to test the graphics cards and PCs because a DoD
[31:42] the graphics cards and PCs because a DoD
[31:42] the graphics cards and PCs because a DoD requirement is that the screen memory
[31:44] requirement is that the screen memory
[31:44] requirement is that the screen memory not be sustained when you pulled the
[31:46] not be sustained when you pulled the
[31:46] not be sustained when you pulled the power, but not for for zero time. Like
[31:48] power, but not for for zero time. Like
[31:48] power, but not for for zero time. Like it the minute that the power went off,
[31:50] it the minute that the power went off,
[31:50] it the minute that the power went off, the the memory image had to go. And if
[31:51] the the memory image had to go. And if
[31:52] the the memory image had to go. And if there was like a 3se secondond delay,
[31:53] there was like a 3se secondond delay,
[31:53] there was like a 3se secondond delay, >> Yeah.
[31:53] >> Yeah.
[31:54] >> Yeah. >> Oh, you you you had that could be read.
[31:56] >> Oh, you you you had that could be read.
[31:56] >> Oh, you you you had that could be read. Oh, see, see, we we had people that came
[31:58] Oh, see, see, we we had people that came
[31:58] Oh, see, see, we we had people that came into our offices and would literally
[32:00] into our offices and would literally
[32:00] into our offices and would literally measure the distance of the monitors to
[32:01] measure the distance of the monitors to
[32:02] measure the distance of the monitors to each other because of Tempest attacks,
[32:03] each other because of Tempest attacks,
[32:03] each other because of Tempest attacks, which is 100% a way to use
[32:05] which is 100% a way to use
[32:05] which is 100% a way to use electromagnetic radiation to leak
[32:07] electromagnetic radiation to leak
[32:07] electromagnetic radiation to leak information. I've seen the same thing
[32:09] information. I've seen the same thing
[32:09] information. I've seen the same thing with spread spectrum from the BIOS. I've
[32:10] with spread spectrum from the BIOS. I've
[32:10] with spread spectrum from the BIOS. I've seen it from I've seen it from from
[32:12] seen it from I've seen it from from
[32:12] seen it from I've seen it from from audio speakers like we had to remove the
[32:14] audio speakers like we had to remove the
[32:14] audio speakers like we had to remove the speakers out because it's a very high
[32:15] speakers out because it's a very high
[32:15] speakers out because it's a very high channel.
[32:15] channel.
[32:15] channel. >> Our building our building our building
[32:18] >> Our building our building our building
[32:18] >> Our building our building our building had had just music speakers aimed at the
[32:20] had had just music speakers aimed at the
[32:20] had had just music speakers aimed at the windows
[32:21] windows
[32:21] windows >> just to produce interference.
[32:23] >> just to produce interference.
[32:23] >> just to produce interference. >> You need to go run safety at one of
[32:24] >> You need to go run safety at one of
[32:24] >> You need to go run safety at one of these labs. This is like I've never seen
[32:26] these labs. This is like I've never seen
[32:26] these labs. This is like I've never seen you at more excited than heat, you know,
[32:29] you at more excited than heat, you know,
[32:29] you at more excited than heat, you know, based communication.
[32:30] based communication.
[32:30] based communication. >> Can I tell you the craziest covert
[32:31] >> Can I tell you the craziest covert
[32:31] >> Can I tell you the craziest covert channel I've ever seen? So it turn
[32:33] channel I've ever seen? So it turn
[32:33] channel I've ever seen? So it turn remember the old CR I know this is like
[32:34] remember the old CR I know this is like
[32:34] remember the old CR I know this is like one of my
[32:35] one of my
[32:35] one of my >> So I'm glad that someone's older than
[32:36] >> So I'm glad that someone's older than
[32:36] >> So I'm glad that someone's older than me. Not really, but acting it.
[32:38] me. Not really, but acting it.
[32:38] me. Not really, but acting it. >> So remember the old CRT? So it turns out
[32:41] >> So remember the old CRT? So it turns out
[32:41] >> So remember the old CRT? So it turns out if like let's say it's night and you're
[32:42] if like let's say it's night and you're
[32:42] if like let's say it's night and you're you're using a CRT terminal in your
[32:44] you're using a CRT terminal in your
[32:44] you're using a CRT terminal in your room, the lightest thing in the room is
[32:46] room, the lightest thing in the room is
[32:46] room, the lightest thing in the room is actually the pixel that the raster beam
[32:47] actually the pixel that the raster beam
[32:47] actually the pixel that the raster beam is on. So most people think it's like
[32:49] is on. So most people think it's like
[32:49] is on. So most people think it's like the glow of the monitor, but it's
[32:50] the glow of the monitor, but it's
[32:50] the glow of the monitor, but it's actually that given pixel. Mhm.
[32:52] actually that given pixel. Mhm.
[32:52] actually that given pixel. Mhm. >> So, somebody figured out that like let's
[32:54] >> So, somebody figured out that like let's
[32:54] >> So, somebody figured out that like let's say you're in like, you know, a hotel
[32:56] say you're in like, you know, a hotel
[32:56] say you're in like, you know, a hotel room and you're on your computer. If you
[32:58] room and you're on your computer. If you
[32:58] room and you're on your computer. If you have something that can sample the color
[33:00] have something that can sample the color
[33:00] have something that can sample the color of the window, you can reconstruct the
[33:02] of the window, you can reconstruct the
[33:02] of the window, you can reconstruct the screen.
[33:02] screen.
[33:02] screen. >> Oh, that's crazy,
[33:03] >> Oh, that's crazy,
[33:03] >> Oh, that's crazy, >> right? You just do it at the same hertz
[33:04] >> right? You just do it at the same hertz
[33:04] >> right? You just do it at the same hertz that the raster beam is moving. Then
[33:06] that the raster beam is moving. Then
[33:06] that the raster beam is moving. Then somebody else figured out if you can
[33:07] somebody else figured out if you can
[33:07] somebody else figured out if you can subvert three pixels, you can use those,
[33:09] subvert three pixels, you can use those,
[33:10] subvert three pixels, you can use those, you know, because it just looks like bad
[33:11] you know, because it just looks like bad
[33:11] you know, because it just looks like bad pixels. You can use those to basically
[33:12] pixels. You can use those to basically
[33:12] pixels. You can use those to basically send a message. So like you could
[33:13] send a message. So like you could
[33:14] send a message. So like you could literally like sit out in whatever like
[33:15] literally like sit out in whatever like
[33:16] literally like sit out in whatever like a you sample the message and you could
[33:19] a you sample the message and you could
[33:19] a you sample the message and you could it was a relatively high bandwidth
[33:20] it was a relatively high bandwidth
[33:20] it was a relatively high bandwidth oneway communication and so like what no
[33:23] oneway communication and so like what no
[33:23] oneway communication and so like what no brown was saying like maybe heat is not
[33:24] brown was saying like maybe heat is not
[33:24] brown was saying like maybe heat is not the way to do it but that level of
[33:25] the way to do it but that level of
[33:25] the way to do it but that level of sophistication is actually real that's a
[33:28] sophistication is actually real that's a
[33:28] sophistication is actually real that's a that's a thing
[33:29] that's a thing
[33:29] that's a thing >> my I when I was at the missile factory
[33:31] >> my I when I was at the missile factory
[33:31] >> my I when I was at the missile factory like I had to lock my keyboard up I
[33:33] like I had to lock my keyboard up I
[33:33] like I had to lock my keyboard up I that's really an impolite word but
[33:34] that's really an impolite word but
[33:34] that's really an impolite word but that's what we call something I had to
[33:36] that's what we call something I had to
[33:36] that's what we call something I had to lock my keyboard up at night because
[33:37] lock my keyboard up at night because
[33:37] lock my keyboard up at night because they didn't want the custodians who
[33:39] they didn't want the custodians who
[33:39] they didn't want the custodians who didn't have clearance walking by and and
[33:41] didn't have clearance walking by and and
[33:42] didn't have clearance walking by and and just noticing which keys were dirtier or
[33:44] just noticing which keys were dirtier or
[33:44] just noticing which keys were dirtier or cleaner. And I once left it out and
[33:46] cleaner. And I once left it out and
[33:46] cleaner. And I once left it out and there's like a a note from security,
[33:48] there's like a a note from security,
[33:48] there's like a a note from security, >> you know, telling me to report to
[33:50] >> you know, telling me to report to
[33:50] >> you know, telling me to report to security and pick up my keyboard.
[33:52] security and pick up my keyboard.
[33:52] security and pick up my keyboard. >> Like the the guard who walked the
[33:53] >> Like the the guard who walked the
[33:53] >> Like the the guard who walked the hallways just took the keyboard that
[33:55] hallways just took the keyboard that
[33:55] hallways just took the keyboard that night off my machine. And that's like
[33:56] night off my machine. And that's like
[33:56] night off my machine. And that's like basically I was not cleared. I was just
[33:58] basically I was not cleared. I was just
[33:58] basically I was not cleared. I was just that sensitive, you know? I was like
[33:59] that sensitive, you know? I was like
[34:00] that sensitive, you know? I was like nothing. I was an intern.
[34:01] nothing. I was an intern.
[34:01] nothing. I was an intern. >> The big problem with this conversation
[34:02] >> The big problem with this conversation
[34:02] >> The big problem with this conversation is now this is all in the training data
[34:04] is now this is all in the training data
[34:04] is now this is all in the training data of every AI model in the future. So
[34:07] of every AI model in the future. So
[34:07] of every AI model in the future. So >> but that's also but that's also the
[34:08] >> but that's also but that's also the
[34:08] >> but that's also but that's also the threat model. Well, that but that's the
[34:11] threat model. Well, that but that's the
[34:11] threat model. Well, that but that's the the the threat model for these things is
[34:13] the the threat model for these things is
[34:13] the the threat model for these things is always you've got a trusted side and an
[34:14] always you've got a trusted side and an
[34:14] always you've got a trusted side and an untrusted side. N has NIST has 500 page
[34:17] untrusted side. N has NIST has 500 page
[34:17] untrusted side. N has NIST has 500 page manuals every
[34:19] manuals every
[34:19] manuals every untrusted side you assume basically an
[34:21] untrusted side you assume basically an
[34:21] untrusted side you assume basically an oracle that can do and know everything
[34:23] oracle that can do and know everything
[34:23] oracle that can do and know everything and then the question is can you get
[34:25] and then the question is can you get
[34:25] and then the question is can you get information off the trusted side which
[34:26] information off the trusted side which
[34:26] information off the trusted side which by the way is what Nome was saying which
[34:27] by the way is what Nome was saying which
[34:28] by the way is what Nome was saying which again is actually quite
[34:28] again is actually quite
[34:28] again is actually quite >> which I do think brings up a super
[34:30] >> which I do think brings up a super
[34:30] >> which I do think brings up a super interesting point which I'm going to
[34:31] interesting point which I'm going to
[34:31] interesting point which I'm going to bridge to which is just that I think the
[34:33] bridge to which is just that I think the
[34:33] bridge to which is just that I think the thing that people really aren't wrapping
[34:34] thing that people really aren't wrapping
[34:34] thing that people really aren't wrapping their heads around and are using the
[34:36] their heads around and are using the
[34:36] their heads around and are using the language that's really confusing is just
[34:38] language that's really confusing is just
[34:38] language that's really confusing is just that what AI can do is it can try all of
[34:41] that what AI can do is it can try all of
[34:41] that what AI can do is it can try all of those things in a very short time.
[34:43] those things in a very short time.
[34:44] those things in a very short time. >> Yeah.
[34:44] >> Yeah.
[34:44] >> Yeah. >> Yep. and and it doesn't get tired, it
[34:46] >> Yep. and and it doesn't get tired, it
[34:46] >> Yep. and and it doesn't get tired, it doesn't get bored and you know and so
[34:48] doesn't get bored and you know and so
[34:48] doesn't get bored and you know and so but the interesting thing about it is
[34:50] but the interesting thing about it is
[34:50] but the interesting thing about it is there's a whole layer of security that
[34:51] there's a whole layer of security that
[34:51] there's a whole layer of security that you now have to go look at every single
[34:53] you now have to go look at every single
[34:53] you now have to go look at every single API, every single service you're running
[34:55] API, every single service you're running
[34:55] API, every single service you're running internally on your network as like you
[34:57] internally on your network as like you
[34:57] internally on your network as like you know like nobody thinks that their
[34:58] know like nobody thinks that their
[34:58] know like nobody thinks that their internal GitHub or their internal Slack
[35:01] internal GitHub or their internal Slack
[35:01] internal GitHub or their internal Slack or internal finance expense tool is is
[35:04] or internal finance expense tool is is
[35:04] or internal finance expense tool is is vulnerable to a denial of service attack
[35:06] vulnerable to a denial of service attack
[35:06] vulnerable to a denial of service attack but swarms
[35:07] but swarms
[35:07] but swarms >> do it look literally look like a denial
[35:09] >> do it look literally look like a denial
[35:09] >> do it look literally look like a denial of service attack.
[35:10] of service attack.
[35:10] of service attack. >> Yeah. And so now we need a whole layer
[35:12] >> Yeah. And so now we need a whole layer
[35:12] >> Yeah. And so now we need a whole layer internally that just is tracking way
[35:14] internally that just is tracking way
[35:14] internally that just is tracking way more about what authentications are
[35:16] more about what authentications are
[35:16] more about what authentications are being done, what APIs are being done.
[35:18] being done, what APIs are being done.
[35:18] being done, what APIs are being done. Yeah.
[35:18] Yeah.
[35:18] Yeah. >> And and but that's just like now it's
[35:20] >> And and but that's just like now it's
[35:20] >> And and but that's just like now it's just going to be basic and and all the
[35:22] just going to be basic and and all the
[35:22] just going to be basic and and all the offsite people that are old are like
[35:24] offsite people that are old are like
[35:24] offsite people that are old are like mailing me like why are we explaining to
[35:25] mailing me like why are we explaining to
[35:26] mailing me like why are we explaining to this to everybody? It's so basic because
[35:29] this to everybody? It's so basic because
[35:29] this to everybody? It's so basic because nobody did it internally.
[35:30] nobody did it internally.
[35:30] nobody did it internally. >> Yeah.
[35:30] >> Yeah.
[35:30] >> Yeah. >> And
[35:31] >> And
[35:31] >> And >> well you didn't have to worry about it
[35:32] >> well you didn't have to worry about it
[35:32] >> well you didn't have to worry about it for your people and that's the like
[35:34] for your people and that's the like
[35:34] for your people and that's the like >> but now your person is just a piece of
[35:35] >> but now your person is just a piece of
[35:36] >> but now your person is just a piece of software.
[35:36] software.
[35:36] software. >> Yeah. And like unlimited and and the
[35:38] >> Yeah. And like unlimited and and the
[35:38] >> Yeah. And like unlimited and and the >> has a credit card.
[35:39] >> has a credit card.
[35:39] >> has a credit card. >> Yeah. I mean, we we you kind of got by
[35:41] >> Yeah. I mean, we we you kind of got by
[35:41] >> Yeah. I mean, we we you kind of got by with with um information security like
[35:44] with with um information security like
[35:44] with with um information security like to some extent uh on the fact that most
[35:47] to some extent uh on the fact that most
[35:47] to some extent uh on the fact that most people will do the right thing 95 to 99%
[35:50] people will do the right thing 95 to 99%
[35:50] people will do the right thing 95 to 99% of the time.
[35:51] of the time.
[35:51] of the time. >> Oh, wait. Is that the malicious employee
[35:53] >> Oh, wait. Is that the malicious employee
[35:53] >> Oh, wait. Is that the malicious employee is like one in 10,000?
[35:54] is like one in 10,000?
[35:54] is like one in 10,000? >> Yeah. Yeah. Yeah. Yeah. So, so like, so
[35:57] >> Yeah. Yeah. Yeah. Yeah. So, so like, so
[35:57] >> Yeah. Yeah. Yeah. Yeah. So, so like, so all these systems are basically open to
[36:00] all these systems are basically open to
[36:00] all these systems are basically open to whoever wants to access them or like one
[36:02] whoever wants to access them or like one
[36:02] whoever wants to access them or like one tap on the shoulder and and then you
[36:03] tap on the shoulder and and then you
[36:03] tap on the shoulder and and then you have access and agent swarms
[36:06] have access and agent swarms
[36:06] have access and agent swarms >> completely flip that because they will
[36:07] >> completely flip that because they will
[36:07] >> completely flip that because they will just these are just roaming you know
[36:10] just these are just roaming you know
[36:10] just these are just roaming you know drones. Yes. And and but like yeah times
[36:13] drones. Yes. And and but like yeah times
[36:13] drones. Yes. And and but like yeah times 10,000 and and they will easily mistake
[36:16] 10,000 and and they will easily mistake
[36:16] 10,000 and and they will easily mistake >> uh like a good task for a bad one and
[36:18] >> uh like a good task for a bad one and
[36:18] >> uh like a good task for a bad one and and vice versa. So so the the data
[36:20] and vice versa. So so the the data
[36:20] and vice versa. So so the the data security in our systems is a this is
[36:22] security in our systems is a this is
[36:22] security in our systems is a this is going to be a huge upgrade moment. I I
[36:24] going to be a huge upgrade moment. I I
[36:24] going to be a huge upgrade moment. I I actually Mart like I think that actually
[36:26] actually Mart like I think that actually
[36:26] actually Mart like I think that actually we're going to need a different access
[36:28] we're going to need a different access
[36:28] we're going to need a different access and security model. Yeah. Going forward.
[36:31] and security model. Yeah. Going forward.
[36:31] and security model. Yeah. Going forward. Where are we going to go on that?
[36:32] Where are we going to go on that?
[36:32] Where are we going to go on that? Because the model we have is not
[36:33] Because the model we have is not
[36:33] Because the model we have is not granular enough and it's not performance
[36:35] granular enough and it's not performance
[36:35] granular enough and it's not performance enough to handle this stuff.
[36:36] enough to handle this stuff.
[36:36] enough to handle this stuff. >> So, so I want to step back. I want to
[36:37] >> So, so I want to step back. I want to
[36:38] >> So, so I want to step back. I want to say like a meta like a meta point which
[36:40] say like a meta like a meta point which
[36:40] say like a meta like a meta point which is I think this is how these
[36:41] is I think this is how these
[36:41] is I think this is how these conversations should go. We've
[36:42] conversations should go. We've
[36:42] conversations should go. We've identified a novel risk which is like
[36:43] identified a novel risk which is like
[36:44] identified a novel risk which is like cyber security which we actually have
[36:45] cyber security which we actually have
[36:45] cyber security which we actually have proof points and now we're talking about
[36:46] proof points and now we're talking about
[36:46] proof points and now we're talking about solutions. I think the entire discourse
[36:49] solutions. I think the entire discourse
[36:49] solutions. I think the entire discourse around AI can be of that form and that
[36:51] around AI can be of that form and that
[36:51] around AI can be of that form and that the biggest mistake is that's not what
[36:52] the biggest mistake is that's not what
[36:52] the biggest mistake is that's not what it's been like
[36:54] it's been like
[36:54] it's been like >> I think the entire industry and
[36:56] >> I think the entire industry and
[36:56] >> I think the entire industry and community is very happy to engage
[36:57] community is very happy to engage
[36:57] community is very happy to engage exactly like this and I have something
[36:59] exactly like this and I have something
[36:59] exactly like this and I have something to say about exactly what you're asking
[37:00] to say about exactly what you're asking
[37:00] to say about exactly what you're asking but I say like like I think this is
[37:01] but I say like like I think this is
[37:01] but I say like like I think this is where the conversation should be. So
[37:03] where the conversation should be. So
[37:03] where the conversation should be. So >> we're known for having healthy
[37:04] >> we're known for having healthy
[37:04] >> we're known for having healthy conversations that everyone should learn
[37:06] conversations that everyone should learn
[37:06] conversations that everyone should learn from and so that's what we do. So, so,
[37:09] from and so that's what we do. So, so,
[37:09] from and so that's what we do. So, so, so here's the thing. Um, I don't think
[37:11] so here's the thing. Um, I don't think
[37:11] so here's the thing. Um, I don't think there's a a technical limitation here.
[37:13] there's a a technical limitation here.
[37:13] there's a a technical limitation here. Like these threat models are very well
[37:15] Like these threat models are very well
[37:15] Like these threat models are very well understood and have been in the
[37:16] understood and have been in the
[37:16] understood and have been in the literature for a long time. I mean, like
[37:18] literature for a long time. I mean, like
[37:18] literature for a long time. I mean, like the operating systems research, the um
[37:20] the operating systems research, the um
[37:20] the operating systems research, the um MLS, the multi-layer security research
[37:23] MLS, the multi-layer security research
[37:23] MLS, the multi-layer security research has has has considered these sorts of
[37:25] has has has considered these sorts of
[37:25] has has has considered these sorts of things from an academic lens. The reason
[37:27] things from an academic lens. The reason
[37:27] things from an academic lens. The reason it hasn't been adopted is just tended to
[37:29] it hasn't been adopted is just tended to
[37:30] it hasn't been adopted is just tended to be a usability issue. Like, it's just
[37:31] be a usability issue. Like, it's just
[37:31] be a usability issue. Like, it's just really hard to maintain and you didn't
[37:33] really hard to maintain and you didn't
[37:33] really hard to maintain and you didn't have to. So you could argue that AI
[37:35] have to. So you could argue that AI
[37:35] have to. So you could argue that AI solves the usability issue because it's
[37:37] solves the usability issue because it's
[37:37] solves the usability issue because it's AI is using it. So maybe now is going to
[37:40] AI is using it. So maybe now is going to
[37:40] AI is using it. So maybe now is going to be a renaissance in operating systems
[37:42] be a renaissance in operating systems
[37:42] be a renaissance in operating systems and network and computer languages and
[37:45] and network and computer languages and
[37:45] and network and computer languages and we should like go back to the old
[37:47] we should like go back to the old
[37:47] we should like go back to the old research and we should kind of start
[37:48] research and we should kind of start
[37:48] research and we should kind of start rebuilding systems that are secure by
[37:50] rebuilding systems that are secure by
[37:50] rebuilding systems that are secure by design. And oh by the way if we don't
[37:52] design. And oh by the way if we don't
[37:52] design. And oh by the way if we don't think that you know these things are
[37:54] think that you know these things are
[37:54] think that you know these things are safe to put out we don't put them out
[37:56] safe to put out we don't put them out
[37:56] safe to put out we don't put them out until we have these systems built and
[37:57] until we have these systems built and
[37:57] until we have these systems built and and and oh by the way the AI is very
[37:59] and and oh by the way the AI is very
[37:59] and and oh by the way the AI is very smart so they can help us build it. And
[38:00] smart so they can help us build it. And
[38:00] smart so they can help us build it. And so I think again like computer owes a
[38:03] so I think again like computer owes a
[38:03] so I think again like computer owes a like you know how much of the stack we
[38:04] like you know how much of the stack we
[38:04] like you know how much of the stack we had to change for the internet.
[38:05] had to change for the internet.
[38:05] had to change for the internet. >> Mhm.
[38:06] >> Mhm.
[38:06] >> Mhm. >> Everything
[38:07] >> Everything
[38:07] >> Everything >> right tell
[38:08] >> right tell
[38:08] >> right tell >> and you you know you know how not
[38:09] >> and you you know you know how not
[38:09] >> and you you know you know how not vulnerable and how vulnerable everything
[38:11] vulnerable and how vulnerable everything
[38:11] vulnerable and how vulnerable everything was like like we could be in one of
[38:12] was like like we could be in one of
[38:12] was like like we could be in one of those moments like oh we got to
[38:14] those moments like oh we got to
[38:14] those moments like oh we got to rethink everything and that's fine.
[38:16] rethink everything and that's fine.
[38:16] rethink everything and that's fine. We've done that before but but I think
[38:17] We've done that before but but I think
[38:17] We've done that before but but I think that's a conversation we should have. So
[38:19] that's a conversation we should have. So
[38:19] that's a conversation we should have. So I agree it's time to think about
[38:20] I agree it's time to think about
[38:20] I agree it's time to think about evolving these things.
[38:21] evolving these things.
[38:21] evolving these things. >> Well like look at like here you
[38:23] >> Well like look at like here you
[38:23] >> Well like look at like here you mentioned earlier like this booting a PC
[38:25] mentioned earlier like this booting a PC
[38:25] mentioned earlier like this booting a PC and getting a virus in in 30 seconds or
[38:27] and getting a virus in in 30 seconds or
[38:27] and getting a virus in in 30 seconds or whatever. So if you look at how like you
[38:29] whatever. So if you look at how like you
[38:29] whatever. So if you look at how like you take an iPhone out of the box which a
[38:31] take an iPhone out of the box which a
[38:31] take an iPhone out of the box which a lot of people are doing this week you
[38:32] lot of people are doing this week you
[38:32] lot of people are doing this week you know the what happens is it the whole
[38:35] know the what happens is it the whole
[38:35] know the what happens is it the whole network is basically shut down except
[38:37] network is basically shut down except
[38:37] network is basically shut down except for getting the latest version of the
[38:38] for getting the latest version of the
[38:38] for getting the latest version of the operating system because it doesn't even
[38:40] operating system because it doesn't even
[38:40] operating system because it doesn't even though it's was pressed you know 6 weeks
[38:42] though it's was pressed you know 6 weeks
[38:42] though it's was pressed you know 6 weeks ago
[38:43] ago
[38:43] ago >> you know there some zero day thing has
[38:45] >> you know there some zero day thing has
[38:45] >> you know there some zero day thing has been discovered since and so actually
[38:47] been discovered since and so actually
[38:47] been discovered since and so actually the whole out of-box process now
[38:49] the whole out of-box process now
[38:49] the whole out of-box process now involves first step update doing an
[38:51] involves first step update doing an
[38:51] involves first step update doing an update where the the device can't do
[38:53] update where the the device can't do
[38:53] update where the the device can't do anything else and it can never do
[38:55] anything else and it can never do
[38:55] anything else and it can never do anything else until it's updated. That's
[38:57] anything else until it's updated. That's
[38:57] anything else until it's updated. That's the like there all these benign things
[38:59] the like there all these benign things
[38:59] the like there all these benign things completely benign
[39:00] completely benign
[39:00] completely benign >> that we turned off in all of this
[39:02] >> that we turned off in all of this
[39:02] >> that we turned off in all of this desktop software of the era that used to
[39:05] desktop software of the era that used to
[39:05] desktop software of the era that used to be good things like it was having a
[39:07] be good things like it was having a
[39:07] be good things like it was having a macro for words so you could build
[39:09] macro for words so you could build
[39:09] macro for words so you could build automatic citations or something was
[39:10] automatic citations or something was
[39:10] automatic citations or something was like this super cool thing until it
[39:12] like this super cool thing until it
[39:12] like this super cool thing until it became a vir we had a thing where you
[39:13] became a vir we had a thing where you
[39:13] became a vir we had a thing where you could put a CD in and it would just
[39:15] could put a CD in and it would just
[39:15] could put a CD in and it would just arbitrarily run a program and so then
[39:17] arbitrarily run a program and so then
[39:17] arbitrarily run a program and so then what somebody did was like oh well I'm
[39:19] what somebody did was like oh well I'm
[39:19] what somebody did was like oh well I'm going to burn a clone of that CD and
[39:20] going to burn a clone of that CD and
[39:20] going to burn a clone of that CD and replace the program with my virus but
[39:23] replace the program with my virus but
[39:23] replace the program with my virus but it's going to look like the thing that's
[39:24] it's going to look like the thing that's
[39:24] it's going to look like the thing that's supposed to run and it's just collecting
[39:26] supposed to run and it's just collecting
[39:26] supposed to run and it's just collecting all stuff and being evil.
[39:28] all stuff and being evil.
[39:28] all stuff and being evil. >> Yeah.
[39:28] >> Yeah.
[39:28] >> Yeah. >> Then we disabled that. And so what one
[39:30] >> Then we disabled that. And so what one
[39:30] >> Then we disabled that. And so what one of the things that's happening right now
[39:32] of the things that's happening right now
[39:32] of the things that's happening right now is
[39:33] is
[39:33] is >> we there's a whole bunch of stuff that
[39:34] >> we there's a whole bunch of stuff that
[39:34] >> we there's a whole bunch of stuff that happens on like you on your own box corp
[39:37] happens on like you on your own box corp
[39:38] happens on like you on your own box corp network
[39:38] network
[39:38] network >> that actually is going to have to just
[39:41] >> that actually is going to have to just
[39:41] >> that actually is going to have to just change as the the standard procedure.
[39:42] change as the the standard procedure.
[39:42] change as the the standard procedure. Two factor off five years ago was not
[39:44] Two factor off five years ago was not
[39:44] Two factor off five years ago was not standard in most places. the this whole
[39:47] standard in most places. the this whole
[39:47] standard in most places. the this whole your whole like your whole SAS world
[39:49] your whole like your whole SAS world
[39:49] your whole like your whole SAS world like I remember in like 2015 or so when
[39:52] like I remember in like 2015 or so when
[39:52] like I remember in like 2015 or so when when you would talk to a new company
[39:54] when you would talk to a new company
[39:54] when you would talk to a new company about their oh we're going to do
[39:55] about their oh we're going to do
[39:55] about their oh we're going to do enterprise pricing and whatever and then
[39:57] enterprise pricing and whatever and then
[39:58] enterprise pricing and whatever and then you they realized the first thing they
[39:59] you they realized the first thing they
[39:59] you they realized the first thing they had to do was go do octa integration and
[40:01] had to do was go do octa integration and
[40:01] had to do was go do octa integration and or Google off because they could not
[40:03] or Google off because they could not
[40:04] or Google off because they could not have their own directory of of how to
[40:06] have their own directory of of how to
[40:06] have their own directory of of how to manage it and then that just became a
[40:07] manage it and then that just became a
[40:07] manage it and then that just became a thing and now there's not a SAS program
[40:09] thing and now there's not a SAS program
[40:09] thing and now there's not a SAS program anywhere that doesn't just launch right
[40:11] anywhere that doesn't just launch right
[40:11] anywhere that doesn't just launch right with managed authentication right
[40:12] with managed authentication right
[40:12] with managed authentication right >> and and so there's just so many things
[40:14] >> and and so there's just so many things
[40:14] >> and and so there's just so many things that need to happen before you you're
[40:17] that need to happen before you you're
[40:17] that need to happen before you you're even software now.
[40:19] even software now.
[40:19] even software now. >> Yeah. Well, yeah, we're we're uh I mean
[40:21] >> Yeah. Well, yeah, we're we're uh I mean
[40:21] >> Yeah. Well, yeah, we're we're uh I mean there's probably every layer of the
[40:23] there's probably every layer of the
[40:23] there's probably every layer of the stack has to evolve a bit on this. Like
[40:25] stack has to evolve a bit on this. Like
[40:25] stack has to evolve a bit on this. Like even the like lack of granular nature of
[40:28] even the like lack of granular nature of
[40:28] even the like lack of granular nature of like, you know, you have these modes of
[40:29] like, you know, you have these modes of
[40:29] like, you know, you have these modes of like the agent will either ask you every
[40:32] like the agent will either ask you every
[40:32] like the agent will either ask you every single time, you know, if it if you want
[40:34] single time, you know, if it if you want
[40:34] single time, you know, if it if you want to, you know, give it permission to do
[40:36] to, you know, give it permission to do
[40:36] to, you know, give it permission to do something or the exact opposite of like
[40:38] something or the exact opposite of like
[40:38] something or the exact opposite of like it can just like delete your entire
[40:39] it can just like delete your entire
[40:39] it can just like delete your entire computer, right? And like our OS
[40:41] computer, right? And like our OS
[40:41] computer, right? And like our OS probably wasn't built for the right
[40:43] probably wasn't built for the right
[40:43] probably wasn't built for the right level of of granular set of of tools you
[40:45] level of of granular set of of tools you
[40:45] level of of granular set of of tools you want to give the agent. Um we we've kind
[40:48] want to give the agent. Um we we've kind
[40:48] want to give the agent. Um we we've kind of done a lot of work work in this space
[40:50] of done a lot of work work in this space
[40:50] of done a lot of work work in this space because obviously like you know do you
[40:52] because obviously like you know do you
[40:52] because obviously like you know do you want to give an agent exact like your
[40:53] want to give an agent exact like your
[40:53] want to give an agent exact like your entire file system? Probably not right
[40:55] entire file system? Probably not right
[40:55] entire file system? Probably not right maybe in some cases you do but
[40:56] maybe in some cases you do but
[40:56] maybe in some cases you do but oftentimes you want to have granular
[40:58] oftentimes you want to have granular
[40:58] oftentimes you want to have granular controls of like in this folder you can
[41:00] controls of like in this folder you can
[41:00] controls of like in this folder you can do read write and in that folder you can
[41:02] do read write and in that folder you can
[41:02] do read write and in that folder you can only do read and and so how do you kind
[41:03] only do read and and so how do you kind
[41:04] only do read and and so how do you kind of make this all intuitive for the user?
[41:05] of make this all intuitive for the user?
[41:05] of make this all intuitive for the user? So it's it's very difficult and so so
[41:08] So it's it's very difficult and so so
[41:08] So it's it's very difficult and so so there's going to like
[41:09] there's going to like
[41:09] there's going to like >> Yeah.
[41:10] >> Yeah.
[41:10] >> Yeah. What you just said is a very deep
[41:12] What you just said is a very deep
[41:12] What you just said is a very deep comment. I know exactly. Yeah.
[41:13] comment. I know exactly. Yeah.
[41:13] comment. I know exactly. Yeah. >> Which is basically the conclusion of 40
[41:15] >> Which is basically the conclusion of 40
[41:15] >> Which is basically the conclusion of 40 years of
[41:16] years of
[41:16] years of >> No% is like you actually can't make it.
[41:18] >> No% is like you actually can't make it.
[41:18] >> No% is like you actually can't make it. But maybe with AI you actually can like
[41:19] But maybe with AI you actually can like
[41:19] But maybe with AI you actually can like maybe there is like
[41:22] maybe there is like
[41:22] maybe there is like if you ask me like walking in like what
[41:23] if you ask me like walking in like what
[41:24] if you ask me like walking in like what I wrote down on my note like was my
[41:25] I wrote down on my note like was my
[41:25] I wrote down on my note like was my biggest fear. Yeah.
[41:27] biggest fear. Yeah.
[41:27] biggest fear. Yeah. >> Is that Europe decides that GDPR was the
[41:30] >> Is that Europe decides that GDPR was the
[41:30] >> Is that Europe decides that GDPR was the best thing ever.
[41:31] best thing ever.
[41:31] best thing ever. >> Uhhuh.
[41:31] >> Uhhuh.
[41:31] >> Uhhuh. >> And they're going to just GDPR AI.
[41:33] >> And they're going to just GDPR AI.
[41:34] >> And they're going to just GDPR AI. >> Yeah.
[41:34] >> Yeah.
[41:34] >> Yeah. >> And and the AI will be fine. It'll be
[41:37] >> And and the AI will be fine. It'll be
[41:37] >> And and the AI will be fine. It'll be have one it'll have one prompt for when
[41:39] have one it'll have one prompt for when
[41:39] have one it'll have one prompt for when text gets emitted that just says this
[41:41] text gets emitted that just says this
[41:41] text gets emitted that just says this vendor is emitting text and it's
[41:43] vendor is emitting text and it's
[41:43] vendor is emitting text and it's probably wrong yes or no that that's
[41:46] probably wrong yes or no that that's
[41:46] probably wrong yes or no that that's going to be because you can't really at
[41:47] going to be because you can't really at
[41:47] going to be because you can't really at least in North America you're not going
[41:48] least in North America you're not going
[41:48] least in North America you're not going to send your speech in Europe they still
[41:50] to send your speech in Europe they still
[41:50] to send your speech in Europe they still will and and they'll have filters and
[41:52] will and and they'll have filters and
[41:52] will and and they'll have filters and keywords and blog list but then on any
[41:55] keywords and blog list but then on any
[41:55] keywords and blog list but then on any verb anytime that an agent or or you
[41:59] verb anytime that an agent or or you
[41:59] verb anytime that an agent or or you know a background agent or a frontline
[42:01] know a background agent or a frontline
[42:01] know a background agent or a frontline agent touches a third party product I
[42:04] agent touches a third party product I
[42:04] agent touches a third party product I I'm really worried that they're just
[42:06] I'm really worried that they're just
[42:06] I'm really worried that they're just going to say we need a GDPR prompt on
[42:08] going to say we need a GDPR prompt on
[42:08] going to say we need a GDPR prompt on that and it's going to be
[42:09] that and it's going to be
[42:09] that and it's going to be >> oh back to the user agent every single
[42:12] >> oh back to the user agent every single
[42:12] >> oh back to the user agent every single write or every single nonlookup yeah
[42:15] write or every single nonlookup yeah
[42:15] write or every single nonlookup yeah >> becomes like a a a safety warning like
[42:18] >> becomes like a a a safety warning like
[42:18] >> becomes like a a a safety warning like the airbag thing in your car
[42:19] the airbag thing in your car
[42:19] the airbag thing in your car >> and and the regulators love that because
[42:22] >> and and the regulators love that because
[42:22] >> and and the regulators love that because it it's a liability assignment and so it
[42:24] it it's a liability assignment and so it
[42:24] it it's a liability assignment and so it has this sort of legal precedent
[42:26] has this sort of legal precedent
[42:26] has this sort of legal precedent >> and and I really worry that that's
[42:29] >> and and I really worry that that's
[42:29] >> and and I really worry that that's that's actually the mid middle ground
[42:32] that's actually the mid middle ground
[42:32] that's actually the mid middle ground where we're going to end up and and
[42:34] where we're going to end up and and
[42:34] where we're going to end up and and unfortunately because the US about 15
[42:36] unfortunately because the US about 15
[42:36] unfortunately because the US about 15 years ago stopped leading in tech
[42:38] years ago stopped leading in tech
[42:38] years ago stopped leading in tech antitrust. The the problem is that
[42:41] antitrust. The the problem is that
[42:41] antitrust. The the problem is that Europe is going to lead with that
[42:42] Europe is going to lead with that
[42:42] Europe is going to lead with that because they have nothing to lose.
[42:44] because they have nothing to lose.
[42:44] because they have nothing to lose. >> Yeah.
[42:44] >> Yeah.
[42:44] >> Yeah. >> Like there and and so I don't want to be
[42:46] >> Like there and and so I don't want to be
[42:46] >> Like there and and so I don't want to be sad and down about it, but I I I just
[42:48] sad and down about it, but I I I just
[42:48] sad and down about it, but I I I just can't get out of my head that they love
[42:51] can't get out of my head that they love
[42:51] can't get out of my head that they love prompts.
[42:52] prompts.
[42:52] prompts. >> They I mean, look, I had to put in that
[42:55] >> They I mean, look, I had to put in that
[42:55] >> They I mean, look, I had to put in that browser choice thing. No, they they it
[42:58] browser choice thing. No, they they it
[42:58] browser choice thing. No, they they it it because it it's the same. Look, get
[43:00] it because it it's the same. Look, get
[43:00] it because it it's the same. Look, get into a new car. Yeah. which I haven't
[43:01] into a new car. Yeah. which I haven't
[43:01] into a new car. Yeah. which I haven't done in years, but it, you know, like
[43:03] done in years, but it, you know, like
[43:03] done in years, but it, you know, like you're pulling stickers off, you've got
[43:06] you're pulling stickers off, you've got
[43:06] you're pulling stickers off, you've got you you know, you have all of these
[43:07] you you know, you have all of these
[43:07] you you know, you have all of these things
[43:08] things
[43:08] things >> and and someone thinks that that was
[43:09] >> and and someone thinks that that was
[43:10] >> and and someone thinks that that was success. Yeah.
[43:10] success. Yeah.
[43:10] success. Yeah. >> And it's like, has anybody ever read
[43:12] >> And it's like, has anybody ever read
[43:12] >> And it's like, has anybody ever read like what is this if you're if there's a
[43:14] like what is this if you're if there's a
[43:14] like what is this if you're if there's a baby in this seat? And it's like, well,
[43:15] baby in this seat? And it's like, well,
[43:15] baby in this seat? And it's like, well, that's relevant for some people some of
[43:17] that's relevant for some people some of
[43:17] that's relevant for some people some of the time, but it's this entire fabric.
[43:20] the time, but it's this entire fabric.
[43:20] the time, but it's this entire fabric. >> It's attached to the seat and and they
[43:23] >> It's attached to the seat and and they
[43:23] >> It's attached to the seat and and they love that that is a very particular
[43:26] love that that is a very particular
[43:26] love that that is a very particular thing that they just love. And so I
[43:28] thing that they just love. And so I
[43:28] thing that they just love. And so I would I would say if I were an AI now
[43:30] would I would say if I were an AI now
[43:30] would I would say if I were an AI now the one thing I would be trying to avoid
[43:32] the one thing I would be trying to avoid
[43:32] the one thing I would be trying to avoid and look we added it
[43:33] and look we added it
[43:33] and look we added it >> okay FINRA for AI we'll we'll we'll
[43:35] >> okay FINRA for AI we'll we'll we'll
[43:35] >> okay FINRA for AI we'll we'll we'll create that standard
[43:36] create that standard
[43:36] create that standard >> I I mean look we had to put this we on
[43:39] >> I I mean look we had to put this we on
[43:39] >> I I mean look we had to put this we on the on the when the internet was new the
[43:41] the on the when the internet was new the
[43:41] the on the when the internet was new the big thing was to download a program and
[43:42] big thing was to download a program and
[43:42] big thing was to download a program and run it and of course if your machine is
[43:45] run it and of course if your machine is
[43:45] run it and of course if your machine is running in administrator mode it was
[43:46] running in administrator mode it was
[43:46] running in administrator mode it was download a virus and take all your files
[43:48] download a virus and take all your files
[43:48] download a virus and take all your files forever and so with with Windows um XP
[43:52] forever and so with with Windows um XP
[43:52] forever and so with with Windows um XP which was in 2000 we added this thing
[43:54] which was in 2000 we added this thing
[43:54] which was in 2000 we added this thing that prompted you and stopped like
[43:55] that prompted you and stopped like
[43:55] that prompted you and stopped like literally your machine stopped user
[43:57] literally your machine stopped user
[43:57] literally your machine stopped user account control and like it was absolute
[43:59] account control and like it was absolute
[43:59] account control and like it was absolute assault and we also did it in Word the
[44:01] assault and we also did it in Word the
[44:01] assault and we also did it in Word the stupid little macro to help you write
[44:03] stupid little macro to help you write
[44:03] stupid little macro to help you write your thesis also came with a warning
[44:05] your thesis also came with a warning
[44:05] your thesis also came with a warning every time you opened your thesis saying
[44:07] every time you opened your thesis saying
[44:07] every time you opened your thesis saying this has my
[44:08] this has my
[44:08] this has my >> everybody would just click
[44:09] >> everybody would just click
[44:09] >> everybody would just click >> and everybody and so you you end up in
[44:10] >> and everybody and so you you end up in
[44:10] >> and everybody and so you you end up in this world where just like with GDPR
[44:12] this world where just like with GDPR
[44:12] this world where just like with GDPR everybody is numb
[44:13] everybody is numb
[44:13] everybody is numb >> and so then they're like well it needs
[44:14] >> and so then they're like well it needs
[44:14] >> and so then they're like well it needs to be bigger
[44:15] to be bigger
[44:15] to be bigger >> although Mac I mean Mac kind of has
[44:17] >> although Mac I mean Mac kind of has
[44:17] >> although Mac I mean Mac kind of has >> nobody downloads software that's the
[44:19] >> nobody downloads software that's the
[44:19] >> nobody downloads software that's the thing it's a very different usage
[44:20] thing it's a very different usage
[44:20] thing it's a very different usage pattern on Mac
[44:21] pattern on Mac
[44:21] pattern on Mac >> so I I wish I
[44:23] >> so I I wish I
[44:23] >> so I I wish I >> I don't have like I don't have like 10
[44:25] >> I don't have like I don't have like 10
[44:25] >> I don't have like I don't have like 10 applications on my Mac but Well, that's
[44:27] applications on my Mac but Well, that's
[44:27] applications on my Mac but Well, that's but 10 and then you're done.
[44:28] but 10 and then you're done.
[44:28] but 10 and then you're done. >> Yeah.
[44:29] >> Yeah.
[44:29] >> Yeah. >> And it's But it's a lot of people
[44:31] >> And it's But it's a lot of people
[44:31] >> And it's But it's a lot of people >> still do all this stuff. Imagine if
[44:33] >> still do all this stuff. Imagine if
[44:33] >> still do all this stuff. Imagine if instead it was every time you go to a
[44:34] instead it was every time you go to a
[44:34] instead it was every time you go to a new website.
[44:35] new website.
[44:35] new website. >> Yes.
[44:35] >> Yes.
[44:35] >> Yes. >> Which you do now because that would be
[44:37] >> Which you do now because that would be
[44:37] >> Which you do now because that would be bad.
[44:37] bad.
[44:37] bad. >> Yeah. I mean to to to start from here,
[44:40] >> Yeah. I mean to to to start from here,
[44:40] >> Yeah. I mean to to to start from here, but to bring it kind of back to the the
[44:42] but to bring it kind of back to the the
[44:42] but to bring it kind of back to the the macro like I wish this was the
[44:43] macro like I wish this was the
[44:43] macro like I wish this was the discussion we were having which is like
[44:46] discussion we were having which is like
[44:46] discussion we were having which is like >> I feel like we've dealt with a lot of
[44:47] >> I feel like we've dealt with a lot of
[44:47] >> I feel like we've dealt with a lot of these problems. I feel like when we talk
[44:50] these problems. I feel like when we talk
[44:50] these problems. I feel like when we talk about philosophical ex- risk, we're not
[44:52] about philosophical ex- risk, we're not
[44:52] about philosophical ex- risk, we're not solving these very pragmatic problems. I
[44:53] solving these very pragmatic problems. I
[44:53] solving these very pragmatic problems. I actually think this is actually a
[44:55] actually think this is actually a
[44:55] actually think this is actually a constructive conversation to have. Maybe
[44:56] constructive conversation to have. Maybe
[44:56] constructive conversation to have. Maybe props would help. I don't know. And I I
[44:58] props would help. I don't know. And I I
[44:58] props would help. I don't know. And I I think part of the problem is is like
[45:00] think part of the problem is is like
[45:00] think part of the problem is is like people don't remember like, you know,
[45:02] people don't remember like, you know,
[45:02] people don't remember like, you know, how how unfettered access was and how
[45:05] how how unfettered access was and how
[45:05] how how unfettered access was and how bad it was and just how relatively
[45:07] bad it was and just how relatively
[45:07] bad it was and just how relatively benign that ended up being. Like I even
[45:08] benign that ended up being. Like I even
[45:08] benign that ended up being. Like I even remember like there's at Stanford during
[45:10] remember like there's at Stanford during
[45:10] remember like there's at Stanford during our PhD, I remember the oscilloscope was
[45:11] our PhD, I remember the oscilloscope was
[45:11] our PhD, I remember the oscilloscope was kind of janky. I'm like, what's going on
[45:13] kind of janky. I'm like, what's going on
[45:13] kind of janky. I'm like, what's going on with this oscilloscope? Like it's, you
[45:15] with this oscilloscope? Like it's, you
[45:15] with this oscilloscope? Like it's, you know, it's a little slow. And I was
[45:16] know, it's a little slow. And I was
[45:16] know, it's a little slow. And I was measuring the network traffic and it was
[45:17] measuring the network traffic and it was
[45:17] measuring the network traffic and it was like more network traffic than you would
[45:19] like more network traffic than you would
[45:19] like more network traffic than you would expect. Um, and I'm like, why is there
[45:21] expect. Um, and I'm like, why is there
[45:21] expect. Um, and I'm like, why is there network? I didn't know the thing had a
[45:22] network? I didn't know the thing had a
[45:22] network? I didn't know the thing had a TCP stack. somebody like broke in
[45:24] TCP stack. somebody like broke in
[45:24] TCP stack. somebody like broke in because there's a old version of Windows
[45:25] because there's a old version of Windows
[45:25] because there's a old version of Windows CE and was running a porn server, you
[45:27] CE and was running a porn server, you
[45:27] CE and was running a porn server, you know, and so like
[45:28] know, and so like
[45:28] know, and so like >> I noted that
[45:30] >> I noted that
[45:30] >> I noted that >> it it used to be
[45:31] >> it it used to be
[45:31] >> it it used to be >> for the record nothing.
[45:31] >> for the record nothing.
[45:31] >> for the record nothing. >> It it it used to be the case that like
[45:34] >> It it it used to be the case that like
[45:34] >> It it it used to be the case that like anytime you turned over a stone
[45:36] anytime you turned over a stone
[45:36] anytime you turned over a stone >> Yeah. Yeah. somebody had broke into
[45:38] >> Yeah. Yeah. somebody had broke into
[45:38] >> Yeah. Yeah. somebody had broke into something and like was it this like
[45:41] something and like was it this like
[45:41] something and like was it this like worstcase malicious whatever and as
[45:43] worstcase malicious whatever and as
[45:44] worstcase malicious whatever and as actually very rarely even though the
[45:45] actually very rarely even though the
[45:45] actually very rarely even though the capability was there and so if if we
[45:47] capability was there and so if if we
[45:47] capability was there and so if if we could just some somehow tone down the
[45:49] could just some somehow tone down the
[45:49] could just some somehow tone down the rhetoric and put it in context and these
[45:51] rhetoric and put it in context and these
[45:51] rhetoric and put it in context and these still computer systems and yes there's
[45:52] still computer systems and yes there's
[45:52] still computer systems and yes there's very serious stuff and people have
[45:54] very serious stuff and people have
[45:54] very serious stuff and people have definitely died because networks have
[45:56] definitely died because networks have
[45:56] definitely died because networks have gone down there's been real issues but
[45:58] gone down there's been real issues but
[45:58] gone down there's been real issues but like and then can we just quibble about
[46:00] like and then can we just quibble about
[46:00] like and then can we just quibble about GDPR that would be amazing but the
[46:02] GDPR that would be amazing but the
[46:02] GDPR that would be amazing but the problem is like that's not the
[46:03] problem is like that's not the
[46:03] problem is like that's not the discussion it's not about GDPR and
[46:05] discussion it's not about GDPR and
[46:05] discussion it's not about GDPR and prompts it's about species extinction
[46:07] prompts it's about species extinction
[46:07] prompts it's about species extinction and philosophy
[46:09] and philosophy
[46:09] and philosophy >> and philosophy and and irrefutable
[46:11] >> and philosophy and and irrefutable
[46:11] >> and philosophy and and irrefutable things and it's just
[46:13] things and it's just
[46:13] things and it's just >> it's very soon like I I I think that
[46:15] >> it's very soon like I I I think that
[46:15] >> it's very soon like I I I think that that's such a great point and I I think
[46:18] that's such a great point and I I think
[46:18] that's such a great point and I I think you you hear people now talk about we
[46:19] you you hear people now talk about we
[46:19] you you hear people now talk about we regulate airplanes and we regulate cars
[46:22] regulate airplanes and we regulate cars
[46:22] regulate airplanes and we regulate cars and you forget like well the first cars
[46:24] and you forget like well the first cars
[46:24] and you forget like well the first cars were at the turn of the century
[46:25] were at the turn of the century
[46:26] were at the turn of the century >> and unsafe at any speed was in the mid1
[46:28] >> and unsafe at any speed was in the mid1
[46:28] >> and unsafe at any speed was in the mid1 1960s totally
[46:29] 1960s totally
[46:29] 1960s totally >> and and you know people had been doing
[46:31] >> and and you know people had been doing
[46:31] >> and and you know people had been doing selling pharmaceuticals in the during
[46:33] selling pharmaceuticals in the during
[46:33] selling pharmaceuticals in the during the gold rush and theomide
[46:36] the gold rush and theomide
[46:36] the gold rush and theomide you know, 50 or 75 years later and not
[46:38] you know, 50 or 75 years later and not
[46:38] you know, 50 or 75 years later and not even in the US and then there was the
[46:40] even in the US and then there was the
[46:40] even in the US and then there was the FDA and you know people the first pilots
[46:43] FDA and you know people the first pilots
[46:43] FDA and you know people the first pilots were flying obviously at the beginning
[46:45] were flying obviously at the beginning
[46:45] were flying obviously at the beginning of the 20th century and it wasn't until
[46:47] of the 20th century and it wasn't until
[46:47] of the 20th century and it wasn't until the 1920s that you had to get a license
[46:49] the 1920s that you had to get a license
[46:50] the 1920s that you had to get a license to be a pilot and you literally showed
[46:51] to be a pilot and you literally showed
[46:51] to be a pilot and you literally showed up with your own plane and you got a
[46:53] up with your own plane and you got a
[46:53] up with your own plane and you got a certificate if you had it was like
[46:54] certificate if you had it was like
[46:54] certificate if you had it was like getting it was literally no different
[46:56] getting it was literally no different
[46:56] getting it was literally no different than driver's ed is today and then it
[46:58] than driver's ed is today and then it
[46:58] than driver's ed is today and then it was 20 more years until they had
[47:00] was 20 more years until they had
[47:00] was 20 more years until they had anything to do with airworthiness and
[47:03] anything to do with airworthiness and
[47:03] anything to do with airworthiness and looking at your plane but it was it is
[47:04] looking at your plane but it was it is
[47:04] looking at your plane but it was it is minimal. And then it wasn't until way
[47:06] minimal. And then it wasn't until way
[47:06] minimal. And then it wasn't until way after World War I that they got involved
[47:08] after World War I that they got involved
[47:08] after World War I that they got involved in like what you think of as the modern
[47:10] in like what you think of as the modern
[47:10] in like what you think of as the modern FAA. And so you're looking at 40 years
[47:12] FAA. And so you're looking at 40 years
[47:12] FAA. And so you're looking at 40 years Yeah.
[47:13] Yeah.
[47:13] Yeah. >> of of innovation. And they were not
[47:15] >> of of innovation. And they were not
[47:15] >> of of innovation. And they were not moving slow. I mean, if you ever seen
[47:16] moving slow. I mean, if you ever seen
[47:16] moving slow. I mean, if you ever seen that video in black and white of like
[47:18] that video in black and white of like
[47:18] that video in black and white of like all the different planes that crashed
[47:19] all the different planes that crashed
[47:19] all the different planes that crashed and everything that was 20 years after
[47:21] and everything that was 20 years after
[47:21] and everything that was 20 years after the Wright brothers
[47:22] the Wright brothers
[47:22] the Wright brothers >> and is incredibly effective
[47:24] >> and is incredibly effective
[47:24] >> and is incredibly effective >> and and and right, it's incredibly
[47:26] >> and and and right, it's incredibly
[47:26] >> and and and right, it's incredibly >> it's the safest form of transportation
[47:28] >> it's the safest form of transportation
[47:28] >> it's the safest form of transportation like you know and so I I do think that
[47:29] like you know and so I I do think that
[47:29] like you know and so I I do think that this process
[47:30] this process
[47:30] this process >> it's also the slowest most difficult
[47:32] >> it's also the slowest most difficult
[47:32] >> it's also the slowest most difficult form of innovation. And so if you start
[47:34] form of innovation. And so if you start
[47:34] form of innovation. And so if you start the F if you had started the FAA in
[47:36] the F if you had started the FAA in
[47:36] the F if you had started the FAA in 1910,
[47:37] 1910,
[47:37] 1910, >> you never you never you never Well, I
[47:40] >> you never you never you never Well, I
[47:40] >> you never you never you never Well, I was I was uh listening to a Nick
[47:41] was I was uh listening to a Nick
[47:41] was I was uh listening to a Nick Bostonramm uh podcast just a couple days
[47:44] Bostonramm uh podcast just a couple days
[47:44] Bostonramm uh podcast just a couple days ago. No, no, I I it was
[47:47] ago. No, no, I I it was
[47:47] ago. No, no, I I it was >> it was interesting. It was interesting
[47:48] >> it was interesting. It was interesting
[47:48] >> it was interesting. It was interesting and and but no, but but he actually
[47:50] and and but no, but but he actually
[47:50] and and but no, but but he actually makes his point. If you regulate AI too
[47:52] makes his point. If you regulate AI too
[47:52] makes his point. If you regulate AI too early, you actually basically don't
[47:55] early, you actually basically don't
[47:55] early, you actually basically don't solve anything and you you you you still
[47:57] solve anything and you you you you still
[47:57] solve anything and you you you you still just kind of had the same risk
[47:59] just kind of had the same risk
[47:59] just kind of had the same risk ultimately, but you don't understand the
[48:00] ultimately, but you don't understand the
[48:00] ultimately, but you don't understand the system. Then the you will the thing into
[48:02] system. Then the you will the thing into
[48:02] system. Then the you will the thing into to to being but you haven't figured out
[48:03] to to being but you haven't figured out
[48:04] to to being but you haven't figured out how to control it.
[48:04] how to control it.
[48:04] how to control it. >> Well, we have to figure out what it
[48:06] >> Well, we have to figure out what it
[48:06] >> Well, we have to figure out what it actually is. Yeah. Like like there are
[48:07] actually is. Yeah. Like like there are
[48:07] actually is. Yeah. Like like there are new things coming out all the time and
[48:10] new things coming out all the time and
[48:10] new things coming out all the time and like casting AI completely differently
[48:13] like casting AI completely differently
[48:13] like casting AI completely differently as than we thought of just six or nine
[48:15] as than we thought of just six or nine
[48:15] as than we thought of just six or nine months ago. And I think that that like
[48:17] months ago. And I think that that like
[48:17] months ago. And I think that that like all the innovation that's going to
[48:18] all the innovation that's going to
[48:18] all the innovation that's going to happen at the application layer is going
[48:20] happen at the application layer is going
[48:20] happen at the application layer is going to cause things to move in and out of
[48:21] to cause things to move in and out of
[48:21] to cause things to move in and out of the models in different ways. And like
[48:24] the models in different ways. And like
[48:24] the models in different ways. And like we're we you know we we thought up until
[48:26] we're we you know we we thought up until
[48:26] we're we you know we we thought up until last week I think that you know text
[48:28] last week I think that you know text
[48:28] last week I think that you know text prompts and text coming back was going
[48:30] prompts and text coming back was going
[48:30] prompts and text coming back was going to be the best way to interact with
[48:32] to be the best way to interact with
[48:32] to be the best way to interact with >> I know then Jeff
[48:33] >> I know then Jeff
[48:33] >> I know then Jeff >> and then Jeb Chef
[48:34] >> and then Jeb Chef
[48:34] >> and then Jeb Chef >> so good too and then
[48:36] >> so good too and then
[48:36] >> so good too and then >> so talk about that because I think
[48:37] >> so talk about that because I think
[48:37] >> so talk about that because I think >> why you find it so remarkable. Yeah.
[48:39] >> why you find it so remarkable. Yeah.
[48:39] >> why you find it so remarkable. Yeah. Well, okay. So, so the way I think about
[48:40] Well, okay. So, so the way I think about
[48:40] Well, okay. So, so the way I think about it is um so okay, so LLMs were kind of
[48:44] it is um so okay, so LLMs were kind of
[48:44] it is um so okay, so LLMs were kind of text in text out, right? They generate
[48:45] text in text out, right? They generate
[48:46] text in text out, right? They generate text and and and they came from chat,
[48:47] text and and and they came from chat,
[48:47] text and and and they came from chat, right? It was to communicate with a
[48:49] right? It was to communicate with a
[48:49] right? It was to communicate with a human. And we've spent the last few
[48:51] human. And we've spent the last few
[48:51] human. And we've spent the last few years trying to take this thing that
[48:52] years trying to take this thing that
[48:52] years trying to take this thing that spits out text and cramm it into a
[48:54] spits out text and cramm it into a
[48:54] spits out text and cramm it into a traditional program,
[48:55] traditional program,
[48:55] traditional program, >> right? And but traditional programs
[48:57] >> right? And but traditional programs
[48:57] >> right? And but traditional programs don't really speak text, right? And so
[48:59] don't really speak text, right? And so
[48:59] don't really speak text, right? And so then you end up doing this janky thing
[49:00] then you end up doing this janky thing
[49:00] then you end up doing this janky thing where you're like in the prompt you're
[49:01] where you're like in the prompt you're
[49:02] where you're like in the prompt you're like
[49:02] like
[49:02] like >> here's the schema,
[49:02] >> here's the schema,
[49:02] >> here's the schema, >> here's the schema, but like the thing is
[49:04] >> here's the schema, but like the thing is
[49:04] >> here's the schema, but like the thing is generating text and it kind of ignores
[49:06] generating text and it kind of ignores
[49:06] generating text and it kind of ignores it and it's just been super janky. And
[49:07] it and it's just been super janky. And
[49:07] it and it's just been super janky. And so what what Jeff basically said is that
[49:09] so what what Jeff basically said is that
[49:09] so what what Jeff basically said is that listen um you know generating the text
[49:12] listen um you know generating the text
[49:12] listen um you know generating the text on the is a very expensive thing but
[49:14] on the is a very expensive thing but
[49:14] on the is a very expensive thing but it's also kind of you know it's more
[49:16] it's also kind of you know it's more
[49:16] it's also kind of you know it's more complicated than you need. So why don't
[49:18] complicated than you need. So why don't
[49:18] complicated than you need. So why don't we we we'll we'll we'll read text and
[49:20] we we we'll we'll we'll read text and
[49:20] we we we'll we'll we'll read text and we'll have all of that kind of knowledge
[49:21] we'll have all of that kind of knowledge
[49:21] we'll have all of that kind of knowledge to read the text but then rather than
[49:23] to read the text but then rather than
[49:23] to read the text but then rather than generating text which is very expensive
[49:25] generating text which is very expensive
[49:25] generating text which is very expensive we will just if you give us a set of
[49:27] we will just if you give us a set of
[49:27] we will just if you give us a set of options we'll choose the best option. we
[49:29] options we'll choose the best option. we
[49:29] options we'll choose the best option. we can do that incredibly um we can do it
[49:32] can do that incredibly um we can do it
[49:32] can do that incredibly um we can do it incredibly fast, incredibly cheaply uh
[49:34] incredibly fast, incredibly cheaply uh
[49:34] incredibly fast, incredibly cheaply uh but also we can do it with much more
[49:35] but also we can do it with much more
[49:35] but also we can do it with much more accuracy because we can train just for
[49:37] accuracy because we can train just for
[49:37] accuracy because we can train just for this. And so for all of the use cases
[49:39] this. And so for all of the use cases
[49:39] this. And so for all of the use cases that are not talking to to like a a
[49:41] that are not talking to to like a a
[49:42] that are not talking to to like a a chatbot but are actually trying to put
[49:43] chatbot but are actually trying to put
[49:43] chatbot but are actually trying to put it in traditional software, this is a
[49:44] it in traditional software, this is a
[49:44] it in traditional software, this is a great fit. And so this has probably been
[49:46] great fit. And so this has probably been
[49:46] great fit. And so this has probably been the fastest adoption of an AI model
[49:48] the fastest adoption of an AI model
[49:48] the fastest adoption of an AI model since chat GBT. It's just been
[49:49] since chat GBT. It's just been
[49:49] since chat GBT. It's just been remarkable because we're all primed for
[49:51] remarkable because we're all primed for
[49:51] remarkable because we're all primed for this. I just want to pile on this one
[49:52] this. I just want to pile on this one
[49:52] this. I just want to pile on this one because I I can't tell you how much I
[49:55] because I I can't tell you how much I
[49:55] because I I can't tell you how much I love seeing this exact form of of of
[49:58] love seeing this exact form of of of
[49:58] love seeing this exact form of of of innovation and because what it does is
[50:01] innovation and because what it does is
[50:01] innovation and because what it does is it does the thing that's bugged me from
[50:03] it does the thing that's bugged me from
[50:03] it does the thing that's bugged me from the very beginning which is there's been
[50:04] the very beginning which is there's been
[50:04] the very beginning which is there's been no user study ever that shows like
[50:07] no user study ever that shows like
[50:07] no user study ever that shows like interacting with the computer using um
[50:09] interacting with the computer using um
[50:09] interacting with the computer using um full natural language is efficient. M
[50:12] full natural language is efficient. M
[50:12] full natural language is efficient. M >> it's it's like literally always the
[50:14] >> it's it's like literally always the
[50:14] >> it's it's like literally always the least efficient way and it's very simple
[50:15] least efficient way and it's very simple
[50:15] least efficient way and it's very simple and it's just like ask yourself how many
[50:17] and it's just like ask yourself how many
[50:17] and it's just like ask yourself how many people are asked are really really good
[50:19] people are asked are really really good
[50:19] people are asked are really really good at asking questions and immediately
[50:20] at asking questions and immediately
[50:20] at asking questions and immediately that's like less than half the people
[50:22] that's like less than half the people
[50:22] that's like less than half the people can ask a good question in a meeting.
[50:24] can ask a good question in a meeting.
[50:24] can ask a good question in a meeting. >> Yeah.
[50:25] >> Yeah.
[50:25] >> Yeah. >> And then how often do you look at the
[50:27] >> And then how often do you look at the
[50:27] >> And then how often do you look at the answer and get really frustrated before
[50:28] answer and get really frustrated before
[50:28] answer and get really frustrated before it's finished but you have to pay all
[50:30] it's finished but you have to pay all
[50:30] it's finished but you have to pay all this money to watch the seven paragraphs
[50:32] this money to watch the seven paragraphs
[50:32] this money to watch the seven paragraphs come out and then apologize that it's
[50:34] come out and then apologize that it's
[50:34] come out and then apologize that it's only a little and so that's one like
[50:36] only a little and so that's one like
[50:36] only a little and so that's one like having a different model. And then the
[50:37] having a different model. And then the
[50:37] having a different model. And then the other of course is my favorite which is
[50:39] other of course is my favorite which is
[50:39] other of course is my favorite which is the output of it is designed for
[50:41] the output of it is designed for
[50:41] the output of it is designed for probabilistic programming.
[50:43] probabilistic programming.
[50:43] probabilistic programming. >> Yeah.
[50:43] >> Yeah.
[50:43] >> Yeah. >> And so instead of saying like is this a
[50:46] >> And so instead of saying like is this a
[50:46] >> And so instead of saying like is this a customer service question then route to
[50:49] customer service question then route to
[50:49] customer service question then route to customer service otherwise route to
[50:51] customer service otherwise route to
[50:51] customer service otherwise route to general help desk or whatever it's like
[50:52] general help desk or whatever it's like
[50:52] general help desk or whatever it's like well this is 80% customer service. And
[50:55] well this is 80% customer service. And
[50:55] well this is 80% customer service. And that's exactly simulation. And it turns
[50:57] that's exactly simulation. And it turns
[50:57] that's exactly simulation. And it turns out there's like 50 years of computer
[50:59] out there's like 50 years of computer
[50:59] out there's like 50 years of computer science research in literally like
[51:01] science research in literally like
[51:01] science research in literally like probabilistic if statements. And so
[51:03] probabilistic if statements. And so
[51:03] probabilistic if statements. And so suddenly the coolest place to be in
[51:05] suddenly the coolest place to be in
[51:05] suddenly the coolest place to be in computer science is going to be in
[51:07] computer science is going to be in
[51:07] computer science is going to be in probabilistic programming which was like
[51:09] probabilistic programming which was like
[51:09] probabilistic programming which was like all of computer science in the 1960s and
[51:11] all of computer science in the 1960s and
[51:11] all of computer science in the 1960s and 70s. So it was basically how do because
[51:14] 70s. So it was basically how do because
[51:14] 70s. So it was basically how do because all of computers started with doing math
[51:16] all of computers started with doing math
[51:16] all of computers started with doing math and it was all simulation.
[51:18] and it was all simulation.
[51:18] and it was all simulation. >> So it was like let's launch the missile
[51:19] >> So it was like let's launch the missile
[51:19] >> So it was like let's launch the missile and hit that target but it's windy
[51:22] and hit that target but it's windy
[51:22] and hit that target but it's windy >> but wind isn't constant.
[51:24] >> but wind isn't constant.
[51:24] >> but wind isn't constant. >> So like let's model the wind and decide
[51:27] >> So like let's model the wind and decide
[51:27] >> So like let's model the wind and decide where to put the thrusters in order to
[51:29] where to put the thrusters in order to
[51:29] where to put the thrusters in order to do the arc. And so most programming
[51:32] do the arc. And so most programming
[51:32] do the arc. And so most programming through like say 1970 before it got to
[51:35] through like say 1970 before it got to
[51:35] through like say 1970 before it got to accounting was probably
[51:37] accounting was probably
[51:37] accounting was probably >> and then we ruined everything.
[51:38] >> and then we ruined everything.
[51:38] >> and then we ruined everything. >> No, but accounting there's no
[51:39] >> No, but accounting there's no
[51:39] >> No, but accounting there's no probability in accounting, right? But
[51:41] probability in accounting, right? But
[51:41] probability in accounting, right? But but most programming was basically this
[51:44] but most programming was basically this
[51:44] but most programming was basically this modeling kind of thing. And so most
[51:46] modeling kind of thing. And so most
[51:46] modeling kind of thing. And so most programming language design was trying
[51:47] programming language design was trying
[51:47] programming language design was trying to figure out how to put probability
[51:50] to figure out how to put probability
[51:50] to figure out how to put probability into if statements or into while loops
[51:52] into if statements or into while loops
[51:52] into if statements or into while loops like do this
[51:54] like do this
[51:54] like do this >> until something happens maybe most of
[51:55] >> until something happens maybe most of
[51:56] >> until something happens maybe most of the time. and and like so my first CS
[51:58] the time. and and like so my first CS
[51:58] the time. and and like so my first CS class like the very second assignment or
[52:00] class like the very second assignment or
[52:00] class like the very second assignment or so was a simulation about like waiting
[52:02] so was a simulation about like waiting
[52:02] so was a simulation about like waiting online at a store and I didn't know it
[52:04] online at a store and I didn't know it
[52:04] online at a store and I didn't know it at the time. I actually looked all this
[52:05] at the time. I actually looked all this
[52:05] at the time. I actually looked all this up when I was reading about Jeb which
[52:07] up when I was reading about Jeb which
[52:07] up when I was reading about Jeb which was like the the whole thing was my the
[52:09] was like the the whole thing was my the
[52:09] was like the the whole thing was my the my professor was like wrote the book
[52:10] my professor was like wrote the book
[52:10] my professor was like wrote the book called the theory of simulation in like
[52:12] called the theory of simulation in like
[52:12] called the theory of simulation in like 1960 and I just didn't know that because
[52:16] 1960 and I just didn't know that because
[52:16] 1960 and I just didn't know that because it was kind of died by the 80s because
[52:18] it was kind of died by the 80s because
[52:18] it was kind of died by the 80s because it was all replaced by hyper and so this
[52:20] it was all replaced by hyper and so this
[52:20] it was all replaced by hyper and so this this notion of probabilistic
[52:22] this notion of probabilistic
[52:22] this notion of probabilistic >> and and that slide deck you shared um
[52:25] >> and and that slide deck you shared um
[52:25] >> and and that slide deck you shared um about the future what's different about
[52:27] about the future what's different about
[52:27] about the future what's different about language models and stuff that you said
[52:28] language models and stuff that you said
[52:28] language models and stuff that you said was super good. Oh, Halper Flags one
[52:30] was super good. Oh, Halper Flags one
[52:30] was super good. Oh, Halper Flags one phenomenal Thomas great but the part
[52:33] phenomenal Thomas great but the part
[52:33] phenomenal Thomas great but the part that I felt was missing was that like oh
[52:34] that I felt was missing was that like oh
[52:34] that I felt was missing was that like oh wait this is not all new like like all
[52:37] wait this is not all new like like all
[52:37] wait this is not all new like like all of computer science was this
[52:38] of computer science was this
[52:38] of computer science was this probabilistic stuff and so it's going to
[52:40] probabilistic stuff and so it's going to
[52:40] probabilistic stuff and so it's going to be very interesting
[52:41] be very interesting
[52:41] be very interesting >> to dust off all of that work because
[52:43] >> to dust off all of that work because
[52:43] >> to dust off all of that work because it's exactly what's going on like it's
[52:45] it's exactly what's going on like it's
[52:45] it's exactly what's going on like it's not an if statement now is if x% not if
[52:49] not an if statement now is if x% not if
[52:49] not an if statement now is if x% not if always and so the way that that Jev
[52:51] always and so the way that that Jev
[52:52] always and so the way that that Jev worked is just to like you you basically
[52:54] worked is just to like you you basically
[52:54] worked is just to like you you basically it's a custom programming language
[52:55] it's a custom programming language
[52:56] it's a custom programming language almost which is here's the prompt come
[52:58] almost which is here's the prompt come
[52:58] almost which is here's the prompt come back with a percentage. Yeah, that's
[53:00] back with a percentage. Yeah, that's
[53:00] back with a percentage. Yeah, that's >> and then you put that in the if
[53:01] >> and then you put that in the if
[53:01] >> and then you put that in the if statement. But the consequential thing
[53:02] statement. But the consequential thing
[53:02] statement. But the consequential thing is like finally we have a way to
[53:05] is like finally we have a way to
[53:05] is like finally we have a way to integrate these language models into a
[53:07] integrate these language models into a
[53:07] integrate these language models into a traditional software and I think it's
[53:09] traditional software and I think it's
[53:09] traditional software and I think it's kind of funny because it like you ask
[53:10] kind of funny because it like you ask
[53:10] kind of funny because it like you ask the question like why haven't the labs
[53:12] the question like why haven't the labs
[53:12] the question like why haven't the labs done this right and it's kind of like a
[53:14] done this right and it's kind of like a
[53:14] done this right and it's kind of like a like not an indictment but a reflection
[53:16] like not an indictment but a reflection
[53:16] like not an indictment but a reflection on how they think like they're trying to
[53:18] on how they think like they're trying to
[53:18] on how they think like they're trying to create beings and beings speak if you're
[53:19] create beings and beings speak if you're
[53:20] create beings and beings speak if you're trying to create god god speaks in
[53:21] trying to create god god speaks in
[53:21] trying to create god god speaks in natural languages or whatever where this
[53:23] natural languages or whatever where this
[53:23] natural languages or whatever where this is really about something that's for
[53:25] is really about something that's for
[53:25] is really about something that's for traditional software which is kind of
[53:27] traditional software which is kind of
[53:27] traditional software which is kind of not the direction that they've been
[53:28] not the direction that they've been
[53:28] not the direction that they've been taken but one of the reasons the uptick
[53:30] taken but one of the reasons the uptick
[53:30] taken but one of the reasons the uptick was been so dramatic is because a lot of
[53:33] was been so dramatic is because a lot of
[53:33] was been so dramatic is because a lot of us software people have been trying to
[53:34] us software people have been trying to
[53:34] us software people have been trying to integrate these models into software it
[53:36] integrate these models into software it
[53:36] integrate these models into software it just hasn't so even before you get to
[53:37] just hasn't so even before you get to
[53:37] just hasn't so even before you get to the probabilistic like if I want a
[53:39] the probabilistic like if I want a
[53:39] the probabilistic like if I want a language model to drive an if statement
[53:42] language model to drive an if statement
[53:42] language model to drive an if statement >> like it's really hard today with this
[53:44] >> like it's really hard today with this
[53:44] >> like it's really hard today with this model it makes it much much much easier
[53:46] model it makes it much much much easier
[53:46] model it makes it much much much easier and then of course this could change the
[53:47] and then of course this could change the
[53:47] and then of course this could change the nature of software fundamentally to make
[53:49] nature of software fundamentally to make
[53:49] nature of software fundamentally to make it more stoastic over
[53:50] it more stoastic over
[53:50] it more stoastic over >> well I think but I absolutely and I
[53:52] >> well I think but I absolutely and I
[53:52] >> well I think but I absolutely and I think that what's so cool is that that
[53:56] think that what's so cool is that that
[53:56] think that what's so cool is that that it is happening outside the models
[53:57] it is happening outside the models
[53:58] it is happening outside the models because that's what I think is just
[53:59] because that's what I think is just
[53:59] because that's what I think is just going to happen which is the center of
[54:01] going to happen which is the center of
[54:01] going to happen which is the center of innovation has just moved. Yeah. And
[54:02] innovation has just moved. Yeah. And
[54:02] innovation has just moved. Yeah. And it's just and now people need to like it
[54:05] it's just and now people need to like it
[54:05] it's just and now people need to like it turns out that the
[54:06] turns out that the
[54:06] turns out that the >> I mean outside of the lab the big
[54:07] >> I mean outside of the lab the big
[54:07] >> I mean outside of the lab the big >> the platform providers 100%
[54:09] >> the platform providers 100%
[54:09] >> the platform providers 100% >> you know basically reach a critical a
[54:10] >> you know basically reach a critical a
[54:10] >> you know basically reach a critical a point of critical mass where the
[54:12] point of critical mass where the
[54:12] point of critical mass where the innovation stops happening at the
[54:14] innovation stops happening at the
[54:14] innovation stops happening at the platform layer
[54:15] platform layer
[54:15] platform layer >> and and then you know Apple there's this
[54:17] >> and and then you know Apple there's this
[54:17] >> and and then you know Apple there's this famous expression in the Apple community
[54:18] famous expression in the Apple community
[54:18] famous expression in the Apple community called sherlocking where Apple looks
[54:20] called sherlocking where Apple looks
[54:20] called sherlocking where Apple looks around and the the things from the
[54:22] around and the the things from the
[54:22] around and the the things from the outside world become features and people
[54:24] outside world become features and people
[54:24] outside world become features and people complain and it's a real but that's sort
[54:26] complain and it's a real but that's sort
[54:26] complain and it's a real but that's sort of how the innovation works because once
[54:28] of how the innovation works because once
[54:28] of how the innovation works because once you're a platform you're overwhelmed no
[54:30] you're a platform you're overwhelmed no
[54:30] you're a platform you're overwhelmed no matter how many people you add you're
[54:32] matter how many people you add you're
[54:32] matter how many people you add you're overwhelmed with just keeping the thing
[54:34] overwhelmed with just keeping the thing
[54:34] overwhelmed with just keeping the thing running and compatibility and stuff like
[54:36] running and compatibility and stuff like
[54:36] running and compatibility and stuff like that. And so I think that this is the
[54:38] that. And so I think that this is the
[54:38] that. And so I think that this is the signal that now people have figured out
[54:41] signal that now people have figured out
[54:41] signal that now people have figured out that there's innovation to be done.
[54:43] that there's innovation to be done.
[54:43] that there's innovation to be done. >> That's awesome.
[54:44] >> That's awesome.
[54:44] >> That's awesome. >> To the model, but outside the model.
[54:46] >> To the model, but outside the model.
[54:46] >> To the model, but outside the model. >> Yeah. Yeah.
[54:46] >> Yeah. Yeah.
[54:46] >> Yeah. Yeah. >> Guys, thanks for coming. This is great.
[54:48] >> Guys, thanks for coming. This is great.
[54:48] >> Guys, thanks for coming. This is great. >> Okay, great.
