AI Can Write the Code. Can It Run the System? | Grepr CEO Jad Naous

AI能写代码，但能运维系统吗？| Grepr首席执行官Jad Naous Martin Casado与Jad Naous深入探讨了AI驱动的代码生成与自主系统运维之间的巨大差异，将纯粹的数字语法与在线分布式环境中厚尾、爆炸式的状态空间进行了对比。Jad分享了Grepr如何利用多智能体CLI工作流和提示词约束实现显著的工程杠杆效应，同时通过在上游实时过滤遥测数据，解决现代微服务可观测性成本失控的问题。他们还辩论了“AI SRE”的可行性，阐明了为何诊断故障需要颠覆传统工作流，从捕捉领先的增量变化入手，而非从滞后的告警倒推溯源。他们最终总结认为，尽管AI能够掌握人类有限的排错启发法和查询优化技巧，但自主运维的实现大概率并非依赖模型去预测混乱的现实世界熵增，而是需要工程师去约束并塑造运维环境，以适配机器的能力。 企业可观测性已演变成一种昂贵的“保险单”，由于微服务遥测数据的急剧膨胀，往往会消耗云基础设施总预算的5%至20%。 (6:50) 编码智能体正在拉大生产力差距而非将其抹平，使普通工程师与顶尖工程师之间的表现差异放大了10倍。 (12:35) 通过使用持久化的CLI子智能体以及提示词驱动的技术负责人，Grepr将一个为期六个月的列式数据切片项目压缩至仅需10天。 (20:00) 现有的AI SRE工具在从滞后告警倒推溯源时往往失效；Grepr颠覆了这一工作流，在上游对PB级遥测数据进行聚类，以捕捉领先的增量指标。 (30:00) 在动态生产环境中部署AI面临着物理世界厚尾的熵增，而代码生成则是在有界的符号模拟中运行。 (37:34)

While generative AI has rapidly advanced in writing software code within bounded digital environments, automating live system operations faces an exponentially larger, computationally irreducible state space. Grepr CEO Jad Naous and investor Martin Casado argue that successful operational AI must avoid chasing lagging incident metrics and instead pair deterministic change-detection with structured, investigative workflows. Observability Costs and the 10x Developer Divergence (6:50) By 2023, observability consumed between 5% and 20% of enterprise infrastructure budgets, functioning as an expensive insurance policy for reliability. However, as monolithic architectures fractured into microservices, exploding telemetry volumes caused the marginal reliability gained per monitoring dollar to flatten. Grepr was conceived to filter and extract high-signal telemetry upstream in real time, reducing downstream data ingestion expenses by 90% while surfacing clean context for automation. Within engineering teams, Naous observed that generative tooling did not democratize development or elevate lower-performing engineers. Instead, it multiplied the performance gap between average and top engineers tenfold. Because frontier models cannot independently design distributed architectures, skilled engineers who decompose systemic complexity gain enormous leverage. Using Claude Code within Windows Subsystem for Linux (WSL) to manage persistent terminal sub-agents, Naous authored a production-grade semi-structured log shredding system in 10 days—a task he estimated would previously take a senior engineer six months. Syntax Versus Semantics and the Operational State Space (15:00) A critical divide exists between syntax—the literal code tokens LLMs master through public training datasets—and semantics, which represent actual machine execution behavior and dynamic cluster states. Casado notes that runtime properties like concurrency, caching, and eventual consistency cannot be inferred from syntax alone. Drawing an analogy to a 4-minute MP3 file, Casado observes that a finite 4MB file yields 2^(4MB) possible states, dwarfing the atoms in the universe. Production operations similarly exhibit combinatoric explosions driven by distributed race conditions, unannounced deployments, and dynamic network traffic. Historically, systems engineers resolved unpredictable complexity not through fragile dynamic algorithms, but through hardware over-provisioning—such as buying more RAM rather than relying on swap memory, or adopting Equal-Cost Multi-Path (ECMP) routing over adaptive routing. Casado argues operational AI faces the same dynamic: models struggle against heavy-tailed edge cases and computational irreducibility, where dynamic runtime states cannot be predicted without running the system. Inverting AI SRE: Leading Indicators Over Backtracking (30:00) Casado and Naous contend that most existing 'AI SRE' tools fail because they rely on lagging indicators. When an alert triggers or a metric spikes, an agent is forced to backtrack through an unmanageably vast state space to diagnose root cause. Grepr inverts this workflow by clustering telemetry at the ingest layer to detect the earliest anomalous delta—such as connection pool exhaustion or buffer depletion—before metrics cross alert thresholds. To manage scale without ungrounded hallucinations, Grepr separates pattern detection from reasoning. A high-throughput deterministic engine scans petabytes of daily logs, isolating novel signatures and passing only unknown anomalies to a generative AI agent. The agent then acts as an investigator: checking code repositories, evaluating call sites, executing runbook heuristics, and formulating testable hypotheses to isolate root causes within a bounded diagnostic loop. Economic Equilibrium and Constraining the Environment (51:48) Drawing parallels to autonomous driving, Casado notes that winning the DARPA Grand Challenge led to expectations of rapid commercial deployment, yet reaching production reality required 20 years and over $170 billion, resulting in difficult unit economics. Applying universal function approximators to the real world is constrained by economic viability rather than raw theoretical feasibility. In enterprise sales, Naous found customers rejected abstract AI pitches but immediately purchased concrete outcomes that reduced telemetry bills and prevented outages. Both speakers predict that rather than waiting for general AI to master infinite operational entropy, industry will adapt the operational environment to match machine strengths—just as modern automated warehouses redesigned physical logistics around robotics. Casado anticipates a decades-long economic feedback loop where human workflows and distributed architectures are intentionally constrained into bounded state spaces that AI models can reliably govern. While AI has proven remarkably effective at code generation, autonomous systems operations remains constrained by real-world entropy, distributed state explosions, and economic friction. Whether AI can fully operate complex infrastructure without human intervention depends on enterprise willingness to redesign and strictly constrain operational environments to fit the bounds of automated reasoning.

Grepr 联合创始人兼首席执行官 Jad Naous 与投资人 Martin Casado 指出，尽管生成式 AI 在代码语法生成上取得了显著突破，但在应对高维度、动态且具备“计算不可约性”的分布式生产运维时，仍面临巨大的状态空间挑战。系统运维的商业化突破不能仅靠泛化的 AI 推理，而必须依靠上游高信噪比遥测、确定性变更检测与聚焦确定性业务结果的工程实践。 可观测性成本陷阱与遥测架构重构 (6:50) 在向微服务架构演变的过程中，现代分布式系统的遥测数据量呈爆炸式增长。企业通常将基础设施总预算的 5% 到 20% 分配给可观测性平台，视其为保障运行稳定性的“保险策略”。然而，由于下游监控费用直接与原始数据吞吐量挂钩，大量低信噪比日志吞噬了巨额预算，监控支出的边际收益严重放缓，并未按比例提升系统可用性。 针对该痛点，Jad Naous 提出在上游数据流中实时解析与过滤遥测数据。Grepr 作为一个即插即用的中间数据层，在数据送入高成本下游存储之前提取结构化高价值信号，声称可将可观测性支出降低 90%，并为后续自动化排障与 AI 推理提供高质量的系统上下文。 语法生成与系统语义的鸿沟 (15:00) 当前大语言模型掌握了海量公开代码的语法表征，使得生成式编码展现出极高效率。Jad Naous 透露，他利用 Windows Subsystem for Linux (WSL) 与 Claude Code 驱动多终端长效代理进行架构设计与 PR 审查，仅用约 10 天便交付了原先需要资深工程师耗时 6 个月的 Apache Parquet 与 Iceberg 数据湖半结构化日志切分（shredding）功能，实现了约 20 倍的人效提升。然而，生成工具不仅没有拉平工程师差距，反而使普通工程师与顶级工程师的技术产出差距拉大了 10 倍，因为分布式架构的顶层心智模型与任务拆解仍然由人类主导。 Martin Casado 与 Jad Naous 强调，代码的“语法”与生产环境的“语义”存在本质脱节。语法仅是编辑器中的字符与逻辑符号，而语义涉及系统的底层运行时表现、访问模式、锁竞争与最终一致性状态。缺乏实际基准测试与动态遥测的语言模型，在做系统内存分配或复杂资源权衡等运维决策时毫无头绪。 颠倒传统 AI SRE：由前置变更检测驱动排障 (30:00) 目前主流的 AI SRE 工具大多依赖指标异常或告警触发等滞后指标，迫使 AI 在近乎无限的状态空间中进行后向回溯，这在复杂的分布式系统中极难准确定位根因。Jad Naous 提出颠倒这一工作流：利用大规模聚类分析，在连接池耗尽、内存临界或缓冲排空等微弱征兆初现时，即时捕捉系统最原始的偏差信号，抢在告警阈值被触发前预警。 在架构设计上，Grepr 明确区分了预测性引擎与生成式代理的职责。每日数十 PB 的海量日志由确定性/预测性模式引擎进行吞吐与基准过滤，一旦发现前所未见的错误特征或模式异动，再将特定信号交给生成式 AI 调查代理。AI 代理随后查阅源代码调用栈、检验历史行为并执行排障实验，将不可预测的运维问题收敛到人类可管理的假设验证流程中。 状态空间爆炸与重尾风险的制约 (37:34) 从计算理论与系统历史来看，Martin Casado 认为分布式运维面临着严峻的组合状态爆炸挑战，其潜在状态数量甚至超过宇宙原子总数，且往往遵循极具偶发性的重尾分布。系统工程的历史多次证明，动态的算法优化（如自适应路由或磁盘内存交换）在实践中容易引发震荡或崩溃，最终产业界往往妥协于直接的硬件过度配置（例如堆叠 RAM 或采用 ECMP 路由）。此外，动态系统存在“计算不可约性”，许多运行时故障无法通过离线分析捷径预测，唯有实际运行系统才能显现结果。 在安全性方面，对抗性威胁的存在进一步放大了该难度，人类攻击者会刻意构造规避特征检测的极端边界条件。针对这些现实阻力，Martin Casado 与 Jad Naous 指出，与其指望 AI 彻底理解复杂的物理现实与组织混乱，未来的工业路径更可能是改造运行环境以适应机器规则，正如自动化立体仓库与限定工业流程的发展脉络。 企业销售现实：聚焦确定性成效而非 AI 标签 (57:08) 在软件工程领域，代码补全助手因直接节省昂贵的工程师人时预算而迅速获得商业普及。与之相比，自动化系统运维工具的商业化推行面临更长周期的验证过程。Martin Casado 以自动驾驶产业耗资 1700 亿美元历经 20 年发展仍受困于单位经济效益为例，警示真实世界长尾分布对自动化的巨大阻力。 Jad Naous 在市场实践中发现，当向企业客户直接推销前沿“AI 运维能力”时，往往遭遇怀疑；而将销售定位转向具体的商业产出——降低云端账单、压缩底层数据量、在重大故障爆发前捕获系统异常时，客户才会迅速签约。企业技术买家对抽象的 AI 技术概念保持审慎，仅愿意为确凿的系统可靠性与直接的成本削减买单。 代码生成与系统运维之间存在巨大的复杂度鸿沟。尽管高级模型已能在人类严格约束的基准与上下文提示下辅助特定模块的优化，但面对具备重尾特征、隐性组织知识与状态组合爆炸的真实生产集群，完全自治的 AI 运维仍然处于极早期阶段。产业界在未来数十年内更可行的演进方向，是通过重构遥测架构与约束系统环境，逐步将运维探索纳入可解释的实验闭环中。

## Background 本期播客由 a16z 通用合伙人 Martin Casado 主持，邀请 Grepr 创始人兼 CEO Jad Naous 共同对谈，于 2026 年 10 月 16 日在 RSS 及各大播客平台发布。节目主要围绕 AI 突破代码编写之后、能否进一步理解并接管复杂的分布式系统与生产环境运维展开深入探讨。 ## TL;DR Jad Naous 分享了其 10 人精简团队如何通过 Claude Code 等命令行多 Agent 协作工作流，将原本需要单人开发半年的数据湖核心项目压缩至 10 天交付，极大放大了顶尖工程师的效能。针对 AI 在运维领域的应用，两人深入剖析了“代码语法”与“运行时语义”之间的本质鸿沟，指出纯依靠大语言模型无法穷举分布式系统的无限状态空间与长尾异常。Grepr 探索出“传统预测模型做早期异常聚类筛选，大语言模型进行源码回溯与因果推演”的混合架构，并通过性能基准成功在查询优化等受限领域取得突破，但面对底层集群的自主掌控仍面临严峻的物理与经济约束。 ## Atmosphere & style 整场对话在轻松熟稔的叙旧与极高密度的硬核技术思辨之间自如切换，斯坦福师兄弟的同门情谊让双方交谈充满默契且毫不避讳分歧。从打趣当年吸烟、用 Windows 和终端写代码的极客偏好，到直切“过度配给”、计算不可约性与自动驾驶 20 年教训的宏大命题，气氛既有工程师攻克难题的务实笃定，又饱含对复杂系统物理规律的敬畏与冷静反思。 ## Bulletin - `11:15` Grepr CEO Jad Naous 指出，AI 将顶尖工程师与普通工程师之间的生产力差距放大了 10 倍，AI 无法让平庸的工程师变优秀，只有大脑能承受更高系统复杂度的人才能高效拆解架构任务并分发给模型。 - `23:45` 借助 Windows WSL、Claude Code 及并行命令行子 Agent，Jad 单兵在 10 天内完成了数据湖 Parquet/Iceberg 列动态提升（Auto-promotion）的核心分布式代码，而传统人力通常需要耗时 6 个月，交付效率提升近 20 倍。 - `30:00` 在 AI SRE 与系统运维路线上，Grepr 放弃了传统从滞后监控指标逆向回溯根因的高难度做法，转为先用确定性模型对每日 PB 级日志进行聚类，捕获细微的新模式后，再交由大模型逆向比对代码与排查故障。 - `37:37` a16z 合伙人 Martin Casado 指出系统工程历史上无数次试图靠算法自治（如虚拟内存换页、动态路由）最终全都妥协于粗暴的“过度配给”（Over-provision），代码生成的解空间与训练分布重合，而现实运维具有无限状态空间与长尾异常，AI 落地将受制于冷酷的经济均衡。 ## Key takeaways - `05:00` 可观测性成本急剧攀升却带来边际可靠性见顶：企业在微服务架构下将 5% 至 20% 的基建预算花在监控上，但购买的本质是可靠性而非监控本身，Grepr 试图在上游实时过滤 90% 的冗余噪声以提纯高信噪比数据。 - `11:15` AI 不会拯救差工程师，而是严重放大了工程师之间的技能鸿沟：顶尖工程师能够理解底层语义并拆解任务，同时驾驭不同框架（Harness）来限制 Agent 滥用 Token，平庸工程师则缺乏这种架构级掌控力。 - `14:48` 代码语法与系统语义存在天然鸿沟：语法仅是输入的字符外壳，而语义代表系统在不同数据规模、负载拓扑和配置下的运行时行为，很多系统语义仅凭静态代码无法直接推导。 - `20:00` 高效的多 Agent 工作流依赖独立的 CLI 命令行会话：通过在终端拉起独立的子 Agent，让主 Agent 与审查 Agent 互相质询架构设计，能有效保持持久上下文，最终再收敛由人类工程师审查合并。 - `23:45` AI 辅助基础设施开发取得 20 倍提速实证：数据湖中的半结构化字段动态粉碎与物理列提升涉及复杂的分布式元数据协调，通过 AI 工具辅助，交付周期从传统的 6 个月压缩至 10 天。 - `28:15` AI 在具备性能基准（Grounding）时擅长逻辑层面的查询优化，但脱离实际物理测量时无法准确推断出合理的物理内存与硬件配给边界。 - `30:00` AI SRE 的有效解法是“前向捕获异常”而非“滞后逆向倒推”：监控指标突增等滞后警报的根因链路极其复杂，系统出故障前往往会抛出未曾见过的细微日志模式，尽早捕捉这些微弱信号是避免线上事故的关键。 - `35:10` 运维排障不应使用单一模型端到端处理，而应采用双层分工架构：由轻量级确定性/预测模型扫描海量日志做聚类检测，再让生成式 AI 发挥长项去阅读源码、推演因果假设与制定修复计划。 - `35:10` AI 替代人类做 SQL 查询优化已经非常接近落地，但在全面接管底层分布式集群管理上仍未达到实用水平。 - `37:37` 计算机系统演进史是一部“用过度配给代替算法预测”的历史：因为现实世界充满重尾分布和长尾故障，无论是虚拟内存还是网络路由，历史上聪明的自适应算法最终大多被廉价硬件的过度配给所替代。 - `40:00` 排障过程与混沌工程的本质差异在于工作流可收敛：虽然现实系统状态比全宇宙原子还多，但工程师遵循的排查假设与实验验证步骤属于低维受限空间，AI 可以通过模拟与因果推演来收窄搜索边界。 - `42:47` 网络安全对抗比系统运维更加残酷：普通运维面对的是大自然的随机性，而安全对抗面对的是恶意攻击者在指数级的离散状态空间中故意寻找死角，防守难度呈几何级上升。 - `50:00` 风险投资的核心壁垒是人际关系驱动的触达与赢得交易，而非形式化的尽职调查：AI 虽能撰写比初级分析师更好的市场分析报告，但这恰恰暴露了风投传统纸面尽调的“歌舞伎表演”本质。 - `51:48` 自动驾驶耗时 20 年与 1700 亿美元的教训证明，在非约束域中单纯追求通用逼近会导致经济性崩溃，现实物理世界的问题必须寻找冷酷的商业与技术平衡点。 - `55:00` 人类倾向于“重塑现实世界以迎合机器”而非硬解混乱：正如自动化立体仓库推倒了传统货架，未来生产系统也会通过重新定义架构边界来降低复杂度，使环境适配 AI 的运行需求。 - `57:32` 企业客户只愿为“降低成本与保障可靠性”买单，不愿为单纯的“AI”标签买单：代码生成因直接绑定最昂贵的人力研发预算而具有立即可见的 ROI，但底层运维系统的价值感知依然存在较高的市场认知门槛。 ## Detailed walkthrough ### `00:00` 序幕与引入：当 AI 从编写代码迈向运行系统 播客一开场抛出了关于人工智能能力的深层叩问：AI 生活的世界完全是数字化的，但这与真实世界并不相同。起初 AI 用于编程时，大家以为它充其量就是个自动补全（autocomplete），没人预料到它竟能构建出极其复杂的软件系统。然而，它迅速打破了人们的预期。但面对生产环境的运维（Operations），情况是否相同？我们能否拥有足够的数据，让模型真正理解全部运维动态，或者说现实系统在复杂度上是否根本无法像代码那样被清晰约束？长期以来，分布式集群能够运转的核心“脏秘密”，其实纯靠人类工程师在幕后救火与维护。那么，AI 如今是否已经具备接管并运行这些分布式服务的能力？ a16z 通用合伙人 Martin Casado 与 Grepr 创始人兼 CEO Jad Naous 展开了深入对谈。作为斯坦福大学同门师兄弟，Martin 邀请 Jad 一起探究：当 AI 写代码的能力日益强大甚至大幅改变其团队开发流程之后，系统运维的未来会走向何方？真实生产系统充满了不可预测的负载波动、突发故障以及极其庞大的可能状态空间。AI 究竟能否像人类资深工程师一样，去主动排查故障、提出假设并进行实验验证？ 在谈话开端，Martin 开玩笑地回忆起当年刚认识 Jad 时的情景：那时 Jad 是实验室新来的博士生，走进办公室，桌上放着一顶摩托车头盔和一包万宝路（Marlboro）香烟，酷劲十足。两人相视而笑，调侃当年的导师 Nick McKeown 撞见抽烟时的微妙反应，甚至回忆起另一位同样抽烟的教授 Balaji Prabhakar。在轻松幽默的叙旧之余，话题迅速切入 Jad 过去在系统与工程领域的深厚技术积累。 - “The world that the AI lives in is all digital and there's the actual world and those things are not necessarily the same.” —— Jad Naous - “The dirty secret for the longest time of running a distributed cluster was just human beings managing it.” —— Martin Casado --- ### `03:00` 从硬件安全、MIT 博士后到回滚系统状态的探索 Jad 随后梳理了自己的技术学术路线。Martin 比他早进斯坦福四年，毕业后创办了知名的软件定义网络（SDN）先锋公司 Nicira。Jad 则留校继续深入网络与系统研究，曾主导 OpenFlow 项目，并深入硬件安全底层。他攻读博士学位期间最有趣的项目之一，是在硬件层面构建了最早一批被考虑采纳的 SHA-3 加密算法实现，借此评估其运行性能。然而，Jad 坦承自己后来对网络本身逐渐失去了兴趣，因为在他看来，网络在本质上只是把比特流从一个地方挪到另一个地方，“这能有多难？” 完成博士学业后，Jad 前往 MIT 师从系统安全领域权威 Nickolai Zeldovich 进行博士后研究，主要聚焦于应用安全。在此期间，他与当时的博士生 Ramesh Chandra 合作并创立了一家公司。这项研究试图解决一个极其艰难的问题：如果一个正在运行的系统遭到攻击或漏洞利用（exploit），能否将与该应用相关的“整个世界的状态”精确回滚到未被入侵的状态，随后在打了补丁的新版程序之上重新应用这些状态，从而彻底抹平攻击？ 这项探索虽然在概念上引人入胜，但在真实工业界落地却异常困难，技术在当时显得过于超前。Martin 也对此产生共鸣，指出这种试图“限定运行时系统状态边界”的难题正是高难度工程学科的典型瓶颈，就像智力上的“捕蝇纸”，让人看到一点进展就深陷其中，最终却往往只能无限渐进于极限而难以彻底攻克。这段直面生产系统极端复杂性的经历，为 Jad 后来理解系统运维的本质打下了深刻的工程烙印。 - “If you get attacked and you get exploited, can you recover from that by rolling back the state of the whole world that's related to that application and then roll it over on top of a patched version of the application to remove the exploit.” —— Jad Naous 描述 MIT 时期的研究课题 - “To what extent can you actually bound the state for operational systems? Which I would argue is very hard.” —— Jad Naous --- ### `05:00` AppDynamics、VC 经历与可观测性的边际成本困境 离开学术界后，Jad 加入了应用性能监控巨头 AppDynamics。在他在职的三年半时间里，公司规模从 200 人迅速扩张至 2000 人，最终被思科（Cisco）收购。成为思科员工仅一天后，Jad 便在 Martin 的极力邀请下加入 a16z 尝试做风投。Martin 当时开玩笑对他说“懂的越少越好”，因为从硬核系统架构师转型为 VC 投资人，就像从读博士转型做业务运营一样，都需要把大脑里固有的思维模式打碎重构（undo the brain damage）。在 a16z 期间，Jad 与 Jennifer、Mark、Ben、Satish、Matt 等人共事，积累了广阔的商业视角。随后他加入 Imply 担任首席产品官（CPO），负责产品、工程、信息安全及设计，将团队从 15 人扩张至 90 多人。当各个板块的管理架构稳固后，日常工作变成了全天候的协调与开会，他便决定离开，开启新的创业探索。 2023 年前后，Jad 开始深入访谈各家企业的工程技术负责人，询问其核心技术开销。当时所有人一致回答：第一是基础架构（Infrastructure），第二是可观测性（Observability）。企业普遍将基础架构预算的 5% 到 20% 砸在可观测性上。然而深入探究其动机后，Jad 发现技术高管真正买单的并不是监控图表本身，而是“可靠性”（Reliability）——可观测性不过是他们追求系统高可用与稳定性的一种代理指标。 过去，投入越多资金在可观测性上，系统可靠性确实越高。但随着云原生与微服务架构的发展，系统被拆解为更碎更小、纵向高度集成的模块，每一层采集的数据量呈现爆炸式增长。可观测性数据的产出速度与成本攀升，远远超过了企业对外输出的功能单元价值，导致每多投入一美元所获得的边际可靠性提升急剧扁平化。企业依旧渴望保障可靠性，但不愿再为海量无用的垃圾数据买单。 ```mermaid flowchart TD accTitle: 可观测性成本与可靠性瓶颈演进 accDescr: 展示了从底层系统数据采集到上游信号提取降本增效的流程演变 subgraph Problem ["传统微服务困境"] A["微服务数量激增"] --> B["遥测数据暴增 (日志/指标/追踪)"] B --> C["可观测性成本飙升 (占基建 5%-20%)"] C --> D["边际可靠性增长彻底见顶"] end subgraph Solution ["上游信号提纯理念"] E["实时数据流采集"] --> F["区分核心信号与冗余噪声 (Fluff/Insurance)"] F --> G["上游完成高维信息提纯与降噪"] G --> H["下游存储与分析成本大幅骤降"] G --> I["提纯信号输入 AI 与自动化运维流程"] end Problem -.->|"推动技术变革"| Solution ``` 由此，Jad 萌生了 Grepr 的核心创想：能否在上游数据流中实时区分真正的“信号”（Signal）与冗余的“噪声”（Noise）？目前采集的大部分可观测性数据本质上只是冗余的“保险杠”，如果能在数据进入下游之前就完成信息提纯与分析，不仅能成倍削减下游存储与查询开销，更重要的是，提纯后的高信噪比实时数据流，能够直接转化为 AI 系统理解与驱动自动化运维的优质养料。 - “Everybody at the time, this was in 23, everybody was saying, hey, we're trying to really spend. And ask them, what are you spending money on? Everybody would say, first was infrastructure and then observability.” —— Jad Naous - “People were spending somewhere between 5% and 20% of what they spend on infrastructure, on observability. But ultimately, what they were really trying to spend it on was reliability. They wanted to buy reliability.” —— Jad Naous - “And the marginal rate or like the marginal increase in reliability for every additional dollar you spending on observability was just flattening out.” —— Jad Naous ### `10:00` Grepr 的定位与团队规模 Jad Naous 介绍了 Grepr 的商业逻辑与架构演进。Grepr 最开始切入的是帮助企业大幅削减遥测（Telemetry）成本——以即插即用（plug-and-play）的方式接入企业的数据流，在数据穿流而过的同时直接为客户削减 90% 的开销。在此基础之上，Grepr 正在构建第二层能力：将这些遥测信号喂给 AI 层，由 AI 主动去进行故障诊断、排查系统异常并完成自动修复。 Martin Casado 赞誉 Jad Naous 是典型的“工程师中的工程师”（the engineer's engineer），不仅做过系统架构师，还亲自带过产品和工程团队。当 Martin 询问 Grepr 当前的规模时，Jad 透露团队目前仅有 10 个人。Martin 由此展开讨论：一个兼具深度架构经验与精简团队的技术掌舵人，究竟如何在实际开发中利用 AI，又从中获得了怎样的生产力跃迁？ ### `11:15` 适应 AI 与驾驭 Harness：优秀工程师的放大效应 回顾软件团队使用 AI 的演进过程，Jad 谈到最初大家都停留在“代码补全”阶段。当时他们在 IntelliJ 中使用 GitHub Copilot，虽然会为自动补全单元测试而惊呼“天哪，看看这个”，但很快意识到那仅仅是初级的 autocomplete。随后技术演进极其迅速，真正优秀的工程师开始以一种截然不同的方式把 AI 融入日常工作流。 ```mermaid flowchart TD accTitle: AI 与工程师的交互适应循环 accDescr: 展示优秀工程师如何通过任务拆解与 Harness 控制与 AI 协同，并形成认知放大效应。 subgraph UserInteraction["认知与控制"] A["高水平工程师<br/>（高复杂度心智模型）"] -->|"拆解复杂架构任务"| B["任务分解与清晰提示词"] B -->|"限制子代理发散<br/>控制 Token 消耗"| C["Agent 运行框架 (Harness)"] end subgraph ExecutionLayer["执行与反馈"] C -->|"调度模型 (Claude Code / Codex 等)"| D["执行代码生成与研究"] D -->|"反馈结果与模型特性"| A end ``` 这种交互甚至演变成一种双向适应——AI 会在无形中“训练”工程师如何向它发号施令。针对 Martin 提出这是否取决于特定模型的问题，Jad 指出除了模型本身的细微差异，更关键的是驾驭不同的运行框架（Harness）。例如在使用 OpenCode、Codex 或 Claude Code 等工具时，它们管理上下文、拆解思考逻辑以及拉起子代理（sub-agents）的机制各不相同。Jad 举例道：“比如用 Claude Code 时，我发现如果我不特意叮嘱它‘别启动 20 个子代理去做基础调研’，它真会一口气拉起 20 个代理到处搜集信息，瞬间把你的 Token 全部烧光。” 随着团队招聘的推进，Jad 发现了一个残酷的现象：普通工程师与顶尖工程师之间的差距被 AI 放大了整整 10 倍。针对 Martin 追问“差工程师能否借 AI 变好”，Jad 给出了坚决的否定： > “It always ends up being like you have to really understand ultimately the setup because the more complexity that your brain can handle, the more you can break down the tasks and kind of assign them.”（归根结底，你必须彻底搞懂底层架构，因为只有你的大脑能承受更高的系统复杂度，你才能够把任务合理拆解并分发给 AI。） - 在 Jad 的观察中，AI 不会让平庸的工程师突然跻身优秀行列，而是让本就优秀的工程师变得更强； - Grepr 的业务重度依赖分布式系统与分布式算法的设计，目前的 AI 大模型本身无法独立想通这种高复杂度问题； - 工程师必须先搞清楚底层语义，把解决路径构思清楚并解释给模型，模型才能去编写诸如查询转换层（query translation layer）这样的具体模块。 ### `14:48` 语法（Syntax）与语义（Semantics）之辩 谈到查询翻译和优化，两人就计算机科学中“语法”（Syntax）与“语义”（Semantics）的差异展开了深入浅出的剖析。Jad 用自然语言做类比：“Dog”这个词的语义是那只四条腿、会汪汪叫的实际动物；而它的语法只是一个由三个不同字母组成的单词外壳。不同词语可以有不同的语法形式，却代表完全相同的概念。 Martin 随即给出了系统工程层面的硬核定义：“语法就像是你敲进去的字符代码，而语义则是机器实际执行的动作，也就是系统的运行时属性（runtime properties）。”两人打趣彼此的背景——Martin 本科读电气工程（EE），后来成了计算机科学家兼 VC；而 Jad 同样有着扎实的计算机体系功底。 Martin 强调，许多系统语义光靠看代码语法是完全推导不出来的，你必须在运行时实测它，而测试结果往往充满变数。例如在 SQL 查询中，相同的 SQL 语法在不同数据规模、底层模式（schema）或配置下，其机器执行行为和性能表现大相径庭。Jad 补充道，系统配置同样会改变执行语义，比如在特定配置下 null 是否被视为 false，只有结合运行环境才能知晓。两人最终达成共识：在接下来的讨论中，“语法”代表实际写的代码，而“语义”统一指代系统在运行时的具体行为与状态。 ### `17:35` 极客的极简工作流：Windows、WSL 与 Claude Code 进入实际开发流程的话题，Martin 提到两人录制前私下闲聊时，Jad 曾透露现在仅凭个人在短短一周内搞定的大型项目，过去往往需要一整支团队耗时数月。当被问及个人的技术栈时，Jad 抛出了一个让 Martin 大跌眼镜的事实：他竟然一直在 Windows 上写代码。 Martin 忍不住惊呼：“什么？难道是微软出的那个 Windows 吗？你是游戏玩家吗，写着写着代码就 Alt-Tab 切去打反恐精英（Counter-Strike）？”Jad 笑着解释，自己不用 Mac 纯粹是因为厌恶苹果的生态设计：“我从没心思去记 Mac 的快捷键，我讨厌单键鼠标，讨厌那些为了按快捷键必须按的一大堆修饰键（modifier keys），我觉得没必要在这上面浪费时间。” Jad 的实际开发环境高度纯粹且硬核： - 底层运行在 Windows 宿主系统上，但日常开发完全依赖 WSL（Windows Subsystem for Linux）； - 系统内完全不安装任何 X-Window 或图形化桌面界面（GUI），纯粹依靠命令行终端（command prompt）进行全套操作； - 核心编码助手使用 Claude Code，通过终端驱动 AI 代理完成整个工程构建流程。 ### `20:00` 多 Agent 协作与命令行架构工作流 Jad Naous 分享了他在开发环境中使用的先进技术组合：利用 Windows WSL（Windows Subsystem for Linux）在 Hypervisor 内运行 Linux 内核，配合 Cloud Code，并采用 Claude Opus 模型进行核心开发。他开玩笑地提到，过去两周里自己一口气输出了整整 20 个大型 PR（Pull Request），然后“直接甩给团队去 Review”。Martin Casado 随即打趣道：“这听起来太惨了，难道现在的职级汇报关系变成了下属替 CEO 审 PR 吗？”Jad 笑着解释，其实团队所有人现在都在这么做，PR 漫天飞舞，每个人都在互相 Review，大家的精力正大幅向测试验证和基准（Golden Expectations）对齐倾斜。 在提升 Agent 上下文效率方面，Jad 摸索出了一个关键技巧：不使用单进程内的子 Agent，而是让 Claude 在命令行（CLI）中直接启动独立的命令行子 Agent。因为命令行子 Agent 能够持久化保持独立的会话状态，工程师甚至可以从不同的终端会话中直接调度它们，这在上下文管理上极为高效。Jad 通常同时打开 3 到 4 个工作窗口；如果并发窗口更多，人类注意力便会脱节，因为任务跑得太快反而容易出现空转等待。 ```mermaid flowchart TD accTitle: Multi-agent CLI development workflow accDescr: 架构展示了从编写设计文档到命令行子 Agent 审查与并行实现的全流程。 subgraph Planning ["规划与架构阶段"] A["主实现 Agent<br/>编写 PRD 与 Design Doc"] B["Tech Lead Reviewer Agent<br/>交叉质询与架构审查"] C["人类工程师 (Jad)<br/>校验劳动拆分与 PR 依赖"] end subgraph Execution ["执行与合并阶段"] D["CLI 并行子 Agent 1<br/>独立上下文会话"] E["CLI 并行子 Agent 2<br/>独立上下文会话"] F["统一合并堆栈<br/>人类逐一 Review 生产上线"] end A <-->|"命令行交互质询"| B B --> C C -->|"识别并行任务"| D C -->|"识别并行任务"| E D --> F E --> F ``` Jad 详细拆解了这一协作机制的运转方式： - **命令行双向质询**：主实现 Agent 会与充当 Tech Lead Reviewer 的 Agent 共同撰写包含 PRD 和详细设计的 PR 文档，Reviewer 会反复提出诸如“这样设计合理吗？”的尖锐问题。 - **架构拆解与校验**：人类工程师仔细核查设计细节，确保工作切分（Division of Labor）合理、PR 间的依赖关系清晰。 - **并行实现与收敛**：确认可并行推进的任务后，主 Agent 派生命令行子进程独立写代码。Jad 强调：“最终你必须把所有东西收敛回一个统一的栈里，由人类逐一进行审查，否则项目会变得非常杂乱分散，无法把控。” --- ### `23:45` 10 天干完 6 个月的活：数据湖“自动提升”上线实战 谈到 AI 辅助开发带来的生产力飞跃，Martin 询问现在的交付周期与几年前相比究竟差距多大，并特别强调讨论的前提是“交付真正的生产级代码（Production Code）”。Jad 举出了最近由他独自完成的一个旗舰级核心项目：数据湖（Data Lake）中的“自动提升”（Auto-promotion，又称字段粉碎 / Shredding）。 在现代数据湖架构中，底层通常基于 Parquet 列式存储与 Apache Iceberg 表格式。然而日志数据大多是半结构化甚至非结构化的，字段极其动态。为了实现最优的存储布局和高速查询，理想状态是将日志事件中的每一个键值对动态提升为物理列，从而允许引擎独立于其他行进行高效过滤。这项工程在分布式存储中复杂度极高，需要深度处理元数据系统的设计、元数据提交协议以及跨多个写入者的并发协调机制。 > “在以前，单兵完成这项工作大概需要 6 个月。而这次，大概只花了 10 天。”——Jad Naous Martin 惊叹于这达到了整整 20 倍的效能提升。这项原本需要长期团队协作或资深架构师闭关半年的核心基础设施代码，如今在极短时间内便保质保量进入了生产环境。 - **核心改进机制**：Auto-promotion / Shredding 将半结构化日志动态映射为 Parquet / Iceberg 列式结构。 - **开发周期压缩**：从传统估算的人均 6 个月大幅压缩至单人 10 天。 - **效率倍数**：在复杂的生产级存储元数据协调系统上实现了接近 20 倍的提速。 --- ### `25:00` 语法生成与运行态鸿沟：分布式系统的复杂性困境 Martin 提出了一个尖锐的系统架构难题：语言模型是在海量代码上训练出来的，生成正确的代码语法合情合理；然而，核心基础设施与分布式系统在运行态（Runtime）的行为，往往完全无法从代码语法表面推导出来。系统的性能表现、可用性以及一致性完全取决于实际运行中的负载特征、访问模式（Access Patterns）和流量激增： - **一致性与可用性权衡**：采用最终一致性（Eventual Consistency）还是悲观锁？若不加锁读取就可能脏读，加锁则会导致系统暂时不可用。 - **缓存与性能折中**：是否引入缓存机制，直接决定了热点流量击穿时的行为。 - **运维数据黑盒**：工业界向来依赖人类 SRE 和运维专家来管理集群、调优数据库，AI 根本无从直接感知这些线下的访问拓扑与硬件环境。 Martin 结合自己的亲身体验大倒苦水：他正在开发一个分布式 3D 高斯泼溅（Gaussian Splatting）系统，需要对数以百万计的高斯球进行流式分发与内存换页，而现有大模型生成的代码“烂透了（It's shit）”，因为系统表现完全依赖于运行环境，没有任何模型能在此类动态场景中做好预判。 --- ### `28:15` 基于性能基准的查询优化与 AI 运维边界 面对动态运行时的复杂性，Jad 阐明了 Grepr 的破局逻辑：关键在于将生产运维规范转化为 AI 可以推理的形式。在 Grepr 的流处理（Stream Processor）系统中，团队预先精确定义了性能基准（Performance Benchmarks）以及系统运行时的硬性约束。 Jad 分享了他正在推进的查询优化（Query Optimization）案例：团队需要将用户输入的 SQL 翻译为流处理流水线代码，最初模型生成的代码十分低效。Jad 直接把模型训斥了一顿：“你生成的代码太烂了，改掉它。”起初模型甚至无法识别代码问题，反问这有什么毛病。Jad 指出：“这里存在重复的连接词（Conjuncts），逻辑项在 `AND` 操作中被冗余计算，早该剔除。”模型在被明确指出优化方向后，迅速推导出一整套代数与逻辑优化路径。 ```mermaid flowchart LR accTitle: Benchmark-grounded query optimization accDescr: 说明 AI 如何结合真实基准测试与数据集进行查询优化，但无法自行决定内存配置。 A["用户 SQL 输入"] --> B["AI 生成基础执行代码"] B --> C{"输入真实查询数据集<br/>与性能基准 Grounding"} C -->|"识别冗余逻辑项与连接词"| D["AI 优化查询流水线"] C -.->|"无法凭空推断硬件边界"| E["动态内存分配配置"] E -->|"必须依赖"| F["人类执行物理实验测量"] ``` Jad 总结了当前 AI 在系统软件上的能力边界： - **基于基准的可行优化**：只要为 AI 提供真实的查询数据集和明确的性能基准（Grounding），模型在查询重写、流水线优化等代数与逻辑层面表现得相当出色。 - **动态物理资源的盲区**：如果脱离测试直接询问 AI“这个系统到底应该分配多少物理内存”，AI 完全没有任何概念（Ed has no idea）。这种底层物理资源维度的配置决策，依然必须扎根于人类工程师在真实生产集群中的实际测试与物理测量。 ### `30:00` AI SRE 的破局思路：从“滞后指标反推”转向“主动变更与异常捕获” 谈及生产环境的运维，Jad Naous 认为目前的系统表现出色的地方在于具有自知之明：当模型意识到自己无法回答某个运维问题时，它会主动设计并运行实验来获取数据。然而，生产系统的行为往往违背预期，这也是事故频发的原因。Martin Casado 随即抛出一个关键问题：业内目前很多尝试将日志喂给 AI、让 AI 充当 SRE（AI SRE）调优系统的实践，业内反馈往往并不理想，Grepr 是否也在做类似的事？其效果究竟如何？ Jad 坦言，Grepr 确实在同时针对运维与安全领域推进 AI SRE 能力，但解题思路与市面上大部分产品截然相反。传统的 AI SRE 往往从“滞后指标（lagging indicators）”切入，例如监控指标突增或警报触发后，再试图逆向倒推庞杂的系统链路寻找根因，而“逆向推导根因是非常成问题的（backtracking into a root cause is actually very problematic）”。Grepr 采取了完全反向的做法：先对海量日志做大规模聚类分析，在“大海捞针”中排查系统中是否出现了此前从未有过的新变化。系统正常扩容通常平稳无声，但一旦出问题，往往会在极早期就抛出极细微的线索，例如“连接耗尽”、“内存超限”或“资源缓冲区排空”等警报。Grepr 的核心就是捕捉这第一条微弱的异常信号，并迅速顺藤摸瓜。 在实际验证中，这种模式表现惊人。Jad 透露，他们在内部系统以及部分客户环境中运行该方案；甚至有几次团队选择忽略 AI 报出的早期异常，结果那些警告最终全都在生产环境中演变成了真实的生产事故。 - 传统 AI SRE 的软肋：依赖指标毛刺（Metric Spike）或告警等滞后结果进行根因回溯，复杂度极高且容易失败。 - Grepr 的检测策略：建立在系统此前运行良好的基线假设上，利用大规模聚类捕捉未曾见过的日志模式作为早期预警。 - 实战警示：无视模型捕获的细微初生异常，已在多次内部与客户测试中直接导致后续线上事故爆发。 --- ### `33:14` 分布式系统的无限状态空间与 AI 运维的边界 Martin 对此表达了深层疑虑。他指出，大语言模型能理解代码语法是因为其训练集覆盖了人类写过的所有代码；然而软件运维的实际状态空间庞大到无法估量，“操作层面的状态增长是成倍增加的，所以很快就会产生比宇宙中原子数量还要多的系统状态”。正因如此，历史上基于产品的自动化运维方案（如混沌工程 Chaos Engineering）往往难以真正奏效，运营分布式集群长期以来的底层秘密其实一直靠“人类肉身维护”。人类工程师排障往往依赖大量隐藏状态，甚至包括“在茶水间偶遇同事时听对方随口提起的部署改动”。在缺乏足够对齐数据和配置的情况下，AI 真的能独立掌管分布式集群吗？ Jad 认为，虽然状态空间理论上是无限的，但人类运维本质上也是依据运行手册（Runbook）行动。目标并不是让 AI 参透一切黑盒，而是提供充分的遥测数据和系统全貌。绝大多数线上故障都源自某种“变更”，只要把精力集中在定位这一变更上，问题就变成了可收敛的确定性工程。 > “操作层面的状态增长是成倍增加的，所以很快就会产生比宇宙中原子数量还要多的系统状态。” —— Martin Casado > > “我们所需要做到的目标，是拥有足够的遥测数据和充分的理解，从而在系统呈现与运行逻辑上让一切变得完全透明可见。” —— Jad Naous ```mermaid flowchart LR accTitle: AI SRE 混合检测与推理架构 accDescr: 展示日志流经传统预测/聚类模型捕获异常模式后，交付大模型进行代码回溯与假设验证的双层工作流。 subgraph Detection["第一层：高吞吐确定性筛选"] Logs["每日 PB 级原始日志流"] --> DetModel["模式聚类与异常检测模型"] DetModel -->|"识别到全新未见模式"| Anomaly["异常信号 / 早期警报"] end subgraph Reasoning["第二层：LLM 认知与排障推理"] Anomaly --> LLM["AI 智能体 (Reasoning Agent)"] LLM --> CodeInspect["逆向定位源码并理解业务逻辑"] LLM --> HypoTest["比对历史行为与运行手册假设"] HypoTest --> Action["输出决策与处置方案"] end ``` --- ### `35:10` 确定性模型与生成式 AI 的分工：查询优化与集群掌管的距离 Martin 提出一个核心疑问：生成式模型本质上并不擅长做确定性的预测或基线异常检测，这是否与 Jad 描述的思路相悖？Jad 澄清，这正是一个普遍的架构误区——异常检测本身绝不由生成式智能体来做，而是由专门的确定性/预测性模型负责。具体架构是：让专门的模型每天扫描 PB 级别的日志，一旦识别出系统从未出现过的新模式，就把该信号提取出来抛给生成式 AI。生成式 AI 随后发挥其推理长项：它去调取并阅读代码，分析是哪一行代码触发了该报错，检查历史上是否发生过，并结合启发式规则与假设进行分析验证，最终判断这是否需要人类介入。 关于落地节奏，两人将场景细分为“查询优化（Query Optimization）”与“分布式集群运维”展开探讨： 1. **查询优化**：Jad 明确表示 AI 替代人类进行查询优化已经“非常接近（Very close）”现实。 2. **分布式集群管理**：面对 Martin 提出的“AI 能否替代类似 MIT 博士 James Cowling 那样支撑全公司生死命脉的分布式集群主管”时，Jad 直截了当地回答：“不，目前还做不到（No, not yet）。” Jad 补充道，AI 辅助编程最初也仅仅是自动补全，当时没人相信它能构建复杂软件，但随着对系统复杂性建模能力的提升，AI 迅速打破了预期。 - 架构解耦：传统预测模型负责高吞吐量的日志聚类与模式突变检测；LLM 负责读取源码、理解业务语义及推理因果。 - 落地预期：确定性更强的 SQL/查询优化即将实现全自动化，但接管底层分布式系统控制权尚非当下模型所能及。 --- ### `37:37` 历史的宿命：算法自适应调优，还是粗暴的“过度配给”？ Martin 从系统工程的发展史切入，提出了一个尖锐的系统论观点。他认为，代码的分布流形（Manifold）相对可控，且能涵盖所有人类代码；但运维完全是另一个领域——充满重尾分布（Heavy-tailed）、极端异常（Exception-based）和幂律分布（Power laws）。回顾计算机系统的历史，工程师无数次想要设计聪明的算法来实现高效自治，例如智能虚拟内存管理、智能拥塞控制、自适应动态路由等。但最终的工程结局毫无例外全是妥协：“我们每一次都选择过度配给（Over-provision）”。 Martin 列举了经典的工程教训： 1. **虚拟内存**：原本设计换页机制（Swap to disk），但一旦真触发磁盘换页，整台机器几乎卡死瘫痪，所以工业界干脆粗暴堆满物理内存（RAM）。 2. **动态路由**：曾设计大量自适应算法来实时响应流量，但往往引发系统剧烈震荡，最终大家认命放弃，全盘采用等价多路径路由（ECMP）加上粗暴的带宽过度配给。 Martin 总结道，正因为系统预测太难，整个系统演进史本质上就是“用过度配给代替预测算法”的历史。而在运维端，要么未来系统继续被粗暴地过度配给，彻底规避复杂的微操运维；要么 AI 能够像资深专家那样处理极端个案。但由于极端场景缺乏训练数据，AI 面临着极为艰难的挑战。面对 Martin 的质疑，Jad 展现出更乐观的底气，他强调自己每天面对的是高度可观测的世界，而在监控完备的环境中，一切物理世界的数据与变更痕迹皆可捕获。 > “每一次……没有任何例外，每一次我们都选择过度配给。” —— Martin Casado > > “系统的历史就是你根本无法预测，所以你只能过度配给，因为你别无选择。” —— Martin Casado ### `40:00` 运维排障、混沌工程与可控的工作流 Jad Naous 和 Martin Casado 讨论了将 AI 引入系统运维的本质区别。Martin 指出，仅仅对大语言模型说“嗨，Claude，这是一堆系统日志，你看这里面出问题了吗”是行不通的，因为现实世界中的运维异常是重尾分布（heavy-tailed）的，错误状态可能非常罕见且从未在训练集中出现过。Jad 对此表示认同，但他指出，运维 AI 的真正价值不是靠单次 prompt 去猜测答案，而是去执行一套排障工作流——运行实验、在系统中探查，并推演各种可能的因果链（causal chains）以验证哪一种与现有数据吻合。 Martin 敏锐地抓住了这个逻辑，并将其与当年兴起但未能彻底颠覆运维的“混沌工程”（chaos engineering）进行类比。混沌工程面临的困境在于系统的状态空间比宇宙中的原子数量还要庞大，而人类工程师的真正能力在于通过一套有条理的排障步骤迅速收敛并约束这个搜索空间。Martin 总结道：“The universe is heavy-tailed. The number of bad things that can happen is probably an exception thing, but the set of steps used to debug it is a manageable thing.” Jad 强调，AI 正在扮演人类工程师的角色：通过因果推断工作流逐步收窄问题范围，而不是指望它像全知机器一样直接吐出结论。 ```mermaid flowchart TD accTitle: AI排障工作流与状态空间收敛 accDescr: 展示异常信号如何通过AI的因果推演与实验验证逐步收窄庞大的系统状态空间 subgraph Input ["输入层"] Sig["监控指标/实体基数异动"] end subgraph AIWorkflow ["AI 探索与验证工作流"] Hypotheses["生成多种可能的因果链假设"] Exp["运行探测实验 / 检索多维度日志"] Verify{"是否与实际系统模式匹配?"} end subgraph Result ["状态收敛"] RootCause["定位根本原因 / 收敛故障空间"] end Sig --> Hypotheses Hypotheses --> Exp Exp --> Verify Verify -- 否 --> Hypotheses Verify -- 是 --> RootCause ``` - 系统故障状态虽呈长尾分布且极度稀疏，但排障的推演过程和验证步骤是结构化且可控的。 - 混沌工程未能普及的原因正是系统状态空间过大，而人类过去靠直觉与标准流程进行人工修剪。 - Jad 透露团队正在网络安全领域尝试类似模式：当实体的统计特征或基数（cardinality）出现异常扰动时，AI 负责枚举潜在因果链，并像“寻找与方形榫头匹配的卯眼”一样去验证实际数据。 --- ### `42:47` 安全对抗与离散状态空间的“4MB MP3 悲剧” 谈及网络安全时，Martin 认为这比日常运维还要困难得多。在普通运维中，极罕见的状态只是大自然本身的随机性；但在安全对抗中，存在存心不良的攻击者，他们会主动且处心积虑地寻找开发者意想不到的罕见死角。两人由“可能的状态空间”展开了幽默的哲学探讨，甚至半开玩笑地扯到了物理学中的普朗克长度（Planck length）以及时空是否离散化。 为了阐明“有限与无限”的错觉，Martin 分享了一段他们在斯坦福研究生时期的真实趣事：同学 Sundar Iyer 曾神情沮丧地跑来对 Martin 说，一首时长 4 分钟的 MP3 音频文件大小大约是 4MB，既然每个字节都是比特构成的有限集合，那就意味着全宇宙所能存在的音乐总数其实是有限的，这让他对“音乐的有限性”感到无比悲哀。Martin 当时在 Stanford Dish 散步时听完大为震惊，忍不住当场吐槽：4MB 的状态空间是 $2^{4 \times 1024 \times 1024 \times 8}$，这早就远远超越了整个宇宙中的原子总数，“你甚至都还没写到 1KB，就已经超越宇宙原子总数了，哥们儿，你到底在伤感什么？放轻松点，这在工程上实际上就是无限的！”借由这个故事，Martin 严肃指出，从业者常常严重低估了复杂分布式系统在实际运行时可能陷入的状态数量，这也是为什么作为风投机构（VC），他认为运维 AI 比单纯写代码的 Coding Agent 要艰难得多。 - 对抗性安全比普通运维更难，因为攻击者是在有意识地构造并遍历罕见边界状态。 - 人类对指数级状态空间的直觉往往是失真的，4MB 文件包含的比特排列组合在物理意义上已形同无限。 - 运维系统面临的不仅是外部不可控的突发负载，还有知识池的极度分散（例如某工程师未经通知直接上线新模块引发死锁）。 --- ### `46:08` 降维表征、计算不可约性与模拟式规划 面对 Martin 的质疑，Jad 表达了对 AI 演进的乐观态度。他认为现实世界遵循概率路径，代码只是解决系统问题的一个微缩缩影（microcosm），人类最初也认为编程包含了无法穷尽的复杂性，但如今 AI 已经打破了这一认知；随着时间推移，模型完全可以通过层层扩展来承载复杂系统问题。Martin 笑着调侃，通常是 VC 在夸夸其谈未来的无限可能，现在角色反而颠倒了过来。 Martin 从机器学习机理出发阐释了自己的世界观：大语言模型之所以能成功，是因为人类的语言将极其高维的宇宙浓缩映射在了一个低维流形（manifold）上，人类的伟大之处在于把复杂的现实降维成了文字和代码。然而，“指令和代码的描述”与“物理世界及系统的真实运转”之间存在本质鸿沟。Martin 引用了计算不可约性（computational irreducibility）的概念指出：“There are things that are computationally irreducible. You kind of have to run it to get the answer. You can't just predict it.” 他认为模型可能存在一个事件视界（event horizon）——凡是不需要通过真实模拟就能预测的问题，模型都能解决；而无法直接跳过计算过程的问题，必须亲自跑一遍。 Jad 随即反驳了“AI 必须一次性直接猜出答案”的前提。他提出，AI 完全有能力进行自我推演、规划执行路径并模拟运行以获取反馈：“Why do you have to just think about it in terms of, hey, it'll have to answer the question now as opposed to being able to simulate and being able to plan its own journey?” Jad 认为，调试过程中的推演步骤本身就是一个低维空间，AI 既可以压缩并驾驭这套步骤，也可以在遇到未知时自动生成代码或模拟器去探测环境，从而逐步构建起对复杂系统乃至更大世界的建模能力。 ### `50:00` AI 能否取代风险投资人：关系驱动与“歌舞伎表演” Jad Naous 顺着讨论抛出了一个尖锐的问题：风险投资（VC）的工作究竟有多少能被 AI 接管？Martin Casado 坦言，在他看来风险投资本质上是“高度依赖人际关系（relationship-driven）”的行业。他坚信市场在绝大多数时候是极其高效的，“所以重点从来不是去发现别人都发现不了的公司，所有人都能找到好公司。” Martin Casado 进一步拆解了自己的投资哲学：他天生对所有技术持乐观态度，从不纠结于所谓的 TAM（潜在市场规模），因为技术总是在创造正向价值。他指出：“我认为关键在于触达能力（access）。它是融资能力、触达能力、赢得交易（win deals）的能力，以及投后赋能的价值。”很多初创公司缺乏独立招募顶级高管等关键资源，而这正是人类投资人发挥不可替代作用的地方。即便能给 AI 一个模拟世界让其通过“奖励作弊（reward hack）”去学习建立信任，真正的工作仍然需要活生生的人来承担。 不过，Martin Casado 随即讽刺了 VC 行业的分析流程。他承认 AI 撰写的分析报告已经不逊于甚至超越了他共事过的任何初级分析师，但这更像是对 VC 自身能力的讽刺：“但我认为这更多是对风投机构‘分析’能力的控诉，而不是说明模型本身有多么强大。我只觉得那全是一场歌舞伎表演和胡扯（kabuki show and bullshit）。”在高效市场中，优秀的投资逻辑简单直接——相信技术浪潮的价值，找到最好的团队，并拿到投资份额。 ```mermaid flowchart TD accTitle: VC核心价值 vs AI能力边界 accDescr: 对比VC的核心竞争壁垒与AI能够胜任的浅层工作 subgraph RelationalCore ["人类投资人的核心壁垒（关系与资源）"] A["高管猎聘与投后赋能"] B["争取交易份额 (Win Deals)"] C["建立深度人际信任与触达 (Access)"] end subgraph AIReplaceable ["AI 可替代的工作（高度形式化）"] D["初级投资分析报告撰写"] E["市场数据搜集与梳理"] F["形式化的尽职调查 (Kabuki Show)"] end RelationalCore -.->|"难以被AI模拟"| G["真实商业闭环"] AIReplaceable -->|"易被模型自动化"| G ``` --- ### `51:48` 约束域与物理世界：自动驾驶 20 年的沉痛教训 Jad Naous 将话题拉回核心分歧：AI 编写代码是一个纯数字化的“抽象模拟世界”，但运维和物理现实却是另一个完全不同的世界。模型训练的“分布内（in-distribution）”数据与真实世界的分布往往脱节。Jad Naous 提出，自动驾驶曾被认为不可能，但如今显然正在实现，因此运维问题终究也会被攻克；那么在代码生成之后，下一个排队被颠覆的领域会是什么？ Martin Casado 提出了一个清晰的判断框架：AI 将在“约束域（constrained domain）”中如野火般迅速蔓延——即问题的解空间分布与训练集的分布完全一致的领域。例如代码语法和图像生成，它们的解空间与训练数据高度重合，AI 在这些领域势如破竹。然而，一旦踏入现实自然世界，经济层面的等价性就会崩溃。 为了说明这种鸿沟，Martin Casado 回忆了自己在斯坦福大学读书时的经历：“我还记得当我在斯坦福的时候，塞巴斯蒂安·特龙（Sebastian Thrun）赢得了 DARPA 大挑战赛。还记得吗？当时我们欢呼：‘万岁！自动驾驶汽车马上就要来了！’说真的，结果花了 20 年时间、1700 亿美元的投资，换来的却是更差的单位经济效益。”面对真实宇宙错综复杂的分布，解决现实问题的难度呈指数级上升。AI 本质上是一个极其通用、效果拔群的“通用函数逼近器（universal function approximator）”，但最终决定哪些场景能跑通的，是冷酷的经济均衡。 --- ### `55:00` 改变世界以迎合 AI：仓储自动化与经济平衡飞轮 针对真实世界的复杂性，Jad Naous 提出了反向思考：很多看似棘手的问题，往往是因为世界原本的构造方式就很难处理，人类试图让 AI 去生搬硬套现状；但历史经验表明，人类往往会选择“改造世界”来适应机器的能力。Jad Naous 指出，这就是现代仓储自动化的演进路径——不再强求机器人在混乱的传统货架间艰难识别，而是把整个仓库推倒重建为纯自动化的结构体系。他推测，未来人类可能会主动重塑业务与系统流程，使其符合 AI 运作所需的规格。 Martin Casado 表示百分之百赞同。他认为“这完全就是机器人技术的发展史”。在纯数字世界中，解空间与训练集同质，问题极易被算法覆盖；而在物理或高度动态的生产世界里，“你可以解决任何你想解决的问题，这仅仅是算力和金钱的问题，而平衡点将是一个经济问题。”改变环境边界、对系统施加人为约束，正是降低复杂度、达成经济平衡的核心手段。Martin Casado 总结道：“我认为我们现在正处于这个元经济平衡飞轮（meta economic equilibrium flywheel）中……我们面临着 30 年的演进周期，期间将创造出巨大的价值。” - **AI 在约束域与现实世界的鸿沟**：代码与图像的训练分布等于解分布；现实世界的数据分布极度发散且充满长尾。 - **自动驾驶的经济账**：历时 20 年耗资 1700 亿美元，但其单位经济模型至今依然严峻。 - **适应模式的反转**：与其训练通用 AI 去应对不可预测的世界，不如改造现实环境（如自动化立体仓库）以降低系统熵增。 --- ### `57:32` 客户只要买“降本与可靠”，不要买“AI”概念 在对未来的宏大畅想之后，Jad Naous 分享了一个团队在销售一线遇到的哭笑不得的真实故事。当 Grepr 团队向企业客户推销产品时，他们开门见山地说：“嘿，我们做了一个 AI 产品，帮助客户提高可靠性。”客户兴致勃勃地安排了会议。但在视频通话中，团队介绍完系统如何优化数据、降低成本并在故障演变成生产事故前将其拦截后，客户突然打断道：“等等，等等，是的，这正是我想买的！但那个 AI 的东西？不不不，不要那个，我们现在想买的是前面那些功能。” Jad Naous 感慨，对于写代码的工具，大家一看就懂，立刻惊呼“这东西太神了”；但到了运维或更深层的系统领域，人们很难直接感知其内在价值并爽快掏钱。Martin Casado 补充分析道，代码生成之所以能迅速爆发出商业价值，是因为它直接等同于“工程研发生产力”，而研发薪酬是大多数科技公司财务报表上最昂贵的开支项目之一，ROI 一目了然。 尽管 Jad Naous 指出，SRE 的生产力以及避免系统宕机所挽回的业务损失同样价值连城，但外界对于这类工具的认知门槛依然很高。Martin Casado 总结说，代码生成之所以带来震撼，是因为其表现太出色且完全超出了人们此前的预期；而 AI 在系统运维与生产运行层面的探索，目前才刚刚揭开序幕。两位嘉宾随后相互致谢，播客在 standard 的免责声明与订阅指引中正式结束。 ## Notable quotes > `33:14` 操作层面的状态增长是成倍增加的，所以很快就会产生比宇宙中原子数量还要多的系统状态。 — Martin Casado > `37:37` 系统的历史就是你根本无法预测，所以你只能过度配给，因为你别无选择。 — Martin Casado > `33:14` 我们所需要做到的目标，是拥有足够的遥测数据和充分的理解，从而在系统呈现与运行逻辑上让一切变得完全透明可见。 — Jad Naous ## Mentioned Martin Casado, Jad Naous, Grepr, a16z, Stanford University, Nick McKeown, Balaji Prabhakar, Nicira, OpenFlow, MIT, Nickolai Zeldovich, Ramesh Chandra, AppDynamics, Cisco, Imply, IntelliJ, GitHub Copilot, OpenCode, Codex, Claude Code, Claude Opus, Windows, WSL, Linux, Parquet, Apache Iceberg, James Cowling, Sundar Iyer, Sebastian Thrun, DARPA ## Video description a16z General Partner Martin Casado sits down with Grepr CEO Jad Naous to ask what comes after AI learns to write code: can it also debug and operate the systems that code creates? Jad shares how AI has transformed engineering inside his 10-person team, cutting some projects from six months to just 10 days. But operating production systems is a harder challenge, with unpredictable workloads, failures, and countless possible states. They discuss whether AI can learn to diagnose problems, test hypotheses, and debug systems like a human engineer. They also explore AI SREs, security, query optimization, and why the best engineers may be gaining the most leverage from coding agents. [Source](https://mgln.ai/e/1344/afp-848985-injected.calisto.simplecastaudio.com/3f86df7b-51c6-4101-88a2-550dba782de8/episodes/c4bfed20-4c94-4742-8309-801df20c4cb9/audio/128/default.mp3?aid=rss_feed&awCollectionId=3f86df7b-51c6-4101-88a2-550dba782de8&awEpisodeId=c4bfed20-4c94-4742-8309-801df20c4cb9&feed=JGE3yC0V)

[33:14] Martin Casado: 操作层面的状态增长是成倍增加的，所以很快就会产生比宇宙中原子数量还要多的系统状态。 [37:37] Martin Casado: 系统的历史就是你根本无法预测，所以你只能过度配给，因为你别无选择。 [33:14] Jad Naous: 我们所需要做到的目标，是拥有足够的遥测数据和充分的理解，从而在系统呈现与运行逻辑上让一切变得完全透明可见。

[0:00] Martin Casado: The dirty secret for the longest time of running a distributed cluster was just human beings managing it. [46:58] Martin Casado: There are things that are computationally irreducible. You kind of have to run it to get the answer. You can't just predict it.

[0:00] The world that the AI lives in is all digital and there's the actual world and those things are not necessarily the same. When we started doing AI for coding, it was autocomplete. Nobody really knew to what extent you can actually build complex software using AI. And then it basically broke through the expectations because we were able to model more and more complexity in the systems. I just don't know if operations is constrained in the same way. Where is even the data to learn how to build a system that understands all of the operations My assumption is that we have all the data. The question, I think, is maybe about complexity. And to what extent can you take the things that you've described that are operational and convert them into something that an AI can reason about? The dirty secret for the longest time of running a distributed cluster was just human beings managing it. Do you think we're at a spot where you could use AI to run one of these services? I think that there's enough to be able to... AI can now write remarkable amounts of production code, but writing software and running it are two very different problems. In this episode, A16Z general partner Martin Cassato sits down with Grepper CEO Jad Naus a systems engineer Martin first met when they were both at Stanford to ask what happens as AI moves from coding into operations Jad explains how his 10 team uses coding agents today, why he thinks the strongest engineers are getting even more leverage from AI, and how work that once took months can now take days. But the harder question is what happens after that code shifts. Production systems face unpredictable workloads, failures, and an enormous number of possible states. Martine and Jad debate whether AI can learn to troubleshoot those systems by detecting what changed, forming hypotheses, and running experiments, or whether operating in the real world is fundamentally harder than generating code. And that leads to the bigger question behind the conversation. How far can AI go when the answer isn't already contained in the world it was trained on? Jad, thanks for joining me on the podcast. For those of you who don't know, Jed is the CEO of Grepper, but also we went to grad school in the same group. And when I first met you, we had a new student. I walked into the office and there was a motorcycle helmet and a pack of Marlboros. I like this guy must be unbelievably cool So my question is what happened to all your coolness Question number one Maybe just for context, maybe give a quick overview of your background. So just for those that are listening, Jed is one of the best systems thinkers I know. I love talking to him. We're going to go kind of to a deep dive of engineering from a systems perspective and how AI is impacting us, especially when it comes to things like operations. But just to ground it, maybe a bit about your background? Sure. So you met me at one of the peaks in my coolness. Yeah, you're pretty cool. Motorcycle and Marlboro cigarettes. I remember thinking, I'm like, wow, are cigarettes even legal? It was like so discordant. I remember Nick seeing me one day smoking outside of outside of Gates and looking at me and going like. Nick McKeown is our advisor. Gates was the The building, yeah. Actually, one of my favorite things, do you remember Balaji Prabhakar, the professor? Of course. My favorite thing about him was that he smoked. I was like the only professor. I just thought it was like, great. Just to be clear, I don't smoke anymore. No, too old, too middle-aged for that now. But let's see. So I went to Stanford with you You were always the guy I had to look up to and you were like hey this the guy I want to be when I grow up Because I started four years before you did So you were four years ahead of me You graduated a while ahead of me. You went and started Nysera. I stayed and stuck around, worked more on OpenFlow, helped build a bunch of stuff on the hardware side from a security perspective. Actually, one of the most fun projects that I worked on there was I built the first set of SHA-3 implementations that people were considering using. And I built them in hardware to kind of show whether they work or not and whatever for one of my projects for my thesis. That was fun. Yeah. And then I decided after a while that I actually don't like networking because it's just about moving bits from one place and putting them in another place. How hard can that be? He has to either be philosophically aligned that is deep or not. Most people assume it's not. I think it's very deep. I mean, there's something interesting about it, but ultimately, to me, that's like, it's not... Yeah, it doesn't do it for you. Yeah. And then I actually went and did a postdoc at MIT with Nikolai Zeljdovic, who's also amazingly deep and super smart. Yeah, he's phenomenal. I did some application security work at the [5:00] time, started a company based off some of the research I had worked on with Ramesh Chandra over there, who was a PhD student at the time. And ultimately, technology was too early. So what we did was, well, ultimately, a lot of the work was actually really Ramesh's. And his PhD was on, well, hey, if you get attacked and you get exploited, can you recover from that by rolling back the state of the whole world that's related to that application and then roll it over on top of a patched version of the application to remove the exploit. Yeah. And so we started doing that in the real world. And we're like, that sounds really hard. It's actually kind of relevant to what we're talking about. To what extent can you actually bound the state for operational systems? Which I would argue is very hard. Yeah. We almost got it working. Almost. I feel like for all long tail disciplines, this is why people get into fly track for smart people. It's like you see a bit of progress, but you're asymptoting. But anyways, I digress. Yes, yes. So then I joined AppDynamics. I was there for about three and a half years. They went from about 200 people to 2000 people got acquired by Cisco I was a Cisco employee for one day And then you like Jad you come work with me Come do this VC thing with me I like what is VC I'm an architect. I do engineering stuff. He's like, great. The less you know, the better. Literally what you said. It's true. And then the first day I started here, you said, slow down. It's actually people that move from actually an operating role to VC. It requires a lot of reworking the brain. Yeah. Just like going from a PhD to operating, by the way. You have to undo the brain damage. This is what real world is like. I know. And then I stayed here for a couple of years, did a bunch of work with you. It was a really fun time. Worked with a lot of great people like Jennifer, Mark, and Ben. Satish. Satish. Matt was here when I first, almost towards the end when I was leaving. Yeah. And then joined Apply as their chief product officer. helped them build their engineering product teams, kind of scaled from about 15 people to over 90 across product engineering, infosec and design. And after about three years, I kind of had a head for each one of these pieces that I owned And I kind of was sitting around just coordinating and doing meetings all day and wanted to go do something else So left and then started talking to a lot of engineering leaders about the problems that they're facing and what they're trying to do. Everybody at the time, this was in 23, everybody was saying, hey, we're trying to really spend. And ask them, what are you spending money on? Everybody would say, first was infrastructure and then observability. And so I started looking at that problem, trying to understand like, why do people think it's too expensive. It turns out there's a few things. First, people were spending somewhere between 5% and 20% of what they spend on infrastructure, on observability. But ultimately, what they were really trying to spend it on was reliability. They wanted to buy reliability. Of spend? Like predictability of spend? Reliability of their infrastructure and applications. I see, I see, I see. And so they were putting money into observability as a proxy for reliability. And for the longest time, the more money you spend, the more reliability you got out. But as architecture has changed and turned into microservices that are much smaller, that are much more vertically integrated, the volume of data that's being collected at every layer expanded, there was much deeper market penetration for observability and kind of the volume of data that you're collecting and the cost of observability was actually starting to grow faster than the units of function that you outputting And the marginal rate or like the marginal increase in reliability for every additional dollar you spending on observability was just flattening out And so that's why people thought it was expensive. But people were still willing to pay five to 20 percent on reliability. And so the question was, can we actually solve that problem and bring back that increase in reliability, that marginal increase in reliability? not necessarily by spending more money on observability, but some other way. And so the thinking was like, okay, great. If we can figure out what is the signal, what is the noise, can we actually turn the observability data that you're collecting that's mostly fluff and mostly insurance at this point and do that kind of extraction of information upstream, do whatever you need upstream and be able to ultimately massively reduce the cost of observability downstream. And if you can extract that signal in real time upstream, then you can one, reduce the cost, but two, you can now feed it into other systems and use it for driving reliability operations afterwards. And the distance at the time, there was something that was called AI coming into the picture. And we're like, well, great. If we are in the data stream and we're extracting the signal, then we are in a strategic. [10:00] position to feed that to whatever is going to come in the future because that's going to be necessary. And so that's what we built. And so Grapper helps companies. We started with helping them reduce their spend on telemetry massively. And so you put us in the. data stream, plug and play, pass the data through, reduce your cost by 90%. And now we're building out that second layer where we're feeding those signals into the AI layer that's now going and troubleshooting and figuring out when things are going wrong. going wrong and fixing them. Oh. I'd love to just talk through kind of your experience on how AI is changing dev generally, which is like a lot of people could ask this question, a lot of people answer, but I just feel like you're kind of like the engineer's engineer. You've been an architect, you've run product, you've run engineering teams. Actually, how big is your company now? How many people? We're 10. Yeah, so you're building a lot of software with a relatively efficient team. senior engineer like you, you know, that has architectural experience, like, like, What sort of gains do you think people are seeing? Maybe talk through like how you use AI, what gains you're actually seeing, what is the stack, what is your process, how you think about it. So maybe I can give you like a historical view of like what's been happening in terms of like running a team and so on. And so, you know, when we first started AR, we're still kind of autocomplete. I remember at the time we were using Copilot and IntelliJ, and we were like, oh my God, look, it autocompleted these tests. And we're like, yeah, but it's just, autocomplete. And very rapidly, things started evolving. And what I've noticed is that really good engineers... are able to somehow... put it into their workflows in a way where the AI starts training them. Oh, interesting. In terms of how to talk to it, how to get the best out of it. And what I noticed... By the way, as an aside, do you think that's model specific? Do you learn how to use AI writ large or do you get efficient at a given model? I think there's some nuances that different models bring. I'm sure looking at your tweets, you've... to work with AI and like Thank you. I think that there's also a way to work with different harnesses. And I think we tried, for example, using OpenCode or different harnesses or using codecs versus Claude. Claude code, different ones have different ways of managing context or thinking through problems or when to spin up the right sub-agents and what to tell it, the kind of prompts. For example, with Claude code, I find that if I don't tell it, don't spin up. 20 Fable agents to do basic research, it'll go and spin up 20 Fable agents to do research and burn all your tokens. And so like a lot of these things, kind of you start learning about... the harness and the models and like how they interact with each other and then how they can interact with the rest of the world. But, I digress. I'm just going to... In my mind, this is kind of what I started noticing is that some people were adapting and learning and kind of doing it that way. And others weren't as... adaptive. And then as we started hiring more people into the team, that disparity between you know, okay engineers and really good engineers just multiply 10x. Can you predict who will... could you predict it would be good? Was it like good engineers got better and bad engineers didn't? Or was it... Really? Okay. Yeah. So like if you're a good engineer, you're just a better good engineer. It's not like there's some like bad engineers all of a sudden became good engineers. No, no, no. It always ends up being like you have to really understand ultimately the setup because the more complexity that your brain can handle, the more you can break down. the tasks and kind of assign them. I think this might actually be like non-consensus opinion that like, I mean, I do hear it sometimes, but the idea that the better engineers are much better, those are less. And you found this to be pretty universally true? I mean, look, my team is still small. We haven't done it across 100 people. In my experience, for the things that we do, that is the case because we build distributed systems and distributed algorithms, and your brain has to handle the complexity because AI doesn't do it. And so for the problems that we solve, none of the models can actually think through the things that we do. Like there's a lot of, for example, we have a query translation layer. We were just talking about query optimization. And so you have to really understand the semantics of the query, [15:00] and how to get to the semantics of the query and then explain it to the model And then have the model go and build the translation layer, let's say. Yeah. So for the people that are listening that may not... - You're so deep in computer science, what do you mean by semantics? So semantics are really the meanings of things. Like the semantic of a dog, of the word dog, is the animal that actually walks on four legs and barks. And the syntax is the fact that it's just a three-letter word and it has three different letters. The shape of the word is different. And that shape can be different between different words. But different words can actually also mean the same thing. So they might have the same semantics, but they might have a different syntax. So a dog and a puppy might be slightly different semantics. They refer to the same. But they're so close to that. So the way, and just for like computer science, the way that I describe it very simply is syntax, you know, it's like the characters you type and semantics is what the machine actually does, right? And so it's like the actual runtime properties of the system. So you did computer science. I did electrical engineering. You became a computer scientist. And I became a VC. That's the problem. I see. Anyways, for people that are listening, I just want to understand, there's the actual syntax, which is writing the code. And then when you talk about syntax, you're just talking about the code itself. And then when you talk about semantics, you're literally talking about you have a machine that interprets that code. whether it's compiled or not, and that machine does something, and semantics is what the machine actually does. And these things are clearly very different. So, for example, some semantics you can't even derive by looking at the syntax. You don't know what the machine is going to do because whatever. You kind of have to test it. You have to test it. And it can go all sorts of different ways when you test it. And a very classic version of this is SQL, for example. You can write some SQL query, but how it actually performs, entirely on, let's say, what the schema looks like, or what the data looks like in the database, etc. But that's, I would not call that semantics though. I would call that... performance characteristics. Oh, sure. Yeah, I think that would be like... I would say the semantics part of that would be like... But depending on what the shape of data is or what the configuration, you might end up with different semantics because, like you said, you want null to be treated as false. And you won't know that unless you also know the configuration of the system and so on. Okay. So for the purpose of this conversation, I think we should just say that there's the actual code you're right and there's how the system behaves. And we'll call the second one semantics just so we can have a conversation. What do you want to call it? sure they're clear. Because there's a lot of... Okay, so let's go back. So... You know, actually, before we started the podcast, you and I were just, shooting the shit, and you were talking about how you can do projects now that take a week where in the past it would take a team and months. Can you maybe walk this through a little bit about how you personally kind of evolved in that? We'll do projects and then kind of the gains that you see. So a little bit about your stack and your process. Yeah. I use Cloud Code. Um... I am going to say something that's actually pretty funny, but I use Windows. What? Was that created by this company, Microsoft? And I use WSL. So I use Linux for everything, like development and so on. There's no like X or UI for it. I just use the command prompt. How do you use Windows? Because I just never really wanted to learn the keyboard shortcuts. Are you a gamer? It's like you tab over to Counter-Strike or whatever. Why else? I'm not hitting on Windows. I mean, I used it in the 90s. Really, like the only reason is I never wanted to learn the shortcuts. I hate the one button mouse. I hate the one button stuff. I hate the number of modifiers you have to type for shortcuts and stuff like that. I just never, I'm like, why? Why waste time? Okay. So I'm like, yeah, I'll install Linux. Use Windows, yeah. And so WSL, I use Cloud Code. What is WSL? That's Windows subsystem for Linux. Great. So they actually run the Linux system. [20:00] kernel within Windows in a hypervisor and provide you access to it. It's great. They've done a great job. Great. Windows WSL. And Cloud Code. And like My latest, the latest thing that I do is I use Opus 5.5. I tell it. Which is a really good model. It's an amazing model. Really good. And it's, I've shipped in the last two weeks, these massive 20 PR things that I just dump on my team to go and review. And they're like, oh. I'm like, don't worry about it. Sounds horrible. It's like the new reporting structure, like the people below review the PRs for the CEO. Is that where we become? No, everybody's like writing the same thing. Oh, everybody. Everybody's like writing. People's PRs too. Yeah, I review my people's PRs. And everybody's like just PRs flying around. Frowning in PRs. Yes. much more on the tests and doing things like golden expectations and are they shifting in different ways and so on. But Opus 5, and the trick that I've recently learned about is that instead of using sub-agents, you basically tell Claude to spin up command line sub-agents because those maintain their persistent session over time. And so you can have it spin up these agents separately. You can actually run them from other sessions and so on. And that allows you to be much more efficient from a context perspective. How many agents do you run? How many? How many? Agents. Agents. Typically, I work on like three or four session windows at the same time because I found that more than that, I... just start losing context between the different things because they start finishing faster and then they wait. And do they interact via the PR? Is that how it works? The sub-agents, like together? Yeah. No, they interact directly through the command line. Oh, wow. Okay. Yeah. So, like, I asked, like, the main thing that does the implementation to have a... tech lead reviewer and like they basically ask it questions does this make sense does this not make sense review my pr etc and so they iterate on these prs together um and then they have a pr doc that basically they work on with a design doc and everything in it like the first thing that i do is i ask that agent to do it ask the tech leads to review the pr doc i go through the the prd i go through the design doc make sure it makes sense go through each piece in like detail make sure that work, the division of labor works, the division between PRs and their relationships work. And then I thought it just. go start doing it. Figure out if there's like parallelization that you can do and then spin up other agents to actually go do them. But ultimately, you have to kind of put it all back in one stack and be able to go through it as a human one by one because otherwise it just gets messy and bushy and you can't really... tackle it you know So how much time, so can you give a sense of like a project you do now versus, I don't know, six years ago, like time differences? Thank you. So the latest project that I shipped was something like... By the way, this is like... to be clear, this is production code. Yeah. Yeah. This is production code. So whatever. So the statement you're about to make is the gains that you get for like shipping production. So I'll give you the example of the last, there, there, there's, The last PR that I worked on or like the last set of PRs was what we call auto promotion in a data lake. Yeah. So in a data lake, you store all the data and typically the data is column oriented, parquet files and iceberg and so on. And when you get data from logs, they're semi-structured, they're arbitrary columns and so on. In order for it to have the best layout of the data and to be able to query it efficiently and fast, the ideal situation is that every field in the log event becomes a column that you can filter on independently of the rest of the rows. And so we call that auto-promotion or shredding. Sure. Um, And doing something like that is incredibly complicated, right? Like you have to go and figure out how does the metadata system work? How do the commits of the metadata system work? How do you coordinate across different writers and so on? And, um... That took maybe about 10 days. In previous days, that would have taken maybe about six months for a single person. And was this just you? That was just me. It is insane. Yeah, sort of like factor of 10... Factor of 20 improvement. Yeah. [25:00] For production code for new products. Yes. You know, one question I always have is, like, I can very much understand... you know, these models are trained on all the code ever written, let's call it, right? So I can understand why they're able to generate code, which is the syntax piece. But then, you're doing core infrastructure. Let's say you're doing distributed algorithms. Like the properties... How it handles itself when it's running is very, very different. Often you just can't even look at the syntax to know what's going to happen. Let's say you're writing some distributed system, and then the way that it will perform, the availability, the correctness even, is going to be dependent on what the workload looks like. What are the access patterns? What is the traffic? performance aspects of it though or are you talking about like this like availability yeah you may have to lock things so it's not available for a period of time i'm talking about correctness like maybe you've decided to do eventual consistency and so like if you do a read it's not going to be consistent because there isn't a lock right um and then performance like you know maybe you cache maybe you don't cache so these are all kind of trade-offs in the design space right yeah and like unless you know like the access patterns you won't know how it's going to perform in these I just think to myself, okay, I've got an AI. I can understand how it will write code for a generic distributed data store, for example. But I don't understand how it will know to... make the trade-offs needed because it doesn't even know what the access patterns are. And it's not like these things can, like, where's even the data to learn how to, like, build a system that understands, like, all of the operations, right? Like, today we rely on human beings to, like, manage clusters and to, like, you know, optimize databases, et cetera. And so can you maybe talk about how you think of, like, I get the writing the code, but, like, you have these deep operational systems. Like, how do you apply AI to that problem? The question I think is... maybe about complexity. And to what extent can you take the things that you've described that are operational and that happen in production and convert them into something that an AI can reason about? Exactly. How can an AI reason about this system? Well, A, first off, to what extent do you see it writing in the code that it can actually predict what's going to happen? How good of a job does it do with that? And the second one is, okay, now how do we make first one. Like, if I use your method, the JAD method of writing code, and I built some big distributed system, like, How often does it actually get it from performance correctness? How often is it actually operationally pretty sensible? And by the way, I work on this distributed Gaussian splatter, and I would say it's shit. It's horrible. Because you've got millions of Gaussian splats that you've got to page in and out and stream. It just totally depends on the running system. None of the models do a good job. So in our world, what I've found is that we've defined basically really well what performance benchmarks look like and how the system should run and what the constraints of actually building within like. our stream processor system, look like to get the right performance. But every once in a while, it'll do, like, my latest project has been all about query optimization and query performance. And so I was like, hey, Classic example of something. I was like, we are basically generating code that corresponds to SQL and we want to run it in a pipeline. And we're like, okay, well, you're generating shit code. Fix it. Yeah. And it started basically going through. And what it didn't realize was that it wasn't actually doing initially any optimization on the code. It's like, oh, that's fine. It looks like what's the problem? And I said, well, look, there's these conjuncts over here. You're doing and and like this particular term is repeated and it should have been taken away. And then it's like, oh, OK, let me figure this out and so on. all the things that it can go and optimize to make this better. Yeah. Um, But ultimately, because it was able to ground that in performance benchmarks, and I was like, look, here's a data set of the, queries and like what the users are asking and so on, go work on it. And so it can basically do a relatively good job. that way. But when it comes to saying something around like, How much memory should I give the system? Yeah. Ed has no idea. And that has to come from actual experiments [30:00] that run in production. And so what I think actually these systems seem to be doing well at this point in terms of operational stuff is that it knows that, hey, I probably won't be able to answer this question. Let me run an experiment somehow. And come up with that information. But of course, when it comes to operations, like production systems don't behave the way that you expect. And ultimately, that's why incidents happen. So have you found, are you doing any of like the AISRE stuff where you actually have a running system that has logs and you feed it to an AI and that kind of helps you like tweak it? Yes. And how well does that work? So we're just... Because anecdotally, my experience with other people doing this has not been that great. Like the reports I've been getting anecdotally have not been great. So a couple of things. The first one is we're doing that for both operational pieces and for security. But the way that we're tackling it is not the way that other people are doing it from an AISRE perspective. A lot of those systems basically start from a lagging indicator like, "Hey, this metric spiked," or like, "This alert happened." go back through a very complex system and try to determine why that happened. Right. Like, yeah, sure. That's why those don't work, because backtracking into a root cause is actually very problematic. And so instead, what we're doing is doing it the other way around, which is, can we actually start by figuring out if there's something different in your environment that has changed? Can we do a large-scale clustering, a large-scale figure out a needle in a haystack, something that is different? So you're assuming that it was working perfectly. fine before and now it's not so something happened But it could be just like a change in like normal. traffic patterns, right? That's fine, too. That's fine, too. And so the thing that we're looking at is like when things go wrong, usually the system will tell you something went wrong. somehow. If it's working fine and scaling as normal, nothing will happen. But sometimes you'll see some message which says, "Error, connections ran out. Or memory exceeded or resource buffers emptied or like some kind of indication, like some kind of really early warning, like something happened. And that's the one message that we want to capture and then go from there and see. Is that an actual problem that we need to investigate and deal with? So how does it do? So far, it's doing pretty well. I mean, we are running it internally in our systems. We run it, we ran it in a couple of our customers' environments. And In the cases where there was a couple of cases where we ignored it, And those turned into production incidents. Really? Oh, wow. You know, it's kind of just from an intuitive perspective, like I can understand how these models understand syntax because, again, like I said, they're trained on all of the code ever written. And it just turns out that covers a lot of useful cases. It's hard for me to actually even think of a bound on the actual operations of software. Right. I mean, it feels like, you know, this is a ridiculous statement, but it's actually kind of true. operationally, it increases by a factor of two. So you very quickly get to more... states than atoms in the universe. And arguably this is why – this is why product approaches to operations have never really worked, right? The chaos engineering stuff never really worked, and we've always kind of relied... Actually, the dirty secret for the longest time of running a distributed cluster was just human beings managing it. So I'm just wondering... And then in order to... even training AI to do it, you have to have like enough data and enough configuration. So I'm just kind of curious, do you think we're at a spot where you could use AI to run a distributed cluster or run one of these services, or do you think that's still on the come? I think that there's enough to be able to... have it start that loop. Right? Like, I mean, Ultimately, I think that I agree with you. There's infinite space. There's a lot of different things. But ultimately, humans also operate on runbooks. And like sure there's a lot of things that happen in the in production that you ultimately need some human to go and like hey this is too complex. There's like some state that's hidden that I know of because I talked to this other guy like at the water fountain the other day and he told me about this thing he deployed. Like there's a lot of stuff that potentially isn't visible in the state of the system. But I'll argue that like. The goal of what we need to be able to do is to get... to have enough telemetry and enough understanding to like make everything really visible from a, hey, here's what the system looks like. Here's how it's working. [35:00] Here's like whatever. And then I think ultimately most of the issues that end up really showing up in operations, assuming you have the system previously running okay, is due to some kind of change. And if you're able to just focus on the goal of figuring out what the change was and kind of going through... Sorry, can I just ask a stupid question on this, which is I never think of these generative models like good at predicting stuff. And it sounds like you're talking a lot about predictive modeling. Like, you predict a change or you baseline and you detect an anomaly. Like, am I thinking about this wrong? No, you're not. Because the goal is not actually having the agent try to predict, but have it get a signal that something changed. Oh, that's deterministically. That's something. I see, I see. Yeah. Wait, wait. So there is a separate model that does the predictive stuff and then that will alert the AI? Yes. Okay. Yeah, so what we do is we basically, you give us your petabyte of logs per day, and we scan all of them and every time we see a new kind of pattern emerge in your logs a new thing that we haven seen before we lift it up to the AI I see So that a predictive model that actually does what predictive models are good at which is determining out things or whatever whatever Exactly. And it goes to the AI, the AI kind of uses that to figure out what to do. Exactly. So the AI basically can then reason and drive that down. So it can say, oh, this message, let me actually look at the code and figure out where it comes from and understand what the code does. let me see if it's actually happened before let me see so it basically runs all these hypotheses and heuristics to try to understand does it make sense does it meaningful in some kind of way and then it can determine what to do with it so how far away do you think we are for it let's just go back to query optimization not like logs or something like so how far away do you think we are from having an AI basically do the job that a human used to do when it comes to query optimization I think very close really yes wow I think running a distributed cluster I just think of like, okay, you've got like James Cowling, PhD from MIT, who runs a distributed cluster and like, if the cluster's down, like the company doesn't work. And then if it's up, it's running. I don't know, like, are we at the point where you would say that an AI can do that? No, not yet. Okay. I think it I think I mean I think like very very simply when we started doing doing AI for coding it was autocomplete And nobody really knew to what extent you can actually build complex software using AI And when we were looking at it, we were like, AI is never going to be able to build X or Y or Z or whatever. And then it basically broke through the expectations because we were able to model more and more complexity in the systems. I just feel like I'm a very simple thinker on this stuff. And to me, these models are distribution in, distribution out. And it turns out you can actually train on all the code ever created. And it's actually like that manifold is pretty manageable. And I just don't know if operations is constrained in the same way. That feels to me more like kind of like the universe, like heavy-tailed and exception-based and power laws and all the tough stuff for these models to learn. But I don't know. I mean, you're the one that's in the field. So maybe that's not the case. Maybe they're manageable. But I will tell you, the history of systems, as you know, is you always want to build a smart algorithm that does the efficient thing. The smart virtual memory management, the smart congestion control, the smart routing. And what do we do instead every single time We build a more complex thing No we over every time Every single time we build uh we over provision we know we over provision every time no literally we every single time like like remember like swap memory where you like swap to disk dude if you did that you were fucked like that's it your computer was dead so like we literally over provision ram and like routing you know how many like adaptive routing algorithms there are like we will respond to traffic but you have oscillations like fuck it we over provision we just use ecmp you know so i just feel like the history of systems is that you don't predict stuff you over-provision because you can't. And I feel like, I mean, you know, so I feel like in the operations side, so either we build this, you know, we will build the systems up front that are over-provisioned so you don't have to, like, do kind of complex operational stuff. Or maybe AI can do it the way that a human being. But I think it's very different than code because you're dealing with, like, exceptions, there's not a lot of data, et cetera. I just feel like a different domain. But I don't know. I'm a VC. I'm just like... You know, I kind of, like, the reason that I'm more optimistic here is because I live in the world where like what we do is actually deal with the observable world. Like I was gonna say you're the literal expert on this. So it's silly for me to even like ask you these questions. No, not at all. Because like in my world, we like my assumption is that we have all the data. Is that like we know exactly when like [40:00] a new thing happens or like an exception emerges or like a workload changes or like whatever. Like in my world, all of that has already been collected and streamed through. Yeah, but the model that is not trained on a grounded source for that stuff. No, no, you're not. That would be really hard. Right. But the story here isn't like, hey... Mr. Claude Here's a bunch of logs. is something going on, right? And it has to be much more able to run experiments and kind of dive in and see, Like, what are the different shapes of possible causal chains that actually fit in the environment that exists? I see. So, yeah, this is what you said before. Now I see what you're saying. You're saying that, like... the universe is heavy-tailed. The number of bad things that can happen is probably an exception thing, but the set of steps used to debug it is a manageable thing. Exactly. That's what you can teach one of these models, and that's what they have. And therefore, if you give it the right signals, it can go through a number of steps to actually constrain this problem. Exactly. Interesting. Exactly. And so it really is taking the role of the human. Because remember the whole chaos engineering wave? Yes. And the reason it didn't go anywhere is you're very so quickly, kind of like, you know, reduce like a state space that was atoms in the universe like super quickly like an act to make that manageable and it was like the human being that would go through steps that would do that on the actual debugging and this is what you're saying the AA would do yes Right. So the goal isn't to like have the model just be like, this is what happens. It's, hey, run through this workflow. to determine what happened. And like we're trying that also on the security side, for example, where like, hey, in security, You have a something different changed in your statistics, like some behavioral difference in terms of what entities exist, the cardinalities of various things that exist. And so that becomes a perturbation in the universe. And so we take that signal and we feed it into an AI and we're like, hey, AI. come up with a bunch of possible causal chains that exist, and then go through and verify if any of those fit the pattern of the data that we actually see. And so that becomes a much more like, hey, here's a, here's like a, I don't know, like fit this square into find the peg or like find the hole that fits this peg. And that seems even harder to me because the operations one, the argument is the system can get into broken states that are kind of like very rare. And there's a lot of those rare states. And so it's just not something that you could train a model on just because there's so many the steps. The security one seems even harder because this isn't just like the natural order of rare states. You actually have an adversary that is trying to intentionally find rare states and intentionally find stuff that you can't do. And so it may be the case that code becomes solved. It takes a little while to do operations. It takes a lot longer. Also think about, I mean, I don't know. To me, the whole world is sort of quantized to a certain extent. The Planck length. Is that what the physicists say? I don't know. I'm not a physicist. I think there's like, actually, I literally don't know what I'm talking about right now. But I literally actually think that like, like, like space time is discretized into a Planck length. But yes, I mean, I agree. Like, to a certain extent, what you just said, but for operations and systems in general. Yeah. Like, there is a discretization of like. the possible states and the past through them that are possible. Yeah. I just think when you're dealing with There might be a ton of them. There might be a ton of them. I'm going to tell you, remember Sundar who was in our grad school group? Yeah. Send their air. So I remember he came to me once and he was looking very sad. He's like, Martine, I just had this really sad realization that, you know, if you take any MP3, remember MP3? Yes. He said, he said MP3s have a finite, let's say you know four minutes it's like four megabytes you know and that means there's a finite number of songs in the universe that can no because you know because it's it's all made of it's all made of bits right and it's four megabytes and that means you can you can just enumerate every one of those concerts i'm so so i'm so sad in this notion of like the the finite of music right literally this is like we're walking on the dish when he had this conversation you've like overcome like the atoms in the universe you haven't even gotten to a k right it's like two to the two to the two to the [45:00] Dude, like, what are you talking about here, man? Be happy. Two to the two to the two, four megabytes. Four megabytes. What are you talking about? Like, there's so much state. Like, be happy. That's effectively infinite, right? It's like, what are we talking about here? So I do think at some level, like, we do underestimate, like, the number of states that, like, you know, systems can get into. And, like, maybe, I will tell you from the perspective of the VC that looks at these companies, It seems much, much tougher than those that are tackling the code because I think it's a much tougher problem. 100% I agree it is a much tougher problem and and it's not just because like the systems themselves are you know there's like external loads that are random that kind of happened to you or that you know somebody just shipped a new thing and rolled it out into production didn't tell anybody or like there's some state that actually happens when like this thing happens and this thing happens and then you end up in a deadlock. who's the coder, it actually also involves... Oh, the pools of knowledge are distributed too. Exactly, and the interactions between what it takes to connect the pieces and put them together. But ultimately, I mean, I think about where AI is and what are the possible futures and the depth and the forward-looking capacity of all the things. I think code is just... To me, it's a microcosm of like, maybe I'm just too fatalistic. Maybe I'm just like too pessimistic. pessimistic about how the world works and and I think of everything as like following some kind of probabilistic path. And so to me, like, If we can make it work in code, we can continue expanding that level of complexity bit by bit and kind of adding layers and layers and layers until we can handle those problems. A lot of people say it's not fatalistic, it's the opposite. It's very optimistic, is what people would say. For me, I don't think the universe gives up its secrets that easily, but we'll see. I mean, this is kind of the big question. And I would say... As yet, it's not an answered question. No, no. But I don't know. A lot of things that as humans we think of as – being random or... impossible to tackle. Yes. And that's what we saw with coding. Initially, people thought the possible world of all these things that you can implement in code are kind of limited, and you can't really get a system to potentially understand the full complexity of a human brain that's required. I think over time, it'll all... It's so funny that I'm actually telling you this and you're like the VC who's looking into the future and being like, no, well, I mean, I'll tell you my view. My view is like the study of these models are you train on data and either that. And that, you know, the... So you train on data and that represents a distribution, that distribution is useful or not. And so when you train on language, the fact that it's useful, that to me says that humans are so remarkable. Because they've looked at the universe, which is a very high-dimensional system, and they've reduced it to a manifold that's actually useful for a model, right? Yes. Right? But that's different than the universe. Like, there's a reduction from this high-dimensionality to the language is less dimensional. And I think that code is a reduction of all the operational states of a machine into, like, the structures that make it run. And so, to me, there's still a disconnect between, like, the instructions or the description and the actual, like, the actual running of the universe and actually running the system. we can make gains toward it. But what I will say is you're kind of converging on simulation. There are things that are computationally irreducible. You kind of have to run it to get the answer. You can't just predict it. And so maybe we just are at an event horizon where models will solve anything that you don't have to actually simulate to get the answer from. But why? Why do you have to just think about it in terms of, hey, it'll have to answer the question now as opposed to being able to simulate and being able to plan its own journey? execution to gain the information that it needs in order to be able to. Yeah, maybe that's where we're going. That's kind of what I'm. Fair enough. And I think like there is a lower dimensional space, which is like these steps that you would take that we can reduce a whole bunch with AI. And then whatever steps that you need that that can't do, it can create the code to do or it can simulate. We need to go and fine to like model the universe into its world bigger. That's [50:00] Probably, I mean, That's probably true. But can it actually simulate a scientist? sufficiently to be able to get to that point. Maybe. Maybe. Cool. Can I ask you a question? How much of the VC work do you think is something that... an AI can drive. So my model for VC is it's very relationship driven. I just have a belief that markets are actually pretty efficient. And so it's not about finding the company that nobody else finds. People all find the right company. I think they're efficient. And then I also have the model VC that I'm... you know, tech is positive and I'm kind of optimistic about all tech and I don't worry about TAM. And so I I think the key is access. It's like raising money, it's access, it's ability to win deals, and it's ability to add value after that happens. And a lot of that is like stuff that, startups just can't naturally do like, you know, the, recruit big execs or help with that. And so... the way that I think about venture, you kind of have to be a human being to do. It's like, you know, um, yeah, Because that's the actual job. So for that, I don't think like an AI... If you gave an AI simulated universe where it could like, you know, like whatever reward hack, like building trust relationships, fine. But I think that that would have to be... That said, I will say I have seen analysis... done by AI that are as good or better than any, you know, gene analysts I've ever worked with. Like, so for the analysis piece, they're great. But I think this is more an indictment on VCs' ability to, like, analyze than it is, like, how powerful, you know, the models are. I just think it's all a kabuki show and bullshit. I just think the right VCs are, like, you know, you know, tech is net positive, you know, TechWaves always add value. Let's find the best company and invest in it. The markets are very, very efficient. And so I just feel like the answer to this question is almost kind of a technicality. My answer is almost a technicality. you know, we started talking specifically about ops. Yeah. But to me, like, there's this world of AI writing code, which is kind of like this abstract simulation world that it lives in. Yeah. And then there's the rest of the world. Yeah, exactly. And the entire discussion is this. It's like, there's like this, there's like the AI, the world that the AI lives in, which is all digital, and like whatever it is, the training data, the in-distribution world, and there's the actual world. And like, those things are not necessarily the same. Right. They're quite often not. And so the question is like, To what extent do you need to train an AI to, to be able to survive in a world? And like, what are the things that it actually, I don't want to answer that question. Like, what do you need to train? But like, I'm curious about the open spaces. So we talked a little bit about operations. You said, oh, I don't know if it's going to be solved or not. I'm like, yeah, Martina, it's going to be solved. Yeah. And... In my mind, there's so many other things. For example, self-driving cars. For a long time, we thought that that's not going to happen. But it's clear that it's actually going to be done. And it's going to happen. It's just taking time because of a lot of different things. And so what are the... What are the other next... stages, the next areas that you think are kind of like lined up right after code. I mean, my sense is, you know, AI is going to continue to take off in a wildfire for a constrained domain where the solution space is in a constrained domain, right? Where the distribution of the solution space is the same as the distribution of the training set. The distribution of the syntax for code is the same as the distribution of the training set. The distribution of the generated images is the same as the distribution of the training set. In those areas, I think it's going to continue to take off as a wildfire. natural world, I think it's going to, you know, have, you know, it's gonna be much harder to take longer. And I think the economic equivalency is going to be much more difficult. Self-driving is a great example. I remember when Sebastian Thrun won the DARPA Grand Challenge when I was at Stanford. Remember that? And we're like, hooray! You know, self-driving cars is around the corner. Seriously. And it took 20 years and $170 billion of investment to get to a worse unit economics like solving real problems where you have to deal with the distribution of the real universe is just much, much harder. And so I view AI as this general [55:00] tool. It's a very effective tool. It's one where like you have a distribution of data and then hopefully you can apply that to like, you know, some problems and we're going to apply it to a bunch of stuff. And I think economic equilibrium is going to determine what works and what doesn't work because I think it's a very, very general tool. It's like a universal function approximator. And wherever that nets out, I mean, again, you know, one of the things that I said earlier was like how AI was training humans to like use it. Yeah. And I very often think about like, Some of the problems that seem intractable is because the world is kind of that way. And then we're trying to fit AI or the tools that we're building to manage the world as it is. But in some situations, we actually tend to change the world to fit to what we can actually do. Oh, 100%. Of course. I think this is the story of robotics. Yes, exactly. It's literally the story of robotics. The automated data warehouse. And so the question, like that always comes to my mind, like when we were looking at these data warehouses and whatever. Like, you know what? Just automate the whole fucking thing. Yeah, yeah, yeah. And... And I wonder to like what extent as we kind of, improve or change how we, or kind of expand the domain of what we want to solve with AI tools and so on, that we actually start modeling the world or changing the world and the things that can actually happen to fit the, what AI needs it to look like. Yeah, I mean, I think that right now a lot of AI just operates in the digital world. Yeah. And for that, it's like literally the solution set is the same distribution as a training set. Yeah. In a sense. And so that's fine. And I think that, again, like in the case of equilibrium, like I think like when you're dealing with the real world, like you can solve any problem you want. It's just a matter of like compute and money. And that the equilibrium point is going to be an economic one. One way you can impact that function that you're doing equilibrium over is to constrain the environment, constrain the system. And so I think that we're going to like we're in now in this meta economic equilibrium flywheel where we're kind of optimizing. This is super general, but very inefficient process with a solution set that people value. And I think we've got 30 years of this and a lot of value is going to be created. You want to hear a funny story? Yeah. built this AI thing and we're like talking to people. We're like reaching out to customers. We're like, hey, we built this AI thing. You know, we're helping customers become more reliable. Here's like what we're doing. They're like, great, we want to talk to you. And we go on the call and we're like, hey, here's what we do. You know, we built this system, helps you optimize your data, helps you reduce your costs. And then it helps you become more reliable because now we can find these problems and solve them before they hit your incident. They're like, wait, wait, wait, wait, yeah that's what I want to buy but the AI thing no no no no that's what we want to buy right now and it's it's interesting like a lot of these uh I always think about it, it's like, oh, for code, people just get it. They're like, oh, shit, it just works. And on the ops side, or anything that's beyond that. Yeah, it's very tough to understand the actual value and put a dollar value to it and pay out the door 100%. And I also think from a coding standpoint, it literally is engineering productivity, which is one of the highest line items on these things. So it's just very straightforward. Exactly. But you can also think about it, it's SRE productivity, or it's like your business going down, maybe you can save an incident. cost of, I don't know. It's weird. I think like how people think about it outside of the I also just think this code was so magical because it was so good, and nobody thought that could happen. And I think the operations stuff now, it's still pretty early. All right. Well, this is a great chat. Thanks, Jeff. Thanks for coming in. Thanks. Thanks for listening to this episode of the A16Z Podcast. If you liked this episode, be sure to like, comment, subscribe, leave us a rating or review, and share it with your friends and family. Go to YouTube, Apple Podcasts, and Spotify. Follow us on X at A16Z and subscribe to our Substack at a16z.substack.com. Thanks again for listening, and I'll see you in the next episode. As a reminder, the content here is for informational purposes only. It should not be taken as legal, business, tax, or investment advice, or be used to evaluate any investment or security, and is not directed at any investors or potential investors in any A16Z fund. investments in the companies discussed in this podcast. For more details, including a link to our investments, please see a16z.com forward slash disclosures. Thank you.