Tim O'Reilly Says Enterprise AI Stalls on Workflows, Not Compute

蒂姆·奥莱利：企业AI落地受阻于工作流程而非算力 由于局部提示词效率无法形成复利效应，企业AI的应用陷入新的索洛悖论，迫使公司重构人员工作流程，并将操作失误制度化为共享的智能体基础设施。 蒂姆·奥莱利深入剖析了现代AI时代的索洛悖论，指出企业生产力的提升取决于对业务流程的重新设计，而非单纯依赖原始算力。结合网络安全公司 Trail of Bits 的实际落地案例，他详细阐述了如何消除员工的身份威胁认同、建立分层能力阶梯，并将操作失误转化为标准化基础设施。他进一步解释了 O'Reilly Media 如何通过模型上下文协议（MCP）服务器将人类专业领域知识融入自主智能体，从而利用成熟框架指导管理层决策。奥莱利最终强调，企业若要建立差异化竞争优势，必须依靠经验丰富的操作人员引导模型突破通用的平均分布，从而解决具有重大影响的组织挑战。 生成式 AI 面临现代版索洛悖论（Solow Paradox）：尽管资本大量投入，但由于组织流程与习惯必须缓慢且刻意地重构，生产力提升依然受阻。 (0:04) 理查德·萨顿的“苦涩教训”（Bitter Lesson）表明，通用算力和搜索始终优于人类精心设计的领域规则；企业必须从浅层的“AI 辅助”工具转向从底层重构问题解决流程的“AI 原生”工作流。 (2:41) 个人 AI 生产力无法自动转化为组织能力，企业需要建立系统性机制将隐性知识与个人提示词沉淀为可复利积累的共享基础设施。 (9:04) 员工抵触情绪源于身份认同受挫及对 AI 错误的零容忍，管理层需以身作则，将 AI 定位为专业能力的放大器，并通过部门技能等级阶梯激励文化转型。 (13:11) 结合伊森·莫里克（Ethan Mollick）的“领导层、实验室与大众”模型，企业可通过每两个月一次的复利黑客松沉淀长效资产，并由专门的中央团队将自下而上的实验标准化为可复用的默认配置。 (23:38)

Tim O'Reilly argues that enterprise AI transformation is stalled by a modern Solow Paradox: while raw compute routinely outpaces manual rules, organizations fail to capture productivity gains until they overcome human resistance, eliminate organizational debt, and restructure workflows around compounding institutional expertise. The Bitter Lesson and Interface Traps (2:41) O'Reilly links stalled enterprise AI productivity to the Solow Paradox—where IT investments took years to surface in macroeconomic statistics—and Richard Sutton's "Bitter Lesson," which demonstrated that compute-driven methods reliably outperform human-curated domain heuristics. In the 1990s, manual web directories like O'Reilly's Global Network Navigator (GNN) and Yahoo were overtaken by Google's dynamic indexing. Technologists continue to fall into interface traps by grafting new models onto obsolete paradigms, as O'Reilly Media did in 2018 by applying Google's BERT to an incremental search interface before ChatGPT demonstrated conversational direct synthesis. To capture value, O'Reilly contends enterprises must transition from "AI-assisted" tools to "AI-native" workflows that re-architect organizational processes. This requires resolving the handoff bottleneck, where personal efficiencies and prompt context remain siloed inside individual accounts rather than compounding into shared corporate capability. Confronting Cultural Inertia and Identity Threat (13:11) Drawing on Dan Guido's work at cybersecurity firm Trail of Bits, O'Reilly argues employee psychology presents a steeper barrier than technical capability. An internal audit at Trail of Bits revealed that even with executive leadership modeling use, only 5% of staff actively drove adoption, while 20% actively resisted and 70% passively resisted due to identity threats, self-enhancing bias, and intolerance for AI errors. When skilled professionals anchor their self-worth to manual syntax or inspection, automated tools provoke defensive resistance, creating what Harper Reed termed "the new Amish." Trail of Bits neutralized this friction by shifting messaging from automation to craft amplification: positioning AI not as an appliance that cooks for you, but as tools making practitioners "better and more dangerous." Guido reinforced this with a department-specific status ladder tying career advancement to AI capability, treating Level 0 resistance as an explicit alignment conflict while rewarding Level 3 contributors who build compounding, reusable enterprise assets. Turning Scar Tissue into Reusable Infrastructure (23:38) To turn individual experimentation into durable enterprise capability, O'Reilly highlights Ethan Mollick's "leadership, lab, and crowd" model. Grassroots crowd experiments surface friction, while a dedicated lab codifies working defaults into shared repositories. This approach eliminates "prompt debt"—brittle, bloated prompts that fail across teams—by systematically turning failure scar tissue into global defaults. Autonomous agents also expose historical "permission debt" and fragmented data layers, demanding clean data pipelines and access controls. Trail of Bits operationalized this through bi-monthly paired hackathons—advancing from repository hygiene to autonomous agents—and securing workflows with sandboxes and seven-day cool-downs on external packages. Trail of Bits reported that client bug resolutions jumped from 15 to 200 per week, AI agents surfaced 20% of reported vulnerabilities, and sales representatives generated an average of $8 million in revenue against an industry benchmark of $2 million to $4 million. Expert Steering and Technology Diffusion (34:56) O'Reilly rejects the view that compute scaling renders domain expertise obsolete. Using a horse-and-jockey metaphor, he argues raw models provide baseline horsepower, but human guidance pulls models out of the generic average distribution. When O'Reilly Media used an AI agent to assess an engineering hiring request, the agent initially produced a standard report detailing ticket volume and capacity. Hooking the agent to an O'Reilly Expert Model Context Protocol (MCP) server transformed the diagnosis: grounding its evaluation in Google's Site Reliability Engineering framework, the agent identified that the team faced roughly 60% operational toil and advised process restructuring instead of hiring. Referencing Technology and the Rise of Great Powers, O'Reilly stresses that historical ascendancy belonged not to societies that invented leading technologies, but to those that facilitated widespread organizational diffusion. O'Reilly concludes that enterprises must look beyond narrow margin gains and raw model benchmarks. Strategic advantage belongs to organizations that build institutional defaults, clean data foundations, and expert context, redirecting human effort from trivial tasks toward larger, consequential problems.

Tim O'Reilly 指出，企业应用生成式人工智能（AI）正遭遇现代版的“索洛悖论”：仅靠算力与通用模型无法自发带来生产力飞跃，决定竞争优势的关键在于重构组织流程、化解文化抗拒，并将隐性专业经验沉淀为可复用的企业基础设施。 苦涩的教训与索洛悖论的重现 (2:41) 强化学习学者 Richard Sutton 提出的“苦涩的教训”（The Bitter Lesson）表明，AI 历史上依赖人类专家经验和手工规则的方法，最终都会被通用算力和大规模搜索所取代。从 1990 年代 O'Reilly 的 GNN 与雅虎的手工分类目录被谷歌动态计算索引击溃，到 2018 年基于 BERT 的“O'Reilly Answers”搜索界面被 ChatGPT 的直接对话合成颠覆，技术演进一再印证了这一规律。然而 O'Reilly 强调，企业战略不能被动等待算力奇迹，经济学家 Robert Solow 提出的“计算机随处可见，唯独没有出现在生产力统计数据中”的悖论如今正在生成式 AI 领域重演。历史上的工业革命表明，地缘与商业竞争的真正赢家往往不是最早发明领先技术的实体，而是最能降低技术扩散门槛、彻底重组业务流程的组织。 化解身份认同威胁与梯级推广体系 (13:11) 技术落地最大的阻力并非算法能力，而是员工心理。在网络安全公司 Trail of Bits，CEO Dan Guido 进行的内部审计显示，仅有 5% 的员工真正积极拥护 AI，20% 处于明确反对状态，约 70% 则是被动抵触的“暗中抗拒者”。前奥巴马竞选团队 CTO Harper Reed 观察到行业中涌现出类似“新阿米什人”的技术群体，他们出于工匠自尊坚持纯手写代码、拒绝机器辅助。Guido 借鉴高端厨具营销经验发现，若将工具宣传为“代替你做饭”，会遭到资深爱好者的抵触；但若定位为“让你成为更优秀的厨师”，采纳率便会大幅提升。Trail of Bits 据此将 AI 重新定义为让员工成为“更强大、更具杀伤力的安全审计员”的赋能工具，并建立了透明的等级阶梯：将第 0 级的抗拒定性为组织目标对齐问题而非技能短缺，向上设立第 1 级个人实验、第 2 级团队工作流重塑以及第 3 级构建组织级持久复用能力的完整晋升机制。 消除提示词债务与建设复用工程体系 (23:38) 孤立的个人效率提升并不能直接转化为组织生产力。O'Reilly 以自身为例说明，他在 Claude 工作区中精调的摘要工作流一旦移交给下属便立刻失效，原因在于上下文、偏好和文件权限等隐性知识被锁在个人环境里。为打破这种壁垒，Trail of Bits 践行沃顿商学院教授 Ethan Mollick 提出的“领导层、实验室与大众”（Leadership, the Lab, and the Crowd）协同框架：高管率先公开使用，员工广泛实验，而专属实验室团队负责将临时修复和试错经验沉淀为系统化默认配置。这种做法能有效遏制作家 Drew Breunig 所指出的“提示词债务”膨胀，即避免模型适配规则演变为臃肿脆弱的个人习惯。团队每八周举办一次目标清晰的复合型黑客松，第一期专注于“海滩清理”，清理遗留代码和统一开发环境，第二期则聚焦自主代理与 MCP 集成；同时通过设置沙箱环境和强制执行 7 天外部开源包冷却观察期，防御供应链安全风险。 专业知识接入与代理驱动的组织重塑 当企业引入自主代理时，过去数十年在数据治理上的走捷径行为将集中爆发为“权限债务”，阻碍自动化推进。前美国首席数据科学家 DJ Patil 指出，搭建统一环境与元数据访问控制等基础数据工程是代理运作的前提。重构流程后，Trail of Bits 的漏洞修复速度从每周 15 个提升至 200 个，20% 的漏洞报告由 AI 自主发现，单名销售代表的年均创收达 800 万美元，远超行业 200 万至 400 万美元的平均水平。针对代理生成的分析报告普遍存在“详列数据却缺乏决策指导”的问题，O'Reilly 团队开发了“O'Reilly 专家 MCP 服务器”。在一次工程团队增加人手的申请中，该工具接入谷歌 SRE 运维体系后，诊断出团队存在约 60% 的结构性无谓劳作（toil），从而指导企业通过流程再造而非盲目扩招来解决问题，成功将深层人类专业知识转化为驱动智能代理的企业级基座。 O'Reilly 总结道，基础大模型仅代表底层的基准动力，真正的差异化来自操作者将其“推离平均分布”的驾驭能力。企业在借助 AI 解决琐碎边缘效率问题的同时，必须警惕被微小胜利所局限，而应将释放出来的生产力转向重构核心流程与攻克更重大的战略挑战。

## Background 该内容由频道 Ai4 于 2026 年 8 月 18 日发布在 YouTube 平台上，演讲者为 O'Reilly Media 创始人兼首席执行官蒂姆·奥莱利（Tim O'Reilly）。简介指出，企业落地 AI 战略最难之处在于割舍旧有模式并直面理查德·萨顿提出的“苦涩的教训”（The Bitter Lesson），奥莱利在演讲中提出了一套包含六大维度的路线图，旨在指导企业调动全体员工经验、驾驭 AI 浪潮并实现战略优势。 ## TL;DR 蒂姆·奥莱利结合 O'Reilly 自身从图书出版、会议到学习平台的转型历程，深入剖析了企业在 AI 时代遭遇的新“索洛悖论”与技术扩散瓶颈。他指出，算力最终会碾压特化规则，但企业胜负的关键不在于模型能力本身，而在于组织架构重组与落地采用。借鉴网络安全公司 Trail of Bits 的实战经验，他系统拆解了化解员工心理阻力、搭建地位阶梯、举办能力复利黑客松、治理 Prompt 与数据债务的落地路径。最后，他强调人类专业经验在引导模型跳出平庸分布中的不可替代性，呼吁企业将“组织伤疤”转化为基础设施，腾出双手去解决更宏大的世界级难题。 ## Atmosphere & style 整场演讲充满了一位硅谷思想领袖兼行业老兵的深沉历史纵深感与务实洞察。奥莱利的语调沉稳、睿智且极富启发性，既没有被科技圈狂热的“AI 药丸”叙事所裹挟，也没有陷入保守悲观，而是用极其接地气的企业转型痛点引发听众强烈共鸣。从技术演进的宏大规律到工程师配置报错的微观细节，叙事兼具思辨高度与实战操作感，收尾处引用里尔克诗句更是将技术讨论升华为对人类卓越追求的精神号召，带给听众一种豁然开朗且充满行动底气的感染力。 ## Key takeaways - [00:04](https://www.youtube.com/watch?v=WGl-myLsPSM&t=4s) 企业的 AI 落地遭遇了新时代的“索洛悖论”（Solow Paradox）：尽管外界大肆鼓吹 AI 的指数级增长与极速颠覆，但企业实际改造业务流程的过程依然痛苦而缓慢。 - [02:41](https://www.youtube.com/watch?v=WGl-myLsPSM&t=161s) “苦涩的教训”（The Bitter Lesson）是整个科技史的底层演进法则：依靠通用算力与暴力计算的动态系统，最终总会超越人类精心构建的手工规则与静态分类体系。 - [06:11](https://www.youtube.com/watch?v=WGl-myLsPSM&t=371s) 企业正从“AI 辅助”迈向“AI 原生”：不能用旧软件思维让 AI 辅助生成传统资产，而应以对话与智能体交互为核心重塑原生业务流。 - [09:04](https://www.youtube.com/watch?v=WGl-myLsPSM&t=544s) 个人 AI 技能与 Prompt 难以在组织内直接推广，核心瓶颈在于大模型上下文无法共享、企业权限割裂以及深藏于业务流中的“隐性知识”壁垒。 - [11:55](https://www.youtube.com/watch?v=WGl-myLsPSM&t=715s) 企业 AI 采纳分为辅助、增强与原生三个层级；真正的 AI 原生旨在建立“能力复利系统”，让每次基于 AI 构建的组件都沉淀为全公司的底层资产。 - [13:11](https://www.youtube.com/watch?v=WGl-myLsPSM&t=791s) 推进 AI 最大的障碍不是技术而是心理与身份威胁（Identity Threat），许多员工和技术专家将自我价值绑定在具体技能上，容易演变为抵制 AI 的“新阿米什人”。 - [16:21](https://www.youtube.com/watch?v=WGl-myLsPSM&t=981s) 组织转型应将 AI 定位为让员工“成为更出色的专家”，并通过公开透明的“地位与能力阶梯”将 AI 深度采纳与员工晋升、声望明确挂钩。 - [19:44](https://www.youtube.com/watch?v=WGl-myLsPSM&t=1184s) 推行企业级 AI 必须先“清理海滩”：理顺内部混乱的基础设施、统一开发环境与数据规范，以避免智能体在配置错误上产生无谓消耗。 - [23:38](https://www.youtube.com/watch?v=WGl-myLsPSM&t=1418s) 采用“领导层、实验室与大众”模型推动组织落地：领导层身先士卒，全员自发实验，“实验室”核心团队负责收拢零散经验并打造全公司标准资产。 - [25:38](https://www.youtube.com/watch?v=WGl-myLsPSM&t=1538s) 将组织“伤疤”转化为基础设施：治理因修补 Prompt 导致的臃肿债务，把每一次踩坑沉淀为全局可复制粘贴的默认配置，消除新人的上手门槛。 - [28:36](https://www.youtube.com/watch?v=WGl-myLsPSM&t=1716s) 构建三级技能仓库（内部、公开、精选）并配置安全防护网，通过多重沙箱、依赖包 7 天安全冷却期以及定期 Agent 评测确保系统稳健。 - [30:18](https://www.youtube.com/watch?v=WGl-myLsPSM&t=1818s) Agent 落地最致命的瓶颈是“权限债务”与数据底座缺陷，必须通过统一环境与精细化访问控制等扎实的基础功课破除数据壁垒。 - [33:40](https://www.youtube.com/watch?v=WGl-myLsPSM&t=2020s) 在 Bitter Lesson 时代，人类专业领域知识仍具不可替代的价值，企业的核心命题是将专家的深刻洞见转化为智能体时代的基础设施。 - [34:56](https://www.youtube.com/watch?v=WGl-myLsPSM&t=2096s) 接入外部专家智能（如 MCP 知识库）能让 Agent 从“列举内部指标的浅层报告”升维为“具备结构性归因与明确执行路径的决策指导”。 - [38:53](https://www.youtube.com/watch?v=WGl-myLsPSM&t=2333s) 历史表明决定技术革命胜负的核心在于“技术扩散与组织重构（Adoption）”，而非单纯追求更强更快的模型算力垄断。 - [41:11](https://www.youtube.com/watch?v=WGl-myLsPSM&t=2471s) 人类应当扮演驾驭 AI 这匹赛马的“骑师”，主动将模型推离平均分布的平庸输出，腾出精力去解决真正重要的世界级大问题。 ## Detailed walkthrough ### [00:04](https://www.youtube.com/watch?v=WGl-myLsPSM&t=4s) 从图书出版到智能体学习平台：O'Reilly 的转型与新索洛悖论 演讲伊始，蒂姆·奥莱利（Tim O'Reilly）现场询问有多少人是因为那些经典的计算机技术图书而认识他的，台下举起了大片双手。他回忆道，从 20 世纪 80 年代到 2000 年，图书出版一直是 O'Reilly 的核心业务。随后公司涉足技术大会领域并运营多年，直到疫情前才停办相关大会。伴随技术浪潮，O'Reilly 经历了无数次转型：虽然至今仍出版《AI 工程》（*AI Engineering*）以及大量探讨智能体编程（agentic programming）的专著，但其核心业务已转变为在线学习平台。多年来，他们始终是重大技术思潮的积极倡导者——从开源软件（Open Source）、Web 2.0、创客运动（Maker Movement）到平台型政府（Government as a Platform），而如今他们全方位押注人工智能（AI）。现在的 O'Reilly 不仅是为人类打造的学习平台，也是一个面向 AI 智能体（agents）的学习平台。 随后，奥莱利指出了当今企业在 AI 落地中普遍遭遇的瓶颈——新时代的“索洛悖论”（Solow Paradox）。经济学家罗伯特·索洛（Robert Solow）曾在 20 世纪 80 年代末提出，计算机革命的影响随处可见，唯独没有体现在生产率统计数据中。如今，科技圈充斥着“AI 药丸”（AI-pilled）拥趸们鼓吹的指数级增长（exponentials）与极速起飞（fast takeoff）论调，声称一切都将被颠覆；但在企业一线的实际操作中，推进 AI 却极其艰难。技术的变革绝非一蹴而就，企业必须经历痛苦、缓慢而扎实的过程去改造业务流程。 - **业务演进路径**：技术图书出版（1980s-2000） $\to$ 行业技术大会 $\to$ 人类与智能体在线学习平台。 - **倡导的技术思潮**：开源软件、Web 2.0、创客运动、政府作为平台（Government as a Platform）、AI 与智能体编程。 - **新索洛悖论**：AI 宣传的指数级生产力爆发与企业实际落地改造业务流程之间的巨大鸿沟。 --- ### [02:41](https://www.youtube.com/watch?v=WGl-myLsPSM&t=161s) “苦涩的教训”：从网络黄页、Google 到 ChatGPT 的范式迭代 奥莱利深入探讨了强化学习先驱理查德·萨顿（Richard Sutton）提出的“苦涩的教训”（The Bitter Lesson）。萨顿在回顾国际象棋、围棋等 AI 发展历程时指出：人类企图将自身专业知识编码进系统的努力，最终都被更通用的算力与暴力计算（brute force）彻底击败。奥莱利强调，苦涩的教训绝不仅局限于 AI 领域，而是整部技术演进史的底层法则。1993 年，O'Reilly 出版了销售达百万册的《全互联网用户指南与目录》（*The Whole Internet User's Guide & Catalog*），并创建了全球首个商业化且包含广告的网站 GNN（Global Network Navigator，首个门户网站）。几年后雅虎（Yahoo）问世，试图列出所有网站；但最终 Google 彻底颠覆了这一切——Google 没有人工维护庞大的目录分类，而是通过海量算力为用户的每一次查询实时动态生成专属目录。 ```mermaid flowchart TD accTitle: The Bitter Lesson across Tech Paradigms subgraph WebEra["互联网信息索引范式演进"] A["人工精选目录<br/>(O'Reilly GNN / 雅虎)"] -->|"算力超越人工分类"| B["算力动态生成结果<br/>(Google 搜索引擎)"] end subgraph AIEra["内容交互与问答范式演进"] C["搜索框+关键词匹配检索<br/>(O'Reilly Answers / BERT)"] -->|"大模型通用对话交互"| D["自然语言生成与交互<br/>(ChatGPT / AI 原生)"] end ``` 这种被“苦涩的教训”击中的经历在 O'Reilly 自身也曾重演。2018 年 11 月 Google 开源了基于 Transformer 架构的语言模型 BERT。当时 O'Reilly 迅速采用 BERT 研发了智能问答产品“O'Reilly Answers”，用自然语言检索旗下所有图书并提取答案。当时奥莱利曾欣喜地认为 Google 开源核心秘诀是一大失误，让 O'Reilly 拥有了匹敌 Google 的站内搜索体验。然而他们当时依然受制于上一代技术（即 Google 搜索框）的界面认知。到了 2022 年 11 月，OpenAI 推出的 ChatGPT 再次贯彻了苦涩的教训——抛弃传统的搜索呈现逻辑，直接以极简的对话框形式让用户直接与模型交互获取一切信息。 - **1993 年**：发布《全互联网用户指南与目录》及 GNN（首个带商业广告的门户网站）。 - **Google 的降维打击**：用海量算力取代人工网站分类目录，按查询动态生成结果。 - **2018 年 vs 2022 年**：O'Reilly Answers 基于 BERT 延续了传统搜索界面，而 ChatGPT 则以直接对话界面打破了旧范式。 --- ### [06:11](https://www.youtube.com/watch?v=WGl-myLsPSM&t=371s) 探索“AI 原生”工作流与技能评估实验 奥莱利指出，行业目前正处于从“AI 辅助”（如 GitHub Copilot）向“AI 原生”（AI native）工作流迈进的探索期，即便是 OpenAI、Anthropic 和 Google 也无法断定未来的终极形态。在 O'Reilly 内部，团队曾推进一个名为“可验证技能”（Verifiable Skills）的项目，旨在帮助企业培训经理评估员工现有技能、目标差距及学习路径。最初团队的思路依然是传统软件思维的 AI 辅助版——“让 AI 帮我们批量生成测试题”。 奥莱利当即否决了这一做法，并进行了一次现场演示：他随意找了一名开发者的公开 GitHub 主页输入给 ChatGPT，让其分析该开发者的技术栈与能力画像；接着又从另一处对话中调出一份 Sora 工程师的职位描述，让 ChatGPT 评估该开发者是否胜任该岗位。模型给出了极其精准深入的匹配分析。这让团队意识到，与其用 AI 辅助出题，不如打造一个对话式的“AI 原生技能路径构建器”（Skill Path Builder）。用户只需与 AI 深入交谈自身现状、学习诉求与目标（例如奥莱利自己就曾使用该工具规划如何微调一个小模型来模仿自己的文风），系统便能动态定制路径。不过，奥莱利也坦言，“AI 原生”是一个不断后退的地平线，所有人都在追赶其演进步伐。 --- ### [09:04](https://www.youtube.com/watch?v=WGl-myLsPSM&t=544s) 企业落地困境：隐性知识转移与复合专业能力 奥莱利分享了他在个人工作流中应用 AI 的经验与阻碍。在他主持的访谈栏目《与蒂姆·奥莱利面对面》（*Live with Tim O'Reilly*）中，他通常会将录音转录文本输入 Claude，提炼社交媒体精华片段并生成摘要总结。由于输入包含了大量他本人的口吻材料，经过反复调试，他训练出的提示词与工作流能生成极为地道的文章，甚至在通过 AI 内容检测器时显示为“100% 人类撰写”。然而，当他试图将这套技能原封不动交接给团队员工以解放自身精力时，这套工作流却失效了。 ```mermaid flowchart LR accTitle: Workflow Handoff Bottlenecks subgraph Individual["个人成功流 (蒂姆·奥莱利)"] T1["播客转录文本"] --> T2["专属 Claude 实例<br/>(积累大量上下文)"] T2 --> T3["100% 人类风格产出"] end subgraph Enterprise["企业交接瓶颈"] E1["上下文无法共享"] -.-> B1["知识孤岛"] E2["权限与工具割裂<br/>(如视频剪辑需找特定同事)"] -.-> B2["隐性知识壁垒"] end ``` 究其原因，核心问题在于大模型实例中积累的丰富历史上下文（context）无法直接跨账号迁移。此外，真实的企业工作流充满了各种隐性知识（tacit knowledge）和权限壁垒——例如提取特定视频素材必须找特定员工 Keith，连 CEO 也无法仅凭智能体直接打通所有环节。理论上的纯智能体工作流在实践中极其复杂。这引出了本次演讲的核心线索：安全咨询公司 Trail of Bits（约 150 人）的 CEO 丹·吉多（Dan Guido）提出的企业 AI 采用路线图。吉多指出，企业落地 AI 的核心要义在于**构建一个能让专业知识不断复合累积的系统**（a system in which expertise compounds）。奥莱利正是通过在播客中对吉多的访谈，将两人的实践与洞察融汇成了本场演讲的方法论框架。 - **个人提示词迁移失败的症结**：模型会话累积的上下文未打通、企业权限分散、业务流中存在大量未显性化的“隐性知识”。 - **企业 AI 核心命题**：如何避免 AI 技能仅停留在个人玩具阶段，将其沉淀并转化为全组织能力不断“复合增益”的系统。 - **方法论来源**：丹·吉多（Dan Guido，Trail of Bits CEO）的企业 AI 采用实践与路线图。 ### [11:55](https://www.youtube.com/watch?v=WGl-myLsPSM&t=715s) 企业 AI 落地的三种境界：从辅助到能力复利 在与网络安全公司 Trail of Bits 创始人兼 CEO 丹·吉多（Dan Guido）深入交流后，蒂姆·奥莱利（Tim O'Reilly）与 Claude 合作撰写了一篇名为《AI 作为企业操作系统》（AI as an Enterprise Operating System）的文章。在探讨企业落地路径时，吉多将 AI 的采纳划分为三个递进的层次：第一层是基础的“AI 辅助”（AI-assisted）；第二层则是“AI 增强”（AI-augmented），即把企业现有的工作流结合智能体流程进行大幅扩展；而最高层级是真正的“AI 原生”（AI-native）。奥莱利强调，AI 原生绝不仅仅是让系统自主执行任务，其核心在于实现“经验与能力的复利”（compounding capabilities）——在企业内部，员工利用 AI 构建的每一个工具、组件或流程，都会沉淀并转化为整家公司的底层能力资产。 ```mermaid flowchart LR accTitle: 三层 AI 采纳模型演进图 subgraph S1["Level 1: AI 辅助"] A["个人工具辅助"] end subgraph S2["Level 2: AI 增强"] B["现有工作流改造"] --> C["引入智能体流程"] end subgraph S3["Level 3: AI 原生"] D["自主系统运行"] --> E["沉淀可复用资产"] --> F["企业能力复利"] end S1 --> S2 --> S3 ``` ### [13:11](https://www.youtube.com/watch?v=WGl-myLsPSM&t=791s) 组织转型的心理阻力与认知偏差：“新阿米什人”与身份威胁 企业推行 AI 最艰难的关卡并非技术，而是员工的心理阻力。吉多在推进变革时总结出的首要法则是：“CEO 必须身先士卒（The CEO has to go first）”。领导者不能只停留在口头宣讲，必须亲自下场使用。然而即使 CEO 带头，企业内部的阻力依然惊人：在转型初期，通常只有约 5% 的员工真正全力投入，20% 的人公开强烈抵制（认为“我们根本不需要这玩意，这很糟糕”），而剩下的 70% 则处于被动抵制状态——表面上随声附和，内心却死守原有的工作方式，成为组织齿轮中最难察觉的“阻力砂砾”。 员工之所以固步自封，背后存在几种根深蒂固的心理偏差与认知威胁： - **自我提升偏差（Self-enhancing bias）**：倾向于把过往的成功完全归功于个人的判断与技巧，而把失败推诿给系统或外部环境。 - **不透明性（Opacity）**：对 AI 模型的决策机理感到困惑与不信任。 - **对不完美的零容忍（Intolerance for imperfection）**：AI 只要犯一次错，就会立刻招致极度严苛的否定与批判。 - **身份威胁（Identity threat）**：这是最核心的阻力源头。员工害怕 AI 不仅会取代他们的工作内容，更会剥夺他们的职业身份认同。 曾担任 2012 年奥巴马竞选团队首席技术官（CTO）的知名黑客哈珀·里德（Harper Reed）曾向奥莱利调侃，在技术社区 Hacker News 上活跃着一群坚守纯手写代码的“新阿米什人”（the new Amish）。许多优秀的程序员将自我价值深度绑定在“手写代码”这一行为本身，因而本能地抗拒 AI 介入核心工作。 ### [16:21](https://www.youtube.com/watch?v=WGl-myLsPSM&t=981s) 重塑身份认同与建立“地位阶梯” 为了化解员工面临的身份威胁，吉多借鉴了一项关于烹饪厨具营销的经典心理学研究。该研究测试了两种宣传定位：第一种主打“这台机器能替你做饭”，第二种主打“这台机器能让你成为更优秀的大厨”。结果显示，主打自动化替代的定位虽然在大众市场有销路，却遭到了真正擅长烹饪的核心用户群体的极力排斥。吉多将这一洞察直接迁移到 Trail of Bits 的安全业务中，他不再宣传 AI 将接管审计，而是向团队明确表态：“AI 将让你成为一名更出色、更具杀伤力的安全审计员（a better and more dangerous security auditor）。” 与其通过高压行政命令强推转型，吉多选择在组织内部搭建一套公开透明的“地位与能力阶梯”（Status and Capability Ladder）。员工在公司内部的专业声望与晋升机会不再取决于资历，而是与其在阶梯上的位置挂钩： | 阶梯等级 | 行为定义 | 组织评估与定位 | |:--- |:--- |:--- | | **Level 0** | **拒绝使用 AI（Resisting AI）** | 非技能问题，而是态度与企业战略目标背道而驰 | | **Level 1** | **尝试与实验（Experimenting）** | 开始在边缘任务中探索和试用 AI 工具 | | **Level 2** | **主动融入工作流（Adapting to work）** | 将 AI 实质性结合进日常生产流程中 | | **Level 3** | **构建组织级能力（Building organizational capability）** | 创造能被跨团队复用的 AI 工具与智能体系统 | 奥莱利指出，Level 0 绝非能力高低的问题，拒绝采纳 AI 实际上是在对抗企业的长期目标。目前，O'Reilly Media 正结合平台庞大的技术与商业技能数据库，将吉多的这套阶梯矩阵提炼为适用于更多行业企业的通用方法论。 ### [19:44](https://www.youtube.com/watch?v=WGl-myLsPSM&t=1184s) 知识复利黑客松：从“清理海滩”到自主智能体 为了让能力阶梯真正运转起来，Trail of Bits 确立了每两个月举办一次内部黑客松的制度。这种黑客松不同于漫无目的的自由发挥，而是具有极其严格的运作机制：提前一周公布公司的核心技术聚焦点与明确的学习目标；工程师与非工程师分流并拿到定制化指引；全员结对（Pairs）协作以确保代码与构想随时得到同行评审；黑客松结束后的核心环节是“沉淀与系统化可复用资产（Harvest reusable artifacts）”。衡量黑客松成败的标准，不在于当天交付了什么原型，而在于次周有多少员工在组织能力阶梯上成功实现了等级跃迁。 第一场黑客松被命名为“清理海滩”（Beach Cleanup）。奥莱利引用了前美国首席数据科学家 DJ·帕蒂尔（DJ Patil）提出的“整洁之屋”（Tidy House）概念——在推行企业级 AI 之前，必须先理顺内部混乱的基础设施与数据资产。Trail of Bits 借此集中清理了庞杂混乱的 GitHub 代码库。同时，奥莱利也分享了自身的切身体会：在分布式办公环境下，由于不同员工的电脑环境各异（例如 macOS 默认终端 Shell 与 Claude 所预期的 Bash 环境不一致），AI 智能体经常在配置错误上反复报错重试，白白消耗大量昂贵的 API Token。因此，统一开发环境与数据规范是极为关键的“清理海滩”第一步。 在随后的第二场黑客松中，主题升级为“自主智能体（Autonomous Agents）与 MCP 集成”： - 推动全员适应让 AI 独立执行多步骤复杂任务，克服人工在每一步过度干预的习惯。 - 针对非技术人员，要求其基于日常使用工具构建可复用的 AI 技能，或开发基于 MCP（模型上下文协议，Model Context Protocol）的工具集成，将个人经验真正沉淀为全公司共享的数字化资产。 ### [23:38](https://www.youtube.com/watch?v=WGl-myLsPSM&t=1418s) 领导层、实验室与大众：Ethan Mollick 的组织推动模型 Tim O'Reilly 继续分享来自 Dan Guido（Trail of Bits 首席执行官）在公司内部 Slack 中的具体实践。Dan 指出，在 AI 采纳过程中，不能只靠大家各自摸索，必须有一部分团队专门去研究和定义“工具环境的标准化”到底应该是什么样。通过类似黑客松的方式发现问题后，就需要有专人把这些零散的经验收拢。 这与沃顿商学院教授 Ethan Mollick（其 Substack 专栏《One Useful Thing》广受欢迎）在一年前提出的核心原则不谋而合——即企业落地 AI 的“领导层（Leadership）、实验室（Lab）与大众（Crowd）”模型： 1. **领导层（Leadership）**：必须身先士卒，率先使用并推动。 2. **大众（Crowd）**：全公司所有在日常工作中自发实验 AI 的员工。 3. **实验室（Lab）**：至关重要的中间团队，负责收集全员的实验成果与经验教训，将其系统化、工具化，并打造出能够在全公司推广复用的标准化资产。 O'Reilly 举例说明这一模式的普遍适用性：他的一位培训师朋友受聘于美国国税局（IRS），第一件事就是集中培训了 IRS 最顶尖的 30 位 IT 和技术专家，将他们打造成核心“实验室小组（Lab group）”。随后由这批人再去培训全机构的其他员工，从而实现自上而下与由点及面的系统化扩展。 ```mermaid flowchart TD accTitle: Ethan Mollick 的 AI 组织采纳模型 subgraph Model["Ethan Mollick 组织落地模型"] A["领导层 (Leadership)<br/>率先使用并提供战略支持"] --> B["实验室 (Lab)<br/>沉淀经验、制定标准配置与工具"] C["大众 (Crowd)<br/>全员自发实验并提供反馈"] --> B B --> D["全公司标准化推广<br/>共享 Skill 库与默认配置"] end ``` --- ### [25:38](https://www.youtube.com/watch?v=WGl-myLsPSM&t=1538s) 将“伤疤”转化为基础设施：治理 Prompt 债务与配置标准化 Dan Guido 提出了一个极具穿透力的概念：“把伤疤转化为基础设施（Turn scar tissue into infrastructure）”。Tim O'Reilly 对此深有共鸣——当他在进行 AI 编码实验时，AI 会不断犯错，虽然最终在提示词调整下把任务磕磕碰碰完成了，但如果只停留在“终于调通了”，这些留在代码和 Prompt 里的补丁就只是未被治愈的“伤疤”。 专栏作家 Drew Breunig 曾专门探讨过“Prompt 债务（Prompt Debt）”：人们为了让 AI 达到目标，把最初简单的 Prompt 越改越臃肿、越改越复杂，最终这些 Prompt 严重绑定于特定模型的特定脾气，完全不具备可移植性。这就是典型的“伤疤组织”。Dan Guido 的思路则是：每一次失败都必须沉淀为一套全局的默认标准（Every failure becomes a default）。 在 Trail of Bits，团队建立了统一的 Claude Code 配置文件仓库。每当 Claude 出现未按预期执行的情况，修复方案就会被沉淀为全公司通用的、可直接复制粘贴的默认配置（Global copy-pasteable defaults）。这样就避免了“员工 A 踩坑自己修好，员工 B 遇到同样问题又用另一种奇怪方式修一遍”的低效内耗。 - **配置冷启动**：Dan Guido 亲自编写第一版基础配置，随后开放全员 Pull Request。 - **专人跟进**：每次黑客松后指定专人收集员工未主动提交但实际踩坑沉淀出的技巧。 - **入职理念**：“如果我明天新招一个人，我绝不希望他重新经历过去一年里我们所有人踩过的所有坑。” - **消除心理门槛**：员工在刚接触 AI 时常会因为一次报错就抱怨“这玩意根本不好用”，开箱即用（out of the box）的健壮默认配置能彻底打破这种心理阻碍。 --- ### [28:36](https://www.youtube.com/watch?v=WGl-myLsPSM&t=1716s) 技能仓库分类与安全沙箱防护网 为了让体系既能规范内部又能赋能外部，Trail of Bits 构建了三级 Skills 代码仓库： 1. **内部仓库**：存放针对公司特定内部工作流的专有技能。 2. **公开仓库**：开源公开的技能。强制团队公开部分内容，能倒逼工程师写出组织内外都能看懂的高质量结构与文档。 3. **第三方精选库**：经过严格审核与策展的外部第三方技能集合。 在安全防护方面，作为一家顶级安全公司，Trail of Bits 建立了严格的安全护栏： - **多重隔离沙箱（Multiple Sandboxes）**：针对不同业务场景配置独立的隔离运行环境。 - **7 天安全冷却期（7-day cool down）**：对所有外部引入的新依赖包实施 7 天冷却机制。统计表明，绝大多数恶意供应链攻击包都会在发布后数小时内被安全研究员捕获，因此 7 天等待期能有效过滤绝大部分供应链风险。 - **Agent 评测体系（Evals）**：正大力投入建立评估机制，因为单纯靠共享配置仍然是在共享“伤疤”，必须像给员工做绩效评估一样定期给 Agent 进行评测（"An agent needs a performance review"）。 --- ### [30:18](https://www.youtube.com/watch?v=WGl-myLsPSM&t=1818s) 数据债务、权限债务与业务成效指标 Tim O'Reilly 指出，AI Agent 落地企业时最致命的瓶颈往往在底层数据，即“权限债务（Permission Debt）”——Agent 想执行任务却没有读取相应数据的权限。一家 500 人的公司（如 O'Reilly Media）数据权限问题就已经一团糟，50,000 人的大型跨国集团更是难以想象。 正如 Dan Guido 的精辟断言：“过去 20 年里一家组织在数据上走过的每一个捷径、偷过的每一次懒，在 Agent 接触到它的瞬间都会暴露无遗。” 美国前首席数据科学家（也是提出“数据科学家”概念的先驱之一）DJ Patil 强调，解决这个问题不需要魔法，而是扎扎实实的“数据 101”基础功课：统一数据环境、清洗数据流、精细化元数据管理与访问控制。DJ 为另一家初创公司搭建了这类底层数据基建，其 CEO 明确表示，他们能在 AI Agent 时代一骑绝尘，完全归功于当年打下的干净数据底座。 Dan Guido 还提出了几条切实可行的管理指标与原则： - **数据复用率指标**：你的团队将百分之多少的数据开放给其他团队复用？你又消费了其他团队多少数据？ - **FAFO 实验时间**：当 Tim 问 Dan 在推进过程中犯过什么错误时，Dan 直言唯一的错误就是“没有给员工足够的自由摸索期（FAFO: Fuck Around and Find Out time）”。 | 落地关键举措 | 具体执行方式 | |:--- |:--- | | **标准化工作流** | 统一采用一套经受住考验的 Agent 运行流，避免各部门工具群魔乱舞。 | | **编制 AI 手册与能力阶梯** | 明确建立能力进阶阶梯（Capability ladder），让掌握 AI 成为员工晋升的明确预期。 | | **短期采纳冲刺** | 举办强迫全员亲自动手（Hands-on）的高强度短周期实战。 | | **沉淀复用资产** | 将技能、配置、依赖包沉淀为代码化资产，配合沙箱确保安全。 | **Trail of Bits 的转型成果与人员演进：** - **工作并未消失，而是转移与升级**：技术编辑不再做逐行校对（Line editing），而是转型为故事架构与读者受众咨询；运营与市场人员开始学习 Git 与命令行；涌现出“Agent 系统质检员”、“Skill 产品经理”等全新岗位。 - **工程效能跃升**：安全漏洞修复能力从过去的每周解决 15 个 Bug，飙升至每周解决 200 个 Bug。 - **AI 发现率**：向客户报告的所有漏洞中，有 20% 最初是由 AI 自动挖掘发现的。 - **销售人效**：销售团队人均创收达到 800 万美元，而同行业的基准线仅为 200 万至 400 万美元。 --- ### [33:40](https://www.youtube.com/watch?v=WGl-myLsPSM&t=2020s) 人类专业经验在 Bitter Lesson 时代的核心价值 尽管“苦涩的教训（The Bitter Lesson）”揭示了算力与通用模型最终会碾压人类精心设计的特化规则，但 Tim O'Reilly 强调，人类的专业领域知识（Human Expertise）在整个过程中依然具有不可替代的核心价值，绝不能消极地坐等通用 AI 算力把人类向前推。 O'Reilly Media CTO Andrew Odewahn 对此提出了极具洞察力的视角。大众通常把 O'Reilly 看作一家出版图书、制作视频或提供体系化学习路径的媒体公司，但其本质实际上是一个“**专业人才与专业需求之间的撮合市场（A matching marketplace for expertise）**”。 公司一直在思考如何帮助专业人才变现其知识：过去图书市场萎缩时，O'Reilly 顺势推出了在线直播培训（Live Training），为专家开拓了传授经验的新渠道；而如今，公司正致力于借助 AI Agent 构建全新的专业知识交付载体，探索如何让专家的深刻洞见与经验更好地融入 Agent 时代。 ### [34:56](https://www.youtube.com/watch?v=WGl-myLsPSM&t=2096s) 将人类专业知识转化为基础设施：O'Reilly 专家智能与 MCP 实践 Tim O'Reilly 分享了团队在构建 MCP（Model Context Protocol）服务器以接入 O'Reilly 内容时的演进过程。最初的第一版非常粗糙，仅限于搜索元数据；但团队很快将其升级为全面接入核心内容库，并借此探索提升组织智能的途径。随着团队执行速度的加快与本地决策成本的降低，传统的中心化计划逐渐被分散的团队级捷径所取代，管理者越来越难获得做出明智决策所需的全局视角。 ```mermaid flowchart TD accTitle: 传统AI报告与引入专家知识后的决策对比 subgraph RawAgent["基础 AI 智能体 (仅接入内部数据)"] A["内部系统指标<br/>(工时/Sprint速度/Backlog)"] --> B["数据驱动报告<br/>(列举工时与工单堆积)"] B --> C["常规结论: 团队超负荷，建议招人<br/>(容易认同，但极难执行)"] end subgraph ExpertAgent["专家增强智能体 (接入 O'Reilly 专家 MCP)"] D["内部系统指标"] --> E["引入 Google SRE 等专业知识库"] E --> F["结构化洞察: 发现 60% 属于操作苦工 (Toil)"] F --> G["决策导向方案: 苦工审计 / 削减目标 / Runbook 归属<br/>(明确可追溯的重构行动)"] end ``` 为了解决这一痛点，团队进行了一次生动的实测。一位工程总监因团队超负荷运转而提出扩招工程师的申请。管理层先利用接入内部系统的普通 AI 智能体生成了一份招聘论证报告，结果虽详尽列举了人员编制、服务归属、Sprint 速度和工单积压等数据，却令人感到一种“低度挫败感”——报告仅仅罗列了“发生了什么”，而没有解释“为什么发生”以及“该怎么做”，属于典型“容易认同却难以付诸行动”的以数据为中心的内容。随后，团队在配置中加入了 O'Reilly 专家 MCP 服务器，重新提问：“O'Reilly 上的行业专家会如何审查此请求？” 输出结果发生了质的飞跃：报告开篇即指出该团队的运营开销是结构性问题而非人员短缺，并直接引用 Google SRE 框架中的“操作苦工（Operational Toil）”概念，判定该团队约 60% 的精力消耗在琐碎苦工上（远超 SRE 文献推荐的干预阈值），进而给出了运行苦工审计、设定具体缩减目标、明确 Runbook 归属权等立竿见影的结构性改善方案。 - **组织智能（Organizational Intelligence）**：将分散的专业经验与“组织伤疤”沉淀为可复用的基础设施。 - **O'Reilly 专家智能（O'Reilly Expert Intelligence）**：目前处于 Beta 阶段，支持分析产品简报、生成工作范围说明书（Scope Statement）、系统替换方案权衡、评估技术迁移与团队路线图等丰富技能。 --- ### [38:53](https://www.youtube.com/watch?v=WGl-myLsPSM&t=2333s) 技术扩散与索洛悖论：组织重构才是制胜关键 Tim 回顾了开场提到的“索洛悖论（Solow Paradox）”并指出，美国当年的生产力飞跃并不是因为计算机运算速度变快（这与纯粹依赖算力的“苦涩教训”逻辑相悖），而是因为企业围绕计算机的能力彻底重组了自身的组织架构。当前业界对模型能力的狂热追逐存在严重误区，许多人误以为只要模型更强、更快，苦涩教训就能自然解决一切问题。然而，技术革命的历史表明，新技术的“技术扩散（Diffusion）”才是最艰难的瓶颈。 正如研究大国崛起的历史文献所揭示的，英国凭借蒸汽机崛起、美国凭借电气化与大规模制造称雄，其背后的核心规律并不是单纯在领先技术上取得垄断，而在于谁能让这项技术实现最广泛、最便捷的普及与应用——**真正决定胜负的门槛是采用（Adoption），而非模型能力本身**。Tim 盛赞了 Trail of Bits 及其负责人 Dan Guido 将内部 AI 落地流程公之于众的胸怀，呼吁更多企业投入到这项必不可少的组织再造工作中。 --- ### [41:11](https://www.youtube.com/watch?v=WGl-myLsPSM&t=2471s) AI 需要人类：把模型推离平均分布，投身真正的大问题 在谈及人类与 AI 的协作关系时，Tim 用赛马进行了形象的比喻：不能把一群马放进赛道就指望它们跑出一场精彩的比赛，马固然重要，但骑师的驾驭同样不可或缺。面对 AI，人类扮演的就是骑师的角色。如果使用者偷懒，只用一句简单的 Prompt 确实能得到及格线水平的输出，但若想追求卓越，就必须主动“将模型推离平均分布（Push the model out of the average distribution）”。 AI 本质上是一种与写作、绘画无异的全新媒介。正如米开朗基罗与梵高使用相同的画具却创造出截然不同的传世杰作一样，每个人都应当通过独特的洞察，将 AI 这门媒介运用到极致。Tim 引用了里尔克的经典诗句：“我们对抗的事物是如此微小；当我们获胜时，它却让我们变得渺小。我们真正渴望的，是被更伟大的存在决定性地击败。” 他以此寄语所有从业者：引入 AI 的终极目的不仅是让企业在现有事务上提高效率，更是为了腾出双手去挑战以往无法企及的宏大目标与真正重要的世界级难题。 - **赛马与骑师理论**：优秀的模型（赛马）需要精湛的人类专业引导（骑师）与良好的落地环境（跑道条件）才能发挥最大价值。 - **对抗平庸**：拒绝停留于平均分布的平庸生成结果，把 AI 视为艺术创作的媒介，发掘其独特深度。 - **追求更宏大的使命**：当 AI 轻松接管了琐碎微小的工作时，人类应该欢庆并转而“投身于真正重要的大事（Work on stuff that matters）”。 ## Notable quotes > [13:11](https://www.youtube.com/watch?v=WGl-myLsPSM&t=791s) CEO 必须身先士卒（The CEO has to go first）。 — Dan Guido > [16:21](https://www.youtube.com/watch?v=WGl-myLsPSM&t=981s) AI 将让你成为一名更出色、更具杀伤力的安全审计员（a better and more dangerous security auditor）。 — Dan Guido > [25:38](https://www.youtube.com/watch?v=WGl-myLsPSM&t=1538s) 把伤疤转化为基础设施（Turn scar tissue into infrastructure）。 — Dan Guido > [25:38](https://www.youtube.com/watch?v=WGl-myLsPSM&t=1538s) 每一次失败都必须沉淀为一套全局的默认标准（Every failure becomes a default）。 — Dan Guido > [25:38](https://www.youtube.com/watch?v=WGl-myLsPSM&t=1538s) 如果我明天新招一个人，我绝不希望他重新经历过去一年里我们所有人踩过的所有坑。 — Dan Guido > [28:36](https://www.youtube.com/watch?v=WGl-myLsPSM&t=1716s) 必须像给员工做绩效评估一样定期给 Agent 进行评测（An agent needs a performance review）。 — Dan Guido > [30:18](https://www.youtube.com/watch?v=WGl-myLsPSM&t=1818s) 过去 20 年里一家组织在数据上走过的每一个捷径、偷过的每一次懒，在 Agent 接触到它的瞬间都会暴露无遗。 — Dan Guido > [33:40](https://www.youtube.com/watch?v=WGl-myLsPSM&t=2020s) 专业人才与专业需求之间的撮合市场（A matching marketplace for expertise）。 — Andrew Odewahn > [38:53](https://www.youtube.com/watch?v=WGl-myLsPSM&t=2333s) 真正决定胜负的门槛是采用（Adoption），而非模型能力本身。 — Tim O'Reilly > [41:11](https://www.youtube.com/watch?v=WGl-myLsPSM&t=2471s) 将模型推离平均分布（Push the model out of the average distribution）。 — Tim O'Reilly > [41:11](https://www.youtube.com/watch?v=WGl-myLsPSM&t=2471s) 我们对抗的事物是如此微小；当我们获胜时，它却让我们变得渺小。我们真正渴望的，是被更伟大的存在决定性地击败。 — Tim O'Reilly > [41:11](https://www.youtube.com/watch?v=WGl-myLsPSM&t=2471s) 投身于真正重要的大事（Work on stuff that matters）。 — Tim O'Reilly ## Mentioned Tim O'Reilly, Dan Guido, Richard Sutton, Robert Solow, Harper Reed, Ethan Mollick, Drew Breunig, DJ Patil, Andrew Odewahn, Rainer Maria Rilke, O'Reilly Media, Trail of Bits, OpenAI, Anthropic, Google, Yahoo, IRS (美国国税局), ChatGPT, Claude, Claude Code, BERT, O'Reilly Answers, GNN (Global Network Navigator), Model Context Protocol (MCP), GitHub, GitHub Copilot, Hacker News, Google SRE ## Video description Tim O'Reilly - Founder and CEO, O'Reilly Media One of the hardest things in getting enterprise AI strategy to work is letting go of what you do today. "The bitter lesson" is Richard Sutton's name for the realization that the progress of AI makes many highly valued skills of today obsolete. But grappling with the bitter lesson is the forge of effective AI corporate strategy. What do you let go of? And what do you embrace? How do you get your employees to ride the whirlwind of AI rather than trying to race it? How do harness the skills and experience of all your employees - not just your AI superstars - to improve their effectiveness and to transform the products and services that you offer to your customers? I'll lay out a six point roadmap for effective AI adoption and its implications for strategy, a roadmap that has already produced enormous success for those who follow it. [Source](https://www.youtube.com/watch?v=WGl-myLsPSM)

[13:11] Dan Guido: CEO 必须身先士卒（The CEO has to go first）。 [16:21] Dan Guido: AI 将让你成为一名更出色、更具杀伤力的安全审计员（a better and more dangerous security auditor）。 [25:38] Dan Guido: 把伤疤转化为基础设施（Turn scar tissue into infrastructure）。 [25:38] Dan Guido: 每一次失败都必须沉淀为一套全局的默认标准（Every failure becomes a default）。 [25:38] Dan Guido: 如果我明天新招一个人，我绝不希望他重新经历过去一年里我们所有人踩过的所有坑。 [28:36] Dan Guido: 必须像给员工做绩效评估一样定期给 Agent 进行评测（An agent needs a performance review）。 [30:18] Dan Guido: 过去 20 年里一家组织在数据上走过的每一个捷径、偷过的每一次懒，在 Agent 接触到它的瞬间都会暴露无遗。 [33:40] Andrew Odewahn: 专业人才与专业需求之间的撮合市场（A matching marketplace for expertise）。 [38:53] Tim O'Reilly: 真正决定胜负的门槛是采用（Adoption），而非模型能力本身。 [41:11] Tim O'Reilly: 将模型推离平均分布（Push the model out of the average distribution）。 [41:11] Tim O'Reilly: 我们对抗的事物是如此微小；当我们获胜时，它却让我们变得渺小。我们真正渴望的，是被更伟大的存在决定性地击败。 [41:11] Tim O'Reilly: 投身于真正重要的大事（Work on stuff that matters）。

[13:11] Dan Guido: the CEO has to go first. [15:40] Harper Reed: the new Amish [15:40] Tim O'Reilly: makes you a better cook [25:33] Dan Guido: turn scar tissue into infrastructure [25:33] Tim O'Reilly: every failure becomes a default. [25:33] Dan Guido: If I hire somebody tomorrow, I don't want them to go through the entire discovery process of the last year. [29:53] Dan Guido: every shortcut an organization took with its data over the past 20 years becomes visible the moment an agent hits it. [32:13] Tim O'Reilly: the work doesn't disappear, it moves. [34:56] Tim O'Reilly: How would the experts on O'Reilly Media review this request? [41:04] Tim O'Reilly: the horse matters, but the jockey matters maybe a lot, too [41:04] Andrew Bruner: pushing the model out of the average distribution [41:04] Rainer Maria Rilke: What we fight with is so small, and when we win, it makes us small. What we want is to be defeated decisively by successively greater beings.

[0:04] Uh how many of you here know me because of the  books? Okay, that's uh a lot of you. Uh so that [0:11] was two that was really starting in the 1980s and  until 2000. Uh that was the heart of our business. [0:20] Uh but uh we had started doing conferences. We  ran conferences like this one for many years. We [0:25] shut that down before the uh pandemic. uh we've  pivoted many many times. You know, we're still a [0:31] book publisher. Uh some of you may have read books  like AI engineering or uh um you know the many the [0:38] many books about agentic programming that we're  publishing. But we're really now largely an online [0:43] learning platform. And we've also been an activist  over the years for big ideas about the future of [0:50] technology, open source software, web 2.0, know uh  the maker movement uh government as a platform and [0:59] right now we're really focused very heavily on  AI as you might expect our core platform is uh [1:06] is an online learning platform but it's an online  learning platform not just for humans now but an [1:11] online learning platform for agents and I'm going  to get to that at the end uh but I want to start [1:17] by saying I'm this talk is drawn very heavily from  the work of someone I met recently a guy named [1:24] um Dan Guido who's the CEO of a small security  company 150 people called Trail of Bits. Uh and [1:32] he has basically built a roadmap for AI adoption  and I think that's really important because one [1:37] of the the things that we're all facing right now  is something that's been referred to as the Solo [1:43] paradox. uh Robert Solo was an economist uh uh  back in the u uh you know in I think it was in in [1:52] the early 90s he said you know the impact of it is  showing up everywhere except in the productivity [1:59] statistics and we're now facing a new solo paradox  where we're being told again and again that AI is [2:08] changing everything and you guys know from where  you sit just how hard that is you know you're [2:13] hearing the AI pill people saying exponentials  and everything's going to change and it's going [2:19] to happen. You know, there'll be fast takeoff and  it's all going to just and it's just technology [2:24] doesn't work that way because you know the hard  the whole the hard slow gritty work of adopting [2:30] a new technology, figuring out your business  processes and that's what I'm going to talk about [2:34] today, how we do that. So, I'm also going to talk  about something that was referred to this morning. [2:41] Um and uh uh I I realized I talked to to the other  thing uh which is called the bitter lesson. And I [2:49] I asked how many people were familiar with the  bitter lesson and very few hands went up. It's [2:54] it's it's something that people who are deep into  AI talk about a lot. Uh because the bitter lesson [3:00] is this uh idea from um uh uh Richard Sutton that  he looked at the history of of AI. This is pre [3:13] transformers. Uh but it was really about this idea  that uh human uh expertise doesn't really matter. [3:22] when they trained uh you know on chess on human  knowledge when they tra train trained on go this [3:28] was eventually beaten by brute force and computing  but the bitter lesson doesn't just apply to AI it [3:35] also applies to just the ordinary progress of of  technology let me talk about my history with the [3:41] worldwide web in 1993 uh in addition of being a  publisher we created the first catalog of websites [3:48] and we were publishers and we listed the best ones  it was the whole internet use guiding catalog. It [3:52] was a book that went on the Senate, a million  copies and it had an accompanying uh you know [3:57] that was the first commercial website, the first  site on the web to have advertising. It grew into [4:04] uh something called the global network navigator  which was the first web portal. And then Yahoo [4:08] came along a couple of years later and they  uh kind of said, "Oh, we're going to list [4:13] everything." But both of us were beaten by Google,  right? And what Google did was to learn the bitter [4:20] lesson. They basically said, "Wow, we'll just  throw a lot of compute at this, you know, and and [4:26] so instead of we had this these big cataloges that  people had to navigate through and Google said, [4:32] well, we'll just make a custom catalog for every  query, right? So that's a really important lesson. [4:38] The bitter lesson is is a real thing and it's not  just in AI. Okay. So, but bit the bitter lesson [4:46] can catch all of us. In November 2018, Google  did the first trans you know they published [4:51] the transformer paper the year before and they  released open as open source BERT that which was [4:56] in some ways the first language model and uh you  know we adopted it pretty quickly at O'Reilly. We [5:02] we built a product called O'Reilly answers and  it looked a like my take when I first saw that [5:08] was oh my gosh this is like Google has just made a  massive mistake. they've open sourced their secret [5:13] sauce and now we can search our content as well  as Google can and that was true when we built [5:18] this very cool product called O'Reilly answers to  help people navigate using plain language to find [5:23] stuff find get answers out of all of our books and  u that was all all fabulous you know and we we had [5:31] uh uh you know this is kind of what it looked  like but we were learning through the interface [5:37] lessons of the previous generation of technology  i.e Google, right? And what happened, as you know, [5:44] was uh in November 22, uh Chat GPT had learned the  bitter lesson relative to Google, and they said, [5:51] "Well, that's not why are we doing it the way  Google did it? We're just going to have a like a [5:56] chat box where you can talk to this thing and and  get whatever you want." So you know uh we all have [6:02] to learn and keep that that um bitter lesson in  mind because uh the bitter lesson is what teaches [6:11] us to think about what AI native might mean right  because we started out and you think about things [6:17] like Microsoft uh you know GitHub copilot it was  clearly AI assisted and you know we're all on a [6:25] journey towards something new even you know uh uh  anthropic and um open AI and Google don't know the [6:37] future. We are building it together and we have  to start thinking about what does AI native look [6:44] like for our workflows, right? Once you have this  magic power, how do you put it to work in a way [6:52] that's that's very simple? And so um you know  when I uh been working with this at my team at [6:59] O'Reilly like we started looking at things. We had  a project that was focused on um what we called um [7:09] um um sorry um verifiable skills and it was really  how do you get people to to to you through a path. [7:20] Our customers were corporate learning managers and  saying we want to know what skills people have, [7:24] where they're trying to get to, and what the path  would look like. And we had a workflow that that [7:29] involved quizzes and things like that. And uh so  our first version, our team's first version was [7:39] well, let's just, you know, we'll get AI to help  us write the quiz questions. We can get a lot more [7:44] done that way, huh? And I was like, no, no, no.  And I I gave the team a demo which I had it in [7:49] the slides at one point but I took it out but I'll  just tell it to you. I went out and I I picked a [7:54] random developer's public GitHub uh profile and I  said to chat GBT tell me about the skills of this [8:02] developer and it just kind of rattled things off  and then I found a random job description over [8:07] was actually on another chat. It was for a Sora  engineer and I said is this guy a good fit for [8:13] this job based on the skills that you've seen? it  was able to answer that and I went dudes this is [8:19] uh you know this is what we're competing against  we have to really change things so now we have you [8:25] know a much more interactive AI what we hope is  towards AI native uh you know skill path builder [8:32] where you kind of just chat with it and you tell  it where you are and what you are trying to do [8:37] what you're trying to learn what you're trying  to accomplish you know like I was just recently [8:40] using it to help me with a a plan for fine-tuning  uh a smaller model to to to write in my voice more [8:48] effectively and uh you know it's pretty good.  It's, you know, you you kind of explain your [8:54] problem and it's more AI native, but I think AI  native is a frontier that recedes and that we're [8:59] all going to be chasing. But so anyway, so the  question then is, okay, how do you get this all [9:04] the way through your organization? So I built  this pretty cool skill for writing. You know, [9:08] there's a lot of posts that we do. For example,  I have this interview show called Live with Tim [9:13] O'Reilly. And every time I do it, I take do  a takeaways post. I take the transcript and I [9:18] feed it into Claude and I say pick out some good  clips for social media uh and give me a summary [9:24] and then I rewrite that summary, you know, but  it's kind of mechanical writing because a lot [9:28] it's like what did I say? What did they what  did my guest say? You know, and I go, yeah, [9:33] I should be able to train a model to do most of  that writing. So, putting my spin on it is is much [9:38] less. And then I've gotten to the point now where  I actually have trained something that writes in [9:44] my voice well enough that I pass it through  an AI detector and it says this is 100% human [9:48] written. Now of course as I say the material the  raw material is a lot of my voice already. So that [9:55] makes it a lot easier. But anyway I gave it over  to members of my my staff and go okay now you do [9:59] this so I don't have to work on these things. And  the skill didn't work. I said why? And I go, "Oh, [10:06] well, it's all this accumulated context that my  instance of Claude has." I go, "How do I get it [10:11] over there?" So, I started thinking about this  problem of enterprises and there's other things [10:16] in that workflow where I go, I can't just do  it agentically. I have to ask Keith to extract [10:21] the video clips, you know, because I don't have  access to them. I I'm the CEO. I should be able [10:26] to figure out where the access is, but somebody in  my organization knows this. And certainly thinking [10:31] about all the tacet knowledge in an organization  and how hard it is, you know, to figure that out, [10:37] you know, in theory I could do all this stuff with  an agentic workflow eventually, but nah, you know, [10:44] in practice it's a lot harder than it looks.  So that led me to this uh talk by this guy Dan [10:51] Guido which I'm going to get to in a moment but  he came up with this notion that the key idea of [10:57] AI in organizations is you have to build a system  in which expertise compounds you know so that when [11:03] I built this system for myself that was helping  me write these kind of mechanical posts with AI [11:12] you know how would I make it so that that actually  just compounds into organizational capability. And [11:19] so I had Dan I I I I saw Dan's um uh give a talk  uh you know it was on on YouTube and he written [11:26] a blog post about it and I so I said well come  on my Live with Tim show. I interviewed him and [11:32] of course I turned that into a takeaway post and  then I turned the takeaway post into this slide [11:37] deck. So this is a lot of Dan Guido in here. Uh  that's Dan Dan over there on the the left side. [11:45] Um, and so, you know, again, a lot of my pros, my  ideas, Dan's ideas, and then, you know, I was able [11:55] to get Claude to build it. But, so let me tell you  what this, you know, message really is, you know, [12:01] and my eyes are not what they used to be. And  so, I go see what slide I'm on. Okay. So, oh, [12:05] this is the post I wrote, you know, I and Claude  wrote about my conversation with Dan. And it was [12:10] called AI as an enterprise operating system. And  Dan talks about the idea that there are three [12:19] levels of AI adoption. We already talked about AI  assisted. What he also says is there's a middle [12:25] level he calls AI augmented. Uh and AI augmented  is you're actually taking it and you're building [12:33] what I was calling AI native earlier. he calls AI  augmented because what I did was take a workflow [12:41] that already existed that I was already doing and  I figured out how to build some agentic processes [12:49] to do a lot more of it. But his point is that AI  native is actually building a system that not only [12:57] does these things autonomously but but compounds  the experience. So everything you build with AI [13:04] becomes part of the corporate capabilities,  right? Anyway, so he he he talked about this, [13:11] but he also talked a lot about the psychology  of AI adoption in enterprises. Uh you know, and [13:17] he what he said was employee resistance was the  first problem he had to solve. And um you know, [13:24] the first thing he he did about that was he said  the CEO has to go first. Well, I was already, [13:30] you know, kind of at the forefront of of what we  were doing at O'Reilly. We we had a lot of people. [13:34] We're a tech company, so, you know, that was that  that part wasn't hard. But I think that's really [13:39] important. If you're in an enterprise and you want  everybody to be doing it, then the CEO can't just [13:45] talk about it. CEO has to do it. And so Dan, you  know, kind of was the first person who did this [13:51] process and then he built it there. But then he  basically noticed that like only 5% of his staff [13:58] was really with him when he's trying to do this AI  transformation. And you know he he said you know [14:05] there was about 20% that were actively resisting  you know like we don't need this stuff it's bad. [14:12] And then about 70% were passively resisting. And  uh you know this is again I find this in you know [14:19] I'm talking about all this time with my senior  executives like who's on board like why are things [14:24] moving so slowly? How many of you have had that  kind of experience right? You have some big idea [14:29] and you just don't know where's this where's the  grit in in the gears in my organization you know [14:34] and it's people who kind of like they're giving  lip service but they really are attached to what [14:39] they do. So Dan started thinking about, you know,  what were the biases that were keeping people [14:45] stuck. And I think this is really great. He he he  talked about um uh uh basically self-reinforcing [14:52] bias, you know, where people basically have  an identity locked up in their job and they [14:58] don't want to lose it, right? He t he talked about  identity threat. Again, actually I may be mixing [15:04] these up and I should have worn my glasses here.  uh but uh the uh the slides here are very small. [15:11] uh he he talked about uh um sorry self-enhancing  bias is the one about crediting your wins to your [15:18] judgment and the failures to you know the system  or whatever identity is this identity threat [15:24] opacity you know we go we don't understand how  these decisions are being made you know and then [15:29] um u you know intolerance for imperfection you  know AI makes a mistake and it immediately gets [15:35] judged very harshly so h how did he deal with  these uh the first one He felt like identity [15:41] threat was the biggest one. People who basically  felt like in their organization that AI was [15:48] somehow going to replace them and not just replace  the work they did but replace who they were. Now [15:54] I had a conversation with Harper Reed who used to  be the CTO of the Obama campaign back in 2012 and [16:01] that he's a very interesting hacker for many  years and he he said you know there's a whole [16:06] community on hacker news that's the the new Amish.  you know they're like we want to code by hand you [16:12] know and you know there are a lot of people who  are attached they have a lot of identity as a [16:15] coder and they don't want to have AI take take  that over so he said well how do you deal with [16:21] identity threat and he he told a story of a study  that was done with two appliances uh two ways of [16:30] of positioning an a cooking appliance and one of  them was it it does the cooking for you and the [16:37] other was it makes you a better cook and the the  one that you know does the cooking for you had a [16:44] big market but the people who were really good at  what they did were resistant so they went oh okay [16:51] so what I have to do trail of bits is a security  company so he said I have to tell everybody this [16:57] will make you a better and more dangerous auditor  security auditor right you know so enhance the [17:02] identity don't take it away and Um but he  also found another kind of idea which was [17:12] how you build a new identity enhancing capability  layer. By the way, just so I say for those of you [17:19] taking pictures, uh Dan has put his entire slide  deck up on a GitHub profile. Look for, you know, [17:25] um Dan Guido Trail of Bits, you know, AI adoption.  you'll it'll lead you actually I have a slide at [17:30] the end with the link and um uh so you can get  his slides you can actually more importantly you [17:38] can look at his GitHub repo for some of the things  we're going to be talking about later uh but he he [17:43] talked about remedies that worked and one of them  was this idea of you have to build a new status [17:50] ladder and it's it's partly a capability ladder  but it's also a status ladder and he had three [17:56] levels he literally identified this idea you're  at level zero you're resisting AI right and this [18:03] you're going to be judged on that right and then  you know one was you're experimenting you know two [18:09] was you're actually actively trying to adapt it  to your work but three was builds AI that build [18:16] that helps build organizational capability so he  said this is how you're going to get status in [18:21] this company you're actually helping to build our  organizational AI capability and It's he described [18:28] it as a ladder not a mandate. All right. And  so um so again we talked about this sort of [18:37] transformative level. And um he also they have  a different version for you know different kinds [18:43] of departments. It's one kind of there's there's  a much more detailed capability matrix that goes [18:48] into well what would a security auditor do? What  would a software developer do? What would a person [18:54] in accounting what was that mean for them? uh and  he but he really did point out that level zero is [19:00] not a skill issue. The other ones are a skill  issue, but if you refuse to use AI, you know, [19:05] you're actually fighting the goals of the company.  And then this is an example of their much more [19:12] detailed skill level, you know, by department.  And we're we're working on a framework for helping [19:18] companies to build something like this that's for  their because we we have massive amounts of skill [19:23] data at O'Reilly uh you know technical skills  and and business skills and so we we kind of [19:29] have the raw material for building something  like this that's tailored to an individual [19:32] organization but this is still something that  we're we're developing. So, we're trying to [19:36] build a a regular methodology based on on Dan's  practice that's applicable across a wide range of [19:44] companies. The next thing they did was was to what  he called hackathons that compound knowledge. So, [19:51] uh what did he mean by that? Well, first of all,  they do a hackathon every two months and they [19:57] announce the it's not just like, oh, get together  and just hack on whatever you want. The hackathon [20:02] has an objective, you know, so the and and that  is a the corporate focus and learning objectives. [20:08] They're defined and announced a week ahead and  there's separate instructions for engineers and [20:14] non-engineers and people where work in pairs so  that everything gets reviewed and then there's [20:21] a demo session at the end like most hackathons.  But then there's also dedicated followth through [20:27] to to to f to harvest the reusable artifacts  and and systematize them, right? And you know, [20:34] success is measured by where people sit on the  capability layer ladder the following week, you [20:40] know? So it's not like what did you ship, but like  are you kind of getting the in the groove, you [20:45] know? But but this idea of the reusable artifacts  is is central to this idea of you're building [20:50] expertise that compounds across the organization.  you're building things that are reusable. Uh, and [20:57] the first hackathon they did just to get everybody  used to the idea was kind of the beach cleanup as [21:02] he he called it. It was just like we're going to  go to the company GitHub repos and we're going to [21:06] look at what a mess they are and we're going to  deal with all kinds of outstanding issues. And I [21:12] I think that the um you know there's a there's a  slide I I come to a little bit later in this talk [21:18] uh about this idea that DJ Patil has called the  the tidy house. You know house cleaning is really [21:26] really important when you get to enterprise AI  adoption. If you if your data is a mess, you know, [21:34] we we took an immediate look at our GitHub repo  and go, "Wow, there's thousands of of of projects [21:40] uh with no real organization, you know, and  engineering teams that use them know their [21:46] way around, but nobody else does. You have to  have a tidy house. So, you know, in some ways, [21:52] the first hackathon is let's get our stuff in  order." And it's not just uh you know say your [22:00] GitHub repos. So like, wow, does everybody have  the same environment? You know, as I started [22:07] working with AI, you know, on on my, you know, you  I'm working on my laptop. We're a very distributed [22:15] company and and I I'm doing stuff with Fable and  Saul even and I I'm watching them fail and retry [22:23] and fail and retry and a lot of it's configuration  stuff. It's like, oh, you know, it turns out like, [22:30] you know, the the max default shell is not  the one that Claude expects. You know, it's [22:36] like it's it's expecting bash and then, you know,  the default one is something else. And, you know, [22:42] it's just again again they're configuration  errors that lead to wasted to expensive tokens [22:48] because nobody has a standard environment.  The the the model and the environment aren't, [22:54] you know, figured out in pairs. So getting that  beach cleanup for everybody I think is a really [23:00] great first hackathon. It's just like get set up  and and you got to keep doing it. And then the [23:05] second hackathon they did two months later was to  get people used to the idea of autonomous agents. [23:09] You know so again it was in each area they said  okay we want you to run uh more independently. [23:16] You know we don't want you just we want you to  understand that these things can get jobs done [23:20] without you checking at every stage. So you had to  put, you know, so it was it that was kind of a a [23:26] twist. They they had it a little different for for  um uh engineers and non-engineers with with the [23:32] non-engineers, they built a reusable skill or MCP  uh integration for a daily tool. But everything [23:38] then goes into a shared skill repo. Anyway, but  this slide is in here and you this is also this is [23:44] Dan Guido's material. this was his slack message  uh to the organization, but he really talks about [23:50] really detailed things like, you know, we're  going to get everybody having, you know, a set of [23:55] reusable defaults and and and and so part of the  team uh was devoted to actually figuring out what [24:03] standardization looked like of the environment  that people are using, you know. So you you kind [24:09] of the the the the beach cleanup hackathon turned  up the issues but then he had a team uh to figure [24:15] that out and that kind of comes to someone else  who's a great guide. How many of you you here read [24:20] Ethan Mollik? A fair number of you all of you  should. He has a a substack called one useful [24:25] thing and uh he he he's a a business professor at  Wharton and he's all in on AI and he's spending [24:31] a lot of time thinking about the implications  for business. Um but one of his principles this [24:37] is a post from a year ago was called leadership  lab and crowd and he made the same point as Dan [24:43] the leadership has to go first uh uh uh and the  crowd all the people who are experimenting in the [24:50] company but you also need a team which he calls  the lab which is is taking those learnings and [24:56] systematizing them and building you know cool  new things that can be socialized around the [25:01] company. So, he sees those three functions and  I, you know, I I I see this again and again. I [25:07] I have a friend who is a an in-person trainer  uh who's working with the uh Internal Revenue [25:13] Service and his first thing was okay, we trained  the top 30 uh executives uh not top 30 IT people [25:23] uh and technologists at the IRS and they became  the lab group, right? then they train as where [25:30] they train other people they're they're getting  more and more systematization. So I think that's [25:33] a really useful way to think about it. But  another wonderful framing that Dan used was he [25:38] said our one of our jobs was to turn scar tissue  into infrastructure. So again, I think about, [25:46] you know, like my, you know, vibecoded uh stuff  that I've done with Saul and I watched it make all [25:52] these mistakes and Dan saying, "Yeah, when you  wait, watch it make all these mistakes and it, [25:57] sure, it corrected. It eventually got through the  process, but look at that. Look at that. Study it [26:03] and figure out how to fix that stuff. You want the  scar tissue not to remain as scar tissue but to to [26:11] um uh to turn into infrastructure. And there's  also a guy Drew Brunig who's very worth following [26:19] again as a substack and he's talks a lot about  something uh he calls prompt debt. You know he [26:25] he talks about the way that people are you know  trying to do something with AI and it doesn't [26:29] do what they want. So the the prompt starts out  simple and gets more and more complex. But the the [26:34] prompt is or sort of tuned to a particular model  and it's not portable, you know, and and so that's [26:40] scar tissue, you know, prompt debt is scar tissue.  So Dan's thing is, okay, how would we turn prompt [26:46] depth into infrastructure by really looking at it,  studying it, and figuring it out. So he says every [26:51] failure becomes a default. So again, you can see  in their repository how they've done it. you know, [26:58] he basically they literally have their cloud code  config repository. Whenever time cla didn't do [27:04] what they wanted, the the fix got loaded into a  global copy pasteable default that everybody had, [27:11] right? You know, it's just so they they built this  as organizational infrastructure. So rather than [27:17] you know, wow, I fixed it for myself and then  the next person comes along and they have to [27:21] fix it for themsel and maybe they fix it in  a slightly different way. It's like this is [27:25] where your lab has to go. No, no, we're going to  come up with with a standard uh you know um set [27:33] of defaults. So Dan built the first version  himself. Then he opened it companywide poll [27:37] requests. Uh but somebody is assigned after  each hackathon to collect what people didn't [27:42] contribute on their own. And um he he says if I  hire somebody tomorrow, I don't want them to go [27:48] through the entire discovery process of the last  year. Right? So, so this is step one of what you [27:58] know Ethan Mlet calls the lab you know it's just  like take this config or whatever you know maybe [28:03] you're not using cloud code maybe you're using  something else but whatever it is you know get [28:08] a team on getting it getting the scar tissue  out of there turned into you know configurable [28:14] defaults that everybody gets so that they get  a working system because this also addresses [28:19] one of these key psychological barriers like this  stuff doesn't work. It made a mistake. You know, [28:24] it's like, no, you want it to work out of the box.  So, you basically need to put in the homework to [28:30] make that happen. Uh, and then they basically put  all this out in public. They have three skills [28:37] repositories. There's an internal one for company  specific workflows, but they also have a public [28:42] repository. They say you know this forces the team  to um uh uh you know to write things others inside [28:50] and outside will actually understand and then  there's a curated list of third party skills. [28:55] So that seems like a pretty reusable pattern. Um  they also have done a lot of work on standardizing [29:02] the safety net you know in terms of the of the  issue that people worry about you know so they [29:07] have multiple sandboxes for different needs you  know they have uh they also have a 7-day cool [29:13] down on every new package from outside that gets  installed uh you know so that you know they say [29:18] most malicious supply chain packages get caught  by security researchers within hours so they're [29:24] security firm and so they got to really be on  top of that but he points he's his point is [29:29] everybody uses the same system but that when they  use it it's safe and it works. So if you have a [29:37] variety of systems across uh that you're going to  allow you know you go okay we're going to use all [29:42] these different different AI systems you got to  do this for all of them. So that maybe says well [29:47] you probably want to standardize to some extent  or you're you're building a big mess but we'll [29:53] figure that out. Um so and of course the next step  that they're working on is a vowels you know how [29:59] do you actually because that you know is right  now it's it's they put in these these uh copy [30:05] pasteable defaults but some of that is actually  still scar tissue it's just everybody's shared [30:10] scared shared scar tissue and uh so they're trying  to do a lot better in investing in in a val agent [30:18] needs a performance review uh but I mentioned  this the data problem earlier So you know [30:25] permission debt you know you basically want to do  something and you don't have permission to access [30:30] the data. This is a common corporate problem,  you know, and and you come across, you know, [30:37] I I look at a 500 company person company like  O'Reilly and I go, it's bad enough there. I go, [30:42] what's it like in a 50,000 person company? I have  no idea how they will deal with this. You know, [30:48] so I love it is every shortcut an organization  took with this data over the past 20 years becomes [30:54] visible the moment an agent hits it. you know  was one of Dan's observations but uh DJ Patil [31:00] who used to be the US chief data scientist who's  one of the guys who coined the term data science [31:05] you know talks about it's really not it's a it's  it's basically data 101 building unified data [31:11] environments you know clean data flows precise  metadata access but that really helps AI function [31:19] DJ designed the data infrastructure for a company  another company started by a friend of mine which [31:23] is doing really really well with AI And the CEO  says, "Yeah, I credit everything to the clean [31:29] data infrastructure. We've been able to do so  much with agents because the data was the data [31:34] infrastructure was there first." So, you really  have to take that one very very seriously. Um, [31:40] Dan also uh came up with what he thinks is a  useful metric, which is what fraction of your [31:45] data did you make reusable for other teams, you  know, and how much of theirs do you use? That's [31:50] a useful concept. The other thing he says,  you just got to put the reps in, you know, [31:55] and and people do need I I asked him at the end  of my interview with him. I said, 'Well, what were [32:00] five mistakes he made? He said, 'Well, there's  only one real mistake, and that's not giving [32:04] people enough uh f and find out time, you know, f  around and find out, fafo time, he called it. So, [32:13] um so people were in a very experimental period.  So, that was a success standardized in one agent [32:20] workflow. you can actually support at least to  start. Write an AI handbook. That's the other [32:24] thing we didn't really talk about. You know,  create a capability ladder uh that makes you [32:30] know improvement in this area expected. You know,  run short adoption sprints uh that force hands-on [32:37] usage. Capture everything in reusable artifacts,  skills, configs, and a curated supply chain. And [32:44] you know, sort of make autonomous agents safe  with sandboxes. He talked a little bit about what [32:50] postAI jobs look like and he you know he discussed  uh um you know the work doesn't disappear it [32:57] moves. He said you know technical editors use  hackathons to build tools that took them out of [33:01] the of line editing. Now they consult on how to  frame a story for the audience. you know ops and [33:06] go to market staff learn get in the command line  you know new roles emerge you know Q&A for agent [33:12] systems skill product management you know and  this is headcount that didn't exist before and he [33:18] said the delivery accountman he said you know for  certain engagements they went from 15 solving 15 [33:24] bugs a week they're they're a security consulting  firm 15 bugs a week to 200 bugs fixed a week [33:29] uh 20% of all bugs reported to clients are now  initially discovered by AI and he said their sales [33:34] team is averaging 8 million ion dollar revenue per  rep against an industry benchmark of two to four. [33:40] So seems like it's working for them. So but I want  to come to another topic which is very dear to my [33:47] heart and goes back to this question of the bitter  lesson. I believe that human expertise does still [33:53] matter in all of this. You know it's not just that  the AI is going to do it all for us. you know, [33:58] it's got if we just sit back and let the progress  of uh of AI carry us forward. And we got a really [34:08] interesting view of this from our CTO Andrew  Odawan. Uh and we've been thinking a lot. You [34:14] know, we are a company. You may think of us as a  company that produces books or learning paths or, [34:20] you know, video, whatever, but what we really are  is we're a matching marketplace for people with [34:26] expertise and people who need that expertise.  And so we think a lot about not just what our [34:35] customers need, but what our talent needs,  our sources of expertise. We introduced live [34:40] training for example because the book market  was shrinking and we went well how can we get [34:44] people to share their you know what's another way  that we can pay people to share their expertise. [34:49] So now we're doing that with agents and Andrew  sort of cracked the code I think. Uh we built [34:56] an MCP server for access to our content and the  first version was pretty pretty lame. You know, [35:02] it was just like search our metadata and then it  was like no actually you can actually get access [35:06] to all our our content. And what we figured  out is that we can use uh it to actually [35:15] increase organizational intelligence in  this way that I've been talking about. [35:19] So um you know Andrew points out everinccreasing  execution speed with a lower cost of of local [35:26] decision-making means that central plans have  splintered into team-based stuff. Projects are [35:30] shortcuts you know coordination with the teams.  It's harder and harder to get the big picture [35:34] view required to make good corporate decisions.  And so he was playing around with this. And now [35:41] this I really can't read now. Uh this is too  small. Can you make that any bigger? Uh he says, [35:46] "Okay, uh not long ago, one of my engineering  directors came to me with a request. His team [35:52] seemed overloaded and he wanted to hire another  engineer. I decided to test a research assistant [35:58] that I built building an AI agent connected to our  internal systems via MCP and asked it to analyze [36:03] the team's workload and write a hiring case." He  said, 'What came back was thorough. Headcount, [36:09] service ownership, sprint velocity, ticket  backlog, capacity duplication, capacity [36:14] allocation, all it was neatly summarized, but  reading it, I felt the same low-grade frustration [36:20] I had felt with every AI generated organizational  report I had seen. It told me what was happening [36:26] without helping me understand why or what I  should actually do about it. It was organized [36:31] around the data rather than around the decision.  It was the kind of response that is easy to agree [36:36] with and difficult to act on. And he said, "Then  they added one more thing to the configuration, [36:42] the O'Reilly expert MCP server. I reran the same  analysis and asked a slightly different question. [36:49] How would the experts on O'Reilly Media review  this request?" He said, "The output was different [36:54] in kind, not just in quality. Instead of leading  with headcount and ticket counts, it opened with [37:01] a finding. The operational overhead problem is  structural, not a staffing deficiency. It cited [37:08] the Google SR framework uh concept of operational  toil. Noted the team was operating approximately [37:15] 60% toil well above the threshold at which the S  sur literature recommended structural intervention [37:20] and made specific concrete recommendations. Run  a toll audit set specific explicit reduction [37:26] targets. Assign operational runbook ownership.  not consider whether to hire rather it gave me a [37:31] grounded traceable argument for doing something  different instead. So that was sort of that's [37:37] what you human expertise can do. It gives you a  unique perspective and we think we've found a way [37:43] to automate that so that can really help agents.  So we're building agents that you know can answer [37:49] questions like what's the status of some project,  help me develop a plan for some task, analyze [37:54] this product brief and help me produce a scope  statement. How can I re replace this system with [37:58] another? What are the trade-offs of adopting some  technology? How long will it take to do something? [38:03] What's involved in retiring some system? Help me  understand this team's workflow and roadmap and a [38:08] whole lot more. So, we're calling this O'Reilly  expert intelligence. And I think it's it's it's [38:13] it's kind of our example of how you know taking  this uh you know scar tissue and turning it into [38:19] infrastructure. Uh and here we're actually trying  to turn it into infrastructure that's reusable [38:24] for other companies. So this is in beta. Uh this  is actually just what I told you about was like [38:29] uh it's just the slide and we can kind of skip  through this but anyway this O'Reilly expert [38:36] intelligence I say is in beta. Uh you can sign up  for it. There's a QR code at the end of the talk [38:42] uh with and we have a bunch of MCP skills  associated with it that uh gonna be there [38:46] at launch. And um but I want to come up with just  end with some closing thoughts that try to make [38:53] sense of this um uh in the in the context I began  with you know so resolving the solo paradox you [39:04] know US productivity you know didn't rise because  computers got faster you know this is contrary to [39:11] the bitter lesson right it rose because companies  reorganized around what computers could do so the [39:17] Solo paradox, you know, disappeared, you know, a,  you know, three or four years after Solo named it, [39:23] you know, um, you know, um, and, you know, today's  people talk too much about model capability as [39:33] if that's going to be the magic weapon. You know,  it's like it's just these things are just going to [39:37] get better and faster. you know the bitter lesson  will win you know and you know what we learn from [39:44] the history of of technology revolutions is  that diffusion is hard there's a great book [39:50] uh I read recently called technology and the rise  of great powers and and it it it kind of contests [39:56] the theory you know which has been around for a  long time that you know technological revolutions [40:01] lead to you know transitions in national power you  know so for example Great Britain took over as the [40:06] world's national power on the back of steam you  know the US uh did on the back of electrification [40:13] and mass manufacturing and you know so on uh  you know he says but this is the this the idea [40:19] that if you dominate the leading sector you will  dominate he says actually what history teaches [40:23] is a different lesson which is the country that  makes it easiest to have widespread adoption of [40:30] a technology is the one that wins it's actually  adoption not capability that is the gating factor [40:38] And I think that that's what we uh are really  faced with. You know, nobody has done the [40:43] organizational work that Trail of Bits did and the  few who have are mostly keeping it to themselves. [40:49] So Dan really needs Dan Solo. Dan really needs to  be um congratulated for his work saying, "Okay, [40:58] here's how we did it. We want everybody to do it."  And so I think we really have to do that necessary [41:04] organizational work. But I want to come back to  this bitter lesson, you know, and just again why [41:11] AI needs us, you know, yes, we need AI. We can use  it to make our companies more productive. Think [41:18] about a horse race. You know, you don't just put a  bunch of horses out there and let them, you know, [41:25] run. They wouldn't actually run a race. You know,  we're in a horse race with our competitors with, [41:31] you know, or trying to figure out what our  customers and the horses need jockeyies, [41:35] right? You know, the horse matters, but the jockey  matters maybe a lot, too. And so does, of course, [41:43] the state of the turf and so on. That, by the  way, is Secretariat RA racing against some other [41:48] horse. Um, anyway, uh, and and Drew Bruner, who  I mentioned earlier, you know, refers to this as [41:55] pushing the model out of the average distribution.  You know, most people, you know, you can if you're [42:03] lazy, you can get adequate results with a simple  prompt. And that may be sufficient for many [42:09] purposes, but if you want to be good, you have to  actually really work at it. And our job is to push [42:17] the model out of the average distribution. Now,  Brew Drew referred to it as making it weird. And I [42:22] don't think it's making it weird. It's just making  it better. Way I describe it is AI is a medium [42:28] like writing or painting. Now how many of you are  artists? You know not that many but you know we [42:35] all can draw a little bit but take a look at what  Michelangelo did and what Van Go did. Each of them [42:40] had the same medium and they drew out something  amazing. And I think the same is true with us. [42:46] We're each unique and we have to get good at you  using the medium of AI in the same way that these [42:53] guys use the medium of painting to make something  brilliant. And I I wrote a post about this on my [42:58] conversations with AI Substack called uh why  AI needs you. Uh and what we choose to work on [43:05] matters a lot. This is a picture of me giving  a talk uh you know 20 years ago now. And and I [43:12] ended with this uh this picture by Delo of Jacob  wrestling with the angel and reading this poem of [43:18] Rilka which ends with this idea. What we fight  with is so small and when we win it makes us [43:23] small. What we want is to be defeated decisively  by successively greater beings. And I just want [43:29] to remind you all that our goal is not just to  make our businesses more efficient. It's to do new [43:36] and wonderful things. uh to try new and wonderful  things that might not have been possible before, [43:44] you know, to work on stuff that matters. The  world is full of great problems. And so if AI [43:50] takes away and makes easy something small, you  know, celebrate it and go work on something big [43:57] with the new powers that we've been given. So that  that's fine. And this is just a for those of you [44:04] who download the slide deck, I kind of describe  the process by which I built the slide deck and [44:08] it's got some credits including a link to Admir  Odawan's organiz organizational intelligence white [44:13] paper that I read you the first four paragraphs  of and uh and to the O'Reilly expert intelligence [44:19] thing and there's a uh QR code if you want to  actually try to sign up. So uh I don't think we [44:27] have well I mean I can stick around. We're at  the end of the conference, so if you guys have [44:30] questions, we can still stick around. And if  not, uh, thank you very, very much. [applause]