Surge 2015 - Lightning Talks

Surge 2015 闪电演讲 本次闪电演讲涵盖了多元的系统工程议题，首先探讨了 LISP 和 NDN 等协议架构、GCC 与 Clang 之间的编译器可调试性对比，以及 Joyent 因 Manta PostgreSQL 事务 ID 回卷 autovacuum 导致的 10 小时故障。演讲者们还探讨了利用多头绒泡菌黏菌进行的生物计算、技术写作技巧、基于 AMQP 的应用服务器，以及编译模型中的 CPU 指令缓存瓶颈。会议还穿插了关于育儿与值班轮换的个人感悟，以及将约吉·贝拉的名言作为 DevOps 原则的讨论，最后以一段再现近期 AWS DynamoDB 元数据故障的默剧表演，以及 Bryan Cantrill 对 SmartOS LX 品牌区域下 jemalloc 崩溃原因的技术剖析（该问题可追溯至 1999 年 Linux madvise 对 POSIX 规范的偏离）圆满收尾。 Joyent Manta 因 PostgreSQL 32 位事务 ID 回卷 autovacuum 引发的故障导致了 10 小时的停机，工程师在运行内存中使用 GDB 动态覆写成本延迟变量后才最终解决该问题。 (9:36) Bryan Cantrill 展示了 Linux 在 1999 年对 Tru64 非标准 MADV_DONTNEED 语义的历史性沿用，如何打破了 POSIX 建议性约定，并导致 Plex Media Server 等应用程序在 LX 品牌区域中发生不明原因的崩溃。 (58:23) 对未优化、已优化以及 Node.js 代码库的基准测试表明，在 DWARF 事后调试中，GCC 解析出的栈局部变量数量最多可达 Clang 的三倍。 (6:09) 对直接编译为原生机器码的机器学习模型进行的硬件分析表明，缓存逐出以及穿过狭窄内存总线引脚引起的指令匮乏，可能导致运行性能相比解释执行下降多达 5 倍。 (35:59)

The Surge 2015 Lightning Talks present a collection of operational postmortems, architectural provocations, and technical critiques, illustrating the pervasive consequences of undocumented side effects in production software. Operational Pitfalls and Technical Debt (9:36) Several presentations highlight how internal database and memory management behaviors can trigger catastrophic outages. David Pacheco details how PostgreSQL’s 'wraparound autovacuum' blocked client traffic in Joyent’s Manta object storage, forcing engineers to manually modify memory variables via a debugger to bypass rigid cost-based delays. Similarly, John describes the 'five stages of allocator grief' encountered when developing custom memory allocators, noting that without robust tooling like AddressSanitizer, production debugging becomes a disorienting, unpredictable process. Semantic Divergence and Kernel Design Bryan Cantrill critiques the long-term impact of 'semantic atrocities' in kernel design, specifically focusing on the divergence between POSIX and Linux. Cantrill argues that the Linux implementation of `madvise(MADV_DONTNEED)`—which treats an advisory hint as an imperative command to discard page data—causes silent data loss and violates the standard. He traces this behavior to an accidental importation of Tru64 UNIX semantics in 1999. This architectural mismatch forced Joyent engineers to spend significant resources troubleshooting crashes in third-party binaries like Plex that relied on Linux-specific memory expectations. Performance Myths and Hardware Realities (35:59) The program challenges industry trends favoring 'native' code translation. An AppNexus engineer argues that compiling machine learning models into native instructions often creates instruction bloat that saturates memory buses and CPU caches, yielding performance worse than interpreted logic. The talk underscores that modern hardware throughput is fundamentally constrained by physical pins and DIMM capacity, meaning bloated code frequently creates instruction-fetching bottlenecks that negate any potential optimization gains. Nature-Inspired and Unconventional Computing Experimental approaches to computation are scrutinized for their practical limits. Dimitris Kavourinis reviews the use of *Physarum polycephalum* (slime mold) to solve spatial routing problems, noting that while the organism effectively prunes redundant paths to mirror transportation networks, it cannot solve NP-complete problems in constant time. Similarly, Josh Schmiedel presents 'Jackalope,' a service-oriented architecture that replaces standard HTTP with RabbitMQ. He acknowledges that while creative, this approach highlights the difficulty of achieving scalability in systems that do not cleanly tolerate network partitions. Professional Standards and Infrastructure Culture The talks also address the human elements of engineering, from the importance of clear writing to the culture of incident response. A guest speaker argues that technical prose is often obscured by the passive voice and excessive adverbs, which obscure accountability in design documents. Additionally, Robert Treat draws parallels between the baseball aphorisms of Yogi Berra and DevOps culture, suggesting that Berra’s observations—such as 'you can observe a lot by watching'—are foundational to modern systems telemetry and the requirement to 'fail safely' during experimentation. The session underscores a recurring theme: software stability often hinges on understanding obscure, legacy behaviors and the physical realities of hardware, rather than merely prioritizing the newest optimization trends.

Surge 2015 闪电演讲汇集了多位一线系统工程师的实战复盘与架构反思，重点揭示了底层语义偏差、运维盲区与硬件物理约束如何对生产系统造成灾难性打击。 内核语义漂移与跨平台兼容灾难 (58:23) Bryan Cantrill 详细回顾了在 SmartOS（illumos）上开发 LX 品牌区以原生运行 Linux 二进制文件时遇到的技术噩梦。团队在排查 Plex 媒体服务器与 Splunk 频繁崩溃的问题时，最终通过可复现的 jemalloc 单元测试将故障根源锁定在 madvise 调用的 MADV_DONTNEED 标志上。按照 POSIX 标准规范，madvise 仅提供建议性提示，内核完全有权忽略，且调用决不能改变内存访问的实际语义。然而 Linux 的实现却将建议转变成了强制性命令：一旦对匿名私有映射调用该标志，内核会立即丢弃物理页面，后续读取则强制返回零填充页面。jemalloc 正是深度依赖了这一非标准语义，导致在严格遵循 POSIX 的系统上产生难以排查的内存损坏。追踪历史后发现，该语义源于 1999 年 SAP 工程师将企业软件从 Tru64 移植到 Linux 时引入的错误理解；Linux 社区不仅直接吸收了该违背 POSIX 的破坏性逻辑，在面对手册与行为不符的质疑时，更选择直接删去手册中“内核可忽略该建议”的说明，而针对该问题的 MADV_FREE 补丁即便在 2014 年基准测试证明性能大幅提升后，至 2015 年底仍未合入主线。 数据库底层机制引发的严重可用性故障 (9:36) David Pacheco 分享了 Joyent 旗下 Manta 对象存储平台的重大服务中断事故。事故根源在于负责存储集群元数据的 PostgreSQL 分片触发了事务 ID（XID）回卷自动清理机制（wraparound autovacuum）。该清理进程强制获取排他锁，导致所有到达主表的客户端读写请求被全量阻塞，引发 API 端两分钟超时与大规模 HTTP 500 报错。然而，由于该机制是防止数据库因事务 ID 耗尽而彻底关机的底层防御，运维团队无法直接终止该任务，只能全力加速其执行。工程师通过内核调度追踪发现，autovacuum 绝大部分时间处于空闲休眠状态：一方面因顺序读取目录内成千上万个离散文件导致文件系统预读失效，团队随后手动发起了预读；另一方面，PostgreSQL 内部内置的“基于开销的清理延迟”（cost-based vacuum delay）机制强制让清理进程周期性休眠以让出资源给业务查询，但当时除该进程外所有业务线程均已卡死在锁上。由于系统缺乏动态配置调整能力，工程师最终借助 GDB 调试器直接修改运行中进程的全局变量内存，将清理速度提升了 5 倍。 编译优化与代码膨胀对系统可观测性的冲击 (6:09) Abel Mathew 与来自 AppNexus 的工程师分别指出了盲目追求编译器与机器码性能优化所带来的副作用。Abel Mathew 强调，工程团队在评估从 GCC 迁移至 Clang 时普遍忽视了“调试可观测性”指标。他在 Ubuntu 15.04 环境下对比 GCC 4.9.2 与 Clang 3.6.0 的测试表明，在开启 -O2 优化后，Backtrace 内部程序中 GCC 仍能解析出约 1,700 个局部变量，而 Clang 仅保留约 500 个；在 Node.js 主干分支测试中，GCC 解析出约 1,600 个局部变量，Clang 则骤降至 475 个，导致崩溃转储分析丢失大量关键上下文。来自 AppNexus 的工程师则驳斥了将机器学习模型直接编译为底层机器码能够提速的行业迷思。他指出，由于内存总线吞吐量物理限制（每 DIMM 每时钟周期仅能传输约 5 字节），庞大的机器指令与 C++ 模板膨胀会导致指令体积远超处理数据，造成严重的 L1/L2 指令缓存未命中，在冷指令缺失时实际每时钟指令数（IPC）甚至跌落至 0.12，使编译模型比常规逻辑慢上 5 倍。 架构抽象的权衡：消息中间件与分布式演进 (28:24) 在分布式架构演化方面，演讲者们展现了从实用黑客手段到模型反思的演进过程。Josh Schmiedel 介绍了其团队构建的 Jackalope 框架，该框架利用自定义客户端 AMQParty 将上层 HTTP 调用伪装并序列化推入 RabbitMQ，后端由 Jackalope 消费队列并模拟 Rack 环境交由 Ruby 处理，以实现跨网络的服务隔离。然而 Josh 坦言，生产实践验证了 RabbitMQ 官方文档所指出的缺陷——扩展性不佳且对网络分区容忍极差，促使他构想转向基于 Apache Kafka 的消费层，并配合底层数据包重写机制将响应直接送回客户端。此外，Dan McDonald 探讨了未来互联网协议演化路径，强调传统 IP 体系自 1981 年以来将“节点身份”与“拓扑定位”绑定在同一地址上的弊端，展示了通过分离身份与定位实现跨数据中心无缝迁移的前景，并展望了以内容命名的命名数据网络（NDN）等长期演进方向。 本场闪电演讲表明，盲目信任编译器优化、直接依赖黑盒数据库维护机制或简单引入网络抽象层往往会在生产环境中引入致命陷阱。真正的系统稳定性取决于对内核语义、内存布局和物理硬件限制的深度认知与调试能力。

## Background 该视频由频道 OmniTI 于 2015 年 11 月 29 日发布在 YouTube 上。内容为 2015 年 Surge 大会的闪电演讲（Lightning Talks）集锦，由 AppNexus 赞助、Geir Magnusson 主持，简介戏称其中还穿插了“意象派现代舞”。 ## TL;DR 本期闪电演讲汇集了多位资深工程师与架构师关于系统底层与工程实践的高能分享。内容涵盖未来网络架构（ILNP 与 NDN）的演进、编译器对可调试性的隐性杀伤、Joyent 遭遇的 PostgreSQL 事务回卷导致 10 小时大宕机的极限排障复盘，以及生物黏菌计算与工程师技术写作法则等多元议题。后半段讲者们深入探讨了硬件内存带宽墙瓶颈、用即兴现代舞拆解 AWS DynamoDB 级联雪崩事故，并由 Brian Cantrill 压轴深入追溯了 Linux 内核因历史借鉴 Tru64 错误语义而导致 madvise 破坏 POSIX 规范、进而引发 jemalloc 崩溃的深层技术罪行。 ## Atmosphere & style 现场氛围极其热烈且充满硬核极客的黑色幽默与自嘲精神。从主持人幽默风趣的控场，到讲者用“灾难大片”视角复盘线上生产事故的惊心动魄，再到现场拉观众上台用现代舞即兴演绎 AWS 宕机事故，全场在技术严肃性与荒诞戏剧感之间自如切换。听众仿佛置身于一场既能学到底层内核黑客级排障技巧、又能与同行心领神会爆笑共鸣的高水平工程狂欢。 ## Bulletin - [09:36](https://www.youtube.com/watch?v=bg6-LVCHmGM&t=576s) Joyent 的 Manta 分布式存储系统因 PostgreSQL 事务回卷自动清理陷入 10 小时全局停机，团队通过挂载 GDB 动态修改内核与延迟变量、调高 autovacuum 阈值才最终化解危机。 - [06:09](https://www.youtube.com/watch?v=bg6-LVCHmGM&t=369s) 编译器基准测试显示，在开启 -O2 优化后，Clang 相比 GCC 会激进消除多达 70% 的栈局部变量，严重削弱事后崩溃分析与 Core Dump 排障的可调试性。 - [35:59](https://www.youtube.com/watch?v=bg6-LVCHmGM&t=2159s) 硬件瓶颈正从算力转向内存带宽：Intel Xeon E5 处理器每时钟周期每个 DIMM 仅约 5 字节带宽，将庞大模型全盘展开为机器码会导致缓存穿透并使 IPC 暴跌至 0.12。 - [58:23](https://www.youtube.com/watch?v=bg6-LVCHmGM&t=3503s) Brian Cantrill 深度复盘 Linux 内核历史，揭示 Linux 因 1999 年照搬 Tru64 的误读实现，使本该仅作性能建议的 madvise(MADV_DONTNEED) 具备强制清零与丢弃数据的破坏性语义，最终引发现代程序跨平台兼容崩溃。 ## Key takeaways - [01:03](https://www.youtube.com/watch?v=bg6-LVCHmGM&t=63s) 下一代互联网架构探索聚焦于两大方向：中期推进 IP 层的“位置与身份解耦”（如 ILNP 和 LISP 协议），长期则转向以内容寻址和自带数字签名为核心的命名数据网络（NDN）。 - [06:09](https://www.youtube.com/watch?v=bg6-LVCHmGM&t=369s) 在评估从 GCC 迁移至 Clang 时，团队必须权衡编译速度与生产环境可调试性，Clang 激进的优化可能导致 Core Dump 中大批局部变量丢失。 - [10:30](https://www.youtube.com/watch?v=bg6-LVCHmGM&t=630s) PostgreSQL 拥有 32 位事务 ID 环形空间（40 亿上限），一旦接近回绕阈值便会强制触发排他性防回绕清理；默认的 2 亿次事务触发门槛在极高吞吐核心业务中极易引发死锁与级联崩溃。 - [13:53](https://www.youtube.com/watch?v=bg6-LVCHmGM&t=833s) 针对 autovacuum 导致的线上停机，Joyent 工程师利用内核调度器跟踪定位磁盘预读短板，并直接挂载调试器改写内存中节流变量以获得 5 倍读吞吐加速。 - [15:55](https://www.youtube.com/watch?v=bg6-LVCHmGM&t=955s) 生物计算（如黏菌实验）虽能基于自组织物理剪切收敛出近似东京地铁的优秀网络拓扑或构建生物或非门，但因约 1 cm/h 的生长速率和无法速解 NP 完全问题，在工程上远无法替代硅基芯片。 - [23:02](https://www.youtube.com/watch?v=bg6-LVCHmGM&t=1382s) 技术写作应当全面向被动语态和冗余副词宣战，通过紧凑的主谓结构与短句提升工程沟通效率和文档可读性。 - [28:24](https://www.youtube.com/watch?v=bg6-LVCHmGM&t=1704s) 试图通过在消息队列上层伪造完整 HTTP 语义来搭建微服务架构存在严重缺陷，RabbitMQ 等组件对横向扩展与网络分区的脆弱性极易成为系统瓶颈。 - [32:08](https://www.youtube.com/watch?v=bg6-LVCHmGM&t=1928s) 从零在用户态编写内存分配器在生产高并发环境下极难调试，缺乏红区保护和编译器 Sanitizer 支持时往往会引发不可追踪的内存踩踏，应优先采用成熟的工业级分配器（如 jemalloc）。 - [35:59](https://www.youtube.com/watch?v=bg6-LVCHmGM&t=2159s) “原生机器码天然更易扩展”是一种工程幻觉：大基本块代码未命中 CPU 缓存时会撞上“内存带宽墙”，导致预取与推测执行全盘失效。 - [42:28](https://www.youtube.com/watch?v=bg6-LVCHmGM&t=2548s) 高频的生产警报值守（PagerDuty）与初为人父半夜照顾婴儿在心理和生理机制上高度相似，唯独婴儿能通过安抚停止哭闹，而硅基系统则冰冷无情。 - [45:51](https://www.youtube.com/watch?v=bg6-LVCHmGM&t=2751s) 约吉·贝拉的经典语录精准映射了现代 DevOps 哲学，涵盖敏捷计划与交付脱节、云端资源争抢、系统度量与可观测性等核心痛点。 - [54:21](https://www.youtube.com/watch?v=bg6-LVCHmGM&t=3261s) 2015 年 AWS DynamoDB 重大故障源于网络微小中断后，存储节点续约超时触发激进重试风暴，最终由于元数据服务被重试洪峰压垮而引发全网级联雪崩。 - [58:23](https://www.youtube.com/watch?v=bg6-LVCHmGM&t=3503s) 在 SmartOS 内核中通过系统调用翻译原生运行 Linux 二进制程序，要求系统开发者必须被迫全盘复现 Linux 历史上所有不合规范的底层反常语义。 - [01:06:57](https://www.youtube.com/watch?v=bg6-LVCHmGM&t=4017s) POSIX 规范明确规定 madvise 仅能作为无副作用的性能优化建议（Hint），绝不允许改变指定内存的访问语义。 - [01:16:18](https://www.youtube.com/watch?v=bg6-LVCHmGM&t=4578s) Linux 对 madvise(MADV_DONTNEED) 的破坏性清零实现源自 1999 年 SAP 移植 Unix 应用时照搬了 DEC Tru64 的误读，导致后续依赖 POSIX 语义的程序产生灾难性数据异常与崩溃。 - [01:18:34](https://www.youtube.com/watch?v=bg6-LVCHmGM&t=4714s) Linux 社区面对 madvise 规范违背问题，历经十余年未能合入符合规范的 MADV_FREE，甚至以直接在 man 手册中删除“内核可以忽略此建议”来掩盖语义缺陷。 ## Detailed walkthrough ### [00:12](https://www.youtube.com/watch?v=bg6-LVCHmGM&t=12s) 闪电演讲开场与规则介绍 主持人 Geir Magnusson 登台宣布 2015 年 Surge 大会的闪电演讲环节正式开始。他提到这个环节在大会一直有着悠久的历史，而且向来非常有趣。与业界常见的每人 2 分钟标准闪电演讲不同，Surge 大会经由 Brian 的提醒将演讲时间放宽到了整整 5 分钟。演讲的主题非常多元，可以搞笑幽默，可以严肃硬核，涵盖任何话题，而且经常会带给观众意想不到的惊喜。Geir 在开场时特别向大会组织者 Robert 及全体筹办团队致谢，感谢他们搭建了这个出色的交流平台，并衷心感谢赞助商 AppNexus 的大力支持。随后，他引出了第一位演讲嘉宾——来自 OmniTI 的 Dan McDonald，演讲主题聚焦于未来互联网架构的两大发展方向。 ### [01:03](https://www.youtube.com/watch?v=bg6-LVCHmGM&t=63s) Dan McDonald：未来互联网的两大演进方向 来自 OmniTI、日常从事操作系统相关工作的 Dan McDonald 登台分享了当前学术界与互联网研究工作组（IRTF）正在探索的两个前沿方向。他指出，经典的互联网协议栈呈现出著名的“协议沙漏”（Protocol Hourglass）结构：最顶层是应用程序，往下依次是表示层、传输层，最狭窄的腰部是 IP 协议，随后再向底层的各类链路层和物理介质展开。除去 IPv6 带来的更大地址空间之外，自 1981 年以来这套沙漏架构其实并未发生根本性的结构变动。当前研究的两个新方向彼此正交，但未来很可能会对运维领域（Ops）产生深远影响。其中第一个方向预期属于中期落地目标，即在 IP 数据包中实现“位置与身份的分离”（Separation of Location and Identity）；第二个方向则是长期目标，即重构整个沙漏模型的“命名数据网络”（Named Data Networking, NDN）。 ```mermaid flowchart TD accTitle: 互联网协议演进对比 accDescr: 对比传统沙漏模型、位置/身份分离模型与命名数据网络架构 subgraph Traditional ["传统 IP 沙漏架构"] T_App["应用程序"] --> T_Trans["传输层 (TCP/UDP)"] T_Trans --> T_IP["IP 层 (兼具位置与身份)"] T_IP --> T_Link["链路层与物理介质"] end subgraph LocId ["位置与身份分离 (ILNP / LISP)"] L_ID["节点唯一身份 (Identifier)"] L_Loc["动态路由位置 (Locator)"] L_ID --- L_Loc end subgraph NDN ["命名数据网络 (NDN)"] N_App["应用程序与安全签名"] N_Chunk["内容块 (Content Chunks)"] N_Strat["分发策略与存储转发"] N_App --> N_Chunk --> N_Strat end ``` 在现有的网络中，IP 地址既代表了节点在骨干网路由中“如何到达”的位置标记（Locator），又同时代表了主机的身份（Identity）。Dan 以自己的公共 Web 服务器（kb.com，运行在其家庭服务器五个 Zone 之一）为例，说明流量如何经由骨干网、在纽约 Verizon 出口路由最终抵达马萨诸塞州北部。如果将身份与位置彻底解耦，节点即可拥有与地理和拓扑位置完全独立的唯一身份标识。关于这项技术在云基础设施中的实际意义，Dan McDonald 明确指出：“Not that AWS would ever go down, but imagine if AWS West goes down and you can reinstantiate your very node with the same identifier just with a different locator in AWS East and if you're really good you could keep those two nodes redundant State up to date with each other.”（并不是说 AWS 会宕机，但设想一下，如果 AWS 美西区域宕机，你可以在 AWS 美东以相同的标识符、仅换用不同的位置重新实例化同一个节点；如果你足够优秀，甚至可以让这两个节点的冗余状态保持同步）。在延迟允许的情况下，这种解耦甚至理论上能维持长连接 TCP 会话不中断，并赋予移动设备真正的 IP 层漫游能力。 目前针对这一方向主要有两个主流提案： - **ILNP（Identifier-Locator Network Protocol，RFC 6740）**：由 Dan 的老朋友兼导师 Ran Atkinson 提出，属于实验性 RFC，核心做法是将 IPv6 地址划分为明确的身份组件与位置组件。 - **LISP（Locator/ID Separation Protocol）**：由 Dino Farinacci 以及 Cisco 相关团队主导，采用“IP in IP”的封装隧道方案，在一个数据包内同时封装标识空间与定位空间。 另一个更加激进的长期演进路线是**命名数据网络（NDN，named-data.net）**，最初由 Van Jacobson 在 PARC（施乐帕洛阿尔托研究中心）发起，后扩展至广泛的学术界研究中。NDN 彻底打破了传统沙漏：应用层下方直接是安全与“内容块”（Content Chunks），中间路由节点演变为全网通用的存储转发（Store-and-Forward）节点，运作模式更接近 Usenet 或电子邮件系统，而非传统的 IP 数据报交换。由于每个内容块都自带数字签名，接收端在拼装数据对象时能直接校验签名，天然具备端到端的安全性。Dan 幽默地畅想，未来如果这种架构铺开，网络中间节点可能会遍布运行着 OmniOS、SmartOS、搭载 ZFS 并部署 Manta 存储系统的 middlebox。 ### [06:09](https://www.youtube.com/watch?v=bg6-LVCHmGM&t=369s) Abel Matthew：编译器的选择与软件可调试性 紧接着登场的是来自 Backtrace 的 Abel Matthew，他的演讲聚焦于一个工程选型中极易被忽视的维度——**可调试性（Debuggability）**。Abel 观察到，很多团队在评估是否从 GCC 切换到 Clang 时，通常只关注编译速度、可执行文件的运行性能以及编译报错信息的友好度（这是 Clang 的常见卖点），却很少有人停下来思考编译器对可调试性到底有什么影响。 为了探究这个问题，Abel 当天下午在 Ubuntu 15.04 上做了一组对照实验，对比了 GCC 4.9.2 与 Clang 3.6.0：他通过遍历崩溃栈轨迹（Stack Trace），统计在各个可访问的栈帧中，有多少局部变量能够被正常解析，还是被编译器完全优化掉了（Optimized out）。实验结果呈现出巨大反差： - 在 Backtrace 的内部应用程序中，无优化编译下 GCC 能解析近 3,000 个变量，而 Clang 只有约 1,200 个；在开启 `-O2` 优化后，GCC 依然能解析约 1,700 个变量，而 Clang 骤降至约 500 个。 - 在对 Node.js 源码主分支（master）的编译测试中，开启优化后，GCC 解析出了约 1,600 个变量，而 Clang 仅解析出约 475 个。 针对这一实验结果在运维与线上排障中的严重影响，Abel Matthew 警告道：“If those variables are optimized out, the less information that you have, the harder it is to figure out what's going on.”（如果这些变量都被优化掉了，你掌握的信息就越少，想要搞清楚究竟发生了什么就会变得极其困难）。当研发团队兴致勃勃地提议切换到 Clang 时，运维团队必须把事后崩溃分析（Post-mortem Crash Analysis）与 Core Dump 排障的难度考虑在内。在随后的现场问答中，Abel 补充说明，这次测试使用了 Backtrace 自研的调试工具，其底层基于 `libunwind` 展开栈轨迹，并利用二进制文件中的 DWARF 调试信息尝试提取自动变量（栈上的局部变量）。 ### [09:36](https://www.youtube.com/watch?v=bg6-LVCHmGM&t=576s) David Pacheco：Manta 灾难日与 PostgreSQL 事务回卷 最后一位登场的演讲者是 Joyent 的资深工程师 David Pacheco，他带来了一场被主持人调侃为“灾难大片”（Disaster Porn）的故障复盘演讲，题目为《Manta 的倒霉日》（Manta's Bad Day）。Manta 是 Joyent 开发的分布式对象存储系统。David 坦言，原本他不知道如何在短短 5 分钟内讲完这样一场严重的生产事故，直到看到了同事 Brian 当天发布的一条推特，该推特仅用两个词就总结了事故的死因——“Wraparound autovacuum”（事务回卷自动清理）。 更具戏剧性的是，就在同一周内，Joyent 甚至并不是第一家因为 PostgreSQL 事务回卷自动清理而遭遇大规模严重宕机事故的公司。David 针对这种冷门却具有毁灭性的故障总结道：“The only way you can remember about wraparound autovacuum is to have actually experienced an outage as a result of it as far as I can tell.”（据我所知，能让你记住事务回卷自动清理的唯一途径，就是你切身经历过一场由它导致的线上宕机事故）。如果运维人员之前从未听说过这个机制，纯粹是因为线上生产环境还没被它彻底整垮过。只要系统依赖 PostgreSQL，未来某一天就很可能会踩中这个深坑。随后，David 开始正式深入拆解 Manta 对象存储服务在此次危机中的具体技术细节。 ### [10:30](https://www.youtube.com/watch?v=bg6-LVCHmGM&t=630s) Manta 元数据层的 PostgreSQL 故障与崩溃初现 演讲者回顾了他们在 Joyent 构建的分布式对象存储系统 Manta。该系统基于 HTTP 提供类似 Unix 目录结构的字节块（byte blobs）的 PUT、GET、DELETE 操作，强调强一致性，其底层高度依赖基于 PostgreSQL 的元数据层。为了支持扩展性，系统进行了分片；为了高可用，每个分片部署了多个 PostgreSQL 实例。系统的所有数据路径操作以及在其上运行的计算任务，全部需要访问这一元数据层。然而在 7 月 27 日 UTC 时间 10:30，一场严重的故障爆发了。 客户端最先感知到延迟急剧飙升，随后服务开始返回大量 500 状态码错误。这是典型的级联超时模式：客户端发出请求，内部处理受阻，API 服务器在等待 2 分钟超时后向客户端回执 500 错误。排查人员迅速将范围锁定在其中一个 PostgreSQL 数据库节点上。奇怪的是，执行简单的探测查询（如 `SELECT now();`）运行完全正常，说明数据库引擎本身并没有彻底崩溃。然而，Manta 核心业务表上的所有查询都被死死锁住。整张表甚至整个数据库中，只有一条查询没有被排他锁阻塞——那是在 10:30 分准时启动的 `autovacuum`（自动清理进程）。 - **架构特性**：基于 HTTP 的强一致性对象存储，具有多分片与多分片内多副本的 PostgreSQL 元数据架构。 - **故障现象**：客户端大面积超时，API 层触发 2 分钟超时后抛出 HTTP 500。 - **核心异常**：全库所有请求排队锁死在核心表上，唯一在运行的只有 10:30 准点触发的 `autovacuum`。 ```mermaid flowchart TD accTitle: PostgreSQL 数据库元数据层阻塞与 500 超时链路 accDescr: 客户端发起请求到 API 触发 2 分钟超时，根因在于 autovacuum 占用全局关键表锁导致所有查询阻塞。 subgraph ClientFlow["客户端交互层"] A["客户端请求 (PUT/GET/DELETE)"] --> B["API 服务器"] B -->|"2 分钟无响应超时"| C["返回 HTTP 500 错误"] end subgraph DBCluster["PostgreSQL 元数据分片"] B -.->|"读写阻塞"| D["业务元数据核心表"] E["SELECT now()"] -->|"成功执行"| F["系统健康探针"] G["autovacuum 进程 (10:30 启动)"] -->|"独占表级锁/资源"| D end ``` ### [11:39](https://www.youtube.com/watch?v=bg6-LVCHmGM&t=699s) 事务环绕风险与内核级 I/O 调优 团队此前就多次遭遇过 autovacuum 不足引发的问题，但由于此前往往是在故障自愈或直接强制中断清理进程后草草了事，一直未能彻底定位根因。这次为了彻底排查，团队抓取了 `pg_locks` 的详细锁状态数据，随后重启了 PostgreSQL。重启后系统恢复了大约 10 分钟，但数据库随即再次触发了一次 autovacuum，并且这次明确标记为“防止事务回绕”（prevent wraparound）。 PostgreSQL 的 32 位事务 ID 是一个环形空间，总量 40 亿，以当前值为基准前后各 20 亿；一旦 ID 回绕越界，过去的事务会变成“未来”发生，造成灾难性数据混乱甚至直接导致数据库停机保护。此时必须让清理进程执行完毕。为了加速该进程，团队开始利用低层工具追踪系统表现，惊讶地发现该 PostgreSQL 进程绝大部分时间竟然处于睡眠状态（sleeping）。 ``` “This is one of those cases where people are like, ‘I'm building an application, why do I need to trace the kernel scheduler?’ Well, because your thing is sleeping all the time and you want to know why.” —— 演讲者 ``` 通过内核调度器追踪，他们定位到进程大部分时间都在等待 I/O。autovacuum 在连续读取文件时由于文件系统自带预读尚能应对，但在同一目录下跨多个文件进行大范围顺序扫描时，底层文件系统完全无法识别这种模式。团队通过主动针对目标文件实施预读（prefetching），成功消除了 I/O 停顿。 - **事务 ID 回绕机制**：32 位整型无符号数构成的环形空间，20 亿前后分界，触发回绕会导致数据库自我关闭以保护一致性。 - **排查手段**：提取 `pg_locks` 锁等待矩阵，利用内核调度器链路跟踪深入分析进程睡眠成因。 - **预读优化突破**：补足底层文件系统无法跨同一目录下多个连续文件进行自动预读的缺陷。 ### [13:53](https://www.youtube.com/watch?v=bg6-LVCHmGM&t=833s) 调试器热修变量与突破 10 小时故障阴霾 解决了底层磁盘 I/O 等待后，团队发现 autovacuum 依然频繁陷入睡眠。这次的元凶是 PostgreSQL 内部的一套自我节流机制——基于成本的清理延迟（Cost-based Vacuum Delay）。该机制原本用于防止清理操作耗尽系统硬件资源，每运行一段时间便强行休眠片刻。但在此刻，除它之外的所有查询都在等待它释放锁，让它主动降速反而成了最坏的策略。 PostgreSQL 原生没有动态关闭该机制的接口，但面对不可承受的停机时间，团队展现了极端的工程手段：既然所有代码和数据都在主机内存中，他们直接挂载调试器（GDB），在运行时硬核改写（whack）了控制清理延迟的全局变量，使清理读吞吐立刻获得了 5 倍的暴增。 清理进程在持续预读数小时后终于结束，服务恢复正常。然而不到 10 分钟，防回绕 autovacuum 竟然再次在同一张表上重新触发，一切又回到了起点。团队意识到必须换个思路：虽然 20 亿是事务回绕的红线，但 PostgreSQL 默认在事务计数达到 2 亿（200 million）时就会保守地启动紧急清理。对于常规运维而言 2 亿很保守，但对于核心系统停机而言这无异于灾难。团队随后果断将这一触发阈值调高到 5 亿（500 million），服务终于稳定并脱离险境。这场长达 10 小时的大停机让演讲者投入了整整一周撰写事后复盘：autovacuum 本身不会直接打垮数据链路，而是上层应用程序某些极为隐蔽的使用模式，在特定场景下意外将其推入了死锁和吞吐崩溃的深渊。 - **内存强修**：使用调试器在运行时直接修改 cost-based vacuum delay 的全局变量，强行移除节流逻辑，获得 5 倍读吞吐加速。 - **阈值调整**：将默认极度保守的 2 亿次事务自动清理门槛调大至 5 亿次，避免短时间内再次陷入死锁循环。 - **故障耗时**：整体服务中断达 10 小时，后续根因分析耗时整整一周。 ### [15:55](https://www.youtube.com/watch?v=bg6-LVCHmGM&t=955s) 生物计算的想象力：从粘菌到东京地铁模型 下一场闪电演讲由 Dimitri Karvonis 带来，主题围绕生物计算展开，副标题定为《生物计算的未来，还是言过其实的淤泥》（The future of biological Computing or overhyped sludge）。主持人 Geir Magnusson 调侃其为“Ruby on slime”，Dimitri 笑称这正好剧透了他 PPT 中仅有的几个笑话之一。演讲从草坪或树干上常见的黄橙色泡沫——粘菌（slime mold）切入。它没有大脑，既不属于动物界、植物界，甚至也不属于真菌界，“霉菌”（mold）一词实为命名误解；它是由类似变形虫的真核单细胞生物构成的自组织网络系统，却能表现出极其复杂的智能行为，甚至引发了诸如“为什么我的粘菌比你的 Hadoop 集群更强”的网络热梗。 Dimitri 详细介绍了日本研究人员中垣俊之（Toshiyuki Nakagaki）等人关于粘菌的经典实验：在实验容器中心放置粘菌母体，并在外围按照东京地铁网络车站的实际地理方位撒上燕麦碎粒。粘菌为了摄取养分迅速向外延伸出密集的管状网络，随后通过管流剪切力逐渐修剪掉非必要的连接边缘，仅保留高效率主干。最终形成的网络拓扑高度贴合现有的东京地铁网络，在冗余性、抗损坏能力与建造成本的综合平衡上甚至比人类工程师设计出的拓扑更为优化。 ``` “In other words, you can just throw down a bunch of grains and the slime mold will find the optimal configuration.” —— Dimitri Karvonis ``` 进一步的研究已经开始探索如何用粘菌实现通用计算。研究人员不仅能将粘菌塑造成任意微流控物理结构，甚至成功利用其原生特性构建出了逻辑“或非门”（NOR Gate）。因为或非门具备逻辑全备性（functional completeness），理论上人类可以利用粘菌构建出任何数字逻辑电路。目前已有名为“PhyChip”的研究项目致力于开发粘菌生物计算芯片，涵盖类比东京地铁网络的最优路径模拟计算，以及基于或非门的数字化逻辑计算。尽管粘菌的生长速度仅有每小时 1 厘米左右，实用性仍然面临极大质疑，但这种生物自组织系统对复杂拓扑求解与替代计算模型的探索，依然展现出迷人的科学潜力。 - **生物属性**：非动物、非植物、非真菌，由类似变形虫的单细胞真核生物组成，无中枢神经却具备去中心化决策机制。 - **经典实验**：东京地铁站食物点拓扑模拟实验，证明其网络自修剪机制能收敛于全局近优解。 - **计算逻辑**：可构建生物或非门（NOR Gate），支撑从模拟图寻优到数字电路的生物芯片（如 PhyChip 项目）研制。 - **物理局限**：生长及反应速率仅约为 1 cm/h，计算收敛极其迟缓。 ### [19:25](https://www.youtube.com/watch?v=bg6-LVCHmGM&t=1165s) 黏菌计算的物理局限与荒诞构想 演讲者继续深入探讨利用黏菌构建计算逻辑门（如 NOR 门）的现实困境与物理局限。他直言不讳地指出，既然当今人类已经能够以更高的良率、更大的规模制造出性能远超生物体的 NOR 门，花费大力气用黏菌来做这件事在工程上显得毫无必要。如今每个人随身携带的智能设备中，所包含的逻辑门数量比用黏菌实际制作出来的要高出几个数量级。 将网络优化问题和黏菌的独特“智慧”联系起来，实际上延续了人类试图借助物理或生物过程来辅助计算的传统（最典型的极致探索便是量子计算）。然而，物理过程往往无法为复杂的计算难题提供真正可靠的捷径。演讲者引用了理论计算机科学家斯科特·亚伦森（Scott Aaronson）的著名实验：将肥皂水倒在夹在玻璃之间的线轴上，肥皂水形成的薄膜并不会自动收敛成最小斯坦纳树（Minimum Steiner Tree）。这有力地打破了“肥皂水能在极短时间内瞬时解决 NP 完全问题”的谣传。 ```mermaid flowchart TD accTitle: 物理与生物过程求解计算难题的局限性 accDescr: 对比肥皂水和黏菌在面对复杂计算问题时的实际表现与理论局限 subgraph SoapBubble["肥皂水实验（Scott Aaronson）"] S1["肥皂水注入玻璃线轴"] --> S2{"求解最小斯坦纳树？"} S2 -->|无法瞬时求解| S3["打破 NP 完全问题速解神话"] end subgraph SlimeMold["黏菌实验"] M1["黏菌觅食网络生长"] --> M2{"求解 NP 完全问题？"} M2 -->|无法精确求解| M3["实验证实：仅能提供 TSP 近似解"] end ``` 黏菌同样无法替人类直接解决 NP 完全问题。不过，实验表明它确实能够为旅行商问题（Traveling Salesperson Problem, TSP）得出一个不错的近似解——演讲者幽默地感叹：“对于一个根本没有大脑的生物来说，这已经相当令人印象深刻了。”大屏幕上展示了黏菌求解 TSP 时的形态演变，随着网络不断演化，逐渐呈现出清晰的网络脉络。 面对“我们能否拥有生活在大自然中、具备自组装、自愈合、适应变化、原料成本极低且零电费开销的自主生物电路”这一设想，演讲者坦言目前人类在工程上还做不到。但在实验室观察黏菌在各种环境下形成不同结构的形态学研究，依然具有正统科学价值。在此基础上，演讲者抛出了一连串带有极客式幽默的调侃与设想： - **提出全新的“黏菌复杂度类”**：衡量计算复杂度的核心标准变成了“解决该问题需要用黏菌覆盖地球多大面积”，并指出在有限的黏菌宇宙中，计算正遭遇物理极限，例如：“要解决包含一百万个节点的最大团问题（Clique Problem），需要吞噬掉几个大洲的陆地面积？” - **复古慢计算营销**：如果算力不够强大，可以将其包装为“手工打造的、纯天然模拟生物计算（Artisanally branded all-natural analog bio-computing）”，主打产自本地森林的“慢计算（Slow Computing）”。 - **开源项目倡议**：发起“Ruby on Slime”项目来引领黏菌革命，其宣传语直接沿用经典梗：“因为你已经在用 Ruby 了（Because you're already using Ruby）”。 - **跨界艺术应用**：展示音乐教授爱德华多·米兰达（Eduardo Miranda）与黏菌共同完成的双钢琴合奏，将黏菌的生物电信号作为音乐艺术创作的灵感源泉。 --- ### [23:02](https://www.youtube.com/watch?v=bg6-LVCHmGM&t=1382s) 工程师写作自救指南：消灭被动语态与副词 随后的讲者带来了一场完全在手机上临时准备、构思于走廊里的即兴闪电演讲。讲者开门见山地表明了自己的双重身份：在现实生活中他是一名工程总监，但同时他还是一名职业作家，长期进行大量写作与文字编辑工作。然而，当他坐在办公室里阅读工程师撰写的设计文档或技术总结时，其惨烈程度让他直呼“痛苦到内心枯萎（makes me die inside）”。更令人绝望的是，当他在 Google Docs 上辛辛苦苦提出修改意见时，工程师们往往直接点击“拒绝修改”，这让他下定决心在大会上为技术人员恶补语法。 讲者指出，工程师写作中面临的头号顽疾在于“极度畏惧主语和动词”，字里行间表现得仿佛主语和谓语会随时从屏幕里跳出来掐死作者一样，这直接导致了通篇充斥着被动语态（Passive Tense）。在语法上，主动语态是主语直接对动词发出动作，而被动语态则是动词反过来作用于主语。讲者犀利地吐槽道：“被动语态的诀窍，就是尽可能让你的主语离动词越远越好，好让谁也注意不到你刚刚写了这句废话——而大多数工程师正是这么写东西的。” 为了证明主动语态的威力，讲者当场拿出手机上的例句进行“代码重构式”的现场改写： | 原始被动语态（拖沓晦涩） | 重构为主动语态（干练清晰） | 表达效果与改写思路 | |:--- |:--- |:--- | | “晚餐时，六只虾被哈里吃掉了。”<br>*(At dinner six shrimp were eaten by Harry.)* | “哈里在晚餐时吃了六只虾。”<br>*(Harry ate six shrimp at dinner.)* | 原句中只要没读到句末的句号，读者就根本不知道是谁吃了虾；改写后主谓清晰，甚至可以查阅词典换上更强烈的动词，如“哈里在晚餐时**啃噬**（gnawed）了六只虾”。 | | “数据中心被美丽的固网工程师巡视着。”<br>*(The data center is roamed by beautiful Network Engineers.)* | “美丽的固网工程师巡查数据中心。”<br>*(Beautiful Network Engineers roam the data center.)* | 原句把人藏在最后，改写后明确执行动作的主体，画面感与责任主体瞬间确立。 | | “今晚监控系统将由我们进行看守。”<br>*(Monitoring is going to be watched by us tonight.)* | “我们今晚盯紧监控，因为系统大概率会崩溃。”<br>*(We're going to watch the monitor tonight because our systems will probably crash.)* | 迅速传递主观意图与关键上下文，信息密度大幅提升。 | 讲者强调：“短小精悍的句子是将观点传递给受众的真正秘诀。”短句更容易消化、阅读门槛更低，能让读者保持更长久的专注，甚至在 Twitter 上获得更多转发。 针对如何消灭被动语态和提高技术文档质量，讲者给出了具体且激进的避坑法则： - **排查标志性介词“by”**：如果在句子中频繁看到“watched by…”等句式，必须立刻拉响警报并重写结构。 - **消灭“to be”动词衍生词**：警惕类似“should have been doing”、“could be”、“must be”等多余的系动词叠加，直接用实义动作代替。 - **全面向副词宣战**：在最后 45 秒的时间里，讲者情绪激动地痛斥技术写作中的滥用副词行为，呼吁彻底删掉“practically（几乎）”、“only（仅仅）”、“very（非常）”以及“just（只是）”等词汇。 - **推荐实用写作工具**：推荐在线应用 Hemingway Editor（hemingwayapp.com），该工具能够自动标红技术文本中的副词、被动语态和冗长病句，倒逼工程师像海明威一样写出干脆有力的文字。 --- ### [28:24](https://www.youtube.com/watch?v=bg6-LVCHmGM&t=1704s) Jackalope：把 RabbitMQ 当成 HTTP 用的奇葩架构 在全场热烈的掌声与笑声中，下一位讲者 Josh Schmida 登台。他在自我介绍时打趣道：“我是一名 CTO，我们公司目前也在招人。”在正式切入正题前，他忍不住先向大会主办方与苹果公司开炮：作为一个技术大会，现场竟然极度缺乏排插，逼得他不得不站在走廊里给自己那台插头反人类的破笔记本充电，并嘲讽苹果作为一家顶尖设计公司应该“长点心”。 Josh 的演讲题目为“Jackalope”（鹿角兔，北美传说中的神秘生物），副标题则是“馊主意，或者仅仅只是个点子”。他提到自己此前听了 Braintree 公司关于其内部网关项目“Broxy”的分享，深受启发（尽管 Broxy 是基于 Redis 实现的），进而萌生了在自家系统里实现特殊面向服务架构（SOA）的想法。为了不让运维工程师发疯，他决定“换一种姿势折磨他们”——自己动手从零编写了一套名为 Jackalope 的全新应用服务器。 ```mermaid flowchart LR accTitle: Jackalope 伪 HTTP 消息队列架构流转 accDescr: 展示请求如何从 Web 入口经过 amqparty 转发到 RabbitMQ，再由 Jackalope 接管并交给 Ruby 应用处理 Client["外部网络请求<br>(Internet Traffic)"] --> Web["Web 接入层"] subgraph Frontend["前端处理"] Web --> RubyApp["Ruby 前端应用"] RubyApp --> AMQ["amqparty 客户端<br>(伪装成 HTTParty)"] end AMQ -->|"AMQP 消息"| Rabbit["RabbitMQ 消息队列"] subgraph Backend["后端服务 (Jackalope)"] Rabbit --> JackalopeServer["Jackalope 应用服务器<br>(替换 Unicorn)"] JackalopeServer -->|"模拟 HTTP 请求"| RubyService["后端 Ruby 业务逻辑"] end ``` 该架构的核心理念在于将底层的消息队列全流程伪装成常规的 HTTP 请求： - **上游发包层（amqparty）**：外部请求进入 Ruby Web 前端后，开发者并不直接使用传统的 HTTP 客户端（如 HTTParty），而是使用自制的封装库 `amqparty`。该库对外暴露完全一致的 HTTP 请求接口，但底层却将数据打包直接发送到 RabbitMQ 队列中。 - **下游承接层（Jackalope）**：在接收端，系统彻底废弃了业界常用的正统 Rack 服务器（如 Unicorn），取而代之的是由 Josh 编写的 Jackalope。Jackalope 直接从 RabbitMQ 中拉取消息，伪造出一套 HTTP 请求上下文并丢给下游的 Ruby 应用运行。 这套把“独角兽（Unicorn）”魔改成“鹿角兔（Jackalope）”的系统令人大跌眼镜，但 Josh 自豪地透露，这套系统目前已经实打实地运行在生产环境中的一个小应用上了，并且“并没有把 DevOps 运维人员完全逼疯——反正他们本来就已经有点神经兮兮了”。 ### [30:44](https://www.youtube.com/watch?v=bg6-LVCHmGM&t=1844s) 放弃 RabbitMQ 与基于 Kafka 的新架构构想 Josh 正在为自己上一段演讲收尾。他坦承团队可能不会继续推进先前的方案。前几年他在 RailsConf 上分享过这个思路，当时全场反应热烈，但现实中项目并没有真正落地。压垮原方案的致命伤在于系统选型——文档第一句话就明确写着 RabbitMQ 并不具备真正的横向扩展能力，且对网络分区（Network partitions）的容错极差。对此 Josh 忍不住吐槽道：“You had one job man.” 随后他抛出了一个新脑洞，希望能征求现场开发者的意见：客户端流量进入 API 服务器后，通过真正的 HTTP 与某个后端服务通信；该服务用正规语言编写（而不是 Josh 口中调侃的“地狱之门”），将数据写入 Kafka。消费者从 Kafka 读取消息并在后端模拟 HTTP，最后利用数据包重写/网络封包篡改（Packet mangling）将流量直接原路送回最初的客户端，避免响应流重新走一遍沉重的中间层链路。Josh 表示如果有人知道如何实现这一整套逻辑且不需要手写大量 Haskell，或者对此构想感兴趣，欢迎在会后找他交流。 ```mermaid flowchart LR accTitle: Kafka 异步管线与封包篡改回程架构 accDescr: 客户端请求经由 API 服务器进入 Kafka 队列，处理完成后通过 Packet Mangling 绕过中间层直连客户端。 subgraph Ingress["入口阶段"] A["客户端 Client"] -->|"API 流量"| B["API 服务器"] B -->|"HTTP"| C["写入服务 Ingestion"] end subgraph Pipeline["消息中间件"] C -->|"写入"| D[("Kafka")] D -->|"消费读取"| E["后端处理模拟 HTTP"] end subgraph Return["直接回程"] E -.->|"Packet Mangling 数据包重写"| A end ``` ### [32:08](https://www.youtube.com/watch?v=bg6-LVCHmGM&t=1928s) 内存分配器之苦与悲伤的五个阶段 讲者 John 登台分享了自己今年花费数月在用户态从零编写内存分配器（Memory allocator）的痛苦经历。该分配器的设计目标是实现极致紧凑、零内存碎片（Zero fragmentation），但也因此导致 Valgrind 等成熟的排错工具几乎失效——一旦某块内存越界溢出到相邻块，工具根本无法捕捉。整个折腾的源头只是一句轻描淡写的“我们自己写个新分配器来支持这个需求吧”，结果让他花了整整三个月在生产级高并发应用中艰难调试。John 感叹道，在没有 Valgrind 这种利器的情况下在生产环境调试内存分配器，简直完美契合存在主义对于“荒诞与虚无”的定义： > “Debugging a memory allocator in production when you can't use Valgrind on it and don't have any good tooling is basically to me like standing in the middle of Times Square in the middle of the day finding a pile of dog crap in the middle of a crosswalk and your only recourse being to stand there and wonder who has done this.” John 借用心理学上的“悲伤五阶段”来拆解这段研发心路： - **否认（Denial）**：盲目乐观，误以为写分配器很简单。事实证明“写一个复杂的内存分配器很容易，但调试它奇难无比，全因最初的傲慢”。 - **愤怒（Anger）**：遇到不可捉摸的内存破坏时彻底抓狂。 - **讨价还价（Bargaining）**：四处寻找救命稻草。祈求能在内存块周围加 Redzones（红区保护），或者希望用 AddressSanitizer（ASan）取代太慢的 Valgrind，但又受制于生产环境缺乏现代编译器的支持。 - **沮丧（Depression）**：陷入怀疑人生的重写泥潭。不仅要自己重写 ASan 的机制和 Redzones 埋点，甚至一度崩溃到想要“摆烂”——既然 64 位有海量虚拟地址空间，不如直接随机分发裸指针。他甚至真做过实验：申请 1TB 地址空间随机派发指针，前 10MB 跑得极顺畅，之后就会迎来灾难性的数据破坏。 - **接受（Acceptance）**：认识并接受自己曾经的傲慢，读完 ASan 论文并补齐监控钩子与 Redzone。最终得到的终极教训是：永远不要自己重造轮子，“Jemalloc is really great they've done a lot of work on it”，用成熟方案才是人间正道。 ### [35:59](https://www.youtube.com/watch?v=bg6-LVCHmGM&t=2159s) 机器码膨胀与内存带宽墙（Native Don't Scale） 主持人随后请出下一位演讲者，带来题为《Native Don't Scale》的闪电演讲。讲者供职于 AppNexus，日常工作专注于性能极限优化。他指出业界当下流行一种危险的工程偏见：认为将逻辑全部用 C++ 模板展开特化，或者将机器学习模型全盘编译为原生机器码（Machine Code），性能就一定会快如闪电。每当公司高管或 CEO 提出“把机器学习模型直接转成机器码跑”时，他都深感绝望。 讲者以生产环境使用的 Intel Xeon E5 芯片实物晶圆图进行深入剖析：现代 CPU 是一项逼近光速极限的工程奇迹，但芯片与外界通信的瓶颈极其狭窄——整个 CPU 只有区区 2000 个引脚，其中大量引脚直连 DIMM 内存插槽，每个内存条仅有约 240 个引脚。在其实际生产机器的带宽下，平均每个时钟周期、每个 DIMM 仅能吞吐约 5 个字节（5 bytes per cycle per DIMM）。 ``` +-------------------------------------------------------------+ | Intel Xeon E5 处理器 (受光速与物理尺寸限制) | | 依靠 ~2,000 个引脚连接外部 | +-------------------------------------------------------------+ | | v v +-----------------+ +-----------------+ | DIMM 插槽 1 | | DIMM 插槽 2 | | (~240 个引脚) | | (~240 个引脚) | +-----------------+ +-----------------+ | | +-----------+-----------+ | v 每时钟周期每个 DIMM 仅约 5 Bytes 带宽 ``` 以经典的向量乘加运算 DAXPY（$ax + y$）为例，计算过程中机器指令本身所占的字节数往往超过了其操作的数据字节数。如果开发者将庞大的机器学习模型直接编译成海量原生机器指令，程序运行时的大部分时间根本不是在等业务数据，而是在等内存将指令读入 CPU。 - **大基本块测试数据**：测试显示，即使是纯 NOP 指令构成的 1KB 大基本块，一旦无法命中缓存而穿透到主内存，理想状态下每时钟周期也仅能执行约 1 到 1.5 条指令（IPC = 1.0~1.5）。 - **指令长度惩罚**：NOP 指令仅占 1 个字节，而 x86 架构下的常规业务指令通常长达 3 到 7 个字节。因此代码一旦脱离 CPU 缓存，IPC 就会暴跌至 0.12 左右，比正常业务代码（通常 IPC 约 0.8~1.5）慢上整整五倍。 - **分支跳转与预取失效**：庞大的代码体积导致 CPU 频繁发生跨内存随机跳转。这比纯数据随机访问更具毁灭性——因为指令未就绪前，CPU 根本无法推断下一步要处理什么数据，硬件预取（Prefetching）和推测执行（Speculation）机制全盘哑火，直接把一台 2015 年的高性能服务器倒退回 90 年代的运行水平。 - **带宽断层**：CPU 的 L1/L2/L3 高速缓存具备极高的吞吐能力，但主内存带宽骤降至仅约十四分之一。讲者告诫在场工程师：“Yes if you want to scale please don't tell me you're just going to turn it all into machine code.” ### [40:11](https://www.youtube.com/watch?v=bg6-LVCHmGM&t=2411s) 瞒天过海的九个月：一名跨国工程师的“秘密生子”记 压轴登台的是 Andrew Wong，他的演讲暂无正式标题。Andrew 开场便幽默地自嘲道：“As Geir alluded to my professional career has been a series of mistakes.” 他现场让观众二选一：是听严肃的技术/运维排障血泪史（“基本上就是 Pager 报警”），还是听一则八卦温情的人文故事。在观众的起哄要求下，他选择了后者。 他自称向来不是一个有长远规划的人，但一年前他和妻子决定要一个孩子。出人意料的是，从确认怀孕起，Andrew 向公司整整隐瞒了九个月。当时他在纽约上班，而妻子远在丹麦哥本哈根。虽然隔着大西洋有 8 小时航程，但他打趣说反正孕期“大部分辛苦活都是妻子在承担”。为了展现支持态度并避免“严重的离婚危机”，他在临产前一个月飞抵哥本哈根陪产，同时向公司请假称离开一阵子很快就回。 巧合的是，女儿恰好在万圣节当天降生，而就在宝宝出生的前一晚，Andrew 还在通宵写补丁修复刚刚搞崩的生产环境。随后，他在刚刚荣升人父、彻夜未眠的状态下准时接入了团队的例行例会，准备当面向同事们揭开这个藏了大半年的天大秘密。 ### [42:28](https://www.youtube.com/watch?v=bg6-LVCHmGM&t=2548s) 育儿与 PagerDuty：无情的硅基机器与嗷嗷待哺的婴儿 上一位演讲者正在分享初为人父的经历与感悟。迎来了女儿的降生，他坦言这种体验棒极了，但他绝不会主动向没有孩子的人强行推销“生娃”，因为这必须是每个人自己深思熟虑后的决定。当团队里的其他组长得知他当了爸爸时，有的惊讶得差点从椅子上摔下来，有的意识到他或许比表面上看起来更为成熟稳重，当然也有人可能觉得他其实超级不负责任。有了孩子后，程序员常被问到的一个经典问题就是：“现在值守寻呼台（Pager）是不是变得更惨了？”毕竟生活中真正被在乎的事情只剩下那一件，而另一边则是 Nagios 发出的刺耳警报。演讲者笑着澄清，自己绝没有把女儿取名为“Nagios”。 实际上，多年的 Pager 轮值反倒为带娃提供了绝佳的预备演练。在他待过的技术团队里，报警往往是先零星来上一两次，随后归于平静，正当你以为自己逃过一劫、心想“肯定是有别的倒霉蛋先爬起来看手机了”的时候，真正的风暴才降临。养孩子的过程完全一模一样，尤其在太阳下山后更是一场鏖战：谁先醒来，谁就得顶上去当值。这对程序员来说尤为残酷，因为他们通常本来就还没睡。不过，他认为伺候孩子比应付故障警报好得多，因为至少系统全面崩溃时还有人陪着你，而且这个小生命见到你时是真心欢喜的。谈及服务器与婴儿的本质区别，演讲者形象地感叹道：“Unlike the machine which was an uncaring silicon god with no forgiveness none at all. The other one you can feed or sing to and it it gently stops yelling at you.”（机器就像是个毫无同情心、铁面无情且绝不饶恕人的硅基神灵；而另一个小家伙，你只要喂喂她或者给她唱首歌，她就会渐渐停止对你大吼大叫。） 演讲进入最后的 19 秒问答环节，台下的 Robert 举手提问。演讲者调侃 Robert 看起来不太友善、周围都没人敢坐，但还是接了他的提问。针对大家关于婴儿名字的猜测，演讲者透露孩子既不叫 Nagios、Circonus、OmniOS，也不叫 SmartOS 或 Linux，而是以希腊神话中掌管迷宫、看守弥诺陶洛斯的女神阿里阿德涅（Ariadne）命名的。 --- ### [45:51](https://www.youtube.com/watch?v=bg6-LVCHmGM&t=2751s) 罗伯特·特里特论约吉·贝拉：被遗忘的 DevOps 真正先驱 随着上一位演讲者下台，主持人欣喜地介绍自愿登台的罗伯特·特里特（Robert Treat）。特里特决定在本次闪电演讲中为一位在他看来在科技会议上从未获得应有认可的伟大人物正名——恰逢本周逝世的传奇棒球名宿约吉·贝拉（Yogi Berra）。特里特指出，多年来业界一直在争论到底是谁发明了 DevOps：有人归功于写出《凤凰项目》的吉恩·金（Gene Kim），以及一同撰写《Visible Ops》的凯文·贝尔（Kevin Behr）与乔治·斯帕福德（George Spafford）；有人认为是约翰·奥尔斯波（John Allspaw）；还有人追溯到丰田生产方式的大野耐一，或是提出约束理论的高德拉特（Goldratt）；更有甚者如约翰·威利斯（John Willis），坚称质量管理大师戴明（W. Edwards Deming）才是 DevOps 理念的开山鼻祖。特里特承认戴明的理论确实可以通过逆向工程与 DevOps 挂钩，但他坚信真正的灵魂人物是约吉·贝拉，因为贝拉的无数传世名言早已精准概括了现代 IT 运维的核心逻辑。 ```mermaid flowchart TD accTitle: 约吉·贝拉名言与现代 DevOps 理念的映射 accDescr: 展示棒球名宿约吉·贝拉的经典语录如何对应系统架构、敏捷研发、成本控制与监控运维。 subgraph Berra["约吉·贝拉哲学 (Yogi-isms)"] Y1["'未来不再是过去那个样子了'"] Y2["'再也没人去那儿了，因为人太挤了'"] Y3["'五分钱再也不值一毛钱了'"] Y4["'别犯太多错误的错误'"] Y5["'光靠观察你就能看出来很多东西'"] end subgraph DevOps["现代系统与 DevOps 实践"] D1["瀑布模型 vs. 敏捷开发（计划永远赶不上变化）"] D2["资源争抢与云迁移（AWS 拥挤 vs. 选择 Joyent）"] D3["硅谷通胀（开发者薪资、住房、BART、卷饼价格飞涨）"] D4["故障容错文化（Surge 会议提倡的良性探索失误）"] D5["系统监控、遥测度量与信息共享"] end Y1 --> D1 Y2 --> D2 Y3 --> D3 Y4 --> D4 Y5 --> D5 ``` 特里特逐一解析了贝拉语录在技术领域的绝妙投射： - **关于计划脱节**：贝拉曾说“未来不再是过去那个样子了（The future ain't what it used to be）”。这精准道破了瀑布流与敏捷开发的本质痛点，无论前期做多么详尽的方案和上线排期，最终的交付结果永远偏离原本设想。 - **关于云计算争夺**：贝拉的名言“再也没人去那儿了，它太拥挤了（Nobody goes there anymore; it's too crowded）”，恰如如今人满为患的 AWS。特里特调侃称，如果是自己今天创业，大概会转投 Joyent，因为那里根本没人用，你能霸占全部的网络带宽和 I/O 资源，完美避开资源竞争的陷阱。 - **关于硅谷泡沫**：贝拉感慨“五分钱再也不值一毛钱了（A nickel ain't worth a dime anymore）”，正好击中了硅谷脱离物理法则的荒诞现实——无论是工程师的薪水、天价房租，还是乘坐旧金山湾区捷运（BART）去买一份卷饼的开销，全都贵得匪夷所思。 - **关于失败文化**：贝拉讲过“我们犯了太多错误的错误”。特里特总告诫自己的团队要犯“不会造成致命伤的错误”，而在 Surge 大会上大家听到的正是这类有价值的经验教训。 - **关于可观测性**：现代 DevOps 极力强调的度量与监控，其核心早已被贝拉一语点破。特里特振振有词地表示：“Damn right, Yogi Berra. You can observe a lot by watching.”（说得太对了，约吉·贝拉早就说过：光靠观察你就能看出来很多东西。） 特里特呼吁，今晚当参会者们走在国家港湾（National Harbor）街头小酌时，别忘了为这位刚刚离世的先驱举杯致敬。 --- ### [50:50](https://www.youtube.com/watch?v=bg6-LVCHmGM&t=3050s) 致敬西奥：AWS 宕机报告的即兴现代舞实验 罗伯特下台后，主持人盖尔·马格努森（Geir Magnusson）接管了舞台，向大家解释接下来这场特别演出的缘由。OmniTI 的创始人兼 Surge 大会发起人西奥·施洛斯纳格尔（Theo Schlossnagle）今年正在环游世界，这是他有史以来第一次缺席 Surge 大会，西奥本人在推特上对此感到非常难过。每年 Surge 大会期间往往会遭遇亚马逊 AWS 的秋季大宕机，导致现场参会者的参会体验大受影响；而今年 AWS 的故障恰好发生在大会开幕前的一个周末。盖尔开玩笑称，这大概是杰夫·贝佐斯（Jeff Bezos）听取了主办方的请求，特意把事故提前，好让工程师们能空出时间在国家港湾痛快畅饮。为了安抚远方的西奥，他们决定献上一场别开生面的行为艺术——将周末 AWS 的官方事故复盘报告（Postmortem）改编为即兴现代舞（Interpretive Dance）。 盖尔重申自己绝不上台跳舞，他只负责念台词，随后便向台下大声招募志愿者：“seriously come on let's go some people want to do this. It won't go to your bosses or employers.”（严肃点，快上台来！总有人想跳的吧！这段录像绝不会寄给你们的老板或雇主的。） 几位参会者纷纷走上台前，盖尔打趣道原本安排了几个托儿，没想到真有纯粹出于热情的观众主动应征。演出内容基于亚马逊公关与市场部门对外发布的故障报告精简版。尽管整场演出毫无排练，但盖尔坚信艺术的真谛在于自发与偶发性。台上一名参会者主动透露自己在大学里跳过三年芭蕾舞，盖尔如获至宝，立刻安排他挑大梁饰演主角——在本次事故中崩溃的核心组件“元数据服务”（Metadata Service）。在全场善意的哄笑与掌声中，盖尔翻开材料，开始宣读这起影响波及无数团队的“DynamoDB 事件”，将时间倒回至周日凌晨 2 点 19 分。 ### [54:21](https://www.youtube.com/watch?v=bg6-LVCHmGM&t=3261s) 现场情景表演：拆解 DynamoDB 元数据雪崩事故 在闪电演讲的舞台上，演讲者把 2015 年著名的 Amazon DynamoDB 大规模宕机复盘变成了一场与现场观众互动的“情景剧”。他当场招募了四名观众上台扮演“存储服务器”（Storage Servers），生动重现了太平洋夏令时间（PDT）凌晨发生的故障链条：起初是一次短暂的网络中断波及了部分 DynamoDB 存储服务器。紧接着，元数据服务（Metadata Service）的响应延迟超出存储服务器所允许的获取与传输时限。无法按时拿到成员资格数据（Membership Data）的存储服务器只能主动退出集群，停止接收用户请求。为了维持容量，台上的“存储服务器”观众在演讲者的调度下一会儿退出、一会儿又被喊回。 随后，由于大量节点同时发起对庞大成员资格数据的并发请求，元数据服务的处理速度被急剧拖垮，最终彻底超时。故障服务器无法完成成员资格续约，全面陷入不可用状态，并开始极其激进地疯狂重试请求。这股重试风暴引发了滚雪球效应：处于高负载重压下的元数据服务瘫痪，连带波及了原本根本未受最初网络中断影响的健康存储服务器。直到清晨 5:06，工程师在经历数次扩容失败后，被迫采取激进手段——直接暂停发往元数据服务的所有请求，强制掐断重试风暴以卸掉负载。当元数据服务终于能够处理管理指令时，团队才得以迅速注入巨额容量。演讲者在此刻大喊让台下更多观众“冲上舞台”（storm the stage）充当扩容资源，并借机调侃道：“我想毕竟是亚马逊，所以启动巨额容量总是很慢的。”最终，DynamoDB 在早上 7:10 恢复，各依赖该服务的 AWS 组件也逐步恢复正常。 ```mermaid flowchart TD accTitle: DynamoDB 故障级联放大过程 accDescr: 展示从最初网络中断到重试风暴最终拖垮元数据服务的连锁反应流程 subgraph Phase1["阶段一：网络抖动与超时"] A["短暂网络中断"] --> B["元数据服务响应超时"] B --> C["部分存储服务器丢失成员资格数据"] C --> D["服务器将自己摘除，停止承接流量"] end subgraph Phase2["阶段二：重试雪崩与大面积瘫痪"] D --> E["大量节点并发拉取庞大成员数据"] E --> F["元数据服务负载过高，响应彻底延宕"] F --> G["重试风暴击垮原本健康的存储节点"] end subgraph Phase3["阶段三：人工阻断与恢复"] G --> H["5:06 AM：暂停元数据服务请求以强行泄洪"] H --> I["恢复管理通道，注入海量计算容量"] I --> J["7:10 AM：错误率降至低位，服务全面复原"] end ``` - **故障关键节点**：清晨 5:06 决定强行暂停发往元数据服务的请求以削减重试洪峰；7:10 服务完全恢复，错误率降至绝大多数客户能够接受的水平。 - **演讲幽默讽刺**：“我想他们的标准确实挺低的，依赖 DynamoDB 的 AWS 基础服务终于恢复了正常运转。明年我们该讲讲 Elastic Beanstalk 了。” --- ### [58:23](https://www.youtube.com/watch?v=bg6-LVCHmGM&t=3503s) Brian Cantrill 登台：在 SmartOS 上实现 Linux 的“反常识罪行” 随着前一场表演结束，主持人 Geir Magnusson 宣布引出当天的压轴嘉宾——来自 Joyent 的 Brian Cantrill。Geir 调侃说，这大概是他第一次看到 Brian 在闪电演讲前“居然提前知道自己要讲什么”。Brian 登台后一边调试自动播放的幻灯片，一边戏谑称 Geir 没向大家说明闪电演讲其实存在一个“Cantrill 例外条款”，但他保证绝不会长篇大论让人厌烦。他将本次演讲的主题定名为一次“反常识的罪行”（a crime against common sense）。 这一罪行的源头，来自于 Joyent 在 SmartOS 内核中构建的 LX 品牌区域（LX Branded Zones）。为了让未经修改的 Linux 二进制程序直接跑在 SmartOS（基于 illumos 内核）的容器内，工程团队直接在内核层实现了一套完整的 Linux 系统调用表。这项成果听起来令人赞叹，但在工程实现上却堪称“人间炼狱”。Brian 强调，在做系统级兼容时，系统工程师没有资格和上层应用讲道理，只能咬牙去复现 Linux 系统所有的底层畸变。 > “在实现 LX 品牌区域的过程中，我们每个人都挣下了该讲一万年闪电演讲的罪过，因为我们把 Linux 的所有恐怖之处、所有的语义暴行全都给实现了，因为我们没法去跟应用程序争辩，我们唯一能做的就是让它老老实实跑起来。” —— Brian Cantrill --- ### [01:00:15](https://www.youtube.com/watch?v=bg6-LVCHmGM&t=3615s) 捉鬼之旅：Plex 崩溃、无符号表之怒与丑陋的 jemalloc 汇编 Brian 选取的具体案例是多媒体服务程序 Plex。作为一个闭源的 Linux 原生程序，Plex 在 LX 区域内运行时经常发生极其诡异的崩溃。在 2015 年 1 月，Joyent 工程师 Patrick Mooney 开始深入调查 Plex 产生的内核转储文件（core dump）。当底层系统调用兼容稍有差池时，用户空间程序往往会以一种匪夷所思的方式自毁。为了排查，团队甚至从软件厂商那里拿到了一个包含调试信息的二进制文件。 在内部聊天记录中，Patrick Mooney 贴出了一段让人啼笑皆非的汇编反编译片段，并发问自己是不是疯了，这段代码实现这么简单的逻辑为什么会冗杂到这种地步。同事 Robert Mustacchi 随口追问这总不至于真的是纯手写汇编，Patrick 回复称这虽然是 Plex 的崩溃现场，但至少该二进制文件里奇迹般地保留了符号表（symbols）和帧指针（frame pointers），堪称“香格里拉世外桃源”。然而，正是这句话刺激到了饱受 Linux 生态缺失帧指针折磨的 Robert，他在群聊里给自己记了一次“自省扣分”（self demerit），因为他被这个残酷的现状彻底激怒并“钓鱼”了。 Brian 随后在现场大屏幕上投射出了这段让所有人瞠目结舌的汇编代码。代码来源于 jemalloc 内存分配器（显然是在极其严苛的未优化调试选项下编译出来的生成物）： ```text movq -0xd80(%rbp), %rax movq %rdx, 0x10(%rax) movq -0xd80(%rbp), %rax movl 0x10(%rax), %eax movl %eax, %edx ``` Brian 带领全场逐行剖析了这段灾难级的机器码：指令首先将栈基址偏置 `-0xd80` 处的内存地址加载到寄存器 `%rax`，接着把 `%rdx` 的值写入 `%rax + 0x10`。荒唐的是，第三条指令紧接着又把完全相同的内存地址再次加载进 `%rax`。Brian 现场模仿微处理器的内心独白：“微处理器那时候真该停下来问一句：‘你认真的？你刚才不是刚读过这个地址吗？你觉得它变了吗？没有吧！我直接把刚才的值给你行不行？’”更离谱的是，随后程序又把刚写入内存的值读回到寄存器 `%eax`，再挪进 `%edx`。尽管这段糟糕透顶的编译器产物给排错带来了极大的噪声，且未能立即解开崩溃之谜，但 Patrick 在极度挫败中顺手修复了 SmartOS core dump 遗漏记录 `%fs` 段寄存器基址（FS base）的缺陷，至少为后续的调试工作补全了一块关键拼图。 ### [01:04:50](https://www.youtube.com/watch?v=bg6-LVCHmGM&t=3890s) 调查陷入僵局与来自社区的突破口 演讲者回顾了当时的团队聊天记录。他笑称自己非常喜欢聊天记录，因为群里总是同时并行着好几个完全不同的话题。比如在 jemalloc 崩溃得一塌糊涂、大家焦头烂额的时候，同事 Robert 突然没头没脑地冒出一句：“总有一天我们会把 QEMU 的 ioctl 替换掉，到时候一定会无比辉煌。” 这句话就像从天花板上掉下来的幸运饼干纸条一样无厘头，让人哭笑不得，但细想一下确实很有道理。与此同时，Patrick 提出了一个极其漂亮的崩溃原因理论，遗憾的是实际数据完全不支持该理论；演讲者调侃道：“理论很完美，可惜被数据推翻了，我最讨厌这种情况。”至此，排查工作仍未取得任何进展。 转机来自于开源社区的一份报告。演讲者由衷感慨开源社区的力量——不同的人在做不同的事情，因此总能带来意想不到的发现。一位使用 Plex 的社区用户反馈称，jemalloc 自身的单元测试无法跑通、会直接报错退出。此前 Splunk 和 Plex 等程序频繁出现随机的 Core Dump，而 jemalloc 总是被静态或动态链接在这些程序中（“总在案发现场”）。如果能够利用 jemalloc 自身稳定失败的测试用例来复现问题，排查工作就有了明确的抓手。 ```mermaid flowchart TD accTitle: jemalloc 故障排查流程 accDescr: 展示从排查 Core Dump 到锁定 jemalloc 单元测试及系统调用的过程 subgraph Problem["故障现象"] A["Splunk 与 Plex 频繁崩溃"] --> B["发现 jemalloc 总在案发现场"] end subgraph Community["社区线索"] C["社区报告: Plex 用户反馈"] --> D["jemalloc 单元测试直接失败"] end subgraph Investigation["分析排查"] B & D --> E["运行并隔离可复现的单元测试"] E --> F["跟踪测试执行的系统调用"] F --> G["发现异常调用: madvise(MADV_DONTNEED)"] end ``` - 演讲者提到，每当在 LX 容器环境下程序发生随机崩溃时，团队总是第一反应怀疑自己的 futex（快速用户空间互斥锁）实现有问题。 - 演讲者澄清道，虽然 futex 本身的设计极其糟糕（“其设计缺陷足以讲上一万场闪电演讲”），但事后证明他们的 futex 实现是完全正确的，与本次故障毫无关联。 - 团队兵分两路：一部分人去跑 futex 压力测试排除嫌疑，另一部分人全力分析失败的 jemalloc 单元测试。 ### [01:06:57](https://www.youtube.com/watch?v=bg6-LVCHmGM&t=4017s) POSIX 标准中的 `madvise`：“建议”而非“命令” 演讲者跟进这个极其稳定复现的 jemalloc 测试后，开始观察其发起的系统调用序列。测试逻辑本身非常简单，但其中调用了带有 `MADV_DONTNEED` 标志的 `madvise`。演讲者带领现场观众重温计算机系统的基础语义：`madvise` 是一项历史悠久的 POSIX 标准接口。顾名思义，“Advice”（建议）是动词“advise”的名词形式，而建议的核心特征在于——它是可以被接收方完全忽略的。 演讲者强调，“建议”与“命令”之间存在本质鸿沟。在系统调用设计中，`madvise(addr, length, advice)` 允许应用程序将自己对未来的预期告知操作系统内核。内核本身很难预测应用程序接下来要做什么，如果应用程序能主动提供线索，内核便能更好地进行性能优化。然而，POSIX 规范对 `MADV_DONTNEED` 的定义非常简略，仅说明应用程序预期在近期内不会访问指定的内存范围。更关键的是 POSIX 规范底部的明确限定：该系统调用绝不能改变指定内存范围的访问语义，它至多只能影响访问该内存时的性能表现。换言之，`madvise` 只是告知内核“我暂时不需要它”，但该虚拟内存映射依然有效，其底层数据理应保持原样。 > “The POSIX madvise function shall have no effect on the semantics of access to memory in the specified range, although it may affect the performance of access.” —— 演讲者引用 POSIX 规范原文指出。 ### [01:09:13](https://www.youtube.com/watch?v=bg6-LVCHmGM&t=4153s) Linux 的语义违背与 jemalloc 的崩溃根源 当演讲者翻看 Linux 的 `madvise(2)` 手册页时，发现了极其离谱的语义颠覆。Linux 手册页开篇写着类似“近期不预期访问该区域”，随后写道：“暂时（For the time being），应用程序已经用完了该范围，因此内核可以释放与其关联的资源。后续对该范围内页面的访问将成功，但会导致从底层映射文件重新加载内容，或者对于匿名映射，按需提供填零页面（zero-filled on demand pages）。” 演讲者犀利吐槽手册页语法混乱不堪，更抨击其实质语义根本不是“建议”，而是强制性的“命令”——如果用户用完了内存且后续需要填零，理应直接调用 `munmap`，而不是让 `madvise` 去破坏内存原有的内容。 ```mermaid flowchart LR accTitle: POSIX 与 Linux 下 madvise 语义对比 accDescr: 对比 POSIX 标准与 Linux 内核对 MADV_DONTNEED 调用的不同行为 subgraph Input["调用 madvise(..., MADV_DONTNEED)"] M["已写入数据 123 的匿名私有映射"] end subgraph POSIX["POSIX 标准语义 (建议)"] P1["将页面标记为可回收候选"] --> P2["若未被回收，读回原值 123"] P1 --> P3["语义不变: 绝不保证重新填零"] end subgraph Linux["Linux 语义 (强制清零)"] L1["强制释放底层物理页面"] --> L2["重新访问时触发缺页并填零"] L2 --> L3["读取结果强制变为 0 (jemalloc 依赖此行为)"] end Input --> POSIX Input --> Linux ``` 为了证明 Linux 这一行为的荒谬，演讲者给出了一个极简示例：先通过 `mmap` 分配私有匿名内存，写入数值 `123`，接着调用 `madvise(..., MADV_DONTNEED)`，随后再次读取该地址，在 Linux 下读出的数值居然变成了 `0`。正常的 POSIX 语义下，如果操作系统面临内存紧缺，内核可以收回该页面；但应用程序绝无理由假定读回来的必然是全零数据——如果内存没被回收，读取到的应该是之前写入的原值。然而，jemalloc 恰恰依赖了 Linux 这种违背标准规范的“强制清零”副作用。正统的 `MADV_DONTNEED` 本该只是极其轻量地将页面挂到可回收链表上，而 Linux 却迫使内核大费周章地走完内存释放与重新填零流程，属于完全本末倒置的设计。 - 演讲者怒斥这种将“建议”擅自改为破坏性操作的行为是“不可理喻的语义灾难（unconscionable semantics）”。 - 演讲者在感到震惊之余去 Google 搜索“madvise MADV_DONTNEED Linux”，试图寻找其他对这种语义破坏感到愤怒的开发者。 - 搜索结果排在第一位的是 Stack Overflow 上的一个提问：“madvise MADV_DONTNEED 是否会指示操作系统延迟写入磁盘？” - 演讲者借此在台上展开了辛辣的吐槽：“提问者从小被告知‘世界上没有愚蠢的问题’，但这只是聪明人编造的谎言，世界上确实存在愚蠢的问题，而这就是其中之一。”他开玩笑称提问者完全有权利享受幸福人生并抚养孩子，但绝对不应该调用 `MADV_DONTNEED`，因为该提问对内存、数据和操作系统的理解与实际原理南辕北辙。 ### [01:13:25](https://www.youtube.com/watch?v=bg6-LVCHmGM&t=4405s) Stack Overflow 与 2005 年的内核邮件：常识的倒退 在追查 Plex 崩溃原因的过程中，演讲者迫切想要得到答案，甚至寄希望于 Stack Overflow 上能出现一锤定音的回复。然而现实却令人啼笑皆非：Stack Overflow 上充斥着那种让人绝望的警告——“千万不要碰 `MADV_DONTNEED`，为了你的身心健康，离内存映射远一点，甚至离操作系统和计算机都远一点，也许去爱达荷州找个农场隐居才是最好的选择”。原帖作者（一位名叫 Damon 的开发者）在回答中带着压抑不住的怒火解释道：在 Linux 内核中，`MADV_DONTNEED` 根本不会被视作“将脏页写回磁盘后再丢弃”的温和优化提示；相反，内核会不由分说地立即丢弃这些页面。更荒谬的是，内核社区并不认为这是一种缺陷，而是将其定义为“刻意为之的设计决策”（deliberate decision）。如果应用程序贸然调用它，就会直接面临数据丢失的风险。 演讲者进一步去翻找 Linux 内核邮件列表的历史记录，想看看当年到底是谁把这套逻辑搞坏的、内核维护者又是否意识到这种语义的破坏性。果不其然，早在 2005 年就有一封标题为《错误的 madvise MADV_DONTNEED 语义》的邮件引起了广泛共鸣。发件人在邮件里直言不讳：Linux 的 man 手册和源代码都宣称 `MADV_DONTNEED` 意味着“应用程序不再关心这些数据，因此可以被丢弃”，但 POSIX 标准规范根本不是这个意思。发帖者几乎是在屏幕前咆哮指出，其他所有成熟的操作系统都遵循标准；比如 Solaris 拥有专门的 `MADV_FREE`，能精准地实现“应用程序处理完毕、可以安全释放或作废内存”的需求。 当时讨论中有人（Robert Love 等）附和提议，应该将 Linux 当前粗暴丢弃内存的行为重命名为 `MADV_FREE`，同时为 `MADV_DONTNEED` 恢复符合 POSIX 规范的真正语义，并同步更正 man 手册。然而这场看似达成共识的讨论，在整整十年之后的 2015 年依然没有取得实质性进展。另一位名叫 Jörn 的开发者在当时的回复中一针见血地抨击道：“在我看来，这依然是一场违背常识的罪行（in my opinion this is still a crime against common sense）。” - Damon 在 Stack Overflow 上痛陈：Linux 下的 `MADV_DONTNEED` 会立即丢弃页面而不是等待写回，这一破坏性行为被官方视作故意设计的特性。 - POSIX 语义中 `madvise` 仅作为性能优化的提示（hint），但 Linux 的实现却演变成了具有破坏性的内存解除映射。 - Solaris 很早以前就提供了专门的 `MADV_FREE` 调用来释放不再使用的页面，语义清晰且安全。 --- ### [01:16:18](https://www.youtube.com/watch?v=bg6-LVCHmGM&t=4578s) 历史破案：Tru64 的幽灵与 SAP 的移植需求 究竟是谁把这个破坏性语义带进 Linux 内核的？演讲者带着听众抽丝剥茧，发现除了 Linux，其他现代操作系统几乎都不这么做——除了一个早已被遗忘的名字：DEC 的 **Tru64 UNIX**。 时间倒流回 1999 年 8 月 20 日。当时 SAP 正在进行各大软件厂商都在做的事情：把自家的企业级 Unix 应用移植到 Linux 平台上。在移植过程中，SAP 工程师发现 Linux 缺少一个类似 `MADV_DONTNEED` 的机制。工程师 Christoph 在向内核邮件列表提需求时，无意中引用了 Tru64 的实现方式，提到他们需要类似 IBM AIX 的 `disclaim` 或者是 Tru64 的 `madvise`。面对 AIX 与 Tru64 的语义分歧，演讲者调侃道这原本是一场“语义决斗”，而不幸的是，Linux 当年偏偏借鉴了最糟糕的那一个。 邮件记录显示，当时就有人看出了异样，发帖指出：虽然业界不少系统都有 `madvise`，但大部分系统仅将其用作页面换出的性能提示；Tru64 把本应用于性能加速的优化调用，改成了直接解除映射并丢弃数据的强制手段，显然是对接口的严重误读与滥用。然而，Linux 内核开发团队最终还是照搬了 Tru64 的错误理解并合并进了内核代码。正如演讲者感叹的那样：“因为某个可怜的 Tru64 工程师误读了调用规范，而这场火灾竟然越过隔离带蔓延进了 Linux，最终导致十多年后 Plex 在我们的 LX 兼容区域里崩溃。” ```mermaid flowchart TD accTitle: Tru64 语义污染 Linux 的历史链条 accDescr: 展示从 Tru64 误读 POSIX 规范，到 SAP 移植应用引入 Linux，最终导致现代系统崩溃的因果关系。 subgraph Origin["1990年代源头"] A["POSIX 规范定义 madvise<br/>（仅作为换页优化的 Hint）"] B["Tru64 工程师误读规范<br/>（实现为强制解除映射并丢弃数据）"] A -->|"误解实现"| B end subgraph LinuxAdoption["1999年 Linux 引入"] C["SAP 移植 Unix 应用至 Linux<br/>（需要类似内存丢弃特性）"] D["Linux 内核合并补丁<br/>（错误采纳 Tru64 破坏性语义）"] B -.->|"参考该行为"| C C -->|"邮件列表提议"| D end subgraph ModernImpact["现代连锁反应"] E["Linux 维持该非标准行为数十年"] F["Plex 在 SmartOS LX 容器中运行<br/>（依赖行为不兼容触发 Crash）"] D --> E E --> F end ``` - 1999 年 8 月 20 日，SAP 工程师在 Linux 内核邮件列表中寻求实现类似 AIX `disclaim` 或 Tru64 `madvise` 的功能。 - Tru64 工程师曲解了 `madvise` 的设计初衷，将原本纯粹用于速度优化的接口改造成了改变映射语义的破坏性操作。 - Linux 采纳了 Tru64 的实现，使得这一历史遗留的错误语义在主流开源内核中扎下根来，成为十余年后 Plex 在 LX branded zones（SmartOS 的 Linux 兼容层）上崩溃的真正病灶。 --- ### [01:18:34](https://www.youtube.com/watch?v=bg6-LVCHmGM&t=4714s) 拖延十年的 MADV_FREE 与 man 手册的“掩耳盗铃”修复 既然问题早已查明，Linux 能否像其他系统一样引入正规的 `MADV_FREE` 呢？历史的演进依然充满戏剧性。直到 2014 年——距离 1999 年过去整整 15 年之后，才终于有人正式提交补丁，建议 Linux 支持 `MADV_FREE`。开发者们结合 jemalloc 内存分配器进行了基准测试，结果显示引入真正的 `MADV_FREE` 后，jemalloc 的运行速度获得了大幅提升。演讲者对此辛辣地吐槽：“如果你们一开始就把语义实现正确，早就不会遇到这些性能烂摊子了。” 然而更为荒诞的是代码集成的进度。补丁早在 2014 年初就已提出，但直到 2015 年大会演讲时，它依然没有被合并进主流发行版中。“大家都吹嘘 Linux 迭代有多快，结果一年半过去了还没合入。”与此同时，官方对 man 手册的修复方式更是堪称教科书级别的“掩耳盗铃”。在原版 Linux man 手册中，曾有一句描述写道：“内核可以自由忽略该建议（the kernel may ignore advice）”，因为按定义 `madvise` 只是建议。然而在面对实际实现具有破坏性强制作废这一矛盾时，维护者的解决办法既不是修复内核逻辑，也不是承认违背规范，而是直接把那句话删掉：“决定将手册里的这句话认定为 bug，只要把它删掉就算修好了！” 演讲者用极具讽刺意味的总结结束了这场闪电演讲：“就这样，问题解决了！现在所有运行 Linux 的人，你们的 madvise 终于被‘修好’了！”全场在爆笑与掌声中落下帷幕。 - 2014 年针对 Linux 的 `MADV_FREE` 补丁在配合 jemalloc 测试时展现出了显著的性能优势。 - 截至 2015 年底演讲进行时，`MADV_FREE` 补丁提出已过去一年半，仍未被主流 Linux 内核整合。 - Linux man 手册所谓的修复方案，仅仅是删除了“内核可以忽略此建议（the kernel is free to ignore the advice）”这一描述，通过修改文档定义来掩盖实现与 POSIX 规范的违背。 ## Notable quotes > [09:36](https://www.youtube.com/watch?v=bg6-LVCHmGM&t=576s) The only way you can remember about wraparound autovacuum is to have actually experienced an outage as a result of it as far as I can tell. — David Pacheco > [35:59](https://www.youtube.com/watch?v=bg6-LVCHmGM&t=2159s) Yes if you want to scale please don't tell me you're just going to turn it all into machine code. — AppNexus 工程师 > [11:39](https://www.youtube.com/watch?v=bg6-LVCHmGM&t=699s) This is one of those cases where people are like, ‘I'm building an application, why do I need to trace the kernel scheduler?’ Well, because your thing is sleeping all the time and you want to know why. — David Pacheco ## Mentioned Geir Magnusson, Dan McDonald, Abel Matthew, David Pacheco, Dimitri Karvonis, Josh Schmida, John, Andrew Wong, Robert Treat, Brian Cantrill, Theo Schlossnagle, Patrick Mooney, Robert Mustacchi, Ran Atkinson, Dino Farinacci, Van Jacobson, Scott Aaronson, Eduardo Miranda, Yogi Berra, Gene Kim, Kevin Behr, George Spafford, John Allspaw, 大野耐一, Eliyahu M. Goldratt, John Willis, W. Edwards Deming, Jeff Bezos, Damon, Robert Love, Jörn, OmniTI, AppNexus, Joyent, Backtrace, Braintree, Amazon Web Services (AWS), Verizon, PARC, DEC, SAP, Manta, PostgreSQL, SmartOS, OmniOS, ZFS, ILNP (Identifier-Locator Network Protocol), LISP (Locator/ID Separation Protocol), NDN (Named Data Networking), GCC, Clang, libunwind, DWARF, GDB, RabbitMQ, Kafka, Unicorn, Jackalope, Broxy, amqparty, HTTParty, Valgrind, AddressSanitizer (ASan), jemalloc, Intel Xeon E5, Nagios, PagerDuty, Circonus, Amazon DynamoDB, Plex, Splunk, QEMU, Tru64 UNIX, IBM AIX, Solaris, Hemingway Editor, PhyChip, Ruby on Slime ## Video description The Surge 2015 Lightning Talks, sponsored by AppNexus, hosted by Geir Magnusson. Now with more interpretive dance. [Source](https://www.youtube.com/watch?v=bg6-LVCHmGM)

[9:36] David Pacheco: The only way you can remember about wraparound autovacuum is to have actually experienced an outage as a result of it as far as I can tell. [35:59] AppNexus 工程师: Yes if you want to scale please don't tell me you're just going to turn it all into machine code. [11:39] David Pacheco: This is one of those cases where people are like, ‘I'm building an application, why do I need to trace the kernel scheduler?’ Well, because your thing is sleeping all the time and you want to know why.

[23:03] The number one problem that really is facing engineering writing is that you are terrified of your subject and you are terrified of your verb.

[0:12] these are the lightning talks for surge [0:14] of [0:15] 2015 it's um an event that has had a [0:18] long history here of of being I think a [0:21] lot of fun um we don't do the standard [0:24] format of 2 minutes 2 minutes 2 minutes [0:27] we've opened it up to 5 minutes thank [0:29] you Brian for correct me again and um [0:32] they can be funny they can be serious [0:34] they can be just about anything um and [0:37] sometimes they're very surprising um I [0:39] want to thank Robert and the rest of the [0:42] uh Team for for allowing me to do this [0:44] uh and putting together such a great con [0:46] uh conference I want to thank the [0:48] sponsor app Nexus for making this [0:50] possible with the uh support they gave [0:53] and um without any further Ado we will [0:56] start with our first talk it's a [0:58] gentleman named Dan McDonald and he's [1:00] speaking about two future internet [1:03] directions um my name is Dan McDonald [1:06] I'm with omniti um I'm the OS I is my [1:10] day job and I'd like to talk about two [1:13] upcoming directions that the Academia [1:16] and research sectors of the internet [1:18] Community are working on um most of you [1:21] know what the internet architecture [1:22] looks like it's the protocol hourglass [1:25] where you have your apps on top your [1:27] transport protocols and your are your [1:30] presentation protocols your transport [1:32] protocols and then you have IP in the [1:34] middle of The Hourglass and then it [1:35] widens back out again over various links [1:38] over various physical media until you [1:40] get to the bottom again and modulo the [1:42] larger addresses of current deployment [1:44] of IPv6 things really haven't changed [1:46] all that much since [1:49] 1981 um two future directions that are [1:51] being worked on by um groups in the irtf [1:55] and in Academia in general they are [1:57] orthogonal to each other but they could [2:00] end up affecting people in Ops at some [2:03] point I think the first one will be [2:05] medium-term and the second one will be [2:07] longterm um the first is the separation [2:10] of location and identity in IP packets [2:13] um and basically it's an improvement on [2:15] the protocol hourglass where IP gets a [2:18] little better and then named data [2:19] networking which they are going to [2:21] reshape The Hourglass and put a [2:24] different thing in the middle narrow [2:26] waist IP addresses they basically serve [2:29] two purposes it's a locator as it [2:32] transits the backbone it tells you how [2:33] you get there so that's a not so Random [2:36] IP address that pretty much if you send [2:38] one from here will'll Transit the [2:39] backbone exit at Verizon in New York [2:41] City and eventually hit Northern [2:44] Massachusetts um it's also kb.com it's [2:48] my uh public web server and one of one [2:50] of six Z five zones on my home Home [2:53] Server um why not separate the identity [2:57] and the locator out from each other [3:00] nodes become uniquely identified [3:01] independent of location and if you can [3:03] imagine that not that AWS would ever go [3:06] down but imagine if AWS go AWS West goes [3:10] down and you can reinstantiate your very [3:13] node with the same identifier just with [3:15] a different locator in AWS East and if [3:18] you're really good you could keep those [3:21] two nodes redundant State up to date [3:24] with each other it is even theoretically [3:26] possible depending on latency and how [3:29] fast you can update things you could [3:31] even especially for longer live longer [3:33] live TCP connections even keep the [3:36] connection State up and running um [3:38] alternatively with separating out [3:40] location and identity you can have [3:41] mobile nodes with real Mobility at the [3:43] IP layer something that's eluded real [3:46] world for a long time um and there are [3:49] probably more things you can do that I [3:51] haven't thought about yet um there are [3:53] two proposals out there the identifier [3:56] locator Network protocol ilmp I'm a [3:58] little biased toward this because ran [4:00] ainson is an old friend and he hired me [4:03] out of grad school so um that's RFC 6740 [4:07] it's an experimental RFC and it [4:09] basically splits the IPv6 address into [4:11] identity and location components um the [4:15] locator ID separation protocol which has [4:17] the uh colliding acronym of lisp um is [4:20] done by Dina Farinacci and a bunch of [4:22] other people from or associated with [4:24] Cisco uh and it basically will do an IP [4:28] packet inside an IP packet so for ipv4 [4:31] you'll have an ipv4 space that's [4:32] identifying and an ipv4 space that's [4:34] locating and those will both go along in [4:37] a single packet looks like a tunneled [4:39] one perhaps um the other approach to [4:43] internet Futures is the name data [4:44] networking where you're going to [4:46] refactor The Hourglass all the apps are [4:48] still on top you have security and then [4:51] you have what are called content chunks [4:53] followed by a strategy for disseminating [4:55] those content chunks traditional [4:58] Protocols are near the bottom and then [5:00] you hit physical [5:01] again um there's a big research group [5:04] associated with this you can visit their [5:05] page at named data.net Van Jacobson [5:08] started this as a project at Park and [5:10] he's expanded it to include more of [5:11] Academia you're universalizing content [5:14] Chucks and chunks and all middle nodes [5:16] become stor and forward nodes it looks a [5:18] little more like uset or email than it [5:20] does traditional IP datagram switching [5:22] um data objects are assembled from [5:24] chunks and they are signed so it will [5:27] make all objects end to end secure [5:30] because when you receive a full object [5:31] you can check it against a signature yes [5:33] there's a lot of infrastructure that's [5:34] going to be involved with this but [5:36] that's why this is a longer term future [5:38] and something still in [5:39] Academia and personally I'm imagining [5:41] ZFS equi middle boxes everywhere and [5:44] maybe they're running omos and maybe [5:45] they're running Manta or smart osm or [5:48] something we'll see I'm kind of excited [5:50] about that possibility um and I've got [5:53] 10 seconds left um thank you any [5:56] questions or get off the stage [6:01] anything two [6:06] one all right Dan thanks very much uh [6:09] next will be uh friend of mine for a [6:11] number of years Abel Matthew on the [6:13] subject of debug [6:16] ability yeah so I'm Abel I work at a [6:19] company called back Trace um and we [6:21] actually think about debuggability quite [6:23] a bit um it's not a question that a lot [6:25] of people ask themselves but uh it's [6:26] something that we do uh and in [6:28] particular we've been asking ourselves [6:30] the question of how do we compare [6:31] compilers and that's because uh some of [6:33] the people that we work with you know [6:35] they're thinking about switching to C [6:37] lank versus GCC you know one of the [6:39] questions that they don't ask themselves [6:40] is what about debuggability you know [6:42] they really care about compilation time [6:44] executable performance you know what [6:46] error messages they're they're getting [6:47] which is a pretty popular one for [6:48] switching to Clank but they don't ask [6:51] themselves uh does this affect debug [6:53] ability and so I wanted to take you know [6:56] a minute to actually describe an [6:58] experiment that I ran you know actually [7:00] this afternoon um [7:02] that essentially what it did is it [7:04] measured the number of resolved [7:06] variables and what I mean that is I walk [7:08] through the stack trace and the [7:09] variables that are reachable uh I see [7:12] whether they're optimized out or not and [7:13] so I'm comparing GCC 4.9.2 versus cink [7:17] 3.6.0 and this is just sorry this was [7:19] what just came on ubun 25 out4 um so app [7:24] here is actually just an internal app uh [7:26] just something that we use at back Trace [7:28] I wanted to see what happened so with no [7:30] optimizations you know you're seeing [7:32] something on the order of near 3,000 uh [7:34] resolved variables with GCC versus um [7:37] something like only a thou uh 20 uh [7:39] 1,200 versus uh with no optimizations [7:43] with C length um on that same app you [7:45] know you're looking at [7:47] 1,700 um with 02 versus around 500 so [7:51] this is actually a pretty significant [7:53] difference you know where this matters [7:55] is let's say you're doing postmortem [7:56] crash analysis you know you're looking [7:58] at a core dump um if those variables are [8:01] optimized out the less information that [8:02] you have the harder it is to figure out [8:04] what's going on you know for people that [8:05] are on Ops teams here you know for [8:07] developers are asking like hey let's [8:09] switch the ceiling this is something [8:10] that they should consider then what we [8:13] did is uh I just took node.js um master [8:16] and I compiled it with GCC versus cink [8:18] and I wanted to see the difference um [8:20] there you're getting around uh 16,000 uh [8:24] sorry 1600 versus four uh 4 475 so still [8:28] pretty pretty significant difference [8:30] um this is obviously a pretty quick uh [8:34] experiment but what I wanted to do was [8:35] follow it up with the blog post because [8:36] it's pretty interesting as to reasons [8:39] why this is happening you know it's not [8:40] necessarily cang's fault or GCC what GCC [8:43] is doing might be something with GDB it [8:45] might be something with the debugger [8:46] that you're using um so yeah follow the [8:49] Twitter and hopefully we'll PL uh post [8:51] the blog post [8:52] soon uh yeah I went that pretty quick [8:55] because I was told I only have two [8:56] minutes so uh [8:59] uh any questions [9:02] anything you tell a [9:09] story so here we're actually using a [9:11] tool that we built at back Trace uh that [9:14] is essentially a debugger uh what it [9:16] does is it uses lib on wine to actually [9:18] walk through the stack TR and then it [9:19] uses dwarf information to actually try [9:21] to resolve the [9:25] variables [9:27] loc just locals just uh stuff on [9:32] automatic cool all right thanks for time [9:36] thank you able uh next up will be Dave [9:39] Peko I hope I pronounced that right with [9:41] uh what I think is going to be disaster [9:43] porn on something titled manta's bad [9:47] day all right thank you my name is David [9:50] poo I'm an engineer at joyant I'm going [9:52] to be talking about manta's bad day uh [9:54] this is a talk about an outage I wasn't [9:56] sure how I was going to jam it into five [9:58] minutes but I was inspired by Brian's [10:00] tweet on that day which is actually just [10:02] a two-word postmortem for the outage [10:03] which is wraparound autov vacuum the [10:05] funny thing about it was we were not the [10:07] first company in that literally in that [10:08] week that had had a major outage that [10:10] was related to a wraparound Auto vacuum [10:12] if you've never heard about this it's [10:13] probably because it's never hosed you in [10:14] production before if you use postgress [10:16] at some point it may and then you will [10:18] wish you remember this talk but you [10:20] won't because the only way you can [10:21] remember about wraparound autovacuum is [10:23] to have actually experienced an outage [10:24] as a result of it as far as I can tell [10:26] um so this is about the Manta storage [10:28] service this is a a an object store that [10:30] we run it's put get delete of bite blobs [10:32] over htdp with a Unix like directory [10:34] structure U Mark cavage and I gave a [10:36] talk about this it surged 2013 um and [10:39] it's a strongly consistent system it's [10:41] built heavily on a postgress based [10:43] metadata tier so we have a bunch of [10:44] shards for scalability and then each of [10:46] the shards has multiple postgress [10:48] databases for availability and basically [10:51] everything in the system all both all [10:53] the data path operations and the compute [10:54] jobs that you can run all access this [10:56] metadata tier so the outage happened at [10:58] 10:30 UTC on July 27th and what happened [11:02] was clients are starting to see very [11:03] high latency followed by a bunch of 500 [11:05] level errors it's a kind of a classic [11:07] case of client makes a request there's a [11:09] problem internally the API servers time [11:11] out after 2 minutes and send back a 500 [11:13] we were able to tell pretty quickly that [11:15] the problem was on one particular [11:16] postgress database um and simple queries [11:18] to that database were totally fine like [11:20] select now is totally fine the database [11:22] itself is not host but all queries on [11:24] this one table which we use for Manta [11:26] were totally stuck in fact there was [11:27] only one query in the entire database [11:29] that was not currently blocked on locks [11:31] for an extended period and that was an [11:33] auto vacuum that started at 10:30 which [11:35] is obviously uh an interesting [11:37] coincidence turns out this was not our [11:39] first issue with autov vacuum we'd had a [11:41] whole bunch of issues related to [11:42] insufficient autov vacuuming so if you [11:44] if you haven't used postgress I don't [11:46] want to get too into the details but um [11:48] vacuuming is a process by which it deals [11:50] with tupal that are basically they were [11:52] part of a table but they're no longer [11:54] part of a table but they might have been [11:55] visible to some transaction concurrent [11:56] with the one that removed them so they [11:58] don't get removed right away so you have [11:59] to do this asynchronous vacuum operation [12:01] but because that's kind of hard to [12:02] manage automatically they they've added [12:04] an auto vacuum system but occasionally [12:06] we've seen service problems resulting [12:08] from autov vacuuming running or I [12:10] shouldn't say we we hadn't root caused [12:11] most of them but Auto vacuum was on the [12:13] scene the outage was either resolved [12:15] before we were able to root cause them [12:16] or we just canceled the auto vacuum [12:18] without figuring out which is obviously [12:19] problematic but there we were so we [12:21] thought in this case well maybe [12:22] canceling the auto vacuum will help but [12:24] we want to be able to root cause it so I [12:25] tried to grab whatever data we could [12:26] from the PG locks table which shows you [12:28] what's blocked um and then we restarted [12:30] postgress and this worked it was great [12:32] for about 10 minutes and postgress [12:35] immediately kicks off another Auto [12:36] vacuum and this time we notice it's [12:37] specifically to prevent wraparound this [12:39] is a much more specific issue this is [12:41] what postrest considers kind of a more [12:43] critical form of autov vacuum where it's [12:45] much more aggressive about database [12:46] locks that it's holding and um if you [12:50] read the the documentation on this it's [12:52] a little scary what what happens is you [12:53] have these transaction IDs there's a [12:55] four it's a 32-bit number so it's a 4 [12:57] billion space but it's a circular space [12:59] there's 2 billion on either side and [13:01] what if you wrap around then you have [13:02] things that were supposed to be in your [13:04] past that look like they're in the [13:05] future it's really bad the database [13:06] shuts down so that's what we understood [13:09] at this point so we said okay we [13:10] actually do have to let this thing [13:11] finish and do its job um but is there [13:13] anything we can do to speed it up so we [13:15] start looking at it with the tools that [13:16] we have and we we see that the postgress [13:18] process which is doing all the work is [13:20] actually spending most of its time [13:21] sleeping and that's kind of weird um and [13:24] at first we see we're we're tracing the [13:25] kernel Schuler this is one of those [13:26] cases where people people are like I'm [13:28] building an application why do I need to [13:30] trace the colel scheduler well because [13:31] your thing is is sleeping all the time [13:33] and you want to know why we were able to [13:34] figure out that it was sleeping on iO [13:36] and the the problem is it's reading a [13:38] whole bunch of files and it's doing [13:39] sequential reads of a file which file [13:41] systems are generally pretty good about [13:42] prefetching but it's also reading tons [13:44] of files in the same directory which [13:46] were also sequential but the file system [13:48] file systems don't generally know about [13:49] that so we were able to prefetch those [13:51] and speed it up quite a bit the next [13:53] thing we did we found it was actually [13:54] still sleeping a lot but this time it [13:56] was in a mechanism within postgress [13:57] called the cost-based vacuum delay this [13:59] is a mechanism that's there to prevent [14:02] the autovacuum process from taking over [14:04] all the resources on the system and so [14:06] it it literally just like sleeps every [14:07] few every period of time for a little [14:10] bit which is exactly the wrong thing to [14:11] do here because this was literally the [14:13] only process in the database which was [14:14] not currently blocked on this process so [14:17] that was actually exactly the wrong [14:18] thing which was unfortunate um but and [14:21] and there's no Dynamic way to change [14:22] this but you know it's our computer we [14:25] have it's all just memory right so we [14:26] just use the debugger to whack these [14:28] Global variables and get a 5x [14:29] Improvement in read through [14:31] quick so we're doing the pre-etching for [14:33] a couple of hours the autov vacuum [14:35] completes the service comes back online [14:37] everything is great for about 10 [14:39] minutes the autov vacuum kicks off again [14:42] to prevent wraparound and um it's [14:45] exactly the same state exactly the same [14:46] table at this point we're thinking we [14:48] need a new approach here because this is [14:50] not converging at all um and and so we [14:52] look back at the postgress stuff to get [14:53] a little bit more information we find [14:55] that although it kicks off this problem [14:56] happens at 2 billion postgress kicks off [14:59] off this emergency operation at about [15:00] 200 million which is considered [15:02] conservative and that's what they talk [15:03] about in the docs it's conservative from [15:05] the perspective of this problem it's [15:07] very aggressive from the perspective of [15:09] our our thing is actually down right now [15:11] because of the autovacuum we actually [15:12] wish we hadn't dealt with this right now [15:13] we ended up tuning that tunable up to [15:15] 500 million and everything came back and [15:18] that actually that did work we're still [15:19] there um and then we went and figured [15:22] out why was our service totally hosed at [15:23] this point um the fund was just starting [15:25] so that was a 10-hour outage I spent [15:27] about a week trying to understand [15:29] everything that happened we were able to [15:30] figure out what the problem was and the [15:32] high order bit for people using [15:33] postgress is um the autovacuum process [15:37] and the wraparound autovacuum process [15:38] does not intrinsically cause your whole [15:40] data path to go down but there are [15:41] certain things you could be doing as an [15:43] application that are very hard to know [15:44] that you're doing that will actually [15:46] cause exactly this problem so all the [15:48] details are um I knew I was going to be [15:49] out of time at this point all the [15:50] details are in our postmortem um [15:52] definitely check that out if you're [15:53] interested for more or come ping me [15:55] thank you the next talk is [15:59] by Dimitri carvonis and it's [16:01] called this was a suggested topic name [16:05] and I thought it was a good one Ruby on [16:16] slime that was actually the punchline to [16:18] one of the few jokes in my presentation [16:20] thanks [16:21] G um well thank you all for [16:25] coming this is Auto [16:30] um my presentation is called the future [16:32] of biological Computing or overhyped [16:35] sludge and my only qualifications coming [16:37] into this are my own curiosity and [16:40] hopefully by the end I will have peaked [16:41] your own curiosity in this subject as [16:44] well perhaps you've woken up one morning [16:46] to find a strange orange or yellowish [16:48] foam growing on a lawn or on a tree this [16:51] is called a slime mold it has no brain [16:55] it doesn't belong to the Animal Kingdom [16:57] the plant kingdom or even the fungus [16:58] Kingdom so the term mold is actually a [17:01] misnomer it's a collection of single- [17:03] celled organisms UK carots that bear [17:06] resemblance most of all to amibas it [17:09] grows in a network structure it's a [17:11] self-organizing system and it's capable [17:13] of complex behavior and this last bit [17:16] has been responsible for a great deal of [17:18] Buzz online about slime molds for [17:21] example it's um responsible for such [17:24] blog titles as why my slime Wold is [17:27] better than your Hadoop cluster [17:30] um it turns out this probably isn't the [17:33] case but where where did this sentiment [17:36] come [17:36] from there's sort of a canonical study [17:39] on this where uh this dot in the center [17:41] is the mass of the slime mold and the [17:44] surrounding dots are little bits of [17:46] grain that have been scattered around it [17:47] in the formation of the subway stations [17:50] in the Tokyo subway system so this slime [17:53] Wold starts spreading out its tendrils [17:55] to consume this nourishment uh forming a [17:58] very dense Network eventually it starts [18:01] pruning the non-essential edges of this [18:03] network so as to have an optimal [18:05] configuration of sorts it turns out that [18:09] this structure actually mirrors the [18:11] design of the existing Tokyo subway [18:14] system to a great deal it's actually [18:16] more optimal and it was later discovered [18:20] that the model used by Tero here for the [18:22] study um will eventually converge to the [18:25] optimal solution I stretch eventually [18:27] because there are some caveats here [18:28] which we'll get to in the general case [18:31] in other words you can just throw down a [18:32] bunch of grains and the slime mold will [18:34] find the optimal [18:37] configuration so there's also research [18:40] done in Computing with slime mold we can [18:42] shape slime mold into arbitrary [18:43] structures we can actually grow nor [18:45] Gates out of it that means you can [18:47] construct any circuit you want out of [18:50] slime molds there is a project called fi [18:53] chip not pie chip um that is dedicated [18:57] to Growing slime oldold chips both in [18:59] the analog style of computing in the [19:01] Tokyo subway system and a more of a [19:03] digital style of computing in terms of [19:05] these nor [19:06] Gates as far as I know they haven't made [19:08] too much progress progress but uh we're [19:12] hopeful so like I said there are some [19:15] caveats slime mold grows at a rate of 1 [19:17] cm per hour um it's dubious to how [19:22] useful this is right and if we can make [19:25] nor Gates out of slime molds why go [19:27] through this trouble when we can make [19:29] nor Gates of better quality and quantity [19:32] today in fact in all of your devices [19:34] right now you have orders of magnitude [19:36] more of these Gates than we can [19:39] practically make with slime wolds [19:42] additionally going back to this network [19:44] optimization problem and these um [19:46] strange smarts present in the Slime [19:48] worldold this is um appearing sort of in [19:51] a tradition of where we look to physical [19:53] processes for potential help with [19:55] computation I guess the epitome of this [19:58] is Quantum computation but it turns out [20:01] that typically these physical processes [20:03] don't give us very good Solutions the [20:06] most popular example of this has been by [20:08] Scott Aronson who showed that if you [20:09] take soapy water and throw it over some [20:13] um spools embedded between glass the [20:16] soapy water is not going to arrive at [20:18] the minimum Steiner Tree in other words [20:20] it's not going to be able to compute an [20:22] NP complete problem nearly [20:24] instantaneously as some people said who [20:26] were spreading rumors this is some [20:29] true or true as well for the slime mold [20:32] it's not going to be able to solve NP [20:34] complete problems for for you but [20:37] experimentally it's been shown to be [20:39] able to arrive at an approximate [20:40] solution for traveling salesperson which [20:43] for an organism with no brain is pretty [20:47] impressive uh this is a slime mold [20:49] solving traveling salesperson you can [20:50] see toward the end it begins to look [20:53] like [20:55] something [20:57] so what are the possibilities could we [21:00] have autonomous circuits living in [21:02] nature that are self assembling [21:03] self-healing adaptive to change have a [21:06] low raw materials cost and no [21:07] electricity cost we really don't know [21:10] right now we [21:12] can't of course um observing slime mold [21:15] in the laboratory is of course a [21:18] legitimate use um observing it form [21:21] various structures and various [21:22] circumstances and observe what [21:25] happens but can we come up with anything [21:27] now [21:29] so I'll offer a few suggestions I think [21:31] offering up slime as a computational [21:33] resource leads to new complexity [21:35] classes primarily my question is how [21:37] much of the Earth do we need to cover in [21:39] slime to solve a given [21:41] problem the size of the slim universe is [21:44] finite and we're running up against the [21:48] physical limits of computing here um you [21:52] might ask a question like how many [21:53] continents have to be engulfed in slime [21:55] to solve for the click problem on a [21:57] million nodes [21:59] I find that this usually results in very [22:02] productive [22:04] conversations if we don't have access to [22:07] very good Computing we can always Market [22:08] it as artisanally branded allnatural [22:11] analog bio Computing slow [22:14] Computing sourced from local [22:17] forests and um I suggest we produce a [22:20] ruby on slime project I think we can get [22:24] a lot of people on board on the slime [22:26] mold Revolution because there's always [22:28] the tagline uh because you're already [22:30] using [22:33] Ruby and here we observe the use of [22:35] slime mold in art this might be one of [22:37] the most interesting applications here [22:39] music Professor Eduardo Miranda is [22:42] performing a piano duet with slime mold [22:46] I I wish I had music for [22:48] you thank [22:50] you the generally lightning talk is a [22:54] night full of mysteries it's a night [22:55] full of mysteries [22:59] and we'll just put this behind [23:01] you that's cool all righty all right I'm [23:03] literally doing running the examples off [23:06] my phone this is fabulous this is as [23:08] prepared as I was uh I came up with this [23:11] lightning talk literally in the hallway [23:13] about an hour before it was time to give [23:15] lightning talks and I am going to talk [23:17] to you about grammar uh because you know [23:20] why not we talk about all kinds of other [23:21] things and grammar is just a scalable as [23:23] anything else uh so um in my real life [23:26] I'm a engineering director but I have a [23:28] second line where I am a professional [23:29] writer and I write all of the time and I [23:33] spend hours and hours and hours editing [23:35] writing and then I go into work and I [23:36] read engineering writing and it makes me [23:38] die [23:39] inside so and then what happens is I sit [23:43] down and I take their writing and I [23:45] attempt to edit it it's usually in [23:46] Google Docs and you know what Google [23:48] Docs will let you do reject people's [23:51] changes magnificent so this happens to [23:54] me all the time so the number one [23:56] problem that really is facing [23:57] engineering writing is that you are [23:59] terrified of your subject and you are [24:01] terrified of your verb it is literally [24:03] going to LEAP out of your computer and [24:05] strangle you at any moment or at least [24:08] that's how you all write which is great [24:10] so this is called writing in passive [24:12] tense I have a whole bunch of examples [24:14] here that I'm going to try to edit up on [24:16] the fly so they a little bit more surgy [24:19] uh so there's two kinds of ways of [24:20] writing a sentence when uh dealing with [24:22] a verb uh there is active tense in which [24:26] the subject is Acts directly on the verb [24:29] and there is passive tense in which the [24:31] verb acts on the subject uh the trick of [24:35] passive tense is that you want to get [24:37] your subject as far away from your verb [24:40] as possible so nobody actually notices [24:43] that you just wrote this sentence and [24:46] that is how most Engineers write so I'm [24:49] going to attempt to edit some sentence [24:50] for you literally on the fly without [24:53] prepo done here so you're getting the [24:55] quality you paid for which is nothing [25:00] so first [25:02] example uh we'll actually use the [25:04] example it's on my phone because I [25:05] really like this example uh so the first [25:08] example is that uh at dinner six shrimp [25:11] were eaten by Harry right so as long as [25:14] we don't know that Harry ever ate those [25:17] six shrimp those six shrimp are safe [25:20] until we get to that period and we can [25:23] refactor that as Harry ate six shrimp at [25:26] dinner now we know what Harry's doing [25:28] he's eating shrimp in fact we can go to [25:30] the sorus right now we can punch up that [25:32] verb and make that sentence way more [25:35] interesting Harry nod six shrimp at [25:38] dinner Harry just gurgled on six shrimp [25:42] at dinner is there fantastic verbs right [25:46] next example the data center is roamed [25:49] by beautiful Network Engineers not the [25:52] sentence on my [25:54] phone we know the data center is there [25:57] we know somebody's roaming it we don't [25:59] want anybody to know that the network [26:01] Engineers are in there we'll edit it for [26:03] them beautiful Network Engineers roam [26:06] the data center now they're there and [26:10] you'll know way you'll get rid of [26:12] them uh so uh let's find a buy uh here's [26:16] a good one uh monitoring is going to be [26:19] watched by us tonight so we know [26:21] there'll be monitoring and eventually be [26:24] watched and eventually we'll get to us [26:26] and it'll be tonight or we say we're [26:29] going to watch the monitor tonight [26:31] because our systems will probably [26:34] crash and this allows us to convey a [26:37] whole more information very quickly [26:40] short and Punchy sentences are the [26:43] secret to actually getting your subject [26:46] across to your audience it also shortens [26:49] your sentences makes them easier to read [26:52] easier to digest and you will have more [26:55] audience longer and they will retweet [26:57] you on interet internet on Twitter I [26:59] tell you right now you want tweets [27:01] active [27:02] tense so how do you get rid of some [27:04] active tense the word by if you use that [27:07] we were watched by yeah that by that's a [27:10] signal right there uh were anything that [27:13] has to be in it to be is your enemy [27:17] right we should have been doing [27:19] something uh we don't want anybody to [27:22] know what we were doing but we should [27:23] have been doing it instead of just doing [27:25] it could be must be [27:29] uh anything of that ilk really terrible [27:31] I only have 45 seconds so I want to rant [27:33] about my other favorite subject of it's [27:34] an engineer writing which is adverbs [27:36] don't use them they are bad just get it [27:40] in your head so if you see practically [27:42] yeah erase that word only get rid of [27:45] that word very yeah that's awful the [27:47] word just yeah you just kill yourself [27:50] don't use that anymore right just edit [27:52] it out 20 seconds there's an app online [27:55] called Hemingway web.com it teaches you [27:58] to write like Ernest Hemingway it's [28:00] awesome it will tell you how bad your [28:02] writing sucks how many adverbs you have [28:04] how much passive tense you have how many [28:06] runon sentences you have if you work [28:08] with this app I guarantee you you will [28:11] not have me ranting at you on stage at [28:13] Serge anymore so Engineers learn to [28:16] write we can write well we can do it [28:17] together yeah go [28:19] [Applause] [28:22] team [28:24] that's um so next we have Josh schmida [28:29] with the mysteriously named [28:32] talk [28:36] [Applause] [28:40] [Laughter] [28:45] Jackalope uh this s is also called bad [28:47] ideas maybe just ideas hey I'm Josh I [28:51] also a CTO I'm also hiring [28:55] [Laughter] [28:58] uh first uh Power hey this is a Tech [29:00] conference where's the power strips why [29:02] I have to say hang on the hall to charge [29:04] my terrible laptop with a terrible plug [29:07] I mean seriously [29:09] Apple what you guys are supposed to be a [29:12] design company get your together [29:14] all right um the thing I would actually [29:16] so like I was watching the uh brain tree [29:18] talk about the uh the wonderfully [29:21] named [29:24] broxy um not that Jack Loves a whole lot [29:27] better to be honest but you know it's a [29:29] unicorn with rabbit you'll see in a sec [29:32] so like the idea is we want to do SOA we [29:35] want to have you know services in the [29:38] internet with like a communicating [29:40] pattern and I don't want to drive my [29:42] office Engineers insane so I drove them [29:44] insane in a slightly different way and I [29:45] wrote a whole new like application [29:48] server uh so the idea is you get traffic [29:52] from the internet through your web thing [29:55] into your Ruby thing because yeah this [29:58] is Ruby uh and then there's a little [30:00] thing there called amq party which [30:02] basically acts like HTT party which is [30:03] an HTP library that actually talks to [30:05] Rabbit even though you think you're [30:07] talking to http uh and then on the other [30:10] end instead of like your web app being [30:12] run by an actual like decent rack server [30:14] like unicorn uh we run it on jackob so [30:17] uh at jackob reads off rabbit and fakes [30:20] out HTTP and you can run your Ruby stuff [30:22] and it's cool and this is actually [30:23] working we actually have this in [30:24] production on one tiny little uh app and [30:28] and it didn't totally Drive our devops [30:30] people crazy uh they're just a little [30:32] bit crazy to start so that that's [30:35] fun uh anyway so uh this is a lot like [30:38] how broxy I guess is structured [30:41] similarly except they were using redis [30:43] which is [30:44] better uh so uh anyway we're probably [30:49] not going to go forward with this I gave [30:50] a talk about this at rails comp a couple [30:53] years ago and everybody was like oh [30:54] that's great and now I'm like yeah we [30:56] didn't do that really so move along now [30:59] uh anyway um the the thing that really [31:01] kind of broke this was U ignore the [31:04] rubies mostly uh this yeah rabbit is not [31:09] really scalable and does not tolerate [31:12] Network partitions well that's right [31:14] there on the very it's the first [31:16] sentence on this dock I mean that's just [31:20] you had one job man um so maybe a better [31:26] idea still probably crazy thing I'm [31:29] thinking about I love your input uh so [31:31] you still you know get the traffic [31:32] traffic traffic API server then it talks [31:34] actual HTTP to something which might be [31:37] written in a halfway decent language and [31:38] not actually a portal to hell Brian [31:41] Kil that writes to kavka and then is [31:45] read from via Kafka and fakes HTTP on [31:47] the back end and then we do some [31:49] wonderful packet mangling to Route the [31:51] traffic right back to the original [31:52] client uh and not actually go back [31:54] through all that pattern so uh this is [31:56] kind of like percolating through my head [31:58] right now uh if you know how to make [31:59] this happen without me having to write a [32:00] lot of has school I'd love to talk to [32:03] you uh or if you think it's is a great [32:05] idea I want to help out I also like to [32:06] talk to you uh anyway that's it thanks [32:08] I'm Josh good luck hi I'm John uh so I [32:12] spent a couple months of my life earlier [32:14] this year writing a memory allocator in [32:15] user space uh it was somewhat [32:18] complicated in that uh it didn't use it [32:22] packed everything together there was [32:23] Zero fragmentation between allocations [32:25] um I knew that going in but that meant [32:27] that a of tooling like Val grind [32:30] wouldn't work as well because an [32:31] allocation that goes off the end of its [32:34] space and overflows into another one you [32:36] can't catch it so this was generated by [32:41] basically the following quote let's [32:42] write a new memory alligator to support [32:44] that and that's how I spent three months [32:46] of my life writing a new memory [32:48] allocator and shoehorning it into a [32:50] moderately High scale production [32:51] application um my favorite definition of [32:54] existentialism is a sense of [32:55] disorientation or meaninglessness in in [32:57] an apparently absurd world and that's [32:59] basically how you feel when you're [33:01] debugging a memory allocator in [33:02] production uh especially when you can't [33:05] use Val grind on it so uh debugging a [33:09] memory allocator in production when you [33:10] can't use Val grind on it and don't have [33:12] any good tooling is basically to me like [33:14] standing in the middle of Time Square in [33:16] the middle of the day finding a pile of [33:18] dog crap in the middle of a crosswalk [33:20] and your only recourse being to stand [33:22] there and wonder who has done [33:25] this oh my God [33:28] and that's all you can do eventually you [33:30] figure it out maybe but maybe you have [33:33] to stand there with a microscope on the [33:34] crosswalk and examine every foot that [33:36] walks by who knows uh so I'm going to [33:39] use the five stages of grief as a way to [33:43] talk about memory allocator and the [33:45] process of writing them um so denial [33:48] yeah I mean you think that it'll be easy [33:50] that's fine you're done uh writing a [33:51] memory allocator it turns out a [33:53] complicated one at that is easy [33:55] debugging it is really hard because you [33:58] were [33:59] arrogant so then then of course comes [34:03] anger we all know how this goes you've [34:05] been there [34:07] bargaining please red zones no one [34:10] started the timer by the way so I'm just [34:11] going to keep going um can I can I have [34:14] red zones can I build red zones that [34:16] might have been a good idea when I [34:17] started anything really literally [34:20] anything that allows me to use Val grind [34:22] or if Val grind's too slow for you or [34:23] you like better error messages address [34:25] sanitizer great project um uh you need [34:28] modern compiler though and that's [34:30] something I can always have um [34:33] bargaining definitely a good stage of [34:35] grief depression at this point you start [34:38] rewriting the entire memory allocator [34:39] from scratch building in proper hooks [34:41] for tooling and proper ways to integrate [34:44] things like user level red zones so you [34:46] start rewriting half of what asan has [34:48] already done for you because you can't [34:50] use asan or you write good hooks for [34:53] it uh at some point you get so depressed [34:56] that you wonder why bother with this why [34:59] not do something else I've got exobytes [35:01] of address space why not just hand out [35:03] random pointers and that's what I tried [35:05] I tried that it's right there it works [35:07] pretty well uh you allocate uh a [35:11] terabyte of address space and if you [35:13] hand out random pointers it works great [35:14] until you allocate about 10 Megs of [35:16] space and then you start to see [35:18] corruption what's [35:19] that exactly um so if you just need a [35:24] real like terrible toy program just hand [35:26] out random pointers [35:29] uh and the last stage of grief is [35:31] acceptance you've realized that you were [35:34] arrogant you've accepted that you wrote [35:36] some proper hooks you wrote some user [35:38] level red zones you read the asand paper [35:41] front to back reimplemented some of it [35:44] and really you just have to remember [35:45] that you shouldn't do this in the first [35:46] place J Malik is really great they've [35:49] done a lot of work on it and that's all [35:52] I have the timer also now says zero so [35:55] don't know when that happened but thank [35:57] you [35:59] his talk is titled native don't [36:08] scale all right thank you thank you so I [36:11] work at AB Nexus M see what I do is I [36:14] make things fast uh and whoa what's [36:18] going on here all right and one of the [36:21] things I I keep hearing is we could do [36:24] C++ templates you get very specialized [36:27] code go fast and like oh oh let me tell [36:29] you about Cash's friends but this is [36:32] getting worse now because we're all [36:33] doing big data what I hear these days is [36:37] we're going to take our machine learning [36:38] model and convert it to machine code and [36:41] it'll be really [36:42] fast and now I just I don't know if I [36:44] should scream or weep uh especially when [36:48] it's your CEO telling you that so [36:51] this or drink yes that happens a lot so [36:54] this is a rant that usually goes through [36:56] my head as I hear d things um this is a [37:00] picture of a die for xon E5 that we run [37:03] in production it's an amazing piece of [37:05] engineering you say things like uh we're [37:08] ConEd by the speed of light here if we [37:09] want to go faster we're going to have to [37:11] have a smarter [37:13] chip and this is what it has to [37:15] communicate with the rest of the world [37:17] that's 2,000 pins that's all we [37:20] have of those 2,000 pins a lot go [37:24] directly to dims and dims have about 200 [37:27] 140 bin that's not a lot to move around [37:30] terabytes of [37:33] data in fact if you look at the Ben [37:35] withd we have on our production machines [37:37] we can run about five bytes per cycle [37:40] per dim that is not a lot now let's look [37:44] at a piece of code that is pretty much [37:46] ideal for current processors that's a a [37:50] Dax spe so you do ax plus y over vectors [37:54] and what you find is that you have more [37:56] bites of [37:58] instructions than bytes of data that it [38:00] accesses so we're all used to thinking [38:03] oh it's going to be slow it has to go [38:04] through all of this data but if you also [38:07] had to go through all this code because [38:09] you decided to convert the machine [38:11] learning model to straight machine code [38:15] you're going to spend most of your time [38:16] waiting for memory not even for your [38:18] data but just for the instruction to [38:20] know what to do with that [38:23] data so I run an experiment I'm like all [38:25] right let's see what happens when I have [38:27] huge basic blocks just [38:29] knobs and as we [38:31] see even with 1 kilobyte basic blocks as [38:36] you hit main memory you can do about one [38:39] 1.5 instructions per clock that's very [38:42] much the ideal and those are one bite [38:46] instructions as we saw here most [38:48] instructions on X6 are about 3 to six or [38:51] seven bytes so we're probably looking at [38:54] about 0.12 IPC as you your code is not [38:59] in cash anymore this is amazing [39:01] performance as you all know uh normal [39:04] business codes gets around I don't know8 [39:07] 1.5 instructions per clock so it's just [39:10] you know five times as [39:12] slow and there are two reasons for this [39:14] Effect one is well you keep jumping [39:15] around in random memory so while we're [39:18] used to thinking oh I'm I'm making [39:20] random accesses it's going to be slow [39:21] well you have the same problem with code [39:23] except it's even [39:24] worse right because if you don't have [39:27] code you don't even know what other data [39:29] you could be prefetching or looking at [39:31] there's no speculation anymore you just [39:34] turn your fancy 2015 computer into [39:38] something that's out of the [39:40] 90s and then there's just well the [39:43] there's no bandwidth cache amazing [39:46] bandwidth that's L1 L2 L3 then you hit [39:49] the main memory and you have something [39:51] like one 14 de bandd so yes if you want [39:55] to scale please don't tell me you're [39:57] just going to turn it all into machine [39:59] code um with [40:01] that I will leave it [40:04] to Andrew Wong and is Untitled lightning [40:11] talk [40:13] hello I'm Andrew [40:16] um so as gear alluded to my uh [40:19] professional career has been a series of [40:21] mistakes [40:28] some of them are less painful than [40:31] others so there are there are two [40:33] options here uh there are software [40:36] developers and Ops people here there's [40:39] also like a human interest part to this [40:41] story we could tell so um which one do [40:44] you want the the operations the software [40:47] developer it's basically pager or the [40:49] human interest [40:51] story not everyone speak at [40:54] once human interest okay [40:59] so not really a [41:01] planner did eventually plan something in [41:04] my life though and uh about a year ago [41:08] we decided we'd have a [41:10] kid so we didn't uh well I we I didn't [41:14] tell the company I worked for that this [41:15] was happening I uh discovered it and [41:18] then spent about nine months keeping it [41:20] a [41:25] secret yeah it is funny isn't it [41:31] so towards the end there's [41:34] the so I was in New York and uh my wife [41:37] was in [41:39] Copenhagen and I mean it it's it's eight [41:42] hours across the pond it's not that bad [41:45] she did most of the [41:49] work so I showed up about a month before [41:52] to be supportive and not risk serious [41:55] divorce [41:58] and I [41:59] uh went through this told the company [42:02] that I would I would show up in a bit [42:04] I'd be [42:05] back so [42:08] uh Halloween my daughter was born I [42:12] spent the night before that patching [42:14] some code that had broken [42:18] production [42:19] um show up to the next meeting the All [42:22] Hands or not the All Hands but the the [42:24] team meeting and uh I I I raised my [42:28] daughter and uh it's wonderful I mean [42:31] it's great uh if you don't have kids I'm [42:34] not going to encourage you to do it it's [42:35] a something you should come to on your [42:39] own some of the fellow team leads uh [42:43] almost fell out of their chair uh others [42:46] realized that I was probably more [42:48] responsible than I let [42:50] on others probably thought I was really [42:53] actually super [42:56] irresponsible um so a leading question [42:58] that you get when you have a kid is um [43:01] how much how much worse is pager [43:05] now because you don't you don't really [43:07] have like there's there's one thing that [43:09] you actually care about and then there's [43:11] like [43:16] nagios I know some of you are thinking I [43:18] named my daughter nagios [43:23] but I didn't [43:30] so Pedra was actually really good [43:32] prep um at least in the organizations I [43:36] worked on because Pages Pages start [43:38] slowly there one [43:40] two dies down you think you're saved [43:44] you're like [43:45] oh some poor other idiot reached for his [43:47] phone [43:49] first and that that's really what [43:51] raising a kid's like right like it's [43:53] it's a battle especially when the sun is [43:56] down uh it's first person to wake up [43:59] that's it you're on [44:01] duty which is hard if you're a [44:03] programmer because you're already [44:11] up so I can I can attest that [44:15] uh that uh pager is worse um and uh it [44:21] really it really wasn't that bad because [44:22] I had someone to keep me company when [44:24] everything was breaking [44:27] and they were really happy to see [44:30] me unlike the machine which was an [44:33] uncaring silicon god with no [44:37] forgiveness none at all the other one [44:39] you can feed or sing to and it it gently [44:44] stops yelling at [44:46] you so that's my talk I hope you enjoyed [44:50] this human interest story if you have [44:52] any questions you should ask them in the [44:54] next 19 seconds [44:58] anyone but [45:03] Robert I mean it doesn't look very [45:05] friendly I mean no one else is sitting [45:07] near [45:08] you but okay Robert take it [45:18] away no [45:21] actually not nagios not circonus not [45:24] omnios not smart OS not [45:28] Linux um named after a a a Greek god [45:32] Arianne who uh who was the goddess of a [45:36] maze and kept a [45:42] Minotaur any other [45:45] questions [45:47] excellent [45:51] Goodbye Mr treat which is a treat I was [45:55] really happy and surprised that he [45:57] volunteered to give a talk [46:03] which is [46:06] topical Yogi Bara [46:09] unsung [46:11] yeah normally I might not do that ready [46:15] okay good all right uh oh hold on slow [46:19] down stop it's backwards all right so [46:21] Yogi Bara uh so so I sort of felt [46:23] compelled to say a few words uh for this [46:26] man who's a great man who I don't think [46:27] gets enough credit especially at [46:29] technology conferences U I really [46:31] haven't heard too many talks about him [46:32] and I think that's a shame so here to [46:34] set the record straight uh We've [46:36] credited a lot of people over the years [46:37] with uh whatever the heck devops is um [46:40] I've heard a lot of people talk about it [46:41] a lot of people say they know what it is [46:42] and who created it and where it's been [46:44] and what it's done um a lot of people [46:46] attributed to Jean Kim WR the Phoenix [46:48] project uh Kevin bear George spafford [46:50] also worked on uh the visible Ops book [46:52] with Jean so they also get some credit [46:54] um I've heard people say like all Spa is [46:56] like the origin of of devops or [46:58] something like that or he's one of the [47:00] new fathers or something so there's a [47:01] lot of credit going around uh tono um [47:05] worked in Toyota manufacturing gold rad [47:07] another sort of person who's done a lot [47:08] of work uh that is sort of attributed to [47:10] being Legacy info on on uh devops if you [47:14] want to learn that um I think probably [47:17] the person who gets the most credit I'm [47:19] not sure it's deserved John Willis tells [47:21] me it is uh Deming is a guy who really [47:23] invented devops right so if you want to [47:25] learn what the core principles and of [47:27] devops is like you go study Deming and [47:29] that's the way of the future um however [47:32] like I'll give Deming some credit like [47:34] he's there you can probably reverse [47:36] engineer his principles into devops [47:37] that's cool um but I think the person [47:40] again who always gets left out of the [47:41] conversation is Yogi [47:43] bar and Yogi bar had a lot of good [47:45] things to say about technology and a lot [47:47] of good things that I think directly [47:48] apply to the world that we live in and [47:49] to devops as a whole uh for example this [47:52] is one of his most famous saying the [47:54] future ain't what it used to be and [47:56] that's so true if you think about it you [47:58] know when you go and like think about [48:00] like waterfall and Agile development [48:02] it's our attempt to try to deal with [48:03] this kind of a problem right the future [48:05] ain't what it used to be we come up with [48:07] plans we do like rollouts and all that [48:09] stuff and it never ends up the way it's [48:11] supposed to be right and this is the [48:13] kind of thing like I that's like I say [48:14] it just totally gets [48:16] overlooked Amazon web infrastructure [48:20] right we've all heard about it many [48:22] people here use it many people think [48:24] it's a great thing you know what I don't [48:27] know if I was starting a business today [48:28] I probably wouldn't use it nobody goes [48:29] there anymore right it's too crowded use [48:32] joyant nobody's using that right like [48:35] you're get all [48:36] kinds you get all the bandwidth and IO [48:39] and whatever you need like it's great I [48:41] would totally do that instead right but [48:44] but think about like that that whole you [48:45] know resource contention is a thing so [48:47] you got to think about that when you're [48:48] trying to like deploy and build systems [48:51] right keep in mind what Yogi's told you [48:54] he's tried to lay this all out for years [48:56] he's been saying this stuff nickel ain't [48:58] worth a dime anymore have you seen [49:00] anything out of Silicon Valley that kind [49:02] of you know follows like modern laws of [49:04] physics when it comes to how much it's [49:05] going to cost whether it's developers or [49:07] houses or you know the BART train and [49:10] getting a burrito like all of that stuff [49:12] way crazy overprice doesn't make any [49:13] sense it's just insane right technology [49:16] industry just it's it's lost its mind [49:19] and Yogi knew this Yogi foresaw this and [49:21] that's what's [49:23] important one of the things I I would [49:25] say you you can't stop all mistakes from [49:28] happening right that's totally true I [49:30] always try to tell people on my team [49:32] like Hey try to screw up in ways that [49:33] don't hurt you right and I'll admit that [49:36] I basically have plagiarized this from [49:37] Yogi right it's okay to make mistakes [49:40] that's fine just don't make too many [49:41] wrong mistakes right those are the ones [49:44] that are bad good mistakes are the kind [49:46] that we hear about at surge conferences [49:48] and I encourage you to make those kind [49:49] of mistakes because that's what we need [49:51] more of in the industry [49:54] absolutely another thing you hear a lot [49:56] about when comes to devops it's all [49:58] about monitoring systems measuring [50:01] sharing that information you know who [50:04] said that first damn right Yogi Bara [50:07] uhhuh you can observe a lot by [50:10] watching I've done this before I've [50:13] watched all kinds of things and I [50:15] observe so many of them and and it's [50:18] just so true like these are fundamental [50:20] truth you cannot disagree [50:24] with so like I said Yogi Bara unsung [50:28] hero he passed away this week that's why [50:30] I'm bringing this news uh as you're out [50:32] on the town tonight or whatever National [50:34] Harbor is as you're out on the town [50:36] tonight you know don't forget to pour [50:38] one out for Yogi he's given so much to [50:41] us it's time that we take a moment to [50:43] reflect upon him and realize his [50:45] greatness and study his teachings thank [50:50] you so um over the you know everybody [50:53] knows who knows [50:54] Theo okay so so Theo was a founder of [50:57] omniti and founded this conference um [51:00] he's he's um I guess slacking off for a [51:02] year or something I think he's wandering [51:04] around the world and uh this is his [51:06] first search con that he's ever missed [51:07] and he was you know sort of broken up [51:08] about it and was tweeting about that [51:12] um you know there was an a AWS outage [51:16] over the weekend right then and [51:18] traditionally um Amazon would would hold [51:21] their uh fall outage during [51:24] sergeon um but this was a problem from [51:27] the organizers because it would you know [51:29] sort of make your experience would surge [51:31] not good every year so um they asked and [51:33] apparently Jeff Bezos complied and they [51:36] they kind of moved the outage to before [51:38] you got here so you you'd have the time [51:41] um to drink out there in that town thing [51:44] that Rober [51:45] mentioned so you know what could we do [51:48] because this is recorded sadly and um [51:51] what could we do you know for Theo so we [51:55] thought that for [51:57] Theo we should do an interpretive dance [52:00] of the postmortem of the weekend's [52:05] outage cannot believe I'm doing [52:09] this oh I'm just going to read stuff oh [52:11] don't no no no I'm not dancing no no no [52:14] no no uh you are yeah so um I'd like to [52:17] get a few [52:18] volunteers [52:21] seriously come on let's go some people [52:24] want to do this [52:29] it won't go to your bosses or employers [52:31] yeah you got people you know and the [52:33] funny thing is we have a bunch of shills [52:34] and now we have people that like are not [52:36] chills so we really don't know what's [52:37] going to [52:38] happen so yeah great thank you all for [52:43] volunteering for Theo foro I didn't [52:46] answer my wedding well we could do this [52:48] at your wedding again if you want [52:52] to all right so um what I have here is I [52:56] I took the the um the postmortem notes [53:00] that were put out by Amazon's marketing [53:02] department apparently um and I cut them [53:05] down a little bit to try to get to to [53:07] sort of important [53:10] pieces uh and what we're going to do is [53:12] is interpret them read them to you so [53:13] it's you know kind of like a postmortem [53:15] we understand what happened um but [53:17] augment [53:19] that with some interpretive [53:22] dance all right so I got to get out of [53:25] the way here because I don't want to [53:26] block this um complely unre oh yeah no [53:31] and in in all honesty this is completely [53:33] unrehearsed we talked about it a few of [53:34] us like should we rehearse um but we [53:37] felt that art is best uh [53:40] spontaneous um and actually because we [53:42] actually had no idea what to do to [53:43] rehearse [53:44] so I actually danced for three years of [53:47] college [53:48] ballet fantastic I'm fantastic okay so [53:53] so there will be a role here which was [53:55] sort of a like the primadonna which in [53:56] this case is the um metadata service so [53:59] you as a professional dancer I guess do [54:03] you need a [54:06] poll all right uh great so it was called [54:10] the Dynamo DB event how many people [54:13] actually use Dynamo DV or did use [54:16] it okay all right so on Sunday at 219 [54:21] a.m. PDT there was a brief Network [54:23] disruption that impacted a portion of [54:25] DB's storage servers okay some of you [54:27] need to volunteer to be storage servers [54:29] like say four of you uh and and sort of [54:31] be storage [54:32] servers all [54:34] right fantastic now when this [54:38] happened a portion of the metadata [54:41] service you there uh responses exceeded [54:44] the retrieval and trans transmission [54:46] time allowed by the storage servers so [54:48] storage servers what are you going to do [54:49] about that with the metad service I mean [54:52] the requests are good uh as a result [54:58] some of the storage servers were unable [55:00] to obtain their membership data and [55:02] remove themselves from taking [55:07] requests all right that's good storage [55:09] servers you need to knock some of you [55:10] should remove yourselves but we still [55:12] need storage servers so maybe one or two [55:13] come back [55:16] good multiple you know what you could be [55:19] a multiple simultaneous [55:22] request multiple simultaneous requests [55:24] for these large memberships cause [55:26] processing to slow [55:28] further and eventually exceed the [55:30] allotted time [55:32] limit this resulted in disrupted storage [55:35] servers failing to complete their [55:36] membership renewal becoming unavailable [55:39] for [55:40] requests and amazingly some of them were [55:42] retrying these [55:49] requests with the metadata service now [55:53] under heavy load [56:03] it can no longer respond as quickly to [56:05] storage servers [56:08] uninvolved in the original Network [56:10] disruption were you involved [56:13] earlier no okay we we need to be precise [56:17] with these [56:19] postmortem where was I great unavailable [56:22] servers continue to retry requests for [56:24] membership data maintaining a high Lo on [56:27] the metadata [56:30] service after feveral sailed attempts or [56:33] several failed attempts at adding [56:34] capacity at 5:6 a.m. we decided to pause [56:39] requests to the metadata service that [56:40] would include reducing the [56:46] load this action decreased retri [56:48] activity which relieved much of the load [56:51] on the metadata service okay there got [56:52] to be some load then [56:58] with the metadata service now able to [56:59] respond to administrative [57:06] requests and this is where we need a [57:07] couple of people to help out we were [57:09] able to add significant [57:12] capacity storm the [57:14] stage significant capacity come [57:21] on I guess it do Amazon so it's slow to [57:24] spool up significant capacity [57:26] [Applause] [57:31] okay at 710 dynb was restored and error [57:34] rates low enough for most [57:36] customers I guess they have very low [57:38] standards and AWS Services depend on [57:40] Dynam Dynam DB to resume normal [57:51] operations next year we will talk about [57:54] elastic beant stock [57:55] [Applause] [58:04] we are [58:05] [Applause] [58:19] done for [58:21] Theo [58:23] okay we have our last one and this has [58:26] become another sort of unspoken [58:30] Unwritten tradition because mainly we [58:32] write this stuff down the day of um the [58:34] last talk is going to be by Brian [58:39] cantrol and I think this is actually the [58:42] first talk that I ever seen him do for a [58:44] lightning talk where he actually knew [58:45] what he was going to say before he [58:47] started the talk so Brian canell thank [58:51] you oh it's autoplay you got it back [58:53] that's right yeah not being SED by the [58:54] auto playay um hey how are you I'm so [58:56] just so you know that there is uh what [58:58] gear did not mention is that there is a [58:59] kilon exception to this little time [59:01] limit here but I advisory um but I [59:04] promise I won't bore you as I walk you [59:07] through what I did not name but was [59:09] named in history a crime against common [59:12] sense so this crime against common sense [59:15] sorry I'll I won't Wander from the mic [59:17] this crime against Common Sense begins [59:20] in our implementation of LX branded zone [59:23] so if you didn't see my talk earlier one [59:25] of the crazy things that we've actually [59:27] managed to pull off at joyant and in [59:28] smart OS is the implementation of a [59:31] Linux system call table on top of smart [59:33] OS so that we can actually run Linux [59:35] binaries in smart OS own which has been [59:36] terrific it sounds great it's great it's [59:38] great when it's done okay it is Hell on [59:41] Earth to implement and we have I have we [59:44] have all earned 10,000 years of [59:46] lightning talks in implementing Alex [59:48] branded zones because we have [59:49] implemented Linux in all of of its [59:53] Horrors all of it its semantic atro [59:56] ities have been implemented because we [59:58] can't actually argue with the [59:59] application we have to Simply get it [60:00] working so this is and actually honestly [60:03] the most difficult part of this [60:04] lightning talk was figuring out which [60:06] atrocity I was going to go into depth on [60:08] so and that this is actually quite [60:10] difficult so hopefully I picked the [60:11] right one but if I pick the wrong one [60:13] don't worry we can just get me spooled [60:14] up on something else a little bit later [60:15] so I what I'm going to talk about is a [60:18] an application called Plex i' never even [60:20] run Plex I don't know what it is [60:22] apparently it's something that fellow [60:24] nerds run at home I like said don't know [60:26] it's a video something or another um all [60:28] I know is that it is a it is a tirely uh [60:31] closed binary that um runs on Linux and [60:35] didn't work very well on LX in [60:36] particular it would blow up in very [60:39] strange ways so um this starts actually [60:42] this is in January um this is a chat uh [60:45] transcript with uh Patrick as Poney [60:48] Patrick's in the room I'm not sure [60:49] Patrick is um and so Patrick is looking [60:52] into this LX cordum and one of the [60:53] challenges about implementing LX zones [60:56] implementing a Linux system call table [60:58] is that when you get it wrong the [61:00] application often just dies and it dies [61:02] in some absolutely batshit insane way so [61:06] this thing has died Plex has died in a [61:08] way that makes no sense and we've [61:11] actually gone back to the um the the [61:14] vendor apparently and extracted a binary [61:18] um that's actually compiled with the [61:19] bugging information and but of course uh [61:21] like like most compilers if you compile [61:23] with the bugging information that [61:24] actually means I actually want you to [61:25] generate the worst code humanly possible [61:28] um and Patrick is saying am I crazy or [61:30] is this overly complicated for what it [61:32] accomplishes uh and the the code that [61:35] he's referring to I'm going to show you [61:36] to it in a second but just to continue [61:38] the transcript um Robert says I assume [61:40] this is not an actual assembly thing [61:42] Patrick says I can't say for sure this [61:43] is Plex blowing up in LX but to their [61:45] credit these binaries actually have [61:47] symbols and frame pointers unlike most [61:48] of the we have to deal with it's a [61:50] shangra law says Patrick to [61:53] which Robert says self demerit you may [61:55] wonder what that is Robert is assigning [61:57] himself a demerit because he has been [61:58] trolled by what Patrick has just said um [62:01] and you can see you and in particular [62:03] there's something that's very very [62:04] important here between those two things [62:07] note the time latency so that 2133 says [62:10] it's a shangra La Robert has [62:13] been trolled into rage about the fact [62:15] that there are no symbol tables and [62:17] there are no stack traces and there are [62:18] no frame pointers so most of the things [62:19] we've done has done no productive work [62:21] and has now come in line to say self [62:23] demerit and assigned himself the demerit [62:25] which I didn't I tuned it out here but [62:26] the bot the chat bot reported his number [62:29] of demerits so that's what's going just [62:31] to give you a little bit of backstory [62:32] there what's going on there okay so it [62:33] turns out Patrick is not crazy this is [62:35] the actual gist that he was mentioning [62:36] and I I'm actually relieved that I am [62:39] not the first person to show assembly [62:40] during the lightning talks also not the [62:42] first person to talk about lib Jem Malik [62:44] but I'm not going to reflect it nearly [62:45] as positively as was reflected earlier [62:47] so um lib Jem Malik is I mean I'm I I'm [62:50] not going to say trash fire but I guess [62:52] I just did but the [62:54] um so lib now this is not to be fair I I [62:58] I think this has to be unoptimized Li J [63:00] Malik I think this has to have been [63:02] compiled under duress because nothing [63:04] else explains what's going on here um [63:05] I'm going to move CU this looks [63:07] complicated but I'm walk you through it [63:08] I'm going to move cue this crazy address [63:09] that ends in d80 off the base pointer [63:11] I'm going to load something into Rax got [63:14] it I have loaded this value from memory [63:16] into Rax okay that's fine and now I want [63:19] to take RDX a different register and I [63:21] want to move that into 0x 10 off RX okay [63:24] fine I'm just highlighting there that's [63:26] fine okay we're happy with that now I [63:28] want to move this same other memory [63:30] location back into Rax and this is one [63:33] where like you wish the the [63:34] microprocessor could just stop and be [63:36] like you mean the one you just loaded [63:39] you want that one yeah yeah I want that [63:41] one do you think it's changed no [63:44] probably not can I just give you the [63:46] same thing I just gave you yeah yeah you [63:48] can do that okay here it [63:52] is but of course the microprocessor [63:54] can't just do that the microprocessor [63:55] has to reference silence and it simply [63:56] gives it um but then this gets so much [63:59] worse because then I'm going to I'm [64:00] going to move uh hex 10 off Rax um into [64:03] eax okay fine um and now I'm GNA I'm [64:06] going to move eax into edx it's like [64:09] didn't we just do [64:11] that all right but you know apparently [64:14] these things are not crimes [64:15] microprocessor has to just run it so on [64:17] on we go okay so um it's not Patrick is [64:19] not crazy but Patrick is also frustrated [64:21] we're frustrated we got a bunch of [64:22] random cord dumps we can't make any I [64:24] mean other than being mildly amused [64:26] about terrible assembly we can't [64:27] actually make any forward progress um [64:29] but then some good comes of it in that [64:31] Patrick is so frustrated that he does [64:32] actually notice that we don't have [64:34] support for FS space in the ctoms he [64:35] puts that back um which I thought was [64:37] was pretty great but it was be it meant [64:39] that we couldn't actually make any [64:39] forward progress on the actual problem [64:41] and here's a chat discussion where we're [64:42] talking about um the I love the fact [64:45] that you've added fa space in the Cump [64:47] and Patrick is saying yeah you know I [64:48] actually I can't actually debug it [64:49] though I'm not actually closer to [64:50] knowing what's going on and this and [64:51] this is what I love about chat chat [64:53] transcripts is there are always like [64:55] conversations happening at once so if [64:57] you take a just a a a small subset of [65:00] them one of those conversations will [65:02] come swooping through and you can just [65:04] be kind of amused by it so Robert [65:06] seemingly opopo of nothing says one day [65:09] we'll replace Kimo's iths and it'll be [65:12] glorious like like a fortune cookie that [65:14] fell from the ceiling you're like okay [65:18] meanwhile je Al is crashing but you know [65:19] it's like Robert's got a point I wonder [65:21] may we will replace that and it will [65:22] actually will be glorious um and but [65:24] Patrick had this beautiful beautiful [65:26] theory about what was going on [65:27] unfortunately that theory was [65:28] contradicted by the data I hate it when [65:29] that happens um so it was a beautiful [65:32] Theory it was so nice but it was wrong [65:33] so we're not any closer to KN what [65:35] broken um which is awful um and then um [65:38] sometime later we get a very interesting [65:40] report and actually this I would take a [65:42] quick aside this is a report from [65:43] someone in the community this is [65:45] tremendously helpful to be open source [65:46] because things are people are doing [65:48] different things and one of our users in [65:50] fact one of the users using Plex [65:52] reported that the gem Alec tests the [65:55] unit test tests bail out so we've got [65:58] all these random cord dumps we know that [65:59] gem malic is always on the scene always [66:02] on the scene being always linked into [66:03] the program not necessarily on the scene [66:04] and the fact that the gem Mal unit tests [66:06] can't complete okay that is definitely [66:09] definitely definitely interesting um and [66:11] so if if we can actually because Splunk [66:14] is dying Plex is dying and boy if we can [66:16] do this with the LI the the gem Malik [66:18] tests let's go figure that out first so [66:20] this is a great place to start this's a [66:22] another CH chat uh transcript um one of [66:25] the things that we've also decided [66:26] whenever anything dies randomly with LX [66:29] we're always nervous that our futex [66:30] implementation is wrong what we have [66:32] learned from this is that our futex [66:33] implementation is definitely right um [66:35] but we always blame the poor fexes fexes [66:38] by the way will be the subject of 10,000 [66:39] other lightning talks um unbelievably [66:42] god- awful broken but whatever it's [66:43] implemented correctly here so it's got [66:44] nothing to do with this particular [66:45] problem but so we're going to kick off [66:47] some fex stress tests that'll be fine um [66:49] and meanwhile we kind of dividing our [66:50] work here um and we're going to go [66:52] actually look at these gem malic tests [66:54] and see if we can understand what's [66:56] going on with the galic failure okay [66:57] this sounds good great go super and we [66:59] find a test that dies really [67:01] reproducibly which is great and it dies [67:03] reproducibly it's a pretty simple test [67:04] I'm looking at the system calls that it [67:05] does and there's an M advise with MF [67:08] don't need that it's doing and that's a [67:09] little weird M advise I you want to kind [67:12] of look into this a little bit and so [67:14] let me just remind you about what M [67:15] advise is so M advise is is an old um [67:18] posix routine um these old andics become [67:22] important in a moment and M advis is [67:26] advice it's advice that's the noun that [67:29] is associated with the verb [67:32] advise this is [67:35] foreshadowing and you can give a lot of [67:38] advice but because it's advice it can be [67:42] ignored that's what advice is and that's [67:45] the difference between advice and a [67:51] command this becomes important in just a [67:53] moment [67:56] so this is M advise which says here's a [67:57] memory region and here's a length and [67:59] here's the advice I wish to I wish to [68:01] offer you and the the advice this is [68:04] actually a great thing because what what [68:05] is happening here is the application is [68:07] trying to tell the operating system [68:08] Colonel something that's very difficult [68:09] to infer namely what am I going to do [68:12] it's very hard for the colonel to [68:13] actually predict the future if the [68:14] application can say like hey here's what [68:16] the future is that's great we can [68:17] actually act so M advis it's a great [68:19] idea the idea of providing advice is [68:21] terrific and M advis don't need now [68:23] admittedly um this this slide is Spartan [68:26] for a reason this is all posic says [68:28] about m m don't need which admittedly is [68:31] it's a little on the tur side specifies [68:33] that the application expects that it [68:35] will not access the specified range in [68:38] the near future okay that's a little [68:40] light but fine M don't need I don't need [68:43] it I don't actually need it go back to [68:44] my previous slide just for a moment [68:46] because I'm going to point out that this [68:48] is advice and in particular if if Mr uh [68:51] prompt will go away um it well it's [68:54] covering up what it's saying here but it [68:55] is to have there we go if you look at [68:57] that last line the postic M advis [68:58] function shall have no effect on the [69:01] semantics of access to memory in the [69:02] specified range although it may affect [69:04] the performance of access fine so em MAV [69:06] don't need is saying I don't need it but [69:08] it's still mapped in that's some advice [69:11] that's some advice [69:13] okay this is the Linux man page for MAV [69:16] don't need okay it starts off somewhat [69:20] promising do not expect access in the [69:23] near future now this is like classic man [69:25] page pigeon in that is that is that are [69:28] you're using the imperative voice in [69:30] there are are you telling me that I [69:31] should not expect access in the near [69:33] future or is that what does that denote [69:36] what the flag does um so okay like let's [69:38] just leave aside the the the man page [69:40] pigeon because this actually has many [69:41] more complete sentences than most L [69:43] manand pages so we should give it that [69:45] all right for the time being the applic [69:47] it's like for the time being this is [69:48] also a very strange thing for the time [69:50] being the application is finished with [69:52] the giving range it's like we're talking [69:54] about a tense design disaster here for [69:57] the time being like in the current tense [69:58] what we talk anyway for the time being [70:00] okay the application is finished with a [70:01] given range so the colonel can free [70:04] resources associated with it uh oh [70:06] subsequent accesses to pages in this [70:08] range will succeed but will result [70:11] either in reloading of the memory [70:13] contents from the underlying map file or [70:16] zero filled on demand Pages for mappings [70:19] without an underlying file boy does that [70:22] not sound like advice [70:26] so what this is saying is that when you [70:28] say MAV don't need I'm actually [70:31] saying mmap that's the same thing is [70:33] saying like I because if you're done [70:35] with it and you don't need it again and [70:36] we need it again it can be B zeroed then [70:38] why don't you unmap it yeah that's not [70:40] advice unmap it that's a command that's [70:42] a system call but no no okay so this is [70:44] a very strange and this is kind of the [70:46] like okay please let this be a bug in [70:48] the man page please this it can't be [70:49] this dumb it can't be this dumb it can't [70:51] be this dumb it oh no no no so this is a [70:56] very simple program that demonstrates [70:57] what I'm talking about so we're going to [70:58] mmap something we're going to map it [71:00] private map a non going to set it to a [71:02] value 123 I'm going to give you the [71:04] advice that I don't need it and I go [71:05] back to to read it again and it's zero [71:08] now this is not what MAV don't need is [71:09] supposed to do what MAV don't need does [71:12] is it says I actually don't need this [71:14] and if the system needs it it can take [71:16] it map don't need actually does violate [71:18] the even the posi notion of semantics if [71:20] the system needs it it can take it but I [71:22] have no right to expect that it is zero [71:24] when I go back to it when if I go back [71:26] to read it if the system needed it it [71:28] would be zero filled if not it won't be [71:30] it would be the old thing that I stored [71:31] I can't rely on the fact that it's zero [71:33] which it turns out is exactly what jeem [71:34] malok is relying on and and the reason [71:37] this is so unconscionable unconscionable [71:40] unconscionable semantics is that the way [71:42] MAV don't need is implemented is [71:44] actually very clever what MAV don't need [71:46] does is it takes this page and it just [71:48] puts it on the list of pages that can be [71:50] reclaimed by the system it doesn't have [71:52] to do anything else whereas MB face palm [71:56] over here has to actually go through the [71:59] memory system and actually effectively [72:01] actually free the page get rid of the [72:03] page it's a lot more work so it's stupid [72:07] why would anyone need this in fact it's [72:09] so stupid that you want to know you went [72:11] I went Googling for it to be like okay [72:14] am I the only one upset by this this [72:16] happens a lot I'm like please tell me [72:18] someone else is enraged by it so you [72:19] Google mvis MAV don't need which I [72:21] encourage you to do it's pretty funny [72:22] for Linux so you you Google Med medis [72:25] MAV don't need Linux and you find all [72:26] sorts of things about how broken the [72:27] semantics are you also find this [72:29] question which is the top hit on stack [72:30] Overflow does m advise MAV don't need [72:34] instruct the OS to lazily write to disk [72:38] it's like oh God you know what you've [72:40] been told questioner your entire life [72:43] that there's no such thing as a stupid [72:45] question but that's a lie that smart [72:48] people tell you there is such a thing as [72:51] a stupid question and this is one of [72:53] them you need to have nothing to do with [72:56] words like memory data write dis OS you [73:00] need to you need and I'm not saying that [73:02] you don't have the right to live a full [73:04] life and to raise kids and to be happy [73:06] in this universe but you may not call [73:08] MAV don't need I'm sorry because no it [73:11] doesn't do any of these things you this [73:12] is so wrong like like leave aside my [73:15] little like rant on stage about the [73:17] difference in some inside baseball [73:18] semantic difference between Linux and [73:20] and other operating systems this has [73:22] just got nothing to do with what you [73:23] think it has to do with and you read [73:24] this like and you're like oh God what's [73:25] the answer to be oh God please let the [73:26] answer be oh thank God oh one of those [73:29] one-word stack Overflow answers like no [73:32] no no no for your own good stay away [73:36] from map donate actually for your own [73:37] good stay away from map actually [73:39] operating systems computers stay away [73:40] from the whole lot maybe you can move to [73:42] a compound in Idaho I know someone [73:44] looking for a roommate out there but the [73:46] so the and actually goes on to explain [73:48] that Linux will not take this as a hint [73:50] to throw Pages away after writing the [73:51] back but to throw them away [73:53] immediately this is not considered a bug [73:56] but a deliberate decision and finally [73:58] I'm finding my people I you can just you [74:01] can feel the simmering anger in there um [74:04] and it's like God bless you Damon you [74:06] may even be in this room Damon whoever [74:08] you are because you were clearly upset [74:10] about this and goes on to explain the [74:12] irony and about how actually there was [74:15] something to go do this but they don't [74:16] do that um and you will lose data of [74:19] used M don't need so don't do this [74:21] essentially so then you keep looking for [74:23] this because you really want to know who [74:25] is to blame here and do the Linux curl [74:27] folks appreciate how broken this is and [74:30] you find that there are many posts like [74:32] this but I like this one wrong madis MAV [74:35] don't need semantics this is in 2005 so [74:38] digging back a little bit to history [74:39] where did this come from and this person [74:41] says hi there's something wrong with the [74:43] current mvis map don't need both the man [74:45] page and the source code say map don't [74:46] need means the application does not care [74:48] about the data so might be thrown away [74:49] but that's not what Pik says it I'm not [74:51] sure if they screamed it when they wrote [74:53] it but I'm screaming it now um and so [74:55] this person goes on to say you know [74:57] every other Opera syst does and it [74:58] actually points out that Solaris has M [75:00] free and I know that Solaris I know it's [75:02] the SW word on Linux kernel but solar [75:04] has something called MB free MB free [75:06] does exactly what you want it to do so [75:08] MB free says actually I'm done with this [75:10] and you can actually go free you can [75:12] actually invalidate it so we actually [75:13] have something to go do that called MB [75:15] free like why don't you use MB free [75:17] jerks um but we we maybe we should add [75:20] MB free and we could change the [75:22] semantics of M don't need to be the [75:23] proper semantics and by the way the man [75:25] page should be corrected too it's like [75:26] Yeah The Man page should be corrected [75:27] absolutely we started on that okay so so [75:31] so um Robert La noell says I like the [75:33] idea of renaming the current MAV don't [75:35] need to MAV free okay that's good and [75:37] then adding a correct MAV don't need [75:39] glad Everyone likes this in 2005 what [75:42] what the hell hats 2015 have we not [75:44] fixed this and as I said the man page [75:46] needs clarification like okay all right [75:48] everyone's wait in okay that's like so [75:50] can we go please make this happen no [75:52] apparently we can't um and then this [75:55] response and God love year I don't know [75:57] who you are because this I when I read [76:00] this response I was still a little angry [76:02] and he responded in my opinion this is [76:04] still a crime against common sense it's [76:07] like God bless you Ur it is a crime [76:10] against common sense and so and the man [76:13] page needs clarification definitely says [76:15] you're an absolutely needs clarification [76:17] where did this come from and I so didn't [76:18] mean to give it away where did this come [76:20] from who did this who is to blame no [76:23] other operating system does it this way [76:25] right [76:26] wrong true [76:28] 64 [76:31] what what seriously what no honestly [76:34] like what seriously true 64 what all [76:37] right so this is a this is a mail from [76:38] August 20th 1999 and now we're getting [76:41] to where this came from and sap was [76:43] doing what people do at sap and what [76:45] people were doing all over at the time [76:47] and they were getting their app their [76:48] Unix app to work on Linux and one of the [76:50] things they were missing was MAV don't [76:52] need and poor Hans or Kristoff I poor [76:55] Kristoff did not mean to do this but [76:57] Kristoff accidentally referred to True [77:00] 64's implementation so we need something [77:02] like A's disclaim or true 64's M advise [77:05] wow a v true 64 in a semantic duel like [77:10] which one you going to root for there in [77:11] this case the right decision was a my [77:14] friends [77:16] um go figure sometimes it is the right [77:18] answer um so we he's saying like listen [77:22] this is what we need to go you should [77:23] add MAV MAV uh don't need and he says M [77:26] advise is known in many systems but they [77:28] do not agree about the meaning of MAV [77:30] don't need that's putting [77:30] euphemistically actually they do agree [77:31] this true 64 is doing something else [77:33] most of them only used as a hint to page [77:35] this area out perhaps a flag MB free or [77:38] M disclaim would be better uh disclaim [77:40] is a specific I know this all right what [77:41] are the comments here and the uh the the [77:46] response to this is um someone says well [77:48] wait a minute MAV don't need it seems [77:49] like true 64 has the wrong semantics and [77:53] although I like m andv [77:55] um I must admit this interpretation is [77:56] really odd misuse of the M advised [77:59] function which is ined for Speed [78:01] optimizations only but to remove the [78:02] mappings of any spanic at all and he [78:04] says yes I think you were right the true [78:06] 64 people misinterpreted the call okay [78:10] so because some poor true 64 engineer [78:13] misread the call and that fire managed [78:16] to LEAP the fire line into Linux because [78:19] Linux implemented it this way anyway [78:22] it's like oh momy and so that's what [78:26] happened it came from True 64 true 64 is [78:29] what crashed Plex on LX branded zones [78:33] um all right so what about M MB free can [78:35] we add MB free well yes we can actually [78:37] add MB free to Linux and this is [78:39] something that this so this was finally [78:41] suggested in 2014 it's so fast forward [78:44] from 1999 maybe we should support m [78:47] advis m free and then they would did [78:48] some benchmarking with li J hey turns [78:50] out Li Jal is a lot faster when we do it [78:52] this way it's like yes you dummies if [78:54] You' want it properly in the first place [78:55] you wouldn't have this problem so of [78:57] course like okay great it's done well no [78:59] it's not done it was done it hasn't been [79:00] integrated anywhere it's 2015 this a [79:03] year and a half ago but Linux moves so [79:04] fast man it's like yeah right okay so [79:07] keep waiting for MB free um but don't [79:10] worry it's been fixed in the man [79:15] page we will drop the sentence that the [79:18] colonel May ignore advice so in the man [79:22] page it used to say the colonel is free [79:25] to ignore the advice because it's [79:26] advice and the bug is determined [79:29] to be well the sentence is the bug we [79:31] should eliminate that oh Mom anyway that [79:35] is there I fixed it m advise it has now [79:38] been fixed for everyone running Linux [79:39] there you go thank you