Porting KVM to SmartOS - Brian Cantrill, Joyent, KVM Forum 2011

将 KVM 移植至 SmartOS - Brian Cantrill，Joyent，2011 年 KVM 论坛 为了填补 illumos 在硬件辅助虚拟化领域的空白，Joyent 将 Linux 的 KVM 子系统移植到了 SmartOS，同时保持了与 QEMU 的用户态兼容性。工程团队克服了早期引导和驱动转换方面的复杂挑战，包括在 MDB 中实现 16 位反汇编、利用 DTrace 检测 VM exit，以及排查线程迁移身份互换和内存复制期间 FPU 寄存器损坏等严重缺陷。Joyent 特意锁定了客户机内存，将不可信的 QEMU 进程封装在非特权的 illumos Zone 中以实现纵深防御，并借助 ZFS 与 Crossbow 提供高扩展性的 I/O 与网络支持。最终的整合方案通过虚拟化监控器级别的 DTrace 探针和实时可视化热图，实现了对客户机调度与中断的高频、客户机外部的可观测性。 Joyent 将 Linux 的 KVM 子系统移植到了 SmartOS（一种 illumos/OpenSolaris 衍生系统），将硬件辅助虚拟化与 ZFS、DTrace、Zone 和 Crossbow 网络虚拟化等原生企业级特性融为一体。 (0:01) 该移植项目将范围严格限定在支持带 EPT 功能的 Intel VT-x 的 64 位 x86 主机上，并严格锁定客户机物理内存，以消除内存超售和换页带来的不可预测延迟。 (4:13) 针对 illumos 无法通过 ioctl 返回匿名文件描述符的问题，Joyent 引入了 KVM_CLONE ioctl，将针对上游 QEMU 的代码修改控制在约十行左右。 (26:30) Joyent 通过在各自独立的精简版 illumos Zone 中运行每个用户态 QEMU 进程来加固部署模型，将虚拟机逃逸漏洞限制在非特权容器内予以化解。 (30:08)

Joyent engineered a port of the KVM hypervisor to SmartOS—an illumos-based operating system—predicated on the belief that a mature, general-purpose operating system provides a superior hypervisor foundation compared to thin hypervisors. By integrating KVM with native illumos technologies including ZFS, DTrace, Zones, and Crossbow, Joyent achieved bare-metal virtualization performance while adding robust multi-tenant containment and deep host-side observability. Hypervisor Philosophy and Project Scope (2:45) Joyent rejected thin hypervisors, arguing that building a hypervisor outside an established operating system repeats two decades of operating system history. Because illumos already possessed enterprise storage (ZFS), dynamic tracing (DTrace), OS-level virtualization (Zones), and network virtualization (Crossbow), porting KVM supplied the missing hardware-assisted virtualization capability needed to run foreign operating systems. To keep the engineering effort tractable, Joyent narrowly scoped the implementation. The host was restricted to 64-bit x86, guest support was limited to 32-bit and 64-bit x86, and CPU virtualization was focused exclusively on Intel VT-x with Extended Page Tables (EPT), deferring AMD SVM. Crucially, the team avoided emulating Linux kernel internals—a pitfall of prior attempts on other platforms—choosing instead a clean internal divergence that preserved compatibility with the upstream userland QEMU/KVM kernel ABI. Debugging Tooling and Kernel Traps (6:04) Early porting relied on compiling KVM code wrapped with preprocessor flags, inserting DTrace probes that triggered on unported branches to log source files and line numbers dynamically. When early guest execution hung in real-mode BIOS code, Joyent enhanced its Modular Debugger (MDB) to map guest physical frame numbers to QEMU virtual addresses and implemented 16-bit disassembly. Coupled with DTrace probes inspecting the Virtual Machine Control Structure (VMCS) via a new `vmregs` variable, developers uncovered and corrected low-level programmed I/O failures, progressing through boot milestones from the illumos kernel debugger (`kmdb`) to Linux, Windows, and Haiku. Cantrill reported that KVM itself exhibited exceptional code quality, with only a minor upstream QEMU defect found regarding `signal()` versus `sigaction()` usage on `SIGIPI`. However, native integration provoked severe self-inflicted bugs. One failure involved incomplete restoration of the `GS_BASE` register during thread migration, leading two physical CPUs to misidentify their own identities under load. Another bug occurred when an illumos FPU save helper cleared the Task Switched (`TS`) flag in `CR0`, preventing Device Not Available (`#NM`) faults; as a result, subsequent memory operations like `memcpy` ran against stale floating-point registers, causing silent memory corruption. Benchmark Parity and Multi-Tenancy Architecture (18:02) CPU benchmark testing initially showed SmartOS KVM running an anomalous 5% faster than Linux KVM on identical hardware. Investigation revealed this was an artifact of Intel Turbo Boost dynamic overclocking reacting to ambient temperature while the Time Stamp Counter remained static; disabling Turbo Boost established identical performance parity across bare metal, Linux KVM, and SmartOS KVM on compute and Sysbench MySQL workloads. Joyent architected its virtualization strategy around multi-tenant cloud economics rather than synthetic consolidation. The implementation locks down guest physical memory entirely and rejects memory overselling, avoiding the high tail latencies caused by host swapping during active resource contention. Where ultra-high density is necessary, Joyent deploys illumos Zones—packing up to 400 active tenant containers onto a single 96 GB machine—reserving KVM exclusively for workloads requiring non-native kernels. For similar operational reasons, Kernel Samepage Merging (KSM) and nested virtualization were omitted as unnecessary or too complex for production hypervisors. Illumos Primitives: Security, Storage, and Tracing (26:30) Bridging illumos to QEMU required addressing the absence of Linux's `anon_inode_getfd` inside driver ioctls. Joyent resolved this by introducing a targeted `KVM_CLONE` ioctl, restricting QEMU code changes to roughly ten lines. vCPU scheduling hooks were mapped directly to illumos context operations (`ctxops`), while KVM timer events were wired to the illumos high-resolution `cyclic` subsystem. The resulting stack derives major operational advantages from illumos subsystems. Storage relies directly on ZFS volumes (`zvols`), replacing userland image formats like QCOW2 with native Adaptive Replacement Cache (ARC) DRAM buffering and instant snapshots. For security, each QEMU process executes isolated within a stripped, unprivileged container (Zone), ensuring that hypervisor escapes break into a restricted sandbox rather than the host kernel. Crossbow virtual NICs (VNICs) enforce rate limiting and anti-spoofing without complex host firewall rules. Finally, replacing KVM tracepoints with DTrace SDT probes enables low-overhead introspection, allowing operators to monitor guest scheduling and interrupt behavior simply by sampling the guest's `%cr3` MMU register from the host. Joyent released its KVM port and lightly modified QEMU branch on GitHub as production-grade software for Intel systems, confirmed when the community booted Windows 7 within 45 minutes of release. Joyent proposed broader collaboration with the KVM community around shared performance diagnostics, automated oddball operating system testing harnesses, and standardized benchmarking, while leaving AMD SVM support and memory overselling to external contributors.

Joyent 工程副总裁 Brian Cantrill 阐述了将 Linux KVM 移植至 SmartOS 的工程实践，主张通用操作系统内核是构建虚拟机监控程序的最佳载体。Joyent 将硬件辅助虚拟化与 illumos 既有的 ZFS 存储、DTrace 动态追踪、Zones 容器及 Crossbow 网络虚拟化结合，旨在构建高密度且高弹性的多租户云计算架构。 架构哲学与移植边界的明确界定 (2:45) Cantrill 提出，构建专用“精简型虚拟机监控程序”（thin hypervisor）是倒退并重复过去 20 年操作系统发展史的错误路径，而成熟的通用操作系统内核才是虚拟化最理想的基石。鉴于 KVM 验证了这一哲学且拥有优异的性能表现，Joyent 选择将其移植至 SmartOS（源自 OpenSolaris 的开源分支 illumos），而非在 illumos 内模拟 Linux 运行环境。为确保工程周期可控，Joyent 严格划定了移植范围：仅支持 64 位 x86 主机平台与 x86/x86_64 客户机，且初期仅聚焦配备扩展页表（EPT）的 Intel VT-x 硬件，暂不实现 AMD SVM 支持。 在接口设计上，团队坚持严格兼容上游 QEMU/KVM 内核 ABI。由于 illumos 内核不允许在设备驱动的 ioctl 处理流程中直接创建并返回匿名文件描述符（Linux 依赖 anon_inode_getfd 实现 vCPU 映射），Joyent 避免了修改大量上游调用的做法，转而引入轻量级的 KVM_CLONE ioctl，仅通过在 QEMU 中增加约十行适配代码便干净地解决了 vCPU 的派生与绑定问题。 引导障碍突破与内核级自损缺陷排查 (6:04) KVM 深度绑定 Linux 内核原语，初期由工程师 Max Bruning 独立攻关代码编译。团队通过 #ifdef XXX 屏蔽尚未移植的代码，并在 #else 分支中植入携带文件名与行号的自定义 DTrace 探针，使未移植依赖在执行触碰时即刻暴露。当虚拟机卡死在实模式 BIOS 时，团队为 MDB 调试器新增了 16 位反汇编功能以及物理页框（PFN）至 QEMU 虚拟地址的映射支持，并利用 DTrace 的 vmregs 变量实时读取虚拟机控制结构（VMCS）硬件寄存器，从而排查并修复了程序化 I/O（PIO）错误，成功让引导流程推进至加载 kmdb、Linux、Windows 以及系统行为特殊的 Haiku OS。 在 Linux 引导攻坚阶段，移植团队排查了两个严重的自损性底层缺陷：一是内核线程跨物理核心迁移时未正确恢复基址寄存器 GS_BASE，导致两个核心在软件标识层面发生身份互换，触发内核断言崩溃；二是移植 FPU 保存例程时，illumos 辅助函数在调用 FXSAVE 后顺带清除了控制寄存器 CR0 中的任务切换位（TS），导致后续浮点指令无法触发“设备不可用”（#NM）中断以惰性恢复寄存器上下文，致使依赖 SSE 指令的高性能 memcpy 相互踩踏数据，造成编译器输出的二进制文件静默损坏。团队最终依托 DTrace 崩溃转储分析彻底定位并清除了这两项隐患。 企业级原生技术整合与容器隔离安全 (21:47) 立足于服务数万客户的多租户生产环境，Joyent 在架构设计中坚决舍弃了内存超卖（Memory Oversell）机制，要求客户机物理内存在初始化阶段全部锁定。Cantrill 认为，在活动租户承载重负载时，内存分页与交换引入的不可预测长尾延迟会彻底摧毁服务可用性；若追求极致多租户密度（如单台 96 GB 内存的 2U 服务器承载 400 个活跃租户），应使用操作系统级容器（Zones），KVM 则专门负责承载异构操作系统内核。为应对多租户安全威胁，SmartOS 将每个运行中的 QEMU 进程完全拘禁在权限受限的独立 Zone 容器中。即便黑客利用漏洞突破 QEMU 虚机边界，攻击者也仅能落入毫无特权的容器沙箱内部，无法直接触碰宿主机内核。 在存储与网络层面，SmartOS 将虚拟磁盘直连至 ZFS 卷（zvol），完全规避了 QCOW2 用户态文件系统的开销与复杂性，赋予虚机即时快照、常数级克隆、自适应替换缓存（ARC）加速以及基于 zfs send/receive 的广域网迁移能力。网络方面，系统通过 Crossbow 虚拟网络接口（VNIC）接入 VirtIO，在免去 Linux 复杂 iptables/ebtables 配置的同时，直接在内核层提供硬件级 MAC 地址防欺骗、确定性带宽切片与细粒度流量监控。 性能对齐验证与无代理调度深度观测 (18:02) 在基准测试中，SmartOS KVM 在纯计算负载上与裸机及 Linux KVM 展现出绝对的性能对齐。针对早期测试中 SmartOS KVM 比 Linux KVM 稳定快 5% 的反常现象，排查确认是 Intel Nehalem 架构的 Turbo Boost 技术因环境温度波动而动态超频所致；当关闭动态超频后，Sysbench MySQL 测试证明两者性能完全一致。在观测层面，移植版本将 Linux 内部的虚机统计数据接入 illumos 的 kstat 框架，团队据此开发的 kvmstat 工具不仅能够每秒打印 exit 计数，还借此查明了初次启动虚机时每秒 1,300 次 VM-exit 是由 5 分钟控制台黑屏超时引发的 EPT 违规刷新造成的。 此外，SmartOS 激活了 QEMU 中原本为 SystemTap 预留的 DTrace 探针，并将 KVM 的 trace 接口全面替换为零开销的 SDT 探针。通过以 999 Hz 的频率采样客户机的 CR3 寄存器（页表基址），宿主机无需在虚机内安装任何监控代理，即可通过 CR3 的数值变动透视客户机操作系统的内部进程调度逻辑与 nice 优先级调节效果，并绘制出微秒级的中断时序与文件系统块写入热力图。 Cantrill 总结指出，SmartOS KVM 移植在 Intel VT-x 平台上已具备生产可用性并在 GitHub 开源，但目前完全缺乏对 AMD SVM 的支持，且出于复杂度考虑明确放弃了 Intel 嵌套虚拟化及内存页面去重（KSM）。Joyent 提议与上游 KVM 社区在底层性能诊断上展开协作，并建议引入 Haiku 和 IBM OS/2 等严苛的边缘操作系统建立全自动化测试矩阵，以验证虚拟化架构的正确性。

## Background 该视频由 Red Hat 于 2011 年 9 月 26 日在 YouTube 平台发布。视频记录了 Joyent 工程副总裁 Brian Cantrill 在加拿大温哥华举行的 KVM Forum 2011 上的主题演讲，内容围绕将 KVM 移植到基于 illumos 的开源操作系统 SmartOS 的历程，涵盖架构继承、工程限制、遇到的障碍以及社区创新。 ## TL;DR Brian Cantrill 深入复盘了 Joyent 团队将 KVM 内核模块及 QEMU 移植至 SmartOS 的工程实践。为了填补 SmartOS 在硬件辅助虚拟化上的空白，团队在严格界定 x86-64 与 Intel VT-x 范围的前提下，依托 DTrace 打桩和 MDB 调试器扩展攻克了引导难题，并排查了 GS Base 错乱与 FPU 状态覆盖等自伤型底层缺陷。演讲还剖析了拒绝内存超卖的设计取舍，展示了结合 ZFS 存储、Zone 容器沙箱与 VNIC 网络虚拟化的架构优势，并通过 DTrace 热力图展示了对客户机调度与中断向量的深度黑盒可观测性。 ## Atmosphere & style 演讲充满极具感染力的极客幽默与犀利吐槽，Cantrill 语言极快、台风狂放且充满表现力，现场笑声与掌声不断。他毫无包袱地调侃前东家收购方、揭露团队在排查诡异崩溃时“濒临绝望”的真实恐慌，将硬核冰冷的底层内核调试化作扣人心弦的技术历险。整场分享兼具深厚的系统底蕴与坦诚的开源情怀，既有对 KVM 坚如磐石品质的崇高致敬，也有对极致黑盒可观测性成果的由衷狂热。 ## Bulletin - [00:01](https://www.youtube.com/watch?v=cwAfJywzk8o&t=1s) Joyent 成功将 Linux KVM 内核模块移植至基于 illumos 的开源操作系统 SmartOS，重点面向具备 EPT 的 Intel VT-x 架构，并保持与 QEMU-KVM 用户态 ABI 的最大兼容。 - [13:11](https://www.youtube.com/watch?v=cwAfJywzk8o&t=791s) 移植团队在排查中未发现任何原生 KVM 内核代码缺陷，仅修复了一个 QEMU 在信号处理上误用 signal() 的缺陷，其核心实现表现出极高的工业级稳定性。 - [18:56](https://www.youtube.com/watch?v=cwAfJywzk8o&t=1136s) 基准测试初期 SmartOS KVM 比 Linux KVM 异常快 5%，最终由 Brendan Gregg 证实系 Intel 睿频加速（Turbo Boost）受温度波动干扰所致；关闭该特性后两者在 Sysbench 测试中性能曲线完全重合。 - [21:47](https://www.youtube.com/watch?v=cwAfJywzk8o&t=1307s) SmartOS 坚持客户机物理内存完全锁定（Locked down）并放弃内存超卖与 KSM 内存去重，将高密度多租户场景交由 illumos Zones（容器）承担，以确保生产环境的低延迟与高确定性。 - [28:47](https://www.youtube.com/watch?v=cwAfJywzk8o&t=1727s) 系统结合 ZFS 实现了常数时间克隆与统一 ARC 读缓存加速，同时将每个 QEMU 进程运行在独立的 Zone 沙箱容器内，有效规避了虚拟机逃逸漏洞带来的宿主机安全风险。 ## Key takeaways - [00:01](https://www.youtube.com/watch?v=cwAfJywzk8o&t=1s) SmartOS 继承了 illumos 的 ZFS、DTrace、Zones 操作系统虚拟化和 Crossbow 网络虚拟化，引入 KVM 是为了补齐硬件级虚拟化的拼图。 - [02:45](https://www.youtube.com/watch?v=cwAfJywzk8o&t=165s) Joyent 坚信最好的虚拟机监控器就是操作系统本身，并认为专有瘦虚拟机监控器（Thin Hypervisor）是在重复走操作系统走过的弯路。 - [06:04](https://www.youtube.com/watch?v=cwAfJywzk8o&t=364s) 初期代码移植采用“#ifdef XXX 打桩结合 DTrace 探测”策略，在未实现代码路径被触发时精准上报源码行号，极大辅助了串行开发阶段的排错。 - [09:38](https://www.youtube.com/watch?v=cwAfJywzk8o&t=578s) 团队扩展了 MDB 调试器，不仅支持将客户机物理页帧（Guest PFN）映射到 QEMU 虚拟地址，还破天荒实现了 16 位实模式反汇编以解决 BIOS 引导死循环。 - [09:38](https://www.youtube.com/watch?v=cwAfJywzk8o&t=578s) 为支持黑盒级观测，SmartOS 为 DTrace 引入了内置变量 vmregs，允许工程师直接从 Intel VMCS 结构中动态抓取客户机寄存器状态。 - [14:15](https://www.youtube.com/watch?v=cwAfJywzk8o&t=855s) 最诡异的自伤型 Bug 发生在线程迁移时未恢复 per-CPU GS Base，导致两个物理 CPU 发生“认知错乱”并认为自己是对方，最终靠 DTrace 死后转储分析破译。 - [16:37](https://www.youtube.com/watch?v=cwAfJywzk8o&t=997s) 移植代码在调用 Illumos FPU 保存函数时误清除了 cr0.TS 位，导致后续 memcpy 无法触发 #NM 异常而直接使用脏寄存器数据，引发随机编译二进制损坏。 - [25:01](https://www.youtube.com/watch?v=cwAfJywzk8o&t=1501s) 由于 illumos 缺少 Linux 的 anon_inode_getfd 机制，移植团队引入了全新的 KVM_CLONE ioctl，以仅仅 10 行 QEMU 修改保留了 vCPU 与文件描述符 1:1 的对应关系。 - [27:13](https://www.youtube.com/watch?v=cwAfJywzk8o&t=1633s) illumos 早年为 SPARC 处理器开发的上下文钩子（Context Ops）被复用于对应 Linux 的 kvm_sched_in/out，而高精度定时器（Cyclic）则负责承载 KVM 单次触发重编程。 - [28:47](https://www.youtube.com/watch?v=cwAfJywzk8o&t=1727s) 每一个 QEMU 用户态进程都被隔离在独立的 Zone 容器中，攻击者即使成功实现虚拟机逃逸，也仅能困在无特权的沙箱环境中，大幅缩减了攻击面。 - [31:14](https://www.youtube.com/watch?v=cwAfJywzk8o&t=1874s) 基于 Crossbow 的虚拟网卡（VNIC）原生支持 MAC 地址防伪造与严格的流量整形带宽控制，同时通过 kvmstat 工具将 vCPU 退出统计实时暴露至 kstat。 - [35:46](https://www.youtube.com/watch?v=cwAfJywzk8o&t=2146s) 宿主机通过 999Hz 采样客户机 CR3 寄存器（MMU 上下文），在无需入侵虚拟机内部的前提下，凭借 DTrace 成功可视化了 Linux 调度器在两个计算任务间的切换轨迹。 - [38:18](https://www.youtube.com/watch?v=cwAfJywzk8o&t=2298s) Cantrill 呼吁未来与社区在 KVM 性能优化、构建冷门系统（如 IBM OS/2）测试农场以及客观基准测试方面展开深入协作。 ## Detailed walkthrough ### [00:01](https://www.youtube.com/watch?v=cwAfJywzk8o&t=1s) 什么是 SmartOS 与 illumos 架构底座 Brian Cantrill 以 Joyent 工程副总裁的身份登台，分享将 KVM 移植到 SmartOS 的技术历程。面对台下可能产生的疑惑，他首先解释了 SmartOS 的来历：它是基于 illumos 的衍生操作系统。illumos 则是从 OpenSolaris 分叉出来的真正开源分支。谈及 illumos 的成立背景，Cantrill 不忘幽默地挖苦前东家的收购方，指出甲骨文当时似乎一心想把员工逼走、蓄意破坏操作系统生态，而这种行为客观上极大地推动了开源运动，“因此再次感谢 Larry（埃里森）”。 作为一个完全独立的开源系统分支，SmartOS 继承了一整套成熟的企业级关键特性： - **ZFS**：最初为 Solaris 设计、后被移植到 FreeBSD 和 macOS 的企业级文件系统。针对“企业级”的定义，Cantrill 直言：“企业级意味着我们在痛苦的熔炉里千锤百炼才把这该死的东西做对。”存储生态系统各层的底层行为极其复杂不可测（Byzantine），唯有经过“血泪之路”（Trail of Tears）的长期打磨才能真正成熟。 - **DTrace**：用于系统动态 ad-hoc 追踪与性能分析的核心工具。 - **基于操作系统的虚拟化（OS-based Virtualization）**：类似于 FreeBSD Jail 注入了“生长激素”，是操作系统内的一等公民，既能以原生硬件速度执行，又具备强隔离边界，能够大幅提升多租户密度。 - **网络虚拟化（Crossbow 项目）**：提供虚拟网卡（vNIC）基础设施，支持细粒度的带宽管理和资源控制。 ```mermaid flowchart TD accTitle: SmartOS技术栈与软硬件虚拟化结构 accDescr: 展示SmartOS基于illumos继承的核心特性，以及引入KVM填补硬件虚拟化空白的架构关系。 subgraph Heritage["SmartOS 继承自 illumos 的核心特性"] ZFS["企业级文件系统 (ZFS)"] DTrace["动态追踪工具 (DTrace)"] OSVirt["操作系统级虚拟化 (Zones)"] NetVirt["网络虚拟化 (Crossbow vNIC)"] end subgraph Missing["缺失拼图与补充"] MissingPiece["缺少硬件级虚拟化"] IntelAMD["Intel VT-x / AMD SVM 硬件支持"] KVMPort["移植 KVM 核心驱动"] end Heritage --> MissingPiece IntelAMD --> KVMPort MissingPiece --> KVMPort KVMPort --> UnifiedOS["统一操作系统底座 (SmartOS)"] ``` --- ### [02:45](https://www.youtube.com/watch?v=cwAfJywzk8o&t=165s) 虚拟化哲学与移植边界的确立 尽管拥有丰富的功能组件，SmartOS 过去唯独缺乏硬件级虚拟化。随着 Intel 和 AMD 在硬件辅助虚拟化上的演进，实现硬件级虚拟化的门槛和开销大大降低。Cantrill 强调了 Joyent 以及 illumos 社区坚守的系统设计哲学： > “我们坚决认为——我甚至敢说 illumos 社区也坚决认为——最好的虚拟机监控器就是操作系统本身。” 在他看来，任何试图构建所谓“瘦虚拟机监控器（thin hypervisor）”的尝试（Cantrill 隐晦表示不点具体专有厂商的名，但大家都心知肚明），本质上都是在把操作系统过去 20 年的发展历史重新踩一遍坑。KVM 不仅共享并率先践行了这一设计理念，而且在基准测试（如 SPEC）中展现出匹敌甚至超越专有方案的最佳性能，并拥有庞大的开源社区。为此，Joyent 决定将 KVM 移植到 SmartOS。 为了保证工程切实可行，团队在立项之初就严格界定了移植范围与约束条件： - **CPU 架构**：业务与资源上优先且仅聚焦于带 EPT（扩展页表）的 **Intel VT-x**。尽管 Joyent 是一家接受 Intel 投资的公司，但 Cantrill 澄清这并非来自安迪·格鲁夫（Andy Grove）的施压，纯粹出于现实工程权衡；同时代码架构上避免封死未来对 AMD SVM 的支持路径。 - **宿主机环境**：明确只支持 **x86-64** 宿主机（仅在考虑类似 Intel Atom 这类低功耗平台时可能才会兼顾 32 位 x86 宿主）。 - **客机环境**：严格限定为 **x86 与 x86-64 客机**。Cantrill 感慨在 Sun 工作了 14 年，看到 RISC 架构或 Power7 架构时难免怀旧伤感，但商业现实必须聚焦于 x86。 - **接口契约**：允许内核具体实现背离 Linux 内部机制以适应 illumos 的内核结构和编码规范，但坚决与 **QEMU-KVM** 之间的用户态接口（ABI/ioctl）保持最大程度兼容。Cantrill 认为，诸如 kvmtool 等项目正在尝试基于该接口构建全新用户态，这非常有助于巩固该 ABI 层的稳固性。 --- ### [06:04](https://www.youtube.com/watch?v=cwAfJywzk8o&t=364s) 编译破冰与基于 DTrace 的打桩追踪法 KVM 在设计之初完全是面向 Linux 内核原生设施的，并未考虑跨平台可移植性。Cantrill 指出，直接在代码中通过垫片去仿真 Linux 内核功能根本行不通（FreeBSD 在 2007 年 Google Summer of Code 项目中曾尝试这一路线），两者的内核内部机制差异实在过大。不过，KVM 2.6.34 展现出了极高的工业级稳定性，Joyent 坚信可以在分叉该实现的同时，维持向上游贡献与拉取补丁的能力。 由于整个移植在编译初期完全无法通过，且依赖关系高度串行化，团队无法直接投入大兵团作战。Joyent 工程师 Max Bruning 于去年深秋独自深入代码前线展开攻坚。为了尽快让代码通过编译，Bruning 采取了大胆的打桩策略： ```c #ifdef XXX /* 尚未移植的 Linux 原生代码块 */ #else DTRACE_PROBE2(kvm_xxx_hit, __FILE__, __LINE__); #endif ``` Cantrill 调侃 `#ifdef XXX` 这种打桩方式大概可以追溯到 IBM 1401 时代。当大批未移植代码块被排除后，为了搞清楚运行逻辑到底会踩到哪些未实现的路径，他们在 `#else` 分支中插入了特殊的 DTrace probe，并将文件名和行号（`__FILE__`, `__LINE__`）作为参数传递。开发团队编写了一段 DTrace 监控脚本：一旦执行新场景发生崩溃，系统能精准上报命中了哪一个未实现的 `XXX` 代码桩，从而指导工程师按需攻坚。 到了次年 3 月下旬，Max Bruning 实现了关键突破：从最初只能执行 3 条指令，发展到能够启动一个持续运行且不触发内核 Panic 的虚拟机。然而，当时的虚拟机处于假死状态，卡在“BIOS 实模式的漆黑深渊”中一动不动。步入 4 月初，工程正式具备了并行化开发的条件，Robert Mustacchi 和 Brian Cantrill 加入攻坚，开始构建专属观测工具以深入洞察虚拟客机内部的运行状态。 - 移植起点基于稳定的 Linux 2.6.34 内核中的 KVM 模块代码。 - 初期的串行攻坚阶段由 Max Bruning 自深秋持续推进至次年 3 月底。 - 4 月初 Robert Mustacchi 与 Brian Cantrill 正式加入，开始多模块并行攻关。 ### [09:38](https://www.youtube.com/watch?v=cwAfJywzk8o&t=578s) 打造调试利器：MDB 扩展与 DTrace 探针 为了攻克移植过程中遇到的诡异底层难题，团队首先对 Illumos 的调试工具进行了大幅扩充。MDB（Modular Debugger）是其内核与用户态通用的活体及死后（postmortem）调试器。团队为 MDB 扩展了将客户机物理页帧号（Guest PFN）映射到 QEMU 虚拟地址（VA）的功能，甚至破天荒地为 MDB 增加了 16 位实模式反汇编支持。Brian Cantrill 直言他从未想过自己这辈子还会亲自写 16 位反汇编器，但这绝非出于学究式的无病呻吟，而是为了排查真正的致命问题——当时 BIOS 引导失败，通过反汇编 BIOS 配合 DTrace 探针，他们看清了执行流卡在特定的 8 条指令中，进而抓出了在 Programmed I/O（PIO）实现上的严重失误。 ```mermaid flowchart TD accTitle: KVM 调试工具链增强 accDescr: 展示 MDB 扩展与 DTrace 探针如何帮助排查 VM 启动与指令执行问题 subgraph MDB["MDB 调试器扩展"] A["Guest PFN 映射到 QEMU VA"] B["16位 实模式反汇编支持"] end subgraph DTrace["DTrace 观测能力"] C["VM-entry / VM-exit 动态探针"] D["内置 vmregs 变量提取 VMCS 寄存器状态"] end subgraph Problem["排查目标"] E["BIOS 引导执行流跟踪"] F["Programmed I/O 实现缺陷修复"] end MDB --> E DTrace --> E E --> F ``` 与此同时，团队在虚拟机的 VM-entry 和 VM-exit 路径上布设了大量 DTrace 探针，并引入了全新的内置变量 `vmregs`，使得工程师能直接通过 DTrace 脚本化地从 Intel VMCS 结构中抓取所需的寄存器状态。整个排查流程就是不断迭代：遇到阻塞性 Bug、发现遗漏或写错的代码组件、补充代码、继续排错。Brian Cantrill 坦言，虽然事后回顾显得很云淡风轻，但当时团队内心的真实状态是极度恐慌：“Of course every time I'm making this sound so clinical, every time you're actually scared out of your mind that this is the bug that is actually going to put you out of business, this is the one that is actually you're not going to be able to find.” ### [11:25](https://www.youtube.com/watch?v=cwAfJywzk8o&t=685s) 移植里程碑与代码中的“XXX”曲线 代码移植的曲折历程完整地投射在了源码中 `XXX` 占位注释的数量变化曲线上。起初，Max Bruning 苦于让庞大的代码成功编译，随着不断引入文件和打桩，`XXX` 数量直线上升；随后 Max 意识到遗漏了整套源文件，补齐后占位符进一步激增；直到 4 月份，团队开始攻克关键难关，逐步清理并剥离占位代码，移植才迎来了破晓时刻。 首个重大里程碑是成功引导 kmdb（Illumos 原生的内核在位调试器）。一旦 kmdb 可以在虚拟机中跑起来，团队就能直接从客户机视角审视其体系结构状态，整体排查效率成倍提升。接着便是成功引导 Linux 内核。回看提交日志，从 kmdb 到 Linux 启动之间仅仅隔了大约 10 天，但 Brian Cantrill 却觉得那十天度日如年：“These were I actually think the 10 most terrifying days of my life... that felt like several years.” 这期间他们被一系列自找的（self-inflicted）极其严酷的 Bug 狠狠折磨。随后他们又迅速跑通了 Windows 和 Haiku OS。Cantrill 对 Haiku 赞不绝口，他提到很多人不知道验证系统改动时该用什么测试客体，而 Haiku 总能对底层硬件做出一些奇特操作（例如把时钟频率拉得极高），是绝佳的检验工具。 - **源码中 XXX 数量曲线**：直观反映了移植阶段从“盲目打桩促成编译”、“补齐漏掉的源码文件导致占位暴增”到“逐步攻坚、全面清理并收尾”的技术演进。 - **kmdb 引导成功**：使得开发者能直接在 Guest 内部断点审查 CPU 寄存器与内存视角，打通了黑盒调试的瓶颈。 - **验证操作系统阵列**：先后跑通了 kmdb、Linux、Windows 以及行为奇特但极其考验硬件虚拟化边界的 Haiku OS。 ### [13:11](https://www.youtube.com/watch?v=cwAfJywzk8o&t=791s) 坚如磐石的 KVM 与唯一的 QEMU 信号缺陷 令人意外的是，作为有着高度怀疑精神的资深工程师，团队在整个移植过程中竟然没有在 KVM 内核代码里发现哪怕一个原生缺陷。Cantrill 对此深表震撼，并在现场向 KVM 维护者 Avi Kivity 及其社区由衷致敬：“In the course of this port we did not discover a single bug that you would call a bug in KVM. Remarkably, remarkably solid.” 在整个 KVM/QEMU 上游代码树中，他们唯一撞上的 Bug 仅仅是 QEMU 在处理核间中断信号 `SIGIPI` 时，错误地调用了历史遗留的 `signal()` 而非规范的 `sigaction()`。Cantrill 戏称关于 `signal` 对比 `sigaction` 的痛苦演化历史以及为什么绝不能用 `signal` 来修改非忽略状态的信号配置，足足可以讲一堂长篇大论课。由于 Linux 与 Illumos 在 `signal()` 的语义细节上存在细微分歧，导致了该问题暴露，但这终究只是微小的上层缺陷。整个 KVM 核心机制在跨平台移植时的惊人稳定性给移植团队留下了深刻印象。 ### [14:15](https://www.youtube.com/watch?v=cwAfJywzk8o&t=855s) 自找的麻烦：错乱的 GS Base 与“我是谁”的 CPU 认知危机 虽然 KVM 本身极其稳固，但移植团队在将 Linux 假设映射到 Illumos 内核时踩到了极其离奇的“自伤型”Bug。在调试 Linux 客户机的那一周内爆发了大约六个致命问题，其中最凶险的一个与 CPU 局部存储（per-CPU GS Base）的恢复有关。在 Illumos 架构中，内核依赖 `GS` 寄存器基地址来定位 per-CPU 数据结构（存储着当前运行的线程、CPU 身份等最核心的状态），而团队的移植代码在某些特定线程迁移（thread migration）场景下，未能正确恢复 GS Base。Cantrill 甚至在现场询问 Avi Kivity Linux 是否也会遇到此问题，在得到现场反馈如果使用类似机制确实可能是个 Bug 后，Cantrill 兴奋地表示那终于可以回馈社区一个补丁了。 该 Bug 展现出的故障现象堪称荒诞至极。在持续高负载压力测试下，系统触发了线程迁移，导致两个物理 CPU 陷入了认知错乱： > “I had two different CPUs, one of which believed that it was the other from a software perspective. Oh my God, the mayhem!” 当一个 CPU 读取错乱的 GS Base 后，它根据 per-CPU 变量认定了自己是“另一个 CPU”，并试图执行另一个 CPU 的线程逻辑，随后迅速撞上最底层的断言崩溃：“等一下，我为什么在这里？我应该在那边才对！”这种诡异的状态让拥有 15 年内核开发经验的 Cantrill 也大开眼界。最终，团队依靠 DTrace 强大的死后转储插桩分析能力（Postmortem Instrumentation）——系统在崩溃前持续记录探针缓冲，并在宕机生成 crash dump 重启后，逆向回放 DTrace 的执行轨迹，才将这一隐秘的时序竞态彻底破译。 ### [16:37](https://www.youtube.com/watch?v=cwAfJywzk8o&t=997s) FPU 寄存器覆盖与 5% 的性能谜题 另一个重磅的自伤 Bug 发生在浮点单元（FPU）状态的保存逻辑上。KVM 原本在 Linux 下有自己的 FPU 保存实现，而团队在移植到 Illumos 时，为了代码重用，将其替换为 Illumos 内部现成的 FPU 保存函数。然而，Illumos 的该函数在执行 `fxsave` 指令的同时，顺带清除了控制寄存器 `cr0` 中的任务切换位（`TS` 位）。 在 x86 架构中，操作系统的浮点状态惰性恢复机制高度依赖 `cr0.TS` 位：当线程尝试使用 FPU 时，硬件会抛出 `#NM` 异常（Device Not Available），内核捕获后负责载入属于该线程的寄存器状态。但因为移植代码清空了 `TS`，CPU 在使用浮点寄存器时完全不触发 `#NM`，直接拿着寄存器中遗留的垃圾数据继续狂奔。而在现代底层系统中，高频调用的 `memcpy` 经常使用浮点/SSE 寄存器作为高速中转缓冲，这直接导致虚拟机在编译大型项目时产生随机破坏的二进制目标文件（corrupted object file）。排查该问题让 Cantrill 痛苦到想给“内核开发者自杀干预热线”打电话寻求心理疏导。 ```mermaid flowchart LR accTitle: FPU 惰性保存破坏机制 accDescr: 描述因清除 TS 位导致缺少 NM 异常进而引发 memcpy 数据损坏的过程 A["调用 Illumos FPU 保存函数"] --> B["执行 fxsave 并清空 cr0.TS 位"] B --> C["线程执行使用 FPU 的 memcpy"] C --> D{"是否触发 #NM 异常?"} D -- 否 (TS已清空) --> E["未恢复线程自有 FPU 状态"] E --> F["直接使用寄存器中的脏数据"] F --> G["内存数据错乱 / 目标文件损坏"] D -.->|正常流程本应触发| H["恢复 FPU 上下文"] ``` 在跨过了重重深坑后，团队针对纯 CPU 性能进行了基准测试。采用固定负载测试显示，SmartOS KVM 的执行性能与物理裸机（Bare Metal）及 Linux KVM 处于完全相同的水准。有趣的是，在测试之初，团队曾测出 SmartOS KVM 比 Linux KVM “快 5%” 且具有 100% 的可复现性。但作为严谨的工程师，他们深知在底层 CPU、主板芯片组和内存完全相同的情况下，虚拟机不可能凭空比物理硬件和其他成熟系统快 5%，这立刻引发了他们对基准测试客观性的深入审查与警惕。 - **FPU 保存缺陷后果**：清空 `cr0.TS` 导致 `#NM` 异常失效，破坏浮点上下文恢复，进而引发由 `memcpy` 造成的广泛数据损坏。 - **基准测试铁律**：面对“SmartOS 虚拟机性能稳定高出 5%”的反常优胜数据，团队没有盲目庆功，而是坚定认为相同硬件下纯计算性能必然持平，并以此为线索追查跑分背后的环境与逻辑差异。 ### [18:56](https://www.youtube.com/watch?v=cwAfJywzk8o&t=1136s) 睿频谜团的解开与基准测试结果 在对比 Linux 与 SmartOS 的 KVM 性能时，团队起初遇到了极其令人困惑的数据波动。最终，Joyent 团队的 Brendan Gregg 成功“破译了谜团”：他们观察到的性能差异源于 Intel Nehalem 处理器的睿频加速（Turbo Boost）动态超频技术。Brian Cantrill 提醒现场所有工程师，在进行严谨的基准测试时必须关掉该特性，因为 CPU 会根据环境温度自动调整主频，而 TSC（时间戳计数器）频率却维持不变，这会导致基准测试结果变得极具误导性。虽然通过 ACPI 寄存器能读到实际频率，但关闭睿频才是消除干扰的最干脆手段。关闭后，双方性能完全持平，Cantrill 调侃道：“我很想告诉你我们快了 5%，但事实并非如此；关掉它后大家速度一模一样，这反倒让我们松了口气。” ```mermaid flowchart TD accTitle: 基准测试中 Intel 睿频引起的频率干扰机制 accDescr: 展示环境温度如何通过 Turbo Boost 改变实际 CPU 频率，从而与固定的 TSC 频率产生脱节，干扰测试结果。 subgraph Hardware["硬件底层行为"] A["环境温度变化"] --> B["Intel 睿频加速 (Turbo Boost)"] B --> C["实际 CPU 主频动态浮动"] D["固定 TSC 频率"] end subgraph Impact["基准测试表象"] C --> E["Sysbench 性能结果波动"] D -.->|"频率脱节导致混淆"| E end subgraph Solution["调试结论"] E --> F["Brendan Gregg 找出根因"] F --> G["彻底关闭 Turbo Boost，数据恢复一致"] end ``` Cantrill 展示了 Sysbench MySQL 基准测试图表，指出裸机性能处于顶端，而 Linux KVM 与 SmartOS KVM 的性能曲线几乎重合，在统计学上完全一致。关于两条曲线在中途出现的交叉，他推测可能与 CPU 调度策略有关，未来会进一步探究。他总结道，基准测试的首要目标只是确保团队没有在移植中犯蠢或自我伤害，“我们一路上自找的麻烦已经够多了，完全有理由相信还会遇到更多，所以我们只想确认自己还在同一梯队里。” - 端口代码已经完全开源，托管在 GitHub 仓库中，并配有微调过的 QEMU 0.14.1 分支。 - 移植质量目前已接近或达到生产级别，但这仅限 Intel VT-x 架构；由于完全未实现 AMD SVM 支持，AMD 平台上尚不可用于生产。 - 就在演讲当天早晨代码发布后仅 45 分钟，另一个 illumos 发行版 OpenIndiana 便直接拉取了最新代码，成功引导了 Windows 7 虚拟机并在社区发布截图；在此之前 Joyent 仅测试过 Windows Server 2008 R2，这次意外的社区验证让 Cantrill “长舒了一口气”。 --- ### [21:47](https://www.youtube.com/watch?v=cwAfJywzk8o&t=1307s) 核心限制：拒绝内存超卖与内存去重 谈及 SmartOS KVM 的功能限制，Cantrill 重点强调了第一大设计决定：**客户机内存必须全额锁定（Locked down）**。Joyent 作为拥有数千台物理机、数万家客户的公有云服务商，长期的运维经验让他们坚信内存超卖在生产中弊大于利。内存超卖对于空闲负载非常理想，但只要有客户开始重度压榨机器，由此引发的长尾延迟与性能抖动便会彻底抵消那点微薄的边缘收益。因此，SmartOS 当前直接锁死客户机内存，一旦内存锁定失败，虚拟机就无法启动。 Cantrill 阐述了 Joyent 在虚拟化层级上的基本哲学：在大规模、高密度的多租户场景下，硬件虚拟化并非最佳解法。面对单机 400 个租户的高并发 PaaS 负载，基于内核的操作系统级虚拟化（illumos Zones/Containers）不仅极其轻量，而且具备硬件虚拟化无法比拟的高性能。Cantrill 直言：“我们在 2U、96GB 内存的机器上跑着 400 个租户，每个人都感觉自己拥有一套高性能的虚拟系统；硬件虚拟化无论多优秀，根本做不到这一点。” ```mermaid flowchart LR accTitle: SmartOS 内存锁定与虚拟化层级架构 accDescr: 区分操作系统级虚拟化与硬件虚拟化的定位，说明 SmartOS 内存锁定的处理机制。 subgraph OSVirt["OS 级虚拟化 (高密度多租户)"] Z["illumos Zones 容器"] -->|"单内核轻量支撑"| H["400 租户 / 2U 节点 (96GB)"] end subgraph HWVirt["硬件级虚拟化 (多操作系统兼容)"] KVM["SmartOS KVM 虚拟机"] -->|"物理内存锁定"| MLock["锁死 Guest 内存"] MLock -->|"锁定失败"| Abort["拒绝启动 VM"] MLock -->|"放弃超卖"| LowLat["保证确定性超低延迟"] end ``` 关于内核同页合并（Kernel Samepage Merging / 内存去重），SmartOS 目前同样未予实现。现场观众提问补充道，Windows XP 这类系统会主动清零大量空闲内存，且不将其占作缓存，去重技术能将这些全零页及只读的代码段（shared text）合并为单页，在桌面虚拟化（VDI）场景下收益极高。Cantrill 对此表示认可，认为未来社区若有强烈需求 illumos 也可以通用化实现，但他格外关注后台页面扫描带来的性能开销，这也是他们当前持观望态度的核心原因。 --- ### [25:01](https://www.youtube.com/watch?v=cwAfJywzk8o&t=1501s) 嵌套虚拟化的取舍与 QEMU 字符设备适配 针对嵌套虚拟化（Nested Virtualization），Cantrill 明确表示 SmartOS 暂不支持，甚至怀疑以后是否值得去实现。他认为在 Intel VT-x 架构上做嵌套虚拟化极其丑陋复杂（相比之下 AMD SVM 稍显优雅）。当 Cantrill 向现场询问该特性在实际业务中的定位时，台下听众直接回应称它“目前就是一个在寻找问题的解决方案”（a solution looking for a problem），引得 Cantrill 大笑并赞同道：“你说出了我想知道的一切。” Cantrill 指出，嵌套虚拟化的补丁代码极其晦涩繁琐，硬件本身根本没有提供足够的底层支持，“去读那些补丁简直是让你的大脑迅速陷入死锁”。虽然嵌套虚拟化对本地开发（例如在 VMware Fusion 中调试虚拟化平台）极具价值，但在公有云生产环境中缺乏真实的迫切需求。 ```mermaid flowchart TD accTitle: vCPU 文件描述符创建机制对比 accDescr: 展示 Linux 原生 anon_inode 机制与 SmartOS 借助 KVM_CLONE ioctl 实现 vCPU fd 映射的区别。 subgraph Linux["Linux KVM 实现"] L1["driver ioctl 调用"] -->|"调用 anon_inode_getfd"| L2["直接在内核创建匿名 FD"] L2 --> L3["每个 vCPU 对应一个独立 FD"] end subgraph SmartOS["SmartOS 改造实现"] S1["QEMU 发起系统调用"] -->|"调用新增 KVM_CLONE ioctl"| S2["驱动实例状态重定向"] S2 --> S3["在 QEMU 用户态补充约 10 行代码"] S3 --> S4["实现 1:1 vCPU 对应 FD 映射"] end ``` 在 QEMU 的移植分歧方面，Joyent 的核心原则是**尽可能减少对上游 QEMU 的侵入式修改补丁**。Linux KVM 在内核中依赖 `anon_inode_getfd` 机制，允许驱动在 `ioctl` 内部直接为新生成的 vCPU 打开并返回一个新的文件描述符（FD），从而维持一个 vCPU 对应一个 FD 的 1:1 映射模式。然而 illumos 内核驱动模型中不存在这种机制。Cantrill 回忆道，团队曾尝试打破这种 1:1 映射，试图把 vCPU ID 作为所有 `ioctl` 的参数传入，但很快发现这会导致与上游接口完全决裂，团队只得紧急撤回方案：“我们发现自己彻底走偏了，于是赶紧回退、回退、再回退。”最终，SmartOS 引入了一个全新的 `KVM_CLONE` ioctl，使得驱动内部实例指向新实体，配合 QEMU 端仅仅十行左右的代码，以极小的代价兼容了原生 vCPU 文件描述符模型。 - Cantrill 在探讨嵌套虚拟化时明确表示：“嵌套虚拟化对我们自己的开发当然极具吸引力，但对于开发用途而言，最合理的做法也是让别人去把它实现出来。” - 现场开发者提到嵌套虚拟化在应用层开发及测试拓扑验证中存在用途，Cantrill 表示会保持关注，观察其未来是否真能演变为正式的生产负载。 ### [27:13](https://www.youtube.com/watch?v=cwAfJywzk8o&t=1633s) 上下文切换钩子与 Cyclic 定时器：Illumos 底层的技术适配 在将 KVM 引入 Illumos 内核的过程中，Joyent 针对虚拟 CPU（vCPU）生命周期管理做出了细致调整。为了在单实例上创建 vCPU，代码会重新打开 `/dev/kvm` 获得全新实例，并在新实例上调用 `KVM_CLONE` 绑定到原有实例，以此支持后续 vCPU 的分配。Brian Cantrill 指出，系统内部内置了严格的防错逻辑，防止单实例上错误创建多个 vCPU；这一设计虽然在 QEMU 内部引入了少量 `#ifdef` 分支，但整体运行极为顺畅。 关于与 Linux 原生实现的细微技术分歧，Cantrill 重点提到了两项源于 Solaris/Illumos 历史遗产的技术： ```mermaid flowchart TD accTitle: Illumos 内核对 KVM 的底层支撑机制 accDescr: 展示 Context Ops 与 Cyclic 定时器子系统在 KVM 移植中的对应关系与工作机制。 subgraph ContextOps["上下文钩子 (Context Ops)"] CO_Hist["历史背景: SPARC 时代虚拟化性能计数器"] --> CO_KVM["KVM 映射: 等价于 kvm_sched_in / kvm_sched_out"] CO_KVM --> CO_Effect["线程脱离 CPU 时自动触发回调"] end subgraph CyclicSubsys["高精度定时器 (Cyclic Subsystem)"] CS_Hist["Cantrill 90 年代末开发的高精度周期触发器"] --> CS_KVM["KVM 映射: 单次触发 (One-shot) 动态重编程"] CS_KVM --> CS_Mig["vCPU 跨核迁移: 不迁移 Cyclic，直接跨 CPU 唤醒 Target CPU"] end ``` - **上下文操作（Context Ops）**：早在早年分析性能极其孱弱的 SPARC 处理器时，Illumos 团队为了虚拟化 CPU 性能计数器而实现了 Context Ops 机制。Cantrill 调侃道：“我们长期虚拟化 CPU 性能计数器，部分原因是为了搞明白 SPARC 这款在市场上毫无竞争力的微处理器到底出了什么问题，所以我们必须榨干系统的每一处性能。”在 KVM 移植中，Context Ops 起到了关键作用，精准对应于 Linux 中的 `kvm_sched_in` 与 `kvm_sched_out`。 - **高精度定时器（Cyclic 子系统）**：Cantrill 在 1990 年代末主导开发的任意精度间隔定时器设施。该设施可直接绑定至特定 CPU 或处理器集，并在不同中断级别触发。如今已支持动态单次触发（One-shot）重编程，所有 KVM 定时器均基于 Cyclic 运行。与 Linux 频繁迁移定时器不同，当 vCPU 跨核迁移时，Cyclic 会直接停留在原核并“戳醒”（poke）目标 CPU，改动极小却性能可靠。 --- ### [28:47](https://www.youtube.com/watch?v=cwAfJywzk8o&t=1727s) ZFS 赋能与 Zone 容器沙箱对 QEMU 的安全加固 谈及 KVM 存储与安全增强，Cantrill 指出 Joyent 根本不需要对 ZFS 做特殊修改，“ZFS 遇上 KVM 就能开箱即用”。用户只需创建一个 ZFS 卷（Zvol），就可以让 QEMU 和 KVM 肆意发挥。Cantrill 借此打趣并吐槽了此前听到的演讲：“借助 ZFS，大家终于可以停止在 QCOW2 里搞用户态文件系统这种令人难受的实验了。” ```mermaid flowchart LR accTitle: SmartOS 结合 ZFS 与 Zone 增强 KVM accDescr: 描述 ZFS 存储特性以及 Zone 容器对 QEMU 进程进行隔离保护的架构逻辑。 subgraph HostStorage["ZFS 存储引擎特性"] Z1["常数时间克隆 (Constant-Time Cloning)"] Z2["统一 ARC 缓存加速 Guest 随机 I/O"] Z3["ZFS Send/Receive 跨广域网极速迁移"] end subgraph SecurityArch["Zone 操作系统级沙箱"] QEMU["QEMU 用户态进程 (唯一进程)"] -->|"发生逃逸 / 漏洞利用"| JPrison["Zone 严格受限容器"] JPrison -.->|"无多余特权"| HostOS["宿主机核心与网络"] end ``` 利用 ZFS 作为 KVM 存储底层带来了三大核心收益： 1. **常数时间克隆（Constant-time cloning）**：虚拟机处于运行状态下即可瞬间创建快照与克隆分支，极大提升了测试和分发效率。 2. **统一自适应替换缓存（ARC）**：Guest 虚拟机的磁盘 I/O 可以无缝被宿主机内存中的 ARC 命中，为随机 I/O 带来潜在的巨大吞吐提升。 3. **ZFS Send/Receive 远程复制**：该机制为未来实现广域网（WAN）环境下的克隆与虚拟机迁移奠定了高效基础。 在安全与资源管理方面，Joyent 将每一个 QEMU 实例单独塞进一个独立的 Solaris Zone（操作系统级虚拟化容器）中。最初这一设计是为了精准计费、QoS、资源限制以及 I/O 限流；但在 Black Hat 黑客大会爆出一系列虚拟机逃逸漏洞后，这一架构展现出惊人的安全防护优势。Cantrill 强调，漏洞实际上并不在内核态的 KVM，而全都在用户态的 QEMU：“在我们的实现中，如果你设法从 QEMU 里钻出来，你会发现自己只是钻进了一个更大、安全级别极高的监狱——也就是容器。你在那里几乎没有任何特权，攻击面被急剧缩减。” --- ### [31:14](https://www.youtube.com/watch?v=cwAfJywzk8o&t=1874s) 基于 VNIC 的网络虚拟化与 KVM 统计工具（KVM Stat） 在网络虚拟化方面，SmartOS 依托 Crossbow 架构为每个 KVM 虚拟机创建专属虚拟网卡（VNIC），并编写了简单的适配代码桥接到 Virtio。Joyent 仅用极少量的工程量就轻松测得 1 Gbps 的单虚拟机吞吐带宽。 相较于 Linux 依赖 iptables 或 ebtables 的复杂规则，基于 VNIC 的网络栈优势显著： - **天然防伪造（Anti-spoofing）**：直接在 VNIC 上指定合法 MAC 地址，任何未授权的数据包均会被直接丢弃。 - **流量整形与共享（Flow Management）**：可以将一张 10 GbE 网卡精准切分给 10 个虚拟机，严格防止个别租户吞噬带宽；Cantrill 强调这在带宽极度昂贵的发展中国家至关重要。 - **细粒度可观测性**：在宿主机端即可直接观测单个 VNIC、单个虚拟机的带宽流量。 ```mermaid flowchart TD accTitle: KVM 统计信息流向 accDescr: 展示从 KVM 内核数据结构到 kstat 以及 kvmstat 工具的用户态输出链路。 Kernel["内核 KVM vCPU 统计 (kvm_vcpu_stat)"] -->|"封装接入"| Kstat["Illumos kstat 子系统"] Kstat -->|"数据消费"| Tool["kvmstat CLI 命令行工具"] Tool --> Display["按秒 / 按 vCPU 实时输出性能计数"] ``` 在内核监控方面，Joyent 将 Linux 的 `kvm_vcpu_stat` 结构体接入 Illumos 的内核统计框架（kstat），并向上游提交了命令行工具 `kvmstat`，支持以秒为单位按 vCPU 实时打印 VM-Exit 数据。 Cantrill 在此分享了一起排障趣事：团队在调试时发现新开机的虚拟机每秒会触发高达 1,300 次 VM-Exit，但排查几分钟后异常指标便神秘消失；重新启动系统，1,300 次退出又再度出现。工程师 Josh Wilsdon 突然反应过来：“等一下，这该不会是屏幕黑屏超时引起的吧？”经过验证，果然是操作系统默认的 5 分钟无操作自动关屏超时设置所致——屏幕变暗前频繁写入显存，引发了大量 EPT（扩展页表）违规陷阱。 --- ### [33:45](https://www.youtube.com/watch?v=cwAfJywzk8o&t=2025s) DTrace 深度下沉与实时云分析可视化 Cantrill 戏谑地指出，QEMU 0.14 版本虽然包含 DTrace 探针，但实际上是给 Linux 的 SystemTap 准备的，在真正的 DTrace 上根本无法直接工作。Joyent 修复并激活了这批探针，并在 SmartOS 的 KVM 内核中全面布放了静态 SDT（Statically Defined Tracing）探针。原 Linux 内核中所有形如 `trace_*` 的调用点均被转换为了 SDT 探针，在未启用时开销为绝对的“零”。 此外，配合之前引入的 `vmregs` 虚拟机寄存器探查功能，开发人员可以用极为简练的 D 语言脚本，按进程 ID、线程 ID 和具体的 Exit 退出原因对虚拟机行为进行动态追踪聚合： ``` # DTrace 聚合监控脚本示例逻辑 dtrace -n 'kvm_exit { @[pid, tid, arg0] = count(); } tick-1s { printa(@); trunc(@); }' ``` “我们发现了一个极具戏剧性的现象：在 Linux 用户态启动一个进程时，它竟然会连续执行三次 `CPUID` 指令。”Cantrill 惊奇地表示，在一个由纯 `fork`、`exec`、`exit` 构成的基准测试负载下，系统每秒产生的 VM-Exit 竟高达 8,000 次。 在演讲尾声，Cantrill 演示了 Joyent 将该能力无缝接入自研“实时云分析框架”（Real-time Analytics Framework）的成果。通过热力图（Heatmap），X 轴代表时间推进，Y 轴代表逻辑卷的扇区偏移量，直观展示了 Guest 虚拟机中 ext3 文件系统反复写入 3 GB 文件、删除再写入的连续寻道轨迹与盘片行为，将底层虚拟化的黑盒内部完全透明化。 - **零探针开销**：SDT 探针在禁用状态下只留下 nop 指令，提供无损的生产环境调试能力。 - **跨层穿透观测**：打通宿主机 DTrace 探针与 Guest 虚拟机用户态执行轨迹，极大简化底层性能瓶颈定位。 - **图形化分析呈现**：以时间-逻辑偏移二维热力图，直接还原 ext3 存储分配策略与实际 I/O 模式。 ### [35:46](https://www.youtube.com/watch?v=cwAfJywzk8o&t=2146s) 从宿主机底层洞察客户机：CR3 寄存器与调度可视化 Brian Cantrill 展示了通过 SmartOS 底层的 DTrace 观察客户机行为的奇妙能力。尽管从宿主机操作系统层面无法直接获知虚拟机内部正在运行的具体进程名称，但可以通过硬件维度的特征进行反推。因为 x86 架构下的 CR3 寄存器代表内存管理单元（MMU）的上下文（即页目录基址），它与具体进程之间存在极强的相关性。Brian Cantrill 直言：“我们无法直接告诉你哪个进程正在运行，但我们知道 CR3 看起来是什么样，而且我们知道 CR3 与进程有着非常强的关联性。我不知道它叫什么名字，但我知道它的十六进制数值。” ```mermaid flowchart LR accTitle: CR3采样与进程行为推导流程 accDescr: 展示宿主机如何通过DTrace采样CR3寄存器值推导客户机内核的进程调度行为 subgraph Host["宿主机 (SmartOS / DTrace)"] DTrace["999Hz 采样 CR3 寄存器"] HexVal["获取十六进制 MMU 上下文<br/>(例如 0x67D800)"] end subgraph Guest["客户机 (Linux Guest)"] ProcA["进程 A (黄色)"] ProcB["进程 B (蓝色 / 被 nice 调优)"] Sched["Linux 调度器 (交替切换)"] end DTrace --> HexVal Sched --> ProcA Sched --> ProcB HexVal -.->|"推导调度交替状态"| Sched ``` 在单 vCPU 的虚拟机实例中，他们启动了两个计算密集型（compute-bound）负载。通过以 999 Hz 的频率采样 CR3 寄存器，可以极其清晰地看到 Linux 调度器在两个计算任务之间来回交替切换的完整过程。随后他们进一步做了实验：对其中一个计算进程执行 `nice` 调整优先级，结果在热图中，该进程的执行窗口迅速被压缩，几乎从主画面中跌落消失，只偶尔浮现一次；再将其反转调换时，则在热图上呈现出规则的阶梯状（stair-step）分布形态。Cantrill 调侃说，关于这种阶梯模式背后的成因，“我确信这里肯定有一篇硕士论文，或者至少有一篇本科毕业论文等着人去写”。 - **可视化坐标定义**：图表的 X 轴表示连续的时间推进，Y 轴表示一秒之内的亚秒级时间偏移（offset in the second）。若将每一秒的垂直条带旋转 90 度并首尾相连，展示出来的就是 Linux 内核最真实的调度轨迹：交替运行黄色进程、蓝色进程，周而复始。 - **虽然匿名但极具洞察力**：即便捕捉到的进程名字仅仅是像 `0x67D800` 这样的十六进制 CR3 物理地址、看似用处有限，但这为系统工程师提供了在不入侵虚拟机内部的情况下前所未有的黑盒可观测性。 --- ### [37:44](https://www.youtube.com/watch?v=cwAfJywzk8o&t=2264s) 亚秒级中断向量分布分析 在展示完调度行为后，Brian Cantrill 拿出了另一个更加疯狂的可视化视图——基于亚秒级时间偏移和中断向量号（Vector）细分的中断请求（IRQ）采样热图。宿主机完全清楚自身正在向虚拟机投递哪些具体的中断，因此通过捕捉并绘制这些 IRQ 事件，能够透视出客户机内部与时间相关的深层行为模式。 热图上呈现出几类截然不同的视觉特征：有些地方呈现出极其平直规整的横向条纹（striations），这代表与绝对时间严格绑定的时钟中断向量；还有一些呈现斜向贯穿的对角线纹理，推测是基于相对时间偏移触发的定时事件；而在另一些区域，则是一大片杂乱无章、涂抹在整秒时间轴上的中断风暴（free-for-alls）。Cantrill 笑着表示，虽然目前“还不确定是否能用这些数据做些什么，但光是盯着看就已经非常有意思了”。 --- ### [38:18](https://www.youtube.com/watch?v=cwAfJywzk8o&t=2298s) 社区合作愿景：性能突破与老旧系统测试农场 针对未来与 KVM 社区的合作，Brian Cantrill 表达了极其热切的期望，并提出了 Joyent 希望与社区重点推进的三个具体方向： 1. **联合攻关 KVM 性能**：借助 SmartOS 的 DTrace，团队对生产环境下各种任意客户机（arbitrary guests）的运行行为拥有极深的可观测性。Cantrill 指出，SmartOS 移植的 KVM 与 Linux 原生 KVM 在核心架构上拥有极大的共性，在一处获得的性能优化很可能会反哺并惠及另一方。他的目标十分激进，直言：“让我们携起手来，通过在生产环境中更深入地理解运行任意客户机的 KVM，去把那些专有闭源虚拟机监视器（proprietary hypervisors）打得落花流水。” 2. **构建冷门系统测试农场（Oddball ISOs）**：团队极度渴望搭建一套自动化测试环境，里面包含各种冷门偏门的操作系统的 ISO 镜像，尤其是 IBM OS/2。Cantrill 分享了一件轶事：他的一位大学好友是 VMware 的极早期初创员工，那位朋友曾告诉他，在 VMware 早期，“只要他们能引导启动 OS/2，他们就可以发布产品了，因为 OS/2 的每一步操作方式都极其刁钻奇葩”。Cantrill 坚信，如果他们在 SmartOS 上能成功引导 OS/2，那么 KVM 的移植与兼容性工作就彻底大功告成了。 3. **基准测试（Benchmarking）**：对于 SPECvirt_sc2010 等虚拟化基准测试，Cantrill 坦言自己与 SPEC 之间有着“漫长且复杂的历史纠葛，就像一段以惨烈方式分手的旧恋情一样”。尽管如此，他依然希望与社区协同建立客观的行业标准测试，不过他强调，相比于基准测试，他更在乎的是性能剖析与底层测试农场这两项前置工作。 --- ### [39:58](https://www.youtube.com/watch?v=cwAfJywzk8o&t=2398s) 致谢与演讲结语 在演讲的尾声，Brian Cantrill 向推动该项目落地的关键人物和技术团队致以诚挚的谢意： - **现场同僚**：来到会议现场的 Joyent 工程师 Josh Wilsdon 与 Robert Mustacchi，他们为项目的实现付出了巨大努力； - **性能与调试大师**：感谢 Brendan Gregg 在 DTrace 性能观测工具链以及可视化超频方面所做的大量突破性工作； - **虚拟化先驱**：感谢 Fabrice Bellard，Cantrill 称其为“会场中不可忽视的无形灵魂（unspoken spirit）”，正是他创造的 QEMU 为后来的虚拟化发展照亮了道路； - **底层硬件与上游支持**：感谢 Intel 提供了极为迅捷强劲的 CPU 硬件架构以及 EPT（Extended Page Tables）支持； - **KVM 缔造者**：感谢现场的 Avi Kivity 及其工程团队，将 KVM 这一伟大的虚拟化平台完整构建出来； - **开源生态**：感谢 Illumos 社区给予该移植项目自始至终的热情支持。 ## Notable quotes > [02:45](https://www.youtube.com/watch?v=cwAfJywzk8o&t=165s) 我们坚决认为——我甚至敢说 illumos 社区也坚决认为——最好的虚拟机监控器就是操作系统本身。 — Brian Cantrill > [28:47](https://www.youtube.com/watch?v=cwAfJywzk8o&t=1727s) 如果你设法从 QEMU 里钻出来，你会发现自己只是钻进了一个更大、安全级别极高的监狱——也就是容器。你在那里几乎没有任何特权，攻击面被急剧缩减。 — Brian Cantrill > [21:47](https://www.youtube.com/watch?v=cwAfJywzk8o&t=1307s) 我们在 2U、96GB 内存的机器上跑着 400 个租户，每个人都感觉自己拥有一套高性能的虚拟系统；硬件虚拟化无论多优秀，根本做不到这一点。 — Brian Cantrill > [28:47](https://www.youtube.com/watch?v=cwAfJywzk8o&t=1727s) 借助 ZFS，大家终于可以停止在 QCOW2 里搞用户态文件系统这种令人难受的实验了。 — Brian Cantrill > [00:01](https://www.youtube.com/watch?v=cwAfJywzk8o&t=1s) 企业级意味着我们在痛苦的熔炉里千锤百炼才把这该死的东西做对。 — Brian Cantrill ## Mentioned Brian Cantrill, Max Bruning, Robert Mustacchi, Brendan Gregg, Josh Wilsdon, Avi Kivity, Fabrice Bellard, Larry Ellison, Andy Grove, Joyent, Red Hat, Intel, AMD, Oracle, Sun Microsystems, IBM, VMware, SmartOS, illumos, OpenSolaris, OpenIndiana, Linux, FreeBSD, macOS, Windows, Windows 7, Windows Server 2008 R2, Windows XP, Haiku OS, OS/2, KVM, QEMU, kvmtool, ZFS, DTrace, Crossbow, MDB, kmdb, Virtio, Sysbench, SPECvirt_sc2010 ## Video description Brian Cantrill, vice president of Engineering at Joyent, presents "Experiences Porting KVM to SmartOS" at the KVM Forum 2011 in Vancouver, Canada. Cantrill talks about SmartOS, a derivative of a truly open source fork of OpenSolaris called Illumos. Key functionality inherited from OpenSolaris/Illumos includes enterprise-grade CFS, dtrace, OS-based virtualization, and network virtualization. Hardware-based virtualization is an ongoing project, though there have been advances by Intel and AMD in this space. Cantrill goes over the process and pains of porting KVM to SmartOS, the limitations and standards of their work, and the roadblocks they encountered along the way. He highlights contributions from members of the Illumos community as well as on-going innovation. More information about the KVM Forum: http://www.linux-kvm.org/page/KVM_Forum_2011 Presented by Red Hat, Inc. [Source](https://www.youtube.com/watch?v=cwAfJywzk8o)

[2:45] Brian Cantrill: 我们坚决认为——我甚至敢说 illumos 社区也坚决认为——最好的虚拟机监控器就是操作系统本身。 [28:47] Brian Cantrill: 如果你设法从 QEMU 里钻出来，你会发现自己只是钻进了一个更大、安全级别极高的监狱——也就是容器。你在那里几乎没有任何特权，攻击面被急剧缩减。 [21:47] Brian Cantrill: 我们在 2U、96GB 内存的机器上跑着 400 个租户，每个人都感觉自己拥有一套高性能的虚拟系统；硬件虚拟化无论多优秀，根本做不到这一点。 [28:47] Brian Cantrill: 借助 ZFS，大家终于可以停止在 QCOW2 里搞用户态文件系统这种令人难受的实验了。 [0:01] Brian Cantrill: 企业级意味着我们在痛苦的熔炉里千锤百炼才把这该死的东西做对。

[30:08] Brian Cantrill: If you somehow burrow out into QEMU, in our implementation you just find yourself having burrowed out into a much larger, much more secure prison, which is a container.

[0:01] [Music] [0:10] great thank you very much um hi good [0:12] afternoon um I am Brian canell I'm the [0:15] VP of engineering I think I'm speaking [0:16] way too loudly um I'm the uh the VP of [0:19] engineering at joint uh and I'm here to [0:22] present on our experiences uh porting [0:25] KVM to Smart OS so we have actually um [0:28] ported KVM to another operating system [0:30] of course the next natural question is [0:33] what the hell is smart OS um smart OS is [0:37] an lumos derived OS um how many of you [0:40] heard of Al lumos at least if you [0:41] haven't all right so not every hand's [0:43] going up um so I should have had a WTF [0:45] is a lumos slide as well um a lumos is [0:49] the what has become of open Solaris so [0:52] lumos is a true open source Fork of open [0:56] Solaris U and has been um aided [0:59] enormously by by Oracle seeming to [1:02] wanting to drive uh employees out of [1:04] Oracle sabotage the operating system and [1:05] so on it's been great for the open [1:07] source effort so thank you again Larry [1:10] um so lumos is completely independent um [1:13] and uh smart OS is an lumos derivative [1:16] um as an lumos derivative um there's a a [1:18] bunch of of key functionality that uh [1:22] smart OS effectively inherits um so how [1:24] many people have heard of ZFS does that [1:27] okay that's that's great um so a lot of [1:29] hands went up CFS is an Enterprise class [1:32] file system that is was originally [1:34] developed for Solaris it's since been [1:36] ported to lots of other systems [1:38] including FreeBSD including the Mac um [1:40] so a lot of other systems a lot of [1:42] really Advanced features built into ZFS [1:45] and again it's Enterprise grade um and [1:47] what does Enterprise grade mean [1:48] Enterprise grade means that we have [1:50] suffered in the Kil of pain getting the [1:52] goddamn thing right um and you know that [1:54] everyone around you in the storage [1:56] ecosystem is effectively Byzantine in [1:58] its Behavior it takes a long long long [2:00] time for a storage stack to get right [2:02] and that's what we've done with ZFS [2:03] we've walked the trail of tear with ZFS [2:06] um D trce people heard of drra is [2:09] Hands-On D TR okay decent number of you [2:11] uh facility for dynamic ad hog [2:13] instrumentation of the system um OS Bas [2:15] virtualization um this is basically um [2:18] you can think of this as jails on growth [2:20] hormone if you're familiar with BSD um [2:23] but it's actually much more than that [2:24] this is really uh os-based [2:26] virtualization as a first class citizen [2:29] in the Opera system operating at [2:30] Hardware speeds but in a highly [2:32] contained way which allows you to drive [2:34] multi tendency very high and we have [2:36] Network virtualization this project [2:37] called crossbow um which gives us a [2:39] virtual Nick infrastructure allows for [2:41] easy bandwidth management and resource [2:43] control so um those features are all [2:45] great but there's one glaring feature [2:47] that's missing um and we've got a great [2:50] feature set um but the thing that we're [2:52] missing that we were missing was [2:54] Hardware based virtualization and now [2:56] thanks to Intel and AMD Hardware based [2:58] virtualization can be now had um [3:02] relatively easier I wouldn't say it's [3:04] easy but it's easier and with remarkably [3:06] high performance and we emphatically [3:10] believe um we joint um we I would I dare [3:13] say the lumos Community emphatically [3:15] believe that the best hypervisor is the [3:18] operating system that is to say anyone [3:20] attempting to implement a quote unquote [3:22] thin hypervisor I won't name any [3:24] proprietary names but we all know who [3:25] I'm thinking of anyone who attempts to [3:27] implement a quote-unquote thin [3:28] hypervisor will go ahead and repeat 20 [3:31] years of os implementation history and I [3:34] think that we would all we can all agree [3:35] that that's actually the wrong path to [3:37] take and indeed we all do agree that [3:39] this is the wrong path to take not only [3:40] does KVM share this Vision KVM pioneered [3:43] this Vision um and I think it was uh a [3:45] huge tribute Obi to you and your team [3:48] that that that you brought this Vision [3:50] to to the world so not only of course [3:52] did kbm bring the vision on to share the [3:54] vision with KVM but KVM itself is best [3:56] of breed a highly competitive [3:57] performance I love watching the the uh [4:00] the spec numbers and watching KVM um is [4:02] either competitive with proprietary [4:04] Solutions or bidding them and a [4:05] community witness our conference here [4:07] with critical mass so the imperative for [4:09] us was clear we had to Port KVM to Smart [4:13] OS sounds like a really easy thing to [4:15] say let's just go do it it's just a port [4:17] right like an editor well no that's much [4:21] so um we wanted to constrain the the [4:23] scope of our port for a little bit first [4:24] of all for business and resourcing [4:25] reasons um we elected to focus [4:27] exclusively on Intel VTX with EPT full [4:30] disclosure joyan is an Intel backed [4:32] company but it wasn't like Andy Grove [4:34] was in my office hanging me to make this [4:36] decision um we um just for practical [4:39] reasons um we decided to focus on Intel [4:42] VTX but we didn't want to do anything [4:43] that would make AMD svm support [4:45] impossible so we we didn't want to screw [4:47] AMD svm but we did want to focus on [4:49] Intel btx now one thing is that we are [4:51] only ever interested in x86 64 host [4:55] support um you know actually just as I [4:57] say that I think actually I should cross [4:58] out we we are most likely ever [5:01] interested in x86 64 host support um [5:04] Adam would be the example of where I [5:05] might be actually interested in x86 host [5:08] support but we are essentially only [5:10] we're willing to do work if we need to [5:11] go do x86 support as opposed to x86-64 [5:13] host support we are only ever interested [5:15] in this I can say emphatically in x86 [5:17] and x86 64 guests love the power 7 stuff [5:21] and you know I I you know I I get a [5:23] little tear in my eye when I think of [5:24] risk having spent 14 years at Sun but um [5:27] the reality is that we are only and x86 [5:30] and X8 664 guests now um we are willing [5:33] to diverge from KVM um to support alumo [5:36] constructs and coding practices but we [5:38] really wanted to main maintain [5:40] compatibility with the Kemo KVM [5:42] interface as much as possible I think [5:43] you guys did a great job of getting a [5:45] really crisp crisp layer in there I mean [5:47] witness the fact even though I think [5:49] it's arguably misguided um witness the [5:52] fact that with the the KVM tool is [5:54] building on top of that interface [5:55] building a wholly novel user land I [5:58] actually think that that is healthy in [5:59] as as much as it it really helps firm up [6:01] that ABI so a nice firm ABI there is in [6:03] everyone's best [6:04] interest so okay let's get started well [6:07] KVM totally rightfully was not designed [6:10] to be portable it shouldn't be designed [6:11] to be portable it is specific to Linux [6:13] it is specific to Linux facilities and [6:15] it became clear to us very quickly like [6:17] immediately like as soon as we popped it [6:19] up in an editor that emulating Linux [6:21] functionality is not going to be [6:22] sufficient and that is actually this is [6:24] the path that the FreeBSD summer of code [6:27] 2007 effort took was the uation Linux [6:29] fun fun ality um that won't work for us [6:31] the the inernal functionality is simply [6:33] too Divergent to actually allow that to [6:35] happen so um that was not going to work [6:38] but the flip side is obvious just to [6:40] your your remarks this morning the KVM [6:43] is stable and you look at 2634 I mean [6:45] you got a very stable base it's in [6:47] production the world over uh and we felt [6:49] confident that we could diverge from the [6:51] Linux implementation with that 2634 [6:53] implementation while still being able to [6:56] to contribute patches um to the KVM [6:58] community and I'll explain why in a [7:00] second you haven't seen any Patches from [7:01] us hint it's good news um and consume [7:04] Patches from the kbm community so we we [7:06] felt that we could we could do this [7:08] realistically now it was also clear that [7:10] just getting something to compile was [7:12] going to be a significant if not [7:13] historic uh undertaking and largely [7:16] serial so you know we would love to have [7:18] a bunch of folks attack this but the [7:19] reality was we just had to have start [7:21] this as something of a small project um [7:24] so joint engineer Max brunning who's in [7:25] the room Max you want to raise your hand [7:27] so um max had the max was kind of thrown [7:30] into the the breach there uh and started [7:33] on this in the late fall of last year [7:35] now to xite compilation and I'm sure [7:37] those of you who have done [7:39] extraordinarily complicated ports but [7:41] will know this that you can't just uh go [7:44] through and get everything to compile [7:45] and kind of Port everything you got to [7:47] pick and choose and kind of ifdef stuff [7:49] out so to expedite just getting the [7:51] compilation we took unported blocks of [7:53] code and we XXX them out um by enclosing [7:57] them in ifdef XXX um I would actually I [8:00] should have looked it up on the internet [8:01] to see if I could find the actually [8:02] historical PR I always grew up with if [8:04] def XXX and I'm sure it dates back to [8:06] the dawn of time uh probably dates back [8:08] to like the IBM 1401 or something but [8:10] anyway the um we we went with the kind [8:13] of the nod to history there if def XXX [8:15] code out now once you've got a bunch of [8:18] XXX code out how do you know what you [8:19] want to go pull in how do you know what [8:21] code you would be hitting what we [8:23] actually did is we took those code sites [8:25] and we put pound else Clauses in there [8:27] that had special Detroit probes with [8:29] under bar file under bar under bar under [8:30] bar line under bar under bar as the [8:32] arguments to that L to that probe then [8:34] we can use this nice little dript here [8:37] to actually tell us what XXX code blocks [8:40] you have hit and what do we need to go [8:42] focus on next and what would often [8:44] happen is we would go to do something [8:46] new the system would fail spectacularly [8:48] and we say oh guess what we just [8:50] executed a new XXX that we need to go [8:52] Implement so this actually happened a [8:54] lot um by now Mac started in late fall [8:58] by late March March um good news max [9:00] could launch a virtual machine uh [9:03] initially it was only three instructions [9:04] then could launch a virtual machine that [9:05] could run forever without panicking [9:08] awesome the problem is it wasn't doing [9:10] anything it was it was stuck somewhere [9:13] God only knows where it is stuck in the [9:15] inky black of the bias in real mode um [9:18] who the hell knows what's going on but [9:20] now we were at the point um in early [9:22] April um where we could paralyze the [9:24] effort a little bit and Robert Baki [9:26] who's also in the room from joyant um [9:28] the um so Robert and I were able to join [9:31] Max in April and and we could really [9:32] paralyze the effort and begin to attack [9:35] these things so we quickly added some [9:36] tooling to understand guas Behavior Uh [9:38] MDB is our debugger um so we added MDB [9:42] support this is a both a postmortem and [9:44] an Institue debugger um for both colel [9:46] and useland we added MD MDB support to [9:49] map guas pfns to Kimo vas I I can't [9:52] believe this but we added MDB support [9:55] for 16bit disassembly I swear I would [9:57] never live to actually be the one doing [9:59] this not only doing this but not at an [10:02] act of like pedantry but actually to [10:04] solve a real honest to God problem and [10:07] to have this thing totally Crack the [10:08] Case um when we added and these two [10:11] things together allowed us to actually [10:12] go disassemble the bias and we could [10:14] actually see thanks to the D probes we [10:16] could actually see oh we're stepping [10:18] through these eight instructions and we [10:20] had screwed up mm so we had screw [10:22] screwed up program bio I guess in that [10:24] case and we're able to figure out how [10:25] we'd screwed up program IO and and get [10:27] it back on its feet um then we also [10:29] added and these were these have been [10:30] huge we just used these all over the [10:32] place on Detroit's probes on VM entry [10:34] and exit and in particular the the [10:37] ability to pull VM state in dra trce [10:39] with a new VM regs variable so this [10:42] allows you to programmatically pull [10:44] exactly the the the the regs out of the [10:46] vmcs that you want to debug a particular [10:48] problem so um we were making progress um [10:51] and what we would do is we would we [10:52] would we would debug the the issue that [10:54] was blocking us almost always due to a [10:56] piece we hadn't pulled in yet we would [10:58] go to pull in the or hadn't done [10:59] properly we would pull in that piece and [11:02] we would when we would iterate iterate [11:03] and iterate of course every time I'm [11:05] making this sound so clinical every time [11:08] you're actually scared out of your mind [11:10] that this is the bug that is actually [11:12] going to put you out of business this is [11:13] the one that is actually you're not [11:14] going to be able to find so I don't mean [11:16] to make this sound too dry um we were [11:18] terrified for most of this um so just I [11:21] want to be clear about that um I don't [11:22] want to have too much machismo here um [11:25] so um but we did actually iterate and we [11:28] and some of these were hard some of [11:29] these were easier and we actually think [11:31] the if you look at the number of xxx's [11:33] in our KVM Port it kind of tells the [11:35] tale of the port here so here you can [11:37] see Max is struggling to get the damn [11:38] thing to compile and uh XXX is climbing [11:41] up here uh and then of course you know [11:43] we're we're pulling some stuff in then [11:45] Max realizes oh there's a whole [11:47] file I haven't pulled in got to pull [11:48] that in that's all XXX out and we're [11:50] kind of rumbling along here um I think [11:52] that is actually what happened there Max [11:53] as I recall um and then we we hit April [11:56] here and we actually begin to make some [11:58] progress and now we're kind of ripping [11:59] stuff out and then we're making really [12:01] good progress actually going in [12:02] completing the port and pulling actually [12:04] everything in cleaning up the xxx's and [12:06] indeed the Milestones you can kind of [12:08] see the Milestones here um we kind of [12:09] broke through in the daylight here [12:11] booing kmdb which is our instit of [12:13] Colonel the bugger um once we could boot [12:15] kmdb we could actually see the guests [12:17] perspective of architecture state that [12:19] allowed us to make a lot faster progress [12:21] um booted Linux you know it's funny I [12:23] actually went back and looked at this [12:24] I'm like that could not have been only [12:26] like 10 days these were I actually think [12:28] the 10 most terrifying days of my life [12:30] um that felt like several years I I it's [12:33] amazing to me that such I had to go back [12:35] and quadruple check the dates because [12:37] this was a particularly brutal problem [12:40] um a very entertaining self-inflicted [12:42] wound which is essentially the story of [12:44] this port um and I'll get to the actual [12:46] problem there in a little bit um we [12:48] booted Windows shortly thereafter um I [12:50] guess I should have highlighted we [12:51] booted haikou in there I've become a [12:52] total fan of haiku I don't know I don't [12:54] know what os's you guys use to actually [12:56] when you go to verify new change but [12:58] Haiku does such weird to the system [13:00] you know it it it's it runs the clock [13:03] really high it just does a bunch of [13:04] weird stuff um and so we I love Haiku [13:07] man I think hiu is awesome um so um the [13:11] uh now and I say the good news the [13:14] reason you haven't gotten any Patches [13:15] from us and I really actually think this [13:16] is amazing I I did not I would not have [13:18] thought this going in uh in the course [13:20] of this port we did not discover a [13:22] single bug that you would call a bug in [13:24] KVM remarkably remarkably solid we are [13:28] cynical Engineers um so that was [13:31] surprising I think to all of us and it's [13:32] a real tribute Obby to you and your team [13:34] in the community that is a it is a solid [13:36] solid solid piece of code the only bug [13:38] that we found that is in KVM or Kimu was [13:41] a bug in Kimu where it was calling [13:44] signal inste set of Sig action on the on [13:46] Sig IPI and if we can have the extended [13:50] lecture on the the tortured history of [13:53] signal versus Sig action and why you [13:55] should never use signal to set signal [13:57] disposition to anything other than [13:58] ignored or TOA suffice it to say they [13:59] weren't um and the semantics on Linux [14:01] versus Al lumos are different there [14:03] totally and it's a very minor bug but [14:05] that was it that's it the rest of the [14:06] code Kimu KVM worked um and again it's a [14:10] huge tribute to this community and this [14:11] code base um that that works as well as [14:14] did I thought you might find some of our [14:15] self-inflicted wounds we certainly found [14:17] them entertaining so I thought you might [14:18] find them entertaining the the week that [14:20] it took us to debug Linux there were [14:21] like six problems in that week um one of [14:24] them was and I highly recommend that you [14:26] do not attempt this on your own [14:29] that we had misconfigured Kimu we [14:57] miscomputernet um Linux obviously does [15:00] not I'm going to assum you what's that [15:02] do then you got a bug okay wow we [15:04] actually might have a bug um if I really [15:07] okay then if you if you got pcbu uh if [15:09] you got a pcbu GS Bas then we actually [15:10] do have a patch for you oh my God I feel [15:12] so happy um the um because the GS Bas is [15:16] not properly restored on some thread [15:19] migrations um so there are some cases [15:21] when you do a thread migration and maybe [15:22] it's impossible maybe you can't hit it [15:24] in Linux for whatever reason but we were [15:26] able to hit it when we poured a bunch of [15:28] load on the system for a while we could [15:29] actually hit this um so the the but the [15:32] manifestation of this was terrific uh [15:34] and I've been doing col imputation 15 [15:36] years I not actually had this particular [15:37] failure mode I've said that a lot of [15:39] course but this particular failure mode [15:41] was a new one I had two different CPUs [15:45] one of which believed that it was the [15:46] other from a software perspective oh my [15:50] God the Mayhem because so like you [15:52] honestly you don't know I mean just [15:54] Linux say this as well you get to that [15:56] per CPU State tells you what thread you [15:58] are tells you who you are well you think [15:59] you're someone else but you're not [16:01] someone else you're you like you you're [16:02] just wrong on that on that perception [16:04] and so you go and you die on the most [16:07] up assertions of like wait a [16:09] minute what am I doing here I'm over [16:10] there it's like no you're not over there [16:11] you're over what the hell is going on [16:14] and it took us a long time and actually [16:15] this is one um we you know we've had a [16:18] lot of these bugs over the years I [16:19] honestly don't know how we would have [16:20] cracked this without Detro um we were [16:21] using Detroit and in particular dri has [16:23] the ability to do postor instrumentation [16:25] where you can have a dri enabling and [16:27] then when the system dies bring up have [16:29] the crash stump come back up and then [16:31] actually go through the Detroit output [16:33] um and so that was key for breaking this [16:35] for cracking this one the other one [16:37] another self-inflicted wound um and this [16:40] is actually this is a good lesson to us [16:41] in terms of being cautious when we go [16:45] reimplement uh functionality in KVM in [16:47] terms of the aluminos equivalence so the [16:49] FPU save routine there is an FPU save [16:51] routine that was in um in KVM or [16:54] actually in Linux and we just and we Dro [16:56] that in to our KVM well we we got [16:58] another one in alumo we might as well [16:59] make the call out to it so we replace [17:02] that with a call to save the FPU [17:03] registers well that call actually is [17:06] slightly different than just an fx save [17:08] it has an fx save and it also clears TS [17:10] and cr0 TS is the bit in cr0 that [17:13] indicates a task switch and that's what [17:15] you you are going to get a an NM you're [17:18] going to get a pound NM when you access [17:20] floating Point State we use that pound [17:21] NM like I imagine you guys do a lot of [17:24] operating systems do we use that pound [17:26] andm to restore your FPU State well if [17:28] you don't hit that pound NM you will go [17:29] flying past your first use of FPU [17:32] registers and you'll just start using [17:34] whatever's there it's like it's got [17:35] nothing to do with your registers it's [17:37] like well how often do you use FPU well [17:39] there's this little thing called mem [17:40] copy comes up a lot uh turns out if you [17:42] call M Copy you're using floating Point [17:44] registers to stage in memory and just [17:46] wild wild exciting things happen this is [17:48] like we're doing a full build in a KVM [17:50] guest and there's a corrupt object file [17:53] and you're just like okay where is the [17:54] colonel developer Suicide Hotline [17:56] because I actually need to call it right [17:57] now and be talked off the roof um but [18:00] fortunately we we gritted that one out [18:02] and nailed it uh in terms of performance [18:04] um you know not surprisingly um although [18:07] there's a an astris there as well um our [18:09] Port when you look at just pure CPU um [18:12] and this is a suspend CPU running uh [18:15] runs same speed as bare metal same speed [18:16] as Linux kbm same speed as smartos KVM [18:19] so that was that was very vindicating it [18:21] actually took us a surprising amount of [18:22] time to get to this result because when [18:23] we first did this the and this was dead [18:26] reproducible smart OS M was 5% faster [18:31] now and and you those of you that have [18:33] done benchmarking know that this is how [18:34] tough benchmarking is cuz hey you're 5% [18:36] faster hey go home you know it's like [18:38] hey you're 5% faster it's like no that [18:40] doesn't make sense like you can be [18:42] awesome but honestly like we're all [18:45] talking to the same Hardware here we [18:47] can't be 5% faster and these are the [18:49] exact same CPUs they are the exact same [18:51] motherboards exact same Dam going down [18:54] every single thing same same same same [18:56] same and finally Brandon Grant on the [18:58] joint team finally cracked the code on [19:00] this we were seeing Dynamic overclocking [19:02] right so the uh Intel's turbo boost [19:04] functionality in the nalum this [19:06] something be very careful of whenever [19:07] you're Bunch marking definitely turn [19:08] that sucker off because what that'll do [19:10] is based on ambient Temp and so on [19:11] that'll overclock you up now your TSC [19:14] frequency is the same so the only way to [19:16] determine your true true CPU frequency [19:18] now I think there actually are acpi regs [19:19] you can get the actual true CPU [19:20] frequency but needless to say you're [19:23] going to end up being very confused when [19:24] you go to compare results um so I would [19:26] love to tell you that we're 5% faster [19:27] but we're not so when we went to turn [19:29] that off everyone was the same speed [19:30] which was a relief so um and in terms of [19:33] we did a and we're still in the kind of [19:35] early stages of doing the benchmarking [19:36] here and I should just say that the the [19:38] point of our benchmarking efforts was to [19:39] make sure that we hadn't anything [19:41] up um because you know we had plenty of [19:43] self-inflicted wounds along the way [19:45] every reason to believe we would have [19:46] some more so we just want to make sure [19:47] that we were in the hunt um and so this [19:49] is a CIS bench MySQL Benchmark um you [19:52] can see that the bare metal performance [19:53] is there and here are uh Linux KVM and [19:56] smartos KVM at essentially the same [19:58] performance here there's kind of this [20:00] interesting Gap here we would want to [20:01] understand that due to CPU scheduling [20:03] perhaps I don't know why there's that [20:04] Crossing there um me the numbers are [20:06] reasonably reproducible um but we view [20:08] these as kind of basically statistically [20:10] being essentially the same thing which [20:11] is good we're delivering the same [20:13] performance that you're delivering and [20:15] you guys are delivering KVM is [20:16] delivering great performance so in terms [20:18] of the port status it's publicly [20:19] available there's a GitHub repo for for [20:20] KVM itself um we've got a um a branch of [20:24] Kimu [20:25] 0.14.1 very slightly different I'm going [20:27] to talk about those differences in a [20:28] second here um and within its scope um [20:32] in terms of the way we have kind of [20:33] defined the scope of this project um [20:35] this port for us is add or near [20:36] production quality if you have AMD svm [20:39] it is not add or near production quality [20:40] because there is no support for AMD svm [20:43] um so it depends on what you're doing [20:44] but for us on this port we we do believe [20:47] that we are approaching rapidly [20:48] production quality if we're not there [20:50] already uh actually very funny this [20:51] morning so we just released this this [20:53] morning um and not 45 minutes after we [20:56] released it um another drro of lumos [20:59] open Indiana picked this thing up on [21:01] latest bits and actually booted a guest [21:04] Windows 7 that we actually hadn't booted [21:06] yet uh and posted the screenshot of [21:08] Windows 7 you know booting and major [21:11] sigh of relief like boy that that could [21:13] have gone the other way um you know we [21:15] we we booted Windows 2008 R2 we booted [21:17] you know various Windows variants but [21:18] not actually gotten to Windows 7 and we [21:21] all know that anything can go wrong so [21:22] that was a relief um but this is [21:24] definitely usable as it is today um so I [21:27] want to go through some of the [21:29] limitations of our Port um some of the [21:31] things that that we have not done uh [21:33] some of things that we don't intend to [21:34] do um I want to go through the some of [21:35] the divergences where we implemented [21:37] things in a slightly different way or [21:38] use a different facility those are just [21:40] more for your curiosity um and then some [21:42] enhancements some ways that we think [21:43] that we're um we're actually doing some [21:45] really interesting things with the KVM [21:46] that we're really excited about so uh [21:47] limitation number one this is an [21:49] important one is guest memory is locked [21:50] down so we are biased by the fact that [21:52] we are a cloud provider so we actually [21:54] have thousands of machines tens of [21:56] thousands of customers uh we've been [21:58] doing this for a long time and for us [22:00] memory oversell is not worth it um [22:02] memory oversell it works great for idle [22:05] workloads but when you have a bunch of [22:07] folks that are actually going to kick [22:09] the snot out of the boxes the actual [22:11] variable latency that you get from [22:13] memory overell just does not pay for [22:15] itself in terms of the marginal revenue [22:16] so it is a business decision at joyant [22:19] and perhaps um something of uh of an [22:22] opinion albeit one that is is grounded [22:24] in fact that we do not believe in memory [22:27] oversell so we currently lock down guest [22:29] memory um and failure to lock down [22:31] memory will result in Failure to start [22:33] um this is a resolvable problem we're [22:36] just not hugely interested in resolving [22:37] it um because we don't we're not [22:38] interested in memory oversell we think [22:39] that someone in the community probably [22:40] will be interested in it and we would [22:42] love to help them out but we're probably [22:43] not going to drive this ourselves in [22:46] part because when you are talking about [22:47] high high multi tendency environments we [22:49] do believe that the hardware is actually [22:51] the wrong level to BR D hardware [22:52] virtualization is great we love it uh [22:54] and it's great when you want to run all [22:55] these different guests and so on but [22:56] when it's time to actually crank multi [22:58] tendency when you run one platform as a [22:59] service and get up the stack actual OS [23:01] level virtualization is a much uh [23:03] lighter way way to uh to virtualize [23:05] youve got a single operating system [23:06] kernel you can punch out these zones and [23:08] we run uh we run production loads with [23:11] 400 tenants per machine on a 2u machine [23:13] two machine 96 G memory has got 400 [23:15] tenants on it each with what what feels [23:17] like a virtual operating system running [23:18] at very high performance you can't do [23:20] that with hard virtualization you can't [23:21] it just can't it doesn't matter how [23:23] great you are um so that's another [23:25] reason why this is not hugely hugely [23:26] interesting for us we Al don't have any [23:28] memory DD um or Colonel Sage mapping in [23:31] Linux parlance uh it's technically [23:33] possible we don't see an acute need um [23:36] maybe we will see an acute need uh in [23:39] part because of the same reason we lock [23:40] down guest memory definitely interest to [23:41] hear experiences about this I mean we [23:42] heard um in an earlier talk that that [23:45] this is being used broadly we would love [23:48] to know what kind of memory savings do [23:49] people see uh are is there one kind of [23:51] guest that sees more I I think it is [23:53] kind of my inference from lore yes [23:56] SA uh uh does a lot of zeroing of free [24:01] free memory and it doesn't use all of [24:03] its memory for cach so if you have [24:04] Windows XP it will collapse all of those [24:07] zero pages into just one page and of [24:09] course all of the shared text will be [24:11] collapsed you can see significant saving [24:13] if you run um virtualized desktops then [24:16] usually they're running the same similar [24:18] workloads the same office we uh the same [24:21] colel so all those can be collapsed and [24:23] of course all those zero pages right [24:25] interesting so I would be very in the [24:27] Windows XP dat point is a very [24:28] interesting one I would just be curious [24:29] about what the rate is there is that is [24:30] that a 25% saving do you get a 25% bump [24:33] do you get a 50% bump do you get a you [24:35] know you get a 3X bump in your what is [24:38] how much do your memory reach how much [24:39] weall the exact numbers but we saw very [24:42] high significant yeah and I think you [24:44] know we'll definitely um this is not [24:46] something that we're we're closed to [24:48] having implemented is and this would be [24:49] a Al lumos thing I mean just the way it [24:51] was in in Linux this would be a generic [24:54] thing effectively um another question [24:55] that we would have is kind of what was [24:56] the performance overhead from p scanning [24:58] um that's an important question for us [25:01] uh we don't support nested [25:02] virtualization currently we not sure [25:04] that we're ever going to implement it uh [25:05] it's it's hard especially on btx on BMX [25:09] um it's just ugly um and I mean svm it's [25:12] it's it's a little more elegant um we [25:14] would love it for own development [25:16] purposes you know VMware if you want to [25:17] despite all of my kind of throwing you [25:19] under the bus if you'd like to implement [25:20] it in VMware Fusion that would be [25:21] awesome um we would love to see n [25:23] virtualization because we would use it [25:25] ourselves for development of course um [25:27] but we don't see the acute need for our [25:28] for ourselves again interested in hear [25:30] in experience experiences with this is [25:32] it being used in production or is this [25:34] primarily for development I mean I would [25:35] be I don't know I'd be in feel free to [25:38] it's currently a solution looking for a [25:40] problem [25:41] and that's that you've told me all I [25:44] ever need solution that is a problem [25:46] because it's incredibly difficult to to [25:48] support um the code is incredibly [25:51] complicated yeah we went through the [25:53] patches and it's like wow that is really [25:55] nasty and it's I mean nasty [25:58] because the hardware the hardware is not [25:59] helping you out I have to say the think [26:01] way to get your mind in a that loock [26:02] very very soon it's horrible uh [26:05] interesting those are very helpful data [26:07] points I don't know if anyone using [26:09] production be certainly using sure of [26:11] course I mean for development it makes a [26:12] ton of ton of sense but for development [26:15] it also makes a ton of sense for someone [26:16] else to go Implement right notation [26:19] development but actually application [26:21] development on top of right right sure [26:23] of course yeah that's your saying yep [26:25] and that's interesting I'll be [26:26] interested to hear if that goes into [26:27] production or to what degree those [26:28] become production loads um one [26:30] Divergence um we um we really want to [26:33] minimize the patches load on top of Kimu [26:35] we don't have an a non iode get FD [26:37] equivalent so there is not a way to [26:40] actually open a new FD from within a [26:42] driver's IO routine effectively which is [26:44] what you guys are doing here um we you [26:47] guys do this to implement the model of a [26:48] one toone mapping um between a file [26:50] script and a vcpu we definitely we [26:52] actually experimented with well let's [26:53] just break that mapping and have the [26:55] vcpu be a parameter every octal it's [26:56] like oh we've completely diverge okay [26:58] never mind unwind unwind unwind what's [27:00] add a new KVM clone ioctl with a one new [27:03] ioctl that makes the the driver State [27:06] and the operated upon instance point to [27:07] effectively another and then to create a [27:09] vcpu we've got slightly more complicated [27:12] code in Kimu it's like you 10 lines of [27:13] code that reopens Dev C does a fresh [27:16] open the KVM calls KVM clone on the new [27:18] instance specifying the old one and then [27:20] is able to go create a vcpu on it we've [27:21] got all the the errors in there that you [27:23] can't create multiple vcpus on a single [27:24] instance and so on so that that has [27:26] worked out very well for us with the the [27:28] caveat that we got a small amount of if [27:30] defy in Kimu which obviously we' like to [27:32] avoid but we can't in this case um one [27:35] very minor Divergence um we've got [27:37] something called Contact stops in lumos [27:39] uh actually these were originally [27:39] implemented kind of interesting [27:40] historical detail for CPU performance [27:42] counter virtualization uh I don't know [27:44] if you guys do the same thing um but we [27:45] virtualized CPU performance counters and [27:47] have for many many years um in part [27:49] because we were trying to understand a [27:50] microprocessor that was enormously [27:52] non-competitive which was spark um so we [27:54] really needed every bit we could get um [27:57] so um we implemented these contact Ops [27:59] and we actually use them to great effect [28:00] in KVM um where we perform the [28:03] equivalent of a KVM SC in KVM SC out um [28:05] so you get code that can be called as [28:06] you're being taken off CPU so that was [28:07] if we didn't have that we would have had [28:09] to implement it which would have been [28:09] Rocky uh for timers uh lumus has got an [28:12] arbitrary resolution Interval Timer [28:13] facility that I implemented way back in [28:15] the day um in the late 90s it's called [28:17] the CX subsystem um CX can be bound to a [28:19] CPU or processor set that can be figured [28:21] to fire different interrupt levels um [28:23] and originally this was designed to be a [28:25] high resolution interval facility um and [28:27] indeed the system clock itself is [28:28] implemented in terms of it um but over [28:30] the years we've actually made some [28:31] enhancements to it and they can now be [28:32] dynamically reprogrammable as one shots [28:35] so all KVM timers are implemented as [28:37] cyclic um one slight difference we don't [28:39] migrate cyclic when a vcpu migrates from [28:41] one CPU to another instead we actually [28:43] just do we poke the CPU poke the target [28:45] CPU from the c or caner so very minor [28:47] Divergence terms of enhancements um you [28:50] know I kind of threw this up there [28:51] because I know this is we believe this [28:53] is an acute problem uh in KVM and in the [28:55] broader world we actually didn't do [28:57] anything thing to Z ZFS ZFS just works [28:59] with KVM so you take a take a ZFS create [29:02] a zall and and have Kimu and KVM go to [29:05] town on it um so that just works but by [29:08] using ZFS uh you get a couple of [29:11] enhancements um for one you can stop the [29:14] experiment of the user level file system [29:16] in KK 2 um which was tough to sit [29:20] through um I mean not it's very Earnest [29:21] work and I don't mean anything against [29:23] the [29:24] presenters the um uh the the Pres ZFS [29:27] allows us to do a lot of things um it [29:29] allows for constant time cloning which [29:31] we've used a lot um as we have uh we've [29:34] been just testing this stuff um it's [29:36] great to be able to just clone an [29:37] instance um and be able to go you can [29:39] clone it while it's up no problem um [29:40] we've got a unified adaptable [29:41] replacement cache The Arc um that allows [29:44] for guest iio to be efficiently cashed [29:46] in the host um and that allows us we [29:48] want to actually use the dram that way [29:50] because it allows um potentially massive [29:52] performance improvements in random iio [29:54] of course it depends on locality um and [29:56] we believe that the CVS remote [29:58] replication facility can provide a [29:59] really uh efficient Foundation to do [30:02] cloning migration in the WAN so that's [30:04] part of where we're going to be taking [30:05] this uh in terms of um we also support [30:08] OS virtualization um our implementation [30:09] does not require um the a to run an OS [30:12] virtualized instance um but um we chose [30:15] to for for our implementation at joyant [30:18] uh and the reason we did that is [30:19] actually because we use it as a [30:20] mechanism for ACC accounting um we use [30:22] it as the basis for quality of service [30:23] Resource Management uh IO throttling [30:25] billing ination Etc so that is the [30:27] reason we actually run the Kimu process [30:29] in his Zone it's the only process [30:30] running in his Zone um but then all [30:33] these KVM vulnerabilities came out um at [30:35] at the the black hat conference and [30:37] which became very interesting to us [30:38] because if you break out of and I [30:40] thought it was actually very interesting [30:41] and again obvious a tribute to you and [30:43] and this team this community the [30:44] vulnerabilities were not in KVM right [30:46] the vulnerability you though it was it [30:47] was Pitch is a kbm vulnerability it's a [30:49] chemu vulnerability you were breaking [30:50] out in the kemu well if you break out in [30:52] the chemo you if you somehow burrow out [30:54] in the chemo in our implementation you [30:55] just find yourself having burrowed out [30:57] into a much larger much more secure [30:59] prison um which is a container so you [31:01] can't actually do anything um you you're [31:04] in fact you actually have less [31:05] privileges there than you do in a normal [31:06] container so we think that the OS [31:08] fertilization really neatly containerize [31:10] chemo um and drastically reduces the [31:12] attack surface there um We've Got Deep [31:14] support for Network virtualization uh we [31:16] create a virtual Nick per KVM guest uh [31:18] and we wrote some simple glue to connect [31:20] this to verdio um and you know we just [31:23] wanted to satisfy ourselves that we [31:24] could push one gbit um to and from a KVM [31:27] G um we were able to do that so we we [31:29] kind of um called it went on to do other [31:31] things um so we we know it performs at [31:34] least adequately but I'm sure there's [31:35] lots we can go to there v-x give us a [31:37] ton of of things that are I would say [31:40] possible um with IP tables EP tables and [31:42] so on but much much much easier um you [31:45] just create a v-neck and you assign it [31:46] to to the the Kimu instance um you get [31:48] anti spoing for example anti spoing just [31:50] kind of comes along for the ride you [31:51] indicate on the phoenic which Macs this [31:53] thing is allowed to have and no compon [31:54] interface that doesn't have that Mac uh [31:56] it gives you flow management so you can [31:57] actually share a 10 GB interface and [31:59] actually chunk that up among 10 [32:01] different guests and assure that no one [32:03] can actually drown it um or if you are [32:05] actually this becomes really critical um [32:07] if you are in certain parts of the [32:08] developing world where bandwidth is is [32:10] at a premium you really need to confine [32:13] bandwidth from a particular guest that [32:14] allows us to do it very easily we also [32:16] get nice resource management [32:17] observability into a pervic and th PR [32:18] guest through butut from the host that [32:20] nicely comes along for the ride uh lumos [32:23] has a has a Kat facility for colonal [32:25] statistics uh we took the KV vcpu stat [32:28] uh structure we implemented that as a [32:29] Kat and then we added a KVM stat tool to [32:32] lumos that that's been pushed U that's [32:34] Upstream now that consumes the K stats [32:36] and displays them per second in per vcpu [32:39] um so we use this all the time when [32:40] we're looking at performance so what we [32:42] do this is for example this is 1 second [32:44] of KVM stat output uh this is a Linux um [32:47] there's a 2 vcpu Linux instance here [32:49] that's idle and a 4 vcpu um uh smart OS [32:53] instance that's actually in the middle [32:54] of booting so you can see and you guys [32:55] recognize all of these these are the [32:57] things that that you added that we've [32:59] effectively exported up um one just side [33:02] note here this can drive you nuts um we [33:04] were seeing um that when a machine first [33:06] booted it would take, 1300 exits per [33:09] second and then as we were debugging it [33:11] it would go away and then what the hell [33:13] you go reboot the instance and then it [33:15] would come back 1,300 exits per [33:17] second and we're debugging and debugging [33:19] debugging it would go away and finally [33:21] Josh Wilston who's a joint engineers in [33:22] the room Josh is like wait a minute I [33:25] wonder if this is due to the screen [33:26] blanking and sure enough it's that five [33:28] minute timeout before the screen blanks [33:30] it's doing um and it we were we were [33:32] seeing uh a bunch of EP EPT violations [33:35] as it was writing to video memory [33:37] presumably so um kind of entertaining [33:40] but a very good way of seeing this [33:41] across many many many vcpus and VMS um [33:45] in terms of dra trce um I think it's [33:46] kind of ironic uh Kimo 014 has drace [33:49] probes even though it's never actually [33:50] worked with drace because the to [33:51] actually support system tab which is [33:53] kind of funny um so we actually lit [33:55] those suckers up and those all lit up [33:56] great right uh then we added a whole [33:58] bunch of St probes at KVM itself all the [34:00] call sites of the trace underbar [34:01] routines became sat probes um gives them [34:04] zero probe effect when they're disabled [34:05] we added the VM regs registers I I [34:07] mentioned and that allows you to [34:09] dynamically safely uh aggregate uh [34:11] instrument on an arbitrary basis per VP [34:13] vcpu VM and so on and it's very nice to [34:16] use this with KVM stat to understand [34:18] workload so for example here's a very [34:19] simple descript that looks at Guest exit [34:22] and Aggregates by process ID thread ID [34:25] and the actual exit reason and then once [34:27] a second kind of prints this thing out [34:28] as a nice report we're running this Fork [34:31] exact exit workload maybe this is this [34:34] is probably not news to anybody but um [34:37] so when you start a Linux process it [34:39] executes CPU ID three times um in [34:42] userland um I maybe that's faster than [34:45] maybe there's no other way to do it I [34:46] was surprised by that so I would to um [34:48] we only discovered this a day or two ago [34:50] um and I would love to hear kind of I'm [34:52] sure there's a story there um so anyway [34:54] you see high exits we're saying 8,000 [34:55] exits per second running this workload [34:57] that just Forex Forex and exits um which [34:59] was kind of interesting um so um just [35:02] and just got a couple more slides to [35:03] wrap up here um totally orthogonal to [35:05] this work but complimentary to it um we [35:08] developed the realtime analytics [35:09] framework that instruments the cloud [35:11] using dce and visualizes the results and [35:13] then we've been extending this facility [35:15] into the new D probes that we built into [35:17] KVM and we've only been experimenting [35:19] with it and only very recently but the [35:21] results have been really interesting so [35:22] here is uh ext3 right offset so this [35:26] this is time on the x- axis this is [35:29] offset on the actual logical volume on [35:32] the Y AIS and this is a workload that is [35:36] writing three gigs over and over again [35:37] it's writing three gigs deletant writing [35:39] three gigs and so you can [35:41] see as it writes up the platter which [35:43] good I mean this is ext3 is doing what [35:46] we would want it to do but this is this [35:47] is interesting that from the kind of the [35:49] bottom looking up we can actually infer [35:51] things about the guest and uh understand [35:53] perhaps a little bit of what the guest [35:55] is doing um here here's one that's [35:57] really wild um so we have this [35:59] interesting idea well we can't tell you [36:01] what process is running but we know what [36:03] cr3 looks like and we know that cr3 has [36:06] a pretty strong correl cr3 is your mmu [36:08] context cr3 has a pretty strong [36:11] correlation to process I don't know what [36:13] it's called but I know it's hex value um [36:15] so what we can actually go do is watch [36:18] thread or process scheduling and this is [36:21] a single vcpu instance um this is [36:23] running two compute B workloads we're [36:24] idle here and then we start writing [36:26] these workloads and you can see the lxer [36:28] trading these things off just as you [36:30] would expect so wow this is interesting [36:32] let's let's amp it up so now at 999 [36:35] Hertz and let's take one of those [36:37] compute bound processes and rice it and [36:39] you can see that this guy basically [36:41] falls off and just kind of appears every [36:42] once in a while um and then we we kind [36:45] of flip it around you've been on too [36:46] long okay you get you get off and now [36:48] we're going to run this other guy and we [36:49] get to see interesting stair step [36:51] pattern I'm sure there's something um [36:53] I'm sure there's a not a doctoral thesis [36:55] but maybe a master's thesis probably an [36:56] undergrad graduate thesis waiting to be [36:57] written on what's going on there um same [37:00] view same sample frequency but now [37:01] horsing around at different ice values [37:03] uh and seeing all sorts of of crazy [37:05] things things that you would expect [37:07] right and so by the I should have I [37:09] should have made this clear what we're [37:10] seeing here what we are seeing is the [37:13] the time on the x- axis and the offset [37:16] in the second on the Y AIS that's [37:19] sampled so what you are effectively [37:20] seeing if you were to take every strip [37:22] and flip it 90 degrees and line them all [37:25] up end to end you would have the Linux [37:27] scheduling Behavior I ran yellow I ran [37:29] blue I ran yellow I ran blue I ran [37:31] yellow I ran blue and so on uh we don't [37:33] know what yellow is the process named 0x [37:36] 67 D800 not terribly useful but um at [37:40] least gives you Insight that you might [37:42] not have had otherwise here's a wild one [37:44] the last one I've got here this is irq [37:46] requests again broken down by subc [37:48] offset and Vector we know what irqs were [37:50] dropping on you let's go look at those [37:53] and here we can see that there's very [37:54] clearly some time related vector here [37:56] these are time vectors here and so when [37:59] you're seeing these striations you've [38:00] got something that's very time based you [38:02] can see some of these diagonals that [38:04] will have that are probably time based [38:06] but relative instead of absolute and [38:07] then you have these kind of [38:08] free-for-alls here where you have a [38:10] bunch of interrupts smeared all across [38:12] the second so again interesting uh not [38:15] sure if you can do anything with this [38:16] but interesting to stare at nonetheless [38:18] uh in terms of engaging the community [38:19] we're very excited um to to engage the [38:22] KVM Community uh a couple of three [38:24] concrete things we would love to work [38:26] with the community on um one is just [38:27] working on KVM performance um with dra [38:30] trce we have much better visibility in [38:31] the guest behavior um and to us it seems [38:34] pretty likely that wins in I mean our [38:36] KVM implementations have got so much in [38:38] common that wins in one are probably [38:40] going to be wins in another so let's [38:42] work together and let's just let's go [38:44] kick the snot out our proprietary [38:46] hypervisors um by understanding KVM [38:49] better and better and better on [38:50] arbitrary guests in production so that's [38:53] something we would really like to work [38:54] with you guys on um we'd also like to [38:55] collaborate on testing um we we dream [38:58] and and perhaps this exists perhaps this [39:00] dream is already a reality of a farm of [39:02] Oddball isos in particular os2 um a [39:05] buddy of mine from school uh was one of [39:07] the very early guys at VMware um and he [39:09] always told me that os2 once they could [39:12] boot os2 they could ship um because os2 [39:15] was so up in every single thing [39:17] it did um so I would love to have an os2 [39:20] ISO I me I fig if we Bo boot os2 we are [39:23] done um so that would be great um we [39:25] would also to and benchmarking I haven't [39:27] taken apart spec at SC 2010 spec and I [39:30] have a a long and complicated history [39:33] with one another um it's like a like a [39:35] girlfriend with whom I had a violent [39:37] breakup so I you know I but I I'm that [39:40] said it's great to see we have this new [39:43] um spec Benchmark and I would like to [39:44] understand it I'd like to understand [39:45] your perspective on it hopefully you [39:47] guys are involved in that um and we [39:49] would like to work with the community [39:50] develop some standard benchmarks this is [39:51] actually a lower priority for us I I [39:53] kind of don't care about that as much I [39:55] really care about these first two things [39:56] quite a bit um just terms of thank you [39:59] um Josh wilon and Rob go from joyan are [40:01] in the room they would instrument on [40:02] this prent Greg um for um for all the [40:05] performance work and on the overclocking [40:06] for Brie bard of course I feel is like [40:08] the kind of the unspoken spirit in the [40:11] room um for lighting the path with Kimo [40:13] uh Intel for a RI and fast CPU along [40:15] with EP a you and your team for putting [40:17] it all together with KVM and of course [40:19] the lumos Community for their [40:20] enthusiastic support so thank you very [40:25] much [40:26] [Music]