Netflix Deploys FreeBSD Trunk in CDN Fleet to Catch Regressions Early
OpenFest Bulgaria
Tracking bleeding-edge development code in production eliminates multi-year upgrade debt and enables rapid detection of upstream regressions that degrade server performance.
AI-assisted summary of Why We Run FreeBSD current at Netflix – Drew Gallatin — original source. Check the original for full context.
Source published:
Key ideas
- Netflix tracks FreeBSD-CURRENT directly in production across its Open Connect CDN, merging upstream code every three to four weeks to catch regressions immediately and eliminate multi-month upgrade debacles. (8:10)
- By co-designing hardware with FreeBSD kernel optimizations—including async sendfile, unmapped mbufs, and NIC TLS offload—Netflix scaled single-server CDN throughput from 100 Gbps in 2017 to 800 Gbps in 2022. (21:20)
- An experimental sub-100-watt, 100 Gbps edge server prototype runs FreeBSD directly on an Nvidia BlueField DPU, bypassing traditional server motherboards and host CPUs entirely to stream video to tens of thousands of subscribers. (22:56)
- A famous upstream boot-time sort optimization exposed a decade-old driver ordering flaw, where an inverted alphabetical tie-break caused an obsolete Pentium 4 thermal driver to attach over Intel SpeedStep and degrade CPU performance by 20%. (27:00)
Detailed notes
## Background Published by OpenFest Bulgaria on YouTube on February 18, 2024, this presentation features veteran FreeBSD committer and Netflix engineer Drew Gallatin discussing Netflix's Content Delivery Network (Open Connect). The talk details why Netflix tracks the development branch FreeBSD-CURRENT in production, outlines their major upstream contributions and bandwidth milestones, and dissects a specific upstream bug that triggered a 20% performance regression.
## TL;DR Drew Gallatin details how Netflix uses FreeBSD to power its global fleet of Open Connect Appliances, pushing bandwidth as high as 800 Gbps per server for static streaming workloads. The team transitioned from tracking FreeBSD-STABLE to running FreeBSD-CURRENT directly in production, enabling rapid upstream collaboration and preventing multi-month upgrade ordeals. Gallatin reviews core architectural enhancements engineered for FreeBSD—including asynchronous sendfile, unmapped mbufs, and Kernel TLS with NIC offload—before walking through the 'Magical Mystery Merge,' a case study where a widely praised boot-time sorting optimization inverted subsystem initialization and caused competing CPU frequency drivers to throttle production machines by 20%.
## Atmosphere & style The presentation carries the warm, unpretentious tone of an experienced systems engineer swapping war stories with peers. Gallatin speaks with understated humor and dry pragmatism, turning what could be dense kernel plumbing into an engaging technical mystery. It feels like an authentic insider's look into high-stakes production engineering, balancing pride in engineering milestones with candid reflections on the operational perils of bisection loops and buggy hardware firmware.
## Bulletin - [08:10](https://www.youtube.com/watch?v=q4TZxj-Dq7s&t=490s) Netflix tracks FreeBSD-CURRENT directly in production across its Open Connect CDN, merging upstream code every three to four weeks to catch regressions immediately and eliminate multi-month upgrade debacles. - [21:20](https://www.youtube.com/watch?v=q4TZxj-Dq7s&t=1280s) By co-designing hardware with FreeBSD kernel optimizations—including async sendfile, unmapped mbufs, and NIC TLS offload—Netflix scaled single-server CDN throughput from 100 Gbps in 2017 to 800 Gbps in 2022. - [22:56](https://www.youtube.com/watch?v=q4TZxj-Dq7s&t=1376s) An experimental sub-100-watt, 100 Gbps edge server prototype runs FreeBSD directly on an Nvidia BlueField DPU, bypassing traditional server motherboards and host CPUs entirely to stream video to tens of thousands of subscribers. - [27:00](https://www.youtube.com/watch?v=q4TZxj-Dq7s&t=1620s) A famous upstream boot-time sort optimization exposed a decade-old driver ordering flaw, where an inverted alphabetical tie-break caused an obsolete Pentium 4 thermal driver to attach over Intel SpeedStep and degrade CPU performance by 20%.
## Key takeaways - [00:18](https://www.youtube.com/watch?v=q4TZxj-Dq7s&t=18s) Netflix Open Connect Appliances function as single-purpose network appliances using dual A/B partition schemes for robust failover during operating system updates. - [03:20](https://www.youtube.com/watch?v=q4TZxj-Dq7s&t=200s) Netflix formats video storage partitions with UFS rather than ZFS because UFS integrates directly with the kernel page cache, avoiding the double-copy overhead of ZFS ARC. - [03:20](https://www.youtube.com/watch?v=q4TZxj-Dq7s&t=200s) Edge streaming workloads are strictly static and pre-encoded into dozens of display variants, eliminating runtime transcoding and focusing server resources entirely on network and disk I/O. - [08:10](https://www.youtube.com/watch?v=q4TZxj-Dq7s&t=490s) Tracking FreeBSD -STABLE releases led to massive merge friction, upstream divergence, and multi-month upgrade efforts, driving Netflix to track the development trunk (-CURRENT) directly. - [11:02](https://www.youtube.com/watch?v=q4TZxj-Dq7s&t=662s) Developing against -CURRENT allows Netflix engineers to submit bug fixes directly upstream for community review rather than maintaining internal forks, shrinking internal tree divergence. - [13:56](https://www.youtube.com/watch?v=q4TZxj-Dq7s&t=836s) Asynchronous sendfile enables the web server to return immediately to its event loop while the storage controller's interrupt completion handler directly hands disk blocks to the network stack. - [13:56](https://www.youtube.com/watch?v=q4TZxj-Dq7s&t=836s) Unmapped mbufs allow kernel buffers to hold references to multiple physical pages without updating page tables, substantially reducing socket-buffer pointer-chasing and CPU cache misses. - [13:56](https://www.youtube.com/watch?v=q4TZxj-Dq7s&t=836s) Hardware-level NIC TLS offload executes payload encryption directly on the network adapter during transmission, reducing CPU crypto overhead to virtually zero. - [16:48](https://www.youtube.com/watch?v=q4TZxj-Dq7s&t=1008s) Custom TCP stacks like RACK paired with high-precision TCP pacing prevent intermediate residential router buffer bloat, prioritizing subscriber streaming smoothness over raw benchmark speeds. - [16:48](https://www.youtube.com/watch?v=q4TZxj-Dq7s&t=1008s) Physical memory pointer (PMP) hooks allow early firewall packet inspection and immediate buffer recycling, dropping tens of millions of DDoS packets per second without kernel allocation overhead. - [21:20](https://www.youtube.com/watch?v=q4TZxj-Dq7s&t=1280s) Single-server streaming capacity evolved from 100 Gbps via Kernel TLS in 2017 to 400 Gbps on single-socket AMD Rome with NIC offload in 2021, reaching 800 Gbps on dual-socket systems in 2022. - [27:00](https://www.youtube.com/watch?v=q4TZxj-Dq7s&t=1620s) Replacing an $O(n^2)$ selection sort with an unstable quicksort during SYSINIT kernel initialization inadvertently reversed the boot-time registration order of tied subsystems. - [29:46](https://www.youtube.com/watch?v=q4TZxj-Dq7s&t=1786s) Because CPU frequency drivers used binary return codes instead of graded probe scores, reverse-alphabetical sorting allowed an obsolete P4 thermal driver to attach first and improperly throttle Intel Xeon cores. - [33:19](https://www.youtube.com/watch?v=q4TZxj-Dq7s&t=1999s) Running upstream trunk code allowed Netflix to identify the subtle frequency control bug immediately, report it directly to the patch author on IRC, and verify an upstream fix within 24 hours. - [36:30](https://www.youtube.com/watch?v=q4TZxj-Dq7s&t=2190s) Operating tens of thousands of edge appliances requires staggered weekly rollouts and cautious validation of hardware firmware after past NIC firmware bugs caused unrecoverable I2C timeouts. - [38:40](https://www.youtube.com/watch?v=q4TZxj-Dq7s&t=2320s) Netflix's continued reliance on FreeBSD is driven entirely by pragmatic engineering performance and stability requirements rather than ideological attachment.
## Detailed walkthrough ### [00:18](https://www.youtube.com/watch?v=q4TZxj-Dq7s&t=18s) Introduction and Overview of Netflix's CDN
Drew Gallatin, introduced at OpenFest Bulgaria, has worked at Netflix for about eight years specializing in FreeBSD kernel optimizations, and has been a FreeBSD committer for more than twenty-five years. He explains that his presentation will cover how Netflix uses FreeBSD, why the company tracks the `FreeBSD-CURRENT` development branch on its content delivery network (CDN), and an instructive case study titled the "Magical Mystery Merge," which illustrates the real-world advantages of running `-CURRENT`.
Netflix calls its CDN Open Connect, and the specialized edge servers deployed across the network are termed Open Connect Appliances (OCAs). Gallatin points out that in operational design, an OCA behaves much more like an appliance—such as a home router—than a traditional office or enterprise server. System software updates are delivered via a dual-partition scheme: the B-side is freshly imaged, and if a boot or update fails, the machine falls back automatically to the A-side partition. Every OCA runs identical base software with no one-off custom packages installed on individual boxes.
```mermaid flowchart LR accTitle: Open Connect Appliance Partition Update Flow accDescr: Diagram showing the A/B image update mechanism used on Netflix Open Connect Appliances. subgraph Update["OCA Storage Partitions"] A["Active Partition (A-Side)"] B["Target Partition (B-Side)"] end Image["New Image Deployment"] -->|"writes to"| B B -->|"boot failure fallback"| A ```
- OCAs can push immense bandwidth, with top-tier units capable of serving up to 400 Gbps. - Netflix deploys three distinct OCA server profiles: - **All-Flash OCAs:** Equipped with NVMe or older SATA SSDs; deployed at internet interchange points and data centers to serve the most popular content. - **Storage OCAs:** Pack hundreds of terabytes of disk capacity to store long-tail assets (from rarely viewed titles to full library catalogs). - **Global OCAs:** Placed directly inside smaller ISP networks to localize streaming traffic, offloading the ISP's upstream transit links when users stream movies.
---
[03:20](https://www.youtube.com/watch?v=q4TZxj-Dq7s&t=200s) Storage Architecture, Filesystems, and Static Workloads
Gallatin addresses one of the most common questions he receives: whether Netflix uses ZFS on its content appliances. The answer is yes, but strictly for the root filesystem. Netflix avoids using ZFS for video content partitions because of caching architecture: ZFS relies on its own Adaptive Replacement Cache (ARC), which sits separately from the operating system kernel's page cache. Serving content directly out of ZFS requires data to be copied from the ARC into the page cache, introducing memory copying overhead and degrading streaming throughput. Consequently, Netflix formats all content partitions with UFS (Unix File System), which integrates directly with the kernel's native page cache.
Netflix pairs UFS with the nginx web server and intentionally runs without software or hardware RAID on its content drives. Drives are simply allowed to fail in place. Gallatin highlights that managing replacement drives through third-party ISPs or dispatching field technicians to remote data centers is operationally expensive and unnecessary; because appliances house enormous aggregate storage, losing one or two drives leaves the machine entirely viable and capable of continuing service.
The appliance workload consists strictly of serving static pre-encoded files. Before any title reaches the CDN, Netflix encodes it into dozens of variants covering different display targets (smart TVs, mobile phones, computers) and varying bitrates. Because every variant is pre-computed to maximize video quality, the CDN edge servers execute zero runtime compute or transcoding, leaving the workload cleanly focused on raw network and disk I/O.
---
[05:26](https://www.youtube.com/watch?v=q4TZxj-Dq7s&t=326s) FreeBSD Lineage and the OCA Dev Team
Gallatin contextualizes FreeBSD's design by comparing it to Linux. While Linux was written from scratch, FreeBSD traces its lineage directly back to the original BSD releases that underpinned 1980s systems like SunOS. FreeBSD emerged from 386BSD, which had fallen into maintenance neglect. Two developer groups maintaining patch kits ultimately diverged: NetBSD formed with an emphasis on code correctness, portability, and multi-architecture support, while FreeBSD prioritized pure performance on x86 32-bit hardware. Gallatin notes that his own involvement as a FreeBSD committer began in the late 1990s while working on FreeBSD's first 64-bit port, the DEC Alpha, which was eventually sunsetted once the vendor discontinued the architecture.
Unlike standard Linux distributions where virtually all software is broken into modular binary packages, standard FreeBSD maintains a traditional separation: the kernel, core utilities (`ls`, `mv`, `cp`), standard shells, and base compilers live unpackaged in the base system root. Third-party utilities (like nginx and Firefox) are handled via packages. Netflix maintains a proprietary, stripped-down distribution that eliminates unneeded compilers, manuals, and packages from the OCAs.
The OCA Dev team responsible for this software stack is a focused group of roughly ten engineers, most of whom are longtime FreeBSD committers or contributors. Gallatin expresses great personal enthusiasm for the group, noting:
> “there's roughly 10 of us and it's a great place to work and it's the best job I've ever had”
---
[08:10](https://www.youtube.com/watch?v=q4TZxj-Dq7s&t=490s) The Pitfalls of Tracking Stable Branches
Historically, Netflix followed the conventional enterprise strategy: tracking a FreeBSD `-STABLE` branch. In FreeBSD terminology, `FreeBSD-CURRENT` is the active development trunk where all changes land first (analogous to the Linus tree in Linux), while `-STABLE` branches act as release and Long-Term Support (LTS) maintenance tracks. FreeBSD targets a major release every two years, though Gallatin observes it often stretches to every three or four years in practice.
Initially, the team tracked a `-STABLE` branch, merging security and bug fixes every few weeks and building internal OCA images. While this cadence functioned smoothly during the lifetime of a single release, upgrading from one major `-STABLE` branch to the next was punishing:
> “what was terrible was when we move from one stable Branch to another and that would take sometimes in the best case the weeks most most times it would take months you know up to six months because you could encounter a problem uh you know either in code that we never managed to Upstream uh being compatible with the new interfaces in the new in the new version or there could be uh some kind of regression introduced Upstream in the new version that was hard to track down and so these these merges every time we updated between stable branches were just awful”
Tracking `-STABLE` also crippled upstream open-source collaboration. Because internal code diverged from `-CURRENT`, contributing a fix or optimization required backporting it to `-CURRENT`, figuring out how to test it without running `-CURRENT` in production, and shepherding it through review—only to wait years to consume it in the next stable branch. This friction led Netflix to hoard internal changes, generating massive technical debt and horrific merge conflicts with every upgrade cycle. Around five to six years prior, the team decided to invert the model entirely and track `FreeBSD-CURRENT` directly in production. Gallatin acknowledges that this initially sounds reckless because FreeBSD operates without subsystem maintainers: any committer with a commit bit has direct commit access across the tree, guided by internal cultural expectations of review, freedom, and responsibility.
[11:02](https://www.youtube.com/watch?v=q4TZxj-Dq7s&t=662s) Tracking FreeBSD-CURRENT and Upstreaming Fast
Drew Gallatin explains that running FreeBSD-CURRENT inside Netflix's Open Connect content delivery network might sound reckless at first glance, but in practice, it is completely transformative. By sticking to a tight merge cadence of every three to four weeks, the team pulls in upstream FreeBSD code constantly. This prevents the traditional two-to-three-year lag where a bug or regression sits buried in a code base until an enterprise finally migrates to a newer release. Instead, regressions are caught immediately. Furthermore, developing directly against CURRENT removes the friction of having to backport features across widely divergent OS versions, allowing Netflix engineers to collaborate seamlessly with the wider open-source community.
Over the years, this dynamic has caused Netflix's internal delta against upstream to steadily shrink. Because their private tree is nearly identical to upstream FreeBSD, Netflix engineers rarely implement small features or minor bug fixes internally first. Instead, they write them directly in the upstream FreeBSD tree, put them through upstream code review, commit them, and simply let them flow back into Netflix on the next three-week merge cycle. This workflow drastically minimizes internal technical debt. Critical security patches or crash fixes are cherry-picked into Netflix's deployment immediately rather than waiting for the cycle. The rare exceptions are massive structural undertakings, like Kernel TLS (kTLS): Drew Gallatin notes that “that was a very large very invasive change and so we kept that to ourselves for close to 5 years,” not out of a desire for proprietary secrecy, but because it took that long to architect it cleanly so that it would be genuinely useful to the broader community rather than just solving Netflix's narrow internal use case.
```mermaid flowchart TD accTitle: Netflix FreeBSD Merge and Contribution Loop accDescr: Diagram showing how Netflix develops features upstream or pulls changes via a three-week merge cycle.
subgraph Upstream ["FreeBSD Upstream (CURRENT)"] UP_REPO["FreeBSD Tree"] UP_REV["Code Review & Commit"] end
subgraph Internal ["Netflix Open Connect Environment"] NF_DEV["Small Feature / Bugfix"] NF_TREE["Netflix Internal Branch"] NF_CHERRY["Critical / Emergency Fixes"] LARGE_DEV["Long-Term Projects (e.g., kTLS ~5 yrs)"] end
NF_DEV -->|"Submit directly upstream"| UP_REV UP_REV --> UP_REPO UP_REPO -->|"Every 3-4 week merge cycle"| NF_TREE UP_REPO -->|"Cherry-pick immediately (security/crashes)"| NF_CHERRY NF_CHERRY --> NF_TREE LARGE_DEV -->|"Refactor for community & upstream"| UP_REPO ```
- A Jenkins job triggers an automated build and test run on every branch push. - Netflix targets two primary server architectures: amd64 (noted humorously as what Microsoft calls amd64 while the rest of the world calls it x86-64) and arm64. - Smoke tests run almost nightly on a dedicated cluster with full kernel debugging enabled: memory corruption detection, use-after-free checking, and lock-order reversal (WITNESS) verification. - Release-candidate testing runs on a broader set of hardware with all debugging checks disabled, focusing strictly on throughput, latency, and performance regressions.
---
[13:56](https://www.youtube.com/watch?v=q4TZxj-Dq7s&t=836s) Major Upstream Contributions: Async Sendfile, Unmapped mbufs, and kTLS
Looking back at Netflix's legacy of contributions to FreeBSD, Drew Gallatin singles out asynchronous `sendfile`—developed by his colleague Gleb Smirnoff in collaboration with NGINX—as the single most impactful architectural advancement for their CDN. Normally, `sendfile` blocks the calling thread while data is fetched from disk into memory before passing it to a network socket. Async `sendfile` completely restructures this interaction: the web server issues the system call and immediately returns to its event loop. The kernel pulls the requested data from storage in the background, and the disk controller's interrupt completion handler directly hands the data off to the network stack. Drew Gallatin emphasizes that “that allows us to avoid having thread pool tools or using async iio or doing any any any sort of thing like that and that's that one thing has is the fundamental optimization it's enabled everything else.”
To build toward kernel-level TLS, the team also revamped FreeBSD's core networking memory buffers by creating unmapped mbufs (modeled similarly to Linux's `skb_frag_t`). Instead of mapping every chunk of memory and pointing to single data blocks, an unmapped mbuf holds references to multiple physical pages without touching the kernel page tables. A single mbuf can encapsulate an entire TLS record containing four or five 4KB pages alongside headers and cryptographic signatures, eliminating socket-buffer pointer-chasing and drastically slashing CPU cache misses even when TLS is unused.
Building atop unmapped mbufs, Netflix implemented Kernel TLS (kTLS), shifting bulk payload encryption out of user-space libraries like OpenSSL and into the kernel itself. This eliminates the expensive double trip across the user-kernel boundary where encrypted data had to be read into user memory and written back down to the kernel. Taking this a step further, modern network adapters—such as Mellanox ConnectX-6 Dx (or newer) and Chelsio T6—support NIC TLS offload. Under this architecture, the network interface card executes the crypto operations autonomously in hardware on transmit. Data travels from storage straight onto the wire while the kernel never maps or reads the payload memory at all, rendering the performance overhead of encryption virtually zero from the host CPU's perspective.
---
[16:48](https://www.youtube.com/watch?v=q4TZxj-Dq7s&t=1008s) Storage Scheduling, Custom TCP Stacks, and DoS Mitigation
Beyond socket primitives, Netflix contributed the CAM I/O scheduler to optimize disk behavior. Under FreeBSD, CAM handles the storage subsystem; Netflix needed a mechanism to fill local Open Connect appliances (OCAs) with fresh catalog titles from peer nodes at the exact same time the server was actively streaming video to users. Heavy write workloads severely degraded read latency on solid-state drives, so the CAM scheduler modulates read and write bandwidth to guarantee playback streams remain fluid without storage-induced jitter.
In networking, Randall Stewart ported Google's BBR congestion control algorithm to FreeBSD and authored RACK entirely from scratch at Netflix. While BBR is famous for its role in HTTP/3 and QUIC, Netflix primarily relies on RACK for its streaming workloads. Alongside RACK, Netflix introduced hardware/software pacing subsystems (such as high-precision timer-based TCP pacing). As Gallatin illustrates, blasting a burst at 100 Gbps toward a residential customer whose cable modem buffer only holds 32KB would instantly overflow the intermediate buffer, dropping packets and triggering cascading TCP retransmissions. Pacing trickles data out smoothly at the exact delivery rate the subscriber connection can sustain.
```mermaid flowchart LR accTitle: Packet Drop via Physical Memory Pointer Hooks accDescr: Flow comparison between standard firewall drops and fast-path recycling via physical memory pointer hooks.
subgraph FastPath ["PMP Hooks (Netflix/XDP-style Fast Path)"] direction TB RX1["NIC Driver receives packet"] --> PMP["Firewall inspects via PMP pointer"] PMP -->|"Drop verdict"| RECYCLE["Driver immediately recycles buffer"] end
subgraph SlowPath ["Traditional Network Stack Drop"] direction TB RX2["NIC Driver allocates new buffer"] --> MAP["Map new DMA / Unmap old"] MAP --> CHAIN["Traverse network stack (Cache misses)"] CHAIN --> FW["Firewall inspects & drops"] FW --> FREE["Free memory pages"] end ```
To counter distributed denial-of-service (DDoS) attacks, Netflix introduced physical memory pointer (PMP) hooks, directly inspired by Linux's XDP (eXpress Data Path). Instead of letting malicious packets flow through standard kernel allocation paths—where the driver must allocate replacement buffers, reprogram DMA descriptors, and pass the data up the stack while burning CPU cache lines—PMP hooks give the firewall an immediate pointer to the packet fresh off the wire. If the packet matches an attack signature, the driver drops it on the spot and immediately recycles the underlying buffer without touching DMA mappings or allocating new memory. This mechanism allows Netflix appliances to discard tens of millions of attack packets per second without stealing a single cycle from legitimate video delivery.
- Scalability improvements upstreamed over the years are so numerous Gallatin remarks they are “probably more than I can remember.” - Warner Losh ported `kboot` (which originally originated on PowerPC 64) to amd64 and arm64, enabling servers running LinuxBIOS without standard UEFI firmware to `kexec` directly into FreeBSD. - Netflix uses its purchasing power to expand hardware support: when acquiring new enterprise hardware that only ships with Linux or Windows drivers, Netflix actively pairs with vendors and provides resources to write and upstream production-grade FreeBSD drivers. - Both Netflix as an organization and the engineers individually contribute financial support to the FreeBSD Foundation. - In his day-to-day role driving performance, Gallatin warns that naive speedups can be counterproductive: stripping out RACK and TCP pacing would theoretically yield higher local throughput benchmarks, but Drew Gallatin emphasizes that “that would make our members quality of experience terrible,” completely violating Netflix's core mission of eliminating video buffering stalls.
[21:20](https://www.youtube.com/watch?v=q4TZxj-Dq7s&t=1280s) Trade-offs in Performance and Hardware Milestones
Drew Gallatin explains the delicate balancing act required when tuning FreeBSD for Netflix’s Content Delivery Network (CDN). Features like TCP pacing consume noticeable CPU cycles, but disabling them comes at a direct cost to user experience: “like if I got rid of the TCP pacing, well CPU would go way down, but our members would see a lot more of the little spinning wheels for rebuffering.” Consequently, performance tuning is not merely about minimizing raw CPU utilization. Instead, the team’s core objectives are maximizing bandwidth output per machine and lowering overall power consumption so identical traffic volumes can be served from smaller footprints.
```mermaid flowchart LR accTitle: Netflix CDN Hardware Performance Milestones accDescr: Timeline of Netflix CDN single-server bandwidth throughput milestones from 2017 to 2022. Y2017["2017: 100 Gb/s<br/>(Kernel TLS)"] Y2020["2020: 200–240 Gb/s<br/>(AMD Rome)"] Y2021["2021: 400 Gb/s<br/>(NIC TLS Offload)"] Y2022["2022: 800 Gb/s<br/>(Dual Socket AMD)"]
Y2017 --> Y2020 --> Y2021 --> Y2022 ```
Gallatin highlights several historical CDN milestones achieved through continuous kernel and hardware co-design:
- **2017 (100 Gb/s):** Believed to be the world's first production 100 Gb/s CDN server, enabled by the introduction of Kernel TLS (kTLS). - **2020 (200 Gb/s):** Deployed AMD Rome-based single-socket systems capable of delivering approximately 240 Gb/s. - **2021 (400 Gb/s):** The same 2020 hardware reached 400 Gb/s once NIC TLS offload was supported in software and enabled across the fleet, moving the cryptographic operations out of the host CPU directly down into the network card. - **2022 (800 Gb/s):** Achieved on a dual-socket system configured with twice the NICs and twice the NVMe storage of the 2020 design.
[22:56](https://www.youtube.com/watch?v=q4TZxj-Dq7s&t=1376s) The Sub-100-Watt 100 Gb/s Server Prototype
Looking toward future efficiency, Gallatin presents an ongoing experimental prototype: delivering 100 Gb/s of traffic under a 100-watt power envelope. The objective is remarkable in perspective: “for the power that you used to have for a light bulb, you can serve 20, 30,000 customers video.” This design bypasses standard server motherboards entirely by running FreeBSD on an Nvidia BlueField DPU (Data Processing Unit).
The system consists of a 100-gigabit BlueField card housing 16 ARM cores and roughly 48 GB of DDR5 memory, plugged into a PCIe expansion backplane alongside NVMe flash storage without any traditional host CPU. While wall power currently measures around 125 watts, Gallatin explains that two factors artificially inflate this number: the lab testbed is powered by an oversized 1200-watt power supply running well below its peak efficiency curve, and the chassis holds excess NVMe drives added to quickly draw traffic during tests. Swapping to an appropriately sized power supply and removing redundant drives is expected to drop system draw below the targeted 100-watt mark.
[24:28](https://www.youtube.com/watch?v=q4TZxj-Dq7s&t=1468s) The Mystery 20% Performance Regression and Git Bisection
Turning to the talk's main case study, Gallatin outlines why tracking upstream FreeBSD-CURRENT is vital by detailing an incident from August. After pulling in an upstream merge that seemed remarkably straightforward—bearing only two trivial merge conflicts—testing on Netflix's baseline 100 Gb/s reference machine revealed an immediate problem. At standard throughput of around 92 Gb/s, CPU consumption jumped roughly 20%, drifting from its typical low-40% baseline up into an erratic, jagged 50%-plus profile. Standard profiling tools revealed no obvious culprits: no single function showed up excessively hot, and no lock contention explained the overhead.
```mermaid flowchart TD accTitle: Git Bisect Process Across Upstream and Netflix Subtree accDescr: The four-hour verification loop used to bisect FreeBSD-CURRENT commits into Netflix's tree. subgraph BisectLoop["Bisection Loop (~4 Hours per Step)"] A["Pick Upstream Git Commit"] --> B["Merge Commit into Netflix FreeBSD Subtree"] B --> C["Resolve Subtree Conflicts & Build Image"] C --> D["Deploy & Run 92 Gb/s Production Load Test"] D --> E{"CPU Normal or 20% High?"} end E -->|"Bad"| F["Mark Bad, Bisect Backward"] E -->|"Good"| G["Mark Good, Bisect Forward"] F --> A G --> A ```
With no clear lead from profiling, Gallatin turned to git bisect to isolate the fault. Because Netflix maintains its operating system source inside a subtree of a larger internal repository, every single bisection step required finding an upstream hash, merging it into the subtree, manually resolving repeated merge conflicts, compiling a full bootable image, deploying it to hardware, and running validation traffic tests. The entire feedback loop took approximately four hours per step, dragging out into what Gallatin jokes was “one eternity later” before finally isolating the offending commit.
[27:00](https://www.youtube.com/watch?v=q4TZxj-Dq7s&t=1620s) Colin Percival’s Famous Boot Sort Optimization
The commit isolated by git bisect appeared completely unrelated to network performance or runtime CPU utilization. Authored by FreeBSD committer and Tarsnap founder Colin Percival, the patch had gained wide attention across the FreeBSD community and Hacker News for dramatically speeding up boot times in cloud environments like Amazon AWS:
> “The funny thing is this is like the most famous FreeBSD commit that I can remember.”
Percival had discovered that kernel initialization spent substantial boot time running an inefficient $O(n^2)$ sorting routine across `SYSINIT` entries, which he initially identified as a bubble sort. In FreeBSD, every kernel subsystem uses the `SYSINIT` macro to register initialization hooks. The linker compiles these into an alphabetical list, which the kernel then sorts at boot time across 79 subsystem tiers and roughly eight sub-priorities per tier (e.g., run first, run second, run last, or don't care). Items with matching subsystem and priority keys are considered tied and intended to execute in arbitrary order.
However, the legacy sorting implementation had actually been an unstable selection sort rather than a bubble sort. As a result, tied entries that historically retained their initial linker alphabetical order were inverted into reverse-alphabetical order under the new quicksort implementation. Percival and Gallatin confirmed this behavior using Percival’s `tslog` boot-profiling utility. Flipping the sort order in the new algorithm immediately restored Netflix's normal runtime CPU performance, yet the core question remained: why would reversing the initialization order of tied subsystems cause an active 20% CPU spike during steady-state network delivery?
[29:46](https://www.youtube.com/watch?v=q4TZxj-Dq7s&t=1786s) Isolating the Driver Subsystem and Frequency Control
To uncover the underlying mechanism, Gallatin modified the kernel to manually steer subsystem ordering through boot arguments rather than rebuilding full kernel images every time. He first bisected across high-level subsystems, which swiftly identified the driver subsystem as the source of the regression. Within the driver subsystem, a second layer of bisection pinpointed the true culprit: competing CPU frequency control drivers attaching to the Intel Xeon processor.
FreeBSD's driver model relies on a probe score mechanism where drivers inspect target hardware and report their suitability:
> “The way drivers are supposed to work in FreeBSD is they're supposed to be a probe score, so every driver kind of gets to look and see, 'Oh yeah yeah, I know what that hardware is, and that's exactly what I was made for and I want that 100%,' or, 'You know, I think maybe I can handle this hardware, but I'm not sure, and if somebody else, you know, better comes along you should give it to them.'”
Instead of utilizing graded probe scores, multiple competing CPU frequency drivers were returning a coarse, binary success status (1 or 0). Consequently, driver attachment defaulted entirely to a first-come, first-served basis, leaving hardware binding dependent on the linker's initial ordering and the subsequent sort behavior.
### [31:40](https://www.youtube.com/watch?v=q4TZxj-Dq7s&t=1900s) Uncovering the Driver Conflict Drew explains that the performance regression was caused by a subtle ordering issue in how drivers attach to the kernel. A newer Intel EST (Enhanced SpeedStep) driver and an older Pentium 4 thermal control driver were both present; alphabetically, the newer driver attached first, suppressing the older one. When the ordering was reversed, the older, less capable driver attached first, effectively blocking the newer one from functioning correctly.
This oversight persisted for nearly a decade because the system "worked accidentally." To prevent such occurrences, his colleague, Warner Losh, is currently implementing explicit "probe scoring" for drivers to ensure ties are broken predictably. The bug was identified because the P4 thermal driver provided nonsensical frequency and power data, causing the CPU to run at an incorrectly throttled, slower speed.
### [33:19](https://www.youtube.com/watch?v=q4TZxj-Dq7s&t=1999s) Community Interaction and Fixes Drew highlights how the collaboration with the FreeBSD community was seamless. After identifying the problematic commit, he reached out to the developer, Colin, via an internal IRC channel. Because the code was fresh, Colin remembered the context, verified the issue, and provided a fix that was merged into FreeBSD-current the following day. Drew immediately cherry-picked the fix into their branch.
He notes that this incident showcases the strategic value of running "current" code. By running server-class hardware at scale, Netflix often acts as an early warning system for the community. Had the bug remained undiscovered for years, it would have been much harder to diagnose or argue for a fix, as developers might have dismissed the report as user error or misconfiguration.
### [36:30](https://www.youtube.com/watch?v=q4TZxj-Dq7s&t=2190s) Deployment and Fleet Operations Responding to an audience question about the time required to upgrade their massive fleet of tens of thousands of servers, Drew explains that they use a staggered rollout. Upgrades are performed in waves by their operations team to mitigate the risk of unforeseen bugs, typically taking about a week to complete.
He shares a cautionary tale regarding the dangers of firmware updates. Years ago, a new firmware blob for a network interface card (NIC) caused rare, one-in-ten-million I2C read timeouts during routine temperature monitoring. This would cause the NIC to reset and go offline. Because this happened at locations without IPMI (Intelligent Platform Management Interface) access, it created a severe operational crisis. This experience led Netflix to be much more conservative, often maintaining their own validated copies of firmware rather than blindly adopting upstream versions.
### [38:40](https://www.youtube.com/watch?v=q4TZxj-Dq7s&t=2320s) FreeBSD vs. Linux and Closing Thoughts Addressing a question about why Netflix chooses FreeBSD over Linux, Drew avoids a "flame war" but emphasizes that the decision is purely pragmatic. He points out that he has extensive experience across various operating systems—including Linux, Solaris, macOS, and VMware—and is not dogmatic about the technology.
“We're not stupid and we wouldn't be running FreeBSD if it didn't work better in this application for us,” he says. He notes that while the initial historical decision may have been influenced by the FreeBSD license, the current choice is driven by the fact that it remains the best tool for their specific needs. The session concludes with the moderator thanking him for sharing his work and presenting him with a token of appreciation from the organizers.
## Notable quotes > [38:40](https://www.youtube.com/watch?v=q4TZxj-Dq7s&t=2320s) We're not stupid and we wouldn't be running FreeBSD if it didn't work better in this application for us — Drew Gallatin
> [21:20](https://www.youtube.com/watch?v=q4TZxj-Dq7s&t=1280s) if I got rid of the TCP pacing, well CPU would go way down, but our members would see a lot more of the little spinning wheels for rebuffering. — Drew Gallatin
## Mentioned Drew Gallatin, Colin Percival, Gleb Smirnoff, Randall Stewart, Warner Losh, Netflix, Open Connect, Open Connect Appliances, FreeBSD, FreeBSD-CURRENT, FreeBSD-STABLE, FreeBSD Foundation, Linux, NetBSD, 386BSD, SunOS, Solaris, macOS, VMware, NGINX, ZFS, UFS, kTLS, CAM I/O scheduler, BBR, RACK, XDP, kboot, Jenkins, Intel Xeon, Intel EST, Pentium 4, Nvidia BlueField DPU, AMD Rome, Mellanox ConnectX-6 Dx, Chelsio T6, Tarsnap, tslog
## Video description We run FreeBSD-current on our CDN at Netflix. By running FreeBSD-current, we can find and fix bugs far more effectively than by running an older FreeBSD-stable branch. This talk describes our team, how we track FreeBSD-current, our contributions to FreeBSD, and some of the performance milestones we've hit. It will end with a description of a finding and fixing very interesting bug in upstream FreeBSD that caused a 20% performance regression.
#OpenFest 2023 I Share Therefore I Am
Website: https://openfest.org/ Facebook: https://www.facebook.com/OpenFestBulgaria Linkedin: https://linkedin.com/company/openfest-bulgaria/ (X) Twitter: https://twitter.com/OpenFestBG Instagram: https://www.instagram.com/OpenFestBulgaria Mastodon: https://mastodon.social/@openfest
[Source](https://www.youtube.com/watch?v=q4TZxj-Dq7s)