Netflix Pushes Single-Server Streaming to 380 Gbps via Hardware TLS

EuroBSDCon

Moving TLS encryption from the host CPU to inline network interface cards eliminates the memory bandwidth bottleneck that caps software-based streaming at 240 Gbps.

AI-assisted summary of Serving Netflix Video at 400Gbps on FreeBSD by Drew Gallatin — original source. Check the original for full context.

Source published: 2021-09-27

Key ideas

Netflix reached 380 Gb/s of TLS-encrypted video delivery from a single FreeBSD server powered by an AMD EPYC processor and Mellanox ConnectX-6 Dx NICs, falling just short of the 400 Gb/s wire target. (26:48)
Software kTLS hit a hard memory bandwidth wall at ~240 Gb/s because each payload byte crosses the memory bus four separate times (storage to RAM, RAM to CPU, CPU to RAM, RAM to NIC), requiring 200 GB/s of bandwidth against a 150 GB/s physical ceiling. (3:45)
Offloading encryption inline to the NIC bypassed the host CPU memory detour entirely, reducing bus traversals to the equivalent of serving unencrypted video and allowing total output to surge past 300 Gb/s. (16:47)
Production deployments require dynamic fallback mechanisms because TCP retransmissions force the NIC to re-DMA full TLS records from byte zero to reconstruct cipher state, turning packet loss into a severe PCIe bus penalty. (20:09)

Detailed notes

## Background Presented by Drew Gallatin of Netflix and published by EuroBSDCon on YouTube on September 27, 2021, this conference presentation examines the engineering push to serve TLS-encrypted video at 400 Gb/s from a single FreeBSD server. According to the talk description, Gallatin covers memory bandwidth bottlenecks in software kTLS, PCIe challenges with hardware crypto offload, NUMA topology optimizations, and comparative benchmark results across AMD Rome, AMD Milan, Ampere Altra arm64, and Intel Ice Lake platforms.

## TL;DR Drew Gallatin details Netflix's architectural journey to scale TLS-encrypted video streaming from 200 Gbps up to a 400 Gbps target on single-socket FreeBSD systems. After hitting a 240 Gbps host memory bandwidth ceiling caused by multi-pass software encryption, Netflix transitioned to inline NIC kTLS on Mellanox ConnectX-6 Dx adapters to eliminate the CPU crypto detour and cut memory bus traffic in half. Gallatin recounts overcoming critical production firmware hurdles—including PCIe Relaxed Ordering defects, session thrashing 'tar pits', and TCP retransmission bandwidth penalties—by pairing hardware offload with network-centric NUMA siloing. The resulting optimizations pushed AMD EPYC systems to 380 Gbps while yielding key architectural insights across alternative Ampere Altra arm64 and Intel Ice Lake configurations.

## Atmosphere & style The presentation carries a candid, pragmatic, and refreshingly unvarnished systems engineering tone, marked by seasoned war stories and wry humor. Gallatin eschews corporate gloss to speak directly as an operating system practitioner, openly dissecting frustrating firmware roadblocks like vendor 'tar pits' and locked-down 'BIOSes from hell'. Watching the talk feels like sitting beside a veteran kernel engineer walking through real hardware counter logs, blending rigorous data analysis with genuine excitement for low-level performance tuning and open-source community collaboration.

## Bulletin - [26:48](https://www.youtube.com/watch?v=_o-HcG8QxPc&t=1608s) Netflix reached 380 Gb/s of TLS-encrypted video delivery from a single FreeBSD server powered by an AMD EPYC processor and Mellanox ConnectX-6 Dx NICs, falling just short of the 400 Gb/s wire target. - [03:45](https://www.youtube.com/watch?v=_o-HcG8QxPc&t=225s) Software kTLS hit a hard memory bandwidth wall at ~240 Gb/s because each payload byte crosses the memory bus four separate times (storage to RAM, RAM to CPU, CPU to RAM, RAM to NIC), requiring 200 GB/s of bandwidth against a 150 GB/s physical ceiling. - [16:47](https://www.youtube.com/watch?v=_o-HcG8QxPc&t=1007s) Offloading encryption inline to the NIC bypassed the host CPU memory detour entirely, reducing bus traversals to the equivalent of serving unencrypted video and allowing total output to surge past 300 Gb/s. - [20:09](https://www.youtube.com/watch?v=_o-HcG8QxPc&t=1209s) Production deployments require dynamic fallback mechanisms because TCP retransmissions force the NIC to re-DMA full TLS records from byte zero to reconstruct cipher state, turning packet loss into a severe PCIe bus penalty.

## Key takeaways - [00:00](https://www.youtube.com/watch?v=_o-HcG8QxPc&t=0s) Netflix aims to maximize single-server efficiency running FreeBSD Current and NGINX to deliver 400 Gb/s of TLS-encrypted video across four 100 GbE interfaces. - [03:45](https://www.youtube.com/watch?v=_o-HcG8QxPc&t=225s) Software kTLS requires 200 GB/s of memory bandwidth to sustain 400 Gb/s of traffic due to four separate memory bus crossings per byte, capping an AMD Rome system at 240 Gb/s. - [06:46](https://www.youtube.com/watch?v=_o-HcG8QxPc&t=406s) Saturating the NUMA interconnect with bulk streaming data acts like highway traffic congestion, delaying urgent cross-domain atomic operations and stalling CPU execution cores. - [10:20](https://www.youtube.com/watch?v=_o-HcG8QxPc&t=620s) Uncoordinated NUMA allocation can force up to four cross-node interconnect traversals per packet, generating up to 200 GB/s of fabric traffic and severely choking throughput. - [12:05](https://www.youtube.com/watch?v=_o-HcG8QxPc&t=725s) Network-centric siloing aligns socket memory, file buffers, and worker threads directly with the incoming NIC port, outperforming disk-centric organization and lifting software kTLS to 280 Gb/s. - [16:47](https://www.youtube.com/watch?v=_o-HcG8QxPc&t=1007s) NIC kTLS (hardware inline offload) eliminates the CPU memory detour by having the network card encrypt TLS records directly on transmit across the PCIe bus. - [20:09](https://www.youtube.com/watch?v=_o-HcG8QxPc&t=1209s) Handling TCP retransmissions in hardware kTLS incurs a heavy PCIe bandwidth penalty because the NIC must re-DMA the entire TLS record from byte zero to regenerate intermediate cipher state. - [23:17](https://www.youtube.com/watch?v=_o-HcG8QxPc&t=1397s) Early ConnectX-6 beta firmware created a 'tar pit' where session state paging across PCIe degraded sustained per-NIC throughput down to 75 Gb/s without triggering health check alerts. - [26:48](https://www.youtube.com/watch?v=_o-HcG8QxPc&t=1608s) Fixing PCIe Relaxed Ordering and enabling Mellanox's tls_optimized firmware setting elevated aggregate system delivery to 380 Gb/s at roughly 60% CPU utilization. - [30:24](https://www.youtube.com/watch?v=_o-HcG8QxPc&t=1824s) A protective kernel policy that evicts lossy connections exceeding a 1% retransmission threshold to software kTLS caused aggregate throughput to drop from 380 Gb/s to 350 Gb/s. - [34:02](https://www.youtube.com/watch?v=_o-HcG8QxPc&t=2042s) Ampere Altra arm64 systems reached 320 Gb/s with hardware kTLS after enabling 8-bit PCIe Extended Tags to increase outstanding in-flight bus transactions. - [37:07](https://www.youtube.com/watch?v=_o-HcG8QxPc&t=2227s) Intel Ice Lake testing was constrained by locked firmware knobs preventing relaxed ordering and an artificial processor memory clock clamp to 2933 MHz. - [42:30](https://www.youtube.com/watch?v=_o-HcG8QxPc&t=2550s) Netflix rejects full TCP Offload Engines (TOE) to retain complete kernel control over TCP congestion control and transport logic, which is critical for Quality of Experience (QoE). - [44:37](https://www.youtube.com/watch?v=_o-HcG8QxPc&t=2677s) FreeBSD's kTLS architecture encapsulates an entire TLS record within a single mbuf, simplifying zero-copy driver handling during TCP packet retransmissions.

## Detailed walkthrough ### [00:00](https://www.youtube.com/watch?v=_o-HcG8QxPc&t=0s) Serving 400 Gbps: Motivation, Workload, and Platform

Drew Gallatin introduces his role at Netflix, where his work focuses directly on CPU efficiency and maximizing the delivery capacity of single-server nodes. He summarizes the overarching mission bluntly: “basically CPU efficiency and trying to serve as many customers as we can from as little hardware as we can, both to save the environment and to save ourselves some costs.” Having spent the preceding period pushing hardware limits, he sets out to explain how close Netflix can get to delivering 400 Gbps of TLS-encrypted traffic out of a single FreeBSD machine. Since 2020, Netflix has deployed production servers capable of serving 200 Gbps of encrypted video. However, those deployed appliances physically house four 100 GbE network interfaces, sparking the engineering effort to determine whether all four could be driven concurrently at full line rate.

The production software stack runs FreeBSD Current (staying within a few weeks of FreeBSD head) and powers the NGINX web server. Netflix serves almost all video content via `sendfile()`, making Kernel TLS (kTLS) an essential design requirement. By handling encryption directly inside the kernel's `sendfile()` pipeline, the system avoids extra boundary crossings where content would otherwise be pulled from storage, transferred into user space for encryption, and then copied back to the kernel. Software kTLS alone yields roughly a 60% savings in CPU utilization.

The core production platform used throughout these benchmarks is built around an AMD EPYC 7502P “Rome” processor featuring 32 cores running at 2.5 GHz. Gallatin notes that conversions between networking units (gigabits per second) and storage/memory bandwidth (gigabytes per second) require tracking an eight-fold factor. The platform includes: - An AMD EPYC 7502P (Rome) 32-core single-socket processor (2.5 GHz). - 8 channels of DDR4-3200 memory, providing 150 GB/s (roughly 1.2 Tbps) of rated memory bandwidth. - 128 PCIe Gen4 lanes, providing approximately 2 Tbps of aggregate I/O throughput. - 2 Mellanox ConnectX-6 Dx network interface cards (NICs), each utilizing a PCIe Gen4 x16 link and providing two 100 GbE ports (for a total of four 100 GbE ports). - An array of PCIe Gen3 NVMe solid-state drives, which were specced prior to the availability of full-speed Gen4 options.

---

[03:45](https://www.youtube.com/watch?v=_o-HcG8QxPc&t=225s) The Memory Bandwidth Wall of Software kTLS

Initial testing on the Rome platform using software kTLS capped server output at approximately 240 Gbps, running straight into a memory bandwidth wall. To prove this ceiling without relying on guesswork, Gallatin tracked memory usage using `amd-pcm` (AMD Processor Counter Monitor), a tool available online that reads system performance counters directly. The measurements confirmed that the machine exhausted its theoretical system memory bandwidth right at 240 Gbps, at which point performance began to collapse.

To illustrate how software encryption devours memory bandwidth, Gallatin maps the end-to-end data lifecycle of a typical client streaming session. When a user streams video, the client-side algorithm issues a request to stay buffered a few megabytes ahead (for instance, requesting the next 2 megabytes). NGINX receives the HTTP request and triggers a `sendfile()` call to fulfill it: 1. **Disk to Memory Read:** The NVMe drives stream raw video data into system memory buffers (aggregating 50 GB/s to sustain a 400 Gbps network pipe). 2. **Memory to CPU Read:** The CPU pulls the data out of the page cache to perform encryption. Gallatin notes an important structural constraint: file data cannot be encrypted in-place. If an in-place transformation occurred on a shared page cache buffer, a subsequent stream to another client would be double-encrypted with mismatched keys, producing garbled data. Consequently, the CPU must allocate a separate kTLS crypto buffer. 3. **CPU to Memory Write:** The encrypted ciphertext is written back into memory at 50 GB/s. 4. **Memory to NIC DMA:** The network interface card reads the encrypted buffer across the memory bus via DMA to push it onto the wire at 50 GB/s.

Because each byte traverses the memory bus four separate times, sustaining a target wire speed of 400 Gbps (50 GB/s) requires a staggering 200 GB/s (1.6 Tbps) of aggregate memory bandwidth—far exceeding the platform's 150 GB/s physical memory limit.

```mermaid flowchart TD accTitle: Memory Bandwidth Multiplier in Software kTLS accDescr: Diagram tracing the four memory crossings required to serve and encrypt file data via software kTLS.

subgraph Storage["NVMe Storage"] D["NVMe SSDs"] end

subgraph Memory["Host Memory (RAM)"] PC["Page Cache Buffer"] CB["TLS Crypto Buffer"] end

subgraph Compute["Host Processor"] CPU["CPU Core (Crypto Engine)"] end

subgraph Network["Network Interface"] NIC["Mellanox ConnectX-6 Dx (Wire)"] end

---

[06:46](https://www.youtube.com/watch?v=_o-HcG8QxPc&t=406s) NUMA Topologies and Interconnect Bottlenecks

Facing memory bandwidth exhaustion in flat memory configurations, Gallatin explored whether non-uniform memory architecture (NUMA) tuning could extract more performance. AMD EPYC platforms allow configuring memory controllers in different NUMA topologies. Public STREAM benchmark results show that running Rome in a standard flat configuration yields around 150 GB/s, whereas partitioning the socket into four nodes per socket (NPS4) improves memory controller efficiency and pushes STREAM bandwidth up to roughly 175 GB/s.

Gallatin recaps NUMA fundamentals: in early symmetric multiprocessing systems, CPUs, memory, and devices were uniformly spaced, meaning software could treat all memory locations identically without latency or throughput variations. However, hardware scaling limits forced designers toward a "divide-and-conquer" approach, interconnecting distinct nodes via a dedicated NUMA bus. When a CPU, memory, storage device, and NIC remain within their local domain, transactions operate at full local speed; crossing the interconnect imposes latency and bandwidth penalties.

On AMD EPYC Rome processors, operating in the most efficient NPS4 mode multiplies the topology into four fully interconnected NUMA domains within a single physical socket. Gallatin explains that these interconnect paths are not without cost: - Inter-domain latency penalties range from 12 to 28 nanoseconds depending on the specific link traversed. - Individual interconnect links are limited to roughly 47 GB/s, with an aggregate interconnect ceiling of about 280 GB/s across the system.

Gallatin outlines his foundational architectural strategy: “Basically, the strategy that I came up with a few years ago is to keep as much of our bulk data off the NUMA fabric as we can.” He uses a highway analogy, warning that allowing high-volume bulk video data to flood the NUMA links acts like filling every lane with tractor-trailers, crowding out the agile commuter traffic. When the NUMA fabric is saturated with bulk stream transfers, urgent low-latency transactions—such as cross-domain atomic counter updates or process synchronization reads—get blocked behind bulk transfers, stalling CPU execution cores while they wait for simple cache lines to clear the bus.

[10:20](https://www.youtube.com/watch?v=_o-HcG8QxPc&t=620s) The NUMA Worst-Case Routing Scenario

Drew Gallatin walks through the catastrophic performance penalties incurred when a multi-node NUMA architecture is left unoptimized. In the worst-case scenario, an incoming client request lands on a network interface card attached to a specific NUMA node (such as the lower-left node). If the media content resides elsewhere, the operating system issues a read request to the storage drive, but foolish memory allocation choices place that buffer on a completely different node (the upper-right node). Because memory was allocated across domains rather than locally, the transaction crosses the interconnect fabric for the first time.

Next, a CPU must read that plaintext data out of memory to perform software TLS encryption. Instead of picking a CPU co-located on the same node, an uncoordinated scheduler picks a core residing on the lower-right node, forcing the data across the NUMA bus a second time. Once encrypted, the resulting ciphertext must be written into an allocated destination crypto buffer. Making another suboptimal placement decision places this buffer on the upper-left node, requiring a third traversal across the interconnect. Finally, when the packet is transmitted out to the client via the original NIC in the lower-left node, the data crosses the NUMA bus a fourth time.

```mermaid flowchart TD accTitle: Unoptimized Software TLS NUMA Traversal accDescr: Shows the four consecutive cross-NUMA bus traversals caused by uncoordinated memory, CPU, and NIC placement.

subgraph NodeLL["Node 0 (Lower Left)"] NIC_IN["Incoming NIC Request"] NIC_OUT["Outgoing NIC Transmit"] end

subgraph NodeUR["Node 1 (Upper Right)"] BufIn["Read Buffer (Plaintext)"] end

subgraph NodeLR["Node 2 (Lower Right)"] CPU_Enc["CPU Encryption Engine"] end

subgraph NodeUL["Node 3 (Upper Left)"] BufCrypto["Destination Crypto Buffer"] end

These compounding round trips severely choke system throughput. Gallatin points out that routing 50 GB/s of data across the interconnect four times pumps 200 GB/s across the bus: “and that's pretty close to the aggregate fabric bandwidth and that causes the fabric to congest and that's uh that's bad.” In an ideal setup, execution stays entirely confined to the local node where the request entered, leading to zero NUMA bus crossings and zero fabric congestion.

[12:05](https://www.youtube.com/watch?v=_o-HcG8QxPc&t=725s) Network-Centric Siloing with FreeBSD and LACP

To reconcile the theoretical ideal of zero crossings with operational realities, Netflix faced structural constraints: each serving appliance operates with only a single IP address and uses LACP (Link Aggregation Control Protocol) bonding across its interfaces. When using LACP, upstream switch link partners hash client flows—typically based on TCP source/destination ports and IP addresses—to shard traffic consistently across the physical links. As long as the link aggregation group (LAG) remains stable without flapping, incoming packets for any given TCP connection always arrive on the exact same physical NIC.

Faced with this behavior, Gallatin evaluated two architectural models: "disk-centric siloing," which organizes processing around where the content physically resides on storage, and "network-centric siloing," which organizes processing around the NIC port where the connection arrives. After pursuing both concepts, Gallatin concluded that network-centric siloing is dramatically superior: “and i believe that network-centric siloing is by far the best way to go.” Under this paradigm, FreeBSD allocates socket memory and network connections with strict affinity to the NUMA node tied to the receiving NIC. Buffers backing media files read from disk are allocated in the local node's memory, crypto buffers are kept local, and TLS worker threads, TCP pacers, and TCP timers are bound with local domain affinity before transmitting back out via the local NIC. All of this infrastructure was contributed directly upstream into the FreeBSD kernel.

- LACP flow hashing relies on TCP 4-tuples (IP addresses and ports), providing stable flow-to-NIC routing. - Media read buffers are populated by DMA directly into memory local to the handling NIC. - TLS crypto workers, TCP timers, and pacing mechanisms run bound to the local NUMA domain. - All optimizations were committed directly upstream to the FreeBSD codebase (FreeBSD 13 / CURRENT).

[13:49](https://www.youtube.com/watch?v=_o-HcG8QxPc&t=829s) Physical Constraints and Real-World NUMA Gains

Applying network-centric siloing to the worst-case scenario produces an immediate architectural improvement. When a connection enters on the lower-left node and requests content stored on a disk attached to the right side, the kernel allocates a local destination buffer on the lower-left node for the storage read, crossing the interconnect only once during disk DMA. Once in local memory, a local CPU performs the TLS encryption, writes the ciphertext into a locally allocated crypto buffer, and hands the packet to the local NIC. Instead of crossing the bus four times, the data traverses the fabric exactly once, dropping fabric load from 200 GB/s down to 50 GB/s.

However, hardware topologies disrupt this clean abstraction. The server contains four NUMA domains but only two dual-port 100GbE NICs (providing a total of 400 Gbps of network bandwidth). Because high-speed NICs are expensive and additional cards would be redundant, two of the four NUMA domains are left without a physical NIC. Gallatin developed kernel workarounds to assign network affinity to non-NIC nodes, but this structural asymmetry reintroduces cross-node hops: - In the worst case, outgoing packets must traverse the interconnect to reach an available physical NIC in another domain, bringing the worst-case bus load to 100 GB/s (two crossings). - In the average case, with four nodes, there is a 75% chance (3 in 4) that the requested disk content resides on a remote node. - With only two NICs distributed across four domains, there is a 50% chance that transmission requires traversing the fabric to reach an active NIC. - Combined, these probabilities average out to approximately 62.5 GB/s of data across the NUMA fabric.

While significantly better than the unoptimized 200 GB/s baseline, the performance impact on network throughput fell short of expectations: “it was no panacea but it gets us from about 240 to about 280 which is you know better than nothing but still you know not anywhere near our goals that might uh you know make it worth letting up a third link but definitely not lighting up all four links.”

[16:47](https://www.youtube.com/watch?v=_o-HcG8QxPc&t=1007s) Eliminating the CPU Detour with NIC kTLS

Hitting the ~280 Gbps ceiling made it obvious that optimizing software-based memory placement was not enough; the core bottleneck lay in the memory bandwidth devoured by TLS encryption itself. Gallatin notes that whether software encryption is executed by the host CPU or offloaded to a look-aside accelerator (such as Intel QAT or a Chelsio CCR card), the underlying limitation remains identical: plaintext must be read out of main memory and the resulting ciphertext written back. By eliminating this memory round trip, system memory bandwidth requirements are immediately cut in half. The data path reverts to the efficiency of serving unencrypted video—streaming straight from NVMe storage into main memory and directly out through the NICs without CPU intervention.

To achieve this, Netflix shifted to NIC kTLS (commonly referred to elsewhere as hardware inline TLS). Under this model, the initial TLS handshake and session negotiation still occur in user space, but once session keys are installed in the kernel, the kernel hands off crypto responsibility directly to the NIC. The network card encrypts TLS records on the fly as data streams across the PCIe bus during packet transmission:

> “so the tls records are encrypted by the nick as the data flows through it on transmit so the positive thing is there's no more detour through the cpu for crypto and we uh and we've reduced our memory bandwidth requirements”

Gallatin acknowledges an architectural tradeoff that may concern security teams: plaintext leaves main memory and traverses the PCIe bus in the clear, whereas traditional user-space TLS keeps payload data encrypted across all internal buses. For Netflix's content distribution workload, however, this exposure is perfectly acceptable.

For hardware enforcement, Netflix selected the Mellanox ConnectX-6 Dx, which provides inline cryptographic offload for both TLS 1.2 and TLS 1.3 using AES-GCM—the dominant cipher on the public web. A crucial operational feature of the ConnectX-6 Dx is its ability to retain cryptographic state within an in-flight TLS record. If the TCP stack sends the first few segments of a TLS record and must pause while waiting for client ACKs, the NIC preserves the cipher state across milliseconds or seconds. When the ACK arrives, TCP resumes transmission mid-record seamlessly. The only expensive exception occurs during out-of-order transmissions, such as TCP packet loss retransmissions, where the NIC no longer holds the state for that arbitrary historical point in the TCP stream and must perform extra processing to reconstruct context.

[20:09](https://www.youtube.com/watch?v=_o-HcG8QxPc&t=1209s) Hardware kTLS: Normal Transmit vs. Retransmit Behavior

Drew Gallatin explains the internal mechanics of hardware-offloaded kernel TLS (inline kTLS) using a diagram of host memory, the PCI Express bus, the network interface card (NIC), and a 100 GbE network. In host memory, an unencrypted TLS record sits inside the socket buffer, demarcated by TCP segment boundaries at byte offsets such as 0, 1448, 2896, and so forth. Under normal transmission, TCP informs the NIC that it wants to send the first few segments of a packet. The NIC then performs direct memory access (DMA) across the PCIe bus to fetch the plaintext data down to the adapter. Gallatin notes that how the NIC organizes this internally is completely abstracted from the operating system:

> “It may do something smarter than this; I don’t know how their NIC works, it may, you know, DMA 4K at a time, it may DMA the individual segments, I have no idea; it’s all hidden from us, but the NIC DMAs the plaintext data down, it encrypts it in the NIC, and then it sends it out on the wire.”

As TCP receives subsequent ACKs and more window space frees up, the stack requests transmission of the remaining segments. The NIC simply DMAs the next chunks, encrypts them on the card, and pushes them out onto the wire. It requires no redundant work when pausing and resuming in the middle of a TLS record; it handles full or partial segments seamlessly.

```mermaid flowchart TD accTitle: Hardware kTLS Transmit vs Retransmit Flow accDescr: Diagram illustrating normal segment transmission versus the full-record DMA penalty required for TCP retransmissions.

subgraph Normal ["Normal Transmission"] direction LR A1["TCP Requests Segments"] --> B1["NIC DMAs Requested Segments"] B1 --> C1["NIC Encrypts on Adapter"] C1 --> D1["Packets Sent to Wire"] end

subgraph Retransmit ["TCP Retransmission Handling"] direction LR A2["Packet Loss / SACK Received"] --> B2["NIC Must Reconstruct State"] B2 --> C2["DMA ENTIRE Record from Byte 0"] C2 --> D2["Encrypt Record up to Target Offset"] D2 --> E2["Encrypt & Transmit Target Segment"] end ```

However, TCP retransmissions break this clean flow entirely. If a client on a flaky connection drops a single segment (e.g., at the 14K offset) and returns a SACK, TCP must retransmit that specific segment. One might expect the NIC to simply DMA that single segment down, encrypt it, and send it. Instead, because the NIC no longer retains the running cipher state for that arbitrary intermediate position in the TCP stream, it must DMA the entire TLS record from byte zero all the way down to the NIC. The card then re-encrypts all the preceding data strictly to regenerate the required crypto state, encrypts the single requested segment, and finally places it on the wire. Gallatin warns that this structural behavior creates an enormous PCIe bandwidth penalty:

> “So that’s a little bit painful, and that means that we have to watch out for TCP re-transmissions because they could easily run us out of PCI Express bandwidth.”

---

[23:17](https://www.youtube.com/watch?v=_o-HcG8QxPc&t=1397s) Mellanox ConnectX-6 Firmware Issues and the "Tar Pit"

Turning from theoretical mechanics to operational "war stories," Gallatin reviews early evaluations of Mellanox ConnectX-6 hardware TLS offload using pre-pandemic beta firmware. While the adapter initially demonstrated an impressive peak throughput of roughly 250 Gb/s aggregate across the system, sustained real-world performance collapsed under production conditions. Netflix edge servers manage traffic distribution via an internal health mechanism powered by a PID controller, which monitors signals like network adapter capacity, CPU saturation, and disk utilization. Because CPU load remained exceptionally low and the network interface reported plenty of headroom, the server continuously reported itself as healthy, requesting more sessions from upstream steering systems.

As connection counts climbed, the NIC's throughput throttled downward until it bottomed out at a meager 75 Gb/s per NIC (150 Gb/s per dual-NIC system). Gallatin illustrates the resulting operational failure mode:

> “As more and more connections arrived, the NIC slowed down more and more, and so we wound up at a floor of about 75 gigs a NIC or 150 gigs per system, and we just kept attracting more and more and more clients and it sort of turned into a tar pit, because this kind of limit is invisible to our PID controller.”

Without an explicit metric signaling this hardware throttling to the control loop, the machine trapped incoming traffic without being able to serve it at rate. Consequently, Netflix deemed this early inline TLS implementation unusable for deployment.

---

[24:48](https://www.youtube.com/watch?v=_o-HcG8QxPc&t=1488s) Investigating Session Scalability and PCIe Relaxed Ordering

Gallatin diagnosed the throughput collapse by analyzing session footprint and adapter memory management. Because the NIC successfully resumes encryption mid-record during standard streaming, it undeniably stores per-connection TLS state. However, Netflix operates at massive scale: every viewing client holds anywhere from one to six concurrent video connections (with buggy client implementations occasionally opening tens or hundreds), alongside dedicated TCP streams for audio and subtitles. Delivering 400 Gb/s demands supporting approximately 400,000 active sessions.

Gallatin suspected that the NIC was running out of on-board memory and thrashing host memory. The Mellanox driver allocates host memory regions on the card's behalf, which the adapter accesses autonomously via DMA. When the number of active sessions exceeds the NIC's local SRAM/DRAM capacity, the card must page session state back and forth across PCIe, decimating throughput. To address this, AMD suggested enabling PCIe Relaxed Ordering, an architectural feature allowing non-dependent PCIe transactions to bypass one another in flight to alleviate bus serialization during state paging. Although Netflix enabled the feature in configuration space, it provided zero performance uplift. Gallatin later discovered why:

> “Later we found out that the ConnectX-6 firmware, even though you could toggle that bit in PCI config space, it didn’t actually—wasn’t actually connected to anything and nothing actually happened because it was hardcoded off.”

---

[26:38](https://www.youtube.com/watch?v=_o-HcG8QxPc&t=1598s) Firmware Breakthroughs: Reaching 380 Gb/s

Eventually, Mellanox delivered updated firmware that genuinely implemented PCIe Relaxed Ordering alongside other internal refinements. The difference was immediate and substantial: sustained bandwidth jumped to 160 Gb/s per NIC, achieving 320 Gb/s total system throughput with parity between peak and sustained traffic. The server no longer degraded into a tar pit under real workloads. For context, while a single adapter using software kTLS could push 190 Gb/s in isolation, host memory bandwidth constraints prevented two adapters from running at that speed simultaneously. Achieving 320 Gb/s aggregate from dual NICs via hardware offload established a brand-new throughput record for Netflix's delivery fleet.

Performance improved even further several months later with a production firmware release introducing an opaque setting:

> “A few months later, they surprised us with a new production firmware which had a knob called `tls_optimized`—I still don’t know what that does except it makes things much better—and now we get 190 gigabits per second a NIC, or 380 gigabits a second total.”

- **Beta Firmware (Pre-Pandemic):** Peaked at ~250 Gb/s; collapsed under load to a sustained 75 Gb/s per NIC (150 Gb/s total) due to unassisted state paging across PCIe. - **Working Relaxed Ordering Firmware:** Delivered 160 Gb/s per NIC (320 Gb/s total) with stable sustained throughput. - **Production Firmware (`tls_optimized`):** Delivered 190 Gb/s per NIC, hitting 380 Gb/s total system throughput.

---

[28:03](https://www.youtube.com/watch?v=_o-HcG8QxPc&t=1683s) Quality of Experience (QoE) and Retransmit Defense Strategies

Despite achieving 380 Gb/s, Netflix could not deploy the technology immediately without rigorous Quality of Experience (QoE) validation. Gallatin emphasizes that raw throughput is meaningless if customer-facing playback degrades. Netflix systematically tracks playback metrics, including: - Rebuffer rate (the frequency of the buffering spinner during streaming). - Time-to-play (the latency between pressing "Play" and video playback beginning). - Time-to-quality (the time required for video resolution to ramp up to decent or excellent fidelity).

Gallatin contrasts the ConnectX-6 results against an earlier hardware TLS NIC evaluation. That earlier adapter lacked the ability to resume crypto state mid-record, requiring FreeBSD developer Randall Stewart to heavily hack the kernel TCP stack to make it TLS-record-boundary aware and force it to segment traffic exclusively along crypto boundaries. That adaptation caused severe QoE degradation, forcing Netflix to abandon the earlier card. In contrast, because the ConnectX-6 cleanly resumes state mid-stream, initial QoE metrics were completely solid, warranting a larger fleet-wide canary deployment.

Finally, before enabling the offload at scale, Netflix had to mitigate the retransmission amplification vulnerability. Because resending dropped segments forces the NIC to DMA an entire TLS record over PCIe just to recreate cipher state, a lossy connection functions essentially as an accidental amplification attack against internal host PCIe bandwidth. To protect system stability, Gallatin outlines their defense strategy: the kernel monitors retransmitted bytes across all active flows, and whenever an individual connection on a lossy path exceeds an acceptable retransmission threshold, the stack dynamically evicts that connection from hardware offload and transitions it back to software kTLS.

[30:24](https://www.youtube.com/watch?v=_o-HcG8QxPc&t=1824s) Mixed Hardware and Software Fallback Penalties

Addressing edge cases and potential exploits, Drew Gallatin explains that the system monitors retransmitted segments in addition to raw byte counts. If a clever attacker requests retransmissions of many tiny segments rather than large byte volumes, the network interface card would still be forced to repeatedly re-DMA all those descriptors and buffers. To prevent this resource drain, Gallatin points out that the kernel enforces strict limits: “and in that case we'll probably kill the connect we kill the connections rather than moving them to software since it's pretty clear it's an attack”.

When evaluating regular traffic with the safety mechanism enabled—moving any TCP connection with more than 1% retransmitted bytes from hardware offload to software encryption—Drew observes an unexpectedly severe performance penalty. This threshold shifts roughly 25% to 33% of active connections into software, causing the maximum stable bandwidth to plunge from 380 Gb/s down to 350 Gb/s. Puzzled by the steep drop, he emphasizes: “just a third of the connections really shouldn't be putting that much stress on the system and remember memory bandwidth looks fine the i the the latency the dma latency looks fine i just don't understand why it's as bad as it is”.

```mermaid flowchart LR accTitle: Connection Handling and Fallback Path accDescr: Flowchart illustrating how TCP connections are evaluated and routed between hardware offload, software fallback, and termination.

subgraph Evaluation ["Connection Inspection"] TCP["Active TCP Connection"] --> Retx{"Retransmission Check"} end

[31:33](https://www.youtube.com/watch?v=_o-HcG8QxPc&t=1893s) Combining Kernel TLS with NUMA Optimizations

Drew describes merging Kernel TLS with NUMA optimizations as a classic “you put your chocolate in my peanut butter kind of moment if you remember those ads from the 80s.” When hardware kTLS is combined with a NUMA topology, the system architecture simplifies dramatically. Previously, data residing on a remote node had to be fetched across interconnects into local memory, processed by the CPU for software encryption, and then shipped over to the NIC. With inline hardware TLS, that entire intermediate crypto detour through local host memory vanishes completely.

In practice, because network cards cannot be installed on every single NUMA node, the worst-case and average-case interconnect crossings remain similar to the non-offloaded setup whenever data must cross over to the NIC on an adjacent node. However, cutting out the software encryption pass saves immense host memory bandwidth:

```mermaid flowchart TD accTitle: NUMA Data Path with Hardware kTLS accDescr: Data flow showing direct DMA from remote memory to NIC bypassing host crypto.

subgraph NodeB ["Remote NUMA Node"] PageCache["File Data in Page Cache"] end

subgraph NodeA ["Local NUMA Node with NIC"] NIC["Mellanox 200GbE NIC\n(Hardware Crypto Engine)"] end

PageCache -->|"Direct PCIe DMA\n(No CPU crypto detour)"| NIC NIC --> Out["Encrypted TLS Packets Out"] ```

Drew presents an “eye chart” summarizing the performance numbers achieved across the AMD configurations: - **Software kTLS (Flat topology):** 240 Gb/s at ~80% CPU utilization (memory bandwidth saturated). - **Software kTLS + NUMA:** 280 Gb/s at ~80% CPU utilization. - **Hardware kTLS (Flat topology, 1 node/socket):** 380 Gb/s at ~60% CPU utilization. - **Hardware kTLS + NUMA:** 380 Gb/s, but CPU utilization drops to just over 50%. - **Hardware kTLS (Flat topology + 1% retransmission fallback safety belt):** 350 Gb/s at high-60% to low-70% CPU utilization.

[34:02](https://www.youtube.com/watch?v=_o-HcG8QxPc&t=2042s) Alternative Platforms: Ampere Altra (arm64)

With about ten minutes remaining, Gallatin turns to alternative platforms evaluated for 400 Gb/s serving, starting with the Ampere Altra—an 80-core ARM Neoverse N1 processor clocked at 3.0 GHz. On paper, the Ampere closely mirrors the AMD EPYC platform, featuring 8 DDR4 memory channels, identical memory capacity, comparable PCIe lane counts, and identical NVMe storage and dual-Mellanox networking. However, switching to a brand-new architecture brought diagnostic challenges on FreeBSD because Drew lacked the established performance counter profiling tools, memory bandwidth monitors, and I/O bus analyzers available on x86, which made him feel like he was “kind of driving blind.”

Initial software kTLS performance on the Ampere Altra was surprisingly sluggish, hovering around 120 to 140 Gb/s. Mark Johnston resolved this bottleneck by implementing a UMA (Universal Memory Allocator) cache zone dedicated to caching 16 KB contiguous TLS crypto destination buffers: “mark johnson uh came up with an idea which is brilliant and which i kick myself for not thinking of but all brilliant ideas are obvious in hindsight and he came up with the idea of using a uma cash zone to cash 16k contiguous tls crypto destination buffers and once we did that we got up to about 180 gigabits a second”.

Moving to hardware kTLS on Ampere immediately hit a PCIe wall at 240 Gb/s with CPU usage sitting at a remarkably low 20%, accompanied by output packet drops on the saturated NICs. Gallatin discovered that PCIe Extended Tags were disabled by default on FreeBSD on the Ampere platform: - PCIe behaves like a network: increasing the tag space is analogous to opening the TCP window, allowing more outstanding transactions in flight. - Standard PCIe default provides a 5-bit tag space (32 outstanding tags). - Enabling PCIe Extended Tags expands the tag space to 8 bits (256 tags). - Once extended tags were turned on, Ampere hardware kTLS bandwidth jumped from 240 Gb/s to 320 Gb/s.

[37:07](https://www.youtube.com/watch?v=_o-HcG8QxPc&t=2227s) Alternative Platforms: Intel Ice Lake Xeon

The final evaluation platform discussed is an Intel Ice Lake Xeon system, which had arrived in the lab just the prior week. Bringing it up was prolonged by storage architecture constraints: the Ice Lake CPU only provides 64 PCIe Gen4 lanes. To achieve full storage throughput without resorting to a massive number of Gen3 drives, Netflix had to source native Gen4 NVMe drives (deploying twenty 4 TB Kioxia NVMe drives) or rely on PCIe switches that exhibited odd hardware bugs.

Drew highlights an aggravating memory frequency limitation on this specific Intel processor model: while populated with DDR4-3200 memory across eight channels, the SKU artificially clamps memory clock speeds to 2933 MHz. Expressing his frustration with artificial market segmentation, he notes: “i personally hate that i wish intel ran all their memory at the same speed like everybody else does across across their product lines but it is what it is”. - **Software kTLS on Intel Ice Lake:** Bottlenecked strictly by memory bandwidth at ~230 Gb/s (scaling directly in line with AMD flat mode adjusted for lower memory frequency). - **Hardware kTLS on Intel Ice Lake:** Could not be benchmarked cleanly. The system shipped with what Gallatin calls the “BIOS from hell,” which locked down crucial PCIe tuning knobs. A firmware fix delivered to enable PCIe relaxed ordering failed to take effect, leaving no verified hardware kTLS figures to present.

| Architecture / Platform | Software kTLS Bandwidth | Hardware kTLS Bandwidth | Primary Bottlenecks / Notes | |:--- |:--- |:--- |:--- | | **AMD EPYC (Rome/Milan)** | ~280 Gb/s | ~380 Gb/s | Memory bus saturated on SW; PCIe limits on HW | | **Ampere Altra (arm64)** | ~180 Gb/s | ~320 Gb/s | Required UMA 16KB cache; required 8-bit PCIe extended tags | | **Intel Ice Lake Xeon** | ~230 Gb/s | Not available | Clamped to DDR4-2933; restricted by locked-down server BIOS |

[39:30](https://www.youtube.com/watch?v=_o-HcG8QxPc&t=2370s) Teasing 800 Gb/s and Community Acknowledgments

Drew shares that Netflix had already built an 800 Gb/s prototype server, intending to unveil its benchmark numbers at the conference. However, severe storms across the United States disrupted freight logistics, preventing the machine from reaching the data center in time to be racked and tested. He quips that he hopes to return the following year to present data on pushing 800 Gb/s out of FreeBSD.

Closing out the session, Drew expresses deep gratitude to his colleagues and FreeBSD open-source collaborators: - **Warren Harrop** at Netflix, who designs and builds prototype content delivery hardware: “warren haroop uh he's one of the people that works on new hardware netflix and is in charge of building a lot of his prototypes and so he's like my santa claus and brings me all my new toys”. - **Mark Johnston**, for foundational work on FreeBSD NUMA optimizations, the virtual memory (VM) subsystem, and Kernel TLS buffer handling. - **John Baldwin**, for the extensive effort of upstreaming kTLS into FreeBSD and collaborating with hardware vendors to design standard kernel/NIC offload interfaces. - **Jeff Roberson**, for his sustained core architectural contributions to FreeBSD's NUMA subsystem.

Reflecting on the collective effort, Drew concludes: “i don't intend to slight anybody but i just love working in the community and every everything that uh everything that we've done is a collaboration”. With a few minutes left on the clock, he opens the floor to audience questions.

[40:56](https://www.youtube.com/watch?v=_o-HcG8QxPc&t=2456s) NIC kTLS Support, NFS over TLS, and PCIe Switch Bandwidth Anomalies

The session transitions into the audience Q&A, with Drew Gallatin navigating to the shared notes tab to address viewer questions directly. The first inquiry addresses whether NIC-based kernel TLS (NIC kTLS) is currently available in the FreeBSD development stream. Drew confirms that NIC kTLS is present in FreeBSD-CURRENT and believes it is also available in FreeBSD 13. However, support remains constrained by hardware driver enablement; within FreeBSD, the only network interface controllers capable of hardware kTLS offload are the Mellanox ConnectX-6 Dx and Chelsio T6 adapters. When asked about performance figures for NFS over TLS, Drew demurs, noting that evaluating that subsystem would require consulting FreeBSD developer Rick Macklem.

The discussion turns to system topology and whether Netflix employs PLX PCIe switches to bridge PCIe Gen 3 storage devices to PCIe Gen 4 root complexes. Drew recalls that they were using Broadcom switches—musing briefly over whether Broadcom acquired PLX—and recounts significant debugging hurdles with switch throughput. In configurations bridging a PCIe Gen 4 x8 upstream link down to four PCIe Gen 3 x4 NVMe SSDs, the hardware exhibited severe throughput anomalies. If only one or two Gen 3 x4 drives were actively transferring, performance matched expectations. However, as soon as more than two drives became active simultaneously, the switch links appeared to degrade to aggregate PCIe Gen 3 bandwidths. These switch quirks proved particularly troublesome when attempting to engineer dense storage appliances constrained to 64 PCIe lanes.

- NIC hardware kTLS offload in FreeBSD is supported exclusively on Mellanox ConnectX-6 Dx and Chelsio T6 adapters. - PCIe switch topologies linking Gen 4 x8 upstreams to four Gen 3 x4 endpoints experienced aggregate bandwidth regressions down to Gen 3 speeds whenever three or more drives were active concurrently. - Storage performance under NFS over TLS is spearheaded within the FreeBSD community by Rick Macklem.

---

[42:30](https://www.youtube.com/watch?v=_o-HcG8QxPc&t=2550s) Full TCP Offload vs. In-House Protocol Tuning, Flow Limits, and Encryption

Drew addresses whether higher-clocked consumer or client-grade DDR memory would help elevate AMD Infinity Fabric throughput. While Infinity Fabric clocking has its own architectural limits, faster memory would directly alleviate memory bandwidth bottlenecks, with Drew remarking that he used to overclock memory on server platforms years ago. Responding to a question about total TCP offload engines (TOE), Drew explains why Netflix deliberately rejects hardware TCP offloading. Netflix maintains an entire transport protocol engineering team—featuring protocol architects like Randall Stewart and Lawrence Stewart (clarifying with a laugh that they are of “no relation”)—focused exclusively on network stack and congestion control innovation. As Drew emphasizes:

> “We one of the things that we depend on is to to control every aspect of the TCP stack and that's what allows us to give such good QoE to our customers.”

Turning to NIC connection scale, Drew discusses hardware session limits. Although he estimates the Mellanox adapter capacity at roughly 128,000 active hardware TLS flows per port, he points out that exceeding this limit does not result in dropped packets or connection failures. If the NIC flow table overflows, the connection simply bypasses hardware crypto and falls back seamlessly to software kTLS on the host CPU. Regarding data encryption on disk, Drew clarifies that stored video assets carry digital rights management (DRM) encryption applied upstream during cloud encoding, but the underlying storage drives utilize no additional layer of drive-level encryption at rest. Finally, regarding network load balancing, Netflix does not steer traffic toward specific NUMA-aligned NIC ports; the servers simply accept the hashing and port selection decisions determined by the upstream switch link partner via LACP.

```mermaid flowchart TD accTitle: TLS Session Handling and Hardware Fallback accDescr: Flowchart illustrating how a TLS connection is routed to hardware offload or falls back to host CPU processing upon flow table exhaustion. subgraph SessionSetup["TLS Session Setup"] A["New Video Stream Connection"] --> B{"NIC Flow Table Space Available? (~128,000 flows/port)"} end subgraph ExecutionPath["Processing Paths"] B -->|"Slot Available"| C["Hardware kTLS Offload (ConnectX-6 Dx / Chelsio T6)"] B -->|"Flow Table Full"| D["Software kTLS Fallback (Host CPU AES-GCM)"] C --> E["mbuf Chain with Full Record Context"] E --> F["Zero-Copy NIC Transmit & Retransmit"] end ```

- Netflix relies entirely on in-house FreeBSD TCP stack customization to optimize streaming Quality of Experience (QoE), ruling out commercial TCP Offload Engines (TOE). - Hardware kTLS flow capacity is estimated at 128,000 sessions per port; overflowing connections gracefully downgrade to software AES-GCM processing. - Content files carry application-level DRM encryption negotiated between cloud encoders and client devices, with no secondary filesystem-level encryption at rest. - Network interface distribution relies strictly on external switch LACP hashing rather than host-directed link steering.

---

[44:37](https://www.youtube.com/watch?v=_o-HcG8QxPc&t=2677s) Performance Monitoring with nstat, FPGA Power Penalties, and FreeBSD's mbuf Advantage

Asked about his methodology for capturing CPU utilization and performance stalls, Drew explains that rather than relying on standard utilities like `vmstat`, he developed a custom tool named `nstat`, which is published in the FreeBSD ports tree. When coupled with Intel PCM (Processor Counter Monitor), `nstat` aggregates system metrics into a single real-time console line, reporting memory bandwidth, core CPU consumption, active TCP connections, interface throughput, context switch rates, and hardware interrupt frequencies.

When queried about Xilinx SmartNICs, Drew notes that Xilinx typically implies FPGA-based logic, whereas Netflix targets ASIC-based adapters due to thermal and power constraints. To illustrate the issue, Drew describes evaluating a prototype TLS offload NIC equipped with an FPGA and 40 to 64 gigabytes of onboard RAM. The massive onboard buffer allowed the adapter to cache transmitted video segments and handle TCP retransmissions entirely on-card without host interaction. While the feature worked well architecturally, its power draw was untenable, with Drew observing:

> “The NIC burned as much power as the rest of the system so it was just not a really good fit for us.”

Drew concludes by contrasting FreeBSD’s kernel architecture with other operating systems. When he initially architected FreeBSD’s kernel TLS subsystem, he explicitly anticipated NIC-based TLS offload. Consequently, he structured FreeBSD's network memory buffers (`mbufs`) so that a single `mbuf` encapsulates an entire TLS record. This unified structural mapping allows network device drivers to perform TCP retransmissions effortlessly, as the driver can locate the complete record context directly within that single `mbuf` without having to reconstruct fragmented crypto state across the operating system boundary. With no further questions in the queue, Drew thanks the audience and concludes the presentation.

- Drew’s `nstat` tool (available in FreeBSD ports) gathers CPU usage, Intel PCM memory bandwidth, network throughput, TCP sessions, interrupts, and context switches into a unified display. - An experimental FPGA-based TLS offload NIC featuring 40–64 GB of onboard packet cache eliminated host-side TCP retransmissions but drew approximately as much power as the entire host server combined. - FreeBSD's kTLS subsystem was architected so that a single `mbuf` encapsulates an entire TLS record, significantly simplifying driver logic during TCP packet retransmissions compared to alternative OS designs.

## Notable quotes > [06:46](https://www.youtube.com/watch?v=_o-HcG8QxPc&t=406s) Basically, the strategy that I came up with a few years ago is to keep as much of our bulk data off the NUMA fabric as we can. — Drew Gallatin

> [20:09](https://www.youtube.com/watch?v=_o-HcG8QxPc&t=1209s) we have to watch out for TCP re-transmissions because they could easily run us out of PCI Express bandwidth. — Drew Gallatin

> [12:05](https://www.youtube.com/watch?v=_o-HcG8QxPc&t=725s) i believe that network-centric siloing is by far the best way to go. — Drew Gallatin

## Mentioned Drew Gallatin, Netflix, FreeBSD, NGINX, AMD, EPYC 7502P, Mellanox, ConnectX-6 Dx, STREAM, Chelsio, Intel, Ampere Altra, Mark Johnston, John Baldwin, Jeff Roberson, Warren Harrop, Randall Stewart, Lawrence Stewart, Rick Macklem, Kioxia, Broadcom, Xilinx

## Video description In this talk, I will discuss the efforts to serve TLS encrypted Netflix video at 400Gb/s from a single server. This will be a follow-on to 2 talks at the 2019 EuroBSDCon: “Numa Optimizations in the FreeBSD Network Stack” and “Kernel TLS and TLS Hardware Offload“. I will provide background on the Netflix video workload, and define key technologies such as NUMA, kernel TLS and hardware kTLS.

I will describe encountering bottlenecks such as: - Memory bandwidth limits for software kTLS - PCIe issues with hardware kTLS - NUMA for software vs hardware kTLS

I will present current and historical performance results from at least: - AMD “Rome” 2nd generation EPYC systems - AMD “Milan” 3rd generation EPYC systems - Ampere Altra arm64 systems

Drew Gallatin Drew started working on FreeBSD at Duke in the 90s, and was one of the people behind the FreeBSD/alpha port. He worked on zero-copy TCP optimizations for FreeBSD and was sending data at over 1Gb/s before gigabit Ethernet was generally available. He spent a decade at Myricom, optimizing their drivers. After a brief hiatus at Google, he landed at Netflix, where he works on optimizing the FreeBSD kernel and network stack for content delivery. He worked on the optimizations to serve unencrypted Netflix traffic at 100Gb/s, and then on more optimizations to send encrypted traffic at ever increasing speeds, from 100Gb/s to 400Gb/s and beyond.

[Source](https://www.youtube.com/watch?v=_o-HcG8QxPc)

中文简报

Netflix借助硬件TLS将单服务器流媒体吞吐量推升至380 Gbps 将TLS加密从主机CPU转移至内联网卡，消除了导致基于软件的流媒体上限停留在240 Gbps的内存带宽瓶颈。 Drew Gallatin详细介绍了Netflix在单路FreeBSD系统上将TLS加密视频流从200 Gbps扩展至400 Gbps目标的架构演进历程。在遭遇由多通道软件加密导致的240 Gbps主机内存带宽瓶颈后，Netflix转而在Mellanox ConnectX-6 Dx适配器上采用内联网卡kTLS，从而消除了CPU加密带来的绕行，并将内存总线流量减半。Gallatin讲述了团队如何通过将硬件卸载与以网络为中心的NUMA隔离相结合，攻克生产环境中关键的固件难题——包括PCIe宽松排序缺陷、会话抖动“焦油坑”以及TCP重传带来的带宽惩罚。这些最终优化将AMD EPYC系统推升至380 Gbps，同时为其他Ampere Altra arm64和Intel Ice Lake配置提供了关键的架构洞见。 Netflix通过搭载AMD EPYC处理器和Mellanox ConnectX-6 Dx网卡的单台FreeBSD服务器，实现了380 Gb/s的TLS加密视频传输，距离400 Gb/s的线速目标仅一步之遥。 (26:48) 软件kTLS在约240 Gb/s处遭遇了硬性内存带宽壁垒，因为每个有效载荷字节都需要四次穿越内存总线（存储到内存、内存到CPU、CPU到内存、内存到网卡），在150 GB/s的物理上限面前需要200 GB/s的带宽。 (3:45) 将加密内联卸载至网卡彻底绕过了主机CPU的内存周转，将总线传输次数降至与传输未加密视频相同的水平，从而使总输出激增至300 Gb/s以上。 (16:47) 生产部署需要动态回退机制，因为TCP重传会迫使网卡从第零字节重新对整个TLS记录进行DMA以重建密码状态，从而将丢包转化为严重的PCIe总线开销惩罚。 (20:09)

Reading summary

Drew Gallatin explains how Netflix adapted FreeBSD and commodity server hardware to deliver 400 Gbps of TLS-encrypted video per node by addressing memory bus saturation, NUMA cross-talk, and hardware offload limits. The architecture relies on inline NIC encryption to eliminate memory round trips while mitigating the PCIe and software penalties inherent to TCP retransmissions. Memory Bus Saturation and NUMA Siloing (3:45) In early benchmarks on an AMD EPYC 7502P (Rome) 32-core system, software Kernel TLS (kTLS) hit a hard ceiling at 240 Gbps, completely saturating the platform's 150 GB/s DDR4 memory bus. Gallatin demonstrates that serving encrypted video through software kTLS forces each byte across the memory bus four times: NVMe storage reads data into page cache (50 GB/s for 400 Gbps network output), the CPU reads it to encrypt, the CPU writes ciphertext into a separate crypto buffer, and the NIC reads it via DMA. This multiplies a 400 Gbps line-rate demand into 200 GB/s of required memory bandwidth. Partitioning the socket into four NUMA domains (NPS4) boosted baseline STREAM bandwidth to roughly 175 GB/s, but uncoordinated memory and core allocations caused bulk video data to cross the interconnect up to four times, consuming around 200 GB/s of fabric bandwidth. Netflix solved this by upstreaming "network-centric siloing" to FreeBSD. By binding media read buffers, crypto workers, TCP pacers, and socket allocations to the NUMA domain tied to the receiving NIC port, interconnect traversals were reduced, raising software kTLS throughput from 240 Gbps to approximately 280 Gbps. Bypassing Host Crypto with Inline NIC kTLS (16:47) Because software kTLS was fundamentally constrained by the memory bus, Netflix adopted inline NIC kTLS using Mellanox ConnectX-6 Dx adapters. By handling AES-GCM encryption on the adapter as data traverses the PCIe bus during packet transmission, the host CPU crypto round trip is eliminated, halving host memory bandwidth consumption and enabling traffic to stream straight from storage to the wire. Early beta firmware severely degraded performance, dropping to a sustained 75 Gbps per NIC (150 Gbps per dual-NIC system) under heavy connection counts. Because the card autonomous paged state across PCIe when running out of onboard memory for hundreds of thousands of sessions, it created an invisible "tar pit" for Netflix's PID traffic controller. Throughput reached 320 Gbps aggregate only after Mellanox delivered firmware with functional PCIe Relaxed Ordering, eventually climbing to 380 Gbps (190 Gbps per NIC) at roughly 60% CPU utilization using production firmware featuring a `tls_optimized` setting. Retransmission Amplification and Hardware Fallback (20:09) While the ConnectX-6 Dx preserves intermediate crypto state during standard pauses, handling packet loss and TCP retransmissions incurs an architectural penalty. Because the card does not retain historical cipher state for arbitrary offsets, resending a single dropped segment forces the NIC to DMA the entire TLS record from byte zero across the PCIe bus and re-encrypt preceding bytes just to recreate state. Flaky client connections effectively become amplification vectors against host PCIe bandwidth. To safeguard the PCIe bus, Netflix designed the FreeBSD kernel to track retransmitted bytes per flow. If an individual connection exceeds a 1% retransmission threshold, the stack dynamically evicts it from the NIC and falls back to host software kTLS, while connections flooding tiny segment retransmissions are terminated as suspected attacks. However, shifting roughly a third of active flows to software dropped aggregate system throughput from 380 Gbps down to 350 Gbps, revealing an unexpectedly steep penalty that Gallatin noted was not fully explained by raw memory or DMA metrics. Cross-Platform Evaluations: Ampere and Intel (34:02) Evaluating alternative 400 Gbps server candidates on FreeBSD exposed platform-specific constraints. On the 80-core ARM-based Ampere Altra, software kTLS was initially restricted to 120–140 Gbps until Mark Johnston introduced a UMA cache zone for 16 KB contiguous TLS crypto buffers, lifting performance to 180 Gbps. Furthermore, hardware kTLS on Ampere stalled at 240 Gbps until FreeBSD enabled 8-bit PCIe Extended Tags (256 outstanding tags versus the 5-bit default of 32), which boosted throughput to 320 Gbps at just 20% CPU utilization. An Intel Ice Lake Xeon platform faced lane and firmware hurdles. Limited to 64 PCIe Gen4 lanes, it required dense Gen4 NVMe arrays to avoid buggy Gen3 PCIe switches that throttled to Gen3 speeds when multiple drives were active. In addition, artificial memory clock down-binning to 2933 MHz capped software kTLS at 230 Gbps, and locked BIOS firmware prevented the activation of PCIe relaxed ordering, precluding viable hardware kTLS evaluation. Combining FreeBSD's network-centric siloing with inline NIC kTLS allowed Netflix to push single-socket systems to 380 Gbps, but pushing toward an 800 Gbps prototype remains bounded by retransmission amplification, PCIe switch routing anomalies, and hardware firmware quirks. Gallatin emphasized that retaining total ownership of FreeBSD's in-house TCP stack—rather than adopting commercial TCP offload engines or power-hungry FPGA designs—remains vital to balancing hardware efficiency with subscriber Quality of Experience.

精炼阅读摘要

Netflix工程师Drew Gallatin在EuroBSDCon演讲中阐述了如何通过FreeBSD内核与硬件协同优化，克服内存带宽瓶颈，实现在单台服务器上提供近400 Gbps的TLS加密视频流分发能力。软件kTLS的内存带宽墙与NUMA瓶颈 (3:45) 在单路32核AMD EPYC 7502P平台上，Netflix最初尝试使用软件内核TLS（kTLS）提供视频服务，但吞吐量在达到约240 Gbps时遭遇物理内存带宽瓶颈。Gallatin解释称，流媒体服务通过sendfile()管道分发数据时，数据无法在共享页缓存中就地加密，必须经历四次内存总线穿越：NVMe读入内存缓存、CPU读取明文执行加密、密文写回内存中的加密缓存，最后由网卡经DMA读出发送。服务400 Gbps（50 GB/s）网络流量需要200 GB/s的内存聚合带宽，而该平台的8通道DDR4内存额定带宽仅为150 GB/s。针对该瓶颈，团队进一步探索了NUMA拓扑调优。在未优化的跨节点路由下，存储DMA、CPU加密和网卡发送若分布在不同节点，数据将四次横跨片内互连总线，产生高达200 GB/s的互连负载并引起总线拥塞。通过在FreeBSD中实施以网络为中心的隔离优化（network-centric siloing），将套接字内存、磁盘读取目标缓存及工作线程严格绑定到接收网卡所在的NUMA节点，互连总线负载降至平均62.5 GB/s。然而，由于4个NUMA节点中仅有2个配备网卡，软件kTLS结合NUMA优化后吞吐量仅提升至280 Gbps，仍远低于400 Gbps的目标。网卡硬件内联kTLS消除内存绕行 (16:47) 为突破内存带宽上限，Netflix转向网卡硬件内联TLS（NIC kTLS）。该架构在用户空间完成握手协商后将密钥注入内核，随即将加密任务下放给网卡硬件（Mellanox ConnectX-6 Dx），在数据经PCIe总线向外发送时由网卡实时执行AES-GCM加密。该方案消除了CPU加密的内存读写往返，使内存路径重回非加密视频的传输效率，系统所需内存带宽直接减半。早期评估中，ConnectX-6 Dx因测试固件未实际启用PCIe Relaxed Ordering，在连接数攀升时由于卡载SRAM/DRAM不足频繁与主机内存换页，导致吞吐量跌入单卡75 Gb/s的“焦油坑”。随后Mellanox交付了正确实现Relaxed Ordering并引入tls_optimized特性的生产固件，单卡带宽跃升至190 Gb/s，双卡整机聚合吞吐量达到380 Gb/s，在CPU占用仅约50%至60%的情况下创下分发新纪录。 TCP重传放大风险与降级回退惩罚 (20:09) 硬件kTLS在正常传输时可无缝暂停与恢复记录内状态，但面对TCP丢包重传时暴露出结构性缺陷。当客户端发生丢包触发SACK重传时，网卡因不再保留历史任意偏移处的加密状态，被迫通过PCIe将整条TLS记录从字节0开始全量DMA至网卡并重新加密，仅为了获取上下文以发出单个丢失分段。这种行为极易耗尽PCIe带宽，甚至可能演变成针对主机总线带宽的重传放大攻击。为防范该风险，内核对活动连接实施监控：对于高频请求微小分段的恶意攻击流量直接终止连接；对于普通丢包链路上重传字节比例超过1%的连接，系统动态将其从硬件卸载踢回至主机软件kTLS。实验显示，这一保护机制将约25%至33%的连接转入软件处理，导致整机稳定吞吐量从380 Gb/s骤降至350 Gb/s。尽管内存带宽与DMA延迟指标表现正常，该性能跌幅的深层原因仍有待进一步排查。异构硬件评估与底层协议栈自研取舍 (34:02) 除AMD平台外，Netflix还评估了Ampere Altra（80核ARM）与Intel Ice Lake平台。Ampere Altra最初受限于内存分配器，在引入16 KB连续TLS缓存区优化并开启PCIe Extended Tags（将标签空间由5位扩充至8位）后，硬件kTLS吞吐量由240 Gb/s提升至320 Gb/s。Intel Ice Lake则因内存频率被人工限制在2933 MHz，且测试样机固件锁死了PCIe关键调优选项，未能成功完成硬件kTLS基准测试。面对网络栈设计，Gallatin明确指出Netflix坚决拒绝商用TCP卸载引擎（TOE），坚持完全掌控自研FreeBSD TCP协议栈以微调拥塞控制与客户体验质量（QoE）。此外，团队评估过配备40至64 GB板载内存、可独立处理重传的FPGA网卡，但因其功耗几乎与整机相当而放弃。最终，FreeBSD在架构上采用单个mbuf封装完整TLS记录的设计，大幅降低了驱动层在TCP重传处理时的复杂度。 Drew Gallatin展示了在FreeBSD上结合硬件内联kTLS与NUMA感知优化实现380 Gb/s吞吐的技术路径，但重传引发的降级惩罚以及异构平台的PCIe/固件调优依然构成工程挑战；Netflix原计划展示的800 Gbps原型机测试因物流受阻延期，成为未来进一步验证的方向。

Detailed notes — zh

## Background 该视频由 EuroBSDCon 于 2021 年 9 月 27 日在 YouTube 平台发布，主讲人为 Netflix 资深性能工程师 Drew Gallatin。根据页面简介，演讲详细复盘了 Netflix 如何在 FreeBSD 系统上攻坚突破软硬件性能瓶颈，最终在单台服务器上实现近 400Gbps 的 TLS 加密视频分发，并涵盖了针对 AMD Rome/Milan、Ampere Altra 及 Intel 平台的评测调优经验。 ## TL;DR 演讲拆解了 Netflix 将单机视频分发吞吐从 200Gbps 翻倍至接近 400Gbps 的工程演进历程。纯软件 kTLS 因 CPU 往返读写加密缓冲区消耗高达 200GB/s 内存带宽，在 240Gbps 处遭遇物理瓶颈，即便配合网络为中心的 NUMA 孤岛化调优也仅能提升至 280Gbps。团队转向 Mellanox ConnectX-6 Dx 硬件内联 TLS 方案，通过固件调优与启用 PCIe 放宽排序（Relaxed Ordering）突破换页瓶颈，最终跑满 380Gbps。针对硬件 kTLS 重传导致 PCIe 带宽放大的隐患，Netflix 设计了动态回退与断开连接的双重防御机制；此外，演讲还横向对比了 Ampere Altra 与 Intel Ice Lake 架构在系统适配上的诸多工程挑战。 ## Atmosphere & style 整场演讲流露出浓厚且硬核的系统级工程师极客气质，主讲人 Drew Gallatin 风格沉稳、老练且极具自嘲幽默感。他以极接地气的打比方方式——如将 NUMA 互联比作公路交通中的重卡与通勤小车、将软硬件整合比作花生酱配巧克力——将深奥的内核与硬件交互讲得引人入胜。观众能清晰感受到团队在面对未公开的固件黑盒陷阱、内存带宽天花板以及硬件缺陷时，抽丝剥茧、精准定位根因的扎实技术底蕴。 ## Bulletin - [04:29](https://www.youtube.com/watch?v=_o-HcG8QxPc&t=269s) Netflix 致力于在单台基于 FreeBSD 的服务器上实现 400Gbps 的 TLS 加密视频输出，但传统软件 kTLS 会因内存读写往返使 50GB/s 的有效负载膨胀至 200GB/s 内存带宽需求，从而撞上物理内存墙。 - [16:47](https://www.youtube.com/watch?v=_o-HcG8QxPc&t=1007s) 通过引入 Mellanox ConnectX-6 Dx 的硬件 Inline TLS，加密计算在 PCIe 发送过程中由网卡硬件实时完成，消除内存往返，整机吞吐在更新固件并开启 Relaxed Ordering 后突破至 380Gbps。 - [20:09](https://www.youtube.com/watch?v=_o-HcG8QxPc&t=1209s) 硬件 kTLS 存在 TCP 重传放大的物理短板：遇到丢包重传时网卡必须重新 DMA 整个 TLS 记录以恢复加密上下文，极易耗尽 PCIe 带宽，Netflix 借此引入了重传率超 1% 动态降级回退至软件 kTLS 的安全防护机制。 - [34:02](https://www.youtube.com/watch?v=_o-HcG8QxPc&t=2042s) 在架构横向对比中，AMD EPYC 达成最优的 380Gbps；Ampere Altra ARM64 经 UMA 连续缓冲区与 8 位 PCIe 扩展标签优化后达到 320Gbps；Intel Ice Lake 则受限于内存降频与 BIOS/PCIe 问题未能测完硬件卸载。 ## Key takeaways - [00:00](https://www.youtube.com/watch?v=_o-HcG8QxPc&t=0s) 单台服务器跑满 400Gbps 的业务初衷是用尽可能少的硬件资源与能耗为更多用户提供服务，硬件基础为配备 4 个 100GbE 端口的单路 AMD EPYC 平台。 - [01:02](https://www.youtube.com/watch?v=_o-HcG8QxPc&t=62s) 软件 kTLS 结合 sendfile 避免了用户态与内核态的数据拷贝，为系统节省了约 60% 的 CPU 开销，但系统在达到约 240Gbps 时即因系统内存带宽饱和而受阻。 - [04:29](https://www.youtube.com/watch?v=_o-HcG8QxPc&t=269s) 由于视频缓存由多连接共享而无法原地加密，400Gbps 吞吐需要经历磁盘写入、明文读出、密文写回及网卡 DMA 四次内存传输，产生 200GB/s（1.6Tbps）的内存带宽开销，远超单路 DDR4 极限。 - [10:20](https://www.youtube.com/watch?v=_o-HcG8QxPc&t=620s) 在 NUMA 架构下缺乏局部亲和性会导致数据跨总线跳跃 4 次并堆积 200GB/s 内部流量，严重拥塞单条仅 47GB/s 的互连链路，甚至阻塞微指令执行。 - [11:51](https://www.youtube.com/watch?v=_o-HcG8QxPc&t=711s) 推行‘以网络为中心的孤岛化’（Network-Centric Siloing）使计算、内存分配与网卡对齐，将最坏情况下的跨总线传输从 4 次减少到 1 次，但因部分节点无物理网卡受限，纯软件吞吐仅提升至 280Gbps。 - [16:47](https://www.youtube.com/watch?v=_o-HcG8QxPc&t=1007s) 硬件 Inline TLS 允许数据从磁盘 DMA 读入内存后直接 DMA 推给网卡硬件进行 AES-GCM 加密，免去了 CPU 内存读写绕道，内存带宽需求瞬间减半。 - [20:09](https://www.youtube.com/watch?v=_o-HcG8QxPc&t=1209s) Mellanox 网卡支持在 TLS 记录内部保持断点加密状态，但一旦发生 TCP 丢包重传，网卡因无历史上下文必须重新 DMA 整个 TLS 记录并重算加密，导致 PCIe 带宽成倍放大。 - [23:18](https://www.youtube.com/watch?v=_o-HcG8QxPc&t=1398s) 早期固件在面对数十万高并发 TLS 连接时，片上 SRAM 耗尽引发网卡与主机内存频繁分页换页，导致系统陷入单卡仅 75Gbps 却不断吸引新连接的‘流沙陷阱’。 - [26:08](https://www.youtube.com/watch?v=_o-HcG8QxPc&t=1568s) 开启 PCIe Relaxed Ordering 并在固件中启用 `tls_optimized` 开关后，网卡状态换页瓶颈解除，系统双卡吞吐一举跃升至 380Gbps。 - [30:24](https://www.youtube.com/watch?v=_o-HcG8QxPc&t=1824s) 为防御恶劣链路或 DoS 攻击耗尽 PCIe 带宽，Netflix 将重传超 1% 的连接回退至软件 kTLS，对异常小包重传连接直接断开，但混合模式实测吞吐因未知压力下滑至 350Gbps。 - [31:33](https://www.youtube.com/watch?v=_o-HcG8QxPc&t=1893s) 硬件 kTLS 与 NUMA 结合后彻底免去了加密内存绕行，系统在保持 380Gbps 峰值带宽的同时，CPU 占用率从 80% 骤降至 50% 刚过。 - [34:02](https://www.youtube.com/watch?v=_o-HcG8QxPc&t=2042s) 在 Ampere Altra 80 核 ARM 平台上，团队通过 UMA 缓存 16KB 连续加密缓冲区将软件 kTLS 提至 180Gbps，并通过开启 8 位 PCIe 扩展标签（256 tags）将硬件卸载吞吐从 240Gbps 解锁至 320Gbps。 - [37:07](https://www.youtube.com/watch?v=_o-HcG8QxPc&t=2227s) Intel Ice Lake 平台因内存降频至 2933MHz 使得软件吞吐被锁在 230Gbps，且受制于 PCIe 通道数少、交换芯片 Bug 及 BIOS 限制，未能完成硬件 kTLS 评估。 - [42:55](https://www.youtube.com/watch?v=_o-HcG8QxPc&t=2575s) Netflix 坚持自主掌控 TCP 协议栈以确保用户 QoE，坚决不采用完整 TCP 卸载引擎（TOE）；当硬件 TLS 会话表溢出时，连接将无缝回退至 CPU 软件处理。 - [44:37](https://www.youtube.com/watch?v=_o-HcG8QxPc&t=2677s) 针对重传缓存问题，FPGA 加大容量板载内存的网卡方案虽然理论可行，但其单卡功耗几乎赶上整台服务器，完全不符合大规模分发的能效比要求。 - [45:59](https://www.youtube.com/watch?v=_o-HcG8QxPc&t=2759s) FreeBSD 在设计 kTLS 底层时让单个 mbuf 完整映射一个 TLS 记录，这种前瞻性内存结构极大简化了硬件网卡驱动在处理丢包重传时的元数据检索流程。 ## Detailed walkthrough ### [00:00](https://www.youtube.com/watch?v=_o-HcG8QxPc&t=0s) 演讲背景与 Netflix 单机 400Gbps 目标 Drew Gallatin 介绍了自己在 Netflix 的职责：专注于 CPU 效率优化，目标是用尽可能少的硬件资源为尽可能多的用户提供视频服务，这既能节约成本，也利于环保。近期他的工作重心一直是在单台服务器上突破性能极限。本次演讲的主题正是如何在 FreeBSD 上将单台服务器的吞吐量推升至接近 400Gbps。 Drew Gallatin 概括了本次演讲的核心脉络： - 解释推动这项工作的动力与业务背景； - 介绍 Netflix 的生产硬件平台与视频分发工作负载特征； - 探讨在实现这些目标的过程中是否必须依赖 NUMA 架构优化； - 介绍硬件级内联 TLS 加密（FreeBSD 社区称之为 NIC TLS）； - 若时间允许，还将探讨除主要生产平台之外的其他备选架构平台（如 AMD Milan 及 Ampere Altra arm64）。 ### [01:02](https://www.youtube.com/watch?v=_o-HcG8QxPc&t=62s) 生产平台规格与软件 kTLS 的性能瓶颈从 2020 年开始，Netflix 便已大规模部署能够分发 200Gbps TLS 加密视频流的服务器。这些机器本身配备了 4 个 100GbE 网络端口，团队自然而然地产生了新的目标：既然硬件上拥有 4 个物理网口，为何不把它们全部跑满，实现单机 400Gbps，而不是止步于 200Gbps？在软件与工作负载层面，Netflix 运行的是与 HEAD 分支仅相差数周的 FreeBSD-CURRENT 内核，Web 服务采用 Nginx，视频传输几乎完全依赖 `sendfile` 系统调用。“And that means that we love kernel TLS because that means the data is encrypted in the kernel as part of the sendfile pipeline, which means we don't have any extra boundary crossings where data is like you know pulled off a disk and encrypted user space and then written back to the kernel,” Drew Gallatin 强调道。仅依靠软件内核 TLS（kTLS），免去数据在用户态与内核态之间的来回拷贝与边界穿越，就为 Netflix 节省了大约 60% 的 CPU 开销。当前采用的生产服务器硬件配置极为强悍： - **CPU**：单颗 AMD EPYC 7502P（代号 Rome），拥有 32 核 64 线程，基频 2.5GHz； - **内存**：8 通道 DDR4-3200，提供约 150GB/s（换算为网络带宽单位约为 1.2Tbps）的理论内存带宽； - **I/O 总线**：128 条 PCIe Gen4 通道，折合网络吞吐量达 2Tbps 级别； - **网卡**：2 张 Mellanox ConnectX-6 Dx，以 PCIe Gen4 x16 接入，每张网卡提供 2 个满速端口，整机合计 4 个 100Gbps 端口； - **存储**：大容量 NVMe 固态硬盘阵列。由于该机器硬件定型较早，彼时市场上尚无成熟的满速 Gen4 SSD，因此系统内挂载了大量 PCIe Gen3 NVMe 驱动器。然而，在纯软件 kTLS 配置下，该平台的实测吞吐量在达到约 240Gbps 时便遭遇严重瓶颈。Drew Gallatin 指出，通过 AMD 提供的性能计数器读取工具（amd-upcm）排查确认，此时系统内存带宽已经彻底耗尽饱和，导致传输性能无法继续向上扩展。 ### [04:29](https://www.youtube.com/watch?v=_o-HcG8QxPc&t=269s) 内存带宽开销剖析：为什么 400Gbps 需要 1.6Tbps 内存吞吐为了说明内存带宽究竟是如何被消耗殆尽的，Drew Gallatin 详细拆解了客户端发起一次点播时内核中的数据流动全过程。当客户端观看电影时，其自适应缓冲算法会预拉取后续数据（例如一次请求接下来的 2MB 视频）。Nginx 收到该 HTTP 请求后，直接通过 `sendfile` 发起传输管道。 ```mermaid flowchart LR accTitle: sendfile 数据流与内存带宽消耗 accDescr: 展示数据从 NVMe 读取到内存、CPU 读出加密后写回新缓冲区，再由网卡 DMA 读出发送的过程。 subgraph Storage["存储层"] Disk["NVMe 固态硬盘阵列"] end subgraph Memory["系统主存 (DRAM)"] PageCache["Page Cache (明文数据)"] TLSBuf["kTLS 目标缓冲区 (密文数据)"] end subgraph Processing["计算与网络"] CPU["CPU (AES 加密计算)"] NIC["Mellanox 网卡 (DMA 读)"] end Disk -->|"1. 写入 (50 GB/s)"| PageCache PageCache -->|"2. 读入 (50 GB/s)"| CPU CPU -->|"3. 写回密文 (50 GB/s)"| TLSBuf TLSBuf -->|"4. DMA 发送 (50 GB/s)"| NIC ``` 在 400Gbps（换算为存储/内存单位即 50GB/s）的网络目标下，数据在系统内存中的往返流向如下： 1. **磁盘读取**：NVMe 驱动器以 50GB/s 的聚合吞吐将明文数据 DMA 写入系统内存的 Page Cache。 2. **CPU 读出明文**：CPU 从 Page Cache 中以 50GB/s 读取明文数据并执行加密运算。 3. **CPU 写入密文**：常规加密算法支持就地（in-place）加密，但在视频分发场景下行不通。Page Cache 中的数据由多个连接共享，如果对原始缓存原地加密，“If I was streaming something and you were streaming something and we encrypted it to my TLS keys, then you would get something that was encrypted doubly and it would just be garbage and it would be a bad day,” Drew Gallatin 解释道。因此内核必须分配全新的独立目标缓冲区，CPU 再以 50GB/s 的速度把密文写回内存。 4. **网卡 DMA 读出**：Mellanox ConnectX-6 Dx 网卡以 50GB/s 从密文缓冲区执行 DMA 读操作，最终推向物理网络。这四次单向传输将 50GB/s 的基础负载直接翻了 4 倍，整机总计需要高达 200GB/s（即 1.6Tbps）的内存带宽。由于这台 AMD Rome 单路服务器的理论极限仅有约 150GB/s，纯软件 kTLS 显然无法支撑 400Gbps 的吞吐需求。 ### [06:46](https://www.youtube.com/watch?v=_o-HcG8QxPc&t=406s) NUMA 架构与互联总线上的“堵车”隐患面对物理内存带宽的硬性限制，Drew Gallatin 开始探索 NUMA（非一致性内存访问）调优是否能帮助系统逼近目标。在基准测试（如 STREAM）中，将 AMD EPYC 处理器配置为 NPS4（每个 Socket 划分 4 个 NUMA 节点）模式能够更充分地调度其内存控制器，实测内存带宽可从平面模式（NPS1）的约 150GB/s 提升到约 175GB/s。 Drew Gallatin 回顾了 NUMA 的技术演进：在早期的多处理器体系中，CPU、内存和外设挂在同一总线上，彼此访问距离完全对称，软件无需感知拓扑，但其可扩展性极差；随后硬件架构转向“分而治之”策略，通过 NUMA 总线将多个独立互联的计算域连接在一起。EPYC 处理器在 NPS4 模式下被切分为 4 个高度互连的 NUMA 域，但跨域访问也引入了性能代价： - **跨域延迟开销**：跨越不同 NUMA 域的访问延迟在 12ns 到 28ns 之间； - **互联带宽上限**：系统内部互联链路单条带宽约为 47GB/s，整机总互联带宽约为 280GB/s。 Drew Gallatin 提出了应对 NUMA 系统的关键准则：“Keep as much of our bulk data off the NUMA fabric as we can.” 他用公路交通打了个形象的比方：如果把整机互联链路想象成高速公路，绝不能让运载海量视频流量的“重型卡车”占满所有车道，否则执行紧急任务的“通勤小汽车”——如跨域更新统计计数器、读取轻量级进程状态等关键微指令——就会被死死堵在路上，最终导致 CPU 核心被迫陷入漫长的跨域等待中。因此，数据流必须尽可能封闭在本地 NUMA 域内完成流转。 ### [10:20](https://www.youtube.com/watch?v=_o-HcG8QxPc&t=620s) NUMA 最坏场景与跨总线拥塞在未进行任何亲和性优化的极端情况下，系统的行为会陷入混乱。Drew Gallatin 演示了一个请求流转的最坏路径：假设一个客户端请求到达了左下角的 NUMA 节点，系统需要从磁盘读取流媒体内容，但因为内存分配策略缺乏感知，数据被错误地分配到了右上角的 NUMA 节点内存中，导致数据首次跨越 NUMA 总线。随后，CPU 需要读取这部分明文数据进行加密，本应就近调度同节点的 CPU，却阴差阳错调用了右下角节点的 CPU，造成数据第二次横跨 NUMA 总线。加密完成后，CPU 需要分配目标加密缓冲区（crypto buffer），系统再次“犯傻”，将缓冲区开辟在左上角的节点内存中，从而触发第三次跨总线传输。最后，发送数据所需的网卡位于最初的左下角节点，密文必须被读取并经由总线发送至该网卡，完成了第四次跨总线跳跃。 ```mermaid flowchart TD accTitle: NUMA 架构下的四次跨总线最坏路径 accDescr: 展示未优化状态下一个请求如何反复横跳导致 NUMA 织构严重拥塞 subgraph TopLeft["左上节点 (Node 0)"] CB["密文缓冲区 (Crypto Buffer)"] end subgraph TopRight["右上节点 (Node 1)"] PB["明文内容缓存 (Plaintext Buffer)"] end subgraph BottomLeft["左下节点 (Node 2)"] Req["请求接入 / 发送网卡 (NIC)"] end subgraph BottomRight["右下节点 (Node 3)"] CPU["加密运算 (CPU Crypto)"] end Req -->|"① 跨总线分配读缓存"| PB PB -->|"② 跨总线拉取明文加密"| CPU CPU -->|"③ 跨总线写入密文缓冲"| CB CB -->|"④ 跨总线送至网卡发送"| Req ``` 这种反复横跨节点总线的模式对互连织构（Interconnect Fabric）造成了致命打击。对于 50 GB/s（400 Gbps）的网络出口吞吐量，如果每次数据转发都要横跨四次，总线上堆叠的内部流量将达到惊人的 200 GB/s。正如 Drew Gallatin 严肃指出的：“那已经非常接近总织构带宽的极限，这会导致互联织构发生严重拥塞，后果非常糟糕。”为了打破这一瓶颈，系统必须采取更为精细的调度，理想状态是完全将请求、内存、计算与网卡限制在单一局部节点内，达成零次跨总线交互。 --- ### [11:51](https://www.youtube.com/watch?v=_o-HcG8QxPc&t=711s) 网络为中心的孤岛化（Network-Centric Siloing）在现实部署中，Netflix 的单台服务器仅分配一个全局 IP 地址，并通过 LACP（链路聚合控制协议）对网卡端口进行绑定。由于链路对端（LACP partner）通常基于 TCP 端口号与 IP 地址进行哈希分流，只要链路聚合组（LAG）未发生震荡或成员增减，特定连接在生命周期内总是固定落入同一物理网卡。基于这一事实，Drew Gallatin 早期探索了两种隔离思路：一种是以内容存储位置为锚点的“磁盘为中心孤岛化”（Disk-Centric Siloing），另一种是以连接接入位置为锚点的“网络为中心孤岛化”（Network-Centric Siloing）。在对二者进行深入评估后，Drew 决定全面推行网络为中心的策略。网络为中心孤岛化的核心原则是：让数据处理的一切环节向接入该连接的网卡所在的 NUMA 节点靠拢。当连接绑定到特定 NUMA 节点后，从磁盘进行 DMA 传输时，直接在本地节点分配媒体文件内存；TLS 加密缓冲区同样在本地分配；负责 TLS 加密的内核工作线程、TCP 步调器（pacers）以及 TCP 定时器均绑定至该节点的 CPU 运行；最终调用本地网卡将加密数据发回客户端。Drew 强调，这套深度的 NUMA 亲和性架构改动早已全部合并回 FreeBSD 上游分支（FreeBSD 13 及 Current）。经过网络中心化优化后，即便遭遇内容存放在对侧远程节点的最坏情况，流程也得到了极大简化： - 客户端请求进入左下节点，系统将磁盘读入的数据直接存入左下节点的本地内存； - 由左下节点的本地 CPU 执行加密计算，并将结果写入左下节点的本地加密缓冲区； - 最终密文直接送往左下节点的本地网卡发出； - 整条链路仅在最初将内容从远端磁盘拉取到本地节点内存时跨越了一次 NUMA 总线，跨总线流量从 200 GB/s 骤降至 50 GB/s。 --- ### [14:48](https://www.youtube.com/watch?v=_o-HcG8QxPc&t=888s) 现实硬件限制与平均吞吐收益然而，实际硬件拓扑往往充满妥协。服务器配备的是双口 100G 网卡（2x 100G），若要跑满 400 Gbps，全机仅需 4 个 100G 端口（即两张物理网卡）。面对拥有 4 个 NUMA 节点的单插槽或双插槽系统，两张物理网卡只能插在其中两个 NUMA 节点上，剩下两个节点根本没有直连网卡。由于 400G 网卡造价昂贵，不可能为了让每个节点都拥有网卡而过度冗余配置。为了让无网卡节点也能分担负载，内核必须引入变通方案，将某一个物理网卡端口在逻辑上“借给”没有直连网卡的 NUMA 节点使用。这一硬件缺憾直接拖累了优化效果。在最坏情况下，某个没有物理网卡的节点处理完数据后，依然必须跨越 NUMA 总线借道其他节点的物理网卡发出，使得跨总线次数从 1 次增加到 2 次，总线流量负荷回升至 100 GB/s。若从统计学概率评估平均情况： - 节点命中本地存储内容的概率为 25%，因而有 75%（3/4）的概率需要跨总线读取远端内容； - 4 个节点中仅有 2 个拥有直连网卡，意味着有 50% 的概率需要跨总线借助其他节点的网卡输出； - 综合折算后，NUMA 织构上的平均数据吞吐约为 62.5 GB/s。尽管 62.5 GB/s 相较最初失控状态下的 200 GB/s 已经有了巨大改观，但性能测试结果却略显骨感。这套软件调度层面的孤岛化优化仅将单机 TLS 发送性能从约 240 Gbps 提升到了 280 Gbps 左右。Drew Gallatin 直言： > “它确实有所帮助，虽然并非灵丹妙药，但它让我们从大约 240G 提升到了 280G，这总比什么都没有要强，但仍然远未达到我们的目标。” 这一成绩充其量只能勉强点亮第三个 100G 链路，离跑满 4 个 100G 链路实现 400 Gbps 的愿景依然相去甚远。 --- ### [16:47](https://www.youtube.com/watch?v=_o-HcG8QxPc&t=1007s) 引入网卡硬件 kTLS（NIC kTLS / Inline TLS）面对纯软件优化的天花板，Drew Gallatin 将目光投向了网卡硬件 kTLS（业界通常称为 Hardware Inline TLS）。在传统的软件 kTLS 架构中，系统存在一个根本性的开销黑洞——CPU 或旁路加速芯片（如 Intel QAT 或 Chelsio CCR）必须将明文从内存中读入缓存进行加密，随后再将密文写回内存。Drew 对此直言不讳：“从演讲一开始那张图起，这两条垂直的绿色线条就一直让我深恶痛绝。”正是明文读出与密文写回的往返“绕路”，平白占用了双倍的内存带宽。如果能彻底消除这个绕道环节，内存带宽需求将瞬间减半，整个数据流将退化成近乎未加密裸流的状态：数据直接从磁盘 DMA 读入内存，再通过 DMA 从内存直接推给网卡，CPU 从繁重的数据搬运中完全抽身。 ```mermaid flowchart LR accTitle: 软件 kTLS 与网卡硬件 Inline TLS 路径对比 accDescr: 对比传统 CPU 加密绕路与网卡硬件直接加密的数据流 subgraph SW["软件 kTLS 路径 (双倍内存读写)"] D1["磁盘 (Disk)"] -->|"DMA 写入"| M1["内存 (明文)"] M1 -->|"读取"| CPU1["CPU / 硬件加速卡"] CPU1 -->|"写入"| M2["内存 (密文)"] M2 -->|"DMA 发送"| N1["网卡 (NIC)"] end subgraph HW["网卡硬件 Inline kTLS (消除绕路)"] D2["磁盘 (Disk)"] -->|"DMA 写入"| M3["内存 (明文)"] M3 -->|"DMA 直推"| N2["网卡 ConnectX-6 Dx (硬件加密)"] N2 -->|"物理线缆"| Wire["网络 (Network)"] end ``` 在网卡 kTLS 模式下，TLS 握手依然在用户空间完成，当加密上下文下沉到内核后，内核就像“传递烫手山芋”（hot potato）一样，迅速将加密任务转交网卡硬件。网卡芯片在数据经由 PCIe 发送上线的物理过程中，硬件级实时对 TLS 记录完成 AES-GCM 加密。这种做法带来了一个常被安全人员议论的技术权衡：数据在操作系统内核中甚至在穿越 PCIe 总线时都是明文状态。Drew 坦言：“我明白有些人可能对这一点心存顾虑，但对于我们的流媒体负载来说这完全不成问题，不过还是有必要在这里明确指出来。” Netflix 在该方案中选用的网卡是 Mellanox ConnectX-6 Dx，它原生支持 TLS 1.2 和 1.3 的 AES-GCM 硬件卸载。此类硬件的一大精妙特性在于**能够在单个 TLS 记录内部保持加密状态**： - 当 TCP 发送完某个 TLS 记录的前几个数据包段后，如果因拥塞控制或等待客户端 ACK 确认而暂停传输（即便停顿数毫秒甚至数秒），网卡依然牢牢记住当前的加密状态上下文； - 待新的 ACK 到达后，TCP 从该 TLS 记录的中间位置继续发送剩余分段，网卡能够无缝续接加密状态，业务毫无感知； - 唯一的性能惩罚场景在于**乱序发送（如 TCP 重传）**：一旦发生重传，网卡无法保留 TCP 流中任意历史位置的瞬时状态，必须通过额外机制重新恢复上下文才能完成加密发送。随后，Drew 准备向观众详细拆解 ConnectX-6 Dx 处理 TLS 发送的具体硬件流程。 ### [20:09](https://www.youtube.com/watch?v=_o-HcG8QxPc&t=1209s) 硬件 kTLS 的发送机制与重传困境在硬件内联 TLS（inline kTLS）架构中，主机内存的套接字缓冲区（socket buffer）中存放着未加密的明文 TLS 记录。TCP 依据分段边界（如 0、1448、2896 字节等）发起发送请求。正常流程下，TCP 请求发送数据包的前几个分段，网卡直接通过 DMA 将明文拉取到网卡内部进行硬件加密，随后打包发送至 100Gb 网络。网卡内部的具体 DMA 策略对系统是黑盒透明的——可能按 4KB 对齐读取，也可能按单个 TCP 分段拉取。只要后续 ACK 推进了拥塞窗口，TCP 继续通知发送剩余分段，网卡便顺畅地断点续传，不产生多余开销。 ```mermaid flowchart TD accTitle: 硬件 kTLS 正常发送与重传机制对比 accDescr: 展示硬件 kTLS 在顺序发送与发生 TCP 重传时 DMA 行为的巨大差异 subgraph Normal["正常发送流程（无额外开销）"] A1["套接字缓冲区: 明文 TLS 记录"] -->|"DMA 拉取分段 (如 1448B)"| B1["网卡硬件加密引擎"] B1 -->|"发出加密数据包"| C1["网络物理链路"] end subgraph Retransmit["重传流程（引发 DMA 放大）"] A2["套接字缓冲区: 明文 TLS 记录"] -->|"DMA 拉取从头到丢失处的全部记录"| B2["网卡重算加密状态并丢弃前序"] B2 -->|"仅加密目标丢失分段"| C2["网卡发出重传包"] end ``` 然而，一旦遇上网络丢包引发 TCP 重传（例如移动端蜂窝网络信号不稳定收到 SACK，要求重发 14KB 位置的某个分段），硬件卸载的致命短板便暴露无遗。网卡本地无法长久保留该 TCP 流在任意历史字节位置的加密上下文状态。Drew Gallatin 强调了这种重传的严峻后果： > “what has to happen is because the nic no longer has the state for this arbitrary position in the tcp stream what has to happen is the nick has to dma the entire tls record from the ver all the way down uh to the neck encrypts it has to encrypt all of that up to the point of the uh of the current uh segment in order to get the crypto state once it gets the crypto state it can encrypt that one segment we want to send and it can send it on the wire” 这意味着为了重传区区 1KB 多的数据包，网卡必须把整个 TLS 记录从开头全部 DMA 下来并在芯片内部跑一遍加密，直至恢复到目标分段的密码学上下文，随后才真正发出重传包。频繁的丢包重传会在 PCIe 总线上形成成倍放大的流量，极易在瞬间耗尽 PCIe 传输带宽。 --- ### [23:18](https://www.youtube.com/watch?v=_o-HcG8QxPc&t=1398s) 早期固件的吞吐“流沙陷阱”与会话分页假说 Netflix 在疫情暴发前拿到了一版 Mellanox 的早期测试版固件。初测时峰值带宽表现亮眼，整机能够达到约 250Gbps。但在真实负载持续灌入后，系统遭遇了灾难性的“流沙陷阱”（tar pit）。Netflix 的流量调度依托于一个基于 PID 控制器的系统健康度评估组件（称为 health），该组件监控网卡容量、CPU 负载以及磁盘饱和度等关键指标来动态引流。在硬件 kTLS 模式下，由于卸载了加密计算，CPU 占用率极低，网络硬件表面上看起来也远未达到极限，健康检查模块因而持续发出“我很健康，请塞更多流量”的指令。现实情况是，随着并发连接数增加，网卡性能发生断崖式下跌，单卡吞吐下探至 75Gbps（整机双网卡共 150Gbps）的底线，而系统却因指标“假健康”继续盲目吸纳更多客户端。Drew Gallatin 说明了这一现象的核心原因： > “and as more and more connections arrived the nick slowed down more and more and so we wound up at a floor of about 75 gigs in nick or 150 gigs per system and we just kept attracting more and more and more clients and it sort of turned into a tar pit because this kind of limit is invisible to our pid controller” 虽然缺乏 Mellanox 官方白皮书的正面确认，但 Drew Gallatin 结合驱动代码给出了技术推测： - 网卡需要为每个会话维持 TLS 状态以支持分段断点续传。 - Netflix 的并发连接规模极为庞大：普通播放客户端会同时建立 2 到 6 个视频连接，外加 1 个音频与 1 个字幕连接，偶尔遇到异常客户端甚至会发起成百上千条连接。以 400Gbps 吞吐折算，系统需要承载约 400,000 个活跃 TLS 会话。 - Mellanox 网卡自带的板载片上内存无法存放几十万个会话上下文。Mellanox 驱动会在主机内存中开辟专用空间供网卡自由使用；当会话规模暴增耗尽板载 SRAM 时，网卡被迫在片上内存与主机内存之间频繁换页（paging），进而导致严重的内存颠簸（thrashing）与吞吐雪崩。 --- ### [26:08](https://www.youtube.com/watch?v=_o-HcG8QxPc&t=1568s) 固件优化历程：PCIe Relaxed Ordering 与 380Gbps 突破为解决状态换页瓶颈，AMD 曾建议开启 PCIe 放宽排序（Relaxed Ordering），允许非依赖事务互相穿插越行，以提升网卡换页读取主机端 TLS 状态的吞吐。Netflix 在配置空间中开启了该特性，起初却毫无改善；后来才排查出 ConnectX-6 的旧版固件虽然暴露了配置位，底层逻辑却直接将其硬编码屏蔽（hardcoded off）。后续的固件迭代带来了一系列阶梯式的性能飞跃： - **正式启用 Relaxed Ordering 的新固件**：网卡真正激活了放宽排序支持，性能一举飙升至单卡 160Gbps（整机 320Gbps）。此时系统表现出高度稳定性，峰值吞吐与持续吞吐几乎拉平，彻底告别了“流沙陷阱”，并首次大幅超越纯软件 kTLS（200+ Gbps）的性能极限。单网卡 160Gbps 的水平已极度逼近该机器单卡跑软件 kTLS 时的物理极限（190Gbps，双卡同时运行会因内存带宽锁死而无法达到该值）。 - **包含 `tls_optimized` 开关的生产版固件**：数月后 Mellanox 提供了具备 `tls_optimized` 控制旋钮的固件。即便 Netflix 团队尚未探明该旋钮底层的具体微架构改动，但其实测表现极其惊人，直接将单卡带宽推至 190Gbps，整机双卡达成 380Gbps 的吞吐新高。 --- ### [28:00](https://www.youtube.com/watch?v=_o-HcG8QxPc&t=1680s) QoE 体验验证与丢包回退防护机制在取得 380Gbps 的基准性能后，Netflix 并没有立刻将硬件 kTLS 推向全量生产。由于流媒体业务对用户体验质量（Quality of Experience, QoE）极度敏感，必须评估核心体感指标：播放缓冲循环动画（rebuffer / 旋转等待图标）的触发频率、点击播放到首帧呈现的启动延迟，以及画质从低阶平滑跃升至 4K/HDR 高清画质的耗时。 Drew Gallatin 回顾了过往测试另一款不支持 TLS 记录断点续传的第三方网卡时的惨痛教训。为了配合那款网卡，网络协议栈专家 Randall Stewart 曾深度入侵 TCP 协议栈底层，强行让 TCP 探测 TLS 记录边界并极力按记录对齐切片发包；这种反常规的 TCP 拥塞控制调整最终摧毁了传输平滑度，导致灾难级的用户 QoE。相比之下，Mellanox 网卡能够原生在记录中间任意断点续传，初期 QoE 测试成绩十分优异。为防范高丢包网络下的潜在风险，Netflix 正在构建针对重传放大的防护体系： > “so whether it's intentional or not if you have a lossy tcp connection it can run you out of pci express bandwidth fast so you want to make sure that this doesn't happen” 这种重传放大效应在恶意滥用或恶劣链路下堪比针对 PCIe 总线的放大攻击。针对这一隐患，Netflix 的工程解决方案是在系统内核实时监控丢包网络下的重传字节比例。一旦某条连接的重传率超过安全阈值，内核将果断把该会话从硬件 kTLS 动态降级回退到纯软件 kTLS 处理，剥离对网卡加密引擎与 PCIe 带宽的非正常消耗，确保高并发骨干链路的绝对稳定。 ### [30:24](https://www.youtube.com/watch?v=_o-HcG8QxPc&t=1824s) 混合硬件与软件 kTLS 的性能损耗与防御机制为了防止恶意攻击者利用重传机制消耗网卡的 DMA 资源，Netflix 设立了严格的重传阈值控制机制。如果某些恶意用户试图绕过字节数限制，故意请求大量极小的数据包或分段重传，迫使网卡不断触发重新 DMA，系统不会费事将这些连接回退至软件加密，而是直接切断连接。正如 Drew Gallatin 所说：“在那种情况下，我们可能会直接断开连接——我们切断连接而不是移到软件，因为很明显这是一次攻击。”（“and in that case we'll probably kill the connect we kill the connections rather than moving them to software since it's pretty clear it's an attack”）然而，当系统在真实环境下运行硬件与软件混合模式时，却出现了意料之外的性能下降。Drew Gallatin 设定了一个保守的安全阈值：只要一条 TCP 连接的重传字节比例超过 1%，该连接就会被移回由软件内核 TLS（kTLS）处理。这一规则导致大约 25% 到 33% 的连接被回退到软件执行。理论上系统的内存带宽和 DMA 延迟都没有达到瓶颈，但系统的最大稳定吞吐量却从 380 Gbps 直线下滑到了 350 Gbps。这种性能降幅远超预期，Drew 坦言自己还未彻底查清根因：“仅仅三分之一的连接真的不应该给系统带来这么大的压力，而且请记住内存带宽看起来没问题，DMA 延迟也看起来没问题，我就是不明白为什么情况会这么糟。”（“just a third of the connections really shouldn't be putting that much stress on the system and remember memory bandwidth looks fine the i the the latency the dma latency looks fine i just don't understand why it's as bad as it is”） - 防御策略：对重传分段异常偏多的连接直接采取 RST/终止连接策略，杜绝针对网卡 DMA 的 DoS 攻击。 - 回退阈值：将重传字节比例上限设为 1%，触发后回退到软件加密。 - 混合模式性能表现：约 25%~33% 的连接被移交至软件，吞吐量由 380 Gbps 下降至 350 Gbps。 --- ### [31:33](https://www.youtube.com/watch?v=_o-HcG8QxPc&t=1893s) 硬件 kTLS 与 NUMA 结合：架构简化与性能总览将硬件 kTLS 与 NUMA 优化结合在一起，被 Drew Gallatin 幽默地比作 80 年代经典 Reese's 花生酱巧克力广告的桥段：“这有点像‘你把巧克力掉进我的花生酱里了’那种时刻。”（“this is a sort of the you put your chocolate in my peanut butter kind of moment if you remember those ads from the 80s”）。在这种架构下，整个数据流转链路大幅精简：当 TLS 连接在系统的某个节点建立，数据从对端节点的内存中拉取过来后，CPU 完全不需要参与任何加解密计算，直接通过 PCIe 交由网卡完成加密并发送。在物理拓扑中，服务器并非每个 NUMA 节点都插有网卡。即便在最坏情况下需要跨越互联总线走到远端节点的网卡，其跨 NUMA 访问开销也与此前无异；系统彻底去除了数据在加密过程中往返内存的迂回惩罚，省下了至关重要的内存带宽。 ```mermaid flowchart LR accTitle: Hardware kTLS with NUMA flow accDescr: Flow diagram showing how data avoids crypto memory detours by directly DMAing to the NIC. subgraph StorageNode ["数据所在节点 (Storage Node)"] Data["页面缓存数据 (Page Cache)"] end subgraph EgressNode ["网卡所在节点 (Egress Node)"] NIC["Mellanox 200G 网卡 (硬件 kTLS 引擎)"] Wire["400G 网络输出 (Wire)"] end Data -->|"跨互连总线 DMA (无 CPU 内存绕行)"| NIC NIC -->|"线速加密并发送"| Wire ``` Drew Gallatin 随后展示了一张详尽的性能演进对比图表，梳理了系统在不同阶段达成的带宽与 CPU 开销： | 模式 / 架构配置 | 稳定网络带宽 | CPU 占用率 | 瓶颈 / 特性说明 | |:--- |:--- |:--- |:--- | | **软件 kTLS（Flat 拓扑）** | 约 240 Gbps | 约 80% | 内存系统带宽饱和，CPU 达到瓶颈 | | **软件 kTLS + NUMA 优化** | 约 280 Gbps | 约 80% | 局部化访问释放带宽，提升 40 Gbps | | **硬件 kTLS（AMD Flat 拓扑）** | 约 380 Gbps | 约 60% | 彻底卸载加密负载，突破内存带宽墙 | | **硬件 kTLS + NUMA 优化** | 约 380 Gbps | 约 50% 刚过 | 带宽维持在 380G 极值，但 CPU 开销进一步显著降低 | | **实际生产部署配置（Flat + 丢包防护）** | 约 350 Gbps | 约 65%~70% | 硬件 kTLS 配合 1% 重传回退安全机制 | --- ### [34:02](https://www.youtube.com/watch?v=_o-HcG8QxPc&t=2042s) 备选平台测试：Ampere Altra 80 核 ARM 服务器调优在评测的替代硬件平台中，Netflix 深度测试了 Ampere Altra 系统。这台服务器配备单颗 3.0 GHz、80 核心的 ARM Neoverse N1 处理器。粗看之下其纸面规格与 AMD EPYC 平台惊人地相似：同样具备 8 通道内存、相同容量的 RAM、相近数量的 PCIe 通道，以及相同的 NVMe 存储和 Mellanox 网卡。唯一的根本变量在于 CPU 架构完全不同。 Drew 指出在 FreeBSD 上适配全新 ARM 架构的最大痛点在于观测性匮乏：“因为这是完全不同的架构，至少在 FreeBSD 上，我目前没办法像在 Intel 和 AMD 上那样做各种好玩的事情，比如查看消耗的内存带宽、运行高级性能分析器、查看正在使用的 I/O 带宽，这让我觉得自己有点在盲开。”（“and because it's very different architecture at least on freebsd i don't currently have a way to do all the fun stuff i do on intel and amd like looking at memory bandwidth being used and running fancy profilers and looking at io bandwidth being used and so it kind of makes me feel like i'm kind of driving blind”）在软件 kTLS 的最初测试中，Ampere 平台表现糟糕，仅跑出了 120~140 Gbps。内核开发者 Mark Johnston 提出了一个关键优化：利用 FreeBSD 的 UMA（通用内存分配器）缓存区预先缓存 16KB 连续的 TLS 加密目标缓冲区，这一改动立竿见影地将软件带宽拉升至 180 Gbps。Drew 盛赞这一方案：“Mark Johnston 想出了一个绝妙的主意，事后看来所有好主意都显而易见，我甚至怪自己怎么没想到，但他想到了用 UMA 缓存区去缓存 16K 连续的 TLS 加密目标缓冲区，一旦我们这样做了，带宽就达到了大约 180 Gbps。”（“mark johnson came up with an idea which is brilliant and which i kick myself for not thinking of but all brilliant ideas are obvious in hindsight and he came up with the idea of using a uma cash zone to cache 16k contiguous tls crypto destination buffers and once we did that we got up to about 180 gigabits a second”）切换到硬件 kTLS 时，团队遇到了新的 PCIe 瓶颈：吞吐量被死死压制在 240 Gbps 左右。此时 CPU 利用率低得惊人（仅约 20%），但网卡传输队列却开始持续丢包。经排查，Drew 发现 FreeBSD 运行在 Ampere 上时没有启用 PCIe 扩展标签（Extended Tags）。PCIe 实际上更类似于网络而非单纯的总线，默认的 5 位标签空间仅允许 32 个并发事务，就像 TCP 窗口太小限制了流水线。开启扩展标签后，标签空间扩大至 8 位（256 个并发标签），流水线瞬间被填满，Ampere 平台的硬件 kTLS 带宽直接跃升至 320 Gbps。 - 处理器规格：Ampere Altra（80 核心 ARM Neoverse，主频 3.0 GHz）。 - 软件 kTLS 调优：由初始 120~140 Gbps 经 UMA 连续 16K 缓冲区优化提升至 180 Gbps。 - 硬件 kTLS 调优：默认仅 32 个 PCIe 标签导致网卡在 240 Gbps 饱和丢包；启用 8 位扩展标签（256 tags）后，带宽突破至 320 Gbps。 --- ### [37:07](https://www.youtube.com/watch?v=_o-HcG8QxPc&t=2227s) Intel Ice Lake 踩坑记、各平台横向汇总与致谢在演讲最后，Drew Gallatin 分享了刚刚运抵数据中心仅一周的 Intel Ice Lake Xeon 平台测试进展。该平台在存储拓扑上遇到了极大的工程麻烦：由于它仅支持 64 条 PCIe Gen 4 通道，无法像 AMD 平台那样挂载大量 PCIe Gen 3 盘，系统不得不改用 Gen 4 盘或 PCIe 交换芯片（PCIe Switch），而测试用的交换芯片又出现了各种诡异 Bug。最终系统搭建为 20 块 Kioxia 4TB NVMe SSD 配对两块 Mellanox 200G 网卡。令 Drew 感到无奈的是内存频率缩水：“这里有一个重要的区别是我们虽然有 8 通道 DDR4-3200，但这一款 Intel Xeon SKU 却把它们降频运行在 2933。我个人很讨厌这一点，我真希望 Intel 能像其他厂商一样在整个产品线中统一内存频率，但现实就是这样。”（“one important difference here is that we have eight channels of ddr3200 but this skew of the intel xeon runs them at 2933 i personally hate that i wish intel ran all their memory at the same speed like everybody else does across across their product lines but it is what it is”）。由于内存频率仅为 2933 MHz，Intel 平台的软件 kTLS 吞吐被严格限制在 230 Gbps。对于硬件 kTLS，测试机遇到了来自 BIOS 的“地狱级”阻碍（“running the bios from hell”），各类底层特性被锁定；演讲前周五收到的松散重排（Relaxed Ordering）补丁未能生效，因此无法公布有效的硬件卸载数据。 Drew 总结了三大平台的实际对比数据，并留下了 800G 系统的悬念： | 平台架构 (CPU) | 软件 kTLS 最大带宽 | 硬件 kTLS 最大带宽 | 现状与主要瓶颈 | |:--- |:--- |:--- |:--- | | **AMD (EPYC Rome/Milan)** | **~280 Gbps** | **~380 Gbps** | 表现最优，达成了准 400G 线速 | | **Intel (Ice Lake Xeon)** | **~230 Gbps** | *测试未完成* | 内存被降频至 2933 MHz；受困于 BIOS 与 PCIe RO 限制 | | **Ampere (Altra ARM64)** | **~180 Gbps** | **~320 Gbps** | 缺少分析工具；通过 UMA 缓存及 8-bit PCIe 扩展标签优化 | 原本计划在演讲结尾展示的 800 Gbps 原型机，因美国境内的恶劣风暴天气导致物流延误，未能赶在大会前上架。在进入问答环节前，Drew 向团队成员表达了真挚的感谢：包括负责硬件原型的 Netflix 工程师 Warren Harrop（被 Drew 赞为“给我送来所有新玩具的圣诞老人”）、FreeBSD 核心开发者 Mark Johnston（在 NUMA、VM 虚拟内存和 kTLS 上的杰出贡献）、John Baldwin（主导了 kTLS 的上游合入并协调各硬件供应商制定接口规范），以及攻坚 NUMA 子系统的 Jeff Roberson。 ### [40:56](https://www.youtube.com/watch?v=_o-HcG8QxPc&t=2456s) 问答环节开启与 FreeBSD 内核对网卡 kTLS 的支持情况演讲进入最后的问答环节，主持人提醒 Drew Gallatin 可以在会议系统的“Shared Notes”（共享笔记）标签页中查看整理好的问题列表。Drew 切换界面后，首先解答了关于 FreeBSD 现阶段对网卡硬件 TLS（NIC kTLS）卸载支持的具体情况。对于“NIC kTLS 是否已进入 FreeBSD -CURRENT 分支”的提问，Drew 给予了肯定答复，并补充表示该特性实际上也已经反向合并到了 FreeBSD 13 稳定分支中。不过，该功能对硬件有较强的依赖性，只能与特定的网卡协同工作。就 FreeBSD 目前的驱动支持而言，它仅适用于 Mellanox ConnectX-6 Dx 以及 Chelsio T6。当被问及“NFS over TLS 的速度有多快”时，Drew 坦承自己并不了解具体数字，建议对此感兴趣的观众向负责该特性的 Rick Macklem 咨询。 - **FreeBSD 分支支持**：NIC kTLS 已合并入 FreeBSD -CURRENT，并存在于 FreeBSD 13 分支中。 - **兼容网卡型号**：目前在 FreeBSD 下仅支持 Mellanox ConnectX-6 Dx 与 Chelsio T6。 - **NFS over TLS 状态**：性能数据需向主要开发者 Rick Macklem 求证。 --- ### [41:43](https://www.youtube.com/watch?v=_o-HcG8QxPc&t=2503s) PCIe 交换芯片带宽异常与内存超频的影响现场观众提出 Netflix 在构建 PCIe 拓扑时是否使用了 PLX 芯片来连接 PCIe Gen3 与 Gen4 设备。Drew 回忆称他们当时使用的是 Broadcom 的 PCIe 交换芯片（Broadcom 此前收购了 PLX），但在使用过程中遇到了令人费解的带宽瓶颈问题。在特定配置下，比如将一个上行 PCIe Gen4 x8 接口拆分连接到四个下行的 PCIe Gen3 x4 接口时，如果只有一到两个 Gen3 x4 设备处于活动状态，系统表现一切正常；但一旦活动的 Gen3 x4 设备数量超过两个，整体吞吐量就会骤降至如同整个链路都受限于 Gen3 的速度。Drew 感叹，正是这种奇特的行为，导致他们在最初尝试用仅有 64 条 PCIe 通道的机器构建高吞吐原型时“遭遇了各种各样的问题”。随后有人询问，如果采用针对客户端/游戏玩家（Client SKUs）的高频 DDR 内存，是否能有助于缓解 Infinity Fabric 的互连带宽限制。Drew 表示，虽然 Infinity Fabric 本身存在固有上限，但更快的内存在内存带宽方面绝对会有直接帮助。他笑着透露，自己以前就经常在服务器上对内存进行超频操作以榨取性能。 --- ### [42:55](https://www.youtube.com/watch?v=_o-HcG8QxPc&t=2575s) 拒绝 TCP 卸载的架构哲学与硬件 TLS 的流溢出处理机制在探讨网络协议栈时，Drew 明确表示 Netflix 并没有采用完整的 TCP 卸载引擎（TOE）。他解释道，Netflix 内部有一个与他们平行的专属团队，专门且排他地专注于传输协议的改进，团队成员包括著名网络协议开发者 Randall Stewart 和 Lawrence Stewart（Drew 特别幽默地顺带澄清：“他们之间没有亲戚关系”）。Netflix 极其看重对 TCP 协议栈每一个细微环节的绝对掌控权。正如 Drew 所阐述的： > “And we one of the things that we depend on is to control every aspect of the TCP stack and that's what allows us to give such good QoE to our customers.”（我们所依赖的核心优势之一，就是能够精准掌控 TCP 协议栈的方方面面，而这正是我们能够向客户提供如此出色体验质量的原因。）针对硬件 TLS 能够支持多少并发流（Flow）以及连接数超限后如何处理的疑问，Drew 解释道，虽然具体的硬件上限他有些模糊（印象中每个端口大约是 128,000 条），但系统的回退机制非常清晰：一旦硬件会话资源溢出（Overflow），系统就不会在网卡硬件上建立 TLS 加密会话，而是无缝回退（Fallback）到基于 CPU 的软件 kTLS 模式来处理后续流量。 ```mermaid flowchart TD accTitle: Hardware kTLS fallback flow accDescr: Diagram illustrating how TCP connections are processed by hardware kTLS or fallback to software kTLS based on hardware capacity. A["新建立的 TLS 连接请求"] --> B{"网卡硬件会话表 是否未满？"} B -->|"是 (资源可用)"| C["启用网卡硬件 kTLS (由 ConnectX-6 Dx / Chelsio 加密)"] B -->|"否 (资源溢出)"| D["优雅降级回退 (Fallback to Software kTLS)"] D --> E["由主机 CPU 内存带宽与指令集加密"] C --> F["数据流出网卡"] E --> F ``` --- ### [43:49](https://www.youtube.com/watch?v=_o-HcG8QxPc&t=2629s) 存储数据加密与 LACP 流量分流现实关于底层存储安全，有观众提问存储介质中的静态数据是否也经过了加密。Drew 澄清说，在存储层面上并没有额外的静态数据加密（Encryption at Rest）。Netflix 的视频资产在进入分发网络前，已经在云端转码阶段通过数字版权管理（DRM）进行了端到端加密，这种加密从云端转码一直持续到用户的机顶盒或手机等终端设备中，因此在边缘缓存服务器的磁盘上并不需要重复加密。随后，观众询问 Netflix 是否会将流量主动调度/导流（Traffic Steering）到最优的网卡上。Drew 直截了当地回答：“不，我们不做导流，我们完全接受链路对端设备做出的哈希决策。”他回忆说，几年前在一次演讲后曾有人找他讨论过可以通过特定协议通知 LACP 链路聚合重定向流量，但 Netflix 并没有实现这套机制。目前服务器完全顺其自然，对端 LACP 交换机把数据包哈希发往哪张网卡，系统就照单全收。 --- ### [44:37](https://www.youtube.com/watch?v=_o-HcG8QxPc&t=2677s) 性能监控工具 nstat 与智能网卡（SmartNIC）的功耗考量在被问及如何监控和统计 CPU 使用率时，Drew 表示他的统计指标在本质上与系统自带的 `vmstat` 相同，但他日常并不直接使用 `vmstat`。为了满足综合观测需求，他亲自编写了一个名为 `nstat` 的工具，并且已经开源提交到了 FreeBSD 的 Ports 软件仓库中。 `nstat` 能够在一行终端输出中，将排查性能瓶颈所需的全部核心指标紧凑地聚合呈现出来，包括调用 Intel PCM 获取的内存带宽、CPU 使用率、当前 TCP 连接总数、网络实时吞吐带宽、上下文切换次数以及中断频率等。对于观众提出的关于 Xilinx 智能网卡（SmartNIC）的看法，Drew 直言他对具体型号并不了解，但指出 Xilinx 让他联想到了 FPGA 方案，而 Netflix 通常更青睐成熟的 ASIC 芯片方案，核心考量是功耗控制。他分享了一个真实的评估经历：Netflix 曾经与一家网卡厂商深入接洽，对方研发了一款基于 FPGA 且板载 40GB 到 64GB 大容量板载内存的硬件 TLS 卸载网卡。这种设计在处理 TCP 重传（Retransmit）时非常理想，因为网卡可以直接在板载缓存中保留并重放所有数据。然而，测试结果让团队望而却步——仅仅一张网卡的功耗几乎就赶上了整台服务器系统的功耗，这彻底违背了能效比原则，因此根本不适合大规模部署。 - **`nstat` 监控指标一览**： - CPU 利用率 - 内存实际带宽（结合 Intel PCM） - 网络吞吐量与 TCP 连接数 - 核心中断（Interrupts）与上下文切换（Context Switches） - **ASIC vs FPGA 网卡**：FPGA 加大内存方案虽能完全解决网卡端重传缓存问题，但其功耗极其夸张，整机能效比难以接受。 --- ### [45:59](https://www.youtube.com/watch?v=_o-HcG8QxPc&t=2759s) FreeBSD 在 NIC TLS 领域的架构超前性与结语在问答的尾声，Drew 总结了 Netflix 对新硬件的开放态度，表示他们始终愿意尝试各种新型网卡，并已经与多家厂商展开了沟通。谈及系统底层实现时，Drew 充满自豪地指出，FreeBSD 在网卡 TLS（NIC TLS）卸载的设计理念上实际上已经走在了行业前列。 Drew 透露，当年他最初着手为 FreeBSD 设计内核态 TLS（Kernel TLS）架构时，脑海中就已经提前构想好了未来支持 NIC 硬件 TLS 卸载的蓝图。因此，他在设计内核内存缓冲区（mbuf）与 kTLS 的交互方式时，特意植入了前瞻性的底层规范：在 FreeBSD 中，单个 mbuf 结构完整地描述并对应一个完整的 TLS 记录（TLS Record）。这种优雅的内存设计为网卡驱动带来了无与伦比的便利，当网络发生丢包重传时，驱动程序只需在同一个 mbuf 的上下文范围内检索，就能直接获取完成该 TLS 记录重传所需要的一切元数据和负载信息，免去了繁琐的内存重组与状态切分。在确认屏幕上没有新增的技术问题后，Drew 礼貌地向在线参与的所有观众道谢，正式结束了整场演讲。 ## Notable quotes > [42:55](https://www.youtube.com/watch?v=_o-HcG8QxPc&t=2575s) And we one of the things that we depend on is to control every aspect of the TCP stack and that's what allows us to give such good QoE to our customers. — Drew Gallatin > [06:46](https://www.youtube.com/watch?v=_o-HcG8QxPc&t=406s) Keep as much of our bulk data off the NUMA fabric as we can. — Drew Gallatin > [28:00](https://www.youtube.com/watch?v=_o-HcG8QxPc&t=1680s) so whether it's intentional or not if you have a lossy tcp connection it can run you out of pci express bandwidth fast so you want to make sure that this doesn't happen — Drew Gallatin ## Mentioned Netflix, FreeBSD, Drew Gallatin, Nginx, AMD, AMD EPYC, Rome, Milan, Mellanox, ConnectX-6 Dx, Chelsio, Chelsio T6, Intel, Intel Ice Lake, Intel QAT, Ampere, Ampere Altra, ARM Neoverse N1, Mark Johnston, Randall Stewart, Lawrence Stewart, John Baldwin, Jeff Roberson, Warren Harrop, Rick Macklem, Broadcom, PLX, Kioxia, Xilinx ## Video description In this talk, I will discuss the efforts to serve TLS encrypted Netflix video at 400Gb/s from a single server. This will be a follow-on to 2 talks at the 2019 EuroBSDCon: “Numa Optimizations in the FreeBSD Network Stack” and “Kernel TLS and TLS Hardware Offload“. I will provide background on the Netflix video workload, and define key technologies such as NUMA, kernel TLS and hardware kTLS. I will describe encountering bottlenecks such as: - Memory bandwidth limits for software kTLS - PCIe issues with hardware kTLS - NUMA for software vs hardware kTLS I will present current and historical performance results from at least: - AMD “Rome” 2nd generation EPYC systems - AMD “Milan” 3rd generation EPYC systems - Ampere Altra arm64 systems Drew Gallatin Drew started working on FreeBSD at Duke in the 90s, and was one of the people behind the FreeBSD/alpha port. He worked on zero-copy TCP optimizations for FreeBSD and was sending data at over 1Gb/s before gigabit Ethernet was generally available. He spent a decade at Myricom, optimizing their drivers. After a brief hiatus at Google, he landed at Netflix, where he works on optimizing the FreeBSD kernel and network stack for content delivery. He worked on the optimizations to serve unencrypted Netflix traffic at 100Gb/s, and then on more optimizations to send encrypted traffic at ever increasing speeds, from 100Gb/s to 400Gb/s and beyond. [Source](https://www.youtube.com/watch?v=_o-HcG8QxPc)

Source quotes

[42:55] Drew Gallatin: And we one of the things that we depend on is to control every aspect of the TCP stack and that's what allows us to give such good QoE to our customers. [6:46] Drew Gallatin: Keep as much of our bulk data off the NUMA fabric as we can. [28:00] Drew Gallatin: so whether it's intentional or not if you have a lossy tcp connection it can run you out of pci express bandwidth fast so you want to make sure that this doesn't happen

Source quotes

[6:46] Drew Gallatin: Basically, the strategy that I came up with a few years ago is to keep as much of our bulk data off the NUMA fabric as we can. [20:09] Drew Gallatin: we have to watch out for TCP re-transmissions because they could easily run us out of PCI Express bandwidth. [12:05] Drew Gallatin: i believe that network-centric siloing is by far the best way to go.

Original transcript

[0:00] so my name is drew gallatin and i'm from [0:03] netflix and i focus on uh [0:06] basically cpu efficiency and trying to [0:08] serve uh as many customers as we can [0:11] from as little hardware as we can [0:14] both to save the environment and to save [0:17] ourselves some costs [0:19] so for the last little bit i've been [0:21] trying to push the envelope with serving [0:22] as much as we can from a single box and [0:24] i'm here to talk to you about getting as [0:26] close as we can to serving 400 gigs from [0:28] a single [0:29] server on freebsd [0:31] so i'm going to talk about you know why [0:33] we want to do this and i'm going to [0:35] describe a little bit about our [0:37] production platform [0:39] and talk about uh the workload that we [0:41] have [0:42] and then talk a little bit about whether [0:43] or not we need pneuma or not to achieve [0:45] these goals [0:46] uh talk a little bit about hardware [0:48] inline tls which is in freebsd we called [0:51] nick tls [0:52] and then if i have time at the end i'll [0:54] loop around and look at some alternate [0:56] platforms in addition to the production [0:57] platform that i'll be talking about for [0:59] most the talk [1:02] all right since 2020 we've been rolling [1:04] out uh some servers that uh have been [1:06] able to serve 200 gigs of uh tls [1:09] encrypted video [1:11] so [1:14] that's really cool um [1:16] the thing is these servers actually have [1:19] four 100 gig network ports and we've [1:22] always thought well wouldn't it be nice [1:23] if we could actually light them all up [1:25] and serve 400 gigs [1:26] from these machines rather than two [1:29] so let's look at [1:30] what [1:31] let's look at our uh workload a little [1:33] bit so [1:35] we run freebsd current uh we're [1:37] generally you know a few weeks behind [1:39] head not not too far we run the nginx [1:42] web server [1:43] and we serve our video uh almost [1:46] entirely the ascend file [1:48] and that means that we [1:50] love kernel tls because [1:52] that uh means the data is encrypted in [1:55] the kernel as part of the send file [1:56] pipeline which means we don't have any [1:58] extra boundary crossings where data is [2:01] like you know pulled off a disk and [2:03] encrypted user space and then written [2:05] back to the kernel you know none of [2:06] those detours and just by itself [2:07] software kernel tls saves us like [2:09] roughly 60 cpu [2:13] um the production platform i'm keep [2:16] referring to is an amd uh epic 7502p [2:20] which is otherwise known as a rome it [2:22] has 32 cores at two and a half gigahertz [2:25] and the more important thing is it has [2:27] eight channels of uh ddr4 3200 [2:31] and that's enough for 150 gigabytes a [2:33] second of memory bandwidth [2:35] and [2:36] now through this talk i'm going to be [2:38] talking about networking stuff which is [2:39] gigabits and [2:42] like storage slash memory bandwidth [2:44] stuff which is typically gigabytes i'm [2:46] going to try to remember to go back and [2:48] forth by that factor of eight and if i [2:50] forget that i apologize [2:52] but i'm just going to try to do both [2:53] things here and mention that that's [2:55] about uh 1.2 terabits of of bandwidth in [2:59] terms of memory bandwidth we have plenty [3:01] of io we have um [3:04] basically 128 lanes of pcie gen 4 [3:08] which is uh about [3:10] two terabits in networking units [3:13] so all that sounds good and like i [3:15] mentioned before we have two uh melanox [3:18] connect x6 dx nix they connect at [3:21] gen 4 [3:22] and by 16 and they have two full speed [3:25] ports each so that's four 100 gig ports [3:28] and hence our motivation we want to use [3:30] all four of those ports [3:32] and we have a whole bunch of nvme drives [3:35] and these machines were specced out way [3:37] before uh we [3:39] you know fund a full speed [3:41] gen 4 drive so we have just a lot of gen [3:43] 3 drives [3:45] so our initial performance results in [3:46] this platform are about 240 gigabits a [3:50] second and this is with software kernel [3:52] tls [3:53] and we're limited primarily by [3:56] memory bandwidth [3:57] and the way that i know that i'm not [4:00] just mystical basically amd has given us [4:03] tools uh [4:05] and you can i think you can get this [4:06] tool yourself if you google for it amd [4:08] up pcm [4:10] and it reads the system counters and it [4:11] tells you what the memory bandwidth is [4:16] so the um [4:18] so that the gist of it is that you know [4:20] we reach the the rated system memory [4:22] bandwidth at about 240 gigabits a second [4:25] and at that point you know things start [4:26] to collapse [4:28] so [4:29] to understand where the bandwidth is [4:31] going i'm going to bring you back to [4:32] this slide that [4:34] should look familiar if you saw my talk [4:36] two years ago in fact a lot of this next [4:38] section is going to look familiar if you [4:40] saw my talk two years ago so i apologize [4:42] for that [4:43] basically um i wish i had a laser [4:46] pointer i was trying to play with my [4:47] mouse and it just doesn't work well [4:48] enough to see but [4:50] essentially the data flow is that [4:52] a netflix client uh you know you're [4:55] watching uh you're watching a movie [4:56] netflix client has an algorithm where it [4:58] says it wants to fetch you know so many [5:00] megabytes ahead so it says it wants to [5:03] fetch the next two megabytes [5:06] and nginx gets their request for that [5:08] the and it launches a uh a send file [5:11] request to satisfy it [5:13] and [5:13] what what happens there is that the the [5:16] data is brought in from those this nvme [5:18] drives in the lower left-hand corner [5:20] and it's you know in aggregate it's read [5:22] 50 gigabytes a second and it's right [5:24] into system memory now once it's right [5:26] into system memory we have to encrypt it [5:28] to send it on the wire [5:30] so the cpu then reads it out of memory [5:32] again uh encrypts it and then allocates [5:36] a separate destination buffer which is [5:38] you know kind of an important aside [5:40] basically [5:41] most crypto does uh crypto in place but [5:44] because we're talking about files here [5:46] that live in the [5:47] in the you know the page cache [5:50] we cannot encrypt in place because if we [5:52] did then that would if uh i was [5:54] streaming something and you were [5:55] streaming something and we encrypted it [5:57] to my uh tls keys then you would get [6:00] something that was encrypted uh doubly [6:03] and it would just be garbage and it [6:04] would be a bad day so we have to encrypt [6:07] to a separate tls crypto buffer [6:09] so once we do that we write it back out [6:11] at 50 gigabytes a second and then the [6:13] network uh card comes along and dmas it [6:16] and sends it out on the wire and that's [6:18] another 50 gigabytes a second so if you [6:20] add all those things up that's um [6:23] that [6:23] that 50 gigabytes a second gets [6:25] multiplied by four so you wind up with [6:27] um [6:29] four [6:30] you wind up with 200 gigabytes a second [6:33] or 1.2 terabits sorry 1.6 terabits a [6:36] second of a bandwidth that you need to [6:39] achieve your goal and obviously we don't [6:41] have that much memory bandwidth [6:44] so [6:46] one of the things i was wondering is can [6:47] numa get us closer so [6:50] one of the things numa does is it makes [6:53] when you run amd machines in numa mode [6:55] and i'm going to explain what numa is in [6:56] a second but if you run amd machines in [6:58] numa mode [6:59] you can end up making more efficient use [7:01] of their memory controllers and so if [7:03] you look at stream results like you know [7:05] on the web you see that for the system [7:07] that we have people generally get about [7:09] 150 gigabytes a second from uh [7:12] in flat mode but if you run the amd [7:14] system in four nodes per socket you get [7:15] about 175 gigabytes a second [7:19] and so [7:21] what is pneuma so numa is uh non-uniform [7:24] memory architecture now this is the [7:25] stuff that these slides are basically [7:27] dupes of what i did two years ago their [7:30] background if you already know this then [7:32] you know get a cup of coffee [7:33] basically it means that uh you know [7:35] memory or devices can be closer to each [7:37] other so if you go back in in the early [7:39] days when we built uh big [7:40] multiprocessors you could just plug in [7:44] cpus and memory and disk wherever you [7:46] wanted to [7:47] everything was equally close to [7:49] everything else [7:50] software people just didn't have to care [7:52] about where things were [7:54] but it didn't scale very well so [7:55] hardware people eventually realized that [7:57] they the best thing to do is to divide [7:59] and conquer [8:00] and so what what they came up with was [8:03] this newa system where essentially you [8:05] can tie multiple computers together with [8:08] the pneumobus so if you if you look at [8:10] the system on the left and the system on [8:13] the right inside those uh inside those [8:15] uh blue circles [8:17] essentially each one of those systems is [8:19] almost a complete computer it has a [8:20] network card it has memory it has disks [8:22] and has a processor [8:24] and ideally you wanted to stay within [8:26] that blue circle if you don't if you [8:28] cross that red dotted line then you wind [8:30] up uh uh crossing the numerabus and your [8:33] access to the things on that side of the [8:35] on that side of the computer ends up [8:37] being slower you know different pneuma [8:39] buses have different efficiencies the [8:41] penalty could be almost nothing or it [8:42] could be really severe it just depends [8:44] on what you're talking about [8:46] so it gets even more complicated with [8:48] with the amd epics because the most [8:51] efficient way to run them is to run them [8:53] in four nodes per socket mode and so now [8:56] we've just sort of multiplied the [8:57] problem and instead of having two number [8:59] domains now we have four and luckily [9:01] they're all interconnected [9:06] so you just get a picture of like how [9:08] complicated things get [9:10] and the latency penalties across these [9:12] dotted lines depending on which ones you [9:13] cross and what you're doing [9:15] uh can be anywhere from 12 to 28 [9:17] nanoseconds so it's it's not free [9:20] and the bandwidth limit is a for the [9:22] system that we have is about 47 [9:24] gigabytes a second per link or 280 [9:27] gigabytes a second total [9:30] so basically the strategy that i came up [9:33] with a few years ago is to keep as much [9:34] of our bulk data off the pneuma fabric [9:37] as we can [9:38] essentially you can sort of think of you [9:39] know if you if you have a highway and [9:41] you want to uh you don't want to fill it [9:43] up with big semi trucks and then block [9:46] the block the commuter cars [9:50] basically if your highway is full of [9:52] this bulb data like these big semi semi [9:54] trucks then the commuter cars can't get [9:56] through on their urgent missions and you [9:58] wind up uh blocking the cpu while you [10:01] wait for small things to get through and [10:03] i'm talking about even simple things [10:04] like you know updating a counter that [10:06] lives in a different human domain or [10:08] maybe a process [10:10] you know needs to read needs to read [10:11] something from some other domain it's [10:14] you want to keep those links as idle as [10:15] you can [10:17] so i'm going to go through and talk [10:18] about kind of the data flow and the [10:20] worst things that could possibly happen [10:22] so in the worst case um a request comes [10:24] in and let's say it comes into the lower [10:26] left newman node and i really wish i had [10:28] a laser pointer because this mixes a lot [10:29] better maybe you can see my [10:31] maybe you can see my mouse moving [10:33] um the request comes in here [10:35] and [10:36] we happen to have the content in a [10:38] different place so [10:40] we asked her we have to oops we asked to [10:42] read the content [10:44] and the content goes into the uh [10:47] into the the note on the upper right [10:48] hand side uh because we allocated the [10:50] the memory in the wrong place since it [10:52] was actually in our local node we [10:53] probably should allocated memory here [10:54] but we were just too dumb to do that so [10:56] now we cross the name of us [10:58] and now uh once it's in memory the cpu [11:01] needs to read it to encrypt it and so [11:03] ideally we pick a cpu that was up here [11:06] and the same node but since we're being [11:07] stupid uh we pick one that's on the [11:09] lower right hand side [11:11] and we cross the pneumabus again [11:14] now after uh after we encrypt it we need [11:17] to put it somewhere so we have to [11:18] allocate a destination crypto buff [11:20] and once we and once we you know if we [11:22] were being smart we'd allocate it right [11:24] where we are but we're not so we [11:25] allocate one and the upper left hand [11:27] side [11:28] and after all is said and done we need [11:31] to send it out and that network card is [11:33] in the lower left hand side [11:35] and so we cross the pneuma bus basically [11:37] four times [11:39] and [11:40] that's pretty close to the aggregate [11:41] fabric bandwidth and that causes the [11:43] fabric to congest and that's uh that's [11:46] bad [11:48] so we don't want to do that we have to [11:49] do something a little bit smarter [11:51] so in the best case in the ideal case we [11:53] just stay in that we just stay in that [11:55] lower left hand side there and never [11:56] leave it and that would lead to zero [11:58] pneuma crossings and nothing on the [12:00] fabric [12:01] um that's where we that's where we want [12:02] to get but how close can we get [12:05] and basically we're constrained to using [12:07] one ip address [12:08] per machine [12:09] and we're using lacp for bonding so [12:13] i came up with two ideas back in the day [12:15] one is to sort of focus on where the [12:16] content lives i call that disc centric [12:18] styling and one is to focus on [12:21] where the connection came in and then i [12:23] call that network centric siloing and [12:26] basically this this word that what the [12:29] lacp partner chose for is essentially [12:30] when you're using lacp [12:32] the link partners will hash uh you know [12:35] on their own criteria typically based on [12:37] tcp ports [12:39] and nip addresses [12:41] to [12:42] to get a good sharding of traffic to all [12:44] the link members and it's a consistent [12:47] as long as the lag doesn't isn't [12:50] isn't fl flapping and members coming and [12:52] going that hashing is consistent so you [12:54] wind up with the same connection going [12:55] the same neck all the time [12:59] so what i moved forward with was the [13:01] network centric siloing and i actually [13:03] fleshed out this disclaimer and i have [13:04] some backup slides at the end if i have [13:06] questions about that i've run that to [13:08] completion and and i believe that [13:09] network-centric styling is by far the [13:11] best way to go so basically you allocate [13:13] um [13:14] the network connections uh with affinity [13:16] to the pneuma nodes uh you allocate [13:18] memory to back the media files [13:20] uh when they're dna from disk local you [13:23] allocate local memory for tls crypto and [13:25] you run tls workers uh h you know tcp [13:28] pacers [13:29] tcp timers and all that kind of stuff [13:31] with domain affinity and you choose a [13:33] local nic to send the to send the [13:36] results back to the client [13:38] so all of this is actually upstream i [13:40] think it might even be in 13 i mean we [13:41] run current i don't track 13 very well [13:44] but i know all of this stuff is is [13:45] actually upstream [13:49] so now let's look at the worst case when [13:51] we're making these better decisions it's [13:53] the same scenario as before [13:55] essentially the [13:58] network connection comes in on the on [13:59] the lower left side [14:01] this time though the content lives on [14:03] the right side just so i have something [14:04] to something to show you [14:06] so what happens is [14:08] uh we allocate a local buffer on the [14:10] lower left hand side to to store the [14:12] content when it comes in from disk um [14:15] now when we go to encrypt it we choose [14:17] that same uh local cpu [14:20] once we've once we write it back we've [14:22] allocated a crypto buffer that's local [14:24] and then we send it [14:26] and we write it back there and then we [14:28] send it back on the network card that it [14:30] came in on [14:31] and boom uh we've only crossed the numa [14:33] bus once rather than four times [14:37] so basically [14:40] the worst case is we cross it once uh we [14:42] only put 50 gigabytes rather than 200 [14:45] gigabytes and all that's good that helps [14:46] a lot [14:48] the real problem though is that uh real [14:50] life is messy i remember i said those [14:52] knicks were two by one hundred [14:54] so that means that [14:56] you can only put [14:58] two nicks in newman domains because [15:01] you have two new domains that don't have [15:03] nyx because [15:04] there's no need for them because we have [15:06] we already have 400 gig links and nicks [15:08] are expensive and we don't want to [15:09] replicate them all around [15:12] so right now i have uh hacks to sort of [15:14] pretend that a nic is the one of the [15:17] ports is in a different domain um and [15:19] this impacts the the worse in the [15:21] average cases and the following way so [15:23] basically that nick uh really isn't [15:26] isn't down on the left-hand side anymore [15:27] actually it's like up on the upper [15:29] left-hand side [15:30] and so rather than uh going right out [15:32] that same pneuma note at the at the end [15:34] you actually have to go across uh across [15:36] pneuma again [15:38] and that impacts uh [15:40] the worst case by now instead of putting [15:42] 50 gigs on on the pneumobus we're [15:44] putting 100 gigs [15:46] now if you if you look at the average [15:48] case it gets a little bit better so if [15:49] you if you think about it [15:51] if you have four nodes um then there's [15:53] basically a one in four chance that the [15:55] content is going to be local so we're a [15:57] three and four chance that you're gonna [15:59] it's not gonna be local so you have a 75 [16:01] percent chance that you're going to [16:02] cross the new bus there to find content [16:03] that's not local [16:05] and then since we have uh two domains [16:07] with nix and two domains without you [16:09] have a fifty percent chance you're gonna [16:10] need to cross the numerabus to go out on [16:12] on on a nick that actually exists [16:15] so that leads us to about sixty two and [16:17] a half gigabytes a second of data on the [16:18] nema fabric which is [16:20] still a whole lot better than than uh [16:23] than the the uh the numbers we the 200 [16:25] gigabytes that we would get if we [16:26] weren't being smart [16:29] so did it help yeah it helped a little [16:31] um it was no it was no panacea but it [16:34] gets us from about 240 to about 280 [16:36] which is you know [16:38] better than nothing but still you know [16:40] not anywhere near our goals that might [16:43] uh you know make it worth letting up a [16:44] third link but definitely not lighting [16:46] up all four links [16:47] so can we do better [16:49] and [16:50] when i'm asking that question basically [16:52] i'm looking at at nick ktls [16:56] so [16:57] if you remember that diagram back at the [16:59] beginning of the presentation [17:01] these these vertical green lines have [17:04] always [17:04] bugged the hell out of me so [17:07] if you can see my mouse moving around [17:08] these lines here [17:10] and here where essentially the cpu has [17:12] to read the uh the data [17:14] out of it has to read the plaintext data [17:16] out of uh memory encrypt it and then [17:18] push it back and it doesn't even need to [17:20] be the cpu it could be a qat [17:23] or a chelseo ccr look aside card [17:26] whatever it is it's got to read the data [17:28] out of memory and write it back and [17:30] that's the heart that's the the part we [17:31] want to get rid of so if we could get [17:33] rid of these this extra kind of detour [17:35] through the cpu or through an [17:36] accelerator card we could essentially [17:38] cut our memory down with requirements in [17:40] half [17:41] and things kind of drop down to the case [17:44] where it's almost like things are [17:45] unencrypted data flows in from the disks [17:48] into memory and out and out the next and [17:50] the cpu just kind of just stays [17:52] uninvolved [17:54] so let's talk about this magic thing [17:57] what is nick ktls so basically it's what [17:59] the rest of the world calls hardware [18:00] inline tls [18:02] um so when normally in kt last [18:04] session's established in user space [18:06] that's still the same [18:07] but now when cryptos move to the kernel [18:09] the colonel kind of turns around and you [18:11] know just does a hot potato and hands [18:13] the crypto off to the nick [18:17] and so the [18:18] the tls records are encrypted by the [18:20] nick as the data flows through it on [18:22] transmit so the positive thing is [18:25] there's no more detour through the cpu [18:26] for crypto and we uh [18:29] and we've reduced our memory bandwidth [18:31] requirements the negative thing for some [18:33] people is that uh all of a sudden your [18:36] data which used to be encrypted in user [18:38] space and then and and then so it was [18:40] always encrypted in the kernel now with [18:41] ktls it's in the clear in the kernel and [18:43] now that moves that even further away so [18:46] it's in the clear on the other going [18:47] across pci express and you know i [18:49] realize some people may have problems [18:50] with that uh for this workload i don't [18:53] but it's just to put it out there [18:56] so the nicks that we're using for this [18:58] are the melon ox connect x6 dx [19:01] they offload tls 1.2 and 1.3 for aes gcm [19:05] that's by far our most uh popular cipher [19:07] and one of the most popular snipers on [19:09] the web [19:10] so [19:11] the thing to re remember about these [19:13] nicks is that they retain cryptostate [19:15] within a tls record [19:17] and that means that [19:19] tcp can send the first couple segments [19:22] off of a tls record and [19:26] stop for a while because it's waiting [19:28] for a knack and then yeah a couple [19:30] milliseconds a couple seconds whenever [19:32] the act later the app can come in and [19:34] then once the app comes in [19:36] tcp can pick right back up in the middle [19:38] of that tls record and then nick will [19:39] have remembered the crypto state and [19:41] things will just keep on going as normal [19:44] the bad case is if a packet is sent out [19:46] of order you know a tcp retransmission [19:48] because at that point the nick no longer [19:50] has the state from some arbitrary point [19:52] in the tcp stream and so it needs to do [19:55] some extra work to recover that state [19:58] so i'm going to go through a little [19:59] diagram which kind of describes [20:02] how [20:04] transmits work with the connectx 60x for [20:06] tls [20:07] all right so this this eye chart here [20:09] essentially what we've got going on is [20:12] in the [20:12] that uh upper rectangle [20:15] is a representation of host memory with [20:19] a tls record unencrypted sitting in the [20:21] socket buffer the numbers you see are [20:24] essentially tcp uh segment boundaries [20:27] so you notice it starts at zero 1448 [20:30] 2896 and so on um [20:32] and then you see the pci express bus uh [20:35] representation of the nick and then a [20:36] hundred gig network [20:38] so [20:39] when uh we go to send something [20:42] basically the nick will the [20:45] tcp will say hey i want to send the [20:46] first you know four or five six segments [20:49] of the packet [20:50] and then the nick will go and dma them [20:52] uh it may do something smarter than this [20:54] i don't know how their nick works it may [20:56] you know dma for 4k at a time it may dma [20:59] the individual segments i have no idea [21:02] it's all hidden from us but the nick [21:03] dmas [21:05] dmas the plain text data down [21:08] it encrypts it [21:09] in the nick [21:10] and then it sends it out on the wire and [21:13] obviously it sends it as packets not [21:14] just a big burst because you know 1500 [21:16] divided to you but [21:17] uh you know my my artistic abilities are [21:19] limited so [21:20] that's what this diagram looks like [21:22] um now tcp comes along later and says [21:26] hey i want to send the next five [21:27] segments from this from this packet and [21:30] so then the nick uh you know as you [21:32] would expect the nick dmas them down [21:34] it encrypts them and it sends them on [21:36] the water [21:37] and then time passes we get some more [21:39] acts more more windows spaces freed up [21:41] tcp says hey i want to send that last uh [21:45] segment and so [21:48] the um [21:50] the dmas it down encrypts it and sends [21:52] it on the wire [21:53] that's all well and good and you can see [21:55] that the nick doesn't have to do any [21:56] extra work when he leaves off in the [21:58] middle [21:59] uh it's just all straightforward he [22:01] could the tcp could do the whole segment [22:03] or tcp just do a part of a segment it [22:05] doesn't matter [22:06] but for a re-transmit [22:08] let's say that uh you know your your [22:10] cell phone was flaky and you lost that [22:13] one chunk out of the out of the video [22:14] stream [22:15] and you send a sack for it and tcp needs [22:17] to resend that uh that that segment at [22:21] 14k [22:22] so what you would expect is you'd expect [22:25] uh to see you expect to see something [22:27] like like this where the nick just [22:29] dmased down encrypts it and sends it in [22:31] the wire [22:32] but no [22:33] what has to happen is because the nic no [22:35] longer has the state for this arbitrary [22:37] position in the tcp stream what has to [22:40] happen is the nick has to dma [22:42] the entire tls record from the ver all [22:45] the way down uh to [22:47] the neck [22:48] encrypts it has to encrypt all of that [22:50] up to the point of the uh [22:53] of the current uh segment in order to [22:54] get the crypto state once it gets the [22:56] crypto state it can encrypt that one [22:58] segment we want to send [23:00] and it can send it on the wire [23:05] so that's a little bit painful and that [23:07] means that we have to watch out for for [23:09] tcp re-transmissions because [23:11] they could easily run us out of pci [23:13] express bandwidth i'm going to talk a [23:14] little bit more about that [23:17] so [23:18] first i'm going to talk about some [23:19] results i mean this is basically a talk [23:21] about war stories there's no there's no [23:23] no no new tales to tell here [23:26] so basically we got a a very early beta [23:28] firmware from melanox uh so i think it [23:30] was like it was way it was before the [23:31] pandemic [23:33] and we had pretty good peak bandwidth we [23:36] got about 250 gigabits a second total [23:38] out of the system [23:39] um [23:41] the problem is that [23:42] the sustained bandwidth was was pretty [23:44] terrible so what happened was basically [23:48] the way our systems work is that [23:50] we [23:51] attract load via uh this thing we call [23:53] health which runs off of a pid [23:55] controller with various inputs [23:57] a lot of the a lot of the inputs are [23:58] things like you know is the network card [24:00] close to capacity is the cpu saturated [24:03] are the disk saturated and whatnot [24:06] um so what happened was as we kept [24:08] adding more and more sessions uh the cpu [24:10] was very low network card was nowhere [24:12] near its limits um and so we said we're [24:14] healthy give me more traffic give me [24:16] more traffic [24:17] uh and [24:18] as more and more connections arrived the [24:20] nick slowed down more and more [24:22] and so we wound up at a floor of about [24:25] 75 gigs in nick or 150 gigs per system [24:28] and we just kept attracting more and [24:29] more and more clients and it sort of [24:31] turned into a tar pit [24:33] because this kind of limit is invisible [24:35] to our pid controller [24:37] so we really didn't like this [24:39] because of that for that reason and it [24:41] wasn't not a big enough advantage for us [24:43] to use to move forward with [24:45] with inline tls [24:47] now [24:48] what exactly was happening so [24:50] all of this is my supposition it's not [24:53] anything that i know from like talking [24:55] to mellox or nda stuff or anything like [24:57] that but basically we know that the nick [24:59] stores tls state procession we know that [25:01] because it can resume in the middle of a [25:02] tls record [25:04] and we know that we at netflix have a [25:06] ginormous number of tls sessions active [25:09] each client has multiple video [25:11] connections anywhere between one one [25:14] typically two to four to six [25:16] buggy buggy clients have had [25:18] tens or hundreds [25:20] uh we have um [25:22] one audio and one subtitle connection so [25:26] by a rule of thumb we kind of estimate [25:27] when we need 400 000 sessions for 400 [25:29] gigabits [25:30] and we noticed that the performance gets [25:32] worse the more sessions we had [25:34] and my supposition is that there's a [25:36] limited amount of memory on the nic [25:37] and that i know from reading the [25:39] melanops driver that it will allocate [25:41] host memory um and [25:45] on behalf of the nick and let the nick [25:47] do whatever it wants with that memory so [25:49] my supposition is that uh what's [25:51] happening is that [25:52] the memory on the neck uh is exceeded [25:55] once we have too many sessions and it's [25:56] just paging in and out of the host [25:58] and things just get really bad once that [26:00] memory is thrashed [26:02] don't know that for sure never got [26:03] confirmation from melanox on that but [26:05] that's what that's what i suspect [26:08] so amd suggested we enable pci relaxed [26:10] ordering um [26:12] and what pci relaxed ordering does is [26:15] basically allows uh things to pass each [26:17] other [26:18] so [26:19] that the theory is that would help with [26:21] with paging in this tls connection state [26:24] so we enabled it um it didn't help [26:26] but later we found out that uh the [26:28] connect x6 firmware even though you [26:30] could toggle that bit in pc uh config [26:32] space [26:33] it didn't actually wasn't actually [26:34] connected anything and nothing actually [26:36] happened because it was hardcoded off [26:38] time passes and we get a new firmware [26:41] and this one's much better [26:43] this one actually enabled relaxed [26:45] ordering and they may have done some [26:46] other things i mean i don't have access [26:48] to their firmware but all i know is this [26:50] one we got 160 gigabits per second per [26:53] nick or 320 gigabits a second total [26:56] uh and at this point we were much [26:58] happier because the peak and the [26:59] sustained were essentially uh the same [27:03] the peak was a little bit higher but [27:04] basically [27:05] our machine wasn't turning it into a [27:07] tarpon [27:08] this is a was a new record for us 320 [27:11] gigs is a lot better than the 200 and [27:12] some gigs we're getting from software [27:14] tls [27:15] and on a per and nick basis it's nearly [27:17] as fast as software tls uh you know 160 [27:20] versus 190 gigabits per nick [27:22] and why i'm talking about per nick i [27:24] mean [27:24] like on one of these machines if you [27:26] just disable one of the knicks and you [27:29] know run it it would software tls you [27:31] can see that the nick can push 190 [27:32] gigabits i mean you can't get 190 [27:34] gigabits from two because the memory [27:36] bandwidth limits [27:38] so anyway um [27:40] a few months later you know we thought [27:42] that was about as good it was going to [27:43] get a few months later uh they surprised [27:45] us with the new uh production firmware [27:47] which had a knob [27:48] called tls optimized i still don't know [27:51] what that does except it makes things [27:52] much better [27:54] and now we get 190 gigabits per second [27:56] in nick or 380 gigabits a second total [28:00] so that's a pretty good number [28:03] let's let's start using it well [28:05] wait a second we're netflix we can't [28:07] actually start using things yet we have [28:09] to do qoe testing [28:11] and qlea is basically the quality of [28:13] experience for the customer so we [28:15] measure things like [28:16] like the rebuffer rate you know like how [28:18] long the like how many times you get [28:19] that little spinny when you're watching [28:20] a netflix [28:21] uh yeah when you're watching a netflix [28:23] video [28:24] how long it takes between pressing play [28:26] and you seeing them you see the video [28:28] show up how long it takes between when [28:30] you when you start playing and when the [28:32] video looks decent [28:34] um or excellent really and [28:37] the initial results from this nick were [28:39] good now that's the contrast we tried to [28:41] nick in the past which did not have this [28:43] feature of being able to pick up in the [28:45] middle of a tls session [28:47] that nick uh [28:49] in order to make it work we had to well [28:51] really randall stewart [28:53] had to essentially hack the [28:55] tcp [28:57] code to become aware of tls record [28:59] boundaries and to try super super hard [29:02] to only send uh [29:04] data around tls record boundaries and [29:06] that that caused really miserable qoe [29:08] and so we couldn't move forward with [29:09] this other nick [29:10] because this nic can can resume where it [29:13] left off [29:14] the qe is fine [29:16] so we did an initial study everything [29:18] went fine now that we have more of these [29:19] deployed we have more experience we're [29:21] going to do a larger more complete study [29:22] before we enable it [29:25] the other thing we need to do before we [29:26] enable it is to come up with a way to [29:29] defend against tcp retransmits so if you [29:31] remember [29:32] those slides uh [29:35] way way way way back when [29:37] where i was talking about what happens [29:39] when we do a re-transmit [29:42] so [29:43] what happens here is [29:45] you wind up with [29:46] you know essentially almost like an [29:48] amplification attack [29:49] so whether it's intentional or not if [29:51] you have a lossy tcp connection it can [29:53] run you out of pci express bandwidth [29:54] fast so you want to make sure that this [29:57] doesn't happen uh you know [29:59] this almost never happens [30:01] so the way we do that is [30:06] oops the way we do that is um to monitor [30:10] the bytes retransmitted uh on lossy [30:12] networks and to [30:14] move connections from hardware back into [30:16] software once uh [30:18] once they exceed a certain threshold of [30:20] retransmits and we also monitor uh [30:24] segments uh that are retransmitted in [30:26] case some hacker gets clever and decides [30:28] well i'm not going to ask for very many [30:29] bytes retransmitted but i might ask for [30:30] a lot of segments and that's still going [30:32] to make the neck re-dna all that stuff [30:34] and in that case we'll probably kill the [30:35] connect we kill the connections rather [30:37] than moving them to software since it's [30:38] pretty clear it's an attack [30:41] so [30:42] one interesting note here is that the [30:44] mixed hardware and software uh [30:46] performance is interesting so i've [30:48] noticed that if i move um [30:50] basically [30:52] and i basically set the threshold to one [30:54] percent meaning that [30:56] if a tcp connection has more than one [30:57] percent retransmitted bytes he gets [30:59] moved to software [31:01] then that moves roughly 25 to 33 percent [31:04] of connections to software [31:06] and that moves our max stable bandwidth [31:08] down from 380 to 350 gigabits a second [31:11] uh and that's a larger impact than i [31:13] would expect [31:15] and that's something that i need to look [31:17] into because [31:18] just a third of the connections really [31:20] shouldn't be putting that much stress on [31:22] the system and remember memory bandwidth [31:23] looks fine the [31:25] i the the latency the [31:27] dma latency looks fine i just don't [31:29] understand why it's as bad as it is [31:33] all right so i'm going to [31:34] now we're going to talk about well what [31:35] happens when we combine these two [31:37] techniques uh you know when we combine [31:39] uh colonel tls and numa this is a sort [31:42] of the you put your chocolate in my [31:44] peanut butter kind of moment if you [31:45] remember those ads from the 80s [31:47] so now this is the same diagram as we [31:49] had before [31:50] except things get a lot simpler so the [31:52] tls connection comes in on the lower [31:54] left hand side [31:55] remember [31:56] again the the [31:58] data lives up in the upper right-hand [32:00] side and we bring it to the lower [32:01] left-hand side [32:03] so now instead of doing any crypto or [32:04] anything boom we just send it out on the [32:06] neck [32:07] so everything's well and good um [32:10] much simpler [32:12] and now in real life if you remember we [32:14] don't have nicks on every node so in [32:16] this case we might need to send it out [32:17] in the nick on the upper left-hand side [32:20] um essentially the worst case is [32:22] essentially the worst case that we had [32:23] before and nothing changes [32:26] uh and the average case is the same [32:28] average case in terms of numeric [32:29] crossings as we had before nothing [32:31] changes [32:32] the only thing really that changes is [32:33] that detour through memory is just [32:35] completely avoided and we save some [32:37] memory bandwidth [32:39] so here's a big eye chart for you of all [32:41] the performance numbers that i've talked [32:43] about so far first we start off with uh [32:46] software tls um [32:48] where we get about 240 gigabits a second [32:51] and we're at about 80 cpu uh one you [32:54] know when the memory system saturates [32:56] and then with software tls and pneuma we [32:58] get to about uh [33:00] 280 again at 80 cpu [33:03] now if we run the amd's in [33:05] flat mode just with one node per socket [33:09] we wind up with [33:11] an and we use uh [33:13] hardware kernel tls we wind up with 380 [33:15] gigs uh at i don't know about 60ish [33:19] percent cpu [33:22] and pneuma [33:25] with uh hardware tls [33:27] doesn't change the bandwidth at all it's [33:28] still about 380 but it drops the cpu to [33:30] just a hair over 50 percent [33:32] um [33:34] and then if we go go back to the the you [33:38] know the configuration that we're [33:39] actually probably going to deploy [33:41] essentially we have a hardware tls and a [33:44] flat topology [33:46] with uh some safety some safety belts to [33:48] keep lossy connections from coming [33:49] things up [33:50] we wind up with about uh [33:53] you know in this in this upper 60 lower [33:55] 70 cpu [33:57] at uh [33:58] about 350 gigs [34:01] so [34:02] in the what do i have left i've got [34:04] about uh [34:05] 10 minutes left bef [34:07] in [34:08] in the end here i'm going to talk about [34:09] some alternate platforms we looked at [34:12] the one of the interesting alternate [34:13] platforms we looked at was uh the ampere [34:16] uh [34:17] ultra which is a [34:19] a three gigahertz 80 cpu arm neoverse [34:23] and if you kind of squint it looks very [34:25] similar to the amd it's got the same [34:27] number of memory channels same size [34:28] memory same number pcie lanes roughly [34:31] the same storage and the same networking [34:34] so really you know the only important [34:35] thing that gets swapped out here is the [34:37] cpu which is a completely different [34:38] architecture [34:40] and because it's very different [34:42] architecture [34:43] at least on freebsd i don't currently [34:46] have a way to do all the fun stuff i do [34:48] on intel and amd like looking at [34:50] memory bandwidth being used and running [34:52] fancy profilers and looking at io [34:55] bandwidth being used [34:56] and so it kind of makes me feel like i'm [34:58] kind of driving blind [35:01] so when we uh initially [35:04] started playing with it we got some [35:05] really poor performance puts up our [35:07] kernel tls i think we got like 120 or [35:09] 140 gigabits a second [35:11] mark johnson uh came up with an idea [35:13] which is brilliant and which i kick [35:15] myself for not thinking of but all [35:16] brilliant ideas are obvious in hindsight [35:18] and he came up with the idea of using a [35:21] uma [35:23] cash zone to cash 16k contiguous [35:26] tls crypto destination buffers and once [35:29] we did that we got up to about 180 [35:31] gigabits a second [35:32] um [35:34] so [35:35] that's pretty good still not amd level [35:37] but it's pretty good um and then we [35:40] tried nik tls the hardware tls [35:43] and we found that we were pcie limited [35:44] at about 240 gigs um the cpu utilization [35:48] was shockingly low it was like you know [35:50] in the 20 range the next but the nix [35:52] were saturated and started uh dropping [35:54] on output [35:57] when i was looking at this i noticed [35:58] that [35:59] one difference between the amd systems [36:01] and the ampere systems [36:02] is that extended tags were not [36:05] were not enabled [36:07] so [36:10] i'm not sure who's responsible for [36:11] enabling them but on ampere with with [36:14] freebsd they just didn't get enabled [36:16] so [36:18] basically the importance of extended [36:19] tags is psi express is almost more like [36:21] a network than a bus [36:23] and just like in tcp when you increase [36:25] the window size if you can increase the [36:27] pci express tag space you can have more [36:29] sort of more balls in the air at the [36:31] same time which uh gives you a bigger [36:34] pipeline and leads to bigger bet leads [36:35] to bigger bandwidth by default the pci [36:38] express tag space is just 5 bits or 32 [36:40] tags [36:41] if you enable extended tags then you get [36:43] uh [36:44] three [36:45] you get 8 bits and 256 tags and once [36:48] once we did that [36:50] we got the band up to 320 gigs a second [36:54] still not amd performance but i don't [36:56] have as much experience on this platform [36:58] i don't have the tools that i have [36:59] everywhere else [37:00] it's possible that we could do better i [37:02] mean i don't want to sell ampere short [37:04] but it's this is where we are right now [37:07] and the last platform i want to talk [37:09] about is a platform i literally uh got [37:12] into a data center like last week [37:14] sometime [37:15] um and it's the uh intel iceland xeon it [37:18] took us forever to get this uh working [37:21] because [37:22] of storage basically it's only [37:25] the difference between uh this system [37:26] you know besides the cpu is that it's [37:28] only got 64 lanes of gen 4 which means [37:31] we can't use you know an insane number [37:33] of gen 3 drives we have to use gen 4 [37:35] drives [37:36] or we have to use the pci express switch [37:39] and the switches that we were using had [37:41] weird problems so [37:43] you know real life messy details this is [37:45] the first time we were able to build [37:46] something close to a full speed system [37:49] so it has 20 uh kioxia four terabyte [37:52] ndme drives and again the same two uh [37:54] melanox nics one important difference [37:57] here is that we have eight channels of [37:59] uh [38:01] ddr3200 but [38:02] this skew of the intel xeon runs them at [38:04] 29.33 [38:06] i personally hate that i wish intel ran [38:10] all their memory at the same speed like [38:11] everybody else does across across their [38:13] product lines but [38:14] it is what it is [38:16] so the intel results are limited by uh [38:19] by memory bandwidth at about 230 [38:21] gigabits a second it makes sense because [38:23] the memory is running at a slower speed [38:24] than on the amd and so the performance [38:26] is basically you would expect the [38:28] performance to be the same as amd uh [38:31] flat mode [38:32] if you if you had the memory running at [38:34] the same speed [38:37] so [38:39] basically [38:41] i wanted to try uh [38:43] hardware tls but i'm running the bios [38:46] from hal uh and we've which is which [38:49] locks out everything i got a fix on like [38:52] friday uh to to enable uh relaxed [38:54] ordering to see if we could get some [38:56] better performance from uh [38:58] from [38:59] ktls and unfortunately the fix didn't [39:02] work so i don't have any results that [39:03] i'm comfortable with presenting [39:05] um [39:07] but here's a little summary of where we [39:09] are with the alternate platforms the [39:11] software tls bandwidth so on amd we have [39:14] about 280. [39:16] um on amp here we have about 180 and on [39:19] intel we have about 230. [39:22] and for the max hardware tls bandwidth [39:24] on amd we have about 380 and an amp here [39:27] we have about 320. [39:30] and i was going to [39:32] end this with something really [39:33] impressive because we built an 800 gig [39:35] machine unfortunately we had some storms [39:37] in the usa [39:38] which delayed shipping which meant that [39:41] the machine didn't get to the data [39:42] center in time to get racked [39:44] so maybe i'll be back next year to talk [39:46] about 800 gigs or something along those [39:49] lines [39:50] in the uh [39:52] in the five minutes left [39:54] well first i want to thank everybody who [39:55] helped especially the netflix side [39:57] warren haroop uh he's one of the people [39:59] that works on new hardware netflix and [40:01] is in charge of building a lot of his [40:02] prototypes [40:03] and so he's like my santa claus and [40:05] brings me all my new toys um [40:07] and i also want to thank the freebsd the [40:10] freebsd developers who have worked with [40:12] over the years on these things [40:13] especially uh mark johnson for all of [40:16] his work on both pneuma [40:18] and the vm system [40:20] and ktls uh john baldwin for doing the [40:23] bulk of the hard work of upstreaming uh [40:26] ktls for us and working with different [40:29] vendors to to hammer out a hard work [40:31] ktls interface [40:32] and also to [40:35] jeff roberson for all of his hard work [40:37] on pneuma [40:38] and i'm sure i'm forgetting people and [40:39] that's [40:40] i don't intend to slight anybody but i [40:42] just love working in the community and [40:44] every everything that uh everything that [40:47] we've done is a collaboration [40:49] i have my slides are up uh after the [40:52] talk and so now i have now i have some [40:54] time for questions i think i have like [40:56] four minutes [40:58] so just start just start reading off the [41:00] uh [41:02] i'll start reading off the panel here if [41:03] i can get my mouse over there [41:06] if you switch to the shared notes tab at [41:08] the top we've organized the questions [41:10] for you oh okay let's see outfit [41:14] it's bubbling up there we go [41:16] all right is nick is nick ktls available [41:18] in current yes uh it is it's available [41:21] in current i actually think it's [41:22] available in 13. um [41:26] but it only works with certain nicks in [41:28] freebsd i believe the only nix it'll [41:29] work with is the connectx 60x and the [41:32] chelsea ot6 [41:35] um [41:36] how fast is nfs [41:37] over tls i don't know that's something [41:40] we'd have to to talk to rick about [41:43] are we using plx chips to connect gen 3 [41:45] to gen 4. [41:47] um we were using um [41:50] broadcom i think broadcom did broadcom [41:52] by plx i don't remember um [41:56] we were having some interesting problems [41:57] with broadcom pci switches where [42:01] they would run at seemingly gen 3 [42:03] bandwidths uh [42:04] if you were if you put like one or two i [42:07] think it was like a [42:08] gen four by eight to four [42:11] gen three by fours and if you put two [42:14] gen three 3x4 is active [42:16] everything was good but if you got over [42:18] if you got over [42:19] two then things were running at gen 3 [42:22] speeds that's why we that's why we had [42:24] all kinds of problems like you know [42:25] building a machine with just 64 pcie [42:27] lanes [42:30] would faster ddr memory available to [42:33] client skus help to bump fabric i'm not [42:36] sure what you mean clients oh do you [42:38] mean like the i mean like gamer machines [42:41] um [42:43] yeah in terms of fabric bandwidth i [42:45] think the infinity fabric is limited but [42:47] it would definitely help with memory [42:48] bandwidth i've i've often like [42:50] i used to overclock memory uh in servers [42:52] a while ago [42:55] um no we didn't know we don't use total [42:57] tcp offloading so we have a essentially [43:00] a whole uh group at netflix that's [43:02] parallel to our group which focuses [43:04] solely on protocol improvements that's [43:06] people like randall stewart uh [43:08] and lawrence stewart no relation [43:10] that do a lot of the tcp work and we one [43:13] of the things that we depend on is to uh [43:14] to control [43:16] every aspect of the tcp stack and that's [43:18] what allows us to give such good qoe to [43:20] our customers [43:22] so how many how many flows can the mix [43:24] handle what happens on overflow [43:26] i kind of talked about that a little bit [43:29] i believe the limit is [43:32] you know i don't even know what the [43:34] limit is i thought it was a hundred and [43:35] twenty eight thousand per port but i'm [43:38] not sure and what happens on overflow is [43:41] that the hardware [43:43] hard the hardware tls session is just [43:44] not set up and it falls back to software [43:49] is the data encrypted in storage too [43:51] no it's i mean that our [43:53] assets are are drm encrypted which is [43:56] between uh you know somewhere some [43:58] encoding in the cloud in your set-top [44:00] box or your phone or whatever but [44:02] there's no extra layer of encryption at [44:04] rest [44:06] do you perform traffic steering to the [44:08] optimal optimal nick uh no we don't we [44:11] we we live with the uh the decision that [44:14] our our link partner uh [44:16] gave us now there somebody grabbed me [44:19] after my last a couple years ago to my [44:21] last talk and said there is a protocol [44:22] to tell the lacp to redirect things but [44:24] we don't have that implemented um and so [44:27] we just um whoops i have all the [44:29] questions just appeared oh there we go [44:30] so we just use the uh whatever lacp [44:33] gives us [44:37] how do we measure percent cpu um [44:39] basically i just use the same numbers [44:41] that as you would see with vmstat [44:44] um [44:45] in terms of front end stalls [44:47] on and actually i don't use vmstat i [44:50] have a tool that i wrote it's in ports [44:51] it's called nstat it shows you various [44:54] things like uh memory bandwidth sorry [44:57] it'll show you memory bandwidth if you [44:58] look at the intel pcm it shows you cpu [45:01] use it shows you [45:02] um number tcp connections it shows you [45:05] network bandwidth it shows your context [45:07] switches interrupts all that kind of [45:08] stuff all in all in one all in one [45:11] one line [45:13] um [45:15] i don't have any thoughts about the [45:17] xilinx uh smartnix i am not aware of [45:20] them um [45:22] typically we try to go for xilinx think [45:25] makes me think fpga and typically we try [45:27] to go for asics just for [45:29] uh power reasons we have we were [45:32] engaging with a nick vendor that was [45:34] building a uh a different kind of tls [45:37] offload nick where they [45:38] had something like 40 or 64 gigs of [45:41] memory and an fpga on the nic [45:43] and that that changed the tcp retransmit [45:45] story because they just were able to [45:46] cache everything in their nic and that [45:48] was nice except the nick burned as much [45:51] i think the nick burned as much power as [45:52] the rest of the system so it was just [45:54] not a really good fit for us [45:59] uh and we're always willing to we're [46:00] always willing to try nicks we've [46:02] engaged with other vendors i think that [46:04] um i actually think the freebsd is ahead [46:06] of the curve on nick tls i know that [46:08] linux uses it um [46:10] but [46:12] i think we we are a little bit ahead of [46:14] the curve because [46:15] one of the things that's that i meant to [46:17] mention is that when um i was first [46:19] working on kernel tls i had this idea of [46:20] nick tls in my head [46:22] and i [46:24] kind of kept that in mind when designing [46:26] the way m-buffs work with with with [46:29] kernel tls and the nice thing about [46:30] freebsd the way embossed work is that a [46:33] single m-buff describes an entire tls [46:35] record so it makes these re-transmits [46:37] like really easy for the driver because [46:38] the driver [46:39] the driver can just look around in the [46:41] same mbuff and it has all of what it [46:43] needs to do a retransmit [46:45] um let's see [46:47] this question is being written [46:48] oh okay that's just the answer is being [46:50] written [46:51] um [46:53] i guess that concludes my talk if [46:54] there's no more questions [46:58] thank you very much [47:00] all right thank you [47:40] you