Cloudflare Rearchitects Data Pipelines to Counter Exponential Traffic Growth

Cloudflare Rearchitects Data Pipelines to Counter Exponential Traffic Growth To keep hardware expansion sustainable while processing roughly two billion events per second, Cloudflare's Data team implemented a series of pipeline optimizations across schemas, compression, replication, sampling, and aggregations, deferring roughly two and a half years of hardware expansion. Dismantling monolithic message schemas in 2022 reduced backbone bandwidth usage by 35% to 40%, and switching compression from LZ4 to zstd in 2024 cut compressed traffic throughput by nearly 40% with minimal CPU impact. Decoupling wide-area disaster resilience from database redundancy in spring 2026 replaced edge double-writing with downstream replication, yielding a net wide-area reduction of 36 GB/s. Decentralizing square-root sampling to the edge collector Logfwdr halved compressed data on top datasets, saving 25 GB/s without degrading data quality for rare keys. Introducing multi-stage ingest rollups for aggregated datasets achieved an 80% wire reduction on HTTP traffic, contributing to a combined 2026 reduction of approximately 70 GB/s across the backbone. Historical Optimizations: Monolithic Schemas and Zstd Compression In 2022, Cloudflare dismantled its single monolithic message schema, which previously required internal consumers to ingest all product fields and filter them downstream. Replacing the shared firehose with product-specific schemas duplicated certain common fields but eliminated irrelevant payloads, cutting backbone bandwidth by 35% to 40%. By 2024, persistent network congestion prompted a migration from LZ4 to zstd compression. Fleet-wide rollout revealed negligible CPU impact while achieving nearly a 40% reduction in compressed throughput, effectively allowing 1.5 times the traffic over the existing pipe capacity. Decoupling Transit Ingest from Analytics Replication Disaster resilience historically required edge collectors (Logfwdr) to transmit two full copies of every non-regional analytics event across wide-area networks to separate centralized regions. In spring 2026, Cloudflare adopted a shared ingest pool where Logfwdr routes an event only once across wide-area transit. Replication was deferred downstream: data is sampled and buffered in Buftee before cross-data-center inserters replicate to secondary regions. Downstream sampling and uniform batch partitioning resulted in eliminating 43 GB/s of transit while adding only 7 GB/s of cross-facility replication, producing a net savings of 36 GB/s. Decentralized Square-Root Sampling at the Edge In summer 2026, Cloudflare expanded sampling by running it directly within Logfwdr before data reaches centralized receivers. The mechanism shuffles records by dataset key to remove bias and retains a sample size proportional to the square root of the group count. Double-sampling at the edge and centralized receivers attenuates high-frequency keys while preserving low-frequency events, causing high-volume keys to scale at roughly x^0.319 instead of x^0.485. Applied to three top datasets, this halved wire traffic to receivers, saving 25 GB/s (over 2 PB per day). Multi-Stage Aggregation and Net Resource Impact For rollup datasets that prioritize exact counts over broad explorative querying, Cloudflare introduced ingest-time aggregation across each stage of the pipeline: within Logfwdr, inside centralized receivers, at ClickHouse inserters, and via ClickHouse aggregating engines. Deployed to HTTP traffic, this reduced transit from over 9 GB/s to under 2 GB/s, an 80% decrease. Across 2026, load balancing, sampling, and aggregations removed roughly 70 GB/s of compressed traffic from the backbone. In Ready-Analytics alone, these changes reduced receiver CPU and memory usage by approximately 50%, saving 750 CPUs and 3.5 TiB of RAM.

对抗指数级流量增长：Cloudflare 数据管道的降本与架构演进 面对每秒约20亿事件的指数级流量增长与硬件线性增长的矛盾，Cloudflare 数据团队在过去五年通过拆分大 Schema、升级压缩算法、重构管道多活复制、去中心化采样以及早期聚合计算等策略，成功将骨干网流量与硬件资源压力大幅缩减，相当于推迟了两年半的硬件扩容需求。 2022年通过将单体 Schema 拆分为各产品专用 Schema，骨干网带宽占用减少了35%至40%。 2024年全量将压缩算法由 LZ4 切换为 zstd，在 CPU 几乎不受影响的情况下降低了近40%的压缩流量吞吐。 2026年重构跨区灾备复制架构，将广域网双写改为共享接入池与下游采样后延迟复制，净削减36 GB/s广域网流量。 将采样下沉至边缘服务 Logfwdr（双重采样），高频事件数据大幅衰减，在三大核心数据集上节约超 25 GB/s（超 2 PB/天）。 在管道全链路引入分布式分层聚合（Ready-Aggregates），使高基数 HTTP 数据集的传输流量骤降约80%。 单体 Schema 拆分与压缩算法演进（2022–2024） 在早期架构中，所有产品字段打包在单条数据流中，下游各系统必须摄取全量数据再过滤。2022年，团队针对 HTTP、安全和 CDN 等各业务定义独立 Schema，尽管产生了部分字段复制，但有效消除了内部无效数据流，骨干网带宽削减了35%至40%。到2024年，面对日益严峻的网络拥塞，团队将压缩算法由 LZ4 改为 zstd；在经过小规模试验证明 CPU 损耗极低后完成全量铺开，直接降低近40%的压缩吞吐，使同等带宽能够承载超1.5倍流量。 管道跨区域复制架构解耦（2026春） 为满足分析服务的区域容灾需求，过去系统在边缘节点（Logfwdr）向多个中心区域进行双写，导致广域网传输量直接翻倍。2026年春，团队借鉴 Logpush 的共享接入池模式，将传输通道与数据库冗余解耦：Logfwdr 仅向可用中心接收节点单写一次，随后在接收端采样并写入 Buftee 缓存层后，再由远程写入器进行跨数据中心延迟复制。由于复制发生在采样之后，且中心端统一分批的数据获得了更高的压缩率，整体实现削减43 GB/s传输流量仅增加约7 GB/s跨机房复制，净节约36 GB/s。 边缘下沉与双重采样机制（2026夏） 为在保障分析质量的前提下进一步减少传输字节，团队在保留中心接收端采样的同时，将采样能力提前部署到 Logfwdr 边缘服务中。针对特定键的数据集，系统通过打乱记录并按开方比例取样；高频出现的“热门”键采样率大幅衰减（增长率由一次采样的 x^0.485 降至双重采样的 x^0.319），而低频“长尾”事件的细节则完整保留。这一策略在率先落地的三大核心数据集中均使网络传输量减半，合计削减 25 GB/s（每天减少超 2 PB）。 全链路早期聚合与资源优化成效（2026秋） 针对依赖 Rollup（固定维度指标）的分析数据集，团队改变了过去仅在 ClickHouse 入库阶段聚合的做法，推出了全链路聚合功能 Ready-Aggregates。数据在 Logfwdr 边缘、中心接收器、ClickHouse 插入器及 ClickHouse 聚合引擎中经历层层部分聚合，在最高基数的 HTTP 数据集上实现了约80%的传输降幅（从超 9 GB/s 降至 2 GB/s 以下）。综合上述改进，2026年从春到秋累计从骨干网卸载了约 70 GB/s 的压缩流量，仅 Ready-Analytics 单一业务接收器便节省了 750 核 CPU 和 3.5 TiB 内存。

More than 20% of the web sits behind Cloudflare’s network, and our team, the Data team, owns the data pipeline responsible for plumbing that data (and more) from our global network to applicable central data centers. The Cloudflare network is a globally distributed system supporting tens of thousands of servers. Data that power [Cloudflare Logs](https://developers.cloudflare.com/changelog/post/2026-09-30-logpush-usage-based-pricing/), [SQL API](https://developers.cloudflare.com/analytics/sql-api/), [observability](https://blog.cloudflare.com/one-observability-platform/), internal analytics, and more flow through this network and use our data pipeline to serve customers. In May of this year, we were providing data delivery at roughly two billion events per second. This includes customer traffic as well as data sent from internal teams. Year after year, we tackle the same problem — exponential traffic growth against linear hardware growth. We have to react to this increase in order to keep resource costs under control. Over the past five years, we’ve made various improvements to compensate for the data volume increase in comparison to hardware allocation, equivalent to deferring two and a half years of hardware expansion. This post details those changes: breaking up our monolithic data schema in 2022, adjusting compression in 2024, rearchitecting pipeline replication in spring 2026, decentralizing sampling in summer 2026, and introducing aggregations into the pipeline in fall 2026. Our pipeline starts with an internal service called Logfwdr, where we collect from highly distributed data sources (think: Workers, [FL](https://blog.cloudflare.com/20-percent-internet-upgrade/)). We then feed this data to a set of centralized receivers, which eventually push them onto our [buffering system](https://blog.cloudflare.com/cloudflare-incident-on-november-14-2024-resulting-in-lost-logs/#buftee), Buftee. Consumers fetch data from Buftee and distribute data to their final destinations, like ClickHouse, which we [rely on heavily](https://blog.cloudflare.com/clickhouse-query-plan-contention/). In 2022, we dismantled our one and only schema. Previously, there was a single stream packed with every product's fields, forcing Cloudflare-internal consumers to ingest everything and demux or discard records via filters. At the time, most of Cloudflare’s data products were built on HTTP, Security and CDN metadata. In 2021, our message schema looked like this: Some fields here belonged under the same product, but some were completely irrelevant to each other. The hypothesis was that none of these teams require 100% of fields in the monolithic schema. In other words, we can create small messages for each product’s required fields. Although we were going to duplicate fields, some of them in every message, replacing this firehose with product-specific schemas actually reduced the backbone bandwidth usage by 35-40%, knocking out roughly a year’s worth of compressed traffic growth. By 2024, as traffic steadily grew, we were tackling network congestion issues on a daily basis. To react to this, we changed our compression algorithm from LZ4 to zstd. We had already experimented with zstd compression before, but we noticed a higher CPU usage at the time. Therefore, we were careful to track the metrics to ensure the trade-off was worth it. Initially, we rolled this change out to a few servers. To our relief, we didn’t observe a noticeable difference in CPU usage. With more confidence, we gradually rolled out to our entire fleet, which also showed little impact to CPUs. In the end, there was nearly a 40% reduction in compressed traffic throughput. The uncompressed bytes remained at the same growth, meaning we could support more than 1.5x the amount of traffic with the same size pipe. Or another way to put it, the estimated growth-equivalent reductions put us at about 10 months prior. Now, in 2026, we attempted three strategies to reduce uncompressed bytes (and thus, compressed bytes). To make analytics resilient against regional disasters, our systems require redundant data to be stored in multiple centralized regions (data subject to [Customer Metadata Boundaries](https://developers.cloudflare.com/data-localization/metadata-boundary/) is stored strictly within its designated region). Historically, the simplest way to achieve disaster resilience was double-writing: Logfwdr transmitted two full copies of every non-regional analytics event — one to each centralized region. Each region then independently processed a complete copy through to ClickHouse. We also apply [sampling](https://developers.cloudflare.com/analytics/sampling/) in each region, an incredibly scalable technique that works for all volumes of data during any traffic conditions. We downsample high-volume traffic while rare events are preserved. This infrastructure was manageable when only a handful of streams used secondary regions. However, as we expanded disaster resilience across almost all analytics streams, Logfwdr traffic doubled across the board. Sending every event, twice, over wide-area transit was unsustainable against our exponential traffic growth. We had already tackled a similar challenge for [Logpush](https://developers.cloudflare.com/logs/logpush/). For Logpush, Logfwdr treats the centralized data centers as a unified, shared pool of receivers: each message is sent to only one region, failing over to another destination if a machine or facility is unreachable. That model had already proven its worth during large incidents, shifting traffic seamlessly from one location to another without human intervention. This automatic failover was possible because Logpush delivery is stateless, as customer destinations do not care which centralized data center pushes the logs. Analytics was trickier. Unlike stateless logs, ClickHouse clusters in both regions still need a full copy of non-regional data to allow instant, independent query failover. The breakthrough was decoupling pipeline transit from database redundancy: **1. Shared Ingest Pool:** We adopted Logpush’s model to the analytics streams. Logfwdr dispatches each event exactly once across wide-area transit to whichever centralized receiver is available. **2. Late Replication from centralized regions:** Instead of replicating traffic at source, replication happens as late as possible in the pipeline. Data lands in our distributed buffering layer, Buftee, and remote inserters stream cross-data-center copies downstream. **3. Resilient Failure Handling:** If a centralized region goes down, Logfwdrs immediately routes all non-regional traffic to the surviving region. If the inter-data center connection blips, each cluster continues ingesting its local share without backpressure; remote inserters catch up from Buftee's retention buffer once the backbone recovers. For the datasets we migrated, the cutover eliminated approximately 43 GB/s of compressed transit traffic while adding only \~7 GB/s of cross-data-center replication on our pipeline. Two architectural factors explain this asymmetry: 1\. **Replication happens downstream of sampling**: In the old model, Logfwdrs transmitted two copies of raw, unsampled events over wide-area transit. In the new model, Logfwdrs send a single copy to centralized receivers, which immediately apply sampling before buffering into Buftee, carrying only this already-compacted stream. 2\. **Better compression through uniform partitioning**: Data at source is fragmented across thousands of individual servers. After centralization, streams are aggregated and partitioned into uniform batches over wider time windows. This structural uniformity allows compression algorithms to achieve dramatically higher compression ratios than Logfwdr transit batches ever could. Overall, this was a net reduction of 36 GB/s across wide-area networks. We rolled this out initially to two datasets, HTTP analytics and Ready-Analytics, our [in-house one-click analytics product](https://blog.cloudflare.com/clickhouse-query-plan-contention/#the-setup-a-petabyte-scale-analytics-platform). We’ve been actively migrating more datasets to this load-balanced architecture since the spring. Another strategy we employed was further expanding our sampling mechanism so that we pulled critical processing earlier into the pipeline. Sampling was already used in our centralized services, particularly the receivers, easing pressure on downstream systems. We can be more resource efficient if we send less data downstream (without sacrificing data quality). The solution here was to simply sample in Logfwdr, in addition to maintaining the sampling in centralized receivers. How the sampling algorithm works here, in short, is each dataset has a key that producer teams have chosen to index and sample the data. When sampling, we first [shuffle the records for each key to eliminate bias](https://blog.cloudflare.com/how-we-make-sense-of-too-much-data). Then, for each group of records, we select a sample size equal to the square root of the total count of the records. The result is a representative sample, whether sampled once or twice. To produce estimates, we scale calculations against the sample rate. For example, if we sample 10 out of 100 records, the sample rate would be 10/100, and the unbiased estimate of the original count would be 10\*10 = 100. To ensure this approach was working well, we compared the number of rows written to ClickHouse against the estimated sum for total count using a variety of keys across a variety of traffic footprints, so that “least seen” keys to “most seen” keys were captured across the spectrum. The result is that for keys with more data, the sample rate decreases. However, keys with less data were sampled comparably. Thus, we still kept a representative sample while delivering and storing fewer events. We’re not compromising the quality of data delivered, even when we deliver less in quantity. If we look at this more closely, using power laws, we can observe that the double-sampled (red) grows at roughly x^0.319 v. sampled-once/control (blue) at x^0.485. Another way to picture this is, a key that has 100x more data than another key would have stored roughly 10x the number of rows if single-sampled. Now, it will only store roughly 4x the number of rows. Concretely, a total of 11.6 million events is represented by 18k rows in ClickHouse as opposed to 79k. Double sampling attenuates storage of the “most seen” keys, while still retaining the same detail per event for “least seen” keys. We initially implemented this strategy against three of our top datasets. Every single one at least halved the amount of compressed data we were originally sending on the wire to centralized receivers. That’s 25GB/s altogether, or over 2PB/day. Cloudflare offers several analytics datasets built on rollups: aggregates of the raw events computed at ingest time. This makes the opposite tradeoff to sampling: rather than answering any question approximately, we answer a fixed set of questions exactly, provided we choose the dimensions and metrics up front. For example, in the HTTP dataset, we measure the number of unique occurrences of IP addresses during a given minute rather than the individual IPs themselves. Another example is we sum the total number of GET, POST, PUT, etc. requests for a given domain, for that minute, grouped by the HTTP method. None of these computations actually necessitate waiting until the tail-end of the pipeline to start calculating. Historically, this aggregation occurred towards the end of the pipeline, inside the inserters which read from Buftee and insert into ClickHouse. Decentralizing this aggregation was an opportunity to spread out load and reduce the amount of data we centralize. Hence, we built an aggregation functionality that is utilized every step of the way, creating repeated opportunities to combine matching partial aggregates as data converges. Much like sampling, records are grouped by a key, which determines which records can be combined. We aggregate on each instance of Logfwdr. We aggregate again after the partially aggregated events reach the centralized receivers, which push these re-aggregated events onto dedicated partitioned buffers. Because these buffers bring together records from multiple receivers, we aggregate another time after they’ve been read in batches by the ClickHouse inserters. Finally, we aggregate in ClickHouse with aggregating table engines. The first dataset we rolled this out to was for the HTTP traffic. In just this one dataset alone, we dropped from 9+ GB/s to \<2 GB/s in compressed bytes traffic. Even though this is currently the highest cardinality dataset, the aggregations on it still gave us an approximate 80% reduction on the wire. This aggregation feature is powerful for datasets that can collapse easily, especially for use cases like usage metrics where accuracy on the sums and counts is paramount. It also means we can offer producer teams another option to reduce resource usage besides sampling, which we did. Like Ready-Analytics, we created a generic in-house one-click aggregating analytics product, called Ready-Aggregates. As we continue to iterate on this new feature in the pipeline, we expect to see savings continue to mount. The starkest proof of this exponential growth is how rapidly traffic growth consumes headroom. Take Ready-Analytics for example, where two of the three mechanisms, load balancing and sampling techniques, were applied. In mid-May, load balancing dropped traffic by 50%, plunging from \~26 GB/s to \~13 GB/s. Two months later, traffic had already started surging back up to \~20 GB/s. That’s about half of the reduction, erased by organic traffic growth. We then created even more headroom by reducing this dataset’s throughput rate to the single digits, \~8GB/, when sampling rolled out. Without both interventions, Ready-Analytics alone would have breached 60 GB/s by September. By processing less data in centralized data centers, we also save a significant amount of CPUs and memory. The following graphs show the impact of load-balancing and sampling in Logfwdr on the CPU and memory usage of just the Ready-Analytics receivers: a \~50% reduction, regaining \~750 CPUs and 3.5 TiB of memory. Across the entire pipeline altogether, from spring to fall, load balancing (\~38 GB/s saved), sampling (\~25 GB/s saved across top datasets), and aggregating (\~7+ GB/s so far), eliminated \~70 GB/s of compressed traffic from our backbone. These cliff drops provide immediate relief. If we zoom out such that we look at five years from now, the changes made in this year alone have just as much impact as the short-term benefits. We’ve halved the amount of data we’d be processing in another half a decade’s time. The structural and algorithmic optimizations we have and will continue to make are essential to keep hardware demands sustainable. If you’re interested in helping us solve these challenges at scale, we’re [hiring](https://job-boards.greenhouse.io/careersday/jobs/8279026?gh_jid=8279026&gh_src=jv4gk9q31us)!