Cloudflare Launches K2 Serverless Event Streams in Public Beta

Cloudflare Launches K2 Serverless Event Streams in Public Beta Cloudflare has introduced Cloudflare K2 in public beta for Workers Paid accounts, providing a fully serverless, durable event streaming primitive designed for high-scale data movement and long-term retention. K2 decouples producers and consumers by storing events as an ordered, durable log on top of Cloudflare R2 object storage. Originally built as an ingestion buffer for Basin Pipelines, K2 avoids traditional systems like Kafka in favor of R2's consistency and durability, accepting around 1 second of produce latency at p99 due to batching. Unlike Cloudflare Queues' fine-grained item tracking and Basin Pipelines' direct-to-storage paths, K2 focuses on batched fan-out consumption and custom processing. K2 is free during public beta, with future pricing set at $0.04 per GB produced, $0.04 per GB consumed, and $0.02 per GB per month for retained data. Architecture on R2 Cloudflare built K2 because traditional distributed software like Apache Kafka does not suit its global edge infrastructure across over 335 cities. To implement append-only logs on R2—which does not natively support appends—K2 batches writes in memory on the edge before persisting them as segment files. This separation of compute and storage delivers 11 9s of durability and strict offset ordering using atomic operations, though local batching introduces roughly 1 second of produce latency at the 99th percentile. Workload Differentiation Cloudflare contrasts K2 with existing asynchronous tools like Queues and Basin Pipelines. While Queues focuses on individual, expensive work units featuring dead-letter queues and message-level retries, K2 handles massive data movement and fan-out through batching. Basin Pipelines is suited for ingesting and writing data directly to R2 or Iceberg tables, whereas K2 is positioned for custom transformations and alternative downstream destinations. Stream Consumption and Availability K2 streams accept arbitrary byte payloads via HTTP API or Worker bindings. Consumers use subscriptions to either split partitions for parallel processing or set up independent subscriptions for pub/sub fan-out, leasing batches for 5 minutes during consumption. K2 is currently available in public beta without billing for Workers Paid users, with anticipated production pricing of $0.04 per GB produced, $0.04 per GB consumed, and $0.02 per GB per month for storage.

Cloudflare推出无服务器事件流服务K2公开测试版 Cloudflare宣布推出基于R2对象存储构建的无服务器持久化事件流服务K2公测版，旨在解决边缘环境中生产者与消费者解耦、海量数据吞吐以及长期持久化缓冲的需求。 K2基于R2对象存储构建分区持久日志，实现计算与存储解耦并提供高持久性保障。 采用边缘内存批处理攒批后再写入R2分段文件的方案，P99写入延迟约为1秒。 与专注于单任务重试的Queues相比，K2面向大规模数据流动、长周期存储与批处理扇出消费。 K2在公测期面向Workers付费用户免费开放，未来预计生产与消费各收取0.04美元/GB。 架构背景与基于R2的实现原理 传统分布式系统如Kafka难以适应Cloudflare跨越335个以上城市的碎片化、瞬态边缘机器架构。为此，团队基于具备强一致性与高持久性的R2构建了K2，将复制与共识下沉至存储层。针对对象存储不支持追加写（append）的问题，K2先在边缘内存中聚合写入，延时极短时间后统一输出为分段文件，并通过R2原子操作保障顺序与递增偏移量，初期P99写入延迟约为1秒。 产品定位与现有产品差异 Cloudflare明确了K2与既有异步原语的区别：Queues围绕单项耗时任务的生命周期设计，提供重试、延迟与死信队列机制；K2则以批次生产消费为中心，以较高的写入延迟和放弃单消息重试为代价，实现高吞吐移动、长效留存和扇出消费。若最终目标是直接转换并写入R2或Iceberg表，则推荐使用Basin Pipelines，自定义处理场景更适合K2。 消费机制与公测计费策略 K2支持通过HTTP API或Worker绑定进行生产，并利用订阅（Subscription）划分消费工作。消费者轮询获取批次事件时可享有5分钟租约，既可通过多读者分流实现并发读取，也可通过独立订阅实现Pub/Sub全量广播。K2现已面向Workers付费用户公测且测试期免费，后续预计对数据生产与消费各按0.04美元/GB计费，数据留存费用为每月0.02美元/GB。

With traditional Remote Procedure Call (RPC) architectures, there exists a core challenge: producers and consumers must align in scale and in time. If your producers send too much data for your consumers to handle or if your consumers or downstream services become unavailable, events are dropped. This problem is compounded with multiple consumers that need to independently process the data. For example, an ecommerce backend may emit events when transactions are completed, which need to be read by an analytics system and a fraud detection service. We can solve this by *decoupling* our producers and consumers — inserting a service in the middle that absorbs writes while allowing independent readers to consume at their own pace. Today we are launching Cloudflare K2 in public beta to solve this problem. K2 is a durable event streaming primitive on the Developer Platform. You send events to a K2 stream, which stores them as an ordered log. Consumers can read them in a variety of ways, for example by splitting up reads across a set of consumers, or delivering all messages to all consumers. It's fully serverless, scales to vast quantities of data, and supports long-term retention, so even long periods of consumer downtime do not lose data. Under the hood, K2 implements a partitioned, durable log on top of R2 object storage, which allows it to scale to huge volumes of storage. If you’re ready to get started, you can create your first stream in seconds by following [the guide here](https://developers.cloudflare.com/k2/get-started/). We first built K2 because *we* needed a durable buffer on the edge, initially to serve as the ingestion layer for [Basin Pipelines](https://developers.cloudflare.com/pipelines/). Pipelines is powered by a [stream processing engine](https://www.arroyo.dev) that operates on a pull-based model, which means some other system has to store events before they are read, transformed, and written to R2. And because we commit to never dropping events once they’re accepted into the Pipelines Stream, that storage has to be durable — meaning it can’t lose data — over potentially long periods of time. This is where most companies would deploy Apache Kafka. However, Pipelines runs on the Cloudflare edge, which spans a huge number of servers across over 335 cities. Our unique architecture means we often cannot run traditional distributed systems software like Kafka, and need to rethink how these systems are built and operated. For stateful services, in particular, Cloudflare’s global infrastructure presents some challenges: we get relatively small slices of machines, those machines are relatively ephemeral, and networking is often over the public Internet. But our infrastructure also has a few superpowers: it’s close to users wherever they are in the world and has an incredible capacity to scale horizontally. In designing the durable buffering system that became K2, we decided to rely on the powerful state primitive we already have: R2. Object storage systems like R2 combine extremely durable storage ([11 9s!](https://developers.cloudflare.com/r2/reference/durability/)) with strongly consistent APIs. Offloading replication and consensus to the storage layer allows us to make the application layer (K2 in this case) radically simpler, cheaper, and higher performance. A secondary benefit is that it separates compute and storage, meaning each can be scaled independently. This allows us to store vast quantities of historical data at low cost. How do we build a log on top of object storage? An immediate issue is that R2 — like other object stores — does not support appends, the standard operation on a log. Instead, we must write complete files, or segments, that are large enough to overcome the cost of writing and reading each one. We do this by first accumulating writes in-memory on an edge service. After waiting a short period for data to arrive, we write all events as a segment file. We achieve ordering and strictly incrementing offsets using R2’s atomic operations without needing a separate coordination service. While building on R2 has many advantages, there is one downside: higher produce latencies. Writing to object storage is slower than a local disk, and we have to wait for the local batch to accumulate before starting the write. In our initial release of K2, this adds up to about 1 second of produce latency at the 99th percentile of response times. We will be sharing more details on the design of K2 in an upcoming technical deep dive. Cloudflare has several existing asynchronous delivery primitives, including [Queues](https://developers.cloudflare.com/queues/) and [Basin Pipelines](https://developers.cloudflare.com/basin-pipelines/). When should you reach for K2 instead of these existing products? There are some superficial similarities between Queues and K2 Streams: both receive events, durably store them, and deliver them to consumers. Queues are designed around tracking individual items of expensive or time consuming work that need to be asynchronously completed. For example, an image processing application may enqueue a user request to be handled by the actual image processing service. They support complex logic on the grain of a particular work item, like retries, delays, and dead-letter queues for failed attempts. K2, by contrast, is designed for high-scale data movement, long-term retention, and fan-out consumption. Messages are produced and consumed as batches — enabling efficient processing at the expense of message-level retries. This batching also drives higher producer latency than for queues. Basin Pipelines is a serverless ingestion service. You can send your Pipeline JSON events, which can be transformed and written to R2 or a Basin Catalog. We recommend Pipelines when the end result is writing your events to object storage or Iceberg tables, and K2 when doing custom processing or writing to other destinations. Using K2 involves first creating a stream. You can have many streams across your account for different use cases or types of events. Streams can be created via [cf](https://blog.cloudflare.com/cloudflare-cf-cli-launch/), Wrangler, the dashboard, or API. Let's take the example of collecting and processing product analytics. First, we'll create a stream with cf: Once we have a stream, we can start producing to it, via an HTTP API or Worker binding. For example, we can [configure a Worker binding](https://developers.cloudflare.com/k2/features/produce/#produce-from-a-worker) and then produce to it like this: K2 represents data as bytes, so you can use whatever format or encoding makes sense for your application. Now that we have events in a stream, we can create a subscription. Subscriptions divide up work between consumers, enabling *read parallelism* — scaling out to multiple readers to handle more load than a single server can manage. We can create a subscription via the HTTP API. With the subscription created, we can then poll it from each of our consumers: When a client calls `consume`, they receive a *lease* for that particular batch of events for 5 minutes. The client can do one of three things: This is one way to consume from K2: splitting work amongst multiple consumers such that each consumer gets a portion of the data. Another way to read is with a separate subscription for each consumer — the pub/sub pattern — in which case each consumer sees all of the messages. Or you can mix-and-match between these two approaches, having multiple independent consumer pools. See the [K2 docs](https://developers.cloudflare.com/k2/) for full details on the APIs. K2 is available today in public beta for accounts with Workers Paid subscriptions, within these limits: If you need higher limits, please reach out to the team on Discord or fill out the [limit increase form](https://docs.google.com/forms/d/e/1FAIpQLSf1_eGOxXFGsh4O72fqdvDbYBjzpV3Q6Vf5X3fqpoTHObGFIA/viewform). Usage of K2 will not be billed during the beta period. Once we begin billing, we anticipate this pricing: | | Pricing | | Data Produced | $0.04 / GB | | Data Consumed | $0.04 / GB | | Data Retained | $0.02 / GB / month | We have an exciting roadmap for K2 over the coming months, including: We’re excited to see what you build on K2! Share your feedback on the [Cloudflare Discord](https://discord.com/invite/cloudflaredev).