Cloudflare Logpush Introduces In-Process SQL Transformers for Log Export

Cloudflare Logpush Introduces In-Process SQL Transformers for Log Export Cloudflare has made Transformers generally available across all Logpush plans, allowing teams to filter, redact, and reshape log records using SQL queries directly within the export pipeline before delivery to external destinations. Transformers embed SQL transformation directly into Logpush's export pipeline to reduce downstream storage costs and eliminate the need for external preprocessing services. Cloudflare chose an in-process Rust-based Apache DataFusion engine over external Cloudflare Workers to prevent network hops and maintain high throughput. Data transfers between Go and Rust across an FFI boundary using Apache Arrow IPC, maintaining explicit memory ownership and achieving ~1.6 million rows per second per goroutine in benchmarks. Transform execution fails closed, retrying with exponential backoff rather than leaking untransformed or unredacted records downstream. SQL-Driven Log Pipeline Transformers allow users to write SQL queries against specific Logpush datasets to remove health checks, nest or rename fields, and redact sensitive information prior to export. Applied to each batch within Logpush's high-throughput pipeline, records matching the query's WHERE clause are filtered, and the SELECT statement produces the exported NDJSON output. Users can configure and version Transformers via Transformer Studio or the Cloudflare API across all plan tiers. Architecture: Apache DataFusion and Arrow IPC To process over 110 million records per second across Kubernetes pods, Cloudflare rejected a Worker-based prototype to avoid network latency and transient timeouts. Instead, the team embedded Apache DataFusion directly into Bufstream, its Go-based delivery engine. Data is passed across the Go-Rust FFI boundary via Apache Arrow IPC, preserving typed columnar representations without row-by-row parsing, while Go and Rust maintain separated, explicit memory lifecycles. Query Planning and Fail-Closed Operations Bufstream caches SQL queries from R2, converts them into reusable DataFusion logical plans against predefined dataset schemas, and executes them concurrently across batches. The transformation mechanism is strictly fail-closed: if an execution error occurs, Bufstream halts the job offset and repeatedly retries with progressive delays rather than sending raw, unredacted logs. Persistently failing jobs are automatically disabled, and successful outputs are compressed and uploaded based on destination concurrency limits.

Cloudflare 在 Logpush 中正式推出 Transformers：基于 DataFusion 的日志导出前 SQL 转换引擎 Cloudflare 宣布 Transformers 正式向所有自助服务和企业计划开放。通过在 Logpush 导出管道内嵌入 Apache DataFusion SQL 引擎，用户可在日志投递至 SIEM 或存储之前完成过滤、字段脱敏和结构重构，从而免除额外维护独立日志处理管道的成本。 Transformers 支持通过 SQL 在导出前对 Logpush 日志进行过滤、脱敏和重构，并输出 NDJSON 格式。 放弃最初基于 Cloudflare Workers 的网络隔离方案，转而在交付服务 Bufstream 中进程内嵌入 Rust 编写的 Apache DataFusion 引擎以保证高吞吐。 Go 与 Rust 之间采用 Apache Arrow IPC 和 C ABI 进行零跨语言所有权绑定的批处理内存交换。 采用严格的 Fail-closed 容错机制：转换失败绝不降级发送原始数据，持久失败任务将被逐步退避并自动禁用。 功能概述与使用场景 Logpush 支持将日志快速导出至 SIEM、分析系统及对象存储。Transformers 允许用户通过 SQL 查询定义过滤条件、脱敏敏感信息以及重组嵌套字段，避免了在下游额外搭建计算服务的麻烦。在导出前移除无用流量（如健康检查）还可降低下游摄取和存储成本。用户可通过 Cloudflare 控制台的 Transformer Studio 或 API 进行可视化编辑与版本管理。 架构选型：进程内 Apache DataFusion Cloudflare 的数据管道每秒摄取超 10 亿事件，Logpush 交付服务 Bufstream 每秒投递超 1.1 亿条记录。初期原型虽尝试使用 Cloudflare Workers 执行 WebAssembly，但引入的网络调用会导致超时和重试延迟。为保证吞吐量与稳定性，团队选择在 Bufstream 进程内嵌入基于 Rust 和 Apache Arrow 构建的 Apache DataFusion 查询引擎，单 Goroutine 压测速度可达约每秒 160 万行。 跨 Go/Rust 边界的高效内存管理 Bufstream（Go）接收 Cap'n Proto 格式日志后，结合数据集规范构建类型化的 Apache Arrow RecordBatch，并通过 cgo 与 C ABI 经由 Arrow IPC 与 DataFusion（Rust）进行交互。两门语言间不共享内存所有权：Go 仅在调用期间传递只读缓冲区指针，转换完成由 Go 拷贝后显式释放 Rust 端分配。该批处理设计规避了按行解析 JSON 的开销，并复用了预先编译的逻辑计划。 容错设计与交付控制 Transformers 设计遵循“故障关闭”（fail-closed）原则，若转换失败，Bufstream 既不会推进日志偏移量，也不会发送未转换的裸日志，以防止下游接收到未脱敏或格式不符的数据。针对持续失败的任务，系统会逐步延长重试延迟并在长期异常后自动禁用。转换完成的 Arrow 数据最终转为 NDJSON 格式，由适配并发速率的 Goroutine 压缩并投递至最终目的地。

Logpush delivers logs in batches as quickly as possible, enabling Cloudflare to provide information almost in real time. During our recent [Birthday Week](https://blog.cloudflare.com/birthday-week-2026-wrap-up/), we made Logpush available on all self-serve plans and [Transformers](https://developers.cloudflare.com/logs/logpush/transformers/) generally available. Those changes were among [eight major updates to Cloudflare Observability](https://blog.cloudflare.com/one-observability-platform/) and reflect our commitment to make [every Cloudflare feature available to everyone](https://blog.cloudflare.com/enterprise-grade-features-for-all/). This post takes a deeper look at how Transformers work inside Logpush's export pipeline. Observability teams use [Logpush](https://developers.cloudflare.com/logs/logpush/) to export Cloudflare logs to security information and event management systems (SIEMs), analytics platforms, and object storage. Before those logs reach their destination, teams may need to remove health checks, omit sensitive fields, or reshape records for dashboards and investigations. Today, this often requires a separate processing service that teams must deploy, scale, monitor, and maintain. For example, a team that collects logs from multiple CDNs can add a static provider field to normalize records before they reach its analytics platform. Teams also pay to export, ingest, and store records they discard as soon as they arrive, so filtering them before delivery can reduce downstream data volume and cost. With Transformers, that processing moves into the Logpush export workflow.Teams define a SQL query to filter records, redact sensitive data, and reshape the output by adding, renaming, removing, or nesting fields. Logpush applies the query before export, so each destination receives data in the form it needs. We made this workflow simple for customers by embedding a SQL transformation engine in Logpush's high-throughput export pipeline while keeping performance and resource use predictable. A Transformer is a SQL query written for a specific Logpush dataset. The query references fields from that dataset and defines which records and fields Logpush should export. Logpush applies the query to each batch before export. Records that do not match the WHERE clause are removed. For the remaining records, the output of the SELECT statement becomes the exported record. Transformers can change the structure and contents of each record, and the transformed results are exported in NDJSON format. Customers can create, preview, and manage Transformers in the Cloudflare dashboard through Transformer Studio or programmatically through the [API](https://developers.cloudflare.com/api/resources/logpush/subresources/transformers/). After saving a Transformer, they can attach it to a compatible NDJSON Logpush job that uses the same dataset. Each update creates a new version, so customers can review how a Transformer has changed over time. The following example reshapes a [Cloudflare HTTP request log](https://developers.cloudflare.com/logs/logpush/logpush-job/datasets/zone/http_requests/) to match a team-defined schema used across CDN providers. It does not convert between serialization formats: both the input and output use NDJSON. ``` { "ClientIP": "203.0.113.42", "ClientRequestHost": "example.com", "ClientRequestMethod": "POST", "ClientRequestPath": "/api/checkout", "ClientRequestUserAgent": "Mozilla/5.0", "EdgeResponseStatus": 502, "EdgeStartTimestamp": "2026-05-21T15:00:00Z", "RayID": "8e2a1c60ef9e1c9a" } ``` The Transformer renames and nests fields, redacts part of the client IP address, adds provider and error information, removes unused fields, and filters out internal and health-check traffic: ``` SELECT extract(epoch FROM to_timestamp(EdgeStartTimestamp)) AS unix_ts, regexp_replace(ClientIP, '\.\d+\.\d+$', '.XXX.XXX') AS redacted_ip, UPPER(ClientRequestMethod) AS method, EdgeResponseStatus AS status, RayID AS ray_id, 'cloudflare' AS provider, named_struct( 'host', ClientRequestHost, 'path', ClientRequestPath ) AS request, EdgeResponseStatus >= 500 AS is_server_error FROM http_requests WHERE ClientRequestHost NOT IN ( 'internal.example.com', 'health.example.com' ) AND ClientRequestPath NOT LIKE '/healthz%' AND ClientRequestPath NOT LIKE '/metrics%' ``` The resulting NDJSON record uses the normalized schema: ``` { "unix_ts": 1779375600, "redacted_ip": "203.0.XXX.XXX", "method": "POST", "status": 502, "ray_id": "8e2a1c60ef9e1c9a", "provider": "cloudflare", "request": { "host": "example.com", "path": "/api/checkout" }, "is_server_error": true } ``` Cloudflare's data pipeline ingests more than one billion events per second. Its Logpush delivery service, which we call Bufstream, delivers more than 110 million Logpush records per second to customer destinations. It runs across hundreds of Kubernetes pods, each claiming a time slice of data from [Buftee](https://blog.cloudflare.com/how-we-make-sense-of-too-much-data), Cloudflare's distributed log-based in-house message broker. This model distributes work horizontally across Logpush jobs. Job volume follows a Pareto-like distribution: most jobs operate at lower volumes, while a small number of high-volume jobs process millions of logs each second. Transformers needed to meet the same operational requirements as the rest of Logpush: reliable delivery, high throughput, and predictable resource use without introducing additional failure scenarios into the export hot path. Our initial prototype ran customer-defined WebAssembly Transformers in Cloudflare Workers, which provides globally distributed, isolated compute. However, invoking a Worker would insert a network call into the Logpush export path. That remote dependency could introduce transient timeouts and retries, reducing delivery throughput and increasing delivery latency. We decided to keep Transformer execution in-process within our delivery service to prioritize reliability and delivery throughput. We evaluated several SQL execution engines and selected [Apache DataFusion](https://datafusion.apache.org/), which already powers [Cloudflare Basin](https://blog.cloudflare.com/cloudflare-basin). Apache DataFusion is a Rust query engine built on Apache Arrow, with a SQL planner and vectorized, columnar execution engine. In a benchmark, executing a prepared Transformer against a synthetic 5,000-row http\_requests batch from a single Go goroutine took about 3 ms, equivalent to roughly 1.6 million rows per second. Attaching a Transformer to a job adds CPU work to its processing path. Bufstream uses per-job transform parallelism so high-volume jobs can execute transform batches concurrently. Embedding DataFusion in Bufstream lets those batches use the service's existing delivery infrastructure, including its scheduling, batching, and destination controls, without introducing a separate transformation tier or network hop. Choosing DataFusion meant connecting Bufstream, which is written in Go, to an engine written in Rust. The next challenge was moving typed batches efficiently across that boundary. The final design of the in-process Transformer pipeline is shown in the diagram below. Bufstream receives compressed Logpush batches containing Cap'n Proto records from Buftee. For a job with a Transformer attached, Bufstream decompresses the batch, parses the eligible logs, and builds a typed Apache Arrow RecordBatch before executing the transform. Apache Arrow is a language-independent, columnar in-memory format designed for efficient analytics on modern CPUs and GPUs. It represents flat and nested data as typed, contiguous columns. Arrow IPC gives Bufstream and DataFusion a standard way to exchange batches across the foreign function interface (FFI) boundary between Go and Rust, exposed through a C application binary interface (ABI) and called from Go using [cgo](https://pkg.go.dev/cmd/cgo), Go's mechanism for calling C code. The Go and Rust runtimes do not share ownership of transform data. Go passes a pointer to its input Arrow IPC buffer to Rust only for the duration of the cgo call, and Rust does not retain that pointer after returning. Rust owns the transformed output buffer; Go copies it into Go-managed memory and then explicitly releases the Rust allocation. This makes ownership and lifetimes explicit across the language boundary, avoiding cross-runtime memory-management bugs. Using Arrow IPC means the Rust DataFusion wrapper can use an Arrow `StreamReader` to read typed `RecordBatch` values, rather than parse individual rows from an intermediate format such as JSON and rebuild typed Arrow columns for execution. Field types, nullability, and nested values are represented consistently on both sides of the boundary, so DataFusion receives data in its native columnar model. Crossing the Go/Rust boundary is not free: batches still need to be serialized, decoded, and managed in memory. Arrow IPC keeps that work batch-oriented and avoids per-record parsing and conversion. This makes the boundary a better fit for Logpush's high-throughput delivery pipeline while retaining schema validation against the Logpush dataset definition. Every Arrow batch needs a schema describing its fields and types. Rather than infer that schema from the logs in a particular batch, Bufstream supplies a generated schema from each Logpush dataset's specification. This gives DataFusion a stable view of the available fields and types, enabling query validation. The table named in a Transformer's FROM clause selects which dataset schema the query uses. For example, this Transformer queries the http\_requests dataset: ``` SELECT ClientIP, EdgeResponseStatus, RayID FROM http_requests WHERE EdgeResponseStatus >= 500 ``` When a customer saves a Transformer through the API or Transformer Studio, Cloudflare stores its SQL query in R2. Bufstream retrieves the query through a multi-layer cache and translates it into a reusable DataFusion logical plan that defines the scan, filters, projections, and expressions. Bufstream prepares this plan once per time slice of a job's data and reuses it for every Arrow batch in that slice. For every new Arrow batch, the Rust wrapper binds the batch as the plan’s table input, executes the query, and returns Arrow batches containing the selected records and fields. Transformers are designed to fail closed. If a transform cannot be executed, Bufstream neither advances the job's delivery offset nor sends the untransformed batch. Delivering raw data in place of transformed data could violate the schema, filtering, or redaction assumptions of downstream pipelines, so Bufstream opts to re-attempt failed transforms instead. To prevent a persistently failing job from consuming capacity needed by healthy jobs, Bufstream progressively delays retries and reports errors on the job. Jobs that remain in a failing state for an extended period are automatically disabled. Customers can monitor successful and failed transformations, data volume before and after transformation, and transformation duration in the Transformer Health view of the Health tab in Logpush. After execution, Bufstream converts the Arrow output into NDJSON logs. It batches, gzip-compresses, and uploads those logs to the configured destination. Destinations accept data at different rates and can temporarily slow down. Bufstream adjusts the number of upload goroutines while respecting destination concurrency limits. By embedding SQL execution in Bufstream, Transformers use Logpush’s existing distributed infrastructure, batching, and destination controls. Cloudflare’s data pipeline handles that complexity, while customers create and apply a Transformer with only a few steps. Open Transformer Studio from the [Logpush page](https://dash.cloudflare.com/?to=/:account/logs) in the Cloudflare dashboard and follow these steps: Transformers is available on Free, Pro, Business, and Enterprise plans. See the [Transformers documentation](https://developers.cloudflare.com/logs/logpush/transformers/) for setup details and the [Logpush pricing documentation](https://developers.cloudflare.com/logs/logpush/pricing/) for current pricing information.