Cloudflare's Logpush Transformers go GA with SQL filtering and redaction on self-serve plans

Cloudflare has taken Logpush Transformers to general availability, giving customers a way to filter, reshape and redact log records with SQL before those records leave the platform. According to the Cloudflare blog, the feature arrived during the company's recent Birthday Week alongside a wider rollout of Logpush itself to all self-serve plans, with the two changes counted among eight updates to Cloudflare's observability products.

A SQL stage inside the export pipeline

Logpush batches Cloudflare logs and delivers them to security information and event management systems, analytics platforms and object storage. The Cloudflare blog notes that teams often need to clean those logs before they land: dropping health-check traffic, hiding sensitive fields, or restructuring records so they match a dashboard or a company-wide schema. Historically that work often required a separate processing service that a team had to deploy, scale, monitor and maintain, and it also meant paying to export, ingest and store records that were discarded as soon as they arrived.

A Transformer is a SQL query written for a specific Logpush dataset. Logpush applies the query to each batch before export: records that fail the WHERE clause are dropped, and the output of the SELECT statement becomes the exported record. Transformations can rename fields, nest them, add computed values such as a static provider identifier for teams normalising logs across multiple CDNs, or mask data like the trailing octets of a client IP. Input and output both remain NDJSON, so Transformers reshape records rather than convert between serialisation formats.

Customers create, preview and manage Transformers through Transformer Studio in the Cloudflare dashboard or programmatically via the API, then attach them to a compatible NDJSON Logpush job that uses the same dataset. Each update to a Transformer creates a new version, so customers can review how it has changed over time.

Keeping transformation out of the critical path

Cloudflare's scale explains why the company was deliberate about where the SQL runs. The blog says the company's data pipeline ingests more than one billion events per second, and its delivery service, Bufstream, delivers more than 110 million Logpush records per second to customer destinations across hundreds of Kubernetes pods. Job volume follows a Pareto-like distribution: most jobs run at low volumes while a small number process millions of records per second.

An early prototype ran customer-defined WebAssembly Transformers in Cloudflare Workers, but invoking a Worker would have inserted a network call into the export path, risking transient timeouts and retries that reduce throughput and add latency. Instead, Cloudflare embedded Apache DataFusion — a Rust query engine built on Apache Arrow that already powers Cloudflare Basin — in-process within Bufstream. In a benchmark cited in the post, executing a prepared Transformer against a synthetic 5,000-row batch from a single Go goroutine took about 3 ms, roughly 1.6 million rows per second. High-volume jobs get per-job transform parallelism so batches can be transformed concurrently, reusing Bufstream's existing scheduling, batching and destination controls rather than adding a separate transformation tier.

Connecting Bufstream, written in Go, to an engine written in Rust was the remaining challenge. Batches arrive from Buftee, Cloudflare's in-house distributed message broker, as compressed Cap'n Proto records; Bufstream decompresses them, parses the eligible logs and builds typed Apache Arrow record batches before executing the transform. Arrow IPC exchanges those batches across a C ABI via cgo. Ownership across the boundary is explicit: Go passes a pointer to its input buffer to Rust only for the duration of the call, and Rust's output buffer is copied into Go-managed memory and then released, an approach Cloudflare says avoids cross-runtime memory-management bugs.

Why it matters

Pre-export transformation is largely a cost and hygiene exercise. Every record filtered before delivery avoids egress, SIEM ingestion and storage fees, and every redacted field reduces the chance of shipping sensitive data such as full client IPs to a third party. For teams normalising logs from several CDN providers into one schema, doing the work inside the export job removes a bespoke service from the infrastructure bill. The implementation is also a notable example of embedding an analytical SQL engine into an operational delivery path, and with Logpush now available on every self-serve plan, the capability is not reserved for enterprise customers.