Conviva replaced mmap with io_uring in its Rust query engine and it got slower

The setup at Conviva

According to a Conviva engineering post that surfaced on the front page of Hacker News, the company analyzes trillions of events a day to diagnose end-user experience, using an event and pattern analysis engine built on DataFusion, Arrow, Rust, Rayon and Tokio. Raw events are transformed, encoded in a mostly-numeric proprietary format and stored in the cloud, then copied to local NVMe and read as large Arrow IPC files of roughly 3–5 GB. A typical query touches six columns across eight batch files, about 1.6 GB per batch and roughly 13 GB per day of data.

Arrow IPC was chosen because its in-memory and on-disk layouts are identical, keeping decode cost minimal, and because arrow-rust natively supports zero-copy reads through mmap. That made mmap the obvious default: huge numbers of files could be lazily read without the application managing memory itself.

Where mmap broke down

At lighter loads mmap worked well, serving queries from raw events in seconds. Under real concurrent load, p95 latency jumped from around 30 seconds to more than 150 seconds, rows scanned per core fell sharply even after accounting for concurrency, the OS page cache shrank as pods consumed more private memory, and adding pods made performance worse rather than better.

The test hardware was a 192-core box with about 750 GB of RAM, using either a two-drive NVMe LVM stripe with a roughly 5.5 GB/s fio ceiling or a 32-drive RAID-0 with about 21 GB/s. The investigation ran on kernel 5.15, with 6.x in production. Conviva notes Linux 6.4+ gained a fast per-VMA lock, while earlier kernels fall back to a slower mmap lock.

To isolate the effect, the team compared one pod versus four pods on the same host under identical query load. For 14-day queries, long enough to fill the page cache, the single pod won by a wide margin: 41% faster at max and more than 20% faster at p95. The diagnosis is that the mmap page cache is implicit shared state at the host level, one cache, one lock hierarchy, one eviction policy shared by every pod, so the pods were fighting for cache rather than CPU. perf record on the same run showed 100% lock contention in the kernel.

A page-fault storm, in numbers

During a stressful run, resident memory climbed to 98.91% of RAM, at which point the kernel began evicting pages that were still needed. Major faults, those requiring physical I/O, spiked to 571/s, 1352/s and 975/s in successive samples. Minor faults ran sustained in the millions per second, peaking around 2.35 million. The post estimates an uncontended minor fault at roughly 0.5–1 microseconds, putting two million per second close to mmap's ceiling, and notes each fault touches a cache line via atomics, which thrashes L1/L2 for an application relying on large cache-resident lookup tables.

Virtual address space had grown to about 3 TB from mapping so many Arrow files, and context switches exceeded two million per second versus 14,000 on a warm-cache run, a 150x difference, as threads constantly blocked on faults and were rescheduled.

The kernel, not the disks, was the bottleneck

perf top made the contrast stark. On a cold run, the kernel's __filemap_add_folio, which inserts a page into the page cache, consumed 78% of samples while actual query code dropped to about 5%; on a warm run the query code took roughly 45%. Off-CPU analysis with bpftrace attributed 30.9% of blocked time to futexes, threads queued behind another thread's page-fault handler, 29.3% to preemption by kernel readahead work, and only 6.9% to actual disk I/O.

The gap to the hardware was large. fio with the io_uring engine on the 32-drive array measured 20.2 GiB/s (21.7 GB/s) with all drives near full utilization, while mmap peaked at 3.44 GB/s, about 16% of what the storage could deliver. Conviva points out the problem is not unique to mmap; any buffered I/O path can hit similar page-cache and lock contention.

The io_uring twist

Conviva then moved to io_uring, drawn partly by its promise of direct user I/O that can bypass the page cache. The outcome is stated plainly in the post's title: it got slower. The published analysis documents in depth why mmap collapsed under load, with the io_uring results and a promised part two on managing page faults carrying the story forward.

Why it matters

For data engineers, the post is a useful corrective to two assumptions: that mmap's zero-copy convenience scales with concurrency, and that a faster I/O interface like io_uring automatically translates into faster queries. Conviva's measurements show a query engine can spend the vast majority of its CPU inside kernel page-cache bookkeeping while NVMe sits mostly idle, and that horizontal scaling can actively hurt when the contended resource is a shared host-level cache. The practical guidance is to profile cold versus warm runs, watch major and minor fault rates plus off-CPU time, and treat the I/O path as a concurrency decision rather than a pure throughput number.