TurboKV: embedded Rust key-value store with atomic batches, range scans and compaction

TurboKV, an open-source embedded key-value store written in Rust, reached the front page of Hacker News via its GitHub repository (kingroryg/turbokv). According to the project's README, the database is fully asynchronous, runs in-process rather than as a separate server, and combines atomic batched writes, ordered range and prefix scans, tunable durability, compression and background compaction. The current release is version 0.6 and installs as a Cargo crate alongside Tokio.

An LSM-style engine with one distinctive detail

The documented component set is the familiar log-structured toolkit: an in-memory memtable that rotates and flushes in the background, a write-ahead log for crash recovery, immutable on-disk SSTables, a manifest, and a block cache for decompressed blocks. Compaction happens in the background. One unusual detail is the persisted Bloom-filter format, which leans on hardware AES instructions; the README tells builders to enable AES plus SSE2 on x86 targets, or AES plus NEON on ARM, or to compile with target-cpu=native when the binary will only ever run on the same CPU family.

Defaults are opinionated. Every durability preset starts with a 64 MiB memtable, a 64 MiB block cache and LZ4 compression, with Snappy, Zstd and no compression as alternatives. Compression applies only to newly written SSTables; existing tables keep whatever encoding they were created with.

Three durability presets

Durability is chosen when the database opens, and the README defines three named presets rather than a wall of knobs:

• DbOptions::fast() acknowledges writes once they are visible in memory, with no WAL at all. Aimed at caches and reproducible data.
• DbOptions::durable(), the recommended default, appends to the WAL without syncing on every write and survives process crashes.
• DbOptions::paranoid() waits for a completed sync_all on the WAL group before acknowledging a mutation. This is the strongest mode, bounded by filesystem and device guarantees.

Individual fields can still be tuned — wal_enabled, sync_writes, memtable_size, block_cache_size and compression — and contradictory combinations, such as requesting write syncs with the WAL disabled, are rejected at open time.

Writes, and what atomic actually means here

write_batch is the headline mutation API: readers see either the state before the batch or the fully applied batch, and the last operation wins for duplicate keys. By contrast, insert_many is explicitly labelled a bulk convenience API, not an atomic visibility transition — a distinction many stores leave undocumented.

Point operations follow byte-oriented semantics. Keys and values are arbitrary byte sequences, an empty value is valid data distinct from a missing key, and remove writes a tombstone even for keys that do not exist. Two caveats from the README stand out: with the WAL enabled, a single record or batch must fit within the WAL's u32 payload length; and a failed or cancelled mutation may already have reached the log, so callers are advised to check the key or reopen the database before retrying a non-idempotent operation.

Scans, snapshots and iterator hygiene

Keys are ordered lexicographically by raw bytes. Eager APIs such as range and scan_prefix allocate the entire result up front, while the streaming variants range_iter and scan_prefix_iter yield guard items whose values load lazily, with corruption surfaced as errors while the iterator advances. The streaming interface supports counting, keys-only collection, pair collection, and offset/limit pagination that traverses skipped entries without copying their memtable values.

The README also documents two costs that are easy to miss: every scan captures a coherent point-in-time snapshot, but creating one can freeze a nonempty active memtable, meaning frequent small scans may increase later flush work; and open iterators pin their snapshot readers, so they should be dropped promptly.

Ownership and shutdown

A single Db or Engine exclusively owns its data directory, and dropping a handle is explicitly not a clean shutdown. Callers must use close() or close_with_status(), while flush() drains pending writes and installs new SSTables and the manifest.

Why it matters

Embedded stores are the plumbing of most server software, and Rust's ecosystem already has established options in this space. TurboKV's pitch is a recognisable LSM design paired with unusually explicit documentation of its durability boundaries — three clearly named presets instead of dozens of interacting flags — plus careful treatment of scan-snapshot semantics and shutdown contracts. At version 0.6 it is young, and the README itself flags sharp edges such as WAL payload limits and non-clean handle drops. For teams that need ordered scans and atomic batches without operating a separate database process, those caveats read less as deterrents and more as evidence of honest engineering, and the project is worth tracking as it matures.