Open-source compressor bzip3 claims big ratio wins over bzip2 and xz in Perl corpus test

What bzip3 is

bzip3, an open-source file compressor that surfaced on the Hacker News front page via its GitHub repository, positions itself as a modern successor to bzip2. Like its predecessor it is strongest on text and source code, but according to the project's README it replaces the old internals with three newer building blocks: an order-0 context mixing entropy coder, a Burrows-Wheeler transform implemented with suffix arrays, and a run-length encoding stage combined with a Lempel-Ziv+Prediction pass that blends LZ77-style string matching with PPM-style context modeling.

The runtime code is copyright Kamila Szewczyk, and the project states it is licensed under LGPLv3 only, with no dual licensing. The suffix-array engine comes from Ilya Grebnov's libsais library under Apache 2.0, and the post-coder for the Burrows-Wheeler output derives from a public-domain implementation by Ilya Muravyov. It builds through a standard autotools chain and is also available through Homebrew on macOS.

The benchmark numbers

The repository's headline test is a corpus the author assembled by downloading every Perl 5 release from CPAN — 262 tarballs — and bundling them into a single archive, then compressing it with several tools:

• xz (LZMA) at level 9 with 16 threads: 2,056,645,240 bytes
• bzip2 at level 9: 3,441,163,911 bytes
• bzip3 with the 256 block setting: 1,001,957,587 bytes
• bzip3 with the 511 block setting: 546,456,978 bytes
• zstd at level 16 with 12 threads: 3,076,143,966 bytes

The largest bzip3 configuration therefore produced roughly a quarter of xz's output and about one-sixth of bzip2's. Wall-clock compression time was around seven minutes for both bzip3 runs, quicker than xz at 12 minutes and bzip2 at 17 minutes, and close to zstd's 6.5 minutes. Decompressing to a hard drive, bzip3 in parallel mode took 4:06, ahead of xz at 4:40 and slightly behind zstd at 3:51, while bzip2 needed 9:22.

A second test first pushed the archive through lrzip for long-range deduplication. That pipeline landed at 60,672,608 bytes with bzip3, versus 64,774,202 bytes for lrzip with LZMA and 75,685,065 bytes for lrzip with bzip2.

Two caveats deserve emphasis. A corpus of hundreds of successive releases of the same codebase is unusually redundant, which flatters compressors that can model similarities across a large window. And the figures are the author's own rather than an independent evaluation, although the README points readers to a third-party benchmark by powturbo covering bzip3, bzip2, bsc and others.

Practical trade-offs

The ratio gains carry a memory cost. In the runs above, the 511-block configuration reported about 12 GB of peak memory and the 256-block run about 18 GB, while zstd used 687 MB. Throughput also depends heavily on the build: the project cites up to roughly 17 MiB/s compression and 23 MiB/s decompression per thread for clang 13 builds on x86-64 Linux, with slower results expected on Windows and 32-bit targets. Testing has covered ten architectures, including x86-64, aarch64, ppc64le, mips, sparc and s390x.

The README also carries an unusually blunt disclaimer. Given the complexity of the algorithms and rare special cases in the code, the author writes, the possibility of remaining bugs cannot be ruled out, and users should not compress data unless they are prepared to accept a small chance that it will not be recoverable. The same section stresses that bzip3 and its library have been carefully constructed and extensively tested.

Why it matters

bzip2 has been a fixture of Unix pipelines and archive formats for decades, and it is showing its age: slow, single-threaded and weak by modern standards. bzip3 is an attempt to keep the Burrows-Wheeler lineage relevant by rebuilding it around suffix arrays, context mixing and parallel processing, and on this evidence it delivers improvements large enough to beat even xz on size at competitive speeds. For developers archiving text-heavy material such as source trees, logs or documentation, that combination is worth testing — with eyes open to the multi-gigabyte memory footprint, the project's relative youth and the fact that the headline numbers come from its own author.