Why concatenating .tar.gz files with cat produces a broken archive

The failure mode

When a project generates several .tar.gz archives that need to become one file, piping them through cat feels like the obvious move. Developer Alex Chan tried exactly that and found the result broken: the merged file decompressed, but only the first archive's contents appeared. In a blog post that reached the Hacker News front page, Chan digs into why, and the explanation runs through magnetic tape hardware and 1990s patent law.

tar carries its tape-drive heritage

tar, short for tape archive, packs multiple files plus their metadata — names, timestamps, directory layout — into a single stream. According to Chan, the format still reflects the physical limits of the magnetic tapes it was designed for: data had to be read sequentially, writes were append-only, and tapes used fixed block sizes.

Internally, an archive stores each file as a header block carrying metadata such as the filename and size, followed by data blocks holding the contents. The archive then closes with at least two blocks filled entirely with zeroes. Those blocks act as an end-of-file marker: a reader treats everything after them as empty tape and ignores it. GNU tar can be told to keep reading with its --ignore-zeros flag, but as Chan notes, archives intended for general distribution should open correctly with default settings.

That structure also explains a few familiar tar quirks. File sizes must be declared in the header before any data is written — Chan describes repeatedly forgetting to set tarinfo.size when using Python's TarFile.addfile, which produces empty entries. Archives may also legitimately contain duplicate filenames: since tape could not be rewritten, updating a file meant appending a new version with the same name, and extraction lets the later copy overwrite the earlier one.

gzip concatenates because it has no EOF marker

gzip, by contrast, compresses a single stream losslessly. Its design, laid out in RFC 1952, was shaped less by hardware than by legal and memory constraints of its era: it was written as a patent-free replacement for compress, whose LZW algorithm was patent-encumbered, it had to run in small, bounded memory so data could be processed in continuous chunks, and it needed to be portable across CPUs and operating systems.

A gzip file is a series of one or more members, each with its own header, compressed data, and a trailer holding a CRC32 checksum and the uncompressed size. There is no end-of-file marker, and — as Chan emphasises — members are not analogous to files. Tools treat them as one continuous data stream, so you cannot list or extract them individually. The practical consequence: concatenating separate .gz files with cat produces a valid file, and gunzip returns the combined uncompressed data as a single stream.

Where the two formats collide

A .tar.gz file layers the two formats: tar turns a directory tree into a stream, and gzip shrinks that stream. Both read sequentially, which is why the combination streams well over a network — files can be processed as they arrive, before the full download completes.

Merging .tar.gz files with cat fails at the seam between them. The gzip layer accepts the concatenated members and decompresses them without complaint, but tar then reads the resulting stream, encounters the first archive's zero-block EOF marker, and stops. Everything after the first archive is silently discarded, even though gzip would have carried on happily.

The fix: rebuild the archive

Rather than splicing bytes together, Chan combines archives member by member using Python's tarfile module. Each input is opened in r:gz mode, its members are iterated via getmembers, and every entry is copied into a fresh archive opened in w:gz mode. The result is a single valid archive with one EOF marker and no reliance on non-default reader flags. It is more code than a simple concatenation, but it is the only approach that yields an archive every tool can read out of the box.

Why it matters

The failure is silent, which is what makes it dangerous. A concatenated .tar.gz decompresses without any error, so pipelines and quick spot checks can pass while everything after the first archive goes missing — a real hazard for anyone generating archives in parts or merging build artifacts. Understanding the internals of both formats also demystifies everyday tar behaviour, from mandatory upfront file sizes to duplicate-name overwrites. And while --ignore-zeros offers an escape hatch for archives with trailing data, shipping files that only unpack correctly with a special flag is a fragile trade-off most projects should avoid.