Apache Iceberg's maintenance gap is the real story in managed data lakes

Iceberg settled the format, not the operations

A guide published on dev.to, framed as a look ahead at the managed data lake in 2027, opens with a blunt premise: Apache Iceberg is now the standard table format for production data lakes, every major engine reads and writes it natively, and the catalog ecosystem has standardized on REST, with data held on commodity object storage the owner controls. The format question is settled, in other words — the remaining problem is operational.

Iceberg, the guide explains, deliberately separates the table format from the system that keeps tables healthy. It ships maintenance procedures — rewrite_data_files, expire_snapshots, remove_orphan_files, rewrite_manifests — but no logic for deciding when to run them, in what order, or how aggressively. Left alone, the author argues, every Iceberg table degrades: small files pile up, snapshot metadata bloats, physical sort orders drift away from how queries actually filter, orphans inflate storage costs, and performance decays until something visibly breaks.

How unmanaged lakes degrade

The first failure mode is small files. Streaming writers such as Flink, Spark Structured Streaming, Kafka Connect, RisingWave and CDC connectors size output by checkpoint interval, not by efficient read size. The guide's worked example: a Flink job committing every 60 seconds across 100 active partitions produces 144,000 files a day at roughly 1–5 MB each, against an efficient-read target of 128–512 MB. Every file carries per-file overhead — S3 GET requests at $0.0004 per thousand, Parquet footer parses, engine task slots — so query compute scales with file count rather than data volume. Batch tables degrade over months; a high-throughput streaming table can degrade within hours of going live.

Snapshots accumulate next. Every commit creates one, which is what enables time travel and rollback, but a table committing every five minutes adds about 288 snapshots a day. Past roughly 1,000–2,000 retained snapshots, planning slows noticeably, and the guide reports production metadata. files exceeding 400 MB. It suggests 3–7 days of retention for typical analytics workloads and 30–90 days for regulated industries, with the planning-time trade-off made explicit.

Manifest fragmentation compounds the planning cost. Each commit appends at least one manifest entry, and months of streaming writes leave hundreds or thousands of small manifests that planners must read before opening any data. The guide argues teams routinely misdiagnose this as an engine problem — tuning memory, scaling coordinators, upgrading versions — when rewriting manifests into fewer, larger files is a high-impact operation few teams automate. Server-side scan planning, introduced in Iceberg 1.11, makes lean manifests more valuable still.

Orphan files are the silent budget line. Failed Spark stages, interrupted compaction, aborted writes and OOM kills during large rewrites leave files that no snapshot references. They are invisible to queries, to Iceberg metadata and to data-quality checks, and can only be found by listing storage prefixes and diffing them against the catalog. According to the guide, production orphan sweeps routinely reclaim 20–40% of a lake's storage; for a 100 TB lake at S3 Standard pricing of $0.023 per GB per month, that is $460–$920 a month of dead weight.

Sort-order drift is the last piece. Bin-pack compaction merges small files but does nothing for layout. A transactions table appended in transaction_id order but queried by customer_id and event_date cannot skip files via Parquet min/max statistics, so a single-customer query scans everything; re-sorted by the filtered columns, the same data lets engines eliminate 95% or more of scan volume. Because query patterns shift as new consumers arrive, manual sort tuning does not scale across hundreds of tables.

The control-plane precedent

The guide points to two companies that paid the full cost of solving this. Netflix built four internal services — Autotune for compaction strategy selection, Polaris for catalog management, janitor services for garbage collection, and Metacat for cross-service observability — each maintained by dedicated teams over multiple years. Google instead engineered automatic compaction and garbage collection directly into BigLake, so its managed Iceberg tables stay healthy however write volumes and query patterns shift. The guide's pitch for 2027 is that this investment no longer needs replicating: “managed” now means renting exactly this control plane, whether a team runs 50 tables or 5,000.

Why it matters

Iceberg plus REST catalogs delivers the portability the lakehouse model promised: one copy of data, many engines, no format lock-in. But this guide locates the real running cost of a lake in maintenance — planning latency, scan volume and storage bills all trace back to compaction, snapshot expiry, manifest hygiene and sort order, none of which the format handles by itself. For cloud data teams evaluating platforms, the checklist is concrete: which maintenance loops run automatically, what snapshot retention ships by default, how often orphan sweeps compare storage against the catalog, and whether sort orders adapt as query patterns change. The difference between a cheap lake and an expensive one is no longer the table format. It is who, or what, runs the maintenance.