Apache Kafka's KIP-1279 embeds cross-cluster mirroring directly in the broker

Apache Kafka is gaining native cross-cluster mirroring under KIP-1279, a change that moves replication between clusters out of external tooling and into the broker itself. According to a Red Hat Developer article that reached the front page of Hacker News, a destination broker fetches committed records from a source cluster using the same fetch protocol followers already use internally, then appends them to local partition logs as raw, unmodified batches — preserving offsets, compression, and consumer group state.

From external tooling to broker-embedded replication

Kafka has always handled data movement within a single cluster well: leaders replicate to followers and consumers pull from any replica with minimal overhead. Copying data between clusters — required for geographic distribution, compliance boundaries, team isolation, or version segregation — has been harder. Since Kafka 2.4, the standard tool has been MirrorMaker 2, which runs as a set of Kafka Connect workers that consume from a source cluster and produce to a destination.

Cluster mirroring removes that layer. Mirror fetcher threads run inside the broker process, so there are no Connect workers to provision, monitor, or scale. A single CLI command, kafka-cluster-mirrors.sh --create, establishes a mirror; --start begins replicating topics. The full lifecycle is managed through the same Admin API used for topics and consumer groups.

Byte-for-byte copies and identical offsets

Two properties define the design. Compressed batches are replicated as raw bytes, so a gzip, snappy, lz4, or zstd batch arrives at the destination in its original form — no decompress/recompress round trip, and the producer's compression choice survives. Offsets are identical across clusters, including gaps left by log compaction, so consumer groups fail over without offset translation: the committed offset on the source is the committed offset on the destination.

Stopping a mirror triggers a deterministic sequence: fetchers are removed, the last mirror epoch is persisted, the leader epoch is bumped, pending transactions are aborted, and a control record expires all producer state. The partition then becomes writable on the destination, with no external coordination or offset queries.

The feature also copes with unclean leader elections on the source. The destination enters a recovery state and waits for all assigned replicas — not just in-sync ones — to converge to the truncated offset before resuming, keeping the two logs consistent even when the source elects a leader with an incomplete log. The article's comparison table lists source compatibility back to Kafka 2.1, versus 2.0+ for MirrorMaker 2.

Three components inside the broker

MirrorMetadataManager is the orchestrator. Running on every broker, it watches the KRaft metadata log and drives state transitions — create, start, stop, pause, resume, recover, delete — when the controller writes a MirrorTopicStateChangeRecord. It also holds an Admin client connection to the source and, every 60 seconds by default, refreshes source metadata: discovering topics matching configured include/exclude patterns, syncing topic configurations, fetching consumer group offsets, and verifying the source cluster ID has not changed. That last check prevents silent data corruption if someone repoints a mirror at a different cluster.

ClusterMirrorCoordinator handles persistence, following the same coordinator pattern as the group and transaction coordinators. Each mirror partition's state is a key-value record in an internal compacted topic, __mirror_state, with defaults of 50 partitions and replication factor 3, guarded by optimistic concurrency control through leader epoch and state epoch fencing.

MirrorFetcherThread performs the actual replication. It extends Kafka's AbstractFetcherThread — the same base class behind intra-cluster replication — and keeps a dedicated NetworkClient with per-mirror authentication credentials, isolating SASL/SSL contexts between mirrors. Threads are keyed by fetcher ID, source broker endpoint, and mirror name, enabling fine-grained load balancing and fast response to source leader changes.

The broker additionally takes care of metadata discovery, configuration syncing, consumer group offset syncing, and ACL propagation. Bandwidth is controllable on both ends: the destination enforces a configurable replication rate limit, while mirror fetch traffic on the source looks like ordinary consumer requests, so existing client quota mechanisms apply unchanged.

A defined partition lifecycle

Mirror partitions progress through a state machine, and any state can move to FAILED on error. LOG_ALIGNMENT first aligns the local log with the source: a brand-new mirror truncates to zero and copies everything, while a resumed one mirrors only the delta, using stored last-mirror offset and epoch values to resolve divergence. EPOCH_FENCING then bumps the local leader epoch by 10, with a re-bump threshold of 3 — without it, destination consumers could initialize with a committed epoch from the source that exceeds the local one and reject the leader.

The Red Hat article walks through two practical scenarios, disaster recovery and cluster migration, and closes with a video demo.

Why it matters

Kafka underpins a large share of real-time data infrastructure, and the painful parts are rarely within-cluster replication — they are disaster recovery, migration, and multi-cluster topologies. KIP-1279 folds those workflows into the broker, eliminating Connect clusters, offset translation topics, and the coordination that made cross-cluster setups fragile. Identical offsets and untouched batches mean consumers and downstream tooling behave the same way after a failover, exactly the guarantee these setups previously lacked. It also continues a broader pattern of Kafka absorbing capabilities that once required a surrounding ecosystem. The details here come from a single vendor article, so specifics may shift as the proposal matures.