JuiceFS rebuilds enterprise filesystem metadata layer for exabyte-scale AI training

What JuiceFS changed

JuiceFS has detailed a sweeping rework of the Enterprise Edition of its cloud-native filesystem, aimed at AI training environments moving from petabytes toward exabytes. According to an engineering post on dev.to, the redesign targets filesystems holding hundreds of billions of files and client counts that can grow from thousands into the millions. The company identifies four pressure points at that scale: the metadata engine, data lifecycle management, the data path, and the client and control planes. The post follows an earlier piece describing how autonomous driving firm Horizon Robotics runs JuiceFS Community Edition at exabyte scale.

A metadata service built around memory

At hundreds of billions of files, the post explains, metadata access hits three walls: hot directories that bottleneck a single node during peak training, added round trips for path resolution and file opens that slow jobs directly, and scale overhead from giant directories — the trash alone can hold tens of billions of entries — plus repeatedly queried global state such as mount configuration and capacity figures.

The answer is a purpose-built metadata service. All core data structures are handled by a single thread, which the company says makes operations atomic by default and avoids latency spikes from lock contention. Metadata is kept entirely in memory, so requests never touch disk and are processed in hundreds of microseconds, while Raft handles replication and automatic failover. One metadata process manages roughly 300 million files using 30 GiB of memory.

Because an in-memory design ties cost directly to memory usage, JuiceFS applied memory pools, manual memory management, serialization and compression of idle directories, and compact representations of small files. Average memory per file dropped from nearly 600 bytes to about 100 — around 27% of an HDFS NameNode's roughly 370 bytes per file, and about 3.7% of a CephFS MDS at roughly 2,700 bytes. The post also cites Alluxio at about 2,100 bytes and JuiceFS Community Edition at about 430 bytes per file.

Horizontal scaling through metadata zones

Beyond a single process, Enterprise Edition partitions the file tree across multiple metadata zones that clients assemble into one namespace. Subtrees are kept whole where possible, so most operations stay inside one zone and fewer distributed transactions are needed. Cross-zone migration uses links and references that remain transparent to POSIX paths, and extremely large directories can be split into virtual subdirectories so a single hot directory does not pin down one zone. Zone information is encoded in the high-order bits of each inode, and clients discover the topology themselves, so adding a zone requires no application changes. Each zone runs three instances — service, hot standby and cold standby — with the service and hot standby using different memory compression strategies, while snapshots and change logs are continuously persisted and can be backed up remotely.

Version milestones and client fan-out

The post traces the scale gains across releases. v5.2 brought multi-zone stability and lower memory use, with a production cluster now managing more than 50 billion files at millisecond-level metadata access latency. v5.3 raised the zone ceiling to 1,024 — enough for a single filesystem to exceed 500 billion files — and added automatic hotspot migration, which splits frequently accessed directories into smaller units migrated concurrently, plus periodic self-checks that repair intermediate states left by cross-zone moves. v5.4 turned to client scale: metadata nodes and network proxies can be deployed together, clients can connect to any node with requests for other zones forwarded internally by the cluster, and connection reuse supports up to a million client connections.

Deletion, recovery and reclamation

Deleting petabytes of data is treated as a first-class workflow, since reclamation is not just about freeing space: the system must also support recovery from accidental deletion, deletion traceability, and audits for orphaned objects. The trash is enabled by default with configurable retention, typically seven days, and restoring a file updates metadata only, without copying data. Since v5.3 the multi-zone trash records each file's original path so restores land in the right place, and v5.4 adds filtering of deleted files by time range and keyword. Reclamation runs on a unified background task framework in which the metadata service schedules work such as chunk compaction, trash cleanup and cross-cloud replication, executed concurrently across clients — so, the post argues, deletion capacity grows with the number of clients.

Why it matters

Training throughput is only as fast as the slowest metadata lookup, and at hundreds of billions of files the per-file memory footprint effectively decides hardware cost: cutting it from roughly 600 bytes to about 100 is an order-of-magnitude reduction in control-plane infrastructure. The design also signals where AI-era storage is heading — scale-out metadata that stays transparent to POSIX applications, client fan-out approaching a million connections, and lifecycle features like trash, recovery and reclamation built into the filesystem rather than bolted on afterward.