pNFS Lattice: Decoupling Metadata, State and Control in Parallel Filesystems

Traditional parallel filesystems scale the data path effectively, but metadata and control plane scalability remain constrained by tightly coupled architectures, static metadata services and complex failover models. This talk presents the architecture behind pNFS Lattice, a new pNFS-based parallel filesystem design developed by PEAK:AIO and Los Alamos National Laboratory that separates the filesystem into independently scalable planes:

* State plane
* pNFS-MDS service plane
* Metadata catalog plane
* Data plane

The design goal is to remove fixed metadata scaling boundaries and allow metadata services to become ephemeral, elastic and dynamically instantiated across existing infrastructure.

Rather than relying on permanently assigned metadata nodes, pNFS Lattice allows lightweight user-space metadata services to be started on demand across generic infrastructure as workload requirements change. In practice, this enables deployments to scale from a small number of metadata services to potentially hundreds or thousands without requiring dedicated metadata islands or tightly coupled failover pairs.

The talk will discuss the architectural separation between state management, metadata servicing and metadata persistence, including how clustered state handling and distributed metadata catalogs are used to decouple metadata processing from metadata storage.

Topics covered include:

* Separation of metadata services
* Clustered state management for distributed metadata services
* Distributed metadata catalog design
* Dynamic metadata service instantiation
* Implications for resiliency and failover
* Metadata scaling
* Operational impact of ephemeral metadata services
* Preserving pNFS compatibility while removing traditional metadata constraints

Rather than proposing another tightly integrated storage stack, pNFS Lattice explores how pNFS metadata services, state handling and metadata persistence can scale independently using loosely coupled service architectures. The project is launched as open source by PEAK:AIO and LANL.