MarFS Multi-Component: Scalable Long-Term Storage on Commodity Disk

As the bandwidth and capacity requirements of HPC long-term storage outpace the capabilities of affordable tape solutions, many now look to disk media as a more economically viable approach. These trends have led to the creation of MarFS, an open source file system which provides a near-POSIX interface and data/metadata separation atop varied underlying storage implementations, whether object-based or otherwise.

The MarFS Multi-Component system is one such implementation. Distributing data and parity blocks across multiple remote systems, Multi-Component provides a flexible and robust means of storing MarFS data objects in a parallel manner. Combined with ZFS storage pools to form a multi-tier erasure stack, the system provides significant reliability, even atop commodity disk media.

This presentation will focus on the specifics of the MarFS Multi-Component implementation as well as lessons learned from its initial deployment in a production setting.