Los Alamos is working to revolutionize how scientific data management is done, moving from large Petabyte sized files generated periodically by extreme scale simulations to a record and column based approach. Along the journey, the NVME Computational Storage efforts became a strategic way to help accomplish this revolution. Los Alamos has been working on a series of proof of concepts with a set of data storage industry partners and these partnerships have proven to be the key to success.

Gary Grider will kick off the talk with a brief background of the challenges currently faced in mass storage systems, and why it is so difficult for modern S3 based workloads to utilize them. Ben McClelland, who architected the system and did the bulk of the development, will introduce the Versity Gateway, an open-source high-performance object-to-file translation tool. He will explain that the Versity Gateway was written from scratch in Go, highlighting the benefits of this choice.

High-performance computing data centers supporting large-scale simulation applications can routinely generate a large amount of data. To minimize time-to-result, it is crucial that this data be promptly absorbed, processed, and potentially even multidimensionally indexed so that it can be efficiently retrieved when the scientists need it for insights.

Given LANL simulations can generate a Petabyte of data per time step with thousands to tens of thousands of time steps, data gravity is a huge concern at the lab. Performing analytics on this data to find and understand interesting features on simulation output is extremely expensive requiring movement of Petabytes of data and a data analytics platform with enough memory, not storage, to hold a full timestep (Petabyte).

The Grand Unified File Indexer (GUFI) is a state of the art file/storage system indexing too that offers both user and storage administrator access in a way that each user can only see the metadata for the files they have access to.  Imagine having an exabyte of data that is in many file system trees in a trillion files in10 billion directories that uses POSIX permissions (UID/GID/rwxrwxrwx) with inheritance.