SkyhookDM: An Arrow-Native Storage System

Learn what is happening in NVMe to support Computational Storage devices. The development is ongoing and not finalized, but this presentation will describe the directions that the proposal is taking. Kim and Stephen will describe the high level architecture that is being defined in NVMe for Computational Storage. The architecture provides for programs based on a standardized eBPF. We will describe how this new command set fits within the NVMe I/O Command Set architecture. The commands that are necessary for Computational Storage will be described. We will discuss a proposed new controller memory model that is able to be used for computational programs.

Kim Malone

Storage Software Architect

Intel

Stephen Bates

CTO

Eideticom

Standardized computational storage services are frequently touted as the Next Big Thing in building faster, cheaper file systems and data services for large-scale data centers. However, many developers, storage architects and data center managers are still unclear on how best to deploy computational storage services and whether computational storage offers real promise in delivering faster, cheaper – more efficient – storage systems. In this talk we describe Los Alamos National Laboratory’s ongoing efforts to deploy computational storage into the HPC data center. We focus first on describing the quantifiable performance benefits offered by computational storage services. Second, we describe the techniques used at Los Alamos to integrate computational storage into ZFS, a fundamental building block for many of the distributed storage services provided for Los Alamos scientists. By developing ZIA, the ZFS Interface for Accelerators, Los Alamos is able to embed data processing elements along the data path and provide hardware acceleration for data intensive processing tasks currently performed on general purpose CPUs. Finally, we describe how computational storage is leading to a fundamental re-architecture of HPC platform storage systems and we describe the lessons learned and practical limitations when applying computational storage to data center storage systems.

Brad Settlemyer

Principal Staff

NVIDIA

There is a new architectural approach to accelerating storage-intensive databases and applications which take advantage of new techniques that dramatically accelerate database performance, improve response times and reduce infrastructure cost at massive scale. This new architecture efficiently stores fundamental data structures increasing space savings—up to 80%—over host- and software-based techniques that cannot address inherent inefficiencies in today’s fastest SSD technology. With novel data structures and algorithms it is a unique data processing approach to fundamentally simplify the storage stack.

Tony Afshary

Sr. Director, Product Line Management

Pliops

The SNIA Computational Storage TWG is driving forward with both a CS Architecture specification and a CS API specification. How will these specification affect the growing industry Computational Storage efforts? Learn what is happening in industry organizations to make Computational storage something that you can buy from a number of vendors to move your computation to where your data resides. Hear what is being developed in different organizations to make your data processing faster and allow for scale-out storage solutions to multiply your compute power.

Bill Martin

SNIA Technical Council Co-Chair/Principal Engineer, SSD IO Standards

Samsung

With the ever-increasing dataset sizes, several file formats like Parquet, ORC, and Avro have been developed to store data efficiently and to save network and interconnect bandwidth at the price of additional CPU utilization. However, with the advent of networks supporting 25-100 Gb/s and storage devices delivering 1, 000, 000 reqs/sec the CPU has become the bottleneck, trying to keep up feeding data in and out of these fast devices. The result is that data access libraries executed on single clients are often CPU-bound and cannot utilize the scale-out benefits of distributed storage systems. One attractive solution to this problem is to offload data-reducing processing and filtering tasks to the storage layer. However, modifying legacy storage systems to support compute offloading is often tedious and requires an extensive understanding of the internals. SkyhookDM introduces a new design paradigm for building computational storage systems by extending existing storage systems with plugins. Our design allows extending programmable object storage systems by embedding existing and widely used data processing frameworks and access libraries into the storage layer with minimal modifications. In this approach data processing frameworks and access, libraries can evolve independently from storage systems while leveraging the scale-out, availability, and failure recovery properties of distributed storage systems. SkyhookDM is a data management system that allows data processing tasks to the storage layer to reduce client-side resources in terms of CPU, memory, and network traffic for increased scalability and reduced latency. On the storage side, SkyhookDM uses the existing Ceph object class mechanism to embed Apache Arrow libraries in the Ceph OSDs and uses C++ methods to facilitate data processing within the storage nodes. On the client-side, the Arrow Dataset API is extended with a new file format that bypasses the Ceph filesystem layer and invokes storage side ceph object class methods on objects that make up a file in the filesystem layer. SkyhookDM currently supports Parquet as its object storage format but support for other file formats can be added easily due to the use of Arrow access libraries.

Jayjeet Chakraborty

IRIS-HEP Fellow

IRIS-HEP and CROSS, UC Santa Cruz

Scientific data is commonly represented by static datasets. There is a myriad of sources for such data: snapshots of the Earth, as captured by satellites, sonars, and laser scanners; the output of simulation models; the aggregation of existing datasets, and more. Some of the problems faced by consumers of that data relate to data transformation (e.g., harmonizing data layout and format), inefficient data transfer over the network (for instance, the creation of derivative data leads to even larger datasets), expensive data processing pipelines (that often preprocess data in advance, hoping that the produced data will be consumed at some point by the applications), CPU occupation, and difficulty to track data lineage (especially when data processing scripts are lost for good). This talk will present a pluggable extension to the HDF5 file format, used extensively by the scientific community, to enable the attachment of user-defined functions that help solve many of the aforementioned problems. Dubbed HDF5-UDF, the project allows scripts written in Lua, C/C++, and Python to be attached into existing HDF5 files. Such scripts are disguised as regular datasets that, once read, execute the associated source code and produce data in a format that existing applications already know how to consume. HDF5-UDF has been designed from scratch to enable seamless integration with hardware accelerators. Its memory allocation model allows for input datasets (i.e., datasets that a user-defined function depends on) to be allocated in device memory (such as a SmartSSD) and the backend architecture enables the incorporation of different programming languages and compiler suites (such as Xilinx's Vitis compiler or Nvidia's nvcc). The talk will present current and future work in the context of computational storage. To date, HDF5-UDF is tied to HDF5, but it is feasible to port this work to other formats such as TIFF (and GeoTIFF). Some guidance will be provided for those wishing to experiment with different data formats. Last, but not least, HDF5-UDF is open source. All examples shown in the presentation are readily available for downloading, modification, and testing. Please visit https://github.com/lucasvr/hdf5-udf and https://hdf5-udf.readthedocs.io for more details on the project covered by this talk.

Lucas Villa Real

Research Software Engineer

IBM Research