Making Real File Systems Faster with Applied Computational Storage
Salon IV
Wed Sep 14 | 10:35am
Abstract
The exploration of computation near flash storage has been prompted by the advent of network-attached flash-based storage enclosures operating at tens of gigabytes/sec, server memory bandwidths struggling to keep up with network and aggregate I/O bandwidths, and the ever-growing need for massive data storage, management, manipulation and analysis. Multiple tasks from distributed analytical/indexing functions to data management tasks like compression, erasure encoding, and deduplication are all potentially more performant, efficient and economical when performed near storage devices. The emerging NVMe Computational Storage standard requires real-world computational storage offload demonstrations to ensure the standard evolves in a useful direction that enables appropriate task offloads for end-user sites. Demonstrating the enablement of a standard’s-based ecosystem for offloading computation to near data storage is a valued contribution to the computing, networking, and storage communities. The goal of the joint Accelerated Box of Flash (ABOF) project (a collaboration between Eideticom, Nvidia, Aeon, SK hynix, and LANL) was to produce a first version of a network-attached computational storage system that allows a host application to directly leverage distributed and programmable computational elements in the ABOF without hiding any of the computation behind a block storage interface. Applications of interest include user-based computation, such as analytics acceleration for popular data formats, and kernel-based computation including functions common to file systems. Both use cases leverage distributed computational offloads near storage. The team has chosen to initially accelerate a commonly deployed kernel-based file system, ZFS, to appeal to the large ZFS community while also making it easy for vendors to deploy accelerated ZFS appliances and creating interesting business opportunities. The technical details of how this solution works, the useful artifacts produced, and the lessons learned from developing and testing this ABOF will be presented by the partners. Some background knowledge on NVMe, computational storage, and disaggregated storage would be beneficial to the audience.
Learning Objectives
- Learning how computational storage disrupts traditional storage architectures
- Learning about co-design for file systems and storage hardware with computational storage
- Learning about the useful open-source artifacts built as part of the development of the ABOF prototype
Related Sessions
Computational Storage
Closing the NVMe-CPCS Stack Gap: An End-to-End Implementation and Per-Workload Break-Even Characterization
The growth of AI and large-scale analytics workloads has made data movement a primary bottleneck of modern storage systems.Near-Data Processing (NDP) addresses this by executing portions of the workload close to where the data resides.
The protocol-level fragmentation that historically blocked NDP adoption has now closed.The SNIA Computational Storage Architecture and Programming Model, the SNIA Computational Storage API, and the NVMe Computational Programs (CP) and Subsystem Local Memory (SLM) command sets — all three now ratified — together define the architecture model, host-side programming interface, and wire protocol for compute offload to storage targets.
The remaining barrier is stack fragmentation.They define the interface but do not deliver an integrated implementation: no documented end-to-end open implementation has combined a SNIA CS API host library, a CP/SLM transport layer, and an NVMe-oF target for the CP/SLM command sets.Without it, break-even goes unmeasured, deployments unvalidated, and future CSD silicon has no conformant reference.
We close this gap with a two-layer reference stack.The host layer exposes the SNIA CS API to applications; the target layer handles CP/SLM commands as an NVMe-oF Computational Storage target.NVMe wire protocol connects them; either layer is replaceable at the protocol boundary without changing the other.
The host layer, libcs, implements the SNIA CS API.Compute requests dispatch via xNVMe (extended with CP/SLM support) over NVMe-oF TCP; transport and target details stay behind a pluggable backend.
The target layer, SPDK CPCS, runs on a JBOF CPU over NVMe-oF TCP; no specialized CSD silicon required.A virtual dispatch layer routes execution to built-in, eBPF, or passthrough runtimes under a single NVMe wire protocol.
We characterize per-workload break-even across aggregation, filtering, vector similarity, and compression by sweeping buffer size, selectivity, dimensionality, and input entropy, identifying per-workload thresholds at which target-side offload outperforms host-side execution end-to-end.
Our contribution is the first publicly described end-to-end NVMe-CPCS stack: libcs and SPDK CPCS connected via xNVMe, integrated on commodity hardware. The per-workload break-even methodology is reusable by any conforming NVMe-CPCS target, including future CSD silicon, without modifying the application layer above libcs.
Computational Storage
Leveraging CMB to Offload Memory Bandwidth for RDMA Direct Applications
High-performance SSDs use onboard DRAM for FTL tables and internal management with DRAM capacities increasing as SSD densities grow. As SSDs adopt newer DRAM technologies like DDR5, a large portion, often 40%-50%, of DRAM bandwidth remains underutilized by internal SSD operations. NVMe Controller Memory Buffer (CMB) and System Local Memory (SLM) features allow this unused DRAM bandwidth to be exposed to host applications. Leveraging CMB and SLM can unlock significant aggregate memory bandwidth across millions of SSDs in data centers, supporting high-performance workloads and new use cases. We'll discuss the architectural considerations and performance benefits of tapping into SSD DRAM bandwidth for enhanced resource efficiency in modern storage systems.
Computational Storage
Computation Offload Specification for AI Frameworks to Emit Partitioning Functions Alongside Large Data-centric Tasks
After surveying large-scale AI and HPC workloads, we have identified a set of principles and best practices widely used in partitioning large sparse tensors and graphs for parallel execution. We present these in the context of a roadmap for standardization of both Data and Compute offload from costly and power-hungry memory and compute tiers to NDP (near data processing) devices. Our work on standardizing partitioning operators will improve the portability and performance of the programming interface and protocol between AI framework backends that emit large tensor operations and the most power-efficient computational memory and storage devices that implement processing elements as low as subarray level and can only handle smaller chunks of data.
