Leveraging Computational Storage for Simulation Science Storage System Design
Salon V
Tue Sep 19 | 4:05pm
Abstract
High-performance computing data centers supporting large-scale simulation applications can routinely generate a large amount of data. To minimize time-to-result, it is crucial that this data be promptly absorbed, processed, and potentially even multidimensionally indexed so that it can be efficiently retrieved when the scientists need it for insights. Currently, despite a transition from HDDs to using all-flash based storage systems in hot storage tiers for a boost in raw storage bandwidth as many recently deployed systems have done, bottlenecks still exist due to legacy software, severe server CPU and memory bandwidth limitations for certain data-intensive operations, and excessive data movement. Computational storage, with its ability to map and distribute storage functions to various computing units along the data processing path, offers opportunities to overcome existing storage system bottlenecks to vastly improve performance and cost. In this talk, we will discuss various computational storage efforts carried out at Los Alamos National Laboratory in collaboration with partners including Aeon Computing, Eideticom, NVIDIA, SK hynix, and Seagate. We will explore topics like transparent ZFS I/O pipeline offloads, analytics acceleration with flash key-value based storage devices, and in-drive SQL like query processing in an erasure coded data lake tier. We will then conclude by discussing lessons learned, next steps, and the need for an open, standards-based approach for computational storage in the form of an object storage system to ease development, adoption, and innovation.
Learning Objectives
- Computational storage use-cases for HPC applications
- Example computational storage implementations
- Lessons learned from exploring computational storage implementations
Related Sessions
Computational Storage
Closing the NVMe-CPCS Stack Gap: An End-to-End Implementation and Per-Workload Break-Even Characterization
The growth of AI and large-scale analytics workloads has made data movement a primary bottleneck of modern storage systems.Near-Data Processing (NDP) addresses this by executing portions of the workload close to where the data resides.
The protocol-level fragmentation that historically blocked NDP adoption has now closed.The SNIA Computational Storage Architecture and Programming Model, the SNIA Computational Storage API, and the NVMe Computational Programs (CP) and Subsystem Local Memory (SLM) command sets — all three now ratified — together define the architecture model, host-side programming interface, and wire protocol for compute offload to storage targets.
The remaining barrier is stack fragmentation.They define the interface but do not deliver an integrated implementation: no documented end-to-end open implementation has combined a SNIA CS API host library, a CP/SLM transport layer, and an NVMe-oF target for the CP/SLM command sets.Without it, break-even goes unmeasured, deployments unvalidated, and future CSD silicon has no conformant reference.
We close this gap with a two-layer reference stack.The host layer exposes the SNIA CS API to applications; the target layer handles CP/SLM commands as an NVMe-oF Computational Storage target.NVMe wire protocol connects them; either layer is replaceable at the protocol boundary without changing the other.
The host layer, libcs, implements the SNIA CS API.Compute requests dispatch via xNVMe (extended with CP/SLM support) over NVMe-oF TCP; transport and target details stay behind a pluggable backend.
The target layer, SPDK CPCS, runs on a JBOF CPU over NVMe-oF TCP; no specialized CSD silicon required.A virtual dispatch layer routes execution to built-in, eBPF, or passthrough runtimes under a single NVMe wire protocol.
We characterize per-workload break-even across aggregation, filtering, vector similarity, and compression by sweeping buffer size, selectivity, dimensionality, and input entropy, identifying per-workload thresholds at which target-side offload outperforms host-side execution end-to-end.
Our contribution is the first publicly described end-to-end NVMe-CPCS stack: libcs and SPDK CPCS connected via xNVMe, integrated on commodity hardware. The per-workload break-even methodology is reusable by any conforming NVMe-CPCS target, including future CSD silicon, without modifying the application layer above libcs.
Computational Storage
Leveraging CMB to Offload Memory Bandwidth for RDMA Direct Applications
High-performance SSDs use onboard DRAM for FTL tables and internal management with DRAM capacities increasing as SSD densities grow. As SSDs adopt newer DRAM technologies like DDR5, a large portion, often 40%-50%, of DRAM bandwidth remains underutilized by internal SSD operations. NVMe Controller Memory Buffer (CMB) and System Local Memory (SLM) features allow this unused DRAM bandwidth to be exposed to host applications. Leveraging CMB and SLM can unlock significant aggregate memory bandwidth across millions of SSDs in data centers, supporting high-performance workloads and new use cases. We'll discuss the architectural considerations and performance benefits of tapping into SSD DRAM bandwidth for enhanced resource efficiency in modern storage systems.
Computational Storage
Computation Offload Specification for AI Frameworks to Emit Partitioning Functions Alongside Large Data-centric Tasks
After surveying large-scale AI and HPC workloads, we have identified a set of principles and best practices widely used in partitioning large sparse tensors and graphs for parallel execution. We present these in the context of a roadmap for standardization of both Data and Compute offload from costly and power-hungry memory and compute tiers to NDP (near data processing) devices. Our work on standardizing partitioning operators will improve the portability and performance of the programming interface and protocol between AI framework backends that emit large tensor operations and the most power-efficient computational memory and storage devices that implement processing elements as low as subarray level and can only handle smaller chunks of data.
