Data Storage Innovations for Scalable AI Infrastructure: From NAND Flash to Exabyte-Scale Hierarchies
Winchester
Mon Sep 28 | 8:30am
Abstract
As AI models grow from billions to trillions of parameters, storage has emerged as a fundamental bottleneck — not just an afterthought — in AI system design. This talk examines the storage and memory demands imposed by modern AI workloads, from large-scale distributed training to high-throughput inference, and presents architectural approaches to address them.
We begin by quantifying the explosive growth in model parameters and training data, and derive estimates of memory requirements for both training (model state, optimizer states, gradients, and activations) and inference (weights and KV Cache). We show that inference — not training — is becoming the dominant long-term storage driver as context lengths and concurrent user counts scale.
We then present a layered memory-storage hierarchy optimized for AI data centers, spanning HBM, DRAM, local NVMe SSDs, remote NVMe over RDMA, and object storage at exabyte scale. We discuss how workload characteristics — burstiness, parallelism, endurance, and latency sensitivity — drive distinct tier requirements for training, checkpointing, inference, RAG, and archival.
Finally, we highlight emerging flash storage innovations including new form factors, liquid-cooled density solutions, storage offload, and GPU-direct data paths that reduce CPU overhead and enable cost-efficient AI scale.
Related Sessions
AI - General
Data Storage Innovations for Scalable AI Infrastructure: From NAND Flash to Exabyte-Scale Hierarchies
As AI models grow from billions to trillions of parameters, storage has emerged as a fundamental bottleneck — not just an afterthought — in AI system design. This talk examines the storage and memory demands imposed by modern AI workloads, from large-scale distributed training to high-throughput inference, and presents architectural approaches to address them.
We begin by quantifying the explosive growth in model parameters and training data, and derive estimates of memory requirements for both training (model state, optimizer states, gradients, and activations) and inference (weights and KV Cache). We show that inference — not training — is becoming the dominant long-term storage driver as context lengths and concurrent user counts scale.
We then present a layered memory-storage hierarchy optimized for AI data centers, spanning HBM, DRAM, local NVMe SSDs, remote NVMe over RDMA, and object storage at exabyte scale. We discuss how workload characteristics — burstiness, parallelism, endurance, and latency sensitivity — drive distinct tier requirements for training, checkpointing, inference, RAG, and archival.
Finally, we highlight emerging flash storage innovations including new form factors, liquid-cooled density solutions, storage offload, and GPU-direct data paths that reduce CPU overhead and enable cost-efficient AI scale.
AI - General
Multi-Layer I/O Profiling and Syscall-Level Deterministic Replay for AI Storage Workloads
As AI workloads—LLM inference, vector-database serving, and KV-cache offloading—dominate data-center resources, characterizing their storage I/O behavior has become critical for optimizing next-generation storage systems. However, the astronomical cost and scarcity of high-end GPU hardware create a significant barrier for storage engineers and researchers. This session introduces ebpf-syscall, an open-source toolset that bridges this gap by enabling researchers to capture complex AI storage patterns once on expensive hardware and replay them infinitely on commodity NVMe systems.
The core of the solution is a coordinated multi-layer profiler that simultaneously instruments six layers of the I/O stack: application intent, the syscall layer, memory-page activity, sub-page memory access (8-byte resolution), the Linux block layer, and the NVMe driver. By correlating these layers, the tool reveals critical phenomena often invisible to single-layer monitoring, such as application-side pre-split I/Os and memory-mapped dirty-page write-back patterns. This work builds upon and complements recent unified VFS monitoring frameworks by adding a capability that monitoring alone cannot provide: deterministic replay.
At the core of the toolset is a faithful reproduction engine that regenerates application I/O behavior at the operating-system boundary. Using lightweight kernel-level instrumentation, every relevant I/O operation is captured along with its parameters and precise timing. The replayer then dispatches these operations in their original order, preserving the exact arguments and sequencing of the source workload. Three design choices enable high-fidelity reproduction: faithful handling of memory-mapped I/O across capture and replay; a remapping layer that keeps file references consistent even when the underlying handles differ between systems; and three playback modes—true-to-original pacing, preserved-gap pacing, and a compressed mode for amplified load testing.
By operating at the syscall boundary, ebpf-syscall fills the gap between block-level tracing—which is too low-level to express application intent—and application-level instrumentation, which is invasive and workload-specific. This positioning allows storage vendors and academic researchers to characterize and optimize test storage configurations against the production I/O fingerprint of sensitive workloads (e.g., fraud-detection pipelines) while remaining compliant with data-protection regulations, since only syscall metadata—not payload data—is captured.
Validation against KV-cache offload workload captured from vLLM and LMCache demonstrates that the replayer reproduces the original IOPS, throughput, and access-pattern histograms of real-world AI traces with high statistical fidelity. Attendees will learn how this toolset (available at https://github.com/SamsungDS/ebpf-syscall) facilitates decoupled SSD firmware validation and performance tuning, significantly reducing the cost of AI-optimized storage development.
AI - General
Rethinking Memory: Strategies to Navigate the DRAM "Pricing Apocalypse"
DRAM supply is tightening and prices are surging in an unprecedented way, translating to painful impacts on infrastructure purchasing decisions and mission-critical workloads. This session will examine a key technical mitigation strategy: AI-powered memory tiering across NVMe flash and DRAM. We’ll share real-world data that illustrates how this approach can sustain performance on existing infrastructure, offering a solution to the DRAM crunch.
