Everything you wanted to ask about Open Source Deep Learning training pipelines (but were afraid to ask?)

Modern deep learning pipelines rely on complex open source storage and data movement stacks, yet the path from NVMe storage to GPU memory is often poorly understood. Performance issues are frequently attributed to “storage” without clear visibility into where the actual bottlenecks occur or how different layers of the pipeline contribute to inefficiency. This talk presents an in-depth experimental analysis and characterization of data flow across diverse high performance data loading libraries such as DALI, MONAI, and FFCV in PyTorch. We study fine-tuning and training workloads across multiple domains, including natural language processing, image processing, speech processing, and medical imaging, using datasets of different modalities and sizes and modern state-of-the-art H100 GPU platforms with high speed NVMe SSDs.

We examine how data moves through the full system stack of storage devices, filesystems, data loaders, preprocessing stages, and GPU compute. The session explores the impact of data loading libraries, data formats, checkpointing mechanisms, and distributed training behavior, along with system tuning options and their effect on compute, memory, and storage utilization, system overheads, and overall pipeline efficiency. Our analysis reveals surprising inefficiencies across different workload domains, where compute utilization and memory overheads vary with storage hardly ever being the bottleneck. We highlight how the choice of library, tuning knobs, and pipeline design decisions can significantly affect throughput, latency, and resource efficiency. 

The study identifies practical opportunities to optimize data movement by aligning frameworks, data formats, and tuning strategies with the capabilities of the underlying hardware platform. This session shares concrete lessons learned for storage developers, systems engineers, and AI infrastructure practitioners, providing guidance for designing, benchmarking, and tuning hardware-aware deep learning pipelines. The attendees will gain a clearer understanding of where performance is lost, how to improve end-to-end efficiency, and how storage architecture influences modern AI - Training and fine-tuning workloads.