A Scalable Artificial Intelligence Data Pipeline for Accelerating Time to Insight

Artificial intelligence (AI) requires processing power and adequate storage while executing various deep learning (DL) frameworks. The training and deployment stages of a DL system have different data and processing needs. On one hand, the large volumes of data during training demands systems with support for massive storage capacity, multiple data formats and protocols for processing dispersed data sets, and sharing of data and models across applications. On the other hand, AI deployment for delivering inference on incoming data requires fast access to the data to meet the demand for AI responsiveness for applications.

The processing and storage needs vary for the different phases of an AI data pipeline comprising of data ingestion, model training and model serving. Disaggregation of GPUs, flash and object storage can enable the delivery of rapid response times and scaling requirements of an AI data pipeline, without compromising on data persistence, quality, durability and cost.