Adaptive Distributed NVMe-oF Namespaces

While NVMe-oF excels at delivering IO to a single remote target in both performance and efficiency, this model of storage devices is inconsistent with large scale storage systems. Storage systems providing rich volume services from distributed pools of storage devices tend to spread logical volumes across many devices in many storage nodes. Host IO must then be delivered to logical volume extents in many storage nodes. Common solutions include bespoke storage clients installed in every host, or dedicated storage gateways to adapt standard storage protocols to the distributed storage service. Intel's Adaptive Distributed NVMe-oF Namespaces (ADNN) is an NVMe-oF extension that enables distributed volumes to be accessed via NVMe-oF in a single fabric hop. Gateways and bespoke storage clients are eliminated. We'll show how ADNN enables hosts to learn which of several equivalent NVMe-oF targets contains each region of a namespace, and adapt to placement changes at runtime. We’ll describe how the ADNN architecture can be used in a single storage node for NUMA optimized placement, and in large scale-out cloud storage systems like Ceph. We’ll show our experimental results using ADNN to access Ceph RBD images. You’ll see how the ADNN hash hint results in host IO delivered directly to the correct OSD node via NVMe-oF, and how the CPU utilization in the host is reduced by eliminating the Ceph block client. We’ll show the results of our hash hint latency POC, which demonstrates that applying the hash hint adds very little latency. We’ll explain how the ADNN hash hint splits the Ceph CRUSH process leaving only the actual hash function to be performed in the NVMe-oF host data path (and the rest performed via Ceph CLI tools only when the cluster map changes). Finally we’ll outline our plans for upstreaming the ADNN reference implementation to SPDK, its current status, and how you can try out the SPDK ADNN building blocks yourself.

Learning Objectives

Drawbacks of gateways and bespoke distributed storage clients,ADNN location hints eliminate the need for gateways and bespoke clients,ADNN can benefits storage systems as small as a single node,ADNN works with Ceph, and reduces client CPU usage.,ADNN is coming to an SPDK git repository near you