NVMe-oF: Scaling up with the Storage Performance Development Kit

The Storage Performance Development Kit (SPDK) Project released an open source NVMe over Fabrics (NVMe-oF) software target in conjunction with release of the NVMe-oF specification in 2016. This target has continued to evolve to ensure linear scaling with the addition of CPU cores, NICs, and NVMe devices, all while maintaining the low latency characteristics of RDMA and FC. This talk will address a number of challenges encountered during this evolution of the SPDK NVMe-oF target including:

• Keeping connections independent from one another to enable a lock-free I/O path
• Maintaining mutable NVMe subsystem state and cross-thread coordination
• Effectively leveraging hardware features in RDMA NICs and Fibre Channel HBAs
• Challenges created by the NVMe over Fabrics protocol itself

Further, benchmarks will be presented showing scaling behaviors in the latest NVMe over Fabrics target against a range of variables including the number of connections, the number of subsystems, and the number of NICs, CPU cores, and NVMe SSDs.

Learning Objectives:
1. Understanding a real NVMe over Fabrics implementation
2. How to scale up with NVMe over Fabrics effectively
3. What SPDK provides to storage developers as a general purpose toolkit