NVMe-oF on RDMA Performance Challenges and Solutions in Commodity Servers

NVMe-oF has become a very successful new networked storage protocol in both the Cloud and Enterprise storage markets. But in order to provide the highest performance NVMe-oF solutions, it is necessary to ensure that all parts of the system are working together as effectively as possible. From the network perspective, industry data has shown that NVMe-oF on RDMA can provide the highest bandwidth and lowest consistent latencies. The goal of this talk is to point out some limitations in commodity servers that can in certain circumstances prevent todays NICs from achieving even higher performance. We will also present the solutions to these limitations that we have incorporated into the NVMe-oF Linux kernel driver. These solutions include, Dynamic Interrupt Moderation and shared Completion Queues. This work demonstrates the importance of continued optimizations to achieve the maximum performance from the commodity server and NIC hardware currently available.

Learning Objectives

Describe the hardware limitations discovered in commodity servers,Present a couple system level solutions,Give insight into the process of performance debug to solution