Optimizing User Space NVMe-oF TCP Transport Solution with both Software and Hardware Methodologies

In this talk, we would like to update the development status of SPDK user space NVMe/TCP transport and the performance optimizations of NVMe/TCP transport in both software and hardware areas. In the recent one year, there are great efforts to optimize the NVMe-oF transport performance in software especially with the kernel TCP/IP stack, such as: Trade-off memory copy cost to reduce system calls to achieve optimal performance of the NVMe/TCP transport on top of the kernel TCP/IP stack Use asynchronized writev to improve the IOPS  Use libaio/liburing to implement group based  I/O submission for write operations. We also spent some efforts to investigate user space TCP/IP stack (e.g., Seastar) to explore the performance optimization opportunity.   In this talk, we also share Intel’s latest effort to optimize the NVMe/TCP transport in SPDK using Application Device Queue (ADQ) technology from Intel 100G NICs, which improves NVMe/TCP transport performance significantly.  We will talk about how SPDK can export the ADQ feature provided by Intel's new NIC into our common Sock layer library to accelerate the NVMe-oF TCP performance and share the performance data with Intel's latest 100Gb NIC (i.e., E810).  ADQ significantly improves the performance of NVMe/TCP transport in SPDK including reduced average latency, significant reduction in long tail latency, and much higher IOPS.

Learning Objectives

Tell the audience the recent 1 year development status of SPDK user space NVMe-oF TCP transport,Share our learned experience on how to accelerate NVMe-oF TCP transport.,Tell the audience the optimization methods on using the Kernel TCP stack (e.g., async writev, group based I/O submission),Tell the audience the possible optimization direction using user space TCP/IP stack.,Share the performance result from using Intel's 100Gb NIC with the ADQ (application device queue) technology to accelerate NVMe-oF TCP transport.