Selecting an NVMe-oF Ethernet Transport - RDMA or TCP?

NVMe-oF is gaining momentum and becoming the de-facto block-interface for accessing remote high-speed, low-latency, shared NVM storage. NVMe-oF offers several choices of transports for Ethernet datacenter networks, namely, remote direct memory access (RDMA) based NVMe-oF (including both RoCE and iWARP RDMA variants), and TCP based NVMe-oF (compatible with commonly available datacenter networks relying on TCP). The selection between these transports depends on a range of criteria such as: existing datacenter network infrastructure, capabilities of the Ethernet network NICs and switches, deterministic storage performance, security requirements, and type of NVM media, host and target environments.
This talk will first describe high-level details of each NVMe-oF Ethernet transport including dependencies on the Ethernet network infrastructure. The talk will then cover determinism, latency, and bandwidth performance characterization of each transport including the impact of supporting Ethernet network components. Finally, we will share a test suite and associated metrics to exercise NVMe-oF and the selected transports to help understand relative performance under different workload characteristics.