p2pmem: Enabling PCIe Peer-2-Peer in Linux

PCIe devices such as GPGPUs, FPGA accerlators, RDMA enabled NICs and NVM Express SSDs are placing a huge strain on the IO subsystem of the CPUs they are connected too. With the emergence of NVMe over Fabrics and heterogeneous compute there is often a desire to move large stream of data between these endpoints without CPU intervention. In this presentation we give an overview of the latest work we have done to enable this PCIe Peer-2-Peer (P2P) communication in the Linux kernel. We also cover a detailed performance comparison between normal data flows and those that avail of P2P. We show how the P2P data-flows offload the IO subsystem of the CPU and lead to better throughput, latency and Quality of Service.