Designing High-Performance Non-Volatile Memory-aware RDMA Communication Protocols for Big Data Processing

Author(s)/Presenter(s):
Library Content Type:
Publish Date: 
Wednesday, September 26, 2018
Event Name: 
Focus Areas:
Abstract: 

Recent studies have shown that the performance of in-memory computing and storage systems can be significantly improved by leveraging the high-performance networking technologies and interconnects. Most of these studies are based on low latency and CPU bypass zero-copy communication over volatile memory (i.e., DRAM). On the other hand, emerging persistent memory technologies that can offer byte-addressability persistence, along with DRAM-like performance, are providing us with opportunities to build novel high-performance in-memory sub-systems for data-intensive applications. In this talk, we will present, NRCIO, a high-performance communication runtime for non-volatile memory over modern network interconnects that can be leveraged by existing Big Data processing middleware. We will show the performance of non-volatile memory-aware RDMA communication protocols using our proposed runtime and demonstrate its benefits by incorporating it into a high-performance in-memory key-value store, Apache Hadoop, and Spark.

Watch video: