The Effect of Non Volatile Memory on a Distributed Storage System

Data workloads generate heavy IOPS and operate on large amounts of data typically stored in a distributed storage system. Traditionally, storage has been the slowest component of a data intensive system, far slower than the CPU and the network. Though with the arrival of faster, persistent storage in the form of NVMe based solid state drives, the I/O performance has increased dramatically, yet the I/O is still slower than the other components in the system. But with the upcoming byte-addressable, non-volatile memory available in DIMM form factor (NVDIMM), this is no longer true; the CPU and the network might soon become bottlenecks. In this paper, we study the effect of using NVDIMM in a representative distributed storage system, Ceph. Through experiments, we demonstrate drastic increase in CPU and network usage, by adding persistent memory to existing storage software.