Benchmarking Distributed Coordination Systems: A Survey and Analysis

Coordination services and protocols are critical components of distributed systems and are essential for providing consistency, fault tolerance, and scalability. However, due to lack of a standard benchmarking tool for distributed coordination services, coordination service developers/researchers either use a NoSQL standard benchmark and omit evaluating consistency, distribution, and fault-tolerance; or create their own ad-hoc microbenchmarks and skip comparability with other services. In this paper, we analyze and compare known and widely used distributed coordination services, their evaluations, and the tools used to benchmark those systems. We identify important requirements of distributed coordination service benchmarking, like the metrics and parameters that need to be evaluated and their evaluation setups and tools.