Hybrid Coded MapReduce for Latency-Constrained Tasks with Straggling Servers

Coded distributed computing (CDC) can efficiently improve the performance of MapReduce by mitigating the effects of straggling servers and decreasing the communication load simultaneously. In this paper, for the latency-constrained computation tasks, we use hybrid coded MapReduce and optimize the hybrid coded scheme to minimize the total latency, including the computation latency in Map phase and the communication latency in Shuffle phase. For the practical scenarios where the number of straggling servers is relatively small compared with the number of all servers, we reformulate the original optimization problem into a non-convex optimization problem. Then we use the successive convex approximation (SCA) algorithm to approximate the non-convex optimization problem into the iterations of a convex optimization problem. We prove the effectiveness and convergence of the proposed iterative SCA algorithm. Besides, the proposed SCA-based algorithm shows the effective convergence and outperforms the enumeration algorithm of the original problem with the approximated total latency and the short execution time via numerical experiments. Furthermore, the hybrid coded scheme achieves significant performance improvement in terms of decreasing the total latency compared with baseline schemes.