Distributed Statistical Machine Learning in Adversarial Settings: Byzantine Gradient Descent

We consider the distributed statistical learning problem over decentralized systems that are prone to adversarial attacks. This setup arises in many practical applications, including Google's Federated Learning. Formally, we focus on a decentralized system that consists of a parameter server and m working machines; each working machine keeps N/m data samples, where N is the total number of samples. In each iteration, up to q of the m working machines suffer Byzantine faults - a faulty machine in the given iteration behaves arbitrarily badly against the system and has complete knowledge of the system. Additionally, the sets of faulty machines may be different across iterations. Our goal is to design robust algorithms such that the system can learn the underlying true parameter, which is of dimension d, despite the interruption of the Byzantine attacks. In this paper, based on the geometric median of means of the gradients, we propose a simple variant of the classical gradient descent method. We show that our method can tolerate q Byzantine failures up to 2(1 + epsilon)q <= m for an arbitrarily small but fixed constant epsilon > 0. The parameter estimate converges in O(log N) rounds with an estimation error on the order of max{root dq/N, root d/N}, which is larger than the minimax-optimal error rate root d/N in the centralized and failure-free setting by at most a factor of root q. The total computational complexity of our algorithm is of O((Nd/m) log N) at each working machine and O(md + kd log(3) N) at the central server, and the total communication cost is of O(md log N). We further provide an application of our general results to the linear regression problem. A key challenge arises in the above problem is that Byzantine failures create arbitrary and unspecified dependency among the iterations and the aggregated gradients. To handle this issue in the analysis, we prove that the aggregated gradient, as a function of model parameter, converges uniformly to the true gradient function.