NetShield: An in-network architecture against byzantine failures in distributed deep learning

There is a growing trend of training deep learning networks on distributed clusters. Unfortunately, distributed deep learning (DDL) is prone to Byzantine failures where some nodes corrupt training by sending malicious gradients to the parameter server (PS). Existing works address this problem by implementing Byzantine defenses on the PS. However, Byzantine defenses come with large computational overhead, seriously affecting the DDL’s training performance. Moreover, malicious gradients are not identified until they are transmitted to the endpoint (PS), which leads to a waste of network resources and a decrease in communication efficiency.