Communication-efficient clustered federated learning via model distance

Clustered Federated Learning (CFL) leverages the differences among data distributions on clients to partition all clients into several clusters for personalized federated training. Compared with the conventional federated algorithms such as FedAvg, existing methods for CFL require either more communication costs or multi-stage computation overheads. In this paper, we propose an iterative CFL framework with almost the same communication cost as FedAvg in each round based on a novel model distance. Specifically, the model distance measures the discrepancy between the client model and the cluster model so that we can estimate the cluster identities for clients on the server side. The proposed model distance considers class-wise model dissimilarity, which enables us to apply it to multi-class classification even when the labels are non-iid across clients. To calculate the proposed model distance, we introduce two sampling methods which generate samples from feature distributions approximately without accessing the raw dataset. Experimental results show that our method can achieve superior and comparable performance on non-iid and iid data respectively with less communication cost compared with the baselines.