Towards Optimal Customized Architecture for Heterogeneous Federated Learning with Contrastive Cloud-Edge Model Decoupling
arxiv(2024)
摘要
Federated learning, as a promising distributed learning paradigm, enables
collaborative training of a global model across multiple network edge clients
without the need for central data collecting. However, the heterogeneity of
edge data distribution drags the model towards the local minima, which can be
distant from the global optimum. Such heterogeneity often leads to slow
convergence and substantial communication overhead. To address these issues, we
propose a novel federated learning framework called FedCMD, a model decoupling
tailored to the Cloud-edge supported federated learning that separates deep
neural networks into a body for capturing shared representations in Cloud and a
personalized head for migrating data heterogeneity. Our motivation is that, by
the deep investigation of the performance of selecting different neural network
layers as the personalized head, we found rigidly assigning the last layer as
the personalized head in current studies is not always optimal. Instead, it is
necessary to dynamically select the personalized layer that maximizes the
training performance by taking the representation difference between neighbor
layers into account. To find the optimal personalized layer, we utilize the
low-dimensional representation of each layer to contrast feature distribution
transfer and introduce a Wasserstein-based layer selection method, aimed at
identifying the best-match layer for personalization. Additionally, a weighted
global aggregation algorithm is proposed based on the selected personalized
layer for the practical application of FedCMD. Extensive experiments on ten
benchmarks demonstrate the efficiency and superior performance of our solution
compared with nine state-of-the-art solutions. All code and results are
available at https://github.com/elegy112138/FedCMD.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要