The Risk of Federated Learning to Skew Fine-Tuning Features and Underperform Out-of-Distribution Robustness
CoRR(2024)
摘要
To tackle the scarcity and privacy issues associated with domain-specific
datasets, the integration of federated learning in conjunction with fine-tuning
has emerged as a practical solution. However, our findings reveal that
federated learning has the risk of skewing fine-tuning features and
compromising the out-of-distribution robustness of the model. By introducing
three robustness indicators and conducting experiments across diverse robust
datasets, we elucidate these phenomena by scrutinizing the diversity,
transferability, and deviation within the model feature space. To mitigate the
negative impact of federated learning on model robustness, we introduce GNP, a
General Noisy Projection-based robust
algorithm, ensuring no deterioration of accuracy on the target distribution.
Specifically, the key strategy for enhancing model robustness entails the
transfer of robustness from the pre-trained model to the fine-tuned model,
coupled with adding a small amount of Gaussian noise to augment the
representative capacity of the model. Comprehensive experimental results
demonstrate that our approach markedly enhances the robustness across diverse
scenarios, encompassing various parameter-efficient fine-tuning methods and
confronting different levels of data heterogeneity.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要