Unraveling the Key Components of OOD Generalization via Diversification
CoRR(2023)
摘要
Real-world datasets may contain multiple features that explain the training
data equally well, i.e., learning any of them would lead to correct predictions
on the training data. However, many of them can be spurious, i.e., lose their
predictive power under a distribution shift and fail to generalize to
out-of-distribution (OOD) data. Recently developed “diversification” methods
approach this problem by finding multiple diverse hypotheses that rely on
different features. This paper aims to study this class of methods and identify
the key components contributing to their OOD generalization abilities.
We show that (1) diversification methods are highly sensitive to the
distribution of the unlabeled data used for diversification and can
underperform significantly when away from a method-specific sweet spot. (2)
Diversification alone is insufficient for OOD generalization. The choice of the
used learning algorithm, e.g., the model's architecture and pretraining, is
crucial, and using the second-best choice leads to an up to 20
in accuracy.(3) The optimal choice of learning algorithm depends on the
unlabeled data, and vice versa.Finally, we show that the above pitfalls cannot
be alleviated by increasing the number of diverse hypotheses, allegedly the
major feature of diversification methods.
These findings provide a clearer understanding of the critical design factors
influencing the OOD generalization of diversification methods. They can guide
practitioners in how to use the existing methods best and guide researchers in
developing new, better ones.
更多查看译文
关键词
Algorithm Design,Diversity,OOD Generalization,Spurious Correlation,Understanding Neural Networks
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要