Not Eliminate but Aggregate: Post-Hoc Control over Mixture-of-Experts to Address Shortcut Shifts in Natural Language Understanding
CoRR(2024)
摘要
Recent models for natural language understanding are inclined to exploit
simple patterns in datasets, commonly known as shortcuts. These shortcuts hinge
on spurious correlations between labels and latent features existing in the
training data. At inference time, shortcut-dependent models are likely to
generate erroneous predictions under distribution shifts, particularly when
some latent features are no longer correlated with the labels. To avoid this,
previous studies have trained models to eliminate the reliance on shortcuts. In
this study, we explore a different direction: pessimistically aggregating the
predictions of a mixture-of-experts, assuming each expert captures relatively
different latent features. The experimental results demonstrate that our
post-hoc control over the experts significantly enhances the model's robustness
to the distribution shift in shortcuts. Besides, we show that our approach has
some practical advantages. We also analyze our model and provide results to
support the assumption.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要