Navigating Noise: A Study of How Noise Influences Generalisation and Calibration of Neural Networks
arxiv(2023)
摘要
Enhancing the generalisation abilities of neural networks (NNs) through
integrating noise such as MixUp or Dropout during training has emerged as a
powerful and adaptable technique. Despite the proven efficacy of noise in NN
training, there is no consensus regarding which noise sources, types and
placements yield maximal benefits in generalisation and confidence calibration.
This study thoroughly explores diverse noise modalities to evaluate their
impacts on NN's generalisation and calibration under in-distribution or
out-of-distribution settings, paired with experiments investigating the metric
landscapes of the learnt representations across a spectrum of NN architectures,
tasks, and datasets. Our study shows that AugMix and weak augmentation exhibit
cross-task effectiveness in computer vision, emphasising the need to tailor
noise to specific domains. Our findings emphasise the efficacy of combining
noises and successful hyperparameter transfer within a single domain but the
difficulties in transferring the benefits to other domains. Furthermore, the
study underscores the complexity of simultaneously optimising for both
generalisation and calibration, emphasising the need for practitioners to
carefully consider noise combinations and hyperparameter tuning for optimal
performance in specific tasks and datasets.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要