Improved Generalization of Weight Space Networks via Augmentations
CoRR(2024)
摘要
Learning in deep weight spaces (DWS), where neural networks process the
weights of other neural networks, is an emerging research direction, with
applications to 2D and 3D neural fields (INRs, NeRFs), as well as making
inferences about other types of neural networks. Unfortunately, weight space
models tend to suffer from substantial overfitting. We empirically analyze the
reasons for this overfitting and find that a key reason is the lack of
diversity in DWS datasets. While a given object can be represented by many
different weight configurations, typical INR training sets fail to capture
variability across INRs that represent the same object. To address this, we
explore strategies for data augmentation in weight spaces and propose a MixUp
method adapted for weight spaces. We demonstrate the effectiveness of these
methods in two setups. In classification, they improve performance similarly to
having up to 10 times more data. In self-supervised contrastive learning, they
yield substantial 5-10
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要