Learning in deep weight spaces (DWS), where neural networks process the weights of other neural networks, is an emerging research direction, with applications to 2D and 3D neural fields (INRs, NeRFs), as well as making inferences about other types of neural networks. Unfortunately, weight space models tend to suffer from substantial overfitting. We empirically analyze the reasons for this overfitting and find that a key reason is the lack of diversity in DWS datasets. While a given object can be represented by many different weight configurations, typical INR training sets fail to capture variability across INRs that represent the same object. To address this, we explore strategies for data augmentation in weight spaces and propose a MixUp method adapted for weight spaces. We demonstrate the effectiveness of these methods in two setups. In classification, they improve performance similarly to having up to 10 times more data. In self-supervised contrastive learning, they yield substantial 5-10% gains in downstream classification.

翻译：深度权重空间学习（DWS）是一个新兴研究方向，它使神经网络能够处理其他神经网络的权重，可应用于二维与三维神经场（如隐式神经表示INR、神经辐射场NeRF），并对其他类型神经网络进行推理分析。然而，权重空间模型普遍存在严重的过拟合问题。我们通过实证分析发现，过拟合的关键原因在于DWS数据集缺乏多样性。尽管同一物体可由多种不同的权重配置表示，但典型的INR训练集未能捕获表征同一物体的隐式神经网络之间的权重变异性。为解决这一问题，我们探索了权重空间中的数据增强策略，并提出了一种适配权重空间的MixUp方法。我们在两种实验设置中验证了这些方法的有效性：在分类任务中，其性能提升效果等同于将训练数据量扩大10倍；在自监督对比学习中，下游分类准确率获得5%-10%的显著提升。