Research on neural networks has largely focused on understanding a single model trained on a single dataset. However, relatively little is known about the relationships between different models, especially those trained or tested on different datasets. We address this by studying how the weight space and underlying loss landscape of different models are interconnected. Specifically, we demonstrate that fine-tuned models that were optimized for high performance, reside in well-defined regions in weight space, and vice versa -- that any model that resides anywhere in those regions also has high performance. Specifically, we show that language models that have been fine-tuned on the same dataset form a tight cluster in the weight space and that models fine-tuned on different datasets from the same underlying task form a looser cluster. Moreover, traversing around the region between the models reaches new models that perform comparably or even better than models found via fine-tuning, even on tasks that the original models were not fine-tuned on. Our findings provide insight into the relationships between models, demonstrating that a model positioned between two similar models can acquire the knowledge of both. We leverage this finding and design a method to pick a better model for efficient fine-tuning. Specifically, we show that starting from the center of the region is as good or better than the pre-trained model in 11 of 12 datasets and improves accuracy by 3.06 on average.

翻译：关于神经网络的研究主要聚焦于理解在单一数据集上训练的单一模型。然而，不同模型之间的关系，尤其是那些在不同数据集上训练或测试的模型，至今仍鲜有探讨。我们通过研究不同模型的权重空间与底层损失景观之间的关联性来填补这一空白。具体而言，我们证明了经过优化以实现高性能的微调模型，会驻留在权重空间中明确定义的区域内，反之亦然——即任何位于这些区域内的模型也能获得高性能。我们进一步表明，在同一数据集上微调的语言模型在权重空间中会形成紧密簇，而在同一底层任务的不同数据集上微调的模型则形成较松散簇。此外，在模型间区域中遍历可生成性能与微调模型相当甚至更优的新模型，即使面对原始模型未经过微调的任务也是如此。这一发现揭示了模型间的潜在关系：位于两个相似模型之间的模型能够同时获取两者的知识。我们利用此发现设计了一种方法，为高效微调选择更优的初始模型。具体而言，在12个数据集中，从该区域中心点出发的训练效果优于或等同于预训练模型的占11个，且平均准确率提升3.06%。