Research on neural networks has largely focused on understanding a single model trained on a single dataset. However, relatively little is known about the relationships between different models, especially those trained or tested on different datasets. We address this by studying how the weight space and underlying loss landscape of different models are interconnected. Specifically, we demonstrate that fine-tuned models that were optimized for high performance, reside in well-defined regions in weight space, and vice versa -- that any model that resides anywhere in those regions also has high performance. Specifically, we show that language models that have been fine-tuned on the same dataset form a tight cluster in the weight space and that models fine-tuned on different datasets from the same underlying task form a looser cluster. Moreover, traversing around the region between the models reaches new models that perform comparably or even better than models found via fine-tuning, even on tasks that the original models were not fine-tuned on. Our findings provide insight into the relationships between models, demonstrating that a model positioned between two similar models can acquire the knowledge of both. We leverage this finding and design a method to pick a better model for efficient fine-tuning. Specifically, we show that starting from the center of the region is as good or better than the pre-trained model in 11 of 12 datasets and improves accuracy by 3.06 on average.

翻译：关于神经网络的研究主要聚焦于理解在单一数据集上训练的单一模型。然而，人们对不同模型之间的关系知之甚少，尤其是那些在不同数据集上训练或测试的模型。我们通过研究不同模型的权重空间与底层损失景观之间的相互联系来解决这一问题。具体而言，我们证明了经过优化以实现高性能的微调模型位于权重空间中定义明确的区域，反之亦然——任何位于这些区域内的模型也能表现出高性能。具体地，我们展示了在同一数据集上微调的语言模型在权重空间中形成紧密聚类，而在同一底层任务的不同数据集上微调的模型则形成较松散的聚类。此外，在模型之间的区域中遍历可达的新模型，其性能可与微调得到的模型相媲美甚至更优，即使在原始模型未经过微调的任务上也是如此。我们的发现揭示了模型之间的关系，表明处于两个相似模型之间的模型能够同时获取两者的知识。我们利用这一发现设计了一种方法，能够为高效微调选择更优的模型。具体而言，我们证明在12个数据集中的11个上，从该区域中心出发的效果与预训练模型相当或更优，平均准确率提升3.06%。