Model merging is a new approach to creating a new model by combining the weights of different trained models. Previous studies report that model merging works well for models trained on a single dataset with different random seeds, while model merging between different datasets is difficult. Merging knowledge from different datasets has practical significance, but it has not been well investigated. In this paper, we investigate the properties of merging models between different datasets. Through theoretical and empirical analyses, we find that the accuracy of the merged model decreases more significantly as the datasets diverge more and that the different loss landscapes for each dataset make model merging between different datasets difficult. We also show that merged models require datasets for merging in order to achieve a high accuracy. Furthermore, we show that condensed datasets created by dataset condensation can be used as substitutes for the original datasets when merging models. We conduct experiments for model merging between different datasets. When merging between MNIST and Fashion- MNIST models, the accuracy significantly improves by 28% using the dataset and 25% using the condensed dataset compared with not using the dataset.

翻译：模型融合是一种通过合并不同训练模型的权重来创建新模型的新方法。先前的研究报告指出，模型融合对于在同一数据集上使用不同随机种子训练的模型效果良好，而不同数据集间的模型融合则较为困难。融合来自不同数据集的知识具有实际意义，但尚未得到充分探索。本文研究了不同数据集间模型融合的特性。通过理论和实证分析，我们发现随着数据集差异增大，融合模型的准确率下降更为显著，且每个数据集的不同损失景观使得跨数据集模型融合变得困难。我们还表明，融合模型需要依赖用于融合的数据集才能实现高准确率。进一步地，我们展示了由数据集压缩生成的压缩数据集可作为原始数据集的替代品用于模型融合。我们在不同数据集间进行了模型融合实验。在MNIST和Fashion-MNIST模型融合时，与不使用数据集相比，使用原始数据集和压缩数据集分别将准确率提升了28%和25%。