Typical deep visual recognition models are capable of performing the one task they were trained on. In this paper, we tackle the extremely difficult problem of combining completely distinct models with different initializations, each solving a separate task, into one multi-task model without any additional training. Prior work in model merging permutes one model to the space of the other then adds them together. While this works for models trained on the same task, we find that this fails to account for the differences in models trained on disjoint tasks. Thus, we introduce "ZipIt!", a general method for merging two arbitrary models of the same architecture that incorporates two simple strategies. First, in order to account for features that aren't shared between models, we expand the model merging problem to additionally allow for merging features within each model by defining a general "zip" operation. Second, we add support for partially zipping the models up until a specified layer, naturally creating a multi-head model. We find that these two changes combined account for a staggering 20-60% improvement over prior work, making the merging of models trained on disjoint tasks feasible.

翻译：典型的深度视觉识别模型只能执行其训练时对应的单一任务。本文针对一个极具挑战性的问题：如何将初始化不同、各自解决独立任务的完全不同的模型，在不经过任何额外训练的情况下合并为一个多任务模型。先前的工作通过将一个模型排列到另一个模型的空间中再相加来实现模型合并。虽然这对训练于相同任务的模型有效，但我们发现这种方法无法处理训练于不相关任务时模型间的差异。为此，我们提出了"ZipIt!"——一种通用的、用于合并两个任意相同架构模型的通用方法，该方法融合了两个简单策略：首先，为处理模型间非共享特征，我们将模型合并问题扩展为允许在每个模型内部通过定义通用"压缩"操作来合并特征；其次，我们支持在指定层之前对模型进行部分压缩，从而自然形成多分支（multi-head）模型。实验表明，这两项改进相较先前工作实现了惊人的20-60%性能提升，使训练于不相关任务的模型合并成为可能。