Fine-tuning pre-trained models provides significant advantages in downstream performance. The ubiquitous nature of pre-trained models such as BERT and its derivatives in natural language processing has also led to a proliferation of task-specific fine-tuned models. As these models typically only perform one task well, additional training or ensembling is required in multi-task scenarios. The growing field of model merging provides a solution, dealing with the challenge of combining multiple task-specific models into a single multi-task model. In this study, we introduce a novel model merging method for Transformers, combining insights from previous work in Fisher-weighted averaging and the use of Fisher information in model pruning. Utilizing the Fisher information of mask nodes within the Transformer architecture, we devise a computationally efficient weighted-averaging scheme. Our method exhibits a regular and significant performance increase across various models in the BERT family, outperforming full-scale Fisher-weighted averaging in a fraction of the computational cost, with baseline performance improvements of up to +6.5 and a speedup between 57.4x and 321.7x across models. Our results prove the potential of our method in current multi-task learning environments and suggest its scalability and adaptability to new model architectures and learning scenarios.

翻译：微调预训练模型在下游任务中具有显著优势。BERT及其衍生模型在自然语言处理中的普适性，也导致针对特定任务微调的模型数量激增。由于此类模型通常仅擅长单一任务，在多任务场景中需要进行额外训练或集成。日益发展的模型合并领域为此提供了解决方案，旨在将多个特定任务模型组合为统一的多任务模型。本研究针对Transformer提出了一种新型模型合并方法，融合了Fisher加权平均与基于Fisher信息的模型剪枝领域的先前研究成果。通过利用Transformer架构中掩码节点的Fisher信息，我们设计了一种计算高效的加权平均方案。该方法在BERT系列各类模型中展现出规律且显著的性能提升，以极低的计算成本全面超越全规模Fisher加权平均，基准性能提升高达+6.5，且各模型加速比介于57.4倍至321.7倍之间。实验结果表明，该方法在当前多任务学习环境中具有应用潜力，并展现出对新模型架构与学习场景的可扩展性与适应性。