Multi-task learning (MTL) compresses the information from multiple tasks into a unified backbone to improve computational efficiency and generalization. Recent work directly merges multiple independently trained models to perform MTL instead of collecting their raw data for joint training, greatly expanding the application scenarios of MTL. However, by visualizing the representation distribution of existing model merging schemes, we find that the merged model often suffers from the dilemma of representation bias. That is, there is a significant discrepancy in the representation distribution between the merged and individual models, resulting in poor performance of merged MTL. In this paper, we propose a representation surgery solution called "Surgery" to reduce representation bias in the merged model. Specifically, Surgery is a lightweight task-specific module that takes the representation of the merged model as input and attempts to output the biases contained in the representation from the merged model. We then designed an unsupervised optimization objective that updates the Surgery module by minimizing the distance between the merged model's representation and the individual model's representation. Extensive experiments demonstrate significant MTL performance improvements when our Surgery module is applied to state-of-the-art (SOTA) model merging schemes.

翻译：多任务学习（MTL）将来自多个任务的信息压缩到一个统一的骨干网络中，以提高计算效率和泛化能力。近期研究直接融合多个独立训练的模型来执行MTL，而无需收集其原始数据进行联合训练，这极大地扩展了MTL的应用场景。然而，通过可视化现有模型融合方案的表征分布，我们发现融合后的模型常常面临表征偏差的困境。也就是说，融合模型与独立模型之间的表征分布存在显著差异，导致融合后的MTL性能不佳。在本文中，我们提出了一种名为“Surgery”的表征外科手术解决方案，以减少融合模型中的表征偏差。具体而言，Surgery是一个轻量级的任务特定模块，它以融合模型的表征作为输入，并试图输出融合模型表征中包含的偏差。随后，我们设计了一个无监督优化目标，通过最小化融合模型表征与独立模型表征之间的距离来更新Surgery模块。大量实验表明，当我们的Surgery模块应用于最先进的模型融合方案时，MTL性能得到了显著提升。