Model merging aims to cheaply combine individual task-specific models into a single multitask model. In this work, we view past merging methods as leveraging different notions of a ''task subspace'' in which models are matched before being merged. We connect the task subspace of a given model to its loss landscape and formalize how this approach to model merging can be seen as solving a linear system of equations. While past work has generally been limited to linear systems that have a closed-form solution, we consider using the conjugate gradient method to find a solution. We show that using the conjugate gradient method can outperform closed-form solutions, enables merging via linear systems that are otherwise intractable to solve, and flexibly allows choosing from a wide variety of initializations and estimates for the ''task subspace''. We ultimately demonstrate that our merging framework called ''Matching Models in their Task Subspace'' (MaTS) achieves state-of-the-art results in multitask and intermediate-task model merging. We release all of the code and checkpoints used in our work at https://github.com/r-three/mats.

翻译：模型合并旨在低成本地将多个特定任务的单任务模型组合成一个多任务模型。在本研究中，我们将以往的合并方法视为利用了不同的"任务子空间"概念，在该子空间中对模型进行匹配后再合并。我们将给定模型的任务子空间与其损失景观相关联，并形式化地将这种模型合并方法理解为求解线性方程组的过程。尽管以往的工作通常局限于具有闭式解的线性系统，但我们考虑使用共轭梯度法来求解。研究表明，使用共轭梯度法可超越闭式解的性能，能够合并原本难以求解的线性系统，并灵活地从多种初始化和"任务子空间"估计中选择方案。最终，我们论证了名为"任务子空间中的模型匹配"（MaTS）的合并框架在多任务和中间任务模型合并中取得了最先进的效果。我们在https://github.com/r-three/mats公开了本研究中使用的所有代码和检查点。