In the era of large-scale training, model merging has evolved into a tool for creating multitasking models efficiently. It enables the knowledge of models to be fused, without the need for heavy computation as required in traditional multitask learning. Existing merging methods often assume that entries at identical positions in weight matrices serve the same function, enabling straightforward entry-wise comparison and merging. However, this assumption overlooks the complexity of finetuned neural networks, where neurons may develop distinct feature compositions, making direct entry-wise merging problematic. We present Decom-Renorm-Merge (DRM), a simple yet effective approach that leverages Singular Value Decomposition to decompose and coordinate weight matrices into an aligned joint space, where entry-wise merging becomes possible. We showcase the effectiveness of DRM across various settings ranging from smaller encoder-based such as ViT and DeBERTa, encoder-decoder-based such as T5, and larger decoder-based such as Llama3.1-8B. Our experimental results show that DRM outperforms several state-of-the-art merging techniques across full finetuning and low-rank adaptation settings. Moreover, our analysis reveals renormalization as the crucial component for creating a robust and even joint space for merging, significantly contributing to the method's performance.

翻译：在大规模训练时代，模型合并已发展为一种高效构建多任务模型的工具。它能够融合不同模型的知识，而无需传统多任务学习所需的大量计算。现有合并方法通常假设权重矩阵中相同位置的条目具有相同功能，从而支持直接的逐元素比较与合并。然而，这一假设忽视了微调后神经网络的复杂性——神经元可能形成不同的特征组合，导致直接逐元素合并存在问题。本文提出分解-重归一化-合并方法，这是一种简单而有效的技术，利用奇异值分解将权重矩阵分解并协调至对齐的联合空间，使逐元素合并成为可能。我们展示了DRM在多种架构中的有效性，包括较小规模的基于编码器的ViT和DeBERTa、基于编码器-解码器的T5，以及更大规模的基于解码器的Llama3.1-8B。实验结果表明，在完全微调与低秩适配设置下，DRM均优于多种先进合并技术。此外，我们的分析表明重归一化是构建稳健且均匀合并联合空间的关键组件，对方法性能提升贡献显著。