Merging finetuned Large Language Models (LLMs) has become increasingly important for integrating diverse capabilities into a single unified model. However, prevailing model merging methods rely on linear arithmetic in Euclidean space, which often destroys the intrinsic geometric properties of pretrained weights, such as hyperspherical energy. To address this, we propose Orthogonal Model Merging (OrthoMerge), a method that performs merging operations on the Riemannian manifold formed by the orthogonal group to preserve the geometric structure of the model's weights. By mapping task-specific orthogonal matrices learned by Orthogonal Finetuning (OFT) to the Lie algebra, OrthoMerge enables a principled yet efficient integration that takes into account both the direction and intensity of adaptations. In addition to directly leveraging orthogonal matrices obtained by OFT, we further extend this approach to general models finetuned with non-OFT methods (i.e., low-rank finetuning, full finetuning) via an Orthogonal-Residual Decoupling strategy. This technique extracts the orthogonal components of expert models by solving the orthogonal Procrustes problem, which are then merged on the manifold of the orthogonal group, while the remaining linear residuals are processed through standard additive merging. Extensive empirical results demonstrate the effectiveness of OrthoMerge in mitigating catastrophic forgetting and maintaining model performance across diverse tasks.

翻译：微调后的大型语言模型（LLM）的融合对于将多样化能力集成至单一统一模型已变得日益重要。然而，当前主流的模型融合方法依赖于欧几里得空间中的线性算术操作，这往往会破坏预训练权重的内在几何特性，例如超球面能量。为解决此问题，我们提出正交模型融合（OrthoMerge），该方法在由正交群形成的黎曼流形上执行融合操作，以保持模型权重的几何结构。通过将由正交微调（OFT）学习到的任务特定正交矩阵映射到李代数，OrthoMerge 实现了一种既考虑适应方向又考虑适应强度的、有理论依据且高效的集成。除了直接利用 OFT 获得的正交矩阵外，我们还通过一种正交-残差解耦策略，将此方法进一步扩展到使用非 OFT 方法（即低秩微调、全参数微调）微调的一般模型。该技术通过求解正交普鲁克问题来提取专家模型的正交分量，随后在正交群的流形上进行融合，而剩余的线性残差则通过标准的加法融合进行处理。大量的实证结果表明，OrthoMerge 在缓解灾难性遗忘和维持模型跨多样化任务性能方面具有显著效果。