Model merging combines knowledge from separately fine-tuned models, yet success factors remain poorly understood. While recent work treats mergeability as an intrinsic property, we show with an architecture-agnostic framework that it fundamentally depends on both the merging method and the partner tasks. Using linear optimization over a set of interpretable pairwise metrics (e.g., gradient L2 distance), we uncover properties correlating with post-merge performance across four merging methods. We find substantial variation in success drivers (46.7% metric overlap; 55.3% sign agreement), revealing method-specific "fingerprints". Crucially, however, subspace overlap and gradient alignment metrics consistently emerge as foundational, method-agnostic prerequisites for compatibility. These findings provide a diagnostic foundation for understanding mergeability and motivate future fine-tuning strategies that explicitly encourage these properties.
翻译:模型合并技术旨在整合来自独立微调模型的知识,然而其成功的关键因素至今仍未得到充分理解。尽管近期研究将可合并性视为模型的内在属性,我们通过一个与架构无关的框架证明:该属性本质上同时取决于合并方法与目标任务对。基于一组可解释的成对度量指标(如梯度L2距离)进行线性优化分析,我们发现了与四种合并方法在合并后性能均存在关联的共性属性。研究发现不同方法间的成功驱动因素存在显著差异(度量指标重叠率46.7%;符号一致性55.3%),揭示了具有方法特异性的“指纹特征”。然而至关重要的是,子空间重叠度与梯度对齐度指标始终作为兼容性的基础性、方法无关的前提条件而显现。这些发现为理解模型可合并性提供了诊断基础,并激励未来开发能显式促进这些属性的微调策略。