With the proliferation of domain-specific models, model merging has emerged as a set of techniques that combine the capabilities of multiple models into one that can multitask without the cost of additional training. In this paper, we propose a new model merging technique, Drop and rEscaLe via sampLing with mAgnitude (DELLA-Merging), that employs a novel pruning technique, MAGPRUNE, which shows significant advantages over DARE and TIES. MAGPRUNE first ranks the parameters in order of their magnitude and assigns higher dropout probabilities (p) to parameters with lower ranks corresponding to lower magnitudes. To approximate the original embeddings, MAGPRUNE employs a rescaling operation on the parameters that survive the random dropping by 1/(1 - p). On three different expert models considered for merging (LM, Math, Code) and corresponding benchmark datasets (AlpacaEval, GSM8K, MBPP), DELLA shows an average improvement of 2.4 points over baseline methods employing delta parameter pruning (an improvement of 3.6 points over TIES, 1.2 points over DARE), and 11.1 points over the no-pruning baseline (TA). We release the source code at: https://github.com/declare-lab/della.

翻译：随着领域专用模型的激增，模型合并已成为一类技术，能够将多个模型的能力整合到一个模型中，使其无需额外训练成本即可执行多任务。本文提出了一种新的模型合并技术——通过幅度采样进行丢弃与重缩放（DELLA-Merging），该技术采用了一种新颖的剪枝方法MAGPRUNE，相较于DARE和TIES展现出显著优势。MAGPRUNE首先根据参数幅度大小对其进行排序，并为幅度较低、排名靠后的参数分配较高的丢弃概率（p）。为逼近原始嵌入，MAGPRUNE对随机丢弃后保留的参数执行重缩放操作，缩放因子为1/(1-p)。在用于合并的三个不同专家模型（语言模型、数学模型、代码模型）及对应基准数据集（AlpacaEval、GSM8K、MBPP）上，DELLA相比采用增量参数剪枝的基线方法平均提升2.4个百分点（较TIES提升3.6个百分点，较DARE提升1.2个百分点），较无剪枝基线（TA）提升11.1个百分点。源代码已发布于：https://github.com/declare-lab/della。