Parameter-efficient fine-tuning (PEFT) techniques make it possible to efficiently adapt a language model to create "expert" models that specialize to new tasks or domains. Recent techniques in model merging and compositional generalization leverage these expert models by dynamically composing modules to improve zero/few-shot generalization. Despite the efficiency of PEFT methods, the size of expert models can make it onerous to retrieve expert models per query over high-latency networks like the Internet or serve multiple experts on a single GPU. To address these issues, we present ComPEFT, a novel method for compressing fine-tuning residuals (task vectors) of PEFT based models. ComPEFT employs sparsification and ternary quantization to reduce the size of the PEFT module without performing any additional retraining while preserving or enhancing model performance. In extensive evaluation across T5, T0, and LLaMA-based models with 200M - 65B parameters, ComPEFT achieves compression ratios of 8x - 50x. In particular, we show that ComPEFT improves with scale - stronger models exhibit higher compressibility and better performance. For example, we show that ComPEFT applied to LLaMA outperforms QLoRA by 4.16% on MMLU with a storage size reduction of up to 26x. In addition, we show that the compressed experts produced by ComPEFT maintain few-shot compositional generalization capabilities, facilitate efficient communication and computation, and exhibit enhanced performance when merged. Lastly, we provide an analysis of different method components, compare it with other PEFT methods, and test ComPEFT's efficacy for compressing the residual of full-finetuning. Our code is available at https://github.com/prateeky2806/compeft.

翻译：参数高效微调（PEFT）技术使得高效适配语言模型以创建针对新任务或领域特化的"专家"模型成为可能。近期模型合并与组合泛化技术通过动态组合这些专家模型模块，提升了零样本/少样本泛化能力。尽管PEFT方法具有高效性，但专家模型的规模导致通过互联网等高延迟网络按查询检索专家模型或在单GPU上服务多个专家模型时存在负担。为解决这些问题，我们提出ComPEFT——一种用于压缩PEFT模型微调残差（任务向量）的新方法。ComPEFT通过稀疏化和三元量化技术，在无需额外重训练的前提下，显著压缩PEFT模块规模，同时保持或提升模型性能。在基于T5、T0和LLaMA的200M-65B参数规模模型上的广泛评估表明，ComPEFT实现了8倍至50倍的压缩比。特别值得注意的是，ComPEFT的压缩效果随模型规模提升而增强——更强模型展现出更高的可压缩性和更优性能。例如，应用ComPEFT的LLaMA模型在MMLU基准上较QLoRA提升4.16%，同时存储体积最高缩减26倍。此外，实验证明ComPEFT生成的压缩专家模型不仅保持了少样本组合泛化能力，还促进了高效通信与计算，并在合并时展现出增强性能。最后，我们对不同方法组件进行了分析，与其他PEFT方法进行了比较，并测试了ComPEFT在全量微调残差压缩中的有效性。代码已开源至https://github.com/prateeky2806/compeft。