We consider the problem of accurate sparse fine-tuning of large language models (LLMs), that is, fine-tuning pretrained LLMs on specialized tasks, while inducing sparsity in their weights. On the accuracy side, we observe that standard loss-based fine-tuning may fail to recover accuracy, especially at high sparsities. To address this, we perform a detailed study of distillation-type losses, determining an L2-based distillation approach we term SquareHead which enables accurate recovery even at higher sparsities, across all model types. On the practical efficiency side, we show that sparse LLMs can be executed with speedups by taking advantage of sparsity, for both CPU and GPU runtimes. While the standard approach is to leverage sparsity for computational reduction, we observe that in the case of memory-bound LLMs sparsity can also be leveraged for reducing memory bandwidth. We exhibit end-to-end results showing speedups due to sparsity, while recovering accuracy, on T5 (language translation), Whisper (speech translation), and open GPT-type (MPT for text generation). For MPT text generation, we show for the first time that sparse fine-tuning can reach 75% sparsity without accuracy drops, provide notable end-to-end speedups for both CPU and GPU inference, and highlight that sparsity is also compatible with quantization approaches. Models and software for reproducing our results are provided in Section 6.

翻译：我们研究了大语言模型（LLMs）的精确稀疏微调问题，即在专用任务上微调预训练LLM的同时诱导权重稀疏性。在精度方面，我们观察到基于标准损失的微调方法可能无法恢复精度，尤其是在高稀疏度场景下。为此，我们系统研究了蒸馏类损失函数，确定了一种基于L2蒸馏的方法（称为SquareHead），该方法能够在所有模型类型的高稀疏度下实现精确恢复。在实际效率方面，我们展示了稀疏LLM可通过利用稀疏性在CPU和GPU运行时实现加速。标准方法通常利用稀疏性减少计算量，但我们发现对于内存受限型LLM，稀疏性还可用于降低内存带宽需求。我们展示了端到端结果：在T5（语言翻译）、Whisper（语音翻译）和开放GPT类型（用于文本生成的MPT）模型上，通过稀疏性实现加速的同时恢复了精度。对于MPT文本生成任务，我们首次证明稀疏微调可在不降低精度的情况下达到75%稀疏度，为CPU和GPU推理提供显著端到端加速，并强调稀疏性与量化方法兼容。重现结果的模型与软件详见第6节。