To overcome the overparameterized problem in Pre-trained Language Models (PLMs), pruning is widely used as a simple and straightforward compression method by directly removing unimportant weights. Previous first-order methods successfully compress PLMs to extremely high sparsity with little performance drop. These methods, such as movement pruning, use first-order information to prune PLMs while fine-tuning the remaining weights. In this work, we argue fine-tuning is redundant for first-order pruning, since first-order pruning is sufficient to converge PLMs to downstream tasks without fine-tuning. Under this motivation, we propose Static Model Pruning (SMP), which only uses first-order pruning to adapt PLMs to downstream tasks while achieving the target sparsity level. In addition, we also design a new masking function and training objective to further improve SMP. Extensive experiments at various sparsity levels show SMP has significant improvements over first-order and zero-order methods. Unlike previous first-order methods, SMP is also applicable to low sparsity and outperforms zero-order methods. Meanwhile, SMP is more parameter efficient than other methods due to it does not require fine-tuning.

翻译：为了克服预训练语言模型（PLMs）中的过度参数化问题，剪枝作为一种直接且简单的压缩方法被广泛使用，通过直接移除不重要的权重来实现压缩。先前的一阶方法成功地将PLMs压缩到极高稀疏度且性能损失极小。这些方法（如移动剪枝）利用一阶信息在剪枝PLMs的同时微调剩余权重。在本工作中，我们认为对于一阶剪枝而言，微调是冗余的，因为一阶剪枝足以使PLMs适应下游任务而无需微调。基于这一动机，我们提出了静态模型剪枝（SMP），该方法仅使用一阶剪枝来使PLMs适应下游任务，同时达到目标稀疏度水平。此外，我们还设计了一种新的掩蔽函数和训练目标以进一步改进SMP。在不同稀疏度水平上的大量实验表明，SMP相较于一阶和零阶方法均有显著提升。与先前的一阶方法不同，SMP同样适用于低稀疏度场景，并且性能优于零阶方法。同时，由于SMP无需微调，其参数效率也高于其他方法。