As their size increases, Large Languages Models (LLMs) are natural candidates for network pruning methods: approaches that drop a subset of network weights while striving to preserve performance. Existing methods, however, require either retraining, which is rarely affordable for billion-scale LLMs, or solving a weight reconstruction problem reliant on second-order information, which may also be computationally expensive. In this paper, we introduce a novel, straightforward yet effective pruning method, termed Wanda (Pruning by Weights and activations), designed to induce sparsity in pretrained LLMs. Motivated by the recent observation of emergent large magnitude features in LLMs, our approach prunes weights with the smallest magnitudes multiplied by the corresponding input activations, on a per-output basis. Notably, Wanda requires no retraining or weight update, and the pruned LLM can be used as is. We conduct a thorough evaluation of our method Wanda on LLaMA and LLaMA-2 across various language benchmarks. Wanda significantly outperforms the established baseline of magnitude pruning and performs competitively against recent method involving intensive weight update. Code is available at https://github.com/locuslab/wanda.

翻译：随着大语言模型（LLMs）规模的增长，网络剪枝方法——在尽可能保持模型性能的前提下丢弃部分网络权重的方法——自然成为其优化方向。然而，现有方法要么需要重新训练（这对十亿级参数的LLMs而言几乎不可行），要么需要依赖二阶信息求解权重重构问题（这可能带来高昂计算成本）。本文提出一种新颖、简洁且高效的剪枝方法Wanda（通过权重与激活进行剪枝），专为预训练LLMs的稀疏化设计。受近期关于LLMs中涌现大规模特征现象的启发，该方法以逐输出为单位，对权重与对应输入激活的乘积中最小值对应的权重进行剪枝。值得注意的是，Wanda无需重新训练或权重更新，剪枝后的LLMs可直接使用。我们在LLaMA与LLaMA-2上针对多种语言基准任务进行了全面评估。实验表明，Wanda显著优于经典的幅度剪枝基线方法，并在性能上与近期涉及密集权重更新的先进方法相当。代码已开源至 https://github.com/locuslab/wanda。