As their size increases, Large Languages Models (LLMs) are natural candidates for network pruning methods: approaches that drop a subset of network weights while striving to preserve performance. Existing methods, however, require either retraining, which is rarely affordable for billion-scale LLMs, or solving a weight reconstruction problem reliant on second-order information, which may also be computationally expensive. In this paper, we introduce a novel, straightforward yet effective pruning method, termed Wanda (Pruning by Weights and activations), designed to induce sparsity in pretrained LLMs. Motivated by the recent observation of emergent large magnitude features in LLMs, our approach prunes weights with the smallest magnitudes multiplied by the corresponding input activations, on a per-output basis. Notably, Wanda requires no retraining or weight update, and the pruned LLM can be used as is. We conduct a thorough evaluation of our method Wanda on LLaMA and LLaMA-2 across various language benchmarks. Wanda significantly outperforms the established baseline of magnitude pruning and performs competitively against recent method involving intensive weight update. Code is available at https://github.com/locuslab/wanda.

翻译：随着模型规模不断增大，大语言模型自然成为网络剪枝方法的理想对象——这类方法通过舍弃部分网络权重来在保持性能的前提下压缩模型。然而现有方法要么需要重训练（这在十亿级参数的LLM上难以承受），要么依赖二阶信息求解权重重建问题，同样可能带来高昂计算成本。本文提出一种名为Wanda（基于权重与激活值的剪枝）的新型方法，这是一种直接高效的剪枝策略，旨在为预训练LLM引入稀疏性。受近期关于LLM中涌现大范数特征的观察启发，我们的方法以输出单元为单位，剪除绝对值最小的权重与其对应输入激活值的乘积。值得注意的是，Wanda无需重训练或权重更新，剪枝后的LLM可直接使用。我们在LLaMA和LLaMA-2的多个语言基准测试上对Wanda进行了全面评估，结果显示该方法显著优于传统幅度剪枝基准，并与需要繁重权重更新的最新方法性能相当。相关代码已开源在https://github.com/locuslab/wanda。