State-of-the-art language models are becoming increasingly large in an effort to achieve the highest performance on large corpora of available textual data. However, the sheer size of the Transformer architectures makes it difficult to deploy models within computational, environmental or device-specific constraints. We explore data-driven compression of existing pretrained models as an alternative to training smaller models from scratch. To do so, we scale Kronecker-factored curvature approximations of the target loss landscape to large language models. In doing so, we can compute both the dynamic allocation of structures that can be removed as well as updates of remaining weights that account for the removal. We provide a general framework for unstructured, semi-structured and structured pruning and improve upon weight updates to capture more correlations between weights, while remaining computationally efficient. Experimentally, our method can prune rows and columns from a range of OPT models and Llamav2-7B by 20%-30%, with a negligible loss in performance, and achieve state-of-the-art results in unstructured and semi-structured pruning of large language models.

翻译：最先进的语言模型在追求大规模文本语料中最高性能的过程中，其规模日益扩大。然而，Transformer架构的庞大尺寸使得模型部署面临计算、环境或设备特定限制的挑战。本文探索了数据驱动的预训练模型压缩方法，作为从头训练小型模型的替代方案。为此，我们将目标损失景观的Kronecker因子曲率近似扩展到大型语言模型中。通过这种方法，我们既能动态分配可移除的结构，又能计算补偿移除影响的剩余权重更新。我们提出了一个适用于非结构化、半结构化和结构化剪枝的通用框架，在保持计算效率的同时，改进了权重更新以捕捉权重间更多相关性。实验表明，我们的方法能够从多种OPT模型和Llamav2-7B中剪除20%-30%的行和列，且性能损失极小，并在大型语言模型的非结构化和半结构化剪枝中取得了最先进的结果。