Large language models have become the cornerstone of natural language processing, but their use comes with substantial costs in terms of compute and memory resources. Sparsification provides a solution to alleviate these resource constraints, and recent works have shown that trained models can be sparsified post-hoc. Existing sparsification techniques face challenges as they need additional data structures and offer constrained speedup with current hardware. In this paper we present SliceGPT, a new post-training sparsification scheme which replaces each weight matrix with a smaller (dense) matrix, reducing the embedding dimension of the network. Through extensive experimentation, we show that SliceGPT can remove up to 25% of the model parameters (including embeddings) for LLAMA2-70B, OPT 66B and Phi-2 models while maintaining 99%, 99% and 90% zero-shot task performance of the dense model respectively. Our sliced models run on fewer GPUs and run faster without any additional code optimization: on 24GB consumer GPUs we reduce the total compute for inference on LLAMA2-70B to 64% of that of the dense model; on 40GB A100 GPUs we reduce it to 66%. We offer a new insight, computational invariance in transformer networks, which enables SliceGPT and we hope it will inspire and enable future avenues to reduce memory and computation demands for pre-trained models. Code is available at: https://github.com/microsoft/TransformerCompression

翻译：大语言模型已成为自然语言处理的基石，但其使用伴随着计算和内存资源方面的巨大成本。稀疏化提供了一种缓解这些资源约束的解决方案，且近期研究表明训练后的模型可以事后进行稀疏化。现有稀疏化技术面临挑战，因为它们需要额外的数据结构，并且在当前硬件上只能提供有限的加速效果。在本文中，我们提出SliceGPT，一种新的训练后稀疏化方案，它将每个权重矩阵替换为一个更小的（稠密）矩阵，从而减小网络的嵌入维度。通过大量实验，我们证明SliceGPT能够移除LLAMA2-70B、OPT 66B和Phi-2模型中高达25%的模型参数（包括嵌入层），同时分别保持稠密模型99%、99%和90%的零样本任务性能。我们的切片模型在更少的GPU上运行，且无需额外代码优化即可实现更快的运行速度：在24GB消费级GPU上，我们将LLAMA2-70B推理的总计算量降低至稠密模型的64%；在40GB A100 GPU上，我们将其降低至66%。我们提出了一个新的见解——Transformer网络中的计算不变性，这一性质使得SliceGPT成为可能，并且我们希望这将启发并推动未来在减少预训练模型内存和计算需求方面的研究。代码可在以下链接获取：https://github.com/microsoft/TransformerCompression