Scaling autoregressive large language models (LLMs) has driven unprecedented progress but comes with vast computational costs. In this work, we tackle these costs by leveraging unstructured sparsity within an LLM's feedforward layers, the components accounting for most of the model parameters and execution FLOPs. To achieve this, we introduce a new sparse packing format and a set of CUDA kernels designed to seamlessly integrate with the optimized execution pipelines of modern GPUs, enabling efficient sparse computation during LLM inference and training. To substantiate our gains, we provide a quantitative study of LLM sparsity, demonstrating that simple L1 regularization can induce over 99% sparsity with negligible impact on downstream performance. When paired with our kernels, we show that these sparsity levels translate into substantial throughput, energy efficiency, and memory usage benefits that increase with model scale. We will release all code and kernels under an open-source license to promote adoption and accelerate research toward establishing sparsity as a practical axis for improving the efficiency and scalability of modern foundation models.

翻译：扩展自回归大语言模型（LLM）推动了前所未有的进步，但也带来了巨大的计算成本。在这项工作中，我们通过利用LLM前馈层（该层占模型参数和执行FLOPs的大部分）内的非结构化稀疏性来应对这些成本。为此，我们引入了一种新的稀疏打包格式以及一套CUDA内核，旨在与现代GPU的优化执行流水线无缝集成，从而在LLM推理和训练过程中实现高效的稀疏计算。为了证实我们的改进，我们对LLM稀疏性进行了定量研究，证明简单的L1正则化可以诱导超过99%的稀疏性，而对下游性能的影响可以忽略不计。当与我们的内核配合使用时，我们表明这些稀疏性水平可转化为显著的吞吐量、能源效率和内存使用效益，且这些效益随模型规模增大而增加。我们将以开源许可证发布所有代码和内核，以促进采用并加速研究，旨在将稀疏性确立为改善现代基础模型效率和可扩展性的实用维度。