This paper introduces SparseOptimizer, a novel deep learning optimizer that exploits Moreau-Yosida regularization to naturally induce sparsity in large language models such as BERT, ALBERT and GPT. Key to the design of SparseOptimizer is an embedded shrinkage operator, which imparts sparsity directly within the optimization process. This operator, backed by a sound theoretical framework, includes an analytical solution, thereby reinforcing the optimizer's robustness and efficacy. Crucially, SparseOptimizer's plug-and-play functionality eradicates the need for code modifications, making it a universally adaptable tool for a wide array of large language models. Empirical evaluations on benchmark datasets such as GLUE, RACE, SQuAD1, and SQuAD2 confirm that SparseBERT and SparseALBERT, when sparsified using SparseOptimizer, achieve performance comparable to their dense counterparts, BERT and ALBERT, while significantly reducing their parameter count. Further, this work proposes an innovative optimizer-compiler co-design strategy, demonstrating the potential of inference acceleration (\textbf{3.37x}, \textbf{6.30x}, and \textbf{7.15x} in comparison with Pytorch, TensorFlow, and LLVM generic compile, respectively) in SparseBERT when paired with an appropriately designed compiler. This study represents a significant step forward in the evolution of efficient, scalable, and high-performing large language models, setting a precedent for future exploration and optimization in this domain. The SparseOptimizer code and SparseALBERT model will be publicly available upon paper acceptance.

翻译：本文提出稀疏优化器（SparseOptimizer），一种利用Moreau-Yosida正则化在BERT、ALBERT、GPT等大型语言模型中自然诱导稀疏性的新型深度学习优化器。该优化器的核心设计在于嵌入收缩算子，可在优化过程中直接赋予稀疏性。该算子依托完善的理论框架，内含解析解，从而增强优化器的鲁棒性与效能。尤为关键的是，稀疏优化器的即插即用特性消除了代码修改需求，使其成为适用于各类大型语言模型的通用适配工具。在GLUE、RACE、SQuAD1与SQuAD2等基准数据集上的实验评估证实：经稀疏优化器稀疏化后的SparseBERT与SparseALBERT，在显著降低参数量的同时，性能可与未稀疏化的BERT、ALBERT相媲美。此外，本研究提出创新的优化器-编译器协同设计策略，展示了当SparseBERT搭配适当设计的编译器时，推理加速效果显著（相较PyTorch、TensorFlow及LLVM通用编译分别达到**3.37倍**、**6.30倍**和**7.15倍**）。该研究为发展高效、可扩展且高性能的大型语言模型迈出重要一步，为后续该领域的探索与优化确立了新范式。稀疏优化器代码及SparseALBERT模型将在论文接收后公开发布。