Large Language Models (LLMs) have achieved remarkable performance across tasks but remain energy-intensive due to dense matrix operations. Spiking neural networks (SNNs) improve energy efficiency by replacing dense matrix multiplications with sparse accumulations. Their sparse spike activity enables efficient LLMs deployment on edge devices. However, prior SNN-based LLMs often sacrifice performance for efficiency, and recovering accuracy typically requires full pretraining, which is costly and impractical. To address this, we propose SpikingMamba, an energy-efficient SNN-based LLMs distilled from Mamba that improves energy efficiency with minimal accuracy sacrifice. SpikingMamba integrates two key components: (a) SI-LIF, a signed-integer spiking neuron that preserves semantic polarity through signed multi-level spike representations. (b) A training-exclusive Smoothed Gradient Compensation (SGC) path mitigating quantization loss while preserving spike-driven efficiency. We employ a single-stage distillation strategy to transfer the zero-shot ability of pretrained Mamba and further enhance it via reinforcement learning (RL). Experiments show that SpikingMamba-1.3B achieves a 4.76$\times$ energy benefit, with only a 4.78\% zero-shot accuracy gap compared to the original Mamba. The model achieves a further 2.55\% accuracy improvement after RL, narrowing the performance gap from 4.78\% to 2.23\%. Code is available at: https://github.com/HuuYuLong/SpikingMamba .

翻译：大型语言模型在各类任务中展现出卓越性能，但因密集矩阵运算而能耗巨大。脉冲神经网络通过用稀疏累加替代密集矩阵乘法提升能效，其稀疏脉冲活动使语言模型能在边缘设备上高效部署。然而，现有基于脉冲神经网络的语言模型常以牺牲性能换取效率，而恢复精度通常需要全量预训练，成本高昂且不切实际。为此，我们提出SpikingMamba——一种从Mamba蒸馏得到的基于脉冲神经网络的节能语言模型，在最小精度损失下提升能效。SpikingMamba集成两大核心组件：(a) SI-LIF有符号整数脉冲神经元，通过带符号多级脉冲表征保留语义极性；(b) 训练专用的平滑梯度补偿路径，在保持脉冲驱动效率的同时减轻量化损失。我们采用单阶段蒸馏策略迁移预训练Mamba的零样本能力，并通过强化学习进一步增强。实验表明，SpikingMamba-1.3B相较于原始Mamba实现4.76倍能效提升，零样本精度仅损失4.78%。经强化学习后模型精度进一步提升2.55%，性能差距从4.78%缩小至2.23%。代码已开源：https://github.com/HuuYuLong/SpikingMamba。