As the size of large language models continue to scale, so does the computational resources required to run it. Spiking Neural Networks (SNNs) have emerged as an energy-efficient approach to deep learning that leverage sparse and event-driven activations to reduce the computational overhead associated with model inference. While they have become competitive with non-spiking models on many computer vision tasks, SNNs have also proven to be more challenging to train. As a result, their performance lags behind modern deep learning, and we are yet to see the effectiveness of SNNs in language generation. In this paper, inspired by the Receptance Weighted Key Value (RWKV) language model, we successfully implement `SpikeGPT', a generative language model with binary, event-driven spiking activation units. We train the proposed model on two model variants: 45M and 216M parameters. To the best of our knowledge, SpikeGPT is the largest backpropagation-trained SNN model to date, rendering it suitable for both the generation and comprehension of natural language. We achieve this by modifying the transformer block to replace multi-head self attention to reduce quadratic computational complexity O(N^2) to linear complexity O(N) with increasing sequence length. Input tokens are instead streamed in sequentially to our attention mechanism (as with typical SNNs). Our preliminary experiments show that SpikeGPT remains competitive with non-spiking models on tested benchmarks, while maintaining 20x fewer operations when processed on neuromorphic hardware that can leverage sparse, event-driven activations.

翻译：随着大型语言模型规模的持续增长，其运行所需的计算资源也在不断增加。脉冲神经网络（SNNs）作为一种能效优化的深度学习方法，通过利用稀疏性和事件驱动激活来降低模型推理的计算开销。尽管SNNs在诸多计算机视觉任务中已具备与非脉冲模型相竞争的能力，但其训练难度仍被证明更高。因此，SNNs的性能仍落后于现代深度学习，其在语言生成任务中的有效性尚未得到验证。本文受接受加权键值（RWKV）语言模型的启发，成功实现了"SpikeGPT"——一种采用二进制事件驱动脉冲激活单元的生成式语言模型。我们在两种模型变体上训练所提出的模型：4500万参数和2.16亿参数。据我们所知，SpikeGPT是目前参数规模最大的通过反向传播训练的SNN模型，适用于自然语言的生成与理解任务。为实现这一目标，我们修改了Transformer模块，用替代机制替换多头自注意力，将随序列长度呈二次方复杂度O(N²)的计算复杂度降低至线性复杂度O(N)。输入令牌以序列方式依次流式进入我们的注意力机制（与典型SNNs一致）。初步实验表明，SpikeGPT在基准测试中与非脉冲模型保持竞争力，同时在采用可利用稀疏事件驱动激活的神经形态硬件处理时，操作次数减少20倍。