As the size of large language models continue to scale, so does the computational resources required to run it. Spiking Neural Networks (SNNs) have emerged as an energy-efficient approach to deep learning that leverage sparse and event-driven activations to reduce the computational overhead associated with model inference. While they have become competitive with non-spiking models on many computer vision tasks, SNNs have also proven to be more challenging to train. As a result, their performance lags behind modern deep learning, and we are yet to see the effectiveness of SNNs in language generation. In this paper, inspired by the Receptance Weighted Key Value (RWKV) language model, we successfully implement `SpikeGPT', a generative language model with binary, event-driven spiking activation units. We train the proposed model on two model variants: 45M and 216M parameters. To the best of our knowledge, SpikeGPT is the largest backpropagation-trained SNN model to date, rendering it suitable for both the generation and comprehension of natural language. We achieve this by modifying the transformer block to replace multi-head self attention to reduce quadratic computational complexity O(N^2) to linear complexity O(N) with increasing sequence length. Input tokens are instead streamed in sequentially to our attention mechanism (as with typical SNNs). Our preliminary experiments show that SpikeGPT remains competitive with non-spiking models on tested benchmarks, while maintaining 20x fewer operations when processed on neuromorphic hardware that can leverage sparse, event-driven activations.

翻译：随着大型语言模型规模的不断增长，其运行所需的计算资源也在持续增加。脉冲神经网络（SNNs）作为一种能效优化的深度学习方法，通过利用稀疏且事件驱动的激活机制来降低模型推理的计算开销。尽管SNNs在诸多计算机视觉任务中已具备与非脉冲模型相竞争的能力，但其训练难度显著更高。因此，SNNs的性能仍落后于现代深度学习技术，其在语言生成任务中的有效性尚待验证。本文受接受加权键值（RWKV）语言模型启发，成功实现了"SpikeGPT"——一种采用二元事件驱动脉冲激活单元的生成式语言模型。我们以两种模型变体（4500万参数与2.16亿参数）对所提模型进行训练。据我们所知，SpikeGPT是迄今为止最大规模的反向传播训练SNN模型，能够胜任自然语言的生成与理解任务。我们通过改进Transformer模块，用线性复杂度O(N)替代多头自注意力机制的二次计算复杂度O(N²)（随序列长度增长），其中输入令牌按顺序流式传入注意力机制（如典型SNNs方式）。初步实验表明，SpikeGPT在测试基准中与非脉冲模型保持竞争力，同时在可充分利用稀疏事件驱动激活的神经形态硬件上，运算操作量降低20倍。