Current Large Language Models (LLMs) are primarily based on large-scale dense matrix multiplications. Inspired by the brain's information processing mechanism, we explore the fundamental question: how to effectively integrate the brain's spiking-driven characteristics into LLM inference. Spiking Neural Networks (SNNs) possess spike-driven characteristics, and some works have attempted to combine SNNs with Transformers. However, achieving spike-driven LLMs with billions of parameters, relying solely on sparse additions, remains a challenge in the SNN field. To address the issues of limited representational capacity and sparsity in existing spike encoding schemes at the LLM level, we propose SDLLM, a spike-driven large language model that eliminates dense matrix multiplications through sparse addition operations. Specifically, we use the plug-and-play gamma-SQP two-step spike encoding method to ensure that the quantization process aligns with the model's semantic space, mitigating representation degradation caused by binary spikes. Furthermore, we introduce bidirectional encoding under symmetric quantization and membrane potential clipping mechanisms, leading to spike trains with no or low firing counts dominating, significantly reducing the model's spike firing rate, while halving the number of time steps. Experimental results show that SDLLM not only significantly reduces inference costs but also achieves state-of-the-art task performance under the spike-based paradigm. For example, compared to previous spike-based LLMs, SDLLM reduces energy consumption by 7x and improves accuracy by 4.2%. Our model provides inspiration for the architecture design of the next generation of event-driven neuromorphic chips.

翻译：当前大语言模型主要基于大规模稠密矩阵乘法。受大脑信息处理机制启发，我们探讨一个基本问题：如何有效整合大脑的脉冲驱动特性于大语言模型推理中。脉冲神经网络具有脉冲驱动特性，已有研究尝试将其与Transformer结合，但在SNN领域实现仅依赖稀疏加法的数十亿参数脉冲驱动大语言模型仍是挑战。为解决现有脉冲编码方案在大语言模型层面表达能力受限和稀疏性不足的问题，我们提出SDLLM——一种通过稀疏加法运算消除稠密矩阵乘法的脉冲驱动大语言模型。具体而言，我们采用即插即用的gamma-SQP两步脉冲编码方法，确保量化过程与模型语义空间对齐，缓解二元脉冲导致的表示退化。同时引入对称量化下的双向编码与膜电位裁剪机制，使脉冲序列中无/低发放率占主导，显著降低模型脉冲发放率的同时将时间步减半。实验表明，SDLLM不仅大幅降低推理成本，还在脉冲范式下达到最先进任务性能。例如，相较先前脉冲大语言模型，SDLLM能耗降低7倍，准确率提升4.2%。本模型为下一代事件驱动神经形态芯片的架构设计提供了启示。