As large language models continue to scale in size rapidly, so too does the computational power required to run them. Event-based networks on neuromorphic devices offer a potential way to reduce energy consumption for inference significantly. However, to date, most event-based networks that can run on neuromorphic hardware, including spiking neural networks (SNNs), have not achieved task performance even on par with LSTM models for language modeling. As a result, language modeling on neuromorphic devices has seemed a distant prospect. In this work, we demonstrate the first-ever implementation of a language model on a neuromorphic device - specifically the SpiNNaker 2 chip - based on a recently published event-based architecture called the EGRU. SpiNNaker 2 is a many-core neuromorphic chip designed for large-scale asynchronous processing, while the EGRU is architected to leverage such hardware efficiently while maintaining competitive task performance. This implementation marks the first time a neuromorphic language model matches LSTMs, setting the stage for taking task performance to the level of large language models. We also demonstrate results on a gesture recognition task based on inputs from a DVS camera. Overall, our results showcase the feasibility of this neuro-inspired neural network in hardware, highlighting significant gains versus conventional hardware in energy efficiency for the common use case of single batch inference.

翻译：随着大规模语言模型规模的快速扩张，运行它们所需的计算能力也在急剧增长。神经形态设备上的事件驱动网络为显著降低推理能耗提供了潜在途径。然而，迄今为止，能在神经形态硬件上运行的大多数事件驱动网络（包括脉冲神经网络）在语言建模任务上的表现甚至未能达到LSTM模型水平。因此，在神经形态设备上进行语言建模似乎遥不可及。本研究首次展示了在神经形态设备——具体为SpiNNaker 2芯片——上实现语言模型的成果，其基础是近期发布的事件驱动架构EGRU。SpiNNaker 2是一款专为大规模异步处理设计的多核神经形态芯片，而EGRU架构则能高效利用此类硬件资源，同时保持具有竞争力的任务性能。这一实现标志着神经形态语言模型首次与LSTM模型性能持平，为将任务性能提升至大规模语言模型水平奠定基础。我们还展示了基于DVS相机输入的手势识别任务结果。总体而言，我们的研究结果证明了这种神经启发网络在硬件中的可行性，并凸显了在单批次推理这一常见应用场景下，相较于传统硬件在能效方面的显著提升。