Spiking neural networks (SNNs) offer a promising pathway to implement deep neural networks (DNNs) in a more energy-efficient manner since their neurons are sparsely activated and inferences are event-driven. However, there have been very few works that have demonstrated the efficacy of SNNs in language tasks partially because it is non-trivial to represent words in the forms of spikes and to deal with variable-length texts by SNNs. This work presents a "conversion + fine-tuning" two-step method for training SNNs for text classification and proposes a simple but effective way to encode pre-trained word embeddings as spike trains. We show empirically that after fine-tuning with surrogate gradients, the converted SNNs achieve comparable results to their DNN counterparts with much less energy consumption across multiple datasets for both English and Chinese. We also show that such SNNs are more robust to adversarial attacks than DNNs.

翻译：脉冲神经网络（SNNs）因其神经元稀疏激活且推理过程为事件驱动，为实现深度神经网络（DNNs）提供了一条更具能效前景的路径。然而，目前极少有工作证明SNNs在语言任务中的有效性，部分原因在于如何以脉冲形式表示词语以及如何通过SNNs处理变长文本并非易事。本研究提出一种用于文本分类SNN训练的“转换+微调”两步法，并提出一种简单而有效的方法将预训练词向量编码为脉冲序列。我们通过实验证明，在使用替代梯度进行微调后，转换得到的SNNs在多个中英文数据集上均能达到与对应DNNs相当的结果，同时能耗显著降低。我们还发现此类SNNs相比DNNs对对抗攻击具有更强的鲁棒性。