Spiking neural networks (SNNs) have manifested remarkable advantages in power consumption and event-driven property during the inference process. To take full advantage of low power consumption and improve the efficiency of these models further, the pruning methods have been explored to find sparse SNNs without redundancy connections after training. However, parameter redundancy still hinders the efficiency of SNNs during training. In the human brain, the rewiring process of neural networks is highly dynamic, while synaptic connections maintain relatively sparse during brain development. Inspired by this, here we propose an efficient evolutionary structure learning (ESL) framework for SNNs, named ESL-SNNs, to implement the sparse SNN training from scratch. The pruning and regeneration of synaptic connections in SNNs evolve dynamically during learning, yet keep the structural sparsity at a certain level. As a result, the ESL-SNNs can search for optimal sparse connectivity by exploring all possible parameters across time. Our experiments show that the proposed ESL-SNNs framework is able to learn SNNs with sparse structures effectively while reducing the limited accuracy. The ESL-SNNs achieve merely 0.28% accuracy loss with 10% connection density on the DVS-Cifar10 dataset. Our work presents a brand-new approach for sparse training of SNNs from scratch with biologically plausible evolutionary mechanisms, closing the gap in the expressibility between sparse training and dense training. Hence, it has great potential for SNN lightweight training and inference with low power consumption and small memory usage.

翻译：脉冲神经网络（SNNs）在推理过程中展现出显著的功耗优势与事件驱动特性。为充分发挥低功耗优势并进一步提升模型效率，研究者们探索了剪枝方法，旨在训练后获得无冗余连接的稀疏SNNs。然而，参数冗余仍制约着SNNs的训练效率。人脑中神经网络的连接重构过程具有高度动态性，而突触连接在大脑发育过程中始终保持相对稀疏。受此启发，本文提出了一种名为ESL-SNNs的高效进化结构学习框架，以从头实现稀疏SNN训练。SNNs中的突触连接在训练过程中动态演化剪枝与再生，同时将结构稀疏性维持在特定水平。通过探索时间维度上的所有潜在参数，ESL-SNNs能够搜索最优稀疏连接结构。实验表明，所提出的ESL-SNNs框架能在降低有限精度损失的同时有效学习稀疏结构的SNNs。在DVS-Cifar10数据集上，当连接密度为10%时，ESL-SNNs仅产生0.28%的精度损失。本工作提出了一种基于生物可解释进化机制的全新SNN从头稀疏训练方法，弥合了稀疏训练与密集训练在表达能力上的差距，为低功耗、小内存占用的SNN轻量化训练与推理提供了巨大潜力。