Spiking neural networks (SNN) have recently emerged as alternatives to traditional neural networks, owing to energy efficiency benefits and capacity to better capture biological neuronal mechanisms. However, the classic backpropagation algorithm for training traditional networks has been notoriously difficult to apply to SNN due to the hard-thresholding and discontinuities at spike times. Therefore, a large majority of prior work believes exact gradients for SNN w.r.t. their weights do not exist and has focused on approximation methods to produce surrogate gradients. In this paper, (1) by applying the implicit function theorem to SNN at the discrete spike times, we prove that, albeit being non-differentiable in time, SNNs have well-defined gradients w.r.t. their weights, and (2) we propose a novel training algorithm, called \emph{forward propagation} (FP), that computes exact gradients for SNN. FP exploits the causality structure between the spikes and allows us to parallelize computation forward in time. It can be used with other algorithms that simulate the forward pass, and it also provides insights on why other related algorithms such as Hebbian learning and also recently-proposed surrogate gradient methods may perform well.

翻译：脉冲神经网络（SNN）近年来因能效优势及更佳的生物神经元机制捕获能力，已成为传统神经网络的替代方案。然而，由于脉冲时刻存在的硬阈值与不连续性，训练传统网络的经典反向传播算法难以直接应用于SNN。因此，大多数先前研究认为SNN相对于权重的精确梯度不存在，并聚焦于近似方法以生成替代梯度。本文中：(1) 通过将隐函数定理应用于离散脉冲时刻的SNN，我们证明尽管SNN在时间上不可微，但其相对于权重的梯度定义良好；(2) 我们提出一种名为“前向传播”（FP）的新型训练算法，可计算SNN的精确梯度。FP利用脉冲间的因果结构，实现计算在时间维度的前向并行化。该算法可与其他前向仿真算法协同使用，并揭示了为何赫布学习及近期提出的替代梯度方法等关联算法可能表现良好。