Backpropagation (BP) is the most important gradient estimation method for training neural networks in deep learning. However, the literature shows that neural networks trained by BP are vulnerable to adversarial attacks. We develop the likelihood ratio (LR) method, a new gradient estimation method, for training a broad range of neural network architectures, including convolutional neural networks, recurrent neural networks, graph neural networks, and spiking neural networks, without recursive gradient computation. We propose three methods to efficiently reduce the variance of the gradient estimation in the neural network training process. Our experiments yield numerical results for training different neural networks on several datasets. All results demonstrate that the LR method is effective for training various neural networks and significantly improves the robustness of the neural networks under adversarial attacks relative to the BP method.

翻译：反向传播（BP）是深度学习中训练神经网络最重要的梯度估计方法。然而，文献表明，通过BP训练的神经网络容易受到对抗性攻击。我们开发了似然比（LR）方法——一种新的梯度估计方法，用于训练包括卷积神经网络、循环神经网络、图神经网络和脉冲神经网络在内的广泛神经网络架构，而无需递归梯度计算。我们提出了三种方法，有效降低了神经网络训练过程中梯度估计的方差。实验给出了在不同数据集上训练多种神经网络的数值结果。所有结果均表明，LR方法能有效训练各类神经网络，并且与BP方法相比，显著提升了神经网络在对抗性攻击下的鲁棒性。