Backpropagation (BP) is widely used for calculating gradients in deep neural networks (DNNs). Applied often along with stochastic gradient descent (SGD) or its variants, BP is considered as a de-facto choice in a variety of machine learning tasks including DNN training and adversarial attack/defense. Recently, a linear variant of BP named LinBP was introduced for generating more transferable adversarial examples for performing black-box attacks, by Guo et al. Although it has been shown empirically effective in black-box attacks, theoretical studies and convergence analyses of such a method is lacking. This paper serves as a complement and somewhat an extension to Guo et al.'s paper, by providing theoretical analyses on LinBP in neural-network-involved learning tasks, including adversarial attack and model training. We demonstrate that, somewhat surprisingly, LinBP can lead to faster convergence in these tasks in the same hyper-parameter settings, compared to BP. We confirm our theoretical results with extensive experiments.

翻译：反向传播（BP）被广泛用于深度神经网络（DNNs）中的梯度计算。通常与随机梯度下降（SGD）或其变体结合使用，BP被视为包括DNN训练和对抗攻击/防御在内的各类机器学习任务中的事实标准方法。近期，Guo等人引入了一种名为LinBP的线性变体反向传播方法，用于生成更具可迁移性的对抗样本以实施黑盒攻击。尽管该方法的黑盒攻击效果已通过实验验证有效，但其理论研究与收敛性分析仍属空白。本文通过为涉及神经网络的学习任务（包括对抗攻击和模型训练）中的LinBP提供理论分析，对Guo等人的工作进行了补充与适度扩展。我们令人惊讶地发现：在相同超参数设置下，LinBP相较于BP在这些任务中能实现更快的收敛速度。我们通过大量实验验证了理论结果。