Backpropagation (BP) is the dominant and most successful method for training parameters of deep neural network models. However, BP relies on two computationally distinct phases, does not provide a satisfactory explanation of biological learning, and can be challenging to apply for training of networks with discontinuities or noisy node dynamics. By comparison, node perturbation (NP) proposes learning by the injection of noise into the network activations, and subsequent measurement of the induced loss change. NP relies on two forward (inference) passes, does not make use of network derivatives, and has been proposed as a model for learning in biological systems. However, standard NP is highly data inefficient and unstable due to its unguided noise-based search process. In this work, we investigate different formulations of NP and relate it to the concept of directional derivatives as well as combining it with a decorrelating mechanism for layer-wise inputs. We find that a closer alignment with directional derivatives together with input decorrelation at every layer significantly enhances performance of NP learning, making its performance on the train set competitive with BP and allowing its application to noisy systems in which the noise process itself is inaccessible.

翻译：反向传播（BP）是训练深度神经网络参数最主流且最成功的方法。然而，BP依赖两个计算性质不同的阶段，未能对生物学习机制提供令人满意的解释，且难以应用于包含不连续性或节点动力学含噪声的网络训练。相比之下，节点扰动（NP）通过向网络激活中注入噪声并测量由此引发的损失变化来驱动学习。NP仅需两次前向（推理）传播，无需利用网络导数，且已被提出作为生物系统学习的候选模型。然而，标准NP因依赖无引导的噪声搜索过程而存在数据效率低下和训练不稳定的问题。本研究探索了NP的不同数学形式，将其与方向导数的概念关联，并引入层间输入去相关机制。我们发现，通过更紧密地契合方向导数并在每一层实施输入去相关，可显著提升NP学习的性能：在训练集上其表现可与BP媲美，且能应用于噪声过程本身不可观测的含噪系统。