Herein the topics of (natural) gradient descent, data decorrelation, and approximate methods for backpropagation are brought into a common discussion. Natural gradient descent illuminates how gradient vectors, pointing at directions of steepest descent, can be improved by considering the local curvature of loss landscapes. We extend this perspective and show that to fully solve the problem illuminated by natural gradients in neural networks, one must recognise that correlations in the data at any linear transformation, including node responses at every layer of a neural network, cause a non-orthonormal relationship between the model's parameters. To solve this requires a method for decorrelating inputs at each individual layer of a neural network. We describe a range of methods which have been proposed for decorrelation and whitening of node output, and expand on these to provide a novel method specifically useful for distributed computing and computational neuroscience. Implementing decorrelation within multi-layer neural networks, we can show that not only is training via backpropagation sped up significantly but also existing approximations of backpropagation, which have failed catastrophically in the past, benefit significantly in their accuracy and convergence speed. This has the potential to provide a route forward for approximate gradient descent methods which have previously been discarded, training approaches for analogue and neuromorphic hardware, and potentially insights as to the efficacy and utility of decorrelation processes in the brain.

翻译：本文将（自然）梯度下降、数据去相关和反向传播的近似方法纳入统一讨论。自然梯度下降阐明了如何通过考虑损失曲面的局部曲率来改进指向最速下降方向的梯度向量。我们拓展了这一视角，并证明要完全解决神经网络中自然梯度所揭示的问题，必须认识到任何线性变换中的数据相关性（包括神经网络每一层的节点响应）会导致模型参数间存在非正交归一关系。解决此问题需要一种在神经网络各独立层对输入进行去相关的方法。我们综述了已提出的节点输出去相关与白化方法，并在此基础上提出一种特别适用于分布式计算与计算神经科学的新方法。通过在多层神经网络中实现去相关，我们不仅证明反向传播训练速度得到显著提升，而且过去曾遭遇灾难性失败的现有反向传播近似方法，其精度与收敛速度也获得实质性改善。这有望为先前被弃用的近似梯度下降方法、模拟与神经形态硬件的训练方案提供新的发展路径，并可能为大脑中去相关过程的效能与作用机制提供见解。