Forward gradient learning computes a noisy directional gradient and is a biologically plausible alternative to backprop for learning deep neural networks. However, the standard forward gradient algorithm, when applied naively, suffers from high variance when the number of parameters to be learned is large. In this paper, we propose a series of architectural and algorithmic modifications that together make forward gradient learning practical for standard deep learning benchmark tasks. We show that it is possible to substantially reduce the variance of the forward gradient estimator by applying perturbations to activations rather than weights. We further improve the scalability of forward gradient by introducing a large number of local greedy loss functions, each of which involves only a small number of learnable parameters, and a new MLPMixer-inspired architecture, LocalMixer, that is more suitable for local learning. Our approach matches backprop on MNIST and CIFAR-10 and significantly outperforms previously proposed backprop-free algorithms on ImageNet.

翻译：前向梯度学习通过计算含噪方向梯度为深度神经网络提供了一种生物可信的反向传播替代方案。然而，当需要学习的参数量较大时，标准前向梯度算法若直接应用会面临高方差问题。本文提出一系列架构与算法改进，使前向梯度学习在标准深度学习基准任务中具备实践可行性。我们证明：通过对激活值而非权重施加扰动，可显著降低前向梯度估计器的方差。进一步地，通过引入大量局部贪心损失函数（每个函数仅涉及少量可学习参数），并结合专为局部学习设计的类MLPMixer架构LocalMixer，有效提升了前向梯度的可扩展性。本方法在MNIST和CIFAR-10数据集上达到与反向传播相当的性能，并在ImageNet数据集上显著优于此前提出的无反向传播算法。