Backpropagation (BP) is widely viewed as biologically implausible, in part because it requires feedback weights to be the transpose of forward weights for error propagation. Interestingly, when training a network with fixed random feedback weights to circumvent this issue, learning aligns the forward weights with the feedback weights, leading the backpropagated error signal to become an approximation of the standard gradient used by BP. This process, called Feedback Alignment (FA), occurs in MLPs and very shallow CNNs but does not scale well to deeper architectures. In this work, we first investigated differences between BP and FA models, trained on CIFAR10, specifically focusing on the effective rank of the signal. We found that the FA error has a considerably lower rank and hence is constrained to a lower-dimensional subspace compared to BP, limiting exploration of the parameter space. Motivated by this observation, we evaluated two mechanisms for increasing the effective dimensionality of FA: Muon, an optimiser that orthogonalises weight updates; and hidden activity normalisation, which promotes activation orthogonality. Across larger architectures and benchmarks, we find that these methods consistently improve over FA baselines, for example, on CIFAR100 with a Resnet-18, accuracy increases by 9 percentage points. Our results identify low-dimensional gradient dynamics as a key obstacle to scaling FA and suggest that inducing higher-dimensional update geometry is a promising route toward scaling alternatives to backpropagation.

翻译：反向传播（BP）因需要反馈权重与前向权重的转置一致以实现误差传播，被广泛认为在生物学上不可信。有趣的是，当使用固定的随机反馈权重训练网络以规避这一问题时，学习过程会使前向权重与反馈权重对齐，导致反向传播的误差信号近似于BP使用的标准梯度。这一过程被称为反馈对齐（FA），它在多层感知机和极浅层卷积神经网络中有效，但难以扩展到更深的架构中。在本工作中，我们首先研究了在CIFAR10上训练的BP与FA模型之间的差异，特别关注信号的有效秩。我们发现，与BP相比，FA误差的秩显著更低，因此被限制在较低维度的子空间中，从而限制了对参数空间的探索。基于这一观察，我们评估了两种提高FA有效维度的方法：Muon（一种正交化权重更新的优化器）和隐藏活动归一化（促进激活正交性）。在更大的架构和基准测试中，我们发现这些方法持续优于FA基线，例如，在CIFAR100上使用Resnet-18时，准确率提高了9个百分点。我们的结果将低维梯度动力学识别为扩展FA的关键障碍，并表明引入更高维度的更新几何是扩展反向传播替代方案的有前景途径。