The brain prepares for learning even before interacting with the environment, by refining and optimizing its structures through spontaneous neural activity that resembles random noise. However, the mechanism of such a process has yet to be thoroughly understood, and it is unclear whether this process can benefit the algorithm of machine learning. Here, we study this issue using a neural network with a feedback alignment algorithm, demonstrating that pretraining neural networks with random noise increases the learning efficiency as well as generalization abilities without weight transport. First, we found that random noise training modifies forward weights to match backward synaptic feedback, which is necessary for teaching errors by feedback alignment. As a result, a network with pre-aligned weights learns notably faster than a network without random noise training, even reaching a convergence speed comparable to that of a backpropagation algorithm. Sequential training with both random noise and data brings weights closer to synaptic feedback than training solely with data, enabling more precise credit assignment and faster learning. We also found that each readout probability approaches the chance level and that the effective dimensionality of weights decreases in a network pretrained with random noise. This pre-regularization allows the network to learn simple solutions of a low rank, reducing the generalization loss during subsequent training. This also enables the network robustly to generalize a novel, out-of-distribution dataset. Lastly, we confirmed that random noise pretraining reduces the amount of meta-loss, enhancing the network ability to adapt to various tasks. Overall, our results suggest that random noise training with feedback alignment offers a straightforward yet effective method of pretraining that facilitates quick and reliable learning without weight transport.

翻译：大脑在与环境交互之前，就已通过类似随机噪声的自发神经活动来优化其结构，从而为学习做好准备。然而，这一过程的机制尚未被充分理解，且尚不清楚该过程是否能有益于机器学习算法。本文通过采用反馈对齐算法的神经网络对此问题进行研究，证明使用随机噪声对神经网络进行预训练能够在不依赖权重传递的情况下提升学习效率与泛化能力。首先，我们发现随机噪声训练能够调整前向权重以匹配后向突触反馈，这对于通过反馈对齐传递教学误差至关重要。因此，经过预对齐权重的网络学习速度显著快于未经随机噪声训练的网络，甚至达到与反向传播算法相当的收敛速度。相较于仅使用数据训练，结合随机噪声与数据的顺序训练能使权重更接近突触反馈，从而实现更精确的信用分配和更快速的学习。我们还发现，在经随机噪声预训练的网络中，各读出概率趋近于随机水平，且权重的有效维度降低。这种预正则化使网络能够学习低秩的简单解，从而减少后续训练中的泛化损失。这也使网络能够鲁棒地泛化至分布外的新数据集。最后，我们证实随机噪声预训练能够减少元损失量，增强网络适应不同任务的能力。总体而言，我们的研究结果表明，结合反馈对齐的随机噪声训练提供了一种简单而有效的预训练方法，可在无需权重传递的情况下实现快速可靠的学习。