Learning relies on coordinated synaptic changes in recurrently connected populations of neurons. Therefore, understanding the collective evolution of synaptic connectivity over learning is a key challenge in neuroscience and machine learning. In particular, recent work has shown that the weight matrices of task-trained RNNs are typically low rank, but how this low rank structure unfolds over learning is unknown. To address this, we investigate the rank of the 3-tensor formed by the weight matrices throughout learning. By fitting RNNs of varying rank to large-scale neural recordings during a motor learning task, we find that the inferred weights are low-tensor-rank and therefore evolve over a fixed low-dimensional subspace throughout the entire course of learning. We next validate the observation of low-tensor-rank learning on an RNN trained to solve the same task by performing a low-tensor-rank decomposition directly on the ground truth weights, and by showing that the method we applied to the data faithfully recovers this low rank structure. Finally, we present a set of mathematical results bounding the matrix and tensor ranks of gradient descent learning dynamics which show that low-tensor-rank weights emerge naturally in RNNs trained to solve low-dimensional tasks. Taken together, our findings provide novel constraints on the evolution of population connectivity over learning in both biological and artificial neural networks, and enable reverse engineering of learning-induced changes in recurrent network dynamics from large-scale neural recordings.

翻译：学习依赖于循环连接神经元群体中突触连接的协同变化。因此，理解学习过程中突触连接性的集体演化是神经科学与机器学习中的核心挑战。近期研究表明，经过任务训练的循环神经网络（RNN）的权重矩阵通常呈现低秩结构，但这一低秩结构在学习过程中如何形成尚不清楚。为解决此问题，我们研究了学习过程中权重矩阵构成的三阶张量的秩。通过将不同秩的RNN模型拟合至动物执行运动学习任务时的大规模神经记录数据，我们发现推断出的权重具有低张量秩特性，即在整个学习过程中始终在一个固定的低维子空间内演化。随后，我们通过直接对真实权重进行低张量秩分解，并证明应用于数据的方法能准确恢复这种低秩结构，进一步验证了基于低张量秩学习行为的实验观察。最后，我们提出了一组数学定理，界定了梯度下降学习动力学的矩阵秩与张量秩，证明低张量秩权重会自然出现在训练以解决低维任务的RNN中。综合而言，我们的发现为生物与人工神经网络学习过程中群体连接性的演化提供了新的约束条件，并能够从大规模神经记录中逆向工程出学习引发的循环网络动力学变化。