Learning relies on coordinated synaptic changes in recurrently connected populations of neurons. Therefore, understanding the collective evolution of synaptic connectivity over learning is a key challenge in neuroscience and machine learning. In particular, recent work has shown that the weight matrices of task-trained RNNs are typically low rank, but how this low rank structure unfolds over learning is unknown. To address this, we investigate the rank of the 3-tensor formed by the weight matrices throughout learning. By fitting RNNs of varying rank to large-scale neural recordings during a motor learning task, we find that the inferred weights are low-tensor-rank and therefore evolve over a fixed low-dimensional subspace throughout the entire course of learning. We next validate the observation of low-tensor-rank learning on an RNN trained to solve the same task. Finally, we present a set of mathematical results bounding the matrix and tensor ranks of gradient descent learning dynamics which show that low-tensor-rank weights emerge naturally in RNNs trained to solve low-dimensional tasks. Taken together, our findings provide insight on the evolution of population connectivity over learning in both biological and artificial neural networks, and enable reverse engineering of learning-induced changes in recurrent dynamics from large-scale neural recordings.

翻译：学习依赖于循环连接神经元群体中突触的协同变化。因此，理解学习过程中突触连接性的集体演化，是神经科学与机器学习领域的关键挑战。近期研究表明，经过任务训练的循环神经网络（RNN）的权重矩阵通常具有低秩特性，但这一低秩结构如何随学习过程展开尚不明确。为此，我们探究了学习过程中由权重矩阵构成的三阶张量的秩。通过将不同秩的RNN拟合至运动学习任务中的大规模神经记录数据，我们发现推断出的权重具有低阶张量秩，因此在学习的全程中始终在一个固定的低维子空间内演化。随后，我们在针对同一任务训练的RNN中验证了低阶张量学习现象的观测结果。最后，我们提出一组数学结论，界定了梯度下降学习动态的矩阵秩与张量秩，表明低阶张量权重在训练求解低维任务的RNN中会自然涌现。综合而言，我们的研究为生物与人工神经网络中群体连接性随学习的演化提供了洞见，并有助于从大规模神经记录中逆向工程学习引发的循环动力学变化。