Representational drift refers to over-time changes in neural activation accompanied by a stable task performance. Despite being observed in the brain and in artificial networks, the mechanisms of drift and its implications are not fully understood. Motivated by recent experimental findings of stimulus-dependent drift in the piriform cortex, we use theory and simulations to study this phenomenon in a two-layer linear feedforward network. Specifically, in a continual online learning scenario, we study the drift induced by the noise inherent in the Stochastic Gradient Descent (SGD). By decomposing the learning dynamics into the normal and tangent spaces of the minimum-loss manifold, we show the former corresponds to a finite variance fluctuation, while the latter could be considered as an effective diffusion process on the manifold. We analytically compute the fluctuation and the diffusion coefficients for the stimuli representations in the hidden layer as functions of network parameters and input distribution. Further, consistent with experiments, we show that the drift rate is slower for a more frequently presented stimulus. Overall, our analysis yields a theoretical framework for better understanding of the drift phenomenon in biological and artificial neural networks.

翻译：表示漂移是指伴随稳定任务性能的神经活动随时间变化的现象。尽管在脑和人工网络中均观察到该现象，但其机制与影响尚未完全明晰。受近期梨状皮层刺激依赖性漂移实验发现的启发，我们通过理论与模拟研究了两层线性前馈网络中的这一现象。具体而言，在持续在线学习的场景下，我们探究了随机梯度下降（SGD）固有噪声所导致的漂移。通过将学习动力学分解为最小损失流形的法向空间与切向空间，我们表明前者对应有限方差涨落，而后者可视为流形上的有效扩散过程。我们解析计算了隐藏层刺激表示的涨落系数与扩散系数作为网络参数和输入分布的函数。进一步地，与实验结果一致，我们证实频率更高的刺激对应更慢的漂移速率。总体而言，我们的分析为理解生物与人工神经网络中的漂移现象提供了理论框架。