Stability in recurrent neural models poses a significant challenge, particularly in developing biologically plausible neurodynamical models that can be seamlessly trained. Traditional cortical circuit models are notoriously difficult to train due to expansive nonlinearities in the dynamical system, leading to an optimization problem with nonlinear stability constraints that are difficult to impose. Conversely, recurrent neural networks (RNNs) excel in tasks involving sequential data but lack biological plausibility and interpretability. In this work, we address these challenges by linking dynamic divisive normalization (DN) to the stability of ORGaNICs, a biologically plausible recurrent cortical circuit model that dynamically achieves DN and that has been shown to simulate a wide range of neurophysiological phenomena. By using the indirect method of Lyapunov, we prove the remarkable property of unconditional local stability for an arbitrary-dimensional ORGaNICs circuit when the recurrent weight matrix is the identity. We thus connect ORGaNICs to a system of coupled damped harmonic oscillators, which enables us to derive the circuit's energy function, providing a normative principle of what the circuit, and individual neurons, aim to accomplish. Further, for a generic recurrent weight matrix, we prove the stability of the 2D model and demonstrate empirically that stability holds in higher dimensions. Finally, we show that ORGaNICs can be trained by backpropagation through time without gradient clipping/scaling, thanks to its intrinsic stability property and adaptive time constants, which address the problems of exploding, vanishing, and oscillating gradients. By evaluating the model's performance on RNN benchmarks, we find that ORGaNICs outperform alternative neurodynamical models on static image classification tasks and perform comparably to LSTMs on sequential tasks.

翻译：循环神经模型的稳定性是一个重大挑战，特别是在开发生物学上合理的神经动力学模型并使其能够被无缝训练方面。传统的皮层电路模型由于动力系统中广泛存在的非线性而极难训练，这导致了一个具有非线性稳定性约束的优化问题，而这些约束难以施加。相反，循环神经网络（RNNs）在处理涉及序列数据的任务方面表现出色，但缺乏生物学合理性和可解释性。在这项工作中，我们通过将动态可除归一化（DN）与ORGaNICs的稳定性联系起来，以应对这些挑战。ORGaNICs是一种生物学上合理的循环皮层电路模型，它能动态实现DN，并且已被证明能够模拟广泛的神经生理现象。通过使用李雅普诺夫间接法，我们证明了当循环权重矩阵为单位矩阵时，任意维度的ORGaNICs电路具有无条件局部稳定性的显著特性。因此，我们将ORGaNICs与一个耦合阻尼谐振子系统联系起来，这使得我们能够推导出电路的能量函数，为电路以及单个神经元旨在实现的目标提供了一个规范性原理。此外，对于一般的循环权重矩阵，我们证明了二维模型的稳定性，并通过实验证明了在高维情况下稳定性依然成立。最后，我们展示了ORGaNICs可以通过时间反向传播进行训练，而无需梯度裁剪/缩放，这得益于其固有的稳定性特性和自适应时间常数，从而解决了梯度爆炸、梯度消失和梯度振荡的问题。通过在RNN基准测试上评估模型的性能，我们发现ORGaNICs在静态图像分类任务上优于其他神经动力学模型，并且在序列任务上的表现与LSTMs相当。