Understanding neural dynamics is a central topic in machine learning, non-linear physics and neuroscience. However, the dynamics is non-linear, stochastic and particularly non-gradient, i.e., the driving force can not be written as gradient of a potential. These features make analytic studies very challenging. The common tool is the path integral approach or dynamical mean-field theory, but the drawback is that one has to solve the integro-differential or dynamical mean-field equations, which is computationally expensive and has no closed form solutions in general. From the aspect of associated Fokker-Planck equation, the steady state solution is generally unknown. Here, we treat searching for the steady states as an optimization problem, and construct an approximate potential related to the speed of the dynamics, and find that searching for the ground state of this potential is equivalent to running an approximate stochastic gradient dynamics or Langevin dynamics. Only in the zero temperature limit, the distribution of the original steady states can be achieved. The resultant stationary state of the dynamics follows exactly the canonical Boltzmann measure. Within this framework, the quenched disorder intrinsic in the neural networks can be averaged out by applying the replica method, which leads naturally to order parameters for the non-equilibrium steady states. Our theory reproduces the well-known result of edge-of-chaos, and further the order parameters characterizing the continuous transition are derived, and the order parameters are explained as fluctuations and responses of the steady states. Our method thus opens the door to analytically study the steady state landscape of the deterministic or stochastic high dimensional dynamics.

翻译：理解神经动力学是机器学习、非线性物理和神经科学领域的核心议题。然而，该动力学具有非线性、随机性，尤其具有非梯度特性，即驱动力无法表示为势函数的梯度。这些特征使得解析研究极具挑战性。常用方法为路径积分方法或动力学平均场理论，但其缺点在于必须求解积分-微分方程或动力学平均场方程，这计算成本高昂且通常不存在闭式解。从对应的福克-普朗克方程角度看，稳态解通常是未知的。本文将稳态搜索视为优化问题，构造与动力学速度相关的近似势函数，发现搜索该势函数的基态等价于运行近似随机梯度动力学或朗之万动力学。仅在零温度极限下才能获得原始稳态的分布。所得动力学稳态分布精确遵循经典玻尔兹曼测度。在该框架下，神经网络固有的淬火无序可通过复制方法进行平均化处理，从而自然导出非平衡稳态的有序参量。本理论复现了著名的混沌边缘结果，进一步推导出表征连续相变的有序参量，并将这些有序参量解释为稳态的涨落与响应。因此，本文方法为解析研究确定性或随机高维动力学的稳态景观开辟了新途径。