Forecasting high-dimensional dynamical systems is a fundamental challenge in various fields, such as geosciences and engineering. Neural Ordinary Differential Equations (NODEs), which combine the power of neural networks and numerical solvers, have emerged as a promising algorithm for forecasting complex nonlinear dynamical systems. However, classical techniques used for NODE training are ineffective for learning chaotic dynamical systems. In this work, we propose a novel NODE-training approach that allows for robust learning of chaotic dynamical systems. Our method addresses the challenges of non-convexity and exploding gradients associated with underlying chaotic dynamics. Training data trajectories from such systems are split into multiple, non-overlapping time windows. In addition to the deviation from the training data, the optimization loss term further penalizes the discontinuities of the predicted trajectory between the time windows. The window size is selected based on the fastest Lyapunov time scale of the system. Multi-step penalty(MP) method is first demonstrated on Lorenz equation, to illustrate how it improves the loss landscape and thereby accelerates the optimization convergence. MP method can optimize chaotic systems in a manner similar to least-squares shadowing with significantly lower computational costs. Our proposed algorithm, denoted the Multistep Penalty NODE, is applied to chaotic systems such as the Kuramoto-Sivashinsky equation, the two-dimensional Kolmogorov flow, and ERA5 reanalysis data for the atmosphere. It is observed that MP-NODE provide viable performance for such chaotic systems, not only for short-term trajectory predictions but also for invariant statistics that are hallmarks of the chaotic nature of these dynamics.

翻译：高维动力系统的预测是地球科学和工程等多个领域的一项基础性挑战。神经常微分方程（NODE）结合了神经网络和数值求解器的优势，已成为预测复杂非线性动力系统的一种有前景的算法。然而，用于NODE训练的经典技术在学习混沌动力系统方面效果不佳。在这项工作中，我们提出了一种新颖的NODE训练方法，能够实现对混沌动力系统的鲁棒学习。我们的方法解决了与底层混沌动力学相关的非凸性和梯度爆炸的挑战。来自此类系统的训练数据轨迹被分割成多个非重叠的时间窗口。除了与训练数据的偏差外，优化损失项还进一步惩罚了预测轨迹在时间窗口之间的不连续性。窗口大小是根据系统的最快李雅普诺夫时间尺度选择的。多步惩罚（MP）方法首先在洛伦兹方程上得到验证，以说明其如何改善损失景观，从而加速优化收敛。MP方法能够以类似于最小二乘影子跟踪的方式优化混沌系统，同时计算成本显著降低。我们将所提出的算法（称为多步惩罚NODE）应用于混沌系统，如Kuramoto-Sivashinsky方程、二维Kolmogorov流以及大气ERA5再分析数据。结果表明，MP-NODE为此类混沌系统提供了可行的性能，不仅适用于短期轨迹预测，也适用于作为这些动力学混沌特性标志的不变统计量。