Two of the many trends in neural network research of the past few years have been (i) the learning of dynamical systems, especially with recurrent neural networks such as long short-term memory networks (LSTMs) and (ii) the introduction of transformer neural networks for natural language processing (NLP) tasks. Both of these trends have created enormous amounts of traction, particularly the second one: transformer networks now dominate the field of NLP. Even though some work has been performed on the intersection of these two trends, this work was largely limited to using the vanilla transformer directly without adjusting its architecture for the setting of a physical system. In this work we use a transformer-inspired neural network to learn a complicated non-linear dynamical system and furthermore (for the first time) imbue it with structure-preserving properties to improve long-term stability. This is shown to be extremely important when applying the neural network to real world applications.

翻译：过去几年神经网络研究的两大趋势包括：（i）学习动力系统，特别是使用长短期记忆网络（LSTM）等循环神经网络；（ii）引入变换器神经网络处理自然语言处理（NLP）任务。这两大趋势均产生了巨大推动力，尤其是第二个趋势：变换器网络如今已主导自然语言处理领域。尽管已有研究尝试交叉这两个方向，但这些工作大多局限于直接使用原始变换器架构，未针对物理系统场景调整其结构。本研究采用受变换器启发的神经网络学习复杂非线性动力系统，并（首次）为其赋予保结构特性以提升长期稳定性。实验表明，在将神经网络应用于实际场景时，这一特性至关重要。