To improve the robustness of transformer neural networks used for temporal-dynamics prediction of chaotic systems, we propose a novel attention mechanism called easy attention. Due to the fact that self attention only makes usage of the inner product of queries and keys, it is demonstrated that the keys, queries and softmax are not necessary for obtaining the attention score required to capture long-term dependencies in temporal sequences. Through implementing singular-value decomposition (SVD) on the softmax attention score, we further observe that the self attention compresses contribution from both queries and keys in the spanned space of the attention score. Therefore, our proposed easy-attention method directly treats the attention scores as learnable parameters. This approach produces excellent results when reconstructing and predicting the temporal dynamics of chaotic systems exhibiting more robustness and less complexity than the self attention or the widely-used long short-term memory (LSTM) network. Our results show great potential for applications in more complex high-dimensional dynamical systems.

翻译：为提升用于混沌系统时序动力学预测的Transformer神经网络的鲁棒性，我们提出一种名为"易注意"的新型注意力机制。由于自注意力仅利用查询与键的内积，研究表明：为获取捕获时序序列中长期依赖所需的注意力分数，键、查询及Softmax并非必要。通过对Softmax注意力分数实施奇异值分解，我们进一步发现自注意力在注意力分数张成的空间中压缩了查询与键的贡献。因此，本文提出的易注意方法直接将注意力分数视为可学习参数。该方法在重构与预测混沌系统时序动力学时取得优异效果，相比自注意力或广泛使用的长短期记忆网络，展现出更强的鲁棒性与更低的复杂度。研究结果表明，该方法在更复杂的高维动力学系统中具有巨大应用潜力。