State-space models (SSMs) have recently emerged as a framework for learning long-range sequence tasks. An example is the structured state-space sequence (S4) layer, which uses the diagonal-plus-low-rank structure of the HiPPO initialization framework. However, the complicated structure of the S4 layer poses challenges; and, in an effort to address these challenges, models such as S4D and S5 have considered a purely diagonal structure. This choice simplifies the implementation, improves computational efficiency, and allows channel communication. However, diagonalizing the HiPPO framework is itself an ill-posed problem. In this paper, we propose a general solution for this and related ill-posed diagonalization problems in machine learning. We introduce a generic, backward-stable "perturb-then-diagonalize" (PTD) methodology, which is based on the pseudospectral theory of non-normal operators, and which may be interpreted as the approximate diagonalization of the non-normal matrices defining SSMs. Based on this, we introduce the S4-PTD and S5-PTD models. Through theoretical analysis of the transfer functions of different initialization schemes, we demonstrate that the S4-PTD/S5-PTD initialization strongly converges to the HiPPO framework, while the S4D/S5 initialization only achieves weak convergences. As a result, our new models show resilience to Fourier-mode noise-perturbed inputs, a crucial property not achieved by the S4D/S5 models. In addition to improved robustness, our S5-PTD model averages 87.6% accuracy on the Long-Range Arena benchmark, demonstrating that the PTD methodology helps to improve the accuracy of deep learning models.

翻译：状态空间模型（SSM）近期成为学习长程序列任务的一种框架。例如结构化状态空间序列（S4）层，其利用HiPPO初始化框架的对角加低秩结构。然而，S4层的复杂结构带来了挑战；为应对这些挑战，S4D和S5等模型采用了纯对角结构。这一选择简化了实现、提高了计算效率，并允许通道间通信。但HiPPO框架的对角化本身是一个病态问题。本文针对这一机器学习中的相关病态对角化问题提出通用解决方案。我们引入一种基于非正规算子赝谱理论的通用后向稳定"扰动-再对角化"（PTD）方法论，可解释为对定义SSM的非正规矩阵进行近似对角化。基于此，我们提出S4-PTD和S5-PTD模型。通过分析不同初始化方案的传递函数，我们证明S4-PTD/S5-PTD初始化强收敛于HiPPO框架，而S4D/S5初始化仅实现弱收敛。因此，我们的新模型对傅里叶模式噪声扰动输入具有鲁棒性，这是S4D/S5模型未能实现的关键特性。除增强鲁棒性外，我们的S5-PTD模型在Long-Range Arena基准测试中平均准确率达87.6%，表明PTD方法论有助于提升深度学习模型的精度。