State-space models (SSMs) that utilize linear, time-invariant (LTI) systems are known for their effectiveness in learning long sequences. However, these models typically face several challenges: (i) they require specifically designed initializations of the system matrices to achieve state-of-the-art performance, (ii) they require training of state matrices on a logarithmic scale with very small learning rates to prevent instabilities, and (iii) they require the model to have exponentially decaying memory in order to ensure an asymptotically stable LTI system. To address these issues, we view SSMs through the lens of Hankel operator theory, which provides us with a unified theory for the initialization and training of SSMs. Building on this theory, we develop a new parameterization scheme, called HOPE, for LTI systems that utilizes Markov parameters within Hankel operators. This approach allows for random initializations of the LTI systems and helps to improve training stability, while also provides the SSMs with non-decaying memory capabilities. Our model efficiently implements these innovations by nonuniformly sampling the transfer functions of LTI systems, and it requires fewer parameters compared to canonical SSMs. When benchmarked against HiPPO-initialized models such as S4 and S4D, an SSM parameterized by Hankel operators demonstrates improved performance on Long-Range Arena (LRA) tasks. Moreover, we use a sequential CIFAR-10 task with padded noise to empirically corroborate our SSM's long memory capacity.

翻译：利用线性时不变（LTI）系统的状态空间模型（SSMs）以其在学习长序列方面的有效性而闻名。然而，这些模型通常面临几个挑战：（i）它们需要专门设计的系统矩阵初始化才能达到最先进的性能，（ii）它们需要在非常小的学习率下以对数尺度训练状态矩阵以防止不稳定，以及（iii）它们要求模型具有指数衰减的记忆以确保LTI系统渐近稳定。为了解决这些问题，我们通过汉克尔算子理论的视角审视SSMs，这为我们提供了SSMs初始化与训练的统一理论。基于此理论，我们为LTI系统开发了一种新的参数化方案，称为HOPE，该方案利用汉克尔算子内的马尔可夫参数。这种方法允许LTI系统进行随机初始化，有助于提高训练稳定性，同时赋予SSMs非衰减的记忆能力。我们的模型通过非均匀采样LTI系统的传递函数来高效实现这些创新，并且与经典SSMs相比需要更少的参数。在Long-Range Arena（LRA）任务上进行基准测试时，由汉克尔算子参数化的SSM相较于HiPPO初始化的模型（如S4和S4D）表现出改进的性能。此外，我们使用带有填充噪声的顺序CIFAR-10任务，从经验上证实了我们SSM的长记忆能力。