Deep neural networks based on state space models (SSMs) are attracting much attention in sequence modeling since their computational cost is significantly smaller than that of Transformers. While the capabilities of SSMs have been primarily investigated through experimental comparisons, theoretical understanding of SSMs is still limited. In particular, there is a lack of statistical and quantitative evaluation of whether SSM can replace Transformers. In this paper, we theoretically explore in which tasks SSMs can be alternatives of Transformers from the perspective of estimating sequence-to-sequence functions. We consider the setting where the target function has direction-dependent smoothness and prove that SSMs can estimate such functions with the same convergence rate as Transformers. Additionally, we prove that SSMs can estimate the target function, even if the smoothness changes depending on the input sequence, as well as Transformers. Our results show the possibility that SSMs can replace Transformers when estimating the functions in certain classes that appear in practice.

翻译：基于状态空间模型（SSMs）的深度神经网络因其计算成本显著低于Transformer，在序列建模领域正受到广泛关注。尽管SSMs的能力主要通过实验比较进行研究，但其理论理解仍较为有限。特别是在SSM能否替代Transformer这一问题上，缺乏统计与定量化的评估。本文从估计序列到序列函数的角度，理论探究了SSMs在哪些任务中可作为Transformer的替代方案。我们考虑目标函数具有方向依赖平滑性的设定，并证明SSMs能以与Transformer相同的收敛速度估计此类函数。此外，我们证明了SSMs能够像Transformer一样，在平滑性随输入序列变化的情况下仍可估计目标函数。我们的结果表明，在估计实践中出现的某些函数类别时，SSMs具有替代Transformer的潜力。