State space models (SSMs) have shown impressive results on tasks that require modeling long-range dependencies and efficiently scale to long sequences owing to their subquadratic runtime complexity. Originally designed for continuous signals, SSMs have shown superior performance on a plethora of tasks, in vision and audio; however, SSMs still lag Transformer performance in Language Modeling tasks. In this work, we propose a hybrid layer named Block-State Transformer (BST), that internally combines an SSM sublayer for long-range contextualization, and a Block Transformer sublayer for short-term representation of sequences. We study three different, and completely parallelizable, variants that integrate SSMs and block-wise attention. We show that our model outperforms similar Transformer-based architectures on language modeling perplexity and generalizes to longer sequences. In addition, the Block-State Transformer demonstrates more than tenfold increase in speed at the layer level compared to the Block-Recurrent Transformer when model parallelization is employed.

翻译：状态空间模型（SSMs）在处理需要建模长程依赖的任务中表现出色，并凭借其次二次时间复杂度的特性高效拓展到长序列。该模型最初针对连续信号设计，已在视觉和音频领域的众多任务中展现出优越性能；然而，在语言建模任务中，SSMs仍落后于Transformer的表现。本研究提出一种名为块状态Transformer（BST）的混合层，该层内部结合了用于长程上下文化的SSM子层和用于序列短期表征的块Transformer子层。我们研究了三种不同且完全可并行化的变体，这些变体整合了SSM与块级注意力机制。实验表明，我们的模型在语言建模困惑度上优于类似的基于Transformer的架构，并能泛化至更长序列。此外，当采用模型并行化时，块状态Transformer在层级别上的速度相较于块递归Transformer提升了超过十倍。