Transformer models have been successful in various sequence processing tasks, but the self-attention mechanism's computational cost limits its practicality for long sequences. Although there are existing attention variants that improve computational efficiency, they have a limited ability to abstract global information effectively based on their hand-crafted mixing strategies. On the other hand, state-space models (SSMs) are tailored for long sequences but cannot capture complicated local information. Therefore, the combination of them as a unified token mixer is a trend in recent long-sequence models. However, the linearized attention degrades performance significantly even when equipped with SSMs. To address the issue, we propose a new method called LongVQ. LongVQ uses the vector quantization (VQ) technique to compress the global abstraction as a length-fixed codebook, enabling the linear-time computation of the attention matrix. This technique effectively maintains dynamic global and local patterns, which helps to complement the lack of long-range dependency issues. Our experiments on the Long Range Arena benchmark, autoregressive language modeling, and image and speech classification demonstrate the effectiveness of LongVQ. Our model achieves significant improvements over other sequence models, including variants of Transformers, Convolutions, and recent State Space Models.

翻译：Transformer模型在各种序列处理任务中取得了成功，但其自注意力机制的计算成本限制了其在长序列中的实用性。尽管存在提高计算效率的注意力变体，但这些方法基于手工设计的混合策略，在有效抽象全局信息方面能力有限。另一方面，状态空间模型（SSM）专为长序列设计，但无法捕捉复杂的局部信息。因此，将两者结合为统一的词元混合器已成为近期长序列模型的发展趋势。然而，即使配备了SSM，线性化注意力仍会显著降低性能。为解决这一问题，我们提出了一种名为LongVQ的新方法。LongVQ利用向量量化（VQ）技术将全局抽象特征压缩为固定长度的码本，从而实现注意力矩阵的线性时间计算。该技术有效保持了动态全局与局部模式，有助于弥补长程依赖问题的不足。我们在Long Range Arena基准测试、自回归语言建模以及图像与语音分类任务上的实验证明了LongVQ的有效性。我们的模型相比其他序列模型（包括Transformer变体、卷积网络及近期状态空间模型）取得了显著改进。