Transformer models have been successful in various sequence processing tasks, but the self-attention mechanism's computational cost limits its practicality for long sequences. Although there are existing attention variants that improve computational efficiency, they have a limited ability to abstract global information effectively based on their hand-crafted mixing strategies. On the other hand, state-space models (SSMs) are tailored for long sequences but cannot capture complicated local information. Therefore, the combination of them as a unified token mixer is a trend in recent long-sequence models. However, the linearized attention degrades performance significantly even when equipped with SSMs. To address the issue, we propose a new method called LongVQ. LongVQ uses the vector quantization (VQ) technique to compress the global abstraction as a length-fixed codebook, enabling the linear-time computation of the attention matrix. This technique effectively maintains dynamic global and local patterns, which helps to complement the lack of long-range dependency issues. Our experiments on the Long Range Arena benchmark, autoregressive language modeling, and image and speech classification demonstrate the effectiveness of LongVQ. Our model achieves significant improvements over other sequence models, including variants of Transformers, Convolutions, and recent State Space Models.

翻译：Transformer模型已在各类序列处理任务中取得成功，但自注意力机制的计算成本限制了其在长序列中的实用性。尽管现有注意力变体提升了计算效率，但其基于人工设计的混合策略在有效抽象全局信息方面能力有限。另一方面，状态空间模型（SSMs）虽专为长序列设计，却无法捕捉复杂的局部信息。因此，将二者融合为统一令牌混合器已成为近期长序列模型的发展趋势。然而，即便配置了SSMs，线性化注意力仍会显著降低性能。为解决该问题，我们提出名为LongVQ的新方法。LongVQ采用向量量化（VQ）技术将全局抽象信息压缩为固定长度码本，实现注意力矩阵的线性时间计算。该技术有效维持了动态全局与局部模式，有助于弥补长距离依赖问题的不足。我们在Long Range Arena基准测试、自回归语言建模以及图像与语音分类任务上的实验证明了LongVQ的有效性。相较于包括Transformer变体、卷积网络及近期状态空间模型在内的其他序列模型，我们的模型取得了显著性能提升。