Ever since their conception, Transformers have taken over traditional sequence models in many tasks, such as NLP, image classification, and video/audio processing, for their fast training and superior performance. Much of these merits result from positional encoding and multi-head attention. However, Transformers fall short in learning long-range dependencies mainly due to the quadratic complexity scaled with context length, in terms of both time and space. Consequently, over the past five years, a myriad of methods has been proposed to make Transformers more efficient. In this work, we first take a step back, study and compare existing solutions to long-sequence modeling in terms of their pure mathematical formulation. Specifically, we summarize them using a unified template, given their shared nature of token mixing. Through benchmarks, we then demonstrate that long context length does yield better performance, albeit application-dependent, and traditional Transformer models fall short in taking advantage of long-range dependencies. Next, inspired by emerging sparse models of huge capacity, we propose a machine learning system for handling million-scale dependencies. As a proof of concept, we evaluate the performance of one essential component of this system, namely, the distributed multi-head attention. We show that our algorithm can scale up attention computation by almost $40\times$ using four GeForce RTX 4090 GPUs, compared to vanilla multi-head attention mechanism. We believe this study is an instrumental step towards modeling million-scale dependencies.

翻译：自提出以来，Transformer凭借其快速训练和优越性能，已在自然语言处理、图像分类、视频/音频处理等诸多任务中取代传统序列模型。这些优势很大程度上源于位置编码与多头注意力机制。然而，Transformer在捕捉长距离依赖方面存在不足，主要原因是其时序和空间复杂度随上下文长度呈二次增长。因此，过去五年中，大量方法被提出以提升Transformer的效率。本文首先退一步审视，从纯数学公式的角度研究并比较现有长序列建模解决方案。具体而言，鉴于这些方法在令牌混合上的共性，我们采用统一模板对其进行总结。随后，通过基准测试，我们证明长上下文长度确实能带来更好的性能（尽管依赖于具体应用），而传统Transformer模型在利用长距离依赖方面存在局限。接着，受新兴大容量稀疏模型的启发，我们提出一个用于处理百万级依赖的机器学习系统。为验证概念，我们评估了该系统核心组件——分布式多头注意力——的性能。实验表明，相较于标准多头注意力机制，我们的算法可在四张GeForce RTX 4090 GPU上将注意力计算速度提升近40倍。我们相信，本研究是迈向百万级依赖建模的重要一步。