Ever since their conception, Transformers have taken over traditional sequence models in many tasks, such as NLP, image classification, and video/audio processing, for their fast training and superior performance. Much of the merit is attributable to positional encoding and multi-head attention. However, Transformers fall short in learning long-range dependencies mainly due to the quadratic complexity scaled with context length, in terms of both time and space. Consequently, over the past five years, a myriad of methods has been proposed to make Transformers more efficient. In this work, we first take a step back, study and compare existing solutions to long-sequence modeling in terms of their pure mathematical formulation. Specifically, we summarize them using a unified template, given their shared nature of token mixing. Through benchmarks, we then demonstrate that long context length does yield better performance, albeit application-dependent, and traditional Transformer models fall short in taking advantage of long-range dependencies. Next, inspired by emerging sparse models of huge capacity, we propose a machine learning system for handling million-scale dependencies. As a proof of concept, we evaluate the performance of one essential component of this system, namely, the distributed multi-head attention. We show that our algorithm can scale up attention computation by almost $40\times$ using four GeForce RTX 4090 GPUs, compared to vanilla multi-head attention mechanism. We believe this study is an instrumental step towards modeling million-scale dependencies.

翻译：自提出以来，Transformer凭借其快速训练和卓越性能，在自然语言处理、图像分类、视频/音频处理等诸多任务中取代了传统序列模型。这一优势主要归功于位置编码和多头注意力机制。然而，Transformer在学习长距离依赖时存在不足，主要因其时间与空间复杂度随上下文长度呈二次增长。为此，过去五年间涌现了大量方法以提升Transformer的效率。本研究首先退一步审视，从纯数学公式角度比较现有长序列建模方案。具体而言，鉴于各类方法在令牌混合层面的共性，我们采用统一模板对其进行归纳。通过基准测试，我们证明长上下文长度确实能提升性能（尽管依赖具体应用），而传统Transformer模型在利用长距离依赖方面存在局限。随后，受新兴大容量稀疏模型的启发，我们提出了一套处理百万级依赖的机器学习系统。作为概念验证，我们评估了该系统核心组件——分布式多头注意力——的性能。实验表明，与标准多头注意力机制相比，我们的算法可在四块GeForce RTX 4090 GPU上将注意力计算规模扩展近40倍。我们相信，这项研究是迈向百万级依赖建模的关键一步。