Ever since their conception, Transformers have taken over traditional sequence models in many tasks, such as NLP, image classification, and video/audio processing, for their fast training and superior performance. Much of the merit is attributable to positional encoding and multi-head attention. However, Transformers fall short in learning long-range dependencies mainly due to the quadratic complexity scaled with context length, in terms of both time and space. Consequently, over the past five years, a myriad of methods has been proposed to make Transformers more efficient. In this work, we first take a step back, study and compare existing solutions to long-sequence modeling in terms of their pure mathematical formulation. Specifically, we summarize them using a unified template, given their shared nature of token mixing. Through benchmarks, we then demonstrate that long context length does yield better performance, albeit application-dependent, and traditional Transformer models fall short in taking advantage of long-range dependencies. Next, inspired by emerging sparse models of huge capacity, we propose a machine learning system for handling million-scale dependencies. As a proof of concept, we evaluate the performance of one essential component of this system, namely, the distributed multi-head attention. We show that our algorithm can scale up attention computation by almost $40\times$ using four GeForce RTX 4090 GPUs, compared to vanilla multi-head attention mechanism. We believe this study is an instrumental step towards modeling million-scale dependencies.

翻译：自概念提出以来，Transformer因其快速训练和优越性能，在自然语言处理、图像分类、视频/音频处理等众多任务中取代了传统序列模型。其优势主要归功于位置编码和多头注意力机制。然而，Transformer在学习长距离依赖方面存在不足，主要原因在于其时间和空间复杂度随上下文长度呈二次增长。为此，过去五年中涌现了大量提升Transformer效率的方法。本研究首先退一步审视，从纯数学公式角度研究并比较现有长序列建模解决方案。具体而言，鉴于它们共享的令牌混合本质，我们采用统一模板进行归纳。通过基准测试，我们发现长上下文长度确实能带来更好的性能（尽管依赖于具体应用），而传统Transformer模型在学习长距离依赖方面存在不足。接着，受新兴超大容量稀疏模型的启发，我们提出了一种用于处理百万级依赖的机器学习系统。作为概念验证，我们评估了该系统关键组件——分布式多头注意力机制的性能。结果表明，与标准多头注意力机制相比，我们的算法可在四块GeForce RTX 4090 GPU上将注意力计算扩展近40倍。我们相信这项研究是迈向百万级依赖建模的重要一步。