Scaling sequence length has become a critical demand in the era of large language models. However, existing methods struggle with either computational complexity or model expressivity, rendering the maximum sequence length restricted. To address this issue, we introduce LongNet, a Transformer variant that can scale sequence length to more than 1 billion tokens, without sacrificing the performance on shorter sequences. Specifically, we propose dilated attention, which expands the attentive field exponentially as the distance grows. LongNet has significant advantages: 1) it has a linear computation complexity and a logarithm dependency between any two tokens in a sequence; 2) it can be served as a distributed trainer for extremely long sequences; 3) its dilated attention is a drop-in replacement for standard attention, which can be seamlessly integrated with the existing Transformer-based optimization. Experiments results demonstrate that LongNet yields strong performance on both long-sequence modeling and general language tasks. Our work opens up new possibilities for modeling very long sequences, e.g., treating a whole corpus or even the entire Internet as a sequence.

翻译：扩展序列长度已成为大语言模型时代的迫切需求。然而，现有方法在计算复杂度或模型表达能力方面存在局限性，导致最大序列长度受到制约。为解决该问题，我们提出LongNet——一种可将序列长度扩展至超过10亿词元且不牺牲短序列性能的Transformer变体。具体而言，我们提出膨胀注意力机制，该机制随着距离增长呈指数级扩大注意力感受野。LongNet具有显著优势：1）线性计算复杂度及序列中任意两词元间的对数依赖关系；2）可作为超长序列的分布式训练器；3）膨胀注意力作为标准注意力的即插即用替代方案，可无缝集成至现有基于Transformer的优化框架中。实验结果表明，LongNet在长序列建模及通用语言任务中均展现强劲性能。本工作为建模超长序列开辟了新可能，例如将整个语料库甚至整个互联网视为一个序列。