The ability of Large Language Models (LLMs) to process and generate coherent text is markedly weakened when the number of input tokens exceeds their pretraining length. Given the expensive overhead of finetuning large-scale models with longer sequences, we propose Dual Chunk Attention (DCA), which enables Llama2 70B to support context windows of more than 100k tokens without continual training. By decomposing the attention computation for long sequences into chunk-based modules, DCA manages to effectively capture the relative positional information of tokens within the same chunk (Intra-Chunk) and across distinct chunks (Inter-Chunk), as well as integrates seamlessly with Flash Attention. In addition to its impressive extrapolation capability, DCA achieves performance on practical long-context tasks that is comparable to or even better than that of finetuned models. When compared with proprietary models, our training-free 70B model attains 94% of the performance of gpt-3.5-16k, indicating it is a viable open-source alternative. All code and data used in this work are released at \url{https://github.com/HKUNLP/ChunkLlama}.

翻译：大型语言模型（LLMs）在输入令牌数量超过其预训练长度时，生成连贯文本的能力会显著减弱。鉴于对长序列进行大规模模型微调的高昂成本，我们提出双块注意力（Dual Chunk Attention, DCA），该方法使Llama2 70B能够在无需持续训练的情况下支持超过10万令牌的上下文窗口。通过将长序列的注意力计算分解为基于块的模块，DCA能够有效捕获同一块内（块内，Intra-Chunk）及不同块间（块间，Inter-Chunk）令牌的相对位置信息，并与Flash Attention无缝集成。除了其卓越的外推能力外，DCA在实际长上下文任务中的性能与微调模型相当甚至更优。与商业模型相比，我们无训练的70B模型达到了gpt-3.5-16k性能的94%，表明其是一种可行的开源替代方案。本研究使用的所有代码和数据均发布于\url{https://github.com/HKUNLP/ChunkLlama}。