We present a series of long-context LLMs that support effective context windows of up to 32,768 tokens. Our model series are built through continual pretraining from Llama 2 with longer training sequences and on a dataset where long texts are upsampled. We perform extensive evaluation on language modeling, synthetic context probing tasks, and a wide range of research benchmarks. On research benchmarks, our models achieve consistent improvements on most regular tasks and significant improvements on long-context tasks over Llama 2. Notably, with a cost-effective instruction tuning procedure that does not require human-annotated long instruction data, the 70B variant can already surpass gpt-3.5-turbo-16k's overall performance on a suite of long-context tasks. Alongside these results, we provide an in-depth analysis on the individual components of our method. We delve into Llama's position encodings and discuss its limitation in modeling long dependencies. We also examine the impact of various design choices in the pretraining process, including the data mix and the training curriculum of sequence lengths -- our ablation experiments suggest that having abundant long texts in the pretrain dataset is not the key to achieving strong performance, and we empirically verify that long context continual pretraining is more efficient and similarly effective compared to pretraining from scratch with long sequences.

翻译：我们提出了一系列支持高达32,768个令牌有效上下文窗口的长上下文大语言模型。该模型系列基于Llama 2，通过持续预训练构建，采用更长的训练序列以及长文本上采样的数据集。我们在语言建模、合成上下文探测任务以及广泛的研究基准上进行了全面评估。在研究基准上，与Llama 2相比，我们的模型在大多数常规任务上取得了一致改进，并在长上下文任务上实现了显著提升。值得注意的是，通过一种无需人工标注长指令数据且成本高效的指令微调流程，70B变体在长上下文任务套件上的整体性能已超越gpt-3.5-turbo-16k。除这些结果外，我们还对我们方法的各个组成部分进行了深入分析，探讨了Llama的位置编码及其在长依赖建模中的局限性，并检验了预训练过程中多种设计选择的影响，包括数据混合和序列长度的训练课程。我们的消融实验表明，预训练数据集中包含大量长文本并非实现强性能的关键；我们通过实验验证，长上下文持续预训练与从头开始使用长序列预训练相比，效率更高且效果相当。