Large Language Models (LLMs) are trained with a pre-defined context length, restricting their use in scenarios requiring long inputs. Previous efforts for adapting LLMs to a longer length usually requires fine-tuning with this target length (Full-length fine-tuning), suffering intensive training cost. To decouple train length from target length for efficient context window extension, we propose Positional Skip-wisE (PoSE) training that smartly simulates long inputs using a fixed context window. This is achieved by first dividing the original context window into several chunks, then designing distinct skipping bias terms to manipulate the position indices of each chunk. These bias terms and the lengths of each chunk are altered for every training example, allowing the model to adapt to all positions within target length. Experimental results show that PoSE greatly reduces memory and time overhead compared with Full-length fine-tuning, with minimal impact on performance. Leveraging this advantage, we have successfully extended the LLaMA model to 128k tokens using a 2k training context window. Furthermore, we empirically confirm that PoSE is compatible with all RoPE-based LLMs and position interpolation strategies. Notably, our method can potentially support infinite length, limited only by memory usage in inference. With ongoing progress for efficient inference, we believe PoSE can further scale the context window beyond 128k.

翻译：大语言模型（LLMs）通常在预定义上下文长度下进行训练，这限制了其在需要长输入场景中的应用。先前为适配长上下文而进行的模型微调通常需按目标长度进行（全长度微调），导致训练成本高昂。为解耦训练长度与目标长度以实现高效的上下文窗口扩展，我们提出位置跳跃式训练（PoSE），通过固定上下文窗口智能模拟长输入。该方法首先将原始上下文窗口划分为若干片段，随后为每个片段设计不同的跳跃偏置项以操控其位置索引。通过为每个训练样本动态调整这些偏置项及片段长度，模型得以适应目标长度内的所有位置。实验结果表明，与全长度微调相比，PoSE在保持性能影响最小化的同时显著降低了内存与时间开销。借助此优势，我们成功将LLaMA模型的上下文窗口从2k训练长度扩展至128k令牌。此外，实验证实PoSE兼容所有基于RoPE的大语言模型及位置插值策略。值得注意的是，本方法理论上可支持无限长度——其限制仅来自推理阶段的内存占用。随着高效推理技术的持续发展，我们相信PoSE能够进一步将上下文窗口扩展至128k以上。