In this paper, we introduce Positional Skip-wisE (PoSE) training for efficient adaptation of large language models~(LLMs) to extremely long context windows. PoSE decouples train length from target context window size by simulating long inputs using a fixed context window with manipulated position indices during training. Concretely, we select several short chunks from a long input sequence, and introduce distinct skipping bias terms to modify the position indices of each chunk. These bias terms, along with the length of each chunk, are altered for each training example, allowing the model to adapt to all positions within the target context window without training on full length inputs. Experiments show that, compared with fine-tuning on the full length, PoSE greatly reduces memory and time overhead with minimal impact on performance. Leveraging this advantage, we have successfully extended the LLaMA model to 128k tokens. Furthermore, we empirically confirm that PoSE is compatible with all RoPE-based LLMs and various position interpolation strategies. Notably, by decoupling fine-tuning length from target context window, PoSE can theoretically extend the context window infinitely, constrained only by memory usage for inference. With ongoing advancements for efficient inference, we believe PoSE holds great promise for scaling the context window even further.

翻译：摘要：本文提出位置跳跃式（Positional Skip-wisE，PoSE）训练方法，用于高效适配大语言模型至极长上下文窗口。PoSE通过固定上下文窗口模拟长输入，在训练时通过调整位置索引将训练长度与目标上下文窗口尺寸解耦。具体而言，我们从长输入序列中选取若干短片段，并为每个片段引入不同的跳跃偏置项以修改其位置索引。每个训练样本中这些偏置项以及片段长度均会变化，使得模型无需在完整长度输入上训练即可适应目标上下文窗口内的所有位置。实验表明，与全长度微调相比，PoSE在保持性能影响最小化的同时大幅降低内存与时间开销。利用此优势，我们成功将LLaMA模型扩展至128k词元。此外，实验验证PoSE兼容所有基于RoPE的大语言模型及各类位置插值策略。值得注意的是，通过将微调长度与目标上下文窗口解耦，PoSE理论上可无限扩展上下文窗口，仅受推理时内存消耗限制。随着高效推理技术的持续进步，我们相信PoSE在进一步扩展上下文窗口方面具有巨大潜力。