So far, expensive finetuning beyond the pretraining sequence length has been a requirement for effectively extending the context of language models (LM). In this work, we break this key bottleneck by Dropping the Positional Embeddings of LMs after training (DroPE). Our simple method is motivated by three key theoretical and empirical observations. First, positional embeddings (PEs) serve a crucial role during pretraining, providing an important inductive bias that significantly facilitates convergence. Second, over-reliance on this explicit positional information is also precisely what prevents test-time generalization to sequences of unseen length, even when using popular PE-scaling methods. Third, positional embeddings are not an inherent requirement of effective language modeling and can be safely removed after pretraining, following a short recalibration phase. Empirically, DroPE yields seamless zero-shot context extension without any long-context finetuning, quickly adapting pretrained LMs without compromising their capabilities in the original training context. Our findings hold across different models and dataset sizes, far outperforming previous specialized architectures and established rotary positional embedding scaling methods.

翻译：迄今为止，要有效扩展语言模型（LM）的上下文长度，通常需要在其预训练序列长度之外进行昂贵的微调。在本研究中，我们通过一种名为“训练后丢弃语言模型位置嵌入”（DroPE）的简单方法，突破了这一关键瓶颈。该方法基于三个关键的理论与实证观察：首先，位置嵌入（PEs）在预训练阶段发挥关键作用，提供重要的归纳偏置，显著促进模型收敛；其次，对显式位置信息的过度依赖恰恰阻碍了模型在测试时泛化至未见长度序列的能力，即使采用流行的位置嵌入缩放方法亦然；第三，位置嵌入并非有效语言建模的内在必需组件，经过短暂的重新校准阶段后，可在预训练完成后安全移除。实证表明，DroPE 无需任何长上下文微调即可实现无缝的零样本上下文扩展，使预训练语言模型快速适应新长度，同时保持其在原始训练上下文中的能力。我们的发现在不同模型与数据集规模下均成立，其性能远超以往专用架构及成熟的旋转位置嵌入缩放方法。