Position embedding is a core component of current Large Language Models (LLMs). Rotary position embedding (RoPE), a technique that encodes the position information with a rotation matrix, has been the de facto choice for position embedding in many LLMs, such as the Llama series. RoPE has been further utilized to extend long context capability, which is roughly based on adjusting the \textit{base} parameter of RoPE to mitigate out-of-distribution (OOD) problems in position embedding. However, in this paper, we find that LLMs may obtain a superficial long-context ability based on the OOD theory. We revisit the role of RoPE in LLMs and propose a novel property of long-term decay, we derive that the \textit{base of RoPE bounds context length}: there is an absolute lower bound for the base value to obtain certain context length capability. Our work reveals the relationship between context length and RoPE base both theoretically and empirically, which may shed light on future long context training.

翻译：位置嵌入是当前大型语言模型（LLM）的核心组件。旋转位置嵌入（RoPE）作为一种通过旋转矩阵编码位置信息的技术，已成为许多LLM（如Llama系列）中位置嵌入的事实标准。RoPE被进一步用于扩展长上下文能力，其方法主要基于调整RoPE的\textit{底数}参数以缓解位置嵌入中的分布外（OOD）问题。然而，本文发现基于OOD理论，LLM可能仅获得表面上的长上下文能力。我们重新审视了RoPE在LLM中的作用，并提出了一种新的长程衰减特性，由此推导出\textit{RoPE的底数决定上下文长度}：要获得特定上下文长度能力，底数存在绝对下界。我们的工作从理论和实验两方面揭示了上下文长度与RoPE底数之间的关系，或将为未来的长上下文训练提供启示。