The extrapolation capability of Large Language Models (LLMs) based on Rotary Position Embedding is currently a topic of considerable interest. The mainstream approach to addressing extrapolation with LLMs involves modifying RoPE by replacing 10000, the rotary base of $\theta_n={10000}^{-2n/d}$ in the original RoPE, with a larger value and providing longer fine-tuning text. In this work, we first observe that fine-tuning a RoPE-based LLM with either a smaller or larger base in pre-training context length could significantly enhance its extrapolation performance. After that, we propose \textbf{\textit{Scaling Laws of RoPE-based Extrapolation}}, a unified framework from the periodic perspective, to describe the relationship between the extrapolation performance and base value as well as tuning context length. In this process, we also explain the origin of the RoPE-based extrapolation issue by \textbf{\textit{critical dimension for extrapolation}}. Besides these observations and analyses, we achieve extrapolation up to 1 million context length within only 16K training length on LLaMA2 7B and 13B.

翻译：基于旋转位置编码的大型语言模型的外推能力目前是一个备受关注的话题。解决LLM外推问题的主流方法涉及修改RoPE，将原始RoPE中$\theta_n={10000}^{-2n/d}$的旋转基数10000替换为更大的值，并提供更长的微调文本。在本研究中，我们首先观察到，在预训练上下文长度内使用更小或更大的基数微调基于RoPE的LLM，可以显著提升其外推性能。之后，我们提出了**基于RoPE的外推缩放定律**，这是一个从周期性角度出发的统一框架，用于描述外推性能与基数值及微调上下文长度之间的关系。在此过程中，我们还通过**外推关键维度**解释了基于RoPE的外推问题的根源。除了这些观察和分析，我们在LLaMA2 7B和13B模型上，仅用16K训练长度就实现了高达100万上下文长度的外推。