Positional encoding is essential for large language models (LLMs) to represent sequence order, yet recent studies show that Rotary Position Embedding (RoPE) can induce massive activation. We investigate the source of these instabilities via a spectral analysis of RoPE, and show that its low-frequency components concentrate structured energy, producing low-rank, over-aligned attention patterns. We theoretically reveal that this low-frequency alignment manifests as activation noise, degrading stability during long-context extrapolation. To mitigate this effect, we introduce Denoising Rotary Position Embedding (DoPE), a training-free method that identifies and suppresses noisy attention heads using truncated matrix entropy, then reparameterizes their attention maps with an isotropic Gaussian distribution. Across a range of settings, DoPE improves length extrapolation performance without fine-tuning, increases robustness to perturbations, and boosts both needle-in-a-haystack and many-shot in-context learning tasks. These results suggest that selective positional encoding is key to robust extrapolation. Our project page is Project: https://The-physical-picture-of-LLMs.github.io

翻译：位置编码对于大型语言模型（LLM）表示序列顺序至关重要，然而近期研究表明旋转位置编码（RoPE）可能引发显著的激活不稳定性。我们通过RoPE的谱分析探究了这些不稳定性的来源，发现其低频分量会聚集结构化能量，产生低秩且过度对齐的注意力模式。我们从理论上揭示了这种低频对齐会表现为激活噪声，从而损害长上下文外推时的稳定性。为缓解此效应，我们提出了去噪旋转位置编码（DoPE），这是一种无需训练的方法：首先通过截断矩阵熵识别并抑制含噪注意力头，随后使用各向同性高斯分布对其注意力图进行重参数化。在多种实验设置中，DoPE无需微调即可提升长度外推性能，增强对扰动的鲁棒性，并显著提升“大海捞针”与多样本上下文学习任务的效果。这些结果表明选择性位置编码是实现稳健外推的关键。项目页面详见：https://The-physical-picture-of-LLMs.github.io