Rotary Position Embedding (RoPE) is the de facto positional encoding in large language models due to its ability to encode relative positions and support length extrapolation. When adapted to vision transformers, the standard axial formulation decomposes two-dimensional spatial positions into horizontal and vertical components, implicitly restricting positional encoding to axis-aligned directions. We identify this directional constraint as a fundamental limitation of the standard axial 2D RoPE, which hinders the modeling of oblique spatial relationships that naturally exist in natural images. To overcome this limitation, we propose Spiral RoPE, a simple yet effective extension that enables multi-directional positional encoding by partitioning embedding channels into multiple groups associated with uniformly distributed directions. Each group is rotated according to the projection of the patch position onto its corresponding direction, allowing spatial relationships to be encoded beyond the horizontal and vertical axes. Across a wide range of vision tasks including classification, segmentation, and generation, Spiral RoPE consistently improves performance. Qualitative analysis of attention maps further show that Spiral RoPE exhibits more concentrated activations on semantically relevant objects and better respects local object boundaries, highlighting the importance of multi-directional positional encoding in vision transformers.

翻译：旋转位置嵌入（RoPE）因其能够编码相对位置并支持长度外推，已成为大语言模型中事实上的位置编码方法。当将其适配到视觉Transformer时，标准的轴向公式将二维空间位置分解为水平和垂直分量，这隐含地将位置编码限制在轴对齐的方向上。我们指出这种方向性约束是标准轴向二维RoPE的一个根本性局限，它阻碍了对自然图像中天然存在的倾斜空间关系的建模。为了克服这一局限，我们提出了螺旋RoPE，这是一种简单而有效的扩展，它通过将嵌入通道划分为多个与均匀分布方向相关联的组，从而实现多方向的位置编码。每个组根据图像块位置在其对应方向上的投影进行旋转，从而允许空间关系在水平和垂直轴之外也能被编码。在包括分类、分割和生成在内的广泛视觉任务中，螺旋RoPE均能持续提升性能。对注意力图的定性分析进一步表明，螺旋RoPE在语义相关物体上表现出更集中的激活，并且更好地尊重了局部物体边界，这凸显了在视觉Transformer中进行多方向位置编码的重要性。