Every Transformer architecture dedicates enormous capacity to learning rich representations in semantic embedding space -- yet the rotation manifold acted upon by Rotary Positional Embeddings (RoPE) has been treated as a fixed, hand-crafted structure, populated only by discrete ordinal indices. We argue that this rotation space is a largely overlooked second dimension of expressivity in the attention mechanism, one whose systematic exploration may open a new door for attention-based architectures. The analogy to complex numbers is instructive: just as introducing the imaginary axis -- orthogonal to and independent of the real line -- unlocked new algebraic structure once believed impossible, treating the rotation manifold as a learnable, signal-conditioned space opens an orthogonal degree of freedom in attention. In this framing, the token embedding encodes the semantic (real) component of a representation -- what a token means -- while the rotation encodes its dynamic (imaginary) component -- how it relates to every other token across time, position, and context. We introduce SIREN-RoPE, a concrete instantiation of this idea, which populates the rotation dimension with heterogeneous signals -- continuous timestamps, cyclical temporal patterns, and categorical metadata -- via a dual-branch Sinusoidal Representation Network (SIREN). As a proof of concept, we evaluate on a production-scale news feed dataset from a major social network using a generative recommender as the ranking model, demonstrating that activating this hidden dimension yields consistent improvements across calibration and ranking objectives with negligible computational overhead. We invite the community to view the rotation space not as a solved positional-encoding detail, but as an untapped axis whose rich structure may prove as consequential for attention as the imaginary unit proved for algebra.

翻译：每个Transformer架构都投入了巨大的容量来在语义嵌入空间中学习丰富的表示——然而，由旋转位置编码（RoPE）作用的旋转流形却被视为一个固定的手工设计结构，仅填充了离散的序数索引。我们认为，这一旋转空间是注意力机制中一个在很大程度上被忽视的第二维度表达能力，对其的系统性探索可能为基于注意力的架构开启新的大门。与复数的类比具有启发性：正如引入虚轴（与实轴正交且独立）解锁了曾被认为不可能的代数新结构一样，将旋转流形视为可学习且受信号调节的空间，为注意力提供了一个正交的自由度。在此框架下，令牌嵌入编码了表示的语义（实）分量——即令牌的含义；而旋转则编码了其动态（虚）分量——即令牌如何随时间、位置和上下文与每个其他令牌相关联。我们提出了SIREN-RoPE作为这一思想的具体实例化，通过双分支正弦表示网络（SIREN），用异构信号——连续时间戳、周期性时间模式和分类元数据——来填充旋转维度。作为概念验证，我们使用来自某主要社交网络的工业级新闻推送数据集，以生成式推荐器作为排序模型进行评估，结果表明激活这一隐藏维度在校准与排序目标上均能带来一致性的改进，且计算开销微乎其微。我们呼吁学界不要将旋转空间视为一个已解决的位置编码细节，而应视其为一个未开发的轴，其丰富结构可能对注意力的意义不亚于虚单位对代数的意义。