It is well noted that coordinate based MLPs benefit -- in terms of preserving high-frequency information -- through the encoding of coordinate positions as an array of Fourier features. Hitherto, the rationale for the effectiveness of these positional encodings has been solely studied through a Fourier lens. In this paper, we strive to broaden this understanding by showing that alternative non-Fourier embedding functions can indeed be used for positional encoding. Moreover, we show that their performance is entirely determined by a trade-off between the stable rank of the embedded matrix and the distance preservation between embedded coordinates. We further establish that the now ubiquitous Fourier feature mapping of position is a special case that fulfills these conditions. Consequently, we present a more general theory to analyze positional encoding in terms of shifted basis functions. To this end, we develop the necessary theoretical formulae and empirically verify that our theoretical claims hold in practice. Codes available at https://github.com/osiriszjq/Rethinking-positional-encoding.

翻译：众所周知,基于协调的 MLPs 通过将坐标位置编码成一系列Fourier 特征,对基于协调的 MLPs 有好处 -- -- 在保存高频信息方面 -- -- 通过将坐标位置编码为一系列Fourier 特征。 Hitherto,这些位置编码有效性的理由完全通过Fourier 镜头来研究。在本文中,我们努力扩大这一认识,显示非四级嵌入功能确实可用于定位编码。此外,我们显示,它们的性能完全取决于嵌入矩阵的稳定级别与嵌入坐标之间的距离保存之间的权衡。我们进一步确定,现在无处不在的 Fourier 特征绘图是满足这些条件的一个特殊案例。因此,我们提出了一个更笼统的理论,用改变的基础功能来分析位置编码。为此,我们开发了必要的理论公式,并从经验上核实我们的理论主张在实践中是否有效。 代码可在 https://github.com/osiriszq/Re thinking-positional-encoding 中查阅。