Prior work has shown that large language models (LLMs) often converge to accurate input embedding for numbers, based on sinusoidal representations. In this work, we quantify that these representations are in fact strikingly systematic, to the point of being almost perfectly universal: different LLM families develop equivalent sinusoidal structures, and number representations are broadly interchangeable in a large swathe of experimental setups. We show that properly factoring in this characteristic is crucial when it comes to assessing how accurately LLMs encode numeric and other ordinal information, and that mechanistically enhancing this sinusoidality can also lead to reductions of LLMs' arithmetic errors.

翻译：先前研究表明，大型语言模型（LLM）基于正弦表示会收敛到精确的数字输入嵌入。本研究进一步量化发现，这些表示实际上具有惊人的系统性，几乎达到近乎完美的通用程度：不同LLM家族会发展出等效的正弦结构，且在大量实验设置中数字表示广泛可互换。我们证明，在评估LLM对数字及其他序数信息的编码准确度时，合理考虑这一特征至关重要，而通过机制性增强这种正弦性还能减少LLM的算术错误。