This study discusses the effects of positional encoding on recurrent neural networks (RNNs) utilizing synthetic benchmarks. Positional encoding "time-stamps" data points in time series and complements the capabilities of Transformer neural networks, which lack an inherent mechanism for representing the data order. By contrast, RNNs can encode the temporal information of data points on their own, rendering their use of positional encoding seemingly "redundant". Nonetheless, empirical investigations reveal the effectiveness of positional encoding even when coupled with RNNs, specifically for handling a large vocabulary that yields diverse observations. These findings pave the way for a new line of research on RNNs, concerning the combination of input-driven and autonomous time representation. Additionally, biological implications of the computational/simulational results are discussed, in the light of the affinity between the sinusoidal implementation of positional encoding and neural oscillations in biological brains.

翻译：本研究利用合成基准任务探讨了位置编码对循环神经网络的影响。位置编码对时间序列中的数据点进行“时间标记”，弥补了Transformer神经网络缺乏数据顺序内在表示机制的不足。相比之下，循环神经网络本身能够编码数据点的时间信息，这使得位置编码的使用看似“冗余”。然而，实证研究表明，即使在循环神经网络中，位置编码在处理产生多样化观测结果的大词汇量时仍具有有效性。这些发现为循环神经网络研究开辟了新方向，涉及输入驱动与自主时间表示的融合。此外，基于位置编码的正弦实现与生物大脑神经振荡之间的相似性，本文还讨论了计算/仿真结果的生物学意义。