Contemporary deep learning models effectively handle languages with diverse morphology despite not being directly integrated into them. Morphology and word order are closely linked, with the latter incorporated into transformer-based models through positional encodings. This prompts a fundamental inquiry: Is there a correlation between the morphological complexity of a language and the utilization of positional encoding in pre-trained language models? In pursuit of an answer, we present the first study addressing this question, encompassing 22 languages and 5 downstream tasks. Our findings reveal that the importance of positional encoding diminishes with increasing morphological complexity in languages. Our study motivates the need for a deeper understanding of positional encoding, augmenting them to better reflect the different languages under consideration.

翻译：尽管当代深度学习模型并未直接整合语言形态学，却能有效处理形态多样的语言。形态学与词序密切相关，而后者通过位置编码被整合到基于Transformer的模型中。这引发了一个根本性问题：在预训练语言模型中，语言的形态复杂度与位置编码的使用是否存在关联？为探究此问题，我们首次开展了涵盖22种语言和5项下游任务的系统性研究。研究结果表明，随着语言形态复杂度的增加，位置编码的重要性逐渐减弱。本研究揭示了深入理解位置编码的必要性，并指出需增强其表征能力以更好地适应不同语言特性。