Since the random language model was proposed by E. DeGiuli [Phys. Rev. Lett. 122, 128301], language models have been investigated intensively from the viewpoint of statistical mechanics. Recently, the existence of a Berezinskii--Kosterlitz--Thouless transition was numerically demonstrated in models with long-range interactions between symbols. In statistical mechanics, it has long been known that long-range interactions can induce phase transitions. Therefore, it has remained unclear whether phase transitions observed in language models originate from genuinely linguistic properties that are absent in conventional spin models. In this study, we construct a random language model with short-range interactions and numerically investigate its statistical properties. Our model belongs to the class of context-sensitive grammars in the Chomsky hierarchy and allows explicit reference to contexts. We find that a phase transition occurs even when the model refers only to contexts whose length remains constant with respect to the sentence length. This result indicates that finite-temperature phase transitions in language models are genuinely induced by the intrinsic nature of language, rather than by long-range interactions.

翻译：自E. DeGiuli提出随机语言模型[Phys. Rev. Lett. 122, 128301]以来，语言模型已从统计力学视角得到深入研究。近期，数值实验证实，在符号间存在长程相互作用的模型中存在Berezinskii-Kosterlitz-Thouless相变。统计力学中早已明确，长程相互作用可诱导相变。因此，语言模型中所观测到的相变是否源于传统自旋模型所不具备的真正语言学特性，这一问题至今尚未澄清。本研究构建了具有短程相互作用的随机语言模型，并对其统计特性进行了数值分析。该模型属于乔姆斯基层级中的上下文相关语法类别，允许显式引用上下文。我们发现，即便模型仅引用相对于句子长度保持恒定的上下文，相变依然会发生。这一结果表明，语言模型中的有限温度相变真正由语言的固有本质所诱导，而非源于长程相互作用。