Whether language models (LMs) have inductive biases that favor typologically frequent grammatical properties over rare, implausible ones has been investigated, typically using artificial languages (ALs) (White and Cotterell, 2021; Kuribayashi et al., 2024). In this paper, we extend these works from two perspectives. First, we extend their context-free AL formalization by adopting Generalized Categorial Grammar (GCG) (Wood, 2014), which allows ALs to cover attested but previously overlooked constructions, such as unbounded dependency and mildly context-sensitive structures. Second, our evaluation focuses more on the generalization ability of LMs to process unseen longer test sentences. Thus, our ALs better capture features of natural languages and our experimental paradigm leads to clearer conclusions -- typologically plausible word orders tend to be easier for LMs to productively generalize.

翻译：语言模型（LMs）是否具有归纳偏好，使其更倾向于类型学上常见的语法特性而非罕见、不合理特性，这一问题通常通过人工语言（ALs）进行研究（White and Cotterell, 2021; Kuribayashi et al., 2024）。本文从两个角度拓展了这些工作。首先，我们采用广义范畴语法（GCG）（Wood, 2014）扩展了其上下文无关的人工语言形式化方法，这使得人工语言能够涵盖已证实但先前被忽视的结构，例如无界依赖和轻度上下文敏感结构。其次，我们的评估更侧重于语言模型处理未见过的更长测试句子的泛化能力。因此，我们的人工语言能更好地捕捉自然语言的特征，且我们的实验范式能得出更清晰的结论——类型学上合理的词序往往更易于语言模型进行生产性泛化。