Accurate syntactic representations are essential for robust generalization in natural language. Recent work has found that pre-training can teach language models to rely on hierarchical syntactic features - as opposed to incorrect linear features - when performing tasks after fine-tuning. We test what aspects of pre-training are important for endowing encoder-decoder Transformers with an inductive bias that favors hierarchical syntactic generalizations. We focus on architectural features (depth, width, and number of parameters), as well as the genre and size of the pre-training corpus, diagnosing inductive biases using two syntactic transformation tasks: question formation and passivization, both in English. We find that the number of parameters alone does not explain hierarchical generalization: model depth plays greater role than model width. We also find that pre-training on simpler language, such as child-directed speech, induces a hierarchical bias using an order-of-magnitude less data than pre-training on more typical datasets based on web text or Wikipedia; this suggests that in cognitively plausible language acquisition settings, neural language models may be more data-efficient than previously thought.

翻译：准确的句法表示对于自然语言的稳健泛化至关重要。近期研究发现，预训练能够使语言模型在进行微调后执行任务时，依赖层次化句法特征（而非错误线性特征）进行表征。我们测试了预训练中的哪些方面对于赋予编码器-解码器Transformer架构以偏好层次化句法泛化的归纳偏置至关重要。研究重点关注架构特征（深度、宽度、参数量）以及预训练语料的体裁和规模，通过两项句法变换任务（英文疑问句生成和被动化）诊断归纳偏置。结果发现：参数量本身并不能解释层次化泛化能力——模型深度比宽度作用更大；同时，使用儿童导向语言等简化语料进行预训练时，仅需网络文本或维基百科等典型数据集十分之一的数据量即可诱导出层次化偏置——这表明在认知合理的语言习得场景中，神经语言模型可能比先前认知更具数据效率。