In English and other languages, multiple adjectives in a complex noun phrase show intricate ordering patterns that have been a target of much linguistic theory. These patterns offer an opportunity to assess the ability of language models (LMs) to learn subtle rules of language involving factors that cross the traditional divisions of syntax, semantics, and pragmatics. We review existing hypotheses designed to explain Adjective Order Preferences (AOPs) in humans and develop a setup to study AOPs in LMs: we present a reusable corpus of adjective pairs and define AOP measures for LMs. With these tools, we study a series of LMs across intermediate checkpoints during training. We find that all models' predictions are much closer to human AOPs than predictions generated by factors identified in theoretical linguistics. At the same time, we demonstrate that the observed AOPs in LMs are strongly correlated with the frequency of the adjective pairs in the training data and report limited generalization to unseen combinations. This highlights the difficulty in establishing the link between LM performance and linguistic theory. We therefore conclude with a road map for future studies our results set the stage for, and a discussion of key questions about the nature of knowledge in LMs and their ability to generalize beyond the training sets.

翻译：在英语及其他语言中，复杂名词短语中的多个形容词呈现出错综复杂的排序规律，这一现象长期成为语言学理论的研究焦点。这些规律为评估语言模型学习涉及句法、语义和语用等传统分野交叉因素的微妙语言规则能力提供了契机。本文系统梳理了解释人类形容词顺序偏好的现有假说，构建了研究语言模型中形容词顺序偏好的实验框架：我们提出了可复用的形容词配对语料库，并定义了适用于语言模型的形容词顺序偏好度量指标。借助这些工具，我们系统研究了训练过程中不同中间检查点上的一系列语言模型。研究发现，所有模型的预测结果相较于理论语言学已识别因素生成的预测，都更接近人类的形容词顺序偏好。同时，我们证明语言模型中观察到的形容词顺序偏好与训练数据中形容词配对的频率高度相关，且对未见组合的泛化能力有限。这凸显了建立语言模型性能与语言学理论之间关联的困难性。最后，我们基于研究结果提出了未来研究的路线图，并就语言模型知识的本质及其超越训练集的泛化能力等关键问题展开讨论。