Language models are typically evaluated on their success at predicting the distribution of specific words in specific contexts. Yet linguistic knowledge also encodes relationships between contexts, allowing inferences between word distributions. We investigate the degree to which pre-trained Transformer-based large language models (LLMs) represent such relationships, focusing on the domain of argument structure. We find that LLMs perform well in generalizing the distribution of a novel noun argument between related contexts that were seen during pre-training (e.g., the active object and passive subject of the verb spray), succeeding by making use of the semantically-organized structure of the embedding space for word embeddings. However, LLMs fail at generalizations between related contexts that have not been observed during pre-training, but which instantiate more abstract, but well-attested structural generalizations (e.g., between the active object and passive subject of an arbitrary verb). Instead, in this case, LLMs show a bias to generalize based on linear order. This finding points to a limitation with current models and points to a reason for which their training is data-intensive.s reported here are available at https://github.com/clay-lab/structural-alternations.

翻译：语言模型通常根据其在特定语境中预测特定词语分布的成功程度进行评估。然而，语言知识还编码了不同语境之间的关系，使得词语分布之间的推理成为可能。我们研究了基于预训练Transformer的大型语言模型（LLMs）在论元结构领域表征这类关系的程度。研究发现，对于预训练期间所见的相关语境（例如动词spray的主动宾语与被动主语）之间新出现的名词论元分布，LLMs表现良好，能够借助词嵌入空间中语义组织的结构特性实现泛化。然而，对于预训练期间未见过的相关语境间泛化（例如任意动词的主动宾语与被动主语之间），尽管这些语境体现了更抽象且被充分验证的结构性泛化，LLMs仍无法完成这一任务。在此情况下，LLMs表现出基于线性顺序进行泛化的偏好。这一发现揭示了当前模型的局限性，并指出其训练需要大量数据的原因。本报告所述结果公开于https://github.com/clay-lab/structural-alternations。