What mechanisms underlie linguistic generalization in large language models (LLMs)? This question has attracted considerable attention, with most studies analyzing the extent to which the language skills of LLMs resemble rules. As of yet, it is not known whether linguistic generalization in LLMs could equally well be explained as the result of analogical processes, which can be formalized as similarity operations on stored exemplars. A key shortcoming of prior research is its focus on linguistic phenomena with a high degree of regularity, for which rule-based and analogical approaches make the same predictions. Here, we instead examine derivational morphology, specifically English adjective nominalization, which displays notable variability. We introduce a new method for investigating linguistic generalization in LLMs: focusing on GPT-J, we fit cognitive models that instantiate rule-based and analogical learning to the LLM training data and compare their predictions on a set of nonce adjectives with those of the LLM, allowing us to draw direct conclusions regarding underlying mechanisms. As expected, rule-based and analogical models explain the predictions of GPT-J equally well for adjectives with regular nominalization patterns. However, for adjectives with variable nominalization patterns, the analogical model provides a much better match. Furthermore, GPT-J's behavior is sensitive to the individual word frequencies, even for regular forms, a behavior that is consistent with an analogical account of regular forms but not a rule-based one. These findings refute the hypothesis that GPT-J's linguistic generalization on adjective nominalization involves rules, suggesting similarity operations on stored exemplars as the underlying mechanism. Overall, our study suggests that analogical processes play a bigger role in the linguistic generalization of LLMs than previously thought.

翻译：大型语言模型（LLMs）的语言泛化机制是什么？这一问题已引起广泛关注，多数研究聚焦于分析LLMs语言能力在多大程度上遵循规则。迄今为止，尚不清楚LLMs的语言泛化是否同样可以通过类比过程来解释——该过程可形式化为对存储范例的相似性运算。先前研究的关键局限在于其关注高度规律性的语言现象，而基于规则与类比的方法对此类现象会作出相同预测。本文转而研究派生形态学，特别是具有显著变异性的英语形容词名词化现象。我们提出一种研究LLMs语言泛化的新方法：以GPT-J为研究对象，通过拟合基于规则与类比学习的认知模型至LLM训练数据，并比较这些模型与LLM对一组虚构形容词的预测结果，从而直接推断其内在机制。如预期所示，对于具有规则名词化模式的形容词，基于规则与类比的模型均能同等解释GPT-J的预测结果。然而，对于具有可变名词化模式的形容词，类比模型表现出显著更好的匹配度。此外，GPT-J的行为对单个词频具有敏感性——即使对于规则形式也是如此，这种行为与规则形式的类比解释相符，却与基于规则的解释相悖。这些发现否定了GPT-J在形容词名词化任务中依赖规则进行语言泛化的假说，表明其内在机制是对存储范例的相似性运算。总体而言，我们的研究表明类比过程在LLMs语言泛化中的作用比以往认知的更为重要。