Large language models (LLMs) have shown the emergent capability of in-context learning (ICL). One line of research has explained ICL as functionally performing gradient descent. In this paper, we introduce a new way of diagnosing whether ICL is functionally equivalent to gradient-based learning. Our approach is based on the inverse frequency effect (IFE) -- a phenomenon in which an error-driven learner is expected to show larger updates when trained on infrequent examples than frequent ones. The IFE has previously been studied in psycholinguistics because humans show this effect in the context of structural priming (the tendency for people to produce sentence structures they have encountered recently); the IFE has been used as evidence that human structural priming must involve error-driven learning mechanisms. In our experiments, we simulated structural priming within ICL and found that LLMs display the IFE, with the effect being stronger in larger models. We conclude that ICL is indeed a type of gradient-based learning, supporting the hypothesis that a gradient component is implicitly computed in the forward pass during ICL. Our results suggest that both humans and LLMs make use of gradient-based, error-driven processing mechanisms.

翻译：大型语言模型（LLMs）已展现出上下文学习（ICL）这一涌现能力。一系列研究将ICL解释为在功能上执行梯度下降。本文提出了一种新的诊断方法，用于判断ICL在功能上是否等同于基于梯度的学习。我们的方法基于逆频率效应（IFE）——一种现象，即误差驱动型学习者在接受低频样本训练时预期会比高频样本产生更大的更新。逆频率效应先前已在心理语言学中得到研究，因为人类在结构启动（人们倾向于使用近期接触过的句子结构的现象）情境中表现出该效应；逆频率效应曾被用作证据，表明人类的结构启动必然涉及误差驱动的学习机制。在我们的实验中，我们在ICL框架内模拟了结构启动，发现大型语言模型展现出逆频率效应，且该效应在更大规模的模型中更为显著。我们得出结论：上下文学习确实是一种基于梯度的学习，这支持了在ICL前向传播过程中隐式计算梯度成分的假说。我们的结果表明，人类和大型语言模型都利用了基于梯度的误差驱动处理机制。