In-context learning (ICL) allows some autoregressive models to solve tasks via next-token prediction and without needing further training. This has led to claims about these model's ability to solve (learn) unseen tasks with only a few shots (exemplars) in the prompt. However, deduction does not always imply learning, as ICL does not explicitly encode a given observation. Instead, the models rely on their prior knowledge and the exemplars given, if any. We argue that, mathematically, ICL fits the definition of learning; however, its full characterisation requires empirical work. We then carry out a large-scale analysis of ICL ablating out or accounting for memorisation, pretraining, distributional shifts, and prompting style and phrasing. We find that, empirically, ICL is limited in its ability to learn and generalise to unseen tasks. Namely, in the limit where exemplars become more numerous, accuracy is insensitive to exemplar distribution, model, prompt style, and the input's linguistic features. Instead, it deduces patterns from regularities in the prompt, which leads to distributional sensitivity, especially in prompting styles such as chain-of-thought. Given the varied accuracies and on formally similar tasks, we conclude that autoregression's ad-hoc encoding is not a robust mechanism for learning, and suggests limited all-purpose generalisability.

翻译：上下文学习（ICL）使得某些自回归模型能够通过下一个词预测来解决任务，而无需进一步训练。这引发了关于这些模型仅通过提示中的少量示例（样本）就能解决（学习）未见任务能力的论断。然而，演绎并不总是意味着学习，因为ICL并未显式编码给定的观察结果。相反，模型依赖于其先验知识和所提供的示例（如果有的话）。我们认为，从数学角度而言，ICL符合学习的定义；然而，其完整特性需要实证研究来阐明。我们随后对ICL进行了大规模分析，通过消融实验或控制变量方法考察了记忆效应、预训练、分布偏移以及提示风格与措辞的影响。我们发现，从实证角度看，ICL在学习及泛化至未见任务方面的能力存在局限。具体而言，当示例数量趋于极限时，模型准确率对示例分布、模型架构、提示风格及输入文本的语言学特征均不敏感。相反，模型通过从提示中的规律性进行模式推演，这导致了分布敏感性——在思维链等提示风格中尤为明显。鉴于在形式相似任务中观察到的准确率差异，我们得出结论：自回归机制的特设编码并非稳健的学习机制，这表明其通用泛化能力存在局限。