In-context learning (ICL) allows some autoregressive models to solve tasks via next-token prediction and without needing further training. This has led to claims about these model's ability to solve (learn) unseen tasks with only a few shots (exemplars) in the prompt. However, deduction does not always imply learning, as ICL does not explicitly encode a given observation. Instead, the models rely on their prior knowledge and the exemplars given, if any. We argue that, mathematically, ICL does constitute learning, but its full characterisation requires empirical work. We then carry out a large-scale analysis of ICL ablating out or accounting for memorisation, pretraining, distributional shifts, and prompting style and phrasing. We find that ICL is an effective learning paradigm, but limited in its ability to learn and generalise to unseen tasks. We note that, in the limit where exemplars become more numerous, accuracy is insensitive to exemplar distribution, model, prompt style, and the input's linguistic features. Instead, it deduces patterns from regularities in the prompt, which leads to distributional sensitivity, especially in prompting styles such as chain-of-thought. Given the varied accuracies on formally similar tasks, we conclude that autoregression's ad-hoc encoding is not a robust mechanism, and suggests limited all-purpose generalisability.

翻译：上下文学习（ICL）使得某些自回归模型能够通过下一个词预测来解决任务，而无需进一步训练。这引发了关于这些模型仅通过提示中的少量示例（样本）就能解决（学习）未见任务的能力的论断。然而，推理并不总是意味着学习，因为ICL并未显式编码给定的观察结果。相反，模型依赖于其先验知识以及可能提供的示例。我们认为，从数学上讲，ICL确实构成了一种学习，但其完整特征需要实证研究来刻画。随后，我们开展了一项大规模分析，对ICL进行了消融实验，并考虑了记忆效应、预训练、分布偏移以及提示风格和措辞的影响。我们发现ICL是一种有效的学习范式，但在学习和泛化到未见任务方面能力有限。我们注意到，当示例数量趋于大量时，准确率对示例分布、模型、提示风格以及输入的语言特征均不敏感。相反，它从提示中的规律性推断模式，这导致了分布敏感性，在诸如思维链等提示风格中尤为明显。鉴于在形式相似的任务上表现出的多样准确率，我们得出结论：自回归的临时编码机制并非鲁棒的方法，并表明其通用泛化能力有限。