Large language models have shown tremendous performance in a variety of tasks. In-context learning -- the ability to improve at a task after being provided with a number of demonstrations -- is seen as one of the main contributors to their success. In the present paper, we demonstrate that the in-context learning abilities of large language models can be recursively improved via in-context learning itself. We coin this phenomenon meta-in-context learning. Looking at two idealized domains, a one-dimensional regression task and a two-armed bandit task, we show that meta-in-context learning adaptively reshapes a large language model's priors over expected tasks. Furthermore, we find that meta-in-context learning modifies the in-context learning strategies of such models. Finally, we extend our approach to a benchmark of real-world regression problems where we observe competitive performance to traditional learning algorithms. Taken together, our work improves our understanding of in-context learning and paves the way toward adapting large language models to the environment they are applied purely through meta-in-context learning rather than traditional finetuning.

翻译：大型语言模型在各类任务中展现了卓越的性能。上下文学习——即在获得若干示例后提升任务表现的能力——被认为是其成功的主要因素之一。本文证明，大型语言模型的上下文学习能力可以通过上下文学习本身进行递归式改进。我们将这一现象命名为元上下文学习。通过考察一维回归任务和双臂老虎机任务这两个理想化领域，我们表明元上下文学习能够自适应地重塑大型语言模型对预期任务的先验知识。此外，我们发现元上下文学习会改变此类模型的上下文学习策略。最终，我们将方法扩展到真实世界回归问题的基准测试中，观察到其与传统学习算法相比具有竞争力的表现。综合而言，我们的工作加深了对上下文学习的理解，并开辟了通过纯元上下文学习（而非传统微调）使大型语言模型适应应用环境的路径。