In-context learning refers to the ability of a model to condition on a prompt sequence consisting of in-context examples (input-output pairs corresponding to some task) along with a new query input, and generate the corresponding output. Crucially, in-context learning happens only at inference time without any parameter updates to the model. While large language models such as GPT-3 exhibit some ability to perform in-context learning, it is unclear what the relationship is between tasks on which this succeeds and what is present in the training data. To make progress towards understanding in-context learning, we consider the well-defined problem of training a model to in-context learn a function class (e.g., linear functions): that is, given data derived from some functions in the class, can we train a model to in-context learn "most" functions from this class? We show empirically that standard Transformers can be trained from scratch to perform in-context learning of linear functions -- that is, the trained model is able to learn unseen linear functions from in-context examples with performance comparable to the optimal least squares estimator. In fact, in-context learning is possible even under two forms of distribution shift: (i) between the training data of the model and inference-time prompts, and (ii) between the in-context examples and the query input during inference. We also show that we can train Transformers to in-context learn more complex function classes -- namely sparse linear functions, two-layer neural networks, and decision trees -- with performance that matches or exceeds task-specific learning algorithms. Our code and models are available at https://github.com/dtsip/in-context-learning .

翻译：上下文学习是指模型根据由上下文示例（对应于某一任务的输入-输出对）和新查询输入组成的提示序列，生成相应输出的能力。关键的是，上下文学习仅在推理时发生，无需对模型进行任何参数更新。虽然GPT-3等大型语言模型展现出一定的上下文学习能力，但尚不清楚成功执行这类任务与训练数据内容之间存在何种关联。为了推进对上下文学习的理解，我们考虑了一个定义明确的问题：训练模型用于上下文学习某一函数类（例如线性函数）。具体而言，给定从该函数类中某些函数导出的数据，我们能否训练模型以从该类中“大多数”函数进行上下文学习？我们的实验表明，标准Transformer可以从头训练，以执行线性函数的上下文学习——即训练后的模型能够从上下文示例中学习未见过的线性函数，其性能可与最优最小二乘估计器相媲美。事实上，即使在两种分布偏移情况下，上下文学习仍然可行：（i）模型训练数据与推理时提示之间的偏移，以及（ii）推理时上下文示例与查询输入之间的偏移。我们还证明，可以训练Transformer上下文学习更复杂的函数类——即稀疏线性函数、双层神经网络和决策树——其性能达到或超过特定任务的学习算法。我们的代码和模型可在 https://github.com/dtsip/in-context-learning 获取。