We study the phenomenon of \textit{in-context learning} (ICL) exhibited by large language models, where they can adapt to a new learning task, given a handful of labeled examples, without any explicit parameter optimization. Our goal is to explain how a pre-trained transformer model is able to perform ICL under reasonable assumptions on the pre-training process and the downstream tasks. We posit a mechanism whereby a transformer can achieve the following: (a) receive an i.i.d. sequence of examples which have been converted into a prompt using potentially-ambiguous delimiters, (b) correctly segment the prompt into examples and labels, (c) infer from the data a \textit{sparse linear regressor} hypothesis, and finally (d) apply this hypothesis on the given test example and return a predicted label. We establish that this entire procedure is implementable using the transformer mechanism, and we give sample complexity guarantees for this learning framework. Our empirical findings validate the challenge of segmentation, and we show a correspondence between our posited mechanisms and observed attention maps for step (c).

翻译：我们研究大型语言模型展现的\textit{上下文学习}（ICL）现象：这类模型无需显式参数优化，仅凭少量标注示例即可适应新学习任务。本文旨在解释在合理的预训练过程与下游任务假设下，预训练Transformer模型如何实现ICL。我们提出一种机制，使Transformer能够实现以下步骤：(a) 接收独立同分布示例序列（该序列已通过潜在歧义分隔符转换为提示），(b) 正确将提示分割为样例与标签，(c) 从数据中推断\textit{稀疏线性回归器}假设，最终(d) 将该假设应用于测试样例并返回预测标签。我们证明该完整流程可在Transformer机制中实现，并给出了该学习框架的样本复杂度保证。实验验证了分割环节的挑战性，并展示了我们提出的机制与步骤(c)中观测到的注意力图之间的对应关系。