Large language models (LLMs) have initiated a paradigm shift in transfer learning. In contrast to the classic pretraining-then-finetuning procedure, in order to use LLMs for downstream prediction tasks, one only needs to provide a few demonstrations, known as in-context examples, without adding more or updating existing model parameters. This in-context learning (ICL) capability of LLMs is intriguing, and it is not yet fully understood how pretrained LLMs acquire such capabilities. In this paper, we investigate the reason why a transformer-based language model can accomplish in-context learning after pre-training on a general language corpus by proposing one hypothesis that LLMs can simulate kernel regression with internal representations when faced with in-context examples. More concretely, we first prove that Bayesian inference on in-context prompts can be asymptotically understood as kernel regression $\hat y = \sum_i y_i K(x, x_i)/\sum_i K(x, x_i)$ as the number of in-context demonstrations grows. Then, we empirically investigate the in-context behaviors of language models. We find that during ICL, the attention and hidden features in LLMs match the behaviors of a kernel regression. Finally, our theory provides insights into multiple phenomena observed in the ICL field: why retrieving demonstrative samples similar to test samples can help, why ICL performance is sensitive to the output formats, and why ICL accuracy benefits from selecting in-distribution and representative samples.

翻译：大型语言模型（LLMs）引发了迁移学习领域的范式转变。与经典的预训练-微调流程不同，使用LLMs进行下游预测任务时，仅需提供少量示例（称为上下文示例），而无需增加或更新现有模型参数。这种LLMs的上下文学习（ICL）能力令人着迷，但预训练LLMs如何获得这种能力仍未被完全理解。本文通过提出一个假设：当面对上下文示例时，LLMs可以利用内部表示模拟核回归，从而探究基于Transformer的语言模型在通用语言语料库上预训练后为何能实现上下文学习。具体而言，我们首先证明，随着上下文示例数量的增加，上下文提示上的贝叶斯推断可以渐近地理解为核回归 $\hat y = \sum_i y_i K(x, x_i)/\sum_i K(x, x_i)$。接着，我们通过实验研究了语言模型的上下文行为，发现ICL过程中LLMs的注意力机制和隐藏特征与核回归行为相符。最后，我们的理论为ICL领域观察到的多个现象提供了洞见：为何检索与测试样本相似的演示示例有助于提升性能、为何ICL性能对输出格式敏感，以及为何选择分布内且具代表性的样本能提升ICL准确性。