We explore the mechanism of in-context learning and propose a hypothesis using locate-and-project method. In shallow layers, the features of demonstrations are merged into their corresponding labels, and the features of the input text are aggregated into the last token. In deep layers, in-context heads make great contributions. In each in-context head, the value-output matrix extracts the labels' features. Query and key matrices compute the attention weights between the input text and each demonstration. The larger the attention weight is, the more label information is transferred into the last token for predicting the next word. Query and key matrices can be regarded as two towers for learning the similarity metric between the input text and each demonstration. Based on this hypothesis, we explain why imbalanced labels and demonstration order affect predictions. We conduct experiments on GPT2 large, Llama 7B, 13B and 30B. The results can support our analysis. Overall, our study provides a new method and a reasonable hypothesis for understanding the mechanism of in-context learning. Our code will be released on github.

翻译：本文探索上下文内学习的机制，并提出一种基于定位-投影方法的假说。在浅层，演示样本的特征被合并到对应标签中，而输入文本的特征则汇聚到最后一个标记。在深层，上下文内头部发挥重要作用。在每个上下文内头部中，值-输出矩阵负责提取标签特征，查询矩阵和键矩阵计算输入文本与每个演示样本之间的注意力权重。注意力权重越大，表示更多标签信息被传递至最后一个标记以用于预测下一个词。查询矩阵和键矩阵可视为两个塔结构，用于学习输入文本与各演示样本之间的相似度度量。基于这一假说，我们解释了标签不平衡和演示顺序影响预测的原因。我们在GPT2 large、Llama 7B、13B和30B模型上开展实验，结果支持了我们的分析。总体而言，本研究为理解上下文内学习机制提供了新方法和合理的假说。我们的代码将发布在GitHub上。