In-context learning (ICL) is a key building block of modern large language models, yet its theoretical mechanisms remain poorly understood. It is particularly mysterious how ICL operates in real-world applications where tasks have a common structure. In this work, we address this problem by analyzing a linear attention model trained on low-rank regression tasks. Within this setting, we precisely characterize the distribution of predictions and the generalization error in the high-dimensional limit. Moreover, we find that statistical fluctuations in finite pre-training data induce an implicit regularization. Finally, we identify a sharp phase transition of the generalization error governed by task structure. These results provide a framework for understanding how transformers learn to learn the task structure.

翻译：上下文学习（ICL）是现代大型语言模型的关键组成部分，但其理论机制仍不清晰。尤其令人费解的是，在任务具有共同结构的实际应用中，上下文学习如何发挥作用。本研究通过分析在低秩回归任务上训练的线性注意力模型来探讨这一问题。在此设定下，我们精确刻画了高维极限下预测的分布与泛化误差。此外，我们发现有限预训练数据中的统计波动会导致隐式正则化。最终，我们识别出由任务结构调控的泛化误差尖锐相变。这些结果为理解Transformer如何学习任务结构提供了分析框架。