In this work, we explore the mechanism of in-context learning (ICL) on out-of-distribution (OOD) tasks that were not encountered during training. To achieve this, we conduct synthetic experiments where the objective is to learn OOD mathematical functions through ICL using a GPT-2 model. We reveal that Transformers may struggle to learn OOD task functions through ICL. Specifically, ICL performance resembles implementing a function within the pretraining hypothesis space and optimizing it with gradient descent based on the in-context examples. Additionally, we investigate ICL's well-documented ability to learn unseen abstract labels in context. We demonstrate that such ability only manifests in the scenarios without distributional shifts and, therefore, may not serve as evidence of new-task-learning ability. Furthermore, we assess ICL's performance on OOD tasks when the model is pretrained on multiple tasks. Both empirical and theoretical analyses demonstrate the existence of the \textbf{low-test-error preference} of ICL, where it tends to implement the pretraining function that yields low test error in the testing context. We validate this through numerical experiments. This new theoretical result, combined with our empirical findings, elucidates the mechanism of ICL in addressing OOD tasks.

翻译：在本研究中，我们探讨了上下文学习（ICL）在训练过程中未接触过的分布外（OOD）任务上的作用机制。为此，我们进行了合成实验，目标是通过使用GPT-2模型的ICL来学习OOD数学函数。我们发现，Transformer模型可能难以通过ICL学习OOD任务函数。具体而言，ICL的表现类似于在预训练假设空间中实现一个函数，并基于上下文示例通过梯度下降进行优化。此外，我们研究了ICL在上下文中学习未见抽象标签的公认能力。我们证明这种能力仅在无分布偏移的场景中显现，因此可能无法作为学习新任务能力的证据。进一步地，我们评估了模型在多个任务上预训练后，ICL在OOD任务上的表现。实证与理论分析均表明ICL存在**低测试误差偏好**，即其倾向于实现能在测试上下文中产生低测试误差的预训练函数。我们通过数值实验验证了这一结论。这一新的理论结果结合我们的实证发现，阐明了ICL处理OOD任务的作用机制。