Pretrained transformers exhibit the remarkable ability of in-context learning (ICL): they can learn tasks from just a few examples provided in the prompt without updating any weights. This raises a foundational question: can ICL solve fundamentally $\textit{new}$ tasks that are very different from those seen during pretraining? To probe this question, we examine ICL's performance on linear regression while varying the diversity of tasks in the pretraining dataset. We empirically demonstrate a $\textit{task diversity threshold}$ for the emergence of ICL. Below this threshold, the pretrained transformer cannot solve unseen regression tasks as it behaves like a Bayesian estimator with the $\textit{non-diverse pretraining task distribution}$ as the prior. Beyond this threshold, the transformer significantly outperforms this estimator; its behavior aligns with that of ridge regression, corresponding to a Gaussian prior over $\textit{all tasks}$, including those not seen during pretraining. These results highlight that, when pretrained on data with task diversity greater than the threshold, transformers $\textit{can}$ solve fundamentally new tasks in-context. Importantly, this capability hinges on it deviating from the Bayes optimal estimator with the pretraining distribution as the prior. This study underscores, in a concrete example, the critical role of task diversity, alongside data and model scale, in the emergence of ICL. Code is available at https://github.com/mansheej/icl-task-diversity.

翻译：预训练Transformer展现出上下文学习（ICL）的卓越能力：它们仅通过提示中提供的少量示例即可学习任务，而无需更新任何权重。这引出一个根本性问题：ICL能否解决与预训练期间所见任务截然不同的全新任务？为探究此问题，我们考察了ICL在线性回归中的表现，同时改变预训练数据集中任务的多样性。我们通过实验证实了ICL涌现所需的**任务多样性阈值**。低于此阈值时，预训练Transformer无法解决未见的回归任务，其行为类似以**非多样化预训练任务分布**为先验的贝叶斯估计器。超过此阈值时，Transformer显著优于该估计器；其行为与岭回归一致，对应所有任务（包括预训练中未见任务）上的高斯先验。这些结果表明，当在任务多样性超过阈值的预训练数据上时，Transformer确实能通过上下文学习解决全新任务。重要的是，这一能力源于它对以预训练分布为先验的贝叶斯最优估计器的偏离。本研究通过具体实例强调了任务多样性（与数据和模型规模并列）在ICL涌现中的关键作用。代码见 https://github.com/mansheej/icl-task-diversity。