In-context learning, where pre-trained language models learn to perform tasks from task examples and instructions in their contexts, has attracted much attention in the NLP community. However, the ability of in-context learning is not fully exploited because language models are not explicitly trained to learn in context. To this end, we propose PICL (Pre-training for In-Context Learning), a framework to enhance the language models' in-context learning ability by pre-training the model on a large collection of "intrinsic tasks" in the general plain-text corpus using the simple language modeling objective. PICL encourages the model to infer and perform tasks by conditioning on the contexts while maintaining task generalization of pre-trained models. We evaluate the in-context learning performance of the model trained with PICL on seven widely-used text classification datasets and the Super-NaturalInstrctions benchmark, which contains 100+ NLP tasks formulated to text generation. Our experiments show that PICL is more effective and task-generalizable than a range of baselines, outperforming larger language models with nearly 4x parameters. The code is publicly available at https://github.com/thu-coai/PICL.

翻译：上下文学习（in-context learning）中，预训练语言模型通过上下文中的任务示例和指令来学习执行任务，这一能力已引起自然语言处理（NLP）社区的广泛关注。然而，由于语言模型并未被显式训练以在上下文中学习，其上下文学习能力尚未得到充分挖掘。为此，我们提出了PICL（面向上下文学习的预训练）框架，该框架通过在通用纯文本语料库上利用简单的语言建模目标对大量“内在任务”进行预训练，从而增强语言模型的上下文学习能力。PICL鼓励模型基于上下文推断并执行任务，同时保持预训练模型的任务泛化性。我们在七个广泛使用的文本分类数据集以及包含100余项文本生成任务的Super-NaturalInstructions基准上，评估了经PICL训练的模型的上下文学习性能。实验表明，PICL相比多种基线方法更有效且更具任务泛化能力，其性能甚至优于参数规模近4倍的更大语言模型。相关代码已公开于https://github.com/thu-coai/PICL。