Large language models (LMs) are currently trained to predict tokens given document prefixes, enabling them to directly perform long-form generation and prompting-style tasks which can be reduced to document completion. Existing pretraining pipelines train LMs by concatenating random sets of short documents to create input contexts but the prior documents provide no signal for predicting the next document. We instead present In-Context Pretraining, a new approach where language models are pretrained on a sequence of related documents, thereby explicitly encouraging them to read and reason across document boundaries. We can do In-Context Pretraining by simply changing the document ordering so that each context contains related documents, and directly applying existing pretraining pipelines. However, this document sorting problem is challenging. There are billions of documents and we would like the sort to maximize contextual similarity for every document without repeating any data. To do this, we introduce approximate algorithms for finding related documents with efficient nearest neighbor search and constructing coherent input contexts with a graph traversal algorithm. Our experiments show In-Context Pretraining offers a simple and scalable approach to significantly enhance LMs'performance: we see notable improvements in tasks that require more complex contextual reasoning, including in-context learning (+8%), reading comprehension (+15%), faithfulness to previous contexts (+16%), long-context reasoning (+5%), and retrieval augmentation (+9%).

翻译：大语言模型（LM）目前通过根据文档前缀预测后续标记进行训练，使其能够直接执行可简化为文档补全的长文本生成和提示型任务。现有预训练流程通过拼接随机选取的短文档集创建输入上下文，但前序文档无法为预测后续文档提供任何信号。为此，我们提出上下文预训练（In-Context Pretraining），这是一种全新方法——语言模型在相关文档序列上进行预训练，从而明确鼓励其跨文档边界进行阅读与推理。通过简单地改变文档排序使每个上下文包含相关文档，并直接应用现有预训练流程即可实现上下文预训练。然而，这一文档排序问题极具挑战性：面对数十亿量级的文档，我们需要在无重复数据的前提下最大化每篇文档的上下文相似性。针对该问题，我们引入近似算法：通过高效最近邻搜索寻找相关文档，并利用图遍历算法构建连贯的输入上下文。实验表明，上下文预训练提供了一种简单且可扩展的方法，能显著提升语言模型性能：在需要更复杂上下文推理的任务中，我们观察到显著改进，包括上下文学习（+8%）、阅读理解（+15%）、对先前上下文的忠实度（+16%）、长上下文推理（+5%）以及检索增强（+9%）。