The recent rapid progress in pre-training Large Language Models has relied on using self-supervised language modeling objectives like next token prediction or span corruption. On the other hand, Machine Translation Systems are mostly trained using cross-lingual supervision that requires aligned data between source and target languages. We demonstrate that pre-training Large Language Models on a mixture of a self-supervised Language Modeling objective and the supervised Machine Translation objective, therefore including cross-lingual parallel data during pre-training, yields models with better in-context learning abilities. As pre-training is a very resource-intensive process and a grid search on the best mixing ratio between the two objectives is prohibitively expensive, we propose a simple yet effective strategy to learn it during pre-training.

翻译：近期大语言模型预训练的迅猛进展主要依赖于自监督语言建模目标，例如下一个词预测或跨度破坏。另一方面，机器翻译系统大多使用跨语言监督训练，需要源语言和目标语言之间的对齐数据。我们证明，在预训练过程中结合自监督语言建模目标与监督机器翻译目标（即在预训练中包含跨语言平行数据）所训练的模型，具有更优的上下文学习能力。由于预训练是一个资源密集型过程，且对两个目标的最佳混合比例进行网格搜索成本过高，我们提出一种简单而有效的策略，在预训练过程中自动学习该比例。