Multilingual pre-training significantly improves many multilingual NLP tasks, including machine translation. Most existing methods are based on some variants of masked language modeling and text-denoising objectives on monolingual data. Multilingual pre-training on monolingual data ignores the availability of parallel data in many language pairs. Also, some other works integrate the available human-generated parallel translation data in their pre-training. This kind of parallel data is definitely helpful, but it is limited even in high-resource language pairs. This paper introduces a novel semi-supervised method, SPDG, that generates high-quality pseudo-parallel data for multilingual pre-training. First, a denoising model is pre-trained on monolingual data to reorder, add, remove, and substitute words, enhancing the pre-training documents' quality. Then, we generate different pseudo-translations for each pre-training document using dictionaries for word-by-word translation and applying the pre-trained denoising model. The resulting pseudo-parallel data is then used to pre-train our multilingual sequence-to-sequence model, PEACH. Our experiments show that PEACH outperforms existing approaches used in training mT5 and mBART on various translation tasks, including supervised, zero- and few-shot scenarios. Moreover, PEACH's ability to transfer knowledge between similar languages makes it particularly useful for low-resource languages. Our results demonstrate that with high-quality dictionaries for generating accurate pseudo-parallel, PEACH can be valuable for low-resource languages.

翻译：多语言预训练显著改进了包括机器翻译在内的多项多语言自然语言处理任务。现有方法大多基于单语数据上的掩码语言建模和文本去噪目标的变体。基于单语数据的多语言预训练忽略了诸多语言对中平行数据的可用性。此外，另有部分工作将人工生成的平行翻译数据整合至预训练过程中。此类平行数据固然有益，即便在资源丰富的语言对中也极为有限。本文提出一种新型半监督方法SPDG，可为多语言预训练生成高质量伪平行数据。首先，在单语数据上预训练去噪模型以执行词语重排序、添加、删除和替换操作，从而提升预训练文档质量；随后，利用词典进行逐词翻译并应用预训练去噪模型为每篇预训练文档生成不同伪翻译版本。由此得到的伪平行数据被用于预训练我们的多语言序列到序列模型PEACH。实验表明，在涵盖监督学习、零样本和少样本场景的多种翻译任务中，PEACH的表现优于用于训练mT5和mBART的现有方法。此外，PEACH在相似语言间传递知识的能力使其对低资源语言尤为实用。结果表明，若能使用高质量词典生成精确伪平行数据，PEACH对低资源语言具有重要价值。