Recent studies have demonstrated the potential of cross-lingual transferability by training a unified Transformer encoder for multiple languages. In addition to involving the masked language model objective, existing cross-lingual pre-training works leverage sentence-level contrastive learning or plugs in extra cross-attention module to complement the insufficient capabilities of cross-lingual alignment. Nonetheless, synonym pairs residing in bilingual corpus are not exploited and aligned, which is more crucial than sentence interdependence establishment for token-level tasks. In this work, we propose a cross-lingual pre-trained model VECO~2.0 based on contrastive learning with multi-granularity alignments. Specifically, the sequence-to-sequence alignment is induced to maximize the similarity of the parallel pairs and minimize the non-parallel pairs. Then, token-to-token alignment is integrated to bridge the gap between synonymous tokens excavated via the thesaurus dictionary from the other unpaired tokens in a bilingual instance. Experiments show the effectiveness of the proposed strategy for cross-lingual model pre-training on the XTREME benchmark.

翻译：近期研究证明，通过训练统一的Transformer编码器处理多种语言，能够实现跨语言迁移的潜力。现有跨语言预训练工作除了包含掩码语言模型目标外，还采用句子级对比学习或额外插入交叉注意力模块来补足跨语言对齐能力的不足。然而，双语语料中的同义词对尚未被利用和对齐，而这对词级任务而言比句子依赖关系的建立更为关键。本研究提出基于多粒度对齐对比学习的跨语言预训练模型VECO 2.0。具体来说，通过序列到序列对齐机制最大化平行对的相似度并最小化非平行对的相似度；进而整合词到词对齐，通过双语实例中的同义词词典挖掘同义词对，以弥合同义词与未配对词之间的语义差距。实验表明，该策略在XTREME基准测试中对跨语言模型预训练具有显著效果。