There are two types of approaches to solving cross-lingual transfer: multilingual pre-training implicitly aligns the hidden representations of different languages, while the translate-test explicitly translates different languages to an intermediate language, such as English. Translate-test has better interpretability compared to multilingual pre-training. However, the translate-test has lower performance than multilingual pre-training(Conneau and Lample, 2019; Conneau et al, 2020) and can't solve word-level tasks because translation rearranges the word order. Therefore, we propose a new Machine-created Universal Language (MUL) as a new intermediate language. MUL consists of a set of discrete symbols as universal vocabulary and NL-MUL translator for translating from multiple natural languages to MUL. MUL unifies common concepts from different languages into the same universal word for better cross-language transfer. And MUL preserves the language-specific words as well as word order, so the model can be easily applied to word-level tasks. Our experiments show that translating into MUL achieves better performance compared to multilingual pre-training, and our analyses show that MUL has good interpretability.

翻译：解决跨语言迁移的方法主要有两类：多语言预训练通过隐式对齐不同语言的隐藏表示，而翻译-测试方法则显式地将不同语言翻译为中间语言（如英语）。相比于多语言预训练，翻译-测试方法具有更好的可解释性。然而，翻译-测试方法的性能低于多语言预训练（Conneau和Lample, 2019；Conneau等, 2020），且由于翻译会改变词序，该方法无法处理词级任务。为此，我们提出一种新的机器创造的世界语（MUL）作为新的中间语言。MUL包含一组离散符号构成的世界语词汇表，以及一个从多种自然语言到MUL的自然语言-MUL翻译器。MUL将不同语言中的共同概念统一为相同的世界语词汇，从而提升跨语言迁移效果；同时保留语言特有词汇及词序，使模型能够轻松应用于词级任务。实验表明，与多语言预训练相比，将文本翻译为MUL可获得更优性能，且分析显示MUL具有良好的可解释性。