Pre-trained multilingual language models have become an important building block in multilingual natural language processing. In the present paper, we investigate a range of such models to find out how well they transfer discourse-level knowledge across languages. This is done with a systematic evaluation on a broader set of discourse-level tasks than has been previously been assembled. We find that the XLM-RoBERTa family of models consistently show the best performance, by simultaneously being good monolingual models and degrading relatively little in a zero-shot setting. Our results also indicate that model distillation may hurt the ability of cross-lingual transfer of sentence representations, while language dissimilarity at most has a modest effect. We hope that our test suite, covering 5 tasks with a total of 22 languages in 10 distinct families, will serve as a useful evaluation platform for multilingual performance at and beyond the sentence level.

翻译：训练有素的多语文模式已成为多语种自然语言处理中的重要基石。在本文件中,我们调查了各种此类模式,以了解它们在不同语言之间传播讲道知识的情况。这是通过对一系列比以前收集的更广泛的讲道层面任务进行系统评估来完成的。我们发现,XLM-ROBERTA模式的组合一贯表现出最佳表现,同时是良好的单语模式,在零景环境中相对较少的贬低。我们的结果还表明,示范提炼可能会损害跨语言转移服刑表现的能力,而语言差异最多会产生一定的效果。我们希望,我们的测试套件,涵盖10个不同家庭总共22种语言的5项任务,将成为在判决一级和之外多语种表现的有用评价平台。