Cross-lingual document representations enable language understanding in multilingual contexts and allow transfer learning from high-resource to low-resource languages at the document level. Recently large pre-trained language models such as BERT, XLM and XLM-RoBERTa have achieved great success when fine-tuned on sentence-level downstream tasks. It is tempting to apply these cross-lingual models to document representation learning. However, there are two challenges: (1) these models impose high costs on long document processing and thus many of them have strict length limit; (2) model fine-tuning requires extra data and computational resources, which is not practical in resource-limited settings. In this work, we address these challenges by proposing unsupervised Language-Agnostic Weighted Document Representations (LAWDR). We study the geometry of pre-trained sentence embeddings and leverage it to derive document representations without fine-tuning. Evaluated on cross-lingual document alignment, LAWDR demonstrates comparable performance to state-of-the-art models on benchmark datasets.

翻译：跨语文文件代表制有助于在多语种背景下理解语言,并允许在文件一级从高资源语言向低资源语言转移学习。最近,如BERT、XLM和XLM-ROBERTA等大型预先培训语言模式在对下游任务进行微调时取得了巨大成功。我们研究这些跨语文模式的几何方法,以便用于文件代表制学习。然而,有两个挑战:(1)这些模式对长文件处理造成高昂费用,因此许多模式有严格的长度限制;(2) 示范微调需要额外的数据和计算资源,而这在资源有限的情况下是行不通的。在这项工作中,我们通过提出未经监督的语文敏感文件代表制(LAWDR)来应对这些挑战。我们研究了预先培训的句的嵌入式,并利用这些模式在不作微调的情况下获取文件代表制。对跨语文文件的校准进行了评价,LADR显示与基准数据集方面最先进的模式具有可比性。