Multilingual Pretrained Language Models (MPLMs) have shown their strong multilinguality in recent empirical cross-lingual transfer studies. In this paper, we propose the Prompts Augmented by Retrieval Crosslingually (PARC) pipeline to improve the zero-shot performance on low-resource languages (LRLs) by augmenting the context with semantically similar sentences retrieved from a high-resource language (HRL) as prompts. PARC improves the zero-shot performance on three downstream tasks (binary sentiment classification, topic categorization and natural language inference) with multilingual parallel test sets across 10 LRLs covering 6 language families in both unlabeled settings (+5.1%) and labeled settings (+16.3%). PARC-labeled also outperforms the finetuning baseline by 3.7%. We find a significant positive correlation between cross-lingual transfer performance on one side, and the similarity between the high- and low-resource languages as well as the amount of low-resource pretraining data on the other side. A robustness analysis suggests that PARC has the potential to achieve even stronger performance with more powerful MPLMs.

翻译：多语言预训练语言模型在近期跨语言迁移学习的实证研究中展现了其强大的多语言能力。本文提出跨语言检索增强提示管道，通过从高资源语言中检索语义相似的句子作为提示来增强上下文，从而改善低资源语言的零样本性能。PARC在无标签设置和有标签设置下，使用覆盖6个语系共10种低资源语言的多语言平行测试集，在三个下游任务（二分类情感分析、主题分类和自然语言推理）上分别提升了零样本性能5.1%和16.3%。有标签PARC方法还比微调基线高出3.7%。研究发现跨语言迁移性能与高资源语言和低资源语言的相似度以及低资源预训练数据量之间存在显著正相关。鲁棒性分析表明，随着更强大的多语言预训练语言模型的发展，PARC有望实现更强的性能表现。