Large-scale multilingual Pretrained Language Models (mPLMs) yield impressive performance on cross-language tasks, yet significant performance disparities exist across different languages within the same mPLM. Previous studies endeavored to narrow these disparities by supervise fine-tuning the mPLMs with multilingual data. However, obtaining labeled multilingual data is time-consuming, and fine-tuning mPLM with limited labeled multilingual data merely encapsulates the knowledge specific to the labeled data. Therefore, we introduce ALSACE to leverage the learned knowledge from the well-performing languages to guide under-performing ones within the same mPLM, eliminating the need for additional labeled multilingual data. Experiments show that ALSACE effectively mitigates language-level performance disparity across various mPLMs while showing the competitive performance on different multilingual NLU tasks, ranging from full resource to limited resource settings. The code for our approach is available at https://github.com/pkunlp-icler/ALSACE.

翻译：大规模多语言预训练语言模型（mPLMs）在跨语言任务中表现优异，但同一mPLM内不同语言间存在显著性能差异。现有研究通过使用多语言数据对mPLMs进行监督微调来缩小这些差异，但获取标注多语言数据耗时费力，且利用有限标注多语言数据微调mPLM仅能封装标注数据特有的知识。为此，我们提出ALSACE方法，通过利用同一mPLM中高资源语言已习得的知识指导低资源语言，无需额外标注多语言数据。实验表明，ALSACE能有效缓解不同mPLMs中语言级性能差异，同时在从全资源到有限资源配置的多语言自然语言理解任务中展现竞争力。本方法代码已公开于https://github.com/pkunlp-icler/ALSACE。