Cross-lingual transfer of language models trained on high-resource languages like English has been widely studied for many NLP tasks, but focus on conversational tasks has been rather limited. This is partly due to the high cost of obtaining non-English conversational data, which results in limited coverage. In this work, we introduce XSGD for cross-lingual alignment pretraining, a parallel and large-scale multilingual conversation dataset that we created by translating the English-only Schema-Guided Dialogue (SGD) dataset (Rastogi et al., 2020) into 105 other languages. XSGD contains approximately 330k utterances per language. To facilitate aligned cross-lingual representations, we develop an efficient prompt-tuning-based method for learning alignment prompts. We also investigate two different classifiers: NLI-based and vanilla classifiers, and test cross-lingual capability enabled by the aligned prompts. We evaluate our model's cross-lingual generalization capabilities on two conversation tasks: slot-filling and intent classification. Our results demonstrate the strong and efficient modeling ability of NLI-based classifiers and the large cross-lingual transfer improvements achieved by our aligned prompts, particularly in few-shot settings. In addition, we highlight the nice results of our approach compared to LLMs such as text-davinci-003 and ChatGPT in both zero-shot and few-shot settings. While LLMs exhibit impressive performance in English, their cross-lingual capabilities in other languages, particularly low-resource languages, are limited.

翻译：跨语言语言模型迁移（如从英语等资源丰富语言的训练模型迁移）已在多种自然语言处理任务中得到广泛研究，但针对对话任务的研究较为有限。这主要源于获取非英语对话数据的高昂成本导致语种覆盖不足。本文提出用于跨语言对齐预训练的XSGD数据集——通过将仅含英语的模式引导对话（SGD）数据集（Rastogi等人，2020）翻译为105种其他语言，构建了一个大规模平行多语言对话数据集。该数据集每种语言包含约33万条语句。为促进对齐的跨语言表征，我们开发了一种基于提示微调的高效方法以学习对齐提示。此外，我们探究了两种分类器（基于自然语言推理的分类器与经典分类器），并测试了对齐提示所实现的跨语言能力。我们在两个对话任务（槽位填充和意图分类）上评估了模型的跨语言泛化能力。实验结果表明，基于自然语言推理的分类器具有强大高效的建模能力，且对齐提示在少样本场景下实现了显著的跨语言迁移改进。特别地，我们的方法在零样本和少样本设置下均优于text-davinci-003和ChatGPT等大型语言模型。尽管大型语言模型在英语上表现优异，但其在低资源语言等非英语语言上的跨语言能力仍存在明显局限。