Existing large language models show disparate capability across different languages, due to the imbalance in the training data. Their performances on English tasks are often stronger than on tasks of other languages. In this paper, we empower pre-trained LLMs on non-English languages by building semantic alignment across languages. We start from targeting individual languages by performing cross-lingual instruction-tuning (CoIT) on LLaMA, i.e. tuning it with translation task data and cross-lingual general task data to obtain cross-lingual models (x-LLaMAs), and formulate underlying scaling laws to investigate the advantages of using scalable translation data. Then we perform multilingual instruction-tuning (MuIT) with mixed resources to build multilingual m-LLaMA. We also illustrate how we leverage the scaling laws to optimize data allocation in a resource-constrained setting. Experiment results on cross-lingual benchmarks XQUAD and MLQA show that x-LLaMAs surpass the English instruction-tuned counterpart (Alpaca) by an average of 27.83% across six non-English languages. Evaluation results on translation dataset Flores-101 show that x-LLaMAs outperform previous LLaMA-based models by an average of 18.89%. Encouragingly, m-LLaMA achieves comparable performance to x-LLaMAs on individual languages and demonstrates the ability to follow multilingual instructions. Further analysis on response content and representation space reveals the alignment of the multilingual semantic space within the middle layers of m-LLaMA.

翻译：现有大语言模型在多种语言上表现出能力差异，这是由于训练数据不平衡所致。它们在英语任务上的表现通常优于其他语言的任务。本文通过构建跨语言语义对齐，增强预训练大语言模型在非英语语言上的能力。我们首先针对特定语言，对LLaMA进行跨语言指令微调（CoIT），即使用翻译任务数据和跨语言通用任务数据进行微调，以获得跨语言模型（x-LLaMAs），并制定底层缩放定律以研究使用可扩展翻译数据的优势。接着，我们采用混合资源进行多语言指令微调（MuIT），构建多语言m-LLaMA。我们还阐述了如何利用缩放定律在资源受限环境下优化数据分配。在跨语言基准测试XQUAD和MLQA上的实验结果表明，x-LLaMAs在六种非英语语言上平均超过英语指令微调模型（Alpaca）27.83%。在翻译数据集Flores-101上的评估结果显示，x-LLaMAs平均超过先前基于LLaMA的模型18.89%。令人鼓舞的是，m-LLaMA在特定语言上达到了与x-LLaMAs相当的性能，并展现出遵循多语言指令的能力。对响应内容和表征空间的进一步分析揭示了m-LLaMA中层内多语言语义空间的对齐。