Instruction tuning has remarkably advanced large language models (LLMs) in understanding and responding to diverse human instructions. Despite the success in high-resource languages, its application in lower-resource ones faces challenges due to the imbalanced foundational abilities of LLMs across different languages, stemming from the uneven language distribution in their pre-training data. To tackle this issue, we propose pivot language guided generation (PLUG), an approach that utilizes a high-resource language, primarily English, as the pivot to enhance instruction tuning in lower-resource languages. It trains the model to first process instructions in the pivot language, and then produce responses in the target language. To evaluate our approach, we introduce a benchmark, X-AlpacaEval, of instructions in 4 languages (Chinese, Korean, Italian, and Spanish), each annotated by professional translators. Our approach demonstrates a significant improvement in the instruction-following abilities of LLMs by 29% on average, compared to directly responding in the target language alone. Further experiments validate the versatility of our approach by employing alternative pivot languages beyond English to assist languages where LLMs exhibit lower proficiency. Our code and data are available at https://github.com/ytyz1307zzh/PLUG.

翻译：摘要：指令微调显著提升了大型语言模型（LLMs）理解和响应多样化人类指令的能力。尽管该技术在高资源语言中取得了成功，但由于预训练数据中语言分布不均导致LLMs在不同语言上的基础能力失衡，其在低资源语言中的应用面临挑战。为解决该问题，我们提出枢轴语言引导生成（PLUG）方法，该方法利用高资源语言（主要为英语）作为枢轴来增强低资源语言的指令微调。该方法训练模型先处理枢轴语言指令，再生成目标语言响应。为评估我们的方法，我们构建了包含4种语言（中文、韩语、意大利语和西班牙语）指令的基准测试集X-AlpacaEval，所有指令均由专业翻译人员标注。实验表明，与直接使用目标语言响应相比，本方法平均将LLMs的指令遵循能力提升了29%。进一步实验通过使用英语以外的替代枢轴语言来辅助LLMs熟练度较低的语言，验证了方法的通用性。代码与数据已开源至https://github.com/ytyz1307zzh/PLUG。