Instruction tuning has shown great promise in improving the performance of large language models. However, research on multilingual instruction tuning has been limited due to the scarcity of high-quality instruction-response datasets across different languages. To bridge this gap, we present Bactrian-X, a comprehensive multilingual parallel dataset of 3.4 million instruction-response pairs across 52 languages. Leveraging this dataset, we train a set of adapters using low-rank adaptation (LoRA), which are lightweight components that seamlessly integrate with large language models. These adapters have a substantially lower parameter count than the base model, making them easily replaceable and usable as plug-ins for different languages or language groups. Extensive experiments in various multilingual evaluation settings demonstrate that models derived from LoRA-based training over Bactrian-X outperform both the vanilla models and existing instruction-tuned models. The code and models are publicly available at https://github.com/mbzuai-nlp/bactrian-x

翻译：指令微调在提升大语言模型性能方面展现了巨大潜力。然而，由于不同语言中高质量指令-响应对数据集的稀缺，多语言指令微调研究仍十分有限。为填补这一空白，我们提出Bactrian-X，一个包含52种语言共340万指令-响应对的综合性多语言并行数据集。利用此数据集，我们通过低秩适应（LoRA）训练出一组适配器——这些轻量级组件可与大语言模型无缝集成。相较于基础模型，这些适配器的参数量显著降低，使其易于替换，并能作为不同语言或语系的可插拔模块使用。在多种多语言评估场景下的广泛实验表明，基于Bactrian-X进行LoRA训练得到的模型在性能上均优于原始模型及现有指令微调模型。代码和模型已于https://github.com/mbzuai-nlp/bactrian-x 公开提供。