A key technology for the development of large language models (LLMs) involves instruction tuning that helps align the models' responses with human expectations to realize impressive learning abilities. Two major approaches for instruction tuning characterize supervised fine-tuning (SFT) and reinforcement learning from human feedback (RLHF), which are currently applied to produce the best commercial LLMs (e.g., ChatGPT). To improve the accessibility of LLMs for research and development efforts, various instruction-tuned open-source LLMs have also been introduced recently, e.g., Alpaca, Vicuna, to name a few. However, existing open-source LLMs have only been instruction-tuned for English and a few popular languages, thus hindering their impacts and accessibility to many other languages in the world. Among a few very recent work to explore instruction tuning for LLMs in multiple languages, SFT has been used as the only approach to instruction-tune LLMs for multiple languages. This has left a significant gap for fine-tuned LLMs based on RLHF in diverse languages and raised important questions on how RLHF can boost the performance of multilingual instruction tuning. To overcome this issue, we present Okapi, the first system with instruction-tuned LLMs based on RLHF for multiple languages. Okapi introduces instruction and response-ranked data in 26 diverse languages to facilitate the experiments and development of future multilingual LLM research. We also present benchmark datasets to enable the evaluation of generative LLMs in multiple languages. Our experiments demonstrate the advantages of RLHF for multilingual instruction over SFT for different base models and datasets. Our framework and resources are released at \url{https://github.com/nlp-uoregon/Okapi}.

翻译：摘要：大语言模型发展的关键技术之一是指令调优，这有助于使模型响应与人类期望对齐，从而实现令人印象深刻的学习能力。目前，两种主要的指令调优方法——监督式微调（SFT）和基于人类反馈的强化学习（RLHF）——被用于打造最优秀的商业大语言模型（如ChatGPT）。为提升大语言模型在研发工作中的可及性，近期也涌现了多种经过指令调优的开源大语言模型，例如Alpaca、Vicuna等。然而，现有开源大语言模型仅针对英语及少数流行语言进行了指令调优，这限制了其对世界许多其他语言的影响力和可及性。在近期少数探索多语言指令调优的研究中，监督式微调是唯一被用于多语言指令调优的方法。这导致基于RLHF的微调模型在多语言领域存在显著空白，并引发重要问题：RLHF如何提升多语言指令调优的性能？为解决这一问题，我们提出Okapi——首个基于RLHF的多语言指令调优大语言模型系统。Okapi为26种不同语言提供了指令及响应排序数据，以促进未来多语言大语言模型研究的实验与开发。我们还提供了基准数据集，用于评估多语言生成式大语言模型。实验表明，在不同基础模型和数据集上，基于RLHF的多语言指令调优相较于SFT具有明显优势。我们的框架与资源已发布在\url{https://github.com/nlp-uoregon/Okapi}。