Large language models (LLMs) demonstrate remarkable ability to comprehend, reason, and generate following nature language instructions. However, the development of LLMs has been primarily focused on high-resource languages, such as English, thereby limiting their applicability and research in other languages. Consequently, we present PolyLM, a multilingual LLM trained on 640 billion (B) tokens, avaliable in two model sizes: 1.7B and 13B. To enhance its multilingual capabilities, we 1) integrate bilingual data into training data; and 2) adopt a curriculum learning strategy that increases the proportion of non-English data from 30% in the first stage to 60% in the final stage during pre-training. Further, we propose a multilingual self-instruct method which automatically generates 132.7K diverse multilingual instructions for model fine-tuning. To assess the model's performance, we collect several existing multilingual tasks, including multilingual understanding, question answering, generation, and translation. Extensive experiments show that PolyLM surpasses other open-source models such as LLaMA and BLOOM on multilingual tasks while maintaining comparable performance in English. Our models, alone with the instruction data and multilingual benchmark, are available at: \url{https://modelscope.cn/models/damo/nlp_polylm_13b_text_generation}.

翻译：大语言模型展现了理解、推理以及遵循自然语言指令生成的卓越能力。然而，当前大语言模型的研究主要聚焦于英语等高资源语言，限制了其在其他语言中的适用性与研究进展。为此，我们提出PolyLM——一个在6400亿token上训练的多语言大语言模型，提供1.7B和13B两种参数规模。为了增强其多语言能力，我们：1）将双语数据整合至训练数据中；2）采用课程学习策略，在预训练阶段将非英语数据的比例从初始阶段的30%逐步提升至最终阶段的60%。此外，我们提出一种多语言自指令方法，自动生成13.27万条多样化多语言指令用于模型微调。为评估模型性能，我们收集了多项现有多语言任务，包括多语言理解、问答、生成及翻译。大量实验表明，PolyLM在多语言任务上优于LLaMA、BLOOM等开源模型，同时在英语任务中保持相当水平。我们的模型、指令数据及多语言基准已开源至：\url{https://modelscope.cn/models/damo/nlp_polylm_13b_text_generation}。