Large language models have exhibited significant proficiency in languages endowed with extensive linguistic resources, such as English and Chinese. Nevertheless, their effectiveness notably diminishes when applied to languages characterized by limited linguistic resources, particularly within the Southeast Asian linguistic landscape, such as Indonesian. The scarcity of linguistic resources for these languages presents challenges associated with inadequate training, restricted vocabulary coverage, and challenging evaluation processes. In response to these exigencies, we have introduced CompassLLM, a large multilingual model specifically tailored for Southeast Asian languages, with the primary aim of supporting the developmental requirements of Shopee. Our methodology encompasses several key strategies. To progressively enhance multilingual proficiencies, we implemented a multi-stage pre-training strategy integrated with curriculum learning, gradually intensifying the focus on low-resource languages. Concurrently, to better accommodate low-resource human instructions, we curated and generated a repository of high-quality multilingual human instructions, culminating the CompassLLM-SFT model through supervised instruction fine-tuning. Finally, to reinforce the model's alignment with human preference behaviors, we have embraced the principle of Direct Preference Optimization (DPO) to obtain CompassLLM-DPO model. Preliminary evaluation of the CompassLLM model yields promising results, with our model surpassing benchmark models like Vicuna-7b-v1.5, Sealion, Falcon and SeaLLM, across diverse evaluation tasks, as verified through both automated and human-driven assessments. Notably, our model exhibits its superior performance in South-east Asia languages, such as Indonesian language.

翻译：大型语言模型在拥有丰富语言资源的语言（如英语和中文）中展现出显著的熟练度。然而，当应用于资源有限的低资源语言时（尤其是东南亚语言环境中的印尼语等），其有效性显著降低。这些语言缺乏语言资源，导致训练不充分、词汇覆盖受限以及评估过程困难。为应对这些挑战，我们推出了CompassLLM，这是一个专为东南亚语言定制的大型多语言模型，主要目标是支持Shopee的开发需求。我们的方法包含多项关键策略。为逐步提升多语言能力，我们实施了结合课程学习的多阶段预训练策略，逐渐加强对低资源语言的训练重点。同时，为更好地适配低资源人类指令，我们策划并生成了高质量的多语言人类指令库，通过监督式指令微调形成了CompassLLM-SFT模型。最后，为增强模型与人类偏好行为的一致性，我们采用了直接偏好优化（DPO）原则，获得了CompassLLM-DPO模型。对CompassLLM模型的初步评估取得了令人鼓舞的结果，在多种评估任务中，我们的模型在自动评估和人工评估中均超越了Vicuna-7b-v1.5、Sealion、Falcon和SeaLLM等基准模型。值得注意的是，我们的模型在东南亚语言（如印尼语）中展现出优越性能。