Recent large language models (LLM) exhibit sub-optimal performance on low-resource languages, as the training data of these models is usually dominated by English and other high-resource languages. Furthermore, it is challenging to train models for low-resource languages, especially from scratch, due to a lack of high quality training data. Adapting pretrained LLMs reduces the need for data in the new language while also providing cross lingual transfer capabilities. However, naively adapting to new languages leads to catastrophic forgetting and poor tokenizer efficiency. In this work, we study how to efficiently adapt any existing pretrained LLM to a new language without running into these issues. In particular, we improve the encoding efficiency of the tokenizer by adding new tokens from the target language and study the data mixing recipe to mitigate forgetting. Our experiments on adapting an English LLM to Hungarian and Thai show that our recipe can reach better performance than open source models on the target language, with minimal regressions on English.

翻译：近年来大型语言模型（LLM）在低资源语言上表现欠佳，因其训练数据通常以英语和其他高资源语言为主。此外，由于缺乏高质量训练数据，为低资源语言训练模型（尤其是从零开始训练）极具挑战性。适配预训练LLM既能减少对新语言数据的需求，又能提供跨语言迁移能力。然而，简单适配新语言会导致灾难性遗忘和分词器效率低下。本研究探索如何高效适配任意现有预训练LLM至新语言，同时避免上述问题。具体而言，我们通过添加目标语言的新 tokens 提升分词器编码效率，并研究数据混合策略以缓解遗忘。通过将英语LLM适配至匈牙利语和泰语的实验表明，我们的方法在目标语言上能达到优于开源模型的性能，同时在英语上仅产生最小程度的性能回退。