Recent large language models (LLM) exhibit sub-optimal performance on low-resource languages, as the training data of these models is usually dominated by English and other high-resource languages. Furthermore, it is challenging to train models for low-resource languages, especially from scratch, due to a lack of high quality training data. Adapting pretrained LLMs reduces the need for data in the new language while also providing cross lingual transfer capabilities. However, naively adapting to new languages leads to catastrophic forgetting and poor tokenizer efficiency. In this work, we study how to efficiently adapt any existing pretrained LLM to a new language without running into these issues. In particular, we improve the encoding efficiency of the tokenizer by adding new tokens from the target language and study the data mixing recipe to mitigate forgetting. Our experiments on adapting an English LLM to Hungarian and Thai show that our recipe can reach better performance than open source models on the target language, with minimal regressions on English.

翻译：近期大语言模型（LLM）在低资源语言上表现欠佳，原因是这类模型的训练数据通常以英语及其他高资源语言为主。此外，由于缺乏高质量训练数据，为低资源语言（尤其是从零开始）训练模型极具挑战性。适配预训练LLM既能减少对新语言数据的需求，又能提供跨语言迁移能力。然而，简单粗暴地适配新语言会导致灾难性遗忘和分词器效率低下。本研究旨在探索如何在不引发上述问题的情况下，高效地将任意现有预训练LLM适配至新语言。具体而言，我们通过添加目标语言的新词元来提升分词器编码效率，并研究数据混合策略以缓解遗忘问题。在将英语LLM适配至匈牙利语和泰语的实验中，我们的方案在目标语言上取得了优于开源模型的表现，同时英语性能下降幅度极小。