The BLOOM model is a large publicly available multilingual language model, but its pretraining was limited to 46 languages. To extend the benefits of BLOOM to other languages without incurring prohibitively large costs, it is desirable to adapt BLOOM to new languages not seen during pretraining. In this work, we apply existing language adaptation strategies to BLOOM and benchmark its zero-shot prompting performance on eight new languages in a resource-constrained setting. We find language adaptation to be effective at improving zero-shot performance in new languages. Surprisingly, we find that adapter-based finetuning is more effective than continued pretraining for large models. In addition, we discover that prompting performance is not significantly affected by language specifics, such as the writing system. It is primarily determined by the size of the language adaptation data. We also add new languages to BLOOMZ, which is a multitask finetuned version of BLOOM capable of following task instructions zero-shot. We find including a new language in the multitask fine-tuning mixture to be the most effective method to teach BLOOMZ a new language. We conclude that with sufficient training data language adaptation can generalize well to diverse languages. Our code is available at https://github.com/bigscience-workshop/multilingual-modeling.

翻译：BLOOM模型是一个大型公开多语言语言模型，但其预训练仅涵盖46种语言。为了在避免高昂成本的前提下将BLOOM的优势扩展到其他语言，有必要使其适应预训练阶段未见过的新语言。在本工作中，我们将现有语言适应策略应用于BLOOM，并在资源受限条件下对其在八种新语言上的零样本提示性能进行基准测试。研究发现，语言适应能有效提升新语言上的零样本性能。令人惊讶的是，在大模型上，基于适配器的微调比继续预训练更为有效。此外，我们观察到提示性能受语言特性（如书写系统）影响不显著，主要取决于语言适应数据的规模。我们还向BLOOMZ（一种经多任务微调、具备零样本指令遵循能力的BLOOM版本）中添加了新语言。研究发现，将新语言纳入多任务微调混合数据集是教会BLOOMZ新语言的最有效方法。我们得出结论：在充足训练数据支持下，语言适应能良好泛化至多样化语言。我们的代码开源于https://github.com/bigscience-workshop/multilingual-modeling。