Transliterating related languages that use different scripts into a common script shows effectiveness in improving crosslingual transfer in downstream tasks. However, this methodology often makes pretraining a model from scratch unavoidable, as transliteration brings about new subwords not covered in existing multilingual pretrained language models (mPLMs). This is not desired because it takes a lot of computation budget for pretraining. A more promising way is to make full use of available mPLMs. To this end, this paper proposes a simple but effective framework: Transliterate-Merge-Initialize (TransMI), which can create a strong baseline well-suited for data that is transliterated into a common script by exploiting an mPLM and its accompanied tokenizer. TransMI has three stages: (a) transliterate the vocabulary of an mPLM into a common script; (b) merge the new vocabulary with the original vocabulary; and (c) initialize the embeddings of the new subwords. We applied TransMI to three recent strong mPLMs, and our experiments demonstrate that TransMI not only preserves their ability to handle non-transliterated data, but also enables the models to effectively process transliterated data: the results show a consistent improvement of 3% to 34%, varying across different models and tasks. We make our code and models publicly available at \url{https://github.com/cisnlp/TransMI}.

翻译：摘要：将使用不同文字的同源语言音译为共同文字，可有效提升下游任务中的跨语言迁移能力。然而，该方法常需从头预训练模型，因为音译会引入现有语言模型（mPLMs）未覆盖的新子词单元。由于预训练需耗费大量计算资源，此类方案并非理想选择。更优途径在于充分利用现有mPLMs。为此，本文提出简洁高效的框架——Transliterate-Merge-Initialize（TransMI），通过利用mPLM及其配套分词器，为音译至共同文字的数据创建强基线。TransMI包含三个阶段：（a）将mPLM词表音译为共同文字；（b）合并新词表与原词表；（c）初始化新子词的嵌入表示。我们将TransMI应用于三种近期强mPLM，实验表明：该框架不仅保留模型处理非音译数据的能力，更使其能有效处理音译数据——在不同模型和任务中，性能提升幅度稳定在3%至34%之间。相关代码与模型已在\url{https://github.com/cisnlp/TransMI}开源。