This study addresses the challenge of extending Large Language Models (LLMs) to non-English languages using non-Roman scripts. We propose an approach that utilizes the romanized form of text as an interface for LLMs, hypothesizing that its frequent informal use and shared tokens with English enhance cross-lingual alignment. Our approach involves the continual pretraining of an English LLM like Llama 2 on romanized text of non-English, non-Roman script languages, followed by instruction tuning on romanized data. The results indicate that romanized text not only reduces token fertility by 2x-4x but also matches or outperforms native script representation across various NLU, NLG, and MT tasks. Moreover, the embeddings computed on romanized text exhibit closer alignment with their English translations than those from the native script. Our approach presents a promising direction for leveraging the power of English LLMs in languages traditionally underrepresented in NLP.

翻译：本研究旨在解决将大型语言模型（LLMs）扩展到使用非拉丁字母的非英语语言时所面临的挑战。我们提出一种利用文本的拉丁化形式作为LLMs接口的方法，假设其非正式使用的普遍性以及与英语共享的令牌能增强跨语言对齐。该方法对英语LLM（如Llama 2）在非英语、非拉丁字母语言的拉丁化文本上持续预训练，随后对拉丁化数据进行指令微调。结果表明，拉丁化文本不仅将令牌生成率降低2至4倍，而且在各类自然语言理解（NLU）、自然语言生成（NLG）及机器翻译（MT）任务中与原生文字表示相当或更优。此外，基于拉丁化文本计算的嵌入向量与对应的英语翻译比原生文字表示更紧密对齐。本方法为利用英语LLM在传统自然语言处理中缺乏代表性的语言上的能力提供了有前景的方向。