Large language models (LLMs) have shown remarkable capabilities in many languages beyond English. Yet, LLMs require more inference steps when generating non-English text due to their reliance on English-centric tokenizers and vocabulary, resulting in higher usage costs to non-English speakers. Vocabulary expansion with target language tokens is a widely used cross-lingual vocabulary adaptation approach to remedy this issue. Despite its effectiveness in inference speedup, previous work on vocabulary expansion has focused on high-resource settings assuming access to a substantial amount of target language data to effectively initialize the embeddings of the new tokens and adapt the LLM to the target language. However, vocabulary expansion in low-resource settings has yet to be explored. In this article, we investigate vocabulary expansion in low-resource settings by considering embedding initialization methods and continual pre-training strategies. Through extensive experiments across typologically diverse languages, tasks and models, we establish a set of strategies to perform vocabulary expansion for faster inference, while striving to maintain competitive downstream performance to baselines. This is achieved with only 30K sentences ($\sim$0.01GB text data) from the target language.

翻译：大语言模型（LLMs）在英语之外的多种语言中已展现出卓越的能力。然而，由于依赖以英语为中心的分词器和词汇表，LLMs在生成非英语文本时需要更多的推理步骤，导致非英语用户的使用成本更高。通过添加目标语言词元进行词汇扩展是一种广泛使用的跨语言词汇适应方法，旨在解决此问题。尽管该方法在加速推理方面有效，先前关于词汇扩展的研究主要集中于高资源场景，假设可获得大量目标语言数据以有效初始化新词元的嵌入表示，并使LLM适应目标语言。然而，低资源场景下的词汇扩展尚未得到充分探索。本文通过研究嵌入初始化方法和持续预训练策略，探讨低资源环境下的词汇扩展。通过对类型多样的语言、任务和模型进行大量实验，我们建立了一套策略，旨在实现更快的推理速度，同时努力保持与基线模型相当的下游性能。这一切仅需使用目标语言的3万条句子（约0.01GB文本数据）即可达成。