Large language models (LLMs) often encode word-form variation (e.g., walk vs. walked) as linear directions in the embedding space. However, standard tokenization algorithms treat such variants as distinct words with different vocabulary entries, quickly filling the size-capped token vocabulary with surface-form variation (e.g., walk, walking, Walk) at the expense of diversity and multilingual coverage. We show that many of these variations can be captured by transformation vectors: additive offsets that yield the appropriate word representation when applied to a base form embedding, in both the input and output spaces. Building on this, we propose a compact reshaping of the vocabulary: instead of assigning unique tokens to each surface form, we compose them from shared base form and transformation vectors (e.g., walked is walk+past tense). Our approach is lightweight, keeping the pretrained backbone frozen and only training small adaptation modules. We apply it across five languages and multiple LLMs in both pretraining and post-hoc adaptation, freeing 10-40% of vocabulary slots to be reallocated where tokenization is inefficient. Importantly, we do so while also expanding vocabulary coverage to out-of-vocabulary words, and with minimal impact on downstream performance. Our findings motivate a rethinking of vocabulary design, towards a representation that better matches the underlying structure of language and the practical needs of multilingual coverage.

翻译：大语言模型（LLMs）常将词形变化（如walk与walked）编码为嵌入空间中的线性方向。然而，标准分词算法将此类变体视为具有不同词汇条目的独立词语，导致受容量限制的符号词汇表迅速被表层词形变化（如walk、walking、Walk）填充，牺牲了多样性与多语言覆盖能力。我们表明，这些变体中的大部分可通过变换向量（transformation vectors）捕获：这类加法偏移量作用于基础词形嵌入时，可在输入和输出空间中生成适当的词语表示。基于此，我们提出了一种紧凑的词汇重组方案：不再为每个表层词形分配独立词元，而是通过共享的基础词形和变换向量组合构建它们（如walked由walk+过去时态构成）。我们的方法轻量化，保持预训练骨干网络冻结，仅需训练小型适配模块。我们在五种语言及多个大语言模型上进行了预训练和后置适配实验，释放了10-40%的词汇槽位，可重新分配至分词效率低下的区域。更重要的是，在此过程中我们同时扩展了词汇覆盖范围至未登录词，且对下游任务性能影响极小。我们的研究结果促使重新思考词汇设计，朝向更贴合语言底层结构与多语言覆盖实际需求的表示方式。