Subword tokenization critically affects Natural Language Processing (NLP) performance, yet its behavior in morphologically rich and low-resource language families remains under-explored. This study systematically compares three subword paradigms -- Byte Pair Encoding (BPE), Overlap BPE (OBPE), and Unigram Language Model -- across six Uralic languages with varying resource availability and typological diversity. Using part-of-speech (POS) tagging as a controlled downstream task, we show that OBPE consistently achieves stronger morphological alignment and higher tagging accuracy than conventional methods, particularly within the Latin-script group. These gains arise from reduced fragmentation in open-class categories and a better balance across the frequency spectrum. Transfer efficacy further depends on the downstream tagging architecture, interacting with both training volume and genealogical proximity. Taken together, these findings highlight that morphology-sensitive tokenization is not merely a preprocessing choice but a decisive factor in enabling effective cross-lingual transfer for agglutinative, low-resource languages.
翻译:子词分词对自然语言处理(NLP)性能具有关键影响,然而其在形态丰富且资源匮乏的语言家族中的行为仍未得到充分探索。本研究系统比较了三种子词范式——字节对编码(BPE)、重叠字节对编码(OBPE)和一元语言模型——在六种资源可用性和类型多样性各异的乌拉尔语系语言上的表现。通过将词性(POS)标注作为受控的下游任务,我们证明OBPE在形态对齐度和标注准确率上始终优于传统方法,尤其是在拉丁文字组语言中。这些优势源于开放词类范畴中碎片化的减少以及跨频率谱的更好平衡。迁移效果还进一步依赖于下游标注架构,并与训练数据量和谱系亲缘关系产生交互作用。综上所述,这些发现表明,形态敏感的分词不仅是一种预处理选择,更是实现黏着语、低资源语言有效跨语言迁移的决定性因素。