Prior work on language modeling showed conflicting findings about whether morphologically aligned approaches to tokenization improve performance, particularly for languages with complex morphology. To investigate this, we select a typologically diverse set of languages: Telugu (agglutinative), Hindi (primarily fusional with some agglutination), and English (fusional). We conduct a comprehensive evaluation of language models -- starting from tokenizer training and extending through the finetuning and downstream task evaluation. To account for the consistent performance differences observed across tokenizer variants, we focus on two key factors: morphological alignment and tokenization quality. To assess morphological alignment of tokenizers in Telugu, we create a dataset containing gold morpheme segmentations of 600 derivational and 7000 inflectional word forms. Our experiments reveal that better morphological alignment correlates positively -- though moderately -- with performance in syntax-based tasks such as Parts-of-Speech tagging, Named Entity Recognition and Dependency Parsing. However, we also find that the tokenizer algorithm (Byte-pair Encoding vs. Unigram) plays a more significant role in influencing downstream performance than morphological alignment alone. Naive Unigram tokenizers outperform others across most settings, though hybrid tokenizers that incorporate morphological segmentation significantly improve performance within the BPE framework. In contrast, intrinsic metrics like Corpus Token Count (CTC) and R\'enyi entropy showed no correlation with downstream performance.

翻译：先前关于语言建模的研究在形态对齐的词元化方法是否能提升性能——尤其是对于形态复杂的语言——方面存在相互矛盾的发现。为探究此问题，我们选取了一组类型学上多样化的语言：泰卢固语（黏着语）、印地语（主要为屈折语，兼具部分黏着特征）和英语（屈折语）。我们对语言模型进行了全面评估——从词元化器训练开始，延伸至微调及下游任务评估。为解释在不同词元化器变体中观察到的持续性能差异，我们聚焦于两个关键因素：形态对齐与词元化质量。为评估泰卢固语词元化器的形态对齐程度，我们构建了一个包含600个派生词形与7000个屈折词形的黄金语素切分数据集。实验表明，更好的形态对齐与基于句法的任务（如词性标注、命名实体识别和依存句法分析）性能呈正相关——尽管相关性中等。然而，我们也发现，词元化算法（字节对编码与Unigram）对下游性能的影响比单纯的形态对齐更为显著。朴素的Unigram词元化器在大多数设定下表现优于其他方法，尽管融合了形态切分的混合词元化器在BPE框架内显著提升了性能。相比之下，语料词元计数（CTC）与Rényi熵等内在指标与下游性能未显示出相关性。