Current Large Language Models (LLMs) mostly use BPE (Byte Pair Encoding) based tokenizers, which are very effective for simple structured Latin scripts such as English. However, standard BPE tokenizers struggle to process complex Abugida scripts due to their structural complexity. The problem is that these tokenizers break complex conjuncts, which are multi-codepoint grapheme clusters, into meaningless sub-character units. This degrades the LLM's reasoning efficiency by forcing it to learn basic orthographic structures at inference time and raises inference costs, resulting in a significant "Token Tax" for the Global South. We propose a new three-layer architecture, the WWHO (Where-What-How Often), and an algorithm named SGPE (Syllable-aware Grapheme Pair Encoding) that separates the linguistic rules of the script from the statistical compression process while enabling seamless multilingual tokenization. Using Sinhala and Devanagari (Hindi/Sanskrit) as highly complex Abugida scripts, we trained WWHO on a cleaned 30-million-sentence dataset and evaluated on a 1,499,950-sentence test set. For Sinhala, SGPE achieves a Token to Word Ratio (TWR) of 1.274 with 4.83 characters per token, representing a 61.7 percent reduction in tokens compared to OpenAI's o200k base. For Hindi, it achieves a TWR of 1.181 (27.0 percent reduction vs o200k). On the mixed-script (Sinhala, Devanagari, and English) dataset, SGPE achieves an overall TWR of 1.240, representing token reductions of 36.7 percent, 39.6 percent, and 60.2 percent relative to o200k base, Llama 4 Scout, and DeepSeek V3, respectively. This effectively extends the usable context window by up to 4.38 times for these Abugida languages while ensuring a Linguistic Zero-Breakage Guarantee, which ensures that no valid syllable is ever split across multiple tokens.

翻译：当前大型语言模型（LLMs）主要采用基于BPE（字节对编码）的分词器，这类分词器对英语等结构简单的拉丁语系文本非常有效。然而，标准BPE分词器因复杂元音附标文字的结构复杂性而难以处理这类脚本。问题在于，这些分词器会将由多码位字素簇构成的复杂连字拆解为无意义的子字符单元。这迫使LLM在推理时必须学习基础正字法结构，不仅降低了推理效率，还增加了推理成本，导致全球南方国家需承担高昂的“分词税”。我们提出一种新型三层架构WWO（Where-What-How Often）及名为SGPE（音节感知字素对编码）的算法，该算法在支持无缝多语言分词的同时，将脚本的语言规则与统计压缩过程分离。以僧伽罗文和天城文（印地语/梵语）作为高度复杂的元音附标文字为例，我们在经过清洗的3000万句数据集上训练WWO，并在包含1，499,950句的测试集上进行评估。对于僧伽罗文，SGPE实现了1.274的词元比（TWR），每个词元对应4.83个字符，相较OpenAI的o200k基准词元数减少61.7%。对于印地语，其TWR为1.181（较o200k减少27.0%）。在混合文本（僧伽罗文、天城文和英语）数据集上，SGPE的总体TWR为1.240，相较于o200k基准、Llama 4 Scout和DeepSeek V3分别实现36.7%、39.6%和60.2%的词元缩减。这使得针对这些元音附标语言的有效上下文窗口最多可扩展4.38倍，同时确保语言零断裂保障——即任何有效音节永远不会被分割到多个词元中。