Traditionally, NLP performance improvement has been focused on improving models and increasing the number of model parameters. NLP vocabulary construction has remained focused on maximizing the number of words represented through subword regularization. We present a novel tokenizer that uses semantics to drive vocabulary construction. The tokenizer includes a trainer that uses stemming to enhance subword formation. Further optimizations and adaptations are implemented to minimize the number of words that cannot be encoded. The encoder is updated to integrate with the trainer. The tokenizer is implemented as a drop-in replacement for the SentencePiece tokenizer. The new tokenizer more than doubles the number of wordforms represented in the vocabulary. The enhanced vocabulary significantly improves NLP model convergence, and improves quality of word and sentence embeddings. Our experimental results show top performance on two Glue tasks using BERT-base, improving on models more than 50X in size.

翻译：传统上，自然语言处理的性能提升主要聚焦于模型改进和参数数量的增加，而词汇构建仍侧重于通过子词正则化技术最大化所表示单词的数量。本文提出了一种新型分词器，利用语义驱动词汇构建过程。该分词器包含一个训练器，通过词干提取增强子词构成，并实施了进一步的优化与适配策略以最小化无法编码的单词数量。同时，编码器经过更新以与训练器集成。该分词器被设计为SentencePiece分词器的即插即用替代方案。实验表明，新分词器使词汇表中表示的词形数量翻倍以上，显著提升了NLP模型的收敛速度，并改善了词嵌入与句子嵌入的质量。基于BERT-base模型，我们在两项GLUE任务中取得了最优性能，超越了体量超过50倍的大模型。