Byte Pair Encoding (BPE) serves as a foundation method for text tokenization in the Natural Language Processing (NLP) field. Despite its wide adoption, the original BPE algorithm harbors an inherent flaw: it inadvertently introduces a frequency imbalance for tokens in the text corpus. Since BPE iteratively merges the most frequent token pair in the text corpus to generate a new token and keeps all generated tokens in the vocabulary, it unavoidably holds tokens that primarily act as components of a longer token and appear infrequently on their own. We term such tokens as Scaffold Tokens. Due to their infrequent occurrences in the text corpus, Scaffold Tokens pose a learning imbalance issue. To address that issue, we propose Scaffold-BPE, which incorporates a dynamic scaffold token removal mechanism by parameter-free, computation-light, and easy-to-implement modifications to the original BPE method. This novel approach ensures the exclusion of low-frequency Scaffold Tokens from the token representations for given texts, thereby mitigating the issue of frequency imbalance and facilitating model training. On extensive experiments across language modeling and even machine translation, Scaffold-BPE consistently outperforms the original BPE, well demonstrating its effectiveness.

翻译：字节对编码（BPE）是自然语言处理领域文本令牌化的基础方法。尽管其被广泛采用，原始BPE算法存在一个固有缺陷：它无意中引入了文本语料库中令牌的频率不平衡。由于BPE迭代地合并文本语料库中出现频率最高的令牌对以生成新令牌，并将所有生成的令牌保留在词汇表中，因此不可避免地会包含那些主要作为较长令牌的组成部分、且单独出现频率较低的令牌。我们将此类令牌称为支架令牌。由于它们在文本语料库中出现频率较低，支架令牌会导致学习不平衡问题。为解决该问题，我们提出Scaffold-BPE，该方法通过对原始BPE进行无需参数、计算量小且易于实现的修改，引入了一种动态支架令牌移除机制。这种新颖方法确保了低频率支架令牌被排除在给定文本的令牌表示之外，从而缓解了频率不平衡问题并促进了模型训练。在语言建模乃至机器翻译的大量实验中，Scaffold-BPE始终优于原始BPE，充分证明了其有效性。