Tokenisation is an integral part of the current NLP pipeline. Current tokenisation algorithms such as BPE and Unigram are greedy algorithms -- they make locally optimal decisions without considering the resulting vocabulary as a whole. We instead formulate tokeniser construction as a linear program and solve it using convex optimisation tools, yielding a new algorithm we call ConvexTok. We find ConvexTok consistently improves intrinsic tokenisation metrics and the bits-per-byte (BpB) achieved by language models; it also improves downstream task performance, but less consistently. Furthermore, ConvexTok allows the user to certify how far their tokeniser is from optimal, with respect to a certain objective, via a lower bound, and we empirically find it to be within 1\% of optimal at common vocabulary sizes.

翻译：Tokenisation是当前NLP流程中的核心组成部分。现有的BPE、Unigram等tokenisation算法均为贪心算法——它们仅基于局部最优决策，而未从整体上考虑最终词汇表。为此，我们将tokeniser构建问题形式化为线性规划，并利用凸优化工具进行求解，由此提出了一种新算法ConvexTok。实验表明，ConvexTok在内在tokenisation指标及语言模型实现的bits-per-byte（BpB）上均有持续改进；在下游任务性能方面同样有所提升，但一致性较弱。此外，ConvexTok允许用户通过下界证明其tokeniser与特定目标函数下最优解的差距，经验性结果表明，在常见词汇表大小下，该差距始终保持在1%以内。