Tokenization is a hardcoded compression step which remains in the training pipeline of Large Language Models (LLMs), despite a general trend towards architectures becoming increasingly end-to-end. Prior work has shown promising results at scale in bringing this compression step inside the LLMs' architecture with heuristics to draw token boundaries, and also attempts to learn these token boundaries with straight-through estimates, which treat the problem of drawing discrete token boundaries as a continuous one. We show that these token boundaries can instead be learned using score function estimates, which have tighter theoretical guarantees due to directly optimizing the problem of drawing discrete token boundaries to minimize loss. We observe that techniques from reinforcement learning, such as time discounting, are necessary to reduce the variance of this score function sufficiently to make it practicable. We demonstrate that the resultant method outperforms prior proposed straight-through estimates, both qualitatively and quantitatively at the $100$ million parameter scale.

翻译：分词作为硬编码的压缩步骤，在大型语言模型（LLMs）的训练流程中依然存在，尽管整体架构设计正日益趋向端到端化。先前的研究已表明，通过启发式方法划定词元边界，将这一压缩步骤纳入LLMs架构内部在大规模应用中展现出良好前景；同时也有研究尝试使用直通估计来学习这些词元边界，将离散边界划定问题视为连续问题处理。本文证明，这些词元边界可以通过评分函数估计进行学习——由于直接优化离散边界划定问题以最小化损失，该方法具有更严格的理论保证。我们发现，必须引入强化学习中的技术（如时间折扣）来充分降低评分函数的方差，使其具备实际可行性。实验表明，在参数量为1亿的规模上，所提出的方法在定性与定量评估中均优于先前提出的直通估计方法。