We propose task-adaptive tokenization as a way to adapt the generation pipeline to the specifics of a downstream task and enhance long-form generation in mental health. Inspired by insights from cognitive science, our task-adaptive tokenizer samples variable segmentations from multiple outcomes, with sampling probabilities optimized based on task-specific data. We introduce a strategy for building a specialized vocabulary and introduce a vocabulary merging protocol that allows for the integration of task-specific tokens into the pre-trained model's tokenization step. Through extensive experiments on psychological question-answering tasks in both Chinese and English, we find that our task-adaptive tokenization approach brings a significant improvement in generation performance while using up to 60% fewer tokens. Preliminary experiments point to promising results when using our tokenization approach with very large language models.

翻译：我们提出任务自适应分词方法，通过调整生成流水线适配下游任务特性，提升心理健康领域的长文本生成效果。受认知科学启示，本方法从多组候选分词结果中采样可变分段，基于任务特定数据优化采样概率。我们构建了专用词汇表构建策略，并引入词汇合并协议，使任务相关标记可集成至预训练模型的分词环节。在中英文心理问答任务上的大量实验表明，该任务自适应分词方法在使用最多减少60%标记量的条件下显著提升生成性能。初步实验表明，本分词方法在大规模语言模型上展现出良好应用前景。