We propose task-adaptive tokenization as a way to adapt the generation pipeline to the specifics of a downstream task and enhance long-form generation in mental health. Inspired by insights from cognitive science, our task-adaptive tokenizer samples variable segmentations from multiple outcomes, with sampling probabilities optimized based on task-specific data. We introduce a strategy for building a specialized vocabulary and introduce a vocabulary merging protocol that allows for the integration of task-specific tokens into the pre-trained model's tokenization step. Through extensive experiments on psychological question-answering tasks in both Chinese and English, we find that our task-adaptive tokenization approach brings a significant improvement in generation performance while using up to 60% fewer tokens. Preliminary experiments point to promising results when using our tokenization approach with very large language models.

翻译：我们提出任务适应性分词方法，旨在根据下游任务特性调整生成流程，并提升心理健康领域的长文本生成效果。受认知科学启发的洞察，我们的任务适应性分词器从多重分词结果中采样可变片段，采样概率基于任务特定数据优化。我们引入一种构建专用词汇表的策略，并提出词汇合并协议，使预训练模型的分词步骤能够集成任务特定标记。在中英文心理问答任务上的大量实验表明，我们的任务适应性分词方法在减少高达60%标记使用量的同时，显著提升了生成性能。初步实验结果显示，该方法与超大规模语言模型结合时展现出令人期待的效果。