We propose task-adaptive tokenization as a way to adapt the generation pipeline to the specifics of a downstream task and enhance long-form generation in mental health. Inspired by insights from cognitive science, our task-adaptive tokenizer samples variable segmentations from multiple outcomes, with sampling probabilities optimized based on task-specific data. We introduce a strategy for building a specialized vocabulary and introduce a vocabulary merging protocol that allows for the integration of task-specific tokens into the pre-trained model's tokenization step. Through extensive experiments on psychological question-answering tasks in both Chinese and English, we find that our task-adaptive tokenization approach brings a significant improvement in generation performance while using up to 60% fewer tokens. Preliminary experiments point to promising results when using our tokenization approach with very large language models.

翻译：我们提出任务自适应分词作为一种方法，通过适应下游任务的具体需求来优化生成流程，从而增强心理健康领域的长文本生成。受认知科学启示，我们的任务自适应分词器从多种分割结果中采样可变分段，并基于任务特定数据优化采样概率。我们引入了一种构建专业词汇表的策略，并提出一种词汇合并协议，允许将任务特定标记集成到预训练模型的分词步骤中。通过在中文和英文的心理问答任务上进行大量实验，我们发现，我们的任务自适应分词方法在生成性能上带来显著提升，同时使用的标记数量减少高达60%。初步实验表明，将我们的分词方法应用于超大型语言模型时，结果十分可观。