We propose task-adaptive tokenization as a way to adapt the generation pipeline to the specifics of a downstream task and enhance long-form generation in mental health. Inspired by insights from cognitive science, our task-adaptive tokenizer samples variable segmentations from multiple outcomes, with sampling probabilities optimized based on task-specific data. We introduce a strategy for building a specialized vocabulary and introduce a vocabulary merging protocol that allows for the integration of task-specific tokens into the pre-trained model's tokenization step. Through extensive experiments on psychological question-answering tasks in both Chinese and English, we find that our task-adaptive tokenization approach brings a significant improvement in generation performance while using up to 60% fewer tokens. Preliminary experiments point to promising results when using our tokenization approach with very large language models.

翻译：我们提出任务自适应分词作为调整生成流程以适应下游任务特性并增强心理健康领域长文本生成的方法。受认知科学洞见启发，我们的任务自适应分词器从多种输出中采样可变分词结果，其采样概率基于任务特定数据优化。我们引入了一种构建专业词汇表的策略，并提出一种词汇合并协议，可将任务特定标记集成至预训练模型的分词步骤中。通过在中英文心理问答任务上的广泛实验，我们发现任务自适应分词方法在生成性能方面带来显著提升，同时使用的标记数量最多可减少60%。初步实验表明，当将我们的分词方法应用于超大规模语言模型时，展现出可观的应用前景。