Prompt tuning, in which a base pretrained model is adapted to each task via conditioning on learned prompt vectors, has emerged as a promising approach for efficiently adapting large language models to multiple downstream tasks. However, existing methods typically learn soft prompt vectors from scratch, and it has not been clear how to exploit the rich cross-task knowledge with prompt vectors in a multitask learning setting. We propose multitask prompt tuning (MPT), which first learns a single transferable prompt by distilling knowledge from multiple task-specific source prompts. We then learn multiplicative low rank updates to this shared prompt to efficiently adapt it to each downstream target task. Extensive experiments on 23 NLP datasets demonstrate that our proposed approach outperforms the state-of-the-art methods, including the full finetuning baseline in some cases, despite only tuning 0.035% as many task-specific parameters.

翻译：提示调优作为一种新兴方法，通过基于学习到的提示向量对预训练基础模型进行任务适配，已展现出高效将大语言模型适配至多个下游任务的潜力。然而，现有方法通常从零开始学习软提示向量，尚未明确如何在多任务学习场景中利用提示向量蕴含的丰富跨任务知识。我们提出了多任务提示调优（MPT），该方法首先通过从多个任务特定源提示中蒸馏知识来学习一个可迁移的共享提示，随后通过学习该共享提示的乘法低秩更新，实现对其高效适配各下游目标任务。在23个NLP数据集上的广泛实验表明，尽管仅调整0.035%的任务特定参数，所提方法仍能超越包括完整微调基线在内的最先进方法。