Prompt tuning approaches, which learn task-specific soft prompts for a downstream task conditioning on frozen pre-trained models, have attracted growing interest due to its parameter efficiency. With large language models and sufficient training data, prompt tuning performs comparably to full-model tuning. However, with limited training samples in few-shot settings, prompt tuning fails to match the performance of full-model fine-tuning. In this work, we focus on improving the few-shot performance of prompt tuning by transferring knowledge from soft prompts of source tasks. Recognizing the good generalization capabilities of ensemble methods in low-data regime, we first experiment and show that a simple ensemble of model predictions based on different source prompts, outperforms existing multi-prompt knowledge transfer approaches such as source prompt fusion in the few-shot setting. Motivated by this observation, we further investigate model ensembles and propose Sample-specific Ensemble of Source Models (SESoM). SESoM learns to adjust the contribution of each source model for each target sample separately when ensembling source model outputs. Through this way, SESoM inherits the superior generalization of model ensemble approaches and simultaneously captures the sample-specific competence of each source prompt. We conduct experiments across a diverse set of eight NLP tasks using models of different scales (T5-{base, large, XL}) and find that SESoM consistently outperforms the existing models of the same as well as larger parametric scale by a large margin.

翻译：提示微调方法通过学习任务特定的软提示来适配冻结的预训练模型，因其参数高效性而受到广泛关注。在大语言模型和充足训练数据的条件下，提示微调的性能可与全模型微调相媲美。然而，在少样本场景下，由于训练样本有限，提示微调无法达到全模型微调的性能水平。本研究聚焦于通过迁移源任务的软提示知识，提升提示微调在少样本场景下的性能。考虑到集成方法在低数据场景下的良好泛化能力，我们首先实验证明：在少样本设置下，基于不同源提示的简单模型预测集成方法，其性能优于现有的多提示知识迁移方法（如源提示融合）。受此发现启发，我们进一步研究模型集成，并提出了样本特定源模型集成方法（SESoM）。SESoM在集成源模型输出时，能够针对每个目标样本分别调整各源模型的贡献权重。通过这种方式，SESoM继承了模型集成方法的优异泛化能力，同时捕获了每个源提示的样本特定能力。我们在涵盖八种不同自然语言处理任务的多样化场景中，使用不同规模的模型（T5-{base, large, XL}）进行了实验，结果表明SESoM在同等乃至更大参数规模的模型中均能持续取得显著优势。