Enhancing the zero-shot performance of instruction-following models requires heavy computation, either by scaling the total number of training datasets or the model size. In this work, we explore how retrieval of soft prompts obtained through prompt tuning can efficiently assist hard prompts in zero-shot task generalization. Specifically, we train soft prompt embeddings for each prompt through prompt tuning, store the samples of the training instances mapped with the prompt embeddings, and retrieve the corresponding prompt embedding of the training instance closest to the query instance during inference. While only adding 0.007% additional parameters, retrieval of soft prompt enhances the performance of T0 on unseen tasks by outperforming it on 10 out of 11 datasets as well as improving the mean accuracy of T0 on BIG-bench benchmark by 2.39% points. Also, we report an interesting finding that retrieving source embeddings trained on similar answer choice formats is more important than those on similar task types.

翻译：提升指令遵循模型的零样本性能通常需要大量计算资源，要么通过扩大训练数据集规模，要么通过增加模型参数量。本研究探索如何通过提示调优获取的软提示检索，高效辅助硬提示实现零样本任务泛化。具体而言，我们为每个提示通过提示调优训练软提示嵌入，存储训练实例与对应提示嵌入的映射样本，在推理阶段检索与查询实例最接近的训练实例所对应的提示嵌入。仅增加0.007%额外参数的情况下，基于软提示检索的方法使T0模型在11个未见任务数据集中的10个上取得更优结果，并在BIG-bench基准测试中将平均准确率提升2.39个百分点。此外，我们报告了一项有趣的发现：相较于任务类型相似的源嵌入，基于相似答案选项格式训练的源嵌入检索更具重要性。