Prompt tuning is a parameter-efficient method, which learns soft prompts and conditions frozen language models to perform specific downstream tasks. Though effective, prompt tuning under few-shot settings on the one hand heavily relies on a good initialization of soft prompts. On the other hand, it can easily overfit to few-shot training samples, thereby undermining generalizability. Existing works leverage pre-training or supervised meta-learning to initialize soft prompts but they fail to data-efficiently generalize to unseen downstream tasks. To address the above problems, this paper proposes a novel Self-sUpervised meta-Prompt learning framework with MEta-gradient Regularization for few-shot generalization (SUPMER). SUPMER leverages self-supervised meta-learning with a diverse set of well-designed meta-training tasks to learn a universal prompt initialization for efficient adaptation using only unlabeled data. Additionally, it jointly meta-learns a gradient regularization function to transform raw gradients into a domain-generalizable direction, thus alleviating the problem of overfitting. Extensive experiments show that SUPMER achieves better performance for different few-shot downstream tasks, and also exhibits a stronger domain generalization ability. The code for SUPMER will be available at https://github.com/beepkh/SUPMER.

翻译：提示调优是一种参数高效的方法，它通过学习软提示并冻结语言模型来执行特定的下游任务。尽管有效，但少样本设置下的提示调优一方面严重依赖于软提示的良好初始化，另一方面容易对少样本训练样本过拟合，从而损害泛化能力。现有工作利用预训练或监督元学习来初始化软提示，但未能以数据高效的方式泛化到未见过的下游任务。为解决上述问题，本文提出了一种新颖的自监督元提示学习框架（SUPMER），结合元梯度正则化用于少样本泛化。SUPMER利用自监督元学习，通过一组精心设计的元训练任务，仅使用无标签数据学习通用的提示初始化，以实现高效适配。此外，它联合元学习了一个梯度正则化函数，将原始梯度转化为领域可泛化的方向，从而缓解过拟合问题。大量实验表明，SUPMER在多种少样本下游任务中取得了更优性能，并展现出更强的领域泛化能力。SUPMER的代码将发布于https://github.com/beepkh/SUPMER。