Prompt tuning for pre-trained masked language models (MLM) has shown promising performance in natural language processing tasks with few labeled examples. It tunes a prompt for the downstream task, and a verbalizer is used to bridge the predicted token and label prediction. Due to the limited training data, prompt initialization is crucial for prompt tuning. Recently, MetaPrompting (Hou et al., 2022) uses meta-learning to learn a shared initialization for all task-specific prompts. However, a single initialization is insufficient to obtain good prompts for all tasks and samples when the tasks are complex. Moreover, MetaPrompting requires tuning the whole MLM, causing a heavy burden on computation and memory as the MLM is usually large. To address these issues, we use a prompt pool to extract more task knowledge and construct instance-dependent prompts via attention. We further propose a novel soft verbalizer (RepVerb) which constructs label embedding from feature embeddings directly. Combining meta-learning the prompt pool and RepVerb, we propose MetaPrompter for effective structured prompting. MetaPrompter is parameter-efficient as only the pool is required to be tuned. Experimental results demonstrate that MetaPrompter performs better than the recent state-of-the-arts and RepVerb outperforms existing soft verbalizers.

翻译：针对预训练掩码语言模型（MLM）的提示调优，在少样本标注的自然语言处理任务中展现出良好性能。该方法通过为下游任务调优提示，并借助言语器桥接预测词元与标签预测。由于训练数据有限，提示初始化对提示调优至关重要。近期提出的MetaPrompting（Hou等人，2022）采用元学习为所有任务特定提示学习共享初始化。然而当任务复杂度较高时，单一初始化难以对所有任务和样本生成优质提示。此外，MetaPrompting需要调优整个MLM，由于MLM通常规模庞大，这给计算和存储带来沉重负担。为解决上述问题，我们采用提示池提取更多任务知识，并通过注意力机制构建实例相关的提示。我们进一步提出新型软言语器RepVerb，该言语器可直接从特征嵌入构建标签嵌入。通过融合提示池的元学习与RepVerb，我们提出MetaPrompter实现高效结构化提示。MetaPrompter具有参数高效性，仅需调优提示池。实验结果表明，MetaPrompter性能优于现有最先进方法，且RepVerb在软言语器中表现最佳。