Prompt tuning for pre-trained masked language models (MLM) has shown promising performance in natural language processing tasks with few labeled examples. It tunes a prompt for the downstream task, and a verbalizer is used to bridge the predicted token and label prediction. Due to the limited training data, prompt initialization is crucial for prompt tuning. Recently, MetaPrompting (Hou et al., 2022) uses meta-learning to learn a shared initialization for all task-specific prompts. However, a single initialization is insufficient to obtain good prompts for all tasks and samples when the tasks are complex. Moreover, MetaPrompting requires tuning the whole MLM, causing a heavy burden on computation and memory as the MLM is usually large. To address these issues, we use a prompt pool to extract more task knowledge and construct instance-dependent prompts via attention. We further propose a novel soft verbalizer (RepVerb) which constructs label embedding from feature embeddings directly. Combining meta-learning the prompt pool and RepVerb, we propose MetaPrompter for effective structured prompting. MetaPrompter is parameter-efficient as only the pool is required to be tuned. Experimental results demonstrate that MetaPrompter performs better than the recent state-of-the-arts and RepVerb outperforms existing soft verbalizers.

翻译：预训练掩码语言模型（MLM）的提示调优在少量标注样本的自然语言处理任务中展现了良好性能。该方法针对下游任务调整提示，并通过动词器（verbalizer）建立预测词与标签预测之间的桥梁。由于训练数据有限，提示初始化对提示调优至关重要。近期，MetaPrompting（Hou等人，2022）利用元学习为所有任务特定提示学习共享初始化。然而，当任务复杂时，单一初始化难以获得适用于所有任务和样本的优质提示。此外，MetaPrompting需要调整整个MLM，而MLM通常规模庞大，这给计算和存储带来沉重负担。为解决这些问题，我们采用提示池（prompt pool）提取更多任务知识，并通过注意力机制构建实例相关提示。我们进一步提出新型软动词器（RepVerb），该模块可直接从特征嵌入构建标签嵌入。通过结合元学习的提示池与RepVerb，我们提出MetaPrompter以实现高效结构化提示。MetaPrompter仅需调整提示池，因此参数效率较高。实验结果表明，MetaPrompter的性能优于当前最新方法，且RepVerb的表现超越现有软动词器。