The large language models have achieved superior performance on various natural language tasks. One major drawback of such approaches is they are resource-intensive in fine-tuning new datasets. Soft-prompt tuning presents a resource-efficient solution to fine-tune the pre-trained language models (PLMs) while keeping their weight frozen. Existing soft prompt methods mainly focus on designing the input-independent prompts that steer the model to fit the domain of the new dataset. Those methods often ignore the fine-grained information about the task and context of the text. In this paper, we propose a multi-level prompt tuning (MPrompt) method for machine reading comprehension. It utilizes prompts at task-specific, domain-specific, and context-specific levels to enhance the comprehension of input semantics at different granularities. We also propose an independence constraint to steer each domain-specific prompt to focus on information within its domain to avoid redundancy. Moreover, we present a prompt generator that incorporates context-related knowledge in the prompt generation to enhance contextual relevancy. We conducted extensive experiments on 12 benchmarks of various QA formats and achieved an average improvement of 1.94\% over the state-of-the-art methods.

翻译：大规模语言模型在多种自然语言任务上展现了优越性能，但这类方法的主要缺陷是对新数据集进行微调时资源消耗巨大。软提示调优提供了一种资源高效的解决方案，可在保持预训练语言模型参数冻结的同时进行微调。现有软提示方法主要集中于设计与输入无关的提示，引导模型适配新数据集的领域，但往往忽略了任务和文本上下文的细粒度信息。本文提出一种面向机器阅读理解的多层级提示调优方法（MPrompt），该方法利用任务级、领域级和上下文级提示，从不同粒度增强对输入语义的理解。我们还设计了独立性约束机制，引导各领域级提示聚焦其领域内信息以避免冗余。此外，我们提出一个提示生成器，在提示生成过程中融入上下文相关知识以增强上下文相关性。我们在12个不同问答格式的基准数据集上进行了广泛实验，相较于现有最优方法实现了平均1.94%的性能提升。