Large language models show impressive results on few-shot NLP tasks. However, these models are memory and computation-intensive. Meta-training allows one to leverage smaller models for few-shot generalization in a domain-general and task-agnostic manner; however, these methods alone results in models that may not have sufficient parameterization or knowledge to adapt quickly to a large variety of tasks. To overcome this issue, we propose meta-training with demonstration retrieval, where we use a dense passage retriever to retrieve semantically similar labeled demonstrations to each example for more varied supervision. By separating external knowledge from model parameters, we can use meta-training to train parameter-efficient models that generalize well on a larger variety of tasks. We construct a meta-training set from UnifiedQA and CrossFit, and propose a demonstration bank based on UnifiedQA tasks. To our knowledge, our work is the first to combine retrieval with meta-training, to use DPR models to retrieve demonstrations, and to leverage demonstrations from many tasks simultaneously, rather than randomly sampling demonstrations from the training set of the target task. Our approach outperforms a variety of targeted parameter-efficient and retrieval-augmented few-shot methods on QA, NLI, and text classification tasks (including SQuAD, QNLI, and TREC). Our approach can be meta-trained and fine-tuned quickly on a single GPU.

翻译：大型语言模型在少样本自然语言处理任务上表现出色。然而，这些模型对内存和计算资源要求极高。元训练允许利用较小的模型以领域通用且任务无关的方式进行少样本泛化，但仅靠这些方法得到的模型可能缺乏足够的参数化能力或知识来快速适应大量不同任务。为解决此问题，我们提出基于演示检索的元训练方法，使用密集段落检索器为每个示例检索语义相似的带标签演示，以获得更多样化的监督信号。通过将外部知识与模型参数分离，我们可利用元训练训练参数高效的模型，使其在更广泛的任务上具有良好的泛化能力。我们从UnifiedQA和CrossFit构建元训练集，并基于UnifiedQA任务提出演示库。据我们所知，本工作是首次将检索与元训练结合，使用DPR模型检索演示，并同时从多个任务中利用演示，而非从目标任务的训练集中随机采样演示。我们的方法在问答（包括SQuAD）、自然语言推理（QNLI）和文本分类（TREC）等任务上，优于多种针对性的参数高效和检索增强少样本方法。该方法可在单个GPU上快速完成元训练和微调。