Prompt learning approaches have made waves in natural language processing by inducing better few-shot performance while they still follow a parametric-based learning paradigm; the oblivion and rote memorization problems in learning may encounter unstable generalization issues. Specifically, vanilla prompt learning may struggle to utilize atypical instances by rote during fully-supervised training or overfit shallow patterns with low-shot data. To alleviate such limitations, we develop RetroPrompt with the motivation of decoupling knowledge from memorization to help the model strike a balance between generalization and memorization. In contrast with vanilla prompt learning, RetroPrompt constructs an open-book knowledge-store from training instances and implements a retrieval mechanism during the process of input, training and inference, thus equipping the model with the ability to retrieve related contexts from the training corpus as cues for enhancement. Extensive experiments demonstrate that RetroPrompt can obtain better performance in both few-shot and zero-shot settings. Besides, we further illustrate that our proposed RetroPrompt can yield better generalization abilities with new datasets. Detailed analysis of memorization indeed reveals RetroPrompt can reduce the reliance of language models on memorization; thus, improving generalization for downstream tasks. Code is available in https://github.com/zjunlp/PromptKG/tree/main/research/RetroPrompt.

翻译：提示学习方法在自然语言处理中取得了显著进展，通过诱导更好的少样本性能，但它们仍遵循基于参数的学习范式；学习过程中的遗忘和死记硬背问题可能导致不稳定的泛化问题。具体而言，原始提示学习在全监督训练中可能难以利用非典型实例进行死记硬背，或在低数据量下过度拟合浅层模式。为缓解这些限制，我们开发了RetroPrompt，其动机是将知识与记忆化解耦，以帮助模型在泛化与记忆化之间取得平衡。与原始提示学习不同，RetroPrompt从训练实例中构建一个开放的知识库，并在输入、训练和推理过程中实施检索机制，从而使模型具备从训练语料库中检索相关上下文作为增强线索的能力。大量实验表明，RetroPrompt在少样本和零样本设置下均能获得更好的性能。此外，我们进一步证明所提出的RetroPrompt在新数据集上能产生更好的泛化能力。对记忆化的详细分析确实揭示了RetroPrompt能减少语言模型对记忆化的依赖，从而提升下游任务的泛化能力。代码可在https://github.com/zjunlp/PromptKG/tree/main/research/RetroPrompt获取。