In recent years, there has been significant progress in developing pre-trained language models for NLP. However, these models often struggle when fine-tuned on small datasets. To address this issue, researchers have proposed various adaptation approaches. Prompt-based tuning is arguably the most common way, especially for larger models. Previous research shows that adding contrastive learning to prompt-based fine-tuning is effective as it helps the model generate embeddings that are more distinguishable between classes, and it can also be more sample-efficient as the model learns from positive and negative examples simultaneously. One of the most important components of contrastive learning is data augmentation, but unlike computer vision, effective data augmentation for NLP is still challenging. This paper proposes LM-CPPF, Contrastive Paraphrasing-guided Prompt-based Fine-tuning of Language Models, which leverages prompt-based few-shot paraphrasing using generative language models, especially large language models such as GPT-3 and OPT-175B, for data augmentation. Our experiments on multiple text classification benchmarks show that this augmentation method outperforms other methods, such as easy data augmentation, back translation, and multiple templates.

翻译：近年来，预训练语言模型在自然语言处理（NLP）领域取得了显著进展。然而，当在小数据集上进行微调时，这些模型往往表现不佳。为解决这一问题，研究者提出了各种适配方法。基于提示的调优可以说是最常见的方法，尤其对于较大的模型。先前研究表明，将对比学习加入基于提示的微调中是有效的，因为这能帮助模型生成更具区分性的类别嵌入，同时由于模型同时从正负例中学习，样本效率也更高。对比学习中最重要的组成部分之一是数据增强，但与计算机视觉不同，自然语言处理中实现有效的数据增强仍具有挑战性。本文提出了LM-CPPF，即对比释义引导的基于提示的语言模型微调，利用生成式语言模型（特别是大型语言模型如GPT-3和OPT-175B）进行基于提示的小样本释义以实现数据增强。我们在多个文本分类基准上的实验表明，这种增强方法优于简单数据增强、回译和多重模板等其他方法。