In recent years, there has been significant progress in developing pre-trained language models for NLP. However, these models often struggle when fine-tuned on small datasets. To address this issue, researchers have proposed various adaptation approaches. Prompt-based tuning is arguably the most common way, especially for larger models. Previous research shows that adding contrastive learning to prompt-based fine-tuning is effective as it helps the model generate embeddings that are more distinguishable between classes, and it can also be more sample-efficient as the model learns from positive and negative examples simultaneously. One of the most important components of contrastive learning is data augmentation, but unlike computer vision, effective data augmentation for NLP is still challenging. This paper proposes LM-CPPF, Contrastive Paraphrasing-guided Prompt-based Fine-tuning of Language Models, which leverages prompt-based few-shot paraphrasing using generative language models, especially large language models such as GPT-3 and OPT-175B, for data augmentation. Our experiments on multiple text classification benchmarks show that this augmentation method outperforms other methods, such as easy data augmentation, back translation, and multiple templates.

翻译：近年来，预训练语言模型在自然语言处理领域取得了显著进展。然而，这些模型在小型数据集上进行微调时往往面临挑战。为解决这一难题，研究人员提出了多种适应性方法。提示式调优无疑是主流方法，尤其适用于大型模型。先前研究表明，在提示式微调中加入对比学习是有效的，因为它能帮助模型生成类别间更具区分度的嵌入表示，同时通过同时学习正负样本提高样本效率。对比学习中最关键的要素之一是数据增强，但与计算机视觉不同，自然语言处理领域有效的数据增强仍具挑战性。本文提出LM-CPPF——基于对比释义引导的语言模型提示式微调方法，该方法利用生成式语言模型（特别是GPT-3和OPT-175B等大型语言模型）进行提示式少样本文本释义，从而实现数据增强。我们在多个文本分类基准上的实验表明，这种增强方法优于简易数据增强、回译以及多模板等其他方法。