Language models (LMs) trained on vast quantities of unlabelled data have greatly advanced the field of natural language processing (NLP). In this study, we re-visit the widely accepted notion in NLP that continued pre-training LMs on task-related texts improves the performance of fine-tuning (FT) in downstream tasks. Through experiments on eight single-sentence tasks and eight sentence-pair tasks in both semi-supervised and fully-supervised settings, we find that conventional continued pre-training does not consistently provide benefits and can even be detrimental for sentence-pair tasks or when prompt-based FT is used. To tackle these issues, we propose Prompt-based Continued Pre-training (PCP), which combines the idea of instruction tuning with conventional continued pre-training. Our approach aims to improve the performance of prompt-based FT by presenting both task-related texts and prompt templates to LMs through unsupervised pre-training objectives before fine-tuning for the target task. Our empirical evaluations on 21 benchmarks demonstrate that the PCP consistently improves the performance of state-of-the-art prompt-based FT approaches (up to 20.1% absolute) in both semi-supervised and fully-supervised settings, even with only hundreds of unlabelled examples. Additionally, prompt-based FT with the PCP outperforms state-of-the-art semi-supervised approaches with greater simplicity, eliminating the need for an iterative process and extra data augmentation. Our further analysis explores the performance lower bound of the PCP and reveals that the advantages of PCP persist across different sizes of models and datasets.

翻译：在大量无标注数据上训练的语言模型（LMs）极大地推动了自然语言处理（NLP）领域的发展。在本研究中，我们重新审视了NLP中一个被广泛接受的观点，即对与任务相关的文本进行持续预训练能提升下游任务中微调（FT）的性能。通过在八项单句任务和八项句对任务上，在半监督和全监督两种设置下进行实验，我们发现传统的持续预训练并不总能带来益处，甚至可能对句对任务或使用基于提示的微调时产生负面影响。为解决这些问题，我们提出了基于提示的持续预训练（PCP），将指令调优的思想与传统的持续预训练相结合。我们的方法旨在通过在对目标任务进行微调之前，利用无监督预训练目标向语言模型呈现任务相关文本和提示模板，从而提升基于提示的微调的性能。我们在21个基准上的实证评估表明，PCP能持续提升最先进的基于提示的微调方法的性能（绝对提升高达20.1%），无论是在半监督还是全监督设置下，即使仅使用数百个无标注示例也能实现。此外，结合PCP的基于提示的微调在更简单的条件下超越了最先进的半监督方法，消除了迭代过程和额外数据增强的需求。我们的进一步分析探讨了PCP的性能下限，并揭示了PCP的优势在不同规模的模型和数据集上均持续存在。