Few-shot learning-the ability to train models with access to limited data-has become increasingly popular in the natural language processing (NLP) domain, as large language models such as GPT and T0 have been empirically shown to achieve high performance in numerous tasks with access to just a handful of labeled examples. Smaller language models such as BERT and its variants have also been shown to achieve strong performance with just a handful of labeled examples when combined with few-shot learning algorithms like pattern-exploiting training (PET) and SetFit. The focus of this work is to investigate the performance of alternative few-shot learning approaches with BERT-based models. Specifically, vanilla fine-tuning, PET and SetFit are compared for numerous BERT-based checkpoints over an array of training set sizes. To facilitate this investigation, applications of few-shot learning are considered in software engineering. For each task, high-performance techniques and their associated model checkpoints are identified through detailed empirical analysis. Our results establish PET as a strong few-shot learning approach, and our analysis shows that with just a few hundred labeled examples it can achieve performance near that of fine-tuning on full-sized data sets.

翻译：小样本学习——即在有限数据条件下训练模型的能力——在自然语言处理领域日益流行，这是因为GPT、T0等大型语言模型已被实验证明，在仅需少量标注样本的情况下即可在众多任务中取得高性能。即使是BERT及其变体这类较小语言模型，当结合模式利用训练（PET）和SetFit等小样本学习算法时，也能在仅利用少量标注样本的情况下展现强劲性能。本研究旨在探索基于BERT模型的替代性小样本学习方法的表现。具体而言，我们针对一系列训练集规模，在多个BERT系列预训练检查点上对比了普通微调、PET和SetFit三种方法。为便于研究，我们考虑了小样本学习在软件工程中的应用场景。通过详尽的实证分析，我们为每个任务甄别出高性能技术及其对应的模型检查点。研究结果证实PET是一种强大的小样本学习方法，分析表明仅需数百个标注样本，其性能即可接近在完整数据集上进行微调的效果。