Few-shot learning and parameter-efficient fine-tuning (PEFT) are crucial to overcome the challenges of data scarcity and ever growing language model sizes. This applies in particular to specialized scientific domains, where researchers might lack expertise and resources to fine-tune high-performing language models to nuanced tasks. We propose PETapter, a novel method that effectively combines PEFT methods with PET-style classification heads to boost few-shot learning capabilities without the significant computational overhead typically associated with full model training. We validate our approach on three established NLP benchmark datasets and one real-world dataset from communication research. We show that PETapter not only achieves comparable performance to full few-shot fine-tuning using pattern-exploiting training (PET), but also provides greater reliability and higher parameter efficiency while enabling higher modularity and easy sharing of the trained modules, which enables more researchers to utilize high-performing NLP-methods in their research.

翻译：少样本学习与参数高效微调（PEFT）对于应对数据稀缺和语言模型规模持续增长的挑战至关重要。这在专业科学领域尤为突出，研究人员可能缺乏将高性能语言模型微调到精细任务所需的专业知识和资源。本文提出PETapter，这是一种新颖方法，能有效结合PEFT方法与PET风格分类头，在不产生全模型训练典型高计算开销的前提下提升少样本学习能力。我们在三个成熟的NLP基准数据集和一个传播学研究的真实数据集上验证了该方法。实验表明，PETapter不仅在使用模式利用训练（PET）时能达到与全模型少样本微调相当的性能，还具备更高的可靠性、更优的参数效率，同时支持更强的模块化设计与训练模块的便捷共享，这将使更多研究者能在其科研中运用高性能NLP方法。