Recent foundational language models have shown state-of-the-art performance in many NLP tasks in zero- and few-shot settings. An advantage of these models over more standard approaches based on fine-tuning is the ability to understand instructions written in natural language (prompts), which helps them generalise better to different tasks and domains without the need for specific training data. This makes them suitable for addressing text classification problems for domains with limited amounts of annotated instances. However, existing research is limited in scale and lacks understanding of how text generation models combined with prompting techniques compare to more established methods for text classification such as fine-tuning masked language models. In this paper, we address this research gap by performing a large-scale evaluation study for 16 text classification datasets covering binary, multiclass, and multilabel problems. In particular, we compare zero- and few-shot approaches of large language models to fine-tuning smaller language models. We also analyse the results by prompt, classification type, domain, and number of labels. In general, the results show how fine-tuning smaller and more efficient language models can still outperform few-shot approaches of larger language models, which have room for improvement when it comes to text classification.

翻译：近期的基础语言模型在零样本和少样本场景下的多项自然语言处理任务中展现了前沿性能。与基于微调的传统方法相比，此类模型的优势在于能够理解自然语言编写的指令（提示），从而无需特定训练数据即可更好地泛化至不同任务与领域。这一特性使其适用于标注实例有限的文本分类问题。然而，现有研究在规模上存在局限，且尚未明确结合提示技术的文本生成模型与微调掩码语言模型等经典方法在文本分类中的性能差异。本文通过针对16个文本分类数据集（涵盖二元、多类及多标签问题）的大规模评估研究填补了这一空白。具体而言，我们比较了大型语言模型的零/少样本方法与小型语言模型的微调方法，并依据提示类型、分类类别、领域及标签数量对结果展开分析。结果表明，泛化能力更强的小型语言模型经微调后依然能超越大型语言模型的少样本方法——后者在文本分类任务中仍有改进空间。