Existing solutions to zero-shot text classification either conduct prompting with pre-trained language models, which is sensitive to the choices of templates, or rely on large-scale annotated data of relevant tasks for meta-tuning. In this work, we propose a new paradigm based on self-supervised learning to solve zero-shot text classification tasks by tuning the language models with unlabeled data, called self-supervised tuning. By exploring the inherent structure of free texts, we propose a new learning objective called first sentence prediction to bridge the gap between unlabeled data and text classification tasks. After tuning the model to learn to predict the first sentence in a paragraph based on the rest, the model is able to conduct zero-shot inference on unseen tasks such as topic classification and sentiment analysis. Experimental results show that our model outperforms the state-of-the-art baselines on 7 out of 10 tasks. Moreover, the analysis reveals that our model is less sensitive to the prompt design. Our code and pre-trained models are publicly available at https://github.com/DAMO-NLP-SG/SSTuning .

翻译：现有零样本文本分类解决方案要么依赖预训练语言模型进行提示学习，但该方法对模板选择较为敏感；要么需要相关任务的大规模标注数据进行元微调。本研究提出一种基于自监督学习的新范式，通过利用无标注数据对语言模型进行微调来解决零样本文本分类任务，称为自监督微调。通过探索自由文本的内在结构，我们提出一种名为"首句预测"的新学习目标，以弥合无标注数据与文本分类任务之间的差距。当模型经过微调学会根据段落其余内容预测首句后，即可对未见任务（如主题分类和情感分析）进行零样本推理。实验结果表明，我们的模型在10项任务中的7项上超越了现有最优基线方法。此外，分析显示我们的模型对提示设计更不敏感。我们的代码和预训练模型已在 https://github.com/DAMO-NLP-SG/SSTuning 公开。