Recently, prefix-tuning was proposed to efficiently adapt pre-trained language models to a broad spectrum of natural language classification tasks. It leverages soft prefix as task-specific indicators and language verbalizers as categorical-label mentions to narrow the formulation gap from pre-training language models. However, when the label space increases considerably (i.e., many-class classification), such a tuning technique suffers from a verbalizer ambiguity problem since the many-class labels are represented by semantic-similar verbalizers in short language phrases. To overcome this, inspired by the human-decision process that the most ambiguous classes would be mulled over for each instance, we propose a brand-new prefix-tuning method, Counterfactual Contrastive Prefix-tuning (CCPrefix), for many-class classification. Basically, an instance-dependent soft prefix, derived from fact-counterfactual pairs in the label space, is leveraged to complement the language verbalizers in many-class classification. We conduct experiments on many-class benchmark datasets in both the fully supervised setting and the few-shot setting, which indicates that our model outperforms former baselines.

翻译：近年来，前缀微调被提出用于高效地将预训练语言模型适配到广泛的自然语言分类任务中。该方法利用软前缀作为任务特定指示符，并通过语言词语化器作为类别标签的提及，以缩小预训练语言模型与下游任务之间的公式化差距。然而，当标签空间显著增大（即多类分类）时，这种微调技术会面临词语化器歧义问题，因为多类标签由语义相似的短语形式词语化器表示。为解决这一问题，受人类决策过程中对每一样本反复斟酌最模糊类别的启发，我们提出了一种全新的前缀微调方法——反事实对比前缀微调（CCPrefix），专门用于多类分类。该方法利用标签空间中事实-反事实对导出的实例相关软前缀，来补充多类分类中的语言词语化器。我们在全监督和少样本设置下的多类基准数据集上进行了实验，结果表明我们的模型优于以往基线方法。