Despite the remarkable success of large-scale Language Models (LLMs) such as GPT-3, their performances still significantly underperform fine-tuned models in the task of text classification. This is due to (1) the lack of reasoning ability in addressing complex linguistic phenomena (e.g., intensification, contrast, irony etc); (2) limited number of tokens allowed in in-context learning. In this paper, we introduce Clue And Reasoning Prompting (CARP). CARP adopts a progressive reasoning strategy tailored to addressing the complex linguistic phenomena involved in text classification: CARP first prompts LLMs to find superficial clues (e.g., keywords, tones, semantic relations, references, etc), based on which a diagnostic reasoning process is induced for final decisions. To further address the limited-token issue, CARP uses a fine-tuned model on the supervised dataset for $k$NN demonstration search in the in-context learning, allowing the model to take the advantage of both LLM's generalization ability and the task-specific evidence provided by the full labeled dataset. Remarkably, CARP yields new SOTA performances on 4 out of 5 widely-used text-classification benchmarks, 97.39 (+1.24) on SST-2, 96.40 (+0.72) on AGNews, 98.78 (+0.25) on R8 and 96.95 (+0.6) on R52, and a performance comparable to SOTA on MR (92.39 v.s. 93.3). More importantly, we find that CARP delivers impressive abilities on low-resource and domain-adaptation setups. Specifically, using 16 examples per class, CARP achieves comparable performances to supervised models with 1,024 examples per class.

翻译：尽管大规模语言模型（如GPT-3）取得了显著成功，但在文本分类任务中，其性能仍明显低于微调模型。这主要归因于：（1）缺乏处理复杂语言现象（如强化、对比、反讽等）的推理能力；（2）上下文学习中允许的标记数量有限。本文提出提示与推理增强方法（CARP）。CARP采用渐进推理策略，专门针对文本分类中涉及的复杂语言现象：首先引导LLMs寻找表层线索（如关键词、语气、语义关系、指代等），并基于此诱导出诊断性推理过程以做出最终决策。为解决标记数量限制问题，CARP使用监督数据集上的微调模型进行上下文学习中的$k$最近邻（$k$NN）示例搜索，使模型既能发挥LLM的泛化能力，又能利用完整标注数据集提供的任务特定证据。值得注意的是，CARP在5个广泛使用的文本分类基准中的4个上取得了新的最先进（SOTA）性能：SST-2为97.39（+1.24）、AGNews为96.40（+0.72）、R8为98.78（+0.25）、R52为96.95（+0.6），在MR上达到与SOTA相当的性能（92.39对比93.3）。更重要的是，我们发现CARP在低资源和领域适应场景中表现出色：具体而言，每类仅使用16个示例时，CARP即可达到每类1024个示例的监督模型相当的性能。