Counterfactual examples are frequently used for model development and evaluation in many natural language processing (NLP) tasks. Although methods for automated counterfactual generation have been explored, such methods depend on models such as pre-trained language models that are then fine-tuned on auxiliary, often task-specific datasets. Collecting and annotating such datasets for counterfactual generation is labor intensive and therefore, infeasible in practice. Therefore, in this work, we focus on a novel problem setting: \textit{zero-shot counterfactual generation}. To this end, we propose a structured way to utilize large language models (LLMs) as general purpose counterfactual example generators. We hypothesize that the instruction-following and textual understanding capabilities of recent LLMs can be effectively leveraged for generating high quality counterfactuals in a zero-shot manner, without requiring any training or fine-tuning. Through comprehensive experiments on various downstream tasks in natural language processing (NLP), we demonstrate the efficacy of LLMs as zero-shot counterfactual generators in evaluating and explaining black-box NLP models.

翻译：反事实样本常用于许多自然语言处理（NLP）任务的模型开发与评估。尽管已有研究者探索了自动化反事实生成的方法，但这些方法通常依赖于预训练语言模型，并需要在辅助性（通常是任务特定）数据集上进行微调。为反事实生成收集并标注此类数据集需要大量人力，因此在实践中难以实现。为此，本文聚焦于一个全新的问题设定：\textit{零样本反事实生成}。我们提出了一种结构化方法，将大语言模型（LLMs）作为通用型反事实样本生成器。我们假设：近期LLMs具备的指令遵循与文本理解能力，可被有效利用来以零样本方式生成高质量反事实样本，无需任何训练或微调。通过在自然语言处理（NLP）中多个下游任务的综合实验，我们证明了LLMs作为零样本反事实生成器在评估与解释黑盒NLP模型方面的有效性。