Models trained with counterfactually augmented data learn representations of the causal structure of tasks, enabling robust generalization. However, high-quality counterfactual data is scarce for most tasks and not easily generated at scale. When crowdsourced, such data is typically limited in scale and diversity; when generated using supervised methods, it is computationally expensive to extend to new counterfactual dimensions. In this work, we introduce DISCO (DIStilled COunterfactual Data), a new method for automatically generating high quality counterfactual data at scale. DISCO engineers prompts to generate phrasal perturbations with a large general language model. Then, a task-specific teacher model filters these generations to distill high-quality counterfactual data. While task-agnostic, we apply our pipeline to the task of natural language inference (NLI) and find that on challenging evaluations such as the NLI stress test, comparatively smaller student models trained with DISCO generated counterfactuals are more robust (6% absolute) and generalize better across distributions (2%) compared to models trained without data augmentation. Furthermore, DISCO augmented models are 10% more consistent between counterfactual pairs on three evaluation sets, demonstrating that DISCO augmentation enables models to more reliably learn causal representations. Our repository is available at: https://github.com/eric11eca/disco

翻译：摘要：通过反事实增强数据训练的模型能够学习任务因果结构的表征，从而实现鲁棒的泛化能力。然而，高质量的反事实数据在大多数任务中十分稀缺，且难以大规模生成。当通过众包方式获取时，此类数据通常在规模和多样性上受限；当使用监督方法生成时，扩展到新的反事实维度则计算成本高昂。在本工作中，我们提出了DISCO（蒸馏反事实数据），一种自动生成大规模高质量反事实数据的新方法。DISCO设计提示词，利用大型通用语言模型生成短语扰动，然后由任务特定的教师模型对这些生成结果进行筛选，以提取高质量的反事实数据。尽管方法是任务无关的，我们将其应用于自然语言推理（NLI）任务，并发现：在诸如NLI压力测试等具有挑战性的评估中，使用DISCO生成的反事实数据训练的相对较小的学生模型，与未使用数据增强训练的模型相比，鲁棒性提升6%（绝对值），跨分布泛化能力提升2%。此外，在三个评估集上，DISCO增强模型的反事实对一致性提升10%，表明DISCO增强能够使模型更可靠地学习因果表征。我们的代码仓库见：https://github.com/eric11eca/disco