In this study, we delve into the realm of counterfactual reasoning capabilities of large language models (LLMs). Our primary objective is to cultivate the counterfactual thought processes within LLMs and rigorously assess these processes for their validity. Specifically, we introduce a novel task, Counterfactual Logical Modification (CLOMO), and a high-quality human-annotated benchmark. In this task, LLMs must adeptly alter a given argumentative text to uphold a predetermined logical relationship. To effectively evaluate a generation model's counterfactual capabilities, we propose an innovative evaluation metric, the LogicAware Counterfactual Score to directly evaluate the natural language output of LLMs instead of modeling the task as a multiple-choice problem. Analysis shows that the proposed automatic metric aligns well with human preference. Our experimental results show that while LLMs demonstrate a notable capacity for logical counterfactual thinking, there remains a discernible gap between their current abilities and human performance.

翻译：本研究深入探讨了大语言模型（LLMs）的反事实推理能力。我们的主要目标是培养LLMs的反事实思维过程，并严格评估这些过程的有效性。具体而言，我们提出了一项新任务——反事实逻辑修改（CLOMO），以及一个高质量的人工标注基准。在该任务中，LLMs必须灵活修改给定的论证文本，以维持预定的逻辑关系。为有效评估生成模型的反事实能力，我们提出了一项创新评估指标——逻辑感知反事实分数（LogicAware Counterfactual Score），用于直接评估LLMs的自然语言输出，而非将任务建模为选择题形式。分析表明，所提出的自动评估指标与人类偏好高度一致。实验结果显示，尽管LLMs展现出显著的逻辑反事实思维能力，但其当前能力与人类表现之间仍存在明显差距。