In this study, we delve into the realm of counterfactual reasoning capabilities of large language models (LLMs). Our primary objective is to cultivate the counterfactual thought processes within LLMs and rigorously assess these processes for their validity. Specifically, we introduce a novel task, Counterfactual Logical Modification (CLOMO), and a high-quality human-annotated benchmark. In this task, LLMs must adeptly alter a given argumentative text to uphold a predetermined logical relationship. To effectively evaluate a generation model's counterfactual capabilities, we propose an innovative evaluation metric, the LogicAware Counterfactual Score to directly evaluate the natural language output of LLMs instead of modeling the task as a multiple-choice problem. Analysis shows that the proposed automatic metric aligns well with human preference. Our experimental results show that while LLMs demonstrate a notable capacity for logical counterfactual thinking, there remains a discernible gap between their current abilities and human performance.

翻译：本研究深入探讨了大型语言模型（LLMs）的反事实推理能力。我们的核心目标是培养LLMs的反事实思维过程，并严格评估这些过程的合理性。具体而言，我们提出了一项新任务——反事实逻辑修改（CLOMO），并构建了一个高质量的人工标注基准。在该任务中，LLMs需熟练调整给定论证文本以维持预定的逻辑关系。为有效评估生成模型的反事实能力，我们提出了一种创新评估指标——逻辑感知反事实评分，直接评估LLMs的自然语言输出，而非将任务建模为多项选择问题。分析表明，所提出的自动指标与人类偏好高度一致。实验结果显示，尽管LLMs展现出显著的逻辑反事实思维能力，但其当前能力与人类表现之间仍存在明显差距。