Counterfactual reasoning, a fundamental aspect of human cognition, involves contemplating alternatives to established facts or past events, significantly enhancing our abilities in planning and decision-making. In light of the advancements in current multi-modal large language models, we explore their effectiveness in counterfactual reasoning. To facilitate this investigation, we introduce a novel dataset, C-VQA, specifically designed to test the counterfactual reasoning capabilities of modern multi-modal large language models. This dataset is constructed by infusing original questions with counterfactual presuppositions, spanning various types such as numerical and boolean queries. It encompasses a mix of real and synthetic data, representing a wide range of difficulty levels. Our thorough evaluations of contemporary vision-language models using this dataset have revealed substantial performance drops, with some models showing up to a 40\% decrease, highlighting a significant gap between current models and human-like vision reasoning capabilities. We hope our dataset will serve as a vital benchmark for evaluating the counterfactual reasoning capabilities of models. Code and dataset are publicly available at https://bzhao.me/C-VQA/.

翻译：反事实推理，作为人类认知的基本方面，涉及对既定事实或过去事件的替代性思考，显著增强了我们在规划和决策方面的能力。鉴于当前多模态大语言模型的进步，我们探讨了它们在反事实推理中的有效性。为便于这项研究，我们引入了一个新颖的数据集C-VQA，专门设计用于测试现代多模态大语言模型的反事实推理能力。该数据集通过向原始问题注入反事实前提构造而成，涵盖了数值和布尔查询等多种类型。它结合了真实数据和合成数据，代表了广泛的难度级别。我们利用该数据集对当代视觉语言模型进行的全面评估揭示了显著的性能下降，部分模型下降幅度高达40%，凸显了当前模型与人类视觉推理能力之间的巨大差距。我们希望我们的数据集能成为评估模型反事实推理能力的关键基准。代码和数据集已在https://bzhao.me/C-VQA/上公开。