Counterfactual reasoning, a fundamental aspect of human cognition, involves contemplating alternatives to established facts or past events, significantly enhancing our abilities in planning and decision-making. In light of the advancements in current multi-modal large language models, we explore their effectiveness in counterfactual reasoning. To facilitate this investigation, we introduce a novel dataset, C-VQA, specifically designed to test the counterfactual reasoning capabilities of modern multi-modal large language models. This dataset is constructed by infusing original questions with counterfactual presuppositions, spanning various types such as numerical and boolean queries. It encompasses a mix of real and synthetic data, representing a wide range of difficulty levels. Our thorough evaluations of contemporary vision-language models using this dataset have revealed substantial performance drops, with some models showing up to a 40% decrease, highlighting a significant gap between current models and human-like vision reasoning capabilities. We hope our dataset will serve as a vital benchmark for evaluating the counterfactual reasoning capabilities of models. Code and dataset are publicly available at https://bzhao.me/C-VQA/.

翻译：反事实推理是人类认知的基本特征，涉及对既定事实或过去事件的替代性思考，显著增强了我们的规划与决策能力。鉴于当前多模态大语言模型的进展，我们探究了其在反事实推理中的有效性。为促进这项研究，我们引入了一个新型数据集C-VQA，专门用于测试现代多模态大语言模型的反事实推理能力。该数据集通过向原始问题注入反事实预设条件构建，涵盖数值型与布尔型等多种查询类型，包含真实数据与合成数据的混合样本，并呈现不同难度级别。我们使用该数据集对当前视觉语言模型进行了全面评估，发现其性能出现显著下降，部分模型降幅高达40%，凸显了现有模型与人类视觉推理能力之间的巨大差距。我们期望该数据集能成为评估模型反事实推理能力的重要基准。代码与数据集已公开于https://bzhao.me/C-VQA/。