Latest methods for visual counterfactual explanations (VCE) harness the power of deep generative models to synthesize new examples of high-dimensional images of impressive quality. However, it is currently difficult to compare the performance of these VCE methods as the evaluation procedures largely vary and often boil down to visual inspection of individual examples and small scale user studies. In this work, we propose a framework for systematic, quantitative evaluation of the VCE methods and a minimal set of metrics to be used. We use this framework to explore the effects of certain crucial design choices in the latest diffusion-based generative models for VCEs of natural image classification (ImageNet). We conduct a battery of ablation-like experiments, generating thousands of VCEs for a suite of classifiers of various complexity, accuracy and robustness. Our findings suggest multiple directions for future advancements and improvements of VCE methods. By sharing our methodology and our approach to tackle the computational challenges of such a study on a limited hardware setup (including the complete code base), we offer a valuable guidance for researchers in the field fostering consistency and transparency in the assessment of counterfactual explanations.

翻译：最新的可视化反事实解释（VCE）方法利用深度生成模型的能力，合成具有令人印象深刻质量的高维图像新样本。然而，由于评估程序差异较大，且往往局限于对单个样例的视觉检查和小规模用户研究，目前难以对这些VCE方法的性能进行比较。在本工作中，我们提出了一个用于系统性定量评估VCE方法的框架及一组最简指标集。我们利用该框架探索了在自然图像分类（ImageNet）VCE中，最新基于扩散的生成模型中某些关键设计选择的影响。我们进行了一系列类似消融实验的测试，生成了数千个针对多种不同复杂度、准确率和鲁棒性分类器的VCE。我们的发现为VCE方法的未来改进和提升指明了多个方向。通过分享我们的方法论以及如何在有限硬件条件下解决此类研究计算挑战的方法（包括完整代码库），我们为该领域的研究人员提供了宝贵指导，促进了反事实解释评估的一致性和透明度。