We present a general framework for evaluating image counterfactuals. The power and flexibility of deep generative models make them valuable tools for learning mechanisms in structural causal models. However, their flexibility makes counterfactual identifiability impossible in the general case. Motivated by these issues, we revisit Pearl's axiomatic definition of counterfactuals to determine the necessary constraints of any counterfactual inference model: composition, reversibility, and effectiveness. We frame counterfactuals as functions of an input variable, its parents, and counterfactual parents and use the axiomatic constraints to restrict the set of functions that could represent the counterfactual, thus deriving distance metrics between the approximate and ideal functions. We demonstrate how these metrics can be used to compare and choose between different approximate counterfactual inference models and to provide insight into a model's shortcomings and trade-offs.

翻译：我们提出了一种评估图像反事实的通用框架。深度生成模型的强大功能与灵活性使其成为学习结构因果模型中因果机制的有力工具。然而，这种灵活性导致在一般情况下无法实现反事实的可识别性。受此问题启发，我们重新审视Pearl关于反事实的公理化定义，以确定任何反事实推断模型必须满足的约束条件：组合性、可逆性和有效性。我们将反事实建模为输入变量、其父变量及反事实父变量的函数，并利用公理约束来限制能够表示反事实的函数集合，从而推导出近似函数与理想函数之间的距离度量。我们演示了如何利用这些度量来比较和选择不同的近似反事实推断模型，并揭示模型存在的不足与权衡取舍。