Observational studies of causal effects require adjustment for confounding factors. In the tabular setting, where these factors are well-defined, separate random variables, the effect of confounding is well understood. However, in public policy, ecology, and in medicine, decisions are often made in non-tabular settings, informed by patterns or objects detected in images (e.g., maps, satellite or tomography imagery). Using such imagery for causal inference presents an opportunity because objects in the image may be related to the treatment and outcome of interest. In these cases, we rely on the images to adjust for confounding but observed data do not directly label the existence of the important objects. Motivated by real-world applications, we formalize this challenge, how it can be handled, and what conditions are sufficient to identify and estimate causal effects. We analyze finite-sample performance using simulation experiments, estimating effects using a propensity adjustment algorithm that employs a machine learning model to estimate the image confounding. Our experiments also examine sensitivity to misspecification of the image pattern mechanism. Finally, we use our methodology to estimate the effects of policy interventions on poverty in African communities from satellite imagery.

翻译：观察性因果效应研究需要对混杂因素进行调整。在表格数据场景中，这些因素被明确定义为独立的随机变量，混杂效应已得到充分理解。然而在公共政策、生态学和医学领域，决策往往基于图像（如地图、卫星或断层扫描图像）中检测到的模式或对象，在非表格化情境中形成。使用此类图像进行因果推断具有独特价值，因为图像中的对象可能与处理变量和结果变量相关。在这些情形中，我们依赖图像来调整混杂，但观测数据并未直接标注重要对象的存在。受现实应用驱动，我们形式化了这一挑战及其处理方法，并明确了识别与估计因果效应的充分条件。通过模拟实验分析有限样本性能，我们采用基于机器学习模型估计图像混杂的倾向性调整算法来估计效应。实验同时检验了图像模式机制误设的敏感性。最后，我们将该方法应用于卫星图像，估计政策干预对非洲社区贫困状况的因果效应。