Evaluating observational estimators of causal effects demands information that is rarely available: unconfounded interventions and outcomes from the population of interest, created either by randomization or adjustment. As a result, it is customary to fall back on simulators when creating benchmark tasks. Simulators offer great control but are often too simplistic to make challenging tasks, either because they are hand-designed and lack the nuances of real-world data, or because they are fit to observational data without structural constraints. In this work, we propose a general, repeatable strategy for turning observational data into sequential structural causal models and challenging estimation tasks by following two simple principles: 1) fitting real-world data where possible, and 2) creating complexity by composing simple, hand-designed mechanisms. We implement these ideas in a highly configurable software package and apply it to the well-known Adult income data set to construct the IncomeSCM simulator. From this, we devise multiple estimation tasks and sample data sets to compare established estimators of causal effects. The tasks present a suitable challenge, with effect estimates varying greatly in quality between methods, despite similar performance in the modeling of factual outcomes, highlighting the need for dedicated causal estimators and model selection criteria.

翻译：评估因果效应的观测性估计器通常需要难以获取的信息：来自目标群体的无混杂干预与结果数据，这些数据需通过随机化或调整产生。因此，在构建基准任务时通常转而采用模拟器。模拟器虽能提供高度可控的环境，但往往因过于简化而难以构成具有挑战性的任务——这要么源于其人工设计缺乏真实数据的复杂性，要么因为它们在拟合观测数据时未引入结构约束。本研究提出一种通用、可复现的策略，通过遵循两个基本原则将观测数据转化为时序结构因果模型并构建具有挑战性的估计任务：1）尽可能拟合真实世界数据；2）通过组合简单的人工设计机制来创造复杂性。我们将这些理念实现为一个高度可配置的软件包，并将其应用于著名的Adult收入数据集以构建IncomeSCM模拟器。基于此，我们设计了多重估计任务与抽样数据集，用于比较现有的因果效应估计器。这些任务呈现出适宜的挑战性：尽管各方法在事实结果建模方面表现相近，但其因果效应估计质量差异显著，这凸显了对专用因果估计器与模型选择标准的需求。