Evaluating observational estimators of causal effects demands information that is rarely available: unconfounded interventions and outcomes from the population of interest, created either by randomization or adjustment. As a result, it is customary to fall back on simulators when creating benchmark tasks. Simulators offer great control but are often too simplistic to make challenging tasks, either because they are hand-designed and lack the nuances of real-world data, or because they are fit to observational data without structural constraints. In this work, we propose a general, repeatable strategy for turning observational data into sequential structural causal models and challenging estimation tasks by following two simple principles: 1) fitting real-world data where possible, and 2) creating complexity by composing simple, hand-designed mechanisms. We implement these ideas in a highly configurable software package and apply it to the well-known Adult income data set to construct the \tt IncomeSCM simulator. From this, we devise multiple estimation tasks and sample data sets to compare established estimators of causal effects. The tasks present a suitable challenge, with effect estimates varying greatly in quality between methods, despite similar performance in the modeling of factual outcomes, highlighting the need for dedicated causal estimators and model selection criteria.

翻译：评估因果效应的观测性估计器需要罕见可获得的信息：来自目标群体的无混杂干预与结果，这些信息需通过随机化或调整产生。因此，在构建基准任务时通常转而采用模拟器。模拟器提供了强大的控制能力，但往往因过于简化而难以构成具有挑战性的任务——其原因在于它们要么是人工设计且缺乏真实数据的细微特征，要么是在拟合观测数据时未引入结构约束。本研究提出一种通用、可复现的策略，通过遵循两个简单原则将观测数据转化为序列结构因果模型并构建具有挑战性的估计任务：1) 尽可能拟合真实世界数据；2) 通过组合简单的人工设计机制创造复杂性。我们将这些思想实现为一个高度可配置的软件包，并将其应用于著名的Adult收入数据集以构建\tt IncomeSCM模拟器。基于此，我们设计了多个估计任务与抽样数据集，用于比较现有的因果效应估计器。这些任务呈现出适宜的挑战性：尽管在事实结果建模方面表现相近，但不同方法对效应估计的质量差异显著，这凸显了对专用因果估计器与模型选择标准的需求。