Counterfactual scenario modeling exercises that ask "what would happen if?" are one of the most common ways we plan for the future. Despite their ubiquity in planning and decision making, scenario projections are rarely evaluated retrospectively. Differences between projections and observations come from two sources: scenario deviation and model miscalibration. We argue the latter is most important for assessing the value of models in decision making, but requires estimating model error in counterfactual worlds. Here we present and contrast three approaches for estimating this error, and demonstrate the benefits and limitations of each in a simulation experiment. We provide recommendations for the estimation of counterfactual error and discuss the components of scenario design that are required to make scenario projections evaluable.

翻译：提出“如果……会怎样？”的反事实情景建模分析是规划未来最常用的方法之一。尽管情景预测在规划和决策中无处不在，但其预测结果却很少被回溯性评估。预测与观测之间的差异源于两个因素：情景偏差和模型校准失准。我们认为后者对于评估模型在决策中的价值最为关键，但这需要估计反事实世界中的模型误差。本文提出并对比了三种估计此类误差的方法，通过模拟实验展示了每种方法的优势与局限。我们提供了反事实误差估计的建议，并讨论了使情景预测具备可评估性所需的情景设计要素。