Understanding causal relationships among the variables of a system is paramount to explain and control its behavior. For many real-world systems, however, the true causal graph is not readily available and one must resort to predictions made by algorithms or domain experts. Therefore, metrics that quantitatively assess the goodness of a causal graph provide helpful checks before using it in downstream tasks. Existing metrics provide an $\textit{absolute}$ number of inconsistencies between the graph and the observed data, and without a baseline, practitioners are left to answer the hard question of how many such inconsistencies are acceptable or expected. Here, we propose a novel consistency metric by constructing a baseline through node permutations. By comparing the number of inconsistencies with those on the baseline, we derive an interpretable metric that captures whether the graph is significantly better than random. Evaluating on both simulated and real data sets from various domains, including biology and cloud monitoring, we demonstrate that the true graph is not falsified by our metric, whereas the wrong graphs given by a hypothetical user are likely to be falsified.

翻译：理解系统中变量间的因果关系对于解释和控制其行为至关重要。然而，对于许多现实世界系统，真实的因果图往往难以直接获取，研究者必须依赖算法或领域专家给出的预测。因此，在使用因果图进行下游任务前，定量评估其质量的度量标准能提供重要的验证依据。现有度量方法通常提供因果图与观测数据间不一致性的绝对数量，但由于缺乏基准参考，实践者难以判断何种程度的不一致性是可接受或可预期的。本研究提出一种通过节点置换构建基准的新颖一致性度量方法。通过将实际不一致性与基准水平进行比较，我们推导出一个可解释的度量指标，用以判断因果图是否显著优于随机图。在模拟数据集及来自生物学、云监控等多个领域的真实数据集上的评估表明，真实因果图不会被本方法证伪，而假设用户给出的错误因果图则很可能被证伪。