Two dominant approaches have emerged for generating probabilistic forecasts of physical systems: generative models, such as diffusion or flow matching; and ensembles of deterministic models with stochasticity injected, trained using the continuous ranked probability score (CRPS) loss. While both approaches have demonstrated strong predictive accuracy, the reliability of their uncertainties has not been systematically assessed. We address this gap by developing a framework to evaluate both approaches across diverse 2D spatiotemporal physical systems, under matched model size and computational budget. We assess the reliability of probabilistic emulation by inspecting the empirical coverage of predictive intervals, while also considering accuracy and computational efficiency metrics. CRPS-trained ensembles typically achieve more reliable uncertainties on both single-step prediction and autoregressive rollouts, demonstrating better coverage than the standard alternative of training generative models in a latent space. Moreover, the CRPS approach offers significantly faster inference. When generative models are trained in ambient rather than a compressed latent space, which is often infeasible for high-dimensional problems, they exhibit comparable coverage to CRPS-trained ensembles, though with substantially larger inference latency. In contrast, when CRPS-trained ensembles are trained in latent space they do not show a marked degradation in coverage with respect to ambient space. Both generative models and CRPS-trained ensembles demonstrate good predictive accuracy. To facilitate future research and application, we release AutoCast, a modular framework implementing both generative models and CRPS-trained ensembles, alongside AutoSim, a flexible dataset generation package for rapid prototyping.

翻译：为物理系统生成概率性预测的两类主流方法已显现：生成模型（如扩散模型或流匹配）与注入随机性的确定性模型集成（采用连续排序概率评分（CRPS）损失训练）。尽管两类方法均展现出卓越的预测精度，但其不确定性预测的可靠性尚未得到系统评估。为填补这一空白，我们构建了一个评估框架，在同等模型规模和计算预算下，对两类方法在多种二维时空物理系统上的表现进行考察。通过审视预测区间经验覆盖率的统计特性，并兼顾精度与计算效率指标，我们评估了概率性仿真的可靠性。研究表明，CRPS训练的集成方法在单步预测与自回归展开中均能生成更可靠的不确定性估计，其覆盖性能显著优于在潜空间训练生成模型的标准方案。此外，CRPS方法在推理速度上具有显著优势。当生成模型在原始空间而非压缩潜空间训练时（这对高维问题往往不可行），其覆盖性能虽可与CRPS训练集成方法相当，但推理延迟大幅增加。相比之下，CRPS训练集成方法切换至潜空间训练时，其覆盖性能较原始空间未出现显著劣化。两类方法均展现出优秀的预测精度。为促进后续研究与应用，我们开源了AutoCast模块化框架（集成生成模型与CRPS训练集成方法），以及支持快速原型开发的灵活数据集生成工具AutoSim。