Recent years have seen increasing efforts to forecast infectious disease burdens, with a primary goal being to help public health workers make informed policy decisions. However, there has only been limited discussion of how predominant forecast evaluation metrics might indicate the success of policies based in part on those forecasts. We explore one possible tether between forecasts and policy: the allocation of limited medical resources so as to minimize unmet need. We use probabilistic forecasts of disease burden in each of several regions to determine optimal resource allocations, and then we score forecasts according to how much unmet need their associated allocations would have allowed. We illustrate with forecasts of COVID-19 hospitalizations in the US, and we find that the forecast skill ranking given by this allocation scoring rule can vary substantially from the ranking given by the weighted interval score. We see this as evidence that the allocation scoring rule detects forecast value that is missed by traditional accuracy measures and that the general strategy of designing scoring rules that are directly linked to policy performance is a promising direction for epidemic forecast evaluation.

翻译：近年来，传染病负担预测工作日益增多，其主要目标之一是协助公共卫生工作者做出明智的决策。然而，关于主流预测评估指标如何反映基于这些预测的政策有效性，相关讨论仍十分有限。本研究探索了预测与政策之间的一种潜在关联：通过分配有限医疗资源以最小化未满足需求。我们利用各区域疾病负担的概率预测来确定最优资源分配方案，进而根据这些分配方案所允许的未满足需求程度对预测进行评分。我们以美国COVID-19住院人数预测为例进行说明，发现该分配评分规则给出的预测技能排名与加权区间分数给出的排名可能存在显著差异。我们认为这一结果表明，分配评分规则能够检测到传统准确性指标未能捕捉的预测价值，且设计直接关联政策绩效的评分规则这一通用策略，为流行病预测评估指明了有前景的发展方向。