Many important computer vision applications are naturally formulated as regression problems. Within medical imaging, accurate regression models have the potential to automate various tasks, helping to lower costs and improve patient outcomes. Such safety-critical deployment does however require reliable estimation of model uncertainty, also under the wide variety of distribution shifts that might be encountered in practice. Motivated by this, we set out to investigate the reliability of regression uncertainty estimation methods under various real-world distribution shifts. To that end, we propose an extensive benchmark of 8 image-based regression datasets with different types of challenging distribution shifts. We then employ our benchmark to evaluate many of the most common uncertainty estimation methods, as well as two state-of-the-art uncertainty scores from the task of out-of-distribution detection. We find that while methods are well calibrated when there is no distribution shift, they all become highly overconfident on many of the benchmark datasets. This uncovers important limitations of current uncertainty estimation methods, and the proposed benchmark therefore serves as a challenge to the research community. We hope that our benchmark will spur more work on how to develop truly reliable regression uncertainty estimation methods. Code is available at https://github.com/fregu856/regression_uncertainty.

翻译：许多重要的计算机视觉应用自然地被表述为回归问题。在医学影像中，准确的回归模型有望自动化各种任务，从而有助于降低成本并改善患者预后。然而，这种安全关键的部署要求模型不确定性估计具有可靠性，即使在实践中可能遇到的多种分布漂移下也是如此。基于此，我们着手研究在各类真实世界分布漂移下回归不确定性估计方法的可靠性。为此，我们提出了一个包含8个基于图像的回归数据集的广泛基准，这些数据集具有不同类型的挑战性分布漂移。随后，我们利用该基准来评估多种最常见的不确定性估计方法，以及来自分布外检测任务的两个最新不确定性评分。我们发现，尽管在无分布漂移时这些方法校准良好，但在许多基准数据集上，它们都变得高度过度自信。这揭示了当前不确定性估计方法的重要局限性，因此提出的基准对研究界构成了一项挑战。我们希望我们的基准能激发更多关于如何开发真正可靠的回归不确定性估计方法的工作。代码可在 https://github.com/fregu856/regression_uncertainty 获取。