Accurately estimating uncertainty is a crucial component of decision-making and forecasting in machine learning. However, existing uncertainty estimation methods developed for IID data may fail when these IID assumptions no longer hold. In this paper, we present a novel approach to uncertainty estimation that leverages the principles of online learning. Specifically, we define a task called online calibrated forecasting which seeks to extend existing online learning methods to handle predictive uncertainty while ensuring high accuracy. We introduce algorithms for this task that provide formal guarantees on the accuracy and calibration of probabilistic predictions even on adversarial input. We demonstrate the practical utility of our methods on several forecasting tasks, showing that our probabilistic predictions improve over natural baselines. Overall, our approach advances calibrated uncertainty estimation, and takes a step towards more robust and reliable decision-making and forecasting in risk-sensitive scenarios.

翻译：准确估计不确定性是机器学习中决策与预测的关键组成部分。然而，现有针对独立同分布（IID）数据开发的不确定性估计方法，在独立同分布假设不再成立时可能失效。本文提出了一种利用在线学习原理的不确定性估计新方法。具体而言，我们定义了一项名为在线校准预测的任务，旨在扩展现有在线学习方法以处理预测不确定性，同时确保高准确性。我们提出了针对该任务的算法，这些算法即使在面对对抗性输入时，也能为概率预测的准确性和校准性提供正式保障。我们通过多个预测任务展示了方法的实际效用，结果表明我们的概率预测优于自然基线方法。总体而言，我们的方法推进了校准不确定性估计，并向风险敏感场景中更稳健、更可靠的决策与预测迈出了一步。