To provide rigorous uncertainty quantification for online learning models, we develop a framework for constructing uncertainty sets that provably control risk -- such as coverage of confidence intervals, false negative rate, or F1 score -- in the online setting. This extends conformal prediction to apply to a larger class of online learning problems. Our method guarantees risk control at any user-specified level even when the underlying data distribution shifts drastically, even adversarially, over time in an unknown fashion. The technique we propose is highly flexible as it can be applied with any base online learning algorithm (e.g., a deep neural network trained online), requiring minimal implementation effort and essentially zero additional computational cost. We further extend our approach to control multiple risks simultaneously, so the prediction sets we generate are valid for all given risks. To demonstrate the utility of our method, we conduct experiments on real-world tabular time-series data sets showing that the proposed method rigorously controls various natural risks. Furthermore, we show how to construct valid intervals for an online image-depth estimation problem that previous sequential calibration schemes cannot handle.

翻译：为给在线学习模型提供严格的量化不确定性分析，我们构建了一个在线环境中可准确控制风险（如置信区间覆盖率、假阴性率或F1分数）的不确定性集合构造框架，从而将保形预测方法拓展应用于更广泛的在线学习问题。即使底层数据分布随时间发生剧烈波动（甚至对抗性波动），该方法仍能保证用户指定风险水平的有效控制。我们提出的技术具有高度灵活性，可适配任何基础在线学习算法（如在线训练的深度神经网络），实施成本极低且几乎不增加额外计算开销。此外，我们将该方法扩展至多风险同步控制场景，确保生成的所有预测集合对所有给定风险均有效。通过真实世界表格型时间序列数据集的实验验证，该方法能严格调控多种自然风险。同时，我们展示了如何为在线图像深度估计问题构建有效置信区间——这是先前序列校准方案无法处理的场景。