Abstract Post hoc recalibration of prediction uncertainties of machine learning regression problems by isotonic regression might present a problem for bin-based calibration error statistics (e.g. ENCE). Isotonic regression often produces stratified uncertainties, i.e. subsets of uncertainties with identical numerical values. Partitioning of the resulting data into equal-sized bins introduces an aleatoric component to the estimation of bin-based calibration statistics. The partitioning of stratified data into bins depends on the order of the data, which is typically an uncontrolled property of calibration test/validation sets. The tie-braking method of the ordering algorithm used for binning might also introduce an aleatoric component. I show on an example how this might significantly affect the calibration diagnostics.

翻译：摘要 通过等渗回归对机器学习回归问题的预测不确定性进行事后重校准，可能会给基于分箱的校准误差统计（例如ENCE）带来问题。等渗回归通常会产生分层的不确定性，即数值相同的子集不确定性。将结果数据划分为等量分箱会引入一个随机成分到基于分箱的校准统计估计中。分层数据的分箱划分取决于数据的顺序，而这通常是校准测试/验证集的一个未受控制的属性。用于分箱的排序算法中的断链方法也可能引入随机成分。我通过一个示例展示了这如何可能显著影响校准诊断结果。