The design and testing of supervised machine learning models combine two fundamental distributions: (1) the training data distribution (2) the testing data distribution. Although these two distributions are identical and identifiable when the data set is infinite; they are imperfectly known (and possibly distinct) when the data is finite (and possibly corrupted) and this uncertainty must be taken into account for robust Uncertainty Quantification (UQ). We present a general decision-theoretic bootstrapping solution to this problem: (1) partition the available data into a training subset and a UQ subset (2) take $m$ subsampled subsets of the training set and train $m$ models (3) partition the UQ set into $n$ sorted subsets and take a random fraction of them to define $n$ corresponding empirical distributions $\mu_{j}$ (4) consider the adversarial game where Player I selects a model $i\in\left\{ 1,\ldots,m\right\} $, Player II selects the UQ distribution $\mu_{j}$ and Player I receives a loss defined by evaluating the model $i$ against data points sampled from $\mu_{j}$ (5) identify optimal mixed strategies (probability distributions over models and UQ distributions) for both players. These randomized optimal mixed strategies provide optimal model mixtures and UQ estimates given the adversarial uncertainty of the training and testing distributions represented by the game. The proposed approach provides (1) some degree of robustness to distributional shift in both the distribution of training data and that of the testing data (2) conditional probability distributions on the output space forming aleatory representations of the uncertainty on the output as a function of the input variable.

翻译：受监督的机器学习模型的设计与测试结合了两种基本分布:(1) 培训数据分布(2) 测试数据分布。虽然这两个分布在数据集无限时是相同和可识别的;当数据有限(且可能腐败)时,它们不完全为人所知(而且可能不同),而且这种不确定性必须考虑到稳健的不确定性定量(UQ) 。我们提出了这一问题的一般决定-理论制导方法:(1) 将现有数据分成一个培训子集和UQ子集(2) 以一组培训组的分包为单位,并培训美元模式(3) 以美元为单位的模型分配;(3) 将UQ数据集分为一个按美元排序的子集,以随机部分确定相应的实证分布 $\mu ⁇ j} (4) 考虑一个对抗性游戏,即玩家I选择一个模型 $\in\left ⁇ 1, m\\right $, 玩家II 选择UQ 分配的稳健(2) 和玩家I 通过对模型的基分配模式评估 Q 和最佳分配率 混合模型 提供最佳分配战略 (5) 提供最佳分配和最优度测试, 最佳分配模式提供最佳分配战略。