Uncertainty estimation is crucial for machine learning models to detect out-of-distribution (OOD) inputs. However, the conventional discriminative deep learning classifiers produce uncalibrated closed-set predictions for OOD data. A more robust classifiers with the uncertainty estimation typically require a potentially unavailable OOD dataset for outlier exposure training, or a considerable amount of additional memory and compute to build ensemble models. In this work, we improve on uncertainty estimation without extra OOD data or additional inference costs using an alternative Split-Ensemble method. Specifically, we propose a novel subtask-splitting ensemble training objective, where a common multiclass classification task is split into several complementary subtasks. Then, each subtask's training data can be considered as OOD to the other subtasks. Diverse submodels can therefore be trained on each subtask with OOD-aware objectives. The subtask-splitting objective enables us to share low-level features across submodels to avoid parameter and computational overheads. In particular, we build a tree-like Split-Ensemble architecture by performing iterative splitting and pruning from a shared backbone model, where each branch serves as a submodel corresponding to a subtask. This leads to improved accuracy and uncertainty estimation across submodels under a fixed ensemble computation budget. Empirical study with ResNet-18 backbone shows Split-Ensemble, without additional computation cost, improves accuracy over a single model by 0.8%, 1.8%, and 25.5% on CIFAR-10, CIFAR-100, and Tiny-ImageNet, respectively. OOD detection for the same backbone and in-distribution datasets surpasses a single model baseline by, correspondingly, 2.2%, 8.1%, and 29.6% mean AUROC. Codes will be publicly available at https://antonioo-c.github.io/projects/split-ensemble

翻译：不确定性估计对于机器学习模型检测分布外（OOD）输入至关重要。然而，传统判别式深度学习分类器对OOD数据会产生未校准的闭集预测。更鲁棒的具有不确定性估计的分类器通常需要潜在的不可获取的OOD数据集进行异常样本训练，或需要大量额外内存和计算资源构建集成模型。本研究提出了一种替代性Split-Ensemble方法，无需额外OOD数据或推理成本即可改进不确定性估计。具体而言，我们提出了一种新颖的子任务拆分集成训练目标，将通用多类分类任务拆分为多个互补性子任务。每个子任务的训练数据可被其他子任务视为OOD样本，从而能够基于OOD感知目标在不同子任务上训练多样化的子模型。子任务拆分目标使我们能够在子模型间共享底层特征以避免参数和计算开销。特别地，通过从共享骨干模型进行迭代式拆分和剪枝，我们构建了树状Split-Ensemble架构，其中每个分支作为对应子任务的子模型。这可在固定集成计算预算下提升各子模型的准确率和不确定性估计性能。基于ResNet-18骨干网络的实验表明，Split-Ensemble在无额外计算成本的情况下，在CIFAR-10、CIFAR-100和Tiny-ImageNet上相比单模型准确率分别提升0.8%、1.8%和25.5%。在相同骨干网络和分布内数据集上的OOD检测平均AUROC分别超越单模型基线2.2%、8.1%和29.6%。代码将于https://antonioo-c.github.io/projects/split-ensemble 公开。